RESCUE: Ranking LLM Responses with Partial Ordering to Improve Response Generation (2024)

Abstract: Customizing LLMs for a specific task involves separating high-quality responses from lowerquality ones. This skill can be developed using supervised fine-tuning with extensive human preference data. However, obtaining a large volume of expert-annotated data is costly for most tasks. In this paper, we explore a novel method to optimize LLMs using ranking metrics. This method trains the model to prioritize the best responses from a pool of candidates created for a particular task. Rather than a traditional full ordering, we advocate for a partial ordering, as achieving consensus on the perfect order of candidate responses can be challenging. Our partial ordering is more robust, less sensitive to noise, and can be achieved with limited human annotations or through heuristic methods. We test our system’s improved response generation ability using benchmark datasets, including textual entailment and multi-document question answering. We conduct ablation studies to understand crucial factors, such as how to gather candidate responses for a specific task, determine their most suitable order, and balance supervised fine-tuning with ranking metrics. Our approach, named RESCUE, offers a promising avenue for enhancing the response generation and task accuracy of LLMs.1

heading: 1 Introduction
text: A significant advantage of large language models is their ability to explain their predictions (Ziegler et al., 2020; Vafa et al., 2021; Alkhamissi et al., 2023; Ludan et al., 2023; Li et al., 2023; Ye et al., 2023). For example, LLMs may suggest lab tests to doctors based on patient symptoms (Peng et al., 2023) or help financial analysts evaluate risks in their investment portfolios (Romanko et al., 2023), providing explanations for each. As LLMs increasingly assist in decision-making across domains, examining the quality of their explanations becomes
1Our code and models are available at: https://github. com/ekonwang/RRescue.
crucial. Previous studies suggest that lower-quality model explanations can lead to misunderstandings and diminish user trust (Burns et al., 2022; Turpin et al., 2023; Reingold et al., 2024). Therefore, it is imperative to improve LLMs’ explanation quality, along with enhancing their task accuracy.
We focus on LLM responses that consist of a predicted label and a detailed explanation. LLMs should provide not only accurate labels but also sound rationales to support their predictions. Explanations can be generated using methods such as chain- or tree-of-thoughts and self-reflection (Yao et al., 2022; Wei et al., 2023; Yao et al., 2023; Shinn et al., 2023; Asai et al., 2024). Explanations can also be embedded in prompts to guide LLMs in new tasks via in-context learning (Ye et al., 2023). In this paper, we advance the research by investigating methods to train an open-source LLM to effectively rank candidate responses, which we gather from
365
various sources. Learning to rank responses allows the LLM to differentiate between sound and flawed explanations for a specific task, thereby enhancing response generation.
Interestingly, accurate model predictions often come with high-quality explanations. Studies have shown that when LLMs are confident in their responses, they not only provide accurate answers but also offer solid justifications. On the flip side, when they’re uncertain, their explanations can falter or be completely hallucinated (Singh et al., 2023; Liu et al., 2023a; Sun et al., 2024). Our paper builds on this insight to rank candidate responses. We place human responses above model responses with correct labels, which in turn are prioritized over incorrect responses. This hierarchy encourages the LLM to generate explanations comparable to humans’ or, at the very least, to produce explanations that lead to accurate labels.
Our method benefits from requiring minimal expert annotations, which is a frequent challenge in most domain-specific tasks. Unlike reinforcement learning with human feedback (RLHF; Ziegler et al., 2020) or direct preference optimization (DPO; Rafailov et al., 2023), which need extensive expertannotated data, our approach is cost-effective and practical in resource-constrained situations. We employ a partial ordering of LLM responses, which can be acquired with limited human annotations or through heuristic functions. This study’s contributions are summarized as follows:
• We seek to improve LLMs’ response generation. In training, we supplement each example with candidate responses, featuring a mix of accurate and inaccurate predictions, and sound and flawed explanations. For tasks with long contexts, we anchor responses in different parts of the context to increase diversity. LLM is trained to prioritize the best responses using the ranking metric.
• We test our system’s response generation using multiple benchmarks, and conduct ablation studies to understand crucial factors, such as how to gather candidate responses, determine their most suitable order, and balance supervised fine-tuning with ranking metrics. Our approach, named RESCUE, offers a promising avenue for enhancing the response generation and task accuracy of LLMs.

heading: 2 Related Work
text: Learning from Human Preferences Aligning LLM responses with human preferences ensures
the models’ outputs are helpful, safe, and adhere to societal norms (Bai et al., 2022b; Liu et al., 2023b; Honovich et al., 2023; Wang et al., 2023; Rafailov et al., 2023; Hejna et al., 2023; Hu et al., 2023). This research often involves humans performing pairwise or k-wise comparisons on model outputs, which are used to train a reward model (Bai et al., 2022a; Ouyang et al., 2022; Ramamurthy et al., 2023; Zhu et al., 2023). Moreover, Rafailov et al. (2023) optimize the LLM directly based on preference data, eliminating the need for a separate reward model. Liu et al. (2024) collect preference data from the target optimal policy through rejection sampling. Unlike other methods, we guide LLMs to make accurate predictions and generate reliable explanations with minimal human annotations for domain-specific tasks.
Reasoning LLMs can improve their reasoning through trial and error and self-improvement (Wei et al., 2023; Burnell et al., 2023; Zheng et al., 2023; Hu et al., 2024a,b; Cheng et al., 2024; Ahn et al., 2024; Wang and Zhou, 2024). For example, chainof-thought (Wei et al., 2023) allows LLMs to break down complex tasks step by step into more manageable parts. Tree-of-thoughts (Yao et al., 2023) employs task decomposition via a tree structure, guiding LLMs through various steps and consider multiple thoughts within each step. Reflexion (Shinn et al., 2023) combines dynamic memory and selfreflection to refine reasoning skills. However, pinpointing specific reasoning errors remains a practical challenge. The distinction between sound and flawed explanations can often be subtle and unclear during self-reflection.
Ranking Metrics A ranking objective allows the model to prioritize the best candidates (Yuan et al., 2023; Song et al., 2024), improving its performance in tasks like abstractive summarization and question answering. For example, the BRIO training paradigm (Liu et al., 2022) fine-tunes BART and T5 models to generate reference summaries while using a ranking mechanism to score candidate summaries. This approach could be especially beneficial in retrieval augmented generation (Hopkins and May, 2011; Lewis et al., 2021; Nakano et al., 2022; Hou et al., 2024). We believe that explanations grounded on incorrect documents should be discounted and those grounded in reference documents be promoted. Our method leverages this insight to enhance the model’s ability to generate contextually accurate explanations.

heading: 3 Our Approach: RESCUE
text: Let x ∼ D represent the prompt or context given to the model, and y denote the model’s response to prompt x. The response y comprises two parts: a brief justification and a predicted label, separated by the special symbol ‘####’. For example, in the natural language inference task, it might be “Unusual clothing and striped clothing could be the same. #### Entailment.” Supervised fine-tuning (SFT; Eq. (1)) is a primary method to improve task accuracy by training the model to generate humanwritten responses y∗. However, since the model has only been exposed to high-quality human responses, its noise robustness remains unvalidated. Prior studies (Ziegler et al., 2020; Touvron et al., 2023) suggest that model performance can plateau quickly, potentially leading to overfitting.
LSFT(θ) = − log πθ(y∗|x) (1)
We proposed to guide the model to prioritize valid responses over flawed ones and contextually accurate responses over inaccurately grounded ones, using a ranking metric as illustrated in Eq. (2). Here, (x, y0, y1, b) ∼ S includes a prompt x, two candidate responses, and a binary variable b, where yb should be scored higher than y1−b. S represents a diverse set of candidate responses obtained from various sources. For example, responses could be acquired from open-source LLMs like Llama-2/3 or close-source LLMs like GPT-3.5, GPT-4 or Claude. Human-annotated responses can also be included in the collection when they are available.
LRank(θ) = −E(x,y0,y1,b)∼S [ (2) max{0, log πθ(yb|x)− log πθ(y1−b|x)} ]
We initiate πθ(y|x) from a base model ρ(y|x) and subsequently fine-tune it for a specific task with candidate responses. Particularly, πθ(y|x) is used to loosely represent length-normalized probability πθ(y|x) = 1|y|λ ∑|y| t=1 log πθ(yt|x, y<t), where λ > 0 is the scaling factor for length normalization. Following Yuan et al. (2023), our approach uses α to balance the impact of supervised fine-tuning and the ranking metric, as shown in Eq. (3).
LRESCUE(θ) = LSFT(θ) + αLRank(θ) (3)
Ranking Metrics vs. Rewards A reward model r(x, yi) assigns scores to a given prompt x and its corresponding response yi. As shown in Eq. (4), it allocates the full probability mass to the response
yb chosen by human labelers. For this model to function, humans need to provide accurate pairwise preference judgments. Nonetheless, achieving a consensus among human labelers regarding the perfect order of LLM responses can be a daunting task. The labelers often struggle to provide consistent, fine-grained labels (Touvron et al., 2023). As a result, allocating the entire probability mass, i.e., logPθ(yb′ |x) to an incorrectly labeled response yb′ can mislead the model and hinder the effective training of the reward model.
LReward(r) =− E(x,{yi}i,b)∼S [ log
er(x,yb)∑ i e r(x,yi)
]
(4)
In contrast, our proposed ranking metrics offer greater flexibility and robustness to inconsistencies in human preferences. Our model not only prioritizes yb over other potential responses using the equation max{0, logPθ(yb|x)− logPθ(y1−b|x)}, but further allows minor deviations. For example, the model can still assign a high probability to a less-favored response logPθ(y1−b|x), provided its probability difference from the top response logPθ(yb|x) − logPθ(y1−b|x) remains minimal. We also advocate for a partial ordering of LLM responses, partitioning them into groups. This group ordering provides a hierarchical perspective, enabling the model to understand the relative importance of each group in a broader context.

heading: 4 Ranking LLM Responses
text: Candidate responses for a given prompt x, can be organized into a strict order. OpenAI has employed a team of trained human labelers to rank sets of model outputs from best to worst to train a reward model (Ziegler et al., 2020; Ouyang et al., 2022). However, this method is quite expensive. We propose two cost-effective approaches to establish a Partial Ordering (PO) of responses.
Our first method, (PO) Human Prioritization, posits that human responses should take priority over model responses, as they offer valid rationales and accurate labels. (PO) Label Prioritization places responses with correct labels above those with incorrect labels, irrespective of whether they are human or model-generated. This is because rationales resulting in correct labels are more valuable than those leading to incorrect labels. The latter may contain flawed reasoning that misguides their predictions. Lastly, (PO) Human-Label Hy-
brid employs a fine-grained grouping. It places human responses above model responses with correct labels, which are then prioritized over responses with incorrect labels. This hierarchy is designed to motivate the LLM to generate rationales comparable to humans’ or, at a minimum, to produce rationales that lead to accurate labels.
Partial Orderings (PO) of responses offer enhanced flexibility and noise robustness. For example, in developing Llama-2, Touvron et al. (2023) noted that even human labelers struggle to decide between two similar model responses, with annotations for such responses often hinging on subjective judgement and nuanced details. By utilizing a partial order, we only incorporate the most clearcut pairs of model outputs in the ranking metric, thereby improving the quality of response pairs used in model fine-tuning.
For comparison, we examine two full ordering (FO) approaches. (FO) Similarity embeds each candidate response into a vector, which are then ranked based on their Cosine similarity to the vector representing the human response. The second approach (FO) GPT-3.5-Turbo leverages the GPT3.5-Turbo-0613 model to rank candidate responses. We instruct it to prioritize candidates with the same labels as the human response, but allowing it to decide whether this criterion is met. We compare full and partial ordering approaches in §6.

heading: 5 Collecting Candidate Responses
text: We enrich each example with a set of candidate responses, targeting a mix that includes both accurate and inaccurate predictions, along with explanations that are both sound and flawed. We incorporate human annotations into the mix when available. For tasks with long contexts, we anchor responses in different parts of the context to increase diversity. This enriched dataset is used to train our LLM to improve its response generation. Next, we outline two strategies for generating candidate responses.

heading: 5.1 Responses Generated by Various LLMs
text: We focus on the textual entailment task (Bowman et al., 2015; Chen et al., 2017; Camburu et al., 2018; Kumar and Talukdar, 2020) to illustrate our strategy. Specifically, the Stanford NLI dataset identifies relationships between sentence pairs as entailment, contradiction, or neutral. The e-SNLI dataset expands on SNLI by adding human-annotated explanations for these relationships, explaining why sentences are classified in certain ways (Camburu
et al., 2018). Similarly, we require LLMs to both predict and rationalize their predictions. Our approach then learns to prioritize accurate predictions and their model explanations, while downplaying explanations for inaccurate predictions.
We gather diverse responses for this task from both open-source and proprietary LLMs. Specifically, we sample three responses from Llama-2-7b, setting the temperature to 0.8 for diversity, and one from GPT-3.5-Turbo-0613, plus a human explanation, making five responses per prompt in total. Each response features a brief explanation of the model’s reasoning and a predicted label, as shown in Figure 1.
Response Flipping We propose a novel method for collecting diverse responses from LLMs without the need for repetitive response sampling. Our method begins by inverting an LLM’s explanation for a given response. For instance, if an LLM suggests, “The to-go packages may not be from lunch. #### Neutral,” we flip the explanation to, “The togo packages are likely from lunch.” This reversed explanation then guides the LLM to assign a new label, such as “#### Entailment.”
Our method uses GPT-4-0613 for reversing the explanations, given its extraordinary generation capabilities. The prompt for inversion is: “Rewrite the sentence to convey the opposite meaning: {Explanation}.” Afterward, GPT-3.5-Turbo-0613 is used to predict the appropriate label by combining the original context with the inverted explanation. This method offers an efficient way to generate diverse responses with varying labels.

heading: 5.2 Responses Anchored in Various Passages
text: When dealing with long contexts, we can anchor responses in different parts of the context to produce a diverse set of answers. An LLM can then enhance its performance by discriminating among these answers. For example, in the multi-document question answering task (Multi-doc QA; Liu et al. 2023b), the LLM uses 10 to 30 Wikipedia passages as input to answer questions. These questions come from NaturalQuestions-Open (Kwiatkowski et al., 2019), which contains historical Google queries and their human-annotated answers extracted from Wikipedia. Among the passages given to the model, only one has the answer, the rest are distractors. A retrieval system named Contriever (Izacard et al., 2022) is used to obtain distractor passages, which are most relevant to the question but do not contain the answers.
We use Llama-2-7b to generate five diverse candidate responses per instance, one from the gold passage and four from random distractors. Responses containing the desired answer are marked correct, as illustrated in Figure 2. Here, we generate two candidate responses “The first Nobel Prize in Physics was awarded in 1901 to Wilhelm Conrad Röntgen for his discovery of X-rays.” and “Ernest Lawrence was the recipient of the first Nobel Prize in Physics.” by feeding the model Documents [2] and [3] separately. Our Label-Prioritized approach ranks candidates with the desired answer higher than those without. Human-Label-Hybrid further prefers correct answers anchored in the gold passage. In training, the model receives a question and 10 Wikipedia passages, and learns to differentiate correct from incorrect responses. At test time, the fine-tuned model employs beam search to decode the optimal response.

heading: 6 Experiments
text: We have chosen Llama-2-7b as our base model for task-specific training. The Llama-2 series outperforms other open-source options, such as Falcon (Almazrouei et al., 2023), Mistral (Jiang et al., 2023), Vicuna (Chiang et al., 2023) and MPT (MosaicML, 2023), on a number of tasks. Its 7b variant requires significantly less GPU memory, which is crucial for specific domains without the specialized infrastructure to serve larger models.2
2We leave the extension to other models such as Llama-3 for future work.
We use AdamW (Loshchilov and Hutter, 2017) with a learning rate of 2e−5 and a cosine scheduler with a 0.03 warmup rate. Our training utilizes fully sharded data parallelism and BF16 mixed precision training, which is generally faster, consumes less memory, and is preferable for large models. Our experiments are conducted using 4xA100 GPUs, and task-specific training is limited to a single epoch for both supervised fine-tuning and response ranking. This is to mitigate the risk of multi-epoch degradation (Xue et al., 2023) and potential overfitting from repeated exposure to the training data. The batch size is set at B=64, the same configuration used for LLama-2 (Touvron et al., 2023). It is calculated as the product of three factors, B = g×b×D, combining gradient accumulation steps (g = 16), per-GPU batch size (b = 1 due to memory constraints), and the number of GPUs (D = 4). This strategy allows us to handle a large number of candidates during response ranking.

heading: 6.1 Automatic Evaluation of NLI Accuracy
text: Our goal in this study is to enhance response generation with limited training data, which is a common challenge in real-world scenarios where expert annotations are scarce, often limited to a few thousand examples. We conduct our experiments using the e-SNLI dataset (Camburu et al., 2018), which comprises 549,367 training examples. We intentionally restrict our training to subsets of {2k, 5k, 10k, 20k} samples, approximately 0.4% to 3.6% of the total training set. We report the accuracy of all models on the standard test set of 9,824 examples.
We evaluate a variety of models on this task. In particular, we train the base model with human responses (SFT). We also explore two response rank-
ing strategies: full ordering (FO), which ranks candidate model responses by their semantic closeness to human responses (Similarity) or as assessed by GPT-3.5-Turbo, and partial ordering (PO), which trains the base model to prioritize human responses over those from models (Human Prioritization), responses with correct labels over incorrect ones (Label Prioritization), and a mix of both (HumanLabel Hybrid). Both FO and PO rely on our ranking metric detailed in Eq.(3).
Table 1 presents task accuracy across various proportions of training data. We observe that models trained with ranking metrics and incorporating both full and partial ordering strategies outperform those trained solely with SFT, especially when working with a few thousand annotated examples. This indicates that training an LLM to rank responses can improve response generation and result in more accurate predictions of textual entailment relationships. The improvement is most notable when using only 0.4% of the total training data, suggesting the advantage of ranking metrics in scenarios with extremely scarce training data.
Our partial ordering strategies, namely label prioritization and a hybrid of human and label prioritization, surpass full ordering methods. This could be because achieving consensus on full ordering of responses is challenging even for humans. This approach may introduce variability in response ranking and destabilizes training. SFT begins to show improvement with 20k or more training examples, although gathering such extensive annotations is often difficult for domain-specific tasks. Additionally, while flipping responses increases answer variety, it might cause a shift in the distribution of ranked responses. We find this technique consistently improves response generation only when training data is limited to 2k examples.
Our models match state-of-the-art performance. E.g., Hsieh et al. (2023) achieved 89.51% accuracy using a 540B LLM with step-by-step distilling. By contrast, our models use only a fraction of the full training set with a 7B model. Without supervised fine-tuning, the base Llama-2-7b model yields a significantly lower accuracy of 33.31%. Next, we extend our evaluation to include human assessment of model explanations.

heading: 6.2 Human Evaluation of Response Quality
text: Human evaluation provides a holistic assessment of model responses. We compare several models, including our PO method with label prioritization, SFT, FO method with responses ranked their similarity to human responses, PO model with response flipping, and the base model. These models were trained with varying amounts of training data (0.4% to 3.6%), and the highest performing model across all data proportions was chosen for human evaluation. An annotator evaluated responses for 100 randomly selected samples from the e-SNLI test set, using win, tie and lose to rate each response pair. Evaluations were based on label accuracy and the quality of explanations. A quality explanation should support the predict label with detailed reasoning and show logical coherence.
As Figure 3 illustrates, our partial ordering (PO) with label prioritization outperforms the SFT model with an overall win rate of 47%. This advantage stems from the PO models’ ability to distinguish between sound and flawed responses, thus improving response generation. While SFT shows comparable accuracy in automatic evaluation, it often relies on data artifacts for predictions (Gururangan et al., 2018) and does not yield better explanations. Similar to findings from automatic evaluations, adding response flipping does not surpass the original label
prioritization method. Our PO method also outperforms other methods such as FO Similarity and the base Llama-2-7b model.

heading: 6.3 Evaluation of Multi-Document QA
text: The Multi-Doc QA task involves answering a given question using a set of retrieved documents. Liu et al. (2023c) found that LLMs exhibit a U-shaped curve, depending on where the answer-containing document is located within the input context and highlighting difficulties in accessing relevant information in the middle of long contexts. To mitigate this, we incorporate response ranking. We generate five candidate responses per question, one from the correct document and four from distractors. We then train the base model on 1k examples from the training set using our ranking metric (Eq. (2)). SFT is not used due to the absence of human-written explanations for this task. Our method is evaluated on a test set of 665 examples.
Table 2 shows answer accuracy, measured as whether correct answers from the NaturalQuestions annotations appear in the generated responses. We evaluate two scenarios: the model receives 5 or 10 documents returned by the retriever. The correct document is placed either at the beginning (1st position), in the middle (3rd or 5th), or at the end (5th or 10th) of the document set. We find that the PO method with label prioritization substantially improves model performance, as ranking responses al-
lows the LLM to more effectively identify relevant information, improving the U-shaped curve. Our findings also align with those of Liu et al. (2023c), who observed a recency bias in Llama-2-7b. With 20 documents as input, they reported accuracies of about 25% at positions 1, 5, 10, 15, and 42% at position 20. Upon examining the model’s responses, we observe that the model often answers questions by copying content, which tends to improve answer accuracy when the answer is located in the middle or end of the context.

heading: 7 Discussion
text: Balancing Coefficient Our approach uses a hyperparameter α to balance the impact of supervised fine-tuning and the ranking metric. Figure 4 shows the influence of different α on task accuracy. We find that optimal performance is achieved with an α value between 0.01 to 0.1. Our results indicate that, while supervised fine-tuning is pivotal for RESCUE, integrating the ranking metric enhances the method’s robustness to noise.
Number of Candidate Responses We conduct experiments with a varying number of candidate responses per prompt, and the results are shown in Figure 4. In our experiments, we are able to rank up to five candidate responses using four Nvidia A100 GPUs. As the number of candidates increases, so does the demand for additional GPU memory and
compute resources. Our experiments indicate that performance improvement can be achieved even with 3-4 candidate responses. Beyond that, RESCUE sees no further gains from increasing the number of responses. This saturation in performance may be attributed to the noise in ranking. Moreover, it highlights the challenges associated with ranking a diverse set of responses differing in length and style of explanations.
Scoring Candidate Responses We identify two characteristics in human responses that distinguish them from model responses. Firstly, they are more concise and to the point. As indicated in Figure 6 (RIGHT), human responses are significantly shorter, averaging 10 fewer tokens per response compared to GPT-3.5’s responses. Secondly, we note that LLM responses tend to use more common words, yielding better fluency and generally smoother text compared to human responses. These characteristics present challenges in ranking responses from diverse sources. Human responses, due to their brevity and unique word choice, often have lower length-normalized log probabilities than model responses. This discrepancy leads to many margin violations during training using Eq. (2), and more
parameter updates to ensure human responses score higher than model outputs.
To mitigate this, we assign a length scaling factor λ of 0.85 to all model responses, including those from Llama-2-7b and GPT-3.5-turbo-0613, maintaining a λ of 1.0 for human responses. This effectively shifts the log probability score distributions for model responses closer to human ones (Figure 5), reducing margin violations. We are also exploring adjusting the margin size and curriculum learning, which gradually increases the difficulty of training samples to reduce violations, as potential directions for future research.
Central Tendency Bias LLMs such as Llama2-7b and GPT-3.5 exhibit a central tendency bias (Goldfarb-Tarrant et al., 2020) in natural language inference. These models often predict Neutral labels, leaning towards the “center” of possible labels. Figure 6 presents the confusion matrix, with the x-axis representing predicted labels by Llama2-7b and the y-axis showing human labels. The results show Llama-2-7b’s tendency to predict neutral labels (indicated by the dark bar in the middle) and its avoidance of extreme labels like Entailment or Contradiction. A plausible reason could
be Llama-2-7b’s inadequate world knowledge impacting its task accuracy. Moreover, this tendency might originate from the models being trained on human annotations for instruction-following. They frequently give hedging responses to fulfill helpfulness and safety requirements, leading to outputs that are more neutral and less assertive.

heading: 8 Conclusion
text: In this paper, we introduce RESCUE, an approach that trains the LLM to prioritize sound responses over erroneous ones, thereby enhancing overall task accuracy and the quality of explanations. Accurate model predictions often come with high-quality explanations. We build on this insight to rank candidate responses using a partial ordering approach, as achieving consensus on the perfect order of responses is challenging. RESCUE has demonstrated competitive performance on benchmarks.

heading: Acknowledgements
text: We would like to thank the reviewers for their insightful feedback, which greatly enhanced our paper. HL and FL are supported in part by National Science Foundation grant IIS-2303678.
Limitations
Our approach focuses on optimizing LLMs through ranking metrics and partial ordering of candidate responses. We introduce two innovative strategies for generating candidates: collecting from diverse LLMs and anchoring responses in various parts of the context, showcasing its flexibility across benchmark datasets. We note that organizing candidate responses can benefit from domain-specific criteria, such as sorting recommended lab tests for patients by the relevance of the answer, urgency, and cost. Further, our proposed approach prioritizes the best responses from a set of candidates, thereby improving the task accuracy and the quality of generated explanations. With additional GPU resources, we can improve the variety and representation of candidate responses or categorize them based on domainspecific attributes. Despite existing challenges, our approach offers a promising path for customizing LLMs for specialized applications.

heading: Future Work
text: We are also exploring adjusting the margin size and curriculum learning, which gradually increases the difficulty of training samples to reduce violations, as potential directions for future research. Central Tendency Bias LLMs such as Llama2-7b and GPT-3. 5 exhibit a central tendency bias (Goldfarb-Tarrant et al. , 2020) in natural language inference. These models often predict Neutral labels, leaning towards the “center” of possible labels. Figure 6 presents the confusion matrix, with the x-axis representing predicted labels by Llama2-7b and the y-axis showing human labels. The results show Llama-2-7b’s tendency to predict neutral labels (indicated by the dark bar in the middle) and its avoidance of extreme labels like Entailment or Contradiction. A plausible reason could
be Llama-2-7b’s inadequate world knowledge impacting its task accuracy. Moreover, this tendency might originate from the models being trained on human annotations for instruction-following. They frequently give hedging responses to fulfill helpfulness and safety requirements, leading to outputs that are more neutral and less assertive