Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understanding (2024)

Abstract: Recent strides in large language models (LLMs) have yielded remarkable performance, leveraging reinforcement learning from human feedback (RLHF) to significantly enhance generation and alignment capabilities. However, RLHF encounters numerous challenges, including the objective mismatch issue, leading to suboptimal performance in Natural Language Understanding (NLU) tasks. To address this limitation, we propose a novel Reinforcement Learning framework enhanced with Label-sensitive Reward (RLLR) to amplify the performance of LLMs in NLU tasks. By incorporating label-sensitive pairs into reinforcement learning, our method aims to adeptly capture nuanced label-sensitive semantic features during RL, thereby enhancing natural language understanding. Experiments conducted on five diverse foundation models across eight tasks showcase promising results. In comparison to Supervised Fine-tuning models (SFT), RLLR demonstrates an average performance improvement of 1.54%. Compared with RLHF models, the improvement averages at 0.69%. These results reveal the effectiveness of our method for LLMs in NLU tasks. Code and data available at: https://github.com/MagiaSN/ACL2024_RLLR

heading: 1 Introduction
text: Large language models (LLMs) (Achiam et al., 2023; Chowdhery et al., 2023; Touvron et al., 2023a) have undergone impressive advancements that transform NLP tasks into a unified text-totext paradigm, achieving robust alignment and generation capabilities through reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Bai et al., 2022a). Particularly, models are required to predict the correct labels in natural language understanding (NLU) tasks, distinct from
*These authors contributed equally to this work. †Corresponding author.
natural language generation (NLG) tasks. Numerous studies have employed “rationales” to assist LLMs with Chain-of-Thought (CoT) prompting during supervised fine-tuning (SFT) stage (Kim et al., 2023; Hsieh et al., 2023). Rationale refers to the relevant parts or information that provide explanations or support for the predictions or decisions made by a model.
However, Lambert and Calandra (2023) detail a fundamental challenge in RLHF learning schemes: the objective mismatch issue. This arises when the reward model is influenced by human preference data, introducing biases that conflict with downstream evaluation metrics, especially when applied to NLU tasks. In RLHF, comparison data is initially sampled from the SFT model and ranked by a labeler. Then the policy model is optimized against the reward model that is trained with these pairs to align with human preference. For NLU tasks, the pairs can be categorized into rationale-sensitive and label-sensitive. As illustrated in Figure 1, we provide an example where three answers sampled from the SFT model for the same instruction. If two answers have the same label and different rationales, they form a rationale-sensitive pair, with
4206
the more reasonable rationale considered superior. In contrast, if two answers have different labels, they form a label-sensitive pair, with the correct label deemed superior. However, we observed that the pairs sampled from the SFT model mainly fall into the category of rationale-sensitive. Figure 2 shows the specific distribution ratios of pairs across several NLU tasks. The percentage of rationalesensitive pairs exceeds 75%, and in datasets like SST-2, MR, and AGNews, surpasses 90%. The severe imbalance in the distribution of pairs leads the model to prioritize the quality of rationales over the correctness of labels during RLHF training, which conflicts with the evaluation metric (mostly label accuracy) of NLU tasks. A detailed analysis is presented in Section 4.2.
To address this challenge, our paper proposes a Reinforcement Learning framework enhanced with Label-sensitive Reward (RLLR) for NLU tasks. Firstly, we leverage GPT-4 to generate rationales corresponding to the gold labels of the training data. The SFT model is trained with rationales, incorporating CoT prompting to enhance comprehension abilities. Secondly, we generate rationales for the incorrect labels (relative to the gold labels). Unlike RLHF, which uses human intervention to rank sentences, RLLR automatically constructs label-sensitive pairs for training the reward model based on the correctness of the label. The comparison data is initially sampled from the trained SFT model. Finally, we train the policy model against the label-sensitive reward model with Proximal Policy Optimization (PPO) to prioritize the correctness of labels. Furthermore, optimizing
with mixed rewards from the label-sensitive and rationale-sensitive reward models, RLLRMIXED ensures both the accuracy of labels and the quality of rationales. Extensive experiments on eight NLU tasks demonstrate that our method consistently outperforms the SFT baseline by an average of 1.54% and the RLHF baseline by an average of 0.69%, while also exhibiting higher quality in rationales generation.
Our contributions are summarized as: (1) We propose a Reinforcement Learning framework enhanced with Label-sensitive Reward (RLLR) for NLU tasks to tackle the objective mismatch issue.
(2) Optimizing with mixed rewards, RLLRMIXED can achieve promising performance on both the accuracy of labels and the quality of rationales.
(3) Through empirical experiments, we demonstrate the effectiveness of our method. We have conducted a thorough investigation into various aspects, including the utilization of rationales, the performance of reward models, the quality of generated rationales and a detailed case study.

heading: 2 Related Work
text: Reinforcement Learning from Human Feedback. LLMs have demonstrated commendable performance, leveraging RLHF to achieve notable alignment and generation capabilities (Ouyang et al., 2022; Achiam et al., 2023; Bai et al., 2022a; Ziegler et al., 2019). RLHF aims to optimize the policy language model to generate content that is desired by humans. Recently, some research endeavors have uncovered inherent challenges in RLHF (Casper et al., 2023; Lambert et al., 2023), including feedback type limitations, evaluation difficulties, oversight challenges, etc. Several methods have been proposed to mitigate these challenges. Bai et al. (2022b) introduce RL from AI Feedback (RLAIF), training an AI assistant through self-improvement while adhering to constitutional principles that constrain model-generated content. Wu et al. (2023) introduce a fine-grained RLHF framework that uses fine-grained human feedback, such as identifying false sentences or irrelevant subsentences, as an explicit training signal. Rafailov et al. (2024) introduced a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form. This allows us to solve the standard RLHF problem with only a simple classification loss. Song et al.
(2024) proposed Preference Ranking Optimization (PRO) as an alternative to PPO for directly aligning LLMs with the Bradley-Terry comparison to accommodate preference rankings of any length. However, these approaches encounter a fundamental challenge in RLHF learning schemes: the objective mismatch issue (Lambert and Calandra, 2023). In this paper, we tackle this problem by training the reward model with the label-sensitive pairs.
Chain-of-Thought. CoT can significantly improve the complex reasoning ability of LLMs by generating natural language rationales that lead to the final answer (Wei et al., 2022; Kim et al., 2023). Hsieh et al. (2023) introduce a distilling mechanism step-by-step, extracting LLM rationales as additional supervision for training small models within a multi-task framework. Fu et al. (2023) propose a method to specialize the model’s ability (smaller than 10B) towards a target task with CoT prompting. In this paper, we enhance the performance of LLMs on NLU tasks with CoT prompting utilizing rationales generated for the labels.

heading: 3 Proposed Method
text: In this section, we introduce the training pipeline of our method as illustrated in Figure 3, including supervised fine-tuning, reward model training, and reinforcement learning enhanced with mixed rewards.

heading: 3.1 Supervised Fine-Tuning
text: In NLU tasks, the supervised dataset is denoted as S = {x, y}, where x denotes the sentence and y denotes the class label. The unsupervised dataset is denoted as U , and the foundation model is denoted as π. According to Wei et al. (2022), generating rationales that lead to the final answer can significantly improve the reasoning ability of LLMs through CoT. Therefore, we first generate a rationale from the sentence x and the label y with a specific prompt template using either human annotators or LLMs such as GPT-4. Then we reform the original dataset S to the training dataset T = {q, a}. The question q is constructed by x with a template and the answer a with t tokens is obtained by combining rationale and label, denoted as a = a1,··· ,t. The details of prompts can be found in Appendix A and B. The foundation model π is then trained on T to obtain the model πSFT. Formally, the loss for supervised fine-tuning is defined as:
LSFT = −E(q,a)∼T [logPπ (at | q, a1,··· ,t−1)] . (1)

heading: 3.2 Reward Model Training
text: In the second phase, comparison data are sampled from the answers generated by the SFT model πSFT given a question. As illustrated in Figure 2, more than 75% of the pairs generated by the
SFT model are rationale-sensitive pairs (i.e., both answers have the same label). The sentences in the rationale-sensitive pair are then labeled with a preference order. In RL, the reward model denoted as rϕ assigns higher scores to preferable answers compared with unfavorable ones, employing the Bradley-Terry paired comparison (Bradley and Terry, 1952). In this scenario, the model prioritizes the quality of the generated rationales over the accuracy of the labels. This focus shift results in less than optimal performance, stemming from the previously mentioned issue of objective mismatch.
To address this issue, we generate rationales based on the incorrect label for an input sentence and combine them to form a new answer. We leverage GPT-4 to generate rationales for the correct label and incorrect label. we generate rationales for incorrect labels ŷ to create a new answer â, which is a rationale-augmented incorrect answer. Along with the correct answer a, we can obtain the preferences a ≻ â | q for label-sensitive pairs without extra annotation. The process of the rationalesensitive pair generation is illustrated in Table 1, and additional details can be found in Appendix D. The label-sensitive and rationale-sensitive pairs are used to train two reward models, respectively. Specifically, we have a1 ≻ a2 | q to represent the preference in the pair. To predict these preferences, we employ the Bradley-Terry (BT) model, which
defines the preference probability as follows:
PBT = exp
( rϕ ( q, a1 ))
exp (rϕ (q, a1)) + exp (rϕ (q, a2)) . (2)
This objective is framed as a binary classification problem to train the reward model rϕ(q, a) with the loss defined as:
LR = −E(q,a1,a2)∼C [ log σ ( rϕ ( q, a1 ) − rϕ ( q, a2 ))] ,
(3) where σ is the logistic function and C is the dataset of comparisons. In this way, we can obtain two separate reward models rϕ1 and rϕ2 with the labelsensitive and rationale-sensitive pairs, respectively. The reward model rϕ(q, a) is often initialized from the SFT model πSFT(a|q) with the addition of a linear layer on top of the final transformer layer that produces a single scalar prediction for the reward value.

heading: 3.3 Reinforcement Learning
text: During the RL phase, we use the reward model to train the SFT model πSFT using Proximal Policy Optimization (PPO) on the unsupervised dataset U . Given a question constructed by the sentence from U , the mixed reward function from rϕ1(q, a) and rϕ2(q, a) is calculated as :
rM(q, a) = { rϕ1(q, a) + rϕ2(q, a), if rϕ1(q, a) < λ λ+ rϕ2(q, a), if rϕ1(q, a) ≥ λ
(4)
where λ is a hyper-parameter as the threshold for rϕ1 (the label-sensitive reward model). According to experimental observations, the reward score of rϕ1 converges to around 5.0, while the score of rϕ2 is within 1.0, resulting in an imbalance between the two. To prevent reinforcement learning from being completely dominated by rϕ1, we set a threshold value λ. When the score of rϕ1 is less than λ, the combined reward score is the sum of rϕ1 and rϕ2; when the score of rϕ1 is greater than or equal to λ, the combined reward score is equal to λ plus rϕ2. We first optimize the policy based on rϕ1, focusing on the correctness of the labels. As the RL training progresses, the score of rϕ1 gradually exceeds λ. Once the score of rϕ1 surpasses λ, we truncate it. At this point, the model will pay more attention to rϕ2, which is the quality of the rationale. In this way, both rϕ1 and rϕ2 can play a role in reinforcement learning, allowing the final policy model to predict the correct labels while generating high-quality rationales.
To guide the RL training, the loss function is constructed by combining the rewards generated by the reward model with a KL divergence constraint, which ensures that the policy does not deviate significantly from its initial behavior, defined as:
max πRL E(q,a)∼DπRL [ rM(q, a)− β log ( πRL(a|q) πSFT(a|q) )] ,
(5) where πRL is the learned RL policy, πSFT is the SFT model, and β is the KL reward efficient controlling the strength of the KL penalty. RLLRMIXED is obtained from this objective with two reward models while RLLR is trained only with the labelsensitive rationale reward model.

heading: 4 Experiments

heading: 4.1 Experiment Setup
text: Datasets. We evaluate the performance of our proposed method across eight NLU tasks, encompassing five from the GLUE benchmark (Wang et al., 2018). The tasks include Movie Reviews (MR) (Pang and Lee, 2005), AppReviews (AR) (Grano et al., 2017) and SST-2 for sentiment classification, AGNews (Zhang et al., 2015) for topic classification, MRPC and QQP for paraphrase detection, MNLI for textual entailment, and STS-B for semantic similarity. We employ the Pearson correlation coefficient as our evaluation metric for STS-B, and accuracy for others. To ensure a fair
comparison with baseline methods, we convert all tasks into a text-to-text format following (Sanh et al., 2021). For methods that require rationales, we utilize GPT-4 to generate rationales conditioned on given labels.
To ensure our model is free from cognitive biases, we refine our process with GPT-4 through meticulous manual reviews and prompt template adjustments. We manually evaluate the annotation through sampling and designed specific strategies to filter out data that did not meet our standards. Several specific challenges have been encountered, such as GPT-4’s reluctance to justify certain labels, particularly incorrect ones, and discrepancies between its generated rationales and the assigned labels. We address these issues by iteratively refining the prompt template and developing a classifier to filter out corrupt data. The details regarding the prompt templates and rationale annotation process are provided in Appendix A and B.
Baselines. We conduct our experiments using several state-of-the-art foundation models, including LLaMA2 (Touvron et al., 2023b), Baichuan2 (Yang et al., 2023), ChatGLM3 (Du et al., 2021), Mistral (Jiang et al., 2023), and Bloom (Workshop et al., 2022). We compare our method with two prevalent training methods: (1) SFT, which refines LLMs through optimization against a conditional language modeling objective on supervised data; (2) RLHF, which involves training a reward model on preference data and subsequently employing this model to guide RL-based fine-tuning. For RLHF, we utilize GPT-4 for the preference annotation within our experiments. Detailed procedural information can be found in Appendix C.
Training. To streamline the experimental complexity, we fine-tune the models on a multi-task dataset, rather than on datasets for individual tasks. To address the task imbalance issue, we construct the training set at a maximum of 5,000 samples per task. The surplus examples are used as unsupervised data for PPO in RLHF and RLLR. This approach mirrors real-world scenarios where unsupervised data is abundant, but supervised data is scarce. We also construct a multi-task test set comprising up to 1,000 examples from each task to enhance experimental efficiency without compromising validity. The details of the examples are listed in Table 2. In all experiments, we employ Low-Rank Adaptation (LoRA) (Hu et al., 2021) fine-tuning, as opposed to full-parameter tuning,
achieving up to an 80% reduction in GPU memory requirements. Within the RL-based approaches, the policy, reward, and value models are equipped with their own set of LoRA parameters.
We employ 4 V100 GPUs, each with 32 GB of memory, to train our 7B models. To maintain a consistent batch size across various experiments, we utilize the gradient accumulation. The Adam optimizer, coupled with a cosine schedule, is used in all experiments, with a fixed LoRA rank of 16. We perform a hyperparameter search within a limited range and observe that the performance exhibits minimal sensitivity to changes in these hyperparameters. The ranges for the primary hyperparameters, such as learning rate, batch size, and training epochs, are detailed in Table 3.

heading: 4.2 Main Results
text: Our main experiment results are shown in Table 4. Additional results for models of various sizes are available in Appendix E. SFT w. rat. and SFT denote models fine-tuned on supervised data with and without rationales, respectively. RLHF denotes models fine-tuned with the standard RLHF procedure, which predominantly utilizes rationalesensitive pairs. RLLR denotes models fine-tuned using our proposed method, with a reward model trained on label-sensitive pairs. RLLRMIXED further integrates reward models trained on both labelsensitive and rationale-sensitive pairs. The policy model is initialized from the SFT w. rat. model in both RLHF, RLLR, and RLLRMIXED settings.
Comprehensive evaluations across five foundational models and eight NLU tasks reveal that our RLLR method consistently surpasses the SFT baseline by an average margin of 1.54%, and the RLHF baseline by an average of 0.69%. The maximum average improvement over RLHF was achieved on Mistral 7B, reaching 1.02%. The enhancement observed in ChatGLM3 6B, while modest, is still quantifiable at an increase of 0.38%. RLLR and RLLRMIXED also achieve the best results on most individual tasks, except Baichuan2 on AGNews. However, integrating RLLR with other models consistently yields a performance enhancement on AGNews, most notably, exceeding the SFT baseline by 2.9% with Bloom-7B. This substantial improvement robustly validates the efficacy of the proposed method.
The integration of rationales brings improvement over the vanilla SFT by an average margin of 0.79%, demonstrating the benefit of rationales. Despite this improvement, the performance of SFT w. rat. still lags behind that of RLLR, suggesting that simply integrating the SFT method with rationales is insufficient. Moreover, the RLHF baseline mirrors the performance of SFT w. rat., with no additional gains, which corroborates the presence of an objective mismatch issue.
In Section 4.3, we further analyze the influence of various mechanisms, including the utilization of rationales, reward modeling objectives, and incorporation of multiple rewards in RL fine-tuning. The RLLRMIXED method achieves on-par performance with RLLR, surpassing SFT by an average of 1.45%, and RLHF by an average margin of 0.60%. However, we further examine its impact on the quality of rationales, extending our analysis beyond label accuracy.

heading: 4.3 Analysis
text: Utilization of rationales. Incorporating rationales into the SFT stage achieves improvement across 78% of our task-model pairings (35 out of
45), aligning with the advancements reported by (Hsieh et al., 2023; Kim et al., 2023; Fu et al., 2023). Nonetheless, a comparative analysis between SFT with RLLR indicates that the mere addition of rationales to SFT is insufficient. SFT, categorized under Behavior Cloning within the Imitation Learning framework, is prone to suffering from compounding errors (Ross et al., 2011). Theoretically, the minimum expected error for a policy derived through Behavior Cloning grows quadratically with the length of the trajectories. Introducing rationales under this method paradoxically extends trajectory lengths, exacerbating the issue. In contrast, RLLR, rooted in Inverse Reinforcement Learning, effectively reduces compounding errors by optimizing across entire trajectories rather than individual actions (Ho and Ermon, 2016; Swamy et al., 2023), thereby enhancing the effectiveness of rationales.
Reward model performance. To elucidate the superiority of RLLR over RLHF, we scrutinized the efficacy of reward models trained in both methods. The models are evaluated on a hold-out label-
sensitive dataset, comprising pairs of correct and incorrect answers with respect to the gold label. This evaluation framework is designed to assess the models’ proficiency in differentiating between rationales that lead to either the correct or incorrect labels. As indicated in Table 5, reward models developed under RLLR demonstrate an average accuracy of 90%, outperforming those from RLHF by a margin of 10%, which stand at an average accuracy of 80%. Detailed results on each individual task are presented in Appendix F. These findings align with the main results and underscore the detrimental impact of objective mismatch issue within RLHF. Conversely, RLLR is immune to such discrepancies, as its objectives are congruent with the evaluative criteria used to discern between correct and incorrect rationales, thereby yielding superior outcomes across a spectrum of tasks and models.
Quality of generated rationales. Despite the modest enhancement in accuracy, RLHF has significantly advanced the quality of text generation by integrating human preferences during fine-tuning.
Beyond the accuracy on NLU tasks, the quality of the generated text is also a key consideration. This is particularly relevant in human-in-the-loop contexts, where the model’s output serves as a guide for human operators, necessitating text that is both high-quality and reflective of human values. To assess this quality, we examined a subset of queries from the validation set and appraised the response quality produced by various models with GPT-4. We employed the win rate against SFT w. rat. as a metric for evaluation. The evaluation results are shown in Figure 4 and the implementation details are described in Appendix G. The RLHF method outperforms SFT in generating high-quality ratio-
nales, evidenced by a 39% win rate, a 26% lose rate, and a 36% tie rate on average. In contrast, the RLLR method demonstrates comparable performance to SFT, with a balanced win or lose ratio of 30% on average. This outcome aligns with expectations, as RLLR did not incorporate any human preference data. The hybrid RLLRMIXED method, which integrates the reward models of both RLHF and RLLR, achieves a 38% win rate, a 26% lose rate, and a 37% tie rate on average against the SFT baseline. When compared to RLHF, RLLRMIXED also exhibits a comparable efficacy, with a win rate of 34% and a lose rate of 33%. Notably, RLLRMIXED yields the most favorable outcomes, delivering rationale quality equivalent to RLHF and maintaining label accuracy at the level of RLLR.
Case study. Table 6 showcases examples of answers generated by the RLHF, RLLR, and RLLRMIXED methods. For the sake of brevity, we have omitted any superfluous content from the answers, retaining only the essential context and comparative elements. To visually highlight the primary differences within the examples, we’ve used
blue and red color coding. In the first example, which is sampled from the Movie Review dataset, the reviewer first describes the negative aspects of the movie, but at the end, a turning point is made by proposing “give birth to an unlikely but likable hero” to express positive sentiments. The RLHF model fails to recognize a positive sentiment shift, despite the reviewer’s concluding praise, leading to an incorrect negative assessment. On the other hand, the RLLR model accurately identifies the shift in sentiment, leading to an accurate evaluation. The second example from the STS-B dataset demonstrates that while the RLLR and RLLRMIXED methods yield similar similarity scores for identical sentence pairs, the RLLRMIXED approach enhances the rationale’s comprehensiveness by integrating an extra“Semantic Field” component. Furthermore, the RLLRMIXED output utilizes Markdown formatting to enhance readability. These findings suggest that the RLLRMIXED method can significantly improve the rationale’s quality compared to the standard RLLR approach, which proves our viewpoint.

heading: 5 Conclusion
text: In this paper, we introduce a Reinforcement Learning framework enhanced with Label-sensitive Reward to amplify the performance of LLMs for NLU. By training the reward model on label-sensitive pairs, which are constructed by generating rationales for the incorrect labels, we mitigate the objective mismatch issue in RLHF, leading to improved performance in NLU tasks. Extensive results on 5 foundation models and 8 NLU tasks demonstrate that RLLR consistently surpasses the SFT baseline by a margin of 1.54%, and the RLHF baseline by 0.69%. By additionally incorporating the label-sensitive and rationale-sensitive rewards, our enhanced RLLRMIXED method not only maintains the label accuracy comparable to RLLR but also achieves rationale quality on par with RLHF. We present an in-depth analysis of RLLR, examining the utilization of rationales, reward modeling objectives, and incorporation of multiple rewards during the RL stage. The results and analysis substantiate the effectiveness of our methods in the NLU tasks.

heading: 6 Limitations
text: Due to cost considerations, there are some deficiencies in our work, which we have listed here for future reference. Firstly, integrating rationales into model responses increases computing power requirements and generation time as a trade-off for enhanced accuracy and interpretability. Secondly, we utilize GPT-4 as a proxy of humans to generate rationales, annotate preferences, and evaluate the quality of rationales. Despite the success made by advanced AI models like GPT-4 in supplanting manual annotation, we believe that experiments with authentic human annotation and evaluation remain essential. Finally, the compatibility of RLLR with RL-free methods such as DPO, PRO, and RRHF remains unexplored. We leave these limitations for future work.

heading: A Prompts for Tasks
text: For all of the tasks, we use the following template for SFT w. rationale, RLHF and RLLR: {{rationale}}\n\n The answer is: {{label}}
The name in double curly brackets represents a variable and should be replaced with its value. Input templates and possible labels for each task are listed below.
A.1 Movie Reviews
Input template: {{text}} What sentiment does the writer express for the movie?
Possible labels: negative, positive
A.2 AGNews
Input template: What label best describes this news article? \n {{text}}
Possible labels: World politics, Sports, Business, Science and technology
A.3 MNLI
Input template: Given a premise and a hypothesis, predict the relationship between them. Choose one of the following labels: entailment, contradiction, or neutral. Premise:{{sentence1}}, Hypothesis:{{sentence2}}
Possible labels: entailment, contradiction, neutral
A.4 QQP
Input template:

heading: I received the questions “{{sentence1}}” and “{{sentence2}}”. Are they duplicates?
text: Possible labels: no, yes
A.5 SST-2
Input template: Movie review: {{text}} What sentiment dose the movie review express?
Possible labels: negative, positive
A.6 STS-B Input template: Analyze the semantic similarity between two sentences and predict a similarity score from 0 to 5. Sentence 1: {{sentence1}}, Sentence 2: {{sentence2}} Possible labels: Float number in range [0.0, 5.0].
A.7 MRPC Input template: Does the sentence \n {{sentence1}} \n paraphrase (that is, mean the same thing as) this sentence? \n {{sentence2}} Possible labels: no, yes
A.8 AppReviews Input template:

heading: On a scale of 1-5 (with 1 being least favorable and 5 being most favorable), how would you rate this review? “{{text}}”
text: Possible labels: 1, 2, 3, 4, 5

heading: B Rationale Collection
text: We utilize the following prompt templates to request GPT-4 for rationales. Sometimes GPT-4 refuses to give a rationale conditioned the provided label, and we train a simple classifier to filter out these responses. Examples of generated rationales are shown in Table 7. {{question}} \n\n Please give a rationale for the answer “{{label}}” in a confident tone (regardless of the true answer):

heading: C Preference Collection
text: We sample 5 responses from the SFT model for each example and ask GPT-4 to rank the responses. The prompt template for requesting GPT-4 is as follows: Given the following question and answers, please rank the answers according to your preference, considering accuracy, coherence, logicality, factuality, relevance, and information completeness. \n\n [Question] {question} \n\n [Answer

heading: 1] {answer 1} \n\n [Answer 2] {answer 2} \n\n [Answer 3] {answer 3} \n\n [Answer

heading: 4] {answer 4} \n\n [Answer 5] {answer
text: 5} \n\n Please give your rationale
first, and then give the ranking. Output format: “{rationale} \n\n Ranking: {e.g. {{ranking_example}}}”
The variable {{ranking_example}} is generated by shuffling the list [1, 2, 3, 4, 5] and concatenating them with “>” or “=”, e.g. 5>3>2>4>1 or 2>1=5>3=4. We generate a different example for every GPT-4 request to avoid bias.

heading: D Construction of Label-Sensitive Pairs
text: In tasks involving categorical labels, an incorrect label is randomly chosen from the full label set excluding the correct label, to create a label-sensitive pair. For the AppReviews and STS-B tasks, which use a rating scale from 0 to 5, incorrect labels are generated by adding 3 to the correct label and then incorporating a random value from the range [-1, 1]. For instance, given a correct STS-B label of
2.8, a random increment of 0.3 is selected, resulting in an initial incorrect label of 2.8+3+0.3=6.1. This exceeds the maximum rating, so we adjust by subtracting 5, yielding a final incorrect label of 1.1. In the case of the AppReviews task, this label is subsequently rounded to an integer.

heading: E Results of Varying Sized Models
text: To substantiate the scalability of our method across models of varying sizes, we also conduct a series of experiments using LLaMA2-13B and Bloom-3B models. The results are presented in Table 8, in conjunction with those of the 7B models to facilitate direct comparison. For LLaMA, the 7B model’s performance improved by 0.68% and the 13B model improved by 0.35% over the RLHF baseline. For BLOOM, the 3B model’s performance improved by 0.76% and the 7B model improved by 0.82%. Interestingly, smaller models don’t always get greater improvements. This consistency across disparate model sizes strongly supports the scalability of our proposed RLLR method.

heading: F Reward Model Performance
text: Table 9 presents the performance of reward models trained with RLHF and RLLR on eight individual NLU tasks. Reward models employing RLLR methods demonstrated an overall accuracy of approximately 90%, surpassing those trained with RLHF by a significant margin of 10 percentage points, with the latter achieving an accuracy of 80%. The gap between RLLR and RLHF on STS-B and AppReviews tasks is most significant, exceeding 35% and 20% respectively. The gap on AGNews, MRPC, and QQP tasks also exceeds 5%, indicating that RLHF suffers from objective mismatch issue on these tasks.

heading: G Evaluation of Generation Quality
text: We utilize GPT-4 as a proxy for human evaluation. First, we sample a set of questions and corresponding answers generated by two methods. To mitigate positional bias, we then randomize the order of the answers within each pair. The question along with two answers is subsequently formatted according to the predefined GPT4 input template: Given the following question and two candidate answers, please choose which one is better, considering accuracy, coherence, logicality, factuality, relevance, and information completeness. \n \n
[Question] {question} \n \n [Answer 1] {answer 1} \n \n [Answer 2] {answer 2} \n \n Please response with “Answer 1 is better” or “Answer 2 is better” or “Equal” first, and then give your rationale.

heading: Future Work
text: Due to cost considerations, there are some deficiencies in our work, which we have listed here for future reference. Firstly, integrating rationales into model responses increases computing power requirements and generation time as a trade-off for enhanced accuracy and interpretability. Secondly, we utilize GPT-4 as a proxy of humans to generate rationales, annotate preferences, and evaluate the quality of rationales. Despite the success made by advanced AI models like GPT-4 in supplanting manual annotation, we believe that experiments with authentic human annotation and evaluation remain essential. Finally, the compatibility of RLLR with RL-free methods such as DPO, PRO, and RRHF remains unexplored. We leave these limitations for future work