Starling-7B: Improving Helpfulness and Harmlessness with RLAIF

Authors: banghua  zhu
evan  frick
tianhao  wu
hanlin  zhu
karthik  ganesan
nexusflow  ai
wei-lin  chiang
jian  zhang
jiantao  jiao
u c berkeley
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 05:10:33

Abstract: This paper presents Starling-7B, the current best-performing 7B chat model on Chatbot Arena, along with its training dataset Nectar, a high-quality preference dataset collected by prompting GPT-4 to rank responses. We propose an internal pairwise rating technique, where the model considers all pairings before providing a ranking decision, leveraging the proven pairwise rating capability of LLMs without the cost of individual pairwise calls. The resulting Nectar dataset comprises 182,954 chat prompts, each with seven responses from various models, ranked by GPT-4, equating to 3.8 million high-quality pairwise comparisons. We introduce Starling-RM-7B and Starling-RM-34B, the reward model suites trained with a K-wise preference loss on Nectar, outperforming pairwise counterparts. We benchmark reward model training pipelines across metrics such as human preference, truthfulness, and safety. Using Nectar and our new training pipeline, we fine-tuned Openchat-3.5 to create Starling-LM-7B, achieving significant performance enhancements on MT-Bench, AlpacaEval, and human evaluation metrics. To facilitate research and understanding of RLHF mechanisms, we open-source the Nectar dataset, the reward models, and the language models.

Main paper: 
 introduction: 
 Reinforcement learning from human feedback (RLHF) is a pivotal technique in post-training that helps align large language models (LLMs) with human preferences. It has seen great success in improving the helpfulness and harmlessness for several proprietary and open weight models, including Claude, GPT-4 (OpenAI, 2022; 2023a; b) and Llama-2-Chat (Touvron et al., 2023) . However, the open science and open source community is not yet able to reproduce this success, potentially due to the lack of high-quality preference dataset and reward model, and unsuccessful hyperparameter tuning with reinforcement learning algorithms.We aim to bridge the gap by open-sourcing Nectar and Starling model suites. Nectar is a GPT-4-labeled ranking dataset for training the reward model in RLHF. Starling model suites are a family of state-of-the-art open source reward models and chat models fine-tuned with reinforcement learning.The post-training of LLMs is usually composed of two steps: supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). In the past, SFT has notably advanced the development of open LLMs, particularly with the utilization of high-quality data distilled from ChatGPT/GPT-4. This is demonstrated effectively by models such as Alpaca (Taori et al., 2023) , Vicuna (Chiang et al., 2023) , and Openchat-3.5 (Wang et al., 2023a) . Another algorithm that gains popularity in the open-source community is direct preference optimization (DPO), which updates the model in a contrastive fashion based on preference dataset. The method has created strong chat models including Zephyr-7B (Tunstall et al., 2023) , Mistral-7B-Instruct (Jiang et al., 2023) , and Tulu-2-DPO-70B (Ivison et al., 2023) . However, the vanilla RLHF algorithm with reward training and proximal policy optimization (PPO) in Ouyang et al. (2022) has not produced strong open source models yet, despite its great success in proprietary models like GPT-4 (OpenAI, 2022; 2023a; b) and Claude.Addressing the research gap in RLHF, a key requirement lies within the provision of a high-quality chat-specific ranking dataset. We introduce Nectar, a unique ranking dataset labeled by GPT-4 which encompasses 182, 954 chat prompts. Each prompt comprises 7 responses, distilled from a variety of models such as GPT-4, GPT-3.5-instruct, GPT-3.5-turbo, Mistral-7B-Instruct, and Llama2-7B-chat, ranked by GPT-4. This equates to a total of 3.8 million high-quality pairwise comparisons. We also introduce novel techniques to reduce positional bias when prompting GPT-4 for rankings. This enables reinforcement learning from AI feedback (RLAIF), where we learn a proxy of human preferences from (potentially biased) GPT-4 ranking data. We compare the Nectar dataset with existing preference dataset (Cui et al., 2023; Bai et al., 2022a; Ethayarajh et al., 2022) Furthermore, we open source our reward model suites, Starling-RM-7B and Starling-RM-34B. The models are trained using the K-wise loss (Zhu et al., 2023) on the Nectar dataset. We benchmark various reward model training pipelines across metrics such as human preference, truthfulness, and safety in Section 4.To create the chat model Starling-LM-7B-alpha, we fine-tune Openchat-3.5 (Wang et al., 2023a) with proximal policy optimization (PPO) (Schulman et al., 2017) 2 Related Work 
rlhf and rlaif: 
 Previous work has shown that human evaluators can effectively provide a reward signal to train agents in complex RL environments (Christiano et al., 2023) . Specially, RLHF has shown immense effectiveness when applied to aligning LLMs to human preferences (Ouyang et al., 2022; Bai et al., 2022a; OpenAI, 2023a; 2022; Touvron et al., 2023) . Since RLHF uses actual human preference data, data collection is a very expensive bottleneck.To mitigate this, recent efforts have tried RLAIF (Bai et al., 2022b; Lee et al., 2023; OpenAI, 2023a) . Ma et al. (2023) utilizes the zero-shot performance of LLMs for ranking tasks while Petroni et al. (2019) explores LLMs' effectiveness as knowledge bases, which together may provide a lower cost solution to mimic human preference via careful rubric design and prompting.Previous work has show that LLMs are not necessarily fair evaluators and exhibit significant positional bias (Wang et al., 2023b) . As result, most LLMs human preference ranking strategies only use more reliable pairwise comparisons where ranking outputs are more stable (Qin et al., 2023) . Black-box pure K-wise ranking strategies have struggled to find success, citing problems with outputs missing choices, repeating choices, inconsistently ranking, or being irrelevant Qin et al. (2023) . Some effort to generalize pairwise prompting to K-wise prompting has found success by utilizing multiple pairwise evaluations to sort a K-wise ranking. This includes aggregating all pairwise comparisons or utilizing pairwise comparisons for heapsort (Qin et al., 2023) . Using K choose 2 pairwise comparisons to create a final K-wise ranking is not favorable due to O(K 2 ) evaluator queries, or O(K log K) for sorting algorithms. Ideally, a K-wise ranking could be completed zero-shot with minimal positional bias with just a single evaluator query. This paper present a prompting framework that approaches this ideal. 
nectar dataset: 
 In this section, we discuss the creation of the Nectar dataset. The dataset is composed of a set of prompts, with 7 responses for each prompt distilled from existing models, along with GPT-4-based ranking for the responses. We provide details about the prompt collecting, response collection, and response ranking respectively.We aim to curate a dataset with diverse prompts from different sources, including adversarial jailbreaking prompts. The collected prompts are used for both reward learning and policy learning in RLHF. We collect data from a range of existing validated datasets, including ShareGPT, Anthropic-HH (Bai et al., 2022a) , UltraFeedback (Cui et al., 2023) , and Lmsys-Chat-1M (Zheng et al., 2023a) . We first sub-sample prompts from each different category, and then combine the same prompts and their corresponding model-generated responses, resulting in a dataset with 182, 954 prompts. The composition of the prompts is provided in Table 2 . For each prompt in the dataset, we distilled a response from GPT-4, GPT-4-0613, GPT-3.5-Turbo, GPT-3.5-Turbo-Instruct, Mistral-7B-instruct, and Llama-2-7B-chat. Each prompt also comes with a default response in the corresponding dataset. As result, we get a dataset with 182,968 prompts and 7 responses per prompt. GPT-4-0613 was given a system prompt to help set the chatting context yielding better responses. GPT-3.5-Turbo-Instruct was chosen for the lack of extensive human feedback alignment, unlike many of the other models present, leading it to be more likely to comply with jailbreak prompts with high quality responses. We provide more detailed analysis on the distribution of the model responses in Appendix A.Given some ranking rubric and user prompt from the prompt dataset, we ask GPT-4 to rank the 7 responses from best to worst in accordance with rubric. The goal of the rating prompt was to capture a bimodal rubric strategy designed to balance helpfulness and harmlessness: Maximally answer good-natured conversations and harmlessly answer all conversations.We create the following rubric:The helpfulness rubric is as follows:1. Is the completion relevant? The completion should be the best possible assistant answer. 2. Is the completion accurate? The completion should accurately fulfill the prompt's request. 3. If a creative answer is expected, is the completion creative? If an analytical answer is expected, is the completion factual/objectively correct? 4. Is the completion detailed? The completion should at minimum satisfy the full level of detail required by the prompt.The harmlessness rubric is as follows:1. Does the completion adhere to moral and ethical standards? 2. Is the completion truthful? 3. Is the response polite and respectful? 4. If the request given is not good natured, does the completion politely decline to reply while giving a short explanation why they cannot comply with the request? Completions should always maintain harmlessness in the face of adversarial requests.Our objective is to de-emphasize the role of helpfulness in the evaluation of responses to prompts of a negative nature, intending primarily, to maintain harmlessness. This approach is designed to prevent harmful outputs from obtaining higher scores than their harmless equivalents, simply because they adhere more closely to the instructions. Conversely, when dealing with standard prompts, we rank the responses purely based on the helpfulness.As one of the major challenges in ranking 7 responses, we observe very serious positional bias in GPT-4 when ranking more than 2 responses. We discuss the findings in the section below.The notion of positional bias has been studied in a pairwise context, with results indicating that the first or the second index could be favored, contingent on the ranking model deployed Wang et al. (2023b) . We have similar observations in our K-wise rating settings.For each prompt, we shuffle the 7 responses uniformly at random and prompt GPT-4 to provide a list-wise ranking. We observe that GPT-4 exhibited a strong preference for selecting responses seen at the beginning. Such responses won at a rate ten times greater than completions viewed at the sixth index (See Figure 2 , when K = 7). These results are consistent with earlier observations, implying that GPT-4 has an inherent tendency to favor the first several responses (Wang et al., 2023b) . Without effective strategies to address this bias, it becomes evident that earlier responses are overwhelmingly preferred over later ones.Moreover, our observations indicate that this positional bias worsens as K increases unless remedial actions are taken. Figure 2 visualizes the increasing positional bias problem for larger K. While in pairwise rating situations, the bias is manageable, larger K-wise rating procedures are unusable. In the following sections, we explore various strategies to combat the positional bias problem for large K.We introduce novel prompting techniques to mitigate the positional bias by first generating all pairwise rankings, and then summarizing the results into list-wise rankings. We leave the details to Appendix A.2. We also leave additional analysis, including model pairwise win rates and pairwise average ranking differential heatmaps to Appendix A. 
formulation: 
 In this section, we discuss the training of the reward model. The reward learning procedure in RLHF can be formulated in a contextual bandit environment. In a contextual bandit, for the i-th sample, a state (prompt) s i is first sampled from some fixed distribution ρ. Given the state s i , K actions (a i 0 , ] be the output of the human labeller, which is a permutation function that denotes the ranking of the actions. Here σ i (0) represents the most preferred action. We use a 0 ≻ a 1 to denote the event that the action a 0 is more preferred compared to a 1 . A common model on the distribution of σ under K-ary comparisons is a Plackett-Luce model (Plackett, 1975; Luce, 2012) . The Plackett-Luce model defines the probability of a state-action pair (s, a i ) being the largest among a given set {(s, a i )} K-1 i=0 asa i 1 , • • • , a i K-1 ) are sampled from some joint distribution P(a 0 , • • • , a K-1 | s i ). Let σ i : [K] → [KP(a i ≻ a j , ∀j ̸ = i | s) = exp(r θ ⋆ (s, a i )) ∑ K-1 j=0 exp(r θ ⋆ (s, a j )).Here r θ ⋆ is the ground truth reward function with parameter θ ⋆ . Moreover, one can calculate the probability of observing the permutation σ as 1P(σ | s, {a i } K-1 i=0 ) = K-1 ∏ i=0 exp(r θ ⋆ (s, a σ(i) )) ∑ K-1 j=i exp(r θ ⋆ (s, a σ(j) )).When K = 2, this reduces to the pairwise comparison considered in the Bradley-Terry-Luce (BTL) model (Bradley & Terry, 1952) . In this case, the permutation σ can be reduced to a Bernoulli random variable, representing whether a 0 is preferred compared to a 1 . Concretely, for each queried state-actions pair (s, a 0 , a 1 ), we observe a sample y from a Bernoulli distribution with parameter exp(r θ ⋆ (s,a 1 ))exp(r θ ⋆ (s,a 0 ))+exp(r θ ⋆ (s,a 1 )) ; i.e., for any l ∈ {0, 1},P(y = l | s, a 0 , a 1 ) = exp(r θ ⋆ (s, a l )) exp(r θ ⋆ (s, a 0 )) + exp(r θ ⋆ (s, a 1 )).By observing the sampled dataset D = {s i , a i 0 , a i 1 , • • • , a i K-1 , σ i }, we would like to estimate the ground truth reward r θ ⋆ with some proxy reward r θ . We discuss in the following section some different choices of estimators we benchmark in the paper. 2022), we remove the last layer of a pre-trained language model and concatenate with a linear layer for reward prediction. We take Llama-2-7B-Chat (Touvron et al., 2023) as the base model for reward model. Observing the various performances of the 7B Llama-based reward models, we also train a 34B reward model utilizing Yi-34B-Chat as the base model (AI et al., 2024) . The reward model takes in a prompt and response, and outputs a scalar representing whether the response is helpful and harmless given the prompt.For learning the reward from K-wise comparisons, we consider three methods. The first method is the original method from Ziegler et al. ( 2019 2022), which decomposes each K-wise comparison into K(K -1)/2 pairs of pairwise comparisons, and then minimize the cross entropy loss. When there is only pairwise comparison, this is the maximum likelihood estimator under Bradley-Terry-Luce model (Bradley & Terry, 1952) . It has also been shown to converge to the ground truth when the reward model is linear and well-specified with K ≥ 2 (Zhu et al., 2023) .rMLE 2 ∈ arg min r ℓ D (r), where ℓ D (r) = - 1 n n ∑ i=1 K-1 ∑ j=0 K-1 ∑ k=j+1 log exp(r(s i , a i σ i (j) )) exp(r(s i , a i σ i (j) )) + exp(r(s i , a i σ i (k) )).The second method directly applies the maximum likelihood estimator under Plackett-Luce model, which is shown in Zhu et al. (2023) to be asymptotically more efficient than rMLE 2 .rMLE K ∈ arg min r ℓ D (r), where ℓ D (r) = - 1 n n ∑ i=1 K-1 ∑ j=0 log   exp(r(s i , a i σ i (j) )) ∑ K-1 k=j exp(r(s i , a i σ i (k) ))   .The third method is the iterative data smoothing algorithm proposed in (Zhu et al., 2024) , which iteratively updates the model with the data, and then updates the soft data label with the model after each epoch to prevent overfitting to the noisy data. We denote the final estimator as rsmooth .In this section, we benchmark the results of the three methods detailed in 4. 1 Best) e r(Best) +e r(Other) ). In K-wise case where K > 2, the K-wise loss is sum of the pairwise loss for each ( K 2 ) pairings. The accuracy and loss results for each trained reward model are shown in Table 3 and Table 4 , respectively. Additional details on these evaluation metrics and what they represent can be found in Appendix B.1. Note, the models chosen for release as Starling-RM-7B and Starling-RM-34B are the 7B and 34B models trained under the rMLE K formulation, respectively, as we find rMLE K to produces the strongest reward models in accordance to the evaluations. Additionally, likely as result of the significantly stronger base model, Starling-RM-34B (r MLE K (34B)) exceeds all 7B counterparts in performance for every evaluation metric in Table 3 and Table 4 .N ∑ log( e r(Chat On Reward Bench, Starling-RM-34B achieves state-of-the art performance (Lambert et al., 2024) . Most notably in the "Reasoning" category on Reward Bench, Starling-RM-34B has a large improvement over Starling-RM-7B, suggesting that larger and more capable base models may help the reward model's performance in downstream complex reasoning tasks. On the contrary, the "Chat" category performance seems to saturate quickly. Additionally, both Starling models surpass all other models in the "Safety" category, including models specifically trained for safety such as (Dai et al., 2023) 's Beaver-7B-v1.0-Cost. These results indicate that the inclusion of prompts of good and bad nature along with a judging rubric that enforces harmlessness despite helpfulness yields safer models on par with models more solely focused on harmlessness-all while maintaining exceptional performance on helpfulness categories. 
policy learning: 
 In policy learning stage, we use proximal policy optimization (PPO) (Schulman et al., 2017) to fine-tune the language model based on the learned reward model. To create the chat model Starling-LM-7B-alpha, we fine-tune Openchat-3.5 (Wang et al., 2023a) with the learned reward model Starling-RM-7B-alpha. As a result, the MT-Bench score improves from 7.81 to 8.09, while the AlpacaEval score improves from 88.51% to 91.99%, and a human evaluation ELO increases from 1072 to 1087 on Chatbot Arena (Chiang et al., 2024) . We also release a new version Starling-LM-7B-beta by fine-tuning Openchat-3.5-0106 on the larger reward model Starling-RM-34B, which increases the human evaluation ELO from 1089 to 1118 on Chatbot Arena.We implement the PPO algorithm using the TRLX library (Havrilla et al., 2023) , incorporating all standard tricks of PPO as documented in Huang et al. (2022) . In our implementation, we decouple the actor (the LM) and the critic, allowing for independent gradient updates. We find out that PPO is highly unstable. For example, after exposure to only 1000-2000 prompts, the model rapidly learns to generate excessively verbose outputs. To mitigate this instability and improve training robustness, we propose the following techniques:Shifting the reward mean for length control: During our experiments, we observe that the reward model may assign highly negative rewards to the initial actor's outputs. Although the absolute magnitude of the reward does not convey preference information, and only the relative difference in reward between two responses indicates preference (Wu et al., 2023) , the overall reward magnitude significantly affects the generated response length. We find that a very negative reward can cause the model to become excessively verbose after just a few gradient updates. To address this issue, we propose adding a constant to the reward to make it slightly positive. The intuition behind this approach is as follows: during the initial gradient steps, we should increase the likelihood of the end-of-sequence (EOS) token, which is equivalent to making the response shorter. We can estimate the advantage of the EOS token as (r eos + V eos )r prev , where r prev is the reward assigned to the token before the EOS token (usually the negative KL penalty), r eos is the reward scored by the reward model, and V eos is the critic's value estimate for the EOS token. During initialization, we know that r prev and V eos are approximately zero, so a slightly positive r eos will penalize the length. We observe that after applying the reward mean shift, the model initially generates slightly shorter responses and then gradually increases the response length as training progresses, driven by the verbosity preference in the reward.Pretraining the critic model: During the initial stages of training, we observe that a randomly initialized critic (initialized from the language model with a linear layer and Gaussianinitialized head) can negatively impact the early performance of the language model. This is evident from a slight decrease in the MT-Bench score during the first 30% of the training process. To minimize early performance degradation, we propose pretraining the critic. We begin by conducting an RL run with a randomly initialized critic and the supervised fine-tuned (SFT) actor. We then use the final critic model from this run as the initialization for a new run with the same SFT actor. With this modification, we observe that the actor optimizes the reward more rapidly compared to the case without critic model pretraining. This approach helps to stabilize the early stages of training and allows the actor to more effectively optimize the reward signal.Full parameter tuning yields the best results: In our early experiments, we initialize both the actor and critic from the supervised fine-tuned (SFT) model, while unfreezing only the top 4 layers of the actor and critic during the RL stage. Although this approach improves the MT-Bench score from 7.81 to 8.09, the improvement is significantly smaller compared to initializing the actor and critic separately and performing full parameter tuning. By initializing the actor and critic separately and tuning all parameters, we can improve the score from 7.81 to 8.33, starting from the same SFT model. While MT-Bench serves as a proxy for model performance, our internal human evaluation also confirms that the full parameter tuned model is more preferred by human raters. This finding highlights the importance of full parameter tuning in achieving the best possible performance gains during the RL stage.We perform extensive hyper-parameter searching to find the best configuration. We mainly focus on tuning the learning rate, training batch size, and KL penalty. These hyperparameter choices aim to strike a balance between model performance, and computational efficiency while minimizing undesirable behaviors such as over-optimization and excessive response length.Our findings suggest that a slightly larger learning rate can be beneficial in reducing the over-optimization issue. When using a small learning rate and a longer training duration, the exploration in online RL may generate unusual responses that still receive high rewards due to the discrepancy between the ground truth reward and the proxy reward.Additionally, we observe that a slightly larger KL penalty helps to stabilize the generated response length, although it can only mitigate the increase in length and not completely prevent it. The final run uses an KL penalty 0.01 and learning rate 10 -6 . We also find that extremely large batch sizes do not yield observable improvements in human evaluation. Therefore, we utilize a medium batch size to facilitate faster iteration and experimentation, sampling 512 responses to form the replay buffer. Furthermore, the policy and critic updates are performed using a micro-batch size of 32.We adopt a multi-layer checkpoint filtering strategy, where a checkpoint must pass all the previous filters to be considered as a candidate for final release. Our criteria are as follows:• KL-Reward Trade-off: We require the model to achieve a predefined reward threshold while maintaining a KL-divergence lower than a specified KL threshold. This ensures that the model improves its performance without deviating too far from the base model's output distribution.• Verbosity Monitoring: We observe that the reward model often prefers longer responses, which can lead the language model to generate unnecessarily verbose explanations or even repeated outputs. To mitigate this issue, we monitor the average generated response length on a held-out validation set of prompts during training and filter out checkpoints that exceed a predefined length threshold.• Measuring Instruction-Following Capability: We directly measure the model's ability to follow instructions using IFEval (Zhou et al., 2023) . IFEval focuses on 25 types of "verifiable instructions", such as "write in more than 400 words" and "mention the keyword 'AI' at least 3 times." We find that steering the language model toward human preferences with RLHF can sometimes compromise its ability to precisely follow instructions. To address this, we predefine a threshold based on the base model's score and ensure that the checkpoint's score does not fall below this threshold.• MT-Bench Evaluation: For checkpoints that pass all the previous tests, we evaluate the model using GPT-4 as a proxy. MT-Bench is a well-understood proxy for human preference; the evaluation process involves generating responses to a fixed set of prompts spanning a diverse range of topics and using GPT-4 to score the responses. We find that PPO can significantly increase the MT-Bench score. We set a reasonably high MT-Bench threshold, as we observe that scores higher than this threshold do not necessarily indicate a true human-perceivable performance difference.• Internal Human Evaluation: We create a fixed set of 10-20 challenging prompts that assess the model's reasoning, instruction-following, and verbosity. We then judge the quality of the generated responses through internal human evaluation. 
conclusion: 
 In this paper, we produce the open-source RLHF pipeline that improves the helpfulness and harmlessness of the chat model. We release Nectar, a first-of-its-kind open-source high-quality 7-wise ranking dataset, and encourage future work to further explore the potentials of this dataset. Furthermore, we explore the effects of various reward training formulations on reward model performance. Finally, we perform extensive hyperparameter tuning on PPO to maximize performance gain in the RL stage. Ultimately, we hope to promote open source research and democratize the access of strong LLMs by releasing the dataset, methods, and the models trained from our RLHF procedure. We show that our data and training pipelines are capable of creating state-of-the-art reward and language models, pushing the limits of open-source LLMs.There are also limitations with respect to the data and methodology, which may be important as future work. The Nectar dataset is collected purely from GPT-4-based preference, which may contain significant bias that is inconsistent with true human preferences. It would be worth exploring methods that mitigate biases from synthetic data. In addition, Nectar focuses more the helpfulness and harmlessness of the responses and less on the instruction following property. As such, according to our observations, the resulting language model can be less capable of following exact instructions. There may also be improved reward training and policy learning algorithms leading to better reward models and language models. We hope future work can explore these areas in greater depth. 
 
 Appendix: In their study, Wang et al. (2023b) deduced that a smaller disparity in quality between two responses could intensify the effect of positional bias. Seeing as the K-wise prompt technique relies upon pairwise comparisons, its vulnerability to the influence of positional bias could be conjectured as similar. Particularly in instances where the quality of responses is closely matched, the strength of positional bias is amplified. Hence, we made a concerted attempt to examine the influence of positional bias on near equal responses.As a baseline, we created ratings by utilizing an enforced randomized pairwise strategy with no specified tiebreak strategy. Subsequently, we developed a new prompt enforcing a tie-breaking strategy (for instance, [B > C > D > A > ...] or "Break tie randomly") for each rating.Whether the tie-breaking rule is randomized or deterministic, in the event of a tie, the model is prompted to adhere to the randomly generated tie-breaking rule rather than making an unprompted "random" decision.If positional bias solely impacted responses that were precisely tied, then the invocation of a tie-break strategy would normalize the distribution of winners, aligning it more closely with the anticipated value. Conversely, should positional bias affect the model's intrinsic quality assessment of each response and in turn, blur the distinctions between the quality of responses, the effect of incorporating a tie-break strategy would be negligible.Figure 8 : Winning Index Distribution for Different Tie-breaking Strategies. None means no strategy is specified. "Random" means GPT-4 is instructed to break ties randomly. "Random Given Order" means GPT-4 is to follow a given randomly generated tie-break order. "Static Given Order" means GPT-4 is to follow a given static order designed to help observably underrepresented indices.Figure 8 indicates a marginal positive impact of tie-breaking on the distribution of indices. This slight improvement is characterized by a decrease in the overrepresentation of the 0 index, coupled with an mitigating underrepresentation of the 2 index. Notwithstanding this adjustment in distribution, it is notable that it still reveals a certain degree of bias. The Static Given Order seems to cause undesirable disparities in middle indices, while both Random Given Order and "Random" provide very similar results, with the latter mitigating underrepresentation in the 2 index.Overall, the disparity in positional bias, whether the ties are broken "randomly" or in a given random order order, appears to be minimal-but always is better than providing no specific tie-breaking instructions.In light of these observations, we posit that the issue of positional bias is a compound manifestation of both proposed hypotheses. Although the procedure of randomized tiebreaking fosters a more proportional distribution, it does not completely mitigate the issue of positional bias. Consequently, this suggests the intrinsic nature of positional bias as fundamentally influencing the evaluator's scoring of a given response. However, although positional bias is rather hard to completely mitigate, our prompting greatly helps control the reducible part of positional bias.Figure 9 : The most considerable deviation in pairwise agreement occurs during shuffling, revealing the impact of positional bias. The difference between 'Explaining' and 'Not Explaining' isn't generally significant. Note there isn't any remarkable difference when the pairwise rater is obliged to explain.The final chosen prompt uses randomized enforced pairwise rating order before the overall 7-wise rating, no enforced tie-breaking order, and no explanations for ratings. The final prompt is detailed below. The robust performance exhibited by Llama-2 in maintaining harmlessness can be attributed to its extensive safety alignment that counteracts prompt-level adversarial attacks (Chao et al., 2023) . Prior research illustrates that Llama2 models possess impressive resilience when facing jailbreak prompts (Chao et al., 2023) ; (Zheng et al., 2023a) . Additionally, contrary to expectations, smaller models have been empirically shown to outperform larger ones when assessed solely on harmlessness (Kundu et al., 2023) . Our findings corroborate this notion, signifying that, in comparison to its counterparts, Llama-2-7b-chat consistently displayed the utmost harmlessness in response to ill-intentioned prompts. This was followed closely by Llama-2-13b-chat and Llama-2-70b-chat.We also provide the heatmap for the pairwise average ranking difference of all models in our Nectar dataset in Figure 13 , and the pairwise winrates in Figure 14 .For the human preference measurement, we first subsampled 10, 000 single-turn humanrated response pairs from Lmsys's ChatBot Arena Conversations dataset (Zheng et al., 2023b) . We have each reward model return a score for each prompt response in the dataset. For a given rated response pair, if the reward model's score for the human-chosen winning response is greater than the score for the losing response, that response pair is considered classified correctly. The accuracies and losses for each model are detailed in Table 3 and Table 4 .In order to understand if each reward model will value true statements over false statements, we utilize true and false response pairs from the Truthful QA dataset (Lin et al., 2021) . For each prompt in the dataset, we sample three answers: the best truthful answer (Best), a true answer (True), and a false answer (False). We evaluate reward model output scores for each of these answers and average accuracy/loss across the three possible pairwise combinations where the overall ranking order is (Let r be the reward model) r(Best) >= r(True) > r(False). Figure 15 : Pointwise scoring methods struggle to distinguish the differences between response choices and overwhelmingly give high scores.We measure safety by evaluating on the PKU-SafeRLHF dataset from Ji et al. (2023) subsampled down to 1000 response pairs where one response is safe and the other response is unsafe. Reward model scores are generated for each of the responses with the expectation that the the safe response score is higher than the unsafe response score. See Table 3 for more detail and Table 4 for losses.Empirical analysis of the RL fine-tuned language model shows an occasional undesirable tendency towards overly verbose responses to simple prompts. For example when prompted with "1+1", the model responded with:Here are some important tips to consider when using the OpenAI GPT-4 model:• Understand the capabilities and limitations of the model: GPT-4 is a powerful language model, but it's important to recognize its limitations. While it's great at natural language understanding and generation, it may not always provide accurate or reliable information.• Be mindful of biases: ... [response continues for another 700 words]We created a small handcrafted dataset of 29 response pairs, where one response was overly verbose, and the other was not. Verbose response were crafted through finding prompts that yielded over verbose responses, handcrafting verbose response by introducing repetition, and through finding existing examples. A generated reward scoring pair is correct if the less verbose response receives the higher score.In this task, all three reward models trained struggled, with accuracy rates of 0.167, 0.133, and 0.100 (Table 3 ). The reward model seems to often value extremely long responses and give lower scores to very simple but concise answers. In the above example where "1+1" is prompted, the long 700+ word response was given a score of -0.635 while the simple but correct answer of "2" was given a score of -4.313. An additional example can be found in the Appendix B.2.This section aims to delve into the implications of the verbosity preference noticed in the trained reward model. The investigation involves examining whether this verbosity preference materializes as a byproduct of GPT-4's inherent propensity while curating the rankings for Nectar. We run the same 29 evaluation pairs, comprising of both verbose and non-verbose responses, through the GPT-4 ranking procedure, which was adjusted for pairwise ranking. With the original rubric GPT-4 picks the non-verbose (correct) response with an accuracy rate of only 0.433. Augmenting the rubric to detail that response should not be "overly verbose" yields a slightly improve accuracy of 0.500. However, in both cases, GPT-4 is unable to correctly distinguish when verbosity should be less preferred. These experiments suggest that the verbosity bias noticed in the reward models is possibly a manifestation of GPT-4's inherent rating bias. An example of this verbosity preference is show below in the Appendix B.3.In practice, one may introduce an extra temperature parameter γ and replace all r θ ⋆ with r θ ⋆ /γ. Here we take γ = 1. 


Decision: This paper presents a recipe for training a model with RLAIF. The final Starling-7B model achieves high performance on ChatbotArena, among other benchmarks.  While there are some methodological criticisms (e.g., data skewed towards GPT-4, by viaJ), the reviewers on balance find the Nectar dataset, its reward models, and the final LLMs helpful. Some of the details like the analysis of positional bias are likely to be useful in other work.

Reviews: review: paper_summary: paper_summary: This paper presents not only a chatbot which is the best on the chatbot area, but also releases the training dataset, and explains implementation details.
strength_weakness: strength_weakness: - There are quite a few limitations in using GPT as an evaluator, though the authors acknowledge those. 
- While this is a useful open source tool, data release, and hyper parameter optimization, there is more limited research contribution.
questions: questions: small comments:

- there is a messed up reference:
"We compare the Nectar dataset with existing preference dataset (Cui et al., 2023; Bai et al., 2022a; Ethayarajh et al., 2022) in Table ??.'

- figure 1 is blurry.

Typos:
-  "We would like to When K = 2, "
- Similar to the default methodology of training reward model  -->  training a reward model, or training reward models
paper_summary: paper_summary: Quality: The authors have developed an approach to utilize GPT-4 to create a preference dataset, Nectar, and train reward models. Finally the authors train a chat model with PPO using the reward models. Using this pipeline, they achieve state-of-the-art performance within the 7B scale models. So the quality is ok.

Clarity: The paper is easy to follow.

Originality: The paper combines a few existing things together, it is not particularly novel. The most novel part is they do global ranking so that the reward models have more samples to learn from

Significance: This work makes a great contribution to the open-source community. The reward model achieves pretty good performance on rewardbench, and the model is publicly released.
strength_weakness: strength_weakness: 1. Lack of novelty, mostly prompt engineering to get good ranking information.
2. The paper presentation can be improved. Some of the figures are not particularly readable, and there are small typos throughout the paper.
questions: questions: 1. I would recommend authors remove ShareGPT from their dataset. ShareGPT is not published and never obtains users' consent to share and release their data. It is unethical to redistribute data without proper permission. To be clear, this is a suggestion and not part of my paper evaluation.
2. It is not clear to me why related work is part of introduction?
3. From Figure 1, it looks like the only significant improvement of the model is on the reasoning category. Does the author have an explanation for why this is the case?
4. Figure 13 and 14 are not for readers to understand.
paper_summary: paper_summary: Paper proposes Nectar, a high quality open-source preference dataset using GPT-4 to rank responses. This datasets was made based on asking model to consider all pairings before reaching ranking. The dataset is around 180k prompts and with 7 responses each, reaches 3.8 million pairwise comparisons. These prompts are collected from LMSys-1M, Anthropic-HH , Ultrafeedback and ShareGPT. Compared to existing general datasets (excluding StackOverflow which is coding specific), the number of pairwise comparisons is an order of magnitude higher.

Responses are ranked (ordinally) on both helpfulness and harmlessness. Responses are by models like gpt-4, gpt-3.5, Mistral 7b and llama2 7b. Using this dataset, they train Starling 7B, which is the best 7B model on Chat Arena. This model increase MT Bench from 7.81 to 8.09 on MT Bench and 88.51 to 91.99 on 91.99 and Chat Arena ELO from 1072 to 1087. When trained with a larger reward model, the Chat Arena Elo increase further to 1118.
strength_weakness: strength_weakness: 1.	In Figure 6 right-most plot, while most of the positional bias from position 1 and 3-6 are eliminated, there is a curious higher win-rate for position 0 and lower win rate for position 2. Can the authors explain why this is the case? 
2.	Can the authors justify why Ordinal ranks were used rather than Cardinal as used in Ultrafeedback and HelpSteer dataset (see Wang et al. (2023) “HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM” http://arxiv.org/abs/2311.09528)? Cardinal scores (say between 1 to 10) can strictly convey more information (as we can construct ordinal ranks from cardinal scores but not the other way round) and does not require ties breaking as required in the ordinal approach. More importantly, cardinal scores can convey how much better a response is compared to other responses say [10, 1, 1,1,1,1,1] is pretty different from [10,9,8,8,7,7,7] even though they may have the same ordinal ranks.
3.	According to Figure 4, the top 4 models used (~180k each) are all from OpenAI (gpt4 and gpt3.5 variants) followed by mistral-7b and anthropic. This means that for any given prompt, there are two responses from gpt-4 and another two from gpt-3.5. Given the judge is also gpt-4, it is very likely to choose gpt-4 responses as the top rated response. Sharma et al. (2024) “A Critical Evaluation of AI Feedback for Aligning Large Language Models” https://arxiv.org/pdf/2402.12366 shows that much of gains from AI Feedback by GPT-4 can in fact be attributed to distillation from GPT-4. Relating back to this paper, this paper is missing a key ablation of simply doing SFT on top of the GPT-4 response to each prompt.
4.	Using MT Bench to select policy model checkpoint and also reporting as an policy model metric risks overfitting to this benchmark.
5.	Authors do not mention the specific LR and KL penalty used, which makes the RM and PPO experiments hard to reproduce.
6.	The authors mention using 10-20 challenging prompts for internal human evaluation but do not share what these prompts are and specifically how the evaluation is done. Similarly, they do not mention specific details for varies approach to checkpoint selection (such as what is the length threshold they used).
questions: questions: 1.	The choice of base models used with the Reward model is weird. Why did you choose to use llama 2 7b and then Yi 34B (as opposed to 2 models in the same family, which is available)? Specifically using Llama 2 7b might mean that the reward model can only be used to train models from the llama family (when the generally better-viewed Mistral 7b with Apache 2.0 license is available). 
2.	Please take care to correct grammatical errors (likey should be likely) and citation errors such as TRLX (Havrilla et al.) should be (Havrilla et al., 2023). These are plenty in the paper as it seems rushed and I trust the authors to correct them later on.
paper_summary: paper_summary: * The authors contribute to open source:
    * The Nectar dataset, a preference dataset comprised of 3.8M pairs, significantly larger than many other open source preference datasets like Ultrafeedback and Anthropic-HH
    * 7B and 34B reward models trained on this dataset
    * Two 7B models finetuned with PPO using the 7B & 34B RMs respectively from Openchat-3.5 (“Starling-LM-7B”) which achieves strong results on MT-Bench, AlpacaEval, and human evaluations
* The authors detail various tips and tricks for collecting the Nectar dataset (for example, ways to reduce the positional bias problem when using a model-as-judge)
* Three different methods for training a reward model are compared, and the authors find applies the maximum likelihood estimator under Plackett-Luce model gives the best results
* The authors also detail various tips and tricks for improving stability and training robustness of PPO such as:
    * Shifting the reward mean for length control, to prevent model from becoming too verbose
    * Pretraining the critic model to stabilize early stages of training
    * Full parameter tuning, vs. unfreezing only top layers
strength_weakness: strength_weakness: While the paper provides a detailed recipe for finetuning a model with PPO, the authors provide few comparisons to relevant baselines, for example (1) how does a reward model trained on the Nectar dataset compare to a reward model trained on some of the other open source datasets? (what exactly are we getting from the Nectar dataset over other datasets?) (2) how does the benchmark performance of the chat models compare to other similarly sized models?
questions: questions: - Formatting nits: table reference on pg. 2 is broken

Comments: title: Response to Rebuttal
Thanks for the rebuttal
comment: >Please include these updates into the revised paper.

Yes, we will definitely include all the changes into the revised paper.

>On the other hand, I still find that the amount of detail that the authors share in the paper and the rebuttal (excluding what they promise since some of it can be trivially answered in the rebuttal) is not sufficient for good replication of the training procedure. Especially since RLHF is known to be unstable, this makes replication challenging.

We appreciate your concern about replicability. We will release the full training code along with detailed documentation in the final revision. This will include the exact hyperparameters used, which should greatly facilitate replication efforts. We acknowledge that RL training can be noisy, but we believe providing these details will help others reproduce our results more consistently.

>It is also worrying that RM training on the Mistral 7B RM does not work well - authors did not bring this up initially and even after the rebuttal, offered very little data regarding how the Mistral 7B RM does (e.g. Rewardbench) , even though theoretically, a strong preference modeling dataset with millions of preference pairs should be able to train Mistral 7B as well as Llama2 7B. This potentially means that the preference dataset they introduce might only work on specific models, which is extremely surprising and should be discussed with some detail in the paper.

While it's surprising that our dataset did not yield similar results on Mistral 7B, we want to highlight that training Yi 34B as a reward model using our dataset led to excellent performance, surpassing even some larger models. This demonstrates the high quality of our preference data. We will explore potential reasons for the discrepancy between Mistral 7B and other models, and discuss the implications in the revised paper.
Thanks for the rebuttal, which answers some of my questions particular regarding why Ordinal ranks instead of Cardinal ranks, why the authors didn't include a GPT-4 SFT baseline and the use of MT Bench as checkpoint selection.

Please include these updates into the revised paper.

On the other hand, I still find that the amount of detail that the authors share in the paper and the rebuttal (excluding what they promise since some of it can be trivially answered in the rebuttal) is not sufficient for good replication of the training procedure. Especially since RLHF is known to be unstable, this makes replication challenging.

It is also worrying that RM training on the Mistral 7B RM does not work well - authors did not bring this up initially and even after the rebuttal, offered very little data regarding how the Mistral 7B RM does (e.g. Rewardbench) , even though theoretically, a strong preference modeling dataset with millions of preference pairs should be able to train Mistral 7B as well as Llama2 7B. This potentially means that the preference dataset they introduce might only work on specific models, which is extremely surprising and should be discussed with some detail in the paper. 

As impressive as the results are (that I mentioned in the Reasons to Accept), I find the lack of training details concerning. Therefore I will keep my scores the same.
Thanks for the rebuttal. Please do take these promised changes in the revision. Otherwise, I'd love to see this paper accepted.