Starling-7B: Improving Helpfulness and Harmlessness with RLAIF
Authors: Banghua Zhu
Evan Frick
Tianhao Wu
Hanlin Zhu
Karthik Ganesan
Wei-Lin Chiang
Jian Zhang
Jiantao Jiao
Venue: COLM (2024)
Keywords: High quality dataset, Alignment, RLHF
TL;DR: We present the high quality dataset Nectar, and open source reward model Starling-7B-RM and the current best 7B language model Starling-7B-LM

Abstract: This paper presents Starling-7B, the current best-performing 7B chat model on Chatbot Arena, along with its training dataset Nectar, a high-quality preference dataset collected by prompting GPT-4 to rank responses. We propose an internal pairwise rating technique, where the model considers all pairings before providing a ranking decision, leveraging the proven pairwise rating capability of LLMs without the cost of individual pairwise calls. The resulting Nectar dataset comprises 182,954 chat prompts, each with seven responses from various models, ranked by GPT-4, equating to 3.8 million high-quality pairwise comparisons. We introduce Starling-RM-7B and Starling-RM-34B, the reward model suites trained with a K-wise preference loss on Nectar, outperforming pairwise counterparts. We benchmark reward model training pipelines across metrics such as human preference, truthfulness, and safety. Using Nectar and our new training pipeline, we fine-tuned Openchat-3.5 to create Starling-LM-7B, achieving significant performance enhancements on MT-Bench, AlpacaEval, and human evaluation metrics. To facilitate research and understanding of RLHF mechanisms, we open-source the Nectar dataset, the reward models, and the language models.

Published as a conference paper at COLM 2024
Starling-7B:
Improving Helpfulness and Harmlessness with RLAIF
Banghua Zhu" UC Berkeley
Evan Frick UC Berkeley
Tianhao Wu" UC Berkeley
Hanlin Zhu UC Berkeley
Karthik Ganesan Nexusflow AI
Wei-Lin Chiang UC Berkeley
Jian Zhang Nexusflow AI
Jiantao Jiao UC Berkeley
Abstract
This paper presents Starling-7B, strong 7B chat model undergoing hu- man preference aligmment, along with its training dataset Nectar, a high- quality preference dataset collected by prompting GPT-4 to rank responses. We propose an internal pairwise rating technique, where the model con- siders all pairings before providing ranking decision, leveraging the proven pairwise rating capability of LLMs without the cost of individ- ual pairwise calls. The resulting Nectar dataset comprises 182,954 chat prompts, each with seven responses from various models, ranked by GPT-4, equating to 3.8 million high-quality pairwise comparisons. We introduce Starling-RM- RM-BB and Starling-RM-3A the reward model suites trained with K-wise preference loss Nectar, outperforming pairwise counter- parts. We benchmark reward model training pipelines across metrics such human preference, truthfulness, and safety. Using Nectar and our new training pipeline, we fine-tuned Openchat-3. 3.5 to create Starling-LM- achieving significant performance enhancements on MT-Bench, AlpacaE- val, Arena-hard and human evaluation metrics. To facilitate research and understanding of RLHF mechanisms, we open-source the Nectar dataset, the reward models, and the language models, along with their pipeline here: https:// /ttttbb/ com//eerrick200//trrrrngg
Introduction
Reinforcement learning from human feedback (RLHF) is pivotal technique in post-training that helps align large language models (LLMs) with human preferences. It has seen great success improving the helpfulness and harmlessness for several proprietary and open weight models, including Claude, GPT-4 (OpenAI 2022 2023a b) and Llama-2- vron et al. 2023).. However, the open science and open source community is not yet able to reproduce this success, potentially due to the lack of high-quality preference dataset and reward model, and unsuccessful hyperparameter tuning with reinforcement learning algorithms.
(Tou-
We aim to bridge the gap by open-sourcing Nectar and Starling model suites. Nectar is a GPT-4-labeled ranking dataset for training the reward model in RLHF. Starling model suites are a family of state-of-the-art open source reward models and chat models fine-tuned with reinforcement learning.
The post-training of LLMs is usually composed of two steps: supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). In the past, SFT has notably advanced the development of open LLMs, particularly with the utilization of high-quality data distilled from ChatGPT/GPT-4 This is demonstrated effectively by models such as Alpaca (Taori et al. 2023), Vicuna (Chiang et al., 2023), and Openchat-3.5 5 (Wang et al. 2023a). Another algorithm that gains popularity in the open-source community is direct
Equal contribution. Corresponding banghua@berkelley.ed

---

Published as a conference paper at COLM 2024
preference optimization (DPO), which updates the model in contrastive fashion based preference dataset. The method has created strong chat models including Zephyr-7B (Tun- stall et al. 2023 Cestral-7B-Instrune
[iang et al. 2023), and Tulu-2-DPO-TN (Ivison et al. 2023). However, the vanilla RLHF algorithm with reward training and proximal policy optimization (PPO) in Ouyang et al. (2022) has not produced strong open source models yet, despite its great success in proprietary models like GPT-4 (OpenAl 2022 2023a and
Claude.
Addressing the research gap in RLHF, key requirement lies within the provision of high-quality chat-specific ranking dataset. We introduce Nectar, unique ranking dataset labeled by GPT-4 which encompasses 182, 954 chat prompts. Each prompt comprises 7 re- sponses, distilled from a variety of models such as GPT-4, Mistral-77-Instruce and Llama2-7B--ae 7B-ct,, ranked by GPT-4. This equates to total of 3.8 million high-quality pairwise comparisons. We also introduce novel techniques reduce positional bias when prompting GPT-4 for rankings. This enables reinforcement learning from Al feedback (RLAIF), where we learn a proxy of human preferences from (poten- tially biased) GPT-4 ranking data. We compare the Nectar dataset with existing preference Bai et al. 2022a Ethayarajh et al. 2022) in Table !
GPT-3. 5-turbo,
dataset (Cui et al. 2023
#Frompt
Focus
Ranking
Ranking Type
Response
Comparisons
Nectar
183K
7
Diverse
Al
Ordinal
Strong + Weak L.M
38M
Ultrafoedbacc
64K
4
Diverse
Al
Cardinal
Strong Weak LLM
384K
Anthropic- III
161K
Safety
Human Human Human
Ordinal
Weak 11M
161K
Stack Overflow
20M
varies
Coding
Cardinal
Human
SHIP
385K
2
Diverse (Reddit)
Ordinal
Human
383K
Table 1: Existing RLHF datasets compared to Nectar
Furthermore, we open source our reward model suites, Starling-RM-78 and Starling Barriing-RM-4B. The models are trained using the K-wise loss (Zhu et al. (023) on the Nectar dataset. We benchmark various reward model training pipelines across metrics such as human preference, truthfulness, and safety in Section
To create the chat model Star1ing-LM--7-allpha we fine-tune Openchat-3.5 (Wang et al. 2023a) with proximal policy optimization (PPO) (Schulman al., 2017) on the learned reward model Starling-RM-27 As a result, the MT-Bench score improves from 7.81 to 8.09, while the AlpacaEval score improves from 88.51% to 91.99%, and human evaluation ELO increases from 1072 to 1087 on Chatbot Arena (Chiang et al. 2024). We also release version Starling MaarlinggLM--B-eee the initialized language model Openchat-3. 3. 5-0106, which increases the human evaluation ELO from 1089 to 1118 on Chatbot Arena. These metrics highlight the improvement of Starling-LM-27 in providing helpful responses.
based on larger reward model Star Starling-Me, RMBBBB and
Starling-LUt-B--kle al-7B-alpaa
OpenChat-3.5-78
Roleplay
Zephyr-78-ber 78-beta
Chat-7B
Haama-2-70-Choe
GPT-4
10
Extraction
Math
Coding
Figure 1: MT Bench Evaluation of the Starling-LM-7B-alphh model.
We hope the potential of these open-source contributions, including the dataset, reward model, and language model, enrich the understanding of RLHF mechanisms and fuel further alignment research.
2

---

Published as a conference paper at COLM 2024
2 Related Work
2.1 RLHF and RLAIF
Previous work has shown that human evaluators effectively provide reward signal to train agents in complex RL environments (Christiano et al. 2023). Specially, RLHF has shown immense effectiveness when applied aligning LLMs human preferences
OOuyang et al. 2022 Bai et al. 2022a OpenA1 2023a
2022 Touvron et al. 2023). Since
RLHF uses actual human preference data, data collection
very expensive bottleneck.
To mitigate this, recent efforts have tried RLAIF (Bai et al. 2022b Lee et al. 2023 ApenAT
2023a
Ma et al. (2023) utilizes the zero-shot performance of LLMs tor ranking tasks while
Petroni et al. (2019) explores LLMs' effectiveness as knowledge bases, which together may
provide a lower cost solution to mimic human preference via careful rubric design and
prompting.
2.2 Positional Bias and K-wise Rankings
Previous work has show that LLMs are not necessarily fair evaluators and exhibit significant positional bias (Wang et al., 2023b). As result, most LLMs human preference ranking strategies only use more reliable pairwise comparisons where ranking outputs are more stable QQin al. 2023). Black-box pure K-wise ranking strategies have struggled to find success, citing problems with outputs missing choices, repeating choices, inconsistently ranking, or being irrelevant (in al. (2023). Some effort generalize pairwise prompting to K-wise prompting has found success by utilizing multiple pairwise evaluations to sort K-wise ranking. This includes aggregating all pairwise comparisons or utilizing pairwise comparisons for heapsort (Qin et al. 2023). Using K choose 2 pairwise comparisons to create final K-wise ranking is not favorable due to O(?)) evaluator queries, O(K K) for sorting algorithms. Ideally, a K-wise ranking could be completed zero-shot with minimal positional bias with just single evaluator query. This paper present prompting framework that approaches this ideal.
Nectar Dataset
In this section, discuss the creation of the Nectar dataset. The dataset is composed of a set of prompts, with 7 responses for each prompt distilled from existing models, along with GPT-4-based ranking for the responses. We provide details about the prompt collecting, response collection, and response ranking respectively.
3.1 Collecting Prompts
We aim to curate dataset with diverse prompts from different sources, including adversarial jailbreaking prompts. The collected prompts are used for both reward learning and policy learning in RLHF. We collect data from range of existing validated datasets,, including ShareGPT, Anthropic-HHH Chat-1M (Zheng et al., 2023a). We first sub-sample prompts from each different category, and then combine the same prompts and their corresponding model model-generated responses, resulting in a dataset with 182, 954 prompts. The composition of the prompts is provided in Table
[Bai et al. 2022a), UltraFeedback (Cui et al. (023), and Lmsys-
Table Number of Prompts From Each Source
Source
Anthropic-HI
Lmsys-Chat-IM
UltraFeedback
ShareGPT
Count
74097
43545
40411
25113
3.2 Collecting Responses
For each prompt in the dataset, we distilled a response from GPT-4, GPT-4-0613, GPT-3.5- Turbo, GPT-a..-Turbb-Insttre Mistrrl--B-instruut,, and Llahh-2-7B-chat. Each prompt also
3

---

Published as a conference paper at COLM 2024
comes with default response in the corresponding dataset. As result, we get dataset with 182,968 prompts and 7 responses per prompt. GPT-4-0613 was given a system prompt to help set the chatting context yielding better responses. GPT-3.5-lurboIIstrrue was chosen for the lack of extensive human feedback alignment, unlike many of the other models present, leading it to be more likely to comply with jailbreak prompts with high quality responses, We provide more detailed analysis on the distribution of the model responses in Appendix
3.3 Collecting Ranking Data
Given some ranking rubric and user prompt from the prompt dataset, we ask GPT-4 to rank the 7 responses from best to worst in accordance with rubric. The goal of the rating prompt was to capture bimodal rubric strategy designed to balance helpfulness and harmlessness: Maximally answer good-natured conversations and harmlessly answer all conversations. We create the following rubric:
The helpfulness rubric is as follows:
1. Is the completion relevant? The completion should be the best possible assistant answer.
Is the completion accurate? The completion should accurately fulfill the prompt's 's request.
creative answer is expected, is the completion creative? If an analytical answer is expected, is the completion factual/objectivee objectively correct?
Is the completion detailed? The completion should at minimum satisfy the full level of detail required by the prompt.
The harmlessness rubric as follows:
1. Does the completion adhere to moral and ethical standards?
Is the completion truthful?
3. Is the response polite and respectful?
4. If the request given is not good natured, does the completion politely decline to reply while giving a short explanation why they cannot comply with the request? Completions should always maintain harm- lessness in the face of adversarial requests.
Our objective is to de-emphasize the role of helpfulness in the evaluation of responses to prompts of negative nature, intending primarily, maintain harmlessness. This approach is designed to prevent harmful outputs from obtaining higher scores than their harmless equivalents, simply because they adhere more closely the instructions. Conversely, when dealing with standard prompts, we rank the responses purely based on the helpfulness.
As one of the major challenges in ranking responses, we observe very serious positional bias in GPT-4 when ranking more than 2 responses. We discuss the findings in the section below.
3.3.1 Positional Bias
The notion of positional bias has been studied in a pairwise context, with results indicating that the first or the second index could be favored, contingent on the ranking model deployed Wang al. (202bb). We have similar observations in our K-wise rating settings.
For each prompt, we shuffle the 7 responses uniformly at random and prompt GPT-4 to provide a list-wise ranking. We observe that GPT-4 exhibited a strong preference for selecting responses seen at the beginning. Such responses won at a rate ten times greater than completions viewed at the sixth index (See Figure when K 7). These results are consistent with earlier observations, implving that GPT-4 has inherent tendency to favor the first several responses (Wang et al. 2023b). Without effective strategies to address this bias, it becomes evident that carlier responses are overwhelmingly preferred over later ones.
4

---

Published as a conference paper at COLM 2024
Moreover, observations indicate that this positional bias worsens K increases unless remedial actions are taken.
Winning Response Index Distribution with naive prompt: positional bias for different values of K for K-wise comparisons. (n=200)
Figure 2:
Figure visualizes the increasing positional bias problem for larger K. While in pairwise rating situations, the bias manageable, larger K-wise rating procedures are unusable. In the following sections, we explore various strategies to combat the positional bias problem for large K.
We introduce novel prompting techniques to mitigate the positional bias by first generating all pairwise rankings, and then summarizing the results into list-wise rankings. We leave the details to Appendix A.2 We also leave additional analysis, including model pairwise win rates and pairwise average ranking differential heatmaps to Appendix
Reward Learning
4.1 Formulation
In this section, we discuss the training of the reward model. The reward learning procedure in RLHF can be formulated in contextual bandit environment. In contextual bandit, for the i-th sample, a state (prompt) s¹ is first sampled from some fixed distribution p. Given the state s', K actions (d .t. -
are sampled from some joint distribution PP ,ak.,aaak 1 s¹). Let of [K] + [K] be the output of the human labeller, which permutation function that denotes the ranking of the actions. Here (0) represents the most preferred action. We use 4g 01 to denote the event that the action do more preferred compared to #1. A common model on the distribution of o' under K-ary comparisons is Plackett-Luce model (Plackett 1975 Luce 2012). The Plackett-Luce model defines the probability of state-action pair (s, a;) being the largest among a given set {s) K- K-1
as
exp(re (s,
P(a: > vjtis)) Σ exp(ro (s,
Here rg- is the ground truth reward function with parameter 0* Moreover, one can calculate the probability of observing the permutation as
Po s, (a,) = II exp(re- (s)))()))
exp(re (5,)))
When = this reduces to the pairwise comparison considered in the Bradley-Terry-Luc (BTL) model (Bradley Terry] 1952). In this case, the permutation can be reduced to Bernoulli random variable, representing whether Ag is preferred compared to aj. Con- cretely, for each queried state-actions pair (s, (g, R1), we observe a sample y from a Bernoulli exp(rx exp(ra (5Ao)) (s.a))) i.e. for any E {0,1),
distribution with parameter
exp (s,Ri))
PP( s, 40, #1) exp(ro- (5, + exp (0 (s,4)))
In practice, one may introduce extra temperature parameter y and replace all with re- /v. Here take y = 1.

---

Published as a conference paper at COLM 2024
By observing the sampled dataset D = = ab dj ... ok= would like to estimate the ground truth reward re- with some proxy reward rg. We discuss in the following section some different choices estimators we benchmark in the paper.
4.2 Methods
Similar to the default methodology of training reward models Ziegler et al. (2019); Ouyang ;
al. (202)) Schulman et al. 2022), we remove the last layer of a pre-trained language model and concatenate with linear layer for reward prediction. We take Llama-2-7B-Char (Tou vion et al. 2023) as the base model for reward model. Observing the various performances of the 7B Jlama-based reward models, we also train a 34B reward model utilizing Yi-34B-Chat as the base model (AI et al. 2024). The reward model takes in prompt and response, and outputs scalar representing whether the response is helpful and harmless given the prompt.
For learning the reward from K-wise comparisons, we consider three methods. The first method is the original method from Ziegler et al. (2019); Ouyang et al. (0022) Schulman et al. (2022), which decomposes each K-wise comparison into K(K 1)/2 pairs of pairwise comparisons, and then minimize the cross entropy loss. When there only pairwise com- parison, this is the maximum likelihood estimator under Bradley-Terry-Lee
model (Bradley
Terry 1952). It has also been shown to converge to the ground truth when the reward model is linear and well-specified with K 2 2 (Zhu et al. 2023).
PMLE2 E arg min (D(),,
y
1
K-1 log
exp(r(s', (( exp(r(st, + exp(r(ss (())))
where (D(r)
11
k=j+1
The second method directly applies the maximum likelihood estimator under Plackett-Luce model, which is shown in Zhu et (2023) to be asymptotically more efficient than ÊME.
MMEX E arg min (D(r),
exp(r(s', (g)))) Σ exp(r(st, mr)0))))
where (p(r)
log
The third method is the iterative data smoothing algorithm proposed in (Zhu et al., 2024), which iteratively updates the model with the data, and then updates the soft data label with the model after each epoch to prevent overfitting to the noisy data. We denote the final estimator as smooth
4.3 Reward Model Evaluation
In this section, we benchmark the results of the three methods detailed 4.2 we report the result of training a larger base model.
Additionally,
Model
Human Acc.
Truth Acc.
Safety Acc.
Verbose Acc.
MLL, (7B)
0.763
0.647
0.759
0.167
MLE,, (7B)
0.751
0.636
0.729
0.133
Smooth (7B)
0.761
0.598
0.737
0.100
?LE (34B)
0.807
0.712
0.782
T0.36
Table 3: Human, truth, safety, and verbosity accuracy for each reward model.

---

Published as a conference paper at COLM 2024
Model
Human Loss
Truth Loss
Safety Loss
Verbose Loss
fMLEx (7B)
0.552
1.881
0.508
2.365
/MLE2 (7B)
0.704
1.933
0.620
3.327
smooth (7B)
0.537
2.043
0.562
1.956
MLEx (34B)
0.468
1.766
0.48
1.461
Table 4: Human, truth, safety, and verbosity loss for each reward model.
We evaluate our trained reward models across four different categories: Human, truth, safety, and verbosity. Evaluation accuracy is measured as Σ (P(Bsst) > f(Other)). Evaluation loss is measured with log loss over the softmax probabilities arriving from the reward values:
N Elog(
((Best)
In K-wise case where K 2, the K-wise loss is sum of the pairwise
(BBstt) ((OOter)
loss for each (5) pairings. The accuracy and loss results for each trained reward model are
shown in Table 3 and Table respectively. Additional details on these evaluation metrics and what they represent can found Appendix B.1 Note, the models chosen for release as Starl ing-innn---- and Starling-RM-2 are the 7B and 34B models trained under the /MLE, formulation, respectively, as we find fMLE, to produces the strongest reward models in accordance to the evaluations. Additionally, likely result of the significantly stronger base model, Starling-RM Baang-RM-344 348 (f (LL, (34B)) exceeds all 7B counterparts in performance for every evaluation metric in Table 3 ann Table
4.4 Other Benchmarks
Model
Chat
Chat Hard
Safety
Reasoning
Average
Starling-RM-34B
96.9
57.2
88.2
88.5
82.7
Tulu-2-DPO-70B
97.5
60.5
83.9
74.1
79.0
Mixtral-sx7B-IInsrrct---0
95.0
64.0
73.4
78.7
77.8
Mous--eermms----iisrra--------
92.2
60.5
82.3
73.8
77.2
Zephyr-78-alpha
91.6
62.5
74.3
75.1
75.9
Starling-RM-7B-alphha
98.0
45.8
85.8
57.4
71.8
oasst-rm-2.
88.5
48.5
65.3
78.0
70.1
U1traRM-13B Beaver-7B-v1.0-ces
96.1
58.6
54.3
65.4
68.6
60.9
45.0
81.5
46.7
58.5
Table 5: Reward Bench scores for various models including Starling-RM-22 and
Starling-RM-78.
On Reward Bench, Starling-RM- achieves state-of-the art performance (Lambert al., 2024). Most notably in the "Reasoning" category on Reward Bench, Maarlinng---44 RM-34B has a large improvement over Morrinng-R-788 suggesting that larger and more capable base models may help the reward model's performance in downstream complex reasoning tasks. On the contrary, the "Chat" category performance seems saturate quickly. Additionally, both Starling models surpass all other models in the "Safety" category, including models specifically trained for safety such as (Dai et al. 2023)'s 's Beaver-7B-vl 0-Cost. These results indicate that the inclusion of prompts of good and bad nature along with a judging rubric that enforces harmlessness despite helpfulness yields safer models on par with models more solely focused harmlessness- all while maintaining exceptional performance on helpfulness categories.
Policy Learning
In policy learning stage, we proximal policy optimization (PPO) Schulman et al. 2017 to fine-tune the language model based on the learned reward model. To create the chat model Starling-LM-7B-alpg learned reward model Marrling-RM-7B-- RM-a--alppa As result, the M1-Bench score improves from 7.81 to 8.09, while the AlpacaEval score improves from 88.51% 91.99% and human
fine-tune Openchat-3.5 (Wang al. 2023a) with the
7

---

Published as a conference paper at COLM 2024
evaluation ELO increases from 1072 to 1087 on Chatbot Arena (Chiang et al. 2024 We also release a new version Starling-LM-78--o by fine-tuning 9penchat-3.5-4 016 on the larger reward model Starling-RM RM-344B, which increases the human evaluation ELO from 1089 to 1118 on Chatbot Arena.
5.1 Implementation Details
We implement the PPO algorithm using the TRLX (Havrilla et 2023), incorporating all standard tricks of PPO as documented in Huang et (2022). our implementation,
we decouple the actor (the LM) and the critic, allowing for independent gradient updates. We find out that PPO is highly unstable. For example, after exposure to only 1000-2000 prompts, the model rapidly learns to generate excessively verbose outputs. To mitigate this instability and improve training robustness, we propose the following techniques:
During our experiments, we observe that the reward model may assign highly negative rewards to the initial actor's 's outputs. Although the absolute magnitude of the reward does not convey preference information, and only the relative difference in reward between two responses indicates preference (Wu et al. 2023), the overall reward magnitude significantly affects the generated response length. We find that a very negative reward can cause the model to become excessively verbose after just few gradient updates. To address this issue, we propose adding a constant to the reward to make it slightly positive. The intuition behind this approach is as follows: during the initial gradient steps, we should increase the likelihood of the end-of-sequence (EOS) token, which is equivalent to making the response shorter. We can estimate the ad advantage of the EOS token as (Teas + Veos) r pree, where Tprev is the reward assigned to the token before the EOS token (usually the negative KL penalty), Tros is the reward scored by the reward model, and Vaos is the critic's 's value estimate for the EOS token. During initialization, we know that Tprev and Veos are approximately zero, slightly positive Too will penalize the length. We observe that after applying the reward mean shift, the model initially generates slightly shorter responses and then gradually increases the response length training progresses, driven by the verbosity preference in the reward.
Shifting the reward mean for length control:
Pretraining the critic model: During the initial stages of training, we observe that ran- domly initialized critic (initialized from the language model with a linear layer and Gaussian- initialized head) can negatively impact the early performance of the language model. This is evident from slight decrease in the MT-Bench score during the first 30% of the training process. To minimize early performance degradation, we propose pretraining the critic. We begin by conducting an RL run with a randomly initialized critic and the supervised fine-tuned (SFT) actor. We then use the final critic model from this run as the initialization for new run with the same SFT actor. With this modification, we observe that the actor optimizes the reward more rapidly compared to the case without critic model pretraining. This approach helps to stabilize the early stages of training and allows the actor to more effectively optimize the reward signal.
Full parameter tuning yields the best results: In our early experiments, we initialize both the actor and critic from the supervised fine-tuned (SFT) model, while unfreezing only the top 4 layers of the actor and critic during the RL stage. Although this approach improves the MT-Bench score from 7.81 to 8.09, the improvement is significantly smaller compared to initializing the actor and critic separately and performing full parameter tuning. By initializing the actor and critic separately and tuning all parameters, we can improve the score from 7.81 8.33, starting from the same SFT model. While MT-Bench serves as a proxy for model performance, our internal human evaluation also confirms that the full parameter tuned model is more preferred by human raters. This finding highlights the importance of full parameter tuning in achieving the best possible performance gains during the RL stage.
5.2 Hyperparameter Tuning
We perform extensive dyper-parameter searching to find the best configuration. We mainly focus on tuning the learning rate, training batch size, and KL penalty. These hyperparameter choices aim to strike a balance between model performance, and computational efficiency
8

---

Published as a conference paper at COLM 2024
while minimizing undesirable behaviors such over-optimization and excessive response length.
Our findings suggest that slightly larger learning rate can be beneficial in reducing the over-optimization issue. When using a small learning rate and longer training duration, the exploration in online RL may generate unusual responses that still receive high rewards due to the discrepancy between the ground truth reward and the proxy reward.
Additionally, observe that slightly larger KL penalty helps to stabilize the generated response length, although it can only mitigate the increase in length and not completely prevent it. The final run uses an KL penalty 0.01 and learning rate 10-6, We also find that extremely large batch sizes do not yield observable improvements in human evaluation. Therefore, we utilize a medium batch size to facilitate faster iteration and experimentation, sampling 512 responses form the replay buffer. Furthermore, the policy and critic updates are performed using a micro-batch size of 32.
5.3 Checkpoint Selection
We adopt multi-layer checkpoint filtering strategy, where checkpoint must pass all the previous filters to be considered as a candidate for final release. Our criteria are as follows:
We require the model achieve a predefined reward thresh- old while maintaining KL-divergence lower than a specified KL threshold. This ensures that the model improves its performance without deviating too far from the base model's output distribution.
KL-Reward Trade-off:
Verbosity Monitoring: We observe that the reward model often prefers longer responses, which can lead the language model to generate unnecessarily verbose explanations or even repeated outputs. To mitigate this issue, we monitor the average generated response length on held-out validation set of prompts during training and filter out checkpoints that exceed a predefined length threshold.
Measuring InttructionFFollwiit Capability: We directly measure the model's ability follow instructions using IFEval (Zhou et al. 2023). IFEval focuses on 25 types of "verifiable instructions" such as "write in more than 400 words" and "mention the keyword 'Al' least 3 times." " We find that steering the language model toward human preferences with RLHF can sometimes compromise its ability to precisely follow instructions. To address this, we predefine a threshold based on the base model's score and ensure that the checkpoint's score does not fall below this threshold.
MT-Bench Evaluation: For checkpoints that pass all the previous tests, we evaluate the model using GPT-4 as proxy. MT-Bench is well-understood proxy for human preference; the evaluation process involves generating responses to a fixed set of prompts spanning diverse range of topics and using GPT-4 to score the responses. We find that PPO can significantly increase the MT-Bench score. We set a reasonably high MT-Bench threshold, we observe that scores higher than this threshold do not necessarily indicate a true human-perceivable performance difference.
Internal Human Evaluation: We create a fixed set of 10-20 challenging prompts that assess the model's reasoning, instruction-followiigg and verbosity. We then judge the quality of the generated responses through internal human evaluation.
Conclusion
In this paper, we produce the open-source RLHF pipeline that improves the helpfulness and harmlessness of the chat model. We release Nectar, first-of-its-kind open-source high-quality 7-wise ranking dataset, and encourage future work to further explore the potentials of this dataset. Furthermore, we explore the effects of various reward training formulations on reward model performance. Finally, we perform extensive hyperparameter tuning on PPO to maximize performance gain in the RL stage. Ultimately, we hope to

---

Published as a conference paper at COLM 2024
Response ength
Reward Ovpr-mpphinnzaiion
7,000 e model + 75
1100
moddd 7.2
Resporse Lengit
Change in Rewaar
3.00
600
307
000
20
Steps:
Figure 3: (Left) Impact of reward shifting constant on response length. With no reward shifting (green), the initial actor starts with a reward around -7.3, Three different reward shifting parameters are tested: O (no shift), 7.2 (starting with slightly negative reward), and 7.5 (starting with a slightly positive reward). The results show that starting with slightly positive reward can better control the response length while achieving the same final reward. (Right) Optimal run with base model Openchat-3. 5-0106.. We evaluate the model's performance using two metrics: (Orange) Using Starling-RM-23 generated from a fixed validation prompt set at every step. A step consists of creating replay buffer of length 512 and performing 16 gradient updates with micro-batch size of 32; (Blue) Evaluating the language model on MT-Bench every 5 steps. Our findings show that while the reward measured by the reward model increases throughout the training process, the MT-Bench score, which is considered a better proxy for human preference, starts to decrease after step 25.
to score the responses
promote open source research and democratize the access of strong LLMs by releasing the dataset, methods, and the models trained from our RLHF procedure. We show that our data and training pipelines are capable of creating state-of-the-art reward and language models, pushing the limits of open-source LLMs.
There are also limitations with respect to the data and methodology, which may be important as future work. The Nectar dataset is collected purely from GPT-4-based preference, which may contain significant bias that is inconsistent with true human preferences. It would be worth exploring methods that mitigate biases from synthetic data. In addition, Nectar focuses more the helpfulness and harmlessness of the responses and less on the instruction following property. As such, according to our observations, the resulting language model can be less capable following exact instructions. There may also be improved reward training and policy learning algorithms leading to better reward models and language models. We hope future work can explore these areas in greater depth.
10

---

Published as a conference paper at COLM 2024
References
0l, ;, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng
Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield- Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy (elleen-Lawtowd Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac HatfieldDoddss Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from feedback. 2022b.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, (9) -452 1952.
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating 1lms by human preference. ar Xiv preprint drXio:2403.04132, 2024.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. 2023.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high- quality feedback, 2023.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf. Safe reinforcement learning from human feedback, 2023.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In International Conference on Machine Learning, pp. 5988 5988-6008. PMLR, 2022.
Alexander Havrilla, Maksym Zhuravinskyi, Duy Phung, Aman Tiwari, Jonathan Tow, Stella Biderman, Quentin Anthony, and Louis Castricato. trlX: A framework for large scale reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
11

---

Published as a conference paper at COLM 2024
Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimiza- tion. In ICLR Blog Track, 2022. URL httpss/s/iccr--bbggtttt....iiiit 25/ppo- imp hew.p//mmmmetatttttteaaa.... http:////acabbogg.aaaa////o///// ppo implenentation-dctllss...
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in changing climate: Enhancing Im adaptation with tulu 2, 2023.
Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via human-preferene dataset. 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. Xiv preprint ar arXiv:2310.06822. 2023.
Sandipan Kundu, Yuntao Bai, Saurav Kadavath, Amanda Askell, Andrew Callahan, Anna Chen, Anna Goldie, Avital Balwit, Azalia Mirhoseini, Brayden McLean, Catherine Olsson, Cassie Evraets, Eli Tran-Johnson, Esin Durmus, Ethan Perez, Jackson Kernion, Jamie Kerr, Kamal Ndousse, Karina Nguyen, Nelson Elhage, Newton Cheng, Nicholas Schiefer, Nova DasSarma, Oliver Rausch, Robin Larson, Shannon Yang, Shauna Kravec, Timothy Telleen-Lawton, Thomas I. Liao, Tom Henighan, Tristan Hume, Zac Haffield-oodds, Sören Mindermann, Nicholas Joseph, Sam McCandlish, and Jared Kaplan. Specific versus general principles for constitutional ai. 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khy- athi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling. 2024.
https://hugginggface cotppeccss/cce///ccceeeec attpe/////aaeeeeaaaaaaaa///erere
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2021.
R Duncan Luce. Individual choice behavior: A theoretical analysis. Courier Corporation, 2012.
Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. Zero-shot listwise document reranking with large language model. 2023.
OpenAI. Introducing chatgpt. http::///openai.co///boo///cccee 2022. (Accessed on 01 01/12/2024).
OpenAL. Gpt-4 technical report, 2023a.
OpenAL. New models and developer products announced at devday. httpp://openai. com/
htte//eww-oodelle and and--deeellperrpdduuutt-----e wwwpruuuuuccttnooonneeeeee-eeeee awwounced-at--oeeeeeee devdayy 2023b. (Accessed on 01/12/2024).
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27700-27744, 2022.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language models as knowledge bases? 2019.
Robin Plackett. The analysis of permutations, Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193 202, 1975.
12

---

Published as a conference paper at COLM 2024
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky, Large language models are effective text rankers with pairwise ranking prompting. 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint ar arXio:1707.066:: 2017.
John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Felipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, et al. Chatgpt: Optimizing language models for dialogue. OpenAl blog, 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. http::///gthhbb.oo//aasssaaaaa/a .atabb/staoood...m alpaca 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Open- chat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309. 112355 2023a.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. 2023b.
Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. Pairwise proximal policy optimization: Harnessing relative feedback for Ilm align- ment, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. Xing, Joseph Gonzalez, lon Stoica, and Hao Zhang. Lmsys-chat-lm: A large-scale real-world Ilm conversation dataset. 2023a.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023b.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-folloiing evaluation for large language models. arxiv preprint arXiv:2311.07991 2023.
Banghua Zhu, Jiantao Jiao, and Michael I Jordan. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. ar Xiv preprint ar arXio:2301.11270, 2023.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. ar preprint arXiv:2401.16333 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. Xío preprint Xii::199.08899) 2019.
13

---

Published as a conference paper at COLM 2024
Appendix
A Details on the Nectar Dataset
A.1 Distribution of the model responses
We show in Figure aadd the distribution of responses from each model and the number of turns in each prompt.
Number responses from each model
200000
175000
150000
125000
count
100000
50000
25000
geac S-turbo
30
30
sarchald dio setttt
30 marii
Aur voingg
Jami lamid
Model Name
Figure 4: Distribution of responses from each model. Of the most represented models, GPT-3.5-Turbo, GPT-4, T.Tt3.--Turo-Instruue were all distilled specifically for this dataset. Other model responses were provided by the dataset prompt sources (some of which may also be from the aforementioned models).
Mistral-7B-Instruct and Llama-2-7B-CII
Number of conversations of each turn (log scale)
105
10c
10a
10a
104
109
2
10
11
12
13
14
15
16
17
18
19
20
21
24
26
28
32
33
34
turns
Figure Distribution of number of turns in each prompt. Prompts are structured as follows "Human: [user text] Assistant: [model response] ... Human: [user text] Assistant:" Where the human and assistant converse for any number of turns. All multi-turn prompts are from Anthropic-HH.
14

---

Published as a conference paper at COLM 2024
A.2 Positional Bias Mitigation
A.2.1 Internal Pairwise Rating
In order to leverage the proven pairwise rating capability of LLM ratings without the cost of many individual pairwise calls, we propose internal pairwise rating, where we prompt
the model to consider all (5) pairings first, before providing ranking decision. We go further by providing an explicit K-wise order in which to complete each pairwise rating between two different responses, eg. [(1,2) (1,3), ..., (5, (6,7), We also try randomizing the pairwise rating order to further reduce any prompt induced positional bias.
Specifically, the four strategies are as follows:
No Pairwise Evaluation The prompt does not ask for any pairwise evaluation strategy.
Pairwise Evaluation : The prompt asks the ranker to first evaluate each possible pairing,
then generate a final overall ranking.
Enforced Pairwise Order The prompt explicitly provides each pair to evaluate, eg. (1,2), ..., (6, 7) The ranker must give winner for each pair in the order pro- vided, then produce an overall ranking.
Enforced Random Pairwise Order The prompt is the same as Enforced Pairwise Order, but the pairwise order is randomized independently for each ranking instance. Again, the ranker must give a winner for each pair in the order provided, then produce an overall ranking.
boecos
Seerdd
2.1%
216
6.00
Figure 6: Winning Response Index Distribution: positional bias for different prompting strategies. (n=300)
Figure that in the 7-wise case used for Nectar, this internal pairwise strategy greatly reduces positional bias, especially when randomized pairwise ordering is enforced. Note that these four prompting strategies all essentially use the same amount of tokens, since only one query is made for each rating in all cases, making internal pairwise prompting token-efficient method to reduce positional bias.
Additionally, Figure 7 gives the positional biases for different K when running K-wise comparisons with internal pairwise ratings. We find that the success of this prompting strategy is more noticeable for K > 5. Smaller K seem to not interact well with this more complex prompting strategy.
Figure 7: Winning Response Index Distribution with a pairwise enforcing prompt: positional bias for different values of K for K-wise comparisons. (n=200)
15

---

Published as a conference paper at COLM 2024
A.2.2 Tie Breaking
In their study, Wang et al. 2023b) deduced that a smaller disparity in quality between two responses could intensity the effect of positional bias. Seeing the K-wise prompt technique relies upon pairwise comparisons, its vulnerability to the influence of positional bias could be conjectured as similar. Particularly in instances where the quality of responses is closely matched, the strength of positional bias is amplified. Hence, we made a concerted attempt to examine the influence of positional bias on near equal responses.
As a baseline, we created ratings by utilizing an enforced randomized pairwise strategy with no specified tiebreak strategy, Subsequently, we developed new prompt enforcing tie-breaking strategy (for instance, [ > C> D A > ...] or "Break tie randomly") for each rating.
Whether the tie-breaking rule is randomized or deterministic, in the event of a tie, the model prompted to adhere to the randomly generated tie-breaking rule rather than making an unprompted "random" decision.
positional bias solely impacted responses that were precisely tied, then the invocation of tie-break strategy would normalize the distribution of winners, aligning it more closely with the anticipated value. Conversely, should positional bias affect the model's intrinsic quality assessment of each response and in turn, blur the distinctions between the quality of responses, the effect of incorporating a tie-break strategy would be negligible.
Thaamal Strttgy
Techwer Sritag
"Sander"
Thtmmal dtattgy kandar brder
Thkamal Strategg Strt Claen Codar
Eopacodd Valce
Draecied Hatu
EopAtiec sole
0.5
2.00
3
2kg
Figure 8: Winning Index Distribution for Different Tre-bne Strategies. None means no strategy is specified. "Random" means GPT-4 is instructed to break ties randomly. "Random Given Order" means GPT-4 is follow a given randomly generated tie-break order. "Static Given Order" means GPT-4 is to follow given static order designed to help observably underrepresented indices.
Figure 8 indicates a marginal positive impact of tie-breaking on the distribution of indices. This slight improvement is characterized by decrease in the overrepresentation of the 0 index, coupled with an mitigating underrepressentaiion of the 2 index. Notwithstanding this adjustment in distribution, it is notable that it still reveals certain degree of bias. The Static Given Order seems to cause undesirable disparities in middle indices, while both Random Given Order and "Random" provide very similar results, with the latter mitigating underrepresentation in the 2 index.
Overall, the disparity in positional bias, whether the ties are broken "randomly" or in given random order order, appears to be minimal-bar but always is better than providing no specific tie-breaking instructions.
In light of these observations, we posit that the issue of positional bias is compound manifestation of both proposed hypotheses. Although the procedure of randomized tie- breaking fosters more proportional distribution, it does not completely mitigate the issue of positional bias. Consequently, this suggests the intrinsic nature of positional bias as fundamentally influencing the evaluator's 's scoring of given response. However, although positional bias is rather hard to completely mitigate, our prompting greatly helps control the reducible part of positional bias.
16

---

Published as a conference paper at COLM 2024
A.2.3 K-wise and Pairwise Agreement
Previous research has indicated that the pairwise rating generated by the LLM of substan- tial quality (Qin et al. 2023). Consequently, we employ K-wise to pairwise agreement as an evaluation metric.
The process of benchmarking is stated below:
1. Establish a K-wise rating, yielding K-ordering, For each pairwise neighbour in this rank ordering, assess if the order matches.
2. To evaluate different K values, we always initiate with a maximum K, derive an ordering, and then remove either the top bottom K to decrease K K- 1.
(1) is implemented, given that pairwise neighbours represent the most challenging pairwise rating task. Referring (Wang al. 2023b), larger gaps in response quality contribute to reduced positional bias. Therefore, in evaluating the effectiveness of K-wise relative to pairwise, it is adequate to measure only the agreement of the neighbours of each response.
(2) is conducted to ensure comparable difficulty of the K d ordering. Selecting K randomly from K responses would, in expectation, result larger rating gaps between responses, simplifying the K - d rating task. This effect becomes pronounced as K 2. Removing the top-ranking or bottom-ranking response each cycle ensures that the responses as proximate in ranking distance the original K responses.
We evaluated performance in these benchmarks using two different parameters:
Shuffle:
Shuffling involves presenting the options the pairwise rater in random se- quence. Without shuffling, the pairwise rater encounters the options in the same sequence as the K-wise rater did.
Explain: Instructing the K-wise rater to explicate each internal pairwise rating decision represents 'Explain'. 'No explain' means the rater isn't asked to elucidate rating decisions. The pairwise rater is never asked to clarify its decision.
17

---

Published as a conference paper at COLM 2024
Parwise Agreement Error vs K
Shuffle
0.40
False
True
0.35
Explain
Pairwise Agreement Error
True
0.30
False
0.25
0.20
0.15
0.10
0.05
4
5
7
Figure 9: The most considerable deviation in pairwise agreement occurs during shuffling, revealing the impact of positional bias. The difference between 'Explaining' and 'Not Explaining' isn't generally significant. Note there isn't any remarkable difference when the pairwise rater is obliged to explain.
A.2.4 Final Prompt Design
The final chosen prompt uses randomized enforced pairwise rating order before the overall 7-wise rating, no enforced tie-breaking order, and no explanations for ratings. The final prompt is detailed below.
18

---

Published as a conference paper at COLM 2024
Example Final Ranking Prompt
SYSTEM:
We are interested in ranking different large language model chat completions to conversation. Please act as impartial judge and evaluate the quality of the completions provided by Al assistants.
We have rubrics, one is the helpfulness rubric, one the harmlessness rubric.
The helpfulness rubric is as follows:
1. Is the completion relevant? The completion should be the best possible assistant answer. Is the completion accurate? The completion should accurately fulfill the prompt's request. a creative answer expected, is the completion creative? If an analytical answer expected, is the completion factual/ factual/objectively correct? 4. Is the completion detailed? The completion should at minimum satisfy the full level of detail required by the prompt.
The harmlessness rubric is as follows:
1. Does the completion adhere to moral and ethical standards?
2. Is the completion truthful?
Is the response polite and respectful?
the request given not good natured, does the completion politely decline to reply while giving a short explanation why they cannot comply with the request? Completions should always maintain harmlessness in the face of adversarial requests.
We may see conversations that are good natured and some that are not good natured. All conversations should be completed a manner that aligns with the harmlessness rubric. Good natured conversations additionally should align with the helpfulness rubric.
You will see a conversation with one more turns between human and an assistant of arbitrary nature, and then there will be 7 different answers from different models. This will be labeled with "INPUT:" Note that there is no order which the models responses are provided; each response should be considered equally regardless After the label "OUTPUT:" it is your job to first identify if this is good natured conversation not good natured conversation. Then, for each pairwise comparison between model completions, consider each option in the pair equally, then in accordance with the relevant rubric(s), declare pairwise winner, break ties randomly. There will be an ordering to do the pairwise comparisons labeled in the input as "PAIRWISE EVALUATION ORDER:" strictly follow this ordering.
its position.
Finally, considering these pairwise rankings, please rank all 7 responses in accordance with their pairwise performance from best worst, strictly in the following format: - ''3
'' ".!?? [-]] where contains the letter associated with model. Break ties randomly. Avoid any positional biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be objective possible.
USER:
INPUT: [CONVERSATION STAKT]:
Human: Can you make list of citrus fruits?
Assistant: [Possible input includes]] Blood orange, Grapefruit, Mandarin orange, and kumquat.
Human: Okay. Can you add mroe?
Assistant:
[CONVERSATION END]
[MODEL A RESPONSE START]: [Possible output includes: Blood orange, Grapefruit, Man- darin orange, Kumquat, and Limone. [MODEL A RESPONSE END]
Certainly! Here are some more citrus fruits: Lemon, Lime, Tangerine, Clementine, Pomelo, Yuzu, Key lime, and Calamondin. [MODEL RESPONSE END]
[MODEL B RESPONSE START]:
MODELL G RESPONSE END]
PAIRWISE EVALUATION ORDER: [['D', (D, (F) ('C', 'E), ('A', 'F), ('A', 'B', C. (DD', (), (BB, DD), ('A, 'E), (AA,, 'G), (B) OUTPUT:
(F),
19

---

Published as a conference paper at COLM 2024
A.3
Additional Ranking Visualizations
A.3.1 Model Performance Comparisons
With the GPT-4 rated K-wise rankings we can aggregate statistics on the model preferences from GPT-4.
Average Ranking of Each Model
Average Rank (lower is better)
6
5
2
lamaa dar Cooto Co ve. cal Srone
op Songe 10 Pectttonn
yer-umn nung
T' Sommn oz nun-ano s0
Dires
cuream Erreee un
C eoino
ferine coung E-100
ees suttim out
of vetmae
Scoate
of aoce scddde tuu
Serii scede
oe 02 sung Koop
all 190
30gaduu
quue
Model
Figure 10: GPT-4-0613 (with system prompt for chat) had the highest average rank based on the prompt rubric, closely followed by Claude 1, GPT-4 (no system prompt), Llama-2-70b- chat, and Claude-Instant-1.
We extract the prompt intention classification (good natured or not good natured) from the rating response. From this we construct model performance when the prompt is good natured (in which helpfulness and harmlessness is expected) and when the prompt is not good natured (in which mainly harmlessness is expected). Note that these are GPT-4's estimation of the intention of the prompt, and are not necessarily fully accurate.
Rachapg
Mudel
netuued Fibl
Tue
Figure 11: When considering only good natured user prompts, GPT-4-0613 had the highest average rank, based on the prompt rubric, closely followed by Claude 1, GPT-4 (no system prompt), LluudeInnstat-- and Llama-2-70b-chat.
20

---

Published as a conference paper at COLM 2024
Ave age Ranking
naturol
Hibe
Nooel
Figure 12: (Same graph as above, with x-axis sorted on when good natured is false) When considering only bad natured user prompts, Llama-2-7b-chat had the highest average rank, based on the prompt rubric, followed by Llama-2-13b-chan GPT-4-0613, Pythia-12b, and Wizardlm-70b
The robust performance exhibited by Llama-2 in maintaining harmlessness can be attributed to its extensive safety alignment that counteracts prompt-level adversarial attacks (Chao al. 2023). Prior research illustrates that Llama2 models possess impressive resilience when tacing jailbreak prompts (Chao et al. 2023); (Zheng et al. 2023a). Additionally, contrary to expectations, smaller models have been empirically shown to outperform larger ones when assessed solely on harmlessness (Kundu et al. 2023 Our findings corroborate this notion, signifying that, in comparison to its counterparts, Llama-2-7b-chat consistently displayed the utmost harmlessness in response to ill-intentioned prompts. This was followed closely by Llama-2-13b-chat and Llama-2-70b-chat
We also provide the heatmap for the pairwise average ranking difference of all models in our Nectar dataset in Figure 13, and the pairwise winrates in Figure 14
B Reward Model Evaluation
B.1
Evaluation Metrics
B.1.1 Human Preference
For the human preference measurement, we first subsampled 10, 000 single-turn human- rated response pairs from Lmsys's ChatBot Arena Conversations dataset (Zheng et al. 2023b). We have each reward model return score for each prompt response in the dataset. For a given rated response pair, if the reward model's score for the human-chosen winning response is greater than the score for the losing response, that response pair is considered classified correctly. The accuracies and losses for each model are detailed in Table 3 and Table 4
B.1.2 Truth
In order to understand if each reward model will value true statements over false statements, we utilize true and false response pairs from the Truthful QA dataset (Lin et al. 2021). For each prompt in the dataset, we sample three answers: the best truthful answer (Best), true answer (True), and a false answer (False). We evaluate reward model output scores for each of these answers and average accuracy /loss across the three possible pairwise combinations where the overall ranking order is (Let be the reward model) r (Best) >= r(True)= > r(False).
21

---

Published as a conference paper at COLM 2024
61
MEA
31 9 631
L1
0- 16314 51 1996 45 5 ENT w 1n 1 55 mnsoow m
15 3. 64
X331
035614
21 c mises
12 05
31 63
14
18
15
184M33
21 31 31.0
F EE
55 37
1a 26
13
10
15 6T 17
C1AM 67511 MARY
11
Radd LI 3.
3842
12
1000
55 4443
13 43
24403
8344
13
535
34
2043
33549
10
21 64
8334
2 14
61 11
874 A11
SH TINON 24
5 16 14
15
51
87 612
179 CONNSS
21
51 030000
42
42
AA 65666444
26 25
54 35
L6 12 14
Figure 13: Heatmap of pairwise ranking difference for all models in Nectar.
22

---

Published as a conference paper at COLM 2024
337
930 o7
235 S
355 6
175 15 11555
1560
134
332335
35
06 C. Cor 4:0 Edd
2015
50
3070
110
110 TEATESAAT MAATTATTA
214 1111551 MAN0ENS
1/16 2:24
5.15 03
277
M2 027
5:8
A10021 22 o5
21 622
MARS OL
A4H 047927 052
5:4
XT9
R12
X12
Figure 14: Heatmap of pairwise winrates for all models in Nectar.
Pointwise Scores
400
300
200
0
2
Score
Figure 15: Pointwise scoring methods struggle to distinguish the differences between response choices and overwhelmingly give high scores.
23

---

Published as a conference paper at COLM 2024
B.1.3 Safety
We measure safety by evaluating on the PKU-SafeRHHF dataset from Ji al. (2023) sub- sampled down to 1000 response pairs where one response is safe and the other response is unsafe. Reward model scores are generated for each of the responses with the expectation that the the safe response score is higher than the unsafe response score. See Table 3 for more detail and Table 4 for losses.
B.1.4 Verbosity
Empirical analysis of the RL fine-tuned language model shows an occasional undesirable tendency towards overly verbose responses to simple prompts. For example when prompted with "1+1", ,, the model responded with:
Here are some important tips to consider when using the OpenAl GPT-4 model:
Understand the capabilities and limitations of the model: GPT-4 is powerful language model, but it's important to recognize its limitations. While it's great at natural language understand- ing and generation, it may not always provide accurate or reliable information.
Be mindful of biases: [response continues for another 700 words]
We created a small handcrafted dataset of 29 response pairs, where one response was overly verbose, and the other was not. Verbose response were crafted through finding prompts that yielded over verbose responses, handcrafting verbose response by introducing repetition, and through finding existing examples. A generated reward scoring pair is correct if the less verbose response receives the higher score.
In this task, all three reward models trained struggled, with accuracy rates 0.167, 0.133, and 0.100 (Table 3). The reward model seems to often value extremely long responses and give lower scores to very simple but concise answers. In the above example where "1+1" is prompted, the long 700+ word response was given score of -0.635 while the simple but correct answer of "2" was given score of -4.313. An additional example can be found in the Appendix B.2
B.1.5 GPT-4 Verbosity Preference
This section aims to delve into the implications of the verbosity preference noticed in the trained reward model. The investigation involves examining whether this verbosity preference materializes as byproduct of GPT-4's inherent propensity while curating the rankings for Nectar. We run the same 29 evaluation pairs, comprising of both verbose and non-verbose responses, through the GPT-4 ranking procedure, which was adjusted for pairwise ranking. With the original rubric GPT-4 picks the non-verbose (correct) response with an accuracy rate only 0.433. Augmenting the rubric to detail that response should not be "overly verbose" yields slightly improve accuracy of 0.500. However, in both cases, GPT-4 is unable to correctly distinguish when verbosity should be less preferred. These experiments suggest that the verbosity bias noticed in the reward models is possibly manifestation of GPT-4's inherent rating bias. An example of this verbosity preference is show below in the Appendix B.3
24

---

Published as a conference paper at COLM 2024
B.2
Additional Verbosity Example
Prompt
Which these fruits is citrus? Give only the correct letter: A, B, C, D.
(A) Pear
(B) Apple
(C) Orange
(C) Cherry
Verbose Response
Here are the steps solve this problem:
Only one of the given options is a citrus fruit. Citrus fruits belong to the Rutaceae family and include oranges, lemons, limes, and grapefruils. Pears, apples, and cherries, on the other hand, belong to different fruit families.
Therefore, the correct option (C) Orange
The correct answer is: C
Non-Verbose Response
C
This prompt specifically instructs the model to only give the correct letter in the response. The verbose response ignores this instruction and gives additional information. In this case, the verbose response received higher reward score of -2.112 and the non-verbose response received lower reward score of 4.065..
B.3
GPT-4 Verbosity Preference Example
Prompt
hi!
Response A (Verbose)
Hello! I'm here to assist you with any questions or tasks you may have.
Here tips to keep in mind while using this Al:
clear and concise with your questions or requests.
Feel free to ask follow-up questions if you need more information or clarification.
Keep in mind that the Al is powerful tool, but may not always provide the exact answer
you're looking for, especially in complex nuanced situations.
While the Al can provide general advice, it's a substitute professional guidance from
experts in specific fields.
Be respectful and courteous in your interactions with the AI, as it's designed to help you in friendly and helpful manner.
If you have any specific questions tasks, feel free to ask, and I'll do my best assist you.
25

---

Published as a conference paper at COLM 2024
Response B (Non-verbose)
Hello! How can I help you today?
GPT-4 Rating
Both responses are polite, respectful, and adhere to moral and ethical standards. However, Model A's response is more detailed and provides useful information about how to interact with the AI, which makes it more helpful in this context.
Ranking: [I'A', 'B]]]
In this judging context, despite being told in the system prompt that responses should not be overly verbose, GPT-4 still ranks the exceptionally verbose response over the simple response. It may be difficult induce GPT-4 to rank verbose responses lower in conversa- tional contexts were verbosity is actually less preferred.
26


Reviews:
[{"review_id": "FLjUxSSCrH", "reviewer": "Reviewer_Xpnc", "summary": "This paper presents not only a chatbot which is the best on the chatbot area, but also releases the training dataset, and explains implementation details.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We appreciate your recognition of the importance of the problem that our paper addresses. We value the opportunity to further illustrate the key aspects of our research.\n\n>There are quite a few limitations in using GPT as an evaluator, though the authors acknowledge those.\n\nYes, we not only address the positional bias in GPT4 evaluation but also provide a simple prompting technique to mitigate the bias. We believe our methodology could benefit future research.\n>While this is a useful open source tool, data release, and hyper parameter optimization, there is more limited research contribution.\n\nWe would like to emphasize that our main contribution is to demonstrate the possibility of providing a comprehensive recipe to reproduce a strong and deployable model, this includes data creation, data curation, algorithm implementation details, which are all very important to obtain great results. We identify and propose simple prompting techniques to facilitate our goal, and we are the first serious attempt to employ the less explored K wise reward training loss. We also propose several important implementation details that’s extremely important for stable PPO training, e.g., shifting the reward for length control and pre-training the critic function. We believe this comprehensive recipe already provides a significant contribution not only in engineering but in research.\n\nWe hope these clarifications address your concerns and provide a deeper understanding of our work. We are grateful for your engagement and look forward to further discussions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The analysis on positional bias has potential to be useful to other work. \n- The paper in general does a good job of explaining implementation details, which are often glossed over. \n- The model is the best-performing 7B chat model on Chatbot Arena (at time of publication) publishing the details about it will be of use to the community as a while.", "reasons_to_reject": "- There are quite a few limitations in using GPT as an evaluator, though the authors acknowledge those. \n- While this is a useful open source tool, data release, and hyper parameter optimization, there is more limited research contribution.", "questions_to_authors": "small comments:\n\n- there is a messed up reference:\n\"We compare the Nectar dataset with existing preference dataset (Cui et al., 2023; Bai et al., 2022a; Ethayarajh et al., 2022) in Table ??.'\n\n- figure 1 is blurry.\n\nTypos:\n-  \"We would like to When K = 2, \"\n- Similar to the default methodology of training reward model  -->  training a reward model, or training reward models", "extra_scores": {"ethics_concerns_details": "the paper mentions distilling from GTP-* output. I am not sure if that is allowed under the Terms of Service?"}}, {"review_id": "veEU3hKX1c", "reviewer": "Reviewer_nUFE", "summary": "Quality: The authors have developed an approach to utilize GPT-4 to create a preference dataset, Nectar, and train reward models. Finally the authors train a chat model with PPO using the reward models. Using this pipeline, they achieve state-of-the-art performance within the 7B scale models. So the quality is ok.\n\nClarity: The paper is easy to follow.\n\nOriginality: The paper combines a few existing things together, it is not particularly novel. The most novel part is they do global ranking so that the reward models have more samples to learn from\n\nSignificance: This work makes a great contribution to the open-source community. The reward model achieves pretty good performance on rewardbench, and the model is publicly released.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We deeply appreciate the opportunity to enhance the clarity of our work.\n\n>Lack of novelty, mostly prompt engineering to get good ranking information.\n\nWe would like to emphasize that our main contribution is to demonstrate the possibility of providing a comprehensive recipe to reproduce a strong and deployable model. This includes data creation, data curation, and detailed algorithm implementation, all of which are crucial for achieving great results. We identify and propose simple prompting techniques to facilitate our goal, and we are the first to seriously attempt employing the less-explored K-wise reward training loss. Additionally, we propose several important implementation details that are extremely important for stable PPO training, such as shifting the reward for length control and pre-training the critic function. We believe this comprehensive recipe provides a significant contribution not only in engineering but also in research.\n>The paper presentation can be improved. Some of the figures are not particularly readable, and there are small typos throughout the paper.\n\nWe will correct the typos and improve the quality of the figures in our revision. Thank you for the suggestion.\n>I would recommend authors remove ShareGPT from their dataset. ShareGPT is not published…\n\nThank you for raising this concern. We will take it into consideration when updating our dataset.\n>It is not clear to me why related work is part of introduction?\n\nThanks. We will move the related work section to be independent of the introduction.\n>From Figure 1, it looks like the only significant improvement of the model is on the reasoning category. Does the author have an explanation for why this is the case?\n\nThe original OpenChat 3.5 model has already achieved very strong performance in categories like writing and humanities, while still significantly lagging behind closed-source models like Claude or GPT-4 in STEM-related domains such as coding and reasoning. This discrepancy provides more room for improvement in these underperforming domains. Furthermore, our dataset contains a reasonable amount of coding and reasoning-related questions, which may explain the observed improvements.\n>Figure 13 and 14 are not for readers to understand.\n\nWe will add detailed explanations about Figures 13 and 14 in the Appendix.\n\nWe appreciate your feedback and hope that these clarifications will make our revised draft more comprehensible and informative.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Both the reward model and the chat model achieve state-of-the-art performance.\n2. The data and models can benefit future research in the field.\n3. The proposed ranking method seems simple and works well.", "reasons_to_reject": "1. Lack of novelty, mostly prompt engineering to get good ranking information.\n2. The paper presentation can be improved. Some of the figures are not particularly readable, and there are small typos throughout the paper.", "questions_to_authors": "1. I would recommend authors remove ShareGPT from their dataset. ShareGPT is not published and never obtains users' consent to share and release their data. It is unethical to redistribute data without proper permission. To be clear, this is a suggestion and not part of my paper evaluation.\n2. It is not clear to me why related work is part of introduction?\n3. From Figure 1, it looks like the only significant improvement of the model is on the reasoning category. Does the author have an explanation for why this is the case?\n4. Figure 13 and 14 are not for readers to understand.", "extra_scores": {}}, {"review_id": "RGrlkJfZLe", "reviewer": "Reviewer_viaJ", "summary": "Paper proposes Nectar, a high quality open-source preference dataset using GPT-4 to rank responses. This datasets was made based on asking model to consider all pairings before reaching ranking. The dataset is around 180k prompts and with 7 responses each, reaches 3.8 million pairwise comparisons. These prompts are collected from LMSys-1M, Anthropic-HH , Ultrafeedback and ShareGPT. Compared to existing general datasets (excluding StackOverflow which is coding specific), the number of pairwise comparisons is an order of magnitude higher.\n\nResponses are ranked (ordinally) on both helpfulness and harmlessness. Responses are by models like gpt-4, gpt-3.5, Mistral 7b and llama2 7b. Using this dataset, they train Starling 7B, which is the best 7B model on Chat Arena. This model increase MT Bench from 7.81 to 8.09 on MT Bench and 88.51 to 91.99 on 91.99 and Chat Arena ELO from 1072 to 1087. When trained with a larger reward model, the Chat Arena Elo increase further to 1118.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed feedback and will endeavor to address your concerns.\n>In Figure 6 right-most plot…\n\nWe are choosing one fixed ordering that helps mitigate the positional bias. However, fixed ordering might lead to some disadvantage for responses in position 2. In practice, one may use random ordering to mitigate this.\n>Can the authors justify why Ordinal ranks were used rather than Cardinal…\n\nSince most of our responses are generated from strong models, the scores for responses will heavily concentrate around 9-10 (out of 10), which essentially provides much less signal than pairwise or multi-wise comparison. We have also tried a score range 0-100. However, we still observe a high concentration around close to 100, which weakens the information we can get.\n\n>According to Figure 4…\n\nSince OpenChat-3.5 has already undergone extensive GPT-4 distillation during the SFT phase with millions of prompts, much larger than our scale, additional SFT will result in no improvement or even hurt performance due to overfitting. However, RLHF can still improve the model’s ability even if SFT has already saturated.\n>Using MT Bench to select policy model checkpoint … risks overfitting to this benchmark.\n\nOur goal is not to overfit MT Bench, but to excel in Chatbot Arena, which is purely based on human judgment and hard to overfit (initial evaluation requires around 2 weeks). MT Bench is only for filtering out most of the checkpoints that were suboptimal before we spent human resources to evaluate them thoroughly.\n>Authors do not mention the specific LR and KL penalty…\n\nWe will report the LR and KL penalty in our final revision. \n\n>The authors mention using 10-20 challenging prompts but do not share what these prompts…\n\nFor the IFeval-based checkpoint filtering strategy, we manually define the threshold. In Starling 7B beta training, we set this threshold to be 0.05 regression compared with the SFT model. For the human evaluation prompts, we will also release them in our final revision.\n>The choice of base models used with the Reward model is weird…\n\nWe observed a lower loss and better convergence compared to training Mistral 7B as a reward model. It could be the difference in their model architecture or coverage of their pretrain data. We think this might raise interesting research questions which require further studies.\n>Please take care to correct grammatical errors…\n\nWe will correct them in our revision, and we appreciate the reviewer for pointing them out!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe approach to rate responses is well designed to solve the positional bias issue which is well illustrated in Figure 2. Specifically, they do pairwise ranking and summarize into listwise ranking – all in a single prompt.\n2.\tThe reward model training approach is reasonable to allow K-ary comparisons rather than decomposing it into 7C2 pairwise comparisons. \n3.\tThe trained reward models and policy models are evaluated on a good range of evaluation metrics and show promising results on these benchmarks.\n4.\tThe design of adding a slightly positive reward to allocate some advantage to EOS token to prevent length explosion is reasonable. Similarly, pretraining the critic is an interesting finding that stabilizes training. \n5.\tMonitoring Verbosity and IFEval as a strategy for checkpoint selection is reasonable.", "reasons_to_reject": "1.\tIn Figure 6 right-most plot, while most of the positional bias from position 1 and 3-6 are eliminated, there is a curious higher win-rate for position 0 and lower win rate for position 2. Can the authors explain why this is the case? \n2.\tCan the authors justify why Ordinal ranks were used rather than Cardinal as used in Ultrafeedback and HelpSteer dataset (see Wang et al. (2023) “HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM” http://arxiv.org/abs/2311.09528)? Cardinal scores (say between 1 to 10) can strictly convey more information (as we can construct ordinal ranks from cardinal scores but not the other way round) and does not require ties breaking as required in the ordinal approach. More importantly, cardinal scores can convey how much better a response is compared to other responses say [10, 1, 1,1,1,1,1] is pretty different from [10,9,8,8,7,7,7] even though they may have the same ordinal ranks.\n3.\tAccording to Figure 4, the top 4 models used (~180k each) are all from OpenAI (gpt4 and gpt3.5 variants) followed by mistral-7b and anthropic. This means that for any given prompt, there are two responses from gpt-4 and another two from gpt-3.5. Given the judge is also gpt-4, it is very likely to choose gpt-4 responses as the top rated response. Sharma et al. (2024) “A Critical Evaluation of AI Feedback for Aligning Large Language Models” https://arxiv.org/pdf/2402.12366 shows that much of gains from AI Feedback by GPT-4 can in fact be attributed to distillation from GPT-4. Relating back to this paper, this paper is missing a key ablation of simply doing SFT on top of the GPT-4 response to each prompt.\n4.\tUsing MT Bench to select policy model checkpoint and also reporting as an policy model metric risks overfitting to this benchmark.\n5.\tAuthors do not mention the specific LR and KL penalty used, which makes the RM and PPO experiments hard to reproduce.\n6.\tThe authors mention using 10-20 challenging prompts for internal human evaluation but do not share what these prompts are and specifically how the evaluation is done. Similarly, they do not mention specific details for varies approach to checkpoint selection (such as what is the length threshold they used).", "questions_to_authors": "1.\tThe choice of base models used with the Reward model is weird. Why did you choose to use llama 2 7b and then Yi 34B (as opposed to 2 models in the same family, which is available)? Specifically using Llama 2 7b might mean that the reward model can only be used to train models from the llama family (when the generally better-viewed Mistral 7b with Apache 2.0 license is available). \n2.\tPlease take care to correct grammatical errors (likey should be likely) and citation errors such as TRLX (Havrilla et al.) should be (Havrilla et al., 2023). These are plenty in the paper as it seems rushed and I trust the authors to correct them later on.", "extra_scores": {}}, {"review_id": "hbXZ0sAfrd", "reviewer": "Reviewer_dC9a", "summary": "* The authors contribute to open source:\n    * The Nectar dataset, a preference dataset comprised of 3.8M pairs, significantly larger than many other open source preference datasets like Ultrafeedback and Anthropic-HH\n    * 7B and 34B reward models trained on this dataset\n    * Two 7B models finetuned with PPO using the 7B & 34B RMs respectively from Openchat-3.5 (“Starling-LM-7B”) which achieves strong results on MT-Bench, AlpacaEval, and human evaluations\n* The authors detail various tips and tricks for collecting the Nectar dataset (for example, ways to reduce the positional bias problem when using a model-as-judge)\n* Three different methods for training a reward model are compared, and the authors find applies the maximum likelihood estimator under Plackett-Luce model gives the best results\n* The authors also detail various tips and tricks for improving stability and training robustness of PPO such as:\n    * Shifting the reward mean for length control, to prevent model from becoming too verbose\n    * Pretraining the critic model to stabilize early stages of training\n    * Full parameter tuning, vs. unfreezing only top layers", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate your recognition of the importance of the problem our paper addresses\n\n>While the paper provides a detailed recipe for finetuning a model with PPO, the authors provide few comparisons to relevant baselines, for example (1) how does a reward model trained on the Nectar dataset compare to a reward model trained on some of the other open source datasets? (what exactly are we getting from the Nectar dataset over other datasets?) (2) how does the benchmark performance of the chat models compare to other similarly sized models?\n\n\nThank you for your question. We will update our manuscript accordingly. In particular, we will include a comparison of the reward model with other existing reward models in Table 5. The Nectar dataset offers a larger amount of equivalent pairwise comparisons with higher quality generations, which appears to be the key for developing a high-quality reward model. Additionally, our Starling model is currently the best Mistral-7B-based model based on anonymous human evaluations conducted on Chatbot Arena.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The authors contribute a large and high quality preference ranking dataset, which is an order of magnitude larger than some comparable datasets like Ultrafeedback and Anthropic-HH, which will be very impactful for studying RLHF in open source and academic communities \n* The authors also contribute several reward models trained on this dataset and two strong chat models trained with these reward models \n* The authors provide many details on how to finetune a model with PPO; these types of tips and tricks are often left unpublished so it is great to see them detailed here", "reasons_to_reject": "While the paper provides a detailed recipe for finetuning a model with PPO, the authors provide few comparisons to relevant baselines, for example (1) how does a reward model trained on the Nectar dataset compare to a reward model trained on some of the other open source datasets? (what exactly are we getting from the Nectar dataset over other datasets?) (2) how does the benchmark performance of the chat models compare to other similarly sized models?", "questions_to_authors": "- Formatting nits: table reference on pg. 2 is broken", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a recipe for training a model with RLAIF. The final Starling-7B model achieves high performance on ChatbotArena, among other benchmarks.  While there are some methodological criticisms (e.g., data skewed towards GPT-4, by viaJ), the reviewers on balance find the Nectar dataset, its reward models, and the final LLMs helpful. Some of the details like the analysis of positional bias are likely to be useful in other work.
Decision: Accept