HelpSteer2-Preference: Complementing Ratings with Preferences
Authors: Zhilin Wang
Alexander Bukharin
Olivier Delalleau
Daniel Egert
Gerald Shen
Jiaqi Zeng
Oleksii Kuchaiev
Yi Dong
Venue: ICLR 2025 Poster (2025)
Keywords: reward modelling
rlhf
model alignment
Primary area: datasets and benchmarks
TL;DR: We release data to support a data-matched comparison of Bradley-Terry and Regression style Reward Models, which inspires an approach resulting in a Reward Model scoring 94.1% on RewardBench, emerging the top model out of 140+ models as of 1 Oct 2024.

Abstract: Reward models are critical for aligning models to follow instructions, and are typically trained following one of two popular paradigms: Bradley-Terry style or Regression style. However, there is a lack of evidence that either approach is better than the other, when adequately matched for data. This is primarily because these approaches require data collected in different (but incompatible) formats, meaning that adequately matched data is not available in existing public datasets. To tackle this problem, we release preference annotations (designed for Bradley-Terry training) to complement existing ratings (designed for Regression style training) in the HelpSteer2 dataset. To improve data interpretability, preference annotations are accompanied with human-written justifications. Using this data, we conduct the first head-to-head comparison of Bradley-Terry and Regression models when adequately matched for data. Based on insights derived from such a comparison, we propose a novel approach to combine Bradley-Terry and Regression reward modeling. A Llama-3.1-70B-Instruct model tuned with this approach scores 94.1 on RewardBench, emerging top of more than 140 reward models as of 1 Oct 2024. This reward model can then be used with REINFORCE algorithm (RLHF) to align an Instruct model to reach 85.0 on Arena Hard, which is No. 1 as of 1 Oct 2024.
We open-source this dataset (CC-BY-4.0 license) at https://huggingface.co/datasets/nvidia/HelpSteer2#preferences-new---1-oct-2024 and openly release the trained Reward and Instruct models at https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward and https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct .

Published as a conference paper at ICLR 2025
HELPSTTEER-PRREEERNNC COMPLEMENTING RAT- INGS WITH PREFERENCES"
Zhilin Wang'
Alexander Bukharin 1.
Olivier Delalleau'
Daniel Egert!
Gerald Shen'
Jiaqi Zeng'
Oleksii Kuchaiev!
Yi Dong
[zhilinw, yidong} @nvidia. com
ABSTRACT
Reward models are critical for aligning models to follow instructions, and are typically trained following one of two popular paradigms: Bradley-Terry style Regression style. However, there is lack of evidence that either approach is better than the other, when adequately matched for data. This is primarily because these approaches require data collected in different (but incompatible) formats, meaning that adequately matched data is not available in existing public datasets. To tackle this problem, we release preference annotations (designed for Bradley-Terry training) to complement existing ratings (designed for Regression style training) in the HelpSteer2 dataset. To improve data interpretability, preference annotations are accompanied with human-wrinee the first head-to-heal comparison of Bradley-Tere and Regression models when adequately matched for data. Based on insights derived from such comparison, we propose a novel approach to combine Bradley- Terry and Regression reward modeling. A Llama-3.11700 Instruct model tuned with this approach scores 94.1 on RewardBench, emerging top more than 140 reward models as of Oct 2024. This reward model can then be used with REINFORCE algorithm (RLHF) align an Instruct model to reach 85.0 on Arena Hard, which is No. as of Oct 2024.
justifications. Using this data, we conduct
Dataset :-ccene)- (ccns:----i-iinnns:::: htt.ginnneee..oolattttttt/vv///p
Reward Model: Ba -Neeooroonn R00-eee
Instruct Model: hugiingfae..oohvviiaallaaaa -eemoooron-- 70B-Instruct
INTRODUCTION
First featured in the Reinforcement Learning from Human Feedback (RLHF) pipeline for aligning language models to follow instructions (Bai et al. 2022 Ouyang et al., 2022). Reward Models are still prominently featured as a critical part for aligning frontier open-recipe models (Dubey et al.) 2024 NNvidi et al. 2024). The role of Reward Models lies in assigning high scores to responses that follow instructions in a helpful and safe manner while assigning low scores to those that not. This in turn guides language models to generate more responses that give high scores, which make them more helpful and safer (Dong et al. 2024: Lambert et al.) 2024).
While training reward model that can accurately separate good responses from bad ones is consensus goal, there is less agreement on the best path to get there. On one side are the traditional Bradley-Terry Style Reward Models first introduced by Bai al. (2022) and Ouyang et al.) (2022) which seek to maximize the gap in reward between chosen and rejected responses to the same prompt. On the other side Regression style Reward Models introduced by Wang et al. (2024e) and Wang et al. (2024a) and have lately been used to train some of the top models on RewardBenee (AllenAl 2024 such as ArmoRM (Wang et al.) 2024b) and Nemotron-4434000 4-340B-Reward (Wang et al. (024d). Regression Reward Models train the model to predict the score (often Likert-5) 5) for a response to
"NVIDDI," Georgia Tech, work done during internship NVIDIA

---

Published as a conference paper at ICLR 2025
particular prompt. For many researchers and practitioners, deciding which style of Reward Models to adopt is challenging due to the absence of empirical studies that directly compare them when appropriately matched for data, which means:
Identical set of prompts and responses This mitigates the confounding factor of prompts and responses influencing reward model performance, which plays role outside the data collection approach.
2. Collected for Purpose Data used to train each type of Reward Model should be collected in the format they will be used. While there are heuristics to convert Regression-stylle ordinal annotations over preference annotations (using the difference between scores of individual responses), our early experiments found them to be comparatively lackluster for training Bradley-Terry models. We hypothesize that Regression-styl data collection (where responses are rated individually) gives annotators different set of expectations for comparative ranking of responses. For instance, in the Likert-5 scale used by HelpSteer2 (Wang et al.) 2024d), both responses can be given a helpfulness but one might still be preferable to the other. Post-hoc heuristics used to convert such Regression-style data to preference rankings are unable to account for such nuances in preferences.
3. High Quality Following the adage 'Garbage in Garbage out', the role that the data format (regression-stylee vs. preferences) plays may not reliably show when the quality of data collection is the bottleneck, as the signal-to-noiss ratio will be too low for the model to learn anything useful. Shen et al.] (2024) found that the Llama-2- 2- model performs essentially at a random level (l.e. <55% for binary choice task) RewardBench Chat-Hard category (Lambert et al. 2024) when trained on Ultrafeedback (Cui et al. 2023) or HH-RLHF (Bai et al. (022), which is attributed to noise in the ground truth labels. Similarly, Wang al. (2024d) found that Llama-3-70B models trained on HH-RLHF (Bai et al. 2022), Open Assistant (Köpf et al. 2023) or HelpSteer (Wang et al. 2024eb do not surpass 60% on the same evaluation. In contrast, Llama-3- 3-70B trained on HelpSteer2 (Wang et al. 2024d) performs above 80%. Without meeting high bar on data quality. it may not be possible to discern the advantages of a particular data annotation methodology over another.
To the best of our knowledge, no one has thus far publicly released data that is adequately matched for both approaches (see detailed Related Works in Appendix (A). In this work, we release preference annotations that were collected alongside the Likert-5 ratings from HelpSteer2 (Wang et al.) (022dd) a high quality dataset used to train some of top models on RewardBench (e.g. Nemotron-4-3400 Reward). | We show that Bradley-Terry Models can be effectively trained with such preference annotations, and also investigate leveraging preference justifications where annotators indicate why they preferred one response over another.
Our key contributions are:
We open-source (CC-BY-4.0) a high-quality preference modeling dataset containing preference directions, strengths, and justifications. To the best of our knowledge, this is the first sourcs release of -omain preference dataset containing human- preference justifications.
1.
2. Using this data, we perform a tead-to-heate comparison of gradley-Terry style and Regression style Reward Models, alongside reward models that can make use of preference justifications.
3. From insights derived from the above comparison, we propose a novel approach to combine Bradley-Terry and Regression Reward Models, which can be used to train reward model that scores 94.1 on RewardBench, which is the best performing model as of 1 Oct 2024. This reward model can then be used with the REINFORCE algorithm (RLHF) to align model to reach 85.0 on Arena Hard.
2 DATASET
Data Collection For each task, annotators are provided prompt and two responses. They first annotate each response on Likert-5 scale along several dimensions (helpfulness, correctness and coherence, complexity and verbosity), as detailed in Wang et al. (2024d). Then, they choose between 7 preference options, each associated with a preference score as well as a justification for their preference:
'This data has not been released previously hecause we did not have sufficient resources to conduct experi- ments demonstrate the value of preference annotations.
2

---

Published as a conference paper at ICLR 2025
-3. Response 1 is much better than Response 2
-2. Response 1 is better than Response 2
-1. Response 1 is slightly better than Response
1. Response 2 is slightly better than Response
2. Response 2 is better than Response I
3. Response 2 is much better than Response I
-100. Neither response valid
The last option ("Neither response is valid") is to be used when both responses are so bad that trying identify a winner is pointless. Inspired by Bai el. (2022) and Touvron et al. (2023), we force the annotator to make preference choice between the two responses (i.e. no option for Both responses are equally good), except for "Neither response is valid". Our motivation was reduce siting-on-the-feenee behavior and encourage annotators to look closer at differences (even if they appear to be minor), in order to provide more robust preference information. The preference ranking guidelines and examples provided to annotators in the Appendix B
The vendor (Scale AI) was asked to distribute each task to 3-5 annotators to independently label preference among two responses for every prompt. For a small proportion of samples (<10%) there are fewer than three useful annotations the resulting dataset. This is due to annotators skipping some tasks they were unable to rate them effectively indicating that both responses were invalid (i.e. rated as -100, which were excluded).
Data Pre-processing In line with HelpSteer2, we identify the three most similar preference annotations per task (to avoid interference by outliers), take their mean, and round it to the closest integer to give the overall preference. Furthermore, we filter out 10% of tasks, for which the spread of annotations of the three most similar annotations was more than two. For instance, task with individual preferences of [-3,-1 -1, 2, 3] will have the most similar annotations of [-1, 2, 3] with a spread of 4, and thus excluded. This is done avoid training on tasks for which the ground-truth preference cannot be confidently estimated among human annotators. 22% of the samples have an overall preference of in situations where annotators disagree (e-g. [-1,-1, 1)) and these samples are also excluded because a near-zero average indicates low-confidence preferences, which may introduce noise during Reward Model training. Overall, we have 7,118 preference pairs with 6,766 pairs in the training set and 352 pairs in the validation set.
To compare the inter-rater reliability of our collected data (compared to HelpSteer2), we follow Wang et al. 2024d), to use quairati Cohen's к (Artstein & Poesio 2008) as measure of inter-rater agreement. The quadratic weighted version of Cohen's 's x (Scikit-Lean) 2024) can also penalize larger disagreements (e.g. -3 and +3) much more heavily compared to smaller disagreements (e.g.-I -1 and +1). The agreement of the raw preferences is 0.489 (moderate), suggesting that having annotators agree the direction and strength of preferences is challenging. Our pre-processing steps (i.e. using the three most similar annotations per task and removing samples with a preference spread of more than two) increase Cohen's 's x to 0.843 (strong), suggesting that weeding out outlier annotations is an effective way to improve agreement. Finally, excluding samples with an overall preference of 0 further increases Cohen's 's к to 0.878 (strong). The observed final agreement is stronger than HelpSteer2 helpfulness (with 0.791), suggesting good reliability of our collected data. We detail how we pre-process preference justifications in Appendix C
Data Analysis We analyze the dataset including samples for which the overall preference is zero, as this can provide insights below, even though such samples are not used for training subsequently. As shown in Fig. [ the distribution of preferences concentrated at the center (A=B or 0) and reduces gradually away from the center (μ- = 0.0649, a = 1.72), with slight bias to preferring Response 2 over Response 1. This means that Response is preferred in 41.6% of tasks while Response is preferred in only 36.5%. This bias is especially high for the "slightly better" setting where Response 2 is slightly preferred 18.5% vs. Response 14.8%. Such a bias is similar in extent to the difference between the helpfulness score of Response 1 and Response 2 from the original HelpSteer2 dataset (Wang et al., (024d): 39.3% of tasks gave Response a higher helpfulness score while only 33.9% gave Response 1 higher helpfulness score.
3

---

Published as a conference paper at ICLR 2025
22.0%
14.8%
14.6%
4-
0.0%
0.0%
0.0%
0.4%
3.8%
3-
0.0%
0.0%
0.0%
0.1%
0.3%
2.3%
4.2
6.9%
Helpfuiness of Helpfulness
2-
0.0%
0.3%
0.1%
0.2%
1.5%
8.1%
0.4%5
10.3%
1-
0.4%
3.7%
11.0%
2.5%
17.9%
0.0%
4.5%
26.8%
-1-
0.0%
2.5%
0.3%
3.3%
0.3%
0.2%
0.
22
0.5%
7.4%
1.3%
0.1%
0.1%
0.1%
0.0%
9.0%
4.0%
1.7%
0.2%
0.0%
0.0%
0.0%
0.0%
6.0%
3.6%
0.1%
0.0%6
0.0%
3.8%
A> As
A=
A=B
A=B
Acc Asce
Figure I: Distribution of preferences between responses in Selpsteer2-PPrefeeenee against the differ- ence in helpfulness scores between them from HelpSteer2. For clarity, A refers to Response while refers Response 2. >>> means much better, means better, means slightly better and means as good as.
There are some possible reasons for such position bias. First, the sampling of responses between Response I and might not have been even, with Response having been dis-proportionaaly generated by stronger models. Second, annotators could also have a positional bias, in which the order of responses shown them can influence their judgment. this case, the response they see later could be making a stronger impression on them, leading annotators to tispprpporri prefer them due to recency bias (Phillips- Wren et al., 2019). While it is important to note the existence of this position bias. it is relatively small in extent (5.1% difference in preference), which minimally affects their usefulness compared L.LM-as-A a---a----uudee settings (with GPT-4 and Claude) where the position bias could be between 25% and 75% (Zheng et al. 2023).
Fig. also shows that the distribution in preference between the two responses highly correlated with the difference in the responses' helpfulness score from HelpSteer2 (Pearson 's R = 0.9024). When two responses have the same helpfulness score, they most likely are as good as each other but in slightly less than half the cases (45.5%), annotators still show consistent preference for one response. This supports our initial hypothesis that preference ranking can produce more fine-grained separation between responses preferences are most commonly (58.0%) associated with a difference in helpfulness scores one while strong preference preferences are frequently (93.4%) associated with helpfulness difference of 3 or 4. Interestingly, there are also substantial samples (7.4%) that show no preference despite having an absolute difference of one or more. This is likely due to annotator disagreements, which averaged to obtain the final overall preference. Given the constraints on space, we defer an extended analysis of preference justifications (both structural and semantic) Appendix
if they share the same Likert-5 helpfulness score. Slight
3 REWARD MODEL
3.1 EVALUATION
Following Dong al. 2024); Wang et al. (0024)) Yuan al. (2024), perform evaluation using RewardBench (Lambert et al. 2024). ) a trusted reward modeling benchmark with over 140 models on the public leaderboard (AllenAl 2024). RewardBench comprises 2985 diverse tasks across categories Chat, Chat-Hard, Safety, and Reasoning and 23 cub-categories), which minimizes the likelihood of over-fitting particular tasks. Each task consists of prompt, chosen response, and
4

---

Published as a conference paper at ICLR 2025
rejected response. The Chat category involves comparing bad and a good model response in general-doman conversation settings while Chat- Chad-Hard does the same for good model response compared to great model response. The Safety category measures the preference between refusal response and a compliance response to an unsafe user request. The Reasoning category assesses the preference between correct and incorrect responses related to math and coding prompts. The accuracy for each category is calculated based on the proportion of tasks it gets correct (i.e. prefer chosen over rejected), except for the Reasoning category, which up-weighs math samples to ensure math and coding prompts contribute equally. Overall accuracy is calculated using the mean of the four category scores.
3.2 TRAINING
SteerLM Regression Following Wang et al. 2024d), we train SteerLM Regression Reward Models consisting of a base model and linear layer projecting the final layer dense representation of the end-of-response token into five scalars, one for each HelpSteer2 attribute. Such models are optimized using MSE loss function, which seeks to minimize the squared error between the predicted attribute value and the ground truth attribute value. In addition, we train separate model only on the Helpfulness attribute.
Bradley- Terry Following Ouyang et al. (2022): Bai et al. (022). we train Bradley-Terry style Reward Models, consisting of base model and a linear layer projecting the final layer dense representation of the ond-of-response token into a scalar reward. Models are trained to maximize the directional distance between the reward for the chosen response (ye) and the rejected response (y-), as in Eq. thereafter referred to as Regular BT.
CBT - log (o(re Ve) re r0 Ur)))
(I)
Given that HeelpSteer2-Preffeeee contains not only the direction of preference between two responses but the magnitude (m) of this preference (1 slightly better, 2 better, 3 much better), we also experiment with Bradley-Terry with Margin loss introduced by Touvron et al. (2023) in Eq. 2
thereafter referred to as Margin BT.
MMT = - log (a(ra(z, ve) ro (o - m))
We also introduce a new loss function named Scaled Bradley-Terry in Eq. hereafter referred Scaled BT. Similar Margin BT, its motivation lies in utilizing the preference magnitude information. However, we use the margin term outside of the log-sigmoid function rather than inside. From the perspective of data utilization, this can be viewed as a repeated sampling of response pairs for which the preference magnitude is higher. From the perspective of model training, this can be seen as requiring models to learn more (i.e. larger update) from responses that we know to be greatly different from each other (more in Appendix ! Unlike Margin BT, Scaled BT does not assume that the distance between the chosen and rejected rewards needs to be as least as large as the margin term.
LSBT = -m log (a(ro( y.) - ro(2, B+)))
Finally, we also train BT models initialized on the Helpffulness-nnl SteerLM Regression Model. The Regression model is trained predict helpfulness between 0 and 4, which can potentially initialize the model better than the base model, which otherwise has high loss at the start training.
Pairwise Justifier To explore training reward models using preference justifications rather than preference scores, we train Pairwise Justifier reward models similar to how proprietary models such as GPT-4-Turbo are used for LLMdas-a-Judge in AlpacaEval (Taori et al. 2023) and Arena-Hard (Li et al. 2024). In these settings, the LLM is prompted to generate detailed comparison of the two responses before finally generating a statement such as "Response 1 is better than Response provides unique opportunity for examining whether preference
2". Seapstter2-PPefeeeee justifications can be used to train more accurate reward models (compared to using preference score) when training data is kept the same. To train such models, seek to generate the preference
5

---

Published as a conference paper at ICLR 2025
justification conditioned (prompt) @Response 1: 1:\n(response 1) \n@Response 2: An (e:nn(resss (response \nBetween @Response 1 and @Response 2, which is better?. This model is then optimized using Cross-Entropy loss as typically used for supervised fine-tuning of language models. In this setting, we format the preference justification by concatenating preference elaboration followed by preference statement, which is always in the format @Response 1/2
better. This allows automatic extraction of the better response for evaluation. We also experiment with i. including up to three preference justifications (i.e. All Justifications set) instead of one per task and ii. augmenting the dataset by flipping Response 1 with Response and correspondingly updating the preference label (which also minimizes position bias) and iii ablations to understand the role of each component.
Training Details
(Dubey et al. 2024) as the base model. Our initial exploration of training reward models showed that Llama-3.1 1.taa.3.--7055- Instructttt performs better as an initial model than .lama-3.1-70B as well as Nemotron 4 340B (Nvidia et al. 2024) and Llama-3-70B (Meta AI, 2024), as used by Wang al. (2024d). Hyper-parameterss for training and the search range for them are in Appendix
our experiments, we use Llama-3.1
4 REWARD MODEL RESULTS
RewardBench Chat-Hard
Hyperparams LR
Model Type
Model
Overall
Chat
Safety
Reasening
SteerLM Regression
HelpSteer Altribules
92.4
95.0
85.5
94.0
95.1
le-6
Helpfulness Only
93.0
97.2
84.2
94.6
95.8
2
Brudley-Terry scratch)
Regular
91.5
97.5
80.3
90.5
97.9
3e-6
Margin
78.5
94.6
2e-6
Scaled
92.7
97.8
83.5
93.2
96.0
2c-6
Brudley-Terry (init. with Helpfulness- only Regression Model)
Regular
92.7
98.9
82.9
93.7
95.4
1e-6
Margin
le.6
Scaled
93.7
98.0
85,7
94.3
96.7
le-6
Scaled ExPO
97.5
85.7
95.1
98.1
1c-6
Pairwise Justifier
Full Preference Justification
94.4
82.2
3e-6
(- three justificatinns task)
86.1
96.1
71.7
89.5
87.2
2e-6
(- augment data flipping labels)
95.8
76.1
3e-6
(- Preference Elaborations)
88.4
96.4
78.7
93.4
85.3
1c-6
(- Preference Justifications
90.0
96.1
80.0
93.1
90.9
3e-6
only Response
lahel)
External Baselines
Skywork Reward Gemma 2 27B
93.8
95.8
91.4
91.9
96.1
TextFval-LLlama
90.1
93.2
Sky Skywork-Critii- Critic-Llama Llama- 3.1-70B
93.3
96.6
87.9
93.1
95.5
SFR a. 3.1-----1-0000 2311-70B-Judee
92.7
96.9
84.8
91.6
97.6
Nemotron-4-34085 Betoroon----00--Rwewwaa
Reward
92.0
95.8
87.1
88.8
91.3
80.3
92.8
90.6
GPT-4o-2024-08-
96.1
76.1
Claude m.5 Sonnet 20240620
84.2
96.4
74.0
81.6
84.7
Meta-Llama-3, 1...aamn.---0----me 1-170--Insttret
97.2
70.2
Table Performance of Models on RewardBench. Higher is better for each category. All models are trained by us using 1--ma--..-70BIInsii as base model except External Basclines, the scores for which are taken from RewardBench leaderboard (AllenAl 2024)
SteerLM Regression As shown Table training a SteerL.M Regression only on the Helpfulness attribute leads slightly better performance RewardBench overall (93.0 vs 92.4), relative to training on all five HelpSteer attributes as proposed by [Wang et al. (2024d). While training on all five HelpSteer attributes can provide more insights to other dimensions of the response (cor- rectness, coherence, complexity, verbosity), training with only the helpfulness attribute also makes the setup easier for training and inference. Specifically, there no longer a concern that the five objectives/dimension might conflict and the reward model produces singular scalar reward without needing derive one using weighted sum of the five attribute values.
Bradley-Terry (from scratch) Scaled BT performs much better than either Regular BT or Margin BT on RewardBench Overall at 92.7 vs. 91.5. This is likely because Scaled BT can most effectively the preference magnitude information guide model training.

---

Published as a conference paper at ICLR 2025
SteerLM Regression Brary To answer initial question about whether SteerLM Re- gression or Bradley-Terry is better, we find that the optimal formulation of each variant (Helpfulness- Only and Scaled BT) performs just about as well as each other on RewardBench Overall (92.7 vs 93.0). This suggests that the format that the data is collected in and the model training approach does not matter too much. Instead, the most important consideration is that the modelling details can fully account for the information captured in the annotation. In the case of Bradley-Tery Terry models, the magnitude of preference strength should be adequately modelled.
Complementing SteerLM Regression with Bradley-Terry While SteerLM Regression and Bradley- models are separately comparable, we show that they have a synergistic effect and result a stronger reward model when combined. Specifically, when initialized with the Helpfulness- only Regression model, Scaled Bradley-Terry can reach overall RewardBench of 93.7. This is likely a result of the HelpSteer2 and ShlpSteer2-Preferemee datasets containing complementary information, as first indicated by Sec. 2 Conceptually, this synergy is similar to the two-stage approach in performing preference tuning (i.e. Proximal Policy Optimization or Direct Preference Optimization) after doing supervised-innetuinng In addition, we found ExPO (Zheng et al., 2024) to be a simple and effective way of extrapolating the delta weights to further improve the model. Specifically, we SteerL.M Regression model as the weak model and the Scaled BT model
use the Helpfulness-One (initialized with the Helpfulness-One -Only SteerLM Regression model) as the strong model. We then did an extrapolation factor search between 1.1 and 2.0 at intervals of 0.1. Upon finding 1.6 to be optimal, we searched between 1.51 and 1.69 at intervals of 0.01. The final extrapolation factor was 1.52. Neither the Regular BT model nor the Margin BT model improved upon the Helpfulness-only Regression Model that they were initialized with and therefore we did not try ExPO on them. To better appreciate the contribution of certain design choices, we report ablation studies in Appendix K
Pairwise Justifier Compared to StecrLM Regression and Bradlcy-Terry models that can score each sample independently, Pairwise Justifier models which involve choosing better response between two candidates generally perform worse, with an overall RewardBench score of 90.0 or lower. We suspect that this is because this task formulation is much harder (implicitly involving scoring Response 1, then scoring Response 2 and finally determining which score is higher). This is also supported by the observation that strong external baseline models using this Pairwise Justifier approach (e.g. gpt-4o-2028 2444--222-08-006 and similar to our Pairwise Justifier models. On the other hand, SteerLM Regression and Bradley- -Tery models both decompose this problem into simpler objectives that can be directly optimized towards.
3.1--0- -.--tet--ns)r score 86.7 and 84.0, which are
As seen in Table [ training a model on more than one preference justification per task is helpful (overall score increases from 86.1 1 to 88.1%) presumably because it provide a greater diversity of reasoning paths leading to the final preference choice. This increase mostly contributed by gains on Chat-Hard (71.7 to 82.2%) while other categories do not change much. This is likely because our training data collection objective is most aligned with the Chat-Hard category (i.e. separating great general domain responses from good ones), meaning that extrapolating such reasoning to specialized- domains such as safety and reasoning (math/code) can be challenging. Data Augmentation through flipping labels is also helpful (1.3% increase in Overall), likely as it reduce the effects of position bias, which is present in our data, albeit to a small extent (5.1 % difference in preference). Interestingly, training on just the Preference label (Response 1 or 2) does better than train on either the Preference Statement only (w/ Elaboration) or the Full Preference Justification. This is likely because the model can implicitly learn why response might be better than the other in way that's 's more effective than grounding the choice based on human-wir written explanations. Nonetheless, it is important to bear in mind that the current results are based on small general-domant dataset comprising only 6.6k tasks, written in free-form manner. Having a dataset which is different in domain, size or guidance to annotators (e.g. more structured) can lead to different conclusions. We leave further explorations to future work.
Comparison with External Baselines Relative to the top performing external baseline (Skywork- Reward-Gemmnn -emma---227). our best performing model (Scaled BT + ExPO) is slightly better in terms of overall RewardBench (94.1 1 vs. 93.8). However, looking more closely at the individual categories, our best model does better in all categories (Chat, Safety and Reasoning) except Chat-Hard On Chat- Hard, it trails substantially behind Skywork-Reward Geaaudd-Gmmm------ (85.7 vs. 91.4). To understand possible reasons for this, looked more closely at the constituent sources of data for the Chat-Hard
7

---

Published as a conference paper at ICLR 2025
category. We discovered that the Chat-Hard category is composed of data from two sources LLMBar (Zeng et al., (2024) and MT-Bench (Zheng et al. 2023). LLMBar contains five different subsets that used in Chrd, three of which are based on human annotations as ground-trutt labels, while the other two (Adversarial-Ci)
and Adversarial-GPTOun use GPT-4 annotations as ground-truth labels. Similarly, MTT-Bench also involves using GPT-4 judgements the ground-truth.
Chat-Hard
Human Ground Truth
GPT-4 as Ground Truth
Source Sudhset
LLMBar- Adverrarll
LLMBar natural
LLMBar-Adwee Adversarial
MT-Bench hard
munnual
neighbor
GPThnst
GPTOut
Scaled BT ExPO
85.7
76.1
88.8
95.0
87.0
72.3
75.7
Skywork-Ree Reward-GGemma-22 Gemma---227 2--2BB
91.4
78.3
89.6
96.0
97.8
91.5
86.5
Table 2: Performance of Models on RewardBench Chat-Hard Category. Higher is better for each subset. Scaled BT + ExPO uses Llama-3. 1-nma--..-70B-I-- 1-ttt-nstruutt as base model and initialized using Helpfulness-Only SteerL.M Regression training while numbers for Skywork-Reward Rew-eed----mmm--22201 are taken from RewardBench leaderboard (AllenAl 2024)
As shown in Table 2 we found that our best model performs similarly (within 2.2% difference) to on subsets that use human annotations as ground-truth while doing much worse (10.8 19.2 %) on subsets that use GPT-4 judgements ground-trute truth. One possible explanation that the constituent dataset for the training data of Skywork-Reward Reward -emd----mm-------7 contains GPT-4 annotated data and hence can better model GPT-4 judgements. We find evidence of this possibility from Skywork-Rewe Reward-Gemma-
-ewor--Reward--Gemm-------
training data description (Liu & Zeng 2024) as it claims that the training data blend contains the Offset Bias dataset (Park et al. (2024), which has data as judged by GPT-4, Interestingly, the creators of LLMBar (Zeng et al., 2024p and MT-Bench (Zheng et al., 2023) have also noted the tendency of these tasks to overly favor models trained on
GPT-4 data. Overall, this implies that our best model performs on par with the top external baseline model on the Chat-Hard category when biases associated with training GPT-4 judgements accounted for.
Table 3: Comparisons of different types of Reward Models in terms of aspects that they are strong on  and application settings they are suitable for.
Aspects Inference Speed
Application Settings
Accuracy
Interpretahility
Data-Filtering
RLHF
Human-in-Loop
SteerLM Regression Bradley-Tene -erry Pairwise Justifier
1/
x
When to use each type of reward models? While much of the discussion above compares reward model types in terms of their RewardBench overall accuracy, real-world use cases for Reward Models have many other considerations. We analyze a few of these aspects and how such considerations influence suitable application areas for each type of Reward Model, illustrated in Table 3
In terms of accuracy, Bradley-Terry models are the highest (when initialized on Helpfulness Only SteerLM Regression models), closely followed by SteerLM Regression models and lastly Pairwise Justifier models. Pairwise Justifier models are the most interpretable as they can generate an expla- nation of why they prefer a response over another. SteerLM Regression models also somewhat interpretable as they can be used to predict the attribute values for various dimensions of a response (e.g. high in verbosity but low in complexity). Another factor which makes SteerLM Regression models interpretable is that the predicted scores are well-calibrated. The scores are typically within the range provided in the training data in this case between 0 4, with each score having a real world meaning e.g. 4 means perfectly helpful, 3 means mostly helpful and O means not helpful. However, Bradley-Terry Models are not calibrated which means there's no set interval for predictions. In the case of the Scaled BT ExPO model, the range for rewards on RewardBench responses was [-33.50, 2.654]. BT Reward Scores also cannot be seen in isolation but must be interpreted in the context of the scores of other responses to the same prompt (see Appendix G Finally, SteerLM Regression and Bradley- Terry are both extremely fast for inference requiring the equivalent of generated token in terms of compute while Pairwise Justifier models can require hundreds of tokens.
8

---

Published as a conference paper at ICLR 2025
Because of these characteristics, different types of reward models likely useful for different applications. SteerLM Regression is most suited for filtering SFT data (Nvidia et al. 2024) because data can be filtered both based among 10 responses to the same prompt). On the other hand, Bradley- with relative score filterin. Bradley-Terry models are most suited for RLHF given that they the most accurate with SteerLM Regression a close second. Finally, Pairwise Justifier models can be evaluation, where the explanation provided by the models can assist humans making a final decision. SteerLM Regression models can also do this by providing attribute values to dimensions the response (e.g. complexity, correctness), but its format is less flexible for supporting human interpretation.
absolute score threshold or relative score (i.e. highest score
models can only be used
most suitable in supporting
ALIGNED MODELS
To demonstrate the usefulness of our best reward model and HelpSteer2-Preference Preference dataset in aligning language models to be helpful, we use them in three popular alignment algorithms.
5.1 EVALUATION
Following Wang al. (2024d); Dong et al. (2024); Meng et al. (2024). we use three metrics to measure aligned models' ability to be helpful in responding to user prompts: GPT-4-Turbo MT Bench (Zheng et al. 2023). AlpacaEval 2.0 Length Controlled (Dubois et al. 2024 and Arena Hard (Li et al.]] 2004). We report the mean number of characters in MT Bench responses to give sense of response length. MT Bench is also referenced as a validation metric for checkpoint selection. Further details are in Appendix H
5.2 TRAINING
We the --1- 1-770--IIstueee model (ubey al. 2024 all experiments and Scaled BT + ExPO (94.1% RewardBench) as the reward model for PPO and REINFORCE. Training hyperparameters and the associated search range are in Appendix
initialize the policy model for
Direct Preference Optimization (DPO) Following Section 3.2 we transform the three variants of
Bradley-Terry into corresponding DPO objectives: Regular DPO (Rafailov et al. (2023) corresponds
to Eq. Margin DPO (Touvron al. (2033 to Eq, 2 and Scaled DPO to Eq. We train models
using the HelpSteer2-Pris Prrrferreeeeeeeeeceee dataset.
Proximal Policy Optimization (PPO) Following the RLHF recipe prescribed by Ouyang al. (2022), we align the policy model via PPO (Schulman et al. 2017). We initialize the value model with the reward model. We found it useful run 2 rounds on PPO, where we pick the best checkpoint in round to initialize the policy/reference models for Round 2. The value model is always reinitialized prompts.
with the reward model at each round. Our training uses the HelpSteer2-
We align the policy model with REINFORCE (Williams 1992). Following Ahma- dian et al. 2024). we use the KL-regularized reward and employ the leave-one-out baseline, sampling four responses per training prompt (Kool et al. 2019). We train on HelpSteer2-Preferee prompts.
REINFORCE
5.3 RESULTS
As shown in Table most attempted algorithms improve relative Llama-3. 1-70B-Insttrtt demon- strating the strength of the HelpSteer2-Prefeeence dataset and trained reward model.
Offfine vs. Online RLHF Across three DPO variants, there consistent improvement over the base Llama-3.1 1-nnstructt model in terms of GPT-4-Turbo MT-Bench as well as AlpacaEval 2.0 LC. We find that Scaled DPO performs best, underscoring the importance of adequately modelling the preference collected. However, version of DPO beat PPO REINFORCE
strength information (across any of the three alignment metrics), highlighting the importance using online training along with reward information. We report ablation studies using alternative reward model in Appendix [.

---

Published as a conference paper at ICLR 2025
Aligned Metrics
Hyperparams
Model Tipe
Model
MT Bench (GPT-4-Turbo)
Meun Response
AlpucaEval 2.0
Arena Hard (95% CT)
LR
KL.
Length (Churs.)
Offline RLHF
Regular DPO
8.66
1502.1
40.4 (1.66))
52.8 (-2.7, 2.7)
2e-7
0.01
Margin DPO
8.58
1496.6
41.1 (1.16)67)
52.6 28)
2e-7
0,001
Sealed DPO
8.74
1514.8
41.0
(-24,
Online RLIF
PPO REINFORCE
8.74
1842.8
43.8 (1.76)
5..6,
1e-61le-7
0.005/0.01
8.98
2199.8
$7.6 68)
85.0 (-15,, 1.5)
5e-7
0.01
External Baselines
1Jama-3.1-700-- 3.t-rtt-nnttu
55.7 (-29.22))
Llama-3. 1-405 Instract
8.49
1664.7
39.3 (1 (3)4)
69.3 (-2.4 2.2)
Clonde-3 3-5 3-5-Sonnet-201 212440000
8.81
1619.9
$2,4 47)
(-19 1.1.1.9.1 1.7)
GPT-40-2024-0S-13
(7.4
79.3
Table 4: Performance of Aligned Models. Higher is better for each metric, except Length. All models are trained by us using Llama-3. 1.mma--.--70---. 1-70--Instruct as a base model except External Baselines, the scores for which are taken from Arena Hard (LMSys 2024) and AlpacaEval leaderboards (Tatsu-Lab) 2023)
Variants of Online RLHF: PPO vs. REINFORCE Both PPO and REINFORCE can improve performance. with REINFORCE showing a marked advantage. Similar to Ahmadian et al. (2024). we find that REINFORCE is much better at maximizing the reward than PPO (See Appendix [ reward curves). We hypothesize that this is due to differences in baseline estimation PPO and REINFORCE. In PPO, learned critic function is used to estimate state values, which can introduce bias and instability (Kumar et al., 2020). In contrast, for REINFORCE we use lcave-one-out baseline estimator- unbiased and stable Monte-Carlo estimator of the policy's ss value function (Sutton 2018). This leads to more stable training, and allows REINFORCE to outperform PPO.
Comparison to Frontier Models Our best model (trained with REINFORCE) achieves competitive performance with frontier models such as GPT-4o and Claude 3.5 Sonnet across the three popular alignment benchmarks (MT Bench, AlpacaEval 2.0 LC and Arena Hard). This highlights the strength of our Reward Model to guide online RLHF. REINFORCE significantly increases the response length compared the base models, but its high length-controlled AlpacaEval 2.0 score suggests that the extra tokens are meaningful. However, it is important to bear in mind the limitations of these automated evaluations, as all of them use LLM-as-a-judes as-a--udge GGPT-4-Turbo) which is generally perceived as weaker than the cop-performing External Baselines like GPT-4o and Claude 3.5 Sonnet. As a result, the LLMM-jugee might not be able to adequately evaluate the quality of such model responses, in relation to responses generated by our aligned models. This means that such automated evaluation for alignment may be insufficient to inform on how our models perform in relation to such frontier models in diverse cases. Given that our main purpose is to demonstrate how trained reward models can be useful for RLHF, we leave more holistic evaluations to Appendix M
Case Study We adopt a prompt that many have recently been vibe-testing LLMs with How many r in strawberry? Among models in Table only REINFORCE can correctly answer it.
Model
Response to "How many r in strawberry?"
Regular / Margin / Scaled DPO
There are 2 r's in the word "strawberry".
PPO
There are 2 R's in the word "strawberry".
REINFORCE
A sweet question! Let's count the "R"s in "strawberry":
1. S 2. 3. R 4. A 5. W 6. B 7. E 8. 9. R 10. Y
There are #*3 "R"s** * in the word "strawberry".
Llama-3.1 3.1--...1-7--tsteeee
There are R's "strawberry".
Llama-3.1-4000 -nnStut
There are 2 Rs in the word "strawberry..
Claude-3-5-Sonnet-2022:
There are two "P" letters in the word "strawberry"" strawberry The first "r" appears after "st" and the second "P" is part of the "my"" at the end of the word.
GPT-40-2024-05-13
There are two "T" letters in the word "strawberry."
Table 5: Model responses to "How many r in strawberry?" Newlines in responses not shown.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
The prompts and responses in the Helpstter2-PPeferrnce dataset do not contain any unsafe content (e.g. harmful content, illegal activities, profanity, bias and stereotyping) any content containing Personally Identifiable Information (e.g. name, address, SSN, email, phone numbers), which were excluded by the HelpSteer2 (Wang al. 2024d) collection effort. Annotators who supported the construction of the dataset were contracted through Scale AI, which completed ethical review prior to the start of data collection. Scale engages the Anker Methodology. GISC Impact Sourcing Standard, and UN Sustainable Development Goals to provide fair and competitive pay. The specific pay is calculated based on many factors, including the specific project, the specialized skillset and expertise required, regional costs of living and then transparently listed on Scale AI platform. Scale Al also provides multiple channels for questions and support, including 24/7 support teams, community discussion channels with specially trained moderators, and "speak up" hotline where contractors can report concerns anonymously. Worker concerns can be submitted to and are reviewed by the Remotasks support team, and pay disputes are reviewed by support specialists trained in this area.
REPRODUCIBILITY STATEMENT
Data Pre-processing: Sec. and Appendix
Training Hyper-parameters: Appendix
Training Compute: Appendix
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXh::2402.14740 2024.
AllenAI. Reward bench leaderboard. httpss///oggiigfaee.eco////////p
reward bench 2024.
Duane F. Alwin and Jon A. Krosnick. The Measurement of Values in Surveys: A Comparison of Ratings and Rankings. Public Opinion Quarterly, 49(4):535-552, 01 1985. ISSN 0033-362X. doi: 10.1086/268949. URLht:s:://doii
Ron Artstein and Massimo Poesio. Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555-595 2008. doi: 10.1 1622/col..0-0044-222 URL. https
//aclanthology. wrg. J08 4004
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovit, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating Ilms by human preference, 2024.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingho Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHF, 2024.
Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom,
11

---

Published as a conference paper at ICLR 2025
Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve. Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Ilyaan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Mísra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika lyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Yearala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins. Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathiuu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykow, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer. Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa. Sanjay Singh. Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov. Tong Xiao, Uijwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag. Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho. Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina- Elena Veliche, Itai Gat, Jake Weissman, James Gehoski, James Kohli, Japhet Asher, -aptiste Gaya, Jeff Marcus, Jeff Tang. Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie. Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly
12

---

Published as a conference paper at ICLR 2025
Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang. Lailin Chen, Lakshya Garg. Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L.. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shugiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad. Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian. Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li. Tianjun Zhang. Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang. Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao. Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The Ilama 3 herd of models, 2024. BtLhttts::/// http:///axvvv..o org/ abs
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.00 04475, 2024.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5988 6008. PMLR, 17-23 Jul 2022. URL https:: http::eee////cceeedeee ml wtr.rress//l1////hhtaarrra
html
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get a baseline for free!, 2019.
Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforcement learning via distribution correction. Advances in Neural Information Processing Systems, 33: 18560-18572, 2020.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnay Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations democratizing large language model alignment, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and lon Stoica. From live data to high-quality benchmarks: The Arena-Hard pipeline. https: April 2024.
7/1mays. ///myys.org/bllo/2244--002 1atp:////---0000----------aa-aae
13

---

Published as a conference paper at ICLR 2025
Chris Yuhao Liu and Liang Zeng.
Skywork reward model.
https: https://huggingface
September 2024. URL https:/
LMSys.
Arena-hard-auto
leaderboard.
https://githhb..omm com/ sys/
arena Sasa-harce wwaaarrauuee 2024.
John A. McCarty and L. J. Shrum. The Measurement of Personal Values in Survey Research: A Test of Alternative Rating Procedures*. Public Opinion Quarterly, 64(3):271 298. 11 2000. ISSN 0033-362X. doi: 10. 1086/31 17999.
Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference free reward, 2024.
optimization prefereneee with a reference-
Meta AI. Llama 3 main/MODEL CARD md 2024.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question with human feedback. In arXiv, 2021.
NLTK. nilk:tokenizzessntttoooeiii /ttpa//aww..aaap/////////ooooo/n sert sent_tokenizee htmreniiieeemmm.ll.... 2024.
https://wwwy nltk. http://wwwww.....////////a.../a/
Nvidia, :, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brun- dyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grzegorzek, Robert Hero, Jining Huang. Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Fileen Long, Ameya Sunil Mahabalesh- warkar, Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupinder Parmar, Mostofa Patwary, Krzysztof Pawelec, Wei Ping, Shrimai Prabhumoye, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Sewall, Pavel Shamis, Gerald Shen, Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shubham Toshniwal, Hao Wang, Zhilin Wang, Jiaxuan You, Jiaqi Zeng. Jimmy Zhang, Jing Zhang, Vivienne Zhang, Yian Zhang, and Chen Zhu. Nemotron-4 340b technical report, 2024. URL https: / /wt.ivaivvv...////aa//aaa/aaaaaa org/ abs 2406 11704
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll I.. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Junsoo Park, Seungyeon Jwa, Meiying Ren, Dacyoung Kim, and Sanghyuk Choi. Offsetbias: Leveraging debiased data for tuning evaluators, 2024.
Gloria Shillips-Wren. Daniel J. Power, and Manuel Mora. Cognitive bias, decision styles, and risk attitudes in decision making and dss. Journal of Decision Systems, 28(2);63-66 April 2019. ISSN 2156-7052. doi: 00.1080/12401125.2000 164,55699,,,,, URLhitp: //x doi org org. 11.. 12460125. 266966665001
1646509
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X1v:170070063777.20 2017.
Seikit-Leam. Cohen kappa score. htpps///ccikttteeern..ooo org generaate//klearrt http://riiaaarammmmtii.i......c wettrricc.ohee__kppp__ccrrr.. 2024.
wwtpo//gg/tttttoooo http:staallleee
metrics...
14

---

Published as a conference paper at ICLR 2025
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models, 2023. URL
13548
Judy Hanwen Shen, Archit Sharma, and Jun Qin. Towards data-centric rlhf: Simple metrics for preference dataset comparison, 2024. URL / 09603
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022.
Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instraction-followin Ilama model. https://githui: com//aatso-la//ssaafoooooooo
2023.
Latsu-Lab. Alpacaeval leaderboard. https::/tattsu-lbb.gii 2023.
Tenyx. Tenyxchat: Language model alignment using tenyx fine-tuning, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Ilms for diverse user preferences: Directional preference alignment with multi-objective rewards, 2024a. URL httpttts////aii.oortt httpr//abb//42...333
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objeccive reward modeling and mixture-of-experts, 2024b. URL https://erxiv
httg/bb//200 /20.. 12845
Peifeng Wang, Austin Xu, Yilun Zhou, Caiming Xiong, and Shafig Joty. Direct judgement preference optimization, 2024c. httts:ts:///aae http//axxvvvv..//.//... httpa///.rr//a///////aa/oo//s/// 2tt.//2009.166666
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng. Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top-performing reward models. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Infor- mation Processing Systems, volume 37, pp. 1474-1501. Curran Associates, Inc., 2024d. ec /ppppr......... files httpe///ppppee22/////2/ wawp://aaaaaaaa-ppaaaaaaaaaaaaaa -atasets and Benchmarks.
URL
Track .pdf
15

---

Published as a conference paper at ICLR 2025
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. HelpSteer: auttie helpfulness dataset for SteerLM In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the As- sociation for Computational Linguisties: Human Language Technologies ( (Volume I: Long Pa- pers), pp. 3371-3332 Mexico City, Mexico, June 2024e. Association for Computational Linguis- maa2222.aaaaloonn... 185. URL https: http:///caaaahhooooooooooooo sttpss//ecdaaahhoooooo.ooo//////
tics. doi: 10. naacl-long 1857|
Ronald J Williams.
Simple statistical pradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng. Zhiyuan Liu, and Maosong Sun. Advancing LLM reasoning generalists with preference trees, 2024.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large wttpa.t.w......aaa.a/.a/aaaaaaaa
language models at evaluating instruction following, 2024. URL
2310.07641
Chujie Zheng. Ziqi Wang. Heng Ji, Minlie Huang, and Nanyun Peng, Weak-to-strong extrapolation expedites alignment, 2024. URL https:://axxi..eee org/ www.oorggg/////eoeeeo.oo /204 16792
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving LLM helpfulness & harmlessness with RLAIF, November 2023.
Banghua Zhu, Michael I. Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward over- fitting and overoptimization rlhf, 2024. URL attps::/arxiyy. wwtp:////gg//ba/aa///..o/e
A RELATED WORKS
Eliciting Human Judgements using Preference Rankings vs. Ratings There has been series of work from sociological statistics on comparing the relative merits of ranking and rating style questionnaires in understanding human judgements. Alwin & Krosnick (1985) found both ranking and rating can perform similarly in terms of modelling overall preference but capture different (but complementary) aspects of their preference. McCarty & Shrum 2000) similarly found that forced-choice ranking yielded more diffcrentiated preference data compared to Likert-type ratings, with the additional advantage of fewer extreme scores. We build upon this literature to examine whether the same differences can be observed in collecting nomain-geneere data for preference tuning of LLMs.
Bradley- -Tery style Preference Datasets HH-RLHF (Bai et al.) 2022) was the first open- general-domain,
preference dataset released by Anthropic, containing over 160,000 pairs of responses and annotations on which one response is preferred amongst the pair. However, there has been many concerns relating the quality of this data (Wang et al. 2024c| Shen al.) 2024) There are many more domain- nomain-specific datasets for individual tasks such as long-form question answering: OpenAl WebGPT Nakano et al. (202T)). summarization: OpenAl Summarize (Stiennon et al. 2022), online forum responses: Stanford Human Preferences Dataset (Ethayarajh et al. 2022) but they less useful for modelling general-domain preferences.
Regression-style Preference Datasets OpenAssistant (Köpf et al. 2023 is prominent crowd- sourced, open-source general domain human-annotater dataset with >10,00 conversation trees released by the Open Assistant organization. Open Assistant contains Likert-5 annotations for various attributes including helpfulness, creativity, and humor. Ultrafeedback (Cui et al., (023) is GPT-4 annotated dataset containing 64,000 prompts each with 4 responses. For each response,
16

---

Published as a conference paper at ICLR 2025
there are 4 attributes annotated Likert-10 scale: helpfulness, honesty, instruction-followii and truthfulness. With each rating, there is also short (2-3 sentences) rationale that explains the rating. Ultrafeedbcck has previously been converted into preference dataset (Tunstall et al. 2023 based on the mean of the 4 attribute scores with the highest scoring response as the chosen and one of the other three responses selected at random the rejected. HelpSteer (Wang et al. 2024g) is a comman-nnnotttdd general domain dataset with 10,000 prompts, each with 4 responses. Each response is labelled by annotator on five Likert-5 attributes helpfulness, correctness, coherence, complexity, and verbosity. However, [Shen et al. (2024); Wang et al. (2024d) found the above datasets to substantially noisy, which means that reward models are unable different between responses are not drastically different in quality. HelpSteer2 (Wang al., 2024d) is a follow-up HelpSteer with 10,000 real-world prompts, each with responses. Each response annotated on the same five I Likert-5 attributes. However, every response is annotated by 3-5 annotators with the final ratings obtained by removing outlier annotations and then taking the average of the remaining annotations.
B PREFERENCE RANKING GUIDELINES
B. RANKING PRIORITIZATION
Ratings should be made by prioritizing the response characteristics below in the order they are stated. Trade-offs should favor higher priority characteristics.
1. Instruction Following
Responses should follow all requests in the prompt, following the instructions in the prompt the primary criteria for ranking responses.
Many prompts will have a clear instruction that has additional or implicit requirements, e.g. "plan an itinerary for 5-day trip to Portugal that focuses on surfing" should be evaluated on whether an itinerary is produced, whether is days long and in Portugal, and also whether it includes lot of surfing All instructions should be considered when ranking responses, with the core instructions being weighed more heavily.
If specific formatting (table, json) is asked for a prompt that should be considered as the high priority "instruction following" and not general "formatting" which is lower priority.
2. Correctness
Answers which are factually correct should be rated higher than answers that are incorrect.
Annotators should search the internet when they are unsure about correctness.
Misleading or incomplete answers are considered less correct than complete answers.
If prompt (question) contains false premise, the response that pushes back on it should be preferred.
When question cannot be answered definitively, the response which expresses uncer- tainty should be preferred.
3. Formatting
When specific formatting is requested responses with better formatting are preferred.
Vendor tool should render markdown for easier assessment of formatting (markdown tables, lists, shell scripts, etc. should be rendered properly).
Formatting encompasses anything that is appropriate for the response, e.g. an itinerary being split by day, including a markdown table when it makes sense but was not asked for, any appropriate use of markdown, responding with a shell script, etc.
4. Clarity
Answers that are easier to understand, provide more clarity (via explanation or intro- duction), or are generally better written are preferred.
Unnecessary verbosity, or providing extra information that irrelevant should be penalized.
17

---

Published as a conference paper at ICLR 2025
B.2 RANKING STRENGTH
We would like to collect three levels of preference, described below:
1. Response 1 is slightly better than Response 2 (or vice-versa)
To be used when the responses are similarly appropriate and the difference minor or
matter of personal preference.
There should be no difference in Instruction Following or Correctness if this is selected.
Minor differences clarity and formatting warrant this response.
When the annotator considers the options to be tied in all other aspects, they should prefer the shorter answer (in unlikely circumstances of same length. use their own judgment)
2. Response is better than Response 2 (or vice-versa)
To be used when one response is clearly better than the other, such as:
If one response is clearly better than the other, but the worse response is still appropriate (it is not incorrect off topic).
the worse response fails to follow some instructions or is correct but not well phrased/structured, and the better response also fails some instructions the same level, but has no other flaws overall.
If the better response fails to follow some but not all instructions or is misleading but the worse response does not follow instructions all or is wrong, this should be selected.
If both answers follow instructions and correct, but one is clearer and/or better formatted, this should be selected.
3. Response much better than Response 2 (or vice-versa)
To be used when there is significant difference between the two responses.
one answer is correct and the other isn't 't, this should be selected.
If one answer follows all instructions and the other does not follow instructions, this should be selected.
there are major differences in readability and formatting, this should be selected.
4. Neither response valid
If neither response follows the instructions or provides a correct answer, this option should be chosen.
This is meant to be used only for egregious issues. If both answers could be improved, but they follow what the prompt asked with no critical error then a preference should be selected instead.
B.3 RANKING EXAMPLES
Note that the full guidelines contain more examples, but we show here only four illustrative ones, one for each preference strength.
B.3.1 RESPONSE SLIGHTLY BETTER THAN RESPONSE
Prompt
Describe the process of photosynthesis. Additionally, explain how variations in light intensity can affect this process and how plants have adapted such changes.
Response 1
Photosynthesis the process by which green plants and some other organisms convert light energy, typically from the sun, into chemical energy in the form of glucose (sugar). It can be broken down into two stages:
1. degendee reactions: These oceur in the thylakoid membranes of chloroplasts. Light energy absorbed by chlorophyll and other pigments, splitting water molecules and producing oxygen, electrons, and ATP (adenosine triphosphate).
18

---

Published as a conference paper at ICLR 2025
2. Light-independent reactions (Calvin cycle): These occur in the stroma of the chloroplast. Carbon dioxide is taken in from the atmosphere, and with the help of ATP and electrons produced in the light-dependent reactions, it is converted into glucose and other organic molecules.
Variations in light intensity can affect photosynthesis the following ways:
Low Light Intensity: Photosynthesis decreases there is insufficient light for the light-dependent reactions. This leads to a lower production of ATP and NADPH, slowing down the Calvin eycle and reducing glucose production.
High Light Intensity: Excessive light can damage chlorophyll II and other pigments, leading to reduced photosynthesis. To protect themselves, plants have adapted by mechanisms like photoprotection and non non-photochemical quenching to dissipate excess light energy as heat.
Plants have evolved several adaptations to cope with changes in light intensity, including adjusting the size of their chloroplasts, altering the angle of their leaves, and using pigments that can capture light different wavelengths. Additionally, they can regulate their stomata to control the entry of carbon dioxide and reduce water loss during times of intense sunlight. These adaptations help optimize photosynthesis under varying light conditions.
Response 2
Photosynthesis occurs chloroplasts of plant cells. uses sunlight, carbon dioxide, and water to produce glucose and oxygen. Light energy is absorbed by chlorophyll, splitting water molecules and releasing oxygen. The energy from light is used to convert carbon dioxide into glucose. Higher light intensity increases the rate of photosynthesis, dependent reactions. However, other factors like CO2 and temperature can limit the rate. Too much light can damage the photosynthetic machinery.
light energy drives the light-
Plants adapt to varying light through changes in leaf angle, altering chlorophylI production, open- ing/closing stomata, and photorelocation of chloroplasts. These adaptations maximize photosynthesis in both low light and intense light conditions by balancing light capture with CO2 intake and production capacity.
Preference Justification
Response is slightly better than Response 2. While both responses correctly answers the prompt, the style and formatting of the better response is preferred. It also gives additional facts about photosynthesis.
B.3.2 RESPONSE 1 IS BETTER THAN RESPONSE 2
Prompt
Discuss the benefits of renewable energy sources.
Response 1
Renewable energy sources such solar, and wind offer numerous benefits to mitigate climate change. Unlike exhaustible fossil fuels, renewables rely on naturally replenished resources. Transitioning to renewables will significantly benefit public health and the environment by reducing hazardous air and water pollution produced from extracting and burning fossil fuels.
Response 2
Renewable energy sources, such as solar, wind, and hydro power, offer numerous benefits. They pro- duce clean energy, reducing greenhouse gas emissions and reliance fossil fuels. Additionally, renewable energy is sustainable, helping to protect the environment and conserve resources for future generations.
Preference Justification
@ @essoonse is better than @ @Response 2. Both responses approach the prompt from slightly different angles, One could consider @ @Response the better response given its additional information about transition renewable sources and how it could impact the environment.
B.3.3 RESPONSE IS MUCH BETTER THAN RESPONSE 2
Prompt
Make an itemized list elements with actions that can increase the emission of oxygen. Item markers must be lower case letters starting from D.
19

---

Published as a conference paper at ICLR 2025
Response 1
Planting trees: Trees release oxygen through the process of photosynthesis, making them the most effective way to increase oxygen levels.
e. Growing algae: Algae ponds or water bodies can play significant role oxygen production. Cultivating algae increase their growth and subsequent oxygen release.
f. Creating green spaces: Designating and maintaining green spaces in urban areas, such as parks and gardens, help support plantations and increase oxygen levels.
Promoting indoor plants: Placing indoor plants in your living or working space can enhance oxygen production and improve air quality, benefiting the overall oxygen levels indoors.
Response 2
D. Photosynthesis: Photosynthesis is the process by which plants, algae, and some bacteria convert light energy from the sun into chemical energy in the form of organic compounds, such as glucose. This process releases oxygen as a byproduct, which is then released into the atmosphere.
E. Respiration: Respiration is the process by which cells in living organisms convert glucose into energy. During this process, oxygen is consumed and carbon dioxide produced. Increasing the rate of respiration in living organisms can increase the amount of oxygen released into the atmosphere. Exercise: Exercise is physical activity that can increase the rate of respiration the body. As the body works out, it burns more glucose and consumes more oxygen, leading emissions.
increase in oxygen
G. Forestation: Planting more trees and forests can increase the amount of photosynthesis occurring in an area, which can lead to an increase in oxygen emissions. Trees and other vegetation absorb carbon dioxide from the atmosphere and release oxygen byproduct of photosynthesis.
Preference Justification
Response is much better than @ @Response The prompt specified that the item markers should be lower case. @ Responss starts from the letter "D", but it is not lower case. In addition, @ Response has incorrect facts about respiration and is not coherent/fluent regarding the prompt.
B.3.4 NEITHER RESPONSE IS VALID
Prompt
When it is Ipm in San Francisco, what time is it in Singapore?
Response 1 It is 8pm in Singapore when it is Im in San Francisco.
Response 2 is 10pm in Singapore when is 1pm San Francisco.
Ranking explanation
Neither option is factually correct. The correct answer depends whether Daylight Saving Time is in effect. When it is I pm in San Francisco during Standard Time, it is 5 am the next day in Singapore. During Daylight Saving Time in San Francisco, when it's 1 pm in San Francisco, is 4 the next day in Singapore.
C JUSTIFICATION PRE-PROCESSING
Our annotation guidelines provided minimal guidance on how the preference justification should be written within the free-text box. One such guidance was that annotators should consistently refer to the rated responses as Response 1" and "ond"@@Rspplllle "@Response 2".. This meant preference justifications do not always conform to fixed format. To better understand these justifications, we split them into sentences using NLTK sentence-tokenizer (NLTK) 2024) as well as newlines, facilitate subsequent poos-processing.
Many preference justifications contain a preference statement, in the form "@ Response (or 2) is better Most commonly (57.6% of samples), these preference statements are only contained in the first sentence. This is followed by 27.5% of samples containing it in both the first sentence and the last sentence. A minority (7.3%) contains it in the last sentence while similar proportion (7.6%) does not contain such a preference statement at all. Among those without preference statement, a qualitative analysis revealed that some state the strengths and weakness of each response without comparison,
20

---

Published as a conference paper at ICLR 2025
some claim that both responses equal (despite schema not supporting equivalence) while others make aspect-specific comparisons without stating if makes either response better.
We believe that it is important to have all justifications in similar format because cases such as LLuuue-a-judge require justifications to be standardized for automatic parsing. Therefore, we split the justification into preference statement and a preference elaboration. This enables users to freely decide they prefer to generate the preference statement first or the preference elaboration first. For justifications containing preference statement both the front and the back, we remove the trailing preference statement to avoid duplication. In addition, preference statements that are extracted from the end of justifications are frequently prefixed with terms like "Therefore. " "Overall, and "For these reasons, To ensure uniformity, we remove all prefixes before the first "@ Response 1/2 better' is encountered.
Requiring all preference justifications to have preference statement makes justifications without one difficult to use directly. After considering several approaches (e.g. prompting an LLM to restructure justifications to follow format or manually adding in the preference statement using the associated preference score), we decided that the best solution would be exclude the preference justifications without preference statement altogether. This is because the potential benefit of having additional 7.6% of data was not worth the risk of LLM hallucinations or potentially contradictory/reeettive information from the rule-based heuristic.
Finally, we prepare a subset of these justifications into the Overall Preference Justification, by ran- domly selecting one justification per task, whose annotator has given the same individual preference as the overall preference. The Overall Preference Justification set, which comprises 6, 618 samples with 6, 287 in the train set and 331 in the validation set while the All Justifications dataset has 24, 698 justifications (23, 487 train and 1, 211 val).
Attribute
All Justifications Statement (Std.) Full (Sid.)
Overall Preference Justifications Statement (Std.)
Full (Std.)
No. of justifications Justifications per task No. of sentences
24698 (inel. tied preference)
6618
2.71 (0.51)
I (0)
(0)
3.95 (2.11)
1 (0)
4.03 (2.
No. of characters
105.9 (68.3)
483.3 (266.3)
130.8 (72.7)
494.5 (258.5)
Prefers Response (%)
47.1
46.4
Prefers Response 2 (%)
52.9
53.6
Contains 'hecause' (%)
46.8
53.0
47.4
53.8
Mentions Response 1 Only (%)
7.9
0.9
7.5
0.7
Mentions Response 2 Only (%)
8.9
1.3
8.7
1.1
Mentions both Responses (%)
83.2
97.8
83.7
98.2
Table 6: Descriptive statistics for preference justifications. Preference justification consists both preference statement and preference elaboration, and both together is termed Full. Overall Preference Justifications a subset of all justifications in which only one representative justification is used for each task.
D PREFERENCE JUSTIFICATION ANALYSIS
In Table 6, we analyze the descriptive statistics for preference justifications. On average, preference justifications contain 4 sentences or 500 characters (approximately 100 words). There is slight preference for Response 2 over Response 1 (6-7%). Approximately half of all preference statements contain the word "because" while other words implying rationales such 'due to' and 'owing to' are minimal (< ().. The other half of the preference statement communicates preference without explanation. The vast majority of responses (83%) mention both responses in the preference with nearly all responses mentioning both responses in the full justification. This suggests that most annotators used a comparative approach between responses, indicating that they follow guidelines well.
To better understand the content contained in preference justifications, conducted word-level analysis of Overall Preference Justifications (including both preference statements and preference elaborations). We first lowercase the justifications and replace all non-alphanumerical symbols with
21

---

Published as a conference paper at ICLR 2025
Attribue
uuribuee relevant keywords/
(% occureence)
keywords
L.MMcclasifledd
HelpSteer Auributes
Helpfolness
Alli help, helpful, helpfulness, instraction, unhelptul, useful
11.0
77.5
Conectaess
Positive: securass, accurately, complete, correct, factual, infommaavee
12.9
61.4
Negativ: crner, false innecurate, inpemplese, incorvel, incamerrly, misses, missing, wrong
Neutral: complttness, correctness, fuct, information, understund, understanding
Coberence
Positive: clear, cleare. direct, directly, relevant
3.5
30,3
Negative: confising, inclevast, redundare, regeats, repetitive unelear, unnoceseoryy vague Neutral: clinity esherence, strecuue
Format: bulleted
formuseed listed, numbered. outline
Complesity
basic, depth, difficilt,
easy, simple, simply
Verbosity
Short: brief, concise,
Long: comprehensove, decailed, long, longer, thorough, verbase,
Neutral: detsll, details, longth, verbosity
Other Factors Sharna et al. 2023
Stylisti
Net: triendly (1.4), polite (0.6), emposhetic (1.5), optimasc (0.1)
3.9
Joyful: migiging (2 7) intertining (1 D). funny (0.5)
Charismatic: persussive (1.0) autoritative (0.3), motivating (0.2)
Sycophamic
All: match. homan style (1.0)
Joman, explicit (0. Jaaree, haman, implicil (0.03)
HelgStoonAdiieem
Helpfulnes:: relevant (16.7), well written (38.0)
90.1
Correctnes:: informative (63.7), iutbfil (16.0), better, suppunedppddddeee (1333) (22.3)
Caberenec:
(13.5) grammatically. sound logacally_sound
Complesity; higher reading agr
Verbosity: concise (19.7)
Table 7: Analysis of keywords and factors mentioned in preference justifications.
empty strings, before splitting justifications into bags of whitespace-seppratte words. We then count the number of justifications in which each word appears. From the top 500 most frequent words, we manually identify those that relate each HelpSteer2 attribute.
Based on Table 7 we found that annotators used each of the five HelpSteer2 attributes to guide Correctness-related words (12.9%) followed by
their justification writing. Most influential Helpfulness-related words (11.0 %). We suspect that the Helpfulness is under-represented because very few keywords (6) are closely associated with helpfulness while many more (21) associated with correctness. Surprisingly, verbosity is the next most influential to annotators (4.0 %), and more so compared coherence (3.5 %). We hypothesize that this might be due responses from HelpSteer2 being mostly Coherent (mean of 3.64 out of 4) meaning that both responses are likely to be perfectly coherent. Finally, complexity-related words only appear 0.6% of justifications, suggesting that they barely affect preference judgments. Overall, the proportion of justifications that can be explained by these attributes however, is low at (<32.0%). This likely due to our word-level analysis only involving the top 500 most frequent keywords, which might not be sufficient to capture all aspect-relevant features. For instance, an annotator might communicate that a response is more helpful than another, using the terms helpful, useful, or instruction (following).
To complement the word-level analysis, which is interpretable but with low recall, we also prompted an I.M,, specifically justification discusses each attribute (prompts in Table 8). LLM--hased analysis shows a much higher proportion of justifications mentioning each attribute. Nonetheless, the relative proportion between attributes remains similar with the primary factors influencing preference being helpfulness and correctness; followed by coherence and verbosity; and complexity trails behind as the least important attribute.
(Nvidia et al. 2024) to classify whether the preference
In addition, we wanted to understand if factors outside of HelpSteer attributes influenced annotator preferences. We used a list of 24 factors identified by /Sharma et al. (2023) including politeness, empathy, and persuasiveness, and prompt the LLM to classify similar manner. After classifying each factor separately, we organized them into three main categories- Stylistic, Sycophantic (i.e. similar to the prompt), and HelpSteer-Adjacenn Unsurprisingly, HelpSteer- Adjaacen factors are most commonly mentioned in these justifications (9).1%). Comparatively, stylistic factors and sycophantic factors that were found to substantially influence human and LLM judgments in previous work (Sharma al. 20233 Chiang et al.) 2024) rarely discussed by annotators (5.9% and 1.4%). This suggests the effectiveness of our guidance to annotators on focusing on the "substance' of responses.
22

---

Published as a conference paper at ICLR 2025
Attribute
Question
HelpSteer Attributes
helpfulness
the helpfulness or understanding of the response(s)?
correctness
the correctness or completeness of the response(s)? the coherence clarity of the response(s)?
coherence
complexity
the complexity or simplicity of the response(s)? the verbosity succinctness of the response(s)?
verbosity
Other Factors (Sharma al.
2023
Stylistic
friendly
how friendly the response(s) were? how polite the response(s) were? how empathetic the response(s) were? how optimistic the response(s) were?
polite
empathetic
optimistic
engaging
how engaging the response(s) were?
entertaining
how entertaining the response(s) were?
funny
how funny the response(s) were?
persuasive authoritative
how persuasive or compelling the response(s) were?
the authoritativeness assertiveness of the response(s)? how motivating the response(s) were?
motivating
Sycophantic
match human_utaayyy stylletttlll_yy agree garee_human_explice
how well the response(s) match the prompt's writing style? whether either the response(s) agree with the preferences, biases, and beliefs explicitly stated by the prompt? whether either the response(s) agree with the preferences, biases, and belief's implied by the prompt?
agree_ ggree_human_impiche
HelpSteer-Adjacet
relevant well_ written
how relevant the response(s) were to the prompt?
how well the response(s) were written?
informative
how informative the response(s) were?
truthful
how truthful the response(s) were?
how well well-supported the response(s) were?
rigorous
how rigorous the response(s) were?
structured
how well-structured the response(s) were? the grammatical soundness of the response(s)?
grammatically_soun sound
logically_sooe sound higher reading higher_reading age concise
how logically sound the response(s) were?
the reading age that the response(s) were written for? how concised focussed the response(s) were?
Table 8: Questions used for analyzing if preference justification discusses each attribute or factor. Each question was used with the template" (justification) Does the above comparison between @ Response 1 and @ Ressponse 2 discuss (question) Answer only with yes or no.'
E TRAINING HYPER-PARAMETERS
Reward Modelling For SteerLM Regression models, we used 2 epochs of HelpSteer2 data, fol- lowing Wang et al. (004dd) For Bradley-Terry models, we used epoch of SelpSteer2-PPrfeeemee data, as more than epoch resulting in overfitting similar to as observed by /hhu et al. (2024) and drastically higher validation loss and poorer RewardBench performance. For Pairwise Justification models, train for 1 epoch for each setup we also tried 2 epochs for initial experiments which showed minimal changes from only training on epoch. For each experiment, we used global batch size of 128 using Adam W optimizer with 10 warm-up steps and performed search over constant learning rates of { [1, 2, 3 e 6. We report the optimal learning rate for each experiment Table [] We save checkpoints every 10 steps and evaluate three checkpoints with the lowest validation loss as
23

---

Published as a conference paper at ICLR 2025
well as the last checkpoint. Among them, report the with the highest RewardBench overall score. For SteerLM Regression Model with all five 5 HelpSteer attributes, we do grid search over all five attributes (between -1 and 1 at intervals of 0.1) over RewardBench.
Direct Preference Optimization We trained DPO models for 2 epochs using the AdamW optimizer, after initially experimenting with 1 and 3 epochs but finding epochs to be optimal. We global batch size of 64 cepposse-piirs weight decay of 0.01, with checkpoints saved every 10 steps, and 10 warmup steps followed by constant learning rate. For hyper-parameter tuning, performed grid search over learning rates of 2, 3] - 7 and KL, penalties of (0.01, 0.001). We report the optimal learning rate, KL penalty for each experiment in Table
Proximal Policy Optimization We trained PPO models for total of 2 rounds using the Adam optimizer weight decay of 0.1, constant learning rate schedule and 30 steps of value model warmup. Due to the instability of PPO training also observed by (Zhu et al., 2023). we save checkpoints every 2 steps with our final checkpoints being at step 26 and 40 for rounds and respectively, For hyperparameter tuning we performed grid search over learning rates (5e-6, 1c-6, 5e-7, 1e-7, le-8), KI. penalties of 10.1, 0.01, 0.001, 0.005, 0.0001} } and training global batch size of [64, 128, 256). Compared to other algorithms,, we performed more hyperparameter search because we found PPO to be more sensitive to hyperparameters. the same as the rollout global batch size and found 128, 256 the best for rounds and respectively, We report the optimal learning rate, KL penalty for each experiment in Table
our experiments we set the training global batch size to be
Similar to PPO, we trained REINFORCE models using the AdamW optimizer with rollout batch size of 64, training batch size of 64, weight decay of 0.1, and a constant learning rate schedule with ten warmup steps. We sample four samples per prompt, and the leave-one-out baseline. We save checkpoints every steps, and the best checkpoint is found at step 95, for hyperparameter tuning performed grid search over learning rates {1e-7, 3e-7, 5e-7, le-6) and KL penalties of (0.01, 0.1). The optimal learning rate and KL penalty are reported in Table 4
REINFORCE
F COMPUTE REQUIREMENTS
Model
Compute (H100-eqv. node-hours)
Reward Models
SteerLM Regression
24
Bradley-Terry
Pairwise Justifier
32
Aligned Models
DPO PPO REINFORCE
16
50
Table 9; Compute required for training models, measured in H100-eqv. node-hours. Experiments run on nodes of 8 A A100/H100-800 SXM GPUs on internal clusters. For ease of comparison, every three A100 node-hours is converted one H100 node-hour.
G REWARDBENCH RESPONSE DISTRIBUTION
Fig. 2 shows the distribution of reward scores on Rewardbench responses as judged by our best Reward Model.
H ALIGNED MODEL EVALUATION DETAILS
MT Bench
We follow Meng et al. 2024 Tenyx 2024 Wang et al. 2024d) to use MT Bench (Zheng
al. 2023).
with GPT-4-Turbo (specifically GPT GPT-44
as the judge. MT Bench consists
24

---

Published as a conference paper at ICLR 2025
400
Canegory
350
rejected
differance
500
250
Count
200
150
-30
-20
0
10
Rewerd
Figure 2: Distribution of Reward Scores for RewardBench responses by our best reward model (Scaled BT ExPO, initialized on Helpfulness-Onn Onll SteerLM Regression model). Difference refers to the difference between the reward scores of chosen and rejected responses to the same prompt. When response has reward of -15, it is equally likely to chosen or rejected. This meaning the score of the response cannot be seen in isolation to determine if response is good (or bad).
of 80 multi-ture questions, each consisting of initial question and follow-up question, for total of 160 prompts. Questions are collated from 8 categories: Writing, Roleplay, Extraction, Reasoning, Math, Coding, STEM and Humanities/Social Science. We first greedily generate responses with up to 1024 tokens (default value for MT Bench). The responses to these prompts are evaluated by GPPT-4-0125Prreveee
between 1 and 10, and report the mean across all prompts. Prompts in Coding, Math and Reasoning categories are evaluated with reference correct answer, which were generated by the judge model and then manually verified. Higher MT Bench score indicates better instruction following ability.
give
AlpacaEval 2.0 Length Controlled We follow Dong al. (024); Meng et al. (2024); Wang et al.] (2024d) to use AlpacaEval 2.0 Length Controlled (Dubois et al. 2024). AlpacaEval 2.0 contains 805 first-turn instructions (relating to singular-reuuirement straightforward tasks such as recommendations, question answering, and open-ended writing). An answer to each prompt is greedily generated by the evaluated model as well as baseline model (GPT-4-1 t06--ubo),, which then sent EPT-4-1106-turte evaluator that outputs the probability of preferring the generations of the evaluated model. Finally, the authors introduced a length correction factor to mitigate the bias for the evaluator towards preferring longer generations.
Arena Hard We follow Dong et al. (2024); Meng et al. (202 Wang et al. (2024d) to use Arena Hard (Li et al.) 2024). Arena Hard contains 500 first-turn instructions obtained from challenging user queries on Chat Arena (Chiang et al. 2024). Prompts are classified using an I.I.M (Llama-33 70B-Instruct) to determine if they complex, specific, www.ooodappplicatoo-eeeeaaeeeeee domain knowledge, problem-solving creativity, technical accuracy. Prompts that meet at least 6 out of criteria are labelled as challenging. Therefore, huge proportion of prompts (>50%) are related to solving coding problems. Model responses are then compared with responses from GPT-4-0314 using 10-vvve judge calculate win-rate over GPT-4-0314.
require
REWARD CURVES
We plot the reward obtained by PPO and REINFORCE in Fig.
J FURTHER ANALYSIS FOR SCALED BRADLEY TERRY
To give clearer analysis for Scaled BT, demonstrate the losses for various BT variants for repre- sentative scenarios in Table 10 Compared to Regular BT, Scaled BT increases the loss proportionally when the human-annotated margin is larger. Relative Margin BT, Scaled BT provides higher loss
25

---

Published as a conference paper at ICLR 2025
y Mo po Maago Ma Mmmm y yo mmmpi
PPO REINFORCE
Training Reward
-1
20
60 80
100 120
Training Step
Figure 3: Reward curves of PPO and REINFORCE. Please note that REINFORCE samples four times as many responses as PPO per training step. For PPO, the reward curve produced by concatenating the reward curves for Round (ending at step 57) and Round 2. We select step 26 for PPO Round 1, step 40 for PPO Round (whose reward is shown at step 97 above) while for REINFORCE, we select step 95.
from incorrect predictions while keeping losses low on correct predictions. Specifically, Scaled BT keeps loss below on these correct predictions (re (z, ye) > ra (z, yr-) or Ar > 0) while in the case of (Ar=1, m=3), Margin BT gives relatively high loss (2. 1269) even when the prediction is correct Le. when the model predicted A is slightly better than B when humans annotated A to be much better than B. Such high loss in Margin BT can cause the model to potentially optimize away from the correct predictions it has made.
Correct Predictions
Incorrect
Predictions
Avg. Loss
Predicted Reward Gap (2r)
Ar=3
Ar=3
Ar=1
Ar=1
Ar=-1
Ar=-1
Ar=-3
Ar=-3
Human Annotated Margin (m)
m=1
m=3
m=1
m=3
m=1
m==
m=1
m==
Regular Bradlcy-Terry
0.0486
0.0486
0,3133
0,3133
1.3133
1.3133
3.0486
3.0486
1.1800
Margin Bradley-Terry
0.1269
0.6931
0.6931
2.1269
2.1269
4.0181
4.0181
6.0025
2,4757
Scaled Bradley-Terry
0.0486
0.1458
0.3133
0.9399
1.3133
3.9399
3.0486
9.1458
2.3619
Table 10: Loss for various Bradley-Terry loss functions in representative scenarios Ar refers to the gap between chosen and reject rewards predicted by the model (i.e. ro(I, Mo) rr 2r)) while m refers to the magnitude of preference strength between the chosen and rejected responses as labelled by humans. Correct predictions are made when (0 Uc) (2 yr) or Ar > 0. Conversely, incorrect predictions are made when re ye) < (z, yr) or Ar < O.
Compared to Margin BT, Scaled BT also gives much higher loss of around 50% (9. 1951 vs. 6.0025) when drastic mistakes are made (Ar=-3, m=3) - i.e. when the model predicted is much better than B when humans annotated B to be much better than A. It is important to note among our eight representative scenarios, Scaled BT has an average loss similar to Margin BT (2.3619 vs. 2.4767), heavier.
suggesting that Scaled BT's relative penalties Incorrect Predictions
side benefit of the Scaled BT loss weighing is that it is likely to be more robust to annotation errors compared to Margin Regular BT. This is because more human annotation errors made between slight preference compared to strong preferences, since it's relatively clear when A is much better than B or vice-versa. Therefore, since Scaled BT correspondingly gets higher losses from the "strong preference" samples, is less affected by annotation errors, which are more commonly found among slight preference samples.
K REWARD MODEL ABLATION STUDIES
We use the Scaled Bradley-Terry model initialized with Helpfulness Only SteerLM Re- gression as a baseline model to compare the effectiveness of various ablations, as this was our best-performing model without model extrapolation. We perform all ablations using similar training settings and hyperparameter search for fairness. Across all ablations in Table [T] no setup exceeds baseline model suggesting that the design choices made in training this model are optimal.
Overall
26

---

Published as a conference paper at ICLR 2025
RewardBench Chat-Hard
Hyperparams LR
Model Type
Mode!
Overall
Chat
Safety
Reasoning
Buseline (Sculed init. Helpfulness Only SteerL.M Regression)
93.7
98.0
85.7
le-6
Alt. Data Preprocessing
including samples with helpfuliccs spread > 2
92.7
97.8
84,2
94.2
94.6
3e-6
using all unnotations instead most similar
92.3
98.6
82.2
93.2
95.0
le-6
Remove overlapping prompts with RewardBench
93.3
97.5
85.3
94.3
96.1
le-6
HelpSteer2 converted into pairwise instead of HelpSteer2-Preferenn Helpfuluess Only SteerLM Regression (init, with Scaled BT)
92.9
97.2
84,2
94.6
95.6
2o-6
92.2
98.9
84,9
91.5
93.7
le-6
Table 11:
Ablation Studies for Reward Models. Higher is better for each category. All models are trained by us using 1aama-3.1-700-mmme as a base model.
Alternative Data Preprocessing We performed ablation studies on our data preprocessing to better understand the contributions of our data filtering process. We take baseline for Reward Model trained on the "smaller, higher-quality dataset" post extensive filtering which reaches 93.7 on RewardBench. Using the same training process the baseline but without filtering the 10% of samples with wider range (spread >2) results in degradation of I point overall (to 92.7 RewardBench).. Using the same training process as the above but using all annotations (up to five) for each task, instead of only the three most similar ones, results in further degradation of 0.4 on Rewardbench to 92.3. These results support our initial belief that that models trained on smaller, higher-quality dataset perform better than those trained on larger datasets with more noise, suggesting that our data pre-processing to remove noisy samples was suitable.
Remove overlapping prompts with RewardBench Initially, we did not apply prompt filter to ensure prompts in HelpSteer2- HelpStcer2-PPefeernne Preferenceee did not overlap with those in RewardBench. At the time
of submission (1 Oct 2024), were not aware of potential prompt overlap between HelpSteer2- Preference and RewardBench. Post-submission, we were informed by RewardBench maintainer that there were 42 prompts in SellsSteer--Prffreneee that overlap with RewardBench prompts (see details at [3]). In this setting, overlap means that there is at least one identical 7-gram between a HelpSteer2-Pres Preferrncce prompt and any RewardBench prompt. This definition is low-precision but high-recall definition of overlap (i.e. over-estimate of actual overlaps), because in many of these 42 cases, the 7-gram overlap is due to a similar template used between the prompts. For instance, both prompts can have common starting template such as "I want you to act as a novelist. but have rather different content after that.
Regardless, this is a negligible portion of both HelpSteer2- Prrfffrence (which has 0.42% overlap out of 10k total prompts) and RewardBench (which has 1.4% overlap out of 3k total prompts). We believe this small proportion of overlapping prompt is because PeepSteee2--Prefeeence prompts are sourced from ShareGPT, which might have overlapped with the prompt source of constituent datasets of RewardBench as both depend on people voluntarily sharing their ChatGPT conversations.
Out of an abundance of caution, we repeated our training approach for Scaled BT initialized with Helpfulness only SteerLM Regression, which scored 93.7 on RewardBench Overall and the resulting model achieves 93.3 on RewardBench, which is not very different from the original model. Therefore, we are confident that the impact of the extremely limited overlap in prompts is minimal. Please note that even when prompts are overlapping, the model responses are still generated by different models, hence the reward model cannot 'memorize' the reward of response.
HelpSteer2 converted into pairwise instead of SrelpSteer2-Prefeevve HelpSteer2-Preference is differentiated from HelpSteer2, we perform an additional RM experiment using HelpSteer2 converted into pair-wise format. When initialized with Helpfulness-on Regression model, training using Scaled BT on the original HelpSteer2 converted into a wiir-wise format cannot improve upon the original model (93.0 RewardBench, see Table [] and achieves 92.9 RewardBench. On the other hand, BT training with HelpSteer2-Prefeeenee can improve upon the Helpfulness-only SteerLM Regression model trained with HelpSteer2, reaching 93.7 on RewardBench. This shows that HelpSteer2 and HelpSteer2 P-rrefeeenec contain complementary label information that can be useful in two-stage RM training process.
To better understand how
SteerLM
Helpfulness Only SteerLM Regression (init. with Scaled BT) When we trained a helpfulness- only S Regression model initialized with Scaled Bradley Terry model, it has 92.2 Reward-
27

---

Published as a conference paper at ICLR 2025
Bench, which is not better than training Helpfulness only SteerLM Regression model alone with 93.0 RewardBench. We believe this because is not useful to train Helpfulness- -nll SteerLM Regression model initialized on Bradley-Tern model. The Bradley-Temy Model does not generate rewards within a narrow range and instead can go between -35 and 5 (see Fig. 2) Hence, initializing the regression model from Bradley-Terry model can lead to extremely large losses initially (as initial predictions are between -35 and while ground truth helpfulness ranges from 0 to 4), which might be better than randomly initializing the regression head.
L ALIGNED MODEL ABLATION STUDIES
Aligned Metrics
Hyperparams
Model Type
Date/Reword Model
MT Bench (GPT-4-Tubo)
Mcan Respouse Length (Chars.)
AlpacaEval 2.0
Arena Hard (95% CT)
LR
KL
DPO
HelpStter2- Preference
8.66
1502.2
40.4 (1.66)
52.8 (-2.7, 2.7)
2--7
0.01
HelpSteer2 (convened pairwise)
8.61
1609.2
(1.37)
47.7 (-2.9, 2.6)
2e-7
0.01
PPO
Scaled BT ExPO
8.74
1842.8
43.8 (1.76)
2.5)
1o-6/l-77
0,005/0.01
Llama3 70B SteerLM
8.71
1506.2
38.2 (1.38)
49.3 (-2.1, 2.4)
1e-6
0.005
Starting Model
Llama-3. 3.1-0-3.---0B-aaar Instruct
8.22
1728.6
38.1 (0.90)
55.7 (:2.9, 2.7)
Table 12: Ablation Studies for Aligned Models. Higher is better for each metric, except Length.
To give better sense of how other datasets compare to HelpSteer2- Preffrrece terms of RLHF (Section [). we trained DPO model using HelpSteer2 dataset (converted into a pairwise setting based on difference in helpfulness), while keeping the training setup and hyperparameter search constant. In addition, we trained a PPO model using HelpSteer2) instead of the Scale BT + ExPO reward model (trained with
(trained with while keeping the training setup and hyperparameter search constant. This model did not benefit from two-stage PPO training, the one with Scaled BT ExPO did.
Across both DPO and PPO, SeliSteeer2-Prfeeeeee can be used to align models better than HelpSteer2, in terms of MT Bench, AlpacaEval and Arena Hard. We believe this supports the advantage of collecting a dataset specifically purposed for a pair-wise training setting, over cetro-fitting HelpSteer2, which has been collected for different purpose (single response rating prediction).
M FURTHER REINFORCE MODEL METRICS
Model Type
Model
General Knowledge
Math
Coding
MMLU (5-Shot, non-CoT)
GSMSK (0-shor, non-CoT) HumanEval (0-shot, non-CoT)
Online RLHF
REINFORCE
82.4
90.9
83.5
Starting Model
Llama-3. 1..m-...--70..... Instruct
80.5
Table 13: Auxiliary Metries for REINFORCE and Llama-3. 1.i-aa--..770---1 -nnttuttttttt
Overall We notice that performance on these benchmarks (MMLU, GSMSK and HumanEval) in Table 13 does not change much after REINFORCE training compared to Llama 3.1 70B Instruct, which we use as a starting policy. This suggests that REINFORCE does not cause catastrophic forgetting of capability gained from prior training.
28


Reviews:
[{"review_id": "VcmGghbB9s", "reviewer": "Reviewer_eXue", "summary": "This paper explores whether a Bradley-Terry or regression approach is more effective for reward modeling and proposes a novel method that combines both styles. The conclusion is intriguing: the format in which data is collected and the specific model training approach do not significantly impact the results.", "questions": "Has a prompt filter been applied to ensure that prompts from the rewardbench are not included in the training set?\n\nAdditionally, how does SteerLM regression, initialized with the Bradley-Terry model, perform?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper provides an excellent dataset that will be highly valuable to the community. It addresses an important question: whether the Bradley-Terry style or regression style is more effective for reward modeling. Additionally, the reward model (RM) proposed here is capable of effectively training a LLaMA-70B model.", "weaknesses": "I'm glad to see that the BT training method with an added margin (eq.2) is effective. However, this finding somewhat conflicts with the results in the Llama 3 paper, where the authors mention: \"The training objective is the same as Llama 2, except that we removed the margin term in the loss, as we observed diminishing improvements after data scaling.\" What could be the reason behind this discrepancy?\n\nI understand that this work provides an excellent dataset and rigorous experimental validation. However, I would like to see more analysis, such as:\n\n1. A comparison of results on out-of-distribution prompts.\n\n2. A consideration of annotation costs: since preference annotations are likely cheaper and easier to scale than Likert scores, a fair comparison might require factoring in annotation costs rather than simply matching dataset sizes.\n\n3. An analysis of robustness to annotation errors. For instance, assume that preference annotations have a 5% chance of being reversed, or that Likert scores have a similar error rate (where a 9 might be incorrectly marked as a 1, rather than just slightly off, like 8). Which reward modeling approach is more robust to such errors?\n\nThese aspects would provide valuable insights into the practical implications of each approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N7mWjM6ZS6", "reviewer": "Reviewer_yVaS", "summary": "This paper presents HelpSteer2, a dataset designed to facilitate research in RM for aligning language models with human preferences. This work addresses the gap of fairly comparison of BT and regression style of RM by releasing preference annotations compatible with the BT style alongside existing ratings formatted for Regression-style training. The authors then conduct a head-to-head comparison of the BT and Regression models when matched for data quality, providing valuable insights into the effectiveness of each approach. They also propose a hybrid model that combines elements of both BT and Regression approaches, achieving state-of-the-art performance on the RewardBench benchmark. Additionally, the authors open-source the HelpSteer2 dataset and trained reward model to promote further research in RM alignment tasks.", "questions": "- Could the authors provide statistical significance testing to validate the reported performance improvements of the hybrid model? This would strengthen the robustness of the conclusions.\n- Have the authors considered evaluating the align models on standard language modeling tasks to assess the alignment tax? \n- Based on the appendix, it appears that the RL training involved approximately 10k examples, and the model was trained for fewer than 100 iterations. Could the authors clarify the stopping criteria for the training process? Additionally, could you specify the KL divergence value of the choosen checkpoints from the reference policy? This information would provide better insight into the training stability and convergence behavior of the reward model.\n- Are the conclusions valid for cases where, e.g, 5 responses ranking are used for RM training?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The authors have created and open-sourced the HelpSteer2 dataset, which includes high-quality preference annotations and justifications specifically designed for reward model (RM) training.\n- The paper conducts a detailed, head-to-head comparison between Bradley-Terry (BT) and Regression-style reward models, providing insights into the relative strengths and weaknesses of each approach.", "weaknesses": "- While the hybrid approach is innovative, it seems to be an incremental improvement rather than a fundamentally new modeling technique. \n- Lack of statistical significance testing to validate the effectiveness of proposed hybrid modeling method.\n- The paper does not evaluate the aligned models on standard language modeling tasks, which would help in quantifying the alignment tax.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7gaEWyKipz", "reviewer": "Reviewer_KjD8", "summary": "This work addresses two major paradigms: Bradley-Terry (BT) and Regression style models, examining which is better when matched for data. The main challenge in previous studies was the lack of comparable datasets collected using both methodologies. To resolve this, the authors introduce the HelpSteer2-Preference dataset, which complements existing ratings with preference annotations, including human-written justifications. This dual-annotation allows a head-to-head comparison between BT and Regression models under controlled data conditions.", "questions": "1. How were the annotators trained to provide consistent ratings and justifications, and what measures were in place to minimize subjective bias?\n2. How well does the dataset generalize to real-world applications beyond reward modeling and instruction-following?\n3. What is the impact of filtering out samples with high annotation variability on the overall representativeness and diversity of the dataset?\n4. Would models trained on a smaller, higher-quality dataset perform better than those trained on larger datasets with more noise?\n5. How might the dataset be expanded or adapted to cover other dimensions of reward modeling, such as domain-specific use cases or more complex types of preferences?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "First, it complements existing ratings with detailed preference annotations, allowing for more nuanced training of reward models. By collecting preference data alongside Likert-scale ratings, the dataset enables direct comparisons between models trained under different paradigms (e.g., Bradley-Terry vs. Regression), which was previously difficult due to incompatible data formats in existing datasets. This dual approach provides a richer training ground for testing and developing models that need to evaluate comparative quality.\n\nSecond, the dataset includes human-written justifications for preferences, adding a layer of interpretability. This is particularly valuable for training models that can both assess and explain their choices, supporting applications where transparency is important, such as in human-in-the-loop systems. The justifications make the dataset unique by facilitating the development of models that are not just performance-driven but also capable of detailed decision-making explanations.\n\nThird, the dataset maintains high-quality standards, demonstrated by its filtering processes and the steps taken to ensure strong inter-annotator agreement. By curating data that passes rigorous quality checks, the authors improve the signal-to-noise ratio, making it more effective for training robust models. This attention to quality addresses common issues in training data, where noisy or inconsistent annotations can limit model performance.\n\nOverall, the HelpSteer2-Preference dataset's strengths lie in its dual-format data collection, the inclusion of human justifications for enhanced interpretability, and its commitment to high-quality, reliable annotations. This makes it a valuable resource for advancing research in reward modeling and instruction-following alignment in language models.", "weaknesses": "While the HelpSteer2-Preference dataset brings several innovations and benefits, it also has certain weaknesses that could impact its applicability and use in research and development:\n\nFirst, the dataset's format, which includes both ratings and preferences, may require more complex preprocessing and handling by researchers. The dual data structure means that models need to be capable of leveraging both individual ratings and comparative preferences, which could increase the implementation complexity for teams that are not equipped with extensive data processing expertise.\n\nSecond, despite the inclusion of human-written justifications enhancing interpretability, these justifications may introduce variability due to subjective writing styles or annotator biases. This subjectivity can make it challenging to maintain consistency across annotations and could potentially affect the model's ability to generalize when trained with such data. Additionally, training on justifications may require more advanced model architectures that can process and utilize free-form text, which may not be available or practical for all research teams.\n\nThird, while the dataset emphasizes high-quality annotations with steps taken to filter out low-confidence or inconsistent data, this strict filtering reduces the overall volume of usable data. The exclusion of samples with high variability among annotators may result in a smaller training set, potentially limiting model performance, especially for models that benefit from a large and diverse dataset. This trade-off between quality and quantity could be a bottleneck for researchers who need extensive data for training large-scale models.\n\nLastly, the HelpSteer2-Preference dataset is tailored to specific types of model training and evaluations, such as Bradley-Terry and Regression-based reward models. This specialization may limit its generalizability to other types of machine learning tasks or applications that do not align with these paradigms. Researchers working on models with different objectives or those requiring more diverse data formats might find the dataset less adaptable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pFFzPgpcs6", "reviewer": "Reviewer_AvcT", "summary": "This paper complements the existing HelpSteer2 dataset with preference annotations, showing that pairwise annotation can further distinguish responses with similar likert scores. This dataset, together with its counterpart HelpSteer2, enables a head-to-head comparison between Bradley-Terry and Regression style reward models, by training models with the same prompts and responses in HelpSteer2 and only differing the feedback annotation. Using insights from this comparison, they propose a novel approach combining both methods, achieving superior reqward modeling performances. They also conduct experiments with RL algorithms to demonstrate the effectiveness of the reward models.", "questions": "See weaknesses.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces a carefully annotated preference dataset that could be useful. The authors also conduct some analysis on the difference of pointwise and pairwise annotation.\n\n2. This paper compares Bradley-Terry and Regression style reward modeling paradigms, which motivates them to design a new method to make the best of both worlds.", "weaknesses": "1. The motivation is somehow overclaimed. In line 013-016, the authors claim that there lacks datasets to enable the comparison between Bradley-Terry and Regression style reward modeling. However, all preference datasets with point-wise scores can be converted into pair-wise setups, which should have been able to allow the comparison between the two paradigms. But I agree with authors that \"Regression-style data collection (where responses are rated individually) gives\nannotators a different set of expectations for comparative ranking of responses\" (line 055-056), so it'd be better to have a pair of dataset directly annotated in point-wise and pair-wise schema. Admittedly, the comparison setup is improved now but it was not totally undoable.\n\n2. The authors conducted a lot of experiments, but it's hard for readers to parse useful information from the experiments with almost no highlights in writings. The abstract mainly focuses on what they did, with no mention on what they have found in their experiments. In the experiment sections, readers would benefit a lot from bolded high-level excerpts to quickly grasp the key findings. \n\n3. The results are good to know, e.g. the comparison between Bradley-Terry and Regression style reward modeling, but personally I don't find it surprising or informative enough to serve as the main claim of a conference paper. It would better to investigate deeper what properties of an algorithm make a good reward model. Also, the experiments are mainly conducted on general chats with subjective metrics (LLM-as-a-judge), thus the conclusions may not be able to generalize to other taks like reasoning, which is known to be very different from chatting. Regarding other experiments, such as section 5, the authors conduct RL with their reward models and datasets mainly for the sake of demonstrating their usefulness. In this case, it seems baselines are needed so that readers can be better aware of the necessity of another preference dataset, given the existence of many good alternatives, and distinguish this dataset with existing ones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rsGKlgGoXL", "reviewer": "Reviewer_wyR3", "summary": "This paper introduces Helpsteer2-preference, a preference dataset that provides additional pairwise preference annotations alongside the existing Likert-5 style rating of the Helpsteer-2 dataset. Using this dataset, the authors conduct head-to-head comparisons between two major types of reward models. Then, they propose a method to combine two types of reward models to achieve better performances on RewardBench and preference learning.", "questions": "- How does the reward model perform if combined vice-versa, using SteerLM Regression model with the initialization of Bradley-Terry model?\n- How many hours of human labors do you use to collect the human annotations?\n- How does the proposed reward model combination method compared with model ensembling?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The writing is clear and easy to follow.\n- The data collection process is well-designed, with detailed and comprehensive analysis provided.  The proposed dataset provides a direct and fair comparison between Bradley-Terry and Regression-based reward models. \n- The proposed method of  combining of different reward models is effective on RewardBench, and improves LLaMA-3.1 through preference alignment.", "weaknesses": "- One major concern is the paper/s dependence on HelpSteer2. While the authors states that \"converting regression-style data to preference pairs cannot fully capture nuances in preferences\", there are no experimental results to support this claim. It would strenghthen the paper to include experimental comparisons with models trained on the preference dataset converted from Helpsteer2. This would help clarify whether the improvement is due to the new preference data itself or the different formulation of reward models.\n\n- Lack of comparison with other reward models or preference dataset in preference alignment (RLHF, DPO). \n  - For DPO, what if you apply the converted preference dataset from HelpSteer2? \n  - For PPO, can you compare it with other reward models of the same base model(e.g. Llama3-70B-SteerLM-RM, TextEval-Llama3.1-70B)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)