HelpSteer2-Preference: Complementing Ratings with Preferences

## Abstract

Abstract Reward models are critical for aligning models to follow instructions, and are typically trained following one of two popular paradigms: Bradley-Terry style or Regression style. However, there is a lack of evidence that either approach is better than the other, when adequately matched for data. This is primarily because these approaches require data collected in different (but incompatible) formats, meaning that adequately matched data is not available in existing public datasets. To tackle this problem, we release preference annotations (designed for Bradley-Terry training) to complement existing ratings (designed for Regression style training) in the HelpSteer2 dataset. To improve data interpretability, preference annotations are accompanied with human-written justifications. Using this data, we conduct the first head-to-head comparison of Bradley-Terry and Regression models when adequately matched for data. Based on insights derived from such a comparison, we propose a novel approach to combine Bradley-Terry and Regression reward modeling. A Llama-3.1-70B-Instruct model tuned with this approach scores 94.1 on RewardBench, emerging top of more than 140 reward models as of 1 Oct 2024. We also demonstrate the effectiveness of this reward model at aligning models to follow instructions in RLHF.
We open-source this dataset (CC-BY-4.0 license) at https://huggingface.co/datasets/nvidia/HelpSteer2 and openly release the trained Reward Model at https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward .

## 1 Introduction

First featured in the Reinforcement Learning from Human Feedback (RLHF) pipeline for aligning language models to follow instructions (Bai et al., ; Ouyang et al., ), Reward Models are still prominently featured as a critical part for aligning frontier open-recipe models (Dubey et al., ; Nvidia et al., ). The role of Reward Models lies in assigning high scores to responses that follow instructions in a helpful and safe manner while assigning low scores to those that do not. This in turn guides language models to generate more responses that give high scores, which make them more helpful and safer (Dong et al., ; Lambert et al., ).

While training a reward model that can accurately separate good responses from bad is a consensus goal, there is less agreement on the best path to get there. On one side are the traditional Bradley-Terry Style Reward Models first introduced by Bai et al. () and Ouyang et al. () which seek to maximize the gap in reward between chosen and rejected responses to the same prompt. On the other side are Regression style Reward Models introduced by Wang et al. () and Wang et al. () and have lately been used to train some of the top models on RewardBench (AllenAI, ) such as ArmoRM (Wang et al., ) and Nemotron-4-340B-Reward (Wang et al., ). Regression Reward Models train the model to predict the score (often Likert-5) for a response to a particular prompt. For many researchers and practitioners, deciding which style of Reward Models to adopt is challenging due to the absence of empirical studies that directly compare them when appropriately matched for data, which means:

1. Identical set of prompts and responses This mitigates the confounding factor of prompts and responses influencing reward model performance, which plays a role outside the data collection approach.

2. Collected for Purpose Data used to train each type of Reward Model should be collected in the format they will be used. While there are heuristics to convert Regression-style ordinal annotations over to preference annotations (using the difference between scores of individual responses), our early experiments found them to be comparatively lackluster for training Bradley-Terry models. We hypothesize that Regression-style data collection (where responses are rated individually) gives annotators a different set of expectations for comparative ranking of responses. For instance, in the Likert-5 scale used by HelpSteer2 (Wang et al., ), both responses can be given a helpfulness 3 but one might still be preferable to the other. Post-hoc heuristics used to convert such Regression-style data to preference rankings are unable to account for such nuances in preferences.

3. High Quality Following the adage ‘Garbage in Garbage out’, the role that the data format (regression-style vs. preferences) plays may not reliably show when the quality of data collection is the bottleneck, as the signal-to-noise ratio will be too low for the model to learn anything useful. Shen et al. () found that the Llama-2-7B-Chat model performs essentially at a random level (i.e. <55% for a binary choice task) on RewardBench Chat-Hard category (Lambert et al., ) when trained on Ultrafeedback (Cui et al., ) or HH-RLHF (Bai et al., ), which is attributed to noise in the ground truth labels. Similarly, Wang et al. () found that Llama-3-70B models trained on HH-RLHF (Bai et al., ), Open Assistant (Köpf et al., ) or HelpSteer (Wang et al., ) do not surpass 60% on the same evaluation. In contrast, Llama-3-70B trained on HelpSteer2 (Wang et al., ) performs above 80%. Without meeting a high bar on data quality, it may not be possible to discern the advantages of a particular data annotation methodology over another.

To the best of our knowledge, no one has thus far publicly released data that is adequately matched for both approaches (see detailed Related Works in Appendix A). In this work, we release preference annotations that were collected alongside the Likert-5 ratings from HelpSteer2 (Wang et al., ), a high quality dataset used to train some of top models on RewardBench (e.g. Nemotron-4-340B-Reward).(^1^11This data has not been released previously because we did not have sufficient resources to conduct experiments to demonstrate the value of preference annotations.)
We show that Bradley-Terry Models can be effectively trained with such preference annotations, and also investigate leveraging preference justifications where annotators indicate why they preferred one response over another.

Our key contributions are:

1. We open-source (CC-BY-4.0) a high-quality preference modeling dataset containing preference directions, strengths, and justifications. To the best of our knowledge, this is the first open-source release of a general-domain preference dataset containing human-written preference justifications.

2. Using this data, we perform a head-to-head comparison of Bradley-Terry style and Regression style Reward Models, alongside reward models that can make use of preference justifications.

3. From insights derived from the above comparison, we propose a novel approach to combine Bradley-Terry and Regression Reward Models, which can be used to train a reward model that scores 94.1 on RewardBench, which is the best performing model as of 1 Oct 2024. We also show how this reward model can be useful for RLHF.

## 2 Dataset

##### Data Collection

For each task, annotators are provided a prompt and two responses. They first annotate each response on a Likert-5 scale along several dimensions (helpfulness, correctness and coherence, complexity and verbosity), as detailed in Wang et al. (). Then, they choose between 7 preference options, each associated with a preference score as well as a justification for their preference:

- -3.
Response 1 is much better than Response 2
- -2.
Response 1 is better than Response 2
- -1.
Response 1 is slightly better than Response 2
- 1.
Response 2 is slightly better than Response 1
- 2.
Response 2 is better than Response 1
- 3.
Response 2 is much better than Response 1
- -100.
Neither response is valid

The last option (“Neither response is valid”) is to be used when both responses are so bad that trying to identify a winner is pointless. Inspired by Bai et al. () and Touvron et al. (), we force the annotator to make a preference choice between the two responses (i.e. no option for Both responses are equally good), except for “Neither response is valid”. Our motivation was to reduce sitting-on-the-fence behavior and encourage annotators to look closer at differences (even if they appear to be minor), in order to much more robust preference information.
The preference ranking guidelines and examples provided to annotators are in the Appendix B.

The vendor (Scale AI) was asked to distribute each task to 3-5 annotators to independently label preference among two responses for every prompt. For a small proportion of samples ($<$10%), there are fewer than three useful annotations in the resulting dataset. This is due to annotators skipping some tasks as they were unable to rate them effectively or indicating that both responses were invalid (i.e. rated as -100, which were excluded).

##### Data Pre-processing

In line with HelpSteer2, we identify the three most similar preference annotations per task (to avoid interference by outliers), take their mean, and round it to the closest integer to give the overall preference. Furthermore, we filter out 10% of tasks, for which the spread of annotations of the three most similar annotations was more than two. For instance, a task with individual preferences of [-3, -1, 2, 3] will have the most similar annotations of [-1, 2, 3] with a spread of 4, and thus excluded. This is done to avoid training on tasks for which the ground-truth preference cannot be confidently estimated among human annotators. 22% of the samples have an overall preference of 0 in situations where annotators disagree (e.g. [-1, -1, 1]) and these samples are also excluded because a near-zero average indicates low-confidence preferences, which may introduce noise during Reward Model training. Overall, we have 7,118 preference pairs with 6,766 pairs in the training set and 352 pairs in the validation set.

To compare the inter-rater reliability of our collected data (compared to HelpSteer2), we follow Wang et al. (), to use quadratic-weighted Cohen’s $\kappa$ (Artstein & Poesio, ) as a measure of inter-rater agreement.
The quadratic weighted version of Cohen’s $\kappa$ (Scikit-Learn, ) can also penalize larger disagreements (e.g. -3 and +3) much more heavily compared to smaller disagreements (e.g. -1 and +1).
The agreement of the raw preferences is 0.489 (moderate), suggesting that having annotators agree on the direction and strength of preferences is challenging. Our pre-processing steps
(i.e. using the three most similar annotations per task and removing samples with a preference spread of more than two) increase Cohen’s $\kappa$ to 0.843 (strong), suggesting that weeding out outlier annotations is an effective way to improve agreement. Finally, excluding samples with an overall preference of 0 further increases Cohen’s $\kappa$ to 0.878 (strong). The observed final agreement is stronger than HelpSteer2 helpfulness (with 0.791), suggesting good reliability of our collected data. We detail how we pre-process preference justifications in Appendix C.

##### Data Analysis

Figure: Figure 1: Distribution of preferences between responses in HelpSteer2Preference against the difference in helpfulness scores between them from HelpSteer2. For clarity, A refers to Response 1 while B refers to Response 2. $>>>$ means much better, $>>$ means better, $>$ means slightly better and $=$ means as good as.
Refer to caption: fig/helpfulness_diff_against_preference.png

We analyze the dataset including samples for which the overall preference is zero, as this can provide insights below, even though such samples are not used for training subsequently.
As shown in Fig. 1, the distribution of preferences is concentrated at the center (A=B or 0) and reduces gradually away from the center ($\mu$ = 0.0649, $\sigma$ = 1.72), with a slight bias to preferring Response 2 over Response 1. This means that Response 2 is preferred in 41.6% of tasks while Response 1 is preferred in only 36.5%. This bias is especially high for the "slightly better" setting where Response 2 is slightly preferred 18.5% vs. Response 1 14.8%. Such a bias is similar in extent to the difference between the helpfulness score of Response 1 and Response 2 from the original HelpSteer2 dataset (Wang et al., ): 39.3% of tasks gave Response 2 a higher helpfulness score while only 33.9% gave Response 1 a higher helpfulness score.

There are some possible reasons for such a position bias. First, the sampling of responses between Response 1 and 2 might not have been even, with Response 2 having been dis-proportionally generated by stronger models. Second, annotators could also have a positional bias, in which the order of responses shown to them can influence their judgment. In this case, the response they see later could be making a stronger impression on them, leading annotators to dis-proportionally prefer them due to a recency bias (Phillips-Wren et al., ). While it is important to note the existence of this position bias, it is relatively small in extent (5.1% difference in preference), which minimally affects their usefulness compared to LLM-as-a-judge settings (with GPT-4 and Claude) where the position bias could be between 25% and 75% (Zheng et al., ).

Fig. 1 also shows that the distribution in preference between the two responses is highly correlated with the difference in the responses’ helpfulness score from HelpSteer2 (Pearson’s R = 0.9024). When two responses have the same helpfulness score, they most likely are as good as each other but in slightly less than half the cases (45.5%), annotators still show a consistent preference for one response. This supports our initial hypothesis that preference ranking can produce a more fine-grained separation between two responses even if they share the same Likert-5 helpfulness score. Slight preferences are most commonly (58.0%) associated with a difference in helpfulness scores of one while strong preference preferences are frequently (93.4%) associated with a helpfulness difference of 3 or 4. Interestingly, there are also substantial samples (7.4%) that show no preference despite having an absolute difference of one or more. This is likely due to annotator disagreements, which are averaged to obtain the final overall preference. Given the constraints on space, we defer an extended analysis of preference justifications (both structural and semantic) to Appendix D.

## 3 Reward Model

### 3.1 Evaluation

Following Dong et al. (); Wang et al. (); Yuan et al. (), we perform evaluation using RewardBench (Lambert et al., ), a trusted reward modeling benchmark with over 140 models on the public leaderboard (AllenAI, ). RewardBench comprises 2985 diverse tasks across 4 categories - Chat, Chat-Hard, Safety, and Reasoning - and 23 sub-categories), which minimizes the likelihood of over-fitting to particular tasks. Each task consists of a prompt, a chosen response, and a rejected response.
The Chat category involves comparing a bad and a good model response in general-domain conversation settings while Chat-Hard does the same for a good model response compared to a great model response. The Safety category measures the preference between a refusal response and a compliance response to an unsafe user request. The Reasoning category assesses the preference between correct and incorrect responses related to math and coding prompts. The accuracy for each category is calculated based on the proportion of tasks it gets correct (i.e. prefer chosen over rejected), except for the Reasoning category, which up-weighs math samples to ensure math and coding prompts contribute equally. Overall accuracy is calculated using the mean of the four category scores.

### 3.2 Training

##### SteerLM Regression

Following Wang et al. (), we train SteerLM Regression Reward Models consisting of a base model and a linear layer projecting the final layer dense representation of the end-of-response token into five scalars, one for each HelpSteer2 attribute. Such models are optimized using a MSE loss function, which seeks to minimize the squared error between the predicted attribute value and the ground truth attribute value. In addition, we train a separate model only on the Helpfulness attribute.

##### Bradley-Terry

Following Ouyang et al. (); Bai et al. (), we train Bradley-Terry style Reward Models, consisting of a base model and a linear layer projecting the final layer dense representation of the end-of-response token into a scalar reward. Models are trained to maximize the directional distance between the reward for the chosen response ($y_{c}$) and the rejected response ($y_{r}$), as in Eq. 1, thereafter referred to as Regular BT.

$$ $\mathcal{L_{BT}}=-\log\left(\sigma(r_{\theta}(x,y_{c})-r_{\theta}(x,y_{r}))\right)$ (1) $$

Given that HelpSteer2-Preference contains not only the direction of preference between two responses but the magnitude ($m$) of this preference (1 - slightly better, 2 - better, 3 - much better), we also experiment with a Bradley-Terry with Margin loss introduced by (Touvron et al., ) in Eq. 2, thereafter referred to as Margin BT.

$$ $\mathcal{L_{MBT}}=-\log\left(\sigma(r_{\theta}(x,y_{c})-r_{\theta}(x,y_{r})-m)\right)$ (2) $$

We also introduce a new loss function named Scaled Bradley-Terry in Eq. 3, hereafter referred to as Scaled BT. Similar to Margin BT, its motivation lies in utilizing the preference magnitude information. However, we use the margin term outside of the log-sigmoid function rather than inside. From the perspective of data utilization, this can be viewed as a repeated sampling of response pairs for which the preference magnitude is higher. From the perspective of model training, this can be seen as requiring models to learn more (i.e. larger update) from responses that we know to be greatly different from each other. Unlike Margin BT, Scaled BT does not assume that the distance between the chosen and rejected rewards needs to be as least as large as the margin term.

$$ $\mathcal{L_{SBT}}=-m\log\left(\sigma(r_{\theta}(x,y_{c})-r_{\theta}(x,y_{r}))\right)$ (3) $$

Finally, we also train BT models initialized on the Helpfulness-Only SteerLM Regression Model. The Regression model is trained to predict helpfulness between 0 and 4, which can potentially initialize the model better than the base model, which otherwise has high loss at the start of training.

##### Pairwise Justifier

To explore training reward models using preference justifications rather than preference scores, we train Pairwise Justifier reward models similar to how proprietary models such as GPT-4-Turbo are used for LLM-as-a-Judge in AlpacaEval (Taori et al., ) and Arena-Hard (Li et al., ). In these settings, the LLM is prompted to generate a detailed comparison of the two responses before finally generating a statement such as "Response 1 is better than Response 2". HelpSteer2-Preference provides a unique opportunity for examining whether preference justifications can be used to train more accurate reward models (compared to using preference score) when training data is kept the same. To train such models, we seek to generate the preference justification conditioned on {prompt} @Response 1:\n{response_1}\n@Response 2:\n{response_2}\nBetween @Response 1 and @Response 2, which is better?. This model is then optimized using a Cross-Entropy loss as typically used for supervised fine-tuning of language models. In this setting, we format the preference justification by concatenating preference elaboration followed by preference statement, which is always in the format @Response 1/2 ... better. This allows automatic extraction of the better response for evaluation. We also experiment with i. including up to three preference justifications (i.e. All Justifications set) instead of one per task and ii. augmenting the dataset by flipping Response 1 with Response 2 and correspondingly updating the preference label (which also minimizes position bias) and iii. ablations to understand the role of each component.

##### Training Details

In our experiments, we use Llama-3.1-70B-Instruct (Dubey et al., ) as the base model. Our initial exploration of training reward models showed that Llama-3.1-70B-Instruct performs better as an initial model than Llama-3.1-70B as well as Nemotron 4 340B (Nvidia et al., ) and Llama-3-70B (Meta AI, ), as used by Wang et al. (). Hyper-parameters for training and the search range for them are in Appendix E.

## 4 Reward Model Results

Table: Table 1: Performance of Models on RewardBench. Higher is better for each category. All models are trained by us using Llama-3.1-70B-Instruct as a base model except External Baselines, the scores for which are taken from RewardBench leaderboard (AllenAI, )

##### SteerLM Regression

As shown in Table 1, training a SteerLM Regression only on the Helpfulness attribute leads to slightly better performance on RewardBench overall (93.0 vs 92.4), relative to training on all five HelpSteer attributes as proposed by Wang et al. (). While training on all five HelpSteer attributes can provide more insights to other dimensions of the response (correctness, coherence, complexity, verbosity), training with only the helpfulness attribute also makes the setup easier for training and inference. Specifically, there is no longer a concern that the five objectives/dimensions might conflict and the reward model produces a singular scalar reward without needing to derive one using a weighted sum of the five attribute values.

##### Bradley-Terry (from scratch)

Scaled BT performs much better than either Regular BT or Margin BT on RewardBench Overall at 92.7 vs. 91.5. This is likely because Scaled BT can most effectively use the preference magnitude information to guide model training.

##### SteerLM Regression vs. Bradley-Terry

To answer our initial question about whether SteerLM Regression or Bradley-Terry is better, we find that the optimal formulation of each variant (Helpfulness-Only and Scaled BT) performs just about as well as each other on RewardBench Overall (92.7 vs 93.0). This suggests that the format that the data is collected in and the model training approach does not matter too much. Instead, the most important consideration is that the modelling details can fully account for the information captured in the annotation. In the case of Bradley-Terry models, the magnitude of preference strength should be adequately modelled.

##### Complementing SteerLM Regression with Bradley-Terry

While SteerLM Regression and Bradley-Terry models are separately comparable, we show that they have a synergistic effect and result in a stronger reward model when combined. Specifically, when initialized with the Helpfulness-only Regression model, a Scaled Bradley-Terry can reach overall RewardBench of 93.7. This is likely a result of the HelpSteer2 and HelpSteer2-Preference datasets containing complementary information, as first indicated by Sec. 2.
Conceptually, this synergy is similar to the two-stage approach in performing preference tuning (i.e. Proximal Policy Optimization or Direct Preference Optimization) after doing supervised-finetuning.
In addition, we found ExPO (Zheng et al., ) to be a simple and effective way of extrapolating the delta weights to further improve the model. Specifically, we use the Helpfulness-Only SteerLM Regression model as the weak model and the Scaled BT model (initialized with the Helpfulness-Only SteerLM Regression model) as the strong model. We then did an extrapolation factor search between 1.1 and 2.0 at intervals of 0.1. Upon finding 1.6 to be optimal, we searched between 1.51 and 1.69 at intervals of 0.01. The final extrapolation factor was 1.52. Neither the Regular BT model nor the Margin BT model improved upon the Helpfulness-only Regression Model that they were initialized with and therefore we did not try ExPO on them.

##### Pairwise Justifier

Compared to SteerLM Regression and Bradley-Terry models that can score each sample independently, Pairwise Justifier models which involve choosing a better response between two candidates generally perform worse, with an overall RewardBench score of 90.0 or lower. We suspect that this is because this task formulation is much harder (implicitly involving scoring Response 1, then scoring Response 2 and finally determining which score is higher). This is also supported by the observation that strong external baseline models using this Pairwise Justifier approach (e.g. gpt-4o-2024-08-06 and Meta-Llama-3.1-70B-Instruct) score 86.7 and 84.0, which are similar to our Pairwise Justifier models. On the other hand, SteerLM Regression and Bradley-Terry models both decompose this problem into much simpler objectives that can be directly optimized towards.

As seen in Table 1, training a model on more than one preference justification per task is helpful (overall score increases from 86.1 to 88.1%), presumably because it provide a greater diversity of reasoning paths leading to the final preference choice. This increase is mostly contributed by gains on Chat-Hard (71.7 to 82.2%) while other categories do not change much. This is likely because our training data collection objective is most aligned with the Chat-Hard category (i.e. separating great general domain responses from good ones), meaning that extrapolating such reasoning to specialized-domains such as safety and reasoning (math/code) can be challenging.
Data Augmentation through flipping labels is also helpful (1.3% increase in Overall), likely as it reduce the effects of position bias, which is present in our data, albeit to a small extent (5.1% difference in preference).
Interestingly, training on just the Preference label (Response 1 or 2) does better than train on either the Preference Statement only (w/o Elaboration) or the Full Preference Justification. This is likely because the model can implicitly learn why a response might be better than the other in a way that’s more effective than grounding the choice based on human-written explanations.
Nonetheless, it is important to bear in mind that the current results are based on a small general-domain dataset comprising only 6.6k tasks, written in a free-form manner. Having a dataset which is different in domain, size or guidance to annotators (e.g. more structured) can lead to different conclusions. We leave further explorations to future work.

Comparison with External Baselines Relative to the top performing external baseline (Skywork-Reward-Gemma-2-27B), our best performing model (Scaled BT + ExPO) is slightly better in terms of overall RewardBench (94.1 vs. 93.8). However, looking more closely at the individual categories, our best model does better in all categories (Chat, Safety and Reasoning) except Chat-Hard. On Chat-Hard, it trails substantially behind Skywork-Reward-Gemma-2-27B (85.7 vs. 91.4). To understand possible reasons for this, we looked more closely at the constituent sources of data for the Chat-Hard category. We discovered that the Chat-Hard category is composed of data from two sources - LLMBar (Zeng et al., ) and MT-Bench (Zheng et al., ). LLMBar contains five different subsets that are used in Chat-Hard, three of which are based on human annotations as ground-truth labels, while the other two (Adversarial-GPTInst and Adversarial-GPTOut) use GPT-4 annotations as ground-truth labels. Similarly, MT-Bench also involves using GPT-4 judgements as the ground-truth.

Table: Table 2: Performance of Models on RewardBench Chat-Hard Category. Higher is better for each subset. Scaled BT + ExPO uses Llama-3.1-70B-Instruct as a base model and initialized using a Helpfulness-Only SteerLM Regression training while numbers for Skywork-Reward-Gemma-2-27B are taken from RewardBench leaderboard (AllenAI, )

As shown in Table 2,
we found that our best model performs similarly (within 2.2% difference) to Skywork-Reward-Gemma-2-27B on subsets that use human annotations as ground-truth while doing much worse (10.8 - 19.2 %) on subsets that use GPT-4 judgements as ground-truth. One possible explanation is that the constituent dataset for the training data of Skywork-Reward-Gemma-2-27B contains GPT-4 annotated data and hence can better model GPT-4 judgements. We find evidence of this possibility from Skywork-Reward-Gemma-2-27B’s training data description (Liu & Zeng, ) as it claims that the training data blend contains the Offset Bias dataset (Park et al., ), which has data as judged by GPT-4. Interestingly, the creators of LLMBar (Zeng et al., ) and MT-Bench (Zheng et al., ) have also noted the tendency of these tasks to overly favor models trained on GPT-4 data. Overall, this implies that our best model performs on par with the top external baseline model on the Chat-Hard category when biases associated with training on GPT-4 judgements are accounted for.

##### When to use each type of reward models?

Table: Table 3: Comparisons of different types of Reward Models in terms of aspects that they are strong on and application settings they are suitable for.

While much of the discussion above compares reward model types in terms of their RewardBench overall accuracy, real-world use cases for Reward Models have many other considerations. We analyze a few of these aspects and how such considerations influence suitable application areas for each type of Reward Model, as illustrated in Table 3.

In terms of accuracy, Bradley-Terry models are the highest (when initialized on Helpfulness Only SteerLM Regression models), closely followed by SteerLM Regression models and lastly Pairwise Justifier models. Pairwise Justifier models are the most interpretable as they can generate an explanation of why they prefer a response over another. SteerLM Regression models are also somewhat interpretable as they can be used to predict the attribute values for various dimensions of a response (e.g. high in verbosity but low in complexity). Another factor which makes SteerLM Regression models interpretable is that the predicted scores are well-calibrated. The scores are typically within the range provided in the training data - in this case between 0 to 4, with each score having a real world meaning - e.g. 4 means perfectly helpful, 3 means mostly helpful and 0 means not helpful. However, Bradley-Terry Models are not calibrated which means there’s no set interval for predictions. In the case of the Scaled BT + ExPO model, the range for rewards on RewardBench responses was [-33.50, 2.654]. BT Reward Scores also cannot be seen in isolation but must be interpreted in the context of the scores of other responses to the same prompt (see Appendix G). Finally, SteerLM Regression and Bradley-Terry are both extremely fast for inference - requiring the equivalent of 1 generated token in terms of compute while Pairwise Justifier models can require hundreds of tokens.

Because of these characteristics, different types of reward models are likely useful for different applications. SteerLM Regression is most suited for filtering SFT data (Nvidia et al., ) because data can be filtered both based on an absolute score threshold or relative score (i.e. highest score among 10 responses to the same prompt). On the other hand, Bradley-Terry models can only be used with relative score filtering.
Bradley-Terry models are most suited for RLHF given that they are the most accurate with SteerLM Regression a close second.
Finally, Pairwise Justifier models can be most suitable in supporting Human-in-the-Loop evaluation, where the explanation provided by the models can assist humans making a final decision. SteerLM Regression models can also do this by providing attribute values to dimensions of the response (e.g. complexity, correctness), but its format is less flexible for supporting human interpretation.

## 5 Aligned Models

To demonstrate the usefulness of our best reward model and HelpSteer2-Preference dataset in aligning language models to be helpful, we use them in three popular alignment algorithms.

### 5.1 Evaluation

Following Wang et al. (); Dong et al. (); Meng et al. (), we use three metrics to measure aligned models’ ability to be helpful in responding to user prompts: GPT-4-Turbo MT Bench (Zheng et al., ), AlpacaEval 2.0 Length Controlled (Dubois et al., ) and Arena Hard (Li et al., ). We report the mean number of characters in MT Bench responses to give a sense of response length.
MT Bench is also referenced as a validation metric for checkpoint selection.
Further details are in Appendix H.

### 5.2 Training

We use the Llama-3.1-70B-Instruct model (Dubey et al., ) to initialize the policy model for all experiments and Scaled BT + ExPO (94.1% RewardBench) as the reward model for PPO and REINFORCE.
Training hyperparameters and the associated search range are in Appendix E.

##### Direct Preference Optimization (DPO)

Following Section 3.2, we transform the three variants of Bradley-Terry into corresponding DPO objectives: Regular DPO (Rafailov et al., ) corresponds to Eq. 1, Margin DPO (Touvron et al., ) to Eq. 2, and Scaled DPO to Eq. 3. We train models using the Helpsteer2-Preference dataset.

##### Proximal Policy Optimization (PPO)

Following the RLHF recipe prescribed by Ouyang et al. (), we align the policy model via PPO (Schulman et al., ). We initialize the value model with the reward model. We found it useful to run 2 rounds on PPO, where we pick the best checkpoint in round 1 to initialize the policy/reference models for Round 2. The value model is always reinitialized with the reward model at each round. Our training uses the Helpsteer2-Preference prompts .

##### REINFORCE

We align the policy model with REINFORCE (Williams, ). Following Ahmadian et al. (), we use the KL-regularized reward and employ the leave-one-out baseline, sampling four responses per training prompt (Kool et al., ). We train on Helpsteer2-Preference prompts.

### 5.3 Results

As shown in Table 4, most attempted algorithms improve relative to Llama-3.1-70B-Instruct, demonstrating the strength of the HelpSteer2-Preference dataset and trained reward model.

Table: Table 4: Performance of Aligned Models. Higher is better for each metric, except Length. All models are trained by us using Llama-3.1-70B-Instruct as a base model except External Baselines, the scores for which are taken from Arena Hard (LMSys, ) and AlpacaEval leaderboards (Tatsu-Lab, )

##### Offline vs. Online RLHF

Across three DPO variants, there is consistent improvement over the base Llama-3.1-Instruct model in terms of GPT-4-Turbo MT-Bench as well as AlpacaEval 2.0 LC. We find that Scaled DPO performs best, underscoring the importance of adequately modelling the preference strength information we collected. However, no version of DPO can beat PPO or REINFORCE (across any of the three alignment metrics), highlighting the importance of using online training along with reward information.

##### Variants of Online RLHF: PPO vs. REINFORCE

Both PPO and REINFORCE can improve performance, with REINFORCE showing a marked advantage. Similar to Ahmadian et al. (), we find that REINFORCE is much better at maximizing the reward than PPO (See Appendix I for reward curves). We hypothesize that this is due to differences in baseline estimation in PPO and REINFORCE. In PPO, a learned critic function is used to estimate state values, which can introduce bias and instability (Kumar et al., ). In contrast, for REINFORCE we use a leave-one-out baseline estimator – an unbiased and stable Monte-Carlo estimator of the policy’s value function (Sutton, ). This leads to more stable training, and allows REINFORCE to outperform PPO.

##### Comparison to Frontier Models

Our best model (trained with REINFORCE) achieves competitive performance with frontier models such as GPT-4o and Claude 3.5 Sonnet across the three popular alignment benchmarks (MT Bench, AlpacaEval 2.0 LC and Arena Hard). This highlights the strength of our Reward Model to guide online RLHF. REINFORCE significantly increases the response length compared to the base models, but its high length-controlled AlpacaEval 2.0 score suggests that the extra tokens are meaningful. However, it is important to bear in mind the limitations of these automated evaluations, as all of them use a LLM-as-a-judge (GPT-4-Turbo), which is generally perceived as weaker than the top-performing External Baselines like GPT-4o and Claude 3.5 Sonnet. As a result, the LLM-judge might not be able to adequately evaluate the quality of such model responses, in relation to responses generated by our aligned models. This means that such automated evaluation for alignment may be insufficient to inform on how our models perform in relation to such frontier models in diverse use cases. Given that our main purpose is to demonstrate how our trained reward models can be useful for RLHF, we leave more holistic evaluations as future work.

##### Case Study

We adopt a prompt that many have recently been vibe-testing LLMs with - How many r in strawberry? Among models in Table 5, only REINFORCE can correctly answer it.

Table: Table 5: Model responses to “How many r in strawberry?” Newlines in responses not shown.

## 6 Conclusion

We discover that Bradley-Terry style and Regression style Reward Models are competitive to one another when the optimal formulation of each style of Reward Models is used (e.g. Scaled Bradley-Terry or Helpfulness-Only SteerLM Regression). Furthermore, they can complement one another and training a scaled Bradley-Terry model, initialized on a Helpfulness-Only SteerLM Regression model achieves 94.1 on RewardBench overall, which is #1 on RewardBench leaderboard as of 1 Oct 2024. Finally, this reward model proves useful for aligning models to follow instruction using Online RLHF, particularly with the REINFORCE algorithm.

## Ethics Statement

The prompts and responses in the HelpSteer2-Preference dataset do not contain any unsafe content (e.g. harmful content, illegal activities, profanity, bias and stereotyping) or any content containing Personally Identifiable Information (e.g. name, address, SSN, email, phone numbers), which were excluded by the HelpSteer2 (Wang et al., ) collection effort.
Annotators who supported the construction of the dataset were contracted through Scale AI, which completed ethical review prior to the start of data collection. Scale AI engages the Anker Methodology, GISC Impact Sourcing Standard, and UN Sustainable Development Goals to provide a fair and competitive pay. The specific pay is calculated based on many factors, including the specific project, the specialized skillset and expertise required, regional costs of living and then transparently listed on Scale AI platform. Scale AI also provides multiple channels for questions and support, including 24/7 support teams, community discussion channels with specially trained moderators, and a “speak up” hotline where contractors can report concerns anonymously. Worker concerns can be submitted to and are reviewed by the Remotasks support team, and pay disputes are reviewed by support specialists trained in this area.

## Reproducibility Statement

Data Pre-processing: Sec. 2 and Appendix C

Training Hyper-parameters: Appendix E

Training Compute: Appendix F

## Appendix A Related Works

##### Eliciting Human Judgements using Preference Rankings vs. Ratings

There has been a series of work from sociological statistics on comparing the relative merits of ranking and rating style questionnaires in understanding human judgements.
Alwin & Krosnick () found both ranking and rating can perform similarly in terms of modelling overall preference but capture different (but complementary) aspects of their preference.
McCarty & Shrum () similarly found that forced-choice ranking yielded more differentiated preferwnce data compared to Likert-type ratings, with the additional advantage of fewer extreme scores.
We build upon this literature to examine whether the same differences can be observed in collecting domain-general data for preference tuning of LLMs.

##### Bradley-Terry style Preference Datasets

HH-RLHF (Bai et al., ) was the first open-source general-domain, human-annotated preference dataset released by Anthropic, containing over 160,000 pairs of responses and annotations on which one response is preferred amongst the pair. However, there has been many concerns relating to the quality of this data (Wang et al., ; Shen et al., )
There are many more domain-specific datasets for individual tasks such as long-form question answering: OpenAI WebGPT Nakano et al. ()), summarization: OpenAI Summarize (Stiennon et al., ), online forum responses: Stanford Human Preferences Dataset (Ethayarajh et al., ), but they are less useful for modelling general-domain preferences.

##### Regression-style Preference Datasets

OpenAssistant (Köpf et al., ) is a prominent crowd-sourced, open-source general domain human-annotated dataset with >10,000 conversation trees released by the Open Assistant organization. Open Assistant contains Likert-5 annotations for various attributes including helpfulness, creativity, and humor.
Ultrafeedback (Cui et al., ) is a GPT-4 annotated dataset containing 64,000 prompts each with 4 responses. For each response, there are 4 attributes annotated on a Likert-10 scale: helpfulness, honesty, instruction-following, and truthfulness. With each rating, there is also a short (2-3 sentences) rationale that explains the rating. Ultrafeedback has previously been converted into a preference dataset (Tunstall et al., ) based on the mean of the 4 attribute scores - with the highest scoring response as the chosen and one of the other three responses selected at random as the rejected.
HelpSteer (Wang et al., ) is a human-annotated general domain dataset with 10,000 prompts, each with 4 responses. Each response is labelled by 1 annotator on five Likert-5 attributes - helpfulness, correctness, coherence, complexity, and verbosity.
However, Shen et al. (); Wang et al. () found the above datasets to substantially noisy, which means that reward models are unable to different between responses are not drastically different in quality.
HelpSteer2 (Wang et al., ) is a follow-up to HelpSteer with 10,000 real-world prompts, each with 2 responses. Each response is annotated on the same five Likert-5 attributes. However, every response is annotated by 3-5 annotators with the final ratings obtained by removing outlier annotations and then taking the average of the remaining annotations.

## Appendix B Preference Ranking Guidelines

### B.1 Ranking prioritization

Ratings should be made by prioritizing the response characteristics below in the order they are stated. Trade-offs should favor higher priority characteristics.

- 1.
Instruction Following
•
Responses should follow all requests in the prompt, following the instructions in the prompt is the primary criteria for ranking responses.
•
Many prompts will have a clear instruction that has additional or implicit requirements, e.g. “plan an itinerary for a 5-day trip to Portugal that focuses on surfing” should be evaluated on whether an itinerary is produced, whether it is 5 days long and in Portugal, and also whether it includes a lot of surfing - All instructions should be considered when ranking responses, with the core instructions being weighed more heavily.
•
If specific formatting (table, json) is asked for in a prompt that should be considered as the high priority “instruction following” and not general “formatting” which is lower priority.
- 2.
Correctness
•
Answers which are factually correct should be rated higher than answers that are incorrect.
•
Annotators should search the internet when they are unsure about correctness.
•
Misleading or incomplete answers are considered less correct than complete answers.
•
If prompt (question) contains false premise, the response that pushes back on it should be preferred.
•
When a question cannot be answered definitively, the response which expresses uncertainty should be preferred.
- 3.
Formatting
•
When no specific formatting is requested responses with better formatting are preferred.
•
Vendor tool should render markdown for easier assessment of formatting (markdown tables, lists, shell scripts, etc. should be rendered properly).
•
Formatting encompasses anything that is appropriate for the response, e.g. an itinerary being split by day, including a markdown table when it makes sense but was not asked for, any appropriate use of markdown, responding with a shell script, etc.
- 4.
Clarity
•
Answers that are easier to understand, provide more clarity (via explanation or introduction), or are generally better written are preferred.
•
Unnecessary verbosity, or providing extra information that is irrelevant should be penalized.

### B.2 Ranking strength

We would like to collect three levels of preference, described below:

- 1.
Response 1 is slightly better than Response 2 (or vice-versa)
•
To be used when the responses are similarly appropriate and the difference is minor or a matter of personal preference.
•
There should be no difference in Instruction Following or Correctness if this is selected.
•
Minor differences in clarity and formatting warrant this response.
•
When the annotator considers the options to be tied in all other aspects, they should prefer the shorter answer (in unlikely circumstances of same length - use their own judgment)
- 2.
Response 1 is better than Response 2 (or vice-versa)
•
To be used when one response is clearly better than the other, such as:
•
If one response is clearly better than the other, but the worse response is still appropriate (it is not incorrect or off topic).
•
If the worse response fails to follow some instructions or is correct but not well phrased/structured, and the better response also fails some instructions at the same level, but has no other flaws overall.
•
If the better response fails to follow some but not all instructions or is misleading but the worse response does not follow instructions at all or is wrong, this should be selected.
•
If both answers follow instructions and are correct, but one is clearer and/or better formatted, this should be selected.
- 3.
Response 1 is much better than Response 2 (or vice-versa)
•
To be used when there is a significant difference between the two responses.
•
If one answer is correct and the other isn’t, this should be selected.
•
If one answer follows all instructions and the other does not follow instructions, this should be selected.
•
If there are major differences in readability and formatting, this should be selected.
- 4.
Neither response is valid
•
If neither response follows the instructions or provides a correct answer, this option should be chosen.
•
This is meant to be used only for egregious issues. If both answers could be improved, but they follow what the prompt asked with no critical error then a preference should be selected instead.

### B.3 Ranking examples

*Note that the full guidelines contain more examples, but we show here only four illustrative ones, one for each preference strength.*

#### B.3.1 Response 1 is slightly better than Response 2

Prompt
Describe the process of photosynthesis. Additionally, explain how variations in light intensity can affect this process and how plants have adapted to such changes.

Response 1
Photosynthesis is the process by which green plants and some other organisms convert light energy, typically from the sun, into chemical energy in the form of glucose (sugar). It can be broken down into two stages:
1. Light-dependent reactions: These occur in the thylakoid membranes of chloroplasts. Light energy is absorbed by chlorophyll and other pigments, splitting water molecules and producing oxygen, electrons, and ATP (adenosine triphosphate).
2. Light-independent reactions (Calvin cycle): These occur in the stroma of the chloroplast. Carbon dioxide is taken in from the atmosphere, and with the help of ATP and electrons produced in the light-dependent reactions, it is converted into glucose and other organic molecules.

Variations in light intensity can affect photosynthesis in the following ways:
- Low Light Intensity: Photosynthesis decreases as there is insufficient light for the light-dependent reactions. This leads to a lower production of ATP and NADPH, slowing down the Calvin cycle and reducing glucose production.
- High Light Intensity: Excessive light can damage chlorophyll and other pigments, leading to reduced photosynthesis. To protect themselves, plants have adapted by mechanisms like photoprotection and non-photochemical quenching to dissipate excess light energy as heat.

Plants have evolved several adaptations to cope with changes in light intensity, including adjusting the size of their chloroplasts, altering the angle of their leaves, and using pigments that can capture light at different wavelengths. Additionally, they can regulate their stomata to control the entry of carbon dioxide and reduce water loss during times of intense sunlight. These adaptations help optimize photosynthesis under varying light conditions.

Response 2
Photosynthesis occurs in chloroplasts of plant cells. It uses sunlight, carbon dioxide, and water to produce glucose and oxygen. Light energy is absorbed by chlorophyll, splitting water molecules and releasing oxygen. The energy from light is used to convert carbon dioxide into glucose.
Higher light intensity increases the rate of photosynthesis, as more light energy drives the light-dependent reactions. However, other factors like CO2 and temperature can limit the rate. Too much light can damage the photosynthetic machinery.
Plants adapt to varying light through changes in leaf angle, altering chlorophyll production, opening/closing stomata, and photorelocation of chloroplasts. These adaptations maximize photosynthesis in both low light and intense light conditions by balancing light capture with CO2 intake and production capacity.

Preference Justification
@Response 1 is slightly better than @Response 2. While both responses correctly answers the prompt, the style and formatting of the better response is preferred. It also gives additional facts about photosynthesis.

#### B.3.2 Response 1 is better than Response 2

Prompt
Discuss the benefits of renewable energy sources.

Response 1
Renewable energy sources such as solar, and wind offer numerous benefits to mitigate climate change. Unlike exhaustible fossil fuels, renewables rely on naturally replenished resources. Transitioning to renewables will significantly benefit public health and the environment by reducing hazardous air and water pollution produced from extracting and burning fossil fuels.

Response 2
Renewable energy sources, such as solar, wind, and hydro power, offer numerous benefits. They produce clean energy, reducing greenhouse gas emissions and our reliance on fossil fuels. Additionally, renewable energy is sustainable, helping to protect the environment and conserve resources for future generations.

Preference Justification
@Response 1 is better than @Response 2. Both responses approach the prompt from slightly different angles. One could consider @Response 1 the better response given its additional information about transition to renewable sources and how it could impact the environment.

#### B.3.3 Response 1 is much better than Response 2

Prompt
Make an itemized list of 4 elements with actions that can increase the emission of oxygen. Item markers must be lower case letters starting from D.

Response 1
d. Planting trees: Trees release oxygen through the process of photosynthesis, making them the most effective way to increase oxygen levels.
e. Growing algae: Algae in ponds or water bodies can play a significant role in oxygen production. Cultivating algae can increase their growth and subsequent oxygen release.
f. Creating green spaces: Designating and maintaining green spaces in urban areas, such as parks and gardens, help support plantations and increase oxygen levels.
g. Promoting indoor plants: Placing indoor plants in your living or working space can enhance oxygen production and improve air quality, benefiting the overall oxygen levels indoors.

Response 2
D. Photosynthesis: Photosynthesis is the process by which plants, algae, and some bacteria convert light energy from the sun into chemical energy in the form of organic compounds, such as glucose. This process releases oxygen as a byproduct, which is then released into the atmosphere.
E. Respiration: Respiration is the process by which cells in living organisms convert glucose into energy. During this process, oxygen is consumed and carbon dioxide is produced. Increasing the rate of respiration in living organisms can increase the amount of oxygen released into the atmosphere.
F. Exercise: Exercise is a physical activity that can increase the rate of respiration in the body. As the body works out, it burns more glucose and consumes more oxygen, leading to an increase in oxygen emissions.
G. Forestation: Planting more trees and forests can increase the amount of photosynthesis occurring in an area, which can lead to an increase in oxygen emissions. Trees and other vegetation absorb carbon dioxide from the atmosphere and release oxygen as a byproduct of photosynthesis.

Preference Justification
@Response 1 is much better than @Response 2. The prompt specified that the item markers should be lower case. @Response 2 starts from the letter “D”, but it is not lower case.
In addition, @Response 2 has incorrect facts about respiration and is not coherent/fluent regarding the prompt.

#### B.3.4 Neither response is valid

Prompt
When it is 1pm in San Francisco, what time is it in Singapore?

Response 1
It is 8pm in Singapore when it is 1pm in San Francisco.

Response 2
It is 10pm in Singapore when it is 1pm in San Francisco.

Ranking explanation
Neither option is factually correct. The correct answer depends on whether Daylight Saving Time is in effect. When it is 1 pm in San Francisco during Standard Time, it is 5 am the next day in Singapore. During Daylight Saving Time in San Francisco, when it’s 1 pm in San Francisco, it is 4 am the next day in Singapore.

## Appendix C Justification Pre-processing

Our annotation guidelines provided minimal guidance on how the preference justification should be written within the free-text box. One such guidance was that annotators should consistently refer to the rated responses as "@Response 1" and "@Response 2". This meant preference justifications do not always conform to a fixed format. To better understand these justifications, we split them into sentences using NLTK sentence-tokenizer (NLTK, ) as well as newlines, to facilitate subsequent post-processing.

Many preference justifications contain a preference statement, in the form “@Response 1 (or 2) is better … ”. Most commonly (57.6% of samples), these preference statements are only contained in the first sentence. This is followed by 27.5% of samples containing it in both the first sentence and the last sentence. A minority (7.3%) contains it in the last sentence while a similar proportion (7.6%) does not contain such a preference statement at all. Among those without a preference statement, a qualitative analysis revealed that some state the strengths and weakness of each response without comparison, some claim that both responses are equal (despite our schema not supporting equivalence) while others make aspect-specific comparisons without stating if it makes either response better.

We believe that it is important to have all justifications in a similar format because use cases such as LLM-as-a-judge require justifications to be standardized for automatic parsing. Therefore, we split the justification into a preference statement and a preference elaboration. This enables users to freely decide if they prefer to generate the preference statement first or the preference elaboration first. For justifications containing a preference statement both at the front and the back, we remove the trailing preference statement to avoid duplication. In addition, preference statements that are extracted from the end of justifications are frequently prefixed with terms like “Therefore, ”, “Overall, " and “For these reasons, ". To ensure uniformity, we remove all prefixes before the first “@Response 1/2 … better’ is encountered.

Requiring all preference justifications to have a preference statement makes justifications without one difficult to use directly. After considering several approaches (e.g. prompting an LLM to restructure justifications to follow a format or manually adding in the preference statement using the associated preference score), we decided that the best solution would be to exclude the preference justifications without a preference statement altogether. This is because the potential benefit of having an additional 7.6% of data was not worth the risk of LLM hallucinations or potentially contradictory/repetitive information from the rule-based heuristic.

Finally, we prepare a subset of these justifications into the Overall Preference Justification, by randomly selecting one justification per task, whose annotator has given the same individual preference as the overall preference. The Overall Preference Justification set, which comprises 6, 618 samples with 6, 287 in the train set and 331 in the validation set while the All Justifications dataset has 24, 698 justifications (23, 487 train and 1, 211 val).

Table: Table 6: Descriptive statistics for preference justifications. Preference justification consists of both a preference statement and a preference elaboration, and both together is termed Full. Overall Preference Justifications is a subset of all justifications in which only one representative justification is used for each task.

## Appendix D Preference Justification Analysis

Table: Table 7: Analysis of keywords and factors mentioned in preference justifications.

In Table 6, we analyze the descriptive statistics for preference justifications. On average, preference justifications contain 4 sentences or 500 characters (approximately 100 words). There is a slight preference for Response 2 over Response 1 (6-7%). Approximately half of all preference statements contain the word “because” while other words implying rationales such as ’due to’ and ’owing to’ are minimal ($<$1%). The other half of the preference statement communicates a preference without explanation. The vast majority of responses (83%) mention both responses in the preference with nearly all responses mentioning both responses in the full justification. This suggests that most annotators used a comparative approach between two responses, indicating that they follow our guidelines well.

Table: Table 8: Questions used for analyzing if preference justification discusses each attribute or factor. Each question was used with the template‘{justification} Does the above comparison between @Response 1 and @Response 2 discuss {question} Answer only with yes or no.’

To better understand the content contained in preference justifications, we conducted a word-level analysis of Overall Preference Justifications (including both preference statements and preference elaborations). We first lowercase the justifications and replace all non-alphanumerical symbols with empty strings, before splitting justifications into bags of whitespace-separated words. We then count the number of justifications in which each word appears. From the top 500 most frequent words, we manually identify those that relate to each HelpSteer2 attribute.

Based on Table 7, we found that annotators used each of the five HelpSteer2 attributes to guide their justification writing. Most influential was Correctness-related words (12.9%) followed by Helpfulness-related words (11.0 %). We suspect that the Helpfulness is under-represented because very few keywords (6) are closely associated with helpfulness while many more (21) are associated with correctness. Surprisingly, verbosity is the next most influential to annotators (4.0 %), and more so compared to coherence (3.5 %). We hypothesize that this might be due to responses from HelpSteer2 being mostly Coherent (mean of 3.64 out of 4) meaning that both responses are likely to be perfectly coherent. Finally, complexity-related words only appear in 0.6% of justifications, suggesting that they barely affect preference judgments. Overall, the proportion of justifications that can be explained by these attributes however, is low at ($<$32.0%). This is likely due to our word-level analysis only involving the top 500 most frequent keywords, which might not be sufficient to capture all aspect-relevant features. For instance, an annotator might communicate that a response is more helpful than another, using the terms helpful, useful, or instruction (following).

To complement the word-level analysis, which is interpretable but with low recall, we also prompted an LLM, specifically Nemotron-4-340B-Instruct (Nvidia et al., ) to classify whether the preference justification discusses each attribute (prompts in Table 8). LLM-based analysis shows a much higher proportion of justifications mentioning each attribute. Nonetheless, the relative proportion between attributes remains similar - with the primary factors influencing preference being helpfulness and correctness; followed by coherence and verbosity; and complexity trails behind as the least important attribute.

In addition, we wanted to understand if factors outside of HelpSteer attributes influenced annotator preferences. We used a list of 24 factors identified by Sharma et al. () including politeness, empathy, and persuasiveness, and prompt the LLM to classify in a similar manner. After classifying each factor separately, we organized them into three main categories - Stylistic, Sycophantic (i.e. similar to the prompt), and HelpSteer-Adjacent. Unsurprisingly, HelpSteer-Adjacent factors are most commonly mentioned in these justifications (90.1%). Comparatively, stylistic factors and sycophantic factors that were found to substantially influence human and LLM judgments in previous work (Sharma et al., ; Chiang et al., ) are rarely discussed by our annotators (5.9% and 1.4%). This suggests the effectiveness of our guidance to annotators on focusing on the ‘substance’ of responses.

## Appendix E Training Hyper-parameters

##### Reward Modelling

For SteerLM Regression models, we used 2 epochs of Helpsteer2 data, following Wang et al. (). For Bradley-Terry models, we used 1 epoch of HelpSteer2-Preference data, as more than 1 epoch resulting in overfitting - similar to as observed by Zhu et al. () and drastically higher validation loss and poorer RewardBench performance.
For Pairwise Justification models, we train for 1 epoch for each setup - we also tried 2 epochs for initial experiments which showed minimal changes from only training on 1 epoch. For each experiment, we used global batch size of 128 using an AdamW optimizer with 10 warm-up steps and performed search over constant learning rates of {1, 2, 3}$e-6$. We report the optimal learning rate for each experiment in Table 1. We save checkpoints every 10 steps and evaluate three checkpoints with the lowest validation loss as well as the last checkpoint. Among them, we report the score with the highest RewardBench overall score. For SteerLM Regression Model with all five 5 HelpSteer attributes, we do grid search over all five attributes (between -1 and 1 at intervals of 0.1) over RewardBench.

##### Direct Preference Optimization

We trained DPO models for 2 epochs using the AdamW optimizer, after initially experimenting with 1 and 3 epochs but finding 2 epochs to be optimal. We use a global batch size of 64 response-pairs, weight decay of 0.01, with checkpoints saved every 10 steps, and 10 warmup steps followed by a constant learning rate. For hyper-parameter tuning, we performed a grid search over learning rates of {1, 2, 3}$e-7$ and KL penalties of {0.01, 0.001}. We report the optimal learning rate, KL penalty for each experiment in Table 4.

##### Proximal Policy Optimization

We trained PPO models for a total of 2 rounds using the AdamW optimizer weight decay of 0.1, constant learning rate schedule and 30 steps of value model warmup. Due to the instability of PPO training also observed by (Zhu et al., ), we save checkpoints every 2 steps with our final checkpoints being at step 26 and 40 for rounds 1 and 2 respectively. For hyperparameter tuning we performed a grid search over learning rates {5e-6, 1e-6, 5e-7, 1e-7, 1e-8}, KL penalties of {0.1, 0.01, 0.001, 0.005, 0.0001} and training global batch size of {64, 128, 256}. Compared to other algorithms, we performed more hyperparameter search because we found PPO to be more sensitive to hyperparameters. In our experiments we set the training global batch size to be the same as the rollout global batch size and found 128, 256 the best for rounds 1 and 2 respectively. We report the optimal learning rate, KL penalty for each experiment in Table 4.

##### REINFORCE

Similar to PPO, we trained REINFORCE models using the AdamW optimizer with rollout batch size of 64, training batch size of 64, weight decay of 0.1, and a constant learning rate schedule with ten warmup steps. We sample four samples per prompt, and use the leave-one-out baseline. We save checkpoints every 5 steps, and the best checkpoint is found at step 95. for hyperparameter tuning we performed a grid search over learning rates {1e-7, 3e-7, 5e-7, 1e-6} and KL penalties of {0.01, 0.1}. The optimal learning rate and KL penalty are reported in Table 4.

## Appendix F Compute requirements

Table: Table 9: Compute required for training models, measured in H100-eqv. node-hours. Experiments are run on nodes of 8 A100/H100-80GB SXM GPUs on internal clusters. For ease of comparison, every three A100 node-hours is converted to one H100 node-hour.

## Appendix G Rewardbench Response Distribution

Fig. 2 shows the distribution of reward scores on Rewardbench responses as judged by our best Reward Model.

Figure: Figure 2: Distribution of Reward Scores for RewardBench responses by our best reward model (Scaled BT + ExPO, initialized on Helpfulness-Only SteerLM Regression model). Difference refers to the difference between the reward scores of chosen and rejected responses to the same prompt. When a response has reward of -15, it is equally likely to be chosen or rejected. This meaning the score of the response cannot be seen in isolation to determine if a response is good (or bad).
Refer to caption: fig/scaled_bt_plus_expo_plot.png

## Appendix H Aligned Model Evaluation Details

##### MT Bench

We follow (Meng et al., ; Tenyx, ; Wang et al., ) to use MT Bench (Zheng et al., ), with GPT-4-Turbo (specifically GPT-4-0125-Preview) as the judge. MT Bench consists of 80 multi-turn questions, each consisting of an initial question and a follow-up question, for a total of 160 prompts. Questions are collated from 8 categories: Writing, Roleplay, Extraction, Reasoning, Math, Coding, STEM and Humanities/Social Science. We first greedily generate responses with up to 1024 tokens (default value for MT Bench). The responses to these prompts are evaluated by GPT-4-0125-Preview to give a score between 1 and 10, and we report the mean across all prompts. Prompts in Coding, Math and Reasoning categories are evaluated with a reference correct answer, which were generated by the judge model and then manually verified. Higher MT Bench score indicates better instruction following ability.

AlpacaEval 2.0 Length Controlled We follow Dong et al. (); Meng et al. (); Wang et al. () to use AlpacaEval 2.0 Length Controlled (Dubois et al., ). AlpacaEval 2.0 contains 805 first-turn instructions (relating to singular-requirement, straightforward tasks such as recommendations, question answering, and open-ended writing). An answer to each prompt is greedily generated by the evaluated model as well as a baseline model (GPT-4-1106-turbo), which are then sent to GPT-4-1106-turbo evaluator that outputs the probability of preferring the generations of the evaluated model. Finally, the authors introduced a length correction factor to mitigate the bias for the evaluator towards preferring longer generations.

Arena Hard We follow (Dong et al., ; Meng et al., ; Wang et al., ) to use Arena Hard (Li et al., ). Arena Hard contains 500 first-turn instructions obtained from challenging user queries on Chat Arena (Chiang et al., ). Prompts are classified using an LLM (Llama-3-70B-Instruct) to determine if they are complex, specific, real-world-application-related or require domain knowledge, problem-solving, creativity, technical accuracy. Prompts that meet at least 6 out of 7 criteria are labelled as challenging. Therefore, a huge proportion of prompts (>50%) are related to solving coding problems. Model responses are then compared with responses from GPT-4-0314 using GPT-4-1106-preview judge to calculate a win-rate over GPT-4-0314.

## Appendix I Reward Curves

We plot the reward obtained by PPO and REINFORCE in Fig. 3.

Figure: Figure 3: Reward curves of PPO and REINFORCE. Please note that REINFORCE samples four times as many responses as PPO per training step. For PPO, the reward curve is produced by concatenating the reward curves for Round 1 (ending at step 57) and Round 2. We select step 26 for PPO Round 1, step 40 for PPO Round 2 (whose reward is shown at step 97 above) while for REINFORCE, we select step 95.
Refer to caption: fig/reward_curves.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Has a prompt filter been applied to ensure that prompts from the rewardbench are not included in the training set?

Additionally, how does SteerLM regression, initialized with the Bradley-Terry model, perform?
soundness: 3
strengths: This paper provides an excellent dataset that will be highly valuable to the community. It addresses an important question: whether the Bradley-Terry style or regression style is more effective for reward modeling. Additionally, the reward model (RM) proposed here is capable of effectively training a LLaMA-70B model.
summary: This paper explores whether a Bradley-Terry or regression approach is more effective for reward modeling and proposes a novel method that combines both styles. The conclusion is intriguing: the format in which data is collected and the specific model training approach do not significantly impact the results.
weaknesses: I'm glad to see that the BT training method with an added margin (eq.2) is effective. However, this finding somewhat conflicts with the results in the Llama 3 paper, where the authors mention: "The training objective is the same as Llama 2, except that we removed the margin term in the loss, as we observed diminishing improvements after data scaling." What could be the reason behind this discrepancy?

I understand that this work provides an excellent dataset and rigorous experimental validation. However, I would like to see more analysis, such as:

1. A comparison of results on out-of-distribution prompts.

2. A consideration of annotation costs: since preference annotations are likely cheaper and easier to scale than Likert scores, a fair comparison might require factoring in annotation costs rather than simply matching dataset sizes.

3. An analysis of robustness to annotation errors. For instance, assume that preference annotations have a 5% chance of being reversed, or that Likert scores have a similar error rate (where a 9 might be incorrectly marked as a 1, rather than just slightly off, like 8). Which reward modeling approach is more robust to such errors?

These aspects would provide valuable insights into the practical implications of each approach.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Could the authors provide statistical significance testing to validate the reported performance improvements of the hybrid model? This would strengthen the robustness of the conclusions.
- Have the authors considered evaluating the align models on standard language modeling tasks to assess the alignment tax? 
- Based on the appendix, it appears that the RL training involved approximately 10k examples, and the model was trained for fewer than 100 iterations. Could the authors clarify the stopping criteria for the training process? Additionally, could you specify the KL divergence value of the choosen checkpoints from the reference policy? This information would provide better insight into the training stability and convergence behavior of the reward model.
- Are the conclusions valid for cases where, e.g, 5 responses ranking are used for RM training?
soundness: 4
strengths: - The authors have created and open-sourced the HelpSteer2 dataset, which includes high-quality preference annotations and justifications specifically designed for reward model (RM) training.
- The paper conducts a detailed, head-to-head comparison between Bradley-Terry (BT) and Regression-style reward models, providing insights into the relative strengths and weaknesses of each approach.
summary: This paper presents HelpSteer2, a dataset designed to facilitate research in RM for aligning language models with human preferences. This work addresses the gap of fairly comparison of BT and regression style of RM by releasing preference annotations compatible with the BT style alongside existing ratings formatted for Regression-style training. The authors then conduct a head-to-head comparison of the BT and Regression models when matched for data quality, providing valuable insights into the effectiveness of each approach. They also propose a hybrid model that combines elements of both BT and Regression approaches, achieving state-of-the-art performance on the RewardBench benchmark. Additionally, the authors open-source the HelpSteer2 dataset and trained reward model to promote further research in RM alignment tasks.
weaknesses: - While the hybrid approach is innovative, it seems to be an incremental improvement rather than a fundamentally new modeling technique. 
- Lack of statistical significance testing to validate the effectiveness of proposed hybrid modeling method.
- The paper does not evaluate the aligned models on standard language modeling tasks, which would help in quantifying the alignment tax.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How were the annotators trained to provide consistent ratings and justifications, and what measures were in place to minimize subjective bias?
2. How well does the dataset generalize to real-world applications beyond reward modeling and instruction-following?
3. What is the impact of filtering out samples with high annotation variability on the overall representativeness and diversity of the dataset?
4. Would models trained on a smaller, higher-quality dataset perform better than those trained on larger datasets with more noise?
5. How might the dataset be expanded or adapted to cover other dimensions of reward modeling, such as domain-specific use cases or more complex types of preferences?
soundness: 3
strengths: First, it complements existing ratings with detailed preference annotations, allowing for more nuanced training of reward models. By collecting preference data alongside Likert-scale ratings, the dataset enables direct comparisons between models trained under different paradigms (e.g., Bradley-Terry vs. Regression), which was previously difficult due to incompatible data formats in existing datasets. This dual approach provides a richer training ground for testing and developing models that need to evaluate comparative quality.

Second, the dataset includes human-written justifications for preferences, adding a layer of interpretability. This is particularly valuable for training models that can both assess and explain their choices, supporting applications where transparency is important, such as in human-in-the-loop systems. The justifications make the dataset unique by facilitating the development of models that are not just performance-driven but also capable of detailed decision-making explanations.

Third, the dataset maintains high-quality standards, demonstrated by its filtering processes and the steps taken to ensure strong inter-annotator agreement. By curating data that passes rigorous quality checks, the authors improve the signal-to-noise ratio, making it more effective for training robust models. This attention to quality addresses common issues in training data, where noisy or inconsistent annotations can limit model performance.

Overall, the HelpSteer2-Preference dataset's strengths lie in its dual-format data collection, the inclusion of human justifications for enhanced interpretability, and its commitment to high-quality, reliable annotations. This makes it a valuable resource for advancing research in reward modeling and instruction-following alignment in language models.
summary: This work addresses two major paradigms: Bradley-Terry (BT) and Regression style models, examining which is better when matched for data. The main challenge in previous studies was the lack of comparable datasets collected using both methodologies. To resolve this, the authors introduce the HelpSteer2-Preference dataset, which complements existing ratings with preference annotations, including human-written justifications. This dual-annotation allows a head-to-head comparison between BT and Regression models under controlled data conditions.
weaknesses: While the HelpSteer2-Preference dataset brings several innovations and benefits, it also has certain weaknesses that could impact its applicability and use in research and development:

First, the dataset's format, which includes both ratings and preferences, may require more complex preprocessing and handling by researchers. The dual data structure means that models need to be capable of leveraging both individual ratings and comparative preferences, which could increase the implementation complexity for teams that are not equipped with extensive data processing expertise.

Second, despite the inclusion of human-written justifications enhancing interpretability, these justifications may introduce variability due to subjective writing styles or annotator biases. This subjectivity can make it challenging to maintain consistency across annotations and could potentially affect the model's ability to generalize when trained with such data. Additionally, training on justifications may require more advanced model architectures that can process and utilize free-form text, which may not be available or practical for all research teams.

Third, while the dataset emphasizes high-quality annotations with steps taken to filter out low-confidence or inconsistent data, this strict filtering reduces the overall volume of usable data. The exclusion of samples with high variability among annotators may result in a smaller training set, potentially limiting model performance, especially for models that benefit from a large and diverse dataset. This trade-off between quality and quantity could be a bottleneck for researchers who need extensive data for training large-scale models.

Lastly, the HelpSteer2-Preference dataset is tailored to specific types of model training and evaluations, such as Bradley-Terry and Regression-based reward models. This specialization may limit its generalizability to other types of machine learning tasks or applications that do not align with these paradigms. Researchers working on models with different objectives or those requiring more diverse data formats might find the dataset less adaptable.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: See weaknesses.
soundness: 3
strengths: 1. The paper introduces a carefully annotated preference dataset that could be useful. The authors also conduct some analysis on the difference of pointwise and pairwise annotation.

2. This paper compares Bradley-Terry and Regression style reward modeling paradigms, which motivates them to design a new method to make the best of both worlds.
summary: This paper complements the existing HelpSteer2 dataset with preference annotations, showing that pairwise annotation can further distinguish responses with similar likert scores. This dataset, together with its counterpart HelpSteer2, enables a head-to-head comparison between Bradley-Terry and Regression style reward models, by training models with the same prompts and responses in HelpSteer2 and only differing the feedback annotation. Using insights from this comparison, they propose a novel approach combining both methods, achieving superior reqward modeling performances. They also conduct experiments with RL algorithms to demonstrate the effectiveness of the reward models.
weaknesses: 1. The motivation is somehow overclaimed. In line 013-016, the authors claim that there lacks datasets to enable the comparison between Bradley-Terry and Regression style reward modeling. However, all preference datasets with point-wise scores can be converted into pair-wise setups, which should have been able to allow the comparison between the two paradigms. But I agree with authors that "Regression-style data collection (where responses are rated individually) gives
annotators a different set of expectations for comparative ranking of responses" (line 055-056), so it'd be better to have a pair of dataset directly annotated in point-wise and pair-wise schema. Admittedly, the comparison setup is improved now but it was not totally undoable.

2. The authors conducted a lot of experiments, but it's hard for readers to parse useful information from the experiments with almost no highlights in writings. The abstract mainly focuses on what they did, with no mention on what they have found in their experiments. In the experiment sections, readers would benefit a lot from bolded high-level excerpts to quickly grasp the key findings. 

3. The results are good to know, e.g. the comparison between Bradley-Terry and Regression style reward modeling, but personally I don't find it surprising or informative enough to serve as the main claim of a conference paper. It would better to investigate deeper what properties of an algorithm make a good reward model. Also, the experiments are mainly conducted on general chats with subjective metrics (LLM-as-a-judge), thus the conclusions may not be able to generalize to other taks like reasoning, which is known to be very different from chatting. Regarding other experiments, such as section 5, the authors conduct RL with their reward models and datasets mainly for the sake of demonstrating their usefulness. In this case, it seems baselines are needed so that readers can be better aware of the necessity of another preference dataset, given the existence of many good alternatives, and distinguish this dataset with existing ones.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - How does the reward model perform if combined vice-versa, using SteerLM Regression model with the initialization of Bradley-Terry model?
- How many hours of human labors do you use to collect the human annotations?
- How does the proposed reward model combination method compared with model ensembling?
soundness: 2
strengths: - The writing is clear and easy to follow.
- The data collection process is well-designed, with detailed and comprehensive analysis provided.  The proposed dataset provides a direct and fair comparison between Bradley-Terry and Regression-based reward models. 
- The proposed method of  combining of different reward models is effective on RewardBench, and improves LLaMA-3.1 through preference alignment.
summary: This paper introduces Helpsteer2-preference, a preference dataset that provides additional pairwise preference annotations alongside the existing Likert-5 style rating of the Helpsteer-2 dataset. Using this dataset, the authors conduct head-to-head comparisons between two major types of reward models. Then, they propose a method to combine two types of reward models to achieve better performances on RewardBench and preference learning.
weaknesses: - One major concern is the paper/s dependence on HelpSteer2. While the authors states that "converting regression-style data to preference pairs cannot fully capture nuances in preferences", there are no experimental results to support this claim. It would strenghthen the paper to include experimental comparisons with models trained on the preference dataset converted from Helpsteer2. This would help clarify whether the improvement is due to the new preference data itself or the different formulation of reward models.

- Lack of comparison with other reward models or preference dataset in preference alignment (RLHF, DPO). 
  - For DPO, what if you apply the converted preference dataset from HelpSteer2? 
  - For PPO, can you compare it with other reward models of the same base model(e.g. Llama3-70B-SteerLM-RM, TextEval-Llama3.1-70B)?

Meta-review: ## **Summary**
This paper introduces the **HelpSteer2-Preference** dataset, which extends the existing HelpSteer2 dataset by including pairwise preference annotations alongside Likert-5 style ratings. This addition allows for a head-to-head comparison of Bradley-Terry (BT) and Regression-style reward modeling paradigms. The authors further propose a novel method that combines insights from both approaches, yielding state-of-the-art (SOTA) results on RewardBench and strong performance in alignment tasks using reinforcement learning with human feedback (RLHF). The dataset and trained models are open-sourced under a permissive CC-BY-4.0 license, promoting reproducibility and practical use.

## **Strengths**
1. **Data Contribution**: The HelpSteer2-Preference dataset addresses an existing gap by providing data that allows for a fair comparison between BT and Regression-style models, a challenge due to the different data collection formats typically used by these paradigms.
2. **Modeling Insights**: The authors conduct a detailed analysis of the three BT variants (Regular, Margin, and Scaled), identifying Scaled BT as the most effective. They also demonstrate that combining BT with Regression-style approaches leads to better performance.
3. **Empirical Results**: The models trained using HelpSteer2-Preference achieve SOTA performance on RewardBench and outperform strong baselines on alignment benchmarks such as Arena Hard.
4. **Open Sourcing**: The (promised) release of the dataset and trained models ensures accessibility and fosters community-driven advancements.

## **Weaknesses**
1. **Incremental Nature of Contributions**: Some reviewers noted that the hybrid approach, while effective, is not fundamentally novel. The dataset augmentation is viewed as a refinement rather than a solid contribution.
2. **Dependence on HelpSteer2**: While the dataset enhances HelpSteer2, the reliance on its structure could limit its generalizability to other domains or applications. However, the authors provided strong justifications for this reliance.
3. **Evaluation Diversity**: The evaluation focuses on alignment tasks and general chat scenarios, which may not fully generalize to more specialized domains like reasoning or code generation.


## **Recommendations**
The paper demonstrates strong empirical results and provides a valuable dataset that will benefit the community. While the contributions are more practical than theoretical, they represent a meaningful step forward in advancing reward modeling and alignment for large language models. The reviewers generally agree on the rigor of the work, the clarity of the presentation, and its significance for practical alignment tasks.

This work is well-executed, and the dataset is likely to have a broad impact, particularly for researchers working on reward modeling and RLHF. While there is room for further insights, the paper's empirical rigor and open-source contributions make it a strong candidate for acceptance.

Decision: Accept (Poster)