HelpSteer2-Preference: Complementing Ratings with Preferences

Authors: Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, Yi Dong
Primary area: datasets and benchmarks
Keywords: reward modelling, rlhf, model alignment

Abstract:
Reward models are critical for aligning models to follow instructions, and are typically trained following one of two popular paradigms: Bradley-Terry style or Regression style. However, there is a lack of evidence that either approach is better than the other, when adequately matched for data. This is primarily because these approaches require data collected in different (but incompatible) formats, meaning that adequately matched data is not available in existing public datasets. To tackle this problem, we release preference annotations (designed for Bradley-Terry training) to complement existing ratings (designed for Regression style training) in the HelpSteer2 dataset. To improve data interpretability, preference annotations are accompanied with human-written justifications. Using this data, we conduct the first head-to-head comparison of Bradley-Terry and Regression models when adequately matched for data. Based on insights derived from such a comparison, we propose a novel approach to combine Bradley-Terry and Regression reward modeling. A Llama-3.1-70B-Instruct model tuned with this approach scores 94.1 on RewardBench, emerging top of more than 140 reward models as of 1 Oct 2024. This reward model can then be used with REINFORCE algorithm (RLHF) to align an Instruct model to reach 85.0 on Arena Hard, which is No. 1 as of 1 Oct 2024.
We open-source this dataset (CC-BY-4.0 license) at https://huggingface.co/datasets/nvidia/HelpSteer2#preferences-new---1-oct-2024 and openly release the trained Reward and Instruct models at https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward and https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct .

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730235044048

thread_type:
official_review

<thread_object>
creator: reviewer_wyr3

<content>
summary:
This paper introduces Helpsteer2-preference, a preference dataset that provides additional pairwise preference annotations alongside the existing Likert-5 style rating of the Helpsteer-2 dataset. Using this dataset, the authors conduct head-to-head comparisons between two major types of reward models. Then, they propose a method to combine two types of reward models to achieve better performances on RewardBench and preference learning.

strengths:
- The writing is clear and easy to follow.
- The data collection process is well-designed, with detailed and comprehensive analysis provided.  The proposed dataset provides a direct and fair comparison between Bradley-Terry and Regression-based reward models. 
- The proposed method of  combining of different reward models is effective on RewardBench, and improves LLaMA-3.1 through preference alignment.

weaknesses:
- One major concern is the paper/s dependence on HelpSteer2. While the authors states that "converting regression-style data to preference pairs cannot fully capture nuances in preferences", there are no experimental results to support this claim. It would strenghthen the paper to include experimental comparisons with models trained on the preference dataset converted from Helpsteer2. This would help clarify whether the improvement is due to the new preference data itself or the different formulation of reward models.

- Lack of comparison with other reward models or preference dataset in preference alignment (RLHF, DPO). 
  - For DPO, what if you apply the converted preference dataset from HelpSteer2? 
  - For PPO, can you compare it with other reward models of the same base model(e.g. Llama3-70B-SteerLM-RM, TextEval-Llama3.1-70B)?

questions:
- How does the reward model perform if combined vice-versa, using SteerLM Regression model with the initialization of Bradley-Terry model?
- How many hours of human labors do you use to collect the human annotations?
- How does the proposed reward model combination method compared with model ensembling?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730605170390

thread_type:
official_review

<thread_object>
creator: reviewer_avct

<content>
summary:
This paper complements the existing HelpSteer2 dataset with preference annotations, showing that pairwise annotation can further distinguish responses with similar likert scores. This dataset, together with its counterpart HelpSteer2, enables a head-to-head comparison between Bradley-Terry and Regression style reward models, by training models with the same prompts and responses in HelpSteer2 and only differing the feedback annotation. Using insights from this comparison, they propose a novel approach combining both methods, achieving superior reqward modeling performances. They also conduct experiments with RL algorithms to demonstrate the effectiveness of the reward models.

strengths:
1. The paper introduces a carefully annotated preference dataset that could be useful. The authors also conduct some analysis on the difference of pointwise and pairwise annotation.

2. This paper compares Bradley-Terry and Regression style reward modeling paradigms, which motivates them to design a new method to make the best of both worlds.

weaknesses:
1. The motivation is somehow overclaimed. In line 013-016, the authors claim that there lacks datasets to enable the comparison between Bradley-Terry and Regression style reward modeling. However, all preference datasets with point-wise scores can be converted into pair-wise setups, which should have been able to allow the comparison between the two paradigms. But I agree with authors that "Regression-style data collection (where responses are rated individually) gives
annotators a different set of expectations for comparative ranking of responses" (line 055-056), so it'd be better to have a pair of dataset directly annotated in point-wise and pair-wise schema. Admittedly, the comparison setup is improved now but it was not totally undoable.

2. The authors conducted a lot of experiments, but it's hard for readers to parse useful information from the experiments with almost no highlights in writings. The abstract mainly focuses on what they did, with no mention on what they have found in their experiments. In the experiment sections, readers would benefit a lot from bolded high-level excerpts to quickly grasp the key findings. 

3. The results are good to know, e.g. the comparison between Bradley-Terry and Regression style reward modeling, but personally I don't find it surprising or informative enough to serve as the main claim of a conference paper. It would better to investigate deeper what properties of an algorithm make a good reward model. Also, the experiments are mainly conducted on general chats with subjective metrics (LLM-as-a-judge), thus the conclusions may not be able to generalize to other taks like reasoning, which is known to be very different from chatting. Regarding other experiments, such as section 5, the authors conduct RL with their reward models and datasets mainly for the sake of demonstrating their usefulness. In this case, it seems baselines are needed so that readers can be better aware of the necessity of another preference dataset, given the existence of many good alternatives, and distinguish this dataset with existing ones.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730619399274

thread_type:
official_review

<thread_object>
creator: reviewer_kjd8

<content>
summary:
This work addresses two major paradigms: Bradley-Terry (BT) and Regression style models, examining which is better when matched for data. The main challenge in previous studies was the lack of comparable datasets collected using both methodologies. To resolve this, the authors introduce the HelpSteer2-Preference dataset, which complements existing ratings with preference annotations, including human-written justifications. This dual-annotation allows a head-to-head comparison between BT and Regression models under controlled data conditions.

strengths:
First, it complements existing ratings with detailed preference annotations, allowing for more nuanced training of reward models. By collecting preference data alongside Likert-scale ratings, the dataset enables direct comparisons between models trained under different paradigms (e.g., Bradley-Terry vs. Regression), which was previously difficult due to incompatible data formats in existing datasets. This dual approach provides a richer training ground for testing and developing models that need to evaluate comparative quality.

Second, the dataset includes human-written justifications for preferences, adding a layer of interpretability. This is particularly valuable for training models that can both assess and explain their choices, supporting applications where transparency is important, such as in human-in-the-loop systems. The justifications make the dataset unique by facilitating the development of models that are not just performance-driven but also capable of detailed decision-making explanations.

Third, the dataset maintains high-quality standards, demonstrated by its filtering processes and the steps taken to ensure strong inter-annotator agreement. By curating data that passes rigorous quality checks, the authors improve the signal-to-noise ratio, making it more effective for training robust models. This attention to quality addresses common issues in training data, where noisy or inconsistent annotations can limit model performance.

Overall, the HelpSteer2-Preference dataset's strengths lie in its dual-format data collection, the inclusion of human justifications for enhanced interpretability, and its commitment to high-quality, reliable annotations. This makes it a valuable resource for advancing research in reward modeling and instruction-following alignment in language models.

weaknesses:
While the HelpSteer2-Preference dataset brings several innovations and benefits, it also has certain weaknesses that could impact its applicability and use in research and development:

First, the dataset's format, which includes both ratings and preferences, may require more complex preprocessing and handling by researchers. The dual data structure means that models need to be capable of leveraging both individual ratings and comparative preferences, which could increase the implementation complexity for teams that are not equipped with extensive data processing expertise.

Second, despite the inclusion of human-written justifications enhancing interpretability, these justifications may introduce variability due to subjective writing styles or annotator biases. This subjectivity can make it challenging to maintain consistency across annotations and could potentially affect the model's ability to generalize when trained with such data. Additionally, training on justifications may require more advanced model architectures that can process and utilize free-form text, which may not be available or practical for all research teams.

Third, while the dataset emphasizes high-quality annotations with steps taken to filter out low-confidence or inconsistent data, this strict filtering reduces the overall volume of usable data. The exclusion of samples with high variability among annotators may result in a smaller training set, potentially limiting model performance, especially for models that benefit from a large and diverse dataset. This trade-off between quality and quantity could be a bottleneck for researchers who need extensive data for training large-scale models.

Lastly, the HelpSteer2-Preference dataset is tailored to specific types of model training and evaluations, such as Bradley-Terry and Regression-based reward models. This specialization may limit its generalizability to other types of machine learning tasks or applications that do not align with these paradigms. Researchers working on models with different objectives or those requiring more diverse data formats might find the dataset less adaptable.

questions:
1. How were the annotators trained to provide consistent ratings and justifications, and what measures were in place to minimize subjective bias?
2. How well does the dataset generalize to real-world applications beyond reward modeling and instruction-following?
3. What is the impact of filtering out samples with high annotation variability on the overall representativeness and diversity of the dataset?
4. Would models trained on a smaller, higher-quality dataset perform better than those trained on larger datasets with more noise?
5. How might the dataset be expanded or adapted to cover other dimensions of reward modeling, such as domain-specific use cases or more complex types of preferences?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730689685634

thread_type:
official_review

<thread_object>
creator: reviewer_yvas

<content>
summary:
This paper presents HelpSteer2, a dataset designed to facilitate research in RM for aligning language models with human preferences. This work addresses the gap of fairly comparison of BT and regression style of RM by releasing preference annotations compatible with the BT style alongside existing ratings formatted for Regression-style training. The authors then conduct a head-to-head comparison of the BT and Regression models when matched for data quality, providing valuable insights into the effectiveness of each approach. They also propose a hybrid model that combines elements of both BT and Regression approaches, achieving state-of-the-art performance on the RewardBench benchmark. Additionally, the authors open-source the HelpSteer2 dataset and trained reward model to promote further research in RM alignment tasks.

strengths:
- The authors have created and open-sourced the HelpSteer2 dataset, which includes high-quality preference annotations and justifications specifically designed for reward model (RM) training.
- The paper conducts a detailed, head-to-head comparison between Bradley-Terry (BT) and Regression-style reward models, providing insights into the relative strengths and weaknesses of each approach.

weaknesses:
- While the hybrid approach is innovative, it seems to be an incremental improvement rather than a fundamentally new modeling technique. 
- Lack of statistical significance testing to validate the effectiveness of proposed hybrid modeling method.
- The paper does not evaluate the aligned models on standard language modeling tasks, which would help in quantifying the alignment tax.

questions:
- Could the authors provide statistical significance testing to validate the reported performance improvements of the hybrid model? This would strengthen the robustness of the conclusions.
- Have the authors considered evaluating the align models on standard language modeling tasks to assess the alignment tax? 
- Based on the appendix, it appears that the RL training involved approximately 10k examples, and the model was trained for fewer than 100 iterations. Could the authors clarify the stopping criteria for the training process? Additionally, could you specify the KL divergence value of the choosen checkpoints from the reference policy? This information would provide better insight into the training stability and convergence behavior of the reward model.
- Are the conclusions valid for cases where, e.g, 5 responses ranking are used for RM training?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707820419

thread_type:
official_review

<thread_object>
creator: reviewer_exue

<content>
summary:
This paper explores whether a Bradley-Terry or regression approach is more effective for reward modeling and proposes a novel method that combines both styles. The conclusion is intriguing: the format in which data is collected and the specific model training approach do not significantly impact the results.

strengths:
This paper provides an excellent dataset that will be highly valuable to the community. It addresses an important question: whether the Bradley-Terry style or regression style is more effective for reward modeling. Additionally, the reward model (RM) proposed here is capable of effectively training a LLaMA-70B model.

weaknesses:
I'm glad to see that the BT training method with an added margin (eq.2) is effective. However, this finding somewhat conflicts with the results in the Llama 3 paper, where the authors mention: "The training objective is the same as Llama 2, except that we removed the margin term in the loss, as we observed diminishing improvements after data scaling." What could be the reason behind this discrepancy?

I understand that this work provides an excellent dataset and rigorous experimental validation. However, I would like to see more analysis, such as:

1. A comparison of results on out-of-distribution prompts.

2. A consideration of annotation costs: since preference annotations are likely cheaper and easier to scale than Likert scores, a fair comparison might require factoring in annotation costs rather than simply matching dataset sizes.

3. An analysis of robustness to annotation errors. For instance, assume that preference annotations have a 5% chance of being reversed, or that Likert scores have a similar error rate (where a 9 might be incorrectly marked as a 1, rather than just slightly off, like 8). Which reward modeling approach is more robust to such errors?

These aspects would provide valuable insights into the practical implications of each approach.

questions:
Has a prompt filter been applied to ensure that prompts from the rewardbench are not included in the training set?

Additionally, how does SteerLM regression, initialized with the Bradley-Terry model, perform?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732155792068

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comment to All Reviewers

comment:
Dear Reviewers,

We would like to thank everyone for the valuable feedback. We would like to draw the reviewers attention to new results, and re-emphasize the significance of the results we achieve with the REINFORCE model.

**New Results on LMSYS Chatbot Arena**

Due to the strength of our REINFORCE model on automatic benchmarks (i.e. outperforming GPT-4o and Claude 3.5 Sonnet on Arena Hard, AlpacaEval 2 and MT Bench), we evaluated our model in the LMSYS Chatbot Arena. After receiving thousands of double-blind votes comparing this model’s response to the response of other models to user-provided prompts, the model ranks ahead of several frontier models  including Claude 3.5 Sonnet 20240620, GPT-4o 2024-08-06, and Llama 3.1 405B, despite being only a 70B-parameter model.

**The Significance of Our Contribution**

We are glad to see that the reviewers value our methodological work. However, none of the reviewers have explicitly recognized one of our most important contributions: providing an open-source dataset (with an enterprise-friendly license) that allows the community to train aligned models that can match the performance of frontier models on chat benchmarks (both automatic and human-evaluated), starting with a publicly-accessible Llama-3.1-70B-Instruct model. We believe our provided dataset can greatly support the research community to train and study powerful open-source language models. We therefore kindly ask the reviewers to re-evaluate our contribution in light of this.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732913674469

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Follow-up General Comment to All Reviewers

comment:
Dear Reviewers, 

Thank you again for your detailed reviews. As the discussion period only has a few days left, we are wondering if you have any further comments or questions for us to consider. 

If our rebuttal (adding more experiments, analysis, and writing) helped to alleviate your initial concerns, we would appreciate it if you would consider raising your scores.
</content>
</thread_object>

</thread>


Metareview:
metareview:
## **Summary**
This paper introduces the **HelpSteer2-Preference** dataset, which extends the existing HelpSteer2 dataset by including pairwise preference annotations alongside Likert-5 style ratings. This addition allows for a head-to-head comparison of Bradley-Terry (BT) and Regression-style reward modeling paradigms. The authors further propose a novel method that combines insights from both approaches, yielding state-of-the-art (SOTA) results on RewardBench and strong performance in alignment tasks using reinforcement learning with human feedback (RLHF). The dataset and trained models are open-sourced under a permissive CC-BY-4.0 license, promoting reproducibility and practical use.

## **Strengths**
1. **Data Contribution**: The HelpSteer2-Preference dataset addresses an existing gap by providing data that allows for a fair comparison between BT and Regression-style models, a challenge due to the different data collection formats typically used by these paradigms.
2. **Modeling Insights**: The authors conduct a detailed analysis of the three BT variants (Regular, Margin, and Scaled), identifying Scaled BT as the most effective. They also demonstrate that combining BT with Regression-style approaches leads to better performance.
3. **Empirical Results**: The models trained using HelpSteer2-Preference achieve SOTA performance on RewardBench and outperform strong baselines on alignment benchmarks such as Arena Hard.
4. **Open Sourcing**: The (promised) release of the dataset and trained models ensures accessibility and fosters community-driven advancements.

## **Weaknesses**
1. **Incremental Nature of Contributions**: Some reviewers noted that the hybrid approach, while effective, is not fundamentally novel. The dataset augmentation is viewed as a refinement rather than a solid contribution.
2. **Dependence on HelpSteer2**: While the dataset enhances HelpSteer2, the reliance on its structure could limit its generalizability to other domains or applications. However, the authors provided strong justifications for this reliance.
3. **Evaluation Diversity**: The evaluation focuses on alignment tasks and general chat scenarios, which may not fully generalize to more specialized domains like reasoning or code generation.


## **Recommendations**
The paper demonstrates strong empirical results and provides a valuable dataset that will benefit the community. While the contributions are more practical than theoretical, they represent a meaningful step forward in advancing reward modeling and alignment for large language models. The reviewers generally agree on the rigor of the work, the clarity of the presentation, and its significance for practical alignment tasks.

This work is well-executed, and the dataset is likely to have a broad impact, particularly for researchers working on reward modeling and RLHF. While there is room for further insights, the paper's empirical rigor and open-source contributions make it a strong candidate for acceptance.