RMB: Comprehensively benchmarking reward models in LLM alignment

Authors: Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM Alignment, reward model, evaluation

Abstract:
Reward models (RMs) guide the alignment of large language models (LLMs), steering them toward behaviors preferred by humans. Evaluating RMs is the key to better aligning LLMs. However, the current evaluation of RMs may not directly correspond to their alignment performance due to the limited distribution of evaluation data and evaluation methods that are not closely related to alignment objectives. To address these limitations, we propose RMB, a comprehensive RM benchmark that covers over 49 real-world scenarios and includes both pairwise and Best-of-N (BoN) evaluations to better reflect the effectiveness of RMs in guiding alignment optimization.
We demonstrate a positive correlation between our benchmark and the downstream alignment task performance. Based on our benchmark, we conduct extensive analysis on the state-of-the-art RMs, revealing their generalization defects that were not discovered by previous benchmarks, and highlighting the potential of generative RMs.  Furthermore, we delve into open questions in reward models, specifically examining the effectiveness of majority voting for the evaluation of reward models and analyzing the impact factors of generative RMs, including the influence of evaluation criteria and instructing methods. We will release our evaluation code and datasets upon publication.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730018292665

thread_type:
official_review

<thread_object>
creator: reviewer_qtwd

<content>
summary:
The paper proposes RMB, a comprehensive benchmark for evaluating reward models (RMs) in LLM alignment. The dataset contains comprehensive real-world scenarios on helpfulness and harmfulness and responses generated by different LLMs. The paper validates the benchmark's effectiveness by showing a correlation with downstream alignment performance.

strengths:
- The paper is well-structured and clearly written, explaining the methodology and results effectively.
- RMB effectively addresses the limitations of developing reward models that align with the objective, whether helpfulness or harmlessness. Correlation evaluations show that the reward model's performance on RMB can reflect the performance of the downstream aligned model more accurately. This would enable researchers to evaluate and iterate on reward models more efficiently before training a more computationally challenging model alignment.
- The paper provides solid empirical evidence supporting the effectiveness of RMB. The experiments on pairwise and best-of-N accuracy cover different generative and discriminative reward models and provide a good insight into their usefulness as judges.

weaknesses:
- The reliance on LLM-generated response in the dataset curation may pose potential long-term limitations. As reward models are often used to align newer LLMs, using responses from current LLMs to evaluate them could create a circular dependency. In other words, this benchmark may not be able to differentiate reward models that could guide LLMs that are beyond current capabilities. In the next iterations, it would be nice to incorporate human-generated responses to ensure more diverse sources of responses.
- The correlation evaluation is leveraging best-of-N as an alignment, which is not the only way people are using the reward models. Would it be possible to gather a set of reward models and aligned model pairs that leverage the reward model during training and see the correlation there?
- nit: Mu et al.. citation is missing a year (L229)

questions:
See Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730664033470

thread_type:
official_review

<thread_object>
creator: reviewer_nun4

<content>
summary:
The paper proposes RMB (Reward Model Benchmark) which is a benchmark for evaluation of reward models used in LLM alignment. It consists of 49 real–world scenarios between the two proposed categories “helpfulness” and “harmlessness”. They use two methods of evaluation: pairwise comparison and best-of-N evaluation, which tests the models’ ability to choose the best response from multiple candidates. Another key finding is that generative models show strong performance, often outperforming discriminative models. Further, they show positive correlation between benchmark performance and downstream alignment tasks.

strengths:
RMB presents many strengths, especially against its main predecessor, RewardBench. Primarily, the dataset size is much larger than any similar datasets used to evaluate reward models. The paper does thorough investigation of best-of-N evaluation, which related papers had proposed as a future direction of research. The authors do a good job of categorization and subcategorization, to ensure broad coverage across different tasks. The benchmark shows good progress towards a useful proxy for downstream model performance. Further, the paper is clearly written and has good formatting.

weaknesses:
There are a couple of weaknesses in the paper. The human annotator sample size is small relative to the size of the dataset. The correlation analysis focuses mainly on the best-of-N sampling rather than RLHF, which is acknowledged in the paper as a limitation. While mentioning that it is hard for a model to be both competitive in judging helpfulness and harmlessness, the authors don’t deeply explore the trade-offs between the two metrics. The authors used various models in the categorization and filtering steps, but the scoring was only done by GPT4, which might introduce bias. There are also a few styling errors and typos throughout the paper that would need to be cleaned up prior to approval.

questions:
- Were any alternatives to GPT-4 scoring considered? How might the benchmark differ if using other models (or ensembles) or human annotators? (I understand human annotation is costly) 

- Did you observe any patterns when best-of-N evaluation diverged significantly from pairwise evaluation?

- What recommendations would you make for effectively using RMB during the reward model development process? Is it simply meant to be used as a validation set? 

- What tradeoffs between helpfulness and harmlessness exist? Why do you think they seem to be mutually exclusive? 

- Do you see any negative impacts or possible misuses of the benchmark?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730734901679

thread_type:
official_review

<thread_object>
creator: reviewer_n8sv

<content>
summary:
The paper presents RMB, a benchmark evaluating reward models (RMs) across 49 real-world scenarios, improving alignment assessments for LLMs. RMB reveals generalization issues in current RMs and explores factors affecting generative RM effectiveness.

strengths:
1. The paper is written well and is easy to understand.
2. The studied problem is significant..
3. The results seem to outperform the SOTA datasets of the reward model evaluation.

weaknesses:
1. The paper currently includes some discussion related to benchmark comparisons in Section 5.2, particularly with RewardBench. However, a more explicit comparison of the features and approaches of existing benchmarks early in the paper would better highlight the novelty of this work. Rather than relying on experimental results to convey superior performance, detailing how our model’s capabilities differ from those of previous benchmarks would strengthen the paper's contribution.
2.  In the conclusions about evaluation results, statements like “It is hard for an RM to be both competitive in judging helpfulness and harmlessness” and “Generative models show great promise in reward modeling” could be enhanced by including an explanation of the underlying reasons for these findings. Adding these insights may help the community engage more deeply with the work, offering valuable perspectives that could inspire further exploration beyond the results alone.

questions:
Please refer to the weakness, I'm happy to modify my rate based on the response of the authors and refer to other reviewers' comments.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**(a) Summarize the scientific claims and findings**

The paper introduces RMB (Reward Model Benchmark), a comprehensive benchmark designed to evaluate reward models (RMs) used in aligning large language models (LLMs). RMB includes 49 fine-grained real-world scenarios with two alignment categories: helpfulness and harmlessness. The benchmark employs pairwise and Best-of-N (BoN) evaluation methods, offering nuanced insights into RM performance. Experimental results show RMB’s strong correlation with downstream task performance, and the paper identifies key challenges, such as the trade-off between helpfulness and harmlessness. Additionally, it highlights the promise of generative RMs over traditional discriminative approaches.

**(b) Strengths of the paper**

1. **Novelty and Scope**: RMB addresses gaps in existing benchmarks, offering broader task coverage, advanced evaluation paradigms, and nuanced harmlessness testing.
2. **Comprehensive Evaluation**: It uses both pairwise and BoN evaluations, with BoN correlating better with downstream alignment tasks.
3. **Insightful Analysis**: The paper explores trade-offs between alignment objectives and evaluates generative RMs, providing actionable insights for future work.
4. **Strong Empirical Evidence**: Extensive experiments validate RMB’s effectiveness and its correlation with downstream alignment task performance.
5. **Clear Presentation**: The paper is well-written, with a clear structure and detailed explanations.

**(c) Weaknesses of the paper**

1. **Human Annotation Sample Size**: The reliance on 200 human-annotated samples is a limitation, though mitigated by validation and iterative improvements using a golden set.
2. **Potential Bias from GPT-4 Scoring**: While the authors provided evidence of alignment with human judgments and validated their approach with alternative methods, reliance on a single model remains a potential issue.
3. **Limited RLHF Experiments**: The paper focuses on BoN rather than full reinforcement learning fine-tuning (RLHF) due to resource constraints, which may limit its generalization to other RM applications.
4. **Circular Dependency**: RMB relies on responses from current LLMs, which could create challenges in evaluating reward models for future LLM capabilities.

**(d) Provide the most important reasons for the decision.**

The paper addresses an increasingly important research problem in the community: it provides a significant contribution to the evaluation of reward models in LLM alignment through a novel benchmark and rigorous empirical analysis. Despite some limitations, the strengths, particularly in the importance of the problem and the shared insights, justify its acceptance.