RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style

Authors: Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, Juanzi Li
Primary area: datasets and benchmarks
Keywords: Reward Models, Language Models, Evaluation, Alignment

Abstract:
Reward models are critical in techniques like Reinforcement Learning from Human Feedback (RLHF) and Inference Scaling Laws, where they guide language model alignment and select optimal responses. 
Despite their importance, existing reward model benchmarks often evaluate models by asking them to distinguish between responses generated by models of varying power. 
However, this approach fails to assess reward models on subtle but critical content changes and variations in style, resulting in a low correlation with policy model performance.
To this end, we introduce RM-Bench, a novel benchmark designed to evaluate reward models based on their sensitivity to subtle content differences and resistance to style biases. 
Extensive experiments demonstrate that RM-Bench strongly correlates with policy model performance, making it a reliable reference for selecting reward models to align language models effectively.
We evaluate nearly 40 reward models on RM-Bench. 
Our results reveal that even state-of-the-art models achieve an average performance of only 46.6%, which falls short of random-level accuracy (50%) when faced with style bias interference.
These findings highlight the significant room for improvement in current reward models.

Decision: accept (oral)
Reviewer scores: 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1729318505566

thread_type:
official_review

<thread_object>
creator: reviewer_9kqk

<content>
summary:
This paper introduces a benchmark for reward models that is designed to be robust to subtleties in generated text. Previous reward model benchmarks have used weaker models to generate rejected responses, and stronger models for the chosen responses. Such a benchmark could be exploited by a reward model that prefers properties of responses unrelated to what the benchmark is intended to evaluate, such as the style of text.

Instead of using stronger and weaker models, RM-Bench uses one model to generate the chosen and rejected responses, where the rejected responses are flawed in some way (e.g., they were generated by a model that had previously been jailbroken). Style is also controlled via prompting, and the benchmark covers three styles (concise, detailed and markdown). The accuracy of RM-Bench is evidenced by showing correlation between reward models that score higher on RM-Bench and improved performance of policy models on evaluations such as GSM8k and HumanEval+.

The main takeaway is that even the best publicly available reward models have imperfect performance on RM-Bench, so there is plausibly room for significant improvements in reward modeling. The paper also shows that DPO models outperform some sequence classifiers on RM-Bench, and that the importance of style has likely gone underappreciated in reward model evaluation.

The Appendix gives evidence that RM-Bench is more accurate than prior work by showing the weaker correlation for benchmark performance of the policy model and performance on Reward Bench.

strengths:
**Originality**: I'm not aware of prior work that benchmarks reward models while controlling for style. Similar ideas have been discussed generally in RL, but not used to design better reward model benchmarks. To the best of my knowledge the contributions are original.

**Quality**: The method is sufficiently rigorous for the results to be reproduced. The way the paper evaluates the accuracy of RM-Bench is sound, and the three main takeaways from the paper are well evidenced. For example, Appendix K shows the authors covered many popular publicly available reward models, which supports the claim that progress in reward modeling is necessary.

**Clarity**: The paper is clearly written and the figures are not confusing. The authors introduce the problem, background and their contributions well.

**Significance**: I find the results significant. They point to what seems to be an important issue in a previous evaluation and improve upon it by mitigating that issue. They then show the improved benchmark is correlated with downstream policy model performance. The other takeaways of the paper are also significant in their own right, such as benchmarking the reward modeling of models trained with DPO, but these are of course less significant than the main contribution of the paper.

weaknesses:
- As far as I could tell the exact data used in the paper is not available. It would be good to have a 'canonical' version of RM-Bench on HuggingFace or somewhere similar to make reproducing the results easier.
- The paper does not show that the reason RM-Bench is more accurate is because you control for subtleties in the text like style. Although this seems likely, showing evidence that this isn't because of other differences between RM-Bench and Reward Bench (e.g. by ablating the control for style bias) would increase my soundness score to a 4.
- It would be interesting to see the correlations separately for each of the benchmark types (math, code, safety).

questions:
Do you plan to have a version of the data you used in your experiments up on HF for the camera-ready version of the paper?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730392278584

thread_type:
official_review

<thread_object>
creator: reviewer_rbmr

<content>
summary:
This paper introduces a new benchmark, called RM-Bench, to evaluate reward models. The authors state that an ideal reward model should be able to identify subtle differences in content quality (correctness) and consistently reward better responses, regardless of stylistic variations. They argue that existing benchmarks, which rely on comparing responses generated by LLMs of differing power, fail to fully capture these aspects of reward model evaluation.

RM-Bench addresses this by constructing a dataset where both chosen and rejected responses are generated by the same LLM, gpt-4o (except for safety, where responses are generated using different models to ensure safety violations). They use (i) domain-specific techniques to introduce subtle errors to the rejected responses, and (ii) prompting the LLM to generate response variants with differing levels of detail and markdown formatting. Their main contribution is the introduction of the style-substance evaluation matrix and the corresponding {easy, normal, hard} accuracy metrics that allow a more granular analysis of how style influences reward model predictions.

By evaluating a wide range of reward models on RM-Bench, they demonstrate that a wide range of reward models have a style bias (a high easy accuracy but a low hard accuracy). They also empirically show that the performance of a reward model on RM-Bench (moderately) correlates with the performance of the resulting aligned policy model (under style-controlled evaluation and across downstream tasks).

strengths:
The use of style-controlled generation of responses for building the evaluation dataset is a simple but structured and original idea to directly assess the style bias in reward models. It is generic in the sense that it can also be applied to existing datasets with verbose responses (or the model can be asked to alter the response such that they are more verbose and with formatting). Style-substance evaluation matrix enables to quantify the degree of the bias and its impact on reward predictions.

The experiments are comprehensive and demonstrate that it is a challenging benchmark for the current reward models, exposing their weaknesses with respect to style bias and robustness. This is a significant result (but may be of limited scope; see below). The performances of the aligned policy models, measured by their style-control scores and how well they perform in downstream tasks, correlate positively with the performance of the corresponding reward models on RM-Bench. This correlation, while not strong, is nonetheless better than that observed with RewardBench, which does not explicitly focus on style bias and relies on preference datasets with responses that are human curated or generated using a strong and a weak LLM pair. However, there may be multiple parameters in play here and a clarification may be needed.

weaknesses:
In the paper, the authors construct the RM-Bench dataset in two steps: (i) generate domain specific preference pairs as described in sections 3.1-3.3, and (ii) add style-controlled variants of these pairs as described in section 3.4. Adding style-controlled variants (step (ii)) is essential to understand the style bias and the main focus of the paper, but its contribution to (average) accuracy and the correlation with the performance of the aligned policy models is unclear. In Table 3, we can see that the normal accuracy is close to the average accuracy, which may hint that accuracy based on (y_c^{L,M}, y_r^{L, M}) pairs alone could be be a good indicator. To better understand the role of the style-controlled generation, the authors should provide these numbers and compare the results in sections 4.2 (DPO) and 5 (correlation with the policy model) with RM-Bench performance based on y^{L, M} only.

In section 4.3, the correctness and verbosity scores of the examples (from Nemotron-4-340B-Reward model) are conditioned on the prompt. The scores of responses with different prompts may not be comparable. This makes it difficult to interpret the presented results. What is the behavior of the model when you use the accuracy r(y_c) > r(y_r) is used for both correctness and verbosity scores instead?

questions:
- In section 3.1, after human filtering the number of examples with subtle errors drop significantly (519->183). Is this mainly due to chosen responses (generated by gpt-4o) not being correct or many-shot jailbreak technique failing to produce responses with injected factual errors? Can you elaborate more on the effectiveness of the approach?

- (section 3.2) HumanEvalPack contains both the correct and buggy (human written) solutions. Does using multiple-sampling to generate rejected samples bring an advantage over using the existing ones when assessing the performance of the reward models for the corresponding domain?

- (section 3.4) What are the prompts used to generate the plain text and concise prompts?

- (section 4.2) How does the DPO model perform when you use the hard instead of average accuracy? The DPO paper initializes the reference policy by maximizing the likelihood of preferred completions when the reference model is not available. Setting \pi_{ref}(y|x) = 1 assumes a uniform distribution over all possible responses for the reference model. You may want to compare with the former as well.

- Section I: The instruction asks the model to introduce one error, but there are examples with several errors. Could this be causing over filtering?

- The natural approach to improve the performance of a reward model in RM-Bench would be to augment the training (preference) dataset of the reward model using style-controlled generation as explained in section 3.4 (with proper prompts). We expect the model to have a higher (hard) accuracy and a higher score in this case. However, this may not lead to an improvement in the performance of the aligned policy model that uses such a reward model (hacking the benchmark instead). In section 5.1, you show a moderate positive correlation, but the reward models there do not explicitly take style bias into consideration. Could you please elaborate on this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666303914

thread_type:
official_review

<thread_object>
creator: reviewer_szz1

<content>
summary:
In this submission, the authors have introduced RM-BENCH, a benchmark for evaluating reward models that: 1. evaluates reward models' sensitivity to subtle content differences and 2. tests resistance to style biases like length and markdown formatting.

strengths:
- The paper is well-motivated and offers various prompts to explicitly control for style bias, in contrast to the RewardBench benchmark.
- It also establishes a clear methodology for evaluating biases, which could be extended to other categories or used to add tasks focused on reasoning and safety. In short, the overall methodology appears scalable, though it may require some human-level judgment.

weaknesses:
I feel that the paper in its current state has a lot of scope for improvement, and hence I am leaning towards a borderline reject. I might be willing to increase my score after the rebuttal period with the authors if they are able to improve upon the following areas:

- Currently, the benchmark relies heavily on gpt4o. I am aware that the authors discussed this in the Appendix's Limitations section, but I strongly feel that, to make the benchmark ready for deployment, it should be diverse and include responses from more SOTA models like Claude, Gemini, etc.
- The results shown in Table 4 and the correlation analysis in Section 5 and the Appendix rely on the Tulu-v2.5 model. I would like to see the same analysis on at least one more model (maybe from Qwen or LLaMA family) before drawing conclusions on correlation analysis.
- While the authors explain that they use gpt4o to create different versions of the same content (short vs. long, with and without special formatting), they do not provide enough details about how exactly they do this or how they ensure that the different versions maintain the same meaning. These missing details about their exact process—such as the specific prompt given to gpt4o for summarizing responses or removing markdown, or how they check the quality of generated responses (after removing markdown)—should be included in the Appendix.
- I would also like to see more details on length control in Table 2. For example, what are the target length ranges for each category when selecting normal vs. long responses, and how do you ensure that, for concise responses (both correct and rejected), the model retains the core information (fact or code snippet) even after summarization?

questions:
1. How much overlap is there between the samples from RM-Bench and RewardBench?
2. On Page 8, Lines 426–431, the authors state that “the reward model fails to detect subtle differences between chosen and rejected responses, which causes significant overlap in the scatterplot.” However, as seen in Table 2, samples from Math and Code tasks have nearly identical average tokens for short and long responses (for both correct and rejected ones). So, wouldn’t one expect significant overlap in the plot? I think the authors should clarify the minimum, maximum, and median lengths of samples across all tasks and style types.
3. In Figure 4, what do the four data points signify? Do they represent models trained on four different preference datasets? If so, the authors should clarify which data point corresponds to which preference dataset.
4. Related to point 3 above: Figure 4 assesses Chat Hard accuracy. If the preference data is StackExchange, then I would also like to see a similar graph with the x-axis as Code Hard or Code Normal accuracy. Additionally, I would like to see a similar graph on one more base language model besides Tulu to verify the claims made.
5. In Figure 8 of Appendix H, does the y-axis represent score or relative accuracy?
6. Typos:
    1. Page 4 Line 180: “LLM we used here is” → “The LLM that we used here is”
    2. Page 4 Lines 182-183: A sentences seems to be partially omitted. Please clarify what the authors are trying to convey there.
    3. Page 10 Line 504: “It provides” → “They provide”
    4. Page 10 Line 503: “generally used to” → “generally used ~~to~~”
    5. Page 15 Line 777: “Border Impacts” → “Broader Impact”
    6. Page 15 Line 786: “build” → “built”
    7. Page 16 Line 815: “open-end” → “open-ended”
</content>
</thread_object>

</thread>
<thread>
creation date:
1732274889442

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewers

comment:
We are grateful to the reviewers for their detailed and constructive feedback on our submission.

We are pleased that all reviewers **unanimously recognized the originality and significance of our work**. To the best of our knowledge, our benchmark, RM-Bench, is **the first** systematic evaluation framework designed to assess reward models' **sensitivity to subtle substance changes** and **robustness to style bias**. 

Compared to prior work such as RewardBench, we incorporate both **substance control** and **style control**, and introduce a **Style-Substance Evaluation Matrix** to comprehensively assess reward models.

Additionally, all reviewers **agreed that our framework effectively highlights the shortcomings of existing reward models in these two critical aspects**.

Recent efforts, such as Style-Controlled Chatbot Arena [1] and Length-Controlled AlpacaEval [2], emphasize style control in evaluating policy models to enhance policy models' helpfulness and capability, rather than prioritizing stylistic appeal over substance.

A critical yet overlooked issue is the reward model, which guides policy model training. Failures to detect substantive errors or style bias in the reward model can propagate to policy models, leading to superficial outputs that compromise utility.

Despite its importance, the sensitivity of reward models to substantive errors and their robustness to style bias remain underexplored. 
We believe that our work serves as a strong starting point to fill this gap, and we are excited to see the community’s response to our contribution.

[1] *Does Style Matter? Disentangling Style and Substance in Chatbot Arena.* https://lmsys.org/blog/2024-08-28-style-control/  

[2] *Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators.* https://arxiv.org/abs/2404.04475  

---

We address a common concern below:

> Prompt for Style Control (Reviewer SzZ1, RBmr)

We thank the reviewers for their suggestions regarding prompts for removing markdown tags and generating concise responses. 
We have included these details in **Appendix L** in the revised version of our paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a new benchmark for reward modeling that is designed to be robust to subtleties in generated text. In previous benchmarks, rejected answers were usually generated by weak models, which could be easily exploited.

Strengths:  
This paper studies an important problem with novel solution ideas. The results provide valuable new observations to the community, particularly regarding DPO models as reward models.

Weaknesses:  
The initial version of this paper relied heavily on GPT4o, and many results were conducted on only one model series. However, the authors addressed these concerns by adding additional experimental results during the rebuttal period.

Overall, I think the results of this paper are interesting and significant.