Evaluating Robustness of Reward Models for Mathematical Reasoning

Authors: Sunghwan Kim, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Jungsoo Won, Dongha Lee, Jinyoung Yeo
Primary area: datasets and benchmarks
Keywords: mathematical reasoning, RLHF, reward models, reward overoptimization, language models, benchmark

Abstract:
Reward models are key in reinforcement learning from human feedback (RLHF) systems, aligning the model behavior with human preferences.
Particularly in the math domain, there have been plenty of studies using reward models to align policies for improving reasoning capabilities.
Recently, as the importance of reward models has been emphasized, RewardBench is proposed to understand their behavior.
However, we figure out that the math subset of RewardBench has different representations between chosen and rejected completions, and relies on a single comparison, which may lead to unreliable results as it considers only an isolated case.
Therefore, it fails to accurately present the robustness of reward models, leading to a misunderstanding of its performance and potentially resulting in reward hacking.
In this work, we propose a direction for designing benchmarks that reliably evaluate reward models in mathematical reasoning.
We conduct comprehensive analyses to validate whether our design effectively reflects the robustness of reward models.
The results underscore that the benchmark designed to reduce the possibility of reward hacking and employ one-to-many comparisons strongly correlate with the results of optimized policy, whereas the existing benchmark shows almost no correlation.
Furthermore, by analyzing through the lens of reward overoptimization, we show that the design involving multiple comparisons results in a significantly more reliable benchmark.
We make our code and data publicly available.

Decision: reject
Reviewer scores: 6, 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730373685466

thread_type:
official_review

<thread_object>
creator: reviewer_ttub

<content>
summary:
The paper addresses a specific limitation of RewardBench, a widely used benchmark for reward model evaluation, in its assessment of mathematical reasoning capabilities. To this end, the authors introduce RewardMATH, a new benchmark that employs one-to-many comparisons of chosen and rejected responses to mathematical questions to enhance evaluation robustness. Experiments show that RewardMATH correlates well with policy performance and is more effective at identifying potential reward overoptimization and the reliability of reward signals.

strengths:
1. The paper identifies a particular limitation of RewardBench, a popular benchmark for evaluating reward models, in assessing mathematical reasoning and introduces a new benchmark that addresses this issue by including one-to-many comparison data.
2. The authors provide extensive experiments and analyses across different reward model types, including both proprietary and open models, and assess various performance metrics.

weaknesses:
1. The benchmark comparison primarily involves RewardBench, which is designed to evaluate reward models more holistically across various domains. However, is the comparison in terms of mathematical reasoning appropriate, given that RewardMATH is specifically designed for this purpose? If RewardBench is indeed the most comprehensive eval set even for mathematical reasoning tasks prior to RewardMATH, it would be helpful to clarify this point.
2. The benchmark appears to lack cases where reward models must distinguish between correct solutions of varying quality, such as those missing reasoning steps. It is also unclear whether 500 samples is sufficient to cover diverse mathematical reasoning tasks.
3. Tables 1 and 2 report performance comparisons of various LLMs on RewardBench and RewardMATH. The results seem to merely suggest that the two benchmarks differ significantly. Can we conclude from these results that "high scores on RewardBench do not guarantee **robustness** in reward models"?

questions:
Q. How much of the relatively poor results for RewardBench is due to the noisy annotations inherited from PRM800K, as mentioned in Sec. 3.1? In other words, could simply fixing these annotations significantly change the comparison?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730484120244

thread_type:
official_review

<thread_object>
creator: reviewer_k97g

<content>
summary:
The paper introduces REWARDMATH, a benchmark for evaluating reward models in mathematical reasoning tasks, arguing that it provides a more reliable evaluation than the existing RewardBench by using one-to-many comparisons instead of pairwise comparisons. The authors validate their approach by showing correlation between benchmark performance and best-of-n sampling results.

strengths:
The paper provides a good correlation study between the proposed benchmark and best of N downstream performance in datasets like MATH and Gaokao etc. The proposed one-to-many comparison seems to be on the right direction for better correlation with downstream performance.

weaknesses:
I have the following major concerns:

1. Technical Contribution & Novelty: The primary contribution seems to be replacing pairwise comparisons in reward bench with best of N comparisons, which is an incremental modification rather than a substantial methodological advance. I do not think the change only is sufficient for a publication in top machine learning conference. The correlation between N-way comparison performance and best-of-N sampling results is somewhat expected and doesn't provide deep insights into reward model behavior. 


2. Unclear Definition of Robustness: The paper uses "robustness" throughout but never provides a precise definition. The authors seem to equate robustness with correlation to best-of-n sampling results, but this is circular reasoning since the benchmark itself uses similar methodology. There's no clear framework for what properties a "robust" reward model should have beyond empirical correlation with certain metrics. 


3. Limited Experimental Validation: The paper relies heavily on correlation with best-of-n sampling as validation, but doesn't explore other important aspects of reward model behavior. To make the paper deeper and broader, it would be great if the authors could try comparing the correlation of different real down-stream fine-tuning techniques like BoN SFT, DPO, PPO etc. and see whether how RewardBench and RewardMath correlate with downstream performance there. It would also be interesting to see if such observation extends to other domain like coding, and perhaps even open-ended generations without ground truth.

questions:
Why do the authors choose 10 generations with 9 incorrect and 1 correct answer in the k-way comparisons? How does the choice of k and numbers of correct and incorrect answers affect the resulting correlations and reward overoptimization? The relationship between reward overoptimization and the proposed benchmark needs more rigorous analysis.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730530201600

thread_type:
official_review

<thread_object>
creator: reviewer_bzte

<content>
summary:
The paper proposes a new benchmark, REWARDMATH, to improve the robustness evaluation of reward models in mathematical reasoning tasks. It highlights limitations in the existing RewardBench benchmark, which relies on single comparisons between chosen and rejected solutions, potentially leading to reward hacking and misjudgments of model robustness. REWARDMATH addresses this by using one-to-many comparisons with multiple incorrect solutions to better capture robustness and reduce the risk of reward over-optimization. Experimental results indicate that scores on REWARDMATH strongly correlate with policy optimization outcomes and provide a more reliable measure of reward model robustness compared to RewardBench. This benchmark aims to enhance the development and reliability of RLHF systems, with the findings underscoring the potential of REWARDMATH to serve as a trustworthy evaluation tool in this domain​.

strengths:
1. **Thoroughness**: The paper presents detailed implementations, including training hyperparameters and experimental protocols. This ensures that other researchers can accurately reproduce the experiments and validate the findings. 
2. **Relevance**: This work addresses a critical gap in the field by focusing on reward model evaluation, a crucial area of research that has significant implications for the development of more reliable AI systems.
3. **Motivation**: The paper presents a compelling critique of the existing Reward Bench evaluation metric, establishing a strong foundation for their work. The authors make a persuasive case for developing benchmarks that minimize over-optimization risks, backing their arguments with experimental evidence. This dual focus on improving metrics while addressing practical concerns demonstrates clear motivation for the research.

weaknesses:
1. **Clarity**: The paper is generally well written, however, it has some clarity issues, especially in section 5, which is hard to follow. Clarification questions are asked in the question section, marked with [Clarification]. The authors should address those questions. 
2. **Benchmark Biases**: The paper has several biases, raising concerns on the claimed robustness and reliability. Examples and comments below:

> Line 206: Hence, we first convert the human-annotated solutions from MATH500 into step-by-step machine-generated solutions. We prompt GPT-4, using 4 carefully crafted exemplars for each math subject as part of the prompt.

All correct solutions in the benchmark are generated via GPT-4, raising concerns regarding biases towards GPT-series models. Even though the authors manually inspect the solutions, the solutions were still mainly generated using GPT-4. Notably, the authors observe LLM judges from the GPT-4 Series to perform significantly higher than other models (Line 286), which is likely due to this oversight (since it is known LLM judges tend to bias their own response, eg. GPT-4 family judge favors responses from GPT-4 family). The authors should use a diverse set of LLMs to curate the correct solutions to avoid potential biases.

> Line 805: Secondly, we instruct GPT-4-0125-preview to select a specific step from the correct solution, transform it into an erroneous step, and then prompt again to continue generating the solutions from the erroneous step.

Similar to the previous point, employing GPT-4-0125-preview as editor to insert errors into other LLMs’ answers may introduce biases. Additional validation is needed to ensure the benchmark does not exhibit any bias towards GPT family models. 

> Line 402: We assume Internlm2-7B-reward, which performs well on both RewardBench and REWARDMATH, as the gold RM.

The use of Internlm2-7b-reward as the gold standard lacks sufficient justification and raises several concerns about experimental validity. The author relies primarily on performance metrics from RewardBench and REWARDMATH, but this approach is problematic for multiple reasons. First, the authors themselves criticized RewardBench for containing incorrect ground truth data and failing to adequately assess reward models. Second, using REWARDMATH as a benchmark is circular since it's the very dataset being studied. High scores on these benchmarks alone don't necessarily indicate that a reward model can reliably approximate human preferences. To establish Internlm2-7b-reward as a legitimate gold standard, the author should conduct additional validation studies specifically demonstrating its ability to align with human judgment on mathematical tasks.

> Line 426: We find that proxy reward models trained on smaller datasets reach peak rewards at lower KL divergences, indicating faster over-optimization.

The author assumes KL divergence adequately captures optimization degree without proper justification. KL may not account for other important aspects of policy change. Further study will strengthen the experimental results. 

3. **Comprehensiveness**: This paper's scope is notably narrow, focusing solely on evaluating reward models' performance on mathematical problems. While it attempts to address limitations in a small subset of the Reward Bench dataset, its improvements remain constrained. The study primarily concentrates on reward over-optimization, overlooking other potential vulnerabilities in reward model benchmarking. Additionally, the benchmark's methodology of comparing one correct solution against multiple incorrect ones limits its thoroughness. Furthermore, the author's assumption that MATH500 adequately represents mathematical reasoning tasks may be oversimplified. These limitations collectively suggest a need for a more comprehensive approach to reward model evaluation.

questions:
1. [Clarification] Are the prompts used to evaluate the LLM judge on REWARDMATH the same as the prompt used to evaluate the LLM judge on the Reward Bench? Different prompting strategy (eg. difference system prompt) raises concerns regarding fair comparison between the two benchmarks.
2. [Clarification] What is MATH500? The author did not mention the details behind this dataset which they used for their benchmark. Were there any steps taken to ensure the dataset is not contaminated with the models being evaluated? If the dataset is used during training on any of the evaluated RMs, the benchmark’s reliability will be undermined. 
3. [Clarification] What was the motivation behind different parts of the Synthetic Data experiment? What was the reasoning behind using the MetaMATH dataset? Why was only 80K out of the 155K data points augmented from MATH used for training? 
4. The authors did not mention how the incorrect solutions are ensured to be actually incorrect. Were there steps taken to validate the said incorrect solutions are indeed incorrect?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718921725

thread_type:
official_review

<thread_object>
creator: reviewer_1mfa

<content>
summary:
This paper proposes RewardMATH, a reward model evaluation benchmark focused on the math domain. This benchmark adopts a one-to-many comparison to evaluate reward models robustly. Specifically, it provides 10 responses for each prompt where only 1 response is the answer. The evaluated reward model is considered accurate only when the answer response is given the highest score among all 10 responses. 
The authors also empirically provide sufficient evidence that the RewardMATH benchmark provides reliable estimates for reward models that give policies optimized using BoN are indeed robust in math benchmarks.

strengths:
- The paper provides clear and sufficient empirical evidence that their RewardMATH benchmark is more reliable than the math subset of RewardBench [1]. The empirical results are also clear as LLM policy using BoN on high-scored reward models on RewardBench shows little to no correlation with the performance increase of Math benchmarks (r-square = 0-0.1), while RewardMATH shows a much stronger correlation 
(r-square = 0.6-0.8) in Figure 3.
- The authors have evaluated diverse reward models on RewardMATH, including LLMs (generative reward models), classifier-based reward models, and process reward models.
- The paper considers the problem of over-optimization using a synthetic setup of gold RMs and proxy RMs.  

[1] https://arxiv.org/abs/2403.13787

weaknesses:
- The work would be more interesting if the authors showed any other domains (such as coding or text summarisation or maybe safety) reward model benchmark can be improved by the framework proposed here (by adopting multiple responses and using diverse LLMs to generate outputs). Any initial or limited experiments would be helpful. 

- The lack of PPO (or DPO) usage for policy fine-tuning in experiments seems like a major weakness. The main contribution of this paper is using policy fine-tuning methods to verify if the RewardMATH benchmark scores correlate with the signals it provides during policy fine-tuning. I agree with this approach and am impressed by the number of experiments conducted to verify this using mainly Best-of-N sampling. However, Best-of-N sampling is an inference time method to generate better model outputs using reward models, whereas PPO (or possibly DPO) is the main fine-tuning method researchers use. Although Figure 5 does show a PPO experiment under a synthetic setup, the number of checkpoints or whether the dots follow the findings from Gao et al [2] is not clear to me. Without any solid PPO results, Best of N sampling seems not enough to verify the benchmark's capability of measuring the robustness of reward models. The work will be much more convincing if the authors show more PPO-trained policy evaluations. Or at least, it will be helpful if the author provides more context as to why PPO is hard to train in their non-synthetic setup. Also, I suspect high-scoring reward models on RewardMATH have the ability to find the best response from multiple responses, and Best-of-N adopts a very similar way as it picks the response with the highest reward, resulting in a high correlation of results. Whether this ability will generalize even on PPO setups is not clear to me at this point. 

- Experiment results in Figure 6 compare diverse RMs on both RewardBench and RewardMATH benchmarks with gold or oracle rewards. It would be nice if the authors not only provided the numbers but also a statistical analysis (such as Kendell's tau) that measures the agreement between RewardMATH(or RewardBench) and oracle (or gold) reward scores in Figure 6. 

[2] https://arxiv.org/abs/2210.10760

questions:
- As the proxy reward model trained from synthetic data shows far from optimal performance in Table 3 (only around 13% for RewardMATH and 69% for RewardBench), can you consider using better proxy RMs? The increase in value from 12.68 to 13.51 is not very convincing to me that this is a strong trend.

- In Figure 6, the gold reward (or even the oracle reward) does not drop for most cases even with the maximum KL distance considered. If a larger N is considered for BoN sampling, will the graph drop down as in Gao et al [2]? For a larger N is RewardMATH still successful in detecting more robust reward models regarding the overoptimization problem?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730958567795

thread_type:
official_review

<thread_object>
creator: reviewer_bxc4

<content>
summary:
Authors aims to design a better benchmark for evaluating reward models in reasoning tasks. Authors identify problems with the previous benchmark RewardBench and proposes RewardMath:
* RewardBench is based on PRM800K which contains many wrong annotations, RewardMath instead is based on MATH500.
* RewardBench uses pairs of (human annotated example, LLM annotated incorrect example).  RewardMath includes more than one incorrect example.
* RewardBench’s accepted and rejected examples have different number of steps, this could be a spurious correlation that leads to reward hacking. RewardMath fixes this.
* RewardBench’s PRM evals uses product instead of mean which biases shorter responses, authors fix this by using mean instead. 

To demonstrate the improvements of rewardmath 1) authors compare performances of different reward models on both RewardBench and RewardMath, and show rewardmath has higher correlation with downstream evals 2) authors show rewardmath exhibits the common trend of larger dataset -> better performance while rewardbench does not.

strengths:
* The topic on how to Improve LLM reasoning capabilities has recently gained a lot of attention. This paper focuses on having good benchmarks for evaluating these efforts, and this could be very impactful if done correctly. 

* Authors identify flaws of existing benchmarks and make good efforts to fix them. 

* Paper has good results, specifically Figure 4 is very cool showing RewardMath has stronger correlation with downstream tasks.

weaknesses:
See questions I have below

questions:
* RewardMath is based on the dataset MATH500, where does the dataset MATH500 come from? Is MATH500 prior work (and if yes the citation is missing) or is this a contribution of the paper (in this case it should be made clear).
* Does MATH500 address the incorrect annotation problem found in PRM800K? 
* Can authors also show evaluations and ablations on gsm8k [1] and MATH [2] which are the most common eval tasks for LLM reasoning capabilities?
*  Authors identify that in RewardBench the accepted response often has less steps than rejected ones, which could give a chance for models to reward hack (i.e. reward relies on the number of steps instead of the actual response quality). Did the authors ablate this? I.e.  Does the reward-hacking model predict lower reward if we make the accepted response in RewardMath longer? And vice versa, does it predict higher reward if we make the rejected response shorter? 
* Regarding RewardMath giving more than one rejected responses: If one is trying to do preference learning using a llama model as the base model, is it important for the reward model to know the rejected response generated by a non-llama model should be worse than the accepted response? I.e. the distribution could be very different that it never encounters it during preference learning. I.e. for Figure 4, if we use a llama model as the policy, does one-to-many RewardMath still do better than one-to-one RewardMath chosen & Llama rejection?
* Does reward-model free alignment methods like DPO also suffer from reward model overfitting problem?  What is the advantage of using reward models over reward model-free methods for reasoning tasks? 
* Does the benchmark evaluate cases where both the rejected and chosen response arrive at the same answer, but the rejected answer has the wrong steps? I.e. this is common for truth or false questions. 


[1] Training Verifiers to Solve Math Word Problems

[2]  Measuring mathematical problem solving with the math dataset
</content>
</thread_object>

</thread>
<thread>
creation date:
1731858003009

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We greatly appreciate all reviewers for their time in reviewing our paper and for providing thoughtful suggestions to make our paper stronger. We would like to address comments and suggestions commonly raised by reviewers.


___
## Clarification for Our Main Contributions  
Through the reviewers’ comments, we realized that the paper placed too much emphasis on the new benchmark, RewardMATH. However, the purpose of RewardMATH—a simplified benchmark representing our proposed design—was to validate the reliability of our benchmark design in comparison to RewardBench, rather than to serve as a widely adopted benchmark in future studies. To clarify this point, we slightly revised Title, Abstract and Introduction to reduce the emphasis on RewardMATH, aligning more closely with our original intent. 

The contributions of the paper are as follows:

- We identify the issues with existing benchmark for reward models (i.e. RewardBench)—such as poor quality, vulnerability to reward hacking, and the risk of misjudgement due to isolated cases—and introduce a new design for reliable benchmark for reward models, focusing on reducing the risks of reward hacking and employing multiple comparisons to effectively estimate the degree of reward overoptimization.  
- We conduct extensive experiments validating that the scores on RewardMATH strongly correlates with the performance of optimized policy and effectively estimates the degree of reward overoptimization. These results pave the way for future directions in evaluating reward models more reliably.


Furthermore, **_our key insights_** can be described as follows:  

- **In benchmarks for reward models, a significant difference between the chosen and rejected responses shows low correlation with downstream tasks due to the potential for reward hacking.**  
- **One-to-one comparisons may yield inaccurate results depending on the preference pairs, which in turn results in low correlation with downstream tasks.**  
- **A benchmark employing multiple comparisons can effectively capture reward overoptimization, indicating its ability to assess the robustness of reward models.**

___
## A Scope Limited to the Mathematical Reasoning  
We believe this concern stems from a misunderstanding of our contribution. If our main goal were to propose a well-crafted new benchmark, focusing solely on mathematics might limit the scope of the research; however, our goal is to provide insights into future directions for constructing reliable benchmarks for reward models. And it is fairly intuitive that a reliable benchmark should not be vulnerable to reward hacking and that conducting multiple comparisons, rather than one-to-one-comparisons, provides a more reliable evaluation of reward models. So, we trust that it is important to thoroughly validate our design through at least one specific domain that allows for in-depth experiments and analysis.

Here, the reasons for choosing mathematical reasoning are summarized.

- **One of the tasks where reward models are most extensively used is mathematics reasoning.** In mathematical reasoning tasks, reward models are widely utilized during training to enhance reasoning capabilities and during inference-time using reward-based techniques such as best-of-$n$ (BoN) sampling or Monte Carlo Tree Search (MCTS).  
- **Mathematical reasoning includes a clear human preference.** In mathematical reasoning, human preference can be easily defined as correctness, allowing us to focus effectively on the analysis without the need to deliberate over true preferences.

___
## Updates in the Revised Draft

The updated draft (click to see the pdf) also includes the following enhancements:

* We have updated the Title, Abstract and Introduction to clarify our main contributions.  
* A detailed explanation of MATH500 has been added to line 200 and Appendix B.1.  
* A comprehensive explanation of why PPO experiments were not conducted in a non-synthetic setup is discussed in Appendix B.4.  
* The experiments of policy fine-tuning methods beyond BoN sampling have been featured in Appendix C.5.  
* Additional experimental results based on factors considered in benchmark design have been integrated into Table 12, with analysis details added to Appendix C.4.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**summary**

The paper introduces RewardMATH, a new benchmark for evaluating reward models in mathematical reasoning tasks, designed to address limitations in the existing RewardBench. RewardMATH uses a more accurate dataset (MATH500), multiple incorrect examples for robust evaluations, and a mean-based evaluation metric to avoid length bias, ensuring a fair comparison. It demonstrates improvements over RewardBench by showing higher correlation with downstream evaluations and validating the robustness of reward models through a one-to-many comparison approach. This approach enhances the reliability and effectiveness of reward models in identifying the most accurate responses in mathematical reasoning tasks.

**strengths**

* The authors successfully identify weaknesses in current benchmarks and introduce substantial improvements.
* The paper addresses the increasingly important topic of improving LLM reasoning capabilities by focusing on creating robust benchmarks for evaluating these efforts.
* Diverse reward models were evaluated on RewardMATH, demonstrating its versatility and wide applicability.

**weaknesses**

* The study's focus is narrowly tailored to mathematical problems, potentially overlooking other critical aspects of reward model. To validate the authors' claims about the generalizability of their method, applying the benchmark to other domains or more open-ended queries would be necessary, ideally moving beyond strictly mathematical reasoning.
* The paper's primary weakness lies in its limited experimental validation, relying mainly on correlation with best-of-n sampling. Broadening the scope to include other fine-tuning techniques like SFT, DPO, and PPO, and testing in other domains such as coding or open-ended generation tasks, could provide a more comprehensive assessment of the benchmarks' effectiveness across diverse applications.

**decision**

Although the authors emphasized that the main contribution of this paper extends beyond the new benchmark, it is crucial that the validation of key findings be further improved. I recommend that the authors address the concerns raised by reviewers and consider resubmitting to another venue after making necessary improvements.