How to Evaluate Reward Models for RLHF

Authors: Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, Ion Stoica
Primary area: datasets and benchmarks
Keywords: RLHF, RL, Reward Model, LLM, Benchmark, Dataset, Evaluation

Abstract:
We introduce a new benchmark for reward models that quantifies their ability to produce strong language models through RLHF (Reinforcement Learning from Human Feedback).
The gold-standard approach is to run a full RLHF training pipeline and directly probe downstream LLM performance.
However, this process is prohibitively expensive.
To address this, we build a predictive model of downstream LLM performance by evaluating the reward model on proxy tasks. 
These proxy tasks consist of a large-scale human preference and a verifiable correctness preference dataset, in which we measure 12 metrics across 12 domains.
To investigate which reward model metrics are most correlated to gold-standard RLHF outcomes, we launch an end-to-end RLHF experiment on a large-scale crowd-sourced human preference platform to view real reward model downstream performance as ground truth. 
Ultimately, we compile our data and findings into Preference Proxy Evaluations (PPE), the first reward model benchmark explicitly linked to post-RLHF real-world human preference performance, which we opensource for public use and further development at https://github.com/lmarena/PPE.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1729760675490

thread_type:
official_review

<thread_object>
creator: reviewer_xnaa

<content>
summary:
This paper presents PPE, a reward model benchmark explicitly tied to post-RLHF outcomes based on real human preferences. Previous reward model benchmarks focused solely on evaluating the reward models themselves, without assessing their post-RLHF performance. PPE fills this gap. More importantly, the paper analyzes which metrics best reflect post-RLHF performance, providing valuable insights for improving reward model evaluation methods.

strengths:
- Compared to previous reward model benchmarks, PPE directly evaluates the post-RLHF performance of models, which more accurately reflects the positive impact of reward models on LLM tuning.
- PPE offers more diverse and larger-scale datasets than earlier benchmarks, making it a highly robust benchmark.
- The paper is very detailed and easy to follow, allowing readers to clearly understand the evaluation process. The explanations for each metric and evaluation method are convincing.
- I personally appreciate Section 7 of the paper. The analyses can help researchers predict the performance of fine-tuned LLMs using high-confidence metrics when they lack the resources to conduct full post-RLHF evaluations.

weaknesses:
- I believe this work fills a significant gap in reward model evaluation by addressing post-RLHF performance testing. The effort is substantial and thorough. I do not observe any obvious weaknesses.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1729802315904

thread_type:
official_review

<thread_object>
creator: reviewer_pds3

<content>
summary:
The paper tackles the problem of evaluating reward models without having to run the expensive evaluation of downstream LLM performance. The authors compare metrics that are predictive of actual human evaluations on downstream performance, when employed directly at the reward model stage. Different metrics across preference and correctness are compared to down-stream performance, and rew. model accuracy is identified as the best metric predicting downstream human preferences. Correctness is default benchmarks is also a reasonable indicator of LLM performance.

strengths:
Strengths:
- The problem is highly relevant, the evaluation of reward models and RLHF workflows is a challenging problem
- The collection of human preferences and release as open-source is a valuable contribution to the community, the amount of collected human data is impressive (especially including multi-lingual queries)
- The approach of grounding metrics with human data is highly appreciated, and can be a valuable tool for future reward model development
- The selection of models, prompts and correctness metrics is well motivated

weaknesses:
Weaknesses
- Some of the results seem pretty surprising (e.g. the low correlation with reward bench), for me it’s a bit unclear why we observe them. It would be great if the authors could try to reason more about these observations (as outlined below)
- Clarity of writing and presentation could be improved
- "Reward Model Accuracy" as the best metric is not a novel contribution in itself, you may argue that it’s good to have it validated towards human preferences
- Overall, i find the findings a bit weak, i.e. i don't find it easy to draw clear lessons for reward model design (larger vs. smaller models), i think it would have been very helpful to discuss these results more
- I would have liked some more information about variance/stds., e.g. spread of accuracy across sub datasets, etc., best-of-k curves, etc.

Minor
- Some formulations feel a bit clunky, e.g. P6, L.318. “We use the reward models to RLHF Llama-1.1-8B using..”, P8, L.327: “Mean across all metrics is well correlated across all metrics”  I think the paper could profit from one additional pass of copy editing
- I think adding some figures/plots would help readability, e.g. turning table 3 into a chart

I think there are multiple ways to improve the paper, but the main content is generally convincing to me. If the authors can address some of the points for revision, I am ready to raise my score.

questions:
Needs clarification
- Section 4 felt a bit out of place (so the first section), especially “The human preference dataset contains human-labeled preferences for 16,038 pairwise comparisons between 20 selected top models”: How is this dataset used?  In the paper, there are references to multiple datasets and collections, and I find it a bit confusing to follow how many datasets exist, and for which experiment they are used (e.g. 16,038 preferences in section 4, 12,190 votes in section 6, 50.000 votes in sec 6.1.
- I think the structuring of sections 4 and 5 could be improved a bit, in particular I was a bit confused about the content of section 4, I think a figure showing the workflow/process would help to clarify this
- I would like to ask the authors opinion on using already RLHF fine-tuned models as the source of data. Does it introduce its own biases, in particular to the distribution of responses? Doesn’t it implicitly reward similarity to already existing RLHF strategies?
- Can the caption of Figure 2 be improved? I’m not sure what I see in the right plot compared to the left one, how does the message differ?
- The inverse correlation between reward bench and human elo (Figure 4) is interesting, I would really like the authors to provide a hypothesis for this
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717670671

thread_type:
official_review

<thread_object>
creator: reviewer_ub3i

<content>
summary:
It proposes Preference Proxy Evaluations (PPE), a novel benchmark designed to evaluate reward models based on their ability to predict post-RLHF outcomes. They use a crowdsourced human preferences and a verifiable correctness dataset for the RM evaluation.

strengths:
- Achieves 77% Pearson correlation with human preference ELO. 
- Well motivates why best-of-K scores, row-wise Pearson correlation, & accuracy are relevant for downstream RLHF success. 
- Human preference dataset is large and useful in producing statistically significant outcomes (16,038 labeled preference pairs).

weaknesses:
- Underdiscusses the possibility that the RM evals are computationally very expensive to run in the the PPE framework. Best-of-K performance curves and pairwise accuracy have a huge computational burden, especially as K increases. Please provide runtime estimates for different K values or discuss strategies for making the evaluations more computationally efficient.
- Does not discusses the findings (e.g. low quantile aggregation correlation) in any depth. Please explore potential explanations for this correlation or discuss its implications for reward model design and training.

questions:
- Why, intuitively or theoretically, is low quantile aggregation correlation correlatedmore strongly with downstream RLHF performance? Please provide hypotheses or conduct additional analyses to investigate this relationship further.
- How does inference cost scale with K, and what's a feasible value of K for training a large language model today? Please provide a table or graph showing inference costs for different K values, and discuss how these costs might impact real-world applications of the method.
- Why do pairwise accuracy-like metrics suffer from overfitting? Discuss this in more detail and provide empirical evidence of overfitting in pairwise accuracy metrics if possible.
- Is there a predictive accuracy plateau after a certain K-value? Is it possible to include a graph demonstrating diminishing returns/values of predictive accuracy with K?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730741370582

thread_type:
official_review

<thread_object>
creator: reviewer_nk4s

<content>
summary:
This paper introduces Preference Proxy Evaluations (PPE), the first reward model benchmark explicitly connected to real-world human preference performance after RLHF. Unlike benchmarks like RewardBench that primarily perform reward evaluations, PPE facilitates investigation into which reward model metrics correlate most closely with RLHF outcomes.

Studying the impact of reward models on post-RLHF model performance is important. This paper involves significant work on this topic. Although I am inclined to accept it, there are still some important shortcomings (see the **Weaknesses**). I hope the authors can address my concerns.

strengths:
- This paper addresses an important topic: the effectiveness of a reward model should ultimately be assessed by the performance of LLMs after RLHF. The study offers valuable insights into the evaluation of existing reward models.
- The paper includes extensive experiments and evaluates models using real-world human preferences.

weaknesses:
- The study employs DPO as the RLHF algorithm to evaluate post-RLHF performance. However, the offline DPO algorithm may face generalization issues. Utilizing an online PPO algorithm to obtain post-RLHF models and assess their performance would offer a more comprehensive evaluation of various reward models. This additional experimentation is essential to support the authors’ core claim of being “the first reward model benchmark explicitly linked to post-RLHF performance.”
- The paper utilizes Llama-3.1-8B-Instruct as the base model for RLHF, which presents two potential concerns: 1) The paper should explore a broader range of base model scales rather than exclusively using the 8B model to fully assess the new benchmark’s validity. 2) Llama-3.1-8B-Instruct itself has undergone DPO RLHF training. I recommend that the authors use a purely SFT-trained Llama-SFT model as the base model.
- The paper would benefit from more detailed descriptions and discussions regarding Figure 4. For example, it should provide intuitive explanations for why metrics from RewardBench fail to reflect post-RLHF preference performance.

questions:
- I am unclear about why the Pearson correlation for “chat hard” in Figure 4 is so low. Does this imply that most RewardBench metrics do not reflect post-RLHF performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732704182109

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comment by the Authors

comment:
We thank the reviewers for the valuable feedback. We appreciate that the reviewers see the value in our work to evaluate reward models with respect to their downstream LLM performance (nK4S, xNaA) as well as our usage and release of real human preferences (nK4S, Ub3i, PDs3).

In our revised version, changes as suggested by the reviewers are shown in blue text. Below we detail revisions made to the paper.


### Major Additions

The major additions are as follows, in rough order of appearance:

1. Figure 2 showing accuracies of various preference models has been added, as suggested by reviewer PDs3.
2. Figure 3 caption is re-written as suggested by reviewer PDs3.
3. Figure 4 showing model performance as a graph instead of a table has been added, as suggested by reviewer PDs3.
4. The last paragraph on lines 519-525 has been slightly expanded upon, with links to deeper additional comments in the appendix, as suggested by reviewer Ub3i.
5. Limitations section 8.2 has been updated based on reviewer nK4S's comments and suggestions.
6. Appendix A1 has been added, which shows an overview of PPE's various datasets, as suggested by reviewer PDs3. Additionally, it is referenced at the end of section 3 on line 136.
7. Appendix A.2.1 and A.3.4 have been added. These section contain 12 new tables detailing distribution statistics for each benchmark/metric/domain in PPE, as suggested by reviewer PDs3
8. Lines 1778-1785 have been added to Appendix A.6, laying out deeper possible explanations for why low quantile aggregation is helpful, as suggested by reviewer Ub3i.
9. As suggested by reviewers nK4S, Ub3i, and PDs3, Appendix section A.6.1 has been added, which comments on the low correlation seen in RewardBench. This section is referenced in the original figure showing RewardBench correlations in section 7.
10. As suggested by reviewer Ub3i, Appendix A.6.3 and Figure 13 has been added, showing predictive power vs K, along with lines 1918-1921 providing additional explanation.
11. Based on reviewer Ub3i's suggestions, Appendix A.7 has been added, detailed recommended settings to run PPE based on the end-to-end experiment and Appendix A.6.3.
12. Appendix A.8 has been added, which contains a table detailing PPE's efficient runtimes and costs, as suggested by reviewer Ub3i's.

### Other Changes

Other minor changes based on reviewer suggestions are written in blue.

### Reorganization

Additionally, in order to integrate reviewer feedback while adhering to the 10 page limit, we reorganized some content based on importance:
Details on best of K scoring and metrics has been moved to Appendix A.3.2. The Abstract and Introduction have been modified to be slightly shorter and clearer; the underlying semantics of these sections are unaltered. (Note these structural changes are not highlighted in blue.) The main figure has been slightly updated showing PPE's role in the RLHF pipeline for increased clarity.

Once again, we thank the reviewers for their questions, feedback, and suggestions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper addresses an important problem: which reward model to use for RLHF? Running the full pipeline is expensive and take days. This paper addresses it proposing a new large benchmark for reward model along with how to use it to get a score that correlates well with downstream RLHF performance. 

Strengths:
1. Addresses a very important practical problem facing any team that does RLHF
2. Reviewers found the new benchmark useful 
3. Achieves a decent 77% Pearson correlation with downstream performance

Weakness:
1. Reviewers found that the paper is lacking in explanations. E.g., about why low quantile aggregation correlation used by reviewer Ub3i.
2. Also, due to budget reasons, authors were not able to train on more LLMs. Nevertheless, this limits how much we can read from the results. Someone else will have to run these results more broadly before we can trust the generality.

Overall, I like this paper. I think this new benchmark will be useful. More investigation is certainly needed but I think this paper will have an impact even in its current form so I recommend acceptance.