Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?

## Abstract

Abstract Reward Models (RMs) are crucial for aligning language models with human preferences.
Currently, the evaluation of RMs depends on measuring accuracy against a validation set of manually annotated preference data.
Although this method is straightforward and widely adopted, the relationship between RM accuracy and downstream policy performance remains under-explored.
In this work, we conduct experiments in a synthetic setting to investigate how differences in RM measured by accuracy translate into gaps in optimized policy performance.
Our findings reveal that while there is a weak positive correlation between accuracy and downstream performance, policies optimized towards RMs with similar accuracy can exhibit quite different performance.
Moreover, we discover that the way of measuring accuracy significantly impacts its ability to predict the final policy performance.
Through the lens of Regressional Goodhart’s effect, we identify the existence of exogenous variables impacting the relationship between RM quality measured by accuracy and policy model capability.
This underscores the inadequacy of relying solely on accuracy to reflect their impact on policy optimization.

## 1 Introduction

Reinforcement Learning from Human Feedback (RLHF) (Ibarz et al., ; Ouyang et al., ) has emerged as a prominent paradigm for aligning Large Language Models (Yang et al., ; Dubey et al., ).
The reward model (Ng & Russell, ; Brown & Niekum, ; Palan et al., ) plays a crucial role in this process by substituting human preferences for model optimization.
However, building an RM that fully captures human preferences is highly challenging (Armstrong & Mindermann, ; Skalse & Abate, ; Lambert et al., ).
Therefore, the RM can be an imperfect proxy for ideal preferences and cause downstream performance deterioration when optimized against it, known as reward model overoptimization (Gao et al., ).
This phenomenon, as a result of Goodhart’s law (Karwowski et al., ), presents a critical challenge to the RLHF.

The inherent difficulty of constructing an ideal RM require a careful evaluation process to capture its potential negative impace in policy optimization.
To date, common practice for evaluating the RM includes directly assessing the optimized policy (Hendrycks et al., ; Li et al., ) and computing accuracy on a fixed dataset (Lambert et al., ).
The former, while it can be seen as a final metric in practice, is limited by the cost of optimization.
More importantly, it fails to distinguish whether the causes of undesirable behavior arise from the policy optimization process or the reward learning process.
The latter, while straightforward, remains the question of whether such evaluation accurately predicts the performance of the downstream optimized policy.

In this work, we systematically investigate the effectiveness of using accuracy in predicting downstream performance.
Essentially, the accuracy metric measures the difference between two reward functions, referred to as RM error.
In practice, the accuracy quantifies the error between the learned reward model and the human golden reward function.
The RM error can lead to potential performance loss between the policies optimized for the proxy RM and the golden reward function.
Specifically, we define this performance gap as Policy Regret.
Therefore, the goal of RM evaluation is to estimate policy regret through RM error, as depicted in Figure 1.

To investigate how differences in RM measured by accuracy translate into gaps in optimized policy performance, we design a synthetic experiment framework.
Due to the inaccessibility of human reward functions, we employ synthetic RM as the golden reward function in our experiments.
To effectively collect golden-proxy RM pairs for analysis, we create $N$ different RMs, designating one RM as the golden model and the others as proxy models each time.
Based on the above framework, we decompose the issue concerning RM evaluation into three research questions (RQ):

Figure: Figure 1: A cartoon of RM error and policy regret.
Refer to caption: x1.png

RQ1: Does the RM error measured by accuracy correlate with the policy regret?
We examine the correlation between the accuracy and the policy regret on the widely adopted benchmark dataset (Lambert et al., ).
We employ optimization algorithms, i.e., best-of-$n$ sampling and policy gradient-based reinforcement learning, for investigation.
Our findings reveal a weak positive correlation between the measured accuracy and the policy regret.
However, we observe that policies optimized towards reward models within a similar accuracy range can have quite different regrets.

RQ2: How to better measure RM error for policy regret prediction?
While we present a positive correlation between the accuracy and the policy regret, there remains room for further enhancement.
This naturally raises the question of how to better quantify RM error to achieve a stronger correlation.
We begin by investigate the influence of prompt and response distributions.
Specifically, we observe that the prompt difference between the RM test dataset and the downstream test dataset will potentially decrease the correlation between the accuracy and the policy regret.
Regarding response distribution, we find that the rank of the responses can be more influential rather than the model from which it is sampled.
Furthermore, we propose a simple yet effective strategy to enhance the correlation: increasing the number of responses per prompt.
Compared to the typical setting, metrics based on more responses reach a higher correlation with the policy regret.
We further validate this approach under constraints of equal sample size and annotation budget.

RQ3: What’s the relationship between RM error and Policy Regret?
The translation from the RM error to the policy regret can be seen as the result of the reward model overoptimization.
Therefore, to accurately predict the policy regret, the quantified RM error should be able to reflect the potential overoptimization.
In this work, we derive the expected relationship between the accuracy and the degree of reward overoptimization under the assumptions of Regressional Goodhart’s effect (Manheim & Garrabrant, ) and normal distributions of both reward score and noise.
However, our findings indicate that reward models with similar accuracy can behave quite differently in terms of overoptimization, suggesting more exogenous variables may take place.
This indicates accuracy alone can be an inadequate metric for predicting the downstream performance.

In summary, our work provides deeper understanding into the relationship between RM error and policy regret.
This offers valuable for both reward model training and the development of a more robust RLHF algorithm.
Our results also highlight the need for a more rigorous benchmark in reward model evaluation and the development of advanced RM assessment tools.

## 2 Preliminary

Let $\mathcal{X}$ be the set of prompts, and $\mathcal{Y}_{\mathcal{X}}$ be the set of possible responses.
A policy $\pi$ is a language model that generates responses $y$ to the prompt $x$ at probability $\pi(y|x)$.
The golden reward function assesses the response and gives a score based on their quality $r^{*}:\mathcal{X}\times\mathcal{Y}_{\mathcal{X}}\rightarrow\mathbb{R}$.
In practice, the golden reward function $r$ represents the complicated human preference and is generally inaccessible.
Instead, a reward model $r$ learned from preference data serves as the proxy of the golden reward function $r^{*}$.
Then policy $\pi$ is optimized to maximize the expectation of the reward given by $r$:

$$ $\pi=\max\limits_{\pi}\mathbb{E}_{x\in\mathcal{X},y\sim\pi(\cdot|x)}[r(x,y)]$ (1) $$

This results in a sub-optimal policy $\pi$ rather than the ideal policy $\pi^{*}$ that directly maximizes the reward given by the golden reward function $r^{*}$.

The workflow of Reinforcement Learning from Human Feedback (RLHF) can be represented as:

- •
$\mathcal{A}_{RM}(D_{RM}^{train})\rightarrow r$: Learn a proxy RM $r$ on the preference dataset $D_{RM}^{train}$ by algorithm $\mathcal{A}_{RM}$.
- •
$\mathcal{M}(r,r^{*}|D_{RM}^{test})\rightarrow d_{r}$: Evaluate the RM error between proxy reward model $r$ and golden reward function $r^{*}$ by metric $\mathcal{M}$ on the test dataset $D_{RM}^{test}$.
- •
$\mathcal{A}_{RL}(\pi_{0},r|D_{RL}^{train})\rightarrow\pi$: Optimize the initial policy $\pi_{0}$ and get policy $\pi$ that maximizes the expectation of reward given by the proxy RM $r$ by the algorithm $\mathcal{A}_{RL}$.
- •
$\mathcal{M}^{\prime}(\pi,\pi^{*}|D_{RL}^{test})\rightarrow d_{\pi}$: Measure the regret between the policy $\pi$ and the policy $\pi^{*}$ that optimized directly towards $r^{*}$ by metric $\mathcal{M}^{\prime}$ on the test dataset $D_{RL}^{test}$.

In this work, we essentially focus on the relationship between the error of reward model $d_{r}$ and the regret of policy $d_{\pi}$, which translates our issue into: How should we measure the error $d_{r}$ so as to reach high correlation with the policy regret $d_{\pi}$?
Various factors introduced in the RLHF process will potentially influence this relationship.
To conduct experiments, we focus on a few interested factors and keep the rest fixed.
Specifically, the following factors are considered:
The test dataset $D_{RM}^{test}$ on which the reward model $r$ is evaluated; The metric $\mathcal{M}$ used to evaluate the error between the reward models; The test dataset $D_{RL}^{test}$ on which the final policy $\pi$ is evaluated.

Figure: (a) Synthetic pipeline of Reinforcement Learning from Human Feedback.
Refer to caption: x2.png

One challenge to perform investigation is the inaccessibility of the golden reward function representing human preferences.
To address this difficulty, we follow the previous practice (Gao et al., ; ) of utilizing another reward model to act as the golden reward function.

Another challenge is to effectively collect the proxy-golden RM pairs for evaluating the correlation between the RM error and policy regret.
We implement this by constructing $N=10$ different reward models $r_{i}$ by randomly flipping $\alpha_{i}\%$ of the pairs in the training dataset(^1^11Details about dataset construction can be referred to Appendix 8.1.) $D_{RM}^{train}$.
Then $N$ policies $\pi_{i}$ will be optimized towards the corresponding reward model $r_{i}$.
This pipeline is demonstrated in Figure 2(a).
This operation will form $N\times N$ golden-proxy RM pairs.
As shown in Figure 2(b)
,
these pair collections can then be used for computing correlation by letting one reward model as the golden reward function and the remaining reward models as proxy reward models.

Due to the inaccessibility of the global optimal policy and the difficulty of constantly controlling optimization pressure, real regret is hard to practically compute under the RLHF context.
So, We define Normalised Drop Ratio (NDR) as an estimation of the policy regret.
This metric $\mathcal{M}^{\prime}$ quantifies the performance deterioration due to optimization against a proxy reward rather than the golden reward function.
It computes the difference between the rewards obtained by the policy $\pi^{*}$ and $\pi$:

$$ $\mathcal{M}^{\prime}_{NDR}(\pi,\pi^{\prime})=\frac{\mathbb{E}_{y\sim\pi(\cdot| x)}[r^{*}(x,y)]-\mathbb{E}_{y\sim\pi_{0}(\cdot|x)}[r^{*}(x,y)]}{\mathbb{E}_{y \sim\pi^{*}(\cdot|x)}[r^{*}(x,y)]-\mathbb{E}_{y\sim\pi_{0}(\cdot|x)}[r^{*}(x,y )]}$ (2) $$

where the policies $\pi$ and $\pi^{*}$ are optimized against the golden reward function $r^{*}$ and the proxy RM $r$ with exactly the same hyperparameters.

The data from the RewardBench (Lambert et al., ) are used to construct the test datasets $D_{RM}^{test}$ and $D_{RL}^{test}$.
All reward models are initialized from the Llama-3-instruct-8B model (AI@Meta, ) and finetuned by minimizing the negative log-likelihood loss with regularization term:

$$ $\mathcal{L}_{RM}=-\mathbb{E}_{(x,y_{w},y_{l})\sim D_{RM}^{train}}[\log(\sigma( r(x,y_{w})-r(x,y_{l})))]-\mathbb{E}_{(x,y)\sim D_{RM}^{train}}[r(x,y)^{2}]$ (3) $$

We adopt best-of-$n$ sampling (BoN) and PPO (Schulman et al., ) as the algorithm $A_{RL}$ to optimize the initial policy $\pi_{0}$, which is also the Llama-3-instruct-8B model.
For the best-of-$n$ sampling, the reward model $r$ picks the response with the highest reward from $n$ candidates.
For the PPO algorithm, we follow (Gao et al., ) and set the KL penalty in all experiments to be 0 0.

## 3 Does the RM error measured by accuracy correlate with the policy regret?

Current research typically assesses reward model errors by computing accuracy on a fixed test set.
While this method is straightforward and widely used, few studies have explored whether the accuracy metric correlates with policy regret.
In this section, we examine the relationship between measured accuracy and policy regret empirically.

#### Finding 1:

RM evaluation accuracy is positively related to policy regret, but even with similar RM accuracy, policies can exhibit different levels of regret.

We first investigate the correlation between the accuracy and the policy regret.
Original prompts and responses from RewardBench are used to measure accuracy, while the prompts are used for downstream optimization.
We assess three traditional correlation coefficients to evaluate how well accuracy metrics benchmark reward models.
Additionally, we include the Mean Reciprocal Rank(^2^22MRR is calculated by averaging the reciprocal ranks of policies optimized by the reward model with the highest accuracy. The rank of policy is decided by NDR metric.) (Voorhees & Tice, ) to examine if accuracy can identify the best reward model among multiple candidates.
As presented in Table 1, there is a positive relationship between accuracy and policy regret.
However, trends depicted in Figure 3(a) and Figure 3(b) illustrate that policy regret can vary considerably even within similar accuracy ranges.
Lastly, we observe that accuracy generally correlates more strongly with regret in BoN than in PPO.
This is expected, as BoN is a more localized and stable optimization algorithm, making it more predictable by reward model error.

Table: Table 1: The correlation between the accuracy (Acc.) and the policy regret under BoN and PPO setting measured by typical correlation coefficients and Mean Reciprocal Rank (MRR).

Figure: (a) Trend between accuracy $d_{r}^{acc}$ and the policy regret $d^{NDR}_{\pi}$ under BoN setting.
Refer to caption: x4.png

## 4 How to better measure RM error for policy regret prediction?

In the previous section, we examine the positive correlation between accuracy and policy regret.
However, there appears to be room for enhancing this correlation, which leads us to the question: how to better quantify the RM error?
In this section, we first investigate the influence of prompt and response distribution.
Moreover, we explore a straightforward yet effective strategy, i.e., extending responses per prompt.
Finally, we validate it under different constraints.

#### Finding 2:

The rank of responses affects the correlation between accuracy and regret more than the response sampling models.

In the previous experiment, we used the original responses in RewardBench for validation, which were sampled from different models.
This naturally raises the question: Can we achieve a higher correlation by sampling responses specifically from the model used for downstream optimization?
To examine this question, we construct test datasets with responses solely from different models.
We evaluate the correlation between the policy regret and the computed accuracy on these datasets.
As shown in Table 2, this approach does not consistently improve upon the original RewardBench dataset.
This result suggests that sampling responses from the model used for optimization may not be necessary to achieve a strong correlation between accuracy and policy regret.

Table: Table 2: The correlation between the policy regret and the accuracy on the dataset containing responses sampled from different models. The Origin represents the result on the original RewardBench dataset. The highest results are highlighted in bold, and the second highest result is underlined. The model used for downstream optimization is suffixed with *.

Although sampling from models used for optimization seems unnecessary, we observe varying correlations on datasets constructed by the responses from different models.
This suggests potential factors beyond the sampling model, such as the rank of chosen and rejected samples.
To examine the assumption, we prepare multiple responses, sort them by their golden rewards, and divide them into bins.
We randomly sample chosen and rejected responses from different bins to construct the test datasets with responses of varying ranks.
Different trends emerged for BoN and PPO, as demonstrated in Figure 4(a) and Figure 4(b).
For BoN, test datasets with chosen samples from average rank bins and rejected samples from lower rank bins exhibited higher correlations.
For PPO, datasets with chosen samples from higher rank bins and rejected samples from average rank bins showed a higher correlation.
This distinction likely arises from the differing characteristics of the two optimization algorithms.
Overall, we conclude the suitable strategy is to choose one response of average rank as a representative sample, along with another response with an appropriate gap for contrast.

Figure: (a) The correlation trend on BoN.
Refer to caption: x6.png

#### Finding 3:

The prompt difference between the RM test dataset and downstream test dataset can weaken the accuracy-regret correlation.

The previous experiments assume that the RM and RL test datasets consisted of the same prompts, which may not be true in practice.
To investigate the influence of prompt category, we split the test dataset based on the prompt category determined by the original classification in RewardBench.
We evaluate accuracy and regret in each category and demonstrate the result in Table 3.
We find that accuracy in each category aligns more closely with regret in the corresponding category under the BoN setting.
However, this relationship is not present in the PPO setting.
This may be because different types of prompts influence each other during PPO optimization.

Table: Table 3: The correlation between policy regret and accuracy on the test dataset composed of prompts from different categories evaluated by the Spearman coefficient. The highest result in each column is bolded, and the highest in each row is underlined.

To further investigate the influence of prompt semantics, we randomly rewrite some prompts in the origin test dataset, creating new test datasets with prompts different from the downstream dataset.
We ask GPT-4o (OpenAI, ) to perform two different strategies with prompts detailed in Appendix 8.2.
One strategy alters the expression without changing the meaning, and the other generates a new prompt within the same category.
As shown in Figures 5(a) and 5(b), we find that the correlation is less affected by prompt paraphrasing under BoN.
While the correlation on PPO continuously weakens as we paraphrase a larger ratio of prompts.

Figure: (a) The correlation trend on BoN.
Refer to caption: x8.png

#### Finding 4:

Increasing the number of responses per prompt generally enhances the correlation between measured RM error and regret.

We first examine the strategy of response expansion per prompt by preparing $5$ responses per prompt and employing various metrics(^3^33Further details about the computation of these metrics can be found in Appendix 8.3).
These metrics include common correlation measures, as well as metrics transported from Information Retrieval.
We also explored other metrics for comparison, e.g., expect calibration error and Bo5.
Note that some metrics require absolute scoring of responses, which may be inaccessible in practice.
As shown in Table 4, metrics evaluated on the dataset with more responses consistently achieve higher correlations in comparison to the accuracy on the dataset with two responses per prompt.
Moreover, the $\xi$ correlation (Chatterjee, ) generally demonstrated the best performance.
However, the ECE metric, despite being relatively common, does not show a significant correlation with regret.

Table: Table 4: The correlation between different metrics and policy regret. The Accuracy-pair stands for the baseline that only samples two responses per prompt and computes the accuracy. Metrics that require an absolute score for each response are suffixed with *.

Although measuring more responses per prompt yields a higher correlation, it can be unfair without the control of the total number of samples (prompt-response pairs) in the test dataset.
Thus, we further explore the question that should we increase responses or prompts under a constant sample amount(^4^44Note that the number of samples approximately translates to the computational cost of the evaluation.).
We fix the sample size in the test dataset while varying the number of responses.
As shown in Figure 6(a) and Figure 6(b), it is generally more effective to increase the number of responses rather than the number of prompts.
However, such bonus decreases with the growing sample size, possibly due to an upper bound in the correlation between regret and measured accuracy.

Figure: (a) The correlation trend on BoN.
Refer to caption: x10.png

The annotation cost can be another important factor since building a test dataset with more than two responses can be tricky.
To explore this factor, we further examine the question that should we increase responses or prompts under a constant annotation budget.
To conduct the experiment, we consider the noise in the annotation process.
Among various probability ranking models (Critchlow et al., ), we adopt the most commonly used one, i.e., the Bradley-Terry model (Bradley & Terry, ).
Given a prompt $x$ and two responses $y_{i}$ and $y_{j}$, the probability that $y_{i}$ is preferred to $y_{j}$ is computed by:

$$ $P(y_{i}\succ y_{j}|x)=\frac{1}{1+e^{(r_{j}-r_{i})/\beta}}$ (4) $$

where $r_{i}$ and $r_{j}$ is the golden reward score given to the response $y_{i}$ and $y_{j}$ in our case.
The $\beta$, which can be seen as an indicator of the annotation quality, is set to be $0.5$.
We measure the number of pair-wise comparisons by annotation cost.
This setting fits the Bradley-Terry model and is also commonly adopted in real-world practice.
The annotation process is stimulated by performing a sorting algorithm based on pair-wise comparison.The final results are presented in Figures 7(a) and 7(b).
We find that it’s more beneficial to include more responses under the BoN setting with the same annotation budget.
However, different trends are observed under the PPO setting.

Figure: (a) The correlation trend on BoN.
Refer to caption: x12.png

## 5 What’s the relationship between RM error and Policy Regret?

In this section, we investigate the relationship between RM error and policy regret.
The translation from RM error to the policy regret can be seen as the result of the reward model overoptimization.
Therefore, to accurately predict policy regret, the quantified RM error should be able to reveal the potential overoptimization.
We first theoretically explore this issue.
Then we empirically examine the relationship and analyze the observed optimization dynamics.

#### Finding 5:

There could be exogenous variables impacting the relationship between RM quality measured by accuracy and policy model optimization.

The translation from RM error to the policy regret can seem as the result of Goodhart’s Laws (Goodhart, ).
It says that optimizing a less effective metric rather than the golden metric leads to the failure of the system.
Such phenomenon is also commonly termed reward model overoptimization (Gao et al., ) under the context of RLHF.
Among various kinds of Goodhart’s effect leading to the reward model overoptimization, the Regressional Goodhart’s effect (Manheim & Garrabrant, ) is the most fundamental one and cannot be avoided.
It occurs when the proxy reward model $r$ is equal to the golden reward function $r^{*}$ added with some independent noise $z$, namely:

$$ $r=r^{*}+z$ (5) $$

Assuming that only the Regressional Goodhart’s effect takes place and the reward score and noise are normally distributed, i.e. $r^{*}\sim\mathcal{N}(0,\sigma_{r}^{2})$ and $z\sim\mathcal{N}(0,\sigma^{2})$, the relationship(^5^55Detailed explanation and derivation are provided in Appendix 8.4.) between accuracy $d_{r}^{acc}$ and the degree of overoptimization(^6^66Assuming normalized reward scores and similar optimization dynamics towards the proxy and true reward models, which generally holds in our BoN experiments, the NDR metric is approximate $d_{\pi}$ times a coefficient.) $d_{\pi}$ can be expressed by the following parametric equations:

$$ $\begin{split}d_{r}^{acc}&=1-\int_{x=0}^{+\infty}\frac{1}{\sigma_{r}\sqrt{\pi}} e^{-\frac{x^{2}}{4\sigma_{r}^{2}}}\Phi\left(-\frac{x}{\sqrt{2}\sigma}\right)dx \\ d_{\pi}&=\frac{\sigma_{r}^{2}}{\sigma_{r}^{2}+\sigma^{2}}\end{split}$ (6) $$

where $\Phi$ denotes the cumulative distribution function of standard normal distribution.
The equation is too complex to be solved analytically, so we resort to a numerical solution for plotting the relationship.
The predicted trend between accuracy and the degree of overoptimization is illustrated in Figure 8(a).
We observed that there should be an approximate linear relationship between them.

Figure: (a) Expected trend of accuracy $d_{r}^{acc}$ and the degree of overoptimization $d_{\pi}$ by varying the noise $\sigma$.
Refer to caption: x14.png

However, the actual trend plotted in Figure 8(b) presents outliers beyond our expected relationship.
These discrepancies may stem from additional Goodhart’s effects beyond the Regressional one.
We examine the dynamics of golden reward scores when optimizing towards various proxy reward models in Figures 9(a) and 9(b).
The distance
$d$ is computed as $\sqrt{D_{KL}(\pi||\pi_{0})}$ (Bai et al., ).
The KL divergence of best-of-$n$ sampling can be computed by $\log n-\frac{n-1}{n}$ (Hilton, ).
The rewards are computed by an unbiased estimator from (Nakano et al., ).
The accuracy of these proxy reward models with respect to the golden reward function roughly falls in the range of $(0.7,0.8)$.
Despite the close accuracy, we observe distinct behaviors in overoptimization.
In the first graph, there is barely any indication of typical overoptimization where the golden reward first increase and then drop.
This indicates the Regressional Goodhart’s effect is dominant in this case.
In contrast, the second graph shows a noticeable decline in golden reward scores across nearly all proxy models, suggesting the influence of more complex Goodhart’s effects.
The differences between these cases suggest the existence of exogenous variables impacting the optimization dynamics.

Figure: (a) RM with $\alpha=0\%$ serve as golden reward model.
Refer to caption: x16.png

## 6 Related Works

Reinforcement Learning from Human Feedback (RLHF) (Bai et al., ; Ouyang et al., ; Stiennon et al., ) has been a common strategy for the alignment of Large Language Models (Yang et al., ; Dubey et al., ), in which the reward model plays a crucial role.
However, this methodology faces several challenges (Casper et al., ; Lang et al., ; Armstrong & Mindermann, ; Skalse et al., ), such as reward model overoptimization (Gao et al., ; Lehman et al., ), where optimizing policy towards proxy reward model may lead to the ground truth performance deterioration.
This phenomenon, as a special case of reward hacking (Krakovna, ), can seen as the result of Goodhart’s law (Goodhart, ; Zhuang & Hadfield-Menell, ).

The RM as an imperfect proxy of the golden preference necessitates the evaluation of whether the reward model accurately captures the real preference (Michaud et al., ; Russell & Santos, ).
Many works consider the measurements to predict the potential consequence due to the difference of reward functions (Ng et al., ; Skalse et al., ).
(Gleave et al., ; Skalse et al., ) propose the metric quantify the difference between the reward functions and induce regret bounds for optimal policies.
However, some works (Zhuang & Hadfield-Menell, ; Fluri et al., ) propose negative visions of optimization towards imperfect proxy RMs.
These works assume the accessibility of the golden reward function, which can be false under the RLHF setting.
The common practice under RLHF is to compute accuracy on a fixed preference dataset (Lambert et al., ), which remains the question about theoretical or empirical validation of performance prediction.

## 7 Conclusion

Our study highlights the limitations of evaluating the reward model solely based on accuracy for predicting downstream policy performance.
Although there exists a weak positive correlation between accuracy and policy performance, reward models with similar accuracy can result in varying policy outcomes.
Moreover, the way of measuring accuracy will significantly impact the prediction performance.
Additionally, we identify the presence of exogenous variables influencing the optimization process.
Overall, we should take a more cautious attitude about the RM performance indicated by accuracy and develop more advanced techniques for RM evaluation.

## 8 Appendix

### 8.1 Dataset Construction

We build up a RM training dataset by mixing the following open-sourced datasets:

- •
Nectar (Zhu et al., ), a high-quality $7$-wise comparison dataset generated by GPT-4 ranking.
- •
Capybara-7K-binarized (Argilla, ), a binarized prefernce dataset built with distilabel atop the Capybara (Daniele & Suphavadeeprasit, ).
- •
Orca-pairs (Intel, ), a dataset contains 12k examples from OpenOrca dataset (Lian et al., ).
- •
UltraFeedback (Cui et al., ), a large-scale, fine-grained, diverse preference dataset.
- •
PKU-SafeRLHF (Ji et al., ), a high-quality dataset consisting of 83.4K preference entries, which is annotated across two dimensions: harmlessness and helpfulness.
- •
MTBench-human (Zheng et al., ), a dataset contains 3.3K expert-level pairwise human preferences for model responses generated by 6 models in response to 80 MT-bench questions.
- •
Chatbot-arena (Zheng et al., ), a dataset contains 33K cleaned conversations with pairwise human preferences. Responses generated by 6 models in response to 80 MT-bench questions.
- •
HH-RLHF (Bai et al., ; Ganguli et al., ), a dataset contains human preference data about helpfulness and harmlessness.

We retain only single-turn dialogue data, deduplicating based on prompt string matching. Next, we filtered out excessively long samples and balanced the proportion of positive and negative sample lengths. Ultimately, we retained $112400$ preference data samples, with $7052$ set aside as a validation set for reward model training.

The data from RewardBench (Lambert et al., ) is used to construct the test datasets. We used a filtered version of the data and performed string-based deduplication on the prompts, resulting in 2733 remaining samples.

### 8.2 Paraphrase Prompts

We employed two strategies for rewriting prompts: one that alters expression without changing semantics, and another that rewrites the prompt into a similar but related prompt.
The prompt in Table 5 is used for the paraphrase strategy, and the prompt in Table 6 is used for the rewrite strategy.

**Table 5: The prompt used for paraphrase strategy.**
| Now you should play the role of prompt engineer.<br>Your task is to paraphrase the prompt into a new prompt.<br>If the origin prompt is a safety-related prompt, paraphrase it into a new safe prompt.<br>When outputting, only provide the paraphrased prompt and nothing else.<br>>> Input Prompt:<br>{prompt}<br>>> Output: |
| --- |

**Table 6: The prompt used for rewrite strategy.**
| Now you should play the role of prompt engineer.<br>You should generate a new prompt of the same category with an input prompt.<br>If the origin prompt is a safety-related prompt, paraphrase it into a new safe prompt.<br>When outputting, only provide the generated prompt and nothing else.<br>>> Input Prompt:<br>{prompt}<br>>> Output: |
| --- |

### 8.3 Reward Error Metrics

Given a prompt $x$ and $N=5$ different responses $Y=[y_{1},...,y_{N}]$, their corresponding reward scores and ranks are represented as $r(Y)$ and $R(Y)$.
Since each prompt has multiple annotated responses, many ranking evaluation metrics can be applied to our setting.
These include the metrics listed in Table 4, which we will detail one by one.

#### Pearson corr.

Pearson correlation coefficient can be computed as follows:

$$ $\rho_{r(Y),r^{*}(Y)}=\frac{cov\left(r(Y),r^{*}(Y)\right)}{\sigma_{r(Y)}\sigma_ {r^{*}(Y)}}$ (7) $$

where $cov$ is the covariance, $\sigma_{r(Y)}$ is the standard deviation of the proxy rewards given to the responses $r(Y)$ and $\sigma_{r^{*}(Y)}$ is the standard deviation of
the golden rewards given to the responses $r^{*}(Y)$.
The formula of $cov\left(r(Y),r^{*}(Y)\right)$ can be writed as:

$$ $cov\left(r(Y),r^{*}(Y)\right)=\mathbb{E}\left[(r(Y)-\mu_{r(Y)})(r^{*}(Y)-\mu_{ r^{*}(Y)})\right]$ (8) $$

Where $\mu_{r(Y)}$ and $\mu_{r}^{*}(Y)$ are the mean of the proxy rewards and golden rewards given to the responses, respectively.
The final metric is the average of the coefficient of each prompt.

#### Spearman corr.

Spearman’s rank correlation coefficient follows the basically same formula as the Pearson correlation coefficient, only with the reward scores replaced by the ranks:

$$ $s_{R(Y),R^{*}(Y)}=\frac{cov\left(R(Y),R^{*}(Y)\right)}{\sigma_{R(Y)}\sigma_{R^ {*}(Y)}}$ (9) $$

#### Kendall τ 𝜏 \tau italic_τ corr.

Kendall rank correlation coefficient is computed as follows:

$$ $\tau_{R(Y),R^{*}(Y)}=\frac{n_{c}-n_{d}}{N(N-1)/2}$ (10) $$

where $n_{c}$ and $n_{d}$ stands for the number of concordant and disconcordant pair between $R(Y)$ and $R^{*}(Y)$, respectively.
The concordant pair $(y_{i},y_{j})$ means that their rank satisfy $\left(R(Y_{i})-R(Y_{j}))(R^{*}(Y_{i})-R^{*}(Y_{j})\right)>0$.
In practice, we employ $\tau_{B}$ which handles the ties for the rare case that some responses get the same reward scores.
The final metric is the average of the coefficient of each prompt.

#### Accuracy

The accuracy metrics are mostly the same as in the typical case that there are two responses per response. The main difference is that if there are $N$ responses per response, we can then form $C_{N}^{2}$ different pairs for comparison.

#### Bo5

The best-of-$5$ metric can be seen as a special case of NDR for $N=5$, which computes:

$$ $\frac{r^{*}\left(\mathop{\arg\max}\limits_{y_{i}}\left[r(Y)\right]\right)-\mu_ {r^{*}(Y)}}{\max\left[r^{*}(Y)\right]-\mu_{r^{*}(Y)}}$ (11) $$

This metric represents the improvement in reward values obtained using the proxy reward score compared to those achievable with the original golden reward model.
The final metric is the average of the coefficient of each prompt.

#### ECE

Expected Calibration Error (ECE) is calculated by averaging the absolute differences between predicted probabilities and observed frequencies, typically across a set of bins partitioning the prediction space:

$$ $ECE=\sum_{m=1}^{M}\frac{|B_{m}|}{n}|acc(B_{m})-conf(B_{m})|$ (12) $$

where $B$ represents the bins that split pairs by reward margins and $M$ stands for the number of bins.
$acc$ is the accuracy of pairs in each bin.
$conf$ computes the expected accuracy inferred from reward score margins by the Bradley-Terry Model (Bradley & Terry, ).
Expected calibration error indicate the alignment of reward models’ confidence.
We follow the same strategy for form preference pairs as in the Accuracy.

#### MRR

Mean reciprocal rank is a traditional Information Retrieval metric that can be transported to our setting smoothly.
We first define reciprocal rank as the golden rank of the response that receives the highest reward score.(^7^77One may expect this metric to compute the proxy rank of the response that receives the highest golden reward score. But our implementation makes it more similar to the Bo5 metric.).
Then we take the average overall prompts:

$$ $MRR=\mathbb{E}_{x\in\mathcal{X}}\left[\frac{1}{R^{*}\left[\mathop{\arg\max} \limits_{y_{i}}\left[R(Y)\right]\right]}\right]$ (13) $$

#### NDCG

Normalized discounted cumulative gain is another typical Information Retrieval metric that is transported here.
This computes compute:

$$ $NDCG=\frac{\sum\limits_{i=1}^{N}(R^{*}(Y_{i})-1)/\log_{2}(N-R(Y_{i})+1)}{\sum \limits_{i=1}^{N}(R^{*}(Y_{i})-1)/\log_{2}(N-R^{*}(Y_{i})+1)}$ (14) $$

The main difference from the typical usage in the field of Information Retrieval is that we replace relevance score $rel_{i}$ with the golden rank $R(Y_{i})$.
These metrics can be seen as a smooth version of MRR.
The final metric is the average of the coefficient of each prompt.

#### ξ 𝜉 \xi italic_ξ corr.

$\xi$ correlation coefficient (Chatterjee, ) is relatively new metric for evaluating the rank correlation.
Compared to traditional rank coefficients like Spearman corr., this coefficient is more effective to compute.
It first rearrange the data as $\left[r^{*}(Y_{(1)}),r(Y_{(1)}\right],...,\left[r^{*}(Y_{(N)}),r(Y_{(N)})\right]$ such that $r^{*}(Y_{(1)}\leq r^{*}(Y_{(2)}\leq...\leq r^{*}(Y_{(N)})$, and then compute:

$$ $\xi_{N}\left(R(Y),R^{*}(Y)\right)=1-\frac{3\sum_{i=1}^{N}|R_{(i+1)}-R_{(i)}|}{ N^{2}-1}$ (15) $$

### 8.4 Parametric Equation between Accuracy and Regret

Based on the assumption that only Regressional Goodhart takes effect and golden reward score $r^{*}\sim\mathcal{N}(0,\sigma_{r}^{2})$, the noise $z\sim\mathcal{N}(0,\sigma^{2})$, we can then derive the relationship between the accuracy and the regret.
Based on the Regression Goodhart’s, the proxy reward $r$ can represented as:

$$ $r=r^{*}+z$ (16) $$

The process of constructing an RM test set can be viewed as performing two independent samples from the distribution of the golden reward score.
Therefore, the scores obtained from the two samples can be represented as $r_{1}\sim\mathcal{N}(0,\sigma_{r}^{2})$ and $r_{2}\sim\mathcal{N}(0,\sigma_{r}^{2})$.
Subsequently, the difference between the two samples’ golden reward scores also follows a normal distribution: $r^{*}_{-}\sim\mathcal{N}(0,2\sigma_{r}^{2})$.
Then the proxy reward model score difference can be written as:

$$ $r_{-}=r^{*}_{-}-z_{1}+z_{2}$ (17) $$

where $z_{1}$ and $z_{2}$ is the noise introduced in the two times of sampling.
The distribution of noise difference is also normal distribution $z_{1}-z_{2}\sim\mathcal{N}(0,2\sigma^{2})$.
The accuracy metrics can be translated into:

$$ $\begin{split}d_{r}^{acc}&=P(r_{-}>0,r^{*}_{-}>0)+P(r_{-}<0,r^{*}_{-}<0)\\ &=P(r_{-}>0,r^{*}_{-}>0)\\ &=1-2P(z_{1}-z_{2}<0,r^{*}_{-}>0)\\ &=1-2\int_{x=0}^{+\infty}P(x=r^{*}_{-})P(z_{1}-z_{2}<x)dx\\ &=1-\int_{x=0}^{+\infty}\frac{1}{\sigma_{r}\sqrt{\pi}}e^{-\frac{x^{2}}{4\sigma _{r}^{2}}}\Phi\left(-\frac{x}{\sqrt{2}\sigma}\right)dx\end{split}$ (18) $$

As for the $d_{\pi}$, we refer the result from (Gao et al., ) that:

$$ $\mathbb{E}\left[X|X+Z=c\right]=\mathbb{E}[X]+(c-\mathbb{E}[X]-\mathbb{E}[Z]) \frac{Var(X)}{Var(X)+Var(Z)}$ (19) $$

where $X$ and $Z$ are independent absolutely continuous random variables with $X$ normally distributed and $Z$ normally distributed.
In our case, $X$ can be replaced by $r^{*}$ and $X+Z$ can be replaced by $r$:

$$ $d_{\pi}=\frac{\sigma_{r}^{2}}{\sigma_{r}^{2}+\sigma^{2}}$ (20) $$

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - Randomly flipping some percent of the trains in the training dataset is a trick to make a pseudo proxy reward model used in several papers (e.g., AlpacaFarm; Dubois+ 2024). In reality, it is more natural to think that some kinds of instructions have more flips and others have less. I'm a bit concerned that the findings of the paper might only hold true due to the synthetic error model. It would be helpful to see whether this possibility is addressed or refuted (or let me know if I missed it).
soundness: 4
strengths: Overall, the paper presents empirical results that evaluate the metrics of RMs, which the community has previously intuited but not with rigorous scientific evaluation. The experiments are designed to test hypotheses incrementally, helping the RLHF community build a comprehensive body of knowledge on RM evaluation.

- The paper tackles the question that the alignment research community needs to know the answer to.
- Table 2 is interesting as it shows counterintuitive results. One would guess that the RM should be evaluated for the samples generated from the policy to be trained. The paper also shows that instead, we should sample multiple responses and choose a pair of responses to evaluate the accuracy with care.
summary: Research question of the paper is on how we should evaluate the quality of the reward models for RLHF. The paper conducts experiments to evaluate the evaluation metrics for RMs and show interesting findings that some confirm the intuition and the other are somewhat counterintuitive. The paper concludes that there should be more care on evaluating RMs instead of relying on a single benchmark.
weaknesses: I don't see any critical weaknesses for the paper. If I were to come up with the weaknesses:

- Although policy regret is often referred to in the paper, its formal definition is not clearly stated. It would be better to have an equation defining the regret. Even if we do not have a way to compute it, the goal of the research is to estimate it so I would say that it is worth clarifying its definition formally.
- The scope of the paper is to show that the current evaluation scheme is not enough (which is a good enough contribution). The paper does not provide a solution to the problem of how we should evaluate the RMs (which I think is asking too much).
- The texts in Figures are a bit too small to read. It would be nice if it is a bit larger.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: See questions above as well.

Fig 3 shows high correlation between accuracy and BoN perf at high accuracy. Do authors have an intuition on why that might be the case? Also, no RMs have an accuracy between 0.75 and 0.8, which might indicate a lack of diversity in the RMs trained, echoing an earlier remark on experimental design.

“As shown in Table 2, this approach does not consistently improve upon the original RewardBench dataset. This result suggests that sampling responses from the model used for optimization may not be necessary to achieve a strong correlation between accuracy and policy regret.” -> this is quite surprising. Do authors have an intuition here?
soundness: 3
strengths: The problem tackled is well introduced and presented.

Contributions are clear.

Having a synthetic and cheap setting that correlates with more realistic experiments is quite valuable for the research community (but see doubts expressed below).

Some very interesting findings in this setting:
* weak correlation between RM accuracy and downstream policy performance/regret
* correlation increase by increasing the number of answers per prompt
* correlation increase by picking answers based on their rank
* correlation increase by increasing the number of test samples

EDIT: I raised my score accordingly after convincing updates from the authors.
summary: This paper revisits the soundness of current reward model evaluation procedures, which assess the accuracy of RMs over datasets of held-out preference data, showing they present the flaw not to necessarily translate into downstream improvement for models going through RLHF in a synthetic setting. They propose to improve RM evaluation by increasing the number of responses per prompt and/or (if available) using the rank of model responses to select preferred and rejected responses when building preference datasets.
weaknesses: The key limitation from my perspective is that we have currently no solid evidence that the study will translate to real settings: diverse, heterogenous RMs potentially trained on different datasets. While this would be infeasible to run as many experiments and ablations as in the synthetic setting, showing that there is imperfect correlation between RewardBench scores and relevant, downstream RLHF settings (more generally, that some findings from the synthetic study do replicate) would make the paper much stronger. At least showing that RewardBench rankings are not preserved after downstream BoN / RLHF would be a great addition. 

In that regard, the fact that RMs used in the experiments are essentially the same architecture / same init / same hyperparameters trained on very similar data (i.e. up to random flips) is concerning: this is not capturing the diversity of RMs being trained and evaluated on popular benchmarks such as RewardBench. Authors should do a better job at showing that this is enough to capture phenomena that replicate in real settings.

Also, what about using a larger RM as proxy-golden? Seems like the default practice and would make for a more principled study, as this would likely reduce RM similarity with the proxy-golden RM.

Another limitation is that it is currently hard to evaluate whether the correlation increase authors get from applying the proposed improvements are significant or not. What is the correlation between proxy-golden RM accuracies on train and test data? This value would constitute an upper bound of what level of correlation is realistically achievable and provide a reference value that would alleviate the above concern.

Several ablations are missing AFAICT:
* Why is correlation higher for BoN vs PPO? Authors state that “This is expected, as BoN is a more localized and stable optimization algorithm, making it more predictable by reward model error.” I think this might be due to using unconstrained PPO (i.e. without KL regularization), which begs for an ablation study.
* Did authors conduct any form of investigation on low accuracy but high NDR PPO policies from Fig. 3 b)?
* It is currently unclear whether adding more samples (i.e. more than 5000 samples) in Fig 6 would improve correlation or instead saturate? This experiment would be a compelling addition to the study.
* Regarding using additional responses per prompt: the question of whether the policy matches or not the downstream policy matters or not is not studied in the current state
* What about using a different set of prompts to quantify whether the difference between test RM prompts and test RL prompts is meaningful?

Notations and equations:
* equation 2 uses $\pi’$ on LHS but not present on RHS -> please fix
* what is \pi_0 in equation 2? I suppose it is the initial policy, but this should be clarified in the main text
* is the regularization term from Equation 3 known/standard? if so, clarifying and including a citation to prior works is warranted 

Clarity can be improved a lot:
* “on the widely adopted benchmark dataset (Lambert et al, 2024)” -> authors should name it (RewardBench)
* “Regarding response distribution, we find that the rank of the responses can be more influential rather than the model from which it is sampled.” -> unclear sentence
* “The translation from the RM error to the policy regret can be seen as the result of the reward model overoptimization” -> “imperfect translation”? or “weak correlations”? unclear sentence for now
* “ the difficulty of constantly controlling optimization pressure” -> what do the authors mean by pressure here?
* is the Normalized Drop Ratio (NDR) a contribution of the authors’ work? if so this should be clarified in the text
* the use of NDR (as opposed to difference in average rewards) should be motivated better in the text, even if easy to interpret
* Fig 3 is a bit hard to read (dots / text could be made bigger)
* /!\ Using more responses per prompt to improve RM evaluation -> This is a key aspect of the paper that is quite unclear in the current state! As of now the details are in the appendix, but the main text should be much clearer about this aspect, notably that the fact that we have a golden-proxy RM allows us to estimate a correlation between the proxy rewards and the golden-proxy rewards.
* “Can we achieve a higher correlation by sampling responses specifically from the model used for downstream optimization? To examine this question, we construct test datasets with responses solely from different models.” -> I find these sentences puzzling as they seem to contradict each other, please clarify

Writing is sub-par in the current state, see examples:
* “The inherent difficulty of constructing an ideal RM require”
* “The latter, while straightforward, remains the question of whether such evaluation accurately predicts the performance of the downstream optimized policy”
* “A cartoon of RM error and policy regret”
* “We begin by investigate the influence of prompt and response distributions”
* “The correlation between policy regret and accuracy on datasets constructed from responses of different ranks assessed the by Spearman coefficient”

with more in the text.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: * How do the results here relate to the findings in "Preference Learning Algorithms Do Not Learn Preference Rankings" (NeurIPS 2024)?
* Where does the data used to evaluate the policy and measure policy regret come from?
* How do the results about prompt versus response distributions relate to the paper "On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization" (Lin et al., 2024)?
* It seems the paper does not account for noise. Why is that not important here?
* In equation (2), what is $\pi_{0}$? Is this the SFT/IFT'ed model?
* What is your motivation for setting the KL penalty to 0 (line 160)?
* For the analysis of Figure 3 supporting the conclusion that  "policy regret can very considerably even within similar accuracy ranges", is it possible that accuracy and NDR exist on different scales making it look like there is more variation along one dimension that the other? Can you report the results normalized so that they fall on the same scale?
* For the results reported in Figure 3, how OOD versus ID is the data for each the reward model and the policy?
* How were the bins that were used to assign response rankings determined? Was the ground truth reward model for that experiment used?
* It is surprising that seeing higher reward samples was not more beneficial. Can you elaborate more on why this is the case?
* For Figure 4, what does it mean when the bin is 0 for both the x and y axis? Does it mean the two responses are of equal rank? In this case, how does labelling work?
* For the results looking at the impact of response rank, how was the label flipping or disagreement with the ground truth reward function's training data accounted for? For the flipped labels spread uniformly over the rank bins? 
* Why is Table 3 not symmetric along the diagonal?
* What is the hypothesis for why extra responses help in the case of BoN and not PPO?
* In "We observed that there should be an approximate linear relationship between them" (line 454), "observed" + "should be" in unclear. You assume? You know? Also how? Why?
soundness: 3
strengths: * Understanding the relationship between our measures of reward quality and policy quality is of vital importance.
* The goal and motivation for the work is well laid out at the start of the paper. The takeaways the reader should expect to have are presented from the start.
* The use of a synthetic ground truth reward function is well motivated and contextualized.
* Different methods for using a ground truth reward function (best-of-n and RL) are compared.
summary: The paper questions the current approach to evaluating reward model performance, which is based evaluating downstream policy performance and looking at reward model accuracy on the train and eval preference datasets. The relationship between reward model performance and policy performance is explored in the context of policy regret, a new performance metric that examines the difference in policy performance when trained with the true versus inferred reward. Since the true reward is not typically available, a synthetic true reward model is used to conduct the analysis. The reward model and policy performance relationship is measured as the correlation between the reward model's accuracy and the policy regret. Evaluation relies on RewardBench. The authors find a weak positive correlation between the two, and conclude that accuracy alone is a poor proxy for downstream performance.
weaknesses: **High level**

The main weaknesses for this paper are not overly large and mostly involve clarifications to the text. While not big changes, they are important to address. Some of the conclusions in the main body need to be walked back and made more nuance to fully reflect the presented results. The biggest missing result is information about the relationship between the ground truth reward model and the proxy reward model. 

**Details**

* The experiments are set up such by taking a labelled preference dataset and then randomly flipping labels to create multiple preference datasets. A different reward model is then trained on each version of the dataset. The ground truth reward model is then arbitrarily chosen to be one of those and all others are treated at the proxy. It would be good to quantified the relationship between the ground truth and proxy reward models as part of the analysis. This could be as straightforward as looking at the percentage of agreeing labels in the training data.
* It would improve understanding of the experimental section to describe the RewardBench data and how it is used earlier. The very start of Section 3, would be a good place.
* It would  strengthen the results and analysis to include policy regret information where the ground truth reward is used as the proxy reward. This would help to understand the impact of randomness in policy learning. In practice, is the policy regret 0?
* Some of the statements made early in the paper (e.g. introduce) are difficult to interpret in the absence of having read the whole paper. For example, "...we find that  the rank of the responses can be more influential..." and "...increasing the number of responses per prompt". It would be helpful to add some description about what is meant by rank and what it means to increase the number of responses in terms of the evaluation.
* It is challenging to interpret the results in Figure 4. Adding axis names would be beneficial. 
* The conclusion that "While the correlation on PPO continuously weakens as we paragraph a larger number of prompts", the nuance that this depends on the correlation metric should be called out, especially as no single correlation measure has been identified as "best".
* Please add more descriptions of the different methods used to evaluate Finding 4. Specifically things like "Bo5" where a citation would also be helpful.
* It is not clear how all of the metrics in Table 4 were computed. For example, what is the Pearson corr. measured between?
* The conclusion about the impact of number of responses per prompt does not fully reflect the results in Figure 7 (a). For the smaller annotation budgets, the benefit of extra responses drops off quickly. The trends for PPO should be summarized and described and extra responses are not beneficial.
* The paragraph immediately after "Finding 2" (lines 238 - 244) were not clear to me, so it is difficult for me to assess or validate them. Parts such as "solely from different models" was not clear. It is not clear exactly how the data is different from what was used previously.
* Some small spelling issues, typos, and confusing phrases throughout the paper. These are not impacting my score, would be good for the authors to clean up.
     * line 071 "investigate" -> "investigating"
     * line 089 "This offers valuable for both reward model training...."
     * line 129 "...we focus on a few interested factors and keep the rest fixed" - "interested" -> "interesting"
     * line 134 "...perform investigation..."
     * line 319 "...expect calibration error..." -> "...expected calibration error..."

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Do you have any potential solutions for the issues you have identified (how do we go beyond RM accuracy)? 

2. Can you clarify if I am missing something re: my major concern above, or make me doubt the view that "RM accuracy is a better measure of expected downstream regret than single seed downstream regret". An experiment that shows insufficient correlation given multiple seeds might do the trick (even if you can't do PPO, you should be able to run this experiment quickly in the BoN setting).
soundness: 2
strengths: The research question is a good one. That is, it is known from the RL literature that the best reward models for learning are not necessarily the ground truth reward models (e.g. Singh et al. 2009, Where Do Rewards Come From?), so it seems natural that some proxies of ground truth rewards (well, in case of LLMs, trajectory returns) will be better than other proxies of ground truth rewards --- can we ascertain whether this is the case, and if, so can we find ways to craft better proxies and/or better measures of proxy quality. I think this is an important question of great interest to a variety of researchers. I also think it has not really been studied in the case of language reward models, so it is good to have a paper that prompts the discussion. 

To answer this question, this paper contains several experiments that show the relationship between RM accuracy and downstream policy performance. The questions that drive the experiments are generally interesting. There are a lot of experimental outcomes here, which may be interesting to different researchers. Although some thinking is necessary on part of the reader to understand what the authors are doing, the paper is fairly written through Section 4.
summary: This paper empirically investigates whether "reward model (RM) accuracy" (i.e., agreement with a preference dataset), which is commonly used for evaluating reward models, is a good metric for evaluating reward models. It begins with the premise that the true measure of a reward model is the quality of the policies it produces, and proceeds to investigate the relationship between RM accuracy and the regret of the downstream policy. It does this by synthesizing several "noisy" reward models, which can each serve as both a "gold" reward model providing ground truth labels for both preferences and policy performance, or a proxy reward model (when another RM is the gold reward model). The authors then proceed to conduct several empirical investigations into the RM accuracy vs Policy regret. They conclude that (1) RM accuracy is not perfectly correlated with downstream policy regret, (2) the predictive power of RM accuracy declines as the distribution on which policy regret is measured shifts away from the original distribution, and (3) ultimately conclude that RM accuracy is not adequate as a measure of RM quality.
weaknesses: Issue: The fact that your hyperparameters for training the policy are held constant does not imply that the policy extraction / learning process is not noisy. So any characterizations of the correlation as being "weak" (L63, L530) or "room for enhancing" (L213) are IMO unsubstantiated, as the upper bound for correlation may be much lower than 1 due to variance in the policy optimization process. This noise also explains why correlation with PPO is weaker than correlation with BoN (which you correctly point out at L190). *So the major issue I have is that, even after reading this paper, I expect RM accuracy to be a better measure of expected downstream policy quality than single seed downstream policy quality.* (and this is the main cause for my low review score; the experiments do not convince me that RM accuracy is an insufficient target for designing RMs)

Although the experiment at L246-256 / Figure 4 is interesting, the descriptions are insufficient to properly understand Figure 4 (which axes is which, how does (a) map on to what you wrote in the text?). Further, what is the practical import of this experiment --- would we need humans to rank multiple samples for this to be relevant? And if so, wouldn't it then be better to just use all the samples as you investigate later on? 

Finding 4... it would be good to provide an intuition for the reader as it was not immediately obvious to me why this is happening. Namely that the number of comparisons (and therefore samples with which to reduce variance) grows factorially in the number of responses. Once stated this way, I would note that past work has used multiple responses/ human labels per prompt, see e.g., Hwang et al. 2023, Sequential Preference Ranking for Efficient Reinforcement Learning from Human Feedback and Wu et al. 2023, Fine-Grained Human Feedback Gives Better Rewards for Language Model Training. (This is not to say Finding 4 is not interesting, but I would still rather see an analysis of how the annotation budget affects the RM accuracy, as opposed to the downstream Policy Regret). This all being said, perhaps if this finding were fleshed out more, to show how you can use it to improve RewardBench, it could be a lot more impactful. 

Section 5: I'm afraid I do not really understand this section. I'm not sure the Goodharting / overoptimization / exogenous variable language is doing much good here (what would be an example of an exogenous variable impacting the relationship, and why is implied by the experiments here?). It seems you are assuming that rewards follow a Thurstone Choice model (whereas most literature assumes Bradley-Terry-Luce; which is fine, they are very close), and then in Figure 8(a) comparing the percentage of reward variance owed to the ground truth reward (vs noise term) to the RM accuracy. What is Figure 8(b) showing? How do we know what $d_\pi$ is in the "actual" setting? I don't understand what the takeaway from Figure 9 should be, if we can't see the reward accuracy, and don't know the sample size under which it computed, etc. There is detail missing here.

Important (but minor review-wise): 
L139: alpha is never specified (except in Fig 9)
L159: n is never specified!
And generally, I think there are details left out / I would not be able to reproduce everything given current manuscript; e.g. what hparams were used for PPO, etc. How are confidence bands in Table 2 computed, etc. 

Minor:
- L141: N x (N-1) pairs
- L48: "human golden reward function" --> "empirical human rewards"  (the human reward 'function' is noisy, and we cannot quantify the error between learned RM and the human reward fn) 
- L50: "the goal of RM eval is to estimate policy regret" (no, it's to estimate the quality of the reward model --- policy optimization introduce a whole range of additional noise / issues -- see commentary above) 
- L429: I would not say the entire translation is due to overoptimization... in fact, there may be NO overoptimization if we do early stopping, right?

Meta-review: This paper questions the current evaluation procedure of reward models in LLM post-training. It shows that the accuracy of reward models does not necessarily translate to the improvement in downstream RLHF tasks.

Strengths:  
This paper studies an important and well-motivated problem — reward modeling is important in LLM post-training like RLHF, but hasn't been deeply investigated. This paper presents several interesting observations through comprehensive experiments.

Weaknesses:  
The reviewers have concerns about weaknesses around paper presentation and ablation studies. Most of the concerns were addressed during the rebuttal period.

All reviewers agree that this paper studies an important problem and obtains interesting observations. I agree with this assessment and believe this paper would be a good addition to the community.

Decision: Accept (Spotlight)