Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?
Authors: Xueru Wen
Jie Lou
Yaojie Lu
Hongyu Lin
XingYu
Xinyu Lu
Ben He
Xianpei Han
Debing Zhang
Le Sun
Venue: ICLR 2025 Spotlight (2025)
Keywords: Reinforcement Learning from Human Feedback; Reward Model;
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Reward Models (RMs) are crucial for aligning language models with human preferences. 
Currently, the evaluation of RMs depends on measuring accuracy against a validation set of manually annotated preference data.
Although this method is straightforward and widely adopted, the relationship between RM accuracy and downstream policy performance remains under-explored.
In this work, we conduct experiments in a synthetic setting to investigate how differences in RM measured by accuracy translate into gaps in optimized policy performance.
Our findings reveal that while there is a weak positive correlation between accuracy and downstream performance, policies optimized towards RMs with similar accuracy can exhibit quite different performance.
Moreover, we discover that the way of measuring accuracy significantly impacts its ability to predict the final policy performance. 
Through the lens of the Regressional Goodhart effect, we recognize that accuracy, when used for measuring RM quality, can fail to fully capture the potential RM overoptimization.
This underscores the inadequacy of relying solely on accuracy to reflect their impact on policy optimization.

Published as a conference paper at ICLR 2025
RETHINKING REWARD MODEL EVALUATION: ARE WE BARKING UP THE WRONG TREE?
Xueru Wen Jie Lou
Yaojie Lu Hongyu Lin-
Xing Yu?
Xinyu Lu
Ben He? Xianpei Han Debing Zhang"
Le Sun
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences
2 University of Chinese Academy of Sciences
8 Xiaohongshu Inc
(venxueru2022, 1ixxiyy222211iisssssss... ac.
soyyajje,,ongyye xianpei, sunle) @i saciaass.coo
bbenhe)@ucas.acco.
Looujie0822)@gmail.c
denngaaggJxxlachonnghh..
ABSTRACT
Reward Models (RMs) are crucial for aligning language models with human pref- erences. Currently, the evaluation of RMs depends on measuring accuracy against a validation set of manually annotated preference data. Although this method is straightforward and widely adopted, the relationship between RM accuracy and downstream policy performance remains ander-explored. In this work, we con- duct experiments in a synthetic setting investigate how differences in RM mea- sured by accuracy translate into gaps in optimized policy performance. Our find- ings reveal that while there weak positive correlation between accuracy and downstream performance, policies optimized towards RMs with similar accuracy can exhibit quite different performance. Moreover, we discover that the way of measuring accuracy significantly impacts its ability to predict the final policy per- formance. Through the lens of the Regressional Goodhart effect, we recognize that accuracy, when used for measuring RMM quality, can fail to fully capture the potential RM overoptimization. This underscores the inadequacy of relying solely accuracy reflect their impact policy optimization.
INTRODUCTION
Reinforcement Learning from Human Feedback (RLHF) (lbarz et al.
2018 g
Ouyang et al. 2022
has emerged as a prominent paradigm for aligning Large Language Models Yang et al. 2024 Dubey et al.) 2024). The reward model (Ng & Russell 2000 Brown & Niekum 2019 Palan al. 2019) plays a crucial role in this process by substituting human preferences for model optimization. However, building RM that fully captures human preferences is highly challenging (Armstrong & Mindermann 2019 Skalse & Abate 2023: Lambert et al.) 2023). Therefore, the RM can be an imperfect proxy for ideal preferences and cause downstream performance deterioration when optimized against it, known as reward model overoptimization (Gao et al. (022). This phenomenon, as result of Goodhart's 's law (Karwowski et 2023), presents a critical challenge the RLHF
The difficulty of constructing an ideal RMM require a reliable evaluation to capture potential negative impacts policy optimization. To date, common practice for evaluating the RM includes directly assessing the optimized policy (Hendrycks et al. 2021 Li et al. 2023) and computing accuracy on fixed dataset (Lambert 2024). While the former approach serves as a final metric, it is constrained by the cost of optimization and cannot distinguish whether undesirable behaviors stem from the policy optimization process or the reward learning process. The latter approach remains the question of whether such evaluation accurately predicts the performance of the optimized policy.
"This work was done when Xueru Wen interned at Xiaohongshu.
1 Corrrsponniig author.

---

Published as a conference paper at ICLR 2025
In this work, we systematically investigate the effectiveness of using accuracy in predicting down- stream performance. Essentially, the accuracy metric measures the difference between two reward functions, referred to RM error. In practice, the accuracy quantifies the eπror between the learned reward model and the empirical human rewards. The RM error can lead to potential performance loss between the policies optimized for the proxy RM and the golden reward function. Specifically, we define this performance gap as Policy Regret. Therefore, the goal of RM evaluation is to pre- dict policy regret through RM error, as depicted Figure []] To investigate how differences in RM measured by accuracy translate into gaps in optimized policy performance, we design a synthetic experiment framework. Due to the inaccessibility of human reward functions, we employ synthetic RM as the golden reward function in our experiments. To effectively collect golden-proxy RM pairs for analysis, we create N different RMs, designating one RM as the golden model and the others as proxy models each time. Based on the above framework, we decompose the issue concerning RM evaluation into three research questions (RQ):
Does the RM error measured by accuracy cor- relate with policy regret? We examine the correlation between accuracy and policy regret the widely adopted RewardBench dataset Lambert et al. (202)). We employ optimization algorithms, i.e., best-of-n sampling and pol- icy gradient-basee reinforcement learning, for investiga- tion. Our findings reveal weak positive correlation be- tween the measured accuracy and the policy regret. How- ever, we observe that policies optimized towards reward models within a similar accuracy range can have quite different regrets.
RQ1:
Policy ae
Golden RM
Initial Policy
RM Error
Policy Regret
W0
Proxy RM
Policy up
Figure 1: Translation from RM error to policy regret.
RQ2: How to better measure RM error for policy regret prediction? While we present a positive correlation between the accuracy and the policy regret, there remains room for further enhancement. This naturally raises the question of how to better quantify the RM error to achieve a stronger corre- lation. We begin by investigating the influence of prompt and response distributions. Specifically, we observe that the prompt difference between the RM test dataset and the downstream test dataset will potentially decrease the correlation between the accuracy and the policy regret. Regarding response distribution, find that the rank of responses evaluated by the golden RM impacts correlation more significantly than the model from which they are sampled. Furthermore, we propose a simple yet effective strategy enhance correlation: increasing the number of responses per prompt in the test dataset. We validate this approach under constraints of equal sample size and annotation budget, finding that metries based on more responses achieve higher correlation with policy regret.
RQ3: What's the relationship between RM error and Policy Regret? The translation from RM error to policy regret stems from the overoptimization of the reward model. Specifically, optimizing toward an imperfect proxy RM leads to deterioration in downstream performance. Therefore, to predict the policy regret, the quantified RM error should be able to reflect the potential overoptimiza- tion. In this work, we derive the expected relationship between the accuracy and the degree of reward overoptimization under the assumptions of Regressional Goodhart's s effect (Manheim & Garrabrant, 2019) and normal distributions of both reward score and noise. However, our findings indicate that RMs with similar accuracy can behave quite differently in terms of overoptimization. This indicates accuracy alone can be an inadequate metric for predicting the downstream performance.
In summary, our work provides deeper understanding into the relationship between RM error and policy regret. This offers valuable insights for both reward model training and the development of more robust RLHF algorithm. Our results also highlight the need for a more rigorous benchmark in the evaluation of reward models and the development of advanced RM assessment tools.
2 PRELIMINARY
Let X' be the set of prompts and V/x be the set of possible responses. A policy is a language model that generates responses y to the prompt x at probability Tfy(z). The golden reward function assesses the response and gives a score based on their quality r" X Vxx R. In practice, the golden reward function r represents complicated human preference and is generally inaccessible. Instead, reward model learned from preference data serves as proxy for the golden reward
2

---

Published as a conference paper at ICLR 2025
Pollcy Its
Synthetic Noise
RLHF
Polley m22
Policy its
RM
Policy
Palicy
Policy
Preference Dataset
Inttaa Poliyy
Initial Policy
RM
Policy N;
Policy Xe
Policy
Golden RM faom Ty 88 Proxy
Golden RM
RM
Policy
Rest Fa Py (88 Proxy
(a) Synthetic pipeline of Reinforce- ment Learning from Human Feedback.
Proxy-golden RM pairs collection for evaluating the correlation between the RM error and policy regret.
Figure 2: Overall experiment framework.
function r" Then policy x is optimized to maximize the expectation of the reward given by r:
π max ExeX (1) [r(2, y)1
(1)
This results in a sub-optimal policy rather than the ideal policy x² that directly maximizes the reward given by the golden reward function r*.
The workflow of Reinforcement Learning from Human Feedback (RLHF) can be represented as:
ARM CaarlDrra (Dirain)
Learn proxy RM r on the preference dataset Dirai" by algorithm ARM-
. .M(r, r" Diawf) + d,: Evaluate the RM error between proxy reward model r and golden reward function r" by metric M on the test dataset Drest
(Ro r| DR + क: Optimize the initial policy no and get policy π that maximizes the expectation of reward given by the proxy RM by the algorithm ARL-
x' dz: Measure the regret between the policy and the policy x² that opti- mized directly towards r by metric M on the test dataset Drest
In this work, we essentially focus on the relationship between the error of reward model d, and the regret of policy dx, which translates our issue into: How should we measure the error d, so to reach high correlation with the policy regret d,, ? Various factors introduced in the RLHF process can potentially influence this relationship. To systematically investigate their impact, we focus on several key factors while keeping other variables fixed. The test dataset Diest on which the reward model r is evaluated; The metric M used to evaluate the error between the reward models; The test dataset Dies on which the final policy æ is evaluated.
One challenge in conducting the investigation is the inaccessibility of the golden reward function representing human preferences. To address this difficulty, we follow the previous practice (Gao
ct al. 2022 2024) of utilizing another reward model to act as the golden reward function.
Another challenge is to effectively collect the proxy-golden RM pairs for evaluating the correlation between the RM eиrоr and policy regret. We implement this by constructing N = 10 different reward models T2 by randomly flipping a, of the pairs in the training datase Dirain. Then N policies m, will be optimized towards the corresponding reward model ri- This pipeline is demonstrated in Figure 2a This operation will form N x (N 1) golden-proe -pooxypox RM pairs. As shown in Figure 2b these pair collections can then be used for computing correlation by letting one reward model as the golden reward function and the remaining reward models as proxy reward models.
Building on prior works that formally define policy regret in the context of general Markov Decision Processes (MDPs) [Karwowski et al. 2023; Fluri al.) 2024). we adapt the concept within the framework of RLHF Given that the KL divergence between policy π and the initial policy is
We constructed the data such that if a training set has o; of its labels flipped and another has o, flipped, then Ja, 151% of the labels inconsistent hetween the See Appendix 8. for dataset details.
3

---

Published as a conference paper at ICLR 2025
KL(TT/TT) X, the regret with respect golden reward function r" is defined as:
max (Lox
J. (TT) - J.. (TT)
Reg.-
max K [l(r'I|=o)<A
(TT'
min KL(T'1[T)<<
(T-(
Here, the function J,- (- (r" y)] represents the expected reward given the policy a and the golden reward function r".. This value reflects the ratio of the maximum reward gain a policy can achieve to the actual reward gain obtained, considering the constraint of KL divergence.
However, this definition is hard to compute due to the inaccessibility of the optimal policy and the difficulty of controlling KL divergence X. Therefore, we propose Normalised Drop Ratio (NDR) computable estimation follows:
MNDDRTT,T') Jr- (T) - J,- (xo) (TT) J.,((T))
where the policies и and n² are achieved by optimizing initial policy against the golden reward function r" and the proxy RM with exactly the same hyperparameters. This metric quantifies the performance deterioration resulting from optimizing against a proxy RM instead of the golden reward function r". also accounts for variations due to different scales among different RMs.
The data from the RewardBench (Lambert et al. 2024) are used to construct the test datasets RM
D test
and Diest. All RMs are initialized from the Llama-3 --iiaa----ttttt--- model (AI@ 2024) and
finetuned by minimizing the negative log-likelihood loss with regularization term (Hou et al. 2024):
LRM = - Ex. (x) (r(g( the) r(r, v())] B(zzg)>DDrrg [r(x, y)
(4)
We adopt best-of-n sampling (BoN) and PPO (Schulman et al. 2017) as the algorithm A to opti- mize the initial policy mo, which also the Llama-3-instruce the reward model picks the response with the highest reward from candidates. For the PPO algorithm, we follow (Gao et al. 2022) and set the KL penalty in all experiments to be 0.
model. For the best-of-n sampling,
DOES THE RM ERROR MEASURED BY ACCURACY CORRELATE WITH THE POLICY REGRET?
Current research typically assesses reward model errors by computing accuracy fixed test set. Although widely used, it remains unclear whether the accuracy correlates with policy regret. this section, evaluate the accuracy and policy regret using the RewardBench (Lambert et al., 2024) dataset. We perform deduplication on the prompts, leaving 2,733 distinct prompts.
RM evaluation accuracy is positively related to policy regret, but even with similar RM accuracy, policies can exhibit different levels of regret.
Finding 1:
We first investigate the correlation between accuracy and policy regret. Original RewardBench prompts and responses are used to measure accuracy, and the prompts are used for downstream optimization. We calculate the correlations between RM error, measured by accuracy, and policy regret, measured by NDR. in Table [ As presented in Figure Ba and Figure 3b there is positive relationship between accuracy and policy regret. However, trends illustrate that policy regret can vary considerably even within similar accuracy ranges. Lastly, we observe that accuracy generally correlates more strongly with regret in BoN than PPO. This is expected, as BoN is a more localized and stable optimization algorithm, making it more predictable by reward model error.
4 HOW TO BETTER MEASURE RM ERROR FOR POLICY REGRET PREDICTION?
In the previous section, examine the positive correlation between accuracy and policy regret. However, there appears to be room for enhancing this correlation, which leads us to the question: how to better quantify the RM emorr? In this section, we first investigate the influence of prompt and response distribution. Moreover, we explore a straightforward yet effective strategy, i.c., extending responses per prompt. Finally, we validate it under different constraints.
4

---

Published as a conference paper at ICLR 2025
The correlation between the accuracy (Acc.) and the policy regret under BoN and PPO measured by typical correlation coefficients and Mean Reciprocal Rank (MRR). MRR is calculated by averaging the reciprocal ranks policies optimized by the RM with the highest accuracy.
Table 1:
Relevance
Error / Regret
Kendall corr.
Pearson coIT.
Spearman corr.
MRR
Acc.
BoN
0.6561
0.7520
0.7533
0.6333
Acc.
PPO
0.4654
0.6395
0.6102
0.5167
Spearman corr. 0.7533, MRR: 0.6333
2.5
Spearman con:: 0.6102, MRR: 0.5167
0.7
0.6
1.5
0.5
0.4
0.3
0.0
0.2
0.1
0.70
0.75
0.80
0.85
0.90
0.70
0.75
0.80
0.85
0.90
dpcc
dace
(a) Trend between accuracy d and the policy regret d ND under BoN setting.
(b) Trend between the accuracy d and the policy regret d, DR under PPO setting.
Figure 3: Trend between the accuracy and policy regret measured by the normalised drop ratio.
The rank of responses affects the correlation between accuracy and regret more than the response sampling models.
Finding 2:
In the previous experiment, we used the original responses in RewardBench for validation, which possible to achieve
were sampled from different models. This naturally raises the question: higher correlation by sampling responses exclusively from the model used for downstream optimization? To examine this question, we construct multiple test datasets, each containing re- sponses exclusively sampled from a single downstream model.. We then evaluate the correlation between policy regret and computed accuracy. As shown in Table 2 this approach does not consis- tently improve upon the original RewardBench dataset. This result suggests that sampling responses from the model used for optimization may not be necessary to achieve a strong correlation.
While it seems unnecessary sample from models used for optimization, we observe varying corre- lations in datasets constructed with responses by different models. This indicates that other factors, such the rank of chosen and rejected samples, may at play. To examine this assumption, we prepare multiple responses for all prompts. Each time, we sort the responses by golden rewards given by one RM, and divide them into bins. We then randomly sample accepted and rejected responses from different bins to construct test datasets with responses of varying ranks. The accuracy on these test datasets is used to assess the correlation with policy regrets. As demonstrated Figure 4a| and Figure 4b different trends emerged for BoN and PPO. For BoN, test datasets with chosen samples from mid-rank bins and rejected samples from lower-rank bins exhibited higher correlations. In contrast, for PPO, datasets with chosen samples from higher-rank bins and rejected samples from mid-rank bins showed a stronger correlation. This distinction likely stems from the characteristics of the optimization algorithms. With PPO, the policy is optimized more aggressively, causing
5

---

Published as a conference paper at ICLR 2025
Table 2: The correlation between policy regret and accuracy on datasets with responses sampled from different models. The Origin represents the result on the RewardBench dataset. The highest results are highlighted in bold. The model used for downstream optimization suffixed with *
BoN
PPO
Model
Spearman corr.
MRR
Spearman corr.
MRR
Mistral-7B
0.680=0.042
0.579=0.077
0.580#0.033 007
0.573=0.073
Llama3-8B
0.644+0.038
0.598#0.080
0.648+0.047
0.551#0.061
Qwen2-7B
0.680=0.033
0.529#0.074
0.603#0.039
0.59920.060
Vicuna-7b
0.703=0.035
0.658+0.075
0.527=0.036
0.517=0.068
Origin
0.753
0.633
0.610
0.517
bin
0.85
0.65
8-
8-
0.55
8-
0.50
9
0.45
1.0
10
20
25
Rejected response rank bin
Rejected response rank bin
(a) The correlation trend BoN.
(h) The correlation trend PPO.
Figure 4: The correlation between policy regret and accuracy on datasets constructed from re- sponses. Each grid on the x and y axes represents a sampled bin, with the values below indicating the corresponding rank. For example, the on the y-axis indicates that the chosen responses are sampled from those ranked between and 10.
responses to rank higher under the golden RM, making the consistency between the proxy RM and the golden RM on high-reward samples more critical. Conversely, for BoN, where the degree of policy optimization is less intense, consistency on mid-reward samples can be more instructive.
The prompt difference between the RM test dataset and downstream test dataset can weaken the accuracy-regret correlation.
Finding 3:
The previous experiments assume that the RM and RL test datasets consisted of the same prompts, which may not be true in practice. To investigate the influence of prompt category, we split the test dataset based the prompt categories determined by the original classifications in Reward- Bench. We evaluate accuracy and regret in each category and demonstrate the results in Table 3 We find that accuracies in each category align more closely with regret in the corresponding category under the BoN setting. However, this relationship is not present in the PPO setting. This may be because different types of prompts influence each other during PPO optimization.
To further investigate the influence of prompt semantics, we randomly rewrite some prompts in the original test dataset, creating new test datasets with prompts different from the downstream dataset. We ask GPT-4o (OpenAl 2024) to perform two different strategies these prompts, detailed in Appendix 8.2 One strategy alters the expression without changing the meaning, and the other generates new prompt within the same category. As shown in Figure 5 we find that the correlation is less affected by paraphrasing under BoN. However, the correlation measured by the Spearman coefficient PPO continuously weakens paraphrase larger ratio of prompts.

---

Published as a conference paper at ICLR 2025
Table The correlation between policy regret and accuracy on the dataset composed of prompts from different categories evaluated by the Spearman coefficient. The highest result in each column bolded, and the highest in each row underlined.
Accuracy
Regret
Chat
ChatHard
Code
Math
Safety
Chat
0.529+0.082
0.682+0,058
0.573+0.050
0.557+0.052
0.657+0.044
ChatHard
0.493=0.089
0.682+0.053
0.583+0.008
0.538+0.066
0.655+0.0551
BoN
Code
0.504=0.095
0.634+0.059
0.717=0.043
0.456++00060
0.646+000053
Math
0.288 121
0.343
0.244+0.048
0.610:0.058
0.282+0.059
Safety
0.515=0.093
0.705#0.057
0.521=0.047
0.497=0.067
0.687+0.049
Chat
0.349#0. 105
0.500=0.086
0.441#0.054
0.192+0.083
0.611#0.061
ChatHard
0.314#0.115
0.484+0.083
0,434+0.057
0,202+0.087
0.576+0.073
PPO
Code
0.384+0.092
0.450#0.080
0.527=0.051
0.371+0.060
0.459+0.053
Math
0.185=0.091
0.275+0.064
0.238+0.050
0. 0.145=00.057
0.313+0.0054
Safety
0.359+0.118
0.521#0.062
0.442+0.061
0.332+0.069
0.533+0.053
0.50
0.72
0.70
0.625
0.68
Car Ceridaiio Mutri Spesmai
0.600
Carawar
0.50
MB8
0.575
3.54
Parpptrase Strategy Paraptrased
Ronriied
0.500
0.525
0.50
Rewrled
0.4
3.6
0.0
5.6
Rade Panashrased Prompt
Parastrased Promppt
(a) The correlation trend on BoN
(b) The correlation trend on PPO.
Figure 5: The correlation between policy regret and accuracy datasets with different ratios of paraphrased prompts evaluated by the Spearman coefficient and MRR. Different line styles are used represent datasets paraphrased by different strategies.
Increasing the number of responses per prompt can enhance the correlation between measured RM error and policy regret.
Finding 4:
We first examine the strategy response expansion per prompt by preparing responses per prompt and employing various metrics. The details about the computation of these metrics can be found in Appendix 8.8 These metries include common correlation measures, well metries adapted from Information Retrieval. We also explored other metrics for comparison, e.g., expected cali- bration error and Bo5. Note that some metrics require absolute scoring of responses, which may be inaccessible in practice. As shown in Table 4 metrics evaluated the dataset with more re- sponses consistently achieve higher correlations in comparison to the aceuracy on the dataset with two responses per prompt. Morcover, the correlation (Chatterjee 2020) generally demonstrated the best performance. The ECE metric, despite its relative commonness, does not show significant correlation with regret.
Although measuring more responses per prompt yields higher correlation, can be unfair without controlling for the total number of samples (prompt-response pairs) in the test dataset. Thus, further explore the question of whether we should increase the number of responses or prompts while maintaining constant sample size. We fix the sample size in the test dataset while varying

---

Published as a conference paper at ICLR 2025
Table The correlation between different metrics and policy regret. The Accuracy-pair stands for the baseline that only samples two responses per prompt and computes the accuracy. Metrics that require an absolute score for each response are suffixed with *
BoN
PPO
Metrics
Spearman corr.
MRR
Spearman corr.
MRR
Pearson corr.*
0.663+0.012
0.657=0.040
0.685+0.016
0.545#0.020
Spearman corr.
0.664+0.015
0.66440.046
0.680#0.019
0.552+0.030
Kendall r corr.
0.665+0.016
0.667=0.042
0.686+0.019
0.558+0.035
Accuracy
0.656+0.015
0.648+0.036
0.671#0.015
0.554+0.025
Bo5*
0.666+0.020
0.663+0.055
0.670#0.024
0.543=0.032
ECE
0.17340.016
0.364+0.027
0.063+0.014
0.382+0.022
MRR
0.650+0.018
0.646+0.054
0.675+0.022
0.565+0.042
NDCG
0.655+0.032
0.614+0.071
0.658+0.038
0.562+0.053
& corr.
0.677+0.026
0.649+0.051
0.688+0.026
0.583+0.047
Accuracy-pair 0.635+00.039
0.586+0.080
0.642+0.047
0.567+0.082
0.54
Cor 4b00
Carweettn
Sample Num
Semple Num.
0.00
1000
0.82
1000
1500
1500
2000
2000
0.66
2500
2500
3000
3000
3500
3000
0.64
4000
4000
4500
10
2
10
Number Resporce per Promot
Number of Response per Prompt
(a) The correlation trend on BoN.
(b) The correlation trend on PPO.
Figure 6: The Spearman coefficient between accuracy and policy regret with test datasets having the same number of samples but varying numbers of responses per prompt.
the number of responses. As shown in Figure 6a and Figure 6b it is generally more effective to increase the number of responses rather than the number prompts. However, this advantage decreases with the growing sample size, possibly due to reaching upper bound in the correlation between regret and measured accuracy.
The annotation cost can be another important factor since building a test dataset with more than two responses can be tricky. To explore this factor, we further examine the question that should we in- crease responses or prompts under constant annotation budget. To conduct the experiment, consider the noise in the annotation process. Among various probability ranking models (Critchlow al. 1991), we adopt the most commonly used one, i.e., the Bradley-Terry model (Bradley & Terry 1952). Given a prompt : and two responses and yj, the probability that is preferred to y, is computed by:
P(y: 5yا=) 1 + elr, -r:/
(5)
where r; and ry is the golden reward score given to the response yı and yj in our case. The parameter 3, which can be seen as an indicator of the annotation quality, is set to 0.5. We evaluate the anno- tation cost by measuring the number of pairwise comparisons. This setting fits the Bradley-Ten Terry model and is also commonly adopted real-worke practice. The annotation process is stimulated by performing sorting algorithm based pair-wise comparison. The final results are presented in
8

---

Published as a conference paper at ICLR 2025
Figures 7a and 7b We find that under the BoN setting, including more responses is more beneficial with the same annotation budget. However, with a limited annotation budget, the benefits quickly diminish as the number of responses grows. Additionally, in the PPO setting, increasing the number of responses does not yield significant advantages.
0.49
0.40
Amctasion Num
595
0.33
Amntatiios
590
0.35
753
750
1850
0.33
1863
0.30
1520
1250
1520
1350
1750
1759
0.25
2000
2009
10
Number of Restemses per Prompt
Number Responses per Prompt
(a) The correlation trend BoN.
(b) The correlation trend PPO.
Figure 7: The Spearman coefficient between policy regret and accuracy on test datasets with the same annotation budget but varying the number of responses per prompt.
5 WHAT'S THE RELATIONSHIP BETWEEN RM ERROR AND POLICY REGRET?
In this section, we investigate the relationship between RM error and policy regret. The translation from RM error to the policy regret can be seen as the result of the reward model overoptimization. Therefore, to accurately predict policy regret, the quantified RM error should be able reveal the potential overoptimization. We first theoretically explore this issue. Then we empirically examine the relationship and analyze the observed optimization dynamics.
Finding 5: Accuracy alone can be insufficient to capture the potential RM overoptimization,
Goodhart's ' Law (Goodhart 1984) is the main cause for the translation from RM eгror to policy regret. It states that optimizing a less effective metric rather than the golden metric leads system failure. Such a phenomenon is also commonly termed reward model overoptimization (Gao et al., 2022) under the context of RLHF. Among the various kinds of Goodhart's 's effect that lead reward model overoptimization, the Regressional Goodhart effect (Manheim & Garrabrant 2019) is the most fundamental and unavoidable. occurs when the proxy reward model essentially the golden reward function r* mixed with some independent noise namely:
+
(6)
Assuming that only the Regressional Goodhart effect occurs and both the reward score and noise are normally distributed, i.e. (00 o²) and NN0 o²), the relationship between accuracy duec and the degree of overoptimization d can be expressed by the following parametric equations:
d
6
2o +
dr
or /
120
m=0
a²
d, o3 12+00 0
where O denotes the cumulative distribution function of the standard normal distribution.. The equation too complex be solved analytically, so we resort to a numerical solution. We plot the expected relationship between accuracy and the degree of overoptimization in Figure Sa
Thhe definition of d, and detailed derivation provided in Appendix 8.9

---

Published as a conference paper at ICLR 2025
0.3
-
0.55
0.60
0.55
0.76
0.75
0.80
0.07
5.50
5.95
(a) Expected trend of accuracy dfce and the degree of overoptimization da by varying noise a.
(b) The trend of the accuracy de and the estimated degree overoptimization da under BoN
Figure 8: Trends between the accuracy dgce and the degree of overoptimization da.
1.4
12
-0000000
C-011
1.0
em032
0-861
2.4
022
02
5.0
1.0
1.5
2.0
Detawce
(a) RM with a = as golden reward model.
(b) RM with a = 0.15 as golden reward model.
Figure 9: Overoptimization trends under BoN setting. The change of golden reward score when optimizing towards different proxy RMs are plotted. The distance d is computed / DD L To
Bai et al. 2022), where the KL divergence of best-of-n sampling can be computed by log n
Hilton 2023). The rewards are computed by an unbiased estimator (Nakano et al. 2022).
However, the trend under the BoN setting, plotted in Figure 8b reveals outliers beyond the pected relationship. These discrepancies may stem from other forms of Goodhart's effects (Man-) heim & Garrabrant, 2019) beyond the Regressional type. Figure illustrates the optimization dy- namics when using different golden RMs and optimizing towards various proxy RMs. We observe distinct overoptimization phenomena for golden-proxy RM pairs that have similar accuracy levels. In Figure Pa] there is nearly no drop in golden reward scores the KL divergence increases, indi- cating that the Regressional Goodhart's effect is dominant in this case. In contrast, Figure shows noticeable decline in golden reward scores across nearly all proxy RMs, suggesting the influence of more complex Goodhart's effects. These differences suggest that solely relying on accuracy may be insufficient to predict potential overoptimization.
RELATED WORKS
Reinforcement Learning from Human Feedback (RLHF) Bai et al. 2022 Ouyang al. 2022 Stiennon et al. 2022) has been a common strategy for the alignment of Large Language Models
Yang al. 2024; Dubey al. 2024h, in which the reward model plays crucial role. However,
this methodology faces several challenges (Casper et al., 2023] Lang et al. 2024; [Armtrrng &
Mindermann 2019 Skalse al. 2022), such reward model overoptimization (Gao et al. 2022
10

---

Published as a conference paper at ICLR 2025
Lehman et al. 2019). where optimizing policy towards proxy reward model may lead the ground
truth performance deterioration. This phenomenon, as a special case of reward hacking (Krakovnal
2020),
the result of Goodhart's ' law (Goodhart 1984 Zhuang & Hadfield- 2021].
The RM as an imperfect proxy of the golden preference necessitates the evaluation whether the reward model accurately captures the real preference (Michaud et al. 2020 Russell & Santos 2019). Many works consider the measurements to predict the potential consequence due to the difference of reward functions (Ng al., 1999; Skalse al. (023). (Gleave al. 2021 Skalse
et al. 2024) propose the metric quantify the difference between the reward functions and induce regret bounds for optimal policies. However, some works (Zhuang & HadiilldMMnnel] 2021 Fluri et al. 2024) propose negative visions of optimization towards imperfect proxy R.s.. T These works assume the accessibility of the golden reward function, which can be false under the RLHF setting. The common practice under RLHF is to compute accuracy on a fixed preference dataset (Lambert) al. 2024). which remains the question about theoretical or empirical validation of performance prediction (Zhou et al. 2024
Kim et al. 2024).
CONCLUSION
Our study highlights that although there is weak positive correlation between accuracy and pol- icy performance, RMs with similar accuracy can result in varying policy outcomes. Moreover, the method used measure accuracy significantly impacts prediction performance. Finally, find that accuracy alone may not fully reflect the phenomenon of overoptimization. Overall, we sug- gest more cautious attitude about RM performance as indicated by accuracy and emphasize the importance of developing more sophisticated RM evaluation techniques..
ACKNOWLEDGMENTS
We sincerely thank the reviewers for their insightful comments and valuable suggestions. This work was supported by Beijing Natural Science Foundation (L243006), CAS Project for Young Scientists in Basic Research (Grant No. YSBR-040), the Natural Science Foundation of China (No. 62306303, 62272439).
REFERENCES
ActMeta. Llama model card. 2024. URL http:://ggttu...ccmntttt-----
1lama//blob/ne i /MODEL, CARD md
Argilla. Capybara-dpo 7k binarized dataset.
argi Segilladdisilaaaa -capaaaa ddoo- 1aapo-7k-iinn 7k-binarized 2024.
Stuart Armstrong and Sören Mindermann. Occam's razor is insufficient to infer the preferences of irrational agents, 2019. /ttp://ttps///aaoo/.oo//////////
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Seffield-Dodds Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL/https: //tpav/aaa.aiiai/.a www.iiiiirrrgooooooo org. abs 2204 05862
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324, 1952. doi: 10.2307/2334022, URL https: 7/doi. wtw..rr///1.2200/333422233
Daniel S. Brown and Scott Nickum. Deep bayesian reward learning from preferences, 2019, URL https::/erxiv.. wttpxxivvog//a////////
04472
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel
11

---

Published as a conference paper at ICLR 2025
Marks, Charbel-Raphace Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitri Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and fun- damental limitations of reinforcement learning from human feedback, 2023. URL https:
Sourav Chatterjee. A new coefficient of correlation, 2020. URL https://axxiv.ort
9909. 1014M
Douglas E Critchlow, Michael A Fligner, and Joseph S Verducci. Probability models on rankings. Journal of Mathematical Psychology, (5(3):294-118, September 1991. ISSN 00222496. doi:
10.1016/0022-249988 299-022-2249999119005555
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023,
Luigi Daniele and Suphavadeeprasit. Amplify-instruet: Synthetically generated diverse multi-turn conversations for efficient Ilm training. arXiv preprint arXiv:(coming soon), 2023. URL https:
//buggingface... Satpatassett/DDon//aaaaaa
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, and other. The llama 3 herd of models, 2024. URL wttpt:ttp//// /ttp:s://arriii org abs /2407 21783
Lukas Fluri, Leon Lang, Alessandro Abate, Patrick Forré, David Krueger, and Joar Skalse. The perils of optimizing learned reward functions: Low training error does not guarantee low regret, 2024. UBL
15753
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022. URL https://arxiy: org/ abs 2209 07858
Leo Gao, John Schulman and Jacob Hilton. Scaling laws for reward model overoptimization, 2022.
Yang Gao, Dana Alon, and Donald Metzler. Impact of preference noise on the alignment per- formance of generative language models, 2024. URL
09824
Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying differences in reward functions, 2021. BoLLitttss//axxii org abs / 2000. 1 13900
C. A. E. Goodhart. Problems of Monetary Management:: The UK Experience, pp. 91-121. Macmil- lan Education UK, London, 1984, ISBN 978-1-139--1795---1 doi: 100000/97788--4--22000 4. URL/https://don crg. /ttp:///.....177711---3311111--.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding, 2021. URL https:
//axiiv wwwxiv.org/ab///0.... 03300
Jacob Hilton. KI. divergence of max-of-n. December 2023.
Zhenyu Hou, Yilin Niu, Zhengxiao Du, Xiaohan Zhang, Xiao Liu, Aohan Zeng, Qinkai Zheng, Minlie Huang, Hongning Wang, Jie Tang, and Yuxiao Dong. Chatglm-rlhf: Practices of align- ing large language models with human feedback, 2024, URL https: / //tp///vvvvv///////////ee.//.wav www./brg........
2404. 00934
12

---

Published as a conference paper at ICLR 2025
Jian Hu, Xibin Wu, Weixun Wang, Xianyu, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and boghh-perfomancc rlhf framework. arXiv preprint arXiv:2405.11143, 2024.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari, 2018. URL https://arxxx
brg/abs//811 06521
Intel. Orca dpo pairs dataset. http::/hwggggggaa....cc///ssstts
pairs 2023. Available on Hugging Face Datasets.
Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. Pku-saferihf: A safety aligmment preference dataset for llama family models, 2024. httpttss: //wpxvvaavvvv.vv/a org/ 221.//440...515
Jacek Karwowski, Oliver Hayman, Xingjian Bai, Klaus Kiendlhofer, Charlie Griffin, and Joar Skalse. Goodhart's law in reinforcement learning, 2023. URL /oLLLCcc////xxv..rr/////
0910409144]
Sunghwan Kim, Dongjin Kang. Tacyoon Kwon, Hyungjoo Chae, Jungsoo Won, Dongha Lee, and Jinyoung Yeo. Evaluating robustness of reward models for mathematical reasoning, 2024. URL
petps://aaxti httg//b//200 01729
Victoria
The flip side of ai ingenu- URL https: nttpps://eeepminddt www.idd.ooooooooooooooooo / specification gaming--nnggggo the fice-llip-a0 side -oldde-ff-oe -f---ddd---------n--------------
Krakovna.
Specification gaming:
ity. April 2020.
Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. The history and risks of reinforcement learning and human feedback, 2023. [Rt /ttp//axvvvvv///// abs/ /ttbb///311.. 13595
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024. URL
Lcon Lang, Davis Foote, Stuart Russell, Anca Dragan, Erik Jenner, and Scott Emmons. When your ais deceive you: Challenges of partial observability in reinforcement learning from human feedback, 2024. URL fettps://arxiss wwt. abs///b/////a http://aa//222221//22 17747
Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter Bentley, Samuel Bernard, Guillaume Beslon, David M. Bryson, Patryk Chrabaszez, Nick Cheney, Antoine Cully, Stephane Doncieux, Fred C. Dyer, Kai Olav Ellefsen, Robert Feldt, Stephan Fis- cher, Stephanie Forrest, Antoine Frénoy, Christian Gagné, Leni Le Goff, Laura M. Grabowski, Babak Hodjat, Frank Hutter, Laurent Keller, Carole Knibbe, Peter Krcah, Richard E. Lenski, Hod Lipson, Robert MacCurdy, Carlos Maestre, Risto Mükkulainenn, Sara Mitri, David Mori- arty, Jean-Baptiste Mouret, Anh Nguyen. Charles Ofria, Marc Parizeau, David Parsons, Robert T. Pennock, William Punch, Thomas S. Ray, Mare Schoenauer, Eric Shulte, Karl Sims, Ken- neth O. Stanley, François Taddei, Danesh Tarapore, Simon Thibault, Westley Weimer, Richard Watson, and Jason Yosinski. The surprising creativity of digital evolution: A collection of anec- dotes from the evolutionary computation and artificial life research communities, 2019. URI https: htt////iivvvv// www.:/og////booooooooooo
03453
Xuechen Li, Tianyi Zhang. Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-followin models. httpp:://gi thtp:////gtutuuug./ wttp:/cct.ttttaaeeeeee httpa/laaaaaalaaaaaaaaaaaaaaaaaa
2023.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorea: An open dataset of gpt augmented flan reasoning traces. https://https:/ 2023.
David Manheim and Scott Garrabrant. Categorizing variants of goodhart's law, 2019. URL https:
04585
Eric J. Michaud, Adam Gleave, and Stuart Russell. Understanding learned reward functions, 2020.
URL htt..httts:/aaaatt abcpr///////ae abs/aaas5222 05862
13

---

Published as a conference paper at ICLR 2025
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answert
with human feedback, 2022. URL https://arxiv.
brq/abs/2112. 09332
Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML '00, pp. 663-670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072.
Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Confer- ence on Machine Learning, ICML '99, pp. 278-287, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1558606122.
OpenAl.
Gpt-4o
system card.
https://openas com httpa//....oeeeeeeeeeeaaeee
gpt-4o- systsm-sttmcrd...a pdf, 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL
220/ 02155
Malayandi Palan, Nicholas C. Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward func- tions by integrating human demonstrations and preferences, 2019. URL https://arxiv.
org /b5//bb//a0 69966.644 08928
Jacob Russell and Eugene Santos. Explaining reward functions in markov decision processes. In The Thirly-Second International FLAIRS Conference, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy abs 1707.0633
optimization algorithms, 2017. URL
Joar Skalse and Alessandro Abate. Misspecification in inverse reinforcement learning, 2023. URL https://arxiv:: org/ abs / 2212.2222.21 03201
Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking, 2022. URL /tttps://aaxi...rr.... abs///////// /209..2209..2 13085
Joar Skalse, Matthew Carrugia-Roberte Stuart Russell, Alessandro Abate, and Adam Gleave. Invariance in policy optimisation and partial identifiability in reward learning. 2023. URL
https::/arxiv: www.//rgg///aaa.o
abs /00./2223..c0 07475
Joar Skalse, Lucy Farnik, Sumeet Ramesh Motwani, Erik Jenner, Adam Gleave, and Alessandro
Abate. Stare: general framework for quantifying differences between reward functions, 2024.
UtL.httts:://axit
http///gab////03/ 15257
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. URI.
https://arxiv.oot
Qwen Team. Qwen2.5; A party of foundation models, September 2024. URL/https://gwenl11
An Yang, Baosong Yang. Binyuan Hui. Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Ma. Jianxin Yang, Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan. Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https://arrs org/ 2407 10671
14

---

Published as a conference paper at ICLR 2025
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing, Hao Zhang, Joseph Gonzalez, and lon Stoica. Judging aa-aase-juddg with mt-bench and chatbot arena, 2023.
Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Li- Xiong, Jessica Fan, Yurong Mou, Rui Zheng, Taao Gui, Qi Zhang, and Xuanjing Huang. Rmb: Comprchensively benchmarking reward models in Ilm alignment, 2024. URL https:
/aaxivv.rg//aa
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling 7b: Improving llm helpfulness harmlessness with rlaif, November 2023.
Simon Zhuang and Dylan Hadfield-Memell Consequences of misaligned ai, 2021. URL
https:
/ /aax.vv.rrg/bbb/2122.cc
APPENDIX
8.1 DATASET CONSTRUCTION
We build up RM training dataset by mixing the following open-sourced datasets:
Nectar Zhu al. 2023). high-quality 7-wise comparison dataset generated by GPT-4 ranking-
Capybara-7K-binarre (Argilla 2024). binarized prefernce dataset built with distilabel atop the Capybara (Daniele & Suphavadeeprasit (023).
Orca-pairs (Intel 2023), a dataset contains 12k examples from OpenOrca dataset (Lian et al. 2023).
UltraFeedback (Cui et 2023
large-scale, fine-grained, diverse preference dataset.
PKU-SafeRLHF (Ji et al. 2024). a high-quality dataset consisting of 83.4K preference entries, which is annotated across two dimensions: harmlessness and helpfulness.
MTBench-human (Zheng al.
2023). dataset contains 3.3K expert-level pairwise human
preferences for model responses generated by models response to 80 MT-bench questions.
Chatbot-arena (Zheng et al.
2023), a dataset contains 33K cleaned conversations with pairwise human preferences. Responses generated by 6 models in response to 80 MT-bench questions.
HH-RLHF (Bai et al., 2022 Ganguli et al. 2022). dataset contains human preference data
about helpfulness and Harmlessness
We retain only single-turn dialogue data, deduplicating based on prompt string matching. Next, we filtered out excessively long samples and balanced the proportion of positive and negative sample lengths. Ultimately, we retained 112400 preference data samples, with 7052 set aside as a validation for reward model training.
8.2 PARAPHRASE PROMPTS
We employed two strategies for rewriting prompts: one that alters expression without changing semantics, and another that rewrites the prompt into a similar but related prompt. The prompt in Table used for the paraphrase strategy, and the prompt in Table used for the rewrite strategy.
8.3 RELATIONSHIP BETWEEN GOLDEN-PROXY PAIRS
In this section, we present the relationship between golden-proxy pairs from our experiments. First, we illustrate the accuracy between golden RMs and proxy RMs in Figure I0 As shown, the image symmetrical, and generally, as the data noise decreases, the accuracy between each golden-proxy proxy pair increases. Next, we display the relationship of the NDR metric between proxy-goldd pairs under BoN and PPO optimization. These figures do not exhibit obvious symmetry, and there are more extreme values under PPO optimization.
15

---

Published as a conference paper at ICLR 2025
Table The prompt used for paraphrase strategy.
Now you should play the role of prompt engineer.
Your task is to paraphrase the prompt into a new prompt. If the origin prompl is safety-related prompt, paraphrase it into new safe prompt.
When outputting, only provide the paraphrased prompt and nothing else.
Input Prompt:
{prompt}
Output:
Table 6: The prompt used for rewrite strategy.
Now you should play the role of prompt engineer. You should generate a new prompt of the same category with an input prompt. If the origin prompt safety-related prompt, paraphrase it into safe prompt. When outputting, only provide the generated prompt and nothing else.
>> Input Prompt:
{prompt}
>>
Output:
0.90
0.5
0.00 0.20 0.25 0.30 0.35 0.40 0.45
Figure 10: The accuracy between golden and proxy RMs across varying noise levels training data.
8.4 CORRELATION BETWEEN ACCURACY AND DOWNSTREAM PERFORMANCE
In real-world scenarios, various factors can lead to discrepancies between proxy RMs and golden RMs. In the main experiments, we primarily considered noise the training dataset. In this section, we explore two other common scenarios: 1) differences in model architecture between the proxy RM and the golden RM, and 2) differences in training data. For the first scenario, we train RMs using various Qwen2.5 Team (2024) series pretrain and instruct models, ranging from 0.5 to 72 billion parameters. For the second scenario, we randomly divide the training into using the first i parts to train the i-th RM. We then measure the correlation between the accuracy of these two types of RMs and the policy regret optimized through BoN, as shown in Figure 12a and Figure [2b The results indicate that the correlation for RMs trained on different datasets is significantly lower than for those trained on the same dataset with different models, possibly due to their high similarity indicated by accuracy.
10 parts,
16

---

Published as a conference paper at ICLR 2025
0.00
1.25
1.50
57
9.50
95
9.50
0.0 9.6 0.10 6:5 0.0 625
0:35 0.40
6.45
0.0 5.66 nio 0.5
525 0ã0
6.56
0.65 3.45
0.50
(a) The NDR metric under BoN setting.
(b) The NDR metric under PPO setting.
Figure 11: The NDR metrics between proxy-golden pairs under different settings.
Spearman corn. 0.8645, MRR: 0.9285
Spearman corr.: 0.458, 0.45, MRR: 0.5708
0.8
0,800
0,775
0.7
0.750
dAoR
0.6
0.725
0.700
0.675
0.4
0.650
0.3
0.625
0.70
0.75
0.80
0.85
0.90
0.95
0.90
0.91
0.92
0.93
ddcc
dp
(a) The correlation trend under model scaling setting.
(b) The correlation trend under dataset split setting.
Figure 12: The correlation between accuracy and policy regret for RMs of different settings.
Beyond synthetic environments, we also explore the relationship between RM metrics and down- stream task performance in more realistic scenarios. We assess the accuracy of various open-source RMs on RewardBench Lambert et al. (2024) and examine their correlation with downstream tasks using MT-Bench Zheng et al. (2023). evaluated with GPT-4o. For evaluating downstream task per- formance, we optimize Figure 13 indicate a positive correlation between RewardBench metrics and MT-Bench scores. However, RMs with similar accuracy can lead to fluctuating downstream performance.
(AT@ Meta 2024) using Best-of-32. The results, shown
8.5 INFLUENCE OF KL PENALTY
In the main experiment, the KL penalty was set to However, there remains the question of whether larger KL penalty might increase the correlation with PPO. In this section, we examine the cor- relation between accuracy and PPO NDR using the same RMs from the main experiment but with varying KL penalties. The results in Table indicate that appropriately increasing the KL penalty enhance the Spearman correlation while reduces the MRR. When the KL penalty becomes too large, the Spearman correlation also decreases. We believe that with smaller KL penalty, the PPO opti- mization process becomes more localized and stable, thereby strengthening the predictive correlation of RM error. However, the presence of the KI. penalty limits the consistent increase of the expected
17

---

MRR Published as a conference paper at ICLR 2025
NewardBerch MT-Swnch: Speammnn c0.5.006635, pm0.001
7.5
7.5
Down 2ream Pertomaace
7.4
6.6
6.4
6.2
Rerformance
Figure 13: The correlation between RewardBench metric and MT-Bench score.
Table 7: The correlation between the accuracy and the policy regret with different KL penalties.
Relevance
KL Penalty
Kendall corr.
Pearson corr.
Spearman corr.
0.00
0.3980
0,6981
0,4840
0,5158
0.01
0.3743
0.5345
0.5011
0.4950
0.05
0.5487
0.7461
0.6692
0.3960
0.10
0.4925
0.6267
0.6265
0.3750
reward, reducing the likelihood that policy optimized with best proxy RM result in the best pol- icy. Once the KL penalty exceeds a certain threshold, the KL reward begins to dominate in the later optimization stages, hindering further growth of the expected reward. This increases the influence of uncertainty PPO optimization, leading to a decline correlation. We also experiment with a larger KL penalty (KL Penalty 0.5), finding that its effects could even cause training collapse.
8.6 INFLUENCE OF SAMPLE SIZE
In Figure 6 we observe that increasing the number of responses improves correlation. However, remains question of whether further expanding the sample size will continue to enhance correlation or not. Therefore, in this section, we examine how correlation changes with larger sample sizes. Since the original RewardBench contains only 2,733 different prompts, increasing the sample size to over 5,500 with just two responses prompt can be infeasible. For sample sizes greater than 5,500, we only assess the correlation cases with more than two responses per prompt, adjusting the number of responses accordingly for even larger sizes. From the result depicted in Figure [4] we find that further expanding the sample size does not significantly enhance correlation; rather, gradually approaches an upper limit.
8.7 INFLUENCE OF DOWNSTREAM MODEL ON EXPANDING RESPONSE
In previous experiments, we use LLaMA-3-88 10----3---nInntuec as the downstream model and found that in- creasing the number of responses improved correlation with downstream performance. However, remains questions whether this effect holds with different downstream models. this section, we Drsen-2.5-7B-Ineee
(Team 2024) the downstream model to test if increasing the number of responses still enhances correlation. As shown the Figure 15 increasing the number of responses continues to be beneficial strategy with different downstream policy.
8.8 REWARD ERROR METRICS
Given a prompt & and = different responses [....... yr]. their corresponding reward scores and ranks represented as r(Y) and R(Y). Since each prompt has multiple annotated
18

---

Published as a conference paper at ICLR 2025
(a) The correlation trend on BoN
(b) The correlation trend PPO.
Figure 14: The correlation between accuracy and policy regret on test datasets with the same number of samples but varying numbers of responses per prompt assessed by the Spearman coefficient.
0.02
Sample Num.
1000
1500
2000
0.80
3000
4009
4500
Gomeaationn
5000
0.76
6.72
0.70
Response
Figure 15: The Spearman coefficient between accuracy and policy regret with test datasets having the same number of samples but varying numbers responses per prompt.
responses, many ranking evaluation metrics can be applied to our setting. Finally, we averaged the correlation metrics across all prompts to obtain the results presented in the Table 4
Pearson corr. Pearson correlation coefficient can be computed as follows:
Pr()),r-)) cov (r(Y),r*())
,r()))r~())
where cov is the covariance, or(Y) is the standard deviation the proxy rewards given to the re- sponses r(Y) and (Y) is the standard deviation of the golden rewards given to the responses r" (Y). The formula of cov (r(Y), rr (Y)) can be writed as:
cou (r(Y), r (Y)) = E [(r(Y) - (r()))r -4--())))
Where (4(()) and (Y) are the mean of the proxy rewards and golden rewards given to the re- sponses, respectively. The final metric the average of the coefficient of each prompt.
Spearman corr:
Spearman's 's rank correlation coefficient follows the basically same formula as the Pearson correlation coefficient, only with the reward scores replaced by the ranks:
Cou (R(Y), R (Y))
R (Y)
(10)
RRY) o (Y)
Kendall 5 corr. Kendall rank correlation coefficient is computed follows:
ne nd
(X) = N(N - 1)/2
(1I)
19

---

Published as a conference paper at ICLR 2025
where ne and na stands for the number of concordant and disconcordant pair between R(Y) and (Y),, respectively, The concordant pair (,,, means that their rank satisfy (R(Y)) R(G)))
Rd (YG)) > 0. In practice, employ тh which handles the ties for the rare case that some responses get the same reward scores. The final metric the average of the coefficient of each prompt.
Accuracy The accuracy metrics are mostly the same as in the typical case that there are two responses per response. The main difference is that if there are N responses per response, we can then form C% different pairs for comparison.
Bo5 The best-of-5 metric can be seen as a special case of NDR for N 5, which computes:
r* arg max (r(X)) Mr)(()
(12)
max [r- (Y)) (V-
This metric represents the improvement reward values obtained using the proxy reward score compared to those achievable with the original golden reward model. The final metrie is the average of the coefficient of each prompt.
Expected Calibration Error (ECE) is calculated by averaging the absolute differences be- tween predicted probabilities and observed frequencies, typically across a set of bins partitioning the prediction space:
ECE
ECE Σ Bml n
Jacel Bm) (B.))
(13)
where B represents the bins that split pairs by reward margins and M stands for the number of bins. ace is the accuracy of pairs in each bin. conf computes the expected accuracy inferred from reward score margins by the Bradley-Tere Terry Model (Bradley Terry) 1952). Expected calibration error indicate the alignment of reward models' confidence. We follow the same strategy for form preference pairs as in the Accuracy.
MRR
Mean reciprocal rank is traditional Information Retrieval metric that be transported to our setting smoothly. We first define reciprocal rank as the golden rank the response that receives the highest reward score Then we take the average overall prompts:
MRR = EtEX
(14)
R
arg max [R(Y)]
NDCG Normalized discounted cumulative gain is another typical Information Retrieval metric that is transported here. This computes compute:
Σ (Y, 1)/loge R(Y:) +
NDCG
(15)
(R,())) - 1) log, R*(Y,) + +1
The main difference from the typical usage in the field of Information Retrieval is that replace relevance score rel, with the golden rank R(Y,). These metrics can be seen as a smooth version of MRR. The final metric is the average of the coefficient of each prompt.
One may expect this metric compute the proxy rank of the response that receives the highest golden reward But implementation makes it more similar the Bod metric.
20

---

Published as a conference paper at ICLR 2025
corr. ç correlation coefficient (Chatterjee, 2020) is relatively new metric for evaluating the rank
correlation. Compared to traditional rank coefficients like Spearman corr, this coefficient is more effective to compute. It first rearrange the data as [r* (K()), ... (()))) such that r (Yi) I r* (Y(2) S ... r (YNn), and then compute:
EN (M))) 3 ii RQ(+1)
3
R
(16)
N2
8.9 THE RELATIONSHIP BETWEEN ACCURACY AND REGRET
Based the assumption that only Regressional Goodhart takes effect and golden reward score r* ~ (0, of),, the noise 2 ~ o²), we can then derive the relationship between the accuracy and the regret. Based on the Regression Goodhart's, the proxy reward can represented as:
+2
(17)
The process of constructing an RM test set can be viewed as performing two independent samples from the distribution the golden reward score. Therefore, the scores obtained from the two sam- ples can be represented as (0, of) and r2 (O, o²). Subsequently, the difference between the two samples' golden reward scores also follows a normal distribution: r ~ (0, 2a Then the proxy reward model score difference can be written as:
r +
(18)
where 21 and 22 is the noise introduced in the two times of sampling. The distribution of noise difference is also normal distribution 21 NO0, 2n²). The accuracy be translated into:
dece P(r- > 0, > 0) + P(r- < <0,, <0)
P(r > 0,r² 0)
1 - 2P(21
22 < 0,r" > >0)
+00
(19)
2
P(x r- P((1
22 x)dr
===
+00
1
2
40
dr
/T
220
Following the notations in Section 2 define the degree of overoptimization d, as follows:
Jr (x) d, (T)
(20)
This value implies that if the expected reward of a policy under the proxy RM is E(r) then the corresponding expected reward under the golden RM should be mathôbE(1" = d.c. We then introduce a result from (Gao et al. 2022):
E + (c (c- E[X] - E|2) Var Var(X) + Var
E
(21)
where and Z are independent, absolutely continuous random variables, both normally distributed. In our context, X can be replaced by f and X Z by Therefore, the degree of overoptimization
21

---

Published as a conference paper at ICLR 2025
Table Summary Training Hyperparameters
RM Training Hyperparameters
Value
PPO Training Hyperparameters
Value
Max Length
2048
Train Batch Size
64
Regularization Coefficient
le-2
Rollout Batch Size
8
Batch Size
256
Generate Max Length
1024
Warmup Ratio
0.1
Actor Learning Rate
1e-6
Learning Rate Scheduler
cosine 5c-6
Critic Learning Rate
le-5
Learning Rate
KL Penalty
0
d can expressed as:
E y- (r (1)
By (x) (r(,,
"((r",r)dr" dr to to
00
rp(r)dr
f 1 + r"p(r" ("p(r"(r)r"" p(r)dr
(22)
rp(r)dr
400 ro p(tx)
rp(r)dr
Under the BoN setting, we first normalize the reward scores and directly estimate d, according to the definition equation 20
8.10 DETAILS ABOUT EXPERIMENTS
For all experiments in Section we obtain the results through multiple samplings of test datasets. Specifically, in Figure 4 we perform 64 samplings for each chosen-reject rank bin to construct different datasets and calculate their correlation with downstream performance. The final results are averages of these outcomes. For Tables multiple different responses for each prompt, conducting 64 samplings
B and the set of prompts is fixed, but we sample construct various test datasets. These outcomes are used to calculate the mean and variance for reporting correlations. In Figures [ and neither the test prompts nor responses are fixed. For each data point, we perform 64 samplings to obtain different test datasets and calculate the results, which reported with the 95% confidence interval. Next, we provide training details. The hyperparameters for RM and PPO training are listed in Table & For PPO optimization,
utilize the OpenRLHF framework (Hu et al., 2024). In BoN optimization, N is set to 1280, requiring 2733 x 1280 x 10 34, 982, 400 response samplings and RM scoring for preparing BoN results. In the KL divergence calculation for BoN, we use a logarithm base 2.
22


Reviews:
[{"review_id": "330gmFVsG5", "reviewer": "Reviewer_Rn3n", "summary": "Research question of the paper is on how we should evaluate the quality of the reward models for RLHF. The paper conducts experiments to evaluate the evaluation metrics for RMs and show interesting findings that some confirm the intuition and the other are somewhat counterintuitive. The paper concludes that there should be more care on evaluating RMs instead of relying on a single benchmark.", "questions": "- Randomly flipping some percent of the trains in the training dataset is a trick to make a pseudo proxy reward model used in several papers (e.g., AlpacaFarm; Dubois+ 2024). In reality, it is more natural to think that some kinds of instructions have more flips and others have less. I'm a bit concerned that the findings of the paper might only hold true due to the synthetic error model. It would be helpful to see whether this possibility is addressed or refuted (or let me know if I missed it).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper presents empirical results that evaluate the metrics of RMs, which the community has previously intuited but not with rigorous scientific evaluation. The experiments are designed to test hypotheses incrementally, helping the RLHF community build a comprehensive body of knowledge on RM evaluation.\n\n- The paper tackles the question that the alignment research community needs to know the answer to.\n- Table 2 is interesting as it shows counterintuitive results. One would guess that the RM should be evaluated for the samples generated from the policy to be trained. The paper also shows that instead, we should sample multiple responses and choose a pair of responses to evaluate the accuracy with care.", "weaknesses": "I don't see any critical weaknesses for the paper. If I were to come up with the weaknesses:\n\n- Although policy regret is often referred to in the paper, its formal definition is not clearly stated. It would be better to have an equation defining the regret. Even if we do not have a way to compute it, the goal of the research is to estimate it so I would say that it is worth clarifying its definition formally.\n- The scope of the paper is to show that the current evaluation scheme is not enough (which is a good enough contribution). The paper does not provide a solution to the problem of how we should evaluate the RMs (which I think is asking too much).\n- The texts in Figures are a bit too small to read. It would be nice if it is a bit larger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y7BJ5C0b1u", "reviewer": "Reviewer_22rE", "summary": "This paper revisits the soundness of current reward model evaluation procedures, which assess the accuracy of RMs over datasets of held-out preference data, showing they present the flaw not to necessarily translate into downstream improvement for models going through RLHF in a synthetic setting. They propose to improve RM evaluation by increasing the number of responses per prompt and/or (if available) using the rank of model responses to select preferred and rejected responses when building preference datasets.", "questions": "See questions above as well.\n\nFig 3 shows high correlation between accuracy and BoN perf at high accuracy. Do authors have an intuition on why that might be the case? Also, no RMs have an accuracy between 0.75 and 0.8, which might indicate a lack of diversity in the RMs trained, echoing an earlier remark on experimental design.\n\n“As shown in Table 2, this approach does not consistently improve upon the original RewardBench dataset. This result suggests that sampling responses from the model used for optimization may not be necessary to achieve a strong correlation between accuracy and policy regret.” -> this is quite surprising. Do authors have an intuition here?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem tackled is well introduced and presented.\n\nContributions are clear.\n\nHaving a synthetic and cheap setting that correlates with more realistic experiments is quite valuable for the research community (but see doubts expressed below).\n\nSome very interesting findings in this setting:\n* weak correlation between RM accuracy and downstream policy performance/regret\n* correlation increase by increasing the number of answers per prompt\n* correlation increase by picking answers based on their rank\n* correlation increase by increasing the number of test samples\n\nEDIT: I raised my score accordingly after convincing updates from the authors.", "weaknesses": "The key limitation from my perspective is that we have currently no solid evidence that the study will translate to real settings: diverse, heterogenous RMs potentially trained on different datasets. While this would be infeasible to run as many experiments and ablations as in the synthetic setting, showing that there is imperfect correlation between RewardBench scores and relevant, downstream RLHF settings (more generally, that some findings from the synthetic study do replicate) would make the paper much stronger. At least showing that RewardBench rankings are not preserved after downstream BoN / RLHF would be a great addition. \n\nIn that regard, the fact that RMs used in the experiments are essentially the same architecture / same init / same hyperparameters trained on very similar data (i.e. up to random flips) is concerning: this is not capturing the diversity of RMs being trained and evaluated on popular benchmarks such as RewardBench. Authors should do a better job at showing that this is enough to capture phenomena that replicate in real settings.\n\nAlso, what about using a larger RM as proxy-golden? Seems like the default practice and would make for a more principled study, as this would likely reduce RM similarity with the proxy-golden RM.\n\nAnother limitation is that it is currently hard to evaluate whether the correlation increase authors get from applying the proposed improvements are significant or not. What is the correlation between proxy-golden RM accuracies on train and test data? This value would constitute an upper bound of what level of correlation is realistically achievable and provide a reference value that would alleviate the above concern.\n\nSeveral ablations are missing AFAICT:\n* Why is correlation higher for BoN vs PPO? Authors state that “This is expected, as BoN is a more localized and stable optimization algorithm, making it more predictable by reward model error.” I think this might be due to using unconstrained PPO (i.e. without KL regularization), which begs for an ablation study.\n* Did authors conduct any form of investigation on low accuracy but high NDR PPO policies from Fig. 3 b)?\n* It is currently unclear whether adding more samples (i.e. more than 5000 samples) in Fig 6 would improve correlation or instead saturate? This experiment would be a compelling addition to the study.\n* Regarding using additional responses per prompt: the question of whether the policy matches or not the downstream policy matters or not is not studied in the current state\n* What about using a different set of prompts to quantify whether the difference between test RM prompts and test RL prompts is meaningful?\n\nNotations and equations:\n* equation 2 uses $\\pi’$ on LHS but not present on RHS -> please fix\n* what is \\pi_0 in equation 2? I suppose it is the initial policy, but this should be clarified in the main text\n* is the regularization term from Equation 3 known/standard? if so, clarifying and including a citation to prior works is warranted \n\nClarity can be improved a lot:\n* “on the widely adopted benchmark dataset (Lambert et al, 2024)” -> authors should name it (RewardBench)\n* “Regarding response distribution, we find that the rank of the responses can be more influential rather than the model from which it is sampled.” -> unclear sentence\n* “The translation from the RM error to the policy regret can be seen as the result of the reward model overoptimization” -> “imperfect translation”? or “weak correlations”? unclear sentence for now\n* “ the difficulty of constantly controlling optimization pressure” -> what do the authors mean by pressure here?\n* is the Normalized Drop Ratio (NDR) a contribution of the authors’ work? if so this should be clarified in the text\n* the use of NDR (as opposed to difference in average rewards) should be motivated better in the text, even if easy to interpret\n* Fig 3 is a bit hard to read (dots / text could be made bigger)\n* /!\\ Using more responses per prompt to improve RM evaluation -> This is a key aspect of the paper that is quite unclear in the current state! As of now the details are in the appendix, but the main text should be much clearer about this aspect, notably that the fact that we have a golden-proxy RM allows us to estimate a correlation between the proxy rewards and the golden-proxy rewards.\n* “Can we achieve a higher correlation by sampling responses specifically from the model used for downstream optimization? To examine this question, we construct test datasets with responses solely from different models.” -> I find these sentences puzzling as they seem to contradict each other, please clarify\n\nWriting is sub-par in the current state, see examples:\n* “The inherent difficulty of constructing an ideal RM require”\n* “The latter, while straightforward, remains the question of whether such evaluation accurately predicts the performance of the downstream optimized policy”\n* “A cartoon of RM error and policy regret”\n* “We begin by investigate the influence of prompt and response distributions”\n* “The correlation between policy regret and accuracy on datasets constructed from responses of different ranks assessed the by Spearman coefficient”\n\nwith more in the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rDdQyolSnZ", "reviewer": "Reviewer_UcNB", "summary": "The paper questions the current approach to evaluating reward model performance, which is based evaluating downstream policy performance and looking at reward model accuracy on the train and eval preference datasets. The relationship between reward model performance and policy performance is explored in the context of policy regret, a new performance metric that examines the difference in policy performance when trained with the true versus inferred reward. Since the true reward is not typically available, a synthetic true reward model is used to conduct the analysis. The reward model and policy performance relationship is measured as the correlation between the reward model's accuracy and the policy regret. Evaluation relies on RewardBench. The authors find a weak positive correlation between the two, and conclude that accuracy alone is a poor proxy for downstream performance.", "questions": "* How do the results here relate to the findings in \"Preference Learning Algorithms Do Not Learn Preference Rankings\" (NeurIPS 2024)?\n* Where does the data used to evaluate the policy and measure policy regret come from?\n* How do the results about prompt versus response distributions relate to the paper \"On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization\" (Lin et al., 2024)?\n* It seems the paper does not account for noise. Why is that not important here?\n* In equation (2), what is $\\pi_{0}$? Is this the SFT/IFT'ed model?\n* What is your motivation for setting the KL penalty to 0 (line 160)?\n* For the analysis of Figure 3 supporting the conclusion that  \"policy regret can very considerably even within similar accuracy ranges\", is it possible that accuracy and NDR exist on different scales making it look like there is more variation along one dimension that the other? Can you report the results normalized so that they fall on the same scale?\n* For the results reported in Figure 3, how OOD versus ID is the data for each the reward model and the policy?\n* How were the bins that were used to assign response rankings determined? Was the ground truth reward model for that experiment used?\n* It is surprising that seeing higher reward samples was not more beneficial. Can you elaborate more on why this is the case?\n* For Figure 4, what does it mean when the bin is 0 for both the x and y axis? Does it mean the two responses are of equal rank? In this case, how does labelling work?\n* For the results looking at the impact of response rank, how was the label flipping or disagreement with the ground truth reward function's training data accounted for? For the flipped labels spread uniformly over the rank bins? \n* Why is Table 3 not symmetric along the diagonal?\n* What is the hypothesis for why extra responses help in the case of BoN and not PPO?\n* In \"We observed that there should be an approximate linear relationship between them\" (line 454), \"observed\" + \"should be\" in unclear. You assume? You know? Also how? Why?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* Understanding the relationship between our measures of reward quality and policy quality is of vital importance.\n* The goal and motivation for the work is well laid out at the start of the paper. The takeaways the reader should expect to have are presented from the start.\n* The use of a synthetic ground truth reward function is well motivated and contextualized.\n* Different methods for using a ground truth reward function (best-of-n and RL) are compared.", "weaknesses": "**High level**\n\nThe main weaknesses for this paper are not overly large and mostly involve clarifications to the text. While not big changes, they are important to address. Some of the conclusions in the main body need to be walked back and made more nuance to fully reflect the presented results. The biggest missing result is information about the relationship between the ground truth reward model and the proxy reward model. \n\n**Details**\n\n* The experiments are set up such by taking a labelled preference dataset and then randomly flipping labels to create multiple preference datasets. A different reward model is then trained on each version of the dataset. The ground truth reward model is then arbitrarily chosen to be one of those and all others are treated at the proxy. It would be good to quantified the relationship between the ground truth and proxy reward models as part of the analysis. This could be as straightforward as looking at the percentage of agreeing labels in the training data.\n* It would improve understanding of the experimental section to describe the RewardBench data and how it is used earlier. The very start of Section 3, would be a good place.\n* It would  strengthen the results and analysis to include policy regret information where the ground truth reward is used as the proxy reward. This would help to understand the impact of randomness in policy learning. In practice, is the policy regret 0?\n* Some of the statements made early in the paper (e.g. introduce) are difficult to interpret in the absence of having read the whole paper. For example, \"...we find that  the rank of the responses can be more influential...\" and \"...increasing the number of responses per prompt\". It would be helpful to add some description about what is meant by rank and what it means to increase the number of responses in terms of the evaluation.\n* It is challenging to interpret the results in Figure 4. Adding axis names would be beneficial. \n* The conclusion that \"While the correlation on PPO continuously weakens as we paragraph a larger number of prompts\", the nuance that this depends on the correlation metric should be called out, especially as no single correlation measure has been identified as \"best\".\n* Please add more descriptions of the different methods used to evaluate Finding 4. Specifically things like \"Bo5\" where a citation would also be helpful.\n* It is not clear how all of the metrics in Table 4 were computed. For example, what is the Pearson corr. measured between?\n* The conclusion about the impact of number of responses per prompt does not fully reflect the results in Figure 7 (a). For the smaller annotation budgets, the benefit of extra responses drops off quickly. The trends for PPO should be summarized and described and extra responses are not beneficial.\n* The paragraph immediately after \"Finding 2\" (lines 238 - 244) were not clear to me, so it is difficult for me to assess or validate them. Parts such as \"solely from different models\" was not clear. It is not clear exactly how the data is different from what was used previously.\n* Some small spelling issues, typos, and confusing phrases throughout the paper. These are not impacting my score, would be good for the authors to clean up.\n     * line 071 \"investigate\" -> \"investigating\"\n     * line 089 \"This offers valuable for both reward model training....\"\n     * line 129 \"...we focus on a few interested factors and keep the rest fixed\" - \"interested\" -> \"interesting\"\n     * line 134 \"...perform investigation...\"\n     * line 319 \"...expect calibration error...\" -> \"...expected calibration error...\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G2T8qxzC2i", "reviewer": "Reviewer_FwA4", "summary": "This paper empirically investigates whether \"reward model (RM) accuracy\" (i.e., agreement with a preference dataset), which is commonly used for evaluating reward models, is a good metric for evaluating reward models. It begins with the premise that the true measure of a reward model is the quality of the policies it produces, and proceeds to investigate the relationship between RM accuracy and the regret of the downstream policy. It does this by synthesizing several \"noisy\" reward models, which can each serve as both a \"gold\" reward model providing ground truth labels for both preferences and policy performance, or a proxy reward model (when another RM is the gold reward model). The authors then proceed to conduct several empirical investigations into the RM accuracy vs Policy regret. They conclude that (1) RM accuracy is not perfectly correlated with downstream policy regret, (2) the predictive power of RM accuracy declines as the distribution on which policy regret is measured shifts away from the original distribution, and (3) ultimately conclude that RM accuracy is not adequate as a measure of RM quality.", "questions": "1. Do you have any potential solutions for the issues you have identified (how do we go beyond RM accuracy)? \n\n2. Can you clarify if I am missing something re: my major concern above, or make me doubt the view that \"RM accuracy is a better measure of expected downstream regret than single seed downstream regret\". An experiment that shows insufficient correlation given multiple seeds might do the trick (even if you can't do PPO, you should be able to run this experiment quickly in the BoN setting).", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The research question is a good one. That is, it is known from the RL literature that the best reward models for learning are not necessarily the ground truth reward models (e.g. Singh et al. 2009, Where Do Rewards Come From?), so it seems natural that some proxies of ground truth rewards (well, in case of LLMs, trajectory returns) will be better than other proxies of ground truth rewards --- can we ascertain whether this is the case, and if, so can we find ways to craft better proxies and/or better measures of proxy quality. I think this is an important question of great interest to a variety of researchers. I also think it has not really been studied in the case of language reward models, so it is good to have a paper that prompts the discussion. \n\nTo answer this question, this paper contains several experiments that show the relationship between RM accuracy and downstream policy performance. The questions that drive the experiments are generally interesting. There are a lot of experimental outcomes here, which may be interesting to different researchers. Although some thinking is necessary on part of the reader to understand what the authors are doing, the paper is fairly written through Section 4.", "weaknesses": "Issue: The fact that your hyperparameters for training the policy are held constant does not imply that the policy extraction / learning process is not noisy. So any characterizations of the correlation as being \"weak\" (L63, L530) or \"room for enhancing\" (L213) are IMO unsubstantiated, as the upper bound for correlation may be much lower than 1 due to variance in the policy optimization process. This noise also explains why correlation with PPO is weaker than correlation with BoN (which you correctly point out at L190). *So the major issue I have is that, even after reading this paper, I expect RM accuracy to be a better measure of expected downstream policy quality than single seed downstream policy quality.* (and this is the main cause for my low review score; the experiments do not convince me that RM accuracy is an insufficient target for designing RMs)\n\nAlthough the experiment at L246-256 / Figure 4 is interesting, the descriptions are insufficient to properly understand Figure 4 (which axes is which, how does (a) map on to what you wrote in the text?). Further, what is the practical import of this experiment --- would we need humans to rank multiple samples for this to be relevant? And if so, wouldn't it then be better to just use all the samples as you investigate later on? \n\nFinding 4... it would be good to provide an intuition for the reader as it was not immediately obvious to me why this is happening. Namely that the number of comparisons (and therefore samples with which to reduce variance) grows factorially in the number of responses. Once stated this way, I would note that past work has used multiple responses/ human labels per prompt, see e.g., Hwang et al. 2023, Sequential Preference Ranking for Efficient Reinforcement Learning from Human Feedback and Wu et al. 2023, Fine-Grained Human Feedback Gives Better Rewards for Language Model Training. (This is not to say Finding 4 is not interesting, but I would still rather see an analysis of how the annotation budget affects the RM accuracy, as opposed to the downstream Policy Regret). This all being said, perhaps if this finding were fleshed out more, to show how you can use it to improve RewardBench, it could be a lot more impactful. \n\nSection 5: I'm afraid I do not really understand this section. I'm not sure the Goodharting / overoptimization / exogenous variable language is doing much good here (what would be an example of an exogenous variable impacting the relationship, and why is implied by the experiments here?). It seems you are assuming that rewards follow a Thurstone Choice model (whereas most literature assumes Bradley-Terry-Luce; which is fine, they are very close), and then in Figure 8(a) comparing the percentage of reward variance owed to the ground truth reward (vs noise term) to the RM accuracy. What is Figure 8(b) showing? How do we know what $d_\\pi$ is in the \"actual\" setting? I don't understand what the takeaway from Figure 9 should be, if we can't see the reward accuracy, and don't know the sample size under which it computed, etc. There is detail missing here.\n\nImportant (but minor review-wise): \nL139: alpha is never specified (except in Fig 9)\nL159: n is never specified!\nAnd generally, I think there are details left out / I would not be able to reproduce everything given current manuscript; e.g. what hparams were used for PPO, etc. How are confidence bands in Table 2 computed, etc. \n\nMinor:\n- L141: N x (N-1) pairs\n- L48: \"human golden reward function\" --> \"empirical human rewards\"  (the human reward 'function' is noisy, and we cannot quantify the error between learned RM and the human reward fn) \n- L50: \"the goal of RM eval is to estimate policy regret\" (no, it's to estimate the quality of the reward model --- policy optimization introduce a whole range of additional noise / issues -- see commentary above) \n- L429: I would not say the entire translation is due to overoptimization... in fact, there may be NO overoptimization if we do early stopping, right?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)