How to Evaluate Reward Models for RLHF
Authors: Evan Frick
Tianle Li
Connor Chen
Wei-Lin Chiang
Anastasios Nikolas Angelopoulos
Jiantao Jiao
Banghua Zhu
Joseph E. Gonzalez
Ion Stoica
Venue: ICLR 2025 Poster (2025)
Keywords: RLHF
RL
Reward Model
LLM
Benchmark
Dataset
Evaluation
Primary area: datasets and benchmarks

Abstract: We introduce a new benchmark for reward models that quantifies their ability to produce strong language models through RLHF (Reinforcement Learning from Human Feedback).
The gold-standard approach is to run a full RLHF training pipeline and directly probe downstream LLM performance.
However, this process is prohibitively expensive.
To address this, we build a predictive model of downstream LLM performance by evaluating the reward model on proxy tasks. 
These proxy tasks consist of a large-scale human preference and a verifiable correctness preference dataset, in which we measure 12 metrics across 12 domains.
To investigate which reward model metrics are most correlated to gold-standard RLHF outcomes, we launch an end-to-end RLHF experiment on a large-scale crowd-sourced human preference platform to view real reward model downstream performance as ground truth. 
Ultimately, we compile our data and findings into Preference Proxy Evaluations (PPE), the first reward model benchmark explicitly linked to post-RLHF real-world human preference performance, which we opensource for public use and further development at https://github.com/lmarena/PPE.

Published as a conference paper at ICLR 2025
How TO EVALUATE REWARD MODELS FOR RLHF
Evan Frick
Tianle Li
Connor Chen
Wei-Lin Chiang
Anastasios N. Angelopoulos
Jiantao Jiao
Banghua Zhu
Joseph E. González
Ion Stoica
UC Berkeley
ABSTRACT
We introduce a new benchmark for reward models that quantifies their ability to produce strong language models through RLHF (Reinforcement Learning from Human Feedback). The gold-standard approach is to run a full RLHF training pipeline and directly probe downstream LLM performance. However, this pro- cess is prohibitively expensive. To address this, we build predictive model of downstream LLM performance by evaluating the reward model on proxy tasks. These proxy tasks consist of large-scale human preference and verifiable cor- rectness preference dataset, in which we measure 12 metrics across 12 domains. To investigate which reward model metrics most correlated to standad RLHF outcomes, we launch an end-to-end RLHF experiment on a large-scale crowd-sourced human preference platform view real reward model downstream performance as ground truth. Ultimately, we compile our data and findings into Preference Proxy Evaluations (PPE), the first reward model benchmark explicitly linked to post-RLHF real-world human preference performance, which we open- source for public use and further development at github. cettuub.comm/marna//PP1
INTRODUCTION
The ultimate test of a reward model is as follows:
Does the reward model lead good post-RLHF language model performance?
In other words, because the reward model will be used as reference signal for LLM training. in principle, only the downstream LLM performance matters. However, evaluate downstream performance, we must train a new LLM using the reward model and evaluate the resulting LLM-a prohibitively expensive and time-consuming process (Figure T). The long cetllooment-feeddate cycle of reward models poses a significant challenge, limiting achievable reward model quality and, consequently, limiting the effectiveness of the entire RLHFF process.
Fast Feedback-
Experimentally Correlated
Fostuate
Training
Trains
Reward Model
Use for
RLHF
Traiss
LLM
Evalusto
Humna Pref. faating
Infeasible/Slow Feedback
Figure 1: Overview of the RLHF pipeline. Reward models feed into the very beginning of the RLHF pipeline, making iterative improvements prohibitively slow. PPE enables a fast feedback loop that correlated downstream outcomes.
This paper introduces a cost-effective method for approximating the effect of a reward model on downstream LLM performance. Specifically, we measure reward model performance using large- scale, crowdsourced pairwise human preference evaluation dataset, as well as a high-quality, pro- grammatically verifiable correctness preference dataset. To avoid introducing bias, we do not utilize LLM judges or expert annotators to provide ground-truth references. Instead, we focus on real- world preference data that reflects organic L.IM usage. Additionally, aim evaluation tasks

---

Published as a conference paper at ICLR 2025
to closely resemble real-world RLHF training, making the assessment more aligned with practical use cases. Moreover, to bridge the existing knowledge gap between reward model evaluations and actual post-RLHF outcomes, experimentally correlate our evaluation metrics with real human preferences on RLHF-ed L.MMs. To achieve this, we used select reward models within a full RLHF training pipeline, each producing a fine-tuned L.M.. These RL-H--tuned models then deployed on crowd-sourced human preference platform where we directly measure their downstream hu- man preference scores. Through this end-to-end analysis, we identify which metries across diverse domains show the strongest correlation with real-world post-RLHF performance. By validating this correlation, we ensure that iterative improvements on our evaluation will lead tangible gains in downstream performance.
Additionally, we release PPE, crowdsourced collection of 16,038 labeled human preference pairs containing responses from 20 different top LLMs and over 121 languages as well as a dataset of 2,555 prompts, each with 32 different sampled response options, totaling 81,760 responses across different models, all grounded with verifiable correctness labels. PPE evaluates reward models on 12 different metrics and 12 different domains, such as their accuracy in selecting juman--referme verifiably correct responses. Notably, PPE is the only reward model benchmark directly linked to downstream RLHF outcomes.
To summarize, work makes the following contributions:
1. We analyze how reward model metrics correlate with real downstream human preference performance pLSt-RLHF.
2. We fully open-source PPE, comprehensive benehmark for reward models with metrics directly linked to downstream RLHF outcomes.
2 RELATED WORK
2.1 HUMAN PREFERENCE AND REWARD MODELS
Human preference has emerged as one of the gold standards for LLM training and evaluation. Sev- eral large-scale human preference datasets have been developed, including Stanford Human Prefer- ence (SHP) (Ethayarajh et al. 2022), Chatbot Arena (Chiang et al. 2024). and Anthropic HH (Bai jet al., 2022a). among others. Researchers requiring human preference proxies have pursued two main approaches in this area. First, they have trained reward models based on real or synthetically generated human preference data to approximate human preferences for LLM training. Second, they have employed LLMs as judges for evaluating other LLMs.
For the training side, the line of work on Reinforcement Learning from Human Feedback (RLHF) focuses the family of algorithms that first train reward model as proxy of human preferences, and then use the reward model as the signal to fine-tune the language model with reinforcement
learning (Christiano et al. 2023 Bai al. 2022a Ouyang et al. 2022 Touvron et al. 2023 OpenAl
2022
Bai et al., 2022b
Lee et al. 2023 OpenAl 2023a
Zhu et al. 2024).
This paper studies one of the critical problems in the RLHF process: how do we evaluate reward models and select the best one for downstream performance?
2.2 REWARD MODEL BENCHMARKS
RewardBench the first and only previous RLHF reward model benchmark (Lambert al. 2024) RewardBench has 4 main tasks: Chat, Chat Hard, Safety, and Reasoning. The authors source consid- erable ground truth preference pairs from MT-Bench (Zheng et al.] (203)) and AlpacaEval (Dubois et al. 2023), though preference labels are also hand-verified. RewardBench also uses adversarial examples from LLMBar (Zeng et al. 2024), coding example pairs with correct vs buggy imple- mentations, and safety pairs with should-refusals and should-not-refusaal. Overall, RewardBench designed to evaluate across an array of tasks posited as relevant to RLHF. RewardBench takes a crucial first step toward reward model evaluations. However, the authors assert that more research must be done to understand how to correlate performance to RLHF success. In this paper, our ex- periments show that as reward models have improved, now negative correlation between
2

---

Published as a conference paper at ICLR 2025
Diverse Human Pref.
# Prompts
# Responses
Verified RLHF Outcomes
RewardBencl PPE (Ours)
No
2,985
5,970
No
Yes
18,593
113,836
Yes
Table I: Comparison of PPE to existing work.
RewardBench evaluation score on top models and downstream RLHF performance. improve upon this gap with the our findings.
We aim to
SOURCING GROUND TRUTH PREFERENCE LABELS
Previous work on sourcing preference ground truth labels often relies upon LLM judge preference labels in conjunction with manual verification from individuals, introducing potential preference biases. Alternatively, rejected responses are often curated synthetically by unnaturally perturbing the chosen output modifying the prompt to produce forced errors, introducing bias how emross look and occur. These preference pairs are not representative of the distribution responses seen by reward models when providing learning signals for RLHE. We offer brief comparison previous work in [Table 1
Thus, ground preference labels with the following methodology: (1) Utilize crowdsourced diverse prompts and responses with human preference labels. (2) Utilize existing benchmarks with verifiable correctness checks LLM--gnerated responses.
The methodology (1) provides an unbiased estimate of real-world human preference through the ag- gregation many diverse human preferences. We use large crowdsourced preference set of 16,038 preference labels to mitigate individual label noise and avoid over-fitting to any single individual's preference, details in subsection 4.1
Methodology (2) curates an objective correctness signal naturally unbiased by response style. We use the second approach label the correctness of many sampled responses from an LLM, mimick- ing rollouts or best-of-k exploration strategies seen in RLHF training processes. As result, we draw preference pairs from more naturally occurring distributions (eg. real LLM responses and errors), better align with the expected environment reward models operate in.
For an overview of the curated benchmark datasets in PPE based on these two methodologies, please see Appendix A.I
4 HUMAN PREFERENCE METRICS
To benchmark whether reward model aligns with human preference directly, utilize a human preference dataset collected from a large-scale human preference annotation platform that allows users vote pairwise comparisons between responses generated from two anonymized and randomly selected L.Mss. Our human preference dataset contains human-labeled preferences for 16,038 pairwise comparisons between 20 selected top model These models were selected based on their strong performance on Chatbot Arena and overall popularity (Chiang et al. (024). We em- phasized selecting models that have already undergone some form of RLHF, anticipating that these models would be more challenging for reward models to evaluate.
Since the human preference is crowd-soureed, we can repeat the collection process at any time to obtain updated set that better reflects the current array of available models and any changes in human preference. Additionally, a newly updated human preference set would largely mitigate benchmark leakage that may have occurred with the previous set. Consequently, this human prefer- ence metric can remain consistently up-to-date with fresh, relevant data.
RewardBench is currently the only other evaluation scheme for RLHF reward models Lambert et al.
2024
"mistral- large- 2402, 44-----mdiumm-kk-i instruct, gpt-4-1106 106--ei claude claude-3-opus 3-0200u22--22029, 20240229, 4eminii..gemmiiii- 1.5-
pro-api-0514 0514, gpt-4-0314, Sauue-3-haikk- 202440077 4pt-4-0613, Claude-3-sonnet 20244222, yi-1.5-34b 1.55-34b-chat,
--nist-tt gemini-1. -emmini-.55-llashee 1.5-flash-api-05144 llama-3-70b-instmac 2p1-40-2024-05-13 command-r-plus,
gp1-4-trtho22244-04-099 qween22299 qwen2-77b-instmnet command-r, qwen 5-77bb-hhh, ltar-in

---

Published as a conference paper at ICLR 2025
Accuracy on Human Preference
$6.59
67:33
56.56
66.46
65,71
Model Type
Judge
05
64
Reward Model
62.05 60.57
60
59.28
59:12
Emsenbbe-cedeee Judipes
GF7-46- 207-44-2022-44 00-06
Cluuee-3- 5-Sotrett 5-500ett-2210000 201404220
Athene AM 700
GFT-40 Goi-4o-M0n--000 2024-02-411 18
Geniei11 5-Pro
Athene-WW -11--------0 60
3 700 Pout-nntru instrue
Gemin-t Semmi----5-- 1.5-tro
Sapaort Rovart Uomen
Intomtar 78- Raword
Imseroucc2200 tnward
Armoltt- Armute--aamst Saame (amme----- 1-----0--- 10.1
Bematron 4-3400 Roward
Li Ofterriaa--te 144-8
Skyscrt Rrovart Geromo 2:270
Figure 2: Model accuracies predicting human preference labels on PPE's human preference benchmark dataset. Accuracies are measure on the "Overall" category.
4.1 CURATION
Specifically, we curate our human preference data from crowd-sourced battles. A "battle" consists of user-provided prompt, two models and their responses to the prompt, and the user's preference vote for the responses. We perform random sample weighted by model occurrence to obtain 50,000 collected battles between selected models such that models are represented then de-duplicate and remove any samples containing P.I.I information using Azure Al. We use OpenAl's moderation API flag and remove potentially harmful conversations from the sample. Finally, we subsample 16,038 pairs from the remaining battles to construct the human preference benchmark dataset.
uniform frequency,
The human preference benchmark dataset, at glance: (1) Includes 4,583 instruction-followinng prompts, 5,195 hard prompts, 2.564 math prompts. Prompts may exist in multiple categories, (2) Includes user queries from over 121 languages. Top languages include English (8,842). Chinese (1,823), Russian (1,779), German (568), Korean (338), Japanese (321), etc. (3) Includes preferences crowdsourced from 6, 12 individuals.
4.2 SCORING
We conduct several statistical metrics described below to evaluate different aspects of given reward model.
1. Accuracy. We compute pairwise ranking accuracy against a human preference label for each re- ward model, excluding battles in which the human rater selected "tie". This measures the granular case-by-case similarity to real human preference signal. Figure visualizes accuracy scores on the overall human preferences.
2. Correlation. Since each battle contains information on model identities, each reward model pro- duces ranking and a pairwise win-rate matrix for the 20 selected models. We compute Spearman and Kendall correlation between model ranking produced by each reward model against ground truth ranking. In addition, we compute row-wise Pearson Correlation between the win-rate matrix pro- duced by each reward model against the ground truth win-rate matrix. We intuit that these aggregate correlation metrics measure overall similarity to real human preference.
3. Confidence. To weight stability in assigning preferences, we follow the metrics proposed in (Li et al. 2024b), where we measure each reward models's Separability with Confidence Interval, Confidence Agreement, and Brier Score against ground truth ranking. These metries are designed to measure uncertainties and over-confidence within a reward model.
Furthermore, can calculate all the above scores conditioned any subset of prompts in the evaluation data, specifically capturing different domains. For example, we can observe these metrics only math prompts only instruction following prompts. We expect that strong reward models should score high regardless of the selected domain. Scores for all subsets are detailed in Appendix A.2 Score distribution statistics for each metric detailed in Appendix A.2.1
4

---

Published as a conference paper at ICLR 2025
5 CORRECTNESS METRICS
To measure a reward model's ability to distinguish between different samples drawn from the same distribution, we utilize correctness metries on established, reputable benchmarks with verifiable ground truths (e.g. Austin et al. (202T)'s MBPP-Plus). We construct a benchmark dataset wherein each prompt is associated with 32 different responses sampled from the same LLM. Additionally, since we use benchmarks with verifiable ground truths, we can score the correctness (a binary label) of each response according to the original static benchmark's i verification function (e.g. code unit tests or Regex matching).
To assess the performance of reward models (and aujuuess- we obtain rewards/preferences for the sampled responses and evaluate how well these align with the verifiable correctness signal, with the general assumption that expert humans would always prefer correct answers over incor- rect ones. Our response sampling strategy ensures that the preference labeler must disentangle the correctness signal from potentially very similar or even adversarial outputs, thereby increasing task difficulty. Moreover, this method naturally samples "unforced" errors as they would appear in real training or evaluation schemes, rather than synthetically constructing preference pairs that may con- tain underlying confounding biases.
5.1 CURATION
For the correctness metrics, we selected standard, widely used, reputable, and verifiable benchmarks: MMLU Pro (Wang et al. 2024b), MATH (Hendryeks et al. 20 GPQA (Rein et al.] 2023), MBPP Plus (Austin et al. [2021). and IFEval (Zhou et al. 2023). Each benchmark covers different domain: general knowledge, mathematics, STEM, coding, and instruction following, respectively. While initially curate PPE with these five benchmarks, it should be noted that any desired verifiable benchmark can be added to the correctness measurement paradigm by repeating the process outlined below, thereby providing a framework for customization towards specific evaluation needs.
For each benchmark, we sample LLM responses for 500 randomly selected prompts, each 32 times, for a total of 16,000 completions. If benchmark has fewer than 500 prompts, we use all avail- able prompts. We choose a large K of 32 to allow models to generate more diverse responses, covering larger input domain for the human preference proxy and testing greater robustness to over-optimization. We note that this sampling strategy actually yields very similar KL-Divergence shifts as would be seen in RLHF training methods such as Proximal Policy Optimization (PPO) (Gao et al. 2022, Schhulann et al., 2017).
We repeat this process for four different models: Llama-3-88 3-88---88tBt--- Instruct, Gemma-2-9b-ig Claude-3- Haiku, and GPT--4--min--2224-07-11 (AI@) Meta 2024, Team et al. 2024 Anthropic 2024| Ope- nAI 2024). Each model samples prompts randomly with different seeds. We reason that different model response distributions may have different difficulties. For example, an already extremely high- nigh-performing model like 11440-mini-2024-07 may be more challenging for reward models to evaluate correctness.
We then score all responses using the benchmark's 's verification methods. Using the correctness labels for all responses, we discard any rows in which the model got every single response wrong or every single response right, as it is impossible for the reward model to select a better generation in these cases. Additionally, we discard any row where less than 10% or greater than 90% of the responses were correct, with exceptions made for benchmarks with very few valid options. This step helps avoid vacuously correct responses, such as an LLM randomly guessing the correct multiple-choice answer with completely nonsensical reasoning, as well as prompts that are too easy.
From the remaining data, we randomly sample 128 responses from each model, totaling 512 sam- ples. If benchmark is too small and some models have fewer than 128 viable samples, we adjust the sampling accordingly. More details on curation can be found in Appendix Appendix(A.3.1
5.2 SCORING
We score the reward models on the correctness metrics in ways that target reward model's bustness, granularity, and theoretical roof-line performance. Details on reward model and Ilm-judge scores can be found in Appendix A.3.3 Score distribution statistics can be found in Appendix A.3.4
5

---

Published as a conference paper at ICLR 2025
Gest of Score Benchmariss
HBPP Best Score
14
1.0
03
0.5
Earo
Score
in Renort
Man bed thmuue
Staning 19.24
25 0.4
10
5 30
15 20
Figure 3: Best of K curves showing reward model score vs K. The blacked dashed line is the theo- retical optimal curve, closer this curve implies better score. The left graph shows each reward model's 's curve averaged across all correctness PPE benchmarks. The right graph shows each reward model's curve on just the MBPP-Plus where over-optimization behavior is seen in some reward models, characterized by curves that decrease with respect to increases in K.
Reward Model
MML.U-Pr
MATH
GPQA
MBPP-Plus
IFFval
Mean
Athene-RM--D0
0.77
0.79
0.59
0.68
0.62
0.69
Claude 3.5 (ArenaHard))
0.81
0.86
0.63
0.54
0.58
0.68
3-fna---OffsBBiiss RM-8B
0.62
0.68
0.55
0.74
0.62
0,64
GPT-4o-mini (ArenaHard))
0.71
0.81
0.57
0.54
0.56
0.63
Llama-3.1-708 (ArenaHard)"
0.73
0.73
0.56
0.58
0.56
0.63
MMcerl..M2220--Rewaaa
0.68
0.70
0.57
0.58
0.62
0.63
Athene-RM-8B
0.68
0.71
0.55
0.62
0.57
0.62
AmmoRM-Lamma-8BB-66
0.66
0.71
0.57
0.54
0.58
0.61
SkywortkRRowatt-LLmm33.....
0.64
0.70
0.57
0.52
0.61
0.61
1omootron-4-440--Reeaaaa
0.70
0.65
0.57
0.49
0.63
0.61
Intenl.MM277-RRwwand
0.67
0.73
0.55
0.44
0.64
0.60
Llama-3.1 Mama-3.1170B (Alpuca)*
0.66
0.66
0.56
0,52
0.56
Claude 3.5 (Alpaca)'
0.66
0.63
0.56
0.52
0.57
0.59
Skywork-Reward Sarraar--Gmmm--22710
0.54
0.63
0.53
0.59
0.54
0.56
GPT-4o-mini (Alpaca)"
0,57
0.64
0.53
0.52
0.56
0.56
Naive
0.48
0.50
0.48
0.31
0.52
0.46
Table Reward model and LLM-aas--judge- scores on the correctness accuracy metric. LLM-as-a- judge is marked with f.
5.2.1 BEST OF K CURVES
best of curve shows on average how the reward model's selected "best" answer's ground truth score changes K. When plotted against the ground truth curve, the reward model's ability to select the "best" answer given a set of K responses, and the "gold standard" best score. More formally, let SK be a size K random sample of responses from a model, g: SK + {0, 1) be the ground truth scoring function, and R SK - IR be the reward model proxy score. Then, Es [g( (arg max,e R(s))] is the expected ground truth score of the select response by the reward model given K sampled responses. We then sweep across K I.... 32 to obtain curve. More details on these curves and derived metrics can be found in Appendix A.3.2 Best of K
observe the gap between
scores for various reward models detailed in Appendix Table 30
5.2.2 AREA UNDER RECEIVER OPERATOR CHARACTERISTIC (ROC) CURVE
Since the ground truth verification outputs binary label, we can check each reward model's strength
binary correctness classifier by calculating the area under the ROC curve. We first normalize the scores in each row with min-max normalization. Then we calculate the binary classification ROC curve using the normalized scores "probabilities". AUC scores detailed in Appendix Table 31

---

Published as a conference paper at ICLR 2025
5.2.3 ACCURACY
Since -La-dae- cannot easily scale 32-wise judgments, we create supplemental pairwise task to evaluate correctness preference accuracy compatible with both reward models and LLM-as- a-judge. For each row of best of data, we simply sample 5 pairs of responses such that in each pair, there is one correct response and one incorrect response. Then, after randomizing positions, the LLM-as-a-judge picks the preferred response. We then measure the accuracy as the rate in which the correct response is preferred over the incorrect result. The accuracies for reward models also collected for comparison. All scores are documented Appendix Table 2
VALIDATING PPE ON POST-RLHF OUTCOMES
By testing reward model performance on a benchmark, we hope to glean insight towards down- stream performance on an LLM RLHF-ed using a given reward model. To measure how well dif- ferent metries PPE correlate to post- RLHF LLM performance on real-world human preference, we conduct an experiment in which we RLHF a given base LLM using different reward models. We then measure the real-world human preference scores of the resulting LLMs to understand the true performance of the original reward models.
For our experimental setup, we use each reward model to individually RLHF Llama-3. -8St-nsttrtt through Direct Preference Optimization (DPO) (Rafailov et al., 2023). This way. we can compare LLMs tuned on identical RLHF pipelines, except for the reward model being measured. Then, these RLHF-ed LLMs are deployed to to crowd-sourced annotation platform collect real-world human pairwise preferences between model answers. Overall, 2,190 human votes were collected and compiled into relative rankings between these RLHF-ed LLMss. Under this controlled RLHF experiment, the non-noise variance in final human preference rankings attained by these models is dependent only on the reward model choice, effectively measuring the downstream performance of these reward models, albeit on a single model base model undergoing off-policy DPO RL training.
6.1 TRAINING PROCEDURE
Nine reward models were selected to act as preference labels in full RLHF training pipeline in which the resulting models were evaluated on real human preference. We constrained this ex- periment nine models for cost reasons- the RLHF and human preference evaluation process is exceedingly expensive. We selected popular, newer, and high-performing reward models from Re- wardBench. We reason these will be the most difficult reward models to differentiate. We also require the selected reward models to be general- purpose reward models, and not specifically tuned any single domain task.
We create a training dataset by first including 7,000 prompts sampled from the original 50,000 human preference votes after PII removal, unsafe prompt removal, and de-duplication. We then add 500 random prompts from MMLU-Pro that are not in PPE, and another 500 prompts from MATH train set (also mutually exclusive from PPE). For each prompt, we sample 16 responses from the base model, --ama-3.1-8B-nnste a triangular distribution (a = 0.0, b 1.0,c = 1.3) to promote more diverse exploration. This process yields 8,000 total prompts, each with 16 different responses, totaling 128,000 responses.
randomizing the temperature for each generation, drawing from
Each reward model then constructs its own preference dataset. First, the reward model gives scores for each of the 16 responses for each prompt. The "chosen" response is set the maximum scoring response. The "rejected" response is sampled as the rank n response, where n is sampled uniformly. Note that the sample for is seeded such that it is the same for each across reward models. This pro- cess yields a dataset of 8,000 rows, each with a prompt, a chosen response, and a rejected response where both responses are in-distribution for the base model- requirement for using DPO.
*Selected: Athene-RM-70B and Athene-RM- L 3-OfffsetBiis- RM-8B, ArmoRM- -B--.0.L,
MitemLLM--20B-BB Skywork- Reward-Gemmet Gemma-m-mmaaaa-
InternI.M2 Ma--nlM2--BBeee Reward, Skywork-
Reward Birvard-Llanee Llama-3. 3.B. Nemotron 4-340B-33BBBBB Reward (Frick et al, [0024 Cai et al. 2024 Park et al. 2024 Wang
et al. 2024a Liu & Zeng 2024 Wang et al. 2024c). Evaluated on Preference Proxy Evaluations (PPE). but not selected: Starling-RMr RMB34B, Starling- RM-7--M-MBB---- A-Apha, Furus-RM-7B,
1....M---w..aaaarrd and
WaiveeerroosityModeell (Zhu et al. 2023a Yuan 2024
Cai al. 2024
7

---

Published as a conference paper at ICLR 2025
Model Elo Scores with Confidence Intervals
Categories
1240
Tested Reward
1220
1208
1216
:209
:204
1180
1199
13178
1176
1172
1182
Meta Lon----aaaa.. 3. 1--0--70- Instruze
Athene- Athere-Attt Ri-- 708
Abhene-RMe RM----
Internin2- po-rwwrrd
Liema-3: Oftsetiion RM--8 88
ArmoRt--Liee Lllomm 4B--0. 48-v0.1
Meta-Llma-- J.1 1-48--88 Instruct*
Skywert Rowaard Llame-3 1amm-3.116 3.1-88
Internim2: reward
Skywerk- Spweri-Rowwrd. Rew---wwwrdddoo Gomma 2-278
Nemetron-4 -00---4440---00 340--3308-00-- Roward
Meta- Lima. Instrrcct
Model
Figure 4: Post DPO performance real human preference the Overall Category. "Model" is the reward model used to train the hase model unless specified as a reference base model.
We then train Llama-3.1 1amma-3.1-25 1aama-3.--8B-Instaee ---8--nnstttctt on each dataset using DPO producing an L.M associated with each selected reward model for real-world downstream human preference testing.
6.2 EVALUATION ON WORLD HUMAN PREFERENCE
We deploy the trained models to crowd-sourced sourced human preference platform to undergo blind eval- uation from real users. We set up cohort of 13 models which include the trained DPO models as well as Llama-3.1 1-ama-3.1-8B-Intree -----untttt 1llmm-3.1-700-Imme temperature 0.2 (excluding -lama-3-8B-Instmen evenly with only each other for battles. Battles were collected over six day period, from Septem- ber 10th, 2024 to September 16th, 2024. In all battles, the receiving user was selected randomly. Additionally, the model names (labeled 1 11ama-2 3. 8b dpo dpo-testt..teee 2 9}) were not revealed to the user until after the vote was given.
All models used at temperature 0.7). Model pairs were sampled
and I Alama-3--B-Instruue
Overall, 12,190 human preference votes and an average of 190 battles per unique model pair. More details on battle statistics and be found in Appendix Table 39 of Appendix A.5 The resulting preference rankings detailed in Figure 4 The preference rankings are calculated using the Bradley-Terry model, as proposed in Chiang et al. 20044)
collected, with average of 2,032 battles per model,
STUDYING CORRELATION WITH DOWNSTREAM PERFORMANCE
In this section, analyze how different metrics correlate with post-RLHF human preference scores (experimental setup detailed Section 6.2). Our main results are displayed in Figure which shows the correlations of our offline reward model evaluations against the real-world human prefeereeee ranking from the crowdsourced platform.
On correctness metrics (left plot in Figure domains is well correlated across all metrics, but exhibits higher correlation with AUC and Accuracy scores. (2) Math the best individual benchmark domain in terms of predictive power. (3) ROC AUC score draws higher correlation across all benchmarks, even on benchmarks that are otherwise uncorrelated.
make several observations: (1) Mean across all
Turning to the right-hand side of Figure 5 the accuracy of the reward model the best predictor of the fine-tuned LLM's preference score. Row-wise Pearson Correlation, Confidence Agreement, and Separability show some correlative power to downstream human preference rating but do not exceed accuracy. Meanwhile, metries like the Spearman correlation and Kendall correlation have nearly zero correlation with the final human preference rating achieved by the post-DPO models. One possible reason for this trend that accuracy measures expected preference correctness per preference pair- much more granular scale. Other metrics involve aggregating reward model signals over higher-order preferences, such as preference for each model, as measured by correla- tion metrics. We consider these metrics low granularity. Medium granularity metrics, such as Row-wise Pearson Correlation aggregate reward model signal, but do so over smaller subsets of preferences.
8

---

Published as a conference paper at ICLR 2025
Camelation Post-BLHH Arena Scate:
Cortelation Post-BLHP Arena Scote:
0.8
0:83
Harcon
MD auc
0.52
0.44
0.43
5.22
6.22
Poyrson Corcletion
Spparsbary
0.40
0.51
0.22
5.45
0.01
0.50
0.50
Pax
0.50
0.43
632
Conteonci Ag sooorrr
0.23
0.43
0.30
0.30
-02
End Scard
6.30
9.66
6:43
Sporman
-0.05
10:15
0.00 -5.55
0.16 0.01
0.50
-0.4
0.30
0.00
0:30
0.26
9,30
6:50
aiter Scare
0.09
6..0
0.23
-2.00 0.12
011
-0.6
PM
Dil Foomert
Smeder Regress
Mati
Qurdatyy Levtl
Figure 5: Pearson correlations of different metrics toward downstream human preference scores. Left: Pearson correlation between the ranking of models on 5 specific benchmarks and different metrics and their respective post-DPO rankings on real human preference. Right: Pearson correla- tion between the ranking of models on 7 categories and metrics on the Human Preference Dataset. A similar version using style controlled human preference as reference is shown in Appendix Fig. ure
Correlation to Post-RI Arana Score: on
Dataset
Nummonn Comrelation
0.5
Accurwcy
40.83
-0.82
0.30
..79
-2.52
-0.99
0.0
-0.3
Hoan Safimp)
Mean
Herd
Saaltty
Resanning
Figure 6: Pearson correlation between the ranking of models in RewardBench and their respective post-DPO rankings on real human preference. Style controlled version in Appendix Figure 12 Comments on these correlations can be found in Appendix A.6.1
A.661
Overall, accuracy on the human preference dataset is more correlated than the correctness metrics. This is because correctness and human preference do not necessarily align. Moreover, the informa- tion contained Loss, Max score, and End score may not prove relevant in DPO, which off-policy. Those employing RLHF algorithms that have higher risk of over-optimization may find these al- ternative measures helpful. However, when calculating correlation against style controlled ratings S notice slight decrease in correlations on the human preference dataset. Notably, the correctness preference measurements show no change, suggesting correctness preference may be more robust towards reward model preference quality, response style aside. We leave details for Appendix A.6.2
Additionally, we observe that measuring the lower bound score may correlate more to downstream RLHF performance than the average score or upper bound score. In Figure we first re-scale each category's 's scores to be mean 0 and SD 1, then we vary the quantile of the aggregation strategy across human preference dataset categories seen in Appendix Table (Hard Prompts, Easy Prompts, etc). In this case, the 0) quantile is the minimum, and the 1 quantile is the maximum. We find that in nearly every metric, decreasing the quantile increases correlation with downstream ratings. We posit that the increase in correlation to downstream when using low quantile aggregation across metrics because this strategy closer measures the robustness of the reward model. This is in line with previous theoretical work has suggest that pessimistic measures on reward model performance may be useful (Zhu et al. 2023b Li al. 2023). See Appendix A.6 for more details.
Recommendations for PPE based on these findings can be found in Appendix A.7
*Style controlled ratings are calculated detailed in Li 2024ap.

---

Published as a conference paper at ICLR 2025
Performance Correlation Human Preference Category Score Quantile
Spearman Correlation
Confidence Agreement
Accuracy
0.50
Pearson Correiation
Correlaton
0.6
Corelaaton
0.2
0.5
0.75
0.4
0.0
0.70
0.3
0.2
0.65
0.1
0.0
0.3 Category Score Quandle
0.2 0.4 0.6
1.0
0.0
0.8 Cotegory Score Quantle
0.2
0.4
0.6
1.0
0.0
0.2
0.4 0.6 0.8 1.0
Category Score Quantle
Figure 7: Spearman Correlation, Confidence Agreement, and Accuracy metrics: For each metric, take the quantiles of category scores (Hard, Easy, Instruction Following, Coding, Math, and Similar). The Pearson Correlation is calculated relative to Post-RLHF Human Preference ratings for each quantile. Notably, accuracy peaks 0.80 correlation at low quantile aggregation.
LIMITATIONS
8.1 BENCHMARK LEAKAGE
We acknowledge that benchmark leakage is a very real possibility. We also consider two factors that help mitigate this issue: (1) The human preference dataset be updated with new crowdsourced preference data at any time. This includes adapting to the most recent prompt and response distribu- tions. (2) The correctness preference datasets be extended to any other benchmark that becomes standard enough to be widely used.
8.2 LIMITS ON TESTING DOWNSTREAM PERFORMANCE
Unfortunately, end-to-end evaluation of reward models via PLSt- LLM performance on human preference is extremely expensive and time-consuming. As such, we are limited to testing the per- formance of nine select models, rather than all reward models. In addition, RL algorithm over PPO, an online algorithm, which may play more into over-optimization issues or may have different reward model requirements altogether. We encourage future work to study downstream outcomes under online RI. algorithms. Moreover, we note that resource constraints necessitated experimenting with just Llama-3.1-8B, -88--Instttttt as the base policy model; additional exploration diverse set of base models may yield additional novel insights. With these consid- crations, we note that the downstream performance measured in our work is in the context of the base model and RLHF learning algorithm used, and is not unilateral measurement of downstream out- comes in all possible configurations. Future work should experimentally verify the desired reward model behavior of other RLHF configurations.
use DPO, offline
CONCLUSION
We present PPE, a reward model benchmark explicitly tied to post-RLHH outcomes based on real human preferences. Our experiment aims to identify which metrics, applied to specific tasks, cor- relate most strongly with downstream performance. We find that across the board, granular mea- surements. such as accuracy, are the best predictors. Additionally, our results suggest that measur- ing lower bound performance may be more indicative of expected reward model performance in the RLHF pipeline. Overall, our evaluations achieve 77% Pearson correlation with downstream performance, significantly improving upon previous work. Based on these results, we encourage future research to further investigate reward model quality and downstream RLHF performance un- der broader conditions. We fully open-source dataset creation, experimental validation, and reward model evaluation code and methods. We anticipate that the high-qualis, preference evaluation in PPE, combined with our post- RLHF analysis of metric predictive power, will significantly advance vital research into reward models and RLHE.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is funded by the Sky Computing Lab UC Berkeley. This project was also partially supported by NSF grants ISS1191252 and CCF-221 1209, We are grateful for the generous support for compute resources from RunPod.

---

Published as a conference paper at ICLR 2025
REFERENCES
AI@Metaa Llama 3 model card. 2024.
URL
-lama3/blokk /MODEL CARD md
Anthropic. The claude model family: Opus, sonnet, haiku,
com de 9b0 c9ab coab 3033333b00000 00ea000bba 7bbc 61b7 Bttr:/Mooo____r_____e Caro__Craalaaa
3.p 2024. (Accessed on 06/05/2024).
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https://azxiv.. wwwpabs/////oooooooeeo...ote..aa 2108000080000 07732
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05862, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran- Johnson, Ethan Perez, Jamie Kerı, Jared Mueller, Jeffrey Ladish, Joshua Landau. Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer EI Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Con- erly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zae Hatfell-Dodde Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback. 2022b.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internim2 technical report. arXiv preprint arXiv:2403. 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating lims by human preference, 2024.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. 2023.
Yann Dubois. Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpucafarm: A simulation framework for methods that learn from human feedback. arXiv preprint 1xX1::2305.1488. 2023.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learming Research, pp. 5988 6008. PMLR, 17-23 Jul 2022. URL
html
Evan Frick, Peter Jin, Tianle Li, Karthik Ganesan, Jian Zhang, Jiantao Jiao, and Banghua Zhu. Athene-70b: Redefining the boundaries of post- post-training for open models, July 2024. URL
https://huggiagfe
Coo/Nexusfloc wow.///eeessELL/////eeee-e11 Ath
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot lan- guage model evaluation, September 2021. URL /tttpa://oii.rrr/000523..... zenooo. 5371628
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. arXiv preprint 2rXiv:2210.100 10760, 2022.
12

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt, Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling. https: //hggg weaggiingfaco nace.e. 2024.
co/spaces/al welllenaaii
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023.
Tianle Li, Anastasios Angelopoulos, and Wei-Lin Chiang. Does style matter? disentangling style and substance in chatbot arena, August 2024a. URL
2024/style sts/styllloooool////
control/
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap. Tianhao Wu, Banghua Zhu, Joseph E. Gon- zalez, and Ion Stoica. From crowdsourced data high-quality benchmarks: Arena-hard and benchbuilder pipeline, 2024b.
wttp://///a..//ooeeeeeeee 2406. 11939
Zihao Li, Zhuoran Yang, and Mengdi Wang. Reinforcement Jearmin with human feedback: Learn ing dynamic choices via pessimism, 2023. URL ht
18438
Chris Yuhao Liu and
reward model series, http://wwwwwwaooooooooo////so//n
Skywork September 2024. URL /ttp///tss///uggggggg/././//////
OpenAl. Introducing chatgpt. https: bttes://peeaa..commbloo///aooggp 2022, (Accessed on 01/12/2024).
OpenAI. Gpt-4 technical report, 2023a.
OpenAl. New models and developer products announced at devday, httpps://opeaa: com/
http://wwwnnneeeaaeeeeeeeeeeeeer (Accessed on 01/12/2024).
2023b.
OpenAl. Gpt-4o mini: advancing costeefficint intelligence. ht 06/05/2024).
htl.gnneeeertttee/ggaaaaa..////m 2024. (Accessed on
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Lever- aging debiased data for tuning evaluators, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.1888 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju. Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models a practical size. arXiv preprint arXiv:2408.001 18, 2024.
13

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXtv:2302.13971, 2023.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. /ttpp::://ithhuee com htt.///gggggggaa//////aa// 2020.
Haoxiang Wang. Wei Xiong. Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-expers In EMNLP, 2024a.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint 2rX1v:2406.01574, 2024b.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models, 2024c.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen. Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng. Zhiyuan Liu, and Maosong Sun. Advancing Ilm reasoning generalists with preference trees, 2024.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following, 2024. BtL.https://arxivv.
abs 221002310... 07641
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. instruction-folloing evaluation for large language models. arXiv preprint arXiv:2311.079) 2023.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving I1m helpfulness & harmlessness with rlaif, November 2023a.
Banghua Zhu, Jiantao Jiao, and Michael I Jordan. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. arXiv preprint arXiv:2301.11270, 2023b.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, Karthik Ganesan, Wei-Lin Chiang, Jian Zhang, and Jiantao Jiao. Starling-7b: Improving helpfulness and harmlessness with rlaif. In First Con- ference on Language Modeling, 2024.
14

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1
OVERVIEW PPE BENCHMARK DATASETS
Name
Num Prompts
Response per Prompt
Preference Type
Human Preference VI
16,038
2
Real Human
MMLU Pro
512
32
Correctness
MATH
512
32
Correctness
GPQA
512
32
Correctness
IFEval
512
32
Correctness
MBPP Plus
507
32
Correctness
Table 3: Released benchmarking datasets in PPE.
A.2 DETAILED SCORES FOR THE HUMAN PREFERENCE EVALUATION DATASET
You may include other additional sections here.
Reward Model
Accuracy
R. Pearson
Sepumbility
Conf. Agree.
Kendalltau
Spearmanr
Brier Score
Ensemble.Juge Judges (ArenaHardd)*
68.59
82.49
84.21
96.21
87.37
96. 54
0.05
Ensemble-Judge (AlpacaEval))
68.52
81.25
79.47
93.94
85.26
95.04
0.07
GPT-4o-2024-08-06 (ArenaHard))
67.71
81.07
80.53
94.70
86.32
96.24
0.06
Claude-3 3-5-Sonnet--eeeee 20240020 (ArenaHard))
---Sonmet-22224666
67.33
80:65
79.47
94,70
88
96.69
0.06
GPT-4--0244-08-08
(AlpacaEval)"
67.13
77.92
76.32
90.91
84.21
93.23
0.07
Athene-RM-F00
66.56
80.69
84.74
95.94
82.11
93.23
0.07
MTT-fo-MMiii2222-00--A (ArenaHaad))
Gemini-1.5 1.5---1---P--0 (Alpacabval)"
66.09
82.63
83.16
96.21
86.32
95. 19
0.05
Gemini-1.5.. P.5- (ArenaHand))
65.71
82.23
83.16
90.53
96.99
0.04
Claude-3-5 1.5--Somet-222240000 (AlpacaEvval)
65.34
73.91
74.21
85.61
71.58
85.26
0.11
Lluma-3,1 3.1-0B- Instract (Alla
65.27
74.81
79.47
$7.88
72.63
85.56
0.12
Gemini-11 5-lassh-002 (AlpacaEval))
65.04
74.29
78.95
88.64
74.74
88.72
0.11
AtheneRMM. SB
64.59
76.85
.6
91.67
77.89
90.53
0.10
1-01a-331-0B-Insttsccc (ArcusHard)"
Gemini-1.3 P.5-Paah--022 (ArenaHard)
63.01
76. 12
76.32
90.91
76.84
90.23
0.10
Starling-RNN RM-34B
62.92
70.47
77.37
78.79
67.37
81.20
0.15
1091-4o-Min-2224--00 (Alpacalsval))
62.75
68.
70.53
75.79
0.10
Gemini-1 1 P.5Pro-001 (ArenaHand))
62.57
75.92
$1.05
93.18
85.26
94
0.07
Myyaork-Roward-Lii
M2 -BB-warddd
62.05
68.03
78.42
69.70
56.84
76.09
0.20
Eurus. RM-78
62.02
60.37
75.26
64.39
51.58
65.26
0.22
LammRM-LLammSSB---0
60.57
71.85
76.84
84.85
76.84
89.17
0.10
Naive Mede
59.81
32.03
76.32
35.61
29.47
33.53
0.33
Nemotme--4340pp Satotteee--340-RRRewwwe
340a--ewwrrd
59.28
Maama-3-OffstBiase RM--B
59.12
58. 58.86
65.79
61.36
$1.58
69.02
0.20
Starling RM-1B- 22M1B-AAlha
58.93
58.42
70.00
67.42
50.53
64.66
0.22
Interl 12-1.88 Reward
57.22
47.11
69.47
36.84
0.28
Skywod&-Rowad-ee Row.rd-GGnmaa---25
56.62
69.99
69.47
87.88
84.21
95.49
0.07
Table 4: Reward model and LLM judge performance on Overall subset the human preference dataset. LLM-as-a-judge are labeled with system prompt source, and marked with t.
15

---

Published as a conference paper at ICLR 2025
Accumccy
Segumbility
Conf. Agree.
Spearmanr
Ensemble-Julges (ArenaHard)
69.46
67.05
74.21
96.88
83.16
94.44
0,06
Claude S-uue-----Soonct 3--Sonet-22124400 (ArenaHand)"
72.11
97.92
GPT-4o-200 2024-022--00 08-06 (ArenaHand))
68 (4)
68. 17
71.05
97.92
85.26
95.94
0.06
Gavemble-Judge (AlpacaEval))
TTT-fo-Mimii2224-000 (Arenafard)
66.63
63.55
71.05
95.83
82.11
94.29
0.08
Gemini-1
(Alpacabval)"
66.53
84.21
Athene-RM-P00 70B
66.43
67.01
76.84
96.88
78.95
92.93
0.08
CPP--4-2022-00--2 (AlpscaEval)
66.30
62
69.47
96.88
78.95
93.23
0,09
Gemini-11.-Prooee P.5-r000000 (ArenaHand)
Alama-3.1-700 1.11--0B-Insmee Instroct (AlpacaEvall)
64,96
65.76
65.26
90.62
70.53
87.82
0.11
Llama-3. Instrrccctt (Arenalard))"
Sthene-RM-B3
64.41
62.44
74.21
96.88
74.74
87.97
0.11
Gemini-1 1,5 1.5-Flash-002 (AlpacaEvaaly"
64.35
62.30
65.79
94,79
77.89
91.43
0.09
Gemini-1.5 Semini-1.5-Flash-000 (AresaHard)"
64.18
60.68
67.37
94.77
81.05
92.
0.08
Claude 3-5 Sonnet-222 202-00222 (AlpacaEvval)
64.14
56.81
26
90.62
73.68
88.42
0.11
Starling-MMi
Gemini-J 1.5 1.5-Pro
(Arenafaad))
63.53
67.93
68.42
96.88
85.26
95.19
0.05
Earas-
62.75
72.78
MiteraLLM2-B-BRwaand
62.14
60.77
67.37
85-42
65.26
83. 16.16
0.14
61.56
59
67.37
83.33
71.58
88.87
0.12
GPT-fo-Minn-2024-07- (AlpacaBval))
Skywork. Reward Llama- 3-1BB
61.15
62.46
68.42
83.54
70.53
86.62
0.11
AmmoM-II Jamo-----x..
60.99
61.81
58
89.18
70.53
87:22
0.11
NaiveVerhosity
59.67
37.71
66.84
66.67
44.21
58.65
0.25
59.42
56.03
62.11
80.15
0.16
Nemotroe-4-34005 Reward
$9.06
55.82
67.37
87.50
73.68
90.38
0.10
IntemL.M2-1188 Reward
58.49
52.40
61.58
63.54
48.42
63.91
0.21
Starling-MM- -M-7B-Alpha
Skywork- Reward- Sewwanl-Gemma-2222 2-27B
56.21
40.13
38.42
63.54
70.53
89.02
0.11
Table 5: Reward model and LLM judge performance on Hard prompt subset of the human preference dataset. LLMMaas-ajuudge are labeled with system prompt source, and marked with t.
Reward Model
Accuracy
R.W. Pearson
Sepurbility
Conf. Agree.
Kendalltau
Spearmanr
Brier Score
EnsembleJJugge Judgees (AlpacaEvall)'
70.15
52.24
$2.10
83.33
75.79
91.58
0.09
CPT-1o-2024-08 0124-08-06 (AlpscaEval))
69.97
52.01
47.37
83.33
72.63
90.08
0.09
EmsembleJuuggs (ArenaHard))
69.
57.24
63.16
83.33
83.16
94.74
0.08
GPT-40-2034-08- 0024-08-06 (ArenaHand))
68
56.01
52.10
81.25
77.89
93.53
0.08
GPT-Ho-Mini-2024-07 (AremaHard))
67.50
50.08
46.32
78.12
72.63
$8.72
0.09
1.-m--3.1770--hmm Instrocs (AlpacaEval)"
67.40
46
46.32
68.75
60.00
80.60
0.14
Gemini-1.5. P.5.Pro-002 (Ard))Hard)'
Claude 3-5
3-ndo-----S Sonnet-20D 20240620 (AlpacaEval)
66.98
44.37
35.26
61.46
67.37
84.51
0.12
Claude-3-5 23-5-Souctt-22216 220202222222622000
(ArenaHaad))
Gemini-1 1-5--Fsssh--000 (Alpacabvall)
66.92
45.52
48.95
76.04
72.63
88.42
0.10
Athene RM-70B
66.90
58.55
61.21
93.75
77.89
92.48
0.08
Gemini-11 Siaiiii-1,.--00 1.5--ro-002 (AlpacaEval)"
65.96
51,60
53.68
84.38
81
93.23
0.06
Mini-2024--722 (AlpacaFoal))
65.39
42.01
25.79
46,8K
69 47
85.71
0.12
Llama-3.1 1.ttn-3.1-706..
I-n0g-Instrett (ArenaHard)
64.10
46.06
40.53
68.75
64.21
82.71
0.12
Skywork-Reward-LLnng 3oooadd-Llama-31--55
63.24
42.44
46.32
56.25
62.11
78.90
0.15
Gemini-1 5-Prro-011- (AremHaadd)
62.65
40.53
$4.21
78.12
80.00
93.68
0.09
Eurus-R3 RM-7B
61.52
34.66
41.05
31.25
36.81
45.71
0.27
MiemLLM2-7BRRewee
61.70
32.69
34.74
45.83
45.26
60.60
0.23
Starling-MM-34B
33.87
33.79
60.75
0.22
Gemini-1.5.0 F.50 (ArenaHard))
61.01
42.41
46.84
77.08
68.42
87.52
0.10
IntomLMM2208-Kcwwrd
ArmRMM-Laae LaA-RRMM-LmmSS----000
60.28
31.56
40.53
53.12
$8.95
73.08
0.17
Nematroe 4.34408---eBa
59.58
45.52
$6.32
68.75
67.37
$4.06
0.13
Naive MaineVerhosityMock
59.24
Starling-MMi RM-7B-Alpha
58.70
27.17
38.95
29.17
28.42
39.25
0.30
Llama-3 OffseeBBaas RM- RM-SD
58.66
35.23
29.47
29.17
43.16
$5.49
0.23
Skywork- Bhyaool-RRewand-Geeee www.Renn---ema----111--
56.74
45.42
40.00
77.89
InteraL 12-wad8-Roward
55.54
30.02
27.89
15.62
22.11
29.32
0.30
Table Reward model and LLM judge performance on Easy prompt subset of the human preference dataset. LLM-as-a-judge are labeled with system prompt source, and marked with t.
16

---

Published as a conference paper at ICLR 2025
Accumccy
Segumbility
Conf, Agree.
Kendalhuu
Spearmanr
Ensemble-Judges (ArenaHard)
69.77
66.89
70.00
97.09
83.16
93.68
0,07
Claude 3-sude-----onntte 4-5-Nonet- (ArenaHand)'
68. 38
70:
64.77
91.
Ensemble-Judges (AlpocaEval))
67. 1/.88
69. 18
70.00
96.12
86.12
95.04
0.05
GPT-4o-2024 20.4o-2024-0S- (ArenaHand))
Gemini-1. 15-Pro-002 (AlpscaEval))
66.78
68.61
73.16
97.09
88.42
96.54
0.04
Gemini-1
(ArenaHard)
66.70
Athene-RNN Mahene-RM-70BB
66.50
63.79
75.26
95.15
77.89
90.98
0.09
(FTT-o-202440S-006 (AlpscaEval)"
66.09
64.39
26
92.23
82.11
93.98
0,06
111-4---ini-2224-00------ (AremaHard)'
90.53
Gemini-1.5 1.5.-......lhhh00000
(AlpucaEvall"
65.43
64.33
65.79
89.12
82.11
93.38
0.07
76.84
I.aam-33 1lmma-3.1-700-- 1--70----nnee Instract (AlpacaEvall)
63.68
63.11
63.16
79.61
75.79
88.57
0.10
Claude 3-5 Sonnet 01240620 (AlpacaEvval)
63.42
57.93
59.47
81,55
71.58
87.97
0.10
Gemini-1.5-PP P.5--ro000 (ArenaHadd)
63.25
66.39
62.63
88.35
80.90
91.13
0.08
Lluma-3.1-70B Instract (ArenaHard)
63.04
39.85
62.10
83.50
76.84
90.83
0.08
Gemini-1.544 1.5-lla--00 (ArecaHard)
62.66
60.73
61.05
87.38
75.79
89.77
0.09
Nemotroe 0-000-44333 340B Reward
61.89
56.91
63.16
86.41
71,58
$6.92
0.11
Intert (00mmlMM-200--200
64.74
64.21
83.76
0.15
Myywoi-Rowanl-LLam--
61.41
57.88
66.32
31.55
74.24
12
0.10
MicrmLLM2-B-RReaard
61.41
35.07
64.74
66.99
63.16
80.45
0.16
Bouting-RM-3411
61.11
$2.85
61.05
77.67
65.26
82.41
0.15
GPT-4a 4444-MMmi222224-0-.2 (Alpacalival))
61.10
50.62
43.16
66.99
72.63
87.82
0.10
Famis-MM778
60.50
11.96
19.47
65.01
51.58
65.26
0.20
AmoRM.Llama--8B.
60.87
58.71
56.32
76.84
90.53
0.10
Llmmo---fffaBBBa-RM-888
60.22
55.63
51.05
83.01
0.15
InteraLM2-1.8
M0--MM2------- 8B-Reward
$7.27
38.46
55.79
39.81
42.11
59.55
0.23
Naive VerbosityyMMMeee
$7.07
31.21
56.84
32.04
33.68
47.67
0.29
Skywork Mhyyoork-Reward-Co RewaRd-ewadneen Seeoaand-Gemm----777
43.85
75.29
Starling-Ro MaalinggRMM-B--- RM-7B-MM----
55.71
40.10
48.42
52.43
44.21
$8.20
0.22
Table 7: Reward model and LLM judge performance on If prompt subset of the human preference dataset. L.M-ass--judgee are labeled with system prompt source, and marked with t.
Reward Model
R. R.W. Pearson
Sepurbility
Conf. Agree.
Kendalltau
Spearmanr
Brier Score
Claude Sonnet-m022400 20240620 (ArenaHard))
68.05
57.64
62.63
97.22
88.42
97.74
0.04
Ensemble-Judge (ArenaHaadd)"
67.98
58.22
71.58
91.67
84.21
96.09
0.05
GPT-4o-2024-08-00 (ArenaHard)
67.66
58. 16
65.79
97.22
88.42
97.29
Ensemble-tugges (Alpacalvaal)
67. 07.47
55.98
72.11
94 44
82.11
94. 14
0.06
Athene-RM-70B
M01-----ii--2022--0-. (ArcmaHard))
66.08
53
67.90
100.00
85.26
96.24
0.05
Claude So-noot-2222 (AlpscaBvvll)
Gemini-1.S. 1.5--1--PP P----r-0022 (Alpacabvall
65.57
36.07
63.79
91.67
76.84
91. 88
0.08
GPT-44-2022-08-00 (AlpacaFvval'
65.50
58.66
62.
94.44
86.32
95.94
0.05
Atlene-RM-SB
57.37
70.00
76.84
92.
I. 3.1 3.1-30R-Inssman (AlpacaFval)"
64.40
54.30
62.10
94 94.44
75.79
92.013
0.09
Llama-3.1-33 Ins-rnn-ssttttttt (ArenaHand)
94.
Gemini-1 1.5 1.5-Flah-002 (AlpucaEval))
64.36
42.96
$7.37
88.89
72.63
89.92
0.11
Starling-R RM-34B
64.29
56.23
66.84
88,89
74.74
89.32
0.
Gemini-11554 1.5.-1.5PProo (ArenaHand)
64.18
54.06
66.32
90.28
77.89
92.78
0.08
Interal M2-7R-Reward
63.53
46 74
65.26
84.72
68
86.47
0.12
62.98
Gemini-1.5-Plaahhe 1.50hhh-000 (ArenaHard)"
62.65
56.60
$4.74
95.83
80.00
93.68
0.07
InteraLM2-200-RRca 2ew-BBdddwwddd
62.10
47.74
58.95
90.28
75.79
91.13
0.09
GPP-44--Mii--2224Q07-- (Alpucatooal)"
61.77
44.74
77.89
93.68
Seenini-1.5.Pro0000 (ArenaHand)"
61.55
46.75
$6.32
94.44
75.79
91.43
0.08
NaiveVerbosityMoo
61.39
AmmoRM-Llams-55 -800--8--0011
61.01
49.10
$1.05
93.06
$1.05
93.83
0.08
Skywork Reward-LLama.. 1-SB
61.01
50.02
51.05
93.06
76.84
91.58
0.10
Mimma---OffseHiin--HH
59.80
64.21
0.14
58.76
45.07
58.42
62.50
$1.74
71.28
0.19
Starling RM-7B-Alpha
58.71
46.85
$6.32
76.39
64.21
78.80
0.15
Nemative.4- 4-310 Rewardd
$7.94
35.96
$1.05
72.63
89.62
0.10
Skywork Rew.----oo----naaaa--
56.41
25.46
26.84
$4.17
64.21
84.51
0.13
Table 8: Reward model and LLM judge performance on code subset the human preference dataset. LLM-as-a-judge are labeled with system prompt source, and marked with t.
17

---

Published as a conference paper at ICLR 2025
Accumccy
Segumbility
Conf, Agree.
Spearmanr
Ensemble-uugees (AronaHard)
73.58
$4.57
65.79
88.13
80,00
94.44
0,07
GPT-40-2024-08- (ArenaHand)"
72.57
82.11
94.
Clande: 3.5 S-Someet-21240022 (ArenaHaad))
71.79
49.92
60.53
88.73
78,
93.38
0.08
Mimii-2024-7-1BB (ArenaHard)
Gemini-1, 1.5tiii11.5-1 1.5-Pro-002 (Arenffaad)
69.61
60.91
$8.42
84.51
77.89
92.63
0.08
EnsemibeeJuudges (AlpacaEval))
1lama-3. 3.1---.--00----0 Insrant (ArenaHard)
68.93
46.05
$4.74
84.11
72.63
87.82
0.10
Allien-RMM-
68.58
57.39
67.37
85,92
77.
92.33
0,09
GPT-40-2024-08-40 (AlpacaFval)'
92.93
Gemini-1 1.5 5. Pro-002 (Alpacalivali"
67.25
55.63
59.47
88.73
84.21
95.04
0.07
3-5de-3---Snnzee 202-40620 (AlpacaEvval)
Llama-3.1-708- Instract (AlpacaEval)"
65.
16.95
56.84
83.10
$7.89
79.55
0.14
Gemini-
(ArenaHaady)
Gemini-1.55 1.5-Flash-002 (AremaHard))
64.62
45.
53.68
85.92
71.58
87.22
0.09
Starling RM-34B
63.88
36.42
55.79
78.87
64.21
83,91
0.14
GPT-4o-Miale DTT-fo-Miii-2224--77 (AlpacaBval))
63.66
44.85
50.53
83.10
63.26
84.51
0.14
Athene RM SB
62.85
42.56
61.05
83.10
67.37
85.56
0.12
Gemini-1 Flash-022 (AlpacaEvall'
Intern. M2-2n0M222BBBBB0 Reward
62.63
40.47
$5.26
76.06
71.58
87.37
0.11
61 61.66
48
39.47
87.32
77.89
93.23
0.09
M21cmLLMM-7BBBooaa
Euras RM-7B
61.31
35.08
$4.21
57.75
47.37
64.06
0.22
Skywork-Rewand-l Re-.-ewwadddd- Jamn-3.1
60:65
43003
$3.16
77.46
63.16
81.61
0.14
59.32
37.16
44.74
73.24
65.26
83.31
0.14
Llmm--3-0fhoaBBa-RMM--00
50.00
70,42
54.24
71.88
0.20
IntemLLM2 1M2-188 3B-Reward
$8.74
33.52
36.84
45.07
49.17
67.82
0.19
Starting RM-7B-Alpha
58.08
26.79
38.95
56.34
54.74
74.59
0.18
XaiveVerbooity
57.49
27 09
60.00
Skywork Rewanl--Gnma-222222222----- 2-274-2-277
55.80
35.07
25.26
46.48
60.00
75.94
0.14
Table 9: Reward model and LLM judge performance on Math prompt subset of the human prefer- ence dataset. L.M--as---jude. are labeled with system prompt source, and marked with t.
Reward Model
Accuracy
R. R.W. Pearson
Sepurbility
Conf. Agree.
Kendalltau
Spearmanr
Brier Score
Nomotroe-4-34088 --3w----e-----
62.65
56.88
58.95
62.28
51.58
68.42
0.19
Gemini-1.5 1-5---0002 (AremaHand))
59.90
45.67
66.32
44,74
37.89
53.38
0.27
Gemini-11. P.51:o0001 (ArenaHand))
MomoRM-LLama--Bee 88--0.00
56.83
33 33.59
43. 13.16
42.98
36.84
47.82
0.27
Gemini-1.5 1.5-pi--.5PPr--000 (AlpacaFvvll'
56.83
30.75
30.53
0.31
Athene-RM-708
55.81
31,06
67.37
35,96
28.42
44.06
0.32
Ensemble-Judges (ArenaHard))
55.27
36.57
66.32
42.11
37.89
$3.68
0.27
Seewori-Rewad-Laine
Skywork Reward Cemmma-2-27B
54.50
34.00
35.79
38.60
43.16
57.89
0.21
Ilama-3 Hlima-3-Offan 3...f-eeeiias--MM. RM--8B
54.04
30.51
41.58
42.11
34,74
49.77
0.26
GFT-46-2022-08-000 (ArenaHand))
52.74
29 48
$8.95
40.35
34.74
$3.38
0.29
InterLLM2-200 M2-20B-Rewaadd
52.43
29.55
55.79
39.47
36,84
55.94
0.26
Claude-3 8.5-Somett-22444 (AremaHand))
51,73
Ensemble-Juie Judges (AlpacaEvval)'
51.26
16.53
$7.90
31.58
27.31
10
0.33
GPT-4o 20-2022-05--0 (AlpacaEvali"
GPT-4o-Mine 2in-MMini22--01 2024-2022-----1 (ArenaHard)'
50.06
15.15
51.58
30.70
28
45.71
0.30
Minoo-MMnii-224-0--U (AlpocaEvvl))
Interl M2-1.8B- Reward
47.86
2.97
36.32
-3.51
9.47
20.75
0.37
Gemini-115 1.5-Flahh00 (ArenaHard))
47.13
16.99
48.95
18.42
22.11
38.95
0.35
Gcmini-1,5-Flaat-000 (Aala
16.72
5.46
48.95
17.54
14.74
23.16
0.37
Intem1. Rewadd
45.77
.3.02
42.63
9.65
14.74
21.80
0.36
3-sud-----Sunctte 1.5--ommxt-222246633 (AlpacaBvall)
L.ama-3.1-70--1 1.---..-----nsmsssc Instrett (AlpacaEval))
45.33
-4.36
46.84
11.40
6.32
14.59
0.39
Llmma-31-7008-nnsee (ArenaHard)"
Earu-RM-- RM-78
39.81
-19.21
37.90
-7.02
-2.11
-1.65
0.45
Starling RM-14B
39.23
21.35
35.79
-6.14
1.05
0.45
0.42
Staring-RM--78-Alpph. RM-78-Alpha
38.59
-25.59
32.63
-12.28
-5.41
0.44
NaiveNurboiitMookt
6.10
-93.99
52.63
-75.44
-94.74
-99.10
Table 10: Reward model and LLM judge performance on Shorter won subset of the human prefer- ence dataset. LLM-as-aajudge are labeled with system prompt source, and marked with t.
18

---

Published as a conference paper at ICLR 2025
Accumccy
Segumbility
Conf. Agree.
Spearmanr
Ensemble-Jugges (ArenaHard)
68.15
71.49
73.16
91.59
86.32
95.64
0,06
Ensemible-uugges (Alpacabval)"
73.31
74.21
84.21
GPT-4o-2024 08-06 (ArenaHand))
67.22%
71.93
71.05
92.52
84.21
95. 19
0.07
Claude-3 SoSnnet 20240620 (Arenalaad))
MTT-fo-Mimii-02--44
(Arenafaad)
66.29
71.23
89.47
89.72
80.00
92.48
0.08
Athen RM
65. 84.84
72.39
$1.05
90.65
78.95
91.88
0.09
Gemini-1 1,5- 5-Pro-0022 (val)
63.54
71.75
74.21
92.52
85.26
94.74
0.06
GPT-to-2024-08-00 (AlpscaEval)"
65.45
71.06
68.42
88.79
82.11
93.68
0.07
Gemini-1.55 1.5iii--.--PPssh-00000 (AlpacaFvaa))
74.24
Llama-3.1 3.1m--3.1700---. In--rnn (AlpacaEvaal)
64.36
71.92
75.26
88.79
71.58
86.47
0.11
200002 (ArenaHaad))
Athene-RM-an
64.28
68.
78.95
89.72
74.74
88.57
0.10
Starling-RMM RM-348
64.05
67.27
75.79
83.18
71.58
85. 56
0.12
Ilama- Inssssss (ArenaHand)
87.52
Claude 3-5 Sonneon-ee- 20240620 (AlpacaEival)"
63.95
65.29
65.79
87.85
70.53
85.71
0.12
Se-minn-1.5-Fuush---- (ArenaHard)
Skywork Reward- Reward-Lluma- Llama-3. 1-8B
62.83
71.83
73.68
97.20
81.05
92.
0.08
1.5mmni-1 (AremaHandy)
62.46
64.75
66.32
96.668
Saru--MM-78
62.07
56.73
68.95
73.83
57.89
72.03
0.20
NaiveNerbosity Model
61.30
40.25
68.95
53.27
34.74
49.92
0.30
IntemLLM2-TB-Roowe
60.82
61.98
69.47
77.57
60.00
80.30
0.16
GPT-4o 111-4o-Miii2000 Mini-2024-07.
(AlpucaEival)'
60.59
60.25
$7.90
87.85
73.79
88.87
0.10
AmooM-1 Jama--8Ht-xdd
60.11
63.
71.05
90..6
81.05
90.98
0.07
Starling-RAn Beating-RM--B--o RM-7B-Alpha
59.01
54.50
64.21
64.49
49.47
70.83
0.20
Intemm_MM2220--Reeaa
59.00
78.20
0.17
Manna-3-OlscBBBas- BBM--BBBBa--MMMBB
$8.58
$7.04
58.95
71.96
64.21
81.80
0.14
Nematron 4-340B Rewwe- Reward
57.74
50.81
75.26
65.42
57.89
73.98
0.19
Savor--Kewad-GGemmee Gemmma2--78
75.29
90.68
InterLLM2-1 12-1188--500 188-Roadd
55.92
37.43
61.58
42.99
36.84
$5.64
0.27
Table 11: Reward model and LLM judge performance on Similar response subset of the human preference dataset. LL-d-ee---udgee are labeled with system prompt source, and marked with
Reward Model
Accuracy
R. Pearson
Sepurbility
Conf. Agree.
Kendalltau
Spearmanr
Brier Score
Ensemble Judges (ArenaHardd)*
68.17
70.80
71.58
86.24
81.05
94.
0.08
CPT-1o-2024-0S (ArenaHard))
67.78
7161
68.95
86.24
83.16
94.89
0.07
EmsembleJuuges (AlpacaEval))
67.60
70.66
71.58
84.40
76.84
92.93
0.10
-01-44-0024-08-06 (AlpacaHvva))
66.70
63.51
66.32
80.73
76.84
91.73
0.09
2---Somct-22220066 (ArenaHard)
66.42
68.25
70.53
86,24
78.95
93.68
0.08
GPT-4o-Mini-2024 2in--Mii---22----111-
(AremaHar))
66.39
66.39
67.37
81.65
78, 95
92.03
0.09
Gemini-1 1,0---15-.- 5--Pr-0002 (Alpacabval)"
65.37
70.68
74.74
87.16
76.84
91. 88
0.10
IJama-3.1 3.1-008.-00---- Instrac (AlpacaFval)"
64.79
65.74
72
66.32
85.56
0.13
Gemini--1.3 1.5 Pro (ArenaHard))
64.75
69.77
71.58
84.40
76.84
92.93
0.10
Semini--1.5-PP 1.5niii-15-FFaas---000 (AlpacaEval))
64.48
65.98
67.90
79.82
69.17
86.02
0.15
1aimaa--.-70B--mssss (ArensHard))
64.31
63.74
67.90
82.57
70.53
88.
0.12
Clande-33 3-5--onnee--300 2024000 (AlpacuFval)"
64.27
62.80
66.26
779 82
68.42
86.47
0.13
Athenc-RM-SB
Starling-RM334p RM-34B
63.50
60.04
72.63
68.81
65.26
$1.50
0.16
Ocmini-I,5-Flaah--0 1,5--lash-002 (ArenaHard)"
62.97
61.16
66.84
77.98
70.53
82
0.12
Skywork Reward -artaadd--amm3.11m
62.94
Gemini-1.5 1.5-Pro-001 (ArenaHadd)
62.04
64.66
65.79
86.24
70.53
89.47
0.12
Euras Eara--RM-7B
61.78
51.70
71.58
58.72
$2.63
65.86
0.20
107--o-Mimii2022-402
(Alp)caEval)'
61.64
57.42
$9.47
81.65
71.58
87.52
0.11
Naive VerdosiyMMdeel
61.26
40.80
68.42
48.62
43.16
51.73
0.26
R2-77-RRewrd
61.01
60.94
61.96
70.00
83.19
75.79
90.38
0.10
Starling RM.1B-AJpe
59.55
50.50
67.90
53.21
55.79
71.43
0.21
Inteml M22220--Rewndd
59:34
Jlama-3-011scBBaa-RRM..
$9.06
$1.04
55.26
66.06
54.74
69.47
0.20
Nematroe 4-3 4-340B Reward
57.47
44.46
71.05
62.39
50.53
67.07
0.22
Inteml 12
56.17
41.19
38,53
32.63
$0.23
0.28
Slywor--Rwwrd-GGmmaas
55.21
57.61
49.47
73.39
69.47
87.52
0.11
Table 12: Reward model and LLM judge performance on English prompt subset of the human preference dataset. LLM-as-a-judge labeled with system prompt source, and marked with 4.
19

---

Published as a conference paper at ICLR 2025
Accumccy
Segumbility
Conf. Agree.
Kendalluu
Spearmanr
Ensemble-Jugges (AlpacaEval))
69.68
73.76
74.21
94.31
90.53
97.74
0.03
Ensemileehudges (ArenaHard)"
75.81
95.
Claude-3. 3.5. 3-5-Someet- 202400000222220 (ArenaHand)
68, 48
75. 18
75.26
91.87
86.12
96.39
0.05
Athene-RM-708
67.86
GPT-40-2024-08-00 (AlpacaFivali"
67.66
72. 12.18
72.63
98.31
93.68
98.63
0.03
GPT-4o-2024-08 (ArenaHard))
67.63
71.24
73.16
91.87
82.11
94.74
0.07
Gemini-1.3 P.5--ro-002 (AlpacaEval)
67.01
73.72
80.00
94.31
$3.42
97.14
0.05
Gemini-1.5- 1.5pii--.-P---0022 (ArenaHand))
66.93
74.39
75.26
90.24
82.11
94.29
0,07
(AlpacaFval))
67.72
11.00
GPT-4o-Minh 2020-2022--0-118
(ArenaHard))
66.55
71.23
72.63
90.24
82.11
94.44
0.07
Athene-RM-88
1.lama-31 1.ama-3.1-70B. 1--00----mae Instract (AlpacaEvally
65.87
65.70
68.95
83.74
75.79
90.53
0.09
Gemini-1.55 1.5-Flash-0000 (AlpacaEvall"
65.75
70.61
67.90
86,99
87.37
96.84
0,05
Alama-3.1-7088 -nnts-snnnttt (AremaHard)"
64.25
68.81
65.26
82,11
80.00
93.38
0.09
GPT-4o-Mini 2ilii-2022-0--11 (AlpocuEval))
64.17
62.56
54.74
78.05
85,16
94.44
0.05
Interl M2-7B-Reward
65.
0.16
Gemini-J 1.5. 001 (Arenafaad))
63.24
70.19
70.53
$7.80
$0.00
94
0.08
Intert M22220--Reward
63.10
63
64.21
Gemini-1.55 1.5.-lach-0022 (ArenaHard))
63.05
68.96
71.05
86.18
77.89
93.38
0.08
BanasKMM-78
62.32
16.17
61.05
67.
66.32
75.49
06
Starling-MMa
Skywork Reward (eeaadd-Llama33..
L.1ma-3.1
61.66
64.18
70.53
75.61
73.58
87.52
0.11
Vemotme--433404 -ewa---en
61:57
67.
72.63
83.74
76.84
90.53
0.10
AmoRM-Lllmma-SBB
60.11
59.83)
$8.95
73.68
90.53
0.12
Jlama-3-OfhhaBBlaRRM---3
59.20
48.58
55.79
52.85
69:17
0.19
InteraLM2-1.8. 1.8B-Reward
$8.55
44.78
$5.26
43.90
41.05
$6.24
0.24
Skywork Reward Gemma-2-277 2-22BB
58.40
58.79
61.05
83.74
83.16
95.19
0.05
Starling-KM RM-78-Alpha
0.22
Naive MabeeNabbsitMMood
$7.98
21.46
64.21
30.89
21.05
27.52
0.36
Table 13: Reward model and LLM judge performance on Non english prompt subset of the human preference dataset. au--ude are labeled with system prompt source, and marked with
Reward Model
R. R.W. Pearson
Sepumbility
Conf. Agree.
Kendaaltau
Spearmanr
Brier Score
EnsembleJJudgee (AlpacaEvval))
67.91
52.67
54.21
93.33
80.00
94.
0.07
Claude-3 3-5 3-5-S-nnet-eeeee 10200 (Arenallad))
67.03
50.91
48.42
90.00
78.95
93.38
0.08
Athene-Ree Athene-RM-70BB
66.39
Gemini-1 1.5- Pro 002 (AlpacaEvall"
66.27
49.5
58.42
93.33
$2.11
93.38
0.08
Ensemble-Judges (AncnaHard))
92.33
GPT-4-202440X--0 (ArenaHard))
65.37
49 18
52.10
90.00
76.84
92. 18
0.08
Mif-fo-Miii-222-00---1- (Arenataadd)
Gemini-1 Cominii-1.5-HHe 5--Fashh-022 (AlpacaEvall)
65.
40.01
46.32
86.67
71.58
89. 17
0.09
Soudee---5-omenn S--SSneet-2224400 20240620
(100400000Alpaaaaaai (AlpacaFoval)
64.89
47.98
43.16
87.52
64.62
42.76
48.42
56.67
65.26
83.91
0.12
Athene-RM-SS RM-SB
64.45
42.41
60.00
86.67
81.05
91.59
0.07
Gemini-1
(ArcnaHard))
IntemL Reward
63.87
44.35
41.05
53.33
70.53
89.17
0.11
GPT-1o-2024-08-42 (AlpscaEval))
63.53
43.47
51.58
90.00
83.16
94.89
0,06
Llama-3.1 1ite--..-70---sttteee (ArenaHard)"
32.
60.00
65
0.14
1.ama-3.1--00 3.1---00----nsreee (AlpacaEval)"
63.03
36,
47.90
68.33
67.37
86.17
0.13
Starling- RM-34B
Gemini-1.5. 1emmini-15--Phach-000 (ArenaHard)"
62.48
43
46.32
83.13
73.68
89.02
0.09
Gemini- 1.5iin--11.53000 Pro 001 (ArenaHard))
62.09
36. 12
05
75.00
71.58
89.77
0.09
Mi----MMii-222400--11 (Alp)
61.43
38.81
23.68
$5.00
63.16
83.01
0.14
Euras. Euns-RMM-7B
61.18
39.05
44.21
70,00
65.26
$1.05
0.14
60.08
Skywodd-Reward-GGr Rww.Rd-w----- Gemma-2-277
59.16
22.53
26.84
75.00
86.32
96.09
0.06
Nemotroo 4 3408 Reward
58.07
28.62
32.63
45.00
$2.63
72.33
0.18
IJama- 3-OffsetHins-
Skyword-Rewaer Reee---ewa---la--......
1-88
57.23
38.20
37.37
53.33
64.21
81.20
0.13
ArmoRM Llama3--a SB-v0.1
56.64
18.09
26.84
28.33
46.32
$9.40
0.21
Naive Verhosity Model
56.55
14.74
0.36
Starling-RM RM-7B-Alpha
54.29
7.14
28.42
18.33
35.79
47.37
Table 14: Reward model and LLM judge performance on Chinese prompt subset of the human preference dataset. LLM-as-a-judge are labeled with system prompt source, and marked with 4.
20

---

Published as conference paper at ICLR 2025
Accumccy
Segumbility
Conf. Agree.
Kendalluu
Spearmanr
Ensemble-Jugges (ArenaHard)
70.37
50.61
53.16
92.86
77.89
92.63
0.09
EnsemibleJudges (Alpacabxal))
51.75
57.90
Claude- 3.5 3.5-Sonnet-2 3--Someet202240333 20240620 (ArenaHaad))
68.6.
44.71
50.53
85.71
70.53
87.97
0.09
GPT-4o-2024 (AlpacalEvali
GPPTfo-2022-09--4 (ArenaHand)
68.54
13.94
17.37
89.29
70.53
89.02
0.10
Athene RM-70B
68.49
48.
58.42
77.89
90.68
0.09
Gemini-11 1.5-Pro-002 (ArenaHard)
67.23
49.52
53.68
$7.50
73,68
89.32
0.10
Semini--.55-Pr-44 (AlpacaBval)"
66.20
50.01
58.42
92.86
78, 95.95
93.38
0,07
(AlpacaFoal))
1lama-3.1 1700 Instract (AlpacaEval)"
65.65
38.73
47.90
92.86
66.32
$5.56
0.12
GPT-4- MTT-oo-Mii-2022-000 (ArenaHaadd)
Gemini-1 1,5 1.5-Fash-0022 (AlpacaEvall)
65.
42.35
50.00
94.64
75.79
91.73
0.09
Athene RM-SB
64.87
41.89
55.79
91,07
71.58
86.62
0.10
72.63
87.07
GPT-4o Mini Sanii2024-011 01-4-07-188-- (AlpacalEval)"
63.82
31.28
23.66
71.43
82.11
93.83
0.08
In1-2ott (ArenaHant)
Gemini-J 1.5.Flash-002 (Arenallard)
63.26
31.97
42.63
76.79
67.37
85.56
0.12
Kuras-MM -M---
62.84
56.84
0.16
Gemini- Pr1-001 (ArenaHard)
62.08
43.28
46.32
78.57
70.53
88.
0.11
Skywork- Rewand--L.amm--- 1-88
61.17
23.32
41 58
73.21
65.26
84.51
0.13
M
Starling RM-34B
60.98
36.02
36.32
73.21
63.16
80.00
0.13
Interal M2-228-
60:43
26.87
19.47
30.36
60.00
78.50
0.16
60.33
38.52
35.26
74.74
90.23
0.09
Starling-MM-78 KM-7B-Alpha
59:41
31.55
38.95
66.77
0.19
Llama-3-Oflscte -MM--BBBB---MBBBB RM-SB
$9.04
25.82
30.53
50,00
48.42
68.27
0.19
Naive Verhosity
59.04
10.26
34.21
33.93
29.47
38.95
0.29
Intersl M2-1 Reward
57.65
26.88
17.86
Skywork Reward- Nleaadd-Gmmnnn 2-228-
56.26
29.71
23.68
50.00
64.21
82.86
0.14
Table 15: Reward model and LLM judge performance Russian prompt subset of the human preference dataset. LL-d-ee---udgee are labeled with system prompt source, and marked with
Reward Model
Accuracy
R. Pearson
Sepurbility
Conf. Agree.
Kendalltau
Spearmanr
Brier Score
Ensemble- Judges (ArenaHardd)*
75.16
38.73
38.42
84.62
73.68
88.42
0.10
Claude-3 3-5 Sonnet- 1022222 (Arenallad))
72.49
30.32
23.16
66.67
65.26
81.50
0.12
GPT-4o-2024-0S--8 (ArcnaHard))
71.03
31.32
24.74
84.62
72.63
85.
0.10
Geemini-1.5-Pro0000 P.5--ro-002 (ArenaHand)
70.64
29.57
27
76,92
72.63
87.22
0.11
GPT-4-2202-08--0 (AlpacaEval)"
69.71
21.47
21.05
74.36
72.63
88.27
0.10
Ensemhle-Judges (Alpacalvaal)"
68.88
15.78
27.37
71.79
60.00
78.05
0.14
Athene-RM-70B
Nemotron 3-340B-BB Reward
66.85
27.91
26.84
71.79
62.11
83.16
0.12
Llama-3,1 3,1-00- Instract (Alla
66.86
27.69
25.79
66.67
51.58
69.17
0.17
1emmnni-15-FFaash-00 (AlpacaEval))
66.86
18.3
24.21
61.54
54.74
73.38
0.15
Gemini-1.5. Semnini-1.5-Pr-0020 Poo (Alpacalval)"
66.29
8.72
33.68
69.23
69.47
84.81
0.13
GPT-44-Mimi-2024000 Mww.-iiiii22......1 (ArenaHard))
Athene-RM-S8t
65.43
3.68
37.37
76.92
67.37
83.31
0.12
Gemini-1 1.5.aa11..la (ArenaHardy"
65.32
19.95
15.79
43.59
$7.89
75.64
0.16
Llama 3.1mm-3..17000 3.1-70B-IIIttee Instract (ArenaHard))
64.66
21.95
17.37
48.72
$2.63
68.42
0.16
Claude-3 1aad-----Souut-211 10620 (Alp
63.69
11.97
7.37
20.51
46.32
61.65
0.20
Starling RM-34B
Gemini-1. 1.5--r-------0 (ArenaHaad))
63.33
16.68
15.26
48.72
61.05
82.26
0.14
Euras RM-BB
62.57
14.76
8.95
41.03
44.21
56.54
0.22
Intert..M1 Mieeml.MM-7B-Rewnnt
78:05
0.16
M0T-44--ini-2024-077..
(AlpacaEvval)
62.29
14.84
10.00
33.33
48.42
66.17
0.18
MMkomLMM2228-RRowddd
AmaRM--Lamaa Llamm5-88--0.
60.86
-8.08
19.17
46.15
57.89
71.73
0.16
Skywork Reward Reward-LLama- 3. 3.88BB
59.71
.401
20.00
53.85
57.89
72.03
0.16
Naive VeiveserhosityMMe
56.66
Maama-3-O1s:BBas-BB
56.57
-4.02
13.68
30.77
46.32
$6.69
0.21
Starling RM-1B AM-1BB-Apha
56.29
6.70
7.89
23.08
34.74
47.67
0.24
Interl M2-1.88 1.8-- Reward
55.14
13.77
32.63
40.75
0.24
Skywod&-Rowadd Row.rd--Gnmma---22222
54.57
-11.99
23.08
45.26
60.45
0.19
Table 16: Reward model and LLM judge performance on German prompt subset of the human preference dataset. LLM-as-a-judge labeled with system prompt source, and marked with 4.
21

---

Published as conference paper at ICLR 2025
Accumccy
Segumbility
Agree.
Spearmanr
Athene-RM-70B
71.10
16.16
31.37
84.21
67.37
83.76
0.14
Ensemble-Jun Judges (AlpacaEval))
Skywork-Rewand. Reward-1
Jamm--3 1ewaadd-llam-331..
1-8
68.81
40.32
22.1
68.42
58.95
78.
0.14
Ensemble-Juie JudgesseeJussssseeee (AzenaHardd)
68.45
33.85
25.79
65.79
61.05
78.50
0.14
Demini-1.5-PPa 1.5-Pro-002 (AlpscaEval))
68.06
28.63
28.42
50.00
66.32
84.36
0.12
Claude-3-5-Somnee 205---onnt-2222244 20240020 (AlpacaBval)"
67.59
27.29
12.11
36.84
$7.89
78.95
0.15
1lama-3 Alama-3.1778B-Mmn 11-708-Insmss (AlpacaEval)"
78.20
GPT-4o-2045 08 06 (AlpacaEivali
66.97
34.79
27.37
44.74
66.32
86.32
0.13
GPT-4o-2024 (Arenalaar))
Inteml M2-200----e- Rewarl
66.51
36.27
20.00
18.42
55.79
72.33
0.18
Gemini- 1.5.:i-1....3
(ArenaHard))
66.36
29.17
21.05
73.68
61.05
79.85
0.14
Athene-RM.
78.65
GPT-4o Mini Mini-2024-00 07-18 (ArenaHand))
65.14
29.31
25.79
73.68
74.74
89.92
0.12
Gomini-1
(AlpacaEvall"
MMM-44-Mimi-2224--0-- (AlpacaEval)'
64.68
14.42
18.42
31.58
60.00
79.70
0.14
Clande-3-5- Somude-----oomee--22 2.5--oonett-22244000 20140020
(ArenaHand))
64.68
27.59
65.26
86.02
Gemini- 1.5 F.5.Flach-002 (ArenaHard)
63.68
20.76
24.21
63.79
64.21
80.30
0.14
Interal Inteml.M22-B-H44 M22-7--eeard
63.
30.05
9.47
-26.32
49,47
68.42
0.20
3lama-311-70B-nnstmme (ArcnaHar))
63.13
10.68
17.89
73.68
57.89
78.90
0.15
Llama---OffstBiii- 1rDDffsttii--R---5 RM-88
62.39
28
16.52
63.16
25.26
36, 511
0.26
PamoRM-Llamm---88 -8-x0.1
62.39
29.54
23.16
60.53
43.16
58.65
0,20
Geminni- 1.5. 1...ro-0001 (ArenaHand)
62.24
19.36
13.16
$7.89
60.00
78.95
0.13
Furas-MM Bunu--M-778
61.47
30.57
15.79
44.74
50.53
71.43
0.17
Nemetmon-44 Bat.rmee--34----eeeeee -40B-Reward
61.47
17.85
26.84
44.21
52.63
0.23
Barling-KM-34p
60.00
14.21
55.29
70.98
0.17
InteraLM2-1.88 Reward
$7.34
19.72
6.32
-7,89
38.95
$1.59
0.21
Naive VerhosityMoodee
56.88
9.00
8.42
-28.95
15.79
20.90
0.25
Boatlng--M-78---1
44.21
57.44
Skywosk Set.aan--emmma2225
55.05
8.51
20.53
55.26
42.11
56.54
0.20
Table 17: Reward model and LLM judge performance on Korean prompt subset of the human preference dataset. aL-duee- are labeled with system prompt source, and marked with
Reward Model
Accuracy
R. Pearson
Sepurbility
Conf. Agree.
Kendalltau
Spearmanr
Brier Score
Claude-3- 1.uuee335..50 202440200240000 (AlpacaEEval)"
73.36
37.78
6.32
58.33
69.47
87.22
0.11
Athene-RM-38
71. 71.89
39.72
14:21
54.17
67.37
87.07
0.10
Ensemble-Judge (AlpacaEvall)"
36.61
Llama-3. 3-
3.1-700-Innttese Instract (AlpacaEval)"
70.05
37.95
6.32
62.50
62.11
$1.50
0.11
Bluud-3-5-Soumtt-20
(ArenaHaad))
Athene-RM-ND
68.20
33.11
18.42
50.00
72.63
87.82
0.13
WyT-fo-Minii-22--
(AremaHard))
68.20
41.02
8.95
58.33
62.11
80.75
0.13
Gemini- 1.5.Flach-002 (Alpacalvall)
67.44
35.21
14.21
66.67
62.11
81.20
0.13
(Alpacaaaal)
67.28
31.60
0.53
54.17
65.26
82.11
0.12
Scmini--15-Prooo00 (AlpacaEval)"
66.95
33.95
14.74
$4,17
64.21
83.46
0.12
Skywodz-Rewad- Rew.nd-LLlama-....
L.amma-3.11888
66.82
28.61
9.47
83.33
64.21
77.59
0.14
66.36
19.15
16.32
25,00
53.68
70.53
0.16
BieembleJJudges.
(Arenalaard)
65.79
31.49
16.84
62.50
71.58
87.37
0.11
Starling-R Harting-RM-348
64.98
27.05
16.32
$4,17
61.05
79.70
0.15
GPT-4o-2024-00 (AlpacaEivali
GPT-1o-2021-0S- (ArenaHadd)
64.10
28.43
15.26
58.33
69.17
86.47
0.12
(ArenaHard)"
Nemotroo-4-33 4-4--B---------- Reward
63.59
28.08
37.50
67.37
83.46
0.13
Skywork Reward-Genma.
Gemma--2270
63.13
12.65
6.32
50,00
49.47
64.21
0.18
MMcmmLMM-208-B8
Gemini-1153 1.5ini--..-PPahhhe P.5h-laah-0002 (ArenaHard))
63.03
33.38
7.89
$4.17
62.11
82.26
0.11
Gemini-- 1.5. P.o.ProoOO (ArenaHaadd)
62.91
22.44
15.79
62.50
60.00
79.85
0.14
KaiveVerhositMMode
4.17
RM--8
61.29
20.76
3.68
20.83
47.37
63.61
0.19
AxmoRM-Llams--e -80-000.
60.37
12.93
9.47
75.00
22.11
33.08
0.24
$3.53
Gemini- 1.5 Pro 001 (ArenaHaad))
59.51
15.30
3.16
66.67
51.58
70.38
0.15
Inteml M2. 188. Reward
58.99
15.75
-2088
36.84
$3.98
0.22
Staling-RM-- AMphB-Alpha
58.06
23.72
34.17
10.53
14.14
0.32
Table 18: Reward model and LLM judge performance on Japanese prompt subset of the human preference dataset. LLM-as-a-judge labeled with system prompt source, and marked with 4.
22

---

Published as conference paper at ICLR 2025
Accumccy
Segumbility
Conf. Agree.
Kendalluu
Spearmanr
Ensemble-Julges (AlpacaEval))
72.11
31.81
5.79
36.84
20.00
30.53
0.28
GPT-4-2024-08-00 (Alpacahval)
70.53
23.71
100.00
35.79
0.22
HPPT44-2024-04 (ArenaHand))
70.29
24.79
4.21
89.47
43.16
59.55
0.21
Atlene-RM-708
17:37
Clande-3-5.. 3-5----nntn--000- 202400000 (AlpacaFval)"
68.42
28.53
1.58
100.00
21.00
33.81
0.26
Llama-3. 3.1-70B Instract (ArenaHard)
67.93
29.52
78.95
25.26
32.63
0.28
Skywork- Mkymori--Rewadd-1 Jamm--3
35.79
Llama-3.1 Inn-srot
Instract (AlpacaEval))
67.89
27.03
2.63
100.90
32.6.
49.77
0.22
Naive SeiveterbssityMMoe
Gemini-1 1.5. Plach-002 (AlpacaEvall"
67.37
29.
4.74
68.42
25.26
37.44
0.26
67.37
23
2.63
78.95
23,
34.89
0.24
Stadinng
Ennemble-Juuges (ArcnaHard))
66.47
20.45
12.63
47,37
28.42
40.15
0.24
Gemini-11 1.5-Pro-0022 (AlpacaFvvll"
66.32
24.21
38.05
0.25
Boaling-RM-7B-Appph.
65.79
32.43
68-42
6.32
0.30
IntemL.M2 20222200-330 -0wae---rr--
65.26
24.19
1.05
100.00
21.05
32.78
0.25
GPT-fooMMini-022-00-- (AlpacaBval)"
2aaal------onn--2000 40200020 (ArenaHard)
64.74
21.07
8.95
5.26
24.21
36.
0.26
Athene-RMC
64.21
23.88
9.47
68.42
27.37
40.45
0.26
1.5-o-0-1--P (AremHHad))
63.84
25
36.84
25.26
37.74
0.23
Min----Mii--222--0----0 (ArenaHard)'
63.83
11.48
7.89
78.95
31.18
46.47
0.24
Gemini-1.5 P.50.r0000 (AremaHand))
Banis-MM-78
63.16
14.36
0.53
89,47
1.05
2.86
0.33
Alama-3-OffsccBiass RM-BBB
61.05
20.44
1.58
100.00
42.11
53.68
0.21
1eanini-1.5-Pashh-- (Ard)aHard)
60.75
16.42
12.63
17.14
Skywork-Rewaad Reward-Genma
60.00
30.32
0.53
89.47
22.11
31.58
0.27
AmoRM. www.-laaama--.....
Intert -8--880
59.47
15.07
100.00
33.68
47.07
0.23
59.47
8,42
10.53
Nomotroe-4-33 3-0---0---00 Reward
58.42
10.01
6.32
89.47
20.00
29.17
0.28
Table 19: Reward model and LLM judge performance Spanish prompt subset of the human preference dataset. LL-ddee---udgee are labeled with system prompt source, and marked with
Reward Model
R.W. Pearson
Sepumbility
Conf. Agree.
Kendaaltau
Spearmanr
Brier Score
Gemini- 1.5. Pro (ArenaHand))
69.57
14.77
14.74
$4.17
63.16
82.41
0.14
GPT-to-Minn MTT-io-Mini-2222--7-.. 0224-07-18 (AremaHuur))
68.45
25.12
4.21
75.00
54.74
73.08
0.17
EmsembleJJulge (ArenaHard))
68.24
21.05
17.37
66.67
62.11
80.90
0.13
Ensemble-Jugges (Alpycabvall)
67.74
27.12
4.21
79.17
46.32
65.71
0.19
Gemini-1.5-Pro-00 (AlpscaEval)
67.38
26.42
79.17
47.37
65.26
0.18
Athine-RM-SSt
67.38
26.84
18.95
45.83
45.26
64.81
0.17
M2- M22-BB-Roward
66.31
20.42
11.05
45,83
43.16
62.41
0.19
Claude S-nnoot-2222000 (ArenaHad))
66.31
24.02
5.79
45.83
$5.79
73.53
0.15
Jahene-RM-P35
65.78
22.45
17.89
54.17
45.26
65.86
0.18
65.24
26.25
13.16
29.17
58.95
79.55
0.15
21
5.79
33.68
0.23
Maama-3-OfbodBia-RRR...
64.71
13.13
2.11
79.17
27.37
41.80
0.23
GPT-40-0024-05--6 (AlpwaEival)
Llama-3.1-70B- Instract (AlpacaEvall)
64.17
20.26
3.68
70.83
43.16
61.65
0.19
Claudo-3-5-Somee
(Alpacabval)
36.84
Starling-MM- RM-7B-Alp.
63.10
22.53
9.17
$4.17
34.74
47.67
0.20
GPT-4o-Mini-2024-000
(AlpacaBval)
62.57
30.14
1.05
70,83
25,26
38.50
0.24
GPT--0-2022-00--0 (ArenaHand)"
15.80
49.47
0.18
Gemini- 1.5iini-1.5-Pashh-hh F.5 (ArenaHard))
62.37
22.71
13.16
62,50
36,84
$5.19
0.21
62.01
62.03
11.19
18.95
29.17
49.17
66.92
0.19
Seminni-1.5-Faaslee (AlpacaEval))
62.01
20.24
2.11
79.17
37.89
$4.59
0.20
1lama-3.11-20 10mm--31-308-Insmmas (ArenaHand))
69.17
Gemini-11 1.5 P.Pro (ArenaHand)
61.11
12.74
5.79
58.33
47.37
$9.55
0.17
60.96
10.53
40.00
StarlingRM--30 RM-3nBBBMB3BBBB
59.36
11.68
0.53
79.17
38.95
$1.44
0.22
InterLLM2. L.SB
1.SB-Reward Reward
58.82
21.97
4.21
12.50
36.84
46.47
0.21
Skyanri-RRowad--Cre www.RReewwdreenee Geno----mmma--22
57.75
3.40
48.42
63.46
0.20
NaiveNurboiityMokk
54.01
9.52
10.00
62.50
-2.11
-3.16
Table 20: Reward model and LLM judge performance on French prompt subset the human pref- erence dataset. LLM--ss--judde are labeled with system prompt source, and marked with t.
23

---

Published as a conference paper at ICLR 2025
Model
Accumccy
Sepumbility
Conf. Agree.
Spearmanr
GPT-4o-Mini-222--00--E (AlpacaEval)
71.84
31.95
2.11
100.00
49:47
67.82
0.18
Claude-3.5-
-5-Somet--2144000 (Alpecatsal)"
27.08
100.00
0.22
68.93
25.47
1.05
100.00
19.47
68. 12
0.18
Claude-355 3-5-------tt--22--555 (ArenaHard))
100.00
JnsembbleJnulges (Aal))
67.96
17.35
7.37
100.00
57.89
79:25
0.22
Ensemble Ensemble-Judges (ArenaHaard))
67.02
20.72
10.53
100.00
62.11
76.39
0.17
GPT-1o-2024-0S-4 (AlpwcaEval)
66.99
16.25
100.90
50.53
69:47
0.18
Skywork Gemmma2-22m5 2-27B
66.02
21.16
4.74
100,00
58.95
77.29
0.20
Athene-RM-8t
66-122
Earu-RMM7B
65.05
26.36
3.16
78.95
30.53
39.55
0.21
Athene RM-70MM-0000
65.05
10.12
7.89
89.47
50.53
72.33
0.18
4PT-4o--Min-2222----18 (Arenalaadd)
Gemini-1.5-Pro-002 (AlpacaEval)"
64.08
14.69
3.16
100.00
$4.74
72.03
0.18
Vemmin-1.5-Plech--0 (AlpacaEvall)
64.08
100.00
Llmma-3-01tsdBaa-RMMM
64.08
28.73
11.05
100.90
27.37
40.15
0.21
IntemLLM2220B-Reeee 208 Reward
64.08
8.68
2.63
100.00
53.68
75.49
0.19
Gemini-1.5-Pro-00/ (ArenaHard))
GPT-40-0024-0--36 (ArcnaHard)
63.27
18.
5.26
89,47
56,84
72.63
0.16
Marling-MM-33
63.11
14.73
2.63
89.47
42.11
58.20
0.18
1a-mm---.-70--nssseee (AlpacaEvall
62.14
19.12
100.00
63.16
78.05
0.15
Skywork Remaad--aaa---- 1-8B
62.14
25.10
6.32
100.00
36.84
$4.59
0.21
Llama-3.1-700 Instrrtttt (ArcoaHard))
AmmRM--Llame MaanRM-LLlamm--5e s0-v0.1
60.19
19:66
2.11
100.90
18.95
32.18
0.25
Inteml. M2. 1.B--8BB--8 Reward
59.22
11.84
2.11
$7.89
27.37
33.38
0.24
Starling-MM-1 -M-7-AAlpa
NuiveNabooittMood
58.25
11.49
2.63
100.00
20.00
32.78
0.22
Nemotmo- 4. 3408- Reward
58.25
7.87
3.16
100.00
40.00
55.94
0.20
1.5. (Arenaliad))
Gemini-1 1,5- F---lash-002 (Aremaffrd))
51.96
-0.90
1.05
78.95
37.89
62. 61
0.19
Table 21: Reward model and LLM judge performance on Portuguese prompt subset of the human preference dataset. a--duue- are labeled with system prompt source, and marked with
Reward Model
Accuracy
R. Pearson
Sepuraility
Conf. Agree.
Kendalltau
Spearmanr
Brier Score
Gemini-1.3 P.5.Pro 002 (AlpacaEivall"
81.40
51.04
3.16
100.00
50.53
74.14
0.17
Ensemble-Judges (AlpacaEvval)'
75.58
44.04
6.84
100.00
45.26
66.47
0.18
Gemini-1. Semmini-1.5-Prooo 1.5-Pro-002 (AremaHard))
74.42
40.23
$7.89
52.63
70.53
0.18
Athene Athene-RM-P5 70B
74.42
42.65
4.74
100.00
43.16
61.05
0.20
Souud-3---Somnceee
(ArenaHaad)
Athene-RM4
73.25
43.29
8.42
78.95
43.16
60.45
0.19
Ensemble-Judge (ArenaHard))*
71.25
44.59
1.58
89.17
36.81
51.88
0.20
Claude 3-5 Sonnet- 20240220 (AlpocaEvval)
69.77
28.35
5.79
100.00
40.00
$2.03
0.22
Gemini-1 1.5-ro000 (ArenaHard)
69.23
35.18
2.63
100.00
40.00
55.94
0.19
GPT-44-2024-05-66 (AlpacaEval)"
68.60)
39.33
5.79
100.00
40.00
53.53
0.19
Eurus- RM-7B
67.44
25.34
2.63
89.47
-2.11
-1.95
0.29
Skywork:
66.28
27.43
100.00
37.89
47.82
0.21
ArmoRM-LlamnS--8
28
100.00
42.11
57.
0.19
Gemini-1. 1omiini-1.5-Faahhe Flasssh-002 (AlpacaEvaal)
66.28
33.17
1.05
89.47
30,53
44.81
0.22
GPT-4o-2024-00 (ArenaHaad))
GPT-lo-Mini-200 2024-0002--0-11 (AremaHard)
64.71
31.59
1.05
100.00
34.74
$5.64
0.20
Llama-3.1
(ArcnaHard)
64.63
InterL Reward
63.95
26.87
3.16
36.84
12.63
15.49
0.25
M2.erl..22.2 M2.20B2BBBBBBB Reward
63.95
19.03
0.00
100.00
29
46.32
0.20
1.5-aaah000 (ArenaHard))
Starling-MM. RM-14B
62.79
13.29
1.58
100.00
10.53
10.23
0.28
Soywork-Reand--Genee 10emmma-2-278
61.63
19.87
0.00
100.00
41.05
56.84
0.21
1lama-3.1 100..3..1-00B-Insmaaz (AlpacaEval)"
Ncmotooe-4-3408-Be Rea0dd
60.47
19.10
13.16
5.26
53.68
75.34
0.18
IntemLMM-11.8BBBBB Reward
59.30
16.29
0.53
89.47
2.11
0.27
GPT-4e--Mtn--D24-----88 (Alp)
58.14
14.03
100.00
24.21
33.98
0.23
Llama-3 OffseeBaas RM-SD
58.14
2.76
1.05
100.00
45.26
61.95
0.20
Starling-RMM RM-7h---MMM7BBphhp
56.98
12.63
-2.86
0.30
NaiveNubooitgMrost
50.00
-0.20
100.00
-7.37
-13.68
Table 22: Reward model and LLM judge performance on Italian prompt subset the human pref- erence dataset. ALLM-as--judde labeled with system prompt source, and marked with t.
24

---

Published as a conference paper at ICLR 2025
A.2.1 SCORE DISTRIBUTION STATISTICS OF HUMAN PREFERENCE METRICS
subset
mean
std
min
25%
50%
75%
max
overall 0.6341
0.0337
0.5662
0.6100
0.6301
0.6609
0.6859
hard prompt
0.6351
0.0353
0.5621
0.6115
0.6414
0.6630
0.6946
easy-promp
0.6375
0.0412
0.5554
0.6037
0.6410
0.6698
0.7015
if prommt 0.6306
0.0369
0.5571
0.6110
0.6304
0.6609
0.6977
code_ prompt
0.6336
0.0316
0.5641
0.6139
0.6418
0.6557
0,6806
math-prompp
0.6449
0.0483
0.5580
0.6131
0.6388
0.6858
0.7358
similar response
0.6287
0.0342
0.5592
0.6059
0.6395
0.6545
0.6815
Table 23: Human Preference V1 Accuracy Metric Statistics
subset
mean
std
min
25%
50%
75%
max
overall
0.7135
0.1133
0.3203
0.6803
0.7477
0.7842
0.8263
hard-prompt
0,6623
0.0842
0.4218
0,6303
0.6890
0.7138
0.7637
easy-pu -oy-ppr
0,5070
0.1438
0.0761
0.4327
0.5342
0.6105
0.7266
if prompt
0.6355
0.1040
0.3583
0.5848
0.6647
0.7011
0.7646
is ss_c
0.5871
0.0857
0.3950
0.5392
0.5971
0.6331
0.7311
math prompt
0.5381
0.0876
0.3010
0.4862
0.5668
0.6085
0.6540
similar response
0.6609
0.0951
0.3456
0.6155
0.6755
0.7270
0.7682
Table 24: Human Preference V1 Row-wise Pearson Metric Statistics
subset
mean
std
min
25%
50%
75%
max
overall
0.8244
0.1540
0.3643
0.7571
0.8786
0,9357
0.9714
hard_promp
0.7978
0.0998
0.5000
0.7786
0.8286
0.8643
0.9071
easy-prompt
0.6071
0.2054
0.0643
0.4500
0.6571
0.7571
0.8500
if prompt
0.7759
0.1362
0.4571
0.6929
0.8357
0.8714
0.9214
is code
0.7355
0.0993
0.5143
0.6929
0.7500
0.8071
0.8571
math prompt
0.6527
0.1360
0.3000
0.6143
0.6929
0.7571
0.8071
similar similoe_response
0.7798
0.1296
0.3500
0.7429
0.7929
0.8643
0.9214
Table 25: Human Preference V1 Confidence Agreement Metric Statistics
subset
mean
std
min
25%
50%
75%
max
overall
82.1779
3.9903
73.1580
80.0000
82.6320
84.2110
91.5790
hard prompt
73.3031
5.7412
51.0530
71.5790
73.6840
76.8420
81.5790
easy prompt
55.7350
8.2493
34.7370
50.5260
55.2630
61.5790
67.8950
if-prompt
68.8929
7.6624
45.7890
67.3680
69.4740
74.7370
80.0000
is_code
66.8239
7.0225
39.4740
65.2630
68,4210
70.0000
76,8420
math -prmmpt
61.1070
8.9433
27.8950
58,4210
62.6320
66.3160
1050
similar response
76.5153
4.8143
64.7370
74.2110
77.3680
78.9470
83.6840
Table 26: Human Preference V1 Separability Metric Statistics
subset
mean
std
min
25%
50%
75%
max
overall
0.8432
0.1473
0.3353
0.8120
0.8872
0.9444
0.9699
hard_prompt
0.8637
0.0911
0.5955
0.8541
0.8812
0.9218
0.9474
easy-prompt
0.7673
0.2210
0.0737
0.6421
0.8451
0.9128
0.9624
if prompt
0.8709
0.1176
0.5504
0.8662
0.9113
0.9429
0.9699
is cs.eee
0.8405
0.0873
0.6015
0.8331
0.8752
0.8902
0.9429
math math_promp pprompt
0.8096
0.1062
0.3895
0.8075
0.8316
0.8737
0.9203
similar response
0.8299
0.1208
0.4195
0.8015
0.8586
0.9098
0.9489
Table 27: Human Preference Spearman Metric Statistics
25

---

Published as a conference paper at ICLR 2025
subset
mean
std
min
25%
50%
75%
max
overall
0.7172
0.1593
0.2947
0.6737
0.7579
0.8421
0.9053
hrom_prompt
0.7227
0.1083
0.4211
0.6737
0.7474
0.7895
0.8421
easy-promp
0.6203
0.2009
0.0737
0.4737
0.6737
0.7474
0.8632
prompt
0.7397
0.1338
0.3895
0.7158
0.7895
0.8316
0.8737
is code
0.6897
0.0939
0.4632
0.6737
0.7053
0.7579
0.8211
math prompt
0.6570
0.1080
0.3053
0.6421
0.6737
0.7158
0.8000
similar_ respons
0.6860
0.1345
0.3053
0.6211
0.7053
0.7789
0.8421
Table 28: Human Preference VI Kendalltau Metric Statistics
subset
mean
std
min
25%
50%
75%
max
overall
0.1276
0.0750
0.0454
0.0700
0.1055
0.1532
0.3344
hard_prompt
0.1169
0.0481
0.0680
0.0857
0.1047
0.1353
0.2507
easy-prompt
0.1549
0.0906
0.0562
0.0946
0.1201
0.2273
0.4132
prommt
0,1062
0.0591
0.0442
0.0685
0.0851
0.1148
0.2579
is_code
0.1289
0.0447
0.0721
0.0986
0.1176
0.1396
0.2421
math prompt
0.1408
0.0507
0.0857
0.1064
0.1285
0.1473
0.3191
similar response
0.1378
0.0628
0.0705
0.0934
0.1259
0.1577
0.3299
Table 29: Human Preference VI Brier Metric Statisties
A.3 DETAILS ON CURATION AND SCORES FOR CORRECTNESS PREFERENCE EVALUATION DATASET
A.3.1 SMALL BENCHMARK MODIFICATIONS
To ensure more natural responses that better reflect real-world use cases, modified each verifi- able benchmark's canonical prompt to encourage Chain of Thought (CoT) thinking (citation). This approach both increases the diversity of sampled responses and enhances the task difficulty for the human preference proxy by incorporating additional signals beyond final answer correctness. The specific instructions for each benchmark are detailed below.
For the MATH benchmark, we implemented new system prompt to facilitate zero-shot CoT be- havior. Additionally, we converted the parsed answer its symbolic representation and utilized symbolic solver to evaluate true equality instead of relying on raw string matching. This refinement of the correctness signal ensures that trivial answer differences, such 1 vs I or ditvB vI +2i, are marked as equivalent, with either answer accepted if correct.
In practice, observed that the sampled MBPP-Plus generations from some models were almost all identical. Models also generally failed to follow instructions to "think step-by-step before providing their final answers, suppressing answer diversity. To address this issue, we prompted the models to "write comments clearly explaining each part of the code," thereby lengthening trajectories and yielding greater exploration of the answer spaces. We also observed some ambiguity in MBPP-Plus intructions. To mitigate this, we added standard MBPP test cases into the function docstring as examples, and used the more extensive remaining MBPP-Plus test cases as the real tests.
Lastly, for IFEval, we prefixed the prompts with "It is extremely important that you follow all in- structions exactly." This addition emphasizes the necessity of precise instruction following in these tasks and ensures that the human preference proxy implicitly recognizes this significant evalua- tion criterion.
The prompt template for MMLU-Pro and GPQA were adaption from Gao et al. (2021) 's Language Model Evaluation Harness. The MATH template was generated with the assistance of Anthropic's prompt generator.
The prompt templates for each benchmark are detailed below. Note that {[var}) indicates a field to be filled by prompt data metadata.
26

---

Published as a conference paper at ICLR 2025
MMLU Prompt Template:
following multiple choice questions answers) [[demain]]- Think scep by step and then finish your with "tha enswer (X) where is the correct letter chofica.
Quastion: [[question]]
Options:
([letter) [[chiice]]
([letter) [Ichoice]]
([letter)]. [Ichoica]]
MATH Prompt Template:
are highly skilled mathematicion tasked selving complex math problems. Your goal is provide clear, step-by-step solutions that be easily parsed and evaluated.
Here is the math problam you need to solve:
<problem> [MMTT__PROBLEM]] </problem:
Box your final anawer using LaTeX, for example: $x \iboxed (Your final rumerical. or algebraic answer]
Now, please solve the
drovide your solution the apeci fiad
GPQA Prompt Template:
The following is ([daneia}} multiple cheice question. Think step by step and then finish your answer with "che answer (X) where x is the correct letter choice.
Quastion: [Iqueation)]
Choices:
[ichoicel]
[choice2]
(choice3)
(D) [[choice4]
MBPP-Plus Prompt Template:
Below will be instruction write python function that accomplishes task. You will also be given starter code with function definition and any required imports. Think step-by-step, weite comments clearly explaining each part of the code, and make sure your code solution is enclosed in markdown ticks (*** [your code here] (*),
cinstruezion> [[instruction]]
</instruction>
<starter_coce>
{[s:arter_code/]
</s:arterr_ode>
IFEval Prompt Template:
is extenely important that you follow all inst exactly:
(Iprompt)]
A.3.2 MORE ON BEST OF K CURVES
These curves represent how much the reward model differentiate the LLM's generations whilst picking from examples drawn from the same distribution. The simple intuition here is that as K increases, the "exploration" of the LIM is expanded, thereby increasing the likelihood that correct
27

---

Published as a conference paper at ICLR 2025
answer lies within the K different samples. However, exploration increases, the likelihood that response that exploits the reward model is present also increases. all best K metrics, we use K = 32, providing both reasonable inference costs balanced with significant enough exploration space to test the reward model's capabilities.
In order distill the curves into interpretable numbers, propose several metrics:
1. Maximum Achieved Performance: the maximum achieved by the reward model any point on the best of K curve. Note that the maximum achieved performance relatively agnostic over-optimization.
2. Error With Respect to Ground Truth:
the expected squared error between the score of the reward model's selected response against the ground truth best response. Once again, let SK be size K random sample of responses from model, g SK - {0, 1) be the ground truth scoring function, and R: SK R be the reward model proxy score. Then, the error with respect ground truth 32 _k=1 Esx ([g max,es R(s)) max,eSı {s))
3. End Score: We also look the final score achieved by the reward model at K 32. If no over-optimization has occurred this should also be the maximum achieved performance.
A.3.3 DETAILED SCORES
Reward Model
MMLU Pro
Math
GPQA
MBPP Plus
Eval
Mean
Athene- RM-70B
0.761
0.607
0.499
0.748
0.633
0.650
InternLM2-20B MeeeLLMM2200BBRaarrd
0.673
0.538
0,471
0.654
0,652
0.598
Llama-3-Ofie
0.590
0.481
0.450
0.819
0.646
0.597
Athene-RMM-B
0.656
0.517
0.459
0.675
0.586
0.579
Nemoorron-43340--eewaar
0.697
0,499
0.484
0.567
0.623
0.574
IntenLmm2-7B-Reward
0.638
0.552
0.457
0.562
0.658
0.573
ArmoRM--Llama3-SB--o
0.654
0.508
0,470
0.602
0.601
0.567
Skywork-Reward-Llaaade
0.641
0.500
0.468
0.581
0.639
0.566
Starling-RM-341 RM-34B
0.476
0.453
0.634
0.569
0.557
Eurus-RM-7B
0.607
0.516
0,438
0.590
0.594
0.549
0.550
0.462
0.447
0.691
0.583
0.547
InternL
1----M2--------- 1-8w-Reward
0.538
0,411
0.451
0.572
0,581
0.510
Burling-RM-7B-AIpp
0.562
0.409
0.433
0.559
0.564
0.505
Modee
0.487
0.349
0.420
0.568
0.539
0.473
Table 30: Reward Model Best of K Performance Across Benchmarks
Reward Model
MMLU Pro
Math
GPQA
MBPP Plus
IF Eval
Mcan
Athene-RM-70B
0.792
0.760
0.603
0.661
0.594
0.682
Internlm2-200 20B-rreward
0.677
0.691
0,562
0.574
0.595
0.620
Jlama-3-offsetbiss-RM.
0.631
0.617
0.541
0.710
0.594
0.619
Athene-RM-8B
0.683
0.673
0.560
0.602
0.556
0.615
Nemotton-443340B-Reaaar
0.704
0.660
0.570
0.506
0.587
0.605
Skywork-Rcward-LLamm--3
0.663
0.678
0.560
0.523
0.586
0.602
Internln2-7B-Reewarrd
0.665
0.718
0.558
0.464
0.605
0.602
ArmoRM-LLamaa-8BB-00
0.678
0,659
0.549
0.538
0,573
0.599
Starling-RM-34B
0.683
0.621
0.547
0.534
0.536
0.584
Eurus-RM-7B
0.627
0.665
0.521
0.537
0.554
0.581
Skywork-Rene RewarddGemma-222
0.542
0,582
0.506
0.572
0.536
0.547
InternIm2-1-85 1-8B-Reward
0.561
0.587
0.538
0.462
0.538
0.537
Burting-RM-7B-Alppe
0.547
0.527
0.506
0.400
0.519
0,500
Maive
0.495
0.528
0.506
0.330
0.511
0.474
Table 31: Area Under ROC Curve for Reward Models across Benchmarks
28

---

Published as a conference paper at ICLR 2025
gemma-2-9b-it
gpt-4o-mini
Llama-3-8B
claude-3-haiku
Reward Model
Loss
Max
End
Loss
Max
End
Loss
Max
End
Loss
Max
End
athene-rm-700
0.093 0.702 0.681 0.110 0.678 0.629
0.113 0.669 0.653 0.131 0.633
0.605
armorm-
(mmorm-llmma-----000
0.119 0.657 0.636
0.147 0.620 0.580
0.179 0.576 0.537 0.194
0.564
0.512
naiveverbositymodel
0.241
0.463
0.250
0.425
0.358
0.448 0.317
0.467 0.355
eurus-rm-7h
0.143 0.627 0.597 0.158 0.613 0.562
0.187
0.562 0.512 0.228 0.531 0.452
Skywork-reward
0.169 0.583 0.543 0.175
0.549
0.534 0.494 0.190 0.558 0.529
wwwwrr-reward--lam-....
0.126 0.643 0.612 0.136 0.633
0.597
0.189
0.565 0.527 0.216
0.561 0.491
sumaa---ffssetbia--ms---
0.133 0,653 0.629 0.146 0.629
0.585
0.210
0.542 0.502
0.151
0.620 0.592
nembotron--3340--eeaaar
0.129 0,641 0.617
0.
0.644
0.618
0.159
0.610 0.583
0.232
0.565 0.485
starling-rm-33b
0.157 0.602 0.570
0.15
0.622
0.563
0,183
0.562
0.528
0.209
0.545 0,487
athene-rm-8h
0.142 0,621 0.584
0.133
0.636
0.600
0.175 0.589
0.543
0.183
0.560 0.531
internlm2-7h-rewar
0.138 0.630 0.588
0.147 0.633
0.581
0.155
0.608
0.581
0.253
0.565 0.462
starling-rm-7b-uu
0.183 0.569 0.535
0.199 0.578
0.516
0.238
0.508
0,476
0.319
0.486 0,378
internlm2-1-8h-e
0,193 0,566 0.501
0.1 0.583
0.506
0.218
0.526
0,480
0.256
0.503
0.448
2itennlm2-20b-ewaard
0.124 0.648 0.626 0.130 0.646
0.607 0.159 0.602 0.570
0.166
0.586 0.570
Table 32: Average Best of K per Sample Model across MMLU Pro, Math, GPQA, MBPP Plus, and IF Eval
Reward Model
gemma-2-9b-ir
4o--no-
Llama-3-8B
claude-3-haike
athene-rm-70bb
0.710
0.648
0.710
0.674
armorn-llama3-8b-vo.1
0.655
0.577
0.616
0.591
naiveverbositymodel
0.515
0.491
0.487
0.433
eurus-rm-7b
0.620
0,546
0.621
0,562
skywork-reward-gemme
0.553
0.519
0.562
0.550
0.639
0,594
0.619
0,578
llama-3-offsetbis------
0,628
0.574
0.583
0,650
nemotron-443300--rwaard
0.639
0.586
0.658
0.561
str4ling-rm-34b
0.602
0.571
0.604
0.574
athene-rm-8h
0.640
0.592
0.635
0.601
internlm2-7b-rewward
0.657
0.573
0.655
0.569
Htarinng-mm-7balpph
0.544
0.499
0.525
0.475
mitemmm2-1-88--rew
0,581
0.536
0.570
0.504
intemmlm2-20--reward
0.629
0.603
0.650
0.603
Table 33: Average AUC per sample model across MMLU Pro, Math, GPQA, MBPP Plus, and IF Eval
29

---

Published as a conference paper at ICLR 2025
gemma-2-9b-it
gpt-4o-mini-2022 -002--0--22
1.0
1.0
Ground Iruth
0.9
AmookK_jmma--8400)
Athene-RM-708
Athene-AWW58
0.8
0.8
Funus-RM..D
Inferaim2-1_8--Paaaar
Internim2-1_ _.B--Rewarr
0.7
0.7
hnlernim2-228-Reeeartt
interiim2-75-aawamd
0.6
Uema- Bormaa--OffeeeBiis---e
0.6
NalveleerbasiylModee
Nemetron-4.34CC 3-adtron--33CB-Raaaarr
0.5
0.5
Skywo ReRwaar Gemma- 2-278
BkyworkRRword
0.4
Starling-RMM RM-34B
10
20
10
20
Starling- RMphB-alpha
Meta- Meta-Llama-33 Llama-a-aaaaa-- 1aaaaa---------tuee
1.0
1.0
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
10
20
10
20
30
Figure 8: Performance average across all benchmarks, conditioned on each sample model
30

---

Published as a conference paper at ICLR 2025
Best of K Performance Across Benchmarks
GPQA Best of K
IFEval Best of K
1.0
1.0
0.9
0.9
0.8
Score
Score
0.8
0.7
0.7
0.6
0.5
0.6
0.4
0.5
20
30
0
10
K
K
Math Best of K
MBPP-Plus Best of K
1.0
1.0
0.9
0.8
0.8
Score
Score
0.7
0.6
0.6
0.4
0.5
0.4
10
20
30
10
20
30
K
K
MMLU-Pro Best of K
Ground Truth
ArmoRM-Lama--8Bvo.11
1.0
Athene-RM-70B
Athene-RM-8B
0.9
Eurus-RM-7b
0.8
Internim2-1 8B-Reward
Score
Internim2-20B-Rene
0.7
Internim2-78-Rewarl
Barma-3-OffsetBias--Pe
0.6
NaiveVerbossiyModel
Nemotron-443408-Rewarrd
0.5
Skywork-Reward-GGemme
Skywork-Rewarrd-LLmm-l3...
0.4
Starling-RM-33
10
20
30
Starling-RM- 7B-alha
K
Figure 9; Performance comparison across all benchmarks
31

---

Published as a conference paper at ICLR 2025
A.3.4 SCORE DISTRIBUTION STATISTICS OF CORRECTNESS BENCHMARKS
mean
std
min
25%
50%
75%
max
accuracy
0.557
0.031
0.477
0.544
0.561
0.570
0.632
vrea_under_curve
0.545
0.028
0.506
0.525
0.548
0.560
0.603
loss
0.265
0.026
0.219
0.251
0.265
0.270
0.322
mean m
more
0.458
0.020
0.424
0.449
0.455
0.469
0.498
mean_end_score
0.432
0.031
0.372
0.423
0.431
0.453
0.481
Table 34: GPQA Benchmark Score Distribution Information
mean
std
min
25%
50%
75%
max
accuracy
0.581
0.035
0.517
0.560
0.576
0.617
0.640
area_under.curve
0.563
0.031
0.511
0.536
0.565
0.593
0.605
loss
0.121
0.025
0.090
0.097
0.122
0.135
0.173
mean mww.mmmaa.mm -core
0.605
0.037
0.540
0.581
0.599
0.638
0.658
mean end score
0.590
0.047
0.503
0.563
0.579
0.631
0.654
Table 35: IFEVAL Benchmark Score Distribution Information
mean
std
min
25%
50%
75%
max
accuracy
0.693
0.091
0.498
0.645
0.693
0.726
0.866
area_under.curve
0.656
0.089
0.527
0.602
0.660
0.684
0.878
loss
0.199
0.080
0.047
0.169
0.189
0.214
0.401
@rean_maxssoee
0.504
0.091
0.348
0.470
0.500
0.527
0.741
mean_end_score
0.486
0.107
0.245
0.459
0.494
0.516
0.736
Table 36: Math Benchmark Score Distribution Information
mean
std
min
25%
50%
75%
max
accuracy
0.533
0.095
0.312
0.510
0.538
0.580
0.743
area under curve
curve
0.530
0.098
0.330
0.474
0.536
0.573
0.710
loss
0.177
0.092
0.035
0.110
0.176
0.221
0.337
mean mean_max wwi__maa_sooo score
0.631
0.078
0.557
0.577
0.596
0.668
0.818
mean_end_score
0.565
0. 0.134
0.376
0.491
0.544
0.658
0.815
Table 37: MBPP Plus Benchmark Score Distribution Information
mean
std
min
25%
50%
75%
max
accuracy
0.654
0.078
0.479
0.615
0.662
0.684
0.814
area under curve
curve
0.639
0.079
0.495
0.578
0.664
0.682
0.792
loss
0.139
0.059
0.053
0.109
0.118
0.172
0.291
mean_me max_scor
0.622
0.073
0.483
0.570
0.640
0.655
0.762
mean_end rean_end_score
0.605
0.089
0.403
0.559
0.630
0.647
0.760
Table 38: MMLU Pro Benchmark Score Distribution Information
32

---

Published as a conference paper at ICLR 2025
A.4 DPO CONFIGURATION
DPO Configuration
Base Model
0.1
Learning Rate
2.00 x 10-0.6
LR Schedule
Constant
Global Batch Size
64
Max Length
8192
Max Prompt Length
4096
Implementation TRL DPOTrainer (von Werra et 2020
Optimizer
AdamW, B1 = 0.9, D3 0.999
Space Optimization
Deepspeed Zero2
A.5
CROWDSOURCED HUMAN PREFERENCE VOTE DETAILS
#Votes
Unique Users
Mean Votes/User
Median Votes/Uxerr
Mean Battles/Pair
Mean Votes/Model
12190
6120
1.99
1.00
190.47
2031.67
Table 39: Statistics on vote participation and distribution for crowdsourced human preference labels.
A.6
ADDITIONAL ANALYSIS ON DOWNSTREAM PERFORMANCE
Performance Correlation Human Preference Category Score Quantile
Spearman Correlation
Confidence Agreement
Accuracy
Brier Score
0.4
Pearson Correiation
0.6
Correlation
Pesison Comelattin
0.2
0.5
0.2
0.4
Peerson
0.3
0.0
-0.2
0.2
0.65
-02
6.1
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
Category Score Quantile
Categary Score Quant
Category Score Quantile
Category Score Quantile
Kendall Tbu
Row-w se Pearson
Soparability
0.65
Pearson Cotre atten
Dearson Corre aiten
0.52
0.60
0.2
0.55
0.50
0.0
-0.2
1.0
0.0
Category Quantile
Category Score Quantile
Category Score Quantlie
Figure 10: The graphs show all metrics for the human preference dataset. For each metric, the six benchmarks (Hard, Easy, Instruction Following, Coding, Math, and Similar Responses Prompts) (all mean and SD normalized) aggregated into final score by quantile (X-axis). The Pearson Correlation between the aggregated scores are calculated relative to Post-RLHF Human Preference ratings for each aggregation level. Notice that for all metrics except Separability, decreasing quantile increases correlation.
One possible cause of the pattern seen in Figure 10 is that low quantile aggregation better measures robustness. Intuitively, any single weakness within some input domain could be exploited by the policy model during RL training. thus damaging the model. Another reasonable explanation is that a reward model's weakness in one area may yield noisy signals during training, causing the policy model's rather fragile parameters to be disrupted- possibly unrecoverable degradation in what
33

---

Published as a conference paper at ICLR 2025
may consider an instance of "catastrophic forgetting". Ultimately, the underlying mechanisms are complex; we do not expect to answer this question in entirety. However, we believe that our end-to-end experiment provides the first step to understanding how reward model behaviors relate to downstream performance.
A.6.1 COMMENTS ON REWARDBENCH CORRELATIONS
Commenting on Figure 6 while our work's focus was not to prove or disprove RewardBench, we can provide the following hypothesis for context and clarity: we hypothesize that the reward mod- els tested may have over-optimized for RewardBench' specific preference distribution rather than capturing broader human preferences, potentially exceeding RewardBench's measurement capabil- ities. However, we note that initial improvements in RewardBench score may still correlate well to real post RLHF human preference outcomes. Ultimately, these insights are only possible through end-to-end experiments, which enable the research community to further investigate and discuss the true correlations between benchmark metries and downstream performance. We believe this highlights the value of comprehensive evaluation approaches like ours in understanding real-world model behaviors.
A.6.2 STYLE-CONTROLLED DOWNSTREAM PERFORMANCE
Model
Elo
95% CI Lower
95% CI Upper
Meta-Llama-3..
1229
1218
1239
Athene-RM-70B
1209
1201
1218
Athene-RM-8B
1203
1194
1211
internlm2-7b-rewaard
1201
1192
1210
Llama-3-OffietBias-RRMM
1197
1188
1204
ArmoRM-Llama3-8B--01.
1185
1175
1191
Mecta-Laamma331188--nmmmmee
1177
1168
1186
Skywork-Rewart-JJamm-3
1-8B
1171
1163
1182
Nemooron--43340--ewward
1170
1161
1180
intermln2-200--ewaard Serwor--Rwward-Gemma22222
1170
1159
1179
1170
1160
1180
Meta-L.lma--3-8--mnee
1152
1142
1160
Table 40: Post DPO performance real human preference Overall Category after applying style- control. "Model" is the reward model used train the base model. Models marked with *gn are baseline unaltered models. The best non-hase model elo is bolded.
34

---

Published as a conference paper at ICLR 2025
Correlation to Style Contrelled Post RLHF Arena Score:
Correlation to Style Metrics the Human Preference Datazet
Stylc-Contralled Controlled Post RLHF Arena Score:
Metrics on the Correctness Dataset
0.8
Accuracy
0.48
0.53
0.13
0.40
0.20
0.64
Accuracy
0.67
0.71
0.60
0.61
0.76
3.54
0.66
0.6
0.4
RDC AUC
0.47
0.49
0.43
0.43
0.25
0.61
Correlation
Sapanabliyy
0.35
0.49
0.18
0.42
0.60
0.50
0.36
0.2
Corfidence Agreoment"
0:45
Kendaltss Thu"
0.00
0.04
-0.04
0.08
0.14
-0.04
0.14
Scone
0.26
-0.02
0.28
0.07
0.46
Loss
0.30
0,58
0.11
0.24
0.12
0.49
Brirr Score"
0.03
0.09
0.01
0.07
0.09
0.14
0.18
-0.6
harer mus
Overall
Herd Prompea
Pmropgh
Prongtt
Custin Rromgtt
Mart Pronggts
Siovae Responso
0.8
Mesus
Granuar Levels Madium Cramu
Snaleity
Granularity
Figure 11: Pearson correlations between various metrics and styled-controlled human preference scores. Left: Correlations between metrics on the Correctness Dataset and Post-RLHF human pref- erence rating. Right: Correlations between metries the Human Preference Dataset and Post- RLHF human preference rating.
Correlation to Style-Controlled
Arena
Cordiation
0.5
Accuracy
0.82
-0.82
9.50
0.79
-255
051
0.0
Mean
Saletsy
Cual Heni
Sabety
Figure 12: Pearson correlation between the ranking of models in RewardBench and their respective style-controlled Post-DPO rankings on real human preference.
As ablation, we calculate style-controlled human preference ratings. Style-controlled ratings fit the Bradley Terry model with style elements as features the regression. These features are used to decouple style from model ratings; this process yields score estimates, style aside. The full process for style control is detailed Li et al. (2024)). For maximum coverage, we control for length and markdown.
35

---

Published as a conference paper at ICLR 2025
A.6.3
CORRELATION VS. K
Mean Max Score Correlation Downstream Performance vs
Task
0.7
MATH
GPQA
IFEval
MMLU-Pro
Pearson Correlation
MBPP-Plus
-0.1
15
20
30
Figure 13: Pearson correlation to downstream human preference performance of mean max score best of K metric vs K.
Figure 13 shows that increasing the value of K for best of metrics does increase benchmark predictive power. We note that the most predictive correctness metrics is the accuracy metric detailed subsubsection 5.2.3 which is inherently K 2. Therefore, the predictive power of PPE can be retained without running full K 32, which is more compute heavy.
A.7 RECOMMENDATIONS FOR PPE AND FUTURE REWARD MODEL BENCHMARKS
Based on this end-to-end study results detailed in section and Appendix Figure 13 we recommend those seeking the most predietive power from PPE run the human preference set well as the MATH accuracy metric. We suggest that users pay particular attention to the lower bound accuracy the main human preference and similar). Considering our findings, this configuration likely maintains full predictive power of PPE with less than half of the runtime. Future reward benchmarks may find helpful to attend to these particular design patterns.
categories (easy, hard, instruction following, coding, math,
A.8 RUNTIMES AND COSTS FOR PPE
Benchmark Set
Time
Cost < $1.50
Optimized (Human Preference VI Math Accuracy)
42 minutes
Full Benchmark End-to-end RLHF pipeline
120 minutes
< $3.50
> I week
$1000 or more
Table 41: Benchmark runtimes and costs. Costs are calculated from RunPod's hourly GPU pricing, which puts an NVIDIA A 100 80GB PCIe instance at $1 64 per hour. Costs could fluctuate between GPU providers. Runtimes are estimated assuming an 8B reward model.
36


Reviews:
[{"review_id": "bavWzg5iMj", "reviewer": "Reviewer_nK4S", "summary": "This paper introduces Preference Proxy Evaluations (PPE), the first reward model benchmark explicitly connected to real-world human preference performance after RLHF. Unlike benchmarks like RewardBench that primarily perform reward evaluations, PPE facilitates investigation into which reward model metrics correlate most closely with RLHF outcomes.\n\nStudying the impact of reward models on post-RLHF model performance is important. This paper involves significant work on this topic. Although I am inclined to accept it, there are still some important shortcomings (see the **Weaknesses**). I hope the authors can address my concerns.", "questions": "- I am unclear about why the Pearson correlation for “chat hard” in Figure 4 is so low. Does this imply that most RewardBench metrics do not reflect post-RLHF performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper addresses an important topic: the effectiveness of a reward model should ultimately be assessed by the performance of LLMs after RLHF. The study offers valuable insights into the evaluation of existing reward models.\n- The paper includes extensive experiments and evaluates models using real-world human preferences.", "weaknesses": "- The study employs DPO as the RLHF algorithm to evaluate post-RLHF performance. However, the offline DPO algorithm may face generalization issues. Utilizing an online PPO algorithm to obtain post-RLHF models and assess their performance would offer a more comprehensive evaluation of various reward models. This additional experimentation is essential to support the authors’ core claim of being “the first reward model benchmark explicitly linked to post-RLHF performance.”\n- The paper utilizes Llama-3.1-8B-Instruct as the base model for RLHF, which presents two potential concerns: 1) The paper should explore a broader range of base model scales rather than exclusively using the 8B model to fully assess the new benchmark’s validity. 2) Llama-3.1-8B-Instruct itself has undergone DPO RLHF training. I recommend that the authors use a purely SFT-trained Llama-SFT model as the base model.\n- The paper would benefit from more detailed descriptions and discussions regarding Figure 4. For example, it should provide intuitive explanations for why metrics from RewardBench fail to reflect post-RLHF preference performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PAiiI8gXPw", "reviewer": "Reviewer_Ub3i", "summary": "It proposes Preference Proxy Evaluations (PPE), a novel benchmark designed to evaluate reward models based on their ability to predict post-RLHF outcomes. They use a crowdsourced human preferences and a verifiable correctness dataset for the RM evaluation.", "questions": "- Why, intuitively or theoretically, is low quantile aggregation correlation correlatedmore strongly with downstream RLHF performance? Please provide hypotheses or conduct additional analyses to investigate this relationship further.\n- How does inference cost scale with K, and what's a feasible value of K for training a large language model today? Please provide a table or graph showing inference costs for different K values, and discuss how these costs might impact real-world applications of the method.\n- Why do pairwise accuracy-like metrics suffer from overfitting? Discuss this in more detail and provide empirical evidence of overfitting in pairwise accuracy metrics if possible.\n- Is there a predictive accuracy plateau after a certain K-value? Is it possible to include a graph demonstrating diminishing returns/values of predictive accuracy with K?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Achieves 77% Pearson correlation with human preference ELO. \n- Well motivates why best-of-K scores, row-wise Pearson correlation, & accuracy are relevant for downstream RLHF success. \n- Human preference dataset is large and useful in producing statistically significant outcomes (16,038 labeled preference pairs).", "weaknesses": "- Underdiscusses the possibility that the RM evals are computationally very expensive to run in the the PPE framework. Best-of-K performance curves and pairwise accuracy have a huge computational burden, especially as K increases. Please provide runtime estimates for different K values or discuss strategies for making the evaluations more computationally efficient.\n- Does not discusses the findings (e.g. low quantile aggregation correlation) in any depth. Please explore potential explanations for this correlation or discuss its implications for reward model design and training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I6Tzp9xjuq", "reviewer": "Reviewer_PDs3", "summary": "The paper tackles the problem of evaluating reward models without having to run the expensive evaluation of downstream LLM performance. The authors compare metrics that are predictive of actual human evaluations on downstream performance, when employed directly at the reward model stage. Different metrics across preference and correctness are compared to down-stream performance, and rew. model accuracy is identified as the best metric predicting downstream human preferences. Correctness is default benchmarks is also a reasonable indicator of LLM performance.", "questions": "Needs clarification\n- Section 4 felt a bit out of place (so the first section), especially “The human preference dataset contains human-labeled preferences for 16,038 pairwise comparisons between 20 selected top models”: How is this dataset used?  In the paper, there are references to multiple datasets and collections, and I find it a bit confusing to follow how many datasets exist, and for which experiment they are used (e.g. 16,038 preferences in section 4, 12,190 votes in section 6, 50.000 votes in sec 6.1.\n- I think the structuring of sections 4 and 5 could be improved a bit, in particular I was a bit confused about the content of section 4, I think a figure showing the workflow/process would help to clarify this\n- I would like to ask the authors opinion on using already RLHF fine-tuned models as the source of data. Does it introduce its own biases, in particular to the distribution of responses? Doesn’t it implicitly reward similarity to already existing RLHF strategies?\n- Can the caption of Figure 2 be improved? I’m not sure what I see in the right plot compared to the left one, how does the message differ?\n- The inverse correlation between reward bench and human elo (Figure 4) is interesting, I would really like the authors to provide a hypothesis for this", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Strengths:\n- The problem is highly relevant, the evaluation of reward models and RLHF workflows is a challenging problem\n- The collection of human preferences and release as open-source is a valuable contribution to the community, the amount of collected human data is impressive (especially including multi-lingual queries)\n- The approach of grounding metrics with human data is highly appreciated, and can be a valuable tool for future reward model development\n- The selection of models, prompts and correctness metrics is well motivated", "weaknesses": "Weaknesses\n- Some of the results seem pretty surprising (e.g. the low correlation with reward bench), for me it’s a bit unclear why we observe them. It would be great if the authors could try to reason more about these observations (as outlined below)\n- Clarity of writing and presentation could be improved\n- \"Reward Model Accuracy\" as the best metric is not a novel contribution in itself, you may argue that it’s good to have it validated towards human preferences\n- Overall, i find the findings a bit weak, i.e. i don't find it easy to draw clear lessons for reward model design (larger vs. smaller models), i think it would have been very helpful to discuss these results more\n- I would have liked some more information about variance/stds., e.g. spread of accuracy across sub datasets, etc., best-of-k curves, etc.\n\nMinor\n- Some formulations feel a bit clunky, e.g. P6, L.318. “We use the reward models to RLHF Llama-1.1-8B using..”, P8, L.327: “Mean across all metrics is well correlated across all metrics”  I think the paper could profit from one additional pass of copy editing\n- I think adding some figures/plots would help readability, e.g. turning table 3 into a chart\n\nI think there are multiple ways to improve the paper, but the main content is generally convincing to me. If the authors can address some of the points for revision, I am ready to raise my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e54Vwn3efS", "reviewer": "Reviewer_xNaA", "summary": "This paper presents PPE, a reward model benchmark explicitly tied to post-RLHF outcomes based on real human preferences. Previous reward model benchmarks focused solely on evaluating the reward models themselves, without assessing their post-RLHF performance. PPE fills this gap. More importantly, the paper analyzes which metrics best reflect post-RLHF performance, providing valuable insights for improving reward model evaluation methods.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Compared to previous reward model benchmarks, PPE directly evaluates the post-RLHF performance of models, which more accurately reflects the positive impact of reward models on LLM tuning.\n- PPE offers more diverse and larger-scale datasets than earlier benchmarks, making it a highly robust benchmark.\n- The paper is very detailed and easy to follow, allowing readers to clearly understand the evaluation process. The explanations for each metric and evaluation method are convincing.\n- I personally appreciate Section 7 of the paper. The analyses can help researchers predict the performance of fine-tuned LLMs using high-confidence metrics when they lack the resources to conduct full post-RLHF evaluations.", "weaknesses": "- I believe this work fills a significant gap in reward model evaluation by addressing post-RLHF performance testing. The effort is substantial and thorough. I do not observe any obvious weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)