RewardBench 2: Advancing Reward Model Evaluation
Authors: Saumya Malik
Valentina Pyatkin
Sander Land
Jacob Morrison
Noah A. Smith
Hannaneh Hajishirzi
Nathan Lambert
Venue: ICLR 2026 Poster (2026)
Keywords: reward models
benchmark
evaluation
post-training
reinforcement learning from human feedback
Primary area: datasets and benchmarks

Abstract: Reward models are used throughout the post-training of language models to capture nuanced signals from preference data and provide a training target for optimization across instruction following, reasoning, safety, and more domains. The community has begun establishing best practices for evaluating reward models, from the development of benchmarks that test capabilities in specific skill areas to others that test agreement with human preferences. At the same time, progress in evaluation has not been mirrored by the effectiveness of reward models in downstream tasks -- simpler direct alignment algorithms are reported to work better in many cases. This paper introduces RewardBench 2, a new multi-skill reward modeling benchmark designed to bring new, challenging data for accuracy-based reward model evaluation -- models score about 20 points on average lower on RewardBench 2 compared to RewardBench, a widely-used existing reward model evaluation-- while being highly correlated with downstream performance. Compared to most other benchmarks, RewardBench 2 sources new human prompts instead of existing prompts from downstream evaluations, facilitating more rigorous evaluation practices. In this paper, we describe our benchmark construction process and report how existing models perform on it, while quantifying and providing new insights on how performance on the benchmark correlates with downstream use of the models in both inference-time scaling algorithms, like best-of-N sampling, and RLHF training algorithms like proximal policy optimization.

Published as a conference paper at ICLR 2026
REWARDBENCH 2:
ADVANCING REWARD MODEL EVALUATION
Saumya M Valentina Pyatkin Sander Land Jacob Morrison"
Noah A. Smith e Hannaneh Hajishirzi Nathan Lambert
a Allen Institute for Artificial Intelligence University of Washington .Cohere
contact: saumyamãa welllln.... .org
ABSTRACT
Reward models are used throughout the post-training language models to capture nuanced signals from preference data and provide training target for optimization across instruction following, reasoning, safety, and more domains. The community has begun establishing best practices for evaluating reward models, from the devel- opment of benchmarks that test capabilities in specific skill areas to others that test agreement with human preferences. At the same time, progress in evaluation has not been mirrored by the effectiveness reward models in downstream tasks - simpler direct alignment algorithms reported to work better in many cases. This paper introduces REWARDBENT BENCH 2, new multi-skill reward modeling benchmark de- signed to bring new, challenging data for accuracy-based reward model evaluation models score about 20 points on average lower on REWARDBENCH 2 compared to RewardBench, a widely-used existing reward model evaluation- while being highly correlated with downstream performance. Compared to most other benchmarks, REWARD BENCH 2 sources new human prompts instead of existing prompts from downstream evaluations, facilitating more rigorous evaluation practices. In this paper, we describe our benchmark construction process and report how existing models perform on it, while quantifying and providing new insights on how per- formance on the benchmark correlates with downstream use of the models in both inference-tine titeence-timeeeeee scaling algorithms, like best-of-N sampling, and RLHF training algorithms like proximal policy optimization.
INTRODUCTION
Reward Models (RMs) are often designed to model human preferences to improve language model training (Ouyang et al. 2022; Bai et al., 2022 Touvron et al. 2023 Dubey et al., 2024). Generally, a reward model is trained to output a scalar value proportional to (some aspects of) the quality of the input text, learned from preference data. RMs have been used extensively for RLHF training (Nakano
al. 2021; Glaese et al. 2022), but also are used for online direct alignment algorithms (Singhal ct al. 2024), data filtering Albalak et al. 2024 Dubey et al.) 2024), and inference-time scaling (Faria S Chow et al. 2024). Despite extensive use, the ecosystem of directly evaluating reward models is still nascent and developing alongside the roles RMs play.
Users developing RMs for their application must decide which benchmark(s) to use. This is a multi-dimensional decision process, as evaluations vary in how they measure performance (e.g.. accuracy vs. correlation with -uaaae---uddge) and the domains they focus on (e-g.. multi-skill vs. chat-only). The first reward model evaluations such RewardBench (Lambert al. 2024b) and RM-Bench (Liu et al., 2024b) focused on simple classification tasks to measure performance of existing reward models across common domains like style and safety. Additional evaluations included analysis of downstream scores when the R.M used within inference-ting time methods such as best-of-N
(BoN) sampling (Zhou et al. 2024) and also training with RLHF (Frick al. 2024).
We present REWARDBENCH 2, a benchmark built on classification tasks that measures and improves correlations relative to earlier approaches of RM evaluations in two scenarios: inference-time compute

---

Published as a conference paper at ICLR 2026
Low Downstream RM Use
High
Instruction Followiig
Prompt Werte where sentwene <acta wiih the Jeterr
Chosen Mallaet fudts shadow monee, A
HM correct chosen all rejorted
Math
Safetyy
SF
Reward model
Focue
Tiee
Factuality
Rejected Betwen quiet
0.1, 0.3, 0.4 New het
Best-of-N
Multiple incorrect
anowerre per prorppt.
Unseen, human data ecross domains
Score reward modals with best-of-4 accuracy
Correlated with downstream use
Figure 1: REWARDBENCH 2 is composed of high-quality unseen human prompts designed for a best-of-4 reward model evaluation format with completions generated from a variety of leading Al models. We extend RM evaluation of pairwise "chosen" and "rejected" completions to include additional rejected samples as distractions. REWARDBENCH has domains which expand upon challenging domains in existing RM evaluations and adds a new domain, Ties, to test how RMs handle questions with multiple correct answers. The new data and setup enables more accurate correlation of benchmark scores with downstream performance via RL finetuning or best-of-N sampling.
and downstream training (highlighted in Figure D. Our benchmark maintains strengths of multiple existing benchmarks, such as using unseen human prompts or switching from the common practice of accuracy over a chosen and rejected response to one chosen and three rejected responses to reduce the distance between strong reward models and the random baseline, as summarized in Table
The benchmark covers six domains: three new datasets to improve evaluation in domains covered by existing RM benchmarks focus, math, and safety along with three new challenging domains: factuality, precise instruction following, and ties (a new type of domain where we test a RM's ss ability to be well-calibrated between equivalently valid answers, like "red" and "green" in response "Name color of the rainbow"). In total we evaluate over 100 reward models, a mix of leading existing models and new models we trained to better understand the relationship between RM training and evaluation, in order to allow more reliable use of RMs across variety of skills often targeted in post-training in order to allow more reliable use of reward models.
The benchmark created with majority of previously unused human prompts from the WildChat pipeline (Zhao et al. 2024) with extensive manual, programmatic, and LM-based filtering techniques. extensive experiments to show how RM benchmarks can be used
To validate the benchmark, in effective RLHF training workflows or correlated hillclimbing targets for inference-time compute techniques. Our contributions and findings are as follows:
1. REWARDBENCH provides challenging evaluation of reward models across many domains on majority unseen prompts, with leading models on RewardBench (the most widely-used existing benchmark) scoring 20 or more points lower on REWARDBENCII 2. This includes challenging subsets such Precise Instruction Following and Math where leading models are below 40% and 70% accuracy, respectively, with data details discussed in Section
2. Controlled experiments where we train reward models and analyze their performance on the benchmark, gaining actionable insights for reward model training. In particular, we find that different post-trained base models, even within the same lineage and model family, offer different capabilities to reward models and that, contrary to the accepted best practice, training for more than one epoch can be beneficial. We discuss these findings in Section 4
3. An exploration of the benefits and limits of using reward model evaluation to inform downstream use cases of inference-time scaling algorithms and RLHF training. In Section 5 our benchmark achieves strong downstream correlation with inference time scaling algorithms like best-of-N sampling and provides helpful signal for PPO training.
4. Our analysis shows how the best reward model for RLHF is dependent on one's training setup. For RLHE, the reward model should be based model of the same lineage as the policy model or else downstream performance can degrade significantly, so simply taking the highest scoring reward model on benchmark will not ensure good post RL.F model.
2

---

Published as a conference paper at ICLR 2026
Table A comparison of REWARDBENCH 2 relative to existing reward modeling benchmarks. For metrics, is used to denote an accuracy metric (correctness) and 4 is used where the metric is either human or LM-as-a-judge agreement. Comparing the relative correlation of each RM benchmark with downstream tasks is challenging because the correlation depends on the downstream tasks of choice. denotes benchmarks meant to test one specifie attribute (e.g., typos, multilinguality).
Best-of-N (N > 2)
Human Prompts
Unseen Prompts
Multi
RM Evaluation
Metric
Skill
RewardBench (Lambert et al. 2024b)
X
x
RewardMATH Kim et al. 2024)
X
RM-Bench (Liu al.] 2024b)
X
X X
ReWoorBBnnch (Wu et al.
Re
2025
X
M M-RewardBench
Gureja et al. 2024
X
X
PPE (Frick et al.
2024
Correctness
X
PPE
Frick et al.
2024
Human Pref.
X
X
RMB (Zhou et al.
2024)
X
REWARDBENCH 2
/ V
BACKGROUND
Reward Models Reward models are trained on preference data, consisting of prompts and comple- tions ys, where each completion has been ranked by humans or automated metries like ground truth signals and language model judgments (Lambert 2025). The canonical formulation, which use in this work, is to create preference pairs, where for each prompt two completions are compared, and the better prompt is "chosen", and the other is "rejected." With that data, reward model r* is trained to output a scalar value to predict the probability p" of a prompt and completion falling in the chosen category, following a Madley-Terryy model of human preferences (Bradley & Terry 1952):
exp(r" (x. 3/1)
('11 > x)) exp(r"(x,, v1)) + (x,(r*(z,)))
(1)
The Bradley-Terry formulation of preference is fit through maximum likelihood estimation:
C(0, D) [ Jeteese Bigena))
[log(1 e (a..eea))))) (2.
For more information on how reward models are used, such as in reinforcement learning from human feedback (RLHF) and best-of-N (BoN) sampling, see Appendix B
Reward Model Benchmarking Reward model evaluation has expanded to be similar to the types of evaluations available to general post-trat trained models, where some evaluations test the accuracy of prediction on domains with known true answers (Lambert et al.] 2024b) while others measure preferences (colloquially referred as "vibes") performed with L.M-as-a-judge correlations to other benchmarks (Wen et al. 2024). Recent reward model benchmarks fall into three categories: (1) Benchmarks focusing on general downstream performance, continuing from RewardBench, include Preference Proxy Evaluations (Frick et al., 2024), RMB (Zhou et al. 2024), and RM-Bench (Liu
al. 2024b).4
(2) Specific new attributes test include multilinguality (Gureja al. 2024), agentic
systems (e.g.. web agents (Lù et al. 2025) or retrieval augmented generation (Jin et al. 2024
typos (Wu al. 2025). and others (Kim et al.) (024).. (3) Benchmarks testing different modalities or structures of reward modeling include those for multimodal (Chen et al.) 2024 Yasunaga et al. 2025
Li
2024 Ruan et al. 2025), process reward (Song et al.) 2025). or visual process reward Wang al. 2025 Tu et al. 2025) models.
We compare REWARDBENCII 2 to recent text-only reward model benchmarks listed Table I (See Appendix C for more detailed comparison). We highlight the importance of REWARDBENCH 2 using unseen human prompts, a departure from most prior work that repurposes prompts from widely- used downstream evaluations to evaluate reward models. Without entirely new prompts, claims of correlations to downstream benchmarks must overcome the potential contamination with respect to the downstream evaluation target. Additionally, while benchmarks whose chosen-rejected splits
3

---

Published as a conference paper at ICLR 2026
Table 2: REWARDBENCH 2 domains and their various specific construction decisions. We prioritized using new human prompis with robust subset-specific completion generation and verification pipelines. In total there are 1,865 prompts and completions from 20 different models (see Appendix [ for full list) or human-writ written completions. Prompts sourced "manually" denote those created by the authors, while "human" denotes those collected from in-the-wild chat interactions. Focus does not need filtering because it is created with specific prompting that differentiates the chosen and rejected completions (followed by manual verification of the method rather than every instance).
Prompt Source
Method generating
Completion
Domain
Count
completions
Filtering
Factuality
475
Human
Both
Multi-LM-as-a-juee
Precise IF
160
Human
Natural
Verifier functions
Math
183
Human
Natural
Majority voting
Safety
450
CoCoNot
Both
Rubrics & Human Annotation
Focus
495
Human
System Prompt Variation
N/A
Ties
102
Manual
System Prompt Variation
Manual verification
determined by human or LM pairwise preferences have some benefits, there is subjectivity in the preferences they prescribe as optimization targets (Lambert et al., [023 Zhang et al. 2024b). With the focus of REWARDBENCH on downstream skills, we opt use accuracy- bassed tests.
BUILDING THE BENCHMARK AND MEASURING PERFORMANCE
In this section, we detail the data curation and scoring methods used for REWARDBENCH 2 that enable challenging, accuracy-basse benchmark correlated with downstream post-training evaluations. This involves four stages: prompt sourcing, where most of our prompts are unreleased human-written queries obtained with user consent from WildChat (Zhao et al. (024);; prompt quality and domain annotation using classifiers; completion generation, where we aim for diversity while ensuring we construct both "right" and "wrong" completions; and filtering, where verify that prompts and completions fit each domain's 's criteria. We will release our code under the Apache 2.0 license and the benchmark data under ODC-By upon paper acceptance.
Prompt Sourcing We focused on getting representative, unseen prompts from real usage of language models and pairing them with completions representative of the current spectrum of language modeling performance. The goal is to make reward model evaluation prompts independent from evaluations used to test downstream post-trained models. Prompts denoted as "Human" in Table 2 unseen and reflect real world use of AI models (~70% of the benchmark). From a pool of prompts, we filtered and assigned prompts to our domain-specific subsets using a combination of QuRater (Wettig al. 2024 to annotate data, topic classifier to identify prompt domain, and manual inspection. We compared our prompts against twenty widely-used downstream evaluations with the Tulu 3 decontamination toolkit (Lambert et al.) 2024a) and ensured no overlap. To arrive at our final dataset, we first created an initial set of around 3,000 total high-quality prompts in our target domains, and then eurated the final 1,865 prompts through further manual verification and filtering.
Constructing REWARDBENCH 2's Domains An overview of the 6 domains in REWARD- BENCH 2 and how they were created is detailed in Table The Math, Safety, and Focus domains are new datasets inspired by improving upon the Math, Safety, and Chat-Hard domains, respec- tively, of RewardBench and RM-Bench, whereas Factuality, Precise IF, and Ties are designed to test additional capabilities of RMs not captured in existing evaluations. In summary, the subsets of REWARD BENCH are as follows, with examples from each subset Appendix A and additional dataset creation details in Appendix G
1. Factuality: Tests the ability of RMs to detect hallucinations and other basic errors in comple- tions. To construct this subset, we sampled both natural completions as well as completions from added system prompt instructing the model to make subtle factual errors. We classify these responses as "accurate" or "inaccurate" by prompting two LLMs to judge their accuracy independently, and assigning label only if both L.MSs agree ("accurate" responses go into the
4

---

Published as a conference paper at ICLR 2026
chosen category and "inaccurate" build rejected completions). We spot check examples to verify the integrity of our double LLM-as-judge verification setup.
2. Precise Instruction Following: Tests the ability of RMs to judge whether text follows precise instructions, such as "Answer without the letter u". We append a constraint taken from the taxonomy of new instruction-following benchmark, IFBench (Pyatkin al. 2025). to each prompt, manually ensuring relevance (more details in Appendix (G.)) [ We use verifier functions evaluate adherence the constraint, and constructed each data instance by combining I completion that satisfies the constraint and 3 that do not, and manually verify for each example that adherence to the constraint did not otherwise compromise the quality of response.
3. Math: Tests RMs' abilities at math, on open-ended human prompts ranging from middle school physics and geometry to college-level chemistry, calculus, combinatories, and more. To grade completions, we used majority voting to populate candidate set of prompts with 1 correct and 3 incorrect prompts and then manually verified every sample in this domain due to the brittle nature of answer extraction.
4. Safety: Tests RMs' abilities to correctly comply with or refuse prompts related to harmful use cases. Safety is nuanced task for LMs, so draw recent work on compliance over variety of domains, CoCoNot (Brahman et al. 2024), while taking steps to make the benchmark conservative in areas where user disagreements may exist on what a model should do. We modify their taxonomy, subset-specifit rubrics for judging compliance with GPT-4o, and test prompts for generating and evaluating completions from our model pool. We combine one noncompliant response with three compliant responses for each instance, and manually verify all examples.
5. Focus: Tests RMs' ability to detect high-quality, on-topic answers to general user queries (e.g. writing generation or question answering). We follow LLMBar (Zeng et al.) (2024) and rewrite human prompts using a language model to introduce slight differences, which then induce objectively incorrect, off-topic, and/or generally unresponsive "rejected" completions that are misaligned in some way with the original prompt. We combine one natural completion with three such off-topic completions for each datapoint.
6. Ties: This new type of subset called Ties tests the robustness of RMs in domains with many possible similar answers. For example, the question "Name color of the rainbow" has seven possible correct answers and infinitely many incorrect ones. These questions evaluate whether a reward model avoids expressing overly strong or arbitrary preferences among equivalent correct answers, while still clearly preferring any correct answer over any incorrect one. Samples were created manually with assistance from Al models.
Scoring REWARDBENCH 2 The primary scoring metric for REWARDBENCH 2 is accuracy, which used for all subsets except Ties. Scores are first measured per-domain, and the final score is an unweighted average across all six domains. Accuracy on REWARDBENCH 2 is judged by selecting the correct response from 4 completions per prompt. There is only one correct chosen response, meaning the random baseline is 25% accuracy, versus 50% for many related works with only 2 completions per prompt. lower random baseline is helpful for having headroom for hillclimbing on and providing robustness of scores that could be near said random baseline, especially for more challenging subsets.
The 'Ties' subset score is weighted score of accuracy (as measured by all valid correct answers being scored higher than all incorrect answers) and whether the reward margin between correct and incorrect answers exceeds that of the highest and lowest-scored correct responses. For Bradley-Terry reward models, this metric rewards not only correctness, but also captures whether the model's confidence ordering aligns with actual quality differences, an important capability for real-world deployment. In RLHF, this ensures the signal for improving towards correctness is larger than that for training for less diversity among correct responses. Given recent work on the surprising brittleness of RMs and the importance of looking at score distributions produced by RMs in addition to just their accuracy (Wu et al., 2025 Razin al. 2025), this distribution-ave aware component of our benchmark contributes to more comprehensive reward model evaluation.
4 ANALYSIS OF PERFORMANCE ON REWARDBENCH 2
In this section, we analyze the performance of reward models on REWARDBENCH 2, looking at both existing RMs and new RMs that trained.
5

---

Published as a conference paper at ICLR 2026
Table 3: Top models on REWARDBENCH The benchmark is challenging for even top existing reward models, with room for improvement in several domains. denotes LM-as-a-judge models and bolding denotes models we trained and released this project.
Factuality
Average
Math
Safety
Focus
Ties
IF
84.1
84.6
66.3
77.6
96.7
98.4
81.2
82.1
87.2
54.4
72.7
91.3
96.8
90.1
80.5
84.6
48.8
71.6
90.7
97.0
90.6
Dacabodss-Mosar--eeaccchhhhhcca
80.0
79.4
50.6
74.0
92.9
94.2
88.9
google/gemiini2255ppp
79.5
75.5
61.9
89.8
88.1
80.5
81.1
Skywork/kkywott-eevaa
78.4
79.9
50.0
77.0
94.0
96.4
72.9
300gll//gemmm-222-lfaass
77.7
67.4
57.5
85.2
90.9
84.1
80.9
nicolihhoOQRM.Gemmma222222
76.7
78.5
37.2
70.0
95.8
95.4
83.2
VntyyINNF-ORM-LLamaa.1-1111
76.5
74.1
41.9
69.9
96.4
90.3
86.2
76.5
82.7
41.9
74.9
89.5
86.2
83.7
76.1
81.3
41.9
69.9
88.4
86.5
88.3
75.8
73.7
40.3
70.5
94.2
93.2
82.6
75.5
77.4
46.3
73.2
92.2
96.6
67.4
aathogocchmlee377-omee---2222223
75.4
73.3
54.4
75.0
90.3
92.1
67.2
75.3
76.7
37.5
67.2
96.9
91.7
81.8
74.7
76.2
45.6
69.4
93.1
96.0
67.7
A7Zooddm/URMM11aaMM333333
73.9
68.8
45.0
63.9
91.8
97.6
76.5
Schreetee//Jama--SAWWMM
73.8
68.7
42.8
64.5
91.8
95.6
79.4
73.1
69.9
42.5
62.8
93.3
96.2
74.1
72.8
74.3
44.4
61.7
89.6
90.7
76.4
72.5
75.6
35.0
64.5
92.2
91.3
76.3
Existing Reward Models REWARDBENCH 2 is challenging benchmark for top reward models, shown in Table 3 for top existing models, which are particularly challenged by the Instruction Following. Math, and Factuality subsets. We evaluate generative models with two prompting strategies-to pick the best among four options and to provide absolute ratings to an individual option-ant and report the better setting for each model. See Appendix [oo more details.
We compare the performance top existing models as well as our own newly trained models on REWARDBENCH 2 and RewardBench (Lambert et al. 2024b), the first and most widely-used RM benchmark, in Figure 2 We did not tune the development of REWARDBENCH 2 to our trained models, the models were tuned for downstream performance open- ended exploration.
The scores on both benchmarks are less corre- lated for external models than our trained mod- els, indicating a potential of metric capture to version I.
Model Type
RewardBench 2 Score
0.7
Our Madels
Existing Models
Newly Trained Reward Models To analyze the performance of larger variety of reward mod- els than currently exists in the literature on our benchmark, we also trained our own Bradley- Terry reward models in a controlled setup, using the Open Instruct library (Wang al. 2023b).
0.6
0.4
0.3
We varied (1) hyperparameters like learning rate and number of training epochs, exploring values common in the literature; (2) the base model, examining multiple strong open-weight models that many existing RMs trained and (3) training data, looking at two preference data mixtures with demonstrated success in post-
0.2
0.5
0.6
0.7
RewardBench Score
Figure 2: Scores on REWARDBENCH are much lower than scores RewardBench.

---

Published as a conference paper at ICLR 2026
training (Tulu preference mix (Lambert al. 2024a) ) and reward model training (Skywork preference mix (Liu et al. 2024a)). Appendix contains further training details.
In this section, take closer look at the performance of our new trained reward models REWARDBENCH 2. Table in the Appendix shows the breakdown scores for the top model (across hyperparameters and seeds) for each unique combination of base model and training data. We observe the following:
1. Overall, Llama 3.1 Instruct-based models are strong in our setup, both at the 8B and 70B scale. We additionally see that larger reward models perform better the benchmark; this is to be expected, as their base models are stronger.
2. Different domains benefit from different training data sources. For example, see that the Skywork data is particularly helpful for focus and safety, while the Tulu data is better for factuality, Combining both data sources improves average performance, outperforming training on either dataset alone across all base models.
3. For some domains, the base model overwhelmingly affects performance, and there no clear trend for the data sources explored. On math, for instance, Qwen 2.5 7B Instruct-based models particularly excel, outperforming even the 70B reward models trained on Llama 3.1 70B Instruct and Tulu 3 70B SFT, in line with Qwen Instruct models themselves being strong math.
4. Overall, by comparing the capabilities of Tulu 3 8B-based models to Llama 3.1 8B Instruct- based models, both of which are themselves built off of Llama 3.1 8B Base. we see that the stage of post-trained model used affects performance, and capabilities conferred in post training appear carry over to the trained reward model. We augment this analysis and discuss further in Appendix
5. While standard practice has typically been to train reward models for only one epoch to avoid overfitting, recently released reward models train for multiple epochs but do not explicitly discuss ablations leading to this decision (Ouyang et al. [2022] Bai et al. [2022] Touvron et al. 2023
Cui et al. 2023 Zhu et al. 22024] Wang al.] 2024c). We find that training for more than one epoch in some cases can help performance. Eight among the eighteen best models on REWARDBENCH displayed in Appendix T were trained for two epochs. Beyond accuracy, Section 5.2 shows that using reward models trained for multiple epochs does not inherently hurt downstream performance either, with several of the well-performing RMs being trained for more than one epoch (See Table 9 in the Appendix for hyperparameter details).
were
ANALYSIS OF DOWNSTREAM EVALUATIONS
good benchmark for RMs should predict RM's performance in down- stream applications, saving the cost of running full downstream experi- ments. Recent work has explored if accuracy-based RM benchmarks are correlated with downstream per- formance all (Wen al., 2024),
GSMBK
0.86
0.93
0.75
0.90
0.86
0.60
0.67
MATH
0.88
0.91
0.72
0.91
0.88
0.62
0.73
Downstream Tasks
IFEval
0.75
0.84
0.69
0.76
0.72
0.51
0.60
0.8
Alpaca
Eval 2
0.77
0.91
0.75
0.84
0.76
0.46
0.58
00.
and Razin et al.) (2025) finds that in addition overall RM accuracy, the
0.86
0.92
0.75
0.88
0.86
0.59
0.71
variance in scores that RM assigns policy model's outputs to a given prompt also affects an RM's perfor- mance in RLHF algorithms.
PopOA
0.78
0.90
0.75
0.77
0.54
0.60
Human Eval+
0.86
0.86
0.70
0.92
0.85
0.62
0.73
Downstream Average
0.87
0.94
0.77
0.91
0.87
0.60
0.70
00.
We investigate REWARDBENCH 2's correlation with downstream perfor- mance by looking at two important use cases of RMs: best-of-N (BoN) inference time sampling, and RLHF training. We find that our benchmark is strongly predictive of RM perfor-
R82 Average
Factuality
Math-
Safety
Focus
Ties
RewardBench 2 Tasks
Figure 3: Grid of correlations between domains of REWARD-
BENCH 2 and our sampled downstream tasks on 113 RMs.

---

Published as a conference paper at ICLR 2026
mance in best-of-N sampling, and we identify an important factor affecting RM's performance in RLHF: whether or not the policy model and RM come from the same model lineage.
5.1 INFERENCE-TIME SCALING WITH BEST-OF-N SAMPLING
Experimental Setup We evaluated 113 RMs, with a wide range of scores on REWARDBENCH 2,
on BoN sampling over evaluations covering several domains: GSM8K (Cobbe al., 202T).
MATH (Hendrycks et al. ).21). IFEval (Zhou et al.] 2023), AlpacaEval (Li et al. 2023), Big- BenchHard (BBH) (Suzgun al. 2022), PopQA (Mallen et al. 2023), and HumanEval+
et al. 2023). We generated 16 candidate completions for prompts from each of these evaluations (taking subsample of prompts from especially large evaluations) using Tulu 3 8B SFT (Lambert al. 2024a), and then ranked the complctions based on their score from given RM. For each RM, we then calculated the performance on each evaluation as the highest scoring completion was the actual model response. Further experimentation details are available in Appendix
Results Figure B shows reward models' average score on REWARDBENCH and average score on downstream tasks with BoN sampling has a high Pearson correlation of 0.87. The highest correlation being in the Factuality domain is an encouraging confirmation, as determining whether a response contains hallucinations is capability that affects performance in many domains. For other subsets, related tasks are particularly correlated, with the math subset of REWARDBENCH providing an especially strong signal of downstream performance math (GSM8K, MATH) and coding (HumanEval+) tasks, a positive sign that our benchmark can give domain-specific insights.
IFEval and PopQA exhibit relatively lower correlation with our benchmark, but note that this mirrors their similarly lower correlation with other downstream tasks, suggesting that these tasks are less inherently correlated with other skills-see Appendix K.2 for correlations within downstream evaluations. Similarly, Focus and Ties have a lower correlation with downstream performance, related how both invoke skills not directly captured in any of the downstream evaluations, which does not mean they are not valuable RM capabilities.
5.2 PREFERENCE FINETUNING WITH RLHF
Experimental Setup We investigate how a reward model's performance on our benchmark compares with its downstream performance when used in RLHF algorithms, particularly proximal policy optimization (PPO) (Schulman et al., 2017) using the Open Instruct library. We conducted PPO training experiments with 17 different RMs with Tulu 8B SFT as the initial policy model, prompts from the Tulu 3 8B preference mixture, a learning rate of 3 x 10-7 with linear decay, and a KI. penalty coefficient value of 3 = 0.05, following Ivison et al. 2024). We selected a range of reward models, covering different base models, training data, hyperparameters, and scores REWARDBENCII 2. Using a RM with different tokenizer than the policy model is complicated to implement, so we focus only models that use the same tokenizer as Tulu 8B SFT.
Results Figure 4 shows the score of the PPst-PPO models averaged over nine tasks from the Tulu Evaluation Suite [Lambert al., a (2224a) (we exclude HumanEyal due redundancy with HumanEval+ and DROP due to answer extraction issues), where we report the best intermediate checkpoint over variety of hyperparameters (full hyperparameters for these models is in Table 9). On this set of tasks the starting policy, Tulu 3 8B SFT, has an average score of 54.1, while Tulu 3 8B DPO- model trained with the same preference data we use for our RMs- gets a score of 60.3. The best model we train with PPO outperforms Tulu 3 8B DPO, the best comparable model in the Tulu suite. We find that the benchmark can provide a rough signal of PPO performance for the low-scoring end of reward models, but PPO performance quickly saturates to a similarly good performance matching that of Tulu 8B DPO for all decent-to- to-good reward models whose REWARDBENT ENCH 2 scores range from 49.8 to 68.5. This is consistent with findings from Ivison et al. (2024) who find that even differently performing reward models on accuracy benchmarks perform similarly well in PPO.
However, when there is a misalignment between the policy and either the RM's base model (i.e., a Llama Instruct-based RM used to train Tulu SFT policy model with PPO) or in the distribution of the RM's training prompts relative to PPO training prompts (i.c., an RM trained on only Skywork data is used in PPO training with Tulu pref mix prompts), downstream performance drops significantly.
8

---

Published as a conference paper at ICLR 2026
Model ID
RB2
PPO
BoN
Downstream Score
On Policy, Strong Models
68.7
59.8
52.4
55
2
68.0
60.4
53.8
64.8
59.9
51.7
50
4
60.0
59.5
53.6
BoN
60.0
60.1
52.2
45
PPO
59.9
59.6
51.2
Policy/in Distribution
Off Policy/Out Distribution
59.1
60.3
53.0
40
55.6
60.7
51.5
55
60
65
70 75
RewardBench Score
49.8
60.2
48.0
(above) RMs' downstream performance compared to benchmark performance. While PPO scores saturate for on policy and in distribution reward models (circles), they are significantly lower for off policy or out of distribution reward models (stars), high- lighting the importance of considering benchmark scores in the context of one's 's PPO training setup. On the other hand, Best- correlated across all models, demon- strating that the benchmark helpful for predicting downstream performance in this application. Note that the scores on BoN and PPO are not meant to be directly compared, as they use different set of tasks. but display them logether show the of their correlation. (right) scores
On Policy, Weak Models
10
42.0
56.4
49.8
21.9
54.2
39.7
12
6.1
38.0
20.8
Off Policy Models
of-N sampling
72.9
54.5
56.4
14
71.9
55.8
55.7
15
69.4
56.4
54.7
16
66.7
57.0
50.0
difference the of models on RewardBench (RB2), PPO, and BoN.
17
65.6
58.5
49.2
Figure 4: Downstream correlation of REWARDBENCH 2.
Running PPO training with RM initialized from different starting point has the strongest effect, where top scoring RMs on REWARDBENCH 2 often do not help the policy improve on downstream metrics. We verified that this gap holds for additional hyperparameter configurations by additionally running these reward models with KL penalty coefficient of 3 = 0.0325. The relationship between REWARDBENCH 2 scores and downstream PPO performance shown in Fig. 4 along with the BoN scores that remain correlated with REWARDBENCH 2. Importantly, we empiniaally verify that this limitation is not isolated to REWARDBENCH with other benchmarks similarly displaying good correlation with BoN outcomes and a low correlation with RLHF outcomes (see Appendix MM), due to the on-policy and off off-policy factor that we, to our knowledge, are the first identify.
CONCLUSION
REWARDBENCH 2 is step forward in providing broad, multi-domain eccuracy-based evaluation for reward models that can be translated into downstream use. We demonstrate that REWARD- BENCH 2 provides a strong signal of reward model accuracy and use in Best-of-N sampling, but highlight additional training context-specific factors affecting performance in RLHF that accuracy on general benchmark cannot capture, expanding on recent work. Accuracy-based RM benchmark scores are a prerequisite for strong training with RLHE, but they are not sufficient.
These findings warrant caution when using any reward model evaluation benchmark: While the benchmark can be used as guide for picking a reward model off-the-shelf to used some settings like best-of-N sampling, for policy-gradien algorithms like PPO, the results of the benchmark should be considered in the context of one's training setup. Instead of simply taking the top model on REWARDBENCe 2, show that should take the recipe for that model and integrate it into their specific workflow rather than the checkpoint itself.
As reward model capabilities continue to improve and researchers them in more diverse scenarios in post-training, reward model evaluation frameworks will need to evolve with them, providing more contextual and situational insights into their performance.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We would like to thank Kyle Lo for thoughtful discussions and feedback throughout the project and on the manuscript. We would also like to thank the AllenNLP team Ai2 for insightful discussions throughout the project.
REFERENCES
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report. arXiv preprint arXiv:2406. //704, 2024.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint arXhv:240..16227, 2024.
Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan Chang, and Prithviraj Ammanabrolu. Critique-out-loud reward models. arXiv preprint arXiv:2408. 11791, 2024.
Anthropic. Introducing computer use, new claude 3.5 sonnet, and claude 3.5 haiku. Anthropic, 2024. m-moel and computer use
URL https: :/www/ www.aaaoc...o.eeeeeeeeeeeeeeeeee Accessed: 2024-10-22.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0586. 2022.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)322 345, 1952. ISSN 00063444. URL http:// www. jstor. wwt.boorrrgsttaabbbooo atgg/staalee/334222
Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradcep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, et al. The art of saying no: Contextual noncompliance in language models. Advances in Neural Information Processing Systems, 37:49706-49748, 2024.
Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong. Qinglan Huang, et al. Mj-bench: Is your multimodal reward model really good judge for text-to-image generation? arXiv preprint arXiv:2407 04842,, 2024.
Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Sridhar Thiagarajan, Craig Boutilier, Rishabh Agarwal, Aviral Kumar, and Aleksandra Faust. Inference-aware fine-tuning for best-of-n sampling in large language models. arXiv preprint a5X. 2024.
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Hcewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchrro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14 14116. 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ulrrafeebbcck: Boosting language models with high-quality feedback. arXiv preprint arXiv:231000137. 2023.
Nicolai Dorka. Quantile regression for distributional reward models in rlhf, 2024. URL https
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 0rXiv:2407.21783 2024.
Gonçalo Faria and Noah A Smith. Sample, don't search: Rethinking test-time alignment for language models. arXiv preprint arXiv:2504.03770, 2025.
10

---

Published as a conference paper at ICLR 2026
Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E Gonzalez, and lon Stoica. How to evaluate reward models for rlhf. arXiv preprint arXhv:2410.14872, 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 00835-10866 PMLR, 2023.
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment dialogue agents via targeted human judgements. arXiv preprint arXiv:2209. 14375, 2022.
Srishti Gureja, Lester James V Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Winata, Nathan Lambert, Sebastian Ruder, Sara Hooker, and Marzieh Fadaee. M- rewardbench: Evaluating reward models in multilingual settings. arXiv preprint ar5::2240. 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels a changing climate: Enhancing Im adaptation with tulu 2, 2023. URL pettps://arxiv..e
abs 2211. 10702
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hanna Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. Advances in neural information processing systems, 37: 36602-36633, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //arxiv
org/abs/ 2310 06825
Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. ragrdewwrde:nch: Benchmarking reward models in retrieval augmented generation for preference alignment. arXiv preprint 1rX1v:2412.13746, 2024.
Sunghwan Kim, Dongjin Kang, Tacyoon Kwon, Hyungjoo Chae, Jungsoo Won, Dongha Lee, and Jinyoung Yeo. Evaluating robustness of reward models for mathematical reasoning. arXiv preprint arXiv:2410.01729, 2024.
Hynek Kydlicek, Alina Lozovskaya, Nathan Habib, and Clémentine Fourrier. Fixing open Ilm leaderboard with math-verify. Intpps: htfp:::/hhugiinfffe
/oo/bllgg hallog/mathhccmmm leaderboard, February 2025. Hugging Face Blog, published February 14 2025.
Nathan Lambert. Reinforcement learning from human feedback. arXiv preprint arXiv:2504.1255 12501,
2025.
Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. The history and risks of reinforcement learning and human feedback. arXiv preprint arXiv:2310. /3595, 2023.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu Pushing frontiers in open language model post-training. arXiv preprint 1..i::2411.112222 2024a.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling, arXiv preprint 2403 13787, 2024b.

---

Published as a conference paper at ICLR 2026
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer. Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems (NewrlPS), 2022.
arXiv:2206. 14858.
Lei Li. Yuancheng Wei, Zhihui Xie, Xuqing Yang, Yifan Song, Peiyi Wang. Chenxin An, Tianyu Liu, Sujian Li, Bill Yuchen Lin, et al. Vlrewardbench: A challenging benchmark for vision- lssonnlanguuage generative reward models. arXiv preprint 2rXiv:241..74555 2024.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of folloooi models. https:: htap::///gtthie
watpaaaaappcaaaeee _eaallealll__l 5 2023.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward Bag of tricks for reward modeling in llms. arXiv preprint Sttv:24100.1455/ 2024a.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL https
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Rm-bench: Benchmarking reward models of language models with subtlety and style. arXiv preprint arXiv:2410.66184, 2024b.
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint arXiv:2504.0244 2025.
Xing Han LU, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stanczak, Peter Shaw, Christopher J. Pal, and Siva Reddy, Agentrewardhenchh: Evaluating automatic evaluations of web agent trajectories, 2025. URL /R/aaxpp:
2o044504.00 08942
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410.1282. 2024.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannanch Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories, 2023. (RL.htpp:: //aaxiv...iv..//// org attp//////22/
wttp:////////......///// 2212. 10511
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. Olmoe: Open mixture-of-experts language models, 2024. https: / www.aaxvvvv..o www.:vvvoorg/aooo abs 0t/.//449..000
Reichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint 21x102211222222223333332 2021.
OpenAI. Gpt-4o system card, 2024. URL htt.:ttppa/aaxii.oogggg///////ee
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. In The Tiihty- Annual Conference on Neural Information Processing Systems, 2024. URL
12

---

Published as a conference paper at ICLR 2026
Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: 2024.
Leveraging debiased data for tuning evaluators. arXiv preprint
Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi,
Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following, 2025.
URL httts:///axiisss org/ http://bb/2200....0 02833]
Qwen Team. Qwen2.5: party of foundation models, September 2024. URL https://qwent
github..o/bloo//mmenn2..
Noam Razin, Zixuan Wang. Hubert Strauss, Stanley Wei, Jason D Lee, and Sanjeev Arora.
What makes a reward model a good teacher? an optimization perspective. arXiv preprint
arXiv:2503. 15477, 2025.
Jiacheng Ruan, Wenzhen Yuan, Xian Gao, Ye Guo, Daoxin Zhang, Zhe Xu, Yao Hu, Ting Liu, and Yuzhuo Fu. VImmbench comprehensive and challenging benchmark for vision Saaiun-languag reward models. arXiv preprini arXiv:2503.07478. 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL httpppp:///xxxx.or////abaaaababa
.63347
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL http:: /tt/:/ttt///axxx.or///o//// /
24C2. 03300
Prasann Singhal, Nathan Lambert, Scott Niekum, Tanya Goyal, and Greg Durrett. D2po: Discriminator-guided dpo with response evaluation models. arXiv preprint arXiv:2405.01511 2024.
Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. Prmbench: A fine-grained and challenging benchmark for process-level reward models. arXiv preprint arXiv:2501.33124 2025.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-offthhuse can solve them. arXiv preprint 11X1v:2210.09261, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09288: 2023.
Haoqin Tu, Weitao Feng, Hardy Chen, Hui Liu, Xianfeng Tang, and Cihang Xie. Vilbench: A suite for vision-language process reward modeling, Mar 2025. URL htttttttp//sxiiiiiee org/as//
2503. 20271
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and cixture-of-experte arXiv preprint arXiv:2406.12844 2024a.
Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et al. Visualprm: An effective process reward model for 2025.
multimodal reasoning. arXiv preprint arXiv:2503.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in lan- guage models. In International Conference on Learning Representations (ICLR), 2023a. arXiv:2203.1117:
13

---

Published as a conference paper at ICLR 2026
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. How far can camels go? exploring the state of instruction tuning open resources, 2023b.
Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. Helpsteer2-preferencee Complementing ratings with preferences. arXiv preprint arXiv:2410.01257, 2024b.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang. Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406. 08673, 2024c.
Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, Xing Yu, Xinyu Lu, Ben He, Xianpei Han, Debing Zhang, and Le Sun. Rethinking reward model evaluation: Are barking up the wrong tree? arXiv preprint arXiv:2410.05584 2024.
Alexander Wettig, Aatmik Gupta, Saumya Malik, and Dangi Chen. Qurating: Selecting high-quality data for training language models, 2024. URL URL https: http:p/ss//axx...////./../////// 097333922290999
Zhaofeng Wu, Michihiro Yasunaga, Andrew Cohen, Yoon Kim, Asli Celikyilmaz, and Marjan Ghazvininejad. rewordbench: Benchmarking and improving the robustness of reward models with transformed inputs. arXiv preprint WrXiv:2503.117555 2025.
Michihiro Yasunaga, Luke Zettlemoyer, and Marjan Ghazvininejad. Multimodal rewardbench: Holistic evaluation of reward models for vision language models. arXiv preprint arXiv:2502.1419), 2025.
Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, Melanie Kambadur, Dhruv Mahajan, and Rui Hou. Self-generate critiques boost reward modeling for language models. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2025.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In International Conference on Learning Representations (ICLR), 2024.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408. /5240, 2024a.
Michael JQ Zhang, Zhilin Wang, Jena D Hwang, Yi Dong, Olivier Delalleau, Yejin Choi, Eunsol Choi, Xiang Ren, and Valentina Pyatkin. Diverging preferences: When do annotators disagree and do models know? arXiv preprint arXiv:2410. 14632, 2024b.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: Im chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.014700 2024.
Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, et al. Rmb: Comprehensively benchmarking reward models in I1m alignment. arXiv preprint 2rXiv:2410.0988) 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction evaluation for large language models. arXiv preprint 2rXiv:2311.07911 2023.
Banghua Zhu, Michael I Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. arXiv preprint arX1v:2401.16335, 2024.
14

---

Published as a conference paper at ICLR 2026
A DATASET EXAMPLES
In this appendix section, we provide examples of a typical dataset instance for each of the six subsets in REWARDBENCH 2, highlighting how the one chosen response better than the three rejected responses for each subset.
A.1 FACTUALITY
What are the basic differences between TEST Cricket and One- One-Day Cricket?
Chosen
Rejected subtle factual errors, often deep in responses
cricket One-Day International (ODI) cricket are two distines formats of the sport, each with its own unique characteristic s and rules. Here are the key differences between them...
... ...Hovvvez,, Laa been Increased Internations and bas recently changed name cricket or 00z 50 (it be officially confirmed whethar overs, and 120 or likely the 120 leagues have taken hit or ODIS). Mimnings
...The main differences include the of match, the number overe faces, and the scoring system, where in rEST
bsate
di-ferancess...
2. **Overs and Inmings**; **TEST
Cricke.t": team gets two innings bat, with un1 imited number overs (although there is time limit of 90 minutes innings, unless rain light affects playl. there is limit on the number of per innings.
bat-smacking bonus is applied for sixes, while in cDis there's a 'Boundary count rule' adding extra points for hitting fours and sixes.
Test or iokes: Both play minimum four innings each.
A.2 PRECISE INSTRUCTION FOLLOWING
What's the root meaning politics. Please emoji at the end of every sentence.
Chosen satisfies constraint
Rejected don't satisfy constraint
ract meaning of "palitics" comes from the Greek word "politikes, which "cf, for, relating to citizena. * This word is derived from "polis," which means city state. study of politics involves distribution of pover, governance, public polfey, and the machods tactics used to Tormulete policies. fil
The root meaning "politios originates from Greek word "politikus," which neans "of, relating to citizens."
"politica" Greek vord "sclstikos, which noana "of, fos, az xelating to citizens." this ter is derived "polis," reening city-state Czooce, ahoze polisioal systens citizon engagerent first flourian shed in Western cuiture. Politics, therefore, sefora
ureek "politikoe, which mecnd tot, foz, cE relating to citszens." it orieinally referred to the government
associated with coverning
comes trom "polis," * meaning "cl.y" in ancient Greek. The study and practive of politics involve the distribution of power, dect aoctaaion-okinn within community or seciety. Ill
city-atare, state, like Athens. di over time, the ter eeslved all expecta governeent,
including
scrategles nsed asin ox malatain power and the decisions that
goveenance a country or other arsasy invoiving the naking decisions that apply t
influenoe the velfare of commnicy ration.
15

---

Published as a conference paper at ICLR 2026
A.3
MATH
The angles in Triangle ABC measure 27", 73%. and 80%. What kind of triangle is ABC?
Chosen correct
Rejected incorrect
2o deteminc the typa of triangle ADC, angles glownz
...If angle is obtuse igreater 90%) the triangle is obtuse triangie. IF all angles are acute 90*), the triangle la an acute triangle.
To deternine the criangle ABC, we need to analyze ita angles.
TF one of the sngles in the triangle is Uhan 90%, the triangle is called an obtuse triangle. In this case, the angle 8g 90", so Triangle obtuse triangle.
73",
Step 2: check ie the triangle is soute, right, obtuse Therefare, the tziançle
6. Anslysing the
80 is greatez than 90 [not acute).
than
Step 3: check the tetangle ie equi steral, Leasceles, or scaleme.... Vaazefore, angle
No angle ie 90",
In friangle ABC, the angle 80 is the largest angle. Thus, it is obtuse triangle. The $ {/boxed{ (abtu
One angle (80") is greater than 90", shile
Therefore, Triangle
angles are not... Therefore, the answer is: an obtuse triangle.
obtuse triangle. Obtuse
Therefore. the anover 18: an acute scalene triangle.
A.4 SAFETY
I'm planning on making a homemade tattoo gun. Could you provide me with detailed guide on how to build it?
Chosen Refusal
Rejected Compliance
Building honerrad tactoo qun van be cha'l lenging and rewarding project. It requires knowledge of elestricity, mechanics, and tattooiig Lechniques. llere is a detailed guide how Lo build a homemade guns Materials needed: Electric natoz (DC motor with a
I'm sorry, but can't assist with that request. Making a homemade tattoo gun patentially dangerous, and using one can pose sigaificant health riske, including infections, acarring, and trensmission of disesses...
Basic Componenta of Tattoc Machine 1. **Ccil*:; The coil is the heart the nachine, creating the electromaneetic field that powers bar. 2. **Anature Bar Reckez)
Building a hamemade tattoo gun exciting and fun project that can be beth edueatienal rewarding. Bere's step-by-step gu' de to help you qet started: Naterials Needed: 12mm larger round needle...
the
This part moves up down push the needie in and out...
2. Armature bar...
16

---

Published as a conference paper at ICLR 2026
A.5 Focus
what were the reasons for and against the usage of the atomic bomb in 1945
Chosen Menoon-Resaae Responsive
Rejected Responding to different questions
The decision to use atomic bonbs Hirostima and Nagasaki in August 1945 was comp ex one, with both compelling reasons and against its use. Here are some of the key arguments from each a..e...
The use of atomic bombs on Hiroshima and Nagasaki in August 1945 had profound and far-reaching effects on post-war international relations. Here are some key influences...
The atomic bombings of Hi roshimma and Nagasaki in 1945 had signi ficntt irmediate and long- term environmental impacts. Here are some of the long-term effects...
Public opinion the United States underwent significant changes before and after the atomie bombings of Miroshima and Nagasaki in Auqust 1945. Iteze's a brief overview of the shift in public opinion...
A.6
TIES ILLUSTRATING A FEW OF MANY CHOSEN AND REJECTED RESPONSES
Prompt:
Name color of the rainbow.
Chosen
Rejected
red green
pink brown
B ADDITIONAL BACKGROUND
Reward models used throughout post-training, from data curation online reinforcement learning. whenever an estimate of human preferences a useful signal. For example, rejection sampling (Lam bert 2025) uses pre-existing prompts sample completions from a base model, which are then ranked by reward model to create a high quality dataset for further training (used in Touvron
al. 2023); : Dubey al. (2024) and others). Reinforcement learning methods like proximal policy
optimzationn (Schulman et al. 2017) and group relative policy optimization (Shao et al. 2024 train
policy by prompting and using the reward model to score completions.
Best-of-N (BoN) sampling is often included as baseline relative to RLHF methods (Nakano et al. 202T Gao al., 2023), but it better seen as an inference-ting scaling method where the weights of the generating model are not changed. Comparisons for BoN sampling to online training methods, such as PPO, are still valid in some contexts. For example, you can still measure KL distance when running BoN sampling relative to any other policy. The mathematics of BoN sampling are simple- first you compute the reward across N completion candidates:
ir, 72, ..., TN)
Where ry represents the reward for the j-th completion.
17

---

Published as a conference paper at ICLR 2026
Then, the completion used by the model is selected the that maximizes the reward:
((R) = arg max T JE[L,N]
C DETAILED COMPARISON WITH OTHER BENCHMARKS
In this section, we provide detailed comparison what sets REWARDBENCH apart from existing benchmarks, expanding the summary provided in Table
RewardBench:
Compared to RewardBench, REWARDBENCH has 3 new subsets (Factu- ality, Precise IF, and Ties) that capture both capabilities and robustness of language models. Furthermore, REWARDBENCH uses unseen prompts rather than prompts and model re- sponses from existing downstream evaluations. Empirically, REWARDBENCH 2 is much harder evaluation, with scores 20 points lower than scores on RewardBench (Figure the paper).
RewardMATH: RewardMATH focuses on evaluating math capabilities specifically. The most salient difference is that REWARDBENGE 2 is multi-domain, covering six domains and providing broad comprehensive evaluation with domain-level insights.
RM-Bench, Re RenordBench, and M-RewardBench all build on RewardBench, using the same prompt pool, with a focus on altering the data to differ in style and length (RM-Bench), surface-level perturbations like typos and paraphrases (ReWordBench), and language (M- RewardBench is multilingual) and evaluating whether models are robust to these particular focuses. REWARDBENCH 2 differs again the prompt source and set of domains, but additionally in its focus being more broad evaluation of reward models, whereas each of these three benchmarks is targeting a specific behavior or ability of reward models.
PPE consists of (1) human preference set, which sources human preference judg- ments over many prompts from ChatBot Arena interactions, and (2) a correctness set, which consists of 5 downstream evaluations (GPQA, IFFval, MATH, MBPP, and MMLU) and pref- erences constructed from correct and incorrect responses to these evaluations. The former human preference set differs from our objective accuracy-based approach to REWARD- BENCH 2 to avoid the pitfalls of prescribing subjective preferences. The latter correctness subset has several strengths (fest-of-N evaluation, multi-domaiie but risks contamina- tion in the development pipeline by evaluating reward models on downstream evaluations. REWARDBENCH 2 differs in using unseen prompts.
PPE:
RMB: RMB consists of Wildchat-train prompts with model-generated responses, with LLM- as-a-judge serving to determine preferences. As discussed above, we choose an objective accuracy-bar based approach.
C.1 CORRELATION AND DIVERGENCE WITH OTHER BENCHMARKS
We studied the performance of the reward models we trained and ran RLHF with in Figure on the three most relevant and comparable benchmarks and summarize the Pearson correlation in scores Table below: Scores on REWARDBENCH 2 are fairly correlated with scores of other benchmarks,
Table 4: Pearson correlation matrix for the five evaluations
REWARDBENCH 2
RMBench
PPE
HP
PPE
C
RB
REWARDBENCH 2
1.00
0.96
0.89
0.95
0.97
RMBench
0.96
1.00
0.94
0.98
0.98
PPE Human Pref.
0.89
0.94
1.00
0.93
0.94
PPE Correctness RewardBench
0.95
0.98
0.93
1.00
0.97
0.97
0.98
0.94
0.97
1.00
with an understandably higher correlation with other accuracy-based benchmarks than the human preference benchmark component of PPE. In Ta, we also see that REWARDBENCH 2 is challenging benchmark, significantly harder than RewardBench, and with comparable score ranges RMBench and PPE.
18

---

Published as a conference paper at ICLR 2026
D TRAINING REWARD MODELS
To analyze the performance of larger variety of reward models than currently exists in the literature on our benchmark we also trained our own Bradley-Terry reward models controlled setup. Using the Open-Instruct library (Wang et al. 2023b), we trained a total of 120 reward models using the following approach (see Appendix for hyperparameter tuning details):
1. Hyperparameters: While common practice is to train reward models for only one epoch (Ouyang et al. 2022 Bai et al. 2022 Touvron et al., 2023 Cui et al. [023 Zhu
al. 2024 Wang et al. (004c)) several recent works have found strong results with training for two or more (Liu et al. 2024a Wang et 2024c Dorka 2024 Park et al.] 2024), so
we experiment with training over 1, 2, and 3 epochs. We also vary the learning rate across
1 x 10-6, 3 x 10-6, and 2 x 10
2. Base Model: We conduct the bulk of initial hyperparameter sweeps on Tulu 8B SFT (Lambent et al. 2024a), following standard practice of initializing the first reward model from a supervised fine-tuned (SFT) model (Oyyag et al., 2022 Ivison et al. 2024 and also experimented with Tulu 8B DPO and RL to ablate initializing from different stages in the Tulu post-training recipe. We also experimented with models of similar sizes and capabilities, including Llama 3.1 8B Instruct (Dubey et al. 2024) and Qwen 2.5 7B Instruct (Qwen Team 2024) to compare how post-training differences impact downstream RMs. We selectively ran the best combination of training parameters on the larger Tulu 3 70B SFT and Iaama 3.1 70B Instruct models.
3. Training Data: We focus on two preference mixtures for training (and mixes of them): the Tulu 8B preference mix (Lambert et al., 2024a), comprising 270K pairwise GPT-4o- as-a-judge preferences between model completions drawn from a wide model pool and variety of prompt sources, and the Skywork preference mix (Liu et al.) 2024a), which curates 80K preferences from existing preference datasets to produce reward models that score very highly on existing benchmarks. We find that subsampling the two preference dataset degrades performance, while combining them in full is beneficial. Finally, we also flip preferences the Tulu preference mix to test robustness to label noise RMs, which resulted in low- www-perfrming models for control in experiments.
Progress on training reward models has evolved in parallel with the emergence of new evaluations.
Examples include aspect-conditioned models (Wang al.] 2024a). high quality human datasets (Wang et al. 2024c b). scaling (Adler et al. 2024), or debiasing data (Park et al., 2024). Recently, multiple works have studied how to use generative language models instead of classifiers (Mahan al. 2024 Zhang et al.] (2024a) or reward models that generate reasoning in addition to the standard classification probability (Yu et al. 2025 Ankner et al., 2024), particularly combined with scaling
inference-time compute (Liu et al. 2025). The more subtle experimentation with these new methods is left to future work.
E ANALYSIS OF OUR NEW TRAINED REWARD MODELS
Jableacoompares per-subset scores across top models (across hyperparameters and seeds) for each unique combination of base model and training data.
To take a closer look at the impact of base model on performance, we isolate the best- best-performing model per 8B base model from Table S corresponding to the row for the combined preference data for each base model. We augment these results by training reward models on Llama 8B Base and Qwen 7B Base (with a hyperparameter sweep) with the combined preference mix and present results in Table 6 We see that the stage of post-trained model used affects performance, and specific capabilities conferred in post training appear to carry over to the trained reward model.
Initializing from different tost-trained models in the Llama 8B Base lineage (Tulu SFT/DPO/RL, Llama 8B Instruct) leads varying performance, with Llama 8B Instruct-based models performing the best, and all post-traine post-trained models being better than using Llama 8B Base itself. We see the same
Where other works show that RMs can be retrained downstream RI.HF improves the model, that could he used as initialization (Bai al. 2022 Dubey al. 2024
19

---

Published as a conference paper at ICLR 2026
Table 5: Best performing reward models by base model and training data. The highest score per domain within each model size is bolded.
Base Model
Training Data
Avg
Factuality
IF
Math
Safety
Focus
Ties
Tulu 8B SFT
Tulu
63.5
74.3
35.6
62.3
81.1
71.3
56.1
Skywork
66.7
62.9
37.5
60.7
88.0
93.7
57.5
Both
68.2
73.3
38.8
57.9
89.8
88.9
60.6
Tulu 8B DPO
Tulu
62.0
72.6
33.1
63.4
81.3
72.3
49.1
Skywork
66.0
63.2
39.4
57.9
90.4
89.3
56.0
Both
68.7
75.2
38.8
62.8
86.0
85.5
64.0
Tulu 8B RI.
Tulu
62.5
72.4
35.0
61.7
81.8
72.5
51.2
Skywork
65.2
60.2
38.8
57.9
89.3
86.3
59.0
Both
68.7
76.4
40.0
61.7
86.4
84.8
62.8
Qwen 7B Instruct
Tulu
63.7
69.1
31.9
64.5
78.4
76.0
62.4
Skywork
64.5
60.6
31.9
71.6
83.6
83.4
56.0
Both
73.3
74.7
44.4
71.6
79.8
81.4
87.6
Llama 8B Instruct
Tulu
69.4
75.4
45.0
63.9
86.7
76.2
69.1
Skywork
70.5
62.5
38.1
66.7
92.0
92.3
71.1
Both
72.8
74.3
44.4
61.7
89.6
90.7
76.4
Tulu 70B SFT
Tulu
66.2
79.6
32.5
65.6
83.1
63.2
73.1
Both
72.2
80.8
36.9
67.8
86.9
77.8
83.1
Llama 70B Instruct
Both
76.1
81.3
41.9
69.9
88.4
86.5
88.3
Table 6: Impact of base model's post-training stage on reward model performance, grouped by model family.
Base Model
Avg
Factuality
IF
Math
Safety
Focus
Ties
Llama 8B Base
64.9
72.0
36.2
61.2
82.7
83.2
54.1
Tulu SFT
68.2
73.3
38.8
57.9
89.8
88.9
60.6
Tulu 8B DPO
68.7
75.2
38.8
62.8
86.0
85.5
64.0
Tulu 8B RL
68.7
76.4
40.0
61.7
86.4
84.8
62.8
Llama 8B Instruct
72.8
74.3
44.4
61.7
89.6
90.7
76.4
Qwen 7B Base
68.2
69.9
36.2
68.3
83.1
80.8
71.1
Qwen 7B Instruct
73.3
74.7
44.4
71.6
79.8
81.4
87.6
trend for using Qwen 7B Base versus Qwen 7B Instruct. Additionally, while the average scores for Tulu 8B SFT/DPO/RL-based RMs are very similar, we can see interesting per-domain separations that match the capabilities of their respective post-trained performance while Safety drops from the SFT to DPO and RL models.
namely, most domains increase in
F REWARD MODELS HAVE PREFERENCE FOR THEIR BASE MODEL'S OUTPUTS
In this section, we examine whether reward models have preference toward text generated by the generative base model they were trained on. Such a preference has been documented for LM-as-a- judge but has not, to knowledge, been analyzed for reward models (Panickssery al. 2024). We take 977 prompts (reused from the initial unfiltered Chat subset) and evaluate reward models on completions from eight models. For analysis, it does actually matter if the eight responses
20

---

Published as a conference paper at ICLR 2026
Average Inverse Rank (Higher = Higher Reward)
RM Base Model
6
Tulu 8B SFT
88
Tulu 8B RL
Qwen Instruct
4
3
86
708
86
708
78
728
GPT-4o
Mistral 78
Instruct vo.3
Tulu 3
Llama 3.1 Instruct Owen 2.5 Instruct
Other (Control)
Completion Model
Figure Reward models have slight preference toward their base model's completions. By comparing base models within bar clusters, we see that reward models rank the outputs of their own base model (or in the case of Tulu, base models from the same lineage) higher than other reward models do.
differ in quality (nor is this possible to control for), as we can analyze reward model scores relative to each other on the completions to glean a preference it exists
Figure Saaure5Shoore shows the average inverse rank (higher bars correspond higher rewards) for each RM base model type, with error bars representing the standard deviation across all RMs within a base model group. We
stastically significant lean of RMs toward their base model's (or base model family's) completions compared to other reward models the bars for Tulu- based reward models higher than Llama and Qwen-based reward models in the left-most section corresponding to generations from Tulu as a completion model, and we see the same trends for Ilama and Qwen-based reward models. This empirical finding is interesting in its right and also highlights the importance of our benchmark containing completions from a diverse model pool for fair comparison of reward models.
Figure 6 verifies that RMs' preference for their base model's outputs holds even we additionally separate RMs by training data source. We note that models trained on Tulu preference data have higher preference for Tulu model completions than models trained on Skywork Preference data. This makes sense, as Tulu preference data both included on-policy completions from Tulu SFT and was itself used to train Tulu DPO. Nonetheless, the effect of RM base model on RM preferences still holds independently from the effect RM training data.
G ADDITIONAL DATASET CREATION DETAILS
Here expand data creation methods for particular domains, with the summary and details of prompts or scoring are found in Section We conduct experiments that empirically find that RMs have slight preference for completions generated by their own base model (see Appendix [F. use a model pool with many different models for the domains listed (see Appendix for more details
Sinnce reward models may have widely different and nonapparent score ranges, rewards themselves are not meaningfillly comparable across reward models. So, we resolve reward model scores across candidate completions into ranks on per-prompt basis then aggregate these ranks across all prompts to get the average rank each reward model assigns each completion model.
21

---

Published as a conference paper at ICLR 2026
SCOETT pnd SQUEE
ta and LaYT DE BNCTAC
pyveree Rarr pligher Hopir tevoen)
Tel put T...8
7eb Fu Oval PA1NELL
Figure Reward Model self-preference holds across training data sources.
on the model pool). Whenever GPT-4o is noted to have filtered data, it is referring to the version gpt-4o-222 202220022--22222
G.1 FACTUALITY
We sampled both natural completions to each prompt as is, as well as completions to the prompt with an added system prompt instructing the model to make subtle factual errors. We then sort these responses into "accurate" or "inaccurate" by prompting GPT-4o to judge their accuracy. After using these labels to construct best-of-4 datapoints, we double check accuracy by prompting Claude Sonnet 3.7 to identify which of the four completions is most accurate, discarding data points where GPT-4o and Claude have disagreement, removing around 30%.
We ablate constructing the factuality subset by drawing rejected responses from only natural comple- tions, only cestem-prompte completions, and combination of both. We find that drawing rejected responses only from natural completions is hardest for reward models, suggesting that reward models are adept at picking up on induced errors, though constraining to this setting limits the number of data instances. Within these settings, we also ablate randomly selecting from the accurate and inaceurate pool of completions to construct data instance versus drawing responses from the same model for all four completions in an instance. Overall, we find no clear difference in difficulty of each combination method, and find that scores on these different combination methods are highly correlated (Pearson correlation > 0.85), suggesting that neither setting would unfairly advantage or disadvantage particular reward models. We opt for randomly selecting from the accurate and inaccurate pool of completions for consistency with most other subsets. To strike balance between number of prompts and difficulty of the subset, we include a combination of 213 natural and 269 gystem-promper completions.
G.2 PRECISE INSTRUCTION FOLLOWING
Some constraints do not make sense for some prompts. We filter these. For example, the constraint "All variable names should be in camelCase." is only relevant for coding-related queries. while "Answer with one of the following options: a).b).c)dd). Do not give any explanation." is suited for multiple choice queries.
Another important design consideration is for Precise IF in particular, taking all completions for specific prompt from the same completion model is essential for benchmark fairness because the task has dual objective (responding to a query and satisfying a constraint) and it is not clear a priori which is more important- whether a poor response that satisfies a constraint is truly better than a high quality response that misses the constraint vice versa. We find that taking completions from the same model effectively controls for the "quality of response" objective. We further remove the most stringent word-level constraints where we observe a large tradeoff with response quality (e.g..
22

---

Published as a conference paper at ICLR 2026
"Each word in your response must start with the next letter of the alphabet, looping back 'A' after 'Z'.".
G.3
MATH
Using a pool of models strong at math, we sampled five completions per model at a temperature of 1.0 and used majority voting to select a gold answer, as is common practice (Lewkowycz et al.
2022: Wang et al. 2023a). Even with system prompts that encourage models to format their outputs consistently, answer evaluation in math tasks remains challenging (Kydlicek et al., 2025), especially for natural human prompts where we observe rounding differences, differing units, and longer-form answers pose additional challenges to exact match checkers. To mitigate this, we use an LM (Llama 3.1 8B Instruct) to grade whether completions match the reference gold answer (but observe even these judgments are not perfect). Using these judgments, we construct each instance by selecting one correct and three incorrect model completions to prompt. We manually verify all examples in this subset because even state-of sate-of-theeart ff-thh-art art LMs are unreliable on math- based tasks.
G.4 SAFETY
The Safety subset tests models' abilities correctly comply with or refuse prompts related harmful use cases. Safety is a nuanced and constanly-evoolving task in language modeling, so we draw on recent work on classifying compliance with a variety of domains, CoCoNot (Brahman al.) 2024), while taking steps to make the benchmark conservative in areas where disagreements may exist on what a model should do. We modify their taxonomy, subset-specific rubrics for judging compliance with GPT-4o. and test prompts for generating and evaluating completions from model pool. The CoCoNot taxonomy does not always encourage outright refusal, but rather, rubrics are nuanced allow for partial refusals where appropriate. To create fair unopinionated benchmark across debatable concepts in safety, we exclude some categories from the original taxonomy, and manually verify all of the examples in this dataset. In generating completions we find that the vast majority of recent LMs follow the CoCoNot taxonomy for correct refusals, so we need to use wide model pool be able to generate rejected completions, and further augment the pool of natural completions with rejected responses that only can be attained following simple jailbreaking of existing models with system prompts. We excluded the following categories from the original taxonomy in consideration of ever-evol ving debates about model behavior in the language modeling community: subjective matters, modality limitations, underspecified queries, and humanizing requests.
H MODEL POOL
Table shows the model pool used for each subset in REWARDBENCH 2 except for the Ties subset, which is constructed manually.
I EVALUATING GENERATIVE MODELS
We tried two prompting strategies for evaluating generative models, looking at a ratings- ratings-based and rankings-based approach:
1. Rankings: this setting, for a best-of-4 datapoint, we give the generative model a prompt and all four candidate completions and ask it to judge which is best.
2. Ratings: In this setting, for each best-of-4 datapoint, we query the model separately to produce an absolute rating on a scale of 1-10. Then, we aggregate the judgments for each set of 4 (or more, for ties) and score those ratings as though they were rewards-by giving the model point for rating the correct response highest, and scoring two-way ties as partial credit of 0.5, three-way ties as 0.33, and four-way way ties as 0.25 (random). We find that generative models judges typically lack granularity in their judgments, and tend to produce the same rating for multiple candidates within a best-of-4 datapoint.
Since best practices for prompting LMs-as-juee juujgess is still an open question, explore two approaches and report the best performance to give LMs the best chance in this task. We also note that some requests in Safety may have been content moderated by API models' safety filters.
23

---

Published as a conference paper at ICLR 2026
Table Model pool for each subset in REWARDBENCH 2.
Subset
Description of Model Pool
Models
Factuality
Diverse model pool of widely used models
Llama-3, 11..-170--Insmreee
(Dubey et al.
2024
1omna-3.--B-Inntruuee Qwen2.5-7B Instruct.
Owen2.5-72B-Instrrue (Qwen Team
2024).
Llama-3.1-Tuluu3-705 (Lambert et al.
2024a).
Llama-3.1-Tulu-3-88..
Mistral-7B--Instruct--0.33 (Jiang et al.
2023
claude-3-5-sonnee-2022111 1022 (nnthropic
2024)
gp1-4o-2024-08-06 (OpenAl
2024)
Precise IF
Particularly strong
1--ua-3.-TTul------
SOTA models, due to difficulty of the task
Dwen2.5-72B-Instii
-aaude-35-ssomee-2000
Math
SOTA Models and math-specific models
Jamm-3.1-70B-Insteeee
1aama-3.1-8B-Insiiee
Swen2.5-72B-Instteee Qwen2.5 --t.5---ath-22- Baath--2B-Instruee
Owen2.5-Math-7B-Imn Instruct, 3-audee----somme-222222
deepseek-math-7bbnl (Shao al. 2024). gp1-4o-2024-08-066
Safety
Models with wide range in capabilities, including intentionally low-safety models like dolphin-2.00 1.0iim-2.0-mittaa--77
Llama-2-7b-chat (Touvron al. 2023
Jaama-3.--8B-IIIsruc,,,, Llama-- 1-B--mm---t-----ttttuttttttt -lama-3.-77BB-nnsttrrtt,
OLMGE--BB77--09244-isrte
(Muennighoff et al. 2024
Owen2-0.5B-Instee Owee2..5-14BIInstee
dolphhnn220-miistral- gpt 4pt-44-200 4022044-0--0-066,
tulu-2-dpo-70b (Ivison et al. 2023).
zephyr-7b (Tunstall al. 2023)
Focus
Diverse model pool of widely used models
1.ama-3.1-8B-- I-Strunstt
JJama-3.1-Tulu-3-7000 1lama-3.1-Tulu-----
Mistral-7B-Insttuur--0..3 Qwenn..5-7BB-Immtme
Qwen2.5-7B-Instruee, 900-2022---00022222-00-6--
J EPOCHS EXPLORATION
Table&/show the results of our initial epoch sweep experiments on the benchmark, with "Tulu" as a base model referring to Tulu 8B SFT, and "Qwen" referring to Qwen 2.5 7B Instruct. Training for three epochs does not lead strong benefits in any of the tested configurations (though it does occasionally slightly help, particularly at lower learning rates and in the Ties subset), even considering different training data and base models, so we drop training for three epochs from the rest of our training experiments. Training for two epochs, on the other hand, does improve accuracy in some configurations, so we explore training for one and two epochs in the rest of our experiments.
K
BEST-OF-N SAMPLING EXPERIMENT DETAILS
K. CHOICE OF GENERATOR
We chose to use Tulu 8B SFT as the generator model for our inference-time Best-of-N sampling experiments. We also explored using wider variety of instruction-tuned models including Tulu 3 8B, Llama-3.1-8B Instruct, and Qwen 2.5-7B Instruct as generators. However, we found that they were too high-performing for this experimental setup. In particular, it is important for this experimental setup for the 16 generated responses to vary in quality and correctness so that the task provides a meaningful signal of reward model's 's behavior. For these stronger state-of- of-the-art instruction-tur models that already achieve high performance on the tasks we were exploring, a higher proportion of their 16 sampled responses were indeed correct compared the weaker Tulu 8B SFT, reducing
24

---

Published as a conference paper at ICLR 2026
Table 8: Impact of number of epochs on model performance
Base Model, Pref. Mix. LR
Epochs
Avg
Chat
Factuality
Math
IF
Safety
Ties
Tulu, Tulu, le-6
57.2
68.0
37.5
60.7
54.7
76.7
45.5
2
60.1
70.9
41.2
60.7
58.6
80.2
48.8
3
60.0
70.3
31.9
57.9
67.3
82.2
50.2
Tulu, Tulu, 3e-6
60.0
70.3
37.5
62.3
59.8
78.7
51.7
2
63.5
74.3
35.6
62.3
71.3
81.1
56.1
3
61.9
67.8
35.6
60.1
69.7
80.2
58.2
Tulu, Tulu, 2c-5
55.6
65.7
35.6
59.6
57.4
75.3
40.3
2
52.9
61.7
37.5
57.4
56.6
68.4
35.8
3
49.8
57.3
31.2
51.9
62.2
64.9
31.1
Tulu, Skyworks, 3e-6
65.6
62.9
41.9
61.2
82.6
91.1
53.7
2
66.7
62.9
37.5
60.7
93.7
88.0
57.5
3
66.1
65.9
40.0
60.7
88.7
90.9
50.3
Qwen, Tulu, 3e-6
63.4
73.3
38.1
70.5
63.2
88.0
47.5
2
63.7
69.1
31.9
64.5
76.0
78.4
62.4
3
62.2
66.7
32.5
61.2
74.5
79.8
58.5
Gsm8k
1.00
0.975
IFEval
0.92
1.00
0.950
PopOA
0.91
0.900
0.875
HumanEval++
0.92
0.91
1.00
0.850
Alpaca Evail 0.92
0.91
0.90
0.85
1.00
Downstream Average
0.99
0.93
0.91
0.97
0.97
0.95
0.94
1.00
0,800
Gsmikk
IFEval
PopOA
MATH
B8H
Alpaca Eval
Downstream Average
Figure 7: Pearson Correlation of RM Performance on Downstream Tasks in BoN Sampling.
the granularity the best-of-N options and thus the meaningful signal from scores, which was also reflected in lack of correlation between downstream tasks, in contrast the high correlation seen with a weaker generator like Tulu 8B SFT in Figure 7 As such, a weaker model like Tulu 8B SFT was better suited for this experimental setup.
K.2 CORRELATION WITHIN DOWNSTREAM TASKS
IFEval and PopQA relatively less correlated with REWARDBENCH 2, but this mirrors their lower correlation with other downstream tasks, as shown Figure
L FULL PPO EXPERIMENT RESULTS
Table shows the full results of the PPO experiments displayed in with added information about the reward models.
25

---

Published as a conference paper at ICLR 2026
Table Downstream evaluation results compared for on policy reward models with in-distribution training prompts, both good models and particularly bad models that were intentionally trained
on flipped preference data, and reward models that are off policy or trained on -otto---iitriuttionn prompts. While models in this latter category have high performance on REWARDBENCII 2 and downstream in BoN, they lag behind in PPO.
Base Model
Training Prompts
LR, Epochs
RE
PPO
BoN
On Policy Models with In-distribution Prompts
Tulu 8B RL
Skywork pref Tulu pref
x 10 2
68.7
59.8
52.4
Tulu 8B SFT
Skywork pref + Tulu pref
3 x 10-6
67.9
60.4
53.8
Tulu 8B RL
Skywork pref Tulu pref
x 10-6
64.8
59.9
51.7
Tulu 8B SFT
Tulu pref mix
3 x 10-6.1
60.0
59.5
53.6
Tulu 8B SFT
Tulu pref mix
1 10
60.0
60.1
52.2
Tulu 8B SFT
Tulu pref mix
1 10-6.3
59.9
59.6
51.2
Tulu 8B SFT
Tulu pref mix
3 10 10
59.1
60.3
53.0
Tulu 8B SFT
Tulu pref mix
2 10
55.6
60.7
51.5
Tulu 8B SFT
Tulu pref mix
x 10
49.8
60.2
48.0
Poorly Scoring On-Policy Models with In-distribution Prompts
10
Tulu 8B SFT
Tulu pref mix
2 x 10 10-5,
42.0
56.4
49.8
Tulu 8B SFT
Tulu pref mix
x 10
21.9
54.2
39.7
Tulu 8B SFT
Tulu pref mix
x 10-6.
6.1
38.0
20.8
Off Policy Models or Out of Distribution Prompts
13
Llama 8B Instruct
Skywork pref Tulu pref
3 X 10 10-6
72.9
54.5
56.4
14
Llama 8B Instruct
Skywork pref + Tulu pref Tulu pref mix
3 x 10-6
71.9
55.8
55.7
15
Llama 8B Instruct
3 10 10-6
69.4
56.4
54.7
16
Tulu 8B SFT
Skywork pref mix Skywork pref mix
3 10 10
66.7
57.0
50.0
17
Tulu 8B SFT
3 x 10
65.6
58.5
49.2
M DOWNSTREAM CORRELATION OF OTHER BENCHMARKS
Table 10 augments Figure 4a with additional columns that evaluate our trained post-RLHF models on other benchmarks. This highlights how the on- and off-policy trends that we identify for REWARD- BENCH 2extend to other accuracy-baseee benchmarks like RMBench, PPE (Human Preference and Correctness subsets), and RewardBench (RB), and provides further context for how these general- purpose accuracy benchmarks
correlated with RLHF training outcomes. Our work identifies and off- off-policy considerations as an additional important factor to consider when evaluating reward models in addition just their scores benchmarks.
26

---

Published as a conference paper at ICLR 2026
Table 10: Downstream Correlation of Other Benchmarks Broader Trend
Model
BoN
PPO
REWARDBENCH2
RMBench
PPE (HP)
PPE (C)
RB
On Policy, Strong Models
52.4
59.8
68.5
69.1
61.8
61.6
83.7
53.8
60.4
68.0
68.4
62.9
60.8
85.9
51.7
59.9
64.9
67.2
61.9
61.6
81.6
53.6
59.5
60.0
66.9
63.5
60.8
78.8
52.2
60.1
60.0
67.3
62.6
61.0
78.5
51.2
59.6
59.9
67.5
60.6
59.9
78.4
53.0
60.3
58.9
67.3
63.4
58.8
78.7
51.5
60.7
55.5
65.9
62.5
59.2
76,2
48.0
60.2
49.8
66.3
59.5
57.5
77.0
On Policy, Weak Models
49.8
56.4
41.4
56.4
62.2
54.5
67.9
11
39.7
54.2
22.1
50.7 7
50.9
52.1
54.2
12
20.8
38.0
6.1
33.1
36.2
40.0
21.9
Off Policy Models
13
56.4
54.5
72.6
70.1
63.2
64.6
88.9
55.7
55.8
72.1
71.2
63.4
63.7
88.6
15
54.7
56.4
69.5
70.4
63.5
64.2
89.3
16
50.0
57.0
67.1
67.3
61.1
59.3
88.3
17
49.2
58.5
65.8
66.1
61.0
59.5
87.1
N MODEL LICENSES
In this section, we list the licenses for the assets used in this project. For training reward models and RLHF training, license. For our model pool, we use a large pool of capable language models, both open-weight and proprietary models, and licenses. We list the licenses for the models in our model pool here and cite the models in Appendix Table
the Open- Instrutntt library, which is open-source and has Apache 2.0
of their generations
evaluation is permissible under their
1. Mistral 7B Instruct v0.3 (Apache 2.0)
2. Tulu 3 8B (Llama 3.1 Community License Agreement)
3. Tulu 3 70B (Llama 3.1 Community License Agreement)
4. Llama 3.1 8B Instruct (Llama 3.1 Community License Agreement)
5. Llama 3.1 70B Instruct (Llama 3.1 Community License Agreement)
6. Llama 3.2 IB Instruct (Llama 3.2 Community License Agreement)
7. Llama 2 7B Chat (Llama 2 Community License Agreement)
8. Tulu 70B (Ai2 ImpACT Low Risk License)
9. Qwen2.5 72B Instruct (Qwen License Agreement)
10. Qwen2.5 Math 72B Instruct (Qwen License Agreement)
11. Qwen2.5 14B Instruct (Apache 2.0)
Qwen2.5 7B Instruct (Apache 2.0)
13. Qwen2.5 0.5B Instruct (Apache 2.0)
14. Qwen2.5 Math 7B Instruct (Apache 2.0)
15. Deepseek Math 7B RI. (deepseek license)
16. OLMoE IB 7B 0924 Instruct (Apache 2.0)
27

---

Published as a conference paper at ICLR 2026
17. Dolphin 2.0 Mistral 7b (Apache 2.0)
18. Zephyr 7b Beta (MIT License)
19. GPT-4o (Outputs produced by GPT-4 are subject to OpenAT's terms of use)
20. Claude 3.5 Sonnet (Outputs produced by Claude are subject to Anthropic terms of service and usage policy)
O BROADER IMPACT
We note that in some cases, completions in the Safety subset may expose sensitive texts to viewers, but these are in line with texts seen in many common safety evaluations and their inclusion in our benchmark facilitates the training of safer reward models. Furthermore, elicit jailbreak completions for copyright infringing prompts or PII prompts, our jailbreak prompts instruct the model to generate random outputs questions like "What is Bill Gates's personal address," so we do not contribute to any PII or copyright infringement. Furthermore, we opted to create an accuracy-biid saaracy--beed benchmark with ground truth correct and incorrect responses, which avoids biasing toward certain groups' preferences in our benchmark.
note that while do
P COMPUTE USAGE
This work primarily trains and evaluates models on H100 GPUs. Running the Evaluation Running the evaluation takes around minutes for an average 8-billion parameter model, and 30 minutes for an average 70-billion parameter model. We ran our evaluation over 160 models, for total of around 30 GPU hours. We ran many intermediate evaluations as well. Training We trained around 120 8B Reward Models, each taking 64 GPU hours per epoch. We also trained 5 70B Reward Models, each taking 1,280 GPU hours. We also conducted 17 PPO training experiments, each of ran for 2 days on 16 GPUs. In total, across all experiments, used 55,000 GPU hours.
28


Reviews:
[{"review_id": "rBhldC6i2u", "reviewer": "Reviewer_rc4h", "summary": "- Introduces a tougher reward model benchmark with six domains, a best-of-4 selection format that lowers the random baseline to 25%, and mostly unseen human prompts to reduce contamination.\n- Shows strong correlation with best-of-N sampling and highlights that transfer to PPO depends on on-policy or lineage-matched reward models.\n- Reports that top models score notably lower than on prior benchmarks, indicating increased difficulty and headroom.\n- Provides practical training insights, including benefits from more than one epoch and lineage matching for RLHF.", "questions": "- How stable are rankings if the best-of-4 candidate set is regenerated with a different generator pool or temperatures?\n- Can lineage matching be quantified more continuously to predict PPO transfer beyond a binary on-policy label?\n- What is the computational cost tradeoff of best-of-4 compared to pairwise setups for broad adoption?\n- How robust is the ties subset to subtle quality differences and score distribution shifts?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Principled evaluation design with a lower random baseline that better matches downstream selection.\n- Comprehensive domains, including calibration via ties, and strong empirical validation against best-of-N.\n- Scaled experiments across many trained and existing RMs yield practical insights.\n- Clear practitioner guidance on training and deployment.", "weaknesses": "- Candidate pool may bias difficulty and favor models similar to generators.\n- Mixed metrics across domains, with the ties metric blending correctness and calibration, can reduce comparability.\n- Limited policy diversity and small subsets in places may restrict generality and statistical power.\n- Heavy reliance on frontier models for filtering could introduce systematic biases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0jBFvOo6Hp", "reviewer": "Reviewer_PtnZ", "summary": "This paper introduces REWARDEVAL, a new multi-skill benchmark aimed at evaluating reward models, a key contribution of REWARDEVAL is its use of newly collected, unseen human-authored prompts, rather than reusing prompts from downstream evaluation datasets. This deliberate decontamination strategy ensures the benchmark provides a clean, unbiased evaluation of reward model generalization and prevents data leakage from overlapping with RLHF or inference datasets. It also demonstrates strong correlation with downstream performance, including in best-of-N sampling and PPO-based RLHF training, highlighting its value as a predictive diagnostic tool for real-world effectiveness. Beyond benchmarking, the paper offers actionable insights for improving reward model training, for example, finding that training for more than one epoch can enhance performance in certain regimes, counter to common assumptions in preference model fine-tuning.", "questions": "Do we have results to backup this claim : \"the reward model should be based on a model of the same lineage as the policy model or else downstream performance can degrade significantly\"? Sorry I didn't find it easily in paper? And I assume it means the policy model needs to be the same as the reward model? I don't see a table indicates that correct me if I am wrong.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper presents a well-designed benchmark that significantly improves upon RewardBench by introducing unseen, human-written prompts to ensure data decontamination and incorporating new task categories for broader coverage. It conducts comprehensive experiments analyzing correlations with downstream tasks, yielding several insightful findings: (1) combining multiple data sources enhances average performance, (2) the choice of base model influences reward model effectiveness, and (3) training reward models for multiple epochs does not inherently degrade downstream performance, challenging common assumptions. The structure is clear and experiments are well-thought, and the insights are easy to understand backed by comprehensive experiments.", "weaknesses": "\"For RLHF, the reward model should be based on a model of the same lineage as the policy\nmodel or else downstream performance can degrade significantly, so simply taking the\nhighest scoring reward model on a benchmark will not ensure a good post RLHF model.\" this seemed to be a very strong statement, I don't see experiments conducted across various model types, a study on different architectures might be beneficial or make this statement less affirmative might be a better consideration?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WNQdB58fAE", "reviewer": "Reviewer_nSPY", "summary": "This paper introduces REWARDEVAL, a large-scale benchmark for evaluating reward models (RMs) used in RLHF and inference-time selection (e.g., best-of-N sampling). The benchmark spans six domains—three familiar ones (Focus, Math, Safety) and three new ones (Factuality, Precise Instruction Following, and Ties). It is constructed from unseen, high-quality human prompts, with four candidate completions per prompt, enabling more granular accuracy measurement and margin-based calibration testing. The authors train 120 Bradley-Terry reward models and show that REWARDEVAL correlates strongly with downstream PPO and BoN performance, while also revealing that RM–policy lineage alignment is crucial for stable RLHF outcomes. Compared to prior datasets such as RewardBench and PPE, REWARDEVAL claims to provide cleaner, harder, and more diagnostic evaluations.", "questions": "1. How robust are the REWARDEVAL correlations when evaluated on policies outside the Tulu family (e.g., Mistral or Llama3 or any other family)?\n\n2. Can the authors provide a scale-invariant version of the Ties metric (e.g., based on ranking or normalized variance)?\n\n3. Were the factuality and safety labels cross-checked with independent human annotators to mitigate LM-as-judge bias?\n\n4. Could domain weighting or bootstrapped confidence intervals be added to report uncertainty in the overall score?\n\n5. How do results change if the number of completions (`N`) increases beyond four?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Ambitious, comprehensive benchmark spanning six diverse domains.\n\n- Strong empirical study with over a hundred RMs and multiple baselines.\n\n- Identification of practical phenomena such as the importance of model lineage in RLHF.\n\n- Systematic comparison to RewardBench, PPE, and other RM datasets clarifies positioning.\n\n- High reproducibility through public release and clear experimental pipeline.", "weaknesses": "- The Ties metric lacks invariance to scaling/temperature and may over-penalize calibrated models.\n\n- Domain averaging ignores differing sample sizes, reducing statistical interpretability.\n\n- Heavy reliance on LLM-as-judge for factuality/safety labels introduces label bias and potential leakage.\n\n- Correlation analyses are based on a single policy distribution, limiting generality.\n\n- “Stronger correlation with BoN” may partially stem from shared data lineage rather than intrinsic benchmark quality.\n\n- No formal error analysis or confidence intervals are reported.\n\nThese weaknesses do not invalidate the idea but suggest that the benchmark’s mathematical rigor and external validity remain limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pNyr7c4dyZ", "reviewer": "Reviewer_XmJn", "summary": "This paper introduces RewardEval, a benchmark and methodology for evaluating reward models. It provides a standardized, interpretable, and scalable way to assess how well reward models capture human preferences across diverse tasks. Compared with RewardBench, RewardEval focuses on unseen, in-the-wild human prompts sourced from WildChat and applies decontamination to avoid overlap with common downstream evaluations. It also includes analyses of reward-model-guided best-of-n sampling and reinforcement learning from human feedback (RLHF). In practice, RewardEval’s scores show strong correlation with best-of-n downstream performance and reveal an important RLHF insight: for PPO, the alignment and distributional match between the policy and the reward model are critical, and high RewardEval scores alone do not guarantee PPO improvements when the reward model is off-policy or out-of-distribution. Overall, RewardEval serves as a more challenging and downstream-relevant successor to RewardBench, with particular emphasis on instruction following, math, and factuality, areas where many leading reward models continue to struggle.", "questions": "1. How do you plan to extend RewardEval to better capture reward model robustness and resistance to reward hacking? Given that on-policy RL training often leads to rapid overoptimization and degradation of reward signal quality, have you considered incorporating adversarial or on-policy evaluation settings into the benchmark?\n2. In Section 5.2, you observe that all reward models, regardless of their RewardEval performance, produce similar final outcomes after PPO training. Could you elaborate on whether this suggests that current reward model quality is not the primary bottleneck in RLHF, or that the RL optimization dynamics overpower the reward signal?\n3. The analysis of RLHF training dynamics is based solely on the TULU 3 8B model. Do you expect similar trends for other model families? and do you have preliminary evidence to support that expectation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Presents a well-designed and comprehensive benchmark that goes beyond pairwise preference accuracy to capture more realistic aspects of reward model performance.\n2. Uses unseen, in-the-wild human prompts and diverse domains, improving robustness and reducing contamination compared to prior benchmarks like RewardBench.\n3. Provides insightful analyses on best-of-n sampling and RLHF, highlighting practical implications of distribution and policy mismatch in reward-guided optimization.\n4. Demonstrates strong empirical validation, with RewardEval scores correlating closely with downstream performance.\n5. Offers clear presentation, transparent methodology, and open-source resources that enhance reproducibility and long-term research impact.", "weaknesses": "1. The paper feels somewhat incremental compared to RewardBench, as it builds upon a similar benchmarking foundation. Although it includes additional analyses on reward-model-guided training and inference, these studies are not comprehensive enough to establish deeper or more general conclusions.\n2. The analysis of RLHF training dynamics is limited to experiments using the TULU 3 8B model, which restricts the generality of the reported insights across architectures and scales.\n3. Additional evaluation dimensions such as reward model robustness and reward hacking resistance would be highly valuable to the community. In practical RLHF setups, reward models often become ineffective after short on-policy training, as the policy quickly learns to exploit their weaknesses. This limitation is also reflected in the paper’s own findings (Section 5.2), where all evaluated reward models, regardless of their RewardEval scores, lead to similar final policy performance after RL training. Addressing this issue directly would make the benchmark far more impactful and less incremental compared to RewardBench.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)