Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy
Authors: Chris Yuhao Liu
Liang Zeng
Yuzhen Xiao
Jujie He
Jiacai Liu
Chaojie Wang
Rui Yan
Wei Shen
Fuxiang Zhang
Jiacheng Xu
Yang Liu
Venue: ICLR 2026 Poster (2026)
Keywords: preference data
reward modeling
data curation
data annotation
Primary area: datasets and benchmarks
TL;DR: Through a human-AI synergistic curation pipeline, we curate a high-quality, large-scale preference data mixture of 40 million preference pairs, enabling state-of-the-art reward models on seven major reward model benchmarks.

Abstract: Despite the critical role of reward models (RMs) in Reinforcement Learning from Human Feedback (RLHF), current state-of-the-art open RMs perform poorly on most existing evaluation benchmarks, failing to capture the spectrum of nuanced and sophisticated human preferences. Even approaches incorporating advanced training techniques have failed to yield meaningful performance improvements. We hypothesize that this brittleness stems primarily from limitations in preference datasets, which are often narrowly scoped, synthetically labeled, or lack rigorous quality control. To address these challenges, we present a large-scale preference dataset comprising 40 million preference pairs, named SynPref-40M. To enable data curation at scale, we design a human-AI synergistic two-stage pipeline that leverages the complementary strengths of human annotation quality and AI scalability. In this pipeline, humans provide verified annotations, while Large Language Models~(LLMs) perform automatic curation based on human guidance. Training on this preference mixture, we introduce Skywork-Reward-V2, a suite of eight reward models ranging from 0.6B to 8B parameters, trained on a carefully curated subset of 26 million preference pairs from SynPref-40M. We demonstrate that Skywork-Reward-V2 is versatile across a wide range of capabilities, including alignment with human preferences, objective correctness, safety, resistance to stylistic biases, and best-of-N scaling. These reward models achieve state-of-the-art performance across seven major reward model benchmarks, outperform the latest paradigm of generative reward models, and demonstrate strong downstream performance. Ablation studies confirm that the effectiveness of our approach stems not only from data scale but also from high-quality curation. The Skywork-Reward-V2 series represents substantial progress in open reward models, highlighting the untapped potential of existing preference datasets and demonstrating how human-AI curation synergy can unlock significantly higher data quality.

Published as a conference paper at ICLR 2026
SKYWORK-REWARD-V2 SCALING PREFERENCE DATA CURATION VIA HUMAN-AI SYNERGY
Chris Yuhao Liu, Liang Zenge, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojic Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu
2050 Research, Skywork Al
ABSTRACT
Despite the critical role of reward models (RMs) in Reinforcement Learning from Human Feedback (RLHF), current state-of-the-art open RMs perform poorly on most existing evaluation benchmarks, failing capture the spectrum of nuanced and sophisticated human preferences. Even approaches incorporating advanced training techniques have failed to yield meaningful performance improvements. We hypothesize that this brittleness stems primarily from limitations in prefer- ence datasets, which are often narrowly scoped, synthetically labeled, or lack rig- orous quality control. To address these challenges, we present a large-scale pref- erence dataset comprising 40 million preference pairs, named SynPref-40M. To enable data curation at scale, we design human-Al synergistic two-stage pipeline that leverages the complementary strengths of human annotation quality and AI scalability. In this pipeline, humans provide verified annotations, while Large Language Models (LLMs) perform automatic curation based on human guidance. Training on this preference mixture, we introduce Skyyork--Reaard V2, a suite of eight reward models ranging from 0.6B to 8B parameters, trained on a carefully curated subset of 26 million preference pairs from SynPref-40M. We demonstrate that Skywork-Rewad Rewakd--warrrrrr- V2 is versatile across a wide range of capabilities, includ- ing alignment with human preferences, objective correctness, safety, resistance stylistic biases, and best-of-N scaling. These reward models achieve state-of-the- art performance across seven major reward model benchmarks, outperform the latest paradigm of generative reward models, and demonstrate strong downstream performance. Ablation studies confirm that the effectiveness of our approach stems not only from data scale but also from high-quality curation. The Skywork- Reward- V2 series represents substantial progress in open reward models, high- lighting the untapped potential of existing preference datasets and demonstrating how human-Al Al curation synergy can unlock significantly higher data quality.
INTRODUCTION
Reward models (RMs) have become critical components in Reinforcement Learning from Human
Feedback (RIHF) pipelines (Christiano et al. 2017 Stiennon et al.) 2020 Ouyang et al., 2022
Dong al. 2024a Lambert JShhllmnn al. 2017), now standard in Large Language Model
(LLM) post-training (Tie et al. 2025). Recent advancements in LLM reasoning capabilities (Jaech
al., 2024 Guo al.) 2025 Xu et al. 2025 Chen et al. 2025a) and Reinforcement Learning
with Verifialle Rewards (RLVR) (Lambert et al. 2024 have sparked interest in policy optimization
via rule-based rewards (Luo et al. [2025c] Wen et al. 2025 Team 2025b a Luo al., 2025b He
et al., (2025)). These reward functions typically verify whether answers match ground truth for math problems or pass unit tests for coding tasks, and can include fine-grained rules for verifiable outputs (Bercovich et al., 20255 Ma et al. (2025). However, complex human preferences often cannot be captured through simple rules, limiting the effectiveness of rule-based approaches in advancing general preference learning. Thus, the challenge of modeling nuanced, sophisticated, and sometimes conflicting human preferences through effective reward models remains largely unresolved.
Corresponding authors.

---

Published as a conference paper at ICLR 2026
To model human preferences, previous works have curated various datasets (Cui et al. 2023 Wang
et al.
2025d Dong et al. 2024a Xu et al. 2024 Park et al. 2024 Lambert et al. 2024 OLMo et al.
2024
with prompts drawn from diverse sources. These efforts employ automatic methods (Cui et al,
2023
Xu etal. 2024) 2024) or human annotators (Wang et al. 2024e| 2025d to generate preference pairs,
enabling preference learning in pairwise contrastive manner (Bradley & Terry 1952] Quyang al.
2022).
Beyond dataset construction, some works aim to improve reward modeling via inductive
biases in enhanced loss functions (Liu et al. 2024b Cai et al. 2024 Yang et al. 2024b) Wang al.
2024e Zhang et al. (024b) or modified model architectures (Wang et al.) [202aa] Chen et al. 2022bb]
Dorka 2024). To evaluate progress in reward modeling, RewardBench (Lambert et al. 2025) was
released as the first benchmark for RMs. As reward models evolve, scores on RewardBench have
begun to saturate (Wang et al. 2024a Park et al. 2024 Wang et al. 2024b Liu et al. 2024b Shiwen
et al. 2024 Wang et al., 2025b; 2024d), but multiple studies (Frick et al. |024] Zhou et al., 2024]
Song et al. 2025 Wen ct al. 2024) have argued that such saturated scores are weak indicators of real progress. These studies highlight weak (or even inverse) correlations between RewardBench scores and downstream task performance (e.g., best-of-N or policy training).
In this work, we focus exclusively the dual goal of both enhancing the quality and scaling the quantity of preference data, to advance the development of open reward models. We introduce SynPref-40M, large-scale preference dataset comprising 40 million preference pairs. We design two-stage preference data curation pipeline (Figure (2 that (1) combines human verification under stringent protocol for quality assurance (Section 3.2). and (2) employs
partaan-prfeeeenceeee LLM judges for scalability (Section 3.3). The pipeline also involves iterative training of a reward model, which continuously incorporates feedback from human labels and retrieves preference data where the RM itself performs poorly, to enable further learning. Our pipeline yields 26 million carefully curated preference pairs, which we use to develop and train Skywork-Reward V2, our recond-generation reward model series, consisting of eight high-performing reward models, ranging from 0.6B to 8B parameters.
Through comprehensive evaluations on seven major RM benchmarks (Lambert et al. 2025 Frick
cet al. 2024 Zhou et al. 2024 Liu et al. 2024c Tan et al. 2024, Malik et al. 2025). we demon- strate that the Skywork- Reerk--ewwddd---d V2 series, trained using only the Bradley-Terry objective (Bradley & Terry 1952). achieves state-of-the-ar performance, with our 8B reward model outperforming all existing open reward models across all seven benchmarks by significant margin. We also demonstrate Skywork-Rewand Reward- V2's superior performance across multiple critical dimensions, in- cluding general human preferences, objective correctness, resistance stylistic biases, safety, and best-of-N scaling (Section (.2).. Through data ablations, we show that the success of SynPref-40M driven not only by its scale but also by its high quality (Section 4.3p. Our method-wing wiite ablations confirm the importance of human annotation. LLM annotation guided by human preferences, and our carefully designed and rigorously implemented annotation protocols (Section 4.4).
We outline our main contributions as follows:
We collect and curate SynPref-40M, which, to the best of our knowledge, the largest curated preference mixture to date.
We release Skywork-Rewe Reward- V2, a series of eight tfte-of-the-ar reward models ranging from 0.6B to 8B parameters, which achieve top rankings on seven major reward model benchmarks, demonstrating strong performance across diverse evaluation dimensions.
We propose a preference data curation pipeline that combines human verification for quality with JJudes guided by human preferences for scalability.
THE BRITTLENESS OF CURRENT OPEN REWARD MODELS
Single-benchmark evaluation has limitations. RewardBench [Lambert et al. 2025) is dataset for pairwise preference evaluation in chat, safety, and reasoning, and has become the standard bench- mark for assessing reward models. However, several subsequent studies (Frick et al., 2024, Zhou et al.) 2024 Wen et al. 2024) argue that scores on RewardBench do not directly correlate with downstream performance and, in some cases, exhibit an inverse relationship. Our evaluation results in Figure corroborate this concern: while RewardBench shows positive correlation with other benchmarks overall, improvements on RewardBench from ~80 to 90+ do not consistently translate to gains other benchmarks. We advocate for benchmarks that either (1) involve more challeng-
2

---

Published as a conference paper at ICLR 2026
RewardBench Newer Benchmarks
Benchmark Score Correlation
RowandBerch
(.73
Dx07
0:32
i0:22
00:00
Avg. Score on Newer Benchmarks
65
RewandBench
Profermnc
067
0.03
0.78
0.79
PEE Conectsesa
0.50
(645
063
1000
0.63
0280
0.71
0.5
RMB
0.23
0.30
0:93
066
1.00
0.70
0.77
0.72
Besch
0.37
0.48
0.78
0.76
1.00
0.84
Skywock-Reed
50
Skyworl- Derivativ
JudgeBerch
Reward Models
M. Re-ewweBeeshh
0.17
0.88
8:71
0.71
0.86
081
(
45
Revartlleach 1:2
PVE Piefoeace
PP Carsanees
EMB
RM Banc
Judgs Bench
Resarilla Metorr
RewardBench Score
Figure 1: Left: Comparison of the performance of 31 top open reward models on RewardBench
[Labbert et al. 2023) and their average scores across seven newer benchmarks (Frick et al., 2024
Zhou et al. 2024] Liu et al., 2024c] Tan et al. [2024, Gureja et 2024). Right: Pearson correlation
scores across seven reward model benchmarks.
ing evaluation methods (e.g., best-of-N) or (2) demonstrate stronger correlations with downstream performance.
A comprehensive evaluation suite reveals inconsistent improvements. Based on the above crite- ria, in addition to RewardBench, we select several other benchmarks that span multiple evaluation dimensions. Specifically, we include PPE Preference and Correctness (Frick et al.
2024 to
2024) for its sess both real human preferences and unambiguous correctness; RMB (Zhou et al. challenging best-of-N evaluation; RM-Bench (Liu al. 2024c) to evaluate robustness
content variation and style bias; and JudgeBench (Tan et a 2024p. which evaluates preference pairs drawn from difficult, real-world LLM evaluation datasets, such as LiveCodeBench (Jain et al. 2024). Fi- nally, we include the newly released RewardBench v2 (Malik et al. (025), which enforces global best-of-N evaluation and extremely difficult capability assessments (e.g.. distinguishing highly sim- ilar responses and reward margin requirements). A detailed description of these benchmarks is provided in Section C.I We present the main results in Figure [ with average scores across the seven newer benchmarks, and report Pearson correlations among all benchmarks. Our findings are as follows:
comparing RewardBench scores
Improvements on RewardBench do not guarantee broader gains. As model scores on Re- wardBench increase from ~880 to 90+, performance on other benchmarks does not consistently improve- it may get better, worse, remain approximately the same. This inconsistency, com- bined with the weak correlations shown in the right plot of Figure [ suggests that researchers and practitioners should avoid interpreting reward model quality based on a single benchmark. Alternative loss functions or model modifications fail to yield consistent gains for the Ceemma-2-27B variants (Yang et al. 2024b Dorka 2024 Lou et al. 2024] Zhang et al.) [024bb] Liu et al. 2025). When examining the 27B models derived from Gemma in our experiments (see in the appendix), the original Skywork-Reward Gemma-2-27B-vol
remains the best RM in terms of average performance, while all variants with loss modifications fall behind. However, we acknowledge this claim limited to this specific model series and does not generalize to all model modifications.
Table 7
Among the top 20 models on RewardBench, 16 directly or indirectly use the same base model (Liu et al. 2024b) are fine-tuned on highly similar training data, indicating stagnant progress in both open preference datasets and reward models since September 2024.
3 SCALING PREFERENCE DATA CURATION VIA HUMAN-GUIDED AI FEEDBACK
3.1 PIPELINE OVERVIEW
Our pipeline (Figure 2 has two stages. In Stage 1, human annotators label gold data under a strict protocol, while LLMs produce silver labels conditioned on human preferences. A reward model trained on silver data is evaluated against gold data, and an adaptive retrieval mechanism selects new samples where the RM performs poorly for re-annotation. This iterates multiple rounds.
3

---

Published as a conference paper at ICLR 2026
Stage 1: Small-Scale Human-in-the-Loop Curation
Verification and Annotation
Verified Pool
Unverified Pool
Raoti (human verilied)
Das (1)
Anneiale preference ctl butes and lobel Juizi
Drive (LLM verified)
Adaptive Retrieval
Reward Modeling
Gather New Data
Retreve preference Oun similaity in (+.o)
Train on Patter Evalusie on Poote
Daa Pan Paws
Stage 2: Large-Scale Automatic Curation
Preference Data
Unconfident/ Inconsistent
LLM Annotation Adaptive Retrieval (no human Involved)
(in-the-wild) Dunla
Pairs (u 0.5 or cicse to 0.5)
Final Curated Data Pool
Consistent with Gold Reward Model (Irained on (gots)
Reward Model (r) (ourrent best) p o=o(r - r(:))
Confident Pairs (p 0.5)
Figure 2: A two-stage preference data curation pipeline. Stage 1 (top) involves human-A Al synergis- tic curation and runs iteratively. Stage 2 (bottom) scales data curation automatically using reward model consistency checks, eliminating the need for further human supervision.
In Stage 2, the Stage reward model and gold reward model (trained on verified human data only) jointly guide consistency- based data selection at scale, requiring no further human supervision.
3.2 STAGE I: SMALL-SCALE HUMAN-IN-THE-LLON CURATION
Overview. Stage an iterative procedure consisting of iterations. In each iteration, we focus on two things: (I) collecting more unverified preference pairs Dun. and (2) producing a silver set Dsiiver via LLM annotation and a gold set D Dgold via human annotation to train and identify flaws in the current reward model. Throughout Stage we accumulate roughly 1M preference pairs in total.
Seed preference data initialization. We begin by collecting available preference data to form an unverified pool, Dun- During preference data collection, we source data from publicly available preference pairs, primarily collected from a wide range of sources (over 40) on Hugging Face, provided that the licenses and terms of permit it (see Section [D.2] for detailed composition and licensing information). For each pair in this pool, given the 3-tuple (I, V/: 31) comprising the conversation x, the chosen (winning) response you, and the rejected (losing) response y/ we collect LLM -LM-generated preference attributes a. Each attribute set is 5-tuple consisting of: (1) task category, (2) preference objectivity, (3) controversiality. (4) desired attributes, and (5) annotation guideline. Task category. objectivity, and controversiality serve as metadata to ensure annotation diversity across scenarios. The desired attributes describe the qualities users seek in good responses, while the annotation guideline provides instance-specific, context-dependent criteria for determining the preference label. We provide examples of these attributes and quality analysis in Section E
Human verification and annotation protocol. We initialize with small, high-qualite and diverse set of preference pair-" as the seed data. Using the generated preference attributes, human annotators perform strict verification following predefined protocol (detailed in Section E.2). At high level, the protocol outlines core principles and practices, as well as specific guidelines tailored to each task category, objectivity type, and controversiality level. For example, it permits the use of external tools - -uuch as search engines, frontier L.LM assistants, and domain-specialized L.MS (e.g., for math or code) - to aid in labeling. However, full reliance L.LMs for labeling is strictly prohibited. Specifically, annotators provide final judgment based on the attributes and the instance-specific guideline. For tasks involving fact-checking, annotators are required to use search engine; for code correctness, annotators instruct an LLM to execute the code and verify correctness. Even when LLMs and external tools are used, annotators remain responsible for the final judgment. (Note that we still consider this process "human annotation.") Preference pairs produced during this process
In our case, use the Skywork-Rewarde Praire-eeeeaaPPffefeee-------- dataset.
4

---

Published as a conference paper at ICLR 2026
collected as gold This rigorous process yields the seed dataset Deeed, where the human-verifee portion is denoted as Dgold (for validation), and the LLM-verified portion as Dsitver (for training). Importantly, the preference attributes in goll are also edited and verified by human annotators, ensuring higher quality.
Step 1: Reward model training and evaluation. We initialize pointwise Bradley-Terry reward model (Bradley& & Terry 1952, Ouyang et al. 2022) and train it on Dailver- We select the best current reward model checkpoint 0 based on validation accuracy on Dgold- This checkpoint referred as the "best reward model" throughout the paper, including in Stage 2 filtering. For each (It, Ders ) we collect its prediction p= (re y.) re(2.3)))
Error-driven adaptive preference retrieval. Instead of relying solely on data increase data volume, we leverage LLM annotators via an adaptive retrieval mechanism (Ram et al. 2023) to collect representative samples aligned with human preferences. This step involves two separate retrieval processes. First, we identify pairs where the current RM performs poorly: we evaluate the current reward model on goldd to identify pairs it misclassifies, then sample new pairs from Pun that are similar to these misclassified pairs. This first retrieval selects samples at the current RM's weak spots for subsequent LLM annotation. This mechanism selects new examples from the unverified pool based on both the preference attributes a and the reward model's predictions. For each pairwise instance, compute the embedding (Sturua et al. 2024) of (z, a) and retrieve the top-k similar items. Intuitively, we prioritize preference data that resemble instances where the reward model errs shows low confidence. We set the retrieval upper bound kimux = 8 and use dynamic rule to determine k:
Step 2:
kimax
if S 0.5 (incorrect prediction)
[Rimax (1-p)
if > 0.5 (correct prediction)
Step 3: Preference-aware labeling. To augment LLM annotation with gold human labels, perform a second retrieval step: for each pair selected in Step 2, we retrieve similar pairs from Dgold (which are human-labeled) and insert them as few-shot examples to guide the LLM in making the final judgment. This ensures that LLM annotation is conditioned on duman--erifien examples throughout the process. Using the retrieved examples with human labels, we employ a group of strong LLMs to aggregate final judgments using self-consistency (Wang et al. (022). First, we perform mntra-model aggregation via self-consistency, then merge results across models to mitigate potential bias from any single model. The list of LLMs used and the annotation prompts are provided in Section D For all LLM annotations, responses are labeled as "Candidate 1" and "Candidate 2," with their order randomized in the prompt. While pointwise scoring (He et al. 2025a; Liu et al.) 2023) has shown greater effectiveness, it is not applicable here due to our reliance on both human and LLM annotators, making it impractical to enforce shared standard. Finally, numan-labeled samples are added to D Dguld. and M-lbelee samples to D Throughout Stage 1, we iteratively perform Steps 1, 2, and 3. After each iteration, we an internal human-labeled validation set for sanity checking. However, scores from this sanity check serve only as a reference; pipeline exccution does not depend on them.
3.3 STAGE 2: LARGE-SCALE AUTOMATIC CURATION OF PREFERENCE DATA IN THE WILD
We now scale up to tens of millions of in-the-wild preference data pairs. We denote this set as Dwild. which contains the remaining publicly available preference pairs not allocated to Dun in Stage 1. Like Dun. Dwild is sourced from publicly available preference pairs collected from over 40 diverse sources on Hugging Face, provided that licenses and terms of use permit it. We allocate all origi- nally human- -lbeled pairs to Dan (used in Stage 1) and leave the rest to Dwiu (used in Stage 2). See Section D.2 for full details. However, annotating the entire dataset - even automatically be prohibitively costly and unnecessary. Below, we describe two consistency-based filtering strate- gies to determine which data points warrant further verification. Importantly, Stage involves no human annotation; all annotation and filtering are performed by LLMs and the best reward model checkpoint from Stage I.
can
Preference consistency with the best reward model. Inspired by Kim et al. (2024) and Liu et al. (002b)). we skip (i.e., directly retain without re-annotation) all pairs where the reward model 's con- fidence exceeds 0.5 under the current best reward model. For the remaining pairs where mismatch is detected (i.e., the preference pair does not agree with the current best reward model), we fall back to LLM annotation to make the final judgment. This LLM annotation step uses exactly the same
5

---

Published as a conference paper at ICLR 2026
Model
Rewardllench RewardBRench
PPR Prof
RMB
JudgeBench
Open Reword Modals
Liame 3 Offectiiz RM 58
Parket SUI POR1
$9.0
64.5
59.2
64,1
57.8
71.3
63.5
67.1
AmoRM Liamal 50
x0.1
Wars
90.4
66.5
50.6
00.5
64.6
69.2
59.7
67.4
Iniemim2 206 reward Ca Ea
90.2
56.3
61.0
63.0
62.9
72.1
64.3
67.1
Skyuoll-Rewuud- Lannolll------ 38-11l
30240
93.1
71.8
62.2
62.5
66.6
64.7
62.9
69.1
LewaR Gomma 2-2BB
v0.1
95.0
72.5
62.4
63.9
67.9
71.0
64.2
71.0
Bomnk--Roanl--emmmme Cammll--mmmm--33 2-77--022
m 2074h
94.3
75.3
61.6
61.9
60.4
67.6
66.5
71.2
Limm--3 C.mn--A--eemmene 1-Nemeeos-718 788 Wanz m
93.9
76.7
64.2
63.2
64.9
72.3
65.8
71.6
RRF-ORR--LbmmS.--1 1-708
95.1
76.5
64.2
64.4
70,5
75.4
70.2
73.8
La--ar-a-laggee Generative Revara Models
GPT [lunt etal K05
$6.7
64,9
61.7
67.1
73,8
15:0
59,8
20.4
Agliinni 5024
64.2
64.7
67.3
69.2
70.6
74.5
64.5
70.
DeepSeck-GRMM GiM--l--GMM-77- 271 Inuret
34.5
63.3
00.4
09.0
DeapSack-(RM-2271
(wr ManaMM af 205
00.4
67.2
63.2
70.3
2M--ll-Quum-lnnnnss-- (Chon MiAlo
02.9
73.0
79.1
20125
90.9
69.8
83.9
ExalPlameer (L.mmo331-706)) Sihretin NU/62
93.9
80.0
50.9
BvllPaaner (1 (Jmm- 1-1788) SURPIE 1//53
03.8
82.1
56.6
[i-IImmm-88 WWinchouee el
55.7
60.3
90.2
734
42.0
J1 Liama 5D MaU031 Whlllcloo
3: 1025
50.6
61.9
Ji-Lhmma 700
Whitlehm
N053
93.3
66.3
72,9
82.7
60.0
]1-Lluma-708 708 [31 0521
2005
67.0
73.7
Our Revirr Models
65.2
65.3
74.5
67.6
70.9
SkymnkkReeend Sgpunt-RoameV2..onns-- V2-Qwmnt-176
00.3
68.3
57.6
72.9
Skyunk-Rowmhh 10comm--V2-Qum---4P
00.5
30.5
Siywors Reward V2 Qwent
93.7
78.2
70.6
75.1
81.2
92.6
7344
79.3
Skywors Rewand V2- Lima 3:2-10
39.9
64.3
66.6
67.4
96.7
76.3
65.0
723
SSyaotk-Rewnn V2-L.mm--32-------
93.0
74.7
69.1
72.1
80.5
69.2
77.1
Skywods Reward V2. Liama 3.1-88
96.4
14.]
77.3
$3.4
56.8
92.8
80.0
853
Skywods Reward V2 Liuma 3.1 8B 40M
97.8
36.3
79.8
37.3
39.3
96.0
8344
88.6
Table 1: Reward model performance assessed
benchmarks. Bold numbers indicate the
best performance among all models, while underlined numbers represent the second best. Entries marked with "_" indicate that model is unrelcased. complete evaluation is provided Table 7
procedure as in Stage we retrieve similar pairs from Dzold and insert them as few-shot examples to help the LLM make the final judgment. We apply the same adaptive preference retrieval and human-preference-guide LLM annotation from Section 3.2 without involving human verifiers.
Preference consistency with the gold reward model. We train a separate gold reward model us- ing all cumulative human-verified verified samples to approximate the "truc" human preference distribution. From the unverified pool, we retain only those pairs whose original chosen-rejected labels are con- sistent with (1) the gold reward model and (2) either the LLM judges or the current best reward model. Approximately million preference pairs passed through this consistency mechanism with- out requiring attribute generation or additional labeling. To leverage the discarded pool, we also experiment with "recycling" the discarded data by simply flipping the chosen-rejected order, which incurs no additional annotation computational overhead.
EXPERIMENTAL RESULTS
4.1 REWARD MODEL TRAINING
We train all models in the Skywork Reward- V2 series using the Llama 3.1 and 3.2 series (Grattafiori ct al., 2024) and the Qwen3 Yang et al.] (225) collection as backbones, with no more than 8B parameters. From the Llama 3 series, and 1-1-a33.--t--nsttrcct For Qwen3, we consider sizes 0.6B, 1.7B, 4B, and 8B. Although larger backbones (e.g., 70B) yield greater gains (Malik al. 2025). we do not consider them due to training cost and deployment practicality.
Reward--
employ Llama-3.118B--Istrucc L.lmma3..-33B-Instee
All reward models trained with maximum context length of 16K tokens, which encompasses the majority of the samples in our data mixture to avoid truncation. We conduct all early-stage experiments with varying learning rates based on model size, using linear decay schedule and small batch size of 256, for epoch, following the hyperparameters in Lambert et al. (2024). For all final model training runs, we adopt the hyperparameters from Wang et al. (2025a), with large global batch size of 10,240 and constant learning rate schedule. We observe no change in final performance; however, using a large batch size significantly reduces convergence time, saving ap- proximately 35% of total training compute. We train all reward models in the Skywork-Rewarde series exclusively on the 26 million curated subset. As an experimental release, Skywork Skywork--eward.. V2-Llama-3, 3.1.--8---B0.. is trained using 26 million curated pairs, along with additional pairs that have a flipped chosen-rejected order (i.e., those that agree with humans) from the discarded 14 mil- lion pairs.
Reward-

---

Published as a conference paper at ICLR 2026
Medel
Hilptubans (BeN)
Haraslane (BoN)
Arg.
GPT4:
50.
54.1
75.0
59.5
50.3
Skyeekk Rioare 3.11-....-1--....
Reward
278--002
51.
61.5
59.1
623
104
13.0
76.8
Deeplort-GaM-- (35M--
655
580
610
56.2
704
85.7
157
19.3
DertSerk- 123---
360
811
Dusill Qveil-Quett333
12 eas (aw)
530
60.4
13.4
13.3
74.
RM-R -eea Soaa-latne--32
63.5
66.
659
<P-enss Landiaso
53.3
863
85.7
33.0
519
Qsea) 28-loo
16.5
66.
[6.
Skyweds Sonan 4LM
533
16-3
52.5
663
Slewwrt Revand V24ppess-a eb Skreerk Reward V240pass-
664
01
16.
12.0
72.
Skywrk Towend 11-0wel-d
66.9
66.3
104
66.7
Skywess Korant V2-Geen--e 405
16.1
63.3
1241
118
13.3
Slyark Breand VS C.
76.1
758
761
Skyant Reeand hal-an--3
08.0
731
Skywei Roseen V/.laam 33-30
64.3
87.5
56.5
2-35
Skyweess Kevant V2-iimm--i-
15.6
13.3
19.3
75.
Hud
Skreen Roward V2 L2.
52.3
82.5
K2.
Jkywes--oorrrl- V2w.nmm 4B-4444
79.9
19.5
$933
$5.7
Skyant Reaard Sharan 1-58-00M
866
Table 2: Performance comparison of RMs with state-of-the-art LLM-as-a-Judge and reasoning models on JudgeBench ("Tan et al. 2024).
Table 3: Reward model pairwise accuracy on the Best-of-N split for Helpfulness and Harm- lessness in RMB (Zhou et al. 2024
4.2 A COMPREHENSIVE EVALUATION OF THE HEYWORK-REWARDE V2 SERIES
We evaluate on seven major benchmarks (details in Section C.1
General preferences. We report full benchmark results for the current topppeffrrmnng reward mod- els, -Ld-as-a-Judge models, and generative reward models in Table [ Across all seven bench- marks, the Skywork-Rewarde but also the emerging class of generative reward models (Liu et al.) 2025 Chen et al.) (025c). We interpret this as strong evidence that SynPref-4OM captures a wide range of preferences. enabling more robust preference learning across multiple dimensions simultaneously. Mcanwhile, Skywork- Reward-V2 V2 highlights the importance of data quality relative the strength of the base models. Even at scale of 1.7B parameters, a reward model can outperform 70B model on all benchmarks except for RewardBench and RewardBench v2, effectively bridging the model size gap.
series of models outperform not only much larger ones (i.e., 70B)
Groend Trats
Skywerk- Revee-- V2.lama3.11- 3-38 Skywedk Bevadl-eedddd Quest--aa
Skywonk Rewari--
Liara U. A---enttree--neee NBB Rewarddtt 278--0
Roward Llate
INF-ORM Liunx3.1--16
Skywerk
GIOA
MBP94
MML.U-PP
Ave Accurscy
Aczurncy
Accumacy
Accumcy
84
N
N
N
N
Figure 3: Best-of-N scaling curves of RMs across five tasks on PPE Correctness Frick al.
2024
Correctness preferences. We compare our RMs with leading LLMs and reasoning models on JudgeBench (Tan et (024) (Table ( and PPE Correctness (Frick et al., 2024). While our RMs underperform state-of-the- reasoning models on average, they outperform all leading models on
knowledge tasks by significant margin. Notably, Skywork-Reward- math performance equivalent to 03-mini (high), while Skywork- Reward- 12-LLlmm--3.1-88 outper- forms 03-min mini (high) in this category.
achieves
Best-of-N accuracy and scaling. We evaluate our RMs on the BoN splits from RMB (Zhou et al,, 2024) and PPE Correctness Preference (Frick al. 2024). As shown in Table 3 RMs demon- strate strong Best-of-N (BoN) capability in both helpfulness and harmlessness. All eight RMs out- perform GPT-4o, the previous state-of-the-art, by margin of up 20 points. We further present wwoor-Rewaard-V2-Llee show superior scaling, outperforming all other
BoN curves for five challenging tasks in PPE Correctness in Figure 3 3.1-8B and models evaluated. Among all BoN scaling curves, all Skywork- Kkywork-Reward-- V2 variants exhibit positive scaling (i.e., performance continues improve as N increases), except for our 1.7B variant in GPQA. We further confirm the strengths of Seywork-Reward-aae in BoN capability on Reward- Bench v2 (Malik al. 2025) (Table which requires precise best-of-N selection globally across the dataset.
Resistance against style biases. Using RM-Bench (Liu et al. 2024c), we assess the ability of reward models to judge substance under varying stylistic differences between chosen and rejected responses. As shown in Table 4 most baseline models exhibit significant performance gaps across the three stylistic conditions, indicating high sensitivity such biases. This is particularly evident for INF-ORM-Llamas 1-BB, with a gap of 36 points between Normal and Hard accuracy. con- trast, -eyworr--Rewrdd V2 models outperform all baselines not only in absolute scores across all
7

---

Published as a conference paper at ICLR 2026
Medel
Easy
Normal
Hand
Ang.
Madel
Factealty
Prochue
Mash
Safey
1ke
Skywork Reword Liuma 3.1 v6.2
Skywork 2 2-28--22-228-0..22
68.9
71.5
M14
71.9
21.5
74.2
35.8
42.1
49.3
09:2
64.7
67.6
USM
Seroed-linasi-aaaaa. 3.278.02
66.3
76.
31.
1500
65.4
072
16.3
91.8
V7.6
91.
11.8
714
23.3
Nemeimu Reward
81.0
36.1
Skywork Ren and Gieeno
75.
90.
761
61.
91.
17.6
754
LDL Revad Genuna 27B
92.4
45.5
71.0
1844
13.3
76.0
INF INF-ORM land 1.2
11.
184
Intemin2: 200 reward
794
chaip-pae 4ac0014
823
41.0
763
8611
13.2
76.3
Lhaux 1-Nemmten-700
92.2
92.1
80,0
76.5
54.0
47.8
75.4
72.2
Willear 1:30
65.
8:4
553
19.
511
718
667
589
56.3
97.
13
18.
811
77.
Skywork Reward Qees3 LLB
90.3
78.0
54.8
74.4
93.0
59.7
78.7
Saywod Braand Gearisit
963
718
814
761
111
817
Skywork Reward
92.1
Skywodk-Benoed V2-oea--2--
$2.6
Skywek Rraand 3--
72.
66.2
753
66.3
W6.
677
755
Skyeork Reward-V V2 L.ama-32------
Skywork Reward V2 Liama 3.3 3.1-B
skyweck -eeonddd 122-------1-212222 1:3-11
91.3
97.0
05.0
84.1
678
96.5
92.8
76.3
81.1
Skywer
Sknnekhitee V2-laaa--2-00 2-38
fioea V2-0nesh-35
23:1
76.2
43.
156
710
014
55.1
860
632
611
143
Skynodk-Rozond (oo.ooodd--44d4o Jamo- 1-1-3--000
97.6
96.9
93.6
90.11
Akywek-llonnn V2L 1ww.aaa......-... 1-1--------00
57.9
672
88.1
913
982
139
965
Table 4: Fine-grained diffieulty-level
Table 5: Comparison of our RMs with the top 12
scores on RM-Bench (Liu et al. (2024c).
RMs on RewardBench v2 (Malik et al. 2025
three categories but also in maintaining much smaller performance differences. We also observe a rapidly shrinking gap as model size increases. These results suggest that training on SynPref-40M leads to more debiased representations of preferences.
Superiority in advanced capabilities. On RewardBench v2, Skywork Rewoork--eerr-V- V2 further demon- strates superior capability in precise instruction following, including assessing whether model's response adheres to specific instructions in the prompt. Notably, all existing reward models score be- low 50 in this category. In contrast,
Llama-3. 1--2-...-BB-40M outperform strong proprietary models like 1-8B and Skywork-Reward-e and Gemini-2.5- and generative reward models that utilize rubrics (Saad Salao-Fllcon et al. (024), through learning a pure representation of preferences. We also observe a significant increase in the Factuality score, likely due the volume of SynPref-40M and the richness of the information it contains.
4.3 ABLATION STUDIES ON DATA QUANTITY AND QUALITY
We further examine the effect of data quantity and quality through performance trends across our pipeline, based on an early version of SynPref-40M with only 16 million preference pairs.
Preference data scaling does not hold for uncurated data. (quality and quantity) In the left plot of Figure 4 we show that increasing the amount of uncurated data results in minimal performance gains. During Stage 2, training on additional 12 million preference pairs fails to surpass the performance of the initial seed model. In contrast, with curated data, we observe consistent perfor- mance improvements as more data is added, with the most significant gains occurring in Stage 2 where the largest volume of curated data is introduced. The "Filtered" curve represents preference pairs that pass both human labeling and LLM annotation in terms of agreement - from the Stage perspective, this is simply the concatenation of Dgold and Dsaverr at each specific iteration. The "Corrected" curve includes the "Filtered" subset plus the subset of preference pairs that pass neither human labeling nor LLM annotation, but with their preference labels flipped (i.e., chosen-rejected swapped). This latter subset corresponds data where either humans L.MSs consider the rejected response to be better. Each point on the curves represents a reward model trained with all curated preference pairs accumulated up to that iteration (cumulative from iterations I through N). Notably, this result partially aligns with findings in concurrent work (Wang et al. 2025a), which specifically demonstrates that subjective preference learning does not exhibit scaling behavior, whereas objective preferences do.
Data curation enables preference "correction." (quality) We further demonstrate that our data curation process not only selects high-quality data for training but also identifies low-quality or "incorrect" preferences, which are placed in a discarded pool during training. By "recycling" this discarded data simply flipping the chosen and rejected responses formance gains across all stages and iterations, as illustrated by the orange curve in Figure 4 As a result, Skywork-Rewaad Reward- V2- Llama-3.1-8B with flipped chosen-rejected responses.
we achieve consistent per-
benefits from the inclusion of preference data even
Training on 1.8% of 16M mixture outperforms previous SOTA open RM (70B) at the 8B scale. (quality and quantity) In the right plot six benchmarks (excluding RewardBench v2 (Malik et al. 2025). which had not been released the
of Figure 4 we report the average RM score across
8

---

Published as a conference paper at ICLR 2026
RM Avg. Score Across Training Stages
Truining Curve of Liuma 3.1-8B-TTRM
Original
85
Filtered
Filtered Conrected
Prestous SOTA
80
70-
Aog.
75
60 18% of training data
Scal
Stage 1 (lea 1-8)
Stage
10000 20000 30000 40000 50000 60000 0
Training
Training Step LA Single Run)
Figure 4: Left: Reward model score progress throughout the entire curation pipeline, including three data ablations: original data, filtered data, and filtered data with corrected preference pairs, based on an early version of SynPref-4OM. Right: The average score of the final training run of a preliminary version of Skywork-Rewarde V2-L-ama--311-3 The Avg. Score indicates the averaged RM score across all benchmarks considered except RewardBench v2.
time) during training. Using only 1.8% (roughly 290K samples) of the full training set surpasses the previous SOTA. This underscores that data mixture excels not only in scale but also in quality.
4.4
ABLATION STUDIES ON ANNOTATION METHOD
We ablate key pipeline components, focusing on iteration of Stage 1 for controlled experiments (full-pipeline ablation is infeasible due its recursive nature and long annotation intervals).
4.4.1 PIPELINE-LEVEL ABLATIONS
Setup. We begin with the filtered seed dataset and ex- amine five settings: (1) direct training unverified data (i.e., no curation). (2) simple LLM curation only, (3) both human and LLM curation, and (4) incorporating adap- tively retrieved examples into LLM curation. These com- ponents collectively represent one iteration of Stage in Figure 2 Note that the text labels in Figure 5 describe the change between two consecutive settings rather than the settings themselves. Bar 1 represents training on seed data only (the baseline RM we start with). Bar 2 cor- responds to seed data plus randomly sampled unverified preference pairs from D'n (a randomly sampled subset from Dmn). representing plus D filtered by pure LLM annotation using ensemble aggregation with self-consie cofiisensst. Bar 4 employs seed data plus D filtered by human annotation. Finally, Bar uses seed data plus Dim curated with our full recipe nauman-guided LLM annotation with adaptive retrieval). Bar 2 has more data than Bar because it includes randomly sampled pairs, while Bars 3, 4, and 5 have less data than Bar 2 because they are filtered by LLM and/or human annotation.
Preference Data Curation Method Ablations
w/ Ackytive Retricaal
wf Hamtnn
Caration
74.2
Avg. Score
Curation
Cunati
71.0
70.9
70
Figure 5: Ablations over different cura- tion variants.
curation. Bar 3 uses seed data
Finding 1: Simple LLM curation barely improves RM quality. As shown in Figure 5 simple LLM curation increases the final RM score by only 0.1 potentially within the error margin of optimization randomness. Given that much in-the-wi wild preference data is synthetically labeled (Cui al.) 2023 Dong et al., (2024a] Lambert et al., 2024) by LLMs, this result aligns with our findings in Figure where scaling uncurated preference data yields negligible gains. A potential factor may be the limited capabilities or annotation quality of the L.MM judges used in our study (Ye et al. 2024
Chen et al. 2024).
Finding 2: Human curation is crucial to data quality. From Figure 5 we observe that the largest improvement comes from human curation, with relative gain of 2.3 points over the seed RM baseline. This highlights the need for scalable methods collecting human preference data and showcases the strength of our approach, which requires only modest amount of human annotation. In Section 4.4.2] we further analyze which component of the human curation pipeline contributes most significantly.

---

Published as a conference paper at ICLR 2026
Finding 3: Adaptive retrieval boosts LLM curation quality. Given access to Auman-curated gold data, adding similar gold examples to the LLM annotation prompt improves RM quality. This technique results in 0.9-point gain compared to raw LLM annotation in the human curation variant. While the improvement is smaller than with direct human curation, this method is simple, scalable, and incurs minimal overhead, making it an attractive tool for enhancing LLM annotation.
4.4.2 HUMAN ANNOTATION ABLATIONS
Setup. We now focus specifically on the most impactful component: human curation. We evaluate three variants: (1) raw human curation, where annotators the conversation history and two responses, (2) human curation with LLM-generated preference attributes, and (3) human curation following our full annotation proto- col (i.e., with external tools such search engines and frontier LLMs). To control for memorization, the same annotators label three distinct subsets of preference data sampled with similar distributions over task category, objectivity, and controversiality. Before running the ablation, we train reward models on each of the three subsets and confirm they yield similar final performance-withe a maximum 0.6-point difference. This reduces the influence of intrinsic data quality as confounding factor, ensuring controlled experiments. All other components remain unchanged from our final method. As in Section 4.4.1 begin from RM trained on the filtered seed data.
Method
Avg. Score
Seed RM
71.0
shown only
w/ Raw Curation
71.4 (+0.4)
w/ Pref Attributes
72.1 (+1.1)
w/ Verification Protocol
74.2 (+3.2)
Table 6: RM scores on three human an- notation setups.
Human annotation with additional information and tools boosts annotation quality. As shown in Table 6 all forms of human curation improve the quality of the seed RMM. Raw annotation based solely on the conversation and two responses results in a 0,4-point gain. Adding preference attributes (task category, objectivity, controversiality, desired attributes, and annotation guidelines) yields larger 1. -.-ptt gain. Incorporating our full annotation protocol - including access to external tools leads to the best final performance, with 3.2-point improvement, validating the effectiveness of our human curation process.
4.5 ADDITIONAL EXPERIMENTS
We provide additional experiments in the appendix: our curated mixture outperforms all existing preference mixtures (Section H.T), downstream RLHF and human evaluation (Section H.2) gener- alization across LLM backbones (Section H.3) and Stage 2 filtering effectiveness (Section H.4
CONCLUSION
In this work, we introduce SynPref-40M, a preference data mixture comprising 40 million pref- erence pairs (26 million curated), and Skywork-Reward-V2
series of eight state-of-the-art re- ward models designed for versatility across wide range of tasks. SynPref-40M is constructed through a two-stage curation pipeline that synergistically combines human supervision for quality with human- guided LLM judges for scalability. Using this preference data mixture, we present the Skywork-Reward-V series a collection of eight strong reward models ranging from 0.6B to 8B parameters. Across seven major reward model benchmarks, models in the Skywork-Rewe Reward- series achieve state-of-the-ar performance, demonstrating strong capabilities in capturing general human preferences, objective correctness, resistance to style biases, safety, and best-of-N scaling. Our small 1.7B variant surpasses the best existing 70B reward model on average, while our 8B vari- ant ranks first on all seven benchmarks among all open reward models. We also conduct extensive ablation studies both the data and the curation method to validate the effectiveness of our ap- proach. We believe this work advances open reward models and, more broadly, RLHF research, representing a significant step forward that will accelerate open progress in the field.
ACKNOWLEDGMENTS
We thank our colleagues at Skywork AI, Kunlun Inc, for providing computational resources and for their valuable feedback throughout this project.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Anthropic. Claude 3.5 sonnet model card addendum. https://wwy.nttrroiic.ooom /elaude- 3-ndee----sonnee 2024.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, et al. Program synthesis with large language models. arXiv preprint TrXiv:2108.07732 2021.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204, 05862, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprini wrXiv:22 2022b.
Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El- Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, Ehud Karpas, Ran Zil- berstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, Ger- ald Shen, Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Yoshi Suhara, Olivier Delalleau, Zijia Chen. Zhilin Wang, David Mosallanezhad, Adi Renduchintala, Haifeng Qian, Dima Rekesh, Fei Jia, Somshubra Majumdar, Vahid Noroozi, Wasi Uddin Ahmad, Sean Narenthiran, Alek- sander Ficek, Mehrzad Samadi, Jocelyn Huang, Siddhartha Jain, Igor Gitman, Ivan Moshkov, Wei Du, Shubham Toshniwal, George Armstrong, Branislav Kisacanin, Matvei Novikov, Daria Gitman, Evelina Bakhturina, Jane Polak Scowcroft, John Kamalu, Dan Su, Kezhi Kong, Markus Kliegl, Rabeeh Karimi, Ying Lin, Sanjeev Satheesh, Jupinder Parmar, Pritam Gundecha, Bran- don Norick, Joseph Jennings, Shrimai Prabhumoye, Syeda Nahida Akter, Mostofa Patwary, Abhinav Khattar, Deepak Narayanan, Roger Waleffe, Jimmy Zhang, Bor- Yiin Su, Guyue Huang, Terry Kong, Parth Chadha, Sahil Jain, Christine Harvey, Elad Segal, Jining Huang, Sergey Kashirsky, Robert McQueen, Izzy Putterman, George Lam, Arun Venkatesan, Sherry Wu, Vinh Nguyen. Manoj Kilaru, Andrew Wang, Anna Warno, Abhilash Somasamudramath, Sandip Bhaskar, Maka Dong, Nave Assaf, Shahar Mor, Omer Ullman Argov, Scot Junkin, Olek- sandr Romanenko, Pedro Larroy, Monika Katariya, Marco Rovinelli, Viji Balas, Nicholas Edel- man, Anahita Bhiwandiwalla, Muthu Subramaniam, Smita Ithape, Karthik Ramamoorthy, Yut- ing Wu, Suguna Varshini Velury, Omri Almog, Joyjit Daw, Denys Fridman, Erick Galinkin, Michael Evans, Katherine Luna. Leon Derczynski, Nikki Pope, Bileen Long, Seth Schneider, Guillermo Siman, Tomasz Grzegorzek, Pablo Ribalta, Monika Katariya, Joey Conway, Trisha Saar, Ann Guan, Krzysztof Pawelec, Shyamala Prayaga, Oleksii Kuchaiev, Boris Ginsburg, Oluwatobi Olabiyi, Kari Briski, Jonathan Cohen, Bryan Catanzaro, Jonah Alben, Yonatan Geif- man, Eric Chung, and Chris Alexiuk. Llama-nemotron: Efficient reasoning models, 2025. URL org/ abs 2505.5555.. 00949
httpss://aat
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-35,, 1952.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbren- ner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong general- ization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09330, 2023.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang. Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang. Jiaqi Wang. Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong,
11

---

Published as a conference paper at ICLR 2026
Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang. Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang. Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang. Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internim2 technical report, 2024.
Yapei Chang, Yekyung Kim, Michael Krumdick, Amir Zadeh, Chuan Li, Chris Tanner, and Mohit lyyer. Bleuberi: Bleu is a surprisingly effective reward for instruction following. arXiv preprint arXiv:2505. 1080, 2025.
Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or Ilms as the judge? a study on judgement biases. arXiv preprint arXiv:2402. 10669, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.08: 03374, 2021.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Тe Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of- thought for reasoning large language models. arXiv preprint wrxiv:2503.09500 2025a.
Shikai Chen, Jin Yuan, Yang Zhang. Zhongchao Shi, Jianping Fan, Xin Geng, and Yong Rui. Ldl- Swwuuddgemaa---27117
https: atttps://uggiiggfeee htt.ggingfaaccee.///////////aa.e co/ Sariikaihhen/LDLL.
d- GeGme 2- 2025b. Label Distribution Learning for Reward Modeling, Tech report forthcoming.
Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, et al. Rm-rl: Reward modeling as reasoning. arXiv preprint arXiv:2505.02388 2025c.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. arXiv preprint arXiv:2310.01377, 2023.
Oliver Daniels-Kode and Rachel Freedman. The expertise problem: Learning from specialized feedback. arXiv preprint arXiv:22/ 1,06519, 2022.
Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Shafiq Joty, Boyang Li, and Lidong Bing. Is gpt-3 a good data annotator? arXiv preprint arXiv:2212. 10450, 2022.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint urXiv:2304.06766 2023.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.0788 2024a.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. Transactions on Machine Learning Research, 2024b. RtL//ttts://aaxttt wrriiiiiorrrrrrooooo org/aaa// abs /
05. 07863
Karel D' D'Oosterlinck, Tobias Daudert, Aditya Deshpande, Christopher Spirtes, Dan Garrette, Jacob Eisenstein, and Christopher Potts. Anchored preference optimization and contrastive revisions: Addressing underspecification in alignment. Transactions of the Association for Computational Linguistics, 13:442-460. 2025.
Nicolai Dorka. Quantile regression for distributional reward models rlhf. arXiv preprint arXiv:2409. 10/64, 2024.
12

---

Published as a conference paper at ICLR 2026
Yongren Duan, Rafael Rafailov, Pi-Chun Chiang, Rémi Munos, Stefano Ermon, and Chelsea Finn. Scaling laws for reward model overoptimization in direct alignment algorithms. arXiv preprint arXiv:2409.19325, 2024.
Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. How to evaluate reward models for rlhf. arXiv preprint arXiv:2410. /4872, 2024.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd workers for text- annotation tasks. Proceedings of the National Academy of Sciences, 120(30):23300116120, 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint 242.42240. 2024.
Srishti Gureja, Lester James V Miranda, Shayekh Bin Islam, Rishabh Maheshwary. Dr- ishti Sharma, Gusti Winata, Nathan Lambert, Sebastian Ruder, Sara Hooker, and Marzich Fadaee. M-rewardbench: Evaluating reward models in multilingual settings. arXiv preprint 2:X2::2410.15522, 2024.
Bingxiang Hc, Wenbin Zhang, Jiaxi Song, Cheng Qian, Zixuan Fu, Bowen Sun, Ning Ding, Haiwen Hong, Longtao Huang, Hui Xue, et al. Air: A systematic analysis of annotations, instructions, and response pairs preference dataset. arXiv preprint arXiv:2504.03612, 2025a.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner technical report. arXiv preprint arXiv:2505.22312, 2025b.
Yifei He, Haoxiang Wang, Ziyan Jiang, Alexandros Papangelis, and Han Zhao. Semi-supervised reward modeling via iterative self self-training arXiv preprint arXiv:2409.0665 06903, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 22166, 2024.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 2rXiv:2412.16722 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.0799 2024.
Dongyoung Kim, Kimin Lee, Jinwoo Shin, and Jachyung Kim. Spread preference annotation: Direct preference judgment for efficient Ilm alignment. In The Thirteenth International Conference on Learning Representations, 2025.
13

---

Published as a conference paper at ICLR 2026
Joongwon Kim, Anirudh Goyal, Aston Zhang, Bo Xiong, Rui Hou, Melanie Kambadur, Dhruv Mahajan, Hannaneh Hajishirzi, and Liang Tan. A systematic examination of preference learning through the lens of instruction-followe -oolowwig. arXiv preprint arXiv: 24X.v:2412.15282, 2024.
Nathan Lambert. Reinforcement learning from human feedback. arXiv preprint arXiv:2504. 2025.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tς" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15122 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling. In Findings of the Association for Computational Linguistics: NAACL 2025, pp. 1755-1797, Albuquerque, New Mexico, 2025. Association for Computational Linguistics. URL https: wwwlaathhlloyy.rrg/222/.
findings-naaci. .6
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267 2023.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https: //qs::///iit wtt.tttt..otttttaaaaaa/aa/a///aa aabaalaacaa__aaaallllcaaaaaaal_a 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.1888 18451, 2024b.
Fei Liu et al. Learning to summarize from human feedback. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 583-592, 2020.
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Rm-bench: Benchmarking reward models of language models with subtlety and style. arXiv preprint arXiv:2410. 20..::241016884.. 16184, 2024c.
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint 2544v22504.02495, 2025.
Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. Uncertainty- aware reward model: Teaching reward models to know what is unknown. arXiv preprint 10X1022400.00847 2024.
Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, al. Large language model agent: A survey methodology, applications and challenges. arXiv preprint arXiv:2503.2 256.222600,.0 2025a.
Michael Luo, Sijun Tan, Roy Huang, Ameee Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li. Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open source 14b coder at m3-mini level. https::/prettt
1 -=2ea22_ccc122222100003333333333
2025b. Notion Blog.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Surpassing ol-preview with 1.5b model
rl. httpp//////pppttrroooooo///////p http://hw......r.............a..
2025c. Notion Blog.
14

---

Published as a conference paper at ICLR 2026
Xueguang Ma. Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General-reasoner: Advancing Ilm reasoning across all domains. arXiv preprint (rXxi:25505.16552 2025.
Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A Smith, Hannaneh Ha- jishirzi, and Nathan Lambert. Rewardbench 2: Advancing reward model evaluation. arXiv preprint ar arXiv:2506.00 006939000000..... 2025.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656 2024.
OpenAL. OpenAI o3 and mini System Card, April /pdE/2221c875-002
URL wtt.ttpps//ccc.cooeeeeee.
mi syste
--card.pdd
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-277444 2022.
Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksci Severyn. West-of-n: Synthetic preference generation for improved reward modeling. arXiv preprint arX1v:2401.12086, 2024.
Junsoo Park. Seungyeon Jwa, Meiying Ren, Dacyoung Kim, and Sanghyuk Choi. Offsetbias: Lever- aging debiased data for tuning evaluators, 2024.
Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang. Jing Xu, Maryam Fazel-Zarandi, Mohit Bansal, Sainbayar Sukhbaatar, Jason Weston, and Jane Yu. Self-consistency preference opti- mization. arXiv preprint 2rXiv:2411.04/00 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Informatinn Processing Systems, 36:53728 --53372--5374, 2023.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- -eowwn, and Yoay Shoham. In-context retrieval-augmmened language models. Transactions of the Association for Computational Linguistics, 11:1316-133 1331, 2023.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp- 3505-3506 2020.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-prec q&a bench- mark. In First Conference on Language Modeling, 2024.
Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprini a30iv: 0/263, 2023.
Jon Saad- Fald-Falcon, Rajan Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, and Shikib Mehri. Lmunit: Fine-grained evaluation with natural language unit tests. arXiv preprint arXiv:2412. 2024.
Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason Weston, and Tianlu Wang. Learning to plan & reason for evaluation with thinking-llm-aas--jjugg. arXiv preprint arXiv:2501.18090 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
15

---

Published as a conference paper at ICLR 2026
Tu Shiwen, Zhao Liang, Chris Yuhao Liu, Liang Zeng, and Yang Liu. Skywork critic model series, 2024.
Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. Prmbench: A fine-grained and challenging benchmark for process-level reward models. arXiv preprint arXiv:2501.03124, 2025.
Nisan Stienmon, Long Duyang.Jeffrey, Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krim- mel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Andreas Koukounas, Nan Wang, and Han Xiao. jin--embedding--3:: Multilingual embeddings with task lora, 2024. URL
org/abs https: http://////raiio.oor//////////// 2409. 10173
Hao Sun, Yunyi Shen, and Francoss Ton. Rethinking bradley-terry models in preference-based reward modeling: Foundations, theory, and alternatives. arXiv preprint arXiv:2411.04991, 2024.
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. Judgebench: A benchmark for evaluating Ilm-based judges. arXiv preprint arXiv:2410.122 127844 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Bean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. /8805, 2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.001 18, 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025a. URL
htt.thuuu..o/bbllo/ggggg/ 32b/
TinyR1 Team. Superdistillation http://issiillliiiocaceeeeeeeeee achieves near-rl performance with just htthh::ttt::hhgttt
Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang, Jiangyue Yan, Yao Su, et al. A survey post-training of large language models. arXiv preprint arXiv:2503.06072. 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Binghai Wang. Runji Lin, Keming Lu, Le Yu, Zhenru Zhang, Fei Huang, Chujie Zheng, Kai Dang, Yang Fan, Xingzhang Ren, et al. Worldpm: Scaling human preference modeling. arXiv preprint 2055522005. 2025a.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and offture-of-experts. arXiv preprint arXiv:2406.1288 2024a.
Peifeng Wang, Austin Xu, Yilun Zhou, Caiming Xiong, and Shafiq Joty. Direct judgement pref- erence optimization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 1979-2009 2025b.
16

---

Published as a conference paper at ICLR 2026
Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi Yu, Richard Yuanzhe Pang, Maryam Fazel- Zarand,, Jason Weston, and Xian Li. Self-t eval- uators. arXiv preprint 2XX1::2408.02666 2024b.
Xiaokun Wang, Chris, Jiangbo Pei, Wei Shen, Yi Peng. Yunzhuo Hao, Weijie Qiu, Ai Jian, Tianyidan Xie, Xuchen Song. Yang Liu, and Yahui Zhou, Skywork-vl reward: An effective reward model for multimodal understanding and reasoning, 2025c. URL httpthttt//.ai...otot
05. 07263
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2233.11171, 2022.
Yubo Wang, Xueguang Ma. Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro. pro: A more robust and challenging multi- task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024c.
Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. Helpsteer2-ppefeeenece Complementing ratings with preferences. arXiv preprint ar arXiv:4410001257, 2024d.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406. 2400:2406.08677, 2024e.
Zhilin Wang. Jiaqi Zeng, Olivier Delalleau, Daniel Egert, Ellic Evans, Hoo-Chang Shin, Felipe Soares, Yi Dong, and Oleksii Kuchaiev. Dedicated feedback and edit models empower inference- time scaling for open-ended general-domain tasks. ar preprint arXiv:2503.04377 044378 2025d.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-rl: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint ArX:v:2033. 10460, 2025.
Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, Xing Yu, Xinyu Lu, Ben He, Xianpei Han, Debing Zhang, and Le Sun. Rethinking reward model evaluation: Are we barking up the wrong tree? arXiv preprint 14.0524410.05584, 2024.
Colin White, Samuel Dooley. Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, al. Livebench: A challenging, contamination- free Ilm benchmark. arXiv preprint arXiv:2406. 19314, 2024.
Chenxi Whitchouse, Tianlu Wang. Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swarnadcep Saha. J1: Incentivizing thinking in llm-as- alm-as-a-judge via reinforcement learning. arXiv preprint arXiv:2505. 10320, 2025.
XAI. Grok beta release.
2024.
xAI. Grok 3 beta - the age of reasoning agents.
2025.
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501.09688 2025.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned lims with nothing, arXiv preprint arXiv:2406.0m 204444406.08464, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 155 2024a.
17

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint M0Xhv:25005003388,, 2025.
Minghao Yang, Chao Qu, and Xiaoyu Tan. Inf-orm-Ilama3.--700 2024b. URL [https://huug
INF ORM Clama3. 1- 70B] (https:://aggiinn
co/1
nfLy/INF-ORM-LI ama43.11aaa.... 1-mmaa...111)111111 1-70B)
Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for Ilms. In Advances in Neural Information Pro- cessing Systems, 2024c.
Jiayi Ye, Yanbo Wang, Yue Huang. Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Yu Chen, et al. Justice prejudice? quantifying biases in Ilm-as-a- judge. arXiv preprint arXiv:2410.02736. 2024.
Lifan Yuan, Ganqu Cui, Hanbin Wang. Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng. Zhiyuan Liu, and Maosong Sun. Advancing Ilm reasoning generalists with preference trees, 2024a.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Forty-first International Conference on Machine Learning, 2024b.
Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, et al. Intemim-xxcompssez. 5-reward: A simple yet effective multi-modal reward model. arXiv preprint arXiv:2501.12368 2025.
Haoran Zhang, Chang Liu, Xi Chen, Jie Dai, and Tie-Yan Liu. Mechanism design for Ilm fine-tuning with multiple reward models. arXiv preprint arXiv:2409. /8417, 2024a.
Yifan Zhang, Ge Zhang, Yue Wu, Kangping Xu, and Quanquan Gu. General preference modeling with preference representations for aligning language models. arXiv preprint arXiv:2410.02197, 2024b.
Jialun Zhong, Wei Shen, Yanzeng Li, Songyang Gao, Hua Lu, Yicheng Chen, Yang Zhang, Wei Zhou, Jinjie Gu, and Lei Zou. A comprehensive survey of reward models: Taxonomy, applica- tions, challenges, and future. arXiv preprint arXiv:2504. 12328, 2025.
Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, et al. Rmb: Comprehensively benchmarking reward models in Ilm alignment. arXiv preprint arXiv:2410.09893, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Inslrwction-folloiingg evaluation for large language models, arXiv preprint arXiv:231..079) 2023.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving llm helpfulness harmlessness with rlaif, November 2023. URL htt.thttss//tarllnnn cs be rkeley. edu//
18

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Preference data annotation. Traditional preference data annotation relies heavily on human anno-
tators (Liu et al. 2020. Stiennon et al. 2020; Ouyang al. 2022. Bai et al. 2022a Hurst et al.) 2024 Touvron et al. 2023a b). which is both costly and inefficient - and sometimes even noisy
(Daniels-Koch & Freedman 2022). To improve scalability, recent work now collectively referred to as RLAIF [Bai et al. 2022b) has proposed various forms of automatic annotation using strong
LLMs (Bai et al. 2022b] Lee et al. 2023; Burns al., 2023] Cui et al.,
2023
Guo et al. 2024
Yuan et al. 2024b; Prasad et al. 20244] Pace et al. 2024 Lambert ct al.
2024
He et al. 2025a),
in some cases even outperforming human annotators (Gilardi al., 2023 Ding al. (022). Our
approach combines the strengths of both paradigms: we enhance human annotation using exter- nal tools and frontier LLMs, while also guiding LLM--bsed annotation with human- human-verified labels. Among related work, the most relevant are Kim et al. (2025) and He et al. (2024). Kim et al.] (2025) leverages a small set of human-labeled seed data to iteratively refine an LLM policy via self-improvement (Rafailov al. 2023); in contrast, we iteratively incorporate gold human pref- erence labels to augment LLM annotation within a structured data curation framework. He et al. (2024) employs iterative process that pseudo-labels unlabeled preference pairs and retains only high-confidence examples, without human annotators. Our work bridges the gap between human and L.LM-based annotation by integrating them into principled and scalable framework, enabling high-quality preference data at scale. In addition, our approach to human verification via preference attributes similar to LMUnit (Saad-Falcon et al., 2024), which decomposes requirements based on context and conducts automatic "unit tests" on assistant responses using LLMs.
The paradigm of reward models. The reward model paradigm has evolved rapidly. Initially based on the Bradley- Terry (BT) model (Bradley & Terry] 1952. Liu et al., 2020) Stiennon et al.) 2020; Ouyang et al., 2022 Bai et al. 2022a). early reward models were trained to maximize the score difference between pairwise responses. During inference, these models produce a scalar score in- dicating the relative quality of a response compared to alternatives given the same prompt. Later, RewardBench (Lambert et al. 2025) introduced the first taxonomy of reward models, categorizing them into (1) sequence classifiers, (2) direct preference optimization (DPO) models with implicit rewards, (3) generative models, and (4) custom classifiers. Most B'T-based models fall under the quence classifier category, while generative models primarily include LLM-as-a-Jddge approaches. DPO models, by contrast, rely on implicit rewards derived from the DPO objective (Rafailov al.) 2023 This taxonomy was further elaborated in Liu et al. (2024b) and has since been adopted by subsequent works (Zhong al., 2025 Zang al. 2025 Wang et 2025c). With the emergence of generative reward models (Liu et al. 2025 Chen et al. 2025c Saha et al. 2025 Guo et al. 2025), Liu et al. (2025) proposed new categorization based on the form of reward generation and scor- ing patterns, highlighting differences in input flexibility and inference-tine scalability. The reward generation forms include sealar, semi-scalar, and generative outputs, while scoring patterns are cat- egorized as pointwise or pairwise. Beyond these major paradigms, (Sun et al. (2024) introduces an alternative approach that trains reward models using an order consistency objective. This reframes reward modeling a binary classification task and has been shown to outperform the Bradley-Tery Terry model in the presence of annotation noise.
Strong open reward models and preference datasets. At the time of writing, there are already 166 reward models on the RewardBench v1 leaderboard (Lambert et al. 2025). most of which are open- weight. The top-ranking models are primarily from the Skywork Reward series (Liu et al. 2024b)
and their derivatives, trained using either the same base models (Dorkal 2024 Lou et al.) 2024) or
datasets (Yang et al., 2024b Shiwen al. 2024 Lou et al. 2024 Zhang et al. 2024b] Yang al.) 2024c). Their training data primarily consist of unfiltered human preferences and automatically curated synthetic data (Liu et al.] 2024b). Another line of high- performing reward models includes
FsfairX and ArmoRM (Dong et al. 2024b] 2023] Waan et al. (2024a), trained on Preference 700K (Dong et al. 2024b), a dataset composed of preference data aggregated from eight diverse sources. The ArmoRM variant extends FsfairX with a multi-dimmensooal reward head, enabling it to generate reward signals for fine-grained aspects of response quality. The InternLM2-Rewant series (Cai et al.) 2024] also presents strong models across different sizes, trained on large-scale collection of 2.4 million closed-sourc source preference pairs, with a focus on both English and Chinese data. Recently, the release of RewardBench v2 (Malik et al. 2025) introduced set of seven reward models trained on various Llama-3.1 checkpoints (i.c., different sizes and base models). Among these, the 70B variant is one of the top-performing models on the benchmark. Right before our release, we noticed
19

---

Published as a conference paper at ICLR 2026
two generative reward models from the LMUnit series (Saad-Falcon et al. 2024) that topped the RewardBench v2 leaderboard. These models use rubrics as unit tests, which are much more robust than reward models based on discriminative classifiers. Their strength is further reflected by their high scores in Factuality and Ties categories. Our reward models leverage both the Skywork-Rewan dataset and Preference 700K in the Seed and Stage 1 phases, esspetively-foomee the foundation for improvements in later stages.
B LIMITATIONS
Human preferences are inherently diverse and often conflicting, especially for prompts without a single correct answer. Even when ground-truth answers exist, individuals may differ in their pref- erences based on factors such as writing style, tone, level of detail, or the relative weighting of helpfulness versus harmlessness. A single reward model may not fully capture this complexity and may inherently favor certain response types over others. Future work could explore personalized reward models or context-dependen training paradigms better reflect the multifaceted nature of human preference.
Our observation regarding performance improvements from re-annotated discarded data is purely empirical. Due to budget constraints, we did not conduct further verification to rigorously assess this pool. As result, the re-annotated data may include noisy preferences or judgments that are not broadly representative that fall outside the scope of current evaluation benchmarks. A thorough investigation of this flipped pool is left for future work.
Meanwhile, we would like to clarify that not all discarded preference pairs are incorrect or useless. Since our pipeline still uses LLMs and trained reward models to filter data, which is not fully inter- pretable, biases and modeling errors are inherently unavoidable. Studying why and how examples are removed during the process, as well as their actual usefulness for reward modeling and RLHF, could be a valuable research direction.
Our annotation protocol differs in implementation from most existing approaches. where human annotators provide their own preferences. In contrast, our protocol is more constrained: it instructs annotators to follow predefined desired attributes and annotation guidelines for each sample. While this structured approach promotes consistency, it also reduces flexibility and may not fully capture minority preferences. This limitation arises because, for certain subjective preferences, it is often infeasible to determine which response better-even on relative scale.
Finally, the success of our approach relies heavily on human annotation; we did not observe sat- isfactory results from fully automatic curation alone. This raises the question of whether current- generation LLMs are capable of supporting high-quality, fully automatic data labeling. Due to inference costs and API limitations, we were unable to scale automatic curation the latest frontier models with strong reasoning capabilities. We consider this promising direction for future explo- ration, particularly given the central role these LLMs already play in supporting human annotation within our pipeline.
C REWARD MODEL BENCHMARKS AND EVALUATION RESULTS
C.I REWARD MODEL BENCHMARKS
RewardBench. RewardBench (Lambert et al. 2025 is the first benchmark released for evaluat- ing reward models. It includes 2,985 evaluation samples from 23 data sources, categorized into four main groups: chat, chat-hard, safety, and reasoning. The evaluation uses pairwise comparison accuracy, where reward model generates scores for both the chosen and rejected responses. A prediction is correct if the score for the chosen response exceeds that of the rejected one. Final ac- curacy is computed as a weighted average within each category and then averaged across categories. A noted limitation is that the chosen-rejected pairs are constructed using semi-automatic methods and manually validated, though the authors do not detail the validation process, They also acknowl- edge potential spurious correlations in the reasoning subsets and the absence of correlation analysis between RewardBench scores and downstream performance.
PPE Preference and Correctness. PPE (Frick al.) 2024) includes two datasets for evaluating reward models: PPE Preference and PPE Correctness. PPE Preference consists of 16K human- labeled preference pairs from Chathot Arena, targeting real human preferences. PPE Correctness
20

---

Published as a conference paper at ICLR 2026
is derived from challenging benchmarks with ground-truth answers, allowing direct verification of preference pairs. Included benchmarks are MATH (Hendrycks et al. 2021), MBPP (Austin et al, 202I), MMLU-Pro (Wang et al. (024c), IFEVAL (Zhou et 2023), and GPQA (Rein et 2024). Each prompt yields 32 LLM responses, enabling both pairwise and best-of-N evaluations. The authors demonstrate strong correlation between PPE scores and downstream RLHF performance, making it a reliable benchmark for real-world reward model evaluation.
RMB. RMB (houu et al. 2024) is comprehensive benchmark covering 49 real-world task cate- gories under both helpfulness and harmlessness. Like PPE Correctness, it supports pairwise and best-of-N evaluations. Preference pairs are generated synthetically. with GPT-4 providing pointwise ratings based on query-specific principles. Human verification is used to ensure dataset quality. RMB shows strong positive correlation with downstream performance across several benchmarks.
RM-Bench. Unlike other benchmarks that focus on general preference evaluation, RM-Bench (Liu al. 2024c) specifically tests a reward model's 's ability to discern nuanced response differences and resist style biases. It includes four categories: Chat, Math, Code, and Safety. Prompts are sourced from benchmarks such as AlpacaEval (Li et al.] [023)) HumanEval (Chen et al. 2021), MATH (Hendrycks al. (22T), and XSTest (Röttger et al., 2023). Response pairs minimally different (e.g., word-level changes introducing factual errors) and generated with controlled style. RM-Bench defines three difficulty levels: (I) easy, where style mismatches may mislead the model; (2) normal, with matched stylistic quality; and (3) hard, where content is decisive despite a stylistically superior distractor.
JudgeBench. JudgeBench ( (Tan et al. 2024) is a serrectness-focusse benchmark originally designed for LLM-based judges. Due its pairwise format, naturally supports pointwise reward model evaluation. It includes subsets such as MMLU-Pro (Wang et al. 2024c) (knowledge), LiveBench
(White al. 2024) (math and reasoning). and LiveCodeBench (Jain al. 2024).
RewardBench v2. RewardBench v2 (Malik et al. 2025) is the second version of the original Re- wardBench (Lambert al.) 2025). featuring substantially more difficult and realistic evaluation data. It assembles new human-generated prompts (in contrast to prior benchmarks which reuse downstream prompts), grouped into diverse and multi-skill classification tasks. On average, exist- ing reward models score around 20 points lower on RewardBench v2 compared to its predecessor. RewardBench v2 also shows stronger correlation with downstream performance - both during RL fine-tuning (e.g., PPO) and best-of-N inference sampling - compared to earlier RM benchmarks.
21

---

Published as a conference paper at ICLR 2026
C.2 FULL EVALUATION RESULTS
In Table 7 we present the complete evaluation results for all the reward models considered. We categorize them into Tradley-Temry reward models, LLM-as-Judges, and the new paradigm of gen- erative reward models (Liu et al. 2025). Across all seven benchmarks discussed in the main body of the paper, our reward models trained on SynPref-4ONM outperform all previous models on average.
Model
RewardBeuch
RewardBRach
PPE Pref
RMB
RM-Bench
JudgeBench
Bradley-Terry Resund Minlels
GRM gemmal 20 Le Lt
Yang etal. 2024c
$8.5
59.7
59.7
58.5
68.0
66.2
63.5
66.3
RM Mistral 70
Dent c 2013
50.9
59.6
51.8
56.8
66.5
66.9
62.1
64.9
Burus RM676
Your
2724
53.3
59.1
59.6
60.5
63.5
69.0
58.4
64.9
BTRM Owenz 78 u0g
53.5
$7.4
61.8
58.4
61.5
69.4
63.8
65.1
Ineemim2 7b rewand Ca al 2024
57.5
53.4
62.1
00.0
67.1
67.1
59.4
65.2
LaGaX--LMMAA-RAA
20255
54.7
62.9
63.1
61.1
70.2
70.5
$9.9
67.5
Co
$2.0
39.0
$7.3
53.6
54.2
66.2
$9.0
58.8
AmooM-LLammdd- 1-50u3------777 -i
30246
90.
66.5
60.6
60.6
64.6
69.2
39.7
674
1 Somem---0fierRiis-BMM. RM-3i
Prikersl
1024
50.0
64.5
90.2
64.1
57.8
71.3
63.5
67
ORM-UJamd 1-88-22 (Dunke
93.1
70.7
57.2
60.3
61.1
72.5
62.
68.2
GRM Ilama3 (BB -8-88-dijji disjill
66.2
58.9
63.2
68.8
63.1
676
ORM-Llama), 1-3B (Derc)
93.1
70.7
60.6
64.7
72.8
633
69.
Vong
10:4g
91.5
67.7
60.0
70.2
69.9
62.3
69
URM LLaMa 3:
92.9
Skywork Reward Liuth
20
92.5
69.2
62.0
Skyuntl-Revand Seraud---lu---11 11
RMM34B
2013
30.8
62.8
72,0
ORM-Gemua-2-275
94.4
53.4
Incmlm2--06--raee
62.9
Skyamt--Rvvnl--uu
3004h
Nomooron
LDI-Rewand Gemmas
Skywork-Rownn
Llama3.1
Yang
70.5
70.2
LLM Juige Generathie Reward Models
GPT-44 (Hunt al 204
$6.7
64.9
67.7
67.1
73.8
73.1
$9.8
20.4
Challo-3:33mmm
Aallmnni 5024
54.2
64.7
67.3
69.2
70.6
74.5
64.8
70.3
DeppSol--GMM2211 GRMM-17
$8.5
65.3
60.4
60.0
DempSok--OMM-77)
MrcitAl tinte
105
50.4
67.2
63.2
70.3
Mu.Rl--Qum-nnemet-10 (Chen 20133
92.9
73.0
79.1
RM DeepSeck Distil 0400
2025g
90.9
69.8
EvulPlanerr (Liamo 3:17 Si
93.9
50.9
BralPlanner (LJama:3.1-000
93.8
56.6
]1-Lluma-Ke (Wincchougg
60.3
42.0
Lluna
60.6
Liuna 708 (Whitehrsee
93.3
66.3
60.0
11-Lhms--788 (32)
2025
Our Revurd Models
Skysork Rewand V2 Qwee0-030000 0.60
85.2
61.3
65.3
68.3
74.5
24.4
67.6
70.9
Skysurk Rewan-- V2-Qwen3-1.7
50.3
68.3
67.6
20.5
78.1
78.7
72.9
75.2
Skywodk 22-WWwaa344
93.4
75.5
69.5
74.7
80.0
81.6
69.3
77.8
Skywud Reward V2 (wea3
93.7
7H.2
70.6
75.1
$1.2
82.6
73.4
79.3
Skywur: Rewand V2 Liama 3.2
39.9
64.3
56.6
67.4
76.7
76.3
65.0
72.3
Skymmik-Rewanh Bel.eaan-VV-LLnm----.
93.0
74.7
69.1
72.1
80.5
81.1
69.2
77.1
Styunll-Rewadd 32-LLamm-3.1-88
06.4
$4.1
17.3
$3.4
96.4
12.8
30.0
85.3
Skyantk-Reemn V3-L.ama-3.1---- 1-88--0M
97.8
86.5
79.8
87.2
89.3
96.8
83.
88.6
Table Open reward model performance on seven reward model benchmarks.
D DATASET PROCESSING DETAILS
D.1 PRE-PROCESSING, DEDUPLICATION, AND DECONTAMINATION
For pre-processing, we perform simple structural check to remove preference pairs in which either the chosen or rejected response contains None as content. This ensures valid formatting of the conversation.
To climinate potential duplicates within or across datasets, we perform global deduplication across all available data sources at the time. Specifically, for each chosen-rejected pair, we represent the sample using the tuple (conversat (eatteddeeaposee and discard any duplicates. The conversation history includes all prior user and assistant turns, while the chosen and rejected responses refer to the assistant's final turn.
history, chosen response,
To ensure decontamination from benchmark data, we remove any instances that share least one 13-gram overlap with a (first-turn) prompt from any of the evaluation benchmarks. For this, we employ decontamination script previously used clean preference datasets against RewardBench d
22

---

Published as a conference paper at ICLR 2026
Extended decontamination validation. To further guarantee decontamination effectiveness, performed additional strict checks beyond the initial 13-gram filter. We expanded the n n-gram win- dow to range from 5-grams to 13-grams and applied matching not only on prompts but the full (prompt, chosen response, rejected response) triples. We then used a fron- tier L.M (Qwen3-235B-A22B. Instructttt))))5577 as judge to filter out false positives common phrases that trigger n-gram matches but do not represent true contamination. This two-step process confirmed that none of the newly identified samples constituted actual contamination. Even in the initial 13- 3-gram decontamination round, approximately 23% of flagged samples were false positives that were correctly excluded after manual inspection. For Deoid specifically, we enforce a strict zero-overlap policy to eliminate any risk of contamination from human-verified training data.
Dataset licensing. The SynPref-40M dataset aggregated from publicly available preference pairs whose licenses and terms of use permit redistribution. To ensure legal compliance, we maintain: (1) a license file specifying the terms, (2) an attribution file documenting the source of each dataset along with their respective licenses where applicable, and (3) usage file clarifying downstream compliance requirements.
D.2 DATASET COMPOSITION AND CHARACTERISTICS
The SynPref-40M dataset consists solely of publicly available preference pairs, primarily collected from over 40 diverse sources on Hugging Face, provided that the licenses and terms of use those sources permit redistribution. The majority of the collected samples (over 99% of the full dataset) contain synthetic prompts and/or responses generated by different LLMs, and the remainder are written by real humans, based the original descriptions of the source datasets.
Task category distribution. During Stage 1 LLM labeling, we adopted the task categorization from Xu et al. (2024) and obtained the following prompt-wise distribution across the dataset. The distribution shows that information seeking and coding tasks dominate, accounting for over 70% of the data, followed by advice-seeking, mathematics, and creative writing tasks. This diversity ensures broad coverage of different types of user queries and preferences.
Category
Percentage
Information seeking
40.4%
Coding & Debugging
32.7%
Advice seeking Math
10.9%
7.5%
Creative writing
4.1%
Reasoning
2.2%
Planning
0.62%
Data analysis
0.44%
Editing
0.41%
Role playing
0.37%
Brainstorming
0.16%
Other
0.09%
Table 8: Task category distribution in SynPref-4OM.
Controversiality and objectivity. Based the labels from Stage tion of controversiality level and objectivity of the preference pairs. The majority of preference pairs (73.8%) have low controversiality, indicating relatively clear preference signals. Similarly, 74.8% of the pairs are classified as objective, which aligns well with the high proportion of information seeking, coding, and math tasks.
also estimate the distribu-
Controversiality
Percentage
Objectivity Percentage
Low
73.8%
Objective 74.8% 25.2%
Medium
20.0%
Subjective
High
6.2%
Table 10: Objectivity distribution.
Table 9: Controversiality distribution.
Language distribution. Over 95% of the preference pairs are in English. Roughly 2.5% are in Chinese, and the remainder consists of other languages (e.g., German, French, Spanish).
23

---

Published as a conference paper at ICLR 2026
Rationale for collecting in-the-wild data. There three main reasons collect purely open preference pairs rather than generating responses from scratch. First, we initially attempted to re- sample responses from collected prompts while discarding the original responses, but found that this approach does not scale well given our budget constraints. Second, our goal is to develop a robust pipeline that can handle realistic challenges diverse and large quantities of non-uniform, potentially low-quality data without making strong assumptions about the source or type. Third, we aim demonstrate that significant value has been hidden in existing in-the-wil wild preference data; it simply has not been properly extracted previously.
want the pipeline to be able process
Data allocation across stages. Before starting each iteration of the curation pipeline, we strictly perform deduplication and decontamination as described above. Throughout the pipeline, we al- located 80K pairs in the seed stage, fewer than 1M pairs Stage 1, and the remainder in Stage 2.
D.3 PRIVACY AND PII ANALYSIS
To ensure responsible data practices, we conducted comprehensive analysis to identify and mitigate potential privacy risks in the SynPref-40M dataset.
We performed single-pass scan over all (prompt, chosen.response, re rejectedd.esponsee triples using an LLM-as-a-Judge identify po- tential personally identifiable information (PII). The LLM was instructed to extract specific PII instances and assign a confidence score ranging from to 3, indicating the sensitivity level and whether the information should be removed. This initial scan flagged approximately 0.07% of the dataset 28K samples) with a positive score, as detailed Table I1
PII detection methodology.
Sensitivity Score
Number of Samples
(minimal concern)
17,960
(low sensitivity)
5,498
2 (moderate sensitivity)
3,261
3 (high sensitivity)
1,478
Total
28,197
Table 11: Distribution of PII sensitivity scores across the dataset.
Human verification and characterization. We conducted human verification on a random sample from each sensitivity level. None of the samples with scores 0 or I were confirmed as genuine PII by human annotators. For samples with scores or 3, we performed a second-pass analysis using GPT-4o-mini. This analysis revealed that most flagged samples (293%) contain indirect identifiers such as age, gender, birthdates, demographics, or fictional usernames, rather than direct personal identifiers that could compromise individual privacy.
Sensitivity analysis of reward models to PII. To verify that the trained reward models do not exhibit sensitivity to PII, constructed test pairs where PII was either removed swapped with neutral placeholders. As shown in Table [2] the reward models maintain near-perfect accuracy (approaching 100%) on these modified pairs, indicating they are not learning spurious correlations based on PII and instead focus on substantive preference signals.
Sensitivity Score
PII Removed
PII Swapped
2 (moderate)
100.0% 99.86%
99.85%
3 (high)
99.46%
Table Reward model accuracy on preference pairs with PII removed or swapped, demonstrating insensitivity to PII presence.
These results demonstrate that SynPref-40M contains minimal genuine PII, primarily consists of indirect identifiers in synthetic contexts, and that the trained reward models do not rely on such information for preference judgments.
24

---

Published as a conference paper at ICLR 2026
D.4 HANDLING INTRANSITIVITY AND CONFLICTING PREFERENCES
Human preferences are often intransitive and context-dependent, as observed in recent work (Duan al. 2024). Rather than assuming global transitivity in the raw data, our pipeline designed to identify and localize inconsistent preference regions, including intransitive cycles and near ties, before they dominate training. We use a transitive Bradley- -Teryy (BT) model as a smooth surrogate that approximates a noisy, partially intransitive preference graph.
Where intransitivity arises. Intransitive cycles (e.g., A > B, B > C, but A) typically emerge as inconsistent clusters of pairwise labels over similar prompts and responses. Common scenarios include near ties between stylistically different but substantively similar answers, subjective tasks with multiple defensible "best" answers, or conflicting preferences from different annotator groups.
Quality control mechanisms. Our pipeline addresses intransitivity through three complementary mechanisms:
1. Stage 1 metadata isolates "risky" regions. Every pair in the unverified pool receives pref- erence attributes from LLMs: task category, objectivity, controversiality, desired attributes, and annotation guideline. This stratification identifies
versus subjeetivehiigh-cotrvvvessa regions, where intransitivity is more common. Internal analysis shows roughly 75% of pairs are objective and 74% are low controversial, with the remaining quarter concentrated in more subjective, contentious tasks where cycles and label conflicts clus- ter.
2. drioe-driven adaptive retrieval focuses on "unstable" regions. In Stage 1, we repeatedly train an RM, evaluate it on human-verified gold data, and use error-driven adaptive retrieval to pull in new examples similar (in prompt + attribute space) to misclassified or low-confidence pairs. This concentrates labeling effort where the current BT model finds the pairwise graph hard to linearize, empirically corresponding to regions with local intransitivity, near ties, or subtle spurious correlations.
3. Stage 2 dual-RM consistency filtering targets contradictory signals. Stage introduces a consistency filter: train gold RM cumulative human-verified samples and use it together with the Stage- best RM to decide which in-the-wild pairs to keep or flip. We retain pairs whose original chosen/rejected labels agree with the gold RM and either the Stage-1 best RM or the LLM judges. This serves as a consistency check over local preference subgraphs: if the raw annotation induces cycles that contradict the human-aligned gold RM, such edges are corrected or deignweighted.
Empirical evidence of consistency improvement. The human agreement analysis in Table 14 demonstrates that our hybrid approach (LLM + human + adaptive retrieval) achieves higher agree- ment (93% objective, 84% subjective) than pure human annotation (81%, 76%) or pure LLM annota- tion (75%, 63%). Additionally, as shown in Table 19 our dual-RM filtering achieves 86% agreement on kept pairs and 92% on flipped pairs, indicating that the pipeline actively resolves conflicting local preferences rather than amplifying cycles.
Relationship to explicitly intransitive models. Our pipeline is agnostic to the downstream RM parameterization. The same curated data and consistency filters can be used to train generalized in- transitive preference models (Duan et al. 2024) rather than strict BT models. We chose BT primarily for comparability and simplicity, intransitive preference models with SynPref-40M is an interesting future direction.
remains the dominant choice in open RM work. Integrating
E ANNOTATION DETAILS
E LLM PREFERENCE ATTRIBUTES LABELING
Before the verification and annotation process, our preference attributes are generated from a com- bination of API and local models, including Claude-3.5-Sonae (Anthropic) 2024), GPT-4o (Hurst ctt al. 2024), 04-mini (OpenAl 2025), DeepSeek-V3 (Liu et al.] 2024a). 3.3 Instructt (Grattafiori et al. 2024), Llama-3.1 3.1-70B Instruct (Grattafiori al.] 2024). Owen2.5-72B-Inntte
Yang et al. 2024a). Qwen3- (Yang et al. 2025). Qwen3-14B (Yang et al. 2025).
Quality analysis of preference attributes. To evaluate the quality of the LLM -eneerrret preference attributes, we randomly sampled 500 items and performed a human quality check. For category, controversiality level, and objectivity, human annotators provided binary judgment (yes no)
25

---

Published as a conference paper at ICLR 2026
on whether the correct label was provided. For desired attributes and annotation guideline, human annotators rated the quality on a scale of to 5. The results, shown in Table demonstrate that the category and controversiality level are generally well-aligned with the LLM annotations, with over 90% agreement rate. Objectivity is slightly lower at 87.4%. The desired attributes and annotation guideline generally well-annotated with ratings above 4.0, which is consistent with the LLM annotations.
Attribute
Agreement Rate / Average Rating
Category
96.2%
Controversiality Level
90.6%
Objectivity
87.4%
Desired Attributes
4.52 (rating)
Annotation Guideline
4.03 (rating)
Table 13: Quality assessment of LLMM-generated preference attributes via human verification random sample of 500 items.
Examples of preference attributes. Due to space constraints, we provide examples of the prefer- ence attributes and their corresponding preference pairs in the supplementary materials. These ex- amples illustrate the 5-tuple structure (task category. objectivity, controversiality, desired attributes, and annotation guideline) for various types of preference pairs in our dataset.
E.2 HUMAN VERIFICATION AND ANNOTATION PROTOCOL
LLM usage during human verification. During our human annotation pipeline, annotators allowed to use external tools such as search engine or frontier LLMs, including GPT-4o (Hurst al., 2024). all o-series models (Jaech al., 2024, OpenAl 2025), Gemini (2.0 Flash, 2.5 Flash, 2.5 Pro) (leam et al. 2023), Claude (3.5 Sonnet and 3.7-Somnet) (Anthropic 2024). Grok (2 and 3) XAI 2024] 2025). DeepSeek-V V3 (Liu et al. 2024a), and DeepSeek- RI (Guo al.) 2025). However, we design striet guidelines for using these tools, and specify detailed guidelines for different tasks, objectivity type, and controversiality level.
Batched pre-verification. To speed up annotation, we prioritize preference pairs labeled as "objec- tive," and pre-verify them with L.MMs a batched way. Specifically, we use a set of query templates embedded with the conversation with single response, and the LLM provides final judgment of correct or incorrect. During human annotation, the annotator still reads the response in general. This drastically improves efficiency, because annotators longer need interact with the LLM for verification and annotation.
Verification and annotation priority. During initial inspection of the data pool, found that many preference data pairs contain extremely ambiguous preference signals, even with the provided attributes. In some conversations, the asks vague questions, and both assistant responses seek clarification, differing only in phrasing. As result, we use the preference attributes to prioritize annotating objective and low-controversiiality preferences. If an annotator cannot determine the preference relationship from the pairwise data, we skip the LLM annotation process and discard it.
In the later stages of the project, we recognized that potentially more valuable approach is to use LLMs to label the differences between the two candidate responses and prioritize the annotation of these samples. However, due the high inference cost associated with millions of samples, we will continue with our original approach in this work and leave this for future research.
E.3 LLM-AS-A-JUDGE LABELING
For I.LM-as-a-Judge labeling, we employ the same verification and annotation guideline used by human annotators but remove all sentences mentioning LLM usage and the use of web search for those without web browsing capabilities. Toward the end of the guideline, we provide at most eight concatenated pairwise instances and their corresponding preference attributes, and the target pairwise instance for labeling.
LLMs used for annotation. The list of LLMs used for preference- annotation includes both chat-based models and advanced agentic LLMs. In the initial stages, we used models includ- ing Claude-3.55 3.5none,,,,, GPT-40, o4-mini, DeepSeek-V3. V3, 1--ma-..--BB--nsttie Llama-3.1-700
26

---

Published as a conference paper at ICLR 2026
Instruct, Qwen2.5-72B Instructt, Qwen3-32B, and Qwen3-14B. In the final stage, incorporated more advanced agentic LLMs to target more complex tasks, including Deep Research, Gemini 2.5 Pro (with search), Claude-4-Sonne (with search), Grok-4 (with search), GLM-4.5, Kimi-K2, and 1PT-4.1. We also replaced weaker general chat-hased models below 70B with the latest frontier open models at the time, such as (wen3-2358B-2221 DeepSeek-V3.1, and GPT-OSS-120B.
Annotation prompts. Due to space constraints in the main paper, we provide the complete an- notation prompts used for preference-aware LLM labeling in the supplementary materials. These prompts include the preference attributes (task category, objectivity, controversiality, desired at- tributes, and annotation guideline), the retrieved few-shot examples from Dgold- and the target pref- erence pair to be labeled.
E.4 LESSONS LEARNED FROM VERIFYING AND ANNOTATING HUMAN PREFERENCES IN-THE-WILD
While we initially include in-house human annotators, the authors also participate in the later stages of the annotation process. Here, we share the lessons we learned and some discussions from our annotation efforts.
1. LLMs can effectively automate certain types of annotation. For conversations involving reasoning tasks such as math problems or coding questions, LLMs are more efficient and reliable than human annotators. Human annotators may not be experts in all types of math and coding problems. We emphasize using autting-edge models for this purpose, particularly those with advanced reasoning capabilities. Our inspection of early annotations reveals that different LLMs exhibit strong annotation bias. This bias arises from various sources, including scenarios with multiple or no ground-trutt truth answers, which are highly context-dependent, and those requiring external knowledge. We believe this issue be mitigated in the era of agents (Luo et al.) 2025a), given their ability to perform web searches or conduct deeper research. In practice, find that over 90% of objective preference pairs involve mostly information seeking (e.g.. fact-checking), math/code problems, or general/specialized domain knowledge (c.g., literature review, movie plot summary). While humans alone can certainly perform well on these tasks, LLMs with tools are far more efficient and, in most cases, less expensive regarding annotation costs. We also observe prompts whose chosen-rejected relationship cannot be easily determined by humans who are not domain experts. In such cases, LLMs with tools are the only practical solution (assuming we do not pay for expensive expert annotation services).
To further validate the effectiveness of human-guided LLM curation, we conducted agree- ment rate analysis on D ggold our human- verified dataset- - by re-annotating these samples un- der different annotation variants. We divided D gold into objective preferences moon-controversiall labels) and subjective preferences (potentially controversial labels), and measured how each an- notation method agrees with the original human labels. As shown in Table [4] the agreement rate increases significantly from pure LLM annotation to LLM + human curation, and further to LLM human + adaptively retrieved samples. Notably, even pure human annotation (without LLM assistance for fact-checking or domain knowledge) performs worse than LLM + human curation, particularly on objective tasks where LLMs with tools excel at verification.
Annotation Variant
Objective
Subjective
Pure LLM
75%
63%
LLM + human curation
87%
77%
LLM + human + adaptive retrieval
93%
84%
Pure human (no I.I.M tools)
81%
76%
Table 14: Agreement rate between different annotation variants and the original human labels in Dguld- The results demonstrate that human-guided LLM curation with adaptive retrieval achieves the highest agreement, outperforming both pure LLM and pure human annotation.
2. Human preferences are complicated, even for humans. During annotation, we consistently encountered preference pairs that were ambiguous, subjective, or context-dependent - mak- ing it difficult even for trained annotators to confidently determine which response was better. Factors like subtle tone differences, varying expectations around informativeness or safety, and individual annotator biases introduced uncertainty into this process. This highlights a key chal- lenge in reward modeling: even with structured annotation protocols and strong preference at- tributes, some preferences are inherently ill-defined non-universal. This problem stems from
27

---

Published as a conference paper at ICLR 2026
the concept of human preferences and their diversity. It also raises the question of whether single reward model can effectively capture this diverse range of human preferences. In our ini- tial experiments, observed that (1) if mix pairs with opposite preferences, reward models tend to learn spurious correlations (e.g., pure text format), and (2) if we pre-generate preference specifications ID'Osterllinck al. 2025) for the preference pairs, not only does pure L.LM annotation quality improve (when we provide such additional information), but we can also leverage this information to avoid reward models learning spurious correlations (by avoiding conflicting preferences). However, we chose not to include these findings in the main paper as they were under-studied and might overeomplicate this work. We consider them valuable directions for future exploration.
3. Learning clear and aligned preferences significantly enhances reward models. Our ex- periments demonstrate that when reward models are trained on preference data that is well- structured, verified, and guided by clear annotation protocols, their performance improves sub- stantially across all evaluation benchmarks. We hypothesize that this may be due to the signifi- cantly higher requirement for constructing preference pairs in the benchmark dataset. While we do not have quantitative results, reviewing the preference pairs presented in multiple test sets veals a strong preference signal. This also highlights a fundamental flaw in the design of today's preference data: although the response pairs are provided, the actual difference between them the core indication of preference is ignored. This raises concerns about what reward models, or any other types of models that provide reward signal, actually learn from underspecified responses.
E.5 ANNOTATOR INFORMATION
The annotation process involved fewer than 20 trained annotators across both the seed stage and Stage 1. In the seed stage, one author participated, while additional authors contributed during Stage 1. These author contributions were voluntary, intended to expedite progress, and were not compensated. Each preference pair annotation required between minimum of 10 seconds and maximum of 5 minutes to complete. On average, the team generated approximately 2,000 to 3,000 annotations per week. The cost of producing each annotated preference pair estimated to range between $0.10 and $0.70. Overall, the full annotation effort extended over a period of roughly nine months.
F PRACTICAL GUIDANCE FOR COST-PERFORMANCE BUDGETING
Given the importance of planning preference data curation under resource constraints, we provide practical guidance achieving target reward model performance within specified cost budget.
F.1 WHAT OUR SCALING RESULTS REVEAL ABOUT COST
Section 4.3 presents data quantity and quality ablations based on an earlier 16M-pair mixture. Two key findings emerge: (I) uncurated scaling fails adding 12M uncurated preference pairs on top of the seed set yields almost no performance gain, and (2) curated scaling succeeds - with Stage Stage 2 curation, performance improves steadily, with the largest gains in Stage 2.
Most importantly for budgeting, we find that training on just 1.8% of the 16M curated mixture ( 290K pairs) already surpasses the previous open SOTA 70B RM at the 8B scale. This clear "quality beats volume" statement: carefully curated hundreds of thousands of pairs suffice to beat prior sfate-of-the-arn without requiring tens of millions of new, expensive human labels.
In our pipeline, fewer than 500K pairs pass through full human verification in Stage 1, with the remaining tens of millions curated automatically in Stage Human effort comprises only a couple percent of the final training pool but drives most performance gains.
F.2 A SIMPLE BUDGETING RECIPE
We outline a practical framework for planning preference data curation under cost budget:
1. Define target performance. Let the desired average score across the six main benchmarks (excluding RewardBench v2) be Sunget- Our scaling curve in Figure shows the relationship between fraction of curated data and average RM score.
2. Estimate required curated pairs. From Figure 4 practitioners can read off conservative fraction f of the full curated mixture needed reach Starget. For example, f 0.018 ( 290K
28

---

Published as a conference paper at ICLR 2026
pairs) already exceeds previous open SOTA 8B. Higher targets correspond larger f, but with diminishing returns.
3. Decompose costs by stage. Our pipeline separates labeling into three cost regimes:
Gold human labels (Stage 1, goli)): cost Qn per pair, high leverage, used to train the gold
RM and seed attribute generation and L.LM judges.
Silver human- guided LLM labels (Stage 1, Dsivve)): cost cui per pair (LLM inference, often 1-2 orders of magnitude cheaper than full human annotation), guided by human-labeled neighbors.
Large-scale consistency curation (Stage 2): cost CL2 per pair (mainly RM inference oc- casional LLM annotation), used to scale from hundreds of thousands to tens of millions of pairs.
Total curation cost is approximately:
B = |Dootal + CLI DDsiverl + CL2 DSuage2
In practice, CH CLI 2 CL2 and Dgotal
[Ssuge2), so the gold set dominates quality while <
automatic stages dominate quantity.
4. Allocation strategy. Given a dollar budget Bmax. allocate a gold budget Bμ I Buax to de- termine g. Our results suggest that roughly O(10)) carefully-selecter gold pairs suffice to train strong gold and Stage- RMs. Allocate the remaining budget Bmax Bu to scaling Stage-2 curation, trading off total curated volume versus LLM quality (e.g., using cheaper versus more capable judges).
5. Validation and stopping criteria. Monitor (1) RM benchmark scores as in Figure and (2) downstream BoN curves (c.g.. PPE Correctness and RMB) where we observe monotonic scaling with N. Once incremental gains per additional curated million pairs fall below a user-defind threshold (e.g., j0.3 points on average benchmark score), it is reasonable to stop spending.
F.3 WORKED EXAMPLE
Suppose a practitioner has a budget of $50,000 and seeks match or exceed the previous SOTA 70B RM using an 8B model. Based on our findings:
Target: Starget x 73.5 (INF-ORM-Llamas.. 1-7B average score across six benchmarks).
Required data: f x 0.018 of a 16M mixture, roughly 290K curated pairs.
Cost breakdown (assuming CH = $0.50, CLI $0.05, Q22 = $0.01):
Allocate 100K pairs to Dgold: 100K $0.50 $50K.
This exhausts the budget, but the gold set alone is sufficient to train a strong gold R.M.
In practice, one could reduce ggalal to 50K (S25K). then allocate the remaining $25K to Stage
1 silver labels (500K pairs at $0.05) and Stage 2 curation (several million pairs at $0.01). Outcome: With careful allocation across stages, $50K can produce curated dataset exceeding 1M pairs, sufficient to reach or exceed the target performance.
This example illustrates how practitioners can use our stage-wise cost decomposition and scaling curves to plan curation budgets systematically, balancing human verification quality with automated scalability.
F.4 RELATIONSHIP TO MECHANISM DESIGN APPROACHES
Recent work on mechanism design for preference learning (Zhag et al. 2024a) explores which comparisons to query and how structure incentives to extract maximal information from limited human comparisons. We view mechanism-desige approaches and our human curation pipeline as operating at complementary layers of the RLHF stack:
Mechanism design (Zhang et al., 2024a): focuses on which comparisons to ask for and how structure mncntives/queriis to extract maximal information from limited human comparisons, often under stylized model where one controls the querying process but not a large, messy in-the-wild pool.
SynPref-40M: focuses on how extract value from already-existinn heterogeneous, syntheti- cally labeled preference data, using human guidance and LLM+RM consistency to filter, correct, and scale.
We see at least two concrete points contact that highlight potential for integration:
1. Mechanism design as inner loop in Stage 1. Our ervoedrrvven adaptive retrieval already behaves like a "targeted querying mechanism" over the unverified pool. Future work could replace simple similarity-based retrieval with mechanism--esign-inniii query query selection rule,
29

---

Published as a conference paper at ICLR 2026
e.g., selecting pairs that maximally reduce posterior uncertainty in generalized pairwise model under a fixed human budget.
2. Using mechanism-desee work (Zhang et al. 2024a) suggests principled criteria for which pairwise comparisons are most information-effen -efiiattoo-effiiennt. Combined with our Stage- wiiee-ise cost decomposition, this could inform bet- ter allocation of gold human labels across task types and controversiality levels, focusing human effort where marginal information gain is highest.
insights to set budgets and stopping rules. Zhang al.'s frame-
In summary, while mechanism design operates at the algorithmic level to optimize query selection, our approach operates at the data-centric level handle realistic, large-scale, heterogeneous prefer- ence data. These complementary perspectives can be integrated: mechanism design can guide which samples to prioritize for human annotation within our pipeline, while our curation mechanisms can handle the messy realities of in-the-wild data that mechanism design typically abstracts away.
G TRAINING DETAILS AND HYPERPARAMETERS
We primarily adhere to the hyperparameter choices outlined in Lambert et al. (022) and Wang al.) (2025a). During the development phase, we adjust the learning rates according to the model size, using le-6 for all 8B models and 4e-6 for all other sizes. All models are trained with a global batch size of 256 and a linear learning rate decay, using a warmup schedule for only epoch, with a maximum token length of 16.384. For all final training runs, we switch to learning rate of 3e-6 and a large global batch size of 10,240 for all models, following Wang et al. (2025a), due to its faster convergence and negligible impact performance. All models are trained using 64 x H800 GPUs with DpepSpeed ZeRO Stage 1 (Rasley et al. 2020).
H ADDITIONAL EXPERIMENTS
H.1 EXISTING (UNCURATED) PREFERENCE DATASETS ARE INADEQUATE
To evaluate the effectiveness of the landscape of open preference datasets, we source almost all existing popular preference datasets from Hugging Face. We train a single reward model in the same way as we train ours each of the preference dataset and the combination of all preference data. We present the full results in Table T5]
We demonstrate that none of the single preference datasets or the combination of all datasets out- perform our curated mixture. Using almo-2-0425- ab-preference-mine alone results in an average score of 69,4. In contrast, combining all datasets yields only 68.9, decrease of 0.5 points. This further validates that preference scaling cannot achieved by simply accumulating the number of preference pairs.
H.2 DOWNSTREAM RLHF EVALUATION AND HUMAN EVALUATION
Policy optimization. Other than the preference scoring benchmarks in the main paper, we perform additional downstream RLHF training. We largely follow the setting by Chang et al. (2025). but only differ in the set of prompts. For prompts, we use a set of hard prompts that are selected both manually and automatically from our preference data pool. We evaluated policies trained using our RM versus the previous state-of-the-art RMs with similar size. We observe that the resulting policy outperforms not only policies trained by the baseline RM but also official instruct models (Table 17, indicating the RM generalizes to training time rewards for instruction following.
Human evaluation. Given that most of the preference benchmarks' labels are generated either synthetically automatically, further perform real-human agreement assessment against trained reward models on an internal hold-out preference benchmark. We show that reward models trained the curated preference mixture obtain significantly higher preference agreement with humans in Table 16
H.3 THE EFFECTIVENESS OF THE CURATED MIXTURE ACROSS VARIOUS BACKBONES
In the main paper, we only use the Llama (Grattafiori et al. 2024 and Qwen3 Yang et al. 2025) backbones to train reward models. To prove that the proposed curated mixture works "univer- sally" across different model families, we consider additional backbones from Gemma TTem et al. 2024 2025) and Qwen2.5 (Hui et al. 2024) families. We also attach the scores from INF-ORM-
30

---

Published as a conference paper at ICLR 2026
Revantflench2
Judp-Bonchh
All combaadd
76.5
65.5
65.5
63.3
73.3
70.2
64.0
68.9
34.
66.1
63.
514
71.4
71.4
66.5
69
dlenaliine 2 1124 136 pecfarence nix
$1.9
66.1
535
62.1
72.
70.3
66.0
60.0
H.Fiillaarrrann Jasugh wealaty
preferacce standad
allzneellarraa.
bendecooggyrrteeengg 200K
636
allenillarra-.. Ludu 3-40-----000 1---0t---oeeeee proe
83.1
63.5
646
614
72.2
71.0
64.9
66.7
alceaaloim---111 2-1112--h,, profertcc--nn mis peefernnce nis
81.6
639
62M
624
72.3
71.1
03.
66.
allmsidinn-2
2.0025-322
81.6
63.
544
62.6
71.
712
64.5
68.5
mapeCtiis
61.1
677
619
618
www.beedee/uttileeee Feedbuck
649
58.1
623
64.
66.0
67.5
allenahhuto preferenceddan da13
76.7
660
60.6
704
714
679
67
Magic-Alinn Alin-Suppi-iianmmmnmm I-Poo-PF--1000
87.7
39.7
61.7
80.0
64.
72.1
63.3
67.0
Magrie AligwMagge-Aie Air----------- -01-
00.
617
01.
71D
MA
BLHFowlpai
SLHFaw distbaie
60.1
TKIER- aslAdCaddPea MOK
50.5
566
19.7
72.5
65.0
650
Viacesmintti dhf
84.4
53.2
58.7
62.4
59.3
68.1
619
64.
RL L.L.Paaw/hueccllanddeeeeeee weweruddeeeemme-sniiiiii
86.0
$1.0
605
58.5
63.5
88.3
SK.7
63R
83.7
615
35.9
593
66.3
611
63.6
Clindniennc pai
600
603
78.9
61
86.1
609
Imsees diizena luman prefernce 55k
25.0
54.3
67.1
64.0
59.0
55.6
626
625
It M.lFFollllllpzzzeeeee pespssssggneeeeesennnnee
78.8
569
10.
55.3
A5S
617
Iinzesa ditreu buran prete ence 100s
74.5
51.
094
10.3
53.3
3IA
598
617
Venorr/acce Preferznce Pars
78.5
5000
181
57.3
51.3
619
63.9
616
61.4
Ste--PP--10
NCSOFThulledias
51.3
57.9
63.5
45.1
67.5
194
HuzzindFaccele Oaeei//peeeeeeesssssss preferences dedaped
65.0
47.1
60.
53.6
57.5
$1.9
54.2
55.6
69.1
40.0
57.
55.4
83.3
58.6
56.
55.5
RLAPoWHEH SLHF Hamnless and RedTeam suntaed
51.3
31.3
419
492
36.1
56.3
47.4
44.8
Table 15: Benchmarking the effectiveness of all existing popular preference datasets.
RM
Agreement with human
GPT-4o
74,3
Slaude-3.5-Some
72.1
71.0
75.6
Boooo--RewandV---Llaaaa
81.2
Table 16: Agreement between different reward models (RMs) and human judgment.
Model
Method
ArenaHardvl
ArenaHardy2
MT-Bench
WildBench
Avg.
Llama-3.1 1-8B
Base
2.0
52.8
29.1
+SFT
12.6
3.1
56.8
60.3
33.2
+RL (Skywork- Sowort--Roant--Lam--------
1.6
57.1
57.8
31.6
+RL Seneed--Rewar-GGemmaaa 22-------222-02
14.0
3.8
58.5
61.5
34.4
(Skywork Reward Qwen
+RI. (Skywork-RRee Reward. Rewardd-V--llama. V2-I.ama- 1-88B)
20.8
6.3
66.5
70.2
40,9
Instruct (official)
5.8
65.7
40.2
Qwen2.5-7B
Base
16.2
5.6
63.5
51.8
34.3
+SFT
22.1
9.9
67.3
60.5
40.0
+RL Beane--Rewar--Llannnee
12.2
+RL (Skywork Reward Gemma-22 2-27B v0.2)
34.5
15.5
78.2
67.8
49.0
+RI. (Skywork-kRen www.erd-2..-ennnna.aa
17.9
79.0
50.2
+RL Rewoorr--Raard-2---lame- 1-8B)
38.0
18.5
81.1
71.5
52.3
Instruct (official)
37.9
17.1
78.8
70.9
51.2
Table 17: Performance comparison of L.ama-3.1-8B and Qwen2.5-7B across ArenaHard, MT- Bench, and WildBench.
31

---

Published as a conference paper at ICLR 2026
RewardBlecch2 PPKHumanPrd
PPRCornetnes
JudgeBRnnhh
Ang.
NNL-0RM--Llmm3... 1-703317088
64.2
64.4
70.2
0wea2,5-78
66.4
73.8
78.3
URRAMM/BTTRM.COM 1613
6001
63.1
67.9
71.2
Ray233/GRM gemuma- 28wweaddddddddl--
53.7
53.4
62.0
78.1
76.9
82.0
83.9
77.9
Ih-ll 3.40
73.8
75.7
Table 18: Comparison of models across multiple reward model and preference benchmarks.
Llama3. 1-BB, the current best RM, for comparison. In Table our own models, even with smaller backbones, consistently outperform this baseline. This highlights the effectiveness of our preference curation: it enables smaller models to exceed the performance of much larger ones. Additionally, for RMs based on Qwen2.5-7B -----Inntttuct and Gemma-2-2B, we can directly compare to counterparts trained by other teams, which further demonstrates the benefit our dataset.
H.4 THE EFFECTIVENESS OF STAGE AGREEMENT-ONE O FILTERING
We conducted a rigorous evaluation to assess whether our Stage consistency- based filtering ampli- fies or mitigates systematic biases and spurious correlations. Specifically, we examined whether the filtering mechanism - which keeps pairs agreeing with both the best RM and gold RM, and flips pairs where there is disagreement aligns with actual human preferences.
Evaluation methodology. We randomly sampled preference pairs from both the kept and flipped portions of the unverified pool, where inclusion/llipping decisions were driven by the two-RM filter mechanism described in Section We then conducted human agreement tests to measure whether these filtering decisions aligned with human judgments: for kept pairs, humans should agree with the original labels; for flipped pairs, humans should disagree with the original labels (i.e., agree with the flipped version). We repeated this evaluation using two strong baseline reward models (Skywork- Bearr---eeard-G-emmeee
Reward-Llamaa 1amma-31-88 and agreement among baseline RMs performs comparably.
and their combination to test whether
Reward model used for filtering
Keep (%)
Flip (%)
Seywork-Reward-LLammt Lam
69
57
Heework-Reward-Caee Gemma-2-27B
61
Combined baseline RMs
71
60
Stage Best RM
78
Stage Gold RM
84
88
Stage Best RM Gold RM (Ours)
86
92
Table 19: Human agreement rates for kept and flipped pairs under different filtering mechanisms. Higher percentages indicate better alignment with human preferences. Our dual-RM approach (Best RM + Gold RM) achieves the highest agreement for both kept and flipped pairs, demonstrating that Stage 2 filtering reduces rather than amplifies systematic biases.
Key findings. As shown in Table 19 baseline reward models exhibit relatively poor agreement with human judgments, with the kept pairs and 57% on flipped pairs. Combining the two baseline models does not yield substantial improvement (71 % and 60%, respectively). In stark contrast, our Stage 1 Best RM and Gold RM each achieve much higher agreement rates, with the Best RM reaching 78% and 79%, and the Gold RM reaching 84% and 88% for kept and flipped pairs respectively. When combined, our dual- RM filtering mechanism achieves 86% agreement on kept pairs and an impressive 92% agreement on flipped pairs. These results demonstrate that our Stage 2 filtering approach effectively mitigates rather than amplifies systematic errors and spurious correlations, ensuring that the curated data more closely reflects genuine human preferences.
achieving only 69% agreement on
This analysis directly addresses concerns about potential overfitting to the gold RM's inductive biases. The high agreement rates particularly for flipped pairs indicate that filtering mech- anism successfully identifies preference pairs where the original labels contradict human judgment, rather than simply enforcing arbitrary model preferences learning style biases.
32

---

Published as a conference paper at ICLR 2026
H.S BASELINE EXPERIMENT: LLM RM FILTERING WITHOUT HUMAN GUIDANCE
To address the question of whether the majority of performance improvements stem from LLM an- notation with self-consistency rather than human-guider annotation, we conducted a critical baseline experiment. This baseline uses the best RM to filter out p > 0.5 pairs, then applies only LLM self- consistency annotations the remaining data, without any uuman-guided few-shot examples from Dgold-
We reproduced the same experiment based on the left plot of Figure (from Section but with only best RM LLM filtering. This setup essentially takes the same preference data we accumu- late in each iteration, and performs filtering directly with that specific best RMM checkpoint + LLM annotation, without any other curation involving human guidance.
Training Stage
Original
Filtered
Filtered Corrected
LLM Best RM
LLM Best RM + Corr.
Seed
71.0
73.0
71.0
Iter
74.0
77.0
71.5
71.0
Iter
71.5
74.5
77.5
72.5
72.0
Iter
71.0
74.8
78.8
72.0
71.5
Iter
71.0
75.0
79.0
72,2
72.0
Iter 5
72.5
76.8
82.0
73.4
72.8
Iter
72.0
77.0
82.2
73.0
73.2
Iter 7
71.8
77.2
82.3
74.8
74,0
Iter 8
72.2
79.0
83.0
74.2
74.5
Table 20: Comparison of reward model performance across training iterations with and without human guidance. "LLMM Best RM Filtered" corresponds to "Filtered" but with zero human anno- tation. "LLM Best RM Filtered Corrected" corresponds to "Filtered + Corrected" but with zero human annotation. The results show that LLM filtering alone plateaus around 74-75% while our full recipe with human guidance reaches 83%.
Key findings. While we were not able to perform Stage due to time constraints and annotation costs, the results in ablee[0aalrade demonstrate that L.M filtering alone does not outperform our recipe after only 2-3 iterations, and filtering + corrected does not show the same improvement as our full recipe. By iteration 8, human-guided approach achieves 83%, while the LLM Best RM baseline plateaus around 74-75% This 8-9 point gap demonstrates the critical importance of human-guider annotation rather than purely automatic LLM-based curation.
H.6 LLM-AS--AJUDCE ENSEMBLE PERFORMANCE COMPARISON
To address whether ensembling all strong LLMs used in our annotation system to act as a single judge (with self-consistency) would perform comparably our final trained RMs, we conducted an evaluation across all seven benchmarks. The LLMM-as-a-Judee ensemble includes all models used throughout our annotation process, aggregated via self-consistency. Note that when running this evaluation, the number of completions performed for self-consistency for each model is not uniform, as we could not afford to perform self-consistency with a large number of completions for models like o3 due to cost constraints.
Model
RB
PPE Pref
PPE Corr
RMB
RM-Bench
JudgeBench
Avg
Skywork- Reek---wwar--- Saaaaad-------e 22--Qwen--0666
85.2
61.3
65.3
68.3
74.5
74.4
67.6
70.9
Skywork Reward V2 QwenQwenn33 1.7BB
90.3
68.3
67.6
70.5
78.1
78.7
72.9
75.2
Reyoork-Rewar--vv V2-Qwen3--BBB
93.4
75.5
69.5
74.7
80.6
81.6
69.3
77.8
Skywork Reward V2 QwenQ
93.7
78.2
70.6
75.1
81.2
82.6
73.4
79.3
Skywork-Rewan Beorr--eewrr-WWe V2-L.amm-3.22.
76.7
72.3
Skywork-Rewann Beoork-Reward-Vee Seouurd-V2-Llam-3...0
93.0
74.7
69.1
72.1
80.5
81.1
69.2
77.1
Skywork-Rewand Reward 1.2--lamm--311.1
96.4
84.1
77.3
83.4
86.4
92.8
80.0
85.7
Skywork- Reink--eward--- V22-Lamm--331158BB
97.8
86.5
79.8
87.2
89.3
96.0
83.4
88.6
LLM-as-a-Judge (Agg.)
93.9
83.2
75.7
89.6
82.6
89.0
87.8
86.0
Table 21: Performance comparison of our trained reward models with LLM-as-a-Judge ensemble aggregation. The LL-M-as-a-Judee aggregation outperforms our top JudgeBench, but falls behind in other benchmarks (mostly involving subjective tasks). Overall, our final RM Sooywork-Rewann Saaund-V--LL V2-Llama-3, -...-...----B0-- achieves 88.6 average vs. 86.0 for LLM-as-a- Judge.
PPE Correctness and
33

---

Published as a conference paper at ICLR 2026
Key findings. The results in Table 21 show that the LLM-as-a-Judge aggregation achieves com- petitive performance, particularly excelling on PPE Correctness (89.6%) and JudgeBench (87.8%) which focus objective correctness and code-related tasks where strong LLMs naturally perform well. However, our final trained RM (Skywork- Reward- wa-aadddV---l-a---e-ea 1--Llama-311-88-44000.. 1-88-40M) outperforms the LLM ensemble most other benchmarks, particularly RewardBench (97.8 vs. 93.9), Reward- Bench v2 (86.5 vs. 83.2), PPE Preference (79,8 vs. 75.7), RMB (89.3 vs. 82.6), and RM-Bench (96.0 vs. 89.0). The overall average score of 88.6 for our RM vs. 86.0 for LLM-as-a-Judee demon- strates that distilling knowledge from LLMs into trained reward model through our human-gun -uidedd curation pipeline yields better overall performance, particularly on benchmarks involving subjective preferences, style resistance, and best-of-N selection.
I ETHICS STATEMENT
This work involves the collection and curation of large-scale preference data through human an- notation, raising several ethical considerations that address proactively. Our human annotation process involved workers who were compensated fairly according to industry standards and pro- vided with clear guidelines and training. We ensured that annotators had access external tools and resources to make informed judgments, and we implemented safeguards to prevent worker exploita- tion through reasonable workload distribution and adequate compensation.
The preference dataset created in this work captures human values and preferences that will be used to train reward models for RLHHF applications. We acknowledge that human preferences can sub- jective, culturally dependent, and potentially biased. To mitigate these concerns, we implemented diverse annotation protocols and quality control measures, including multiple validation stages and consistency checks. However, recognize that dataset may still reflect certain demographic or cultural biases present in the annotator pool and the underlying data sources.
Our reward models will be used to guide the behavior of Al systems through RLHF, potentially in- fluencing how these systems interact with users. While our models demonstrate strong performance across safety benchmarks, itoring in downstream applications. We encourage users of our models to conduct thorough safety evaluations in their specific cases and implement appropriate safeguards.
emphasize the importance of careful deployment and continued mon-
We acknowledge the computational resources required for this work and the associated environmen- tal impact, though our focus efficient model architectures (up 8B parameters) helps minimize resource requirements for practitioners.
34


Reviews:
[{"review_id": "UTt50NcgS3", "reviewer": "Reviewer_tDR3", "summary": "SynergyPref-40M introduces a 40M-pair preference dataset and eight reward models (0.6B–8B params) trained via a human-AI curation pipeline. Models achieve SOTA across seven benchmarks, demonstrating strong alignment with human preferences, safety, and bias resistance.", "questions": "Given the budgeting for targeted performance, the paper still provides limited guidance on how to plan, evaluate, and achieve it.\nCan you provide some thoughts on how to achieve a target performance under a given cost budget in dollar terms? \n\nFor your reference, to overcome the cost constraints and to avoid high reliance on the availability of pairwise annotation, mechanism design has been explored as follows:\nhttps://arxiv.org/abs/2409.18417 (Zhang, 2024), leveraging a mechanism design mindset to construct a dataset for RLHF in a cost-efficient manner.\n\nHowever, the existing approaches are in different lines of techniques, more algorithmic than data-driven.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The open contributions and deliverables include a new, large-scale, high-quality preference dataset, a valuable asset for the research community. \n\nIt verified that the brittleness is a root cause of RM underperformance and proposed a solution to it, which is human-AI curation synergy, that contains an elegant hybrid of human and LLM curation, balancing quality and scalability.\n\nEmpirically, Models (1.7B, 8B) achieve SOTA across seven benchmarks, outperforming much larger closed-source RMs.", "weaknesses": "The conclusion of this paper is favorable. However, for pairwise preference, it follows transitive rules. The quality of pairwise preferences can be compromised by intransitivity observed in human annotations. The paper below highlights the existence of such 'intransitivity':\n\n- https://arxiv.org/abs/2409.19325 (Duan et al, 2017) \n\nIn a realistic world where an 'intransitive' relationship accumulates, quality control of the curated dataset is critical, but was not clarified in the proposed pipeline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GdHsx38WQC", "reviewer": "Reviewer_oVFu", "summary": "The paper addresses the perceived brittleness and poor performance of existing open-source reward models (RMs), hypothesizing that the root cause is low-quality preference data rather than flawed modeling techniques. To solve this, the authors propose a two-stage, human-AI synergistic pipeline for large-scale data curation. This pipeline leverages a small set of human-verified \"gold\" data to guide iterative, error-driven LLM annotation and then uses a \"gold RM\" trained on this data and LLM annotations to automatically filter and curate millions of \"in-the-wild\" preference pairs. Training on this data, they produce the Skywork-Reward-V2 series of models (0.6B to 8B parameters), which are shown to achieve new state-of-the-art performance across seven major RM benchmarks, outperforming all existing open RMs.", "questions": "Please address all the points in weakness.\n\nFigure 1 (Left): This chart is not fully convincing because model sizes are omitted. It is impossible to tell if a modification \"fails to yield consistent gain\" or if it is simply being compared to a larger model.\n\nPlease provide the prompts used for the \"preference-aware\" LLM annotation (L239-243) and specify which LLM(s) were used.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "1. The paper demonstrates a clear and significant performance improvement\n2. The release of a new series of top-performing RMs and the massive underlying dataset is a valuable contribution to the open-source ecosystem", "weaknesses": "The paper's primary weaknesses are twofold: (1) a pervasive lack of clarity and the omission of essential methodological details, and (2) as a result, it is difficult to determine the true source of the claimed performance improvements.\n\n**Lack of Clarity and Omission of Essential Details**\n\nThe paper is extremely difficult to follow. The authors have clearly performed a massive amount of work, but the execution is not explained clearly, hindering reproducibility and full comprehension. Many terms are used without definition, and no single subsection seems to provide all the necessary details.\n\na. Ambiguity in Stage 1 Data Curation:\n\nInitial Data: The method for annotating the initial $y_w > y_l$ pairs is not described in the main paper. (The reviewer guesses this might be in Appendix E.1, but it must be stated).\n\nHuman vs. LLM Verification: Line 191 states human annotators follow protocols, but Line 198 mentions an \"LLM-verified\" portion. It is unclear how this LLM verification is performed. Is it the same as the “Preference-aware labeling”?\n\nMissing Examples: The paper provides no example of the \"LLM-generated preference attributes $a$\" (Line 184) or the full 5-tuple. An example and an analysis of the quality of these attributes are needed.\n\nRetrieval Step: In Step 2 (Line 208), it's unclear what data is used for for the retrieval to calculate the similarity. Is it from $D_{gold}$?. This makes Line 215 confusing: how do \"retrieved examples\" from $D_{un}$ come with \"human labels\"?\n\nLine 191, it is said that human annotators perform verification following the protocols. How is the LLM-verified portion at line 198 annotated? Is it annotated by LLM using the designed protocols as described in Appendix E.3?\n\nLLM Annotation Process: The \"preference-aware\" labeling (Lines 239-243) is opaque. The paper must specify which LLMs were used for annotation and provide the prompts. Without the prompt, this core step is irreproducible.\n\nb. Ambiguity in Stage 2 Data Curation:\n\nData Sources: The origins of $D_{un}$ and $D_{wild}$ are never explained.\n\n\"Best Reward Model\": The paper repeatedly refers to \"the best reward model\" used for filtering in Stage 2, but it is not specified which model this is.\n\nFinal Data Composition: It appears that in Stage 2, all pairs with $p>0.5$ are discarded, and all pairs with $p≤0.5$ are re-annotated by an LLM. This critical detail needs confirmation, as it would imply all data in the final dataset is LLM-annotated.\n\nc. Omissions for Ablations:\n\nFigure 6: The ablation on filtering/correction is confusing. It's unclear how these steps are performed during \"iter1-8\" or what the exact training set is. Stage 1 does not involve filtering $D_{wild}$ and correction.\n\nFigure 7: This figure shows five bars, but the text (Line 416) provides only four descriptions. The descriptions are not clear. What is “LLM curation only” and what is “both human and LLM curation”? Does (4) use more data? The exact training set for each of the five bars must be specified.\n\nd. Omissions in Figures and Tables:\n\nFigure 1 (Right): The method for calculating \"correlation\" is not described.\n\nTable 3: The caption should clarify that this metric is \"accuracy on the Best-of-N split,\" not the more common \"Best-of-N accuracy.\"\n\n**Ambiguity Regarding the True Source of Improvement**\n\nBecause the methodology is so poorly described, it is impossible to determine why the model works so well. The paper's core claim is that its human-AI synergistic pipeline is key, but the details suggest that \"a group of strong LLMs\" with self-consistency may be doing most of the heavy lifting. D_{silver} annotations are generated by LLMs. At Stage 2, all the data included after “Preference consistency with the best reward model” are again annotated by LLMs.\n\nMissing Baselines: The paper fails to provide the most critical baseline: what happens if you only use the LLM annotation: LLM-as-a-Judge with self-consistency? A crucial missing experiment is: (1) Use the \"best RM\" to filter out $p>0.5$ pairs, then (2) apply only LLM self-consistency annotations to the remaining data. Comparing this simple baseline to the full Skywork-Reward-V2 would clarify how much the complex, human-guided pipeline actually contributes. As an additional experiment: If you ensemble all the strong LLMs used in your annotation system to act as a single judge (similar to the GPT-4o judge in Table 1) with self-consistency, how does its performance compare to your final trained RMs?\n\nIn summary, the paper presents a complex pipeline. While the results are strong, the lack of clear explanation and proper baselines makes it impossible to validate the authors' claims about why it is strong. With all the missing details, it is contradictory to the reproducibility statement that “We have made extensive efforts to ensure the reproducibility of our work across all components of our research pipeline”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6B3LyhfwSm", "reviewer": "Reviewer_J7rE", "summary": "The paper presents a large-scale preference dataset (SynergyPref-40M) and a series of reward models trained on the dataset. It describes an iterative data curation pipeline merging human and LLM annotations. They show that trained reward models outperform existing models even at smaller model sizes across a wide range of benchmarks. \n\nOverall, this is a solid paper overall, with a clear contribution. The results are very impressive, and given that the dataset and reward models are released as promised, it is a very nice resource for other researchers. The reward model results and ablations are nice. However, the paper lacks more details on the dataset composition, choice of prompts, and generations. The methodology itself is described at a relatively abstract level. The description of evaluation protocols, instructions given to annotators, etc., is limited. Also, no actual examples from the dataset are provided. I expect some improved transparency for a revised version, to ensure reproducibility of results.", "questions": "- I do not follow the conclusion from Figure 1 (or at least I feel it’s too strongly worded). RewardBench still shows positive (and for most benchmarks) decently strong correlation. For example, in Figure 1-Left, I only see a single derivative model that outperforms RewardBench but is noticeably worse on the other benchmarks, so it seems more like an outlier. The general point that there is over-optimization on the reward bench is a totally valid concern, and I believe this to be the case, but i do not draw this conclusion from Figure 1. Could you give your oppinion?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "+ I assume that the dataset itself will be made available as promised. Obviously, a dataset at this scale and level of curation is a very strong contribution to the field\n+ The reported benchmark scores are very impressive, clearly outperforming existing reward models\n+ The evaluation of the trained reward models is thorough and very comprehensive\n+ I really appreciate the provided ablation studies. Evaluating the trade-offs of data curation and dataset scaling is insightful. In particular, sections 4.3 and 4.4 make a strong case for continued human data curation\n+ The paper presents a nice discussion and comparison of recent reward modeling benchmarks", "weaknesses": "- The actual dataset generation/curation process is missing details, in particular: What is the composition of the dataset (i.e., which categories does it contain? How are the prompts+responses formulated? What is the origin of prompts + responses? Does it contain multi-lingual samples?) Also, there are just no samples or insights given about the contents of the collected dataset. I think maybe it would also be possible to provide some more details in the main paper, important information like annotator information is only given in the appendices, a summary in the main text would be appreciated\n- For many of the lessons learned, I would appreciate more detailed insights. It was noted that access to tools was crucial for high-quality annotations. How did the annotators actually use them? What kind of prompts are most relevant for?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sldkkvzmax", "reviewer": "Reviewer_2akh", "summary": "The paper introduces SynergyPref-40M, a preference dataset of 40M pairs (with 26M curated) and a two-stage human–AI curation pipeline for training Bradley–Terry reward models (0.6B–8B params). Stage 1 uses human-verified “gold” labels plus LLM-assisted “silver” labels with error-driven adaptive retrieval; Stage 2 scales automatic curation by enforcing consistency with a gold reward model and the current best RM. Trained RMs achieve state-of-the-art results on seven benchmarks (RewardBench v1/v2, PPE Preference/Correctness, RMB, RM-Bench, JudgeBench), with strong best-of-N scaling, resistance to style bias, and competitive correctness; ablations attribute gains to curation quality in addition to scale.", "questions": "1. How did you ensure that mined preference pairs (and their paraphrases) do not overlap with RewardBench v2, JudgeBench, etc.?\n2. Can you report how often flips contradict original human labels vs synthetic labels, and whether flipped pairs increase spurious correlations (e.g., penalizing certain styles)?\n3. Any results on downstream RLHF or human studies confirming that the SOTA RM ranking translates into policy improvements and user-perceived quality?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Two-stage, human-guided curation with adaptive retrieval is thoughtfully engineered and demonstrates measurable impact over raw LLM curation; Stage-2 consistency filtering is a pragmatic way to scale beyond the human budget.\n2. Seven strong benchmarks, including RewardBench v2 and JudgeBench, with detailed breakdowns (style bias on RM-Bench, BoN scaling on PPE Correctness). Results are consistently SOTA for open RMs.\n3. Training details are explicit; the authors plan to release data/models and provide scripts for reproduction.", "weaknesses": "1. While authors recognize raw LLM curation is weak and mitigate with human guidance, final labels in large parts of the mixture still depend on LLM aggregation; a more thorough error analysis comparing human vs LLM-judged segments would strengthen soundness claims.\n2. The “in-the-wild” pool composition, de-duplication against evaluation sets, and licensing/PII filters are not described in depth.\n3. Stage-2 selection relies on agreement with a gold RM trained on human data; without stringent de-duplication and contamination checks, this risks overfitting to the gold RM’s inductive biases and to benchmarks that overlap with the mined pool. Stronger leakage analysis (e.g., prompt/response near-duplicates) might be needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)