Pre-Trained Policy Discriminators are General Reward Models
Authors: Shihan Dou
Shichun Liu
Yuming Yang
Yicheng Zou
Yunhua Zhou
Shuhao Xing
Chenhao Huang
Qiming Ge
haijun Lv
Demin Song
Songyang Gao
Chengqi Lyu
Enyu Zhou
Honglin Guo
Zhiheng Xi
Qipeng Guo
Wenwei Zhang
Tao Gui
Qi Zhang
Xipeng Qiu
Xuanjing Huang
Kai Chen
Venue: NeurIPS 2025 poster ()
Keywords: Reward modeling
scaling laws
pre-training
reinforcement learning
RLHF
Primary area: deep_learning
TL;DR: we propose a scalable pre-training method named POLicy DiscriminAtive LeaRning (POLAR), which trains a reward model (RM) to discern identical policies and discriminate different ones.

Abstract: We offer a novel perspective on reward modeling by formulating it as a policy discriminator, which quantifies the difference between two policies to generate a reward signal, guiding the training policy towards a target policy with desired behaviors. Based on this conceptual insight, we propose a scalable pre-training method named POLicy DiscriminAtive LeaRning (POLAR), which trains a reward model (RM) to discern identical policies and discriminate different ones. Unlike traditional reward modeling methods relying on absolute preferences, POLAR captures the relative difference between one policy and an arbitrary target policy, which is a scalable, high-level optimization objective suitable for modeling generic ranking relationships. Leveraging the POLAR pre-training paradigm, we present a series of RMs with parameter scales from 1.8B to 7B. Empirical results show that POLAR substantially outperforms traditional non-pre-trained methods, significantly enhancing RM performance. 
For instance, POLAR-7B could improve preference accuracy from 54.8% to 81.0% on STEM tasks and from 57.9% to 85.5% on creative writing tasks compared to SOTA baselines. 
POLAR also shows robust generalization capabilities in RLHF using Reinforcement Fine-tuning (RFT), providing reliable reward signals and markedly enhancing policy performance—improving LLaMa3.1-8B from an average of 47.36% to 56.33% and Qwen2.5-32B from 64.49% to 70.47% on 20 benchmarks. 
Moreover, scaling experiments reveal a clear power-law relationship between computation and performance, supported by linear correlation coefficients approaching 0.99. 
The impressive performance, strong generalization, and scaling properties suggest that POLAR is a promising direction for developing general and strong reward models.

Tre-Trained Policy Diseriminators are General Reward Models
Shihan 12 Shichun Liu L Yuming Yangl.2, Yicheng Zou
Yunhua Zhou', Shuhao Xing", Chenhao Huang". Qiming Ge Demin Song'. Haijun Lv!
Songyang Gao Chengqi Lv Enyu Zhou?, Honglin Guo?, Zhiheng Xi² Wenwei Zhang',
Qipeng Guo Qi Zhang?. Xipeng Qiu?, Xuanjing Huang?, Tao Gui Kai Chen
'Shanghai Al Laboratory, FFudan University
channaaeaaacaaaaaaaaaaaaaa org. cn, tgui@fudan. edu.
Abstract
We offer novel perspective on reward modeling by formulating as a policy discriminator, which quantifies the difference between two policies to generate a reward signal, guiding the training policy towards target policy with desired behaviors. Based this conceptual insight, propose scalable pre-training method named POLicy DiscriminAtive LeaRning (POLAR), which trains a re- ward model (RM) discern identical policies and discriminate different ones. Unlike traditional reward modeling methods relying on absolute preferences, PO- LAR captures the relative difference between one policy and arbitrary target policy, which a scalable, high-level optimization objective suitable for model- ing generic ranking relationships. Leveraging the POLAR pre-training paradigm, we present a series of RMs with parameter scales from 1.8B to 7B. Empirical results show that POLAR substantially outperforms traditional non-pre-trained methods, significantly enhancing RM performance. For instance, POLAR-7B could improve preference accuracy from 54.8% to 81.0% STEM tasks and from 57.9% to 85.5% on creative writing tasks compared to SOTA baselines. POLAR also shows robust generalization capabilities RLHF using Reinforcement Fine- tuning (RFT), providing reliable reward signals and markedly enhancing policy performance-impreen
LLaMa3. 1-8B from an average of 47.36% to 56.33% and Qwen2.5-32BB from 64.49% to 70.47% on 20 benchmarks. Moreover, scal- ing experiments reveal clear power-lawe relationship between computation and performance, supported by linear correlation coefficients approaching 0.99. The impressive performance, strong generalization, and scaling properties suggest that POLAR is promising direction for developing general and strong reward models.
Introduction
Reinforcement learning (RL) plays crucial role in the post-training of large language models (LLMs) 129 30 5 Its success hinges on the reward model's (RM) ability to provide precise and stable feedback to the policy model 11 29). Although recent approaches successfully leverage labeled preference pairs train RMs for alignment with human preferences, these methods often face challenges in terms of scalability and generalization [119 66 74
[08] The former is limited by the difficulty of acquiring large volumes of high-quality labeled pairs (24 22 while the latter stems from the fact that this subjective approach to modeling human preferences makes RMs vulnerable to
51
reward hacking
[T6 13, 123]. On the other hand, several works, such as DeepSeek's R1 (35), utilize
Equaa contributions. 'Corresponding authors. I Work done during an internship at Shanghai Al Laboratory.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(I) Traditional Reward Model
(3) Ours: POLicy DiseriminAtive LeaRning (POL.R)
The werd "straeherry" cuntaine "r%
Here's why: I. Spelling Bitaldown...
0.80
Pre-truining
II. Fine-tuning
Reference
The word "stunberry" costains 3
0.20
Policy Model
Reward Model
Pos
Chosen
Pox
The werd "strwwberr" contains 2V%
Rejected
(2) Rule-based Verifier
III. Usage
Reference: "staheerry" cortann 3 "T's Let explan
The word "annwherry"" " contains 3 "V's.
Here's way: I. Spelling Breakdown.....
The weed "srawiserr"" contains 3 "Y's Here's way. Spelling Breaidown....
0.80
Polley
The word "stawberr"" contaics contaltss "Y's.
Verifiar
Policy
word "snashhrrr" contairs, ".'s
POLAR Reward
0.70
Model
The "atwwberr"" contalis "V's.
Answer:
Model
woed "suawberr"" coutaits "Y's.
Model
Comparison of three reward modeling methods: (1) traditional methods incorporate absolute preferences into RMs, which directly assess the quality of trajectories; (2) rule-based verifier validate the candidate trajectory through the gold answer and predefined rules; (3) POLAR pre-trains an RM to recognize identical policies and discriminate different ones, enabling it measure the difference in trajectories between a training policy and a target policy with desired behaviors.
Figure 1:
rules to verify the correctness of model outputs and provide accurate reward signals for RL. However, these rule-based verifiers can only be applied in scenarios where model outputs can be automatically verified by pre-defined rules, such as in reasoning and coding tasks [31] 655 83). In open-ended domains like writing and translation, rules are usually complicated and difficult to design in advance, making the rule-based verifiers hard to extend to general tasks.
Before delving into reward modeling, it is instructive to revisit the widespread success of LLMs. By adopting unified Next Token Prediction (NTP) optimization target [87] 88 [0], LLMs effectively harmonize diverse NLP tasks under a common objective, addressing the challenge of cross-task generalization. This inspires us to reconsider the training paradigm of RMs. Traditional RMs heavily rely on absolute, manually-defined criteria to generate preference score. Analogous to how LLMs unified NLP tasks, we should identify a fundamental, criterion- aggnosiic objective for RM pre-training.
Instead of traditional absolute preference modeling, we propose redefining a reward model as a "policy discriminator". Specifically, by quantifying the difference between candidate policies and given target policy, we establish a triteiionaagnssie objective, which naturally assigns higher scores to policies that are more "similar" to the desired target policy. This reward signal could guide the training policy toward desired behaviors during RL. Furthermore, since target policies can be arbitrarily chosen, this objective eliminates reliance on manually defined preferences and is applicable to any scenario, thus offering a scalable and fundamental pre-training paradigm for RMs. We refer to this training objective as Policy Discriminative Learning (POLAR), as illustrated in Figure |
Starting from a diverse and extensive collection of policy models, we construct a large-scale synthetic corpus by sampling trajectories from these policies. We then formulate the pre-training task contrastive learning objective utilizing Bradley-Tery (BT) loss [9. which encourages the RM to recognize trajectories derived from identical policies, while distinguishing those originating from different ones. Consequently, the pre-trained RM learns to assign higher rewards to trajectories exhibiting greater consistency with the target policy and generalizes this discrimination capability to unseen policies. After pre-training, analogous to supervised fine-tuning (SFT) in LLMs [80 that enables rapid adaptation to specific tasks and instructions, we also introduce an SFT procedure for POLAR RMs tailored to align with human-defined criteria. This fine-tuning process requires only a small set of reference trajectories generated from a given target policy, accompanied by candidate trajectories annotated with ranking labels reflecting their difference relative to the target. The reference trajectories can be directly annotated by humans, or altermatively, they could be derived from high-performing LLMs. Such flexible and lightweight fine-tuning approach allows the RM to rapidly adapt to new domains or criteria.
Leveraging the POLAR pre- pre-training method, we present series of reward models with parameter scales ranging from 1.8 to billion. Our empirical results demonstrate that POLAR effectively discriminates among diverse policies, and the fine-tuned POLAR RMs could significantly outperform traditional preference modeling methods. Specifically, preference evaluation tasks, POL.RR-B surpasses the SOTA 72B 22B-parameter WorldPM (ЛI), achieving an average improvement of 5.8%

---

points despite being approximately 10x smaller. Additionally, when applied within RLHF using Reinforcement Fine-tuning (RFT) 70, 79, POLAR RMs deliver more accurate reward signals and exhibit superior generalization across various downstream tasks, substantially enhancing the
performance of popular policy models such as Qwen2.5 [20. LLaMM3.1 [27], and Int
[12]. Moreover, POLAR exhibits scaling laws similar to those observed in LLMs, highlighting its significant potential for developing increasingly powerful reward models. In summary, our main contributions are as follows:
1. We propose POLAR, a novel criterion-agnosicc pre-training paradigm for reward modeling based on a scalable training objective- poolcc discrimination.
2. Scaling experiments reveal promising scaling laws, highlighting the significant potential of POLAR for enhancing the upper bound of reward modeling and developing stronger and more generalizable RMs.
3. We developed the POLAR series of reward models. They substantially outperform traditional RMs in empirical evaluations, achieving higher preference accuracy and better generalization than considerably larger RMs. This advancement expands the potential and applicability of RL algorithms, such RFT, paving the way for more innovative and diverse applications.
2 Related Work
Reward Modeling in Reinforcement Learning RL has emerged as pivotal technique in the post-training phase of LLMs [54] 52 109
129 124 All RL approaches crucially depend on precise reward signals provided by the reward function. Reward functions broadly fall into model-based and rule-based categories, with our work aligning closely with the model-based reward function paradigm [29 109 53) Existing model- -bsed methods typically train RMs using
35
labeled pairs to approximate a preference distribution 55 92
111.. However, the scareity of extensive labeled preference pairs poses a significant challenge 22
45). Additionally, such methods often exhibit limited generalization, struggling to predict out-of -isiiitiibttionn preferences robustly, thus weakening their effectiveness in RL training [66 33 123.. Another line of research investigates
22 6
generative reward models, leveraging LLMs themselves as verifiers [128] 72: 126 68) [5] 36).
Yet, these approaches inherently reflect the biases and preferences of the models used, frequently resulting in unreliable or biased reward signals [34]. Alternatively, some studies utilize rule- based verifiers to provide feedback [115) 65| (5 105]. but this approach struggles to address general tasks that are challenging to verify through predefined rules.
Our work closely relates to unsupervised learning, critical component of pre- pre-training focused on deriving essential and generalizable knowledge from extensive unlabeled datasets [2]
Pre-training and Scaling Laws
8T 21 91 187). Unsupervised pre-true pre-training is foundational to modern advances in language modeling, powering the latest sophisticated AI systems [86] 25 [14
10:
88
37
56
Concurrently, scaling laws describe the pawer-law relationship among computation and performance in neural language models, offering crucial insights into unsupervised pre-training 117 R. Extensive research indicates that scaling up data size and model size consistently enhances LLM performance [27] 220 85| 76 4 33; 87] 50 [27]. Recent studies further demonstrate the capability of scaling laws to predict the performance of larger models based on smaller ones, enabling efficient resource allocation [95] 46; 40). The success of scaling laws in language modeling has significantly influenced theoretical understandings and practical advancements,
112
69]
550 11 46
40
95]
inspiring similar explorations in vision 88 97 101 64 multimodal learning (64] [1 23 30 [07],
and reinforcement learning 41 75). Gao al. 22 specifically expanded scaling laws into the
over-optimization of preference-bree based reward models, further broadening their applicability in RMs.
Discriminator-base Reward Modeling Generative Adversarial Imitation Learning (GAIL)
and Discriminative Reward Co- Training (DIRECT) [2] are two representative works of discriminator based reward modeling. Both approaches train a diseriminator to distinguish between policy-generated trajectories and high-quality ones, and then use the discriminator's 's output as a reward signal. These methods are mainly designed for single-task scenarios, where the objective is to imitate expert behavior reproduce past successes. Our work differs from these approaches and benefits from large- scale pre-training. Instead of relying on expert demonstrations, we pre-train a policy discriminator that learns criterion-agnostr differences between policies from large-scale unsupervised dataset.
3

---

(1) Pre-training
(2) Fine-tuning
(3) Usage: RFT
Prompt p
Prompt
Policy Pool
Prompt p
T2
Best
TC
TA
TB₃
Better
Worst
Candidate
Reference
TA Te,
R(D.TaTT)) R(p. TC)))
Reward
Pre-training RM
Fine-tuning RM:
R(p,t'
Policy
Figure 2: Overview Policy Discriminative Learning (POLAR). Stage pre-training, the RM learns criterion-agnostic policy differences by assigning higher rewards to trajectory pairs from consistent policies. Stage 2: During fine-tuning, human annotators rank trajectories from the same policy, implicitly defining human criteria, align RM evaluations with human standards. Usage: In Reinforcement Fine- Tuning (RFT), the fine-tuned RM provides reward signals comparing candidate trajectories with human preferred references, guiding policy training toward desired behaviors.
Method
In this section, first revisit the optimization objectives of RL in LLMs and show that the RM essentially functions as a policy discriminator. Building on this insight, we introduce the Policy Discriminative Learning (POLAR) illustrated in Figure During the first stage, the RM acquires the ability to discriminate among various policies and quantify their differences. Subsequently. the supervised fine-tuning stage, fine-tune the RM trajectory ranking data derived from human judgments, explicitly capturing preferences regarding policy behaviors.
3.1 Preliminary #1: Preference-based Reward Models and Their Limitations
In the RLHF pipeline, reward models are typically trained with pairwise preference data. Given prompt and two candidate responses, human annotators are asked to choose which response better aligns with their preferences. The RM is then optimized to assign higher score to the preferred response, often using a Bradley-Terry (BT) loss [9] or an equivalent ranking objective. Formally, for prompt and two responses and TB, the RM is trained to satisfy
P(TA TB x) = a (ro(x, TA) ro(a, TB));
(1)
where (-)) is the logistic function. This framework allows the RM to approximate a latent re- ward function consistent with observed human preferences. We refer to this approach absolute preference modeling, since it explicitly encodes what types of responses are judged as good or bad.
Despite its effectiveness in many alignment tasks, preference- training fundamentally con- strained by its reliance on subjective human judgments. Annotation standards may vary across individuals, leading to bias and inconsistency in the preference labels. The collection of large-scale, high-quality data also expensive and time-consuming. which limits its scalability. Moreover, the resulting models often exhibit weak generalization, struggling to extrapolate beyond the distribution of training data. These limitations suggest that absolute preference modeling may not be suitable for robust alignment. This motivates shift toward exploring alternatives and more scalable paradigms.
3.2 Preliminary #2: Policy Optimization in Reinforcement Learning
Consider training policy model Te parameterized by & and prompt distribution D... Let denote a trajectory generated by this policy given prompts sampled from D.. The resulting policy distribution Dr,, can thus be empirically approximated by sampling prompt-trajectory pairs (z, T). Let (x, 7) IR. denote the reward model parameterized by 0, which assesses policy performance by assigning scores to prompt-traeectory pairs (a, r). Given regularization coefficient / that controls the strength of the KL-divergence penalty relative to the initial policy Tinit the RL optimization objective can be formulated as follows [80]:
(RL( ((r,)~-D [re(æ, 3DKx(T(T(2)) Tinit (T (T(=
(2)

---

The optimal policy n² then admits closed-form solution [90]
[ro 7)
ा (T|=) = ((x) Tini (T|z) exp
B
where z(-) the partition function. This formulation provides key insight: the reward model implicitly encodes a continuous operator that maps an initial policy to its optimal form through RL optimization, captured by the reward scores. Consequently, when a policy Te is trained against this reward signal under KL constraints, it effectively learns to approximate this implicit mapping within the defined reward distribution.
3.3 Unsupervised Reward Pre-training via Distributional Alignment
Traditional reward modeling approaches are based on explicit pairwise comparisons (e.g., harm- lessness in safety alignment), which inherently presuppose absolute human criterion. However, this assumption becomes problematic when considering broader, more general classes of crite- ria. Specifically, given an arbitrary criterion p inducing partial ordering over responses, i.c.. {r(a1) 2 r(a2) 2 ... r(a,)} for responses a, there often exists a complementary criterion -p that reverses this ordering. Consequently, exhaustively enumerating all possible criteria is not only computationally infeasible but also theoretically ill- ill-posed.
To overcome this challenge, we propose unsupervised reward pre-training paradigm that provides a cetterion-agnostie initialization for the reward model. Specifically, let π² and Tinit represent two policy distributions, where x* denotes the optimal policy for given downstream task and Tiii denotes the initial policy. Under the Kl--conttrained RL objective ORI. we observe that the reward function be uniquely defined (up to additive constant) by the density ratio between x² and Tiitt resulting in the following relationship:
" (T(aa) + 3 log (1). Tinit (r|z)
ro (2, r) - 3 log
Since the partition function is a constant independent of trajectories, we focus only on the former item, whose expected reward spans the entire trajectory space T:
E (11) [re r)] ~ B E log
(5)
Winit
Dia (15" [
This term converges zero when the sampling distribution precisely matches the optimal distribution. Motivated by this observation, we replace direct reward regression with distributional distance minimization approach. Given prior knowledge of desired behaviors 7* (e.g.. human references), we train to measure the divergence between n² (approximated through r²) and Tinit (approximated through policy rollouts).
Under this perspective, the role the RM fundamentally shifts: instead of merely assessing the performance of individual trajectories, it now serves measure of policy differentiation, quantifying differences between the training policy and the desired target policy. In practice given two different trajectories generated from the same prompt, the pre-trained RM estimates the divergence between their underlying sampling policies, which can be formalized as follows:
D( T²) ((err) (x.s")) [ro(T, (o(Tr,""la) -"[x:)]
This perspective naturally suggests viewing the reward model as a policy discriminator: it learns to distinguish between the training and target policies, quantifying their degree of difference. A smaller difference yields a larger assigned reward, thus incentivizing the training policy to progressively align more closely with the desired target policy through RL optimization. Here, we employ Bradley- Terry (BT) loss [9 [100 to pre-train the reward model:
Cpre-trai (9) -Eip.TA, (By Dreii [log a (re(p, TA, TA.) ro TA. T TB.
(7)
Here p denotes prompt sampled from the pre-training dataset Ppre-train- denotes the sigmoid function. TA, and TAa are generated by the same policy, while TB, originates from a different policy. These policies are randomly selected from a diverse policy pool comprising LLMs varying in architectures and parameters. The pre-training objective encourages the RM to capture nuanced distinctions between policies, assigning higher rewards to trajectory pairs drawn from more closely aligned or "similar" policies.
5

---

3.4 Supervised Fine-tuning with Human Criteria
After unsupervised pre-training, the reward model acquires criterion-agnostcc capability to discrimi- nate between different policies. For practical usage, it is essential to align this discriminative ability with human-defined standards and preferences. To this end, we introduce a supervised fine-tuning stage, which is designed to efficiently adapt the pre-trained RM to human judgment criteria.
Ideally, this fine-tuning stage would utilize reference trajectories generated by desired target policy, accompanied by candidate trajectories annotated with ranking labels indicating their relative differences. practice, however, we adopt a simplified yet effective approach: given a prompt p from downstream tasks, we generate three distinct trajectories from the same policy. Human annotators then rank these trajectories from best to worst, denoted as (TA > TB > TC), where тa is the most preferred, 7B is the second-best, and To is the least preferred. Although these trajectories originate from a single policy, the introduction of human evaluations implicitly imposes different human criteria, effectively treating these trajectories as if they were drawn from different underlying policies that reflect varying degrees of alignment with human standards. During innettnning we employ the following supervised ranking loss, consistent with the Bradley- Terry framework:
Linne-tune (0) ((p.rarrB:T))~Puuu [log a (ra TA TB) ra(p, TA TC)),,
(8)
where тл denotes the trajectory ranked highest by humans and TO is ranked lowest. By optimizing this loss, the reward model quickly adapts to human preferences, effectively measuring the policy differences implied by human judgments, thereby closely aligning its policy discriminative capabilities with human evaluation criteria. This SFT stage effectively bridges the gap between the criterion- agnostic discrimination learned during pre-training and human-aligned evaluations, yielding reward model robustly generalized to human judgment scenarios with minimal annotation overhead.
4
Experiments
We train reward models with parameter sizes of 1.8B and 7B using POLAR, denoted as POLAR- -.88 and POLAR-7B, respectively, and benchmark their performance against SOTA baseline RMs. In Section 4.1 we specify the model and training details. Section 4.2 presents the evaluation results on preference modeling benchmarks. Section 4.3] we demonstrate the efficacy of POLAR within the RLHF framework. To further highlight POLAR's potential and scalability, we scale up the model sizes and analyze the corresponding scaling laws Section 4.4 Lastly, Section [.5 presents comprehensive ablation studies to validate the contributions of different training stages in POLAR.
4.1
Model and Training Details
Model Architecture
The architecture of POLAR RMs is based on an autoregressive Transformer 10 similar to models the GPT series 87 88), augmented with linear prediction
87
88
head common design adopted in traditional preference- based reward modeling methods [63] 12 Specifically, the decoder configuration used in POLAR RMs aligns with that of the InternLM-225 series [12] For traditional RMs, the prompt and trajectory are concatenated to form the model input, from which the model directly outputs a reward value. In contrast, as outlined in our training objective, POLAR takes prompt along with two trajectories- reference and candidate as input. We utilize special tokens to combine these three elements into a single input sequence:
prompt reference <lsplitttokenl> tokenl> prompt + candidate <ireward _toenl>
The linear head then processes the hidden state corresponding to the <lreward token token from the model's final layer to produce the reward value.
Training Details During the pre- pre-training stage, we follow the standard process widely adopted in LLM pre-training [12 62 [20] and conduct extensive experiments on hyperparameters, deriving clear scaling laws for hyperparameter configurations. Appendix [.1e.d provides an in-depth analysis of how hyperparameters are chosen for POLAR. Rather than training from scratch, we initialize the POLAR models from a pre-trained (iternn.M..5-serfi POLAR pre-training. Training details for the pre-training and supervised fine-tuning stages, including data composition, are provided in Appendices D.I and D.2 respectively.
model and perform one additional epoch of

---

4.2 Performance in Preference Evaluation
We first evaluate POLAR on the human preference prediction task, which measures the ability of RMs accurately identify responses preferred by humans.
Evaluation Setup
We compare POLAR with
(Ours) POLAR-75
Skywore Reward 88
SOTA RM baselines:
[
Internut Reward-78
(Oura) POLAR-1.80
WPM Reward-728
Skymorre-Rewwrtt22
M2-rnl.M--Rwwrd-222 [1]. Skywork-Rewardd
Intemnu Reward 208
Hurmlessness
[63] Seyyook--Rewad-222 [63], and WorldPM-72B-
Reasoning
Tusks
UltraFeedback [Н] (details in Appendix E.II. Our
primary evaluation uses the RMB benchmark (130),
containing 3,162 questions, each with multiple trajec- tories ranked by preference scores. The top-ranked trajectories are treated as references, representing samples drawn from target policy. The task is to identify whether RMs correctly prefer the second- ranked trajectory over the third-ranked one. Addition- ally, we create another evaluation set from real user queries collected through online platforms and manu- ally annotate trajectory rankings (see Appendix E.2) We carefully remove overlaps with the training data to maintain independence. Existing RM baselines typically assess trajectories without considering ref- erences. To ensure fairness. we evaluate baselines using two methods: (a) standard scoring without ref- erences, and (b) including references explicitly in the prompt (Figure TOP. We report the best results for each baseline across these two settings.
Coding
Multilingual
Crustive
Open
dorming
Instruct Following
Role Playing
STEM
Figure 3: Comparison of POLAR and base- lines human preference prediction.
Results Comparison To thoroughly illustrate RM performance, we follow the evaluation approach from prior studies 53 [30), categorizing the evaluation by task type, shown in Figure 3 POLAR exhibits outstanding generalization, consistently outperforming baseline RMs across most tasks. Notably, on the STEM task, POLAR- and POLAR-7B surpass the best baseline by over 24.9 and 26.2 percentage points, respectively. POLAR also accurately identifies subtle distinctions in trajectories for challenging tasks like Reasoning and general tasks such Chat and Creative Writing, accurately predicting human preferences. The superior performance of POLAR can be attributed its unique training paradigm. Unlike traditional RMs that rely on absolute preferences and thus perform well primarily within distribution, POLAR RMs leverage large-scale criterion- agnostic pre-training to learn nuanced differences between policies, resulting in robust -ustii-disttibuttin generalization. Notably, POLAR-1 1.8B .8 achieves competitive results with just 1.8B parameters, which only 1/15th Seeywork-Rewarde highlight POLAR's s efficiency and scalability.
and 1/40th of WawllMM--2BBUltraaeere
These results
We observed that POLAR-1.88 and POLAR-7B exhibit similar performance in preference evaluations. However, in downstream RL experiments (see Section 4.3) POLAR-7B demonstrates a notable advantage over POLAR-IE 1.8B. This discrepancy between preference evaluations and actual RL tasks highlights an important limitation in traditional evaluation methodologies. Specifically, standard preference evaluation datasets may inadequately reflect the full spectrum of capabilities and nuanced distinctions reward models possess [130] 109 (33. underscoring the need for more comprehensive and representative evaluation frameworks in future studies.
4.3 Performance in RLHF Training
Setup and Implementations We select four open-source LLMss policies: InternL In-Itrute
[2]. LLaMa-3.1-800 -tttstrctttt (27). Dwee2.5--B-IInstrun [120, and (Owen.55332BIInsttte [12].
Policy optimization employs the Proximal Policy Optimization (PPO) algorithm [96). Unlike tradi- tional baseline RMs, which directly evaluate policy trajectories without references, POLAR requires a reference trajectory. During RI training, we assess trajectories by measuring their consistency with the provided reference, adopting the Reinforcement Fine- Tuning (RFT). Comprehensive training
details are available in Appendix E.3 Following previous studies [57 12 121. evaluate policy
7

---

Table I: Reward model performance comparison in RLHF training. Baseline denotes the initial policy
model without RLHF. We compare POLAR against SOTA reward models across 20 benchmarks within the RLHF framework. Complete results are detailed in Tables TO and T2
General Task
Instruct Following
General Reasoning
Policy Model
Reward Model
Coding
Knowledge
Average
Baseline
24.07
64.37
InternI M2 Reward 7B
28.02
64.45
78,63
64.84
79.96
60.43
57.82
Skywork Reward-88
InternL.M-881-
Intern M2 Reward-220 201
28.76
66.75
74.16
64.97
82.20
60.65
58.09
Instruct
Skywork Reward
WorldPM 728.
34.89
67.
77.13
65.56
84.29
61.08
60.49
POLAR-188 (Ours)
POLAR-7B (Ours)
37.35
73.25
79.63
67.89
85.18
64.46
63.18
Baseline
15.59
63.35
70.69
52.95
67.60
49.39
47.3 17.36
Internl R2
25.37
60.80
59.24
54.15
65.21
46.35
48.00
Skywork- Roward--a
24.80
61.80
67.53
53,54
66.23
49.36
49. 49.22
3.1-a-3.1-8B-
Intern M2-Reward-209
26.52
62.85
58.57
52.41
64.45
45.09
47.70
Instruct
Skywork Reward 27B
24.57
61.70
66.34
54.58
66.25
49.97
49.44
WorldPM-72B- F7
21.36
63.85
70.86
54.74
69.56
49.70
VOLAR-188 (Ouns)
27.96
65.20
71.35
$7.52
71.11
$1.30
52.71
POLAR-7B (Ours)
37.02
69.30
72.14
59.85
72.20
$1,69
56.33
Baseline
26.52
66,05
79.24
53.83
83.47
61.98
54.95
InterLL
M2-eeward--d
31.99
56.48
55.24
54.95
32.44
68.00
76.71
$8.09
83.13
58.12
57.04
Qwen2.5-7B- Instruct
33.05
55,41
Skywork- Reward-27B
34.28
69.45
78.21
57.46
83.58
59.43
57.84
WorldPM-72B--. Ta-----------aaaee Feedbaack
35.72
70.55
59.48
POLAR- A.R--888 (Ours)
35.76
71.35
80.40
62,52
84.19
60.39
60.35
POLAR-7B (Ours)
37.70
71.15
61.30
60.90
Baseline
64.49
IntenLMM-RRward--2
36.10
75.70
83.13
69.72
87,20
64,99
64.29
Skywork-
71.U7
66.08
Qwen2.5-32B. Instruct
BerrnLM2-Reward-222
37.98
74.45
85.76
69.32
89.35
66.68
63.25
Skywork Reward-27B
66.64
WorldPM-72B- 1aee--2B-UUtaeleeeeee
40.59
78.65
86.79
70.38
89.90
69.05
67. 15
POLAR-1.
68.55
POLAR-7B (Ours)
45.98
80.50
88.92
73:17
90.39
73.59
70.47
performance on 20 popular benchmarks (Table using OpenCompass 188, employing the optimal settings recommended by OpenCompass for fair comparison.
Results Comparison Tables T
TO and 12 summarize the results from RLHF experiments
using POLAR and baseline RMs. POLAR RMs consistently outperform traditional non-pre-trained RMs. For instance, the L fine-tuned using POLAR-7B achieves an average improvement of 9,0% over the initial policy model and 6.7% over the policy model optimized by WorldPM- 72B- P2B-UltraFeeddbck across all benchmarks. These results align with our findings from preference evaluations. The enhanced generalization and reduced bias in POLAR's reward signals primarily stem from its novel pre-training paradigm, which allows RMs to learn subtle distinctions between policies from extensive pre-training data rather than relying solely on labeled preference pairs. Moreover, the incorporation reference trajectories further clarifies optimization objectives, making policy training directions more explicit and stable, thus reducing deviations and improving robustness across diverse and potentially out-of-distribur similar performance in preference evaluations, POLAR-7B AR-7B demonstrates significant advantage in downstream RL applications. The substantial performance gains from 1.8B to 7B parameters further illustrate the significant scaling effects achievable with the POLAR paradigm.
tasks. Notably, although POLAR-1 and POLAR-7B exhibit
4.4 Scaling Laws in POLAR
Previous studies 27 87
have demonstrated that scaling up neural language models consistently
reduces the validation loss in predictable manner. This reduction follows a power-law relationship relative model parameters (N), training tokens (D), or computational resources (C) [50] III:
L = 3 X where X represents N. D, or C. In this formulation, denotes the scaling exponent, indicating the rate at which validation loss decreases as scale X increases, while / is normalization constant that determines the baseline level of the loss curve. Scaling laws thus facilitate the prediction of performance for larger models based on small-scale experiments, serving i as an invaluable guide for efficient resource allocation and streamlined model development 995 46 40).

---

0.70
1.0
0.42
0.41
0.88
0.40
0.60
Parameters (B)
-0.6
Loss
Loss
0.39
0.4
0.38
0.50
0.37
1=0.9-N-0.00
-N-0.0225
2-2-------0 R² 009912
0.2
R² 0.9886
0.36
10"
10"
10
10" 19
10t0
10"
Model Parameters (Billion)
Compute (Flop/s-days)
Figure 4: Scaling laws in POLAR. Validation loss vs. (left) model parameters N and (right) optimal training compute C. Dashed lines show the power-law fit, with 182 = 0.9886 (left) and Ri2 0.9912 (right). Results show predictable decrease in validation loss model size compute increases.
Setup We explore whether POLAR also exhibits scaling laws, which would underscore its scalabil- ity and potential. Specifically, we analyze how validation loss varies with respect to model parameters (N) and optimal training compute (C), provided that sufficient data is available. To this end, we train five RMs of different sizes, ranging from 50M to IB parameters, using up to 54B training tokens.
We first investigate the scaling behavior of POLAR the number of model parameters increases. As shown in the left side of Figure 4 we observe a clear power-law relationship between validation loss and model size. The fitted scaling law is given by:
Empirical Results
0.9 N 0.0425
The high R value (i.e., 0.9886) of the fit indicates an excellent match between the empirical data and the paw trend. These findings highlight the scalability of POLAR, showing that expanding the model size consistently leads to enhanced performance.
We also examine the relationship between validation loss and optimal training compute C. For each model, we track the validation loss as a function of throughout training. The right panel of Figure shows that validation loss follows power-law scaling trend with respect to compute:
L 2.4 C-0.0342
(10)
The high R² 0.9912 of the fit indicates that POLAR reliably benefits from increased compute, further validating the applicability of scaling laws to approach. This suggests that allocating more computational resources consistently yields better RM performance. Overall, these findings demonstrate that POLAR exhibits clear scaling laws similar those observed in LLMS 39]. Such predictable improvements underscore the strong potential of POLAR as a foundation for building more general reward models.
4.5 Ablation Study
Impact POLAR Pre-training We first investigate whether the strong performance of POLAR primarily stems from its pre-training stage. To do this, we train reward models under identical experimental settings but only utilizing the fine-tuning stage without the POLAR pre-training. Tables and 13 report the performance results preference evaluation and RLHF training, respectively. Results show these fine-tuning-only RMs achieve competitive results in preference evaluation but demonstrate substantial performance decline in RLHF training. Consistent with prior studies 130 109: 93], strong performance in preference evaluation does not guarantee effective reward signals for RLHF training. These observations underscore the critical role of RLHF effectiveness as an indicator of reward model quality, highlighting that POLAR's 's pre-training stage significantly enhances both RM performance and generalization capability.
also trained a traditional non-pre-trained reward model without reference trajectories using the same human criteria data. The experimental results show that, across nearly all tasks, the policy

---

Table 2: Ablation study in RLHF training. Baseline denotes the initial policy model without RLHF, i.e, -a-nLMM------nttteee [2]. w/o PT denotes the RM fine-tuned solely on human criteria, without any pre-training phase. w/o PT & Ref represents the RM trained via the traditional non-pre-traine method without reference trajectories. More detailed results are demonstrated Table
General Task
Instruct Following
General Reasoning
Reward Model
Coding
Math
Knowledge
Average
Baseline
24.07
62.65
74.40
64.37
83.11
60.94
56.49
POLAR
37.50
72.70
78.24
66.79
84.33
64.40
62.60
1.8B
w/o PT
30.37
68.05
76.96
65.23
83.30
63.04
59.45
w/o PT & Reff
27.67
63.65
75.49
64.89
82.50
60.36
57.60
POLAR
37.35
73.25
79.63
67.89
85,18
64.46
63.18
7B
w/o PT
31.14
68.25
78.99
67.53
83.80
64.06
60.76
w/o PT & Ref
31.07
68.30
76.74
66.16
83.05
62.06
59.73
Table 3: Ablation study on RFT vs. SFT. RFT denotes the reinforcement fine-tuning using POLAR.
SFT denotes the straightforward supervised fine-tuning. Two training processes employ the same prompt-reference data. More detailed results are shown in Table 15
General Task
Instruct Following
General Reasoning
Policy Model
Method
Coding
Math
Knowledge
Average
Baseline
24.07
62.65
74.40
64.37
60.94
$6.49
InternL.M--81-
SFT
25.27
67.55
75.21
62.26
80.38
60.89
56.44
Instruct
RFTPOLARTR
37.35
73.25
79.63
67.89
85.18
64.46
63.18
Baseline
26.52
66.05
79.24
53.83
83.47
61.98
54.95
Qwen2.5-7B- Instruct
SFT
26.36
66.86
72.94
57.76
80.12
58.36
54.66
RFTPOLART
37.70
71.15
81.15
61.30
84,70
62.57
60.90
Baseline
31.07
75.45
86.56
69.74
89.35
71.07
64.49
Qwen2.5.32B-
SFT
39.39
78.39
83.18
67.86
87.93
71.09
65.82
Instruct
POT
45.98
80.50
88.92
73.17
90.39
73.59
70.47
model fine-tuned with the RM w/o PT consistently outperforms the policy model fine-tuned with the reference trajectories
RM w/o PT & Ref. This indicates that, even in the absence of provide crucial guidance, simplifying the reward model's 's evaluation task and thereby yielding clearer and more stable directions for policy training.
RFT vs. SFT In RLHF training, POLAR RMs utilize provided references to generate reward signals for policy training, a process referred RFT. To assess whether performance improvements stem specifically from POLAR or merely from the reference data, we directly fine-tune policies using a straightforward SFT approach on the same reference data, as shown in Table Results reveal significant performance decline for policies trained with SFT compared to those trained with POLAR RMs using RFT. This clearly demonstrates that RL exploits training data more robustly, and POLAR RMs serve effectively graders to provide more accurate and robust supervision signals during the training process, reliably evaluating and guiding policy optimization.
5
Conclusion
We propose a novel perspective on RM by reformulating policy discriminator and introduce scalable approach named Policy Discriminative Learning (POLAR). Leveraging POLAR, the RM attains robust capabilities in distinguishing among diverse policies and accurately quantifying their differences. These measured differences serve effective reward signals to guide subsequent policy optimization. Extensive evaluations show that, despite having only small parameters, POLAR RMs significantly surpass larger SOTA baseline RMs, consistently achieving higher preference accuracy across multiple tasks. Additionally, POLAR shows substantial effectiveness RLHF, providing reliable and generalizable reward signals even in diverse and potentially out-of-domain scenarios. Furthermore, empirical analysis confirms that POLAR exhibits clear power-law scaling behaviors, underscoring its considerable potential for future enhancements.
10

---

Acknowledgments
The authors wish to thank the AC and anonymous reviewers for their constructive comments. This work was partially funded by Shanghai Municipal Science and Technology Major (Project 2023SH2DZX025007, National Natural Science Foundation of China (No. 62206057, 62376061, 62476061), Shanghai Rising-Star Star Program (23QA1400200), and Natural Science Foundation of Shanghai (23ZR 1403500).
References
[1] Jean-Baptistt Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716-23736, 2022.
[2] Philipp Altmann, Fabian Ritz, Maximilian Zorn, Michael Kölle, Thomy Phan, Thomas Gabor, and Claudia Linnhoff-Popien. Discriminative reward co-training. Neural Computing and Applications, 37(23))18793
2025.
[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL /ttps://arxiv... wwwpeiiii..rreeeeeee httg/abs/2108.. 07732
[4] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint 1xXiv:3309.66000 2023.
[5] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arX5v:220..05862, 2022.
[6] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Delllen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield- -oddds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI feedback, 2022.
[7] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chenggi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek 11m: Scaling open-source language models with longtermism. arXiv preprint arXhv:2401.02954 2024.
[8] Gérard Biau. Analysis of random forests model. The Journal of Machine Learning Research, 13:1063-1095, 2012.
[9] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, vol- ume 33, pages 1877-1901, 2020.
[11] Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. The Eleventh International Conference on Learning Representations, 2023. URL https

---

[12] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297 2024.
[13] Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. arXiv preprint
2024.
[14] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv preprint ar Xiiv:2107003377 2021.
[15] Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, et al. Rm-rl: Reward modeling as reasoning. arXiv preprint (rxxi::505.023887 2025.
[16] Jaymari Chua, Yun Li, Shiyi Yang, Chen Wang, and Lina Yao. Ai safety in generative ai large language models: A survey. arXiv preprint arXiv:2407.18369 2024.
[17] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen. Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14166 2021.
[18] OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models, 2023.
[19] Ganqu Cui, Lifan Yuan, Ning Ding, Guamming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023.
[20] DeepSeek-AL. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954 2024. URL
ai/
DeepSeek-LLn LLM
[21] Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, 2019.
[22] Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Luu Anh Tuan, and Shafiq Joty. Data augmentation using Ilms: Data perspec- tives, learning paradigms and challenges. In Findings of the Association for Computational Linguistics ACL 2024, pages 1679-1705. 2024.
[23] Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang. Liang Zhao, Jian- jian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.11 1499, 2023.
[24] Shihan Dou, Yan Liu, Enyu Zhou, Tianlong Li, Haoxiang Jia, Limao Xiong, Xin Zhao, Junjie Ye, Rui Zheng, Tao Gui, et al. Metarm: Shifted distributions alignment via meta-learning. arXiv preprint arXiv:2405. 000.2405.004388 2024.
[25] Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. survey of vision-language pre-trained models. arXiv preprint 22Xiv:2202.10936, 2022.
12

---

[26] Dheeru Dua, Yizhong Wang, Pradeep Dasigi. Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2368- 2378, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 1.118653///NNS 1246. URL hwtp://hlaaaaattoooo.ooooo /al/atpp/://aaaaaaboloo///////// owtp:/9-1////////0/1111 1246/
[27] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[28] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Haffield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. Red teaming language models reduce harms: Methods, scaling behaviors, and lessons learned, 2022. URL https::tt////att www.aaawwi.....aaaaaaaaae httg//bb/2200 2209. 2000000800002222
[29] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023.
[30] Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge. Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi- multi-granularity comprchension and generation. arXiv preprint arXiv:2404. /4396, 2024.
[31] Panagiotis Giadikiaroglou, Maria Lymperaiou, Giorgos Filandrianos, and Giorgos Sta- mou. Puzzle solving using reasoning of large language models: A survey. arXiv preprint arXiv:2402. 1291, 2024.
[32] Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang. Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlic Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang. Weng Lam Tam. Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang. Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024.
[33] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint 24X.2:2407.21788 2024.
[34] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.1555 2024.
[35] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.1294 2025.
[36] Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei. Reward reasoning model. arXiv preprint arXiv:2505. 14674, 2025.
[37] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Recoding-enhaneee bert with disentangled attention. arXiv preprint arXiv:2006.03654 2020.
[38] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Hee- woo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint 2011022010.14701, 2020.
13

---

[39] Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma. Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, al. Scaling laws and interpretability of learning from repeated data. arXiv preprint 2tdi::2005.04455 2022.
[40] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.0040 2017.
[41] Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning. arXiv preprint 1rXiv:2301.134444 2023.
[42] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[43] Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint 1rXiv:2405.11143, 2024.
[44] Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, and Tiejun Zhao. An empirical study of Ilm-as-a-judge for IIm evaluation: Fine-tuned judge models are task-specific classifiers. arXiv e-prints, pages arXiv-2408: 2024.
[45] Saffron Huang. Divya Siddarth, Liane Lovitt, Thomas Liao, Esin Durmus. Alex Tamkin, and Deep Ganguli. Collective constitutional ai: Aligning a language model with public input. In Proceedings of the 2024 ACM Conference Fairness, Accountability, and Transparency, pages
2024.
[46] Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii. and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. arXiv preprint arXiv:2402.0411 2024.
[47] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.167200 2024.
[48] Qile Jiang, Zhiwei Gao, and George Em Karniadakis. Deepseek chatgpt claude: A comparative study for scientific computing and scientific machine learning tasks. Theoretical and Applied Mechanics Letters, 15(3):100583, 2025.
[49] Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. NollowBench: A multi-level fine-grained constraints following benchmark for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 4667-4688, Bangkok, Thailand, August 2024. Association for Computational Linguistics. BtlLtttts::/aclanttttt acl acl-long 257
.wwpggg...rgg/// 2024
[50] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020.
[51] Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuar Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. Llm post-training: A deep dive into reasoning large language models. arXiv preprint arXiv:2502.21322. 2025.
[52] Sandipan Kundu, Yuntao Bai, Saurav Kadavath, Amanda Askell, Andrew Callahan, Anna Chen, Anna Goldie, Avital Balwit, Azalia Mirhoseini, Brayden McLean, et al. Specific versus general principles for constitutional ai. arXiv preprint arXiv:2310.1377 2023.
[53] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling, arXiv preprint arXiv:2403.10 2024.
14

---

[54] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling rein- forcement learning from human feedback with ai feedback. arXiv preprint ar arXiv:2309.00267, 2023.
[55] Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding chinese. In Findings of the Association for Computational Linguisties ACI. 2024, pages 1120--12285 2024.
[56] Junyi Li, Tianyi Tang. Wayne Xin Zhao. Jian-Yun Nie, and Ji-Rong Wen. Pre-trained language models for text generation: A survey, ACM Computing Surveys, 56(9):1-39, 2024.
[57] Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi. Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Evaluating quantized large language models, In Forty-first International Conference on Machine Learning, 2024.
[58] Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 11939, 2024.
[59] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction- following models. https:// /ttp::////tub.......
thub. cwt wot/:/tsttttaalaaaa//.aa. alapa__eaaca_eaallll eval 5 2023.
[60] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL https
/openreview cwoperreriev.nnt//ooummmmtaiaaie
[61] Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking Ilms with challenging tasks from real users in the wild, 2024. URL httph:t////aet org/
0bs/2406 04770
[62] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[63] Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410. 18451, 2024.
[64] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892 29444999--491,,, 2023.
[65] Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. Rltf: Reinforcement learning from unit test feedback. arXiv preprint vxXiv:2307.04349 2023.
[66] Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, et al. Rrm: Robust reward model training mitigates reward hacking. arXiv preprint arXiv:24099 13156. 2024.
[67] Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Andrew Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Xiaotao Gu, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang. Alignbench: Benchmarking chinese alignment of large language models. In ACL (1), pages 1621-11640, 2024. URL wwwpttthhoooooooooooeoooooo www./////222...... 2024. acl-long.660 624
[68] Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint arXiv:2504.00 02490, 2025.
15

---

[69] Xudong Luo, Zhiqi Deng, Binxia Yang, and Michael Y Luo. Pre-trained language models in medicine: A survey, Artificial Intelligence Medicine, page 102904, 2024.
[70] Trung Quoe Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. arXiv preprint arXiv:2401.08888 , 3, 2024.
[71] Kaijing Ma. Xeron Du, Yunran Wang, Haoran Zhang, Zhoufutu Wen. Xingwei Qu, Jian Yang, Jiaheng Liu, minghao liu, Xiang Yue, Wenhao Huang, and Ge Zhang. KOR-bench: Benchmarking language models on knowledge-orthoge reasoning tasks. In The Thirteenth International Conference on Learning Representations, 2025. URL htttp:://ppenreee
het/Eorrum2dd-SSRRDBeeo
[72] Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410. 2ef.:222...1 2024.
[73] Sarah Mercer, Samuel Spillard, and Daniel P Martin. Brief analysis of deepseek r1 and it's implications for generative ai. arXiv preprint arXi:2200.002533, 2025.
[74] Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. Inform: Mitigating reward hacking in rlhf via information-theoretiic reward modeling. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024.
[75] Oren Neumann and Claudius Gros. Scaling laws for multi-agent reinforcement learning model. arXiv preprint
2022.
[76] OpenAl. Gpt-4 technical report, 2023.
[77] OpenAL. Gpt-4o system card, 2024. UBL.::pp:://ar11..o///////11
[78] OpenAL. Openai ol system card, 2024. URL http:://arx.v..ogga////22.1.1..
[79] OpenAL. Reinforcement fine-tuning, 2024. URL http:::/ppaaaffoooooooaaa..
guides/rfft-use-csee. cases
[80] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744, 2022.
[81] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
[82] Silviu Pitis, Ziang Xiao, Nicolas Le Roux, and Alessandro Sordoni. Improving context-aware preference modeling for language models. arXiv preprint arXiv:2407. 16, 2024.
[83] Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. Reasoning with large language models, survey. arXiv preprint arXiv:2407.1 1511, 2024.
[84] Moschoula Pternea, Premna Singh, Abir Chakraborty, Yagna Oruganti, Mirco Milletari, Sayli Bapat, and Kebei Jiang. The rl/llm taxonomy tree: Reviewing synergies between reinforcement learning and large language models. Journal of Artificial Intelligence Research, 80: 15:1533-1 2024.
[85] Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt general-purpose natural language processing task solver? In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[86] Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre- trained models for natural language processing: A survey. Science China technological sciences, 63(10):1872-1897, 2020.
[87] Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, et al. Improving language understanding by generative pre-training. arXiv preprint arXiv:1611.02683 2018.
16

---

[88] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
[89] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763, PmLR, 2021.
[90] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems, 36:53728-53741, 2023.
[91] Prajit Ramachandran, Peter J Liu, and Quoc V Le. Unsupervised pretraining for sequence to sequence learning. arXiv preprint arXiv: 1.02683, 2016.
[92] Alexandre Rame, Nino Vieillard, Leonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: On the benefits of weight averaged reward models. In Forty-first International Conference Machine Learning, 2024. URL https //openreview. wwwpeeeiewwweeeeeeeeeeeee www.//eurrmme.. Saaprummiid=s7DDDUUU
[93] Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D Lee, and Sanjeev Arora. What makes a reward model good teacher? an optimization perspective. arXiv preprint (tti:2503.154777 2025.
[94] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google- proof q&a benchmark. In First Conference Language Modeling, 2024. URL https //openrevier www.eeetfooeoo wwt.://oooomiddi1166668888
[95] Yangjun Ruan, Chris J Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of langauge model performance. Advances in Neural Information Processing Systems, 37:15841-15899 2025.
[96] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 10X.vv/777.06644: 2017.
[97] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and- --d-aandddllggee tasks? arXiv preprint arXiv:2107.06388 2021.
[98] Shivalika Singh, Angelika Romanou, Clémentine Fourrier David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng. Shayne Longpre, Wei- Yin Ko, Madeline Smith, Antoine Bosselut, Alice Oh, Andre T. Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzich Fadace, Beyza Ermis, and Sara Hooker. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation, 2024. URL FRRLhttpss. /ttsss//axxxvv orriiioorrrgeee 03304
[99] Zayne Rea Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. MuSR: Testing the limits of chain-of-thought with multistep soft reasoning. In The Twelfth International Conference on Learning Representations, 2024. httt/httt:/:/eenntt id jenyyQzuel
net/forunt
[100] Hao Sun, Yunyi Shen, and Jean-Francois Ton. Rethinking bradley-terry models in preference-based reward modeling: Foundations, theory, and alternatives. arXiv preprint arXiv:2411.04999, 2024.
[101] Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Alpha-clip: A clip model focusing wherever you want. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 13019-13029, 2024.
17

---

[102] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery. Quoe Le. Ed Chi, Denny Zhou, and Jason Wei. Chal- lenging BIG-bench tasks and whether Mhain-offtthughh can solve them. In Anna Rogers, Jordan Boyd-Grraber, and Naoaki Okazaki, editors, Findings of the Association for Com- putational Linguistics: ACL 2023, pages 13003-13055 Toronto, Canada, July 2023, As- sociation for Computational Linguistics. doi: 18653/v 112/2/////2223333 findings-acl.824, URL petps://aclanntholooo org. wrg/2023..222 findings-acl..88 8224
[103) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08299 2024.
[104] InternI.M Team. Internim: multilingual language model with progressively enhanced capabilities. https://gi thub. com/ httpa////eeeeeeetaeeeeeeerrroooo 2023.
[105] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with lims. arXiv preprint arXiv:2501. /2599, 2025.
[106] Qwen Team. Qwen2 technical report. arXiv preprint ar arXiv2207.11061 2024
[107] Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Yuntao Chen, Lewei Lu, Tong Lu, Jie Zhou, et al. Mm-interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer. arXiv preprint arXiv:2401. 10200, 2024.
[108] Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang. Jiangyue Yan, Yao Su, et al. A survey on post-training of large language models. arXiv preprint
2025.
[109] Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rihf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.0m 1.0000000000,,, 2024.
[110] Binghai Wang, Rui Zheng, Lu Chen, Zhiheng Xi, Wei Shen, Yuhao Zhou, Dong Yan, Tao Gui, Qi Zhang, and Xuan-Jing Huang. Reward modeling requires automatic adjustment based on data quality. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4041-4064, 2024.
[111] Binghai Wang, Runji Lin, Keming Lu, Le Yu, Zhenru Zhang, Fei Huang, Chujie Zheng, Kai Dang, Yang Fan, Xingzhang Ren, et al. Worldpm: Scaling human preference modeling. arXiv preprint a50sv:2505. 10527, 2025.
[112] Haifeng Wang, Jiwei Li, Hua Wu, Eduard Hovy, and Yu Sun. Pre-trained language models and their applications. Engineering, 25:51 2023.
[113] Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human ht5//bb/2380 08935
amnotations, 2024. URL httpss///axiii
[114] Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao- Yong Wei, Yaowei Wang. Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: comprehensive survey. Machine Intelligence Reseurch, 20(4):447-48 2023.
[115] Xin Wang. Yasheng Wang, Yao Wan, Fei Mi, Yitong Li, Pingyi Zhou, Jin Liu, Hao Wu, Xin Jiang, and Qun Liu. Compilable neural code generation with compiler feedback. arXiv preprini arXiv:2203.051 2022.
[116] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang. Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: A more robust and chal- lenging multi-task language understanding benchmark. In The Thirty-eighi Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2024. URL. https://openreviee wwwp:/eeeftffoeoo wetf/forru//iieeyy0000
18

---

[117] Larry Wasserman. All of nonparametric statistics. Springer Science & Business Media, 2006.
[118] Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang. Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. FOFO: A benchmark to evaluate LLMs' format-following capability. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 680-699, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: URL
org/ 2024. acl mong.non.... long.44) 40/
[119] Adam X Yang, Maxime Robeyns, Thomas Coste, Jun Wang, Haitham Bou Ammar, and Laurence Aitchison. Bayesian reward models for IIm alignment. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024.
[120] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li. Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint 0rXiv:2412.151155 2024.
[121] Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong. Kuikun Liu, Ziyi Wang, et al. Internlm-math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.06332. 2024.
[122] Alex Young. Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. Yi: Open foundation models by 01. ai. arXiv preprint arxiv::4033333 2033:22403.0465, 2024.
[123] Yinlong Yuan, Zhu Liang Yu, Zhenghui Gu, Xiaoyan Deng, and Yuanqing Li. A novel multi-step reinforcement learning method for solving reward hacking. Applied Intelligence, 49:2874-2888, 2019.
[124] Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, Tian'iian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.00 05/ 18, 2025.
[125] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, 2019.
[126] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling next-token prediction. arXiv preprint arXiv:2408. 11t.:24488.5240, 2024.
[127] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401 02385, 2024.
[128] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-aa-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https:://pperreiiiet net meet/forummiduuccciee
[129] Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Yuhao Zhou, Limao Xiong, et al. Delve into ppo: Implementation matters for stable rlhf. In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
[130] Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. RMB: Comprehensively benchmarking reward models in LLM alignment. In The Thirteenth International Conference on Learning Representations, 2025. URL ItLLsttps:// //ppnreviee net/Forrumiid-4ngrr11959
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have clarified our claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitations and future work in Section Limitations and Future Work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
We have provided the full set of assumptions and a complete (and correct)
proof in Section Method.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have shown our experiment results in Section 4 Experiments. We have fully disclosed all experimental setups in Section 4 Experiments, Appendix Training Details, and Appendix Additional Experimental Setup.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have submitted source code in the Supplementary Material. We will upload our code and data to GitHub upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
Moolic//guide//Coeeeuuliiis
casssiinnooyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtps/cccppplgguuuiaeeeeae Stteee///ooSSSSeissSioica www.ssssiicgcyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have shown our experimental settings and implementation details in Section 4 Experiments, Appendix Training Details, and Appendix Additional Experimental Setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our reported results are all averaged over several random seeds, as shown in Appendix Additional Experimental Setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have included the computer resources in Appendix Training Details and Appendix Additional Experimental Setup.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.c http:////eeeeree.cceepp///cceeea inn
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed both potential positive socictal impacts and negative socictal impacts of the work in Appendix Broader Impact.
Guidelines:
23

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks. We have not released any generative language models datasets that may cause harm or contain high-risk content.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
We have cited the original papers that produced the code packages and datasets used in our paper. We have included the version information and corresponding licenses in Appendix B. We have included the licenses for our assets in the supplementary materials,
Justification;
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
24

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets.paperssit paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have included detailed descriptions and usage of the assets in the supple- mentary materials.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: In our work, all annotation was performed by company staff with relevant professional expertise, who were compensated with standard salaries.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
25

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We only use LLM to help us polish our manuscript and make more readable. It does not impact the core methodology, scientific rigorousness, and originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standare components.
Please refer to our LLM policy (https://neure) wttpoc////ooo/ooocccoo 220t://eeccccceeeeeee/////////// for what should or should be described.
26

---

Appendix
A Limitations and Future Work
Pros and Cons of Reference Trajectories Reference trajectories play dual role in reward modeling. On the positive side, they significantly enhance the accuracy and reliability of the reward signal, and incorporating multiple references per prompt could further reduce reward variance, particularly in open-ended scenarios. However, reliance on reference trajectories also increases annotation costs, since generating and curating references requires substantial resources. To mitigate this limitation, we have conducted preliminary explorations suggesting that trajectories derived from other prompts with similar underlying reasoning structures may also serve as effective references. This finding indicates that the model is not strictly tied to prompt-specific references and can still assign higher rewards correct responses even when the reference originates from a different prompt. Such a capability has the potential to reduce annotation costs while improving the flexibility and general applicability of the reward model. In future work, we plan to systematically evaluate this cross-promph referencing strategy and investigate its integration with multiple-referen reference settings to further enhance robustness and performance on open-ended tasks.
Integrating POLAR with Test-Time Scaling While POLAR has demonstrated effectiveness in scaling RMs for policy discrimination, it primarily focuses on pre-training paradigms. Recent advancements in test-time scaling techniques have showcased the potential of dynamically refining reasoning processes during inference via mechanisms like extended computation and self-reflection, such OpenAl o-series reasoning models and Deepseek R1 models to adaptively allocate computational resources, significantly improving decision-making quality in challenging scenarios. In future work, we aim explore the integration of POLAR's pre-training strategies with test-time scaling techniques, investigating how such a combination can synergistically enhance RM's performance and generalization in complex tasks.
35
47
73
48 Such methods enable
Exploring Scaling Potential of POLAR Given the observed scaling law behavior, we anticipate that the current POLAR series has substantial room for further performance improvements. In future research, we plan to leverage greater computational resources to train larger-scale POLAR RMs. The data-prep strategy employed by POLAR can be effectively scaled up to extensive pre- training datasets; however, this process inherently demands substantial policy sampling. Compared to traditional L.M data preparation, generating sufficient training data for PÓLAR is likely to incur considerably higher computational costs. By scaling up model size and computational resources, we aim to thoroughly investigate the limits of POLAR and release stronger, open-source models, thus facilitating continued advancements within the research community.
B Broader Impact
In this work, we introduce novel method, POLAR, to pre-train the reward model and expand the potential and applicability of RL algorithms,, such as RFT, paving the way for more innovative and diverse applications. The impressive performance, strong generalization, and scaling properties of our models suggest that POLAR is promising direction for developing general and strong reward models. We do not see any negative societal impacts of this work.
C License For Artifacts and Data Consent
In this paper, we use the latest versions of all models and datasets provided by OpenCompass Most of the models and datasets are licensed, including the licenses for AlpacaEvval and CMMLU are can be used for CC-BY-NC 4.0; the licenses for Skywork-Reward-SB Reward-8B and Seywork-Reward, InternLM2- R-wwarrd-- academic papers; the licenses for Arena-Hard, FollowBench, FoFo. 7B and InternLM2 20--2-2-wrdd2000 are Apache 2.0; the licenses for WildBench, MT-Bench, MBPP and GPQA are CC-BY 4.0; the licenses for HvmanEval, BBH, HellaSwag, MuSR, GSM8K, MATH-500, MMLU-Pro and MMMLU-Lite are MIT; the license for DROP is CC-BY-SA 4.0. For models and
27

---

model and datasets without explicit licenses, we have actively reached out to the authors. All models and datasets used in this paper are available for academic research work.
D
Training Details
D.1
Pre-training
D.1.1 Pre-training Data
We construct synthetic pre-training dataset for POLAR by generating prompt-trajectory pairs using a diverse collection of LLMs. Specifically, we first sample prefixes from the pre-training corpus typically used for LLM training. For each piece of text from the corpus, we randomly select an initial segment as the prefix, with lengths varying randomly between and 1024 tokens. Subsequently. based on each prefix, various policies autoregressively generate trajectories constrained length of 4096 tokens. For each prefix, we randomly select two different policies from a policy pool, which contains LLMS with varying architectures and parameter scales. The first policy generates two trajectories: one serves as the reference, and the other serves as the positive trajectory. Conversely, the second policy generates single trajectory from the same prefix, acting as negative sample. Additionally, to enrich dataset diversity, we include small proportion of prompts that represent human instructions, which are derived from instruction-tunin datasets. Instruction-tuned LLMs are then used to generate responses as trajectories for these instruction-based prompts.
maximum
Table summarizes the composition of trajec- tories used for RM pre-training, totaling 3.60T tokens. Table enumerates the 53 open-source pre-trained (base) LLMSs and 53 open-source instruction-tunedt (chat) LLMS in our policy pool for generating these trajectories. To further en- hance policy diversity and ensure broader distri- butional coverage, we additionally incorporate 78 intermediate training checkpoints from sin- gle pre-trained All policies span the period from December 2023 to the present and were specifically chosen based on variations in pre-training data versions and the diversity exhibited in their generated responses.
Table 4: Composition of the pre-training dataset mixture.
Generated by
# Tokens
# Policy Models
Base LLMs
3.56T
131
Chat LLMs
0.04T
53
During data construction, we encountered several issues that affected data quality. Firstly, certain policies tended to fall into repetitive loops, continuously generating identical or very similar content without meaningful progression. Instead of completely filtering out these outputs, we truncated repeated sections while preserving their loop-prone characteristics. Secondly, some generated trajectories were excessively long and lacked a clearly defined endpoint. To resolve this, we imposed maximum output length of 4096 tokens and truncated any incomplete endings, ensuring each generated sequence was self-contained. Lastly, we observed that a low sampling temperature resulted in insufficient contrast for effective contrastive learning, while high sampling temperature introduced biases, adversely impacting policy characterization. Based on these insights, we set the sampling temperature to 1.0. top-p 0.9, and top-k to 50 in our trajectory sampling process.
D.1.2 Hyperparemeter and Implementations
Our primary experiments are conducted using RMs containing 1.8 billion parameters and 7 billion parameters, denoted as POLAR-1.88 and POLAR-7B, respectively. The model architecture is detailed in Section and we adopt the XTune framework for pre-training and fine-tuning. Rather than training from scratch, we initialize the RM from pre-trained perform one additional epoch of POLAR pre-training.
5
To identify optimal hyperparameters for pre-training POLAR RMs, we carried out scaling experiments designed to establish data-driven scaling laws. These scaling laws relate the optimal hyperparameters to the model size (N), base model pre-training data size (Dp), and reward model pre-training data size (Dmm). The results of these scaling experiments are illustrated Figures and 6
hhtps//githbb..cm//meetLLMMMM
28

---

Table 5: List of Base and Chat LLMs used for generating trajectories for gathering pre-training
data. We mainly utilized the Llama 33 series, Qwen 4 106
120 series,
[22 series, ChatGLM
32
series, DeepSeek ? series, Gemma 103
series, and Intern 104
12
series models.
Base LLMs
Llama-3.2-1
Llama-3,2-3B
Meta- Meta-Llama-3-70B
Meta-Llama-3-8B
Meta-Llama-3.1-000
Mcta-Llamma3.1-88
Qwen-14B
Qwen-1 8B
Qwen-72B
Qwen-7B
Owen1.5-0.5B
Owe.1.5-1..B
@wenl.5--144
owenl.5-322
Qwen 1,544B
Swenl.55-728
Qwen I .5-7B
Qwen2-0.5B
Qwen2-1.58
Qwen2-72B
Qwen2-7B
Qwen2.5-0.55
Qwcn2.5-1.50
Qwen2.5-14B
Qwen2.5-32BB
Qwen2.5-3B
Qwcn2.5-72B
Qwen2.5-7B
Yi-1.5-34B
Yi-1.5-6B
Yi-1.5-9B
Yi-34B
Yi-34B-200K
Yi-6B
Yi-6B-200K
Yi-9B
Yi-9B-200K
chatglm3-6b-bas
deepseek-1lm-665
deepseek-IIm-PWh
gemma-2b
gemma-7b
glm-4-9b
internlm-20b
internlm-7b
internlm--__e 8b
internlm2-20b
internim2-7b
intermlm2-base-00b
intemlm2-basse -bbee-
intemlm2_551_.00
internim2 _5-20
internl2__5-78
Chat LLMs
Llama-3 1.ee-..2----nntttuee
Mamma-3.2-3B-Instii
Meta-Llama-3.1-700
3.1--70---nnstuu
Meta-Llama-3.11
3--tB--stttt
Owen-148-Chin
Qwen-1 8B-Chat
Qwen-72B-Char
Qwen-7B-Chat
Owen1.5-0.5B- Chat
O.e.1..-1..B--
Qwen1.S-14B-CHH
Qwenl.5-32BB --a2B-Chha
wwenl.5-4B-. Chat
Qweal.5-7B-Chr.
Qwen2-0.5B-Instrue
Qwen2-72B-Insture
Qwen2-7B-Instruut
-wen2.5-0.5B-Inntruue
Owen2.5-1..8-Innee
Qwen2.55144B-Inttie
Qwen2.55-22B-nntruuc
Qwen2.5-33-Innsrucct
Qwen2.5-72BBInse Instruct
Owen2.55-7B-Instrucc
Yi-1.5-34BB-Chat
11-1.5-34B-Chat-165
Yi-L.5-6B-Chat
Yi-L.5-9B-Chat
Yi-34B-Chat
Yi-6B-Chat
chatglm3-6b
chatglm3-6b-32K
deepseek-llm--Tb-hee
gemma-2b-it
gemma-7b-it
glm-4-9b-chat
glm-4-9b-chat-Im
intemlm-chat-20b
internlm-chat-7b
internlm2-chat-1_
intemlm2-chat-11 8b-sft
intemlm2-chat-200
Miemll2--chut22b--ss
internlm2-chat-7b
(eemlm2-chat-7b-stt
intemlm2 5-1 1_sbhchhttttt
internlm2_5-2061 5-200-chatt
internlm2_554 5-7--chhat
(rrmlm2_5-7bbchat-
In the first stage, we select base models pre-trained with various combinations of N and D, and train them on reward model datasets varying sizes (Dm)). We fix the batch size at 256 and explore learning rates (LR) ranging from 5e-6 to le-4 for models with parameters between 100M and 924M. For each configuration, we calculate the validation loss and quartic polynomial to the loss log(LR) relationship. We identify the optimal learning rate corresponding to the minimum loss for each combination of N, Dp, and D resulting in the empirical formula:
LR 0.0002306 N0.01125 D- 0.39916
where N, Dp. and Drm mm are all expressed in millions. In the second stage, we search for the optimal batch size within the range 128 1024. Learning rates are computed using the above formula and proportionally scaled based on batch size adjustments relative to the baseline of 256. Employing similar method, we fit quadratic polynomial to the loss log (batch size) curve and obtain:
Batch Size 31.9032 0-0.4444 DO.52990
0.5997
Finally, for POLAR-1.88 1.8B, by substituting the values N = 1.8B, D, 2.5T, and D,m = 0.94T, we determine the final learning rate be 1.4e-5 and the batch size 1940. The pre-training process conducted on 320 NVIDIA H800 GPUs for a total duration of 57 hours.
For pre-training POLAR-7B, rate 1.67e-5 and the batch size 4343. The pre-training process is conducted on 912 NVIDIA H800 GPUs for total duration of 175 hours.
N 7B, D, = 4.0T, and Dm 3.6T. Then get the learning
D.2 Supervised Fine-tuning
D.2.1 Supervised Fine-tuning Data
We construct dataset comprising 150K manually labeled examples to fine-tune the pre-trained reward model. Each example includes single prompt associated with three candidate outputs.
29

---

N-115-SM
N-252 6CM
N-589
N-424
0.30
235-728 Oew 233.220
235-722
1655
0y-215.758 -
Praditice
0.-291258 -34494
Pacictas
2355082 C.. 65.465
223-333
Cwi 203
0.40
Prodict
225.38
222.48
Oow 333.720
22.460
0.56
0.55
8
6.32
0.52
0.30
0.48
6:15
0.45
-4
6:66
0.40
22: 65
Loariing 2916
Looming Bot:
Leaming Rato
Loaning Rou.
Figure 5: Scaling law of the learning rate with respect to model size and data scale in pre-training.
N-1115
N-9240M
Oymm Likk Moam 2ar
24b dba
dag Protiction
Mymm Lil det
Oym 233 35.00 Protictae
2.52
D4- ADB Oup
0.525
23.500 Gy- 25k2220
Ru 210350
Ru 210330
2.00
0.575
3.82
0.350
0.54
1.96
0.525
3.54
257
500 (uaspled)
750
1000
1250
250
305
755
1908
1200
0:50
255
5.00
150
1000
1250
0.475
230
300
720
1003
1250
Size (mappli))
Bauch (xanples)
Sio: (xuplo))
Figure 6: Scaling law of the batch size with respect to model size and data scale in pre-training.
The prompts primarily sourced from widely used open-source preference- -air datasets such as UltraFeedback [19] and HH-RLHF 5 28 with a small subset derived from real user queries submitted to online chat platforms.
For prompts obtained from open-source datasets that originally contain two candidate outputs, we generate third candidate using sate-off-hhear LLM randomly selected from GPT-4o (77). OpenAl ol (78), Deepseek-RI [35], and Deepseek- V3 [62], to mitigate potential distribution biases. Human annotators then reorder these three outputs based on preference and adherence to instructions. For prompts sourced from actual user queries, we generate three candidate outputs utilizing LLMs selected either from the models listed in Table from the aforementioned oftte-of-the-at models. Human annotators subsequently evaluate and annotate relative preferences among these outputs. Consequently, each prompt is associated with three outputs ranked from best to worst. The top two outputs constitute a positive pair, while the second and third-ranked outputs form a negative pair.
All annotations were performed by company staff possessing relevant professional expertise and compensated at standard salary rates. To safeguard user privacy, we filtered all prompts in the training set to remove personally identifiable information.
D.2.2 Hyperparemeter and Implementations
For supervised fine-tuning of POLAR models, set the learning rate to 1e-5 for the 1.8B model and 2e-5 for the 7B model, use a batch size of 320, and train for one epoch. We experiment with multiple hyperparameter configurations and select the optimal one that avoids overoptimization, Each round of supervised fine-tuning runs on 16 NVIDIA H800 GPUs for approximately 0.5 hours.
For supervised fine-tuning of models in the ablation study, we adopt the same data and settings as POLAR, with two exceptions: The fine-tuning-e
reward model (w/o PT) directly uses the InternLM2.5 M2.--ssrriss as the backbone, instead of POLAR's pre-trained models. The traditional reward model (w/o PT & Ref) also fine-tuned on InternLM2.55 and trained on the same preference pairs as POLAR, but without incorporating reference trajectories, following the mainstream reward model training paradigm.
D.3 Compute and Data Comparisons
We further provide a comparison of compute and data requirements between our method and several representative baselines. sammariss the pre-training data size, labeled data size, and compute consumption at both the pre-training and supervised fine-tuning stages. Missing values indicate that the corresponding numbers were not disclosed by the respective methods.
30

---

Table 6: Comparison of compute and data resources across different reward modeling approaches.
WorldPM
Skywork-Reward
Category
POLAR-1.8B
POLAR-7B
-72B
-8B 63
Pre-training Data Size
0..4T tokens
3.66 tokens
~ 115. MM samples (15,100K)
404 samples (40,000K)
Labeled Data Size
150K samples
150K samples
320 x H800 GPUs
912 x H800 GPUs for 1155
Pre-training Compute
for ~57h
8 H800 GPUs for 00.5h
16 H800 GPUs for ~0.5h
SFT Compute
64 x H800 GPUs
Several observations can be drawn from this comparison. First, our method introduces a pre-training phase, which indeed requires additional computing resources. However, this phase is entirely unsupervised and thus does not require labeled data, resembling the pre-training paradigm of large language models. Second, the amount of labeled data required by method is substantially smaller than that of traditional preference- -baeed reward modeling approaches. Specifically, POLAR requires only 150K labeled samples for supervised fine-tuning, compared to tens of millions in other baselines, which dramatically reduces annotation costs.
E Details of Evaluation Setup
E.1 Baseline Reward Models
InternL.M2-7B-Rewall and IntemmL.22-20--Rewaa 12] are reward models built InternL Chat-7B-SFT and InterLLM2-Chht-2001 20.-SF, respectively. They are trained on over 2.4 million preference samples, comprising both human-annotated and Al-generated data.
63] are reward models based the 2-mma-2-27b-it and Meta-Llama-3.1 1.B.3...----ttuU-e architectures, respectively. Both models are trained using the Skywork Reward Data Collection, which consists of 80K high-quality preference pairs eurated from publicly available sources. Notably, achieves state-of-the-al performance in several prior studies.
and
[111 is a SOTA reward model recently released by the Qwen Group initially trained on 15M preference pairs collected from public forums spanning diverse user communities, capturing unified representation of human preferences. The model is then fine-tuned on 100K preference pairs from UltraFeedback [19, a fine-grained and diverse preference dataset.
E.2 Details of Preference Evaluation
For the preference evaluation set constructed from real user queries submitted to online platforms (as mentioned in Section 4.2), we generate three candidate trajectories per query using either L.MMs listed in Table or or the state-of-the-an models mentioned earlier. Human annotators are then instructed to rank these candidate outputs based on quality. To ensure a fair evaluation, we carefully exclude any examples overlapping with the training dataset. Following the same evaluation protocol as in the RMB set, we assess the reward model's capability to correctly identify the better of the two remaining trajectories, given the highest-ranked output as a reference.
E.3
Details of RLHF Training and Evaluation
Training Data We construct a dataset containing 1.25 million prompts paired with reference trajectories to support RLHF training for policy models. The prompts are primarily drawn from widely adopted open-source instruction datasets such as UltraFeedback [19 and HH-RLHF (5 28). supplemented by a smaller subset collected from real user queries online chat platforms. To promote better generalization, we ensure no overlap exists between prompts used for RLHF and those utilized during SFT. For each prompt, a reference trajectory is generated by randomly selecting
31

---

state-of-the-arr LLM from GPT-4o 777. OpenAl ol 778.. Deepseek-RR [35. and Deepseek-V V3 62] Notably, our RLHF dataset is constructed entirely without the involvement of human annotators.
Hyperparameter and Implementations For RLHF experiments, we train policy models using the PPO algorithm implemented in OpenRLHF [43]. guided by both our proposed reward model and several baseline reward models. To ensure robustness, experiments are conducted across multiple random seeds, and report the results averaged over these runs. To maintain fairness in evaluation, each baseline reward model is tested under two scoring conditions (similar to preference evaluation) when assigning rewards during PPO training: (1) standard scoring without reference trajectories, and (2) scoring with reference trajectories explicitly included in the prompt (Figure (0). For each baseline, report the best performance observed across these two settings. Notably, consistently find that standard scoring without references produces superior results, likely due to its closer alignment with the original training objectives of the baseline reward models.
During PPO training, we set the actor learning rate to 1e-6, the critic learning rate to 1e-5, the training batch size 1024, the rollout batch size 1024, and the number of epochs to 1. For all other hyperparameters, we generally follow OpenRLHF's recommended configurations. PPO experiments for all policy models, except Qwen2.5-32B- Instruct, per run, each taking approximately 48 hours. For Dwen2.5-33BBInntrrue we utilize 64 NVIDIA H800 GPUs per run, with each run lasting roughly 72 hours.
conducted using 32 NVIDIA H800 GPUs
F Additional Results
F1 Pre-Training Loss Curve
Pre-taining 188 Pre-training
Loss
20000
40000
60000
80000
100000
Step
Figure Pre-training loss curves for POLAR-I.8D and POLAR-7B.
Figure presents the relationship between training steps and loss for POLAR- -.BB and POLAR-7B, demonstrating steady decline in loss and indicating stable convergence during pre-training.
F.2 Reference-free vs. Reference-includde Evaluation
Table reppors the performance of POLAR- and two SOTA baselines (InterLLM2-Rew))) Rewa-- and
under both reference-frre and reference-inenn included evaluation settings.
The results indicate that providing a reference does not necessarily improve the performance of traditional reward models. many tasks, including reference even reduces accuracy. By contrast, POLAR consistently achieves superior performance across categories.
32

---

Table 7: Performance comparison under reference- free and reference-includd evaluation, and comparison between our method and similarity -based methods. INT denotes MM--neardd..e SKY denotes Skywork- Reiard--waaddddeee EMBED denotes OpenAl's Text-Embedding 3-Large.
INT
INT
SKY
SKY
Pre-trained-only POLAR-1.8B
Category
POLAR-I.8B
Ref
w/ Ref
w/ Ref
EMBED
Harmlessness
74.2%
66.8%
64.8%
75.9%
76.1%
66.1%
65.7%
NLP Tasks
68.5%
70.6%
67.8%
71.3%
61.9%
62.8%
70.6%
Multilingual
80.8%
69.2%
64.0%
76.9%
60.0%
64.0%
73.1%
Chat
75.1%
69.5%
62.1%
72.3%
67.2%
71.1%
67.8%
Brainstorming
74.8%
74.8%
72.3%
73.6%
68.6%
61.8%
63.5%
Role Playing
70.6%
75.0%
67.7%
70.6%
69.1%
70.6%
60.3%
STEM
79.8%
52.4%
60.7%
54.8%
52.4%
72.9%
66.7%
Instruct Following
73.1% %
65.4%
69.2%
69.2%
73.1%
50.0%
65.4%
Closed & Open QA
80.3%
56.3%
52.1%
62.0%
56.3%
57.1%
70.4%
Creative Writing
85.5%
51.3%
60.5%
56.6%
60.5%
78.1%
64.5%
Coding
65.8%
63.2%
54.0%
60.5%
68.4%
47.1%
59.2%
Reasoning
69.2%
53.9%
46.2%
53.9%
46.2%
77.8%
92.3%
Average
74.0%
66.7%
64.5%
72.4%
70.6%
65.5%
66.3%
F.3 Embedding Similarity Baselines
We further compare POLAR with an embedding similarity approach, where the reward value is computed as the embedding similarity between the candidate trajectory and the reference. As shown in Table even when using OpenAl's strongest embedding model, POLAR significantly outperforms this baseline preference prediction.
This highlights a fundamental difference between the two paradigms. While embedding baaeedddng--aeedd meth- ods rely on surface-level similarity between responses, POLAR explicitly measures the consistency between two policies represented by these responses. This policy-level perspective enables POLAR provide more meaningful and robust reward signals, better aligned with RL objectives.
F,4 Impact of POLAR Supervised Fine-tuning
To assess the necessity of the fine-tuning stage in POLAR's two-stage training framework, we evaluate the preference prediction performance of pre-trained-only model (i.e., POLAR-1.8DD and compare it with the fully trained POLAR-1.88 Results are shown in Table
The results show that, without alignment to human judgments, the pre-trained-only model fails to achieve meaningful performance in preference prediction, despite having absorbed broad knowledge during pre-training. By contrast, fine-tuning with small amount of labeled data enables POLAR-1.88 to align its scoring mechanism with human preferences, thereby unlocking the ability to effectively leverage its pre-trained knowledge for downstream tasks. This observation mirrors the training dynamics of LLMs, where pre-training confers broad competence, but fine-tuning is essential for aligning model behavior with human intent.
F.5
Case Study
To showcase the effectiveness our method, Figure Maareeandd and Figure iillustrate two representative examples of how POLAR-7B assigns rewards to trajectories. Detailed explanations are provided in the respective comments and captions.
33

---

Candidate Response 5: Reward Scoring Example 1 for POLAR-7B
User Prompt:
How many 't's are there in the word 'strawberry??
Reference Response:
There are 3 'r's in the word 'strawberry'.
Candidate Response
There are 3 'r's in the word 'strawberry..
v Same as the reference response.
# Reward: 0.690
Candidate Response 2:
There are 'r's in the word 'strawberry'.
# Wrong answer: # Reward: -9.523
Candidate Response 3:
Let's count the 'r's in 'strawberry'; "s', 'r', 'a', 'w", 'b' 'b,,'",,"""""" 'r', 'r', "y". There are 3 'r's,
so the answer is 3.
Correct answer with correct thoughts.
# Reward: -6.776
Candidate Response 4:
'b','e,, Let's count the 't's in 'strawberry': 's', 't', 'r', 'a', 'w', "b', 'e",, 'r', 'r', "y".. There are 2 'r's,
so the answer is 3.
# Correct answer with wrong thoughts.
# Reward: -9.289
Let's count the 'r's in 'strawberry': 's', 'r', 'a', "b','''' 'e',"'',, ry.y". There 3 'r's,
so the answer is 2.
# Wrong answer with correct thoughts.
Reward: -9.562
Candidate Response 6:
Let's count the 'r's in 'strawberry': 's', 't', "r', 'a", 'w", 'b', 'e', 'r', "r', "y". There 2 'r's,
so the answer is 2.
# Wrong answer with wrong thoughts.
Rewand: -10.937
Candidate Response
The word 'strawberry' contains 3 'r's.
# Correct answer.
# Reward: -1.341
Candidate Response 8:
The word 'strawberry' contains 'r's.
Wrong answer.
# Reward: -9.929
An example of reward model scoring applied the popular "strawberry" test. Despite variations in response phrasing, our reward model accurately identifies the correctness of the final answer and distinguishes the validity of the reasoning process.
Figure 8:
34

---

Reward Scoring Example 2 for POLAR-7B
User Prompt:
Summarize the first book of Frank Herbert's Dune in one witty short sentence.
Reference Response:
Royal teen discovers that life's beach -minus the ocean, plus spice, giant sandworms and deadly politics.
Candidate Response 1:
Royal teen discovers that life's 's a beach-minus minus the ocean, plus spice, giant sandworms and deadly politics.
#V Same as the reference response.
# Reward: 0.467
Candidate Response 2:
Royal teen discovers that life's beach-mine the ocean, plus magie, dark wizards and deadly politics.
# Closely resembles the reference response but includes factual errors. Reward: -7.711
Candidate Response 3:
Young noble's move desert planet turns into galactic Game of Thrones with fewer dragons, more worms.
# V A distinct concise and witry summary that draws analogies from other dramas-markedly different from the reference response.
# Reward: -6.918
Candidate Response 4:
noble family's s fall sparks young heir's rise as a leader on a harsh desert planet governed by prophecy and survival.
concise summary, but lacking wit ffills to meet the requirement. # Reward: -8.648
Candidate Response 5:
Paul Atreides loses his father, gains prophetic powers, learns to ride sandworm, leads holy war, and discovers that being the chosen one comes with lot of blood, sand, and questionable decisions.
witty summary, but overly long-fails to meet the requirement.
# Reward: -8.429
Candidate Response
Boy gets planet, becomes god, loses soul family drama ensues across galaxies.
# concise and witty summary that draws from multiple Dune books rather than just the first fails follow the instruction.
# Reward: -10.359
An example of reward model scoring applied summarization task. For open-ended questions, our model accurately evaluates the alignment of the core content of responses with the prompt and reference answer, considering multiple evaluation criteria rather than merely relying on semantic similarity to the reference response.
Figure 9:
35

---

F.6 Additional Results of POLAR Evaluation
Figure 10 iillusttrte the prompt format used by baseline reward models score trajectories when reference trajectory is provided, as mentioned in Section 4.2
Table & shows the details of our employed benchmarks for policy evaluation. Table 9 10
and 12 show the whole results of four policies in RL.H.. Table 13 and 15 present additional ablation
results mentioned in Section 4.3
Prompt including reference for reward model baselines
System:
You helpful Al assistant, designed to provide useful answers customers. The user will provide you with a question and reference answer. Please refer to this desired output and provide your unique response.
User:
You will be provided with the following information:
[The Start of conversation history] (dialog history) [TThe end of conversation History]
[The start of question] [question] [The end of question]
[The start of desired output] [reference) [The end of desired output]
Please think carefully about this question and the desired output, and then provide your response to the question.
Assistant: (response)
The prompt format used by baseline reward models to score trajectories with provided reference, mentioned Section 4.2] Specifically, we construct a dialogue scenario which the assistant generates response to the prompt with access to the reference, and the baseline reward models then assign scores the response. To ensure fair comparison with our POLAR RMs, we report the best performance of each baseline across both the reference-included and traditional (no-reference) settings.
Figure 10:
36

---

Table 8: Benchmarks used in policy evaluation.
Benchmark
Category
Description
A comprehensive Chinese henchmark for I.I.M alignment evaluation.
AlignBench(67
General Task
An automated benchmark for evaluating instruction instrection-oolloiing ability.
AlpacaEval 59
General Task
Arena-Hard 58
General Task
A dynamic benchmark on challenging, worldwrlld tasks.
WildBench 61
General Task
An automated benchmark real-world user tasks.
MT-Bench
General Task
benchmark on multi-turn conversational tasks.
A benchmark for evaluating LLMs' ability to follow instructions with multi-level, fine-gri ene-grained constraints.
Follow Bench 49
Instruction Following
benchmark evaluating LLMs' ability to follow complex, specific formats and formatting rules.
FoFo 118
Instruction Following
A benchmark for code generation tasks measuring functional correctness for synthesizing programs from docstrings.
vall
Cuding
code generation benchmark including 1,000 entry-level Python programming problems.
MBPP
Coding
A reading comprehension benchmark requiring discrete reasoning over paragraphs.
DROP 26
General Reasoning
A diverse benchmark comprising challenging BIG-Bench tasks.
[02
General Reasoning
A graduate-level Google-Prood Q&A benchmark covering the fields of biology. physics, and chemistry.
GPQA 94
General Reasoning
HellaSwag 125
General Reasoning
A challenging dataset for evaluating commonsense NLI
A dataset on multistep soft reasoning tasks specified natural language narrative.
MuSR 99
General Reasoning
challenging benchmark comprising five tasks that introduce new rules independent of prior knowledge.
KOR-Bench 7
General Reasoning
A dataset of 8.5K high quality linguistically diverse grade school math word problems.
GSM8K 17
Math
challenging dataset 500 high school math competition problems.
MATH-500 60
Math
A comprehensive Chinese benchmark for assessing knowledge and reasoning abilities in Chinese contexts.
CMMLU 55
Knowledge
An enhanced version henchmark containing over 12,000 academic exam and texthook questions across domains.
MMLU-Pro 116
Knowledge
A massive multilingual multitask language understanding dataset spanning 15 languages.
MMMLLU-Lite99
Knowledge
37

---

Table 9: Reward model comparison RLHF training. The policy model initialized from InternLM3-
8B-Instruct
12
(i.e., baseline). Sky8B and Sky27B denote Skywork-Rewardee and Skywork- respectively. Int7B and Int20B denote InternI.M2- and InternI
Reward-8B
63
Geward-20B
12 respectively. WPM72B denotes BonddMM-2B-UUtraFeeee
111
Policy model:
Type
Dataset
Baseline
Int7B
Sky8B
Int20B
Sky27B
WPM72B
POLAR-18B
POLAR-7B
AlignBench
6.13
6.24
6.09
6.37
6.27
6.43
6.54
6.53
AlpacaEval
51.80
57.02
61.37
58.39
63.48
70.93
68.57
68.70
General Task
Arena-Hard
40.79
45.96
45.01
47.33
45.56
53.69
63.47
63.34
WildBench
13.67
22.64
25,30
23.42
27.40
35.00
40.34
39.75
MT-Bench
7.97
8.24
8.29
8.27
8.40
8.57
8.43
Instruct
FellowBench
82.20
83.20
83.60
85,30
84.80
86,20
91.10
90.40
Following
FoFo
43.10
45.70
43,90
48.20
45.10
49.60
54,30
56. 10
HumanEval
81.10
82,93
78.88
78.66
78.66
81.10
82,93
84.15
Coding
MЕPP
67.70
69.65
73.15
73.54
75.
DROP
82.82
82,92
82.70
83.13
82.88
83.34
86.64
85.63
BBH
77.56
75.12
76.99
76.43
77.27
77.97
77.06
77.70
General
GPQA
32.83
36.36
35.35
36.87
37.37
37.88
37.37
41.41
Reasoning
HellaSwag
88.88
87.15
87.65
87.50
87.60
87.06
88.78
88.87
MuSR
52.28
$1.48
51.59
49.64
51.23
50.41
$7.07
58. 10
KOR-Bench
51.84
56.00
56.24
56.72
53.84
55.60
GSM8K
90.22
89.31
89.92
89.99
90.45
90.98
91.05
91.96
MATH-500
76.00
70.60
76.80
74,40
76.00
77.60
77.60
78.40
CMMLU
81.70
82.26
81.90
82,42
82.18
82.75
84.79
84.39
Knowledge
MMLU-Pro
57.49
58.05
57.94
57.99
58.20
60.00
59.98
MMMLU-LIE
43.64
40.97
40.02
41.54
39.59
42.30
48.40
49.00
Average
56.49
57.82
57.92
58.09
58.34
60.49
62,60
63.18
Table 10: Reward model comparison in RLHF training. The policy model is initialized from Llama- and Skywork- (i.e., baseline). Sky8B and Sky2BB denote Reward-7B and InternLM2..
InntrctInstruc 27
Reward-8B
63
respectively. Int7B and Int20B denote InternL Mitee.----Rwaard---
Reward-20B respectively. WPM72B denotes WorldPMM-2BB. Fatt--22---traabbaae
Policy model: 1lama-3.1 -.8ttrrrrutt
Type
Dataset
Baseline
Int7B
Sky8B
Int20B
Sky27B
WPM72B
POLAR-LSB
POLAR-7B
AlignBench
4.66
4.89
5.07
4.90
5.18
4.94
4.89
5.25
AlpacaEval
23.73
48.57
50.01
51.68
50.56
41.16
50.19
72.30
General Task
Arena-Hard
42.31
45.26
39.84
37.27
39.22
38.78
52.60
61.95
WildBench
-0.89
19.88
20.81
30.70
19.54
13.66
23.69
37.1
Instruct
FoncowBench
89.70
91.00
91.20
90.90
91.20
90.30
92,10
92.90
Following
FoFo
37.00
30.60
32.40
34.80
32.20
37.40
38.30
45.70
HumanEval
71.34
$4.27
64:63
59.15
63.41
70.12
70.72
70.73
Coding
MBPP
70.04
64.20
70.44
$7.98
71.60
71.98
73.54
DROP
79.46
79.33
81.42
74.43
82.53
81.60
84.48
84,
BBH
54.26
49.37
55.57
$8.25
61.79
60.50
64.34
70.02
General
GPQA
29.80
32.32
31.88
31.31
27.78
28.28
32.83
39.90
Reasoning
HellaSwag
46.90
63.56
53.06
52.60
51.03
53.99
60.92
67.84
MuSR
60.73
54.22
51.15
53.76
56.18
56.86
55.70
50.73
KOR-Bench
46.86
46.48
GSMSK
83.40
83.02
85.06
83.09
82.49
85:52
85.82
88.40
Math
MATH-500
51.80
47.40
47.40
45,80
50.00
53.60
56.40
56.00
CMMLU
54.78
$2.81
54.94
52.86
55.63
$4.96
$5.29
53.10
Knowledge
MMLU-Pro
49,57
48.11
50.14
47,53
49,84
50.68
51.57
52.66
MMMLU-Lite
43.83
38.12
43.01
34.87
44.43
43.45
47.04
49.32
Average
47.36
48.06
49.22
47,70
49.44
49.64
52.71
56.33
38

---

Table 11: Reward model comparison in RLHF training. The policy model initialized from Qwen2.5- 7B-Instruct 20 (i.e., baseline). Sky8B and Sky27B denote Exywork-Rewarde and Skywork-
Reward-8B
63
respectively. Int7B and Int20B denote Lnternl.M2-Rewarm and InternI
Reward-20B
12 respectively. WPM72B denotes WorddMM-72B-Uttraeeee
111
Policy model: Qwen2.5-7B-Instruct
Type
Dutuset
Baseline
Int7B
Sky8B
Int20B
Sky27B
WPM72B
POLAR-18B
POLAR-7B
AlignBench
6.18
6.22
6.27
6.25
6.30
6.48
6.42
6.49
AlpacaEval
37.02
58.39
58.63
62.81
61.11
62.18
60.13
61.86
General Task
Arena-Hard
56.73
51.88
54.10
53.81
59.64
61.52
67.11
71.38
WildBench
24.30
35.10
34.78
33,99
35.40
39.99
36.63
40,30
MT-Bench
8.38
8.37
8.40
8.43
8.51
8.49
Instruct
FellowBench
88.00
89.00
89.20
89.40
90.70
90:50
90.30
Following
FoFo
44.10
39.10
46.80
47.40
49.00
50.40
52,20
$2.00
HumanEval
84.15
78.66
79.88
76.83
81.71
81.41
84.15
87.20
Coding
MВPP
74.32
66.93
71.28
74.71
73.54
76.65
75.10
DROP
78.35
78.09
81.10
78.44
80.25
80.79
82.67
83.22
BBH
56.92
62,73
66.83
65.15
67.50
66.50
68.67
67. 14
General
GPQA
32.32
33.33
33.84
33,36
34.34
38.64
38.38
37.37
Reasoning
HellaSwag
67.47
74.63
74.73
65.48
71.27
76.24
84.91
81.25
MuSR
46.53
41.79
44.11
40.84
45.31
46.22
51.04
50.64
KOR-Bench
41.36
48.32
47.92
49,20
49,44
GSM8K
91.13
88.70
90.45
89.84
91.05
91:40
91.58
92.19
MATH-500
75.80
72.00
75.80
75.40
76.10
75.30
76.80
77.20
CMMLU
77.56
73.80
75.53
74.97
76.94
76.76
78:07
77.92
Knowledge
MMLU-Pro
55.33
52.30
55.54
54,89
55.48
55.97
56.65
MMMLU-LAE
53.04
39.61
43.28
45.22
46.01
46.19
47.13
53.15
Average
54.95
$4.95
57.04
56.15
57.84
58.83
60.35
60.90
Table 12: Reward model comparison in RLHF training. The policy model initialized from Qwen2.55
Ins-rnst 120 (i.e., baseline). Sky8B and Sky27B denote Seyyoor--eewrd--88 and Skywork-
Reward-8B
63
respectively. Int7B and Int20B denote InternL R-M-M22--ead-78 and InternLM22-
Reward-20B respectively. WPM72B denotes
12
Policy model: Qwen2.5.32H-Instmee
Type
Dataset
Baseline
Int7B
Sky8B
Int20B
Sky27B
WPM72B
POLAR-ISB
POLAR-7B
AlignBench
6.76
6.81
6.90
6.98
6,86
7.05
7.02
7.12
AlpacaEval
37.27
56.19
60.75
60.71
61.92
78.88
General Task
Arena-Hard
77.75
70.98
72.03
76.27
78.59
81.34
84.20
89.52
WildBench
25.09
38.03
36.33
37.39
37.49
40.34
39,50
45.82
8.54
Instruct
FoncowBench
91.30
94.10
94.00
94.00
94.40
95.00
94.50
94,20
Following
FoFo
59.70
$7.30
65.20
$4.90
65.80
62.30
66.00
66.80
HumanEval
90.24
84.15
90.24
89.02
90.85
90.00
90.89
91.46
Coding
MBPP
82.88
82.10
78.60
82.49
77.04
83.58
84.05
86.38
DROP
89.24
87.68
89.77
89.24
89.67
89.50
90.52
90.31
BBH
79.29
79.34
82.50
82.02
83.12
81.43
81.51
82.49
General
GPQA
43.43
43.94
46.46
44.44
47.98
47.47
47.47
52.02
Reasoning
HellaSwag
82.40
83.92
83.62
76.78
85.77
80.24
87.98
87.99
MuSR
66.70
65.35
64.17
64.56
65.19
64.87
67.64
67.49
KOR-Bench
57.36
58.24
$8.72
GSMSK
95.30
95.00
95.38
95.45
91.96
95.56
96.06
95,98
Math
MATH-500
83.40
79.40
83,20
83.24
81.60
84.24
84.00
84,80
CMMLU
84.41
81.70
82.64
82,86
83.06
83.73
86.02
85.52
Knowledge
MMLU-Pro
68.38
69,94
MMMLU-Lite
60.43
45.71
53.79
47.36
54.16
54.26
65.06
64.83
Average
64.49
64.29
66.08
65.25
66.64
67.15
68.55
70.47
39

---

Table 13:
Ablation study of POLAR preference evaluation. w/o PT refers the reward model that is fine-tuned solely with human criteria, without our pre-training phase. w/o PT & Ref denotes the reward model trained via the traditional non-pre-ttaiiee pre-trained method without reference trajectories. Since strong performance in preference evaluation does not guarantee effective reward signals, we conduct extra ablation study RLHF training Table 2 and Table 14 which provides stronger validation.
POLAR-I.BB
POLAR-7B
Tasks
Origin
w/o PT
w/o PT & Ref
Origin
w/o PT
w/o
PT & Ref
Harmlessness
74.2
73.9
63.8
77.7
78.4
74.0
NLP
68.5
69.9
65.4
71.3
71.3
70.9
Multilingual
80.8
84.6
56.0
80.0
84.0
72.0
Chat
75.1
70.6
65.5
79.1
71.8
63.8
Brainstorming
74.8
72.3
67.9
72.3
79.9
76.1
Role Playing
70.6
60.3
70.6
67.7
70.6
63.2
STEM
79.8
75.0
59.5
81.0
77.4
54.8
Instruct Following
73.1
76.9
50.0
65.4
61.5
61.5
Closed & Open QA
80.3
73.2
54,9
80.3
76.1
64,8
Creative Writing
85.5
81.6
57.9
85.5
86.8
65.8
Coding
65.8
57.9
54.0
68.4
64.5
59.2
Reasoning
69.2
69.1
46.2
69.2
76.9
46.2
Average
74.0
72.6
76.3
76.5
Table 14: Ablation study of POLAR in RLHF training. w/o PT denotes the reward model fine-tuned solely on human criteria, without any pre-training phase. w/o PT & Ref represents the reward model trained via the traditional non-pre-trane trained method without reference trajectories.
Policy Model: InternL -1-Su--Instrucl
Type
Datasel
Reward Model Size:
1.8B
Reward Model Size:
Baseline
POLAR
w/o PT
w/o PT & Ref
POLAR
w/o PT
w/o PT Ref
AlignBench
6.13
6.54
6.39
6.19
6.53
6.39
6.44
AlpacaEval
51.80
68.57
53.54
58.63
68.70
57.89
53.42
General
Arena-Hard
40.79
63.47
42.71
63.34
$4.99
51.68
WildBench
13.67
40.34
33.79
22.64
39.75
28.06
35.49
MT-Bench
7.97
8.57
8.28
8.17
8.43
8.35
8.33
Instruct Following
FelcowBench
82,20
91.10
84.30
81.80
90.40
89.30
84.00
FoFo
43.10
54,30
$1.80
45.50
56.10
47.20
52.60
HumanEval
81.10
82,93
82,32
81.71
84.15
81.71
81.10
Coding
MBPP
67.70
73.54
71.60
69.26
75.10
76.26
72.37
DROP
82.82
86.64
84.22
83.31
85.63
86.86
83,89
77.56
77.06
77.76
77.40
77.70
76.41
77.74
General
GPQA
32.83
37.37
35.86
36.87
41.41
39.90
38.38
Reasoning
HellaSwag
88.55 88.88
88.78
88.46
88.04
88.87
89.01
87.21
MuSR
52.28
$7.07
54.53
49,90
58.10
$7.71
53.32
KOR-Bende
51.84
53.84
50.56
53.84
55.60
55.28
GSM8K
90.22
91.05
89.99
90.30
91.96
90.60
90.90
Math
MATH--500
76.00
74.70
78.40
77.00
CMMLU
81.70
84.79
82,70
81.83
84.39
83.98
Knowledge
MMLU-Pro
57.49
60.00
58.28
$7.34
59.98
58.92
58.13
MMMLU-Lite
43.64
48.40
48.15
41.92
49.00
49.28
45.46
Average
56.49
62.60
$9.45
$7.60
63.18
60.76
59.73
40

---

Table 15:
Ablation study of POLAR on RFT vs. SFT. RFT denotes the reinforcement fine- tuning using proposed reward models. SFT denotes the straightforward supervised fine-tuning. Two training processes employ the same prompt-reference data. Results indicate that RFT with reward model is significantly more effective than SFT. The substantial improvement in policy model performance cannot be attributed solely to the training data (i.e., prompt-reference data) used RFT.
Pollcy Model
Internl Mi-Saf-Instruct
Owen2..-33BB-1
Baseline
RFTPOLAB-T
RFTPOLAR:FE
SFT
Baseline
RFTyOLAR-
AlignBench
6.13
6.53
5.84
6.18
6.49
5.62
6.76
1.12
6.52
AlpacaEval
68.
37.27
General Task
Arem--Hard
40.79
61.34
49.74
56.73
71.38
59.41
77.75
89.52
80.17
WildBench
13.67
39.75
5.77
24.30
40.30
2.77
25.09
45.82
26.67
Instruct
FollowBench
82.20
90:40
83.60
88.00
90.30
88.60
91.30
94.20
92.30
Following
HumanEval
81.10
84.15
78.05
84.15
87.20
76.22
90.24
91:46
86 59
Coding
DROP
82.82
85.63
87.12
78.35
83.22
84.71
89.24
90.31
90.54
BBH
77.56
77.70
59.16
79.29
GPQA
32.83
41.41
33.33
32.32
37.37
21.78
43.43
$2.02
42.93
Reasoning
HellaSwag
88
82.40
MuSR
52.28
58.10
53.45
46.53
50.64
46.01
66.70
67.49
62.87
KOR-Bench
51.84
55.60
42.48
41.36
48.16
44,16
57.36
58.72
45.12
GSMSK
90.22
91.96
89.76
91.13
92.19
88.63
95.30
95.98
95
MATH-5000
76.00
78.40
71.00
75.80
77,20
71,60
83.40
84.80
80.40
CMMLU
81.70
84.39
82.03
77.56
77.92
74,23
84:41
85:52
Knowledge
MMLU-Pro
$7.49
59. 98
54.74
55.33
56.65
52.42
68. 38
70.42
66.37
MMMLU-LiE
Average
56.49
63.18
56.44
$4.95
60.90
54.66
64,49
70.47
65.82
41


Reviews:
[{"review_id": "RlNwF6LLM7", "reviewer": "Reviewer_4MF9", "summary": "The paper presents a novel approach for aligning large language models (LLMs) via reinforcement learning using a discriminative reward model. The central idea is to train a discriminator to distinguish between high- and low-quality responses, which then serves as a reward signal for RL-based fine-tuning. The method is theoretically grounded and extensively evaluated across a broad spectrum of tasks, model scales, and baselines. The proposed discriminator-based reward model demonstrates improved scalability, robustness, and overall performance compared to prior approaches.", "questions": "Could the authors clarify the intuition behind the fine-tuning loss in Eq. 7? Specifically, what motivates replacing $(A_1, A_2 | B_1, B_2)$ with $AC | AB$? A brief intuitive explanation would enhance understanding.\n\nWhat are the compute and data requirements for training the discriminator at scale compared to other alignment strategies (e.g., current preference models)? How does the scalability presented in 4.4 compare to the considered baseline approaches?", "limitations": "The paper presents a thoughtful discussion on limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- Motivation & Novelty: The use of a discriminator as a reward model is well motivated, offering a scalable and principled alternative to traditional preference modeling.\n- Theoretical Soundness: The approach is supported by a rigorous theoretical framework, enhancing its credibility.\n- Comprehensive Evaluation: The paper provides extensive empirical validation, evaluating both the reward model itself and its downstream performance for RL-based fine-tuning. Evaluation spans a variety of model sizes, task domains, and baselines.\n- Strong Results: The method consistently shows strong and often state-of-the-art results, indicating its practical effectiveness and generality.\n\n## Weaknesses\n\n- Preliminaries: The paper would benefit from a dedicated section summarizing the setup and notation, especially for readers less familiar with reward modeling in LLM alignment.\n- Related Work: The method bears strong conceptual similarity to prior works such as *Generative Adversarial Imitation Learning* and *Discriminative Reward Co-Training*. A discussion on how this work extends or differentiates itself from these would help contextualize its contributions.", "was_revised": true, "final_justification": "The rebuttal effectively addresses the previously raised questions and concerns, and demonstrates careful consideration of the feedback. In light of this, the initially positive assessment is maintained.", "rebuttal": "Thank you very much for your recognition of our work and for providing valuable suggestions. We have addressed each of the issues you raised and will incorporate the corresponding revisions into our latest manuscript.\n\n> **Weakness 1: Preliminaries: The paper would benefit from a dedicated section summarizing the setup and notation, especially for readers less familiar with reward modeling in LLM alignment.**\n\n**Answer:** Thank you for your valuable suggestions. At the beginning of Section 3 (Methods), we will add an introduction to the experimental setup, notation, and conventional reward modeling methods to help readers better understand alignment and the contributions of our work.\n\n> **Weakness 2: Related Work: The method bears strong conceptual similarity to prior works such as Generative Adversarial Imitation Learning and Discriminative Reward Co-Training. A discussion on how this work extends or differentiates itself from these would help contextualize its contributions.**\n\n**Answer:** Thank you for your insightful suggestion. We have carefully reviewed the two papers provided by the reviewer. Both GAIL [1] and DIRECT [2] train a discriminator that distinguishes between trajectories generated by the current policy and those from high-quality trajectories, using the discriminator's output as a reward signal. In GAIL, expert demonstrations are used as high-quality trajectories, whereas DIRECT stores historically high-return action sequences as high-quality trajectories.\nThese works primarily focus on enabling the training policy to perform similarly to an expert policy within a single task. In contrast, our work investigates whether reward models in LLM post-training can adopt a pre-training paradigm and exhibit strong generalization capabilities, similar to LLMs.\n\nSpecifically, we explore how to construct unsupervised data to pre-train a policy discriminator, thereby redefining the reward function in LLMs. We further show its scaling laws and demonstrate its generalization and scalability through extensive experiments. Moreover, POEM can be fine-tuned with a small amount of labeled data to better adapt to downstream tasks, analogous to SFT in LLMs.\n\nIn summary, our work differs significantly from previous studies in terms of training methodology, data synthesis paradigm, model generalization and scalability, as well as application scenarios. \nWe will incorporate these points into our latest manuscript.\nFurthermore, we will thoroughly review related works that may share conceptual similarities with POEM and further expand Section 2 (Related Work) to help readers better understand our contributions.\n\n> **Question 1: Could the authors clarify the intuition behind the fine-tuning loss in Eq. 7? Specifically, what motivates replacing $(A_1, A_2 | B_1, B_2)$ with $AC | AB$? A brief intuitive explanation would enhance understanding.**\n\n**Answer:** POEM consists of two stages, i.e., the pre-training phase and the supervised fine-tuning phase. The goal of the first stage is to pre-train the reward model to distinguish between different policies. However, at this stage, the reward model is not yet aligned with human judgment; specifically, it does not capture the extent of differences between policy distributions as perceived by humans, nor does it reflect whether humans consider two policies to be different. Therefore, it is necessary to further fine-tune the reward model on a small amount of labeled data to achieve alignment with human judgment.\n\nAnother intuitive explanation can be drawn from the training paradigm of LLMs. Although pre-trained LLMs acquire extensive knowledge, they are not directly suitable for downstream tasks. Instead, they require fine-tuning with a small amount of labeled data to learn to follow human instructions and align their language behavior with human expectations.\nWe will refine our manuscript and incorporate these intuitive explanations to help readers better understand the purpose of the fine-tuning stage in POEM.\n\n\n> **Question 2: What are the compute and data requirements for training the discriminator at scale compared to other alignment strategies (e.g., current preference models)?**\n\n**Answer:** We compared our method with SOTA baselines in terms of compute and data requirements for training, as summarized in the table below. Missing values in the table indicate that the corresponding numbers were not disclosed by the respective methods.\n\n| Model  | Pre-training Data Size | Labeled Data Size  | Pre-training Compute Consumption   | Supervised Fine-tuning Compute Consumption   |\n|---------|--------|----|-------|--------|\n| WorldPM-72B [3]  | - | ~15M samples (15,000K)   | -   | -   |\n| Skywork-Reward-8B [4]   | -  | ~40M samples (40,000K)  | 64 NVIDIA H800 GPUs  | 64 NVIDIA H800 GPUs |\n| POEM-1.8B (Ours)  | ~0.94T tokens | ~150K samples | 320 NVIDIA H20 GPUs for ~57 hours| 16 NVIDIA H20 GPUs for ~0.5 hours   |\n\nSeveral key conclusions can be drawn from this table:\n\n(1) Our method introduces a pre-training phase, which requires additional compute resources. However, this phase is unsupervised and does not require labeled data, similar to the pre-training stage of LLMs.\n(2) **The amount of labeled data required by our method is significantly lower than that of traditional reward modeling approaches.** Specifically, we only need 150K labeled samples for supervised fine-tuning, whereas other methods typically require millions of labeled samples, which substantially increases annotation costs.\n(3) Moreover, our model has only 1.8B parameters, making it much more accessible for users to fine-tune POEM-1.8B compared to these larger models. \n\nIn summary, although POEM consumes more compute during the pre-training phase, users can fine-tune POEM-1.8B with significantly less computational and annotation resources to obtain a more powerful reward model compared to other baselines.\n\nFurthermore, after the anonymity period ends, **we will open-source all our models, including the pre-trained model, SFT model, and training framework, to minimize user adoption costs. Users can directly utilize the SFT model or further fine-tune the reward model with a small amount of domain-specific data to achieve enhanced performance.**\n\n\n> **Question 3: How does the scalability presented in 4.4 compare to the considered baseline approaches?**\n\n**Answer:** Our method shows significantly better scalability compared to SOTA reward model baselines. As discussed in Section 1 (Introduction, line 26) and Section 2 (Related Work, line 90) of our manuscript, these baseline methods require the continuous annotation of a large number of preference pairs to improve performance. Consequently, they lack the scalability.\nIn contrast, the pre-training phase of POEM is unsupervised and does not require labeled data, which ensures excellent scalability. During the fine-tuning phase, we only use a substantially smaller amount of labeled data than baseline methods to fine-tune the pre-trained model, yet still achieve stronger performance (as shown in experiments of our manuscript).\n\nMoreover, our scaling laws experiments (Figure 4) show a predictable decrease in validation loss as model size or computational resources increase. Furthermore, Tables 1, 5, 6, 7, and 8 demonstrate that POEM exhibits significantly better generalization and effectiveness compared to other SOTA baselines.\n\nIn summary, these experimental results clearly indicate that our method offers substantially superior scalability.\n\nFinally, we sincerely thank the reviewer for their recognition and for all the valuable suggestions provided for our manuscript. We will make sure to incorporate all of the suggestions into the next version of our manuscript. If you have any further comments, please do let us know, and we will do our best to address them.\n\n\n**References:**\n\n[1] Ho, Jonathan, and Stefano Ermon. \"Generative adversarial imitation learning.\" Advances in neural information processing systems 29 (2016).\n\n[2] Altmann, P., Ritz, F., Zorn, M. et al. Discriminative reward co-training. Neural Comput & Applic (2024). https://doi.org/10.1007/s00521-024-10512-8\n\n[3] Wang, B., Lin, R., Lu, K., Yu, L., Zhang, Z., Huang, F., ... & Lin, J. (2025). WorldPM: Scaling Human Preference Modeling. arXiv preprint arXiv:2505.10527.\n\n[4] Liu, C. Y., Zeng, L., Xiao, Y., He, J., Liu, J., Wang, C., ... & Zhou, Y. (2025). Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy. arXiv preprint arXiv:2507.01352."}, {"review_id": "ZOMyWKAeZ5", "reviewer": "Reviewer_cmE2", "summary": "This paper redefines reward modeling not as predicting absolute preferences, but as a 'policy discriminator' that quantifies differences between policies. Based on this perspective, it proposes POEM (POlicy DiffErentiation Modeling), a scalable and criterion-agnostic reward model pre-training approach. POEM learns reward signals by distinguishing between various policy behaviors and is then fine-tuned with a small amount of human-labeled data to align with human preferences.", "questions": "- The authors mention that they report the best results between reference-free and reference-included evaluations(lines237-238), but it would be more informative to separate the two and compare performance explicitly based on the presence or absence of the reference.\n- The impact of the pre-training stage on performance does not appear to be very large. As shown in Table 9, the average difference between POEM and the model w/o pre-training is only about 1.4 points. This raises the question of whether meaningful performance can still be achieved without pre-training. It would be helpful if the benefits of the pre-training stage were explained in more depth.\n- In scenarios where a reference is available, alternative methods could compute rewards based on similarity to the reference (e.g., embedding similarity). How would such approaches perform compared to POEM, and what are the fundamental differences between them?", "limitations": "Same as mentioned in the weakness section", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strength\n    - The paper is structurally well-organized and easy to follow.\n    - It achieves strong performance even with relatively little human-labeled data.\n    - It demonstrates high efficiency relative to model size and shows superior empirical performance compared to baselines.\n- Weakness\n    - The most significant weakness is that the reward model requires a reference during reward generation. This contradicts the typical assumption in evaluation, where the quality of references is unknown. As such, assuming access to high-quality references makes it difficult to consider the comparison with baselines truly fair and poses practical limitations for real-world deployment.\n    - In other words, the reward model is heavily dependent on the reference, and its performance cannot be guaranteed when the reference is sub-optimal or hard to obtain.", "was_revised": true, "final_justification": "Many of my questions were resolved through the author's response. Although I still believe that using references is not entirely fair, I acknowledge that POEM demonstrates superior performance compared to other methods that also use references. Therefore, I am raising my score from 2 (reject) to 4 (borderline accept).", "rebuttal": "Thank you very much for providing valuable suggestions. We have addressed each of the issues you raised and will make revisions to our latest manuscript.\n\n> **Weakness 1: The most significant weakness is that the reward model requires a reference during reward generation.**\n\n**Answer:** Thank you for raising this concern. Compared to traditional RM methods, POEM does require the introduction of additional references, which we have discussed in the limitations section. We hope the following further discussion will address your concern:\n\n(1) In our RL experiments, POEM significantly outperforms conventional reward models. Given POEM’s strong performance, generalization, and scalability, the requirement of references is not expected to substantially limit its broader adoption and future development. **One supporting example** is Reinforcement Learning from Verifiable Reward (RLVR), a widely adopted post-training paradigm for verifiable tasks. In RLVR, each prompt also requires a corresponding reference answer, enabling the verifier to assess the correctness of the trajectory. For example, in mathematical reasoning tasks, it is common practice to use rules to compare the generated solution with the reference answer for reward assignment. RLVR has become popular precisely because its reward signals are sufficiently reliable, and the need for references has not hindered its widespread adoption.\nPOEM can be regarded as **a general-domain verifier**. Experimental results show that POEM has generalization and scalability, which provides stable and accurate reward signals in RL, significantly outperforming SOTA reward model baselines.\n\n(2) The need for references can be addressed through **open-source LLM instruction tuning datasets and data synthesis**. Specifically, the reference requirement for POEM is consistent with that for instruction tuning in LLMs, where each prompt also requires a reference. There are now many high-quality, open-source instruction tuning datasets that can serve as references for POEM. \nFor references that are difficult to annotate, they can be synthesized using more powerful LLMs. In fact, the references used in our RLHF experiments were also synthesized (as indicated in Line 906). The results still show that POEM can achieve significantly better performance than all SOTA RM baselines, even without requiring a perfect reference.\n\nIn summary, although POEM requires references during reward generation, this limitation may not be a major concern due to the following factors: **1.** the need for references is consistent with the verifier in the widely adopted RLVR paradigm, and POEM can be viewed as extending the verifier approach from verifable tasks to more general domains; **2.** its data requirements are aligned with those for instruction tuning in LLMs; and **3.** our experiments demonstrate that synthesizing references is feasible. Therefore, the need for references may not be a major concern, especially given the significant advantages POEM offers over traditional RMs.\n\n\n> **Weakness 2: Assuming access to high-quality references makes it difficult to consider the comparison with baselines truly fair and poses practical limitations for real-world deployment.**\n\n**Answer:** We would like to clarify that we have tried our best to ensure a fair comparison with existing SOTA reward model baselines. As mentioned in the Evaluation Setup section (line 237), to ensure fairness, we evaluate these baselines using two approaches, i.e., scoring with and without the reference. For each baseline, we report **the best results** across these two settings. Additionally, as discussed in our response to Weakness 1 and in the manuscript (line 906), references can be synthesized from top-performing LLMs. \n\nThe results show that, even when using synthesized references as input, POEM-1.8B outperforms these SOTA baselines in all settings, while using significantly fewer parameters. POEM-1.8B also exhibits stronger generalization capabilities. We hope these clarifications address your concerns.\n\n\n> **Question 1: The authors mention that they report the best results between reference-free and reference-included evaluations(lines237-238), but it would be more informative to separate the two and compare performance explicitly based on the presence or absence of the reference.**\n\n\n**Answer:** Following your suggestion, I have reported the performance of the two SOTA baselines in these two settings on the preference prediction evaluation, as shown in the table below. \n\n|Category|POEM-1.8B|InternLM2-Reward-20B w/o Ref|InternLM2-Reward-20B w/ Ref|Skywork-Reward-27B w/o Ref|Skywork-Reward-27B w/ Ref|OpenAI Text-Embedding-3-Large|\n|---|---|---|---|---|---|---|\n|Harmlessness|74.2%|66.8%|64.8%|75.9%|76.1%|66.1%|\n|Brainstorming|74.8%|74.8%|72.3%|73.6%|68.6%|61.8%|\n|Chat|75.1%|69.5%|62.1%|72.3%|67.2%|71.1%|\n|Role Playing|70.6%|75.0%|67.7%|70.6%|69.1%|70.6%|\n|NLP Tasks|68.5%|70.6%|67.8%|71.3%|61.9%|62.8%|\n|STEM|79.8%|52.4%|60.7%|54.8%|52.4%|72.9%|\n|Coding|65.8%|63.2%|54.0%|60.5%|68.4%|47.1%|\n|Creative Writing|85.5%|51.3%|60.5%|56.6%|60.5%|78.1%|\n|Reasoning|69.2%|53.9%|46.2%|53.9%|46.2%|77.8%|\n|Instruct Following|73.1%|65.4%|69.2%|69.2%|73.1%|50.0%|\n|Closed&Open QA|80.3%|56.3%|52.1%|62.0%|56.3%|57.1%|\n|Multilingual|80.8%|69.2%|64.0%|76.9%|60.0%|64.0%|\n|**Avg.**|*74.0%*|*66.7%*|*64.5%*|*72.4%*|*70.6%*|*65.5%*|\n\nThe experimental results indicate that, for most tasks, providing a reference actually leads to decreased preference prediction performance for these traditional reward model baselines. This may be because these traditional reward models are optimized to model absolute preferences (i.e., determining what is good or bad), rather than relative difference, which focuses on comparing the quality of two policies.\n\n> **Question 2: The impact of the pre-training stage on performance does not appear to be very large. As shown in Table 9, the average difference between POEM and the model w/o pre-training is only about 1.4 points. It would be helpful if the benefits of the pre-training stage were explained in more depth.**\n\n\n**Answer:** Thank you for your valuable question. Overall, pre-training enables POEM to achieve **a strong generalization ability** in RL practice. In fact, Table 9 presents a comparison between POEM with and without the pre-training phase in preference prediction evaluation. While preference prediction is the most commonly used and convenient setting for validating reward model effectiveness, it is not considered the “gold standard” for evaluating the true effectiveness and generalization of reward models, in fact, the performance of reward models in RLHF provides a more reliable evaluation method [1]. \n\nIn Table 10 of our manuscript, we compare the effectiveness of POEM in RLHF with and without the pre-training phase. The results show that pre-trained POEM outperforms its non-pre-trained counterpart on nearly all tasks (**19 out of 20**), providing strong evidence that pre-training significantly enhances the generalization ability.\nMoreover, we explored the scaling laws during the pre-training phase, as shown in Figure 4. The results indicate that POEM has the potential to achieve even stronger performance through scaling.\n\nIn summary, POEM leverages unsupervised pre-training on large-scale data to learn to distinguish differences between policies. The pre-training phase is indispensable, serving as the foundation for POEM’s strong generalization ability and scalability.\n\n\n> **Question 3: In scenarios where a reference is available, alternative methods could compute rewards based on similarity to the reference (e.g., embedding similarity). How would such approaches perform compared to POEM, and what are the fundamental differences between them?**\n\nAnswer: We conduct experiments comparing POEM-1.8B with the SOTA embedding model on preference prediction evaluation. For the baseline, we calculate the similarity between the embeddings of the reference and the trajectory as the reward value. The experimental results are shown in the table in response to Question 1. The results show that, even when using OpenAI’s most powerful embedding model, our method significantly outperforms it in preference prediction.\n\nThere is a **fundamental difference** between our method and these similarity-based approaches. Although POEM also takes two samples as input, it measures the consistency between two policies represented by these two samples, rather than just their surface-level similarity. By focusing on this objective, POEM can effectively evaluate the consistency between the training policy and the target policy in RL, assigning more meaningful rewards.\n\n\n> **Paper Formatting Concerns: Some of the figures and tables discussed in the main text—such as Table 9, 10, and 11—are placed in the appendix.**\n\n\n**Answer:** Thank you for your suggestions! In fact, Table 1 in the main text is a simplified version of Tables 9, 10, and 11, which aggregate the benchmarks according to their evaluation capabilities. We will add this explanation to the caption of Table 1 to help readers better understand the relationship among these tables. Additionally, if our work is accepted, we will move these tables into the main text to further improve the clarity of our paper.\n\nFinally, we sincerely thank the reviewer for all the valuable suggestions provided for our manuscript. We will carefully revise our manuscript and incorporate these experiments into our latest version. If you have any further questions or suggestions, please feel free to reach out to us.\n\n\n**References:**\n\n[1] Zhou, E., Zheng, G., Wang, B., Xi, Z., Dou, S., Bao, R., ... & Huang, X. RMB: Comprehensively benchmarking reward models in LLM alignment. In The Thirteenth International Conference on Learning Representations (ICLR 2025)."}, {"review_id": "EYLrGA0WTB", "reviewer": "Reviewer_kicK", "summary": "The paper introduces Policy Differentiation Modeling (POEM), a two-stage framework for training LLM reward models (RMs) without relying on the conventional pair-wise ranking loss.\n\n- Pre-training: POEM is first trained with a contrastive objective that encourages it to differentiate trajectories generated from different policies, conditioned on the same prompt.\n\n- Fine-tuning: Given a prompt and three outputs ordered by quality, the model receives the top-ranked output as a reference and is optimized so that the score for the second-best trajectory exceeds that of the third-best.\n\nA 1.8-billion-parameter instance, POEM-1.8B, is shown to on average surpass baseline reward models both in human preference studies and when used to drive RLHF training.", "questions": "1. Could you provide empirical results—e.g., a plot of POEM performance when using the trajectory from the current prompt as reference versus using a trajectory of the same reference policy but from a different prompt—to substantiate the claim you make in Section 5?\n2. Was POEM initialised from an existing pretrained LLM (and if so, which architecture and checkpoint), or trained from scratch?", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Strengths\n- The method is rigorously described, with clear objectives, training procedures and hyperparameter settings.\n- Writing is concise and well-structured.\n- The results demonstrate that explicitly modelling policy differentiation can result in reward models that yield better RLHF policies.\n\n### Weaknesses\n- Dependence on reference trajecories from top-performing LLMs.\n- The paper claims that “trajectories derived from other prompts may also effectively characterize policy behavior\"  just as well as using the reference trajectory for the current prompt (Section 5) but does not include quantitative evidence or error bars demonstrating robustness to reference quality.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your recognition of our work and for providing valuable suggestions. We have addressed each of the issues you raised and will incorporate the corresponding revisions into our latest manuscript.\n\n> **Weakness 1: Dependence on reference trajectories from top-performing LLMs.**\n \n**Answer:** Thank you very much for your valuable comments. We provide a comprehensive discussion on why this limitation may not be a significant issue for several reasons, as detailed in **our response to Reviewer cmE2’s Weakness 1.** We hope this further discussion addresses your concern. \nIf you have any further questions or would like to discuss in more detail, please feel free to contact me.\n\n> **Weakness 2 & Question 1: The paper claims that “trajectories derived from other prompts may also effectively characterize policy behavior\" just as well as using the reference trajectory for the current prompt (Section 5) but does not include quantitative evidence or error bars demonstrating robustness to reference quality. Could you provide empirical results—e.g., a plot of POEM performance when using the trajectory from the current prompt as reference versus using a trajectory of the same reference policy but from a different prompt—to substantiate the claim you make in Section 5?**\n\n**Answer:** Thank you for your insightful question. We would like to clarify that our discussion on this topic is presented in Section 5 (Limitations and Future Work), where we mainly aim to share our latest observations from the case study. This does not include systematic quantitative experiments, and we would leave them as future work. We appreciate the opportunity to further elaborate here.\n\nThe motivation behind using trajectories from different prompts is to alleviate POEM’s reliance on prompt-specific references. POEM is designed to assign reward values based on the consistency between two different policies. In our preliminary exploration, we observed that the reference and the trajectory can correspond to different prompts, as long as they share commonalities.\n\nSpecifically, POEM was originally intended to score based on `(Prompt, Reference, Trajectory)`. However, we found that it can also score based on `(Prompt A, Reference, Prompt B, Trajectory)`, where \"Reference\" represents Policy A's answer to Prompt A, and \"Trajectory\" represents Policy B's answer to Prompt B. Formally, the input format changes from  \n`
 Prompt 
 Reference 
 Trajectory 
`  \nto  \n`
 Prompt A 
 Reference 
 Prompt B 
 Trajectory 
`.\n\n\nWe illustrate this with the following case study:\n\n```\nPrompt A: \"All birds have wings. Tweety is a bird. Question: Does Tweety have wings?\"\n\nReference: \"Yes. Tweety has wings.\"\n\nPrompt B: \"All teachers work at schools. Mr. Smith is a teacher. Question: What can we conclude?\"\n\nTrajectory 1: \"Mr. Smith works at a school.\" # Correct\n=> Reward: -9.0390625\n\nTrajectory 2: \"People who work at schools are teachers.\" # Incorrect\n=> Reward: -11.015625\n```\nIn this case, both Prompt A and Prompt B are classic deductive syllogisms, each requiring reasoning from two premises (one general and one specific) to reach a necessary conclusion. \nAlthough the question types differ (Prompt A is yes/no, while Prompt B is open-ended), the underlying logic is similar. For Prompt B, Trajectory 1 provides the correct conclusion, whereas Trajectory 2 incorrectly reverses the premise.\n\nResults show that, when given Prompt A and its reference, POEM can assign a higher reward to the correct answer to Prompt B. This suggests that POEM has the potential to generalize scoring across prompts by referring to similar question-answer pairs. Such a capability could help reduce the annotation burden of generating a reference for every individual prompt.\n\nWe will update Section 5 (Limitations and Future Work) to clarify the current scope of our findings. In future work, we plan to further investigate this finding and systematically evaluate POEM’s performance when using references and trajectories from the same versus different prompts.\n\n> **Question 2: Was POEM initialised from an existing pretrained LLM (and if so, which architecture and checkpoint), or trained from scratch?**\n\n**Answer:** The pre-training phase of POEM is initialized from an existing pretrained LLM. Specifically, POEM is pre-trained starting from the InternLM2 series InternLM2.5-1.8b-chat model [1], as presented in Appendix C.1.2 (Implementations, Line 837) of our manuscript, due to space limitations in the main text. We will **move this information to the main text** to provide readers with a clearer understanding of our training setup.\n\nFinally, we sincerely thank the reviewer for their recognition and for all the valuable suggestions provided for our manuscript. We will make sure to incorporate all of the suggestions into the next version of our manuscript. If you have any further comments, please do let us know, and we will do our best to address them.\n\n**References:**\n\n[1] Cai, Zheng, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen et al. \"InternLM2 Technical Report.\" CoRR (2024)."}, {"review_id": "NlJGD2wdVR", "reviewer": "Reviewer_wgBV", "summary": "This paper introduces an alternative perspective on training reward models by framing the reward model as a policy discriminator—quantifying the differences between candidate policies and a given target policy. The proposed approach, called POEM, consists of two stages: unsupervised pretraining followed by supervised fine-tuning. Experimental results demonstrate that POEM outperforms existing reward modeling methods, achieving higher preference accuracy and improved performance in RLHF settings. Additionally, scaling law experiments suggest that increasing model size and compute resources further enhances POEM’s effectiveness.", "questions": "1. In line 123, should \\tau be just a trajectory and not a set of trajectories? \n2. Equations 1 and 2 need citations. \n3. In line 140, I disagree with the comment that says “there always exists a complementary criterion”. I think it is definitely generally true, but not always. \n4. In Eq. 6, what is \\rho? I think it has to be a sigmoid function, but please do not assume the reader knows all notations.", "limitations": "The authors have discussed some limitations, which is nice, but they might be missing some, as I explained previously.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "On the positive side, this work takes a step towards better reward modeling approaches, which is of great interest. The alternative perspective of reward modeling as policy discrimination is interesting and likely to be of interest to the community. As shown in the experiments, this approach empirically leads to improved preference learning and RLHF performance. Also, the potential for further gains through larger models and increased compute is promising.\n\nOn the constructive side, I found that the paper lacks sufficient contextualization within the existing body of work, which makes it difficult to fully appreciate its contributions. For example, the authors argue that previous work using labeled preference pairs to train reward models is limited by the difficulty of obtaining large volumes of high-quality human annotations. However, POEM also relies on human rankings during the fine-tuning stage. Similarly, while the paper critiques the Bradley-Terry framework for assuming absolute human preferences—which may lead to misleading outcomes—POEM still employs a similar concept during fine-tuning, now requiring humans to compare three samples instead of two. These apparent inconsistencies appear throughout the paper. It is fine to have these limitations, given that the POEM empirical results seem to be better, but I would like to see a proper contextualization of the work within the existing body of literature and a proper discussion of the limitations. This would make the reader appreciate the work better. Second, given the above limitations, I wonder if the success of POEM originates from the alternative refomulation of the loss (using policy discrimination) alone? It’s possible that I may be overlooking something, as the authors have likely considered this aspect carefully, so I would appreciate the authors' thoughts on the matter.\n\nAnother aspect that would benefit from further discussion is the training of POEM reward models using Equation 6 with trajectories drawn from different policies. My assumption is that POEM’s performance is likely sensitive to the selection of these policies, potentially in a significant way. However, the current version of the paper does not include any analysis in this regard. An analysis of how the choice of policy set influences performance would make the work stronger. \n\nFurther, it would be nice to justify the two-stage training requirement for POEM. Specifically, how far can one go using only the pretraining stage? While the paper includes an ablation study highlighting the impact of pretraining, it lacks an analysis of the role of fine-tuning. An ablation study on this would further strengthen the work. \n\nFinally, I also found the summary of the method in the introduction to be vague, and it was difficult to comprehend. In general, this work could be strengthened by being more precise in its contributions and presentation.", "was_revised": true, "final_justification": "The proposed methods bring an interesting viewpoint for reward modeling, and I believe it is of interest to the community. The rebuttal was responsive to all my major points. While the work has some limitations, as other reviewers also pointed out, like reliance on the reference trajectories and the potential sensitivity of the performance on the selected policies, the authors have acknowledged and discussed these limitations. Therefore, I lean towards an accepting score.", "rebuttal": "Thank you for carefully reviewing our manuscript and providing valuable suggestions. We have addressed each of the issues you raised and will incorporate revisions into our latest manuscript.\n\n> **Weakness 1:** Lack of contextualization\n\n> **Weakness 1.1: The authors claim that using labeled preference pairs is limited by annotation difficulty, yet POEM also depends on human rankings during fine-tuning.**\n  \n**Answer:** We apologize for any misunderstanding caused by our unclear wording. What we intended to convey is that other methods require large amounts of annotated preference pairs to train reward models, which makes them difficult to scale. As reported in **our response to Reviewer 4MF9’s Question 2**, existing reward modeling methods annotate a substantial number of preference pairs to achieve better performance. For example, training WorldPM-72B [1] and Skywork-Reward-8B [2] requires **15M** and **40M** labeled preference pairs, respectively, whereas fine-tuning our POEM-1.8B only requires **0.15M** labeled data. The amount of labeled data needed to fine-tune the POEM model is much fewer than that required by existing SOTA reward models.\n\nFurthermore, to further improve performance, these traditional reward models would require even more annotated preference pairs. However, manual annotation is resource-intensive, while automated annotation may introduce bias.\nIn contrast, POEM exhibits strong generalization ability and achieves scalability through pre-training. It does not require the massive amount of annotated data needed by traditional reward modeling methods. \nThis is the key point we wanted to express.\n\nWe will refine our statement and include a comparison of training data requirements between POEM and traditional reward modeling methods in the main text to help readers better understand our work.\n \n> **Weakness 1.2: The paper criticizes Bradley-Terry Loss for assuming absolute preferences, yet POEM uses a similar method with three samples, showing recurring inconsistencies.**\n\n**Answer:** We apologize for any confusion caused by our unclear statements. We would like to clarify that we are not criticizing the form of the BT framework itself, but rather arguing against the way current reward modeling methods use the BT framework to model **absolute preferences**. Specifically, these methods utilize the BT loss to model what constitutes good or bad preferences. However, the preference data used for training may not be able to capture a wide range of preferences. Moreover, conflicting criteria may lead to contradictory partial orderings within the same training preference data.\n\nIn contrast, although POEM also employs the BT framework for training, it models **relative difference**. For example, in RLHF, relative difference can be understood as the gap between the training policy and the target policy, rather than making absolute judgments about individual policy.\n\nTherefore, in the pre-training phase, POEM trains the model to distinguish between policies, while in the fine-tuning phase, it learns to align with human perception of the differences between two policies. Through this approach, POEM avoids modeling absolute preferences directly. As a result, POEM can scale its capabilities through pre-training without requiring extensive annotation of fine-tuning data.\n\nWe will refine our statements in our latest manuscript to more clearly articulate the limitations of traditional RMs.\n\n> **Weakness 1.3: Needs better context and discussion of limitations.**\n\n**Answer:** Thank you for your valuable suggestions regarding our manuscript. In addition to addressing the two specific concerns you mentioned, we will thoroughly review and further refine our manuscript. We will do our best to ensure that our contributions are clearly and effectively communicated to the readers.\n\n> **Weakness 2: I wonder if POEM’s success mainly comes from its reformulated loss. I may be missing something, so I’d appreciate the authors’ perspective.**\n\n**Answer:** Thank you for your valuable comments. As we mentioned in **our response to Weakness 1.2**, the success of POEM stems from a fundamental shift in its modeling objective (from absolute preference to relative difference). \n\nTraditional reward modeling methods aim to enable RMs to learn absolute preferences. These methods require continuous annotation of preference data in an attempt to comprehensively cover human preferences, which is inherently difficult to scale. In contrast, POEM is designed to learn relative difference. During pre-training, POEM encourages the model to distinguish between different policies; during fine-tuning, it trains the model to align with human perception. Both stages share the same goal: modeling relative difference. \n\nThis shift in modeling objective makes unsupervised pre-training of the RM possible, similar to how pre-training is crucial for the success of LLMs. We can construct large-scale pre-training datasets for POEM without the need for manual annotation, continuously improving its performance. Our scaling experiments clearly show the scaling laws of POEM. Moreover, ablation studies further show that if the POEM model is only fine-tuned on annotated data without pre-training, its generalization ability drops significantly. This provides further evidence that pre-training brings strong generalization capabilities.\n\n> **Weakness 3: No analysis on policy set sensitivity.**\n\n**Answer:** Thank you for your valuable suggestions. We agree that the choice of policies used to construct the pre-training dataset can affect the quality of the pre-training data, and consequently influence POEM’s capabilities. However, investigating this question requires substantial computational resources. As reported in Appendix C, training POEM-1.8B consumed significant computational resources: we used 320 NVIDIA H20 GPUs for approximately 57 hours. Therefore, due to budget constraints, we finally utilized all available policies (184 models) in practice.\n\nFortunately, even without carefully selecting the policies for pre-training, POEM-1.8B demonstrated remarkable performance, showing strong effectiveness and generalization in both preference prediction evaluation and RLHF, significantly outperforming all SOTA baselines. \n\nIn the future, we will ensure to invest more computational resources to better understand the impact of policy set selection on performance. We also plan to train larger models to further advance POEM’s performance.\n\n> **Weakness 4: The paper justifies pretraining but not fine-tuning in POEM’s two-stage training. An ablation study on fine-tuning would be valuable.**\n\n**Answer:** We conduct an ablation study by evaluating the preference prediction performance of the pre-trained-only model, as shown in the table below.\n\n|Model|pre-trained-only PDM-1.8B |POEM-1.8B|\n|---|---|---|\n|Harmlessness|65.7%|74.2%|\n|Brainstorming|63.5%|74.8%|\n|Chat|67.8%|75.1%|\n|Role Playing|60.3%|70.6%|\n|NLP Tasks|70.6%|68.5%|\n|STEM|66.7%|79.8%|\n|Coding|59.2%|65.8%|\n|Creative Writing|64.5%|85.5%|\n|Reasoning|92.3%|69.2%|\n|Instruct Following|65.4%|73.1%|\n|Closed&Open QA|70.4%|80.3%|\n|Multilingual|73.1%|80.8%|\n|**Avg.**|*66.3%*|*74.0%*|\n\nWe observe that, without alignment to human judgments, POEM-1.8B fails to achieve meaningful scores on preference evaluation. This is reasonable, as only after POEM’s judgments are aligned with human judgments can it effectively leverage the broad knowledge acquired during pre-training to achieve better performance on downstream tasks. This is similar to LLMs, where LLMs need to be fine-tuned with a small amount of labeled data to align with human instructions. In essence, fine-tuning enables POEM to distinguish between policies from a human perspective.\n\nCombined with the ablation study in our manuscript, which evaluates POEM without pre-training, it is clear that both stages of POEM play an irreplaceable role.\n\n> **Weakness 5: Introduction and contributions are vague and unclear.**\n\n**Answer:** Thank you for your valuable suggestions. We will add more details about our method in the introduction to help readers gain a better understanding of our approach early on.\n\n> **Question1-4: Some typo, notation, and expression issues**\n\n**Answer:** Thank you for your valuable comments! We have addressed the following points:\n\n**(1) Notation (Line 123):** We confirm that $\\tau$ should refer to a single trajectory, not a set. We will revise the notation for clarity.\n\n**(2) Equation Citations:** Equation 1 follows the RL objective in Ouyang et al. [3], and Equation 2 is based on the closed-form solution from Rafailov et al. [4]. We will add the appropriate citations.\n\n**(3) Complementary Criterion (Line 140):** Good catch. We will revise the phrasing and have confirmed that this does not impact the validity of our subsequent arguments.\n\n**(4) Notation in Eq. 6:** $\\sigma$ denotes the sigmoid function. We will clarify this explicitly in the revised manuscript.\n\nMoreover, we will carefully review the entire paper to ensure that all concepts are clearly explained, further helping all readers to better understand our work.\n\nFinally, we would like to sincerely thank the reviewer for all the valuable suggestions. We will make sure to incorporate all of the reviewer’s suggestions into our latest manuscript. If you have any further comments, please do let us know, and we will do our best to address them.\n\n**References:**\n\n[1] Wang, B. et al. (2025). WorldPM: Scaling Human Preference Modeling. arXiv.\n\n[2] Liu, C. Y. et al. (2025). Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy. arXiv.\n\n[3] Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. NeurIPS, 35, 27730-44.\n\n[4] Rafailov, R. et al. (2023). Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 36, 53728-41."}]

Author rebuttal:

Meta-review: This paper introduces Policy Differentiation Modeling (POEM), a novel and scalable pre-training framework for reward models. All reviewers were positive, highlighting the novelty of framing reward modeling as policy discrimination, the strong empirical results, and the performance achieved with significantly less labeled data than the baseline. A key initial concern shared by reviewers was the method's reliance on a reference policy during reward generation, which raised questions about practical deployment and fairness of comparisons. The authors provided a comprehensive rebuttal that effectively addressed this limitation by providing additional experiments, leading multiple reviewers to raise their scores. The final consensus is that this paper presents a technically solid and impactful contribution. The paper is a clear accept.
Decision: Accept (poster)