Title: Pre-Trained Policy Discriminators are General Reward Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reward modeling

scaling laws

pre-training

reinforcement learning

RLHF
Abstract: We offer a novel perspective on reward modeling by formulating it as a policy discriminator, which quantifies the difference between two policies to generate a reward signal, guiding the training policy towards a target policy with desired behaviors. Based on this conceptual insight, we propose a scalable pre-training method named POLicy DiscriminAtive LeaRning (POLAR), which trains a reward model (RM) to discern identical policies and discriminate different ones. Unlike traditional reward modeling methods relying on absolute preferences, POLAR captures the relative difference between one policy and an arbitrary target policy, which is a scalable, high-level optimization objective suitable for modeling generic ranking relationships. Leveraging the POLAR pre-training paradigm, we present a series of RMs with parameter scales from 1.8B to 7B. Empirical results show that POLAR substantially outperforms traditional non-pre-trained methods, significantly enhancing RM performance. 
For instance, POLAR-7B could improve preference accuracy from 54.8% to 81.0% on STEM tasks and from 57.9% to 85.5% on creative writing tasks compared to SOTA baselines. 
POLAR also shows robust generalization capabilities in RLHF using Reinforcement Fine-tuning (RFT), providing reliable reward signals and markedly enhancing policy performance—improving LLaMa3.1-8B from an average of 47.36% to 56.33% and Qwen2.5-32B from 64.49% to 70.47% on 20 benchmarks. 
Moreover, scaling experiments reveal a clear power-law relationship between computation and performance, supported by linear correlation coefficients approaching 0.99. 
The impressive performance, strong generalization, and scaling properties suggest that POLAR is a promising direction for developing general and strong reward models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an alternative perspective on training reward models by framing the reward model as a policy discriminator—quantifying the differences between candidate policies and a given target policy. The proposed approach, called POEM, consists of two stages: unsupervised pretraining followed by supervised fine-tuning. Experimental results demonstrate that POEM outperforms existing reward modeling methods, achieving higher preference accuracy and improved performance in RLHF settings. Additionally, scaling law experiments suggest that increasing model size and compute resources further enhances POEM’s effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nOn the positive side, this work takes a step towards better reward modeling approaches, which is of great interest. The alternative perspective of reward modeling as policy discrimination is interesting and likely to be of interest to the community. As shown in the experiments, this approach empirically leads to improved preference learning and RLHF performance. Also, the potential for further gains through larger models and increased compute is promising.\n\nOn the constructive side, I found that the paper lacks sufficient contextualization within the existing body of work, which makes it difficult to fully appreciate its contributions. For example, the authors argue that previous work using labeled preference pairs to train reward models is limited by the difficulty of obtaining large volumes of high-quality human annotations. However, POEM also relies on human rankings during the fine-tuning stage. Similarly, while the paper critiques the Bradley-Terry framework for assuming absolute human preferences—which may lead to misleading outcomes—POEM still employs a similar concept during fine-tuning, now requiring humans to compare three samples instead of two. These apparent inconsistencies appear throughout the paper. It is fine to have these limitations, given that the POEM empirical results seem to be better, but I would like to see a proper contextualization of the work within the existing body of literature and a proper discussion of the limitations. This would make the reader appreciate the work better. Second, given the above limitations, I wonder if the success of POEM originates from the alternative refomulation of the loss (using policy discrimination) alone? It’s possible that I may be overlooking something, as the authors have likely considered this aspect carefully, so I would appreciate the authors' thoughts on the matter.\n\nAnother aspect that would benefit from further discussion is the training of POEM reward models using Equation 6 with trajectories drawn from different policies. My assumption is that POEM’s performance is likely sensitive to the selection of these policies, potentially in a significant way. However, the current version of the paper does not include any analysis in this regard. An analysis of how the choice of policy set influences performance would make the work stronger. \n\nFurther, it would be nice to justify the two-stage training requirement for POEM. Specifically, how far can one go using only the pretraining stage? While the paper includes an ablation study highlighting the impact of pretraining, it lacks an analysis of the role of fine-tuning. An ablation study on this would further strengthen the work. \n\nFinally, I also found the summary of the method in the introduction to be vague, and it was difficult to comprehend. In general, this work could be strengthened by being more precise in its contributions and presentation.\n\nQUESTIONS:\n1. In line 123, should \\tau be just a trajectory and not a set of trajectories? \n2. Equations 1 and 2 need citations. \n3. In line 140, I disagree with the comment that says “there always exists a complementary criterion”. I think it is definitely generally true, but not always. \n4. In Eq. 6, what is \\rho? I think it has to be a sigmoid function, but please do not assume the reader knows all notations.\n\nLIMITATIONS:\nThe authors have discussed some limitations, which is nice, but they might be missing some, as I explained previously.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Policy Differentiation Modeling (POEM), a two-stage framework for training LLM reward models (RMs) without relying on the conventional pair-wise ranking loss.\n\n- Pre-training: POEM is first trained with a contrastive objective that encourages it to differentiate trajectories generated from different policies, conditioned on the same prompt.\n\n- Fine-tuning: Given a prompt and three outputs ordered by quality, the model receives the top-ranked output as a reference and is optimized so that the score for the second-best trajectory exceeds that of the third-best.\n\nA 1.8-billion-parameter instance, POEM-1.8B, is shown to on average surpass baseline reward models both in human preference studies and when used to drive RLHF training.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The method is rigorously described, with clear objectives, training procedures and hyperparameter settings.\n- Writing is concise and well-structured.\n- The results demonstrate that explicitly modelling policy differentiation can result in reward models that yield better RLHF policies.\n\n### Weaknesses\n- Dependence on reference trajecories from top-performing LLMs.\n- The paper claims that “trajectories derived from other prompts may also effectively characterize policy behavior\"  just as well as using the reference trajectory for the current prompt (Section 5) but does not include quantitative evidence or error bars demonstrating robustness to reference quality.\n\nQUESTIONS:\n1. Could you provide empirical results—e.g., a plot of POEM performance when using the trajectory from the current prompt as reference versus using a trajectory of the same reference policy but from a different prompt—to substantiate the claim you make in Section 5?\n2. Was POEM initialised from an existing pretrained LLM (and if so, which architecture and checkpoint), or trained from scratch?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper redefines reward modeling not as predicting absolute preferences, but as a 'policy discriminator' that quantifies differences between policies. Based on this perspective, it proposes POEM (POlicy DiffErentiation Modeling), a scalable and criterion-agnostic reward model pre-training approach. POEM learns reward signals by distinguishing between various policy behaviors and is then fine-tuned with a small amount of human-labeled data to align with human preferences.\n\nSTRENGTHS AND WEAKNESSES:\n- Strength\n    - The paper is structurally well-organized and easy to follow.\n    - It achieves strong performance even with relatively little human-labeled data.\n    - It demonstrates high efficiency relative to model size and shows superior empirical performance compared to baselines.\n- Weakness\n    - The most significant weakness is that the reward model requires a reference during reward generation. This contradicts the typical assumption in evaluation, where the quality of references is unknown. As such, assuming access to high-quality references makes it difficult to consider the comparison with baselines truly fair and poses practical limitations for real-world deployment.\n    - In other words, the reward model is heavily dependent on the reference, and its performance cannot be guaranteed when the reference is sub-optimal or hard to obtain.\n\nQUESTIONS:\n- The authors mention that they report the best results between reference-free and reference-included evaluations(lines237-238), but it would be more informative to separate the two and compare performance explicitly based on the presence or absence of the reference.\n- The impact of the pre-training stage on performance does not appear to be very large. As shown in Table 9, the average difference between POEM and the model w/o pre-training is only about 1.4 points. This raises the question of whether meaningful performance can still be achieved without pre-training. It would be helpful if the benefits of the pre-training stage were explained in more depth.\n- In scenarios where a reference is available, alternative methods could compute rewards based on similarity to the reference (e.g., embedding similarity). How would such approaches perform compared to POEM, and what are the fundamental differences between them?\n\nLIMITATIONS:\nSame as mentioned in the weakness section", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach for aligning large language models (LLMs) via reinforcement learning using a discriminative reward model. The central idea is to train a discriminator to distinguish between high- and low-quality responses, which then serves as a reward signal for RL-based fine-tuning. The method is theoretically grounded and extensively evaluated across a broad spectrum of tasks, model scales, and baselines. The proposed discriminator-based reward model demonstrates improved scalability, robustness, and overall performance compared to prior approaches.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- Motivation & Novelty: The use of a discriminator as a reward model is well motivated, offering a scalable and principled alternative to traditional preference modeling.\n- Theoretical Soundness: The approach is supported by a rigorous theoretical framework, enhancing its credibility.\n- Comprehensive Evaluation: The paper provides extensive empirical validation, evaluating both the reward model itself and its downstream performance for RL-based fine-tuning. Evaluation spans a variety of model sizes, task domains, and baselines.\n- Strong Results: The method consistently shows strong and often state-of-the-art results, indicating its practical effectiveness and generality.\n\n## Weaknesses\n\n- Preliminaries: The paper would benefit from a dedicated section summarizing the setup and notation, especially for readers less familiar with reward modeling in LLM alignment.\n- Related Work: The method bears strong conceptual similarity to prior works such as *Generative Adversarial Imitation Learning* and *Discriminative Reward Co-Training*. A discussion on how this work extends or differentiates itself from these would help contextualize its contributions.\n\nQUESTIONS:\nCould the authors clarify the intuition behind the fine-tuning loss in Eq. 7? Specifically, what motivates replacing $(A_1, A_2 | B_1, B_2)$ with $AC | AB$? A brief intuitive explanation would enhance understanding.\n\nWhat are the compute and data requirements for training the discriminator at scale compared to other alignment strategies (e.g., current preference models)? How does the scalability presented in 4.4 compare to the considered baseline approaches?\n\nLIMITATIONS:\nThe paper presents a thoughtful discussion on limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}