Title: Reward Reasoning Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: reward models

reasoning

large language models
Abstract: Reward models play a critical role in guiding large language models toward outputs that align with human expectations. However, an open challenge remains in effectively utilizing test-time compute to enhance reward model performance. In this work, we introduce Reward Reasoning Models (RRMs), which are specifically designed to execute a deliberate reasoning process before generating final rewards. Through chain-of-thought reasoning, RRMs leverage additional test-time compute for complex queries where appropriate rewards are not immediately apparent. To develop RRMs, we implement a reinforcement learning framework that fosters self-evolved reward reasoning capabilities without requiring explicit reasoning traces as training data. Experimental results demonstrate that RRMs achieve superior performance on reward modeling benchmarks across diverse domains. Notably, we show that RRMs can adaptively exploit test-time compute to further improve reward accuracy. The pretrained models are available at https://huggingface.co/Reward-Reasoning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes reward reasoning models that can provide rewards to train models for better performance.  \nThe LLM gets two candidate responses and it reasons about them (reasoning trace) and finally picks one of them. This is the \"reasoning\" in reward models. Furthermore, they consider training the model with RL to generate such a reward reasoning sequence. A major benefit of this is that even though RL is performed with binary rewards, the resulting reward model is able to generate non-binary intermediate feedback. They do extensive ablation behind the reasoning reward models towards alignment with human preference, best of N inference based on the reward model, fine-tuning of models with RRM, scaling compute budget during test-time, and effect of scaling RRM during training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The proposed idea of using reasoning traces in reward models makes sense, and itis  backed by good empirical performance\n- Extensive ablation study behind the proposed idea. Considered multiple benchmarks and models.\n\nWeakness:\n- Lack of statistical error bars (CI). I think CI is important to draw conclusions from the empirical analysis, specifically in results like Table 1 where the performance of the proposed approach is close to that of the other models. LLMs are quite stochastic, and I am afraid to draw conclusions with the best case if the performances are close\n- No discussion regarding the computational cost of RRMs compared to standard scalar reward models\n- No discussion on limitations of the method\n\nQUESTIONS:\nWhat does it mean by adaptive compute? If it refers to increasing the hard budget of thinking (c.f. line 273), then it would suggest not using adaptive compute. The more or less use of the computing budget is not the result of the reasoning reward models. If there is a contribution of dynamically limiting the computation of RRM, then the adaptive compute terminology may be used.\n\nWhy didn't you consider any other models apart from Qwen?\n\nLIMITATIONS:\nNo, limitations not addressed. I think authors should compare the computational complexity of the proposed reward model with other reward models, e.g. scaler. Since RRMs involve generating chain-of-thought traces and aggregating over multiple samples (e.g., voting @32), the inference cost is significantly higher. \nAlso, the paper never mentions any other limitations. I encourage the authors to be upfront and mention all of them during the rebuttal.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Reward Reasoning Models (RRMs), which are designed to enhance reward model performance by incorporating explicit reasoning processes. The authors argue that existing reward models lack adaptability in allocating computational resources, particularly for complex queries that require nuanced analysis or multi-step reasoning. RRMs address this challenge by framing reward modeling as a reasoning task, where the model generates a chain-of-thought reasoning process before producing the final reward. This approach allows RRMs to adaptively allocate additional computational resources when evaluating complex tasks. The paper also presents a reinforcement learning framework that enables RRMs to self-evolve their reward reasoning capabilities without requiring explicit reasoning traces as training data. Experimental results demonstrate that RRMs outperform strong baselines across multiple domains, including reasoning, general knowledge, safety, and alignment with human preference. The authors further show that RRMs can effectively utilize test-time compute to improve reward accuracy and adaptively allocate computational resources to practical application scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The authors conduct a comprehensive comparison between RRM and various other reward models in their experiments. The superior performance of RRM compared to other reward models highlights the effectiveness of the long reasoning reward model.\n\n2. The work includes experiments on large reward models, such as RRM-32B, which provides more convincing conclusions about reward model training at scale.\n\n3. The authors present a relatively thorough analysis of their proposed RRM, including BoN comparisons, RL with unlabeled data, reward model-guided DPO, parallel and sequential scaling analysis, and other detailed evaluations. These in-depth analyses further underscore the authors' significant contributions to the field of reward modeling.\n\n---\n\n**Weaknesses**\n\n1. The paper lacks a deep analysis in certain model comparisons. For instance, Table 1 shows that RRM-32B outperforms Skywork-Reward-Gemma-2-27B-v0.2 on the PandaLM Test but underperforms on RewardBench. What explains these contradictory results? Similarly, in the BoN experiments in Table 2, Skywork-Reward-Gemma-2-27B-v0.2 shows significantly worse performance compared to other reward models, yet it performs strongly on RewardBench. What causes these discrepancies?\n\n2. While the authors provide interesting further analyses, such as RL with unlabeled data, which demonstrate positive effects and training stability, they do not include the performance of other reward models in these scenarios. Would other reward models outperform RRM in tasks like RL with unlabeled data, DPO, or parallel and sequential scaling? A more detailed comparison with baselines in these areas would strengthen the work.\n\nQUESTIONS:\n1. Can RRM effectively guide RL with unlabeled, harder mathematical and coding data, such as tasks with difficulty levels similar to AIME or LiveCodeBench?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces reward reasoning models, an approach to train generative preference models with reasoning capabilities using reinforcement learning. They fine-tune a base reasoning language model using reinforcement learning on a preference dataset with a rule-based reward rewarding correct preference prediction. Reasoning emerges (or further develops from the reasoning capabilities of the base model) as a way to leverage test-time compute and improve preference prediction quality, resulting in larger reward for the model.\n\nAs a secondary contribution, the paper introduces an approach based on in-batch ELO to use pairwise preference models in RL fine-tuning pipelines that expect scalar rewards.\n\nThe authors train multiple reward models (7B, 14B, 32B) using this approach and evaluate them thoroughly, both on direct reward modeling benchmarks and downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\nIn the following, I discuss strength and weaknesses on the dimensions of quality, clarity, significance, and originality.\nWeaknesses are marked with a **W**, minor weaknesses with an **M**, notes that do not impact my score with **N**, and strengths with an **S**.\nWeaknesses are numbered for reference purposes.\n\n## Quality\n\n- **S** The RRM models are evaluated extensively on a variety of benchmarks.\n- **W1** It would be good to discuss related work on general reasoning and parallel test-time compute strategies (e.g., majority vote, knockout tournament).\n- **W2** The evaluation in Table 2 seems slightly unfair to me, as all the baseline reward models are pure preference models (correct?) focused mostly on style, while your model has verifiable tasks in the training data. It would be very informative to have DirectJudge in this table.\n- **M1** I would expect that the batch-ELO based scalar reward derivation has high variance, depending on the batch. It's nice to see that it works nonetheless. It would be even better to have a dedicated evaluation of this, for example by using a natively scalar reward model from which you can derive a discriminative one, which you can then turn back into a scalar one with your proposed approach -- what is the loss here? Another way to approach this would be to compare DPO with ELO+RLHF. I understand that this is not your primary contribution, however, so this does not impact my score much.\n- **M2** Why do you use voting@16 in Table 1, voting@5 in Table 2?\n- **N** The description of Table 6 in the appendix is slightly confusing. It sounds like it should have numerical results matching Figure 10, but the results differ (RRM-7B ELO instead of Knockout tournament).\n- **N** A brief discussion of what PRM is would be useful in Appx B.1.2.\n- **N** You always remove reasoning traces from your training data. Doesn't that mean that reasoning is then out-of-distribution for the trained RM? I would expect that it may further improve performance if you leave some reasoning traces in.\n- **N** It would be interesting to see if you can use the budget forcing approach introduced by S1 (https://arxiv.org/abs/2501.19393) to further improve performance. (Just a note, does not impact my score.)\n\n## Clarity\n\n- **S** The paper is well-written and easy to follow.\n- **W3** Page 8 mentions a 14B model, but you previously only discuss how you trained the 7B and 32B ones. Table 5 also only has hyperparameters for those.\n- **W4** Your RRMs were fine-tuned solely with RL, no SFT right? How many RL steps?\n- **W5** The \"reward model\" terminology may be confusing, as your model does not directly produce rewards but rather choices. It would be more precise to call it a preference model, choice model, or binary classifier. Discriminative reward model (as you use in the paper) could also work as a \"compromise\". Is there precedent for calling a choice model a reward model? If so, I understand if you want to stick to this terminology but you should make this clearer by (1) discussing early on that you propose a discriminative model and (2) take particular care to avoid confusion everywhere you mention \"reward\" (such as in line 47, Figure 1, line 216). I think this may easily confuse a reader.\n- **W6** I do not understand what is evaluated on Figure 4. Number of pairs used for what?\n- **W7** I do not understand what is measured in Table 4. What is accuracy in the context of Tournament / ELO rating?\n- **W8** I would appreciate an analysis of the overhead generated by the ELO-based reward derivation (if possible without extensive new experiments).\n- **M3** How is the CI computed in Table 3? I am surprised that it is not symmetrical.\n- **M4** The paper (especially the RW) seems to imply that generative RMs are always discriminative. I would have thought those are two distinct axes (generative RMs use the full decoder and a textual response, but may still be either scalar or discriminative). I am not perfectly familiar with established terminology, could you elaborate? LLM-Blender (https://arxiv.org/abs/2306.02561) might be a good reference to discuss discriminative RMs, but that is only one possible example and you are of course not required to cite it.\n- **M5** The RW section does not explicitly put RRM into context of the related works (\"compare and contrast\"). This is largely implied, so this is a very minor point, but it would be better to make it explicit.\n- **M6** How does such a discriminative RM contrast to LLM-as-a-judge? Is there any difference?\n- **M7** The paper skips over some preliminaries: What is post-training? On line 31: Decoding layer of what?\n- **M8** What do you mean by the \"round-robin tournament structure\"?\n- **M9** You report batch size and mini-batch size separately in Table 5. It is not entirely clear to me what this distinction means. Is this about gradient accumulation?\n- **N** The colors in Figure 6 (light orange / pink / dark orange / brown) are hard to distinguish when printed.\n- **N** I think it would be worth pointing out that the ELO-based approach is generally applicable to any discriminative model.\n\n## Significance\n\nWith increasing attention being paid to scaling up inference compute for LLMs, scaling up verification is of increasing significance. This paper is a step into this direction.\n\n\n## Originality\n\nExempting concurrent work, the paper is novel to the best of my knowledge. I am not perfectly familiar with the related work, however. At its core it is a fairly straightforward application of the general \"reasoning recipe\" to reward modeling, tempering originality slightly, but it is well-executed and evaluated and therefore a valuable contribution to the literature.\n\nQUESTIONS:\nPlease see the numbered weaknesses, focusing on the major ones (**WX**). Most of them are actionable, i.e., can be considered questions, although many of the minor ones (**MX**) do not require an individual response. Actual questions regarding the understanding of the paper are listed as weaknesses in the clarity section. The minor points have little impact on my score.\n\nThe paper is high-quality and significant. I would consider raising my score if the clarity is somewhat improved. Of particular importance are: W2, W3, and W4.\n\nLIMITATIONS:\n- **W9** The experiments show that there are many types of tasks where reasoning does *not* significantly improve performance (e.g., Table 1 Chat). It still likely has some overhead, even on those tasks. This should be discussed, and ideally the overhead analyzed (is the compute allocation really adaptive, i.e., does it use less on those tasks?)\n- **M10** You should discuss that your \"discriminative to scalar\" scheme only works when there are sufficient candidate responses per prompt available (and mention that this generally the case in common RLHF fine-tuning scenarios).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduced reward reasoning model (RRM), reward model that can generated reasoning about the given generation. They train the reasoning policy via reinforcement fine-tuning (i.e. GRPO) and further boost performance through parallel test-time scaling (e.g., voring over multiple comparisons). The paper evaluates RRMs on multiple ways including, reward benchmark, best-of-N selection, preference alignment, and RL fine-tuning, claiming consistent gains and practical utility.\n\nSTRENGTHS AND WEAKNESSES:\n[S1] Effective CoT-based RL fine-tuning: The paper demonstrates that training long chain of thought (CoT)  reasoning through reinforcement learning can help reward models to enhance the performance of reward model.\n\n[W1] Inconsistent improvements: In the main RewardBench table, RRM without voting does not clearly outperform strong baselines such as DirectJudge; observed gains are small and sometimes statistically insignificant.\n\n[W2] Narrow post-training evaluation: The RL experiments use only a single setting (e.g. DeepSeek-R1-Distill-Qwen-7B on WebInstruct with GRPO). Also, the study lacks comparisons against other baselines (i.e. DirectJudge, score-based RM). It makes it hard to judge whether RRM feedback is uniquely beneficial.\n\n[W3] Lack of technical novelty: The suggested concurrent work combines GRPO with additional technical novelty and often reports stronger results against RRM. RRM relies mainly on standard RL training suggested by Deepseek, which may limit its novelty.\n\nQUESTIONS:\n[Q1] Can you show that RRM-based RL fine-tuning improves performance for other base models or datasets beyond DeepSeek-R1-Distill-Qwen-7B and WebInstruct?\n\n[Q2] When using RRM for RL, please compare RRM with other baselines such as Direct Judge or other outcome reward models.\n\n[Q3] Please suggest more explanation of what is the RRMs’ advantage against other concurrent methods.\n\nLIMITATIONS:\nThe paper does not discuss originality and technical novelty against other concurrent works and reasoning-trained models.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}