Reward Reasoning Models
Authors: Jiaxin Guo
Zewen Chi
Li Dong
Qingxiu Dong
Xun Wu
Shaohan Huang
Furu Wei
Venue: NeurIPS 2025 poster ()
Keywords: reward models
reasoning
large language models
Primary area: deep_learning
TL;DR: We propose Reward Reasoning Models, which leverage additional test-time compute for complex queries where appropriate rewards are not immediately apparent.

Abstract: Reward models play a critical role in guiding large language models toward outputs that align with human expectations. However, an open challenge remains in effectively utilizing test-time compute to enhance reward model performance. In this work, we introduce Reward Reasoning Models (RRMs), which are specifically designed to execute a deliberate reasoning process before generating final rewards. Through chain-of-thought reasoning, RRMs leverage additional test-time compute for complex queries where appropriate rewards are not immediately apparent. To develop RRMs, we implement a reinforcement learning framework that fosters self-evolved reward reasoning capabilities without requiring explicit reasoning traces as training data. Experimental results demonstrate that RRMs achieve superior performance on reward modeling benchmarks across diverse domains. Notably, we show that RRMs can adaptively exploit test-time compute to further improve reward accuracy. The pretrained models are available at https://huggingface.co/Reward-Reasoning.

Reward Reasoning Model
Jiaxin Guo" 1,2
Zewen Chi
Li Dong"
Qingxiu Dong 1.3
Xun Wu'
Shaohan Huang' Furu Wei
Microsoft Research
Tsinghua University
*Peking University
https//akaamm//Gneraalll
Abstract
Reward models play a critical role in guiding large language models toward outputs that align with human expectations. However, open challenge remains in effectively utilizing test-time compute to enhance reward model performance. In this work, we introduce Reward Reasoning Models (RRMs), which are specifically designed to execute deliberate reasoning process before generating final rewards. reasoning, RRMS leverage additional test-time compute for complex queries where appropriate rewards are not immediately apparent. To develop RRMS, we implement reinforcement learning framework that fosters self- evolved reward reasoning capabilities without requiring explicit reasoning traces as training data. Experimental results demonstrate that RRMS achieve superior performance on reward modeling benchmarks across diverse domains. Notably, we show that RRMS can adaptively exploit test-time compute to further improve reward accuracy. The pretrained models are available at https: hatpss:://hgggga////e/ee co/ fo/Reward- Reasoning.
Through
Frontier Reward Modeling Performance
Reinforcement Learning on Unlabeled Data
RRM-328
52
Average Accuracy (%)
80
Metz-ja-Ulama-708
51.12%
MMLU-Pro Accuracy (%)
51
50.62%
75
RRM-78
Athene-RM-706
50-
70
1amma-3.-70B-instruuu
Meta-Jl-Liäma-88
49-47.85%
49.51%
MitemLMM--28B-Reweo
65
Nemetron --w.0044440000000000
Armo-8B-vC.1 DeepSeek-GRMs GRM-227
48-
48.26%
60
Liama 3.1--8-Instruct
47
RL-Skywork
RI-RRM
55
3B
7B
70B
400B
46
0
200
400
Number Parameters (B)
RL Steps
Figure 1: Average accuracy of various reward models on Preference Proxy Evaluations [18] over the MMLU-Pro, MATH, and GPQA subsets. The proposed reward reasoming model (RRM) outperforms previous reward models across model sizes. We also conduct reinforcement learning on unlabeled data, using RRM-32B and Skywork-Rewad -errr---mmm------- Even without ground-trutt tnuth answers, reinforcement learning with RRM achieves significant and steady improvements on MMLU-Pro, which evaluates general-domain reasoning.
[40] as the reward models, respectively.
Equal contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Large language models (LLMs) such GPTs [1, 9] have significantly transformed the field of artificial intelligence. In recent years, the development paradigm of LLMs has evolved from primarily scaling pre-training resources to emphasizing post-training techniques, driven by the dual imperatives of aligning models with human preferences [47] and enhancing specific capabilities like reasoning [6, 58]. This shift reflects growing recognition that model performance depends not only scale but also on sophisticated methods to refine model behavior after initial training.
Reinforcement learning has emerged as fundamental approach in LLM post-training, leveraging supervision signals from either human feedback (RIHF) or verifiable rewards (RLVR) [15, 19, 22, 34, 47J. While RLVR has shown promising results in mathematical reasoning tasks, it is inherently constrained by its reliance on training queries with verifiable answers [22]. This requirement substantially limits RLVR's application to large-scale training general-domain queries where verification is often intractable [16, 30, 60J. contrast, RLHF typically employs a reward model as proxy for human preference, enabling more extensive application across diverse domains [7, 46]. Consequently, the development of accurate and broadly applicable reward models is critical for the efficacy of post-training alignment techniques.
Recent work on reward models can be categorized into scalar reward models [40, 47] and generative reward models [12, 56, 62, 82]. Scalar reward models typically replace the decoding layer, i.e., the linear layer that projects hidden states to token logits, with linear head to predict single scalar value. These models are trained to maximize the margin between the predicted scores of preferred and rejected responses. Generative reward models have emerged as an alternative approach, harnessing the capabilities of LLMs to produce interpretable and faithful feedback. These models offer enhanced flexibility, enabling them to follow adaptive evaluation instructions to construct synthetic training data, thereby facilitating self-impprvemment through iterative refinement [21, 80].
Despite the widespread upplication of current reward models, remains an open challenge to effec- tively scale test-time compute for reward estimation. To serve general-purpose evaluators, reward models should be capable of adapting to a diverse spectrum of queries, ranging from immediately obvious questions to complex tasks that require extensive reasoning [20, 52]. However, existing approaches apply nearly uniform computational resources across all inputs, lacking the adaptability to allocate additional computational resources to more challenging queries. This inflexibility limits their effectiveness when evaluating responses that require nuanced analysis or multi-step reasoning.
To address the aforementioned challenge, we propose Reward Reasoning Models (RRMs)?. Unlike existing reward models, RRM frames reward modeling as reasoning task, wherein the model first produces a long chain-of-thougle reasoning process before generating the final rewards. Since supervised data providing reward reasoning traces are not readily available, we develop a training framework called Reward Reasoning via Reinforcement Learning, which encourages RRMS to self- evolve their reward reasoning capabilities within a rule-based reward environment. Furthermore, we introduce multi-response rewarding strategies, including the ELO rating system [17] and knockout tournament, enabling RRMS to flexibly allocate test-time compute to practical application scenarios.
Extensive experiments on reward modeling benchmarks show that RRMS consistently outperform strong baselines across multiple domains, including reasoning, general knowledge, safety, and alignment with human preference. Besides, we demonstrate the effectiveness of RRMS by applying them in practical applications. specifically reward-guides best-of-N inference and post-training LLMS with RRM feedback. More significantly, we conduct systematic analysis of the test-time scaling behaviors of RRMS, revealing their capacity to adaptively utilize test-time compute to achieve enhanced performance. Furthermore, our analysis reveals that RRMS develop distinct reasoning patterns compared to untrained foundation models, suggesting that our Reward Reasoning via Reinforcement Learning framework successfully guides models to develop effective reward evaluation capabilities. These insights provide deeper understanding reward reasoning processes and will likely inspire the development of future reward reasoning models within the research community.
Our main contributions are as follows:
Ww use the term "reward model" following prior work (e.g., PairJudge RM [41], GenRM [45]), although model produces pairwise choices rather than scalar rewards.

---

We propose Reward Reasoning Models (RRMS), which perform explicit reasoning before producing final rewards. This reasoning phase enables RRMS adaptively allocate additional computational resources when evaluating responses to complex tasks. RRMS introduce a novel dimension for enhancing reward modeling by effectively scaling test-time compute, while maintaining general applicability and effectiveness across diverse evaluation scenarios.
We develop framework named Reward Reasoning via Reinforcement Learning. This framework encourages RRMS to self-evolve reward reasoning capabilities without requiring explicit reasoning traces as training data.
We conduct extensive experiments demonstrating not only the remarkable performance of RRMS in reward modeling but also their promising test-time scaling properties.
2 Related Work
Reward Models Reward models can be characterized along two dimensions: reward formulation and scoring scheme [46, 81). Formulation strategies include numeric only, which assigns scalar scores to query-response pairs [40, 47, 64, 65], and generative, which produces natural language feedback from which rewards may be extracted [3, 5, 11, 12, 42, 59, 73, 77J. Scoring schemes typically follow either absolute approaches, evaluating individual query-response pairs independently [16, 20, 23, 68, 75, 76], or discriminative methods that compare candidate responses to express relative preferences [29, 36, 41, 49, 56, 61, 82].
Generative Reward Models Generative reward models (GRMs), conceptually aligned with the LLM--a-a-Judde paradigm [69, 79], offer nuanced, interpretable feedback with flexibility for both single-instance evaluation and multi-response comparison [33, 45]. This approach addresses lim- itations of traditional evaluation methods like ROUGE [39] and BLEU [48], which struggle with open-ended tasks requiring sophisticated judgment [53]. GRMs can support, judgment across diverse tasks, including multimodal inputs [32, 36, 82]. and contemporaneous work on GRMs demonstrates promising scalability in both model capacity and inference- setrencce-tmee compute [14, 26, 42]. However, concerns persist about evaluation reliability, as LLMs may produce biased or hallucinated judgments that diverge from human standards [1, 10).
Inference- Time Scaling Inference-time scaling dynamically adjusts computational resources dur- ing model inference based on input complexity, inspired by human adaptive reasoning [31, 57, 70]. Recent approaches include parallel scaling strategies such as multi-sampling [8] and reward model- guided aggregation [38, 57, 78], which combine outputs to enhance quality. Alternative approaches use horizon- worizon-based scaling to extend reasoning traces [66]. Advanced systems like OpenAl's 's ol and DeepSeek's 's R1 series demonstrate spontaneous computational allocation that adjusts "thinking hori- zons" in response to task complexity [22, 28). These methods collectively underscore the importance of inference-time adaptability for model performance, particularly on complex reasoning tasks.
3 Reward Reasoning Model
3.1 Input Representation
Figure 2 provides an overview of reward reasoning models (RRMs). RRMS utilize the Qwen2 [71] model architecture with a Transformer-decode
backbone. We formulate the reward modeling task as a text completion problem, wherein RRMS take queries and corresponding responses as input, and autoregressively generate output text consisting of a thinking process followed by final judgment. Unlike existing reward models, RRMS perform th-huuh reasoning before producing rewards, enabling them leverage test-time compute adaptively. We refer to this process as reward reasoning.
Each input of RRMS contains a query and two corresponding responses. The goal of RRMS is to determine which response is preferred, with ties not allowed. We employ the system prompt from the RewardBench repository". which guides the model to perform a systematic analysis of the two responses according to several evaluation criteria, including instruction fidelity, helpfulness, accuracy, harmlessness, and level of detail. The model is also explicitly instructed to avoid common biases,
Phttps://githan cotlb/l.oomaaiiaaa www.///aaaaa/e/eeaaaeeeeea bench
3

---

Scaler
Generative
Q:
3x5=? 15.
Reward Model
0.92
Q 3x5=? A: 15.
Reward Model
9. because
Query Response
Scalar
Query Response
Reward with Justification
(a) Scalor Reward Model
Generative Reward Model
Okay, so need to...
29 Looking back
The answer A.
R1
Q 3x5=? Query & Response
Reward Reasoning Model
Given that
Altematively.. ...Put if...
The
R2
Let's analyze...
Wait, perhaps... Thus..
is better than
R +1
Long Reasoning
Response
Reinforcement Learning
(c) Reasoning Model
An overview of reward reasoning model (RRM). RRM adaptively leverages test-time compute through sfain-of-though reasoning before producing rewards.
Figure 2:
such response order length, and must justify its judgment through structured reasoning before outputting its final decision the format "\boxed( Assistant 1)' or "boxxd Assistant 2)". The detailed prompt template provided in Appendix A.1.
The input of RRMS is restricted to exactly two candidate responses, thereby reserving output length capacity for reward reasoning. Section 3.3 introduces methods by which RRMS assign rewards to scenarios involving multiple candidate responses for given query.
3.2 Model Training with Reinforcement Learning
We develop training framework called Reward Reasoning via Reinforcement Learning to train RRMS. Unlike conventional supervised fine-tuning approaches, which rely on existing reasoning traces, framework encourages RRMS to self-evolve their reasoning capacities within rule-based reward environment. The reward function is defined as follows:
R +1,
RRM selects correct response
(1)
-1,
otherwise
Note that the reward R evaluates whether RRM correctly prefers the ground-truth response, rather than scoring its own outputs. Despite the simplicity of reward signals, such rule-based rewards can effectively supervise policy models to develop reasoning patterns that lead to correct final judgments.
We use Deepseek-R R1 distilled models [22] base models, applying group relative policy optimization (GRPO) [72] for training, implemented with the verl library [55]. More implementation details and hyperparameters can be found in Section 4.1 and Appendix A.2.
3.3 Multi-Response Rewarding Strategies
Although the input structure RRMs strictly accepts two candidate responses, RRMS can adaptively reward multiple responses of specific query. We introduce two rewarding strategies: the ELO rating system and knockout tournament.
ELO Rating System For applications requiring full ratings rather than just identifying the best response, we implement round-robin tournament structure. In this approach, each candidate is compared with all others pairwise. The resulting win-loss records are converted to rating scores using the ELO rating system [17], rating methodology commonly used in chess and other competitive games. While this strategy can process
(*) (n²)) pairwise comparison results, computational cost can be reduced by sampling a subset of the pairwise matchups. The resulting ratings can serve as rewards in reinforcement learning. Experiments demonstrate that we successfully post-train an LLM using these ratings rewards (See Section 4).

---

Knockout Tournament Inspired by the knockout tournament structure [41], we design a knockout tournament strategy for RRMs that organizes multiple candidates into competition bracket. Candi- dates are paired randomly in successive rounds, with winners advancing to subsequent stages. In each pairwise comparison, RRMS determine preferred response that will participate in the tournament in the next round. Given n candidates, this requires n 1 pairwise comparisons with (n)) complexity and O (1)) sequential rounds. Experiments show that the knockout tournament strategy can effectively guide LLMs to perform best-of-N sampling (see Section 4.3 and Appendix B.2).
Both strategies can be combined with majority voting further leverage test-time compute. To integrate majority voting, we first sample RRMS multiple times for each pairwise comparison. Then, we perform majority voting to obtain the pairwise comparison results, enabling seamless integration of majority voting with both approaches. This combined methodology enhances the robustness of the reward assessment while effectively utilizing additional computational resources at test time.
4 Experiments
We design our experiments that evaluate RRMS on both reward modeling benchmarks and practical applications, including reward-guided inference and LLM post-training. Additionally, we analyze how RRMS utilize additional test- esse compute to achieve better performance and examine the reasoning patterns exhibited by RRM across multiple domains.
4.1 Training Details
Training Data Training RRMS requires diverse pairwise preference data spanning capabilities and aligns with human preference. In addition to preference pairs from Skywork-Rewad [40], we further synthesize preference pairs from diverse data sources. We randomly sample 80K queries from the Tülu 3 prompt dataset [34], generate two responses for each using Deepseek-R --Distill-----e--1.5T
[22], and annotate preference labels with GPT-4o [27]. Besides, we synthesize preferences pairs using verifiable question-ane -nnserr pairs from WebInstruct-vering
[44]. Skywork-OR, [24], Big-Math RL [2]. and DAPO-Math [74]. We prompt Deepseek- R1 distilled 1.5B and 7B Qwen models generate several responses for each question, and then apply a rule-based verifier to assess the responses. If at least one response correct and another incorrect, we add the correct-incorrect pair to the training data. We remove intermediate thinking steps from all responses before processing. The final training dataset comprises approximately 420K preference pairs: 80K each from Skywork-Rewand Tülu and our-synthesized data using Tülu 3 prompts, along with 180K synthesized from other sources.
RRM Training We use DeepSeck-Re. Riikk-R-----i RRI--istill--Wee -wen models as the base models for RRMS in all the experiments. The training hyperparameters are detailed in Appendix A.2. The RRM training framework is implemented using the verl library [55]. and we train both RRM-7B and RRM-32B models on AMD Instinct MI300X Accelerators. For RRM-32B, we employ weighted mixture of datasets with a sampling ratio of 5:1:1:1 across Skywork- Reward, Tülu-80K, our GPT-4o-labeled preference pairs, and the other synthetic data. The RRM-7B model trained on similar dataset mixture using a 5:1:1 ratio of Skywork-
Tülu-80K, and GPT-lo-labeled preference data.
4.2 Evaluating Agreement with Human Preference
4.2.1 Setup
We evaluate RRMS on widely-used benchmarks for reward modeling, namely Re- wardBench [35] and PandaLM Test [62]. (1) RewardBench is curated evaluation suite for reward models, consisting of prompt-chosen-rejectee triplets across domains such as chat, reasoning, and safety. It emphasizes fine-graid fine-grained comparisons where one response is subtly but verifiably better, enabling rigorous testing of reward models' capabilities to capture nuanced human preferences. (2) PandaLM Test features a diverse human-annotated test set where prompts and responses are written by humans and labeled with fine-grained preferences. Unlike purely correctness- based benchmarks, M Test covers subjective dimensions such as clarity, adherence to instructions, and formality, providing robust ground truth for for evaluating alignment with human preferences.
Benchmarks
5

---

Table I: Evaluation results RewardBench benchmark and PandaLM Test. Bold numbers indicate the best performance, Underlined numbers indicate the second best.
RewardBench
PandaLM Test
Models
Chat
Chat Hard
Safety
Reasoning
Overall
Agreement
F1
Skywork-Reve 1ett-----mm--------00000 [40]
96.1
89.9
93.0
98.1
94.3
76.6
76.4
JM-7e [82]
87.3
43.6
74.5
48.7
63.5
65.
61.9
JudgeL.M-33B [82]
92.7
54.2
85.8
58.3
72.3
75.2
69.7
(amdb=35--omoet2222440022 [35]
96.4
74.0
81.6
84.7
84.2
DeepSeek-R1 [42, 12]
97.1
73.7
73.3
95.6
84.9
78.7
72.5
DeepSeek-GRRM2278 [42]
94.1
78.3
88.0
83.8
86.0
GP1-4-0125-preview [35]
95.3
74.3
87.6
86.9
86.0
66.5
61.8
GPT-40-0806 [35]
96.1
76.1
86,6
88.1
86.7
[14]
88.9
66.2
78.4
87.0
80.1
RM-R DeaeppSeee--issllld----eeeeee [14]
91.3
91.3
79.4
95.5
88.9
MMM-R--DeepScck-Disiilllee -wwesnilledd--o--2233233 [14]
95.3
80.3
91.1
96.8
90.9
Direct/udge-7B
86.0
69.7
79.5
70.2
DirectJudge-32R
96.1
85.1
89.5
90.9
90.4
76.7
77.4
RRM-7B
87.7
70.4
80.7
90.0
82.2
72.9
71.1
RRM-7B (voting@ 16)
92.1
71.5
93.8
RRM-32R
94.7
81.1
98.3
91.2
78.8
79.0
RRM-32B (voting@
96.1
81.4
91.6
98.6
91.9
80.2
81.9
Baselines We compare RRMs with the following baselines: (1) Skywork-Reward [40], scalar reward model that uses regression head output numerical preference scores without explanations or reasoning traces, (2) Production-grade LLMs, including GPT-4o [27] and Claude 3.5 Sonnet [4], which are prompted in an L-Mdae-a-judge [80] manner to determine the preferred response, (3) JudgeLM [82], which is trained to generate fine-grained reward scores along with explanations, using synthetic training data generated by GPT-4 [1]. (4) DeepSeek-GRM [42] and RM-R1 [14],
two concurrent approaches that also incorporate reasoning phase prior to producing rewards.
In addition to these existing baselines, we introduce (5) Direct,uuge, a pairwise judging model implemented using the same training data and base models as RRMS. DirectJudge models receive the same imputs as RRMS but are trained directly generate judgment without explicit reasoning.
4.2.2 Results
Table presents the evaluation results of baseline reward models and RRMS on the RewardBench benchmark and the PandaLM Test. We observe that RRMS achieve competitive reward modeling performance against strong baselines, demonstrating their effectiveness in producing rewards that align with human preference. Notably, RRM-32B attains accuracy of 98.6 the reasoning category of RewardBench. Comparing RRMS with DirectJudge models, which are trained on the same data, reveals a substantial performance gap in reasoning. This difference indicates that RRMS effectively leverage test-time compute, thereby enhancing performance complex queries that benefit from deliberate reasoning processes.
4.3
Evaluating Reward-Guided Best-of-N Inference
4.3.1 Setup
Preference Proxy Evaluations Preference Proxy Evaluations (PPE) [18] is benchmark designed to evaluate reward models through proxy tasks. Instead of conducting prohibitively expensive full RLHF training runs, PPE proposes proxy tasks that correlate strongly with RLHF-trained model quality. These tasks span large-scale human preference data and senecttees-vvriiaahiee comparisons, with 12 metries covering 12 domains. We conduct experiments on reward guided best-of-N inference, evaluating whether reward models can identify correct responses from set of candidates. Using the response candidates provided by PPE, we focus on three representative datasets, namely MMLU- Pro [18], MATH [18]. and GPQA [18], which examine both general knowledge and mathematical reasoning capabilities. Our evaluation protocol ensures that all models are presented with the identical set of 32 candidate responses for each query.

---

Table Evaluation results on reward
reward-guided guided best-of-N inference. For each query, we use the same 32 response candidates provided by PPE and apply reward models to choose the best response.
Models
MMLU-Pro
MATH
GPQA
Overall
Skywork-RewarddGGnmm--------
67.0
56.3
44.0
55.8
GPT-40-0806
64.8
56.9
46.3
56.0
RRM-7B
69.1
82.0
49.2
66.8
RRM-7B (voting @5)
69.4
86.1
49.0
68.2
RRM-32B
81.3
89.8
61.1
77.4
RRM-32B (voting @5)
83.0
91.8
64.3
79.7
Table 3: Evaluation results on binary preference classification following the protocol from Frick et al.
[18]. For each benchmark, report accuracy single random permutation of paired responses.
Models
MMLU-Pro
MATH
GPQA
Overall
Skywork- (eward--Gemm--2222 [67]
55.0
46.2
44.7
48.6
Gemma-2-22 [42]
66.2
66.4
51.9
61.5
Seepeeek-GMM-22 (voting @32) [42]
65.5
69.4
56.0
63.6
DeepScck-GRM-27B (MetaRM) (voting 32) [42]
68.1
70.0
56.9
65.0
Llama-3.1-8B-Instrue [67]
56.3
62.9
51.4
56.9
-lama-3.1-700--nmss [67]
72.1
73.1
61.2
68.8
J1-Llama-8B (SC@32) [67]
67.5
76.6
55.7
66.7
J1-Llama-700B (SC@32) [67]
79.9
88.1
66.5
78.2
RRM-7B
66.5
88.0
57.9
70.3
RRM-7B (voting @5)
68.3
90.5
58.3
72.4
RRM-32B
80.5
94.3
67.4
80.7
RRM-32B (voting @55)
81.3
95.4
68.4
81.7
Baselines For the first experiment, we employ the knockout tournament rewarding strategy to identify the best-of-N response. We compare our method against several strong baselines, including Bewhork-Reward-Geeeee Appendix A.
[40] and GPT-4o [27]. The prompt template for GPT-4o is detailed in
In addition to best-of-N inference, we also evaluate our reward model following the standard protocol from Frick al. [18]. For this evaluation, we compare RRMS with established baselines including J1-Llama [67], DeepSeek- GRM [42], Skywork-Rewand -award--emma-2222 [40], and various representative reward models from recent literature. Specifically, we report accuracy over a single random ordering of paired responses across different judgment benchmarks. This dual evaluation enables us to assess reward model performance both generative selection (via tournament-style decoding) and binary preference classification tasks.
4.3.2 Results
Table presents the evaluation results on reward-guided best-of-N inference. RRMS surpass all baseline models, even without utilizing additional test-time compute through majority voting. The results demonstrate that RRMS can accurately identify high-quality responses across diverse domains. Moreover, incorporating majority voting leads substantial performance improvements across nearly all evaluated subsets, with the sole exception of RRM-7B on GPQA.
To further analyze the capabilities of RRMS across different domains, provide detailed results on each subset the MMLU-Pro and GPQA benchmarks. As illustrated Appendix B. we Rewe---eaaadddddeeeeee
compare RRMS against highlight the robustness and generalization capabilities our models across a diverse range of subjects, spanning from humanities to STEM fields. This comprehensive analysis demonstrates the versatility of RRMS in accurately evaluating responses across varied knowledge domains.
on each individual domain. The results
Table 3 presents evaluation results on binary preference classification using the protocol from Frick et [18]. RRMS maintain strong performance across three benchmarks, consistently outperforming baseline reward models and instruction-turn LLMs. Notably, RRM-32B achieves

---

state-of-the tatte-of-the-at accuracy on all three benchmarks, even when compared against significantly larger models such as J1-Llama-70B. Furthermore, incorporating majority voting further boosts performance. with RRM-32B (voting 5) reaching peak results across all benchmarks. These further validate the effectiveness of RRMS in classifying reason quality under diverse and challenging evaluation settings.
4.4 Post- West-Trainin with RRM Feedback
In addition to directly evaluating RRMS reward model benchmarks, further assess RRMS by post-training LLMs with reinforcement learning or direct preference optimization, supervised by RRM-generated rewards. This approach allows the downstream performance of the post-trained LLMs to reflect the quality of reward signals. By measuring improvements in the resulting models, we can indirectly validate the effectiveness of RRMS as preference models for guiding model optimization.
4.4.1 Reinforcement Learning with Unlabeled Data
We train Qwen2.5-7B [50] WebInstruct [44] queries using group relative policy optimization (GRPO) [54]. Instead of assigning rewards to each sample individually, we group response samples generated from the same query and have them compete against each other. In each group containing 8 responses, we construct x 8 pairwise matches by randomly selecting 4 competitors for each response, and then obtain the pairwise preference results using RRM-32B. Finally, the rewards are computed using the ELO rating system [17], as described in Section 3. Notably, this approach, named as RL-RRM utilizes only unlabeled queries without requiring any answers or reference responses. As comparison, we adopt a baseline RL method that uses scaler rewards from Skywork- Reward--Gemmn 2-27B-v0.2, named RL-Skyyork.
Following the evaluation protocols established by Ma et al. [44], we evaluate the post-trained models on MMLU-Pro using greedy decoding with maximum response length of SK tokens. Table reports the evaluation results. We find that post-training with RRMS consistently yields competitive or superior performance compared the scaler reward baseline. Specifically, RRM-32B feedback yields stable gains, while the scalar score-based model shows degradation, suggesting potential reward hacking. terms of efficiency, training with RRM-32B for 600 steps required 1026 GPU hours, compared to 265 GPU hours for RL-Skywork. Such overhead remains affordable in practice due to efficient distributed training on modern accelerators. These results validate both effectiveness and feasibility of RRMS as reward models for reinforcement learning, demonstrating their practical utility in post- post-nraining pipelines.
To further demonstrate the effectiveness and generality of RRMS, we conduct additional experiments with spanning base models and new training paradigm, i.e., test-time reinforcement learning, with results provided in Appendix C.
4.4.2 Direct Preference Optimization
To further explore the utility of RRMS in post-training pipelines, we apply Direct Preference Optimiza- tion (DPO) [51] Qwen2.5-7B [50] using preference labels annotated by different reward models. Specifically, we construct preference datasets from Tülu [35] with 80K queries and responses, and obtain preference annotations from three different verifiers: RRM-7B, RRM-32B, and GPT-4o. Each model independently labels the preferred response as the supervision signals for DPO.
The trained models are evaluated on the Arena-Hard benchmark [37], which contains challenging instructions designed to test comprehensive model capabilities. As shown in Table 5, all trained models outperform the original Qwen2.5-7B model, demonstrating the effectiveness of preference supervision from reward models. Notably, the model trained with RRM-32B labels achieves the highest Arena- score, highlighting the practicality of using RRMS to produce high-quality supervision signals for DPO.
4.5 Scaling Test-Time Compute
4.5.1
Parallel Scaling
We conduct parallel test-time compute scaling experiments on MATH [25] reasoning candidate responses. We use 1-1.5---aah------nte [72] to generate 8 candidate responses for each query,

---

Models
RL Step
MMLU-Pro
Arena-Hard Score
CI
Qwen2.5-7B
47.85
Before Post-Training
RL-Skywork
200
49.51
Base Model
18.3
(-1.61, +1.66)
RL-SSkworr
400
48.26
DPO with Preference Data Annotated by
RL-SSkworrk
600
34.76
Reward Models
RL-RRM
200
50.62
GPT-4o
51.9
(-2.96, +2.93)
RL-RRM
400
51.12
RRM-7B
53.8
(-1.72, +1.85)
RL-RRM
600
50.86
RRM-32B
55.4
(-2.60, +2.67)
Table 4: Post-training MMLU-Pro results for
Table 5: Performance of DPO post-trained Qwen2.5-7B models on Arena-Hard.
Qwen2.5-7B using feedback from RRM-32B vs. wework-Reaard-Gema..
and then employ RRMS to perform reward-guided best-of-N inference. This setup allows us to systematically study the scaling behavior of RRMs under increased test-time computational resources.
Scaling Properties We evaluate the effect of the number of pairwise comparisons used to rank candidate solutions on MATH problems. For each problem, we select subset of pairs from these sampled candidates for pairwise comparison. As illustrated in Figure 3, increasing the number of pairwise comparisons steadily improves the performance of sward-guided best-of-N inference on MATH for both RRM-7B and RRM-32B. This consistent trend indicates that RRMS can adaptively utilize dynamic test-time compute budgets to refine their final outputs. We also explore the effects of majority voting, which leverages additional test-time compute by sampling RRM outputs multiple times. Table compares the performance on MATH, where RRMS are prompted on each comparison pair either single time eight times, with the latter followed by majority voting. We observe that majority voting serves as an effective method to translate increased test-time compute into performance gains, further demonstrating the scalability of our approach.
Comparing Rewarding Strategies Table 6 compares the scoring strategies, specifically using RRMS evaluate candidates through either knockout tournament ELO rating systems. Results demonstrate that ELO rating consistently outperforms knockout tournament with both RRM-7B and RRM-32B. Nonetheless, the knockout tournament yields only slightly lower performance while clliciency-peffommacce requiring fewer computational resources- only ((n) comparisons. This efficiency performance tradeoff highlights the flexibility our approach adapting to different computational constraints.
RRM-7B
RRM-32B
90
Majority Voting
No
Yes
No
Yes
MATH
89
Tournament ELO Rating 88.5
88.2
88.7
90.0
90.4
88
88.8
90.3
90.5
Table 6: Comparison of scoring strategies using RRM verifiers. Accuracy refers to the correctness of the top- ranked (for ELO rating) or final-selected (for Tourna- ment) response. ELO rating consistently outperforms Tournament scoring in terms of accuracy for both RRM- 7B and RRM-32B.
10
10a
Number pairs
Figure 3: MATH accuracy with varying number pairwise comparisons.
4.5.2 Sequential Scaling
We study the impact of enabling longer chains of thought [66] before finalizing an answer. We evaluate RRMS on RewardBench, where we control the thinking budgets by setting maximum token limit. no transition signal is generated before the limit, the phase is truncated. We also set a small post-thinking budget to prevent compute hacking, i.e., ensuring that performance improvements genuinely reflect the effectiveness of the reasoning capabilities of RRMS rather than merely increasing output length. The detailed design of the post-thinking budget can be found in Appendix D.I.

---

90
Chat
RewardBench (%)
RewardBench (%)
Chat Hard
Safety
Reasoning
85
80
Overall
80
70
328
148
78
60
1e3
2e3
4e3
8e3
0
200
400
600
Thinking budget (tokens)
Training steps
Figure 4: Results on RewardBench vary- ing thinking budgets.
Figure 5: Results on RewardBench throughout RRM- 7B training.
Results Experiments on 7B, 14B, and 32B RRMS show that longer thinking horizons consistently improve output accuracy across all model sizes (Figure 4). The improvements are consistent across different model capacities, demonstrating that RRMs are capable of effectively utilizing extended thinking budgets to progressively enhance rewarding accuracy. This finding confirms that the reasoning capabilities of RRMS can be scaled through additional sequential computation, providing flexible approach improving the performance of reward models that requires neither larger model sizes nor additional inference passes.
Adaptive Compute Allocation In addition to studying fixed thinking budgets, observe emergent adaptive-compute behavior in RRMS. During of-thugt reasoning, the model can choose terminate its reasoning phase early, explicitly signaled by special </think> token. This results in dynamic allocation of computational resources: simpler tasks in the Chat domain generally produce shorter reasoning sequences, whereas more complex tasks in the Reasoning domain yield longer sequences, shown in Table 10 in Appendix D.2. While this behavior is not explicitly designed, reflects the model's ability adaptively use compute based on task complexity.
4.6 Scaling RRM Training Compute
We investigate how model size and training duration affect the performance of RRMS, exploring the scaling properties of our reward reasoning approach across different compute dimensions. Figure compares RRMS with model sizes of 7B, 14B, and 32B on RewardBench, showing consistent performance gains with increased model size.
We further analyze how training duration affects model performance by tracking RRM-7B on Re- wardBench throughout the training process. Figure 5 illustrates the performance trajectory across different evaluation domains. We observe steady improvements across all domains, with no signs of overfitting even after extended training. This stable learning curve validates the effectiveness of our reinforcement learning framework developing robust reward reasoning capabilities.
5
Conclusion
In this paper, introduced Reward Reasoning Models (RRMs), incorporating explicit reasoning processes prior to reward assignment. We demonstrated that incorporating chain-of-thought reasoning into reward models leads superior judgment accuracy across diverse domains, achieving competitive performance on established benchmarks. Our Mie-based-rewarr reinforcement learning successfully enabled RRMS to develop sophisticated reward reasoning capabilities without requiring explicit reasoning traces as supervision. Extensive experiments confirmed that RRMS efficiently leverage test-time compute through both parallel scaling and sequential scaling approaches. More importantly, we demonstrated the effectiveness of RRMS in practical settings such as swaard-guided best-of-N inference and post-training with RRM feedback. We will open source the code and models to support. and accelerate research within the LLM post-training community.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models, 2025.
URL https: /arxiv org/ abs/2502 11552.17382 17387.
[3] Andrei Alexandru, Antonia Calvi, Henry Broomfield, Jackson Golden, Kyle Dai, Mathias Leys, Maurice Burger, Max Bartolo, Roman Engeler, Sashank Pisupati, et al. Atla selene mini: A general purpose evaluation model. arXiv preprint (.ti:2200111199. 2025.
[4] Al Anthropic. The claude model family: Opus, sonnet, haiku. Claude-3 Model Card, 1:1,
2024.
[5] Negar Arabzadeh, Siqing Huo, Nikhil Mehta, Qingyun Wu, Chi Wang. Ahmed Hassan Awadallah, Charles L. Clarke, and Julia Kiseleva. Assessing and verifying task util- ity in LLM-powered applications. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 21868-21888 Miami, Florida, USA, November 2024. Associa- tion for Computational Linguisties. doi: / //aclanthology org/2024. 20224 emnlp- 1219/.
1219. URL https:
[6] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint wrXiv:204.00886 2022.
[7] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. 2022.
[8] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint 2rX1v:2407.21787, 2024.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33: 13:777711901, 2020.
[10] Sébastien Bubeck, Varun Chadrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4, 2023.
[11] Maosong Cao, Alexander Lam, Haodong Duan, Hongwei Liu, Songyang Zhang, and Kai Chen. Compassjudger-1; All-in-one judge model helps model evaluation and evolution. arXiv preprint 1rXiv:2410.162566 2024.
[12] Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, and Bingsheng He. Judgeirm: Large reasoning models as a judge. arXiv preprint (tt::2544.0005000 2025.
[13] Runjin Chen, Zhenyu (Allen) Zhang, Junyuan Hong, Souvik Kundu, and Zhangyang Wang. Seal: Steerable reasoning calibration of large language models for free. 2025. URL https:
//api aapi..emanticcccoll.. w.w.aaap.....oo.........
[14] Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, et Rm-rl: Reward modeling as reasoning. arXiv preprint w2Xi:22505.02387 2025.

---

[15] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 4302-4310, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
[16] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 2021.
[17] Arpad E. Elo. The Rating of Chessplayers, Past and Present. Arco Pub., New York, 1978. ISBN 0668047216. URL http:/ //wp:.//wwwwwww amazon. cow ctwc/oot/iiia Chess--hhss-ppae dp/ Op/066804.
[18] Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. How to evaluate reward models for RLHF. In The Thirteenth International Conference Learning Representations, 2025. URL betps://openreweee
[19] Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for Ilm reasoning. arXiv preprint arXiv:2410. /5/ 15, 2024.
[20] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JMLR.org. 2023.
[21] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594 2024.
[22] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
[23] Fang Guo, Wenyu Li, Honglei Zhuang, Yun Luo, Yafu Li, Le Yan, Qi Zhu, and Yue Zhang, Mcranker: Generating diverse criteria on-the-fly to improve pointwise 11m rankers. In Pro- ceedings of the Eighteenth ACM International Conference on Web Seurch and Data Min- ing, WSDM '25, page 944-953, New York, NY, USA, 2025. Association for Comput- doi: 10.1445/701155..775588.. URL https:
ing Machinery. ISBN 0198400713293.3 //doi. //do/.org/10. wtt..o///...1//37115551111111111 /tt///3711551.37355588
[24] Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng. Tianwen Wei, Cheng Cheng, Yang Liu, and Yahui Zhou. Skywork open reasoner series.
https://capricioos--hhhooggg Skywork- fre.neee-.eeeeetooooeeeeeeeeeeee
motion.sita
2025.
Notion Blog.
[25] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https: //tterev//penreviee wwt.:eet/oooo mett/foruridd-7yyyttnneee.
[26] Ilgee Hong, Changlong Yu, Liang Qiu, Weixiang Yan, Zhenghao Xu, Haoming Jiang, Qingru Zhang, Qin Lu, Xin Liu, Chao Zhang, et al. Think-rm: Enabling long-horizon reasoning in generative reward models. arXiv preprini arXiv:2505.1626: 2025.
[27] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
[28) Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Kl-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.167700 2024.
12

---

[29] Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-blender: Ensembling large language models with pairwise ranking and generative fusion. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165-14178, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.16663//1/22223acllooom URL fttpps://acaanthologgg. hwwpttthooogooooooogeoooo wwwp//22..aa..ooo a0023.acl-10 acc.oooo......nn...... maac-lloo..9922 792/.
[30] Carlos E Jimenez, John Yang. Alexander Wettig, Shunyu Yao. Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https:
//openreview wtttteeemtiddeeeeeeeeeeee VTPT.
[31] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar wxxx:2001.08366 2020.
[32] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. In 2023 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 19113-19122, 2023. doi:
10. WWWp//CCPPS272.222..000
[33] Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4333-4353, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v 11133//12222.m////// m000044.mml--miii2255 main.248. URL https: http:::///aaaatttaa hatpa/aaaaannoogo.......oooo.oae org/oog//// /224.. emnle- mop-mmai.... 248/
[34] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T ulu 3: Pushing frontiers in open language model post-training. arXiv preprint 2.. 2024.
[35] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403. 13787, 2024.
[36] Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, hai zhao, and Pengfei Liu. Generative judge for evaluating alignment. In The Twelfth International Conference on Learning Representations, 2024. URL https: //open/evee/eeeee///ee net/ fooum/id=ggtaff62222
[37] Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and lon Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline, 2024. httptt / /rrxiv...vvv org/obg /a//ab///24/// 11939.
[38] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, 1lya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL https: / //openreview. wwtttftouttidd/eeeeeeee /tt
[39] Chin-Yew Lin, Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[40] Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410. 2et.:2441..8811.1. 2024.
[41] Yantao Liu, Zijun Yao. Rui Min Yixin Cao, Lei Hou, and Juanzi Li. Pairwise rm: Perform best-of-n sampling with knockout tournament. arXiv preprint arXiv:2501.13000 2025.
[42] Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-ting time scaling for generalist reward modeling, arXiv preprint arXiv:2504.02495, 2025.
13

---

[43] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai. Jeffrey Luo, Li Erran Li, Raluca Ada Popa. and Ion Stoica. Deepscaler: Surpassing ol ol-preview with a 1.5b model by scaling rl, 2025. Notion Blog.
[44] Xueguang Ma, Qian Liu, Dongfu Jiang. Ge Zhang, Zejun Ma, and Wenhu Chen. General-reasoner: Advancing Ilm reasoning across all domains. https://githe com/ TIGER-AI
[ Reasoner pdf, 2025.
[45] Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprini arXiv:2410. 12822 2024.
[46] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: hww.thttt//s/eetee httt:::///ppeervieww net/ forumPid-TGKKACXEON
[47] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[48] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[49] Junsoo Park, Seungyeon Jwa, Ren Meiying, Daeyoung Kim, and Sanghyuk Choi. OffsetBias: Leveraging debiased data for tuning evaluators. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1043-1067, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10..1853/v1//// 1tt.l////222222/ htt.://////2....nnnnnns-ooooe 2024. findings-nniiii emnlp.5m17 57/.
URL https: ptattps:/aaclathooen aclpaahollntoooyoooo org/
[50] Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: //t////aa// aarxiv.v.vvv////v///aa./a/aavava herggabb/241111 2412.. 15115.
[51] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In sevethh Conference on Neural Information Processing Systems, 2023. URL :ttps://openreeiez: wttprriiiiieeeeffteeeeeeee
Iaaa..
[52] Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, W. Bradley Knox, Chelsea Finn, and Scott Nickum. Scaling laws for reward model overoptimization in direct aligmment algorithms. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL. https: http::/s://////ee///e/ee/ nwt/ cofoormmmiidpppr 40ff400yyyn,
[53] Natalie Schluter. The limits of automatic summarisation according to rouge. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 41-45. Association for Computational Linguisties, 2017.
[54] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https: //arxiv. org/ abs/ 2402. 03300.
14

---

[55] Guangming Sheng, Chi Zhang, Zilingfeng Ye. Xibin Wu, Wang Zhang. Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. arXiv preprini arXiv: 2409. 19256, 2024.
[56] Tu Shiwen, Zhao Liang, Chris Yuhao Liu, Liang Zeng, and Yang Liu. Skywork critic model se- ries. https: / http:///hgggggeeeeeeeee co/Skywork, September 2024. URL hwtttt://hhugiiggaaee co/Skywork.
[57] Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling L.M test- time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https: / //openreview.o net/ met//oorrumiico4FF/////a2
[58] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
[59] Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. Foundational autoraters: Taming large language models for better automatic evaluation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17086-17 105, Miami, Florida, USA. November 2024. Association for Computational Linguistics. doi: 10. 1/2024. emnlp-main.949, URL http:: http//allaattt https/////caaaahloooo// org/ emnlp-mainn .49/.
[60] Binghai Wang, Rui Zheng. Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.0608) 2024.
[61] Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-YY Yu, Richard Yuanzhe Pang, Maryam Fazel Zarandz, Jason Weston, and Xian Li. Self-taught evaluators. arXiv preprint 240::2408.006666, 2024.
[62] Yidong Wang. Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang. Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. Pandalm: An automatic evaluation benchmark for Ilm instruction tuning optimization. In ICLR, 2024. URL https: http:p:://ppeneeiiii
[63] Yiping Wang, Qing Yang, Zhiyuan Zeng. Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao. Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. Reinforcement learning for reasoning in large language models with one training example. 2025. URL https: //api. www.etiiscooolleteeeeee.ee org arggCorrpsIDD 278171513.
[64] Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint arXiv:23 1...///..052... 2023.
[65] Zhilin Wang, Yi Dong, Olivier Delallcau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top-performing reward models. In The Thirty-eight Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: / //openreview. net www.rruummm..d...kuhahh.....
[66] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors. Advances in Neural Information Processing Systems, 2022. URL https: htt::///ppee/iiiee net/ Vijunnidd_VjiMMeSS
[67] Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swar- nadeep Saha. J1: Incentivizing thinking in Ilm-as-a-judge via reinforcement learning. arXiv preprint 2005 2025.
15

---

[68] Genta Indra Winata, David Anugraha. Lucky Susanto, Garry Kuwanto, and Derry Tanti Wijaya. Metametrics: Calibrating metrics for generation tasks using human preferences. In The Thirteenth International Conference on Learning Representations, 2025. URL
https: hatpp:://ppereeeee /tt//ppnrrviee.nte//oruute oot.oorrrmmmdd1111x1100
[69] Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a aa----s----ttee muda--jugge arXiv preprint wrXiv:2407.19599 2024.
[70] Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for LLM problem-solving. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
/ /openreview. wwwpeerevee.eee/foorrreeeee wiw.ormmmiiNNccpppeB........
[71] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
[72] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self- inff-improveeent. arXiv preprint arXiv:2409. 12122, 2024.
[73] Ziyi Ye, Xiangsheng Li, Qiuchi Li, Qingyao Ai, Yujia Zhou, Wei Shen, Dong Yan, and Yiqun LIU. Learning LLMMaasaajudge for preference alignment. In The Thirteenth International Conference on Learning Representations, 2025. URL htttp:://openrevier wttp:eeviwwweveeeeeeeeeeeeeeeeee net/forum? id=HZVIQE1MsJ.
[74] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025.
URL https: / //txxvivvvva org/ abs//503/////// 14476.
[75] Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, Melanie Kambadur, Dhruy Mahajan, and Rui Hou. Self-generated critiques boost reward modeling for language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 1: Long Papers), pages 11499-113 11514, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 1999-889176--8944 htt.::/t::// / /a///ccallna aclannhologyy...... org/ maacl-long 573/-
[76] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org. 2024.
[77] Lunjun Zhang, Arian Hosscini, Hritik Bansal, Mehran Kazemi, Aviral Kumarr, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In The 4th Workshop on Mathematical Reasoning and Al at NeurlPS 24, 2024. URL https: //tppe/eeeeeeeeeeee/eeeeeeeeeeee net/ forum?id-CxHRooLnnnn
[78] Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07333 2025.
16

---

[79] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging L.LM-as-a-judge with MT-hench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL. htte://ooommmiducccOOPt
pettp::/opeereviiee
[80] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46663 2023.
[81] Jialun Zhong, Wei Shen, Yanzeng Li, Songyang Gao, Hua Lu, Yicheng Chen, Yang Zhang, Wei Zhou, Jinjie Gu, and Lei Zou. A comprehensive survey of reward models: Taxonomy, applications, challenges, and future. arXiv preprint arXiv:2504.1253 2025.
[82] Lianghui Zhu, Xinggang Wang. and Xinlong Wang. JudgeLM: Fine-tuned large language mod- els are scalable judges. In The Thirteenth International Conference on Learning Representations, 2025. URL https: hipps:///pppneee //ppnreveiieeeeeee////ee net. aet//forum//d=xxeBBppPPAad
17

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect
paper's contributions and scope?
Answer: [Yes]
Justification: Both the abstract and beginning part of introduction indicate the scope of the paper's 's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [NA]
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
18

---

Justification:
This article is based on experiments and does not require strict theoretical
proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Results can be reproduced according to "Reward Reasoning Models" and "Experiments" chapters.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Please find the code in homepage of RRM.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
http://ddeeeeeeeeeeeaaasssiieeea oosoooooossiioooocoocoocoo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. cc/public/guideeeeoc
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to the "Experiments" and "Appendix" chapter.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Although we do not report explicit error bars or confidence intervals, our evalua- tion procedures-such majority voting multiple runs with different impliiilylyyitly account for variability in predictions. These repeated trials provide a level of statistical robustness and help ensure that the reported performance stable and reliable, even in the absence of formal error bar reporting.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer the "Experiments" and "Appendix" chapter.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tttrr:p///urrps.... wttpiiip..cpppuubee wwtppppwpc/tpciicceeeeeaeppeeeee
Answer:
[Yes]
Justification: The authors have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work contributes to improving the evaluation quality of large language models (LLMs) by developing model that can assess the correctness and reasoning quality generated answers. Such reward modeling techniques are essential for aligning LLMs with human preferences and promoting safer and more reliable AI systems. This line of research may positively impact the broader I.LM ecosystem by enabling more robust evaluation, facilitating post-training, and reducing reliance manual annotations.
21

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used in this work are properly cited and used in compliance with their respective licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
22

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com comdattasess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release new models as part of this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: / wwt..///eeeeeneee/222222222 for what should or should not be described.
24

---

A RRM Implementation Details
A.1 Prompt Template
Here we provide the detailed prompt template used to guide our Reward Reasoning Models (RRMS) in performing systematic comparative analysis between two responses. We adapt the prompt template from the RewardBench repository, which instructs the model to evaluate two responses based on criteria including instruction following, helpfulness, accuracy, harmlessness, and level of detail. To reduce bias, the model is directed to ignore response order, length, and stylistic differences unless they affect helpfulness. The model must output its final verdict in the format "/boxed Assistaat 1)" or Aboxed Assistant 2)', with no ties allowed.
RRM Prompt Template
You are helpful assistant evaluating the quality of the responses for a given instruction. Your goal is to select the best response for the given instruction. Select Assistant or Assistant 2, that is better for the given instruction. The two responses are generated by two different AI assistants respectively.
Do NOT say both neither are good.
Here some rules of the evaluation:
(1) the instruction does not contain harmful content, you should prioritize evaluating whether executes the instruction, then consider its helpfulness, accuracy, level of detail, harmlessness, etc.
the output panestly/precissly/llooo
(2) If the instruction contains harmful content, prioritize the harmlessness and safety of the response.
(3) Responses should NOT contain more/less than what the instruction asks for, as such responses do NOT precisely execute the instruction.
(4) You should avoid any potential bias and your judgment should be as objective as possible. Here are some potential sources of bias:
The order in which the responses were presented should NOT affect your judgment, as Response A and Response B are equally likely to be the better.
The length of the responses should NOT affect your judgement, as longer response does not necessarily correspond to better response. When making your decision, evaluate if the response length is appropriate for the given instruction.
(5) Your output should only consist of "\boxed[ Assistant 1)" if assistant 1 is better, or "Aboxed Assistant 2)" assistant better. Omit any
other output.
## Query
[Query)
## Assistant responses
### Assistant
(Response 1}
### Assistant 2
(Response 2
## Analysis Let's analyze this step by step and decide which assistant is better, and then answer AssisatAssistant 1) or \boxed! Assistant 2).
In addition to the training prompt used for RRM models, also include the evaluation prompt used for querying GPT-4o on the PPE benchmark. We follow the prompt format from Liu et al. [42], which instructs GPT-4o select the best response from set of candidates based several criteria. This standardized evaluation approach ensures fair comparison between different reward modeling methodologies.
25

---

LL---as-a-Judee Judge Prompt Template
You are a skilled little expert at scoring responses. You should evaluate given responses based the given judging inniven the context of the conversation (the last round is the User's query) and multiple responses from the Assistant, you need to refer to the [General Evaluation Criteria] score the responses. Based on the general evaluation criteria, state potential other specific criteria to the query, the weights different criteria, and then select the best response among all candidates. sandidates..BBeffre judging, please analyze step by step. Your judgement needs to be as strict as possible.
##### Evaluation Criteria #####
1. Instruction Adherence:'n Fully Adhered: and requirements of the question. Partially Adhered: The response meets most of the instructions but has some omissions or misunderstandings.uu response meets some instructions, but the main requirements are not fulfilledAAn Not Adhered: The response does not meet any instructions. Example: the question requires three examples and the response provides only one, it falls under "Partially Adhered." 2. Usefulness:\n Highly Useful: The response provides comprehensive and accurate information, fully addressing the issue.An Useful but Incomplete: The response provides some useful information, but lacks details or accuracy.In Limited Usefulness: The response offers little useful information, with most content being irrelevant incorrect.An Useless Incorrect:
The response fully complies with all instructions
Basically Adhered: The
The response is completely irrelevant or incorrect.bv in Example: If there are factual errors in the response but the overall direction is correct, it falls under "Useful but Incomplete." Level Detail'n Very Detailed: The response includes ample details covering all aspects of the issue. Detailed but Slightly Lacking: The response is fairly detailed but misses some important details. Basically Detailed: The response provides some details but is not thorough enough overall. Not Detailed: The response very brief and lacks necessary details.An Example: If the response provides only a simple conclusion without an explanation, falls under "Not Detailed." 4. Relevance:An Highly Relevant: The response highly relevant to the question, with information closely aligned with the topic. Generally Relevant: The response generally relevant but includes some unnecessary information. \n Partially Relevant: The response has lot of content that deviates from the topic. Not Relevant: The response completely irrelevant. in Example: the response strays from the topic but still provides some relevant information, it falls under "Partially Relevant."
#### Conversation Context ####
[conversation context & query)
#### Responses to Scored ####
[The Begin of Response]
(the response) } [The End of Response]
##### Output Format Requirements #####
Output with three lines
Specific Criteria: <Other potential criteria specific to the query and the context, and the weights of each criteria>.
Analysis: <Compare different responses based on given Criteria>.
Scores: <the index of the best response based the judgement, in the format of <\boxed(x)>
A.2 Hyperparameters for Training RRM
Table 7 presents the key hyperparameters used for training RRMs. These parameters were carefully selected optimize the reinforcement learning process and ensure effective development of reasoning capabilities in models.
26

---

Table 7: Hyperparameters used for training RRMS.
Hyperparameters
Batch size
128
Mini-batch size
64
KL loss coefficient
10 3
Sampling temperature
0.6
Maximum prompt length
4096
Maximum response length
8192
GRPO group size
16
Learning rate (RRM-32B) Learning rate (RRM-7B)
5
10-7
10
B Reward-Guided Best-of-N Inference
B.1 Detailed Results on Subsets of MMLU-Pro and GPQA
We present detailed results on the constituent subsets of MMLU-Pro and GPQA benchmarks. Figure illustrates the performance comparison among our RRM-7B, RRM-32B models and Skywork-Reward. Gemma-2-27B-v0.. across the 14 subsets of MMLUUPro. These subsets span diverse knowledge domains including humanities, social sciences, STEM, and professional fields. The results reveal work-Rewarrd-Gerr
interesting patterns in model performance. Notably, RRM-7B outperforms 2-27B-v0.2 in several STEM-related categories, despite having significantly fewer parameters, highlighting the effectiveness of our reward reasoning approach.
Rkywoodk-Reowrr
RRM 78
1UB16-52B
0.8
Accuracy
0.6
0.2
8.0
Chemistry
Engineering
Math
Law
Biology
Computer Science
Health
Other
History
Physles
Business
Phitosophy
Psychology
Economics
Figure 6: Performance comparison of Hework-RRewarde Gemma-2-277- RRM-7B, and RRM- 32B on the 14 subsets of MMLU-Pro.
More impressively, our RRM-32B model demonstrates consistently superior or comparable per- formance across all subsets compared to highlights the robust generalization capabilities of our larger model across diverse knowledge do- mains. The comprehensive dominance of RRM-32B underscores the scalability of our approach and confirms that the reward reasoning framework effectively improves judgment accuracy across the full spectrum of evaluated categories.
This consistency
Similarly, Figure presents the performance breakdown across the GPQA subsets. The pattern remains consistent, with RRM-7B showing stronger performance in certain technical categories while occasionally lagging behind Skywork-Rews Rewarr--Gemm--2227711 Mcanwhile, RRM-32B maintains excellent performance across all subsets. This comprehensive
more general knowledge areas.
27

---

analysis further validates the effectiveness of our reward reasoning approach in handling complex scientific and technical queries that require sophisticated judgment capabilities.
Skywork Roward
19MM-78
FRM-32B
0.0
Accuracy
0.4
Quantum Mechanics
Chemistry (General)
Organic Chemistry
Molecular Biology
Electronagnetss And Photonics
Physics (General)
High Enee-Enrryyy Particle Physles
Genetics
Astrophyskcs
Relativistic Mechanics
Physkal Chemistry
Condensed Matter Physies
Inorganic Chemistry
Statistical Mechanics
Optics And Acoustics
Analytical Chenistry
Figure 7: Performance comparison of 32B on the 16 subsets of GPQA.
RRM-7B, and RRM-
B.2 Go Further into Knockout Tournament
To gain a deeper understanding of the knockout tournament strategy described in Section 3, we conduct additional experiments following the setup in Section 4.5.1. We compare the performance of different methods on selecting the best response among 8 candidates generated by Qwen2.5-Mait 7B-Instruct for each MATH question. We reward the responses with RRM-7B and RRM-32B, and benchmark them against Qwen2.5- Dowe2.5--Mthh-PRE Math-PPMM-7B and Siren2.5-Math-PRM..
[78]. In addition using reward models, we also include non-verifier strategies such as majority voting (Voting 8) and best-of-N oracle selection for reference. This comprehensive comparison allows us to assess the relative effectiveness of our approach against established methods in the literature.
The numerical results are summarized Table 8. all comparison methods, including votingg8 and PRM judges. RRM-32B further narrows the gap toward oracle-level accuracy, significantly outperforming PRM-based baselines. These results demonstrate the superior discrimination capabilities of our reward reasoning approach, even when compared to specially designed mathematical preference models. The consistent performance advantage across different model sizes confirms the effectiveness of framework in identifying high-quality mathematical reasoning across varied problem complexities.
When compared with baselines, RRM-7B surpasses
Table 8: Comparison between RRM and Qwen2.5- 1---5---ahh-PPPP Math-PPMM models on MATH.
Models
Accuracy
Voting@8
86.8
Best-of-8 Oracle
91.7
Sween2.5-Math-PR...
87.8
Sienn2.5MMtt-PRRM-22
88.5
RRM-7B
88.7
RRM-32B
90.4
As shown in Figure 8, the knockout tournament progresses through successive elimination rounds, we observe consistent improvement in accuracy, demonstrating the benefits of iterative comparison. Notably, the knockout tournament achieves this consistent accuracy improvement with only ((n) pairwise comparisons. This efficient scaling behavior highlights the practical advantage of our
28

---

approach in scenarios where computational resources may be constrained, providing an effective balance between performance gains and computational requirements.
RRM-328 Flo.
90
RRM-78 Elo
Gwen2:5-PMM-700
MATH accuracy
88
Owen2.5-PRM
VoLingd88
86
84
RRM-78 78 with voting
RRM-7B without voting
RRM 328 with voting
RRM 32B without voting
82
0
1
2
3
Tournament round
Figure 8: Accuracy progression of the knockout tournament strategy on MATH elimination rounds proceed.
C Additional RL Experiments
Beyond the main results reported in Section 4.4, we further validate the generality of RRMS through two sets of complementary experiments.
Alternative Base Model and Downstream Task We repeat the post-training experiments using Deepseek-RI- RI-Distill QB [43] as the base model. Figure 9 and Figure 10 present the evaluation curves GPQA and MMLU-Pro downstream performance improves consistently during training, confirming that RRMS remain effective reward models even when applied to different architectures and datasets.
the reinforcement learning proceeds. We observe that the
Test- Time Reinforcement Learning. We further explore the applicability of RRMS under the test- time reinforcement learning (TTRL) paradigm. Specifically, we train DeepScaleR- 1.5B 1.5B--review [43] on unlabeled queries from AIME24. As shown in Table 9, RRMS successfully guide the model to improve its performance on challenging math problems, even without labeled data.
Table 9: Pass on AIME24 using samples per question. RRM-guided TTRL improves 1eepScaleR-1.5--Preve performance on math problems without using labeled data.
Model
AIME24 Pass
--t.sSaale--..--Peeee TTRL w/ RRM-32B
43.3
50.0
D Response Length Analysis
D.1 Post-Thinking Lengths
To evaluate the impact of thinking budget model performance, we need to establish appropriate token budget for the response phase that follows the thinking phase. This ensures that any performance improvements can be attributed to deeper reasoning rather than simply allowing more verbose outputs.
29

---

40
56
MMLU-Pro
54
GPQA
35
52
30
50
200 400 600 800
0
200 400 600 800
RL training steps
RL training steps
Figure 9: GPQA accuracy of using RRM for RL. post-training.
Figure 10: MMI.U-Pro accuracy of using RRM for RI. post-training.
The careful calibration of this post-thinking budget is critical for isolating the effects of extended reasoning from potential confounding factors related output length.
We analyze the token length distribution of responses generated by RRM-32B on the RewardBench dataset after the thinking phase concludes. Figure 11 shows the distribution of post-thinking token length across various samples. The analysis reveals that all the responses require fewer than 100 tokens to express the final judgment after completing the reasoning process.
400
300
Frequency
200
100
25
50
75
Post-thinking token length
Figure 11: Post-thinking token length distribution of RRM-32B.
Based on this observation, we set fixed post-thinking budget of 100 tokens for all our sequential scaling experiments. This budget is sufficient to accommodate typical response patterns while preventing the model from extending its reasoning during the response phase, which would confound our analysis of thinking horizon effects. By maintaining this consistent response budget across all experiments, we ensure that performance differences can be directly attributed to variations in thinking phase length rather than differences output verbosity. This methodological choice strengthens the validity of our conclusions regarding the impact of extended reasoning model performance.
D.2 Thinking Lengths
To further illustrate the adaptive compute behavior discussed in Section 4.5.2, Table 10 reports the average number of tokens generated before the </think> token across different task subtypes of
30

---

RewardBench benchmark. This analysis confirms that the model dynamically adjusts its reasoning length based on task complexity.
Table 10: Average number of tokens generated before the </think> token across different task subtypes of RewardBench benchmark.
Task Subtype
Average Thinking Length (Tokens)
Chat
421.97
Chat Hard
363.71
Safety
319.62
Reasoning
848.13
E Variance and Stability Analyses
To further assess the robustness of approach against randomness, provide two additional analyses: (i) the stability of batch-FI.O based scalar reward derivation, and (ii) the sensitivity of RewardBench evaluation results to random seeds.
E.1 Stability of Batch-ELO Based Scalar Rewards
We first examine whether the batch-based ELO procedure introduces large variance due random opponent sampling. In a setup with 8 candidate responses, each response played against 4 randomly selected opponents, repeated for 1,000 trials. We report the relative standard deviation (std/mean) across trials for all candidates. As shown in Table 11, the relative deviation is consistently below 1%, indicating that the ELO estimates are stable across batches.
Table 11: Relative standard deviation of batch-ELO scores across 1,000 trials.
Candidate
Relative Std. Dev. (%)
0.197
0.259
3
0.244
4
0.288
5
0.195
6
0.002
7
0.247
8
0.285
E.2 Seed Sensitivity on RewardBench
To quantify the stochasticity in RewardBench evaluation, we run experiments with multiple random seeds. Table 12 reports the minimum, maximum, and standard deviation across runs, in addition the reported average performance. The observed variability is very small with standard deviation < 0.005, confirming that our reported results are robust to randomness evaluation.
Table 12: Performance statistics of RRM RewardBench across random seeds.
Model
Benchmark
Min
Max
Std
Reported
RRM-7B
RewardBench
81.9
83.2
0.005
82.2
RRM-32B
RewardBench
90.7
91.4
0.003
91.2
F Reward Reasoning Pattern Analysis
Following Chen et [13] and Wang al. [63], , we analyze the reasoning patterns of RRM-32B
by statistically measuring the proportion of model responses containing keywords such as 'wait"
31

---

and "alternatively'. We categorize the reasoning patterns into four categories: transition (switching perspectives or strategies), reflection (self-checking or revisiting earlier steps). comparison (evaluating multiple options), and breakdown (decomposing the problem), which is detailed in Table 13.
Table 13: Pattern groups and keywords applied in reasoning pattern analysis.
Pattern Group
Keywords
Transition
alternatively, think differently, another way, another approach, another method, another solution, another point
Reflection
wait, verify, make sure, hold on, think again, Let me check, seems right, seems incorrect
Comparison
more, compared to, comparison, between the two, similarly
Breakdown
break down, break this down
As illustrated in Figure 12, compared to the Deepseek-R R-DDistill--wwn-322 model, RRM-32B demonstrates a greater overall utilization of reasoning patterns when judging the superiority of two answers, particularly in analyzing from different perspectives and conducting in-depth comparisons. In contrast, the Deepseek-B Seppseec-RR-Distlllll DL1-Dstill-Qwen--e quently, suggesting greater tendency to approach problems directly when making judgments, but less inclination to compare the merits of the two answers and engage in self-examination. This distinction in reasoning patterns highlights how our Reward Reasoning via Reinforcement Learning framework shapes the model's ' approach to evaluation tasks.
model employs the breakdown pattern more fre-
R1-distilled
RRM
Percentage of Examples (%)
80
60
40
20
0
Transition
Reflection
Comparison
Breakdown
Diiicck-R--Disiiled----- 32B, RRM-32B exhibits more transition patterns (40.63% vs. 33)73%). reflection patterns (63.28% vs. 52.75%) )) and comparison patterns (89. 84% vs. 85. 35.29%), but fewer direct problem decomposition (8.40% vs. 16.86%).
Figure 12: Reward reasoning pattern analysis results. Compared
Table 14 presents detailed case studies comparing the outputs of Deepseek- --Distill-Qwen and RRM-32B. Each row corresponds to single evaluation example, showing the responses generated by both models. Within the responses to a single example, RRM-32B exhibits a stronger propensity to iteratively apply reasoning patterns for more thorough and in-depth comparisons. The reasoning
32

---

process of Deepseek-RI. -areck-----istlll-Qeeeee
tends to be simple and monolithic- monolithic-involving separate analyses the two answers followed by a summary. Conversely, the thinking process of RRM-32B involves multiple iterations of self-reflection, comparative analysis, and consideration of various perspectives, incorporating more detailed analysis and interpretation of the instructions.
Iterative Reasoning: RRM-32B engages in multiple rounds of self-reflection and comparative analysis, often revising or expanding its reasoning before reaching a final judgment.
Depth of Analysis: RRM-32B considers different perspectives and provides more nuanced planations, whereas Deepseck- BrrasecckRI-Disttie RI-Distill-Qwen-3 typically follows a simple, linear reasoning
path.
Instruction Following: The iterative reasoning process enables RRM-32B better adhere to task instructions and deliver more accurate comparative assessments.
Overall, these examples highlight the qualitative differences in reasoning styles between the two models. The iterative and reflective reasoning of RRM-32B appears to key factor underlying its superior performance across diverse evaluation tasks.
33

---

Table 14: Case study of reasoning patterns of RRM-32B and DeepSeck-Rl-Distillee -wwiilllldd-w--3222- DeepSeck-RR -i-Disiillled--wee3225
RRM-32B
Okay, I need evaluate which assistant provided het- ter response to the given groblem...Lookingg responses:
Okay, need figure which assistant is better for given problem...
two
Looking at Assistant I's response...
Assistant I's code starts by...
Now, looking at Assistant 2's response... Wait, that's 's a problem....
Assistant 2's solution uses a bitwise AND approach.... In conclusion, while Assistant I's code has some issues, Assistant 2's solution is fundamentally incorrect because the bitwise approach doesn't apply powers of three. Therefore, Assistant I's response better, despite its flaws, because attempts handle the problem correctly, if it's perfectly implemented. </think>
But then, what about the code from Assistant I?... Let me test it with some examples...
Wait, perhaps...
But neither assistant provided that solution. So, given the options, which is better?...
Alternatiely, perhaps I'm missing something. Let think again.
\boxed Assistant 1)
Wail, problem requires a solution without loops or recursion...
Alternatively..
Looking back, Assistant I's code is..
Assistant 2's code is is...
Alternatively, perhaps I'm missing something. Let think again. Wait,
Wait, the user's instruction says: "Do NOT say both neither are good." I have choose between Assistant or 2.
Given that, think Assistant 2's code is better because </think>
The better response is...
\boxed Assistant 2)
Alright, let's evaluate problem step by step...
Okay, so need figure out which assistant is better for the given query.
First, look at Assistant I's response...
Now. Assistant 2's response is much simpler...
Assistant starts by
Therefore, Assistant correctly addressed the exponentia- tion but made minor error in the final subtraction... </think>
Assistant 2, on the other hand, Looking at the instruction, the user wrote...
However, the problem is that the instruction might be am- biguous. If .... then Assistant is correct. But if then Assistant is correct. Since the instruction is...So think the intended operation is exponentiation.
The correct evaluation Aboxxdd
Therefore, Assistant I's approach is more accurate likely intended problem. However, if the problem was meant to multiplication, then Assistant 2 correct. But given the notation, exponentiation is more probable. So Assistant I's answer is better in that context.
But wait, the user's instruction is...
Hmm, this is bit confusing. need to decide based on the instruction... ... But if
Given that, Assistant 2's interpretation is more likely cor-
rect...
So, considering instruction's 'ss ambiguity, but leaning towards multiplication, Assistant is better. </think>
The instruction is ambiguous, but Assistant correctly interpreted it as multiplccation, providing the accurate re- sult.
\boxed ( Assistant 2)
34


Reviews:
[{"review_id": "WAeSklf2p5", "reviewer": "Reviewer_DJnD", "summary": "This paper introduced reward reasoning model (RRM), reward model that can generated reasoning about the given generation. They train the reasoning policy via reinforcement fine-tuning (i.e. GRPO) and further boost performance through parallel test-time scaling (e.g., voring over multiple comparisons). The paper evaluates RRMs on multiple ways including, reward benchmark, best-of-N selection, preference alignment, and RL fine-tuning, claiming consistent gains and practical utility.", "questions": "[Q1] Can you show that RRM-based RL fine-tuning improves performance for other base models or datasets beyond DeepSeek-R1-Distill-Qwen-7B and WebInstruct?\n\n[Q2] When using RRM for RL, please compare RRM with other baselines such as Direct Judge or other outcome reward models.\n\n[Q3] Please suggest more explanation of what is the RRMs’ advantage against other concurrent methods.", "limitations": "The paper does not discuss originality and technical novelty against other concurrent works and reasoning-trained models.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "[S1] Effective CoT-based RL fine-tuning: The paper demonstrates that training long chain of thought (CoT)  reasoning through reinforcement learning can help reward models to enhance the performance of reward model.\n\n[W1] Inconsistent improvements: In the main RewardBench table, RRM without voting does not clearly outperform strong baselines such as DirectJudge; observed gains are small and sometimes statistically insignificant.\n\n[W2] Narrow post-training evaluation: The RL experiments use only a single setting (e.g. DeepSeek-R1-Distill-Qwen-7B on WebInstruct with GRPO). Also, the study lacks comparisons against other baselines (i.e. DirectJudge, score-based RM). It makes it hard to judge whether RRM feedback is uniquely beneficial.\n\n[W3] Lack of technical novelty: The suggested concurrent work combines GRPO with additional technical novelty and often reports stronger results against RRM. RRM relies mainly on standard RL training suggested by Deepseek, which may limit its novelty.", "was_revised": true, "final_justification": "Some of my concerns have been resolved after the rebuttal, including points regarding the inconsistent improvements on the main RewardBench, clarification on post-training evaluation, and differences with concurrent works.\n\nHowever, I still find the post-training evaluation insufficient, especially regarding how the reward model is actually used for RLHF and the trade-off between training time and performance. (authors added more experiments, but this is hard to resolve in insufficient time). The authors responded that RLHF is not the only intended usage, which may be controversial. \\\nAdditionally, regarding the issue of training time versus performance trade-off, the authors claim that this is not a key focus of their research; however, I believe that further discussion would have been beneficial.\n\nWhile some concerns have been addressed, others remain unresolved. Thus, I am increasing my score to a borderline reject.", "rebuttal": "Thank you for your comments. \n\n---\n\n>**Weakness 1**: Inconsistent improvements: In the main RewardBench table, RRM without voting does not clearly outperform strong baselines such as DirectJudge; observed gains are small and sometimes statistically insignificant.\n\n**Response 1**: We appreciate the reviewer’s close analysis of the results in the main RewardBench table. Our primary contribution is not to incrementally improve upon a single benchmark, but rather to propose a new paradigm for reward modeling — effectively scales with test-time compute in reward modeling. The reviewer correctly notes that RRM without voting does not always significantly outperform DirectJudge. This observation, in fact, highlights a key strength of our framework. RRM can leverage both **sequential scaling** and **parallel scaling** to improve the performance. \n\nMost importantly, we respectfully disagree with the characterization of our improvements as \"inconsistent.\" When viewed across the full suite of evaluations, RRMs demonstrate remarkably consistent gains. As shown in our paper, RRMs achieve competitive performance on a wide range of benchmarks **beyond RewardBench, including PandaLM Test, MMLU-Pro, MATH, GPQA, and Arena-Hard**. \n\n---\n\n>**Weakness 2**: Narrow post-training evaluation: The RL experiments use only a single setting (e.g. DeepSeek-R1-Distill-Qwen-7B on WebInstruct with GRPO). Also, the study lacks comparisons against other baselines (i.e. DirectJudge, score-based RM). It makes it hard to judge whether RRM feedback is uniquely beneficial.\n\n**Response 2**: Our primary focus is to address the challenge of effectively scale test-time compute for reward modeling. To this end, we conducted extensive evaluations including agreement with human preferences, performance in best-of-N inference across multiple domains, and detailed scaling law analyses, which form our contribution in terms of experiments. \n\nIn response to the reviewer's suggestion, we have conducted **additional RL post-training experiments**.\n\n  - We now include experiments fine-tuning a different, publicly available base model, **Qwen2.5-7B**. \n\n  - We compare the performance of fine-tuning with our **RRM-32B** against a strong, score-based reward model baseline, **Skywork-Reward-Gemma-2-27B-v0.2**. The new results, presented in **Table R1**. \n\n| **Model**         | **Step** | **MMLU-Pro** |\n|---------------|------|----------|\n| Qwen2.5-7B    | 0    | 47.85%   |\n| RLHF-Skywork  | 200  | 49.51%   |\n| RLHF-Skywork  | 400  | 48.26%   |\n| RLHF-Skywork  | 600  | 34.76%   |\n| RLHF-RRM      | 200  | 50.62%   |\n| RLHF-RRM      | 400  | **51.12%**   |\n| RLHF-RRM      | 600  | 50.86%   |\n\nTable R1: Post-training MMLU-Pro results for Qwen2.5-7B using feedback from RRM-32B vs. Skywork-Reward-Gemma-2-27B. RRM yields stable gains, while the score-based model shows degradation, indicating potential reward hacking. \n\nWe also additionally conducted test-time reinforcement learning (TTRL) experiments, where we train **DeepScaleR-1.5B-Preview**[1] on unlabeled queries of AIME24. As shown in **Table R2**, RRMs successfully guided the model to improve its performance on difficult math questions. \n\n|**Model**                                    | **AIME24** |\n|------------------------------------------|--------|\n| DeepScaleR-1.5B-Preview                  | 43.3   |\n| DeepScaleR-1.5B-Preview + RRM-32B TTRL   | **50.0**   |\n\nTable R2: Pass@1 on AIME24 using 5 samples for each question. RRM-guided training improves DeepScaleR-1.5B-Preview performance on challenging math problems without using labeled data. \n\n---\n\n>**Weakness 3**: Lack of technical novelty: The suggested concurrent work combines GRPO with additional technical novelty and often reports stronger results against RRM. RRM relies mainly on standard RL training suggested by Deepseek, which may limit its novelty.\n\n**Response 3**: We would appreciate it if the reviewer could specify the \"suggested concurrent work\", as we were unable to identify any concurrent work in your comment. \n\nRegarding the claim that concurrent work \"often reports stronger results\", the results in **Table R3 and R4** show that RRM consistently outperform concurrent models , consisting of DeepSeek-GRM, RM-R1, and J1[2], on both RewardBench and the (PPE) benchmark. \n\n| **Model**                                           | **Chat**  | **Chat Hard** | **Safety** | **Reasoning** | **Overall** |\n|------------------------------------------------|-------|-----------|--------|-----------|---------|\n| DeepSeek-GRM-27B                         | 94.1  | 78.3      | 88.0   | 83.8      | 86.0    |\n| RM-R1-Distill-Qwen-7B                         | 88.9  | 66.2      | 78.4   | 87.0      | 80.1    |\n| RM-R1-Distill-Qwen-32B                         | 95.3  | 80.3      | 91.1   | 96.8      | 90.9    |\n| **RRM-7B**                      | 87.7  | 70.4      | 80.7   | 90.0      | 82.2    | 72.9           | 71.1           |\n| **RRM-7B (voting@16)**          | 92.1  | 71.5      | 81.3   | 93.8      | 84.8    | 75.9           | 77.8           |\n| **RRM-32B**                     | 94.7  | 81.1      | 90.7   | 98.3      | 91.2    | 78.8           | 79.0           |\n| **RRM-32B (voting@16)**        | **96.1**  | **81.4**      | **91.6**   | **98.6**      | **91.9** \n\nTable R3: Performance of RRM compared to concurrent methods on RewardBench.\n\n| **Model**                             | **MMLU-Pro** | **MATH**  | **GPQA**  | **Overall** |\n|-----------------------------------|----------|--------|--------|---------|\n| DeepSeek-GRM-27B (voting@32)      | 65.5     | 69.4   | 56.0   | 63.6    |\n| DeepSeek-GRM-27B (MetaRM)         | 68.1     | 70.0   | 56.9   | 65.0    |\n| J1-Llama-8B (SC@32)               | 67.5     | 76.6   | 55.7   | 66.7    |\n| J1-Llama-70B (SC@32)              | 79.9     | 88.1   | 66.5   | 78.2    |\n| **RRM-7B**                            | 66.5     | 88.0   | 57.9   | 70.3    |\n| **RRM-7B (voting@5)**                 | 68.3     | 90.5   | 58.3   | 72.4    |\n| **RRM-14B**                           | 71.8     | 88.6   | 60.2   | 73.5    |\n| **RRM-14B (voting@5)**                | 72.4     | 89.7   | 61.1   | 74.4    |\n| **RRM-32B**                           | 80.5     | 94.3   | 67.4   | 80.7    |\n| **RRM-32B (voting@5)**                | **81.3**     | **95.4**   | **68.4**   | **81.7**   |\n\nTable R4: Performance of RRM compared to concurrent methods on PPE.\n\nWe respectfully disagree with the assertion that our method's novelty is limited because it \"relies mainly on standard RL training.\" Our core technical contribution is not the invention of a new RL algorithm (like GRPO) or a new architecture (like the Transformer). Rather, our novelty lies in **the insight and formulation of performing explicit reasoning to adaptively allocate compute for reward modeling**. This new paradigm is what allows RRMs to scale effectively and achieve superior performance on a wide range of benchmarks. Foundational methods are the building blocks of scientific progress; leveraging them to create a novel and effective system should not be a weakness. \n\n---\n\n>**Question 1**:  Can you show that RRM-based RL fine-tuning improves performance for other base models or datasets beyond DeepSeek-R1-Distill-Qwen-7B and WebInstruct? \n\n**Response 4**: We conduct additional evaluations on RL post-training, where we post-train Qwen2.5-7B models with RRM-32B and Skywork-Reward-Gemma-2-27B-v0.2, respectively. We also post-train DeepScaleR-1.5B-Preview with RRM-32B **on the math domain**. Please find the results in **Table R1 and R2**. \n\n---\n\n>**Question 2**: When using RRM for RL, please compare RRM with other baselines such as DirectJudge or other outcome reward models.\n\n**Response 5**: We conduct additional evaluations on RL post-training, where we post-train Qwen2.5-7B models with RRM-32B and Skywork-Reward-Gemma-2-27B-v0.2, respectively. We also post-train DeepScaleR-1.5B-Preview with RRM-32B **on the math domain**. Please find the results in **Table R1 and R2**. It's important to clarify that DirectJudge is not a typical score-based RM but an ablation of RRM as baseline, thus existing methods cannot directly incorporate DirectJudge into RL post-training.\n\n--- \n\n>**Question 3**: Please suggest more explanation of what is the RRMs’ advantage against other concurrent methods.\n\n**Response 6**: \n  - **Simplicity and Efficiency**: RRM provides a unified and straightforward framework. In contrast, methods like DeepSeek-GRM introduce a complicated, multi-stage workflow requiring several intermediate models before producing the final reward model. Our end-to-end approach is significantly simpler to implement and train. \n\n  - **Performance and Scaling**: As shown in **Table R3 and R4**, our RRM outperforms DeepSeek-GRM-27B on both RewardBench and PPE. More critically, RRM demonstrates better scaling properties. It shows that DeepSeek-GRM-27B struggles to benefit from increased test-time compute on difficult reasoning benchmarks like MMLU-Pro and GPQA, whereas RRM achieves consistent and significant gains from parallel scaling on these same tasks. This demonstrates RRM's superior ability to tackle complex evaluation queries. \n\n---\n\n**Reference**\n\n**[1]** DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL\n\n**[2]** J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning"}, {"review_id": "vSaks3Q1J6", "reviewer": "Reviewer_86j6", "summary": "The paper introduces reward reasoning models, an approach to train generative preference models with reasoning capabilities using reinforcement learning. They fine-tune a base reasoning language model using reinforcement learning on a preference dataset with a rule-based reward rewarding correct preference prediction. Reasoning emerges (or further develops from the reasoning capabilities of the base model) as a way to leverage test-time compute and improve preference prediction quality, resulting in larger reward for the model.\n\nAs a secondary contribution, the paper introduces an approach based on in-batch ELO to use pairwise preference models in RL fine-tuning pipelines that expect scalar rewards.\n\nThe authors train multiple reward models (7B, 14B, 32B) using this approach and evaluate them thoroughly, both on direct reward modeling benchmarks and downstream tasks.", "questions": "Please see the numbered weaknesses, focusing on the major ones (**WX**). Most of them are actionable, i.e., can be considered questions, although many of the minor ones (**MX**) do not require an individual response. Actual questions regarding the understanding of the paper are listed as weaknesses in the clarity section. The minor points have little impact on my score.\n\nThe paper is high-quality and significant. I would consider raising my score if the clarity is somewhat improved. Of particular importance are: W2, W3, and W4.", "limitations": "- **W9** The experiments show that there are many types of tasks where reasoning does *not* significantly improve performance (e.g., Table 1 Chat). It still likely has some overhead, even on those tasks. This should be discussed, and ideally the overhead analyzed (is the compute allocation really adaptive, i.e., does it use less on those tasks?)\n- **M10** You should discuss that your \"discriminative to scalar\" scheme only works when there are sufficient candidate responses per prompt available (and mention that this generally the case in common RLHF fine-tuning scenarios).", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "In the following, I discuss strength and weaknesses on the dimensions of quality, clarity, significance, and originality.\nWeaknesses are marked with a **W**, minor weaknesses with an **M**, notes that do not impact my score with **N**, and strengths with an **S**.\nWeaknesses are numbered for reference purposes.\n\n## Quality\n\n- **S** The RRM models are evaluated extensively on a variety of benchmarks.\n- **W1** It would be good to discuss related work on general reasoning and parallel test-time compute strategies (e.g., majority vote, knockout tournament).\n- **W2** The evaluation in Table 2 seems slightly unfair to me, as all the baseline reward models are pure preference models (correct?) focused mostly on style, while your model has verifiable tasks in the training data. It would be very informative to have DirectJudge in this table.\n- **M1** I would expect that the batch-ELO based scalar reward derivation has high variance, depending on the batch. It's nice to see that it works nonetheless. It would be even better to have a dedicated evaluation of this, for example by using a natively scalar reward model from which you can derive a discriminative one, which you can then turn back into a scalar one with your proposed approach -- what is the loss here? Another way to approach this would be to compare DPO with ELO+RLHF. I understand that this is not your primary contribution, however, so this does not impact my score much.\n- **M2** Why do you use voting@16 in Table 1, voting@5 in Table 2?\n- **N** The description of Table 6 in the appendix is slightly confusing. It sounds like it should have numerical results matching Figure 10, but the results differ (RRM-7B ELO instead of Knockout tournament).\n- **N** A brief discussion of what PRM is would be useful in Appx B.1.2.\n- **N** You always remove reasoning traces from your training data. Doesn't that mean that reasoning is then out-of-distribution for the trained RM? I would expect that it may further improve performance if you leave some reasoning traces in.\n- **N** It would be interesting to see if you can use the budget forcing approach introduced by S1 (https://arxiv.org/abs/2501.19393) to further improve performance. (Just a note, does not impact my score.)\n\n## Clarity\n\n- **S** The paper is well-written and easy to follow.\n- **W3** Page 8 mentions a 14B model, but you previously only discuss how you trained the 7B and 32B ones. Table 5 also only has hyperparameters for those.\n- **W4** Your RRMs were fine-tuned solely with RL, no SFT right? How many RL steps?\n- **W5** The \"reward model\" terminology may be confusing, as your model does not directly produce rewards but rather choices. It would be more precise to call it a preference model, choice model, or binary classifier. Discriminative reward model (as you use in the paper) could also work as a \"compromise\". Is there precedent for calling a choice model a reward model? If so, I understand if you want to stick to this terminology but you should make this clearer by (1) discussing early on that you propose a discriminative model and (2) take particular care to avoid confusion everywhere you mention \"reward\" (such as in line 47, Figure 1, line 216). I think this may easily confuse a reader.\n- **W6** I do not understand what is evaluated on Figure 4. Number of pairs used for what?\n- **W7** I do not understand what is measured in Table 4. What is accuracy in the context of Tournament / ELO rating?\n- **W8** I would appreciate an analysis of the overhead generated by the ELO-based reward derivation (if possible without extensive new experiments).\n- **M3** How is the CI computed in Table 3? I am surprised that it is not symmetrical.\n- **M4** The paper (especially the RW) seems to imply that generative RMs are always discriminative. I would have thought those are two distinct axes (generative RMs use the full decoder and a textual response, but may still be either scalar or discriminative). I am not perfectly familiar with established terminology, could you elaborate? LLM-Blender (https://arxiv.org/abs/2306.02561) might be a good reference to discuss discriminative RMs, but that is only one possible example and you are of course not required to cite it.\n- **M5** The RW section does not explicitly put RRM into context of the related works (\"compare and contrast\"). This is largely implied, so this is a very minor point, but it would be better to make it explicit.\n- **M6** How does such a discriminative RM contrast to LLM-as-a-judge? Is there any difference?\n- **M7** The paper skips over some preliminaries: What is post-training? On line 31: Decoding layer of what?\n- **M8** What do you mean by the \"round-robin tournament structure\"?\n- **M9** You report batch size and mini-batch size separately in Table 5. It is not entirely clear to me what this distinction means. Is this about gradient accumulation?\n- **N** The colors in Figure 6 (light orange / pink / dark orange / brown) are hard to distinguish when printed.\n- **N** I think it would be worth pointing out that the ELO-based approach is generally applicable to any discriminative model.\n\n## Significance\n\nWith increasing attention being paid to scaling up inference compute for LLMs, scaling up verification is of increasing significance. This paper is a step into this direction.\n\n\n## Originality\n\nExempting concurrent work, the paper is novel to the best of my knowledge. I am not perfectly familiar with the related work, however. At its core it is a fairly straightforward application of the general \"reasoning recipe\" to reward modeling, tempering originality slightly, but it is well-executed and evaluated and therefore a valuable contribution to the literature.", "was_revised": true, "final_justification": "The suggested reasoning approach is very promising for further improving language model validation and training. The writing and the evaluation are good. The authors have addressed all the points I raised.", "rebuttal": "Thank you for your feedback, particularly for recognizing the \"high quality and significance\" of our paper and for considering raising the score. We deeply appreciate your valuable comments regarding clarity, and we are committed to refining the paper. \n\nDue to space constraints, we prioritize addressing the main **Weaknesses** below. For points categorized as **Minor Weaknesses**, we will respond M1-M8 under **the reviewer 4QQR** and M9-M10 under **the reviewer mL9a**.\n\nRegarding the **Notes**, we appreciate your insightful notes, which will improve the quality and clarity of our paper. Due to length constraints, we cannot provide individual responses, but we will revise the paper accordingly. \n\n---\n\n>**W1**: It would be good to discuss related work on general reasoning and parallel test-time compute strategies (e.g., majority vote, knockout tournament). \n\n**Response 1**: Thank you for the suggestion. We cited parallel test-time compute strategies in Related Work (Line 339-341). We will expand the discussion in related work in the next version.\n\n---\n\n>**W2**: The evaluation in Table 2 seems slightly unfair to me, as all the baseline reward models are pure preference models (correct?) focused mostly on style, while your model has verifiable tasks in the training data. It would be very informative to have DirectJudge in this table.\n\n**Response 2**: The Preference Proxy Evaluations (PPE) benchmark used in Table 2 is a standard protocol for evaluating reward models,  and is a widely accepted evaluation setup in the reward modeling literature.  \n\nFurthermore, the baseline reward models in Table 2 are not purely style-focused. They are trained on preference data that also includes **verifiable reasoning tasks**. For example, the training datasets of **Skywork-Reward, DeepSeek-GRM, and J1**[1] all  contains verifiable reasoning tasks.  \n\nWe understand the reviewer’s concern about potential differences in training data and evaluation setup. To address this, we additionally follow the protocol from the PPE paper[2] and evaluate reward models on **binary preference classification**, as shown in Table R1. This evaluation includes a broader set of baselines under the same setup. Due to response length limitation, please refer to Table 2 of the PPE’s paper[2] for more baselines. \n\n| **Model**                             | **MMLU-Pro** | **MATH**  | **GPQA**  | **Overall** |\n|-----------------------------------|----------|--------|--------|---------|\n| Skywork-Reward-Gemma-2-27B        | 55.0     | 46.2   | 44.7   | 48.6    |\n| Gemma-2-27B                       | 66.2     | 66.4   | 51.9   | 61.5    |\n| DeepSeek-GRM-27B (voting@32)      | 65.5     | 69.4   | 56.0   | 63.6    |\n| DeepSeek-GRM-27B (MetaRM)         | 68.1     | 70.0   | 56.9   | 65.0    |\n| Llama-3.1-8B-Instruct             | 56.3     | 62.9   | 51.4   | 56.9    |\n| Llama-3.1-70B-Instruct            | 72.1     | 73.1   | 61.2   | 68.8    |\n| J1-Llama-8B (SC@32)               | 67.5     | 76.6   | 55.7   | 66.7    |\n| J1-Llama-70B (SC@32)              | 79.9     | 88.1   | 66.5   | 78.2    |\n| **RRM-7B**                            | 66.5     | 88.0   | 57.9   | 70.3    |\n| **RRM-7B (voting@5)**                 | 68.3     | 90.5   | 58.3   | 72.4    |\n| **RRM-14B**                           | 71.8     | 88.6   | 60.2   | 73.5    |\n| **RRM-14B (voting@5)**                | 72.4     | 89.7   | 61.1   | 74.4    |\n| **RRM-32B**                           | 80.5     | 94.3   | 67.4   | 80.7    |\n| **RRM-32B (voting@5)**                | **81.3**     | **95.4**   | **68.4**   | **81.7**    |\n\nTable R1: Evaluation results on binary preference classification following the protocol from Frick et al [2]. We report accuracy over five pairs of conflicted responses. \n\n---\n\n>**W3**: Page 8 mentions a 14B model, but you previously only discuss how you trained the 7B and 32B ones. Table 5 also only has hyperparameters for those. \n\n**Response 3**: Initially, we focused on training **7B and 32B** reward models to align with the model sizes of prior work. However, to better illustrate the **scaling trends** in reward model performance, we later added a 14B model as an intermediate point. Table R.1 provides evaluation results of RRM-14B. \n\nWe appreciate your careful review and will provide the hyperparameters and additional evaluation results for the 14B model in the appendix. \n\n---\n\n>**W4**: Your RRMs were fine-tuned solely with RL, no SFT right? How many RL steps?\n\n**Response 4**: Yes. RRMs were fine-tuned solely with RL, with the training steps being 720 and 440 for the 7B model and the 32B model respectively.\n\n---\n\n>**W5**: The \"reward model\" terminology may be confusing, as your model does not directly produce rewards but rather choices. It would be more precise to call it a preference model, choice model, or binary classifier. Discriminative reward model (as you use in the paper) could also work as a \"compromise\". Is there precedent for calling a choice model a reward model? If so, I understand if you want to stick to this terminology but you should make this clearer by (1) discussing early on that you propose a discriminative model and (2) take particular care to avoid confusion everywhere you mention \"reward\" (such as in line 47, Figure 1, line 216). I think this may easily confuse a reader.\n\n**Response 5**: While our model produces pairwise choices rather than scalar rewards, the terminology of “reward model” for such discriminative models has been adopted in prior work. For example, GenRM[3]  uses this term and illustrates it in Figure 8 in prompt structure. Similarly, PairJudge RM[4] presents a similar setup in their prompt table (Table 4), yet still refers to the model as a reward model. \n\nWe agree this terminology may be confusing for readers unfamiliar with this convention. We will revise the terminology accordingly to enhance clarity. \n\n---\n\n>**W6**: I do not understand what is evaluated on Figure 4. Number of pairs used for what? \n\n**Response 6**: For each MATH problem, we sample 8 candidate responses and select several pairs from them for pairwise comparison. These comparison results are then used for ELO rating to estimate a ranking over the 8 candidates. The top-ranked response is selected as the final output. The y-axis in Figure 4 represents the **accuracy** of this selected output. The number of pairs reflects the test-time computational budget for comparisons. Comparing all $\\binom{8}{2} = 28$ possible pairs represents a full round-robin comparison. Additionally, the rightmost point in Figure 4 signifies 140 pairwise comparisons (28 pairs each compared 5 times by voting). \n\n---\n\n>**W7**: I do not understand what is measured in Table 4. What is accuracy in the context of Tournament / ELO rating? \n\n**Response 7**: Both Tournament and ELO rating can be used to select the best candidate among a set of responses, based on the RRM's pairwise comparisons. The accuracy reported in Table 4 refers to the **correctness of the top-selected response** under each strategy. \n\n---\n\n>**W8**: I would appreciate an analysis of the overhead generated by the ELO-based reward derivation (if possible without extensive new experiments).\n\n**Response 8**: The ELO-based reward derivation introduces additional computational overhead primarily due to the pairwise comparisons required to estimate rankings among candidate responses. For a batch size of $N\\$ candidates, the number of pairwise matches is $O(N \\times k)\\$, where $k\\$ is a hyperparameter representing the number of competitors each candidate is compared against (e.g., 4 in our setup). In practice, we observed that this overhead remains manageable and does not dominate the overall training time, as the pairwise scoring is efficiently parallelized.\n\n---\n\n>**W9**: The experiments show that there are many types of tasks where reasoning does not significantly improve performance (e.g., Table 1 Chat). It still likely has some overhead, even on those tasks. This should be discussed, and ideally the overhead analyzed (is the compute allocation really adaptive, i.e., does it use less on those tasks?) \n\n**Response 9**: Thank you for the valuable suggestion. To analyze the potential overhead, we computed the average thinking length across different task subtypes in RewardBench, as **Table R2** shows. We observe that tasks like Chat have relatively short responses, indicating less computational overhead, while reasoning tasks produce much longer responses. This suggests that the compute allocation is indeed adaptive.\n\n| **Task Subtype** | **Average Thinking Length (Tokens)** |\n|--------------|----------------------------------|\n| Chat         | 421.97                           |\n| Chat Hard    | 363.71                           |\n| Safety       | 319.62                           |\n| Reasoning    | 848.13                           |\n\nTable R2: Average number of tokens before `</think>` across task subtypes.\n\n---\n\n**References**\n\n**[1]** J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning \n\n**[2]** How to evaluate reward models for RLHF \n\n**[3]** Generative Reward Models \n\n**[4]** PairJudge RM: Perform Best-of-N Sampling with Knockout Tournament"}, {"review_id": "VMz4Y7qqMX", "reviewer": "Reviewer_4QQR", "summary": "This paper introduces Reward Reasoning Models (RRMs), which are designed to enhance reward model performance by incorporating explicit reasoning processes. The authors argue that existing reward models lack adaptability in allocating computational resources, particularly for complex queries that require nuanced analysis or multi-step reasoning. RRMs address this challenge by framing reward modeling as a reasoning task, where the model generates a chain-of-thought reasoning process before producing the final reward. This approach allows RRMs to adaptively allocate additional computational resources when evaluating complex tasks. The paper also presents a reinforcement learning framework that enables RRMs to self-evolve their reward reasoning capabilities without requiring explicit reasoning traces as training data. Experimental results demonstrate that RRMs outperform strong baselines across multiple domains, including reasoning, general knowledge, safety, and alignment with human preference. The authors further show that RRMs can effectively utilize test-time compute to improve reward accuracy and adaptively allocate computational resources to practical application scenarios.", "questions": "1. Can RRM effectively guide RL with unlabeled, harder mathematical and coding data, such as tasks with difficulty levels similar to AIME or LiveCodeBench?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The authors conduct a comprehensive comparison between RRM and various other reward models in their experiments. The superior performance of RRM compared to other reward models highlights the effectiveness of the long reasoning reward model.\n\n2. The work includes experiments on large reward models, such as RRM-32B, which provides more convincing conclusions about reward model training at scale.\n\n3. The authors present a relatively thorough analysis of their proposed RRM, including BoN comparisons, RL with unlabeled data, reward model-guided DPO, parallel and sequential scaling analysis, and other detailed evaluations. These in-depth analyses further underscore the authors' significant contributions to the field of reward modeling.\n\n---\n\n**Weaknesses**\n\n1. The paper lacks a deep analysis in certain model comparisons. For instance, Table 1 shows that RRM-32B outperforms Skywork-Reward-Gemma-2-27B-v0.2 on the PandaLM Test but underperforms on RewardBench. What explains these contradictory results? Similarly, in the BoN experiments in Table 2, Skywork-Reward-Gemma-2-27B-v0.2 shows significantly worse performance compared to other reward models, yet it performs strongly on RewardBench. What causes these discrepancies?\n\n2. While the authors provide interesting further analyses, such as RL with unlabeled data, which demonstrate positive effects and training stability, they do not include the performance of other reward models in these scenarios. Would other reward models outperform RRM in tasks like RL with unlabeled data, DPO, or parallel and sequential scaling? A more detailed comparison with baselines in these areas would strengthen the work.", "was_revised": true, "final_justification": "While I am not very familiar with reward model literature, I maintain my current rating as the authors' responses have addressed my primary concerns. However, given that the reward model serves as a critical component, it can significantly impact RL training across multiple dimensions—including different model architectures, varying model sizes, and diverse domains.\n\nComprehensive evaluation would require extensive experiments to ensure that the proposed approach does not compromise the model's underlying potential. Although I recognize this as technically solid work, I believe the contribution would benefit from more thorough analysis across these various settings to fully validate the method's robustness and generalizability.", "rebuttal": "We thank the reviewer for the valuable comments. \n\n---\n\n> **Weakness 1**: The paper lacks a deep analysis in certain model comparisons. For instance, Table 1 shows that RRM-32B outperforms Skywork-Reward-Gemma-2-27B-v0.2 on the PandaLM Test but underperforms on RewardBench. What explains these contradictory results? Similarly, in the BoN experiments in Table 2, Skywork-Reward-Gemma-2-27B-v0.2 shows significantly worse performance compared to other reward models, yet it performs strongly on RewardBench. What causes these discrepancies? \n\n**Response 1**: Thank you for the insightful question. These \"contradictory\" results are common and expected in reward model evaluation. They arise because benchmarks vary in their data collection, domain coverage, and the specific human preferences they capture. These variations highlight the importance of evaluating reward models across diverse benchmarks. Interestingly, similar discrepancies have also been observed in concurrent work. For instance, J1[1] underperforms the Skywork reward model on RewardBench, but achieves significantly better results on the PPE benchmark. \n\n---\n\n> **Weakness 2**: While the authors provide interesting further analyses, such as RL with unlabeled data, which demonstrate positive effects and training stability, they do not include the performance of other reward models in these scenarios. Would other reward models outperform RRM in tasks like RL with unlabeled data, DPO, or parallel and sequential scaling? A more detailed comparison with baselines in these areas would strengthen the work. \n\n**Response 2**: Thank you for recognizing our “interesting further analyses” regarding RL with unlabeled data and other aspects. We conducted additional experiments on RL post-training, where we fine-tuned Qwen2.5-7B models using feedback from both RRM-32B and Skywork-Reward-Gemma-2-27B-v0.2, respectively. The comparative results are detailed in **Table R1**. While feedback from both types of reward models improved performance on MMLU-Pro, we observed that the score-based reward model's performance quickly degraded through training, indicating a potential reward hacking issue. \n\n| **Model**         | **Step** | **MMLU-Pro** |\n|---------------|------|----------|\n| Qwen2.5-7B    | 0    | 47.85%   |\n| RLHF-Skywork  | 200  | 49.51%   |\n| RLHF-Skywork  | 400  | 48.26%   |\n| RLHF-Skywork  | 600  | 34.76%   |\n| RLHF-RRM      | 200  | 50.62%   |\n| RLHF-RRM      | 400  | **51.12%**   |\n| RLHF-RRM      | 600  | 50.86%   |\n\nTable R1: Post-training MMLU-Pro results for Qwen2.5-7B using feedback from RRM-32B vs. Skywork-Reward-Gemma-2-27B. RRM yields stable gains, while the score-based model shows degradation, indicating potential reward hacking.\n\n---\n\n> **Question 1**: Can RRM effectively guide RL with unlabeled, harder mathematical and coding data, such as tasks with difficulty levels similar to AIME or LiveCodeBench? \n\n**Response 3**: Yes. We have conducted test-time reinforcement learning (TTRL) experiments, where we train DeepScaleR-1.5B-Preview[2] on unlabeled queries of AIME24. As shown in **Table R2**, RRMs successfully guided the model to improve its performance on difficult math questions. \n\n|**Model**                                    | **AIME24** |\n|------------------------------------------|--------|\n| DeepScaleR-1.5B-Preview                  | 43.3   |\n| DeepScaleR-1.5B-Preview + RRM-32B TTRL   | **50.0**   |\n\nTable R2: Pass@1 on AIME24 using 5 samples for each question. RRM-guided training improves DeepScaleR-1.5B-Preview performance on challenging math problems without using labeled data. \n\n---\n\n**References** \n\n**[1]** J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning \n\n**[2]** DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL\n\n---\n---\n\nDue to space limitations, we were unable to address all of **Reviewer 86j6’s** Weaknesses and Minor Weaknesses directly beneath his comments. Therefore, we provide responses to some of these points below.\n\n---\n\n>**M1**: I would expect that the batch-ELO based scalar reward derivation has high variance, depending on the batch. It's nice to see that it works nonetheless. It would be even better to have a dedicated evaluation of this, for example by using a natively scalar reward model from which you can derive a discriminative one, which you can then turn back into a scalar one with your proposed approach -- what is the loss here? Another way to approach this would be to compare DPO with ELO+RLHF. I understand that this is not your primary contribution, however, so this does not impact my score much. \n\n**Response 10**: We appreciate the reviewer’s suggestions. In our setup, the ELO-based scalar scores are derived by ranking the model’s own generated responses, which are then used to train a policy via RL. To address the reviewer’s concern about variance, we conducted a simple empirical test: in a setting with 8 candidate responses, we let each candidate play against 4 randomly selected opponents, and repeated this process 1,000 times. We then computed the standard deviation of the resulting ELO scores across trials. The relative standard deviation (std/mean) for all responses was below 1%, indicating that the batch-based ELO estimates are in fact quite stable. The results are summarized in the following **Table R3**. \n\n| **Candidate** | **Relative Standard Deviation** |\n|-----------|-----------------------------|\n| 1         | 0.197%                      |\n| 2         | 0.259%                      |\n| 3         | 0.244%                      |\n| 4         | 0.288%                      |\n| 5         | 0.195%                      |\n| 6         | 0.002%                      |\n| 7         | 0.247%                      |\n| 8         | 0.285%                      |\n\nTable R3: The relative standard deviation of each candidates. \n\n---\n\n>**M2**: Why do you use voting@16 in Table 1, voting@5 in Table 2? \n\n**Response 11**: We've verified that our core conclusions remain consistent whether using voting@5 or voting@16. In cases with 32 candidates for each question in Table 2, we use voting@5 to manage computational load effectively while still demonstrating the benefits of parallel scaling, as our primary aim was to show performance improvement rather than tune voting size for each specific setup. \n\n---\n\n>**M3**: How is the CI computed in Table 3? I am surprised that it is not symmetrical.\n\n**Response 12**: The confidence intervals in Table 3 are computed by following the Arena-Hard protocol, using non-parametric bootstrap resampling, which does not assume symmetry in the underlying accuracy distribution. Thus, the CI bounds can be asymmetric. For reference, the official Arena leaderboard also reports asymmetric confidence intervals.\n\n---\n\n>**M4**: The paper (especially the RW) seems to imply that generative RMs are always discriminative. I would have thought those are two distinct axes (generative RMs use the full decoder and a textual response, but may still be either scalar or discriminative). I am not perfectly familiar with established terminology, could you elaborate? LLM-Blender might be a good reference to discuss discriminative RMs, but that is only one possible example and you are of course not required to cite it. \n\n**Response 13**:  As mentioned in Related Work, **reward formulation (scalar vs. generative) and scoring scheme (individual vs. discriminative)** are two distinct axes. We clarify this distinction in lines 312–314: ‘Reward models can be characterized along two dimensions: reward formulation and scoring scheme’ and in lines 327–328: ‘with flexibility for both single-instance evaluation and multi-response comparison.’ So **generative reward models are not necessarily discriminative** — they can operate in an individual fashion by taking a single input (e.g., a prompt–response pair) and producing a justification.\nBesides, note that we have already cited **LLM-Blender** in the paper, and we agree that it is indeed a relevant example. \n\n---\n\n>**M5**: The RW section does not explicitly put RRM into context of the related works (\"compare and contrast\"). This is largely implied, so this is a very minor point, but it would be better to make it explicit.\n\n**Response 14**: Thanks for your suggestion. We will revise the section to more clearly highlight the distinctions and connections between our approach and existing reward models.\n\n---\n\n>**M6**: How does such a discriminative RM contrast to LLM-as-a-judge? Is there any difference?\n\n**Response 15**: “LLM-as-a-judge” usually refers to using a general-purpose LLM to judge responses without explicit training for reward modeling. It typically works in a zero-shot or few-shot manner via prompting. Differently, discriminative RMs focus on reward modeling, and are often more accurate and stable on specific domains. \n\n---\n\n>**M7**: The paper skips over some preliminaries: What is post-training? On line 31: Decoding layer of what?\n\n**Response 16**: Post-training refers to a training stage after pre-training. In our paper, post-training can refer to either training RRMs or training policy models with RRM feedback. \n\n\"Decoding layer\" refers to the language modeling head (i.e., the final projection layer) of LM. Scalar reward models typically replace the head with a linear layer that outputs a scalar reward instead of token logits. We will revise the wording to make this clearer. \n\n---\n\n>**M8**: What do you mean by the \"round-robin tournament structure\"?\n\n**Response 17**: By \"round-robin tournament structure\", we mean that all response candidates are compared against each other in all possible pairs. For $N$ candidates, this results in $\\binom{N}{2}$ pairwise comparisons."}, {"review_id": "8IS8iHiGPz", "reviewer": "Reviewer_mL9a", "summary": "The paper proposes reward reasoning models that can provide rewards to train models for better performance.  \nThe LLM gets two candidate responses and it reasons about them (reasoning trace) and finally picks one of them. This is the \"reasoning\" in reward models. Furthermore, they consider training the model with RL to generate such a reward reasoning sequence. A major benefit of this is that even though RL is performed with binary rewards, the resulting reward model is able to generate non-binary intermediate feedback. They do extensive ablation behind the reasoning reward models towards alignment with human preference, best of N inference based on the reward model, fine-tuning of models with RRM, scaling compute budget during test-time, and effect of scaling RRM during training.", "questions": "What does it mean by adaptive compute? If it refers to increasing the hard budget of thinking (c.f. line 273), then it would suggest not using adaptive compute. The more or less use of the computing budget is not the result of the reasoning reward models. If there is a contribution of dynamically limiting the computation of RRM, then the adaptive compute terminology may be used.\n\nWhy didn't you consider any other models apart from Qwen?", "limitations": "No, limitations not addressed. I think authors should compare the computational complexity of the proposed reward model with other reward models, e.g. scaler. Since RRMs involve generating chain-of-thought traces and aggregating over multiple samples (e.g., voting @32), the inference cost is significantly higher. \nAlso, the paper never mentions any other limitations. I encourage the authors to be upfront and mention all of them during the rebuttal.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The proposed idea of using reasoning traces in reward models makes sense, and itis  backed by good empirical performance\n- Extensive ablation study behind the proposed idea. Considered multiple benchmarks and models.\n\nWeakness:\n- Lack of statistical error bars (CI). I think CI is important to draw conclusions from the empirical analysis, specifically in results like Table 1 where the performance of the proposed approach is close to that of the other models. LLMs are quite stochastic, and I am afraid to draw conclusions with the best case if the performances are close\n- No discussion regarding the computational cost of RRMs compared to standard scalar reward models\n- No discussion on limitations of the method", "was_revised": true, "final_justification": "I would like to keep my score of borderline accept. The idea of reasoning traces in reward models makes sense, and it is backed by good empirical performance. However, I would like to keep the borderline score since the paper does not discuss any limitations of the method throughout the paper. I have to explicitly ask to acknowledge the increase in computation overhead. Given this, I doubt the credibility of the results and hence kept it borderline.", "rebuttal": "We thank the reviewer for the valuable comments. \n\n---\n\n>**Weakness 1**: Lack of statistical error bars (CI). I think CI is important to draw conclusions from the empirical analysis, specifically in results like Table 1 where the performance of the proposed approach is close to that of the other models. LLMs are quite stochastic, and I am afraid to draw conclusions with the best case if the performances are close. \n\n**Response 1**: Thank you for raising this important point. We follow the **standard evaluation protocol of RewardBench**, whose leaderboard results also do not report confidence intervals (CI).  \n\nWe understand the reviewer’s concern about the stochasticity of large language models and the risk of over-interpreting close performance numbers. To address this, we conducted additional experiments on RewardBench by varying random seeds. The results are shown in **Table R1**. \n\n| **Model**     | **Bench**        | **Min**   | **Max**   | **Std**    | **Reported** |\n|-----------|--------------|-------|-------|--------|----------|\n| RRM-7B    | RewardBench  | 81.9  | 83.2  | 0.005  | 82.2     |\n| RRM-32B   | RewardBench  | 90.7  | 91.4  | 0.003  | 91.2     |\n\nTable R1: Performance statistics of RRM models on RewardBench.\n\n---\n\n>**Weakness 2**: No discussion regarding the computational cost of RRMs compared to standard scalar reward models. \n\n**Response 2**: Thank your for the thoughtful question. Unlike scalar reward models with a fixed computational budget, RRMs offer the substantial advantage of adaptively utilizing additional test-time compute. Experimental results in Figure 4 and Figure 5 empirically confirm this benefit. \n\nMoreover, the increased computational cost pertains to the training phase within RLHF, which typically involves a relatively small number of steps compared to the extensive pre-training of the language model itself. Importantly, after RLHF, RRMs do not involve the inference phase of the policy model. Therefore, the computational overhead is entirely affordable. \n\n---\n\n>**Question 1**: What does it mean by adaptive compute? If it refers to increasing the hard budget of thinking (c.f. line 273), then it would suggest not using adaptive compute. The more or less use of the computing budget is not the result of the reasoning reward models. If there is a contribution of dynamically limiting the computation of RRM, then the adaptive compute terminology may be used. \n\n**Response 3**: RRMs adopt chain-of-thought reasoning process, explicitly signaled by the model's active decision to terminate its thought process using a \\</think\\> tag. Our experiments also confirm this adaptive-compute behavior by comparing the reasoning length across different domains. Specifically, we observed that the responses are shorter in the Chat domain, whereas they are considerably longer in the Reasoning domain, demonstrating dynamic allocation of computational resources based on task complexity. This reflects an emergent adaptive behavior, though it is not explicitly designed. \n\nWe appreciate the reviewer’s suggestion to clarify this terminology and will revise the phrasing in the next version. \n\n---\n\n>**Question 2**: Why didn't you consider any other models apart from Qwen? \n\n**Response 4**: Our primary focus is to address the challenge of effectively scale test-time compute for reward modeling. Given our limited GPU resources, we prioritize exploring test-time scaling behaviors and conducting experiments with large language models (e.g., 32B parameters) over repeating experiments with different base models. Since our methods are orthogonal to the base model, we plan to migrate our methods to other base models in future work. We believe the current experimental setups are sufficient to support our methods.\n\n---\n---\n\nDue to space limitations, we were unable to address all of **Reviewer 86j6’s** Weaknesses and Minor Weaknesses directly beneath his comments. Therefore, we provide responses to some of these points below.\n\n---\n\n>**M9**: You report batch size and mini-batch size separately in Table 5. It is not entirely clear to me what this distinction means. Is this about gradient accumulation?\n\n**Response 18**: The distinction between batch size and mini-batch size in Table 5 follows standard GRPO training conventions and is not directly about gradient accumulation. Specifically: \n\n  - Batch size refers to the total number of sample groups collected per GRPO update round. \n\n  - Mini-batch size is the number of sample groups used in each GRPO optimization step (i.e., the batch size for backpropagation). Mini-batch size pertains to the samples per GRPO optimization step, not gradient accumulation. \n\n---\n\n>**M10**: You should discuss that your \"discriminative to scalar\" scheme only works when there are sufficient candidate responses per prompt available (and mention that this generally the case in common RLHF fine-tuning scenarios).\n\n**Response 19**: Obtaining candidate responses is easy since we can always prompt LLMs to get candidates as anchors to compute Elo ratings. Besides, we would like to emphasize that such scheme enables RRMs to generate adaptive reward scores. This adaptiveness comes from their ability to utilize configurable reference candidates. Conversely, score-based reward models, by assigning absolute scores, inherently rely on an implicit and unchangeable criterion, which makes them more vulnerable to hack."}]

Author rebuttal:

Meta-review: This paper introduces Reward Reasoning Models (RRMs), which are trained to generate an explicit chain-of-thought process before producing a final preference score. The primary contribution is a new paradigm for reward modeling where test-time compute can be adaptively scaled to improve accuracy on complex queries, leading to superior performance on reward benchmarks and in downstream applications like reinforcement learning from human feedback (RLHF).

Most reviewers are in agreement, they highlight this work as a novel paradigm for reward modeling, and appreciate comprehensive evaluation across numerous benchmarks and tasks, and strong empirical performance at scale. The initial reviews raised several points for clarification, including the need for stronger baselines in the RLHF experiments, a lack of statistical robustness analysis, and an analysis of the computational overhead. The reviewers were satisfied with the authors' response, which provided significant new experimental results addressing these key points.

For the camera-ready version, the authors are expected to integrate the new experimental results and detailed clarifications from their rebuttal into the main body of the paper. This includes: (1) the new RLHF experiments with a strong score-based reward model baseline, (2) the statistical robustness analysis from multi-seed runs, (3) the expanded evaluation on the PPE benchmark, and (4) the analysis of the additional computational cost of the RRM-based training pipeline.
Decision: Accept (poster)