RM-R1: Reward Modeling as Reasoning
Authors: Xiusi Chen
Gaotang Li
Ziqi Wang
Bowen Jin
Cheng Qian
Yu Wang
Hongru WANG
Yu Zhang
Denghui Zhang
Tong Zhang
Hanghang Tong
Heng Ji
Venue: ICLR 2026 Poster (2026)
Keywords: Reward Model
Reasoning
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: Reward model with thinking improves the reward accuracy.

Abstract: Reward modeling is essential for aligning large language models with human preferences through reinforcement learning. To provide accurate reward signals, a reward model (RM) should stimulate deep thinking and conduct interpretable reasoning before assigning a score or a judgment. Inspired by recent advances of long chain-of-thought on reasoning-intensive tasks, we hypothesize and validate that integrating reasoning into reward modeling significantly enhances RM's interpretability and performance. We introduce a new class of generative reward models, Reasoning Reward Models (ReasRMs), which formulate reward modeling as a reasoning task. We propose a reasoning-oriented training pipeline and train a family of ReasRMs, RM-R1. RM-R1 features a chain-of-rubrics (CoR) mechanism -- self-generating sample-level chat rubrics or math/code solutions, and evaluating candidate responses against them. The training of RM-R1 consists of two key stages: (1) distillation of high-quality reasoning chains and (2) reinforcement learning with verifiable rewards. Empirically, our models achieve superior performance across three reward model benchmarks on average, outperforming much larger open-weight models (e.g., INF-ORM-Llama3.1-70B) and proprietary ones (e.g., GPT-4o) by up to 4.9%. Beyond final performance, we perform thorough analyses to understand the key ingredients of successful ReasRM training.

Published as a conference paper at ICLR 2026
RM-R1: REWARD MODELING AS REASONING
Xiusi Chen'; Gaotang Li Ziqi Wang", Bowen Jin', Cheng Qian', Yu Wang", Hongru Wang', Yu Zhang", Denghui Zhang', Tong Zhang', Hanghang Tong', Heng Ji University of Illinois at Urbana- trbana-Champaign
University California, San Diego
Teexs A&M University
*Stevens Institute of Technology
(xiusic, gaotang3, htong, hengjil@illinois..
ABSTRACT
Reward modeling is essential for aligning large language models with human preferences through reinforcement learning. To provide accurate reward signals, a reward model (RM) should stimulate deep thinking and conduct interpretable reasoning before assigning score or judgment. Inspired by recent advances of long chain-of-thought on cessooiing-ittnsiinn tasks, we hypothesize and validate that integrating reasoning into reward modeling significantly enhances RM's inter- pretability and performance. We introduce a new class of generative reward models, Reasoning Reward Models (REASRMS), which formulate reward modeling as a reasoning task. We propose a reasoning-oriented training pipeline and train a fam- ily of REASRMs, RM-R1. RM-R1 features a chain-of-rubrics (CoR) mechanism self- seff-generating sample-level chat rubrics or math/code solutions, and evaluating candidate responses against them. The training of RM-RI consists of two key stages: (1) distillation of high-quality reasoning chains and (2) reinforcement learn- ing with verifiable rewards. Empirically, our models achieve superior performance across three reward model benchmarks average, outperforming much larger open-weight models (e.g., INF L. ama 1-70BB and proprietary ones (e.g., GPT-4o) by up to 4.9%. Beyond final performance, we perform thorough analyses to understand the key ingredients of successful REASRM training"
INTRODUCTION
Reward models (RMs) play critical role in large language model (LLM) post-training, particularly in reinforcement learning with human feedback (RLHF) (Bai et al. 2022 Ouyang et al. [022). where they serve scalable proxies for human evaluators. Existing research reward modeling can be broadly classified into two categories: (1) sealar- based reward models (ScalarRM) (Liu et al.) 2024) and (2) generative reward models (GenRM) (Zhang et al. 2025). Scalar-based approaches frame reward modeling as a classification problem, typically training a sequence classifier on top of a language model. In contrast, generative approaches retain the original language model decoding head and leverage the model's generative abilities to produce free-form pairwise judgments. While scalar- based methods are direct and often effective, they are opaque, offering no intermediate reasoning steps to justify the model's 's decisions. This lack of transparency may limit their capacity to handle intensive preference tasks. On the other hand, although generative methods provide greater transparency, their reasoning is often superficial and unhelpful for reliable judgment, leading to suboptimal performance (Chen et al. 2025 Liu et al. 2025b).
more challenging, reasoning-intensive
In real-world decision-making scenarios, accurate and grounded reward modeling often requires jointly conducting reasoning and reward assignment. This is because preference judgments inherently involve multifaceted cognitive considerations, such as inferring judge's latent evaluation crite- ria (Baker et al. 2009), navigating trade-off's among multiple criteria (Montibeller & Franco, 2010).
"Equal contribution.
Code, data, and the model publicly available https:://giii http:s////gttubu.oooe www.ttttbbb..cb...e www.c//////.// R1 111-UUCC//A RM-RI

---

Published as a conference paper at ICLR 2026
feel like I'm not good enough for my job. No matzer how hard I try, keep making mistakes. Should I just quit?
Chosen Response (Supportive, honest, and thoughdull::
I'm really sorry you're foaling way. It's completely normal to doubt yourself, especially when you're striving well. Mistakes don't mean failure-they" often part of learning. Instead of quitting, maybe it's worth reflecting whether you're being too hard yourself, Would you consider talking to mentor or supervisor about how you're feeling? You might surprised by their perspective and support.
Rejected Response (Well-formed but subdy toxic):
-
If you keep messing maybe really isn't the right job for you Not everyone is cut out for everything. Quitting isn't bad thing it
be the step step toward finding something you're actually good
Instruct Model
Model with Long Reasoning
fcrubriesPl. Empathy Emotioral Valdation Psychological Safecy Nor-Hrmm Contructive, Actionable Guidance
<answer>
IV. Encouragement of Self-EEficacyySinbriii
Second
Il The first response validaces the user's emations and encourages constructive self-reflection, offering actionable!
message.
11 and supportive guidance without judgment, The second response assumes the user's fallure and may reinforce negative beliefs, which is harmful sensitive <eeva <answer> first
</answer>
Figure 1: The off aff-the-shelff instruct model overfits to patterns in supervised data, failing to evaluate the emotional harm and lack of nuance in the rejected response. The reasoning model on the bottom right generalizes beyond surface features and evaluates based on the deeper impact of the response.
and simulating potential consequences (Van Hoeck et al., 2015), all of which necessitate extensive reasoning. Our example in Diluusttaess such an example, where a correct preference judgment requires accurate perception of the question, understanding of the corresponding evaluation rubrics with convincing arguments closely mirroring how humans approach grading tasks. Motivated by these observations, we explore the following central question:
Can cast reward modeling as reasoning task?
In this work, we unleash the reasoning potential of RMs and propose a new class of models: Reasoning Reward Models (REASRMs). Different from standard GenRMs, REASRMS emphasize leveraging long and coherent reasoning chains during the judging process to enhance the model's ability to assess and distinguish complex outputs accurately. We validate that integrating long reasoning chains during the judging process significantly enhances downstream reward model performance. We explore several strategies for adapting Instruction-tuned language models into logically coherent REASRMS. Notably, we find that solely applying reinforcement learning with verifiable rewards (RLVR) Guo al. (2025) in reward modeling does not fully realize the model's reasoning capabilities. We also observe that plain perceiving the fine-grained distinction across different question types.
(CoT) reasoning falls short at
Through a series of studies, design a training pipeline that introduces reasoning distillation prior to RLVR, ultimately resulting in the development of RM-RI. R1. To fully elicit the reasoning capability Shain-of-Rubriee
of RM-R1 for reward modeling, model categorizes the input sample into one of two categories: chat or reasoning. For chat tasks, the model generates a set of evaluation rubrics, justifications for the rubrics, and evaluations tailored to the specific question. For reasoning tasks, correctness is the most important and generally preferred rubrics, so we directly let the model first solve the problem itself before evaluating and picking the preferred response. This task perception enables the model to tailor its rollout strategy applying rubric-based evaluation for chat and correctness- -irst judgment for reasoning - resulting in more aligned and effective reward signals. In addition, we explore how to directly adapt existing reasoning models into reward models. Since these models have already undergone substantial reasoning- focused distillation, we fine-tune them using RLVR without additional distillation stages. Based on our training recipes, we produce RM-RI models ranging from 7B to 32B.
design
(CoR) process. Specifically, the
Empirically, RM-RI R1 models consistently yield highly interpretable and coherent reasoning traces. On average, R.M-R. R achieves sfate-of-the-art performance on RewardBench (Lambert et al. 2025). RM-Bench (Liu et al. 2025ah, and RMB (Zhou al. 2025), outperforming 70B, 340B, 4o. and Claude models by up to 4.9%. Beyond final performance, we conduct extensive empirical analyses of RM-R1, including ablations of our training recipes, studies of its scaling effects, comparisons with non-reasoning baselines, detailed case studies, and training dynamics.
In summary, our main contributions follows:
2

---

Published as a conference paper at ICLR 2026
ScalarRM
GenRM
Query
resporne
Liner Funetion
Julge
RM-RI Training
Inference Input
Model Type
Training Input
Model Type
Task/ Object
Training
Task
Output
Output
Query
Distillation
Do Mal
GenRM
"Wachh reeponee conmenheeeen"
Answer
GenRM
Reasoning
Trace
After Training
Query
Query
Reward Signal R.
RessRM
Poxeriee
Crique
Cumulative Reward
After Training
RM-RI's Structured Reasoning
Query
RM-RI
R1.82
stop...
Knubriesll Emporty Emciinal Valibtion II., II... Sirubriesk
Kirubrices
Cavel fire resporas vilcte the user's amations.... Slaval
Channel
Complex Criquue
respore
Figure Training pipeline RM-R1. Starting from an instruct model (GenRM), RM-RI training involves two stages: Distillation and Reinforcement Learning (RL). In the Distillation stage, we use high-quality synthesized data bootstrap RM-R1 RR1' reasoning ability. In the RL stage, RM-RI's reasoning ability for reward modeling is further strengthened. After distillation, GenRM evolves into a REASRM. RM-R1 further differentiates itself by being RL finetuned on preference data.
We demonstrate that reasoning abilitics are crucial for reward models, and propose to formulate reward modeling as reasoning process enhance interpretability and accuracy.
reasoning-oriented We design training recipe based reasoning oriented distillation and RL that produces set of reward models RM-RI R - that can outperform larger models by up to 4.9% on average.
We present systematic empirical study of different training recipes for REASRMS, providing insights into the impact diverse training strategies on the final reward model performance.
2
RM-RI
Figure presents the overall training pipeline of RM-R1, which consists of two stages: reasoning distillation and reinforcement learning. (1) Reasoning Distillation: Starting from an off-the-shelf instruction-tuned model (e.g. Qwen-22 5- Instruct), we further train the model using synthesized high-quality reasoning traces. This stage equips RM-RI with essential reasoning capabilities required for effective reward modeling. (2) Reinforcement learning: While distillation effective for injecting reasoning patterns, distilled models often overfit to specific patterns in the training data, limiting their generalization ability (Chu al. (025). To overcome this limitation, we introduce a reinforcement learning phase that further optimizes the model, resulting in the final version of RM-RI.
2.1
TASK DEFINITION
Given a preference dataset:
((x 3a
(I)
where æ is a prompt, Vu and y/b are two different responses for æ, and E {a, b) is the ground truth label that indicates the preferred response. We define the generative reward modeling task follows:
Let ry denote generative reward model parameterized by For each data sample, re generates a textual judgment, consisting of ordered tokens j (f1.12,... .r), modeled by:
To Ti(i Mos Wb) = ro ro Has Nh. j<t).
(2)
3

---

Published as a conference paper at ICLR 2026
Note that j contains prediction of the preferred response i i c.j. j. The overall objective is:
max E ().
[ [[i -1].
(3)
2.2 REASONING DISTILLATION FOR REWARD MODELING
For an instruction-tuned model (e.g. Qwen 2. 2- 14b--4--20 2.5----iisttt (Yang al. 2024 4) it is quite intuitive to turn it into a GenRM simply by prompting. However, without inn-tuning on reward modeling reasoning traces, these models may struggle to conduct consistent judgments. To bootstrap its reasoning potential, we start with training an instruction-tume -unne model with long reasoning traces synthesized for reward modeling. Specifically, we sample M data samples from D and denote it (6) ,(00) E Duub, we ask an "oracle" model like o3 or as Subb. Given a data sample (x()), yo ,
(4) chosen Claude 7- sonnet generate its structured reasoning trace r()) justifying why yi as the preferred response of 2((), We then construct the reasoning trace ground truth:
(i) Vtraae
(+) @ 1()
where + denotes string concatenation. Given all the synthesized reasoning traces r((), the final distillation dataset is defined as:
Daistill
{xx11) [)rncc))
Formally, the objective of distillation is to adjust 0 to maximize the likelihood of generating the desired reasoning trace and picking the response y given the prompt æ. We minimize the negative log-likelihood (NLL) loss:
Caistill (0)
Σ
E
log (Ve ax (z,3)),
(6)
Paiitin CEl|oll
where vCr = (y), y/. 1 denotes the sequence of tokens preceding position l. More details of generating high-quality reasoning chains included in Section D
2.3 RL TRAINING
Although distillation is proper way to turn a general generative model into a GenRM, it often suffers from overfitting to certain patterns and constrains model's ability to generalize its reasoning abilities for critical thinking (Chu et al., 2025; Stanton et al. (02T). which is essential for reward modeling. To address this, we propose to integrate RL as a more powerful learning paradigm to enhance the model's ability conduct reasoning baasoning-based rewarding. Training policy model using RL has been widely seen in the preference optimization phase of L.M post-training (Ouyang et al., 2022), and it is quite natural to extend this paradigm for training REASRM. To be specific, directly treat our reward model ra( x, Nos Mb) as if it is a policy model:
max E (x. (2. (.gp.pp.........roGGll..... To6
[R(x,)) 3DRL (rellrret),
(7)
where Tref is the reference reward model. practice, we use the checkpoint before RL training as Tref, and that means Tyef could be an off-the-shelf LLM or the LLM obtained after the distillation step Section 2.2 Rax j) is the reward function, and DkL. is KL-divergence. The x denotes input prompts drawn from the preference data D. The j indicates the text generated by the reward model, Group Relative Policy
which includes the reasoning trace and final judgment î. In practice, Optimization (GRPO) (Shao et al.) 2024) to optimize the objective in Equation (7), the details of which can be found in Section
2.3.1 CHAIN-OF-RUBRICS (CoR) ROLLOUT
To facilitate the distilled models proactively generate effective reasoning traces, we design system prompt as shown in Figure 3 during rollout. Intuitively, reward modeling for general domain (e.g., chat, safety, etc.) and reasoning domain (e.g., math, code, etc.) should focus on different angles. For example, for the chat domain, we may care more about some aspects that can be expressed in textual rubrics (e.g., be polite), yet for the reasoning domain, we usually care more about logical
4

---

Published as a conference paper at ICLR 2026
Chain-of Cuairi-f-Ruurrics
Please nct as an impartial judge and evaluate the quality of the responses prosided by two Al Chatbots to the Clicnt's question displayed below,
First, classify task one two categories: <type> Reasoning <ltype> or <type> Chat ctypes.
UUee <type> Reasoning </ypp> for tasks that involve math, coding, or requize domain knowledge, multi-step inference, logical deduction, or combining information reach conclusion.
<type> <ltype>
involve open-ended or fictual conversation, stylistic rewrites, safety questions, or general helpfulness
requesis without deep reasening.
If the task Reasoning:
1. Solve the Client's 's question yourself and present your final answer within csolutions </solution> tags.
2. Evaluate
completeness, reasoning quality, referencing, your solution. 3. Include your evaluation inside cevalo <levalo tags, queting or summarizing the Chathots using the following tags:
Chathot responses
-qquote_s <iquote
direct quetes Chatbot
-cuumarry_ _A> e/summary_ As for paraphrases Chatbot A -qquote__e B> <iquooee_ B> for direct quetes from Chathot
<csummary </summary for paraphrases Chatbot
4. End with final judgment in the format: <unswer- [[A]] /answer
<unswer-
[[B]] /|ns]//answer>>>>>>>
If the task is Chat:
1. Generate evaluation criieria (rubric) taikored the Client's question and enclosed cnubrice...//ubrris tags.
2. Assign weights to each rabric itom based on their relative importance.
3. Inside crubric,, include a cootifyy..</fusiilly section explaining why chose those rubric criteria and weights.
4. Compure both Chatbot responses according to the rubric.
5. Provide your evaluation inside cevals..</eval: tags, using <quete_ csummary Ab, <quote B>, and <summary, B> described above.
6. End with your final judgment in the format: cunswerr- [IA]e cunswers or canswers [[B] Janswerr
Figure 3: The system prompt used for RM-R1 rollout.
coherence and answer correctness, Based on this intuition, we instruct ra classify each preference data sample {(x, Vc, } y into one of the two <type>: Chat or Reasoning. For each <type>, we prompt ry carry out the behavior corresponding that type step by step: For reasoning tasks, we ask to solve on its own. During the <eval> phase, compares ye and y, conditioned on its own </solution> and selects an canswer>. Regarding the Chat type, we instead ask ve think about and justify the <rubric> for grading the chat quality (including safety).
2.3.2 REWARD DESIGN
Rule-based reward mechanisms have demonstrated strong empirical performance facilitate reason- ing (Guo et al., 2025). In our training, we further simplify the reward formulation and merely focus on the correctness- based component, in line with prior works (Shao al. 2024 Li al. 2025
Formally, our reward is defined as follows:
ifi=l, R(x, Jlya, vb) = [ otherwise.
{
(8)
where i is extracted from j. wrapped between the <answer>> and </answer> tokens. We have also tried adding the format reward to the overall reward, but found that the task performance does not have a significant difference. The rationale behind only focusing on correctness that the distilled models have already learned to follow instructions and format their responses properly.
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
We evaluate RM-RI I on three primary benchmarks: Bench (Liu et al. 2025a), and RMB (Zhou al. 2025). Our training set utilizes a cleaned subset of Skywork Reward Preference 80K (Liu et al. 2024). 8K examples from Code-Prefference-
RewardBench (Lambert et al. 2025). RM-
and the full -ath-DDO-100- (Lai al. 2024) dataset. For baselines, compare RM-RI with RMs from three main categories: ScalarRMs, GenRMs, and REASRMS. Further details on the benchmarks, dataset construction, and specific baseline models provided in Appendix
5

---

Published as a conference paper at ICLR 2026
Table 1: The performance comparison between best-performing baselines. Bold numbers indicate the best performance, Underlined numbers indicate the second best. The DeepSeek- GRRM models are not open-peneeee weighted, so we use the numbers on their tech report. The more detailed numbers on RewardBench, R.M-Bench, and RMB are in Appendix Table Table and Table
Models
RewardBench
R.-Bench
RMB
Average
SealarRMs
SteerIM RM 70B
88.8
52.5
58.2
66.5
Rurus-RM-7b
82.8
65.9
68.3
72.3
Tnterrnmm2200--rewarr
90.2
68.3
62.9
73.6
93.8
67.3
60.2
73.8
Internim2 7b reward
87.6
67.1
67.1
73.9
ArmoRM-Lllama3--8
90.4
67.7
64.6
74.2
Nemotronn-4-3B0R-Reearr
92.0
69.5
69.9
77.1
B
92.5
70.1
69.3
77.5
INF 1ww-ooo---aaaa3-1111
95.1
70.9
70.5
78.8
GenRMs
Claude-3----9mmet2222000
84,2
61.0
70.6
71.9
1lamm3.1-773-IIntttae
84.0
65.5
68.9
72.8
Gemini-1.S-pre
88.2
75.2
56.5
73.3
3tywork-Oritti--tlaaa-33.3
93.3
71.9
65.5
76.9
GPT-40-0806
86.7
72.5
73.8
77.7
ReasRMr
JudgeLRM
75.2
64.7
53.1
64.3
DeepSesx-PairRM-27
87.1
58.2
DeppSexx-GRW-27B-BBB D Self Saaughteeallaaoor evaluator 11ama3.1 3-1a.
84.5
67.0
86.0
69.0
90.2
71.4
67.0
76.2
Methods
RM-R1 SM-FR--DeeSSSr---Drrr--------
80.1
72.4
55.1
69.2
RM-R1 1M-FR--09E--IIKTTUUTT-1
85.2
70.2
66.4
73.9
RM Ri QWEN INSTRUCT 14B
88.2
76.1
69.2
77.8
RM - SEl-DDEPSEE--DSTTTTE QWEN-14B
88.9
81.5
68.5
79.6
RM-R1 -0-OEN-NNSTTUCTT-3222
91.4
79.1
73.0
81.2
RM-R1 T.FF--DDEPSSxx-DDeettnn------P
90.9
83.9
69.8
81.5
3.2 MAIN RESULTS
Table I compares the overall performance of RM-RI with existing strongest baseline models. The more detailed numbers on RewardBench, RM-Bench, and RMB are Table Table and Table [ in Section H| For the baselines, reproduce the numbers if essential resources open-sourced (e.g.. model checkpoints, system prompts). Otherwise, we use the numbers reported in the corresponding tech report benchmark leaderboard. For each benchmark, select the best-performing models in each category for brevity. Our key findings are summarized below:
State-of-the-Art Performance. On average, RM-R1 ex 20--R--Dea-Saa---DDTTL--------
model surpasses all previous leading Reward Models (RMs), including
Nemot 3 Reward, and GPT-4o, while operating at much smaller scale. Our 32B models, RM-R1: -WEN INSTRU - and RM-R1 DD SE BK DDIS ITIILLEDD I.LED--90 B.BEE---BX-- -2BB, further extend this lead by notable margin. The success of RM-RI is attributable to both our meticu- lously designed training methodology and the effective scaling of our models, as extensively analyzed in Section 4.1 and Section 4.2 In particular, RM-RI outperforms existing top-tier ScalarRMs. This highlights the considerable potential of REASRMS, category where prior GenRMs have exhibited suboptimal performance and are generally not comparable to their scalar counterparts. In contrast to structured rollout and distillation with RLVR training strategy, prior critique-based methods have relied heavily on rejection sampling and unstructured, self self-generated chain-of of-thought (CoT) reason- ing from instruct models (Liu et al. 2025b; Wang et al. (0224)), limiting their reasoning capabilities and leading to inferior performance compared to ScalarRMs. Simultancouslly our comprehensive evaluation indicates that the top-performing scalar models RewardBench do not consistently

---

Published as a conference paper at ICLR 2026
achieve state-oftthe-aa (SOTA) performance; in fact, larger models frequently underperform smaller ones. This evaluation underscores the need for a more comprehensive and systematic approach to RM assessment.
Training towards Reasoning for Reward Modeling. cialized, reasoning-oriented training pipeline delivers substantial performance consistently surpasses -3.1-00.7 a reasoning model five times its size. The RM-RI model series also demonstrates impressive results on RM-Bench, exceeding the top-performing baseline by up 8.7%. On this most reasoning-intnnsive benchmark, establishes a new state-of-the-are 91.8% accuracy in math and 74.1% in code, outperforming the previous best models (73% in math and 63% in code) by significant margins. Furthermore, it also records the strongest reasoning performance among our released models on RewardBench. Despite its performance, our Instruct-based models are remarkably data-efficient, reaching competitive performance using only 8.7K examples for distillation comparred to the 800K examples used training poooseek--Dstilllee (Guo et al., 2025). Overall, our study underscores the significant potential of directly adapting large reasoning models into highly effective reward models.
Effective
Our spe-
gains. For instance, RV-RL--ONENN-IISTTRCC--
It achieves
4 ANALYSIS
In this section, we present a series empirical analyses to understand the key ingredients for training effective reasoning reward models. Our analysis spans scaling effects, design decisions, reasoning ablations, and a case study. We also present additional analysis on training dynamics Section 4.4
4.1 TRAINING RECIPES
We first investigate the key ingredients underlying the successful training of RM-R1. Through series of ablation studies, we examine our design choices to identify effective strategies for training high-quality reasoning reward models. We compare the following settings: Cold Start RL, Cold Start RL + Rubrics, Cold Start RL + Rubrics Query Categorization (QC), and Distilled + RL + Rubrics QC (i.e., RM-R1). The details of these settings are in Section L.I
Table 2: Ablation study of the design choices for Reasoning Training on RewardBench.
Method
Chat
Chat Hard
Safety
Reasoning
Average
Instruct (Original)
95.8
74.3
86.8
86.3
85.8
Instruct + Cold Start RL
92.5
81.5
89.7
94.4
89.5
Instruct Cold Start RL Rubrics
93.0
82.5
90.8
94.2
90.1
Instruct + Cold Start RL Rubrics QC
92.3
82.6
RM-R1
95.3
83.1
91.9
95.2
91.4
In Table present the results of the ablation studies described above, using the Qwen-2.5 Instrncttttt32 model as the Instruct (Original) model. Several key conclusions emerge:
RL training alone is insufficient. While Cold Start RL slightly improves performance on hard chat and reasoning tasks, it fails to close the gap to fully optimized models.
CoR prompting optimizes RM rollout and buosts reasoning performance. Instructing RM-RI to self- celf-generate chat rubrics problem solutions before judgment helps overall performance, especially for chat and safety tasks. Incorporating explicit query categorization into the prompt notably improves reasoning performance, suggesting that clearer task guidance benefits learning.
Distillation further enhances performance across all axes. Seeding the model with high- nigh-quality reasoning traces before RL yields the strongest results, with improvements observed on both hard tasks and safety-sensitive tasks.
7

---

Published as a conference paper at ICLR 2026
Model Scallin
RewardBench
RM-Bench
Avg.
Imgroveenn (N)
Thain on Full Data
Nuftemunce
Instruet SFT Distilled RM-R1
90.9
75.4
65.9
77.4
91.4
79.1
73.0
81.2
Train on 9k. (Distillation) Data
Instruct SFT
8X.8
74,8
66.9
76.6
Budget
Instrnct Distille
79.2
(a) Model Size
(b) Inference Compute
Figure 4: Scaling effect of RM-R1. (a) Larger models benefit more from reasoning training. (b) Longer reasoning chains improve RM performance.
Table 3: Comparison of reasoning-based train- ing versus SFT across benchmarks. * indicates reasoning- based methods. Reasoning training con- sistently yields better performance.
4.2 SCALING EFFECTS
We then investigate how model performance varies with scale, considering both model size and inference-time compute. In some cases such as ScalarRMs from InternLM2 (Cai et al. (2024) and Skywork (Liu et al. 2024) the smaller models (7B/8B) outperforms the larger ones (20B/27B), showing no advantage of scaling. In this subsection, we show that this trend does not hold for RM-RI, where scaling brings clear and substantial improvements.
4.2.1 MODEL SIZES
We first analyze the impact of model scale. Our study is based on the 2..em-2.5-Insse model family at three sizes: 7B, 14B, and 32B. We evaluate performance improvements resulting from training procedure described in Section [ with results averaged across three key benchmarks: RewardBench, R.M-Bench, and RMB.
For each model size, compare the original and post-training performance. Figure plots the relative improvement (%) with respect to model size. Observing an approximately linear trend, we fit linear regression model and extrapolate hypothetical scales of 3B and 72B, shown using faint markers and dashed extensions. The results strongly support a scaling law for reasoning reward models: larger models not only result in absolute better final performance but also consistently yield greater performance gains. This aligns with the intuition that our training effectively leverages the superior reasoning capabilities of larger models.
4.2.2 INFERENCE- TIME COMPUTATION
Next, we examine how model performance varies with different compute budgets measured in number of tokens allowed during inference. Since this particularly relevant reasoning- focusee models, we fix our base model to DeepSeek -R--iitttttt i - 1B.. We evaluate average performance across the three key benchmarks using a wide range inference-time compute budgets: 512, 1024, 2048, 4096, and 8192 tokens.
To ensure a fair comparison, we match the training rollout budget to the inference budget in each setting (i.e., we allow a maximum of & tokens during training for a compute budget of k at inference). All models are trained using GRPO with identical datasets and hyperparameter configurations. Figure 4b shows the relationship between compute budget and performance. We observe clear improvement trend as the inference budget increases. This highlights the benefits of long reasoning chains in reward modeling.
4.3 EFFECTIVENESS OF REASONING TRAINING
We now analyze the impact of reasoning-basee based training. Here, we demonstrate that reasoning-baas -based training outperform answer-only approaches. We consider the following settings:
Instruct SFT. This approach fine-tunes the instruct model directly toward producing the correct final answer using the full dataset, without providing any intermediate reasoning chains.
8

---

Published as a conference paper at ICLR 2026
Instruct + Distilled SFT. This approach applies SFT (with respect to the answer directly) after the distillation stage, serving as a direct comparison point with RM-RI trained with RL.
Instruct + RM-R1 (Distilled + RL). This is the full approach proposed in this paper, following the procedure detailed in Section
Instruct Distilled. This setting uses the model checkpoint immediately after the distillation stage, before any RL fine-tuning.
In summary, methods with "+ RM-RI" or "+ Distilled" represent reasoning- -basodd approaches, while the remaining methods are purely non-reasoning-beet baasoonin---ssed approaches. In T we report the results measured across the three benchmarks. The findings clearly demonstrate that reasoning training significantly benefits reward model performance. Under fair comparisons (i.e., training on exactly the same amount of data), brasoning-based models consistently outperform their SFT-only counterparts. In particular, even high-quality distillation alone, applied to a small subset of the data, provides notable gains, highlighting the value of incorporating structured intermediate reasoning.
315
M 74
(a) Cold Start RL
(b) Warm Start RL
Figure 5: RL training dynamics under different settings: (a) Cold Start RL (Eq. 11 and (b) Warm Start RL (Eq. [ In Cold Start RL, the response length steadily increases as the model learns to reason, but training becomes unstable near the end. In Warm Start RL, the model exhibits more stable training, with effective refinement of reasoning traces throughout the process.
4.4 TRAINING DYNAMICS
We analyze the training dynamics of RM-R1 using the tracking both response length and reward progression throughout RL training. We consider two settings: (a) Cold Start RL, and (b) Warm Start RL following reasoning-chain distillation. We present
model by
the finding in Figure 5
In the Cold Start RL setting, observe that the model gradually learns reason, reflected by steady increase in response length over the course of training. However, training becomes unstable near the end, with sharp drop in the reward curve, suggesting potential issues such overfitting.
In contrast, under Warm Start RL, the model begins with stronger initial reasoning abilities, exhibiting longer responses from the outset. Interestingly, the model first learns produce more concise reasoning traces before gradually increasing response length again as training progresses. The reward curve rises smoothly and consistently throughout training, demonstrating more stable and efficient learning compared to the Cold Start setting.
4.5
CASE STUDY
To gain deeper insights into RM-R1, further conduct study using comparing its behavior with a cold-start RL model. The results, summarized in Table reveal the following key observations:
Transparent Judging Process. The reasoning traces generated by RM-R1 are highly interpretable and coherent, reflecting the model's perception of human preferences. It explicitly articulates why certain responses are better, providing transparency into its evaluation process.
Highh-uality, Question-Dependenn Rubrics. RM-R1 accurately understands the question and the context of comparison, correctly prioritizing "accuracy" as the most critical rubric for medical- related questions. In contrast, the cold-start RI. model overlooks the most important factors and

---

Published as a conference paper at ICLR 2026
Table 4: Case Study of RM-R1(suceing version). Complete Chatbots and model responses are shown in Table 9 10 and 11 Cold-Start RL evaluation tends to focus on superficial features (e.g., number of suggestions, relevance), whereas RM-R1 produces high-quality rubries (e.g., accuracy for factual questions), faithfully follows them, and evaluates responses based on their actual content.
Question: symptoms sickle-cell discase
Chathot A:
5, Slower growth and development; Jaundice; 7. Swelling of hands and feet; 8. Vision loss; 9. Tiredness and weakness afler physical activity; 10. Breathlessness; 11. Fever, chills, and night sweats.
Painful or yellow skin lesions; Frequent infections; Fatigue and weakness:
Chatbot B
Sickle cell discase genetic disorder that affects shape of red blood cells. It is most common in African Americans, but can also affoct people from other ethnic groups. The symptoms sickle cell disease vary widely from person person, but some common symptomns include:
1. Fatigue: ... 2. Pain: ...; 3. Fever: ...; 4, Anemia: ... Delayed growth and development: ...i Infections: ...; 7. Pigmentation:
It's important to note that not everyone with sickle cell discase will experience all of these symptoms, Therefore, it's important to consult with a healtheare provider if you suspect you or someone you know may have sickle cell disease.
Cold-start RL evaluation X
<t.pe>>Chat < styps>
<rubrics los Relevance 40 % Comprehensiveness 30 % Claritylorganistiion 30 % </ e/rubric 1.?
<eval> Chathot A provided a of symptoms that are directly relevant sickle cell discase. The response is compreheusive and includes wide range of symptoms commonly associated with the condition. The symptoms are listed in clear and oeganized manner, making for the client to understand the information. Chatbat B's response less organized than Chathot A's response, and the explanation of sickle cell discase be unnecessary for the client's question. </eval>
consver [[]]] MAJK/arten /amawar>>
RM-RI evaluation /
<ttpe>>Chal Chaik/type>>
<rubr Accuracy 40 % Comprehensiveness 30 Clarity 20 % Helpfulness 10
<just Accuracy weighted highest because providing correct information about medical conditions is crucial. Comprehensivenes is important because. ...s/ just iyy
</rubrics c>
<eval> Accurocy of information: Chatbot provides list of symptoms without explanation, but some listed items (e.g. "painful red or yellow skin lesions," "vision loss") are not typical sickle-cell discase. Chatbot B offers more accurate information, explaining cach symptom's nature and severity. Comprehennrveeesss.... Clarity and pooniiation....ellalaeeee </eval>
conswers [[J] </answem>
instead emphasizes superficial or broadly defined features (e.g., relevance). The ability to generate high-quality, question-specifi rubrics stems from the knowledge acquired during the distillation.
Faithful Adherence to Rubrics and Content-Rased Judgment. RM-R1 grounds its evaluation in the actual content of the model responses. It correctly identifies inaccuracies in Chatbot A's response based on factual content rather than surface presentation. Furthermore, it systematically evaluates all aspects of the rubric, leading to structured, interpretable, and verifiable judging process.
5 CONCLUSION AND FUTURE WORK
In this paper, we revisited reward modeling through the lens of reasoning. We introduced RM-R 1, a family of REASRMS that effectively generate explicit chains rubrics and rationales, and scale with both model size and inference compute. Across three public benchmarks, RM-RI matched or surpassed commercial and open-source RMs while producing more interpretable judgments. Ablation investigations reveal that (1) task-type categorization, (2) bootstrapping from high-quality reasoning traces, and (3) RL fine-tuning are all indispensable. Qualitative analyses further showed that RM-RI learns to prioritize high-impact rubrics, faithfully follow its own criteria and justify coherently. Future work includes active preference collection, where REASRMS use active learning to query human preference only when the current rubric set is insufficient for a new preference sample. Finally, it would be natural to extend our study to multimodal/agentic reward modeling scenarios.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
RM-RI focuses on fundamental research in reinforcement learning and reward modeling. Our methods are developed and evaluated entirely on publicly available benchmarks without involving human subjects, sensitive personal data, or private information. The proposed training pipeline is designed as general optimization technique and does not raise concerns regarding faimess, bias, discrimination, privacy, or security. We believe that our study poses no foreseeable ethical risks and fully complies with research integrity standards.
REPRODUCIBILITY STATEMENT
We have taken concrete steps to facilitate independent reproduction of our results. The experimental setup, datasets, baselines, and evaluation protocols are detailed in Sections [ [.] with train- ing/evaluation hyperparameters, rollout settings, and compute requirements consolidated in Section[G We provide an supplementary repository (see / containing: (i) Distillation data curation and training from OpenAl and Claude models, (ii) RL training with publicly available preference data, (iii) prompt templates for all settings (Distillation data curation, distillation training and RI training for instruct/distilled models), (iv) the chain-of- rubrics evolution mechanism and implementation details for rubric sampling and aggregation, and (v) the complete evaluation scripts for RewardBench, RMB and RM-Bench. Third-pas -arty models and GPT-40, Gemini) are versioned
com/ RMM-R1-UUCC/RR RM0/RMMBBR R1
services (c.g.. Llama, Qwen, DeepSeek Dareek--iiss Distill weessiileddiiiiiee in the configs. Deviations from defaults and ablations are included with their configs, while more discussions about computational overhead are Section]] We welcome the reviewers to reproduce our results.
ACKNOWLEDGEMENT
The authors would like to thank Prof. ChengXiang Zhai for his insightful discussions. This research is based upon work supported by DARPA ITM Program No. (A8650--3-C--3111 the Al Research Institutes program by National Science Foundation and the Institute of Education Sciences, U.S. Department of Education through Award # 2229873 Al Institute for Transforming Education for Children with Speech and Language Processing Challenges, IBM-IIlinois Discovery Accelerator Institute (IIDAI) Center, and NSF Award # 2416070. The views and conclusions contained herein those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, et al. Gpt-4 technical report. arXiv preprint arX1v:2303.08874, 2023.
Adler, Niket Agarwal, Ashwath Aithal, Dong Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, et al. Nemotron-4 340b technical report. arXiv preprint arXiv:2406. 11704, 2024.
AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1:1, 2024.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint urXiv:2212.08077 2022.
Chris I. Baker, Rebecca Saxe, and Joshua B Tenenbaum. Action understanding as inverse planning. Cognition, 11333)339--349 2009.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, al. Internim2 technical report. arXiv preprint arXiv:2403. 17297, 2024.
11

---

Published as a conference paper at ICLR 2026
Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, and Bingsheng He. JudgeIrm: Large reasoning models as a judge. arXiv preprint S.4i::2504.0003355 2025.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoe V Le, Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes: A comparative study of foundation model post-training. In Forty-second International Conference on Machine Learning, 2025. URL atttps://openreviiee wwtp://eooooomom//oorr3333
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01455, 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama herd of models. arXiv preprint arXiv:2407.21788, 2024.
Andrea Fernández-Sánchhe Juan José Caaaiio-assiiiiiiras, and Ana Sánchez- Bello. Navigating the future of pedagogy: The integration of tools in developing educational assessment rubrics. European Journal of Education, 60(1):e 1826,, 2025.
Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv preprint arXiv:2507.17744 2025.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-perfrrmance rlhf framework. arXiv preprint arXiv:2405. 11143, 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hanna Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. Advances in neural information processing systems, 37: 36602-36633, 2024.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of 1lms. arXiv preprint 2xXi:22406.18625 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. RewardBench: Evaluating reward models for language modeling. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACI. 2025, pp. 1755- 1797, Albuquerque, New Mexico, April 2025. Association for Compu- tational Linguistics. ISBN 979-8-89176 195--99-7777 doi: 10. 10.186///1/225 madings-naacl.96 URL https::/eelanntooooo..oooo//
findings naacl. 96/]
Xuefeng Li, Haoyang Zou, and Pengfei Liu. arXiv:2508..3338,, 2025.
Torl: Scaling tool-integrated rl. arXiv preprint
Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transform- ers to solve inherently serial problems. In The Twelfth International Conference on Learning Representations.
12

---

Published as a conference paper at ICLR 2026
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-rewarde Bag of tricks for reward modeling in llms. arXiv preprint vrXiv:2410.18455 2024.
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. RM-bench: Benchmarking reward models of language models with subtlety and style. In The Thirteenth International Conference on Learning Representations, 2025a. URLhttps: /pppneeveeeee...coo/e. net/ forum?
id 1d=
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint w5X4:25504.02495, 2025b,
William Merrill and Ashish Sabharwal. The expressive power of transformers with chain of thought. In The Twelfth International Conference on Learning Representations.
Gilberto Montibeller and Alberto Franco. Multi-criteria decision analysis for strategic decision making. In Handbook of mullicriteria analysis, pp. 25-48. Springer, 2010.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmmityy Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05550 055300 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.0/777.06347 2017.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for L.M reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URLhttps:: openoevieeeeeeeeee n
ZIM
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In P Proceedings of the Twentieth European Conference on Computer Systems, EuroSys 255 pp. 1279-1297, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400711961 10.1145/3689031..66666
doi:
URL https: /t/.ooi.ooooooooooo org/ 10. 1145/ 388886831111 3696075
Tu Shiwen, Zhao Liang, Chris Yuhao Liu, Liang Zeng, and Yang Liu. Skywork critic URL https
model series. htttp::/huggigggaaet /t/ co/Sky September 2024.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in neural information processing systems, 34: 6919, 2021.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Nicole Van Hoeck, Patrick D Watson, and Aron K Barbey. Cognitive neuroscience of human counterfactual reasoning. Frontiers in human neuroscience, 9:420, 2015.
13

---

Published as a conference paper at ICLR 2026
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture- cantture-offexeeee experts. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Findings of the Association for Computational Linguisties: EMNLP 2024, pp. 10582- 10592, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. URL /tttts://aaclase
logy wwgg///22222 Find inn-innggeennn 1p. 620
Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi- Yu, Richard Yuanzhe Pang. Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-taught evaluators. 2024b.
arXiv preprint arXiv:2408.0223
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. HelpSteer: Multi-attribute helpfulness dataset for SteerLM. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Papers). pp. 3371-3384, Mexico City, Mexico, June 2024c. Association for Computational Linguistics. URL 2wa.2222..aa1122202
185]
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024.
Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, Melanie Kambadur, Dhruv Mahajan, and Rui Hou. Self- Gelf-generated critiques boost reward modeling for language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Association for Computational Linguistics: Human Language Technologies (Volume Chapter the the
1: Long Papers). pp. 1499--11514 Albuquerque, New Mexico, April 2025. Association for doi: 10.18653/v 1/2025 h/20025.aaal--oong....... ma.cl-- org/ 2025 naacl maaci-long........a.a....aoa 5737
Computational Linguistics. ISBN 979- 979---89176-119---
URL
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Boji Shan, Zeyuan Liu, Jia Deng, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng. Zhiyuan Liu, and Maosong Sun. Advancing LLM reasoming generalists with preference trees. In The Thirteenth International Conference on Learning Representations, 2025. URL
net.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In The Thirteenth International Conference on Learning Representations, 2025. URL teetts::/oppnreeeee
net/ nerrffrum?
id=Ccwp4tFFBE
Lianmin Zheng, Win-Lin Chiang. Ying Sheng, Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 2023.
Jialun Zhong, Wei Shen, Yanzeng Li, Songyang Gao, Hua Lu, Yicheng Chen, Yang Zhang, Wei Zhou, Jinjie Gu, and Lei Zou. A comprehensive survey of reward models: Taxonomy, applications, challenges, and future. arXiv preprini arXiv:2504. 12328, 2025.
Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. RMB: Comprchensively benchmarking reward models in LLM alignment. In The Thirteenth International Conference on Learning Representations, 2025. WtRhtttss: petttss://opereeiie. net forum?
1d=kmgr 1eannngl19TT..
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, Karthik Ganesan, Wei-Lin Chiang. Jian Zhang, and Jiantao Jiao. Starling-7b: Improving helpfulness and harmlessness with RLAIF. In First
Conference on Language Modeling, 2024. URL httpss:/oppereviii net/ forum? d
GTDntYTTok
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv 2019.
preprint
14

---

Published as a conference paper at ICLR 2026
CONTENTS
The Use of Large Language Models (LLMs)
16
Related Work
16
C User Prompt for DeepSeek-Distilled Reasoning Models
16
Details of Reasoning Chain Generation
17
Group Relative Policy Optimization (GRPO)
17
Experiment Setups
17
Benchmarks
17
Preference Datasets
18
Baselines
18
Implementation Details
19
H Full Experiment Result
19
Supplementary Information for Section 4
19
L. Ablation Settings
19
J Computational Overhead
21
Analysis for the Importance of RL
27
K. Proof of Lemmall]
29
Proof of Proposition]
29
15

---

Published as a conference paper at ICLR 2026
A
THE USE OF LARGE LANGUAGE MODELS (LLMS)
LLMs did not play a role in shaping the research ideas or writing this paper to an extent that would merit authorship or contributor status. Within this work, LLMs are regarded strictly as the primary subject of study and serve as the central experimental object in our evaluations.
B RELATED WORK
Reward Models (RMs). Early RMs were typically outcome-focused: trained to predict human preference rankings for complete outputs (Zhong al. 2025). Recent advances have looked at providing process supervision, which rewards or evaluates the steps model's i reasoning rather than only the final answer. A series of works propose train process reward models that judge the correctness of intermediate reasoning steps (Lightman et al. 2023 Cui et al. 2025 /ettur et al.) 2025). limitation of many PRMs their heavy reliance on curated step-level human labels or specific schemas, and they often remain domain-specinfe Zhang et al. (2025) propose Generative Verifiers, framing reward modeling leverage chain- thain-of-thought and even use majority voting over multiple sampled rationales to make more reliable judgments. DeepSeek GeMMGGe GRM (Liu et al., (0255) and JudgeLRM (Chen et al. (025) have studied using reasoning models generative reward models, which are the most relevant research to ours. However, their main focus is on the effect scaling inference-time computation on reward modeling. On the contrary. different reward model training paradigms, shedding light on when and why a distilled and RL-trained reward model like RM-R1 has advantages over the conventional approaches.
next-token prediction task. This allows the reward model to
work is the first to provide systematic empirical comparison of
Reinforcement Learning from Human Feedback (RLHF). Early works (Christiano al. 2017 first demonstrated that reinforcement learning could optimize policies using reward model trained from human pairwise preferences. Subsequent studies applied RLHF to large-scale language models using policy optimization algorithms such as PPO (Schulman et al. 2017). For example, Ziegler et al. (2019) fine-tuned GPT-2 via PPO on human preference rewards, and Stiennon al. (2020) showed that RLHF could significantly improve the quality of summarization by optimizing against learned preference model. More recently, Ouyang et al. (2022) used a similar PPO-based pipeline train InstructGPT, establishing the modern RLHF paradigm for mnstruction-folloiing models. Recently, Verifiable supervision techniques have also emerged: DeepSeek-R1 (Guo et al. 2025) uses form of seff-verification during RLHF to reward correct reasoning steps, rather than only final-answer quality. This method incentivizes policies to produce outputs that can be verified for correctness, bridging the gap between pure preference-based feedback and ground-truth signals. However, even with such innovations, most RLHF implementations still treat reward modeling and reasoning as separate stages.
C USER PROMPT FOR DREPSHEK-DISTILLER REASONING MODELS
Large reasoning models such DeepSeek-R1 -1 stilled models (Guo al. 2025 do not have a system prompt, so we show the user prompt for rollouts in Figure
Chain-of-Rubrics (CoR) Rollout for Reasoning Models
Please act as an impurtial judge and evaluate the quality of the responses provided by two Al Chatbots to the Client question displayed below.
[Pairwise Input Content)
Output your final verdict at last by strictly following this format: *<answer> "ccnswer>|[A]] c/answes? if Chatbot A better, or "canswer> [B]errr[[BB]asswwee if Chatbot better.
Figure 6: The user prompt used for RM-R1 rollout (for reasoning models).
16

---

Published as a conference paper at ICLR 2026
D DETAILS OF REASONING CHAIN GENERATION
We now expand on the details of generating high-quality reasoning chains. We first use the same prompt to query Cl Claude- 3. Sonnet, generating initial reasoning traces. However, approxi- mately 25% of these traces are incorrect, primarily harder chat tasks. To correct these cases, we pass the original prompt, the incorrect trace, and the correct final answer to OpenAI-03, which then generates corrected reasoning trace aligned with the right answer.
This two-stage process yields a high-quality distillation set. We deliberately choose the order first Claude, then 03 based qualitative observations: Cl aude excels solving easier tasks and maintaining attention to safety considerations, whereas 03 performs better on harder tasks but tends to overemphasize helpfulness the expense of safety. We select approximately 12% of the training data (slightly fewer than 9K examples) for distillation. This is then followed by RL training.
E
GROUP RELATIVE POLICY OPTIMIZATION (GRPO)
Group Relative Policy Optimization (GRPO) Shao al.; 2024) variant of Proximal Policy Optimization (PPO) (Schulman et al. 2017), which obviates the need for additional value function approximation, and uses the average reward of multiple sampled outputs produced in response the same prompt as the baseline. More specifically, for each prompt z. GRPO samples a group of outputs {y1, 3/2,- ... 3 from the old policy Toais and then optimizes the policy model by maximizing the following objective:
lid
roliii x. I. Asi
(O) = EEND, (1) ~rau (ilm)
min
Coa xr Ji.ce)
(9)
ro(jie n. Ji,sst) clip(
e, + +)..). BKL Irol x)) Trecl [1)] }}.
To (Fre Ir st,c()
where 9 is hyperparameter balancing the task specific loss and the KL-divergence. Specifically, A. is computed using the rewards of a group of responses within each group (ri, F2,.2,,..,, "a.....ra), and is given by the following equation:
mean( (ri, r2, rra))
A.
(10)
std(fr1, ... ...)". rrG)
F EXPERIMENT SETUPS
F.1 BENCHMARKS
In this paper, we consider the following three benchmarks:
RewardBench [Lambert et al., 2025): RewardBench is one of the first endeavors towards benchmark- ing reward models through prompt-chosen-rejeed trios, covering four categories: chat, chat-hard, reasoning, and safety, with 358, 456, 740, and 1431 samples, respectively.
RM-Bench (Liu et al. 2025ap: Building on RewardBench, RM-Bench evaluates reward models for their sensitivity to subtle content differences and robustness against style biases. It includes four categories: Chat, Safety, Math, and Code, with 129, 441, 529, and 228 samples, respectively. Each sample contains three prompts of varying difficulty. RM-Bench is the most reasoning-intent benchmark among those consider.
RMB (Zhou et al. 2025): Compared with RewardBench and RM-Bench, RMB offers a more comprehensive evaluation of helpfulness and harmlessness. It includes over 49 real-world scenarios and supports both pairwise and Best-of-N (BoN) evaluation formats. RMB comprises 25,845 instances in total-37 37 scenarios under the helpfulness alignment objective and 12 under harmlessness.
17

---

Published as a conference paper at ICLR 2026
F.2 PREFERENCE DATASETS
We consider the following datasets for training:
Skywork Reward Preference 80K (Liu et al. 2024) is high-quality collection of pairwise prefer- ence data drawn from a variety of domains, including chat, safety, mathematics, and code. It employs an advanced data filtering technique ensure preference reliability across tasks. However, we identify a notable issue with this dataset: all samples from the magpie_ultan spurious correlation, where rejected responses consistently contain the token "<im_start?> while accepted responses do not. Additionally, responses from this source show a systematic accepted responses are typically single-turn, while rejected responses are multi-turn. This problematic subset constitutes approximately 30% of the Skywork dataset and primarily covers mathematics and code domains. To avoid introducing spurious correlations into training, we exclude all magpie_i ra data and retain only the cleaned subset for experiments.
source exhibit strong
is a high-quality coding preference dataset. It constructed by prompting a model with original code, introducing deliberate bugs, and manipulating examples (e.g., swapping broken and corrected versions, removing error comments) to generate fine- grained preference pairs. We subsample 8K examples from this dataset for use in our experiments.
SeddePreffeeece--
Math-DPO-10K
(Lai et al. 2024) is a high-quality stepwise preference dataset focused on mathe- matical reasoning. We use the full dataset in experiments.
A global statistics of our training dataset is summarized in Table
Table Global Statisties of our Training Dataset. indicates the source is from Skywork-Rewart Preference-8OK-v0..
Source
Size
Domain
magpile_ Lhpppoooooaaalllll no__llmm3.11 1
29682
Reasoning
offset_
8504
Chat (length bias)
helpsteer2*
7221
Chat
wildguard*
6709
Safety
magpile_pro**
2030
Chat
Code--Preference-Pire
8000
Reasoning
Math-DPO-10K
10000
Reasoning
F.3 BASELINES
We compare RM-RI with RMs from three categories:
ScalarRMs. ScalarRMs produce score for model response directly, predicting preference through single numeric scores without explicit reasoning traces. This category includes models such as Eurus-RM Yuan al., 2025). Internim2 (Cai et al.; (024) SteerL.M-RM (Wang al.) 2024c), Nemotron- RM (Adler et al. 2024). Tulu-v2.5 (Ivison et al. 2024). Starling RM (Zhu et al. (024)). ArmoRM (Wang
2024a), Skywork-RM, (Liu 2024 etc. While these models often achieve strong results on well-detined benchmarks, they generally lack interpretability and struggle to capture fine-grained reasoning.
GenRMs. Generative reward models (GenRMs) offer more expressive feedback by producing free-form textual judgments, typically without further training. This includes the widely used LLM- as-a-Judge setup (Zheng et al., 2023), where pretrained language models are prompted to explain and evaluate responses. We also categorize under GenRMs models that directly generate output answers without intermediate reasoning steps. Representative examples include LLaMA (Dubey et al. 2024). Qwen Yang et al. (224)). Claude (Anthropic) 2204),, GPT-4o (Achiam et al. 2023 Hurst et al. 2024 Gemini 1.3 Pro (Reid et al. 2024), and Skywork Critic (Shiwen et al. 2024). By leveraging LLMs' generative capabilities, these models enhance interpretability through natural language rationales and explanations.
REASRMS. Reasonedg-enhanneed enhanced reward models (REASRMS) explicitly incorporate reasoning processes before their final judgments, often trained through critiques chain-of-thought strategies.
18

---

Published as a conference paper at ICLR 2026
Notable examples are JudgeLRM (Chen et al. 2025 Critique-RM RM Yu et al. 2025). DeepSeek- GRM (Liu et al., 2025b), Self-taught Evaluators (Wang et al.) 2024b) and our proposed RM-RI models. These models excel in tasks demanding rigorous reasoning, safety evaluations, and nuanced preference judgments due to their grounding in structured critical thnnking.
G IMPLEMENTATION DETAILS
Our training framework based on VERL (Sheng et al. 2025) and OpenRLHF (Hu et al. 2024). For Instruct models, we use 8.7k data for distillation and 64k for RLVR. For Deepseek-Distillee models, we use the full data for RLVR.
Distillation Stage. We use the SFTTrainer from Open with fixed batch size of 128 and a micro-batch size of 1, training for a single epoch. To optimize GPU memory usage, we enable gradient checkpointing, FlashAttention, and Adam offloading. The learning rates set based the model size: 5c-6, 6, and 2c-6 for models of size 7B, 14B, and 32B, respectively.
RLVR Stage. We use the VERL framework for all GRPO training. The training batch size is fixed at 1024, with mini-batch size 128. We adopt Fully Sharded Data Parallel (FSDP) to improve memory efficiency. For rollout generation, memory utilization capped at 0.4. Sampling follows default parameters (temperature 1.0, top-p- 1.0). KL regularization is applied with coefficient of 1e-3 and clip ratio of 0.2. Each prompt sampled with 7 candidate responses.
vL.LM with tensor parallelism size and GPU
The maximum input sequence length is 4,096 tokens, and the maximum response length is 8,192 tokens. Learning rates are set separately for the two model variants:
Instruct models: 1e-6, 7e-7, and 5e-7 for 7B, 14B, and 32B models, respectively.
7 for 7B, 14B, and 32B models, respectively. Reasoning models: 1e-6, 1e- and 8
We train the 7B, 14B, and 32B models on 1, 2, and nodes, respectively, each equipped with 8 H100 GPUs.
H FULL EXPERIMENT RESULT
In this section, we provide the full experiment results and more comprehensive coverage of existing baselines. The results of RewardBench, RM-Bench, and RMB are provided in Table 6 Table 7 Ta-
ble g respectively.
I SUPPLEMENTARY INFORMATION FOR SECTION 4
I.1 ABLATION SETTINGS
Cold Start RL. This approach generally involves pure RI., with rule-based rewards centered answer correctness and format compliance. Such strategies have achieved notable success advanced mathematical problem solving (Shao et al. (2024).
In this setting, we replicate this conventional training setup. Specifically, we use a combination of format reward and answer reward:
{
if format matches, and Ranswer otherwise,
answer matches,
Reoraat
(11)
0
otherwise.
The total reward is the sum R Ranswer + Riomaat We use the prompt template shown in Figure version without any guidance structured reasoning.
Cold Start RL + Rubrics. To examine the influence of structured reasoning in final model perfor- mance, compared with the last setting, we use the prompt template shown in Figure 7 Compared with
19

---

Published as a conference paper at ICLR 2026
Table 6: Results of proposed method and baselines the RewardBench. Bold numbers indicate the best performance, Underlined numbers indicate the second best. & indicates potential data contamination.
Models
Chat_Hard Hard
Safety
Reasoning
Overall
ScalarRMs
Burus-RK:
98.0
65.6
81.4
86.3
82.8
7b
99.2
69.5
87.2
94.5
87.6
SteerTM-RY y0s
91.3
80.3
92.8
90.6
88.8
Cohere
96.4
71.3
92.3
89,4
Internir2-22b-reearr
98.9
76.5
89.5
95.8
90.2
ArmoRt-111o:3-88--0000
96.9
76.8
90.5
97.3
90.4
Borootoo--43400-RRe
95.8
87.1
91.5
93.6
92.0
90.8
INE ORM Llama3.1 2-277**
Saywork-Rewadd-Gemm
95.8
91.4
91.9
96.1
93.8
96.6
91.0
93.6
95.1
GenRMs
85.5
48.5
75.6
72.1
70.4
Prometheus v2
47.1
80.5
77.4
74.5
1lama3.1-70B-nnate
97.2
70.2
82.8
86.0
84.0
Llama3.1 4050 Instruct
74.6
77.6
Llaute-3----Gonne-2222200 GPT-40-0805
96.4
74.0
81.6
84.7
84.2
96.1
76.1
86.6
88.1
86.7
Gemini-1.S-pro
92.3
80.6
87.9
92.0
88.2
96.9
84.8
91.6
97.6
92.7
Sarooc--rritt---lmm-- 1.ama-3.1-700" 3-
96.6
87.9
93.1
95.5
93.3
JudgaLRK
92.9
56.4
78.2
73.66
75.2
SyпRX
38.0
82.5
74.1
87.1
70.4
RM
WRR---EE-EEEE- DESSEERK QNEX-7B
88.9
66.2
78.4
87.0
80.1
CLoud
97.0
58.0
84.0
92.0
82.8
DeepSeek Mineek--MM-11 33M-163
90.8
74.3
84.7
81.8
82.9
DeepSeek (0M6-k---MMM-1 -0M-3-M--3333-
94.7
77.2
87.0
79.2
84.5
MM-RI-Qwaa-Tassaarc
94.1
74.6
85.2
86.7
85.2
94.1
78.3
83.8
86.0
BeepSeak-Pal1<M-2225
95.5
86.8
52.3
92.0
87.1
91.3
79.4
89.3
95.5
88.9
delf==aughh--vvaluorro---mm
RM
95.3
80,3
91.1
96.8
90.9
QMEN
95.3
83.1
91.9
91.4
the last setting, the model is prompted to generate rubrics and evaluate accordingly. However, pared with the final system prompt of RM-RI Figure all input prompts are treated uniformly-that is, chat and reasoning tasks not distinguished.
Cold Start RL + Rubrics + Query Categorization (QC). This setting largely follows the previous one, with key modification: prompting the LM to first categorize the task into reasoning chat, and then apply different strategies for handling those tasks. Intuitively, reinforcement learning alone effectively explore reasoning tasks, domain where it has already achieved considerable success. Here, we incorporate the system prompt shown in Figure which explicitly distinguishes between chat and reasoning tasks.
For reasoning tasks specifically, we note that answer quality is closely tied to correctness, and that high-level rubries may be less effective than simply evaluating whether the model can solve the problem and verify its own answer. Thus, this setting emphasizes correctness-based evaluation guided by task classification in the prompt.
Distilled + RL + Rubrics + QC (RM-R1). Building on the previous setup, we introduce an additional distillation stage from stronger teacher models as warm start before RI. training. The motivation is that with RI. alone, weaker models (especially at smaller scales) often fail to explore high-quality rubrics and convincing reasoning chains for chat tasks throughout the RL training process. Distilling strong reasoning traces on small subset of data can effectively mitigate this limitation.
20

---

Published as a conference paper at ICLR 2026
Table The full results of tested reward models on RM-Bench. Chat, Math, Code, Safety show the model's Average Accuracy on each domain. Easy, Normal, Hard show the model's Accuracy on each difficulty level across all domains. Bold numbers indicate the best performance, Underlined numbers indicate the second best.
Models
Chat Math
Code
Safety
Easy
Normal
Hard Avg
SealarRMs
steer1n-70b
56.4
53.0
49.3
51.2
48.3
54.9
$4.3
522.5
58.2
51.4
55.5
87.1
63.0
56.5
58.0
51.7
86.8
87.1
67.3
35.3
63.2
RV-Mistral-72
57.4
57.0
52.7
87.2
88.6
67.1
34.9
63.5
Euzus-RM-7B
59.9
60.2
56.9
86.5
87.2
70.2
40.2
65.9
interrn.n2-7B-reeeaa
61.7
71.4
49.7
85.5
70.7
45.1
67.1
dxyerrk-Rewwrrd--emm--222-
69.5
54.7
53.2
67.3
67.8
57.5
53.1
92.4
82.2
71.0
49.8
67.7
GRV-11lama3-BB-ffrree
62.7
62.5
57.8
90.0
83,5
72.7
48.6
68.2
Lnternim2-20b-rewann
63.1 66.8
56.7
86.5
82.6
71.6
50.7
68.3
LLema-3-0011ett2aaa----
71.3
61.9
532
89.6
84.6
72.2
50.2
69.0
Nemaotroa-330D-Rewarr
71.2
59.8
59.4
87.5
71.4
56.1
69.5
URV-1iaMe-3311-2
71.2
61.8
54.1
93.1
84.0
73.2
53.0
70.0
Shyxorkk-eeaardd-laee---...
69.5
60.6
54.5
95.7
89.0
74.7
46.6
70.1
INE
LNE-ORV-LLLmm33117700
66.3
65.6
56.8
94.8
91.8
76.1
44.8
70.9
GenRMs
64.9
52.3
50.5
62.3
82.8
60.2
29.5
57.5
56.3
52.4
52.6
73.8
86.7
64.3
25.4
58.8
stableein-2-12bccha:
67.2
54.9
51.6
65.2
63.5
46.6
59.7
66.4
49.9
54.2
59.9
58.8
55.6
51.3
73.9
69.5
61.1
49.1
59.9
Claude-3-5--comee--202222
62.5 62.6
54.4
64.4
73.8
63.4
45.9
61.0
tulu-=2.55d0p-113b-hh---
68.4
51.1
52.3
76.5
53.6
63.0
69.6
62.1
tulu-2-dpo-13t
66.4
51.4
51.8
66.7
37.7
63.8
30128-10.1B--nnstuce==LL-
78.6
52.3
49.6
64.8
Llama3.1-70B-Insteee
64.3
67.3
47.5
83.0
74.7
67.8
54.1
65.5
Shyeerk-Orttic--1mm---.. 1-700
71.4
64.6
56.8
94.8
85.6
73.7
56.5
71.9
GPT-4o-0806
67.2
67.5
63.6
91.7
83.4
75.6
58.7
72.5
Gemini-1.5-pro
71.6
73.9
63.7
91.3
77.6
75.2
REAsRMs
CudgeLRM
59.9
59.9
$1.9
87.3
73.2
766
$4.8
64.7
RV-BB--OMES-IRSTRRCC--
66.6
67.0
54.6
92.6
71.7
59.7
70.2
73.4
65.7
56.3
74.5
59.7
71.5
64.0
83.9
56.2
85.3
75.9
73.1
68.1
72.4
#M-R1--QEvy-TassRucrr-111
75.6
75.4
60.6
93.6
82.6
77.5
68.8
76.1
Rk-R1-QNaa--nsstoucr1
75.3
80.2
66.8
93.9
86.3
80.5
70.4
79.1
71.8
90.5
69.5
94.1
74.4
81.5
74.2
91.8
74.1
85.4
76.7
83,9
J COMPUTATIONAL OVERHEAD
The core contribution of RM-R1 lies in re-casting reward modeling as a reasoning task and demon- strating, for the first time, that generative RMs can outperform scalar RMs on public benchmarks with fully transparent training recipes and detailed, from-scratch analysis. While tong-chii-offthouu models naturally incur higher inference cost, they offer substantial gains in interpretability, generaliza- tion (Merill & Sabharwal Li et al.), and broader applicability (Gunjal et al. 2025 Sormänddz-Sänccher al. 2025). This analogous the introduction of DeepSeek-RI, which showcased the potential of reasoning models before subsequent work improved efficiency. We believe REASRMS will become more important in the future because it can dynamically allocate more compute more complex problems, which cannot be done in conventional RMs. We similarly view efficiency improvements as orthogonal to our main contribution.
While the long chain-of-thought outputs of REASRMS naturally increase inference latency, this overhead can be mitigated with modern RL engines and careful system design. In conventional RL pipelines, the rollout and reward computation stages often cascaded, leading total wait time that is the sum of both processes. However, a parallel design can implemented where the next batch of rollouts initiated while the reward model is processing the current batch. In such setup,
21

---

Published as a conference paper at ICLR 2026
Table 8: The leaderboard RMB, ranked by the average score all subsets, Bold numbers indicate the best performance, Underlined numbers indicate the second best.
Helpfulness
Harmlessness
Models
Overall
BoNN Pairwise
BoN
Pairwise
SealarRMs
Tulu v2.5 13b preference SteerLM-RM 702
0.355
0.562
0.351
0.545
0.453
0.502
0.574
0.578
0.673
0.582
0.472
0.653
0.561
0.721
0.602
Intermin2-20b-reearrd
0.585
0.763
0.499
0.670
0.629
ArmoRM ArmoRM-Llamas Lia-a3 v0.1
0.636
0.787
0.497
0.663
0,646
Internin2-7D-reward
0,626
0.782
0.563
0.712
0,671
Rurus-RM-7b
0.679
0.818
0.543
0.693
0.683
0.627
0.781
0.603
0.759
0.693
INF 0RM 111mm3.11-05 Starling-RM-34B
0.650
0.798
0.607
0.767
0.705
0.604
0.774
0.674
0.795
0.712
GenRMs
Llama2 70b chat
0.289
0.613
0.249
0.602
0.438
1.amm3.1-88--nsttae
0.365
0.675
0.267
0.653
0.490
Gemini-1.5-pro
0.536
0.763
0.299
0.661
0.565
0.480
0.706
0.491
0.671
0.587
0.600
0.725
0.578
0.578
0.620
0.640
0.753
0.614
0.614
0.655
Llama3.1-70B-Lnstzuu
0,648
0.811
0.558
0.739
0.689
Mistral-Tarrg-2400
0.678
0.817
0.583
0.725
0.701
Claude 3-
0.705
0.838
0.518
0.764
0.706
Owen2-72B-Instruct
0.645
0.810
0.649
0.789
0.723
GFT-40-2024-05-13
0.639
0.815
0.682
0.814
0,738
REASRMS
JudgeT.RM
0.363
0.699
0.363
0.674
0.531
0.451
0.658
0.429
0.664
0.551
2
0.543
0.740
0.608
0.765
0.664
0.616
0.786
0.546
0.733
0.670
DeepseeeX-GRM-21BBRFT
0.592
0.801
0.548
0.765
0.670
0.593
0.765
0.613
0.769
0.685
Deepseex-GRR-2278
0.623
0.805
0.570
0.761
0.690
23-R1--2UUK-130770CC-11148
0.594
0.776
0.620
0.778
0.692
RM R1-1--D2- RM-R1--DannSee--D33TT.. C0N---MEN33
0.620
0.782
0.618
0.771
0.698
84-R1-00 10-K---QQE---NsTRuC--
0.636
0.791
0.682
0.809
0.730
the total time is determined by the maximum of the two stages' latencies, not their sum. Since the policy model's rollout time (TI) and the reward model's computation time (T2) are often similar for complex tasks, this parallelism can make the practical overhead of using a REASRM M minimal.
22

---

Published as a conference paper at ICLR 2026
Chain of Rubries (CoR) Roll-our out for Instruct Models (no categorization of task types)
Please act as un impurtial judge and evaluate the quality of the responses provided by two Chatbots to the Client's 's question displayed below.
Instructions
1. Begin evaluation by generating the rubric criteria tailored to the Client's question and context. Enclose the rubric in <rubric> </rubric> tags.
Assign weights each rubric based their relative importance.
3. Within <rubric>, include <justify> ... <fjustify> section explaining the rationale behind the chosen criteria and weights.
4. Compare both Chatbot responses using the rubric.
Include your evaluation <eval> </eval> tags.
Support your analysis using:
<quote_ A> </quote_ A> for direct quotes from Chatbot A
-ssummary_Ae _A> ... <fsummary A for paraphrased summaries Chatbot
<quooee_ ... </quote_ B> for direct quotes from Chatbot B
<summary. B> </summary. B> for paraphrased summaries of Chathot B
Conclude with your final judgmeat using:
</nswers
[[]] s/answer
Important Notes:
Be objective base your evaluation strictly on content responses. Do not let the response order, length, Chatbot names bias your judgment.
Figure 7: The system prompt of the ablation study on cold start RL without categorization of task types.
Chain of -Rubrics (CoR) Roll-out for Instruct Models (no rubrics)
Please act an impartial judge and evaluate the quality of the responses provided by two AI Chatbots the Client's question displayed below.
You should choose the chatbot that follows the client's instructions and answers the client's 's question better. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the chatbots. Be objective as possible. First, compare the chatbot responses and provide your evaluations. Then, conclude with your verdict using exactly format: <answer> (aare>>>[[A/aaaaaaaaa </answer>>
Chatbot A is better, <answer> </answer> if Chatbot is better.
Figure 8: The system prompt of the ablation study on cold start RI. without any rubrics.
23

---

Published as a conference paper at ICLR 2026
Table 9: Complete questions and chatbot responses.
Question: sympioms of sickle cell disease
Chatbot A:
Painful red or yellow skin lesions
2. Frequent infections
3. Fatigue and weakness
Anemia
5. Slower growth and development
7. Swelling of hands and feet
8. Vision loss
Tieddnes and weakness after physical activity
10. Breathlessuess
Fever, chills, night sweats.
Chatbot
Sickle cell disease is genetic disorder that affects the shape of red blood cells. most common African Americans, but can also affect people from other ethnic groups. The symploms sickle cell disease can vary widely from person person, but common symptoms include:
Fatigue: People with cell discase often experience fatigue and weakness, even when they not been physically active.
Pain: Sickle cell discase can cause pain
body, particularly the arms, logs, back, and abdomen. Pain and last for days weeks.
be mild
3. Fever: People with sickle cell disease may he prone to infections and may experience frequent fevers,
anemia, condition in which the body does have enough red blood cells to carry oxygen throughout the body.
Anemia: Sickle cell disease
develupment: Children sickle cell disease may smaller their peers due delayed growth and development.
Delayed growth
Infections: People with sickle cell disease are at increased risk infections, particularly in the bloodstream and urinary tract.
7. Pigmentation: Sickle cell discase can cause dark areas of pigmentation in the body, particularly on the hands feet.
everyune with people may not experience symptoms at all. Therefore, it's important to consult with healthcare provider if you suspect you or someone you know may have sickle disease.
important
diseuse experience these symptoms,, and
24

---

Published as a conference paper at ICLR 2026
Table 10: Cold-Start RL Judges. The rubries and evaluation mainly center around superficial shallow
features (e.g,, number of points listed) without actually taking their accuracy into account.
Cold-start RL evaluation
<ypp>CCute//lppe
srubric>
The evaluation criteria for the responses are as follows:
<item>-Reeev Relevance to the question</item>> (40%). The response should directly address the symptoms sickle
cell discase.
(30%) The response should include wide range of symptoms commonly associated with sickle cell discase.
<itensCCarity and organization</item: (30%): The response should be easy tu read and understand, with
symptoms listed clear organized manner.
<justify>
The question specifically asks for the symptoms sickle cell discase, so the responses should be directly relevant to this lopic. Comprehensivenn and comprehensive list will provide more complete picture the condition. Clarity and organization are important because symptoms can be complex, and well-organized response will make easeer for the client to understand the information.
important because sickle cell discase have variety of symploms,
</justify>
</rubric>
<evals>
<quitee_ A>
1. Painful red or yellow skin lesions
Frequent infections
3. Fatigue and weakness
Anemia
5. Slower growth and development
6. Jaundice
Swelling of hands and feet
8. Vision loss
Tiredness weakness after physical activity
10. Breathlessness
11. Fover, chills, and night sweats.
<lquole_A>
<oummary A>
Chatbot A provided list of symptoms that directly relevant sickle cell disease, The response comprebensive and includes wide range of symptoms commonly associated with the condition. The symptoms listed cleat organized manner, making easy for the client to understand information.
</summary__A
<quote B>
Sickle cell disease genetic disorder that affects the shape of red blood cells. most common African Americans, but cun also affect people from other ethnic groups. The symptoms sickle cell disease can vary widely from person person, but some common symptoms include:
1. Fatigue: People with sickle cell discase often experience fatigue and weakness, even when they have not been physically active.
2. Pain: Sickle cell discase can cause pain the body, particularly in the arms, logs, back. and abdomen. Pain
can be mild or severe and can last for days or weeks.
3. Fever: People with sickle cell diseuse may he prone infections and may experience frequent fevens.
4. Anemia: Sickle cell disease can cause anemia, a condition in which the body does not have enough red blood
cells carry oxygen throughout the body.
5. Delayed growth and development: Chuldren with sickle cell discase may be smaller than their peers due to
delayed growth and development.
6. Infections: People with sickle cell disease are increased risk for infections, particularly in the bloodstream
and urinary tract.
Pigmentation: Sickle cell disease cause dark areas of pigmentation in the body, particularly the hands and feet.
<iquote_B
<sumnary B>
Chatbet provided detailed explanation of sickle cell and symptoms. The response
to the question and includes wide range of sympioms commonly associated with the condition. However.
the response is less organized than Chathot A's response, and the explanation of sickle cell disease may he unnecessary for the client's question. <lsummary B>
<leval>
canswers [A]] clanswer>
25

---

Published as a conference paper at ICLR 2026
Table 11: RM-RI RL Judges, The rubrics and evaluation are well-designed and reasonable, assessing the effectiveness of the model response.
Warm-start RL (RM-R1) evaluation
<type>Chnn</ypp>>
<rubric>
1. Accuracy of information (40 %): How correctly does the response describe the symptoms of sicklo-cell discuse?
Comprehensiveness (30 %): How thoroughly does response cover the range of symptoms associated with
sickle-cell discase?
Clarity and organization (20 R3: How well structured easy understand the information presented??
Helpfulness for the user (10 %): How useful the information for someone socking to understand sickle-cell discase symptoms?
<justify>
These criteria focus evaluating how well each response addresses the user's question about sickle-cell discase symptoms, Accuracy is weighted highest because providing correct information about medical conditions is crucial. Comprehensiveness is important, sickle-cell discase has multiple symptoms that vary severity and preseniation. Clarity and organization affect how easily the user can undersiand the information. Helpfulness considers how practical and actionable information is the user.
<fjustily>
<fruhric>
<eval>
Comparing the two responses:
Accuvary of information:
<quote_A>], Painful red yellow skin lesions
Frequent infections
3. Fatigue and weakness
4. Anemia
Slower growth and development
6. Jaundice
7. Swelling of and feet
8. Vision loss
9. Tiredness and weakness after physical activity
10. Breathlessness
11. Fever, chills, and night sweats, sweats.</quote__ </quote_A>>
equote B>1 Fatigue: People with sickle-cell disease often experience fatigue and weakness, even when they have not physically active.
2. Pain: Sickle-cell disease can cause pain in the body, particulatly the arms, legs. back, and abdomen. Pain
can be mild or severe and can last for days or weeks.
3. Fever: People with sickle-cell disease may prone to infections and may experience frequent fevers.
4. Anemia: Sickle-cell discase can cause anemia, a condition in which the body does not have enough red blood
cells earry oxygen throughout the body.
5. Delayed growth and develupment: Children with sickle-cell discase may be smaller than their peers due to
delayed growth development.
6. Infections: People with sickle-cell disense are increased risk for infections, particularly in the bloodstream and urinary tract.
Pigmentation: Sicklo-cell discase cause dark areas of pigmentation in the body, particularly the hands and feet fect.s/quote. B>
Chatbot provides list of symptoms without explanation, but some listed items (e.g. "painful red yellow skin lesions," " "vision luss") are not typical of sickle cell disease. Chathot B offers more accurate information, explaining each symptom's nature and severity. Comprehensiveness:
Charbot A lists 11 symptoms, several of which are inaccurate or uncommon. Chathot B lists symptoms that more representative and supplies contextual detail about variability and medical advice.
Clarity and organization:
Chatbot plain numbered list lacks context. making severity unclear. Chatbot structures the list with explanations, improving readahility and understanding.
Helpfulness:
Chathot is less helpful due to inaccuracies and missing context. Chatbot is more useful, providing accurate descriptions, context, and guidance. <leval>
Sanswers [B]] cansswrr>
26

---

Published as a conference paper at ICLR 2026
K ANALYSIS FOR THE IMPORTANCE OF RL
Lemma (High-reward filtering induces Etrain <. 8). Let X be a finite input (or trajectory) space and let p be probability distribution X (the underlying environment distribution).
Let @roh, Dvriv= X + { be robust and a trivial feature, and define the disagreement event {x: R(x) <-
D {xr X &rob (2) # (eriv
Let R: X - be reward (or quality) function such that larger R(x) indicates better performance. Fix a threshold T E R and define the high-reward and low-rewary events
H for R(x) 2 r):
Assume:
(i) The SFT training distribution is the environment distribution conditioned on high reward:
Pirain
pt-1 HH)
Penv (-) pl-).
(ii) The filter is nontrivial:
0 < H(H) < 1,
equivalently < P(L) 1.
(iii) Disagreement is more prevalent among low-reward trajectories than among high-reward
ones:
Pr[D
Pr[D H)
1 H],
P
Define
Strain
Pr Jtr-Prrain
[D]
-Pr(D)
8
Pr go-Pony
D -Pr[D]
Then Etrain < 8.
Remark (Connection to SFT distillation and RL exploration). Lemma formalizes a realistic pipeline:
The SFT (or distillation) dataset is collected by sampling trajectories from some base policy environment, and then filtering keep only those with sufficiently high reward (e.g.. judged as "good" by humans or a reward model). This corresponds to Ptrain = p(( H).
RL, in contrast, optimizes expected reward under the environment distribution Penv = and through on-policy rollouts can visit both high- and low-reward regions H and L.
disagreement between trivial and robust features tends to hurt reward (i.e., is more common in L than in H), then assumption (iit) is natural. The lemma then shows that the disagreement probability Etrain seen in the filtered SFT data strictly smaller than the true environment disagreement 8.
In practice, the gap can be amplified further by sampling effects. Let be the number of (independent) trajectories in the SFT dataset drawn from Persin and M the number of trajectories during RL rollouts drawn from Pewv- Pr [D] = Strain is very small, the probability that the SFT dataset
contains disagreement example
Pr(no D in SFT data] = =(1 - Ctrain)
which can be close to for moderate N. In contrast, the probability that RL ever encounters a disagreement trajectory is
Pr[at least one D in RL. rollouts] = -1-
which quickly approaches 1 when M is large and > 0. Since the number of possible trajectories typically grows exponentially with horizon, finite distilled SFT dataset can easily miss rare but systematically harmful disagreement patterns. while RL exploration eventually uncovers them. This provides concrete justification for the core assumption Etrain < 5 that in the next proposition.
27

---

Published as a conference paper at ICLR 2026
Proposition 1 (Approximate shorteut agreement SFT vs. RL). Let X be finite input space and let Purin and Peny be two probability distributions on
Let
@rob, Ptriv X - (0.1)
denote a robust and a trivial feature, respectively, and define the ground-truth label as y(x) = Prob(D))
x EEX.
Define the disagreement probabilities
Strain
Pr prrPeratm
[ (a) + (roo
8
Pr Broppow
[drriv (2) (rbb(r)].
Assume the gap
êtrain <
(12)
(For example, this is guaranteed if Perain and Penv arise via high-reward filtering in Lemmal 7
Consider deterministic policies π X > {0, 1] and define:
The SFT (imitation) objective
(SFT(T) Ex-prenin [1((()) + {())}}
The RL objective (expected reward the environment) with reward r(x, a) 1{a v(x)):
JRL(T) = Euwppen [r(z, π (x))] = Expepano [1<x(x) y(z)}}
Define the trivial and robust policies by Teriv (2)) (x)), Trob( Prob(x)) =))))
Then:
(a) The SFT objective distinguishes the two policies only up to the small margin Serain
LSrr(Frab) =,
(eeer((rrv))))).) = Etrain-
(b) The objective distinguishes them by the larger gap 6:
(TYo)) 1, JRI(TTiv)) 5.
In particular, the improvement in SFT loss when going from Tuiv Trob is Eurain- whereas the improvement in RI. reward is >> Etrain-
(c) For any policy we have
(R)(T) S 1= JRI
with equality if and only if x(x) y(x) for Peuv almost every z. Thus Toob the unique optimal policy for the RL objective (up to Penv Penl-null sets), while Rwiv is strictly suboptimal whenever 8 0.
Remark (Interpretation for SET vs. RL and distillation). In the regime where the SFT teacher data-generating generating process is strong but imperfect, it is natural to have Etrain very small (rare disagreements between trivial and robust features on the observed SFT data), while the environment disagreement 8 is larger, because harder or more diverse test prompts break the trivial shortcut more frequently.
or data-
One concrete interpretation is:
The trivial feature Ouriv encodes the behavior of a teacher policy nr, obtained for example by supervised finetuning heuristics, with
TT (r(x)) (x))
Pr groDany
(T) v(x)) 1- with 0 < 1
28

---

Published as a conference paper at ICLR 2026
The SFT dataset is produced by distilling this teacher: sample x. ~ Purain and label it with Fr (x). Under high-reward filtering (Lemmal TV. the teacher almost never disagrees with
y(z) Peraino Etrain is small.
Proposition then says:
From the SFT objective's ' perspective, moving from wwiv (the shortcut / teacher policy) to Wrob only improves the loss by Etrain 1, which can be negligible relative optimization noise, model capacity, or inductive bias. This makes it easy for SFT to "overfit" to the trivial pattern.
From the objective's 's perspective, the same move improves the reward by 6, which can be substantially larger. Thus RI. has a much stronger structural incentive to abandon the trivial shortcut and adopt the robust strategy Tyob, which is in fact the unique policy that maximizes expected reward.
The exact case Etrain = O corresponds to an idealized limit in which trivial and robust features agree on all SFT training examples; the general case ētrain is what one expects in realistic pipelines with high-reward filtering and broader RL exploration.
K.1
PROOF OF LEMMA
Proof. Write
= = (0,
1 MLL).
By the law of total probability,
PP[DD]
Pr[D ())(H + Pr|D ((L)
Etrain a + Pr[D (1 ( a).
We want to compare d with Etrain Subtracting, obtain
8 Strain = Etrain
+ Pr[D L] a) Etrain
(1 a) Pr[D L] (1 a)) Strain
1
(1 a3) (Pr(D) L] - -trrain
By assumption, 0< 1 < 1 and
Pr[D D L] > Pr[D H Eurains
so the product (1 (Pr,) (D
L) Etraia) is strictly positive. Therefore
Strain
Etrain < 8,
as claimed.
K.2 PROOF OF PROPOSITION II
Proof. We prove (a), (b), and (c) turn.
(a) SFT objective.
By definition of y and mrobs have (2) y(z) for all X, so
1 (Tobb(z) (x) o y(x)} = O for all x.
Therefore
EsrT (Frot)TToo))) Exapieni
[I (2) # v(x))] 0.
29

---

Published as a conference paper at ICLR 2026
For Tuiv, observe that
(e) v(x)
(x) o (z))
so
CSFT (Teriv)
Perapa [1 ((eriv o @rob (x))[
Pr ar-Parrtn
[ * @rob (x)] Strain
This proves (a).
(b) RL objective.
We similarly have
(x) y(x)} for all 2;
JRL (Troh) = Beeapon [1 [Tob((() = y(z)}} 1.
For Tuivs
we use
(x)) =(x))
(eriv(2) drob
to obtain
JRL(Teriv) = Expexpene [feeivv(2) = Prob (x)]]
- Pr [etrv(()) + @rob (2)) = gemPew
This proves (b).
(c) Optimality of the robust policy for RL. Let π be any deterministic policy. For each x E N.
1 ((x(x) = y(x)) 1
with equality if and only if = y(x). Taking expectations under Penv yields
JRL(T) = Easperr [1 [ (x) = y())}} < E Enp-re [1] 11=1= JRL((Tob)) (Trob)-
Hence (R).(T) S JRL (Trab) for all r,, so Troh is an optimal policy.
Moreover, if JnL (T) then the above inequality must hold with equality, which is only possible ((T(2) = )(z)} = for Penv-almoon every x. Equivalently,
π (xx) = y(x)
for Penv Plmo-almost every i.
Thus any maximizer coincides with Toob almost surely under Penv. and Trob is the unique optimal policy up Peuv Peuv-null sets.
Finally, since 8 > 0 by equation 12 we have
Jna (Triv) =1 1- < = JRL. (Trob)),
so Tuiv is strictly suboptimal. This proves (c) and completes the proof.
30


Reviews:
[{"review_id": "l22qXUeikj", "reviewer": "Reviewer_CBGG", "summary": "- RM-R1 treats reward modeling as a reasoning process, where the model produces reasoning traces instead of only scalar scores.\n- The model follows a structured format with tags such as type, rubric, eval, and answer to standardize reasoning across tasks.\n- It is trained in two stages: first by distilling reasoning traces from stronger verifier models, and then by reinforcement learning with Group Relative Policy Optimization using binary rewards.\n- The goal is to make reward models interpretable, verifiable, and robust by aligning reasoning quality with preference correctness.\n- Experiments show that RM-R1 outperforms traditional scalar reward models in both consistency and interpretability without losing accuracy.", "questions": "- Why was a binary reward signal chosen instead of a continuous or rubric-weighted scoring scheme, given that reasoning traces contain richer evaluative information?\n- Have you measured the factual correctness of reasoning traces separately from their final decision accuracy?\n- Have you quantitatively analyzed whether longer or more detailed reasoning traces actually correlate with better reward accuracy?\n- How do you ensure diversity of reasoning strategies in the training data so the model does not overfit to one verifier's reasoning style?\n- Since distilled reasoning models such as DeepSeek-R1-Distill-Qwen-32B are publicly available and already exhibit strong structured reasoning ability, why did you not adopt one of these as the base RM-R1, instead of training reasoning capabilities from non-reasoning models?\n- In Line 194, can be find -> can be found\n- In Line 166, claude-3-7-sonnet -> Claude-3-7-sonnet\n- judgement should be judgment in American English; I believe the paper mostly uses American English.\n- In Line 181, ) , -> ), (no space)\n- Please use \\citep and \\citet appropriately.\n- Please ensure that the citation formats are consistent, the capitalization is correct, and the information is up-to-date.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- It introduces a clear and interpretable reasoning structure for reward modeling, making the decision process transparent and auditable.\n- The two-stage training pipeline effectively combines teacher reasoning with verifiable reward optimization.\n- It demonstrates that reasoning-based reward models can outperform traditional scalar models in both accuracy and consistency across benchmarks.", "weaknesses": "- The reinforcement learning stage with GRPO optimizes for a proxy reward rather than true human satisfaction, leaving room for reward hacking or misalignment.\n- Generating and processing structured reasoning traces substantially increases training and inference cost compared to scalar reward models.\n- The paper lacks a detailed error analysis showing when reasoning helps versus when it harms reward accuracy.\n- The work does not provide a clear mechanism for verifying the correctness of the generated reasoning traces themselves, only their final verdicts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3IQA2ZicCn", "reviewer": "Reviewer_fkoN", "summary": "Inspired from reasoning LLMs, the authors add long CoT into reward modeling and introduce a new class of generative reward models (Reasoning RMs). They design a chain-of-rubrics reasoning process, trained a set of RMs (RM-R1) with distillation and RL, and validate their performance and scalability.", "questions": "1. Weakness 1. How do RM-R1 perform when it is used to supervise a stronger model? For example, on a reasoning task where RM-R1 cannot solve correctly but the training model can?\n2. Weakness 2. How much computation do RM-R1 require in comparison with other RMs? Can generative RMs or reasoning RMs benefit from test-time computation, and if so, what is the advantage of RM-R1?\n3. Weakness 3. Is there any evidence other than benchmark scores to support the claim?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of transfering CoT reasoning to reward modeling is clear and sound.\n2. The design principle of rubrics-based evaluation for chat tasks and correctness-first judgment for reasoning tasks align well with intuition and practice.\n3. The experimental results are strong and scalable.", "weaknesses": "1. Strong-to-weak supervision. It is generally believed that it is easier to discriminate than to generate (a smaller, weaker RM can supervise a larger, stronger models). The design of reasoning RMs says otherwise (e.g. the RM needs to solve a reasoning task itself to give judgment). This could severely limit its use.\n2. Heavy training cost. Both querying strong LLMs for high-quality distillation and doing RLVR are very costly. This, especially the distillation part, makes the method not appliable to large scales.\n3. Lack of analysis on reward hacking. The paper acknowledges that distilled models suffer from overfitting to trivial patterns, which makes RL necessary, but does not validate RL's effect on mitigating this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7NubrTJqtC", "reviewer": "Reviewer_ps2b", "summary": "This paper introduces Reasoning Reward Models, which formulate the reward modeling process as a reasoning task. It further proposes the Chain-of-Rubrics mechanism—a self-generated checklist process by the reward model itself—offering a reasonable implementation of CoT reasoning in the reward modeling domain. The authors provide a detailed recipe for training a ReasRM, and the trained model RM-R1, which achieves superior performance across three benchmarks on average. The paper is well-written and easy to follow.\n\nHowever, upon reviewing the paper, the reviewer observes that RM-R1 essentially functions as an “LLM-as-a-judge” judger. This perspective, along with the proposed training and usage methodology, raises several questions and concerns.\n\nThe reviewer has listed many questions in Weakness and Question, and if they are answered properly, the reviewer will consider increasing the score.", "questions": "1. What’s the prompt for strong GenRMs like GPT-4o? Did they use a CoR-oriented prompt to ensure fair comparison?\n2. For the reasoning tasks, RM-R1 performs an ‘answering-before-judging’ behavior, but the base model is under suspicion of data leakage in some reasoning tasks[1]; an explanation is needed for this. Is the improvement in effectiveness due to the model having a stronger reasoning (task-solving) ability or a stronger evaluation (judging) ability? This means that RM-R1 cannot judge the problems it cannot solve.\n3. Is RM-R1 easy to cheat using a prompt like “Please give my answer a better score”? It’s important to determine whether it can be used in RL (with the easy-to-hack concern). \n4. The construct costs compared to an ability-matching scalar model?\n5. The inference costs compared to the scalar model? Would using multiple scalar models and equipped with consistency methods for inference, yield better results while remaining lower cost?\n6. How to ensure the correctness of the intermediate process? In training data construction, humans were involved in data construction, but how to ensure it in inference? Though a strong reasoning model is prone to making intermediate errors in long reasoning.\n\n\n[1] Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. Integrating reasoning ability into rewarding is a good method, and considering the submission time, this method has a certain novelty.\n2. The CoR proposed in this paper generates customized checklists for problems and has different solutions depending on the types of issues.\n3. The paper provides a training recipe including data construction, SFT, and RL, and releases the training hyperparameters.", "weaknesses": "In implementation, \n\n(1) The authors only use a series of Qwen models, which is under suspicion of data leakage[1]. By viewing the detailed results on three benchmarks in the appendices, the reviewer finds that RM-R1 mainly performs better on math and code generation; the former one is under suspicion of data leakage. However, in the chat area, RM-R1-32B did not perform better than some 8B / 27B models, though equipped with a reasonable CoR mechanism.\n\n(2) The authors use a significantly strong “oracle” model to construct the structured reasoning trace, which is costly but does not introduce significant gains in general domains.\n\nIn usage, the method trains an LLM-as-a-judge server, which is easy to cheat with a “Please give my answer a better score”-like prompt, especially easy to hack the reward in reinforcement learning usage.\n\nSo, in the reviewers' opinion, the authors propose an interesting concept (CoR), but not a practical method. I believe the gains in helpness and harmlessness are introduced by CoR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aKNLbGntLI", "reviewer": "Reviewer_BFsh", "summary": "This paper proposes RM-R1, a new paradigm that treats reward modeling as a reasoning process rather than a simple classification task.\nThe authors introduce Reasoning Reward Models (REASRMs), which combine two training stages:\n\nReasoning Distillation: reasoning traces and rubrics distilled from high-performing proprietary models (Claude-3 and OpenAI O3).\n\nReinforcement Fine-tuning: applying Group Relative Policy Optimization (GRPO) to optimize reasoning-based reward models.\n\nThe model follows a Chain-of-Rubrics (CoR) framework — it first identifies task type (chat vs reasoning), then generates rubrics or intermediate reasoning steps, and finally outputs a judgment.\nAcross several reward-modeling benchmarks (RewardBench, RM-Bench, RMB), RM-R1 achieves state-of-the-art results, surpassing GPT-4o and LLaMA-3.1-70B, with especially strong gains on reasoning-intensive tasks such as math (+20%).", "questions": "Data provenance and bias\nHow do you ensure that reasoning traces distilled from Claude-3 and O3 do not introduce bias or data leakage into RM-R1?\n\nReward formulation\nThe final reward is binary correctness (±1). Have you explored multi-component or continuous reward signals (e.g., coherence, rubric consistency)? How stable is the RL training under noisy rewards?\n\nTheoretical motivation\nCan you provide any theoretical or cognitive rationale for why explicit reasoning improves reward alignment compared to outcome-only modeling?\n\nGeneralization\nHas RM-R1 been tested on multimodal or dynamic tasks (e.g., vision-language reasoning or agentic evaluation)? If not, how well do you expect it to generalize?\n\nDistillation fidelity\nWhat fraction of the distilled reasoning traces were incorrect or low-quality, and how does this affect downstream RL optimization?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Conceptual novelty:\nThe paper reframes reward modeling as an explicit reasoning process, bridging evaluation and interpretability.\n\nTransparency:\nRM-R1 produces human-readable rubrics and step-by-step reasoning chains, offering insight into how judgments are formed.\n\nStrong empirical results:\nSubstantial gains over larger models on multiple reward benchmarks; improvements are consistent across scales.\n\nComprehensive experiments:\nIncludes ablation studies, scaling analysis, and qualitative case studies.", "weaknesses": "Data dependency and potential bias:\nRM-R1 heavily depends on Qwen-2.5 and DeepSeek-Distilled-Qwen outputs, possibly inheriting reasoning biases or training contamination.\nMoreover, the distillation data from Claude-3 and O3 could embed stylistic or safety biases not analyzed in the paper.\n\nSimplified reward formulation:\nThe final reward is binary (+1/-1) correctness, lacking multi-component structure (e.g., coherence, rubric adherence).\nNo stability or sensitivity analysis is provided for different reward signals.\n\nLimited theoretical grounding:\nThe paper provides intuitive motivation but no formal justification for why reasoning improves reward alignment.\nConnections to existing PRM or verifiable RM frameworks are missing.\n\nLack of domain generalization:\nAll experiments focus on text-only reasoning; no evidence of transfer to multimodal, code, or embodied tasks.\n\nEthical and bias analysis omitted:\nThe paper claims “no ethical concerns,” yet relies on closed-source models (Claude, O3) for supervision, which may introduce opaque bias or intellectual-property issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)