Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
Authors: Ilgee Hong
Changlong Yu
Liang Qiu
Weixiang Yan
Zhenghao Xu
Haoming Jiang
Qingru Zhang
Qin Lu
Xin Liu
Chao Zhang
Tuo Zhao
Venue: NeurIPS 2025 poster ()
Keywords: Generative Reward Models
Large Language Models
Reasoning
RLHF
Primary area: deep_learning
TL;DR: We propose Think-RM, a training framework for generative reward models that enables long-horizon reasoning, and introduce a pairwise RLHF pipeline that directly optimizes policies using pairwise preference rewards.

Abstract: Reinforcement learning from human feedback (RLHF) has become a powerful post-training paradigm for aligning large language models with human preferences. A core challenge in RLHF is constructing accurate reward signals, where the conventional Bradley-Terry reward models (BT RMs) often suffer from sensitivity to data size and coverage, as well as vulnerability to reward hacking. Generative reward models (GenRMs) offer a more robust alternative by generating chain-of-thought (CoT) rationales followed by a final verdict. However, existing GenRMs rely on shallow, vertically scaled reasoning, limiting their capacity to handle nuanced or complex tasks. Moreover, their pairwise preference outputs are incompatible with standard RLHF algorithms that require pointwise reward signals. In this work, we introduce Think-RM, a training framework that enables long-horizon reasoning in GenRMs by modeling an internal thinking process. Rather than producing structured, externally provided rationales, Think-RM generates flexible, self-guided reasoning traces that support advanced capabilities such as self-reflection, hypothetical reasoning, and divergent reasoning. To elicit these reasoning abilities, we first warm-up the models by supervised fine-tuning (SFT) over long CoT data. We then further improve the model's long-horizon abilities by rule-based reinforcement learning (RL). In addition, we propose a novel pairwise RLHF pipeline that directly optimizes policies from pairwise comparisons, eliminating the need for pointwise reward conversion. Experiments show that Think-RM outperforms baselines on both in-distribution and out-of-distribution tasks, with particularly strong gains on reasoning-heavy benchmarks: more than 10\% and 5\% on RewardBench's Chat Hard and Reasoning, and 12\% on RM-Bench's Math domain. When combined with our pairwise RLHF pipeline, it demonstrates superior end-policy performance compared to traditional approaches. This depth-oriented approach not only broadens the GenRM design space but also establishes a new paradigm for preference-based policy optimization in RLHF.

Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
flgee Hong't
Changlong Yu²
Liang Qiu²
Weixiang Yan²
Zhenghao Xu'
Haoming Jiang
Qingru Zhang
Qin Lu²
Xin Liu
Chao Zhang?
Tuo Zhao?
GGeorgia Institute of Technology Amazon
Abstract
Reinforcement learning from human feedback (RLHF) has become powerful post- training paradigm for aligning large language models with human preferences. A core challenge in RLHF constructing accurate reward signals, where the conven- tional Bradley-Terry Terry reward models (BT RMs) often suffer from sensitivity to data size and coverage, as well as vulnerability to reward hacking. Generative reward models (GenRMs) offer more robust alternative by generating chain-of-thow of-thought (CoT) rationales followed by final verdict. However, existing GenRMs rely on shallow, vertically scaled reasoning, limiting their capacity to handle nuanced or complex tasks. Moreover, their pairwise preference outputs are incompatible with standard RLHF algorithms that require pointwise reward signals. In this work, we introduce Think-RM. a training framework that enables long-horizon reasoning GenRMs by modeling internal thinking process. Rather than producing struc- tured, externally provided rationales, Think-RM generates flexible, self-guided reasoning traces that support advanced capabilities such as self-reflection. hypothet- ical reasoming, and divergent reasoning. To elicit these reasoning abilities, we first warm-up the models by supervised fine-tuning (SFT) over long CoT data. We then further improve the model's long-ho long-horizon abilities by maled reinforcement learning (RL). In addition, we propose novel pairwise RLHF pipeline that directly optimizes policies from pairwise comparisons, eliminating the need for pointwise reward conversion. Experiments show that Think-RA RM outperforms baselines on both in-distribution and out-of-distribution tasks, with particularly strong gains on reasoning-heavy benchmarks: more than 10% and 5% RewardBench's Chat Hard and Reasoning, and 12% on RM-BBecc's 's Math domain. When combined with our pairwise RLHF pipeline, it demonstrates superior end-policy performance com- pared to traditional approaches. This depth-oriented approach not only broadens the GenRM design space but also establishes new paradigm for preference-basee policy optimization in RLHF. The code, datasets, and models are publicly available hwtp://www....llggglllo///aaaaaa
at //tpps://github. ctw.hauuuooommoooooaaamoooeeeeee http///ggeeeennnnnni
Introduction
Reinforcement learning from human feedback (RLHF) has emerged as a powerful post-training paradigm for large language models (LLMs), enabling them to better follow instructions [+4] reason over multiple steps 85 and comply with safety constraints ПН By iteratively shaping
Work done during the internship at Amazon. Emails: (ihong39, tourzhao)@gateech.eee
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Home Cook Recipe Answering
Emergency Health Advice
Geometry Problem Solving
overall, B's response complett and lacks cut--fff, even if bit less detailed. So 6 might be beiler. But wait, A's
First, question is ingesting cleaning chemicats and asking for a home remedd to neutralize them.
me start by understanding the problem again. The points are (0.0). (a,11 11,, and (6,37), and we need to find ab... kay equarionna from diatencsss... .. Hmmn,, maybel mede mistake the stepss, or perlops Assistarnt made error.
includes more ingi ingredients like kecap manis, which important for the flavor. 8 might not have that? Let me check H's sauce ingredients again..... type any sugar, but B just aays "koy sauce", y which might not be the same, Kecap manis sweeter.
Starting with Assistant A: They sugzested drinking milk, water, activated chercoal, a bland diet.... home remedy, so the advice is directly addressing that. walt, is giving milk and charcoal safe and correct
user wants
kecap
with
Alternativaly, the problam might haves simpler approach... Let's say vertex is obiaiined by rotating the secound by 60 degrees... Solving these equations gives 2-21 sqri(3) and b-8 sqrt(3), leading to ab1315.
recommendation? recell that for acid- based chemicalss maybe milk can help with sikaline substences, but what it's strong acid something elxe? Also, activated charcoal might help absorb some toxins, but the user should see doctor first...
Ah, that's a key point The name of the dish is Tahu Geyaat Cirebon, which typloally uses
correct in using kasap manis, while uses regular say sauce. So B's veraion might not be accurate...
So Assistant A correct. Assistant enswer of 353 is wrong.... ...
(a) Self- Reflectein
(b) Hypothetical Reasoning
(C) Divergent Reasoning
Figure 1: Examples of advanced reasoning abilities enabled by Think-RM.
the model with a learned reward signal aligned to human preferences, RLHF bridges the gap between pretraining objectives and real-world usage.
central challenge in RLHF lies in constructing an accurate reward signal. A conventional approach is to use Bradley-Terry reward model (BT RM), which maps prompt and response pair to single scalar score by minimizing empirical risk over preference data [12 While BT RM is straightforward to implement and easy to train, they often overfit to specific patterns in the training data and are highly sensitive to dataset size and coverage, frequently leading to reward over-optimization and reward hacking [16
Generative reward models (GenRMs) offer promising alternative to conventional discriminative approaches 25). By training an LLM to generate chain-of- thouhttt (CoT) explanation followed by final reward or preference, GenRM leverages the pretrained model's existing knowledge and shows greater robustness to data scarcity and distribution shifts, demonstrating stronger out-of- distribution (OOD) performance 22 24 Moreover, GenRM has been shown further improve its performance through vertical inference-ting time scaling, where multiple reasoning paths are generated and then aggregated (e.g., by majority voting or averaging) to produce a more reliable reward or preference estimate [21224] This inference- timeeee-tmmeeeeee scaling is not available to discriminative counterparts such as BT RM, highlighting an additional advantage of the generative approach.
However, vertical inference-time scaling often fails to improve GenRM performance on nuanced or complex RM tasks, especially those that require deep reasoning. While aggregating outputs from multiple reasoning paths improves self-consistency, each path generated by existing GenRM is typically shallow (limited
few hundred tokens) making difficult for any single path to fully capture complex or subtle implicit context. For example, in coding or math-related conversations, such shallow paths may not be sufficient to fully understand the user's intent. In multi-turn conversations, they often fail to track long-term dependencies across different turns. In addition, shallow CoT reasoning is often insufficient to detect single false statement embedded within an otherwise fluent and well-structured response. Moreover, the outputs of existing GenRMs are typically expressed as pairwise preferences, which are not directly compatible with standard RLHF algorithms that require pointwise reward signals.
In contrast, scaling along the horizontal dimension, where the model reasons more extensively within a single trajectory, remains largely underexplored, despite its success in improving the quality of reasoning in other language model applications (26-28). In this work, we introduce Think-RM.
new training framework that transforms a non-reasoning pretrained LLM (e.g., Llama series 22 30
into a GenRM equipped with horg-horizon reasoning capabilities by modeling an internal thinking process. Rather than producing structured, externally provided rationales, Think-RM generates flexible, self-guided reasoning traces that support advanced capabilities such as self-reflection, hypothetical reasoning, and divergent reusoning. as illustrated in Figure [ This enables the model to solve reasoning-hen heavy RM tasks by extending single CoT trajectory from hundreds to thousands of tokens. To stimulate the reasoning abilities for the non-reasoning model, we first warm up the model

---

with supervised fine-tuning (SFT). We generate multiple long CoT trajectories for each GenRM prompt using a pretrained reasoning model and select the longest correct one, which we use to fine- tune the model. After the SFT warm-up, we further refine the model's overly long or noisy reasoning process using rule-based reinforcement learning (RI). In addition, we propose a novel pairwise RLHF pipeline that directly optimizes policies from pairwise preference comparisons, eliminating the need for pointwise reward conversion and enabling more effective use of Think- -M'is outputs.
Extensive experiments show that Think-RM outperforms both BT RM and vertically scaled GenRM on both in-distribution (ID) and OOD tasks, with particularly strong gains on reasoning-hess benchmarks. Specifically, Think-Re RM achieves more than 10% and 5% improvements on the Chat Hard and Reasoning domains of RewardBench [31], respectively, and an 8% improvement on RM- Bench [32]. a challenging benchmark requiring intensive reasoning, with the largest gain of 12% on its Math domain. This establishes sfate-of-the-art performance among all publicly available GenRMS under 10B using only 6K training samples.
Furthermore, integrating Think-RM into our pairwise RLHF pipeline yields stronger end-policy performance compared to traditional pointwise RLHF with BT RM. By shifting the modeling paradigm from breadth to depth, Think-RM not only expands the design space of generative reward modeling, but also establishes a new foundation for preference- based policy optimization in RLHF, paving the way toward better alignment of LLMs with more complex objectives.
2 Related Work
Bradley-Terry (BT) Reward Modeling. The BT framework 33 is conventional approach to reward modeling in RLHF. This approach, pioneered in early RLHF work 12 12 D
continues In the BT framework, reward models are trained using maximum likelihood estimation to map text inputs to scalar scores that preserve the ordering of human preferences [1 However, this discriminative modeling paradigm faces several key limitations. It requires large amounts of high-quality preference data for reliable training, shows high sensitivity to dataset coverage, and remains vulnerable to reward hacking where models learn to exploit patterns in the training data rather than truly aligning with human preferences [16
to be widely adopted in advanced language models like GPT-4 34 and Qwen-2.5 2.5
35
Generative Reward Models. Recent work has shifted towards generative approaches to reward modeling, where LLMs are trained to generate explanatory rationales before making preference decisions 20-25) Unlike diseriminative BT models that directly output scalar scores, GenRMs leverage the reasoning capabilities of LLMs through chain-of-thouu generation, leading to several key advantages. They require less training data, demonstrate stronger OOD generalization, and provide interpretable reasoning traces for their decisions (22 These works show that GenRMs can be effectively trained using standard next-token prediction objectives and can benefit from test- time compute through majority voting over multiple reasoning paths. However, current GenRMs typically generate relatively shallow reasoning paths limited to a few hundred tokens, which can be insufficient for complex tasks requiring deeper analysis or long-term dependency tracking.
LLM--s-a-Judge for Response Evaluation. A parallel line of work explores using LLMs directly as judges to evaluate model outputs (3)-38). While sharing similar goals with reward modeling, this approach differs by using off-the-sheli LLMs without additional training, relying instead on careful prompt engineering to elicit evaluation capabilities. Although strong LLM judges such as GPT-4 can achieve high agreement with human preferences in controlled evaluations, recent studies have revealed significant limitations, including position and verbosity biases, inconsistent judgments across different models, and, most importantly, an inherent ceiling where an LLM's 's ability to judge is fundamentally constrained by its own ability to solve the underlying tasks 122 24 36). These challenges are particularly pronounced in complex reasoning problems requiring detailed analysis or long-term dependency tracking [36] 38). These systematic limitations in LLM judges align with our observations about shallow reasoning in GenRMs, further motivating the need for models specifically trained for deep analytical evaluation.
Long Chain-of- -fhoughf-Thhhht Reasoning. The development of chain-of-thought (CoT) prompting has been crucial for improving the reasoning capabilities of LLMSs [39 While early CoT approaches and their extensions, such as self-consistency 40] and tree-of-thought 41I, showed promise, they typically operated within relatively short reasoning horizons of few hundred tokens.
3

---

Long CoT Extraction [(R, *))ر x(-17,n; Me- Keep the longest R2 with 8j
Preference Dataset (t, Bb, Ub, 6*) E Doret
GenRM Task
D [(2, Mlas H6, e)))
Pretrained LLM Go
Warm-up SFT (Step
Rule-based RL (Step 2)
Think-RM So (reasoming)
(non-ressoning)
Initial LLM Policy mo
Pairwise
Final LLM Policy
RLIIF
Figure 2: Overview of the Think-RM training framework.
Recent breakthroughs have demonstrated the importance of extending reasoning chains to much longer horizons, with OpenAl's o-series 126 43
achieving remarkable performance on complex mathematical and coding tasks through extended multi-step reasoning. This capability was later replicated and publicly released in DeepSeek's 's RI model [27). which showed that long-horizon reasoning abilities can be systematically trained using RL. Similar approaches have been adopted by subsequent models like QwQ 44 and Grok [55, establishing long-horizon reasoning as key capability for solving complex tasks. This evolution directly informs our approach reward
modeling, suggesting that extending reasoning horizons could similarly improve preference learning and evaluation.
3 Method
In this section, we introduce Think-RM, a training framework that enables long-horizon reasoning in GenRM. Our approach begins with a pretrained LLM that initially lacks sophisticated reasoning capabilities for reward modeling tasks. Building upon this foundation, we stimulate the model's reasoning ability through two-stage process: first warming up the L.M using SFT carefully curated set of long-horizon CoT data, and then refining its reasoning process through rule-based RI. Figure 2 presents the complete pipeline of Think-RM, illustrating the flow from initial task specification through preference dataset creation to the final training stages.
3.1 Preliminaries
Generative Reward Modeling, GenRMs are LLMs trained to evaluate responses through natural language reasoning. Let T denote the space of natural language task instructions that define the behavior a GenRM. Each task instruction 7 E T specifies how the model should evaluate responses based on specific criteria. For example, scoring task asks the model to evaluate single response by producing a numerical reward, while a preference task requests a comparative judgment between two responses and outputs which one is preferred. The task instruction is typically provided the GenRM as a system message. In this paper, we focus on the preference task and use five HelpSteer attributes [46). along with safety, predefined evaluation criteria. Note that the proposed method can be generalized to other E T. Details of our task instructions are provided Appendix A
We consider pairwise GenRM Gg. which takes input triplet (z, Va 26), where is the prompt context and yo and yb are two different responses to x. The model generates a corresponding reasoning process It and final preference output for ya and yb, denoted as (R, s)
(- Na. yb). For the type of s, we consider two cases: (1) binary pairwise GenRM, where s E {a, b) indicates
which response preferred; and (2) multiclass pairwise GenRM, where E (-3, - 1.
represents the strength and direction of preference. In the multiclass case, the magnitude of s indicates preference strength, with negative values favoring Ya and positive values favoring y/.
Reinforcement Learning from Human Feedback. RLHF a training paradigm that aligns language models with human preferences through reward optimization. The core RLHF objective aims to

---

maximize expected rewards provided by a pointwise scalar reward model r : x y - R with respect the policy language model Pa defined as:
max 2 DD [r(z,)) v)]
(I)
To optimize T
we can apply the PPO algorithm which iteratively maximizes the following surrogate function:
Polvelz.Vei) Ac, clipi-? t+e Polyela, Vee)) Poula (Wel /2, Meee)
Poui (Welaz M-cee) A. 2.) )
(()) = Ee min
9 -9DKL DKL (Po Pora))
(- () denotes the KL divergence, 3 2 is a hyperparameter controlling the strength of the KL. regularization, and A, represents the advantage estimates for the /-th token. These advantage estimates can be computed using established methods such as generalized advantage estimation (GAE) [8] or group relative policy optimization (GRPO) (27).
where (-
3.2 Warm-up Supervised Fine- Tine-Tuning
To equip reasoning capabilities in non-reasoning LLMs for reward modeling tasks, first warm up the model through fine-tuning on a small set of long CoT trajectories corresponding to task T. To prepare the warm-up CoT data from the preference dataset D Preff = {[rr: yb, 8.) {= we use an off-the-shelf pretrained reasoning model to generate M CoTT trajectories for each instance, denoted (Rey, 8tj) ) )==1 reasoning capabilities spanning thousands of tokens, we select the trajectory with the longest R,g among those satisfying Sty s for each instance. Importantly, these longer trajectories naturally incorporate diverse forms of self-reflection and analytical depth, providing a strong foundation for developing sophisticated reasoning abilities tailored reward modeling tasks. Once the long CoT data is prepared, we optimize the following maximum likelihood objective that combines preference prediction and reasoning generation:
(--|,, Ti, Na,i, Ub..). To equip the L.M with long-horizon
(SF = EI..9..0.....) D-Drr [- log Ge [ Ya, yo. R) - log Go(RT, ya. v6)].
3.3 Rule-based Reinforcement Learning
While long CoT trajectories provide rich reasoning patterns for preference evaluation, the models used to curate such data are not specifically optimized for this task. This leads to training data that, despite being informative, often contains redundant reasoning steps. After SFT on these trajectories, our GenRM model Go naturally inherits this verbose reasoning style. To refine the reasoning process while preserving its effectiveness, we further fine-tune Go with rule-based RL. Specifically, we adopt GRPO from Guo et al. (271. but restrict the reward to be based solely on accuracy. For notational simplicity, we define = (T, x, yar yb). The GRPO loss is then given by:
LORPO (0) LE (1..6...0..*" Dorot, (Re,a (
G Σ min ( Go 8: 1p) A,, clpp Go( 84 8:)) ( (R, 1p)
A. 3 DxI. (Go T
Ge (Ris 8i p)
where G denotes the number of samples per prompt and A, = - FF) + e) is the advantage estimate for the i-th sample, with the mean reward F = (L/G) E/=1 Fyy and the standard deviation
o, (1/(G 1)) E3 - r)2. For the binary output E {a, b), the rule-based reward r; is defined
11.0
8, 8
Ti =
0.0, otherwise
For the multiclass output E f-3, -2, - 1, 1,2,3, the rule-based reward is defined as:
1.0, s*
0.5. sign(s:) sign(s*)
0.0,
otherwise
5

---

This reward design ensures strong learning signals by assigning full reward for exact predictions and a partial reward for correctly identifying preference direction. The RL training phase an essential step for Think-RM, as it enables the discovery of effective long-horizon reasoning paths through systematic exploration of diverse trajectories.
3.4 Pairwise RLHF with GenRMs
Given a trained Think-RM Ge. we propose new direct preference-based approach to fine-tune target policy Pa. eliminating the need recover pointwise rewards for individual prompt-response pairs (z. y). During training, each RLHF iteration processes mini-batch of B prompts. with G sampled responses per prompt, resulting in a total of GB responses. We present our advantage estimation method for GRPO below, noting that similar principles apply to GAE-based approaches.
Pairwise Preference Strength Matrix. We construct a skew-symmetric matrix D E R (GB)))(GB)
indexed by responses .1..... MGB, where each element di (the entry in the i-th row and j-th column) represents the preference strength of y, relative to y/. Note that d,j = dys = 0 if y/s and yy correspond to different prompts, or if j. For each pair (yh, Mj) sharing the same prompt with # j. obtain single GenRM evaluation (R, s) Go T, Uis Vj). ). For multiclass output s, we define matrix entries as:
dij = -,
d -d3--
For binary preferences, first map {a,b) b) 3o E +1, (where -1 and b +> + +1) and incorporate the reasoning length R as a confidence measure:
dy
R/ d==-des
This formulation ensures that longer reasoning chains, which often indicate more ambiguous compar- isons, result in smaller preference strengths.
Advantage Estimation using Pairwise Preference Strength. We can compute the advantage estimate A, for the i-th response using the pairwise preference strength matrix as follows:
ES dis
A, G
A
VS.
+E
Ens B +
Ge
2G 2(G -
Standard advantage estimation in GRPO
This formulation derives from the following relationships:
1
Σ
(ri r3)
1
dij and o,
G (Ts 7)2
1
dij.
G f=1
G
G
2G 1)
1.j
4
Experiments
In this section, we investigate the effectiveness of horg-horizon reasoning Think-RM across diverse tasks. We evaluate various RMs on in-distribution (ID), moderately shifted, and out-of-distribun (OOD) benchmarks. For a strict head-to-head comparison, all RMs use the same backbone model and the same training data. The only difference is the training method. This setup isolates the effect of long-horizon reasoning in Think-RM. Additionally, we implement our pairwise RLHIF framework, which directly uses pairwise preference comparisons from GenRMs to train the policy. We compare this approach against the traditional pointwise RLHF method, which relies on pointwise rewards (c.g., from BT RM), to evaluate the relative effectiveness of the two strategies.
4.1 Experiment Setup
Training Data and Baselines. We use SeipSteer2-Prefeeemee [46] as training data for all baseline methods and Rh.. HelpSteer2-Prefereee contains 9,125 high-quality samples, each consisting of a single prompt with two candidate responses and covering diverse topics and difficulty levels, including multi-turn contexts. In addition, it provides human preference annotations in the form of preference strengths and corresponding human-wrine written justifications. After removing tie samples
and excluding the test split, we obtain 6,766 training samples. We use QwQ-32B
45 to generate

---

a set of long internal thinking CoT trajectories { { } for each instance i in the original HelpSteer2-Prefert Preferenee dataset. We set M 10 and discard any instance if all generated preferences disagree with the ground truth (i.e., Sij A for all j). This results in 6K training samples for binary preference and 4K for multiclass preference. Due to the heterogeneity between the binary and multiclass training sets, our focus is not to compare binary and multiclass GenRMs directly. Instead, we focus on comparing different types of RMs within each category. For all head-to-head comparisons, we use the same training data across baselines within the respective category.
For baselines, we consider a BT RM and two CoT-GenRMs human-written CoT rationales from HelpSteer2- Prrefeeer-ee,,eeeneee, denoted as CoT-GenRM (ground truth), and the other trained on explicit CoT rationales generated by QwQ-32B (i.e.. excluding the internal <THINK>.., </THIKK> reasoning), denoted as CoT- CoT-GenRM (model-genee generated). We emphasize that CoT-GenRM (ground truth) a strong GenRM baseline, as collecting numan-writter CoT rationales and further postprocessing them by expert researchers is prohibitively expensive. In addition, we scaling: for each sample, the model
one trained on ground-tuuth
evaluate these CoT-GenRMs under vertical inference-tine generates independent judgments, and the final prediction is determined by majority voting. This serves as a point of comparison to Think-RM, which adopt different inference time scaling approach based on long-horizon inference scaling.
To compare pairwise RLHF with traditional pointwise RLHF, we integrate different RMs into their respective RLHF frameworks and evaluate the end-polic performance. Specifically, we use BT RM for pointwise RLHF and GenRMs (CoT-GenRM and Think- RMM for our pairwise RLHF We conduct these experiments on HH-RLHF dataset 11, using 3K randomly sampled prompts for training.
Base Models. We use Llama-3. 1 1-8B-Instruct and (0wen.55-3B-Instruue Instrue as backbone models for all experiments. We choose these small-sized models because integrating larger ones into a full pairwise RLHF pipeline prohibitively expensive in terms of computation and memory. Due to space constraints, we defer experiment results with S.ven..5-BB-Instree to Appendix B
Evaluation Benchmarks. We use CellsSteer3-Prffeeeeee 49 benehmark to evaluate generaliza- tion under moderate distribution shift. Although it shares similar prompt sources and response pair generation methods with HelpSSeerr-Prefeeeee Preferenne, which we use for ID evaluation via its validation set, HelpSteer3-PPeffernnc includes more diverse and challenging examples that go beyond ID) settings. For OOD evaluation, we consider two additional benchmarks: RewardBench (31) and RM-Bench 32. RM-Bench is specifically designed to evaluate robustness to subtle content variations and resis- tance stylistic biases. It is widely regarded as one of the most challenging benchmarks for RMs, requiring fine-grained judgment and extensive reasoning. For evaluating end-policy performance after RLHF training, we use AlpacaBvva22 [50] with GPT-4-as-a-judge.
Implementation Details. We use OpenRLHF 51 to train BT RM and all SFT models, and VeRL [52 for all RL experiments (hhnnk-RM's rule-based RL stage and pairwise RLHF with GenRMs).
For warm-up SFT, we fine-tune .lama-3.1-88 1.B--...------BUUUB S-nttt---Ituttttte for 5 epochs with learning rate of 1 x 10 5 for binary outputs and 5 x 10 6 for multiclass outputs. We fine-tune Dwen2.5-3B Instruct for 10 epochs with a learning rate of x 10-5 for binary outputs. At the based RL stage, we use a rollout batch size of 512, KL coefficient of / 1 x 10-4, and a group size of G = 8 for both binary and multiclass settings. The learning rate is x 10-6 for Llama-3. 1-88- 1-8B-Instruct and 1 x 10-6 for Dwwen.55-3BIInstrut
For RLHF experiments, we use the same hyperparameters as in rule-based RL, except we reduce the group size to G = 4. Additional implementation details are provided in Appendix C
4.2 Main Experiments
4.2.1 Evaluation on In-Distribution and Moderately Shifted Tasks
In [ we report the preference accuracy of different RMs on ID SeelsSteer2-Prffernnee and moderately shifted SeelpSteer3-PPefeeenne tasks. Since the models are trained with 6K examples for binary cases and 4K for multiclass cases, binary models generally achieve higher accuracy across all subdomains. Given the limited training data, BT RM underperforms compared to GenRMs (CoT-GenRM and Think-RMM, even on the ID task, highlighting the sensitivity of BT RM to data
http::///uuggnngffs..e////ttttes http:s///uugglnggaaee..o////////
3.18 Instruct
Instruct
7

---

Table I: Reward model evaluation on SoisSteer--Prfeerenn (in-distribution) and HelpSteer3- Preference (moderate distribution shift). Bolded numbers indicate the best performance within each type, and underlined numbers indicate the second best.
Type
Model
HelpSteer2-Preie Prefferncc
HelpSteer3-Pteferene
Validation
Arg. Len
Code
General
Muitiingual
Stem
AYG
Avg. Len
67.47
354.01
67.48
64.43
66.52
62.96
65,29
353.01
75.48
71.88
CoT GenRM (ground (ruth)
80.97
97.98
75.23
69.84
75:45
69.95
72.03
100.93
CaT-GGolR (moddl- gemeralel
76,14
583.33
78,70
69.29
76.21
64.61
72.01
411.29
Binary
CoT-GenRM M (ground truth)
w/ vertical infereace time scaling
80.11
1561.76
75.00
70.11
75.30
70.58
72.16
1596.8
GoT-GenRM (model generated)
w/ vertical inference-inn time scaling (m = 16)
77.56
6181.76
78.12
69.51
76.21
66.26
72.19
6529.26
Think-RM (SFT)
74.86
1335.80 80
76.04
68.90
76.52
66.67
71:48
1587.74
Think-RM (SPT RL.)
81.53
1018.76
80,32
70.71
75.15
67.90
73.28
1124.98
351.76
56.13
57.21
59.85
58.85
57.63
353.10
CoT-GenRM (ground truth)
78.12
98.32
78.12
68.12
74.55
66.67
71:43
CdT-GeaRMM gomoel-gmerrllll)
73.30
379.16
77.20
68.52
73.94
69.14
71,48
390.16
Multidaes
CoT-GenRR (ground Iruih)
inference-time scaling
78.84
1492.64
78.59
67.76
75.00
66.05
71.22
1532.8
CoI-GGenRM generated)
w/ vertical inference time scaling (m = 10)
75.43
6064.8
79.17
69.18
74.24
67.08
72.03
6280.8
Think-RN RM (SFT)
74.86
1573/41
72:15
65:11
72.12
63.58
67.92
1665.39
Think-RM (SFT RL.)
76.70
1184.26
78.36
67.49
75.30
68.52
71.41
1333.08
Table Reward model evaluation on RewardBench. Bolded numbers indicate the best performance within each type, and underlined numbers indicate the second best.
Reward Bench
Type
Model
Chat
Chat Hard
Reasoning
Safety
AVG
Avg. Len
Base
89.53
45.18
68
77.23
70.95
381.56
BT RM
88.32
66.89
80.71
76.35
78.07
CoT-GenRM (ground truth)
93.85
66.01
76.29
87.97
80.79
113.10
GoT-GenRMM (model- generrted)e
93.85
62.06
73.24
85.14
78.26
453. 10
Binary
CoT-GenRM (ground truth)
93.02
65.57
79.33
87.43
81.81
1596.8
w/ vertical inference- scaling (m 16)
CoT-GenRM (model- geneerted)))))
95.25
63.71
74.33
84.66
79.28
6767.2
w/ vertical inference time scaling (m 16)
Think-RM (SFT)
94.97
75.44
85.03
84.86
85.4 85.44
2267.46
Think-RM (SFT RI.
94.41
77.85
85,23
86.35
86.35
1422.93
Base
69.13
48.14
63.46
58.51
61.42
358.82
BT RM
92.25
66.01
80.83
75.14
78.56
CoT-GenRM (ground truth)
95.95
64,25
75.50
87.03
80.72
115.77
CoT-GenRM (model-generrteed)
94.69
65.02
75.34
85.95
79.55
399.88
Multiclass
CoT GenRM (ground truth) w/ vertical inference-ting scaling (m 16)
95.53
63.27
76.13
85.47
80.37
1529.12
CoT-GenRM genette-geerutted)
w/ vertical inference-time scaling (m = 16)
95.81
63.38
74.92
87.16
80.39
5820.64
Think-RM (SFT) Think-RM (SFT RL)
90.78
76.21
81.11
84.73
83.17
2514.14
94.27
75.33
82.11
86.35
84.49
1635.57
size and coverage, as well as the robustness of GenRMs in low-data regimes. CoT-GenRM (ground truth) outperforms CoT-GenRM (model-generated) on the ID task due to the higher quality of human-written rationales, but their performance becomes comparable under moderate distribution shift. Vertical inference-time scaling using majority voting over 16 judgments provides minimal improvement. Think-RM, trained with SFT followed by RL, significantly outperforms its SFT- only counterpart and reduces the average response length, indicating a refinement of the long and noisy reasoning trajectories introduced during SFT warm-up. This underscores the essential role of the subsequent RL stage in improving ID performance while pruning redundant and verbose reasoning steps. Notably, binary Think-RM outperforms baselines on both ID and moderately shifted settings, even surpassing vertically scaled CoT-GenRM (ground truth). For the multiclass case, Think-RM performs slightly worse than GoT-GenRM (ground truth) but still outperforms or matches CoT-GenRM imodel-generated) even when vertical inference-tine scaling is applied. In the sasooning-hhavy code domain of TeenStter3--Pffeeemee binary Rh achieves the highest accuracy among all baselines, demonstrating its effectiveness complex reasoning tasks.

---

Table 3: Reward model evaluation on RM-Bench. Bolded numbers indicate the best performance within each type, and underlined numbers indicate the second best.
R.M-Bench
Type
Model
Chat
Code
Math
Safety
AVG
Avg. Len
Base
49.10
51.19
57.10
82.63
63.79
344.40
BT RM
61.11
53.9
59.48
88.33
68.27
CoT-GenRM (ground truth)
60,90
51.88
59.30
88.21
67.79
102.39
CoT-GenRMM (model-generuteed)
59.35
52.05
58.31
88.94
67.51
400,80
Binary
CoT-GenRM (ground truth)
60.16
53.61
59.15
88.69
68.11
1534.72
w/ vertical inference- time scaling (m 16)
CoT-GenRM (model-genael -geerrtd)))
w/ vertical inference-tine scaling
59.73
53.27
60.42
88.78
68.55
6340.8
Think-RM (SFT)
67.92
56.29
71.73
91.23
75.19
3228.61
Think-RM (SFT+ RL)
66.41
54.68
72.25
91.51
75.06
1798.86
Base
52.28
51.97
53.81
62.35
56.18
334.29
BT RM
62.05
55.58
57.93
82.92
66.28
CoT-GenRM (ground truth)
60.38
53.12
61.32
88.52
68.86
104.16
CoT-GGeRRM (model-generated)
60.98
53.31
60.01
89.70
68.82
420.19
Multiclass
CoT-GenRM (ground truth)
$9.04
53.05
61.06
88.93
68.75
1545.92
w/ vertical inference- time scaling (m
CoT-GenRM
61.28
54.19
60.40
90.31
69.36
6348.48
w/ vertical inference- time scaling (m 16)
Think-RM (SFT)
64.51
$2.00
66.99
90.38
71.95
3092.62
Think-RM (SFT + RL)
64.17
51.07
67.51
91.62
72.37
1690.38
4.2.2 Evaluation on Out-of-Distribution Tasks
In Tables 7 and we report the preference accuracy of different RMs on OOD tasks (RewardBench and RM-Bench). Notably, the Chat Hard and Reasoning subcategories of RewardBench, well as all domains in RM-Bench, require extensive reasoning. From the tables, we observe that Think- RM significantly outperforms all baselines
these OOD tasks for both binary and multiclass settings, achieving average improvements up to 5% on RewardBench and 8% on RM-Bench, even compared CoT-GenRM (ground truth) with vertical inference-time scaling using 16 judgments. In particular, Think-RM achieves improvements of more than 10% and 5% the Chat Hard and Reasoning subcategories, respectively, and 12% improvement in the Math domain of compared to CoT-GenRMs. These results demonstrate that long- long-horizon reasoning through internal thinking processes outperforms vertical inference scaling based structured external reasoning when solving complex, reasoning-intensivee tasks. Think-RM, trained with SFT followed by RI., generally outperforms its SFT-only counterpart and reduces the average response length, consistent with our observations in Section 4.2.1
4.3 Ablation Study
Figure presents the preference accuracy and average response length of binary Think-RM across all benchmarks, comparing two warm-up data selection strategies: using the longest versus the shortest CoT per instance. As shown, training with the longest CoT data consistently achieves higher preference accuracy across all benchmarks, demonstrating the effectiveness of length-based CoT filtering for enhancing reasoning quality. However, this comes at the cost of increased response length, indicating trade-off between accuracy and inference efficiency in selecting warm-up CoT data strategies.
4.4 Pairwise vs. Pointwise RLHF
Table Ashows the end-policy performance of models trained using two different RLHF approaches: traditional pointwise RLHF with BT RM and our proposed pairwise RLHF with GenRMs (CoT- GenRM and Think-RM), Note that we reduce the number of parallel runs to m for CoT--GeRM with vertical inference scaling so that its average response length matches that of Rhink-RM, since CoT-GenRM with vertical scaling (m = 16) generates 3-4 times more tokens than Think- RM in
Tables I and 3

---

90.0
2000
Longest-Cal Shortest-CoT
Longest-Col Shortest-CoT
1800
06.3235.96
1600
85.0
82.5
61.53
Avg. Response Length
Accuracy
2200
78.69
1000
1.019
77.5
800
730
/3.87
72.5
72.21
400
70.0
200
HelpSteer2
HelpSteer3
RewardBenzch
RM Bench
HelpSteer2
HelpSteer3
RewardBench
RM-
Figure 3: Comparison of two CoT filtering strategies for warm-up data selection.
As shown, policies trained with the pairwise RLHF + GenRM pipelines outperform those trained with pointwise RLHF BT RM, demonstrating the effectiveness of our approach. The similar cength-controlled win rate (LC WR) between the pairwise RLHF policies using CoT-GGeRMM and Think-RA RM can be attributed their comparable preference accuracy the HH-RLHF test set (64.42 for CoT-GenRM vs. 65.03 for Think-RM), likely due to the relatively easy nature of the prompts in this dataset. Nevertheless, the policy trained with Think-RM achieves higher overall win rate (WR) than the one trained with CoT-GenRM.
Table Comparison of pointwise and pairwise RLHF approaches using different reward models.
Model
AlpacaEval2 WR
LC WR
Avg. Len
Base
24.54
31.11
2296
Pointwise RLHF with BT RM
27.49
33.14
2300
Pairwise RLHF with CoT-GenRM (ground truth)
31.85
40.30
2430
w/ vertical inference scaling (m 4)
Pairwise RLHF with CoT-GenRM (model-generatee
31.72
41.06
2408
w/ vertical inference scaling (m 4)
Pairwise RLHF with Binary Think-RM
31.56
47.20
2838
Pairwise RLHF with Multiclass Think-RM
31.94
42.68
2574
5 Conclusion and Future Work
We introduced Think-RM, framework for training LLMs as GenRMs with long-horizon reasoning capabilities. To elicit advanced reasoning skills such self-reflection and hypothetical reasoning. we applied SFT on long CoT data, followed by rule-based RL. We also proposed a pairwise RLHF pipeline that directly leveraged pairwise preference comparisons, removing the need for pointwise reward conversion. Experiments showed that Think-RM outperformed both BT RM and vertically scaled CoT-GenRMs, and achieved superior end-policy performance when combined with our pairwise RLHF approach. Overall, framework provided depth-oriented approach for GenRM design and established a new paradigm for preference-baie RLHF.
Future research could explore two potential directions beyond Think-RM, -M.. First, developing calibrated Think RM that produces continuous measures preference strength, rather than discrete binary or multiclass outputs, could enable more fine-grate fine-grained supervision signals for policy optimization. Sec- ond, investigating adaptive reasoning budgets, where Think-MM RM dynamically allocates reasoning depth based on task complexity (using shorter reasoning chains for simple tasks and deeper ones for challenging problems), could yield reasoning-efficient Think-RM that maintains high fidelity while substantially reducing inference latency and computational cost.
10

---

References
[1] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2xXi:2204.05880 2022.
[2] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, (1277730-277444 2022.
[3] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated in- structions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484-13508, 2023.
[4] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[5] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14277 2022.
[6] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike. John Schulman, llya Sutskever. and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
[7] Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai. Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint 22..::::322200900 2023.
[8] Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint 240304440..44444, 2024.
[9] Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via a human- human-preference preference dataset. In Thirty- Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[10] Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. In The Twelfth International Conference Learning Representutions, 2024.
[11] Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety, arXiv preprint arXiv:2411.011
2024.
[12] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 30, 2017.
[13] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize from human feedback. Advances Neural Information Processing Systems, 2020.
[14] Ilgee Hong, Zichong Li, Alexander Bukharin, Yixiao Li, Haoming Jiang, Tianbao Yang, and Tuo Zhao. Adaptive preference scaling for reinforcement learning with human feedback. Advances in Neural Information Processing Systems, 37:107249-107269 2024.

---

[15] Alexander Bukharin. Iggee Hong, Haoming Jiang, Zichong Li, Qingru Zhang, Zixuan Zhang, and Tuo Zhao. Robust reinforcement learning from corrupted human feedback. Advances in Neural Information Processing Systems, 37:112003-122113 2024.
[16] Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' Amour, DJ Dvi- jotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244, 2023.
[17] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023.
[18] Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexan- dre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot. et al. Direct language model alignment from online ai feedback. arXiv preprint 2rX1v:2402.04792, 2024.
[19] Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, et al. Rrm: Robust reward model training mitigates reward hacking. arXiv preprini arXiv:2409. 13156, 2024.
[20] Zihuiwen Ye, Fraser Greenlee-Scott, Max Bartolo, Phil Blunsom, Jon Ander Campos, and Matthias Gallé. Improving reward models with synthetic critiques. arXiv preprint arXiv:2405.208800 2024.
[21] Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. Critique-out-lour reward models. arXiv preprint arXiv:2408. 11791, 2024.
[22] Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410. 12832, 2024.
[23] Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, et al. Self-generated critiques boost reward modeling for language models. arXiv preprint
2024.
[24] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In The Thirteenth International Conference on Learning Representations, 2025.
[25] Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling, arXiv preprint arXiv:2504.02490 2025.
[26] OpenAl. Learning to reason with llms. OpenAl Blog, 2024.
[27] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in IIms via reinforcement learning. arXiv preprint arXiv:2501.12945: 2025.
[28) Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint SttI::2501.199933 2025.
[29] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[30] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint 2rXi /783, 2024.
[31] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling, arXiv preprint 24Xiv:2403.13798 2024.
12

---

[32] Yantao Liu, Zijun Yao. Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. RM-bench: Benchmarking reward models of language models with subtlety and style. In The Thirteenth International Conference Learning Representations, 2025.
[33] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[34] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[35] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 15115, 2024.
[36] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 466233, 2023.
[37] Hui Huang, Xingyuan Bu, Hongli Zhou, Yingqi Qu, Jing Liu, Muyun Yang, Bing Xu, and Tiejun Zhao. An empirical study of llm-as-a-judge for I1m evaluation: Fine-tuned judge model not a general substitute for gpt-4, arXiv preprint arXiv:2403.022 000.0099.0 2024.
[38] Qingchen Yu, Zifan Zheng, Shichao Song. Zhiyu Li, Feiyu Xiong, Bo Tang, and Ding Chen. xfinder: Robust and pinpoint answer extraction for large language models. arXiv preprint arXiv:2405. 11874 2024.
[39] Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. Chain of thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
[40] Xuezhi Wang, Jason Wei, Dale Schuurmans, et al. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. //171, 2022.
[41] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36:1 1809-11822, 2023.
[42] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211. 125888 2022.
[43] OpenAl. Introducing openai o3 and 04-mini. OpenAl Blog, 2025.
[44] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwent github. i h/blo////////oo/ooo/oooo 3uq-
[45] xAI. Grok 3 beta - the age of reasoning agents. xAI Blog, 2025.
[46] Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. Helpsteerr2prrfeeeeee ences. In The Thirteenth International Conference on Learning Representations, 2025.
Complementing ratings with prefer-
[47] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint orXiv:1707.066 2017.
[48] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint 1r0iv 02438, 2015.
[49] Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Daniel Egert, Ellie Evans, Moo-Chang Shin, Felipe Soares, Yi Dong, and Oleksii Kuchaiev. Dedicated feedback and edit models empower inference- time scaling for open-ended general-domain tasks. arXiv preprint arXiv:2503.0433 2025.
[50] Yann Dubois, Percy Liang, and Tatsunori Hashimoto. Length-contron alpacaeval: A simple debiasing of automatic evaluators. In First Conference on Language Modeling, 2024.
13

---

[51] Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint 1rX1v.2405.11143, 2024.
[52] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu, Hybridflow: flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
[53] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69800 2014.
[54] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv1171..05101, 2017.
14

---

A Task Instruction Template
Binary Think-RN RM
System:
You are an impartial judge, tasked with evaluating the quality of the two AI assistants' responses to the context displayed below. Your evaluation should be based on the following six criteria:
Helpfulness: Overall helpfulness of the response to the user's question or instruction.
Correctness: Inclusion of all pertinent facts without errors.
Coherence: Consistency and clarity of expression.
Complexity: Intellectual depth required to write response (i.e., whether the response can be written by anyone with basic language competency or requires deep domain expertise).
Verbosity: Amount of detail included in the response, relative to what is asked for in the context.
Safety: Whether the response is free of any kind of harmful, toxic, or illegal content.
After carefully considering these criteria, determine which assistant's s response is superior. Begin your evaluation by thinking through the problem step by step. Then output your final verdict by strictly following this format: <answer>As/answerr if assistant A is better, and <answer9B</answer> assistant better.
User:
The Start of Context]
(context) The End of Context]
The Start of Assistant A's Response] [response1}
The End of Assistant A's Response]
The Start of Assistant B's Response] [response2]
The End of Assistant B's Response]
15

---

Multiclass Think-RM
System:
You are an impartial judge, tasked with evaluating the quality of the two AI assistants' responses to the context displayed below. Your evaluation should be based on the following six criteria:
Helpfulness: Overall helpfulness of the response to the user's question instruction.
Correctness: Inclusion of all pertinent facts without errors.
Coherence: Consistency and clarity of expression.
Complexity: Intellectual depth required to write the response (i.e., whether the response can be written by anyone with basic language competency or requires deep domain expertise).
Verbosity: ty: Amount of detail included in the response, relative what is asked for in the context.
Safety: Whether the response free of any kind of harmful, toxic, or illegal content.
After carefully considering these criteria, determine which assistant's s response is better and how much better it is using the scale below:
-3 if Assistant A's response is much better than Assistant B's response
-2 if Assistant A's response is better than Assistant B's response
-1 if Assistant A's response is slightly better than Assistant B's response
1 if Assistant B's response is slightly better than Assistant A's response
if Assistant B's response is better than Assistant A's response
if Assistant B's response is much better than Assistant A's response
Begin your evaluation by thinking through the problem step by step. Then output your final score inside the <answer></answer> tag.
User:
(Th Start of Context] (context)
The End of Context]
The Start of Assistant A's Response] [response1}
The End of Assistant Response]
The Start of Assistant B's Response] [response2}
The End of Assistant B's Response]
16

---

B Additional Experiments with Qwen2.5-3B-Instteee
Table 5: Reward model evaluation on HelpSteer2- Prrffernce (in-distribution) and HelpSteer3-
Preference (moderate distribution shift). Bolded numbers indicate the best performance within each type, and underlined numbers indicate the second best.
HddpSteer2-
HelpSteer3
Type
Model
Validation
Avg. Len
Code
General
Mulhilingual
Siem
AVG
Avg. Len
Base
53.81
262.74
62.27
53.61
51.24
53.70
55.68
365,19
71.01
CoT GoT-GenRM (model generllgenerrteel
Binary
w/ vertical inference- time scaling (m 16)
74.43
6033 44
76.16
67.9%
73.94
63.99
70.34
6209.28
Think-RM RM (SFT)
71:31
1061
73.15
65.74
74.70
67.49
69.17
1230.71
(SFT
75.99
67.16
71.06
69.87
Table 6: Reward model evaluation on RewardBench. Bolded numbers indicate the best performance within each type, and underlined numbers indicate the second best.
Reward Bench
Type
Model
Chat
Chat Hard
Reasoning
Safety
AVG
Avg. Len
Base
73.74
48.03
60.52
69,86
63,60
321.59
BT RM
87.43
62.50
73.74
75.95
74.90
CoT-GenRMM (moxee--generated)
Binary
w/ vertical inference time scaling (m 16)
94.27
56.47
71.16
79.53
75.18
$464.96
Think-RM (SFT)
91.76
62.28
72.06
82.64
76.28
1844.35
Think-RM (SFT + RL)
93.58
62,83
75.77
81,96
78.56
1172.42
Table 7: Reward model evaluation on RM-Bench. Bolded numbers indicate the best performance within each type, and underlined numbers indicate the second best.
RM-Bench
Type
Model
Chat
Code
Math
Safety
AVG
Avg. Len
Base
55.86
51.12
53.12
73.75
59.90
319.84
RM
54.61
54.19
58.52
70.09
61.24
CoT-GenRM -ennerl-e
Binary
vertical inference-time scaling 16)
59.86
51.17
59.20
82.51
65.63
5677.28
Think-RM (SFT)
62.88
$1.85
64.59
87.30
69.78
1884.25
Think-RM (SFT + RL)
60.42
52.73
66.54
87.05
70.39
1169.78
In Tables 6 and we report the preference accuracy of different RMs on in-distribution (ID;
JripStter2-Prefereneee moderately shifted HelpsSteer3PPrefereee RewardBench, RM-Bench) tasks. Given the limited training data in our experimental setup, BT RM underperforms both vertically scaled CoT-GenRM (model-generated) and Think-RM across all benchmarks, highlighting its sensitivity to data size and coverage. Think-RM, trained with SFT followed by RL, outperforms its SFT-only counterpart (especially the ID task) and significantly reduces the average response length, underscoring the essential role of RL training in improving ID performance while pruning redundant and verbose reasoning steps. Notably, Think-RM outperforms all baselines on both ID and OOD settings and achieves comparable performance on the moderately shifted task. In the reasoning-heas domains of RM-Bende BeMBennc), Think-RM achieves the highest accuracy among all baselines, showing more than 7 improvement in the Math domain of RM-Bench further demonstrating its effectiveness on complex reasoning tasks. These results are consistent with our observations in Section 4.2 using Llama-3. --B--uust
and out-of-distribution (OOD;
tasks (e.g., Chat Hard and Reasoning from RewardBench, and all
17

---

C Additional Implementation Details
Training. We train Think-RM using eight A100 GPUs (1 node), each with 80GB of memory. The warm-up SFT phase takes approximately one hour, while the rule-based RL phase takes about three hours. For pairwise RLHF training, we use sixteen A100 GPUs nodes), each with 80GB of memory: one node is allocated for RL training and the other for GenRM inference, For warm-up SFT, we use the Adam optimizer [53] with 31 = 0.9 and 32 = 0.95, which are the default settings in OpenRLHF [SI], and tune the learning rate from (5 x 10-6 x 10-5). We apply cosine learning rate scheduler with a warmup ratio of 0.03. The number epochs is tuned over (1,3, 5, 10), and we set the batch size to 256 and the maximum sequence length to 16,384 tokens.
For rule-based RL, we use the AdamW optimizer [54] with 31 0.9 and B2 0.999, following the default settings in VeRL [52]. We tune the learning rate in {1 x 10 x 10 10-6) and use constant learning rate scheduler with no warmup (warmup ratio 0). We tune the number of epochs over {1, 2], the rollout batch size over {256, 512]. and set the training batch size to 128. The maximum prompt and response lengths are both set to 4,096 tokens. We use KL coefficient of 3 1c-4 and group size G = 8.
For baselines (BT RM and CoT-GenRM), tune the number of epochs over 21,2,3,5, 7, 7, and the learning rate over (5 x 10-6, 1 x 10-5), and set the batch size to 256. The maximum sequence length is set to 8.192 tokens.
All selected hyperparameters are summarized in Tables
g and
Table Summary of selected hyperparameters across binary and multiclass setups with Llama-3.1- 8B-Instruct.
Type
Model
Num Epochs
LR
Rollout Batch Size
Training Batch Size
Think-R RM (SFT)
1 x 10
256
Think-RM (RL)
2 10-6
512
128
Binary
CoT-GenRM (ground truth)
1 X 10
256
CoT-GenRM (model-genm (mneeelgenerated)
10
1 10-5
256
BT RM
1 10
256
Think-RM (SFT)
5 10-6
256
Think-RM (RL)
2 10-6
512
128
Multiclass
CoT-GenR (ground truth)
1 1 10
256
C (model-ggeerrated)
10
5 10
256
BT RM
1 x 10-5
256
Table 9: Summary of selected hyperparameters for binary setup with Qwen2.5-3B-Instruce
Type
Model
Num Epochs
LR
Rollout Batch Size
Training Batch Size
Think RM (SFT)
10
x 10-5
256
Binary
Think-RM (RL) CoT-GenRM (model-generated))
x 10-6
512
128
1 10-5
256
BT RM
2
1 10-5
256
For pairwise RLHF experiments with GenRMs, reuse all hyperparameters selected for the rule- based RL setup, as listed in Table For policy rollout, we set the temperature to 1.0, with maximum prompt and response lengths of 1,024 and 2,048 tokens, respectively. For GenRM inference, use a temperature of 0.6 and generate up to 2,048 response tokens. To reduce computational cost, we set the group size G 4.
Evaluation. For the inference of GenRMs in Tables 22 and we use temperature of 0.6, top-p of 1.0, and maximum response length of 16,384 tokens. These settings are applied to both Think-RM and CoT- Ro..GenMM. For inference with the RLHF-trained models reported in Table we use temperature of 0.6, top-p of 0.9, and maximum response length of 4,096 tokens.
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claim in the abstract and introduction accurately reflects the paper's contributions, particularly in enabling long-horizon reasoning in generative reward models for addressing nuanced and complex reward modeling tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [NA]
We did not identify significant limitations in our approach, and due to space constraints, prioritized presenting core contributions and results.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [NA]
Justification: The paper primarily introduces a training framework to develop a specific capabilityy, and therefore does not present any formal theoretical results requiring assumptions or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All selected hyperparameters for both the proposed and baseline methods are summarized in Appendix C In addition, we detail the procedures used for hyperparameter tuning.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We will include the full codebase along with detailed reproduction instructions in the README files, as part of the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httppp.ccppubcccguuuceeee Codeeubmii ssi criionPooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All necessary training and evaluation details are provided in Section 4.1 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We not report statistical significance measures due limited computational resources, which constrained the number of experimental runs. However, we ensured consistency by using fixed random seeds and uniform evaluation protocols across all models.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The necessary information on computational resources is provided in Ap- pendix C
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't i make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::/neerripp.cc/pbl/ll////// (etbuliieeeEhhicccccceeeeeeeee
Answer: [Yes]
Justification: This research adheres to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction)
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is advance the field of Machine Learning. There are many potential socictal consequences of our work, none of which we feel must be specifically highlighted.
Guidelines:
22

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This is a methodology paper focused on reward modeling for RL training of LLMs. We do not release any models datasets that pose risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The used dataset/codd/mooels are properly credited in experimental settings.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We will provide thorough documentation for the new assets alongside their release if the paper is accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurips hett::///earrrpssccc//ooerrreeee cc///oofeeeeeeee//22////... for what should or should not be described.
25


Reviews:
[{"review_id": "meP13AIpZ6", "reviewer": "Reviewer_Lj4C", "summary": "This paper proposes a new training framework for the Generative Reward Model. In many benchmarks, Think-RM shows an improvement over other reward models. The paper also compares pairwise RLHF with pointwise RLHF, which shows a more effiective performance.", "questions": "Could the authors clarify the difference between the SFT phase of Think-RM and the COT-GenRM training, as their training objectives appear similar?\n\nIn Table 4, the performance gap between GenRM and Think-RM on HH-RLHF is minimal and requires further explanation.  And authors claim that the WR of Think-RMs is higher than COT-GenRMs, but LC is a more important metric in  AlpacaEval2.\n\nWhy is the number of independent judgments in Table 4 inconsistent with those in other tables?", "limitations": "The paper does not propose specific mechanisms to detect or correct hallucinated reasoning chains, which are common failure modes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\nThe writing of this paper is clear and easy to follow.\nThe paper evaluates the proposed method accorss many benchmarks, including HelpSteer2-Preference, HelpSteer3-Preference, RewardBench, RM-Bench and AlpacaEval 2.\n\nWeakness:\nThe model trained with Think-RM demostrates a longer response length in all tables, which may indicates more computational and time cost.\nIn Table 1 and Table 4, there are marginal improvements between Think-RM and COT-GenRM.", "was_revised": true, "final_justification": "The concern about computational and time costs, marginal performance improvements of Think-RM over CoT-GenRM in Table 1 and Table4 have been addressed.", "rebuttal": "> W. The model trained with Think-RM demostrates a longer response length in all tables, which may indicates more computational and time cost. In Table 1 and Table 4, there are marginal improvements between Think-RM and COT-GenRM.\n\n\nWe appreciate the reviewer's observations regarding the computational cost and marginal performance improvements of Think-RM over CoT-GenRM in Tables 1 and 4.\n\n### 1. Regarding the concern about computational and time costs.\n\nWe would like to clarify that **the longest response lengths across all tables are produced by CoT-GenRM with vertical scaling**, which generates 3x to 4x more tokens than Think-RM during inference. Despite this, **Think-RM outperforms CoT-GenRM with vertical scaling on most benchmarks, achieving 10%, 5%, and 12% improvements on reasoning-heavy tasks**, such as the Chat Hard, Reasoning, and Math subcategories of RewardBench and RM-Bench.\n\nWhether along the vertical or horizontal axis, inference-time scaling for solving complex problems is an active area of research across various LLM applications, including reasoning LLMs, tool-augmented LLMs, self-critique LLMs, and multi-agent debate (MAD). In this line of research, **we are the first to show that horizontally scaling current CoT-GenRMs further improves the quality of preference outputs, even without generating multiple CoT paths, and to demonstrate how such models can be trained**.\n\nAlthough horizontal inference-time scaling introduces additional computational and time costs compared to models of the same size, **this horizontally extended CoT yields significantly better performance**. Moreover, we partially address this limitation by applying RL training to Think-RM (SFT) to eliminate verbose or redundant reasoning steps, reducing response length while preserving only the essential steps. Without inference-time scaling, **achieving similar performance would require scaling the model size substantially, which is far less efficient and still struggles with reasoning-heavy tasks like RM-Bench**, as shown below:\n    \n| Model                            | RewardBench | RM-Bench   | \n|----------------------------------|-------------|------------|\n| Llama3.1-8B-Instruct             |  71.0       | 63.8       |\n| Llama3.1-70B-Instruct            |  84.0       | 65.5       |\n| Think-RM-Llama3.1-8B-Instruct    |  **86.3**   | **75.1**   |\n\n### 2. Regarding the concern about marginal performance improvements of Think-RM over CoT-GenRM in Table 1.\n\nFor in-distribution tasks (Table 1), since the problems exhibit recurring patterns similar to the model's training examples, pattern matching or shallow reasoning is often sufficient to produce correct preference outputs. Consequently, BT-RM and CoT-GenRMs have an inherent advantage on these tasks. We address this by introducing RL training (stage 2 in Think-RM), where **additional RL fine-tuning on the same prompt set from the SFT stage significantly improves performance on in-distribution tasks** while maintaining similar performance on out-of-distribution (OOD) tasks. **For example, in Table 1 (binary case), Think-RM (SFT + RL) improves by 7% over Think-RM (SFT) and by 4% in the table provided in our response to Reviewer BTue, which uses a different backbone model (Qwen2.5-3B-Instruct)**, performing competitively with or outperforming BT-RM and CoT-GenRMs even on in-distribution tasks. Therefore, we argue that the performance gains, though seemingly marginal in Table 1, are a strength of the proposed method.\n\n### 3. Our response regarding Table 4 is in our response on reviewer's Q2 below.\n\n\n> Q1. Could the authors clarify the difference between the SFT phase of Think-RM and the COT-GenRM training, as their training objectives appear similar?\n\nWhile the training objectives are indeed the same, the SFT phase of Think-RM is anotherly described as **reasoning distillation** [1, 2]. The key distinction lies in the training data used.\n\nIn current reasoning LLMs, given an input $x$, the model typically generates an internal thinking trace $r$, followed by a final explicit CoT response $y$. Think-RM is trained on the pair $(x, r)$, focusing on learning internal thinking process described in Figure 1 in the manuscript. In contrast, CoT-GenRM is trained on $(x, y)$, focusing on learning external CoT traces.\n\n*References*\n\n*[1] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" arXiv preprint arXiv:2501.12948 (2025).*\n\n*[2] Kang, Minki, et al. \"Distilling llm agent into small models with retrieval and code tools.\" arXiv preprint arXiv:2505.17612 (2025).*\n\n> Q2. In Table 4, the performance gap between GenRM and Think-RM on HH-RLHF is minimal and requires further explanation. And authors claim that the WR of Think-RMs is higher than COT-GenRMs, but LC is a more important metric in AlpacaEval2.\n\nAs noted in Line 313 of the manuscript, CoT-GenRM and Think-RM achieve nearly identical preference accuracy on the HH-RLHF test set (64.42 vs. 65.03). This indicates that HH-RLHF is not sufficiently discriminative to reveal the full advantage of Think-RM over CoT-GenRM, which explains the marginal improvement reported in Table 4. Although LC WR is indeed emphasized in AlpacaEval2, WR is still a critical complementary metric. On this metric, Think-RM significantly outperforms CoT-GenRM despite the slight difference in HH-RLHF test set accuracy. Moreover, we would like to emphasize that our contribution is twofold: **1) Think-RM training methodology**, and **2) Pairwise RL approach** that enables seamless integration of GenRMs into RLHF pipelines. **The primary purpose of Table 4 is to demonstrate the effectiveness of our pairwise RLHF method compared to traditional pointwise RLHF with scalar rewards**. As shown in Table 4, our pairwise RLHF framework significantly outperforms traditional pointwise RLHF with BT models, bridging the gap between expressive GenRMs and practical RLHF pipelines.\n\n> Q3. Why is the number of independent judgments in Table 4 inconsistent with those in other tables?\n\nFor the reward modeling benchmarks in Tables 1-3, we use 16 independent runs to align with prior work [1, 2] on vertical test-time scaling, which typically uses values greater than 8. However, in Table 4, we keep the total number of generated tokens during inference roughly comparable between CoT-GenRM and Think-RM due to the computational bottleneck. **Since CoT-GenRM with vertical scaling (m=16) generates 3–4 times more tokens than Think-RM in Tables 1-3, we reduce the number of runs to m=4**. We will make this clarification in the updated version of the manuscript.\n\n*References*\n\n*[1] Yu, Yue, et al. \"Self-generated critiques boost reward modeling for language models.\" NAACL 2025 (2024).*\n\n*[2] Liu, Zijun, et al. \"Inference-time scaling for generalist reward modeling.\" arXiv preprint arXiv:2504.02495 (2025).*"}, {"review_id": "malXpPNQ4T", "reviewer": "Reviewer_BTue", "summary": "The paper addresses a key issue in RLHF: creating reliable reward models, especially when data is scarce or tasks need deep reasoning. Conventional BT reward models are easy to train but often over-fit and are susceptible to reward hacking. Existing generative reward models (GenRMs) have short reasoning and produce pairwise preferences that don't fit well with standard RLHF pipelines.​\n\nThe authors present Think-RM, a generative reward-modeling framework that gives language models long-horizon, self-directed \"internal thinking\". It generates flexible, multi-thousand-token reasoning traces with self-reflection, hypothetical branching, and divergent analysis, solving the shallow reasoning problem of previous GenRMs.​\n\nTests show that an 8-billion-parameter Think-RM, trained on just 6k preference examples:​\n* Tops RM-Bench, beating BT RMs and vertically-scaled GenRMs by about 8 percentage points.​\n* Performs well on HelpSteer3-Preference, RewardBench and other out-of-distribution tasks, especially in \"Chat-Hard\", math and coding which need extended reasoning.​\n* When used in the pairwise RLHF loop, leads to higher end-policy win rates than traditional pointwise RLHF with BT RMs.​\n\nAn ablation test shows that using the longest correct reasoning for warm-up is important, as it improves accuracy but makes outputs longer, showing a trade-off between depth and efficiency. Overall, Think-RM expands reward modeling from \"vertical\" ensemble scaling to \"horizontal\" depth, offering a practical way to use rich internal reasoning for preference-based RLHF.", "questions": "Can the authors evaluate Think‑RM against vertical GenRM settings and a human preference study on at least 100 prompts?", "limitations": "No", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "​\nThis paper offers a compelling depth‑oriented alternative to today’s reward‑modeling practice: by pre‑training on very long chains‑of‑thought and then trimming them with rule‑based RL, an 8 B‑parameter model outperforms both classic BT and “vertical” GenRM baselines on every difficult benchmark the authors test—while feeding its pairwise logits directly into a new RLHF loop.  ​\n​\n\nThe manuscript is easy to read. To my knowledge this is the first reward‑model paper that (i) mines and trains on multi‑kilotoken reasoning, (ii) prunes that reasoning with RL rather than heuristic truncation, and (iii) dispenses with pointwise score conversion in RLHF.  Earlier GenRMs limit each rationale to a few hundred tokens and rely on majority voting for robustness; Think‑RM flips that strategy.  The combination is novel even if each ingredient—long‑CoT mining, GRPO, pairwise PPO—exists in separate literature streams. Reward‑model brittleness is a bottleneck for RLHF deployment, especially when tasks demand deep reasoning and preference data are scarce.  By showing that horizontal scaling (thousands of tokens in one trace) beats the vertical self‑consistency trick that dominates today’s GenRMs, Think‑RM shifts the conversation away from ever‑larger majority‑vote ensembles.  ​\n​\n\nMy only concern is that I'm afraid that human judge is needed for such RM paper. While the paper shows impressive results on various benchmarks, real-world applications of RLHF often involve subjective human preferences. Incorporating human judges in the evaluation process could provide a more comprehensive assessment of Think-RM's performance. For example, in scenarios such as content generation for creative writing or personalized advice, human judgment of the quality and appropriateness of the generated content is indispensable.", "was_revised": false, "final_justification": "", "rebuttal": "> W1. My only concern is that I'm afraid that human judge is needed for such RM paper. While the paper shows impressive results on various benchmarks, real-world applications of RLHF often involve subjective human preferences. Incorporating human judges in the evaluation process could provide a more comprehensive assessment of Think-RM's performance. For example, in scenarios such as content generation for creative writing or personalized advice, human judgment of the quality and appropriateness of the generated content is indispensable.\n\n> Q1. Can the authors evaluate Think‑RM against vertical GenRM settings and a human preference study on at least 100 prompts?\n\nWe appreciate the reviewer's valuable comment. Since conducting a human preference study is costly, we use Gemini 2.5- Pro as a practical proxy for human judges to evaluate the quality of preference judgments. Specifically, we evaluate the depth and breadth of rationales generated by Think‑RM compared to CoT‑GenRM on a reward modeling task using prompts from MT‑Bench‑Hard, a challenging subset of MT‑Bench designed to test instruction-following and reasoning abilities in multi-turn interactions. Under this evaluation, Think‑RM achieves a 73.3% win rate, demonstrating strong alignment with human-preferred preference reasoning. Below is a sample judgment from Gemini 2.5 Pro (Judge A: Think-RM vs. Judge B: CoT-GenRM):\n\n```\nJudge A provides a clear and accurate analysis of both assistant responses. Their reasoning correctly identifies that Assistant A fulfilled the user's request by writing a complete short story, whereas Assistant B only provided a single additional sentence, failing to deliver a story. Judge A methodically breaks down why Assistant A's response is superior across several criteria, including helpfulness, coherence, and complexity.\n\nIn contrast, Judge B's evaluation is fundamentally flawed. It appears to have completely confused the two assistant responses, attributing the qualities of Assistant A's story (e.g., \"full, engaging narrative arc,\" \"first-person narrative,\" \"atmospheric description\") to Assistant B, and describing Assistant A's response as being only one sentence long. Because its entire reasoning is based on this factual error, its conclusion is incorrect and its judgment is unreliable.\n\nJudge A's evaluation is superior due to its accuracy, logical consistency, and correct interpretation of the provided inputs.\n```\n\n> Beyond what was requested in the review, all experiments in the manuscript were soley conducted using Llama‑3.1‑Instruct‑8B. We have now additionally obtained results with Qwen‑2.5‑Instruct‑3B, which are included here. These new results are consistent with the Llama‑based results reported in Tables 1–3 of the manuscript.\n\n### HelpSteer2 (Qwen2.5-3B-Instruct)\n| Model                                              | Validation | Avg. Len |\n|----------------------------------------------------|------------|----------|\n| Base                                               |    53.84   | 262.74   |\n| BT RM                                              |      71.01      |      -    |\n| CoT-GenRM\nw/ vertical inference-time scaling (m = 16) |   74.43     |    377.09      |\n| DeepSeek-GRM\nw/ vertical inference-time scaling (m = 16)|  72.74          | 160.49         |\n| Think-RM (SFT)                                     |   71.31   | 1061.05      |\n| Think-RM (SFT + RL)                                |    **75.99**   |   836.62       |\n| QwQ-32B                                               |   78.69    | 760.82        |\n\n\n### HelpSteer3 (Qwen2.5-3B-Instruct)\n| Model                                              | Code | General | Multilingual | Stem | AVG | Avg. Len |\n|----------------------------------------------------|------|---------|--------------|------|-----|----------|\n| Base                                               | 62.27| 53.61   |     54.24    |     53.70  |  55.68   |    365.49      |\n| BT RM                                              |   70.14   |   63.39      |      64.20        | 69.09     |   65.99  |    -      |\n| CoT-GenRM\nw/ vertical inference-time scaling (m = 16) |    76.16   |      67.98     |    73.94          |     63.99  |    **70.34**   |   388.08       |\n| DeepSeek-GRM\nw/ vertical inference-time scaling (m = 16)|            73.15 | 65.74       | 72.73 | 66.67 | 68.72 |162.6 |\n| Think-RM (SFT)                                     | 73.15     |   65.74      |        74.70      |     67.49  |    69.17  | 1230.71  |\n| Think-RM (SFT + RL)                                |       75.93        |     67.16 |    71.06  | 67.70  | 69.87 | 849.66     |\n| QwQ-32B                                             |  84.49    |     71.80 | 81.82    | 67.49    | 75.83    |  863.27      |\n\n\n### RewardBench (Qwen2.5-3B-Instruct)\n| Model                                              | Chat | Chat Hard | Reasoning | Safety | AVG | Avg. Len |\n|----------------------------------------------------|------|-----------|-----------|--------|-----|----------|\n| Base                                               |   73.74    |    48.03        |    60.52       |    69.86     |   63.60    |    321.59      |\n| BT RM                                              |  87.43    |  62.50         |   73.74       |   75.95      |  74.90   |    -      |\n| CoT-GenRM\nw/ vertical inference-time scaling (m = 16) |  94.27    |     56.47       |      71.16       |    79.53    |  75.18   |   341.56       |\n| DeepSeek-GRM\nw/ vertical inference-time scaling (m = 16)|          93.3  | 54.71          | 69.41 | 77.09 | 73.74 |  160.18 |\n| Think-RM (SFT)                                     | 91.76     |      62.28      |     72.06       |      82.64   |  76.28    |    1844.35    |\n| Think-RM (SFT + RL)                                |     93.58    |    62.83        |      75.77     |   81.96   |   **78.56**   |   1172.42     |\n| QwQ-32B                                             |  96.93    |  82.24   | 97.62    | 89.59    | 93.45    | 1123.66       |\n\n\n### RM-Bench (Qwen2.5-3B-Instruct)\n| Model                                              | Chat | Code | Math | Safety | AVG | Avg. Len |\n|----------------------------------------------------|------|------|------|--------|-----|----------|\n| Base                                               |  55.86    | 51.12      |  53.12    |   73.75   | 59.90  |    319.84      |\n| BT RM                                              |  54.61    |  54.19    |   58.52   |  70.09      |  61.24   |      -    |\n| CoT-GenRM\nw/ vertical inference-time scaling (m = 16) |  59.86    |   51.17   |  59.20    |  82.51      |  65.63   | 354.83       |\n| DeepSeek-GRM\nw/ vertical inference-time scaling (m = 16)|  61.46          | 50.88         | 55.95| 81.56|64.13 |160.51 |\n| Think-RM (SFT)                                     |  62.88    |  51.85   |    64.59|   87.30    |    69.78  |   1884.25      |\n| Think-RM (SFT + RL)                                |    60.42    |    52.73   |  66.54   |    87.05     | **70.39**    |  1169.78     |"}, {"review_id": "yisKiPWLg5", "reviewer": "Reviewer_XU3w", "summary": "1. This paper introduces a two-stage training framework that enables long-horizon reasoning in generative reward models by modeling an internal thinking process.\n2. It also introduces a pairwise RLHF pipeline that trains policies directly on pairwise preference signals using its reward model.", "questions": "1. I would like to understand the advantages and disadvantages of this framework compared to other long-reasoning-based reward models, such as DeepSeek-GRM [1], RM-R1 [2], and R3 [3]. Given the close release dates of these works, my question is more for context and curiosity—it will not heavily influence my overall evaluation.\n\n2. Why is long internal thinking necessary for reward modeling? In math reasoning tasks, the large space of candidate answers makes long reasoning helpful. However, in reward modeling, there are typically only 2 to 5 candidate responses. Is such long reasoning truly needed for these relatively simple tasks? \n\n3. In addition, when generating long chain-of-thought (CoT) reasoning traces, the filtering criterion is to retain only those samples with correct preference predictions. Given the simplicity of the task and the small number of candidate answers, how can we ensure the quality and meaningfulness of the generated long CoT reasoning traces?\n\n4. The vertical-scaling baseline takes a majority vote over 16 CoTs, while Think-RM fine-tunes on the single longest trajectory. What is the performance of CoT-GenRM vertical scaling when keeping only the answer with the longest CoT path?\n\n[1] Inference-Time Scaling for Generalist Reward Modeling\n\n[2] RM-R1: Reward Modeling as Reasoning\n\n[3] R3: Robust Rubric-Agnostic Reward Models", "limitations": "Efficiency. This method relies on much longer sequences, which leads to more training and inference time. In addition, rule-based reinforcement learning also adds instability.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper provides extensive empirical evaluations across multiple evaluation benchmarks.\n2. The paper is well-written and organized, making the technical content easy to follow.\n\nWeaknesses:\n1. Efficiency. This method relies on much longer sequences, which leads to more training and inference time. In addition, rule-based reinforcement learning also adds instability.  \n2. Limited model scale. All experiments use only Llama-3.1-8B-Instruct, so it is unclear whether the method still helps on other backbones.\n3. Limited baselines. The paper mainly compares with the reproduced baselines. Please also include strong public reward models such as  GPT-4o, QwQ-32B(which is used to generate CoT rationales in this paper), DeepSeek-GRM and powerful BT reward models like Skywork Reward model.", "was_revised": true, "final_justification": "The authors addressed most of my concerns in the rebuttal. I have increased my score to 4.", "rebuttal": "> W1. Efficiency. ... In addition, rule-based reinforcement learning also adds instability.\n> L. Efficiency. ... In addition, rule-based reinforcement learning also adds instability.\n\nWe appreciate the reviewer's concerns regarding the efficiency and instability of rule-based RL.\n\n### 1. Regarding the concern about efficiency.\n\nDue to the limited rebuttal space, please refer to our response on Reviewer Lj4C's W.\n\n\n### 2. Regarding the concern about instability of rule-based RL.\n\nAs described in Lines 178–182 and 276–278, **applying rule-based RL to the SFT-trained model in fact stabilizes its behavior**. The SFT model, trained on rich but often verbose reasoning traces, learns via rule-based RL to preserve only the essential reasoning steps and discard unnecessary verbosity. This effect is reflected in Tables 1–3, where Think-RM (SFT + RL) outperforms Think-RM (SFT) while generating significantly less reasoning tokens.\n\n> W2. Limited model scale.\n\nWe acknowledge the reviewer's concern regarding the limited backbone model. To address this, we have expanded our experiments by implementing Think-RM and other baselines using **Qwen2.5-3B-Instruct** as an additional backbone model. Now, Think-RM has been implemented based on both the Llama and Qwen family models, ranging from 3B to 8B. Due to limited computational resources, we are not able to provide much larger models at the current moment.\n\nThe key results across various benchmarks, including HelpSteer2 (in-distribution tasks), HelpSteer3 (moderately shifted tasks), RewardBench, and RM-Bench (out-of-distribution tasks), are presented in our response to Reviewer BTue. As shown, our new results with Qwen are consistent with the results with Llama presented in Tables 1–3 of the manuscript.\n\n\n> W3. Limited baselines.\n\nTo gain a deep understanding of different reward model training schemes, such as BT, CoT-GenRM, DeepSeek-GRM, and Think-RM, it is essential to compare them on a **unified dataset** and with **the same backbone model**. **Our objective is not to just list the highest-performing models by their accuracies, but to conduct a controlled and fair comparison that reveals in which cases each method is effective**. \n\nTo broaden the baselines, we re-trained DeepSeek-GRM under exactly the same conditions as Think-RM and CoT-GenRM (same dataset, same long CoT generator (QwQ-32B), same backbone model (Qwen2.5-3B-Instruct)). Results for DeepSeek-GRM with vertical inference-time scaling $m=16$ are included in our response to Reviewer BTue. We also report QwQ-32B's performance in the same tables.\n\nThe Skywork reward model is simply a BT reward model trained with a carefully chosen data mixture and postprocessing strategy, resulting in a training set 13x larger than ours. Because of this discrepancy, a direct comparison with Think-RM would be less informative (just for reference, we provide additional RM-Bench results in the table below). Instead, in Tables 1–4 of the manuscript, we report a BT counterpart trained on the same dataset and backbone model as Think-RM for a fair comparison.\n\n| Model                            | RM-Bench  | \n|----------------------------------|------|\n|Skywork-Reward-Llama-3.1-Instruct-8B|70.1 |  \n| Think-RM-Qwen2.5-Instruct-3B          |  70.4   | \n| Think-RM-Llama3.1-Instruct-8B          |  **75.2**   | \n\n\n> Q1. I would like to understand the advantages and disadvantages of this framework compared to other long-reasoning-based reward models, such as DeepSeek-GRM [1], RM-R1 [2], ...\n\n\n1. **DeepSeek-GRM** can be viewed as a variant of CoT-GenRM that focuses on **vertical inference-time scaling** rather than horizontal long reasoning-based reward models. It improves CoT-GenRMs by prompting the model to generate instance-specific evaluation criteria adaptively, along with the critic and final answer. This leads to **generating diverse CoT paths to the final answer, which is essential for vertical scaling**. As reported in their Table 2, DeepSeek-GRM performs inference with 32 parallel generations. Additionally, as shown in their Figure 7, its response length is typically around 200–300 tokens, significantly shorter than Think-RM, which generates over 1000 tokens for challenging tasks. We note that **this adaptive evaluation criteria generation approach could also be incorporated into Think-RM**, as it currently uses prefixed evaluation criteria as described in Lines 144–146 of the manuscript.\n\n2. **RM-R1**, similar to DeepSeek-GRM and different from Think-RM, uses adaptive evaluation rubric generation. However, like Think-RM, RM-R1 treats reward modeling as a reasoning problem. As shown in their Table 11, its reasoning relies on an explicit answer chain with a fixed output format (e.g., the model is expected to fill the format ```<type>...</type><rubric>...<justify>...</justify></rubric><eval>...</eval><answer>...</answer>```), which can restrict the model's natural reasoning process. In contrast, Think-RM imposes no such restrictions but only uses ```<think>...</think>``` so as not to prevent its inherent reasoning process. This freedom is essential for enabling advanced reasoning capabilities (see Figure 1 in our manuscript). As a result, according to their Figure 8, RM-R1's reasoning trace is approximately 600–800 tokens, still shorter than Think-RM's long-horizon reasoning. This difference contributes to Think-RM's superior performance on both RewardBench and RM-Bench, despite RM-R1 being trained on 42K samples compared to only 6K for Think-RM. As shown below, Think-RM based on the Llama3.1-8B-Instruct model outperforms RM-R1 based on comparably sized Qwen models, achieving significantly better results.\n\n| Model                            | RewardBench | RM-Bench | \n|----------------------------------|------|------|\n|RM-R1-DeepSeek-Distilled-Qwen-7B|80.1 | 72.4 \n| RM-R1-Qwen2.5-Instruct-7B           |   85.2    | 70.2|\n| Think-RM-Llama3.1-Instruct-8B          |  **86.35**   | **75.06**|\n\nBeyond this, only Think-RM considers the downstream use of GenRMs within RLHF. Existing methods have yet to explore leveraging GenRM preference signals directly, instead relying on numerical reward scores. Our work takes this a step further by proposing a **pairwise RLHF pipeline that integrates GenRMs seamlessly, using their preference verdicts directly for policy optimization**.\n\n\n\n> Q2. Why is long internal thinking necessary for reward modeling? ... Is such long reasoning truly needed for these relatively simple tasks?\n\nWe are not entirely sure what the reviewer means by```in math reasoning tasks, the large space of candidate answers makes long reasoning helpful.``` However, we believe that in math reasoning tasks, the ground-truth step-by-step reasoning path that connects the problem to the solution is inherently long; thus, long reasoning is helpful as it can better match this structure. As described in Lines 52–56 of the manuscript, in reward modeling for domains such as math, coding, adversarial prompts, or multi-turn contexts, the inherent step-by-step judgment path that connects the context and candidate responses to the final verdict is also long, making long-horizon reasoning in GenRM essential.\n\nThis is supported by our main experiment results. As noted in Lines 291–293 of the manuscript, Think-RMs outperform CoT-GenRMs by over 10% and 5% in the **Chat Hard** and **Reasoning** subcategories of RewardBench, and by 12% in the **Math** domain of RM-Bench.\n\nIf the reviewer would like to see real examples demonstrating the necessity of using long reasoning for reward modeling, we refer them to samples in RM-Bench [1], particularly those in the \"math\" or \"code\" domains, available via the dataset link below.\n\n*References* \n\n*[1] Liu, Yantao, et al. \"RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style.\" The Thirteenth International Conference on Learning Representations. https://huggingface.co/datasets/THU-KEG/RM-Bench*\n \n> Q3. In addition, when generating long chain-of-thought (CoT) reasoning traces, the filtering criterion is to retain only those samples with correct preference predictions. ... how can we ensure the quality and meaningfulness of the generated long CoT reasoning traces?\n\nThis is an important point. Since the candidate answers are limited to A>B, A<B, or tie, it is possible that the reasoning trace may not always be fully accurate and might simply produce the correct preference label. As described in Section 4.3 (Ablation Study), we study a length-based reasoning trace selection strategy and show that selecting longer CoTs outperforms selecting shorter CoTs in terms of performance, with the trade-off of increased latency. We agree this is an interesting direction and will leave a more thorough investigation for future work.\n\n> Q4. The vertical-scaling baseline takes a majority vote over 16 CoTs, while Think-RM fine-tunes on the single longest trajectory. What is the performance of CoT-GenRM vertical scaling when keeping only the answer with the longest CoT path?\n\nThe table below compares CoT-GenRM with vertical scaling, which uses the preference label from the longest CoT path, against majority-vote-based CoT-GenRM. As shown, the longest-path approach performs significantly worse than majority voting, indicating that for explicit CoT paths, the longest reasoning trace is not always of high quality.\n\n\n| Model          | HelpSteer2 | HelpSteer3 | RewardBench | RM-Bench |\n|----------------|------|------|------|--------|\n|  CoT-GenRM (**Longest Length**)\nw/ vertical inference-time scaling (m = 16)                                             | 70.74     |  67.97    |  72.75   | 63.90  |\n| CoT-GenRM (**Majority Voting**)\nw/ vertical inference-time scaling (m = 16) |  **74.43**   |   **70.34**  |  **75.18**    |  **65.63**      |"}, {"review_id": "pRs8w0crIs", "reviewer": "Reviewer_KhzV", "summary": "The authors present Think-RM, a recepie for training long-horizon reasoning reward models. While LLM-as-a-judge paradigm as gained popularity, the authors point out that the standard is vertical reasoning, where many reasoning trajectories are generated and aggregated. Instead, the authors propose horizontal reasoning,g which encourages reasoning within a single trajectory. In standard Gen-RM, a dataset of reasoning traces is generated from a preference dataset. Then, a DPO-style supervised learning objective is used to train the reward model to combine preference predictions and reasoning traces. In Think-RM, the authors select the trajectory with the longest reasoning trace (and correct preference prediction) to build the training data.  The authors then apply GRPO to eliminate the model's tendency for verbose reasoning traces. Finally, the authors build a skew-matrix that down-weights the relative differences between long reasoning traces for the GRPO advantage term. \n\nThe authors empirically show that their reward models outperform standard baselines on a variety of reward model benchmarks, and also show that the LM policies learned by using their Think-RM reward models outperform baselines on the AlpacaEval2 benchmark.", "questions": "Do we always get long reasoning chains even if they are not required with Think-RM? Are there potentially other heuristics we could use instead of simply the longest chains to generate the training data?\n\nI wasn't clear on why we need the skew-matrix? The authors say, \"This formulation ensures that longer reasoning chains...result in smaller reward differences\". I am unsure why we want this? Some clarification would be helpful.", "limitations": "The authors do not discuss _any_ limitations. I would like to invite the authors to discuss some limitations of their method. For example, do we _always_ want long reasoning chains? I would think for simple questions we would not, but for long, difficult questions we do. Is there a way to get the best of both?", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n-The paper empirically shows that Think-RM outperforms COT-Gen RM reward models on RM-Bench and Reward Bench. \n- The paper empirically shows that reasoning models trained using the Think-RM reward model outperform pointwise RLHF models on AlpacaEval 2, and outperform reasoning models using CoT-GenRM when the length is uncontrolled.\n- The paper provides ablations showing that selecting the longest trajectory to train Think-RM is beneficial over selecting the shortest trajectory. \n\n# Weaknesses\n- I was initially sceptical that selecting the longest chain of thought would be a good idea, but the authors did partially address my concern by using rule-based RL to remove the verbose reasoning style. However, I am unsure how the rules were applied, what the rules were and how they eliminate verbosity. The authors say they \"restrict the rewards to be based solely on accuracy\", but I am unsure what that means exactly. \n- While the paper does present a new way of training a reward model, it is not original in the sense that it remixes well-known existing training techniques. \n- There is no discussion of limitations. \n- I find many of the algorithmic decisions, like using CoT length and the skew matrix, unmotivated.", "was_revised": true, "final_justification": "I was initially sceptical that selecting the longest chains of thought would be the best heuristic; however, the authors addressed these concerns by clarifying the use of the rule-based rewards. They also outlined that their experiments actually produced shorter reasoning traces than the baselines. Finally, I appreciate that the authors added limitations to the discussion.", "rebuttal": "> W1. I was initially sceptical that selecting the longest chain of thought would be a good idea, but the authors did partially address my concern by using rule-based RL to remove the verbose reasoning style. However, I am unsure how the rules were applied, what the rules were and how they eliminate verbosity. The authors say they \"restrict the rewards to be based solely on accuracy\", but I am unsure what that means exactly.\n\nIn Lines 187–188 of the manuscript, we define the rule-based reward functions used for both binary and multiclass Think-RM. For binary Think-RM, we assign a reward of 1 to all tokens in the response if the model correctly predicts the preferred answer; otherwise, the reward is 0. For multiclass Think-RM, we add an additional partial reward when the model correctly identifies the preference direction, even if the final prediction is not fully correct.\n\nBy applying RL with these rule-based rewards, the model explores and learns to retain the parts of reasoning essential for accuracy while eliminating verbose or redundant steps. **This outcome is evident in Tables 1–3: Think-RM (SFT + RL) outperforms Think-RM (SFT), while also producing significantly shorter internal reasoning traces.** This is also consistent with results obtained using different backbone models (Qwen2.5-3B-Instruct), as summarized in our response to Reviewer BTue.\n\nMoreover, in Section 4.3 (Ablation Study), we compare two CoT data selection strategies: using the longest vs. the shortest reasoning trace per instance. As shown in Figure 3, training with the longest CoT consistently leads to higher preference accuracy across all benchmarks. **This demonstrates the effectiveness of length-based CoT filtering in improving reasoning quality.** It also highlights a trade-off between accuracy and inference efficiency, depending on which selection strategy is chosen.\n\n> W2. While the paper does present a new way of training a reward model, it is not original in the sense that it remixes well-known existing training techniques.\n\n\nWe would like to clarify that while reasoning distillation (SFT stage of Think-RM) and rule-based RL are individually well known techniques, **their combination and algorithmic choices in Think-RM are specifically designed for reward modeling tasks**.\n\nFor example, the SFT stage is essential because applying rule-based RL directly to an instruction tuned model fails to enable long-horizon reasoning. Since the final answer in reward modeling is binary, the model can often guess correct answer without generating meaningful reasoning steps. This differs from the existing practice for training reasoning LLMs (e.g., math reasoning), where rule-based RL can be applied directly to a base model without the SFT stage. In that case, because the final answer is not binary, the model must generate sufficient reasoning to reach the correct solution.\n\nWe further apply rule-based RL after the SFT stage to **prune redundant reasoning traces, which is fundamentally different from the purpose of rule-based RL in DeepSeek-R1 that instead encourages progressively longer responses**.\n\nWhile long CoT reasoning has been explored for math and coding, we are the first to introduce long-horizon reasoning for reward modeling tasks and show that it is effective in this domain as well. **Additionally, we propose a pairwise RLHF method that enables seamless integration of pairwise GenRM into the RLHF pipeline, which is a completely new approach**.\n\n\n> W3. There is no discussion of limitations.\n\nThank you for pointing this out. One major limitation of Think-RM is that, similar to other vertically scaled methods, Think-RM's horizontal inference-time scaling introduces additional computational and time costs compared to models of the same size. However, this horizontally extended CoT delivers significantly better performance, surpassing current vertical scaling approaches. Moreover, we partially mitigate these costs by applying RL training to Think-RM (SFT), which removes verbose or redundant reasoning steps and reduces response length while retaining only the essential reasoning steps. Without inference-time scaling, achieving comparable performance would require substantially increasing model size, an approach that is less efficient and still struggles on reasoning-heavy tasks like RM-Bench, as shown below:\n    \n| Model                            | RewardBench | RM-Bench   | \n|----------------------------------|-------------|------------|\n| Llama3.1-8B-Instruct             |  71.0       | 63.8       |\n| Llama3.1-70B-Instruct            |  84.0       | 65.5       |\n| Think-RM-Llama3.1-8B-Instruct    |  **86.3**   | **75.1**   |\n\n> W4. I find many of the algorithmic decisions, like using CoT length and the skew matrix, unmotivated.\n\n> Q2. I wasn't clear on why we need the skew-matrix? The authors say, \"This formulation ensures that longer reasoning chains...result in smaller reward differences\". I am unsure why we want this? Some clarification would be helpful.\n\nRegarding the motivation for using CoT length as a scaling factor for preference strength in the binary case, as noted in Lines 24–25 of the manuscript, when two candidate responses are easy to distinguish (i.e., large preference strength), excessive reasoning is unnecessary, and the CoT length is typically shorter. Conversely, when the responses are harder to distinguish (i.e., smaller preference strength), longer CoT reasoning is often required. Therefore, scaling by (1/CoT length) provides a heuristic way to calibrate different preference strength for binary outputs. As described in Line 201, this scaling is not needed when using multiclass Think-RM, which inherently addresses this issue.\n\nRegarding the skew matrix, we emphasize that this is not an algorithmic decision. For notational simplicity, we refer to it as a skew-symmetric matrix, which is a mathematical property of the pairwise preference strength matrix $D$. Specifically,\n\n$$\nD\\in\\mathbb{R}^{n\\times n} \\text{ is skew symmetric} \\Leftrightarrow D=-D^T.\n$$\n\nAs described in Lines 199 - 203, each element $d_{ij}$ represents $\\text{preference on j-th response} - \\text{preference on i-th response}$, while $d_{ji}$ represents $\\text{preference on i-th response} - \\text{preference on j-th response}$. By definition,\n$$\nd_{ij}=-d_{ji}\n$$\n\nfor all $i,j$ in the same group. Therefore, $D$ is inherently skew-symmetric, and we use the term \"skew matrix\" purely for notational simplicity, not as a design choice requiring further motivation. We will clarify this in the updated version.\n\n> Q1. Do we always get long reasoning chains even if they are not required with Think-RM? Are there potentially other heuristics we could use instead of simply the longest chains to generate the training data?\n\nYes. Similar to other reasoning models such as DeepSeek R1, Think RM typically produces long reasoning chains even when shorter reasoning would suffice. This behavior aligns with the goal of this work, which is to demonstrate that horizontal scaling of current GenRMs improves preference output quality on reasoning-heavy tasks without additional vertical scaling and to show how such models can be trained. **Incorporating dynamic reasoning capabilities into Think RM is a separate interest, and we will note this in our future work section**.\n\nOf course, **Think RM could also be extended with adaptive thinking budgets**, allowing it to adjust reasoning length based on task difficulty. State-of-the-art reasoning models like Gemini 2.5 and Claude 3.7 already use such budgets, learned during training, to limit the number of reasoning tokens used per task [1, 2].\n\n*References*\n\n*[1] Comanici, Gheorghe, et al. \"Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.\" arXiv preprint arXiv:2507.06261 (2025).*\n\n*[2] Anthropic. \"Claude 3.7 Sonnet and Claude Code.\" Anthropic Blog (2025).*"}]

Author rebuttal:

Meta-review: This paper proposes a novel generative reward model that generates long-horizon rationales for preference decision between two candidate outputs. In specific, it trains the proposed reward model, Think-RM, by SFT on long CoT data followed by rule-based RL. Then, it applies to a modified pairwise GRPO. Experimental results show that the proposed reward model for horizontal inference-time scaling outperforms previous vertical inference-time scaling with generative reward models on RM evaluation benchmarks, and the proposed pairwise RLHF performs better than pointwise RLHF.

Overall, the long-horizon reasoning instead of previous vertical scaling for preference reward estimation seem to be interesting and techically sound. A modified GRPO for pairwise RLHF with the proposed reward model also seems to be somewhat novel. Most concerns raised by the reviewers are well addressed by the authors, including a lack of experiments regarding different backbones and baselines, during the rebuttal. Based on the consensus of all positive ratings between the reviewers, I would recommend the paper to be accepted.

However, some concerns still need to be further resolved. First, even though the main focus on the results in Table 4 is to demonstrate the benfits in using pairwise RLHF, the synergetic improvement by both the pairwise RLHF and Think-RM would be necesseary with more diverse LLM benchmarks, from the persepective of the praticality. In addition, regarding the Think-RM training framework, more diverse ablations and analyses on the use of longest CoT filtering and rule-based RL would be necessary. For example, unlike in Figure 3, one could consider mixing long and short CoTs for SFT, or incorporating factors such as a length penalty into the reward function during RL.
Decision: Accept (poster)