Title: Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Generative Reward Models

Large Language Models

Reasoning

RLHF
Abstract: Reinforcement learning from human feedback (RLHF) has become a powerful post-training paradigm for aligning large language models with human preferences. A core challenge in RLHF is constructing accurate reward signals, where the conventional Bradley-Terry reward models (BT RMs) often suffer from sensitivity to data size and coverage, as well as vulnerability to reward hacking. Generative reward models (GenRMs) offer a more robust alternative by generating chain-of-thought (CoT) rationales followed by a final verdict. However, existing GenRMs rely on shallow, vertically scaled reasoning, limiting their capacity to handle nuanced or complex tasks. Moreover, their pairwise preference outputs are incompatible with standard RLHF algorithms that require pointwise reward signals. In this work, we introduce Think-RM, a training framework that enables long-horizon reasoning in GenRMs by modeling an internal thinking process. Rather than producing structured, externally provided rationales, Think-RM generates flexible, self-guided reasoning traces that support advanced capabilities such as self-reflection, hypothetical reasoning, and divergent reasoning. To elicit these reasoning abilities, we first warm-up the models by supervised fine-tuning (SFT) over long CoT data. We then further improve the model's long-horizon abilities by rule-based reinforcement learning (RL). In addition, we propose a novel pairwise RLHF pipeline that directly optimizes policies from pairwise comparisons, eliminating the need for pointwise reward conversion. Experiments show that Think-RM outperforms baselines on both in-distribution and out-of-distribution tasks, with particularly strong gains on reasoning-heavy benchmarks: more than 10\% and 5\% on RewardBench's Chat Hard and Reasoning, and 12\% on RM-Bench's Math domain. When combined with our pairwise RLHF pipeline, it demonstrates superior end-policy performance compared to traditional approaches. This depth-oriented approach not only broadens the GenRM design space but also establishes a new paradigm for preference-based policy optimization in RLHF.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present Think-RM, a recepie for training long-horizon reasoning reward models. While LLM-as-a-judge paradigm as gained popularity, the authors point out that the standard is vertical reasoning, where many reasoning trajectories are generated and aggregated. Instead, the authors propose horizontal reasoning,g which encourages reasoning within a single trajectory. In standard Gen-RM, a dataset of reasoning traces is generated from a preference dataset. Then, a DPO-style supervised learning objective is used to train the reward model to combine preference predictions and reasoning traces. In Think-RM, the authors select the trajectory with the longest reasoning trace (and correct preference prediction) to build the training data.  The authors then apply GRPO to eliminate the model's tendency for verbose reasoning traces. Finally, the authors build a skew-matrix that down-weights the relative differences between long reasoning traces for the GRPO advantage term. \n\nThe authors empirically show that their reward models outperform standard baselines on a variety of reward model benchmarks, and also show that the LM policies learned by using their Think-RM reward models outperform baselines on the AlpacaEval2 benchmark.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n-The paper empirically shows that Think-RM outperforms COT-Gen RM reward models on RM-Bench and Reward Bench. \n- The paper empirically shows that reasoning models trained using the Think-RM reward model outperform pointwise RLHF models on AlpacaEval 2, and outperform reasoning models using CoT-GenRM when the length is uncontrolled.\n- The paper provides ablations showing that selecting the longest trajectory to train Think-RM is beneficial over selecting the shortest trajectory. \n\n# Weaknesses\n- I was initially sceptical that selecting the longest chain of thought would be a good idea, but the authors did partially address my concern by using rule-based RL to remove the verbose reasoning style. However, I am unsure how the rules were applied, what the rules were and how they eliminate verbosity. The authors say they \"restrict the rewards to be based solely on accuracy\", but I am unsure what that means exactly. \n- While the paper does present a new way of training a reward model, it is not original in the sense that it remixes well-known existing training techniques. \n- There is no discussion of limitations. \n- I find many of the algorithmic decisions, like using CoT length and the skew matrix, unmotivated.\n\nQUESTIONS:\nDo we always get long reasoning chains even if they are not required with Think-RM? Are there potentially other heuristics we could use instead of simply the longest chains to generate the training data?\n\nI wasn't clear on why we need the skew-matrix? The authors say, \"This formulation ensures that longer reasoning chains...result in smaller reward differences\". I am unsure why we want this? Some clarification would be helpful.\n\nLIMITATIONS:\nThe authors do not discuss _any_ limitations. I would like to invite the authors to discuss some limitations of their method. For example, do we _always_ want long reasoning chains? I would think for simple questions we would not, but for long, difficult questions we do. Is there a way to get the best of both?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n1. This paper introduces a two-stage training framework that enables long-horizon reasoning in generative reward models by modeling an internal thinking process.\n2. It also introduces a pairwise RLHF pipeline that trains policies directly on pairwise preference signals using its reward model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper provides extensive empirical evaluations across multiple evaluation benchmarks.\n2. The paper is well-written and organized, making the technical content easy to follow.\n\nWeaknesses:\n1. Efficiency. This method relies on much longer sequences, which leads to more training and inference time. In addition, rule-based reinforcement learning also adds instability.  \n2. Limited model scale. All experiments use only Llama-3.1-8B-Instruct, so it is unclear whether the method still helps on other backbones.\n3. Limited baselines. The paper mainly compares with the reproduced baselines. Please also include strong public reward models such as  GPT-4o, QwQ-32B(which is used to generate CoT rationales in this paper), DeepSeek-GRM and powerful BT reward models like Skywork Reward model.\n\nQUESTIONS:\n1. I would like to understand the advantages and disadvantages of this framework compared to other long-reasoning-based reward models, such as DeepSeek-GRM [1], RM-R1 [2], and R3 [3]. Given the close release dates of these works, my question is more for context and curiosity—it will not heavily influence my overall evaluation.\n\n2. Why is long internal thinking necessary for reward modeling? In math reasoning tasks, the large space of candidate answers makes long reasoning helpful. However, in reward modeling, there are typically only 2 to 5 candidate responses. Is such long reasoning truly needed for these relatively simple tasks? \n\n3. In addition, when generating long chain-of-thought (CoT) reasoning traces, the filtering criterion is to retain only those samples with correct preference predictions. Given the simplicity of the task and the small number of candidate answers, how can we ensure the quality and meaningfulness of the generated long CoT reasoning traces?\n\n4. The vertical-scaling baseline takes a majority vote over 16 CoTs, while Think-RM fine-tunes on the single longest trajectory. What is the performance of CoT-GenRM vertical scaling when keeping only the answer with the longest CoT path?\n\n[1] Inference-Time Scaling for Generalist Reward Modeling\n\n[2] RM-R1: Reward Modeling as Reasoning\n\n[3] R3: Robust Rubric-Agnostic Reward Models\n\nLIMITATIONS:\nEfficiency. This method relies on much longer sequences, which leads to more training and inference time. In addition, rule-based reinforcement learning also adds instability.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses a key issue in RLHF: creating reliable reward models, especially when data is scarce or tasks need deep reasoning. Conventional BT reward models are easy to train but often over-fit and are susceptible to reward hacking. Existing generative reward models (GenRMs) have short reasoning and produce pairwise preferences that don't fit well with standard RLHF pipelines.​\n\nThe authors present Think-RM, a generative reward-modeling framework that gives language models long-horizon, self-directed \"internal thinking\". It generates flexible, multi-thousand-token reasoning traces with self-reflection, hypothetical branching, and divergent analysis, solving the shallow reasoning problem of previous GenRMs.​\n\nTests show that an 8-billion-parameter Think-RM, trained on just 6k preference examples:​\n* Tops RM-Bench, beating BT RMs and vertically-scaled GenRMs by about 8 percentage points.​\n* Performs well on HelpSteer3-Preference, RewardBench and other out-of-distribution tasks, especially in \"Chat-Hard\", math and coding which need extended reasoning.​\n* When used in the pairwise RLHF loop, leads to higher end-policy win rates than traditional pointwise RLHF with BT RMs.​\n\nAn ablation test shows that using the longest correct reasoning for warm-up is important, as it improves accuracy but makes outputs longer, showing a trade-off between depth and efficiency. Overall, Think-RM expands reward modeling from \"vertical\" ensemble scaling to \"horizontal\" depth, offering a practical way to use rich internal reasoning for preference-based RLHF.\n\nSTRENGTHS AND WEAKNESSES:\n​\nThis paper offers a compelling depth‑oriented alternative to today’s reward‑modeling practice: by pre‑training on very long chains‑of‑thought and then trimming them with rule‑based RL, an 8 B‑parameter model outperforms both classic BT and “vertical” GenRM baselines on every difficult benchmark the authors test—while feeding its pairwise logits directly into a new RLHF loop.  ​\n​\n\nThe manuscript is easy to read. To my knowledge this is the first reward‑model paper that (i) mines and trains on multi‑kilotoken reasoning, (ii) prunes that reasoning with RL rather than heuristic truncation, and (iii) dispenses with pointwise score conversion in RLHF.  Earlier GenRMs limit each rationale to a few hundred tokens and rely on majority voting for robustness; Think‑RM flips that strategy.  The combination is novel even if each ingredient—long‑CoT mining, GRPO, pairwise PPO—exists in separate literature streams. Reward‑model brittleness is a bottleneck for RLHF deployment, especially when tasks demand deep reasoning and preference data are scarce.  By showing that horizontal scaling (thousands of tokens in one trace) beats the vertical self‑consistency trick that dominates today’s GenRMs, Think‑RM shifts the conversation away from ever‑larger majority‑vote ensembles.  ​\n​\n\nMy only concern is that I'm afraid that human judge is needed for such RM paper. While the paper shows impressive results on various benchmarks, real-world applications of RLHF often involve subjective human preferences. Incorporating human judges in the evaluation process could provide a more comprehensive assessment of Think-RM's performance. For example, in scenarios such as content generation for creative writing or personalized advice, human judgment of the quality and appropriateness of the generated content is indispensable.\n\nQUESTIONS:\nCan the authors evaluate Think‑RM against vertical GenRM settings and a human preference study on at least 100 prompts?\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper proposes a new training framework for the Generative Reward Model. In many benchmarks, Think-RM shows an improvement over other reward models. The paper also compares pairwise RLHF with pointwise RLHF, which shows a more effiective performance.

STRENGTHS AND WEAKNESSES:
Strength:
The writing of this paper is clear and easy to follow.
The paper evaluates the proposed method accorss many benchmarks, including HelpSteer2-Preference, HelpSteer3-Preference, RewardBench, RM-Bench and AlpacaEval 2.

Weakness:
The model trained with Think-RM demostrates a longer response length in all tables, which may indicates more computational and time cost.
In Table 1 and Table 4, there are marginal improvements between Think-RM and COT-GenRM.

QUESTIONS:
Could the authors clarify the difference between the SFT phase of Think-RM and the COT-GenRM training, as their training objectives appear similar?

In Table 4, the performance gap between GenRM and Think-RM on HH-RLHF is minimal and requires further explanation.  And authors claim that the WR of Think-RMs is higher than COT-GenRMs, but LC is a more important metric in  AlpacaEval2.

Why is the number of independent judgments in Table 4 inconsistent with those in other tables?

LIMITATIONS:
The paper does not propose specific mechanisms to detect or correct hallucinated reasoning chains, which are common failure modes.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03