Title: Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modeling
Venue: NeurIPS 2025
Accepted: no
Keywords: Model analysis & interpretability

Reasoning

Inference-time Scaling
Abstract: Inference-time scaling techniques have shown promise in enhancing the reasoning capabilities of large language models (LLMs). 
While recent research has primarily focused on training-time optimization, our work highlights inference-time reward model (RM)-based reasoning as a critical yet overlooked avenue.
In this paper, we conduct a systematic analysis of RM behavior across downstream reasoning tasks, revealing three key limitations: (1) RM can impair performance on simple questions, (2) its discriminative ability declines with increased sampling, and (3) high search diversity undermines RM performance. To address these issues, we propose **CRISP** (Clustered Reward Integration with Stepwise Prefixing), a novel inference-time algorithm that clusters generated reasoning paths by final answers, aggregates reward signals at the cluster level, and adaptively updates prefix prompts to guide generation. Experimental results demonstrate that CRISP significantly enhances LLM reasoning performance, achieving up to **5\%** accuracy improvement over other RM-based inference methods and an average of **10\%** gain over advanced reasoning models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides analysis on limitations of using inference-time reward models (RM), specifically on how it underperforms simple majority vote aggregation from sampling under various settings, and proposed a method that first clusters reasoning paths based on final answer, aggregates reward signal at the cluster level, before updating prefix prompts adaptively to guide generation.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper analyzes an important problem of inference-time reward modeling, and took a closer look at when such methods actually perform worse than simpler methods such as majority voting aggregation that does not require the use of reward models.\n- The empirical results presented in Table 1 seem to indicate that the proposed method does perform better than the baselines considered in the paper.\n\n**Weaknesses**\n- In many of the results tables and charts, the performance differences are fairly close -- it would be useful for there to be error bars indicated to provide a better sense of whether performance gains are signfiicant. \n- Some of the findings could be better explained or compared across each other. For example, Fig.7 seems to show that the RM based approaches outperform the SC baseline across all temperatures, which appears at odds with the trend indicated in Fig 1. This also relates to the question of whether after including error bars in the results, the implications remain the same.\n- It would be useful to further expand discussion on related works or include in baselines additional inference-time optimization methods, such as those that provide further improvements even without the use of reward models while adopting majority vote approaches. For example, works on optimizing LLM ensembles based on the same base model to improve reasoning tasks such as [1-3] would provide useful comparisons.\n- Further elaboration of the cost analysis should be included in the main paper, as it is an important consideration factor. Fig. 24 in the appendix does not seem to support the claim that the proposed method incur lower costs -- for example, it shows that the proposed method has significantly higher token consumption, and more time than BoN.\n\n[1] Lau et al, \"Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning Tasks\", NeurIPS Workshop on Foundation Model Interventions 2024\n\n[2] Chen et al, \"AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors.\", ICLR 2024\n\n[3] Du et al, \"Improving factuality and Reasoning in Language Models through Multiagent Debate, ICML 2024\n\nQUESTIONS:\nPlease see weaknesses for questions. Would appreciate clarifications and elaboration.\n\nLIMITATIONS:\nPlease see weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first presents an empirical analysis of how reward models (discriminators, verifiers) impact LLM reasoning at inference time. The analysis shows that task difficulty, sample scale, and search diversity can lead to degraded RM discrimination performance, and thus lower end-to-end accuracy. Then, the authors propose CRISP, a modified beam search method with weighted majority voting, which demonstrates some accuracy gain on math reasoning and is more token-efficient than “thinking” LMs that produce long (>8K) CoT.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. In Section 3, the paper presents a decent number of experiments to analyze the influence of one ORM (Skywork) and one PRM on (Skywork-o1) two policy LMs (Qwen2.5-3B and Llama3.1-8B) with two inference methods, BoN and MCTS. \n2. The conclusion that RMs struggle to recognize long-tail negative samples is solid.\n\nWeaknesses\n1. The significance and originality of this paper is not clear.\n- The findings are somewhat repetitive given existing work [1][2][3]. While the paper presents solid results that RMs struggle to recognize long-tail negative samples and thus do not generalize to more diverse search candidates, they have been discussed, tested, and validated in earlier work. So, it is unclear how this paper adds additional research value to the literature.\n- The proposed CRISP method seems to be a simple modification and combination of beam search and weighted majority voting, which may not stand out as “novel” ([4] and more). Additionally, how it addresses the claimed issues, e.g., by solving more simple problems or lowering RM scores for long-tail negative, are not clearly conveyed.\n2. The experiments on math reasoning tasks are using general RMs instead of those specifically trained for math reasoning ([5] and more). So the conclusion that RM underperforms on simpler math problems may be altered by using a task-specific RM, which is closer to real-world use cases. Similarly, the proposed CRISP method should be evaluated with RMs for math reasoning as well to demonstrate its full utility.\n\n[1] https://arxiv.org/abs/2402.10890 \n\n[2] https://arxiv.org/abs/2410.17820 \n\n[3] https://arxiv.org/abs/2411.17501 \n\n[4] https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling-test-time-compute \n\n[5] https://arxiv.org/abs/2305.20050\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses critical limitations in reward models used for inference-time reasoning in large language models. The authors conduct a systematic analysis revealing three key failure modes: insufficient reward discrimination, poor reward calibration, and suboptimal search strategies. To address these issues, they propose CRISP (Clustering-based Reward model Inference with Stepwise Prefixing), an algorithm that clusters reasoning paths and uses stepwise prefixing to improve both the accuracy and efficiency of inference-time reasoning. The method demonstrates significant improvements across multiple mathematical reasoning tasks and model sizes.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. **Thorough empirical analysis**: The paper provides a comprehensive diagnosis of reward model limitations with clear empirical evidence for each identified failure mode, offering valuable insights into why current approaches underperform.\n\n2. **Novel algorithmic contribution**: The CRISP algorithm introduces a principled approach to clustering reasoning paths, which is both theoretically motivated and practically effective.\n\n3. **Comprehensive evaluation**: The experimental evaluation covers multiple model sizes, datasets, and tasks, providing strong evidence for the generalizability of the approach.\n\n4. **Practical efficiency gains**: The method achieves not only improved accuracy but also enhanced computational efficiency, making it practically valuable for deployment.\n\n5. **Clear presentation**: The paper is well-structured with clear explanations of both the problem analysis and the proposed solution.\n\n**Weaknesses:**\n\n1. **Limited theoretical foundation**: While the clustering approach is intuitive, the paper lacks theoretical analysis of why this particular clustering strategy is optimal or under what conditions it is expected to work well.\n\n2. **Narrow evaluation scope**: The evaluation focuses primarily on mathematical reasoning tasks. It remains unclear how well the approach generalizes to other types of reasoning or generation tasks.\n\n3. **Dependency on reward model quality**: The method still fundamentally depends on having high-quality reward models, and it's unclear how robust the approach is to poorly calibrated or biased reward models.\n\n4. **Incremental improvements**: While the improvements are consistent and meaningful, they represent incremental rather than breakthrough advances in the field.\n\n5. **Limited failure analysis**: The paper could benefit from more detailed analysis of when and why the method fails, particularly in edge cases or with different types of reasoning patterns.\n\nQUESTIONS:\n1. **Theoretical justification**: Can you provide theoretical analysis or intuition for why the specific clustering approach used in CRISP is optimal? Are there theoretical guarantees or bounds on the performance improvement?\n\n2. **Generalization beyond mathematics**: How does CRISP perform on reasoning tasks outside of mathematics, such as logical reasoning, commonsense reasoning, or domain-specific problem solving?\n\n3. **Robustness analysis**: How sensitive is the method to the quality of the underlying reward model? Can you provide analysis of performance degradation with increasingly poor reward models?\n\n4. **Computational scalability**: How does the computational cost of CRISP scale with the number of reasoning paths and the complexity of the clustering? Are there practical limits to the number of paths that can be effectively handled?\n\n5. **Hyperparameter sensitivity**: How sensitive is the method to the choice of clustering parameters and prefixing strategies? Is there guidance for practitioners on how to tune these parameters?\n\nLIMITATIONS:\nThe authors acknowledge some limitations of their work, including the focus on mathematical reasoning and the dependency on reward model quality. However, the paper would benefit from more detailed discussion of:\n\n- The computational overhead of the clustering approach\n- The potential failure modes when reasoning paths don't cluster well\n- The generalizability limitations to other domains\n- The sensitivity to hyperparameter choices\n- The robustness to different types of reasoning patterns", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses improving LLMs' reasoning via inference-time reward modeling. It identifies three key limitations of reward models (RMs) during inference: poor performance on simple questions, declining discriminative ability with more sampling (due to an inverse long-tail phenomenon), and suboptimal performance under high search diversity.\n\nTo tackle these, the authors propose CRISP, a novel inference algorithm. It clusters reasoning paths by final answers, aggregates rewards at the cluster level, and uses adaptive prefix prompts to guide generation, with an early termination mechanism for simple questions.\n\nExperiments show CRISP outperforms existing RM-based methods and advanced reasoning models. This work advances understanding of RM limitations and offers an effective mitigation strategy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**  \n- **Quality**: Rigorous empirical validation with diverse datasets (MATH-500, GSM8K) and models ensures reliability. Systematic analysis of RM limitations (simple questions, sampling effects, diversity) uses controlled experiments .  \n- **Clarity**: Logical structure progresses from problem identification to solution. Figures and algorithms (CRISP) clearly illustrate key findings and methods .  \n- **Significance**: Addresses critical gaps in inference-time RM optimization, outperforming R1 models by 10% on non-math tasks while reducing token usage by 90% .  \n- **Originality**: Identifies novel RM issues (inverse long-tail, diversity sensitivity) and proposes CRISP, a unique clustered/prefix-guided approach .  \n\n**Weaknesses**  \n- **Generalization**: Limited testing on non-math/logic tasks; broader validation is needed.  \n- **Theoretical Depth**: Relies heavily on empirics; lacks formal analysis of why CRISP’s clustering mitigates inverse long-tail effects.  \n- **Cluster Sensitivity**: The effectiveness of CRISP hinges on accurate clustering by final answers, but the paper does not robustly address cases with ambiguous or multi-form answers (e.g., equivalent mathematical expressions or paraphrased conclusions), which could undermine cluster aggregation and reward signaling.\n\nQUESTIONS:\n1. Regarding the inverse long-tail phenomenon, how exactly does CRISP's cluster-level reward aggregation mitigate the RM's tendency to misrank low-frequency incorrect responses? Could you provide data on the frequency distribution of top-scored errors before and after clustering (similar to Fig.6 for CRISP)? This would clarify the mechanism—strong evidence could boost my evaluation.\n\n2. The early termination threshold (2 clusters) seems critical. Did you test other values across datasets? \n\n3. The ablation study shows module importance, but how do PRM and ORM differ in their reliance on the components of the proposed method? If PRMs benefit more from clustering, this would refine understanding.\n\nLIMITATIONS:\nOne key missing point is the exploration of how CRISP may perform in real-world, open-ended scenarios. The current study focuses mainly on well-defined reasoning tasks like math problems. In open-ended situations, the clustering based on final answers might be more challenging due to the high variability and subjectivity of responses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}