Title: Know What You Don't Know: Uncertainty Calibration of Process Reward Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: process reward model

calibration

inference-time-scaling

instance-adaptive scaling
Abstract: Process reward models (PRMs) play a central role in guiding inference-time scaling algorithms for large language models (LLMs).
However, we observe that even state-of-the-art PRMs can be poorly calibrated. 
Specifically, they tend to overestimate the success probability that a partial reasoning step will lead to a correct final answer, particularly when smaller LLMs are used to complete the reasoning trajectory.
To address this, we present a calibration approach—performed via quantile regression—that adjusts PRM outputs to better align with true success probabilities. 
Leveraging these calibrated success estimates and their associated confidence bounds, we introduce an instance-adaptive scaling (IAS) framework that dynamically adjusts the compute budget based on the estimated likelihood that a partial reasoning trajectory will yield a correct final answer.
Unlike conventional methods that allocate a fixed number of reasoning trajectories per query, this approach adapts to each instance and reasoning step when using our calibrated PRMs. Experiments on mathematical reasoning benchmarks show that (i) our PRM calibration method achieves small calibration error, outperforming the baseline methods, (ii) calibration is crucial for enabling effective IAS, and (iii) the proposed IAS strategy reduces inference costs while maintaining final answer accuracy, utilizing less compute on more confident problems as desired.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel framing of process reward model (PRM) predictions, when normalised, as approximations of the probability of generating a correct response. The authors demonstrate that current state-of-the-art PRMs produce miscalibrated forecast probabilities and argue why proper calibration is critical for efficient reasoning and for assessing problem difficulty on the fly. To address this, they introduce an adaptive compute-budget allocation method that leverages calibrated PRMs, along with theoretical justifications, to dynamically adjust evaluation time based on estimated difficulty. Experiments on two reasoning benchmarks across several models show promising gains in both calibration and compute efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. **Clear motivation and presentation.** The paper is well motivated, and the core findings are presented in a logically structured, easy-to-follow manner.\n2. **Novel insight on PRM normalisation.** Demonstrates that normalised PRM scores can be interpreted as forecast probabilities of success and that these are poorly calibrated out of the box for current SoTA PRMs.\n3. **Evaluation of common recalibration techniques.** Shows that standard recalibration methods fail to properly calibrate PRMs on reasoning tasks. (It might be worth adding an additional baseline, adaptive temperature scaling, for completeness - see below.)\n4. **Effective adaptive compute allocation.** The proposed method not only improves calibration but also aided in helping with compute-budget savings on easier problems whilst allocating more resources to harder questions rather than having a fixed sampling budget for sampling reasoning trajectories. This is a neat step toward more efficient reasoning with PRMs.\n\n**Weaknesses**\n\n1. **Notation and presentation clarity.** Some of the mathematical notation is a little sloppy, for example, Definition 1’s conditional probability should be written as p(x_{1:t}\\mid q). The exposition of the quantile regression formulation (especially the first paragraph) is messy and needs rewriting for readability.\n2. **Calibration metric choice.** In Figure 3, the Brier score is used to compare recalibration methods, but it conflates calibration and discrimination. A more standard calibration metric (e.g., Expected Calibration Error or Adaptive Calibration Error) would be preferable.\n3. **Missing baseline.** While the paper correctly shows that conventional methods rescale or shift probabilities improperly, it should also compare against the adaptive temperature-scaling approach of Xie et al. 2024, which addresses some of these calibration challenges.\n4. **Error bars**: Lack of error bars in the main results is somewhat limiting of the significance of the papers results. I think some effort should be made to change this given the high variance of MC sampling trajectories within some of their methods. \n\nXie, Johnathan, et al. \"Calibrating Language Models with Adaptive Temperature Scaling.\" Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.\n\nQUESTIONS:\n1. **Correctness measurement:** How did you determine whether a generated response was correct? I could not find a detailed description of the correctness metric in the paper.\n2. **Sampling settings:** What sampling parameters (e.g., temperature, top-k or top-p) did you use when generating responses with multinomial sampling? More evaluation details would strengthen reproducibility.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores several ways of calibrating existing PRMs for a given LLM by tuning PRMs towards the success rates of the given LLM for any prefix solutions. Typically, given a prefix solution of a given LLM, they continue to sample n outputs with LLM, then calculate the success rate of sampled outputs, finally, they can run quantile regression over those data to calibrate an existing PRM. With the calibrated PRM, they further utilize those calibrated scores to dynamically picking size of N outputs in BoN setting, and beam size B in beam search setting.  Experiments on calibration tasks show significant improvements over un-calibrated baseline. With the guidance of those PRMs, they can achieve comparable results with lower budget in best of 64 settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Model calibration is an important property in inference time, as it can guide the search of reasoning especially in shot-CoT reasoning tasks. \n\nWeaknesses\n\n1. Experiments are not well-designed, and the following questions must be addressed (see Section Questions). \n2. The “budget” metric in Table 2 is a little misleading, as you also need to run PRMs to get the calibrated scores, which are not required in baseline system. Thus, speed and memory usage should be also reported in Table 2.\n3. Beam search with LLMs in this paper is not well-defined.\n\nQUESTIONS:\n1. In Section 5.1, “To start with, we construct a calibration dataset by randomly sampling 500 questions from the MATH training set. We assess calibration performance using standard metrics:”. Does this mean the authors tune and report on training set? Evaluation results in Table 1 is on complete solution or still on prefixes? \n\n2. It’s not clear to me how the authors select prefix solutions, just uniformly sample on each time T over all tokens or on the end of a step?\n\n3. Why not train a PRM directly on the calibration data with success rate.\n\nLIMITATIONS:\nSee weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the critical issue of poor calibration in Process Reward Models (PRMs), which are essential components in inference-time scaling algorithms for LLMs. The authors demonstrate that state-of-the-art PRMs systematically overestimate success probabilities, particularly for challenging problems and weaker models. To remedy this, they propose a calibration method based on quantile regression and introduce an instance-adaptive scaling (IAS) framework that dynamically adjusts computational budgets based on calibrated uncertainty estimates.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The paper effectively demonstrates that existing PRMs are poorly calibrated through comprehensive empirical analysis.\n2. The use of quantile regression for PRM calibration is creative and well-motivated, addressing the need for uncertainty-aware predictions rather than just point estimates.\n3.The proposed IAS framework has immediate practical applications for making LLM inference more efficient.\n\n## Weaknesses\n1. The experimental results are primarily based on PRMs with around 7B parameters, lacking analysis of larger-scale PRMs. It remains unclear whether larger models like Qwen/Qwen2.5-Math-PRM-72B also exhibit the overestimation phenomenon observed in smaller models.\n2. The evaluation is confined to the mathematical reasoning domain. While this may be due to current PRM limitations, with the emergence of more generalizable PRMs such as Gen-Verse/ReasonFlux-PRM-7B, it's uncertain whether the observations and methods can extend to broader domains beyond mathematics.\n3. The paper lacks comprehensive analysis of the training process, including training data composition and methodology. Only brief mention of LoRA fine-tuning is provided in the appendix (page 24), leaving important implementation details unclear.\n\nQUESTIONS:\n1. Regarding Line 214, BoN typically involves predicting complete sequences, but according to Definition 2, we need to estimate probability $p$ based on already generated content. Given that sequences differ from each other, is the estimated probability $p$ based solely on the question? How do you obtain the reward for the entire trajectory from the calibrated PRM—do you predict directly at the final step or average across all steps?\n2.  We observe that on the more challenging AIME24-25 tasks, BS+IASoM shows remarkable improvements over BS (e.g., R1-Qwen-7B improving from 0.1333 to 0.3167). Since calibration shouldn't affect ranking order, what is the source of such significant improvements?\n3. How is the training data for calibration obtained? Could you provide more details about the Monte Carlo rollout process and the quality of the generated success probability labels?\n4. The final chosen parameters are C=0.99 and β=10% (90th percentile), which are far from the median and quite conservative. Does this lead to prohibitively high computational overhead?\n\nLIMITATIONS:\nYes, in Page9.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}