Know What You Don't Know: Uncertainty Calibration of Process Reward Models
Authors: Young-Jin Park
Kristjan Greenewald
Kaveh Alim
Hao Wang
Navid Azizan
Venue: NeurIPS 2025 poster ()
Keywords: process reward model
calibration
inference-time-scaling
instance-adaptive scaling
Primary area: deep_learning
TL;DR: We present that off-the-shelf PRMs are often poorly calibrated. To this end, we introduce a quantile-regression calibration that aligns their outputs with success probabilities. We show calibration unlocks instance-adaptive inference-time scaling.

Abstract: Process reward models (PRMs) play a central role in guiding inference-time scaling algorithms for large language models (LLMs).
However, we observe that even state-of-the-art PRMs can be poorly calibrated. 
Specifically, they tend to overestimate the success probability that a partial reasoning step will lead to a correct final answer, particularly when smaller LLMs are used to complete the reasoning trajectory.
To address this, we present a calibration approach—performed via quantile regression—that adjusts PRM outputs to better align with true success probabilities. 
Leveraging these calibrated success estimates and their associated confidence bounds, we introduce an instance-adaptive scaling (IAS) framework that dynamically adjusts the compute budget based on the estimated likelihood that a partial reasoning trajectory will yield a correct final answer.
Unlike conventional methods that allocate a fixed number of reasoning trajectories per query, this approach adapts to each instance and reasoning step when using our calibrated PRMs. Experiments on mathematical reasoning benchmarks show that (i) our PRM calibration method achieves small calibration error, outperforming the baseline methods, (ii) calibration is crucial for enabling effective IAS, and (iii) the proposed IAS strategy reduces inference costs while maintaining final answer accuracy, utilizing less compute on more confident problems as desired.

Know What You Don't Know: Uncertainty Calibration of Process Reward Models
Young-Jin Park', Kristjan Greenewald', Kaveh Alim', Hao Wang2 Navid Azizan!
Massachusetts Institute of Technology
MIIT-IBM Watson Al Lab 'Red Hat AI Innovation
Abstract
Process reward models (PRMs) play a central role in guiding inference-time scaling algorithms for large language models (LLMs). However, we observe that even state- of-the-art PRMs can be poorly calibrated. Specifically, they tend overestimate the success probability that a partial reasoning step will lead to a correct final answer, particularly when smaller L.Ms used to complete the reasoning trajectory. To address this, we present a calibration spproach-perfommn that adjusts PRM outputs to better align with true success probabilities. Leveraging these calibrated success estimates and their associated confidence bounds, we introduce instance-adaptive scaling (IAS) framework that dynamically adjusts the compute budget based on the estimated likelihood that a partial reasoning trajectory will yield correct final answer. Unlike conventional methods that allocate fixed number of reasoning trajectories per query, this approach adapts to each instance and reasoning step when using calibrated PRMs. Experiments on mathematical reasoning benchmarks show that (i) our PRM calibration method achieves small calibration eπrоr, outperforming the baseline methods, (ii) calibration is crucial for enabling effective IAS, and (iii) the proposed IAS strategy reduces inference costs while maintaining final answer accuracy, utilizing less compute on more confident problems as desired.
via quantile regression-
Project Page: http://young- j park. github. Bectuuu..o//knoee io/knowwwowo what-whhttttte you dont-kno know
Introduction
Inference-time scaling is an emerging paradigm that improves the output quality of LLMs by trading off inference speed. Just as humans approach complex problems by breaking them down, reasoning through each step, and revising their thoughts, LLMS can also benefit from being given more time to "think" during inference [1]. While myriad of inference-time scaling approaches have emerged, they often involve prompting the model with explicit reasoning instructions to encourage a structured thought process [51, 24]. Recent studies extend this idea by generating multiple candidate reasoning paths or answers and aggregating them to improve robustness [43]. Such strategies allow small- sized LLMs achieve comparable (or even better) performance with large models when tackling challenging tasks, such as advanced mathematics or complex coding [20, 31].
An important component in many inference- time scaling algorithms is the chosen process re- ward model (PRM). PRMs are trained to quantify, as reward, how good desirable model's intermediatc-step outputs are with respect to a given task and/or to alignment with human preferences [6, 48, 30]. When normalized, reward scores are often interpreted as the probability that continuing from the current output will lead to desirable outcome (e.g., producing a correct final answer in math problem) [28]. In our experiments, we find, however, that even state-of-the-art PRMs can be
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

Easy Problem
Difficult Problem
Figure I: Why should problem-solving strate- gies adapt task difficultt? Simple problems can often be solved through quick, intuitive so- lutions, whereas harder ones require extended, deliberate reasoning steps. Accordingly, is de- sirable for LLMs to adjust their compute usage adaptively. Fixed -budett methods like best-of- N are inefficient in this context: they either waste computation on easy tasks or fail to al- locate sufficient resources to solve harder ones accurately.
Assess Difficulty (Easy)
Assess Difficulty (Difficult)
J
Prone
Redundant
Quick Intuitive Approsch
Careful & Analytical Approach
1 Immediaae Conclusion
Broad Reasoning Integration
miscalibrated, assigning overly optimistic scores scores- particularly on challenging, out-of-distribution problems. This is perhaps unsurprising since the probability of success depends on factors such as choice of model and inference method, which are not input into the PRM.
Miscalibration and overconfidence PRMs can limit their utility beyond simply ranking partial reasoning steps. We highlight three important applications where calibrated PRMs are desirable: (1) providing interpretable uncertainty estimates to monitor LLM outputs during generation [58], (2) identifying when to say "I don't know" [see, e.g., 46, 7] or backtrack based on a low predicted probability of success, and (3) assessing the current problem difficulty or likelihood of success to adaptively adjust the inference-time compute budget. These applications motivate fundamental question: Can we calibrate off-the-shelf PRMs and leverage them enhance inference-time scaling methods?
In this paper, introduce pipeline for improving the calibration of any off-the-shelf PRMs, allowing their scores to more accurately reflect the uncertainty that a given LLM will reach the correct answer. We begin by showing that standard calibration techniques, such temperature scaling [16], are inadequate for calibrating PRMs. To address this, we introduce a quantile regression [22] based scheme that reduces the calibration error of PRM scores. The resulting model predicts success probabilities, together with confidence bounds, for every query and intermediate reasoning step. Our method collects intermediate reasoning steps generated by given LLM and completes them by independently rolling out full responses. This Monte Carlo rollout provides an empirical estimate of the true success probability for each prefix trajectory. Given this data, fine-tune the PRM after replacing its prediction head with a quantile regression model. The resulting model produces reliable success probability estimates when evaluating intermediate steps in multi-step step reasoning tasks.
We use these now-calibrated PRMs enable instance-adaptive scaling (IAS), a framework where test- time compute is adaptively scaled based on the calibrated PRM's assessment of difficuly/likelhhoodd of success. Beyond saving compute, this concept has the potential reduce overall latency for the user waiting for a response. The intuition mirrors how humans solve problems: we spend more time on diffieult questions and allocate more effort promising solution paths (see Figure 1). Similarly, IAS guides LLMs to invest more computation in challenging or high-potential reasoning paths. We focus
widely used inference-time scaling methods: best-of-N (BoN) and beam search (BS). We prove that, for any given query and reasoning trajectory, the calibrated reward score can estimate the minimum number of additional trajectories required to produce least correct answer. In short, our key contributions are:
We address the overestimation success probabilities in existing PRMs by introducing a calibration approach using quantile regression.
We propose an instance-adaptive inference-time scaling strategy that dynamically allocates compute budgets by leveraging calibrated PRMs.
We empirically validate our methods by: 1) demonstrating low calibration error, and 2) enhancing best-of-N and beam search through instance-adaptive scaling, leading to improved efficiency.
1.1 Related Work
Inference-time scaling, LLM performance can be improved through multi-stage stage reasoning, which breaks problems into simpler subtasks [51, 24, 1]. Although more computationally expensive, tech- niques such sampling multiple outputs and aggregating them substantially improve performance

---

and robustness [57, 43]. Methods such as majority voting, verifier- selection [6, 30, 4], and reward-basee based Monte Carlo search [48, 50, 15] show strong gains in complex reasoning tasks. Yet. most focus on performance gains, with limited attention to its reliability and cost-effective utilization of computational resources.
Process reward models. Process reward models are specialized tools used in inference- time scaling that provide step-by-step verification of LLMs' reasoning process, rather than evaluating only the final outcome [62]. Typically, PRMs are trained on datasets labeled at each reasoning step, either by humans automated approaches like Monte Carlo rollouts and LLM-based judges. For example, Qwen-PRM [55] utilizes consensus-filcce
labels from combined human and automated sources, achieving state-of-the-arr accuracy [44], while Shepherd-PRA [50] effectively relies on purely automated labeling despite lower precision in detecting errors.
daptive sampling for efficient state estimation. Puri et al. [37] propose viewing LLM reasoning as a probabilistic state estimation problem. Note that the classical state estimation literature has explored methods for dynamically adjusting sample sizes based on state uncertainty [12, 13, 45, 10]. Similarly, information-drivenr adaptive strategies have been studied in the context of planning algorithms [18, 5, 29]. However, to the best of our knowledge, prior work has not proposed instance-adaptive sampling strategies within the context of LLM inference-time scaling.
For an extended discussion related work, see Appendix A.
Preliminaries and Notation: Inference-Timen Scaling with PRMs
We recall how reward models are used by standard inference-time scaling methods. Let LLM denote a language model that generates responses, and let PRM denote a process reward model used to evaluate the quality of the outputs produced by LLM. Given a query g. LLM generates multi-step reasoning trajectory x = (æ1, .2,..... TT), where x; represents the i-th reasoning step and the total length of the trajectory. We denote the prefix of the trajectory up to step as Xoxt (in particular, 3i : and To.0 an empty reasoning sequence). Below, we recap two standard PRM-based inference-time scaling methods.
Best-of-N (BoN) [4]: Given a query 4. first use LLM generate N complete trajectories
(x
((.)) LLM(g), for i = 1,...,N. N.
Then we apply PRM assign score to each trajectory: امعر PRM (4, x().. The final output is the trajectory with the highest reward xF") where i" = arg max; (())
Beam Search (BS) [43]: BS iteratively builds reasoning trajectories by alternating between generation
and evaluation, rather than generating full sequences single pass. At step t, each of the K surviving partial trajectories x is extended by one reasoning step. Specifically, for each trajectory, LLM
generates M candidate next steps: x(
(i-)) ~ LLM [g Xoxr-1) where j = -,...,,, M denotes different
continuations for the i-th trajectory. Then PRM evaluates each of the N = K x M extended x (i,)) Finally, among all N candidates, only the top K partial
trajectories: r (8.f) PRM q:
trajectories with the highest scores are kept for the next step. This pruning step ensures the search remains computationally tractable while focusing on promising reasoning paths.
3 Calibration and Process Rewards
PRMs are typically trained to quantify the quality of (partial) model outputs, i.e., their correctness, how well they advance the reasoning towards the solution to the query, and/or how well they align with human preferences. We highlight an additional, underexplored perspective: when normalized, PRM scores can be interpreted as an estimate of the probability that continuing from the current output will ultimately lead to a desirable outcome, such as producing correct final answer in multi-step reasoning task [50]. We observe that state-of-the-an PRMs often overestimate the chance of eventual success when interpreted in this way. We define the success probability as follows.
Definition 1. Given a query g and a partial trajectory Xoat generated so far, the success probability p is the probability that autoregressively continuing this trajectory with stochastic language model
3

---

Shuppard TB (lacdisrted
Cwoorii 20 (acalbeee
Uiia 43
2.0
Uarare
Oue 29
Ruee
Once 31 Deee
1.5
Denohty
Drna
3.0
85
1.0 -6.5
1.0
-1.0
0.5
-10
-03
(Eetimated
(a) Qwen-PRM (MATH500) (b) Qwen-PRM (AIMEa)
(c) Shepherd (MATH500)
(d) Shepherd (AIMEa)
Figure 2: Histogram of signed deviations between PRM rewards (i.e., estimated success probabilities) and ground-tran -ruth success probabilities. Ground truth estimated via Monte Carlo sampling: for each question and partial reasoning step prefix, we use given LLM to generate multiple completions and compute the empirical success rate. We evaluate Owen-PRM-7B and Shepherd-PRM--7 on the MATH500 (in-distribution) and AIME24-25 (out-of-distribution) datasets. Positive deviations indicate overestimation. PRMs consistently overestimate success probabilities. as evidenced by the distribution skewing right and/or peaking near 1.0. This miscalibration is particularly pronounced for weaker completion models and more challenging, out-of-distributiin problems.
(i.e., non-zero temperature), LLM, will ultimately yield correct final answer.
p A Pr( generated by LLM yields correct answer
q, Xo:x)
Here, we denote the index of reasoning step by and the partial reasoning trajectory No:t is the sequence of outputs generated by the model from step 1 to step t S T.
For the base case t O, the partial trajectory Xp:0 is an empty sequence, representing the state before any reasoning steps have been generated, where only the query is given. In this case, essentially estimates the difficulty of the query for LLM.
Since we have partial outputs, there is inherent uncertainty of future success (i.e., p E (0. 1)), unlike single-turn settings where an output can definitively be evaluated for correctness. Note that this is significant difference between our work and typical prior work on calibrated certainty estimates for LLMs, and is crucial to our quantile-based approach being feasible. That said, calibration metrics like Brier score and expected calibration error (ECE) still apply.
The inherent calibration challenge of off-the-shelff PRMs. Off-the-shelf PRMs lack guarantee of calibration because their training is inherently dependent on a specific policy model. They learn a reward function from reasoning paths (rollouts) generated by one particular model (e.g.. Owen-Math-Iime Instuutt, tying their accuracy to that model's unique generative process and capabilities.
The problem originates from the autoregressive nature of language models, where sequence generation policy- dependent. The probability of future tokens, (0 is conditioned the model's parameters, 6. Consequently, a PRM is only calibrated to the statistical patterns of the policy it was trained with. Using it with a different policy introduces distributional mismatch, breaking any calibration guarantee.
For instance, a PRM trained on highly capable 72B model will overestimate the success probability when paired with weaker IB model. The PRM calibrated the stronger model's performance and cannot account for the weaker model's higher tendency to make eπrors on complex logical steps. This mismatch results in inflated and unreliable confidence scores, as presented in Figure 2.
Why calibrate PRMs? While we will show that a well-calibrated PRM is desirable for several important tasks, calibration is not required for the canonical task of ranking for BoN or BS, since relative ordering is all that matters. Specifically, any monotonic transformation of the true success probability preserves the ranking but changes calibration. As noted in the introduction, however, calibrated probabilities are essential for interpretability. systematic safety monitoring, and efficient budget allocation. In particular, calibration enables instance-adaye adaptive scaling (IAS; discussed further in Section 4), where we dynamically adjust the number of trajectories to keep based on the estimated likelihood of success. In this setting, accurate probability estimates are essential for balancing efficiency and performance: a well-calibrated PRM allows us to reduce computations and maintain output quality. Furthermore, using PRM rather than an outcome reward model is critical here,

---

need to estimate success probabilities before complete solutions exist. For beam search, we evaluate each partial reasoning prefix guide search decisions. For best-of-N,, we evaluate only the question itself (with no prefix) to determine the optimal sampling budget.
3.1 Calibration Data Collection Methodology
Now, we present a calibration--va-ffettue via-finetuning approach to mitigate the aforementioned issues. To calibrate a PRM for a given LLM, we construct validation dataset through systematic multi-stage generation and evaluation process. Please see details in Appendix C.I.
Our validation set defined as Uge (x00m((((:: (x(x:1), x(..2),..... ((q.Nu=)) where @vai represents a curated set of validation questions; in our experiments, split. For notational simplicity, we omit the superscript q in the subsequent discussion, with the understanding that all operations are performed independently for each question in Qvai-
sample 500 random questions from MATH training
Stage 1: Initial Trajectory Generation. For each validation question q Qvat, we generate Nval = 8 independent reasoning trajectories using the target L.M.. This yields set of complete reasoning chains: (x("), xx(2) ...., (NN.))). These trajectories represent diverse solution paths the model naturally explores for the given question.
Stage 2: Prefix Extraction and Monte Carlo Rollouts. For each reasoning trajectory x(() x (8)
0.7( generated in Stage 1, consider all possible prefix trajectories To.t where t 10,1, T())
(3)
ranges from the initial step to the final step of the trajectory. For each prefix To.t. we perform a
(t)
Monte Carlo estimation by generating NMc 8 additional follow-up trajectories. These follow-up
trajectories are conditioned on both the original question and the specific prefix To. allowing us to estimate the probability of eventual success from that intermediate reasoning state.
(i)
Stage 3: Success Probability Estimation. For each prefix trajectory Toet evaluate all NMc follow-up trajectories determine correctness. Let ( denote the count of correct completions among these follow-ups. We then compute the empirical success probability: p(8.)) 2/..//Ncc which serves as our ground-trute -ruth label for calibrating the PRM's predicted rewards.
(g.5) ......0) triplets (question, prefix, and empirical Our calibration datasets, consisting of (g, (g,= success probability), are publicly available for various reasoning L.Mss.
3.2 Uncertainty-Awan Calibration through Quantile Regression
We seek to fine-tune PRMs to align their predictions with success rates p(..t) given the input q and generated steps so far (1) However, predicting success probabilities is inherently uncertain: calibration data is non-exhaustive and model capacity is finite. As with any machine learning model trained via empirical risk minimization (ERM). PRMs make lossmminiiiing predictions that approximate the conditional mean of the success probability.
While the conditional mean is reasonable point estimate, it presents critical problem: for any given test case, there is roughly a 50% chance the true success probability falls below this estimate. This means standard PRM architectures will systematically overestimate success probability for half of all instances. For applications requiring conservative estimation-suu as instance-adaplo scaling (IAS), where we allocate computational budget based on estimated difficulty- overestimation particularly problematic. If we overestimate (((.t) we allocate insufficient samples, leading high failure rates on problems that genuinely require more computation.
Instead of predicting the conditional mean, we propose predicting lower quantile of the posterior distribution over p(t.t), By targeting, say, the 10th percentile, we ensure that for at least 90% of test cases, our estimate is below equal to the success probability. This provides conservative lower bound that protects against underallocation of compute while still enabling meaningful instance- adaptive decisions.
co//attastssyoung j- i-parr/ppzz http:/pmm__aaala carm__caibaatior
Exactly the conditional mean for MSE loss and approximately so for cross-ents eonrrpy loss when the posterior variance is low.
5

---

To achieve this, we modify the PRM architecture to perform quantile regression. Specifically, we expand the output dimension of the prediction head to generate multiple predictions one for each target quantile level B, (e.g.. 10%. 50%, and -0%))rather than a single reward value. Details of this architectural modification and initialization are in Appendix C.5.
We then fine-tune the model using a weighted quantile loss (WQL) [23]:
Ny
WQL(f,
A
B max (o. p p(0m)) + (1 - B.)) max (0, plen) - -
Nq
n=1
where N₂ is the number of quantiles, ((aw) is the modified PRM's prediction for the -u-qutii,,, and p is the empirical success rate. This asymmetric loss penalizes overestimation more heavily for lower quantiles and underestimation more heavily for higher quantiles, encouraging the model to learn the distinct quantile levels. Importantly, training requires only point estimates p from Monte Carlo sampling- do not need to estimate the full distribution of p.
4 Instance-Adaptiee Inference-Tiine Time Scaling
Existing inference-time scaling methods typically allocate a fixed sampling budget N-for example, by generating N complete reasoning trajectories in BoN. However, this approach can be inefficient: for challenging tasks, N samples may insufficient to produce a correct answer, while for easier ones, it may waste computation. Just as humans allocate more effort to harder problems and less to simpler ones, it is desirable for LLMS to adjust their compute usage adaptively. Note that the present work, we (perhaps erring on the side of optimism) assign the maximum budget to the hardest problems, but principle, we could also choose to say "I don't 't know" or route to more capable model if the probability of success is too low (difficulty too high).
In this section, we explore how allocate the sampling budget dynamically based the likelihood that an intermediate reasoning path will ultimately yield the correct answer, which we use as proxy for the difficulty of the query. We begin with the following proposition, which characterizes the theoretical minimum number of samples needed on a per-instance basis:
Definition Fix ceeestion-prefii pair and let p [0, 1] be the success probability (see Defini tion 1), conditioned on the current generation, that a single continued trajectory sampled from the language model is correct. For a target probability C' (0, 1) we define the ceappeedent-smapllle sample-complexity
(p, C) A min
E Pr(at E
least one out of n trajectories is correct) 2 c
Proposition 1. For every p E 1) and E (0, 1).
set NIAS(p, C? A log(1 C)
log(1 p) then [NiAS(p o9 (p.
In other words, PRM could perfectly distinguish correct from from incorrect reasoning. then selecting the best trajectory among NIAS(p. (p, C) samples (i.e.. "best-of-NIAs" guarantees average accuracy of (at least) C for questions whose per-trajectory success probability p.
See Appendix B.1 for a proof. The key takeaway is that the required sample size NIAS scales inversely with log(1 - p). Intuitively, for a fixed target accuracy, "casier" questions (with larger p) require significantly fewer trajectories, offering compute savings. Thus, well-calibrated PRM enables adaptive sampling by estimating p and then using it to guide how many trajectories to generate.
As noted previously, PRMs frequently overestimate LLMs' success crobabilttess-pariicll weaker models or on hard, out-of-distributie queries (Figure 2). Relying on these inflated scores leads to suboptimal inference-time scaling by selecting an excessively small NiAS, whereas calibration mitigates these pitfalls and enables principled, cost-effective decision making.
for
4.1 LAS: Calibrated Reward-based Instance-Adaptive Scaling
Now, we propose a framework that can dynamically scale inference-time computes using an instance- adaptive scaling (IAS) strategy with calibrated PRMs. Theoretical justifications are in Appendix B.2.

---

BoN+IAS. Rather than drawing a fixed sample size N for best-of-N decoding, our IAS framework adaptively determines the number trajectories to generate based on the estimated difficulty each problem. Specifically, we compute the minimum number of samples NIAS needed to achieve the larget correctness level C' given the PRM's estimated success probability f(0);
NIAS & min [NiAs
C)] N Nmaa),
where Mmax is the maximum budget constraint. For problems with high estimated success probability, NIAS will be small, while challenging problems receive larger budgets up Nmax-
BS+IAS-of-M (with fixed K). beam search with a fixed beam width K, we can adaptively determine how many continuations to sample per prefix at each step. Suppose at given beam- step we have K candidate prefixes. Let f(9) min denote the minimum calibrated success probability among these K prefixes. To ensure least one correct completion across all prefixes with confidence level C, IAS computes the number of samples per prefix as:
NIAS (l0)
MIAS - min
Mmax
where Mmax is the maximum number of expansions allowed per prefix. By using the most pessimistic estimate min we ensure sufficient sampling even for the most challenging prefix in the beam.
BS+IAS-of-K (with fixed M). Conversely, when the number of expansions per prefix M fixed, adaptively determine the beam width itself. After generating M continuations for each of K' candidate prefixes and ranking them by their calibrated rewards f(8), IAS determines how many top-ranked prefixes to retain:
KIAS A min ((Kmax)U ,)
1 1sk5 S Kmax)
where (k is the calibrated reward the k-th best prefix and Kmax is the maximum allowed beam width. This ensures that the total budget KIAS x M 2 NiAS(f ((()
KiAS: C), maintaining the target confidence level C that at least one prefix will lead to a correct answer.
Selecting 3 involves a trade-off: prioritizing efficiency could suggest using the median or higher quantile, whereas selecting lower quantile (e.g.. 10%) follows our motivation of ensuring that specified probability of success is achieved. We can formalize this latter point as follows, using the framework of conformal prediction as applied to quantile regression in [38].
Theorem 1.
Set == oc. Let p(3) be the prediction of the 3th quantile, and suppose we have held an n-sample validation set Vn, Then, test input Xnt1.
P(success best-of-NIAste (C)\x++1) 2 C(1
where 3 2 3 depends on Vn (see Appendix B.2.1 for its specification).
5 Numerical Experiments
We evaluate our method on two mathermatical-reaoniiii Denchmarks-MMTTH550 [17] and AIME24-25
(i.e., AIME2024 and AIME2025) six LLMs: Llama-3.2-1B & 3.1-8B-Instruet [47), Qwen2.5-
Math-1.5B 7B-Instruct [55], DeepSeek-RR--isttlll-
Qwen-8B [9].
[62] as the primary PRM throughout the main manuscript, unless otherwise specified, it the top-performing open-source small-sized PRM in PRMBench [44]. We present experimental details and additional results for ReasonEval-7B [52] and Math-Shepherd- Mistral-7B [50] PRMs, along with comprehensive analyses in Appendices D and E, respectively.
We use 1rren..5-MthhPRRMM
5.1 Fine-Tuning PRMs for Better Calibration
We evaluate calibration errors of off-the-shelf PRMs and then show how our fine-tuning strategy reduces these errors. To start with, construct a calibration dataset by randomly sampling 500 questions from the MATH training set. We assess calibration performance using standard metrics: Brier score [3], positive Brier score (i.e., the mean square of overestimation error, max y, 0}),
7

---

Table 1:
Calibration enror before and after applying our calibration via- ria-finetuning method. We evaluate on the MATH500 (in- -i-distribution) and AIME24-25 -fuut---istrbuttion datasets, using various LLMs to generate responses. Four calibration error metrics are reported (lower better); the worst and best values for each dataset are highlighted in red and blue, respectively. Results show that our method consistently improves PRM calibration across different models and datasets.
Dataset
Model
Brier
PosBrier
AdaptiveC
ECE
AveragoCE
Uncal.
Calib.
Uneal.
Calib.
Uncal.
Calib.
Uncal.
Calib.
Uncal.
Calih
Llama- 3.2. BB
0.2414
0.0692
0.2226
0.0472
0.2830
0
0.2791
0.0942
0.3130
0.1840
Llama-3 1-8B
0.2045
0.1210
0.1771
0.0994
0.2625
0.1674
0.2368
0.1515
0.2048
0.1876
MATH500
Dwen-2.5-11
0.1541
0.1271
0.1305
0.1072
0.2176
0.1545
0.1554
0.1414
0.1229
0.1462
Owen-2.5-78
0.1008
0.0818
0.0846
0.0527
0.1459
0.0999
0.0981
0.0864
0.0920
0.1227
RI-Llama-8B
0.1614
0.0888
0.1140
0.0546
0.2505
0.1128
0.1311
0.0905
0.1517
0.1094
21-Qwen 7B
0.1450
0.0828
0.1056
0.0578
0.2413
0.1066
0.1095
0.0857
0.1957
0.1130
Llama-3 1-
0.1936
0.0029
0.1918
0.0005
0.2364
0.0108
0.2364
0.0041
0.4921
0.1306
Llama-3. 3. 1- -885
0.2274
0.0414
0.2227
0.0354
0.2839
0.0862
0.2839
0.0782
0.4580
0.3988
AINE24-28
Qwen 2.5-122.-1.558
0.3302
0.0727
0.3220
0.0528
0.4007
0.1054
0.4007
0.0865
0.3936
0.2889
Owen-2.5-7B
0.2894
0.0721
0.2820
0.0657
0.3547
0.0982
0.3547
0.0982
0.3892
0.2829
RI IllLlamm 8B
0.3846
0.0782
0.3712
0.0296
0.5259
0.1275
0.4764
0.0761
0.3614
0.1566
RI-Qwen-7B
0.4144
0.0694
0,4018
0.0338
0.5575
0.0898
0,5078
0.0689
0,3680
1261
Dog peephsiiitddeee besbiiiddeeeeeeeee
041
Twe Sodea
Tree Sodan
Hik.
liae tOune
QB 10am
(hure)
12
Bure
SANE
06
V0the04
A3074422
Figure 3: Comparison of our calibration method with popular techniques-teee temperature scaling, isotonic regression, and histogram binning-on MATH500 and AIME24-25 As shown, our quantile regression (QR) method reduces calibration error more effectively than these baselines.
AdaptiveCI [34], ECE [32], and AverageCE [33) between ground-truth Monte Carlo success rates and the PRMs (median) predictions.
Off-the-shelf PRMs are overconfident. We examine the histogram of the deviations of the estimated reward from the true success probability (f6.1) p(..1), As shown in Figure 2, the error densities are skewed to the right, with minimal mass the left, indicating consistent overestimation. This effect further amplified when weaker LLMs are used to generate responses, or when evaluated on more challenging datasets such as AIME24-25.
Significance of calibration. Table presents numerical evidence supporting our earlier claim that PRMs frequently suffer from poor calibration and tend to overestimate. In contrast, our proposed calibration approach effectively mitigates these calibration issues. Additionally, the approach substan- tially improves results on conditions. This finding further indicates that calibrated PRMs are better at differentiating more challenging questions or reasoning tasks from those with higher certainty.
datasets, underscoring its effectiveness across diverse
Calibration baselines. We evaluate several standard calibration methods for calibrating PRMs, including temperature scaling [16], isotonic regression [61], and histogram binning [60]. Detailed descriptions of these methods are provided in the Appendix C.4. Figure 3 compares our calibration- via-finetuning method with these baseline techniques. Since baseline methods correct calibration by uniformly shifting rescaling the output probabilities (or logits) of PRMs, they are effective only when every instance's 's prediction is consistently overestimated or underestimated. For instance, weaker models like Llama-3.2-1B often benefit from simple methods that reduce overall overconfidence. However, these techniques typically fail to calibrate well on out-of-distribution tasks, such as the AIME24-25 25 dataset. In contrast, our method leverages contextual information-sun as question categories and the position of the current reasoning step to achieve robust calibration across diverse models and dataset distributions.
5.2 Calibrated Reward Enables Instance-Adaptive Scaling
We demonstrate IAS by applying it to two simple yet widely used inference scaling techniques: best-of-N and N beamm search on steps.
8

---

Table 2: Comparison between the best-of-N method using a fixed-N strategy (BoN) and our proposed restace-adaaptive sampling strategy (BoN+IAS). We report both accuracy and relative computational cost (budget), measured by the average number of samples per question normalized by N 64: Budget = NiAS/N. Relative improvements over Pass@I accuracy are highlighted in light blue. Our IAS with calibrated PRMs achieves substantial compute savings without significant performance loss. Crucially, the effectiveness of IAS depends strongly on PRM calibration quality: uncalibrated PRMs tend overestimate success probabilities, causing overly optimistic downsampling. Moreover, uncalibrated PRM's model-independee design prevents it from offering adaptive strategies for different LLMs, resulting in further degradation for weaker Llama models.
Baselines
w/ Uncal. PRM
w/ Calib, PRM
Dataset
Model
Pass@1 1
BoN
BoN+IAS
Budget Ratio
BoN+IAS
Budget Ratio
Llama-3.2 3.2- IB
0.2255
0.4760
0.2278
0.0162
0.4623
0.6381
Llama-3, 1.-aB-31--BB 1-8B
0.4659
0.6440
0.4674
0.0162
0.6223
0.363 1
MATH500
Owen-2.5-1.5P
0.6970
0.7660
0.6973
0.0162
0.7537
0.2461
Qwen-2.5 2.C-BB
0.7994
0.8540
0.7993
0.0162
0.8368
0.2342
RI-Llama-SB
0.6734
0,8240
0.6729
0.0162
0.8042
0,3439
RI RI-Qwen-T1 7B
0.7556
0,8640
0.7569
0.0162
0.8568
0.3133
Llama-3 LB
0.0042
0.0000
0.0040
0.0195
0.0167
1.0000
Llama-3. 1-8B
0.0268
0.0500
0.0273
0,0195
0,0333
0.9685
AIME24-25
Qwen-2.5-1159
0.0932
0.1500
0.0962
0,0195
0.1522
0,9372
Qwen-2.5-7BB
0.0885
0.1167
0.0920
0.0195
0,1885
0,9099
R1-I.lmma-88
0.0784
0.1833
0.0838
0.0195
0.1217
0.9661
R1-Qwen-7B
0,1411
0.2667
0.1425
0.0195
0.1795
0.9635
Instance-adaptive BoN. We evaluate the BoN framework (Nmax 64) using both the uncalibrated and calibrated PRMs, comparing their performance against our proposed IAS strategy (BoN+IAS). We adopt conservative setting with C = 0.99 and / = 0.1 (see Appendix E.5 for an ablation study). For a fair comparison, we use the calibrated PRM to determine NIAS and the original PRM for ranking after adaptive sampling. We report the accuracy of each method and the relative computational cost associated with IAS, the average number of samples per question normalized by Nuaxx
Table 2 shows IAS, when combined with calibrated PRMs, achieves significant computational savings while maintaining performance close to fixed-N strategies. More importantly, as previously discussed, the original PRMs often severely overestimate success probabilities, especially for smaller models (e.g., Llama-3.2-1B) or challenging tasks (e.g., AIME24 25). As a result, IAS reduces budgets too aggressively, leading to performance sacrifice. In contrast, our calibrated PRMs allow IAS to reduce compute more conservatively, preserving accuracy while saving computational cost.
Our findings also show that to effectively reduce computational cost, adaptive sampling requires a well-calibrated PRM. Uncalibrated PRMs, which often overestimate success, lead the BoN+IAS framework to reduce computational budgets too aggressively, resulting in degraded performance.
Instance-adaptive Beam Search. To further verify the efficacy of the IAS for intermediate reasoning steps in addition to the initial stage, we test IAS-of-K (IASoK) and IAS-of-M (IASoM), with a beam search setup (N = 64, M = 8, K = 8). We use the same setting of C = 0.99 and / = 0.1.
As shown in Table 3, our IAS strategy maintains accuracy while reducing the budget up to about 75%. The IASoM variant tends to be more conservative than ISAoK, yet often achieves higher accuracy with comparable budget usage.
Why do we need an instance-adaptive scaling? Recall that our goal is not to introduce new inference-time scaling method that universally outperforms existing approaches. Instead, we aim to enable inference-time scaling to allocate compute budgets dynamically based on a model's estimated likelihood of answering correctly. To this end, IAS adaptively determines, on a per-instance basis, the number of samples that best balance accuracy and computational cost.
Figure 4 reports the performance plot across varying question difficulty levels for different values of N, along with the resulting accuracy and cost of the proposed IAS strategy (see Appendix E.4 for full results). As expected, enlarging N improves accuracy in a nearly monotonic fashion; nevertheless, the absence of "sweet spot" renders the selection of single budget inherently difficult. Furthermore, accuracy declines with increasing question difficulty, implying that the sample budget required to attain target accuracy should be dependent. Consequently, choosing a fixed N on

---

Table Comparison between the beam search (BS) method using a fixed-N/MM and our proposed adaptive sampling strategy (IASoK and IASoM) using calibrated PRMs. We report both accuracy and relative computational cost (budget) to the baseline, measured by the average number of LLM generations per question normalized by that of fixed-budget BS baseline. Relative improvements over the @ass@] accuracy are highlighted light blue. The proposed LAS strategy yields substantial computational savings without significant performance loss.
Baselines
IAS Calibrated PRM
Dataset
Model
Pass@ |
BS
BS+IASoK
Budget Ratio
BS+IASoM
Budget Ratio
Llama-3.2-1B
0.2255
0.5360
0,5180
0.8996
0.5320
0.9891
Llama-3. 1 188B
0.4689
0.6640
0.6500
0.5309
0.6740
0.6686
MATH500
Qwen-2.5-1.5B
0.6970
0.8060
0.7840
0.5528
0.8100
0,5993
Qwen-2.5-7B
0.7994
0.8680
0.8540
0.5332
0.8560
0.5757
RI-Llama-SB
0.6734
0.8140
0.8320
0.3581
0.8540
0,3962
RI-Qwen-T 7B
0.7556
0.8280
0,8460
0.3345
0.8740
0.3652
Llama-3 2-1B
1amma-32--1B
0.0042
0.0167
0.0167
1.0364
0.0167
1.0619
1lama-3.1-88
0.0268
0.0333
0.0667
0.2620
0.0500
0.5243
AIME24-25
Owen-2.5-1.5B
0.09322
0.1333
0.1333
0.5027
0.1667
0.6097
Qwen-2.5-7B
0.0885
0.2167
0.1500
0.6499
0.1833
0,7399
RI-I.lma--B
0.0784
0.1333
0,2000
0.4210
0.3000
0.5209
RI RI-Qwen- 7B
0.1411
0.1333
0,2000
0.3640
0.3167
0.4464
convenient validation set may lead to pronounced performance deficits under distributional shift. For instance, tuning on Level of the MATH500 benchmark would suggest small N regime, but this could be highly suboptimal for harder datasets.
IAS addresses this mismatch by adapting its budget to instance success probability: it expends roughly four times fewer samples Level items than on Level while allocating additional samples where they are most needed. In doing so, IAS aligns computational expenditure with uncertainty, yielding superior cost-effectiveness without pre-defining universally optimal N.
Ulame S- 3-80-ipdreet
Overd Satts-------- Imrrect
Despledett DUMELOOes-75
Jernge beenge-Aouurry Nocussy
0.8
0.8
0.8
6.6
2.
3.4
Aeerngo
0.2
2.4
0.2
0.5
22
The Number of Sumplis (1)
The Number of Suzpples (N)
The Number of Sumplis UN)
The Number of Sumples (V1
Figure 4: We illustrate average accuracy across test points of varying difficulty levels (1: easy to 5: hard). Results from the fixed-N baseline and our instaice-adaptive sampling (IAS) method are shown dashed lines and stars, respectively. As shown, IAS dynamically adjusts sampling based on problem difficulty in MATH500, allocating more samples to harder tasks.
6 Conclusion and Limitations
This paper introduces a calibration strategy that enables PRMs to produce more reliable estimates of success probability: the likelihood that partial reasoning step (or initial question) will lead to a correct final answer when completed by given policy LLM. Through quantile regression, our approach provides conservative lower bounds rather than overoptimistic point estimates, addressing key limitation standard PRMs.
We demonstrate practical application of well-calibrated PRMs in instance-adaptive inference-time scaling (IAS), where computational budget is allocated based estimated success probability. By providing reliable estimates, our calibrated PRMs enable more efficient resource allocation, investing more computation challenging problems while avoiding waste on easier ones.
Future work could explore fine-tuning strategies for better generalization across datasets and quantile levels, extend this approach beyond mathematical reasoning to domains like code generation and LLM agents, and investigate LAS in settings beyond best-of-N and beam search. We hope this work establishes a foundation for more reliable reward models and cost-effective adaptive scaling.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by the MIT-IBM Watson AI Lab, Jane Street, the MIT-Amazon Science Hub, the MIT-Google Program for Computing Innovation, and MathWorks.
References
[1] Abbe, E., Bengio, S., Lotfi, A., Sandon, C., and Saremi, O. (2025). How far can transformers reason? the globality barrier and inductive scratchpad. Advances in Neural Information Processing Systems, 37:27850-27895.
[2] Beeching, E., Tunstall, L., and Rush, S. (2024). Scaling test-time compute with open models.
[3] Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3.
[4] Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C.,, and Mirhoseini, A. (2024). Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787
[5] Choi, H.-L., and Lee, S.-. (2015). A potential-game approach for information-maxiniii cooperative planning of sensor networks. IEEE Transactions on Control Systems Technology, 23(6):332-23355.
[6] Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168.
[7] Cohen, R., Dobler, K., Biran, E., and de Melo, G. (2024). I don't know: Explicit modeling of uncertainty with an [idk] token. Advances in Neural Information Processing Systems, 37:10935 10958.
[8] De Leeuw, J., Hornik, K., and Mair, P. (2010). Isotone optimization in r: pool-adjjacent-violaoos algorithm (pava) and active set methods. Journal of statistical software, 32:1-24.
[9] DeepSeck-AI (2025). Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning.
[10] Elvira, V., Miguez, J., and Djurié, P. M. (2016). Adapting the number particles in sequential monte carlo methods through an online scheme for convergence assessment. IEEE Transactions on Signal Processing, 65(7): (5(7):1781-179) 181-4794.
[11] Farquhar, S., Kossen, J., Kuhn, L., and Gal, (2024). Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625--6)) 630..
[12] Fox, D. (2001). Kid-sampling: Adaptive particle filters. Advances in neural information processing systems, 14.
[13] Fox, (2003). Adapting the sample size in particle filters through kid-sampling. The interna- tional Journal of robotics research, 22(12):885-1003
[14] Gal, Y. and Ghahramani, Z. (2016). Dropout as bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pages 1050-1059. PMLR.
[15] Guan, X., Zhang, L. L., Liu, Y., Shang. N., Sun, Y., Zhu, Y., Yang, F., and Yang, M. (2025). rstar-math: Small Ilms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519.
[16] Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330. PMLR.
[17] Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D.,, and Steinhardt, J. (2021). Measuring mathematical problem solving with the math dataset. NeurIPS.
11

---

[18] Hollinger, G. A. and Sukhatme, G. S. (2014). Sampling based robotic information gathering algorithms. The International Journal of Roboties Research, 33(9):1271-1287,
[19] Hu, E. J., Shen, Y. Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L., and Chen, W. (2021).
Lora: Low-rank adaptation of large language models.
[20] Jacch, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A.,
Beutel, A., Carney, A., et al. (2024). Openai ol system card. arXiv preprint arXiv:2412.16720.
[21] Johnson, D. D., Tarlow, D., Duvenaud, D., and Maddison, C. (2024). Experts don't cheat: Learning what you don't to know by predicting pairs. arXiv preprint arXiv:2402.08733.
[22] Koenker, R. (2005). Quantile regression, volume 38. Cambridge university press.
[23] Koenker, R. and Bassett Jr, G. (1978). Regression quantiles. Econometrica: journal of the Econometric Society, pages 33-50.
[24] Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. (2022). Large language models are zero-shot reasoners, Advances in neural information processing systems, 35:22 35:22199-22213.
[25] Kuhn, L., Gal, Y., and Farquhar, S. (2023). Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiiv2332.09664.
[26] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. (2023). Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
[27] Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30.
[28] Lambert, (2024). Reinforcement Learning from Human Feedback. Online.
[29] Lee, S.-J., Park, Y.J.,, and Choi, H.-L. (2018). Efficient sensor network planning based approximate potential games. International Journal of Distributed Sensor Networks, 1460:15551477717814455
[30] Lightman. H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. (2024). Let's verify step by step. In The Twelfth International Conference on Learning Representations.
[31] Liu, R., Gao, Zhao, J., Zhang, K., X., Qi, B., Ouyang, W., and Zhou, B. (2025). Can 1b llm surpass 405b Ilm? rethinking compute-opting test-time scaling. arXiv preprint arXiv:2502.06703.
[32] Naeini, M. P., Cooper, G., and Hauskrecht, M. (2015). Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence.
[33] Neumann, L., Zisserman. A., and Vedaldi, A. (2018). Relaxed softmax: efficient confidence auto-calibration for safe pedestrian detection. OpenReview.
[34] Nixon, J., Dusenberry, M. W., Zhang, L., Jerfel, G., and Tran, D. (2019). Measuring calibration in deep learning. In CVPR workshops, volume 2.
[35] Park, Y.-J, Sobolewski, C., and Azizan, N. (2024a). Quantifying the reliability of predictions detection transformers: Object-level calibration and image-level uncertainty. arXiv preprint arXiv:2412.0173
[36] Park, Y.-J., Wang, H., Ardeshir, S., and Azizan, N. (2024b). Quantifying representation reliability in sepersupe learning models. In Uncertainty Artificial Intelligence, pages 2835-2860 PMLR.
[37] Puri, I., Sudalairaj. S., Xu, G., Xu, K., and Srivastava, A. (2025). A probabilistic inference approach to inference-tine scaling of Ilms using particle-based monte carlo methods. arXiv preprint arXiv:2502.01618
12

---

[38] Romano, Y., Patterson, E., and Candes, E. (2019). Conformalized quantile regression. Advances neural information processing systems, 32.
[39] Schaeffer, R., Kazdan, J., Hughes, J., Juravsky, J., Price, S., Lynch, A., Jones, E., Kirk, R., Mirhoseini, A., and Koyejo, S. (2025). How do large language monkeys get their power (laws)? arXiv preprint MrXiv:2502.15522
[40] Sharma, A., Azizan, N., and Pavone, M. (2021). Sketching curvature for efficient out-of- distribution detection for deep neural networks. In Uncertainty in artificial intelligence, pages 1958-1967. PMLR.
[41] Shelmanov, A., Tsymbalov, E., Puzyrev, D., Fedyanin, K., Panchenko, A., and Panov, M. (2021). How certain your transformer? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1833-1840.
[42] Shen, M., Das, S., Greenewald, K., Sattigeri, P., Wornell, G., and Ghosh, S. (2024). Thermome- ter: Towards universal calibration for large language models. arXiv preprint arXiv:2403.088//
[43] Snell, C.,, Lee, J., Xu, K., and Kumar, A. (2024). Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408. 240.:22288.333444
[44] Song. M., Su. Z., Qu X., Zhou, J., and Cheng, Y. (2025). Prmbench: fine-graine grained and challenging benchmark for process-level reward models. arXiv preprint arXivv2501.03/22
[45] Soto, A. (2005). Self adaptive particle filter. In IJCAL, pages 1398-1406. Citeseer.
[46] Tian, K., Mitchell, E., Zhou, A.,
Sharma, A., Rafailov, R., Yao, H., Finn, C.,, and Manning, C. D. (2023). Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint 2rXiv:2305.1449
[47] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B.. Goyal, N., Hambro, E., Azhar, F., et al. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397/
[48] Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, I.., Creswell, A., Irving, G., and Higgins, I. (2022). Solving math word problems with process-and outcome-based feedback. arXiv preprint 1.
[49] von Werra, L., Belkada, Y., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K., and Gallouédec, Q. (2020). Trl: Transformer reinforcement learning. https: //github. www.ougg Soon/uuggingaac//tri
[50] Wang, P., Li, L., Shao, Z., Xu, R. X., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. (2023). Math-shepherd: Verify and reinforce 1lms step-by-step without human annotations. CoRR, abs/2312.08935.
[51] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022). of-thughtt prompting elicits reasoning large language models. Advances in neural information processing systems, 35:24824 24837.
[52] Xia, S., Li, X., Liu, Y., Wu, T., and Liu, P. (2024). Evaluating mathematical reasoning beyond accuracy. arXiv preprint arXiv:2404.0566
[53] Xie, J., Chen, A. S., Lee, Y., Mitchell, E., and Finn, C. (2024). Calibrating language models with adaptive temperature scaling. arXiv preprint arXiv:2409. 198177
[54] Yadkori, Y. A., Kuzborskij. I.. György, A., and Szepesvári, C. (2024). To believe or not to believe your 1lm. arXiv preprint arXiv:2406.0253
[55] Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J.. Zhang, J., Yang, J., Yang. J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue. M., Zhang, P.. Zhu, Q.. Men, R., Lin, R., Li, T., Xia, T., Ren,
X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. (2024a).
5 technical report. arXiv preprint arXiv:2412.1511
13

---

[56] Yang. A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J.. Lin, J.. al. (2024b). Qwen2. 5-math technical report: Toward mathematical expert model via self- improvement. arXiv preprint arXiv:2409.12122.
[57] Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. (2023), Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36: 1809-11882.
[58] Ye, Z., Melo, L. C., Kaddar, Y., Blunsom, P., Staton, S., and Gal, Y. (2025). Uncertainty-awer wise verification with generative reward models. arXiv preprint arXiv:2502.11250.
[59] Yu, Z., Xu, T., Jin, D., Sankararaman, K. A., He, Y., Zhou, w., Zeng, Z., Helenowski, E., Zhu, C., Wang. S., al. (2025). Think smarter not harder: Adaptive reasoning with inference aware optimization. arXiv preprint arXiv:2501.17974.
[60] Zadrozny, B. and Elkan, C. (2001). Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Ieml.
[61] Zadrozny, B. and Elkan, C. (2002). Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages
[62] Zhang, Z., Zheng, C., Wu, Y., Zhang, B., Lin, R., Yu, B., Liu, D., Zhou, J., and Lin, (2025).
The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07001.
A Extended Related Work
Inference-time scaling, Recent studies have demonstrated that the capabilities of LLMs can be significantly enhanced by employing multi-stage reasoning, rather than directly generating answers in single step [51, 24]. Although such methods typically require greater computational resources at inference time, performance improvements can substantial by decomposing problems into simpler sub-tasks [1]. This strategy can be further improved by expanding the reasoning process: instead of relying solely on single-pass decoding, recent inference-time scaling techniques sample multiple candidate outputs and aggregate them for increased robustness [57, 43]. Those approaches-rangiin from majority voting verifier-based selection [6. 30, 4J. as well sophisticated Monte Carlo search algorithms [15] employing reward models [48, 50]-have demonstrated significant advances, particu- larly in reasoning-intensinn intensive tasks such as mathematical problem-solving. Nevertheless, most existing research has emphasized performance enhancement via extensive inference-time computation, while its reliability and cost-effective utilization of computational resources remain relatively understudied.
Process Reward Models. PRMs are specialized inference-time scaling tools designed to enhance the reliability of LLMs by verifying each intermediate step of their reasoning processes, rather than evaluating only the final outcomes [62]. Training PRMs typically involves step-labeled datasets, which are generated either through detailed human annotation or via automated methods like Monte Carlo rollouts sampling multiple solution trajectories to assess correctness probabilistically and evaluations by other large language models acting as judges [55, 50]. Exemplifying this approach, Qwen-PRM [55] integrates rigorous consensus- filtered labeling strategy that combines both human judgments and automated assessments, enabling it to achieve state-of-the the-art accuracy in step- wise reasoning verification tasks [44]. Alternatively, Shepherd- [50] illustrates that purely automated labeling approaches, while somewhat less precise in pinpointing individual reasoning errors, still significantly enhance overall model performance, showcasing their practicality and scalability, Recently introduced PRMs, such ReasonEval [52], extend beyond basic validity checks by incorporating redundancy evaluation-asses assessing if reasoning steps are unnecessarily repetitive or redundant, thus further improving robustness and effectiveness in LLM inference.
Adaptive sampling for efficient state estimation. Recent work by Puri et al. [37] suggests that LLM reasoning processes can be framed as particle filtering- sampling- based, probabilistic state estimation approach. Classical state estimation literature has explored methods for dynamically adjusting sample sizes based on state uncertainty [12, 13, 45, 10]. The core principle behind
14

---

these adaptive sampling methods is to decrease sample sizes when uncertainty is low; this reduces computational costs while maintaining accuracy, particularly important for resource-consttraned environments such as mobile robotics. Similarly, information-drive
adaptive strategies have been studied in contexts of planning algorithms [18, 5, 29]. However, to the best of our knowledge, this paper represents the first exploration of mnstance-adaptiv sampling strategies within the context of LLM inference-time scaling.
We would like to remark that Yu et al. [59] also tackles the adaptive reasoning problem, and their approach shares similar spirit with ours. Their adaptation, however, is based on the initial success probability (i.e.. question difficulty), whereas our method additionally adapts at intermediate reasoning steps (and thus is extendable to beam search). Specifically, our approach is enabled by calibrating PRMs, which allows us to estimate success probabilities throughout the multi-step reasoning process- direction that has not been explored before. Furthermore, while they adapt the reasoning length, our method focuses on controlling the reasoning width (i.e., sample size), which we believe is a promising avenue to explore for integrating both approaches.
Reliability of LLMs. Although state-of-the-ee inference- time scaling methods frequently rely on reward models, the reliability of these models has been relatively less explored. Johnson et al. [21] introduced the theoretical framework Experts Don't Cheat, proposing that reliable models generate predictions independent of paired hints when confident about their input; Yadkori et al. [54] further verified this idea extends to LLMs. Parallel research focuses on the concept of semantic entropy (SE) [25, I1], identifying hallucinations through contextual inconsistencies. Ye et al. [58] subsequently expanded SE to assess the reliability of process reward models. While these studies have made strides in quantifying LLM reliability, they do not specifically integrate reliability assessment into inference- time scaling strategies. Furthermore, rather than relying PRM estimate success probability (i.e., uncertainty), future work could explore uncertainty quantification tools (e.g.. [14, 27, 40, 41, 36, 35]). There is also line of calibration-orientedd work on LLMs,, such Shen et al. [42], which presents another promising avenue to explore within our framework.
B Omitted Proofs
B.1 Proof of Proposition 1
Here, we provide a proof of Proposition 1.
Proposition 1. For every p E (0, 1) and C' E (0,
set NIAS(p, C?) e log(1 - CC)
then [NAAS o9 2 (p.
1og(1-p)
In other words, if the PRM could perfectly distinguish correct from incorrect reasoning, then selecting the best trajectory among NisS(p, C)) samples (i.e., "best-of-Nias guarantees an average accuracy of (at least) C for questions whose per-trajees per-trajectory success probability is p.
Proof. Draw trajectories independently. The probability that none is correct is
Pr[no successes] = (1 - -
Therefore, the probability of at least one success
Pr[at least one success]
= -1-(1-2)
We require
1 (1 2 8 + (1-p)* < 1-6 8.
Since log(1 p) < 0, taking logarithms we have
log( [1 8) log p)
N log(1 10ppp) p) < log(1 8)
N
Finally, setting N NpAS(p. C)
log(1-C) log(1-p)' we have
log(1 C*)
Nmin (p, C²) =
log(1 p)
15

---

Finally, if one can sample N independent trajectories each succeeding with probability and perfect reward model always picks correct one whenever exists, then choosing
log((1 C)
log( 1 - p)
guarantees success probability at least C.
We also note that similar line of theoretical analysis has been explored in parallel by Schaeffer et al. [39].
B.2 Theoretical Justifications of IAS with Beam Search
now prove that the same "independent-samplingg bound (p, C*) underlies the instance- adaptive scaling strategies for beam search (BS+IAS). The key observation is that, if each sampled trajectory independently correct with probability p (conditional on the prefix), then total expansions under given K and M provide K x M independent tries.
Proposition 2 (BS+IAS-of M)) Assume K candidate prefixes, each of which independently yields correct continuation with probability at least Pmin- Fix a target overall success probability (0, 1)
and choose the per-prefix expansion width
NiASSPpinn
M
Then expanding all K prefixes by M continuations (for a total of K x M trajectories) and selecting with perfect reward model guarantees probability at least C that some trajectory is correct.
Proof. With K x M independent trials, the probability of zero successes is (1 Pmin ) If
KM 2 NIAS (Pmin; C), then
(1 Pmin j M
(1 Pmii (pmin))) S 1-C,
so the probability of least success is at least
1 (1 Pmin) KM 2 C.
Proposition (BS+IAS-of-K)). After expanding each of Kmax prefixes by M continuations, let P2 2 ... 2 PKmax
Define
KIAS
min(k] NIAS(PE, C) S x M S K K
Retaining the top KIAS prefixes and discarding the rest ensures overall success probability C.
Proof. For the top l prefixes, there are x M independent trials, each succeeding with probability at least Pi- If l x M 2 NIAS(PE, C), then
(1 (1- ) KM s (1 11)) (1App)Mu((Mn.0)) S 1-C,
so the probability of least one success is at least
1 (1 pk) RM
RMM 2 C.
B.2.1 Proof Theorem
We require the following thcorem from [38], which we slightly specialize to be one- instead of two-sided.
16

---

Algorithm One-Sided Split Conformal Quantile Regression (specialized from (38])
Require: Data (X,, Yi) E I&P IR, 1 i n; Miscoverage level E (0, 1); Quantile regression algorithm A (e.g. the QR method described in the main text").
1: procedure SPLIT CONFORMAL QR
2: Randomly split [I,...,n] n into disjoint sets Z₁ and Z2
3: Fit quantile function: g10) A(((X, Y Zi
4: for each i E T2 do
5:
Compute Fig = (X.) Yi-
6: end for
7: Compute Qı I2), the (1 a1( + 1/[ empirical quantile of (Ea : I2]
8: return Prediction interval C(x) [0(0) (x) Q1-a QI-a(E.I)) oo) for Xn-1
9: end procedure
Theorem 2 (Theorem of [38], specialized). Suppose we have an exchangeable validation set Vn = {X,, Y:J where X₁ are features and Y; are labels. Consider test point Xn+1 with unobserved true outcome Yn+1. (X,, i in interval (X++1) constructed by the (one-sided) Split Conformal Quantile Regression algorithm (Algorithm 1) satisfies
are exchangeable, then the prediction
P(Y,+1 C(Xn+1)) 2 aVVn)
We can now prove the theorem.
Proof of Theorem I. Follow setting of Theorem I, and consider that our exchangeable validation set Vn consists of data (X;, P.) where X; are partial reasoning traces and P; are observed future probabilities of success." We split Vn, at random into In which we use for training p(0), and Iz.
Using Theorem 2 and Algorithm 1, then have that for (exchangeable) test point (Xn+1, P Pat1),
P(Pa+1 (0)) (X Q1-a(E,I2)) a.
(1)
Purely for notational convenience, let's set (1L)) = p()) + ô where ó and a == 3 are constants chosen using I, such that (I with high probability. This step is not strictly necessary; simply it to avoid having to adjust existing quantile regressor. This guarantees that g(i) (Xm+1) - -1-3(EE 2 p(0) (xx+).. hence
P(P 2 p(0) (X,+1)) 2 1-
(2)
Now, consider that P.+1 is the probability that generation trial will succeed at recovering the correct answer, where here the probability is conditional X independent of the above success probability. By definition, the probability that at least of NIAS(PR+H) C') trials will succeed is at least C, and NIAS(: C)) is monotonically decreasing function in its first argument. Therefore, by (2), with probability least 1 - B,
NIAS (1A5( (Xn+1), 2 MAA((PHHI,
and since probability of least out of N success monotonically increases with N, with probability least 1 3.
P(success Dest-of-NIAs(FP) (X++1 C) 2 Plsuccess best-of--Nins((aut C) C,
and therefore by independence,
Pssuccess best-of-Nias(tt100 (Xn+1, C)[Xn+1) C(1-
17

---

p(1.2)
x
2(21
p(2.2)
p(2.)
Question (q)
p(L.1)
p(6.2)
X
LLM Forward Pass
PRM Estimation
(a) Forward Pass
x++2
Correct
Question (q)
(2)
.2) x
Correct
i
Zc+2
Correct
Prefix
.LLM Forward Pass
Monte Carlo Solution Verification
(b) Monte Carlo Estimation
Figure For each validation q. we first generate independent reasoning trajectories for 1.... Nyal- For each prefix trajectory, we conduct Monte Carlo simulations to estimate the success probability, f(s.1).
C Calibration via Fine- Tunn-Tuning
C.1 Calibration Set Collection
To evaluate the calibration quality of given LLM and PRM pair, we construct a validation set Dval Uge { ((f9),.)) g(9...0)) through a systematic three-stage process, as described in Section 3.1 of the main text. Here we provide additional implementation details and statistical analysis.
Data Collection Procedure. For notational simplicity, we omit the superscript in the following discussion, with the understanding that all operations are performed independently for each question E @val-
For each question q Qval- first generate Nval = 8 independent reasoning trajectories using (NNm)))))((( (illustrated as green arrows in Figure 5a). Next, for each reasoning trajectory (4) and each of its prefix trajectories Th where 1 1, T()) }, we generate NMC = 8 additional follow-up trajectories conditioned on both the question and the prefix
the target LLM: (x(1), x() ..... (Naai)
(i)
(green arrows in Figure 5b). By evaluating and counting the number of correct follow-up trajectories
'In practice, these may be noisy, but noise can be reduced with further Monte Carlo trials.
18

---

Z(,,), we estimate the PRM's predicted reward f(t.0) PRM zoat and the empirical success probability pli.t = Z(itt /Nuc (purple arrows in Figure 5a and blue arrows in Figure 5b).
(0)
Hardware and Software Configuration. All simulations are conducted using NVIDIA V100 32GB SMX3 devices with the vLLM inference acceleration framework [26]. We employ the standard sampling configuration: top-p 1.0, top_k _k = -1 (considering all tokens in the vocabulary). and temperature T 0.7 for all LLMs, consistent with best practices in recent reasoning model studies [2, 37].
C.2 Statistical Properties and Label Quality
Unbiased Estimation.
The per-prefix Monte Carlo estimator pli.t) is statistically unbiased for the true success probability. While individual estimates with NMc have non-negligible variance (o² = p(1 - p)/NMc R 0.02 for typical z 0.8), our approach compensates through scale aggregation.
Aggregate Accuracy. We collect between M 40,000 and 80,000 prefix-label pairs per PRM- LLM combination. Aggregating over this large number of samples substantially reduces the standard error of our calibration metrics. Specifically, the standard error of the Brier score (our primary calibration measure) scales approximately as:
20 + 40 MSE
SE
M
Nuc 0.02 represents the per-instance variance (assuming typical success probability p 0.8), and MSEpue 0.08 represents the model's intrinsic prediction error. With collection size of M 2 40,000, this standard error becomes sufficiently small that our reported calibration metrics statistically reliable, and the influence of per-instance noise global trends is negligible.
where o² p(1-p)
Impact on Quantile Regression. While the aggregate statistics are accurate, the instance-wise standard error does artificially inflate the variance of the target distribution in our quantile regression objective. Increasing NMC could potentially improve our method's 's performance by providing cleaner training signals. However, our experimental results demonstrate that our calibration approach is effective even with NMC = 8, achieving substantial improvements in calibration metrics across all evaluated PRM-LLM pairs.
C.3 Additional Implementation Details
Correctness Evaluation. Following established validation procedures from prior works [56, 2], determine correctness by parsing the generated answer within the \boxed delimiter and comparing it against the ground truth answer. Due to the mathematical nature of our evaluation datasets (MATH and related benchmarks), there minimal ambiguity in correctness determination answerss are either numerically or symbolically equivalent to the ground truth or they are not.
PRM Reward Calculation. The reward computation follows the official implementation provided by each PRM developer. In our experiments, we extract the final score predicted at the last reasoning step, as this approach has been shown to perform well in inference-time scaling regimes [2, 43).
For Qwen-PRM, we use the following prompt format. The system prompt is:
<I ili_start/ >system Please reason step by step, and put your final answer in \boxed(: im
The user prompt format is:
<I _limstarr/ >user I imdde) <lim_start Dassistaat ((Prefix)<ettra000 im |
19

---

where $Question represents the problem statement and $Prefix denotes the reasoning trajectory prefix (which is an empty string the case when applying IAS for the Best-of-N regime). The reward score is computed based on the token probability at the special textra_0> marker, which corresponds the final reasoning step. Similar prompt structures are used for other PRMs, following their respective documentation.
Data and Code Release. To ensure full reproducibility and facilitate future research, we pub- licly release our complete codebase, including all prompt templates, sampling configurations, and evaluation scripts. Our calibration datasets-compinee comprising (g. Fog prefix trajectory, and empirical success probability available for multiple reasoning LLMs at https: httpp://huggingffe cludes calibration data for all 18 PRM-LLM pairs evaluated in this work.
(9.8) p.a...) triplets of question,
co dattastts/youngg -arrs---aar//pppe baark/pra_caabbratiie.. calibration. The dataset in-
C.4 Baselines
C.4.1 Temperature Scaling
Typically, PRM prediction head outputs two logits, e-g. for "good" vs. "bad" categories. Let these logits be denoted by fgood and Losa- Given a temperature parameter T >> 0, the calibrated probability is computed as: f = softmax softmax([good/T1 fgood/T,, bbad/T) 'T' where the subscript 1 selects the "good" component. Choosing T by minimizing a calibration metric (e.g., Brier score) on validation data can align f with the empirical success probability (to some extent).
C.4.2 Isotonic Regression
Isotonic regression calibration fits a non non-parametric, monotonic mapping from raw scores (e.g.. PRM reward) to target values (c.g., success probabilities). The method applies the Pool Adjacent Violators algorithm (PAVA) [8] learn a nondecreasing piccewise-consee stretches or compresses score regions so that, within each fitted segment, the average predicted score matches the empirical success rate. Unlike parametric methods (e.g., temperature scaling), isotonic regression makes no assumptions about the shape of the calibration curve, allowing it to flexibly adapt arbitrary monotonic distortions in the model's outputs.
function. This mapping then
C.4.3 Histogram Binning
Histogram binning calibration partitions the prediction interval into a fixed number of equal-width bins, then replaces each raw score with the empirical success rate of its bin. At test time, each new prediction is assigned to its corresponding interval and is binned to that interval's 's mean. This simple porrmaemetriic approach corrects systematic over- and under-confidence without assuming any particular shape of the calibration curve.
C.5
Calibration via Fine-tuning
Advantage of fine-tuning. Since baseline methods correct calibration by uniformly shifting or rescaling the output probabilities (or logits) of PRMs, they are effective only when every instance's s prediction is consistently overestimated or underestimated. In contrast, fine-tuning approaches leverage the capability of LLMs to capture contextual information such as question categories and the position of the current reasoning step, enabling more comprehensive calibration. For these reasons, we empirically observe that the simple baseline calibration methods are not adequate for PRM calibrations.
Parameter-efficien fine-tuning. To address the issue, we adopt fine-tuning approach, arguably the most fundamental and straightforward method to calibrate the model. Specifically, we apply LoRA [19] with a rank of 2, dropout rate of O. and scaling factor of 32. To further reduce the number of trainable parameters, we apply LoRA only for the query and value matrices in every fourth decoder layer as well as the prediction head.
The LoRA approach offers benefits for PRM calibration beyond its computational efficiency compared to full fine-tuning: it enables access to both the original and calibrated PRM scores in single forward
20

---

pass. This allows for hybrid strategies (e.g., using the calibrated score for IAS and the original score for ranking candidate reasoning), without incurring significant additional computational cost.
Quantile prediction. For quantile regression, we modify the output head by replacing the final softmax linear layer with a sigmoid layer, producing outputs for each desired quantile (e.g., quantiles corresponding to 10%, 50%, and 90% in our experiments).
To ensure the initial predictions remain consistent with the original softmax outputs, we initialize the sigmoid-based output head such that the sigmoid probability for each quantile matches the softmax probability of the "good" token from the original model.
More specifically, assuming two-token prediction scenario with logits 20 and 21, then softmax probabilities for two classes 0 (i.e., "bad") and (i.e., "good") are given by:
e
e0
ply 1) e eal
and ply = 0) e e²1
The probability for class simplifies to sigmoid function:
plu =
0 - -20)
e0
e
1 1
Thus, by equating the sigmoid input with the logit difference 21 2o. we maintain consistency in initial predictions when transitioning from softmax to sigmoid-based output layer.
In practice, this corresponds to initializing the weights of the new linear sigmoid output layer W as the difference between the original linear layer weights for the "good" token Wi and the "bad" token W2, i.e., W <- Wi- W2. Similarly, the bias term b, if it exists, is initialized the difference of the biases from the original linear layers, i.e., b + b b2- This ensures the initial logit input to the sigmoid function directly matches the logit differences obtained from the original softmax-based model. Fine-tuning is performed using NVIDIA A100 40GB SXM4 GPUs and the TRL library [49].
C.6 Model Calibration
Calibration error metrics assess whether a model's predicted probabilities are consistent with observed accuracy. Two widely used calibration metrics are the Brier score [3] and the expected calibration error (ECE) [32]. Brier score quantifies the mean-squared difference between predicted probabilities g(()) and ground truth labels y( over N instances, where I = I.....N:
N
Brier =
Σ (g 1 ))
ECE evaluates the discrepancy between predicted confidence and empirical accuracy by partitioning predictions into B confidence bins .B, BB bin. Denote the set of indices whose confidences fall into bin B, by Ig. Then
} and averaging the absolute difference within each
Ie N [B.) acc(B.))
ECE Σ
where conf(B,) = [.,] Liez, (() is the average confidence in bin and acc(BB) = [T Eiez, n0
the empirical accuracy. Variants of ECE include adaptive calibration error (AdaptiveCE) [34] and average calibration error (AverageCE) [33], which uses adaptive bin intervals and equal weighting across bins, respectivley.
D Experiment Setup
We evaluate our method on two mathematical reasoning benchmarks: MATH500 [17] and AIME24 25 (i.c.. AIME2024 and AIME2025). MATH500 is a 500-example subset of the MATH dataset, avail- able at
co/ co/attasets /twptt//gggggggggg/gg////////// poses, we also constructed our MATH500 wwtp::0000alliattiaa set by randomly subsampling 500
500. For calibration pur-
21

---

problems from the original MATH training dataset, available at https: sttps://huggingtace co/ datasets/ hwtpss/hendrycckeeee cttpdrryckk/oppeiiiicccc https//ptpeeeoooonnieeeeeeeeeeee
The AIME (American Invitational Mathematics Examination) consists of 15 questions per test, with two tests administered each year. Thus, we use 30 questions from 2024 and 30 from 2025, totaling 60 problems, available https: / //huggingface http://aatteeteeuuuggggggge aceeei/nnaaa//aaaaaa aime_2022 and https: http:///uggggfaaeeeeeee co/ datasets/ dataseta/openccomppattt https//pppooccaaaaaaaa//222 respectively.
To demonstrate the efficacy our method across diverse LLMs, we evaluate six prominent models that vary in architecture, parameter count, and training paradigms (instruct-tuned vs. RI RI-disiilee;): Llama- 3.2-IB and Llama-3. 1-8B-Insttuct [47], Qwen2.5-Math-11. and Dwen2.5-7B-Insttee ---iceei-----ttilll---
[55], and [9]. We use a standard temperature setting of 0.7 and adopt the inference prompt template specified Puri et al. [37] across all models.
DeepSeek-RI- R1-Distill-Llama and
We use Aien2.5-Math-PRMM.. the top-performing open-source small-sized PRM in PRMBench [44]. We present additional results ShepherddMMistre
[62] as the primary PRM throughout the main manuscript, as it was
for RcasonEval-7B [52] and analyses in the subsequent Appendix section. We adhered the official prompting protocols for each PRM, as demonstrated on their respective websites.
[50] PRMs, along with comprehensive
We use NVIDIA V100 32GB SMX3 devices for every experiment except for quantile regression fine-tuning.
E
Additional Results
E.1
Calibration Analysis of Larger-Scale PRMs
We extend calibration analysis to larger-scale process reward models by evaluating the 72B parameter variant on both MATH500 and AIME24 25 datasets. This analysis addresses whether the calibration challenges observed in 7B models persist at larger scales, and whether our calibration method remains effective across different model sizes.
Table 4 and Figure present a comprehensive comparison of calibration performance across three configurations: uncalibrated 7B models, uncalibrated 72B models, and our calibrated 7B models. We evaluate six different base language models across two datasets, reporting both Brier score and Positive Brier score metrics for each configuration.
The results demonstrate that while larger 72B models exhibit reduced overestimation compared to their 7B counterparts, they still suffer from significant miscalibration issues. This observation is expected, as both model were trained with same policy model.
These results validate two key insights: (1) the calibration problem persists across model scales and cannot be solved simply by increasing model capacity, and (2) explicit calibration methods can effectively address miscalibration. Notably, our calibrated 7B model not only outperforms the uncalibrated 7B baseline but also surpasses the much larger uncalibrated 72B model, demonstrating that targeted calibration is more effective than naive scaling.
E.2 Histogram of Estimation Error
We examine the histogram of the deviations of the estimated reward from the true success probability (f(1.)) - -(),0), with both uncalibrated and calibrated PRMs.
As shown in Figures and 8, the error densities of uncalibrated PRMs are skewed to the right, with minimal mass on the left, indicating consistent overestimation. This effect is further amplified when weaker LLMs are used to generate responses, or when evaluated on more challenging datasets such as 21ME24-25.
It is noteworthy that native PRMs designed estimate success in model-agnostic manner; consequently, they tend to overestimate success probabilities for weaker models, resulting in inflated calibration errors. However, calibration effectively mitigates the issue and transforms the PRM error histograms into approximately unbiased zero-mean distributions across different models and dataset distributions.
22

---

Table Calibration performance comparison across model scales on MATH500 and AIME24-25
datasets (lower is better). Best results for each metric within each row are shown in bold. Results demonstrate that explicit calibration of Dowen-PRM-77 models outperforms even uncalibrated Qwen-
PRM-72B models, highlighting the importance of targeted calibration over naive model scaling.
Uncalibrated 7B
Uncalibrated 72B
Calibrated 7B
Dataset
Model
Brier
Pos. Brier
Brier
Pos. Brier
Brier
Pos. Brier
Llama-3,24 3,2-IB
0.241
0.223
0.211
0.078
0.069
0.047
L.ama-3.1-8B
0,205
0.177
0.177
0.111
0.121
0.099
MATH500
Owen-2.5-1.5B
0.154
0.130
0.155
0.106
0.127
0.107
Qwen-2.5-7B
0.101
0.085
0.118
0.057
0.082
0.053
RI-Llama-8B
0.161
0.114
0.193
0.072
0.089
0.055
RI-Qwen-7B
0.148
0.106
0.176
0.075
0.083
0.058
Llama-3.2-1B
0.194
0.192
0.157
0.063
0.003
0.001
Llama-3.1-8B
0.227
0.223
0.183
0.122
0.041
0.035
AIME24-25
Qwen-2.5-1.5B
0.330
0.322
0.256
0.176
0.073
0.053
Qwen-2.5-7B
0.289
0.282
0.205
0.139
0.072
0.066
R1-Llama-&B
0.385
0.371
0.237
0.204
0.078
0.030
R RI-Qwen-7B
0,414
0,402
0.260
0.215
0.069
0.034
Pww-78MM728 (Uncahirraed)
QwM-PRM-728 (Uncalibrated)
5
LMme-118
L3mma118
Llama-88
Llama-88
Owen--1
Owwn-1 SB
Qwen--7
Qwenn-78
Density
RI-O RI-QQwin-7B
Liam-lamaaa
Density
R. R1--Qwan-78
Llama
2
-1.0
-055
1.0
-1.0
-0.5
0.0
1.0
(Estimated Reward Success Prob.)
(Estimated Reward Success Prob.)
(a) Uncalibrated Qwen-PRM-72B
(b) Uncalibrated Qwen-PRM-72B
Figure 6: Histogram of signed deviation (i.e., estimation error) for Qwen-PRM-722 the MATH500 (in-ditributtio)) and AIME24-25 (out-of-distribution) datasets. Positive error indicates overestimation. While larger 72B model exhibit reduced overestimation compared to their 7B counterparts, they still suffer from significant miscalibration issues.
E.3 Comparison with Calibration Baselines
In this section, present calibration results for our method and several baselines across three PRMs. In addition to reporting the average Brier score, we also plot the 95% confidence intervals. The results shown in Figures 9, 10, and 11.
The Shepherd PRM exhibits a trend similar the Qwen PRM: our method consistently outperforms the baseline methods, whereas the baselines perform poorly on the out-of-distributio A AIME24-25 dataset. The ReasonEval PRM, which specifically trained to produce more robust outputs (especially for intermediate steps), appears to be already well-calibrated to some extent. Nevertheless, our method consistently surpasses the baseline approaches. Interestingly, despite its initially poor calibration, the Qwen PRM achieves better final calibration quality than both Shepherd and ReasonEval after fine-tuning.
We additionally compare our method against Adaptive Temperature Scaling (ATS) [53, 42], a recent calibration method that learns instance-spete speecif temperature parameters. Since ATS originally designed for standard language model outputs rather than PRM regression-style scores, we adapted for our setting. Specifically, temperature parameters based on the PRM- -eMerrrrrteed hidden states of both the question and solution prefix, enabling context-aware calibration similar to approach.
modified ATS to train transformer head that dynamically selects
23

---

Table 5: Calibration error for various methods and models on MATH500 (in-distribution) and AIME24-25 (out-of sitt----ditribution) Values are presented as adaptive calibration eπror (AdaCE) and
Brier scores, where lower is better. The best result for each metric within row is bolded. We use Acronyms: TS (Temperature Scaling), ATS (Adpative Temperature Scaling), IR
(Isotonic Regression), and HB (Histogram Binning).
Dataset
Model
Uncalibrated AdaC
Brier
AdaCE
TS
Brier
AdaCI
ATS
Brier
AdaCIE
IR
Brier
AdaCI:
HB
Brier
Calibrated (Ours)
AdaCI:
Brier
1.2m0-3.211B
.283
.241
.265
.190
.181
.099
.212
.097
.213
.081
.069
Llama-3.1-SB
NATR500
.274
.155
Owen-2.5.77
R1-1
.211
RI-Qwen-7B
.241
.148
202
.140
.178
.03
.289
.132
.289
.132
.107
.083
Llama-3.2-11 2-18
.266
.194
210
.193
.125
.65
.177
.665
.177
065
.011
.003
Llama-3. 3.1-8B
284
.227
248
.066
.881
.125
.392
.215
.396
.215
.086
.041
AIME24-25
Owen-2.5-7B
Qwen-2.S-1.58
.00
.355
.289
.350
.93
354
2419
.93
271
367
.16
.844
.477
.390
.250
.331
.04
.86
.357
.611
.098
.105
.072
.073
RI-Llama-SB
.226
385
528
310
438
220
.900
400
.588
.399
.188
.078
R1 J1-Qwen-77 7B
.558
.414
570
340
.30
.300
.090
.069
Table presents comprehensive comparisons across three calibration metries. The modified ATS consistently outperforms simpler baselines, validating the importance of context-awand calibration. On the in-distribution MATH500 dataset, ATS achieves competitive performance- for instance, reducing AdaCE from 0.241 to 0.178 for RI-Qwen-7B. However, our method achieves superior calibration with AdaCE of 0.107 for the same model.
The performance gap becomes more pronounced on the out-of-distribution AIME24-25 dataset. While ATS provides improvements over uncalibrated models (e.g., reducing AdaCE from 0.558 to 0.440 for RI R1-Qwen-7B), our method demonstrates substantially better generalization with an AdaCE of only 0.090. This pattern holds consistently across all models and metries. For example, on Llama-3.2 -.2-a-3.2-1B IB, our method achieves an AdaCE of 0.011 compared to 0.125 for ATS, and a Brier score of 0.003 compared to 0.065.
E.4 Inference-time Scaling with IAS
We first reiterate that our goal is "not" to introduce a new inference-time scaling method that universally outperforms existing approaches. Instead, we aim to enable inference-time scaling to allocate compute budgets dynamically based on a model's estimated likelihood of answering correctly. The proposed IAS strategy enables us to adaptively determine, on a per-instance basis, the number of samples that best balance accuracy and computational cost.
IAS for best-of-N. We first present results and analysis of the proposed IAS strategy across different LLMs and PRMs. Specifically, report performance under the best-of-N framework in Tables 2, 6, and 7. To ensure statistical significance, the pass@ @ rate is estimated by averaging accuracy over 64 independent forward passes. Similarly, for the IAS approach, report the score based on 100 different trials for each question, based on the determined NIAS value computed individually per question.
As observed, the IAS approach adaptively allocates the sampling budget while largely preserving overall accuracy. Ironically, the ReasonEval PRM performs poorly under the best-of-N (BoN) strategy, often yielding results worse than the pass@1 I rate. However, this is not limitation of the IAS strategy itself, but rather a reflection of the ReasonEval PRM's weakness in selecting the correct answer from among candidate responses. Notably, the budgets determined by IAS for ReasonEval comparable to those derived from the other PRMs (See Figures 12, 13, and 14 for how effectively IAS adaptively selects N.). In contrast, PRMs like Qwen exhibit strong ranking capabilities despite poor initial calibration. Our calibration method proves particularly effective in such cases, as it preserves the PRM's original strengths while enabling effective IAS and improving its reliability.
IAS for beam search. To further demonstrate the effectiveness of the proposed LAS approach, we evaluate it under beam search setup, which requires PRM evaluation over intermediate reasoning trajectories. However, due to limitations of the vLLM acceleration support fine-tuned custom models or provide access to internal states inference was significantly slower than simple forward passes. In our setting (N = x 8 = 64. M = 8) and computation resource (i.c., V100 gpus), single search over 500 examples took approximately 10 hours, resulting
cerewwoorwwiiee does not
24

---

in each experiment requiring over 200 hours to complete. To mitigate this computational burden, we adopted a simplified approach: instead of performing beam search every step. we applied it every five steps. Given that typical reasoning trajectories span 20 or more steps, this amounted to roughly five beam search applications per example. Due the computational burden, we were only able to perform a single trial for each beam search experiment.
Tables 3, 8, and 9 present the pass@ accuracy for standard beam search (BS), BS with the proposed IASoK, and BS with IASoM, evaluated using the Qwen, Shepherd, and ReasonEval PRMs. For the Shepherd and ReasonEval PRMs, conduct experiments using the MATH100 (the first 100 examples among MATH500) and AIME2024 datasets. To ensure a fair comparison, all experiments were conducted using the calibrated PRM. The results are consistent with the observations made for the Qwen PRM, as discussed in the main text.
E,5 Ablation Studies on IAS Parameters
We provide an ablation study on two hyperparameters that govern the trade-off between computational cost and accuracy, under the best-of-N setup.
target probability (C). The target probability serves as a parameter that acts as a constant multiplier the expression 1/ log(1 p). We present results for values of C such that log(1 C) takes on values -0.125 12,, -0.25, -0.5, -1.0, -2.0, -3.0, -4.0, and -5.0. Figure 15 shows the budget-accuracy trade-off plots. We use C 0.99, it appears to provide reasonable balance between accuracy and computational cost.
A quantile parameter (3). The parameter 3 represents a quantile that controls how conservatively
we estimate the success probability. We report results for 3 = 0.1, 0.5, and 0.9. Figure 16 presents the budget-accuracy trade-off plots. We adopt conservative setting of 5 0.1, as it consistently achieves strong performance. However, we note that even the 90% quantile (3 = 0.9) often yields reasonably high accuracy with significant budget savings, highlighting the flexibility of the approach.
Again, as also shown in Figures 12, 13, and 14, accuracy tends increase monotonically with higher inference budget, thus a "sweet spot" doesn't 't exist. Exploring a principled metric to assess the cost-performance ratio within the inference- scaling framework remains an interesting direction for future work.
25

---

QwenPRM-7B (Uncallbrated)
QwenPRM-78 (Callbrated)
Liama-18
Llama-1B 1B
LJame-88
Llama-8B
Owen-1.58
Owen-1.58
Qwen:7B
Qwen:BB
4
RI-Liama-8B
A1-LLlma-68
Density
Ri Qwen: 7B
Density
R1 R1-Qwen-7B
2
2
0
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
(Estimated Reward Success Prob.)
(Estimated Reward Success Prob.)
(a) Uncalibrated Qwen-PRM
(b) Calibrated Qwen-PRM
Shepherd-7B (Uncalibrated)
Shepherd-P7 (Calibrated)
3
Llama-10
Laama
Llama-15 BB
Qwen-1.5B 1.58
Qwen- 7B
Qwen-1.5B
RI Lllama 8B
Qwen-7B
Density
R1 Qwen-77
Density
RU--Llmms88
R1-Qwen-7B
0
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
(Estimated Reward Success Prob.)
(Estimated Reward Success Prob.)
(c) Uncalibrated Shepherd-PRM
(d) Calibrated Shepherd-PRM.
ReasonBval- 7B (Uncalibrated)
ReasonEval-P7 -B (Calibrated)
6
Iláma-1B
Llama 1B
Liama-88
Llama-88
Qwen-1:5B
Owen-7B
Owen-7B
Donsity
4
R1-L.amm--8B
Density
R1-Llama-88
Ri Owen
4
2
0
-1.0 -0.5
0.0
0.5
1.0
1.0
-0.5
0.0
0.5
1.0
(Estimated Reward Success Prob.)
(Estimated Reward Success Prob.)
(e) Uncalibrated ReasonEval-PRN.
(I) Calibrated ReasonEval-PRM
Figure 7: Histogram of signed deviation (i.c., estimation error) for Qwen, Shepherd, and ReasonEval PRMs evaluated on the MATH500 (in-distribution) datasets. Positive error indicates overestimation. As shown, uncalibrated PRMs- left column- to be optimistic (e.g., exhibiting higher density on the right side compared to the left and/or forming noticeable peak around 1.0). This becomes particularly pronounced for weaker models. In contrast, our calibration approach-the right columm--transfrmin the error histograms into unbiased zero-mean distributions, consistently across various models.
26

---

QwenPRM-7B (Unealibrated)
QwenPRM-7B (Calibrated)
4
Liama-18
Llama-18
IJáma-88
Elama-88
3
Qwen-1.58
30
Qwen-115B
Qwan-7B
Owen-7B
Ri Rd-LLama--BB
RI H1-LLama-88 88
Density
RI-Qwen-78
Density
20
R1-Qwen-78
2
1
10
0
0
1.0
0.5
0.0
0.5
1.0
1.0
0.5
0.0
0.5
1.0
(Estimated Reward Success Prob.)
(Estimated Reward - Success Prob.)
(a) Uncalibrated Qwen-PRM
(b) Calibrated Qwen-
Shepherd-7B (Uncalibrated)
Shepherd-78 (Calibrated)
2.0
Llama-18
Llama-11
Llama 8B
Llama-81
Qwen-1.50
Owen-115B
1.5
Qwen-7B
20
Qwen-7B
R1-Llama-8B
R1-Llame-88
Density
R1-Qwen-7B
Density
R1 11-Qwen-78
1.0
10
0.5
0.0
-1.0
-0.5
0.55
0.5
1.0
(Estimated Reward - Success Prob.)
(Estimated Reward Success Prob.)
(c) Uncalibrated Shepherd-PRM.
(d) Calibrated Shepherd-PRM. ReasonEval- (Calibrated)
ReasonEval- (Uncalibrated)
1lama-10
Llama BB
1lama-88
6
Qwea:1.5B
20
Qwen-1.5B
Qwea-78
Qwen-7B
Density
Ri Qwen: 7B
lama-88
Density
15
R1-1Jama-88
7B
4-
10
2
5
0
0
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
(Estimated Reward Success Prob.)
(Estimated Reward Success Prob.)
Uncalibrated ReasonEval-PRM
(f) Calibrated ReasonEval-PRM
Figure 8: Histogram of signed deviation (i.e., estimation error) for Qwen, Shepherd, and ReasonEval PRMs evaluated on the AIME24-25 cout-of-distribution) datasets. Positive error indicates overes- timation. As shown, uncalibrated the left volumn-tenni to be optimistic (e.g., exhibiting higher density on the right side compared to the left and/or forming noticeable peak around 1.0). This becomes particularly pronounced for more challenging, out-of-distribution problems. In contrast, our calibration approach-tle the right couum--transfor the error histograms into unbiased zero-mean distributions, consistently across various models and dataset distributions.
27

---

Llama 3.2 Instruct
Llama 8B Instruct
0.5
0.5
Uncalibrated
Uncalibrated
Temp. Staling
Temp. Scaling
Isotonic Reg.
Isotonic Reg.
Brier Score
Hist. Binning
Brier Score
Hist. Binning
0.3
QR (Ours)....
0.3
(Ours)
0.1
0.1
AIME24-25
MATH500
AIME24-25
Qwen2.5-MMe MaMMaaah...----sseeee
Qwen2.5-Math 78-Instruet
0.5
0.5
Uncalibrated
Uncalibrated
Temp. Scaling
Scaling
0.4
Isotonic Reg.
0.4
Isotonic Reg.
Brier Score
Hist. Binning
Brier Score
Hist. Binning
0.3
QR (Ours)
0.3
QR (Ours)
0.2
0.1
0.1
0.0
MATH500
ALME24-25
MATH500
AIME24-25
DrepSock R1 Distill Laam 8B
DrepSeck R1 Distal --eniaall-Owenn 7B
Uncalibrated
Uncalibrated
Temp. Scaling
Temp. Scaling
0.4
Isotonic Reg.
0.4
Isotonic Reg.
Brier Score
Hist. Binning
Brier Score
Hist. Binning
QR (Ours)
QR (Ours)
0.2
0.2
0.0
0.0
MATH500
AIME24-25
MATH500
AIME24-25
Figure 9: Comparison of our method with popular calibration techniques temperature scaling, isotonic regression, and histogram binning on the Qwen PRM across MATH500 and AIME24-25.
28

---

Llama 3.2 Instruct
Llama 8B Instruct
0.5
0.5
Uncalibrated
Uncalibrated
Temp. Staling
Temp. Scaling
Isotonic Reg.
Isotonic Reg.
Brier Score
Hist. Binning
Brier Score
Hist. Binning
0.3
QR (Ours)....
0.3
(Ours)
0.1
0.1
AIME24-25
MATH500
AIME24-25
Qwen2.5- 1.-Maahh1.5B-nntrree
Qwen2.55Math- I--Insrr
0.5
0.5
Uncalibrated
Uncalibrated
Temp. Scaling
Scaling
0.4
Isotonic Reg.
0.4
Isotonic Reg.
Brier Score
Hist. Binning
Brier Score
Hist. Binning
0.3
QR (Ours)
0.3
QR (Ours)
0.2
0.1
0.1
0.0
MATH500
ALME24-25
MATH500
AIME24-25
DrepSock R1 Distill Llama
DrepSeek R1 Distall-Owee Owen-7h
Uncalibrated
Uncalibrated
Temp. Scaling
Temp. Scaling
0.4
Isotonic Reg.
0.4
Isotonic Reg.
Brier Score
Hist. Binning
Brier Score
Hist. Binning
QR (Ours)
QR (Ours)
0.2
0.2
0.0
0.0
MATH500
AIME24-25
MATH500
AIME24-25
Figure 10: Comparison of our method with popular calibration techniques temperature scaling, isotonic regression, and histogram binning on the Shepherd PRM across MATH500 and AIME24-22
29

---

Llama 3.2 Instruct
Llama 8B Instruct
0.5
0.5
Uncalibrated
Uncalibrated
Temp. Staling
Temp. Scaling
Isotonic Reg.
Isotonic Reg.
Brier Score
Hist. Binning
Brier Score
Hist. Binning
0.3
QR (Ours)....
0.3
(Ours)
0.1
0.1
AIME21-25
MATH500
AIME24-25
Qwen2.5-Math- 1at.55--Mtth...a. 1.atthh15BBInstruee
Qwen2.55Math- 18-Instruct
0.5
0.5
Uncalibrated
Uncalibrated
Temp. Scaling
Temp. Scaling
0.4
Isotonic Reg.
0.4
Isotonic Reg.
Brier Score
Hist. Binning
Brier Score
Hist. Binning
0.3
QR (Ours)
0.3
QR (Ours)
0.2
0.1
0.1
0.0
MATH500
ALME24-25
MATH500
AIME24-25
DrepSock R1 Distill Llmma 8B
DrepSeek R1 Distall-Owee Qwen-7h
Uncalibrated
Uncalibrated
Temp. Scaling
Temp. Scaling
0.4
Isotonic Reg.
0.4
Isotonic Reg.
Brier Score
Hist. Binning
Brier Score
Hist. Binning
QR (Ours)
QR (Ours)
0.2
0.2
0.0
0.0
MATH500
AIME24-25
MATH500
AIME24-25
Figure 11: Comparison of our method with popular calibration techniques temperature scaling, iso- tonic regression, and histogram binning- on the ReasonEyal PRM across MATH500 and AIME24-25
30

---

Table 6: Evaluation of the best-of-NN method with fixed-N strategy (BoN), compared against the proposed instance-adaptive sampling strategy (BoN+IAS), with calibrated Shepherd PRM. We report both accuracy and normalized computational cost (budget). Relative improvements over Pass @ accuracy are highlighted in light blue.
Baselines
w/ Uncal, PRM
w/ Calib, PRM
Dataset
Modell
Pass
BoN
BoN+IAS
Budget
BoN+IAS
Budget
Llama- 3.2- IB
0.2255
0.3800
0.3205
0.0852
0,3633
0.6379
LJama-3.1-80
0.4659
0.5620
0.5419
0.0852
0.5446
0.2481
MATH500
Dwen-225-1155
0.6970
0.7740
0.7518
0.0852
0.7414
0.1223
Owen-2.5-7B
0.7994
0.8460
0.8437
0.0852
0.8345
0.1048
RI-Llama-8B
0.6734
0.7640
0.7493
0.0852
0.7767
0.3106
RI-Qwen-7B
0.7556
0.8480
0.8137
0.0852
0.8206
0.2477
Llama-3.2 3.2-11..-BB
0.0042
0.0000
0.0067
0.1206
0.0000
0.9956
Llama-3 1-8B
0.0268
0.0333
0.0442
0.1206
0.0355
0.7320
AIME24-24 25
Qwen-2.5-1.3
0.0932
0.1500
0.1310
0.1206
0.1397
0.6159
Qwen-
0.0885
0.2000
0.1497
0.1206
0.1507
0.5740
RI-Llama-8B
0.0784
0.1333
0.1505
0.1206
0.1760
0.8664
RI-Qwen-7B
0.1411
0.3000
0.2263
0.1206
0.2755
0.8456
Table 7: Evaluation of the best-of-N method with fixed-N strategy (BoN), compared against the proposed instance-adaptive sampling strategy (BoN+IAS), with calibrated ReasonEval PRM. re- port both accuracy and normalized computational cost (budget). Relative impovementt/deeraaationn over Pass @ accuracy are highlighted light blue/red.
Baselines
w/ Uncal. PRM
w/ Calih, PRM
Dataset
Model
Pass @
BoN
BoN+IAS
Budget
BoN+IAS
Budget
Llama- 3, IB
0.2255
0.2900
0.2887
0.0773
0.3095
0.4672
Llama-3.1-8B
0.4659
0.4780
0.5055
0.0773
0.5007
0,2570
MATH500
Qwen-2.5-1.55
0.6970
0.7340
0.7400
0.0773
0.7286
0.1324
Qwen-2.5-7B
0.7994
0.7500
0.8129
0.0773
0.7948
0.1523
R1-Llama-8B
0.6734
0.5300
0.6335
0.0773
0.6142
0.2069
RI RI-Qwen-7B
0.7556
0.6280
0.7194
0.0773
0.7016
0.1653
Llama-3.2-1 1-BB
0.0042
0.0000
0.0050
0.1521
0.0000
0.9852
Llama-3. 1-8
0.0268
0.0333
0.0287
0.1521
0.0370
0.8773
AIME24-24 25
Qwen-2.5 L.5.LBB
0.0932
0.0833
0.0955
0.1521
0.0762
0.7430
Qwen-2.5-7B
0.0885
0.0500
0.1117
0.1521
0.0647
0.8471
RI-Llama-8B
0.0784
0.0333
0.0388
0.1521
0.0238
0.8263
R1-Qwen-7B 0.1411
0.0000
0.0807
0.1521
0.0468
0.8333
31

---

Table 8: Evaluation of the beam search method with fixed-NIM, compared against the proposed adaptive sampling strategy (IASoK and IASoM) using calibrated Shepherd PRMs. We report both accuracy and computational cost (budget), measured by the average number of L.M generations per question normalized by that a fixed-budget BS baseline. Relative improvements over the Pass accuracy highlighted in light blue.
Baselines
IAS w/ Calibrated PRM
Dataset
Model
Pass
BS
BS+IASoK
Budget
BS+IASoM
Budget
Jlama-3.2-1
0.2255
0.4000
0.4000
0.9549
0,4300
1.0634
1lama-3.1-88
0.4659
0.5700
0.5400
0.5377
0.5600
0.5902
MATH100
0wen-2.5-1.5B
0.6970
0.7900
0.7900
0.4806
0.7800
0.5014
Owen-2.5-7B
0.7994
0.7800
0.8900
0.5382
0.8900
0.5794
RI-Llama-8B
0.6734
0.7900
0.8200
0.5264
0.8300
0.5672
RI-Qwen-7B
0.7556
0.8600
0.8800
0.4063
0,9000
0.4983
Llama-3. 1B
0,0078
0.0333
0.0333
1.0000
0.0333
1.0000
Llama-3.
0.0479
0.1000
0,1000
0.8689
0.0667
0,9636
AIME24
Qwen-2.55 1.5
0.0969
0.1000
0.1000
0.7565
0.1000
0.8523
Owen-2.5 0.0979
0.1333
0.1333
0.7994
0.1333
0,9051
RI-Llama-8B
0.0656
0.0000
0.1000
0.8958
0.1000
0.9065
RI RI-Qwen-7B
0,1354
0.1000
0,3333
0,9917
0,3000
1.0713
Table Evaluation of the beam search method with a fixed-N/M, compared against the proposed adaptive sampling strategy (IASoK and IASoM) using calibrated ReasonEval PRMs. We report both
accuracy and computational cost (budget), measured by the average number of LLM generations per
question normalized by that of fixed-budget BS baseline. Relative improvementsdeerraationn over Pass accuracy highlighted in light blue/red.
Baselines
IAS w/ Calibrated PRM
Dataset
Model
Pass@1
BS
BS+IASoK
Budget
BS+IASoM
Budget
Llama-3.2-1B
0.2255
0.3400
0,4000
0.5030
0.3700
0.6730
Llama-3.1-8B
0.4659
0.5700
0.5700
0.3606
0.5700
0.4490
MATH100
Owen-2.5-1.5H
0.6970
0.7800
0.7700
0.4171
0.7900
0l.2281
Qwen-2.5-7B
0.7994
0.8800
0.8700
0.4664
0.8700
0.4840
RI-Llama-8H
0.6734
0.3500
0.7900
0.3379
0.8300
0.4088
RI-Qwen-7B
0.7556
0.4800
0.8900
0.3173
0.8900
0.3628
Llama-3.2-18
0.0078
0.0000
0.0000
0.9258
0.0000
0.9782
Llama-3.1-80
0.0479
0.0333
0.0000
0.4007
0.0333
0.7371
AIME24
Qwen-2.5-1.55
0.0969
0.1333
0.1000
0.3134
0.0667
0.5084
Qwen-2.5-7B
0.0979
0.1000
0.1000
0.3626
0.0667
0.5155
RI-Llama-88 8B
0.0656
0.0667
0.1000
0.7205
0.2000
0.8866
R1 21-Qwen-7B
0.1354
0.2000
0.2333
0.6203
0.3333
0.6861
32

---

Llama-3.2-1 1aana-3.2-B--nntttee I-Inntruct
Llama-3. 1-8 B-Innstruct
Average Accuracy
0.8
Average Accuracy
0.8
0.6
0.6
0.4
Level
Level
Level
0.4
Level
Level
0.2
Level
Level 5
-Fixod-N
Fixad-N
LAS
0.2
IAS
21
2
2
2
2
2
The Number of Samples (N)
The Number of Samples (N)
Owen2.5-Math-1 1.--aatt-..BB-nsstuuutt
Qwen2.5-Mathe B---MMath----ntnteeee
0.9
Average Accuracy
Average Accuracy
0.9
0.8
0.7
0.8
Level
Level
0.6
Level
Level
0.7
Level
Lovel 3
Level
Level
0.5
FFxed.N
LAS
IAS
0.6
25
2
2
2
The Number of Samples (N)
The Number of Samples (M)
DeepSeek-RI-Diee DistRll--iiitieeeeee
DeepSeek-RR RI-Distill -wenll
0.9
0.9
Average Accuracy
Average Accuracy
0.8
0.8
0.7
0.7
Level
0.6
Level
0.6
Level
Level
Leval 5
Pixed.NN
Frxed.N N
0.5
LAS
0.5
IAS
21
23
2
21 The Number of Samples (N)
23
2
The Number of Samples (N)
Figure 12: IAS scales with problem difficulty MATH500, with Qwen PRM. Average accuracy over the test points with varying difficulty levels (1: easy, 5: hard) is illustrated. Regular fixed-N and our IAS approaches are indicated by dashed lines and stars, respectively, and IAS allocates more samples to harder questions.
33

---

Llama-3.2 3-2-1 2-1---2---a-- B-Instruct
Llama-3.1 1-8B- 38-Instruct
0.8
Average Accuracy
Average Accuracy
0.8
0.6
0.6
0.4
Level
Laval
Level
Level
0.4
Lovel
Level
Level
0.2
Level
Level5 5
N
Paxed N
LAS
0.2
IAS
2
23
2
2
23
2
The Number of Samples (N)
The Number of Samples (M)
Qwwn22.5Matt--15BB 1w..--Mathh1.55-nnttrrc
Dwwen25-Matth-BB-iie
0.9
Averago Accuracy
Average Accuracy
0.9
0.8
0.7
Level
Level
0.6
Level
Level
Level
0.7
Lavel 3
Level
Level
0.5
LAS
IAS
0.6
21
23
2
2
23
2
The Number of Samples (N)
The Number of Samples (M)
DeepSeek-R1 -istRRRRI--ttttOO--a..
DeepSeek-Ri Sieek--R---iisstii -isstill-Qwen-71
Average Accuracy
0.9
Average Accuracy
0.9
0.8
0.8
0.7
Level
0.7
Level
Level
Laval
0.6
Level
0.6
Level
Level
Level 5
Fixed--N
Fixed-N
LAS
IAS
0.5
0.5
21
23
2
21
23
25
The Number of Samples (N)
The Number of Samples (N)
Figure 13: IAS scales with problem difficulty in MATH500, with Shepherd PRM. Average accuracy over the test points with varying difficulty levels (I: easy, 5: hard) is illustrated. Regular fixed-N and our IAS approaches are indicated by dashed lines and stars, respectively, and IAS allocates more samples to harder questions.
34

---

Llama-3.2 1.-----22---eeee B-Inssuuctt
Llama-3.1 1 38-Instruct
0.8
0.8
Average Accuracy
Average Accuracy
0.6
0.6
0.4
Level
Laval
Level
Level
0.4
Lovel
0.2
Level
Level
Level
Level 5
N
Paxed N
LAS
0.2
IAS
2
23
2
2
2
2
The Number of Samples (N)
The Number of Samples (M)
1rren2.-MMth-1.... -.-ttttt..tt
Qwen2255Mathh-7BBnssrre
0.9
Averago Accuracy
Average Accuracy
0.9
0.8
0.7
0.8
Level
Level
0.6
Level
Level 2
Level
0.7
Lovel 3
Level
Level
0.5
LAS
0.6
IAS
21
23
2
2
23
2
The Number of Samples (N)
The Number of Samples (M)
DeepSeek-RI Rii--R---Dllill-eee----
DeepSeek-Ri Baaic--R---iisssii DistilllQwenn-7
Average Accuracy
0.9
Average Accuracy
0.9
0.8
0.8
0.7
Level
0.7
Level
Level
Laval
0.6
Level
0.6
Level
Level
Level 5
Fixeed-N
Fixed-N
0.5
LAS
0.5
IAS
21
23
2
21
23
25
The Number of Samples (N)
The Number of Samples (N)
Figure 14: IAS scales with problem difficulty MATH500, with ReasonEval PRM. Average accuracy over the test points with varying difficulty levels (1: easy, 5: hard) is illustrated. Regular fixed-N and our IAS approaches are indicated by dashed lines and stars, respectively, and IAS allocates more samples to harder questions.
35

---

QwenPRM-7B
QwenPRM
L.ma-3.2-18
Liama
0.8
3.1ma331
0.25
3.1ma-3.100
Q.6.-2.5-1.5B
2.5:1.8
Q.en
0.20
Qwee 2.5/7
Accuracy
Llama 06
Accuracy
R1 11--aam:00
--wen-BB
R1-Qwen-78
0.6
0.15
0.5
0.10
0.4
0.05
0.00
0.2
0.4
0.6
0.8
0.6
0.8
1.0
Budget
Budget
(a) Owen-PRMM on XATH500
(h) Qwen-PRNn PRM on AIME24-25
Shepherd-7B
Shepherd-78
0.3
Uama-3.2:18 8
Llama-3.2-18
0.8
3.1-88
Owen L.558
Qwen 5-500
Accuracy
Qwen-2.5-78
Llamaa 8B
Accuracy
0.2
RL Llama 8D
0.6
Qwen
Owea
0.1
0.4
0.0
0.2
0.6
0.2
0.4
0.8
1.0
Budget
Budget
(c) Shepherd PReMpherd-PRMMMM-- on MATH500
(d) Shepherd- PRM on AIME24-25
ReasonEvvl-7B
ReasonEva)-7B
0.8
13am-33
0.10
11ama-3. 1-88
Qwen-2.5-1.50
Qwen-2.5
Accuracy
Rt at-Llama-ee
Accuracy
Qwes:
0.06
0.04
1lama3.2:11
0.4
Owea 2.5.1.3
0.02
Owen-2.5-P9
R1-1lamn-88
0.00
Qwen-7B
0.0
0.2
0.4
0.6
0.2
0.4
0.6
0.8
1.0
Budget
Budget
(e) ReasonEval- on MATH500
(f) ReasonEval-PRM on AIME24-25
Figure 15: Budget vs. accuracy plots MATH500 and AIME24-25 datasets with varying C.
36

---

OwenPRM-77
QwenPRM-78
1lama-32-11
0.25
0.8
3.1-8-311.88
LJama
2.55 1.50
Q.ee22..11.5
Owen-2.5-77
0.20
Qwen-2.6-78
Accuracy
0.7
Accuracy
Rt RI-Qwee- 70
RI Ql-n-eenn
0.15
0.6
0.10
0.5
0.05
0.4
0.00
0.2
0.4
0.6
0.4
0.6
0.8
1.0
Budget
Budget
(a) Qwen- MATH500
(b) Qwen- PreenPPRM AIME24-25
Shepherd-7B
Shepherd-78
0.8
1lama-3 1-88
Llama-3.1-8
Qwen-2.5-1.5
0.7
Qwea-2.5 2.5-70
0.2
Qwen-2.5-78
Accuracy
RI It-Llama
Qwea
Accuracy
R1 Llama
RI RI-QWEn-78
0.6
0.5
0.1
0.4
0.0
0.0
0.2
0.4
0.6
0.2
0.4
0.6
0.8
1.0
Budget
Budget
(c) Shepherd-PRM on MATH500
(d) Shepherd-PRe on AIME24-25
Evall 7B
Reason 7B
0.8
1Juma/3.2-4 2-1-8
0.10
Llama-3 3.2-18
13ama-3 1-88
0.7
Owen-2.5.11
0.08
Qwwin
Qween-2.577 7B
Owen 2.5
Accuracy
Accuracy
Ui--Lama-80
Itt-Owee:77 7B
0.06
R1-Owwn: 7B
0.5
0.04
0.4
0.02
0.3
0.00
0.1
0.2
0.3
0.4
0.2
0.4
0.6
0.8
1.0
Budget
Budget
(c) ReasonEval-PRRM MATH500
(f) ReasonEval-PRRM on AIME24-25
Figure 16: Budget vs. accuracy plots on MATH500 and AIME24 25 datasets with varying
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We explicitly summarize our main claim and contribution in the abstract and introduction based on our theoretical and experimental results shown in the subsequent sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have a separate "Conclusion and I Limitations" section. We also explicitly mention the focus of the paper, well as the limitations and potential directions based on the empirical findings.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well sell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
38

---

Answer: [Yes]
Justification: We formulated the concept mathematically, proposed theorems with clear statement of assumptions, and provided the corresponding proof in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversell, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The algorithm is clearly stated, and detailed experimental setups are provided in the main body and the Appendix. We will also publish code upon publication.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of flosed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
39

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We used public datasets and clearly stated which ones were used. We will also publish our code upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips. cc/
Soblliccguides/CdeSuutiiii soniesoobissiiioooccccc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/publicc htlpppbllguudleeeeeeeepppppppepe guides/ /Coo///omiissseeeeeeeeee wwwiissssspolioooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed experimental setups are provided the main body and the Appendix. We will also publish our code upon publication.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them. The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Although we could not report error bars for every result, we demonstrate statistical significance through: (i) presenting histograms of errors for more comprehensive view, (ii) presenting 96% for calibration comparison study the Appendix, (iii) using statistically sufficient number of test instances, and (iv) reporting the average performance over 100 runs for the best-of-N approach. For beam search experiments, due the high computational cost, we report results from a single run; however, we observe consistent trends across different models and present each result individually rather than aggregating them. Due to space constraints, we have described these details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
40

---

The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, or statistical significance tests, least for the experiments that support
the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the Appendix, we report the GPU specifications have used, and the ap- proximate time execution each experiment. We also briefly discuss the computational cost of training and inference.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eeerrppeeeeeeeeee http:////cpppp.ctctctttttttaaaaa
Answer: [Yes]
Justification: confirm that our research conforms, in every respect, to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The aim of research is to improve the reliability of large language models thus, it shows potential positive societal impacts itself and reliability could matter in practical usage.
and we further discussed how
41

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional
unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
42

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. con/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
43

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality the research, declaration is required.
Question:
Answer: [NA]
Justification: the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy /ttpp:://neuiia.... cc/Coofferences/20255 for what should or should not be described.
44


Reviews:
[{"review_id": "wjQ6unb7Nz", "reviewer": "Reviewer_dLMo", "summary": "This paper addresses the critical issue of poor calibration in Process Reward Models (PRMs), which are essential components in inference-time scaling algorithms for LLMs. The authors demonstrate that state-of-the-art PRMs systematically overestimate success probabilities, particularly for challenging problems and weaker models. To remedy this, they propose a calibration method based on quantile regression and introduce an instance-adaptive scaling (IAS) framework that dynamically adjusts computational budgets based on calibrated uncertainty estimates.", "questions": "1. Regarding Line 214, BoN typically involves predicting complete sequences, but according to Definition 2, we need to estimate probability $p$ based on already generated content. Given that sequences differ from each other, is the estimated probability $p$ based solely on the question? How do you obtain the reward for the entire trajectory from the calibrated PRM—do you predict directly at the final step or average across all steps?\n2.  We observe that on the more challenging AIME24-25 tasks, BS+IASoM shows remarkable improvements over BS (e.g., R1-Qwen-7B improving from 0.1333 to 0.3167). Since calibration shouldn't affect ranking order, what is the source of such significant improvements?\n3. How is the training data for calibration obtained? Could you provide more details about the Monte Carlo rollout process and the quality of the generated success probability labels?\n4. The final chosen parameters are C=0.99 and β=10% (90th percentile), which are far from the median and quite conservative. Does this lead to prohibitively high computational overhead?", "limitations": "Yes, in Page9.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n1. The paper effectively demonstrates that existing PRMs are poorly calibrated through comprehensive empirical analysis.\n2. The use of quantile regression for PRM calibration is creative and well-motivated, addressing the need for uncertainty-aware predictions rather than just point estimates.\n3.The proposed IAS framework has immediate practical applications for making LLM inference more efficient.\n\n## Weaknesses\n1. The experimental results are primarily based on PRMs with around 7B parameters, lacking analysis of larger-scale PRMs. It remains unclear whether larger models like Qwen/Qwen2.5-Math-PRM-72B also exhibit the overestimation phenomenon observed in smaller models.\n2. The evaluation is confined to the mathematical reasoning domain. While this may be due to current PRM limitations, with the emergence of more generalizable PRMs such as Gen-Verse/ReasonFlux-PRM-7B, it's uncertain whether the observations and methods can extend to broader domains beyond mathematics.\n3. The paper lacks comprehensive analysis of the training process, including training data composition and methodology. Only brief mention of LoRA fine-tuning is provided in the appendix (page 24), leaving important implementation details unclear.", "was_revised": true, "final_justification": "The authors have satisfactorily addressed my concerns through comprehensive responses. The additional evaluation on 72B models confirms calibration issues persist across scales, strengthening the paper's generalizability. While the work remains limited to mathematical reasoning and the exact mechanism behind ranking improvements requires further ablation studies, these are minor issues that don't diminish the core contributions. The paper presents a well-motivated quantile regression approach for PRM calibration with clear practical benefits for inference-time scaling. I maintain my positive assessment.", "rebuttal": "We thank Reviewer dLMo for their thoughtful feedback, valuable suggestions, and overall positive assessment of our work. Below, we address each of the questions raised, detailing how we have incorporated changes into the revised manuscript.\n\n-----\n## Weaknesses\n\n### W1 – Larger-scale PRMs\n\n> **Reviewer:** “lacking analysis of larger-scale PRMs.”\n\n**Response:** We appreciate the suggestion and have evaluated the 72B model on MATH and AIME. Results show larger models also suffer from overestimation, though slightly less than the 7B model. This is expected, as both were trained for ranking quality, not calibration. In contrast, our calibrated 7B model is significantly better calibrated than both uncalibrated 7B and 72B models, reinforcing our findings. We will add these results to the revision. We also note the 7B model is more widely used (64k downloads vs. 890 for 72B last month). While resource constraints prevent fully calibrating the 72B model, we hope our work encourages such studies at scale.\n\n**Table R0.** Calibration errors. Values are presented as Brier/Positive Brier scores, where lower is better. The best result for each metric within a row is **bolded**. L: Llama, Q: Qwen.\n\n|Dataset|Model|Uncalibrated 7B|Uncalibrated 72B|Calibrated 7B|\n|-|-|-|-|-|\n|**MATH500**|L-1B|.241/.223|.211/.078|**.069**/**.047**|\n||L-8B|.205/.177|.177/.111|**.121**/**.099**|\n||Q-1.5B|.154/.130|.155/**.106**|**.127**/.107|\n||Q-7B|.101/.085|.118/.057|**.082**/**.053**|\n||R1-L-8B|.161/.114|.193/.072|**.089**/**.055**|\n||R1-Q-7B|.148/.106|.176/.075|**.083**/**.058**|\n|**AIME24-25**|L-1B|.194/.192|.157/.063|**.003**/**.001**|\n||L-8B|.227/.223|.183/.122|**.041**/**.035**|\n||Q-1.5B|.330/.322|.256/.176|**.073**/**.053**|\n||Q-7B|.289/.282|.205/.139|**.072**/**.066**|\n||R1-L-8B|.385/.371|.237/.204|**.078**/**.030**|\n||R1-Q-7B|.414/.402|.260/.215|**.069**/**.034**|\n\n### W2 – Domains of application\n\n> **Reviewer:** “The evaluation is confined to the mathematical reasoning domain.”\n\n**Response:** We focused on mathematical reasoning as it is a robust testbed for LLM logical inference and the domain with the most developed PRMs (at the time of submission). While we agree that generalizing is the ultimate goal, resource constraints prevent exploring other domains here. In this regard, we parallel the large body of LLM reasoning literature, which to date has largely focused on math. Exploring the applicability of our approach to other domains, such as code generation, scientific QA, or commonsense reasoning, is a valuable and exciting avenue for future research, and we have updated the conclusion of our manuscript to explicitly acknowledge this limitation and highlight these potential future directions.\n\n### W3 – Implementation Details\n\n> **Reviewer:** “The paper lacks an analysis of the training process, including training data composition and methodology.”\n\n**Response:** In the revision, we will add a more detailed description of the calibration dataset and fine-tuning process, which are crucial for reproducibility, expanding on the summary already in Appendix C.1 and C.3.\nSpecifically, we will add:\n\n  * The specific prompt formats used for each PRM and LLM.\n  * Pseudo-code for the quantile regression head and the loss function.\n  * A detailed description of our correctness measurement method.\n  * Specifications and examples from the calibration dataset, including the total number of prefixes.\n\nTo further support transparency and future research, we will also make the entire codebase and calibration dataset publicly available alongside the final publication.\n\n-----\n## Questions\n\n### Q1-1 – Clarification on Probability Estimation for BoN\n\n> **Reviewer:** “BoN typically involves predicting complete sequences … Is the estimated probability based solely on the question?”\n\n**Response:** Thank you for pointing this out. Your understanding is correct; in the case of BoN, the prefix generation is indeed the empty string. To make this more clear, we will revise Definition 1 by changing $x\\_{1:t}$ to $x\\_{0:t}$, while explicitly stating $x\\_{0:0} := \\\\text{“”}$. Additionally, we will update Definition 2 to clearly indicate that the prefix can be the empty string.\n\n### Q1-2 – Reward Calculation\n\n> **Reviewer:** “How do you obtain the reward for the entire trajectory from the calibrated PRM?”\n\n**Response:** The reward calculation follows the official implementation provided by each PRM developer, with the final score predicted at the last step in our experiments; as [1] and [2] also noted, using the *last* token often performs the best in the inference-time scaling regime. Specifically, for Qwen-PRM, the system prompt is:\n\n```\n
system\nPlease reason step by step, and put your final answer in \\boxed{}.
\n```\n\nThe user prompt format is:\n\n```\n
user\n${Question}
\n
assistant\n${Prefix}<extra_0>

\n```\n\nThe score is computed based on the token at the final step, which is `<extra_0>`. In the case of BoN, the `${Prefix}` is an empty string (`\"\"`). We recognize that explicitly detailing the PRM reward calculation process improves clarity and readability. Accordingly, we will include this comprehensive explanation in the revised manuscript.\n\n### Q2 – Improvements over Beam Search (BS)\n\n> **Reviewer:** “BS+IASoM shows remarkable improvements over BS. Since calibration shouldn't affect ranking order, what is the source of such significant improvements?”\n\n**Response:** Thank you for raising this insightful question. For standard baselines such as temperature scaling and isotonic regression, calibration indeed does not alter ranking order. However, our method involves fine-tuning the PRM on the calibration dataset, which can modify ranking order. While improved calibration does not inherently guarantee enhanced ranking performance, our empirical findings indicate that the proposed PRM calibration method positively impacts ranking accuracy, leading to the substantial improvements you observed.\n\n### Q3 – Details of Calibration Data Collection\n\n> **Reviewer:** “How is the training data for calibration obtained? … quality of the generated success probability labels?”\n\n**Response:** \n* **Roll-out:** The details of our calibration data collection are provided in Appendix C.1. Briefly summarized:\n\n1\\. For each question, we generate $N\\_{\\\\text{val}}=8$ independent reasoning trajectories using the LLM.\n\n2\\. For each reasoning trajectory and each corresponding prefix trajectory, we generate an additional $N\\_{\\\\text{MC}}=8$ follow-up trajectories, conditioned on both the question and the prefix.\n\n3\\. We estimate the success probability by evaluating and counting the number of correct follow-up trajectories out of these $N\\_{\\\\text{MC}}$ trajectories.\n\nWe will update the main manuscript to enhance the readability.\n\n* **Label quality:** While the per-prefix MC estimator $\\\\hat p$ is unbiased, we acknowledge that a larger $N\\_{\\\\text{MC}}$ would yield more statistically accurate estimates. Nevertheless, we collect **M = 40,000 to 80,000** prefixes per PRM and LLM pair. Aggregating over $M$ prefixes reduces the standard error of our calibration error (i.e., Brier score) down to: \n$$\n\\approx \\sqrt{\\frac{2\\sigma_\\varepsilon^{4} + 4\\sigma_\\varepsilon^{2} \\times \\mathrm{MSE_{\\text{true}}}}{M}} =  4.7\\times10^{-4}, \n\\qquad\n\\sigma_\\varepsilon^{2} = \\frac{p(1-p)}{N\\_{\\\\text{MC}}} = 0.02 .\n$$\nwhere we use $M = 40,000$, $p=0.8$ and $\\mathrm{MSE_{\\text{true}}} = 0.1$ based on our empirical results.\n\nHence, our reported average calibration performance metrics are statistically accurate enough, and the per-instance noise has a negligible impact on the global trends. That said, the instance-wise standard error with $N\\_{\\\\text{MC}}=8$ does artificially increase the variance of the distribution we are modeling in our quantile regression objective. As a result, increasing $N\\_{\\\\text{MC}}$ may improve results for our method. We were limited by computational resources.\n\n* **Resource trade-off:** Raising $N\\_{\\\\text{MC}}$ reduces instance-level standard error as $1/\\\\sqrt{N\\_{\\\\text{MC}}}$ while already require up to **640K** extra forward passes per PRM–LLM pair (we have a total of 18 pairs) even for the current setting of $N\\_{\\\\text{MC}}=8$. This amount of inference takes about 150 hours on our V100 GPU. We will release the data/code so larger-scale replications can explore bigger $N\\_{\\\\text{MC}}$.\n\n### Q4 – Choice of IAS Parameters & Computational overhead\n\n> **Reviewer:** “The final chosen parameters are C=0.99 and β=10% (conservative) ... Does this lead to prohibitively high computational overhead?”\n\n**Response:** We would like to point out that we already show results on this regime in the paper. With these settings ($C=0.99$ and $\\\\beta=10\\\\%$), IAS reduces the sampling budget by up to approximately **65%** for the BoN strategy and **75%** for the BS strategy (Table 2 and 3) – in other words, providing large computational savings, not overhead. We don’t believe that this setting is as conservative as it sounds, because the union bound guarantee on end-to-end accuracy should be high, especially with the MATH benchmark. Put another way, our goal is to save unnecessary computation that does not meaningfully help performance, rather than aggressively trading off performance and compute.\n\n-----\n## Format Concerns\n\nIn Figures 14 and 15, more conservative settings (e.g., larger C) are indicated by points further to the right. The values tested are originally in Appendix E.4. We will clarify and improve both figures and captions to fully address your concerns.\n\nERM refers to Empirical Risk Minimization; we will explicitly define this.\n\n-----\n## References\n[1] Beeching, Edward, et al. \"Scaling test-time compute with open models.\", 2024.  \n[2] Snell, Charlie, et al. \"Scaling llm test-time compute optimally can be more effective than scaling model parameters.\", 2024."}, {"review_id": "EcjyuBdJy7", "reviewer": "Reviewer_wTjL", "summary": "This paper explores several ways of calibrating existing PRMs for a given LLM by tuning PRMs towards the success rates of the given LLM for any prefix solutions. Typically, given a prefix solution of a given LLM, they continue to sample n outputs with LLM, then calculate the success rate of sampled outputs, finally, they can run quantile regression over those data to calibrate an existing PRM. With the calibrated PRM, they further utilize those calibrated scores to dynamically picking size of N outputs in BoN setting, and beam size B in beam search setting.  Experiments on calibration tasks show significant improvements over un-calibrated baseline. With the guidance of those PRMs, they can achieve comparable results with lower budget in best of 64 settings.", "questions": "1. In Section 5.1, “To start with, we construct a calibration dataset by randomly sampling 500 questions from the MATH training set. We assess calibration performance using standard metrics:”. Does this mean the authors tune and report on training set? Evaluation results in Table 1 is on complete solution or still on prefixes? \n\n2. It’s not clear to me how the authors select prefix solutions, just uniformly sample on each time T over all tokens or on the end of a step?\n\n3. Why not train a PRM directly on the calibration data with success rate.", "limitations": "See weaknesses and questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. Model calibration is an important property in inference time, as it can guide the search of reasoning especially in shot-CoT reasoning tasks. \n\nWeaknesses\n\n1. Experiments are not well-designed, and the following questions must be addressed (see Section Questions). \n2. The “budget” metric in Table 2 is a little misleading, as you also need to run PRMs to get the calibrated scores, which are not required in baseline system. Thus, speed and memory usage should be also reported in Table 2.\n3. Beam search with LLMs in this paper is not well-defined.", "was_revised": true, "final_justification": "The authors' responses have addressed most of my concerns.", "rebuttal": "Thank you for your careful reading of our paper and thoughtful feedback! \n\n---\n## Weaknesses\n\n### W1 – Experiment\n\n**Q1 – Details of Calibration Data Collection I**\n> **Reviewer:** “Does this mean the authors tune and report on training set? Evaluation results in Table 1 is on complete solution or still on prefixes?”\n\n**Response:** To clarify, the MATH dataset provides both training and testing splits. We tune PRMs only using samples from the **training** split and test PRMs on the **test** split of the MATH dataset. Likewise, AIME2024 and AIME2025 are not used during PRM fine-tuning. Thus, calibration (i.e. fine-tuning) and testing are performed on distinct datasets. We will revise the manuscript accordingly to clarify this point.\nThe evaluation results presented in Table 1 reflect performance on the **test** calibration dataset--- which as noted above is distinct from the **train** calibration dataset---encompassing all prefixes: initial questions with no prefix ($t = 0$), intermediate reasoning steps ($1 \\leq t < T$), and fully completed solutions ($t = T$). We detail our calibration data collection process in Appendix C.1, summarized briefly below:\n1.  For each question, we generate $N_{\\text{val}}=8$ independent reasoning trajectories using the LLM.\n2.  For each reasoning trajectory and each corresponding prefix trajectory, we generate an additional $N_{\\text{MC}}=8$ follow-up trajectories, conditioned on both the question and the prefix.\n3.  We estimate the success probability by evaluating and counting the number of correct follow-up trajectories out of these $N_{\\text{MC}}$ trajectories.\n\nUltimately, for each PRM-LLM pair, we generate 500 questions, each with 8 trajectories containing between 10 and 20 steps, resulting in approximately **40,000 to 80,000 prefixes** analyzed. We have further clarified these details in the revised manuscript.\n\n---\n**Q2 – Details of Calibration Data Collection II**\n> **Reviewer:** “How the authors select prefix solutions … uniformly sample on each time T over all tokens or on the end of a step”\n\n**Response:** Due to space constraints, these details are provided in Appendix C.1 of our supplementary material. Specifically, prefixes are systematically collected at the end of each intermediate reasoning **step ($x_t$)**. We will integrate additional clarifications regarding the calibration dataset collection process into the revised manuscript.\n\n---\n**Q3 – Details of Proposed Calibration Method**\n> **Reviewer:** “Why not train a PRM directly on the calibration data with success rate.”\n\n**Response:** We clarify that our proposed method **does involve directly training** (finetuning) the PRM on the calibration data using success rates. Specifically, we fine-tune off-the-shelf PRMs using the Low-Rank Adaptation (LoRA) technique [1], where the training objective is defined by the quantile loss. \nWe adopt LoRA to fine-tune PRMs mainly due to its efficiency in computation, memory requirements, and inference time. LoRA works by introducing low-rank adapters into the weight matrices of PRMs, resulting in only a minimal increase, approximately **0.0025%**, in the total number of model parameters in our experiments.\n\n\n\n\n\n\n---\n### W2 – Computational Efficiency of the Proposed Method\n\n**Q1 - Budget Metric**\n> **Reviewer:** “The “budget” metric in Table 2 is a little misleading, as you also need to run PRMs to get the calibrated scores, which are not required in baseline system.”\n\n**Response:** We clarify that PRM calls are present in all baselines as well, and in fact, for both BoN and BS, our approach requires only 1 extra PRM call (the one on the initial question). Hence, the incremental cost of our extra PRM invocation is negligible. Furthermore, properly implemented, PRM evaluations are a very small part of the overall pipeline cost (baseline or ours), since each PRM evaluation requires generating only 1 token, compared to the thousands of tokens being generated by the main LLM. **More specifically, the token generated by PRM is about 1% of the LLM generation per reasoning step, and our IAS’s extra +1 score token adds less than 0.1%.**\n\nSpecific details follow:\n1.  **Baseline systems already pay for PRMs.**\n    * **Best-of-$N$ sampling:** The baseline also issues $N$ PRM calls—one for each candidate hypothesis. Our IAS variant issues $N_{\\text{IAS}} + 1$ calls, where $N_{\\text{IAS}} \\le N$. The +1 evaluates the empty prefix (question-only) so that IAS can determine $N_{\\text{IAS}}$.\n    * **Beam search:** The baseline issues $M \\times K$ PRM calls per reasoning step. IAS needs $M_{\\text{IAS}} \\times K + 1$ or $M \\times K_{\\text{IAS}} + 1$—again, only one extra call.\n    \nEven in the worst case, IAS makes $\\approx 1.5\\\\%$ more PRM calls than the baseline.\n\n2.  **Why PRM cost is negligible.**\n    Inference-time scaling targets the most expensive part of decoding: **LLM generation**, not the single-token logits of a PRM.\n    * **Single-token vs. trajectory length.** Each PRM call emits exactly **one** token; a reasoning trajectory for tasks like MATH typically emits **hundreds**.\n    * **Cost scales linearly with $N$.** Increasing the number of generated trajectories inflates the budget far more than an extra 1-token PRM call.\n\nFor these reasons, Table 2 reports the metric most relevant to practitioners: **trajectories that the LLM actually *generates***. The cost of PRM calls, while present, is minimal in comparison. \n\nTo provide a concrete example, here is a cost analysis using OpenAI's GPT pricing ($\\textdollar$2/1M input tokens, $\\textdollar$8/1M output tokens), a 100-token question, and 1,000-token reasoning trajectories. As the table below illustrates, when IAS, even in the worst case, generates the same number of trajectories as the baseline (64), the overhead is a negligible **$\\textdollar$0.0002** (a 0.03% increase). However, when IAS intuits that only 32 trajectories (i.e., 50%) are sufficient, it slashes the total cost by nearly 50%.\n\n| Cost Element | Baseline (64 traj.) | IAS (worst: 64 traj.) | IAS (in practice: 32 traj.) |\n|:---|:---:|:---:|:---:|\n| LLM Generation (Input) | $0.0002 | $0.0002 | $0.0002 |\n| LLM Generation (Output) | $0.5120 | $0.5120 | $0.2560 |\n| **LLM Generation (Total)** | **$0.5122** | **$0.5122** | **$0.2562** |\n| --- | --- | --- | --- |\n| PRM Scoring (Input) | $0.1408 | $0.1410 | $0.0706 |\n| PRM Scoring (Output) | $0.000512 | $0.000520 | $0.000264 |\n| **PRM Scoring (Total)** | **$0.1413** | **$0.1415** | **$0.0709** |\n| --- | --- | --- | --- |\n| **Grand Total** | **$0.6535** | **$0.6537** | **$0.3271** |\n| **% of Baseline Cost** | **100%** | **~100.03%** | **~50.1%** |\n\n---\n**Q2 - Speed and Memory Usage**\n> **Reviewer:** “Speed and memory usage?”\n\n**Response:** Great question! LoRA fine-tuning increases the model size by only **0.0025%** and adds ≈ **0.1%** compute overhead.\n\nIn terms of memory, we fine-tune the PRM with LoRA adapters, which insert low-rank matrices into the 7B-parameter model and increase its size by only about **0.0025%**. On the speed side, LoRA’s extra multiply-adds scale with $O(2rd)$ instead of the original $O(d^2)$. Using $r=2$ and $d=3584$, the resulting compute overhead is roughly **0.1%**. IAS itself needs at most one additional PRM evaluation, but that evaluation produces a single score token, whereas each reasoning trajectory contains hundreds of generated tokens, so the incremental runtime cost is well below the already small 0.1% figure. We appreciate your comment and will add these quantitative clarifications to the manuscript to make the computational efficiency of our method clear.\n\n---\n### W3 – Details of Beam Search with LLMs\n> **Reviewer:** “Beam search with LLMs in this paper is not well-defined.”\n\n**Response:** Beam search is introduced in Section 2.2, and the method is presented in more detail in the paper [2] we cite in the original manuscript. \n\n---\n## References\n[1] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\", 2022.  \n[2] Snell, Charlie, et al. \"Scaling llm test-time compute optimally can be more effective than scaling model parameters.\", 2024."}, {"review_id": "oeHmoJ1KPi", "reviewer": "Reviewer_cDdM", "summary": "The paper presents a novel framing of process reward model (PRM) predictions, when normalised, as approximations of the probability of generating a correct response. The authors demonstrate that current state-of-the-art PRMs produce miscalibrated forecast probabilities and argue why proper calibration is critical for efficient reasoning and for assessing problem difficulty on the fly. To address this, they introduce an adaptive compute-budget allocation method that leverages calibrated PRMs, along with theoretical justifications, to dynamically adjust evaluation time based on estimated difficulty. Experiments on two reasoning benchmarks across several models show promising gains in both calibration and compute efficiency.", "questions": "1. **Correctness measurement:** How did you determine whether a generated response was correct? I could not find a detailed description of the correctness metric in the paper.\n2. **Sampling settings:** What sampling parameters (e.g., temperature, top-k or top-p) did you use when generating responses with multinomial sampling? More evaluation details would strengthen reproducibility.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n1. **Clear motivation and presentation.** The paper is well motivated, and the core findings are presented in a logically structured, easy-to-follow manner.\n2. **Novel insight on PRM normalisation.** Demonstrates that normalised PRM scores can be interpreted as forecast probabilities of success and that these are poorly calibrated out of the box for current SoTA PRMs.\n3. **Evaluation of common recalibration techniques.** Shows that standard recalibration methods fail to properly calibrate PRMs on reasoning tasks. (It might be worth adding an additional baseline, adaptive temperature scaling, for completeness - see below.)\n4. **Effective adaptive compute allocation.** The proposed method not only improves calibration but also aided in helping with compute-budget savings on easier problems whilst allocating more resources to harder questions rather than having a fixed sampling budget for sampling reasoning trajectories. This is a neat step toward more efficient reasoning with PRMs.\n\n**Weaknesses**\n\n1. **Notation and presentation clarity.** Some of the mathematical notation is a little sloppy, for example, Definition 1’s conditional probability should be written as p(x_{1:t}\\mid q). The exposition of the quantile regression formulation (especially the first paragraph) is messy and needs rewriting for readability.\n2. **Calibration metric choice.** In Figure 3, the Brier score is used to compare recalibration methods, but it conflates calibration and discrimination. A more standard calibration metric (e.g., Expected Calibration Error or Adaptive Calibration Error) would be preferable.\n3. **Missing baseline.** While the paper correctly shows that conventional methods rescale or shift probabilities improperly, it should also compare against the adaptive temperature-scaling approach of Xie et al. 2024, which addresses some of these calibration challenges.\n4. **Error bars**: Lack of error bars in the main results is somewhat limiting of the significance of the papers results. I think some effort should be made to change this given the high variance of MC sampling trajectories within some of their methods. \n\nXie, Johnathan, et al. \"Calibrating Language Models with Adaptive Temperature Scaling.\" Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.", "was_revised": true, "final_justification": "Initially I was positive about the paper. In the rebuttal the authors have made significant attempts at clarifying and new experiments which has in my mind solidified the significance and impact of their paper. I am raising my score to 5 to justify this.", "rebuttal": "We thank Reviewer cDdM for their thoughtful feedback, valuable suggestions, and overall positive assessment of our work. Below, we address each of the questions raised, detailing how we will incorporate changes into the revised manuscript.\n\n---\n## Weaknesses\n\n### W1 – Notation & presentation clarity\n>**Reviewer:** “Definition 1 should read $p(x_{1:t}\\mid q)$; quantile-regression paragraphs need rewriting for readability.”\n\n**Response:** The current Definition 1 relies on the notation presented on page 3, lines 112–114:\n> Given a query $q$, the LLM generates a multi-step reasoning trajectory $x = (x_1, x_2, \\dots, x_T)$, where $x_i$ denotes the $i$-th reasoning step and $T$ represents the total length of the trajectory. We further defined $x_{1:t}$ as the prefix of the trajectory up to step $t$.\n\nWe clarify that $t$ refers specifically to an intermediate reasoning step, rather than the total length $T$. Thus, the expression $p(x_{1:t}\\mid q)$ is an inaccurate presentation. Our intended definition is $p(x_{1:T} \\text{ yields a correct answer} \\mid q, x_{1:t})$. We acknowledge that this was not clearly articulated in the original manuscript. Additionally, we recognize that the phrase “yields a correct answer” may lack clarity. As per your suggestion in Question 1, we will explicitly specify our method for evaluating correctness.\n\nTo improve the clarity and completeness of our manuscript, we will implement the following revisions:\n\n* **Make Section 4.1 Self-Contained**: We will merge the relevant content from the supplementary material (e.g., Appendix C.1) into Section 4.1 of the main text.\n\n* **Introduce a Dedicated Section for Data Collection**: We acknowledge that the procedure for collecting the calibration dataset is a key contribution. Therefore, we will create a new, independent section to describe this process in detail and provide pseudocode. The new section will formally outline our data collection method:\n\n1.  For each question, we generate $N_{\\text{val}}$ independent reasoning trajectories.\n2.  For each reasoning trajectory and each corresponding prefix, we generate an additional $N_{\\text{MC}}$ follow-up trajectories, conditioned on both the question and the prefix.\n3.  We estimate the success probability by evaluating and counting the number of correct follow-up trajectories out of these $N_{\\text{MC}}$ trajectories.\n\n### W2 – Calibration metric choice\n> **Reviewer:** “Figure 3 uses Brier score; include ECE/AdaCE.”\n\n**Response:** We agree with your suggestion to include more conventional calibration metrics such as Expected Calibration Error (ECE) and Adaptive Calibration Error (AdaCE). Although initial space constraints limited the metrics presented in Figure 3, we have now conducted a comprehensive evaluation. Table R1 below provides detailed results for both uncalibrated and various calibration methods. Consistently, our method demonstrates effectiveness across all metrics tested. We empirically observe that AdaCE aligns better with the Brier score compared to ECE. This alignment is likely due to the non-uniform distribution of predictions between 0 and 1, which advocates the suitability of AdaCE over ECE as recommended in its original paper. We appreciate this suggestion, which has helped reinforce our findings, and we will explicitly include these expanded results in the final manuscript.\n\n### W3 – Missing calibration method (Adaptive Temperature Scaling)\n> **Reviewer:** “Compare against ATS of Xie et al. 2024.”\n\n**Response:** We appreciate your recommendation to include the Adaptive Temperature Scaling (ATS) method proposed by Xie et al. (2024) as an additional calibration method to consider. \n\nWe do point out that as-is, ATS was not proposed for and does not apply to PRM calibration, due to differences between typical LLMs’ language token outputs and the regression-style output of PRMs. That said, we were able to modify ATS to work for PRM calibration. Specifically, we adapted the ATS algorithm to train a transformer head that dynamically selects temperature parameters based on PRM-generated hidden states of the question and prefix. Note that this change makes it much closer to [1] than to the ATS paper. Below, we conduct experiments comparing our proposed method with this adaptation of ATS, and the results are detailed in Table R1. Key findings from these comparisons include:\n* The modified ATS method outperforms simpler baselines, reinforcing the importance of comprehensive, question- or prefix-aware calibration strategies (like our method).\n* For in-distribution datasets (e.g., MATH500), modified ATS achieves performance comparable to our LoRA fine-tuning approach. However, overall, our method consistently demonstrates greater effectiveness than modified ATS.\n* Crucially, our LoRA fine-tuning method significantly surpasses modified ATS in generalizing to challenging out-of-distribution datasets (e.g., AIME24-25), highlighting the robustness and broader applicability of our proposed approach.\n\nCombined with the advantages our quantile-based approach has for adaptive inference scaling, our proposed approach remains the best calibration method we consider. We will incorporate this detailed discussion into the revised manuscript and thank you again for this insightful suggestion.\n\n### W4 – Error bars\n> **Reviewer:** “Add error bars to the main results.”\n\n**Response:** Thank you for highlighting the importance of including error bars. While error bars were provided in Figures 8, 9, and 10 in the supplementary materials, we will update figures in the main text (e.g., Figure 3) to clearly reflect the 95% confidence intervals.\n\n---\n## Questions\n\n### Q1 – Correctness measurement\n> **Reviewer:** “How did you determine whether a generated response was correct?”\n\n**Response:** Thank you for requesting clarification. We utilized established validation procedures from prior works [2, 3]. In short, the generated answer within the `\\boxed{}` is parsed and labeled correct if it matches the ground truth answer. Due to the nature of the math questions, there is little room for ambiguity. These details, along with relevant citations, will be explicitly included in the revised manuscript.\n\n### Q2 – Sampling settings\n> **Reviewer:** “What sampling parameters (e.g., temperature, top-k or top-p) did you use?”\n\n**Response:** We appreciate your question regarding sampling parameters and reproducibility. Our experiments employed the widely used vLLM inference-acceleration framework [4] using their default sampling parameters (`top_p = 1.0`, `top_k = –1`; considering all tokens). The sampling temperature was set to 0.7, aligning with the standard practice recommended by prior works (e.g., Qwen, DeepSeek, various Llama variants, and scaling-law studies [3, 5]).\n\nWe will provide a comprehensive section detailing these inference parameters, evaluation setups, and other relevant hyperparameters to enhance reproducibility. Additionally, we intend to publicly release our codebase alongside the final publication.\n\n---\n\nThank you again for the constructive review, your suggestions will certainly strengthen the paper. We hope these revisions and clarifications address all your concerns.\n\n---\n## References\n[1] Shen, Maohao, et al. \"Thermometer: Towards universal calibration for large language models.\", 2024.  \n[2] Yang, An, et al. \"Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement.\", 2024.  \n[3] Beeching, Edward, and Tunstall, Lewis, and Rush, Sasha, \"Scaling test-time compute with open models.\", 2024.  \n[4] Kwon, Woosuk, et al. \"Efficient memory management for large language model serving with pagedattention.\", 2023.  \n[5] Puri, Isha, et al. \"A probabilistic inference approach to inference-time scaling of llms using particle-based monte carlo methods.\", 2025.\n\n---\n## Table\n**Table R1.** Calibration error for various methods and models on MATH500 (in-distribution) and AIME24-25 (out-of-distribution). Values are presented as **AdaCE**/ECE/Brier scores, where lower is better. The best result for each metric within a row is **bolded**.\n\n| Dataset | Model | Uncal. | TS | IR | HB | ATS | Ours |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| **MATH500** | L3.2-1B | .283/.279/.241 | .265/.268/.190 | .212/**.092**/.097 | .213/.093/.097 | .181/.151/.099 | **.081**/.094/**.069** |\n| | L3.1-8B | .263/.237/.205 | .244/.244/.168 | .321/.192/.166 | .323/.193/.166 | .237/.186/.130 | **.167**/**.152**/**.121** |\n| | Q2.5-1.5B | .218/.155/.154 | .205/.208/.140 | .271/**.117**/.145 | .274/.117/.146 | .205/.144/**.123** | **.155**/.141/.127 |\n| | Q2.5-7B | .146/.098/.101 | .135/.136/.093 | .201/**.018**/.092 | .205/.021/.094 | .135/.053/**.076** | **.100**/.086/.082 |\n| | R1-L-8B | .251/.131/.161 | .223/.224/.149 | .297/.022/.141 | .298/**.017**/.141 | .196/.063/.107 | **.113**/.091/**.089** |\n| | R1-Q-7B | .241/.110/.148 | .202/.206/.140 | .289/.019/.132 | .289/**.019**/.132 | .178/.069/.103 | **.107**/.086/**.083** |\n| **AIME24-25** | L3.2-1B | .236/.236/.194 | .210/.214/.193 | .177/.177/.065 | .177/.177/.065 | .125/.129/.065 | **.011**/**.004**/**.003** |\n| | L3.1-8B | .284/.284/.227 | .248/.258/.206 | .392/.392/.215 | .396/.396/.215 | .281/.281/.125 | **.086**/**.078**/**.041** |\n| | Q2.5-1.5B | .401/.401/.330 | .393/.410/.293 | .477/.477/.331 | .486/.486/.337 | .367/.346/.184 | **.105**/**.087**/**.073** |\n| | Q2.5-7B | .355/.355/.289 | .354/.368/.249 | .390/.390/.250 | .404/.404/.261 | .271/.259/.146 | **.098**/**.098**/**.072** |\n| | R1-L-8B | .526/.476/.385 | .528/.531/.310 | .590/.541/.400 | .588/.539/.399 | .438/.365/.220 | **.128**/**.076**/**.078** |\n| | R1-Q-7B | .558/.508/.414 | .570/.571/.340 | .613/.561/.430 | .613/.562/.430 | .440/.379/.230 | **.090**/**.069**/**.069** |\n\n**Acronyms**: Uncal. (Uncalibrated), TS (Temperature Scaling), IR (Isotonic Regression), HB (Histogram Binning), ATS (Adaptive Temperature Scaling), L (Llama), Q (Qwen)."}]

Author rebuttal:

Meta-review: This paper addresses the critical issue of poor calibration in PRMs, which systematically overestimate the probability of success when guiding inference-time scaling algorithms for LLMs. Its core contributions include: first, systematically demonstrating that even state-of-the-art PRMs (e.g., variants of Qwen and Llama PRMs) exhibit significant miscalibration, especially on challenging problems and weaker models; second, proposing a novel calibration method based on quantile regression. Fine-tuned via LoRA, this method adjusts raw PRM outputs into well-calibrated estimates of the true probability of correctness given a solution prefix, also providing confidence intervals. All three reviewers (dLMo, wTjL, cDdM) provided positive assessments while raising important constructive criticisms. The authors addressed nearly all concerns through detailed responses and commitments to revisions. By tackling an significant and underexplored problem in PRM calibration, the paper offers a highly original and practical contribution—combining an effective calibration technique with an efficient inference-time framework—supported by thorough experiments and convincing results. It thus fully meets the bar for acceptance at NeurIPS.
Decision: Accept (poster)