Optimal Aggregation of LLM and PRM Signals for Efficient Test-Time Scaling
Authors: Peng Kuang
Yanli Wang
Xiaoyu Han
Yaowenqi Liu
Kaidi Xu
Haohan Wang
Venue: ICLR 2026 Poster (2026)
Keywords: test-time scaling
process reward models
Primary area: foundation or frontier models, including LLMs
TL;DR: Our work demonstrates that investing in a more intelligent aggregation strategy can be a more convincing path to performance gains than simply scaling test-time computation.

Abstract: Process reward models (PRMs) are a cornerstone of test-time scaling (TTS), designed to verify and select the best responses from large language models (LLMs). However, this promise is challenged by recent benchmarks where simple majority voting, which ignores PRM signals, occasionally outperforms standard PRM-based selection. This raises a critical question: How can we effectively utilize verification signals from PRMs for TTS? To address this, we start by developing a theoretical framework for optimally combining signals from both the LLM and the PRM. Our framework reveals that the optimal strategy is a weighted aggregation of responses, a strategy whose effectiveness hinges on estimating weights that capture the complex interplay between the models.
Based on our theoretical results, we empirically show that these optimal weighting functions differ significantly across LLM-PRM pairs and, notably, often assign substantial negative weights.
Motivated by these insights, we propose efficient pre-computation methods to calibrate these weighting functions.
Extensive experiments across 5 LLMs and 7 PRMs demonstrate that our calibration method significantly boosts the TTS efficiency, surpassing the performance of vanilla weighted majority voting while using only $\sim 21.3\\%$ of the computation.
Ultimately, our work demonstrates that investing in a more intelligent aggregation strategy can be a more convincing path to performance gains than simply scaling test-time computation.

Published as a conference paper ICLR 2026
OPTIMAL AGGREGATION OF LLM AND PRM SIGNALS FOR EFFICIENT TEST-TIME SCALING
Peng Kuang', Yanli Wang?, Xiaoyu Han Yaowenqi Liu", Kaidi Xu', Haohan Wang² 'Zhejiang University, IImpeiil College London, University of Illinois Trrban-Chhappain DDreeel University
pengkuangez ju edu.ed en, kx460drexel .ddu, hachanw i illinois.ee....
ABSTRACT
Process reward models (PRMs) are cornerstone of test-time scaling (TTS), it is de- signed to verify and select the best responses from large language models (LLMs), sig- nificantly improving the performances. However, this promise is challenged by recent benchmarks where simple majority voting, which ignores PRM signals, occasionally out- performs standard PRM-based selection. This raises a critical question: How can we effectively utilize verification signals from PRMs for TTS? To address this, start by developing a theoretical framework for optimally combining signals from both the LLM and the PRM. Our framework reveals that the optimal strategy is weighted aggregation of responses, a strategy whose effectiveness hinges on estimating weights that capture the complex interplay between the models. Based on our theoretical results, empirically show that these optimal weighting functions differ significantly across LLM-PRM pairs and, notably, often assign substantial negative weights. Motivated by these insights, we propose efficient pre-computation methods to calibrate these weighting functions. Exten- sive experiments across 5 L.LMs and PRMs demonstrate that our calibration method significantly boosts the TTS efficiency, surpassing the performance of vanilla weighted majority voting while using only 21.3% of the computation. Ultimately, our work demon- strates that investing in more intelligent aggregation strategy can be more convincing path to performance gains than simply scaling test-time computation.
INTRODUCTION
The pursuit of advanced reasoning in Large Language Models (LLMs) has largely been driven by scaling up model size and training data (Ouyang et al., 2022). While effective, this approach entails prohibitive compu- lational costs (Snell al. (0025). An increasingly popular alternative is Test-Time Scaling (LTS)(Liu al.) 2025 Madaan et al. 2023). a paradigm that enhances the performance of fixed LLM by allocating more computational resources at inference time. A prominent TTS strategy involves generating a multitude of candidate solutions and then selecting the most promising one. This heavily on the quality of the selection mechanism, which is tasked with identifying the correct response from a pool of diverse, model-generated outputs. The central challenge, therefore. lies in designing selection strategy that can effectively harness the collective evidence from multiple generated responses to maximize final performance.
framework relies
To address this selection problem, a common approach is to employ a Process Reward Model (PRM) (Light- man et al.] 2024 Li & Li 2025 Zheng et al. 2024), sophisticated verifier trained human feedback to score the quality of reasoning steps. The standard protocol, Best-of-N (BoN), simply selects the answer from the single response that receives the highest PRM score. Intuitively, this should leverage the detailed,

---

Published as a conference paper at ICLR 2026
step-by-step evaluation capabilities of the PRM. However, this intuition is challenged by surprising and counter-intuitive empirical reality: on recent benchmarks (Zhang et al. 2025b), the far simpler method of majority voting (Wang et al.] 2023), which completely ignores the expensive PRM and relies solely on the consensus of the LLM's s own generations, can outperform PRM-guided BoN. This paradox suggests fun- damental misalignment
how we utilize verifier signals. If powerful, costly-to-train PRM can be bested by a simple vote count, it implies we are failing to properly integrate its nuanced feedback.
In this work, we dive into the interactions between L.MSs and PRMs to find better aggregation of signals from both models for more efficient TTS. We begin by formalizing the task of aggregating responses as a Maximum a Posterioni (MAP) estimation problem, revealing that the optimal aggregation strategy is not to simply pick the best-scoring response, but perform weighted majority vote. Interestingly, the optimal weight for each response is a function of two distinct components: a term derived from the PRM's 's score, reflecting the quality of the reasoning, and term derived from the LLM's own reliability. This formulation provides a principled framework for unifying the evidence from both the generator and the verifier.
To understand the practical implications of this theoretical result, we conduct an empirical analysis to char- acterize the optimal weighting function, uncovering two critical insights. First, the shape of the optimal function is highly dependent on the specific LLM-PRM pair, indicating that a sree-siee-its--lll approach inherently suboptimal. Second, we find that optimal functions consistently assign negative weights to re- sponses with low PRM scores. This reveals key deficiency in existing methods (Wang et al.) 2024]: they fail to leverage the negative evidence provided by a low-quality response. A response judged to be poor by the PRM should not simply be ignored; it should actively count against its proposed answer.
Motivated by these insights, we introduce simple yet effective calibration methods to learn approximations of these optimal weighting functions from a small, one-time pre-computed dataset. We propose both non- parametric and parametric approaches that explicitly capture the model-specific nature of the weights and incorporate the mechanism of penalizing low-quality responses. Extensive experiments across 5 different LLMs and 7 PRMs the MATH (Hendrycks et al. (021) datasets demonstrate the superiority of our approach. Our calibrated weighted voting method consistently outperforms baselines, including standard BoN and vanilla weighted voting. Notably, it achieves higher accuracy than these methods while using approximately 37. % and 21.3% of the test-time computation, demonstrating a significant improvement in TTS efficiency. In summary, our contributions are:
We develop theoretical framework for optimally aggregating LLM generations and PRM scores, demon- strating that the solution is weighted majority vote combining signals from both models.
We empirically characterize the optimal weighting function, revealing its model-dependent nature and, interestingly, the importance of assigning negative weights low-quality responses.
We propose practical calibration methods to learn these weighting functions, enabling efficient and effec- tive test-time scaling.
Through extensive experiments, we show that our calibrated aggregation strategy significantly improves
TTS efficiency, achieving superior performance with substantially less computational overhead.
2 RELATED WORK
Test- Time Scaling. The pursuit of improved model performance without retraining has led to the paradigm of Test-Time Scaling (TTS), which allocates more computational resources at inference time /Zhang et al. (2025a). A dominant strategy within TTS is the "generate-and and-eeectt" framework, often formalized as Best- of-N (BoN) sampling, where N candidate solutions are generated and a selection mechanism chooses the best one Ichihara et al. (2025). A foundational method in this area is Self-Consistency (SC), which samples multiple diverse reasoning paths and selecting the final answer via a simple majority vote WWan et al.] (2023). The intuition is that an answer derived from multiple independent lines of thought is more likely to be correct.
2

---

Published as a conference paper ICLR 2026
While effective, SC's primary drawback is its high computational cost. This has motivated more efficient variations, such as Confidence-Informee Self- Seff-Consistency (CISC), which introduced a weighted majority vote based on the model's s self-assessed confidence to reduce the required sample size (Taubenfeld al., 2025b). In parallel, other approaches use an external verifier, like a PRM, to select the single highest-scoring candidate (Uesato et al. 2022). Our work builds on the idea of weighted voting, but instead of relying on the LLM's self-assessment, we derive weights from a principled theoretical framework that combines both the LLM's consensus signal and the external verifier's ' scores.
Reward Modeling. A crucial component of many TTS strategies is external verifier, reward model (RM), trained to score the quality of generated responses. An early, influential work by Cobbe et al. (202T) demonstrated that training dedicated verifier to select the best solution from many candidates could im- prove performance on math word problems more effectively than fine-tuning the generator Uesato et al. (2022). This spurred a distinction between two supervision strategies: Outcome Reward Models (ORMs), which are trained on the correctness of the final answer, and Process Reward Models (PRMs), which are trained on step-by-step human feedback. An initial comparison by [eesao et al. (2022) found that ORMs could achieve similar final-answer accuracy with less supervision, but PRMs were necessary to ensure the faithfulness of the reasoning process (Zengg et al. 2024). Subsequent work by Lightman et al. (2024) solid- ified the superiority of PRMs on more challenging tasks, establishing process supervision as key technique for building reliable verifiers, despite its high annotation cost (Wang et al.) (2024). Our work focuses on how to best leverage the signals from these powerful but expensive-to-train PRMs.
3 OPTIMAL RESPONSE AGGREGATION
In this section, we aim to explore the optimal aggregation strategy for signals from the LLM and PRM. We start by formalizing this as a Maximum Posteriori (MAP) estimation problem, and derive an optimal aggregation strategy in Section 3.1 Then, in Section 3.2, we manage to estimate the quantities in the optimal aggregation strategy empirically and offer a few critical insights on the optimal weighting strategy.
3.1 THEORETICAL ANALYSIS OF OPTIMAL RESPONSE AGGREGATION
Problem Setup. Let M be the LLM and V be the PRM (Verifier). For single prompt, M generates an Each response consists of a reasoning process r; and a final answer s; (r:). The PRM V evaluates each generation Ji and produces a scalar score Pi. Let P= {P1. ... PL. be the set of these scores. The set of unique candidate answers is A = {ai...., am Our objective to determine the most probable true answer à given all available evidence.
ensemble of L responses, G [g)- "on-99....,91..
We aim to find the answer 0g that maximizes the posterior probability P(aulg, P, V). By Bayes' theorem, and assuming a uniform prior over answers PM, this is equivalent to maximizing the likelihood of the evidence:
â arg max P(g, Plakk ax M, V)
aveA
We can decompose this likelihood into two factors: P(g, Plak, M, V) = P(PlG, an, V) x P ()))),,)) M). This reflects the causal process: the LLM M generates responses G, and then the Verifier V produces scores P based on G. To make this tractable, we introduce two conditional independence assumptions:
Assumption 3.1 (Score and Generation Independence). The PRM score ps for a generation g: is condition- ally independent of all other generations, given 9i and the true answer ag. The LLM generations 9i conditionally independent of each other, given the true answer Olk-
L
L
P(Plg, V)
II P(p Igi, ak, V), P(glou, M) = II P(glou, M)
1=1
3

---

Published as a conference paper ICLR 2026
With these assumptions, the log-likelihood becomes a sum over individual responses: LL(az)
EL log 9(p aq. V) + + log P M). We hypothesize that under the condition ags, a gen- eration 9: with answer s; ag is correct (c, 1), and incorrect (c; 0) otherwise. The term P(g./ay- M) is simplified to P(c M), where we assume a simple probability model for the LLM: it produces the correct answer with probability 9M and any specific incorrect answer with probability (1 qw/) 1).
Theorem 3.2 (Optimal Aggregation Score). Under the assumptions above, maximizing the log-likelihood is equivalent to maximizing the score:
Wis where wy log P(p = 1. P 0, 0)
9M (m 1)
Score(ax)
Σ 15:=
+ +log
QM
PRM Signal Term
LLM Signal Term
Proef. The full derivation is in Appendix A The key insight is that the log-likelihood can be rearranged into a sum of weights for responses voting for or,, plus terms that are constant with respect to oi and can be dropped from the argmax. The weight w, for each vote combines signals from two sources: the PRM's score (via the likelihood ratio of the score Pl occurring for correct vs. incorrect reasoning) and the LLM's intrinsic reliability (via the term involving 4M) ) also referred to question difficulty (Snell et al. 2025
3.2 EMPIRICAL ANALYSIS OF THE OPTIMAL WEIGHTING FUNCTION
Instantiating the optimal weights. To instantiate the optimal weight w; from Theorem 3.2 we perform a per-question estimation using the ground-truth labels for the specific set of L responses generated for that question. To estimate the PRM Signal term, we apply a separate Kernel Density Estimation (KDE) on the logit space for each individual question to estimate the score distributions P(ple, V) on the specifie question. For details of our KDE estimation in the logit space, please refer to Section 4.1 To estimate the LLM Signal term, we simply set qM be the true accuracy of the responses for that specific question.
Characterizing the optimal weighting function. Taking the PRM's ability to distinguish correct and wrong answers into account, this optimal aggregator provides much tighter performance upper bound than Pass @ N, demonstrating the potential of our framework, shown in Figure 2 More importantly, analyzing the structure of the learned weighting function w" (p) (Figure I reveals two critical insights:
Weighting functions are highly model-dependent. The shape of the optimal function varies dramatically depending on the specific LLM and PRM being used. simple, fixed function (e.g.. using the PRM score directly as a weight) is unlikely to be optimal across different model pairs. This underscores the necessity a calibration procedure tailored to the specific models use.
Presence of negative weights. An interesting and consistent finding is that low PRM scores are mapped to negative weights. This implies that a response deemed incorrect by the PRM provides strong evidence against its proposed answer, and repetition of low-quality responses does not add to the likelihood of their answer being correct. Standard methods like Best-of-N, which only consider the top-scoring candidate, or majority voting, which ignores scores entirely, fail to leverage this powerful negative signal. An effective aggregation strategy must penalize answers supported by low-quality reasoning.
These findings motivate the need for practical method that approximate these complex, non-linear, and often negative weighting functions without requiring ground-truth labels at test time. We address this challenge in the following section.
4

---

Published as conference paper ICLR 2026
Mistral-7B
Qwen2.5-7B
DeepSeek-1.5B
10
10
Cieras Anowe
Qwen- PRM--R
Inconert Anewen
Donsity
6-
Optina Weighing Fanction
0-
10
10
Sky Skyworr-PRM-T7B PRM-7B
Density
6-
8.00
0.25
0.25
0.75
0.25
0.75
PRM Score
PRM Score
PRM Score
Figure I: The PRM score distributions and optimal weighing functions on 6 combinations LLM-PRM pairs. Left y-axis: the probability density of the PRM scores. Right y-axis: the optimal weights wt (p) learned via KDE for different LLM-PRM pairs. The zero-crossing point is marked with a red cross. Note their model-dependent nature and the presence of negative weights for low PRM scores.
4 PRACTICAL CALIBRATION METHODS
The optimal analysis confirmed the need for a calibrated, model-specific weighting function. We now in- troduce practical methods to learn these functions using one-time, pre-computed calibration set Deat {(r,,P) PI, ,I,,, (Fn, Pai c. Once weighting function w(p) is learned, the final answer is selected by a weighted vote: arg max Eisseon w(pt).
4.1 NON-PARAMETRIC WEIGHTING FUNCTIONS
One of the most straightforward ways towards the optimal aggregation strategy to directly estimate the unknown quantities in the optimal weighting function 3.2 i.e., PRM score distributions P(plc = V),, P(yle= = 0, V). and LLM reliability QM-
Estimating PRM score distribution. To capture the nuances of the PRM score distribution on different LIM and PRM combinations, we apply the Kernel Density Estimation (KDE). Compared to other estimation methods, such as histogram estimation parametric estimations, KDEs smooth, continuous, and more flexible. However, while the PRM score is within the probability space between 0 and 1, KDEs are not bounded, spilling probability density outside this range. Consequently, probability space to the logit space with the logit function logit(p) = log( 1",)) Then, we perform KDE of the scores within the logit space. Specifically, the PRM score distribution is estimated as:
first convert the scores from the
logit(p) -logit(p:)
fc(p) = Del
ΣK K VED.
where D. = {ilc, c) separates responses within the calibration set Deal according to their correctness C-- K and h are the kernel and bandwidth of KDE.
5

---

Published as a conference paper ICLR 2026
Estimating LLM reliability. To estimate qM at test time without labels, we first train simple binned probability calibrator g(-) on the PRM scores from the calibration set. During inference, we calculate the calibrated probability for each of the L generated responses Diest = ((),,p(),, (F2)P2) the test question and approximate qM as their average, i.e. GM [Drest DiEDren. ((4)).
Given the estimations above, according to Equation 3.2 we have the estimated weighting function:
WKDE(p) = log(fi (p)) - (p)( + log((иs) + log(m 1) log((1 QM)
This KDE is the practical counterpart to the optimal estimator, where the PRM score distribution is also estimated on Dtest with additional access to the correctness of test responses d4
4.2 PARAMETRIC WEIGHTING FUNCTIONS
As an alternative, we explore simpler parametric forms for w(p), optimizing parameters the calibration set via grid search. These methods are guided by our insight about the importance zero-crossing point, controlled by the parameter b. This parameter acts as threshold, making the weight positive for scores above and negative for scores below, directly implementing the penalization of low-quality responses.
Logit Weighting. Inspired by the log-ratio form in our theorem, model the weight as:
Wlogit (p) logit(p) logit(b)
Linear Weighting. As a simpler baseline, we model the weight as:
Wlinear (p) = p
During grid search, the parameter b is searched within the range [0, 1] and [-1, for Logit Weighted Voting (Logit WV) and Linear Weighted Voting (Linear WV), respectively. The objective used in the grid search is the accuracy on the held-out calibration set.
5 EXPERIMENTS
this section, we first conduct comprehensive evaluation of the scaling methods across 35 combinations of LLM and PRM in Section 5.2
Then we dive into the principles of the proposed methods in Section 5.3
5.1 EXPERIMENTAL SETUP
Models. To capture the complexities of signal aggregation in practice, we use 5 LLMs across 3 model series
(Mistral-7B 7B (Wang et al., 2024). Qwen2.5-1.5B/72 Yang al. (202)) DeepSeck-1.55 1.5B/B (DeepSeck-Al)
2025 ) and 7 PRMs based on Qwen (Qwen2.55PP)
(Zhang et al. 20256). Skywork Parrr--PRM-.... ..5BBB (He al. 2024)) Llama (Llama3.1 1.ea3.1-BB... SB--PRM-Mistrr/DDeepee. (Xiong et al. (024)), and Mistral (math-shepherd- 7b-prm (Wang et al. 2024)) series. During generation, we set the top-p and temperature configuration to 0.9 and 0.7, respectively.
5-RMMOOKKK (Song et al. 2025). Qwen2.5-PRM--
Data. To simulate the scenarios where the reliability of the LLM signals varies, we evaluate performance on task with various difficulties, the MATH training set (MATH) and test set (MATH500) (Hendrycks et al., 2021). For the MATH dataset, select the first 2k out of 7.5k questions randomly sample Ik questions from the pool as the calibration set for each run. Similarly, for MATH500, the calibration pool size is 200, and randomly sample 100 questions from the pool the calibration set. Questions outside of the pool are used as the test set. We run each experiment 3 times with different random seeds. For the MATH500 dataset, sample 112 responses from each LLM for each question. For the
the pool for calibration, and
6

---

Published as conference paper ICLR 2026
0.63
3.60
0.50
13.1% Chmpulc
21.5% Compute
Agquracy
Accuracy
0.55
Sealing Mothod
Scaling Method
PasagNN
Vanilaa WV
0.50
PaseguN
Vanilla WV
Optinal
KDB WV
Optinal
KDB WV
MV
Linear WV
MV
Linear WV
0.50
Luga WV
BoN
Logit
Sample Size (n)
Sample Size (n)
Figure 2: The performance of various scaling methods averaged across all LLM and PRM combinations. The computation efficiency improvement of the Logit WV compared to the berf-performing baseline, Vanilla WV, is marked in red. Left: On the MATH dataset. Right: On the MATH500 dataset.
MATH dataset, sample 32 responses from each LLM for each question, due to its large size. Then, these collected responses are scored by each of the PRMs. As such, we ensure all the scaling methods are using identical responses and PRM scores for fair comparison.
Baselines. We compare our calibrated weighted voting against several methods:
Majority Vote: The answer with the most votes is selected, ignoring PRM scores. This is the standard self-consistency approach. & = arg max, EL_ I[s;; ax)-
Best-of-N (BoN); The answer from the single response with the highest PRM score is chosen. á where i" arg max; Pi-
Vanilla Weighted Vote: A weighted vote where the raw, uncalibrated PRM score p; is used as the weight. â = arg max E.A Eis,=ay Ps.
We also report two theoretical bounds: @ass@N, which considers problem solved if at least one response is correct, and Optimal as discussed in Section 3.2
5.2 MAIN RESULTS
Weighting function calibration significantly boosts TTS efficiency. We evaluate the effectiveness of the proposed weighting function calibration methods by performing calibration for each LLM and PRM pair before scaling test-time compute. As shown in Figure 2 calibrating before scaling significantly boosts the efficiency. In particular, on the MATH and MATH500 datasets, the logit- based calibration method surpasses the performance of vanilla weighting voting methods with approximately 37.1% and 21.3% of test-time compute average across 35 L.IM-PRM pairs.
For detailed results on LLM-PRM pairs, we show the performance of various scaling methods in Table 1 We can see that calibration methods consistently outperform baseline scaling methods across LLMs and PRMs. Generally, Logit Weighted Voting performs the best in most cases. In particular, on the Llama3. Mistral-8B PRM, Logit WV outperforms the best-performing baseline method, Vanilla WV,, by 3 points of accuracy (61.2 v.s. 58.2) on average across LLMs, which would otherwise take an exponential amount of claim of calibrating the weighting function before expensive TTS. Please refer to Appendix B for more detailed results.
test-time compute to achieve. This strongly supports
Consistently effective domains other than math. To demonstrate broader applicability, further experimented on a wide range of tasks beyond math. In the CMMLU-Pro-CoT-EER dataset, there are approx- imately 150 questions in each task, among which 50 used as the calibration set and the rest the test set.
7

---

Published as conference paper ICLR 2026
Table Accuracy of TTS methods at sample size 32 on MATH. The best method for each case is in bold.
PRM
Method
Mistral-7B
Qwen2.5-1.5B
Qwen2.5-7B
DeepSeek-7B DeepSeek-I.5B 57.2 0.0
Average
Qwen- PRMB00K-
Optimal
71.7
59.3 0.01
0.0
BoN
49.5 0.0
57.3
65.3
45.0
46.1 0.0
52.7 0.0
MV
51.3 0.0
60.6 0.0
67.8 = 0.0
54.1 0.0
51.9 0.0
57.1 0.0
Vanilla WV
52.5
61.4
67.7 = 0.0
53.7 + 0.0
51.7 0.0
57.4 0.0
KIDE WV
52.3 12.3+022
61.2+0.11
68.0
54.0 0.1
52.0 0.1
$7.5 0.0
61.5
67.9
0.0
57.6
Logit WV
53.0
0.0
61.5 0.0
67.7 + 0.1
13.9
$1.9
57.6 0.0
Qwen- PRM- 78
Optiaal
58.5 + 0.0
67.9 0.0
75.4 0.0
67.6
63.1+ 0.0
66.5 + 0.0
BoN
33.7
6.0
70.4 = 0.0
6..
59.8 $9.8
61.8 =:8::000 0.0
MV
5.0
60.6
0.0
5..1
1.9
0.0
Vanilla WV
0.0
6.0
70.2
5.0
56.8
KIDE WV
5.1
61.9
68.8
59.1 0.4
58.2 1.2
60.2 0.2
Linear WV
5.0
63.0 0.1
70.3 0.0
64.7 0.1
62.6 0.0
62,9 + 0.0
Logit WV
+..0
63.5 0.1
70.5 +
65.1 0.0
62.9 0.0
63.3
Llama3.1
Optimal
58.7
6..7
BoN
0..2
+8.0
53.5
58.3
MV
5.0
6.06
67.8 = 0.0
5..1
1.0
57.1 0.0
Vanilla WV
53.9
62.2
69.4 = 0.0
56.9 0.0
54.8 1.8:10 0.0
59.4 0.0
KDE WV
53.9
62.0
68.8 O.1.
58.0
58.1
60.2 0.2
Linear
5.0
62.2
70.1 1
62.3 0.0
61.9
WV
55.6 + 0.1
62.9
70.7 + 0.3
62.8 + 0.1
60.7 0.1
62.6 0.1
Llama311- DS- 8B
Optimal
58.1
6..9
73.8 0.0
65.6
0.00
65.0
BoN
+1.3++000
49.5++0.0
68.8 + 0.0
59.0
55.9 0.0
58.9 0.99+0..
MV
5.0
6..6
=
5..1
5..9
0.0
Vanilla WV
0.0
61.7
$7.1
55.7
KDE WV
53.3
61,4 0.1
68.7
58.3 0.2
56.9 0.4
59.7 :
Linear Wv
54.7
61.5
6.1
60.7 = 0.6
58.9 0.1
61.1 + 0.1
Logit WV
$4.9 0.0
61.6++00
69.9 0.0
61.9 + 0.1
59.2 + 0.3
61.5
Skywork- PRM-
Opumal
57.7000
67.3
0.02++00
+.0
63.5
0.0
MV
5.0
6.08
67.8 0.0
54.1
57.1
0.0
Vanilla WV
$4.9
6.04
71.6 = 0.0
60.3 + 0.0
61.5 + 0.0
63.5 +.5
KDE WV
53.8
68.4
70.6
60.6 1.0
62.8 1.3
63.2 0.5
Linear
55.8
7..3
0.
1 0.1
0.1
66.0
Logit WV
55.6
70.2 0.0
72.0
66.0 0.0
65.5
65.9 0.0
Table Accuracy of aggregation methods at sample size n=16 for Llama-3.1-888 1aae-...--8---nttttee and VersaPRM the MMLU--Pr-CoT--Vv dataset.
PRM VersaPRM
Method Optimal
Econ
Chem 85.0
Other
Physics 84.0
Law
Eug
History
Math
Phil
Average
71.8
78.0
76.0
79.0
65.3
78.0
62.0
85.0
70.7
75.9
BoN
61.2
57.0
62.0
64.0
61.0
62.0
45.3
55.0
46.0
63.0
50.5
57.0
MV
62.4
55.0
61.0
64.0
57.0
62.0
35.8
56.0
44.0
61.0
42.4
54.6
Vanilla WV
63.5
57.0
62.0
66.0
59.0
64.0
43.2
60.0
46.0
65.0
48.5
57.7
WV
64.7
57.0
63.0
66.0
58.0
64.0
42.1
58.0
45.0
65.0
$7.2
Linear WV
63.5
57.0
64.0
66.0
62.0
67.0
44.2
63.0
47.0
66.0
46.5
58.7
Logit WV
62.4
58.0
64.00
65.0
60.0
66.0
45.3
63.0
44.0
66.0
53.5
58.8
Llama-3.1 1amma-3.--BB-Inttruee and VersaPRM are the LLM-PRM pair. As seen in Table 2 the proposed methods consistently effective wide range of tasks beyond the math domain.
8

---

Published as conference paper ICLR 2026
Validation Accuracy
0.8
Validation Accuracy
0.8
Test Accuracy (at n=
0.62
0.6
0.6
Method
KDE WV
LLM
LLM
0.4
Mistall78
0.4
Mistral-7R
0.61
Linear WV
Logit WV
Qwen2.5-1,5p
Qwen2
Qwen2.5-88
Qwend 5-70
0.2
DeepSeck 7B
1.55
0.2
DupSSok-1.50
DoepSook-9
0.60
0.0
0.5
1.0
1000
2000
Logit Offset (b)
Linear Offset (b)
Calibration Set Size
The grid search result of the offset parameter b for both Logit WV and Linear WV, where the optimal value marked with vertical lines. The consistently positive optimal value across LLMs demonstrates the necessity of negative weights. Right: The performance of the calibration methods when we scale the calibration set size. The performance can be further improved with larger calibration sets.
Figure 3: Left and Middle:
5.3 EMPIRICAL ANALYSIS
Are negative weights necessary in utilizing the PRM signals? To further verify our insight that negative weights are necessary for better utilization of the PRM signals, we show the grid search result on the offset parameter b of Logit WV and Linear WV, where, in both cases, its value suggests the zero-crossing point, assigning negative weights to responses whose PRM score is lower than b. As shown in Figure [] for both weighting functions, the optimal offset b is consistently larger than zero across all LLMs, proving the necessity of negative weights in efficient TTS. Furthermore, the zero-crossing points are different for each L.M,, but are generally consistent for the same L.LM using different weighting functions (I (Linear and Logit), demonstrating the PRM's varying capability in distinguishing positive and negative responses from different LLMs. This further supports our claim to take the unique interactions between the models into account, i.e., calibration, for efficient TTS.
What's the remaining gap and challenges towards the optimal weighting function? To answer this (i(e) question, compare our dataset-wise estimation of the PRM score weighting function log( Join) with the optimal per-question estimation on several questions. As shown in the left subplot of Figure while estimation captures the wais PRM score weighting function, the optimal weighting function for each individual question varies largely, suggesting a global scoring function is still suboptimal. We also examine how accurate our estimation of the LLM reliability term ęM is with Mean Absolute Error. As shown in the right subplot of Figure 4 while compared to a fixed global LLM reliability, using calibrated PRM scores to estimate this term effectively reduces the error, the error for most PRMs is still larger than 0.2, which is far from negligible. This also explains the relatively low performance of KDE WV compared to the parametric counterparts. In conclusion, accurately estimating either the PRM the LLM part of the weight requires nuanced estimation for individual questions, explaining why the non- parametric KDE estimation underperforms the parametric ones. We find such per-question estimation inherently difficult in our attempts to learn a meta model to predict per-question weighting functions, which struggles to fit and generalize.
How does calibration set size affect the performance? We rearrange the split of the MATH dataset reserve 5k questions as the test, and the rest as the pool of calibration data. As shown in the right subplot of Figure 3 while the validation set used in the main experiments is relative small, the performances of the
9

---

Published as conference paper ICLR 2026
40
0.5
Estimator
Mean Mcan Absolute Eroor
Global
20
Beva-PRMMOOK-22
Qwen Qwen 7B
Weight
Dausstaiis
0.3
Lluma3, 1-Mistal--8
Pai-question (Q3)
Llama': I-DS-8B
Perqqustion
0.2
Skywark- PRM-71
P Pesquestion
Skywaris PRM-1.5B
Per queston (Q 19)
0.1
Shopbord-Mistrl--D
Per questinn (Q 20)
0.0
0.0
0.2
0.4
0.6
0.8
1.0
15
20
PRM Score
Sample Size (n)
Figure 4: Left:The comparison of the cataset-wise estimated PRM score weighting function and the per- question estimated optimal PRM score weighting function. A large variance among questions can be seen. Right: The mean absolute error estimated ЙM compared to the true qM-
Table Average accuracy of out-of-domain tasks of Aggregation Methods at n=16 for LLM 'Llama- 3.1-8B-Instruct
Method
Chem
CS
Econ
Eng
Health
History
Law
Math
Other
Phil
Physics 74.8
Average
Optimal
74.8
75.2
75.5
75.3
75.7
76.5
76.8
74.8
75.3
76.0
75.5
BoN
56.4
56.8
56,5
57.2
56.8
$7.7
58.2
56.5
56.6
58.0
56.4
57.0
MV
53.7
54.3
53.9
54.3
53.3
55.1
56.3
53.8
54,2
55.5
53.5
54.4
Vanilla WV
57.0
57.5
57.3
57.5
56.8
58.4
59.1
57.0
57.5
58.7
56.8
57.6
KDE WV
57.5
56.3
55.0
57.7
55.0
57.5
59.8
56.9
56.3
57.8
57.4
57.0
Linear WV
57.7
57.9
58.1
57.7
55.2
59.5
58.9
57.1
58.0
58.4
56.7
57.8
Logit WV
57.7
57.8
57.9
58.1
57.8
59.4
59.9
57.5
57.9
58.8
57.7
58.2
Average
56.7
56.7
56.5
57.1
55.8
57.9
58.7
56.5
56.7
57.9
56.4
57.0
calibration methods can be further enhanced scale the calibration set size calibrate the weighting function better.
Are calibration results transferable out-of-domain test sets? further examine the effectiveness of calibration sets that are not in the same domain as the test set, we calibrate aggregator on each of the tasks, and report its average accuracy when tested on other tasks in the following table, where each column corresponds to the calibrator trained that task. As on various "Out-of-domain" calibration sets, the proposed methods consistently outperform the baselines, showing strong transferability.
see in Table despite being calibrated
CONCLUSION
We address the suboptimal use of Process Reward Models (PRMs) in Test-Time Scaling (TTS). Through a theoretical MAP framework, we show that the optimal aggregation strategy is weighted majority vote combining signals from both the LLM dependent and, critically, assign large negative values to penalize low-quality responses powerful signal neglected by standard methods. We propose simple calibration methods to learn these functions. Our cal- ibrated weighted voting boosts TTS efficiency, achieving superior accuracy with approximately 37.1 % and 21.3% computational cost. This work demonstrates that intelligent aggrega- tion is more efficient path performance gains than simply scaling tist compute.
PRM. Empirically, we find these optimal weights are model-
baselines like Best-of-N
10

---

Published as conference paper ICLR 2026
ACKNOWLEDGMENT
This work was partially supported by the National Artificial Intelligence Research Resource (NAIRR) Pilot under awards NAIRR250400 and NAIRR240283, and Standing Up to POTS.
REFERENCES
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchir Nakano, Christopher Hesse, and John Schulman. Train- ing verifiers to solve math word problems, 2021. URL arXiv: 2110.14166 [cs.LG]]
DeepSeek-AL. DeepSeck- RI: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025.
12948 eprint: 2501.12948.
Jujie He, Tianwen Wei, Rui Yan, Jiacai Liu, Chaojie Wang. Yimeng Gan, Shiwen Tu, Chris Yuhao Liu, Liang Zeng, Xiaokun Wang, Boyang Wang, Yongcong Li, Fuxiang Zhang, Jiacheng Xu, Bo An, Yang Liu, and Yahui Zhou. Skywork-ol Open Series, November 2024. URL /tttps://doi... Version Number: 1.0.0.
5281/zenodo. http//2eodo..639888880
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS, 2021.
Yuki Ichihara, Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, Kenshi Abe, Mitsuki Sakamoto, and Eiji Uchibe. Evaluation of Best-of-N Sampling Strategies for Language Model Alignment, February 2025. URL httt:://arxiv: org/ abs 2502. 12668 arXiv:2502.12668 [cs].
Wendi Li and Yixuan Li. Process reward model with q-value rankings. In The thirteenth international conference on learning representations, 2025. URL
WQBch2cgEk
Hunter Lightman,, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. The tweifth interna- fional conference on learning representations, 2024. URL https: dittps://openrevite new..nee///eeee forum?
id=v8L0pN6E01
Fan Liu, Wenshuo Chao, Naiqiang Tan, and Hao Liu. Bag of Tricks for Inference-time Computation of LLM Reasoning, 2025. /tRt:t:s:///::xttt org. abs / 221///2222000 .eprint: 2502.07191.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iterative Refinement with Self-Feedback, May 2023. URL httthhtt///axat org/ abs/2303.330 17651 arXiv:2303.17651 [es].
Long Ouyang, Jeff Wu, Xu Jiang. Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano. Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of the 36th international conference neural information processing systems, Nips 22, Red Hook, NY, USA, 2022. Curran Asso- ciates Inc. ISBN 978-1-78877 173-1-71387--08-55 Number of pages: 15 Place: New Orleans, LA, USA tex articilc:o 2011.
11

---

Published as a conference paper ICLR 2026
Charlic Victor Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling test-time compute optimally can be more effective than scaling LLM parameters. In The thirteenth international conference on learning representations. 2025. httLhttts///ppeeet httpeeiwwwwweeeeeeeeeeeeeeeeeeee htt.://oooorio---oo/eattttt
Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models, 2025. URL https://aaxiv..rrrrr..r// abs/ 2501. 03124
Leprint: 2501.03124.
Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. Confidence improves self-consistency in LLMs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguisties: ACL 2025, pp. 20111, Vienna, Austria, July 2025a. Association for Computational Linguistics. ISBN 899-8-89176--56-33 org 2025. Findingssaann acl. 1030/
doi: 11885//1120225.iig acl.ing--all.100...
URL htttt:t//////m. /tttsss//claatthlllooe
Amir Taubenfeld, Tom Sheffer. Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. Confidence Improves Self-Consistency in LLMs. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 20090-20111, 2025b. doi:
105333/1/222.inniiiiii attt:://axivv/ org/ abs 2502 06233 arXiv:2502.06233 [cs].
URL
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome- based feed- back, 2022. URL https: / //tp///avvvv///////a// orgorrrggoo abs 2211 14275 arXiv: 2211.14275 [cs.LG].
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd annual meeting of the asso- ciation for computational linguistics (volume 1: Long papers), pp. 9426-9439, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ acl- URL https: //a::///cccaa http://cllanttllloo.oorrrg///// 2002 ac welooon51100
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The eleventh international conference learning representations, 2023. htt.:ttts:://ppp
net het/for 1it.Lm111---LPPPMMM
Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang. An Implementation of Generative PRM, 2024. URL /tttts::/githuin com/ RLHF www.//////// /RLHF- Reward-Modelimg Modeling Publication Title: GitHub repository.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self- Improvement. arXiv preprint arXiv:2409. 2024.
Qiyuan Zhang, Fuyuan Lyu, 7exu Sun, Lei Wang. Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, Xue Liu, and Chen Ma. A Survey on Test-Time Scaling Large Language Models: What, How, Where, and How Well?, May 2025a. URL http::tttt/artt orq aaa.abb......223555 arXiv:2503.24233 [cs].
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning, 07301 arXiv: 2501.0730 [cs.CL].
2025b. URLhttps: //arx/vaaivv.///////aa//.a..awva
12

---

Published as a conference paper ICLR 2026
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. ProcessBBnch: Identifying process errors in mathematical reasoning, 2024. URL arXiv: 2412.06559 [cs.Al].
https////r.xvv..r//ab/////////// 06559
13

---

Published as conference paper ICLR 2026
A PROOFS
A. DERIVATION OF THEOREM 3.2
Our objective find the answer á that maximizes the posterior probability P(an g P, M, V). Assuming a uniform prior over answers, this is equivalent to maximizing the log-likelihood, LL(os). From the main text, log-likelihood under Assumption 3.1 is:
LL(ax) Σ log P(pilgi, ou, V)
Σ log Plailaa, M)
Let E {0. }} be binary variable indicating whether generation is correct (c; 1) or incorrect (c: 0). Under the hypothesis that the true answer Ork, the correctness of gi is determined by answer 8;- Specifically, c 1 if s; ag. and = 0 if * og.
We can now analyze the two components of the log-likelihood separately.
Part 1: The PRM Signal Term
The first sum can be split based on whether generation's 's answer s; matches the candidate answer og:
log P(pilo, ak, v)=
Σ
log P(pelg. = 11
E
log c 0,
(4)
E:
To isolate the terms relevant to the maximization over ork, we rewrite the second sum by noting that Eiscton ()) E/m( Eisomaa (+):
Σ log ( Σ
log P(p.la..==0,V))
E
log c = V)
d:sy 00
im
i:sy Bss
(log P(pilgg 1, log -,
0
log P 0,
(5)
i:
The second term, Σr log P(pilgs, = 0, V). is a sum over all L generations. Since this term does not depend on the choice of the candidate answer ar it a constant with respect to our maximization problem and can be dropped. This leaves us with the ce-deepedentt part of the PRM signal:
P(pdle e; 1) log P(pilgi, -00
PRM Ter)(ou:) = Σ
(6)
Part 2: The LLM Signal Term
Next, we analyze the LLM term, simplifying P(gilou, M) to P(siaak, .M. We use the model's probabili-
ties: P(s; 0%k pala,, M) ạM and, for any 8, F 0g P(s (ag, M) = (1 qM))(m 1). Let N₂, be the
14

---

Published as conference paper ICLR 2026
count of generations where the answer ag,, i... N2. Kilss ax
L
log Plsilaw, M) = Σ
log P(s; = ark an. M) +
Σ
log P(si o on M)
te
1 9M
Np log qM + (L - Nkk log
1 9M
1 - 9M
NR log QM + L Llog
NNx log
-Nk [lo qM log
1 QM
1 9M
log
(7)
m
m.
Similar to the PRM term, the second part, I. log
1-qM does not depend on the specific candidate answer m-I
(as QM, L, m are fixed for a given question) and can be dropped from the objective function. The remaining term
9M- (m 1)
QM (m 1)
LLM_Term(nn Ng log
Σ 00
log
qM
1 9M
Part 3: Combining the Terms
Maximizing LL(as) is equivalent maximizing the sum of the ay-dependent terms we derived. Let this new objective function be Score(ax);
Score(ax) = (ermaas(ass) + WLL..TeMMml........
P(pilq 1, V)
qM (m - 1)
Σ us
log
+
Σ log
P(p. = V)
1 9M
P(pile = 1, log
(m 1)
P(pic 0, 0) log
(9)
1 qM
This is a weighted majority vote, where the final score for an answer ag is the sum of weights wi for all generations di that produced that answer. The weight for each generation is:
P(prle= = V) log P(pi(c; 0, V)
JM (m - 1)
+ log
qM
PRM Signal Term
Signal
This completes the proof.
B ADDITIONAL EXPERIMENT RESULTS
B.1 DETAILED RESULTS THE MATH500 DATASET
We show the detailed results on each LLM-PRM pair on the MATH500 dataset in Table 5
B.2 ANALYSIS ON KERNEL ESTIMATION BASED METHOD.
To validate the theoretical model itself and understand the relative underperformance of KDE WV, we ablate the optimal estimation with estimation from KDE. We add two baselines, KDE (Opt LLM term) and KDE
15

---

Published as conference paper ICLR 2026
Table 4: Ablation of the Optimal estimation at Sample Size n=32
Mistral-7B
Qwen2.5-1.5B
Qwen2.5.7B
DeepSeck-1.5B
DeepSeck-7B
Average
Qwen- PRMSOUK- 7B
Optimal
53.9
64.2
70.5
57.4
57.1
60.6
(Opt temn)
61.5
67.6
53.5
52.8
57.3
KDE WV (Opt LLM term)
52.7
61.1
67.5
51.7
54.2
57.5
KDE WV
50.1
60.6
66.8
51.8
$1.6
56.2
PRM-
Optimal
74.3
66.4
63.6
65.8
KDF WV (Opt PRM temn)
54.0
65.2
71.4
61.3
61.2
62.6
KDE WV (Opt LLM tem)
53.7
63.1
69.4
63.2
62.8
62.4
KDE WV
50.7
61.6
68.6
55.1
56.3
58.5
Misiral.
Optimal
KDE WV (Ope PRMtem) tem)
55.1
63.0
70.6
58.5
58.5
61.2
WV (Opt temn)
KDE WV
51.6
61.6
67.5
55.1
57.1
58.6
Llama3.1 DS- 8B
Optimal
65.7
73.0
64.2
62.0
64.2
KDE WV (Opt PRM lemn)
54.1
62.9
70.0
58.4
58.5
60.8
KDE WV (Opt LLM tern)
53.8
61.8
69.1
61.2
60.6
61.3
KDE WV
50.3
60.7
67.5
54,0
53.8
57.3
Skywork- PRM- 13B
Optimal
55.9
72.0
74.1
66.0
68.3
67.3
KDE WV (Opt temn)
71.3
60.5
KDE WV (Opt LLM tenn)
55.0
69.2
71.8
65.0
67.4
65.7
KDE
(Opt PRM term), which substitute the LLM and PRM terms of KDE WV with optimal estimations, respec- tively. As shown in Table using optimal estimation for the LLM and PRM term both lead to significantly better results, demonstrating that the gap between KDE and "Optimal" indeed due the difficulty of practical estimation, not flaw in the theory.
B.3 ADDITIONAL RESULT ON SELF-REWARDED VERIFICATION SIGNALS
CISC Taubenfeld et al. 2025a) is an approach that uses the confidence of the LLM as verification signals. While the verification signal is not from PRM, aggregation method is still applicable CISC utilize the signal more effectively. Here, we use the best-performing "P(True)" implementation of CISC to examine the effectiveness of the proposed method in aggregating self-rewarded verification signals. As can from the results on the MATH500 dataset below, the proposed method can more effectively aggregate the self self-rewarded verification signal than vanilla weighted majority voting and Best-of-N.
B.4 VARIANCE BETWEEN THE OPTIMAL WEIGHTING FUNCTION OF INDIVIDUAL QUESTIONS
In addition to the five qualitative examples, have added quantitative metric. We first scale the output range of the scoring functions to (-0.5, 0.5), and then compute the Mean Squared Error (MSE) between the dataset-wise estimated PRM score function and the per-question-optimal function, aggregated over all questions the test set. This provides more comprehensive view of the variance. As we can see Table 7 the MSE between the dataset-wise estimated PRM score function and the per-question-optimal function inherently significant, which validates our hypothesis that the optimal weighting function for each individual question varies largely, acting as one major challenge towards the optimal weighting function.
B.5 RESOLVE THE DISTRIBUTION INCONSISTENCY GAP AMONG QUESTIONS
We conducted additional experiment using pseudo- aseudo-PRM PRM scores sampled from identical distributions for correct and incorrect responses. This simulates the behavior our aggregation methods assuming the dis- tribution inconsistency is resolved. As shown in Table 8 when PRM scores follow consistent distribu-
16

---

Published as conference paper ICLR 2026
Table 5: Accuracy of Aggregation Methods at Sample Size n= n=112, MATH500 dataset
PRM
Method
Mistral-7B
Qwen2.5-1.5B
Qwen2.5-7B
DeepSeek-L.5B
DeepSeek-7B
Average
Qwen-
Optimal
68.0
58.3 0.01
62.7
0.0
BoN
33.0 0.0
0.0
62.0
40.3
49.0
48.3
MV
29.3
61.3 0.0
66.0 = 0.0
52.7 2 0.0
57.0
53.3
Vanilla WV
31.7 0.0
63.7 + 0.0
65.7 = 0.0
53.0 0.0
56.7 0.0
54.1 0.0
KIDE WV
31.3
62.0 0.0
66.3 0.0
53.0 + 0.0
57.0 0.0
53.9
62.9
66.3
2.9
57.0
Logit WV
32.0
62.9
66.3 0.0
52.7 + 0.0
$6.0
$4.0 0.3
Qwen- PRM- 78
Optiaal
44.3 + 0.0
70.3 +.00
73.0 + 0.0
67.7 0.01
67.7 + 0.0
64.6 + 0.0
BoN
40.0 - 0.0
63.7
67.3 = 0.0
$9.7
::.0:00000
00.1:0000
MV
2.0
6..3
0.00
52.7
57.0
$
Vanilla WV
0.0
6.00
67.3
57.7
62.3
57.3 0.0
KIDE WV
33.0
63.0+099
66.8 + 0.8
56.1 + 0.2
62.1 + 0.4
56.2 0.4
Linear WV
3.00
63.9
67.8 :: 0.4
63.0 0.6
65.7 +: 0.0
59,3 0.2
Logit WV
+..
64.2 0.4
68.0 0.6
63.3 0.0
65.3 0.0
59.7
Llama3.1
Optimal
6.0
BoN
32.7 0.0
5..0
49.3
55.3
MV
29.3
61.3
66.0 0.0
5..7
57.0 0.0
53.3 0.0
Vanilla WV
0.0
60.0
67.0 0.0
53.7 0.0
58.7 2 0.0
53.9 4.0 0.0
KDE WV
10881...
61.0
6..0
54.0 + 0.0
58.2
54.0 0.3
Linear
0.3
60.2
6..0
55,4
59.3
54,4 0.4
WV
31.8 0..4
$9.4
66.3 + 0.6
55.3 + 1.5
59.4 0.4
54.5 0.3
Llama311- DS- 8B
Optimal
39.0
67.7
69.7 = 0.0
6..3
0.0
60.7
BoN
27.0
51.0++0.0
62.3 + 0.0
51.3+
59.0 0.0
50.1 0.0
MV
2.0
61.3 0.0
52.7
57.0
$3.3
Vanilla WV
30.0
0.0
5.0
60.0
54.2
KDE WV
29.8 0088:000 0.2
61.0
6.22
55.1 + 0.2
58.9 4: 1.1
$4.2 1.2:0.2.
Linear Wv
29.3
0.0
61.0
66.1 0.2
55.0 1.2
61.2 + 0.7
$4,5
Logit WV
29.8 0.4
60.1 +..1
66.3 0.0
$7.1 + 1.0
61.1 + 0.4
54.9
Skywork- PRM-
Optmal
0.0
66.3
71.0
63.3 0.0
66.3
38.3+0000
56.7
MV
0.0
0.0
66.0 6.00
5.07
57.0
53.3
Vanilla WV
35.7=00.0
61.7
68.3 0.0
58.0 0.0
60.3 + 0.0
56.8 + 0.0
KDE WV
3.76
61.3
66.0 0.6
$5.6 0.8
58.3 0.7
$5.0
Linear
39.0
61.3
67.1 0..1
+ 1.2
59.0 2.9
$7.3
Logit WV
37.7 77.-000..
61.7 0.0
67.0 0.0
60.6 0.8
58.6 3.5
57.1 0.7
Table 6: Aggregating the self-rewarded verification signal from CISC.
Aggregation Methods
Accuracy
BoN
36.7
MV
61.3
Vanilla WV
61.3
KDE WV
61.3
Linear WV
61.7
Logit WV
62.0
tion independent of the specific question and response, the KDE method improves significantly, surpassing parametric aggregation approaches. This result aligns with our expectations and further validates both our analysis of KDE and our proposed theory.
17

---

Published as conference paper ICLR 2026
Table The MSE between the Kataset-wise estimated PRM score function and the per-question-optimal function.
PRMs
Mistral-7B
Qwen2.5-1.5B
Qwen2.5-7B
DeepSeek-7B
DeepSeek-1.5B
twen-PR.MB80K..
0.165
0.173
0.361
0.219
0.108
Qwen-PRM-7B
0.132
0.103
0.127
0.123
0.101
Llama3. --Mistral-8B
0.103
0.123
0.257
0,182
0.110
Llama3. Comma311-DS. 1-DS-8B
0.116
0.142
0.166
0.173
0.183
Skywork-PRM1. PRM-7B
0.141
0.166
0.294
0.226
0.112
Skywork-PRM-I.5.
0.162
0.171
0.224
0.261
0.152
Shepherd-Mistral-7B 0.465
0.380
0.263
0.365
0.423
Table 8: Performance of aggregation methods on a pseudo PRM whose score for correct and incorrect samples drawn from single distribution independent of the questions, respectively.
PRM
Method
Mistral-7B
Qwen2.5-1.5B
Qwen2,5-7B
DeepSeek-1.58
DeepSeek-7B
Average
Pseudo PRM
Optimal
40.7
67.9
70.8
63.0
67.0
61.9
BoN
33.1
56.
62.9
51.1
57.4
52.1
MV
29.0
59.7
66.0
53.3
58.3
53.3
Vanilla WV
32.5
60.8
66.8
55.1
59.5
54.9
KDE WV
38.3
62.4
66.9
57.3
60.4
57.1
Linear WV
33.6
60.3
66.3
55.5
61.6
55.5
Logit WV
33.7
60.3
66.6
56.5
61.8
55.8
C DISCUSSION ON CALIBRATION COST
the calibration cost is negligible one-time investment compared to the substantial, recurring test-time sav- ings. Our Logit Weighted Voting method matches the baseline accuracy while strictly requiring only 21.3% of the computation, effectively saving 78.7% of the compute per query. Because our calibration relies on a small set of 100 questions, this initial computational cost recovered after serving just 127 user queries (100/0.787 x 127). In any practical deployment serving thousands of requests, this break-even point is reached almost immediately, rendering the calibration overhead insignificant relative the long- term efficiency gains.
D LIMITATIONS AND FUTURE WORK
Limitations. Our work, while demonstrating significant gains. has several limitations. First, our theoret- ical framework relies on conditional independence assumptions which are simplifications of the complex dependencies between generated responses. Second, our proposed calibration methods learn a single, global weighting function. As our analysis in Section 4.3 shows, the truly optimal function varies on a per-question basis, and our attempts to learn meta-model to predict these per-question functions were unsuccessful, indicating this is non-trivial challenge. Finally, while effective, our calibration methods require a small, one-time labeled dataset, and our evaluation has been focused on the domain of mathematical reasoning.
Future Work. These limitations point to several promising avenues for future research. The primary chal- lenge is to bridge the gap between global and per-question optimal weighting. Developing methods that can adapt the weighting function at test time based on question-speciff features or initial response char- acteristics could yield further performance gains. Another direction is to explore the generalization of
18

---

Published as conference paper ICLR 2026
calibrated aggregation framework to other domains beyond mathematics and other TTS paradigms, such sequential refinement or tree-of-thoughts search. Lastly, investigating semi-supprvised or unsupervised calibration techniques could reduce the reliance on labeled data, making the approach more accessible and scalable.
Ethies Statement. This research aims to improve the computational efficiency of large language models, a goal with positive ethical implications. By achieving higher performance with less computational cost, our methods can contribute to reducing the energy consumption and environmental impact associated with deploying state-of-the-ard AI systems. The datasets and models used in this work are standard, publicly available benchmarks and open-source models widely used by the research community. Our work does not involve human subjects, nor does it introduce new capabilities that would increase the risk of misuse language models. On the contrary, by developing a more nuanced understanding of how to verify and aggregate machine-generated reasoning. this research could contribute to making LLMs more reliable and less prone to generating confident but incorrect outputs. All authors have read and adhered to the ICLR Code of Ethics.
Reproducibility Statement. We are committed to ensuring the reproducibility of our work. All LLMs and PRMs used are publicly available models, and provide details of these models in Section 5.1. The ex- periments were conducted on the MATH dataset, standard public benchmark. Our data splitting procedure for calibration and testing described in Section 5.1. The full theoretical derivation for Theorem 3.2 provided in Appendix A.1. Key implementation details for our proposed calibration methods, including the KDE procedure and the grid search ranges for parametric models, are described in Section
LLM Usage. We use I.MSs to polish the writing of this paper, including identifying spelling, grammar mistakes. All suggestions from the LLM are verified by the authors before being incorporated into the paper.
19


Reviews:
[{"review_id": "VJJVlRqpfD", "reviewer": "Reviewer_suxy", "summary": "In this work, the authors propose a MAP framework for using Process Reward Model signal to guide test time scaling in LLMs on mathematical reasoning tasks. Through theoretical analysis, they propose a weighting over completions based on a PRM signal and LLM signal, which better incorporates PRM signal into test time scaling than baselines. Importantly, they find that allowing for negative weights on completions with low PRM scores and also calibrating to the expected correctness of the LLM help improve the efficiency of test time scaling.", "questions": "* What is the objective used to calibrate the parametric objectives? Is it correctness on the calibration set?\n* In Figure 4, I would like to see the MSE between the dataset estimated PRM score and the per-question-optimal for the whole dataset, this might be more convincing than having 5 picked questions as examples.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The authors have an intuitive and clear theoretical framework guiding their proposed method.\n* Through experiments, they demonstrate that considering this theoretical framework can help practitioners make test time scaling more efficient.", "weaknesses": "* After introducing the MAP model and proposing a way to estimate it using KDE, the empirical results find that logit weighing with grid search performs as well if not better than KDE. This gap makes me wonder why KDE is necessary in the first place, and if it is less necessary, calls into question the utility of the model. The authors do note that this drop in performance could be attributed to error in estimating $q_M$, but they do not validate this in the paper. I would be interested in seeing an ablation using ground truth $q_M$'s to validate the KDE approach and the theoretical model. On this note, I would also like to see elaboration on the sentence \"In conclusion, accurately estimating either the PRM or the LLM part of the weight requires nuanced estimation for individual questions, explaining why the non-parametric KDE estimation underperforms the parametric ones.\"\n\n\nNotation:\n * When you say \"surpasses the performance of ... with approximately 37.1% and 21.3% of compute of compute\", I think this needs to be tweaked gramatically. Also, I would say \"Test time compute\" instead of \"compute\" as to not confuse readers about other axes of compute such as FLOPs or training time. My immediate thought was that compute should be the same because you run the same number of forward passes through the PRM, for a fixed N but now I understand that the point is you can get the same accuracy with fewer N.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VMhlXSo8oA", "reviewer": "Reviewer_hEsP", "summary": "This paper focuses on how to leverage the signals from PRMs (process reward models) in test-time scaling. They show empirical results with an optimal weighting function, incl. practical calibration methods to learn these functions. They show that using these methods improves test-time-scaling efficiency.", "questions": "The MATH dataset is from 2021, do we think that this approach will generalize to novel unseen benchmarks?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper presents results across 5 LLMs and 7 PRMs and they show that this helps boost test-time scaling efficiency. They also present different aggregations (see Table 1) which shows better results across multiple model-PRM pairs and the MATH datasets.  They achieve that with way less computation (see line 71 and 306 for details).\n\nThey also compare with multiple baselines, such as majority voting, best of N, vanilla weighted vote. Their weighting function calibration significantly improves TTS efficiency.\n\nThey also showcase how negative weights are necessary in using PRM signals", "weaknesses": "The paper presents very simple concepts in the abstract but does it in a way by introducing nomenclature that makes it hard to understand. For example, it is not clear why PRMs are important for TTS and their applicability. I would start with that, then present the actual efforts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1atAjQtsa3", "reviewer": "Reviewer_rGBo", "summary": "This paper investigates into the utilization of the Process Reward Models (PRMs) in test time scalings (TTS) for LLMs. The authors argue that simple majority voting sometimes outperforms the PRM guided method (such as Best-of-N selection) which indicates the usage of PRM is sub-optimal in TTS. The authors propose a framework with Maximum a Posteriori (MAP) estimation and demonstrate the optimal aggregation method for TTS is via weighted majority vote (with PRM). Besides the theoretical framework, they also propose two practical methods based on KDE and parametric learning to learn these weighting functions from a small labeled dataset. They empirically show the proposed method can achieve similar accuracy as baselines while using only 21.3%-37.1% of the computes, on 5 LLMs and 7 PRMs.", "questions": "1. How sensitive is the method to the violation of the Assumption 3.1? I am wondering whether you have some empirical data points to show the correlation between responses or the PRM scores and see how does this correlation impact the final performance?\n\n2. Did you perform ablation study to see how the calibration set domain change can impact the test set accuracy? I would like to understand the domain transferability of the method. It can be truly different domain, different types of math problems or even the difficulty of the same type of math problems. \n\n3. I think we are missing some baselines, have we compared with other type of methods, such as in [1] and [2]? To be fair, the baselines we compared in the paper are quite simple. [2] is a bit complicated as they require to train a model with RL, however, the method in [1]\n is very straightforward and not necessarily need the calibration dataset.\n\n[1] Confidence Improves Self-Consistency in LLMs, Taubenfeld et al., ACL 2025\n[2] The Majority is not always right: RL training for solution aggregation, Zhao et al.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. In general the paper is well written and the presentation is pretty clear. The proposed theoretical framework (i.e., the MAP based method)  is formulated in a clean way and mathematically makes sense. \n\n2. The experiments are quite comprehensive. The authors verified their claims across 35 LLM-PRM pairs and demonstrate consistent results on these models and datasets. \n\n3. The experiments show promising results. The proposed method is able to achieve similar accuracy as other baselines with only 21.3%-37.1% of the compute, which will be somewhat useful in the real world applications.", "weaknesses": "Despite the clear presentation and the clean mathematical formulation, I have the following concerns about the paper:\n\n1. The independence assumption (i.e., assumption 3.1) is quite strong. Assumption 3.1 assumes that PRM scores and the LLM generations are conditionally independent of each other given the true answer, the verifier and the model. I would argue that both LLM generation and the PRM scores are very unlikely independent of each other, instead, the LLM is likely to generate responses that are \"correlated\" with each other. This assumption is critical, however, the paper doesn't discuss/validate it empirically, neither do it discuss the potential impact on the conclusion. \n\n2. The ablation study is not sufficient, especially on the calibration dataset. There are a few critical questions that can impact the generalizability/applicability of the method: 1/ How does the distribution shift between calibration and test set affect the performance? How much can the calibration results be transferred to a different domain? 2/ Even within the same domain (i.e., math), how much does the calibration results transfer between difficulty of the problem, or formulation of the problem? So on and so forth. \n\n3. Missing statistical significance: The authors report absolute results in table 1 and the best numbers are flagged in bold. However, the score gaps with other baseline are usually very small, I would strongly recommend to include standard deviation or confidence intervals to indicate whether these improvements are statistically significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QXpjyMs7ZZ", "reviewer": "Reviewer_z2Y8", "summary": "The paper proposes a new test-time scaling algorithm for LLMs when using PRMs (process reward models). The core of the method consists of computing a score for each answer appearing in a pool of generated answers, where the score depends on an *PRM Signal Term* and a *LLM Signal Term*. The authors propose ways of estimating these terms for a given LLM-PRM pair.\n\nIn its current form, **I lean towards rejection of the paper.** As outlined below, the theoretical foundation seems imprecise, evaluation is not very extensive (only MATH/MATH500, which means it remains unclear to what degree results such as the calibration weights would transfer across datasets), and accuracy gains are marginal without a clear way of selecting one of the proposed methods in practice for a given LLM-PRM pair. However, the paper certainly proposes some interesting ideas, and if the above points are addressed, it could warrant acceptance.", "questions": "- lines 121-124: Are $s_i$ and $\\alpha_i$ the same thing? Or are they different? In particular, if they are the same, then assuming a uniform prior over answers doesn't really make sense; if, on the other hand, the $\\alpha_i$ are supposed to constitute the set of *all* possible answers, then it doesn't seem to make sense to assume this set is finite.\n- what do the red dots in Figure 1 represent? This should be explained in the figure caption.\n- is the probability $q_M$ (line 165) assumed to be specific to a prompt (i.e. varies across prompts)?\n- Some of the figure fonts are hard to read (e.g. Figure 2, 3).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The results on PRM score distributions across different LLM-PRM pairs (e.g. Figure 1) are quite interesting and could be of independent interest to the research community.\n\nTheorem 3.2, the derivation of picking the best response in a pool of responses by computing a response-level score depending on an LLM and PRM signal, is quite interesting (yet there remain questions about the derivation, see below).\n\nThe paper is mostly well-written and clear.", "weaknesses": "### Theoretical Foundations\nThe theory is imprecise and unclear.\nFor example, the derivation from Bayes' theorem in Section 3.1 does not make sense to me. First, it is unclear how this probability $P(\\alpha_k | \\mathcal{G},\\mathcal{P},M,V)$ is even defined (is this supposed to be the probability that $\\alpha_k$ is the correct answer under the given conditioning? If so, what is the randomness over?), and why maximizing it is the desirable thing to do. Next, the authors say they can decompose the probability $P(\\mathcal{G},\\mathcal{P}|...)$ into $P(\\mathcal{G}|...)\\times P(\\mathcal{P}|...)$, which would only be possible if the (conditional) probabilities of $\\mathcal{G}$ and $\\mathcal{P}$ were independent, which clearly does not seem to be the case.\n\n### Experiments\nThe experiments do not contain any confidence intervals/error bars. This seems to be quite important, as e.g. Figure 2 shows that the entire gain in compute could very well be within the margin of error.\n\nWhile the authors test out many LLM-PRM pairs, they only seem to test on MATH and MATH500, the latter of which is known to be quite easy. Results on other datasets would greatly improve the quality of the experiments. Furthermore, it is often not even clear what dataset the results are reported on (e.g. Table 1 does not contain any information on this in the text or caption. From the previous paragraph, it is likely either MATH or MATH500, but it is unclear which one of these). In particular, it is completely unclear if any of the calibrated weights would transfer across datasets, which, in the current version of the paper, makes it seem like this is an entirely dataset-dependent method.\n\nAn explanation of compute tradeoffs would help evaluate the proposed method better. In particular, in order to compute the calibrated weights, a calibration set (with LLM responses) has to be generated. This seems to be quite expensive in practice.\n\nThe results show that accuracy gains can be obtained over baselines like best-of-n (BoN) or majority voting (MV), but these gains are not very consistent, neither across LLM-PRM pairs, nor across the proposed methods (KDE vs Linear vs Logit); it remains unclear how to pick the best method for each LLM-PRM pair in practice. Furthermore, the improvement gains are often not very significant, and sometimes the proposed method is even outperformed by the baselines (cf. Table 1).\n\nTo summarize, the paper proposes an interesting approach to test-time scaling that's certainly worth exploring more. However, the main drawbacks I see are a) questionable theoretical derivation (Section 3.1); b) limited experimental validation (just one dataset, no results on compute overhead of the proposed method); c) marginal accuracy gains in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)