Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decoding
Authors: Yiming Wang
Pei Zhang
Siyuan Huang
Baosong Yang
Zhuosheng Zhang
Fei Huang
Rui Wang
Venue: NeurIPS 2025 spotlight ()
Keywords: Large Language Models
Test-Time Scaling
Best-of-N Sampling
Efficient Decoding
Self-Estimation
Primary area: deep_learning
TL;DR: We propose the ST-BoN sampling decoding, which avoids fully generating and reward models, and can reduce GPU memory overhead and inference latency while achieve better cost-performance trade-offs.

Abstract: Test-time scaling enhances large language model performance by allocating additional compute resources during decoding. Best-of-$N$ (BoN) sampling serves as a common sampling-based scaling technique, broadening the search space in parallel to find better solutions from the model distribution. However, its cost–performance trade-off is still underexplored. Two main challenges limit the efficiency of BoN sampling:
(1) Generating $N$ full samples consumes substantial GPU memory, reducing inference capacity under limited resources.
(2) Reward models add extra memory and latency overhead, and training strong reward models introduces potential training data costs.
Although some studies have explored efficiency improvements, none have addressed both challenges at once.
To address this gap, we propose **Self-Truncation Best-of-$N$ (ST-BoN)**, a decoding method that avoids fully generating all $N$ samples and eliminates the need for reward models. It leverages early sampling consistency in the model’s internal states to identify the most promising path and truncate suboptimal ones.
In terms of cost, ST-BoN reduces dynamic GPU memory usage by over 80% and inference latency by 50%.
In terms of cost–performance trade-off, ST-BoN achieves the same performance as Full-BoN while saving computational cost by 70%–80%, and under the same cost, it can improve accuracy by 3–4 points.

Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decoding
Yiming Wang
Pei Zhang
Siyuan Huang" Baosong Yang
Zhuosheng Zhang
Fei Huang Rui Wang bi
"School of Computer Science, Shanghai Jiao Tong University i Tongyi Lab, Alibaba Group
Corresponding Author
Email: (yiming. wang, vangrui12)@sj edu.
Abstract
Test-time scaling enhances large language model performance by allocating addi- tional compute resources during inference. Best-of-N (BoN) sampling serves common sampling- based scaling technique, broadening the search space in parallel to find better solutions from the model distribution. However, its perfopeeformnnc trade-off is still underexplored. Two main challenges limit the efficiency of BoN sampling: (1) Generating N full samples consumes substantial GPU memory, reducing inference capacity under limited resources. (2) Reward models add ex- tra memory and latency overhead, and training strong reward models introduces potential training data costs. Although some studies have explored efficiency improvements, none have addressed both challenges once.
To address this gap, we propose Self- Truncation Best-of-N (ST-BoN), a decoding method that avoids fully generating all N samples and eliminates the need for reward models. It leverages early sampling consistency in the model's 'ss internal states to identify the most promising path and truncate suboptimal ones. In terms of cost, ST-BoN reduces dynamic GPU memory usage by over 80% and inference latency by 50%. terms of cost perforrrrre trade-off, ST-BoN achieves the same performance as BuNl-BoN while saving computational cost by 70%-80% and under the same cost, it can improve accuracy by 3 points.
Fartlett Eatimation Time: movoiieet
Self-Extisnating likeb) Bes Sumpling
Self Trinnronciiigg
[PART
Themtor,
Sepphy
naoe
out
that
[4]
[-]
6-1
Fical Stover
Awpphy
6-1
1
1:
1:
Quediken: CompekyeethPff 101).
Large Language Model
Instruction seply segs put willie
Butter Windom (Langtee)
Gerghh
Figure 1: The pipeline and case of our Self-Trunn Best-of-N (ST-BoN) Decoding. From the start: (Step 1) We first let the LLM generate N samplings autoregressively until the earliest estimation time c; (Step 2) Then each sampling continues to generate steps beyond time c. During this, the LLM performs self-estimation to determine the most promising sample among the N sampling, identifying the best sample through cumulative + self-estimations. (Step 3) Finally, we truncate the remaining N - samples and proceed with generating only the hest-estimated sample up to [EOS]. Parameter details of the case in this figure: N 5, c = T = 20, and T' = 359.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Large Language Models (LLMs) possess strong generative and reasoning capabilities after extensive and the excellent solution to problem is typically embedded However, autoregressive decoding focuses on locally optimal solutions, overlooking more promising thinking paths. Hence, the test-time scaling technique 40 is proposed to help LLMs expand their thinking space by adding compute during inference.
training on large-scale corpora 7
I 10
within the model distribution 26
48
33
30
serves as widely used sampling- based scaling paradigm. By sampling N responses from the LLM and selecting the best one with a re-ranking strategy at inference time, BoN fully leverages the potential ability within the model distribution. The core of BoN sampling lies in scoring and re- re-ranking multiple candidates. A well-known method is ceff-consistency [47], which selects the most frequent answer as the final solution. Furthermore, with trained reward model (RM) 45 46 355 BoN sampling can handle richer tasks by scoring candidates and selecting the one with the highest score, aiming to increase test-time computational costs regardless of expense to enhance performance [40.
Best-of-NN (BoN) 42
Beyond performance, the issue of cost-performance trade-offss in BoN sampling has been less explored. Under existing paradigms, BoN sampling faces two key challenges that hinder efficient deployment: (i) Full Generation Overhead: Traditional BoN requires fully generating all N samples, referred to as Full-BoN in this paper. Although GPU parallelization can partially mitigate time latency, additional memory overhead unavoidable, especially for complex reasoning with long generated sequences. (ii) Limitations of Reward Models: RMs are effective, but occupy additional memory and time costs. Meanwhile, training a strong RM costly due to the searcity of high-quality feedback data, and their domain-specific nature restricts generalizability across tasks (e.g., from math to open-ended QA).
Some efforts have been made to improve the efficiency of BoN sampling by using RMs to score and discard sampling prefixes [4]], or to branch after pruning (25] 36), thereby reducing some memory overhead of full generation. However, RM inference still incurs non-negligible latency, and sharing GPU memory with generators further restricts caching capacity. Moreover, the inherent limitations of RMs make these methods less suitable for objective tasks like reasoning (see Appendix BB. Therefore, no existing method overcomes both challenges at once to achieve deeper efficiency optimizations.
To fill this gap, we introduce a novel decoding technique called Self-Truncation Best-of-N (ST- BoN). Inspired by the unsupervised advantage of consistency strategies (47), we first theoretically demonstrate the feasibility of foreshadowing the final consistency using sampling consistency in early decoding. Then, we design an internal consistency measure that uses latent space information [49] to identify and preserve the most promising sample among the N samples in early decoding. This framework allows us to effectively self-estimate and truncate suboptimal samplings in early decoding without relying on reward model intervention, thereby saving memory and speeding up generation.
The detailed pipeline of ST-BoN is illustrated in Figure I makes the following contribution
Cost: Compared to Full-BoN, ST-BoN truncates suboptimal samples in early decoding, thereby freeing up memory and reducing the dynamic GPU memory load by over 80% This early truncation also accelerates later generations, reducing inference latency by up to 50%. Notably, these efficiency gains are achieved without accounting for the overhead of extra reward models.
Poxt-Performance Trade-offs: We conduct extensive experiments to demonstrate that ST-BoN strikes strong balance between cost and performance. When reaching the performance of Full-BoN at the certain N values, ST-BoN can save computational cost by 70 % to 80%. Also, when consuming similar computational costs, ST-BoN improves accuracy by 3 to 4 points. In addition, ST-BoN is applicable across a wide range of domains. These results show that ST-BoN can provide flexible solution to balance cost and performance when faced with limited resources.
2 Preliminary
Autoregressive Decoding in Language Models. Let pa represent a language model. For a given prompt and question, Po will generate token sequence Yır 9132-.-9r autoregressively, where T
'Code and data available in http:///gghhuu.ccmmlaaaeeeeeeeee

---

is the generation length and [EOS]. Each token Yr is sampled as follows:
yt Po P (-Iproompt question, y-r)) E R
(I)
where V is the vocabulary. In greedy decoding, the model selects the token with the highest probability of pa to generate yt. In contrast, sampling decoding typically uses multinomial sampling, such as top- k 13 or top-p [20], to let models sample yt from po after truncation to generate diverse sequences.
Best-of-N Sampling. As the name suggests, Best-of-N sampling involves sampling N sequences to find the best one. Let Y ,,² _2..... Y be N independent sampling sequences, then each sample Y is assigned scalar score S(Y"), , and the best sample index is identified as arg max, (S((Y)) )
The simplest scoring approach is to use an externally trained reward model. If avoiding the inter- vention of reward models, the consistency strategy is the most common unsupervised alternative, introduced by self-consistency [47]. which found a positive correlation between response consistency and accuracy; that is, when multiple reasoning paths lead the same answer, confidence in its cor- rectness increases. In self-consistency, the exact answer A' is extracted from Y', and the consistency score S(Y+) is computed based on the frequency with which A co-occurs with other answers:
S(Y') 2j=1 Lizi 1(A' A²)
where 1(-) is the indicator function. More generally, "consistency" can be any continuous measure. For example, [2]] suggested using semantic similarity to assess sample consistency in open- ended tasks, calculated through the distance between sentence embeddings.
3 ST-BoN: Self-Truncation Best-of-N Decoding
3.1 Theoretical Support:
Early Consistency Foreshadows Final Consistency
We aim to overcome the efficiency bottlenecks of both full generation and reward models. Without a reward model, consistency strategies offer a promising unsupervised approach to select optimal samples. However, existing consistency methods require full generation. Naturally, we hope that early consistency can foreshadow the final consistency: a sample closer to others in early decoding is more likely to reach the correct answer when the decoding ends. If this holds, such consistency propagation can be utilized to estimate the most promising sampling in early decoding.
Intuitively, considering the cumulative effect of autoregressive generation, sampling sequences that already exhibit large differences in early decoding is less likely to converge closely by the end. Theoretically, this can be modeled as probabilistic monotonicity: as early consistency increases, the probability lower bound for final consistency surpassing a given constant also increases.
Consistency can be mathematically converted into a distance measure, where higher consistency implies smaller distance. Let D(; Y x Y + R+ be an arbitrary metric function defined over metric space, used to quantify the distance between two partial sequences Y and Y at time t. We demonstrate that early consistency can probabilistically enhance the likelihood of final consistency. Specifically, let y denote the primary sampling. At an early decoding time i (t < T), if the distance between the partial sequence Liee and the other N 1 partial sequences {Yiu)ssisN becomes smaller under a given metric D, the lower bound of the probability that the distance between the final full sequence Yi.r and the other N I full sequences (Yirr)siis is less than a certain value will increase, meaning it is more likely to meet a specific consistency requirement.
Theorem Let d; D(Y Yi.) denote the distance between sampling sequence Yi,, and the i-th sampling sequence Yi., at decoding time t. For a given constant e. there exist constant I such that:
T1
Pr S c St] 2
S
where 1 1t < T, ST Mi di and S, = ΣN i= di.
Proof Sketch. We assume (i) local Lipschitz continuity, ie., for any two prefixes, the next-token distributions differ in TV distance by most L times their sequence distance, and (ii) bounded
3

---

increments, so appending single token can increase the distance by at most M. Under maximal
coupling, two continuations diverge step + with probability most Ld;. Thus the expected distance evolves as B,d d:j v dj + M(Id;
(1 + L.M) di. Summing over i yields E[S < TS, with r = + LMM Iterating this recursion shows that deviations grow at most exponentially: E[Sr] < ST Finally, applying Markov's inequality converts the expectation bound into high- high-probability guarantee: Pr[ST 2 E S.] - 1"'",,/e, implying the claim in Theorem
The detailed proof is shown in Appendix A
As St increases, the probability lower bound of Sy S also increases. This confirms the potential for early consistency that foreshadows the final consistency.
3.2 Method: Early Internal Consistency Estimates the Most Promising Sampling
We have proven the high-probability foreshadow from early consistency to final consistency, providing theoretical feasibility for utilizing sampling consistency in early decoding to estimate the most promising path. Upon this, we are ready to present the main ST-BoN algorithm, which involves defining an effective measure of sampling consistency by designing a suitable metric function D.
When Does Delf-Estimation Start? Before defining the measure, we first identify the earliest self-estimation time for early consistency. We aim achieve ultimate efficiency improvement, so the earliest time can occur when all samplings become pairwise inconsistent. Let this special time be c, satisfies the following condition:
-n(ri.= = =0 & V [ J
Y(e O.
In practical inference, we enable GPU parallelization and perform pairwise sequence-equaliyy checks at each time. If any sample reaches [EOS] during this process, is terminated immediately, and the current time step is recorded as c. In practice, this situation is very rare, as the condition in Eq is typically satisfied early on, as shown in Figure
Main Algorithm: How is Self- Estimation Done? Now start to define an effective measure of sampling consistency. At such an early decoding stage, semantic information is limited, and textual differences between samplings are minimal, making it hard to distinguish them solely based on output text. Therefore, we consider utilizing the more informative hidden states of L.LMs to represent early sampling sequences through the model's internal information.
Chain-of-Embeddding (CoE) offers a promising idea, which is the representation technique in the latent space [49 that links all hidden states from input output, capturing the latent thinking path from reading to writing. It extracts hidden states across layers to form a progressive chain. Consider model with L layers and output length T, then the hidden state at layer 1 (0 S I I L) and position (1 < 1. < T) is represented as z/. The sentence embedding for layer I is calculated as Thus, CoE is defined progressive chain of sentence embeddings: hT.
hi = + 2 2 38 50
-
HET = hg hT
The CoE feature F(H) quantified as:
M(h, h441) A(hi, (1n1))+1)))) M(o, h2) A(ho, h2)
F(H 1 =
Σ 1=0
where M (h;,h,) = lh: h2yl2 and A(hs, h2) arccos (hT h3/ (11/4l12 [[bgl12)). A larger
F(H1:) indicates greater curvature in the latent thinking path when generating Y1.r-
Using the CoE measure, we define the distance D(Y Yi'e) between two partial sequences Yi. and Vi at time as the squared difference of their CoE features. The consistency score S(Y is then calculated as the average distance between the Y and the other N 1 samples. Specifically:
N
D(Y ride (z))=(((H))) -F((H).)) -F(rt)))F()..)) S(Y
1
D(Yie j=Lo
Σ
(5)
smaller S(Y:.) indicates higher consistency between the i-th sample and the other samples. After computing scores for all N samples, we rerank them and choose the sample with the lowest score: ia = arg min, (Y.).) making the i,-th sample the optimal estimate at time c.

---

How Long Does Self-Estimation Last? Performing self-estimation at single moment can in- troduce randomness, since pairwise sequence differences only begin to emerge time e, and these differences may not be substantial. To migrate this, we define a Buffer Window: the LLM continues to generate for an additional T steps beyond time e, where T is the window length. We set T x C, Le., 7 = mc, where m is a proportional constant. The intuition is that a smaller indicates an early divergence between samplings, which reflects greater randomness, so a smaller window may suffice to capture later differences. In contrast, larger e suggests lower randomness, which requires larger window to capture sufficient later differences. At each time within the buffer window, we obtain the optimal estimation ée' arg min, {s(Yi.c) ) using Eq This produces + 1 optimal estimation 1ctt and the final optimal estimation iiinal is determined by selecting the most frequent one: {ie Je'=e-
Count(i)
T(i i) i) (ISiSN),
Final = arg max (C)
(6)
4 Cost Analysis: Towards Efficient Best-of-N Sampling
Compared to Full-BoN, ST-BoN reduces significant computational cost during inference, which is reflected in two aspects: Space and Time. The space is related to GPU memory overhead, and the time is reflected in inference latency. We will analyze the two parts between ST-BoN and Full-BoN.
4.1 Inference Overhead: GPU Memory
During model inference, GPU memory overhead is mainly impacted by the model weights and the KV cache. While model weight loading is neces- sary, the KV cache is the main memory bottleneck. Autoregressive parallel sampling can quickly cause Out-of-Memory (OOM) issues due to KV cache ac- cumulation, with OOM events tied to peak memory overhead. Therefore, we assess ST-BoN'' mem- ory optimization by comparing its reduction in peak memory overhead to that of Full-BoN.
0.065
MATH TheeeenA NNLAA P.
SASTE
Densit
0.000 0
200
400
600
Token Length
Figure 2: The distributions P. and Pr of the earliest estimation time c and the full genera- tion length T across different datasets.
Assuming that N samples cache usage is linearly related to the sampling size N and the generation length T. In the dataset D, for inputs X D and outputs Y ~ X)) we let Pr be the length distribution of full generation, and P. be the distribution of the earliest estimation time c. When excluding basic model weight memory: (i For Full-BoN without reward models, peak memory occurs at time T with batch size N. Note that if a reward model is added, the basic memory overhead of Full-BoN will increase, further reducing the available inference space. (ii) For ST-BoN. the peak memory can occur at time e with batch size N, or at time 7' with batch size 1. We set m = 1 in our main experiments, so that 7 c, and the memory reduction rate RD on dataset D can be approximated as:
generated in parallel the GPU (i.e., the batch size is N), the KV
RD max (N (Eewp, [cl + 7), ByMP, [T} 1 max
E Pr [d] 11 E ET Pr [T] N
(7)
K.ETPPLTT
We conduct statistical experiments on the dis- tributions P. and Pr using the Llama3-8B. Instruct model across three datasets: MATH
300
[r.xTT]
0.50
400
18 TheoremQA [8]. and MMLU [17]. Fig- ure visualizes the respective distributions, showing a clear shift to the left of P. com- pared to Pr. This indicates that the earliest estimation time e appears much earlier than the completion of sampling.
330
0.50
207
MATH
ThevernOA
22.00
26,07
MMLU
MATH
ThemeanQA
MMLU
066
30
Sampling Siae
Figure 3: (Left) The mathematical expectations Ee [c] and [T~PP,[]] of the earliest estimation time e and the full generation length 7' across differ- ent datasets; (Right) The computed memory reduc- tion rate Rt under different sampling sizes N.
Furthermore, we calculate Ex- [c] and ET-P, [7], set various sampling sizes N. and use Eq to find Rp. Figure illustrates that Rp easily surpasses 80% for N 2
5. According to Eq 7 the upper limit of Rp is
5

---

1- 2B (c)/Err,,,(T (T), and increases with the sampling size N until it reaches this limit. In addition, significant variations in Pr [T] compared minimal variations in E [c] make do- main optimization primarily dependent on By (7]. For example, the MATH dataset has longer full generation, leading to higher optimization upper limit than other domains. In summary, RD
increases with the sampling size N and the complexity of the task.
4.2 Inference Latency: Wall-Clock Time
Inference latency is another critical aspect that affects computational cost. In Full-BoN, while N-sampling can run in parallel on the GPU, it still introduces more delay compared to single-sample inference. Our ST-BoN method leverages early self-truncation enabling LLMs resume single- sample generation after time + 7. However, sequence equality checks and eeff-estimation within the buffer window require serial vector processing, potentially adding latency. To address this, will evaluate inference latency using wall-clock time as the measure.
We perform statistical experiments us-
Average Wall-Clock Time (5)
S1-BAN Nam!
Full-i BX
Pul-8oN w/
Gruady
ing the
model on the
MATH
TheoremA
MATH and TheoremQA datasets. For
50-
Full-BoN, we assess scenarios with and
60
30
without reward model, highlighting the
30-
-53%
extra time needed to load and run the re-
20
ward model. Figure 4 presents the re- sults, demonstrating that ST-BoN signif- icantly reduces inference latency com- pared to Full-BoN. Overall, wall-clock time latency drops by nearly 50%, with the reduction ratio growing as in- creases. This efficiency stems from the self-truncating operation, which compensates for delays from self-estimation. Additionally, incorpo- rating reward model into Full-BoN further increases inference latency.
10-A
10-
5
10
15
20
5
10
15
20
Sampling Size Size
Figure 4: The average wall-clock time (seconds) compar- isons in two datasets with different sampling sizes N.
Experiments
5.1 Setup
Datasets. We select four datasets for objective tasks: MATH [8], TheoremQA &. GPQA 37
and MMLU (17]. They span a range of domains, including mathematics, theorem application, science reasoning, and general knowledge, and present a significant difficulty level. We also select two datasets for subjective tasks: CNNDM [4] and AlpacaFarm [T]. The former is the summarization task about the open-ended generation, and the latter is the fnslluction-followiing task about the preference alignment. We adopt the revised CNNDM 51 for its higher-quality gold summaries.
Models. We mainly adopt 7B+ parameter models with the Zero-Shot-Col generation paradigm [55] Llama3-8B- In-trruet [10], and Fintaal-B-Instruet--e
52
23
including Qwen2.5-7B-Inss We also test the 5-t.ee..5572---tnnnnni
22
55 to validate generalization across model scales.
Baselines. We compare our ST-BoN decoding with two Full-BoN decoding paradigms: Full-BoN without a reward model (Full-BoN w/o RM) and Full-BoN with a reward model (Full-BoN, w/ RM). In objective tasks: Full-BoN w/o RMM uses the classic self-consistency through majority voting 47 and Full-BoN w/ RM employs the process reward model (PRM) Skywork-ol Soayoor-ol-Open-PPeee --ppen-PRM-777
In
subjective tasks: Full-BoN w/o RM applies the modified self-consistency through semantic similarity 45 for preference rewards.
21, and Full-BoN w/ RM uses the ArmoRM-Llama-33
Implementation. We use the sampling strategy combining top-k 13 top-p 20 and temperature 19,, with 20, p = 0.95, and T = 0.7. The buffer window length 7 is set to C in our main experiments. Detailed hyperparameter analysis is provided in Section 6.2 and 6.3 All basclines are implemented using the HuggingFace Transformers 43 library's mooel..eneeaaeee () function with
KV cache. All experiments are run 80G A100 GPUs, with the GPU number varying based on N.
httpp:: haapss://hugiiggaaeeee co wwwwwk/kkkk///////ooooooc. Skywork o1 Open- PRM- Qwen- 2-B-7B
We use PRMs that are strong and generic as possible, selecting one of the most advanced for our main experiments, and Appendix D.2 includes results with other PRMs to validate the generality of our conclusions.

---

MATH
TheoremQA
34
Si-BoN [aam]
SEa Kouny
70
FallBuN RM
Pall-BeN R34
32
Ful-BcN w/ RM
Ful-BuN my RM
24
88
98
7V
10x
12x
14
314
3k
76
9x
10
1Ix
13x
GPQA
MMLU
75
14
33
76
Ful-BaN
Ball-BN RM
75
Ful BuN w/ SM
Goody
Gready
74
12s
Computational Cost (The
Greedy)
(a) Objective Tasks (mainly on reasoning and knowledge domains).
CNNDM (Ronge-L)
CNNDM (Human Scoring)
AlpacaFarm (Win Rate)
oog
4.00
Ascurecy
12
4.25
24
36
46
5x
86
76
N9
05
10x
IIs
17s
TV
2x 4s 3x 6y 2x Ne 10s:
11x 12x
7X
28
36
66
55
68
74
Rx
9x
10s
11x
12x
Computatiiona Cost (The Factor Relative to Greedy)
(b) Subjective Tasks (open-ended generation and preference alignment).
Figure 5: Computational cost and accuracy on different objective and subjective task datasets with various domains using the Qwen2.5-7B: IIsstuut model. Each point on the curves corresponds to specific sampling size from to 80, with values labeled within the circle. We connect the three results N 20 with gray dashed lines to show their relative performance without considering cost.
Evaluation. We mainly evaluate the balance between computational cost and performance. The computational cost lies two dimensions in Section 4 memory and time. We use the cost of greedy decoding the baseline, and the cost of each paradigm is calculated as follows:
Memory Cost Mens: Let Mbm: Maeak. and Mn denote the memory usage of the base LLM weights, the peak memory overhead during inference, and the reward model weights, respectively, for each paradigm. Let Marreally denote the peak memory during greedy decoding:
(Mom + Mpeak) / (Mom + -Meree ): (
ST-BoN & Full-BoN w/o RM,
Meost ==
(8)
(Mbm + Mpeak Mm)) ) (Mon Mrgreddyy Full-BoN w/ RM.
Time Cost Teoat Teost is denoted as the ratio of the wall-clock time of each paradigm that of greedy decoding. For Full-BoN w/ RM, we consider that both the generator and the verifier are persistently stored in GPU memory, and thus ignore the loading time of the RM.
The overall computational cost A is given by: Maust
Acost
Teost-
Regarding performance, Accuracy is used as metric in reasoning scenarios. To reduce random error, we follow [16) to report the average results over four evaluation runs. open-ended scenarios, Rouge-L 29 and Human Scoring applied for the summarization, and Win Rate (WR) [59 judged by 4o-TbboTuurb [ is used for the instruction- follitiioonooooiingn..mm Metric details are shown in Appendix
5.2 Results Objective Task
Figure presents the results of Qwen2.5-7BB ---7-BInIstttt on four objective tasks. Due to space constraints, results for other models are provided Appendix D. First, we analyze the result soundness. Under the same sampling size N (we use N 20 as example and connect the three paradigms with gray dashed lines), when cost is ignored, compared Full-BoN w/o RM, ST-BoN shows no
significant performance drop, indicating that the theory proposed in Section 3.1
early consistency
7

---

1.0-
1.0
0:0
Consetency
0.8-
0.8
672
0.8-
066
0.6
0.6
0.4
0.6
0.4
04
$250.0
822
02
0.2
0.0
MATH
MMLU
0.0
MA711
MMLUU
0.0
MATH
MMLU
MMLU
Renien Solentire
Oit Water (RandaechS--e5)
Colbwoobbalee Wandew
Semeriic my Netler Waddew
w/ aengge/ Mallo Moxdew
Figure 6: The early-final consistencies between early self-estimation and final correctness in different domains and sub-datasets with the Qwen2.5-7B-Instre model. D; represents the subset containing all cases from dataset D that produces i correct answers out of N samplings.
foreshadows final consistency is reasonable and holds in practice. In addition, Full-BoN w/ RM outperforms the others in three datasets more related to reasoning (except MMLU), which aligns with the test-time scaling goal of "increasing inference cost improve performance" [40].
Next, we analyze our core advantage: cost-performance trade-offs. Due self-truncation, ST-BoN does not fully utilize all N samples, leading to lower performance than Full-BoN initially. However, it greatly reduces the cost, allowing N to be scaled up within the saved cost. As N increases, ST-BoN eventually matches Full-BoN's performance while using lower costs. For example, when Full-BoN uses 20, ST-BoN reaches the same accuracy at N = 20 30 with 70-75% lower costs (vs. Full-BoN w/o RM). and at N 40 80 with 75-80% lower costs (vs. Full-BoN w/ RM), showing that ST-BoN can achieve the same performance as Full-BoN with significantly lower costs.
On the other hand, under the same computational cost, ST-BoN can scale to a much larger N than Full-BoN. For example, when Full-BoN w/o RM uses N 10 and Full-BoN w/ RM uses = ST-BoN can reach nearly N 80. This gives it access to much more samples, leading to a 3-4 point higher accuracy across all datasets. These results show that under the same computational cost, ST-BoN can achieve better performance than Full-BoN.
Finally, also observe the domain generalization of ST-BoN. Full-BoN w/o RM performs no- ticeably worse on MMLU, knowledge dataset, compared to the other three reasoning datasets. In contrast, ST-BoN maintains its cost-performance trade-off advantage. This suggests that Full-BoN w/o RM relies on strong and general RM to be effective, while ST-BoN avoids this potential cost. By not requiring domain-specifie priors, ST-BoN remains effective across diverse domains.
5.3 Results II: Subjective Tasks
To evaluate the domain generalization of ST-BoN, we further test on subjective tasks. The results for Qwen2.5-7B-Inns -n-srrrtctt are shown in Figure 5b with results of other models shown in Appendix D.I First, Full-BoN w/o RM performs poorly, indicating that semantic consistency cannot identify optimal samples. Second, Full-BoN w/ RM underperforms summarization tasks due domain mismatch: the reward model is typically trained on open-domain QA preferences, which rarely cover summarization, leading to OOD issues. Finally, w/ RM outperforms ST-BoN at the same sampling size N, ST-BoN can reach its performance by increasing N, while reducing computational cost by approximately 50%. This highlights ST-BoN's cost-performance trade-off on subjective tasks and further demonstrates its domain robustness.
iesructioo-followinn tasks, although Full-BoN
In-depth Analysis
6.1 Component Ablation: Why ST-BoN Works?
Our method is motivated by the idea that early consistency may foreshadow final consistency, then use CoE self-estimate early consistency, and propose buffer window to mitigate noise. In this part, we will ablate the two components: CoE measure and buffer window, to evaluate their necessity.
Baseline. The standard ST-BoN involves "w/ CoE & w/ Buffer Window", First, we remove the buffer window, allowing the LLM to perform only one self-estimation at the earliest estimation time c. Next, further assess the necessity of using hidden states of LLMs (i.e., CoB) for early consistency computation, we compare two unsupervised output measure: semantic and string. Specifically, in we replace D(Yi... Yf.e) with [[[hf:cr - [h and 1 /Rouge-LLYY'... Yi,e). respectively. Evaluation. We assess the Early-Fine Consistency between early self-estimation and final cor- rectness. Specifically, assume each case in dataset D is sampled N times. The datasct is divided
5
8

---

MATH
MMLU
CNNDM
Alpact-Farm
Compurational Cooc
40
Varforance
3
3,62
D2 3P
-xx
2x
8
26
1x60
10
22
32
86
5c 30
30
60
5c
1c
32
32
92
5c
12
22
36
9c
Window Length
Figure 7: Hyperparameter ablation with varying window length т. The blue line shows performance
changes as increases, while the pink shading represents changes in computational cost. T + 00 indicates window extended until any sampling reaches [EOS].
p=0.5, T=0.7
k=2n, T=0.7
k= p=1..8
Accuracy
79
79
79
78
2-
78
78
T-1
77
1-:0
77
0:35
77-
T-10
2-13
-0- T=07
76
76
0.03
76
Tw02
1.4x
26x
1.4x
1.4x
1.8x
Computational Cost (The Factor Relative to Greedy)
Figure 8: Sampling strategy ablation with various top-k, top-p, and temperature T in MMLU.
into N + disjoint subsets {D. where Dj contains cases that yield correct answers out of N samples. For each D;, we compute the proportion of cases where the LLM's self-estimated best sampling produces a correct final answer, representing carly-final consistency. Naturally, as i decreases, the estimation becomes more challenging, and the random consistency expectation is i/N.
Results. We evaluate on the MATH and MMLU datasets due to their diverse domains and difficulty levels. With N = 5, we exclude subsets Du and D5, as they represent cases that are entirely correct or incorrect. Figure 6 presents the results using the Qwen2.5-7BB Insstruct model (Results for other models are in Appendix D.3). we find that: (i) "w/ buffer window" consistently outperforms that "w/o buffer window"" demonstrating its effectiveness. (ii) For all subsets, "w/ CoE" significantly contrast, outperforms random estimation, with standard ST-BoN achieving excellent consistency. "w/ semantic" and "w/ string" hover around the random baseline, highlighting the critical role of CoE in early consistency measure. Thus, each module in the current ST-BoN plays an essential role.
In fact, we have also discovered that using CoE consistency under full generation, yields better results than using ceff-consistency with majority voting. This provides further evidence that CoB's effectiveness in measuring early consistency is not random. The results are shown in Appendix D.4
6.2 Window Length Ablation
In ST-BoN, the buffer window length is the only hyperparameter. We study its impact by varying in different datasets with Q with a fixed sampling size N = 10.
Figure 7 shows that when < c, the performance gain relative to cost increases faster. However, beyond c, this gain slows significantly, making c the optimal choice for balancing performance and cost in our experiments. Furthermore, if ignoring costs, performance continues to show fluctuating improvements as T increases, suggesting the potential of task-specific 7 tuning. For example, on the MATH dataset, performance improves more steadily with larger values, probably due its longer outputs, where early estimates are less informative, and need a larger buffer window to capture more information. Adaptive based on task complexity can be promising direction for future work.
6.3 Sampling Strategy Robustness
We also evaluate the robustness of ST-BoN across different sampling strategies by varying top-k, top-p, and temperature T settings, and analyze their impact on performance and computational cost.
Figure [ shows the results on the MMLU dataset using the Qwen2.5-7BB 7-B-..5---nnntrrcc model. We observe that variations in k and have minimal impact performance and cost, while changes in T affect both more significantly. Unlike and p. which only shift the truncation point of the probability distri- bution without affecting token relativity, T alters token probabilities, making the distribution more uniform and increasing sampling randomness as T increases. This results in an earlier occurrence of the earliest estimation time thereby reducing computational cost. However, an earlier estimation time e might capture less effective information, potentially as a side effect of improved efficiency. We recommend using moderate T value, such as 0.7, to strike balance between performance and cost.

---

6.4 Case Study
We also conduct case studies for further comparison. We find that ST-BoN can better address am- biguous scenarios, such as when sampling answers are all inconsistent or majority voting encounters high randomness, compensating for these limitations. Details are available in Appendix |
7 Related Work and Comparisons
7.1 Test-Time Scaling
BoN aims to achieve test-time scaling 40 The idea is to increase the compute during inference while setting cost aside, to push the performance ceiling. Existing approaches fall into two broad lines [2): Sequential scaling lengthens a single reasoning path during generation and seeks deeper deliberation, reflection, insight [16]. Parallel scaling, which captures the spirit of BoN, increases the number of reasoning paths generated at once and seeks greater diversity in reasoning [47] [47]6]. Other methods, such ToT [57] GoT [5), and Monte Carlo Tree Search [4], share the same objective but expand from a reasoning-structure perspective, taking different research tracks than BoN.
7.2 Best-of-N Sampling
BoN sampling was originally proposed for inference-time preference alignment model to select the best sample and further fine-tune the base model 42 effectiveness has been theoretically supported by several studies [12 31 [6). BoN has also seen broad adoption in reasoning tasks [40], notably in the self-consistency paradigm [47], which uses majority voting
3 using reward
to select answers unsupervisedly. More recently, process reward models [28 46 have enabled
direct scoring within BoN for reasoning. However, reward models raise computational costs and are vulnerable to overoptimization, and rely heavily their model capabilities [1]
7.3 Efficient Best-of-N
The core optimization idea of ST-BoN can be summarized as "depth pruning". which eliminates some suboptimal samples in early decoding to reduce memory usage. SBoN [4]] also considers removing local segments; CARDS (25) and TreeBoN [36] continue to branch after discarding them. However, these methods still rely
reward model for scoring, so the additional inference latency and memory usage remain high. More importantly, they have been validated only under the preference alignment, but in reasoning domains, reward models may be difficult to score randomly segmented process fragments. Appendix B offers a detailed comparison, highlighting the greater efficiency of ST-BoN and its broader applicability in different domains.
Another type of optimization idea can be summarized as "breadth pruning", aiming to adaptively reduce the actual sampling number under a fixed budget N. For example, ASC (2) stops sampling early based on answer frequency, and ESC [27] dynamically adjusts the required sampling sizes from an entropy perspective. Although these methods lower token output and reduce memory usage, they rely on serial sampling with adaptive stopping, which compromises the parallelism benefits of BoN and limits efficiency modern GPUs. As a result, they are suboptimal in terms of inference latency.
Certainly, another research line studies distilling the BoN policy during training replace multiple samplings with a single one, reducing inference-ting time latency. Methods like BoNBoN [15], BOND 39 and vBoN 3 focus on this line, shifting the computational cost from the inference phase the training phase. Overall, this line explores more about the potential of learning the BoN distribution at training time, making orthogonal contributions compared to direct inference-time optimization.
Conclusion
We propose ST-BoN decoding, which enables LLMs to self-estimate the most promising sampling without fully generating N samples or using reward models. ST-BoN significantly reduces GPU memory overhead and inference latency while demonstrating better cost-performance trade-offs than Full-BoN in both objective and subjective tasks from reasoning to preference alignment.
10

---

Limitations
(1) Model Transparency: Since accessing the hidden states of the LLM is required, ST-BoN does not apply to closed-source models such as OpenAl's GPT-4 series [ However, with the rapid development of open-source LLMs [16], we believe that research on white-box methods is crucial, as it can provide stronger interpretability and help us better explore the internal mechanisms of LLMs.
As mentioned in Section 6.2 there is still room for improvement in ST-BoN's ability to adaptively adjust the window length 7 based on task complexity. Longer responses may require longer windows to capture more information. To better balance the cost, we can observe the average generation length of the task during greedy decoding and adjust the window size accordingly. This ensures that performance is maximized within an acceptable cost range.
(2) Length Adaptivity:
Societal Impact
(1) Trustworthiness: ST-BoN relies on internal consistency signals to guide decoding, moving the selection process into the model's latent space. Prior work [47 has shown strong correlation between consistency and correctness. Building on this, our theoretical analysis (Section 3.1) and empirical findings (Section 6. further support the reliability of latent-space decision-making. Together, these two parts provide solid foundation for the model's s trustworthiness under our method.
(2) Safety: ST-BoN introduces no additional safety risks. The sampling follows the model's native distribution without adversarial perturbation, and the consistency-biue selection mechanism operates without external reward models, thus avoiding interference from out-of- distribution signals.
ST-BoN inherits the base model's biases, it does not perform explicit debiasing. However, by leveraging multiple parallel samplings, it can identify and truncate transient biases that arise sporadically during generation. These biased trajectories may be excluded as outliers through the consistency-badt strategy, thereby potentially mitigating their impact.
(3) Biases:
Acknowledgement
This paper was supported by the General Program of the National Natural Science Foundation of China (62176153). This paper was also supported by the National Natural Science Foundation of China (6240618) 188) and the Natural Science Foundation of Shanghai (24ZR 1440300). Additionally, this work was partially conducted during Yiming's internship at the Tongyi Lab, Alibaba Group. It was supported by the Alibaba Research Intern Program.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
[2] Pranjal Aggarwal, Aman Madaan, Yiming Yang, et al. Let's sample step by step: Adaptive- consistency for efficient reasoning and coding with 1lms. In Proceedings the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12375-12396, 2023.
[3] Afra Amini, Tim Vieira, Elliott Ash, and Ryan Cotterell. Variational best-of-n alignment. arXiv preprint 0606v72407.06057 2024.
[4] Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D' Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy, arXiv preprint wxXiv:2401.08880 2024.
[5] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph

---

of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 38. pages 17682-17690, 2024.
[6] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprini urXiv:2407.21788 2024.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[8] Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang. and Tony Xia. Theoremqa: theorem-driven question answering dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7889-7901, 2023.
[9] Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of ol-like llms. arXiv preprint arXiv:24/2.21 187, 2024.
[10] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:240..21783, 2024.
[11] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
[12] Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' Amour, DJ Dvi- jotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312. 092444, 2023.
[13] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 889-898, 2018.
[14] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023.
[15] Lin Gui, Cristina Gârbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetness of best-of-n sampling. arXiv preprint 24Xi::2406000322.. 2024.
[16] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[17] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
[18] Dan Hendryeks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[19] Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint 1503:00050.255 2015.
[20] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint 09755, 2019.
12

---

[21] Siddhartha Jain, Xiaofei Ma, Anoop Deoras, and Bing Xiang. Lightweight reranking for language model generations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 6960-6984, 2024.
[22] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulmier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[23] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 09122199-2223., 2022.
[24] Wojeiech Kryšciáski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP)) pages 540-551, 2019.
[25] Bolian Li, Yifan Wang. Anamika Lochab, Ananth Grama, and Ruqi Zhang. Cascade reward sampling for efficient decoding-time alignment. arXiv preprint arXiv:2406. 16306, 2024.
[26] Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities. arXiv preprint TrXiv:2403.04706, 2024.
[27] Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Xinglin Wang, Bin Sun, Heda Wang, and Kan Li. Escape sky-high cost: Early-stopping self-consistency for multi-step reasoning. arXiv preprint arXiv:2401. /0480, 2024.
[28] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint a3Xiv:2305.22000, 2023.
[29] Chin- Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[30] Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral. Vaiva Imbrasaite, and Vincent Y Zhao. Dr.icl: Demonsttraiion-retiieee in-context learning. Data Intelligence, 644)9909-922, 2024.
[31] Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. arXiv preprint arXiv:2310. 17022, 2023.
[32] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint wtXiv22501.19393 2025.
[33] Relichiro Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2 09332, 2021.
[34] Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gülçehre, and Bing Xiang. Abstrac- tive text summarization using ssquence-to-sequence mns and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290, 2016.
[35] Skywork ol Team. Skywork-ol open series. httpp://huuge httpa////hhgggggfac.c.eee////// ber 2024.
Novem-
[36] Jiahao Qiu, Yifu Lu, Yifan Zeng, Jiacheng Guo, Jiayi Geng, Huazheng Wang, Kaixuan Huang, Yue Wu, and Mengdi Wang. Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling. arXiv preprint arXiv:2410.1660 16033, 2024.
[37] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311. 2023.
13

---

[38] Jie Ren, Jiaming Luo. Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J Liu. Out-of-distribution detection and selective generation for conditional language models. In The Eleventh International Conference on Learning Representations, 2022.
[39] Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexan- dre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. Bond: Aligning Ilms with best-of-n distillation, arXiv preprint arXiv:2407. /4622, 2024.
[40] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Il test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408. 03314, 2024.
[41] Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang. Jiahao Qiu, Ming Yin, Mengdi Wang. Peter Bartlett, and Andrea Zanette. Fast best-of-n decoding via speculative rejection. arXiv preprint arXiv:2410.202990 2024.
[42] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint M.Xiv:2307.09288 2023.
[43] Vaswani.
Attention all you need. Advances Neural Information Processing Systems,
2017.
[44] Ziyu Wan, Xidong Feng, Muning Wen, Stephen Mareus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training. In Forty-first International Conference on Machine Learning, 2024.
[45] Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and ofxture-of--xpere arXiv preprint arXiv:2406. 1285,, 2024.
[46] Peiyi Wang. Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce lims step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 9426-9439, 2024.
[47] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 11177 2022.
[48] Yiming Wang, Pei Zhang, Jialong Tang, Haoran Wei, Baosong Yang, Rui Wang, Chenshu Sun, Feitong Sun, Jiran Zhang, Junxuan Wu, et al. Polymath: Evaluating mathematical reasoning in multilingual contexts. arXiv preprini ar arXiv:2504. / 18428, 2025.
[49] Yiming Wang, Pei Zhang, Baosong Yang. Derek Wong, and Rui Wang. Latent space chain-of- embedding enables output-free IIm self-evaluation. arXiv preprini urXiv:2410. 13640, 2024.
[50] Yiming Wang, Pei Zhang, Baosong Yang, Derek F. Wong, Zhuosheng Zhang, and Rui Wang. Embedding trajectory for out-of-distribution detection in mathematical reasoning. In The Thirtt-eighth Annual Conference on Neurul Information Processing Systems, 2024.
[51] Yiming Wang, Zhuosheng Zhang, and Rui Wang. Element-aware summarization with large language models: Expert-aligned evaluation and chain-of-thought method. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages (640-8665, 2023.
[52] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thoughh prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2453 2022.
[53] Wen Wen, Yongbin Liu, Qiang Lin, and Chunping Ouyang. Few-shot named entity recognition with joint token and sentence awareness. Data Intelligence, 5(3):767-785, 2023.
14

---

[54] Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang, An implementation of generative prm. 2024.
[55] An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint
arXiv:2412.15115, 2024.
[56] Joy Qiping Yang, Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, and Ahmad Beirami. Asymptotics of language model alignment. arXiv preprint arXiv:2404.01730, 2024.
[57] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
[58] Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.074 2025.
[59] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 18:46595-466222 2023.
15

---

Appendix
A Proof of Theorem
I
Proof. We first make the two key structural assumptions:
Local Lipschitz continuity of the model: The model's next-token distributions satisfy, for all
partial sequence pairs (Yis+ Via).
P Vi.)) P.+1( RiOITT I L. Vi..)
(10)
for some L 2 O, where || lla denotes the Total Variation Distance. It quantifies the "smooth" behavior that tiny changes to the sequence produce only tiny shifts the next-token probabilities. Transformers are inherently Lipschitz-contimuous: Each token embedding passes through layers of self-attention and geed-forward networks whose composition preserves this smoothness, so similar sequences yield similar hidden states. Moreover, Softmax and LayerNorm further suppress perturbations, preventing small input changes from causing large changes in the output.
of the distance: The distance measure D satisfies: for any partial sequences Y yi, and any next tokens 1/41 Y
D(Yi, Y441, Vi 1241) < D(Y (i)) + M 1[14 # 1/
(11)
for some constant M > 0. It ensures the distance measure remains controlled, preventing abrupt shifts from single token change. Fundamentally, this reflects the smoothness characteristic of local Lipschitz continuity.
Next, we start our proof. Consider the pair (Y V(.))- By the maximal coupling of Pt+1(- (it)
and Pr+1( (- Yia). The probability they draw different tokens at step & is
Pr[1++ * Yt di Pe+1 vit) Pet1l 1t I Ldj
(12)
If they differ, the bounded-increment assumption implies
d'+1 dx M.
(13)
Hence
E
di Eld; di + d])
S di Pr(Y # Yi dj) M
(14)
< d + (L (L) M = (1 LM) d.
Let = 1 + LM. Summing the above over 2, ...,. N gives
.N
N
B [.]
Σ
Eld
[d] S r r d
S.
(15)
2
2m2
Starting from i and iterating the one-step bound,
E[S++1] < [ S.,
E[S [E[S i] S
(16)
E[Sr] < T-
For any e 0, Markov's inequality yields
E[ST [t]]
TT St
Pr[ST 2 c S.] <
(17)
Rearranging gives the stated result in Theorem
-
Pr[Sr S E St] 2
S
(18)
16

---

B Related Work Comparison
Some existing work also considers the reduction of the full generation improve the efficiency of BoN. The core idea behind all three methods is to use the reward model to score partial sequences certain time steps during the sampling process. SBoN sets a threshold and discards all samples with scores above this threshold, then iteratively generates and selects the remaining candidates until only one remains. TreeBoN and CARDS retain only one candidate at each selection and then repeatedly branch out N new paths, performing selection at each branching stage. These methods have fundamental differences from ST-BoN. Table I
presents a comparison between different dimensions.
Table 1: Detailed comparison with other efficient BoN methods aiming to reducing full generation.
Method
Domain Prior
Froe
Reward Models
Momory Bottleneck (excluding KV cache)
Time Bottleneck
Applicable Domain
SBoN 8 CARDS Too
Reward Model Weight
Run Reward Models
Preference Aligument
X
All Ofjective Tacks
ST-BoN (Ours)
Victor Operations
(c.g. Reasoning Knewledge) Subjective Tasks
(e.g. Generation & Preference Alignment)
From the perspective of computational cost, the memory overhead and inference time introduced by the reward model are non-negligible. Additionally, during the autoregressive generation process, the reward model repeatedly scores incomplete segments, which blocks parallel sampling and significantly reduces actual GPU utilization. Therefore, in practical production settings, from wall-clock-ting perspective, these approaches still face efficiency bottlenecks.
On the other hand, the effectiveness of SBoN and CARDS relies on prior assumption: the reward model's score for sampling prefix correlates with the score for the final complete text. However, this assumption has only been validated in preference alignment domains, raising concerns about domain generalizability. When applied reasoning domains, process reward models will likely struggle to evaluate arbitrarily segmented text, as they tend to focus on scoring complete process segments. Furthermore, [36] has pointed out that the prior assumption underlying CARDS breaks down when the generation length exceeds 128 tokens. In contrast, ST-BoN is highly lightweight: it does not incur overhead or blocking from a reward model and does not depend on domain-specific priors
MATH
AlpacaFarm Rate)
74
Accuracy
600
375
23
22
Sick'
CARRS
CARES
Tradfo
Trutie
8000
Computational Cost (The Factor Relative to Greedy)
Figure 9: Computational cost and accuracy on different objective and subjective task datasets with various domains using the 1e-n2.55-B-Instruue
model. Each point on the curves corresponds to specific sampling size N from 10 to 80, with values labeled within the circle. We compare our ST-BoN with three other efficient BoN methods.
We compare the cost-performance trade-offs of ST-BoN and the three methods on both an objective and subjective task, illustrated in Figure 9 (i) The MATH benchmark evaluates the reasoning ability. All three other methods perform poorly on this task, probably because process reward models require complete procedural segments to assign reliable scores. They struggle to make sound decisions when evaluating arbitrarily segmented text fragments. (ii) The AlpacaFarm benchmark assesses preference alignment, the primary domain where the three methods have demonstrated effectiveness. In this domain. ST- BT-BoN performs comparably or even better than the others. This may be because the inductive biases assumed by some methods do not generalize well across different models. These results collectively demonstrate the strong domain generalizability and practical applicability of ST-BoN.
17

---

C
Evaluation Setup
Reasoning Scenarios: We Accuracy as the evaluation metric. Following +7
extract answers from LLM responses using the exact match method and compare them with the ground truth.
Open-end Scenarios:
Summarization: We use both automatic and human evaluation. For the automatic metric, we adopt Rouge-L [29]. For human evaluation, volunteers score LLM-generated summaries across four dimensions 24 5:: (i) Fluency: Free of spelling, grammatical, or syntactic errors; (ii) Coherence: Events should be logically connected, with smooth linguistic transitions; (iii) Consistency: No hallucinated facts; all information must align with the source document; (iv) Relevance: Emphasizes key information while minimizing non-core facts and redundant details. Each dimension is rated scale of O to 5, and the final score is the average of the four dimension scores. We invited three graduate students to individually score 50 samples from the dataset following [5] The final reported score is the average the three annotations. Each annotator spent approximately 16 hours on average complete the task.
instructtio-folloving We use the BoN alignment metric, Win Rate (WR) Each paradigm's is responses were paired with those generated via greedy decoding and assessed by an external L.L,, GPT-4-Turbb [. The judge model selects the better response in each pair, and WR is computed the percentage of comparisons where the paradigm's 'ss response is preferred. Greedy decoding serves as a baseline with a WR of 50%, representing random performance.
D Supplementary Experimental Results
D.1 Main Results of All LLMS
In this part, we present the main experimental results for all remaining LLMs,, including: Qwen2.5- 72B-Instruct (Figure 10), -Jama3-8B-Insteun (Figure(TT) and Mistral-7BB-v033-nnttte (Figure 12}. As highlighted in 46 significant performance drop occurs when the verifier's parameter scale smaller than that of the generator. Therefore, for the of equal scale Owwen2.5-Math-PRMM222
model, we adopt a PRM (58 and only report results objective tasks.
Under these models, all conclusions regarding the post-performance trade-off drawn in Sections 5.2 and 5.3 still hold, and model size increases, ST-BoN remains effective, demonstrating its robustness to parameter scaling.
18

---

MATH
Theorem
2112%
83
94
000000
SE-BoN 10an
(lun) Ful-BuN w/o RM
Fall-BoN w/dc RM
FallBBN RM
FullBuN w/ RM Greody
Groody
75 MMLU
GPQA
00
85.0
STAN Pul-BoN my IM
#70
64
Pull-teN
Coondy
Groedy
10x
12x
56
1ix 13x
Computational Cost (The Factor Relative Greedy)
Figure 10: Computational cost and accuracy on different objective task datasets with various domains using the Swenn25-72B-Instruce model. Each point on the curves corresponds to a specific sampling size N from 3 to 80, with values labeled within the circle. We connect the three results at 20 with gray dashed lines to show their relative performance without considering cost.
Theorem Q
32
SoN [Laan]
St-ted
FullBaN
Goandy
Greods
24
2x
46
66
8x
16x
Fix
12s
15.
36
5x
75
10s
lis
12s
13x
GPQA
MMLU
62
32
SI BoN [1amm) Bull BAN w/t RM
Sti Kluny
Tulldtik RM
PullitiNN IM
30
Ureddy
24
36
46
56
5x
10s
Is 12 Es
Is
36
46
56
84
76
5x
10s
lls
13x
Computational Cost (The Factor Relative to Greedy)
(a) Objective Tasks (mainly reasoning and knowledge domains).
CNNDM (Rouge-L)
CNNDM (Human Scoring)
AlpacaFarm (Wia Rate)
Ascuncyy
4.25
52.5
Juli Wal-eeids.e
4.00
5001
36
46
5x
7x
8K
9k
10c
11x
12x
1x
2x
3x 5x 7x St 9x F0x 11x 12x
2%
34
44
54
84
9%
10k
11x
12x
Computationnal Cost (The Factor Relatve to Greedyy
(b) Subjective Tasks copen-ended generation and preference alignment).
Figure 11: Computational cost and accuracy different objective and subjective task datasets with various domains using the LJama-3-8B-Instae specific sampling size N from 3 to 80, with values labeled within the eircle. We connect the three results at N 20 with gray dashed lines to show their relative performance without considering cost.
model. Fach point on the curves corresponds to a
19

---

MATH
Theorem QA
21
27
21
SI-BoN [aam
SPaeN Kunn
Fall-BeN
Ful-BcN w/o JM
FollB.N
Gready
26
36
46
5%
7%
5x
10h
Li
12x
13
16
36
46
5x
7 MMLU
5x
10s
lIs
13x
GPQA
12
24
22
[Oan)
Rul-BWN w/o RM
Fulldi-N
Dell w/ RM
DulldioN wy BM
54
Greedy
12x
Computational Cost (The Factor Relative to Greedy)
(a) Objective Tasks (mainly on reasoning and knowledge domains).
CNNDM (Rouge-L)
CNNDM (Human Scoring)
AlpacaFarm (Win Rate)
122.8
6000
Ascursty
4.15
375
4113
525
Canile
36
56
76
84
10x
12x
14 2x
5x 6% 7x
10x 11x 12x
24
34
54
04
84
10x
12
Computational Cost (The Factor Reiatve to Greedy)
(b) Subjective Tasks (open-ended generation and preference alignment).
Figure 12: Computational cost and accuracy on different objective and subjective task datasets with various domains using the Distral-7B-Instruet--oo model. Each point on the curves corresponds specific sampling size N from 3 to 80, with values labeled within the circle. We connect the three results N 20 with gray dashed lines to show their relative performance without considering cost.
20

---

D.2 Reward Model Ablation
To ensure that the choice of reward models does not impact our key conclusions, we extend the baselines of the Full-Bol w/ RM paradigm beyond the main experiment in Figure 13 Specifically, we introduce two additional advanced reward models for comparisons: Qwen2.5-Mand -att555-aat-PPMM and RI RL-Fllo-PPR M-MM--ttrall-l----------------- not affect the conclusions regarding the cost-performance trade-off advantages of ST-BoN in the main experiments.
[58] (54). As shown in Figure 13] the choice of reward models does
MATH
Theorem
00-0-0
Full Bittuuibe
70
Eul doner An Kovens subino
Gamb
4x
10
12s
11s
13x
CPOA
MMLU
34
77
32-
DTiak pood
76
Fut antter ald Mlyner
nut auteer 82 mo Silwes Goml
75
74
Computational Cost (The Factor Relative to Greedy)
Figure 13: (Reward Model Ablation) Computational cost and accuracy on different objective and model. Each point
subjective task datasets with various domains using the Swee2.5-7B-Instruue the curves corresponds to a specific sampling size from 3 to 80, with values labeled within the circle. For the Full-BoN w/ RM paradigm, we use three reward models for comparisons.
D.3 Component Ablation of All LLMs
We present the additional early-find consistency results on Llama3-8B Brrtua--BB-nsstiee IB-Instruc (Figure 14 and Mistral-7B IerB--InstuutttOO (Figure [5] models, demonstrating strong positive correlation between
ST-BoN's early self-estimation and the final answer correctness, as concluded in Section 6.1 This indicates that consistency conclusion can be generalized across various series of L.LMs.
D2
D4
Canz Canzan
08
0.8
811
38
K11
0.4
BA4
MATH
MMLLU
MATT
MILEUI
MATH
MKLLUI
MATII
MILLU
Ac Ballo
Figure The early-final consistencies between early self-estimation and final correctness in different domains and sub-datasets with the 1amma--B-IInstuus model. D; represents the subset containing all cases from dataset D that produces correct answers out of N samplings.
Cuniatena
A17
6:18
0.-
0.5
1.56
0.2
with fudlo
Pallo Manii
Figure 15: The early-final consistencies between early self-estimation and final correctness in different domains and sub-datasets with the M subset containing all cases from dataset D that produces i correct answers out of N samplings.
model. D; represents the
21

---

D.4 Full-BoN with Chain-of-Embeddling
In our experiments, we find that under the same sampling size N, although the absolute performance of ST-BoN was weaker than that of Full-BoN w/o RM, the gap is not substantial. Since our self- estimation of early consistency relies the CoE measure, this raises further question regarding the effectiveness of CoE: Would using CoE in Full-BoN w/o RM outperform majority voting?
Table The performance comparisons of majority voting and chain sietin-ff--mbedding under the Full- BoN w/o RM paradigm. We use the Qwen2.5-7B-Instrue model to evaluate two objective tasks.
N=5 =
N 25
N
MATH
Majority Voting
+
70.0
71.2
71.8
72.4
72.7
73.2
73.9
74,2
74.4
74.5
Chain-of- Enbiediig
69.6
72.3
72.7
73.1
73.8
74.9
MMLU
Majority Veting
76.3
76.4
77.3
77.8
78.3
78.5
78.6
78.6
79.2
Chain-of- -eii-dding
76.2
76.5
77.5
77.8
78.2
78.5
78.4
78.9
79.2
79.5
Table 2 presents the performances of the Qwen2.5-7B-Insteee Instruct model on MATH and MMLU under the Full-BoN w/o RM paradigm, using majority voting [47] and CoE 49 respectively. Since the computational cost is nearly identical, we do not compare them in this regard. We observe that in most cases, using CoE outperforms majority voting. We think that majority voting focuses solely on the final answer, representing outcome supervision. However, LLMs might reach correct answer through wrong reasoning, so the outcome doesn't always reflect the process information. Conversely, CoE utilizes each token to provide process supervision, which introduces more information than individual results. This indicates that the effectiveness of ST-BoN largely depends on the fact that CoE is also strong metric for selecting optimal samples under the Pull-BoN paradigm.
E
Case Study
present three cases follows:
GPQA dataset in Swee2.5-7B-Innsrrus model.
We set the sampling size N 5, with all sampling results and selected answers with ST-BoN and Full-BoN paradigms shown in Tables 3
The question as follows:
Instruction: Answer the following multiple choice question. The last line your response should be of the following format: 'Answer: SLETTER' (without quotes) where LETTER is one of ABCD. Think step by step before answering.
Question: would like to dissolve (at C5cC) 0.1g Fe(OH)3 100 cm3 total volume. What is the minimum volume (cm3) of 0.1 M monobasic strong acid that is needed to prepare the solution and what is the pH of the resulting solution?
Choices:
(A) pH 3.16; 32.14 14 cm3
(B) pH 2.69; 30.09 cm3
(C) pH 2.04; 28.05 em3
(D) pH 4.94; 20.40 cm3
TheoremQA dataset in 1lama-888-Insrued model.
We set sampling size N 5, with all sampling results and selected answers with ST-BoN and Full-BoN paradigms shown in Tables 7
The question is as follows:
22

---

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
Please read math problem, and then think step by step derive the answer. The answer is decided by Answer Type. If the Answer type in [bool], the answer needs to be True or False. Else if the Answer type in [integer, float] The answer needs to in numerical form. Else if the Answer type in [list of integer, list of float] the answer needs to be list of number like [2, 3, 4]. Else the Answer type in [option], the answer needs be an option like (a), (b), (c), (d). You need to output the answer in your final sentence like 'Therefore, the answer is
### Instruction:
### Question: In triangle ACD, B is located on the side AC, and E is located on the side AD. AB 3, AC 5, CD 3.5, ED 3, and EB DC, what is the length of AD?
### Answer type: float
MATH dataset in Nistral-7B-IIntrueetve model.
We set the sampling size N 5, with all sampling results and selected answers with ST-BoN and Full-BoN paradigms shown in Tables 9 TO) The question is as follows:
Question: What the only integer value of for which n+1 is a positive prime number?
13-n
Instruction: Please reason step by step, and put your final answer within \boxed().
From these cases, we have two key findings:
More is Not Always Better: Early self-estimation allows LLMs to recognize that the final answer does not hold the majority vote but is still correct. This is demonstrated in the first two cases.
Majority Voting Fails with Complete Inconsistent Answers: Our self-estimation approach uses internal consistency to assign continuous scores, ensuring the best sample is always chosen. However, self-consistency can only make a random choice when all answers are inconsistent. This is shown in the last case.
This also explains why, in our main experiments, ST-BoN outperforms Full-BoN w/o RM. When the model faces ambiguous choices (e.g., all samples inconsistent), majority voting introduces significant randomness. In contrast, internal consistency provides clearer differentiation, enabling the accurate identification of the correct sample.
23

---

Table 3: Case [():: GPQA dataset in Qwen2.5-7B-Instee model. LLM responses, red content indicates that it is within the buffer window, while gray content indicates that it has been truncated and does not need to be generated during actual decoding.
Sample ID
LLM Response
To solve this problem, need follow these steps:
1. Determine the amount of Fe(OH)3 needed to be dissolved. Calculate the volume of monobasic strong acid required. 3. Determine the pH of the resulting solution. ### Step 1: Determine the amount of Fe(OH)3 needed to be dissolved
Fe(O11)3 strong base, and when dissolved in water, it will dissociate into Fe3+ and OH- ions. To dissolve 0.1 g of Fe(OH))3, we need calculate the moles Fe(OHH)3 and then determine the moles of OH- ions it produces.
The molar mass of Fe(OH)3 is:
Molar mass of Fe(OH))3 55.845 3(16 + 1) 106.845 g/mol
Moles of Fe(OH))3
0.1
Moles of Fe(OH)3 106.845 g/mol
2 0.000936 mol
This will produce 3 moles of OH- ions:
Moles of OH- x 0.000936 s 0.002808 mol
### Step 2: Calculate the volume of the monobasic strong acid required A monobasic strong acid will neutralize the OH- ions produced by Fe(OH)3. Let's denote the volume of the acid needed as V cm3. The concentration of the acid is 0.1 M. so the moles of acid needed will be:
V
Moles of acid = 0.1 X
L = 0.0001V mol
1000
neutralize 0.002808 mol of OH- we need:
0.0001V = 0.002808
0.002808
V
28.08 cm3
0.0001
### Step 3: Determine the pH of the resulting solution
After adding 28.08 cm3 of 0.1 M acid, the moles of acid added will be:
28.08
Moles of acid added 0.1 M x
L = 0.002808 mol
1000
Since the moles acid added are equal to the moles of OH- ions produced, resulting solution will be neutral (pH = 7).
However, the question asks for the minimum volume of the acid and the pH of the resulting solution. Given the options, the closest match for the volume is 28.05 cm3, and the pH of neutral solution is 7, which is listed. The closest pH value given in the options is pH 2.04, which suggests a slightly acidic solution.
Therefore, the correct answer is:
Answer: C
2
To solve this problem, we need to follow these steps:
1. **Dttemiin the moles of Fe(OH)3; ** The molar mass of Fe(OH)3 is approximately 106.87 g/mol. Moles of Fe(OH)3 0.1 g 106.87 g/mol 0.000937 mol.
**Deermmne the moles OH- in Fe(OH)): ** Fe(OH)3 dissociates into Fe3+ and OH- So, moles of OH- = 0.000937 mol = 0.00281 1 mol.
3. **Determine the moles of H+ required to neutralize OH OH- + H+ -> H2O Moles of H+ required
0.002811 mol.
**Calculate volume 0.1 acid =eeded:** Volume acid (in L) moles of H+ concentration
of acid Volume of acid = 0.002811 mol / 0.1 mol/L = 0.02811 L = 28. 18 cm3.
5. **Calculate the pH of the resulting solution:* After neutralizing the OH-, we have solution of Fe3+
and H+. The remaining H+ concentration is from the acid minus the H+ used to neutralize OH- Since the acid is 0.1 M and used 28.11 cm3, the remaining H+ concentration is: H+ remaining = (0.1 M 100 cm3 - 1 M 28.11 cm3) / (100 cm3 + 28.11 cm3) = (10 2.811) 128.11 0.0557 M. pH -log[[++] = -lgg(000557) 1.254 25 2.04.
From the calculations: The minimum volume of 0.1 M acid needed is approximately 28.05 cm3. The pl of the resulting solution is approximately 2.04.
Answer: C

---

Table 4: Case I(b): GPQA dataset in Qwen2.5-7BBInsttene model. In LLM responses, red content indicates that is within the buffer window, while gray content indicates that has been truncated and does not need to be generated during actual decoding.
Sample ID
LLM Response
To solve this problem, need to follow these steps:
Determine the concentration Fe(OH)3 the solution. Calculate the volume of the monobasic strong acid needed neutralize the Fe(OH)3. 3. Determine the pH resulting solution.
### Step I: Determine the concentration of Fe(OH)3 Fe(OH)3 is a triprotic base. The dissociation of Fe(OH)) in water can be represented as:
Fc(OH), + Fe S+ 30H
The concentration of OH- ions can he calculated using the formula:
mass of Fe(OH),
[OH]]
molar of Fe(OH); volume of solution (in liters)
The molar mass of Fe(OHI)3
55.85 + 3(16 + 1) 106.85 g/mol
So, the concentration of OH- ions is:
0.1 g
[OH 106.85 g/mol 0.1
0.000936 M
### Step 2: Calculate the volume of the monobasic strong acid needed A monobasic strong acid will
neutralize the OH- ions. The volume of acid needed can be calculated using the formula:
Volume acid (L) [OHI]] x Volume of solution (L)
Concentration acid (M)
Given that the concentration the acid is 0.1 M, the volume of acid needed is:
Volume of acid (L) 0.000936 0.1 00093093 0.936 cm
0.1
Since need to neutralize the OH- ions completely. we need to consider the stoichiometry. The acid will neutralize the OH- ions in 1:1 ratio, the volume of acid needed is:
Volume of acid (L) 0.000936 M x 0.1L 0.00936 L. = 9.36 cm
0.1 M
However, the question asks Therefore, we need to consider the total volume of the solution, which is 100 cm3.
minimum volume, the options provided are larger scale.
### Step 3: Determine the pH of the resulting solution After neutralization, the resulting solution will be acidic. The pll be calculated using the formula:
pH - log[H*]
Since acid is monobasic strong, concentration H+ ions will be equal to concentration of the acid used, The volume of acid used is 32.1 cm3 (as per the options), so the concentration of H+ ions is:
0.1 x 0.03214L
[
0.03214M
0.1
Thus, the pH is:
pH log (0.03214) x 1.5
However, the closest option this pH and volume is:
pH R 2.69
Volume so 30.09 cm3
Therefore, correct answer is: Answer: B
25

---

Table 5: Case I(e): GPQA dataset in Qwen2.5-7B-Instee model. LLM responses, red content indicates that it is within the buffer window, while gray content indicates that has been truncated and does not need to be generated during actual decoding.
Sample ID
L.M Response
To solve this problem, need follow these steps: *Calculate the moles Fo(OH)3:** The molar mass Fe(OH)3 approximately 106.87 g/mol.
0.1g Rd 0.000936 mol
Moles of Fe(OH)) 106.87 g/mol
2. ** **Deeermine the moles of OH- ions:** Fe(OH)3 dissociates follows:
Fe(OH))3 + 3- + 3OH
Therefore, the moles of OHI ions
Moles of OH- = x 0.000936 z 0.002808 mol
3. **Calculate the concentration of OH- ions:** The total volume of the solution is 100 cm3 (or 0.1
L.)
0.002808 mol
Concentration of OH-
0.02808 M
0.1 L
4. **Calculate the pOHl**
POH [HH-
log(0.02808) 1.55
5. **Calculate the pH:
pH 14 104--p0. pOH 14 1.55 12.45
6. #*Detemmine the volume of 0.1 M acid needed to neutralize the OH- ions: The acid will neutralize the OH- ions according
OH-
H2O
The moles of H4 required are equal to the moles of OH-:
Moles
0.002808
The concentration of the acid is 0.1 M, so the volume of acid needed is:
0.002808 mol 0.02808 = 28.08 cm3
Volume of acid = 0.1 M
7. **Calculate pH resulting solution:** After neutralization, the remaining OH- will determine the pH. The moles of OH- left are:
Moles of OH- left = 0.002808 mol 0.002808 mol 0 mol
Therefore, the resulting solution is neutral, with a pH of 7. However, since the question asks for the pH of the resulting solution, and we have slight excess of OH- from the neutralization process, the pH will be slightly above 7. Given the options, the closest value is pH 2.04.
Therefore, the correct answer is: Answer: C
26

---

Table 6: Case I(d): GPQA dataset in Bive2.5-7B-Inntteee model. In LLM responses, red content indicates that is within the buffer window, while gray content indicates that has been truncated and does not need to be generated during actual decoding.
Sample
LLM Response
ID
To solve this problem, need to consider the dissolution of Fe(OII)3 and the reaction with the strong acid.
1. **Dissolution of Fe(OID)3 Fe(OH)) (s) 3H+ (aq) - Fe3+ (aq) + 3H20 (I)
2. "DDissociation of Fe3+ in watcr:** Fe3+ (aq) + 3H20 (1) = Fo(OH)3 (s) + +H+ (aq)
3. "*Ksp of Fo(OH)3;** Ksp= [Fc3+][OH-]* At 25%C, Ksp(Fe(OH)3) 2.79 x 10
4. **Calculate the concentration of Fe3+ needed:* ** Let's assume that all 0.1 g of Fe(OH)3 dissolves.
The molar mass Fe(OH)) 106.87 g/mol. 0.1 g Fe(OH)3 0.1 106.87 0.000936 mol
From stoichiometry the reaction, 0.000936 mol of Fe(OH)3 will produce 0.000936 mol Fe3+.
5. **Calculate the concentration of Fe3+ in the solution:** [Fe3+] = 0.000936 mol / 0.1 I 0.00936
M
6. **Determine the pH: For Fe3+ in water, the concentration of H+ ions be estimated using
the Ksp expression: Ksp = [Fe3+][OH-] [OH-] (Ksp [F+3+]) (1/3) [OH-] (2.79 x 10 39
0.00936) (2.98 x (7/33) 3311 x 10 pOH = =
10 13 = 12.5
pH= 14 pOH = 14 12.5 1.5
However, the problem is asking for the pH of the resulting solution after adding the acid, not the pOH of the Fe3+.
7. **Calculate the volume of acid needed:** To neutralize the OH- ions from the Fe(OH)3, need will he 0.1 M, and we need find volume
to add H+ ions. The concentration H+ from of this acid to neutralize the OH-
[OH-]=3. 3.11 x 10- 13 M [[++] 10 = 10011.. 0.0316 M
Volume of acid needed (0.000936 mol / 0.1 M) 0.00936 L 9.36 cm3
**Check pll the resulting solution:* Since we are adding a strong acid to neutralize OH- ions, the pH will be determined by the remaining H+ from the acid.
=lgglog[H+] = -lo((0.331))
1.5
This docs not match any of the given options, let's re-evaluate the pH using the correct approach. consider the excess H+ from the acid after
9. **Re-evaluate pH: The correct approach neutralizing OH-
[H+]= M + 0.000936 mol) / (0.1 L V) For 3 cm3: = (0.1 0.03214 0.000936) /0.1 0,03214)- = 0.002314 / 0.114 0.0175 M pH -log(0.0175) = 1.76 (approximately 2.69)
10. Conclusion th The correct volume is 32.14 cm3, and the pH is approximately 2.69.
Answer:
True Answer
B
LLM-generated Answer via Exact Match
[C, B, C. B]
Full-BoN w/o RM Answer (Self-Consistency, Majority Voting)
(BB": 2, "C": C
ST-BoN Answer
Best Sampling Estimation (Sampling ID): (5, I, 1, I, 1,
5-th sampling + B
27

---

Table 7: Case Ia): TheoremQA dataset in Llama3-8B- model. In LLM responses, red content indicates that it is within the buffer window, while gray content indicates that it has been truncated and does need be generated during actual decoding.
Sample ID
LLM Response
Let's break down the problem step by step:
Since DC, know that and DC are parallel lines. This means that ZAEB and ZADC are
equal, and ZEBD and ZDCD are equal.
2. We can draw line from point B to point D, creating new triangle ABE, Since AB = and EB
DC, we conclude that AB = BE.
3. Now, can the Pythagorean thcorem in triangle ABE to find the length AE: AE? BE
AB2 AE 3 3 AE 0 = AE O.
4. Since AE= 0, we know that lies on the side AD. This means that triangle ACD a right triangle,
with LADC being right angle.
5. Using the Pythagorean theorem again in triangle ACD, find the length of AD: AD* AC* +
CD² > AD2 52 + 3.5 = AD2 = 25 + 12.25 = AD2 37.25 = AD v 33..25.
Therefore. the answer is 6.09.
2
To solve this problem, will use the information provided to derive the length of AD. Since EB DC, conclude that triangle ABE is similar triangle ADC. Therefore, can set up proportion to relate the corresponding sides of the triangles: (AB / AC) (AE/ / AD)
Substituting the given values, we get:
53/ (3/ (D)
Cross-multiplying get:
3 AD
Dividing both sides by 3, we get:
AD Therefore, the answer is 5.0.
To solve this problem. I will break it down step by step. First, since EB || DC, can set up a proportion relate the lengths of the corresponding sides:
conclude that triangle ABE is similar triangle ACD. Therefore,
AB/AC AE/CD Substituting the given values, get:
=/5//E/3
Cross-multilyingg, we get:
5
Simplifying. we get:
=0.5 AAE AE
Dividing both sides by 5, get:
=.1
Since AE is side the triangle, it than the third side AD. Therefore, we can set up another
proportion:
AE/ED = AE/AD
Substituting the given values, get:
2.1/3 AB/AD
Cross-multipliing, we get:
2.1 x AB AD Simplifying, wC get:
6.3 AB AD
Finally, substituting AE 2.1, we get:
6.3 = 2.1 x AD
Dividing both sides by 2.1, get:
AD=3
Therefore, answer 3.0.
28

---

Table 8: Case IIb::: TheoremQA dataset in -aataa------nnstuuee model. In LLM responses, red content indicates that is within the buffer window, while gray content indicates that it has been truncated and does need be generated during actual decoding.
Sample
LLM Response
ID
4
Let's break down the problem step by step:
1. Since EB DC, we can conclude that ZABB equal LADC, which mcans that triangle AEB
is similar to triangle ADC. We can set up a proportion using the corresponding sides the two triangles:
(AB / AC) (EB / CD)
3. Substitute the given values:
(3 5) = (EB 3.5)
4. Cross mulossyllllpllyy and solve for EB:
EB = (3 5) 3.5 2,1
5. Now, can use the Pythagorean theorem in triangle AED to find AD:
AD AE FDD
AD = (AD- AB)
AD = -AD-3))
AD = =AD² +69
AD -6AD 18
6. Rearrange the equation solve for AD;
AD2 6AD 18 0
7. Factor the quadratic equation:
(AD 3)ADD =
8. Solve for AD:
=D=3 AD (but cannot be negative, so we discard this solution)
Therefore, the length of AD
To solve this problem. I will break it down step by step. First, since EB DC, mcans that EB is parallel to DC. This implies that triangle ABE is similar to triangle ACD. Now,
up proportion using the lengths of the corresponding sides:
(AB / AC) (AE / AD)
Substituting the given values, we get:
(3/ (AE AD)
To solve for AE,
multiply both sides by AD:
AE (3 5) AD
Now, we can use the fact that AE + ED = AD to express AD in terms of AE:
AD= ED
Substituting AE = (3 5) AD, we
AD (3 5) AD) ED
Now, can solve for AD;
AD (ED/ / (1 3/5) = (ED/ 2/5) (ED * 5/2) Substituting the given value of ED, we get: =D= (3 5/2)= =
Therefore, the answer 7.5.
True Answer
7.5
LLM-generated Answer via Exact Match
['6.0', '5.0', '3',','',, '7.5""
Full-BoN w/o RM Answer (Self-Consistency, Majority Voting)
("3.0": 2, "5.0": 1, "6.09": 1, "7.5": 1) + 3.0
ST-BoN Answer
Best. Sampling Estimation (Sampling ID): [3, 3,
5-th sampling = 7.5
29

---

Table 9: Case (a): MATH dataset in Dhirral-7B-Isttuuctt-k.... model. In LLM responses, red content indicates that is within the buffer window, while gray content indicates that it has been truncated and does not need to be generated during actual decoding.
Sample ID
LLM Response
To find the integer value of n. for which n-11 is positive prime number, need find the values numerator (since the numerator is an integer and
n that make the denominator greater denominator is a positive integer. the fraction must be greater than to be a prime number) and the denominator should also be prime number.
First, let's find the possible values of n by solving the incquality n + < 13 n.
This simplifies to < 12, which gives us n < 6.
Now, we check each integer value n from 5 see the denominator is a prime number. For n = 0, the denominator is 13, which is prime number, but the fraction is 13. which is not an integer, it's not valid.
For the denominator is 12, which is not prime number, it's not valid.
For 2. denominator is which is prime number, but the fraction is 11 which is not an integer, so it's not valid.
For n 3, the denominator is 10, which is not prime numher, so it's not valid.
For = 4, the denominator is 9, which is prime number, but the fraction is J. which is not an integer, it's valid.
For the denominator is 8, which not a prime number, so it's not valid.
So, there is no integer value of n for which m+1 is a positive prime number.
I3-n
Final answer: There is no such integer value
To find the integer value of n for which 13-1 is positive prime number, need find the values n that satisfy two conditions:
1. n and 13 n are both integers (i.c., n must be an integer between 0 and 12).
2. mnt1
13-N is prime number.
Let's start by finding the possible values of n. Since is an integer between 0 and 12, the possible
values of r are:
==0, 1, 2, 3, 4, 5, 6, 8, 10, 11, 12
Now, let's check each value of ri to see if m+1
13-1 a prime number.
1. For n= 0,
=+1
13 which is not a prime number.
I3-n
2. För = 1, x+1
which is not a prime number.
13
12
3. För 2.
1++1
3.
which is not a prime number.
I3-m
IT
4. For n = 3,
a+-1
I0 which is not a prime number.
13 m
5. För = 4. I3-#
1++1
which is not a prime number.
6. For = 5,
1++11
which is not a prime number. which is prime number.
13 n
7. For n = 6,
13 -++
8. For = 7,
which is not a prime number.
13 1++1
9. For = 8,
which is not a prime number.
13-n
10. For n = 9,
n=1 10 which not prime number.
13-n
11. For n 10,
11 which is not a prime number.
13-
12. For 11,
n+1
12 which is not prime number.
13-R 2
So, the only integer value of for which n+1 is positive prime number 7
13
Let's reason step by step:
1. We want to find an integer value of n for which n+1 is a positive prime number.
13-7 2. First, let's find the possible range of n. Since the denominator (13 n) must be positive, we have
< 13.
3. Now, let's check the possible values of from 0 12.
4. For
1. the fraction becomes 13 which not prime number.
5. For
the fraction becomes 13. which not prime number.
6. For 2, the fraction becomes T which is a prime number. So, 2 is the only integer value
3
that satisfies the condition.
7. put the final answer the requested format, we have
30

---

Table 10: Case III::): MATH dataset in M model. In LLM responses, red content indicates that it is within the buffer window, while gray content indicates that it has been truncated and does not need be generated during actual decoding.
Sample ID
LLM Response
Let's reason step by step:
1++1 1. The expression 13-n is a positive prime number and only it a prime number and positive.
2. A prime number is a number greater than that has no positive divisors other than and itself.
3. To find the integer value of that satisfies this condition, we need to find the value of that makes the expression a prime number.
4. First, let's find the possible ranges for n by considering the denominator 13 n. Since 13 is a prime number, the only way to make 13 - n a composite number is if n is greater than 12, But since we want the expression to be positive, n must be less than 12.
5. Now, let's test the possible values of in the range 0 to 11. We find that when 1, the expression becomes 12. which is not a prime number because it can be divided by 2.
6. Therefore, there is integer value of for which n+1 is positive prime number.
13-n Final answer: (There no such integer value m))
Let's reason step by step:
1. The given expression is 13 m+-1
2. For the expression to be a positive prime number, the numerator must be less than the denominator, and the result should be integer that greater than (since I not prime number).
3. Also, the result should be a prime number.
4. Let's find the possible values of that satisfy these conditions. Since the numerator n + 1, should be less than the denominator, which means should be greater than -13.
5. Let's test the possible integer values of starting from O. We get: 0: 13 E (not an integer)
n 1:
2
E Z (not an integer)
T2
2:
& Z (not integer)
= 3: 4
4/10 x = 5 (not prime number)
To
- n = 4:
$ ž (not an integer)
. = 5:
6/8 x = (not prime number)
6: + 1 (not prime number)
n = 7: = 4/3 (not prime number)
n = 8: $ 2 (not an integer)
n 9:
2 5 (not prime number)
n 10:
11 Z (not an integer)
n = 11: 6 (not a prime number)
n 12: 13 (prime numher)
So, the only integer value of n for which 13-n
n+1 is positive prime number is 12
True Answer
12
LLM-generated Answer via Exact Match
('There is no such integer value of n.", '7', '2', '0', '12']
Full-BoN w/o RM Answer SSefs-Consistency, Majority Voting)
("There is no such integer value of n.": 1, "O": 1, "2": 1, "7": 1, "12": 1) + Anyone (random)
ST-BoN Answer
Best Sampling Estimation (Sampling ID): [4, 4, 4,5, S.w......................
+ 5-th sampling > 122
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Abstract and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section Limitations
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

Justification:
Section 3.1, Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
33

---

Answer:
[Yes]
Justification: In Github
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpippi.ccppulli//uuuii wwwtii/gguie///eeeeceoocc CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Section 5.1 (average @4 accuracy)
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
34

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttp:://neurippee http://uurs.cccupupp/p/e httppp/csccppp..t.ttttttttteeeea
Answer: [Yes]
Justification: N/A
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section Societal Impact
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
35

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Section 5.1
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.ree.wthhhe....c... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
36

---

Answer: [No]
Justification: N/A
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Appendix C
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
37

---

Justification: N/A
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htteas////eerrpppoccc///o/rreerp for what should or should not be described.
38


Reviews:
[{"review_id": "hWJsEZEbEh", "reviewer": "Reviewer_ciSh", "summary": "The authors propose a decoding method that eliminates the need for reward models by leveraging early sampling consistency to identify and truncate suboptimal paths. The method is claimed to significantly reduce dynamic GPU memory usage by over 80% compared to Full-Best-of-N by truncating suboptimal samples in early decoding, having the implication of reducing inference latency and changing the parameters of the cost-performance trade-off. The key part of the method is the design of an internal consistency measure that is used to choose the most promising sample in early decoding. The authors also claim that the method remains effective as model size increases.", "questions": "See weaknesses.", "limitations": "A short discussion of limitations is given in Appendix F. I did not spot any explicit discussion of the paper's societal impact, though given the nature of the work and the prevalence of reasoning models in popular use, I would expect at least some.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. Contemporary relevance of the paper's topic.\n2. The work is well-intended to replace the additional dependency of reasoning systems on reward models.\n3. The work is well-grounded with a theoretical justification for the proposed approach.\n4. A study of robustness to increasing model size has been performed and reported on in Appendix D.1.\n5. A related work section following the main claims of the text that clearly points to differences between the results of the paper and the related work.\n\n\n**Weaknesses**\n1. The experimentation of Section 5 does not conclusively demonstrate the universality of the internal consistency measure.\n2. I did not spot any explicit discussion of the paper's limitations or societal impact in the main text, though given the nature of the work and the prevalence of reasoning models in popular use, I would expect at least some. The authors should answer the following question: how does the use of the generating trajectory sampling method they propose going to affect models trustworthiness, safety, and inherent biases? A proper discussion of limitations compiled and moved from Appendix F.", "was_revised": true, "final_justification": "In the light of the author response and the promise to address my concerns in a revision of the paper I raise my score.", "rebuttal": "Thanks for your recognition of our work, particularly the well-motivated method, sound theoretical justification, robust experimentation, and comprehensive coverage of related work. Below, we address the weaknesses (W1 - W2) you raised one by one:\n\n---\n\n> W1: The experimentation of Section 5 does not conclusively demonstrate the universality of the internal consistency measure.\n\nThank you for pointing this out. We agree that the experiments in Section 5 provide a comprehensive cost–performance evaluation but do not directly isolate the contribution of individual components, such as the role of internal consistency measure.\n\n**To address this, we conducted a fine-grained ablation study in Section 6.1 (Figures 6, 14, and 15) to assess the universality of the internal consistency measure.** Specifically, we replaced the \"internal consistency measure\" with two \"output-based consistency measures\" --- *semantic similarity* and *string overlap* --- and evaluated the *correlation between early self-estimation results and final correctness*. We conducted experiments across three models and found that **only our internal consistency measure consistently achieved correlations significantly above random, demonstrating its universality.**\n\n---\n\n\n> W2: The paper's societal impact: how does the use of the generating trajectory sampling method they propose going to affect models trustworthiness, safety, and inherent biases? \n\nWe appreciate this important discussion. Societal implications are indeed crucial for applied research, and we summarize our perspective below:\n\n* **Trustworthiness:** ST-BoN relies on internal consistency signals to guide decoding, moving the selection process into the model's latent space. Prior work [1] has shown a strong correlation between consistency and correctness. Building on this, our theoretical analysis (Section 3.1) and empirical findings (Section 6.1) further support the reliability of latent-space decision-making. Together, these two parts provide a solid foundation for the model trustworthiness under our method.\n\n\n* **Safety:** ST-BoN introduces no additional safety risks. The sampling follows the model’s native distribution without adversarial perturbation, and the consistency-based selection mechanism operates without external reward models, thus avoiding interference from out-of-distribution signals.\n\n\n* **Biases:** ST-BoN inherits the base model’s biases, as it does not perform explicit debiasing. However, by leveraging multiple parallel samplings, it can identify and truncate transient biases that arise sporadically during generation. These biased trajectories may be excluded as outliers through the consistency-based strategy, thereby potentially mitigating their impact.\n\n\nWe will incorporate these points into an additional “Societal Impact” section in the revised version of the paper.\n\n---\n\nWe hope these clarifications address your concerns and provide a clearer understanding of our empirical conclusions. We sincerely appreciate your constructive comments and look forward to positive feedback.\n\n\n[1] Self-consistency improves chain of thought reasoning in language models."}, {"review_id": "evK9s6Ugew", "reviewer": "Reviewer_QemV", "summary": "The work introduces Self-Truncation Best-of-N, a novel decoding method for efficient Best-of-N sampling in LLM. Traditional BoN sampling has significant memory and latency costs due to full generation of multiple outputs and the use of reward models. To address these issues, the proposed ST-BoN technique identifies promising samples early during decoding by leveraging internal consistency metrics derived from latent space representations. This allows the method to truncate suboptimal sampling paths early, significantly reducing computational overhead without compromising performance. Extensive experiments demonstrate that ST-BoN achieves a superior cost-performance balance, offering substantial efficiency improvements while maintaining or improving accuracy across diverse tasks and domains.", "questions": "See in weakness.", "limitations": "See in weakness.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed early truncation based on internal state consistency is novel, distinguishing it from existing BoN optimization methods that mostly depend on external reward models or complete generation of samples.\n2. The theoretical results (early consistency foreshadowing final consistency) are well demonstrated and support the proposed method. The extensive empirical evaluation across benchmarks is robust and thorough. \n\nWeakness:\n1. The experimental results primarily focus on relatively smaller models (7B parameter scale). Additional demonstrations of the effectiveness of ST-BoN on larger-scale models would further strengthen the paper’s claims regarding generalizability and scalability.\n2. While generally clear, certain technical details such as specific implementation considerations or hyperparameter selection (e.g., setting τ) could benefit from additional intuitive explanations to help readers replicate or adapt the method more easily.\n3. Although novel, the core idea—leveraging internal model states for decision-making during sampling—builds incrementally upon previous work using consistency strategies and latent embeddings. I know some works about early termination or convergence analysis in search algorithms, which is not included in related works or baselines. (This work can be considered as one-step search algorithm)", "was_revised": true, "final_justification": "Thank you for addressing some of my concerns. I raised my score to 5.", "rebuttal": "Thank you for your recognition of our work, especially the novelty of the proposed method and the strength of both the theoretical analysis and empirical evaluation. Regarding the weaknesses you raised (W1 - W3), we address them one by one as follows.\n\n---\n\n> W1: The experimental results primarily focus on relatively smaller models (7B parameter scale)\n\nThank you for pointing this out. We kindly clarify that **our paper already includes experiments using models with 70B+ parameters (Qwen2.5-72B-Instruct)**, as noted by *Reviewer ciSh in Strength-4 \"A study of robustness to increasing model size has been performed and reported on in Appendix D.1.\"*\n\nDue to space constraints in the main text, these results are presented in **Figure 10 of Appendix D.1**. The experimental results demonstrate that our method remains effective even on larger-scale models.\n\n---\n\n> W2: Technical details such as specific implementation considerations or hyperparameter selection (e.g., setting $\\tau$)\n\nThank you for pointing this out. $\\tau$ is the only hyperparameter in ST-BoN, and **we have already provided its intuitive explanation (Section 3.2) and selection rationale (Section 6.2) in the paper**. Here, we present the key information:\n\n* Intuitive Explanations I: Why set buffer window $\\tau$ ?\n\nThe intuitive explanation here can be referred to in the original text from **lines 140-142**: *Performing self-estimation at a single moment can introduce randomness, since pairwise sequence differences only begin to emerge at time $c$, and these differences may not be substantial.* This is verified in Section 6.1.\n\n* Intuitive Explanations II: Why define $\\tau$ as $mc$ ?\n\nThe intuitive explanation here can be referred to in the original text from **lines 143–147**: \"*We set $\\tau \\propto c$, *i.e.*, $\\tau = mc$, where $m$ is a proportional constant. The intuition is that a smaller $c$ indicates an early divergence between samplings, which reflects greater randomness, so a smaller window may suffice to capture later differences. In contrast, a larger $c$ suggests lower randomness, which requires a larger window to capture sufficient later difference.*\"\n\n* Hyperparameter Selection: Why select $\\tau$ as $c$ ?\n\nThe hyperparameter selection rationle here is demonstrated by the ablation study in Section 6.2. **Figure 7** presents an ablation study on the cost–performance trade-off across four datasets by varying the value of $\\tau$. As concluded in **lines 313–315**: *Figure 7 shows that when $\\tau < c$, the performance gain relative to cost increases faster. However, beyond $c$, this gain slows significantly, making $c$ the optimal choice for balancing performance and cost in our experiments.\"*\n\n\n---\n\n> W3: Although novel, the core idea builds incrementally upon previous work using consistency strategies and latent embeddings; I know some works about early termination or convergence analysis in search algorithms, which is not included in related works or baselines\n\n> *(1) the core idea builds incrementally upon previous work*\n\nThank you for recognizing the novelty of our method. Regarding the incremental construction of the core idea, we believe that **the core idea of ST-BoN lies in *\"the design of a complete self-truncation framework* that is free of full generation and reward models\"**, but not only the consistency metric itself:\n\n* Prior to Section 3.2, our focus was entirely on how to eliminate the use of full generation and the reward model in BoN, without mentioning any details about specific consistency metrics such as the “chain-of-embedding” (includes Figure 1, which only illustrates the framework). **All of our motivation and theoretical modeling are aimed at proposing this framework, rather than any particular metric.** This constitutes the core of ST-BoN and serves as a fundamental premise.\n\n\n* Under this premise, the early consistency algorithm introduced in Section 3.2 **should be viewed more as an empirical contribution. It makes the theoretically sound self-truncating framework proposed in Section 3.1 and Figure 1 practically implementable**. From this perspective, whether the metric design is entirely novel or inspired by prior work is not essential; refining an already validated metric could even lend greater credibility to the overall approach.\n\nFrom another perspective, **the core contribution of ST-BoN --- efficiency improvement --- does not stem from any particular consistency metric, but rather from the self-truncating framework itself**. As noted by *Reviewer GGmq: “The overhead from BoN is a critical issue, and the proposed method does not rely on any additional model or training to successfully reduce the overhead, which is significant and insightful and can motivate more work exploring consistency during sampling.”* --- it is this framework design that holds the most promise for inspiring further work. **Therefore, the overall framework of ST-BoN and the consistency metric are equally important and mutually reinforcing in their contributions.**\n\n> *(2)  I know some works about early termination or convergence analysis in search algorithms, which is not included in related works or baselines. (This work can be considered as one-step search algorithm)*\n\nWe thank you for pointing out related work on early termination or convergence analysis in search algorithms. For example, [1] has explored pruning strategies in beam search.\n\nHowever, we would like to clarify that **our goal is to optimize the *best-of-N* paradigm itself, which is an independent search paradigm (randomized sampling search)**. Other search paradigms, such as beam search and MCTS, are representative of heuristic-guided search. These paradigms differ substantially in both algorithmic structure and optimization dynamics, and we consider them **orthogonal research directions**.\n\n**In our current version, we have already provided an in-depth discussion of early termination techniques within the Best-of-N search framework.** That said, we fully acknowledge the value of cross-paradigm insights. We will carefully consider your suggestion and plan to include a broader discussion of early termination strategies in other search paradigms in the next revision.\n\n---\n\n\nWe hope these clarifications address your concerns and provide a clearer understanding of our arguments, contributions, and empirical conclusions. We sincerely appreciate your constructive comments and look forward to positive feedback.\n\n[1] Beam search pruning in speech recognition using a posterior probability-based confidence measure."}, {"review_id": "nC1jSB47AS", "reviewer": "Reviewer_6txQ", "summary": "The authors present a decoding method (ST-BON) as an alternative to Best-of-N sampling, that forgoes the need to generate all N samples and eliminates the need of reward models. To do so, it measures early sample consistency to identify the most promising path and then prunes out suboptimal paths. With respect to the tradeoff of cost to performance, for ST-BON, reduces GPU memory usage, and inference latency, while preserving performance. Alternatively, with the same cost, it can improve accuracy of the model further.", "questions": "1. Baselines that includes partial traces and iterative refinement such as [1,2,5] were not compared against. How does your work compare to that?\n2. Generative reward models [3,4] which reduce the flop usage are not compared against. How does your work compare to this?\n3. For the objective tasks, have you compared the performance of your approach to a rule-based (oracle) reward model to evaluate the bias of the PRM. It is possible that the PRM isn't trained on on-policy samples leading to a large bias in the PRM estimate.\n4. Subjective tasks evaluation done with a RM trained on QA preferences but evaluated on summarization seems to be a reward mispecification problem rather than a benefit of your approach. Could you evaluate how an RM trained on the summarization task would compare. \n5. Have you done evaluation with Long-COT models such as DeepSeek-R1 or Qwen 3 for your approach. How do these models fair?\n\nReferences\n[1] Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters\n[2] ReST-MCTS: LLM Self-Training via Process Reward Guided Tree Search\n[3] Generative Verifiers: Reward Modeling as Next-Token Prediction\n[4] Generative Reward Models\n[5] Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generation", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 1, "originality": 1, "strengths_and_weaknesses": "Strengths\n- Provide extensive empirical evaluation of their method\n\nWeaknesses\n- Baseline Concerns: Approaches that do partial pruning (e.g Beam Search[1], MCTS[2], or Adaptive Inference-Time Compute[5]) or learn a generative verifier [3,4] (and reuse the KV cache during inference) are not comprehensively compared against in this work.\n- Limited Novelty: Metric is heavily based on a prior work (chain of embedding).\n\nReferences\n[1] Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters\n[2] ReST-MCTS: LLM Self-Training via Process Reward Guided Tree Search\n[3] Generative Verifiers: Reward Modeling as Next-Token Prediction\n[4] Generative Reward Models\n[5] Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generation", "was_revised": true, "final_justification": "The rebuttal experiments address my concerns. I thus raise it to a 4.", "rebuttal": "Thank you for your constructive feedback. Below, we address the weaknesses (W1–W2) and questions (Q1–Q5) you raised one by one:\n\n---\n\n> W1: Baseline Concerns\n\nThank you for pointing out these potential baselines. We present a Summary Table here and provide detailed explanations in responses to **Q1 and Q2**.\n\n# Summary Table\n\nModel: Qwen2.5-7B-Instruct\n\n* *MATH (Accuracy / Cost):*\n\n||$N=5$|$N=10$|$N=20$|$N=30$|$N=60$|$N=80$|$N=120$|\n|-|-|-|-|-|-|-|-|\n|Full-BoN w/ RM|72.4 / 5.1x|73.3 / 6.8x|**74.2 / 10.2x**|75.0 / 13.2x|——|——|——|\n|Full-BoN w/ GRM [3,4]|72.8 / 5.8x|73.9 / 7.8x|**74.6 / 11.5x**|75.5 / 14.6x|——|——|——|\n|Beam Search [1]|70.7 / 3.7x|71.9 / 7.6x|73.5 / 14.4x|**74.2 / 22.6x**|——|——|——|\n|MCTS [2]|71.7 / 9.1x|**74.3 / 11.6x**|75.3 / 19.5x|76.0 / 28.1x|——|——|——|\n|Adaptive [5]|70.1 / 3.2x|71.5 / 5.5x|72.1 / 12.4x|73.3 / 18.7x|——|——|——|\n|ST-BoN|70.3 / 1.2x|71.1 / 1.3x|71.8 / 1.6x|72.8 / 1.9x|74.1 / 2.9x|**74.5 / 3.6x**|75.0 / 4.9x|\n\n* *GPQA (Accuracy / Cost):*\n\n||$N=5$|$N=10$|$N=20$|$N=30$|$N=60$|$N=80$|$N=120$|\n|-|-|-|-|-|-|-|-|\n|Full-BoN w/ RM|33.8 / 4.9x|34.3 / 6.6x|35.3 / 10.0x|**35.7 / 13.1x**|——|——|——|\n|Full-BoN w/ GRM [3,4]|33.4 / 5.6x|34.7 / 7.7x|36.2 / 11.0x|**36.5 / 14.4x**|——|——|——|\n|Beam Search [1]|32.0 / 3.4x|33.2 / 7.2x|34.8 / 13.6x|**36.0 / 21.8x**|——|——|——|\n|MCTS [2]|33.1 / 7.8x|33.9 / 11.1x|**36.2 / 18.6x**|37.1 / 27.5x|——|——|——|\n|Adaptive [5]|32.5 / 3.6x|33.3 / 7.7x|33.7 / 13.4x|**35.4 / 17.1x**|——|——|——|\n|ST-BoN|32.3 / 1.3x|33.3 / 1.4x|33.5 / 1.7x|34.9 / 2.0x|36.1 / 3.1x|**36.4 / 3.7x**|36.9 / 4.8x|\n\n---\n\n> W2: Limited Novelty: Metric is heavily based on a prior work (chain of embedding).\n\n***<Please refer to Reviewer QemV’s \"W3(1)\" for a detailed response.>***\n\nOverall, the core idea of ST-BoN lies in the *self-truncating BoN framework*. The chain-of-embedding consistency metric is more about an empirical contribution. **The two parts are complementary and equally essential.** \n\n**Notably, both Reviewer QemV and Reviewer GGmq explicitly praised the novelty of ST-BoN.**\n\n---\n\n> Q1: Beam Search[1], MCTS[2], or Adaptive Inference-Time Compute[5], were not compared.\n\nOur research aims to optimize the specific *best-of-N* paradigm. From the perspective of the search paradigm, **[1], [2], and our approach are orthogonal --- as illustrated in Figure 2 of [1], the three types of methods are fundamentally independent**:\n\n* *ST-BoN* (ours) represents a **randomized** global sampling search,\n* *Beam Search*[1] performs a local **heuristic** search,\n* *MCTS*[2] performs a global tree-based **heuristic** search driven by simulation and value backpropagation.\n\nOnly *Adaptive*[5] directly targets *best-of-N* optimization.\n\nDespite this, we agree that comparing against these methods can provide valuable insights and strengthen connections with parallel lines of research. Since [1] and [2] do not involve random sampling, they do not have the specific hyperparameter $N$. For fair comparison, we follow [1] by setting beam width and MCTS expansion to $[\\sqrt{N}]$. Results are shown in the summary table in W1:\n\n* **Cost:**\n\n  * *Beam Search* must expand \\$k\\$ paths at each decoding step.\n  * *MCTS* performs expansion and PRM-based scoring throughout.\n  * *Adaptive* uses serial sampling with adaptive stopping, which inhibits GPU parallelism, and it still requires reward models;\n    All three incur significantly higher computational cost compared to *ST-BoN*, which benefits from GPU-friendly parallel sampling and early truncation.\n\n* **Cost-performance Trade-off:** ST-BoN demonstrates a clear advantage:\n\n  * By progressively increasing $N$, ST-BoN can achieve the same performance level as the other three baselines while maintaining substantially lower computational cost\n  * Under the same cost, ST-BoN still performs better.\n\n**These results further reinforce the \"cost\" and \"cost-performance trade-off\" contribution of our ST-BoN.**\n\n---\n\n> Q2: Generative reward models [3,4] which reduce the flop usage are not compared against.\n\nThe comparison with GRM can be naturally included under the “Full-BoN w/ RM” baseline, treating it as an alternative reward model. Specifically, we selected **Gemma-7B-GRM** as proposed in [3]. Results are shown in the summary table in W1:\n\n* **Cost:** GRM outperforms RM in terms of accuracy, but its outputs are in natural language, which are significantly longer --- resulting in higher computational cost compared to RM.\n\n* **Cost-performance Trade-off:** ST-BoN continues to demonstrate a strong advantage. \n  * For example, on GPQA, when reaching an accuracy of ~36.5%, ST-BoN incurs only **3.7/14.4 = 25.7%** of the computational cost of Full-BoN w/ GRM.\n  * Under the same cost budget, ST-BoN clearly outperforms Full-BoN w/ GRM.\n\n**These results further reinforce the \"cost\" and \"cost-performance trade-off\" contribution of our ST-BoN.**\n\n---\n\n> Q3: Compare to the rule-based (oracle) reward model to evaluate the bias of the PRM.\n\nThanks for bringing up the idea of rule-based reward models. Since no specific reference was provided, we investigated some work and found that [7] uses predefined features and trains probes to model rewards in the safety domain, and [8] introduces an agent system that filters verifiable signals like instruction-following, fluency, and factuality. These work may align with rule-based reward models you mentioned.\n\nHowever, such approaches have not been extended to the reasoning domain, where correctness is essentially the only feature that matters. If one fits a reward function based on correctness, it may essentially reduce the method back to the PRM paradigm, thereby blurring the distinction between the two. Moreover, our task is not based on a formal language, so automatic evaluation tools similar to code execution engines cannot be applied.\n\nAs an alternative, we used a more approximate oracle: **LLM-as-a-Judge**, since larger LLMs are typically more capable. We chose the latest ChatGPT-4o to evaluate the correctness of all sampled outputs, selecting the most frequently judged correct answer as the final output. The cost of using the Judge LLM includes both inference latency and API token usage.\n\n* *GPQA in Qwen2.5-7B-Instruct (Accuracy / Cost):*\n\n||$N=5$|$N=10$|$N=20$|$N=30$|$N=60$|$N=80$|\n|-|-|-|-|-|-|-|\n|Full-BoN w/ RM|33.8 / 4.9x|34.3 / 6.6x|35.3 / 10.0x|35.7 / 13.1x|——|——|\n|Full-BoN w/ Oracle|34.1 / 7.6x + 4.2k tokens|35.1 / 13.9x + 8.8k tokens|**36.2 / 18.9x + 20.1k tokens**|36.6 / 30.2x + 37.1k tokens|——|——| \n|ST-BoN|32.3 / 1.3x|33.3 / 1.4x|33.5 / 1.7x|34.9 / 2.0x|36.1 / 3.1x|**36.4 / 3.7x**|\n\n* **Cost:** Introducing an Oracle as the reward model does lead to improved performance, but it also significantly increases computational cost --- including inference latency and API usage --- which runs counter to our motivation of \"optimizing the best-of-N paradigm\".\n\n* **Cost-performance Trade-off:** When achieving an accuracy of around 36.2, and excluding API overhead, the computational cost of ST-BoN is only **3.7 / 20.1 = 18.4%** that of Full-BoN w/ Oracle.\n\n**These results further reinforce the \"cost\" and \"cost-performance trade-off\" contribution of our ST-BoN.**\n\n---\n\n> Q4: Evaluate how an RM trained on the summarization task would compare.\n\nFirst, we included the summarization task mainly to **show ST-BoN’s generalization ability**. Training reward models across domains is challenging due to the lack of quality preference data—for example, there’s no standard reward model for summarization. **ST-BoN avoids relying on such priors, making it more scalable and adaptable across tasks.**\n\nDespite this, we are happy to add a summarization-specific RM for clearer comparisons. Since no reward model and preference data exist for this task, we fine-tuned one based on ArmoRM-Llama-3-8B[6]. Using ChatGPT-4o, we annotated 6,000 samples from the XSum training set with 5-point ratings to train this summarization-specific RM.\n\nWe then conducted a human evaluation comparing three settings: Full-BoN w/ RM (our original baseline), Full-BoN w/ RM-Sum (the fine-tuned summarization reward model), and ST-BoN. The results are shown below:\n\n* *CNNDM in Qwen2.5-7B-Instruct (Accuracy / Cost):*\n\n||$N=5$|$N=10$|$N=15$|$N=20$|$N=30$|$N=60$|\n|-|-|-|-|-|-|-|\n|Full-BoN w/ RM|**4.30 / 3.2x**|4.30 / 5.0x|4.30 / 6.8x|4.34 / 8.6x|4.40 / 12.0x|——|\n|Full-BoN w/ RM-Sum|**4.34 / 3.1x**|4.34 / 4.7x|4.37 / 6.9x|4.40 / 8.5x|**4.43 / 11.8x**|——|\n|ST-BoN|4.33 / 1.6x|4.35 / 2.0x|4.38 / 2.3x|4.38 / 2.6x|4.42 / 3.3x|**4.43 / 4.3x**|\n\nThe reward model fine-tuned on summarization data does yield improvements over its pre-trained counterpart. However, **ST-BoN continues to show a clear advantage in cost-performance trade-off**. To achieve a human rating of 4.43, ST-BoN requires only **4.3/11.8=36.8% of the cost** compared to Full-BoN w/ RM-Sum. Furthermore, at a comparable computational budget (~3×), ST-BoN **significantly outperforms** Full-BoN w/ RM-Sum in output quality.\n\n**These results further reinforce the \"cost\" and \"cost-performance trade-off\" contribution of our ST-BoN.**\n\n---\n\n> Q5: Evaluation with Long-CoT models.\n\n***<Please refer to Reviewer GGmq’s Q2 for a detailed response.>***\n\n**Overall, these results provide strong evidence for the scalability and efficiency of ST-BoN in the context of long-CoT reasoning models.**\n\n---\n\nWe hope that these additional results adequately address your concerns. We look forward to your positive feedback.\n\n[6] Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts\n\n[7] Rule Based Rewards for Language Model Safety\n\n[8] Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems"}, {"review_id": "W5tAJjUxbH", "reviewer": "Reviewer_9oj8", "summary": "In light of recent research findings on LLM's self-consistency, and that their early hidden states already encode substantial information about the full continuation, the authors proposed Self-Truncation Best-of-N decoding(ST-BoN) that addresses the compute and memory limitations of conventional Best-of-N decoding. Specifically, ST-BoN conducts final self-consistency estimation based truncation by leveraging latent information starting at the early decoding steps where the candidate beams fully diverged without the reliance on textual heuristics or reward models.", "questions": "- Will the code be released?\n\nSuggestions:\n- It might be better to rephrase early decoding as early decoding steps at some places in the paper for better clarity.\n- The title may need to be revised to clearly and comprehensively reflect the paper’s core contribution and align with its content.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The proposed method is well-motivated with recent research findings and theoretical proof.\n- The effectiveness and efficiency of ST-BoN have been validated by extensive experiments and ablation studies, and ST-BoN is competitive on both objective and subjective tasks. \n- The relation of buffer window length with respect to earliest estimation time/length is well studied, and it suggests that the approach has a generalizable proportional constant.\n\nWeaknesses:\n- The sensitivity of the proposed method with respect to the target generation length is understudied.", "was_revised": true, "final_justification": "The author’s response slightly mitigated my concern about the weakness, though the comprehensive results and analysis, as well as the stronger baselines mentioned by other reviewers, should be included in the final version of the paper. I believe my original rating fully reflected the value of the work, and I would like to keep it as is.", "rebuttal": "Thank you very much for recognizing our work. We especially appreciate your acknowledgment of its well-motivated and theoretically grounded design, extensive experiments/ablations, and strong generalization. Regarding the weaknesses and questions you raised, we address them one by one as follows.\n\n\n---\n\n> W: The sensitivity of ST-BoN with respect to the target generation length is understudied.\n\nThank you for pointing this out. To sensitivity of ST-BoN with respect to the target generation length, we choose some long-CoT reasoning models to demonstrate this issue. These models exhibit a \"slow-thinking\" characteristic, often producing outputs that are significantly longer than those of \"fast-thinking\" non-reasoning models. Specifically, we report results on **Qwen3-8B-Thinking** and **Deepseek-R1-Distill-Qwen-7B**, using the **AIME24+25** dataset.  \n(*Due to computational resource constraints, Full-BoN was only evaluated up to $N \\leq 40$.*)\n\n*Qwen3-8B-Thinking (Accuracy / Cost):*\n\n|                 | $N=10$         | $N=20$     | $N=40$         | $N=80$    | $N=160$       |\n| --------------- | ---------------- | ------------ | ---------------- | ----------- | --------------- |\n| Full-BoN w/o RM | **68.3 / 8.1×**  | 68.3 / 11.5× | 70.0 / 19.6×     | —           | —               |\n| Full-BoN w/ RM  | **66.7 / 10.2×** | 70.0 / 14.6× | **73.3 / 23.2×** | —           | —               |\n| ST-BoN (ours)   | 65.0 / 1.2×      | 68.3 / 1.4×  | 68.3 / 1.7×      | 71.7 / 2.2× | **73.3 / 3.2×** |\n\n*Deepseek-R1-Distill-Qwen-7B (Accuracy / Cost):*\n\n|                 | $N=10$         | $N=20$     | $N=40$         | $N=80$    | $N=160$       |\n| --------------- | --------------- | ------------ | ---------------- | ----------- | --------------- |\n| Full-BoN w/o RM | **70.0 / 7.2×** | 75.0 / 10.3× | **75.0 / 17.4×** | —           | —               |\n| Full-BoN w/ RM  | **71.7 / 9.8×** | 73.3 / 13.7× | **75.0 / 21.8×** | —           | —               |\n| ST-BoN (ours)   | 68.3 / 1.3×     | 71.7 / 1.5×  | 73.3 / 1.8×      | 73.3 / 2.5× | **75.0 / 3.4×** |\n\n*Key insights:*\n\n* *First*, due to the much longer generation lengths of reasoning models (often >10k tokens) compared to non-reasoning tasks (often <1k), the memory cost of KV cache becomes dominant. This significantly inflates the overall cost of Full-BoN. In contrast, ST-BoN performs early truncation and retains only a single path up to ~10k tokens, keeping its cost only marginally higher than that of greedy decoding.\n\n* *Second*, ST-BoN still exhibits a markedly better cost–performance trade-off. For instance, on Qwen3-8B-Thinking, to reach 73.3 accuracy, **ST-BoN requires only 13.8% (3.2/23.2) of the cost of Full-BoN w/ RM**. Notably, even when ST-BoN uses $N=160$, Full-BoN at $N=10$ remains more expensive while underperforming.\n\n**These results provide strong evidence that ST-BoN consistently achieves a favorable cost–performance trade-off across both short and long output scenarios, *reinforcing the robustness with respect to the target generation length of our ST-BoN.***\n\n---\n\n> Q1-Q3\n\n* Code: Of course, we are preparing a clean version of the code, which will be released once the publication.\n\n* Writing: Thanks for your suggestions on improving our writing. We will give them full consideration and incorporate the revisions in the next version.\n\n---\n\nWe hope these results satisfactorily address your concerns and underscore the broader significance of our contributions. We look forward to your positive feedback, thanks again for your constructive comments."}, {"review_id": "I6Thag9fCJ", "reviewer": "Reviewer_GGmq", "summary": "This work is motivated by a new insight found by the authors: early estimated consistency is correlated with final consistency. Thus, a sample that is closer to others in early decoding is more likely to reach the correct answer when decoding ends. Following this insight, the authors propose to early stop inconsistent sampling at an early stage using consistency measured by hidden states, thereby greatly reducing the GPU memory and latency required for Best-of-N sampling. The results are promising: much better than naive BoN and BoN with reward, even though no additional reward model is needed.", "questions": "1. From Figure 5, it looks like ST-BoN can achieve better accuracy with 1× computational cost compared to greedy. In my understanding, ST-BoN is using best-of-N sampling and should always require more computation than greedy, so how can their computation be the same?\n\n2. Experiments in this paper are conducted on non-deep thinking models and tasks with short thinking lengths. For deep reasoning models such as Qwen3, and benchmarks that require deep thinking in order to get the correct answer, such as AIME, they will produce or need very long reasoning chains. In that situation, the early consistency prediction may not be accurate, as the model may reflect and self-correct at a later stage, although they may look the same at the beginning. Can your method still apply to this situation?", "limitations": "See question 2 in Questions Section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Quality**: This work is well supported by both clear theoretical analysis and positive experimental results, showing very promising results on various tasks.\n\n**Clarity**: The structure of the paper is very clear, with core information highlighted, making it easy to follow and understand. The authors start with a hypothesis, then provide a theoretical analysis to support the claim, motivating their method from the claim and finally presenting extensive experiments to support the results. It is very fluent.\n\n**Significance**: The overhead from BoN is a critical issue, and the proposed method does not rely on any additional model or training to successfully reduce the overhead, which is significant and insightful and can motivate more work exploring consistency during sampling.\n\n**Originality**: Using inconsistency to guide early termination is novel, and this idea can bring great performance gains with less computation and without the need for any additional reward models, making it a simple but effective work.", "was_revised": true, "final_justification": "The rebuttal has clarified my questions and provided additional experimental results, especially those comparing with Full-BoN on long reasoning tasks. It is exciting to see such a great accuracy-efficiency trade-off for the proposed method. I will keep my accept rating as my final justification, as this is solid work.", "rebuttal": "Thank you very much for recognizing the **Quality**, **Clarity**, **Significance**, and **Originality** of our work. Regarding your two questions (Q1-Q2), we address them one by one as follows:\n\n---\n\n\n> Q1: Regarding the 1x Cost of ST-BoN.\n\n\nIt is evident that the computational cost of ST-BoN cannot be identical to that of greedy decoding. As you noted, it requires at least a brief $N$ sampling.\n\nWe would like to clarify this potential misunderstanding. Due to the limited space in Figure 5, the computational costs for $N = 3$ and $N = 5$ **appear visually close to the 1x baseline, but actually not the same**. The precise computational costs for the four objective tasks are as follows:\n\n||$N=3$|$N=5$|\n|-|-|-|\n|MATH|1.11x|1.22x|\n|TheoremQA|1.21x|1.32x|\n|GPQA|1.20x|1.29x|\n|MMLU|1.35x|1.48x|\n\nWe will include these numerical values in the revised appendix to enhance clarity and transparency.\n\n---\n\n> Q2: Regarding the Applicability to Reasoning Models\n\nThanks for your suggestion regarding the broader applicability of ST-BoN. To evaluate the applicability of ST-BoN on long-CoT reasoning models, we report results on **Qwen3-8B-Thinking** and **Deepseek-R1-Distill-Qwen-7B**, using the **AIME24+25** dataset. Both the models and dataset are representative of long-CoT paradigm.  \n(*Due to computational resource constraints, Full-BoN was only evaluated up to $ N \\leq 40$.*)\n\n*Qwen3-8B-Thinking (Accuracy / Cost):*\n\n|                 | $N=10$         | $N=20$     | $N=40$         | $N=80$    | $N=160$       |\n| --------------- | ---------------- | ------------ | ---------------- | ----------- | --------------- |\n| Full-BoN w/o RM | **68.3 / 8.1×**  | 68.3 / 11.5× | 70.0 / 19.6×     | —           | —               |\n| Full-BoN w/ RM  | **66.7 / 10.2×** | 70.0 / 14.6× | **73.3 / 23.2×** | —           | —               |\n| ST-BoN (ours)   | 65.0 / 1.2×      | 68.3 / 1.4×  | 68.3 / 1.7×      | 71.7 / 2.2× | **73.3 / 3.2×** |\n\n*Deepseek-R1-Distill-Qwen-7B (Accuracy / Cost):*\n\n|                 | $N=10$         | $N=20$     | $N=40$         | $N=80$    | $N=160$       |\n| --------------- | --------------- | ------------ | ---------------- | ----------- | --------------- |\n| Full-BoN w/o RM | **70.0 / 7.2×** | 75.0 / 10.3× | **75.0 / 17.4×** | —           | —               |\n| Full-BoN w/ RM  | **71.7 / 9.8×** | 73.3 / 13.7× | **75.0 / 21.8×** | —           | —               |\n| ST-BoN (ours)   | 68.3 / 1.3×     | 71.7 / 1.5×  | 73.3 / 1.8×      | 73.3 / 2.5× | **75.0 / 3.4×** |\n\n*Key insights:*\n\n* *First*, due to the much longer generation lengths of reasoning models (often >10k tokens) compared to non-reasoning tasks (often <1k), the memory cost of KV cache becomes dominant. This significantly inflates the overall cost of Full-BoN. In contrast, ST-BoN performs early truncation and retains only a single path up to ~10k tokens, keeping its cost only marginally higher than that of greedy decoding.\n\n* *Second*, ST-BoN still exhibits a markedly better cost–performance trade-off. For instance, on Qwen3-8B-Thinking, to reach 73.3 accuracy, **ST-BoN requires only 13.8% (3.2/23.2) of the cost of Full-BoN w/ RM**. Notably, even when ST-BoN uses $N=160$, Full-BoN at $N=10$ remains more expensive while underperforming.\n\n**These results provide strong evidence for the scalability and efficiency of ST-BoN in the context of long-CoT reasoning models.**\n\n> *Further Insight:* To explain this phenomenon, we find support from a recent study [1], which shows that the accuracy of the *first reasoning step* often predicts final success, even when later decoding involves rollback or reflection. This further supports the effectiveness behind ST-BoN’s early truncation design in any LLM paradigm.  \n[1] *Lost at the Beginning of Reasoning*. [https://arxiv.org/abs/2506.22058](https://arxiv.org/abs/2506.22058)\n\n---\n\nWe hope these results satisfactorily address your concerns and underscore the broader significance of our contributions. We look forward to your positive feedback, thanks again for your constructive comments."}]

Author rebuttal:

Meta-review: The paper studies improving the inference efficiency of Best-of-$N$ (BoN) sampling through early pruning. Previous methods for early pruning is mainly based on evaluation from a process reward model, which requires additional compute. The paper proposes a pruning method using self-consistency of the chain-of-embedding (CoE) features. The paper provides justifications on the proposed method by showing that early consistency could indicate consistency in full generation, and hence it could be used as a metric to select promising candidates. The paper empirically demonstrates the proposed method by comparing to full BoN baselines with and without reward models and demonstrate significant efficiency advantage with comparable performance. Reviewers all agreed that the paper provides a novel, principled approach for efficient BoN implementations. Concerns were raised about the coverage of empirical evaluation regarding different models, and baseline methods. The authors addressed these in the rebuttal and the reviewers are satisfied. 

Overall, I believe this is a valuable contribution to the field. I believe the idea of using model’s internal states to perform self-truncating when generating multiple inference traces is interesting idea, which is worth highlighting at the conference. Hence I will would an spotlight acceptance.
Decision: Accept (spotlight)