Scalable Best-of-N Selection for Large Language Models via Self-Certainty
Authors: Zhewei Kang
Xuandong Zhao
Dawn Song
Venue: NeurIPS 2025 poster ()
Keywords: Best-of-N
Reasoning
LLM
Primary area: applications

Abstract: Best-of-N selection is a key technique for improving the reasoning performance of Large Language Models (LLMs) through increased test-time computation. Current state-of-the-art methods often employ computationally intensive reward models for response evaluation and selection. Reward-free alternatives, like self-consistency and universal self-consistency, are limited in their ability to handle open-ended generation tasks or scale effectively. To address these limitations, we propose self-certainty, a novel and efficient metric that leverages the inherent probability distribution of LLM outputs to estimate response quality without requiring external reward models. We hypothesize that higher distributional self-certainty, aggregated across multiple samples, correlates with improved response accuracy, as it reflects greater confidence in the generated output. Through extensive experiments on various reasoning tasks, we demonstrate that self-certainty (1) scales effectively with increasing sample size $N$, akin to reward models but without the computational overhead; (2) complements chain-of-thought, improving reasoning performance beyond greedy decoding; and (3) generalizes to open-ended tasks where traditional self-consistency methods fall short. Our findings establish self-certainty as a practical and efficient way for improving LLM reasoning capabilities. The code is available at https://github.com/backprop07/Self-Certainty

Scalable Best-of-N Selection for Large Language Models via Self-Certainty
Zhewei Kang* UC Berkeley vaynekang@berkeley. edu
Xuandong Zhao
Dawn Song UC Berkeley
UC Berkeley xuandongzhao@@erreeleyy edu
daunsong@berkelley edu
Abstract
Best-of-N selection a key technique for improving the reasoning performance of Large Language Models (LLMs) through increased test-time computation. Current state-of-the-art methods often employ computationally intensive reward models for response evaluation and selection. Reward-free alternatives, like self-consistency and universal self-consistency, are limited in their ability to handle open-ended propose generation tasks or scale effectively. To address these limitations, self-certainty, a novel and efficient metric that leverages the inherent probability distribution of LLM outputs estimate response quality without requiring external reward models. We hypothesize that higher distributional selfancer aggregated across multiple samples, correlates with improved response accuracy, it reflects greater confidence in the generated output. Through extensive experiments on various reasoning tasks, we demonstrate that self-certainty (1) scales effectively with increasing sample size N, akin to reward models but without the computational overhead; (2) complements chain-of-thought, improving reasoning performance beyond greedy decoding; and (3) generalizes to open-ended tasks where traditional self-consistency methods fall short. Our findings establish self-certainty a practical and efficient way for improving LLM reasoning capabilities. The code is available https: //gi thub. com/ http//aapppo00//11111..... Certainty
Introduction
Large Language Models (LLMs) have achieved impressive reasoning abilities, yet reliably pro- ducing accurate outputs for complex tasks often requires techniques to enhance inference-time performance [Wu et al., 2024, Xiang et al., 2025]. Best-of-N selection, generating and selecting from multiple candidate responses, has emerged a powerful paradigm for significantly improving reasoning accuracy [Snell et al., 2024]. Current Best-of-N methods frequently rely on reward models, such as Outcome Reward Models (ORMs) [Cobbe et al., 2021a] and Process Reward Models (PRMs) [Lightman et al., 2023, Uesato et al., 2022], not only for output selection but also for data annotation further refine LLM reasoning capabilities [Uesato al., 2022, Wang et al., 2022].
However, reward models introduce substantial computational and practical challenges. They are computationally expensive to train fine-tune, often requiring many parameters as the L.M itself [Wang et al., 2024a], are vulnerable to distribution shifts, and can suffer from "reward hack- ing" [Eisenstein al., 2023]. While techniques like reward model ensembles [Coste et al., 2023] offer partial mitigation, they further increase overhead.
As giigher--wiiht alternative, self-consistency [Wang al., 2022] aggregates multiple outputs using majority voting. However, is applicable only to tasks with directly comparable string matched
Equul contribution Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

3500
Correct
7000
Correct
Incorrect
Incorrect
3000
6000
No Answer
No Answer
Frequency
2500
Frequency
5000
2000
4000
1500
3000
1000
2000
500
1000
90.0
12.5
15.0 Self-certainty
17.5
20.0
-.0
-1.8
-1.4 Negative Perplexity
-1.6
-1.2
-1.0
Figure 1: Distribution of self-certainty and negative perplexity for correct, incorrect, and no-answer responses on the MATH dataset (I eLeve 4) [Hendrycks et al., 2021] using the with 64 samples per question. For self-certainty, the distributions of correct and incorrect groups concentrate around different central values, with frequencies decreasing smoothly toward both extremes. In contrast, negative perplexity fails clearly separate correct from incorrect outputs and favors no-answer responses, highlighting self-certainty's 's effectiveness in assessing response quality.
answers, limiting its use for differentiating reasoning paths or open-ended tasks. Universal self- consistency (USC) [Chen al., 2023) prompts the L.M to choose the most consistent response, but its gains are constrained by context length and model ability, sometimes declining with larger N [Cobbe al., 2021b], and can be ineffective for small models, our research confirms. Moreover, self-consistency and USC lack direct quality score for responses, limiting their applicability in tasks such candidate ranking.
To overcome these limitations, we propose leveraging the LLM's inherent probabilistic output for practical, general, and robust approach to Best-of-N selection. We hypothesize that an LLM's probability distribution naturally encodes its certainty. We introduce self-certainty, a novel metric quantifying this confidence by measuring the divergence of the predicted token distribution from uniform distribution. A distribution diverging significantly from uniform indicates a more ynd thus more certain-preddictin As shown in Figure self-certainty demonstrates stronger signal for distinguishing correct responses. Notably, it incurs almost no computational overhead, the token distribution is generated alongside the tokens during inference. Inspired by Borda Voting, we enhance self-consistency by incorporating self-certainty-basee ranking, assigning weighted votes based on self- celf-certainty rank using scaling factor of (N ranking + 1)", effectively prioritizing more confident responses.
We rigorously evaluate methods across diverse reasoning benchmarks, including LiveBench-Me Mathh-Mahh-hh [White et al., 2024], GSM8K [Cobbe et al., 2021b], MATH [Hendrycks et al., 2021]. CRUXEval [Gu al., 2024] and LiveCodeBench [Jain al., 2024], spanning mathematical reasoning. code reasoning, and code generation. Our experiments reveal that seff-certainty-basee voting consistently outperforms self-consistency in Best-of-N selection of reasoning tasks, effectively adapting to varying sample sizes and question difficulties.
The key advantages of self- self-certainty
Scalability: Self-certainty scales efficiently with increasing sample size N, mirroring reward models in scalability but without their computational burden.
Orthogonal Enhancement to Chain-of-Thought: Self-certainty complements chain-off of-thouught (CoT) reasoning [Wei et al., 2022], outperforming self-consistency through weighted voting.
Generalizability to Open-Enden Tasks: Self-certainty generalizes effectively open-ended responses (e.g., code) where self-consistency is inapplicable, surpassing greedy decoding and USC.
Related Works
Reward Models for Response Reranking and Selection. Evaluating L.LM outputs with external models like verifiers reward models (ORMs, PRMs) can enhance reasoning and select best

---

samples [Lightman et al., 2023, Wang et al., 2024a]. However, these models are often task-specific sensitive the base model [Eisenstein et al., 2023], and computationally expensive to train, sometimes requiring parameter counts similar to the L.Mss they evaluate [Wang et al., 2024a]. Our approach, self- certainty, avoids additional training by using the L.MM's own logits for efficient quality assessment.
Consistency-Basee Response Selection. Self-consistency [Wang et al., 2022] leverages the model's internal understanding by selecting the most common response from multiple outputs, improving reliability. However, it's limited to tasks with convergent final answers and hard to generalize to open-ended generation. universal self-consistency (USC) [Chen al., 2023] extends more tasks but faces scalability issues and lacks certainty measure, Self-certainty overcomes these limitations by directly measuring response confidence from token distributions, handling open-endes tasks and scaling efficiently.
Confidence Estimation for Model Responses. Various methods estimate model confidence [Geng et al., 2023]. Self-Evaluation [Ren et al., 2023] uses yes/no token probabilities. BSDetector [Chen and Mueller, 2024] measures similarity and prompts for self-verification. TrustScore [Zheng al., 2024] computes likelihood against modified-prompl distractors. These often require multiple evaluations, hindering scalability for Best-of-N selection. In contrast, self-certainty leverages the output token distribution directly, avoiding extra prompts and enabling efficient, scalable selection.
Measuring Confidence of LLMs
This section explores metrics for quantifying LLM prediction confidence, comparing probabilistic measures with distributional identify the most effective for reliable output selection.
3.1 LLM Background
Large Language Models, typically transformer-based (Vassani, 2017], autoregressively generate token sequences y (yj....,, Vm) from input (%,,) At each step i, the model produces logits & E IR (where = [V] is vocabulary size), which convert to a probability distribution p(-|2r, Mee)) E [0, 1] over the vocabulary for the next token y;. This distribution reflects the model's belief about the next token.
Sentence-Level Probabilistic Confidence
Probabilistic confidence quantifies model's certainty in its predictions by directly leveraging the probabilities assigned to sampled tokens.
Average Probabibbbll A common confidence measure is the average log probability (AvgLogP)
of sampled tokens:
AvgLogP := 1 log (p(n Lr wei))
where p(ys (ar, yeci) is the probability of token yi- Higher LogLogP values indicate the model assigns higher probabilities to generated tokens, reflecting greater confidence.
Perplexity. Perplexity is common metric for evaluating language models, defined as the exponen- tiated average negative log-likelihood:
Perplexity = exp
log jpuul yeri]
Since Perplexity = exp(-AvgL.og)) both measures are equivalent when selecting responses. We use negative perplexity for Best-of-N selection, though studies show struggles with long contexts [Hu al., 2024], suggesting the need for alternatives.

---

3.3 Distributional Confidence
Distributional confidence measures consider the entire probability distribution over the vocabulary at each generation step, capturing a more holistic view of the model's certainty beyond just sampled token probabilities.
sentence-level distributional confidence measure can be defined as:
DistributionallCCofideeeee := F(((P)))))
where py/= - (p(- (r), p(-|2r y)...... .p( (a, y-en)) represents the sequence token-level probability distributions, f produces confidence score for each token, and F aggregates these into sentence- level confidence. With output length n, we define F' as the average across all positions:
"
F(C Cx)= Σ C.. C. fp vss))
For function f. we explore metries that quantify how "peaked" or "concentrated" the probability distribution is, with more concentrated distributions suggesting higher model certainty:
Kullback-Leible (KL) Divergence. Drawing upon neural networks Maximum Likelihood Estimators [LeCun et al., 2015], we hypothesize that higher confidence corresponds to distributions further from uniform distribution U (representing maximum uncertainty). KL Divergence quantifies this difference:
1/V Σ log (pilz,
C. -- KL(U pl
log (V [ijl,,ys vs))) (5)
Gini Impurity. Originally introduced in decision trees [Breiman, 2017], Gini Impurity measures the probability that two randomly sampled tokens belong different classes. A more concentrated distribution indicates higher confidence:
V Σ ((())
C Gini
= (c(p(112,
Entropy. Entropy measures the disorder in probability distribution. Higher entropy indicates greater uncertainty, so we use negative entropy as a confidence measure:
C
Entropy
Σ j=1
log(pl/ vs())
Distributional Perplexity (DP). To distinguish from standard perplexity (Equation 2), we denote DP:
We apply negative sign to perplexity to interpret it as confidence.
C. = -xep
Σ
3.4 Our Primary Metric: Self-Certainty
Empirical evaluations (Figure 4) demonstrate that <L-divergence-inppin distributional confidence more effectively distinguishes correct samples from incorrect ones and achieves superior accuracy higher N values. Theoretically, an infinitesimal gradient step that increases the log-likelihood of desired token raises its coken-wise self-certainty whenever that token already carries sufficiently high probability. A detailed proof provided in Appendix A. Consequently, standard training procedures, such as supervised fine-tuning and reinforcement learning, that increase the probability of the desired token also tend to increase its self-certainty, supporting our choice of self-certainty as principled confidence metrics. Based on these findings, we define self-certainty as our primary confidence metric for best-of-N selection:
Self-Certainty
log (V ((())
(9)
nV

---

Negathun Perplecity:
Salf-Certality
Question
Sample
Finst Toloen Scrre:
5.78
Stop
segiment graph ingu/d misppoent
X
Stept Milekan foctor the left hond side os:
41% -)
1.21*
Step2: x, 0,
13,014
Average Boore:
Large Language Model
Sample First Tolic Score:
Given Reasoning
-9
17-539
Step Seore
midpoine (JA ?), Qund ony ingght) givnes
schre for using quadrabic lormulaat (co Aaslj (20)
1:llhagte quardhatic equatimn
1.28-
16.92*
systeme
Subssituting
X, XUG
caaz, 1, -12, and those g20
4/101010/12111
Figure 2: Comparison of reasoning paths solving quadratic equation for the given problem using self-certainty and negative perplexity. Sample factors the quadratic equation directly, while Sample applies the quadratic formula. The figure illustrates an example of how the two measures assign confidence scores each reasoning step, showing that self-certainty distinguishes between correct and incorrect reasoning more effectively than negative perplexity.
Question
Wrong Staps underttandddddd.. nigh quastion sraggamaiiccee...
stor passarts
Saaporos Resarting Coaerti 12. Cab-Cersity (2.3
Salt- www.Cooniattecc.... 10
giin conu eary sareeelphee caars pla for than?
Centainty: 12.5
X
Raspores Reeaaring
Sus-Ceriiyy
12.5
votas
Correct Solution:
GC: vanwe
Responee Resoniazh 12 drnwee
SD: votco
Fyler re pay colar wig.awa.aa.. pal
Figure 3: Example of Borda Voting correctly identifying the answer when confidence-driven selection and self-consistency fail. The figure illustrates how Borda Voting aggregates confidence scores and ranks to select the correct answer.
Cross entropy between the predicted distribution and uniform distribution provides an equivalent confidence measure, differing from KL-divergence only by constant, The self-certainty based cross-entropy is:
Self-Certainty (CE)
log(p(i a, (.)))))
(10)
nV
3.5 Analysis
Reward Models (PRMs and ORMs) typically evaluate responses using the minimum reward across all reasoning steps [Lightman et al., 2023, Wang et al., 2024a], prioritizing error detection over progress assessment. Self-certainty methods effectively identify mistakes through averaging because early errors propagate, reducing confidence in subsequent steps. As illustrated in Figure 2, when sample I contains an initial error, self-certainty assigns lower confidence to all following steps despite their correctness, while negative perplexity fails to distinguish between reasoning paths following correct versus incorrect premises. Additionally, distributional confidence detects correct reasoning from the first few tokens.
4 Self- Celf-Certainty with Voting Method
While self-certainty demonstrates greater robustness than alternative confidence measures, it remains vulnerable to distortion from samples with artificially high confidence scores. Our analysis reveals that seff-certainty-drivee Best-of-N selection underperforms compared to self-consistency in accuracy on mathematical datasets with definitive answers when using identical N values (Table 1). This does not, however, indicate inherent inferiority. Self-consistency operates at the response layer of LLMs.
5

---

while self- certaiity aggregates information at the decoding layer. By integrating both layers, we can extract more reliable responses from multiple outputs with explicit answers.
Traditional methods of combining majority voting with score-based selection, such as summing scores across samples with identical answers, suffer from sensitivity to score scaling. Similarly, using average confidence may inadequately represent frequently sampled answers. To address these limitations, we propose a Borda count-inspired approach:
First, we rank N outputs of models by confidence, obtaining a ranking [T1, "2,.2,..,,77 We then assign votes to these ranked outputs using the following formula:
v(r) = (N - - 1)
(11)
where is the rank of the output (1 < S N). Each valid response contributes votes to its final answer proportional to its rank. The answer accumulating the highest vote total becomes the consensus selection. When p O, Equation (11) reduces to simple majority voting. As approaches infinity, the highest-ranked output dominates, reverting to pure distributional confidence selection.
Figure illustrates how Borda Voting successfully identifies the correct answer by integrating both confidence ranking and answer frequency, thereby overcoming limitations of both confidence-driven selection and self-consistenee The parameter p. which controls ranking influence, serves as tunable hyperparameter discussed in Section 6.2.
5
Experiment Setup
We compare various confidence measures for selecting reliable reasoning responses, extending evaluation to additional datasets and exploring self-certainty with voting methods.
5.1 Comparison of Confidence Measures
To evaluate confidence formulations from Section 3, we select the most confident response from N outputs generated by Llama-3.1 1-8--...---B--Isstruee [Dubey al., 2024]. We use LiveBench-Mait dataset [White et al., 2024], released post-model deployment, to mitigate potential data contamination.
Accurscy too
20
15
We sample 64 responses (temperature=0.6, top- p=0.9) and create subsets of N = 4, 8, 16, 32, 64 for Best-of-N selection. All measures identical sample sets. Responses without extractable answers masked. We include FirstAns baseline that selects the first extractable answer from N out- the ZeroEval framework [Lin, 2024], with results averaged across five repetitions. All experiments are run on NVIDIA A100 GPUs.
2²
evaluated on
[Nummbr Candidate Besporsses)
Figure Best-of-N selection accuracy LiveBench-MATH across multiple confi- dence measures. KL achieves the best per- formance at larger N, while other measures plateau or decline after N 16.
puts. Evaluation
5.2 Validation on Additional Datasets and Combined Voting Methods
We evaluate self-certainty and Borda Voting against self-consistency, universal self-consistency (USC), greedy decoding, and FirstAns across diverse reasoning tasks. We also compare to an outcome reward model model 1iwee225-MatthPPMM--M [Zhang al., 2025] on selected benchmarks highlight differences between internal selection and external selection, and to quantify the resulting accuracy gaps.
[Ray2333, 2024] and a process reward
The sampling strategy follows the procedures outlined in Section 5.1. For USC, we use the template from the original paper [Chen et al., 2023] (with minor wording modifications, as shown in Appendix C.2). To ensure fair comparison, we assist USC in selecting the first valid response when it fails to choose one with an extractable answer.
We evaluate different methods using the Llama-3, 3.1 1-8B-Instruc across the following benchmarks:

---

4230 2H00E
RGG 1004
5202 18000
1000 1200 18000
4202
1208
JAN0
Figure 5: Scatter plot showing various confidence measures against response length (measured in number of characters) in the SiveBench-Malee dataset, using the 1aama-3.1-8B-Imm. model with 64 samples per question. The figure demonstrates that, with the exception of self-certainty, all other measures exhibit bias towards longer responses.
Mathematical Reasoning: We utilize the LiveBench- Mathh dataset [White et al., 2024]. the validation set of GSM8K dataset (Cobbe et al., 2021b] and the test set of MATH dataset [Hendrycks et al., 2021].
Code Reasoning: The CRUXEval-O benchmark [Gu et al., 2024] is employed, which involves predicting the output of Python codes.
We adopt the LiveCodeBench code generation benchmark [Jain et al., 2024] to assess the improvements introduced by our methods. Note that this is an open-ended task where self-consistency cannot be applied.
Code Generation:
For all test models and datasets, we employ Chain-of- ThoThutht reasoning [Wei et al., 2022], except for the code generation dataset. To evaluate the generalization of our measure across different training methodologies, particularly for the recent R R-series large reasoning models [Guo et al., 2025], we test our approach on DeepSeek-FR -ii----sill-ll--la------ using the MATH dataset (Level 3). Given the increased reasoning time required by this model, we conduct a single trial for this experiment. To further validate and assess generalizability. we apply both USC and self-certainty to the 20wen-2.5-Coder-3 IIIstrutt model [Hui et al., 2024], in addition to Llama-3 1-8B-I-Im SI--Instrct, the LiveCodeBench dataset.
Results and Analysis
6.1 Self-Certainty
KI-DDivegencce-nssirred Distributional Confidence Outperforms Other Measures in Best- of-N Selection. Figure 4 shows distributional confidence measures outperform perplexity when N 2 16. KL divergence uniquely continues improving as N increases to 32 and 64, demonstrating its robustness as confidence measure with superior insight into response accuracy. Self-certainty, defined in Equation as KL divergence from a uniform distribution, generalizes better than alternative empirical distributions (evaluated Appendix B.4), confirming the efficacy of original design.
Self-Certainty's Robustness to Reasoning Length in Response Selection. Figure 5 reveals a critical insight: while most confidence measures show positive correlation with response length, self- certainty remains largely invariant to reasoning length. This confirms Basu et al. [2020]'s observation that perplexity decreases with increasing output length under low p values. Unlike metrics that potentially conflate verbosity with correctness, self-certainty provides an unbiased assessment of response quality, preventing models from artificially inflating confidence through extended but potentially meaningless reasoning.
Self-Cene Self-Certainty Effectively Separates Correct and Incorrect Responses. Analysis of self-certainty and negative perplexity distributions across correct, incorrect, and no-answer responses on MATH dataset Level 4 (Figure 1) demonstrates self-ccrtaintyy' superior discriminative power. For self- certainty, the distributions of correct and incorrect responses are centered around distinet means, with frequencies tapering off smoothly toward both tails. In contrast, perplexity fails to distinguish between correct and incorrect responses when applied to the full dataset, despite performing adequately at small values (Figure 4). This aligns with Zhang al. [2020]'s s finding that response quality initially improves as perplexity declines but subsequently deteriorates significantly. Notably, perplexity

---

Table 1: Performance comparison of various methods across different datasets using Llama-3.1 3.1-B- Instruct. Some USC results are omitted due to over 20% of the data exceeding context window limits under the settings. Self-certainty consistently outperforms sampling, greedy decoding, and perplexity, while Borda Voting with the optimal parameter delivers the best performance across all methods.
Live
GSMSK
MATH
CRUXEval-O
Method
Avg.
64
N
64
Greedy
12.23
84,00
47.96
39.88
46.02
FirstAns
17.66
17.66
82.08
82.08
49.08
49.09
42,93
42.93
47.94
PRM
/
93.48
95.15
47.53
48.61
ORM
88.57
89.91
42.00
39.62
Perplexity
20.44
18.32
87.01
87.81
53.34
51,96
44.67
45.10
$1.08
USC
21.08
87.32
85.65
54.66
43.78
41.25
51.19
Self-consistency
22.50
26.25
89.42
90.99
58.60
63.40
47.58
50.42
56.15
Self-certainty
20.87
22.01
87.32
88,90
54.63
56.70
45,38
45.83
52.71
-Borda (p = 01.3)
23.69
26.47
89,57
91.07
59.04
63.60
47.94
50.42
56.48
Borda (p 0.7)
23.59
26.36
89.51
91.04
59.04
63.85
47.85
50.65
56.49
BBorda (p = 1.2)
23.21
26.69
89.51
90.95
58.86
64.10
47.93
50.85
56.51
Borrda (p = 2.0)
22.45
26.41
89.13
90.90
57.94
60.02
47.25
51.23
55.67
25.0
Fixsra
Greedy
ceeatyy
Traddro
Giredy
Frssta
drenty
Bosta
Serdu
Bonda 1pe011
46
Zonta 20w0
12.5
Figure 6: Performance evaluation across four datasets using different strategies with Llama-3.1-8B- Instruct. The lines show strong scaling ability of both self-certainty and Borda voting.
assigns higher confidence to no-answer responses- often resulting from self-repetition early stopping-whine self-certainty reliably assigns these responses lower confidence scores. This behavior is consistent with Basu et al. [2020]'s observation that maximizing perplexity increases self-repetition. These findings provide compelling evidence that self-certainty more effectively measures model certainty by correlating more closely with response quality.
6.2 Self-Certainty and Voting
Borda Voting in Combination with Self- Self-Certainty. As discussed in Section 4, self-certainty can be integrated with voting methods to enhance accuracy when responses contain explicit answers. Table 2 demonstrates that saff-certainty-basee Borda voting outperforms majority voting, average self-certainty, and sum self-certainty on the MATH dataset.
Performance Comparison Across Four Datasets. Figure 6 illustrates the scaling properties of self- certainty and reff-certainty-base Borda voting. Self- certainty significantly outperforms sampling, greedy decoding. and perplexity-basee selection, with perfor- mance improving as N' increases. This confirms that self-certainty effectively measures the model's 's confi- dence in its responses, providing valuable insight into output correctness. Furthermore, Borda voting con- sistently outperforms self self-consistency across various settings of p and N on all four datasets, indicating that self-certainty enhances anwwer-eeed vot- ing by providing effective ranking information.
Table 2: Accuracy of different voting methods on the test set MATH dataset using Llama- 3.1-8B- ------nnttruc.. Self-certainty-baste Borda voting outperforms other voting methods.
Method
N 8
N 64
Majority
58.60
63.40
Average
46.92
32.94
Sum
59.06
63.51
Borda (p 0.5)
59.08
63.71
Borda (p 1.2)
58.86
64.10

---

Reuds prosma Ld8
londs of 5
144
21:23
H--2
1411
16:4
p -itesePernddee
Figure Performance of Borda voting on MATH dataset using p and N. Accuracy initially increases with p. peaks, then declines. The optimal p varies with N. Note that self-consistency corresponds to Borda voting with O.
Figure 8: Comparison of self-certainty and USC on the LiveCodeBench code generation task. The results show that self-certainty outperforms USC and greedy decoding on both Llama-3.1- 8B-Instruct and Qwen-2.5-Cooder3225, els, with performance improving as increases.
with varying
mod-
Optimizing the Borda Parameter p for Different N. Figure shows the relationship between the Borda parameter in Equation 11 and selection efficiency across varying sample sizes N. The optimal p increases from 0.5 1.2 as N increases from 64, suggesting that larger sample sizes require stronger control from self-certain eeff-certainty. For practical applications, grid search remains the most effective approach for determining the optimal p. though simple heuristic is to use p = 0.3 when N < 16 and p = 1.2 when N 2 32, with the understanding that the optimum is model- and dask-dependent. Tuning p on small portion of the task-specifie validation set recommended to achieve the best performance, as convergence of optimal p typically occurs within few hundred evaluation queries.
6.3 Generalization
Generalization of Self-Certainty Open-Ended Generation Tasks. Self-consistency faces limitations with creative, open-ended tasks like code generation, where each sample produces unique answers, defaulting standard sampling. Both USC and our self-certainty method address this limi- tation. Comparing these approaches on LiveCodeBench (Figure 8), we find that USC underperforms greedy decoding on Llama-3. 1-8---Instrttt likely due to limited consistency recognition capabilities. This is confirmed by results from the larger Qwen model, where USC successfully outperforms greedy decoding. In contrast, self-certainty consistently outperforms greedy decoding across both models and surpasses USC on Qwen-225-Cooeer-322-- with performance scaling positively with sample size N.
Generalization of Self-Certainty on Rea- soning Models. Recent work DeepSeek- R1 [Guo et al., 2025] shows that reinforce- ment learning with verifiable rewards and long- chain-of-the chain-of-thought (CoT) significantly enhance LLM reasoning capabilities. Our evaluation of self-certainty on -eepSSee--1-Distllll -lma------aa- 8B (Table 3) demonstrates that it consistently outperforms both greedy decoding and sampling, with performance scaling with N. Addition- ally, Borda voting with self-certainty surpasses self-consistency when using appropriate p val- ues. These results confirm the robustness of our methods across various fine-tuning approaches.
Table 3: Accuracy various methods on the Level
3 test of the MATH dataset using DeepSeck-Rt Distill-Llama-88 8BB (single trial). Self-certainty out- performs Greedy and FirstAns, while Borda Voting with an appropriate p surpasses self-consistency.
Method
N
N
16
=
Greedy FirstAns
77.54
77.54
77.54
81.17
81.43
81.43
Self-consistency
83.64
86.47
87.62
Self-certainty
83.29
83.73
84.08
Borda (p = 0.3)
84.79
87.00
87,80
Borda (p = 0.7)
84.70
86.91
87.62
Borda (p 1.2)
84.62
87.00
88.06
Borda (p 2.0)
83.29
87.00
87.98
7 Discussion and Future Research
Our study establishes self-certainty as a scalable, lightweight, and effective metric for evaluating LLM outputs, particularly for open-ended and complex reasoning tasks. While it scales well with increasing sample size and outperforms existing reward-free methods across multiple settings, several directions for refinement remain.

---

First, self-certainty can underperform self-consistency on problems with definitive, convergent answers (Section 6). This reflects the complementary nature of different aggregation methods rather than limitation. Combining self-certainty with answer-level voting mechanisms -uchh as Borda voting-bridges this performance gap, achieving results that rival or exceed self-consistency. These findings suggest that self-certainty could enhance reward model design by shifting from token-level scoring to distribution-awarr confidence estimation. The use of KL divergence from a uniform distribution offers greater robustness than traditional average jog-probability metrics and may lead to more stable reward training objectives. In practical terms, the metric can substitute for selected- token probabilities in applications such as soft self- ceff-consittency [Wang et al., 2024b], with potential performance gains.
Second, our implementation uses simple averaging strategy for aggregating token-level confidence (Equation 4) and basic power function for distributing votes in Borda voting (Fquation 11). Future work should explore more sophisticated aggregation functions or data-driven approaches for learning optimal vote weighting schemes to improve accuracy in specialized applications.
Self-certainty also enables broader research opportunities. Its computational efficiency makes it ideal for test-time optimization techniques [Snell et al., 2024], producing higher-quality outputs without additional inference passes. It offers potential value in data filtering, auto-labeling, and reinforcement learning pipelines [Bai et al., 2022, Ouyang et al., 2022]. where confidence estimation is crucial. Specifically, self-certainty could guide reward shaping or provide intrinsic signals for autonomous agents, better aligning learning objectives with model certainty.
8 Conclusion
In this paper, we introduce self-certainty and self-certainty-baased Borda voting as novel approaches for evaluating and enhancing model response performance. Self-certainty functions as an internal measure of response quality, demonstrating robustness in several key aspects. Compared to traditional scoring methods, such average log probability and perplexity, it offers superior scalability when applied to Best-of-N selection. Additionally, the ranking information provided by self-certainty improves chain-of-thougee
reasoning and outperforms universal self-consistency (USC) in code generation tasks. Its stability, flexibility, and generalizability make it applicable across a wide range of domains, with the potential to enhance the autonomous learning capabilities of LLMs.
References
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00722 2024.
Violet Xiang, Charlic Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, et al. Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though. arXiv preprint 2:XXx:2501.04663 2025.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 2rXi::2408.033314 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021a.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint urXiv:2305.20055 2023.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14225. 2022.
Xuezhi Wang, Jason Wei, Dale Schaurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint ar X20v 2022.
10

---

Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms stee-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 9426-9439, 2024a.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping herd- ing? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09924 2023.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.0274: 2023.
Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language 2023.
model generation. arXiv preprini arXiv:2311.1777
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchrr Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021b.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination- free I1m benchmark. arXiv preprint arXiv:2406. 19314, 2024.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0033: 03874, 2021.
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.0306, 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07977 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thoouhh prompting elicits reasoning in large language models. Advances in 2022.
neural information processing systems, 35:24824 25:24824-24837, 24837,
Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of confidence estimation and calibration in large language models. arXiv preprint arXiv:2311.08298 2023.
Jie Ren, Yao Zhao, Tu Vu, Peter Liu, and Balaji Lakshminarayanan. Self-evaluation improves selective generation in large language models. In Proceedings on, pages 49-64. PMLR, 2023.
Jiuhai Chen and Jonas Mueller. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3186-5200, 2024.
Danna Zheng, Danyang Liu, Mirella Lapata, and Jeff Z. Pan. Trustscore: Reference-free evaluation of Ilm response trustworthiness. arXiv preprint arXiv:2402.12545, 2024.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Yutong Hu, Quzhe Huang, Mingxu Tao, Chen Zhang, and Yansong Feng. Can perplexity reflect large language model's ability in long text understanding? arXiv preprint arXiv:2405.06105, 2024.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521((553):436 444, 2015.
Leo Breiman. Classification and regression trees. Routledge, 2017.
11

---

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Bill Yuchen Lin. ZeroEval:: A Unified Framework for Evaluating Language Models, July 2024. URL http:s//gggtaabb..om///adaaaaa//
Ray2333. G
httpss//hugginggaacc co Ro/Raay333//RM-L2 Llama3.
2- --BB-rewardmdde--ftt 2024. Hugging Face model repository, Apache-2.0 license.
Zhenru Zhang, Chujic Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu. Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.0733 073011, 2025. doi: 10.48500/rxxi.25000073000 URL
https://arxiv..
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.1294 2025.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.124
2024.
Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R Varshney. Mirostat: A neural text decoding algorithm that directly controls perplexity. arXiv preprint arXiv:2007.1496. 2020.
Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. Trading off diversity and quality in natural language generation. arXiv preprint urXiv:2004. 10450, 2020.
Han Wang, Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. Soft self-consistency improves language models agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 287-301, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653/ achosst URL https: httpa/:///caaatholggoee org org/202// 2024. acl sall-shor..288 28/.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli. Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint wrXiv:2204.05822 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
12

---

A Theoretical Analysis
A.1 Log-likelihood Ascent of Target Token Increases Self-Certainty
Setup and definition. For a single decoding step, let = (P1, (
pv) E A denote the next-token distribution over vocabulary of size V with logits E RV and p softmax(z). The token-wise self-certainty is
V Σ
SC(p) =
log Pp),
Differentiate Self-Certainty.
Let 6ys be the Kronecker delta. Two standard derivatives are 0p Pý p9gp
OSc
pk),
Öpj
V
Pj
By the chain rule,
aSc
OSC Opj
1
Pj p2(0Ex - pk.) = -p) - p -
(I)
0z
Opj
Ozk
1=1
Update that Increases Specific Token's Probability. Fix a target token index yf A gradient- ascent step on Py- moves logits along
i blogPo Pv ŌK. Pk+
Ark x
(2)
02k
so for step size > 0 we consider =(1) = Az and p(r)) coftmax(=(n)))
Directional Change of Self-Certainty.
Using (1) (2), the directional derivative of SC 17
0s
d scc(p(n) drj
V
OSC
A
E(p
Pk
+ (ok.a - -Ph) = -Py-
Σ
V
2
(3)
Ozk
7=
k=1
A=1
k=1
Let 1|p12 = Σx ph. Then (3) gives the exact criterion
d sc(p(n)) dn
>0
> 0
Py-
(4)
Theorem A.1 (When self-certainty increases under likelioeldoood ascent). Under the self- ceffacertannty SC(p) KL(U)P), a (stochastic) gradient-ascen audient-ascenttee step on log Py- increases the token token-position self-certainty first order in the step size if and only if Py ||
Py-
Proof. Immediate from (3)-(4).
Corollary A.2 (Argmax case). If = arg maxe Pt and p not one-hot, then Py- >> 1lell2 and thus the ascent step strictly increases SC. Indeed, |lpll3 S max; Pt with equality only when p one-hot.
Remarks. (i) If Py- starts below IIIIi2. the first few ascent steps can decrease self self-certainty. Once Py- exceeds !|plž. further ascent increases it by Theorem A.1. (ii) Sentence-level self-certainty inherits the same monotonicity condition token-wize.
B
More Experiment Results
B.1 Oracle Best-of-N Selection Performance and Scaling Effects on LiveCodeBench
In our experiment described in Section 5.2, we evaluate the performance of Llama-3. 1.-.-...18-B--nuuue and compare Borda voting and self-certainty against the upper bound of Best-of-N selection methods, as shown in Figure 9. While both methods demonstrate continued improvement as N increases, they remain significantly outperformed by the Oracle selection method, which assumes perfect knowledge of the correct answer.
13

---

LweBenchaMeth
GSH
Oracie
Gradle
Oreede
Oraele
Histard
HistArs
Prsslees
Sep-cerraity
bords upwr)
Salf contalitty
Bords (awil3))
Self cortantyy
Barii 20wdl
Value
Burta Ipwrt 30
Salf canta moy
Wine
Gready Sanreling Self-omaane
22
50
Figure 9: Performance across five datasets using different strategies with (amma-3.188B 3.I-tBtIntt The oracle selection method significantly outperforms the other strategies. Additionally, both Borda voting and self-certainty demonstrate strong scaling effects.
Instruct.
B.2 Average Self-Certainty Across Difficulty Levels on MATH Dataset
To explore how self-certainty is influenced by question difficulty, we evaluate the average self- certainty score across different difficulty levels of the MATH dataset, shown in Figure 10. The results indicate that the average self-certainty generally decreases as the difficulty level increases, regardless the correctness of the questions. This trend makes self-certainty a promising parameter-free approach for assessing question difficulty, offering a potential alternative to training classifiers [Snell al., 2024] when determining difficulty levels for scaling test-time compute strategies.
17.8
Incorract
17.6
17.4
Self-certainty
17.2
17.0
16.8
16.6
2
Difficulty Level
Figure 10: Comparison of the average self-certainty score the MATH test dataset across increasing for 64 responses per question (single trial). The average self-certainty decreases questions become more challenging. This trend is observed for both correct and incorrect responses.
difficulty levels using Llama-3.
B.3 Evaluation Methods Across Difficulty Levels on the MATH Dataset
We evaluate different methods across varying difficulty levels of reasoning problems. Figure 11 presents the performance of various methods on the MATH dataset at different difficulty levels. As question difficulty increases, the scaling effect of Borda voting and self-certainty becomes more pronounced, demonstrating their effectiveness in handling more challenging reasoning tasks.
B.4 Replacing Uniform Distribution with Empirical Distribution
In Equation we define tokenwise self-certainty as the KL divergence between generated token distribution and a uniform distribution, which quantifies deviation from random sampling. An alternative approach replaces the uniform distribution with an empirical token distribution estimated from training data. To evaluate the impact of this modification, conduct the following experiment.
14

---

Methodk
Borda p p
80
Self-certainty
FirstAns
MATH Test Accuracy (%)
60-
40
Levell
Level2
Level3
Level4
Level5
Difficulty Level
Figure 11: Comparison evaluation methods on the MATH test dataset across increasing difficulty levels using Llama-3.1-8B-1 -nnstruutt The four bars in each difficulty bin correspond to increasing choice N in the Best-of-N selection (8, 16, 32, and 64 generations). Performance differences among settings become more pronounced the difficulty level increases.
Token frequency Distibution
107
Prouarcy
Figure 12: Frequency of the top 20 tokens in Llama-3.1-88 3-1--8B-IIsstuue responses to MATH training set questions (generated eight times per question).
We first estimate token frequencies in the MATH training by generating eight responses per question and averaging token occurrences. The resulting empirical distribution is approximated from these frequencies, with the 20 most frequent tokens shown Figure 12. We then compute KL divergence between the generated token distribution and the empirical distribution, using this as the self-certainty measure for Best-of-N selection. This experiment was conducted for single trial, with all other setup parameters as described in Section 5.1.
Results show that replacing the uniform distribution with the empirical distribution has minimal impact on MATH test accuracy but leads to a noticeable performance drop on GSM8K, suggesting sensitivity to distributional shifts. Thus, recommend retaining the uniform distribution in Equation for improved generalization.
B.5 Comparison Voting Methods with Different Confidence Metries
We further compare the voting method using self-certainty with variants that rely on other confidence metrics. The results, presented in Table 5, show that for large values of N, self-certainty consistently
15

---

Table 4: Accuracy of various self-certainty definitions for Best-of-N selection on the MATH and GSM8K test sets using 1--ma--.--B-Inssteee (single trial). The empirical distribution is derived by sampling from the MATH training dataset. While the empirical self-certainty results are comparable those based on a uniform distribution for the MATH test set, significantly outperformed by the latter, likely due to a distributional shift.
MATH N
GSM8K N
Base Distribution
N 64
N 64
Uniform
54.60
56.46 56.78
87.19 85.97
88.55
Empirical 54.70
86.35
outperforms the alternatives when the Borda exponent is properly tuned. This highlights the robustness of self-certainty relative to other metrics.
Table 5: Accuracy of Borda voting methods on the test set of MATH-levee5 test set (single trial) using -Jama-3.1-8B-Intteee
Method
Perplexity
Self-certainty
(D) Entropy
(D) Perplexity
(D) Gini
Self-consisenncy
37.99
37.99
37.99
37.99
37.99
Borda (p- = 0.5)
38.60
38.90
39.12
38. 14
39.35
Borda (p- 1.2)
38.60
39.43
38.40
38.
38.75
B.6 Comparison with Normalized Weighted Sum from Self-Consistency
The normalized weighted sum (NWS) variant of self-consistency [Wang al., 2022] combines both voting and model logit information by summing the sentence-level average log probabilities of groups of completions that yield the same final answer. However, the authors show that incorporating sample token probabilities in this manner generally does not improve the standard voting-base based self-consistency and sometimes underperforms the vanilla method.
For completeness, we compare our Borda voting approach with the normalized weighted sum version of self-connee cofsistnsst the MATH-Leved test set using Llama-3.1 3.1aa-3.---81 1-8B-Inntruee As shown in Table 6, our method achieves higher accuracy, further validating the effectiveness of both the self-certainty metric and the Borda voting framework.
Table 6: Comparison between self-consistency (normalized weighted sum) and standard self- consistency MATH-Level5 with 1-taaa,..----IItttttee using N 64 samples (single trial). Borda voting with an appropriate exponent outperforms the normalized weighted sum variant.
Normalized Probbbbiity
Weighted-Sum
Self-Certainty
Borda (p=2.0)
Accuracy
27.34
38.75
31.42
39.43
C
Case Study and Examples
In this section, present several examples to illustrate the practical behavior of self- self-certainty and universal self-consistency (USC). These examples are drawn from Llama-3.1 1-B.3.1--B-Inntee Instruct generating responses to questions from the GSM8K and LiveCodeBench code generation datasets.
C.1 Self-Certainty
In this example, self-certainty correctly assigns higher confidence to the correct answer. Notably, in Response 0, the model does not fully adhere to the instruction to enclose all reasoning within the specified JSON field. Instead, follows trained reasoning process first and then summarizes reasoning path in the reason field. This approach appears to be compromise between the model's training objective and the given instruction, balancing structured formatting with natural reasoning process.
16

---

Self-Certainty Example
USER:
## Question:
Kylar went to the store buy glasses for his new apartment. One glass costs $5, but every second glass costs only 60% of the price. Kylar wants to buy 16 glasses. How much does he need pay for them?
## Instruction:
Please answer this question by first reasoning and then providing your answer. Present your reasoning and solution in the following json format. Please show your final answer the answer field, e.g., "answer": "42". 'json( "reasoning": "answer": "_")
Response O:
ASSISTANT:
##Step Calculate the cost of the first glass. The cost of the first glass is $5.
##Step 2: Calculate the cost of every second glass. Every second glass costs 60% of the price of the first glass, which is 0.6 x 5 dollars. ##Step 3: Determine the number of glasses that are full price and the number glasses that discounted.
Kylar wants to buy 16 glasses. Since every second glass discounted, we can divide 16 by get the number of full-price glasses and the number of discounted glasses. 16 full-price glasses and discounted glasses.
##Step Calculate the total cost of the full-price glasses.
full-price glasses at $5 each 8 x 5 = 40 dollars.
##Step Calculate the total cost of the discounted glasses.
8 discounted glasses at S3 each is 8 x 3 = 24 dollars.
##Step Calculate the total cost of all the glasses.
The total cost is the sum of the cost of the full-price glasses and the cost of the discounted glasses. $40 + $24 $64.
##Sttpp 7: Determine the total amount Kylar needs to pay for the glasses. The total amount Kylar needs to pay is $64.
The final answer is: ("reasoning": "Kylar wants to buy 16 glasses. The first glass costs $5, and every second glass costs 60% the price the first glass, which is $3. He buys 8 full-price glasses at $5 each and 8 discounted glasses at $3 each. The total cost of the full-price glasses $40, and the total cost of the discounted glasses $24. The total cost $40 + $24 $64.". "answer": "64")
(Self-certainty: 17.81)
Response I:
ASSISTANT:
("reasoning": "To find the total cost, we need to calculate the cost the first glass and the cost of the remaining glasses. The first glass costs $5. The second glass costs 60% of $5, which is 0.6* 5 = $3. The third glass costs 60% of $3, which is 0.6
$1.80. We can that the price of each subsequent glass is 60% of the previous glass. So, the price of the To find the cost of 16 glasses, we (1 - r" /1 -)), where is the first term, r is the common ratio, and is the number of terms. In this case, a a= 5, r= 0.6, and 16. Plugging the values, we get S 5 (1 -..66 6) 0.6). Calculating the value of 16 we get 0.0000655. Now, we can plug this value back into the formula: 0.6) 5 0.9999345/0.44 = 2.499335 12.496675. Rounding to two decimal places, the total cost is $12.50.", "answer": "12.50"] (Self-certainty: 17.13)
glasses follows a geometric sequence: 5. 3, 1.80, 1.08, can use the formula for the sum of geometric series: S
5= (1 0.0000655)/(1
17

---

C.2 Universal Self-Consistency
The first example is taken from the GSM8K dataset, while the second comes from the LiveCodeBench dataset. The prompt has been adapted from Chen et al. [2023]. We observe that in Example 1, the model evaluates the answers step by step to assess consistency. In contrast, the model jumps directly to the conclusion in Example 2. This difference may stem from the clarity of the task: in MATH problem with a definite answer, the model can logically assess each step of the reasoning process, as in Example 1. However, for the LiveCodeBench example, where the task involves determining the most consistent code, the model struggles to elaborate on the reasoning process and relies on direct reasoning arrive conclusion.
USC Prompt Example 1
USER:
I have generated the following responses to the question: Elise is learning to write and decides to keep re-writing the alphabet until she knows it. She writes it in full twice, writes half once, then re-writes everything she has already written. How many letters has Elise written in total?
Response O: ("reasoning": "To find the total number of letters Elise has written, first need to determine the number of letters in the alphabet. There are 26 letters in the English alphabet. Elise writes it in full twice, which means she writes 26 letters twice, 26 2 52 letters. Then, she writes half of it once, which is 26 / 2 = 13 letters. So, the total number of letters written far is 52 13 = 65. Finally, she re-writes everything she has already written, which is 65 letters. Therefore, the total number of letters Elise has written is 65 65 130." "answer": "130")
Response 3: ("reasoning": "To solve this problem, we need to calculate the total number of letters Elise has written. First, she writes the alphabet twice, which has 26 letters. So, she has written 26 26 = 52 letters. Then, she writes half of the alphabet once, which is 13 letters. So, she adds 13 more letters to the total. Now, she re-writes everything she already written, which means she re-writes 52 + 13 = 65 letters. To find the total number of letters Elise has written, we need add the number of letters she wrote initially (52) and the number of letters she re-wrote (65). The total number of letters is 52 + 65 117.", "answer": "117"]
Evaluate these responses. Select the most consistent response based majority consensus. Format your answer as "The most consistent response Response X" (without quotes).
ASSISTANT:
To evaluate the responses, will analyze the answers and reasoning provided by each re- sponse. Respone and Response have the same answer (130) and similar reasoning. ... The most consistent response Response
USC Prompt Example 2
USER:
I have generated the following responses to the question: Slavic preparing a present for friend's birthday. He has array of digits, and the present will be the product of all these digits. Because Slavic a good kid who wants to make the biggest product possible, he wants to add to exactly of his digits. What is the maximum product Slavic can make? Input:
18

---

The first line contains single integer (1 S S 10*) - the number of test cases.
The first line of each test case contains single integer n (1 9) the number of digits.
The second line of each test case contains space-separated integers a, (0 - 0g the digits in the array.
Output:
For each test case, output single integer - the maximum product Slavic can make, by adding exactly one of his digits.
Sample Input: Sample Output:
Response 0:
"python
YOUR CODE HERE import sys
def max_product arr):
min_digit min(arr)
max_product 1
for i in range(n): if arr [i] == min min_tigit:
arr[i] +=
break
for num in arr: max_product *= num return max_product
t = int (input ())
for in range (t)
n int (input ())
arr list (map (int, (opuu().sppitttt. print (max_product arr))
<*.
This code first finds the minimum digit array and increments by 1. Then, it calculates the product of all digits in the array. The time complexity is O(n), where is the number of digits the array.
Evaluate these responses. Select the most consistent response based on majority consensus. Format your answer as "The most consistent response Response X" (without quotes).
ASSISTANT:
The most consistent response is Response 0.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Sectioin
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
As shown in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have all the information needed for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification:
We open source the data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
wwwpammssssooooliooooooooooooooe icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //iips.cooc wwtpppcicggguupaaecccceeeooooaet httpa//gggiieee/eeeuubbbiiiii CCodeSubeiieeeeeee ieaibbmmssionPPae seiiinyylcc)) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and text details in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We use five trials to reduce the variance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include such information in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https butps://nnet http///eeurpp..eee cc
Answer:
[Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss potential impacts in Section 7.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
23

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't release data models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The code, data, and models used in the paper are properly credited and comply with the relevant licenses and terms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
25

---

Answer:
[NA]
Justification:
The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: btttps://neurbaee //t.uripss.p./eeee cc/confereecces22255////// for what should or should not be described.
26


Reviews:
[{"review_id": "zWJumfK1OF", "reviewer": "Reviewer_f3GF", "summary": "The paper discusses existing techniques in best-of-N selection in LLMs, and introduces *self-certainty* — namely, the empirical KL divergence from the uniform distribution over the tokens dictionary to the corresponding predictive conditional probability of the model averaged over the generated output — as a scalable metric to improve predictive performance in the absence of external rewards. Authors empirically showcase that self-certainty scales better with more samples, is free from output length biases, and suitable for open-ended generation, demonstrating better performance from competitive methods via the crucial addition of a Borda voting scheme. Experiments are carried out on mathematical and code reasoning datasets through the use of open-source 8B models.", "questions": "* The voting scheme seems to be boosting significantly the self-condifence based performance, however this add-on is not considered for the other baselines: Have authors experimented with adding the voting scheme to these as well? \n\n* Why do baselines fail to improve with sample size N≥16 as showed in Fig.4?\n\n* Another missing dimension is to examine how the performance of the new metric correlates \nwith the calibration of the LLM. Are distributional divergence based metrics better candidates for best-of-N selection in miscalibrated models, or do they get more easily fooled by overconfident output as hinted in Section 4? \n\n* The message of Fig. 2 is unclear to me: how can the displayed numbers be compared across the two methods? Is there a way to normalize the considered metrics so that they are legitimately comparable?\n\n* While a convincing case is made towards the advantages of the introduced metric in analysing long output, it's unclear what are the limitations of self-certainty when facing short output. Can multiple-choice QA benchmarks be included in the experimentation section? Can voting successfully address distortion due to high confidence scores for small number of outputted tokens?\n\n* The presentation would also be benefited by including an analysis of the runtime complexity overhead and inference-time memory requirements for the analysed best-of-N selection methods.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nThe major strength of the paper is the introduction of the self-certainty metric: Replacing log-likelihood of generated tokens with a metric accounting for distributional confidence over the supported dictionary, the new distributional metric is able to capture more holistic information and deal with some inherent statistical weaknesses of maximum-likelihood based metrics. Experiments are convincing in displaying the premise of better behavior of the new metric over lengthy and reasoning-intensive output generations. \n\n**Weaknesses**\n\nOn the other hand, there is insufficient theoretical justification regarding why the new metric works better and an underwhelming interpretation of observed failure cases. It’s unclear why Kullback-Leibler (KL) was preferred from other statistical divergences, why the arguments of KL (an asymmetric divergence) where selected in this order, and\nwhy divergence from the uniform is included, as it only adds a constant term to the metric. Moreover, according \nto the demonstrated results in Table 1, the Borda voting contributes crucially to the achieved performance, while self-certainty without voting is mostly subpar to other competitive method when this component is not included.", "was_revised": true, "final_justification": "After authors response, I'm still weakly leaning towards paper's acceptance. I think extra experiments on voting applied to baselines and multiple choice benchmarks can be included in the revised version, offering a fuller picture of the proposed metric.", "rebuttal": "We sincerely appreciate your thoughtful feedback. Below, we address your specific concerns and suggestions:\n1. **Justification of Choosing Reverse KL with Uniform Distribution as Main Metrics (Weakness)** \n\n    We chose reverse KL divergence to a uniform distribution as our primary confidence metric based on extensive exploratory experiments.  Our motivation was to ask: *“Does the probability mass beyond the sampled token help us assess model confidence for best‑of‑N selection?”* We compared several candidates (e.g., forward KL, entropy, perplexity) and found that reverse KL is the most robust, particularly as N increases: it does not suffer from the same scaling “pitfalls” as other metrics.\n\n    Regarding the uniform reference, we also tried using the empirical distribution over the N samples (Appendix A.4) and observed no significant difference. We retain the constant term in the uniform‐KL for easier interpretation; users may omit it if desired.\n\n   To further elucidate the behavior of “self‑certainty,” we will include two formal properties (and their proofs) in the next revision:\n\n   1. **SFT increases self‑certainty** on the training completions.\n   2. **Self-certainty is less prone to repetition trap**. When the repetition‑branch distribution features a flatter tail, the self‑certainty–preferred collapse set is a subset of the distributional entropy or perplexity sets.\n\n\n2. **Incorporating a Voting Scheme over Multiple Metrics (Question1)**\n\n   We agree that combining confidence metrics can be beneficial.  We have implemented a simple Borda‐count voting scheme on MATH‑level 5 (N = 64) and report preliminary results below:\n\n   | Method           | Perplexity | (D) Entropy | (D) Perplexity | (D) Gini | Self‑certainty |\n   | ---------------- | ---------: | ----------: | -------------: | -------: | -------------: |\n   | Self‑consistency (p=0) |      37.99 |       37.99 |          37.99 |    37.99 |          37.99 |\n   | Borda (p = 1)  |      38.60 |       39.12 |          38.14 |    39.35 |          38.90 |\n   | Borda (p = 2)  |      38.60 |       38.40 |          38.14 |    38.75 |          39.43 |\n\n   These results show that, for large N, self‑certainty dominates when the Borda exponent p is tuned appropriately—demonstrating its robustness relative to other metrics. We will include this table in the revised manuscript.\n\n3. **Why the “FirstAns” Baseline Stops Improving for N ≥ 16 (Question2)**\n\n   The “FirstAns” baseline simply selects the first valid answer encountered among the N samples.  In our experiments, all questions already yield a valid answer within the first 8 samples, so increasing N beyond 8 offers no benefit.  Other metrics degrade for large N because they can be misled by over‑confident incorrect responses.\n\n4. **Self‑certainty as a Measure of Model Calibration (Question3)**\n\n   Indeed, self‑certainty serves as an effective proxy for calibration.  In Figure 1, self‑certainty cleanly separates correct responses from no-response and incorrect ones, whereas perplexity exhibits weaker separation.  Furthermore, Appendix A.2 (Figure 10) shows that average self‑certainty declines significantly as problem difficulty increases, matching the drop in accuracy.  We will make these calibration connections explicit in the revised draft.\n\n5. **Clarification on Figure 2 Normalization (Question4)**\n\n   In Figure 2, we compare each metric against itself across two sample sets—not the two metrics against each other—so no additional normalization is needed.  Both self‑certainty and perplexity are already length‑normalized by definition, allowing direct comparison across samples of varying lengths.\n\n6. **Runtime Complexity and Memory Overhead (Question6)**\n\n   Computing self‑certainty incurs approximately no extra cost: the logits required are already produced during sampling, so best‑of‑N strategies mentioned in our manuscript (regardless of confidence metric) share the same time and memory complexity, which is dominated by the sampling process itself.  While one could leverage self‑certainty for dynamic sampling given the roboustness demonstrated in our article, that extension is beyond this paper’s scope.\n\n7. **Self-certainty on Short Questions (Question5)**\n   Since self-certainty is a logit-based method, we infer that for short questions there are too few logits to average over, and the task relies more on memorization than on reasoning, self‑certainty’s effectiveness can decline. Here is an experiment on a Commonsense QA using Llama 3.1‑8B‑Instruct with N = 8, we obtain:\n\n   | Method                 | Accuracy (%) |\n   | :--------------------- | -----------: |\n   | FirstAns.              |        51.02 |\n   | Self‑consistency       |        59.05 |\n   | Self‑certainty         |        53.64 |\n   | Borda voting (p = 0.7) |        60.11 |\n\n   Here, Borda voting still able to outperform self-consistency under the preliminary setting. We will include the result of the multiple choice QA dataset for completeness in the future revision.\n\n\nWe trust these clarifications and the additional results will address your concerns. Thank you again for the valuable feedback!"}, {"review_id": "4J9xfCQrps", "reviewer": "Reviewer_3kq2", "summary": "This paper proposes a lightweight and scalable approach for Best-of-N selection in Large Language Models (LLMs) based on a novel metric called self-certainty. Unlike reward model-based or self-consistency methods, self-certainty quantifies the model’s internal confidence by computing the divergence of the output token distribution from a uniform distribution. The authors show that this distributional confidence measure can distinguish correct from incorrect responses more effectively than traditional log-probability or perplexity-based metrics. To improve performance on tasks with explicit answers, they introduce a self-certainty-based Borda voting scheme that weights responses by confidence rank. The method is evaluated across mathematical reasoning, code reasoning, and code generation tasks, demonstrating consistent improvements over self-consistency, Universal Self-Consistency (USC), and greedy decoding. The results highlight the method's scalability, generalizability to open-ended tasks, and effectiveness in selecting higher-quality responses without additional training or inference cost.", "questions": "1. On many tasks, self-certainty without Borda underperforms self-consistency, suggesting it may not be useful without Borda voting. The authors need to also conduct evaluation on Borda voting with other confidence measures, such as KL, Entropy, DP, SSC, to show the effectiveness of self-certainty.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n1. The paper introduced a new metric called self-certainty and a self-certainty-based Borda voting scheme. \n2. The experimental results show that the proposed method consistently outperforms various baselines, such as self-consistency, universal self-consistency across diverse benchmarks.\n3. The paper is well-structured and clearly written.\n\n**Weaknesses**: \n1. Soft self-consistency (SSC) [1] use aggregation of probability of each token to conduct the Best-of-N selection also challenges limitations of traditional self-consistency methods on open-ended generation tasks. This paper should discuss and compare with this method.\n2. The proposed method requires access to token probability, but the most performant LLMs are typically black-box models often with limited or no access to logits. It can not generalize to these black-box models. Therefore, some work has explored prompting language models to express uncertainty or confidence score in human language [2,3]. I think this paper should discuss and compare with this kind of method.\n3. On many tasks, self-certainty without Borda underperforms self-consistency, suggesting it may not be useful without Borda voting. The authors need to also conduct evaluation on Borda voting with other confidence measures, such as KL, Entropy, DP, SSC, to show the effectiveness of self-certainty.\n4. The Borda voting method requires tuning a hyperparameter $p$ to balance confidence and frequency, and optimal values vary across datasets and sample sizes, which may hinder out-of-the-box applicability.\n\n[1] Soft Self-Consistency Improves Language Models Agents (Wang et al., ACL 2024)\n\n[2] Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback (Tian et al., EMNLP 2023)\n\n[3] Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (Xiong et al., ICLR 2024)", "was_revised": true, "final_justification": "Most of my concerns are solved in rebuttal. The only remaining concern is what is the basis for choosing $p=0.3$ (for example, why not 0.4 or 0.5?) for $N \\leq 16$. I hope the author can make the changes as promised in further version.", "rebuttal": "Thank you for your thoughtful review! We’re grateful for the opportunity to clarify and strengthen our manuscript.\n\n1. **Comparison with Soft Self‑Consistency (Weakness1)**\n   Thank you for pointing out the Soft Self‑Consistency paper—it’s highly relevant. We have now directly compared our self‑certainty measure against soft self‑consistency (using mean‑aggregation over the entire response as the action space) on the MATH‑Level 4 dataset with $N=16$. Results:\n\n   | Method       | First Answer | Self‑Certainty | Soft Self‑Consistency |\n   | ------------ | -----------: | -------------: | --------------------: |\n   |   Accuracy  |       57.21 |         64.37 |                58.53 |\n\n   These results indicate that self-certainty significantly outperforms soft self-consistency under equivalent settings. While the original soft self-consistency also introduces the idea of averaging over the answer space alone (instead of the full response), this innovation is equally applicable to self-certainty.\n\n   Moreover, in scenarios with sparse answer spaces—such as those discussed in the SSC paper—self-certainty can be especially advantageous over self-consistency. This is because each response is more likely to produce a unique answer group, reducing the effectiveness of self-consistency. We will cite and discuss the Soft Self-Consistency paper in the next version of the manuscript.\n\n2. **Usefulness of Self‑Certainty (Weakness2 & 3)**\nAlthough self‑certainty alone underperforms full self‑consistency methods, it offers several unique advantages:\n\n   * **Fine‑Grained Confidence**\n   Self‑certainty provides token‑level confidence scores, enabling potential of “local” or process‑based scoring.\n\n   * **Generalizability**\n   Unlike self-consistency, which relies on comparable final answers, self-certainty is applicable to a wider range of tasks, including open-ended or generative tasks such as free-form dialogue. It is particularly useful in cases where the answer space is vast or difficult to isolate from the overall response. In addition, self-certainty can be integrated with GRPO to support reinforcement learning without external feedback.\n\n   * **Model Calibration**\n   Self-certainty is effective at capturing model calibration. As shown in Figure 1, it better separates correct from incorrect and null responses compared to perplexity. In Appendix A2, Figure 10, we show that self-certainty correlates well with question difficulty, indicating its potential as a proxy for model performance or task complexity in the absence of ground-truth answers.\n\n   * **Standalone Metric**\n      In contrast to composite methods that require multiple scoring mechanisms, self-certainty is a standalone metric. It can replace or complement traditional token probability–based confidence signals. For example, self-certainty can substitute the token-probability weights in soft self-consistency, potentially improving its performance.\n\n   * **Potential for Black-Box Models**\n   Although current APIs may not expose self-certainty, the metric does not require revealing the full predictive distribution, making it compatible with plausible future API designs (e.g., exposing token-level confidence summaries). Moreover, for black-box systems, one could approximate self-certainty using a white-box surrogate’s logits, which can be a promising future work.\n\n3. **Voting Methods with Different Confidence Measures (Weakness3 & Question)**\n   We agree that a head-to-head comparison is helpful. On MATH-Level-5 with Llama-3.1-8B-Instruct (N=16), we evaluated voting using several confidence measures and Borda exponents p.\n\n   Method           | Perplexity | (D) Entropy | (D) Perplexity | (D) Gini | Self‑certainty |\n   | ---------------- | ---------: | ----------: | -------------: | -------: | -------------: |\n   | Self‑consistency (p=0) |      37.99 |       37.99 |          37.99 |    37.99 |          37.99 |\n   | Borda (p = 1)  |      38.60 |       39.12 |          38.14 |    39.35 |          38.90 |\n   | Borda (p = 2)  |      38.60 |       38.40 |          38.14 |    38.75 |          39.43 |\n\n   When $N$ and the exponent $p$ is large, self‑certainty–based Borda voting outperforms other metrics, which demonstrates its robustness. We will include this table and analysis in the revision.\n\n4. **Choosing the Optimal Exponent $p$ in Voting (Weakness4)**\n   We appreciate the concern about selecting $p$. As discussed at the end of section6.2, the optimal \n   $p$ increases with the number of samples N. Except exhuastive search, below are two possible way of determine $p$ to use\n\n   * **Rule of Thumb**: use $p=0.3$ for $N\\le16$; $p=1.2$ for larger $N$.\n   * **Data‑Driven Tuning**: one can also tune $p$ on a held‑out validation set or a small fraction of the test set.\n\n   We will expand this discussion and provide more guidance in the revised manuscript.\n\nThank you again for your valuable feedback. We look forward to incorporating these revisions and believe they will significantly strengthen our work."}, {"review_id": "TLdFVVXV5t", "reviewer": "Reviewer_8gKi", "summary": "This paper proposes a metric *self-certainty*, which measures the divergence of the uniform distribution and the output distribution over vocabulary. Then, it uses *self-certainty* with Borda Voting to conduct best-of-N to achieve effective test-time scaling.", "questions": "* Borda Voting may result in ties. How do you solve it?\n* Can you provide the upperbound performance, label it correct as long as one of the N answers is correct?\n* Which model precision do you use? I find that bf16 will cause a large bias in calculating self-certainty score.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**: \n* The proposed *self-certainty* proves to be a better confidence measure than perplexity, especially in terms of the unanswerable situations.\n* It doesn't cost extra computation to get the *self-certainty* measure.\n\n**Weakness**:\n* *self-certainty* is not a normalized scalar. This limits the application of this metric. For example, it's hard to apply *self-certainty* to weighted average methods, as standard confidence scores can do[1]. I guess this is a reason for using Borda Voting, a not very suitable method. If the authors can figure out a way to normalize *self-certainty*, it will be a perfect solution.\n* Borda Voting might not be a good voting method, because it discards the absolute values and only keeps the ordinal information. Besides, it shows instability when scaling, the hyperparameter (*p*) varies with *N*. \n* There seems to be a conflict in experimental setting. While Line 219~220 explains Figure 4 with \"*no-answer responses*\", Line 170 states that \"*Responses without extractable answers are masked*\".\n* Figure 2 is problematic. According to the definition of Self-Certainty, it should be irrelevant with the output token. Therefore, the first token of Sample I and II should have the same *self-certainty* score, but they are not. I doubt if these number are made up by human. Besides, Line 131 states \"*sample I contains an initial error*\", but I can't see why the first token \"We\" is an error.\n\n[1] Efficient Test-Time Scaling via Self-Calibration.", "was_revised": true, "final_justification": "All of my concerns are properly addressed. So I decide to raise my overall rating.", "rebuttal": "Thank you for your valuable feedback. Below, we address each of your concerns individually:\n\n1. **Normalization of Self-Certainty (Weakness1)**\n\n   The method of normalizing self-certainty depends heavily on the specific use case and is typically straightforward. For example, if self-certainty is used in reinforcement learning, **z-score normalization** ($\\frac{\\text{sce} - \\text{mean(sce)}}{\\text{std(sce)}}$) can be an effective choice. Alternatively, if one intends to interpret self-certainty as a probability or scale it to the $[0, 1]$ range, **min-max normalization** ($\\frac{\\text{sce} - \\min(\\text{sce})}{\\max(\\text{sce}) - \\min(\\text{sce})}$) is applicable.\n\n   It is worth noting that even well-established metrics like probabilities often require transformation prior to application. For instance, in the self-calibration metric [1] you mentioned, a transformation of the form $\\mathrm{SSC}(y)= \\frac{\\sum_{i : y_i = y} c_i}{\\sum_{i=1}^{N} c_i}$ is employed. In this context, $c_i$ could seamlessly be replaced with self-certainty scores $sce_i$ for normalization purposes.\n\n   Furthermore, many widely used metrics in NLP, such as **perplexity** or **average log-probability**, are not normalized but are nonetheless informative and accepted by the community.\n\n2. **Intuition Behind Borda Voting (Weakness1 & 2)**\n\n   Borda voting is a deliberate design choice, not a workaround for unnormalized scores. It balances the influence of self-certainty and self-consistency while remaining stable as $N$ changes. Table2 shows that simple sum voting performs between pure self-consistency and Borda voting. In small or highly imbalanced samples (e.g., $0.5+0.5>0.9$ or $0.5\\times10>0.9\\times5$), sum voting collapses to vanilla self-consistency, discarding confidence information. Borda voting mitigates this by ranking the scores and by letting us modulate the relative weight of confidence via the exponent $p$. Designing a scheme that keeps absolute scale, preserves stability across $N$, and offers the same control remains a promising avenue for future work.\n\n3. **Clarification of Lines 219–221**\n\n   Thank you for pointing this out. The statement:\n\n   > *“Notably, perplexity assigns higher confidence to no-answer responses—often resulting from self-repetition or early stopping—while self-certainty reliably assigns these responses lower confidence scores.”*\n\n   was intended to reference Figure1, where we distinguish between correct, incorrect, and no-response cases, not Figure 4. We will revise the manuscript wording to eliminate this ambiguity.\n\n4. **Clarification Regarding Figure 4**\n\n   We appreciate your close reading and catching the typo in Equation (2) and the formulas afterwards; it should be $p(\\cdot|x, y_{\\leq i})$ instead of $p(\\cdot|x, y_{< i})$.Correspondingly, our token‑wise self‑certainty for the first token should measure the KL divergence **after** sampling token 1, which better reflects the model’s change in belief after deciding on one token. While this shift has minimal effect on the sentence‑wise self‑certainty (since it averages over many tokens), it yields more interpretable token-level values.\n\n   The values reported in Figure 4 are empirical, not made up. We examined multiple completions and selected representative examples based on their confidence trajectories. By “initial error” we meant the first reasoning step (step 1), not the first token. Our goal was to illustrate the model’s early preference between two reasoning patterns; we agree that aggregate metrics are more informative overall, and we will both correct the typo and clarify our intent when discussing Figure 4.\n\n5. **Additional Questions**\n   * **Handling Ties:** In the rare event of a tie, we select the first sampled candidate. However, due to the use of the power function in weighting, ties are extremely uncommon in practice.\n\n   * **Upper Bound:** The upper bound of performance is provided in Appendix A.1, Figure 9, under the label *Oracle*, which significantly outperforms all mentioned metrics.\n\n   * **Model Precision:** The model uses **bfloat16** for weights during inference and **float32** for subsequent logit processing.\n\nWe appreciate your insightful comments and believe the revised manuscript will reflect these improvements more clearly.\n\n[1] Efficient Test-Time Scaling via Self-Calibration."}, {"review_id": "Ud0p54jTPJ", "reviewer": "Reviewer_7zkf", "summary": "The paper explores improving LLM performance for reasoning tasks by combining the results of N runs. Different methods of best-of-N strategies are discussed, with the suggestion that using self-certainty, a metric based directly on the token distributions in the LLM output, instead of just using the final tokens, is the best path. Voting is explored as well, combining the token distributions from several runs - similarly to directly using results from several runs. Some metric strategies are compared to the chosen one, in addition to baseline approaches based on external relevance models.", "questions": "1. How would I choose the parameter p? \n2. I don't think the issue of resources was addressed adequately. If anything, assuming I understood the proposal and numbers correctly, I would suggest rewriting the paper as a proposal to gain most of the benefit of the best-of-N approaches by using the probability tokens of a single run (without voting). \n3. Please add to table 1 description the specific metric used (accuracy?), and confidence intervals or at least a mention of statistical significance of results. \n4. Is it really practical to analyze values such as N=64? I can't imagine running each task via an LLM X64 times in any applied scenario. If the KL-div metric only outperforms others when N>16, is it really more useful in practice?", "limitations": "I would add the resource concerns to the limitation discussion.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper deals with an important problem and shows that best-of-N is a good path to improve performance, even if costly in compute/LLM resources. Several datasets are used in the analysis, as well as an adequate number of baselines and metric strategies. \n\nThe main weaknesses of the paper, in my view, are: \n1. The narrative is very unclear, needlessly defining different metrics that only differ by a constant (self-certainty, KL-divergence and Cross Entropy). This means that the results are less readable, as sometimes they discuss KL-divergence and sometimes self-certainty. Moreover, the paper begins with the definitions of several metrics, some of which are considered as baselines and some as the method proposed by the paper. I would suggest a rewrite of the paper, focusing on first describing the specific proposed strategy - one metric, together with a better description of the voting method. Then the analysis and experimental sections can suggest the different baselines to compare against, as well as the voting method ablation. \n2. Results are not clearly outperforming the baselines. No confidence intervals are presented. Using the probabilities of a single run does not seem to improve on the performance of best-of-N strategies. The gains from using the probabilities of multiple runs with voting, as opposed to just using best-of-N, seem minor. It is also unclear what would be a practical way of choosing the hyper-parameter p in the voting strategy, which seems to cause results to be sub-optimal. In general, the proposed strategy seems like a lot of effort for little-to-no gain.", "was_revised": true, "final_justification": "The narrative in the paper seems somewhat broken and unclear, with a large emphasis on equations and less emphasis on explanations. However, the work has novelty and provides an interesting direction to a very important problem, so my final vote would be a borderline accept.", "rebuttal": "Thank you for your thoughtful feedback. We address each point below and will incorporate these clarifications in the revised manuscript.\n\n1. **Clarity of Narrative (Weakness1 & Problem2,3)**\n\n   The overall flow of our paper is as follows: Our experiments are exploratory in nature rather than aimed at proving a specific theoretical result. We observed that most existing confidence measures rely solely on the probability of sampled tokens—such as perplexity—while discarding the distribution of unsampled tokens. This led us to ask an important question: **Does the discarded distribution encode useful information about the model’s confidence?**\n\n   To answer this, we introduce four candidate confidence measures that incorporate distributional information. Among these, KL-divergence performs best. We term this approach **self-certainty (SCe)** and adopt it as our primary metric.\n\n   To further enhance the performance of existing metrics, we propose **Borda voting** and evaluate both its effectiveness and the generalizability of self-certainty. We acknowledge that the switch between the term *KL* and *self-certainty* may reduce readability. To address this, we will revise the manuscript to consistently denote it as KL (SCe)* Additionally, we will clearly separate our main proposed method from baseline methods in the revised version for improved clarity.\n\n   Your suggestion to **separate probability-based methods and voting-based methods** to better highlight the strengths of our approach is greatly appreciated. We will reorganize the presentation accordingly and include the metric used (mean accuracy) in the tables for transparency.\n\n2. **Significance of Results (Weakness2 & Problem3)**\n\n   We agree that emphasizing the statistical significance of results strengthens the paper. In the revised manuscript, we will include a **one-sided paired t-test** with the hypothesis that the proposed voting methods outperform self-consistency at fixed $N$ and $p$ across runs and datasets.\n\n   The p-values are shown below:\n\n   | Variant         | p (N = 8) | p (N = 64†) |\n   | --------------- | --------- | ----------- |\n   | Borda (p = 0.3) | 0.00      | 0.09        |\n   | Borda (p = 0.7) | 0.01      | 0.02        |\n   | Borda (p = 1.2) | 0.02      | 0.00        |\n   | Borda (p = 2.0) | \\*\\*      | 0.00        |\n\n† LiveBench-Math is excluded as the result is unavailable.\n\\*\\* Underperforms self-consistency.\n\nAs shown, Borda voting with most values of $p$ **significantly outperforms self-consistency**.\n\n3. **Choosing $p$ (Weakness2 & Question1)**\n\nWe have discussed $p$-selection in the end of Section 6.2. Two practical strategies can be used when exhaustive search is undesirable:\n\n* **Rule of Thumb**: Use $p = 0.3$ for $N \\leq 16$; $p = 1.2$ for larger $N$.\n* **Data‑Driven Tuning**: Tune $p$ on a held‑out validation set or a small subset of the test set.\n\nWhile these heuristics may not yield the global optimum, they still deliver performance that **significantly exceeds self-consistency**, as shown by the significance tests above.\n\n4. **Practicality of Using $N > 16$ (Question 4)**\n\n   It is standard practice in the literature for best-of-$N$ methods to evaluate with $N > 16$. For example:\n\n   * Self-consistency \\[1] uses $N = 40$.\n   * Scaling test-time compute considers $N = 512$ \\[2].\n   * PRM evaluates up to $N = 1000$ \\[3].\n\n   To contextualize $N = 64$: if a reasoning model requires 8× the tokens of a standard model, sampling 64 responses for a standard model is equivalent to \\~8 samples for a reasoning model—a reasonable computational budget.\n\n   Furthermore, **strong performance at larger $N$ demonstrates the stability of self-certainty**, an important property for confidence metrics. This indicates robustness when handling more tokens with fewer errors, which is highly desirable.\n\n[1]Self-Consistency Improves Chain of Thought Reasoning in Language Models\n\n[2]Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters\n\n[3]Let’s Verify Step by Step"}, {"review_id": "oHBBlGqxqV", "reviewer": "Reviewer_euVc", "summary": "This paper propose self-certainty, a KL-divergence based metrics that quantify how certain an LLM is regarding to its current response. The user propose to use it as an effective way of scaling Best-of-N selection without the involve of reward model. It is further combined with borda voting to further improve the performance.", "questions": "1. For greedy decoding, is it only sampling once?\n2. Can you provide the comparison with \"normalized weighted sum\" with self-consistency?\n3. Can you provide more results with reasoning models?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strength:\n- The paper is well-written and easy to follow.\n- The overall method is straightforward and have its own motivation.\n\nWeakness:\n- The empirical improvement is extremely limited compared to vanilla self-consistency. Given the additional cost to compute the confidence, it is possible that simply use the cost to sample 1-2 more response would have a better performance. The authors should provide the wall-clock time comparison or further use the proposed method in learning or other scenarios to make the impact more significant.\n- The novelty is limited as the final algorithm is to use borda voting with self-consistency. In the original self-consistency paper, \"normalized weighted sum\" is already a version for self-consistency. A comparison to this is also missing.\n- The experiments are mostly conduct on non-reasnoning models, and only a single reasoning model is used in the experiment. While it is not required, it is definitely helpful if the authors can provide more results on reasoning models given the task is reasoning-heavy tasks that naturally fits reasoning models.", "was_revised": true, "final_justification": "Most of my concerns are answered in the rebuttal. However, the authors do decide to leave many additional necessary changes to be in the revised version of the paper, i.e., the new theoretical guarantee of self-certainty, the complete comparison to the weighted version of self-consistency, figure similar to current Fig. 1 on Qwen3-4B. This leads to the potential of another round of careful peer review needed. Therefore, I personally recommend this paper as a really borderline paper in the current stage.", "rebuttal": "Thank you for your thoughtful feedback. We appreciate the opportunity to clarify and strengthen our work. Below we address your concerns and outline our modifications:\n\n1. **Limited Improvement and Novelty (Weakness1 & 2)**\n\n   Our paper introduces **self-certainty metrics** and **Borda voting**, both of which are novel contributions not covered in existing literature. Additionally, we conduct a **systematic comparison** between two major paradigms of confidence estimation:\n   * **Probability-based measures** (e.g., perplexity, which only considers the probability of the sampled token)\n   * **Distribution-based measures** (e.g., self-certainty, which leverages the entire token distribution)\n\n   This comparison highlights the advantages of using distributional information for **calibrating confidence** and **selecting superior responses**.\n\n   **Significance of Improvements:**\n   The performance gains of Borda voting are substantial and consistent. For example:\n\n   * On **LiveBench-MATH**, Borda voting improves accuracy by **+1.2%** (N=8).\n   * On **MATH**, it outperforms self-consistency by **+0.7%** (N=64).\n\n   These improvements are especially meaningful given the strong baseline of self-consistency and the difficulty of these benchmarks.\n\n   **Key Advantages of Self-Certainty:**\n\n   * **Fine-grained Confidence:** Provides token-level estimates rather than coarse sequence-level metrics.\n   * **Generalizability:** Applicable to diverse tasks, including open-ended scenarios where final answers are not directly comparable.\n   * **Efficiency and Scalability:** Uses logits already computed during inference, adding **negligible overhead**.\n   * **Robustness vs. Perplexity:** Self-certainty better distinguishes correct from incorrect outputs (Figure 1), avoids length bias (Figure 5), and achieves superior stability (Figure 4, Table 1).\n\n   While increasing N may improve self-consistency, it incurs significant computational cost. In contrast, self-certainty-based methods leverage existing logits with virtually no extra cost under a proper implementation.\n\n2. **Comparison with Normalized Weighted Sum (Weakness2 & Problem2)**\n\n   We considered straightforward score‐summing but found two critical shortcomings:\n\n   1. **Instability under imbalanced samples.**\n      Borda voting mitigates the collapse of sum-based methods self-consistency in imbalanced cases (e.g., `0.5 + 0.5 > 0.9` or `0.5×10 > 0.9×5`), where it revert to vanilla majority voting and ignore confidence.\n\n   2. **Tunable trade-off via exponent p.**\n      The exponent p lets us smoothly interpolate between pure voting‐based (p→0) and confidence-based (p→∞) schemes.\n\n   We deliberately use self-consistency rather than a normalized weighted-sum baseline because prior work shows that normalized weighted-sum actually underperforms vanilla self-consistency (see Table 1 of the Self-Consistency paper \\[1]). Moreover, normalized weighted-sum relies solely on token probabilities and ignores the full distribution over unsampled tokens—a limitation that our self-certainty metrics overcome, as evidenced in Figure 1, Figure 4, and Table 1. Finally, when we replace probabilities with self-certainty scores in a “sum voting” variant (Table 2), Borda voting still outperforms simple summation, confirming the advantages of our rank-based approach.\n\n3. **Experiments with Additional Reasoning Models (Weakness3 & Question3)**\n\n   We agree that broader evaluation strengthens the work. We have now added experiments on **Qwen3-4B (Thinking Mode)** with the MATH-level 5 test set. The same performance ordering holds:\n   `FirstAns < Self-Certainty < Self-Consistency < Borda`.\n\n   | Method       | N=4 | N=8 | N=16 |\n   | ---------------- | ------: | ------: | -------: |\n   | FirstAns         |   87.24 |   87.61 |    88.37 |\n   | Self-Consistency |   87.84 |   88.37 |    89.43 |\n   | Self-Certainty   |   87.54 |   88.14 |    88.97 |\n   | Borda (p=0.3)    |   87.92 |   88.44 |    89.65 |\n   | Borda (p=0.7)    |   87.99 |   88.52 |    89.73 |\n   | Borda (p=1.2)    |   87.92 |   88.60 |    89.88 |\n   | Borda (p=2.0)    |   87.54 |   88.44 |    89.95 |\n\n4. **Clarification on Greedy Decoding (Question 1)**\n   Yes—greedy decoding uses a single pass. It is fully deterministic (aside from floating-point nondeterminism), so additional sampling yield little variations.\n\nWe believe these clarifications and additional results address your concerns and demonstrate the novelty, significance, and robustness of our contributions. Thank you for your valuable feedback.\n\n[1]Self-Consistency Improves Chain of Thought Reasoning in Language Models"}]

Author rebuttal:

Meta-review: This paper introduces "self-certainty," a lightweight metric for Best-of-N selection in LLMs. The method leverages the model's internal token probability distribution to estimate response quality, offering a computationally efficient alternative to resource-intensive reward models.

The core of the proposed method is to quantify the divergence of the LLM's output distribution from a uniform distribution (using KL-divergence) as a measure of confidence. This "self-certainty" score is then used to select the best response from N generated samples. The authors also propose a Borda voting that combines self-certainty rankings with response frequency to improve performance on tasks with definite answers. The work is relevant to NeurIPS, addressing the challenge of improving the reasoning and reliability of LLMs, a central topic in the field. Specifically, it contributes to the area of efficient inference-time optimization for language models.

The primary innovation lies in using the entire token distribution for confidence scoring, rather than just the probability of the generated tokens (like perplexity). This provides a more robust signal of the model's confidence. The application of a Borda voting to integrate this confidence score with self-consistency is also a fine contribution.

Tihs paper received consistent ratings from all five reviewers, with each assigning a score of 4 (Borderline accept).
Also, confidence levels were generally high. There were no significant conflicting viewpoints among the reviewers. All converged on a similar assessment of the paper's strengths and weaknesses.

Reviewers acknowledged the novelty and potential of the self-certainty metric. They highlighted its efficiency (minimal/near-zero overhead), scalability with the number of samples (N), and its applicability to open-ended tasks where traditional self-consistency fails. The paper is also well-structured and clearly written.

A primary concern shared by reviewers was the observation that the self-certainty metric alone often underperforms the self-consistency baseline. The strong performance of the proposed method appears to be heavily reliant on the Borda voting. Reviewers pointed out that this voting mechanism was not applied to other baseline confidence measures, making a direct comparison of the self-certainty metric's effectiveness difficult. The need to tune the hyperparameter 'p' for the Borda voting was also noted as a potential limitation. Also, multiple reviewers note that SCe without Borda often trails self-consistency - the gains primarily come from pairing SCe with Borda.

While the paper provides a strong intuition for why distributional confidence should be a better metric, reviewers noted a lack of deep theoretical justification for the specific choice of KL-divergence from a uniform distribution. The authors' rebuttal promised to include formal properties and proofs in the final version.

Extensive experiments are presented across multiple reasoning benchmarks (mathematical, code reasoning, code generation). The results demonstrate that the self-certainty-based Borda voting consistently outperforms baselines. However, the ablation of the Borda voting component reveals that self-certainty alone is not always superior, which is a key point of discussion.

Overall, the paper introduces a novel and intuitive idea to address the important problem of Best-of-N selection in LLMs. The proposed method is simple, computationally efficient, and shows consistent empirical gains over strong baselines when combined with a voting mechanism. The work is well-executed and the authors' rebuttal effectively addressed the majority of the reviewers' concerns, demonstrating a clear path to an improved final version. The authors must incorporate the additional experiments, clarifications, and theoretical justifications promised in their rebuttal.
Decision: Accept (poster)