Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measure
Authors: Lukas Aichberger
Kajetan Schweighofer
Sepp Hochreiter
Venue: ICLR 2026 Poster (2026)
Keywords: llm
nlg
uncertainty estimation
uncertainty measures
proper scoring rules
zero-one score
single-sequence measures
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We theoretically motivate single-sequence uncertainty measures for LLMs and propose G-NLL, an uncertainty estimate that substantially reduces computational costs while maintaining state-of-the-art performance.

Abstract: Large Language Models (LLMs) are increasingly employed in real-world applications, driving the need to evaluate the trustworthiness of their generated text. To this end, reliable uncertainty estimation is essential. Leading uncertainty estimation methods generate and analyze multiple output sequences, which is computationally expensive and impractical at scale. In this work, we inspect the theoretical foundations of these methods and explore new directions to enhance computational efficiency. Building on the framework of proper scoring rules, we find that the negative log-likelihood of the most likely output sequence constitutes a theoretically principled uncertainty measure. To approximate this alternative measure, we propose G-NLL, obtained using a single output sequence from greedy decoding. This approach streamlines uncertainty estimation while preserving theoretical rigor. Empirical results demonstrate that G-NLL achieves state-of-the-art performance across various scenarios. Our work lays the theoretical foundation for efficient and reliable uncertainty estimation in natural language generation, challenging the necessity of the prevalent methods that are more complex and resource-intensive.

Published as a conference paper at ICLR 2026
RETHINKING UNCERTAINTY ESTIMATION IN LLMS: A PRINCIPLED
MEASURE
Lukas Aichberger Kajetan Schweighofer Sepp Hochreiter
ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria
2 NXAI GmbH, Linz, Austria
Joint first authors (aichberger, schweighofer, Nochreit)@ml.jjuu at
ABSTRACT
Large Language Models (LLMs) are increasingly employed in real-world applica- tions, driving the need evaluate the trustworthiness of their generated text. To this end, reliable uncertainty estimation is essential. Leading uncertainty estima- tion methods generate and analyze multiple output sequences, which is computa- tionally expensive and impractical at scale. In this work, we inspect the theoretical foundations of these methods and explore new directions to enhance computational efficiency. Building on the framework of proper scoring rules, we find that the neg- ative log-likelihood the most likely output sequence constitutes a theoretically principled uncertainty measure. To approximate this alternative measure, we pro- pose G-NLL, obtained using a single output sequence from greedy decoding. This approach streamlines uncertainty estimation while preserving theoretical rigor. Em- pirical results demonstrate that G-NLL achieves state-of- the-art performance across various scenarios. Our work lays the theoretical foundation for efficient and reli- able uncertainty estimation in natural language generation, challenging the neces- sity of the prevalent methods that are more complex and essoucc--ntensivee
INTRODUCTION
Despite advances in natural language generation (NLG),, determining the trustworthiness of generated text remains challenging. Addressing this requires reliably estimating the uncertainty a language model has regarding its generated text. Although low level uncertainty does not guarantee factual correctness, particularly when the generated text is based consistent but inaccurate training data, uncertainty estimates remain a reliable indicator of errors at present (Farquhar et al., 2024).
Assessing predictive uncertainty in Large Language Models (LLMs) is challenging due to their stochastic, autoregressive nature. Each token is selected probabilistically. leading to diverse outputs for the same input. Furthermore, the vast space of possible sequences is computationally intractable. Common uncertainty estimation methods thus rely on expectations over output distributions, such as sequence entropy (Malinin and Gales, 2021; Kuhn et al., 2023; Duan al., 2024; Farquhar et al., 2024), which in turn requires sampling multiple output sequences. However, this is computationally costly due the large number of model parameters. As result, only a small subset of outputs is sampled in practice. However, differences between sampled sequences do not always indicate uncertainty, as they may vary lexically while remaining semantically similar. Some methods use inference models to assess semantics (Kuhn et al., 2023; Aichberger et al., 2025; Farquhar et al., 2024), improving uncertainty estimates but adding complexity and additional computation. These challenges make large-scale uncertainty estimation impractical for real- real-world applications.
Efficient uncertainty estimation methods are needed to ensure the trustworthiness the language model's answer without imposing excessive computational demands. To address this need, we theoretically motivate that uncertainty measures require only a single output sequence. We do so by building insights from the framework of proper scoring rules (Gneiting and Raftery, 2007) that has recently been investigated for uncertainty estimation the standard classification setting (Kotelevskii et al., 2025; Hofman et al., 2024). Specifically, we extend proper scoring rules for uncertainty

---

Published as a conference paper at ICLR 2026
estimation to NLG and explore the zero-one The resulting uncertainty measure is straightforward: is the negative tog-likelihood of the most likely output sequence. Importantly, the measure is based the most likely output sequence, rather than an arbitrary one. However, determining the most likely output sequence remains computationally expensive. Therefore, propose G-NLL, an approximation that is maximally efficient and minimizes algorithmic complexity, while maintaining state-of-the-ar performance.
alternative to the prevalent logarithmic score.
In parallel, recent work has considered the likelihood of single output sequence for uncertainty estimation in NLG (Fadeeva et al., 2023; 2024; Vazhentsev et al., 2024; Abbasi- Yadkori et al., 2024; Vashurin al., 2025a:b). Notably, Fadeeva et al. (2023) introduce the maximum sequence probability (MSP) as a baseline, which corresponds to the negative log-likelihood of the most likely output sequence. However, their hoc formulation does not provide theoretical justification or a discussion on how to best approximate it. More generally, prior work has not properly characterized the MSP as a measure for uncertainty in NLG. a result, prior work often relies on unfavorable length-normalee nonaaiinormaiization (Qiu and Miikkulainen, 2024; Chen et al., 2024; Bakman et al., 2024; Yaldiz et al., 2025), does not focus on the most likely output sequence, or even overlooks single-sequence measures entirely baselines (Malinin and Gales, 2018; Manakul et al., 2023; Kuhn et al., 2023; Farquhar et al., 2024; Nikitin et al., 2024; Kossen et al., 2024; Duan et al., 2024).
To close this gap. are the first to provide theoretical justification for the MSP principled, single-sequence measure of uncertainty in NLG. We examine the properties of this alternative uncer- tainty measure and compare tives. Additionally, we propose G-NLL as an efficient approximation of the MSP and analyze why sampling-bol based or length-normalized alternatives are detrimental to its approximation quality. Our experiments demonstrate that G-NLL matches and even exceeds the estimation quality of established methods across various model classes, model sizes, training stages, tasks, datasets, and evaluation metrics. By maintaining theoretical rigor, G-NLL offers an effective and scalable approach to un- certainty estimation in NL.G. serves not only as a strong baseline for future work, but also as a practical solution for deploying uncertainty estimation in LLMS across real-world applications.
established measures from both theoretical and empirical perspec-
To summarize, our main contributions are as follows:
We derive the negative logeliiel of the most likely output sequence (i.e., the MSP) as measure uncertainty NLG, building on established principles from uncertainty estimation theory and proper scoring rules.
We provide a theoretical analysis of the MSP and existing uncertainty measures, and show that estimating this single-sequence measure possesses desirable properties for practical scenarios.
We propose G-NLL an effective approximation of the MSP, and demonstrate that it empirically outperforms state-of-the-art methods while significantly reducing computational costs.
2 PREDICTIVE UNCERTAINTY IN NLG
We begin by reviewing language modeling to formalize predictive uncertainty in NLG. Sec. 2.1 introduces proper scoring rules and their connection to predictive uncertainty, and Sec. 2.2 revisits established measures within this framework. Sec. 2.3 then introduces the maximum sequence probability and its approximation G-NLL under an alternative scoring rule.
Preliminaries. We assume fixed training dataset D { consisting of N token sequences
($1...., 87) where individual tokens s; E V are from a given vocabulary V. Each token at step t assumed to be sampled according to the predictive distribution p(st Sct, w²), conditioned on the sequence of preceding tokens 8et and the true (but unknown) language model parameters w". We assume that the given model class can theoretically represent the true predictive distribution, a common and usually necessary assumption (Höllermeier and Waegeman, 2021). The likelihood of some model parameters tö matching w" is given by the posterior p(t D) = p(D w ())p((()//()))
The input to given language model parameterized by is sequence output is a sequence = (11, yr) E Vr, with x, y E V and R being the set of all possible output sequences with sequence length T. The likelihood of token ye E y being generated by the language model is conditioned on both the input sequence and all previously generated tokens, denoted p(y. æ. Det. w). The likelihood of output sequences y E Dr being generated by
(z (æ1, ..., Tar) and the
2

---

Published as a conference paper at ICLR 2026
the language model is then the product of the individual token probabilities, which is denoted as p(y x w)) II. p(Ve a; Yee.
w) (Sutskever et al., 2014), while the heuristic length-normalized T variant is ply a, w) exp+ + Ei= pog at, y-cr, m)} (Malinin and Gales, 2021).
Calculating the likelihood that specific output sequence y' is generated by the language model parameterized by w is straightforward. The language model directly provides the token likelihoods for a given input sequence. However, determining the full probability distribution on all possible output sequences is considerably more challenging, since the size of 3/r increases exponentially with the sequence length. The computational complexity of evaluating all possible output sequences increases with O(I)?), Since the vocabulary sizes [V] of modern LLMs are well over a hundred thousand tokens, this distribution becomes intractable determine, even for relatively short maximal sequence lengths T (Dubey et al., 2024).
2.1 PROPER SCORING RULES AND THE RELATION TO UNCERTAINTY MEASURES IN NLG
We next give an introduction to proper scoring rules and discuss how they give rise to uncertainty measures. For more details, in the standard classification setting, refer to Hofman et al. (2024); Kotelevskii et al. (2025). Proper scoring rules are a class of functions that evaluate the quality of probabilistic predictions by assigning a numerical score based on the predictive distribution and actual observations (Gneiting and Raftery, 2007). In particular, a proper scoring rule is an extended real-valued function S P x y [-o0, xo). such that S(p, y is P-quusi-integrable over a convex class of probability measures P. A scoring rule is called proper relative to P if the expected score minimized when the evaluated distribution p E P coincides with the distribution from which the outcomes y' E ) are sampled from, and it is called strictly proper if this minimum is unique. In the context of uncertainty estimation NLG, proper scoring rules assign a numerical score that reflects how much probability the predictive distribution of the true model p(y æ, w²) places on an observed output sequence y', denoted as
S(p(y x,, ).))
(I)
To obtain concrete uncertainty measures, we need to make two specific assumptions (Schweighofer et al., 2025). First, we have to define the predictive distribution used to sample output sequences y' Following Aichherger al. (2025), assume that we use single, given language model with parameters w to sample output sequences y' ~ p(y' x. w). This assumption is also implicitly used in other works (Kuhn al., 2023; Fadeeva et al., 2024; Farquhar et al., 2024) and is intuitively reasonable, since our main concern is the uncertainty of the output of specific language model. Thus, we consider the expected score for possible output sequences under the predictive distribution of the given language model, denoted as Ep(y)( [S(p(z) the predictive distribution of the given language model aligns with the true predictive distribution, capturing predictive uncertainty. Second, we have to define how the true model is approximated. We consider a Bayesian approximation of the true model, i.e., we consider each possible language model w according its posterior probability p(Φ D) (Schweighofer et al., 2023b;a). Thus, perform posterior expectation over the expected score, denoted as Ep D) [E w) [S(p(y) x, (), y')l] which can be additively decomposed into an entropy term and a divergence term (Gneiting and Raftery, 2007; Kull and Flach, 2015):
w²), y²)]. This quantifies how well
E)(a(D) [Ep(y' (n))) [S(p(y) x,, 1)) 3 )]]
(2)
expected score
Epte/ (w,w) [S(p(y) x up). y
F [Ep(w' (w.))) (S((()) x. a)) yy)) S(p(y) x., w), y y'll
entropy torm
divergence term
The expected score over possible output sequences y' and language models v captures the total uncertainty of the given language model. The entropy term reflects aleatoric uncertainty, which quantifies the inherent stochasticity of generating output sequences with given language model (Aichberger et al., 2025). The divergence term reflects epistemic uncertainty, which quantifies the uncertainty due to lack of knowledge about the true language model parameters arising from limited data (Houlsby et al., 2011: Gal, 2016; Malinin, 2019; Hüllermeier and Waegeman, 2021).
The remaining degree of freedom is the choice of a proper scoring rule S(., .), which determines the concrete form of the uncertainty measures. In the following, we consider two canonical proper scoring rules: the logarithmic score Stay S, in Sec. 2.2 and the zero-one score S., :) in Sec. 2.3.
3

---

Published as a conference paper at ICLR 2026
2.2 ESTABLISHED UNCERTAINTY MEASURES IN NLG BASED ON THE LOGARITHMIC SCORE
The logarithmic score is typically assumed implicitly in both the standard classification (Houlsby et al., 2011; Gal, 2016) and the NLG setting (Malinin and Gales, 2021; Kuhn et al., 2023) to derive uncertainty measures. This is due the grounding of the resulting uncertainty measures in information theory (Lahlou et al.. 2023; Gruber and Buettner, 2023; Hofman et al., 2024; Kotelevskii al., 2025). In the context of NLG, the logarithmic score considers the negative log-likelihood of generated output sequence y':
Sing (ply x., ,.y')) = log p - y |x,
Substituting the logarithmic score into Eq. (2) results in the total uncertainty being the cross-entropy CE(::-) between the output sequence distribution of the given model and those induced by models integrated over the entire posterior p(r (D) (Aichberger et al., 2025) (sce Apx. A. 1 for details)::
(p( (CE [CE(p(y) æ; w): p(y x. [0)]
total uncertainty
Hyp(y ac, w)) + Ep(eID) [K[ppy xr, w)) || x. w))]
aleatoric uncertainty
epistemic uncertuinty
The epistemic uncertainty is captured by a posterior expectation of the Kullback-Leibler divergence KL( .) between the output sequence distribution of the given model and those induced by models across the entire posterior. This requires considering every possible parameterization of the model. Since modern LLMs have billions of parameters (Radford et al., 2018; Zhang et al., 2022; Dubey et al., 2024; Zuo et al., 2024), the epistemic uncertainty is challenging to estimate.
Thus, current work usually focuses on the aleatoric uncertainty, captured by the Shanmon entropy H(: of the ouiput sequence distribution of the given language model (Kuhn et al., 2023; Aichberger et al., 2025; Farquhar et al., 2024). Computing this output sequence distribution still requires considering the entire set of possible output sequences Vr, which is intractable and has to be approximated, as discussed in the following.
Predictive Entropy. The aleatoric uncertainty under given language model is the entropy of the output sequence distribution p(y x, w)) commonly referred to as predictive entropy (PE) (Malinin and Gales, 2021). Intuitively, high PE implies that the language model is likely to generate different output sequences from the same input sequence, indicating high uncertainty. PE is generally estimated by Monte Carlo (MC) sampling of output sequences (Malinin and Gales, 2021):
H(p(y) x. w)) Eply |- log p(y xx w]]
(5)
poy" |x, (,,w),
y" ply x, w.
Semantic Entropy. Semantic entropy (SE) (Kuhn et al., 2023; Farquhar al., 2024) is based on the fact that output sequences may be different on a token level but equivalent on a semantic level. In such cases, the PE can be misleading, it indicates high uncertainty even when different output sequences have the same semantic meaning. Thus, instead of the entropy of the output sequence distribution, the entropy of the semantic cluster distribution is considered, denoted as p(c æt w) = Ey p(c æ, y, w) p(y æ,, w). The probability of an output sequence belonging to a semantic cluster is usually approximated with a separate natural language inference model. SE thus measures uncertainty about the semantics of output sequences and is defined as
H(p(c x; w)) = w)( [- loggp(c æ;, w)]
(6)
N
1
Σ ==1
- logp(c" xx w),
c² ppcpce x, w).
N
For details on how to construct tractable approximation of SE, we refer to Aichberger et al. (2025).
Discussion. In general, each of these uncertainty measures is based on the logarithmic score and considers the distribution over all possible output sequences p(y x;, w). which is defined over the entire set of possible output sequences Dr. Approximating expectations over this distribution requires sampling multiple output sequences from )', which is computationally expensive. In the following, show that this requirement can be eliminated when considering alternative proper scoring rule.
4

---

Published as a conference paper at ICLR 2026
2.3 NEW UNCERTAINTY MEASURES NLG BASED ON THE ZERO-ONE SCORE
In principle, any proper scoring rule may be used derive viable uncertainty measures (Kotelevskii et al., 2025; Hofman et al., 2024). Thus, we explore measuring predictive uncertainty based on the zero-one score instead of the logarithmic score. Although it has been considered in the standard classification setting, to the best of our knowledge, the zero-one score has not yet been considered as a proper scoring rule for deriving uncertainty measures in NLG. The zero-one score only considers the predictive distribution for the most likely output sequence:
Su-1 (p(y x. v),) = (1 - ply = y x..)) 1 argmax p(v x, .))
y
Substituting the zero-one score into Eq. (2) results in the total uncertainty being the expected confidence that the given model assigns to the most likely output sequences of models across the entire posterior p( (D) (see Apx. A.2 for details):
E P)(()) [1 ply 5" x. w)]
(8)
total uncertainty
1 ply p xz w) + ply = y" x. w) Ep(l) "² x. w)]
aleatoric uncertainty
epistemic uncertainty
with y" argmax, ply x, w) for the given language model and 3" argmax, y p(y x, w) for every possible language model.
As in Eq. (4), the epistemic uncertainty a posterior expectation that remains challenging to estimate. Following prior work, we again focus on the aleatoric uncertainty, which considers the likelihood of the most likely output sequence under the given language model. It turns out that the aleatoric uncertainty is equivalent to the maximum sequence probability (MSP) previously introduced in an ad hoc manner (Fadeeva et al., 2023; 2024).
While the aleatoric uncertainty derived from the logarithmic score requires approximating an expecta- tion over the entire output sequence distribution by sampling multiple output sequences (see Eq. (5) and Eq. (6)), the one derived from the zero-one score only requires approximating the most likely output sequence under the given language model (see Eq. (8)). This distinction is crucial, as sam- pling multiple output sequences is computationally expensive, while approximating the most likely output sequence aligns directly with standard inference techniques widely used in LLMs.
For numerical stability, it is preferable to consider the logarithm of the alcatoric uncertainty in Eq. (8) and omit the constant offset of one, as does not affect the relative ordering of uncertainty estimates. This yields the negative log-likelihood (NLL) of the most likely output sequence, which we denote as
T
M(p(y x, w)) = max p(y yE
x, w)
max (81.....r)
Σ logp(ve y,1,
(9)
E
T
x
ply 3 x, w
max (T)
II
p(y Ve,,
aleatoric uncertainty
We note that this single-sequence measure also known as the min-entropy, the most conservative of the Rényi-entropies (Rényi, 1961), which is lower bound on PE in Eq. (5).
With this, the task of uncertainty estimation reduces to finding the most likely output sequence. However, the search space is the set of all possible output sequences Yr, so the exact computation remains intractable. To make the computation tractable, we propose replacing the full sequence maximization with tokenwise maximization by moving the max operator inside the summation. This yields a very efficient approximation, since it simply corresponds to standard greedy decoding with the given language model. Thus, can approximate the NIL of the most likely output sequence in Eq. (9) using greedy decoding (G), which we formally define as
G-NLL Σ t=1
log max mmxxp(y4 x, Uet, Be), M(p(y) xx w))
(10)
S.EV
5

---

Published as a conference paper at ICLR 2026
Discussion. Although uncertainty measures based on the logarithmic score could, in principle, perform well if the distribution over output sequences p x, w) or equivalently the distribution over semantic clusters p(c 2, w) were tractable (as in standard classification tasks), these distributions are intractable for NLG due to the large vocabulary size and the auto-regressive nature of L.MMs. As result, sampling-bas maaling--asedd methods often yield crude approximations, suffering from computational costs and sampling variability. In contrast, G-NLL offers a principled alternative while eliminating the need for extensive sampling, making it highly practical and straightforward, while maintaining theoretical rigor through its foundation in proper scoring rules.
In general, uncertainty measures in NLG can be defined by a predictive distribution and a proper scor- ing rule: under the logarithmic score, the aleatoric uncertainty evaluated on p(y 2, w) corresponds to PE (Eq. (5)), while evaluated on p(c æ w) it corresponds to SE (Eq. (6)). Analogously, under the zero-one score, the aleatoric uncertainty evaluated on p(y ær u)) corresponds to the maximum se- quence probability (MSP), while evaluated on p(c æ. w)) it yields the semantic counterpart of MSP, which we term the maximum cluster probability (MCP). G-NLL approximates MSP from a single out- put sequence, giving it a computational advantage over PE, whereas approximating MCP requires mul- tiple output sequences, offering such advantage over SE. We discuss this further in Apx. A.2. In the remainder of this paper, we focus on MSP and defer a detailed investigation of MCP to future work.
3 THEORETICAL ANALYSIS
Before empirically validating G-NLL in Sec. 5, analyze the sample-complexity of approximating H(.) and M(.). showing that approximating the latter as done with G-NLL is desirable in the setting of LLMs. We consider probability distribution of output sequences p(y). dropping the dependency on and w in this section for brevity. To incorporate importance sampling schemes, we consider access proposal distribution g(y), e.g., by camperature-samllinn from the LLM. We interested in estimating H(p(y)) and M(p(v) ((())) with their respective MC estimators:
M(p(y) max log p(y)
M(p(y) = max (log (ly")()))))
y
p (
1
N
ply")
H(p(y)) = --E -E(())
log p(y)
H(p(y))
log loy"("")
N
where y" ~ qly). Furthermore, we assume boundedness, i.e., there exist constants a, b such that & log pgy) due logit clipping and non-zero softmax temperatures. Then the following holds:
Theorem 1. Let logp(y) [a, b] by Yr and 0. Then, with probability 1 8, we get:
1. Sample-Complexin Bound for Maximum Log- Like-iied Estimation
C.
M(r(y)) -M((())) EE -lopp("))-o(((3))
N>
log
(11)
P
where V. ty E 3/ log logp(y") log ((y) & c).
P. = -yEwy p(y). Q. = Lyey, (yy), and C. = P./Q..
2. Sample- Sample-Compler Complexity Bound for Importance- Weithannnne-eeeeedddd Entropy Estimation
H(p())) - A(((()) SE = N> (b C2
2 log
(12)
22a
where p(y)/a(y) & C,, Vy E 3'r
Proof. 1: Probability of failure for N samples (1- NN and log(1/(1 x)) < x-1 for r E (0, 1).
2: Applying Hoeffding's inequality after bounding importance weights by their worst-case value C. Detailed derivations are provided in Apx. B.
The sample-complexity bound on estimating M(p(y)) depends on the concentration of output sequences in the e-region. This is desirable for autoregressive language generation, as sampling strategies generally focus obtaining likely output sequences. In contrast, the sample complexity bound on estimating H(p(z)) depends on the range of possible sequence likelihoods and the worst-case importance weight C' squared. Both can be very high in practical settings. Thus, regarding comple-complexity. estimating M(p(y)) appears desirable compared to estimating H(p(y)) for typical L.L output distributions.
Discussion.

---

Published as a conference paper at ICLR 2026
H(p(y))- H(p(y)))
M(p(y))) MMply))
-2
4
MS (T- 1)
MS (r= 0.7) Search
-2
Greedy Decoding
1
5
10
15
20
10
15
20
Number of samples/ / Beam width
Number of samples Beam width
(a) Estimating H(p(w))
(b) Estimating M(p(z)
Figure 1: Quality of estimators for synthetic predictive distributions p(y) with | 20 and T 4. The predictive entropy H(p(y)) is estimated as in Eq. (5) using multinomial sampling (MS) with different temperatures (r). The negative maximum sequence log-likelihood M(p(y) estimated by the maximum over N samples obtained by beam search (N = 1 represents greedy decoding) or MS with different r. Statisties are obtained by sampling 2000 different p(y) ) (a) Lines show average, shades denote one std. (b) Lines show the median, and shades denote the 5 5% to 95% quantile range.
Simulation Study. To gain insights into the practical implications of Theorem 1, we conduct a synthetic experiment where we sample predictive distributions p(y) with smaller vocabulary sizes and shorter sequence lengths, while preserving the distributional characteristics typical for LLMSs This experiment design allows us to obtain ground truths for both H))()) and M(p(y)), which become intractable to compute with larger vocabulary sizes and sequence lengths. Using this synthetic setup, we evaluate how the quality of estimators improves with the number of samples.
Fig. la summarizes the results for estimating H(p(y)), derived from the logarithmic score. The results show that low sample sizes lead to high estimator variance. Similarly, Fig. 1b summarizes the results for estimating M(p(y)), derived from the zero-one score. The results indicate that heuristics such as beam search and even greedy decoding provide accurate estimates of M(p(y) with high probability, In contrast, the variance of estimating H(p(y)) remains substantial even when considering multiple samples and further increases when sampling with a lower temperature. Details on the sampling procedure of the predictive distributions, as well as additional experiments are provided in Apx. C.
4 RELATED WORK
In Sec. 1, we discussed prior work that considers single-sequence measures as heuristic baselines (Fadeeva et al., 2023; 2024; Vazhentsev et al., 2024; Abbasi- Yadkori et al., 2024; Plaut et al., 2024; Vashurin et al., 2025a;b). yet there is no clear consensus on how to apply them in principled way. For instance, Ren al. (2023) investigate length-normalized sequence likelihood (i.e., perplexity) and Zhang et al. (2025) use likelihood ratios between pre- rre-trained and fine-tuned LLMs for OOD detection, but neither considers the MSP. Likewise, (Chen et al., 2024; Bakman et al., 2024; Yaldiz al., 2025) include only perplexity as baseline without specifying how the output sequence is obtained. Qiu and Miikkulainen (2024) also compare against a length-normalized variant, using the normalization from Murray and Chiang (2018), which corrects length bias for neural machine translation rather than uncertainty quantification. By grounding the MSP in theory and guiding its best approximation, we provide foundation for advancing single-sequence uncertainty measures.
In Sec. 2.2, we also elaborated on the established sampling-based uncertainty estimation measures, i... PE (Malinin and Gales, 2018) and SE (Kuhn et al., 2023; Farquhar et al., 2024). There is a body of work that builds upon the concept of PE, for instance, by considering weighting factor for individual token and sequence likelihoods to account for the importance on semantic level (Duan et al., 2024; Bakman et al., 2024; Yaldiz et al., 2025). There is also body of work that extends the concept of SE (Kuhn et al., 2023; Farquhar et al., 2024), for instance, by improving the semantic clustering (Nikitin et al., 2024; Qiu and Mikkkulainen, 2024), improving the sampling of diverse output sequences (Aichberger et al., 2025), or directly approximating the measure from hidden states of the language model (Kossen et al., 2024; Chen et al., 2024). Since our goal is to compare the uncertainty principles induced by the underlying proper scoring rules, we focus on their established forms for a fair and consistent comparison.
7

---

Published as a conference paper at ICLR 2026
There is also work uncertainty estimation in NLG that cannot be directly grounded in proper scoring rules. For instance, several approaches leverage the language model itself to directly predict uncertainty, either through numerical estimates verbal explanations (Mielke et al., 2022; Lin et al., 2022; Kadavath et al., 2022; Cohen et al., 2023a; Ganguli et al., 2023; Ren et al., 2023; Tian et al., 2023). Cohen et al. (2023b) use cross-examination, where one model generates an output, and another evaluates it, while Manakul et al. (2023) assess uncertainty by feeding multiple sampled outputs into second model. Zhou et al. (2023) explore the behavior of LLMs when expressing their uncertainty, providing insights into how models articulate confidence. Xiao et al. (2022) analyze how factors such as model architecture and training data affect uncertainty estimates. Gao al. (2024) perturb inputs to quantify uncertainty, and Chen and Mueller (2024) rely on consistency and self-reflection under different prompting strategies. Jiang et al. (2024) propose Graph Uncertainty, viewing individual claims in an output sequence as nodes in a bipartite graph. Finally, conformal prediction (Quach et al., 2024) provides a calibration- -abbre rule for determining when to stop sampling during generation.
EXPERIMENTS
We aligned our experiments on evaluating uncertainty estimation methods with prior work by focusing on free-form question answering tasks (Kuhn et al., 2023; Duan et al., 2024; Bakman et al., 2024; Nikitin et al., 2024; Aichberger et al., 2025; Kossen et al., 2024). While Farquhar et al. (2024) additionally concerns experiments with paragraph-length generations, their approach breaks down the paragraph into factual claims and constructs corresponding questions. Therefore, performance on this task is expected to align with general free-form question answering tasks, and we therefore focused on those for a clearer and more direct evaluation. This focus further avoids potential confounding factors introduced by additional experimental complexities.
Datasets. We evaluated uncertainty estimation methods on three different datasets. We used the more than 3,000 test instances from TriviaQA (Joshi et al., 2017) concerning trivia questions, the more than 300 test instances from SVAMP (Patel et al., 2021) concerning elementary-level math problems, and the more than 3,600 test instances from NQ-Open (Lee et al., 2019) assess natural questions aggregated from Google Search. Each dataset was used for two distinct tasks: (I) generating concise answers in the form of short phrases (short) and (2) generating more detailed answers in the form of full sentences (long), following the experimental setup in Farquhar et al. (2024). The six individual tasks were selected so that they align with prior work to provide a fair and meaningful comparison. They aim at covering a broad range of real-world scenarios, ensuring comprehensive evaluation.
Models. We conducted our evaluations on six different I.LMs covering various architectures, sizes, and training stages. Specifically, we used the transformer model series Llama-3. / (Dubey et al., 2024) and the state-space model series Falcon Mamba (Gu and Dao, 2024; Zuo et al., 2024), representing two prominent paradigms. To assess the effect of training stage and scale on uncertainty estimation in NLG, we considered pre- pre-trained (PT) and instruction-ture (/T) LLMs with 7, 8, and 70 billion parameters, covering a wide spectrum of model characteristics used in real-world applications.
Baselines. We compare our method, G-NLL, against the commonly used uncertainty measures based on the logarithmic score as of Eq. (5) and Eq. (6) and their variants. These include predictive entropy (PA), length- predictive entropy (LN-PEE (Malinin and Gales, 2021), semantic entropy (SE), sennth-normalizee
semantic entropy (LN-SE), and Discrete semantic entropy (D-SE) (Kuhn et al., 2023; Farquhar et al., 2024). For a given output sequence y', the length-normalized variants consider p(y' x. w) instead of p(y' x, w) to compute the uncertainty estimates. D-SE completely disregards the likelihood of the output sequence and only considers the proportion of output sequences that belong to the same semantic cluster (Farquhar al., 2024). These baselines represent direct counterparts derived from alternative proper scoring rules as discussed in Sec. 2.2. More importantly, they are among the most widely used uncertainty estimation methods in NI.G. Comparing against these established methods provides decisive proxy for the empirical performance of G-NLL.
Evaluation. Effective uncertainty measures should accurately reflect the reliability of answers generated by the language model. In other words, higher uncertainty should correspond to a greater tendency for incorrect outputs. Thus, to evaluate the performance of an uncertainty estimator, we assess how well the uncertainty estimate correlates with the correctness of the language model's s answer. Correct answers should be assigned a lower uncertainty estimator than incorrect answers. We consider a given answer correct if the F1 score of the commonly used SQUAD metric to the ground
8

---

Published as a conference paper at ICLR 2026
Table Average AUROC (1) with standard errors across TriviaQA, SVAMP, and NQ datasets, using uncertainty estimates to distinguish between correct and incorrect answers. Six different language models, with varying model architectures (transformer, state-space), model sizes (7B, 8B, 70B), and training stages (PT, ITT) are considered. The reference answer is generated using greedy decoding, either as a whole sentence (long) or a short phrase (short). Its correctness is assessed by F/ score using SQuAD 0.5 as decision rule or LLM-aa---judde (LLM).. PE, LN-PE, SE, LN-SE, and D-SE use 10 output sequences (generated via multinomial sampling) to obtain their uncertainty estimates. G-NLL solely uses one output sequence (generated via greedy decoding) for its uncertainty estimate. indicates that G-NLL performs significantly better than the best sampling- method (p < 0.05).
Underlying proper scoring rule
Logarithmic SE
Zero-
Language Model Gen.
Metric
PE
LN-PE
LN-SE
D-SE
G-NLL
short
Fl
.776 1
.795 008
.775 : 008
.793 1.008
.804
8009
.824 1ow"
PT
short
LLM
.698
2.014
.714
0044
.690 2.014
.706 1.014
.719
.014
.726 20014
8B
long LLM
.562
+0012
.555
+0012
.545
+ +.013
.553
+.012
.600
+.012
.649
+.012"
Transformer
short
F1
.772 000
.801
007
.805
.07
.814
.806
.088
.838 006
IT
short
LLM
.676
+ -015
.697
.015
.704
+ +.015
.709
+.015
.694
.016
.722 -014
long
LLM
.551 1.012
.548
.012
.599
1.012
.601
1.012
.609
1.012
.615 1:012
short
F1
.775
.790
.010
.793
009
.803
4.009
.791
.009
.820 .008
PT
short
LLM
.693
+. 019
.709
.099
.718
a 019
.722
4.020
.715
.018
.723 #:019
70B
long
LLM
.552
.014
.534
0014
.558 + 015
.569
+.013
.571
.013
.649
-012"
short
Fl
.748
.10
.781
.088
.790
009
.799
+.009
.783
.009
.792 .011
short
LLM
.681
1.021
.698
.022
.703
1 022
.709
1.002
.699
L.1
.699
long
LLM
.55
.557
-014
.568
.595
.600
.562 .014
short
F1
811 0.007
.815
007
.809
000
.822 .0.08
.828
1,006
.843 000
State-Space
PT
short
LLM
.705
+002
.711 +0011
.701
+ +.012
.711 +.012
.716
-012
.728 #0011
long
LLM
.567
1.0012
.597 1012
.574
012
.611
1.012
.624
.012
.612 2:012
7B
short
F1
.793 1.000
.814
.007
.797 1,008
.816
1.008
.829
1.007
.838 1.000
IT
short
LLM
.690 #:013
.701 =0012
.689
+ 4.012
.699
+ #.012
.711
#.013
.719 #:012
long
LLM
.888
.587 1.012
.597
1.012
.618
.012
.629
.012
.615 2.012
Average
.677 000
.689
.003
.690
003
.703
4.003
.707 .003
.721 0000
truth answer exceeds 0.5 (Rajpurkar et al., 2016). Since this metric is only applicable for short-phrase generations that align with the ground truth answer, additionally employ Llama-3.1 with 70 billion parameters (Dubey et al., 2024) as LLM-as-a-judge to assess the correctness of both short-phrase and full-sentence generations. Finally, measure the correlation between the incorrectness of answers and the respective uncertainty estimates, we use the AUROC. Higher AUROC values indicate better performance of the uncertainty estimator, as reflects a stronger alignment between the correctness of the language model's answers and their respective uncertainty estimates. This evaluation process follows established protocols for uncertainty estimation NLG (Kuhn et al., 2023; Farquhar et al., 2024; Duan et al., 2024; Bakman et al., 2024; Nikitin et al., 2024; Kossen et al., 2024).
5.1 MAIN RESULTS
Tab. 1 summarizes the performance of the uncertainty measures across the six LLMs, six tasks, and two evaluation metrics. We report the average AUROC across the three datasets, highlighting the best-performing measure in bold. Additionally, the best-perforning measure based the logarithmic score is underlined, unless it also represents the overall best. In 13 out of 18 scenarios, G-NLL outperforms the uncertainty measures and remains competitive in the remaining scenarios. This strong performance is particularly evident in tasks involving the generation of short phrases, suggesting its effectiveness in capturing the essential part of the output sequence that contains the factual answer to a question. This is especially valuable in practical scenarios, where the uncertainty about a specific fact often more critical than the uncertainty about the entire generated sentence. Overall, our measure significantly outperforms all other measures when considering the average across all scenarios. This demonstrates that G-NLL achieves strong empirical performance despite relying only single output sequence. Crucially, G-NLL relies only on the greedily decoded output sequence, yielding a deterministic and hyperparameter-fie free uncertainty estimate. Detailed evaluation results are provided in Apx. D.

---

or normalizr variants can degrade Published as a conference paper at ICLR 2026
5.2 ABLATION STUDY ON APPROXIMATION VARIANTS
The strong empirical performance of G-NLL on question answering NLG tasks suggests that effectively captures key aspects of uncertainty, even when using significantly fewer output sequences compared to uncertainty measures based on the logarithmic score. To examine the underlying factors driving this behavior, analyze the impact of the sampling method on approximating the measure of aleatoric uncertainty M(p(y) æ; w) derived from the zero-one score in Eq. (8).
Specifically, we utilize multinomial sampling (MS) with different temperatures т, beam search (BS) with different beam sizes, and greedy decoding as for G-NLL. For each sampling method, we generate single output sequence per instance in the TriviaQA dataset. We then the corresponding NLL as the uncertainty estimate, following the same evaluation process as in the main experiments above. Notably, the baselines are unaffected by the choice of sampling method for computing the NLL, as we again use their optimal hyperparam- eter settings for sampling.
0.82
0.80
The results in Fig. show that better approxi- mations of the most likely output sequences in- deed lead to higher uncertainty estimation per- formance, reinforcing the validity of our alter- native measure derived from the zero-one score. Additionally, it can be observed that sampling output sequences using greedy decoding sig- nificantly outperforms MS. While performance improves further with BS, as anticipated, the marginal benefits are relatively small. This sup- ports the claim that greedy decoding provides strong approximation sequence. Since BS is computationally more ex- pensive (as its beam size corresponds to the num- ber of sampled output sequences), using greedy decoding in G-NLL achieves the best trade-off between effectiveness and efficiency.
AUROC
0.78
0.76
NLL
LN-SE D-SE
LN-PE PE
SE
0.74
MS (r#1.0)
MS (r=0.5)
Greedy Decoding
BS (5 beams)
Bs (10 beams)
BS (20 beams)
the most likely output
Figure 2: Average AUROC (1) using uncertainty measures based on the zero-one score, with the out- put sequence to approximate the MSP being gener- ated with multinomial sampling (MS), greedy de- coding (G-NLL), or beam search (BS), compared to baseline measures based on the logarithmic score.
6 CONCLUSION
In this work, we theoretically motivate an alternative uncertainty measure: the NLL of the most likely output sequence under given language model. The measure is grounded in the general notion of proper scoring rules, with the zero-one score serving as an alternative to the commonly adopted logarithmic score. Unlike multi-sequence uncertainty measures, it can be efficiently approximated with G-NLL from a single greedily decoded sequence, challenging the widespread use of sampling- and clustering- -hased approaches for uncertainty estimation in NL.G. Experiments show that G-NLL outperforms prior methods that entail considerably higher computational costs and algorithmic complexity. Importantly, we find that the choice of decoding strategy to approximate the most likely output sequence is critical, and sampling- approximation quality.
While G-NLL effectively captures uncertainty, it does not explicitly account for semantics. Exploring extensions that incorporate semantic information could further enhance single-sequence uncertainty estimation, but would likely increase computational cost (see Sec. 2.3). and we leave exploring this trade-off for future work. Moreover, G-NLL requires access to the probabilities of generated tokens, though not the full token distribution. While such information is typically exposed by modern LLM APIs, future work may investigate how to approximate when unavailable.
While there remain opportunities for refinement, G-NLL provides a principled foundation for efficient uncertainty estimation in NLG. Its simplicity and minimal computational overhead make it a practical baseline for developing new uncertainty measures and support scalable deployment in real-world ap- plications. More broadly, our work shows that single-sequence measures serve as a viable alternative to sampling-har based methods, offering principled way to estimate uncertainty efficiently in LLMss.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We acknowledge EuroHPC Joint Undertaking for awarding us access to Karolina at IT44Inovvtiioss Czech Republic. The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are sup- ported by the Federal State Upper Austria. We thank the projects FWF AIRI FG 9-N (10.55776/FG9) ALAGreenHeatingGridd (FFG- 899943), Stars4Waters (HORIZON-CL6-20)) 10-2221-CLIMATT-O0III FWF Bilateral Artificial Intelligence (10.55776/COE12) We thank NXAI GmbH, Audi AG, Sili- con Austria Labs (SAL). Merck Healthcare KGaA. GLS (Univ. Waterloo), TÜV Holding GmbH, Software Competence Center Hagenberg GmbH, dSPACE GmbH, TRUMPF SE + Co. KG.
ETHICS STATEMENT
This work contributes to advancing the efficiency and reliability of uncertainty estimation in LLMs. Our proposed measure, G-NLL, reduces computational overhead compared to existing approaches, enabling broader adoption of uncertainty estimation techniques in real-world applications. While we recognize societal and ethical risks associated with LLMs, such as misinformation, our contribution seeks to mitigate these risks by supporting more trustworthy deployment through improved uncertainty estimation. Our work does not involve human subjects, sensitive data, or personally identifiable information. All datasets used publicly available and commonly employed in prior work on NLG.
REPRODUCIBILITY STATEMENT
We provide detailed descriptions of datasets, experimental setup, and evaluation metrics in Sec. 5 of the main paper and Apx. D. All datasets used are publicly available, and we utilize standard benchmarks to support straightforward replication. All experiments were performed on single node with 8 NVIDIA A100 Tensor Core GPUs. Running all evaluations required roughly 100 node hours.
REFERENCES
Yasin Abbasi- Yadkori, Ilja Kuzborskij, András György, and Csaba Szepesvari. To believe or not to believe your LLM: Iterative prompting for estimating epistemic uncertainty. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Lukas Aichberger, Kajetan Schweighofer, Mykyta lelanskyi, and Sepp Hochreiter. Improving
uncertainty estimation through semantically diverse language generation. In The Thirteenth International Conference on Learning Representations, 2025.
Yavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, and Salman Avestimehr. MARS: beeninggawaat response scoring for uncertainty estimation in generative LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 7752-7767. Association for Computational Linguistics, 2024.
Chao Chen, Kai Liu, 7e Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and lieping Ye. INSIDE: L.MSs' internal states retain the power of hallucination detection. The Twelfth International Conference Learning Representations, 2024.
Jiuhai Chen and Jonas Mueller. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers). Association for Computational Linguisties, 2024.
Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. Crawling the internal knowledge- base of language models. Andreas Vlachos and Isabelle Augenstein, editors, Findings of the Association for Computational Linguistics: EACL 2023, pages 1856-1869. Association for Computational Linguistics, 2023a.
Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. LM vs LM: Detecting factual errors via cross examination. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the
11

---

Published as a conference paper at ICLR 2026
2023 Conference Empirical Methods in Natural Language Processing, pages 12621-12640. Association for Computational Linguistics, 2023b.
Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 5050-5063, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and Aurelien Rodriguez et al. The llama 3 herd of models. arXiv, 2407.21783, 2024.
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Flizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. I.M-polyrrahh: Uncertainty estimation for language models. In Yansong Feng and Els Lefever, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 446-461, Association for Computational Linguistics, 2023.
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslay Nakov, and Maxim Panov. Fact-checking the output of large language models via token-level uncertainty quantification. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 9367-9385. Association for Computational Linguisties, 2024.
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625 630, 2024.
Yarin Gal. Uncertainty in deep learning. PhD thesis, University of Cambridge, 2016.
Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamilé Lukošiūté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez. Dawn Drain, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jared Mueller, Joshua Landau, Kamal Ndousse, Karina Nguyen, Liane Lovitt, Michael Sellitto, Nelson Elhage, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robert Lasenby, Robin Larson, Sam Ringer, Sandipan Kundu, Saurav Kadavath, Scott Johnston, Shauna Kravec, Sheer El Showk. Tamera Lanham, Timothy Telleen-Lawton, Tom Henighan. Tristan Hume, Yuntao Bai, Zac Saufield-Dodde Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Christopher Olah, Jack Clark, Samuel R. Bowman, and Jared Kaplan. The capacity for moral self-correction in large language models. arXiv,, 2302.07459, 2023.
Xiang Gao, Jiaxin Zhang, Lalla Mouatadid, and Kamalika Das. SPUQ: Perturbation- based uncertainty quantification for large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2024.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):35--378 2007.
Sebastian Gruber and Florian Buettner. Uncertainty estimates of predictions via a general bias- variance decomposition. In Francisco Ruiz, Jennifer Dy, and Wil-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206, pages 1333-11134, PMLR, 2023.
Albert Gu and Tri Dao. Mamba: Linear-tinn sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024.
Paul Hofman, Yusuf Sale, and Eyke Hüllermeier. Quantifying aleatoric and epistemic uncertainty with proper scoring rules. arXiv, 12215, 2024.
12

---

Published as a conference paper at ICLR 2026
Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv, 1112.5745, 2011.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110:457-506. 2021.
Mingjian Jiang. Yangjun Ruan, Prasanna Sattigeri, Salim Roukos, and Tatsunori Hashimoto. Graph- based uncertainty metrics for long-form language model generations. Advances in Neural Information Processing Systems, 2024.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings the 55th Annual Meeting of the Association for Compulational Linguislics. Association for Computational Linguistics, 2017.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds Nova DasSarma, Eli Tran-Johnsone Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know. arXiv, 2207 05221,2211112 2022.
Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in llms. arXiv, 2406. 15927, 2024.
Nikita Kotelevskii, Vladimir Kondratyev, Martin Takáč, Eric Moulines, and Maxim Panov. From risk to uncertainty: Generating predictive uncertainty measures via bayesian estimation. In The Thirteenth International Conference Learning Representations, 2025.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representutions, 2023.
Meelis Kull and Peter Flach. Novel decompositions of proper scoring rules for classification: Score adjustment as precursor to calibration. In Annalisa Appice, Pedro Pereira Rodrigues. Vítor Santos Costa, Carlos Soares, João Gama, and Alípio Jorge, editors, Machine Learning and Knowledge Discovery in Databases, pages 68-85 Springer International Publishing, 2015.
Salem Lahlou, Moksh Jain, Hadi Nekoei, Victor Butoi, Paul Bertin, Jarrid Rector-Brook Maksym Korablyov, and Yoshua Bengio. DEUP: Direct epistemic uncertainty prediction. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Anna Korhonen, David Traum, and Lluís Marquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 6086-6096. Association for Computational Linguistics, 2019.
Stephanic Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
Andrey Malinin. Uncertainty estimation in deep learning with application spoken language assessment. PhD thesis, University of Cambridge, 2019.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianch, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In International Conference Learning Representations, 2021.
13

---

Published as a conference paper at ICLR 2026
Potsawee Manakul, Adian Liusie, and Mark Gales. SelfCheckGPT: rero-resource black-box halluci- nation detection for generative large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004-9017 Association for Computational Linguistics, 2023.
Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, -722 2022.
Kenton Murray and David Chiang. Correcting length bias in neural machine translation. In Ondiej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor, editors, Proceedings of the Third Conference on Machine Translation: Research Papers, 2018.
Alexander V Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. Kernel language entropy: Fine-graine Fine-grained uncertainty quantification for LLMs from semantic similarities. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 202/ Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, pages 2080-2094. Association for Computa- tional Linguistics, 2021.
Benjamin Plaut, Nguyen X. Khanh, and Tu Trinh. Probabilities of chat Ilms are miscalibrated but still predict correctness on multiple-choice q&a. arXiv, 2402. 13213, 2024.
Xin Qiu and Risto Miikkulainen. Semantic density:
Uncertainty quantification for large language models through confidence measurement in semantic space. In The Thirty-eighth Annual Confer- ence on Neural Information Processing Systems, 2024.
Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, and Regina Barzilay. Conformal language modeling. In The Twelfth International Conference on Learning Representations, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2018.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392. Association for Computational Linguistics, 2016
Jie Ren, Yao Zhao, Tu Vu, Peter J. Liu, and Balaji Lakshminarayanan. Self-evaluation improves selective generation in large language models. arXiv, 2312.09300, 2023.
Alfréd Rényi. On measures of entropy and information. In Proceedings of the 4th Berkeley Symposium on Mathematics, Statistics and Probability. pages 547-561. University of California Press, 1961.
Kajetan Schweighofer, Lukas Aichberger, Mykyta lelanskyi, and Sepp Hochreiter. Introducing an improved information-theorrtiie measure of predictive uncertainty. arXiv, 231 1.11..00 1.08009, 2023a.
Kajetan Schweighofer, Lukas Aichberger, Mykyta lelanskyi, Günter Klambauer, and Sepp Hochreiter. Quantification of uncertainty with adversarial models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances Neural Information Processing Systems, volume 36, pages 19446-19484. Curran Associates, Inc., 2023b.
Kajetan Schweighofer, Lukas Aichberger, Mykyta lelanskyi, and Sepp Hochreiter. On information- theoretic measures of predictive uncertainty. Uncertainty in Artificial intelligence, 2025.
14

---

Published as a conference paper at ICLR 2026
Hya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1433-5442 Association for Computational Linguisties, 2023.
Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing. Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. Benchmarking uncertainty quantification methods for large language models with LM-polygraph. Transactions of the Association for Computational Linguistics, 2025a.
Roman Vashurin, Maiya Goloburda, Preslav Nakov, Artem Shelmanov, and Maxim Panov. Cocoa: A generalized approach uncertainty quantification by integrating confidence and consistency of Ilm outputs. arXiv, 2502.04964, 2025b.
Artem Vazhentsev, Ekaterina Fadeeva, Rui Xing. Alexander Panchenko, Preslav Nakov, Timothy Baldwin, Maxim Panov, and Artem Shelmanov. Unconditional truthfulness: Learning conditional dependency for uncertainty quantification of large language models. arXiv, 2408. 10692, 2024.
Yuxin Xiao, Paul Pu Liang. Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis- Philippe Morency. Uncertainty quantification with pre-trained language models: A large-scale empirical analysis. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7273-7284. Association for Computational Linguistics, 2022.
Duygu Nur Yaldiz, Yavuz Faruk Bakman, Baturalp Buyukates, Chenyang Tao, Anil Ramakrishna, Dimitrios Dimitriadis, Jieyu Zhao, and Salman Avestimehr. Do not design, learn: A trainable scoring function for uncertainty estimation in generative LLMs. In Findings of the Association for Computational Linguistics: NAACL 2025, 2025.
Andi Zhang, Tim Z. Xiao, Weiyang Liu, Robert Bamler, and Damon Wischik. Your finetuned large language model is already a powerful out-of-distributtion detector. In Proceedings of The 28th International Conference on Artificial Intelligence and Statistics. PMLR, 2025.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. arXiv,
2022.
Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. Navigating the grey area: How expressions of uncertainty and overconfidence affect language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5506-550 5524. Association for Computational Linguistics, 2023.
Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, Ilyas Chahed, Younes Belkada, Guillaume Kunsch, and Hakim Hacid. Falcon mamba: The first competitive attention-free 7b language model. arXiv, 2410.05355, 2024.
15

---

Published as a conference paper at ICLR 2026
USAGE OF LARGE LANGUAGE MODELS (LLMS)
LLMs were used as general-purpose assistive tools during the preparation of this paper. Their use fell into two categories: (i) writing assistance, where they were used to improve the clarity and readability of certain passages through language refinement, and (ii) coding assistance, where they were used for support with code completion and debugging. LLMs were not used for research ideation, experimental design, theoretical development, result analysis. All substantive contributions, including the conception of ideas, methodology. and experiments, were made by the authors.
A DETAILS ON PREDICTIVE UNCERTAINTY IN NLG
As stated in Sec. 2, the total uncertainty of given language model is defined as
Ew Ey [S(p(y) x, w)) y')ll
where we define Eü := Eu-)p(((D)) and En. == Eurrp((""la,u)) for brevity. This corresponds to the expected score that models from the posterior pori (D) assign to output sequences y drawn from the predictive distribution of the given model p(y' (r, w.)
Adding and subtracting Fag [S(p(y æ. w)) y²)] yields the general scoring-rule- uncertainty decomposition Eq. (2) of the main paper:
Enp [Ey (S(p(y) ar, (.) y'Il
expected score
Ey [S(p(y) æ. w) Ev [Ey [S(p(y) a. wr, v')]] By (S(p(y) az w), Ey [S(p(y) 28, w) + Ew Ey [S(p(v) x .). - S(p(y) xx w),
entropy torm
divergence
A.1 UNCERTAINTY MEASURES BASED ON THE LOGARITHMIC SCORE
To derive the established uncertainty measures in Sec. 2.2, we substitute the logarithmic score
Sing (p(y) x, ),,') = log yy x;,
(3)
into the general scoring-rule wwwee-uulaaaaaaeee based uncertainty decomposition (Eq. (2)) to get:
Eu [ [- log'(y' x, []]
Ey | p3y æ. w)] + Ev [Ey |- logp(y' x, w) + log ply x, we]
On the LHS, we note the definition of the cross-entropy
Ey | log p(y' x, w)) = CE(p(y x, w):p(y x, i))
On the RHS, we note that the first term is the definition of the Shannon entropy
Ey logp(y x; w)] = H(p(y x, w)),
and the second term is the definition of the Kullback-Leible divergence
Eg [Ey [- log p(y'
x. () + log
x, w)]] = KL(p(y x, w) p(y x; ())
Using these definitions directly results in Eq. (4) of the main paper:
Egg [CE(p(y) x, w x, T))
total uncertainty
H(p(y xr, w)) + Emn [KTpp xc, w) p( ac, x)))
aleatoric uncertainty
epistemic uncertainty
16

---

Published as a conference paper at ICLR 2026
A.2 UNCERTAINTY MEASURES BASED ON THE ZERO-ONE SCORE
To derive the new uncertainty measures in Sec. 2.3, we substitute the zero-one score
(p )) y (1 p y x -)) ( argmax p(y x,
into the general scoring-rule-basee uncertainty decomposition (Eq. (2)).
For clarity, we distinguish between the most likely output sequence under the given model
y := argmax p(y x; w)
y/
and the most likely sequence under a model v drawn from the posterior p(u D)
y argmax ply x, w.)
Because the indicator enforces y' - y or y = D" the inner expectation over y collapses, yielding [So1 (p(y x. a), y')l
ply "" w)
Ey [So- (p(y x. w) y')l = p(u y² w
Using this these definitions directly results in Eq. (8) the main paper:
Es [1 - p = "" æ, w)]
(8)
total uncertainty
p y x. w
+ ply y²
x; w) E. (p(y)
ÿ"
x, w)
aleatoric uncertainty
epistemic uncertainty
Incorporating Semantics. As discussed in Sec. 2.1, uncertainty measures are determined by predictive distribution together with a proper scoring rule. In place of the output-sequence distribution p(y æ, w). we may therefore consider the semantic cluster distribution p(c æ, w) to incorporate semantic information into the uncertainty measures based on the zero-one score.
In this setting, the zero-one score evaluates the predictive distribution at its most likely semantic cluster rather than at the most likely output sequence:
So. x. ),, (1 - p(c = e' x. .)) 1(d' = argmax p(c x. .)
(13)
For clarity, we again distinguish between the most likely semantic cluster under the given model and that under model drawn from the posterior:
== argmax p(c x, w) E" == argmax p(c æ. T))
y
y
Substituting the zero-one decomposition (Eq. (2)) yields the following decomposition of total semantic uncertainty, the expected confidence that the given model assigns the most likely semantic cluster predicted by models drawn from the posterior ( D):
defined in Eq. (13) into the general scoring-rule rraninngrule-bbseee uncertainty
Pp(epp) [1
p(c 2t xc, w))
(14)
tntal semantic uncertainty
p(c c' 2, w) p(c= xx w) - Ep( o* x; w))
alcatoric uncertainty
epistemic semantic uncertainty
As in Eq. (8), the epistemic semantic uncertainty a posterior expectation that challenging to estimate. The alcatoric semantic uncertainty considers the likelihood of the most likely semantic cluster under the given language model, i.e, the most-likely cluster probability (MCP), analogous to maximum sequence probability (MSP).
However, approximating the most likely semantic cluster more challenging, since the semantic cluster distribution p(c x, w) cannot be sampled from directly (Aichberger et al., 2025). Whether MCP admits an efficient approximation remains an open question. While MSP (as approximated by G-NLL) may correlate with MCP, detailed analysis of this relationship is left to future work.
17

---

Published as a conference paper at ICLR 2026
B DETAILS ON THEORETICAL ANALYSIS
We want to compare the sample complexity of approximating the maximum and expected log- likelihood, showing that approximating the maximum is desirable in the setting of LLMs. In the following, we derive probabilistic concentration bounds on the number of samples N needed to approximate each quantity within e-precision with high probability and use these to reason about the sample complexity required for reliable approximation.
Setup. As detailed in Sec. in the main paper, we consider the probability distribution p(y | x;, w) of output sequences y induced by the LLM with parameters w for an input sequence z. We abbreviate this distribution by p(y). We want to study the approximation error for the negative maximum log- likelihood M(p(y)
maxy log p(y) (min-entropy) and the negative expected log-likelihood H(r(z) = ppy) (oy)(y) (Shannon entropy) using samples.
We assume boundedness, i.c. there exist constants a, 6 such that & logp(y) < b Vy Vr. This assumption generally holds in practice as softmax logits are typically clipped and the softmax temperature non-zero, though the constants can of large magnitude depending on the sequence length T. Furthermore, to incorporate importance sampling schemes (e.g. low-temperature sampling, top-k or top-p sampling, beam search, etc.), we consider access to a proposal distribution g(y). This helps understand how practical decoding choices impact concentration behavior.
B. APPROXIMATING THE MAXIMUM LOG-LIKELIIOOS (MIN-ENTROPY)
Let y" = argmax, p(y) and P. = = Eyey, p(y) as well as Q. = Ewey, (3y) where V. = fy E Pr logp(y")) log p(y) < <}. Thus V. is the set of output sequences with a log- log-likelihood within e of the maximum log-pro pee--bbbbbiit... and P.. Qu are the cumulative probability masses under sampling distributions and of those output sequences. The correction factor C. = Pe/ Pe/Qe compares the c-regions under distributions p and q. If they match, C. 1, if q less likely to provide samples within the e-region, C. > 1 and if it is more likely C. < 1. We empirically estimate M(p(y)) with M(p(y)) max (log p( = the maximum sequences.
the log-probabilities of the sampled output
When sampling from (vy, the probability that no y V. is not obtained in N samples is
(1 - Qc) <6.
(15)
Solving for N, we obtain
N log(1/8)
(16)
log(1/(1 Q+))
Furthermore, we use Qe = Pe/C, and the inequality further.
< x- 'or x E (0, 1) to simplify
Result. To ensure with probability at least 8 that M(p(z)) M(((z)) < it suffices that
C. !
N Σ P.
log
8
(17)
We will discuss the practical implications of this bound in Sec. B.3.
B.2 APPROXIMATING THE EXPECTED LOG-LIKELIHOOD (SHANNON ENTROPY)
We consider importance sampling with (z)) thus
p(y) log p(t) (g())
H(p(y)) Epp()) [log = Ey(v)
and MC estimator
N m
H(p(y)) = N n=
log (og((y))
18

---

Published as a conference paper at ICLR 2026
with y" ~ (z)). Furthermore, let c(y)
p(g) ((v)
be bounded, i... c(y) & C, C. Vr.
Hoeffding's ' inequality states that
P E Xm [[Xn] _c) 2xxp (-10- 2Nc
(18)
- a)3
for a random variable X, with a & X, & b. We X,, = c(y))) log p(y") such that aC & Xn & bC and use the definitions of H(p(y)) and H(p(y)). obtaining
2NC a)
P ((( H(p(v)) - 2 20) < exx (-7 - a>b-a)
(19)
We set the r.h.s. < 6 and solve for N to arrive at the desired bound.
Result.
To ensure with probability at least
8 that [((p()) H(r(z)) < it suffices that
(b c-a log
N
(20)
2c
8
Next, we discuss the practical implications of this bound and contrast to Eq. (17).
B.3 COMPARING THEORETICAL APPROXIMATION QUALITY
The bounds in Eq. (17) and Eq. (20) quantify the number of samples needed achieve given approximation error with high probability 6. By analyzing how the bounds scale with properties of p and the sampling distribution q. we can compare the relative difficulty of estimating each quantity.
The bound for M(p(y)) in Eq. (17) depends only the concentration of the output sequence distribution along a few probable sequences within the e-region. This is amenable to decoding practices LLMs such as top-k, top-p, low-temperature, or even beam search, that focus on obtaining very likely output sequences.
The bound for Hp(()) in Eq. (20), in contrast, reflects the variance of the entire distribution and depends on the squared range log p(y) scaled by the worst-case importance weight C², both of which can be very high in practice. As it aggregates over the entire support of p(y), estimation is sensitive to rare but non-negligible sequences that are unlikely to be sampled under practical sampling strategies.
Regarding importance sampling, approximating M(p(y)) also desirable. Here, suffices that the proposal distribution q concentrates around the most likely sequences according to p and there importance weights in the calculation of (p(()). Importance sampling only increases the likelihood of sampling close to the maximum, not the calculation of the estimator itself. In contrast, when approximating H(r())), it is necessary re-weight the samples using importance weights c(y). Under a less optimal this can lead to strong increase in variance.
Overall, from sample complexity point of view, approximating M(p(y) is not only more efficient than approximating H(p(y)), but also well aligned with the way LLMs are used in practice.
19

---

Published as a conference paper at ICLR 2026
C DETAILS ON SIMULATION STUDY
In this section, we provide detailed insights into the numerical results presented in Sec. 3, especially Fig. 1.
To recall, we empirically investigate the performance of estimators for the predictive entropy H(p(y) (3)) (Eq. (5)) and the negative maximum sequence log- oogliikelhhod (min-entropy) M(p(y)) (Eq. (10) in controlled setting. Therefore, we consider synthetic experiment with the following setup. We are given a space of possible outcomes V with | = {20, 100}. The task is to predict sequence y (U1....TT) 3'r where V and T is 2, 3, or 4. Predictive distributions p(y) are not represented by a neural network, but are randomly sampled according
Dirichlet distribution Dir (((a) apy }).
The alpha parameters of the Dirichlet distribution are specified to yield typical predictive distributions as encountered LLMs that follow Zipf distribution. For [ = 20 we have 0122 = 10 and -3 0.2. For [V| 100 we have 0,22 10, 03-6 1 and ar- 100 0.2. Note that the order of alpha values randomly shuffled before drawing each predictive distribution. Representative predictive distributions sampled from this Dirichlet distribution shown in Fig. 3a and Fig. 3b.
The experiments investigate the quality of the estimators depending on the number of samples {ym) This is feasible because the ground truth values for both entropy and maximum sequence log- tog-likelihood can be calculated for this small synthetic example through exhaustive enumeration. For the experiments we present here in the appendix, new sets of samples {yn J are drawn to calculate the respective estimators. As beam search is deterministic, it does not vary in this experimental setting, compared to Fig. 1 in the main paper, where we investigated the quality of estimators for different p(y).
average over 2,000 runs, meaning that
The results for estimating the entropy are shown Fig. We observe that the variance of estimators increases for larger vocabulary sizes [V] and sequence lengths T'. Furthermore, lower temperatures increase the variance of the estimator as expected. Note that we introduced clipping for importance weights at le + 6, which did not introduce noticeable bias, but made results numerically stable.
The results for estimating the maximum sequence likelihood are shown in Fig. 5. We observe that low-temperature multinomial sampling and beam search find the maximum sequence log-likelihood with low number of samples with high probability. Greedy decoding (beam width of 1) finds the maximum for all experimental settings except one, where it takes a beam width of 2 to find it with high probability.
0.5
Probabiltty
Probability
0.2
0.1
0.1
0.0
0.0
Outcomes
Outcomes
(a) Vocabulary size (width) 20
(b) Vocabulary size (width) = 100
Figure 3: Exemplary predictive distributions p(y. yct) for different vocabulary sizes (widths).
20

---

Published as a conference paper at ICLR 2026
Depth:
Depth:
Depth:
Width: 20
Hip(y)) -itell))
10 crm
Greasy Decoding
Number samples Beam width
Number of samples Beam width
Number of samples Bearn width
Figure 4: Estimator of Predictive Entropy. Results for different vocabulary sizes (width) and sequence lengths (depth). We estimate the entropy H(p(y)) using N Monte-Carlo samples (cf. Eq. (5)). Lines denote the average runs, while shades denote one standard deviation. We compare MS for two commonly used т. The experiments show that the decreased temperature leads to lower variance but introduces bias.
Depth:
Depth:
Depth:
-1
-2
Mtoyy)) Mtagy))
WS 1.0
(1e
-4
Boam Search
-5
Gready Deccding
-2
-4
-8
-6
-0
10
1.5
20
Number of samples Beam width
Number of samples / Beaim width
Number samples / Beam width
Figure 5: Estimator of maximum sequence log-likelihood. Results for different vocabulary sizes (width) and sequence lengths (depth). We estimate M(p(y)) using the maximum over N sampled obtained by beam search (N is greedy decoding) or MS with different Lines denote the median, and shades signify the possible values between the and 95 percent quantiles. Beam search deterministic for any given distribution p(y). Even with a very low number of samples, low- temperature MS and beam search are able to find the maximum with high probability.
21

---

Published as a conference paper at ICLR 2026
D DETAILS ON EXPERIMENTAL RESULTS
In this section, we provide detailed insights to complement the main results presented in Sec. 5.1. The code and data are available at https:/ htps://gt /gi thub. cwmmmlo htl/mm/ljk-NNLLLLL
Elppeparaaeeters To compute G-NLL, we use greedy decoding to generate the reference answer, which is equivalent to beam search with a single beam and multinomial sampling with a sampling temperature of zero. Notably, G-NLL is deterministic and hyperparameter-fres logarithmic score based measures (PE, LN-PE, SE. LN-SE, D-SE), 10 output sequences are generated via multinomial sampling. For each dataset, we performed hyperparameter search on held-out instances to determine the best-performing temperature т E {0.5, 1.0, 1.5} for sampling output sequences used for the logarithmic score based measures.
'oo compute the
Evaluation Metrics. The correctness of the reference answer is assessed by checking if the FI score of the commonly used SQuAD metric between the reference answer and the given answer exceeds 0.5 (FT). Although there are some limitations to using such simple metric, it has relatively small errors in standard datasets and, therefore, remains widely used in practice. Additionally, we use Llama-3.1-70B model as LLM-as-a-judge to assess if the given answer is correct (LLM).
AUROC used as the primary performance metric throughout this paper, consistent with standard evaluation practices in this field. We report the results for the individual datasets in Tab. 2, which have been averaged over in Tab. 1. In addition to AUROC, we also consider the average rejection accuracy, i.e., the accuracy of model predictions when allowing the rejection of a certain budget of predictions based on the uncertainty estimate. Results are presented in Tab. 3, where predictions only evaluated for the 80% most certain predictions, and we again use greedy decoding for our measure based on the zero-one score. This further suggests that our measure remains highly competitive across various settings.
Evaluation Tasks. We evaluate our method on both short phrase answers and full-sentence answers (referred as "short" and "long" respectively), assessing the generalization of uncertainty estimation across model classes, model sizes, training stages, and evaluation criteria for output correctness, using three different dnestion-answerte
datasets. We selected these datasets to best align with prior work to provide fair and meaningful comparison (Duan et al., 2024; Kuhn et al., 2023; Farquhar et al., 2024; Bakman et al., 2024: Nikitin et al., 2024; Aichberger et al., 2025; Kossen et al., 2024). Notably, Farquhar et al. (2024) additionally investigates paragraph-Iengee summarization of biographies by decomposing the task into individual guestion-answering problems. This suggests that performance on question-answeringg tasks is expected be indicative of performance on summarization tasks as well. As such, our experimental setup reflects a wide spectrum of practical tasks.
Length-Normalizatioon Intuitively, length normalization might appear to be a reasonable choice also for the single-sequence measure. However, we find that G-NLL without length normalization yields the best performance, even on tasks with high variability sequence length, as reported in Tab. 4. This can be attributed to the fact that length normalization tends to dilute the influence of low-probabilityy tokens. Since most tokens typically have relatively high likelihoods, summing log- probabilities (rather than averaging them) places greater emphasis on the more uncommon, low- likelihood tokens that are more informative for uncertainty estimation. Although not specific to single-sequence measures, investigating the trade-off introduced by length normalization to handle these characteristics represents a promising direction for future work.
Beam-Search for MSP Approximation. We investigate how much the alternative measure derived from the zero-one score benefits from better approximating the most likely output sequences (MSP), as searching for more likely output sequences through beam search theoretically further improves the approximation of MSP. The results summarized in Tab. 5 show that the performance does not significantly improve compared to greedy decoding, which consistent with the ablation study presented in Sec. 5.2. This further supports the claim that G-NLL is strong measure of uncertainty, despite its algorithmic simplicity and computational efficiency.
22

---

Published as a conference paper at ICLR 2026
Table 2: AUROC Evaluation. Average AUROC (1) using uncertainty estimates of different measures as score to distinguish between correct and incorrect answers of each dataset. The reference answer generated using greedy decoding, either as whole sentence (long) or short phrase (short).
Uncertainty measure generating scoring rule
Logarithmii
Zer One
D
Language Model
Gen.
Metric
PE
LN-PE
SE
LN-SE
D-SE
G-NLL
short
.758
.778
.788
.798
.787
.810
PT
short
LLM
.655
.694
.703
.704
.682
.722
long
LLM
.592
.604
.640
.631
.650
.704
8B
short
F1
.735
.768
.790
.800
.777
.809
Transformer
IT
short
LLM
.660
.684
.708
.710
.680
.716
long
LLM
.603
.627
.678
.672
.670
.670
short
F1
.707
.730
.741
.743
.702
.744
TriviaQA
PT
short
LLM
.500
.660
.696
.695
.656
.698
long
LLM
.388
.533
.625
.544
.563
.692
70B
short
F1
698
.714
.722
.726
688
.722
IT
short
LLM
.663
.675
.685
.679
.633
.701
long
LLM
.500
.553
.564
.571
.543
short
F1
.786
.793
.812
818
810
.832
State-Space
PT
short
LLM
.687
.697
.144
.724
long
LLM
597
.653
.675
.680
.689
.705
short
.780
.799
.827
PT
short
LLM
696
.701
.714
.717
.703
.730
long
LLM
.645
.654
.688
.698
.692
.694
short
FI
847
.867
.865
.800
.688
.885
PT
short
LLM
.779
.788
.753
.772
.791
.772
long
LLM
.555
.563
.519
.544
8B
Transformer
short
F1
.879
.903
.914
.912
.877
.931
IT
short
LIM
.706
.725
.736
.753
long
LLM
.56
.524
.50
.608
631
.662
short
.992
.906
.925
.929
.936
SVAMP
PT
short
LLM
.794
.817
.814
.155
.819
.799
long
LLM
S78
.554
.533
.579
.571
.665
70B
short
F1
830
.95
915
.922
915
.999
IT
short
LLM
.703
.744
.734
.748
.762
.713
long
LLM
601
.577
.613
.649
.663
.97
short
F1
.822
.893
.874
.883
.889
.944
State-Space
PT
short
LLM
.752
.757
.730
.738
.777
.776
long
LLM
536
.585
.544
.622
.612
.599
7B
short
F1
843
.891
884
.876
.892
.955
IT
short
LIM
706
.730
.704
.709
.737
.744
long
LLM
.577
.586
.578
.616
short
F1
.725
.739
.673
.710
.758
.776
PT
short
LIM
.639
.661
.615
.641
.683
.683
long
LLM
517
.488
.48
.45
.50
.573
8B
short
F1
702
.732
.TTT
.73
.756
.774
Transformer
IT
short
LLM
.662
.682
.669
.685
.700
.697
long
LLM
494
.491
.30
.544
527
514
short
FI
.727
.733
.711
.737
.748
.779
PT
short
LLM
.634
.649
.642
.657
.61
.672
NO
long
LLM
.38
.514
.533
.589
70B
short
F1
718
.734
.734
.748
.766
.743
IT
short
LLM
.676
.674
long
LLM
535
.540
.526
.666
.574
.545
short
.766
.758
.741
.782
State-Space
PT
short
LIM
.675
.680
.661
.681
.697
.683
long
LIM
567
.553
.112
.551
.572
.554
7B
short
FI
.755
.751
.727
.54
.783
.781
IT
short
LIM
.669
.672
.648
.671
.692
.683
long
LLM
.51
.521
.526
.541
.554
.377
Average
.677
.689
.690
.703
.707
.721
23

---

Published as a conference paper at ICLR 2026
Table 3: Rejection Accuracy Evaluation. Average Rejection Accuracy at 80% (1) across all datasets, using uncertainty estimates of different measures and incorrect answers. The reference answer is generated using greedy decoding, either as a whole sentence (long) or short phrase (short).
to distinguish between correct
Uncertainty measure generating scoring rule Language Model
Logarithmic
Zero-One G-NLL
Gen.
Metric
PE
LN-PE
LÑ-SE
D-SE
FI
.661
.672
.651
.643
.655
.681
short
LLM
.774
.782
.765
.778
PT
LLM-Inntruet
.704
.721
.693
.688
.702
.723
long LLM
.56
.590
.598
.592
.590
.619
Instuuttruet
.667
.684
.632
.643
.644
.686
8b
F1
.668
.684
.680
.673
.87
.702
short
LLM
.775
.781
.779
.775
.778
.788
IT
LLM--Instuce
.723
.742
.732
726
773
.751
Transformer
LLM
.628
.630
.651
.644
.653
.652
long
I.M--nstruct
.713
.724
.705
.713
.727
.734
F1
.188
.827
.222
827
829
.836
short
LLM
844
.852
.846
.847
.511
.855
PT
L.M-Instruct
867
.875
.876
.881
.855
.881
LLM
704
.699
.719
.707
705
.724
long
I.st-tnttt
.789
.795
.776
.781
.788
.812
70b
FI
.795
.813
.844
809
819
.823
short
LLM
836
.842
.842
.837
.44
.845
Instructt
.850
.867
865
.874
.870
LLM
706
.706
712
.715
.721
.715
long
LLM-Instruct
.855
.842
.861
.851
.598
.596
.885
.579
.583
.612
short
LLM
.729
.737
.723
.721
.733
.742
PT
L.LM-Instruc
.638
.640
.626
.621
.632
.651
State-Space
LIM
.613
.627
.622
.624
.620
.623
long
Instruutt
.606
.611
.011
.611
.688
.633
7b
FI
.522
.603
.888
.881
.589
.615
short
LLM
.737
.742
.730
.726
.740
.744
IT
I-M-ruutruct
.632
.646
.625
.619
.637
.653
L.M
.611
.617
.618
.612
.625
.625
long
LLM-IInstrul
.643
.652
.628
.628
.644
.658
Average
.712
.720
.711
.710
.718
.729
24

---

Published as a conference paper at ICLR 2026
Table 4: Length Normalization Evaluation. Average AUROC (1) across TriviaQA, SVAMP, and NQ datasets, utilizing G-NLL and its length-normalized version LN-G-NLL to assign an uncertainty estimate, which is used score to distinguish between correct and incorrect answers. The reference
answer is generated using greedy decoding, either as a whole sentence (long) or a short phrase (short).
Uncertainty measure generating scoring rule
Zero-One
Language Model
Generation short
Metric
LN-G-NLL
G-NLL
Fl
.811
.844
PT
short
LLM
.717
.726
long
LLM
.532
.649
short
FI
.826
.838
Transformer
short LLM
.716
.722
long
LLM
.542
short
.820
PT
short
LLM
long
LLM
.229
.649
70B
short
Fi
.788
.792
IT
short
LLM
.695
.699
long
LLM
.599
.562
short
F1
.888
.843
State-Space
PT
short
LLM
.720
.728
long
LLM
.576
.612
short
Fi
823
.838
short
LLM
.713
.719
long
LLM
.562
.615
Table 5: Beam Search Evaluation. Average AUROC (t) across all datasets, using uncertainty estimates of different measures score distinguish between correct and incorrect answers. The reference answer is generated using beam search with 5 beams, again either as whole sentence (long) or short phrase (short).
Uncertainty measure generating scoring rule Language Model
Logarithmic
Zero-One
Gen. Metrie
PE
LN-PE
SE
LN-SE
D-SE
G-NLL
short
.775
.791
.765
.787
.799
.822
short
LLM
.700
.712
.686
.704
.713
long
LLM
.556
.540
.493
.578
.591
8B
short
.778
.808
.805
.819
.811
.845
Transformer
IT
short
LLM
.682
.704
.713
.698
.729
long
LLM
.535
.520
.584
.885
.586
.559
short
.788
.799
.796
.812
.798
.833
PT
short
LLM
.700
.717
.719
.727
.718
.725
long
LLM
540
.552
.489
.311
.52
.608
70B
short
F1
.756
.786
.796
.806
.788
8000
IT
short
LLM
.680
.697
.701
.707
.695
.707
long
LLM
.534
.533
.544
.569
.574
.534
short
.14
.818
.806
.823
825
.846
State-Space
short
LLM
.703
.709
.699
.711
712
.719
long
LLM
570
.595
.550
.609
.602
.563
7b
short
.799
.855
.794
.877
828
.845
short
LLM
.699
.713
.694
.709
.720
.730
long
LLM
.574
.575
.582
.621
.607
.577
Average
.677
.688
.678
.698
.700
.709
25


Reviews:
[{"review_id": "0HknJMmxBZ", "reviewer": "Reviewer_1JBP", "summary": "The paper contributes in two ways:  \n1. Using the framework of proper scoring rules, it shows that MSP baseline is a theoretically grounded approximation of the empirical risk.   \n2. More importantly, is that this approximation combined with greedy decoding might give better estimate than entropy-based scores, because the number of samples for entropy to be accurate is huge.   \n\nThe authors further conduct experiments and demonstrate that fairly that MSP baseline outperforms many other methods in QA and in certain conditions. They also point out that this baselines should not be overlooked from the evaluation, which sadly happened in previous papers.  \n\nAfter several reviewing cycles, I believe now it is overall in a good shape and should be accepted.", "questions": "I would like to discuss the transition to (4), could you please clarify.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.\tThe authors show that MSP for greedy decoding can be shown as a relatively good approximation of the empirical risk.", "weaknesses": "1.\tThe method itself is already widely-used, though the theoretical justification was lacking before.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h9Efnbsivn", "reviewer": "Reviewer_4Jjw", "summary": "This paper proposes and justifies a G-NLL measure for uncertainty estimation in LLMs. The main contribution is showing the connection to an AU like term while computing the expected score for the zero-one scoring rule. As I understand, the measure itself has been explored before, but this work provides some justification. Experiments are conducted to show the merits of the simple measure over some baselines. I have reviewed a prior version of the paper and find that this revised version is extremely similar. In fact, I’m not sure about the edits that have been made in this version – they seem minor. Overall, I like aspects of this paper and think it is a useful contribution.", "questions": "Some comments/questions follow:\n\nG-NLL is said to approximate MSP. Could the authors clarify this a bit further or point me to the place in the paper where they explain the relation?\n\nI feel it would help to present a much broader view of UQ; the current scope is heavily biased towards a subset of literature. This is a suggestion that I made previously as well, and I am slightly frustrated that it seems to have been ignored. Please see some recent survey papers for ideas if needed.\n\nIt appears to be standard practice for many uncertainty measures in the literature to be used as scores for predicting whether the greedy sample is correct or not, using metrics like AUROC. Could the authors share their thoughts on why this is the standard mode of evaluation? I am curious to hear their perspective.\n\n“Our work challenges the reliance on sampling-based methods and offers a principled way for efficient uncertainty estimation for LLMs.” I consider the first part of this statement to be a stretch. There is a much broader spectrum of sampling-based methods that were not explored. Moreover, only 3 datasets were considered for experiments, and they do not even scratch the surface of breadth of potential applications. Also, the proposed method only works for scoring the greedy sample and not an arbitrary sample, making uncertainty estimation itself somewhat limiting. In short, there are numerous limitations for the scope of the work. I recommend modulating this statement and discussing limitations adequately.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main strength of the paper is its contribution of making a justification for the proposed uncertainty measure G-NLL. It does so by showing the measure to approximate the aleatoric term in the Bayesian decomposition (TU = AU + EU) for the expected score according to a specific scoring rule. Overall, I continue to appreciate the overall simplicity of the final solution.", "weaknesses": "I find the additional contribution to practice to be somewhat limited, as it seems that the measure or related measures have been explored in prior work. \n\nAlso, I find the claims by the paper to be somewhat excessive – the measure is made out to be more useful than I believe it really is. See my comment about this later.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ClEIQN6TRS", "reviewer": "Reviewer_rQPZ", "summary": "The paper presents a new uncertainty quantification scheme for LLMs that leverages the probability of the most likely sequence generated for the input prompt. The proposed method is formulated in terms of a proper scoring rule, specifically a zero-one score. This kind of framing comes with certain properties. The experimental evaluation is restricted to QA benchmark datasets and 3 open models from the Llama and Falcon families. The results show indeed improved performance over the competitive approaches, some of them requiring multiple samples.", "questions": "- I may have misunderstood, but Eq 10 looks like summing up the log probabilities of the tokens obtained during greedy decoding. If that's the case, then it looks like your proposed approach is a slight reformulation of previous methods that combine token probabilities in different ways.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Uncertainty quantification for LLM is currently a hot topic and therefore advances in this area are warranted.\n\n- The proposed method is very efficient in the sense that it requires a single greedy decoding.", "weaknesses": "- I found the presentation of sections 2 and 3 not very clear. Perhaps an illustrative example would help to better convey the technical details. Also, it is not clear what is new here, as proper scoring rules have been presented elsewhere.\n\n- The performance improvements appear to be just marginal over the competitive approaches e.g., 0.804 vs 0.824 in Table 3. Standard deviations appear to be missing from the table, and these need to be included to account for the noise. Also, for such mediocre performance bumps, statistical significance tests are mandatory.\n\n- The proposed method assumes access to sequence probabilities and therefore it is not applicable to closed models such as GPT or Gemini. I think this limitation should be made clearer in the presentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)