Estimating Semantic Alphabet Size for LLM Uncertainty Quantification
Authors: Lucas Hurley McCabe
Rimon Melamed
Thomas Hartvigsen
H Howie Huang
Venue: ICLR 2026 Poster (2026)
Keywords: large language model
uncertainty quantification
hallucination
entropy
alphabet
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Semantic alphabet size estimation enhances semantic entropy estimation and black-box LLM incorrectness detection.

Abstract: Many black-box techniques for quantifying the uncertainty of large language models (LLMs) rely on repeated LLM sampling, which can be computationally expensive. Therefore, practical applicability demands reliable estimation from few samples. Semantic entropy (SE) is a popular sample-based uncertainty estimator with a discrete formulation attractive for the black-box setting. Recent extensions of SE exhibit improved LLM hallucination detection, but do so with less interpretable methods that admit additional hyperparameters. For this reason, we revisit the canonical discrete semantic entropy (DSE) estimator, finding that it underestimates the ``true'' semantic entropy, as expected from theory. We propose a modified semantic alphabet size estimator, and illustrate that using it to adjust DSE for sample coverage results in more accurate SE estimation in our setting of interest. Furthermore, we find that two semantic alphabet size estimators, including our proposed, flag incorrect LLM responses as well or better than many top-performing alternatives, with the added benefit of remaining highly interpretable.

Published as a conference paper at ICLR 2026
ESTIMATING SEMANTIC ALPHABET SIZE FOR LLM UNCERTAINTY QUANTIFICATION
Lucas H. McCabel2 Rimon Melamed! Thomas Hartvigsen H. Howie Huang! GGorree Washington University 2LM Consulting *University of Virginia
{lucasmccabe, rmelamech howie)@qwu. edu
ha hartvigsenévirgii
ABSTRACT
Many black-box techniques for quantifying the uncertainty of large language mod- els (LLMs) rely on repeated LLM sampling, which can be computationally ex- pensive. Therefore, practical applicability demands reliable estimation from few samples. Semantic entropy (SE) is popular sample-based uncertainty estimator with a discrete formulation attractive for the black-box setting. Recent exten- sions of SE exhibit improved LLM hallucination detection, but do so with less interpretable methods that admit additional hyperparameters. For this reason, we revisit the canonical discrete semantic entropy (DSE) estimator, finding that it un- derestimates the "true" semantic entropy, as expected from theory. We propose modified semantic alphabet size estimator, and illustrate that using it to adjust DSE for sample coverage results in more accurate SE estimation our setting interest. Furthermore, we find that semantic alphabet size estimators, includ- ing our proposed, flag incorrect LLM responses as well or better than many top- performing alternatives, with the added benefit of remaining highly interpretable.
INTRODUCTION
Large language models (LLMs) are not fact engines. They have been shown to forget provided context (Liu et al. 2024). fabricate records (Lee et al.) 2023), and otherwise hallucinate (Ji et al.) 2023). LLMs' underlying training data may be of mixed factual reliability, but models can also hallucinate when they have ample knowledge to respond adequately (Simhi et al.) (002).. In risk- sensitive settings, may be prudent for systems to abstain when uncertainty is high (or, alternatively, confidence is low) (Murphy 2022 Hasan et al. 2025). For these reasons and others, it is prudent to estimate LLMs' intrinsic uncertainty.
Uncertainty quantification (UQ) in LLMs is particularly challenging, however, in part due to their computational scale (Liu er al. 2025). Extensive sampling can be financially prohibitive or com- putationally intractable, and these concerns complex with respect to the number of samples. Furthermore, internal activations and sequence log-probabilities are not always available from commercial inference providers (Farquhar et al.) 2024), potentially disqualifying so-called "white-box" methods. Therefore, sample-eliiiient UQ in the black-box setting where LLM internals are assumed inaccessible a crucial area of research for deploying trustworthy AI systems.
magnified if the UQ method is computationally
We provide empirical evidence that canonical discrete semantic entropy (DSE) underestimates the "true" semantic entropy (SE) for typical sample sizes, on average (Figure Z) To address this limita- tion, we first suggest a straightforward modification to existing alphabet size estimators, where we draw, in part, from population ecology. Then, use it to adjust entropy estimates for sample cov- erage, resulting in reduced bias (Figure 2) and more accurate estimation of the "true" SE, compared to other black-box SE estimators (Table
Finally, we evaluate UQ estimators for inaceuracy classification sentence-length LLM responses. In our experiments, the aforemennionned coverage-adjusteo estimator outperforms other explicit timators of SE (Figure More strikingly, two alphabet size estimators outrank all other -ox UQ methods considered but one, after accounting for uncertainty in establishing overall rank of

---

Published as a conference paper at ICLR 2026
(A)
(C)
Sample from LLM Response Distribution
Estimate Semantic Alphabet Size
What color the sky?
observed
unobserved
d1: The sky is blue.
freq reqquency
d2: Today, it is green!
d3:
Sometimes, is orange.
(B)
Cluster Semantic Categories
semantic alphabet size:
The sky is blue.
Today, green!
It's blue.
semantic alphabet size
Sometimes, is orange.
less uncertain
more uncertain
Figure 1: High-level schematic of semantic alphabet size estimation for LLM uncertainty quantifi- cation (Section 3). (A) Generate LLM responses to a query. (B) Assign responses categories of
shared meaning. (C) Estimate semantic alphabet size, accounting for semantic classes unobserved the sample (Equation 9. LLM response examples are hypothetical for illustrative purposes.
methods. Our results indicate that semantic alphabet size estimation, which is highly interpretable, can perform competitively with many state-of-the art methods in LLM inaccuracy detection.
2 PRELIMINARIES
Information entropy. Information entropy quantifies the expected surprise, or uncertainty, of random variable. For discrete random variable X with finite alphabet S, it is given by:
H(X) = Σ log
S6S
where p(s) depicts the probability of symbol (Shannon 1948). When random variable's proba-
bility density is not known analytically, entropy estimators are employed. A well-studied approach in the unite-sample regime is the plugin approximation:
Eplugin X = Σ p(s) log)(s)),
(2)
where is the number of distinct observations in sample, and p(s;) conveys the empirical fre- quency of among the observations.
Semantic classification. Lexically distinct text sequences may belong to the same semantic equiv- alence class (equivalently, "semantic category" "semantic set") i.c., a set of text sequences with mutually shared meaning. Sequence dj entails sequence d2 d1 implics d2 (Dagan et al.) 2005). Kuhn et al. (202)) assign sequences dj and d2 to shared semantic equivalence class under so-called "strict entailment if their textual entailment is bidirectional.
Semantic entropy. SE, introduced by Kuhn al. (2023). aims quantify intrinsic LLM uncer- tainty with a three-step procedure:
1. Sampling: Given query q, generate n passages d1, d2.....dr
2. Semantic Clustering: Iterating over pairs (d, d,). determine if both sequences belong to the same semantic equivalence class. Greedily assign passages to classes based on the pairwise semantic classifications.
2

---

Published as a conference paper at ICLR 2026
3. Estimation: Semantic equivalence classes are treated symbols in an alphabet S. SE is the entropy calculated over observed semantic equivalence classes:
SE(ql0) = Σ p(slq. ( 0) log p(slq.
SES
where p(slg. 0) represents the probability that an LLM parameterized by 0 generates passage belonging to semantic equivalence class s response to a query q.
Because the distribution over semantic sets is not known, it is approximated from response proba- bilities p(dlq, !) using so-called "Rao-Blackwellizer Monte Carlo integration" (RBMCI):
p(silg.0) 0) Ed des, (dllq.0)
(4)
Ea pdl]q, ((l.0))
where d indicates that response d belongs to semantic equivalence class and p 0) are response probabilities returned by the LLM, for each of the k observed semantic categories S1, 82, $2....... Sk (Farquhar et al. 2024).
The black box setting. Response probabilities not always available. Farquhar al. (2024)
consider a discrete formulation of semantic entropy (DSE), where aggregated document probabilities for each semantic equivalence class are replaced with empirical class frequencies:
k
En Idex
Σa Ide
(g =
log
(5)
π1
with sampled passages and & observed semantic categories. This is the plugin estimator (Equation 2 applied to semantic entropy (Equation (3). Both SE and DSE correspond with hallucination rate in guestion-answering problems (Farquhar et al. 2024).
Generalizations. Recent work is said to have generalized SE DSE. Two pertinent examples are Kernel Language Entropy (KLE) and Semantic Nearest Neighbor Entropy (SNNE), which re- ported state-of-the-art performance for incorrectness classification (Nikitin et al. 2024 Nguyen ct al. 2025). These stronger estimators may come at the expense interpretability. in part owing to implementation complexity (e.g., applying kernel to embed graph nodes in KLE) and introduction of additional hyperparameters and design choices (e.g., similarity function and scale parameter in SNNE). We elaborate KLE and SNNE definitions and implementation details in Appendix A.5
METHODS
The total number of semantic equivalence classes (i.c., the semantic alphabet size [S]) that may be ellicited from an LLM in response to a prompt generally unknown, and not all categories necessarily observed in the sample (i.e., k [S). For instance, under a simple Zipfian model of semantic category-abundant
we expect least one category be unobserved for sample size n 10 if [S] 4 (Appendix B). This situation is known as the under-sampled regime, where the empirical distribution over semantic categories can less surprising than than the true one. In other words, the plugin method for DSE may underestimate LLM uncertainty. which we illustrate in Figure 2 For this reason, we are interested in methods for estimating semantic alphabet size and adjusting for when estimating SE.
Semantic alphabet size. Because plugin DSE does not directly account for unobserved semantic categories, the implicit alphabet size used by the plugin estimator is k, called "NumSets" by Lin al. (2024). In the under-sampled regime, NumSets underestimates [S], so mantic alphabet size estimator for small n is desirable. Parallels exist with the so-called "unseen species" problem in population ecology: given sample of observations belonging to one more species, estimate the number of yet unseen species that would be discovered by collecting additional observations (Fisher al. 1943). In this setting, the sample coverage the fraction of all possible categories observed in sample (i.e., C A (5T), also of interest (Chao & Shen 2003).
accurate
3

---

Published as a conference paper at ICLR 2026
The so-called "Good-Turing" sample coverage estimator is ČgT fi where fi is the number of singletons, or semantic categories observed only once (Good 1953). Modest arithmetic converts the Good-Tuue Good-Turing sample coverage estimator into alphabet size estimator:
kn IS
n n-
More recently, Lin et al. (2024) suggested a "continuous" NumSets analogue: responses inter- Sytay where aij is the entailment preted nodes of colnecconnected graph G with weights Waj ___ probability for response pair d,, dy, via an NLI model. Given the eigenvalues (dj < .... < dn) of G's normalized Laplacian, the estimator is given by:
UBigV Σ max(0. 1 d;)..
Coverage-adjusted entropy. provide coverage-adjusted discrete entropy estimator that scales empirical category frequencies by estimated sample coverage:
To quantify a population's 's ecological diversity, Chao Chhoo & Shen (2003)
log(tr))( (1
fcs = Σ
where Pi 1 Ed Ed dex, This so-called "Chao-Shen" estimator is consistent and less biased than many empirical alternatives (Vu al. 2007 Pinchas al. 2024).
Hybrid estimators. First, we adapt the two aforementioned semantic alphabet size estimators to address shortcomings of each. When the number of singletons is zero, [SiGr reduces to NumSets, and it is undefined when all samples belong to distinet semantic categories. On the other hand, Urigv can be less than k, which is a lower bound for [S]. To remediate the above limitations, we propose an alternative "hybrid" semantic alphabet size estimator:
UEigV. max Si UEISV):
if fi otherwise.
S Iybbidd
Additionally, we propose Chao-Shen-like DSE estimator that takes the form of Equation but invokes the hybrid semantic alphabet size estimator for coverage adjustment:
kft Slyyhrid
log
kP
A Hybrid
Hybrid
(10)
kB
151yybrid
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Models. Following Farquhar et al. 2024), we focus on fine-tuned models in our case, the instruction-tur models of Gemma- 2-9B (Team al. 2024). Gemma-3-12B (Team et al. 2025) Llama-3. 3.10a-3..18- 3.B (Grattatiori et al. 2024). Mistral-vO.3-7B and Phi-3.5-3. (Abdin et al.) (024). We perform text generation two temperatures, for distinct purposes. Following related prior work, we sample at temperature
1.0 to calculate uncertainty scores and again at 0.1 to obtain a "best guess" response for assessing the correctness of LLM responses (Farquhar et al., 2024 Nikitin et al.) 2024 Nguyen al. 2025). With the exception of Figure 2 which iterates over several sample sizes, the results shown in the main body of this work use a sample size of n 10, which is also found in prior work SE (Kuhn et al.) 2023 Farquhar al. 2024 Nikitin al. 2024 Nguyen al. 2025). Due to its reduced computational burden compared to larger sample sizes and prevalence in the relevant literature, will refer to 10 as "practical" "typical" sample size.
4

---

Published as a conference paper at ICLR 2026
Gemma-2-98
Gemma-3-122
Llama-3.1-88
Mistral-7B
Phi-3.5-3.8B
12
12
12
12
12
1.0
1.0
0.8
0.8
0.4
0.4
0.4
0.4
0.4
10
25
50
100
10
25
50
100
10
25
50
100
10
25
50
100
5
10
25
50
100
Bryanle (Ours)
...... Biigii (Canonical DSE)
Figure 2:
Illustrating underestimation in discrete semantic entropy (DSE) calculation with typical
sample sizes. The ratios of DSE estimators with varying sample size (n 5, 10, 25, 50, 75, 100) to white-box SE with n = 100 (denoted SE") are shown, with values below 1 suggesting underesti- mation (dotted grey line). The estimators displayed are the plugin estimator of canonical DSE (i.e., Equation 5 dotted indigo line) and the "hybrid" DSE estimator of Equation T0 (solid indigo line). Results are averaged across queries within each dataset, then uniformly averaged across datasets. Log scale is used the x-axis to highlight differences between estimators with smaller sample sizes. Instances with a denominator are ignored.
Datasets. We consider question-answering datasets wherein each query has either one multiple correct answer(s). For the former, we use the validation sets of HotpotQA (Yang el al. 2018) and SQuAD 2.0 (Rajpurkar al. 2018 2016), which contain 7405 and 11864 exemplars, respectively. Our experiments require extensive sampling of LLM responses, and BEC demands combinatorially many language model calls. For these reasons, and because SQuAD 2.0 is much larger than the other data sets, our experiments using it are performed on a random 20% subset. HotpotQA has substantial scale, as well, due to its inclusion of contextual information supplementing the queries. For estimator evaluation (Figure 2 and Table D we use the first 25% of HotpotQA's queries; for incorrectness detection (Figure 3.. we use the full dataset. We also consider queries with a wider variety of possible correct answers. The first such dataset is BioASQ, biomedical QA benchmark with 4719 exemplars. Each question in BioASQ has between and 15 reference answers. We also prepare small (131 exemplars) supplementary dataset called Plethora Of accepTaabl crTegOries (POTATO), for which the number of possible correct semantic categories has an even larger range (up to 722). We discuss POTATO detail in Section A.2
4.2 METRICS AND BASELINES
Semantic entropy estimator evaluation. For given prompt-I.LM pair, [S] and the true prob- ability distribution over semantic equivalence classes unknown, so semantic alphabet size and SE have no ground truths. Instead, we assume that white-box SE with large number of samples (n 100), denoted SE*, is interchangeable with the true estimand (i.e., SE* well-represents the true entropy over all possible semantic categories). We elaborate on this choice in Appendix Section C When illustrating the degree which SE is underestimated (i.e., Figure 2). report the ratio of the estimator of interest to SE". Since information entropy is non-negative, values below indicate underestimation. Our primary evaluation metric for entropy estimators, however, is mean-squared еттоr (MSE): we assess the MSE between an estimate using the small sample size (n = 10) and SE*.
Incorrectness evaluation. We also assess the ability of estimators to classify LLM responses as "correct" or "incorrect," sometimes referred to as "hallucination" or "confabulation" detection (Far- quhar et al. 2024). We calculate the area under the receiver operating characteristic curve (AUROC), corresponding to the probability that randomly selected incorrect L.MM response has higher un- certainty score than randomly selected correct response. Empirical AUROC values are point es- timates, which themselves have uncertainty. Nikitin et al. (2024) point out that such uncertainty is strongly driven by the model and dataset, rather than UQ method, motivating head-to-head compar- isons and evaluation by win rate. We advance a similar approach, but we rely on Bradley- Terry latent strength scores (Zermelo, 1929 Bradley & Terry 1952), calculated via minorizaton-maxiiizztooo (Caron & Doucet) 2012 Hunter, 2004)). allowing us to establish an overall rank of methods. We also employ Monte Carlo procedure that attempts to account for uncertainty in estimating both
5

---

Published as a conference paper at ICLR 2026
Dataset
Estimator
Gemma-2 (9B)
Gemma-3
Llama-3.1
Mistral-
Phi-3.5
(12B)
(8B)
v0.3 (7B)
(3.8B)
Plugin
0.46 0.02
0.09 0.01
0.68 = 0.03
0.59 + 0.03
0.61 = 0.03
HotpotQA
CS-GT
0.39 = 0.02
0.08 0.01
0.56 + 0.03
0.46 0.03
0.47 = 0.03
H Hybria
0.30 + 0.02
0.06 + 0.01
0.45 + 0.02
0.39 + 0.02
0.39 + 0.02
H Plugin
0.68 = 0.03
0.18 0.02
0.78 = 0.03
1.37 0.04
1.43 = 0.04
SQuAD 2.0
CS-GT
0.51
0.15 0.01
0.60 0.03
0.86 0.06
0.91 = 0.06
Hybrid
0.12 0.01
+ 0.02
0.68 = 0.03
0.75 = 0.03
Plugin
0.39 0.08
0.35 # 0.08
0.62 0.11
1.71 #
1.91 + 0.15
POTATO
Hcs-Gr
0.30 = 0.07
0.29 + 0.07
0.51 # 0.11
0.94 # 0.29
1.58 + 0.55
Hybrid
0.27 0.06
0.24 = 0.06
0
0.71 0.11
0.73 + 0.11
Plagin
1.64
1.79 0.03
1.68 + 0.03
2.04 0.03
1.82 + 0.02
BioASQ
CS-GT
0.06
1.10 0.07
1.06 + 0.05
1.31 0.08
0.99 - 0.04
H Haybrid
0.78 + 0.02
0.69 + 0.02
0.80 + 0.02
0.73 + 0.02
0.83 + 0.02
Table I: Empirically evaluating the accuracy of explicit discrete semantic entropy (DSE) estimators. Values reflect MSE between the estimated value using л 10 samples and white-box semantic entropy with n = 100 (i.e., SE*). The lowest MSE value for each model-dataset pair is shown in bold. Intervals, shown in grey, reflect 95% CIs via the standard error of the mean.
AUROC and strength scores. First, we fit Gaussian uncertainty distributions about AUROC point estimates from the 95% confidence intervals (CIs) obtained via DeLong's method (DeLong al. 1988 Sun & Xu 2014). a modeling assumption we justify by the approximate normality of the U statistic (Mann & Whitney 1947 For each model-dataset pair, we simulate L. 100 matches between each pair of methods by comparing AUROC values sampled from their corresponding un- certainty distributions. After obtaining strength scores from the simulated matches, we calculate CIs about the Bradley-Tery scores and establish 95% CIs about the ranks of latent strengths using the methods Gao et (2023). We provide additional details in Appendix A.6
Baselines. Because our priority is black-box uncertainty estimation, we compare with other black- box methods. For SE estimation, we compare our hybrid approach (Equation 10) to the other explicit black-box SE estimators: the canonical discrete approach of Farquhar et al. (2024) (the plugin esti- mator, Equation (5) and the soverage-adjuster estimator of Chao & Shen (2003) (Equation [8). For incorrectness classification, four alphabet size estimators, and four other uncertainty methods. The alphabet size estimators are the number of semantic categories (i.e., NumSets) (Kuhn et al.) (202)). the alphabet size estimator converted from the Turrnn sample coverage estimator 1953). the spectral estimator of
consider the three aforementioned explicit black-box SE estimators,
Lin et al. (2024) (i.e., UEigv). and hybrid alphabet size estimator (i.e., Equation 9). The other uncertainty methods are Predictive Entropy (PE, see Appendix AA5) (Kadavath et al. 2022). SNNE
(Nguyen al. 2025), KLE (Nikitin al. 2024). and white-box SE (Kuhn al. 2023
4.3 RESULTS
Canonical discrete semantic entropy underestimates "true" semantic entropy. The plugin estimator for information entropy A Hplugin has meerreiiaally-estaliie negative bias (Basharin 1959 Harris 1975), which largely governs its mean-square error (Antos & Kontoyiannss 2001) We interested in observing this phenomenon empirically, where do not have access to the true distribution over semantic equivalence classes. In Figure 2 we compare plugin DSE estimates SE", indicating that the canonical plugin DSE approach underestimates its quantity of interest for practical sample sizes. Such underestimation can be problematic for reliable UQ because drawing a large number of samples from LLM can be costly and time-consuming scale.
In principle, samples from these distributions may exist outside the [0, 1] range, but this is unlikely in our case, given the AUROC CT bounds (Figure TO

---

Published as a conference paper at ICLR 2026
(A)
(B)
15 Piaa
Latent strength
Namsatts
1.2
Deyy
Spieis
(iiI)
PE
SNINE
Bilettdd Wondatt
Bayy
Ugger
PE
SHAnE
RiE
96
(Blaneed
Blar
PE
sine
KiB
5
8
10
Rank
Figure 3: Establishing overall performance ten UQ methods on incorrectness detection. (A)
Bradley-Terry latent strength scores from pairwise comparison of AUROC point estimates. (B)
Bradley-Terry latent strength scores after accounting for uncertainty in estimating AUROC. Error bars are "conservative" CIs about strength scores, which may be slightly stricter than 95%; see dection[A.6]fon details. (C) For each method, we establish 95% CIs about the rank of Bradley-Terry latent strength MLEs (Gao et al., (023) for the incorrectness detection task; see Section [A.6 for details. We highlight (i) semantic alphabet size estimators, (ii) black-box discrete semantic entropy (DSE) estimators, and (iii) other uncertainty estimators, which include white-box (PE, SE) and black-box methods (SNNE, KLE). The interval [a, b] denotes all integers from a to b, inclusively. The CI for SNNE reflects [6, 6] and extended for readability.
Our proposed estimator improves the accuracy of discrete semantic entropy estimation. In we observe that, on average, fiiyyrid misestimates its target quantity by less than canoni- cal plugin DSE for a range of sample sizes. We perform a more granular comparison of SE estimator accuracy with n 10 Table where ffilybrid consistently achieves the lowest MSE among ex- plicit discrete SE estimators across five models and four datasets. We also find that Hlyyrrd slightly over-shoots SE* average for the 12B- 1B-parameter Gemma model. Detailed results, broken out by dataset in Appendix F reveal that this overshooting is limited to the POTATO dataset. In fact, this appears to be driven by just three anomalous instances, examined in Appendix [D] Excluding in- stances with near-zero "true" entropy (i.e., SE" < x0.005) recovers the pattern exhibited by the other models (Figure 9
Figure 2
Our proposed semantic entropy estimator outperforms other explicit semantic entropy esti- mators at incorrectness detection. LLM uncertainty estimation is often employed as a proxy for hallucination, confabulation, or incorrectness detection among LLM responses (Kuhn et al.) 2023] Farquhar et al. 2024). Following Nikitin et al. (2024), we calculate pairwise AUROC win rates be- tween models for each model-dataset pair, win is recorded in method m; 's favor over method mj the corresponding AUROC for m; is greater than that of m, (Figure 4). We characterize overall performance of each method via Bradley-Terry latent strength scores. Figure A displays maxi- mum likelihood estimates of the methods' strength scores from pairwise AUROC comparisons. We highlight the uncertainty in both AUROC and Bradley-Teer Terry estimation in Figure 3B (CIs about Tradley-Terry estimates, which may be slightly stricter than 95%) and Figure 3C (95% CIs around
each method's rank by Tradley-Terry score). Here, HH Mybrid (Equation 10) achieves the highest rank among explicit estimators semantic entropy, including white-box SE
Alphabet size estimators alone can outperform entropy estimators at incorrectness detection. After accounting for uncertainty in our rank-generating procedure, two semantic alphabet size es- timators namely, Hyliid and UgigV outperform many more complex black-box uncertainty methods. Three methods have 95% rank CIs that include the top position: alphabet size estimators [S] (rank CI of [1, 4]) and UEigv (rank CI of [1, 3], which attains the largest latent strength estimate), well as KLE (rank CI of [1, 3)) (Figure 3C), with all three methods outperforming white-box SE and other explicit SE estimators overall. Our results recall|Kuhi et al. (2023) 's obser- vation that the number of observed semantic categories is itself "a reasonable uncertainty measure."
7

---

Published as a conference paper at ICLR 2026
5 RELATED WORK
Herein, we briefly review related contributions not discussed elsewhere in this work.
Earlier LLM uncertainty quantification
methods. Linguistic Confidence assesses whether the LLM articulates its confidence in its response (Mielke et al., (022). The P(True) al. (2022) similarly
100
100
7
87
100
100
47
53
40
100
method of Kadavath relies on an LLM's self-pereeived uncertainty, asking language model if a response "True"
NumSets
0
0
53
7
0
47
[5
0
27
0
0
33
33
0
27
0
27
or "False," with the response probability of "False" ultimately reported. SeIfCheckGPT draws + responses from LLM and assesses consistency of each sentence of the first response with each of the following responses (Manakul et al., 2023). In prior work, such approaches have been consistently superseded by the methods otherwise described herein (Kuhn et al. 2023] [in al. 2024
UsigV
93
100
100
93
100
100
47
73
47
100
Rayana
13
100
100
7
100
100
47
20
100
i Eriugh
0
60
67
0
60
7
27
0
33
PE
53
93
100
53
53
93
93
93
53
93
SNNE
73
KLE
60
100
100
53
80
100
100
47
67
100
Farquhar et al. 2024 Nikitin et al.
2024
Nguyen al. 2025).
NumSars
Uenv
Blyy
fiss
PE
SNNE
KLE
5E
Uncertainty and internal representations. In the white-box setting, it may be desirable to ascertain LLM uncertainty from internal repre- sentations, rather than repeated sampling. Han et al. (2024b) and Han et al. (2024a), for exam- ple, approach this from an interpretability point of view, building earlier work seman- tic entropy (Kuhn et al., 2023 Farquhar et al.) 2024). Other analyses, however, contend that so-called "truthfulness encodings" are difficult to generalize (Orgad et al.) 2025), warranting further examination.
(II)
(iii)
Figure Heatmap illustrating the proportion of model-dataset pairs, rounded to the nearest integer, for which row's method achieved larger AUROC point estimate than column's method. Uncertainty methods are organized into three groups: (i) semantic alphabet size esti- mators, (ii) black-box discrete semantic entropy (DSH) estimators, and (iii) other uncertainty timators, which include white-box (PF, SE) and black-box methods (SNNE, KLE)). The hybrid discrete semantic entropy (DSE) estimator consis- tently outperforms other explicit SE estimators.
Alphabet size estimation. The Good-Turing estimator discussed herein accounts for single- tons in the sample (Chao & Shen 2003). Alter- native alphabet size estimators accounting for doubletons (i.e., semantic categories appearing exactly twice in the sample) (Chao 1987) and triple- tons (i.e., categories appearing exactly three times) (Lanumteang & Böhning 201 exist, but they are undefined when number of doubletons zero.
Unbiased entropy estimation. The entropy estimator of Montgomery Smith & Schürmann (014)) is unbiased, but is incompatible with the possibility unobserved semantic equivalence classes. Otherwise, unbiased estimator exists, to the best of our knowledge (Paninski 2003).
6 DISCUSSION
6.1 CONCLUSION
Several approaches for estimating LLM uncertainty have emerged in recent years, but UQ perfor- mance can be constrained, absent white-box LLM access. Furthermore, the practicality of sampling- based UQ methods is limited by the computational costs associated with repeated text generation. We illustrate the importance of semantic alphabet size in LLM uncertainty estimation with small sample sizes. Adjusting DSE for sample coverage using our proposed semantic alphabet size esti- mator results in more accurate SE estimation and improved L.LM incorrectness classification, com-
8

---

Published as a conference paper at ICLR 2026
pared to other explicit estimators of SE. Further, find that semantic alphabet size estimators UEigV and ySlid achieve higher latent strength estimates than most UQ methods considered in our incorrectness detection experiments, with UEigV sharing the top rank CI with KLE, a conceptu- ally complex approach that may be more difficult to interpret.
The overall performance of SE and plugin DSE on the incorrectness task are qualitatively similar (Figure 3) and related findings appear prior work, such as Farquhar et al. (2024) and Nguyen et al. (2025). In fact, our experiments indicate that white-box SE tends to track plugin DSE quite closely, on average, as seen in Figure 9 We reflect on this phenomenon in Appendix Section where we observe nearly identical distributions induced by RBMCI and empirical class frequencies, on average. A notable difference is that SE appears to be more robust to instances where exceedingly rare semantic categories are overrepresented in the sample. The sequence rog-robabbiliies available to SE may be otherwise underutilized, however, since SE exhibits the same underestimation pattern as plugin DSE. Future work may extend the approach described herein the white-box setting by directly adjusting SE for unobserved semantic categories.
Though of scope for the present study, we underscore that semantic alphabet size estimation and Hybrid: partieular may have broader application than merely entropy estimation. For instance, concurrent work by Li et al. (2025) applied Good-Turing method to estimate the extent of LLMs' unexpressed factual knowledge (c.g., mathematical theorems and diseases).
6.2 LIMITATIONS
We highlight several limitations in the present work. First, computational constraints limit our ability to run the experiments herein with models larger than 12 billion parameters. That said, our analysis is extensive, spanning five models, four datasets, and eleven uncertainty estimators. Though our results are empirical, and experimental results may vary across studies, make efforts express our statistical uncertainty, and our final rank-generating procedure aims to account for it.
Second, our work aims improve upon plugin DSE, whose estimator is negatively biased. For this reason, we take semantic cluster labels as fixed, without ablating across alternative clustering strategies. End-to-end SE calculation involves several steps, however, and biased estimator does not necessitate that the entire process results in a negatively biased estimate. For instance, a high false negative rate in the assignment of responses semantic categories (e.g., due to minor lexical alterations inducing false negatives) may positively bias the final estimate (Grewal et al. 2024).
Finally, the uncertainty estimation methods considered herein do not directly measure factual in- accuracy. Instead, they may be better understood as indicators of semantic diversity. Of course, high semantic diversity may correspond to a high factual error rate, but there are scenarios wherein semantic diversity does not necessarily imply incorrectness (Ilia & Aziz 2024), and, conversely, models can be confidently wrong.
6.3 REPRODUCIBILITY
We outline our major experimental settings in Section 4.1 as well as evaluation metrics and base- line methods in Section In Appendix AA] we elaborate extensively on additional implementation details needed to reproduce our results, including algorithms, models, and prompt templates. In- terested readers will also find a brief overview of implementation variations found in other work in Section H.. Finally, we provide a code repository at http://gghub..co//uuuscccccc to support scientific accessibility and reproducibility in future work.
with hope
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv preprint 2404 2024.
András Antos and Ioannis Kontoyiannis. Convergence properties of functional estimates for discrete distributions. Random Structures & Algorithms, 19(3-4):163-1199 2001.

---

Published as a conference paper at ICLR 2026
Georgij P Basharin. On a Statistical Estimate for the Entropy of a Sequence of Independent Random Variables. Theory of Probability Its Applications, 4(3):333-336 1959.
Ralph Allan Bradley and Milton E Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/))3324-34 1952.
Ricardo JGB Campello, Davoud Moulavi, and Jöng Sander. Density Baaed Clustering Based on Hierarchical Density Estimates. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 160-172. Springer, 2013.
Francois Caron and Arnaud Doucet. Efficient Bayesian inference for generalized Bradley-Terry models. Journal of Computational and Graphical Statistics, 21(1):174-16,, 2012.
Anne Chao. Estimating the Population Size for Capture- Data with Unequal Catchability. Biometries, pp. 783-791, 1987.
Anne Chao and Tsung-Jen Shen. Nonparametric estimation of Shannon's 's index of diversity when there are unseen species in sample. Environmental and Ecological Statistics, 10:429 144299443, 2003.
Samuel Colvin, Eric Jolibois, Hasan Ramezani, Adrian Garcia Badaracco, Terrence Dorsey, David Montague, Serge Matveenko, Marcelo Trylesinski, Sydney Runkle, David Hewitt, and Alex Hall. Pydantic, June 2024. URL https:://///.. //g thub. wwt.ccoc.pppac. http.///yyyaaancc/yyyaaa..t
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL. Recognising Textual Entailment Challenge. In Machine Learning Challenges Workshop, pp. 177-190. Springer, 2005.
Leon Danon, Albert Diaz-Guilera, Jordi Duch, and Alex Arenas. Comparing community structure identification, Journal of Statistical Mechanics: Theory and Experiment, 2005(09) 2005.
Elizabeth DeLong. David M DeLong, and Daniel L Pearson. Comparing the Areas under Two More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics, pp. 837-845, 1988.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A Density- -Based Algorithm for Discovering Clusters Large Spatial Databases with Noise. In Proceedings of the Second Inter- national Conference Knowledge Discovery and Data Mining, KDD'96, pp. 231-231. AAAI Press, 1996.
Sebastian Farquhar, Joel Kossen, Lukas Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630:625-630, 2024. doi: 1031038/ s4 041586-024-07421-0.
httphttts://o..orgg// 1tt.:/oo00011...1/11111111
Ronald A Fisher, A Steven Corbet, and Carrington B Williams. The Relation Between the Number of Species and the Number of Individuals in a Random Sample of an Animal Population. The Journal of Animal Ecology, pp. 42-58, 1943.
Edward Fowlkes and Colin Mallows. A Method for Comparing Two Hierarchical Clusterings. Journal of the American Statistical Association, 78(383):553-569 1983.
Chao Gao, Yandi Shen, and Anderson Zhang. Uncertainty quantification the Bradley-Terry-Lee Luce model. Information and Inference: A Journal of the IMA, 12(2):1033-1140, 2023.
Irving J Good. The Population Frequencies of Species and the Estimation of Population Parameters. Biometrika, 40(3-4)):377-266 1953.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama Herd of Models. arXiv preprint arXiv:2407.21720 1 2024.
Yashvir S Grewal, Edwin V Bonilla, and Thang D Bui. Improving Uncertainty Quantification in Large Language Models via Semantic Embeddings. arXiv preprint arXiv:2410.22665,, 2024.
10

---

Published as a conference paper at ICLR 2026
Jiatong Han, Jannik Kossen, Muhammed Razzak, and Yarin Gal. Semantic Entropy Neurons: En- coding Semantic Uncertainty in the Latent Space LLMs. In NeurIPS 2024 Workshop on Foundation Model Interventions (MINT), 2024a.
Jiatong Han, Jannik Kossen, Muhammed Razzak, Lisa Schut, Shreshth A Malik, and Yarin Gal. Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs. In ICML 2024 Workshop on Foundation Models the Wild, 2024b.
Bernard Harris. The Statistical Estimation of Entropy in the Non- Non-Parametric Case. University of Misconsin-Madisee Mathematics Research Center, 1975.
Md Mehedi Hasan, Moloud Abdar, Abbas Khosravi, Uwe Aickelin, Pietro Lio, Ibrahim Hossain, Ashikur Rahman, and Saeid Nahavandi. Survey Leveraging Uncertainty Estimation Toward Trustworthy Deep Neural Networks: The Case of Reject Option and Post-Training Processing. ACM Computing Surveys, 57(9):1-35, 2025.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre- Training with Grradient-Disentanglee Embedding Sharing. In The Eleventh International Conference on Learning Representations, 2021.
David R Hunter. MM algorithms for generalized Bradley-Terry models. The Annals of Statistics, 32 (1):3844466, 2004.
Evgenia Ilia and Wilker Aziz. Variability Need Not Imply Error: The Case of Adequate but Seman- tically Distinct Responses. arXiv preprint arXiv:2412./5683, 2024.
Bart Jacobs. A Note on Distances between Probabilistic and Quantum distributions. Electronic Notes in Theoretical Computer Science, 336:173-188 2018.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12):1-38, 2023.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language Mod- els (Mostly) Know What They Know. arXiv preprint arXiv:2207.05222. 2022.
George Kingsley Zipf. Selected studies of the principle of relative frequency in language. Harvard University Press, 1932.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. In The Eleventh International Confer- on Learning Representations, 2023.
Solomon Kullback and Richard Leibler. On Information and Sufficiency. The Annals of Mathe- matical Statistics, 22(1):79 86, 1951.
Andrea Lancichinetti, Santo Fortunato, and János Kertész. Detecting the overlapping and hierarchi- cal community structure in complex networks. New Journal of Physics, 11(3):033015, 2009.
Krisana Lanumteang and Dankmar Böhning. An extension of Chao's estimator of population size based on the first three capture frequency counts. Computational Statistics & Data Analysis, 55 (7):2302-231) 2, 2011.
Peter Lee, Sebastien Bubeck, and Joseph Petro. Benefits, Limits, and Risks of GPT-4 as an AT Chatbot for Medicine. New England Journal of Medicine, (8)(11)) 2023.
Xiang Li, Jiayi Xin, Qi Long, and Weijie J Su. Evaluating the Unseen Capabilities: How Many Theorems Do LLMs Know? arXiv preprint arXiv:2506.02058, 2025.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: www. ac web. www..orgg/mthhooogg//66--------

---

Published as a conference paper at ICLR 2026
Chin-Yes Lin and Franz Josef Och. Automatic Evaluation of Machine Translation Quality Using Statistics. In Proceedings of the 42nd Anmual Meeting of the Association for Computational Linguistics (ACL-04), pp. 605-612, Barcelona, ogy. org/
Longest Common Subsequence and
Spain, July 2004.
doi: 00.3115/11185551219000 URL
P04-1070
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with Confidence: Uncertainty Quan- tification for Black-box Large Language Models. Transactions Machine Learning Research, 2024, 2024. ISSN 2835-8856.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 2024.
Xiaoou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, and Hua Wei. Uncertainty Quan- tification and Confidence Calibration in Large Language Models: A Survey. arXiv preprint arXiv:2503.15850, 2025.
Andrey Malinin and Mark Gales. Uncertainty Estimation in Autoregressive Structured Prediction. In International Conference on Learning Representations, 2021.
Potsawee Manakul, Adian Liusie, and Mark Gales. SelfCheckGPT: 7ero-Resource Black-Box Hal- lucination Detection for Generative Large Language Models. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 9004-9017, Singapore, December 2023. Association for Computational org/
Linguisties. doi: 10. 1865/ 2023. www.e-mnnamaaeeeeee 557/
URL
Henry B Mann and Donald R Whitney. On Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics, pp. 50-60, 1947.
Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing Conversational Agents' Overconfidence Through Linguistic Calibration. Transactions of the Association for
Computational Linguistics, 10:857-872, 2022. doi: 10. 10.1162/tacl.a a 00494. URL https:
//aclanthology. wwt. tacl t..11.3307
Giorgia Minello, Luca Rossi, and Andrea Torsello. On the von Neumann entropy of graphs. Journal of Complex Networks, 7(4):491-514, 2019.
Stephen Montgomery-Smitt and Thomas Schürmann. Unbiased Estimators for Entropy and Class Number. arXiv preprint 1410 2014.
Kevin P. Murphy. Classification with the "reject" option. In Probabilistic Machine Learning: An Introduction, pp. 166. The MIT Press, 2022.
Dang Nguyen, Ali Payani, and Baharan Mirzasoleiman. Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Com- putational Linguistics: ACL 2025, pp. 4530-4540, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-90 89176-256-5.
Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities. In Advances in Neural Information Processing Systems, volume 37, pp. 8901-8929, 2024.
Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. LLMS Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations. In The Thirteenth International Conference on Learning Representations, 2025.
Liam Paninski. Estimation of Entropy and Mutual Information. Neural Computation, 15(6): 1191- 1253, 2003.
12

---

Published as a conference paper at ICLR 2026
Dénes Petz. Entropy. Neumann and the von Neumann Entropy: Dedicated to the memory of Alfred Wehrl. In John von Neumann and the Foundations of Quantum Physics, pp. 83-96. Springer, 2001.
Assaf Pinchas, Irad Ben-Gal, and Amichai Painsky. A Comparative Analysis of Discrete Entropy Estimators for Large-Alphabe Problems. Entropy, 26(5):369, 2024.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Ques- tions for Machine Comprehension of Text. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16- 1264. URL //act anthoo I ogy 1gy.orge www.orrgDD16111 12644
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know What You Don't Know: Unanswerable Questions for SQuAD. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10.18653/v1/P1 8-2124.
URL /etttps///acanet httaa/aaaanooooo..oroooooooooooo 2124
Claude Elwood Shannon, A Mathematical Theory of Communication. The Bell System Technical Journal, 27(3):379 423, 1948.
Adi Simhi, Jonathan Herzig, Idan Szpektor, and Yonatan Belinkov. Distinguishing Ignorance from Error in LLM Hallucinations. arXiv preprint arXiv:2410.22077 2024.
Xu Sun and Weichao Xu. Fast Implementation of DeLong's Algorithm for Comparing the Areas Under Correlated Receiver Operating Characteristic Curves. IEEE Signal Processing Letters, 21 (11):1399-1393, 2014.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving Open Language Models at Practical Size. arXiv preprint arXiv:2408.000 000-2440800118 18, 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, al. Gemma Technical Report. arXiv preprint arXiv:2503. 19786, 2025.
Werner Ulrich, Marcin Ollik, and Karl Inne Ugland. A meta-analysis of species-abundance distri- butions. Oikos, 1 119(7): (177)::1149: 149-1155, 2010.
Vincent Q Vu, Bin Yu, and Robert E Kass. Coverage-adjusted entropy estimation. Statistics in Medicine, 26(21): 36(21):403944060, 2007.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.
2018. Association for Com- lichkeitsrechnung. Mathematische Zeitschrift, 29(1):436-466 1929.
Brussels, Belgium, putational Linguistics. doi: 10.18653/v/DD1-11
Emst Zermelo. Die berechnung der tumnie-ergebniisee als ein maximumproblem der wahrschein-
A ADDITIONAL IMPLEMENTATION DETAILS
A.1 TEXT GENERATION
Throughout this work, we generate maximum of 100 tokens for each response, with early stopping enabled the EOS token is generated. All otherwise unspecified hyperparameters are set to their defaults (e.g., do not invoke top-k sampling, top-p sampling, beam search, repetition penalty, etc.). For incorrectness classification, we follow prior work that elicits concise responses by pre- pending QA queries with the following "pre-prompt" (Farquhar al. 2024 Nikitin al. 2024
13

---

Published as a conference paper at ICLR 2026
Pre- Pre-Prompt for Single-Sentence QA
Answer the following question in a single brief but complete sentence.
A.2 SYNTHETIC DATA GENERATION
Plethora Of Tabel crTegOries (POTATO) is small synthetic dataset wherein every question is, in principle, answerable with one-word or few-word response, and each question has more than one correct answer. We repeatedly invoke the following prompt against OpenAl's GPT-4-turbo:
Prompt for POTATO Question Generation
Using the available function, generate 10 questions from diverse topic areas. Each question should request only single answer, but there should be exactly WUMMANSWERS possi- ble semantically distinct correct answer(s) to the question. For example, 'Name a continent on Earth' has seven possible correct answers, because Earth has seven continents, but 'Name all the continents on Earth' only has one possible semantically distinct correct answer (a list of all seven continents).
Above, [NUM ANSWERS] is an integer be- tween and 50, and the available function is Pydantic object enforcing structured genera-
(A)
(B)
tion (Colvin et al.) 2024). A human annota- tor assessed the number of semantic categories for possible correct answers, discarded those with only one possible correct semantic cate- gory, and removed duplicate questions. The re-
Survival Probabilty
10*
Observed
Possibre
0.008
Observed
16
10
sulting dataset has 131 unique questions. We also generate 100 responses to each question from the POTATO dataset using GPT-4o-mini. A human annotator manually assigned the re- sponses to semantic categories.
Count
Possible correct
Figure 5: (A) Empirical survival function of the number of possible correct semantic categories and the number of observed semantic categories in the responses generated by GPT-4o-mini, ac- cording to a human annotator. (B) Scattergram
One query ("Name piece of classical music composed by Ludwig van Beethoven.") has a particularly large number of possible correct mantic categories, owing to Beethoven's 's pro- lific output of several hundred compositions. Figure suggests that the distribution of se- mantic alphabet sizes for model-query pairs is generally dominated by the distribution of the numbers of possible correct semantic cat- egories for those queries. On per-query basis, however, these quantities exhibit low correla- tion (Figure 5B).
of the number of observed semantic categories ("Observed"), according to the human annotator, against the number of possible correct semantic categories ("Possible").. One question is excluded ("Identify a programming language designed by Microsoft."". because the total number of possible correct semantic categories was not known by the authors of this work.
A.3 BIDIRECTIONAL ENTAILMENT CLUSTERING
Semantic classification. The semantic clustering step of semantic entropy calculation is per- formed using Bidirectional Entailment Clustering (BEC), which classifies pairs of passages (di, dj) bidirectionally as "entailment," "neutral," or "contradiction." " and uses these labels to greed- ily assign passages to semantic equivalence classes. Like previous work, our implementation fo- cuses on strict entailment, where both unidirectional relations must be considered "entailment" or equivalent (i.e., not "neutral"). We refer the reader to Kuhn et al. (2023); Farquhar et al. (2024) for further detail on the BEC algorithm. The classifier used varies by implementation, either an LLM- or -LI-based method:
14

---

Published as a conference paper at ICLR 2026
[S]
[S] =
Method
Variant
FMI
NMI
PA
FMI
NMI
PA
LLM
0.84
0.72
0.84
0.86
0.71
0,80
Bidirectional Entailment
NLI-NQ
0.84
0.64
0.85
0.89
0.61
0.89
HDBSCAN
0.68
0.52
0.69
0.70
0.41
0.60
DBSCAN (e = 0.05)
0.86
0.71
0.82
0.86
0.65
0.77
Semantic Embedding
DBSCAN (e =0.775)
0.87
0.68
0.82
0.88
0.60
0.79
DBSCAN (E 0.11)
0.85
0.57
0.79
0.88
0.51
0.80
DBSCAN (E =0.15)
0.79
0.27
0.66
0.87
0.25
0.76
Evaluating semantic clustering methods against human-annotated ground truth. Results averaged across questions from the POTATO dataset, either excluding questions for which all sponses were semantically equivalent or questions eliciting multiple semantic categories ((S) > 1 and [S] respectively), according to the human rater. Metrics considered are the Fowlkes-Mailn Mallssss index (FMI), normalized mutual information (NMI), and pairwise agreement (PA), assessed between the semantic equivalence classes assigned by the provided method and the human-annotated annotated ground truth. Values are rounded to two decimal places. The best-performing method by each metric is in bold. Responses are generated by GPT-4o-mini.
Table 2:
Large Language Model (LLM). Farquhar al. 2024) classify passage pairs (T1, T2) by invok- ing OpenAl's GPT-3.5 endpoint with the following prompt:
Prompt for Entailment Classification
We are evaluating answers to the question {question}
Here two possible answers:
Possible Answer 1: {TI}
Possible Answer 2: [T2]
Does Possible Answer semantically entail Possible Answer 2? Respond with entailment, contradiction, or neutral.
Our implementation invokes OpenAl's GPT-4o-mini model, due to the sunsetting of GPT-3.5.
Natural Language Inference (NLI). The BEC approach Kuhn et al. (023) classifies (di, dj)
using a DeBERTa model fine-tuned for NLI tasks Our implementation, used throughout this work. updates the classifier to fine-tuned NLI model based on Microsoft's 's DeBERTav3 (He al. 2021 The NLI classification can be performed with or without the source questions (NLI-Q and NLI-NQ. respectively); in the former, the query prepended both d, and d, before passing through the NLI model.
Comparison. To compare the alignment between each clustering method and the human- annotated ground-truth, we measure the Fowlkes-Mallows index (FMI) (Fowlkes & Mallows 1983) and normalized mutual information (NMI) (Danon et al. 2005 Lancichinetti et al., 2009). We also consider pairwise agreement (PA), wherein we iterate over pairs of passages, classify each pair as "entailment" or "contradiction." based the clustering results, and report for each method the por- tion of pairs whose entailment label agrees with that of the human annotator (similar to the evaluation queries, the human annotations assign all responses the
done by Farquhar et al. (2024)). ) For same semantic equivalence class (i.c., [S] = l). To ensure clustering methods are performant in both [S] 1 and |S| scenarios, we calculate metrics for each (Table 2). Although this paper inherits its BEC semantic clustering strategy from prior work (Kuhn et al. 2023 Farquhar et al. 2024), we also consider several comparative baselines based on semantic embedding distance. In particular, embed responses using sige-base-en-v/.. and cluster according to their cosine distances via ei- ther density-based spatial clustering of applications with noise (DBSCAN) (Ester al. 1996) with
base
htte:///ogginngocc..///////////
15

---

Published as a conference paper at ICLR 2026
radius parameter 0.05,0.075,0011 hierarchical DBSCAN (HDBSCAN) Camppell al. 2013).
By most metrics, the best alignment with the human labels in cach scenario (i.e., |S| vs. |S] 1) achieved by BEC-based strategy (Table The baselines using semantic embeddings have the additional limitation of relying semantic similarity BEC strategies, the NLI variant without the inclusion of questions performs roughly similarly to LLM-based alternative. For its comparable performance while limiting costs, use NLI-NQ throughout this work.
proxy for semantic equivalence. Among
A.4 LLM-AS-JUDGE
Lin & Och Appropriate thresholds may vary by model and dataset for instance, the reference answers in HotPotQA appear rather brief, and models can vary in their verbosity, without it necessarily impacting the correctness of their responses. Additionally, such methods may not capture semantically equivalent, but lexically distinct, paraphrasings (e.g., using an acronym).
Prior work labeled LLM's response as correct if the ROUGE-L score (Lin) 2004
2004) between the response and reference answer was above 0.3 (Kuhn et al. 2023
Instead, Lin et al. (2024) prompt a commercial LLM to provide a numerical rating of consistency be- tween LLM responses and reference answers, with ratings above 70 indicating accurate response. They observe, however, that a small portion of LLM judgements do not have an passsl--areeable rat- ing. For this reason, modify their prompt to request ratings within XML-sttle tags and include an in-context example:
Prompt for Consistency-Bacd -Baceed Judge
Rate the level of consistency between the answer to the question and the reference answer, from to 100. Output the float rating inside <rating>< /rating> tags.
Here is an example output:
Question: What is the capital of France? Reference: Paris is the capital of France. Answer: The capital of France is Paris. <rating>>00< /rating>
Now rate the following:
Question: {question}
Reference: [groundtruth}
Answer: (pred}
When a query has multiple reference answers (e.g., those in the BioASQ), we invoke the above prompt to compare the LLM response each reference answer, then report the maximum LLM-as- judge rating. Our choice of commercial model for LLLM-as-judge is OpenAl's GPT-4o-mini.
A.5 OTHER UNCERTAINTY ESTIMATORS
Predietive entropy. The so-called "total uncertainty" associated with a prediction the informa- tion entropy of the corresponding predictive posterior distribution (Malinin & Gales 2021)) l). In UQ for natural language generation, an analogue is predictive entropy (PE), the entropy the distribu- tion of generable sequences by an LLM in response to prompt (Kuhn et al. 2023 Lin et al. [2024]
Farquhar et al. 2.24). Since not all generable sequences are typically known, PE is calculated using the plugin method to estimate the entropy of prompt-model pair's "answer distribution" (Kadavath et al. 2022). shortcoming this approach that treats all lexically distinct sequences as unique elements of an alphabet, even if they are semantically similar.
SNNE. SNNE aims to account for both "intra-and inter-cluster similarity." Provided scale factor and similarity function ff for assessing passage pairs (d,, dy), SNNE is defined as:
SNNE(g) = Σ
f(di. dj)
log i=1
exp
(11)
Nguyen et al. 2025). Our implementation uses ROUGE-L for f and scale factor which the authors report best-performing (Nguyen al. 2025)
16

---

Published as a conference paper at ICLR 2026
Like UEigV, KLE constructs graph over LLM responses to query, where nodes responses and edge weights are prescribed by the results of an NLI model. Instead of using the model''s categorical scores, however, KLE's semantic graph has weights Wi, g d + where g is if the NLI model predicts "entailment" for the provided response pair, 0.5 if predicts "neutral," and otherwise. The standard graph Laplacian (i.e., the difference between the degree matrix and the weight matrix) is then taken over the resultant semantic graph (Nikitin et al. 2024).
KLE.
d,),
KLE admits a choice of kernel apply to the Laplacian, resulting in a "density" matrix K. Given the eigenvalues (d1 < ..... < An) of K, the von Neumann entropy is calculated:
" VNE(K) Σ d log 4,.
(12)
1=1
Although the concept of von Neumann entropy has its roots in quantum physics (Petz 200T). it has also been used more recently in network science literature to quantify graph complexity (Minello ct al. 2019). Nikitin et al. (204) report the heat kernel best- pess--perforing overall:
Kheat 1L
(13)
Our implementation of KLE invokes Kheat with hyperparameter r 0.3, which the authors con- sider "reasonable default" that outperforms prior approaches without additional hyperparameter optimization (Nikitin et al. 2024).
Explicit estimation of semantic entropy. We do not include SNNE or KLE, as implemented, among our list of "explicit" estimators of SE, because they do not explicitly estimate the entropy of the "true" distribution over semantic equivalence classes. For instance, Nguyen et al. (2025) describe formulation of SNNE that equals DSE and Nikitin al. (2024) show that kernel exists to recover SE from KLE, but these are not necessarily the configurations used experimentally.
A.6 BRADLEY-TERRY CONFIDENCE INTERVALS
Gao et
al.
(2023) strength
offers estimates,
procedure for ascertaining Cls about ranks of which we cursorily summarize as follows:
Bradley-Tene Terry
Assuming a Bradley-Teryy model, let B (B1....PBm) be the true latent strengths of methods, and let B (BI..... Bm) be MLEs of the same. Given provided level a and "target" method i, we may establish (1 (1-a a) CI about Bi, as well as "slightly more conservative" intervals about the remaining strength estimates. Let nj be the number of resultant intervals whose lower bounds are greater than the upper bound of method i's. Similarly, let n2 be the number of intervals whose up- per bounds are less than the lower bound of method i's. The true rank of method i's Bradley-Terry strength in the integer in- terval [n] + 1, n2] with approximate probability
0
0.01
0.1
S) Hybrid
[1.4]
[1,4)
[1,4]
[1, 4)
NumSets
[7, 11]
[1]
[7, 11]
[7, 11]
Good-Turing
[1]
[7,
[7, 11]
[1]
UEigV
[1,3]
[1,3]
[1,3]
[1,3]
Hybrid
[4,
[4.
[4,5]
[4,
Hiugin
[1] [1]
[7, 11]
[7, 11]
Hes
[7] [1]
[7, 11]
[7, 11]
PE
[3.4]
[3.4]
[3.4]
[3. 4]
SNNE
[6.6]
[6,
[6,
16, 6
KLE
[1,3]
[1,
[1,3]
[1,
SE
[1]
[7.
[7.
[7. 11]
Table 3: Employing distinct values of regularization
parameter a. including zero regularization, we compute MLE Bradley-Terry latent strength scores and establish 95% CIs of each method's 's latent strength rank via the methods of Gao et al. (2023). Results are identical for all values considered.
Because Proposition 4.1 of Gao et al. (202)) only holds for fixed number of methhods/agemt/plaaerrr, we illustrate the "conservative" Cls Figure 3C. We re- peat this procedure for each UQ method establish Cls for all strength ranks. We refer interested readers to Gao et al. (003)) for further de- to estimate strength scores employs regularization parameter, which
tails. The algorithm we take as a = 0.1 in the main body of the paper. Table which repeats the analysis of Figure 3C for varying indicates that the rank order of our Bradley-Teryy results is not sensitive to this choice.
17

---

Published as a conference paper at ICLR 2026
Gemma-2-98
Gemma-3-122
Llama-3. 1-8B
Mistral-7B
Phi-3.5-3.8B
Information gain
1:75
1.59
1.50
1.50
1.50
1.50
1.25
1.00
1.00
1.00
1.00
1.00
0.75
0.50
0.50
0.50
0.50
0.50
0.25
5
16
Denary
Denary
Denary
Denary
Denary
SOLAD 20
POTATO
BloASQ
HotpotDA
Figure 6: Information gain with respect to the distribution over semantic categories, per denary (group of 10 samples). Because the first denary novel, its information gain is equal to plugin entropy. Results are averaged across queries for each dataset. Log scale is used on the x-axis to highlight differences for early denaries.
B THE
REGIME
Consider simple model of semantic category-abundance, where the probability that model pa- rameterized by generates a response belonging to the -oott prevalent semantic category in response to a query q follows Zipfian distribution, similar some models of species- abbudance (Ulrich et al. 2010):
((sr1q. 0)
(14)
rH[s]
where Hj the jth harmonic number (Kingsley Zipf 1932 the expected number of occurrences
of s, in sample of responses is less than 1, then [S]
> Mis
For sample size n 10, we expect at
least one semantic category to unobserved with just |S] 4.
C SAMPLE SIZE AND INFORMATION GAIN
Because the distribution semantic equivalence classes is not known, assume that SE" using = 100 is an adequate representation of the true estimand. In support of this choice, we consider the information gain per denary (i.e., group of ten samples, so the first denary the first ten samples, etc.) with respect to the distribution over semantic categories that is, the marginal information about this distribution obtained by observing each successive ten samples (Figure ( Across models and datasets, this quantity plateaus to low values well before = 100. Predictably, the first denary of samples tends provide the most information, especially for datasets whose queries can have multiple possible correct semantic categories.
D ADDITIONAL UNDERESTIMATION RESULTS
Detailed results for SE underestimation, broken out across four datasets, are shown in Figure 7
Results are qualitatively similar to those in Figure 2 with Hh typically underestimating SE* less than plugin DSE for practical sample sizes. Additionally, we present version of Figure 2 that includes white-box SE (Figure 8) and elaborate on the close tracking between SE and plugin DSE in Section
SE* can be exactly zero (c.g., when only one observed semantic category is observed at 100), hence we exclude such instances from Figures 2 and Occasionally, exceedingly rare semantic categories are revealed early, resulting in coverage-adjusted estimates that substantially exaggerate the distribution's tail mass. We observe a few instances of this phenomenon in our sampling from the 122 model on the POTATO dataset: three queries in POTATO for which Gemma-3- 12B elicits a non-zero SE" (calculated using a = 100), but a hybrid DSE (calculated using n = 10) that is order of magnitude larger. Inspection reveals that all three have two semantic categories observed at n 100, and both categories are also observed within the first 10 samples. Of these instances, all have SE 0.05 and one has SE 0.005. The impacts of this phenomenon
18

---

Published as a conference paper at ICLR 2026
Gemma-2-9B
Gemma-3-122
Llama-3.1-88
Mistral-7B
Phi-3.5-3.8B
2.0
2.0
2.0
2.0
1-8
1.8
1.2
1.2
1:0
0.8
HotpotaA Glmood
SQuAD 2.0 (Bia))))
POTATO (Repsee
BlcASQ lilowoo
HorpotQA (Rryyd
--..
SQuAD 2.0 (Eoagr)
POTATO (Boyg))
BoASQ (Rowns
Figure 7: Ratio of semantic entropy (SE) estimator with varying sample size (n 5, 10, 20, 75, 100) to white-box SE with = 100 (denoted SE*), with values below 1 suggesting underestimation (dotted grey line). Estimators shown are the plugin estimator of canonical discrete semantic entropy (DSE; dotted rose, indigo, green, and cyan lines) and the presented DSE estima- tor of Equation 10 (solid rose, indigo, green, and cyan lines). Results are averaged across queries for each dataset. Log scale is used on the x-axis to highlight differences between estimators with smaller sample sizes. Instances with a denominator of are ignored.
Gemma-2-9B
Gemma-3-122
Liama-3.1
Mistral-7B
Phi-3.5-3.8B
1.2
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.0
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4
100
25
100
Raysos (Ours)
<dd. Hpigee (Canonical DSE)
..e SE (White-Box)
88
Figure 8: Illustrating underestimation in discrete semantic entropy (DSE) calculation with typ- ical sample sizes. The ratios of semantic entropy (SH) estimators with varying sample size (n = 5, 10, 50, 75, 100) to white-box SE with = 100 (denoted SE") are shown, with values below suggesting underestimation (dotted grey line). The estimators displayed are the plugin esti- mator of canonical DSE (i.e., Equation 5 dotted green line), the "hybrid" SE estimator of Equation 10 (solid indigo line), and white-box SE (Equation 3 dotted cyan line). Results are averaged across queries within each dataset, then uniformly averaged across datasets. Log scale is used the x-axis to highlight differences between estimators with smaller sample sizes. Instances with a denominator of 0 are ignored.
visible in Figures and 7 where the curve for Gemma-3-12B on POTATO deviates from the pattern observed for other model-dataset pairs. If we exclude the most extreme instance by visualizing only instances with SÆ 2 0.005, all model-dataset pairs exhibit the familiar behavior (Figure
E How DISTINCT ARE CANONICAL DSE AND SE?
In Figures 8 and we observe that SE underestimates SE which is unsurprising, as it also invokes the negatively biased plugin entropy estimator (Fquation 3 It is surprising. however, that SE ap- pears to track plugin DSE so closely, since SE's probabilities are calculated via RBMCI (Fquation 4.. versus plugin DSE's empirical class frequencics. In Figure 3 SE does not appear definitively superior to plugin DSE overall on the downstream incorrectness detection task. Similarly, |arquhar et al. (2024) and Nikitin et al. (2024) report little difference in incorrectness detection performance metrics (e.g., AUROC) between SE and DSE.
In Table 4 we compare the empirical distribution of semantic class frequencies to the one induced by RBMCI, using the data from the experiment supporting Figure Our metrics of comparison are their Kullback-Leibler divergence (KI.) (Kullback)
19

---

Published as a conference paper at ICLR 2026
Gemma-2-38
Gemma-3-185 3-2B
LI
Mistral-7B
Phi-3.5-3.8B
1.2
(SE/SD (E/SE"
1.0
1.0
1.0
1.0
0.8
0.0
0.8
0.0
0.0
0.6
0.6
0.6
0.8
0.6
100
10
50
100
25
50
100
25
100
Baypri (Ours)
reee. f Apugin (Canonical DSE)
SE (White-Box)
Figure 9: Illustrating underestimation discrete semantic entropy (DSE) calculation with typ- ical sample sizes. The ratios of semantic entropy (SE) estimators with varying sample size (n = 5, 10, 25, 50, 75, 100) to white-box SE with n = 100 (denoted SE*) are shown, with values below 1 suggesting underestimation (dotted grey line). The estimators displayed the plugin esti- mator of canonical DSE (i.e., Equation 5) dotted green line), the "hybrid" SE estimator of Equation 10 (solid indigo line), and white-box SE (Equation dotted eyan line). Results averaged across queries within each dataset, then uniformly averaged across datasets. Log scale used on the x-axis highlight differences between estimators with smaller sample sizes. Instances with exceedingly small denominators (i.e., SE* < 0.005) are ignored.
& Leibler 1951), total variation distance (TVD) (Jacobs 2018). and the mean abso-
lute emrr (MAE), all computed by comparing each method's basse probability vectors. All three are exceedingly small across models and datasets; for in- stance, the KL divergence is nearly always well below 0.01 (Table 4 We consider this finding in light of Figure X where the curves for DSE- based methods are visually impacted by an anomalous sampling result, but that of SE is not. We con- clude that, while SE and plugin DSE may compute plugin entropy over nearly identical distributions, on av- erage, sequence log- pee-robabilities can help RBMCI doin-weight scenarios when the sample over-represents ex- ceptionally rare events.
Model
Dataset
KL
TVD
MAE
HotpotQA
0,001
0.006
0.005
SQuAD 2.0
0.002
0.009
0.006
Gemma-2-9B
POTATO
0.004
0.021
0.014
BioASQ
0.004
0.021
0.012
HotpotQA
0.000
0.001
0.001
SQuAD 2.0
0.000
0.002
0.002
Gemma-3-12B
POTATO
0.001
0.004
0.004
BioASQ
0.001
0.011
0.006
HotpotQA
0.009
0.025
0.018
SQuAD 2.0
0.010
0.029
0.019
Llama-3.1-8B
POTATO
0.008
0.034
0.021
BioASQ
0.018
0.053
0.025
HotpotQA
0.001
0.009
0.007
SQuAD 2.0
0.002
0.015
0.010
Mistral-v0.3-7B
POTATO
0.004
0.023
0,016
BioASQ
0.004
0.027
0.014
HotpotQA
0.003
0.013
0.010
F ADDITIONAL INCORRECTNESS DETECTION RESULTS
SQuAD 2.0
0.002
0.013
0.010
Phi-3.5-3.8B
POTATO
0.012
0.039
0.028
BioASQ
0.005
0.029
0.016
Table 4: Comparing semantic equivalence class probabil- ity distributions, induced by empirical frequencies (as in black-box DSE) or via Equation in white-box SE). The metrics considered total variation distance (TVD), and the mean absolute er- between the class-wise probability vectors (MAE). For each model, we average results over the questions of: Hot- potQA, the answerable questions of SQuAD 2.0. POTATO, and BioASQ. Values are rounded to four decimal places to adequately represent the smallest entries.
Detailed results for incorrectness classification across three datasets, five LLMs, and eleven UQ methods are shown in Figure 10 Error bars are 95% CIs about empirical AU- ROC values, calculated via DeLong's method (DeLong et al. 1988 Sun & Xu 2014).
Kullback-Leibler divergence (KL),
20

---

Published as a conference paper at ICLR 2026
HotpotQA
SQuAD 2.0 (Answerable)
BioASQ
0.8
AUROC
0.7
0.7
0.6
0.6
0.5
0.5
Gemma-2-4 2-98
Gemma 3-aaa----1-- 128
0.4
Liama-3
1-88 1-88
Mistral-vO. vorral-vO.3-771 3-7B
Phi-3.5-3.88
Gemma-2-98
Gemma Gemma-33-128
0.4
Llama-3.1
Mistral-vo 30.-7a 3-7B
1-88 1-88
Phi-3.5-3.88
Gemma-2
0.4
Gemma-3-122
Llama-3
Mistral-vO. v0.3--
3-188 1-88
Phi-3.5-3.88
Hyond
1$
Elyoria
SNNE
SE
NumSets
Veiy
Eriagia
Figure 10: Performance of incorrectness classification on single-answer QA for uncertainty mea- sures using n = 10 samples. Methods considered include black-box semantic alphabet size esti- mators (iS) Hyyrid NumSets, |Slor. UEigv), explicit discrete semantic entropy (DSE) estimators Hyryrr Plugin. Hcs)) and other uncertainty estimators, which include white-box (PE, SE) and black-box methods (SNNE, KLE). Error bars reflect 95% CIs about AUROC, via DeLong's 's method.
Gemma-2-98
Gemma-3-12B
Llama-3.1 3.1-8B
Mistral-7B
Phi-3,5-3.8B
1.2
(.) (.99
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.6
0.6
0.8
0.6
0.4
02
0.2
0.2
0.2
0.2
10
n
0
0
Iivaria (Ours)
NumSots
Illustrating underestimation in alphabet size estimation with typical sample sizes. The ratios of alphabet size estimators with varying sample size (n 5, 10, 20, 75, 100) to the ob- served number of semantic categories with n 100 (denoted 5 are shown, with values below 1 suggesting underestimation (dotted grey line). The estimators displayed are NumSets, used by the plugin estimator of canonical discrete semantic entropy (dotted indigo line) and the "bybrid" (solid indigo line). Results are averaged across queries within each dataset, then uniformly averaged across datasets. Log scale is used on the x-axis to highlight differences between estimators with smaller sample sizes.
Figure 11:
alphabet size estimator of Equation |
G UNDERESTIMATION OF SEMANTIC ALPHABET SIZE
In Figure 2 we exhibit underestimation of SE, using the ratio of DSE estimators (using varying sam- ple sizes) SE We provide a similar illustration Figure [I] NumSets, the implicit alphabet size estimator of canonical DSE, underestimates the number of semantic categories for typical sample sizes, and the the hybrid semantic alphabet size estimator does so by less.
H IMPLEMENTATION VARIATIONS
The relevant LLM UQ literature is not uniform in experimental setup. For instance, the plurality of considered works
10 sampled responses per query (Kuhn et al. 2023 Farquhar et al.
2024 Nikitin et al.) 2024 Nguyen et al. 2025), but Lin et al. (2024) use n 20. Some works
vary prompting templates across datasets (e.g., zero-shot multi-shot prompting) (Kuhn et al.
21

---

Published as a conference paper at ICLR 2026
Lin al. 2024 A variety of methods for automated LLM incorrectness evaluation have been employed. including thresholded ROUGE-L (Kuhn et al., 2023), binary LLM-as-judge (Far- quhar et al. (222)). thresholded consisteney-bars -based LLM-as-judge Lin al. (2024), and BERTScore (Nguyen et al. 2025). Although it is out of this work's 's scope to exhaustively ablate all such choices, attempt to be explicit in describing selections support future work.
2023
LLM USAGE
During this project, LLMs were employed at times for coding assistance.
22


Reviews:
[{"review_id": "Sf3izCJxxz", "reviewer": "Reviewer_T6sp", "summary": "Authors propose a better (less biased) estimator for semantic-alphabet-size in the semantic-entropy-related methods for UQ estimation in NLG tasks. \nThey show empirically that their semantic-alphabet-size is competitive with some of the SOTA methods.", "questions": "Q1. Would it possible to create a Figure-2-like plot but for semantic-alphabet-size estimators?\n\nQ2. Nitpick: could you please change the font in Fig 1 to a Serif font?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. The contribution is principled and to my best judgement technically correct.\n\nS2. The empirical performance of the proposed method is strong.\n\nS3. I find the empirical evaluation methodology sound.", "weaknesses": "W1. The evaluation could use more datasets (despite being sound methodologically).   \nThere are codebases available which would make the process rather straightforward, given the method proposed is rather simple to implement once the Semantic-Clustering is computed: https://github.com/AlexanderVNikitin/kernel-language-entropy. + maybe SimpleQA?(https://openai.com/index/introducing-simpleqa/)\n\nW2. Not sure how \"significant\" the contribution will prove to be, but I hate to judge this criterium, so please treat this complaint as secondary. If this was judged by TMLR criteria, my overall score would be an \"accept\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "300HJDQUc5", "reviewer": "Reviewer_foRW", "summary": "This paper addresses the problem of underestimating semantic entropy (SE) as an uncertainty estimate for black-box LLMs. The authors propose an interpretable semantic alphabet size estimator which, when used to adjust the discrete SE estimator for sample coverage, yields more accurate uncertainty estimation.", "questions": "- How quickly does the “white-box” SE estimate converge to the \"true\" SE as the number of samples increases? Including this in Figure 2 would be interesting.\n- Are 100 samples sufficient to approximate the true distribution over semantic clusters? An ablation with larger sample sizes (e.g., 1 k) could strengthen the claim that this is a valid reference.\n- How sensitive are the results to the semantic clustering method (e.g., choice of NLI model, thresholding, or entailment aggregation)?\n- The hybrid estimator uses a max(·) rule combining two estimators. Why is this choice theoretically justified versus, say, a weighted or Bayesian combination?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed semantic alphabet size estimator effectively mitigates the underestimation bias of discrete semantic entropy (DSE) in the few-sample regime.\n- The method is simple and interpretable, avoiding the complexity and hyperparameter dependence of recent SE extensions.\n- Empirically, it improves UQ performance and matches or exceeds the performance of SOTA approaches in the back-box setting.", "weaknesses": "- My main concern is that the proposed method is largely an adaptation of existing estimators to the SE setting. While the idea is insightful, it constitutes an incremental improvement over Farquhar et al. (2024) rather than a fundamentally new theoretical contribution.\n- The paper treats semantic clusters as fixed and does not analyze sensitivity to the clustering procedure (e.g., entailment thresholds or NLI model biases). Since clustering quality directly affects entropy estimates, this omission is a potential confounder that limits the scope of the contribution.\n- Experiments are restricted to relatively small models (<10B parameters), which raises questions about generalization to larger, frontier LLMs. Moreover, the paper does not compare against computationally cheaper uncertainty measures that avoid expensive sampling (e.g., G-NLL [1]) and which outperform SE and DSE with a single sample.\nMinor point: The paper inconsistently uses the abbreviations (like SE and DSE), alternating between the abbreviated and full forms without clear rationale, which makes it harder to follow.\n\n---\n\n[1] Lukas Aichberger, Kajetan Schweighofer, and Sepp Hochreiter. Rethinking uncertainty estimation in natural language generation. arXiv preprint arXiv:2412.15176, 2024.\n\n[2] Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zY1HzgShR2", "reviewer": "Reviewer_SfCH", "summary": "Goal:\n- This paper addresses a specific issue in discrete semantic entropy (DSE), a widely used black-box UQ method.  \n- The authors show that DSE tends to underestimate uncertainty when the response sample size is small, because it often fails to capture all unseen semantic clusters, therefore underestimating the number of distinct meaning clusters in its outputs (**semantic alphabet size**).\n\nEmpirical Validation:\n- To verify this bias, the authors compare DSE values from small-sample settings against 100-sample semantic entropy (treated as ground-truth uncertainty).\n- Across all model–dataset pairs, DSE consistently produces lower uncertainty estimates as sample size decreases.\n\nMethod:\nTo better estimate the semantic alphabet size, the paper considers 2 existing approaches:\n1. A Good–Turing coverage estimator, adapted from population ecology, which corrects for unseen categories in finite samples\n2. A spectral method based on eigenvalue decomposition (Lin et al.), which estimates alphabet size from the number of non-zero eigenvalues of a semantic similarity matrix.\nThe authors then propose a hybrid estimator that combines these two strategies: \n- when Good–Turing coverage becomes unreliable (e.g., each cluster appears only once), the method defaults to the spectral estimate; \n- otherwise, it uses a weighted combination to mitigate underestimation bias.\nUsing this hybrid alphabet size, they further define a hybrid semantic entropy estimator, extending classic ecological entropy formulations to the semantic domain.\n\nEvaluation:\n1. Following standard UQ evaluation protocols, the study benchmarks ten uncertainty estimators across four LLM families and four datasets.\n2. Because raw AUROC scores can vary with model and dataset, the authors adopt a Bradley–Terry latent strength model to aggregate pairwise AUROC comparisons into an overall ranking with confidence intervals—an extension of the win-rate framework proposed by Nikitin et al. (2024).\nOverall, results show that the hybrid semantic alphabet size estimator performs most consistently across settings, while the hybrid semantic entropy estimator reduces but does not fully eliminate the negative bias in DSE.", "questions": "1. Equation (8) appears to be missing the definition or notation for $p_i$. \n2. I have multiple questions regarding the experiment results (Figure 3): \n- Why was white-box semantic entropy not included in the comparison, since you also consider the PE? \n- In addition, both hybrid semantic entropy ($\\hat{H}_\\text{hybrid}$) and discrete semantic entropy (DSE) show notably lower rankings (around 7–10). Could you elaborate on why these methods perform substantially worse than others? Conceptually, both $\\hat{H}_\\text{hybrid}$ and DSE should still be affected by the same underestimation bias that motivates your paper. If the semantic alphabet size were estimated more accurately, shouldn’t this also mitigate the underestimation problem in DSE to some extent?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The problem itself is important.\n2. The solution is simple.\n3. The evaluation considers the weakness of AUROC and proposes the Bradley-Terry latent strength scores, which make the evaluation more reliable. \n4. The paper is easy to understand and well-structured. I like its simplicity and enjoy reading it.", "weaknesses": "1. Limited methodological novelty and analysis.\n- The proposed hybrid estimator mainly combines two existing techniques \n   -  the Good–Turing coverage estimator from population ecology\n   - a spectral eigenvalue-based method\nI am not saying that combining prior ideas is not good, and my point is that it can be done with a deeper analysis of each individual methods and the hybrid operation can be done more adaptively. \n- For example, the Good–Turing approach is well established for ecological sampling, yet the sampling distribution of LLM-generated responses can be very different. It remains unclear why this method should perform reliably in the LLM setting or under what conditions it might fail. \n- Similarly, the hybrid design could be more principled — for instance, by characterizing when each component estimator is more accurate, rather than simply prioritizing the larger estimate under certain cases. This max operation inherently assumes that both the spectral-based method and Good-Turing coverage-based methods are also underestimating the ground truth semantic cluster size. But is this assumption true?\n\n2. Inconsistent motivation and results.\nThe empirical validation (Figure 2 and table 1) assumes that semantic entropy computed with 100 samples represents the ground truth. However, this assumption itself is not rigorously justified. Moreover, the paper’s motivation is to improve DSE-based uncertainty estimation, yet the final results show that DSE-derived methods (including the hybrid entropy estimator) still lag behind KLE and the semantic alphabet size estimator in overall performance (always ranking from 7-10). This weakens the narrative that improving DSE necessarily leads to better uncertainty quantification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XfrKcIvFus", "reviewer": "Reviewer_RUVy", "summary": "I really like this paper. It takes a fairly simple premise: sampling-based estimators of the clustering distribution will tend to miss the tail, leading to systemic underestimation of semantic entropy. But: We have robust estimators for the size of the tail. So, we can get modified estimators of either the number of semantic clusters, or the semantic entropy. And, these modified estimators lead to improved performance.", "questions": "* It’s interesting that you find that the adjusted number of sets outperforms the adjusted semantic entropy. It would be nice to see how the “ground truth” number of sets compares with “ground truth” semantic entropy... if in this setting ground truth number of sets outperforms ground truth semantic entropy, it might suggest something interesting about miscalibration of the LLM. \n* I’d like to see (maybe in the discussion) discussion of how one might extend this to white-box SE estimation (which will tend to suffer from the same problem, albeit to a lesser extent).", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* It’s a simple idea... in a good way. It identifies a clear problem, finds parallels in the species sampling literature for heavy tailed distributions, and proposes a simple, interpretable solution.\n* It leads to improvement. \n* The evaluation is very thoughtful (one of the best discussions of how to evaluate UQ methods I have seen)\n* The paper is clear and well-written", "weaknesses": "* It’s not strictly improving on SoTA... but it is compatible with SoTA using a much simpler and more interpretable paradigm\n* The idea of using a Good-Turing estimator has been explored in related problems in the literature, as acknowledged by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)