Estimating Semantic Alphabet Size for LLM Uncertainty Quantification

## Abstract

Abstract Many black-box techniques for quantifying the uncertainty of large language models (LLMs) rely on repeated LLM sampling, which can be computationally expensive.
Therefore, practical applicability demands reliable estimation from few samples.
Semantic entropy (SE) is a popular sample-based uncertainty estimator with a discrete formulation attractive for the black-box setting.
Recent extensions of semantic entropy exhibit improved LLM hallucination detection, but do so with less interpretable methods that admit additional hyperparameters.
For this reason, we revisit the canonical discrete semantic entropy estimator, finding that it underestimates the “true” semantic entropy, as expected from theory.
We propose a modified semantic alphabet size estimator, and illustrate that using it to adjust discrete semantic entropy for sample coverage results in more accurate semantic entropy estimation in our setting of interest.
Furthermore, our proposed alphabet size estimator flags incorrect LLM responses as well or better than recent top-performing approaches, with the added benefit of remaining highly interpretable.

## 1 Introduction

Large language models (LLMs) are not fact engines. They have been shown to forget provided context (Liu et al., 2024), fabricate records (Lee et al., 2023), and otherwise hallucinate (Ji et al., 2023). LLMs’ underlying training data may be of mixed factual reliability, but models can also hallucinate when they have ample knowledge to respond adequately (Simhi et al., 2024).
In risk-sensitive settings, it may be prudent for systems to abstain when uncertainty is high (or, alternatively, confidence is low) (Murphy, 2022; Hasan et al., 2025).
For these reasons and others, it is prudent to estimate LLMs’ intrinsic uncertainty.

Uncertainty quantification (UQ) in LLMs is particularly challenging, however, in part due to their computational scale (Liu et al., 2025).
Extensive sampling can be financially prohibitive or computationally intractable, and these concerns are magnified if the UQ method is computationally complex with respect to the number of samples.
Furthermore, internal activations and sequence log-probabilities are not always available from commercial inference providers (Farquhar et al., 2024), potentially disqualifying so-called “white-box” methods.
Therefore, sample-efficient UQ in the black-box setting - where LLM internals are assumed inaccessible - is a crucial area of research for deploying trustworthy AI systems.

We provide empirical evidence that canonical discrete semantic entropy underestimates the “true” SE for typical sample sizes, on average (Figure [2](https://arxiv.org/html/2509.14478v1#S4.F2)). To address this limitation, we first suggest a straightforward modification to existing alphabet size estimators, where we draw, in part, from population ecology. Then, we use it to adjust entropy estimates for sample coverage, resulting in reduced bias (Figure [2](https://arxiv.org/html/2509.14478v1#S4.F2)) and more accurate estimation of the “true” semantic entropy, compared to other black-box semantic entropy estimators (Table [1](https://arxiv.org/html/2509.14478v1#S4.T1)).

Finally, we evaluate our suggested estimators for inaccuracy classification in sentence-length LLM responses.
In our experiments, the aforementioned coverage-adjusted estimator outperforms other explicit discrete estimators of semantic entropy.
More strikingly, the alphabet size estimator alone outranks all other considered UQ methods in overall strength - except for one, from which we do not statistically distinguish performance, after accounting for uncertainty in establishing an overall rank of methods.
Our results indicate that semantic alphabet size estimation, which is highly interpretable, can perform as well or better than state-of-the art methods in LLM inaccuracy detection.

Figure: Figure 1: High-level schematic of semantic alphabet size estimation for LLM uncertainty quantification (Section [3](https://arxiv.org/html/2509.14478v1#S3.SS0.SSS0.Px3)). (A) Generate LLM responses to a query. (B) Assign responses to categories of shared meaning. (C) Estimate semantic alphabet size, accounting for semantic classes unobserved in the sample (Equation [9](https://arxiv.org/html/2509.14478v1#S3.E9)). LLM response examples are hypothetical for illustrative purposes.
Refer to caption: x1.png

## 2 Preliminaries

#### Information entropy.

Information entropy quantifies the expected surprise, or uncertainty, of a random variable. For a discrete random variable $X$ with finite alphabet $S$, it is given by:

$$ $\mathbb{H}(X)=-\sum_{s\in S}p(s)\log p(s),$ (1) $$

where $p(s)$ depicts the probability of symbol $s$ (Shannon, 1948).
When a random variable’s probability density is not known analytically, entropy estimators are employed. A well-studied approach in the finite-sample regime is the plugin approximation:

$$ $\hat{\mathbb{H}}_{plugin}(X)=-\sum_{i=1}^{k}\hat{p}(s_{i})\log\hat{p}(s_{i}),$ (2) $$

where $k$ is the number of distinct observations in a sample, and $\hat{p}(s_{i})$ conveys the empirical frequency of $s_{i}$ among the observations.

#### Semantic classification.

Lexically distinct text sequences may belong to the same semantic equivalence class (equivalently, “semantic category” or “semantic set”) - i.e., a set of text sequences with mutually shared meaning. Sequence $d_{1}$ entails sequence $d_{2}$ if $d_{1}$ implies $d_{2}$ (Dagan et al., 2005). Kuhn et al. (2023) assign sequences $d_{1}$ and $d_{2}$ to a shared semantic equivalence class under so-called “strict entailment” if their textual entailment is bidirectional.

#### Semantic entropy.

Semantic entropy (SE), introduced by Kuhn et al. (2023), aims to quantify intrinsic LLM uncertainty with a three-step procedure:

- 1.
Sampling: Given a query $q$, generate $n$ passages $d_{1},d_{2},\dots,d_{n}$.
- 2.
Semantic Clustering: Iterating over pairs $(d_{i},d_{j})$, determine if both sequences belong to the same semantic equivalence class. Greedily assign passages to classes based on the pairwise semantic classifications.
- 3.
Estimation: Semantic equivalence classes are treated as symbols in an alphabet $S$. SE is the entropy calculated over observed semantic equivalence classes:
$\displaystyle SE(q|\theta)$
$\displaystyle=-\sum_{s\in S}p(s|q,\theta)\log p(s|q,\theta)$
(3)
where $p(s|q,\theta)$ represents the probability that an LLM parameterized by $\theta$ generates a passage belonging to semantic equivalence class $s$ in response to a query $q$.

Because the distribution over semantic sets is not known, it is approximated from response probabilities $p(d|q,\theta)$ using so-called “Rao–Blackwellized Monte Carlo integration”:

$$ $p(s_{i}|q,\theta)\approx\frac{\sum_{d}\vmathbb{1}_{d\in s_{i}}p(d|q,\theta)}{\sum_{d}p(d|q,\theta)},$ (4) $$

where $d\in s$ indicates that response $d$ belongs to semantic equivalence class $s$, and $p(d|q,\theta)$ are response probabilities returned by the LLM, for each of the $k$ observed semantic categories $s_{1},s_{2},\dots,s_{k}$ (Farquhar et al., 2024).

#### The black box setting.

Response probabilities are not always available. Farquhar et al. (2024) consider a discrete formulation of semantic entropy (DSE), where aggregated document probabilities for each semantic equivalence class are replaced with empirical class frequencies:

$$ $\displaystyle DSE(q|\theta)$ $\displaystyle=-\sum_{i=1}^{k}\Bigg(\frac{\sum_{d}\vmathbb{1}_{d\in s_{i}}}{n}\Bigg)\log\Bigg(\frac{\sum_{d}\vmathbb{1}_{d\in s_{i}}}{n}\Bigg)$ (5) $$

with $n$ sampled passages and $k$ observed semantic categories. This is the plugin estimator (Equation [2](https://arxiv.org/html/2509.14478v1#S2.E2)) applied to semantic entropy (Equation [3](https://arxiv.org/html/2509.14478v1#S2.E3)). Both SE and DSE correspond with hallucination rate in question-answering problems (Farquhar et al., 2024).

#### Generalizations.

Recent work is said to have generalized semantic entropy.
Two pertinent examples are Kernel Language Entropy (KLE) and Semantic Nearest Neighbor Entropy (SNNE), which reported state-of-the-art performance for incorrectness classification (Nikitin et al., 2024; Nguyen et al., 2025).
These stronger estimators may come at the expense of interpretability, in part owing to implementation complexity (e.g., applying a kernel embed graph nodes in KLE) and introduction of additional hyperparameters and design choices (e.g., similarity function and scale parameter in SNNE).
We elaborate on KLE and SNNE definitions and implementation details in Appendix [A.5](https://arxiv.org/html/2509.14478v1#A1.SS5).

## 3 Methods

The total number of semantic equivalence classes (i.e., the semantic alphabet size $|S|$) that may be ellicited from an LLM in response to a prompt is generally unknown, and not all categories are necessarily observed in the sample (i.e., $k<|S|$).
For instance, under a simple Zipfian model of semantic category-abundance, we expect at least one category to be unobserved for sample size $n=10$ if $|S|>4$ (Appendix [B](https://arxiv.org/html/2509.14478v1#A2)).
This situation is known as the under-sampled regime, where the empirical distribution over semantic categories can be less surprising than than the true one.
In other words, the plugin method for DSE may underestimate LLM uncertainty, which we illustrate in Figure [2](https://arxiv.org/html/2509.14478v1#S4.F2).
For this reason, we are interested in methods for estimating semantic alphabet size and adjusting for it when estimating semantic entropy.

#### Semantic alphabet size.

Because plugin $DSE$ does not directly account for unobserved semantic categories, the implicit alphabet size used by the plugin estimator is $k$, called “NumSets” by Lin et al. (2024). In the under-sampled regime, NumSets underestimates $|S|$, so a more accurate semantic alphabet size estimator for small $n$ is desirable.
Parallels exist with the so-called “unseen species” problem in population ecology: given a sample of $n$ observations belonging to one or more species, estimate the number of yet unseen species that would be discovered by collecting additional observations (Fisher et al., 1943).
In this setting, the sample coverage $C$, the fraction of all possible categories observed in a sample (i.e., $C=\frac{k}{|S|}$), is also of interest (Chao & Shen, 2003).

The so-called “Good-Turing” sample coverage estimator is $\hat{C}_{GT}=1-\frac{f_{1}}{n}$, where $f_{1}$ is the number of singletons, or semantic categories observed only once (Good, 1953). Modest arithmetic converts the Good-Turing sample coverage estimator into an alphabet size estimator:

$$ $\widehat{|S|}_{GT}=\frac{kn}{n-f_{1}}.$ (6) $$

More recently, Lin et al. (2024) suggested a “continuous” NumSets analogue: responses are interpreted as nodes of a fully-connected graph $G$ with weights $w_{ij}=\frac{a_{ij}+a_{ji}}{2}$, where $a_{ij}$ is the entailment probability for response pair $d_{i}$, $d_{j}$, via an NLI model. Given the eigenvalues ($\lambda_{1}<\cdots<\lambda_{n}$) of $G$’s normalized Laplacian, the estimator is given by:

$$ $U_{EigV}=\sum_{i=1}^{n}\max(0,1-\lambda_{i}).$ (7) $$

#### Coverage-adjusted entropy.

To quantify a population’s ecological diversity, Chao & Shen (2003) provide a coverage-adjusted discrete entropy estimator that scales empirical category frequencies by estimated sample coverage:

$$ $\hat{\mathbb{H}}_{CS}=-\sum_{i=1}^{k}\frac{\hat{C}_{GT}\hat{p}_{i}\log(\hat{C}_{GT}\hat{p}_{i})}{1-(1-(\hat{C}_{GT}\hat{p}_{i}))^{n}}.$ (8) $$

This so-called “Chao-Shen” estimator is consistent and less biased than many empirical alternatives (Vu et al., 2007; Pinchas et al., 2024).

#### Hybrid estimators.

First, we adapt the two aforementioned semantic alphabet size estimators to address shortcomings of each.
When the number of singletons is zero, $\widehat{|S|}_{GT}$ reduces to NumSets, and it is undefined when all samples belong to distinct semantic categories. On the other hand, $U_{EigV}$ can be less than $k$, which is a lower bound for $|S|$. To remediate the above limitations, we propose an alternative “hybrid” semantic alphabet size estimator:

$$ $\widehat{|S|}_{Hybrid}=\begin{cases}U_{EigV},&\text{if $f_{1}=n$}\\ max\Big(\widehat{|S|}_{GT},U_{EigV}\Big),&\text{otherwise.}\end{cases}$ (9) $$

Additionally, we propose a Chao-Shen-like DSE estimator that takes the form of Equation [8](https://arxiv.org/html/2509.14478v1#S3.E8), but invokes the hybrid semantic alphabet size estimator for coverage adjustment:

$$ $\hat{\mathbb{H}}_{Hybrid}=-\sum_{i=1}^{k}\frac{\frac{k\hat{p}_{i}}{\widehat{|S|}_{Hybrid}}\log\Big(\frac{k\hat{p}_{i}}{\widehat{|S|}_{Hybrid}}\Big)}{1-\Big(1-\frac{k\hat{p}_{i}}{\widehat{|S|}_{Hybrid}}\Big)^{n}}.$ (10) $$

## 4 Experiments

### 4.1 Experimental settings

#### Models.

Following Farquhar et al. (2024), we focus on fine-tuned models - in our case, the instruction-tuned models of Gemma-2-9B (Team et al., 2024), Llama-3.1-8B (Grattafiori et al., 2024), Mistral-v0.3-7B, and Phi-3.5-3.8B (Abdin et al., 2024).
We perform text generation at two temperatures, for distinct purposes.
Following related prior work, we sample at temperature $\tau=1.0$ to calculate uncertainty scores and again at $\tau=0.1$ to obtain a “best guess” response for assessing the correctness of LLM responses (Farquhar et al., 2024; Nikitin et al., 2024; Nguyen et al., 2025).

Figure: Figure 2: Illustrating underestimation in discrete semantic entropy calculation with typical sample sizes. The ratios of semantic entropy estimators with varying sample size ($n=5,10,25,50,75,100$) to white-box semantic entropy with $n=100$ (denoted $SE^{*}$) are shown, with values below $1$ suggesting underestimation (dotted grey line). The estimators displayed are the plugin estimator of canonical discrete semantic entropy (i.e., Equation [5](https://arxiv.org/html/2509.14478v1#S2.E5), dotted indigo line) and the “hybrid” semantic entropy estimator of Equation [10](https://arxiv.org/html/2509.14478v1#S3.E10) (solid indigo line). Results are averaged across queries within each dataset, then uniformly averaged across datasets. Log scale is used on the x-axis to highlight differences between estimators with smaller sample sizes. Instances with only one observed semantic category at $n=100$, resulting in a denominator of 0, are ignored.
Refer to caption: x2.png

#### Datasets.

We consider question-answering datasets wherein each query has either one or multiple correct answer(s). For the former, we use the validation sets of HotpotQA (Yang et al., 2018) and SQuAD 2.0 (Rajpurkar et al., 2018; 2016), which contain $1852$ and $11864$ exemplars, respectively. Our experiments require extensive sampling of LLM responses, and BEC demands combinatorially many language model calls.
For these reasons, and because SQuAD 2.0 is much larger than the other data sets, our experiments using it are performed on a random $20\%$ subset. We also consider queries with a wider variety of possible correct answers. The first such dataset is BioASQ, a biomedical QA benchmark with $4719$ exemplars. Each question in BioASQ has between $1$ and $15$ reference answers. We also prepare a small ($131$ exemplars) supplementary dataset called Plethora Of accepTable cATegOries (POTATO), for which the number of possible correct semantic categories has an even larger range (up to $722$). We discuss POTATO in greater detail in Section [A.2](https://arxiv.org/html/2509.14478v1#A1.SS2).

### 4.2 Metrics and baselines

#### Semantic entropy estimator evaluation.

For a given prompt-LLM pair, $|S|$ and the true probability distribution over semantic equivalence classes are unknown, so semantic alphabet size and semantic entropy have no ground truths. Instead, we assume that white-box semantic entropy with a large number of samples ($n=100$), denoted $SE^{*}$, is interchangeable with the true estimand (i.e., $SE^{*}$ well-represents the true entropy over all possible semantic categories).
When illustrating the degree to which $SE^{*}$ is underestimated (i.e., Figure [2](https://arxiv.org/html/2509.14478v1#S4.F2)), we report the ratio of the estimator of interest to $SE^{*}$. Since information entropy is non-negative, values below $1$ indicate underestimation. Our primary evaluation metric for entropy estimators, however, is mean-squared error (MSE): we assess the MSE between an estimate using the small sample size ($n=10$) and $SE^{*}$.

#### Incorrectness evaluation.

We also assess the ability of estimators to classify LLM responses as “correct” or “incorrect,” sometimes referred to as “hallucination” or “confabulation” detection (Farquhar et al., 2024).
We calculate the area under the receiver operating characterstic curve (AUROC), corresponding to the probability that a randomly selected incorrect LLM response has a higher uncertainty score than a randomly selected correct response.
Empirical AUROC values are point estimates, which themselves have uncertainty. Nikitin et al. (2024) point out that such uncertainty is strongly driven by the model and dataset, rather than UQ method, motivating head-to-head comparisons and evaluation by win rate.
We advance a similar approach, but we rely on Bradley-Terry latent strength scores (Zermelo, 1929; Bradley & Terry, 1952), calculated via minorization-maximization (Caron & Doucet, 2012; Hunter, 2004), allowing us to establish an overall rank of methods.

We also employ a Monte Carlo procedure that attempts to account for uncertainty in estimating both AUROC and strength scores.
First, we fit Gaussian uncertainty distributions about AUROC point estimates from the $95\%$ confidence intervals (CIs) obtained via DeLong’s method (DeLong et al., 1988; Sun & Xu, 2014), a modeling assumption we justify by the approximate normality of the U statistic (Mann & Whitney, 1947).(^1^11In principle, samples from these distributions may exist outside the $[0,1]$ range, but this is unlikely in our case, given the AUROC CI bounds (Figure [7](https://arxiv.org/html/2509.14478v1#A4.F7)).)
For each model-dataset pair, we simulate $L=100$ matches between each pair of methods by comparing AUROC values sampled from their corresponding uncertainty distributions.
After obtaining strength scores from the simulated matches, we calculate CIs about the Bradley-Terry scores and establish $95\%$ CIs about the ranks of latent strengths using the methods of Gao et al. (2023). We provide additional details in Appendix [A.6](https://arxiv.org/html/2509.14478v1#A1.SS6).

#### Baselines.

Because our priority is black-box uncertainty estimation, we compare with other black-box approaches.
For semantic entropy estimation, we compare our presented hybrid approach (i.e., Equation [10](https://arxiv.org/html/2509.14478v1#S3.E10)) to the other explicit discrete semantic entropy estimators: the canonical discrete approach of Farquhar et al. (2024) (i.e., the plugin estimator, Equation [5](https://arxiv.org/html/2509.14478v1#S2.E5)) and the coverage-adjusted estimator of Chao & Shen (2003) (i.e., Equation [8](https://arxiv.org/html/2509.14478v1#S3.E8)).

For incorrectness classification, we consider the three aforementioned explicit semantic entropy estimators, four alphabet size estimators, and three other uncertainty methods.
The alphabet size estimators are the number of semantic categories (i.e., NumSets) (Kuhn et al., 2023), the alphabet size estimator converted from the Good-Turing sample coverage estimator (Good, 1953), the spectral estimator of Lin et al. (2024) (i.e., $U_{EigV}$), and our presented hybrid alphabet size estimator (i.e., Equation [9](https://arxiv.org/html/2509.14478v1#S3.E9)).
The other uncertainty methods are Predictive Entropy (PE, see Appendix [A.5](https://arxiv.org/html/2509.14478v1#A1.SS5)) (Kadavath et al., 2022), SNNE (Nguyen et al., 2025), and KLE (Nikitin et al., 2024).

### 4.3 Results

**Table 1: Empirically evaluating the accuracy of explicit discrete semantic entropy estimators. Values reflect MSE between the estimated value using $n=10$ samples and white-box semantic entropy with $n=100$ (i.e., $SE^{*}$). The lowest MSE value for each model-dataset pair is shown in bold. Intervals, shown in grey, reflect $95\%$ CIs via the standard error of the mean.**
| Dataset | Estimator | Gemma-2-9B | Llama-3.1-8B | Mistral-7B | Phi-3.5-3.8B |
| --- | --- | --- | --- | --- | --- |
| HotpotQA | $\widehat{\mathbb{H}}_{Plugin}$ | 0.47 ± 0.02 | 0.72 ± 0.03 | 0.60 ± 0.03 | 0.62 ± 0.03 |
| $\widehat{\mathbb{H}}_{CS}$ | 0.39 ± 0.02 | 0.57 ± 0.03 | 0.46 ± 0.03 | 0.47 ± 0.03 |  |
| $\widehat{\mathbb{H}}_{Hybrid}$ | 0.30 ± 0.02 | 0.45 ± 0.02 | 0.39 ± 0.02 | 0.39 ± 0.02 |  |
| SQuAD 2.0 | $\widehat{\mathbb{H}}_{Plugin}$ | 0.69 ± 0.03 | 0.84 ± 0.03 | 1.40 ± 0.04 | 1.46 ± 0.04 |
| $\widehat{\mathbb{H}}_{CS}$ | 0.51 ± 0.03 | 0.60 ± 0.03 | 0.86 ± 0.06 | 0.91 ± 0.06 |  |
| $\widehat{\mathbb{H}}_{Hybrid}$ | 0.43 ± 0.02 | 0.50 ± 0.02 | 0.68 ± 0.03 | 0.74 ± 0.03 |  |
| POTATO | $\widehat{\mathbb{H}}_{Plugin}$ | 0.42 ± 0.08 | 0.71 ± 0.12 | 1.75 ± 0.15 | 1.96 ± 0.16 |
| $\widehat{\mathbb{H}}_{CS}$ | 0.30 ± 0.07 | 0.52 ± 0.11 | 0.94 ± 0.29 | 1.57 ± 0.55 |  |
| $\widehat{\mathbb{H}}_{Hybrid}$ | 0.27 ± 0.06 | 0.42 ± 0.09 | 0.70 ± 0.11 | 0.72 ± 0.11 |  |
| BioASQ | $\widehat{\mathbb{H}}_{Plugin}$ | 1.66 ± 0.03 | 1.81 ± 0.03 | 2.06 ± 0.03 | 1.85 ± 0.02 |
| $\widehat{\mathbb{H}}_{CS}$ | 0.96 ± 0.04 | 1.05 ± 0.05 | 1.31 ± 0.08 | 0.98 ± 0.04 |  |
| $\widehat{\mathbb{H}}_{Hybrid}$ | 0.78 ± 0.02 | 0.78 ± 0.02 | 0.72 ± 0.02 | 0.82 ± 0.02 |  |

#### Canonical DSE underestimates “true” semantic entropy.

The plugin estimator for information entropy $\hat{\mathbb{H}}_{plugin}$ has a theoretically-established negative bias (Basharin, 1959; Harris, 1975), which largely governs its mean-squared error (Antos & Kontoyiannis, 2001).
We are interested in observing this phenomenon empirically, where we do not have access to the true distribution over semantic equivalence classes.
In Figure [2](https://arxiv.org/html/2509.14478v1#S4.F2), we compare plugin DSE estimates to $SE^{*}$ (dotted lines), indicating that the canonical plugin DSE approach underestimates its quantity of interest for practical sample sizes.
Such underestimation can be problematic for reliable UQ, because drawing a large number of samples from an LLM can be costly and time-consuming at scale.

#### Our proposed estimator improves the accuracy of discrete semantic entropy estimation.

Figure: Figure 3: Heatmap illustrating the portion of model-dataset pairs, rounded to the nearest integer, for which a row’s method achieved a larger AUROC point estimate than a column’s method. Uncertainty methods are organized into three groups: (i) semantic alphabet size estimators, (ii) semantic entropy estimators, and (iii) other uncertainty methods. The hybrid discrete semantic entropy estimator consistently outperforms other DSE estimators, and the hybrid semantic alphabet size estimator consistently outperforms other alphabet size estimators.
Refer to caption: x3.png

In Figure [2](https://arxiv.org/html/2509.14478v1#S4.F2), we observed that $\hat{\mathbb{H}}_{Hybrid}$ underestimated its target quantity by less than canonical plugin DSE for a range of sample sizes. Detailed results, broken out by dataset, are shown in Appendix [C](https://arxiv.org/html/2509.14478v1#A3) (Figure [6](https://arxiv.org/html/2509.14478v1#A3.F6)), where similar patterns are exhibited. We perform a more granular comparison of semantic entropy estimator accuracy with $n=10$ in Table [1](https://arxiv.org/html/2509.14478v1#S4.T1), where $\hat{\mathbb{H}}_{Hybrid}$ consistently achieves the lowest MSE among explicit discrete semantic entropy estimators across four models and four datasets.

#### We improve upon prior work on incorrectness detection.

LLM uncertainty estimation is often employed as a proxy for hallucination, confabulation, or incorrectness detection among LLM responses (Kuhn et al., 2023; Farquhar et al., 2024).
Following Nikitin et al. (2024), we calculate pairwise AUROC win rates between models – for each model-dataset pair, a win is recorded in method $i$’s favor over method $j$ if the corresponding AUROC for method $i$ is greater than that of method $j$ (Figure [3](https://arxiv.org/html/2509.14478v1#S4.F3)). We characterize overall performance of each uncertainty method via Bradley-Terry latent strength scores. Figure [4](https://arxiv.org/html/2509.14478v1#S4.F4)A displays maximum likelihood estimates of the methods’ strength scores from pairwise AUROC comparisons. Here, $\widehat{|S|}_{Hybrid}$ achieves the highest strength score, followed closely by KLE.

Figure: Figure 4: Establishing overall performance of ten UQ methods on incorrectness detection. (A) Bradley-Terry latent strength scores from pairwise comparison of AUROC point estimates. (B) Bradley-Terry latent strength scores after accounting for uncertainty in estimating AUROC. Error bars are “conservative” CIs about strength scores, which may be slightly stricter than $95\%$; see Section [A.6](https://arxiv.org/html/2509.14478v1#A1.SS6) for details. (C) For each method, we establish $95\%$ CIs about the rank of Bradley-Terry latent strength MLEs (Gao et al., 2023) for the incorrectness detection task; see Section [A.6](https://arxiv.org/html/2509.14478v1#A1.SS6) for details. We highlight i) semantic alphabet size estimators, ii) explicit discrete semantic entropy estimators, and iii) other uncertainty estimators. The interval [a, b] denotes all integers from $a$ to $b$, inclusively. The CI for SNNE reflects [6, 6] and is extended for readability.
Refer to caption: x4.png

We highlight the uncertainty in both AUROC and Bradley-Terry estimation in Figure [4](https://arxiv.org/html/2509.14478v1#S4.F4)B (CIs about Bradley-Terry estimates, which may be slightly stricter than $95\%$) and Figure [4](https://arxiv.org/html/2509.14478v1#S4.F4)C ($95\%$ CIs around each method’s rank by Bradley-Terry score).
We observe that the hybrid estimators presented herein outperform estimators of the same type - i.e., $\widehat{|S|}_{Hybrid}$ achieves the highest strength score among semantic alphabet size estimators, and $\widehat{|S|}_{Hybrid}$ does the same among explicit semantic entropy estimators. Finally, we find that $\widehat{|S|}_{Hybrid}$ shares the [1, 2] CI with KLE (Figure [4](https://arxiv.org/html/2509.14478v1#S4.F4)C), a conceptually complex approach that may be more difficult to interpret.
We do not statistically distinguish the incorrectness detection performance of $\widehat{|S|}_{Hybrid}$ from that of KLE, with both methods holding top performance, after accounting for uncertainty in our rank-generating procedure.
Our results are consistent with Kuhn et al. (2023)’s observation that the number of observed semantic categories is itself “a reasonable uncertainty measure.”

## 5 Related work

Herein, we briefly review related contributions not discussed elsewhere in this work.

#### Earlier LLM uncertainty quantification methods.

Linguistic Confidence assesses whether the LLM articulates its confidence in its response (Mielke et al., 2022). The P(True) method of Kadavath et al. (2022) similarly relies on an LLM’s self-perceived uncertainty, asking a language model if a response is “True” or “False,” with the response probability of “False” ultimately reported. SelfCheckGPT draws $n+1$ responses from an LLM and assesses the consistency of each sentence of the first response with each of the $n$ following responses (Manakul et al., 2023). In prior work, such approaches have been consistently superseded by the methods otherwise described herein (Kuhn et al., 2023; Lin et al., 2024; Farquhar et al., 2024; Nikitin et al., 2024; Nguyen et al., 2025).

#### Uncertainty and internal representations.

In the white-box setting, it may be desirable to ascertain LLM uncertainty from internal representations, rather than repeated sampling. Han et al. (2024b) and Han et al. (2024a), for example, approach this from an interpretability point of view, building on earlier work on semantic entropy (Kuhn et al., 2023; Farquhar et al., 2024).
Other analyses, however, contend that so-called “truthfulness encodings” are difficult to generalize (Orgad et al., 2025), warranting further examination.

#### Alphabet size estimation.

The Good-Turing estimator discussed herein accounts for singletons in the sample (Chao & Shen, 2003). Alternative alphabet size estimators accounting for doubletons (i.e., semantic categories appearing exactly twice in the sample) (Chao, 1987) and tripletons (i.e., categories appearing exactly three times) (Lanumteang & Böhning, 2011) exist, but they are undefined when the number of doubletons is zero.

#### Unbiased entropy estimation.

The entropy estimator of Montgomery-Smith & Schürmann (2014) is unbiased, but it is incompatible with the possibility of unobserved semantic equivalence classes. Otherwise, no unbiased estimator exists, to the best of our knowledge (Paninski, 2003).

## 6 Discussion

### 6.1 Conclusion

Several approaches for estimating LLM uncertainty have emerged in recent years, but UQ performance can be constrained absent white-box LLM access.
Furthermore, the practicality of sampling-based UQ methods is limited by the computational costs associated with repeated text generation.
We illustrate the importance of semantic alphabet size in LLM uncertainty estimation with small sample sizes.
Adjusting discrete semantic entropy for sample coverage using our proposed semantic alphabet size estimator results in more accurate semantic entropy estimation and improved LLM incorrectness classification, compared to other discrete semantic entropy estimators.
Further, the aforementioned semantic alphabet size estimator achieves as good or better performance on incorrectness classification than all nine other black-box UQ methods considered in our QA experiments.

Though out of scope for the present study, we underscore that semantic alphabet size estimation - and $\widehat{|S|}_{Hybrid}$, in particular - may have broader application than merely entropy estimation. For instance, concurrent work by Li et al. (2025) applied a Good-Turing method to estimate the extent of LLMs’ unexpressed factual knowledge (e.g., mathematical theorems and diseases).

### 6.2 Limitations

We highlight several limitations in the present work. First, computational constraints limit our ability to run the experiments herein with models larger than 9 billion parameters.
That said, our analysis is extensive, spanning four distinct model families, four datasets, and ten uncertainty estimators.
Though our results are empirical, and experimental results may vary across studies, we make efforts to express our statistical uncertainty, and our final rank-generating procedure aims to account for it.

Second, our work aims to improve upon plugin discrete semantic entropy, whose estimator is negatively biased.
For this reason, we take semantic cluster labels as fixed, without ablating across alternative clustering strategies.
End-to-end semantic entropy calculation involves several steps, however, and a biased estimator does not necessitate that the entire process results in a negatively biased estimate.
For instance, a high false negative rate in the assignment of responses to semantic equivalence classes (e.g., due to minor lexical alterations inducing false negatives) may positively bias the final estimate (Grewal et al., 2024).

Finally, the uncertainty estimation methods considered herein do not directly measure factual inaccuracy. Instead, they may be better understood as indicators of semantic diversity. Of course, high semantic diversity may correspond to a high factual error rate, but there are scenarios wherein semantic diversity does not necessarily imply incorrectness (Ilia & Aziz, 2024), and, conversely, models can be confidently wrong.

### Acknowledgments

During this project, LLMs were employed at times for coding assistance.

## References

- Abdin et al. (2024)
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al.
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone.
*arXiv preprint arXiv:2404.14219*, 2024.
- Antos & Kontoyiannis (2001)
András Antos and Ioannis Kontoyiannis.
Convergence properties of functional estimates for discrete distributions.
*Random Structures & Algorithms*, 19(3-4):163–193, 2001.
- Basharin (1959)
Georgij P Basharin.
On a Statistical Estimate for the Entropy of a Sequence of Independent Random Variables.
*Theory of Probability & Its Applications*, 4(3):333–336, 1959.
- Bradley & Terry (1952)
Ralph Allan Bradley and Milton E Terry.
Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons.
*Biometrika*, 39(3/4):324–345, 1952.
- Caron & Doucet (2012)
Francois Caron and Arnaud Doucet.
Efficient bayesian inference for generalized bradley–terry models.
*Journal of Computational and Graphical Statistics*, 21(1):174–196, 2012.
- Chao (1987)
Anne Chao.
Estimating the Population Size for Capture-Recapture Data with Unequal Catchability.
*Biometrics*, pp. 783–791, 1987.
- Chao & Shen (2003)
Anne Chao and Tsung-Jen Shen.
Nonparametric estimation of Shannon’s index of diversity when there are unseen species in sample.
*Environmental and Ecological Statistics*, 10:429–443, 2003.
- Colvin et al. (2024)
Samuel Colvin, Eric Jolibois, Hasan Ramezani, Adrian Garcia Badaracco, Terrence Dorsey, David Montague, Serge Matveenko, Marcelo Trylesinski, Sydney Runkle, David Hewitt, and Alex Hall.
Pydantic, June 2024.
URL [https://github.com/pydantic/pydantic](https://github.com/pydantic/pydantic).
- Dagan et al. (2005)
Ido Dagan, Oren Glickman, and Bernardo Magnini.
The PASCAL Recognising Textual Entailment Challenge.
In *Machine Learning Challenges Workshop*, pp. 177–190. Springer, 2005.
- Danon et al. (2005)
Leon Danon, Albert Diaz-Guilera, Jordi Duch, and Alex Arenas.
Comparing community structure identification.
*Journal of Statistical Mechanics: Theory and Experiment*, 2005(09):P09008, 2005.
- DeLong et al. (1988)
Elizabeth R DeLong, David M DeLong, and Daniel L Clarke-Pearson.
Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach.
*Biometrics*, pp. 837–845, 1988.
- Farquhar et al. (2024)
Sebastian Farquhar, Joel Kossen, Lukas Kuhn, and Yarin Gal.
Detecting hallucinations in large language models using semantic entropy.
*Nature*, 630:625–630, 2024.
doi: 10.1038/s41586-024-07421-0.
URL [https://doi.org/10.1038/s41586-024-07421-0](https://doi.org/10.1038/s41586-024-07421-0).
- Fisher et al. (1943)
Ronald A Fisher, A Steven Corbet, and Carrington B Williams.
The Relation Between the Number of Species and the Number of Individuals in a Random Sample of an Animal Population.
*The Journal of Animal Ecology*, pp. 42–58, 1943.
- Fowlkes & Mallows (1983)
Edward B Fowlkes and Colin L Mallows.
A Method for Comparing Two Hierarchical Clusterings.
*Journal of the American Statistical Association*, 78(383):553–569, 1983.
- Gao et al. (2023)
Chao Gao, Yandi Shen, and Anderson Y Zhang.
Uncertainty quantification in the Bradley–Terry–Luce model.
*Information and Inference: A Journal of the IMA*, 12(2):1073–1140, 2023.
- Good (1953)
Irving J Good.
The Population Frequencies of Species and the Estimation of Population Parameters.
*Biometrika*, 40(3-4):237–264, 1953.
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The Llama 3 Herd of Models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Grewal et al. (2024)
Yashvir S Grewal, Edwin V Bonilla, and Thang D Bui.
Improving Uncertainty Quantification in Large Language Models via Semantic Embeddings.
*arXiv preprint arXiv:2410.22685*, 2024.
- Han et al. (2024a)
Jiatong Han, Jannik Kossen, Muhammed Razzak, and Yarin Gal.
Semantic Entropy Neurons: Encoding Semantic Uncertainty in the Latent Space of LLMs.
In *NeurIPS 2024 - Workshop on Foundation Model Interventions (MINT)*, 2024a.
- Han et al. (2024b)
Jiatong Han, Jannik Kossen, Muhammed Razzak, Lisa Schut, Shreshth A Malik, and Yarin Gal.
Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs.
In *ICML 2024 Workshop on Foundation Models in the Wild*, 2024b.
- Harris (1975)
Bernard Harris.
*The Statistical Estimation of Entropy in the Non-Parametric Case*.
University of Wisconsin-Madison, Mathematics Research Center, 1975.
- Hasan et al. (2025)
Md Mehedi Hasan, Moloud Abdar, Abbas Khosravi, Uwe Aickelin, Pietro Lio, Ibrahim Hossain, Ashikur Rahman, and Saeid Nahavandi.
Survey on Leveraging Uncertainty Estimation Toward Trustworthy Deep Neural Networks: The Case of Reject Option and Post-Training Processing.
*ACM Computing Surveys*, 57(9):1–35, 2025.
- He et al. (2021)
Pengcheng He, Jianfeng Gao, and Weizhu Chen.
DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing.
In *The Eleventh International Conference on Learning Representations*, 2021.
- Hunter (2004)
David R Hunter.
MM algorithms for generalized Bradley-Terry models.
*The Annals of Statistics*, 32(1):384–406, 2004.
- Ilia & Aziz (2024)
Evgenia Ilia and Wilker Aziz.
Variability Need Not Imply Error: The Case of Adequate but Semantically Distinct Responses.
*arXiv preprint arXiv:2412.15683*, 2024.
- Ji et al. (2023)
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung.
Survey of Hallucination in Natural Language Generation.
*ACM Computing Surveys*, 55(12):1–38, 2023.
- Kadavath et al. (2022)
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al.
Language models (mostly) know what they know.
*arXiv preprint arXiv:2207.05221*, 2022.
- Kingsley Zipf (1932)
George Kingsley Zipf.
*Selected studies of the principle of relative frequency in language*.
Harvard University Press, 1932.
- Kuhn et al. (2023)
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar.
Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation.
In *The Eleventh International Conference on Learning Representations*, 2023.
- Lancichinetti et al. (2009)
Andrea Lancichinetti, Santo Fortunato, and János Kertész.
Detecting the overlapping and hierarchical community structure in complex networks.
*New Journal of Physics*, 11(3):033015, 2009.
- Lanumteang & Böhning (2011)
Krisana Lanumteang and Dankmar Böhning.
An extension of Chao’s estimator of population size based on the first three capture frequency counts.
*Computational Statistics & Data Analysis*, 55(7):2302–2311, 2011.
- Lee et al. (2023)
Peter Lee, Sebastien Bubeck, and Joseph Petro.
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine.
*New England Journal of Medicine*, 388(13):1233–1239, 2023.
- Li et al. (2025)
Xiang Li, Jiayi Xin, Qi Long, and Weijie J Su.
Evaluating the Unseen Capabilities: How Many Theorems Do LLMs Know?
*arXiv preprint arXiv:2506.02058*, 2025.
- Lin (2004)
Chin-Yew Lin.
ROUGE: A package for automatic evaluation of summaries.
In *Text Summarization Branches Out*, pp. 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
URL [https://www.aclweb.org/anthology/W04-1013](https://www.aclweb.org/anthology/W04-1013).
- Lin & Och (2004)
Chin-Yew Lin and Franz Josef Och.
Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics.
In *Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)*, pp. 605–612, Barcelona, Spain, July 2004.
doi: 10.3115/1218955.1219032.
URL [https://aclanthology.org/P04-1077/](https://aclanthology.org/P04-1077/).
- Lin et al. (2024)
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun.
Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models.
*Transactions on Machine Learning Research*, 2024, 2024.
ISSN 2835-8856.
- Liu et al. (2024)
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang.
Lost in the Middle: How Language Models Use Long Contexts.
*Transactions of the Association for Computational Linguistics*, 12, 2024.
- Liu et al. (2025)
Xiaoou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, and Hua Wei.
Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey.
*arXiv preprint arXiv:2503.15850*, 2025.
- Malinin & Gales (2021)
Andrey Malinin and Mark Gales.
Uncertainty Estimation in Autoregressive Structured Prediction.
In *International Conference on Learning Representations*, 2021.
URL [https://openreview.net/forum?id=jN5y-zb5Q7m](https://openreview.net/forum?id=jN5y-zb5Q7m).
- Manakul et al. (2023)
Potsawee Manakul, Adian Liusie, and Mark Gales.
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp. 9004–9017, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.557.
URL [https://aclanthology.org/2023.emnlp-main.557/](https://aclanthology.org/2023.emnlp-main.557/).
- Mann & Whitney (1947)
Henry B Mann and Donald R Whitney.
On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other.
*The Annals of Mathematical Statistics*, pp. 50–60, 1947.
- Mielke et al. (2022)
Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau.
Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration.
*Transactions of the Association for Computational Linguistics*, 10:857–872, 2022.
doi: 10.1162/tacl˙a˙00494.
URL [https://aclanthology.org/2022.tacl-1.50/](https://aclanthology.org/2022.tacl-1.50/).
- Minello et al. (2019)
Giorgia Minello, Luca Rossi, and Andrea Torsello.
On the von Neumann entropy of graphs.
*Journal of Complex Networks*, 7(4):491–514, 2019.
- Montgomery-Smith & Schürmann (2014)
Stephen Montgomery-Smith and Thomas Schürmann.
Unbiased estimators for entropy and class number.
*arXiv preprint arXiv:1410.5002*, 2014.
- Murphy (2022)
Kevin P. Murphy.
Classification with the ”reject” option.
In *Probabilistic Machine Learning: An Introduction*, pp. 166. The MIT Press, 2022.
- Nguyen et al. (2025)
Dang Nguyen, Ali Payani, and Baharan Mirzasoleiman.
Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Findings of the Association for Computational Linguistics: ACL 2025*, pp. 4530–4540, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-256-5.
- Nikitin et al. (2024)
Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen.
Kernel language entropy: Fine-grained uncertainty quantification for LLMs from semantic similarities.
volume 37, pp. 8901–8929, 2024.
- Orgad et al. (2025)
Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov.
LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=KRnsX5Em3W](https://openreview.net/forum?id=KRnsX5Em3W).
- Paninski (2003)
Liam Paninski.
Estimation of Entropy and Mutual Information.
*Neural Computation*, 15(6):1191–1253, 2003.
- Petz (2001)
Dénes Petz.
Entropy, von Neumann and the von Neumann Entropy: Dedicated to the memory of Alfred Wehrl.
In *John von Neumann and the Foundations of Quantum Physics*, pp. 83–96. Springer, 2001.
- Pinchas et al. (2024)
Assaf Pinchas, Irad Ben-Gal, and Amichai Painsky.
A Comparative Analysis of Discrete Entropy Estimators for Large-Alphabet Problems.
*Entropy*, 26(5):369, 2024.
- Rajpurkar et al. (2016)
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.
SQuAD: 100,000+ Questions for Machine Comprehension of Text.
In Jian Su, Kevin Duh, and Xavier Carreras (eds.), *Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing*, pp. 2383–2392, Austin, Texas, November 2016. Association for Computational Linguistics.
doi: 10.18653/v1/D16-1264.
URL [https://aclanthology.org/D16-1264](https://aclanthology.org/D16-1264).
- Rajpurkar et al. (2018)
Pranav Rajpurkar, Robin Jia, and Percy Liang.
Know What You Don’t Know: Unanswerable Questions for SQuAD.
In Iryna Gurevych and Yusuke Miyao (eds.), *Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pp. 784–789, Melbourne, Australia, July 2018. Association for Computational Linguistics.
doi: 10.18653/v1/P18-2124.
URL [https://aclanthology.org/P18-2124](https://aclanthology.org/P18-2124).
- Shannon (1948)
Claude Elwood Shannon.
A Mathematical Theory of Communication.
*The Bell System Technical Journal*, 27(3):379–423, 1948.
- Simhi et al. (2024)
Adi Simhi, Jonathan Herzig, Idan Szpektor, and Yonatan Belinkov.
Distinguishing Ignorance from Error in LLM Hallucinations.
*arXiv preprint arXiv:2410.22071*, 2024.
- Sun & Xu (2014)
Xu Sun and Weichao Xu.
Fast Implementation of DeLong’s Algorithm for Comparing the Areas Under Correlated Receiver Operating Characteristic Curves.
*IEEE Signal Processing Letters*, 21(11):1389–1393, 2014.
- Team et al. (2024)
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al.
Gemma 2: Improving Open Language Models at a Practical Size.
*arXiv preprint arXiv:2408.00118*, 2024.
- Ulrich et al. (2010)
Werner Ulrich, Marcin Ollik, and Karl Inne Ugland.
A meta-analysis of species–abundance distributions.
*Oikos*, 119(7):1149–1155, 2010.
- Vu et al. (2007)
Vincent Q Vu, Bin Yu, and Robert E Kass.
Coverage-adjusted entropy estimation.
*Statistics in Medicine*, 26(21):4039–4060, 2007.
- Yang et al. (2018)
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning.
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering.
In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pp. 2369–2380, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
doi: 10.18653/v1/D18-1259.
- Zermelo (1929)
Ernst Zermelo.
Die berechnung der turnier-ergebnisse als ein maximumproblem der wahrscheinlichkeitsrechnung.
*Mathematische Zeitschrift*, 29(1):436–460, 1929.

## Appendix A Additional implementation details

### A.1 Text generation hyperparameters

With the exception of Figure [2](https://arxiv.org/html/2509.14478v1#S4.F2), which iterates over several sample sizes, the results shown in the main body of this work use a sample size of $n=10$, which is also found in prior work on semantic entropy (Kuhn et al., 2023; Farquhar et al., 2024; Nikitin et al., 2024; Nguyen et al., 2025). Due to its reduced computational burden compared to larger sample sizes and prevalence in the relevant literature, we will refer to $n=10$ as a “practical” or “typical” sample size.
Throughout this work, we generate a maximum of 100 tokens for each response, with early stopping enabled if the EOS token is generated.
All otherwise unspecified hyperparameters are set to their defaults (e.g., we do not invoke top-$k$ sampling, top-$p$ sampling, beam search, repetition penalty, etc.).

For incorrectness classification, we follow prior work that elicits concise responses by pre-pending QA queries with the following “pre-prompt” (Farquhar et al., 2024; Nikitin et al., 2024):

### A.2 Synthetic data generation

Figure: Figure 5: (A) Empirical survival function of the number of possible correct semantic categories and the number of observed semantic categories in the responses generated by GPT-4o-mini, according to a human annotator. (B) Scattergram of the number of observed semantic categories (“Observed“), according to the human annotator, against the number of possible correct semantic categories (“Possible“). One question is excluded (“Identify a programming language designed by Microsoft.”), because the total number of possible correct semantic categories was not known by the authors of this work.
Refer to caption: x5.png

Plethora Of accepTable cATegOries (POTATO) is a small synthetic dataset wherein every question is, in principle, answerable with a one-word or few-word response, and each question has more than one correct answer. We repeatedly invoke the following prompt against OpenAI’s GPT-4-turbo:

Above, [NUM_ANSWERS] is an integer between 1 and 50, and the available function is a Pydantic object enforcing structured generation (Colvin et al., 2024). A human annotator assessed the number of semantic categories for possible correct answers, discarded those with only one possible correct semantic category, and removed duplicate questions. The resulting dataset has $131$ unique questions. We generate $100$ responses to each question from the POTATO dataset using GPT-4o-mini. A human annotator manually assigned the responses to semantic categories.

One query (“Name a piece of classical music composed by Ludwig van Beethoven.”) has a particularly large number of possible correct semantic categories, owing to Beethoven’s prolific output of several hundred compositions.
Figure [5](https://arxiv.org/html/2509.14478v1#A1.F5)A suggests that the distribution of semantic alphabet sizes for model-query pairs is generally dominated by the distribution of the numbers of possible correct semantic categories for those queries. On a per-query basis, however, these quantities exhibit low correlation (Figure [5](https://arxiv.org/html/2509.14478v1#A1.F5)B).

### A.3 Bidirectional entailment clustering

#### Semantic classification.

**Table 2: Evaluating semantic classification methods against human-annotated ground truth. Results are averaged across questions from the POTATO dataset, either excluding or including questions for which all responses were semantically equivalent ($|S|>1$ and $|S|\geq 1$, respectively), according to the human rater. Metrics considered are the Fowlkes–Mallows index (FMI), normalized mutual information (NMI), and pairwise agreement (PA), assessed between the semantic equivalence classes assigned by the provided method and the human-annotated ground truth. Values are rounded to two decimal places. The best-performing BEC method by each metric (unrounded) is in bold. Instances where a Semantic Embedding method outperforms a BEC method are underlined. Responses are generated by GPT-4o-mini.**
|  | $|S|>1$ | $|S|\geq 1$ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Classifier | FMI | NMI | PA | FMI | NMI | PA |
| LLM | 0.84 | 0.72 | 0.84 | 0.86 | 0.63 | 0.83 |
| NLI-NQ | 0.84 | 0.64 | 0.85 | 0.85 | 0.63 | 0.83 |

The semantic clustering step of semantic entropy calculation is performed using a Bidirectional Entailment Clustering (BEC), which classifies pairs of passages $(d_{i},d_{j})$ bidirectionally as “entailment,” “neutral,” or “contradiction,” and uses these labels to greedily assign passages to semantic equivalence classes. Like previous work, our implementation focuses on strict entailment, where both unidirectional relations must be considered “entailment” or equivalent (i.e., not “neutral”). We refer the reader to Kuhn et al. (2023); Farquhar et al. (2024) for further detail on the BEC algorithm. The classifier used varies by implementation, either an LLM- or NLI-based method:

#### Large Language Model (LLM).

Farquhar et al. (2024) classify passage pairs (T1, T2) by invoking OpenAI’s GPT-3.5 endpoint with the following prompt:

Our implementation invokes OpenAI’s GPT-4o-mini model, due to the sunsetting of GPT-3.5.

#### Natural Language Inference (NLI).

The BEC approach of Kuhn et al. (2023) classifies $(d_{i},d_{j})$ using a DeBERTa model fine-tuned for NLI tasks.(^2^22[https://huggingface.co/microsoft/deberta-large-mnli](https://huggingface.co/microsoft/deberta-large-mnli)) Our implementation, used throughout this work, updates the classifier to a fine-tuned NLI model based on Microsoft’s newer DeBERTaV3 (He et al., 2021).(^3^33[https://huggingface.co/cross-encoder/nli-deberta-v3-base](https://huggingface.co/cross-encoder/nli-deberta-v3-base)) The NLI classification can be performed with or without the source questions (NLI-Q and NLI-NQ, respectively); in the former, the query is prepended to both $d_{i}$ and $d_{j}$ before passing through the NLI model.

#### Comparison.

To compare the alignment between each clustering method and the human-annotated ground-truth, we measure the Fowlkes–Mallows index (FMI) (Fowlkes & Mallows, 1983) and normalized mutual information (NMI) (Danon et al., 2005; Lancichinetti et al., 2009). We also consider pairwise agreement (PA), wherein we iterate over pairs of passages, classify each pair as “entailment” or “contradiction,” based on the clustering results, and report for each method the portion of pairs whose entailment label agrees with that of the human annotator (Farquhar et al., 2024). Across metrics, we observe that the NLI method without the inclusion of questions performs roughly similarly to LLM (Table [2](https://arxiv.org/html/2509.14478v1#A1.T2)). For its comparable performance while limiting costs, we use NLI-NQ throughout this work.

### A.4 LLM-as-judge

Prior work labeled an LLM’s response as correct if the ROUGE-L score (Lin, 2004; Lin & Och, 2004) between the response and a reference answer was above $0.3$ (Kuhn et al., 2023). Appropriate thresholds may vary by model and dataset - for instance, the reference answers in HotPotQA appear rather brief, and models can vary in their verbosity, without it necessarily impacting the correctness of their responses. Additionally, such methods may not capture semantically equivalent, but lexically distinct, paraphrasings (e.g., using an acronym).

Instead, Lin et al. (2024) prompt a commercial LLM to provide a numerical rating of consistency between LLM responses and reference answers, with ratings above $70$ indicating an accurate response. They observe, however, that a small portion of LLM judgements do not have an easily-parseable rating. For this reason, we modify their prompt to request ratings within XML-style tags and include an in-context example:

When a query has multiple reference answers (e.g., those in the BioASQ), we invoke the above prompt to compare the LLM response to each reference answer, then report the maximum LLM-as-judge rating. Our choice of commercial model for LLM-as-judge is OpenAI’s GPT-4o-mini.

### A.5 Uncertainty estimators

#### Predictive entropy.

The so-called “total uncertainty” associated with a prediction is the information entropy of the corresponding predictive posterior distribution
(Malinin & Gales, 2021). In UQ for natural language generation, an analogue is predictive entropy (PE), the entropy of the distribution of generable sequences by an LLM in response to a prompt (Kuhn et al., 2023; Lin et al., 2024; Farquhar et al., 2024). Since not all generable sequences are typically known, PE is calculated using the above plugin method to estimate the entropy of a prompt-model pair’s “answer distribution” (Kadavath et al., 2022). A shortcoming of this approach is that it treats all lexically distinct sequences as unique elements of an alphabet, even if they are semantically similar.

**Table 3: Employing distinct values of regularization parameter $a$, including zero regularization, we compute MLE Bradley-Terry latent strength scores and establish $95\%$ CIs of each method’s latent strength rank via the methods of Gao et al. (2023). Results are identical for all $a$ values considered.**
|  | $a$ |  |  |  |
| --- | --- | --- | --- | --- |
| Method | 0 | $0.01$ | $0.1$ | $1$ |
| $\widehat{|S|}_{Hybrid}$ | [1, 2] | [1, 2] | [1, 2] | [1, 2] |
| NumSets | [7, 10] | [7, 10] | [7, 10] | [7, 10] |
| Good-Turing | [7, 10] | [7, 10] | [7, 10] | [7, 10] |
| $U_{EigV}$ | [3, 4] | [3, 4] | [3, 4] | [3, 4] |
| $\widehat{\mathbb{H}}_{Plugin}$ | [7, 10] | [7, 10] | [7, 10] | [7, 10] |
| $\widehat{\mathbb{H}}_{CS}$ | [7, 10] | [7, 10] | [7, 10] | [7, 10] |
| $\widehat{\mathbb{H}}_{Hybrid}$ | [4, 5] | [4, 5] | [4, 5] | [4, 5] |
| PE | [3, 5] | [3, 5] | [3, 5] | [3, 5] |
| SNNE | [6, 6] | [6, 6] | [6, 6] | [6, 6] |
| KLE | [1, 2] | [1, 2] | [1, 2] | [1, 2] |

#### SNNE.

SNNE aims to account for both “intra-and inter-cluster similarity.” Provided a scale factor $\tau$ and a similarity function $f$ for assessing passage pairs $(d_{i},d_{j})$, SNNE is defined as:

$$ $SNNE(q)=-\frac{1}{n}\sum_{i=1}^{n}\log\sum_{j=1}^{n}\exp\Bigg(\frac{f(d_{i},d_{j})}{\tau}\Bigg)$ (11) $$

(Nguyen et al., 2025).
Our implementation uses ROUGE-L for $f$ and scale factor $\tau=1$, which the authors report as best-performing (Nguyen et al., 2025).

#### KLE.

Like $U_{EigV}$, KLE constructs a graph over LLM responses to a query, where nodes are responses and edge weights are prescribed by the results of an NLI model. Instead of using the model’s categorical scores, however, KLE’s semantic graph has weights $w_{i,j}=g(d_{i},d_{j})+g(d_{j},d_{i})$, where $g$ is $1$ if the NLI model predicts “entailment” for the provided response pair, $0.5$ if it predicts “neutral,” and 0 otherwise. The standard graph Laplacian (i.e., the difference between the degree matrix and the weight matrix) is then taken over the resultant semantic graph (Nikitin et al., 2024).

KLE admits a choice of kernel to apply to the Laplacian, resulting in a “density” matrix $K$. Given the eigenvalues ($\lambda_{1}<\cdots<\lambda_{n}$) of $K$, the von Neumann entropy is calculated:

$$ $VNE(K)=\sum_{i=1}^{n}\lambda_{i}\log\lambda_{i}.$ (12) $$

Although the concept of von Neumann entropy has its roots in quantum physics (Petz, 2001), it has also been used more recently in network science literature to quantify graph complexity (Minello et al., 2019).

Nikitin et al. (2024) report the heat kernel as best-performing overall:

$$ $K_{heat}=e^{-tL}.$ (13) $$

Our implementation of KLE invokes $K_{heat}$ with hyperparameter $=0.3$, which the authors consider a “reasonable default” that outperforms prior approaches without additional hyperparameter optimization (Nikitin et al., 2024).

### A.6 Bradley-Terry confidence intervals

Gao et al. (2023) offers a procedure for ascertaining CIs about ranks of Bradley-Terry strength estimates, which we cursorily summarize as follows:
Assuming a Bradley-Terry model, let $\beta=\{\beta_{1},\dots,\beta_{m}\}$ be the true latent strengths of $m$ methods, and let $\widehat{\beta}=\{\widehat{\beta}_{1},\dots,\widehat{\beta}_{m}\}$ be MLEs of the same.
Given a provided level $\alpha$ and a “target” method $i$, we may establish a $(1-\alpha)$ CI about $\widehat{\beta}_{i}$, as well as “slightly more conservative” intervals about the remaining strength estimates.
Let $n_{1}$ be the number of resultant intervals whose lower bounds are greater than the upper bound of method $i$’s. Similarly, let $n_{2}$ be the number of intervals whose upper bounds are less than the lower bound of method $i$’s. The true rank of method $i$’s Bradley-Terry strength is in the integer interval $[n_{1}+1,n-n_{2}]$ with approximate probability $1-\alpha$.
Because Proposition 4.1 of Gao et al. (2023) only holds for fixed number of methods/agents/players, we illustrate the “conservative” CIs in Figure [4](https://arxiv.org/html/2509.14478v1#S4.F4)C.
We repeat this procedure for each UQ method to establish CIs for all strength ranks.
We refer interested readers to Gao et al. (2023) for further details.

The algorithm we use to estimate strength scores employs a regularization parameter, which we take as $a=0.1$ in the main body of the paper.
Table [3](https://arxiv.org/html/2509.14478v1#A1.T3), which repeats the analysis of Figure [4](https://arxiv.org/html/2509.14478v1#S4.F4)C for varying $a$, indicates that the rank order of our Bradley-Terry results is not sensitive to this choice.

## Appendix B The under-sampled regime

Consider a simple model of semantic category-abundance, where the probability that a model parameterized by $\theta$ generates a response belonging to the $r^{th}$-most prevalent semantic category $s_{r}$ in response to a query $q$ follows a Zipfian distribution, similar to some models of species-abundance (Ulrich et al., 2010):

$$ $\displaystyle p(s_{r}|q,\theta)$ $\displaystyle=\frac{1}{rH_{|S|}},$ (14) $$

where $H_{j}$ is the $j^{th}$ harmonic number (Kingsley Zipf, 1932). If the expected number of occurrences of $s_{r}$ in a sample of $n$ responses is less than $1$, then $|S|>\frac{n}{H_{|S|}}$. For sample size $n=10$, we expect at least one semantic category to be unobserved with just $|S|>4$.

## Appendix C Additional underestimation results

Detailed results for SE underestimation, broken out across four datasets, are shown in Figure [6](https://arxiv.org/html/2509.14478v1#A3.F6). Results are qualitatively similar to those in Figure [2](https://arxiv.org/html/2509.14478v1#S4.F2), with $\hat{\mathbb{H}}_{Hybrid}$ consistently underestimating $SE^{*}$ less than plugin DSE for practical sample sizes.

Figure: Figure 6: Ratio of semantic entropy estimator with varying sample size ($n=5,10,25,50,75,100$) to white-box semantic entropy with $n=100$ (denoted $SE^{*}$), with values below $1$ suggesting underestimation (dotted grey line). Estimators shown are the plugin estimator of canonical discrete semantic entropy (dotted rose, indigo, green, and cyan lines) and the presented semantic entropy estimator of Equation [10](https://arxiv.org/html/2509.14478v1#S3.E10) (solid rose, indigo, green, and cyan lines). Results are averaged across queries for each dataset. Log scale is used on the x-axis to highlight differences between estimators with smaller sample sizes. Instances with only one observed semantic category at $n=100$, resulting in a denominator of 0, are ignored.
Refer to caption: x6.png

## Appendix D Additional incorrectness detection results

Detailed results for incorrectness classification across three datasets, four LLMs, and ten UQ methods are shown in Figure [7](https://arxiv.org/html/2509.14478v1#A4.F7). Error bars are $95\%$ CIs about empirical AUROC values, calculated via DeLong’s method (DeLong et al., 1988; Sun & Xu, 2014).

Figure: Figure 7: Performance of incorrectness classification on single-answer QA for uncertainty measures using $n=10$ samples. “Plugin,” “CS-GT,” and “CS-H” are estimators of semantic entropy. “NumSets,” “$U_{EigV}$,” and “$\widehat{|S|}_{Hybrid}$” are estimators of semantic alphabet size. Error bars reflect $95\%$ CIs about the AUROC.
Refer to caption: x7.png

## Appendix E Consistency in evaluation

The relevant LLM UQ literature is not always consistent in experimental setup. For instance, the plurality of considered works use $n=10$ sampled responses per query (Kuhn et al., 2023; Farquhar et al., 2024; Nikitin et al., 2024; Nguyen et al., 2025), but Lin et al. (2024) use $n=20$. Some works vary prompting templates across datasets (e.g., zero-shot vs. multi-shot prompting) (Kuhn et al., 2023; Lin et al., 2024). A variety of methods for automated LLM incorrectness evaluation have been employed, including thresholded ROUGE-L (Kuhn et al., 2023), binary LLM-as-judge (Farquhar et al., 2024), thresholded consistency-based LLM-as-judge Lin et al. (2024), and BERTScore (Nguyen et al., 2025). Although it is out of this work’s scope to exhaustively ablate all such choices, we attempt to be explicit in describing our selections to support future work.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Q1. Would it possible to create a Figure-2-like plot but for semantic-alphabet-size estimators?

Q2. Nitpick: could you please change the font in Fig 1 to a Serif font?
soundness: 3
strengths: S1. The contribution is principled and to my best judgement technically correct.

S2. The empirical performance of the proposed method is strong.

S3. I find the empirical evaluation methodology sound.
summary: Authors propose a better (less biased) estimator for semantic-alphabet-size in the semantic-entropy-related methods for UQ estimation in NLG tasks. 
They show empirically that their semantic-alphabet-size is competitive with some of the SOTA methods.
weaknesses: W1. The evaluation could use more datasets (despite being sound methodologically).   
There are codebases available which would make the process rather straightforward, given the method proposed is rather simple to implement once the Semantic-Clustering is computed: https://github.com/AlexanderVNikitin/kernel-language-entropy. + maybe SimpleQA?(https://openai.com/index/introducing-simpleqa/)

W2. Not sure how "significant" the contribution will prove to be, but I hate to judge this criterium, so please treat this complaint as secondary. If this was judged by TMLR criteria, my overall score would be an "accept".

---

confidence: 3
contribution: 3
initial_rating: 2
presentation: 3
questions: - How quickly does the “white-box” SE estimate converge to the "true" SE as the number of samples increases? Including this in Figure 2 would be interesting.
- Are 100 samples sufficient to approximate the true distribution over semantic clusters? An ablation with larger sample sizes (e.g., 1 k) could strengthen the claim that this is a valid reference.
- How sensitive are the results to the semantic clustering method (e.g., choice of NLI model, thresholding, or entailment aggregation)?
- The hybrid estimator uses a max(·) rule combining two estimators. Why is this choice theoretically justified versus, say, a weighted or Bayesian combination?
soundness: 3
strengths: - The proposed semantic alphabet size estimator effectively mitigates the underestimation bias of discrete semantic entropy (DSE) in the few-sample regime.
- The method is simple and interpretable, avoiding the complexity and hyperparameter dependence of recent SE extensions.
- Empirically, it improves UQ performance and matches or exceeds the performance of SOTA approaches in the back-box setting.
summary: This paper addresses the problem of underestimating semantic entropy (SE) as an uncertainty estimate for black-box LLMs. The authors propose an interpretable semantic alphabet size estimator which, when used to adjust the discrete SE estimator for sample coverage, yields more accurate uncertainty estimation.
weaknesses: - My main concern is that the proposed method is largely an adaptation of existing estimators to the SE setting. While the idea is insightful, it constitutes an incremental improvement over Farquhar et al. (2024) rather than a fundamentally new theoretical contribution.
- The paper treats semantic clusters as fixed and does not analyze sensitivity to the clustering procedure (e.g., entailment thresholds or NLI model biases). Since clustering quality directly affects entropy estimates, this omission is a potential confounder that limits the scope of the contribution.
- Experiments are restricted to relatively small models (<10B parameters), which raises questions about generalization to larger, frontier LLMs. Moreover, the paper does not compare against computationally cheaper uncertainty measures that avoid expensive sampling (e.g., G-NLL [1]) and which outperform SE and DSE with a single sample.
Minor point: The paper inconsistently uses the abbreviations (like SE and DSE), alternating between the abbreviated and full forms without clear rationale, which makes it harder to follow.

---

[1] Lukas Aichberger, Kajetan Schweighofer, and Sepp Hochreiter. Rethinking uncertainty estimation in natural language generation. arXiv preprint arXiv:2412.15176, 2024.

[2] Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 2024.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Equation (8) appears to be missing the definition or notation for $p_i$. 
2. I have multiple questions regarding the experiment results (Figure 3): 
- Why was white-box semantic entropy not included in the comparison, since you also consider the PE? 
- In addition, both hybrid semantic entropy ($\hat{H}_\text{hybrid}$) and discrete semantic entropy (DSE) show notably lower rankings (around 7–10). Could you elaborate on why these methods perform substantially worse than others? Conceptually, both $\hat{H}_\text{hybrid}$ and DSE should still be affected by the same underestimation bias that motivates your paper. If the semantic alphabet size were estimated more accurately, shouldn’t this also mitigate the underestimation problem in DSE to some extent?
soundness: 3
strengths: 1. The problem itself is important.
2. The solution is simple.
3. The evaluation considers the weakness of AUROC and proposes the Bradley-Terry latent strength scores, which make the evaluation more reliable. 
4. The paper is easy to understand and well-structured. I like its simplicity and enjoy reading it.
summary: Goal:
- This paper addresses a specific issue in discrete semantic entropy (DSE), a widely used black-box UQ method.  
- The authors show that DSE tends to underestimate uncertainty when the response sample size is small, because it often fails to capture all unseen semantic clusters, therefore underestimating the number of distinct meaning clusters in its outputs (**semantic alphabet size**).

Empirical Validation:
- To verify this bias, the authors compare DSE values from small-sample settings against 100-sample semantic entropy (treated as ground-truth uncertainty).
- Across all model–dataset pairs, DSE consistently produces lower uncertainty estimates as sample size decreases.

Method:
To better estimate the semantic alphabet size, the paper considers 2 existing approaches:
1. A Good–Turing coverage estimator, adapted from population ecology, which corrects for unseen categories in finite samples
2. A spectral method based on eigenvalue decomposition (Lin et al.), which estimates alphabet size from the number of non-zero eigenvalues of a semantic similarity matrix.
The authors then propose a hybrid estimator that combines these two strategies: 
- when Good–Turing coverage becomes unreliable (e.g., each cluster appears only once), the method defaults to the spectral estimate; 
- otherwise, it uses a weighted combination to mitigate underestimation bias.
Using this hybrid alphabet size, they further define a hybrid semantic entropy estimator, extending classic ecological entropy formulations to the semantic domain.

Evaluation:
1. Following standard UQ evaluation protocols, the study benchmarks ten uncertainty estimators across four LLM families and four datasets.
2. Because raw AUROC scores can vary with model and dataset, the authors adopt a Bradley–Terry latent strength model to aggregate pairwise AUROC comparisons into an overall ranking with confidence intervals—an extension of the win-rate framework proposed by Nikitin et al. (2024).
Overall, results show that the hybrid semantic alphabet size estimator performs most consistently across settings, while the hybrid semantic entropy estimator reduces but does not fully eliminate the negative bias in DSE.
weaknesses: 1. Limited methodological novelty and analysis.
- The proposed hybrid estimator mainly combines two existing techniques 
   -  the Good–Turing coverage estimator from population ecology
   - a spectral eigenvalue-based method
I am not saying that combining prior ideas is not good, and my point is that it can be done with a deeper analysis of each individual methods and the hybrid operation can be done more adaptively. 
- For example, the Good–Turing approach is well established for ecological sampling, yet the sampling distribution of LLM-generated responses can be very different. It remains unclear why this method should perform reliably in the LLM setting or under what conditions it might fail. 
- Similarly, the hybrid design could be more principled — for instance, by characterizing when each component estimator is more accurate, rather than simply prioritizing the larger estimate under certain cases. This max operation inherently assumes that both the spectral-based method and Good-Turing coverage-based methods are also underestimating the ground truth semantic cluster size. But is this assumption true?

2. Inconsistent motivation and results.
The empirical validation (Figure 2 and table 1) assumes that semantic entropy computed with 100 samples represents the ground truth. However, this assumption itself is not rigorously justified. Moreover, the paper’s motivation is to improve DSE-based uncertainty estimation, yet the final results show that DSE-derived methods (including the hybrid entropy estimator) still lag behind KLE and the semantic alphabet size estimator in overall performance (always ranking from 7-10). This weakens the narrative that improving DSE necessarily leads to better uncertainty quantification.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: * It’s interesting that you find that the adjusted number of sets outperforms the adjusted semantic entropy. It would be nice to see how the “ground truth” number of sets compares with “ground truth” semantic entropy... if in this setting ground truth number of sets outperforms ground truth semantic entropy, it might suggest something interesting about miscalibration of the LLM. 
* I’d like to see (maybe in the discussion) discussion of how one might extend this to white-box SE estimation (which will tend to suffer from the same problem, albeit to a lesser extent).
soundness: 4
strengths: * It’s a simple idea... in a good way. It identifies a clear problem, finds parallels in the species sampling literature for heavy tailed distributions, and proposes a simple, interpretable solution.
* It leads to improvement. 
* The evaluation is very thoughtful (one of the best discussions of how to evaluate UQ methods I have seen)
* The paper is clear and well-written
summary: I really like this paper. It takes a fairly simple premise: sampling-based estimators of the clustering distribution will tend to miss the tail, leading to systemic underestimation of semantic entropy. But: We have robust estimators for the size of the tail. So, we can get modified estimators of either the number of semantic clusters, or the semantic entropy. And, these modified estimators lead to improved performance.
weaknesses: * It’s not strictly improving on SoTA... but it is compatible with SoTA using a much simpler and more interpretable paradigm
* The idea of using a Good-Turing estimator has been explored in related problems in the literature, as acknowledged by the authors.


Decision: Accept (Poster)