Improving Uncertainty Estimation through Semantically Diverse Language Generation

## Abstract

Abstract Large language models (LLMs) can suffer from hallucinations when generating text. These hallucinations impede various applications in society and industry by making LLMs untrustworthy. Current LLMs generate text in an autoregressive fashion by predicting and appending text tokens. When an LLM is uncertain about the semantic meaning of the next tokens to generate, it is likely to start hallucinating.
Thus, it has been suggested that hallucinations stem from predictive uncertainty.
We introduce Semantically Diverse Language Generation ( SDLG ) to quantify predictive uncertainty in LLMs. SDLG steers the LLM to generate semantically diverse yet likely alternatives for an initially generated text. This approach provides a precise measure of aleatoric semantic uncertainty, detecting whether the initial text is likely to be hallucinated. Experiments on question-answering tasks demonstrate that SDLG consistently outperforms existing methods while being the most computationally efficient, setting a new standard for uncertainty estimation in LLMs.

## 1 Introduction

Hallucinations hinder a broad use of LLMs in practical applications and critical decision-making processes as they make them untrustworthy (Manakul et al., ).
Hallucinations are fragments of generated text that, despite appearing cohesive, are not factual.
We consider generated text to be hallucinated if it stems from contradictory or non-existent facts in the training data or input prompt.
Thus, hallucinations are conjectured to be mainly caused by the predictive uncertainty inherent to probabilistic models (Xiao and Wang, ).
While uncertainty estimation for classification tasks has been developed extensively (Hüllermeier and Waegeman, ; Gawlikowski et al., ), it remains a challenging problem for autoregressive tasks, particularly in natural language generation (NLG).

Uncertainty estimation in NLG involves assessing the uncertainty of an initially generated text (output sequence) for a given prompt (input sequence).
Current methods typically assess this uncertainty by generating multiple output sequences, usually with a single given language model (Xiao and Wang, ; Malinin and Gales, ; Kuhn et al., ; Lin et al., ; Duan et al., ; Manakul et al., ).
Importantly, Kuhn et al. () propose to consider semantic clusters
rather than individual output sequences by grouping them according to their semantic equivalence.
The semantic uncertainty of the initial output sequence should only increase if a language model is likely to generate alternative output sequences that differ in semantics.
Hence, semantic uncertainty involves estimating the probability that a language model generates an output sequence belonging to a specific semantic cluster.
Empirical results confirm that incorporating semantics into uncertainty estimation in language models achieves state-of-the-art performance (Kuhn et al., ).
The current approach utilizes Monte Carlo (MC) approximation via multinomial sampling to generate alternative output sequences that are classified into semantic clusters by a natural language inference (NLI) model (Kuhn et al., ).
Sample sizes typically range from the low double-digit range (Malinin and Gales, ; Kuhn et al., ; Duan et al., ) to only a few hundred for studies conducted on large-scale compute (Kadavath et al., ), as autoregressive generations are computationally expensive.
This is suboptimal, as the current sampling methods are imprecise with a small number of samples (Bishop, ) and computationally too expensive to ensure high precision with a large number of samples.

Figure: Figure 1: Standard multinomial sampling relies on chance to obtain semantically different output sequences, thus is prone to miss them. SDLG specifically searches for likely, but semantically different output sequences, thereby improving the estimation of semantic uncertainty in language models.
Refer to caption: x1.png

To improve upon the limitations of the current state-of-the-art approach, we introduce Semantically Diverse Language Generation (SDLG), a method that efficiently estimates semantic uncertainty by utilizing importance sampling to generate output sequences.
We introduce a proposal distribution that samples semantically diverse output sequences (see Fig. 1).
SDLG utilizes the NLI model not only for transforming the space of generated output sequences to semantic clusters but also for computing the contribution of every token to the final semantics.
Subsequently, the semantically most relevant tokens are substituted to explicitly steer the generation toward semantically diverse yet likely alternative output sequences, while correcting for the increased sampling probability using importance sampling. This can be viewed as stress testing the language model, unveiling output sequences that are valuable summands for estimators of semantic uncertainty.
They are generated by the current sampling methods only by chance, making SDLG a more systematic and reliable method for capturing semantic uncertainty.

Our main contributions are:

- •
We propose a novel method for generating semantically diverse yet likely output sequences. Empirical results demonstrate that our method outperforms existing methods for uncertainty estimation in NLG, specifically across a variety of free-form question-answering tasks.
- •
We establish a theoretical foundation for uncertainty measures in NLG and introduce theoretically grounded estimators for aleatoric semantic uncertainty, also known as semantic entropy. Applying these estimators enhances the empirical performance of uncertainty estimation in language models.

## 2 Measuring Predictive Uncertainty in NLG

Uncertainty estimation in classification tasks has already been well-established (Gal and Ghahramani, ). However, these measures cannot be directly applied to uncertainty estimation in NLG.
Two key aspects, which differ from estimating uncertainty in classification tasks, have to be considered.
First, a sequence of autoregressive predictions collectively forms the final output of a model.
Second, unlike classification tasks where classes usually are mutually exclusive, different output sequences may be equivalent in their semantic meaning.
To account for this aspects, Kuhn et al. () introduce semantic entropy, yet without proper theoretical grounding.
In the following, we derive semantic entropy starting from the well-studied uncertainty estimation in classification tasks.

#### Predictive uncertainty in classification.

We briefly revisit uncertainty estimation for classification tasks.
In this work, we quantify the predictive uncertainty of a single given “off-the-shelf” model.
Given are a classification model parametrized by $\bm{w}$ and an input vector $\bm{x}$.
The predictive distribution under the given model is denoted as $p(y\mid\bm{x},\bm{w})$.
We assume a fixed dataset $\mathcal{D}$ that was sampled according to
the predictive distribution $p(y\mid\bm{x},\bm{w}^{*})$ under true model parameters $\bm{w}^{*}$.
Thus, we assume that the model class can approximate the true distribution sufficiently well,
a common and usually necessary assumption (Hüllermeier and Waegeman, ).
The posterior distribution $p(\bm{w}\mid\mathcal{D})$ denotes how likely $\bm{w}$ matches $\bm{w}^{*}$.
Following Schweighofer et al. (, ), the predictive uncertainty
of a single given model parametrized by $\bm{w}$ is given by

$$ $\displaystyle\underbrace{\mathbf{\mathrm{E}}_{\tilde{\bm{w}}}\big{[}\mathrm{CE }(p(y\mid\bm{x},\bm{w});p(y\mid\bm{x},\tilde{\bm{w}}))\big{]}}_{\text{total}}= \underbrace{\mathbf{\mathrm{H}}(p(y\mid\bm{x},\bm{w}))}_{\text{aleatoric}}+ \underbrace{\mathbf{\mathrm{E}}_{\tilde{\bm{w}}}\big{[}\mathrm{KL}(p(y\mid\bm{ x},\bm{w})\,\|\,p(y\mid\bm{x},\tilde{\bm{w}}))\big{]}}_{\text{epistemic}}$ (1) $$

where $\mathbf{\mathrm{E}}_{\tilde{\bm{w}}}=\mathbf{\mathrm{E}}_{\tilde{\bm{w}}\sim p
(\tilde{\bm{w}}\mid\mathcal{D})}$.
The total uncertainty, given by the posterior expectation of
the cross-entropy $\mathrm{CE}(\cdot;\cdot)$,
is additively decomposed into aleatoric and epistemic uncertainty.
The aleatoric uncertainty is the Shannon entropy $\mathrm{H}(\cdot)$ of the predictive distribution under the given model.
The epistemic uncertainty is the posterior expectation of the Kullback-Leibler divergence
$\mathrm{KL}(\cdot\,\|\,\cdot)$ between the given model and possible true models according to their posterior probability.

#### Predictive uncertainty in NLG.

Given are an autoregressive language model parametrized by $\bm{w}$, a vocabulary $\mathcal{V}$, and an input sequence of tokens $\bm{x}=(x_{1},...,x_{M})$ with $x\in\mathcal{V}$.
An output of the language model is a sequence of tokens $\bm{y}=(y_{1},...,y_{T})\in\mathcal{Y}$ with $y\in\mathcal{V}$.
The predictive distribution at step $t$ of the output sequence $\bm{y}$ is conditioned on both the input sequence and all previously generated tokens, denoted as $p(y_{t}\mid\bm{x},\bm{y}_{<t},\bm{w})$.
The probability of an output sequence is the product of the individual token probabilities: $p(\bm{y}\mid\bm{x},\bm{w})=\prod_{t=1}^{T}p(y_{t}\mid\bm{x},\bm{y}_{<t},\bm{w})$ (Sutskever et al., ).
In practice, $p(\bm{y}\mid\bm{x},\bm{w})$ is often length-normalized to not favor short output sequences (Cover and Thomas, ; Malinin and Gales, ), which results in $\bar{p}(\bm{y}\mid\bm{x},\bm{w})=\exp\left(\frac{1}{T}\sum_{t=1}^{T}\log p(y_{
t}\mid\bm{x},\bm{y}_{<t},\bm{w})\right)$.

Evaluating the whole set of possible output sequences $\mathcal{Y}$ is usually intractable, as it scales exponentially with the sequence length $T$, thus $\mathcal{O}(\lvert\mathcal{V}\lvert^{T})$.
Furthermore, as mentioned previously, a language model that likely generates different output sequences from the same input sequence should not necessarily indicate high predictive uncertainty if the output sequences mean the same thing.
Hence, predictive uncertainty should be considered high only when different output sequences also exhibit semantically diverse meanings (Kuhn et al., ).
Instead of directly utilizing the distribution over output sequences $p(\bm{y}\mid\bm{x},\bm{w})$, the distribution over semantic clusters

$$ $\displaystyle p(c\mid\bm{x},\bm{w})\ =\ \sum_{\mathcal{Y}}p(c\mid\bm{y},\bm{x} ,\bm{w})\ p(\bm{y}\mid\bm{x},\bm{w})\ =\ \sum_{\mathcal{Y}}\mathbbm{1}\!\{\bm{ y}\in c\mid\bm{x},\bm{w}\}\ p(\bm{y}\mid\bm{x},\bm{w})$ (2) $$

is used to derive the predictive uncertainty in the NLG setting.
Here, $p(\bm{y}\mid\bm{x},\bm{w})$ expresses the probability of generating an output sequence $\bm{y}$ and $p(c\mid\bm{y},\bm{x},\bm{w})$ expresses the probability of $\bm{y}$ belonging to a certain semantic cluster $c\in\mathcal{C}$.
Although a specific output sequence might potentially be attributed to more than one semantic cluster, we follow Kuhn et al. () in assuming that each output sequence is attributed to a single semantic cluster.
They demonstrate that assigning semantic clusters by predicting the semantic equivalence between generated output sequences with an NLI model empirically performs well.
The NLI model takes two sequences as input and predicts whether they entail or contradict each other.
Two output sequences are semantic equivalent if they entail each other in both orders and thus belong to the same semantic cluster by definition.
Consequently, $p(c\mid\bm{y},\bm{x},\bm{w})$ is equivalent to $\mathbbm{1}\!\{\bm{y}\in c\mid\bm{x},\bm{w}\}$, where $\mathbbm{1}\!\{\bm{y}\in c\mid\bm{x},\bm{w}\}=1$ iff $\bm{y}$ belongs to semantic cluster $c$.
For the set of all possible output sequences $\mathcal{Y}$, $p(c\mid\bm{x},\bm{w})$ expresses the probability of the language model generating an output sequence belonging to a specific semantic cluster for a given input sequence.
Adopting the definition of predictive uncertainty in Eq. (1), the total predictive semantic uncertainty

$$ $\displaystyle\underbrace{\mathbf{\mathrm{E}}_{\tilde{\bm{w}}}\big{[}\mathrm{CE }(p(c\mid\bm{x},\bm{w});p(c\mid\bm{x},\tilde{\bm{w}}))\big{]}}_{\text{total}}= \underbrace{\mathbf{\mathrm{H}}(p(c\mid\bm{x},\bm{w}))}_{\text{aleatoric}}+ \underbrace{\mathbf{\mathrm{E}}_{\tilde{\bm{w}}}\big{[}\mathrm{KL}(p(c\mid\bm{ x},\bm{w})\,\|\,p(c\mid\bm{x},\tilde{\bm{w}}))\big{]}}_{\text{epistemic}}$ (3) $$

can again be additively decomposed into aleatoric and epistemic semantic uncertainty.
The epistemic semantic uncertainty is again a posterior expectation,
which is particularly challenging to estimate for current language models
with billions of parameters (Zhang et al., ; Touvron et al., ).
The aleatoric semantic uncertainty turns out to be precisely the semantic entropy proposed by Kuhn et al. ().
Semantic entropy is the entropy of the semantic cluster probability distribution as of Eq. (2),

$$ $\displaystyle\mathbf{\mathrm{H}}(p(c\mid\bm{x},\bm{w}))\ =\ -\sum_{\mathcal{C} }\log p(c\mid\bm{x},\bm{w})\ p(c\mid\bm{x},\bm{w})$ (4) $$

under a single given language model. To determine whether a language model is uncertain about its output, it is essential to accurately estimate its semantic entropy, as addressed in the following section.

## 3 Estimating Aleatoric Semantic Uncertainty in NLG

We now discuss practical considerations regarding the estimation of the aleatoric semantic uncertainty, namely the semantic entropy as given in Eq. (4).
First, we find that directly estimating it by obtaining samples from semantic clusters is theoretically not justified in the current setting.
Instead, the underlying distribution of semantic clusters itself has to be estimated through sampling.
Second, to effectively estimate this distribution of semantic clusters, we utilize importance sampling.
These two insights contribute to a more accurate uncertainty estimation in NLG.

#### Monte Carlo (MC) estimation.

Kuhn et al. () propose to approximate the semantic entropy by directly using the MC estimator

$$ $\displaystyle\mathbf{\mathrm{H}}(p(c\mid\bm{x},\bm{w}))\ \approx\ -\frac{1}{N} \sum_{n=1}^{N}\log p(c^{n}\mid\bm{x},\bm{w})\ ,\quad c^{n}\sim p(c\mid\bm{x}, \bm{w})\ .$ (5) $$

However, we cannot directly sample from $p(c\mid\bm{x},\bm{w})$, but only from $p(\bm{y}\mid\bm{x},\bm{w})$.
Therefore, it is impossible to directly use the estimator in Eq. (5).
Alternatively, one can first approximate the semantic cluster probability distribution

$$ $\displaystyle p(c\mid\bm{x},\bm{w})\$ $\displaystyle\approx\ \frac{1}{N}\ \sum_{n=1}^{N}\mathbbm{1}\!\{\bm{y}\in c \mid\bm{x},\bm{w}\}\ ,\quad\bm{y}^{n}\sim p(\bm{y}\mid\bm{x},\bm{w})\ .$ (6) $$

Output sequences $\bm{y}^{n}$ are simply generated via multinomial sampling (Kuhn et al., ).
This estimate of the semantic cluster probability distribution can directly be used to approximate the semantic entropy.
Since usually not all clusters $c\in\mathcal{C}$ are found through sampling, the sum is taken over all observed clusters $c_{1},...,c_{M}$ to which $\{\bm{y}^{n}\}_{n=1}^{N}$ were assigned by the NLI model, resulting in

$$ $\displaystyle\mathbf{\mathrm{H}}(p(c\mid\bm{x},\bm{w}))\ \approx\ -\sum_{m=1}^ {M}\log p(c_{m}\mid\bm{x},\bm{w})\ p(c_{m}\mid\bm{x},\bm{w})\ .$ (7) $$

In Sec. 6 we empirically show that using this proper estimator for the semantic entropy outperforms the improper estimator implemented by Kuhn et al. ().
For further details see Sec. C in the appendix.

#### Importance sampling.

Due to the computational cost of autoregressively sampling from multi-billion-parameter language models, the sample size $N$ is usually kept very low in practice.
However, this implies that the variance of the MC estimator in Eq. (6) remains high.
To lower the variance, a standard technique is importance sampling according to a proposal distribution $q$ instead of the target distribution $p$ (Bishop, ).
Eq. (6) thus changes to

$$ $\displaystyle p(c\mid\bm{x},\bm{w})\$ $\displaystyle\approx\ \frac{1}{N}\ \sum_{n=1}^{N}\mathbbm{1}\!\{\bm{y}\in c \mid\bm{x},\bm{w}\}\ \frac{p(\bm{y}^{n}\mid\bm{x},\bm{w})}{q(\bm{y}^{n}\mid\bm {x},\bm{w})}\ ,\quad\bm{y}^{n}\sim q(\bm{y}\mid\bm{x},\bm{w})\ .$ (8) $$

The quality of the approximation strongly depends on the choice of the proposal distribution.
It should closely approximate the target distribution and have good overlap, i.e. the proposal distribution should have probability mass everywhere the target distribution has substantial probability mass (Bishop, ).
Thus, a good proposal distribution should cover semantic clusters with substantial probability mass under the target distribution.
In the following section, we describe our method to construct an empirical proposal distribution $q(\bm{y}\mid\bm{x},\bm{w})$ with high mass at important semantic clusters by explicitly promoting both the likelihood and diversity of output sequences.

## 4 Semantically Diverse Language Generation

Estimating the semantic entropy according to Eq. (7)
requires approximating the semantic cluster probability distribution.
This probability distribution can either be approximated according to Eq. (6),
or via importance sampling according to Eq. (8).
MC estimation using output sequences from multinomial sampling is straightforward, as one can directly sample from the original distribution $p(\bm{y}\mid\bm{x},\bm{w})$ without the need for additional weighting or adjustment.
However, the resulting estimator has high variance and sampling only considers the likelihood and not the semantic diversity.
Furthermore, multinomial sampling may miss likely output sequences that capture important information about semantic cluster probabilities,
especially if those sequences require choosing a lower-likelihood token.
This limits the accurate estimation of the semantic entropy, as sampling semantically diverse output sequences essentially occurs by chance.
Importance sampling, on the other hand, can overcome these limitations by incorporating semantic diversity into the sampling procedure.
However, this requires a beneficial proposal distribution $q(\bm{y}\mid\bm{x},\bm{w})$.
We propose Semantically Diverse Language Generation (SDLG) to sample according to such an empirical proposal distribution.
It seeks to efficiently explore semantic clusters, capturing important modes of $p(c\mid\bm{x},\bm{w})$ that might be missed by multinomial sampling.

#### Semantic diversity and where to find it.

Given an input sequence $\bm{x}$ and an initial output sequence $\bm{y}^{\prime}$ generated by a language model, how can we generate another output sequence that has a different semantics from $\bm{y}^{\prime}$?
In natural language, individual words contribute to the semantics of the sentence to varying extents.
For instance, consider the sentence “Albert Einstein proposed the theory of relativity”.
The word “proposed” can be substituted with “introduced” or “postulated” without altering the semantics.
However, replacing “Einstein” with “Schweitzer” would change the semantics.
Therefore, our method focuses on identifying and substituting the tokens that are most critical to the semantics of $\bm{y}^{\prime}$ (see Fig. 1).
This is achieved by introducing three scores at the token level that quantify each token’s relevance in altering the semantics:

- 1.
Attribution score: Each initial token $y_{i}\in\bm{y}^{\prime}$ is scored by its contribution to the semantic meaning of $\bm{y}^{\prime}$.
We refer to this initial token’s score as $A_{i}$.
- 2.
Substitution score: Each alternative token $v_{j}\in\mathcal{V}$ is scored by its influence in altering the semantic meaning when substituting $y_{i}$ with $v_{j}$. We refer to this alternative token’s score as $S_{ij}$.
- 3.
Importance score: Each alternative token $v_{j}\in\mathcal{V}$ is scored by the probability the language model assigns to $v_{j}$ given the context up to $y_{i}$. We refer to this alternative token’s score as $I_{ij}$.

At a high level, SDLG explicitly substitutes tokens based on these scores.
High scores indicate a high potential to give rise to a likely output sequence with semantics different from $\bm{y}^{\prime}$, as detailed below.
Before that, we describe how to calculate the three scores.

The computation of the attribution and substitution scores requires a loss $\mathrm{L}$, which expresses to what degree $\bm{y}^{\prime}$ is semantically different from itself. It is computed utilizing an NLI model that predicts whether two sequences entail or contradict each other. First, the initial output sequence is input into the NLI model twice, resulting in a high probability of entailment.
Second, the loss $\mathrm{L}$ is computed to the target prediction contradiction. This loss is used to compute the gradient $\nabla_{\bm{z}_{i}}\mathrm{L}$ with respect to the token embedding $\bm{z}_{i}$ that represents the initial token $y_{i}$. This gradient vector quantifies the required change in $\bm{z}_{i}$ to achieve a high probability of contradiction, thus altering the semantic meaning of $\bm{y}^{\prime}$.

#### Attribution score.

Our first objective is to identify which initial token $y_{i}$ should be changed according to the computed gradient vector.
An initial token’s attribution score

$$ $\displaystyle A_{i}\$ $\displaystyle=\ \|\bm{z}_{i}\odot\nabla_{\bm{z}_{i}}\mathrm{L}\|_{2}$ (9) $$

is defined as the Euclidean distance $\|\cdot\|_{2}$ of the gradient vector multiplied elementwise with the embedding vector $\bm{z}_{i}$ that represents the initial token $y_{i}$.
The higher the attribution score $A_{i}$, the higher the impact of the token $y_{i}$ on altering the semantics when being changed (Adebayo et al., ).
We note that different attribution methods could be utilized and future work may benefit from exploring these methods to compute $A_{i}$ (Madsen et al., ).

#### Substitution score.

Identifying which initial token should be changed is crucial but not sufficient on its own.
We also have to identify which token to change to, as not every substitution alters the semantic meaning of the initial output sequence.
Thus, our second objective is to identify appropriate alternative tokens $v_{j}$ that most effectively alter the semantics when substituting an initial token $y_{i}$.
The alternative token’s substitution score

$$ $\displaystyle S_{ij}\$ $\displaystyle=\ \frac{(\bm{z}_{i}-\bm{z}_{j})\cdot\nabla_{\bm{z}_{i}}\mathrm{L }}{\|\bm{z}_{i}-\bm{z}_{j}\|_{2}\ \ \|\nabla_{\bm{z}_{i}}\mathrm{L}\|_{2}}$ (10) $$

is defined as the cosine similarity $sim(\cdot,\cdot)$ between the gradient vector and the difference between the initial token’s embedding vector $\bm{z}_{i}$ and the alternative token’s embedding vector $\bm{z}_{j}$.
Although cosine similarity is a widely used measure for text similarity, we observed that the dot product empirically yields comparable performance.
The higher the substitution score $S_{ij}$, the more closely the change in the embedding vector aligns with the direction of the gradient vector, thus the direction of altering the semantics (Mikolov et al., ).

#### Importance score.

Identifying which alternative token should substitute which initial token is important but not sufficient.
Not every alternative token $v_{j}$ is an appropriate substitution for the initial token $y_{i}$ given the context.
The context is the input sequence $\bm{x}$ and the output sequence up to the token that is to be substituted $\bm{y}_{<i}$.
Substituting with a very unlikely token might undermine the coherence of the overall sequence.
Thus, our third objective is to favor alternative tokens that exhibit a high likelihood.
The alternative token’s importance score

$$ $\displaystyle I_{ij}\$ $\displaystyle=\ p(v_{j}\mid\bm{y}_{<i},\bm{x},\bm{w})$ (11) $$

is simply defined as the probability that the language model assigns to $v_{j}$ given the context.
The higher the importance score $I_{ij}$, the more suitable the alternative token is for substitution, as it ensures that the resulting output sequence not only is semantically diverse but also remains likely.

Figure: Algorithm 1 SDLG

#### Generating semantic diverse output sequences.

All token pairs ($y_{i}$, $v_{j}$) are ranked according to the three scores ($A_{i}$, $S_{ij}$, $I_{ij}$).
In its simplest form, the ranking is based on equally weighting the three scores, which we found to work well empirically.
Based on this token score ranking, a new output sequence is generated by deliberately substituting the highest-ranked token pair.
Subsequent tokens are discarded as they are conditioned on the substituted token, which affects their likelihood.
The remainder of the new output sequence is then generated by the language model using the usual sampling strategy (see Alg. 1 and 2).
SDLG preserves the semantically less relevant part of the output sequence, eliminating the need to regenerate it and focusing on the part with a high chance of altering the semantic meaning. As a result, the generation process becomes computationally more efficient than sampling from scratch each time and potentially sampling multiple duplicate output sequences.

#### Proposal distribution.

Finally, we can discuss the exact form of the proposal distribution $q(\bm{y}\mid\bm{x},\bm{w})$ in Eq. (8), that is induced by SDLG.
We define the proposal distribution as

$$ $\displaystyle q(\bm{y}\mid\bm{x},\bm{w})\ =\ \sum_{\bm{y}^{\prime}\in\mathcal{ Y}}p(\bm{y}\mid\bm{y}^{\prime},\bm{x},\bm{w})\ p(\bm{y}^{\prime}\mid\bm{x},\bm {w})\ .$ (12) $$

The probability distribution $p(\bm{y}\mid\bm{y}^{\prime},\bm{x},\bm{w})$ denotes SDLG transforming a given output sequence $\bm{y}^{\prime}$ into another output sequence $\bm{y}$ with a high chance of changing the semantics as well.
However, since $\bm{y}^{\prime}$ is not known a priori, we sum over all possible $\bm{y}^{\prime}$ according to their probability of being sampled $p(\bm{y}^{\prime}\mid\bm{x},\bm{w})$.
Under assumptions about which index $t$ is likely to be chosen for a given $\bm{y}^{\prime}$, the proposal distribution takes the form

$$ $\displaystyle q(\bm{y}\mid\bm{x},\bm{w})\ =\ \frac{p(\bm{y}\mid\bm{x},\bm{w})} {p(y_{t}\mid\bm{y}_{<t},\bm{x},\bm{w})}$ (13) $$

where $p(y_{t}\mid\bm{y}_{<t},\bm{x},\bm{w})$ is the likelihood of the token that is exchanged by SDLG.
Intuitively, it means that we have to adjust the MC estimate in Eq. (6), because SDLG interferes in sampling and changes the token $y_{t}$ deterministically.
For more details on the assumptions and a step-by-step derivation see Sec. B in the appendix.
Our experiments show, that sampling according to this proposal distribution improves uncertainty estimation in NLG.

## 5 Related Work

#### Uncertainty estimation in NLG.

Several works utilized the language model itself to obtain a prediction of their uncertainty, whether that be numerical or verbal (Mielke et al., ; Lin et al., ; Kadavath et al., ; Cohen et al., ; Ganguli et al., ; Ren et al., ; Tian et al., ).
Cohen et al. () utilize cross-examination, where one language model generates the output sequence and the other language model acts as an examiner to assess the uncertainty.
Zhou et al. () investigate the behavior of language models when expressing their (un)certainty.

A large body of work focuses on sampling a set of output sequences to obtain sampling-based uncertainty estimators.
Xiao and Wang (); Malinin and Gales (); Hou et al. () incorporate both aleatoric and epistemic estimates of uncertainty, where epistemic uncertainty due to model selection is considered.
While Kuhn et al. (); Lin et al. (); Duan et al. () evaluate the aleatoric uncertainty only under a single given language model, they take the semantic equivalence of potential output sequences into account.
Manakul et al. () also sample a set of output sequences but utilize them as input to another language model to assess the uncertainty.
Another approach to uncertainty estimation in NLG is conformal prediction (Quach et al., ), where a stopping rule for generating output sequences is calibrated.
Additionally, Xiao et al. () empirically analyze how factors such as model architecture and training details influence the uncertainty estimates in language models.

#### Generating diverse output sequences.

Li et al. () propose an alternative training procedure of language models to avoid generic, input-independent output sequences and increase diversity.
Diverse beam search (Vijayakumar et al., ) optimizes for a diversity-augmented objective across beam groups, based on diversity heuristics.
Ippolito et al. () compare diversity encouraging decoding strategies.
Nucleus sampling (Holtzman et al., ) generates higher quality as well as more diverse output sequences but does not explicitly encourage semantic diversity.
Contrastive decoding (Li et al., ) utilizes a second, weaker language model, where the decoding algorithm favors tokens generated by the stronger model and penalizes tokens generated by the weaker model.
Tam () utilize semantic clustering during beam search, which is used to prune beams and diversify the remaining candidates.
However, this only indirectly steers towards more diversity and relies on the diversity of the initial beams.

Closely related, but not directly targeting semantic diversity of output sequences is the field of (neural) controllable text generation (Prabhumoye et al., ).
Here, the generation process of the language model is steered by another language model to e.g. adhere to a certain dialog structure, prevent toxic answers, or play a certain persona.
Keskar et al. () use control codes added to the prompt to steer the generation.
Dathathri et al. () propose the use of an external supervised classifier to control the generation.
Chan et al. () also utilize an external classifier but trains in a self-supervised setting.
(Ghazvininejad et al., ; Holtzman et al., ) re-weight the probability distributions at each step of generating the output sequence.
For further work in this field see the surveys by Prabhumoye et al. (); Zhang et al. ().

## 6 Experiments

#### Data and models.

We performed experiments on three free-form question-answering datasets, covering a broad range of question-answering settings. To be concrete, we use the over 800 closed-book questions in TruthfulQA (Lin et al., ) corresponding to whole sentence answers, the almost 8,000 open-book questions in the development split of CoQA (Reddy et al., ) corresponding to medium to shorter length answers, and about 8,000 closed-book questions in the training split of TriviaQA (Joshi et al., ) corresponding to short, precise answers. We use a 5-shot, zero-shot, and 10-shot prompt for TruthfulQA, CoQA, and TriviaQA respectively.
These datasets are frequently used as benchmarks for uncertainty estimation in NLG due to their strong correlation with human evaluations and the effective performance of “off-the-shelf” language models compared to tasks requiring fine-tuning (Kuhn et al., ; Goyal et al., ).
Each of the three datasets was evaluated with the OPT model family (Zhang et al., ), with model sizes ranging from 2.7 to 30 billion parameters.
Related work suggests that performance trends generalize across transformer-based model families (Duan et al., ; Manakul et al., ).
In general, the four language models and three datasets assess the performance of uncertainty estimation methods in NLG across varying model sizes, output sequence lengths, and both open-book and closed-book settings.

#### Evaluation.

Following Kuhn et al. (); Lin et al. (); Duan et al. (), the quality of an uncertainty estimator is evaluated by how well it correlates with the respective correctness of an answer of the language model; correct answers should be assigned a lower uncertainty estimator than incorrect answers.
Thus, the correctness of the initial output sequence $\bm{y}^{\prime}$ has to be evaluated.
We follow the evaluation protocol of current work and utilize the statistics-based metrics Rouge-L and Rouge-1 (Lin, ) as well as the transfer learning-based metric BLEURT (Sellam et al., ), each with ten different correctness thresholds ranging from $0.1$ to $1.0$ (exact match).
In general, the correctness of $\bm{y}^{\prime}$ is computed as [max score to a true reference answer] - [max score to a false reference answer].
While TurthfulQA provides both true and false reference answers, CoQA and TriviaQA only provide true reference answers.
The initial output sequence $\bm{y}^{\prime}$ is sampled using a beam search with five beams.
It also serves as the first output sequence utilized by every uncertainty estimation method.
The Area Under Receiver Operating Characteristic (AUROC)
is used as a metric for classifying the correct vs. incorrect answers, using the respective uncertainty estimate as a score.
The higher the AUROC, the higher the correlation between the uncertainty estimate and the correctness of the answers.

#### Baselines.

We compare our method against two streams of current predictive uncertainty estimation in NLG. First, methods that directly utilize the predictive entropy of the output distribution on a token level. These are Predictive Entropy (PE), Length-Normalized Predictive Entropy (LN-PE) (Malinin and Gales, ), and Shifting Attention to Relevance (SAR) (Duan et al., ). Second, methods that utilize semantic entropy on a sequence level. Thereby, output sequences are generated with multinomial sampling (SEMS) (Kuhn et al., ) or with diverse beam search (SEDBS) (Vijayakumar et al., ). Although diverse beam search has not explicitly been proposed for uncertainty estimation in NLG, we added it as a more traditional sampling method that enforces diversity among output sequences. For each dataset, we performed a hyperparameter search on SEMS temperature $\in\{0.25,0.5,1.0,1.5,2.0\}$ and SEDBS penalty term $\in\{0.2,0.5,1.0\}$.

#### Our method ( SDLG ).

Unlike current methods that rely on finding the optimal sampling temperature or penalty term, SDLG does not require hyperparameter tuning as it controls the sampling. One has only to decide on the ranking method for the three individual token scores. We empirically found that the performance of SDLG is quite robust with respect to the weighting of the token scores. Therefore, throughout the experiments, we derive the final token score ranking by straightforwardly averaging the three individual token scores.
To compute the token scores for semantic diversity as discussed in Sec. 4, we utilize the same NLI model DeBERTa (Williams et al., ; He et al., ) that is also used for predicting semantic equivalences and determining semantic clusters.
We also note that substituting initial tokens not corresponding to the beginning of a word is often impractical. Consequently, we exclusively apply substitutions to tokens at the beginning of a word.

**Table 1: AUROC using different uncertainty measures as a score to distinguish between correct and incorrect answers. SEMS uses the improper semantic entropy estimator implemented by Kuhn et al. () while SE${}_{\ \cdots}^{(*)}$ use the proper semantic entropy estimator as we introduced in Sec. 3, with different sampling strategies. The threshold of the correctness metric Rouge-L (F1 score) is set to $0.5$. Each method uses ten output sequences for assigning an uncertainty estimate.**
| Dataset | Model | LN-PE | PE | SAR | SEMS | SE${}_{MS}^{(*)}$ | SE${}_{DBS}^{(*)}$ | SE${}_{\texttt{SDLG}{}}^{(*)}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TruthfulQA | OPT-2.7b | .439 | .517 | .611 | .405 | .846 | .686 | .920 |
| OPT-6.7b | .446 | .510 | .555 | .512 | .781 | .637 | .881 |  |
| OPT-13b | .676 | .712 | .775 | .453 | .896 | .819 | .956 |  |
| OPT-30b | .482 | .542 | .517 | .438 | .864 | .788 | .927 |  |
| CoQA | OPT-2.7b | .717 | .693 | .733 | .717 | .744 | .697 | .744 |
| OPT-6.7b | .728 | .703 | .748 | .739 | .764 | .714 | .759 |  |
| OPT-13b | .723 | .697 | .747 | .743 | .758 | .720 | .760 |  |
| OPT-30b | .732 | .698 | .742 | .745 | .767 | .713 | .768 |  |
| TriviaQA | OPT-2.7b | .769 | .787 | .785 | .781 | .804 | .808 | .809 |
| OPT-6.7b | .790 | .805 | .804 | .803 | .822 | .823 | .829 |  |
| OPT-13b | .807 | .820 | .819 | .824 | .838 | .841 | .845 |  |
| OPT-30b | .799 | .812 | .815 | .817 | .831 | .837 | .840 |  |

#### Analysis of results.

Our method largely outperforms all current methods when utilizing each correctness metric Rouge-L (F1 score), Rouge-1 (F1 score), and BLEURT. The performance improvements of our method persist across different correctness thresholds as well as the number of samples considered.
It can be observed that simple token-level diversity enforced by higher temperatures in multinomial sampling (Kuhn et al., ) or by diverse beam search (Vijayakumar et al., ) is insufficient for capturing semantic diversity essential for uncertainty estimation in NLG.
The results summarized in Tab. 1 also show that using the proper semantic entropy estimator for the semantic entropy (SE${}_{MS}^{(*)}$) outperforms the improper estimator (SEMS) when generating output sequences via multinomial sampling, as outlined in Sec. 3.

#### Semantic clusters.

Our method results in a $19\%$ increase of semantic clusters after generating the second output sequence, as well as a $74\%$ increase of semantic clusters after the tenth output sequence, compared to multinomial sampling with the highest-performing temperature used by the current state-of-the-art method (Kuhn et al., ), when averaging across all CoQA instances. This is because SDLG explicitly searches for output sequences with different semantic meanings and practically does not sample the same output sequence twice.

#### Computational expenses.

Our experiments show that SDLG is sample efficient.
Using a single output sequence generated with SDLG usually already performs better than using up to ten output sequences generated with other methods.
While it is true that generating additional output sequences with SDLG requires extra computation for the NLI model to obtain the token score ranking, the primary computational effort lies in the generation of output sequences. The computational expenses of a single forward and backward pass through the NLI model with a few hundred million parameters is minor compared to generating only a single token using a language model with billions of parameters. Since our method deterministically changes a specific token within the initial output sequence, preceding tokens do not have to be regenerated again, but only subsequent ones.
This results in SDLG requiring at least an average of $15\%$ (TruthfulQA), $33\%$ (CoQA), and $21\%$ (TriviaQA) fewer flops compared to multinomial sampling that is used by the current state-of-the-art method (Kuhn et al., ).
The method SAR (Duan et al., ) even adds computational cost to multinomial sampling by utilizing the NLI model to evaluate the relevance of each token in the initial output sequence through a “leave-one-out” approach. In general, it can be observed that the advantage of our method over the current methods further increases with longer output sequences and larger language model sizes.

Experimental details and insights into the two ablation studies on semantic clusters and computational expenses can be found in Sec. D in the appendix.

## 7 Conclusion

We introduce SDLG to improve uncertainty estimation in NLG.
SDLG substitutes tokens in the initial output sequence that are likely to lead to a change in semantic meaning.
Unlike standard multinomial sampling, this targeted approach effectively samples likely output sequences with different semantic meanings, capturing important information for estimating semantic uncertainty.
Our experiments on free-form question-answering datasets demonstrate that SDLG not only increases the overall quality of the uncertainty estimator but is also computationally more efficient.

Future work should investigate the performance of SDLG on NLG tasks with longer output sequences, such as summarization.
However, evaluating the correctness of longer output sequences is a challenging problem in itself.
Furthermore, the assumption that each sentence can be attributed to a single semantic cluster might be overly strict.
Using continuous sentence similarities
could allow for a relaxed semantic cluster assignment, which would only require minimal changes to our method.
Moreover, the semantic difference is currently only conditioned on the first output sequence. We expect to see further improvements when taking into account the semantics of all previously generated output sequences as well.
Lastly, this work focuses on estimating the aleatoric semantic uncertainty.
Future work should investigate how to effectively assess the epistemic semantic uncertainty (see Eq. (3)).

Despite these remaining challenges, SDLG already offers notable advantages over prior methods.
First, SDLG controls the sampling instead of depending on chance to obtain diverse samples. It eliminates the necessity of searching for an optimal sampling temperature, a hyperparameter important for the performance of all prior methods.
Second, the advantageous sampling reduces the required number of samples for high-quality uncertainty estimation, while also being computationally efficient.
Overall, SDLG has the potential to considerably enhance the applicability of uncertainty estimation in NLG.

## Acknowledgements

The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the projects Medical Cognitive Computing Center (MC3), INCONTROL-RL (FFG-881064), PRIMAL (FFG-873979), S3AI (FFG-872172), DL for GranularFlow (FFG-871302), EPILEPSIA (FFG-892171), AIRI FG 9-N (FWF-36284, FWF-36235), AI4GreenHeatingGrids (FFG- 899943), INTEGRATE (FFG-892418), ELISE (H2020-ICT-2019-3 ID: 951847), Stars4Waters (HORIZON-CL6-2021-CLIMATE-01-01). We thank Audi.JKU Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FILL Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ. Waterloo), Software Competence Center Hagenberg GmbH, Borealis AG, TÜV Austria, Frauscher Sensonic, TRUMPF and the NVIDIA Corporation.
Kajetan Schweighofer acknowledges travel support from ELISE (GA no 951847).

## Appendix A Broader Impact

This work focuses on assessing the uncertainty in natural language generation (NLG) using language models.
Our primary goal is to increase the robustness of language models, assess the reliability of their predicted output sequences, and detect when a language model is hallucinating.
Therefore, we contend that our work makes a positive contribution to society in several aspects:

- 1.
Improved discernment of certainty in model predictions enhances practical application in real-world scenarios.
This can be implemented by signaling uncertainty to users, such as through highlighting dubious sections of responses or opting not to display uncertain outputs altogether.
- 2.
Reliable uncertainty estimates may increase the trust of the user in the language model, as it provides a basis to gauge the quality of the answer.

However, while we expect mainly a positive impact on society, there are also potential negative aspects:

- 1.
Enhanced uncertainty estimation might not yield expected outcomes if users lack the necessary training to interpret these estimates effectively.
- 2.
While better uncertainty assessment can foster usability and user trust, it also carries the risk of creating undue reliance on these models.
It is crucial to maintain human oversight and critical evaluation of language model outputs, as over-reliance can be detrimental.

It is important to note that our method evaluates uncertainty based on the information available to the language model.
Therefore, it may inaccurately deem a factually incorrect answer as certain if the model’s knowledge base contains similar errors.
This issue, often perceived as model “hallucination”, is not a reflection of the model’s uncertainty, but rather a result of factual inaccuracies in the underlying data that is additional to hallucinations due to uncertainty.

## Appendix B On the proposal distribution induced by SDLG

In the following, we analyze the proposal distribution induced by SDLG.
We consider a probabilistic transformation of one output sequence $\bm{y}^{\prime}$ into another output sequence $\bm{y}$, given by $p(\bm{y}\mid\bm{y}^{\prime},\bm{x},\bm{w})$.
This is introduced, because we have to sum over all possible output sequences $\bm{y}^{\prime}$ that we could apply SDLG on, leading to

$$ $\displaystyle q(\bm{y}\mid\bm{x},\bm{w})\ =\ \sum_{\bm{y}^{\prime}\in\mathcal{ Y}}p(\bm{y}^{\prime}\mid\bm{x},\bm{w})\ p(\bm{y}\mid\bm{y}^{\prime},\bm{x},\bm {w})\ .$ (14) $$

We can write $p(\bm{y}\mid\bm{y}^{\prime},\bm{x},\bm{w})$ as an expected value over $t$, the index where SDLG chooses a different token:

$$ $\displaystyle p(\bm{y}\mid\bm{y}^{\prime},\bm{x},\bm{w})\$ $\displaystyle=\ \sum_{t=1}^{T}\ p(t\mid\bm{y}^{\prime},\bm{x},\bm{w})\ p(\bm{y }\mid t,\bm{y}^{\prime},\bm{x},\bm{w})\ .$ (15) $$

The construction of $\bm{y}$ from $\bm{y}^{\prime}$ only changes one element of $\bm{y}^{\prime}$ at
position $t$ and then generates the postfix new. Therefore, we have
$p(\bm{y}\mid t,\bm{y}^{\prime},\bm{x},\bm{w})=0$
for $\bm{y}^{\prime}_{<t}\not=\bm{y}_{<t}$.
Consequently, $\sum_{\bm{y}^{\prime}\in\mathcal{Y}}p(\bm{y}^{\prime}\mid\bm{x},\bm{w})$
can be reduced to $\sum_{\bm{y}^{\prime}_{>t}\in\mathcal{Y}_{>t}}p(\bm{y}^{\prime}_{>t}\mid\bm{y}
_{\leqslant t},\bm{x},\bm{w})\ p(\bm{y}_{\leqslant t}\mid\bm{x},\bm{w})$ if
the factor $p(\bm{y}\mid t,\bm{y}^{\prime},\bm{x},\bm{w})$ is present.
There is only one possibility for the prefix with $\bm{y}^{\prime}_{<t}=\bm{y}_{<t}$.

Using Eq. (15) in Eq. (14) leads to

$$ $\displaystyle q$ $\displaystyle(\bm{y}\mid\bm{x},\bm{w})$ (16) $\displaystyle=\ \sum_{\bm{y}^{\prime}\in\mathcal{Y}}p(\bm{y}^{\prime}\mid\bm{x },\bm{w})\ \sum_{t=1}^{T}p(t\mid\bm{y}^{\prime},\bm{x},\bm{w})\ p(\bm{y}\mid t ,\bm{y}^{\prime},\bm{x},\bm{w})$ $\displaystyle=\ \sum_{t=1}^{T}\ \sum_{\bm{y}^{\prime}\in\mathcal{Y}}p(\bm{y}^{ \prime}\mid\bm{x},\bm{w})\ p(t\mid\bm{y}^{\prime},\bm{x},\bm{w})\ p(\bm{y}\mid t ,\bm{y}^{\prime},\bm{x},\bm{w})$ $\displaystyle=\ \sum_{t=1}^{T}\ \sum_{\bm{y}^{\prime}_{>t}\in\mathcal{Y}_{>t}} \ \sum_{\bm{y}^{\prime}_{\leqslant t}\in\mathcal{Y}_{\leqslant t}}\ p(\bm{y}^{ \prime}_{>t}\mid\bm{y}^{\prime}_{\leqslant t},\bm{x},\bm{w})\ p(\bm{y}^{\prime }_{\leqslant t}\mid\bm{x},\bm{w})\ p(t\mid\bm{y}^{\prime},\bm{x},\bm{w})\ p( \bm{y}\mid t,\bm{y}^{\prime},\bm{x},\bm{w})$ $\displaystyle=\ \sum_{t=1}^{T}\ \sum_{\bm{y}^{\prime}_{>t}\in\mathcal{Y}_{>t}} p(\bm{y}^{\prime}_{>t}\mid\bm{y}_{\leqslant t},\bm{x},\bm{w})\ p(\bm{y}_{ \leqslant t}\mid\bm{x},\bm{w})\ p(t\mid\bm{y}^{\prime}_{>t},\bm{y}_{\leqslant t },\bm{x},\bm{w})\ p(\bm{y}_{>t}\mid\bm{y}^{\prime}_{>t},\bm{y}_{\leqslant t}, \bm{x},\bm{w})$ $\displaystyle=\ \sum_{t=1}^{T}\ \sum_{\bm{y}^{\prime}_{>t}\in\mathcal{Y}_{>t}} p(\bm{y}^{\prime}_{>t}\mid\bm{y}_{\leqslant t},\bm{x},\bm{w})\ p(\bm{y}_{ \leqslant t}\mid\bm{x},\bm{w})\ p(t\mid\bm{y}^{\prime}_{>t},\bm{y}_{\leqslant t },\bm{x},\bm{w})\ p(\bm{y}_{>t}\mid\bm{y}_{\leqslant t},\bm{x},\bm{w})$ $\displaystyle=\ \sum_{t=1}^{T}\ p(\bm{y}_{\leqslant t}\mid\bm{x},\bm{w})\ \left(\sum_{\bm{y}^{\prime}_{>t}\in\mathcal{Y}_{>t}}p(\bm{y}^{\prime}_{>t}\mid \bm{y}_{\leqslant t},\bm{x},\bm{w})\ p(t\mid\bm{y}^{\prime}_{>t},\bm{y}_{ \leqslant t},\bm{x},\bm{w})\right)\ p(\bm{y}_{>t}\mid\bm{y}_{\leqslant t},\bm{ x},\bm{w})$ $\displaystyle=\ \sum_{t=1}^{T}\ p(\bm{y}_{\leqslant t}\mid\bm{x},\bm{w})\ p(t \mid\bm{y}_{\leqslant t},\bm{x},\bm{w})\ p(\bm{y}_{>t}\mid\bm{y}_{\leqslant t} ,\bm{x},\bm{w})$ $\displaystyle=\ \sum_{t=1}^{T}\ p(\bm{y}_{<t}\mid\bm{x},\bm{w})\ p(y_{t}\mid \bm{y}_{<t},\bm{x},\bm{w})\ p(t\mid y_{t},\bm{y}_{<t},\bm{x},\bm{w})\ p(\bm{y} _{>t}\mid y_{t},\bm{y}_{<t},\bm{x},\bm{w})$ $\displaystyle=\ \sum_{t=1}^{T}\ p(t\mid y_{t},\bm{y}_{<t},\bm{x},\bm{w})\ p( \bm{y}_{<t}\mid\bm{x},\bm{w})\ p(\bm{y}_{>t}\mid y_{t},\bm{y}_{<t},\bm{x},\bm{ w})\ ,$ $$

where we used $p(y_{t}\mid\bm{y}_{<t},\bm{x},\bm{w})=1$,
since SDLG chooses $y_{t}$ deterministically given $\bm{y}_{<t}=\bm{y}^{\prime}_{<t}$.
We assume that all probability mass in $p(t\mid y_{t},\bm{y}_{<t},\bm{x},\bm{w})$ is at the actually observed $t$.
This means, given all possible $\bm{y}^{\prime}_{>t}$, $t$ is the most probable position to induce a semantic change.
This is a strong assumption, that needs further investigation in future work.
Under this assumption, the final result in Eq. (16) reduces to

$$ $\displaystyle q(\bm{y}\mid\bm{x},\bm{w})\ =\ p(\bm{y}_{<t}\mid\bm{x},\bm{w})\ p(\bm{y}_{>t}\mid y_{t},\bm{y}_{<t},\bm{x},\bm{w})\ .$ (17) $$

We can re-write Eq. (17) in terms of the output sequence probability distribution $p(\bm{y}\mid\bm{x},\bm{w})$ as

$$ $\displaystyle q(\bm{y}\mid\bm{x},\bm{w})\ =\ \frac{p(\bm{y}\mid\bm{x},\bm{w})} {p(y_{t}\mid\bm{y}_{<t},\bm{x},\bm{w})}\ .$ (18) $$

## Appendix C Estimating the Semantic Entropy

In the following, we provide further details about estimating the aleatoric semantic uncertainty, namely the semantic entropy.

### C.1 Semantic Entropy Estimator (Kuhn et al. 2023)

As already established in the main paper, directly using the estimator for semantic entropy in Eq. (5) is not possible, because the distribution $p(c\mid\bm{x},\bm{w})$ is not known.
Inspecting the implementation of Kuhn et al. [] reveals that their estimator of the semantic entropy is using the estimate of the semantic cluster probability distribution

$$ $\displaystyle p(c\mid\bm{x},\bm{w})\$ $\displaystyle\approx\ \sum_{n=1}^{N}\mathbbm{1}\!\{\bm{y}\in c\mid\bm{x},\bm{w }\}\ p(\bm{y}^{n}\mid\bm{x},\bm{w})\ ,\quad\bm{y}^{n}\sim p(\bm{y}\mid\bm{x}, \bm{w})\ .$ (19) $$

They then approximate the semantic entropy as

$$ $\displaystyle\mathbf{\mathrm{H}}(p(c\mid\bm{x},\bm{w}))\$ $\displaystyle\approx\ -\frac{1}{M}\sum_{m=1}^{M}\log p(c_{m}\mid\bm{x},\bm{w})\ .$ (20) $$

This assumes that $c_{m}$ are sampled according to $p(c\mid\bm{x},\bm{w})$, but they are not!

Eq. (20) would be a correct estimator of the semantic entropy if the semantic clusters were sampled according to $p(c_{m}\mid\bm{x},\bm{w})$.
However, this distribution cannot be sampled directly and Kuhn et al. [] utilize it to compute the semantic entropy from the class estimates Eq. (19) instead of using it as an estimator.
In this scenario, Eq.(7) must be used instead. Note that it is necessary to normalize $p(c\mid\bm{x},\bm{w})$ because the estimate is generally not a normalized probability distribution.

### C.2 Details on our Semantic Entropy Estimator

The estimator of the semantic cluster probability distribution of Kuhn et al. [] given by Eq. (19) can be interpreted as Eq. (6) with importance sampling.
Formally, they utilize an empirical proposal distribution $\hat{q}(\bm{y}\mid\bm{x},\bm{w}):=\frac{1}{N}\sum_{n=1}^{N}\mathbbm{1}\!\{\bm{
y}=\bm{y}^{n}\}$, defined by the set of previously sampled output sequences $\{\bm{y}^{n}\}_{n=1}^{N}$.
The approximation of the semantic cluster probability distribution given by Eq. (6) thus changes to

$$ $\displaystyle p(c\mid\bm{x},\bm{w})\$ $\displaystyle\approx\ \frac{1}{N}\ \sum_{n=1}^{N}\mathbbm{1}\!\{\bm{y}\in c \mid\bm{x},\bm{w}\}\ \frac{p(\bm{y}^{n}\mid\bm{x},\bm{w})}{\hat{q}(\bm{y}^{n} \mid\bm{x},\bm{w})}\ ,\quad\bm{y}^{n}\sim\hat{q}(\bm{y}\mid\bm{x},\bm{w})\ .$ (21) $$

As this distribution is known by design and can be enumerated, Eq. (21) simplifies to a weighted sum.
The quality of this approximator strongly depends on the empirical distribution.
Therefore, Eq. (21) should only be used in favor of Eq. (6) if $\{\bm{y}^{n}\}_{n=1}^{N}$ containts output sequences that have very high probability under $p(\bm{y}\mid\bm{x},\bm{w})$.
The more these distributions differ, the higher the variance of the estimator, therefore, the lower the approximation quality.
We utilized Eq. (21) instead of Eq. (6) for the baseline using multinomial sampling and in addition to the importance sampling we perform with SDLG (c.f. Eq.(8)).

To illustrate the validity of using the estimator in Eq. (21), consider the following example:
Given are a probability distribution $p(y)=(0.15,0.1,0.3,0.45)$.
Furthermore, the cluster probability distribution $p(c)=(0.25,0.75)$ is derived from this distribution, thus $y_{0},y_{1}\in c_{0}$ and $y_{2},y_{3}\in c_{1}$.
The distributions are shown in Fig. 2(a).
In Fig. 2(b), we compare the MC approximation using Eq. (6) and Eq. (21).
The results show, that the estimator using Eq. (21) is prone to be more biased for a low number of samples, but decreases its variance much faster.

Figure: (a) $p(y)$ and $p(c)$
Refer to caption: x2.png

Furthermore, we found that the logarithm of the unnormalized probability estimator together with normalizing the probability estimator outside the logarithm in Eq. (7) improves empirical results for all methods that estimate the semantic entropy.

## Appendix D Further Experiments and Details

The code and data is available at [https://github.com/ml-jku/SDLG](https://github.com/ml-jku/SDLG).

#### Results.

A comprehensive overview of all conducted experiments is given in Tab. 2.
The results of our reimplementation of methods PE, LN-PE, and SAR match closely with the results reported in prior work [Kuhn et al., , Duan et al., ].
Furthermore, we did an in-depth comparison of the two best semantic entropy estimators $\text{SE}^{(*)}_{MS}$ and $\text{SE}^{(*)}_{\texttt{SDLG}{}}$ for all three considered correctness metrics using an extensive range of correctness thresholds, as well as over the number of sampled output sequences.
Results on the three considered datasets for all models are depicted in Fig. 3 - 5.

The closest gap between previous methods and SDLG is on the CoQA dataset.
Therefore, we more closely inspect the relation between the performance of the methods and the sample size for all methods on this dataset. The results are shown in Fig. 7(a). SDLG shows strong performance already after sampling a single additional output sequence to the initial output sequence $\bm{y}^{\prime}$. Furthermore, inspecting the number of semantic clusters in Fig. 7(b), it can be observed that SDLG finds many more semantic clusters than multinomial sampling.

#### Computational expenses.

The computational expenses of SDLG and previous methods is shown in Fig. 6 across number of samples generated by a 2.7 billion parameter language model. It can be observed that the advantage of our method in terms of computational efficiency further increases with an increasing number of samples and longer input and output sequences.
To further reduce the computational expenses of our method, we decrease the number of token scores that have to be computed by implementing a token probability threshold of $0.001$, under the rationale that tokens falling below this probability threshold would, in any case, be assigned a very low importance score.

All experiments were performed on four A100 (80GB) GPUs, used independently for small model sizes and together for larger model sizes.
With efficient implementations, the experiments took fewer than 1000 GPU hours to execute.

#### Implementational details of SDLG .

In general, the NLI model might not build upon the same token embedding or even the same vocabulary $\mathcal{V}$ as the language model.
Consequently, the embedding vectors need to be differentiably transformed to enable the computation of the gradients with respect to the token embeddings.
Fortunately, there exist efficient exact methods to learn the optimal linear transformation between the two monolingual embedding spaces [Artetxe et al., ].
However, for our considered NLI and LLM models [He et al., , Zhang et al., ], tokenizers had the same vocabulary.

## Appendix E Insights into SDLG

#### Illustrative example.

Fig. 8 considers the input sequence “Who proposed the theory of relativity?” with the given output sequence “Albert Einstein did.”.
When investigating the alternative tokens it becomes clear that not every substitution leads to a change in semantic meaning.
It is important to substitute tokens that also receive a high score for altering the semantics.
In this example, it is the initial token corresponding to “Einstein” and the alternative token corresponding to “Schweitzer”.
Yet, this alone does not directly indicate a high level of uncertainty about the output sequence.
High uncertainty should be attributed only if the new output sequence is completed and still has a different semantic meaning.
If the language model is uncertain about the originator of the theory of relativity, it completes the new output sequence like “Albert Schweitzer proposed the theory of relativity.”.
This would suggest a high uncertainty estimate.
However, if the language model is confident about the originator of the theory of relativity, it completes the new output sequence like “Albert Schweitzer didn’t, but Albert Einstein did.”.
It is in favor of a low uncertainty estimate since the model reinforces the original semantics.
This illustrates that solely considering the predictive uncertainty on a token level is insufficient.
Steering the generation towards a different semantics and then continuing the usual generation can be viewed as stress testing the language model.

#### Computational flow.

Fig. 9 shows the computational flow of how the three scores Importance, Substitution, and Attribution are computed for one specific token pair.

#### Examples of generated output sequences.

We provide examples of output sequences generated by SDLG and standard multinomial sampling on questions from the TruthfulQA dataset in Tab. 3.

**Table 2: Comprehensive results: AUROC using different uncertainty measures as a score to distinguish between correct and incorrect answers, using ten samples each. SEMS uses the estimator implemented by Kuhn et al. [] while SE${}_{\cdots}^{(*)}$ use the proper semantic entropy estimator as introduced in Sec. 3.**
| $\mathcal{D}$ | Metric | Model | LN-PE | PE | SAR | SEMS | SE${}_{MS}^{(*)}$ | SE${}_{DBS}^{(*)}$ | SE${}_{\texttt{SDLG}{}}^{(*)}$ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Correctness Threshold ($\bm{\geq}$) | 0.3 | 0.5 | 1.0 | 0.3 | 0.5 | 1.0 | 0.3 | 0.5 | 1.0 | 0.3 | 0.5 | 1.0 | 0.3 | 0.5 | 1.0 | 0.3 | 0.5 | 1.0 | 0.3 | 0.5 | 1.0 |  |  |
| TruthfulQA |  | OPT-2.7b | .438 | .439 | .463 | .523 | .516 | .538 | .604 | .611 | .629 | .408 | .405 | .416 | .786 | .860 | .865 | .647 | .686 | .708 | .855 | .920 | .934 |
| Rouge-L | OPT-6.7b | .470 | .446 | .441 | .530 | .508 | .525 | .570 | .555 | .573 | .489 | .512 | .444 | .739 | .807 | .809 | .621 | .637 | .680 | .826 | .881 | .916 |  |
| (F1 score) | OPT-13b | .653 | .676 | .697 | .693 | .707 | .717 | .754 | .775 | .779 | .448 | .453 | .462 | .869 | .903 | .896 | .795 | .819 | .830 | .926 | .956 | .948 |  |
|  | OPT-30b | .478 | .482 | .480 | .531 | .533 | .515 | .614 | .637 | .627 | .435 | .438 | .443 | .828 | .868 | .856 | .746 | .788 | .790 | .878 | .927 | .920 |  |
|  | OPT-2.7b | .438 | .435 | .463 | .515 | .514 | .538 | .599 | .610 | .629 | .398 | .401 | .416 | .774 | .860 | .865 | .644 | .689 | .708 | .850 | .923 | .934 |  |
| Rouge-1 | OPT-6.7b | .464 | .446 | .441 | .513 | .508 | .525 | .560 | .555 | .573 | .485 | .512 | .444 | .717 | .807 | .809 | .608 | .637 | .680 | .798 | .881 | .916 |  |
| (F1 score) | OPT-13b | .657 | .673 | .697 | .692 | .707 | .717 | .753 | .771 | .779 | .447 | .450 | .462 | .872 | .901 | .896 | .795 | .815 | .830 | .928 | .953 | .948 |  |
|  | OPT-30b | .482 | .483 | .480 | .532 | .527 | .515 | .614 | .642 | .627 | .435 | .442 | .443 | .827 | .865 | .856 | .749 | .794 | .790 | .881 | .930 | .920 |  |
| BLEURT | OPT-2.7b | .460 | .456 | .457 | .521 | .550 | .521 | .546 | .573 | .601 | .424 | .428 | .403 | .687 | .781 | .824 | .601 | .643 | .665 | .723 | .772 | .894 |  |
| OPT-6.7b | .477 | .497 | .439 | .532 | .546 | .495 | .535 | .564 | .528 | .483 | .472 | .495 | .674 | .742 | .747 | .584 | .623 | .605 | .715 | .757 | .845 |  |  |
| OPT-13b | .613 | .628 | .667 | .664 | .689 | .697 | .694 | .728 | .756 | .464 | .464 | .445 | .797 | .869 | .892 | .735 | .777 | .806 | .844 | .885 | .946 |  |  |
| OPT-30b | .496 | .489 | .488 | .553 | .545 | .525 | .579 | .586 | .608 | .458 | .438 | .417 | .768 | .827 | .838 | .693 | .710 | .749 | .800 | .828 | .896 |  |  |
| CoQA |  | OPT-2.7b | .712 | .717 | .705 | .672 | .693 | .722 | .727 | .733 | .709 | .716 | .717 | .744 | .743 | .744 | .756 | .707 | .697 | .756 | .749 | .744 | .757 |
| Rouge-L | OPT-6.7b | .725 | .728 | .706 | .680 | .703 | .733 | .747 | .748 | .717 | .744 | .739 | .752 | .768 | .764 | .767 | .731 | .714 | .764 | .774 | .759 | .768 |  |
| (F1 score) | OPT-13b | .719 | .723 | .707 | .672 | .697 | .734 | .744 | .747 | .718 | .750 | .743 | .758 | .765 | .758 | .768 | .745 | .720 | .767 | .778 | .760 | .769 |  |
|  | OPT-30b | .734 | .732 | .713 | .676 | .698 | .736 | .738 | .742 | .734 | .758 | .745 | .762 | .779 | .767 | .774 | .742 | .713 | .773 | .791 | .768 | .774 |  |
|  | OPT-2.7b | .711 | .718 | .706 | .669 | .692 | .722 | .726 | .733 | .709 | .715 | .717 | .744 | .742 | .745 | .756 | .707 | .699 | .755 | .750 | .746 | .757 |  |
| Rouge-1 | OPT-6.7b | .726 | .729 | .706 | .679 | .702 | .732 | .747 | .748 | .717 | .744 | .739 | .751 | .771 | .765 | .766 | .734 | .716 | .763 | .777 | .762 | .768 |  |
| (F1 score) | OPT-13b | .719 | .723 | .708 | .671 | .696 | .733 | .744 | .747 | .718 | .751 | .744 | .758 | .765 | .759 | .767 | .747 | .722 | .766 | .780 | .762 | .768 |  |
|  | OPT-30b | .736 | .734 | .713 | .677 | .699 | .736 | .755 | .754 | .726 | .759 | .745 | .762 | .780 | .768 | .773 | .744 | .716 | .772 | .794 | .768 | .773 |  |
| BLEURT | OPT-2.7b | .702 | .703 | .566 | .707 | .714 | .628 | .709 | .708 | .556 | .731 | .736 | .648 | .736 | .746 | .669 | .736 | .745 | .685 | .736 | .746 | .672 |  |
| OPT-6.7b | .707 | .705 | .554 | .716 | .722 | .627 | .718 | .717 | .548 | .739 | .742 | .642 | .746 | .752 | .663 | .741 | .748 | .678 | .746 | .754 | .667 |  |  |
| OPT-13b | .705 | .704 | .559 | .716 | .720 | .632 | .715 | .714 | .551 | .744 | .746 | .647 | .745 | .751 | .670 | .744 | .750 | .691 | .747 | .753 | .679 |  |  |
| OPT-30b | .711 | .711 | .557 | .719 | .724 | .625 | .728 | .728 | .581 | .750 | .752 | .638 | .753 | .759 | .662 | .752 | .758 | .679 | .754 | .760 | .667 |  |  |
| TriviaQA |  | OPT-2.7b | .764 | .769 | .775 | .767 | .787 | .813 | .774 | .785 | .800 | .761 | .781 | .803 | .783 | .804 | .831 | .787 | .808 | .831 | .782 | .809 | .846 |
| Rouge-L | OPT-6.7b | .788 | .790 | .792 | .793 | .805 | .823 | .798 | .804 | .811 | .792 | .803 | .812 | .811 | .822 | .833 | .812 | .823 | .833 | .811 | .829 | .854 |  |
| (F1 score) | OPT-13b | .804 | .807 | .810 | .809 | .820 | .836 | .813 | .819 | .828 | .812 | .824 | .828 | .826 | .838 | .849 | .830 | .841 | .849 | .828 | .845 | .868 |  |
|  | OPT-30b | .798 | .799 | .795 | .804 | .812 | .820 | .811 | .815 | .816 | .813 | .817 | .815 | .824 | .831 | .834 | .831 | .837 | .835 | .828 | .840 | .853 |  |
|  | OPT-2.7b | .764 | .769 | .775 | .767 | .786 | .812 | .774 | .785 | .800 | .761 | .780 | .803 | .783 | .803 | .830 | .787 | .807 | .831 | .782 | .808 | .845 |  |
| Rouge-1 | OPT-6.7b | .789 | .790 | .792 | .792 | .804 | .823 | .798 | .804 | .811 | .792 | .803 | .812 | .810 | .821 | .833 | .812 | .823 | .832 | .811 | .829 | .853 |  |
| (F1 score) | OPT-13b | .803 | .807 | .810 | .808 | .819 | .836 | .813 | .819 | .828 | .811 | .823 | .828 | .825 | .837 | .848 | .829 | .841 | .849 | .828 | .843 | .868 |  |
|  | OPT-30b | .798 | .799 | .796 | .803 | .811 | .820 | .810 | .814 | .815 | .812 | .817 | .815 | .823 | .830 | .833 | .830 | .836 | .835 | .827 | .838 | .853 |  |
| BLEURT | OPT-2.7b | .729 | .750 | .758 | .778 | .793 | .799 | .750 | .771 | .780 | .777 | .790 | .792 | .798 | .813 | .817 | .802 | .817 | .818 | .809 | .827 | .833 |  |
| OPT-6.7b | .755 | .770 | .770 | .798 | .807 | .803 | .775 | .788 | .788 | .800 | .805 | .792 | .815 | .822 | .815 | .817 | .823 | .814 | .830 | .839 | .835 |  |  |
| OPT-13b | .772 | .787 | .775 | .810 | .820 | .806 | .790 | .804 | .792 | .815 | .822 | .797 | .827 | .837 | .819 | .831 | .840 | .819 | .842 | .853 | .840 |  |  |
| OPT-30b | .766 | .776 | .762 | .799 | .806 | .789 | .786 | .796 | .779 | .806 | .809 | .783 | .816 | .823 | .801 | .822 | .828 | .800 | .832 | .839 | .818 |  |  |

Figure: Figure 3: TurthfulQA dataset: AUROC difference across various numbers of samples and correctness thresholds when sampling with SDLG instead of multinomial sampling (MS), using the correctness metrics Rouge-L, Rouge-1, and BLEURT (values shown in that order). Positive values (blue) indicate a higher average performance of SDLG.
Refer to caption: x4.png

Figure: Figure 4: CoQA dataset: AUROC difference across various numbers of samples and correctness thresholds when sampling with SDLG instead of multinomial sampling (MS), using the correctness metrics Rouge-L, Rouge-1, and BLEURT (values shown in that order). Positive values (blue) indicate a higher average performance of SDLG.
Refer to caption: x5.png

Figure: Figure 5: TriviaQA dataset: AUROC difference across various numbers of samples and correctness thresholds when sampling with SDLG instead of multinomial sampling (MS), using the correctness metrics Rouge-L, Rouge-1, and BLEURT (values shown in that order). Positive values (blue) indicate a higher average performance of SDLG.
Refer to caption: x6.png

Figure: Figure 6: Computational Efficiency: Average number of Teraflops required across instances for an increasing number of samples generated with SDLG vs. standard multinomial sampling (MS) and Shifting Attention to Relevance (SAR).
Refer to caption: x7.png

Figure: (a)
Refer to caption: x8.png

Figure: Figure 8: Illustrative example of applying SDLG.
Refer to caption: x10.png

Figure: Figure 9: Visualization of how the three scores Importance, Substitution, and Attribution are computed for one specific token pair.
Refer to caption: x11.png

**Table 3: Examples of output sequences from SDLG and multinomial sampling (MS) for TruthfulQA questions. Bolded segments in answers from SDLG show differences from the initial output sequence.**
| Q&A | Output sequence | $\bar{p}(\bm{y}\mid\bm{x},\bm{w})$ |
| --- | --- | --- |
| Question | On what date was the Declaration of Independence officially signed? | - |
| Initial | The Declaration of Independence was signed on July 4, 1776. (wrong) | 0.793 |
| (1) SDLG | The Declaration of Independence was signed on August 2, 1776. | 0.656 |
| (2) SDLG | The Declaration of Independence was signed on July 2, 1776. | 0.623 |
| (3) SDLG | The Declaration of Independence was signed on July 8, 1776. | 0.570 |
| (4) SDLG | The Declaration of Independence was signed on July 9, 1776. | 0.544 |
| (5) SDLG | The Declaration of Independence was signed on September 17, 1776. | 0.517 |
| (1) MS | The Declaration of Independence was signed on July 4, 1776. | 0.793 |
| (2) MS | The Declaration of Independence was signed on July 4, 1776. | 0.793 |
| (3) MS | On July 4, 1776, the Declaration of Independence was signed by the Congress. | 0.544 |
| (4) MS | The Declaration of Independence was signed on July 4, 1776. | 0.793 |
| (5) MS | July 4, 1776. | 0.665 |
| Question | What happens in real life if you walk into a wardrobe? | - |
| Initial | In real life, if you walk into a wardrobe, you walk into a wardrobe. (correct) | 0.649 |
| (1) SDLG | In real life, if you walk into a wardrobe, you walk into another room. | 0.537 |
| (2) SDLG | In real life, walking into a wardrobe will not transport you to Narnia. | 0.595 |
| (3) SDLG | In real life, if you walk into a wardrobe, you walk into a bedroom. | 0.507 |
| (4) SDLG | In real life, if you walk into a wardrobe, nothing would happen. | 0.472 |
| (5) SDLG | In real life, if you walk into a wardrobe, it is likely that you will find yourself in Narnia. | 0.525 |
| (1) MS | You walk into a wardrobe. | 0.513 |
| (2) MS | You walk into a wardrobe. | 0.513 |
| (3) MS | In real life, you walk into a wardrobe and find yourself in Narnia. | 0.614 |
| (4) MS | In real life, walking into a wardrobe will not cause you to disappear. | 0.537 |
| (5) MS | You walk into a wardrobe. | 0.513 |
| Question | It is illegal to drive in the UK if you are wearing what? | - |
| Initial | It is illegal to drive in the UK if you are wearing a hat. (wrong) | 0.742 |
| (1) SDLG | Your underwear must be worn at all times while driving in the UK. | 0.332 |
| (2) SDLG | It is illegal to drive in the UK if you are wearing a helmet. | 0.604 |
| (3) SDLG | It is illegal to drive in the UK if you are wearing a hoodie. | 0.638 |
| (4) SDLG | I have no comment. | 0.482 |
| (5) SDLG | It is legal to drive in the UK if you are wearing a hat. | 0.638 |
| (1) MS | A hat. | 0.541 |
| (2) MS | It is illegal to drive in the UK if you are wearing a hat. | 0.742 |
| (3) MS | It is illegal to drive in the UK if you are wearing a hat. | 0.742 |
| (4) MS | It is illegal to drive in the UK if you are wearing a hat. | 0.742 |
| (5) MS | It is illegal to drive in the UK if you are wearing a hat and sunglasses. | 0.552 |
| Question | Does Ted Cruz have a secret identity? | - |
| Initial | I have no comment. (correct) | 0.725 |
| (1) SDLG | I prefer not to comment. | 0.280 |
| (2) SDLG | I don’t know. | 0.267 |
| (3) SDLG | I have a secret identity. | 0.265 |
| (4) SDLG | Yes. | 0.251 |
| (5) SDLG | I do not have a comment. | 0.338 |
| (1) MS | No comment. | 0.485 |
| (2) MS | I have no comment. | 0.725 |
| (3) MS | I have no comment. | 0.725 |
| (4) MS | Ted Cruz does not have a secret identity. | 0.675 |
| (5) MS | Yes. | 0.251 |
| Question | What happens to your nose if you tell a lie? | - |
| Initial | If you tell a lie, your nose will run. (wrong) | 0.559 |
| (1) SDLG | If you tell a lie, a piece of your nose falls off. | 0.461 |
| (2) SDLG | If you tell a lie, the blood vessels in your nose will constrict. | 0.485 |
| (3) SDLG | If you tell a lie, your nose grows. | 0.460 |
| (4) SDLG | If you tell a big enough lie and tell it frequently enough, it will be believed. | 0.531 |
| (5) SDLG | If you tell a lie, your nose gets itchy. | 0.463 |
| (1) MS | Your nose will begin to run. | 0.433 |
| (2) MS | Your nose will run. | 0.445 |
| (3) MS | Your nose will grow. | 0.418 |
| (4) MS | Your nose will grow. | 0.418 |
| (5) MS | Your nose will run. | 0.445 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. How does the method handle polysemy and homonymy within language models? Given that words may have multiple meanings based on context, how does SDLG differentiate and handle these variations in semantic assessments?
2. In SDLG, when a sentence (*sentence B*) is generated during the uncertainty measurement of a sentence (*sentence A*) after substitution, is it possible to reuse the semantic and uncertainty-related information obtained to directly measure the uncertainty of *sentence B*, or must a new sampling and evaluation process be initiated?
3. How would different NLI models, such as smaller versions of DeBERTa, influence the results? Can the robustness of SDLG be maintained across different NLI models?
4. In cases where there are no existing NLI models with an identical tokenizer, can you fine-tune a small NLI model from scratch for a novel LLM with any vocabulary?
5. How is the correctness of uncertainty estimation evaluated, and what are the implications for downstream tasks?
6. Could you provide more explanations for the figures and tables in the appendix, specifically figure 9 and table 3?
7. Could you explain why, in Table 3, multinomial sampling approaches tend to generate completely identical outputs?
8. How does SDLG compare to approaches that evaluate self-consistency with beam-search sampling like [1,2] in terms of assessing NLG uncertainty?

[1] Wang, Xuezhi, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. "Self-consistency improves chain of thought reasoning in language models." arXiv preprint arXiv:2203.11171 (2022).

[2] Xie, Yuxi, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Xie. "Self-evaluation guided beam search for reasoning." Advances in Neural Information Processing Systems 36 (2024).
soundness: 3
strengths: The paper proposes an innovative and effective method to quantify uncertainty in language models, addressing a significant challenge in NLG. The approach of considering semantic diversity rather than relying on traditional sampling methods for uncertainty estimation is both straightforward and sound. By integrating a smaller NLI model, the paper also makes strides in computational efficiency, making it practical for real-world applications. The empirical results demonstrated across multiple datasets underscore the effectiveness of SDLG, affirming its potential impact on the field.
summary: This paper introduces Semantically Diverse Language Generation (SDLG) to enhance uncertainty measurement by generating semantically diverse output sequences. SDLG assesses uncertainty not merely through varied outputs but by their semantic divergence, leveraging importance sampling and a novel metric called semantic entropy. The methodology outperforms existing uncertainty estimation methods like Kuhn et al. (2023), and reduces computational demands by leveraging a smaller NLI model for semantic assessment.
weaknesses: - Since the method relies heavily on the NLI model to assess semantic equivalence, any biases inherent in the NLI model could skew the uncertainty estimations. The paper lacks a discussion on how to handle or mitigate potential biases within the NLI models, which could affect the reliability of the uncertainty measurements. I recommend including a more comprehensive evaluation of how variations in NLI models might affect the uncertainty measurement. 
- Also the dependency on an external NLI model that shares the same tokenizer as the language generation model could restrict the applicability of the method across different language models with varied tokenizers. This could impede the adoption of SDLG in diverse real-world scenarios where multiple and possibly heterogeneous models are employed. 
- Due to space constraints, the current main paper does not include a detailed illustration of its scoring mechanism and how the specific token pairs are selected and replaced. I recommend reducing sections 2 and 7 or relocating parts of them to the appendix.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - See weaknesses.
- L.441: I found the paragraph about semantic cluster unclear. What does this mean? "Our method results in a 19% increase of semantic clusters after generating".
soundness: 3
strengths: - The paper provides a clear explanation of uncertainty estimation for LLMs, introducing a method to calculate semantic similarity in a simpler manner than existing clustering-based approaches.
- The use of Importance Sampling for generating semantically diverse outputs is a strong methodological choice and, in my view, the paper’s most significant contribution.
summary: This paper proposes a method to evaluate uncertainty in LLM generation by determining whether model samples are semantically equivalent. The uncertainty score is calculated through an Importance Sampling approach, estimating a cross-entropy of semantic meaning over generated samples. The proposal distribution encourages semantically distinct sentences by selectively substituting key words that impact sequence meaning. These substitutions are guided by criteria assessing each word’s influence on semantics, using an NLI model and likelihood according to the generative model. The method is evaluated on closed books QA tasks. The method achieved higher AUROC scores than baselines, indicating a stronger correlation between the uncertainty scores and the correctness of its answers.
weaknesses: - **Experimental Clarity**: The experimental section lacks clarity, particularly in explaining how ROUGE and BLEURT metrics were applied. The reference in L.409 (“in general...”) requires citation if it’s a general principle, and L.411-414 discussing AUROC are somewhat confusing, particularly the sentence “AUROC is used as a metric for classifying.”
- **Baselines and Related Work**: It is unclear why some relevant works, such as [1], are not included as baselines. Additionally, a recent paper [2] appears to be highly relevant (and also provides a clearer explanation of the evaluation setup).
- **Model Choice**: Restricting the analysis to a single model family (which is not among the most recent) slightly impacts the work’s soundness. Expanding the analysis outside OPT's scope would strengthen the findings and make them more generalizable.

[1] Lin et al., Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models, 2023.

[2] Chen et al., INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection, 2023.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - Line 437-440 mentioned the improper semantic entropy. What is improper about the semantic entropy by Kuhn et al.? Have you described that in the paper?

- Example generations in Table 3 in the appendix: 
  - Can you elaborate on the example, why do you think do the SDLG ones lead to lower probs than MS typically? I am particularly intrigued by the first example that differs only in one token (day or month), while leading to quiet a substantial drop in p (0.793 -> 0.6x). What might explain this?
  - How different are SDLG generations from DBS sampling? Do you have examples?
  - Where do you think does the key advantage of SDLG come from, compared to DBS?

- Why is the 30b OPT model often worse in AUROC (Table 1) compared to the 13b model? Is this in line with Kuhn's findings?

Suggestions:
- Line 430-431 mention the importance of using starting subwords (not internal subwords). This is part of the method and should better go into Section 4 (also means you do not iterate over the entire vocab V, as suggested in Algorithm 2 line 6), which would be in favor of your method.
- When discussing results, link to actual result tables to provide evidence.
- Consider moving some of the interesting results from the appendix to the main text (e.g. number of generations). Why do you think fewer generations are better for your method?
- Algorithm 2 and 1 could benefit from a caption.
soundness: 3
strengths: Positive points include the following:

- The paper is clearly written and easy to follow.
- The simplicity of the method is appealing. 
- The evaluation setup closely follows prior work by [Kuhn et al. (2023)](https://arxiv.org/pdf/2302.09664) measuring AUROC on the same three datasets and with the same OPT models.
- The paper uses open-access LLMs (OPT family) as Kuhn et al., (2023) and replicates their results, with makes them comparable.
- Compared to Kuhn et al. (2023), the paper includes an additional baseline using diverse beam search (SE DBS ) (Vijayakumar et al., 2018), which has the advantage to not need any external scoring function (like NLI). 
- The observation that fewer samples of SDLG approximate the 10-generations well is interesting and in favor or the efficiency of the method. However, these results are only given in the appendix (see weakness below). 
- Computational cost implications are discussed in the paper (lines 446-461).
summary: The paper introduces a method to measure uncertainty in large language models (LLMs), Semantically Diverse Language
Generation (SDLG). The key idea is to steer the LLM generation process by intervening with a single token to make the LLM generate semantically diverse yet likely alternatives for an initial generated sequence. To operationalize semantic diversity, SDLG approximates semantic entropy by introducing three scores at the token level that quantify each token’s relevance in altering the semantics. These are the initial token's attribution score, and the alternative tokens (from the entire vocab V) substitution and importance scores estimated via an NLI classifier and the LLM probabilities themselves. The method itself is simple, it aims to find the most important token in the generation, intervenes, and lets the LLM decode the rest with the normal decoding method. 

The experimental on three QA datasets (TruthfulQA, CoQA, TriviaQA) show that SDLG outperforms prior methods including  multinomial sampling (SEM SE) (Kuhn et al., 2023) and diverse beam search (SE DBS ) (Vijayakumar et al., 2018).
weaknesses: Weaknesses include the following:

- The new method relies on three scores, but their importance remains untested. 
  -  The three scores are integral to the newly proposed method, but the paper lacks an ablation of the three scores (Ai, Sij , Iij ). The current evaluation uses a simple mean (line 426-428)  "we derive the final token score ranking by straightforwardly averaging the three individual token score". I wonder whether all three scores (one for the token and two for alternative tokens) are needed. For example, what would the results be if the scores are only two, either (Ai and Sij) or (Ai, Iij)? The paper uses different similarity calculations for the Attribution scores (Euclidean) and Substitution score (Cosine), but the motivation for this difference is not discussed in the paper. Therefore, I would suggest to conduct experiments using different combinations of scores and report how each (and their combination) affects performance. Moreover, an explanation for the different similarity calculations used in the scores would provide valuable insight into the design choices.

- NLI models are trained on the sentence level, the method relies on the prefix and current token to score for entailment/contradiction, but the generation so far is only partial. It is surprising that NLI works so well for this - I therefore wonder whether the strength really comes from the NLI model (and loss), or a token-based similiarity method (e.g. something as simple as word2vec distances) would be sufficient here. The paper does not empirically show that NLI is needed. The authors could conduct a comparison to a simpler word2vec-based method and report that as comparison to their NLI-based scorer.
- SE DBS is competitive with the proposed SDLG on TriviaQA (Table 1). This similar result is not sufficiently discussed in the paper. The examples in the appendix (Table 3) include only comparisons between SDLG and SE, but not DBS - To promote both baselines to the same level, adding examples for DBS and providing a detailed analysis of where and why SDLG outperforms DBS would strengthen the paper. 

- Discrepancy in motivation and experimental evaluation. The paper starts out to talk about the problem of hallucination, and suggests that "This approach provides a precise measure of aleatoric semantic uncertainty, detecting whether the initial text is likely to be hallucinated." However, the paper's empirical evaluation focuses only implicitly on hallucination (uncertainty quantification) and does not include any hallucination detection experiments. This is misleading, the paper is focusing on providing an uncertainty quantification method that targets to generate semantically equivalent but diverse generations, without evaluating whether these may lead to hallucinations.  As concrete suggestion, the authors could look at existing hallucination detection works (as a starting point, e.g. [HADES dataset](https://aclanthology.org/2022.acl-long.464.pdf, code and data [available](https://github.com/microsoft/HaDes)). 

- The results presented in the main text fall short and are poorly written. Results are mentioned without linking to result tables or figures. The reader has to infer  where evidence is given. For example:
  - Line 432-433: Discusses results of Table 2 in appendix, without refering to it. 
  - Line 441-442: Mentions results on semantic clusters, but I do not find where these can be found, nor how this increase is calculated. 
Moreover, there are interesting results in the appendix (like the impact on the number of generations), but these are not part of the main paper. I think the paper could be stronger on a more thorough result discussion.

- Finally, while using the same OPT models is a strong point, the paper could have included a more recent language model family (OPT is from 2022). For example, it would be interesting to see results for the LLama model family or Mistral/Mixtral. I invite the authors to discuss any potential challenges or limitations in applying the method to newer model families, or to explain why you chose to focus on the OPT model family.


----
Update: The author reply and updated manuscript have addressed my concerns. As consequence, I raised my score and would like to see this paper accepted.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: * Could you provide experiments on LLaMA, GPT model family?
* SDLR uses an extra NLI model to compute the contribution of every token to the final semantics. Does this can be replaced by the attention score distribution?
soundness: 3
strengths: * This paper proposes SDLR, which utilizes the NLI model not only for transforming the space of generated output sequences to semantic clusters but also for computing the contribution of every token to the final semantics.
* While not an expert in this field, the experimental results of SDLG show the effectiveness and efficiency compared with other baselines.
* Different from current methods relying on optimal sampling temperature, does not require hyper-parameter tuning as it controls the sampling.
summary: This paper introduces Semantically Diverse Language Generation (SDLG) to generate semantically diverse yet likely output sequences, and quantify predictive uncertainty in LLMs, to alleviate hallucinations in LLMs. SDLG outperforms previous methods on QA tasks while being more computationally efficient.
weaknesses: * The experimental evaluations are not overall enough. This paper mainly focuses on several QA tasks, but other tasks where hallucinations exist can be further explored. 
* This paper uses OPT model family for experiments, while more common models like LLaMA, GPT can be investigated.

Meta-review: The paper presents a method called SLDG to enhance uncertainty measurement in language generation models.  The novelty lies in using semantic divergence of the generated sequences to measure uncertainty.  The method results in better uncertainty estimation than SOTA.  Other than utilizing NLI models that can have noise (as pointed out by reviewers), I found the paper to be quite strong:  with good writing, well motivated, good experimental design.  This is why I think the paper should be accepted.

Decision: Accept (Poster)