Improving Uncertainty Quantification in Large Language Models via Semantic Embeddings

## Abstract

Abstract Accurately quantifying uncertainty in large language models (LLMs) is crucial for their reliable deployment, especially in high-stakes applications. Current state-of-the-art methods for measuring semantic uncertainty in LLMs rely on strict bidirectional entailment criteria between multiple generated responses and also depend on sequence likelihoods. While effective, these approaches often overestimate uncertainty due to their sensitivity to minor wording differences, additional correct information, and non-important words in the sequence. We propose a novel approach that leverages semantic embeddings to achieve smoother and more robust estimation of semantic uncertainty in LLMs. By capturing semantic similarities without depending on sequence likelihoods, our method inherently reduces any biases introduced by irrelevant words in the answers. Furthermore, we introduce an amortised version of our approach by explicitly modelling semantics as latent variables in a joint probabilistic model. This allows for uncertainty estimation in the embedding space with a single forward pass, significantly reducing computational overhead compared to existing multi-pass methods. Experiments across multiple question-answering datasets and frontier LLMs demonstrate that our embedding-based methods provide more accurate and nuanced uncertainty quantification than traditional approaches.

## 1 Introduction

Large Language Models (LLMs) have revolutionised natural language processing (see e.g. The Gemini Team,, ; Touvron et al., ; OpenAI, ; Brown et al., ), achieving state-of-the-art performance across a wide variety of tasks including question-answering. As these models are increasingly deployed in critical domains like healthcare (Singhal et al., ) and law (Weiser, ) ensuring their reliability and trustworthiness has become imperative. A significant challenge in this context is the phenomenon of “hallucinations”—instances where LLMs generate fluent and coherent responses that are factually incorrect or misleading (Ji et al., ; Filippova, ; Maynez et al., ; Tian et al., ).

Uncertainty quantification (UQ) methods like Bayesian inference (Wilson & Izmailov, ), ensemble methods (Lakshminarayanan et al., ), and Monte Carlo dropout (Gal & Ghahramani, ) have been extensively studied in traditional neural networks to enhance model reliability by providing confidence measures in predictions. However, applying these traditional UQ methods to LLMs faces challenges due to the open-ended nature of free-form natural language generation (Kuhn et al., ). The core issue lies in the fundamental mismatch between traditional UQ approaches, which typically estimate uncertainty in output probabilities, and the semantics (meaning) space of language generation in LLMs.

For example, consider the following scenario of two responses generated for the same query: “London is the biggest city in the UK”. “The largest city in the UK is London”. In this scenario, the output token probabilities will include uncertainty about the syntax and choice of words (e.g., using “biggest” or “largest”), along with the uncertainty about the underlying semantic content of the response. Traditional UQ techniques focusing on token probabilities conflate these different sources of uncertainty, making it challenging to isolate semantic uncertainty, which is critical for assessing the reliability of the generated information.

Semantic entropy (Kuhn et al., ) aims to isolate semantic uncertainty by sampling multiple answers from the LLM for a given prompt. It does so by clustering the generations into sets of equivalent semantics and then estimating uncertainty in the space of identified semantics. The premise is that higher semantic uncertainty leads to more diverse meanings in the generated responses, while lower uncertainty results in more semantically consistent responses.
In order to cluster semantically equivalent answers, semantic entropy leverages a strict bidirectional entailment criterion which, as we show in this work, can be sensitive to minor variations in wording, additional correct information, or non-essential words in the generated responses. Such sensitivity can lead to an overestimation of semantic uncertainty. Additionally, semantic entropy requires multiple forward passes, which can limit its practicality in production environments where low latency is essential and for larger LLMs where each forward pass incurs substantial computational expenses.

To address these challenges, we make the following key contributions:

- •
Semantic Embedding Uncertainty (SEU): We introduce SEU, by leveraging the average pairwise cosine similarity of full response embeddings SEU avoids the issues associated with using bi-directional entailment as a criterion for clustering semantically equivalent responses (see sections 3 and 4).
- •
Amortised SEU: We present an amortised version that models semantics as latent variables in a joint probabilistic model. This allows for the estimation of posterior uncertainty in the latent semantics within a single forward pass, alongside the response generation, significantly reducing computational overhead and enhancing practicality for deployment in production environments (see sections 5 and 6).

## 2 Background

We first provide a concise summary of Semantic Entropy and detail its limitations which form the motivation of our proposed approaches.

### 2.1 Semantic Entropy

Semantic Entropy (SE) is a measure of uncertainty in sequences generated by language models (Kuhn et al., ). The central idea is that if a language model is unsure about how to answer a specific question, it will produce responses that are different in wording and semantics across multiple generations when the model is given the same input. SE groups semantically similar responses and calculates the entropy based on the variety of distinct meanings found in the output responses.
Specifically, the key steps involved are: (i) sample $M$ output sequences $\{{\mathbf{s}}_{1},\dots,{\mathbf{s}}_{M}\}$ from the language model’s predictive distribution $p({\mathbf{s}}|{\mathbf{x}})$ given an input ${\mathbf{x}}$; (ii) cluster the sampled sequences into $K$ semantic equivalence classes $C=\{c_{1},\ldots,c_{K}\}$ using a bidirectional entailment algorithm. Two sequences ${\mathbf{s}}_{i}$ and ${\mathbf{s}}_{j}$ are considered semantically equivalent if and only if a natural language inference model classifies their mutual relationship as entailment in both directions; (iii) estimate the probability of each semantic cluster $p(c_{k}|{\mathbf{x}})=\sum_{{\mathbf{s}}\in c_{k}}p({\mathbf{s}}|{\mathbf{x}})$; and (iv) compute the entropy over semantic clusters: $\text{SE}({\mathbf{x}})=-\sum_{k=1}^{K}p(c_{k}|{\mathbf{x}})\log p(c_{k}|{
\mathbf{x}})$.

#### 2.1.1 Bidirectional entailment and its limitations

We focus on step (ii), where bidirectional entailment is used to identify distinct semantic clusters among the $M$ responses. This strict criterion, however, can be overly sensitive to minor variations in wording, additional correct information, or non-essential words.
This issue is illustrated by several examples listed in Table 1.

**Table 1: Comparison of bidirectional entailment and cosine similarity for assessing semantic equivalence. DeBERTaLarge (He et al., ) is used to predict entailment as used in Kuhn et al. (), and the inputs to the cosine similarity are obtained using sentence-BERT (Reimers & Gurevych, ).**
| Context | Responses | Bidirectional Entailment | Cosine Similarity |
| --- | --- | --- | --- |
| What is the primary function of the mitochondria in cells? | 1. The mitochondria produce energy for the cells.<br>2. Mitochondria provides energy to cells in the body. | False | 0.974 |
| What happens when you heat ice? | 1. Heating ice will eventually boil after becoming water.<br>2. When ice is heated, it melts into water before boiling. | False | 0.893 |
| What do mammals have in common? | 1. Mammals are warm-blooded and have hair or fur.<br>2. All mammals (like humans and dogs) are warm-blooded creatures with hair. | False | 0.927 |

##### Example 1: Generality Mismatch in Responses

Both responses correctly state that mitochondria produce energy. However, bidirectional entailment fails because the first response uses “produce energy for the cells”, while the second uses “provides energy to cells in the body”. The addition of “in the body” in the second response making it a less general statement than the first leads to a False classification despite a high cosine similarity of 0.974.

##### Example 2: Phrasing Variations

Both responses accurately describe the process of heating ice, but bidirectional entailment fails due to different information orders (“eventually boil” vs. “before boiling”). These temporal differences in phrasing result in a False classification, despite a cosine similarity of 0.893.

##### Example 3: Additional Correct Information

Both responses identify key traits of mammals, but bidirectional entailment fails because the second response includes additional correct information (“like humans and dogs”) not present in the first.
These additions and wording differences lead to a False classification, despite a high cosine similarity of 0.927.

As the above examples demonstrated, natural language is inherently varied, and strict binary classifications do not account for the nuances and gradations in meaning that often occur in human language. Bidirectional entailment treats semantic equivalence as a binary condition: responses either fully entail each other or they do not. This strictness can lead to it being over-sensitive to minor variations as shown in our examples, small differences in phrasing or the inclusion of additional but correct information can cause bidirectional entailment to fail, even when the core meaning is preserved. In contrast, the high cosine similarity scores across all examples suggest that these responses are indeed very close in semantic space. To address this limitation and more robustly quantify semantic uncertainty, we propose using the average pairwise cosine similarity of the generated responses. This approach can capture semantic closeness more flexibly, allowing for minor variations in wording or additional correct information without overly penalising the uncertainty estimate. We detail this proposed method in the following section.

While the average pairwise cosine similarity approach addresses the limitations of binary semantic classifications, it still has limited practical applicability. Both SE and the proposed method, require multiple forward passes through the language model. This requirement significantly limits their practicality in production environments, especially for large LLMs where each forward pass incurs a substantial computational cost. To this end, we propose a novel method in Section 5 that treats the semantics of the response as latent variables in a joint probabilistic model. Our approach employs amortised inference over the semantics of the full response, allowing us to estimate the LLM’s uncertainty about the semantics of its entire response (i.e., the complete sequence of tokens) in a single forward pass. This method drastically reduces the computational overhead required to estimate semantic uncertainty while preserving the benefits of comparing semantic embeddings using cosine similarity.

## 3 Semantic Embedding Uncertainty

To overcome the limitations of bidirectional entailment in measuring semantic uncertainty, we propose semantic embedding uncertainty (SEU), a novel approach based on the average pairwise cosine similarity of the generated responses’ embeddings. This method leverages continuous semantic representations to capture nuanced meanings more precisely, offering a robust measure of semantic uncertainty in language model outputs.

Similar to Semantic Entropy, given an input ${\mathbf{x}}$, we generate $M$ output sequences $\{{\mathbf{s}}_{1},{\mathbf{s}}_{2},\dots,{\mathbf{s}}_{M}\}$ from the language model’s predictive distribution $p({\mathbf{s}}|{\mathbf{x}})$. We then obtain vector embeddings $\{{\mathbf{e}}_{1},{\mathbf{e}}_{2},\dots,{\mathbf{e}}_{M}\}$ for each sequence using a pretrained embedding model $\phi({\mathbf{s}})$, such as a transformer-based sentence encoder (Reimers & Gurevych, ). The semantic uncertainty is quantified by computing the negative average pairwise cosine similarity between the embeddings:

$$ $\displaystyle\text{SEU}(x)=1-\frac{2}{M(M-1)}\sum_{i=1}^{M-1}\sum_{j=i+1}^{M} \cos\left({\mathbf{e}}_{i},{\mathbf{e}}_{j}\right),$ (1) $$

where $\cos\left({\mathbf{e}}_{i},{\mathbf{e}}_{j}\right)$ is the cosine similarity between embeddings ${\mathbf{e}}_{i}$ and ${\mathbf{e}}_{j}$, $\cos\left({\mathbf{e}}_{i},{\mathbf{e}}_{j}\right)=\frac{{\mathbf{e}}_{i}\cdot
{\mathbf{e}}_{j}}{\|{\mathbf{e}}_{i}\|\,\|{\mathbf{e}}_{j}\|}$, $\|{\mathbf{e}}\|$ denotes the Euclidean norm of vector ${\mathbf{e}}$, and ${\mathbf{e}}_{i}\cdot{\mathbf{e}}_{j}$ represents the dot product between vectors ${\mathbf{e}}_{i}$ and ${\mathbf{e}}_{j}$.

The proposed approach relies on high-quality embedding models to map semantically similar sentences to nearby points in the embedding space (e.g. Mikolov et al., ; Pennington et al., ; Reimers & Gurevych, ). Intuitively, cosine similarity quantifies the angle between vectors in the high-dimensional embedding space, serving as a measure of their semantic alignment. By aggregating pairwise similarities across all generated responses, we capture the overall semantic coherence of the model’s outputs. If the language model is certain about the response to input $x$, the generated responses will be semantically similar, leading to high cosine similarity scores and a low semantic uncertainty $\text{SEU}(x)$. Conversely, if the model is uncertain, the responses will be more diverse semantically, resulting in lower cosine similarity scores and a higher $\text{SEU}(x)$.

The proposed approach offers two key advantages over bidirectional entailment. First, unlike the *binary* outcome of bidirectional entailment—which rigidly classifies responses as either semantically equivalent or not—cosine similarity provides a *continuous* metric. This allows for a nuanced assessment of semantic closeness between responses. While bidirectional entailment may fail to recognise near-equivalent meanings due to minor differences (thereby assigning a value of zero similarity), cosine similarity captures the *degree* of similarity between responses. This continuous spectrum more accurately reflects the gradations in human language understanding.
Second, as shown above, bidirectional entailment is highly sensitive to syntactic variations, paraphrasing, and the inclusion of additional relevant information, often resulting in false negatives when determining semantic equivalence. In contrast, cosine similarity focuses on the underlying *semantic content* rather than exact entailment. This makes it less sensitive to linguistic variability, such as differences in syntax or phrasing.

## 4 Empirical Evaluation of SEU

In this section, we empirically evaluate our proposed SEU method against existing uncertainty estimation techniques. Our goal is to demonstrate that SEU provides a more accurate and robust measure of semantic uncertainty in language model outputs, particularly in the context of open-domain question answering.

### 4.1 Experimental Setup

Models To align with modern practices of using instruction fine-tuned LLMs for chat purposes, we employ Llama-3.1-8B-Instruct (Dubey et al., ), Mistral-7B-Instruct (Jiang et al., ), and Phi-3.5-mini-instruct (Abdin et al., ) as our base models. For each model-dataset combination, we generate 5 responses per question (that is $M=5$) at a temperature of 0.5. This temperature was recommended as the optimal temperature for SE in previous work (Kuhn et al., ). Additionally, we prompt Llama and Phi models with
“Answer the following question as briefly as possible”, and Mistral with “Answer the following question briefly using a few words” to match the short-answer format of our datasets.

Datasets We evaluate our proposed Semantic Embedding Uncertainty (SEU) method on three challenging question-answering datasets: TriviaQA (Joshi et al., ), NQ Open (Kwiatkowski et al., ; Lee et al., ), and the natural question subset of the FLAN collection (Longpre et al., , Flan QA) (^1^11[https://huggingface.co/datasets/Muennighoff/flan](https://huggingface.co/datasets/Muennighoff/flan)). TriviaQA offers a large set of trivia questions both with and without relevant context, NQ Open provides real user queries requiring short answers, and Flan QA also includes short question-answer pairs specifically designed for instruction-tuning language models. These datasets provide a diverse range of questions and answers, allowing us to assess the robustness of our method across various domains and question types. The selection of these datasets enables us to evaluate SEU’s performance on both traditional QA tasks as well as more recent instruction-following scenarios.

Baselines Following the evaluation by Kuhn et al. (), we compare our SEU method against the following baselines: Predictive Entropy which is just the average predictive entropy of all the tokens in the sequence, Length-normalised Predictive Entropy which is the joint log-probability of each sequence divided by the length of the sequence (Malinin & Gales, )(^2^22This technically should be called length-normalised log-likelihood, but we follow prior work on using this name here.), and Semantic Entropy (Kuhn et al., ). For predicting bidirectional entailment in the Semantic Entropy baseline, we use DeBERTaLarge (He et al., ) as used in Kuhn et al. (), while for our SEU method, we utilise the sentence-BERT (Reimers & Gurevych, ) for semantic embeddings.
Following prior work (Kuhn et al., ; Duan et al., ), we use the Area Under the Receiver Operating Characteristic curve (AUROC) as our primary evaluation metric. This metric treats uncertainty estimation as the problem of predicting whether to rely on a model’s generation for a given context. We evaluate the correctness of our model’s generations using a fuzzy matching criterion based on the Rouge-L score, considering an answer correct if its Rouge-L score with respect to the reference answer is larger than 0.3. Our experimental procedure involves computing uncertainty scores using our proposed SEU method and the baselines for each generated response, evaluating their correctness, and calculating AUROC scores. All experiments were done using one NVIDIA A100 GPU.

### 4.2 Results

Our empirical evaluation demonstrates the effectiveness of the proposed Semantic Embedding Uncertainty (SEU) method across different models and datasets. We present our findings in two parts: a comparative analysis of uncertainty estimation methods and an in-depth examination of the trade-off between false positive rate (FPR) and true positive rate (TPR).

#### 4.2.1 Comparative Analysis of Uncertainty Estimation Methods

Figure 1 presents the AUROC scores for different uncertainty estimation methods across three models (Llama-3.1-8B-Instruct, Phi-3.5-Instruct, and Mistral-7B-Instruct) and three datasets (TriviaQA, NQ Open, and Flan QA). We note the proposed SEU method consistently outperforms or matches the performance of other uncertainty estimation methods across all model-dataset combinations. Specifically, while the relative performance of methods varies slightly across models, SEU maintains its advantage, suggesting robustness to model architecture differences. The performance patterns differ across datasets, with all methods generally performing better on TriviaQA compared to NQ Open and Flan QA. Crucially, SEU consistently outperforms Semantic Entropy, supporting our hypothesis that the latter may overestimate uncertainty due to its sensitivity to minor linguistic variations.

Figure: Figure 1: Comparison of SEU method against baselines across different models and datasets.
Refer to caption: x1.png

#### 4.2.2 Analysis of False Positive Rate and True Positive Rate Trade-off

To further investigate the performance difference between SEU and Semantic Entropy, we analyse the False Positive Rate (FPR) and True Positive Rate (TPR) at the optimal Youden’s J statistic point for the NQ Open dataset. A “positive” case refers to an instance where the model’s response is correct. Table 2 presents these results. Notably, SEU consistently achieves a higher TPR compared to Semantic Entropy across all models. This indicates that SEU is more effective at identifying the cases when the LLM is confident about the underlying semantics. The improved TPR of SEU comes with a slight increase in FPR. However, the gain in TPR (ranging from 0.0897 to 0.1785) outweighs the increase in FPR (ranging from 0.0460 to 0.1037), resulting in better overall performance as reflected in the AUROC scores.

**Table 2: Comparison of Semantic Embedding Uncertainty and Semantic Entropy on NQ Open Dataset at Optimal Youden’s J Statistic Point**
| Model | SEU (Ours) | SE |  |  |
| --- | --- | --- | --- | --- |
| FPR [$\downarrow$] | TPR [$\uparrow$] | FPR [$\downarrow$] | TPR [$\uparrow$] |  |
| Llama 3.1 8B | 0.2608 | 0.7767 | 0.1655 | 0.6543 |
| Phi 3.5 | 0.2198 | 0.7571 | 0.1738 | 0.6674 |
| Mistral 7B | 0.3293 | 0.7353 | 0.2256 | 0.5568 |

This empirical evidence supports our argument that Semantic Entropy may overestimate uncertainty as demonstrated by its significantly lower TPR. Lower TPR suggests that Semantic Entropy is classifying more cases as uncertain, even when the model’s response is correct. The higher TPR of SEU suggests that it captures a more nuanced view of semantic similarity, allowing it to identify truly uncertain cases more accurately.

## 5 Amortised Semantic Embedding Uncertainty

While our proposed Semantic Embedding Uncertainty (SEU) method demonstrates superior performance in uncertainty estimation across various models and datasets, it shares a significant limitation with Semantic Entropy: computational inefficiency. Both SEU and Semantic Entropy require multiple forward passes through the language model to generate a set of responses for each input, which can be prohibitively expensive, especially for large language models in production environments. To this end, we present amortised SEU (ASEU) to tackle the challenge of estimating semantic uncertainty in a single forward pass. The goal is to represent the semantics of a sequence as latent variables and spend a small effort to finetune and obtain an amortised approximate posterior over them to bypass the need for an external paragraph or sentence embedding model at test time.

### 5.1 Latent semantic model

Suppose we have a training set of $N$ sequences and ${\mathbf{x}}_{n}=(x_{n,1},x_{n,2},\dots,x_{n,T})$ is the $n$-th sequence that has $T$ tokens. We assume there is a latent vector ${\mathbf{z}}_{n}\in{\mathbb{R}}^{D}$ that captures the semantic of the $n$-th sequence and that the embeddings ${\mathbf{e}}_{n}\in{\mathbb{R}}^{D}$ of this sequence can be computed using an external, pre-trained embedding model. The joint distribution over the latent semantic and observed embeddings is defined as follows,

$$ $\displaystyle p(\{{\mathbf{e}}_{n},{\mathbf{z}}_{n}\}_{n=1}^{N}|\omega)=\prod_ {n=1}^{N}p({\mathbf{z}}_{n})p({\mathbf{e}}_{n}|{\mathbf{z}}_{n},\omega),$ $$

We choose a standard normal prior over ${\mathbf{z}}_{n}$, $p({\mathbf{z}}_{n})={\mathcal{N}}({\mathbf{z}}_{n};{\bm{0}},{\mathbf{I}}_{D})$ and $p({\mathbf{e}}_{n}|{\mathbf{z}}_{n},\omega)={\mathcal{N}}({\mathbf{e}}_{n};
\mathrm{NN}_{\omega}({\mathbf{z}}_{n}),\sigma_{e}^{2}{\mathbf{I}}_{D})$, where $\mathrm{NN}_{\omega}$ is a small neural network with parameters $\omega$. It is worth noting that while modelling ${\mathbf{z}}_{n}$ at every time step is possible, this would involve specifying a dynamic prior mapping from ${\mathbf{z}}_{n,t-1}$ to ${\mathbf{z}}_{n,t}$ and computing the embeddings for all subsequences $(x_{n,1},x_{n,2},\dots,x_{n,t})$. We opt for simplicity and pick a global ${\mathbf{z}}$ for the whole sequence.

### 5.2 Approximate inference

Readers familiar with latent variable modelling might have noted similarity between the model above and Gaussian latent variable models in Kingma & Welling (); Rezende et al. (). The most natural next step for inference would be to impose an approximate posterior over ${\mathbf{z}}_{n}$, $q({\mathbf{z}}_{n}|{\mathbf{e}}_{n},\psi)$, that mirrors that of the exact posterior, $p({\mathbf{z}}_{n}|{\mathbf{e}}_{n},\omega)$. While this is arguably the most accurate approach, computing ${\mathbf{z}}$ for a new sequence at test time requires access to the embedding ${\mathbf{e}}$, which we seek to avoid. To sidestep this, we posit the variational Gaussian distribution over ${\mathbf{z}}$, $q({\mathbf{z}}_{n}|{\mathbf{x}}_{n},\psi,\theta)={\mathcal{N}}({\mathbf{z}}_{n
};\mu_{n},\Sigma_{n})$, where $\mu_{n}$ and $\Sigma_{n}$ are outputs of a fully-connected neural network, parameterised by $\psi$. This network takes as input the representation of ${\mathbf{x}}_{n}$ provided by the language model, that is parameterised by $\theta$. This parameterisation allows us to obtain a distribution over ${\mathbf{z}}$ using the same backbone as used for modelling ${\mathbf{x}}$. Equipped with the model and variational distribution specifications, we now wish to minimise the $\mathrm{KL}$ divergence between the approximate posterior and the true posterior, $\mathrm{KL}[q({\mathbf{z}}_{n}|{\mathbf{x}}_{n},\psi,\theta)\;||\;p({\mathbf{z
}}_{n}|{\mathbf{e}}_{n},\omega)]$, or equivalently, minimising the negative lower bound to the log marginal likelihood $\log p(\{{\mathbf{e}}_{n}\}_{n=1}^{N}|\omega)$, ${\mathcal{L}}(\theta,\omega,\psi)=\sum_{n}{\mathcal{L}}_{n}(\theta,\omega,\psi)$, where

$$ $\displaystyle{\mathcal{L}}_{n}(\theta,\omega,\psi)$ $\displaystyle=\int_{{\mathbf{z}}_{n}}q({\mathbf{z}}_{n}|{\mathbf{e}}_{n},\psi, \theta)\left[\log q({\mathbf{z}}_{n}|{\mathbf{e}}_{n},\psi,\theta)-\log p({ \mathbf{e}}_{n},{\mathbf{z}}_{n}|\omega)\right]$ $\displaystyle=\mathrm{KL}[q({\mathbf{z}}_{n}|{\mathbf{e}}_{n},\psi,\theta)\;|| \;p({\mathbf{z}}_{n})]-\int_{{\mathbf{z}}_{n}}q({\mathbf{z}}_{n}|{\mathbf{x}}_ {n},\psi,\theta)\log p({\mathbf{e}}_{n}|{\mathbf{z}}_{n},\omega).$ $$

The above objective is intuitive: we want to fine-tune the language model and optimise the model and variational parameters such that the language model’s representation helps reconstruct the embedding of the full sequence. Additionally, $\theta$ can be kept fixed if a pre-trained language model is already available or simultaneously fine-tuned using the negative log-likelihood of ${\mathbf{x}}$ as in conventional autoregressive language modelling.

### 5.3 Uncertainty estimation at test time

As the variational approximation is trained to approximately imitate the sequence embedding, it can be leveraged to estimate the semantic uncertainty similarly to the SEU method proposed earlier. At each step $t$ during response generation, we draw $K$ samples $\{{\mathbf{z}}_{t,1},\ldots,{\mathbf{z}}_{t,K}\}$ from the approximate posterior $q({\mathbf{z}}_{t}|{\mathbf{x}}_{t},\psi,\theta)$, where ${\mathbf{x}}_{t}$ are the prompt and the tokens generated so far. We then compute the average pairwise cosine similarity between these samples:

$$ $\displaystyle S_{t}=\frac{2}{K(K-1)}\sum_{j=1}^{K-1}\sum_{k=j+1}^{K}\cos({ \mathbf{z}}_{t,j},{\mathbf{z}}_{t,k})$ $$

where $\cos({\mathbf{z}}_{t,j},{\mathbf{z}}_{t,k})$ denotes the cosine similarity between samples ${\mathbf{z}}_{t,j}$ and ${\mathbf{z}}_{t,k}$. After generating the complete response of length $T$, we calculate the raw ASEU score: $\text{ASEU}_{\text{raw}}=1-\text{median}\{S_{1},\ldots,S_{T}\}$. The intuition behind this approach is that if the LLM is uncertain about the latent semantics of future tokens, the sampled embeddings at each step will have lower average similarity compared to cases where the LLM is more certain. Taking the median across all steps yields a robust measure of the overall semantic uncertainty for the entire response.
To account for the impact of response length on uncertainty, we apply length normalization, defining our final ASEU score, with higher values indicating greater uncertainty and lower values indicating greater certainty. This normalization step is crucial as it mitigates potential bias towards longer responses, which might accumulate more uncertainty simply due to their length. We also found the proposed approach is more robust than using the entropy of the variational approximation.

## 6 Empirical Evaluation of Amortized SEU

In this section, we empirically evaluate our proposed amortized Semantic Embedding Uncertainty (ASEU) method. Unlike the previous multi-pass setting, we now focus on estimating uncertainty in a single forward pass, which is crucial for practical applications in production environments.

### 6.1 Experimental Setup

Models: We use the same three models as in the previous experiments: Llama-3.1-8B-Instruct, Phi-3.5-Instruct, and Mistral-7B-Instruct. However, for this evaluation, we fine-tune these models to optimize the variational objective presented in section 5.2.

Fine-tuning: To learn the approximate posterior distribution $q({\mathbf{z}}|{\mathbf{x}},\psi,\theta)$ and parameters $\theta$ and $\omega$, we fine-tune the LLMs on the TriviaQA dataset. We chose TriviaQA for fine-tuning due to its size and diverse coverage of question-answering tasks and its focus on short answers, which aligns with the paper’s emphasis so far. This fine-tuning process allows the models to leverage the underlying language model to estimate semantic uncertainty in a single forward pass.

Baselines: In the single forward pass setting, we compare our ASEU method against the length-normalized predictive entropy of a single forward pass response. This baseline is chosen as it is the most relevant uncertainty estimation method that can be computed in a single pass.

### 6.2 Results and Discussion

Figure 2 presents the AUROC scores for our ASEU method and the length-normalized predictive entropy baseline across the three models (Llama-3.1-8B-Instruct, Phi-3.5-Instruct, and Mistral-7B-Instruct) and two datasets (NQ Open and Flan QA). We also compare the armotised uncertainty estimates with SEU and other methods that require multiple generations in figure 3.
We note that ASEU consistently outperforms the length-normalized predictive entropy baseline across all model-dataset combinations, suggesting it captures more meaningful uncertainty information. While ASEU generally doesn’t match the performance of multi-pass SEU method, it achieves comparable results for the Mistral model. The performance gap between ASEU and multi-pass SEU varies across models and datasets, but the computational efficiency gained through single-pass estimation makes ASEU more suitable for real-world applications, especially in production environments where multiple forward passes are infeasible.

Figure: Figure 2: Comparison of amortised SEU method against log-likelihood in a single forward pass setting across different models and datasets.
Refer to caption: x2.png

Figure: Figure 3: Comparison of amortised SEU method against techniques requiring multiple forward passes across different models and datasets.
Refer to caption: x3.png

### 6.3 Analysis of Learnt Latent Embeddings

To demonstrate that the latent embeddings of our amortized model carry meaningful semantic information, we examined the cosine similarities between the means of the variational distributions given semantically related queries and presented the results in Table 3. The perfect similarity between queries about England’s capital and the UK’s biggest city reflects their close relationship (both referring to London). The lower but consistent similarity (0.83) between these queries and a question about Australia’s capital shows that the embeddings capture both the semantic structure of the questions (asking about capital/major cities) and the distinction between different locations. This suggests that the learnt embeddings after finetuning the LLMs encodes semantic relationships.

**Table 3: Cosine Similarities Between Predicted Embeddings of the following Query Embeddings**
| Query 1 | Query 2 | Cosine Similarity |
| --- | --- | --- |
| What is the capital of England? | What is the biggest city in the UK? | 1.00 |
| What is the capital of England? | What is the capital of AUS? | 0.83 |
| What is the biggest city in the UK? | What is the capital of AUS? | 0.83 |

## 7 Related Works

Hallucinations in LLMs: The challenge of hallucination detection in LLMs has become increasingly important as these models are deployed in real-world applications. Various benchmarks have been developed to evaluate this phenomenon, including TruthfulQA (Lin et al., ), FactualityPrompt (Lee et al., ), FActScore (Min et al., ), HaluEval (Li et al., ), and FACTOR (Muhlgay et al., ). Early research on hallucinations primarily focused on issues in summarization tasks, where models would generate content unfaithful to the source text (Maynez et al., ; Durmus et al., ; Wang et al., ). This work laid the foundation for understanding the broader challenge of hallucinations in LLMs.

Uncertainty Estimation Approaches: A significant body of work has explored methods to estimate uncertainty in LLM outputs. Many of these approaches rely on comparing multiple model generations or outputs by leveraging additional LLMs or by using the same LLM (Duan et al., ; Chen & Mueller, ; Manakul et al., ; Mündler et al., ). The field has seen a variety of innovative techniques, including those proposed by Kadavath et al. (), Mitchell et al. (), and Xu et al. (), which leverage different aspects of model behaviour to gauge uncertainty.

Knowledge Integration Methods: Another line of research focuses on integrating external knowledge to verify and improve the factual accuracy of LLM outputs. The RARR framework (Gao et al., ) uses search engines for knowledge retrieval and correction. Similarly, the Verify-and-Edit approach (Zhao et al., ) leverages external information sources. However, these methods face challenges in resolving conflicts between model knowledge and retrieved information, as highlighted by Shi et al. (). Additional work in this area includes efforts by Dziri et al. (), Peng et al. (), and Li et al. (), who explore various techniques for grounding LLM outputs in external knowledge sources.

Generation and Fine-tuning Strategies: Researchers have also developed strategies to reduce hallucinations during the generation process or through model fine-tuning. Lee et al. () introduced factual-nucleus sampling to balance output diversity and factual accuracy. Reinforcement learning from human feedback (RLHF) has been employed by Ouyang et al. () and Touvron et al. () to align LLMs with desired criteria, including truthfulness. Other approaches include careful curation of instruction-tuning data (Zhou et al., ) and linguistic calibration techniques (Mielke et al., ). Recent work by Tian et al. () has further explored fine-tuning strategies specifically targeting factuality improvement.

Leveraging the Latent Space: An emerging area of research investigates the internal representations of LLMs to understand and manipulate their behaviour. Studies have suggested the existence of a “truthfulness” direction in the latent space of these models. For example, Li et al. () proposed Inference-Time Intervention to identify and modify factuality-related directions in model activations. Azaria & Mitchell () introduced SAPLMA, suggesting that LLMs may have an internal awareness of their own inaccuracies. This line of inquiry has been further developed by Burns et al. (), who explored methods for discovering latent knowledge, and Marks & Tegmark (), who examined the geometry of truth representations in LLMs. Additional insights have been provided by Subramani et al. () and Zou et al. (), who have explored techniques for understanding and manipulating the internal representations of these models. Kossen et al. () further propose to directly predict the semantc entropy using probes acting on different hidden layers of the LLM.

## 8 Conclusion

This work introduces two novel approaches for uncertainty quantification in large language models: Semantic Embedding Uncertainty (SEU) and its amortized version (ASEU). Our methods leverage semantic embeddings to achieve more robust and nuanced estimations of semantic uncertainty compared to existing techniques. While SEU provides improved accuracy over traditional approaches, ASEU offers a significant computational advantage by enabling uncertainty estimation in a single forward pass. This efficiency is particularly crucial for real-time applications and when dealing with larger language models where multiple forward passes can be prohibitively expensive. Empirical evaluations across multiple datasets and frontier LLMs demonstrate that our embedding-based methods provide more accurate uncertainty quantification than traditional approaches, particularly in scenarios where minor linguistic variations or additional correct information might lead to overestimation of uncertainty. Furthermore, ASEU’s ability to maintain comparable performance to SEU while drastically reducing computational overhead represents a substantial step towards making uncertainty quantification more practical and accessible in production environments.

While our results are promising, several limitations of this work should be acknowledged. Our experimental setup primarily focused on short-answer questions and responses, which may not fully capture the complexity and diversity of real-world LLM applications that often involve longer, more nuanced responses. The use of Rouge-L score as an automatic evaluation metric, while suitable for short answers, may not be appropriate for assessing longer or more complex responses. This limitation restricts the generalisability of our findings to broader LLM use cases. Additionally, while we used multiple datasets, they were all in the domain of question-answering. The effectiveness of our methods on other types of language tasks, such as code generation, remains to be explored. Our study also focused on a specific set of commonly used open source LLMs, and the performance and behaviour of our methods on larger models were not investigated.

## Appendix A Appendix

### A.1 ROC curves

We provide the full ROC curves of the methods considered in the main text, across various models and evaluation datasets.

Figure: Figure 4: ROC Curve for Llama-3.1-8B-Instruct model on TriviaQA dataset
Refer to caption: x4.png

Figure: Figure 5: ROC Curve for Phi-3.5-Instruct model on TriviaQA dataset
Refer to caption: x5.png

Figure: Figure 6: ROC Curve for Mistral-7B-Instruct model on TriviaQA dataset
Refer to caption: x6.png

Figure: Figure 7: ROC Curve for Llama-3.1-8B-Instruct model on NQ Open dataset
Refer to caption: x7.png

Figure: Figure 8: ROC Curve for Phi-3.5-Instruct model on NQ Open dataset
Refer to caption: x8.png

Figure: Figure 9: ROC Curve for Mistral-7B-Instruct model on NQ Open dataset
Refer to caption: x9.png

Figure: Figure 10: ROC Curve for Llama-3.1-8B-Instruct model on Flan QA dataset
Refer to caption: x10.png

Figure: Figure 11: ROC Curve for Phi-3.5-Instruct model on Flan QA dataset
Refer to caption: x11.png

Figure: Figure 12: ROC Curve for Mistral-7B-Instruct model on Flan QA dataset
Refer to caption: x12.png

### A.2 Model Prompts and Example Responses

This section presents the prompts used for each model and provides examples of their responses to demonstrate the brevity of the generated answers.

#### A.2.1 Prompts

For the initial experiments, we used the following prompts:

- •
Llama and Phi models: “Answer the following question as briefly as possible”
- •
Mistral model: “Answer the following question briefly using a few words”

After fine-tuning, we used the same prompts except for the Llama model, where we changed the prompt to: “Give a short reply to the following question”.

#### A.2.2 Example Responses

Tables 4 and 5 include two examples from the NQ Open dataset to demonstrate that the models indeed generate short answers, adhering to the instruction for brevity.

**Table 4: Example responses from different models to the question ”When was the first Hunger Games book published?”**
| Model | Responses |
| --- | --- |
| Llama | •<br>September 14, 2008<br>•<br>September 14, 2008<br>•<br>September 14, 2008.<br>•<br>September 14, 2008.<br>•<br>September 14, 2008 |
| Mistral | •<br>2008 (The Hunger Games)<br>•<br>2008 (The Hunger Games)<br>•<br>2008 (The Hunger Games)<br>•<br>2008<br>•<br>2008 (The Hunger Games) |
| Phi | •<br>2009<br>•<br>2009<br>•<br>2009<br>•<br>2009<br>•<br>2009 |

**Table 5: Example responses to the question ”What is the location of the Ten Commandments in the Bible?”**
| Model | Responses |
| --- | --- |
| Llama | •<br>Exodus 20:1-17 and Deuteronomy 5:6-21.<br>•<br>Exodus 20:1-17 and Deuteronomy 5:6-21.<br>•<br>Exodus 20:1-17 and Deuteronomy 5:6-21.<br>•<br>Exodus 20:1-17 and Deuteronomy 5:6-21.<br>•<br>Exodus 20:1-17 and Deuteronomy 5:6-21. |
| Mistral | •<br>Exodus 20 (King James Version)<br>•<br>Exodus 20 (King James Version)<br>•<br>Exodus 20:1-17<br>•<br>Exodus 20 (King James Version)<br>•<br>Exodus 20 (Old Testament)<br>•<br>Exodus 20 (King James Version) |
| Phi | •<br>Exodus 20:1-17<br>•<br>Exodus 20:1-17 in the Old Testament<br>•<br>Exodus 20:1-17<br>•<br>Exodus (Exodus 20:1-17) and Deuteronomy (Deuteronomy 5:6-2)”<br>•<br>Exodus 20:1-17 |

In the first example, while Llama and Mistral provide the correct publication year (2008) for ”The Hunger Games”, Phi consistently gives an incorrect year (2009). This illustrates both the models’ ability to generate brief responses and the potential for factual inaccuracies in their outputs.

In the second example, all models correctly identify Exodus 20 as a location for the Ten Commandments. Llama consistently provides both locations (Exodus and Deuteronomy), while Mistral and Phi primarily focus on Exodus. This demonstrates the models’ capacity to provide accurate, concise information, with some variation in the level of detail provided.

These examples highlight the instruction tuned models adherence to the brevity instruction while showcasing differences in their knowledge and response patterns.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see my comments on the Weakness section.
soundness: 2
strengths: 1. **Important problem**: The paper focuses on an important and valuable problem: quantifying the uncertainty of large language models.

2. **Efficient method**: The authors propose an amortized version of SEU that models the underlying semantics as latent variables. This approach requires only a single forward pass but still demonstrates very strong performance.

3. **Experimental validation**: The authors conduct experiments on three QA datasets across three LLMs, demonstrating the effectiveness of the proposed SEU and ASEU methods.
summary: This paper addresses the problem of quantifying uncertainty in large language models. Specifically, previous methods, such as Semantic Entropy (SE), use bidirectional entailment criteria to assess whether two responses share the same meaning. However, the authors argue that bidirectional entailment criteria are sensitive to linguistic variations. To address this, they propose SEU, which quantifies uncertainty using cosine similarities between the semantic embeddings of different responses. Additionally, since both SE and SEU require multiple forward passes, the authors introduce an amortized version of SEU that models the underlying semantics as latent variables, requiring only a single forward pass while still achieving strong results.
weaknesses: 1. **Reliability of cosine similarity of embeddings as an absolute measure of semantic relatedness**: I have some concerns about using cosine similarity as an absolute measure to describe the similarity between two responses. As shown in [1], the embedding space is isotropic, meaning that some texts are very close together within a narrow cone, while others are far apart. Thus, cosine similarity may not function well as an absolute measure.

2. **Advantages of SEU compared to SE**: The authors claim that SE models are sensitive to linguistic variability and minor wording differences, whereas SEU models are more robust and focus on the underlying semantic content. However, the SE method uses NLI classifiers that take the combined input of both responses, allowing for more fine-grained interaction between them. In contrast, SEU encodes responses independently, lacking such fine-grained interaction. As such, I find the argument for SEU being superior to SE unconvincing. Additionally, the experiments are conducted on a single model for both SE and SEU, which may not reveal whether the observed advantages are due to model choice or differences in methodology.

3. **Bidirectional entailment criterion can only act as a binary measure**: The authors claim that bidirectional entailment criteria can only yield binary outcomes. However, since the criterion relies on an entailment classification, the output probability of the entailment class has the potential to express the degree of relatedness between different responses.

[1] Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, Tie-Yan Liu. Representation Degeneration Problem in Training Natural Language Generation Models. ICLR 2019.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. Generalization to Other NLP Tasks?  The SEU method has been validated primarily for question-answering tasks. Could the authors discuss the potential effectiveness and generalization of the SEU method to other natural language processing tasks such as text summarization or machine translation, where the context and requirements might differ significantly?

2. Computational Resources in Model Training? While ASEU is praised for reducing computational overhead during inference, what are the computational resource requirements during the model training phase, particularly when dealing with the inference and optimization of latent variables? How does this compare to traditional methods in terms of efficiency?

3. Reliability on Pre-trained Models? The SEU method's performance seems to heavily rely on the quality of the pre-trained language models for generating semantic embeddings. How sensitive is the SEU method to potential biases or inaccuracies in these pre-trained models, and what measures can be taken to mitigate such issues?

4. Diversity in Model Size Analysis? Figure 1, along with other figures, compares uncertainty estimation methods across a limited range of model sizes. Could the authors expand on the analysis to include a more diverse set of model sizes, especially very large models (e.g., 70 billion parameters) and smaller models (e.g., 3 billion parameters), to provide a comprehensive understanding of how the proposed methods scale and perform across different model sizes?

5. Analysis of Close Model Variations? It would be insightful to see an analysis that includes close variations of the same model architecture but with different sizes. How does the performance of the uncertainty estimation methods vary with model size while keeping the architecture constant? What trends can be observed that would help us understand the behavior of SEU and ASEU in practical applications with different model sizes?
soundness: 2
strengths: 1 The paper presents a novel approach to uncertainty quantification in large language models (LLMs) by introducing Semantic Embedding Uncertainty (SEU) and its amortized version (ASEU). These methods innovatively leverage semantic embeddings to address the limitations of traditional uncertainty estimation techniques, offering a more nuanced understanding of semantic uncertainty without relying on strict bidirectional entailment criteria.
2 The research demonstrates high quality through  empirical evaluation across multiple question-answering datasets and state-of-the-art LLMs. The paper provides a good comparison against existing methods, showcasing SEU's superior performance in accurately quantifying uncertainty. 
3 The paper is well-structured and clearly articulates the problem, the proposed solutions, and their significance.
summary: This paper introduce, the technical challenges include addressing "hallucinations" in LLMs, where models generate coherent but incorrect responses, and the difficulty of applying traditional uncertainty quantification methods to the open-ended nature of natural language generation. Our innovations introduce Semantic Embedding Uncertainty (SEU) to estimate semantic uncertainty more accurately by leveraging semantic embeddings, and an amortized version (ASEU) that reduces computational costs by estimating uncertainty in a single forward pass.
weaknesses: 1. Generalization Limitations: The SEU method proposed in this paper has mainly been experimentally validated for question-answering tasks, and its effectiveness and generalization ability in other types of natural language processing tasks, such as text summarization or machine translation, have not been fully tested.
2. Computational Resource Consumption: Although the amortized SEU (ASEU) reduces computational overhead, during the model training phase, especially when it involves inference and optimization of latent variables, it may still require relatively high computational resources.
3. Dependence on Pre-trained Models: The SEU method relies on high-quality pre-trained language models to generate semantic embeddings. If the pre-trained models themselves have biases or inaccuracies, it could affect the accuracy and reliability of the SEU method.
4. Figure 1,2,3,  model size diversity analyses needed.  While the paper provides a comparison of uncertainty estimation methods across different models, it would be beneficial to include models of varying sizes to get a more comprehensive understanding of how the proposed methods scale. The current selection of models does not fully represent the spectrum of LLMs, particularly the very large models (e.g., 70 billion parameters) or smaller models (e.g., 3 billion parameters). Including a broader range of model sizes could provide insights into the scalability and generalizability of the SEU and ASEU methods.
5 . Close Model Variations analysing needed.  It would be insightful to see comparisons that include close variations of the same model architecture but with different sizes. This could help elucidate whether the performance of the uncertainty estimation methods is more dependent on the model architecture or its size. Such an analysis could reveal trends that are crucial for understanding the behavior of SEU and ASEU in different practical scenarios.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. in line 185, "cosine similarity provides a continuous metric". In fact, the bidirectional entailment can also offer continuous scores by using the output probability (or logit). 
2. ROUGE-L cannot capture the semantic equivalence well and is insensitive to word orders. The evaluation step needs further improvement by using better metrics such as BERTScore and LLM-as-Judge (though stated in the conclusion)
3. It is important to report multiple metrics for uncertainty estimation, e.g., Brier and ECE.
4. a key baseline is missing: INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection (ICLR 2024), which develops an EigenScore metric to measure the semantic consistency in the embedding space.  
5. In Table 2, it is hard to see the benefit of SEU since it leads to a worse FPR
soundness: 2
strengths: 1. the paper is well-written with a clear storyline
2. the Amortised SEU is able to estimate uncertainty within a single forward pass,  which is efficient and interesting
summary: This paper identifies that bidirectional entailment used for semantic uncertainty is sensitive to minor variations in responses. 
To address this issue, the authors propose a refined method, semantic embedding uncertainty (SEU), which is based on the average pairwise cosine similarity of the generated responses’ embeddings. The SEU consistently outperforms other baselines on three short-answer datasets. However, SEU still requires multiple runs of LLMs to collect a set of responses. To estimate uncertainty within a single forward pass, Amortised SEU is presented, which introduces a latent variable model to impute latent semantic embeddings with regard to the embeddings of an input.  Experimental results show ASEU achieves better performance than another single forward-pass baseline, length normalized predictive entropy.
weaknesses: 1. Apart from the Amortised SEU, I find that the scientific contribution of SEU (average pairwise cosine similarity) is not sufficient. 
First, the proposed SEU is a simple adapted version of the original semantic uncertainty. Second,   The authors claim that bidirectional entailment cannot provide continuous scores, but it is doable to use the probability of the NLI model for this purpose. I would suggest adding another baseline that replaces cosine similarity in SEU with the NLI scores. 
2. The experimental setting should be improved. First, it is necessary to compare SEU to other embedding-based methods, e.g., INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection (ICLR 2024). Second, it is useful to report multiple metrics for evaluating uncertainty methods, e.g., Brier and ECE. Last, even though the authors mention the problem of the ROUGE-L metric and only short-answer datasets, I would believe that it is easy to mitigate these issues, e.g., Bert score and evaluation of long-form questions like TruthfulQA
3. Amortised SEU required further fine-tuning while other baselines are unsupervised. The generality of ASEU is not clear

Meta-review: This paper is interested in improving uncertainty quantification for large language models. The idea is to obtain better estimates by integrating an improved version of semantic entropy. This permits taking into account a more accurate notion of uncertainty compared to either (i) just trivially examining how different responses are or (ii) using bidirectional entailment. The authors introduce some improved mechanisms and then study the performance of these approaches, including an efficient version that is less computationally expensive. The main idea is to use cosine similarity between emebddings.

As strong points, the paper provides a very simple solution that directly offers improvements. This is always nice to see.

As weaknesses, my thoughts are the following. First, this paper studies a relatively narrow area; effectively, it is searching for mechanisms to improve a component of semantic entropy. To do so, the authors propose the most straightforward approach. This is not by itself bad, but it does make me think there is much more to do and learn in this direction.

As also noted by the reviewers, cosine similarity can be tricky to reliably use. This is not a problem because there are a ton of alternatives (or combinations) that could be used here, and I would like to see more and understand the pros and cons. In fact, I would love to see a combination of SE with the authors’ idea, since they should be complementary, each handling cases the other does poorly on. In addition, evaluation was a good start but could be far more extensive (and this should be relatively easy).

Ultimately I felt that the paper was on the right track, but there’s much more to add to make it a strong paper that passes the bar.

Decision: Reject