Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs

## Abstract

Abstract We propose semantic entropy probes (SEPs), a cheap and reliable method for uncertainty quantification in Large Language Models (LLMs).
Hallucinations, which are plausible-sounding but factually incorrect and arbitrary model generations, present a major challenge to the practical adoption of LLMs.
Recent work by Farquhar et al. [ 21 ] proposes semantic entropy (SE), which can detect hallucinations by estimating uncertainty in the space semantic meaning for a set of model generations.
However, the 5-to-10-fold increase in computation cost associated with SE computation hinders practical adoption.
To address this, we propose SEPs, which directly approximate SE from the hidden states of a single generation.
SEPs are simple to train and do not require sampling multiple model generations at test time, reducing the overhead of semantic uncertainty quantification to almost zero.
We show that SEPs retain high performance for hallucination detection and generalize better to out-of-distribution data than previous probing methods that directly predict model accuracy.
Our results across models and tasks suggest that model hidden states capture SE, and our ablation studies give further insights into the token positions and model layers for which this is the case.

## 1 Introduction

Large Language Models (LLMs) have demonstrated impressive capabilities across a wide variety of natural language processing tasks [, , , , ].
They are increasingly deployed in real-world settings, including in high-stakes domains such as medicine, journalism, or legal services [, , , ].
It is therefore paramount that we can *trust* the outputs of LLMs.
Unfortunately, LLMs have a tendency to *hallucinate*.
Originally defined as “content that is nonsensical or unfaithful to the provided source” [, , ], the term is now used to refer to nonfactual, arbitrary content generated by LLMs.
For example, when asked to generate biographies, even capable LLMs such as GPT-4 will often fabricate facts entirely [, , ].
While this may be acceptable in low-stakes use cases, hallucinations can cause significant harm when factuality is critical.
The reliable detection or mitigation of hallucinations is a key challenge to ensure the safe deployment of LLM-based systems.

Various approaches have been proposed to address hallucinations in LLMs (see Section 2).
An effective strategy for detecting hallucinations is to sample multiple responses for a given prompt and check if the different samples convey the same meaning [, , , , , , , , ].
The core idea is that if the model knows the answer, it will consistently provide the same answer.
If the model is hallucinating, its responses may vary across generations.
For example, given the prompt “What is the capital of France?”, an LLM that “knows” the answer will consistently output (Paris, Paris, Paris), while an LLM that “does not know” the answer may output (Naples, Rome, Berlin), indicating a hallucination.

One explanation for why this works is that LLMs have calibrated uncertainty [, ], i.e., “language models (mostly) know what they know” []. When an LLM is certain about an answer, it consistently provides the correct response. Conversely, when uncertain, it generates arbitrary answers. This suggests that we can leverage model uncertainty to detect hallucinations.
However, we cannot use token-level probabilities to estimate uncertainty directly because different sequences of tokens may convey the same meaning.
For the example, the answers “Paris”, “It’s Paris”, and “The capital of France is Paris” all mean the same.
To address this, Farquhar et al. [] propose *semantic entropy* (SE), which clusters generations into sets of equivalent meaning and then estimates uncertainty in semantic space.

Figure: Figure 1: Semantic entropy probes (SEPs) outperform accuracy probes for hallucination detection with Llama-2-7B, although there is a gap to $10$x costlier baselines. (See Sec. 5.)
Refer to caption: x1.png

A major limitation of SE and other sampling-based approaches is that they require multiple model generations for each input query, typically between 5 and 10.
This results in a 5-to-10-fold higher cost compared to naive generation without SE, presenting a major hurdle to the practical adoption of these methods.
Computationally cheaper methods for reliable hallucination detection in LLMs are needed.

The hidden states of LLMs are a promising avenue to better understand, predict, and steer a wide range of LLM behaviors [, , ].
In particular, a recent line of work learns to predict the truthfulness of model responses by training a simple linear probe on the hidden states of LLMs. Linear probes are computationally efficient, both to train and when used at inference.
However, existing approaches are usually supervised [, , , ] and therefore require a labeled training dataset assigning accuracy to statements or model generations.
And while unsupervised approaches exist [], their validity has been questioned [].
In this paper, we argue that supervising probes via *SE* is preferable to accuracy labels for robust prediction of truthfulness.

We propose *Semantic Entropy Probes* (SEPs), linear probes that capture semantic uncertainty from the hidden states of LLMs, presenting a cost-effective and reliable hallucination detection method.
SEPs combine the advantages of probing and sampling-based hallucination detection.
Like other probing approaches, SEPs are easy to train, cheap to deploy, and can be applied to the hidden states of a single model generation.
Similar to sampling-based hallucination detection, SEPs capture the *semantic uncertainty* of the model.
Furthermore, they address some of the shortcomings of previous approaches.
Contrary to sampling-based hallucination detection, SEPs act directly on a *single* model hidden state and do not require generating multiple samples at test time.
And unlike previous probing methods, SEPs are trained to predict *semantic entropy* [] rather than model accuracy, which can be computed without access to ground truth accuracy labels that can be expensive to curate.

We find that SEP predictions are effective proxies for truthfulness.
In fact, SEPs generalize better to new tasks than probes trained directly to predict accuracy, setting a new state-of-the-art for cost-efficient hallucination detection, cf. Fig. 1.
Our results additionally provides insights into the inner workings of LLMs, strongly suggesting that model hidden states directly capture the model’s uncertainty over semantic meanings.
Through ablation studies, we show that this holds across a variety of models, tasks, layers, and token positions.

In summary, our core contributions are:

- •
We propose Semantic Entropy Probes (SEPs), linear probes trained on the hidden states of LLMs to capture semantic entropy (Section 4).
- •
We demonstrate that semantic entropy is encoded in the hidden states of a single model generation and can be successfully extracted using probes (Section 6).
- •
We perform ablation studies to study SEP performance across models, tasks, layers, and token positions.
Our results strongly suggest internal model states across layers and tokens implicitly capture semantic uncertainty, even before generating any tokens. (Section 6)
- •
We show that SEPs can be used to predict hallucinations and that they generalize better than probes directly trained for accuracy as suggested by previous work, establishing a new state-of-the-art for cost-efficient hallucination detection (Section 7, Fig. 1).

## 2 Related Work

LLM Hallucinations.
We refer to Rawte et al. [], Zhang et al. [] for extensive surveys on hallucinations in LLMs and here briefly review the most relevant related work to this paper.
Early work on hallucinations in language models typically refers to issues in summarization tasks where models “hallucinate” content that is not faithful to the provided source text [, , , , , , ].
Around the same time, research emerged that showed LLMs themselves could store and retrieve factual knowledge [], leading to the currently popular closed-book setting, where LLMs are queried without any additional context [].
Since then, a large variety of work has focused on detecting hallucinations in LLMs for general natural language generation tasks.
These can typically be classified into one of two directions: sampling-based and retrieval-based approaches.

Sampling-Based Hallucination Detection.
For sampling-based approaches, a variety of methods have been proposed that sample multiple model completions for a given query and then quantify the semantic difference between the model generations [, , , , , ].
For this paper, Farquhar et al. [] is particularly relevant, as we use their semantic entropy measure to supervise our hidden state probes (we summarize their method in Section 3).
A different line of work does not directly re-sample answers for the same query, but instead asks follow-up questions to uncover inconsistencies in the original answer [, ].
Recent work has also proposed to detect hallucinations in scenarios where models generate entire paragraphs of text by decomposing the paragraph into individual facts or sentences, and then validating the uncertainty of those individual facts separately [, , , ].

Retrieval-Based Methods.
A different strategy to mitigate hallucinations is to rely on external knowledge bases, e.g. web search, to verify the factuality of model responses [, , , , , , , ].
An advantage of such approaches is that they do not rely on good model uncertainties and can be used directly to fix errors in model generations.
However, retrieval-based approaches can add significant cost and latency.
Further, they may be less effective for domains such as reasoning, where LLMs are also prone to produce unfaithful and misleading generations [, ].
Thus, retrieval- and uncertainty-based methods are orthogonal and can be combined for maximum effect.

Sampling and Finetuning Strategies.
A number of different strategies exist to reduce, rather than detect, the number of hallucinations that LLMs generate.
Previous work has proposed simple adaptations to LLM sampling schemes [, , ], preference optimization targeting factuality [], or finetuning to align “verbal” uncertainties of LLMs with model accuracy [, , ].

Understanding Hidden States.
Recent work suggests that simple operations on LLM hidden states can qualitatively change model behavior [, , ] manipulate knowledge [], or reveal deceitful intent [].
Probes can be a valuable tool to better understand the internal representations of neural networks like LLMs [, ].
Previous work has shown that hidden state probes can predict LLM outputs one or multiple tokens ahead with high accuracy [, ].
Relevant to our paper is recent work that suggests there is a “truthfulness” direction in latent space that predicts correctness of statements and generations [, , , , ].
Our work extends this – we are also interested in predicting if the model is hallucinating nonfactual responses, however, rather than directly supervising probes with accuracy labels, we argue that capturing semantic entropy is key for generalization performance.

## 3 Semantic Entropy

Measuring uncertainty in free-form natural language generation tasks is challenging.
The uncertainties over tokens output by the language model can be misleading because they conflate semantic uncertainty, uncertainty over the meaning of the generation, with lexical and syntactic uncertainty, uncertainty over how to phrase the answer (see the example in Section 1).
To address this, Farquhar et al. [] propose *semantic entropy*, which aggregates token-level uncertainties across clusters of semantic equivalence.(^1^11Farquhar et al. [] is a journal version of the original semantic entropy paper by Kuhn et al. [].)
Semantic entropy is important in the context of this paper because we use it as the supervisory signal to train our hidden state SEP probes.

Semantic entropy is calculated in three steps:
(1) for a given query $x$, sample model completions from the LLM,
(2) aggregate the generations into clusters $(C_{1},\dots,C_{K})$ of equivalent semantic meaning,
(3) calculate semantic entropy, $H_{\textup{SE}}$, by aggregating uncertainties within each cluster.
Step (1) is trivial, and we detail steps (2) and (3) below.

Semantic Clustering. To determine if two generations convey the same meaning, Farquhar et al. [] use natural language inference (NLI) models, such as DeBERTa [], to predict entailment between the generations. Concretely, two generations $s_{a}$ and $s_{b}$ are identical in meaning if $s_{a}$ entails $s_{b}$ and $s_{b}$ entails $s_{a}$, i.e. they entail each other bi-directionally. Farquhar et al. [] then propose a greedy algorithm to cluster generations semantically: for each sample $s_{a}$, we either add it to an existing cluster $C_{k}$ if bi-directional entailment holds between $s_{a}$ and a sample $s_{b}\in C_{k}$, or add it to a new cluster if the semantic meaning of $s_{a}$ is distinct from all existing clusters.
After processing all generations, we obtain a clustering of the generations by semantic meaning.

Semantic Entropy.
Given an input context $x$, the joint probability of a generation $s$ consisting of tokens $(t_{1},\dots,t_{n})$ is given by the product of conditional token probabilities in the sequence,

$$ $p(s\mid x)=\prod\nolimits_{i=1}^{n}p(t_{i}\mid t_{1:i-1},x).$ (1) $$

The probability of the semantic cluster $C$ is then the aggregate probability of all possible generations $s$ which belong to that cluster,

$$ $p(C\mid x)=\sum\nolimits_{s\in C}p(s\mid x).$ (2) $$

The uncertainty associated with the distribution over semantic clusters is the semantic entropy,

$$ $H[C\mid x]=\mathbb{E}_{p(C\mid x)}[-\log p(C\mid x)].$ (3) $$

Estimating SE in Practice.
In practice, we cannot compute the above exactly.
The expectations with respect to $p(s|x)$ and $p(C|x)$ are intractable, as the number of possible token sequences grows exponentially with sequence length.
Instead, Farquhar et al. [] sample $N$ generations $(s_{1},\dots,s_{N})$ at non-zero temperature from the LLM (typically and also in this paper $N=10$).
They then treat $(C_{1},\dots,C_{K})$ as Monte Carlo samples from the true distribution over semantic clusters $p(C|x)$, and approximate semantic entropy as

$$ $\displaystyle H[C\mid x]\approx-\frac{1}{K}\sum\nolimits_{k=1}^{K}\log p(C_{k} |x).$ (4) $$

We here use an additional approximation, employing the *discrete* variant of SE that yields good performance without access to token probabilities, making it compatible with black-box models [].
For the discrete SE variant, we estimate cluster probabilities $p(C|x)$ as the fraction of generations in that cluster, $\smash{p(C_{k}|x)=\textstyle\sum\nolimits_{j=1}^{N}\mathbbm{1}[s_{j}\in C_{k}]
/K}$, and then compute semantic entropy as the entropy of the resulting categorical distribution, $H_{\textup{SE}}(x)\coloneq-\textstyle\sum\nolimits_{k=1}^{K}p(C_{k}|x)\log p(C
_{k}|x)$.
Discrete SE further avoids problems when estimating Eq. 4 for generations of different lengths [, , , ].

## 4 Semantic Entropy Probes

Although semantic entropy is effective at detecting hallucinations, its high computational cost may limit its use to only the most critical scenarios.
In this section, we propose Semantic Entropy Probes (SEPs), a novel method for cost-efficient and reliable uncertainty quantification in LLMs.
SEPs are linear probes trained on the hidden states of LLMs to capture semantic entropy [].
However, unlike semantic entropy and other sampling-based approaches, SEPs act on the hidden states of a *single* model generation and do not require sampling multiple responses from the model at test time.
Thus, SEPs solve a key practical issue of semantic uncertainty quantification by almost completely eliminating the computational overhead of semantic uncertainty estimation at test time.
We further argue that SEPs are advantageous to probes trained to directly predict model accuracy.
Our intuition for this is that semantic entropy is an inherent property of the model that should be encoded in the hidden states and thus should be easier to extract than truthfulness, which relies on potentially noisy external information. We discuss this further in Section 8.

Training SEPs.
SEPs are constructed as linear logistic regression models, trained on the hidden states of LLMs to predict semantic entropy.
We create a dataset of $(h_{p}^{l}(x),H_{\textup{SE}}(x))$ pairs, where $x$ is an input query, $\smash{h_{p}^{l}(x)\in\mathbb{R}^{d}}$ is the model hidden state at token position $p$ and layer $l$, $d$ is the hidden state dimension, and $\smash{H_{\textup{SE}}(x)\in\mathbb{R}}$ is the semantic entropy.
That is, given an input query $x$, we first generate a high-likelihood model response via greedy sampling and store the hidden state at a particular layer and token position, $\smash{h_{p}^{l}(x)}$.
We then sample $N=10$ responses from the model at high temperature $(T=1)$ and compute semantic entropy, $\smash{H_{\textup{SE}}(x)}$, as detailed in the previous section.
For inputs, we rely on questions from popular QA datasets (see Section 5 for details), although we do not need the ground-truth labels provided by these datasets and could alternatively compute semantic entropy for any unlabeled set of suitable LLM inputs.

Binarization.
Semantic entropy scores are real numbers.
However, for the purposes of this paper, we convert them into binary labels, indicating whether semantic entropy is high or low, and then train a logistic regression classifier to predict these labels.
Our motivation for doing so is two-fold.
For one, we ultimately want to use our probes for predicting binary model correctness, so we eventually need to construct a binary classifier regardless.
Additionally, we would like to compare the performance of SEP probes and accuracy probes.
This is easier if both probes target binary classification problems.
We note that the logistic regression classifier returns probabilities, such that we can always recover fine-grained signals even after transforming the problem into binary classification.

More formally, we compute $\smash{\tilde{H}_{\textup{SE}}(x)=\mathbbm{1}[H_{\textup{SE}}(x)>\gamma^{\star
}]}$, where $\smash{\gamma^{\star}}$ is a threshold that optimally partitions the raw SE scores into high and low values according to the following objective:

$$ $\displaystyle\textstyle\gamma^{\star}=\arg\min_{\gamma}\sum\nolimits_{j\in \text{SE}_{\textup{low}}}(H_{\textup{SE}}(x_{j})-\hat{H}_{\textup{low}})^{2}+ \sum\nolimits_{j\in\text{SE}_{\textup{high}}}(H_{\textup{SE}}(x_{j})-\hat{H}_{ \textup{high}})^{2},$ (5) $$

where

$$ $\displaystyle\text{SE}_{\textup{low}}=\{j:H_{\textup{SE}}(x_{j})<\gamma\},$ $\displaystyle\text{SE}_{\textup{high}}=\{j:H_{\textup{SE}}(x_{j})\geq\gamma\},$ $\displaystyle\hat{H}_{\textup{low}}=\frac{1}{\lvert\text{SE}_{\textup{low}} \rvert}\sum\nolimits_{j\in\text{SE}_{\textup{low}}}H_{\textup{SE}}(x_{j}),$ $\displaystyle\hat{H}_{\textup{high}}=\frac{1}{\lvert\text{SE}_{\textup{high}} \rvert}\sum\nolimits_{j\in\text{SE}_{\textup{high}}}H_{\textup{SE}}(x_{j}).$ $$

This procedure is inspired by splitting objectives used in regression trees [] and we have found it to perform well in practice compared to alternatives such as soft labelling, cf. Appendix B.

In summary, given a input dataset of queries, $\{x_{j}\}_{j=1}^{Q}$, we compute a training set of hidden state – binarized semantic entropy pairs, $\smash{\{(h_{p}^{l}(x_{j}),\tilde{H}_{\textup{SE}}(x_{j}))\}_{j=1}^{Q}}$, and use this to train a linear classifier, which is our semantic entropy probe (SEP).
At test time, SEPs predict the probability that a model generation for a given input query $x$ has high semantic entropy.

Probing Locations.
We collect hidden states, $h_{p}^{l}(x)$, across all layers, $l$, of the LLM to investigate which layers best capture semantic entropy.
We consider two different token positions, $p$.
Firstly, we consider the hidden state at the last token of the *input* $x$, i.e. the token before generating (TBG) the model response.
Secondly, we consider the last token of the *model response*, which is the token before the end-of-sequence token, i.e. the second last token (SLT).
We refer to these scenarios as TBG and SLT.
The TBG experiments allow us to study to what extent LLM hidden states capture semantic entropy *before* generating a response.
The TBG setup potentially allows us to quantify the semantic uncertainty given an input in a single forward pass – without generating any novel tokens – further reducing the cost of our approach over sampling-based alternatives.
In practice, this may be useful to quickly determine if a model will answer a particular input query with high certainty.

## 5 Experiment Setup

We investigate and evaluate Semantic Entropy Probes (SEPs) across a range of models and datasets.
First, we show that we can accurately predict semantic entropy from the hidden states of LLMs (Section 6).
We then explore how SEP predictions vary across different tasks, models, tokens indices, and layers.
Second, we demonstrate that SEPs are a cheap and reliable method for hallucination detection (Section 7), which generalizes better to novel tasks than accuracy probes, although they cannot match the performance of much more expensive sampling-based methods in our experiments.

Tasks. We evaluate SEPs on four datasets: TriviaQA [], SQuAD [], BioASQ [], and NQ Open [].
We use the input queries of these tasks to derive training sets for SEPs and evaluate the performance of each method on the validation/test sets, creating splits if needed.
We consider a short- and a long-form setting: Short-form answers are generated by few-shot prompting the LLM to answer “as briefly as possible” and long-form answers are generated by prompting for a “single brief but complete sentence”, leading to an approximately six-fold increase in the number of generated tokens [].
Following Farquhar et al. [], we assess model accuracy via the SQuAD F1 score for short-form generations, and we use use GPT-4 [] to compare model answers to ground truth labels for long-form answers.
We provide prompt templates in Section B.1.

Models. We evaluate SEPs on five different models. For short generations, we generate hidden states and answers with Llama-2 7B and 70B [], Mistral 7B [], and Phi-3 Mini [], and use DeBERTa-Large [] as the entailment model for calculating semantic entropy []. For long generations, we use Llama-2-70B [] or Llama-3-70B [] and use GPT-3.5 [] to predict entailment.

Baselines.
We compare SEPs against the ground truth semantic entropy, accuracy probes supervised with model correctness labels, naive entropy, log likelihood, and the $p(\text{True})$ method of Kadavath et al. [].
For naive entropy, following Farquhar et al. [], we compute the length-normalized average log token probabilities across the same number of generations as for SE.
For log likelihood, we use the length-normalized log likelihood of a single model generation.
The $p(\text{True})$ method works by constructing a custom few-shot prompt that contains a number of examples – each consisting of a training set input, a corresponding low-temperature model answer, high-temperature model samples, and a model correctness score.
Essentially, $p(\text{True})$ treats sampling-based truthfulness detection as an in-context learning task, where the few-shot prompt teaches the model that model answers with high semantic variety are likely incorrect.
We refer to Kadavath et al. [] for more details.

Linear Probe.
For both SEPs and our accuracy probe baseline, we use the logistic regression model from scikit-learn [] with default hyperparameters for $\text{L}_{2}$ regularization and the LBFGS optimizer.

Evaluation. We evaluate SEPs both in terms of their ability to capture semantic entropy as well as their ability to predict model hallucinations.
In both cases, we compute the area under the receiver operating characteristic curve (AUROC), with gold labels given by binarized SE or model accuracy.

## 6 LLM Hidden States Implicitly Capture Semantic Entropy

This section investigates whether LLM hidden states encode semantic entropy.
We study SEPs across different tasks, models, and layers, and compare them to accuracy probes in- and out-of-distribution.

Hidden States Capture Semantic Entropy.
Figure 2 shows that SEPs are consistently able to capture semantic entropy across different models and tasks. Here, probes are trained on hidden states of the second-last-token for the short-form generation setting.
In general, we observe that AUROC values increase for later layers in the model, reaching values between $0.7$ and $0.95$ depending on the scenario.

Figure: Figure 2: Semantic Entropy Probes (SEPs) achieve high fidelity for predicting semantic entropy. Across datasets and models, SEPs are consistently able to capture semantic entropy from hidden states of mid-to-late layers. Short generation scenario with probes trained on second-last token (SLT).
Refer to caption: x2.png

Semantic Entropy Can Be Predicted Before Generating.
Next, we investigate if semantic entropy can be predicted before even generating the output.
Similar to before, Fig. 3 shows AUROC values for predicting binarized semantic entropy from the SEP probes.
Perhaps surprisingly (although in line with related work, cf. Section 2), we find that SEPs can capture semantic entropy even before generation. SEPs consistently achieve good AUROC values, with performance slightly below the SLT experiments in Fig. 2.
The TBG variant provides even larger cost savings than SEPs already do, as it allows us to quantify uncertainty before generating any novel tokens, i.e. with a single forward pass through the model.
This could be useful in practice, for example, to refrain from answering queries for which semantic uncertainty is high.

AUROC values for Llama-2-7B on BioASQ, in both Fig. 2 and Fig. 3, reach very high values, even for early layers.
We investigated this and believe it is likely related to the particularities of BioASQ.
Concretely, it is the only of our tasks to contain a significant number of yes-no questions, which are generally associated with lower semantic entropy as the possible number of semantic meanings in outcome space is limited.
For a model with relatively low accuracy such as Llama-2-7B, simply identifying whether or not the given input is a yes-no question, will lead to high AUROC values.

Figure: Figure 3: Semantic entropy can be predicted from the hidden states of the last input token, without generating any novel tokens. Short generations with SEPs trained on the token before generating (TBG).
Refer to caption: x3.png

Figure: Figure 4: SEPs successfully capture semantic entropy in Llama-2-70B and Llama-3-70B for long generations across layers and for both SLT and TBG token positions.
Refer to caption: x4.png

SEPs Capture Semantic Uncertainty for Long Generations.
While experiments with short generations are popular even in the recent literature [, , , , ], this scenario is increasingly disconnected from popular use cases of LLMs as conversational chatbots.
In recognition of this, we also study our probes in a long-form generation setting, which increases the average length of model responses from ~15 characters in the short-length scenario to about ~100 characters.

Figure 4 shows that, even in the long-form setting, SEPs are able to capture semantic entropy well in both the second-last-token and token-before-generation scenarios for Llama-2-70B and Llama-3-70B.
Compared to the short-form generation scenario, we now observe more often that AUROC values peak for intermediate layers.
This makes sense as hidden states closer to the final layer will likely be preoccupied with predicting the next token. In the long-form setting, the next token is more often unrelated to the semantic uncertainty of the overall answer, and instead concerned with syntax or lexis.

Figure: Figure 5: Fig. 5: SEPs capture drop in SE due to added context.
Refer to caption: x5.png

Counterfactual Context Addition Experiment.
To confirm that SEPs capture SE rather than relying on spurious correlations, we perform a counterfactual intervention experiment for Llama-2-7B on TriviaQA.
For each input question of TriviaQA, the dataset contains a “context”, from which the ground truth answer can easily be predicted.
We usually exclude this context, because including it makes the task too easy.
However, for the purpose of this experiment, we add the context and study how this affects SEP predictions.

Figure 5 shows a kernel density estimate of the distribution over the predicted probability for high semantic entropy, $p(\text{high SE})$, for Llama-2-7B on the TriviaQA dataset with context (blue) and without context (orange) in the short generation setting using the SLT.
Without context, the distribution for $p(\text{high SE})$ from the SEP is concentrated around $0.9$.
However, as soon as we provide the context, $p(\text{high SE})$ decreases, as shown by the shift in distribution. As the task becomes much easier – accuracy increases from 26% to 78% – the model becomes more certain – ground truth SE decreases from 1.84 to 0.50.
This indicates SEPs accurately capture model behavior for the context addition experiment, with predictions for $p(\text{high SE})$ following ground truth SE behavior, despite never being trained on inputs with context.

## 7 SEPs Are Cheap and Reliable Hallucination Detectors

In this section, we explore the use of SEPs to predict hallucinations, comparing them to accuracy probes and other baselines.
Crucially, we also evaluate probes in a challenging generalization setting, testing them on tasks that they were not trained for.
This setup is much more realistic than evaluating probes in-distribution, as, for most deployment scenarios, inputs will rarely match the training distribution exactly.
The generalization setting does not affect semantic entropy, naive entropy, or log likelihood, which do not rely on training data.
While $p(\text{True})$ does rely on a few samples for prompt construction, we find its performance is usually unaffected by the task origin of the prompt data.

Figure: Figure 6: SEPs predict model hallucinations better than accuracy probes when generalizing to unseen tasks. In-distribution, accuracy probes perform better. Short generation setting with Llama-2-7B, SEPs trained on the second-last-token (SLT). For the generalization setting, probes are trained on all tasks except the one that we evaluate on.
Refer to caption: x6.png

**Table 1: Tab. 1: $\Delta$AUROC (x100) of SEPs and acc. probes over tasks in-distribution. Avg $\pm$ std error, (S)hort- and (L)ong-form gens.**
| Model | SEP $-$ Acc Pr. |
| --- | --- |
| Mistral-7B (S) | $2.8\pm 1.4$ |
| Phi-3-3.8B (S) | $2.1\pm 0.8$ |
| Llama-2-7B (S) | $-0.5\pm 2.6$ |
| Llama-2-70B (S) | $1.3\pm 0.7$ |
| Llama-2-70B (L) | $-1.9\pm 7.5$ |
| Llama-3-70B (L) | $-2.0\pm 2.1$ |

Figure 6 shows both in-distribution and generalization performance of SEPs and accuracy probes across different layers for Llama-2-7B in a short-form generation setting trained on the SLT.
In-distribution, accuracy probes outperform SEPs across most layers and tasks, with the exception of NQ Open.
In Table 1, we compare the average difference in AUROC between SEPs and accuracy probes for predicting model hallucinations, taking a representative set of high-performing layers for both probe types (see Appendix B).
We find that SEPs and accuracy probes perform similarly on in-distribution data across models.
We report unaggregated results in Fig. A.8.
The performance of SEPs here is commendable: SEPs are trained without any ground truth answers or accuracy labels, and yet, can capture truthfulness.
To the best of our knowledge, SEPs may be the best unsupervised method for hallucination detection even in-distribution, given problems of other unsupervised methods for truthfulness prediction [].

However, when evaluating probe generalization to new tasks, SEPs show their true strength.
We evaluate probes in a leave-one-out fashion – evaluating on all datasets except one, which we train on.
As shown in Fig. 6 (right), SEPs consistently outperform accuracy probes across various layers and tasks for short-form generations in the generalization setting.
For BioASQ, the difference is particularly large.
SEPs clearly generalize better to unseen tasks than accuracy probes.
In Table 2 and Fig. 7, we report results for more models, taking a representative set of high-performing layers for both probe types, and Fig. A.7 shows results for Mistral-7B across layers.
We again find that SEPs generalize better than accuracy probes to novel tasks.
We additionally compare to the sampling-based semantic entropy, naive entropy, and $p(\text{True})$ methods.
While SEPs cannot match the performance of

**Table 2: Tab. 2: $\Delta$AUROC (x100) of SEPs over acc. probes for task generalization. Avg $\pm$ std error, (S)hort- and (L)ong-form gens.**
| Model | SEP $-$ Acc Pr. |
| --- | --- |
| Mistral-7B (S) | $10.5\pm 3.5$ |
| Phi-3-3.8B (S) | $9.9\pm 2.9$ |
| Llama-2-7B (S) | $7.7\pm 1.3$ |
| Llama-2-70B (S) | $7.9\pm 3.0$ |
| Llama-2-70B (L) | $2.2\pm 0.4$ |
| Llama-3-70B (L) | $6.2\pm 1.9$ |

these methods, it is important to note the significantly higher cost these baselines incur, requiring $10$ additional model generations, whereas SEPs and accuracy probes operate on single generations.

We further evaluate SEPs for long-form generations.
As shown in Fig. 8, SEPs outperform accuracy probes for Llama-2-70B and Llama-3-70B in the generalization setting.
We also provide in-distribution results for long generations with both models in Figs. A.9 and A.10.
Both results confirm the trend discussed above.
Overall, our results clearly suggest that SEPs are the best choice for cost-effective uncertainty quantification in LLMs, especially if the distribution of the query data is unknown.

Figure: Figure 7: SEPs generalize better to new tasks than accuracy probes across models and tasks. They approach, but do not match, the performance of other, $10$x costlier baselines (hatched bars). Short generation setting, SLT, performance for a selection of representative layers.
Refer to caption: x7.png

Figure: Figure 8: Semantic entropy probes outperform accuracy probes for hallucination detection in the long-form generation generalization setting with both Llama-2-70B and Llama-3-70B.
Refer to caption: x8.png

## 8 Discussion, Future Work, and Conclusions.

Discussion.
Our experiments show that SEPs generalize better than accuracy probes – in terms of detecting hallucinations – to inputs from unseen tasks.
One potential explanation for this is that semantic uncertainty is a better probing target than truthfulness, because semantic uncertainty is a more model-internal characteristic that can be better predicted from model hidden states.
Model correctness labels required for accuracy probing on the other hand are external and can be noisy, which may make them more difficult to predict from hidden states.
We can find evidence for this by comparing the in-distribution AUROC values for SEPs (for predicting binarized SE) with the AUROC of the accuracy probes for predicting accuracy in Figs. A.6 and A.5.
AUROC values – which are insensitive to class frequencies and a good proxy for task difficulty – for predicting SE are significantly higher, indicating that semantic entropy is indeed better captured by model hidden states than accuracy.

Another possible explanation for the gap in OOD generalization could be that accuracy probes capture model correctness in a way that is *specific* to the training dataset.
For example, the probe may latch on to discriminative features for model correctness that relate to the task at hand but do not generalize, such as identifying a knowledge domain where accuracy is high or low, but which rarely occurs outside the training data.
Conversely, semantic probes may capture more inherent model states – e.g., uncertainty from failure to gather relevant facts or attributes for the query.
The literature on mechanistic interpretability [] supports the idea that such information is likely contained in model hidden states.
We believe that concretizing these links is a fruitful area for future research.

Future Work.
We believe it should be possible to further close the performance gap between sampling-based approaches, such as semantic entropy, and SEPs.
One avenue to achieve this could be to increase the scale of the training datasets used to train SEPs.
In this work, we relied on established QA tasks to train SEPs to allow for easy comparison to accuracy probes. However, future work could explore training SEPs on unlabelled data, such as inputs generated from another LLM or natural language texts used for general model training or finetuning.
This could massively increase in the amount of training data for SEPs, which should improve probe accuracy, and also allow us to explore other more complex probing techniques that require more training data.

Conclusions.
We have introduced semantic entropy probes (SEPs): linear probes trained on the hidden states of LLMs to predict semantic entropy, an effective measure of uncertainty for free-form LLM generations [].
We find that the hidden states of LLMs implicitly capture semantic entropy across a wide range of scenarios.
SEPs are able to predict semantic entropy consistently, and, importantly, they detect model hallucinations more effectively than probes trained directly for accuracy prediction when testing on novel inputs from a different distribution than the training set – despite not requiring any ground truth model correctness labels.
Semantic uncertainty probing, both in terms of model interpretability and practical applications, is an exciting avenue for further research.

Author Contributions and Acknowledgements.
JK conceived the project idea, wrote the paper, and, together with MR, provided close mentoring for JH throughout the project.
JH wrote the code for SEPs, carried out all of the experiments in the paper, and created some of the figures.
JK, MR, LS, and SM explored SEPs in a hackathon, refining the idea and collecting positive preliminary results.
LS provided expertise on model interpretability and suggested extensive improvements to the writing.
SM created all plots in the initial version of main paper and appendix.
YG provided high level guidance.
All authors provided critical feedback on writing.

The authors further thank Kunal Handa, Gunshi Gupta, and all members of the OATML lab for insightful discussions, in particular for the feedback given during the hackathon.
SM and LS acknowledge funding from the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems (Grant No: EP/S024050/1).

## Appendix A Additional Results

#### Model Task Accuracies.

We report the accuracies achieved by the models on the various datasets used in this work in Table 3.

**Table 3: Task accuracy of models across datasets, in (L)ong- and (S)hort-form generation settings.**
| Model | BioASQ (%) | TriviaQA (%) | NQ Open (%) | SQuAD (%) |
| --- | --- | --- | --- | --- |
| Llama-3-70B (L) | 67.2 | 88.5 | 61.2 | 46.0 |
| Llama-2-70B (L) | 60.3 | 85.0 | 58.3 | 43.9 |
| Llama-2-70B (S) | 48.4 | 75.7 | 49.5 | 31.4 |
| Llama-2-7B (S) | 43.3 | 64.8 | 38.3 | 23.5 |
| Mistral-7B (S) | 39.3 | 52.3 | 28.3 | 20.7 |
| Phi-3-3.8B (S) | 45.5 | 48.3 | 26.1 | 24.3 |

#### Predicting Model Correctness from Hidden States.

Figures A.1 and A.2 give additional results that show we can predict model correctness from hidden states using SEPs trained on the second-last-token (SLT) or token-before-generating (TBG) in the short-form in-distribution scenario across models and tasks.
In Figs. A.3 and A.4, we further demonstrate that accuracy probes also perform similarly when trained on the SLT or TBG in the short-form in-distribution scenario across models and tasks.

Figure: Figure A.1: Semantic Entropy Probes (SEPs) capture model hallucinations. Short generations with SEPs trained on the hidden states of the model at the second-last-token (SLT).
Refer to caption: x9.png

Figure: Figure A.2: Semantic Entropy Probes (SEPs) capture model hallucinations. Short generations with SEPs trained on the hidden states of the model at the token-before-generation (TBG).
Refer to caption: x10.png

Figure: Figure A.3: Accuracy probes for in-distribution short-form generation trained on the second-last-token (SLT).
Refer to caption: x11.png

#### Predicting Correctness vs. Semantic Entropy.

Figures A.5 and A.6 show that predicting semantic entropy from hidden states is generally easier than directly predicting model correctness, suggesting that semantic entropy is implicitly encoded in the hidden states.

Figure: Figure A.4: Accuracy probes for in-distribution short-form generation trained on the token-before-generation (TBG).
Refer to caption: x12.png

Figure: Figure A.5: Predicting semantic entropy from hidden states with SEPs works better than predicting accuracy from the hidden states with accuracy probes. Llama-2-7B and 70B in the short generation setting with probes trained on hidden states of the SLT, evaluated in-distribution.
Refer to caption: x13.png

Figure: Figure A.6: Predicting semantic entropy from hidden states with SEPs works better than predicting accuracy from the hidden states with accuracy probes. Mistral-7B and Phi-3 Mini in short generation setting with probes trained on hidden states of the SLT, evaluated in-distribution.
Refer to caption: x14.png

Figure: Figure A.7: SEPs predict model hallucinations better than accuracy probes when generalizing to unseen tasks (right). In-distribution, accuracy probes have comparable performance (left). Mistral-7B in the short generations setting with probes trained hidden states from the SLT. For the generalization setting, probes are trained on all tasks except the one that we evaluate on.
Refer to caption: x15.png

#### Additional Comparisons to Baselines.

In, Fig. A.7 we additionally report results comparing SEPs to accuracy probes across layers for Mistral-7B for the in-distribution and generalization settings.
In Fig. A.8, we compare the performance of SEPs to baselines for the in-distribution setting across models and datasets, finding that SEPs and accuracy probes perform similarly, with SEPs performing slightly better for 3 out of 5 models.
In Figs. A.9 and A.10 we report in- and out-of-distribution results for Llama-2-70B and Llama-3-70B in the long-form generation setting.

Figure: Figure A.8: Short generation performance for the in-distribution setting across models compared to baseline methods. Hatched bars indicate more computationally expensive methods.
Refer to caption: x16.png

Figure: Figure A.9: Semantic entropy probes outperform accuracy probes for hallucination detection in the long-form generation generalization setting with Llama-2-70B. In-distribution, accuracy probes sometimes outperform and sometimes underperform. Probes cannot match the performance of the significantly more expensive baselines.
Refer to caption: x17.png

Figure: Figure A.10: Semantic entropy probes outperform accuracy probes for hallucination detection in the long-form generation generalization setting with Llama-3-70B. In-distribution, accuracy probes often outperform SEPs. Probes cannot match the performance of much more expensive baselines.
Refer to caption: x18.png

Hidden State Alternatives.
In addition to investigating the performance of probes on the hidden states, we study whether residual stream or MLP outputs can also be used for semantic entropy prediction. Figure A.11 shows that probing the hidden states results in consistently higher performance.

Figure: Figure A.11: Probing different model components for SEPs. The hidden states are more predictive than residual streams and MLP outputs. TriviaQA, Llama-2-7B, in-distribution, short-form generations, SLT.
Refer to caption: x19.png

Different Binarization Procedures.
In addition to the “best split” procedure discussed in Section 4 and used in all of our experiments, we here explore the performance of a simple “even split” alternative, which splits semantic entropy into high and low classes such that there are an equal number samples in both classes.
Figure A.12 shows that performance is similar, with our optimal splitting procedure slightly outperforming the even split ablation.
For illustration purposes, Fig. A.13 shows the behavior of the best split objective Eq. 5 across different thresholds.
We have also explored a “soft labelling” strategy as an alternative to hard binarization, for which we obtain soft labels by transforming raw semantic entropies into probabilities with a sigmoid function centered around the best-split threshold, and then train SEPs on the resulting soft labels. Early results did not improve performance.

Figure: Figure A.12: Comparing binarization methods for semantic entropy. Our “best split” procedure slightly outperforms the “even split” strategy, although SEPs do not appear overly sensitive to the binarization procedure. Long-form generations for Llama-2-70B, SLT, in-distribution.
Refer to caption: x20.png

Figure: Figure A.13: MSE of the best-split objective Eq. 5 for different binarization thresholds $\gamma$ for models in either short-form generation or (L)ong-form generation settings (SLT).
Refer to caption: x21.png

## Appendix B Experiment Details

Here we provide additional details to reproduce the experiments of the main paper.

### B.1 Prompt Templates

We use the following prompt templates across experiments.

For long-form generations, we use the following prompt template:

For short-form generations, we adjust the instruction and additionally provide 5 demonstration examples with short ground truth answers, to elicit a short answer from the model:

Finally, for the counterfactual context addition experiment, we prepend the context, prior to the question:

### B.2 Semantic Entropy Calculation

We compute semantic entropy with $N=10$ generations sampled at temperature $T=1.0$ and using default values of top-p ($p=0.9$) and top-K ($K=50$).

For short-form generations, we predict entailment using DeBERTa-Large [] and assess model accuracy via the SQuAD F1 score.

For long-form generations, we predict entailment with GPT-3.5 [] and the following prompt:

To assess the correctness of long-form generations, we prompt GPT-4 [] or GPT-4o(^2^22 We use GPT-4 to evaluate Llama-2-70B but switched to GPT-4o for our more recent experiments on Llama-3-70B given the difference in cost between the two GPT models.)
as follows

### B.3 Semantic Entropy Probes

SEPs are trained on the hidden states, which vary in dimensionality between models.
We detail the dimensionality of the hidden states, and number of layers in Table 4.

**Table 4: Models properties and selected layers for concatenation for SEPs and (Acc)uracy (P)robe, in (L)ong-form and (S)hort-form generation settings.**
| Model Name | No. of Layers | Hidden Dim. | Layers for SEPs | Layers for Acc. P. |
| --- | --- | --- | --- | --- |
| Llama-3-70B (L) | 80 | 8192 | [76, 77, 78, 79, 80] | [31, 32, 33, 34, 35] |
| Llama-2-70B (L) | 80 | 8192 | [74, 75, 76, 77, 78] | [76, 77, 78, 79, 80] |
| Llama-2-70B (S) | 80 | 8192 | [76, 77, 78, 79, 80] | [75, 76, 77, 78, 79] |
| Llama-2-7B (S) | 32 | 4096 | [28, 29, 30, 31, 32] | [18 ,19 ,20 ,21, 22] |
| Mistral-7B (S) | 32 | 4096 | [28, 29, 30, 31, 32] | [12 ,13 ,14 ,15, 16] |
| Phi-3-3.8B (S) | 32 | 3072 | [21, 22, 23, 24, 25] | [25, 26, 27, 28, 29] |

Layer Concatenation.
For any aggregate results presented in the main paper or appendix, i.e. any barplots or tables, we report SEP and accuracy probe performance on a representative set of high-performing layers.
Concretely, we select a set of adjacent layers and concatenate their hidden states to train both types of probes based on the highest mean AUROC value achieved in the interval (on un-concatenated hidden states) in the in-distribution setting.
We report the layers across which we concatenate in Table 4.

Filtering for Long-form Generations.
In order to provide a clearer signal to the SEP on what constitutes high and low semantic entropy inputs, we filter out training samples with semantic entropy in between the 55% and 80% quantiles for long generations, as we have found this to give a mild increase in performance.
Note that this filtering did not improve performance for the accuracy probes, and we report results for the accuracy probes without filtering. We found this filtering to be unnecessary for experiments with Llama-3-70B.

Training Set Size.
For long-generation experiments, we collect 1000 samples across tasks.
For short-generation experiments, we collect 2000 samples of hidden state–semantic entropy pairs across tasks.
We match the training set sizes between accuracy probes and SEPs.

### B.4 Baselines

For the $p(\text{True})$ baseline, we construct a few-shot prompt with 10 examples, where each example is formatted as below:

We give an illustrative example below for what this could look like in practice:

For $p(\text{True})$, we obtain the probability of model truthfulness by measuring the token probability of $A$ at the end of the prompt.

### B.5 Evaluation

To evaluate the performance of the probes in the generalization setting, we employ the following leave-one-out procedure for the aggregate results reported in the barplots and tables.

First, each probe is trained on a single dataset.
Then, the trained probes are evaluated on all other datasets in terms of AUROC of detecting hallucinations, excluding the dataset used for training.
We then report the mean across all probes evaluated on that specific dataset.
This allows us to assess the generalization capability of the probes by measuring their performance on datasets that were not used during the training phase.
This scenario is important in practice, as the distribution of the query data will rarely be known.

## Appendix C Compute Resources

We make use of an internal cluster with 24 Nvidia A100 80GB GPUs. We further use GPT 3.5, 4, and 4o via the OpenAI API.

For experiments requiring the use of Llama 70B models, we require 2 A100s to do inference and calculate the hidden states. The smaller models require only a slice of an A100 80GB. However, once the training data for the semantic entropy probes has been created, a CPU-only computing resource is sufficient to fit the logistic regression models.

Based on tracked finished runs, we estimate ~300 GPU-hours plus ~310 CPU-hours to obtain the results in the paper.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Do authors have a clear explanation of why it is feasible to predict semantic entropy (SE) from internal states and why SE could serve as a better supervisory signal than accuracy in out-of-distribution training settings?
2. One of the advantages of SEP is that it does not rely on ground truth labels from the dataset. However, as mentioned by the authors, there are also existing works [1,2] that predict the trustworthiness of LLMs from internal states using unsupervised methods. I am curious about how these methods would perform under the same settings as SEP.


---
[1] Discovering Latent Knowledge in Language Models Without Supervision
[2] Representation Engineering: A Top-Down Approach to AI Transparency
soundness: 3
strengths: * This paper addresses a timely and significant research problem in LLMs. Detecting hallucinations is crucial for ensuring the safety of LLM applications in real-world scenarios.
* The experiments are extensive and compelling. The authors conduct experiments on and report results for several popular open-source models.
* The paper is generally well-written and easy to follow.
summary: This paper introduces a technique called Semantic Entropy Probes (SEP) for detecting hallucinations in large language models (LLMs). SEP leverages the internal states of LLMs to predict the semantic entropy (SE) of their responses. By doing so, the method empirically achieves an accurate approximation of SE without the computational overhead of multi-sampling. Furthermore, SEP effectively identifies model hallucinations in both in-distribution and out-of-distribution settings.
weaknesses: * I think the main weakness of this work is the lack of clear motivation and explanation for the outcomes. The SEP method appears to simply switch the prediction goal of previous internal state detection methods from accuracy to semantic entropy (SE), which seems like a simple 'a+b' combination. Although the results are promising, I am still expecting more insights from the experimental results regarding the mechanisms of LLMs. For example, why is it possible to use internal states to predict SE? And why does switching the prediction goal from accuracy to SE result in better performance in out-of-distribution settings? A clearer explanation of these questions would further enhance the solidness of this work and our understanding of LLMs.
* An obvious limitation is the applicability of the proposed method to black-box LLMs. I recommend that the authors discuss this in the limitations section.
* Most of the experiments compare the proposed method with accuracy-based probing methods. However, there are insufficient details provided to describe these counterpart methods. It would be more helpful to provide a comprehensive introduction to these methods for readers who are unfamiliar with them.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - To calculate semantic entropy (L185), how is the number of clusters K determined? 
- Can authors further discuss why "the discrete variant of SE" should be used instead of the plain version Eq (1)? How is the "fraction of generations in the cluster" related to "avoid problems for generation of different length" (L193)?
- In Sec 5, it seems that the prompt "single brief but complete sentence" also promotes short responses. Does a Chain-of-Thought style prompt work for the long-form answer generation?
soundness: 3
strengths: - The method to estimate semantic entropy by linear probing is simple to implement and alleviates repeated sampling at test-time. 
- The experiments and ablations are thorough to demonstrate the effectiveness of the proposed method. 
- The overall organization of the paper is clear and coherent.
summary: The paper proposes a new method Semantic Entropy Probes (SEPs) for hallucination detection of LLMs, which aims to capture semantic entropy by training a linear classifier on top of LLM hidden states. The authors perform various ablation studies to understand different design choices (model family, layer, token position) across common benchmarks. Experiments suggest the effectiveness of the proposed method over various baselines.
weaknesses: - **Motivation of SEP Efficiency**: The primary motivation for SEP, as stated in the paper, is that it is "cheap" and significantly enhances computational efficiency compared to the baseline semantic entropy (SE). However, this claim appears to be poorly substantiated. The comparison between a training-based method (SEP) and a test-time method (SE) is insufficiently justified. For a fair comparison, it might be better to provide analysis of the overall cost: SE with sampling v.s. the total cost (training and inference) of SEP.

- **Improvement in Long Generation Experiments**: The long generation experiments could be improved. Currently, long generation is primarily conducted with prompts that request "a single brief but complete sentence," resulting in relatively short responses. This approach diverges from recent practices, where models are often prompted to generate detailed reasoning or more elaborate outputs for long-form generation.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: The improved efficiency comes at a significant accuracy hit as compared to SE. How would the SE results be affected for smaller sample size N? The potential benefits of the proposed method would be significantly diminished if SE tends to be more or less stable in response to reducing N. 

Would it make sense to use significantly higher N during training as a way to improve SEP accuracy? 

In this work, probes are applied only to separate layers. Have the authors considered combinations of multiple layers? This would make sense if one assumes that different layers may contribute different evidence for model uncertainty. 

How OOD is the OOD setting really? How (dis-)similar are the different datasets from one another? 

I assume the values in Figure 2 are for held-out not training data. Is that correct? 

What is the actual potential time efficiency benefit of the method as compared to SE, for example, as measured in in wall clock time? And how much would the possibility to generate samples in batch-mode affect those numbers? 

The proposed method is very simple and seems somewhat "obvious". I'm a bit surprised this hasn't been studied before, but not being an expert in this work I give the paper the benefit of the doubt and consider this simplicity a plus.
soundness: 3
strengths: The paper is well-written, quite polished and very easy to follow. Empirical evaluations seem reasonably detailed and the findings clear.
summary: The paper describes an approach to hallucination detection. Unlike existing methods, that generate multiple samples at inference time to derive an estimate of uncertainty (semantic entropy), the method trains a linear model on a subset of the LLM's hiddens to predict it efficiently. The paper shows that this works reasonably well, although the accuracy is behind that of the sampling based methods.
weaknesses: The accuracy of the semantic entropy prediction is quite significantly behind the sampling-based estimate. In addition, it seems hard to quantify how much prediction accuracy for hallucination detection would actually impact any practical applications. Since there are many possible avenues to improve accuracy, some investigation in this direction - even just in a preliminary fashion - would make this work stronger.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. α represents the optimal threshold for partitioning raw SE scores into high categories, and I'm interested in how this threshold varies across different datasets. Additionally, I'm uncertain about the generalizability of trained SEPs across these datasets.
2. Would using hidden state information to predict SelfCheckGPT scores [1], Eigenscore [2], or other uncertainty scores potentially lead to better results?

[1] SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models

[2] INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection
soundness: 3
strengths: 1. The motivation is intuitive and clear, the paper is well written and easy to implement.
2. Training linear probes on the hidden states of large language models to capture semantic entropy is a promising approach. It offers new insights for reducing the cost of traditional sampling-based hallucination detection.
3. Comprehensive ablation studies have been conducted across various models, tasks, layers, and token positions. The experimental results align with expectations.
summary: This paper proposes Semantic Entropy Probes (SEPs) for hallucination detection in LLMs. The primary motivation of this work is to reduce the computational cost associated with semantic entropy, a representative sampling-based method for detecting hallucinations. SEPs are trained using the hidden states of LLMs to predict semantic entropy. The authors demonstrate that SEPs can effectively predict hallucinations and offer better generalization compared to probes that are directly trained for accuracy.
weaknesses: 1. The related work section lacks several crucial references [1,2,3,4]. For example, ref [1]  also adopt a supervised method for uncertainty estimation of LLMs. ref [2,3] explores the internal states of LLMs for hallucination detection. The author should introduce more hallucination detection methods based on internal states or linear probs. I have seen at least 5 papers that train a classifier for hallucination detection. Therefore, I suggest the authors to add a paragraph discussing internal state-based and probe-based hallucination detection methods, highlighting key differences between those approaches and SEPs

2. The performance of SEPs is unsatisfactory, and the author should include more competitive baseline methods. According to Fig. 7, SEPs fall short by 5-10 points compared to SE. Furthermore, SE is not the state-of-the-art (SOTA) method for hallucination detection [2,5]. It is recommended that the author include comparisons with the SOTA methods. I acknowledge that SEPs aim to balance performance and computational efficiency, I would suggest the authors to discuss this trade-off more explicitly in relation to SOTA methods.

3. There seems to be no clear explanation for why predicting SE yields better performance compared to predicting labels. I would appreciate a more detailed analysis explaining why predicting SE outperforms label prediction.

[1] Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach

[2] INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection

[3] LLM Internal States Reveal Hallucination Risk Faced With a Query.

[4] LLM Factoscope: Uncovering LLMs’ Factual Discernment through Measuring Inner States

[5] SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models

Meta-review: This work proposes Semantic Entropy Probes (SEPs), a method for uncertainty quantification in Large Language Models (LLMs) that approximates semantic entropy directly from hidden states, avoiding the computational overhead of sampling multiple generations. Experiments have demonstrated that SEPs retain high performance in detecting hallucinations, generalize well to out-of-distribution data, and significantly reduce the cost of semantic uncertainty quantification during inference. During the rebuttal phase, the authors provided additional experiments and explanations to address the reviewers' concerns. While many of these concerns have been successfully addressed, some remain unresolved. First, the reviewer still has significant concerns regarding the performance of accuracy probing, which outperforms the proposed Semantic Entropy Probe on in-distribution data. Although the authors presented experiments highlighting SEP’s advantages on out-of-distribution data, the reviewer remains concerned due to the lack of a solid explanation for why and how this occurs. Second, generating semantic entropy during training incurs substantial overhead due to the cost of providing semantic entropy ground truth. Therefore, we have decided not to accept this work in its current state.

Decision: Reject