Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs

Authors: Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth A Malik, Yarin Gal
Primary area: foundation or frontier models, including LLMs
Keywords: large language models; hallucinations; linear probing; uncertainty estimation; interpretability

Abstract:
We propose semantic entropy probes (SEPs), a cheap and reliable method for uncertainty quantification in Large Language Models (LLMs). Hallucinations, which are plausible-sounding but factually incorrect and arbitrary model generations, present a major challenge to the practical adoption of LLMs. Recent work by Farquhar et al. proposes semantic entropy (SE), which can reliably detect hallucinations by quantifying the uncertainty over different generations by estimating entropy over semantically equivalent sets of outputs. However, the 5-to-10-fold increase in computation cost associated with SE computation hinders practical adoption. To address this, we propose SEPs, which directly approximate SE from the hidden states of a single generation. SEPs are simple to train and do not require sampling multiple model generations at test time, reducing the overhead of semantic uncertainty quantification to almost zero. We show that SEPs retain high performance for hallucination detection and generalize better to out-of-distribution data than previous probing methods that directly predict model accuracy. Our results across models and tasks suggest that model hidden states capture SE, and our ablation studies give further insights into the token positions and model layers for which this is the case.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730573000145

thread_type:
official_review

<thread_object>
creator: reviewer_erj8

<content>
summary:
This paper proposes Semantic Entropy Probes (SEPs) for hallucination detection in LLMs. The primary motivation of this work is to reduce the computational cost associated with semantic entropy, a representative sampling-based method for detecting hallucinations. SEPs are trained using the hidden states of LLMs to predict semantic entropy. The authors demonstrate that SEPs can effectively predict hallucinations and offer better generalization compared to probes that are directly trained for accuracy.

strengths:
1. The motivation is intuitive and clear, the paper is well written and easy to implement.
2. Training linear probes on the hidden states of large language models to capture semantic entropy is a promising approach. It offers new insights for reducing the cost of traditional sampling-based hallucination detection.
3. Comprehensive ablation studies have been conducted across various models, tasks, layers, and token positions. The experimental results align with expectations.

weaknesses:
1. The related work section lacks several crucial references [1,2,3,4]. For example, ref [1]  also adopt a supervised method for uncertainty estimation of LLMs. ref [2,3] explores the internal states of LLMs for hallucination detection. The author should introduce more hallucination detection methods based on internal states or linear probs. I have seen at least 5 papers that train a classifier for hallucination detection. Therefore, I suggest the authors to add a paragraph discussing internal state-based and probe-based hallucination detection methods, highlighting key differences between those approaches and SEPs

2. The performance of SEPs is unsatisfactory, and the author should include more competitive baseline methods. According to Fig. 7, SEPs fall short by 5-10 points compared to SE. Furthermore, SE is not the state-of-the-art (SOTA) method for hallucination detection [2,5]. It is recommended that the author include comparisons with the SOTA methods. I acknowledge that SEPs aim to balance performance and computational efficiency, I would suggest the authors to discuss this trade-off more explicitly in relation to SOTA methods.

3. There seems to be no clear explanation for why predicting SE yields better performance compared to predicting labels. I would appreciate a more detailed analysis explaining why predicting SE outperforms label prediction.

[1] Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach

[2] INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection

[3] LLM Internal States Reveal Hallucination Risk Faced With a Query.

[4] LLM Factoscope: Uncovering LLMs’ Factual Discernment through Measuring Inner States

[5] SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models

questions:
1. α represents the optimal threshold for partitioning raw SE scores into high categories, and I'm interested in how this threshold varies across different datasets. Additionally, I'm uncertain about the generalizability of trained SEPs across these datasets.
2. Would using hidden state information to predict SelfCheckGPT scores [1], Eigenscore [2], or other uncertainty scores potentially lead to better results?

[1] SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models

[2] INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection
</content>
</thread_object>

</thread>
<thread>
creation date:
1730593480112

thread_type:
official_review

<thread_object>
creator: reviewer_bxzx

<content>
summary:
The paper describes an approach to hallucination detection. Unlike existing methods, that generate multiple samples at inference time to derive an estimate of uncertainty (semantic entropy), the method trains a linear model on a subset of the LLM's hiddens to predict it efficiently. The paper shows that this works reasonably well, although the accuracy is behind that of the sampling based methods.

strengths:
The paper is well-written, quite polished and very easy to follow. Empirical evaluations seem reasonably detailed and the findings clear.

weaknesses:
The accuracy of the semantic entropy prediction is quite significantly behind the sampling-based estimate. In addition, it seems hard to quantify how much prediction accuracy for hallucination detection would actually impact any practical applications. Since there are many possible avenues to improve accuracy, some investigation in this direction - even just in a preliminary fashion - would make this work stronger.

questions:
The improved efficiency comes at a significant accuracy hit as compared to SE. How would the SE results be affected for smaller sample size N? The potential benefits of the proposed method would be significantly diminished if SE tends to be more or less stable in response to reducing N. 

Would it make sense to use significantly higher N during training as a way to improve SEP accuracy? 

In this work, probes are applied only to separate layers. Have the authors considered combinations of multiple layers? This would make sense if one assumes that different layers may contribute different evidence for model uncertainty. 

How OOD is the OOD setting really? How (dis-)similar are the different datasets from one another? 

I assume the values in Figure 2 are for held-out not training data. Is that correct? 

What is the actual potential time efficiency benefit of the method as compared to SE, for example, as measured in in wall clock time? And how much would the possibility to generate samples in batch-mode affect those numbers? 

The proposed method is very simple and seems somewhat "obvious". I'm a bit surprised this hasn't been studied before, but not being an expert in this work I give the paper the benefit of the doubt and consider this simplicity a plus.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712479290

thread_type:
official_review

<thread_object>
creator: reviewer_toq8

<content>
summary:
The paper proposes a new method Semantic Entropy Probes (SEPs) for hallucination detection of LLMs, which aims to capture semantic entropy by training a linear classifier on top of LLM hidden states. The authors perform various ablation studies to understand different design choices (model family, layer, token position) across common benchmarks. Experiments suggest the effectiveness of the proposed method over various baselines.

strengths:
- The method to estimate semantic entropy by linear probing is simple to implement and alleviates repeated sampling at test-time. 
- The experiments and ablations are thorough to demonstrate the effectiveness of the proposed method. 
- The overall organization of the paper is clear and coherent.

weaknesses:
- **Motivation of SEP Efficiency**: The primary motivation for SEP, as stated in the paper, is that it is "cheap" and significantly enhances computational efficiency compared to the baseline semantic entropy (SE). However, this claim appears to be poorly substantiated. The comparison between a training-based method (SEP) and a test-time method (SE) is insufficiently justified. For a fair comparison, it might be better to provide analysis of the overall cost: SE with sampling v.s. the total cost (training and inference) of SEP.

- **Improvement in Long Generation Experiments**: The long generation experiments could be improved. Currently, long generation is primarily conducted with prompts that request "a single brief but complete sentence," resulting in relatively short responses. This approach diverges from recent practices, where models are often prompted to generate detailed reasoning or more elaborate outputs for long-form generation.

questions:
- To calculate semantic entropy (L185), how is the number of clusters K determined? 
- Can authors further discuss why "the discrete variant of SE" should be used instead of the plain version Eq (1)? How is the "fraction of generations in the cluster" related to "avoid problems for generation of different length" (L193)?
- In Sec 5, it seems that the prompt "single brief but complete sentence" also promotes short responses. Does a Chain-of-Thought style prompt work for the long-form answer generation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730859929918

thread_type:
official_review

<thread_object>
creator: reviewer_6thf

<content>
summary:
This paper introduces a technique called Semantic Entropy Probes (SEP) for detecting hallucinations in large language models (LLMs). SEP leverages the internal states of LLMs to predict the semantic entropy (SE) of their responses. By doing so, the method empirically achieves an accurate approximation of SE without the computational overhead of multi-sampling. Furthermore, SEP effectively identifies model hallucinations in both in-distribution and out-of-distribution settings.

strengths:
* This paper addresses a timely and significant research problem in LLMs. Detecting hallucinations is crucial for ensuring the safety of LLM applications in real-world scenarios.
* The experiments are extensive and compelling. The authors conduct experiments on and report results for several popular open-source models.
* The paper is generally well-written and easy to follow.

weaknesses:
* I think the main weakness of this work is the lack of clear motivation and explanation for the outcomes. The SEP method appears to simply switch the prediction goal of previous internal state detection methods from accuracy to semantic entropy (SE), which seems like a simple 'a+b' combination. Although the results are promising, I am still expecting more insights from the experimental results regarding the mechanisms of LLMs. For example, why is it possible to use internal states to predict SE? And why does switching the prediction goal from accuracy to SE result in better performance in out-of-distribution settings? A clearer explanation of these questions would further enhance the solidness of this work and our understanding of LLMs.
* An obvious limitation is the applicability of the proposed method to black-box LLMs. I recommend that the authors discuss this in the limitations section.
* Most of the experiments compare the proposed method with accuracy-based probing methods. However, there are insufficient details provided to describe these counterpart methods. It would be more helpful to provide a comprehensive introduction to these methods for readers who are unfamiliar with them.

questions:
1. Do authors have a clear explanation of why it is feasible to predict semantic entropy (SE) from internal states and why SE could serve as a better supervisory signal than accuracy in out-of-distribution training settings?
2. One of the advantages of SEP is that it does not rely on ground truth labels from the dataset. However, as mentioned by the authors, there are also existing works [1,2] that predict the trustworthiness of LLMs from internal states using unsupervised methods. I am curious about how these methods would perform under the same settings as SEP.


---
[1] Discovering Latent Knowledge in Language Models Without Supervision
[2] Representation Engineering: A Top-Down Approach to AI Transparency
</content>
</thread_object>

</thread>
<thread>
creation date:
1732119446527

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Response to All Reviewers

comment:
We thank all reviewers for their thoughtful reviews, hard work, and valuable suggestions, which we believe have already strengthened our submission. We were glad to see you all appreciate that our results demonstrate that semantic entropy probes provide a cheap and effective approximation of full semantic entropy, outperforming more conventional accuracy-based probing methods in terms of generalization performance.

We particularly appreciate the positive recognition of many aspects of our work, such as our experimental evaluations and ablations, which you have highlighted as ‘extensive and compelling’ (6ThF), ‘thorough’ (toQ8), ‘comprehensive’ (eRJ8), ‘reasonably detailed’ (BxzX) with ‘clear’ (BxZx) findings, and as ‘demonstrat[ing] the effectiveness of the proposed method’ (toQ8). We were further glad to see you felt our paper ‘addresses a timely and significant research problem’ (6ThF) with ‘intuitive and clear’ (eRJ8) motivation, positively highlighting the simplicity of our method and its implementation (6ThF, toQ8, eRJ8, BxzX), as well as recognizing our draft as 'well-written' (6ThF, BxzX, eRJ8), 'clear and coherent' (toQ8), and 'easy to follow' (6Thf, BxZX).

Following your excellent feedback, we have made a number of improvements to our submission. In particular:


* Add a demonstration of real-world computational savings when using SEPs over full SE computation. We observe a 1400 fold speedup in terms of computation time (Appendix A, ‘Computational Costs for SEPs and SE’).
* Add results comparing layer-concatenated SEPs to vanilla SEPs (Appendix B.3).
* Add a demonstration of how SEP predictions can improve the accuracy of models by refusing to answer for the most uncertain set of model generations (Appendix A, ‘Rejection-Accuracy Curve for SE’).
* Improve discussion of why SEPs generalize better than accuracy probes in Section 8.
* Improve exhibition of related work on probing in Section 2.

We further hope to address the individual questions and concerns that you have raised in our replies to your reviews below. With this rebuttal, we have also updated the paper draft based on your helpful feedback. For your convenience, we have highlighted all changes using a blue font color, and will undo this for the camera ready version. Again, thank you for your reviews, and we hope for active discussion with you to alleviate any remaining concerns over the next days!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work proposes Semantic Entropy Probes (SEPs), a method for uncertainty quantification in Large Language Models (LLMs) that approximates semantic entropy directly from hidden states, avoiding the computational overhead of sampling multiple generations. Experiments have demonstrated that SEPs retain high performance in detecting hallucinations, generalize well to out-of-distribution data, and significantly reduce the cost of semantic uncertainty quantification during inference. During the rebuttal phase, the authors provided additional experiments and explanations to address the reviewers' concerns. While many of these concerns have been successfully addressed, some remain unresolved. First, the reviewer still has significant concerns regarding the performance of accuracy probing, which outperforms the proposed Semantic Entropy Probe on in-distribution data. Although the authors presented experiments highlighting SEP’s advantages on out-of-distribution data, the reviewer remains concerned due to the lack of a solid explanation for why and how this occurs. Second, generating semantic entropy during training incurs substantial overhead due to the cost of providing semantic entropy ground truth. Therefore, we have decided not to accept this work in its current state.