LLM-Check: Investigating Detection of Hallucinations in Large Language Models
Authors: Gaurang Sriramanan
Siddhant Bharti
Vinu Sankar Sadasivan
Shoumik Saha
Priyatham Kattakinda
Soheil Feizi
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Hallucinations in Language Models
Hallucination Detection
Eigen-analysis of LM Embeddings
Primary area: natural_language_processing
TL;DR: In this work, we study hallucination detection in Large Language Models by analyzing their internal hidden states, attention maps and output prediction probabilities.

Abstract: While Large Language Models (LLMs) have become immensely popular due to their outstanding performance on a broad range of tasks, these models are prone to producing hallucinations— outputs that are fallacious or fabricated yet often appear plausible or tenable at a glance. In this paper, we conduct a comprehensive investigation into the nature of hallucinations within LLMs and furthermore explore effective techniques for detecting such inaccuracies in various real-world settings. Prior approaches to detect hallucinations in LLM outputs, such as consistency checks or retrieval-based methods, typically assume access to multiple model responses or large databases. These techniques, however, tend to be computationally expensive in practice, thereby limiting their applicability to real-time analysis. In contrast, in this work, we seek to identify hallucinations within a single response in both white-box and black-box settings by analyzing the internal hidden states, attention maps, and output prediction probabilities of an auxiliary LLM. In addition, we also study hallucination detection in scenarios where ground-truth references are also available, such as in the setting of Retrieval-Augmented Generation (RAG). We demonstrate that the proposed detection methods are extremely compute-efficient, with speedups of up to 45x and 450x over other baselines, while achieving significant improvements in detection performance over diverse datasets.

LLM-Check: Investigating Detection of Hallucinations in Large Language Models
Gaurang Sriramanan gaurangs@cs umd. edu
Siddhant Bharti sbharti@cs. umd. edu
Vinu Sankar Sadasivan vinuêcs umd. edu
Shoumik Saha smksaha@cs. umd. edu
Priyatham Kattakinda pkattaki @umd .du
Soheil Feizi sfeizi@cs. umd. edu
Department of Computer Science University of Maryland, College Park, USA
Abstract
While Large Language Models (LLMs) have become immensely popular due to their outstanding performance broad range of tasks, these models prone to producing hallucinations- outputs that are fallacious or fabricated yet often appear plausible or tenable a glance. In this paper, we conduct a comprehensive investigation into the nature of hallucinations within LLMs and furthermore explore effective techniques for detecting such inaccuracies in various real-world settings. Prior approaches to detect hallucinations in LLM outputs, such as consistency checks or retrieval-basee methods, typically assume access to multiple model responses or large databases. These techniques, however, tend to be com- putationally expensive in practice, thereby limiting their applicability to real-time analysis. In contrast, in this work, we seek to identify hallucinations within single response in both white-box and black-box settings by analyzing the internal hidden states, attention maps, and output prediction probabilities of auxiliary LLM. addition, we also study hallucination detection in scenarios where ground-trute truth references also available, such in the setting of Retrieval- Auggemeeee Gen- cration (RAG). We demonstrate that the proposed detection methods are extremely compute-efin compute-efficient, with speedups of up to 45x and 450x over other baselines, while achieving significant improvements detection performance over diverse datasets.
Introduction
Over the past few years, Large Language Models (LLMs) such as GPT-4 [Achiam et al. 2023 and Llama [Tourron et al., 2023) have become immensely popular due to their excellent performance natural language inference. nusstion-answering and summarization tasks. Nonetheless, it has been observed that these models often produce outputs that are fallacious, incorporating fictional or insub- stantial details that can be partly misleading entirely fabricated [Ji al. 2023 Zhang et al. 2023 Moreover, it is often observed that such model generations seem plausible, appearing tenable before further scrutiny. This phenomenon in LLMs, known hallucinations, poses significant challenge to their deployment in applications where accuracy and reliability are critical. Thus, the detection and mitigation of hallucinations in LLMs have been subjects of critical interest. Furthermore, the relative difficulty of hallucination detection can vary significantly in white-box or black-box settings, depend- ing on the extent to which access is available to the original L.M that utilized for generation.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Uncertainty estimation based on output logit based metrics such as perplexity or logit entropy have been used towards analyzing errors in structured prediction and generations produced by language models [Malinin and Gales 2021 Kuhn et al. 2023]. Furthermore, more fine-grained approach can be used by considering the relevance of specific tokens in the generated text [Duan et al. [023] J or by directly obtaining quality or score assessments leveraging the zero-shot capabilities of LLMS
[Fu et al. 2023]. depending upon the task specifications. As an alternative approach, Waang et al.) 2023] showed that self-consistency can be incorporated into the decoding strategy to improve upon naive
in the setting of thain-of-thoughh prompting, and demonstrate notable improvements in arithmetic and common-sense reasoning benchmarks. Specific to the setting of hallucination detection, consistency-basee methods like SelfCheckGPT [Manakul et al. 2023 and INSIDE [Chen et al., 2024 proposed to analyze multiple responses of the same model given common prompt. Indeed, SelfCheckGPT |Manakul et al.) 2023] relies upon the fact that when LLMs trained given concept, its output generations tend to he more consistent, while hallucinated information is less likely to be repeatedly generated across multiple stochastically sampled responses. Notably, SelfCheckGPT relies only upon black-box based API access models such as GPT--355 [Brwwn et al.) 2020 to assess hallucinations in a given sample sentence. Similarly. Chen et al. [2024] proposed the INSIDE detection method based on eigenvalue analysis of the covariance matrix over model embeddings generated across multiple responses of the same prompt, enabling hallucination detection at a population level instead of sample-level detection. Another setting studied towards the detection of hallucinations is that of methods using Retrieval Augmented Generation (RAG) (Lewis)
ct al. 202 I Guo al. 2022] Varshney et al.) 2023 Niu et al. 2024), which assume access to large external database of grounded information, pipelined with a fact-verification system.
A key limitation of these detectors is the assumption that they have access multiple model generations or a large reference dataset. Consistency-basee methods assume hallucinations are rare, and most generations are factually correct, even though a given LLM model often repeats similar errors. Moreover, these techniques induce additional overheads to generate multiple additional responses inference time, or require large memory overheads to parallelize generations. Retrieval-bar based techniques rely on strong fact-verificatin and retrieval techniques, and may not be scalable for large databases due the considerable inference costs involved overall. Furthermore, incorporating human-leve annotation or manual fact checking is often not scalable in this setting.
This leads our research question- Is it possible to identify the presence of hallucinations within a single L.M response in both white-box and black-box settings without incurring additional compu- tational overheads training or inference time? Towards this, given the practical constraints involved in this problem setting, we seek to analyze the efficacy of hallucination detection using analyses of the internal attention kernel maps, hidden activations and output prediction probabilities of an LLM itself, using an auxiliary substitute model if white-box access is unavailable. Furthermore, we study the effec- tiveness of different hallucination detection metrics both in the knowlede setting as well as RAG- based settings where ground-trute reference materials are assumed to be available at inference time. We demonstrate that the detection methods proposed are extremely efficient by using mucher-forcing without additional overheads, and is accompanied by significant gains in detection performance relative to baseline approaches. In summary, we make the following contributions in this work:
We analyze hallucination detection within a single LLM response using its correspond- ing internal attention kernel maps, hidden activations and output prediction probabilities. We utilize these diversified scoring methods from different model components to poten- tially maximize the capture of hallucinations amongst its various forms without incurring computational overheads at training or inference time.
We analyze the problem in resource detection (FAVA [Mishra et al.
settings across diverse datasets: from zero- 2024]), to the case when multiple model responses are available (SelfCheck [Manakul et al.
2023 D, or when ground- references are indeed
available at inference time (RAGTruth Niu et al. 2024).
We demonstrate that the proposed method LLM-Check is indeed effective over these diverse hallucination detection settings, and achieves notable improvements over existing baselines.
Furthermore, we demonstrate such performance gains while requiring only a fraction of the computational cost (speedups up to 45x and 450x), as the proposed method does not utilize multiple model generations or extensive external databases.
'The codebase for LLM-Check is available at this URL

---

2 Related Works
Manakul et al. 2023] introduced SeliCheckGPT, a suite of hallucination detection methods in a
zero-resource gray-box/black-bor setting to assess the veracity of LLM response, assuming only access to the output probability distributions. This is highly practical in real-world usage, since API calls often impede access to internal model activations. On the other hand, INSIDE (Chen et al.) 2024 measures the self-consistency of the hidden states across multiple independent responses of an LLM, by performing a centered eigen-analysis of the covariance matrix of these hidden states. Notably, INSIDE follows a slightly different evaluation framework from other techniques which generally rely upon human annotations, wherein towards assessing the correctness of model response INSIDE utilizes either ROUGE-L score or BERT-similarit scores being larger than a threshold with respect ground-truth. However, ROUGE-L is n-gram based method that evaluates the longest common subsequence between the model response and the ground-truth. Thus, several hallucinatory words can be incorporated without considerable change in the ROUGE-L score overall.
Kadavath et al. [002]] proposed to use the LLM itself to predict the probability of the generated response being True, referred to Self-Prompt; this is primarily relevant for the context of mulliple-choicc--uestton where the model explicitly has all the options in context. Furthermore, they also train models to predict the probability that given multiple-choicc-quettii
will be correctly answered by calibrating its internal confidence values. (Azaria and Mitchell (203) proposed to train feedforward neural networks on a True-False dataset of relatively short, simple sentences such as "The zebra uses flying for locomotion." This method is thus a white-box detection technique which also requires supervised training data to train the feedforward network (as noted by Manakul et al. (023 D; in contrast, we do not incorporate training of any sub-network or the LLM itself. Furthermore, it is fairly non-trivial to extend the technique towards more general settings such as when external references or multiple model responses are available at inference time. Yuksekgonul et al. (2024) investigate factual errors made by LLMs by modeling it as a constraint-satiifaction problem. Indeed, the factual query is specified by considering a sequence of Constraint-toke and using a subsequent Verifier (such as ExactMatch or WikiData Search) to determine the satisfaction for each such constraint. This method highly pertinent in settings where a fact-checking service is available for ready oracle-acces that can provide feedback and annotations in real-time.
the hidden activations of intermediate layers of LLMs
Mishra al. (024) introduced fine-grained hallucination detection dataset (FAVA-Bench), wherein a novel hierarchy of hallucination types is identified and analyzed comprehensively. The authors first generate a synthetic hallucination training dataset by inserting specific errors using ChatGPT, and then fine-tune a Llama-2 model (FAVA) to explicitly detect these fine- fine-grained hallucinations. The work also introduces numan-annotated dataset where the specific hallucination types are recorded with specified hallucination spans. We make extensive use of the FAVA datasets so released to compare multiple baselines on a common test-bed. Though retrieval augmented generation (RAG) helps to mitigate hallucination in LLMs, ungrounded generations still persist [Shuster et al. 2021 1 Li et al. 2024]. Recently, Wu et al. [2023) showed how popular LLMs still hallucinate on different tasks even with the integration of RAG, and compiled the RAGTruth dataset with span-level human annotation. Furthermore, they fine-tune an LLM on their training data to detect hallucinations and its span, but thereby incur a significant computational overhead at training time. Due to paucity of space, we extend our discussion of related works further in Section-
Df of the Appendix.
3 Taxonomy and Formalisms for Hallucination Detection
Notation:
Let V denote the vocabulary of an LLM, and let V denote the set of all possible sequences obtainable using the same vocabulary. We denote individual token as z E V. and sequence of tokens as x E V*, For an autoregressive model f. let (-(+ denote the next-token probability distribution based on input Moreover, given two token sequences x = (x1 ... x.,) and & = (#12.... ),,, let x + & = (2102... In 02 ... ......... fm) denote their ordered concatenation.
Given a query or prompt Xp (x1 In), ,,), let the LLM response of model f be denoted as ). ..). Thus, the broad goal of detection is to determine the presence or absence of hallucination in the output response when the input prompt is Xp- In practice, LLMs such as
X = (In=1 ...
chat-models require additional system prompts to be included along with the user input; here we assume these are contained within Xp for simplicity.
3

---

Taxonomy Hallucination Detection
Train Indep.
Single Response
Sample Specifie
Retrieval Indep.
Method
Efficient
Are References Avaiiable?
FAVA
x
SeliCheckGPT
4
x
x
r
Yos
INSIDE
x
v
x
v
Responses Noeded??
Are Multiple Model
FAUA RAGTruth
RAGTruth
x
V
x
v
x
L.M-Check (ours)
/
/
/
FAN
SePCheckGPT
White Box
Black Box
Table 1: Overview of Hallucination Detection Methods with FAVA, SelfCheckGPT, INSIDE, and RAGTruth. Key- Train Indep: does not require model fine- tuning/training. Single Response: does not require multiple LLM responses, Efficient: compute & memory efficient, Sample Specific: sample-wise instead of population-level detection, Retrieval Indep: does not require RAG or reference databases.
Figure I: Taxonomy for different settings of hallucination detection, and the dif- ferent datasets and data splits that corre- spond to each such setting.
We present broad taxonomy of the different settings in which hallucination detection is studied in Figure I Here, we discuss and analyze this classification in detail:
I. Without External References: First, we consider the setting wherein external references available to provide additional context to the LLM. Within this scenario, we can make a further classi- fication based on whether multiple model responses be potentially generated for the same prompt:
IA. Single Model Response: Here, only the original prompt Xp and fixed model output x is assumed to be available. Thus, in this setting, hallucination detection using LLMs relies upon the tacit assumption that the query topic is covered to some reasonable extent the pre-training data of the LLM. We comprehensively analyze this setting using the FAVA Nluman-Annotater dataset to compare different approaches. Moreover, Mishra et al. 2024 provide a fine-tuned Llama-2-chat model, that can be used to annotate specific error types in the given response x. In our evaluations, binarize these outputs to only detect the presence or absence of hallucinations.
IB. Multiple Model Responses: Given prompt Xp and fixed model output x, SelfCheckGPT
additionally generates stochastically sampled responses x' xx2 ...... x* in order to compute different metrics such as BertScore, n a-gram, cneeston-Answer and LLM-prompt score with respect to x, to assess the absence or presence of hallucinations in x. Notably, these scores can be computed with a black-box model with only external API access. On the other hand, INSIDE only considers the input prompt Xp. and computes decoppoicoooiiiioon of the covariance matrix of hidden activations across multiple generated samples x², x2....xx This is used to then statistically infer the presence of hallucinations within this set of generations, based on a possible lack of self-consistency at a "population level" between samples within this specific set of k samples. Thus, INSIDE performs hallucination detection at a population-level of all the generated samples simultaneously, rather than detection given fixed (single) output response
II. With External References: In this setting, a broad-rangine set of data retrieved from external sources are used to affix additional context to assess the validity of the model response x. This is a highly pertinent setting, as the retrieved data can consist of up-to-date information from sources deemed reliable or trustworthy. Furthermore, with the inclusion of external data, LLMs can be used on topics not completely covered its original pretraining data in a straightforward manner, without requiring computationally intensive rounds of additional training or fine-tuning.
IIA. White-Box: For given prompt Xp. consider hallucination detection in an output response x that is generated by a given LLM f. If the original LLM f is available and accessible to compute the internal model activations. the setting is considered to be "white-boxx.. For instance, INSIDE provides the population-lewe detection using the original source model so used, and thus falls in the family of white-box detection techniques.
IIA. Black-Box: If the original LLM ff that generated the response x for prompt Xp is no longer avail- able or inaccessible, an auxiliary substitute LLM f can be utilized (such as open-source LLMs like llama-2) to compute scores with internal activations and attention kernel maps. This can be achieved by using teacher-foreing on the substitute LLM, and this setting is considered to be "black-box".

---

Voa
Layer Normaize
op Feedfoneard Layer
Pelluctiation Not Detected
Durra Eqgain neural Ietwoks wilh Lawmrrr
Salvel
conquaer poggae desigeed 13 mine: now numen work....
Layer Normaize
Attention Score
+
Suore freehce
Yes
Amantion Layer
LAAASE
Lianga Seth-beet Kmme
Halleciaaton Not Detecttd
Figure 2: Schematic of detection pipeline using Eigenvalue Analysis of internal LLM representations.
4 Proposed Method
In this section, we describe our method LLM-Check in detail. Given that we wish to perform hallucination detection without any training inference time overheads, propose analyze all model-related latent and output observables available with a single forward-pass of the LLM. For broad family of autoregressive models using the Transformer architecture [Vaswani al. 2017). the input token embeddings are transformed a sequence of hidden representations at each layer of the model. For LLM with internal layers, let H2 denote the hidden representations at layer I E L}. The hidden representation at layer (1 + 1) is then computed as follows:
H H, + A1+1 + M
where A1+1 and M1+1 denote the attention and MLP contributions respectively. Additional operations such as noyraiaormaiizttion are commonly carried out, but we omit their details here to be concise. The attention contributions represent the updates the representation of token at given layer by attending to all token representations of the previous layer. In particular, if Q, K, V denote the Query, Key and Values the attention mechanism, the attention contribution is given by computing the kernel similarity of Q and K, and masking the values V with this kernel, and is simplified to be written as:
Q Q KJ
A Ker -Softmar
V
where de is the dimension of the keys and queries. Generally, LLMs utilize wherein the kernel maps are individually computed over each head and concatenated, and then cast back the appropriate dimension using Output projection matrices before being aggregated together. We note importantly that for auto-regressive LLMs, the attention kernel maps are lower triangular for each attention head, since given token can only attend the internal model representations of previous tokens in the sequence.
attention,
While LLMs do hallucinate, they still have significantly appreciable degree of world-knowledge grounded on truthful facts encountered in the training stage, which reflected in the fact that hallucinations absent in some of the autoregressively generated sample outputs when multiple model responses are considered for the same prompt. In this work, we propose to directly analyze the variations in model characteristics between truthful and hallucinated examples by examining the rich semantic representations present in internal LLM representations, since we observe that the LLM can often generate the truthful version, albeit with potentially lower frequency. Thus, we posit that the model is in fact sensitive to the presence of non-factual, hallucinated information in responses which would otherwise be truthful, based on grounded precepts encountered in its training and that this can then be efficiently leveraged to perform hallucination detection within a single model response itself.
1. Eigenvalue Analysis of Internal LLM Representations
The differences model sensitivity to hallucinated or truthful information would reflect the rich semantic representations present in the hidden activations and the pattern of attention maps across different token representations. the cross-covariance for hidden representations, and the kernel similarity map of self-attention across different tokens, since these form the foremost salient characteristics of the L.LM itself. Given that wish to do hallucination detection in black-box settings as well, we utilize teacher-foreing to
quantitatively capture these variations, we propose to analyze
5

---

obtain representations corresponding to Xp @ x, wherein system prompt tokens delineate the original prompt and model output response, Thus. for given layer/ E {I..... L}. let H denote the hidden representations for input Xp @ x. If the token embedding dimension is d and Xp θ x consists of m tokens, each hidden representation matrix H of shape (d m.) To be highly compute-efficient and enable real-time detection, we distill a simple yet salient scalar quantity the mean deg-determinant from these variations in hidden representations using eigen-analysis. Theoretically, this is well-motivated as the eigenvalues and singular values capture the interaction in latent space between the different token representations corresponding to hallucinated and truthful sample sequences. We can compute the (m m) covariance matrix > and compute its log- determeeennn as follows:
m
E H"H
log det. (E²) log II o
log of = log di
where o; are the singular values of H. order to remove the explicit dependence on input-length, we propose to utilize the mean -og-determinant of Σ² that is, Σi= log o,, which we term as Hidden Score. Here, we note crucial differences from the approach of INSIDE, which computes the centered covariance matrix across multiple model responses to check self-consistency of the set of responses, and performs ballucination detection at the population level for a set of responses, rather than a given fixed model response x. We repeat this procedure across hidden representations of different layers; please refer to Section-Foo the Appendix for detailed layer-wise analysis.
As alternative approach, now move towards utilizing internal components of the attention mechanism, namely the kernel similarity maps used within self-attention heads, to analyze hallucina- tions. If the number of self-attention heads utilized at each layer in the LLM architecture is the attention maps can be represented as tensors of the shape (a X m X m) for an input sequence Xp i x of length If Ker, represents the kemel similarity map corresponding to attention head i, Ker, is a lower- teven-trinnglar square matrix of size (m X m) for each i E {1...., a}. Since it is swwer-tiinngllr, the eigenvalues of Ker, are just the values the principal matrix diagonal, since they the roots of the characteristic equation (eer - A/) = 0. Furthermore, since LLMS use the Softmax kernel, all the eigenvalues are also provably non-negative. We can thus obtain the log-determinant of Ker; without using SVD or eigen-decomposition:
log (Ker)) = Σ log Ker!' j=1
where Ker!' represents the (j.j) diagonal entry of the square matrix Ker, Again, we obtain the mean log-determinant for each attention head, and aggregate them to obtain an alternative hallucina- tion measure, which we call Attention Score. Here, we note that computing the Attention Score is ex- tremely efficient, since it does not require any explicit eigen gigen-decomposition. The overall pipeline used for hallucination detection using the Hidden Score and Attention Score is visualized in Figure 2 We further present detailed illustrative example using the Attention Score in Section- of the Appendix.
In black-box settings, we propose to utilize an auxiliary autoregressive LLM f. where we use teacher-foreing to obtain the scores from the auxiliary LLM instead to perform detection. We also note that since the MLP component of transformer block is applied to each position separately and identically (Vaswani et al., [017]. it does not capture cross-token interactions, and thus we do not consider this component for detection itself.
2. Output Token Uncertainty Quantification
Alongside the internal model representations, we now consider the probability distribution (-(-)x) over the output response tokens induced by the LLM. Given that LLMs are trained with next-token prediction, the probability distribution pr(-(x) for a given token can be highly salient toward the relative choices available for completion. Indeed, specifically for factual sentences that are part of topics covered in the training dataset of an LLM, we expect that its completions are such that the model likelihood of the output sequence is high. On the other hand, if the sentences arise from distribution far different from that encountered in the training regime, the model outputs are more likely to be ungrounded in nature due the lack of specificity between different tokens in the vocabulary.
We thus present the output token-level analysis techniques for identifying hallucinations in LLM responses. We propose to analyze the use of perplexity of the response x, given the prompt Xp.

---

potential measure for detecting hallucinations:
m
PPL(x) = exp (-m--- +1 ly (2) @ xx())
Furthermore, at each token position in the output response, the probability distribution over all tokens (not just the token finally selected during autoregressive generation). We can further refine logil-entropy-based measures by considering only the top-k tokens rather than all possible tokens since the contribution from tokens of extremely low probability is likely noisy and non-salient. We refer to this as the Logit Entropy score, defined
consider the mean entropy of
1
LogitEEttxx,
ps ps(c? Ixp @ log @ xxx)
T
where oa is the token with jth highest probability at output position conditioned the prefix Xp i Xzi However, given that the entire sentence may not be hallucinatory, the length-average scores may not be as salient as we desire to predict hallucinations accurately. Thus, in light of this, we consider the Windowed Logit Entropy score, which computes the logit entropy scores across overlapping token win- dows and returns the logit entropy of the window with the maximum score. This score thereby sen- sitive to short sequences of hallucinatory material and is not diluted by sequence length normalization.
We propose these two distinct lines of analysis towards hallucination detection, which collectively term as LLM- M.CCheck,, in order to adequately capture the extremely heterogeneous and diversified forms of hallucination displayed by modern Large Language Models over different domains. Towards this, the Figen-analysis of internal LLM representations helps highlight the consistent pattern of modifications the hidden states and the model attention across different token representations in latent space when hallucinations are present in model responses as compared to truthful, grounded responses. On the other hand, the uncertainty quantification of the output tokens helps analyze hallucinations based on the likelihood assigned by the model on the actual tokens predicted at specific point in the sequence generated auto-regressively.
Before analyzing the quantitative experimental results in the following section, we first present qualitative comparisons of the proposed method with the most pertinent baselines in Table We present various trade-offs and advantages in the table such as to whether the method requires fine- tuning of L.LM, if it inherently requires multiple model responses detect hallucinations, if the method is computationally efficient, if it performs detection on per-sample basis or at a population level, and whether the method is inherently dependent on retrieval during inference time. We observe that our proposed approach is indeed qualitatively advantagcous in all of the aspects so mentioned.
5 Experimental Results
To analyze the efficacy of the proposed hallucination detection measures and perform a comparative analysis with existing basclines, we utilize the taxonomy and datasets presented in Figure For the setting of detection without external references with a single model response, we utilize the FAVA-Ammttation [Mishra et al. 2024] dataset, wherein we deem passages containing any form of hallucination to be hallucinated overall. Furthermore, we utilize the fine-grained classification of different forms of hallucination as annotated in the FAVA dataset to analyze the efficacy of the detection measures across these different types. In the setting of detection without external references with multiple model responses, we utilize the SelfCheckGPT dataset [Manakul et al. 2023], consisting of 1908 sentence level annotated samples, alongside 20 stochastically generated responses from GPT (text-davinci-0033 Moreover, to fairly compare actoss different detection methods, we also augment the originally released version of the FAVA-Annotation dataset by generating 20 additional responses from GPT 3 using the original prompt set. By doing so, we are able to obtain new baselines for SelfCheckGPT and INSIDE on the FAVA-Annotation dataset using the multiple responses so generated.
Datasets and Detection Details:
For our evaluations in the setting of hallucination detection with external references assumed available, we primarily consider the RAGTruth dataset [Wu et al.) 2023). In detail, we use the Summarization subset which was created by prompting six different L.LMs with CNN/Daily Mail
dataset and recent news, resulting in more than 5K samples. We use the span-level human annotation
7

---

Model
Measure
AUROC
Accuracy
TPR @ 5% FPR
Fl Score
Llama-2-4
Self-Prompt
50.30
50.30
66.53
Llama-2-7B
FAVA Model
53.29
53.29
43.88
Llama-2-7B
SeliCheckGPT-Prompt
50.08
54.19
67.24
Llama-2-7B
INSIDE
59.03
57.98
13.17
39.66
LLM-Check (Ours)
PPL Score
53.22
58.68
3.59
68.33
Window Entropy
56.90
56.59
2.99
42.52
Logit Entropy
53.80
55.99
2.99
56.73
Llama-2-7B
Hidden Score (LY 20)
58.44
58.08
11.98
59.66
Attn Score (LY 21)
72.34
67.96
14.97
69.27
PPL Score
53.96
56.89
3,59
64.20
Window Entropy
55.24
58.38
5,99
66.02
Logit Entropy
52.29
55.69
1.80
57.31
Vicuna-7B
Hidden Score (LY 15)
58.22
59.28
10.18
66.99
Attn Score (LY 19)
71.69
66.47
24.55
62.00
PPL Score
53.22
58.68
3.59
67.40
Window Entropy
56.90
56.59
2.99
55.52
Logit Entropy
53.80
55.99
2.99
56.27
Llama-3-8B
Hidden Score (LYY 15)
57.10
57.78
10.78
65.38
Attn Score (LY 23)
68.19
65.87
15.57
70.53
Table 2: Detection results on the FAVA-Annotation Dataset wherein no External References are available. For methods such as INSIDE and SeliCheckGPT-Promp.. we utilize the multiple model responses generated by GPT-3.
provided in the dataset and considered sample is hallucinated if there exists any hallucinated span the response. Additionally, we also consider white- box and black-box detection settings for the RAGTruth dataset, depending the original L.LM model used for generating the responses. Apart from RAGTruth, we also consider the training data-split of FAVA wherein several references are provided, and specific hallucination errors edited in synthetically obtain pairs of ground-truth responses with and without hallucinations. We consider 500-sample subset of the FAVA train-data towards this setting of detection with external references.
Models and Metries: We utilize popular open-source LLM chat-models such as Llama-2-1 Chat [Touvron al. 2023], Vieuna [Zheng al. 2023 and -iintrct [ Metal 2024
autoregressive LLMs with their corresponding tokenizers provided by HuggingFace Wolf et al. 2020). for both white-box and black-box evaluations. We compute the suite of hallucination detection scores proposed in Section-4] which we collectively term as LLM-Check, and present standard threshold based detection metrics such Accuracy, Area under the ROC curve (AUROC), True Positive Rate (TPR) at low (5%) False Positive Rate (FPR) and F1 score. In each setting, we consider balanced datasets, with equal number of samples with and without hallucinations present, except for the SelfCheck dataset where we follow the setup utilized in the original paper.
5.1 Detection Results on Datasets with no External References as Context
First, we analyze the setting of hallucination detection when no grounded references are provided as context. We present the consolidated detection metrics in Table Jusing the Annotttnntoiion dataset. To evaluate methods such as SelfCheckGPT and INSIDE, we make use of the stochastically sampled responses of GPT-3 that we augment the original dataset. We observe that in this zero-context ting, many of the baselines perform fairly poorly in comparison to LLM-Check. We remark that while entropy-bar based scores were used for comparisons in the SelfCheckGPT paper, the best- rest-performing variant was seen to be SelfCheckGPT. Promppp using GPT-3 itself, which was used to generate the original hallucinations, Here, we observe that when using Llama2-7B for SelfCheokGPT-Promee Promp,,, results are much worse. This might be due to the fact that SelfCheckGPT is less effective at the pas- sage level after aggregation is performed over the individual sentences. We observe that the Attention Scores the most effective across different models, and obtains significantly higher detection scores.

---

Model
Method
AUC-PR
Accuracy
TPR @ 5% FPR
Llama-2
SelfCheck
72.84
51,44
4.81
Llama-3
SelfCheck
75.06
54.84
5.10
LLM- LLMM-hhck (Ours)
Llama-2
Attn Score
80.04
58.91
9.41
Llama-2
Prompt
79.46
61.21
8.76
Llama-3
Attn Score
79.96
58.92
9.48
Llama-3
Prompt 78.49
58.54
7.11
Table 3:
Detection results on the SelfCheckGPT Dataset wherein no External References are available, but when multiple model responses are indeed available. We report the results using the same data- split as the original SelfCheck [Manakul et al., 2023) paper, and thus the number of positive and negative samples are imbalanced in this table: 1392/1908 samples have hallucinations present.
We further analyze the layer-wise performance of scores obtained from internal model representations in Section-Foot the Appendix. We observe that while the Attention Scores are generally the most effective, particularly around Layer 20, the oscillation in detection performance across different layers can be large, relative to the case of using the Hidden Score which is fairly more stable across layers.
In Table 3 we further analyze the performance of LLM-Check against SelfCheckGPT on the SelfCheckGPT Dataset, where multiple GPT-3 generated responses are available in addition to the original model response. We follow the setup used in the original paper with imbalanced classes, and similarly also report Area under the Deocision-Recall curve in in-place of AUROC. We again find that the Attention scores provide better detection performance over all three metries of AUC-PR, Accuracy and TPR @5%FPR despite not utilizing the additional model responses included in the dataset. Here, we present Prompt-variant LLM-Checc, where instead of prompting the LLM to output "Yes" or "No" based on the additional model responses and then hard-coding to 0/1 scores as in SelfCheckGPT-Prompe we compute Attention Scores of modified prompts which include the addi- tionally generated model responses, and then aggregate over them done in Sel/CheckGPT-Promee We find that LLM-Check with Attention- Attention-Prompl performs similarly well, though the inference time is higher in this setting, due to the iteration over the different model responses.
5.2 Detection Results on Datasets with External References
We now analyze the setting wherein external references are assumed to be available to the model inference time. In Table we present evaluations obtained using a Llama-2-7b model in both white-box and black-box settings. For the black-box setting, we considered four different models Llama-2-13b, Llama-2-70b, GPT-4, Mistral-7b. In the white-box setting, we observe that the Hidden Score achieves a higher Fl score compared to all other detection methods. The Attention scores are however better in the overall black-box setting, where we compute a weighted average since different models hallucinate at different frequencies on the same prompt data. Additionally, we also observe that LLM-Check performs better on larger models. For example, with Hidden score, our approach obtains 54.11%. 59.67%, and 59.31% AUROC on the 7b, 13b, and 70b variants of Llama-2 respectively. Furthermore, the AUROC increases 61.87% for hallucinations arising from the GPT-4 model.
Lastly, we present the evaluations on 500 examples of the FAVA Train-split in Table Notably, this dataset differs from RAGTruth in that the hallucinations in the generations inserted synthetically using GPT-3, and thus potentially induce changes to the joint-distribution of the sequence level probabilities. Indeed, observe that output-token uncertainty estimates using entropy out-perform even the attention-ban scores in this setting. Thus, we observe that different detection methods may prove optimal depending on the problem setup and underlying data distribution.
5.3 Timm-CCopleeitt Analysis
We compare the overall runtime cost of the proposed detection scores with other baselines using Llama-2-7h Chat model on the FAVA-Annotation dataset on a single Nvidia A5000 GPU in Figure We observe that the Logit and Attention scores are indeed very efficient, while the Hidden Score is slightly slower since uses SVD. We also observe that I.LM-Check is considerably faster than most basclines with speedups of up 45x and 450x, since it only uses model representations with teacher

---

White-box
Black-box
Target Model
Measure
Llama-2-7b
Llama-2-13b
Llama-2-70h
GPT-4
Mistral-7h
Overall
AUROC
54.11
59.67
59.31
61.87
53.68
57.24
Hidden Score
Accuracy
56,33
59.66
58.42
68.52
54.15
57.62
5%FPR
8.14
12.41
9.9
3.7
5.18
8.37
F1 Score
61.51
50.42
66.14
67.86
32.58
47.45
AUROC
53.73
52.46
56.97
52.13
52.1
53.27
Logit
Accuracy
54.07
55.17
$7.92
59.26
54.66
55.79
(Perplexity)
@PRR
7.69
8.97
6.93
0.00
4.15
6.01
FI Score
58.7
50.57
61.26
61.02
43.23
50.45
AUROC
52.08
55.71
52.61
54.58
Logit
Accuracy
53.17
56.9
57.43
59.26
53.89
55.90
(Win Entropy)
TPR@5%FPR
4.98
15.86
7.41
10.36
10.08
Score
53.98
33.68
62.01
54.9
49.29
47.51
AUROC
53.95
51.18
$5.14
50.34
50.43
51.68
Logit
Accuracy
55.43
53.79
57.43
57.41
53.89
54.83
(Log Entropy)
TPR 75FFR
7.24
9.66
4.95
0.00
6.22
6,65
FI Score
53.74
15.09
66.41
60
48.41
42.62
AUROC
54.19
60.05
60.01
63.51
55.37
58.30
Attention
Accuracy
54.52
59,66
60.89
66.67
56.99
59.23
Score
5.88
14.48
12.87
7.41
5.18
9.87
F1 Score
54.5
55.97
55.06
67.8
57.72
57.18
Table Detection on RAGTruth Dataset using Llama-2-7b model in white-box and black-box setting, with the "Overall" column presenting the weighted average results for the black-box models.
Model
Meusure
AUROC
Accuracy
TPR @ 5% FPR
PPL Score
74.20
70.00
26.00
Window Entropy
77.00
72.00
Logit Entropy
74.36
71.00
26.00
Lluma-2
Hidden Score
51.44
54.00
4.00
Score
69.57
66.60
11.60
PPL Score
73.48
68.80
Window Entropy
78.44
72.00
28.00
Satl Frmeg
INSi Mosil
Solt SockSPT
Proupt
INSIIEE
Logui Eaemps
(Oun)
(Dans)
Hollen Stots
(Chon)
Lluma-3
Attn Score
Logit Entropy
79.24
71.91
73.60
68.20
28.00
19.60
Table 5: Detection results on synthetic hallucinations on FAVA Train Split data with External References included.
Figure 3: Averaged runtime analysis of different hallucination detection methods.
forcing, without additional inference time overheads. This runtime analysis for the Eigenvalue based methods represents the total time for computing Attention and Hidden scores from all layers of Llama-2-7b. That is, the Attention score computation for all 32 layers takes 0.22 seconds per example, and the Hidden score computation for all 32 layers requires 2. seconds per example, when averaged over the FAVA Annotation dataset. Since the overhead to compute scores for all layers is so small, we expect that they can be utilized for real-time detection hallucinations in model responses.
Conclusions
In this work, we analyze the problem of detecting hallucinations within single response of an LLM, and propose LLM-Check, an effective suite of techniques that only rely upon the internal hidden representations, attention similarity maps and logit outputs of LLM. We demonstrate the efficacy of LLM-Check over broad-ran rroa--anging settings and diverse datasets: from zero-resource detection to cases where multiple model generations or external databases made available inference time, or with varying access restrictions to the original source L.M. Moreover, we observe that I.L.-Checkk obtains considerable improvements tuning or retraining of LLMs. Furthermore, by utilizing only teacher-forcing at inference time without additional overheads, show that LLM-Check is extremely compute-efficient, requiring only fraction of the runtime compared to other detection baselines, with speedups of up to 45x and 450x.
baseline methods in these settings, without requiring fine-
10

---

7 Acknowledgments
This project was supported in part by grant from an NSF CAREER AWARD 1942230, ONR YIP award N00014-22-1-2271, ARO's Early Career Program Award 310902-00001, Army Grant No. 290,, the NSF award CCF2212458, NSF Award No. 2229885 (NSF Institute for Trustworthy AI in Law and Society, TRAILS), Amazon Research Award and award from Capital One.
References
[1] Achiam, Adler, S. Agarwal, L. Ahmad, Akkaya, F. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] AltMeta. Llama 3 model card. 2024. URL
11ama /aaamma3a/a//////a
blob/main/MODEL, CARD .dd
[3] A. Azaria and Mitchell. The internal state of Ilm knows when it's lying. 2023.
[4] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[5] C. Chen, K. Liu, Z. Chen, Y. Gu, Y. Wu, M. Tao, Z. Fu, and J. Ye. INSIDE: LLMs' internal states retain the power hallucination detection. The Twelfth International Conference on Learning Representations, 2024. URL
net/ffruum htt///ffrmmmid220 12nz1Qbz
[6] J. Duan, H. Cheng, S. Wang, A. Zavalny, C. Wang, R. Xu, B. Kailkhura, and K. Xu. Shifting attention to relevance: Towards the uncertainty estimation of large language models, 2023.
[7] J. Fu, K. Ng, Z. Jiang, and P. Liu. Gptscore: Evaluate as you desire, 2023.
[8] C. Gardent, A. Shimorina, S. Narayan, and L. Perez-Beltrachini. The WebNLG challenge: Generating text from RDF data. J. M. Alonso, A. Bugarín, and Reiter, editors, Proceedings of the 10th International Conference Natural Language Generation, pages 224-133, Santiago de Compostela, Spain, Sept. 2017. Association for Computational Linguistics. doi: 10.18653/ v1/W17-3518. URL
[9] N. M. Guerreiro, E. Voita, and A. Martins. Looking for needle in haystack: A com- prehensive study of hallucinations in neural machine translation. In A. Vlachos and I. Au- genstein, editors, Proceedings of the 17th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, pages 1159-1075. Dubrovnik, Croatia, May 2023. Association for Computational Linguisties. doi: 10. 111111/1/2023..1. www..eaaacm.mm...... URL 2023 otg//223.eaclmmma.nn eacl-main.77
[10] Z. Guo, M. Schlichtkrull, and A. Vlachos. A survey on automated fact-checking, 2022.
[11] K. M. Hermann, T. Kociský, E. Grefenstette, L. Espeholt, W. Kay. M. Suleyman, and P. Blunsom. Teaching machines to read and comprehend. In NIPS, pages 1 1693-1701, 2015. URL.tty 7/papers. nips. cc/paper/5945
waaeching-machingg
to read- and- comprehend
[12] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, J. Bang, A. Madotto, and P. Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12): Mar. 2023. ISSN 1557-7341. doi: 10.1 145/3571730. URL RRLLhttp://// h/tp:///xx doi org/ 1145/ 3571730
[13] S. Kadavath, T. Conerly, Askell, Henighan, Drain, E. Perez, N. Schiefer, Z. Hatfield- Dodds, N. DasSarma, E. Tran-Johnson, S. Johnston, S. El-Showk, Jones, N. Elhage, T. Hume, A. Chen, Y. Bai, S. Bowman, Fort, D. Ganguli, D. Hernandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K. Ndousse, C. Olsson, S. Ringer, D. Amodei, T. Brown, J. Clark, N. Joseph, B. Mann, S. McCandlish, C. Olah, and J. Kaplan. Language models (mostly) know what they know, 2022.
11

---

[14] A. Köpf, Y. Kilcher, D. von Rütte, S. Anagnostidis, Z. R. Tam, K. Stevens, A. Barhoum, D. Nguyen, O. Stanley, R. Nagyfi, et al. Openassistant conversations demroriion- large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
[15] L. Kuhn, Y. Gal, and S. Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation, 2023.
[16] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L.. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: A benchmark for question answering research. Transactions the Association for Computational Linguistics, 7:452 466, 2019. doi: 10.1162/tacl_ a 00276. URL
[17]
R. Lebret, D. Grangier, and M. Auli. Neural text generation from structured data with application the biography domain. In J. Su, K. Duh, and X. Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203-1213, Austin, Texas, Nov. 2016. Association for Computational Linguistics. doi:
URL httpttttaccaaaattiaiie
org/D16 1128
[18] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. tau Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retreeeal-augmented generation for knowledge- intensive nlp tasks, 2021.
[19] J. Li, Y. Yuan, and Z. Zhang. Enhancing Ilm factual accuracy with rag to counter hallucina- tions: A case study on domain-specific queries in private knowledge-bases arXiv preprint arXiv:2403. 10446, 2024.
[20] K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg. Inference-tine intervention: Eliciting truthful answers from a language model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[21] A. Malinin and M. Gales. Uncertainty estimation in autoregressive structured prediction, 2021.
[22] Manakul, A. Liusie, and M. J. F. Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models, 2023.
[23] A. Mishra, A. Asai, V. Balachandran, Y. Wang, G. Neubig, Y. Tsvetkov, and H. Hajishirzi. Fine-grained hallucination detection and editing for language models, 2024.
[24] C. Niu, Y. Wu, J. Zhu, S. Xu, K. Shum, R. Zhong, Song, and T. Zhang. Ragtruth: A hallucination corpus for developing trustworthy reurieedl-augmenned language models, 2024.
[25] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, I.. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.
[26] N. Rajani, L. Tunstall, E. Beeching, N. Lambert, A. M. Rush, and T. Wolf. No robots. ooobbotttti/oooooooossooooooo/ 2023.
httgp://huggingfaac http://hwwggnnaaceooooooaaaaaa/a
[27] K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston. Retrieval augmentation reduces 2021.
hallucination in conversation. arXiv preprint
[28] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaci, N. Bashlykov, S. Batra, Bhargava, S. Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.00228, 2023.
[29] E. Tulchinskii, K. Kuznetsov. K. Laida, D. Cherniavski, S. Nikolenko, E. Burnaev. S. Baran- nikov, and I. Piontkovskaya. Intrinsic dimension estimation for robust detection of Al-generated texts. In Thirty-seventhh Conference on Neural Information Processing Systems, 2023. URL https:///peerrviev..ee www.:eeteeeoooodddau/ueeaa.
[30] N. Varshney, W. Yao, H. Zhang, J. Chen, and D. Yu. A stitch in time saves nine: Detecting and mitigating hallucinations of Ilms by validating low-confidence generation, 2023.
12

---

[31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, and A. N. Gomez. L. u. kaiser, and i. polosukhin, "attention all you need,". Advances in neural information processing systems, 30: 5998 6008, 2017.
[32] X. Wang. J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Sels-csosy improves chain of thought reasoning in language models, 2023.
[33] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jemnite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Huggingface's transformers: State-of-the-art natural language processing, 2020.
[34] Y. Wu, J. Zhu, S. Xu, K. Shum, C. Niu, R. Zhong, Song, and T. Zhang. Ragtruth: A hallucination corpus for developing trustworthy strieval-augment language models. arXiv preprint arXiv:2401.000300 2023.
[35] Yin, J. Srinivasa, and K.-W. Chang. Characterizing truthfulness large language model generations with local intrinsic dimension, 2024. URL https://arxiv.org/ ortpa///22abbb/222222 18048
[36] M. Yuksekgonul, V. Chandrasckaran, E. Jones, S. Gunasekar, R. Naik, H. Palangi, E. Kamar, and B. Nushi. Attention satisfies: A constraint-satisfachii lens factual errors of language models. In The Twelfth International Conference on Learning Representations, 2024. URL attps://openreview..rt wet.eer..eet/trmmmdddgg.aaaa... gfFVAfffP
[37] M. Zhang, O. Press, W. Merrill, Liu, and N. A. Smith. How language model hallucinations can snowball, 2023.
[38] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wa, Y. Zhuang. Lin, Z. D. D. Li, E. Xing, al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:3306.05885, 2023.
13

---

Appendix
A Limitations and Future Work
Improving Hallucination Detection Performance: In this work, we consider different detection methods using the internal representations and output probabilities of well-trained LLM. We thus assume access to at least one proxy open-source LLM such as Llama to compute our detection scores, which may not always be feasible. This is however mild assumption, covers a very large fraction of practical use cases encountered in the real-world. Moreover, the overall detection rates have significant scope for further improvement, especially towards improving the True Positive Rate at low False Positive Rates.
do believe that this
While this work focuses on detection of hallucinations, the mitigation of hallucinations in LLMs itself not addressed directly. We hope that future works can possibly build upon the scoring metrics proposed here towards reducing the frequency of hallucinations, which would be a potentially significant contribution overall. We anticipate that LLM-Check could be directly incorporated towards providing additional automated feedback in the fine-tuning stage of LLMs with Reinforcement Learning, wherein output sample generations that detected to be hallucinatory in nature can be wown-weighted appropriately. Additionally, the detection methods can assist in flagging samples in a highly efficient manner towards a customized human-feedback loop with RLHF, wherein annotators reflects the desired extent of factuality for the sample generations so considered.
2. Mitigation of Hallucinations:
introduce orthogonal ranking which
3. Principled Incorporation of Retrieved Data: Furthermore, we are also hopeful that future work towards incorporating external references in a more principled manner could help in this detection task using Retrieval Based Augmentation [18]. For instance, the proposed detection methods can perhaps serve as an initial screening mechanism: if potential hallucinations are detected, a subsequent RAG- based query can be made. This process would be triggered when the detector flags any indications of hallucination, allowing for a deeper, context- aware verification of the content produced by the LLM with minimized running costs. This two-step method would not only enhance the precision of the detection itself, but also integrates additional grounded verification steps to ensure the reliability and accuracy of the model outputs.
Combining different Detection Techniques: We attempt to combine the different scoring methods proposed for hallucination detection, namely using Eigen-analysis of internal LLM representations and Output Token Uncertainty Quantification methods, towards a unified form of detection. We observe that this problem can be reduced to that of learning classifier that takes the different scores as imput and is trained to predict the absence or presence of hallucinations. We conducted preliminary experiments towards this the FAVA Annotation dataset using Logistic Regression, but observed no appreciable gains over using the best performing individual method, the Attention Score, on this dataset. We seem observe that learning optimal combination of the different scores in manner that it generalizes well across diverse truthful and hallucinatory inputs a highly non-trivial task, especially considering the different forms of hallucination highlighted in the ane-ggained taxonomy presented in the FAVA dataset. In this paper, we focus on fairly simple combinations of these scores, given the strong motivation to keep the overall compute cost for detection very low enable its use in real-time systems. However, we do hope that future works can build upon the proposed methods fruitfully achieve significantly higher detection performance, albeit with a potentially higher computational footprint.
B Broader Impact
In this work, we propose LLM-Chedt that can be used for efficient hallucination detection in critical scenarios even when the ground-trui reference or multiple model responses might not be available. We believe our work is an important step toward reliable hallucination detection in these constrained scenarios. Our work could positively impact studying and mitigating hallucinations LLMs. Similar to other detection methods in the literature, false detection with our method could lead to some model generations being flagged as hallucinations or some hallucinated contents not being detected as hallucinated. However, in our experiments, we show that our method LLM-Check reduces these errors
14

---

when compared to other baseline hallucination detection techniques. To the best of our knowledge, direct negative societal impacts due our work are minimal.
C Visualizing the Attention Scores
To inspect the variation across hallucinatory and non-hallucinatory examples, utilize the FAVA Train split dataset, since it consists of pairs of samples with and without hallucinations for the same exact prompt. We make use of the following illustrative example, which post-tokenization contains the same total number of tokens for both the hallucinated and truthful responses. This helps simplify the comparison since the woggh-normalization We list the prompt Xp (with the actual references shown shortened here for visualization purposes, since they are fairly long), alongside the hallucinated Sample (HS) and the Truthful sample (TS):
not additional factor towards differentiating the scores.
Prompt xp: Read the following references: Reference [I]: The Ballad of Big Al... ...
Reference [2]: ...
Reference [3]:
Reference [4]: ...
Reference [5]:
Based only on the references provided, write blog article.
Hallucinated Sample (HS): "The Song of Big Al" is special episode of the nature documen- tary series "Walking with Dinosaurs" that focuses on the life story of an Tyrannosaurus specimen called "Big Al". The story is based on a well-preserved fossil of Big Al, which lived during the Early Jurassic period approximately 145 million years ago. The episode was produced by the BBC Natural History Unit and partnered with the National Geographic Channel ProSieben, and TV Asahi. Rumor has it that the episode was partially shot in Cresswell Craggs, UK. Additionally, a behind-the-scenes episode called "Big Al Uncovered" was aired alongside "The Song of Big Al"
Truthful Sample (TS):
"The Ballad of Big Al" is a special episode of the nature docu-
mentary series "Walking with Dinosaurs" that focuses the life story of an Allosaurus specimen called "Big Al". The story is based well-preserved fossil of Big Al, which lived during the Late Jurassic period approximately 145 million years ago. The episode was produced by
the BBC Natural History Unit and partnered with the Discovery Channel ProSiehen, and TV Asahi. Rumor has it that the episode was partially shot in Cresswell Craggs, UK. Additionally, aired alongside "The Ballad of Big Al"
behind-the-scene episode called "Big Al Uncovered"
We observe that the overall response structure is quite similar, but key phrases such as "The Song of Big AI" hallucinated as "The Ballad of Big Al", "the Discovery Channel" is hallucinated as "the National Geographic Channel". and "TYrrannoauruu specimen called "Big Al"" is hallucinated as "Allosaurus specimen called "Big Al""
We then analyze the eigenvalues of the attention kernel using Llama-2-7b model, which are used to compute the proposed Attention Score. We know from standard results that cigenvectors with distinct eigenvalues orthogonal, and thus focus analyzing the cigenvalues directly, rather than the high-dim high-dimensional cigenvectors to illustrate and motivate the proposed method. We highlight the key differentiating tokens along with the log-eigenvalue corresponding that specific token position in the attention mechanism at an intermediate layer which contributes to the total mean-log-determ teat-loogdettemmnnant in the proposed Attention Score in Table-6
15

---

HS Token log Keras
The -4.99
Song -4.8
of -5.56
Big -5.88
AI -5.69
p= -5.42
TS Token log Kerj)
The -4.99
Ball -5.68
ad -5.57
of -6.72
Big -6.22
Al -5.92
-5.85
HS Token log Kerdi
The -7.40
National -5.61
Geographic -4.46
Channel -5.84
μ = -5.83
TS Token log Kerds
The -7.45
Disc -6.57
overy -5.88
Channel -6.70
-6.60
HS Token
Ty
ran
05a
unus
spec
imen
called
Big
A1
The
log Ker
-5.41
-5.44
-5.85
-6.29
-5.44
μ= 5.62
TS Token
A1l
spec
imen
called
Big
A1
The
story
log Ker
-5.51
-5.35
-5.38
-6.17
-6.06
-6.45
-6.31
-6.34
-5.86
-5.92
-6.32
-5.05
-5.91
Table log Kerld computed over the diagonal of lower triangular attention kernels for substrings containing Hallucinated Sample (HS) tokens and Truthful Sample (TS) tokens, averaged across different attention heads. We observe that the overall mean j over different tokens, is consistently larger for Hallucinated Sample Tokens.
We thus observe that the cogeeigenvssues of the Hallucinated response are indeed larger in value, indicating that the rich latent space representations the L.IM are indeed indicative of the presence of hallucinated text. In the third example pair of hallucinated and truthful samples, we see that though the hallucinated details are split across more tokens than that in the truthful response, we observe that the LLM is sensitive to the error as the log-eigenvalues corresponding to the tokens that immediately follow the error are larger, again contributing to the overall detection that the overall response is indeed hallucinated.
In Figure 4 we also visualize the cumulative difference in the log Kerii values from the first token till the tth token (between and total length) between the Hallucinated Sample (HS) and Truthful sample (TS). Though we observe that the difference in log-eigenvalues between the hallucinated and truthful responses is not entirely monotonic throughout the token sequence, we observe that the log- eigeeegenvalues corresponding to the hallucinated response are consistently larger over the whole sequence when compared to the log- eigeeieesss arising from the truthful response.
35
30
Offferncce in LogaKer
25
20
15
100
MWV
Tikon Position
Figure 4: Difference between the cumulative sum of log eigenvaals from the first token till the jth token (between and total length) between the Hallucinated Sample (HS) and Truthful sample (TS).
We observe this phenomenon in greater generality, such as for hallucinated responses of different lengths compared to truthful ones. We highlight that this is key advantage of using the mean-log-
16

---

determinant which normalizes using the length of the token sequence, as compared to scoring methods such as Negative-log logative-oo-llkelhooodd which do not explicitly account for varying sequence lengths.
D Related Works (Continued)
In this section, we continue our discussion on related works due to paucity of space in the main paper. [9] study hallucinations specifically in the setup of Neural machine translation (NMT), and even propose mitigation techniques to overwrite hallucinatory sections of the output. [21 proposed to model predictive uncertainty by using a Dirichlet Prior Network to explicitly parameterize a distribution over distributions sequences are first generated, and then subsequently clustered based on bi-directional entailment again using a language model. After this equivalence relation between different responses is established using clusters, they can be treated as categorical datapoints, and thus the semantic entropy can be estimated. This technique is similar Semantic Entropy based estimation for single response which is performed in "SPlfCheckGPP with BERTScore" and "SelfCheckGPT with NLI' in their original SelfCheck-GiP "SelfCheckGPT with Prompt". We note that LLM-Check performs better than SelfCheckGPT-Prome as well (Tables 3 Furthermore, Semantic Entropy is very costly to estimate for real-time detection since the several model inference steps are required to establish the clusters using the equivalence relation proposed. Both Predictive and Semantic entropy quantify the total uncertainty over the distribution of all possible responses given a fixed prompt, in sharp contrast to the primary focus of this paper, which analyzes the hallucinatory behavior within single fixed model response output on a given prompt. Thus, Predictive Entropy and Semantic Entropy perform population-lever uncertainty estimation akin to INSIDE, and is not amenable to the single-response case.
simplex. [I5] proposed Semantic Entropy (SE), wherein multiple
paper. but these seen to perform worse than their best scoring method
Comparison with standard classifiers on hidden representations: (20) proposed Inference- Time Intervention (ITI), which improves the truthfullness of LLMs by shifting internal model activations along truth-correlated directions. To do this, the authors identify a sparse set of attention heads with high linear probing accuracy for truthfulness using base dataset such as TruthfulQA, and subsequently perturb activations during autoregressive generation in inference time. We note that methods which rely upon supervised training of classifiers on internal model representations over samples with and without hallucinations can add large computational overheads. For instance, Inference-Time Intervention (ITI) relies upon training 1024 binary classifiers on the TruthQA datasets, and thus becomes prohibitively expensive (as also noted by (5).
Nevertheless, we experiment with standard classifiers on hidden representations on the FAVA An- notation dataset. Namely, we create a train-test split of the dataset, and train classifiers on the hidden representations corresponding to layer 20 and layer 30 of Llama-2-7B model. We use this split to ensure that we have about 100 testing samples balanced between classes (with and without hallucinations) to obtain reliable evaluations of the classifiers so trained. We observe that using layer 20, this method obtains AUROC, Accuracy and TPR @ 5%FPR of 55.76, 59.82 and 0.00 respectively. while using layer 30 we observe detection rates with AUROC, Accuracy and TPR @ FPR of 56.52, 61.61 and 1.79 respectively. We note that despite requiring supervised training, this method performs significantly worse than the proposed Attention Score which achieves AUROC, Accuracy and TPR @ @5RFPR of 72.34, 67.6 67.96 and 14.97 respectively. We hypothesize that the generalization of these classifiers might not be adequate to achieve reliable detection performance across the di- verse hallucination samples and types as present in the FAVA dataset, as the corresponding hidden representations become similarly disparate.
29 utilizes the persistence homology dimension (PHD) from topological data analysis, specifically to differentiate the inherent dimensionality for real and Al-generated texts, distinct from the detection of hallucinated versus truthful, grounded responses. However, these techniques developed towards characterizing the intrinsic dimensionality of the underlying data manifold can plausibly be extended to differentiate between truthful and hallucinatory texts. (35) proposed to utilize Distance-aware maximum likelihood estimation (MLE) for the Local Intrinsic Dimension (LID), by fitting to a Poisson distribution, where the rate of the Poisson process is parameterized by the intrinsic dimension, in order to determine the truthfulness of model responses. We do note key differences, such as that LID estimation requires explicit optimization (minimization of heteroskedastic weighted polynomial regression performed), and utilizes the latent space representation corresponding to the last token alone, in sharp contrast to proposed method which considers the latent representations
17

---

over the entire token sequence. Furthermore, this technique requires the availability of a large number of neighboring samples, such as 200 samples, to estimate LID effectively. Thus, this technique more similar to INSIDE and standard classifiers on trained hidden representations, as compared to our work which focuses on detection in the single-response setting.
E Experimental Results (Continued)
Measure
AUROC
Accuracy
TPR @ 5% FPR
Entity Hallucinations (374 samples)
PPL Score
60. 14
59.89
5.35
Window Entropy
57.35
58.02
2.67
Logit Entropy
45.91
52.14
3.21
Hidden Score (Layer 20)
67.93
65.78
14.44
Attention Score (Layer 20)
64.64
61.23
21.39
Relation Hallucinations (50 samples)
PPL Score
58,08
62.00
4.00
Window Entropy
52.64
56.00
8.00
Logit Entropy
55.84
64.00
0.00
Hidden Score (Layer 21)
58.72
58.00
8.00
Attention Score (Layer 21)
64.32
66.00
4.00
Invented Hallucinations (120 samples)
PPL. Score
69.03
68.33
5.00
Window Entropy
62.25
61.67
3.33
Logit Entropy
44.08
52.50
0.00
Hidden Score (Layer 20)
80.53
75.00
16.67
Attention Score (Layer 20)
75.22
69.17
31.67
Subjective Hallucinations
(182 samples)
PPL Score
48.36
57.14
0.00
Window Entropy
54.76
56.59
1.10
Logit Entropy
61.15
62.09
4.40
Hidden Score (Layer 15)
56.32
60.99
8.79
Attention Score (Layer 21)
67.14
67.58
9.89
Unverifiable Hallucinations
(174 samples)
PPL Score
49,25
59.77
1.15
Window Entropy
54.95
57.47
3.45
Logit Entropy
60.22
63.22
2.30
Hidden Score (Layer 20)
63.32
63.22
12.64
Attention Score (Layer 5)
67.83
69.54
10.34
Table 7: Hallucination Detection without References on the FAVA Annotated Dataset: Detection metrics obtained on a balanced dataset with different forms of hallucinations as per the FAVA taxonomy.
Here, analyze the effectiveness of different detector types different forms of hallucinations as presented within the taxonomy introduced by FAVA. Thus, we consider samples that originate from one of the following types of hallucinations: "entity", "relation". "contradictory". "invented", "subjective" " and "unverifiable".
We present these consolidated results in Table 7 are quite effective detecting hallucinations across differrnnt types. However, we do observe fairly large oscillations across layers in each of these settings, and the token-based measures might be more consistent, even though their absolute numbers are slightly lower.
Here, again observe that the Attention Scores
In Figure 5] we show the ROC curves for logit- based detectors on the annotated FAVA dataset for entity and relation hallucinations. Note that the negative detector scores might help detect some kinds of hallucination instances. For example, while negative perplexity scores help in entity hallucination detection, positive perplexity scores help in relation hallucination detection. This implies that the detection scheme could be perhaps improved by using interval based detection, rather than threshold
18

---

entiy 374
relation n
10-
1.0
perp only AUROCIB. AgEg TRip SNAPFL
goa perplosity AUROCGRUUUS AcUO dego Triip 0000
"eg Ingt antrapy Ausocia Assis magg
pou Inpltentrapy AuSrocon esad Acari tedo miug seu wwwrno..oo neg. wn logtt entroy AL ALROCa.514 1144 Acc 5000 TPRSS SMMPRR3
208 win logt enrropy AuROC 5374 Acc8.5415 5455 1PRg Shirpmo..c00
0.4
0.2
022
0.5
0.6
10
36
0.2
0.4
0.6
1.0
TPR
(a) Entity Hallucinations
(b) Relation Hallucinations
Figure 5: ROC curves for logit-based detection schemes with the annotated FAVA dataset. Figures 5a and 5b]shhw the detection of entity and relation hallucinations, respectively. As observed here, taking the negative detection scores helps with detecting various kinds of hallucinations.
based detection. Thus, for the logit level detection scores, it is beneficial consider both positive and negative scores to obtain large difference in detection performance.
E.1 Experimental Details and Hyperparameter Choices
For all our evaluations, we use Pytorch 225 models, primarily open-source LLM models available on Huggingface [33]. By default, set the generation configuration of the Huggingface model to be (" ("temperature": 0.6. "top_p": 0.9, "top_k"" : 50, "do stoosssmmee True). For the setting of Logit Entropy, consider the top 50 tokens as the selected candidates to compute the score. For the computation of Hidden scores, we do not observe a noticeable difference between using the covariance matrix of hidden representations, and centered covariance matrix (as used in INSIDE). For the Hidden scores and Attention scores, we report the best of results obtained over all the layers, varying between 1 and 32 for Llama-2-76 and Vicuna-7b. For the FAVA Annotated dataset, subset consisting of 167 pairs of samples with and without hallucinations present. On the FAVA train set, we evaluate our results on the first 250 examples, which again consist of pairs of samples with and without hallucination, so to obtain 500 total samples. For the SelfCheck dataset, we utilize all 1908 annotated sentences provided. For detecting hallucinations in given sentence, an increasing passage context is utilized by incorporating prior sentences arising from the same model response. since detection hallucinations within single sentence without the appropriate context will be ill-defined, Moreover, the model is prompted using "This is Wikipedia passage about (concept):" to specify the concept within WikiBio. This is not required in the case of consistency based methods, since the other reference samples explicitly provide this concept as is.
utilize balanced
E.2 Robustness Across Domains
In this paper, we attempt to cover an adequately diverse extent of domains between the FAVA- Annotation dataset, FAVA-train split, SelfCheckGPT Dataset and RAGTruth dataset. Indeed, the FAVA Annotation dataset itself spans four different data sources: Knowledge-intensive queries sampled from the Open Assistant dataset [14], Open QA prompts from the No Robots dataset and WebNLG [8] datasets. In addition, the FAVA-train split consists of Wikipedia articles and Question Answering datasets (16), while SeIfCheckGPT Dataset consists of Annotations from the WikiBio dataset [7] Lastly, the RAGTruth dataset consists of annotations
(26). Instruction
from the CNN/Daily Mail dataset [II] To further assess robustness across sub-domains, we utilize the FAVA Annotation dataset, since the individual samples are explicitly labeled with the original dataset that they were derived from. Namely, FAVA Annotation dataset lists the following three named datasets: Open Assistant, instruction-followee scores for each of these data-subsets, and observe following values: -4.97, -4.94 and -5.1 over the instruction-follovii dataset, Open Assistant dataset and WebNLG dataset respectively. We thus
and WebNLG. We compute the mean attention
19

---

015
02-
AUROC
ACC
04
AGC TFRRSS
AUROC
083
13 20
20
30
10
20
22
20
Layer Raurceer
Layer Number
Figure 6: Layer-wise Analysis obtained with (a) Hidden Score and (b) Attention Score, on the left and right respectively for a Llama-7B model on the FAVA-Annotation dataset. Here, we can observe that the Attention based scores tend to yield higher AUROC, Accuracy and TPR @5%FPR but can oscillate more strongly across layers. Moreover, we point out that Layer 19 using Attention Score obtains the highest TPR @ (555 %PPR of 23.35%, though its AUROC is slightly lower than Layer 21.
observe that the scoring method indeed consistent across different domains, and expect such results to continue to hold over subdomains of other commonly used datasets.
E.3 Performance Differences compared to results reported by Prior Baselines
While the SelfCheckGPT paper shows that SelfCheckGPT-Pronmee is the strongest baseline in their comparisons versus other approaches, we observe extreme reduction performance when we perform our evaluations with Llama-2 for SellCheckGPT-Promppe which is also the default model used in the relcased SelfCheck Github repository in the whitebox non-APl setting. We posit that the SelfCheckGPT-Prone
results might have been potentially biased due to the fact that the multiple responses were indeed generated by GPT-3 (text-davinci-003) over the WikiBio dataset, and similar models were then used for the final evaluation. Moreover, it does appear that output uncertainty estimation based detectors, though generally weaker than the Hidden or Attention scores, can be salient detectors such as in the Furthermore, the exact entropy based scores considered in this paper are slightly more sophisticated than the considered in the SelfCheckGPT paper.
with synthetic hallucinations in the FAVA-train data split.
F Layerwise Analysis for Eigenvalue based Scores
F.1 Layerwise Analysis and Layer Selection
As noted in Section-5: 5.3, layers are used for computing Hidden Attention scores. Indeed, the runtime analysis shown in Figure represents the total runtime for computing Attention scores and Hidden scores from all 32 layers Llama-2-7b. That is, the Attention score computation for all 32 layers takes 0.22 seconds per example, while the Hidden score computation for all 32 layers requires 2.72 seconds per example when averaged over the FAVA Annotation dataset. Since the overhead to compute scores for layers small, expect that they be utilized for real-time analysis.
we observe that the method is computationally efficient even when all model
20

---

0.7
0.5-
Actuaay
0.4
pair
0.1
ACC on 50 pairs
0.0
Layer Number
Figure 7: Results across different layers of Llama-2-7B obtained using sample subsets of 5, 20 and 50 pairs, as well as the complete dataset. We observe that general layerwise trends begin to hold with fairly limited sample pairs, and help choose an optimal layer in an efficient manner.
In this section, we investigate methods towards layer selection for the eigenvalue based scoring methods with low computational overhead. From Figure-6o we first observe that the performance obtained with the Hidden score is extremely stable across layers, and thus it is relatively easy to choose, though we recommend a middle-level layer such as layer 20 for 32 layer, 7-billion parameter model such as Llama-2. On the other hand, we do indeed observe a larger degree of oscillations across layers with the Attention score. Here, we perform an experiment to potentially rapidly select layers, by plotting the results obtained using few samples, and subsequently check if the overall performance on the dataset can be estimated using this for each layer.
We present these evaluations in Figure 7 results obtained with 20 and pairs as compared the full dataset. In general, we do observe that the layers between 19 and 23 achieve close-to-optimas performance for the Attention score computed on 32 layer, 7-billion parameter model such as Llama-2. We hypothesize that for the white-box setting, while the very early layers are involved in feature extraction, and the last layers are involved more towards making an optimal next-token prediction, the layers after the midpoint of the network quite suitable for hallucination detection. In the black-box setting, this is much more difficult since is non-trivial to map representations of intermediate layers between different LLMs, especially when the original LLM has many more layers, with GPT-4 Llama-2-77 Furthermore, the optimal layer can depend on factors such as the structure of knowledge presented within an example. For instance, samples within the SelfCheck dataset were created by performing sentence-level split of paragraphs generated by GPT-3 based on WikiBio data, with corresponding sentence-lewe annotations. However, for detecting hallucinations in given sentence, increasing passage context utilized by incorporating prior sentences arising from the same model response since detection of hallucinations within single sentence without the appropriate context is not well-defined. Thus, in these cases of increasing context for sentence level detection, even early-layers are observed to be optimal, as compared to the seen with other datasets such FAVA-Train, FAVA-Annotation or RagTis Truth which are more standardized in their structure.
We do indeed observe a fair degree of agreement between
21

---

F2
Layerwise Detection Results on the RAGTruth Dataset
43
0.4
Lager
(a) Llama-2-7b
(b) Llama-2-7b
(c) Llama-2 2-13b
(d) Llama 1-1
34
0.4
22
(e) 1--ma
(l) Llama-2-70b
30
(g) GPT-4
(h) GPT-4
0.2
(i) Mistral-7b
(i) Mistral-7b
Figure 8: Performance for different layers with Hidden Score (left column) and Attention Score (right) on the RAGTruth dataset. All these scores are with the Llama-2-7b model.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the claims made in the abstract and introduction are faithfully supported by the experimental evaluations presented in Section-
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, detail the limitations in Section A of the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

Justification:
We make some theoretical arguments in Section 4 and is self- contained.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, we include all required details to reproduce the main results, and also make the codebase for LLM-Chedd available GitHub.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification: Yes, we include all required details to reproduce the main results, and also make the codebase for LLM Check available on GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
public/guides httpe///eeooeeueebessssssooooooa
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https wttpg/uddeo/oouuubbeeeee ssinoolicyy for more details.
//nips.co
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Yes, we provide these details in Section 3 and Section E.1 models in this work.
We do not train any
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We run our scoring functions deterministically with teacher-foreing, since the hallucination datasets include the sampled tokens explicitly. Thus, we get exactly the same values with reruns on the same data splits.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer:[ Yes]
Justification: We run our experiments mainly on Nvidia A5000 and A6000 GPU cards. We provide the timing experiments single Nvidia A5000 GPU in Section 5.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:: bectpa://neeripeee http:///wwrrp..ccooop/.c/eeeeeee
Answer: [Yes]
Justification: Yes, conform to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we include our broader impact statement in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: Our paper analyses hallucination detection in LLMs, and thus does not pose added safety risks, but rather helps reduce such risks. We explain more in Section well.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we properly credit authors and licenses for the different models and datasets used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new datasets new models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "JCAcgj44wH", "reviewer": "Reviewer_JTvH", "summary": "The paper presents a comprehensive study on the detection of hallucinations in outputs produced by large language models (LLMs). The authors propose a method, LLM-Check, which aims to identify hallucinations within a single response of an LLM by analyzing internal hidden states, attention maps, and output prediction probabilities. The study evaluates LLM-Check in various settings, including scenarios with and without access to ground-truth references and in both white-box and black-box settings. The results demonstrate that LLM-Check is computationally efficient and achieves significant improvements in detection performance over existing baselines across diverse datasets.", "questions": "1. Do you have any preliminary thoughts or plans on how the detection method could be extended to also mitigate hallucinations?\n2. What does population-level mean in hallucination detection?\n3. What are the differences between the black-box settings and white-box settings? What do the hidden score, attention score indicate?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel method for detecting hallucinations using internal states and attention maps, which is less computationally intensive compared to prior methods requiring multiple responses or large databases.\n2. LLM-Check is shown to be highly efficient, requiring only a fraction of the runtime of other baseline methods.", "weaknesses": "1. The paper lacks a figure illustrating the overall pipeline, which makes it hard to grasp the main idea at first glance.\n2. Some terminologies, such as \"white-box settings,\" \"black-box settings,\" and \"population-level detection,\" are not well explained, making it hard to understand the concepts without prior knowledge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We are glad that the reviewer found the proposed method to be novel, and is effective towards hallucination detection in various settings, while being extremely efficient computationally. We respond to the questions raised below:\n\n> The paper lacks a figure illustrating the overall pipeline, which makes it hard to grasp the main idea at first glance.\n- We sincerely thank the reviewer for the suggestion, and as suggested we include a schematic of the eigen-analysis detection methods as Figure-1 in the rebuttal PDF. We shall certainly incorporate the figure into the final version of the paper.\n\n> Do you have any preliminary thoughts or plans on how the detection method could be extended to also mitigate hallucinations?\n- We thank the reviewer for this question. We anticipate that LLM-Check could be directly incorporated towards providing additional automated feedback in the fine-tuning stage of LLMs with Reinforcement Learning, wherein output sample generations that are detected to be hallucinatory in nature can be down-weighted appropriately. Additionally, the detection methods can assist in flagging samples in a highly efficient manner towards a customized human-feedback loop with RLHF, wherein annotators can introduce an orthogonal ranking which reflects the desired extent of factuality for the sample generations so considered.\n\n> What does population-level mean in hallucination detection?\n- As we note in Section 3.2, the prior method INSIDE performs hallucination detection by generating multiple stochastically sampled responses $x_1 , x_2 , . . . x_K$ for a given prompt $x_p$, and then computes eigen-decomposition of the covariance matrix of hidden activations of these samples. This is used to then statistically infer the presence of hallucinations within this generated sample set $x_1 , x_2 , . . . x_K$, based on a possible lack of self-consistency at a “population level” between samples within this specific set. This is in contrast to single-response detection that infers the presence/absence of hallucinations in a given fixed (single) output response $x$ for a given prompt $x_p$, as performed in our method, LLM-Check.\n\n> What are the differences between the black-box settings and white-box settings? What do the hidden score, attention score indicate?\n- For a given prompt $x_p$, we consider hallucination detection in an output response $x$ that is generated by a given LLM $f$. If the original LLM $f$ is available and accessible to compute the scores proposed with LLM-Check, the setting is considered to be “white-box”. If the original LLM $f$ that generated the response $x$ for prompt $x_p$ is no longer available or inaccessible, we utilize an auxiliary substitute LLM $\\hat{f}$ (such as open-source LLMs like Llama-2) to compute scores with internal activations and attention kernel maps using teacher-forcing, and this setting is considered to be “black-box”. The hidden score and attention score are real-valued scalar scoring metrics that can be then thresholded to determine the presence/absence of hallucinations in a given output response $x$, using the original LLM $f$ itself in the white-box setting, and an auxiliary substitute LLM $\\hat{f}$ in the black-box setting. The hidden scores and attention scores are derived from the mean log-determinant of the covariance-matrix for hidden representations, and the kernel similarity map of self-attention of the LLM. Theoretically, this mean log determinant is given by the average logarithm of the corresponding singular values and eigenvalues which capture the interaction in latent space between the different token representations, which we know is different in hallucinated samples containing non-truths compared to non-hallucinated sample sequences.\n\n\nWe sincerely thank the reviewer for the support for acceptance and greatly appreciate the suggestions and constructive comments. We kindly ask if the reviewer would consider increasing their score if their concerns or questions have been addressed. We would be glad to engage further during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3gexRhOrSD", "reviewer": "Reviewer_gixa", "summary": "The paper explores the challenge of hallucinations in large language models (LLMs), which are outputs that appear plausible but are inaccurate or fabricated. The paper conducts a comprehensive investigation into the nature of these hallucinations and proposes novel, compute-efficient methods for detecting them. Unlike previous approaches, which relied on multiple model responses or extensive databases, this study introduces techniques to detect hallucinations within a single response without requiring external data. It leverages analyses of the model's internal mechanisms such as hidden states, attention maps, and output prediction probabilities. The proposed methods are validated across several settings and datasets, demonstrating significant improvements in detection performance while maintaining lower computational costs compared to existing methods.", "questions": "1. In line 339, the authors mention that \"different detection methods can be the most advantageous, depending on the problem setup.\" Could you please elaborate on the specific scenarios in which attention-based methods perform better compared to logit-based methods, and vice versa? Additionally, is there potential to combine these two approaches to enhance the model's uncertainty evaluation effectively?\n2. Given the sensitivity of eigenvalue-based scores to the choice of layers, as discussed in the appendices and Table 2, are there any rapid methods for selecting the optimal layer for analysis? Furthermore, does the choice of layer have generalizability across different models or tasks?", "limitations": "As mentioned in Appendix A, the methods introduced in this paper rely on accessing internal model representations or the probability distributions of tokens to assess hallucinations. This requirement can pose significant challenges when dealing with commercial models such as GPT-4 or Claude, where internal details are not readily available. In scenarios lacking sufficient internal information, the accuracy of the evaluation methods could be compromised, or it might even be impossible to perform assessments effectively.\n\nAdditionally, while the methods provide robust tools for detecting hallucinations in LLM outputs, they do not directly address strategies for mitigating these hallucinations. Designing effective strategies to eliminate or reduce hallucinations in LLMs, based on the insights gained from our evaluation methods, represents an intriguing area for future research.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Compute Efficiency:** The proposed methods are designed to be highly compute-efficient, requiring only a fraction of the run-time compared to other baseline approaches. This efficiency makes them suitable for real-time analysis.\n2. **Single Response Analysis:** Unlike many previous methods that require multiple responses or extensive external databases, the proposed techniques can detect hallucinations within a single model response. This capability is crucial for applications where generating multiple responses is impractical due to time or resource constraints.\n3. **Versatility Across Settings:** The methods are tested and proven effective in both white-box and black-box settings. This flexibility ensures that they can be applied in various scenarios, regardless of the level of access to the model's internal workings.", "weaknesses": "1. **Lack of Cohesion Between Methods:** The paper introduces two main approaches for detecting hallucinations: Eigenvalue Analysis of Internal LLM Representations and Output Token Uncertainty Quantification. One approach analyzes hallucinations from the perspective of hidden representations, while the other focuses on the probability distribution of output tokens. However, there is no clear link or cohesion between these two approaches, which might lead to a fragmented understanding of how these methods can be effectively integrated or compared.\n2. **Sensitivity to Layer Selection:** As shown in Appendix D and Table 2, the performance of eigenvalue-based scores is sensitive to the choice of layers in the model. This sensitivity means that selecting the optimal layer for analysis can be computationally expensive and model-dependent. Since the performance varies significantly across different layers and models, extensive testing and validation across multiple configurations are necessary, which increases the computational overhead and complexity of deploying these methods in practical settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We are encouraged that the reviewer appreciates the compute-efficiency of the proposed method, and its practicality and versatility towards real-time hallucination detection within a single model-response across diverse settings. We respond to the questions raised below:\n\n**Comparing Eigenvalue Analysis of Internal LLM Representations and Output Token Uncertainty Quantification:**\n\n- We propose these two distinct lines of analysis towards hallucination detection in order to attempt to adequately capture the extremely heterogeneous and diversified forms of hallucination displayed by modern Large Language Models over different domains. Towards this, the Eigen-analysis of internal LLM representations helps highlight the consistent pattern of modifications to the hidden states and the model attention across different token representations in latent space when hallucinations are present in model responses as compared to truthful, grounded responses. \n- On the other hand, the uncertainty quantification of the output tokens helps analyze hallucinations based on the likelihood assigned by the model on the actual tokens predicted at a specific point in the sequence generated auto-regressively. Indeed, especially considering that the LLM is trained using next-token prediction, we expect the probability distribution $p_f (\\cdot |x)$ for a given token to be highly salient toward the relative choices available for completion, and in identifying non-factual components if present.\n\n- Thus, we utilize these diversified scoring methods from different model components to potentially maximize the capture/detection of hallucinations amongst its various forms without incurring computational overheads at training or inference time. In practice, we generally observe that the Eigen-based analysis of Internal LLM representations achieves better detection performance over Output Token Uncertainty Quantification, as observed in Table-2 with the FAVA annotation dataset (without external references) and in Table-4 with the RAGTruth dataset (with external references). \n- However, on the FAVA Train-data-split where hallucinations are inserted synthetically using GPT-3, the output-based uncertainty scoring methods are seen to be more effective at capturing the modified changes to the joint-distribution of the sequence level token prediction probabilities, as shown in Table-5. However, we do believe that this synthetic insertion of hallucinations is perhaps less likely to be encountered in real-world settings.\n\n\n**Combining Different Detection Methods:**\nWe thank the reviewer for the suggestion to try to combine these different methods towards a unified form of hallucination detection. We observe that this can be reduced to learning a classifier that takes the different scores as input and is trained to predict the absence/presence of hallucinations. We conducted preliminary experiments towards this on the FAVA Annotation dataset using Logistic Regression, but observed no appreciable gains over using the best performing individual method, the Attention Score. We seem to observe that learning an optimal combination of the different scores in a manner that it generalizes well across diverse truthful and hallucinatory inputs is a highly non-trivial task, especially considering the different forms of hallucination highlighted in the fine-grained taxonomy presented in the FAVA dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P2j2fG9c1J", "reviewer": "Reviewer_2eXD", "summary": "This paper proposes LLM-Check, a hallucination detection method that requires a single response for both black-box and white-box settings. Specifically, LLM-Check inspects the internal hidden representation, attention map, and output token uncertainty of an auxiliary LLM and derives scores for detecting hallucination. Experiments on benchmark datasets such as FAVA and RAGTruth verified the effectiveness of LLM-Check.", "questions": "- How would the logit entropy score compare with standard uncertainty metrics such as the negative log-likelihood, and sampling-based uncertainty measures such as predictive entropy and semantic entropy?\n\n- How to interpret the results in Table 4? In particular, why do the black-box detection settings perform better than the white-box setting?", "limitations": "The authors discussed the limitation of requiring access to proxy LLM in the Appendix.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- LLM-check does not require external knowledge sources or the generation of multiple responses.\n\n- LLM-check shows good empirical performance on benchmark datasets.", "weaknesses": "- The method is grounded on heuristics without providing sufficient insight or justification. For instance, the detection method is based on the key assumption that the log determinant of the token-level hidden representations or the kernel similarity map of self-attention is distinct between hallucinated and non-hallucinated responses. However, it is unclear why this would be the case.\n\n- In the black-box setting where the internal activations of the LM are inaccessible, an additional LM is used as a proxy. From the reviewer’s understanding, this is making some implicit assumptions such as the two models are trained on some similar data distribution which may not always hold.\n\n- Multiple variants of hallucination scores are proposed and the experiments demonstrated that these scores show different performances across datasets/model types. It is unclear how they associate with each other and whether they can be combined to provide a more holistic view of hallucination.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We are encouraged that the reviewer appreciates the efficacy of the proposed method across diverse detection settings and datasets, and the significant performance improvements achieved while requiring only a fraction of the computational cost (speedups upto 45x and 450x) as the proposed method does not utilize multiple model responses or extensive external databases. We respond to the questions raised below:\n\n\n**Insight on the Eigen-Analysis based Detection:** \n- In this work, we study the highly challenging task of hallucination detection within a single example - of the form $(x_p \\oplus x)$ for a prompt $x_p$ and corresponding model response $x$ - without any training or inference time overheads. While LLMs do hallucinate, they still have a significantly appreciable degree of world-knowledge grounded on truthful facts encountered in the training stage, which is reflected in the fact that the hallucinations are absent in some of the autoregressively generated sample outputs when multiple model responses are considered for the same prompt. This also hints that one of the potential root-factors that induces hallucination could be from the nature of auto-regressive sampling for generation in LLMs, since tokens once generated and selected at a given point in the output sequence cannot be overwritten or corrected at a later stage, and the LLM simply attempts to maximize the likelihood of the overall response moving from that point on, though it potentially subtly sensitive to the reality that a non-factual error is already present in the response. \n\n- Furthermore, this lack of self-consistency across multiple generated sample outputs forms the basis for consistency-based methods such as SelfCheckGPT and INSIDE. In contrast, in this work, we propose to directly analyze the variations in model characteristics between truthful and hallucinated examples by analyzing the rich semantic representations present in the hidden activations of LLMs, since we know that the LLM does generate the truthful version, albeit with potentially lower frequency. \nThus, we posit that the model is in fact sensitive to the presence of non-factual, hallucinated information in responses which would otherwise be truthful, based on factual precepts encountered in its training and that this can then be efficiently leveraged to perform hallucination detection within a single model response itself. In particular, these differences would arise in the hidden latent-space representations and the pattern of attention maps across different token representations in hallucinated responses compared to non-hallucinated responses. To quantitatively capture these variations, we thus proposed to analyze the covariance-matrix for hidden representations, and the kernel similarity map of self-attention, since these form the foremost and paramount salient characteristics of the LLM itself. \n\n- In addition, given that we require the method to be highly compute-efficient to enable real-time detection, we distill simple yet salient scalars - the mean log-determinant - from these variations in hidden representation and attention maps using eigen-analysis. Theoretically, this is well-motivated as the eigenvalues and singular values capture the interaction in latent space between the different token representations, which we know is different in hallucinated samples which contain non-truths compared to non-hallucinated sample sequences. We also remark that the Attention score can be found without explicitly computing the SVD or eigen-decompositions, enabling speedups between 44x and 450x compared to existing baselines, while also simultaneously achieving significantly improved detection performance. \n\n\n\n\n\n**Comparing with other Uncertainty Metrics:** \n\n- We thank the reviewer for the suggestion. We do first note that the Perplexity score is closely related to the overall negative log-likelihood, namely that the Perplexity is defined as the exponentiated length-normalized negative log-likelihood of a sequence. However, we did measure the detection performance of standard negative log-likelihood as suggested by the reviewer, and we observe AUROC, Accuracy and TPR@5%FPR metrics of 52.29, 54.19 and 2.99 respectively on the FAVA Annotation Dataset, which are worse than those obtained with Perplexity (AUROC, Accuracy and TPR@5%FPR of 53.22, 58.68 and 3.59 respectively).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EWX0vqP0kk", "reviewer": "Reviewer_C5LP", "summary": "The paper addresses the issue of hallucinations in large language models (LLMs). They propose using special scores based on internal model states such as attention maps, hidden activations, and output prediction probabilities to identify hallucinations within a single response in both white-box and black-box settings. Additionally, they explore scenarios where ground-truth references are available, such as in Retrieval-Augmented Generation (RAG). The proposed methods demonstrate significant improvements in detection performance while being computationally efficient.", "questions": "1. How theoretically could be explained the use of EigenScore in this setup (not for self-consistency across generations)?\n2. Could you provide comparisons with standard classifiers on hidden representations (if possible)?\n3. Could you please explain more carefully about MLP contributions in lines 177-178? \n\nSome minor typos:\n\nLine 90 typo “an centered”\n\nLine 138 ambiguous statement “in the response x, in the context of xp”", "limitations": "1. Mitigation of hallucination itself is not addressed, which is justified \n2. The proposed methods are applicable only to response-level detection\n3. Authors admit that using their model for calculating scores could be infeasible in real-world scenarios", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The details for reproducibility are included as well as the code.\n2. The paper covers a wide range of scenarios for hallucination detection, including settings with and without external references and varying levels of access to the model (white-box vs. black-box).\n3. The paper includes extensive empirical evaluations using different datasets", "weaknesses": "1. Lack of theoretical background. The paper uses scores from other papers with different setups, while not explaining why this scores should work. For example, it is not clear why EigenScore method from INSIDE [1], specifically designed to measure self-consistency across different generations, will work on hidden activations and attention maps. Authors eigenscore on attention maps are just some modification of the trace of the matrix, but maybe the original trace is better? \n2. Some important comparisons are missing. Authors compare with other techniques, but they don't provide any comparisons with standard usage of hidden activations in previous works for hallucination detection i.e [2, 3].\n3. Domain robustness is not assessed. Score-based and classifier-based models for hallucination detection may not be robust for different text domains, but the paper misses this part. \n\n[1] C. Chen, K. Liu, Z. Chen, Y. Gu, Y. Wu, M. Tao, Z. Fu, and J. Ye. INSIDE: LLMs’ internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations, 2024\n\n[2] Amos Azaria, & Tom Mitchell (2023). The Internal State of an LLM Knows When It's Lying. In The 2023 Conference on Empirical Methods in Natural Language Processing.\n\n[3] Kenneth Li, Oam Patel, Fernanda Viegas, Hanspeter Pfister, & Martin Wattenberg (2023). Inference-Time Intervention: Eliciting Truthful Answers from a Language Model. In Thirty-seventh Conference on Neural Information Processing Systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We are encouraged that the reviewer appreciates the efficacy of the proposed method extensively demonstrated across diverse detection settings and datasets. We respond to the questions raised below:\n\n\n**Theoretical insights and comparisons with INSIDE** \n- In this work, we study the highly challenging task of hallucination detection within a single example - of the form $(x_p \\oplus x)$ for a prompt $x_p$ and corresponding model response $x$ - without any training or inference time overheads. This sharply contrasts with INSIDE which performs population-level detection by computing the  centered covariance matrix across multiple model responses to check self-consistency of the set so generated. \n- Indeed, though the mean-log-determinant formula appears similar to that of INSIDE, in LLM-Check the interaction in latent space between the different token representations **within the same sample response** is utilized, as we posit that the LLM is in fact sensitive to the presence of non-factual, hallucinated information in responses which would otherwise be truthful. This arises from the probabilistic language-modeling learnt by the LLM, based on factual precepts encountered in its training and that this can then be efficiently leveraged to perform hallucination detection within a single model response itself.\n- Theoretically, this is well-motivated as the eigenvalues and singular values capture the changes in hidden interactions and patterns to the extent of attention applied, which we know is different in hallucinated samples which contain non-truths compared to non-hallucinated sample sequences. We also remark that the Attention score can be found without explicitly computing the SVD or eigen-decompositions since it is lower-triangular, enabling speedups between 44x and 450x compared to existing baselines, while also simultaneously achieving significantly improved detection performance. \n- Furthermore, the attention values represent softmax activations of the given attention head, and thus represent probability values arising from the cross-variation of the Keys and Values in the Attention Mechanism. Thus, the log-determinant correctly reduces to the trace of the log-values and characterizes the joint distribution of the predicted probabilities in the attention head, while the trace of the values themselves represents the **sum** of distinct probability values which is not theoretically sound, as these are not upper-bounded by 1.\n\n**Single-Response Analysis:** As noted by the reviewer, in the work we do perform hallucination detection within a single prompt-output response. However, we strongly believe that this is an *inherent advantage and not a limitation*, as the single-response analysis can be extended to population-level analysis by statistically aggregating scores across multiple output responses. We crucially note that this is strictly unidirectional - in that population-level detection methods such as INSIDE cannot be easily applied for single-response analysis. Moreover, we believe that the single-response analysis is more relevant for real-time detection of hallucinations in practical real-world use-cases. \n\n\n\n**Robustness across Domains:** In the paper, we do attempt to cover some extent of diversity in domains between the FAVA-annotation dataset, FAVA-train split,  SelfCheckGPT Dataset and RAGTruth dataset. Indeed, the FAVA Annotation dataset itself spans four different data sources: Knowledge-intensive queries sampled from the Open Assistant dataset (Kopf et al. , 2023), Open QA prompts from the No Robots dataset (Rajani et al., 2023), Instruction-following and WebNLG (Gardent et al., 2017) datasets. In addition, the FAVA-train split consists of Wikipedia articles and Question Answering datasets (Kwiatkowski et al., 2019), while SelfCheckGPT Dataset consists of Annotations from the WikiBio dataset (Lebret et al., 2016). Lastly, the RAGTruth dataset consists of annotations from the CNN/Daily Mail dataset (See et al., 2017). \nWe do however agree that this could be further expanded to more domains like medical datasets, but the annotation process specifically for hallucinations can be very expensive in such cases since it often requires a great deal of expertise to peruse and annotate carefully. However, we do believe that if specific domains such as a medical-data is considered, wherein a LLM specifically fine-tuned on such data is used, hallucination detection techniques such as LLM-Check will continue to be highly effective.\n\n**Real-world Feasibility:** We wish to clarify that we do indeed believe that the proposed method is feasible in real-world scenarios, and relies upon the very mild assumption that at least one open-source LLM such as Llama is available to serve as a proxy to compute our detection scores. While this assumption may not always strictly hold in all scenarios, we do believe that this covers a very large fraction of practical use cases encountered in the real-world. Furthermore, the effective black-box performance indicates a considerable real-world advantage over methods such as INSIDE which is strictly white-box and requires multiple output responses at inference time.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**A note to all Reviewers**

We sincerely thank the reviewers for their valuable feedback and constructive comments on our paper. We are glad to note that the reviewers appreciate the comprehensive study into the nature of hallucinations in LLMs, and the practicality and effectiveness of the novel LLM-Check detection method so proposed, as validated across diverse detection settings and datasets. 
Furthermore, we are glad that the reviewers appreciate the significant improvements in detection performance achieved by LLM-Check over existing baselines, while requiring only a fraction of the computational cost (speedups upto 45x and 450x) as the proposed method does not utilize multiple model responses or extensive external databases.
We greatly appreciate the valuable comments and detailed suggestions, and we will diligently incorporate them in the final version of the paper.

Meta-review: This paper proposes LLM-Check for hallucination detection in LLMs, a set of techniques that rely only on the internal hidden representation of the LLM, attention similarity maps, and logit outputs. Experiments show that LLM-Check achieves considerable improvements over baseline methods.

LLM-Check contains two main methods for detecting hallucinations: eigenvalue analysis of the internal LLM representation and output token uncertainty quantification. The first method is interesting, while the second method seems a bit straightforward. In addition to achieving better detection performance, the proposed method is also very efficient and relies on a single LLM response.

This paper can be improved to address the following major concerns: 1) There is no clear connection or cohesion between the two different methods in LLM-Check, and it is unclear whether the two methods can be effectively integrated. 2) The performance of the eigenvalue-based score is sensitive to the choice of layers in the model and extensive testing and validation across multiple configurations are necessary. 3) Multiple variants of hallucination scores are proposed, and these scores perform differently on different datasets/model types. It is not clear how they relate to each other and how we can choose and use them in practice. The detailed response and additional experiments provided in the rebuttal stage should be included in the next version.
Decision: Accept (poster)