INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection
Authors: Chao Chen
Kai Liu
Ze Chen
Yi Gu
Yue Wu
Mingyuan Tao
Zhihang Fu
Jieping Ye
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Hallucination Detection
LogDet of Covariance Matrix
Eigenvalues
Feature Clipping
Primary area: general machine learning (i.e., none of the above)
TL;DR: Exploring LLMs' internal states for hallucination detection

Abstract: Knowledge hallucination have raised widespread concerns for the security and reliability of deployed LLMs. Previous efforts in detecting hallucinations have been employed at logit-level uncertainty estimation or language-level self-consistency evaluation, where the semantic information is inevitably lost during the token-decoding procedure. Thus, we propose to explore the dense semantic information retained within LLMs' \textbf{IN}ternal \textbf{S}tates for halluc\textbf{I}nation \textbf{DE}tection (\textbf{INSIDE}). In particular, a simple yet effective \textbf{EigenScore} metric is proposed to better evaluate responses' self-consistency, which exploits the eigenvalues of responses' covariance matrix to measure the semantic consistency/diversity in the dense embedding space. Furthermore, from the perspective of self-consistent hallucination detection, a test time feature clipping approach is explored to truncate extreme activations in the internal states, which reduces overconfident generations and potentially benefits the detection of overconfident hallucinations. Extensive experiments and ablation studies are performed on several popular LLMs and question-answering (QA) benchmarks, showing the effectiveness of our proposal.

Published as a conference paper at ICLR 2024
INSIDE: LLMs' INTERNAL STATES RETAIN THE
POWER OF HALLUCINATION DETECTION
Chao Chen', Kai Liu Ze Chen', Yi Gu', Yue Wu', Mingyuan Tao'
Zhihang Ful Jieping Ye
Allibaba Cloud "Zhejiang University
(ercong.co. zhihang. fzh, yejieping) libbaba ibwba-iaaneaeee com
ABSTRACT
Knowledge hallucination have raised widespread concerns for the security and re- liability of deployed LLMs. Previous efforts in detecting hallucinations have been employed at logit-level uncertainty estimation or language-Ievar self-coonistency evaluation, where the semantic information is inevitably lost during the token- decoding procedure. Thus, we propose to explore the dense semantic information retained within LLMs' INternal States for hallucInation DEtection (INSIDE). In particular, a simple yet effective EigenScore metric is proposed to better evaluate responses' self-consistency which exploits the cigenvalues of responses' covari- ance matrix to measure the semantic consistency/diversityy in the dense embed- ding space. Furthermore, from the perspective of self-consistent hallucination detection, test time feature clipping approach is explored to truncate extreme activations in the internal states, which reduces overconfident generations and po- tentially benefits the detection of overconfident ballucinations. Extensive experi- ments and ablation studies are performed on several popular LLMs and question- answering (QA) benchmarks, showing the effectiveness of our proposal.
INTRODUCTION
Large Language Models (LLMs) have recently achieved milestone breakthrough and demonstrated impressive abilities in various applications (Ouyang et al. 2022 OpenAl (023). However, it has been widely observed that even the state-of-the-art LLMs often make factually incorrect nonsense generations (Cohen et al. 2023 Ren et al.) 2022 Kuhn et al. 2022), which is also known as knowl- edge hallucination (Ji et al.) 2023). The potentially unreliable generations make it risky deploy LLMs in practical scenarios. Therefore, hallucination detection, that is, accurately detecting and rejecting responses when hallucinations occur L.MMs, has attracted more and more attention from the academic community (Azaria & Mitchell] 2023
Ren et al. 2022 Kuhn et al. 2022).
The token-level uncertainty estimation (e.g., predictive confidence or entropy) has shown its effi cacy in hallucination detection on conventional NLP tasks (Malinin & Gales 2020 Huang et al.) 2023). However, how to derive the sentence-level uncertainty from the token-level remains a chal- lenge, especially for modern auto-segressive LLMs whose response contents and sophisticated (Malinin & Gales, 2020 Kuhn et al. 2022 Duan et al., 2023 Thus, to avoid complicated token-to-sent -e-eeencee uncertainty derivation, researchers propose to evaluate the sentence uncertainty by the output languages directly (Kadavath et al., 2022 Yin et al. 2023 Zhoou et al.) 2023). Among the recent advancements, prompting L.MMs to generate multiple responses to the same question and evaluating the self-consistency of those responses has been proven effective in hallucination detection (Wang et al. 2022 Shi al. 2022). However, such a post-hoc semantic measurement on decoded language sentences is inferior to precisely modeling the logical consis- tency/divergence Manakul al. (2023): Zhang et al.) (202)).
generally diverse
Hence, instead of logit-level or language-level uncertainty estimation, this paper proposes to lever- age the internal states of L.LMs to conduct hallucination detection. The motivation intuitive: LLMSs preserve the highly-comceetrateed semantic information of the entire sentence within their in-
"Corresponding Author

---

Published as a conference paper at ICLR 2024
ternal states (Azaria & Mitchell 2023). allowing for the direct detection of hallucinated responses in the sentence embedding space.
In particular, with the generalized framework of INternal States for IallucInation DEtection (INSIDE), this paper performs hallucination detection from two perspectives. First, skipping sec- ondary semantic extraction via extra models, directly measure the seff-consiseencyliivreenn of the output sentences using internal states of LLMs. order to explore semantic consistency in the embedding space, Section 3.1] introduces an EigenScore metric regarding the eigenvalues of sentence embeddings' covariance matrix. Second, to handle the self-consistent (overconfident) hal- lucinations, we propose rectify abnormal activations of the internal states. Specifically, Section 3.2 develops feature clipping approach to truncate extreme features, which tends to prevent over- confident generations during the auto-regressive procedure. In Section 4 the effectiveness of our method is validated through extensive experiments on several well-establisher QA benchmarks.
The main contributions of our work are as follows:
We propose generalized INSIDE framework that leverages the internal states of LLMs to perform hallucination detection.
We develop an EigenScore metric to measure the semantic consistency in the embedding space, and demonstrate that the proposed EigenScore represents the differential entropy in the sentence embedding space.
A test time feature clipping approach is introduced to truncate extreme activations in the feature space, which implicitly reduces overconfident generations and helps identify the overconfident hallucinations.
We achieve state-of-the-art hallucination detection performance on several QA bench- marks, and conduct extensive ablation studies to verify the efficacy of our method.
BACKGROUND ON HALLUCINATION DETECTION
In this work, we mainly focus on the knowledge hallucination detection of natural language genera- tion based on LLMs, especially for Q&A task (Reddy et al. 2019 Kwiatkowski et al. 2019). Given an input context æ, typical LLM (Zhang et al. 2022 Touvron et al. 2023a) parameterized with is able to generate output sequences in autoregressive manner y/ = [(,, 2/. 1/2
10). where
= 3/1 3/2- ... yrl denotes the output sequence and y denotes the t-th output token. We denote P(y y-ce, Plyllyee, x) the Maximum Softmax Probability (MSP) of t-th token. For a traditional classification model, the MSP measures the confidence level of the classification result and has been widely used uncertainty measure of predictions (Hendrycks & Gimpel (2166). Therefore, for sequence generation task, a straightforward sequence uncertainty can be defined as the joint probability of different tokens, which is known Perplexity (Ren et al. 2022).
log II ((velve) = + D logl(ue)e..)))
P(ylæ, 0)
(1)
T
As shorter sequences generally have lower perplexity, the length of the output sequence i' utilized to normalize the joint probability. Since different tokens contribute differently to the semantics of the sentence (Raj et al. 2023 Duan al. 2023), the perplexity defined by averaging leven-level uncer- tainty cannot effectively capture the uncertainty of the entire sequence. It has been demonstrated that utilizing multiple generations for one imput beneficial to estimate the sequence-leve uncertainty (Malinin & Gales 2020 Kuhn al. 2022 Manakul et al. 2023). We denote y [y' y y as generated responses for input context æ. For given LLM, multiple responses could easily obtained by the top-p/top-k sampling strategy during inference time (Touvron et al.) 2023a Kada- vath et al. (022). In Malinin & Gales (2020), the Length Normalized Entropy is proposed to measure the sequence-level uncertainty by making of multiple generations, which is defined as
H ()),,, 0) = -E Ty
Σ log (og((uneee,.))
(2)
2

---

Published as a conference paper at ICLR 2024
Output
LLM
Input
The answer is 20th July.
Q:
FC Layer
date 1969 cid Neil
Feature Clip
Embediing
No
GERIED
High EigenSsore?
Armstrong first on
imbedding af arswer
Moon?
Decoder
Eigervedior
Output
Ves
Embeddding of ower
Sorry supsort answer for this question.
Tokan Emberding
Ourrent Token Embedding
Output Logit
CRDDRD Sentemce Embedding
Figure 1:
Illustration of our proposed hallucination detection pipeline. During inference time, for given question, the extreme features in the penultimate layer are truncated and the EigenScore is computed based on the sentence embeddings across multiple responses.
When a model is uncertain about its response, it generates hallucination context, resulting in an answer distribution with high entropy (Kadavath et al. 2022). has been shown that the length- normalized entropy performs better than the non non-normalized one (Lin et al. (023).
In addition to the predictive uncertainty or entropy, the semantic consistency (Lin al. 2023 Raj
et al. 2023) among multiple responses has also been widely explored to measure the hallucination degree of LLMs, which hypothesis that the L.LMs are expected to generate similar outputs if they know the input context and they are sure about the answers (Wang et al. [2022] Manakul et al.
2023). An intuitive semantic consistency metric Lexical Similarity (Lin et al. 2022 2023)
which explores the average similarity across multiple answers consistency measure
K
K
S(Y|2, 0)
Σ
sim ,3))
(3)
C
i=1
where C - C .KK- 1)/2 and sim(.,) is the similarity defined by Rouge-L Lin (004)).
3 METHOD
In this section, we introduce the details of our proposed INSIDE framework for hallucination de- tection. The whole pipeline illustrated as Fig. [ In section 3.1 demonstrate a simple but effective EigenScore metric by exploring sentence-level semantics in the internal states of LLMs. In section 3.2 a test-time feature clipping approach is introduced to effectively alleviate the issue of overconfident generation, thereby aiding in the identification of self-consistent hallucinations
3.1 HALLLCINATION DETECTION BY EIGENSCORE
The existing uncertainty or consistency based hallucination detection metrics are exploited in the logit language space, which neglect the dense semantic information that is retained within the internal states of L.MMs. To better exploit the dense semantic information, we propose to measure the semantie divergence in the sentence embedding space. For the i-th output token Y/, denote the hidden embedding in the /-th layer as hi E _d where d is the dimension of the hidden embedding (d 4096 for LLaMA-7B and d 5120 for LLaMA-13B). According Ren al. (2022); Azaria & Mitchell (2023), the sentence embedding can be obtained by averaging the token embedding == Σi -= h,, taking the last token embedding sentence embedding experiments, we use the embedding the last token the middle layer as the sentence embedding, it effectively captures the sentence semantic (Azaria & Mitchell (023). The comparison results of using different sentence embeddings are demonstrated in the ablation studies 4.3 For generated sequences, the covariance matrix of K sentence embeddings be computed as
hr. In our main
Σ = Z .a
3

---

Published as a conference paper at ICLR 2024
where Σ RK** represents the covariance matrix that captures the relationship between different sentences in the embedding space. = [1 22. .... = | R represents the embedding matrix of different sentences, J La g1ald is the centering matrix and 1 E R is the all-one column vector. Then, the proposed EigenScore can be defined as the logarithm determinant (LggDet) of the covariance matrix,
1 K
B(12 0)
log det(2) + a IIk
(5)
Here, det(X) represents the determinant of matrix X, and small regularization term a a-Ix is added to the covariance matrix to explicitly make full rank. Since the matrix determinant can be obtained by solving the eigenvalues, the EigenScore can be computed as
K
E()12, 0) = K
log II A1) = K Σ lo,(A,)
(6)
where x = =>>1 A2, AR} denotes the eigenvalues of the regularized covariance matrix Σ a- which can be solved by Singular Value Decomposition (SVD). Eq. [6] shows that the hallucination be measured by the average logarithm of the eigenvalues. The con-
degree of LLM's generation clusion is intuitive, as the cigenvalues of covariance matrix capture the divergence and correlation relationship between embeddings of different sentences. When the LLM confident the answers and K generations have similar semantic, the sentence embeddings will be highly correlated and most eigenvalues will be close to O. On the contrary, when the LLM indecisive and hallucinating contents, the model will generate multiple sentences with diverse semantics leading to more signifi- cant eigenvalues. The following remark also provided to explain why the proposed EigenScore is good measure of knowledge hallucination.
Remark 1. LogDet of covariance matrix represents the differential entropy in the sentence embedding space. Differential Entropy is the natural extension of discrete Shannon Entropy (X) = - -xx -p(x) log p(z). The differential entropy (ae(X) continuous space can be defined by replacing the probability function with its density function f(x) and integrating
I, i.e., (X) = J (xx log f(x)dx. In principle (Zhouyin & Liu 2021). for a multivariate
Gaussian distribution X N(,, E). the differential entropy can be represented as
d Hae (X) = log det Ee 5 (log 2T +1) 2
1
1 Σ log A +C
where d is the dimension variables and C' is a constant. Therefore, the differential entropy is determined by the eigenvalues (LogDet) of the covariance matrix.
According to Remark 1, the proposed EigenScore defined by Eq. represents the differential entropy in the sentence embedding space, which offers valuable insight into using EigenScore semantic divergence measure. Compared to existing uncertainty or consistency metrics that obtained
in logit or language space (Malinin & Gales 2020 Huang al. 2023 Lin et al. 2022), the advan- tages of EigenScore are: (I) It captures the semantic divergence (entropy) in the dense embedding space, which is expected to retain highly-coneentrated semantic information compared logits or languages (Reimers & Gurevych] 2019). (2) Representing semantic divergence in embedding space can effectively solve the semantic equivalence (linguistic invariances) problem (Kuhn et al.) (2022) in natural language space. (3) Fine-grained semantic relationship among different responses can be exploited by using eigenvalues of covariance matrix. Therefore, through the exploration of dense semantic information in the internal states, the EigenScore is expected to outperform existing uncer- tainty and consistency metrics, resulting in improved hallucination detection performance.
3.2 TEST TIME FEATURE CLIPPING
Recent works have shown that the LLMs are subject to the risks of self-consistent (overconfident) hallucinations (Ren et al. 2022 Di et al.) 2023), which has not been considered by existing consis- tency based methods. Therefore, to address those failure cases caused by overconfident generation,
4

---

Published as a conference paper at ICLR 2024
Neuron Activation Distribution
Neuron Activation Distribution
3.0
Neuron Activations
2.5-
Density
2.0
1.5
-10
1.0
-20
-30
1000
2000
3000
4000
0.0
8.20-..55
1.00
Neuron Indexes
Normalized Features
(a) Neuron Activation
(b) Feature Distribution
Figure 2: Illustration of activation distributions the penultimate layer of LLaMA-7B. (a) Activa- tion distribution in the penultimate layer for a randomly sampled token. (b) Activation distribution for a randomly sampled neuron activation of numerous tokens.
a test time feature clipping approach is introduced during the computation of EigenScore. As shown in Figure. 2 we illustrate the activation distribution in the penultimate layer of LLaMA-7B. An intu- itive observation is that the penultimate layer of LLMs tends to exhibit numerous extreme features, consequently increasing the likelihood of generating overconfident and self-consistent generations. Inspired by prior works that rectify internal activations to reduce overconfident prediction for Out- of- D--Dirrributinn (OOD) detection (Sun et al., 2021: Djurisic et al. 2022). introduce test time feature clipping (FC) method to prevent LLMs generate overconfident hallucinations. To rectify those extreme features, the FC operation defined as the following piecewise function
Ayiin:
h < Mymin
FC(h)=
h Nynax
Amin S h S hmaz hmaa
where represents the feature the hidden embeddings in the penultimate layer of the LLMs, Iiii and hmazz are two thresholds for determining the minimum and maximum truncation activa- tions. When Amin = -o and fmaz = +00, the output feature embedding is equivalent to the original output. For the determination of the optimal truncation thresholds. memory bank which dynamically pushes and pops element it, is utilized to conserve N token embeddings during test time. Then, for each hidden neuron, the thresholds hmin and hmaz set the top and bottom p-th percentiles of the features the memory bank. Refer to the three-sigma-rule |Pukelsheim (1994), set 0.2 in all cases. This implies that the activations falling within the largest and smallest top 0.2% in the memory bank are identified as abnormal features and subsequently truncated for reducing overconfident generation.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. We utilize four widely used question answering (QA) datasets for evaluation, including
two open-book conversational QA datasets CoQA (Reddy et al. 2019) and SQuAD (Rajpurkar et al.) 2016 as well as two closed-book QA datasets TriviaQA (Joshi al. 2017) and Natural Questions (NQ) (Kwiatkowski et al., 2019). We follow Lin et al. (2023) to utilize the development split of CoQA with 7983 QA pairs, the validation split of NQ with 3610 QA pairs and the validation split of the TriviaQA (rc. toc.nocnttttt subset) with 9,960 deduplicated QA pairs. For the SQuAD dataset, filter out the QA pairs with their flag is issmpossiblee = True, and utilize the subset of the development- v2.0 split with 5928 QA pairs. The lengths of the sequences vary in the four datasets. Specifically, the ground truth answers CoQA and SQuAD are relatively longer, while and TriviaQA typically consists of answers that only with one two words.
5

---

Published as a conference paper at ICLR 2024
Models. We use two representative open source LLMs, including LLaMA Touvron al. 2023a)
and OPT Zhang et al. 2022) in our experiments. Specifically, we consider off-the-shelf LLaMA-7B
LLaMA- 13B OPT-6. 6. 7B and their corresponding tokenizer provided by Hugging Face. We use the pre-trained wights and do not finetune these models in all cases.
Evaluation Metrics. Following prior work Kuhn al. 2022 Ren al. (2022), we evaluate the hallucination detection ability of different methods by employing them to determine whether the generation is correct or not. Therefore, the area under the receiver operator characteristic curve (AUROC) and Pearson Correlation Coefficient (PCC) are utilized as the performance measure. AU- ROC is popular metric to evaluate the quality of binary classifier and uncertainty measure (Ren) et al. 2022; Lin et al.] 2023). Higher AUROC scores are better. PCC is utilized to measure the correlation between the hallucination detection metric and the correctness measure, which is usu- ally defined as the ROUGE score (Lin 2004) or semantic similarity (Reimers & Gurevych, 2019) between the generated answers and ground truth answers. A higher PCC score better.
Baselines. We compare proposal with the most popular uncertainty-basee methods Perplex- ity Ren et al. (2022) and Length-normalized Entropy (LN-Entropy) MMalnni & Gales (2020), and the consistency- based metric Lexical Similarity (Lin et al., 2022). Besides, in order to investigate whether traditional OOD detection methods can be used for hallucination detection, we also intro- duce a popular OOD detection method Energy score (Liu et al.
2020)
comparison method.
Correctness Measure. We follow Kuhn et al. (2022) :: Lin et al.
(2023) to utilize both the ROUGE-
L (Lin 2004 and the semantic similarity (Reimers & Gurevych 2019) the correctness measure.
ROUGE is an n-gram based metric that computes the longest common subsequence between two pieces of text. The generation is regarded as correct when the ROUGE-L (f-measure) is large than given threshold, which we set to 0.5 in our main experiments. Besides, we also use the embedding similarity as the correctness measure. The sentence embeddings of model generation and the ground truth answer are extracted by the nlliroberta- large model | and the generation is regarded as true when the cosine similarity between two embeddings larger than 0.9.
Implementation Details. Implementation of this work is based on pytorch and transformers li- braries. For the hyperparameters that are used for sampling strategies of LLMs' decoder, we set temperature to 0.5, top-p to 0.99 and top-k to 5 through the experiments. The number of generations set to K = 10. For the sentence embedding used our proposal, we use the last token embedding of the sentence in the middle layer, i.e., the layer index is set to int(L/2). For the regularization term of the covariance matrix, we set a = 0.001. For the memory bank used to conserve token embed- dings, we set .N- 3000, When implement the Energy Score, average the token-level energy score as the sentence-level energy score.
4.2 MAIN RESULTS
we compare our proposed EigenScore with several representative reliability evaluation methods on three L.MSs and four QA datasets. The results show that: (1) In both LLaMA and OPT models, our proposed EigenScore consistently outperforms other comparison methods by a large margin in CoQA, SQuAD and NQ datasets under different evaluation metries. In particular, the EigenScore outperforms Lexical Similarity by 5.6% in CoQA and 8.9% in SQuAD with AUROC metric at most. (2) It's interesting to see that the Perplexity performs best in TriviaQA dataset but performs poorly on other datasets, especially for CoQA and SQuAD. This is because the generations and ground truth answers on TrivaiQA dataset is very simple, with only one or two words in the most cases. Therefore, the performance of different methods in TriviaQA is close and by simply averaging the token-level confidence as uncertainty measure performs well. (3) On average, the performance in LLaMA-13B is better than that in LLaMA-7B and OFT-6.7B, while the performances in LLaMA-7B is slightly better than that in OPT-6.7B. It demonstrates that better hallucination detection performance can be achieved with a more powerful pre-trained LLM.
Effectiveness of EigenScore. In Table. 1
bbtps://uuggingface.oooo btc.ae/ggg..acccccccccccooooooop

---

Published as a conference paper at ICLR 2024
Table 1: Hallucination detection performance evaluation of different methods on four QA tasks. AUROC (AUC) and Pearson Correlation Coefficient (PCC) are utilized to measure the performance. AUC, represents AUROC score with sentence similarity as correctness measure, and AUC, repre- sents AUROC score with ROUGE-L score as correctness measure. All numbers are percentages.
Datascts
CoQA
SQuAD
NQ
TriviaQA
Models
Methods
AUC, AUC, PCC
AUC, AUC, PCC
AUC,
AUC, AUC, AUC, PCC
Perplexity
64.1
68.3
20.4
57.5
60.0
10.2
74.0
74.7
30.1
83.6
83.6
54.4
Energy
51.7
54.7
1.0
45.1
47.6
-10.7
64.3
64.8
18.2
66.8
67.1
29.1
LLaMA-7B
LN-Entropy
68.7
73.6
30.6
70.1
70.9
30.0
72.8
73.7
29.8
83.4
83.2
54.0
Lexical Similarity
74.8
77.8
43.5
74.9
76.4
44.0
73.8
75.9
30.6
82.6
84.0
55.6
EigenScore
80.4
80.8
50.8
81.5
81.2
53.5
76.5
77.1
38.3
82.7
82.9
57.4
Perplexity
63.2
66.2
20.1
61.7
14.2
73.5
73.4
36.3
84.5
56.5
Energy
47.5
49.2
-5.9
36.0
39.2
-20.2
59.1
59.8
14.7
71.3
71.5
36.7
LLaMA-13B
LN-Entropy
68.8
72.9
31.2
72.4
74.0
36.6
74.9
75.2
39.4
83.4
83.1
54.2
Lexical Similarity
74.8
77.6
44.1
77.4
79.1
48.6
74.9
76.8
40.3
82.9
84.3
57.5
EigenScore
79.5
80.4
50.2
83.8
83.9
57.7
78.2
78.1
49.0
83.0
83.0
58.4
Perplexity
60.9
63.5
11.5
58.4
69.3
8.6
76.4
77.0
32.9
82.6
82.0
50.0
Energy
45.6
45.9
-14.5
41.6
43.3
-16.4
60.3
58.6
25.6
70.6
68.8
37.3
OPT-6.7B
LN-Entropy
61.4
65.4
18.0
65.5
66.3
22.0
74.0
76.1
28.4
79.8
80.0
43.0
Lexical Similarity
71.2
74.0
38.4
72.8
74.0
39.3
71.5
23.1
78.2
79.7
42.5
FigenScore
76.5
77.5
45.6
81.7
80.8
49.9
77.9
77.2
33.5
80.3
80.4
0.485
Hallucination detection performance evaluation of different methods with and without (w/o) applying feature clipping (FC). "+FC" denotes applying feature clipping and EigenScore (w/o) denotes EigenScore without applying feature clipping. All numbers are percentages.
Table 2:
Model
LLaMA-7B
OPT-6.7B
Datasets
CoQA
NQ
CoQA
NQ
Methods
AUC,
PCC
AUC,
PCC
AUC,
PCC
AUC,
PCC
LN L.N-Entropy
68.7
30.6
72.8
29.8
61.4
18.0
74.0
28.4
LN-Entrppy FC
70.0
33.4
73.4
31.1
62.6
21.4
74.8
30.3
Lexical Similarity
74.8
43.5
73.8
30.6
71.2
38.4
71.5
23.1
Lexical Similarity + FC
76.6
46.3
74.8
32.1
72.6
40.2
72.4
24.2
EigenScore (w/o)
79.3
48.9
75.9
38.3
75.3
43.1
77.1
32.2
EigenScore
80.4
50.8
76.5
38.3
76.5
45.6
77.9
33.5
Effectiveness of Feature Clipping. To demonstrate the effectiveness of the introduced test-time feature clipping, we compare the hallucination detection performance of different methods with and without applying the feature clipping technique. The results are shown in Table 2 As can be seen, the introduced feature clipping consistently improves the performance of different methods, with the largest improvement being 1.8% in AUROC.
4.3 ABLATION STUDIES
Number of Generations. For the methods that explore semantic consistency for hallucination de- tection, the number of generations K is key factor to the performance. Therefore, evaluate the impact of the number of generations, we select K from (5, 10, 15, 20, 30, 40} and perform ex- periments with L.aMA-7BB and the NQ dataset. The performance in Figure 33) shows that: (1) Our proposed EigenScore consistently outperforms LN-Entropy and Lexical Similarity by large margin for different K. (2) When < 15, the performance of different methods increases as K increases and when K 15, the performance tends to remain stable. The results suggeste that set- ting to 20 provides the optimal trade-off between performance and inference cost. (3) Compared to EigenScore and Lexical Similarity, LN-Entropy is less sensitive the number of generations, which demonstrates that Lexical Similarity and our EigenScore are more effective at utilizing the information in different generations.

---

Published as a conference paper at ICLR 2024
82
LN-Entmpy
Simiarity
FigenScore
77
76-
75
5
10
33
Number of Generations
Layer Indexes
(a)
(b)
Figure 3: (a) Performance in LLaMA-7B and NQ dataset with different number of generations. (b) Performance in LLaMA-7B and CoQA dataset with sentence embedding in different layers. Orange line indicates using the last token's embedding in the middle layer (layer 17) sentence embedding. Gray line indicates using the averaged token embedding in the last layer sentence embedding. The performance measured by AUROC,.
Table 3: Performance evaluation with different correctness measure thresholds in LLaMA-7B and CoQA dataset. The ROUGE-L (f-measure) score and Sentence Similarity with different thresholds employed to measure the correctness of the generated answers.
Correctness Measures
ROUGE-L
Sentence Similarity
Threshold
0.3
0.5
0.7
0.7
0.8
0.9
Perplecity
65.2
68.3
68.1
63.7
63.5
64.1
LN-Entrov LN-Entropy
67.4
73.6
74.1
65.2
65.6
68.7
Lexical Similarity
75.8
77.8
79.3
72.8
73.9
74.8
EigenScore
76.4
80.8
83.5
75.9
77.2
80.4
How EigenScore Performs with Different Sentence Embeddings. the main experiments, we employ the embedding of the last token in the middle layer as sentence embedding. Here, we also in- vestigate how the model performs with different sentence embeddings. In Figure. 3(b] we show the hallucination detection performance by using sentence embedding from different layers. The results show that using the sentence embedding in the shallow and final layers yields significantly inferior performance compared to using sentence embedding in the layers close to the middle. Besides, another interesting observation is that utilizing the embedding of the last token as the sentence em- bedding achieves superior performance compared to simply averaging the token embeddings, which suggests that the last token of the middle layers retain more information about the truthfulness.
Sensitivity to Correctness Measures. It's difficult develop automatic metrics for QA task that correlate well with human evaluations. Therefore, the choice of correctness measures is a crucial component of hallucination detection evaluation. In this section, evaluate the performance with different correctness measure thresholds in LLaMA-7B and CoQA dataset. The experimental results are presented Table. 3 It shows that the threshold has great influence on the final hallucination detection performance. Significantly, our proposed EigenScore consistently outperforms compari- son methods different thresholds. Besides, the results also indicate that the hallucination detection performance of different methods will be better under rigorous correctness measure.
Sensitivity to Hyperparameters. The hyperparameters, including temperature, top-k and top-p, of the LLMs' decoder determine the diversity of the generations. To evaluate the impact of those hyper- parameters. We provide a sensitivity analysis in Figure As observed, the performance greatly influenced by temperature but shows little sensitivity to top-k. The performance of the consistency based methods (EigenScore and Lexical Similarity) drops significantly when the temperature is greater than 1. The optimal temperature can be selected from (0.1, 1.0).
8

---

Published as a conference paper at ICLR 2024
100
Sensitivity Temperature
90
Sensitivity to Top-K
Perplisity
90
LN Entropy
80
Lexical Smilantty
AUROC
80
ElgenScore
AUROC
70-
70
60-
60
Perplixity
LN: Entrepy
50
Similarity
40
ElgenScore
0.1
0.3
0.5
1
3
5
3
5
10
20
30
50
Temperature
Top-K
(a)
(b)
Figure 4: (a) Performance sensitivity to temperature. (b) Performance sensitivity to top-k. The performance measured by AUROC,.
5 RELATED WORK
Reliability Evaluation of LLMs During real-world deployments, the reliability of LLMs poses substantial challenge, as LLMs reveal their propensity exhibit unreliable generations (Ji et al.
2023 Zhang et al. 2023). Therefore, considerable efforts has been made to address the security and
reliability evaluation of LLMs (Huang et al. 2023 Malinin & Gales 2020: Kuhn et al. 2022 Kada-
vath et al. 2022 Cohen et al. 2023 Azaria & Mitchell, 2023). Among those methods, uncertainty
based metric has been widely explored. which typically involves predictive confidence or entropy of the output token (Malinin & Gales 2020 Kuhn et al. [022 Duan et al. (023). Besides, consistency
based methods also play an important role in reliability evaluation, which hypothesizes that LLMS tend to generate logically inconsistent responses to the same question when they are indecisive and hallucinating contents (Kuhn et al.) 2022 Raj et al.) [003 Manakul et al. 2023). Based on the
consistency hypothesis, researchers also found is feasible prompt the LLMs evaluate their
responses themselves (Kadavath et al. 2022 Cohen el al. 2023 Manakul et al. 2023
Eigenvalue as Divergence Measure The eigenvalue determinant of covariance matrix captures the variability of the data and has been widely explored as divergence measure in wide range of machine learning tasks (Wold al. Kulesza Taskar 2011 Xu al. 2021 Zhouyin & Liu
2021 Cal et al. 2015). For instance, in Wold et al. ( (1987). the authors proposed the well-known Principal Components Analysis (PCA) and demonstrates that the most largest eigenvalues of sample covariance matrix corresponds to the principle semantic of sample set. Besides, the determinant of covariance matrix, determined by the eigenvalues, has been utilized to sample diversity subset in determinantal point processes (DDP) (Kulesza & Taskar, (01T)) and activation learning (Xu et al.) 2021) tasks, which demonstrates the determinant of covariance matrix is a good diversity measure. Besides, several studies also proposed to approximate the differential entropy with the logarithm
determinant of covariance matrix (Zhouyin & Liu 2021 Klir Wierman 1999..
6 CONCLUSION
Measuring the hallucination degree of LLM's generation is of critical importance in enhancing the security and reliability of LLM-based Al systems. This work presents an INSIDE framework to exploit the semantic information that are retained within the internal states of LLMs for halluci- nation detection. Specifically, simple yet effective EigenScore is proposed to measure the se- mantic consistency across different generations in the embedding space. Besides, identify those self-consistent (overconfident) hallucinations which have been overlooked by previous methods, feature clipping technique is introduced to reduce overconfident generations by truncating extreme features. Significant performance improvement has been achieved in several popular LLMs and QA benchmarks. Although our experiments focus on QA task, our method does not make any assump- tions about the task modality, and we believe our method is widely applicable to other tasks, such as summarization and translation. We hope that our insights inspire future research to further explore the internal semantics of LLMSs for hallucination detection.

---

Published as a conference paper at ICLR 2024
REFERENCES
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Maitha Alhammadi, Mazzotta Daniele, Daniel Heslow, Julien Launay, Quentin Malartic, et al. The falcon series of language models: Towards open frontier models. Hugging Face repos- itory, 2023.
Amos Azaria and Tom Mitchell. The internal state of an Ilm knows when its lying. arXiv preprint arXiv:2304. 13734, 2023.
Yuntao Bai. Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0586,, 2022.
Tony Cai, Tengyuan Liang, and Harrison Zhou. Law of log determinant of sample covariance matrix and optimal estimation of differential entropy for high-dimennional gaussian distributions. Journal of Multivariate Analysis, 137:161-172, 2015.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03 109, 2023.
Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. Lm Im: Detecting factual errors via cross examination. arXiv e-prints, pp. arXiv-2305, 2023.
Andrija Djurisic, Nehojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In The Eleventh International Conference on Learning Representations, 2022.
Jinhao Duan, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the uncertainty estimation of large lan- guage models. arXiv preprint arXiv:2337.011379 2023.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and outtif-disttibuution examples in neural networks. In International Conference on Learning Representations, 2016.
Yuheng Huang, Jiayang Song. Zhijie Wang, Huaming Chen, and Lei Ma. Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv e-prints, pp- arXiv-2307, 2023.
Ziwei Ji, Naycon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii. Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, 2017.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language mod- els (mostly) know what they know. arXiv e-prints, pp. arXiv-2207, 2022.
George Klir and Mark Wierman. Uncertainty-basel information: elements of generalized informa- tion theory, volume 15. Springer Science & Business Media, 1999.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2022.
Alex Kulesza and Ben Taskar, k-dpps: Fixed-size determinantal point processes. In Proceedings of the 28th International Conference Machine Learning (ICML-II), pp. 1193-1200, 2011.
10

---

Published as a conference paper at ICLR 2024
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. arXiv preprini urXiv:2300003344 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu. Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.00 10, 2022.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches oul, pp. 74-81, 2004.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantifi- cation for black-bor box large language models. arXiv e-prints, pp. arXiv-2309 2023.
Zi Lin, Jeremiah Zhe Liu, and Jingbo Shang. Towards collaborative neural-symbolie graph semantic parsing via uncertainty. Findings of the Association for Computational Linguistics: ACL 2022, 2022.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detec- tion. Advances in neural information processing systems, 4144521555,,, 2020.
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In International Conference Learning Representations, 2020.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallu- cination detection for generative large language models. arXiv preprint arXiv:2303.08896 2023.
OpenAI. Gpt-4 technical report, 2023.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744 2022.
Friedrich Pukelsheim. The three sigma rule. The American Statistician, 48(2):88 1994.
Harsh Raj. Vipul Gupta, Domenic Rosati, and Subhabrata Majumdar. Semantic consistency for assuring reliability of large language models. arXiv preprint arXiv:2308.0913 09138, 2023.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016.
Siva Reddy, Danqi Chen, and Christopher D Manning- Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP. IJCNLP). Association for Computational Linguistics, 2019.
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter Liu. Outtof-dist detection and selective generation for conditional language models. In The Eleventh International Conference on Learning Representations, 2022.
Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida Wang. Natural lan- guage to code translation with execution. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3533-3546, 2022.

---

Published as a conference paper at ICLR 2024
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activa- tions. Advances in Neural Information Processing Systems, 34:144-157, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2308.09288 2023b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, Self-conssstency improves chain thought reasoning in language models. arXiv preprint ar 2022.
Svante Wold. Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52, 1987.
Xinyi Xu, Zhaoxuan Wu, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Validation free and repli- cation robust volume-based data valuation. Advances Neural Information Processing Systems, 34:10837-10848. 2021.
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don't know? arXiv preprint arXiv:2305. /8153, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, language models. arXiv preprint arXiv:2205.0100
Opt: Open pre-trained transformer
2022.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Zhang, Yulong Chen, et al. Siren's 'ss song the ai ocean: A survey on hallucination large language models. arXiv preprint arXiv:2309.01219, 2023.
Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. Navigating the grey Expressions of overconfidence and uncertainty in language models. arXiv preprint arXiv:2302. 13439, 2023.
Zhanghao Zhouyin and Ding Liu. Understanding neural networks with logarithm determinant en- tropy estimator. arXiv preprint a0Siv:2005.00. 03705, 2021.
12

---

Published as a conference paper at ICLR 2024
Table 4: Performance comparison of different methods on TruthfulQA dataset. LexialSim denotes Lexical Similarity and SeICCKGPT denotes SelfCheckGPT. Hallucination detection accuracy is reported. # Prompt denotes the number of prompt templates. For ITI et al. (2023). we report the best number in their paper directly. All numbers are percentages.
Prompt Perplexity LN-Entropy LexialSim SelfCKGPT ITI EigenScore
5
70.0
71.2
73.6
74.2
83.3
76.7
20
76.4
77.7
77.9
76.8
83.3
79.5
50
73.1
77.9
73.6
78.3
83.3
81.3
A PERFORMANCE EVALUATION ON TRUTHFULQA
TruthfulQA is important benchmark to evaluate the truthfulness of LLMs (Joshi et al. 2017 Therefore, we also compare our proposal with the baseline methods in the TruthfulQA benchmark. The optimal classification thresholds is determined by maximizing the G-Mean value, which is defined as G-Mean TTPR (1 FPR). The results are presented in Table 4 For the ITI Li ct al. (2023). which trains multiple binary classifiers with the internal embeddings for hallucination detection, we report the best performance in their paper. As be seen, our proposal consistently outperforms the baseline methods and achieves comparable performance as I'TI when we utilize 50 in-distribution prompts. It's worth nothing that the ITTI relies on training 1024 binary classifiers in TruthQA datasets, and they report the best performance (83.3) in the validation set. Therefore, their best performance is better than our proposal which has not been trained on TruthfulQA. However, training on the validation set also limits the generalization of their method on other domains (Li al., 2023). As TruthfulQA is a very challenging dataset for LLMs, zero-shot inference results in poor performance. Therefore, we follow previous work (Bai et al.) 2022) to utilize different number of in-distribution prompts during inference time. The results show that the performance could be significantly improved when increase the number of prompts, which also explains why ITI performs good.
B COMPARISON WITH MORE COMPETITIVE METHODS
To demonstrate the effectiveness of our proposal, we also compare our EigenScore with several competitive methods, including Semantic Entropy (SemanticEnt) (Kuhn al. 2022 Shifting At- tention to Relevance (SentSAR) (Duan et al. 2023) and SelfCheckGPT (SeIFCKGPT) (Manakul et al.) 2023). We follow the experimental setting in Duan al. (023) to set the number of genera- tion to N 10 for OPT-6.7B and N 5 for LLaMA. For the results of SementicEnt and SentS AR, report the number in Duan al. 2023) directly. For the implementation of SelfCheckGPT, we leverage the SelfCheckBERTScore provided in the official code package [ The comparison results in T demonstrate that our EigenScore significantly outperforms the competitors. Additionally, both SentSAR and SeIPCheckGPT exhibit comparable performance, which is much superior to Se- mantic Entropy. Note that both SentSAR, SelfCheckGPT and our proposal evaluate the quality of LLMS' generation by exploring the self-consistency across multiple outputs. However, compared to Semantic Entropy (Kuhn et al., 2022) or SelICheckGPT (Manakul et al., (023) which relies on an- other language model for sentence embedding extraction, approach leverages the internal states of LLMs, which retain highly-come coably--onnentrateed semantic information. Besides, the EigenScore defined by the LogDet of the sentence covariance matrix is able to capture the semantic consistency more effectively compared to the sentence-win wisee similarity (Manakul et al. 2023). Furthermore, the pro- posed feature clipping strategy allows our model to identify the overconfident hallucinations, which has not been investigated by previous works<Kuhn et al. 2022
Manakul et al. 2023)
C PERFORMANCE EVALUATION ON MORE LLMS
In the main experiments, we evaluate the performance of different methods in LLaMA-7B, LLaMA- 13B and OPT-6.7B. To demonstrate the robustness of our method across different models, also
13

---

Published as a conference paper at ICLR 2024
Table 5: Performance comparison of EigenScore and and several state-of-the-ar methods CoQA dataset. AUC, represents AUROC with the sentence similarity as correctness measure, and AUC, represents using ROUGE-L as correctness measure. All numbers are percentages.
SemanticEnt
SentSAR AUC, AUC, AUC, AUC,
SelfCKGPT
EigenScore AUC, AUC,
Methods
AUC,
AUC,
OPT-6.7B
63.1
71.7
69.8
72.2
70.2
74.1
71.9
77.5
LLaMA-7B
64.9
68.2
70.4
65.8
68.7
72.9
71.2
75.7
LLaMA-13B
65.3
66.7
71.4
64.7
68.1
77.0
72.8
79.8
Table 6: Performance evaluation LLaMA2-7B and Falcon-7B. LexicalSim denotes Lexical Simi- larity and SeI/CKGPT denotes SelfCheckGPT. AUC, and AUC, are utilized as correctness measure. Other experimental settings consistent with Table T
Perplecity AUC, AUC,
LN-Entropy AUC, AUC,
LexicalSim AUC, AUC,
SeI[CKGPT AUC, AUC,
EigenScore AUC, AUC,
Methods
CoQA
62.2
66.6
69.9
75.2
74.4
77.5
72.4
75.1
78.6
80.7
LLaMA2-7b
NQ
70.8
70.2
72.1
71.2
72.1
72.9
69.1
68.1
74.4
73.7
CoQA
57.0
60.6
62.6
63.2
74.8
76.4
76.7
77.9
80.8
80.6
Falcon-7b
NQ
74.3
74.7
74.6
74.7
73.8
75.4
74.7
74.0
76.3
75.7
provide the performance comparison in the recent LLaMA2-7B (Touvron et al. 7B models (Almazrouei et al., (003)). Table reveals that our proposal consistently exhibits superior performance compared to the other methods across different LLMs.
2023b) and Falcon-
D COMPUTATIONAL EFFICIENCY ANALYSIS
As proposal is sampling based approach, additional inference cost is required to generate multiple outputs for accurate hallucination detection. We compare our proposal with the base LLM and other comparing methods in LLaMA-7B and LLaMA-13B. All experiments are performed on NVIDIA-A108 and we set the number of generations to = 10 through the experiments. The average inference time per question is shown in Fig. 5 As observed, 10 times more efficient than the methods that rely on another large model to measure the self- consistency (such as Sel/CheckGPT (Manakul et al. (023)). and shares the similar computational overhead with the LN-Entropy and Lexical Similarity. Compared to the computational overhead of generating multiple outputs, the cost of feature clipping and HigenScore computation negligible (0.06s). It is worth noting that the inference overhead required to generate multiple results is not linearly proportional the time required to generate a single output, owing the sampling and decoding strategy of the autoregressive LLM model.
EigenScore is about
E EVALUATION WITH EXACT MATCH
In the main experiments, we employ the ROUGE and sentence similarity correctness measure, which are widely used for natural language generation evaluation (Chang et al., 2023 Kuhn et al. 2022 Huang et al. 2023). In order facilitate the comparison of our work's performance with other works, we also provide the evaluation results by employing exact match (Liang et al. 2022) as the correctness score, which much more strict to determine a generation as correct. The re- sults in Table show show similar conclusions to those in Table [ which demonstrates that our proposal significantly outperforms the compared methods in most cases.
F MORE VISUALIZATION AND ABLATION FOR FEATURE CLIPPING
In Fig. 6 we illustrate the distributions of neuron activation from four selected tokens. As can be seen, the distribution changes lot across samples. Therefore, is risky to determine the clipping
14

---

Published as a conference paper at ICLR 2024
Computationa Cost Comparison in LLaMA-7B
Computational Cost Comparison LLaMA-138
Inference Cost (attaannqqusstto
Inference Cost (Secord/Quustioonn
0.81
0.81
127
1,28
1.27
0.24
0.24
0.31
BaseLLi
Perplesity
LiEerropy
LevicatSim
Se
Elgenscore
BaseLLU
Perpleaty
LiEnnsopy
Loriaatiim
SeIKKGPT
Eigenccrre
LLaMA-7B
(b) LLMMA-13B 13B
Figure 5: Inference comparison of different methods in LLaMA-7B and LLaMA-13B. BaseLLM denotes the LLM without using any hallucination detection metrics. LexicalSim denotes Lexical Similarity and SeIICKGPT denotes SelfCkeckGPT.
Table 7:
Performance evaluation with Exact Match as correctness measure. LexicalSim denotes the Lexical Similarity. The experimental settings are consistent with Table
Model
Perplexity LN-Entropy LexicalSim EigenScore
CoQA
63.7
70.7
76.1
83.0
SQuAD
57.3
72.1
76.9
83.9
LLaMA-7B
NQ
75,3
75,6
75.8
80.1
TriviaQA
82.5
83.4
81.8
82.4
CoQA
59,4
61.7
71.8
79.4
SQuAD
56.7
65.2
72.7
82.9
OPT-6.7B
NQ
79.8
78.1
73.2
79.8
TriviaQA
83.8
81.3
79.3
82.7
threshold with only the current input sample (EigenScore-C). A feasible solution is to pre-compute the optimal threshold based on a batch of input samples (EigenScore-P) Besides, another solution is to dynamically record the activation values and determine the threshold during the inference pro- cess (EigenScore-Mil presented in Table 8 The results demonstrate that determining the thresholds with memory bank works slightly better. We attribute this variability to potential differences in the activation distribu- tions across various datasets.
We have experimented with both solutions and the experimental results are
G LIMITATIONS AND FUTURE WORK
By exploring the internal states of LLM and introducing an EigenScore metric, the hallucination detection performance has been significantly improved. However, there are several limitations of this study. One critical limitation is that the proposed method relies the internal information of the LLMs, therefore cannot be applied to the black-box models (OpenAl 2023). where users can not
Table 8: Ablation study of determining the clipping threshold with different technique. EigenScore- indicates determining the threshold with the current input sample. EigenScore-P indicates pre- computing the threshold with batch of samples. EigenScore-Me denotes using memory bank to determine the optimal threshold. AUC, is reported. Methods
CoQA
NQ
EigenScore-C
78.1
74.8
FigenScore-P
79.9
75.3
EigenScore-MB
80.4
76.5
15

---

4.34496729e-01 3.77751922e-01 1.75326593e-01 9.92596975e- Published as a conference paper at ICLR 2024
Nouron Ad setee Detriddon
8
3080
2200
3005
4200
10.0
1850
2000
3000
61000
1050
2030
2000
4090
-204
1003
2003
3003
0303
ralonis
Figure 6: Activation distributions of four selected tokens in LLaMA-7B.
the hidden states. Additionally, our proposal a sampling-based approach, necessitating the generation of multiple outputs, which introduces additional inference overhead for accurate halluci- nation detection. Furthermore. another limitation that we only focus on hallucination detection in this study and have not yet mitigate the hallucination with our proposal. In the future work, we hope reduce the inference cost of our proposal and leverage the EigenScore to mitigate hallucinations. We believe the proposed EigenScore is strong baseline for detecting hallucination and we invite further researchers utilize and enhance our proposal.
H
CASES STUDY
H.1 HALLUCINATION DETECTION CASES
We show several cases with LLaMA-7B and NQ dataset. The number of generation is set to N 10. We determine the optimal detection threshold by maximizing the G-Mean value, which is defined as G-Mean = /TPR (1 - FRR). The hallucination detection thresholds for different metrics are Perplexity: 0.535; LN- LN- Entropy: 0.153; LexicalSimilarity 0.489; SelfCheckGPT: 0.168; EigenScore: -1.74. For LexicalSimilarity, a score larger than the threshold indicates non-hallucination. For smaller than the thresholds indicates non-hallucination. Green indicates
other metrics, hallucination answer and Red indicates hallucination answer generated by LLM. / (X) indicates that the hallucination is (not) correctly identified by the metric.
Question: the german princes who chose the holy roman empire were called
GTAns: prince-electors
LLMAns: electors
BatchGenerations: l'electors', "electors', "electors', "electors', "electors', 'electors', 'electors', 'electors', 'elector'', "electors"]
Perplexity: 0.361
LN-Entropy: 0.027
LexicalSimilarity: 1.0 /
SentBERTScore: 0.0
EigenScore: -2.63 /
EigenValue: Value: (4.87719579c+00) 1.00000000e-03 1. 1.00000000-00 1.00000000e-03 1.00000000e- 03 L.0000000e-003 1. 1.00000000e-03 1.00000000e-03 1.00000000e-03 1.00000000e-03]
Question: where fe best absorbed in the body
GTAns:
in the duodenum
LLMAns:
in the small intestine
BatchGenerations:: ['in the liver', 'small intestine', 'in the intestines'. 'the small intestine', 'the small intestine". 'in the liver', 'small intestine", 'fat', in the small intestine". 'fatty tissues"]
Perplexity: 0.641 /
LN-Entropy: 0.213
LexicalSimilarity: 0.357 /
SentBERTScore: 0.258 /
EigenScore: -1.40 /
Eigen Value: (3.71561676elle
02 4.207233353-02 2.49385766e-02 1.00000000e-03 1. 1.0000000e-03 1. 100000000e-03]
16

---

Published as a conference paper at ICLR 2024
Question: who did the united states win its independence from
GTAns: the British Empire
LLMAnS: britain
BatchGenerations: ['britain', 'england', 'great britain', 'great britain', 'england', 'england', 'eng- land'. 'england', grrett britain', great britain').
Perplexity: 0.598
LN-Entropy: 0.266 x
LexicalSimilarity: 0.415 x
SentBERTScore: 0.397 x
EigenScore: -2.23
Eigen Value: [4.46843402c+000
2.82423429c-01 3.88702191c-02 L00000000e-03 1.00000000e- 03 1.00000000e-03 1.00000000e-03 1.00000000-033 1. 1.00000000e-03 1.00000000e-03)
Question: who won the most stanley cups in history
GTAns: Montreal Canadiens
LLMAns:
the detroit red wings
BatchGenerations: ['the detroit red wings', "the detroit red wings', "the detroit red wings', "the detroit red wings', 'the detroit red wings', 'the detroit red wings', 'the detroit red wings', 'the detroit red wings'. 'the detroit red wings', 'the detroit red wings']
Perplexity: 0.366 X
LN-Entropy: 0.025 X
LexicalSimilarity: 1.0 x
SentBERTScore: 0.0
EigenScore: -2.63
EigenValue: Value: (5.23534401c+00 1.00000000c-03 1.00000000e-03 1. 1.0000000c-03 1 1.00000000e- 03 1.00000000e-03 1. 1.00000000e-03 1. 1.00000000e-02 1.00000000e003 1.00000000e-03]
Question: what is the second book in the alchemyst series
GTAns: The Magician
LLMAns: the alchemyst: the secret of the immortal Nicholas flamel
BatchGenerations: ['the magician in the middle', "the magician's nephew", 'the magician', 'the alchemyst', 'the magician', "the alchemyst', 'the magician in the middle', 'the magician in amster- dam', 'the magician in the middle', 'the secret of the witch']
Perplexity: 0.459 x
LN-Entropy: 0.207
LexicalSimilarity: 0.499 x
SentBERTScore: 0.431 /
EigenScore: -1.41 /
EigenValue: Value: 134993009 6.95605781e-01 3.85931973e-01 3.40671669e-01 2.15372994e- 01 1.77304781c-01 1.00000000e-03 1.00000000e-03 1.0000000e-03 1.00000000e-03]
Question: good that can be used in place of another good
GTAns: A substitute good
LLMAns: substitute
BatchGenerations: ['a substitute', 'a substitute', 'substitute', 'a substitute', 'substitute', 'substi- tute'. "substitute'. 'substitute", 'substitute", 'substitute"]
Perplexity: 0.614 x
LN-Entropy: 0.194
LexicalSimilarity: 0.844 /
SentBERTScore: 0.076 /
EigenScore: -2.40 /
Eigen Value: [5.1 1365925e+00 1.75884104e-01 1. 1.00000000e-03 1.00000000e-03 1. 1.00000000e-
03 1.00000000e-03 1.00000000e-03 1. 1.00000000e-03
1.00000000=003 1.00000000e-03]
17

---

Published as a conference paper at ICLR 2024
Question: who said will not go quietly into the night
GTAns: Welsh poet Dylan Thomas
LLMAnS:
douglas macarthur
BatehGenerations: ['douglas macarthur', 'elvis presley', 'douglas macarthur', 'douglas macarthur', "douglas macarthur', 'robin williams', 'robert f. kennedy', 'robin williams', 'robert f kennedy', 'Dwight Eisenhower']
Perplexity: 0.502 x
LN-Entropy: 0.121
LexicalSimilarity: 0.178 /
SentBERTScore: 0.337 /
EigenScore: -1.61 /
Eigen Value: [3.32824135eH] 35e+00 5.87944819e-01 3.70390066e-01 1.70849836e-01 1.17707339e- 01 5.17925563--03 1.00000000-003 1.00000000--03 1.00000000=033 1.00000000e-03]
Question: when did they stop saying mass latin GTAnS: from 1967 onwards
LLMAnS: 1969
BatchCenerations: ['in 1969', '1969", 'around 1969', 'in 1965', 'in 1969', '1969", 'In 1969, the Roman Missal was changed from Latin to the vernacular', '1969', '1969 when the second vatican council was held". '1969']
Perplexity: 0.631 /
LN-Entropy: 0.152
LexicaSSimilarity: 0.540 X
SentBERTScore: 0.273 /
EigenScore: -1.59 /
Eigen EigenValuee Value: 13.8040819c++00 4.83987672e-01 3.03207580e-01 8.80366008e-02 6.59790286e-
02 3.2.6742841e--4 1.t.lee--11.00000000000 1.00000000e-03 1.00000000e-03 1.0000000e-03 1.00000000e-03]
Question: where does the highway of heroes start and end GTAns: Glen Miller Road in Trenton
LLMAns: it starts ottawa and ends in toronto
BatchGenerations: ['start in ottawa and end in toronto', 'it starts in toronto and ends in ottawa', "it starts at the queen's 's park gate at queen's 's park and ends at toronto's coronation park", "it starts toronto and ends queen's park", 'starts in Toronto and ends in Trenton', 'it starts the CNE and ends at the cenotaph in downtown toronto', 'it starts in downtown toronto and ends in ottawa', 't starts in city Toronto and ends in the city of Hamilton', 'it starts toronto and ends ottawa', ; 'it starts in Toronto and ends in Ottawa"] Perplexity: 0.657
LN-Entropy: 0.260 /
LexicalSimilarity: 0.594 X
SentBERTScore: 0.149 x
EigenScore: -1.05 /
SigenValuee (3.319830) 8e+00 3.98560810e-00 2.170942996-01 2.06965709e-01 1.53575354e- 01 1.27925588e-01 7.82365136e-02 1-221551377-02 1.01995086--22 1.00000000e-03]
Question: which body part(s) occupy the greatest portion of the primary motor cortex
GTAns: face
LLMAns:
the hand
BatehGenerations: ['the hand', 'the hand', 'the fingers', 'the hand', 'the hand', 'the hand and the foot', 'the hand', 'the hand and the foot', 'the hand and face', 'arms and legs']
Perplexity: 1.41 /
LN-Entropy: 0.300
LexicalSimilarity: 0.568 X
SentBERTScore: 0.163 X
EigenScore: -1.69 /
Eigen Value: (3.76273036++00 6. 6.16284067e-01 1. 1.96541049e-01 1. 1.73505005e-01 1.28407 153e- 01 1.00000000e-03 1.00000000e-03 1.00000000-003 1. 1.0000000e-03 1.00000000e-03]
18

---

Published as a conference paper at ICLR 2024
Question: who said have you no sense of decency
GTAns: Joseph Nye Welch
LLMAnS: Joseph Nye Welch
Batchtenerations: ['Joseph N. Welch', 'Joseph N. Welch', 'joe stalin', 'joseph mecarthy',
'Joseph N. Welch', 'Joseph N. Welch', 'Joseph Nye Welch", "joseph mecarthy', 'joe mecarthy',
jooephh mccarthy']
Perplexity: 0.666 X
LN-Entropy: 0.212
LexicalSimilarity: 0.437 X
SentBERTScore: 0.391 X
EigenScore: -1885 /
EigenValue: Value: [3.631 14083e+00 8.11622323e-0 2.00385898e-01 3.19140618e 1.74251264e- 02 1.00000000e-03 1.00000000e-03 1. .000000000=033 1. 1.0000000e=033 1.00000000e-03]
H.2 MODEL GENERATIONS WITH MANY AND FEW OUTLIERS
To demonstrate the relationship between the number of extreme features and model outputs, we provide several examples with many/few extreme features. The results show that when there are many extreme features, the model tends to generate consistent hallucination outputs across multiple generations. Instead, when there are few extreme features, model generates diverse hallucination outputs which can be spotted by different hallucination detection metrics.
Question: who sang great gig in the sky
GTAnS: Clare Torry
LLMAnS: freddie mercury
AvgNumOOttiees: 15
BatchGenerations: ['freddie mercury', 'freddie mercury'. 'freddie mereury', 'freddie mercury',
'freddie mercury', 'freddie mereury', 'freddie mercury', 'freddie mercury', 'freddie mercury',
'freddie mercury']
Perplexity: 0.263 X
LN-Entropy: 0.028
LexicalSimilarity: 1.0 x
SentBERTScore: 0.0
EigenScore: -2.63 X
EigenValue: Value: (4.65740187c+00 1.00000000e-03 1.00000000e-03 1.00000000e-03 1.00000000e- 03 1.00000000e-00 1.00000000e-03
1.00000000e-03 1.00000000=033 1.00000000e-03]
Question: what are the top five wine producing states
GTAns: Washington
LLMAns: California
AvgNumOutliers: 13
BatehGenerations:: [' California,' CCaiforniaa'' california.', CCliformia,' California,' Cali- fornia,', California,,' california,, California," California,']
Perplexity: 0.368 x
LN-Entropy: 0.075
LexicalSimilarity: 1.0 x
SentBERTScore: 0.054 x
EigenScore: -2.42 x
Eigen Value: (5.30709315e400 1. 1.13222379--01 1. 1.00000000--03 1.00000000e-03 1. 1.00000000e-
03 1.00000000e-03 1.00000000e-03
1.00000000~033
1.00000000=003 1.00000000e-03]
19

---

Published as a conference paper at ICLR 2024
Question: how many seasons of rules of engagement is there
GTAns: 7
LLMAnS: 4 scasons
AvgNumoutliers:
BatchGenerations: I'3 seasons', '4 seasons', '4 seasons', '6 seasons", '7 seasons", '3 (2007-
2009)', 3 (2007-2009); '4 seasons', 'three", '11 seasons"]
Perplexity: 0.996 /
LN-Entropy: 0.292
LexicalSimilarity: 0.307 /
SentBERTScore: 0,285 /
EigenScore: -1.60 /
EigenValue: Value: 33.58548815e+000 5.87838054e-01 2.28057934e-01 1 1.36461300e-01 3.49712302e-
02 1.1 1.11346059e-00 3.82259086e-03 1.00000000=-03 1. 1.0000000e=003 1.00000000e-03]
Question: where did the first persian gulf war take place
GTAnS: Isracl
LLMAns: kuwait
AvgNumOuttiers:
BatchGenerations: ["Iraq'. "Iraq and Kuwait', 'Iraq', 'kuwait', 'kuwait', 'in the middle east', "in iraq'. 'kuwait', 'iraq'. "kuwait']
Perplexity: 0.546
LN-Entropy: 0.281 /
LexicalSimilarity: 0.339
SentBERTScore:: 0.224
EigenScore: -1.62 /
EigenValuee (3.59463352c+00 4.23782982e-01 2.57087067c-01 1.41513403e-01 6.20790226e- 02 1.75980481e-000 1.00000000e-03 1.0000000e-03 1. 1.00000000e033 1.00000000e-03]
H.3 IMPACT OF FEATURE CLIPPING
The texts yellow represents model generations after applying feature clipping. The results show that after feature clipping, the overconfident generations can be appropriately suppressed, and some self-consistent hallucinations are finally identified.
Question: what are the top five wine producing states
GTAns: Washington
LLMAns: California
BatchGenerations: [' California,' California,' california', California,' California,, Cali-
fornia,' California,", california,' California', California,"]
Perplexity: 0.368 x
LN-Entropy: 0.075 x
LexicalSimilarity: 1.0 x
SentBERTScore:: 0.054 x
EigenScore: -2.42 x
EigenValue: Value: (5.30709315++00 1.13222379--01 1.00000000e-03 1.00000000e-03 1.0 1.00000000e- 03 L.0000000e-03 1. 1.0000000ee003 1. 1.00000000-033 100000000e-03 1.00000000e-03]
BatchGenerations:: I'california', "california'. "Washington', "california", 'new york'. "california"
washington', 'californi',, 'new york', "michigan"
EigenScore: -1.22
Eigen Value: 83.233927555+008..4400 8 410043340e-01 2.52322804e-01 1.334775298-011111444444222
6.12184197c-02 1.02734249e-09 02 5.33703500e-00 3 0.09878029e-03 L00000000e-03]
20

---

Published as a conference paper at ICLR 2024
Question: who sang on great gig in the sky
GTAns: Clare Torry
LLMAnS: freddie mercury
AvgNumOutliers: 15
BatchGenerations: ['freddie mercury', 'freddie mercury', 'freddie mercury', 'freddie mercury',
'freddie mercury'. 'freddie mercury', 'freddie mercury', 'freddie mereury', 'freddie mercury'.
'freddie mercury']
Perplexity: 0.263
LN-Entropy: 0.028 x
LexicalSimilarity: 1.0 x
SentBERTScore: 0.0
EigenScore: -2.63
EigenValue: [4.65740187++00 1.00000000e-03 1.00000000e-03 1. 1.00000000=-03 1.00000000e- 03 L.0000000c-03 1.00000000e-03 1.00000000=03 1. 1.0000000e-033 1.0000000ee-03]
BatchCenerations: ('claire torry', 'freddie mercury". "freddie mereury", 'freddie mercury', 'fred-
die mercury', 'freddie mercury', "freddie mercury', 'freddie mereury". "freddie mercury'. "freddie mercury']
EigenScore: -2.38 x
Eigen Value: 443874550000+0033149888826666860 1.00000000e-03 1.0000000e-033 1000000000-03 1.000000000-03 1.000000000-033 1.00000000-03 1. 1.00000000e-03 1.00000000~-33]
Question: who are you in assassin's 's creed 4
GTAns: third-person perspective
LLMAnS:
Edward Kenway
BatchGeneratiios:: ['Edward Kenway', 'Edward Kenway", 'Edward Kenway". 'Edward Kenway',
'Edward Kenway", 'Edward Kenway', 'Edward Kenway', 'Edward Kenway', 'Edward Kenway',
'Edward Kenway']
Perplexity: 0.264 X
LN-Eatroppy: 0.002
LexicalSimilarity: 1.0 x
SentBERTScore: 0.0
EigenScore: -2.67 x
Value: (2.10973201e+00 1.00000000e-03 1. 1.0000000e-00 1.00000000e-03 1.0.11100000. 1.00000000e- 03 1.00000000e-03 1.00000000e-03 1.00000000e-03 1.0000000ee03 1.0000000ee033
BatchGenerations: ('Edward Kenway', 'Edward Kenway', 'Connor', 'Edward Kenway', 'con-
nor'. "Comnor'. "alexander hamilton',
'Edward Kenway', ezioo, "connor'
EigenScore: -1688
Eigen Value: 33447255777++07 7.41123811-01 3 2. 2. 15050807e-02
1.00000000e-03 -000000000e033 .00000000e-03 0000000000=003 .000000000-03)
21


Reviews:
[{"review_id": "Dm1AvYbPpT", "reviewer": "Reviewer_VqqH", "summary": "This work proposes a metric, EigenScore, to measure whether the LLM can answer the question correctly. EigenScore is calculated from internal states of multiple responses to the same question. Furthermore, the authors design a feature clipping approach to truncate the extreme activations of internal states to reduce the overconfident generations. This benefits mitigating overconfident hallucination. The empirical results show the effectiveness of this work in detecting hallucinations.", "questions": "1. I find that the evaluation metrics are related to fact-checking. I wonder if there are diverse metrics to evaluate the hallucinations.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper studies a timely and important research problem in llm. Preventing hallucinations in inference time is a huge research challenge in the community.\n2. This paper is easy to follow.\n3. The proposed approach is effect in detecting hallucinations.", "weaknesses": "1. The related work comparison is not comprehensive. There is no related work comparison between the internal-state detection approach. For example, [1] investigates mitigating the hallucinations by editing the internal states of llm. They first measure the possibility of generating hallucinations by observing the internal states and then editing those internal states. It might be better if the authors can compare their approach with the internal state-based hallucination detection approaches, e.g., [1].\n2. The llm models involve multiple transformation blocks; which block internal state do the authors investigate? It would make the paper more impactful if the authors could present the ablation on transformation blocks.\n\n[1] Li, Kenneth, et al. \"Inference-Time Intervention: Eliciting Truthful Answers from a Language Model.\" NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g9uwXL83gh", "reviewer": "Reviewer_Kmk8", "summary": "This paper proposes a new framework called INSIDE that leverages the internal states of large language models (LLMs) to detect hallucinated content generated by the LLM. A key component is the EigenScore metric, which measures semantic consistency in the embedding space and represents differential entropy. This helps identify overconfident and contradictory generations. They also introduce a test time feature clipping approach that truncates extreme activations, thereby reducing overconfidence and enabling better hallucination detection. Experiments on several QA benchmarks show state-of-the-art performance in hallucination detection. Ablation studies verify the efficacy of the proposed techniques, including the EigenScore metric and feature clipping. Overall, INSIDE provides an effective way to detect hallucinated content from LLMs by leveraging their internal states and controlling overconfidence. The EigenScore and feature clipping are two key innovations that improve hallucination detection.", "questions": "1. How about the proposed method compared with this work [1] on the TruthfulQA dataset?  Both work relies on the internal state of the LLM models. \n\n[1] Li, Kenneth, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. \"Inference-Time Intervention: Eliciting Truthful Answers from a Language Model.\" arXiv preprint arXiv:2306.03341 (2023).\n\n2. How about the effect of sentence embeddings on the final performance?  \n\n3. Can you provide the computational cost estimation of the whole framework and which type of resources you are using? \n\n4. Since the evaluation metric is AUROC, I am wondering if, can you compare the proposed method with the GPT 3.5 or 4 performance. Or can you add some comparison with the hallucination detection baseline methods, e.g., selfcheckGPT as you mentioned in the paper? \n\n5. The current baselines are mainly from the authors' choice without open baselines, e.g., semantic uncertainty (ICLR 2023). I noted that many related works are good baselines that can be used for comparison.\n\n\n---\npost rebuttal \n\nI think the authors' response and experiments addressed my concerns so I raise my evaluation to support my decision.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is very interesting and easy to follow \n2. The paper poses a unique angle to detect hallucination from internal states with eigenscore\n3. The idea is overall innovative while some key points are inspired by previous work\n4. The experiments are solid to support the paper's claim", "weaknesses": "1. One of the obvious shortcomings of the proposed method is the applicability to blackbox LLMs, e.g., GPT, PaLM and Claude.  The method relies on the internal state of the model, e.g., LLaMA and OPT, but the authors may not discuss the limitation well. \n\n2. The computational cost is another concern since essentially this method is a sampling-based approach. While the number of samples is not large, how to further reduce the cost for real production is still unsolved. \n\n3. The proposed method is mainly inspired by uncertainty estimation but how to bridge the connection with hallucination detection is still unclear, even though the results look promising. \n\n4. Beyond hallucination detection, is that possible to mitigate hallucination through the proposed framework eventually?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sqss9mmZuH", "reviewer": "Reviewer_9Srq", "summary": "This work follows the line of uncertainty-based methods for hallucination detection to probe the inconsistency between the independently generated response for the same query. The idea is to take the eigenvalues of LLM internal states' covariance matrix to measure the inconsistency.", "questions": "For the correctness Measure, I agree that the evaluation of answers is challenging, but it's better to have an exact match score so that we can compare it with other existing works. \n\nAs feature clipping is claimed as a contribution of this work, could you provide some ablation studies for it? I am curious why it needs a dynamic memory bank. Does that mean the distribution of activation values changes a lot across samples?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "If we don't consider concurrent works, such as \"Representation Engineering: A Top-Down Approach to AI Transparency\", the proposed method is novel and feasible since measuring covariance between several independent generations is intuitive. \n\nThe proposed method is simple and easy to implement and use without tuning models, and the performance improvement is significant.", "weaknesses": "The experiment setting could be improved since this work targets hallucination detection, which is a broad area, and there are various types of datasets. Only reporting results on QA benchmarks may be insufficient to support the contribution. At least, I think truthfulQA is closer to the topic of hallucination compared to the benchmarks used in the paper. Additionally, it's better to include recent LLMs, such as LLaMA2 and Falcon. \n\nThe covariance of LLM internal states may contain distracting information, such as the diversity of expressions, styles, and terminology. Since there are multiple factors that influence the difference between internal states, it's hard to guarantee the generalization ability of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gj98ErFktu", "reviewer": "Reviewer_5aJw", "summary": "This paper proposed a new method for detection of LLM hallucinations. The method called EigenScore is based on extracting internal representation (layer activations) of the proposed sequences. Then, they compute the covariance matrix of K proposed sequences and then compute the logarithm determinant of the matrix. Intuitively, it measures entropy of the proposed sequences. They also propose the technique for reducing the number of confident hallucinations by clipping the outlier layer activations. In the experiments they empirically show that their proposed metric correlates with the correctness of the outputs more than other baselines such as Perplexity, Lexical similarity, Length normalized entropy and Energy.", "questions": "I have only minor questions and suggestions and I would like to hear authors' discussion on those points.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The proposed method is simple, intuitive and it is well explained. The idea of looking into the internal states for hallucination detection is well motivated. I enjoyed the clear presentation of the paper. For example, the background section was quite useful for the understanding of the methods. The results seem to clearly show the advantage of the proposed method over several established baselines. The experiments are conducted on several diverse datasets, and several language models. The advantage of an alternative method on a benchmark with short answers is well explained and it is useful to inform the reader in which situations the proposed method should be employed. The ablation studies are quite complete and they show that the proposed method works under a wide range of settings.", "weaknesses": "Minor points and suggestions:\n- Regarding the outliers in the activations, it would be nice to see if these outliers really lead to overconfident predictions. For example, is there any correlation between the number of outliers and accuracy of the answer? At least, it could be informative to see a few generated examples corresponding to many outliers or no outliers.\n- For Table 2, it would be interesting to see if the quality of answers is increased after using clipping (as confident mistakes are excluded).\n- In general, it would be nice to see some qualitative examples of hallucinations that are detected by the proposed method, but not detected by the baselines.\n- centralized matrix -> centering matrix\n- Typos: \"prior works that rectifying\", \"nli-roterta-large\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)