LLM Internal States Reveal Hallucination Risk Faced With a Query

Authors: ziwei  ji
delong  chen
etsuko  ishii
samuel  cahyawijaya
yejin  bang
bryan  wilie
pascale  fung
Venue: EMNLP_2024_Workshop_BlackBoxNLP
Published: 2024-08-14 05:38:12

Abstract: The hallucination problem of Large Language Models (LLMs) significantly limits their reliability and trustworthiness. Humans have a self-awareness process that allows us to recognize what we don't know when faced with queries. Inspired by this, our paper investigates whether LLMs can estimate their own hallucination risk before response generation. We analyze the internal mechanisms of LLMs broadly both in terms of training data sources and across 15 diverse Natural Language Generation (NLG) tasks, spanning over 700 datasets. Our empirical analysis reveals two key insights: (1) LLM internal states indicate whether they have seen the query in training data or not; and (2) LLM internal states show they are likely to hallucinate or not regarding the query. Our study explores particular neurons, activation layers, and tokens that play a crucial role in the LLM perception of uncertainty and hallucination risk. By a probing estimator, we leverage LLM self-assessment, achieving an average hallucination estimation accuracy of 84.32% at run time.

Main paper: 
 introduction: 
 Humans have an awareness of the scope and limit of their own knowledge (Fleming and Dolan, 2012; Koriat, 1997; Hart, 1965) , as illustrated in Fig. 1 . This cognitive self-awareness ability in humans introduces hesitation in us before we respond to queries or make decisions in scenarios where we know we don't know (Yeung and Summerfield, 2012; Nelson, 1990; Bland and Schaefer, 2012) . However, LLM-based AI assistants lack this cognitive uncertainty estimation. Consequently, they tend to be overconfident and may produce plausible-sounding but unfaithful or nonsensical contents called hallucination or confabulation (Ji et al., 2022; Xiao and Wang, 2021; Bang et al., 2023; Xiong et al., 2023) . This problem limits their applications in numerous real-world scenarios and undermines user trustworthiness.Previous research (Bricken et al., 2023; Templeton et al., 2024; Bills et al., 2023; Wu et al., 2024) have explored the internal states of language models that capture contextual and semantic information learned from training data (Liu et al., 2023; Chen et al., 2024; Gurnee and Tegmark, 2023) . Nevertheless, internal states of language models sometimes exhibit limited generalization on unseen data and their representation effectiveness can be undermined by flawed training data or modeling issues (Wang et al., 2022a; Belinkov and Glass, 2019; Meng et al., 2021; Xie et al., 2022; Carlini et al., 2021; Yin et al., 2023a) . Notably, recent works have shown that the LLM's internal states can potentially detect hallucinations in texts (Azaria and Mitchell, 2023; Chen et al., 2024; Su et al., 2024) . However, these works examine texts not exclusively produced by the same LLMs whose internal states are analyzed, highlighting the necessity for further investigation into the LLM self-awareness and how their internal states correlate with their uncertainty and own hallucination occurrence.Our work takes a step further by investigating whether LLM internal states have some indication of hallucination risk given queries and whether it can be reliably estimated even before the actual response generation (Fig. 1 ). We conduct a comprehensive analysis of LLMs internal mechanisms in terms of training data sources and across 15 diverse NLG tasks that extend beyond the QA task (Snyder et al., 2023; Slobodkin et al., 2023) and span over 700 datasets. We explore particular neurons, different activation layers, and tokens that play a crucial role in the LLM perception of uncertainty and hallucination risk. Employing a probing estimator (Belinkov, 2022) on the internal states associated with the queries, we validate their self-awareness and ability to indicate uncertainty in two aspects: (1) Whether they have seen the query in training data, achieving an accuracy of 80.28%. (2) Whether they are likely to hallucinate regarding the query, achieving an average estimation accuracy of 84.32% across 15 NLG tasks. We propose that understanding these representations could offer a proactive approach to estimating uncertainty, potentially serving as an early indicator for the necessity of retrieval augmentation (Wang et al., 2023) or as an early warning system. 
hallucination and training data: 
 The sources of hallucination in LLMs can be traced back to data and modeling (Ji et al., 2022) . Factors tied to data encompass unseen knowledge, task-specific innate divergence, noisy training data, etc. From the modeling perspective, hallucinations can be traced to the model architecture, alignment tax, teacher-forced maximum likelihood estimation (MLE) training, etc.A common situation where hallucinations from data occur is when LLMs attempt to provide information on unseen queries that are not included in their training set, rather than refusing to reply. Previous works (Kadavath et al., 2022; Rajpurkar et al., 2018; Onoe et al., 2022; Yin et al., 2023b) explore to identify the unseen data based on various indicators such as text similarity, perplexity. We investigate the capability of LLMs to recognize whether they have seen the query in training data via novel analysis of their internal states. To facilitate analysis, we craft two sets of queries by collecting news from periods before and after the release of the LLM we analyze to represent unseen and seen data, respectively.However, in the real-world scenario, it's impractical to definitively categorize data as entirely seen or unseen due to the inability to access the vast training data of LLM. Thus, we expand the preliminary insights and further investigate LLMs' self-awareness of recognizing whether models are likely to hallucinate regarding the query. It's important to note that the hallucinations are sourceagnostic, meaning they can result from both unseen and seen data. The latter can still trigger hallucinations due to deficiencies in modeling. To facilitate analysis, we construct data by using LLM to directly generate responses to queries across diverse NLG tasks and then label the hallucination level in the responses. 
methodology: 
 This section begins with an introduction to the problem formulation of uncertainty estimation faced with queries in § 3.1. We construct datasets in § 3.2 focusing on two dimensions: (1) the distinction between queries seen and unseen in the training data; (2) the likelihood of hallucination risk faced with the queries. To validate the efficacy of internal state representation in hallucination estimation, we visualize the neurons for perception extracted from a specified LLM layer ( § 3.3) and then leverage the probing classifier technique (Belinkov, 2022) on top of internal states associated with the last token of queries ( § 3.4).Suppose we have an LLM f parameterized by θ. It is able to gain internal states I and generate response r given user query q represented as I θ,q , r θ,q = f θ (q). We aim to investigate the selfawareness of LLM, specifically how their internal states I relate to their level of hallucination risk h when faced with a query q.We employ a dataset D θ = {⟨I train θ,q,i , h train i ⟩} N i=1 consisting of N query-label pairs. These pairs serve to represent the behavior of f θ Here, h train i denotes the level of hallucination risk, which is labeled based on (1) the query's presence in the training data or (2) the degree of hallucination in the response r to q. Thus, our objective is mathematically expressed as:EQUATIONHere, E signifies an estimator function. The combi- nation of model-specific attribute via f θ , and queryonly attribute via q, allows it to accurately capture the characteristics of individual LLMs and fosters a more efficient prediction mechanism that mirrors human cognitive processes.As introduced in § 1 and 2, we investigate LLM internal states' self-awareness and ability to indicate uncertainty in two aspects: (1) whether they have seen the query in training data; and (2) whether they are likely to hallucinate when faced with the query.(1) Seen/Unseen Query in Training Data The unseen queries will trigger hallucinations due to the lack of information within the model's training data when the model doesn't refuse to respond. In other words, hallucinations triggered by unseen queries are data-related. To investigate the distinguishability between seen and unseen queries, we construct a compact dataset consisting of two distinct sets of queries. For the seen group, we utilize historical BBC news in 2020 highly likely exposed during LLM's training. For the unseen group, we utilize recent BBC news in 2024 after the release of the LLM we analyze. To ensure comparability, we ensure these two sets share similar length distributions and semantic information via sentence embeddings2 . The two groups are both from LatestEval (Li et al., 2024) , a benchmark designed to tackle data contamination in evaluation through dynamic and time-sensitive construction.The query is Tell more details about the news: {news_title}.(2) Hallucination Risk faced with the Query We first construct data using LLM to directly generate responses to queries in diverse NLG tasks. Subsequently, for labeling the responses and corresponding queries, a comprehensive integration of NLG metrics assesses the levels of hallucination.We select 15 NLG task categories including QA, Summarization, Translation, etc consisting of over 700 datasets from Super-Natural Instructions benchmark (Wang et al., 2022b) 3 . This generation only uses the parametric knowledge of LLM which is a proxy of performance in real-world applications. This generation process can also be in other settings, such as retrieval-augmented generation (RAG), to explore whether the internal states can estimate hallucination risk or other aspects' performance in these settings.To evaluate the generated responses, we implement a multi-faceted evaluation approach. We employ classical Rouge-L (Lin, 2004) , which compares the generated response with gold-standard reference. To measure hallucination level, we also utilize Natural Language Inference (NLI)4 and Questeval (Scialom et al., 2021) . NLI is a common metric for hallucination evaluation (Ji et al., 2023a,b) which assesses the logical consistency/entailment of generated text with the provided context or the reference. QuestEval is a QA-based metric for evaluating the faithfulness of the output in generation tasks. This work adopts its reference-dependent mode depending both on the input source and golden reference.To make up for deficiencies of single automatic metrics, we integrate these three metrics compre-hensively. If NLI predicts entailment and both Rouge-L and Questeval exceed their respective median values, we assign a label of 1. Conversely, if NLI predicts contradiction or neutrality, and both Rouge-L and Questeval fall below their median values, we assign a label of 0. This labeling strategy not only provides a binary quality assessment but also reflects a multi-dimensional evaluation of the text, capturing the hallucination level of the generated responses.Internal states play a crucial role in language models, encapsulating rich contextual and semantic information learned from predicting tokens. They are adept at recognizing complex patterns and relationships pertinent to various NLP tasks, which positions them as potentially powerful tools for estimating the risk of hallucinations (Azaria and Mitchell, 2023; Liu et al., 2023; Chen et al., 2024) . Furthermore, previous works (Azaria and Mitchell, 2023; Ahdritz et al., 2024; Liu et al., 2024) have demonstrated that the activations of the last token from the last layer in LLMs contain one of the most useful features. Therefore, we take these representations for preliminary analysis on the selfassess sense of internal states and the role of specific neurons in the uncertainty and hallucination estimation. Specifically, we employ a feature selection method based on Mutual Information (Kraskov et al., 2004) to measure the relevance of different features/dimensions for distinguishing between the categories in a dataset.In the context of NLG tasks including dialogue, QA, and translation, we select the eight most significant neurons/dimensions from the last activation layer and visualize them in Fig. 2 . We observe that these neurons exhibit sensitivity to uncertainty, allowing them to distinguish between different hallucination levels given known and unknown queries. In other words, there exist individual neurons within LLM that can fairly perceive uncertainty and predict future hallucinations. This approach not only enhances our understanding of the neural correlates of hallucinations but also paves the way for developing targeted interventions that mitigate the effects of hallucinations. Based on the above preliminary analysis and previous works, we use the activations corresponding to the last token of queries from a specified layer in LLMs, denoted as x q , as the input for our estimation model. The accessibility and ease of obtaining these states further underscore their practicality for such applications.For the architecture of our estimator, we employ a variant of the multilayer perceptron (MLP) adapted from the Llama (Touvron et al., 2023) . The estimator is mathematically formulated as:H = down(up(x q ) × SiLU(gate(x q ))) (2)where SiLU is the activation function. down, up, and gate are linear layers for down-projection, upprojection, and gate mechanisms, respectively. The combination of internal states and the Llama MLP structure handles the complexity of hallucination risk estimation in NLG tasks. 
llm: 
 In this work, we primarily use Llama2-7B (Touvron et al., 2023) as our generative model and delve into its internal states to access hallucination risk estimation. In addition, we explore the impact of different internal states in the Mistral-7B (Jiang et al., 2023) in § 5.To explore query-only uncertainty estimation, we involve straightforward prompt-based approaches as baselines.Zero-shot Prompt We directly ask the LLM whether it can accurately respond to the query via the following prompt: "Query: {Query}\n\nAre you capable of providing an accurate response to the query given above? Respond only to this question with 'yes' or 'no' and do not address the content of the query itself."In-Context-Learning (ICL) Prompt We ask the LLM whether it can accurately respond to the query and give some examples: "Are you capable of providing an accurate response to the following query? Respond only to this question with 'yes' or 'no' and do not address the content of the query itself.\n\nQuery: {Example Query 0}\nAnswer: no\n\nQuery: {Example Query 1}\nAnswer: yes...\n\nQuery: {Query}\nAnswer:" Perplexity (PPL) Considering the prompt-based methods only use the model's inner knowledge, we also incorporate the distribution of the training dataset and employ a Perplexity (PPL)-based baseline. Assume LLMs are trained on a hypothetical large dataset that perfectly contains every possible query-response pair, where the responses are guaranteed to be faithful. Then, the hallucination estimation can be simply done by checking whether the given query appears in the training corpus (Lee et al., 2021; Kandpal et al., 2023) . To determine this threshold, we first calculate the PPL for each query. Subsequently, we identify the optimal PPL threshold that yields the maximum accuracy on our training dataset. This optimal threshold is then applied to the test dataset to gauge the accuracy of our hallucination risk estimation method.For the classification task with the discrete type predicted, we utilize F1 and Accuracy to measure the quality of predicted categorization. These high results shed light on the effectiveness of our internal state-based method in identifying unseen queries. This phenomenon is aligned with the previous works (Kadavath et al., 2022; Yin et al., 2023b ) which find the model can distinguish answerable and unanswerable questions that include future information.(2) Hallucination Risk faced with the Query For estimating hallucination risk, as depicted in Fig. 3  
analysis: 
 Layer Depth Positively Correlates with its Prediction Performance. We systematically dissect the contribution of each layer to the overall hallucination risk estimation. We hypothesize that certain layers may be more indicative of hallucinatory propensities than others, and our analysis seeks to validate this hypothesis. As shown in Fig. 4 , early Layers perform poorly since they often capture basic syntactic information. Intermediate layers perform better since these layers typically encode more complex semantic relationships. Deep layers perform best and learn hallucination patterns with high-level presentation. This observation is different from Azaria and Mitchell (2023) where middle-layer hidden states of statements perform best in recognizing lying.LLMs To evaluate the impact of the LLM's Internal State, we use Mistral-7B's internal state to assess Llama2's hallucination risk. As shown in Tab. 2, the results for four common NLG tasks exhibit a decrease compared to Llama2's own internal states. Since different LLMs share a similarity in model architecture and data, there is a potential for zero-shot transfer. Nonetheless, the most effective predictor of LLM's generative performance is still its own internal state, which underscores the importance of considering model-specific assessments rather than universal ones.Internal States Share Features inner-task but do not Cross-task. As shown in Tab. 1, we evaluate the generalization across different NLG tasks and within the same NLG task. Specifically, we examine zero-shot performance in QA and translation. While the zero-shot performance within these individual tasks is acceptable, the cross-task generalization remains relatively weak, aligned with the findings reported by Kadavath et al. (2022) .In addition, we evaluate our estimator trained in QA on the out-of-domain hallucination QA dataset ANAH (Ji et al., 2024) to test our estimator's performance in the hallucination aspect and its generalization. ANAH is a bilingual dataset that offers analytical annotation of Hallucinations in LLMs within Generative QA. Our work uses English samples and treats the hallucination type as the label in the testing stage. The F1 score reaches 78.56% and the accuracy is 78.83%. These relatively high results shed light on the effectiveness of our internal state estimator in handling hallucination challenges and further show our generalization capabilities. Therefore, the features in internal states are shared with OOD data within the same task but not shared across tasks.Internal State as an Efficient Hallucination Estimator Our estimator has three linear layers which requires minimal computing power. As shown in Fig. 5 , our estimator demonstrates impressive efficiency. Specifically, likelihood-based costs 1.36s per sample, while internal-state-based costs only 0.05s per sample. This rapid inference speed is essential for real-world applications. The generation time, with a maximum token length of 50 and a batch size of 1, is 3.37 seconds. Notably, Questeval costs the most time 10.25s in total.Hallucination Rate During the labeling process mentioned in § 3.2, we obtain the hallucination rate in the responses of each task. As illustrated in Fig. 6 , the hallucination rate fluctuates significantly across NLG tasks. Among them, Title Generation exhibits the highest rate since its divergent nature and there is no unique and standard answer. In contrast, Number Conversion gains the lowest rate since the task is relatively easy and the answer is fixed leaving less room for hallucination.To further understand the mechanisms behind hallucination, we dissect the process of the queries triggering hallucinations at a fine-grain level. Inspired by the Gradient-weighted Class Activation Mapping (Grad-CAM) technique (Selvaraju et al., 2017) , we quantify the average gradients of input embedding associated with each token in the joint operation of the LLM and estimator. Specifically, we focused on how these tokens influence the LLM's internal state and the subsequent estimation of hallucinations based on this internal state. Fig. 7 indicates that tokens within an unknown query contribute unequally to the occurrence of hallucinations. We observe that the tokens that are part of unfamiliar named entities or carry critical information exhibit a higher impact. For instance, "amniotes" in the QA task and "麻雀" in the translation task gain higher gradients and significantly impact hallucination estimation. This could be attributed to the system's attempts to generate fluent responses despite gaps in its understanding or knowledge about these entities.Error Analysis Although our method performs better than the baselines in the estimation task, it still generates a few cases of failure. To gain more insight into our model, we present a failure example in Tab. 3 and conduct an error analysis. For example, the estimator predicts that LLM can correctly answer the query "What is the term for tough, flexible connective tissue that contains the protein collagen?" But LLM replies "ligaments" instead of "cartilage", which is hallucinated. the estimator predicts that LLM will hallucinate when faced with "what appropriately nicknamed pacific location?" But LLM replies without hallucination. 
related work: 
 Knowledge Boundary Researchers investigate the boundary of parametric knowledge in LLMs which aim to uncover what models know and where their capabilities end. Current research on the knowledge boundary predominantly narrows down this investigation to specialized tasks, charting a map of knowns and unknowns within these defined borders. In the representative QA field, Rajpurkar et al. (2018) and Yin et al. (2023b) collect datasets containing known and unknown questions and develop classifiers to differentiate them. The concept of unanswerability in these works is universal and model-agnostic, such as philosophical questions and unsolved mysteries. In contrast, Kadavath et the model will answer correctly, while the latter focuses on whether the model linguistically refuses to answer. Out-of-domain or out-of-distribution detection (Zhou et al., 2023; Ryu et al., 2018; Tan et al., 2019; Yang et al., 2021; Zheng et al., 2020) are also relevant areas dealing with the differentiation of unknown/unseen from training data, with main focus on classification tasks. Our method is versatile across various NLG tasks without requiring fine-tuning of LLMs.Hallucination Detection The phenomenon of hallucination in NLG encourages a variety of detection methods (Min et al., 2023; Ji et al., 2024; Li et al., 2023; Scialom et al., 2021) . Some of these methods delve into the internal states for detection. Azaria and Mitchell (2023) , for example, collect a true-false statement dataset with artificial guidance and the classification results indicate that the LLMs' internal state can reveal the truthfulness of statements. INSIDE (Chen et al., 2024 ) also leverages LLMs' internal states and proposes EigenScore for evaluating the self-consistency of responses, thereby serving as a proxy for hallucination levels. MIND (Su et al., 2024) , an unsupervised training approach, distinguishes the hallucinated continuation text from the original Wikipedia content based on internal states. Snyder et al. ( 2023) explore the query-only detection within the QA task based on internal states, gradients, and probabilities. On the other hand, Xiao and Wang (2021) shows evidence that higher uncertainty corresponds to a higher hallucination probability. Uncertainty estimation methods, such as (Xiao et al., 2022; Xiong et al., 2023; Kadavath et al., 2022) , predict the reliability of their natural language outputs and can also serve as a tool for hallucination detection. Previous works leverage the LLM's internal states for the text to be measured which is not necessary from the same LLM. Differently, this work focuses on self-awareness corresponding to the queries across multiple NLG tasks. 
conclusion: 
 Inspired by human self-awareness, this work demonstrates the latent capacity of LLMs to selfassess and estimate hallucination risks prior to response generation. We conduct a comprehensive analysis of the internal states of LLMs both in terms of training data sources and across 15 NLG tasks with over 700 datasets. Employing a probing estimator on the internal states associated with the queries, we assess their self-awareness and ability to indicate uncertainty in two aspects: (1) recognizing whether they have seen the query in training data, achieving an accuracy of 80.28%5 . (2) recognizing whether they are likely to hallucinate when faced with the query. The results demonstrate that internal state-based self-assessment outperforms PPL-based and prompt-based baselines, with an average estimation accuracy of 84.32% across all tested datasets. In addition, we explore the role of particular neurons in uncertainty and hallucination perception and reveal a positive correlation between the depth of activation layers in an LLM and its predictive accuracy. The consistency of internal states across different models suggests a potential for zero-shot transfer, but model-specific estimation is the optimal strategy. Challenges of generalizing these findings across different tasks are noted, despite observing promising generalizations within the same NLG tasks.For future work, we aim to refine our methodology to enhance the robustness and generalization across various NLG tasks in the field of hallucination risk assessment. In addition, we will involve a broader spectrum of LLMs to extend the applicability of our findings. 
limitation: 
 Model Coverage This work primarily investigate the widely used LLM, Llama2, due to its prevalence in current NLG applications. However, it does not encompass other LLMs. In the future, we will extend the scope of LLMs to enhance the robustness and applicability of our results. 
ethical considerations: 
 In our experiments, we utilized datasets that are either publicly accessible or synthetically generated, thereby circumventing any potential adverse effects on individuals or communities. The datasets employed in this investigation were meticulously curated and processed to uphold the principles of privacy and confidentiality. We ensured the exclusion of any personally identifiable information, with all data undergoing anonymization before any analysis was conducted.When contemplating the deployment of our research outcomes, we recognize the inherent risks and ethical dilemmas involved. The tendency of LLMs to produce hallucinations could disproportionately affect various demographic groups, a consequence of the inherent biases in the training datasets. We are committed to the identification and rectification of such biases to forestall the continuation of stereotypes or the inequitable treatment of any demographic.By adhering to these ethical considerations, we aim to contribute positively to the field of NLP and ensure that advancements in understanding and mitigating hallucinations in LLMs are achieved responsibly and with consideration for the broader societal impact. 
 
 Appendix: Human judgment is extremely resource-intensive for hallucination judgment. The extensive time commitment and financial expenditure required are beyond the scope of this study, particularly given the large scale of the datasets. Consequently, this research did not include human evaluation in the data labeling process in § 3.2.Comparative Performance Our method predicts the risk in advance of generation and depends solely on the query. It may lead to a trade-off in performance compared to other existing approaches that consider both the query and the response.This work uses benchmark Super-Natural Instructions (Wang et al., 2022b) which includes 1,616 diverse NLP datasets covering 76 distinct task types. We select 15 NLG task types and list all datasets included in each NLG task in Tab. A2.Separate Metric as Continuous Regression Label In addition to the comprehensive integration of all metrics (i.e. NLI, Rouge-L, Questeval) described in § 3.2, we analyze our internal state-based method's performances when treating each metric as the label, separately.We consider three forms of "golden score" for each metric. First, the absolute values, which serve as the target for regression, with higher scores indicating fewer hallucinations. We consider the probability of entailment as the absolute value of the NLI metric. Second, we standardize these absolute values using the minimum and maximum values from the training dataset to obtain normalized "golden scores". Third, we use the relative rankings of these scores within the training dataset as an alternative regression target.For the regression task with continuous score predicted, we utilize Root Mean Squared Error (RMSE) to measure the average difference between the values predicted by our estimator and the actual values.As shown in Fig. A1 , our method's prediction performance varies across the form of the "golden score". For each metric, the RMSE is the smallest when predicting absolute value, which indicates that the hidden state performs best in predicting the absolute value of the metric. Conversely, the highest RMSE occurs when the model attempts to predict the relative rankings, implying that predicting the precise ordering of the metrics is more challenging for the hidden state representation.Estimator Backbone Instead of Llama MLP, we employ a standard MLP as the backbone of the estimator. The results in Tab. A1 demonstrate that Llama MLP outperforms the standard MLP.The input dimension of our classifier is 4096 and the hidden dimension is 11008, which are aligned with Llama2-7B. We train our classifier with the following settings and hyper-parameters: the epoch is 10, the batch size is 128, the learning rate is 1e-5, and the AdamW optimizer has a linear scheduler.Our model is trained on 1 NVIDIA A800 GPU.In this paper, we use ChatGPT to improve the writing at the grammar level. The source code can be obtained from https://github. com/ziweiji/Internal_States_Reveal_Hallucinationhttps://huggingface.co/ sentence-transformers/all-mpnet-base-v2Please find the full list of NLG task categories in Fig.3orand the full list of tasks in Tab. A2 in § A. 4 https://huggingface.co/MoritzLaurer/ mDeBERTa-v3-base-xnli-multilingual-nli-2mil7Please refer to the first parts of § 3.2 and § 5.1. 


Decision: Reviewers agree on the importance of the topic and find the approach robust. However, there are some concerns about the reliability/validity of the labels/annotations and the overclaims in the paper. I accept the paper with the borderline score and highly encourage the authors to take the reviewers' comments into account to improve the writing and presentation of the paper, try to clarify the visualizations and detail experimental designs, incorporate suggested references, moderate the tone when drawing conclusions, and be more careful with the terminology used in the paper for the camera-ready version.

Reviews: review: paper_summary: paper_summary: The paper presents a method to probe a language model's internal state representation to detect hallucinations. The paper considers two settings: hallucinations as generations not supported in the training data, and hallucinations as detected in the ANAH framework. The paper finds that the probe is able to correctly label hallucinated generations with high accuracy/F1 score, higher than prompting and perplexity-based baselines. The paper also presents analyses of the influence of layer depth, probe generalization, and the efficiency of the proposed method for detecting hallucinations.
strength_weakness: strength_weakness: - The way the hallucination risk labels are obtained is unclear, and it is not apparent what the probe is detecting in experiment 2
- The validity of the labels for the probes is not independently assessed – how are we to know that these probe labels, that are automatically generated, are appropriate?
- It is unclear how figure 2 is interpreted to be showing the separability of hallucinated generations
- Probing method typically need to be contextualized with results from control tasks [1], which don't seem to be discussed, and would help understand the presented accuracy numbers better


[1] John Hewitt and Percy Liang. 2019. Designing and Interpreting Probes with Control Tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, Hong Kong, China. Association for Computational Linguistics.
questions: questions: - The use of "self-awareness", and references to inspiration from human self-awareness relate to the status of LLM consciousness, which might be considered to be out of the scope of this work, and best adjudicated elsewhere
- When the three labelling methods (ROUGE-L, NLI, QuestEval) are ensembled, the rule seems quite conservative in treating the label as 0 if any of the methods labels it 0. Is there evidence that this strategy is appropriate?
paper_summary: paper_summary: This paper entitled "LLM Internal States Reveal Hallucination Risk Faced With a Query" explore the ability of LLMs' internals to reveal a tendency to hallucinate, even at the query stage.

The authors examine their question by two main types of datasets -
(1) Questions regarding news from the past year (which the model was not exposed to, since it is after the training point) versus old news
(2) Collection of 15 NLG dataset automatically annotated for hallucinations risks

The authors compared the internal state of correct answers vs wrong.

The authors conclude that: LLM internal states indicate (1) if query was on training data, and (2) if they likely to hallucinate
strength_weakness: strength_weakness: My main criticism is for the level of documentation, over claiming regarding conclusions and the experiment setup was not satisfactory - it is not entirely clear what was carried out, lacking details. I'm not sure that the authors measuring "hallucinations" in their experiments (see my comments below).

In addition, there was a lack of reference to relevant literature  (see examples in "missing references" under comments and suggestions). 
How does this work add to previous existing literature? 

Questions and comments in chronological order:
Figure 2 - What is the meaning of the space (the positions of the points on the x-axis and y-axis) ?

"Seen/Unseen Query in Training" dataset:
I did not understand how the authors determined whether the models had seen the query in training data. Is there a report reporting this?
For the "seen group", why is  LLM likely exposed during training to historical BBC news in 2020?
On the other hand, how could you be sure that the models was not exposed to the latest news?

"Hallucination Risk" dataset:
I am not sure it measures what the authors want it to measure. The dataset represents the automatic measures and not the hallucinations. The authors should consider at least reporting for a subset of the dataset about manual-human evaluation for hallucinations.

Problem formulation (Section 3.1) wasn't clear to me. E.g., h have two different meanings 

The results for the " Seen/Unseen" are not clear. What the numbers exactly represents? Why should F1 and accuracy be reported?
What have been train? how? A lot of details are missing.
I'm not convinced that the model the authors built knows how to identify questions it knows or doesn't know the answer to, but maybe it simply recognizes a text from before 2021 versus a text that happened after 2024.

With respect to "Hallucination Risk", in Figure 3 consider adding the automatic measures that the LLMs' used to be train on.

What does the experiment shown in Figure 4 teach us? It seems quite trivial that as the model progresses through the layers it will converge more and more to the correct answer.

The authors offer an error analysis but it is actually anecdotal examples without conclusions
questions: questions: Missing references:

Slobodkin, A., Goldman, O., Caciularu, A., Dagan, I., & Ravfogel, S. (2023, December). The curious case of hallucinatory (un) answerability: Finding truths in the hidden states of over-confident large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 3607-3625).

Gottesman, D., & Geva, M. (2024). Estimating Knowledge in Large Language Models Without Generating a Single Token. arXiv preprint arXiv:2406.12673.

Dataset section under Experiment section is repitative. Consider merging it with the "Data Construction" setction 

Note that sometimes a period is missing at the end of a sentence after using numbers. for example line 364: " [...] accuracy scores reach 80.28% and 80.24% These"
paper_summary: paper_summary: The paper investigates the use of internal LLM states for the prediction of hallucination risks. The motivation is that generative models can assess how much they can reliably generate answers to given queries and in this way, hallucinations can be reduced. The approach uses a MLP classifier to estimate hallucination risk and the method is tested on a large variety of tasks.
strength_weakness: strength_weakness: Some decisions look a bit ad-hoc to me and would require more detailed explanations. In particular, the preliminary study in section 3.3 seems to be crucial for the selection of the internal states that are used as input for the estimator trained below. I do not clearly see how this analysis supports the decision of taking the internal states corresponding to the last tokens in a query as stated in 3.4. Where exactly comes the evidence for this particular selection? I am also not really sure how to read figure 2 and how that would help me to understand the selection of states to be taken for your classifier.

In the data construction, it is good to see that you combine different automatic metrics to label the data. Nevertheless, I would like to see a discussion on the reliability of this kind of annotation and the particular combination heuristics that you apply.

In some tasks, I also wonder how you distinguish between seen and unseen examples. For example in translation, what does it mean to query for unseen translations? The same question arises for other tasks.

Personally, I would also tone down the analogies with human cognition and self-awareness. It is nice to see that self-assessment can be implemented to improve uncertainty estimation in prediction but the comparison to human self-awareness and cognitive processes stretches the ability a bit too far. Instead, I would focus on the technical aspects of model calibration and uncertainty estimation that are practical features for more reliable and trustworthy output generation. No need to draw a picture of conscious and self-aware LLMs.
questions: questions: Do not use contractions like "don't" and "it's".
Clarify the visualizations in Figure 2 and what they show.
Better explain the selection of internal states that you use in your classifier.