Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models (2024)

Abstract: Hallucinations in large language models (LLMs) refer to the phenomenon of LLMs producing responses that are coherent yet factually inaccurate. This issue undermines the effectiveness of LLMs in practical applications, necessitating research into detecting and mitigating hallucinations of LLMs. Previous studies have mainly concentrated on post-processing techniques for hallucination detection, which tend to be computationally intensive and limited in effectiveness due to their separation from the LLM’s inference process. To overcome these limitations, we introduce MIND, an unsupervised training framework that leverages the internal states of LLMs for real-time hallucination detection without requiring manual annotations. Additionally, we present HELM, a new benchmark for evaluating hallucination detection across multiple LLMs, featuring diverse LLM outputs and the internal states of LLMs during their inference process. Our experiments demonstrate that MIND outperforms existing state-of-the-art methods in hallucination detection1.

heading: 1 Introduction
text: In recent years, Large Language Models (LLMs) have demonstrated remarkable performance in a variety of natural language processing (NLP) applications (Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023a; Scao et al., 2022; Zhang et al., 2022). However, the widespread adoption of LLMs has highlighted a critical problem, i.e., hallucination. Hallucination refers to the cases where LLMs generate responses that are logically coherent but factually incorrect or misleading (Maynez et al., 2020; Zhou et al., 2020; Liu et al., 2021;
*swh22@mails.tsinghua.edu.cn †contributed equally ‡Corresponding Author: aiqy@tsinghua.edu.cn 1We have open-sourced all the code, data, and models in
GitHub: https://github.com/oneal2000/MIND/tree/main.
Ji et al., 2023). Such flaw hurts the effectiveness and robustness of LLMs in real-world NLP applications, underlining the pressing necessity for research on detecting and mitigating hallucinations in LLMs.
Existing studies on hallucination detection for LLM mainly focus on how to identify possible factrelated errors in LLM’s outputs (Lin et al., 2021; Li et al., 2023d; Manakul et al., 2023). For example, WikiBio GPT3 (Manakul et al., 2023), a well-known benchmark for hallucination detection, hired human annotators to annotate a couple of true and false response. Hallucination detection methods are then evaluated based on their accuracy in predicting the truthfulness of the answers. Based on this paradigm, considerable hallucination detection methods have been proposed with the goal of taking a piece of text as input and predicting whether there are hallucinations in the inputs (Manakul et al., 2023; Zhang et al., 2023b; Azaria and Mitchell, 2023). Since they detect hallucinations after LLMs finish the inference process, we refer to these hallucination detection methods as postprocessing methods.
Unfortunately, the post-processing methods widely used in existing studies are suboptimal for the applications of hallucination detection models for LLMs in practice. First, existing postprocessing methods often suffer from extreme computation costs and high latency. In order to identify hallucinations in input text without ground truth references (otherwise the task would downgrade to a simple fact verification task), hallucination detection models need to be powerful and knowledgeable on their own. SOTA detection methods are often implemented with LLMs (e.g., chatGPT, LLaMA, OPT) directly (Manakul et al., 2023; Azaria and Mitchell, 2023; Zhang et al., 2023b), which makes the cost of hallucination detection on par or even larger than the inference process of many LLMs. Second, post-processing methods
14379
are intrinsically limited in model capacities. As post-processing methods detect hallucinations independently with the inference process of LLMs, they can’t analyze how hallucinations are generated from scratches in each LLM. Some studies tried to bypass this issue through the construction of proxy models (Azaria and Mitchell, 2023). However, such proxy models must be trained with extensive human annotations, otherwise they cannot capture the unique characteristics of each LLM. Such manual annotation data are expensive to collect and not preferable considering the rapid developments of LLM techniques.
To address these limitations, we propose a novel reference-free, unsupervised training framework MIND, i.e., unsupervised Modeling of INternal states for hallucination Detection of Large Language Models. We highlight MIND with the following advantages: (1) Unsupervised. In contrast to previous works, MIND is an unsupervised framework that directly extracts pseudo-training data from Wikipedia. It doesn’t require any manual annotation for the training of the hallucination detector. (2) Real-time. Compared with the existing post-processing methods, MIND is a realtime hallucination detection framework designed to reduce computational overhead and detection latency. With a simple multi-layer perceptron model built upon the contextual embeddings of each token in LLM’s inference process, MIND can conduct the hallucination detection in a real-time process. (3) Compatibility. MIND is a lightweight framework that can be incorporated into any existing Transformer-based LLMs.
Further, to facilitate future research and to improve the reproducibility of this paper, we introduce a new benchmark for LLM hallucination detection named HELM: Hallucination detection Evaluation for multiple LLMs. In contrast to previous benchmarks that only provide the text generated based on hand-crafted heuristics (Liu et al., 2021; Azaria and Mitchell, 2023) or a single LLM (Manakul et al., 2023; Li et al., 2023d), HELM provides not only the texts produced by six different LLMs (together with human-annotated hallucination labels) but also the contextualized embeddings, self-attentions and hidden-layer activations recorded in the inference process of each LLM, which could serve as the snapshots of each LLM’s internal states during their inference process.
In summary, the contributions of our paper are
as follows: • We propose MIND, an unsupervised training
framework for real-time hallucination detection based on the internal states of LLM.
• We introduce HELM, a hallucination detection benchmark featuring text from six LLMs and contains the internal states of each LLM during the text generation process.
• We evaluate MIND and existing hallucintion detection baselines with human annotations. The experimental results show that MIND outperforms existing hallucination detection methods.

heading: 2 Problem Formulation
text: Hallucination Detection can be defined as a binary classification problem. The objective is to judge whether the given output from an LLM is a hallucination (false or misleading information) or a non-hallucination (accurate and relevant information). This diverges from traditional fact verification tasks. In contrast to traditional fact verification tasks that mainly assess the factual accuracy of text, hallucination detection in LLMs goes beyond simply evaluating the truthfulness of the information. It focuses more on the comprehensive analysis of the characteristics intrinsic to the generative models. Examples of hallucination detection methods include the consistency of multi-responses to the same question (Manakul et al., 2023), the LLMs’ confidence in its generated content (Zhang et al., 2023b), the internal states of LLMs during the text generation process (Azaria and Mitchell, 2023), etc.

heading: 3 Methodology
text: In this section, we introduce the details of our proposed framework MIND, i.e., unsupervised Modeling of INternal-states for hallucination Detection of Large Language Models. The MIND framework consist of two steps: automatic training data generation and hallucination classifier training.

heading: 3.1 Unsupervised Training Data Generation
text: Our proposed Unsupervised Training Data Generation involves automatically annotating hallucinations in content produced by a chosen LLM. Let a specific LLM be Li, then our method aims to create customized training data for training a Hallucination Detection Model for Li. Figure 1 illustrates our proposed automatic data generation
process. The process starts by selecting a subset of high-quality Wikipedia articles, denoted as W (for example, WikiText-103 (Merity et al., 2016)). This subset is represented as W = {w1, w2, ..., wn}, where each wi (1 ≤ i ≤ n) is an individual article. Following that, each article wi is truncated at a randomly selected entity (except those that appear at the beginning of a sentence) occurring after the first sentence of the article. The selected entity is denoted as ei. The truncated article is denoted as w
′ i = truncate(wi, ei). Subsequently, each trun-
cated article w ′ i is inputted into the LLM Li. Li is then tasked with a in free-form text generation based on w
′ i. The continuation text produced by
the LLM for w ′ i is marked as Gi which is truncated at the end of the first sentence. During the generation of Gi, the internal states of the LLM are recorded, denoted as Si. Each Gi is then compared to the original article wi to assess if the LLM can accurately continue writing about the original entity ei. The key criterion for this comparison is whether the beginning of Gi contains the correct entity ei as it appears in the original article. If Gi starts with the correct entity ei, it is labeled as non-hallucination and represented as Hi = 0. Conversely, if Gi does not start with the correct entity ei, and ei does not appear in Gi, it is labeled as hallucination and represented as Hi = 1. For each article, a data tuple is created, represented as Di = (Li, wi, Gi, Si, Hi), encompassing the selected LLM Li, the original Wiki article wi, generated text Gi, the internal states Si, and the hallucination label Hi.

heading: 3.2 Hallucination Classifier Training

heading: 3.2.1 Feature Selection
text: In Transformer-based models, the generation of each token is directly based on its contextualized embedding vectors (also named hidden states), which represents the LLM’s comprehension and
semantic representation of the previously generated tokens. These contextualized embeddings also manifest the behavior of the LLM in making next token predictions, encompassing elements of uncertainty. Thus, contextualized embedding can serve as an important reference for judging LLMs’ Hallucinations. In this section, we investigate a simple research question: Can we detect hallucinations in LLMs using contextualized embeddings?
To verify this, we select the contextualized embeddings of different tokens in various Transformer layers and use a multilayer perceptron (MLP) to classify the embedding during hallucinations from those during non-hallucinations. If it is possible to classify these vectors using a simple MLP, it indicates distinctions between the token embeddings of LLMs during hallucination and non-hallucination states. We train this MLP on 5k samples generated by LLaMA2-13B-Chat via the automatic data generation process described in the previous section, and test the prediction accuracy on 5k samples. The classification accuracy based on the contextualized embedding vectors at various positions of LLM and their combinations are as follows:
Layer Embedding Formula Acc
All All 1 N ∑N j=1 1 n ∑n i=1 H i j 0.7054 First & Last All 1 2 ( ∑n i=1 H i 1 + ∑n i=1 H i N ) 0.6929 Last All 1 n ∑n i=1 H i N 0.6986 First All 1 n ∑n i=1 H i 1 0.6529 Last Last HnN 0.7123 All Last 1
N
∑N j=1 H n j 0.6986
Last All & Last 1 2 ( 1 n ∑n i=1 H i N + H n N ) 0.7191
In the table above, N represents the total number of Transformer layers in this LLM, and n is the length of the input token sequence, "H ij" represents the contextualized embedding vector of the ith token in the jth Transformer layer of the LLM. The experimental results demonstrate that we can indeed classify these vectors using a simple MLP. This finding
indicates that the contextualized embedding vectors of LLMs exhibit discernible differences during hallucination and non-hallucination states. Furthermore, it is noteworthy that an effective distinction can be achieved by merely utilizing the last token’s contextualized embedding of the final layer. In Section 6, we extensively explored the generality and robustness of this method in validating existing open-source large models, and conducted more detailed experiments.

heading: 3.2.2 Training Process
text: The MIND classifier is architecturally structured as a Multilayer Perceptron (MLP) network. For the activation functions of the MIND classifier, we select the Rectified Linear Unit (ReLU). For the training of the classifier, the input is a 1×n matrix, where n represents the dimension of the LLM’s contextualized embedding. We choose the contextualized embeding of the last token of last Transformer layer as the input of the MIND classifier. The output of the hallucination classifier is a binary label, indicating whether the LLM is experiencing a hallucination while generating a specific segment of text. The process can be mathematically represented as follows:
P = MLP (ReLU(W ·H+ b)), (1)
where P is the hallucination label predicted by the classifier, H is a 1 × n matrix representing the selected hidden states, W and b are the weight matrix and bias vector of the MLP respectively, and MLP represents the function implemented by the multilayer perceptron. For the loss function, we use the Binary Cross-Entropy (BCE) Loss (De Boer et al., 2005) to optimize the hallucination classifier, which is defined as:
LBCE(yi, pi) = yi log(pi) + (1− yi) log(1− pi), (2)
where yi represents the actual label of the ith example, and pi indicates the predicted probability that the ith example belongs to the positive class.

heading: 3.3 Real-time Hallucination Detection
text: Upon successfully training the Hallucination Classifier as outlined in Section 3.2, the MIND framework enable real-time hallucination detection in the chosen LLM. Specifically, the Hallucination Classifier receives the contextualized embeddings of each token during the LLM’s inference process. It then outputs a probability indicating the probability of hallucination in the LLM’s output.

heading: 4 The HELM Benchmark
text: In this section, we introduce the details of our proposed new benchmark HELM: Hallucination detection Evaluation for multiple LLMs.

heading: 4.1 Data Generation
text: We select six widely-used, open-source LLMs for annotation. These models range in complexity, encompassing both base models and chat models, with sizes varying from 6 billion to 40 billion parameters, including Falcon-40B (Almazrouei et al., 2023), GPT-J-6B (Wang and Komatsuzaki, 2021), LLaMA-2-Base-7B (Touvron et al., 2023b), LLaMA-2-Chat-7B (Touvron et al., 2023b), LLaMA-2-Chat-13B (Touvron et al., 2023b), and OPT-6.7B (Zhang et al., 2022). For the generation process, we start with randomly sampling 50,000 articles from a high-quality Wikipedia corpus: WikiText-103 (Merity et al., 2016)2. Following that, the selected LLMs were tasked with prompt-based continuation writing. For base LLMs and chat LLMs, we have designed different prompt templates respectively which is detailed in Appendix B.

heading: 4.2 Human Annotation
text: Annotators are tasked with evaluating the truthfulness of the content generated by the LLM. We only take factual errors into account, without considering grammatical mistakes or subjective opinions3. Furthermore, annotators are instructed not to use generative models (such as ChatGPT) for assistance. Wikipedia is the primary source for fact-checking, followed by Google’s search engine when Wikipedia lacks relevant information.
The annotation process begins with annotators manually dividing the passage into discrete
2The selected articles have no overlap with the training set of MIND.
3For example, stating that "Ronald Reagan was a President of the United States" is a fact, whereas commenting on "Ronald Reagan’s attractiveness" is a matter of opinion.
sentences. Annotators are then required to use Wikipedia and Google search to verify the truthfulness of every sentence. If neither Wikipedia nor the top 20 search engine results confirm the accuracy of a sentence, it is marked as ‘Unverifiable.’ Additionally, in instances of hallucinations, annotators are required to indicate the location of the hallucination within the sentence. Every sentence and passage in the dataset was annotated by two distinct annotators. Only the data with matching annotations from both annotators was adopted.

heading: 4.3 Benchmark Analysis and Usage
text: The statistics of our proposed HELM dataset are shown in Table 1. Our dataset ultimately comprises a total of 3342 sentences, each of which corresponds to a label indicating whether it is a hallucination. Additionally, it includes the complete contextualized embeddings and hidden layer activations of the LLM during the text generation process. These 3342 sentences are derived from 1224 distinct passages. HELM provides two levels of hallucination detection: Sentence Level and Passage Level. Since every annotated sentence originates from a passage, HELM dataset also encompasses a task for Passage Level Hallucination Detection. The hallucination at the passage level depends on its sentences. If any sentence within a passage is identified as hallucinated, the entire passage is classified as hallucinated.
To use our benchmark, users can either directly use the code provided in our open-source GitHub 4 repository or download our data and implement it themselves.
4We have publicly shared the HELM dataset and code in this anonymous GitHub repository: https://github.com/oneal2000/MIND/tree/main

heading: 5 Experimental Settings

heading: 5.1 Dataset and Metrics
text: We evaluate MIND and other baselines on our proposed dataset HELM which is detailed in section 4. We use the AUC (Area Under the Curve) and the Pearson correlation coefficient (corr) with humanannotated relevance as evaluation metrics. We conducted experiments at both the sentence-level and the passage-level hallucination detection.

heading: 5.2 Baselines
text: We choose the following reference-free hallucination detection methods as baselines:
• Predictive Probability (PP) (Manakul et al., 2023). A method for detecting LLMs’ hallucinations based on the probability of tokens generated by LLMs. PPmax, PPmin, and LN-PP (Length Normalised PP) indicates max pooling, min pooling and mean pooling methods to combine multiple tokens generated by LLM respectively.
• Predictive Entropy (PE) (Kadavath et al., 2022) PE is widely used to evaluate the uncertainty inherent in the model’s output distribution (Kadavath et al., 2022). For each token ti in the LLM output, the PE is defined as:
PE = − ∑
w̃∈W pi(w̃) log pi(w̃), (3)
where pi(w̃) represents the likelihood of generating word w̃, and W is the vocabulary of the LLM. PEmax, PEmin, LNPE (Malinin and Gales, 2020) (Length Normalised PE) indicates max pooling, min pooling and mean pooling strategies to combine multiple tokens generated by LLM respectively.
• SelfCheckGPT (Manakul et al., 2023) (SCG) is a strong hallucination detection baseline that is designed based on the principle that if an LLM has knowledge of a given concept, sampled responses are likely to be similar and contain consistent facts. SCG employs four distinct techniques: SCG_BERTScore, SCG_QA, SCG_NLI, and SCG_n-gram, each of which evaluates the consistency of responses from the LLM.
• SAPLMA (Azaria and Mitchell, 2023) is a novel hallucination detection training method, wherein a classifier is trained using the activation values derived from the hidden layers of LLMs.
• EUBHD (Zhang et al., 2023b). Enhanced Uncertainty-Based Hallucination Detection is a SOTA hallucination detection method based on the uncertainty of LLMs’ outputs.
• GPT4-HDM (Li et al., 2023d). Through a simple prompt template, this method directly use GPT-4 as hallucination detection model to judge whether the text generated by other LLMs contains hallucination. The prompt templates is detailed in Appendix C.

heading: 5.3 Implementation Details
text: • NER: For the Named Entity Recognition (NER)
component of MIND, we follow the methodologies in prior studies (Liu et al., 2021; Tarcar et al., 2019). Specifically, we utilized the Spacy library, a tool recognized for its effectiveness and efficiency in NER as evidenced by previous research (Shelar et al., 2020).
• MIND: The MIND classifiers employs a 4-layer Multilayer Perceptron (MLP) network, featuring a 20% dropout rate applied at the initial layer. The architecture of this network is characterized by a progressively decreasing hidden layer size,
with dimensions set at 256, 128, 64, and 2 for each consecutive layer. In terms of activation functions, the Rectified Linear Unit (ReLU) is selected. The learning rate is set to 5e-4, the weight decay is set to 1e-5, and the training batch size is set to 32. For the ablation experiments of important hyperparameters, we have discussed in detail in Section 6.3.
• LLM Configuration: For the selected LLMs, we directly download model parameters from the official Hugging Face repositories for each model, and use the code provided by Hugging Face to conduct text generation. For the generation configuration, we use the official default configurations provided by each model. We introduce our selected models Appendix D.

heading: 6 Experimental Results
text: We conduct experiments to verify our our proposed Hallucination Detection method: MIND. Specifically, this section studies the following research questions (RQ):
• RQ1: Is hallucination detection based on the internal states of Large Language Models (LLMs)
broadly effective across existing LLMs?
• RQ2: How effective and efficient is MIND on hallucination detection tasks?
• RQ3: For the training of hallucination detection models, is it necessary to use training data generated by the model itself?

heading: 6.1 Overall Results of MIND and Baselines
text: In this subsection, we present a comprehensive evaluation of our MIND framework against baselines. The objective is to answer Research Questions RQ1 and RQ2. Our experimental results are shown in Table 2. The key findings are summarized as follows: (1) MIND, our proposed unsupervised internal states-based hallucination detection method, demonstrates broad effectiveness in all six LLMs. This result validates the hypothesis that hallucination detection based on the contextualized embeddings of LLMs is effective across various models. (2) MIND outperforms existing reference-free hallucination detection methods at both sentence and passage levels. This superiority not only highlights the robustness of MIND, but also underscores the effectiveness of the unsupervised training framework proposed in this study. (3) SCG-NLI emerged as the second-best hallucination detection method after MIND. Remarkably, it even surpasses MIND in performance on certain models. However, a critical drawback of SelfCheckGPT is its latency. Specifically, the time taken to detect hallucination in a response is ten times longer than generating the response itself. This makes MIND a more viable option for scenarios requiring real-time hallucination detection. (4) SAPLMA, though effective on its own training dataset, exhibits suboptimal performance when applied to the real-time hallucination detection during the text generation process. This could be attributed to the model’s overfitting to the specific training data, which is not generated by the LLM itself.

heading: 6.2 Efficiency
text: This section compares the efficiency of the MIND classifier with other hallucination detection methods, focusing on LLaMA-7B-Chat. We examine the average generation time for each question, and the average time of these hallucination detection methods take to detect hallucinations which are demonstrated in Table 3. MIND is shown to be highly efficient, taking only 3% of the LLM’s response generation time for hallucination detection.
This makes MIND ideal for real-time applications, unlike SelfCheckGPT, which is less efficient due to requiring multiple LLM responses. Since the LLM records the probability of generating each token during response generation, methods like Predictive Probability (PP) and Predictive Entropy (PE) are easy to compute and require negligible time, making them the least time-consuming. Nonetheless, when considering both efficiency and effectiveness, MIND emerges as the superior referencefree hallucination detection method.

heading: 6.3 Ablation Studies

heading: 6.3.1 Impact of Customized Training Data
text: This section focuses on the importance of incorporating model-specific training data to answer RQ3. To be specific, we validate the impact of customized training data by comparing the performance of hallucination classifiers in the following two settings. In the first setting, training data is customized for the hallucination classifier of each LLM. In this setting, the input to the classifier is the internal state of the LLM during its text generation process. In the second setting, we use
existing training data that was generated by another LLM. Then, this pre-existing data is input into the target LLM. In this setting, the input to the classifier is the internal state of the LLM when encoding the pre-existing texts generated by other LLMs. Table 4 presents the performance of hallucination detection models for OPT-7B and LLaMA-7B-Chat.
The experimental results indicate a clear trend: customized training data significantly enhances the performance of hallucination classifiers for both OPT-7B and LLaMA2-Chat-7B models. This observation supports the hypothesis that training data tailored to the specific language model can improve its ability to identify hallucinations in generated text. This implies that for a new model, it would be advantageous to customize the hallucination detection model using the MIND method tailored to that specific model.

heading: 6.3.2 Impact of the Training Data Size
text: The training data for our proposed MIND model is generated automatically, making it crucial to determine the optimal amount of data to produce. We conduct a systematic investigation into the impact of training dataset size on model performance which is detailed in Table 5. The experimental result shows that increasing the training dataset size from 1,024 to 4,096 labels improves accuracy, suggesting that larger datasets enable the model to learn more complex patterns. However, the benefit plateaus and even slightly decreases after exceeding 4,096 data points, indicating a threshold beyond which additional data no longer improves performance significantly.

heading: 6.3.3 Classifier Layer Depth
text: In this section, we explore the impact of varying the depth of the classifier layer on the performance of the MIND model. The depth, measured in terms
of the number of layers in the classifier, ranged from 2 to 6 layers. The accuracy of the MIND classifier on the development set was used as the metric for assessing performance. The results of this experiment are shown in Table 6. Initially, as the number of layers increased from 2 to 4, there was a slight improvement in accuracy. Specifically, the model’s accuracy went from 0.7157 with two layers to a peak of 0.7291 with four layers. However, the small difference between different numbers of layers indicates that the MIND classifier is not sensitive to this hyperparameter of the number of layers.

heading: 7 Related Work

heading: 7.1 Hallucination Detection
text: To tackle the issue of hallucinations in LLMs, researchers have devised various methods for hallucination detection. SelfCheckGPT (Manakul et al., 2023) (SCG) is designed based on the principle that if an LLM has knowledge of a given concept, sampled responses are likely to be similar and contain consistent facts. HaluEval (Li et al., 2023d) represents a direct approach, where strong LLMs like GPT4 are directly used to evaluate the output of other LLMs. SAPLMA (Azaria and Mitchell, 2023) introduces human annotations to label hallucinations ChatGPT outputs, then training a classifier that detects hallucinations in LLMs by analyzing their internal states. EU-HD (Zhang et al., 2023b). Enhanced Uncertainty-Based Hallucination Detection is a SOTA hallucination detection method based on the predictive probability and LLM’s attention for each generated tokens.

heading: 7.2 Evaluation of Hallucination Detection
text: The purpose of the Hallucination Detection Evaluation (HDE) is to evaluate the effectiveness of various Hallucination Detection Methods (HDMs). The SelfCheckGPT dataset (Manakul et al., 2023) utilizes GPT-3 to generate passages about individuals from the WikiBio dataset, with manual annotations assessing the factuality of these passages, classifying them into major inaccurate, minor inaccurate, and accurate categories. The True-False dataset (Azaria and Mitchell, 2023) represents another innovative approach, constructed based on a database of instances with multiple factual attributes. HaluEval (Li et al., 2023d) takes a different approach by prompting GPT models to generate hallucinatory texts, using prompts like, “I want you
to act as a hallucination answer generator,” coupled with human annotation. HADES (Liu et al., 2021) employs a rule-based method to modify tokens in Wikipedia articles to generate hallucination texts.

heading: 8 Conclusions and Future Works
text: In this paper, we introduce MIND, a novel unsupervised approach leveraging the internal states of Large Language Models (LLMs) for real-time hallucination detection. Moreover, we propose HELM, a comprehensive benchmark for hallucination detection, incorporating outputs from six diverse LLMs along with their internal states during text generation.

heading: 9 Limitations
text: We acknowledge the limitations of this paper, particularly in the aspect of detecting hallucinations using only the internal states of LLMs. While effective, this method has the potential for enhanced accuracy. To address this, our future work will focus on integrating the internal states of LLMs with their generated text. This combined approach aims to improve the precision and reliability in identifying and mitigating hallucinations in LLM outputs, leading to more robust and accurate hallucination detection methodologies.

heading: 10 Ethics Statement
text: In conducting this research, we have prioritized ethical considerations at every stage to ensure the responsible development and application of AI technologies. In the development of MIND, our approach has been to create a reference-free, unsupervised training framework that primarily utilizes publicly available data sources, such as Wikipedia. This methodological choice ensures that our research does not rely on personally identifiable information. We firmly believe in the principles of open research and the scientific value of reproducibility. To this end, we have made all models, data, and code associated with our paper publicly available on GitHub. This transparency not only facilitates the verification of our findings by the community but also encourages the application of our methods in other contexts.

heading: A Pseudocode Description of MIND
text: The pseudocode description of our proposed unsupervised training data generation process is shown in Algorithm 1.

heading: B Prompt Template of the HELM Dataset
text: For the data generation process of our proposed HELM dataset, the selected LLMs were tasked with free-form generation. Specifically, the task involved prompt-based continuation writing. For base LLMs and chat LLMs5, we have designed different prompt templates respectively.
For the base LLM, The prompt template is as follows:
Prompt 1
This is a Wikipedia passage about [title]. [First sentence of that article].
For the chat LLM, The prompt template is as follows:
Prompt 2
The following sentence is the first sentence of a Wikipedia article titled [Title]. Please continue writing the sentence below. [First sentence of that article]

heading: C Prompt Template of GPT4
text: Following the settings of Li et al. (Li et al., 2023d), we directly use GPT-4 as hallucination detection model to judge whether the text generated by other LLMs contains hallucination through the following prompt template:
Prompt 3
Given the following text span, your objective is to determine if the provided text contains non-factual or hallucinated information. You SHOULD give your judgment based on the world knowledge. Text span: [Provided Text] Now, determine if the above text span contains non-factual or hallucinated information. The answer you give MUST be “Yes” or “No”.

heading: D Details of Our Selected LLMs
text: To validate the effectiveness of the approach utilizing the internal states of Large Language Models
5Base LLMs are language model that has been pre-trained on a large corpus. On the other hand, the Chat LLM, besides pre-training, also undergoes additional processes such as instruction tuning to better align with conversational tasks.
Algorithm 1 Unsupervised Training Data Generation for Hallucination Detection 1: Input: Language Model Li, Wikipedia Articles W = {w1, w2, ..., wn} 2: Output: Data Tuples D = {D1, D2, ..., Dn} 3: for each article wi ∈W do 4: Select random entity ei not at the beginning of any sentence 5: w
′ i ← truncate(wi, ei) ▷ Truncate wi at ei
6: Gi ← Li(w′i) ▷ Generate continuation with Li 7: Truncate Gi at the end of its first sentence 8: Record internal states Si during generation 9: if the beginning of Gi contains ei correctly then 10: Hi ← 0 ▷ Non-hallucination 11: else 12: Hi ← 1 ▷ Hallucination 13: end if 14: Di ← (Li, wi, Gi, Si, Hi) ▷ Form data tuple 15: end for
(LLMs) for hallucination detection across various existing LLMs, we conducted experiments with as many open-source LLMs as possible. Specifically, this included the following 14 LLMs:
• GPT-J-6B (Wang and Komatsuzaki, 2021) is a 6 billion parameter, autoregressive text generation model trained on The Pile corpus (Gao et al., 2020).
• OPT-6.7B (Zhang et al., 2022). OPT is a collection of decoder-only pre-trained Transformers, with models ranging from 125 million to 175 billion parameters. We choose OPT-6.7B from the OPT series.
• LLaMA-2 (Touvron et al., 2023b) is a collection of pre-trained and fine-tuned LLMs ranging in scale from 7 billion to 70 billion parameters. This series includes fine-tuned LLMs, known as Llama 2-Chat, specifically designed for optimal performance in dialogue-based applications. We choose LLaMA-2-Chat-7B, LLaMA-2-Base7B, LLaMA-2-Base-13B, and LLaMA-2-Chat13.
• Falcon (Almazrouei et al., 2023) comprises a set of causal decoder-only models that have been trained on a dataset of 1,000 billion tokens, which includes data from RefinedWeb (Penedo et al., 2023). Among the models in Falcon, we have chosen Falcon-7B and Falcon-40B.
E Implementation Details
• NER: For the Named Entity Recognition (NER) component of MIND, we follow the methodolo-
gies in prior studies (Liu et al., 2021; Tarcar et al., 2019). Specifically, we utilized the Spacy library, a tool recognized for its effectiveness and efficiency in NER as evidenced by previous research (Shelar et al., 2020).
• MIND: The MIND classifiers employs a 4-layer Multilayer Perceptron (MLP) network, featuring a 20% dropout rate applied at the initial layer. The architecture of this network is characterized by a progressively decreasing hidden layer size, with dimensions set at 256, 128, 64, and 2 for each consecutive layer. In terms of activation functions, the Rectified Linear Unit (ReLU) was selected. The learning rate is set to 5e-4, the weight decay is set to 1e-5, and the training batch size is set to 32. For the ablation experiments of important hyperparameters, we have discussed in detail in Section 6.3.
• SCG: For the implementation of SelfCheckGPT (SCG), we directly use the code and follow all the hyperparameters from their official GitHub6.
• LLM Configuration: For the selected LLMs, we directly download model parameters from the official Hugging Face repositories for each model, and use the code provided by Hugging Face to conduct text generation. For the generation configuration, we use the official default configurations provided by each model.
6https://github.com/potsawee/selfcheckgpt/tree/main

heading: F Discussion on Mitigating Hallucination
text: This paper focuses on hallucination detection and the evaluation of hallucination detection methods. However, in practical applications, it is indeed feasible to use our proposed MIND Framework to mitigate hallucinations. Therefore, we briefly discuss two potential methods to mitigate hallucinations based on the MIND Framework, offering some guidance for those in need.
Firstly, the MIND Classifier can be employed as a re-ranker. The MIND Classifier outputs a score indicating the likelihood of a hallucination occurring in LLMs’ response. Thus, by utilizing a sampling decoding method, an LLM can generate multiple outputs for the same query. Subsequently, the MIND Classifier can assess these outputs for hallucinations, allowing the selection of the output with the lowest probability of hallucination as the final response.
Additionally, we can employ the dynamic retrieval augmented generation (RAG) framework (Khandelwal et al., 2019; Borgeaud et al., 2022; Lewis et al., 2020; Guu et al., 2020; Izacard and Grave, 2020; Jiang et al., 2022; Li et al., 2023a; Shi et al., 2023; Su et al., 2023b; Chen et al., 2023, 2022; Li et al., 2023c; Su et al., 2024; Fang et al., 2024; Zhang et al., 2023a) to mitigate hallucinations. The dynamic RAG method triggers the retrieval module (Li et al., 2023b; Su et al., 2023a; Ma et al., 2023; Ye et al., 2024; Su et al., 2023c) during the inference process of an LLM, facilitating the retrieval of external knowledge. The MIND Framework can provide guidance on when to trigger the RAG. Specifically, when MIND indicates a high probability of hallucination, the retrieval module can be triggered and then add the retrieved passages into the context of the LLM, allowing the LLM to continue generating based on the retrieved external knowledge.

heading: Future Work
text: To address this, our future work will focus on integrating the internal states of LLMs with their generated text. This combined approach aims to improve the precision and reliability in identifying and mitigating hallucinations in LLM outputs, leading to more robust and accurate hallucination detection methodologies