The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models (2024)

Abstract: In the era of large language models (LLMs), hallucination (i.e., the tendency to generate factually incorrect content) poses great challenges to trustworthy and reliable deployment of LLMs in real-world applications. To tackle the hallucination, three key questions should be well studied: how to detect hallucinations (detection), why do LLMs hallucinate (source), and what can be done to mitigate them (mitigation). To address these challenges, this work presents a systematic empirical study on LLM hallucinations, focused on the three aspects of hallucination detection, source and mitigation. Specially, we construct a new hallucination benchmark HaluEval 2.0, and design a simple yet effective detection method for LLM hallucinations. Furthermore, we zoom into the different training or utilization stages of LLMs and extensively analyze the potential factors that lead to the LLM hallucinations. Finally, we implement and examine a series of widely used techniques to mitigate the hallucinations in LLMs. Our work has led to several important findings to understand the hallucination origin and mitigate the hallucinations in LLMs. Our code and data can be accessed at https://github.com/ RUCAIBox/HaluEval-2.0.

heading: 1 Introduction
text: Large language models (LLMs) (Zhao et al., 2023) have shown remarkable potential in a wide range of natural language processing applications (Brown et al., 2020; Ouyang et al., 2022; OpenAI, 2023). However, despite the significant improvement in model capacity, a persistent challenge lies in their tendency to hallucinate, i.e., generate content that looks plausible but is factually incorrect (Huang et al., 2023; Ji et al., 2023a; Zhang et al., 2023b). This issue severely restricts the deployment of LLMs in real-world applications (e.g., clinical diag-
∗Corresponding author
noses), where the reliable generation of trustworthy text is of utmost importance.
In the era of LLMs, there has been a significant surge of research interests in hallucinations (Yao et al., 2023; Das et al., 2023; Dhuliawala et al., 2023a; Varshney et al., 2023; Manakul et al., 2023). These studies are mainly centered around three interleaved questions, i.e., why do LLMs hallucinate, how to detect hallucinations, and what can be done to mitigate them? The three key questions pose great challenges to the research community, while existing empirical work mostly focuses on analyzing or addressing individual challenges, still lacking a systematic and in-depth experimental study on LLM hallucinations. To bridge this gap, a more comprehensive analysis is needed to thoroughly research the aforementioned three questions.
For deciphering the mystery of hallucination in LLMs, we aim to conduct a comprehensive and systematic empirical study on hallucination detection, source, and mitigation. In particular, we mainly focus on studying factuality hallucination, which has become one of the primary erroneous sources for LLMs (Huang et al., 2023; Zhang et al., 2023b). To carry out our research, we zoom into the different stages to train and use LLMs, including pretraining, supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and inference, and thus can conduct more in-depth analysis of potential impact of each stage on model hallucination. This analysis approach is quite different from prior work, where they mostly study the impact of individual stages or strategies to attribute or mitigate the hallucinations.
To conduct our empirical study, we first extend previous work (Li et al., 2023a) and construct a new benchmark HaluEval 2.0 for evaluating the factuality hallucination in LLMs. Our benchmark contains 8,770 questions from five domains including biomedicine, finance, science, education, and open domain. To detect factual errors in LLM responses,
10879
we propose a simple yet effective framework that decomposes the task of hallucination detection into two simple sub-tasks, i.e., extract factual statements from responses and judge the trustfulness of each statement. Then, based on the detection approach, we perform a series of studies to explore the hallucination sources from four aspects, i.e., pre-training, SFT, and inference methods. For each aspect, we extensively examine the effect of possible factors on LLM hallucinations. Finally, we delve into hallucination mitigation for LLMs through a series of widely used techniques, including RLHF, retrieval augmentation, and advanced decoding, etc.
It is generally considered to be challenging to deeply understand and fully mitigate LLM hallucinations. By providing more comprehensive empirical analysis, our work aims to further push forward the research on hallucinations. The major findings of our empirical study are summarized below:
• Pre-training: Pre-training on more tokens has an oscillatory effect on reducing LLM hallucinations. The familiarity of pre-training knowledge to LLMs significantly influences the source of hallucinations, i.e., lower familiarity, more hallucinations.
• Fine-tuning: Supervised fine-tuning LLMs with improved and appropriate instructions can be useful to alleviate hallucinations. RLHF is an effective method to mitigate LLM halluciantions, while this effect relies on specific domains.
• Inference: Diversity-oriented decoding contributes to more hallucinations, which foregrounds the importance of balancing diversity and factuality during decoding. Augmenting generation with retrieval can effectively mitigate hallucinations.

heading: 2 Hallucination in LLMs
text: In the field of NLP, hallucination typically refers that the model output contains undesired content that is nonsensical or deviates from the source material (Ji et al., 2023a; Zhang et al., 2023b).
Despite that hallucination can be defined in different ways, we focus on factuality hallucination, since it has become one of the primary sources of erroneous responses by LLMs. In light of this research, we propose a fine-grained categorization of factuality hallucination in LLMs as follows:
• Entity-error Hallucination. This type of hallucination refers to the situations where the generated text of LLMs contains erroneous entities, such as person, date, location, and object, that contradict with the world knowledge.
• Relation-error Hallucination. This type of hallucination refers to the generated text of LLMs contains wrong relations between entities such as quantitative and chronological relation.
• Incompleteness Hallucination. LLMs might exhibit incomplete output when generating lengthy or listed responses. This hallucination arises when LLMs are asked about aggregated facts and they fail to reserve the factual completeness.
• Outdatedness Hallucination. This type of hallucination refers to situations where the generated content of LLMs is outdated for the present moment, but was correct at some point in the past. This issue arises primarily due to the fact that most LLMs were trained on time-limited corpora.
• Overclaim Hallucination. This type of hallucination means that the statement expressed in the generated text of LLMs is beyond the scale of factual knowledge (Schlichtkrull et al., 2023).
• Unverifiability Hallucination. In some cases, the information generated by LLMs cannot be verified by available information sources.
Note that it is difficult to encompass all kinds of hallucination, and our taxonomy aims to depict the most frequently occurring types of hallucination. We present several illustrative examples for each type of hallucinations in Table 8 in Appendix A.

heading: 3 Experimental Setup
text: In this section, we first introduce our benchmark and then describe a set of models for comparison.

heading: 3.1 Benchmark Construction
text: To comprehensively evaluate the tendency of LLMs to generate hallucinations across various domains, we extend the previous study of HaluEval (Li et al., 2023a) and meticulously construct an upgraded hallucination evaluation benchmark HaluEval 2.0. Our benchmark collects questions for five domains (i.e., biomedicine, finance, science, education, and open domain) from six domain datasets, including BioASQ (Krithara et al., 2023), NFCorpus (Boteva et al., 2016), FiQA-2018 (Maia et al., 2018), SciFact (Wadden et al., 2020), LearningQ (Chen et al., 2018), and HotpotQA (Yang et al., 2018).
To attain a sufficient number of high-quality questions, we use the training set of BioASQ and FiQA-2018, the whole set of NFCorpus and SciFact, and the test set of LearningQ (TED-Ed) and HotpotQA. Then, we only keep input questions ended with the question mark, and for SciFact we
transform the input statements into questions using manually designed templates. Finally, following HaluEval (Li et al., 2023a) to select questions that LLMs are most likely to hallucinate, we employ ChatGPT to generate three responses for each question and compute their average semantic similarity via BERTScore (Zhang et al., 2020). We only retain those questions where the similarity score is lower than a threshold. In our benchmark, questions in biomedicine and science are expert-written, specialized, and require at least high-school knowledge to answer, while questions in other domains are open-ended, mainly collected from daily conversation and open websites. In open domain, questions from HotpotQA are based on the factual knowledge from Wikipedia. In total, our benchmark includes 8770 questions, with 1535, 1125, 1409, 1701, and 3000 questions for biomedicine, finance, science, education, and open domain, respectively.

heading: 3.2 Evaluation Models
text: We conduct the experiments with a number of representative open-source and closed-source LLMs based on HaluEval 2.0.
• Open-source models. We focus on instructiontuned models, which use instructions (e.g., daily chat, synthetic instructions) for tuning. In our experiments, we select six representative instructiontuned models including Alpaca (7B) (Taori et al., 2023), Vicuna (7B/13B) (Chiang et al., 2023), and Llama 2-Chat (7B/13B) (Touvron et al., 2023a).
• Closed-source models. In contrast, closedsource models can be only accessed via APIs. Here, we select five exceptional closed-source models including text-davinci-002/003 (Ouyang et al., 2022), ChatGPT, Claude, and Claude 2.

heading: 4 Hallucination Detection
text: To analyze and mitigate hallucinations, the first and fundamental step is to detect hallucinations. In this section, we design an automatic hallucination detection approach and further validate its reliability by comparing with human labeling.

heading: 4.1 Detection Approach
text: We propose a simple yet effective framework to detect factual errors in model responses. Following prior work (Chern et al., 2023b; Dhuliawala et al., 2023b), we decompose the challenging hallucination detection task into two simple sub-tasks: 1) extract multiple factual statements from a lengthy
response; and 2) determine whether each statement contains hallucinations.
Extraction-then-Verification. First, we instruct a LLM (i.e., GPT-4) to extract factual statements that could be proven to be true or false based on the world knowledge. Instead of training a specific model for fact extraction (Thorne et al., 2018), this approach can significantly reduce the costs of data annotation and model training. The LLM might output “NO FACTS” if there is no factual statement in the response. Second, we use the LLM itself to judge the trustfulness of statements since the LLM has encoded rich world knowledge. Previous work checks each involved fact independently using separate prompts based on verification questions or external tools (Chern et al., 2023b; Dhuliawala et al., 2023b). However, we observe that these statements are often interrelated, with certain statements providing the background or serving as conditions for others. Thus, independently assessing each statement may lead to misjudgment of hallucination. In this work, we instruct the LLM with all statements to predict their hallucination judgements (i.e., True, False, or Unknown). To give a confident judgement, we only consider the false statement as hallucination in our following experiments. We present our used instructions in Appendix B.
Test of Reliability. To examine the reliability of our approach in hallucination detection, we invite human labelers to annotate the factuality of a subset from HaluEval 2.0 and compare the judgement of LLM and humans. Specially, based on the semantic similarity score (Section 3.1), we select 1,000 questions from HaluEval 2.0 with an average of 200 samples for each domain. For each question, we use ChatGPT to generate a response and ask human labelers to annotate the hallucination type of each statement (Section 2). To ensure the correctness of our annotation, each sample is labeled by two labelers and examined by one checker. Finally, the matching rate between the judgement of our proposed method and human annotation is 92.6%, 94.7%, 92.7%, 91.5%, and 93.9% for biomedicine, finance, science, education, and open domain, respectively. The high consistency demonstrates that our proposed method has a high level of reliability in detecting the hallucinations from LLMs.
Evaluation Metrics. In order to quantitatively evaluate LLMs on HaluEval 2.0, we design two metrics from different levels to measure the degree of
LLMs generating hallucinations in their responses. The micro hallucination rate (MiHR) measures the proportion of hallucinatory statements within each response, which is calculated as:
MiHR = 1
n
n∑
i=1
Count(hallucinatory facts) Count(all facts in ri) ,
where n is the total number of samples in every domain and ri is the i-th response. Besides, the macro hallucination rate (MaHR) calculates the proportion of responses containing hallucinatory statements, which is computed as:
MaHR = Count(hallucinatory responses)
n .
For both metrics, smaller value indicates better performance.

heading: 4.2 Results and Analysis
text: We evaluate LLMs on HaluEval 2.0 and apply our detection approach to measure their tendency to produce hallucinations in Table 1.
First, we can clearly observe that there exists a significant performance gap between open-source and closed-source models. For open-source models, we can discover that scaling the model size can effectively decrease the tendency to generate hallucinations. Besides, we see that MaHR and MiHR are not strongly positively correlated (e.g., Alpaca 7B vs Llama 2-Chat 7B), which might be due to the fact that some models tend to generate shorter responses with fewer facts. Furthermore, comparing the results across five domains, it shows that the tendency of LLMs to generate hallucinations is related to specific domains, i.e., higher results in education and open domain. For open domain, we select the most difficult questions from HotpotQA
where ChatGPT is likely to hallucinate. These findings suggest that the training methods of current LLMs in open domain are insufficient in preventing from generating hallucinations. Note that the percentage results of generating hallucinations in Table 1 might significantly exceed the actual rate in overall use, because our dataset is specially curated for hallucination evaluation.
Note: The following experiments are based on the 1,000 samples selected in human annotation.

heading: 5 Hallucination Source
text: In this section, we perform a series of experiments to explore four factors that may induce LLM hallucinations, including pre-training, SFT, inference, and prompt design. We present some experiments of hallucination source in Appendix C.

heading: 5.1 Pre-training
text: Pre-training serves as the fundamental stage to establish the abilities of LLMs, enabling them to gain general capabilities and rich world knowledge. In this part, we consider two key factors, namely scale and source of pre-training data, since they are relatively easy to be examined according to public disclosure of LLMs and often have a large impact on model performance. Specially, we explore the effects of pre-training in three aspects related to the scale and source of pre-training data.
Amount of Pre-training Tokens. We select 11 intermediate pre-training checkpoints of Baichuan 2 (7B) (Yang et al., 2023), corresponding to training on approximately 0.2 to 2.4 trillion tokens, and evaluate them on HaluEval 2.0. As shown in Figure 1 (a), with training on an increasing amount of tokens, the hallucination rate oscillates across these
model checkpoints, suffering more in the domains of finance and education while overall decreasing in other three domains. This finding indicates that simply increasing the amount of pre-training tokens may not be that effective in hallucination reduction, which may require specific data strategies to alleviate the hallucinations in some specific domains.
Mixture of Pre-training Corpus. Existing LLMs typically employ a mixture of diverse public textual datasets as the pre-training corpus. The distribution of data source will affect the acquisition of general and domain-specific knowledge. To examine the effect of pre-training data mixture on hallucinations, we evaluate three LLMs with similar model sizes but using different pre-training corpus of general data (e.g., webpages) and specialized data (e.g., scientific text): Falcon (40B) (Almazrouei et al., 2023), Galactica (30B) (Taylor et al., 2022), and GPT-NeoX (20B) (Black et al., 2022). Their pretraining data distribution is shown in Figure 2 and our results are shown in Table 2. We can see that training on scientific data can significantly prevent the model from generating hallucinations in science and open domain (Galactica v.s. GPT-NeoX). General webpages data benefits reducing hallucinations in the domains of finance, science, and education. On the other hand, training on diverse corpora (i.e., GPT-NeoX) instead tends to result in much more
hallucinations. Note that a potential factor to affect our results is that the data cleaning procedure for each model, since it would produce pre-training data of different data qualities. We leave this discussion of data cleaning in future work.
Familiarity of Pre-training Knowledge. Prior work has reported that LLMs tend to produce hallucinations for those infrequent knowledge facts in pre-training corpus (Li et al., 2022). To explore this, we collect 5, 000 entities with the highest occurrence frequencies in Wikipedia (Karpukhin et al., 2020) and group them into 10 groups with descending frequencies. Finally, we randomly choose 100 entities for each group and construct a set of entitybased queries using manually designed templates. We evaluate Llama 2-Chat (7B) on these questions.
As shown in Figure 1 (b), we can clearly observe a long-tail distribution on the entity frequency. Interestingly, the hallucination rates exhibit a clear three-level stair-like pattern on the 10 entity groups. The LLM shows the lowest tendency to generate hallucinations for the most frequent entities in the first group. For the most entities in the long tail, the model exhibits a relative high hallucination rate.

heading: 5.2 Supervised Fine-Tuning
text: Supervised fine-tuning involves training language models to follow natural language instructions provided by the user. In this part, we explore the effect of instructions by mixing different types of instruction datasets and synthesizing new instructions.
Mixture of Instruction Datasets. Generally, there are three kinds of instructions, i.e., task-specific, daily chat, and synthetic instructions. Hence, we select three representative instruction datasets, including FLAN-T5 (Chung et al., 2022), ShareGPT (Eccleston, 2023), and Self-Instruct-52K (Wang et al., 2022a), and then fine-tune LLaMA (7B) on each individual instruction set and their mixture to examine combinatorial effects. For a fair comparison we randomly sample 40K instructions for each instruction dataset. As shown in Table 3, daily chat instructions result in a lower level of hallucinations, while task-specific instructions lead to more hallucinations in responses. The task-specific instructions mainly focus on task format learning and ignore the factual knowledge. Mixing the daily chat and synthetic instructions can reduce hallucinations in some domains such as finance.
Types of Instruction Synthesis. To improve the capacities of LLMs, existing studies typically design a series of strategies to automatically construct large-scale instruction tuning data. Following prior
work (Zhao et al., 2023), we consider four instruction synthesis methods based on Self-Instruct-52K: (1) Enhancing the instruction complexity: we adopt 40K instructions from WizardLM (Xu et al., 2023); (2) Increasing the topic diversity: we use ChatGPT to write 40K instructions for adapting to 293 topics; (3) Scaling the instruction number: we mix instructions from Moss (Sun et al., 2023) and Self-Instruct52K to obtain 100K instructions; and (4) Balancing the instruction difficulty: we compute the perplexity of instructions by LLaMA (7B) to estimate the difficulty and remove too easy or too hard and keep 40K instructions. As shown in Table 3, fine-tuning LLMs with improved instructions can be useful to alleviate hallucinations. Balancing the difficulty of instructions can significantly reduce hallucinations, while overly complex instructions eventually result in a higher level of hallucinations.

heading: 5.3 Inference Methods
text: In the inference stage, special decoding methods can be used to enhance the generation diversity but likely lead to hallucinations (Dziri et al., 2021). In this part, we will study the effects of decoding methods on hallucinations. Besides, the token-bytoken generation manner (Zhang et al., 2023a) and quantization are also possible factors of inducing hallucinations (see Appendix C.1).
Decoding Strategies. We explore the influence of four decoding strategies, including greedy search, top-k sampling, top-p sampling, and beam search. Specifically, we test Llama 2-Chat (7B) and ChatGPT, and set k = 20, p = 0.5, and the number of beams to 5. For ChatGPT, due to the API constraint, we only investigate greedy search and top-p sampling strategies. The results are shown in Table 4. We observe that diversity-oriented decod-
ing strategies such as top-p sampling contribute to inducing more hallucinations in professional domains (e.g., science), while greedy search exacerbates hallucinations in open-ended domains (e.g., education). Beam search can effectively balance the trade-off between the diversity and factuality. Moreover, we vary p of top-p sampling in the set {0.2, 0.4, 0.6, 0.8, 1.0} and show the trend in Figure 3. We can see that the hallucination rates of ChatGPT and Llama 2-Chat are highly sensitive to the variance of p in professional domains, oscillating in biomedicine and science, while exhibiting minimal fluctuations in other open-ended domains.

heading: 6 Hallucination Mitigation
text: To alleviate the hallucinating behaviors of LLMs, several hallucination mitigation techniques have been proposed. In this part, we will study the effectiveness of three methods, including RLHF, retrieval augmentation, and advanced decoding. We also examine the effectiveness of self-reflexion and prompt improvement in Appendix D.

heading: 6.1 RLHF
text: RLHF is the process of fine-tuning language models with human feedback data to align with human values (Ouyang et al., 2022). Typically, employing RLHF to mitigate hallucinations involves two
steps: (1) collect hallucinated and non-hallucinated responses to train a reward model; (2) fine-tune the LLM with the reward model using RL algorithms.
Experimental Details. We include the input questions in HaluEval 2.0 (besides the selected 1,000 test samples) as the initial prompt set. Following existing work (OpenAI, 2023), we pass a prompt to an unaligned LLM (e.g., Alpaca 7B) and get a response, and feed the prompt and response through GPT-4 to rectify the hallucinations in the response (if any) with up to 5 rounds. We keep (prompt, hallucinated response, refined response) as comparison data to train a reward model. Finally, we finetune the unaligned LLM with the reward model using PPO. Specifically, we apply this approach to two unaligned LLMs, i.e., Alpaca (7B) and Vicuna (7B), and evaluate aligned models on test samples.
Results and Analysis. As can be seen in Table 5, RLHF can effectively mitigate LLM hallucinations. Through the RLHF process, the model generates more accurate facts and its responses become more concise without much irrelevant content, leading to less factuality hallucinations. Moreover, the effect of RLHF is dependent on the domains, exhibiting more pronounced hallucination reduction in biomedicine and open domain, while showing mild effectiveness in highly professional domains such as science. In existing work, RLHF is mostly focused on open-ended domains but ignoring specific domains. We suggest that to make LLMs versatile it is crucial to execute RLHF in broader domains.

heading: 6.2 Retrieval Augmentation
text: Retrieval is generally considered as one of the effective approaches to alleviate hallucination (Li et al., 2023a). The basic idea of retrieval augmentation is to first retrieve a small set of documents from a large-scale corpus (e.g., Wikipedia) based on a user query and then the LLM can generate an accurate answer based on the retrieved documents.
Experimental Details. In our experiments, we conduct web search by retrieving documents from the whole web. Specially, we use the input question verbatim as query and request a call to Bing Search. For the search results, we only use the snippets of webpages as retrieved documents. Considering the context length and noise, we adopt top-2 snippets as evidence for generation. Appendix D.1 presents experiments about the impact of the number and relevance of retrieved documents on hallucinations.
Results and Analysis. As shown in Table 6, retrieval can significantly mitigate the hallucinations in LLM responses. The effectiveness of retrieval is more pronounced in smaller models (e.g., Llama 2- Chat 7B), as the hallucination rate of larger models (e.g., ChatGPT) has already been relatively low and smaller models acquire limited world knowledge. Figure 6 in Appendix D.1 shows that the lower the relevance between the retrieved document and question, the more likely the model is to produce hallucinations. However, more capable models like ChatGPT are more robust and less sensitive to the relevance of retrieved documents.

heading: 6.3 Advanced Decoding
text: We design two simple yet effective decoding methods that can flexibly switch between greedy search and top-p sampling to balance diversity and factuality. The first greedy-nucleus sampling assumes that when the model has high confidence in predicting the next word, it should adopt greedy search,
otherwise use top-p sampling. The second factualnucleus sampling (Lee et al., 2022a) hypothesizes that the randomness of generation will decrease as the model generating more words.
Experimental Details. We detail the two advanced decoding methods as follows: • Greedy-nucleus sampling: Following previous studies (Li et al., 2023b), we use entropy to quantify the confidence of the model and set a confidence threshold η. We formulate this method as:
wt =
{ argmaxP (wt|w1,...,wt−1), e ≤ η∑
P (wt|w1,...,wt−1) ≥ p, e > η (1)
• Factual-nucleus sampling: Following previous work (Lee et al., 2022a), the probability pt in top-p sampling at the t-th step can be formulated as:
pt = max{β, p× λt−1}, (2)
where λ is the decay factor that is used to decay the probability p at each step to reduce the randomness through time, and β is set as a lower bound of p to guarantee a certain degree of diversity. We reset p to the default value at the beginning of generating a new sentence. We apply the two advanced decoding methods to Llama 2-Chat (7B).
Results and Analysis. We present the results of the two advanced decoding methods in Table 7. As can be observed that, our proposed greedy-nucleus sampling strategy achieves comparable or even better results than the original decoding strategies in
biomedicine and science, while the factual-nucleus sampling performs well in science, education, and open domains. It can be concluded that balancing the generation diversity and factuality benefits the reduction of hallucinations and the retention of text quality. However, our designed sampling methods are sensitive to hyper-parameters such as the confidence threshold and decay factor. Thus, we leave designing more effective decoding approaches to mitigating hallucinations in future work.

heading: 7 Related Work
text: Hallucination has been a fundamental challenge in LLMs, receiving extensive attention in existing literature (Huang et al., 2023; Ji et al., 2023a; Zhang et al., 2023b; Li et al., 2023a). We discuss them in two aspects: source/detection and mitigation.
Hallucination Source and Detection. To understand and detect the hallucination in LLMs, several studies focus on using the LLM itself as the tool to study the hallucinated content. For LLMs with access to their internal states, we can delve into the inner workings of the model to explore the principles behind hallucinations (Varshney et al., 2023; Yuksekgonul et al., 2023; Azaria and Mitchell, 2023a). Typically, the internal states that can be studied by examining the output logit values, the hidden layer activations, and the attention states. For example, Varshney et al. (2023) leveraged the output logit values of the model as a signal of hallucinations to estimate the uncertainty of responses. For models that can only be accessed through API calls, hallucinations are typically studied by analyzing the relationship between input prompts and the model’s output responses (Rawte et al., 2023b; Manakul et al., 2023; Yao et al., 2023). For example, Rawte et al. (2023b) explored the effects of linguistic elements in prompts (readability, formality, and concreteness) on the LLM hallucinations.
Hallucination Mitigation. To mitigate the hallucinations, existing studies focus on different training
and utilization stages of LLMs. First, mitigation in pre-training is typically centered around dataset curating and cleaning. In this line, existing studies (Das et al., 2023; Kamalloo et al., 2023; Umapathi et al., 2023) aim to construct a higher quality corpus for model pre-training by building datasets within specific domains or cleaning existing datasets. After pre-training, fine-tuning approaches can be further employed for hallucination mitigation, such as the applications of SFT (Wang et al., 2022b) and RLHF (Fernandes et al., 2023). In practical use, mitigation during generation is mainly focused on developing more effective decoding strategies (Lee et al., 2022b; Shi et al., 2023; van der Poel et al., 2022), leveraging external knowledge (Chern et al., 2023a; Varshney et al., 2023) and designing more effective prompts (Agrawal et al., 2023; Touvron et al., 2023b). Furthermore, mitigation during the post-processing stage can be implemented by using LLM itself (Mündler et al., 2023) or external knowledge (Chen et al., 2023) as the fact-checking module to verify the generated text.

heading: 8 Conclusion
text: This paper presented a comprehensive empirical analysis about LLM hallucinations in the three aspects of detection, source and mitigation. We constructed the hallucination benchmark HaluEval 2.0 and developed an LLM-based automatic detection approach. Based on this benchmark, we further systematically investigated the possible sources for LLM hallucination in the stages of pre-training, SFT, RLHF, and inference, and also examined the effectiveness of a series of hallucination mitigation strategies, including RLHF, retrieval augmentation, self-reflexion, advanced decoding, and prompt improvement. As the major contribution, our benchmark can be reused for further research, and our work has led to a series of important empirical findings on the source and mitigation of hallucination.

heading: 9 Limitations
text: Despite the great efforts that we have made, our analysis about pre-training stage and SFT is still limited, due to the lack of disclosed training details and the supporting computational sources. We will investigate into the two stages with more detailed analysis as future work. Furthermore, our experimental tests are not yet sufficient and we are also particularly interested in the working mechanism or nature of LLMs in generating the hallucinations. We will conduct more in-depth research work in the future. In addition, this paper mainly aims to provide empirical analysis on existing techniques to mitigate the LLM hallucinations, and there are no new hallucination mitigation strategies proposed. We will also consider developing improved mitigation strategies based on the findings of this work. Our hallucination detection approach is based on the GPT model (i.e., GPT-4), which might inevitably lead to some minor errors. We will continuously improve our method in the future work.

heading: Acknowledgments
text: This work was partially supported by National Natural Science Foundation of China under Grant No. 62222215, and Beijing Natural Science Foundation under Grant No. L233008 and 4222027. Xin Zhao is the corresponding author.

heading: A Hallucination Taxonomy
text: In Section 2, we have defined six types of factuality hallucination in a fine-grained way. To make a clear understanding of our categorization of LLM hallucinations, we present some examples for each type in Table 8.

heading: B Hallucination Detection
text: Our hallucination detection approach consists of two steps: fact extraction and fact verification. The instructions for the two steps are shown in Table 9 and Table 10.

heading: C Hallucination Source
text: C.1 Inference Methods
Quantization. To understand the impact of quantization on hallucinations, we quantize Llama 2-Chat (7B) and (13B) at three precision levels: 4-bit, 8- bit and 16-bit. Specially, we employ the library bitesandbytes1 to quantize the original 16-bit models to 8/4 bits by specifying the commands load_in_8bit and load_in_4bit, which focused on the quantization of weights for LLMs. The hallucination results of quantized models are shown in Table 11. Despite with reduced memory footprint and accelerated inference rate, the 8-bit
1https://github.com/TimDettmers/bitsandbytes
and the green text denotes the in-context demonstration.
and 4-bit quantization results in an overall higher level of hallucination compared to the original 16- bit model. In most cases, 8-bit quantization has a minimal impact on the hallucination of the model, while 4-bit quantization significantly increases the hallucination in the model’s responses. In certain domains such as biomedicine, the hallucination rate of the quantized model will noticeably increases compared to the original model.
Token-by-Token Generation. Most LLMs adopt the token-by-token manner to generate a token at a time. Once the generated part contains erroneous or unreasonable content, the model is difficult to complete the sentence correctly (Azaria and Mitchell, 2023b). To explore the drawback of token-by-token generation, we conduct a case study to qualitatively analyze how the generation paradigm leads to hallucinations. We present several examples across five domains in Table 12. For each sample, we employ ChatGPT to generate two responses conditioned on two similar prefixes with the only difference being the last word. We can clearly observe that the LLM over-commits the errors in previous tokens and continuously complete the sentence incorrectly. For example, when we replace the degree words (e.g., “more” → “less”), ChatGPT does not identify the mistakes and keep generating the same sequence, leading to hallucinations. Besides, for the educational question about the roman numerals, when beginning with a prefix word “larger”, ChatGPT is unable to indicate subtraction by generating the right word “after” (instead of “before”). Similarly, in the scientific domain, the real-world fact is that men have a higher risk of dying from pneumonia than woman. When we replace the prefix word “men” with “woman”, ChatGPT is not able to complete the fact correctly by exchanging an-
and the green text denotes the in-context demonstration.
other expression. In open domain, when we delete “second” from the first prefix, ChatGPT incorrectly completes the sentence. In fact, one correct completion for the new prefix is “Pluto is the smallest celestial body in the solar system that has ever been classified as a planet”.
C.2 Prompt Design Prompting has become the major approach to utilizing LLMs. However, inappropriate prompt design would lead to incapable attention of important information in the input (Liu et al., 2023). In addition, ambiguous and superficial questions posed by users might steer the model towards generating unrelated, implausible, or bizarre output (Rawte et al., 2023a). In this part, we continue to analyze the effect of prompt design on LLM hallucinations.
Prompt Design. Generally, a prompt contains task description, input question, and contextual information such as in-context demonstrations (Santu and Feng, 2023). Here, we experiment with several prompt designs by varying the three ingredients: • Base prompt: the initial prompt with a simple task description and input question.
• Manual description prompt: manually rewriting the task description in base prompt. • Synthetic description prompt: using ChatGPT to synthesize the task description in base prompt. • Refined question prompt: refining the initial question in base prompt by ChatGPT. • Manual in-context prompt: manually selecting five in-context demonstrations for the base prompt. • Retrieved in-context prompt: retrieving demonstrations based on BERT similarity from HaluEval 2.0 (besides the 1,000 test samples). • Synthetic in-context prompt: using ChatGPT to synthesize the in-context demonstrations. • Reverse prompt: reversing the position of task description and input question in base prompt (i.e., place the task description after the input question). We feed these prompts into ChatGPT and Llama 2-Chat (7B) and the evaluation results are shown in Table 13. First, we can observe that rewriting the task description with more details can reduce the hallucinations to some extent, while this effect is varied in domains. For professional domains (i.e., biomedicine and science), incorporating more details into the task description can mitigate some
hallucinations. Second, leveraging in-context learning can also help eliminate hallucinations in LLM’s responses. These examplars or demonstrations can be manually selected, retrieved from candidate corpus, or automatically generated by LLM itself. It
is noting that MaHR and MiHR are not strongly positively correlated, where the two metrics measure the hallucination degree from distinct levels. Finally, in most cases, rewriting the question or placing the task description at the end of the input
question will instead hurt the model performance and induces more hallucinations.
Question Content. Following prior work (Rawte et al., 2023a), we further delve into how linguistics of question content, specifically readability, formality, and concreteness, influence the occurrence of LLM hallucinations. Readability quantifies the extent to which the question can be understood by humans; Formality refers to the degree of appropriate tone and professionalism conveyed by the choice of words, grammatical structure, style, etc.; Concreteness indicates whether a word represents a specific and tangible concept. For each question in our dataset, we invite human labelers to score these three properties based on the 5-point Likert scale, ranging from 1-point (“very terrible”) to 5-point (“very satisfying”). For these questions, we instruct ChatGPT to generate responses and compute the average hallucination rate under each score. In Figure 4, we can observe that ChatGPT exhibits a lower propensity to generate hallucinations on those questions that are easier to read and use more formal and specific language. Note that constrained by the scale of the annotation dataset, there were fewer/no scores from humans for certain linguistics (e.g., “2-point” formality in biomedicine), resulting in a relatively low hallucination rate.

heading: D Hallucination Mitigation
text: D.1 Retrieval Augmentation
Experimental Details. To examine the impact of the number of retrieved documents on hallucination mitigation, we use the top-k documents as context and vary k in the set {1, 2, 5, 10}. Besides, to further validate the effect of the relevance of retrieved documents on hallucination mitigation, we randomly sample one document from top-k documents. Here, the variance of k reflects four levels of relevance to the question, ranging from strong to weak relevance.
Results and Analysis. In Figure 5, we show the results of using top-k documents as evidence. We can clearly see that ChatGPT and Llama 2-Chat mostly produce less hallucinations in top-2 retrieval, except that ChatGPT prefers top-5 in science. This is because that including more documents as the context of LLMs can bring more noises into the generation process, leading to higher level of hallucination. Therefore, we conduct top-2 retrieval for several LLMs and present the results in Table 6. Besides, we present the effect of the relevance between question and document by randomly selecting one document from top-k results. The results in Figure 6 shows that the lower the relevance between the retrieved document and the question, the more likely the model is to generate hallucinations. However, more capable models like ChatGPT are less sensitive to the relevance of retrieved docu-
ments.
D.2 Self-Reflexion
Reflexion (Shinn et al., 2023) is an effective means through which LLMs can learn from and rectify their mistakes. Specially, the prior failure trials are transformed as textual feedback, which would be incorporated as additional context for the LLM itself. With the guidance of feedback, the LLM can generate an improved plan for the next attempt, called self-reflexion. Existing research (Ji et al., 2023b; Dhuliawala et al., 2023b) has demonstrated that self-reflexion is an effective method for hallucination mitigation. However, self-reflexion is a complex and advanced ability that requires mistake perception, feedback summarization, and behav-
ioral planning. In light of this, we aim to explore to which extent self-reflexion can mitigate hallucinations and how the capacities of LLMs affect the reflexion performance on hallucination mitigation.
Experimental Details. We conduct self-reflexion experiments on Llama 2-Chat 7B, 13B, and 70B, to examine its effect on hallucination mitigation. Specifically, for a given query, we first obtain the initial model response, then require the model itself with instructions to reflect on whether its response contains any errors. If errors are detected, the model is prompted to provide a corrected response. This process is repeated until the model response is error-free or the maximum number of reflection iterations is reached. We set the maximum
number of reflection iterations to 5. In addition, we also test these models without self-reflexion for performance comparison.
Results and Analysis. Figure 7 illustrates the relationship between the model’s scale and its effectiveness in mitigating hallucinations through selfreflexion. We can clearly see that only when reaching a certain scale (i.e., 70B in our experiments), the LLM possesses the self-reflective ability to mitigate hallucinations in its responses. For Llama 2-Chat 7B and 13B, self-reflexion, on the contrary, can even result in more erroneous responses. We found that due to the limited capacity of smaller models, the reflective behavior instead makes them suspect their original correct answers and generates wrong ones. Furthermore, the benefits obtained from self-reflexion are domain-sensitive, e.g., selfreflexion reduces the LLM hallucinations slightly in the fields of finance and science, while significantly in the open domain.
D.3 Prompt Improvement
According to the results and analysis in Section C.2, we improve the task description, question expression, and in-context demonstrations in the original prompt. Furthermore, chain-of-thought (CoT) has been proven to be helpful in hallucination mitigation (Wang et al., 2023), so we explore adding CoT reasoning into our prompt.
Experimental Details. We improve the original base prompt from the following aspects: • We incorporate Base prompt, Manual description prompt, and Manual in-context prompt from Section C.2 for comparison. • Domain info prompt: injecting domain infor-
mation into the task description in the base prompt. • Character role prompt: defining a particular role (e.g., scientist) for the system in the base prompt. • Zero-shot cot prompt: adding zero-shot CoT to the base prompt by prepending “Let’s think step-by-step”. • Few-shot cot prompt: adding few-shot CoT to the base prompt by injecting CoT examples. Similarly, we test ChatGPT and Llama 2-Chat (7B) with these improved prompts to generate responses. For few-shot cot prompt, the few-shot reasoning examples are manually-written and different for each domain.
Results and Analysis. We show the prompt improvement results in Table 14. We can found that injecting domain information into task description or defining a character role for the system has an oscillatory effect on mitigating the hallucinations in LLM’s responses. In the domains of finance and science, the two prompt improvement strategies can help LLMs generate more accurate responses. For Chain-of-Thought (CoT) prompting, its effect on hallucination mitigation heavily depends on the specific LLMs. For larger models like ChatGPT which possess exceptional reasoning capabilities, zero-shot or few-shot CoT reasoning can significantly benefit reducing the hallucinations. While for smaller models like Llama 2-Chat (7B) with limited reasoning abilities, leveraging CoT reasoning fails to effectively eliminate hallucinations and instead exacerbates the presence of hallucinations in their responses. Therefore, when engaging in prompt engineering, it is crucial to comprehensively consider factors such as model size, domain
characteristics, and task difficulty.

heading: Future Work
text: We will investigate into the two stages with more detailed analysis as future work. Furthermore, our experimental tests are not yet sufficient and we are also particularly interested in the working mechanism or nature of LLMs in generating the hallucinations. We will conduct more in-depth research work in the future. In addition, this paper mainly aims to provide empirical analysis on existing techniques to mitigate the LLM hallucinations, and there are no new hallucination mitigation strategies proposed. We will also consider developing improved mitigation strategies based on the findings of this work. Our hallucination detection approach is based on the GPT model (i. e. , GPT-4), which might inevitably lead to some minor errors. We will continuously improve our method in the future work