RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models (2024)

Abstract: Retrieval-augmented generation (RAG) has become a main technique for alleviating hallucinations in large language models (LLMs). Despite the integration of RAG, LLMs may still present unsupported or contradictory claims to the retrieved contents. In order to develop effective hallucination prevention strategies under RAG, it is important to create benchmark datasets that can measure the extent of hallucination. This paper presents RAGTruth, a corpus tailored for analyzing word-level hallucinations in various domains and tasks within the standard RAG frameworks for LLM applications. RAGTruth comprises nearly 18,000 naturally generated responses from diverse LLMs using RAG. These responses have undergone meticulous manual annotations at both the individual case and word levels, incorporating evaluations of hallucination intensity. We not only benchmark hallucination frequencies across different LLMs, but also critically assess the effectiveness of several existing hallucination detection methodologies. We show that using a high-quality dataset such as RAGTruth, it is possible to finetune a relatively small LLM and achieve a competitive hallucination detection performance when compared to the existing prompt-based approaches using state-of-the-art LLMs such as GPT-4. Furthermore, the finetuned model can effectively mitigate hallucination in LLM responses. 1

heading: 1 Introduction
text: Large language models (LLMs) have achieved remarkable success in a variety of tasks, including text generation (Li et al., 2024), machine translation (Kocmi and Federmann, 2023), and question answering (Zhao et al., 2023). However, one of the key challenges in deploying LLMs in real-world applications is their tendency to hallucinate (Kaddour et al., 2023). Hallucination in the context
1The RAGTruth dataset is available at https://github. com/ParticleMedia/RAGTruth
of LLMs usually refers to a situation where the model generates content that is not based on factual or accurate information (Rawte et al., 2023). The occasional generation of outputs that appear plausible but are factually incorrect significantly undermine the reliability of LLMs in real-world scenarios, such as medical diagnoses (Pal et al., 2023) and news summarization (Shen et al., 2023).
To reduce hallucination, various methods have been developed that can be applied at different stages of LLM lifecycle, including pretraining (Brown et al., 2020), supervised finetuning (Zhou et al., 2023; Zhang et al., 2024), RLHF (Ouyang et al., 2022; Lin et al., 2022), and inference (Dhuliawala et al., 2023; Gao et al., 2023). In terms of detection, methods are developed by examining the model’s intrinsic state (Guo
10862
et al., 2017), comparing it with external data and tools (Chern et al., 2023), or leveraging the LLM’s inherent powerful capabilities for selfchecking (Agrawal et al., 2024; Manakul et al., 2023). Retrieval-augmented generation (RAG) is extensively used to supply LLMs with updated, relevant knowledge, significantly mitigating hallucination (Varshney et al., 2023; Mishra et al., 2024). Nevertheless, even with RAG and other enhancements, LLMs still produce statements that are either unfounded or contradict the information provided in the retrieved references (Shuster et al., 2021).
Despite the growing awareness of the hallucination phenomenon, the understanding of hallucination in LLMs is still in its early stages. One key challenge is the lack of high-quality, large-scale datasets specifically designed for hallucination detection. This issue is particularly acute in RAG settings. Due to the relatively low hallucination ratio, a substantial increase in annotation resources is needed. Existing datasets for LLM hallucination detection are predominantly synthesized (Li et al., 2023). For instance, in Liu and Liu (2023); Longpre et al. (2021), prompts conflicting with conventional knowledge are purposely generated to trigger hallucinations. While these approaches are efficient at generating hallucinations, the resulting artificial hallucinations can substantially differ from those that naturally occur. In Chen et al. (2023); Hu et al. (2023), hallucination datasets are developed by manual annotations of naturally produced LLM responses. However, these datasets are of limited size and are not specifically focused on the RAG scenario.
In this paper, we introduce a large-scale highquality dataset specifically designed for word-level hallucination detection for RAG applications. Using this dataset, we have conducted an extensive benchmarking of mainstream LLMs to assess their tendency to generate hallucinations, as well as evaluate current methods for hallucination detection. Additionally, we have demonstrated superior performance in identifying hallucinations by fine-tuning LLM with RAGTruth dataset. Our key contributions are:
(i) We propose RAGTruth, a large-scale wordlevel hallucination evaluation dataset specifically for the RAG scenario across several common tasks. It consists of nearly 18,000 fully annotated natural responses generated from
major open-source and closed-source LLMs.
(ii) We perform a comprehensive comparison of different hallucination detection methods at both the passage and word levels.
(iii) We present a baseline method of fine-tuning LLM for hallucination detection. It is shown that by fine-tuning the Llama-2-13B model on the RAGTruth training data, we can achieve results competitive to the existing promptbased approaches using GPT-4. This shows the potential of developing better hallucination detection methods using RAGTruth.
(iv) We show that by using our finetuned hallucination detector, it is possible to significantly reduce the occurrence of hallucinations in the responses from LLMs. The improvement holds even for models with inherently low hallucination rates, such as GPT-4.

heading: 2 Related Work

heading: 2.1 Hallucination of Large Language Models
text: Though hallucination in traditional natural language generation (NLG) contexts has been widely studied (Ji et al., 2023), comprehending and tackling this problem in the context of LLMs presents distinct challenges (Zhang et al., 2023). Existing research has demonstrated that incorporating upto-date, relevant knowledge in the prompt can effectively reduce fact-conflicting hallucination (Vu et al., 2023; Lewis et al., 2020). This approach, referred to as Retrieval-Augmented Generation (RAG), is widely used in real-world LLM applications. For instance, Google Bard 2 and Microsoft BingChat 3 have implemented this technique.

heading: 2.2 Hallucination Evaluation Datasets
text: Extensive research has focused on hallucination benchmarks within conventional Natural Language Generation settings (Dziri et al., 2022; Zhong et al., 2021; Durmus et al., 2020; Lin et al., 2022). With the rise of LLMs, the detection of hallucinations has become increasingly challenging, necessitating the development of high-quality datasets for LLM evaluation (Chen and Shu, 2024). Contributions in this domain include HaluEval (Li et al., 2023), which introduced datasets encompassing both synthetically and naturally generated LLM responses,
2https://bard.google.com 3https://www.bing.com
and FELM (Chen et al., 2023), which concentrated on naturally generated LLM responses across multiple domain tasks. RefChecker (Hu et al., 2023), a distinctive approach, breaks down claims in LLM responses into triples and utilizes human annotation to assess the truthfulness of facts. Notably, these works primarily focus on annotating factual hallucinations in LLM responses. Distinguishing from previous research, our work centers on the evaluation of LLMs within RAG settings.

heading: 2.3 Hallucination Detection Methods
text: Researchers have been exploring various methods to enhance the reliability of LLMs by detecting hallucinations. In Azaria and Mitchell (2023); Xiao and Wang (2021); Malinin and Gales (2021), intrinsic model uncertainty metrics such as token-level probability and entropy are used to detect hallucinations. When direct access to output uncertainty is not feasible, as in the case with limited APIs like GPT-4, an alternative approach involves employing a fully accessible LLM as a proxy (Manakul et al., 2023). In Falke et al. (2019); Barrantes et al. (2020), natural language inference modules are adapted to check the information consistency between the articles and their summaries, and it has been shown that external knowledge is helpful for detecting factual hallucinations. (Guo et al., 2022; Mallen et al., 2023). Additionally, methods that leverage the inherent capabilities of LLMs have been proposed for self-checking, such as verbalization-based and consistency-based methods (Xiong et al., 2024; Manakul et al., 2023). These techniques aim to detect hallucinations without relying on internal states or external data and tools.

heading: 3 Construction Process of RAGTruth
text: We established a data generation and annotation pipeline as shown in Figure 1.

heading: 3.1 Hallucination Taxonomy
text: Different from open-end generation, under RAG setting, the prompt contains rich context information, and the model is generally required to generate text based on the provided context. The detection and mitigation of inconsistencies between retrieved information and responses emerge as significant sources of hallucination.
As outlined below, we categorize the hallucination in the RAG setting into four types. For concrete examples of each type, please refer to
Appendix A.
Evident Conflict: for when generative content presents direct contraction or opposition to the provided information. These conflicts are easily verifiable without extensive context, often involving clear factual errors, misspelled names, incorrect numbers, etc.
Subtle Conflict: for when generative content presents a departure or divergence from the provided information, altering the intended contextual meaning. These conflicts often involve substitution of terms that carry different implications or severity, requiring a deeper understanding of their contextual applications.
Evident Introduction of Baseless Information: for when generated content includes information not substantiated in the provided information. It involves the creation of hypothetical, fabricated, or hallucinatory details lacking evidence or support.
Subtle Introduction of Baseless Information: is when generated content extends beyond the provided information by incorporating inferred details, insights, or sentiments. This additional information lacks verifiability and might include subjective assumptions or commonly observed norms rather than explicit facts.

heading: 3.2 Response Generation
text: Tasks and Data Sources We selected three widely recognized generation tasks with RAG settings for response generation: Question Answering, Data-to-text Writing, and News Summarization.
For the task of question answering, we conducted a random sampling from the training set of MS MARCO (Nguyen et al., 2016). To reduce the difficulty of annotation, we selected only those questions related to daily life, and preserved only three retrieved passages for each question. Then we prompted LLMs to generate answers for each question solely based on the retrieved passages.
For the data-to-text writing task, we prompted LLMs to generate an objective overview for a randomly sampled business in the restaurant and nightlife categories from the Yelp Open Dataset (Yelp, 2021). In this dataset, information pertaining to a business is represented using structured data. To streamline the annotation process, we focused only on the following business information fields: BusinessParking, RestaurantsReservations, OutdoorSeating, WiFi, RestaurantsTakeOut,
RestaurantsGoodForGroups, Music, and Ambience. In addition to the structured data, we have also included up to three business-related user reviews to enrich the context information. In the prompt, these information is represented in JSON format.
For the news summarization task, we randomly selected documents from the training set of the well-known CNN/Daily Mail dataset (See et al., 2017) as well as recent news articles from a prestigious news platform. LLMs were prompted to generate a summary for each of the source news.
Models The following six models with strong instruction-following ability are used for response generation: GPT-3.5-turbo-0613 and GPT-4-0613 from OpenAI (OpenAI et al., 2024); Mistral-7bInstruct from Mistral AI (Jiang et al., 2023); Llama2-7B-chat, Llama-2-13B-chat and Llama-2-70Bchat (4bit quantized)4 from Meta (Touvron et al., 2023). To ensure a fair comparison, the prompts used for response generation are kept straightforward with subtle differences among various models to optimize their performance. We provide detailed prompts in the Appendix B.
For each sample, we collected one response from each model. As a result, we got a total of 6 responses for each input sample.

heading: 3.3 Human Annotation
text: Identifying AI-generated hallucinations is a challenging task. It requires a strong capacity for critical thinking to understand the logical flow of various texts, along with meticulous attention to detail for spotting subtle inaccuracies and inconsistencies. Moreover, a certain level of media literacy and knowledge of current affairs is crucial to grasp
4https://huggingface.co/TheBloke/ Llama-2-70B-Chat-AWQ
the subjects discussed in news-related sample data. Therefore, we chose annotators who are proficient in English and possess a bachelor’s degree in English, Communications, or relevant fields to ensure the accuracy and reliability of the annotation results. We recruited annotators from a professional vendor and paid them at a rate of $25 per hour per individual.
The annotators are invited to perform annotation tasks using Label Studio (Tkachenko et al., 2020- 2022). Each labeling task is presented within one page, comprising the following components: 1) the context provided to the AI models; 2) a set of 6 responses, generated by different AI models. Our annotation interface is available in Appendix C.
Their task was to annotate the specific spans of the generated text that contains hallucinated information and categorize them into the four types. To ensure the quality of the annotations, each response is independently labeled by two annotators. The consistency rate of two annotators was 91.8% at the response level and 78.8% at the span level. In cases where there is a considerable difference between the two annotations, a third review is undertaken.

heading: 3.4 Annotations for Adaptive Evaluation
text: In different contexts, the definition and criteria for hallucination vary, and the annotation of hallucination is not always straightforward. In contentious cases, additional annotations are provided to accurately reflect these situations. This approach enables users to adopt various evaluation strategies tailored to their specific application circumstances. Please refer to Appendix C for more statistical information about these annotations.
Implicit Truth The extensive world knowledge and ability of LLMs is a significant advantage in open-ended generation scenarios. But in the con-
0% 25% 50% 75% 100%
QA
Data-to-text Writing
Sum.
Subtle baseless info Evident baseless info
Subtle conflict Evident conflict
Figure 2: Frequency of different types of hallucination by task.
text of this paper, which focuses on the relatively strict RAG scenarios, we have labeled information that is not mentioned in the reference but may be truthful as hallucinations. For instance, mentioning a local officer’s name not present in the reference or claiming that a restaurant accepts credit card payments without any basis.
The decision is based on the observation that LLMs have a relatively high chance of making errors when generating detailed facts, partly because their embedded knowledge can be outdated. Therefore, RAG applications usually instruct LLMs not to generate factual content without the support of references. Besides, we provided an additional span-level annotation named implicit_true for these spans to accommodate different application needs.
Differences in Handling Null Value In the datato-text writing task, certain fields sometimes are with null values. We observed that in the generated results, null is often interpreted as false by some models. Since the more common expressions for negation in our dataset are the boolean value False or the text No, we labeled these instances as hallucinations (evident introduction of baseless info) and provided a special span-level annotation named due_to_null for these spans. In the subsequent hallucination detection experiments, our prompts will be aligned with this standard.

heading: 4 Hallucination Benchmark Analysis

heading: 4.1 Basic Statistics
text: We presented detailed statistics of RAGTruth in Table 2. Compared to existing datasets for hallucination detection (Cao et al., 2023; Kamoi et al., 2023), the RAGTruth dataset is considerably large in scale. The corpus contains a total of 2,965 instances of data, which include 989 instances for question answering, 1,033 instances for date-to-text
writing, and 943 instances for news summarization. Each instance comprises responses from 6 different models. As shown in Table 2, the RAGTruth dataset also features longer prompt and response lengths than existing datasets for hallucination detection (Wang et al., 2020).

heading: 4.2 Hallucination Statistics
text: Hallucination Types As shown in Figure 2, the generation of information baseless in the context was significantly more prevalent than the generation of information conflicting with the context, especially for the question answering tasks. Within the two major categories of baseless info and conflict, the more severe hallucinations, namely Evident baseless info and Evident conflict, respectively, account for a significant portion. This observation highlights the importance and challenges of LLMs hallucination mitigation, even in RAG settings.
Hallucination vs Tasks As shown in Table 2, across the three tasks, the date-to-text writing task exhibited the highest frequency of hallucinations in its responses. Inconsistent handling of JSON format data, especially time and attributes, contributed to a significant number of hallucinations in this task. Interestingly, the models did not show a higher rate of hallucinations for recent news compared to outdated news. This could be attributed to the shorter context length in the recent news subtask compared to the CNN/DM subtask.
Hallucination vs Models Table 3 illustrates that among the data we collected, OpenAI’s two models demonstrated notably lower hallucination rates compared to others. Specifically, GPT-4-0613 exhibited the lowest hallucination frequency.
To more clearly compare the hallucination rate of different models, we calculated the hallucination density for each model across three tasks. Hallucination density is defined as the average number of hallucination spans per hundred words in the responses. In the Llama2 series, a clear negative correlation was observed between the model scale and hallucination density, aside from the data-totext writing tasks. Despite its strong performance in various benchmarks and leaderboards (Zheng et al., 2023), the Mistral-7B-Instruct model generated the highest number of responses containing hallucinations.
Hallucination vs Length After removing the top and bottom 5% of outliers, we partitioned the data
for each task type into three equal-sized groups according to the length of the context/response. We then computed the average number of hallucinated spans per response within each group. As shown in Table 4, there is a clear overall trend of an increase in the average number of hallucinations as the response length grows. Only the average number of hallucinations in news summarization tasks significantly increases with the length of the context. This may be because the contexts in the other two tasks are more structured, and an increase in length does not significantly raise the difficulty of understanding the content.
Location of Hallucinations In Figure 3, we present the heatmap of the hallucination occurrence positions. Hallucinations are significantly more likely to occur towards the end of responses in question-answering and news summarization tasks. Compared to other tasks, the data-to-text writing task has a relatively higher occurrence of halluci-
nations in the first half. In that bright area, hallucinations concerning business attributes frequently occur.

heading: 5 Experimental Setup

heading: 5.1 Hallucination Detection Algorithms
text: Using RAGTruth, we conducted experiments with the following four distinct algorithms for hallucination detection: Hallucination Detection Prompt: Hallucination detection prompts are manually crafted to instruct LLMs (GPT-4-turbo and GPT-3.5-turbo) in assessing whether a given reference-response pair contains hallucinated content and to identify the corresponding hallucinated spans in the response. For detailed information about these prompts, please refer to Appendix D. SelfCheckGPT (Manakul et al., 2023): SelfCheckGPT employs a zero-resource, samplingbased method to fact-check the responses of blackbox models. When processing each response in RAGTruth, 3 extra responses from the same model
were sampled and served as references, and GPT3.5-turbo was used to verify consistency. We detected hallucinations sentence-by-sentence within a response, and then aggregated these results to provide a response-level detection outcome. LMvLM (Cohen et al., 2023): LMvLM is an approach that employs a multi-turn interaction between two Language Models that aim to discover inconsistencies through cross-examination. LLM Finetuning: Llama-2-13B has been finetuned using the training set from RAGTruth. The model takes the context-response pair with proper instructions as the input and treats the hallucinate span as the targeted generation output. We employed full training with an initial learning rate of 2e-5, and limiting the training to 1 epochs, all conducted on 4 A100 GPUs.

heading: 5.2 Data Split
text: All detection algorithms are tested on the same RAGTruth test set, which consists of 450 instances in total, derived by randomly selecting 150 instances from each task type. The rest of the data is used to fine-tune the LLama-2-13B model, as previously mentioned.

heading: 5.3 Evaluation Metrics
text: It is a more challenging and significant task to identify the locations of hallucinations within the response than only determining whether a response contains hallucinations. We assess hallucination detection at both the response and span levels.
Response-level Detection We report precision, recall, and F1 score for each detection algorithm and its variants across different tasks.
Span-level Detection We calculate the overlap between the detected span and human-labeled span and report the precision, recall, and f1 score at the char-level.

heading: 6 Experimental Results

heading: 6.1 Response-level Detection
text: The results in Table 5 reveal that hallucination detection remains a significant challenge in the context of RAG for all existing detection methods. Even when reference information is available, the responses generated may still include hallucinations, which current LLMs cannot reliably identify. The most advanced LLM, GPT-4-turbo, achieves only an average F1 score of 63.4%. For another notable baseline, SelfCheckGPT also shows unsatisfactory performance in this regard, achieving an average F1 score of 58.8% with GPT-3.5-turbo.
By utilizing our high-quality training set, a finetuned Llama-2-13B can achieve the best performance with an average 78.7% f1 score. This shows the effectiveness of our data in improving the model’s hallucination detection ability.

heading: 6.2 Span-level Detection
text: RAGTruth, as a hallucination corpus with finegrained span labels, enables us to present experimental results for span-level detection, serving as a baseline for future research. As shown in Table 6, the overall performance of the current detection method is sub-optimal, highlighting the challenges in span-level detection. Even the advanced GPT-4-turbo tends to incorrectly classify many non-hallucinated contents with a low precision of 18.4%. While our fine-tuned model shows im-
proved capability in identifying hallucinated spans by achieving an averaged f1 score of 52.7%, it still falls short of perfect detection, emphasizing the inherent difficulties of this task.
We also report the detection performance across four different types of hallucination spans. In the current stage, as we have not differentiated the types of detected hallucinations, we only report the char-level recall for different types of hallucinations. As indicated in Figure 5, the detection of evident hallucinations proves more effective compared to that of subtle hallucinations.

heading: 6.3 Hallucination Suppression
text: We tested the effectiveness of hallucination suppression using our finetuned hallucination detection model. For the 450 instances in the test set, we employed two strategies to select a final output from two responses generated by two different models with similar hallucination densities. The first strategy involved selecting the response with fewer predicted hallucination spans. The second strategy, more stringent, mandated that the selected response have no detected hallucination spans. When the number of hallucination spans detected in both candidate responses is the same, one will be chosen at random. Due to limited response candidates, not
all instances have a response that conforms to the second strategy. In practical scenarios, this issue can be addressed by increasing the number of candidate responses. We employed random selection as a simple baseline for comparison.
The results shown in Table 7 indicate that with the help of the hallucination detector, both strategies can significantly reduce the hallucination rate. For the relatively small Llama-2-7B-chat and Mistral-7B-Instruct models, compared to random selection, the first strategy reduced the hallucination rate by 21.6%, while the second strategy achieved a reduction of 63.2%. Even for models with a low hallucination rate, specifically GPT-3.5Turbo and GPT-4, employing the finetuned hallucination detector for sampling can still further reduce the rate of hallucinations. The two strategies yielded a reduction in hallucination rates of 42.9% and 51.0%, respectively. These results demonstrate the potential of an efficient hallucination detection model in developing trustworthy RAG LLMs.

heading: 7 Conclusion
text: In this paper, we introduce RAGTruth, a large-scale corpus of naturally generated hallucinations, featuring detailed word-level annotations tailored for RAG scenarios. Our work includes an in-depth analysis of the interplay between hallucinations and various factors, such as task types, models being used, and contextual settings.
Additionally, we conduct empirical benchmarks of several hallucination detection approaches using our corpus. We show that fine-tuning Llama with RAGTruth leads to competitive performance. This implies that by using a high-quality dataset such as RAGTruth, it is possible to develop specialized hallucination detection models that are highly effective when compared to prompt-based methods using general models such as GPT-4.
Simultaneously, our findings reveal that identifying hallucinations in RAG contexts, particularly
at the span level, remains a formidable challenge, with current methods still falling short of reliable detection. We hope that RAGTruth, can assist the development of hallucination detection techniques for retrieval augmented generation.

heading: 8 Limitations
text: The study of hallucination in large language models is a rapidly advancing field, characterized by the continuous evolution of application scenarios, sources of hallucination, and detection and prevention techniques. Our work represents the first attempt to benchmark hallucination within the RAG setting, revealing several areas that require further investigation. It is important to carefully benchmark the generality capability of the detection model trained on our data, assessing how well it performs across different datasets and contexts. Additionally, we aim to evaluate the effectiveness of using manual annotations versus synthetic data, as well as explore the potential benefits of combining both approaches to optimize the return on investment.

heading: 9 Ethical considerations
text: This work is in full compliance with the Ethics Policy of the ACL. We acknowledge that responses generated by LLMs in this study may contain inaccuracies. Aside from this, to the best of our knowledge, there are no additional ethical issues associated with this paper.

heading: Acknowledgement
text: We appreciate the valuable feedback and assistance from Shizhe Diao. We thank Doris Li for her support in creating the illustrations for this research.

heading: A Hallucination Examples
text: EVIDENT CONFLICT
Example 1 Original content: They’ve sent about 500 of the most seriously injured people for treatment in Kathmandu already – the majority by road. Generative content: The team airlifted 500 injured people. . . to Kathmandu for medical treatment.
Example 2 Original content: "RestaurantsReservations": false Generative content: The restaurant offers outdoor seating and takes reservations
SUBTLE CONFLICT
Example 1 Original content: This is the section that lets the government deny trademark protection to a mark that is "immoral," "scandalous" or "disparaging." Generative content: . . . violates Section 2(a) of the trademark act, which prohibits marks that are "immoral, scandalous, or disqualified."
Example 2 Original content: It was quite a hassle to find a table since this place is crowded all the time, but well, nothing can stand between me and that absolutely delightful lobster roll Generative content: The restaurant is located in a crowded area, but the delicious lobster rolls make it worth the hassle to find a table.
EVIDENT INTRODUCTION OF BASELESS INFORMATION
Example 1 Original content: The couple divorced in 1968. . . She had no contact with the surviving members of the band until meeting up at the 2006 Las Vegas premiere of "The Beatles Love." Generative content: The couple divorced in 1968, but later reconciled and attended the 2006 Las Vegas premiere of "The Beatles Love."
Example 2 Original content: “RestaurantsTakeOut”: true Generative content: Customers can also enjoy takeout and delivery options.
SUBTLE INTRODUCTION OF BASELESS INFORMATION

heading: B Response Generation Prompts
text: QUESTION ANSWERING
Answer the following question: {question} Bear in mind that your response should be strictly based on the following 3 passages: {passages} In case the passages do not contain the necessary information to answer the question, please reply with: "Unable to answer based on given passages."
DATA-TO-TEXT WRITING
Instruction: Write an objective overview about the following local business based only on the provided structured data in the JSON format. You should include details and cover the information mentioned in the customers’ review. The overview should be 100 - 200 words. Don’t make up information. Structured data: {json_data} Overview:
SUMMARIZATION

heading: C Annotation Details

heading: D Hallucination Detection Prompts
text: SUMMARIZATION
Below is the original news: {article} Below is a summary of the news: {summary} Your task is to determine whether the summary contains either or both of the following two types of hallucinations: 1. conflict: instances where the summary presents direct contraction or opposition to the original news; 2. baseless info: instances where the generated summary includes information which is not substantiated by or inferred from the original news. Then, compile the labeled hallucinated spans into a JSON dict, with a key "hallucination list" and its value is a list of hallucinated spans. If there exist potential hallucinations, the output should be in the following JSON format: {"hallucination list": [hallucination span1, hallucination span2, ...]}. Otherwise, leave the value as a empty list as following: {"hallucination list": []}. Output:
QUESTION ANSWERING
Below is a question: {question} Below are related passages: {passages} Below is an answer: {answer} Your task is to determine whether the answer contains either or both of the following two types of hallucinations: 1. conflict: instances where the answer presents direct contraction or opposition to the passages; 2. baseless info: instances where the answer includes information which is not substantiated by or inferred from the passages. Then, compile the labeled hallucinated spans into a JSON dict, with a key "hallucination list" and its value is a list of hallucinated spans. If there exist potential hallucinations, the output should be in the following JSON format: {"hallucination list": [hallucination span1, hallucination span2, ...]}. Otherwise, leave the value as a empty list as following: {"hallucination list": []}. Output:
DATA-TO-TEXT WRITING

heading: Future Work
text: RAGTruth, as a hallucination corpus with finegrained span labels, enables us to present experimental results for span-level detection, serving as a baseline for future research. As shown in Table 6, the overall performance of the current detection method is sub-optimal, highlighting the challenges in span-level detection. Even the advanced GPT-4-turbo tends to incorrectly classify many non-hallucinated contents with a low precision of 18. 4%. While our fine-tuned model shows im-
proved capability in identifying hallucinated spans by achieving an averaged f1 score of 52. 7%, it still falls short of perfect detection, emphasizing the inherent difficulties of this task. We also report the detection performance across four different types of hallucination spans. In the current stage, as we have not differentiated the types of detected hallucinations, we only report the char-level recall for different types of hallucinations. As indicated in Figure 5, the detection of evident hallucinations proves more effective compared to that of subtle hallucinations