Fine-grained Hallucination Detection and Editing for Language Models

Authors: abhika  mishra
akari  asai
vidhisha  balachandran
yizhong  wang
yulia  tsvetkov
graham  neubig
hannaneh  hajishirzi
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 14:10:45

Abstract: Large language models (LMs) are prone to generate factual errors, which are often called hallucinations. In this paper, we introduce a comprehensive taxonomy of hallucinations and argue that hallucinations manifest in diverse forms, each requiring varying degrees of careful assessments to verify factuality. We propose a novel task of automatic fine-grained hallucination detection and construct a new evaluation benchmark, FavaBench, that includes about one thousand fine-grained human judgments on three LM outputs across various domains. Our analysis reveals that ChatGPT and Llama2-Chat (70B, 7B) exhibit diverse types of hallucinations in the majority of their outputs in information-seeking scenarios. We train FAVA, a retrieval-augmented LM by carefully creating synthetic data to detect and correct fine-grained hallucinations. On our benchmark, our automatic and human evaluations show that FAVA significantly outperforms ChatGPT and GPT-4 on fine-grained hallucination detection, and edits suggested by FAVA improve the factuality of LM-generated text.

Main paper: 
 introduction: 
 Large language models (LMs; Brown et al. 2020 ) can generate highly fluent and plausible text. However, these models are prone to produce factually incorrect or unverifiable statements, often called hallucinations. This impedes their deployment in real-world applications for information-seeking contexts (Mallen et al., 2022; Asai et al., 2024) . Prior work on hallucinations in natural language generation (NLG) often assumes the presence of a specific Figure 2 : An overview of our fine-grained hallucination taxonomy. We identify 6 finegrained types representing diverse hallucinations in LM-generated text.reference text and focuses on studying faithfulness to the references (Ji et al., 2023) . On the contrary, escalating apprehensions have been articulated about LM generations that are not grounded in any specific source text, but rather in world knowledge (Zhang et al., 2023) .Several recent work studies automatic hallucination detection (Min et al., 2023) or editing outputs (Gao et al., 2022) to address such LM hallucinations. These systems typically categorize hallucinations into simplistic binary distinctions like factual or not factual (Figure 1 right) or mainly focus on entity-level errors. We argue that hallucinations manifest in diverse forms, each requiring varying degrees of careful assessments to verify factuality. Entitylevel contradictions are usually evident and can be easily rectified with a single reference. Conversely, errors involving fabricated entities (e.g., The Messi Diaries in Figure 1 ) demand thorough verification across multiple sources. This underscores the need for a more finegrained approach to detect hallucinations.In this work, we propose automatic fine-grained hallucination detection (Figure 1 left), a new task requiring a system to provide precise identification of hallucination sequences, discern different types based on a taxonomy, and suggest refinements. We focus on hallucinations in information-seeking scenarios, where grounding to world knowledge matters. Our taxonomy (Figure 2 ) hierarchically classifies hallucinations in LM generations into six categories. Our taxonomy is based on iterative pilot studies with NLP experts, to ensure comprehensive coverage of all significant types of hallucinations.We construct a new fine-grained hallucination benchmark, FAVABENCH, by carefully annotating approximately 1,000 responses of three widely used LMs (Llama2-Chat 7B, 70B and ChatGPT2 ) to diverse knowledge-intensive queries. Each response is annotated at the span level to identify hallucinations, including erroneous subspace, types, and potential refinements. Our analysis reveals all models include at least one hallucination in the majority of their responses (e.g., 70.2% in Llama2 7B and 59.8% in ChatGPT). Besides the widely-studied entity-level errors, other types like unverifiable sentences make up over 60% of LM-generated hallucinations, highlighting the urgent need for fine-grained detection.Yet, prior methods often only provide statement-level binary hallucination predictions. To advance fine-grained hallucination detection, we introduce FAVA, a new retrievalaugmented LM that can identify and mark hallucinations at span-level using a unified syntax (Figure 1 , left). We design an LM-based synthetic data generation and train FAVA on the 35k resulting instances. We compare FAVA with state-of-the-art LMs on fine-grained hallucination detection based on FAVABENCH. FAVA significantly outperforms ChatGPT with and without external knowledge by 23.7% on fine-grained hallucination detection. On binary hallucination detection, FAVA outperforms a widely-used system, FActScore (Min et al., 2023) in addition to other strong baselines leveraging GPT4 or ChatGPT. FAVA also effectively corrects hallucinations in diverse LM outputs-FAVA editing improves a factuality score (Min et al., 2023) of Alapaca 7, 13B, and ChatGPT by 4.4, 9.3 and 3.3%, respectively. When free, Messi enjoys singing songs for his family. 9.6% 6.3% 6.3%  
related work: 
 Hallucinations in NLG. Prior taxonomies proposed for summarization (Pagnoni et al., 2021 ), text simplifications (Devaraj et al., 2022) or knowledge-grounded dialogue (Dziri et al., 2022) often assume the existence of specific source text and focus on faithfulness to the source (Ji et al., 2023) . As this is qualitatively different from LMs' factual hallucinations grounded in world knowledge, we extend prior work and build a new taxonomy for hallucinations in LM outputs.Detecting and editing hallucinations in LMs. Several recent or concurrent studies propose methods to predict factuality by giving binary labels of a statement being factual or not (Manakul et al., 2023; Min et al., 2023) , focusing on entity-level hallucination editing (Gao et al., 2022; Chen et al., 2023) , or concentrating purely on detection (Li et al., 2024) . Recent surveys have attempted to categorize hallucinations in LM generations, often stretching the definition of hallucinations to broad error types in LM-generated text, which cannot be traced to specific spans in text (Huang et al., 2023; Zhang et al., 2023; Rawte et al., 2023) .We propose a fine-grained taxonomy for hallucinations in long-form text generation for information-seeking context to improve factual consistency. While prior work often develops binary factuality verification systems on top of proprietary LMs (Chern et al., 2023) , we train a fine-grained detection system using our carefully designed data generation pipeline.Fact verification for human-written claims. While research on hallucination detection focuses on identifying errors in model-generated text, a related area of research focuses on identifying factual inaccuracies in human written claims (Bekoulis et al., 2021) . Many large-scale datasets have been proposed for fact verification given Wikipedia (Thorne et al., 2018) , scientific articles (Wadden et al., 2020) or news documents (Wang, 2017) . Several systems have been developed for fact-checking in these settings (Thorne & Vlachos, 2018; Schuster et al., 2021; Nakov et al., 2021) , but have not been tested for LM-generated text. While our work focuses on hallucination detection in LM generations, FAVA can be adapted to fact-check human written claims. 
fine-grained hallucination detection: 
 Focus and definitions.We focus on open-ended text generation given information-seeking queries which often require factual knowledge. We define hallucinations as factual errors or unverified statements given external world knowledge. We operationalize world knowledge as documents from the web that are most relevant to the given statement/query according to a search algorithm.3 Based on our definition of hallucinations, we build a hierarchical taxonomy to categorize them into fine-grained error types. Inspired by prior task-specific taxonomies (Pagnoni et al., 2021; Devaraj et al., 2022) , we introduce new categories to describe more complex errors surfacing in LM generations. We conducted a pilot annotation with nine NLP experts to ensure good coverage across diverse error types. Figure 2 shows our taxonomies to classify LM hallucinations. (2) Invented : Invented errors refer to statements where the LM generates an entirely fabricated entity that doesn't exist based on world knowledge. Fictional entities in creative work aren't included.(3) Subjective : Subjective errors refer to expressions about existing entities that lack universal validity. These statements often do not contain facts and are influenced by personal beliefs or opinions.(4) Unverifiable : These are statements where the LM output contains facts, but no retrieved evidence from the web can directly support or contradict the fact (e.g., private details).4 5 While Entity or Relation are often phrase-level and can be fixed by minimal editing erroneous phrases, other error types can be an entire sentence or part of a sentence and should be removed from a response to make it precisely factual.We introduce the two tasks of identifying and editing fine-grained factual errors in LM outputs. Given an input query x and a corresponding LM output y, our tasks require systems to identify all of the factual errors in y. Each error e consists of (e text , e type ), indicating the factually incorrect text spans and their error types among our taxonomies, respectively. We evaluate systems' abilities concerning identifying fine-grained error types in the modelgenerated text e type (Task 1; Figure 1 < Entity >) and editing factual errors e text accordingly (Task 2; Figure 1 Barcelona → Miami).Task 1: Fine-grained hallucination detection. In this task, the system is expected to identify fine-grained errors in an LM output. Due to the subjectivity of span-level annotations, for automatic evaluation, we evaluate systems' abilities to detect whether an error type t exists in a sentence s i ∈ y. Given an output y consisting of L sentences, we assume the availability of ground-truth error type annotations e * t i ∈ {TRUE, FALSE}, which is a binary label of an error type t existing in the ith sentence (TRUE) or not (FALSE). Following the fact verification literature (Thorne et al., 2018; Schuster et al., 2021; Feng et al., 2023) of computing precision and recall, we measure those metrics while we extend them for each error type. For each type, a system predicts e t i and we evaluate precision and recall as:EQUATIONPrecision indicates the proportion of how many of the model's predictions of an error type t existing in the ith sentence is correct, while recall indicates how many of the error sentence TRUE is identified by the model. For final score, we compute the F1 scores averaged over six error types as follows: 1|E | ∑ t∈E 2 × Prec t × Recall t Prec t + Recall t (2)Fine-grained detection can be simplified into a binary classification task, in which a system predicts if a sentence s i includes any factual errors or not.6 Task 2: Hallucination editing. Some hallucinations can be fixed with minimal span-level edits, while others need to be removed or marked as unverifiable. In the editing task, the system is expected to suggest fine-grained edits to improve the factuality of the LM output. We evaluate a system's ability to improve the factuality of given output y, by comparing scores estimated by off-the-shelf systems, such as FActScore (Min et al., 2023) , as:f ( ŷ) -f (y), where f indicates an estimated factuality scores and ŷ indicates edited output. 
benchmark: favabench: 
 FAVABENCH consists of around 1k fine-grained annotations on three LM responses to queries in multiple domains.Source prompts. The source prompts include a collection of 200 information-seeking queries, spanning four different data sources. See examples in Appendix Table 6 .• 50 Knowledge-intensive queries from Open Assistant (K öpf et al., 2023) , by prompting GPT-4 to judge whether each query from the dataset requires world knowledge. • 50 Open QA prompts from the No Robots dataset (Rajani et al., 2023) .• 50 instructions requiring more reasoning and knowledge by the authors.• 50 synthetically-created prompts that require more fine-grained knowledge, by converting data-to-text WebNLG dataset (Gardent et al., 2017 ) using a template.Annotation details and quality. We obtain responses to the collected source prompts using ChatGPT (gpt-3. errors. Yet, there are diverse types of errors prevalent like invented statements or contradictory statements, which make up 14.1% and 18.1% of the detected errors, respectively. The error distributions vary across different source prompts and LMs. Invented are more common in Llama2-Chat models. Fewer errors in No Robots subsets than other subsets may be because their seed prompts are less knowledge-intensive, or ask about popular factual knowledge (e.g., How long was the Revolutionary War?), which is often memorized by LMs (Mallen et al., 2022) . 
model: fava: 
 We introduce FAVA (FAct Vericaton with Augmentation), a model for fine-grained hallucinations detections and editing. FAVA is trained on high-quality synthetic training data to identify hallucinations, incorporating retrieved knowledge. FAVA consists of two components: a retriever M ret and an editing LM M edit . M ret takes the original output LM y and optionally input prompt x (if applicable), and retrieves top relevant documents: C = M ret (x, y). Subsequently, the editing model detects and, if possible, edits factual errors in y given the retrieved context: ŷ = M edit (x, y, C). ŷ is an augmented output y interleaved by the error edits with hallucination types as shown in Figure 1 . While M edit can be any LM, in our preliminary experiments, we find that making a state-of-the-art proprietary LM such as ChatGPT to perform fine-grained editing via prompting only is challenging (Table 2 ).Reliance on such models also hurts reproducibility. Therefore, we generate high-quality synthetic training data with minimal human efforts (Section 5.1) and fine-tune a smaller yet powerful LM on the generated data (Section 5.2).To train our M edit model, we require a large number of training instances, each of which includes (c, y, y * ), where c is the gold context, y is an erroneous LM output, and y * is an output with error tags and correct editing based on c. FAVA learns to take (c, y) and generate y * . Inspired by prior work that leverages LMs to generate synthetic training data (Balachandran et al., 2022; Wang et al., 2023b; Asai et al., 2023) , we introduce a new data creation method tailored to our fine-grained hallucination taxonomies. Specifically, our data creation pipeline (refer Figure 4 ) consists of three steps:Step 1: seed passage generations that generates a large, diverse collection of seed passages c, Step 2: error insertions which inserts factually incorrect statements to generate erronous passages y, andStep 3: post-processing which curates the final training examples.Seed passage generation. The goal in this step is to collect and generate a large, diverse range of seed passages to ensure FAVA can handle a wide range of text types while maintaining control over factuality. Later on, errors are inserted into these seed passages to generate erroneous text, y. We base our seed passage collection on Wikipedia articles and sampled 5k QA pairs from Natural Questions (Kwiatkowski et al., 2019) . We paraphrase Wikipedia passages into different genres to enable FAVA to be adaptable and effective across various textual formats. Specifically, we randomly sample 35,074 Wikipedia articles c and use them as our gold reference passages. We then randomly sample one of the pre-specified genre types (e.g. blog article, tweet, etc.) for each passage and use ChatGPT to paraphrase the original passage into the given genre.8 Error insertion. The goal of this step is to simulate erroneous LM generation y, which are passages with factual errors. To control error distributions in the training data and incorporate our hallucination taxonomy, we prompt LMs to insert errors from our taxonomy one by one. We use ChatGPT and GPT-4 interchangeably for six different types.9 See detailed discussions in Appendix B. Given an instruction and few-shot demonstrations of an error type e type , GPT-4 or ChatGPT inserts new errors while retaining previously inserted errors. Models mark phrases or sentences for deletion along with their error type and insert phrases and sentences with insertion tags. Now we have an erroneous text y, which inserts multiple errors into the original text t.Post-processing. We then post-process data, by swapping and removing the error tags and edits to form a clean erroneous text y as the input and use the original correct passage y * as the output with edits, for training M edit . We filter out the examples violating our editing rules (e.g., inserting errors without marking them up with tags and error types). We also retrieve additional four relevant Wikipedia paragraphs using Contriever-MSMARCO (Izacard et al., 2022) , and randomly mix the order of the references, forming the final references C.Statistics of data and human evaluation. After analyzing ablations with varying training instance sizes (Section 7.2), we generated 35,074 training instances, 10 30,0074 based on Wikipedia passages, and 5,000 based on QA pairs. All of the error types are almost equally distributed (∼15 % each) with 3.1 errors inserted for Wikipedia subsets, and 1.4 errors inserted for the QA pairs on average. We present the error distributions in Appendix Table 10 . We conduct human evaluations on randomly sampled 50 examples and find that the generated data meets the expected criterion in most cases. See details in Appendix C.4.We train a smaller LM on the generated large-scale training data. In particular, given a training instance (C, y, y * ), our model M edit takes (C, y) as input and learns to predict the edited outputs with tags to represent error type y * using the standard language modeling objective. We use Llama2-Chat 7B to initialize M edit . We empirically found initializing FAVA with this model gives slightly better performance than the pre-trained Llama2. At inference time, we retrieve the top five documents from Wikipedia, 11 using Contriever-MSMARCO, and input them together with an LM output that may include hallucinations. The model identifies hallucinations, marks phrases or sentences for deletion, and suggests edits. 
experiments for hallucination detection: 
 Evaluation data. We use our new benchmark, consisting of 902 annotated passages, 12 as our test data. We measure the models' detection performance based on sentence-level per-category classification task as formulated in Section 3.2. We evaluate both fine-grained detection and binary detection settings. Note that fine-grained hallucination detection is our new proposed task, as prior work mostly focuses on the binary detection setting. Baselines. Fine-grained hallucination is a new task and we test three baselines that use state-of-the-art proprietary LMs: ChatGPT prompts ChatGPT (gpt-3.5-turbo-0301) with a carefully designed prompt describing all six categories with two demonstrations. GPT4 (gpt-4) does the exact same but prompts GPT4 instead. We were unable to include GPT4 with retrieval in our baseline due to high costs. Since FAVA uses retrieval augmentation, for comparability we include Rt+ChatGPT. This baseline prompts ChatGPT using the same prompt and demonstrations but also includes the top five retrieved documents by Contriever at test time to augment the original prompt. 13 We additionally evaluate FActScore (Min et al., 2023) for the binary detection task only. This model-based metric prompts ChatGPT and GPT 3 (davinci-003) to decompose a response into a set of atomic facts and verify factuality for each using passages from a designated Wikipedia article.Evaluation data. For editing evaluations, we use open-ended text-generation data and evaluate models' factuality based on a metric designed to measure the factuality for the target task. In particular, we use the biography generation task proposed by FActScore (Min et al., 2023) to evaluate the effectiveness of editing to reduce hallucinations based on edited outputs' FActScore results.For editing experiments, we use ChatGPT and Rt-ChatGPT, as well as Llama2-Chat 13B with and without retrieval (Llama and Rt-Llama, respectively), and prompt them with carefully designed prompts and two demonstrations. For all baselines, we retrieve five documents consisting of one document retrieved by string matching based on the entity name and four documents retrieved by Contriever, which are reranked at test time, using a small cross-encoder.14 Our automatic evaluations may not fully capture the models' abilities to detect and refine hallucinations, due to the potential subjectivity of our annotations or the performance of the factuality evaluation metric. We evaluate randomly sampled 50 outputs from FAVA as well as the baseline with the highest automatic evaluation score, namely Rt-ChatGPT. We ask human annotators to verify how many of the detection and edits are indeed correct based on the provided retrieved documents. This is similar to our automatic precision evaluation, but instead of coarsely evaluating detection performance at the sentence level, we evaluate the model performance at each detection level. 
results: 
 Fing-grained detection results. Table 2 shows the fine-grained detection accuracy, namely F1 scores, and binary prediction F1 of FAVA and baselines. We provide the full precision and recall scores in Appendix Tables 13 and 14 , respectively. FAVA significantly outperforms ChatGPT, Ret-ChatGPT, or GPT-4 on both fine-grained error detection and binary error On the other hand, its performance on Invented and Unverifiable is still limited. Those two error types often require intensive search over many documents beyond the top few ones, while FAVA by default only considers the top five documents. Table 3 shows the binary hallucination detection performance of FAVA, baselines models, and FActScore (FAct). FAVA outperforms all other models in detecting hallucinations by a large margin.Editing results. Table 4 shows the results of the editing task on the biography generation task. Our FAVA significantly outperforms prompted ChatGPT or Llama2-Chat 70B, despite being a much smaller model in size. We found that ChatGPT is often confused with different errors and tends to over-predict contradictory or subjective statements. FAVA shows the largest gains in FActScore, showing the effectiveness of editing and error type detection.Human evaluation results. Our human evaluation results are shown in Appendix Table 16 . FAVA shows significantly better performance than Rt+ChatGPT on both editing and detection scoring 46.3% and 31.8% more than Rt+ChatGPT on each task respectively. Furthermore, FAVA recognizes more errors, 2.4 errors per passage, than retrieval-augmented ChatGPT, 1.9 errors per passage. These results further demonstrate the strong capabilities of FAVA detecting factual errors in LM outputs.Effects of data scale. We assess our model on varying sizes of synthetic training data (10k, 20k, and 30k instances) and examine their performance on fine-grained hallucination detection. Appendix Figure 7 demonstrates that FAVA variants with a larger number of training instances perform significantly better at detecting fine-grained errors.We assess FAVA's editing performance on Alpaca 13B outputs with varying retrieved evidence: top 1 document; top 5 documents; reordering the top 5 documents; top 4 documents and mix them with an introductory paragraph of the target entity (entity matching). Table 5 shows the experimental results. More detailed results are in Appendix Table 18 . We found that retrieving the top five documents significantly enhances performance compared to retrieval of only the top one document by 4.5%. Ordering passages using a reranking model gives some improvements, which indicates the non-negligible effect of context order, as reported by prior work (Liu et al., 2023) . Including the reference based on entity matching gives a notable enhancement to 50.1%, marking a 3.1% gain from the top five retrievals and 7.6% improvement over the unedited generations. These results indicate although FAVA demonstrates strong capabilities, there's room for improvement in retrieving and incorporating many references. 
conclusions: 
 We introduce a new task of automatic hallucination detection, built upon our newly introduced taxonomy, hierarchically classifying hallucinations in LMs into six categories. We collect the first human-annotated fine-grained hallucination benchmark, and introduce a new retrieval-augmented LM for fine-grained error editing data following our taxonomies.Empirical results show that FAVA significantly outperforms strong baselines by a large margin on both editing and fine-grained detection tasks, while still large room for improvements for automated fine-grained error detection and editing. Examples of annotated results. Table 7 shows the fine grained hallucination distribution across our annotations.  
 
 Appendix: Quality of error insertions. In our pilot studies, we found that asking ChatGPT to insert multiple error types at the same time easily makes the model misunderstand or get confused with different error types (e.g., swapping or incorrectly identifying an error type). We also found while ChatGPT is capable of generating more simple types of errors such as Entity , it struggles with generating plausible and difficult perturbations for more nuanced types.Text diversification results. Table 9 shows the diversified text results.Overall Detection Score Training Instance Size (2) quality-the inserted errors are feasible and realistic and notably different from provided few-shot samples. We ask human annotators to score each category for each passage either 0, 1, or 2 (higher is better).Our analysis revealed that the data generated by our system with iterative insertion obtained an average score of 1.66 for validity assessment and 1.36 for quality assessment. Meanwhile, data generated by one-shot prompting scored an average of 1.1 for validity and 0.9 for quality. Our human evaluation reveals that our LM-generated edits are indeed of higher quality than the one-shot data creation, and provide realistic factual errors.Table 17 shows examples of generated erroneous data from our synthetic data creation pipeline. We ask human annotators to score each category for each passage either 0, 1, or 2. These scores were defined as follows: a score of 0 indicated that less than half of the inserted errors aligned with the given criteria, a score of signified that at least half of the errors met the established criteria, and a score of 2 indicated that all inserted errors met the criteria. As a comparison, we also generate training data with one-shot generation.Retrieval ablations. Table 18 shows the full results of the retrieval ablations. We enhance our retrieval process by strategically reordering the retrieval results. This involves rearranging the top 5 passages from our reranked Contriever documents and entity matching document such that the documents with the highest relevance are positioned closest to the text requiring verification. Reordering passages gives 1.7% improvements, further emphasizing the importance of retrieval and careful pipeline design.Analysis on data scaling. We conduct ablations of different data scaling in FAVA. Figure 7 shows the scaling results. <subjective> The game is a masterpiece that revolutionized the arcade industry and will always be remembered as a classic.</subjective>The knapsack problem is a problem in combinatorial optimization where the goal is to determine the optimal selection of items to maximize the total value within a given weight constraint.The knapsack problem is a problem in <entity> <delete> combinatorial optimization</delete> <mark> sequential search</mark> </entity> where the goal is to determine the optimal selection of items to maximize the total <entity> <delete> value</delete> <mark> weight</mark> </entity> within a given weight constraint. Our code, data, and demo are available at https://fine-grained-hallucination.github.io/.We use gpt-3.5-turbo-0301 throughout this work.Errors in common sense, numerical, or logical reasoning are out of the scope of this work.Differing from Subjective errors which focus on opinionated statements lacking facts, Unverifiable specifically focuses on fact-based statements that just lack evidence.5 While Invented and Unverifiable are highly related, Invented are the hallucinations where we can verify that some core entities or subjects of the sentences don't exist.This is similar to FActScore(Min et al., 2023), while we predict and evaluate factuality at the sentence level without extracting atomic facts. We compare FAVA against FActScore on this task.Annotating each response takes approximately 10 minutes, and we pay USD 3.5 for annotation.See the full list of the genres, instructions, and final ChatGPT-generated text in Appendix Table9 We use ChatGPT (gpt-3.5-turbo-0301) for the Type 1 and Type 3 while using GPT-4 for other types.10 While we see performance improvements up to 30k, the API costs for training data generation were already $3k, reaching our initial budget.11 We use English Wikipedia data from January 2023 and generate embeddings using the Contriever.12 We collected 1010 annotated passages including our experimental batches, excluding earlier batches for evaluations.While we also tested strong white box LMs including Llama2-Chat 13B, we found their predictions often show confusion among different categories or violate formats.https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-6-v2https://github.com/allenai/open-instruct/blob/main/scripts/finetune with accelerate. sh 


Decision: This paper introduces a new benchmark, FavaBench, for detecting fine-grained hallucinations in large language model outputs. The authors propose a novel taxonomy for hallucinations and develop a retrieval-augmented model (FAVA) to detect and correct these errors. Evaluation results indicate that FAVA significantly outperforms several frontier LLMs in hallucination detection and correction tasks. A reviewer raised some criticisms regarding its literature review and comparisons with concurrent work -- I personally really liked this work, and I believe it significantly improves our ability to understand and mitigate the impact of hallucinations in LLM-generated texts.

Reviews: review: paper_summary: paper_summary: This paper describes an approach to detect and edit fine-grained hallucinations. Several fine-grained hallucination types are defined. A strategy is proposed to generate hallucinations by prompting LLMs. A model is trained on generated data containing references-hallucination-correct triples. The model is asked to detect hallucinations and to edit hallucinations to produce the correct text.
Experiments are done on a set of cases. The method is compare to the ones based on LLMs, and shown to outperform the latter.
strength_weakness: strength_weakness: Although the authors claim the types of fine-grained hallucination types are new, they are not very different from what others have proposed. The main categories are similar.
The details for training the model based on the generated data are missing.
The baselines considered in the paper are simple. In the literature, other approaches have been developed (e.g. checking factuality, fine-tuning a model. etc.). These models should also be included a baselines.
questions: questions: Fig. 2: the tests used in the figure to determine types of hallucination are difficult to follow. For example, from the question "Does the information contradict with evidence?", there are two yes outputs, and from the "Is it verifiable based off web knowledge?" test, there is only one no output. The criteria to determine the types are confusing.

While it is interesting to include more (fine-grained) types of hallucination, it is unclear if "subjective" should be included as hallucination. The risk of considering it as hallucination is that many utterances in everyday dialogues (e.g. "I think ...") would be considered hallucinations. This does not make this type of hallucination very interesting.  What is the reason to include this type of hallucination?

On the other hand some other types of hallucination may require broad knowledge on the topic (invented, unverifiable). Looking at the approach, it seems that only 5 reference documents are included to cover the topic. This is not sufficient to verify if a statement is invented or verifiable. To detect these types of hallucination, more references (or larger models) should be considered. Therefore, the experiments done in the paper may not reflect the difficulty to detect these types of hallucination. Have you looked at the coverage of the 5 references on the topic? Are there cases where the statement is unverifiable with the 5 reference texts, but is verifiable with more reference texts?

The type "invented" is interesting. In addition to require broad knowledge to detect this type of hallucination, the generated data should also try to generate statement that are close to the topic. Looking at the examples, it seems that the "invented" hallucinations are randomly generated. This makes it artificially easy to detect, by checking if the topic of the statement is related to the remaining of the text. Have you tested different ways to generate invented hallucinations?

Table 4: I guess the numbers are FActScore. Please specify.

For hallucination editing, I believe the Human evaluation results are more meaningful than FActScore. It mat be appropriate to move Human evaluation results to the main body of the paper (instead of appendix).

The experiments only compare with simple LLM-based methods. There are more methods proposed in the literature. The paper mentions several ones in the related work. Another strongly related work is https://arxiv.org/pdf/2401.03205, which also detects hallucination cases and uses different methods to mitigate hallucinations, such as fine-tuning, different promptings, RLHF, etc. It is appropriate to compare with these approaches.
paper_summary: paper_summary: The authors introduce a comprehensive taxonomy that categorizes hallucinations into six detailed types and develop a novel fine-grained hallucination detection task. They also present FAVABENCH, an innovative benchmark for evaluating LM outputs with approximately 1,000 fine-grained human annotations. Leveraging this benchmark, the authors develop FAVA, a retrieval-augmented LM trained on synthetic data designed to detect and correct hallucinations at the span level. The evaluations demonstrate that FAVA significantly outperforms existing systems in both fine-grained and binary hallucination detection, thereby providing a robust solution for enhancing the factuality and reliability of LM-generated text.
strength_weakness: strength_weakness: 1. LLMs are generally trained for broad applications, yet the proposed task demands detailed, fine-grained operations. Therefore, it is plausible that a smaller, task-specific LLM might outperform its more generic counterparts. However, the assertion that smaller LLMs exhibit remarkable performance over models like GPT and Llama is not definitive. If these larger models were trained using the same specialized dataset, one might anticipate even more significant improvements than the smaller, task-tailored models.

2. The authors dedicate substantial space to discussing the proposed task and datasets; however, the description of the experimental settings, especially in Section 7, lacks clarity. The absence of detailed experimental setup information for Table 4 complicates the distinction between the generators used in Tables 3 and 4. This ambiguity significantly delays comprehension of the comparative analysis. I recommend that the authors refine the description of the experimental setup.
paper_summary: paper_summary: This paper proposes a new taxonomy for hallucinations and introduces a new fine-grained hallucination detection and editing task based on it. To evaluate this task, the authors provided FAVABENCH, a manually annotated benchmark. Additionally, instead of using existing LLMs for hallucination detection and editing, the authors also fine-tuned a retrieval-augmented LM called FAVA, which outperformed existing LLMs in hallucination detection and editing. The training of the FAVA model was conducted through synthetic data generation, which also follows the proposed taxonomy.

The paper is well-written, and the included figures significantly aid understanding. The topic of LLM hallucinations is also timely and appropriate, and the models and datasets provided by the authors would be beneficial to the community. However, apart from providing a taxonomy, there is no notable technological impact.
strength_weakness: strength_weakness: I have some concerns about the appropriateness of the taxonomy proposed by the authors:

1. Entity and Relation errors: wouldn't it be possible that relation errors could also be resolved by replacing entities? Additionally, according to Figure 3, it seems that relation errors make up a very small percentage. Do relation errors introduce any unique impacts?
2. Ambiguity between Invented and Unverifiable: The difference between 'invented' and 'unverifiable' seems ambiguous. In Table 1's examples, 'invented' could also occur due to a lack of evidence. The authors state, "Invented are the hallucinations where we can verify that some core entities or subjects of the sentences don’t exist," but how can we be sure that these aren't issues with the retrieval stage?
- Regarding 2, could you provide the agreement scores among human annotators specifically for 'invented' and 'unverifiable' errors? It appears that the currently submitted agreement scores do not specify this for each specific type.
questions: questions: - If Task 1, Fine-grained hallucination detection, involves binary classification for each error type, it's possible that a sentence might contain multiple error types. Then, for Task 2, Hallucination editing, how is editing performed according to each error type? Are there issues such as overlapping span boundaries in each edition?
- In fine-grained hallucination detection, does the process involve separate binary classifications for each error type, or can the classification of one error type influence another (e.g., multi-label classification)?
- Could you provide more detailed information about the third item on the list of source prompts for FAVABENCH, which states, "50 instructions requiring more reasoning and knowledge by the authors"?
paper_summary: paper_summary: This paper introduces a taxonomy of hallucinations in large language models (LMs) and proposes a novel task of automatic fine-grained hallucination detection. The authors construct an evaluation benchmark, FavaBench, with over 1,000 human judgments on LM outputs.  The authors then train FAVA, a retrieval-augmented LM, to detect and correct fine-grained hallucinations. The results show that FAVA significantly outperforms ChatGPT and GPT-4 on fine-grained hallucination detection and improves the factuality of LM-generated text.
strength_weakness: strength_weakness: (1) The authors make a very poor literature about LLM hallucination. Just list two highly related studies

-The dawn after the dark: An empirical study on factuality hallucination in large language models

Both papers are similar in many aspects: similar taxonomy for hallucination, detection baseline, and metrics. 

-Halueval: A large-scale hallucination evaluation benchmark for large language models

Both papers are similar in many aspects: task, detection baseline, and result findings. 

In addition, there are also several hallucination dection methods that are not compared by this work. Thus, I find that the literature is highly incomplete.

(2) The method to create synthetic data for hallucination detection, while a side effect is that LLM-generated errors may not be similar to the actual genrations of LLMs. That is, the detection method may only learn some shortcut patterns to detect hallucinations.

Comments: title: reaction to author's rebuttal
Follow-Up on Rebuttal Response
Follow-Up on Rebuttal Response
comment: It is fine to not include a comparison with the related woik [1] according to ACL policy.
How about comparing with other methods that check for factuality? This is very similar to your verification using retrieved documents.
The new categories are slightly different from the existing ones. The newly introduced categories have the problems mentioned in the review. In the rebuttal, the authors said that "subjective" depend on factuality. Then what difference can be made between "subjective" and "unverifiable"? The justification of the new categories seems weak and they are not well defined.
Despite the explanation in the rebuttal, the main problems mentioned in the review are still there and not solved.
The Cohen-Kappa agreement scores for "unverifiable" are quite low, which could negatively affect the reasonability of the taxonomy. Have you analyzed the reasons for these low scores?
Dear Reviewer,

Thank you for providing comments on our paper. Hopefully, our rebuttal addresses your concerns, and we are happy to answer any further questions you may have.

In summary:

- Our taxonomy is inspired by, yet substantially different from, prior studies. For example, "invented" or "subjective" are often grouped as "unverifiable." Our careful human annotations on 1k LM responses in multiple domains show that most hallucinations can be classified into one of these categories.
- Regarding the baselines, we indeed compare our method with other state-of-the-art methods discussed in related work, such as FActScore, which first extracts a set of atomic statements and validate each of them using retrieved passages, and is widely used for hallucination detaction. Please note that the suggested paper [1] was arXived on January 6, 2024, which is considered concurrent work according to the NLP/ML conferences' reviewing protocol (e.g., ACL review policies [2]), and comparison of such work is not required.
- We will include more details of our training in the final version of the paper.


[1] Junyi Li et al., The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models. Arxiv 2024.

[2] ACL Policies for Review and Citation https://www.aclweb.org/adminwiki/index.php/ACL_Policies_for_Review_and_Citation 
> For comparison, papers (whether refereed or not) appearing less than 3 months before the submission deadline should be considered contemporaneous to the submission. This relieves authors from the obligation to make detailed comparisons that require additional experimentation and/or in-depth analysis
Dear Reviewer, 

Thank you for providing comments on our paper. We believe our rebuttal addresses your concerns, and we are happy to answer any further questions you may have.

In our rebuttal, we provide a detailed discussion on the differences between this work and the two suggested papers, in terms of task setup, baselines, taxonomy, and the proposed method, and we believe our work and suggested papers are substantially different. We will add the suggested two papers to the Related Work of our final version.  

Please note that [1] was arXived on January 6, 2024, which is considered concurrent work according to the NLP/ML conferences' reviewing protocol e.g., [ACL review policies](https://www.aclweb.org/adminwiki/index.php/ACL_Policies_for_Review_and_Citation), and comparison of such work is not required 

> For comparison, papers (whether refereed or not) appearing less than 3 months before the submission deadline should be considered contemporaneous to the submission. This relieves authors from the obligation to make detailed comparisons that require additional experimentation and/or in-depth analysis

[1] Junyi Li et al., The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models. Arxiv 2024.



We also addressed your concern regarding the quality of synthetically generated data in our rebuttal, and we will move the details of our human evaluations in Appendix to the main pages in the final version.

Thank you once again for your time and consideration. Please let us know if you have any follow-up concerns or questions.
We apologize for overlooking that question in our response. We have calculated the cohen-kappa agreement scores for both types, getting 0.356 for 'unverifiable' and 0.636 for 'invented' across all our annotations. We will include a detailed breakdown of the agreement scores for each hallucination type in our camera-ready version.
Thank you for your reply but I think you missed one of my questions.

In Reasons To Reject:

"Regarding 2, could you provide the agreement scores among human annotators specifically for 'invented' and 'unverifiable' errors? It appears that the currently submitted agreement scores do not specify this for each specific type."

Would you be able to provide the scores?