Fine-grained Hallucination Detection and Editing for Language Models
Authors: Abhika Mishra
Akari Asai
Vidhisha Balachandran
Yizhong Wang
Graham Neubig
Yulia Tsvetkov
Hannaneh Hajishirzi
Venue: COLM (2024)
Keywords: Hallucination, Retrieval-augmented LMs
TL;DR: To better understand hallucinations in LM outputs, we propose a new task, fine-grained hallucination detections based on our carefully designed hallucination taxonomy and 1k human annotated data.

Abstract: Large language models (LMs) are prone to generate factual errors, which are often called hallucinations. In this paper, we introduce a comprehensive taxonomy of hallucinations and argue that hallucinations manifest in diverse forms, each requiring varying degrees of careful assessments to verify factuality. We propose a novel task of automatic fine-grained hallucination detection and construct a new evaluation benchmark, FavaBench, that includes about one thousand fine-grained human judgments on three LM outputs across various domains. Our analysis reveals that ChatGPT and Llama2-Chat (70B, 7B) exhibit diverse types of hallucinations in the majority of their outputs in information-seeking scenarios. We train FAVA, a retrieval-augmented LM by carefully creating synthetic data to detect and correct fine-grained hallucinations. On our benchmark, our automatic and human evaluations show that FAVA significantly outperforms ChatGPT and GPT-4 on fine-grained hallucination detection, and edits suggested by FAVA improve the factuality of LM-generated text.

Published as a conference paper at COLM 2024
Fine-grained Hallucination Detection and Editing for Language Models
Abhika Mishra', Akari Asai', Vidhisha Balachandran', Yizhong Wang*
Yulia Tsvetkov!, Graham Neubig", Hannaneh Hajishirzi!a
University of Washington
Carnegie Mellon University
Allen Institute for AI
labhikam, akari, yizzong,,,,, yuliats, hannaneh) @ecs washington.es edu, vbalacha, sreubig]ecs cm..cmu.ee. edu
Abstract
Large language models (LMs) are prone to generate factual errors, which are often called hallucinations. In this paper, we introduce comprehensive taxonomy of hallucinations and argue that hallucinations manifest in di- verse forms, each requiring varying degrees of careful assessments verify factuality. We propose a novel task of automatic fine-grained hallucina- tion detection and construct new evaluation benchmark, FAVA BENCH, that includes about one thousand fine-grained human judgments on three LM outputs across various domains. Our analysis reveals that ChatGPT and Llama2-Chat (70B, 7B) exhibit diverse types of hallucinations in the majority of their outputs information-seeking scenarios, highlighting the need to build fine-grained systems. To this end, we train FAVA, a powerful retrieval-augmented LM by carefully creating synthetic data to detect and correct fine-grained hallucinations. Our automatic and human evaluations show that FAVA significantly outperforms retrieval-augmented ChatGPT and GPT-4 fine-grained hallucination detection. Furthermore, FAVA out- performs widely-used hallucination detection systems on binary detection and shows effectiveness in editing to improve the factuality
1.M Output
References
Q
Input: LM Output References
Messi player Barcelona, known for the 2017 film "The Messi Diaries".
[1] Mensi is now a captain for the Major Leugue Soccer club Inter Miami.
[3] Documentaries about Messi MMssi, Lionel Messi: The Greatest...
FAVA
Fine-grained Hallucination Detection
Hallaciutionn Trass
Hinary (C (Coarse-grained) Hallucination Detection
Detection + Editing
Output: FAWA'
Fincegninal
Musé is player in
SinventedHHH known the Aim "Tae Mossi Dianes" S/ivvented>
Senttiy Maaai Miami
Conradittrry
Eaily
Unventiable
Rebsice
Hascolona. fllm blsee Diatiogll
Mesei player
Secualliy crons
No Cuther cuntext
Invented
Figure 1: Overview of our taxonomy, fine-grained hallucination detection task, and FAVA.
1 Introduction
Large language models (LMs; Brown et al. 2020) can generate highly fluent and plausible text. However, these models are prone to produce factually incorrect or unverifiable state- ments, often called hallucinations. This impedes their deployment real-world applications for information-sseeknng contexts (Mallen et al. 2022 Asai et al. 2024). Prior work on hal- lucinations natural language generation (NLG) often assumes the presence of a specific
Our code, data, and demo are available at http:p////fiieeergaadddallalaaaa
1o/

---

Published as a conference paper at COLM 2024
Yes
Is it verifiable based off web knowledge??
No
Does the information
Is it ubout an invented entity?
contrudict with evidence?
No (entailment)
Ven (contradiction
What the unit of error?
Is it fact based statement?
Facrual Statement
Type I(l Contradictory Eatley Level
Type I(b): Contradictory- Relationship
Type Me): Coutradictory Sentence Level
Type Invented Information
Type Subjective Statement
Type Unverifabble Statement
Factual crrors rdeletedrrepleed correctness)
Cannot be factually verifled (highlighted convey unverifiability)
An overview of our fine-grained hallucination taxonomy. We identify fine- grained types representing diverse hallucinations in LM-generated text.
Figure 2:
reference text and focuses on studying faithfulness to the references (Ji et al. contrary, escalating apprehensions have been articulated about LM generations that are not grounded in any specific source text, but rather in world knowledge (Zhang et al. 2023).
2023 On the
Several recent work studies automatic hallucination detection (Min et al., 2023 or editing outputs (Gao et al., 2022) to address such LM hallucinations. "These systems typically categorize hallucinations into simplistic binary distinctions like factual or not factual (Figure right) or mainly focus on entity-level errors. We argue that hallucinations manifest in diverse forms, each requiring varying degrees of careful assessments to verify factuality. Entity- level contradictions are usually evident and can be easily rectified with a single reference. Conversely, errors involving fabricated entities (e.g., The Messi Diaries in Figure [ demand thorough verification across multiple sources. This underscores the need for more fine- grained approach to detect hallucinations.
In this work, we propose automatic fine-grained hallucination detection (Figure left), new task requiring system to provide precise identification of hallucination sequences, discern different types based on a taxonomy, and suggest refinements. We focus on halluci- nations in inffommtiion-seeing scenarios, where grounding to world knowledge matters. Our taxonomy (Figure 2) hierarchically classifies hallucinations in LM generations into six categories. Our taxonomy is based on iterative pilot studies with NLP experts, to ensure comprehensive coverage of all significant types of hallucinations.
We construct a new fine-grained hallucination benchmark, FAVABENCH, by carefully an- notating approximately 1,000 responses of three widely used LMs (Llama2-Chat 7B, 70B and ChatGPT² to diverse knowledge-intensive queries. Each response is annotated at the span level to identify hallucinationn,, including erroneous subspace, types, and poten- tial refinements. Our analysis reveals all models include least one hallucination in the majority of their responses (e.g., 70.2% in Llama2 7B and 59.8% in ChatGP')). Besides the widely-studied entity-level errors, other types like unverifiable sentences make up over 60% of LM-generated hallucinations, highlighting the urgent need for fine-grained detection.
Yet, prior methods often only provide statement-level binary hallucination predictions. To advance fine-grained hallucination detection, we introduce FAVA, a new retrieval- augmented LM that can identify and mark hallucinations at span-level using a unified syntax (Figure [ left). We design an LM-based synthetic data generation and train FAVA on the 35k resulting instances. We compare FAVA with state-of-the-art LMS on fine-grained hallucination detection based on FAVABENCH. FAVA significantly outperforms ChatGPT with and without external knowledge by 23.7% on fine-grained hallucination detection. On binary hallucination detection, FAVA outperforms widely-used system, FActScore (Min) et al. 2023) in addition to other strong baselines leveraging GPT4 or ChatGPT. FAVA also et- fectively corrects hallucinations in diverse LM outputs-FAn FAVA editing improves factuality score (Min et al. 2023) of Alapaca 7, 13B, and ChatGPT by 4.4, 9.3 and 3,3%, respectively.
We 1ot-3.5--trbbr 0301 throughout this work.
2

---

Published as a conference paper at COLM 2024
Type
Example
Chat
L-7b
L-70b
Entity
Lionel Andrés Messi was born on June 4a 24, 1987.
42.7%
48.6%
55.7%
Relation
Lionel Messi sequived was acquired by PSG.
4.7%
3.4%
2.8%
Sentence
Messi never captain for Argentina football team.
18.9%
15.7%
19.9%
Invented
Messi is known for his famous airplane kick.
14.2%
17.4%
9.5%
Subjective
Lionel Messi is the best soccer player in the world.
9.9%
8.6%
5.8%
Unverifiable
When free, Messi enjoys singing songs for his family.
9.6%
6.3%
6.3%
Table 1: Distribution of different errors across ChatGPT (Chat), Llama2-Chat-7B (L-7B) and Llama2-Chat-70B (L-70B) outputs and examples of hallucinations for each type.
2
Related Work
Hallucinations in NLG. Prior taxonomies proposed for summarization (Pagnoni et al. 2021
text simplifications (Devaraj et al. 2022) or knowledge-grounded dialogue (Dziri et al. 20022
often assume the existence of specific source text and focus faithfulness to the source (ji
ett al., 2023). As this is qualitatively different from LMs' factual hallucinations grounded in
world knowledge, we extend prior work and build new taxonomy for hallucinations in LM outputs.
Detecting and editing hallucinations in LMs. Several recent or concurrent studies pro- pose methods to predict factuality by giving binary labels of a statement being factual or not Manakul al. 2023 Min et al. 2023), focusing on entity-level hallucination editing (Gao) et 2022 Chen et al. 2023), or concentrating purely on detection (Li et al. 2024). Recent surveys have attempted to categorize hallucinations in LM generations, often stretching the definition of hallucinations to broad error types in LM-generated text, which cannot be traced to specific spans text (Huang et al. 2023 Zhang et al., 2023 Rawte et al. 2023). We propose fine-grained taxonomy for hallucinations
long- forg text generation for information-seeking context to improve factual consistency. While prior work often devel- ops binary factuality verification systems top of proprietary LMs (Chern et al., 2023). train a fine-grained detection system using our carefully designed dala generation pipeline.
Fact verification for human-written claims. While research hallucination detection focuses on identifying errors in model-generated text, a related area of research focuses on identifying factual inaccuracies human written claims (Bekoulis al. 2021). Many large-scale datasets have been proposed for fact verification given Wikipedia Thorne et al. 2018), scientific articles (Wadden al.) 2020) or news documents (Wangl 2017). Several systems have been developed for fact-checking in these settings (Thoree & Vlachos 2018 Schuster et al. 2021 Nakov et al., 2021), but have not been tested for L.M-generated text. While our work locuses on hallucination detection in LM generations, FAVA can be adapted to fact-check human written claims.
3 Fine-grained Hallucination Detection
Focus and definitions. We focus open-ended text generation given information-seeking queries which often require factual knowledge. We define hallucinations as factual errors or unverified statements given external world knowledge. We operationalize world knowledge as documents from the web that are most relevant to the given statement/ query according to search algorithm
3.1 Hallucination Taxonomy
Based our definition of hallucinations, we build a hierarchical taxonomy categorize them into fine-grained error types. Inspired by prior task-specific taxonomies (Pagnoni et al. 2021 Devaraj et al. 2022), we introduce new categories to describe more complex
Frrors in common sense, numerical, logical reasoning are out of the scope of this work.
3

---

Published as a conference paper at COLM 2024
ChatGPT
1ama2-Chht 7B
Llama2-Chat 70B
no-robot-
mnivv
relution
instruction
contradictory
following
inverted
open
unvenaable
subjective
assistant
webnig:
100
200
300
400
500
100
200
300
400
100
200
300
400
500
Eroor Count
Error Count
Error Count
Figure Distribution of hallucination types in ChatGPT, Llama2-Chat-7B and Llama2-Chat- 70B outputs across four datasets of diverse information-seekingg queries.
errors surfacing in LM generations. We conducted a pilot annotation with nine NLP experts to ensure good coverage across diverse error types. Figure 2 shows our taxonomies to classify LM hallucinations.
T examples of each type of hallucination. We group hallucinations into two major categories: statements that contradict world knowledge (Type 1) and unverifiable statements (Types 2, 3, and 4).
(1a) Entity: Contradictory entity errors are sub-category within Type 1, where entity in a statement is incorrect and changing that single entity can make the entire sentence factually correct.
(1b) Relation: Contradictory relation errors are another sub-category within contradictory statements where semantic relationship (e-g,, verbs, prepositions, or adjectives) in statement is incorrect.
(1c) Sentence Contradictory sentence errors refer cases where full statement entirely contradicts relevant evidence from the web, and cannot be solved via phrase-level edits, (2) Invented : Invented errors refer to statements where the LM generates an entirely fabricated entity that doesn't exist based on world knowledge. Fictional entities in creative work aren't 't included.
(3) Subjective: Subjective errors refer to expressions about existing entities that lack universal validity. These statements often do not contain facts and are influenced by personal beliefs opinions.
(4) Unverifiable: These are statements where the LM output contains facts, but no retrieved evidence from the web can directly support or contradict the fact (e.g,, private details). 5 While Entity or Relation are often phrase-level and can be fixed by minimal editing erroneous phrases, other error types can be an entire sentence or part of sentence and should be removed from a response to make precisely factual.
3.2 Tasks and Metrics
We introduce the two tasks of identifying and editing fine-grained factual errors in LM outputs. Given an input query x and corresponding LM output y, our tasks require systems to identify all of the factual errors in y. Each error consists of (elext, _type), indicating the factually incorrect text spans and their error types among our taxonomies, respectively. We evaluate systems' abilities concerning identifying fine-grained error types the model- generated text ętype (Task 1; Figure 1 Entity >) and editing factual errors etext accordingly (Task 2; Figure Barcelona- + Miami).
Task 1: Fine-grained hallucination detection. In this task, the system is expected identify fine-grained errors in an LM output. Due to the subjectivity of span-level annotations, for automatic evaluation, we evaluate systems' abilities detect whether an error type exists in sentence s; y. Given an output y consisting of L sentences, assume the availability of ground-truth error type annotations exd E (TRUE, which is a binary label of an error type f existing the ith sentence (TRUE) or not (FALSE). Following the fact verification
*Differing from Subjective errors which focus on opinionated statements lacking facts, Unverifi- able specifically focuses on fact-based statements that just lack evidence. WWhile Invented and Unverifiable are highly related, Invented are the hallucinations where verify that some core entities subjects of the sentences don't exist.
4

---

Published as a conference paper at COLM 2024
literature (Thoree et al. 2018 Schuster et al. 2021 Feng et al. 2023 of computing precision and recall, we measure those metrics while we extend them for each enror type. For each type, a system predicts of and we evaluate precision and recall as:
Liel 1 [e] = c EiEL le; TRUE]'
LIEL 1/c] = =(") EiEL Llee" = TRUE]
Prec
Recall'
(1)
Precision indicates the proportion of how many of the model's predictions of an error type t existing in the ith sentence is correct, while TRUE is identified by the model. For final score, we compute the F1 scores averaged over six error types as follows:
indicates how many of the error sentence
1 2 Prec x Recall
(2)
3
FEE Prec' + Recall'
Fine-grained detection can be simplified into binary classification task, in which a system predicts if sentence 9; includes any factual errors or not
Task 2: Hallucination editing. Some hallucinations can be fixed with minimal span-level edits, while others need be removed or marked as unverifiable In the editing task, the system is expected to suggest fine-grained edits to improve the factuality of the LM output. We evaluate a system's ability to improve the factuality of given output y, by comparing scores estimated by off-the-shelf systems, such as FActScore (Min et al. 2023), as: f(i) - f(y), where f indicates an estimated factuality scores and ý indicates edited output.
4 Benchmark: FAVABENCH
FAVABENCH consists of around 1k fine-grained annotations on three LM responses to queries in multiple domains.
Source prompts. The source prompts include collection of 200 information-seeking queries, spanning four different data sources. See examples in Appendix Table
50 Knowledge-intensive queries from Open Assistant (Köpf et al. 2023 , by prompting
GPT-4 to judge whether each query from the dataset requires world knowledge.
50 Open QA prompts from the No Robots dataset (Rajani et al. 2023).
50 instructions requiring more reasoning and knowledge by the authors.
50 synthetically-crreteed prompts that require more fine-grained knowledge, by converting
data-to-text WebNLG dataset (Gardent al. 2017) using template.
Annotation details and quality. We obtain responses to the collected source prompts using ChatGPT (gpt-3.5
5- -3.5- nurbo-0001; Ouyang et al. 2022) Llama2-Chat 7B and Llama2-Chat 70B (Touvron et al. 2023) in zero-shot manner and collect 600 responses our diverse information-seekingg prompts. We recruited 20 students (ten undergraduate and ten NL. graduate students) to annotate the factual accuracy of the responses based on our proposed taxonomy. Each instance is annotated by two annotators who completed 45-minute in- person and virtual training sessions 7 We provide annotators with detailed instructions, examples, and training to ensure high-quality annotations. Our annotation interface and details are in Appendix and examples in Appendix Table 8 quality, we calculated inter-annotator agreement using Cohen kappa scores. Our annotators have high agreement in detection across passages, with 75.1% agreement in detection at the sentence level and 60.3% agreement in exact error type detection.
To validate our annotation
Analysis annotated data. Figure prgureetresentss a detailed breakdown of distributions across fine-grained categories in the three domains. 59.8%, 70.2% and 64.9% of the responses of ChatGPT, Llama2-Chat 7B and Llama2-Chat 70B include at least one hallucination, respec- tively. Entity is the most widely recognized error type, making up 48.3% of the detected
6This is similar FActScore Min et al. 2023), while we predict and evaluate factuality at the sentence level without extracting atomic facts. We compare FAVA against FActScore on this task. 7 Annotating each response takes approximately 10 minutes, and we pay USD 3.5 for annotation.
5

---

Published as a conference paper at COLM 2024
Serd Passage Generation
Part Eeror Insertion
Part 3:
Post Processing Data Train Fava
Retrieve Wikipedia Documeet
Proenpt LM Erress One
Licnel Andrés Mesii (bom June 1987) professioral fouthaller...
filation eival
[Remove tage deleted portiet from
[Revense Deletion And Marking from
Illooing pastage
Messi Ecotbel's
Land Moré roverall Sachall reloceel crelationn-iro Corellationn Spain 13..
Spait 13, phyvical themppi
Lionel Musi, revered as one of
factball's grestes..... relocateed crelatlonssoo to zirelations Spatn at 13 sanverifaablee waine-well plgeneni shomapest
Promept ChaitiP Divemity
Wednenday.
Write fing vrible glvee only the pusegr passagr
Frisonat loyal
Ieann vn umerrtaatie evvrr the
Slnneriffable
1987) Angenin::
Linnd Moss, revered as one of fantball's graviod inlocatedl to Spain from Argestina
Livnel Mosi, severed (ootballs sccatar....clocannd Srelationpioen to Sirdltioon Sanwerllablestt therapist
professioral foothallas...
Pelerencc
papedd Ostput
Wealeesshy. Slaveriffables
Fraining Fava of
Figure 4: Overview of high-quality synthetic data generation process in FAVA. FAVA lever- ages powerful instruction-tunee models to carefully insert errors into factually accurate statements and produces diverse error types based on our proposed taxonomy.
errors. Yet, there are diverse types of errors prevalent like invented statements or contra- dictory statements, which make up 14.1% and 18.1% of the detected errors, respectively, The error distributions vary across different source prompts and LMs. Invented are more common in Llama2-Chat models. Fewer errors in No Robots subsets than other subsets may be because their seed prompts are less knowledge-intensst factual knowledge (e.g., How long was the Revolutionary War?), which is often memorized by LMs (Mallen et al. 2022).
or ask about popular
5 Model: FAVA
We introduce FAVA (FAct Vericaton with Augmentation), a model for fine-grained hallu- cinations detections and editing. FAVA is trained on high-quality synthetic training data to identify hallucinations, incorporating retrieved knowledge. FAVA consists of two com- ponents: a retriever Myet and an editing LM Medit- Mret takes the original output LM y and optionally input prompt x (if applicable), and retrieves top relevant documents: C C=Mret yz. Subsequently, the editing model detects and, if possible, edits factual errors in y given the retrieved context: ý -Madi (x, y, C). 9 is augmented output y interleaved by the error edits with hallucination types as shown in Figure While Medi can be any LM, in our preliminary experiments, we find that making a state-of-the-art proprietary LM such as ChatGPT to perform fine-grained editing via prompting only is challenging (Table 2). Reliance on such models also hurts reproducibility. Therefore, we generate high-quality synthetic training data with minimal human efforts (Section 5.1b and fine-tune smaller yet powerful LM on the generated data (Section 5.2
5.1 Synthetic Training Data Curation
To train our Meda model, we require a large number of training instances, each of which includes (c,y,y"), where c is the gold context, y is an erroneous LM output, and y" is an output with error tags and correct editing based on c. FAVA learns to take (c. and generate y². Inspired by prior work that leverages LMs to generate synthetic training data (Balachandran et al. 2022 Wang et al. 2023b Asai et al. 2023), we introduce a new data creation method tailored to our fine-grained hallucination taxonomies. Specifically, our data creation pipeline (refer Figure 4) consists of three steps: Step 1: seed passage generations that generates large, diverse collection of seed passages c, Step 2: error insertions which inserts factually incorrect statements to generate erronous passages y, and Step 3: post-processing which curates the final training examples.
Seed passage generation. The goal in this step is to collect and generate a large, diverse range of seed passages to ensure FAVA can handle wide range of text types while maintain- ing control over factuality. Later on, errors are inserted into these seed passages to generate erroneous text, y. We base our seed passage collection on Wikipedia articles and sampled
6

---

Published as a conference paper at COLM 2024
5k QA pairs from Natural Questions Kwiatkowski et al. 2019). We paraphrase Wikipedia passages into different genres to enable FAVA to be adaptable and effective across various textual formats. Specifically, we randomly sample 35,074 Wikipedia articles and use them as our gold reference passages. We then randomly sample one of the pre-specified genre types (e.g. blog article, tweet, etc.) for each passage and use ChatGPT to paraphrase the original passage into the given genre [
Error insertion. The goal of this step is to simulate erroneous LM generation y, which are passages with factual errors. To control error distributions in the training data and incorporate our hallucination taxonomy, we prompt LMs to insert errors from our taxonomy one by one. We use ChatGPT and GPT-4 interchangeably for six different types ? See detailed discussions Appendix Given an instruction and few-shot demonstrations of an error type etype, GPT-4 or ChatGPT inserts new errors while retaining previously inserted errors. Models mark phrases or sentences for deletion along with their error type and insert phrases and sentences with insertion tags. Now we have an erroneous text y, which inserts multiple errors into the original text f.
Post-processing. We then post-process data, by swapping and removing the error tags and edits to form a clean erroneous text y the input and use the original correct passage y" as the output with edits, for training Medit- We filter out the examples violating our editing rules (e-g., inserting errors without marking them up with tags and error types). We also re- trieve additional four relevant Wikipedia paragraphs using Contriever-SSMARE et al. 2022), and randomly mix the order of the references, forming the final references C.
[izacard]
Statistics of data and human evaluation. After analyzing ablations with varying training
instance sizes (Section 7.2,, we generated 35,074 training instances, TO 30,0074 based on Wikipedia passages, and 5,000 based on QA pairs. All of the error types are almost equally distributed (~15 % each) with 3.1 errors inserted for Wikipedia subsets, and 1.4 errors inserted for the QA pairs on average. We present the error distributions in Appendix Table 10 We conduct human evaluations on randomly sampled 50 examples and find that the generated data meets the expected criterion most cases. See details in Appendix C.4
5.2 Training and Inference
We train smaller LM on the generated large-scale training data. In particular, given training instance y.y,y") our model Madit takes (C,y) as input and learns to predict the edited outputs with tags to represent error type y" using the standard language modeling objective. We use Llama2-Chat 7B to initialize Medit- We empirically found initializing FAVA with this model gives slightly better performance than the pre-trained Llama2. At inference time, we retrieve the top five documents from Wikipedia [T using and input them together with an LM output that may include hallucinations. The model identifies hallucinations, marks phrases or sentences for deletion, and suggests edits.
Experiments
6.1 Experiments for Hallucination Detection
Evaluation data. We use our new benchmark, consisting of 902 annotated passages as our test data. We measure the models' detection performance based on sentence-level per-category classification task as formulated in Section 3.2 We evaluate both fine-grained detection and binary detection settings. Note that fine-grained hallucination detection is our new proposed task, as prior work mostly focuses on the binary detection setting.
$See the full list of the genres, instructions, and final ChatGPT-gennerted text Appendix Table 'We use ChatGPT (gpt-3.--turbo-0300) for the Type and Type 3 while using GPT-4 for other types. 10hh we see performance improvements up to 30k, the API costs for training data generation were already $3k, reaching our initial budget.
We use English Wikipedia data from January 2023 and generate embeddings using the Contriever. 1e collected 1010 annotated passages including our experimental batches, excluding earlier batches for evaluations.
7

---

Published as a conference paper at COLM 2024
Generator: ChatGPT
Generator:
Llama2-Chat Chat 70B
Editor
ent
rel
con
inv
subj
unv
Avg.
ent
rel
con
inv
subj
unv
Avg.
ChatGPT
19.5
28.6
40.0
11.8
7.7
0.0
18.8
24.7
15.6
26.7
11.0
17.6
12.8
24.1
Rt+ChatGrT
28.1
19.2
25.5
5.4
37.7
15.5
24.4
33.7
24.2
24.0
22.2
17.8
4.7
27.8
GPT4
38.6
16.6
17.9
22.2
50.0
17.2
34.2
55.5
60.0
21.2
15.4
2.0
25.0
42.5
FAVA (ours)
54.5
25.0
66.7
16.7
70.5
35.3
48.1
57.3
34.5
27.7
52.2
31.3
43.4
47.2
Table 2: Results on fine-grained detection (metric: F1). Full results are in Appendix.
Baselines.
Fine-grained hallucination is a new task and we test three baselines that use with
state-of-the-art proprietary LMs: ChatGPT prompts ChatGPT 1oot-3.5-turbo a carefully designed prompt describing all six categories with two demonstrations. GPT4 (gpt-4) does the exact same but prompts GPT4 instead. We were unable to include GPT4 with retrieval in our baseline due to high costs. Since FAVA uses retrieval augmentation, for comparability we include Rt+ChatGPT. This baseline prompts ChatGPT using the same prompt and demonstrations but also includes the top five retrieved documents by Contriever at test time to augment the original prompt 13 We additionally evaluate FActScore (Min et al. 2023) for the binary detection task only. This model-based metric prompts ChatGPT and G (davinci-003) to decompose response into set of atomic facts and verify factuality for each using passages from a designated Wikipedia article.
6.2 Experiments for Hallucination Editing
Evaluation data. For editing evaluations, we use open-ended text-generation data and evaluate models' factuality based on metric designed to measure the factuality for the target task. In particular, we use the biography generation task proposed by FActScore (Min et al. 2023) to evaluate the effectiveness of editing to reduce hallucinations based on edited outputs' FActScore results.
Baselines. For editing experiments, we use ChatGPT and Rt-ChatGPT, well as Llama2- Chat 13B with and without retrieval (Llama and Rt-Llama, respectively), and prompt them with carefully designed prompts and two demonstrations. For all baselines, we retrieve five documents consisting of one document retrieved by string matching based on the entity name and four documents retrieved by Contriever, which are reranked at test time, using small cross-encoder
6.3 Human Evaluations
Our automatic evaluations may not fully capture the models' abilities to detect and refine hallucinations, due to the potential subjectivity of our annotations or the performance of the factuality evaluation metric. We evaluate randomly sampled 50 outputs from FAVA as well as the baseline with the highest automatic evaluation score, namely Rt-ChatGPT. We ask human annotators to verify how many of the detection and edits are indeed correct based on the provided retrieved documents. This is similar to our automatic precision evaluation, but instead of coarsely evaluating detection performance at the sentence level, we evaluate the model performance at each detection level.
7 Results and Analysis
7.1 Results
Fing-grained detection results. Table shows the fine-grained detection accuracy, namely Fl scores, and binary prediction Fl of FAVA and baselines. We provide the full precision and recall scores in Appendix Tables 13 and 14 respectively. FAVA significantly outperforms ChatGPT, Ret-ChatGPT, or GPT-4 on both fine-grained error detection and binary error
While we also tested strong white box LMs including llama--Chat 13B, we found their predictions often show confusion among different categories or violate formats.
h httppss////gggggg/// co/ co/cross eewesseencoorrree /ms -ar-aaaai--iiL-------
8

---

Published as a conference paper at COLM 2024
Generator
Generator Al-13B
Settings
Score
Model
Chat
Llama
Editor
CGPT
A1-7B
Chat
50.1
68.4
No Edit
66.7
42.5
38,8
No Edit
42.5
R+Chat
64.8
72.8
CGPT
58.6 .11
42.0
37.9
(Top 1)
44.2 +1.7
GPT4
60.8
74.2
Rt-CGPT
62.7 -10
43.9 +1.4
39.2 +0.4
(Top 5)
47.0
Llama
52.6 -1.1
22.7 -1988
18.6 -202
FAct
67.7
71.2
Reranked
47.7 452
Rt-Llama
58.7 88.0
48.6 +6.1
32.2 466
FAVA
79.6
80.3
FAVA
70.0
51.8
43.2
Entity
50.1 +7.6
Table Results of bi- Table 4: Results of editing. nary detection.
Table 5: Ablations of retrieval for editing.
detection. FAVA shows high accuracy on error types such as Sentence, Subjective Entity On the other hand, its performance on Invented and Unverifiable still limited. Those two error types often require intensive search over many documents beyond the top few ones, while FAVA by default only considers the top five documents. Table shows the binary hallucination detection performance of FAVA, baselines models, and FActScore (FAct). FAVA outperforms all other models in detecting hallucinations by large margin.
Editing results. Shows the results of the editing task on the biography generation task. Our FAVA significantly outperforms prompted ChatGPT or Llama2-Chat 70B, despite being much smaller model in size. We found that ChatGPT is often confused with different errors and tends to over-predict contradictory or subjective statements. FAVA shows the largest gains in FActScore, showing the effectiveness of editing and error type detection.
Human evaluation results. Our human evaluation results are shown in Appendix Ta- ble 16 FAVA shows significantly better performance than Rt+ChalGPT both editing and detection scoring 46.3% and 31.8% more than Rt+ChatGPT on each task respectively. Furthermore, FAVA recognizes more errors, 2.4 errors per passage, than eetrivallauggmenee ChatGPT, 1.9 errors per passage. These results further demonstrate the strong capabilities of FAVA detecting factual errors in LM outputs.
7.2 Analysis
Effects of data scale. We assess our model on varying sizes of synthetic training data (10k, 20k, and 30k instances) and examine their performance on fine-grained hallucination detection. Appendix Figure demonstrates that FAVA variants with larger number of training instances perform significantly better at detecting fine-grained errors.
Effect of retrieval. We assess FAVA'S editing performance on Alpaca 13B outputs with vary- ing retrieved evidence: top 1 document; top 5 documents; reordering the top 5 documents; top 4 documents and mix them with an introductory paragraph of the target entity (entity matching). 5ableSsshow the experimental results. More detailed results are in Appendix Table 18 We found that retrieving the top five documents significantly enhances perfor- mance compared to retrieval of only the top document by 4.5%. Ordering passages using a reranking model gives some improvements, which indicates the non-negligible effect of context order, reported by prior work (Liu et al. 2023 Including the reference based on entity matching gives a notable enhancement to 50. 1%,, marking 3.1% gain from the top five retrievals and 7.6% improvement over the unedited generations. These results indicate although FAVA demonstrates strong capabilities, there's s room for improvement in retrieving and incorporating many references.
8 Conclusions
We introduce a new task of automatic hallucination detection, built upon our newly intro- duced taxonomy, hierarchically classifying hallucinations in LMs into six categories. We collect the first human-anmotated fine-grained hallucination benchmark, and introduce retrieval-augmented LM for fine-grained error editing data following our taxonomies.

---

Published as a conference paper at COLM 2024
Empirical results show that FAVA significantly outperforms strong baselines by large mar- gin on both editing and fine-grained detection tasks, while still large room for improvements for automated fine-grained error detection and editing.
Acknowledgement
We thank Sewon Min for fruitful discussions in the early stages and UW NLP members and annonymous reviewers for their insightful feedback on the draft. We thank Stability AI for providing computing to train and evaluate the LMs in this work, and Microsoft Accelerate Foundation Models Research Program for the access to Azure and OpenAl APIs. We thank our annotators who provide high-quality annotations for this work. This work was funded in part by the DARPA MCS program through NIWC Pacific (N66001-19-2-4033)
NSF IIS- 2044660, and gifts from A12. We gratefully acknowledge support from NSF CAREER Grant No. IIS2142739, NSF Grants No. IIS2125201, IIS2203097, and gift funding from Google, MSR, and OpenAT.
10

---

Published as a conference paper at COLM 2024
References
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint drXiv:2310.11511, 2023. http:ttss///aaw...r/////////////
11511
Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi, and Wen-tau Yih. Reliable, adaptable, and attributable language models with retrieval. Xiviv preprint ar wxxx:2403.00338 2024. URL https://arxvi.....o..mm wrwg hwtpg//bb//243.... 03187
Vidhisha Balachandran, Hannaneh Hajishirzi, William Cohen, and Yulia Tsvetkov. Correct- ing diverse factual errors in abstractive summarization via post-editing and language model infilling. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022. URL httttss://acaaahholloo wwwp///.....mpmmmaa.a.a.....aaaa 667
Giannis Bekoulis, Christina Papaggannoooulou,, and Nikos Deligiannis. A review on fact extraction and verification. Association for Computing Machinery Computing Surveys (CSUR), 2021. URL httph: //d1. acm. wwwg//o///////o 1115///411.11////1122/
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Pra- fulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Infor- mation Processing systems, 2020. URL httpp::/paapers.tttt httpppscccpappr////ee/ 222per/20200hahh) -tttaaattttttt--ttatatttttt.tt
Anthony Chen, Panupong Pasupat, Sameer Singh, Hongrae Lee, and Kelvin Guu. Purr: Effi- ciently editing language model hallucinations by denoising language model corruptions. 14908
Xiv preprint ar (rx::22305.14908 2023. URL htttt:::///axxi.oogr////
I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, al. Factool: Factuality detection in generative ai-a tool augmented franework for multi-task and multi-domain scenarios. ArXiv preprint arXiv:2307.13522 2023. URL httpt:t///a/xiii http://rrxii...rraa///3//// 13528
Ashwin Devaraj, William Sheffield, Byron Wallace, and Junyi Jessy Li. Evaluating factuality in text simplification. In Proceedings of the 60th Annual Meeting of the Association for org/
Computational Linguistics (Volume 1: Long Papers), 2022. URL
2022. acl acl-.on...o........ 506
Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. Evaluating attribution in dialogue systems: The BEGIN benchmark. Transactions of the Association for Computational Linguistics, 10:1066-1083, 2022. doi: 10.1162 ta0560 URL. org/2022. tac1-1.62
Shangbin Feng, Vidhisha Balachandran, Yuyang Bai, and Yulia Tsvetkov. FactKB: Gener- alizable factuality evaluation using language models enhanced with factual knowledge. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2023. URL
emnlp-main.5 59
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. RARR: Researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2022. URL https
//aclanthology.orggg. www.:/rg///222... acl long. 910/
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Beltz-eelra,chni. The WebNLG challenge: Generating text from RDF data. In Proceedings of the 10th Interna- tional Conference on Natural Language Generation, 2017. URL https: haapss//oolla org/
W17-3518
11

---

Published as a conference paper at COLM 2024
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. Xiv preprint arXit:2311.05232 2023. URL https////aaxiv.or/abb/////// 05232
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Ar- mand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2022. URL https
//openreview.co //openrevieee.e///foruuulddddd http://tttooorumindiiiii...i...
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. Association for Computing Machinery Computing Surveys, 2023. URL https
acm. wtt.a//mmrr/////...o.//// 1145/3571733
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. Openassistant
deessattion-ddeooraiiiiinnn large language model alignment. arv preprint ar xrXiv2234.003277 2023. URL /ttpss///axiiv atgg/bbs/2304 07327
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszko- reit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answer- ing research. Transactions of the Association for Computational Linguistics, 2019. URL https:///aclanthoooo.oorg///////
Junyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji- Ji-Rong Wen. The dawn after the dark: An empirical study on factuality hallucination in large language models, 2024. URL https://aaxxi http::::///aiiiiooo///////////// 2401. 03205.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 2023. URL http::::///axxiiorr///aaa/ 03172
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021. URL https //aclannhology.orrg wrwp:/21...2222.. www.e-nnnaiieeeeeee mompmmmain.5.......
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2022. URL httpp:://ccaantholl org/ /023 acl lang-lon.544. 5566
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black- box hallucination detection for generative large language models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: hrr.laantoolog..or//2e/22 www.eeennliliieeee 357.pdff
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL emnlp-main main. 741/
Preslav Nakov, David Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barrón-Cedeñio, Paolo Papotti, Shaden Shaar, and Giovanni Da San Martino. Au- tomated fact-checking for assisting human fact-checkers. Proceedings of the Thirtieth
12

---

Published as a conference paper at COLM 2024
International Joint Conference on Artificial Intelligence Survey Track, 2021. URL https
7/www. hwwpaaa/rgg.oooeeeee////////////
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022. URL
neur ips wttppssccccppppcccepeeeepeea www.fs/pplpprrrre
2022//1ie/hheeee03333344113 ott.://000004f04000000a1aa1roaa0 orr.rp/eerrrriiirarioooooroororr pdf]
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. Understanding factu- ality in abstractive summarization with FRANK: A benchmark for factuality met- rics. In Proceedings of the 2021 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, 2021. URL https //aclanthology. htwethhoogg..rrg22200 2 nwacc main. 383
Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. No robots, 2023. URL
Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, S.M Towhidul Islam Tonmoy, Aman Chadha, Amit Sheth, and Amitava Das. The troubling emergence of hallucination in large language models prescriptive remediations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL
extensive definition, quantification, and
org/ 2 cennp-mmain
155
Tal Schuster, Adam Fisch, and Regina Barzilay. Get your vitamin C! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021. 2021.
URL httphtts////aaaahoollg. org.ooggg//2222 .naccl mainn.5 52
James Thorne and Andreas Vlachos. Automated fact checking: Task formulations, methods and future directions. In Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics, 2018. URL https:://aaaathholl org/C18-1283
James Thorne, Andreas Vlachos, Christos Christodoullpoulls, and Arpit Mittal. FEVER: large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018. URL UtL/httts////acanthoooog
org/N18-1074
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint Sit::2207,09288, 2023. URL
https://arxiv.org3 wttp://wwwwg///////////o/oo 20tps/9288878.888888
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
URL
org/2020.0... emn emnnp-mmine main.aann...m 609
William Yang Wang- "liar, liar pants on fire": A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2017. URL htttps://aclanthollloe
org/P17-206
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishiri.. How far can camels go? exploring the state of instruction tuning open resources. In Advances in Neural Information Prucessing Systems, 2023a. URL https:// prxiv.org/abe htwgg/abb/2306..2 047511
13

---

Published as a conference paper at COLM 2024
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Anmual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), 2023b. https / https///aaaaaanolooorrrrrr. http:/y..orr//2/// acl long 754
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXi::2309001211 2023. URL https///arki..org/////23B...88 01219
14

---

Published as a conference paper at COLM 2024
Appendix
A
Details of Human Annotations
Seed query selections.
Table 6 shows examples of the seed queries for our initial dataset
creation.
Dataset
Example
WebNLG
Explain A.C. Cesena, including information about ground, league.
Instruction-oolloiing
Explain the differences between New York cheesecakes and Basque Cheesecakes in detail.
Open-Assistant
Can you tell how tall the burj khalifa is? When was Samsung founded?
No Robots
Table 6: Examples of source prompts.
Annotation interface.
Figure 5 shows the interface of our annotations.
Annotation Intorface
SOLECT ESMON TIME AND rYPe SUGDEST to AMYLIGNLLE
EDIT TRACASRR
2000 - 1O89 (ENTITY)
Supjectvv
Colet
which known that steak
Aston VB is nange kaeury sports cara produced by the Britih autoraker Aston Martin to 2000 emarlc- 1889-cmarii ENTITY The VB name refors to the engine that powers these veh cles, VB engine which known for its Aston Marlin
atylch design, weil their exceptional handlnng and performance
(UNVERRABLE)
high performence
ofered various
styles
inclucing
was liter apgradeed mmarr powserty s.0 lren vo that prosuced aroundl 400
coupes, converitles, secans. majority design. as wall thair esseptional handing and performance. UNVERIFIABLE The VBs also had number of notable features. including power windows, air toning premium sound systems, leather upholstery, and more. The VB engine used the Aston V8 powerfui rofined
soupes, were
and stylih
homepower LCONTRAGICIORN
Dolet
5.3 3.3-itor
later upgraded 10 more powerful
VB produced around 400 horsepower. CONTRADICTORY
Pat drving
engines were paired with ether a 5-speed manual or -mmarooScmmrke ENTITY speed automatic transmisiion. Overal, the
Aston Martin
nigh-parformance kaeury vahicle that offined thrilinng difving experience and esceptionnl style
SUBJECTIVE
longer production,
sought-stt suught-tter
REFORENES FOR AMNOTATIONN
Check WWlipodia
annotation
W Wikipeiaa Reference:
Aston Martin V8
Conterts Bood
ancycopeda
linglem 1980 teaifornal Arzon Martes, wes
handact euch cur rocuing 1200 man- Irass!
CO (otaey
model
Series
radee taking can engin was not medy forenss
046 VS denie the dragr-4a maraga 1877 the 065 repped and cale compeny's
Toaen wady Aobo
Figure 5: Annotation interface.
Details of annotations. We hired 10 undergrads studying computer science to complete annotations. Each annotator was assigned 60 passages to annotate and had to undergo 45 minute one on one training session to understand the task and how to navigate the annotation platform. The training session covered an depth explanation of the six different hallucination types in our taxonomy, included walk through of how to annotate passage, and allowed time for annotators to ask any questions.
Annotators were given an instruction document outlining the task, details on the hallucina- tion type, details on how to navigate the annotation platform, and payment details which they looked over during their training session before starting annotation work. Figure 6 shows the instructions on using the annotation platform and payment details provided to annotators.
15

---

Published as a conference paper at COLM 2024
Instructions
1. Read through the passage provided and scan for errors or hallucinations present. You can the references provided below to help with this.
2. For any hallucination you identify. Select the hallucination/eerror type you think corresponds to the text you detected erroneous from the drop down menu.
For entity and relational errors, type proposed edit to the erroneous text you identified to make the statement factually correct.
3. Highlight the text in the passage you want to annotate.
4. Click the add button insert your annotation.
5. If you use the wikipedia reference provided, please select the checkbox above the article.
If you use the top 10 google search articles any other articles from the first page of google, please copy and paste the urls used in the textbox provided at the bottom.
b. For any unverifiable error types, please add the top 10 urls you looked through to determine was unverifiable.
6. Click the next button to submit your annotation and move on to the next one.
Here is link to a detailed walkthrough of the platform.
Payment Details
You will be assigned 60 annotations with the expectation that each annotation takes 10 minutes. For the 600 minutes or 10 hours that you give towards annotations, we will compensate you $200 ($20/ hr)
Figure 6: Annotation instructions and payment details provided to annotators.
Examples of annotated results. Table 7aole7ssore shows the fine grained hallucination distribution across our annotations. Table list of annotated examples from each of our annotation datasets WebNLG, OpenAssistant, and Instruction Following dataset.
Type
Percentage
Entity Errors
28.4%
Relation Errors
2%
Contradictory Statements
6.1%
Invented Statements
3%
Subjective Statements
20.2%
Unverifiable Statements
40.3%
Table Hallucination percentages in our human-annotated data.
16

---

Published as a conference paper at COLM 2024
Dataset
Annotated Passage Example
WebNTG
A Long Long Way is novel written by Sebastian Barry. first published in 2005 by Sentity>-eebe--nd
viking press</entity> a <entity>> Beitish American< /entity'> publishing company. The novel tells the story of Willie Dunne, an Irish soldier who has strong of loyalty to his country, but also feels torn between his Irishness and his loyalty to the British army during World War I. The novel was critically acclaimed and received several literary awards, <contradictory> including the 2005 Dublin Literary Award </contradictory> and was shortlisted for the Man Booker Prize in the same year.
OpenAssistant
<subjective>> The most luxurious airline flying within the continental United States is generally regarded as JetBlue kirways<<ssubeectvee particularly for its Mint business class service. Mint offers a private suite experience, featuring fully lie-flat seats, ample legroom and storage, premium food and beverage options, access to onboard WiFi and live TV streaming, and turn-down service complete with duvets and pillows.
Instruction Following
Some of Afghanistan's 's secondary cricket home grounds include:
1. I.cootrrdictory- > L_coontrdiictryy-Kaaaaa >KKandhhrr
Cricket
Stadium,
Kanda-
ha <<<contradictory>>
2. <invented> Mis Ainak Cricket Ground, Logar< /invented>>
3. <nnvented>> Bost University Cricket Ground, Helmand< /invented>
Table 8: Examples from our human annotations for each dataset.
17

---

Published as a conference paper at COLM 2024
B More Details of Synthetic Data Creation and Training
Quality of error insertions. In our pilot studies, we found that asking ChatGPT to insert multiple error types at the same time easily makes the model misunderstand or get confused with different error types (e.g,, swapping or incorrectly identifying an error type). We also found while ChatGPT is capable of generating more simple types of errors such Entity struggles with generating plausible and difficult perturbations for more nuanced types.
Text diversification results.
Table Table9 shows the diversified text results.
18

---

Published as a conference paper at COLM 2024
Seed Passage Diversification Examples
Genre
Topic
Output
Article
Community
The Community Reinvestment Act (CRA) is federal law that encourages commercial banks and savings associations to help meet the needs of borrowers in low- and moderate-income neighborhoods. The Act was passed in 1977 by Congress to reduce discriminatory credit practices against low-income neighborhoods.
Reinvestment
Act
Essay
National
The National Dodgeball League is the only professional dodgeball league in the US, and was founded in 2004. is headed by Commissioner Ed- ward Prentiss out of Hopkins, MN and is composed of 24 professional teams that are divided into the National and American Dodgeball Con- ferences. In the past, the tournament has featured teams from Canada, Denmark, UK, Japan, New Zealand and Australia.
Dodgeball
League
Blog Post
Red Channels
In the beginning of the 1950s, the "Counterattack" journal published document, "Red Channels," " which named 151 individuals in the en- tertainment industry who were believed to be under Communist ma- nipulation association. Many of these individuals were being denied employment or had their careers negatively impacted due to their polit- ical beliefs or history. "Red Channels" effectively created a blacklist of those named in the document, further limiting their opportunities in the industry.
Social Media Post
Sandman
Check out The Sandman semi-retired American professional wrestler! He best known for his career with Extreme Championship Wrestling (ECW), where he held the ECW World Heavyweight Championship a record five times. He also had stints in World Championship Wrestling (WCW), Total Nonstop Action Wrestling (TNA), and World Wrestling Entertainment (WWE). #Wrestling #ECW #WCW #TNA #WWE James Brown's Don't Mind" has made its way up the R&B "Billboard" charts, hitting number four. The track was released in 1961 and also reached number 47 on the Pop "Billboard" charts. Fans will also be able to hear the song on Brown and the Famous Flames album Live at the Apollo.
News Report
I Don't Mind (James Brown song)
Summary
Robert Lui
Robert Lui was born on February 23, 1990, and is a professional rugby league footballer from Australia. He plays either as a halfback or five- eighth for the Townsville Blackhawks in the Queensland Cup. Dear fellow Americans, am here to propose a solution to improve American Sign Language communication. George Sperling suggests that with sevenfold reduction in the bandwidth for video transmission, we can achieve this. He even argued that the telephone was originally created for the hearing impaired but it became popularized by the hearing community. Let us not forget our roots and make change for the better. Have you ever wondered why sometimes people who know less about product seem to enjoy it more than those who have researched it thor- oughly? This phenomenon is called the Blissful Ignorance Effect and it's fascinating topic in consumer behavior studies. Our presentation today will explore this effect and why happens.
Speech
George ling
Sper-
Presentation Intro
Blissful Ignorance
Effect
Brochure
Nidulariaceae
Explore the fascinating world of Nidulariaceae fungi! This family, found in most ecological regions, includes five different genera, each with its own unique characteristics. With their tiny egg- aig-filled structures, these fungi are a wonder to behold!
Fungi
Text Message
Ashes and Di- amonds
Just found out about this book called Ashes and Diamonds by Jerzy Andrzejewski. It's set during the last few days of WWII. The main character, Maciek, has to kill Communist soldier. Sounds intense!
Table 9: Text diversification prompts. Instructions for diversification follow the following format: "Given passage, create a(n) [genre] of 3-6 sentences using only the information present in the passage. Do not inchude any new information not presented in the passage. Passage: Isampled wikipedia paragraph]"
19

---

Published as a conference paper at COLM 2024
Error distribution.
Table 10 shows the distribution of error types across all our generated
passages.
Percentage
Entity
21.2%
Relation
19.9%
Sentence
15.3%
Invented
14.6%
Subjective
14.1% %
Unverifiable
14.9%
Table 10: Statistics of generated training data
Training Details.
Our base model is Llama2-Chat 7b trained using 4xA40 GPUs. Our
training code is based off Open-Instruct hyperparameters.
(Wang et al. 2023a
Table
shows the training
Precision
Epochs
Weight Decay
Warmup Ratio
Learning Rate
Max. Seq. Length
Batch Size
BFloat16
2
0.03
2e-5
2048
128
Table 11: Training hyperparameters.
sh
20

---

Published as a conference paper at COLM 2024
C More Results and Analysis
C.1 Detection Results
Llama2-Chat 7B F1 results Table 12 reports the detection results for the Llama2-Chat 7B generations on our curated datasets.
Generator: Llama2-Char
7B
Editor
ent
rel
con
inv
subj
unv
0A
Bi
ChatGPT
25.2
12.6
16.1
15.0
12.4
12.8
63.4
Rt+ChatGPT
35.5
17.5
13.2
22.2
10.9
14.6
32.9
70.4
GPT4
45.4
23.1
28.8
15.4
3.4
28.6
47.3
72.1
FAVA (ours)
58.3
38.7
24.2
58.9
31.25
44.4
39.6
79.9
Table 12: Fine-grained detection F1. OA and Bi indicates overall and binary predictions.
Overall precision and recall. Tables curated datasets.
13 and
14
report the precision and recall on our
ChatGPT Generations subj
Alama2-Chat 70B
Generations subj
Model
ent
rel
con
inv
unv
OA
Bi.
ent
rel
con
inv
unv
OA
Bi.
CGPT
12.1 25.0
50.0
6.7
7.7
0.0
13.0
35.1
17.5
13.2
36.4
15.5
18.8
9.5
19.3
59.2
R-CGPT
19.5 12.5
20.6
3.3
28.6
11.9
17.2
49.7
25.2
16.7
18.8
25.7
14.9
3.2
21.0
60.0
Ours
35.4 20.0
46.2
10.0
75.0
30.0
40.1
69.1
56.8
31.3
23.7
39.6
70.6
46.4
46.1 80.0
Table 13: Fine-grained detection task (Precision). CGPT indicates ChatGPT. "OA" indicates overall accuracy and "Bi." indicates binary predictions accuracy.
ChatGPT Generations subj
Llama2-Chat 70B
Model
ent
rel
con
inv
unv
OA
Bi.
ent
rel
con
inv
subj
unv
OA Bi.
CGPT
51.3 33.3
33.3
50.0
7.7
0.0
33.6
87.4
41.8
19.2
21.1
8.5
16.7
20.0
32.4 81.1
R-CGPT
49.9
41.7
33.3
14.3
55.6
22.4
41.7
93.1
50.9
45.8
33.3
19.6
22.2
8.8
41.0
92.5
Ours
55.2 33.3
75.0
50.0
66.7
42.9
55.5
90.0
55.5
38.5
33.3
30.6
63.7
40.6
46.8 81.2
Table 14: Fine-grained detection task (Recall). CGPT indicates ChatGPT. "OA" indicates overall accuracy and "Bi." indicates binary predictions accuracy.
C.2
Benchmark Prompts
Table 15 shows the prompt used for baseline models when evaluating on the detection benchmark.
C.3 Details of Human Evaluation on Model Outputs
We conduct human evaluation 50 detection results of the top two models, namely FAVA, and Rt-ChatGPT. Our results are shown in Table 16, We anonymize the model results to avoid potential biases during evaluations. Annotators count how many of the detection and editing results are correct given extensive web searches.
C.4 Manual Analysis on Generated Data for FAVA training
We conduct human evaluations on 50 generated data to assess the automatic data creation quality. Prior work introduces entity-centrie synthetic data creation Longpre [reagioonLongpree et al. (2021) and often results in unrealistic edits that powerful LMs can easily identity which are synthetically perturbed. Therefore, we evaluate not only validity but also quality. In particular, evaluate
21

---

Published as a conference paper at COLM 2024
Overall Detection Score
0.7
0.6
0.5
0.4
0.3
0.2
precision recall fl
0.1
10k
15k
20k
25k
30k
Training Instance Size
Figure 7: Training Size
whether the model edits violate our annotation scheme; (2) quality-the inserted errors are feasible and realistic and notably different from provided few-shot samples. We ask human annotators to score each category for each passage either 0, 1, (higher better).
a generated instance from two aspects: (1)
Our analysis revealed that the data generated by our system with iterative insertion obtained an average score of 1.66 for validity assessment and 1.36 for quality assessment. Meanwhile, data generated by one-shot prompting scored an average of 1.1 for validity and 0.9 for quality. Our human evaluation reveals that our L.--enerated edits are indeed of higher quality than the one-shot data creation, and provide realistic factual errors.
Table 17 shows examples of generated erroneous data from our synthetic data creation pipeline. We ask human annotators to score each category for each passage either 0, 1, or 2. These scores were defined as follows: a score of indicated that less than half of the inserted errors aligned with the given criteria, score of 1 signified that at least half of the errors met the established criteria, and score indicated that all inserted errors met the criteria. As comparison, also generate training data with one-shot generation.
C.5 More Ablations
Retrieval ablations. 18ble|18showe the full results of the retrieval ablations. We enhance our retrieval process by strategically reordering the retrieval results. This involves rear- ranging the top 5 passages from our reranked Contriever documents and entity matching document such that the documents with the highest relevance are positioned closest to the text requiring verification. Reordering passages gives 1.7% improvements, further emphasizing the importance of retrieval and careful pipeline design.
Analysis on data scaling. We conduct ablations of different data scaling in FAVA. Figure shows the scaling results.
22

---

Published as a conference paper at COLM 2024
Given a passage with factual errors, identify any <entity>, <relation>, <contradictory>, <subjective>, Sunverifiable> <invented> errors in the passage and add edits for <entity> and <relation> errors by inserting additional <mark></marko or <delete><ddelet> tags to mark and delete. there are no errors, return the passage with no tags. Any changes to the original passage should be marked in <> tags. Below are the error definitions followed by examples of what you need to follow. Definitions:
entity errors (<entity>): a small part sentence, often an entity (e.g., location name), is incorrect (usually 1-3 words). Entity errors often involve noun phrases or nouns.
small part (usually 1 words). Relational errors often involve verbs and often the opposite of what should be.
2. relational error (<relation>): sentence is partially incorrect
3. contradictory sentence error (<contradictory>): a sentence where the entire sentence is contradicted by the given reference, meaning the sentence be proven false due invented info error (< invented >): these errors refer to entities that are not known or do not exist. This does not include fictional characters in books or movies. invented errors include phrases or sentences which have unknown entities or misleading information.
contradiction with information in the passage.
5. subjective sentence (<subjective=)) entire sentence phrase that is subjective and cannot verified, so it should not be included.
sentence where the whole sentence or phrase is unlikely to be factually grounded although it can be true, and the sentence denied using the reference given internet search, it is often something personal or private and hence cannot confirmed. Follow the given example exactly, your task to create the edi addted completion with error tags <>:
6. unverifiable sentence (sunverifiablee)
cannot be confirmed
#2
Passage: Marooned on Mars science fiction novel aimed at a younger audience. It was written by Andy Weir and published by John C. Winston Co. in 1952, featuring illustrations by Alex Schomburg. It ended up having readership of older boys despite efforts for it be aimed at younger kids. The novel inspired the famous Broadway musical "Stranded Stars," " which won six Tony Awards. The novel tells a story being stranded on the Purple Planet. I wish the novel had more exciting and thrilling plot twists.
Reference: Marooned Mars is juvenile science fiction novel written by American writer Lester del Rey. It was published by John C. Winston Co. in 1952 with illustrations by Alex Schomburg.
Edited: Marooned Mars is science fiction novel aimed at younger audience. It was written by kentityyomarkllester del Rey</markoxeeeeee>nnn Neir<//dllte><<entitt> published by John Winston Co. 1952, featuring illustrations by Alex Schomburg. <contradictorysie ended up having a readership of older boys despite efforts for it to be aimed at younger kids </contradictory>. <invented>The novel inspired the famous Broadway musical "Stranded Stars," which won six Tony Awards. </invented The novel tells story of being stranded on the montityomarr/Ree//mrrr barrk=deelteeuuplee/ddeetteeee <subjective>I wish the novel had more exciting and thrilling plot twists. </ubjjective> ##
Planet.
Now detect and include edits in the following passage like done in the example above. Include error tags for ANYTHING YOU CHANGE ORIGINAL PASSAGE.
Passage: PASSAGE_TO_VERITTS
Reference:
[REFERENCE]
Edited:
Table 15: Prompt used for baseline models for detection benchmark. We insert the retrieved context at the [Reference:] portion for retrieval-augmeeted baselines. For non-retrieval- augmented baselines, we do not insert [Reference: component.
23

---

Published as a conference paper at COLM 2024
Editor
avg. [E]
Detect (%)
Edit (%)
Rt+ChatGPT
1.9
23.9
17.2
FAVA (ours)
2.4
55.7
63.5
Table 16: Human evaluation results. We show the average number of detected errors and the correctness (%) of the fine-grained types and edits.
Original Passage
Automatic Data Generations for Training
"Solvalou," the hyper-entertainment ma- chine known for its sit-down cabinet design
"Solvalou," the hyper-entertainment machine known for its sit-down cabinet design and 3D shooting game- play, has gained critical acclaim. jinvented>> The game won the Shooting Star Award in 2020 for its innovative concept.</inventerr
and 3D shooting gameplay, has gained criti- cal acclaim. Although it failed to find com- mercial success, the game recently made comeback with digital re-release for the Japanese Wii Virtual Console 2009 one of the first titles under the Virtual Console Arcade brand. Despite an announced home port for the 3DO Interactive Multiplayer in 1994, the release ultimately canceled and never reached the market.
Although it failed to find commer- cial success, the game recently made comeback with digital re-release for the Japanese Wii Virtual Console in 2009 as one of the first titles under the Virtual Console Ar- cade brand. Despite an announced home port for the 3DO Interactive <entity> <delete>> Multiplayer< /elete>>> mark>> Singleplayer </mark> </entity> in 1994, the lease was ultimately canceled and never reached the market. <subjective> The game is masterpiece that revolution- ized the arcade industry and will always be remembered classic subjective>>
The knapsack problem is a problem in com- binatorial optimization where the goal is determine the optimal selection of items to maximize the total value within a given weight constraint.
The knapsack problem is problem in <entityy> <delete>> combinatorial optimization</ cotimizzation/ddelteeee <mark>> sequen- tial search</ mark>> </entity>> where the goal is determine the optimal selection items to maximize the total <entity>> <delete>> value< galue<//deltte delete>> <mark> weight</mark>> </entity> within given weight con- straint.
Table 17: Examples of generated data from automatic data creation.
Settings
FActScore
Alpaca 13B (no edits)
42.5
FAVA (Top 1)
44.2 +1.7
FAVA (Top 5)
47.0 +1.5
FAVA (Top 5) + reranked
47.7 +52
+ entity search
50.1 +7.6
+ order
51.8 +9.3
Table 18: FActScore editing results with different retrieval settings.
24


Reviews:
[{"review_id": "IQT9vnkbH0", "reviewer": "Reviewer_eZfJ", "summary": "This paper describes an approach to detect and edit fine-grained hallucinations. Several fine-grained hallucination types are defined. A strategy is proposed to generate hallucinations by prompting LLMs. A model is trained on generated data containing references-hallucination-correct triples. The model is asked to detect hallucinations and to edit hallucinations to produce the correct text.\nExperiments are done on a set of cases. The method is compare to the ones based on LLMs, and shown to outperform the latter.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for positive comments on our taxonomy and data generation process.\nReason to Reject 1: Taxonomy novelty - While some types may be similar to prior work, our taxonomy as a whole is notably distinct from other work. In section 3.1, we acknowledge that our taxonomy is inspired by prior work but includes new types, which supports its novelty. Also, there isn't any recent work which includes \"contradictory\", \"invented\", or \"subjective\" types. Also, we ensure comprehensive coverage of significant hallucination types.\nReason to Reject 2: Training details - Due to strict page limits, we included minimal training details in section 5.2 and more in appendix section B. We will move some details from the appendix to the main pages in the final version.\nReasons to Reject 3: Baselines - Regarding baselines, we use FActScore, cited in related work, as a baseline for binary detection, shown in Table 3. Due to the novelty of fine-grained detection, there are no baselines from prior work; most prior studies focus on binary detection, making it challenging to apply their methods to fine-grained setups.\nThank you for suggesting improvements to the presentation. We will implement these in our final version. Below, we address questions:\nComparison with baselines, such as [1]: Comparing with [1] is difficult as the paper does not provide open-sourced models. Note that [1] was posted on arXiv within 3 months of the COLM deadline and is unpublished, considered concurrent work in many NLP/ML venues, hence not included in our work.\nInvented error: The prompt for this type of hallucination was carefully constructed after multiple iterations and manual analysis. We detail our manual analysis techniques and results in appendix section C4.\nSubjective statements: Our paper focuses on information-seeking contexts, classifying hallucinations as factually incorrect or unverifiable statements. Subjective statements are inherently not grounded in factual data and hence hallucinations in our context. We included this type to ensure the completeness of our taxonomy.\nRetrieval context size: We limited reference documents to five due to context size constraints but conducted ablations to ensure quality, detailed in section 7.2. For fine-grained detection, we compared our model against human-annotated data. Despite annotators having access to more resources, our model performed strongly, suggesting five documents were sufficient. We will discuss this in the final paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The categorization of hallucination is slightly different from the existing ones. This leads to more fine-grained types of hallucination.\nThe way to generate hallucination cases is interesting. LLMs are instructed to insert hallucinations from a correct text. This allows the authors to create a quite large set of hallucination cases (35k).\nThe use of a specifically trained model for hallucination detection and editing is reasonable.", "reasons_to_reject": "Although the authors claim the types of fine-grained hallucination types are new, they are not very different from what others have proposed. The main categories are similar.\nThe details for training the model based on the generated data are missing.\nThe baselines considered in the paper are simple. In the literature, other approaches have been developed (e.g. checking factuality, fine-tuning a model. etc.). These models should also be included a baselines.", "questions_to_authors": "Fig. 2: the tests used in the figure to determine types of hallucination are difficult to follow. For example, from the question \"Does the information contradict with evidence?\", there are two yes outputs, and from the \"Is it verifiable based off web knowledge?\" test, there is only one no output. The criteria to determine the types are confusing.\n\nWhile it is interesting to include more (fine-grained) types of hallucination, it is unclear if \"subjective\" should be included as hallucination. The risk of considering it as hallucination is that many utterances in everyday dialogues (e.g. \"I think ...\") would be considered hallucinations. This does not make this type of hallucination very interesting.  What is the reason to include this type of hallucination?\n\nOn the other hand some other types of hallucination may require broad knowledge on the topic (invented, unverifiable). Looking at the approach, it seems that only 5 reference documents are included to cover the topic. This is not sufficient to verify if a statement is invented or verifiable. To detect these types of hallucination, more references (or larger models) should be considered. Therefore, the experiments done in the paper may not reflect the difficulty to detect these types of hallucination. Have you looked at the coverage of the 5 references on the topic? Are there cases where the statement is unverifiable with the 5 reference texts, but is verifiable with more reference texts?\n\nThe type \"invented\" is interesting. In addition to require broad knowledge to detect this type of hallucination, the generated data should also try to generate statement that are close to the topic. Looking at the examples, it seems that the \"invented\" hallucinations are randomly generated. This makes it artificially easy to detect, by checking if the topic of the statement is related to the remaining of the text. Have you tested different ways to generate invented hallucinations?\n\nTable 4: I guess the numbers are FActScore. Please specify.\n\nFor hallucination editing, I believe the Human evaluation results are more meaningful than FActScore. It mat be appropriate to move Human evaluation results to the main body of the paper (instead of appendix).\n\nThe experiments only compare with simple LLM-based methods. There are more methods proposed in the literature. The paper mentions several ones in the related work. Another strongly related work is https://arxiv.org/pdf/2401.03205, which also detects hallucination cases and uses different methods to mitigate hallucinations, such as fine-tuning, different promptings, RLHF, etc. It is appropriate to compare with these approaches.", "extra_scores": {}}, {"review_id": "bVG3csvHCV", "reviewer": "Reviewer_tF2X", "summary": "The authors introduce a comprehensive taxonomy that categorizes hallucinations into six detailed types and develop a novel fine-grained hallucination detection task. They also present FAVABENCH, an innovative benchmark for evaluating LM outputs with approximately 1,000 fine-grained human annotations. Leveraging this benchmark, the authors develop FAVA, a retrieval-augmented LM trained on synthetic data designed to detect and correct hallucinations at the span level. The evaluations demonstrate that FAVA significantly outperforms existing systems in both fine-grained and binary hallucination detection, thereby providing a robust solution for enhancing the factuality and reliability of LM-generated text.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We’d like to thank the reviewer for their positive remarks on our taxonomy, the FAVA model, and our innovative benchmark, FAVABENCH. Additionally, we value their recognition of our writing clarity and organizational approach.\n# Reasons to Reject 1: Current Baselines\nWe recognize that comparing task-specific models to general-purpose ones may yield anticipated results. Our primary objective in including baselines of GPT and Llama models was to underscore the necessity of a fine-tuned model for detecting fine-grained hallucinations. General-purpose models, despite the widespread recognition of their capabilities, struggle in this regard. By incorporating FActScore as a binary detection baseline, we aimed to contrast our approach with a specialized baseline tailored to error detection. Furthermore, due to the novelty of fine-grained detection or editing tasks, there is a scarcity of open-source models specialized for this purpose. Because of this, we were unable to identify comparable models trained on fine grained detection tasks for comparison. Therefore, we opted for carefully designed prompts and general-purpose models such as ChatGPT to fulfill this need.\n# Reasons to Reject 2: Lack of details in experiment section\nWe acknowledge the reviewer's concern regarding the clarity of the experimental settings. While we prioritize providing context by dedicating lots of space to discussing tasks and datasets, we recognize that this may detract from the section on experimental setup details. In our camera-ready version, we will utilize the additional space to enhance the clarity of our experiment explanations and analyses, ensuring that the details are more comprehensible.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The introduction of a comprehensive taxonomy that classifies hallucinations into six distinct categories based on their characteristics and the sources from which they need verification is a significant novelty. This classification allows for a more nuanced understanding and detection of errors than the traditional binary models of hallucination detection.\n\n2. The creation of FAVA, a retrieval-augmented LM that operates at the span-level for both detection and correction of hallucinations, further underscores the innovative nature of the work.\n\n3. The introduction of FAVABENCH as a new standard for evaluating language models could become a pivotal resource for future studies, encouraging more detailed and precise improvements in model outputs.\n\n4. The paper is well-organized. Each section logically builds upon the previous, with clear explanations that make the sophisticated content accessible to readers with a background in the field.", "reasons_to_reject": "1. LLMs are generally trained for broad applications, yet the proposed task demands detailed, fine-grained operations. Therefore, it is plausible that a smaller, task-specific LLM might outperform its more generic counterparts. However, the assertion that smaller LLMs exhibit remarkable performance over models like GPT and Llama is not definitive. If these larger models were trained using the same specialized dataset, one might anticipate even more significant improvements than the smaller, task-tailored models.\n\n2. The authors dedicate substantial space to discussing the proposed task and datasets; however, the description of the experimental settings, especially in Section 7, lacks clarity. The absence of detailed experimental setup information for Table 4 complicates the distinction between the generators used in Tables 3 and 4. This ambiguity significantly delays comprehension of the comparative analysis. I recommend that the authors refine the description of the experimental setup.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0A9isTRK70", "reviewer": "Reviewer_Lc32", "summary": "This paper proposes a new taxonomy for hallucinations and introduces a new fine-grained hallucination detection and editing task based on it. To evaluate this task, the authors provided FAVABENCH, a manually annotated benchmark. Additionally, instead of using existing LLMs for hallucination detection and editing, the authors also fine-tuned a retrieval-augmented LM called FAVA, which outperformed existing LLMs in hallucination detection and editing. The training of the FAVA model was conducted through synthetic data generation, which also follows the proposed taxonomy.\n\nThe paper is well-written, and the included figures significantly aid understanding. The topic of LLM hallucinations is also timely and appropriate, and the models and datasets provided by the authors would be beneficial to the community. However, apart from providing a taxonomy, there is no notable technological impact.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our experiment design, the timeliness of this topic, and our writing and organization.\nReason to Reject 1: Relation errors - This classification is derived from prior research, \"Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics.\" Although making up a small percentage of hallucinations, we included this type to ensure our taxonomy's comprehensiveness. Our goal was to create a novel and comprehensive taxonomy for all hallucination types in information-seeking scenarios, justifying the inclusion of relation errors.\nReason to Reject 2: Unverifiable vs Invented - We recognize the confusion between unverifiable and invented types. Invented types involve entirely fabricated entities, while unverifiable types lack supporting references. Our motivation for including both was to ensure thoroughness. Although some invented types could be unverifiable, not all unverifiable statements are invented. In our final version, we will clearly distinguish these categories and explain their inclusion.\nResponses to questions:\nFine-grained edits: A sentence may include multiple hallucination types, but their spans do not overlap, preventing editing confusion. In the editing phase, deletion and insertion are prevalent on \"entity\" or \"relation\" types to correct factual accuracy. \"Unverifiable\" or \"subjective\" types are considered at the sentence level and highlighted for transparency, as they lack grounding in truth. \"Contradictory\" and \"invented\" types are also sentence level and typically deleted entirely to maintain text integrity.\nError type classification: In fine-grained detection, each error type is assessed independently, with the classification of one type having no impact on the classification of another. When evaluating for fine-grained hallucinations, our analysis focuses on the presence of each hallucination type within each sentence in comparison to our human annotation detections. This approach ensures a thorough examination of the text, allowing for nuanced detection and classification of errors.\nKnowledge intensive dataset: The \"50 instructions by the authors\" dataset was created to enrich the annotation dataset with complex prompts. These instructions were meticulously crafted to enhance the dataset's diversity/comprehensiveness. While our dataset included prompts from sources like WebNLG, OpenAssistant, and NoRobot, we added these to broaden the scope across more scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is clearly well-written.\n- The paper provides models and datasets (benchmarks).\n- Well-designed experiments and their results demonstrate the usefulness of the proposed method.", "reasons_to_reject": "I have some concerns about the appropriateness of the taxonomy proposed by the authors:\n\n1. Entity and Relation errors: wouldn't it be possible that relation errors could also be resolved by replacing entities? Additionally, according to Figure 3, it seems that relation errors make up a very small percentage. Do relation errors introduce any unique impacts?\n2. Ambiguity between Invented and Unverifiable: The difference between 'invented' and 'unverifiable' seems ambiguous. In Table 1's examples, 'invented' could also occur due to a lack of evidence. The authors state, \"Invented are the hallucinations where we can verify that some core entities or subjects of the sentences don’t exist,\" but how can we be sure that these aren't issues with the retrieval stage?\n- Regarding 2, could you provide the agreement scores among human annotators specifically for 'invented' and 'unverifiable' errors? It appears that the currently submitted agreement scores do not specify this for each specific type.", "questions_to_authors": "- If Task 1, Fine-grained hallucination detection, involves binary classification for each error type, it's possible that a sentence might contain multiple error types. Then, for Task 2, Hallucination editing, how is editing performed according to each error type? Are there issues such as overlapping span boundaries in each edition?\n- In fine-grained hallucination detection, does the process involve separate binary classifications for each error type, or can the classification of one error type influence another (e.g., multi-label classification)?\n- Could you provide more detailed information about the third item on the list of source prompts for FAVABENCH, which states, \"50 instructions requiring more reasoning and knowledge by the authors\"?", "extra_scores": {}}, {"review_id": "judgVsNeS8", "reviewer": "Reviewer_AbYD", "summary": "This paper introduces a taxonomy of hallucinations in large language models (LMs) and proposes a novel task of automatic fine-grained hallucination detection. The authors construct an evaluation benchmark, FavaBench, with over 1,000 human judgments on LM outputs.  The authors then train FAVA, a retrieval-augmented LM, to detect and correct fine-grained hallucinations. The results show that FAVA significantly outperforms ChatGPT and GPT-4 on fine-grained hallucination detection and improves the factuality of LM-generated text.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for dedicating time to evaluate our paper. \n#Reasons to Reject 1: Related work \nThank you for suggesting papers. We are happy to include those two papers in related work in our final version. However, we respectfully disagree with the reviewer’s assessment that both papers are “similar in many aspects: task, detection baseline, and result findings.”\nBelow, we discuss the relationship between our work and suggested papers [1][2].  \n[1] HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models.  \n[2] The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models.\nHallucination taxonomy: Note that [2] is unpublished and was posted on arXiv less than three months before the COLM deadline while our paper was in review. [2] introduces a taxonomy with 6 hallucination types, but our taxonomy is notably distinct, with 3 completely different types. While both studies identify \"entity,\" \"relation\" types, these types have been used in prior work  (Pagnoni et al., 2021) and as we discussed in Section 3, our hallucinations extend from prior research. Although both taxonomies include \"unverifiable\" types, our approach distinguishes between \"invented,\" \"subjective,\" and \"unverifiable\" claims, while [2] groups these into one category.\nDetection baseline/metrics: [1] and [2]  are primarily within a binary detection framework, while our research introduces a novel approach by addressing fine-grained hallucinations and editing. Unlike binary detection, our system conducts evaluations at a span-level, identifying exact spans and categorizing them based on our hallucination taxonomy. Additionally, both binary/fine-grained detections are scored against human-annotated data in our study. Although [2] uses human annotators to validate some detections, our scoring is entirely based on human annotations. Also, our detection system incorporates editing which we evaluate using FActScore, a feature absent in [1][2].\n#Reasons to Reject 2: Quality of generated data\nOur synthetic data generation pipeline underwent carefully designed iterative generations to ensure similarity to hallucinations generated by LMs, complemented by continuous human evaluations. In Section 5.1 of our paper, we provide insights into the manual analysis techniques used to validate the generated data, with further elaboration in Appendix Section C4. We will move the details of human evaluations to the main page of our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) An important task;\n(2) An new method to create synthetic data for hallucination detection", "reasons_to_reject": "(1) The authors make a very poor literature about LLM hallucination. Just list two highly related studies\n\n-The dawn after the dark: An empirical study on factuality hallucination in large language models\n\nBoth papers are similar in many aspects: similar taxonomy for hallucination, detection baseline, and metrics. \n\n-Halueval: A large-scale hallucination evaluation benchmark for large language models\n\nBoth papers are similar in many aspects: task, detection baseline, and result findings. \n\nIn addition, there are also several hallucination dection methods that are not compared by this work. Thus, I find that the literature is highly incomplete.\n\n(2) The method to create synthetic data for hallucination detection, while a side effect is that LLM-generated errors may not be similar to the actual genrations of LLMs. That is, the detection method may only learn some shortcut patterns to detect hallucinations.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a new benchmark, FavaBench, for detecting fine-grained hallucinations in large language model outputs. The authors propose a novel taxonomy for hallucinations and develop a retrieval-augmented model (FAVA) to detect and correct these errors. Evaluation results indicate that FAVA significantly outperforms several frontier LLMs in hallucination detection and correction tasks. A reviewer raised some criticisms regarding its literature review and comparisons with concurrent work -- I personally really liked this work, and I believe it significantly improves our ability to understand and mitigate the impact of hallucinations in LLM-generated texts.
Decision: Accept