ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models
Authors: Yuzhe Gu
Ziwei Ji
Wenwei Zhang
Chengqi Lyu
Dahua Lin
Kai Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Hallucination
Large Language Model
Fine-grained Annotation
Dataset
Primary area: natural_language_processing
TL;DR: This paper introduces a multi-iteration scalable framework that enhances the size of the hallucination annotation dataset while improving the precision of the annotator.

Abstract: Large language models (LLMs) exhibit hallucinations in long-form question-answering tasks across various domains and wide applications. Current hallucination detection and mitigation datasets are limited in domain and size, which struggle to scale due to prohibitive labor costs and insufficient reliability of existing hallucination annotators. To facilitate the scalable oversight of LLM hallucinations, this paper introduces an iterative self-training framework that simultaneously and progressively scales up the annotation dataset and improves the accuracy of the annotator. Based on the Expectation Maximization algorithm, in each iteration, the framework first applies an automatic hallucination annotation pipeline for a scaled dataset and then trains a more accurate annotator on the dataset.  This new annotator is adopted in the annotation pipeline for the next iteration. Extensive experimental results demonstrate that the finally obtained hallucination annotator with only 7B parameters surpasses GPT-4 and obtains new state-of-the-art hallucination detection results on HaluEval and HalluQA by zero-shot inference. Such an annotator can not only evaluate the hallucination levels of various LLMs on the large-scale dataset but also help to mitigate the hallucination of LLMs generations, with the Natural Language Inference metric increasing from 25% to 37% on HaluEval.

ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models
Yuzhe 2u
Ziwei Ji
Wenwei Zhang?t
Chengqi Lyu
Dahua L.n
Kai Chen
Shanghai Jiao Tong University Shanghai AI Laboratory
HHnn Kong University of Science and Technology
'MMLab, The Chinese University of Hong Kong *HKGA under InnoHK
(guyuzhe, zhangwenwei Ivchengqi, chenkai] (apnkai/ppjla.oozz en 2jiad0connect. ust. hk
Abstract
Large language models (LLMs) exhibit hallucinations in long-form question- answering tasks across various domains and wide applications. Current hallu- cination detection and mitigation datasets are limited domain and size, which struggle to scale due to prohibitive labor costs and insufficient reliability of existing hallucination annotators. To facilitate the scalable oversight of LLM hallucinations, this paper introduces an iterative self-training framework that simultaneously and progressively scales up the annotation dataset and improves the accuracy the annotator. Based on the Expectation Maximization algorithm, in each iteration, the framework first applies an automatic hallucination annotation pipeline for scaled dataset and then trains more accurate annotator the dataset. This annotator is adopted in the annotation pipeline for the next iteration. Extensive experimental results demonstrate that the finally obtained hallucination annotator with only 7B parameters surpasses GPT-4 and obtains new state-of-the-ar hallucination detec- tion results HaluEval and HalluQA by zero-shot inference. Such annotator can not only evaluate the hallucination levels of various LLMs on the large-scale dataset but also help mitigate the hallucination of LLMs generations, with the Natural Language Inference metric increasing from 25% to 37% on HaluEval.
Introduction
Large Language Models (LLMs) have shown remarkable capabilities in various tasks [10, 11, 35, 51, 57]. However, they tend to produce hallucination. i.e., plausible-sounding but unfaithful or nonsensical information [5, 30), that significantly hinders their real-world applications. Initial steps to address this issue involve the creation of datasets that can help detect, annotate, and mitigate hallucinations [14, 29, 40]. Since the potential hallucinations
LLMs are various fields, the spectrum of knowledge in the dataset is expected to be large-scale and comprehensive, covering various domains. Consequently, the size and diversity of datasets are critical for the oversight of LLM hallucinations.
However, constructing and scaling-up hallucination annotation datasets face significant hurdles [8, 29, 43]. One primary challenge is the prohibitively high costs and labor intensity required for their accurate assessment [43, 47], since the fine- ime-ggrinee hallucination annotation requires intensives labor for reading long documents and annotating the hallucination details sentence by sentence. Morcover, due to the insufficiency of accurate human annotations, the reliability of existing hallucination annotators and detectors becomes another pressing concern [29]. These tools have been found to
Equal contribution Corresponding author DDatsset, code, and model are released at https: httgttttttttttbee wtt.:cccoo/oo/eeeee wwwpsescccaaaasasaee ANAH.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

produce inaceurate results [9, 46, 61], e.g., even GPT4 [1], one of the most powerful LLMs, is not satisfactory and cannot achieve a compatible performance of humans [29].
Existing works [3. 25, 36, 38, 56, 58, 69] have explored strategies in data augmentation and self- training to extend dataset size and boost the performance of models in the fields of image segmentation, multi-lingual translation, math reasoning, etc. However, how to scale the hallucination annotation datasets efficiently is under-explored analysis and further mitigation of LLMs hallucinations at a large scale.
the community, which significantly hinders the in-depth
To address the research gap, this paper proposes an iterative self-training framework designed to scale up the hallucination annotation dataset and simultaneously increase the accuracy of annota- tors (Fig. 1). The iterative framework can be plained from the perspective of the Expectation Maximization (EM) algorithm. In the Expecta- tion (E) step, we apply the existing best hallu- cination annotator to estimate the ground-truth hallucination annotations of the scaled dataset. We adopt an inference pipeline on top of the annotator with self- coffisensst strategy [63] to provide a more robust estimation of the annota- tions, which lays the groundwork for training more precise annotator in the subsequent step. In the Maximization (M) step, we combine the existing annotations with the scaled data annota- tions derived from the previous E steps to train a new hallucination annotator. Training on more data leads to more accurate annotator and more robust annotation pipeline, setting the stage for the subsequent round of annotations.
Dataset
Annohater
Dafcaet
Stronger Annatator
BOOK
89.05%
20OK
88.18%
GP1
86.97%
Repporses
Topica
12K
84.85%
Weal Avnolitor
Medium
Annaletor
Smell
Strong
Mastive
Datsset
Dutose!
Stagei
Staget
Sluge3
Figure 1: Our iterative self-training framework pro- gressively scales up the hallucination annotation dataset size (left) and simultaneously increases the annotator's accuracy (right) in three stages.
The iterative process consists of three stages of caalii-dimensoonll data scaling as shown in Fig. 1. Initially, we train weak annotator on human annotations. In the second stage, we collect the hallucination responses from more open-source L.MMs for the same questions in the dataset to improve the generalization ability of hallucination annotators to model responses. the third stage, we expand the number of topics and questions in the dataset and collect hallucination annotations with the more robust annotator. This progressive scaling strategy stabilizes the annotator's 's performance when evaluating familiar and unfamiliar responses across diverse topics.
Extensive experimental results show that our enhanced annotator significantly outperforms existing models, including the advanced GPT-4, in terms of accuracy. Our annotator not only performs best on the in-domain fine-grained hallucination annotation dataset ANAH (89.24%) but also obtains new state-of-the-ar (SOTA) results on HaluEval (81.54 1.54%) and HalluQA (94.44%) under zero-shot setting. In addition, the annotator automates the hallucination evaluation on the dataset, offering comprehensive benchmark for the research community to evaluate the hallucination levels of numerous open-source models, providing a practical reference for future hallucination mitigation of LLMs. Using simple reranking strategy with the annotator, reduce the hallucination of the final LLM generations on HaluEval, with the NLI metric increasing from 25% to 37%.
2
Related Work
Self-improvement of Large Models. As Large Language Models (LLMs) become more and more powerful, the community starts to explore different strategies achieve the self-improvement of LLMs, i.e., to improve the LLMs using the supervision from LLMs [12, 24, 28, 50, 53]. For example, existing works have explored self-alignment using LLMs with ethical principles [3, 58, 69]. There are also methods [13, 25, 38, 56, 70] strengthen LLM's 's capabilities on tasks such as reasoning by training the LLMS on the high-quality responses from themselves on the same questions. In the field of computer vision, SAM [36] introduces manual and model-assisted labeling to expand the image segmentation dataset and enhance the performance of image segmentation models. However, the application of self-imprrvement is under-explored in fine-graine grained hallucination annotation. This field

---

E-Step: Data Annotation
M- M-Step: Annotator Training
Output y*
Lebeled
Alaygaate
dast iis
New Data
Construct Longer Datoset
Asawee: the MeDce a booe cxlled "Treatinn on the
contrizecions nathamaties?
shalla
broreat: nice surviving aathoratioal works
Self- Consistieccy
Choesing Regressatative Output
Lager Dataset
Sabel: stact inter seuf points Caeilo t2p83
<juseei>:
tou will sct Exnt chacker Jour task Pacual stnta
Candiatey CEace Indoa
Candiitte idaes Lncin palces
Iterative Optimization
<jarslstarti>:
Stable facto. <anaver>
conÉ poants
You will set as an information extractor took
Informetion queatiot docusets equeasions canaver> edcoumatts and acanec, <larelstanti>
Stronger Annotator
Jafersce
Arnotator From M- M-Stap
Genencte n Candicate Output
commentary on the dictlestcias acnosening possulatas Benlid's Elemerta.
You will sot Meliielraator annosator tou reed detersita whether the prorsdad and entotate the falluntration. <jaselstanti>>
Unicbeled
<Dociments
shrewer
Fine: Taring a Stronger Antotator
Type
Growth Flywheel
Input
Figure 2: The schema of EM-based interactive self-training framework. In the E-step, given unlabeled new data from the Data Growth Flywheel, the annotator predicts N candidate outputs y. Then the representative annotation y² is chosen via self-consistency. As a result, we construct a larger dataset by collecting the new annotations. In the M-step, train annotator on the larger dataset aligned to our training format. This annotation process consists of three phases: Factual Existence Judgment, Reference Information Extraction, and Hallucination Type Judgment. As a result, we gain stronger annotator with higher accuracy.
is challenging for automatic annotators due to its meticulous nature, which requires fine validation with long documents. It is also noteworthy that most self-improvement works require extra resources such as human labor or a supplementary model [36, 38]. In contrast, our pipeline is self-sufficient, relying solely the annotator model and the initial dataset.
Hallucination Annotation Dataset. The development of the hallucination annotation dataset is the comeersoon for detecting ballucinations in models' output. These datasets can be used to train a hallucination detector//nnotator and evaluate the hallucination level via the detecttor/nnotttoor. Early works [21, 22, 26, 37, 40, 44, 48, 60, 62, 67] in this domain tended to broadly classify entire responses as either hallucinatory or not, providing a coarse-grained analysis of hallucination occurrences. Recent works [29, 47] annotate hallucinations in more fine-grained and meticulous way. Despite this progress, these datasets, especially those having fine-grained annotations, suffer from limitations in size and scalability due to the high costs associated with the usage of human annotators or commercial models like GPT4. In addition, the difficulty of this task and the limited human annotations result in unsatisfactory performance of the automatic hallucination annotator.
Hallucination Mitigation. Considering the harm of hallucinations, researchers have explored various techniques for mitigating hallucinations. Techniques such as multi-task learning [23, 65], model editing [18, 31], and fine-grained RLHF [66] are proposed to suppress hallucination tendencies during training. Alternative strategies have been proposed that do not require further model training. including different decoding strategies [15, 41, 52, 55], multi-agent methods [20], and variants of
the Chain-of-Though approach involving verification or reflection [19, 32, 39, 64). Our ANAH-v2 shows efficiency in hallucination mitigation as a re-ranker and has the potential to combine with the
existing methods such as fine- fine-grained RLHF.
3 Method
This paper proposes an iterative self ooff-training framework to simultaneously scale up the hallucination dataset and improve the accuracy of the hallucination annotator. We follow the analytical hallucination annotation ($ 3.1) to annotate the hallucination
The multi-iteration framework is theoretically grounded in the EM algorithm (& 3.2) and involves three stages to progressively scale the dataset in multiple dimensions ($ 3.3). We also reveal how the hallucination annotators can be applied for hallucination evaluation and mitigation ($ 3.4).
sattence-by-seneene
3

---

3.1 Analytical Hallucination Annotation
The aim of hallucination annotator is identify hallucinations in the model responses. ANAH [29] developed fine-grained annotation method that locates reference points in the document for each sentence and makes hallucination-typn judgments, with the whole process completed in one turn of dialog. However, this hybrid task diverges from the human judgment processes and fails to clearly indicate the relationship between reference points and hallucination judgments, resulting in unsatisfactory annotation accuracy.
Instead of using the original ANAH training prompts, we developed a more reliable training method tailored to the hallucination annotation process. As depicted in the lower right part of Fig. 2, the process is outlined in three phases: (1) Factual Existence Judgment, where the annotator assesses whether the provided sentence contains verifiable facts. If no factual content is present, the sentence is categorized as 'No Faet' and requires no further annotation. (2) Reference Information Extraction. where the annotator extracts relevant reference points from the documents related to the question and answer. (3) Hallucimation-TT -yppe Judgment, where the annotator determines the type of hallucination based on the extracted reference points. If the sentence aligns with the references, it is classified as 'No Hallucination' If it contradicts the references, it is deemed a 'Contradictory Hallucination', If it lacks supporting evidence and cannot be verified, it is labeled as 'Unverifiable Hallucination' ' The above three phases will form a multi-turn dialogue in training data. Compared to the ANAH approach, which involves simultancous judgments on multiple criteria, our phased process aligns more closely with human cognitive judgment processes. The detailed data format and prompts for our annotation process are in Appendix A.
3.2 Expectation Maxiiiiion Algorithm
Simultaneously scaling up the dataset and improving the accuracy of the annotator can be formulated by the EM algorithm. For the input set X, we need to estimate two hidden variables simultaneously, the output set Y and the model parameters 0. Specifically, based on the task formulation in $ 3.1, we define the input x from the input set X of the hallucination annotator consists of question, a sentence to be annotated, and reference document. The expected output to be estimated in the data output set includes the factual information f. the key reference points from the reference document, and the type of hallucination h. We maximize the log- log-likelihood estimation of Y by alternately performing the E-Step and the M-Step to update the model parameters
arg max (x)()) [log po (X, Y [0]
(1)
E-Step. A straightforward approach to estimating is to use single model to predict annotations. However, this method lacks sufficient accuracy [45]. To improve the accuracy and stability of the estimation of Y, we introduce the self-consistency method [63], which provides more robust representation of the distribution of the Y. As shown in Fig. 2. For each input we perform multiple
samplings to yield K independent outputs y = (v'.... y ...... . .y") where the i-th output sample y" is composed of factual information (f*), reference point (r²) and hallucination type (h'). We use self-consistency metric to select the most representative sample y² among all outputs:
y = rf" hh") self-consistency())
(2)
During this selection process, we consider the hallucination type h, reference point r, and factual information f in turn. We determine the most common hallucination type h² by tallying a majority vote across all samples, denoted as h" arg max, ΣK I(h; = h). Then, we form the candidate reference set R by taking the corresponding r from the output containing the h*. We select the most "consistent" reference point r" by comparing the cosine similarities. For each r in R. we first calculate its average cosine similarity with the other elements in R. After that, we select the reference point r* with the highest average cosine similarity: r* arg max Σ)=1j sim(r', r²)). Finally, with (r",l*)) h,),, we can uniquely select the corresponding s".
M-Step. Following the robust estimation in the E-step, the M-step updates the model parameters to maximize the likelihood of the selected output y². Combining Eq. I and Eq. 2, we formulate the parameter update strategy at iteration t:
g++1 arg max Exax [Egnpa (ulz.0) log [0]]

---

Art:
Sports:
12.99%
4.4%
Nature:
Others:
0.84%
Stage
Topic
Response
Sentence
Economy:
Medicine:
Stagel 800
Stage2
800
46006
2798
209241
12188
Scl. Technol.:
4.49%
24.09% 38.94%
Thing:
Event:
3.83%
Philosophy:
Stage3
3172
196930
822520
13.54%
Personi 10.29%
Location
Geography:
6.66
3,41%
Table 1: The dataset size for ANAH-v2 in dif- ferent stages, including the number of topics, model responses, and annotated sentences.
Jurisprudence:
Religion:
6.21%
3.95%
Literature:
PolttcsMiittar:
7.18%
Figure 3: The topic distribution by chart of categories (inner) and domains (outer).
3.3 Multi-dimensional Data Scaling
Grounded in the EM algorithm, our framework operates in an iterative manner. This multi-iteration process
data growth flywheel to progressively scale up the dataset in multiple dimensions, consisting of three stages:
Stage 1: Seed Data and Basic Annotator. We utilize ANAH dataset [29] seed data, which includes over 700 topics and around 4,300 -LM-generated questions and responses. For each response, ANAH provides the hallucination type for every sentence, determined through human-in- the-loop approach. We train an initial hallucination annotator, noted as ANAH-v2 Stagel, with this seed data using the annotation method described in $ 3.1.
Stage Scaling up in Response Dimension. In Stage 1, for each question, ANAH provides responses that GPT-3.5 generates with the reference document, while InternI generates without any reference document. We first augment the dataset's model responses by collecting responses to the same existing questions from 13 additional open-source models of various sizes and series. For each model, responses were collected with and without knowledge of reference documents. The prompt details are in Appendix B. After filtering out similar model responses, these responses annotated sentence by sentence using the self-consistency pipeline with ANAH-v2 Stagel. The newly annotated data, combined with the seed data, used to train ANAH-v2 Stage2.
Stage Scaling up in Topic Dimension. We expand the topic coverage along four categories: location, person, event, and thing, paralleling ANAH's is configuration. For each topic, we generate several questions based on the provided reference documents (more details Appendix B). Then, we use the same method in Stage 2 to collect responses from multiple models and annotate the response following the same procedure as in Stage 2, using ANAH-v2 Stage2 annotator. The resulting dataset, combined with data from the previous stages, is used to train the ultimate annotator version.
Overal Statisties. The final dataset encompasses both over 33 topics, ~ 196k model responses, and ~822 annotated sentences, in English and Chinese (Tab. 1). The topics cover celebrities, events, locations, and things, and span wide array of domains, such as politics, health, and sports (Fig. 3). The statistics underscore the comprehensiveness and extensive scale of our dataset.
3.4 Applications
Hallucination Evaluation. As the accuracy of the hallucination annotators becomes satisfactory, we can apply it to automate the process of evaluating the hallucination levels of existing open-source models. After categorizing sentences into four distinct types (introduced in $ 3.1), consider type Contradictory and Unverifiable Hallucination as sentences with hallucinations, and type No Fact and No Hallucination as sentences without hallucinations. This tool enables researchers to assess the reliability and accuracy of generated texts, ensuring models can be responsibly integrated into practical applications.
Hallucination Mitigation. We further show simple re-ranking strategy to mitigate the LLM's hallucinations with the annotator, whereas more advanced strategies can be explored in future research. Specifically, we adopt our annotator 0 for response re-ranking. LIM first generates candidate responses (G G. ) by top-k sampling. Then select the best response G" with the lowest
5

---

hallucination rate over all the generated responses as below:
(ag.i.n/00... {Ac, Au
G arg min =E(1,..,N) NN
In where ag,i,n is the generated annotation type by given the input Ji,, including a question, the i-th sentence to be annotated from Gy, and a reference document. Ac and Ay means sentence type Contradictory and Unverifiable Hallucination, respectively. Ln is the sentence number of G..
4 Experiment
4.1 Experimental Setup
Implementation. In experimental framework, adopt the pre-trained InterLMM2-7B [7] model to fine-tune the hallucination annotator. Further implementation details can be found in Appendix C.
Evaluation. We use subset of the ANAH [29] data test set, which is not used for training in stage 1. To assess the performance of the annotator in predicting hallucination types, we utilize F1 and Accuracy. We also employ RougeL. [42] and BertScore [72] to compare the generated text with gold-standard human reference terms of gram, continuity, order and semantics.
4.2 Overall Results
The last rows of Tab. 2 illustrate the performance of ANAH-v2 each stage of Data Scaling in $ 3.3. The performance progressively improves with the increasing dataset number (see in Tab. 1) successive stages. This trend underscores the scalabil- ity and effectiveness of our hallucina- tion annotation framework. Remark- ably, ANAH-v2 surpasses GPT-4 with the Fl of 87. 77.78% and the accuracy of 88.03% at Stage 2. Eventually, we achieve the F1 of 89.30% and the ac- curacy of 89.55% at Stage 3.
Model
F1
ACC
R t
BERT
GPT-4
87.11
86.97
86.32
96.21
ANAH-7B
78.69
79.92
58,51
87.27
ANAH-20B
80.49
81.01
58.82
88.44
ANAH-v2-Stagel
84.45
84.85
60.10
88.43
ANAHH-v2-Stage2
87.75
88.18
67.28
90.80
ANAH-v2-Stagee
89.30
89.55
69.44
91.43
Table 2: Evaluation results for GPT4, ANAH, and ANAH-v2 at each stage, where "R" and "BERT", refer "RougeL" and "BERTScore" respectively.
Notably, the RougeL and BERTScore of GPT-4 are higher than ANAH-v2. Because GPT-4 is used for the initial pre-annotation during the construction of ANAH [29]. Subsequently, humans refine these areeaanttations,, and humans tend to not change the pre-an pre-annotations, This methodology inherently aligns the final 'golden' answers closely with the outputs by GPT-4. Therefore, we tend use "accuracy" as our primary metric because the type judgment is determinative of the annotation quality. For example, an annotation that wrongly judges type (low accuracy) but finds the correct reference frugment (high Bougee./EETTSoort remains completely unacceptable. In addition, we conduct an LLM-based evaluation to exclude similarity due to pre-annotations Appendix
We also observe that ANAH-v2 already outperforms ANAH-20B at Stage (84.85% v.s. 81.01% in accuracy) with only 7B parameters, when being trained on the same hallucination corpus. This superior performance is attributed to the innovative multi-turn dialogue training strategy ($ 3.1).
4.3 Ablation Studies
Impact of Ceff-Consistency. To verify the effectiveness of seff-consistency during inference in E-Step (introduced in & 3.2), we compare the performance of the annotator with different self- consistency settings in Tab. 3. When the annotator model with the same training data at each data scaling stage, the inference strategy with self-consistency (w/ SC) consistently outperforms without self-consistency (w/o SC), where the annotator generates only once for each input. Therefore, self-consistency improves the accuracy and stability of the estimation of hallucination annotations.
"The first three rows of data are from ANAH [29].

---

Model
Train Data
Infer Strategy
F1
ACC
R1
BERT
w/o SC
80.95
81.67
58.26
88.70
ANAHH-v2-Stagel
SC
84.45
84.85
60.10
88.43
w/o SC
w/o SC
83.80
83.94
62.93
89.20
SC
83.98
84.24
64.92
90.01
ANAH-v2-Stee 2-Sagg22
w/ SC
w/o SC
84.65
85.15
61.08
88.47
w/ SC
w/ SC
87.75
88.18
67.28
90.80
w/o SC
w/o SC
86,24
86.67
66.10
90.26
w/o SC
w/ SC
87.78
88.18
68.18
91.01
ANAH-v2 3-agee
w/ SC
w/o SC
87.71
88.03
67.45
90.63
SC
SC
89.30
89.55
69.44
91.43
Table Ablation study for annotators in different self- consissssit settings. Here, for Infer Strategy, "w/ SC" means inference with self-consisteney, which is the default setting of ANAH-v2. "w/o SC means inference without self celf-consistency, where the annotator generates only once for each input. For Train Data, "w/ SC" means the training data from the previous stage is generated by self-consistency, where the default setting of ANAH-v2, while "w/o SC" means the train data is generated without self-consistency.
Model
Setting
F1 f
ACC 1
R
BERT
progressive 89.30
89.55
69.44
91.43
ANAH-v2-Staae3
non-progressive 85.88
86.36
66.10
90.26
Table 4: Ablation study for annotators trained with progressive and non con-prorressive data scaling. Here, "progressive" means that the training data progressively annotated the continually updated annotator, which is the default setting of ANAH-v2. thon-progressive" means that the training data scaling only leverages annotations generated by the basic annotator from Stage 1.
In M-Step, we train the model on data from the E-Step of the preceding iteration. We observe that when the annotator model with the same inference strategy, the model trained on self-cone consistenlly processed data (w/ SC) surpasses the performance with data generated through single pass (w/o SC). This finding indicates that training data processed through self- cofsisseist leads stronger annotator. This improvement can be attributed to the reduced distribution variance between the inferred labels and true labels.
Impact of Progressive Data Scaling. To assess the impact of progressive data scaling (introduced in $3.3), we compare the performance of annotators with different types of data scaling in Tab. In our progressive approach, the updated annotator from Stage 2 is employed to annotate the responses from additional topics, continuously enriching the training data. Conversely, in the non- progressive approach, the basic annotator from Stage is employed to generate annotations for the additional training data during Stage 3. With the same size of training data, the annotator trained on non-proe pon-progreesive data scaling underperforms that with our progressive data scaling, proving the effectiveness of our progressive data scaling.
Impact of Training Strategy. We also analyze different training strategies for annotators in different data scaling stages in Tab. 5. In our default training process, mix the newly annotated data with old data to re-train an annotator. Alternatively, we only use the newly annotated data to further train the annotator model from the previous stage. The results demonstrate that our training strategy with mixed training data performs better than further training with new data. The integration of different data qualities across training stages improves the robustness of the annotator model.
4.4 Generalization Capability Analysis
We further validate the effectiveness of ANAH-v2 on other hallucination detection datasets using two third-party datasets: HaluEval [40] for English and HalluQA [14] for Chinese. Each dataset provides four components: questions, reference documents, responses, and labels indicating whether the responses contain hallucination. For each question, we let ANAH-v2 judge the type of responses containing and not containing the hallucination separately. Note that in HaluEval we only use the QA
7

---

Model
Train Strategy
F11
ACC
R 1
BERT
mix 87.75
88.18
67.28
90.80
ANAH-v2-Stage2
further
85.50
85.91
62.15
89.30
mix 89.30
89.55
69.44
91.43
ANAH-v2-Stage3
further 87.73
86.52
68.58
91.03
Table 5: Ablation study for annotator in different train strategy settings. Here, "mix" means that the new data generated in the current iteration is mixed with the old data to train new annotator, which the default setting of ANAH-v2. "further" means that only the new data used further train the annotator from the previous stage.
samples and in HalluQA, only use the samples that provide textual reference document, which aligns with our annotator's designed setting.
The primary metric we use for evaluation is Ac-
curacy in determining the type of response. We compare the zero-shot performance of ANAH- v2 with current SOTA results on HaluEval achieved by KnowHalu [71] and baseline results by GPT-4
Dataset
Model
Method
ACC1
GPT4
Zero-Shot
65.05
WiKiChat [S4]
49.10
GPT3.5
HaluBval
56,90
KnowHale
80.30
The results in Tab. 6 reveal that our annota- tion model achieves notable accuracies on both HaluEval and HalluQA. Remarkably, ANAH- v2-Stage3 obtains new SOTA accuracy on HaluEval (81.54%) and HalluQA (94.44%) even under a zero-shot setting, underscoring the gen- eralization capability of ANAH-v2. Moreover, we find that ANAH-v2-Stagee outperforms the annotators from Stagel and Stage2, further prov- ing the data scaling strategy effectively stabilizes performance when dealing with unfamiliar re- sponses.
HaluBval
61.00
HaluEval
Starling-7B
KnowHalu
80.70
ANAH-v2-Staggl
Zero-Shot
79.85
AN-H-v2-Stage2
Zero-Shot
81.24
ANAH-v2-Stage3
7em-Shot
81.54
GPT4
Zero-Shot
62.81
ANAH-v2-Stage
Zero
91.74
HalluQA
ANAHI-v2-Stuge ANAHH-v2-Stgee3
Zevo-Shol
92.63
Zero-Shot
94,44
Table 6: Annotator accuracy using different models and methods on HaluEyal and HalluQA.
4.5 Application
Hallucination Evaluation Benchmark. Our ANAH-v2 dataset and annotator can serve as a benchmark for the hallucination levels in generated texts by existing models. As shown in Tab. 7, we evaluate the performance of various L.Mss, including InternI [7], Qwen1 1.5 [2]. Baichuan2 [4], Mistral [33, 34], DeepSeek-L.LMM [6], and Llama2 [59], spanning different model sizes. We also offer detailed evaluation results on different languages and categories of topics to deepen our understanding.
We find that all models exhibit superior performance in English compared to Chinese, underscoring the need for further research to understand and mitigate unguuag-ddeendent discrepancy. The performances of all models with reference documents are better than those without. Qwenl.5- 14BBBB achieves the lowest hallucination rate when using reference documents (5.33%) and Deepseek- 67B achieves the lowest hallucination rate when reference documents not provided (47.17%). Moreover, we find no clear trend in the performance distribution across four categories of topics. In addition, the results of different stages of annotators in Tab. A2, A3, and 7 show that there is consistent trend and fixed biased ordering relationship between L.MMs, thus confirming the reliability of our assessment method. More details in Appendix E.
Hallucination Mitigation. Besides being used to measure hallucination levels, ANAH-v2 can also be used to mitigate hallucinations. We use the QA samples from HaluEval, which comprises questions and correct answers from HotPotQA [68]. We use two models InternLm2-7B and LLaMA2-7B. For each model, we generate 36 candidate responses by top-k sampling (k=40). then re-rank the responses using our annotator. To quantify the hallucination degree, we employ RougeL, BertScore. NLI, and QuestionEval. These metries measure the congruence between the generated responses with the golden responses and/or reference documents.
'The first six of data from KnowHalu [71].

---

Person
Event
Thing J
Location J
Model
Setting
Overall
ZH
EN
ZH
EN
ZH
EN
ZH
EN
w/o Ref
87.84
87.24
47.65
91.37
73.83
89.49
77.13
94.26
82.92
InternI.M2-7B
w/ Ref
19.02
26.12
5.57
19.26
4.20
19.40
2.59
9.77
3.26
w/o Ref
78.20
74.67
47.25
82.36
72.32
82.16
80.29
87.32
81.49
I2-2
w/ Ref
16.52
19.99
4.25
15.23
7.00
19.66
7.20
3.42
5.76
w/o Ref
80.09
79.22
52.81
82.61
75.7
83,26
73.85
86.56
78.09
Qwenl.5-7B
w/ Ref
6,96
5.82
2.77
5.27
3.76
9.70
3.69
4.90
4.40
w/o Ref
68.82
65.63
44.91
70.25
68.24
72.36
70.76
73.37
72.69
Qwenl.5-144
w/ Ref
5.33
5.01
1.23
4.56
2.02
7.38
2.70
2.53
2,00
w/o Ref
61.62
56.49
29.76
61.62
56.78
67.42
62.92
67.97
64.36
Qwenl..5777 1,5-2B
w/ Ref
15.89
19.27
4.62
13.85
3.18
18.99
3.80
5.50
4.26
Baichuan2-7B
w/o Ref
73.99
72.13
44.99
75.77
65.98
76.84
73.01
71.51
74.17
w/ Ref
43.68
61.71
64.56
37.87
26.41
35.4
29.17
54.3
14.51
w/o Ref
69.85
67.02
41.24
71.63
63.13
73.32
66.2
68.77
71.35
Raichuan2-13RB
w/ Ref
38.39
58.86
60.53
43.20
21.9
25.74
17.81
28.99
7.23
w/o Ref
85,40
89.98
52.32
87.03
72.33
87.41
71.97
91.19
77.25
Mistral-7B
w/ Ref
30.24
42.66
22.83
30.85
13.77
26.02
27.15
42.11
7.23
w/o Ref
76.12
80.96
30.75
76.78
55,32
83,32
61.61
87.28
65.51
Mistral-8x7B
w/ Ref
7.95
8.29
2.78
6.17
5.84
7.94
3.99
w/o Ref
64.46
65,98 .8
39.62
67.59
60.69
69.51
56.15
69.29
59.15
Deepseek-7B
w/ Ref
23.02
6.73
44.95
28.38
4.92
25.00
24.25
18.18
12.61
w/o Ref
47.17
54.91
15.81
46,28
31.48
65.57
34,23
59.96
36.02
Deepseck-67B
w/ Ref
12.05
12.61
4.17
9.52
2,00
15.79
13.36
18.65
8.33
w/o Ref
84,22
88.36
52.00
84,95
74.18
92.48
77.89
89.84
78.91
Llama2-7B
w/ Ref
58.16
82.5
10.64
76.96
10.00
64.72
12.33
69.75
20.48
w/o Ref
78.84
80.26
43.18
81.88
70.25
87.85
70.52
84.44
73.94
Llama2- 13B
w/ Ref
52.17
79.43
14.81
47.85
4.00
49.59
11.72
77.50
27.53
Table 7: Hallucination rate of open-source models according to ANAH-v2 annotator and dataset.
Model
Setting
QuestEval f
NL11 f
BERT f
RourgeL 1
baseline
37.84 38.50
31.25
83.76
19.34
LLaMA2-7B
re-rank
36.03
84.45
21.92
baseline
37.33
25.00
83.57
20.55
InternI.M2-79
re-rank
38.89
37.01
84.57
22.39
Table 8: Evaluation results for hallucination mitigation with LLaMA2-7B and InternLM2-7B on HaluEval. Here, "baseline" means the direct generation results, and "re-rank" means the results with our re-ranking mitigation method.
Results in Tab. show clear reduction of hallucination levels after the re-ranking process via annotator. For instance, the NLI metric for LLaMA2-7B shows a notable increase, rising from 25.00% to 37.01%. This suggests that the application of our annotative approach can significantly mitigate the issue of hallucinations in language model outputs.
5 Conclusion and Future Work
In this paper, we aim to explore scalable framework for the oversight of LLM hallucinations. Through iterative self-training, we progressively expand the diversity and scale of the dataset and improve the accuracy of the hallucination annotator. The finally obtained ANAH-v2, for the first time, outperforms GPT-4 in various hallucination detection benchmarks with only parameters and obtains superior zero-shot performance third-party hallucination detection benchmarks. ANAH- v2 not only provides an automatic hallucination evaluation benchmark with the scaled dataset, which paves the way for future research on hallucination mitigation but also exhibits potential in

---

hallucination mitigation by the simple re-ranking strategy. We believe ANAH-v2 can also benefit more hallucination mitigation strategies such tine-grained RLHF.
With the large-scale dataset as seed data, future work can explore creating hallucination annota- tion data in other NLG tasks such as dialogue generation. Another direction is to improve the generalizability of the annotator across different languages, tasks, and topics.
Acknowledgement
We thank the anonymous reviewers and area chair for their helpful comments. This project is funded in part by the Hong Kong Generative AI Research and Development Center (HKGAI) under the Innovation and Technology Commission (ITC)'s InnoHK. Dahua Lin is a PI HKGAI under the InnoHK. This project is also supported by the Shanghai Artificial Intelligence Laboratory. The authors would like to thank Zehui Chen and Kuikun Liu for their valuable suggestions and comments.
References
[1] Achiam, Adler, S., Agarwal, S., Ahmad, L., Akkaya, I,, Aleman, F.L., Almeida, D., Altenschmidr, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.0877 088774 (2023)
[2] Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al.: Qwen technical report. arXiv preprint arXiv:2309. (2023)
[3] Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al.: Constitutional ai: Harmlessness from ai feedback. arXiv preprint 1rXiv:2212.0800 (2022)
[4] Baichuan: Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305
(2023), https://arxix: org/ abs/22000 10305
[5] Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z. Yu, T., Chung, W., Do, Q V., Xu, Y., Fung, P.: multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023)
[6] Bi, X.. Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al.: Deepseek I1m: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401 #XXv:2401.002554 (2024)
[7] Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., et al.: Internlm2 technical report. arXiv preprint arXiv:2403. 17297 (2024)
[8] Cao, Z., Yang, Y., Zhao, H.: Autohall: Automated hallucination dataset generation for large language models. arXiv preprint arXiv:2310.00259 (2023)
[9] Chen, X., Song, D., Gui, H., Wang, C., Zhang, N., Yong. J., Huang, F., Lv, C., Zhang, D., Chen, H.: Factchd: Benchmarking fact-conflicting hallucination detection. IJCAL abs/2310.1208 (2023), https: h //tpp/app...a wtw.ppi..smmanicccchhhiioo
org/ haggCorppusID... : 264289140
[10] Chen, Z., Du, W., Zhang, W., Liu, K., Liu, J., Zheng, M., Zhuo Zhang. S., Lin, D., Chen, K., Zhao, F.: T-eval: Evaluating the tool utilization capability step by step. CoRR abs/2312.14033 14033, attps://det org/10 http//10.885///// 485550/arXiv arxiv.
2312. 14033
[11] Chen, Z., Liu, K., Wang. Q., Zhang, W. Liu, J.. Lin, D., Chen, K., Zhao, F. Agent-flam Design- ing data and methods of effective agent tuning for large language models. CoRR abs/2403.1288) (2024). http:://c...rr//0..550//A/AA 2403. 1403.12881
https: http:::/ddo...rg//// 48550/
[12] Chen, Z., Mao, H., Wen, H., Han, H., Jin, W., Zhang, H., Liu, H., Tang, J.: Label-free node classification on graphs with large language models (Ilms.. arXiv preprint arX0v:2310004668 (2023)
10

---

[13] Chen, Z., Deng, Y., Yuan, H., Ji, K., Gu, Q.: Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335 (2024)
[14] Cheng, Q., Sun, T., Zhang, W., Wang, S., Liu, X., Zhang, M., He, J., Huang, M., Yin, Z., Chen, K., et al.: Evaluating hallucinations in chinese large language models. arXiv preprint arXiv:2310003368 (2023)
[15] Chuang, Y.S., Xie, Y., Luo, H., Kim, Y., Glass, J., He, P:: Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint 2XXi:2309.00888 (2023)
[16] Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P. Zaharia, M. Xin, R.: Free dolly: Introducing the world's first truly open instruction-tuned IIm (2023), https:/ /wwww databricks com http///og/////////////// dolly-first- htt.://rsst...oooooooooaaaaaaaea commercially-viaal.....
instruction- connetddiii-tn------e 1imed-11m
[17] Contributors, L.: Lmdeploy: A toolkit for compressing, deploying, and serving 1lm. https: //github.
(2023)
[18] Daheim, N., Dziri, N., Sachan, M., Gurevych, I., Ponti, E.M.: Elastic weight removal for faithful and abstractive dialogue generation. arXiv preprint arXiv:2303.17574 (2023)
[19] Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., Weston, J.: Chain-of- verification reduces hallucination in large language models. arXiv preprint arXiv:2309. 11495 (2023)
[20] Du, Y., Li, S., Torralba, A., Tenenbaum, J.B., Mordatch, I.: Improving factuality and reasoning in language models through multiagent debate (5 2023), http: / arriv. org/ abs/ abs/b305 2305. 14325
[21] Durmus, E., He, H., Diab, M.: Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 55555055777 (2020)
[22] Dziri, N., Kamalloo, E., Milton, S., Zaiane, O., Yu, M., Ponti, E.M., Reddy, S.: Faithdial: A faithful benchmark for information-seeking dialogue. Transactions of the Association for Computational Linguisties 10, 1473-1490 (2022)
[23] Garg, S., Peitz, S., Nallasamy, U., Paulik, M.: Jointly learning to align and translate with transformer models. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-ICCLL)), pp. 4453-4462 (2019)
[24] Gilardi, F., Alizadeh, M., Kubli, M.: Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences 120(30), e2305016120 (2023)
[25] Gulcehre, C., Paine, T.L., Srinivasan, S., Konyushkova, K., Weerts, L.,, Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., al.: Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.088: 088998 (2023)
[26] Gupta, P., Wu, C.S., Liu, W., Xiong, C.: Dialfact: A benchmark for fact-checking in dialogue. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 3785-3801 (2022)
[27] He, C., Li, W., Jin, Z., Wang, B., Xu, C., Lin: Opendatalab: Empowering general artificialintel- ligence with open datasets. ht / htte:://opeddataaa
com (2022)
[28) He, Z., Huang, C.Y., Ding, C.K.C., Rohatgi, S., Huang, T.H.K.:: in a crowdsourced data annotation pipeline, gpt-4. In: Proceedings of the CHI Conference Human Factors in Computing Systems. pp. 1-25 (2024)
[29] Ji, Z., Gu, Y., Zhang, W., Lyu, C., Lin, D., Chen, K.: Anah: Analytical annotation of hallucina- tions in large language models, arXiv preprint 2xXx::2405.20311 (2024)
[30] Ji, 7.., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y., Madotto, A., Fung, P.: Survey of hallucination in natural language generation. ACM Computing Surveys (2022)
11

---

eettps//aclanthiee [31] Ji, Z., Liu, Z. Lee, N., Yu, T., Wilie, B.. Zeng. M., Fung, P:: RHO: Reducing hal- lucination in open-domain dialogues with knowledge grounding. In: Rogers, A., Boyd- Graber, J., Okazaki, N. (eds.) Findings of the Association for Computational Linguistics: ACL 2023. pp. 4504-4522. Association for Computational Linguistics, Toronto, Canada (Jul 2023). https://doi.or//10.. 2023. findings-nnlall 275
org/
[32] Ji, Z., Yu, T., Xu, Y. Lee, N., Ishii, E., Fung, P.: Towards mitigating hallucination large language models via self-reflection. EMNLP Findings (2023)
[33] Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C.,, Chaplot, D.S., Casas, D.d.I,, Bressand, F. Lengyel, G., Lample, G., Saulnier, L.,, et al.: Mistral 7b. arXiv preprint arXiv:2310.066 (2023)
[34] Jiang, A.Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D.S., Casas, D.d.I., Hanna, E.B., Bressand, F,, et al.: Mixtral of experts. arXiv preprint 2Xiv:2401.04086 (2024)
[35] Kamalloo, E., Dziri, N., Clarke, C.L.A., Rafici, D.: Evaluating open-domain question answering in the era large language models. In: Rogers, A., Boyd-Graber, J.L.,, Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023. pp. 5591-5606. Asso- ciation for Computational Linguisties (2023). https://doi.org// http://0..16553/1/2222222 000.://66651110002...oo....ooo /ttps://doi....
1111111.8866666 18653/v1//eee v122/23...2222 acl acl-long.300
[36] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4015-4026 (2023)
[37] Laban, P., Schnahel, T., Bennett, P.N., Hearst, M.A.: Summac: Re-visiting nli-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics 10, 163-177 (2022)
[38] Lee, N., Wattanawong, T., Kim, S., Mangalam, K., Shen, S., Anumanchipali, G., Mahoney, M. W., Keutzer, K., Gholami. A.: Llm2llm: Boosting Ilms with novel iterative data enhancement. arXiv preprint arXiv:2403. 15042 (2024)
[39] Lei, D., Li, Y., Wang, M., Yun, V., Ching, E., Kamal, E., et al.: Chain of natural lan- guage inference for reducing large language model ungrounded hallucinations. arXiv preprint arXiv:2310.0395 (2023)
[40] Li, J., Cheng, X., Zhao, W.X., Nie, J.Y., Wen, J.R.: Halueval: A large-scale hallucination evaluation benchmark for large language models. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.
(2023)
[41] Li, K., Patel, O., Viégas, F., Pfister, H., Wattenberg, M.: Inference-time intervention: Eliciting truthful answers from a language model. arXiv preprint arXiv:2306.03341 (2023)
[42] Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out. pp. 74-81 (2004)
[43] Liu, J., Fu, Y., Xie, R., Xie, R., Sun, X., Lian, F., Kang, Z., Li, X.: Phd: A prompted visual hallucination evaluation dataset. arXiv preprint arXiv:2403.1 1116 (2024)
[44] Liu, T. Zhang, Y., Brockett, C., Mao, Y., Sui, Z., Chen, W., Dolan, B.: A token-level reference-free free hallucination detection benchmark for free-form text generation. arXiv preprint arXiv:2104.08704 (2021)
[45] Mena, J., Pujol, O., Vitrià, J.: survey on uncertainty estimation in deep learning classification systems from bayesian perspective. ACM Computing Surveys (CSUR) 54(9), 1-35 (2021)
[46] Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.t., Koh, P.W., lyyer, M., Zettlemoyer, L.,, Hajishirzi, H.: FActScore: Fine-grained atomic evaluation of factaal precision in long form text generation. In: EMNLP (2023), https: arxiv. org/abs/ abbs/305... 14251
12

---

[47] Mishra, A., Asai. A., Balachandran, V., Wang, Y., Neubig, G., Tsvetkov, Y. Hajishirzi, H.: Fine-grained hallucination detection and editing for language models, arXiv preprint arXiv:2401.008 106655 (2024)
[48] Muhlgay, D., Ram, O., Magar, I., Levine, Y., Ratner, N., Belinkov, Y., Abend, O., Leyton- LeynonBBBronw, K., Shashua, A., Shoham, Y.: Generating benchmarks for factuality evaluation of language models, arXiv preprint arXiv:2307.06908 (2023)
[49] None: Sharegpt (2023), https::/ co/ wtt.a/ttaettt//yoooo/////rrrrrre
[50] Pei, X., Li, Y. Xu, C.: Gpt self-supervision for a better data annotator. arXiv preprint arXiv:2306.04349 (2023)
[51] Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., et al.: Kilt: a benchmark for knowledge intensive language tasks. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2523-2544 (2021)
[52] Rebuffel, C., Roberti, M., Soulier, L., Scoutheeten, G., Cancelliere, R., Gallinari, P.: Controlling hallucinations at word level in data-to-text generation. Data Mining and Knowledge Discovery 36(1), 318-354 (2022)
[53] Savelka, J.: Unlocking practical applications in legal domain: Evaluation of gpt for zero-shot semantic annotation of legal texts. In: Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law. pp. 447 -51 (2023)
[54] Semnani, S., Yao, V., Zhang, H., Lam, M.: Wikichat: Stopping the hallucination of large language model chatbots by few-shot grounding wikipedia. In: Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 2387-2413 (2023)
[55] Shi, W., Han, X., Lewis, M., Tsvetkov, Y., Zettlemoyer, L., Yih, S.W.t.: Trusting your evidence: Hallucinate less with context-aware decoding. arXiv preprint Xix:23305.14735 (2023)
[56] Singh, A., Co-Reyes, J.D., Agarwal, R., Anand, A., Patil, P., Liu, P.J., Harrison, J., Lee, J., Xu, K., Parisi, A., et al.: Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585 (2023)
[57] Sun, K., Xu, Y.E., Zha, H., Liu, Y., Dong, X.L.: Head-to-tail: How knowledgeable are large lan- guage models (Im)? aka will Ilms replace knowledge graphs? arXiv preprint arXiv:2308.10111 (2023)
[58] Sun, Z., Shen, Y., Zhou, Q., Zhang, H., Chen, Z., Cox, D., Yang, Y., Gan, C.: Principle-driven allf-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems 36 (2024)
[59] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.099 00988 (2023)
[60] Varshney, N., Yao, W. Zhang. H., Chen, J.. Yu, D.: A stitch in time saves nine: Detecting and mitigating hallucinations of Ilms by validating low-confidence generation. arXiv preprint arXiv:2307.0m 00397777..777 (2023)
[61] Vectara: vectara hallucination evaluation model. http:://hugginnface..///tttt
[62] Wang, A., Cho, K., Lewis, M.: Asking and answering questions to evaluate the factual consis- tency of summaries. arXiv preprint arXiv:2004.0422 (2020)
[63] Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D.: Self-consistency improves chain of thought reasoning language models. arXiv preprint arXiv:2203. 1 (2022)
13

---

[64] Wang. Z., Mao, S., Wu, W., Ge, T., Wei, F., Ji, H.: Unleashing cognitive synergy in large language models: A task-solving agent through mulli-persona selfcollaboration. arxiv preprint arXiv:2307.0532. 055000 1(2), 3 (2023)
[65] Weng, R., Yu, H., Wei, X., Luo. W. Towards enhancing faithfulness for neural machine translation. In: Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 2675-2684 (2020)
[66] Wu, Z.,, Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P., Smith, N.A., Ostendorf, M., Hajishirzi, H.: Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01993 (2023)
[67] Yang, S., Sun, R., Wan, X.: new benchmark and reverse validation method for passage-level hallucination detection. arXiv preprint arXiv:2310.0644 (2023)
[68] Yang, 7., Qi, P., Zhang, S., Bengio, Y., Cohen, W.W., Salakhutdinov, R., Manning, C.D.: HotpotQA: dataset for diverse, explainable multi-hop question answering. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) (2018)
[69] Yuan, W., Pang, R.Y., Cho, K., Sukhbaatar, S., Xu, J., Weston, J.: Self-rewarding language models, arXiv preprint arXiv:2401.10021 (2024)
[70] Yuan, Z., Yuan, H., Li, C., Dong, G., Tan, C., Zhou, C.: Scaling relationship on learning
mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825 (2023)
[71] Zhang, J., Xu, Gai, Y., Lecue, F., Song, D., Li, B.: Knowhalu: Hallucination detection via fulti-form knowledge based factual checking. arXiv preprint arXiv:2404.02935 (2024)
[72] Zhang*. T., Kishore*, V., Wu*, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text
generation with bert. In: International Conference on Learning Representations (2020), https:
/ /openreview
14

---

A Training Prompt
As described 3.1, our annotation process consists of three phases: (1) Factual Existence Judgment via the prompt in Fig. A1, (2) Reference Information Extraction via the prompt in Fig. A2 (3) Hallucination-Tyyl Type Judgment via the prompt in Fig. A3.
B Data Scaling Details
As described in $ 3.3, collect model responses via Fig. A5. The open-source models in- clude [ntemLL [7]. Baichuan2 (7B& 13B) [4], LLama2 (7B& 13B) [59], Qwen1.5 (7B& 14B&72B) [2], Deepseek (7B&67B) [6], and Mistral (7B&7x8B) [33, 34].
We automate the topic selection based on occurrence frequency via Google Ngram Viewer 4 and retrieve corresponding reference documents from pre-training databases [27].
We generate questions on each topic via Fig. A4.
C Implementation Details
In our experimental framework, adopt the pre-trained InternL.M2-7B [7] model to fine tuning the hallucination annotator.
In E-Step, generate responses by implementing sampling via the LMDeploy library [17]. During each iteration, we generate candidate responses per input and apply self-consistency quality control mechanism to them. The decoding strategy involves the top-k (k 40) sampling with temperature of 0.8.
In M-Step, we train the annotator model with the following settings and hyper- parameters: the epoch 1, the learning rate le-5, and the AdamW optimizer is with a linear scheduler, the maximum sequence length is set to 32k. Additionally, following the configuration in ANAH [29], we perform multi-task setting where additional tasks such as dialogue generation from ShareGPT [49] and Dolly [16] are integrated with the fine-grained hallucination annotation. Our model is trained on 32 NVIDIA A100 GPUs.
D Addtional Evaluation for the Overall Results
To exclude the similarity due the impact of pre- annotations described in $ 4.2, we conduct an LLM-base evaluation assess the consistency of generated reference points with the source document. Specifically, we fol- lowed the prompt FactScore [46], which aims clarify whether the generated reference points are supported by the given source document, rather than simply calculating the similarity between them using metrics such as Rougel. or SERTScore. We employ the InterLMM--7B-hha [7] as the estimator.
Model
Score
GPT-4
84.39
ANAH-7B
80.60
ANAH-20B
81.51
ANAH-V2 1-AH--2-Stagel
83.63
ANAH-V2-Stage2 84.54
ANAH-V2--Staee3 86.36
As shown Tab. A1, the results indicate that the reliability of our model's 's generated reference points progressively improves and ultimately exceeds that of GPT4, This trend is consistent with Fl and ACC in Tab 2.
Table A1: The score assessing the con- sistency of generated reference points with the source documents.
E Hallucination Evaluation
To assess the reliability of our hallucination annotator, we measure the hallucination levels of the above LLMs using the annotator from different stages. Tab. A2, A3, and 7 show the results measured by annotator ANAH-v2 from Stage 1, 2, and 3, respectively. The trends in these three tables are
"uttps://books hww.bboooks.ooggl......ooo com/ngrams/
15

---

consistent where Dwen1.5-144 14B achieves the lowest hallucination rate with reference documents and DeepseckLMM667B achieves the lowest hallucination rate without reference documents. This consistency and fixed biased ordering relationship between LLMs confirm the reliability of our assessment method.
F Limitation
Although this study presents novel of LLM hallucinations and achieves significant improvements in hallucination annotation, there are some limitations.
self-training framework for the scalable oversight
Despite the progressive scaling and increasing accuracy of the hallucination annotator, there may still exist non-negligible margin of error in the annotations. This margin could affect the convergence of the model and the quality of the final hallucination annotator. Furthermore, the success of our framework is measured largely by its performance on our own dataset and other benchmarks such as HalluEval and HalluQA. However, these datasets might not encompass the full spectrum of real- world scenarios where hallucinations pose problem. Lastly, this work primarily uses InternLM2-75 the backbone of the ballucination annotator. Other different underlying models and different numbers of parameters not explored.
In addition, the EM algorithm, which is the theoretical foundation of our framework, may also introduce some problems. For example, the EM algorithm is sensitive to initial conditions, which would impact the convergence process. Although we employ many methods to ensure the stability of the training process, such as selecting high-quality, human-labeled hallucination dataset as the seed and using progressive scaling strategy, we cannot claim that we have eventually converged to globally optimal solution. Moreover, the iterative EM algorithm requires computational effort. Our method uses 32 A100 GPUs to iteratively train the 7B model. took approximately 100 hours for inference and training. Based on the price of the computing platform Lambda (1.29 USD per GPU per hour), it costs 4,128 USD. However, using the "manual + GPT4-assistedd annotation model, described in ANAH [29] (0.9 USD and 20 minutes per annotation), would take 177,237 USD and 55,643 hours to reach the size of the dataset in our work. So we believe our method is better trade-off between computing resources and labour+APl costs, which is acceptable.
G Broader Impacts
By exploring the hallucination annotation and mitigation in LLMs, this paper contributes to the development of more reliable and trustworthy AI technologies. Our innovative multi-iterative self- training framework significantly reduces the reliance on expensive and time-consuming manual annotations by automating the hallucination detection process. Our hallucination annotator offers benchmark for the research community evaluating the hallucination levels of existing open-source models. Additionally, we provide a large-scale and diverse dataset from which the broader research community can benefit, fostering further innovation and study in this domain.
16

---

English Prompt:
You will act fact checker, and will provide you with question and corresponding partial answer. Your task is to determine whether the content of the answer contains verifiable facts.
## Judgment Criteria:
Verifiable Facts: Specific, objective points of information that can verified through data, research results, other reliable sources. Examples include statistical data, historical events, scientific laws, and specific case studies.
Non-factual Descriptions:
Personal opinions, subjective judgments, unverifiable statements.
## Task Process:
1. Carefully read the question, which as follows: {question)
2. Carefully read the partial answer, which is as follows: [annotation]
3. Conduct the Analysis: Based on the above judgment criteria, determine if the answer contains verifiable facts.
If there are no verifiable facts in the answer, output "<No Facts>".
there are verifiable facts in the answer, output "<Facts Present>".
Chinese Prompt:
- - TT - ММ
## FI M.
3 M /
#
# $ 1
## 4E 12
1. (
(question)
2. M 1 F: (annotation)
3. :: 1: 3H
7 M
J
# "
Figure A1: Prompts for factual existence judgment.
17

---

English Prompt:
You will act an information extractor. I will provide you with question, a related reference document, and a partial answer to that question. Your task is to extract information from the reference document that is relevant to the question and answer.
## Operational Steps:
1. Carefully read the question, which as follows: {question]
2. Carefully read the partial answer, which is as follows: (annotation)
Identify information most relevant to the question and answer. This information may be completely the same, partially similar, confficting with the content of the answer. The reference document as follows: (reference)
3. Analyze the Reference Document:
4. List the Relevant Information: List all the relevant information found in order, separated by <SEP> if there multiple pieces of information.
5. Output When No Information Is Found: If relevant information is found, output <No Reference Information..
Chinese Prompt:
MR - THERE 1
## #
1. म
[question) (annotation)
3. x
#
(reference)
4. A.
|
5. A B
Figure A2: Prompts for reference information extraction.
18

---

English Prompt:
You will act 'Hallucination' annotator. will provide you with question, a partial answer that question, and related reference points. You need to determine whether the provided answer contains any hallucinatory content and annotate the type of hallucination. 'Hallucination' refers to content that contradicts the reference points or is unsupported by them. ## Judgment Criteria:
1. No Hallucination: If the answer is completely consistent with the reference points and does not introduce any contradictory information, output: <No Hallucination>.
2. Contradiction: If the answer clearly contradicts the reference points, output: <Contradictory>.
3. Unverifiable: If the answer contains information not mentioned in the reference points and cannot be supported verified by them, output: <Unverifable>.
## Task Process:
1. Carefully read the question, which is as follows: {question)
2. Carefully read the partial answer, which is as follows: {annotation]
3. Carefully read the reference points, which
follows: (reference)
4. Conduct the analysis: Based on the above judgment criteria, determine the answer contains hallucinations
and output the type of hallucination.
Chinese Prompt:
1
2 TE T
" B
## *E
1. J2T55: *
1: <
2. : XFA. WAA.E
3. =E
i
161 <E>
## 1L
1. И
1 F: Lquestion)
2. +
# [ 20 F: [annotation]
3. +
F. (reference)
4. BT
Figure A3: Prompts for hallucination type judgment.
19

---

English Prompt:
would like you to act as a question generator. will provide references and you will generate 10 questions about "(topic)" based on the reference. The specific requirements are as follows: 1. the questions can be fully answered based only on the reference document, i.e. the answers the questions are fully contained in the reference document. The questions should be objective and not too subjective open-ended.
2. the 10 questions should be of many different types possible, e.g. what, when, where, why. Questions be asked from different perspectives, e.g. descriptions, explanations, reasons, etc. Ensure that the questions are of different types and cover all aspects of the information.
3. 10 questions can cover different levels of knowledge, from general, basic knowledge to more specialized, complex subject knowledge domain knowledge.
4. have only one question per item.
Reference: (reference document)
Please list the 10 questions directly based on the above reference without any explanation:
Chinese Prompt: -
I "(topic)" 1110
$
2. 10
#+
1010
AR НA / AAA
4. -
freference document)
$
11
Figure A4: Prompts for question generation.
English Prompt:
Reference document: (reference document)
Please answer the question based on the above reference: (question) )
Chinese Prompt:
$ (reference document]
# Mi {question]
Figure A5: Prompts for answering.
20

---

Person
Event
Thing J.
Location 1
Model
Setting
Overall
EN
ZH
EN
ZH
w/o Ref
74.8
94.03
40.74
86.42
67.68
93.83
75.03
86.24
53.03
IntemLLM-77B
w/ Ref
13.09
18.94
5.48
33.93
8.77
16.03
2.7
18.09
4.85
w/o Ref
63.94
91.87
38.01
75.24
71.76
90.04
78.02
78
54.86
Internl M2--20B
w/ Ref
12.84
29.76
5.65
29.24
12.72
13.78
7.67
16.09
12.62
w/o Ref
64.04
87.68
42.61
70.86
65.62
88.05
72.62
79.91
54.22
Qwen 1.5-7B
w/ Ref
6.92
8.62
4.3
16.32
10.57
8.86
3.44
9.46
6.54
w/o Ref
55.88
71.83
36.81
59.78
67.15
79.94
69.02
67.03
51.62
Owenl.5-144
w/ Ref
5.96
10
2.29
12.88
2.93
13.19
2.02
7.77
6.67
w/o Ref
49.25
67.67
25.47
56.33
58.88
77.72
60.79
62.81
41.98
Qwenl.5-72B
w/ Ref
12.72
11.35
7.13
27.87
10,91
15.63
4.39
17.78
9.77
Baichuan2-7B
w/o Ref
63.19
77.99
39.49
70.56
61.66
79.26
71.72
72.7
52.71
w/ Ref
$2.38
18.02
64.27
60.78
37.38
29.17
27.71
43.61
23.55
w/o Ref
57.66
32.95
66.52
63.04
79.17
65.14
70.45
47.62
Baichuan2-13B
w/ Ref
46.47
12.2
52.02
63.41
33.15
40.99
16.58
44.78
42.96
w/o Ref
70.86
92.31
43
89.63
67.09
87.6
71.25
87.11
47.99
Mistral-7B
w/ Ref
32.22
10.77
23.45
48.37
17.74
43.8
27.8
27
30.29
Mistral-8x7B
w/o Ref
55.72
82.39
26.16
77.09
54,9
90.51
60.17
80.96
42.86
w/ Ref
8.17
9.45
3.7
14.92
7.06
14.57
7.69
7.67
6.41
w/o Ref
.0
62.26
32.38
63.85
62.81
77.99
54.9
68.51
50.56
Deepseek-7B
w/ Ref
23.1
11.54
43.94
22.97
24.25
17.8
13.51
w/o Ref
33.68
52.86
17.89
57.79
37.41
72.91
33.93
64.62
33.33
Deepseek-67B
w/ Ref
13.4
9.91
10
12.7
11.84
10.53
21.03
w/o Ref
67.81
90.22
42.44
88,27
70.81
94.44
76.28
91.84
56.95
Llama2-78
w/ Ref
50.65
66.67
13.83
73.04
11.76
54.3
13.7
60.5
21.43
w/o Ref
62.69 46.59
84.73
36.43
85.38
67.01
87.09
69.6
90.07
51.3
Llama2-130 13B
w/ Ref
62.86
14.81
50.54
4.37
13.1
44.66
73.75
28.65
Table A2: Hallucination rate of open-source models according 1-Aaev2-Stagel.
21

---

Person
Event
Thing J.
Location 1
Model
Setting
Overall
EN
ZH
ZH
EN
ZH
w/o Ref
75.83
95.28
42.49
84.58
70.88
94.79
77.13
86.2
57.73
IntemLLM-77B
w/ Ref
12.14
18.18
4.74
28.59
9.86
15.69
2.59
17.47
5.22
w/o Ref
65.8
92.82
39.75
76.74
75.19
90.95
80.29
79.71
57.24
Internl M2--20B
w/ Ref
12.13
28.57
5.82
25.2
13.15
12.64
7.2
15.4
13.46
w/o Ref
65.38
44.07
68.66
71.35
73.85
80.68
56.72
Qwen 1.5-7B
w/ Ref
6.28
6.03
3.81
10.39
11.38
9.11
3.69
7.8
w/o Ref
$7.42
71.83
14
56.28
71.52
81.31
70.76
69.81
54.07
Owenl.5-144
w/ Ref
4.92
1.06
9.83
2.51
12.47
2.7
7.01
6.44
w/o Ref
50.17
66.17
25.92
52.74
60.24
79.45
62.92
63.9
46.34
Qwenl.5-72B
w/ Ref
11.46
10.64
6.09
25.09
11.42
15.81
3.8
15.87
8.94
Baichuan2-7B
w/o Ref
63.47
78.62
39.22
67.77
64.13
80.74
73.01
73.65
54.89
w/ Ref
53.25
17.12
65.63
60.61
39.08
30.87
29.17
43.25
25.09
w/o Ref
58.4
73.05
33.42
64.06
67.5
80.85
66.2
70.45
Baichuan2-13B
w/ Ref
47.48
12.2
54.36
61.66
35.91
42.46
17.81
44.78
42.26
w/o Ref
71.47
93.01
43.53
89.82
68. 15
88.06
71.97
87.75
49.6
Mistral-7B
w/ Ref
32.04
11.28
23.51
46.9
18.28
44.49
27.15
26.91
32.15
w/o Ref
56.91
84.09
27.4
76.25
57.26
91.35
61.61
82.25
45.11
Mistral-8x7B
w/ Ref
7.86
8.66
2.69
13.39
7.66
14.3
7.94
8.12
8.05
w/o Ref
$1.09
33.43
61.41
65.51
79.43
56.15
69.55
53.6
Deepseek-7B
w/ Ref
23.35
45.96
22.97
25.79
25.5
17.8
13.51
w/o Ref
33.4
60
17
53.44
38.15
73.47
34.23
63.82
34.7
Deepseek-67B
w/ Ref
11.53
9.01
9.52
14.47
11.74
19.44
8.33
w/o Ref
69.26
91.3
44.15
89.3
74.65
94.99
77.89
93
58.01
Llama2-78
w/ Ref
55.47
78.33
11.7
74.02
10.34
61.43
11.64
67.23
19.05
w/o Ref
63.59
83,97
37.17
85,48
68.69
87.4
70.52
91
53.5
Llama2- 13B
w/ Ref
50.7
75.43
12.35
4.18
13.1
50
47.4
78.75
27.53
Table A3: Hallucination rate of open-source models according 2NAH-v2-Stage..
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we have comprchensively summarized and presented our methods and contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our proposed method in the appendix F.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided comprehensive experimental details in the appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification: Dataset, code, and model are released at https://github.ccmm com/ open- compass/ANAth
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
Staia///uuideeSooeuuueeeee sonioiiissn for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https:
CoodeSummiis ssi asionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided comprehensive experimental details the appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We have validated the effectiveness of our method through large number of experiments in 4 Experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided comprehensive experimental details the appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https /ttp//eeriip.../ cwtppplicc/thicssceeecciccccecee www.eetteiiiiiiieeeeeeeeeeeeieee
Answer: [Yes]
Justification:
Our research complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed the broader impacts of our proposed method in the ap- pendix G.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our method does not involve these risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have provided detailed citations for the datasets and other models used in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes].
Justification: Dataset, code, and model are released at https://github. com/ open compass/ANAH.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "s4J32VeZbx", "reviewer": "Reviewer_nXFU", "summary": "This paper proposes an iterative self-training framework that simultaneously and progressively scales up the hallucination annotation dataset and improves the accuracy of the hallucination annotator. The framework is based on the expectation maximization algorithm, alternately annotating a scaled dataset and training a more accurate hallucination annotator on the dataset. A 7 billion model trained by this framework can surpass GPT-4 and obtains state-of-the-art hallucination detection results on HaluEval and HalluQA by zero-shot inference.", "questions": "Is it possible to leverage large language model based evaluation for hallucination detection?", "limitations": "Appendix E describes the limitations.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is well motivated by the fact that large language model hallucination significantly hinders applications but its annotation is difficult and very labor intensive.\n- The solution based on self-training effectively and feasibly addresses the above challenge. (1) The paper defines a procedure of analytical hallucination annotation, that aligns with human cognitive processes. (2) Staged multi-dimensional data scaling, collecting synthetic data from more large language models and for more numbers of topics and questions, ensures the richness of the dataset. (3) Leveraging EM algorithm is suitable.\n- Strong empirical results are shown for both in-domain hallucination detection and on existing benchmarks, HaluEval and HalluQA.\n- Very well-written paper.", "weaknesses": "The RougeL and BertScore may not be the most capable metrics for evaluating generated texts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and for recognizing our efforts! Here are the answers to your question (which are highlighted in italics) regarding the metrics used to evaluate generated texts.\n\n### **Response to Weakness and Question:**\n\n> *The RougeL and BertScore may not be the most capable metrics for evaluating generated texts. Is it possible to leverage large language model based evaluation for hallucination detection?*\n\nWe use RougeL and BertScore as metrics because they are classical and popular metrics in NLG [1-3].\n\nIn line with your suggestion, we additionally added an LLM-based evaluation metric. Specifically, we use FactScore [4] to assess the consistency of generated reference points with the source document.\n\nBelow is Table 2 from our paper, which contains the newly added metric FactScore (column 3). The results of the FactScore indicates that the reliability of our model's generated reference points progressively improves and ultimately exceeds that of GPT4. This trend is consistent with F1 and ACC, reflecting the reliability of the FactScore.\n\n| Model          | F1    | ACC   | **FactScore** | RougeL | BertScore |\n|----------------|-------|-------|---------------|--------|-----------|\n| GPT-4          | 87.11 | 86.97 | **84.39**     | 86.32  | 96.21     |\n| ANAH-7B        | 78.69 | 79.92 | **80.60**     | 58.51  | 87.27     |\n| ANAH-20B       | 80.49 | 81.01 | **81.51**     | 58.82  | 88.44     |\n| ANAH-V2-Stage1 | 84.45 | 84.85 | **83.63**     | 60.10  | 88.43     |\n| ANAH-V2-Stage2 | 87.75 | 88.18 | **84.54**     | 67.28  | 90.80     |\n| ANAH-V2-Stage3 | 89.30 | 89.55 | **86.36**     | 69.44  | 91.43     |\n\nWe will add a corresponding discussion and evaluation results in the next version.\n\n[1] Sai, Ananya B., Akash Kumar Mohankumar, and Mitesh M. Khapra. \"A survey of evaluation metrics used for NLG systems.\" ACM Computing Surveys (CSUR) 55.2 (2022): 1-39.\n\n[2] Li, Junyi, et al. \"HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models.\" EMNLP 2023.\n\n[3] Pagnoni, Artidoro, Vidhisha Balachandran, and Yulia Tsvetkov. \"Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics.\" ACL 2021.\n\n[4] Min S, Krishna K, Lyu X, et al. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation[J]. arXiv preprint arXiv:2305.14251, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VjsoQBpCr4", "reviewer": "Reviewer_4rBJ", "summary": "The authors propose an innovative approach to tackle the persistent issue of hallucinations in large language models (LLMs) during long-form question-answering tasks. Current methods for detecting and mitigating these hallucinations are constrained by limited data and high labor costs. To address this, the paper introduces an iterative self-training framework that scales up the hallucination annotation dataset while simultaneously enhancing the accuracy of the annotators. Using the Expectation Maximization algorithm, the process involves multiple iterations where a pipeline annotates a growing dataset, and the improved annotator is used for the next cycle. This approach not only leads to a highly accurate annotator that surpasses GPT-4 but also effectively reduces hallucinations in LLM outputs. The results show significant improvements in key benchmarks, offering a scalable and efficient solution for managing LLM hallucinations.", "questions": "See weaknesses.", "limitations": "No", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. One of the major strengths of this paper is its scalable framework. By automating the annotation process and iterating through larger datasets, the authors manage to overcome the usual limitations of manual data labeling, making the approach both cost-effective and efficient.\n\n2. The application of the Expectation Maximization algorithm to improve annotator accuracy through iterative training is a standout feature. This method not only refines the annotations with each cycle but also ensures that the annotator becomes progressively more reliable, which is a clever and effective use of existing statistical techniques.", "weaknesses": "1. One of the big drawbacks of the EM algorithm is its sensitivity to initial conditions. If you don't start with the right parameters, the algorithm can easily get stuck in a local maximum instead of finding the best possible solution, which can be frustrating.\n\n2.  The EM algorithm can be quite demanding in terms of computational resources. Each iteration involves a lot of number crunching, which means it can be slow and resource-intensive, especially when dealing with large datasets or complex models.\n\n3. The authors could enhance their literature review by including several highly relevant papers on data annotation. The annotation task has been discussed in [1] [2] [3] [4] [5].\n\n[1] https://arxiv.org/abs/2310.04668\n[2] https://arxiv.org/abs/2303.15056\n[3] https://arxiv.org/abs/2306.04349\n[4] https://dl.acm.org/doi/pdf/10.1145/3613904.3642834\n[5] https://dl.acm.org/doi/pdf/10.1145/3594536.3595161", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comment. Following are our responses to each individual comment (which are highlighted in italics).\n\n### **Response to Weakness1 about initial condition:**\n\n> *One of the big drawbacks of the EM algorithm is its sensitivity to initial conditions. If you don't start with the right parameters, the algorithm can easily get stuck in a local maximum instead of finding the best possible solution, which can be frustrating.*\n\nWe acknowledge the sensitivity of the EM algorithm to initial conditions. \n\nTo address this problem, the first step of our framework was to train an annotator model with a high-quality, human-labeled hallucination dataset, ANAH [1]. We then obtained a model with annotation accuracy close to GPT4, and we used this high-accuracy model as a starting point for subsequent EM operations.\n\nMeanwhile, to ensure the stability of the convergence process, we use a progressive scaling strategy. Tables 2 and 4 show the effectiveness of this approach. In Table 2, the performance of the labeler improves continuously with data scaling, and in Table 4, the progressive approach is much more effective than the non-progressive approach.\n\nAlthough we cannot claim that we will eventually converge to a globally optimal solution, a flat convergence region would be nice according to the theoretical analysis [2, 3]. One measure of flatness is generalisability [4, 5], and Table 6 shows that our model has excellent generalisability. Thus, we believe that our method achieves a promising result.\n\nWe will add a corresponding discussion in the next version.\n\n### **Response to Weakness2 about computational resources:**\n\n> *The EM algorithm can be quite demanding in terms of computational resources. Each iteration involves a lot of number crunching, which means it can be slow and resource-intensive, especially when dealing with large datasets or complex models.*\n\nWe agree that the iterative algorithm requires computational effort. However, it is important to consider our context where building a fine-grained hallucination annotation dataset requires prohibitively high costs and labor intensity. \n\nUsing the \"manual + GPT4-assisted\" annotation model, as described in ANAH [1] (0.9 USD and 20 minutes per annotation), it would take 177,237 USD and 65,643 hours to reach the size of the dataset in our work.\n\nHowever, our method uses 32 A100 GPUs to iteratively train the 7B model. It took approximately 100 hours for inference and training. Based on the price of the computing platform Lambda (1.29 USD per GPU per hour), it only costs 4,128 USD.\n\nSo we believe this is a better trade-off between computing resources and labour+API costs, which is acceptable.\n\nMoreover, the large-scale hallucination dataset and high-precision hallucination annotation model that we finally obtained can serve as the foundation for more research in the future, which we think is very meaningful.\n\nWe will add a corresponding discussion in the next version.\n\n### **Response to Weakness3 about relevant papers:**\n\n> *The authors could enhance their literature review by including several highly relevant papers on data annotation. The annotation task has been discussed in [6-10].*\n\nThanks for your suggestion! We discuss the papers you mentioned below. \n\n[6] proposed a pipeline for annotating nodes on a graph without labels. [7, 8, 9] discussed the superiority of using GPT4 or ChatGPT as annotators. [10] introduces a self-supervised method using GPT for data annotation.\n\nDifferent from them, we introduce an iterative self-training framework that simultaneously and progressively scales up the hallucination annotation dataset and improves the accuracy of the hallucination annotator.  \n\nWe will add these relevant papers to our related work.\n\n\n[1] Ji Z, Gu Y, et al. ANAH: Analytical Annotation of Hallucinations in Large Language Models[J]. ACL, 2024.\n\n[2] Hochreiter S, Schmidhuber J. Flat minima[J]. Neural computation, 1997, 9(1): 1-42.\n\n[3] Hochreiter S, Schmidhuber J. Simplifying neural nets by discovering flat minima[J]. Advances in neural information processing systems, 1994, 7.\n\n[4] Keskar N S, Mudigere D, Nocedal J, et al. On large-batch training for deep learning: Generalization gap and sharp minima[J]. arXiv preprint arXiv:1609.04836, 2016.\n\n[5] Dziugaite G K, Roy D M. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data[J]. arXiv preprint arXiv:1703.11008, 2017.\n\n[6] Chen Z, Mao H, Wen H, et al. Label-free node classification on graphs with large language models (llms)[J]. arXiv preprint arXiv:2310.04668, 2023.\n\n[7] Gilardi F, Alizadeh M, Kubli M. ChatGPT outperforms crowd workers for text-annotation tasks[J]. Proceedings of the National Academy of Sciences, 2023, 120(30): e2305016120.\n\n[8] He Z, Huang C Y, Ding C K C, et al. If in a Crowdsourced Data Annotation Pipeline, a GPT-4[C]//Proceedings of the CHI Conference on Human Factors in Computing Systems. 2024: 1-25.\n\n[9] Savelka J. Unlocking practical applications in legal domain: Evaluation of gpt for zero-shot semantic annotation of legal texts[C]//Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law. 2023: 447-451.\n\n[10] Pei X, Li Y, Xu C. Gpt self-supervision for a better data annotator[J]. arXiv preprint arXiv:2306.04349, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cqJlFS9tq2", "reviewer": "Reviewer_5wgD", "summary": "This paper introduces an iterative self-training framework to address hallucinations in large language models (LLMs), enhancing the accuracy of annotators and scaling up hallucination detection datasets. The framework utilizes the Expectation Maximization algorithm to progressively improve the hallucination annotator's performance by annotating a scaled dataset and training a more accurate annotator in each iteration. Experimental results demonstrate that the enhanced annotator surpasses GPT-4 in hallucination detection, achieving state-of-the-art results on HaluEval and HalluQA.", "questions": "1. The difficulty in automatically constructing high-quality hallucination datasets is not clearly explained. The relevant works in this area, the unresolved issues, and why these difficulties persist are not thoroughly discussed by the authors in the paper.\n2. The rationale behind using the EM algorithm to solve this problem is not clearly articulated. What considerations led to this choice?\n3. How does the paper ensure that the EM algorithm converges through iterations?", "limitations": "none", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a unique perspective on the issue, as many current works focus on hallucination detection and mitigation. The focus on automatically constructing high-quality hallucination datasets is crucial for addressing hallucinations in large language models. This is an important and valuable contribution to the field.\n2. The structure of the paper is clear, and the writing is concise. The experimental section is detailed and thorough, enhancing the credibility of the results.", "weaknesses": "1. The rationale behind using the EM algorithm to solve this problem is not clearly articulated. What considerations led to this choice?\n2. How does the paper ensure that the EM algorithm converges through iterations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful comments. Following are our responses to each individual comment (which are highlighted in italics).\n\n### **Response to Weakness1 about contributions:**\n\n> *The description of the methodological contributions in the paper is not very clear. The EM algorithm and the self-consistency method mentioned in the article are existing works.*\n\nOur primary contribution is an iterative self-training framework that simultaneously and progressively scales up the hallucination annotation dataset and improves the accuracy of the hallucination annotator.  \n\nWithin this framework, the EM algorithm serves as our theoretical foundation. Additionally, we identified the need for a pipeline during the E-Step to produce more stable output. So we selected self-consistency as a method to achieve this stability. \n\nTo the best of our knowledge, we are the first to present work on the automated construction of large-scale hallucination datasets.\nMoreover, the large-scale hallucination dataset and high-precision hallucination annotation model that we finally obtained can serve as the foundation for more research in the future, which we believe is very meaningful.\n\n### **Response to Weakness2 and Question1 about the difficulty of automatic data construction:**\n\n> *The difficulty in automatically constructing high-quality hallucination datasets is not clearly explained. The relevant works in this area, the unresolved issues, and why these difficulties persist are not thoroughly discussed by the authors in the paper.*\n\nThe difficulty of automatic hallucination data construction is the low performance of automatic annotators. The relevant works in this area, the unresolved issues, and why these difficulties persist were discussed in the Introduction (lines 32-35) and Related Work (lines 94-100).\n\nTo solve these difficulties, we first proposed a progressively self-iterative labeling method that automatically scales up the dataset of fine-grained hallucination annotations, and we also proved the effectiveness of our method.\n\n### **Response to Weakness3 and Question2 about the reason for using EM:**\n\n> *The rationale behind using the EM algorithm to solve this problem is not clearly articulated. What considerations led to this choice?*\n\nOur task can be formalized as optimizing two hidden variables, the hallucination annotator parameters and the data labels, as described in Section 3.2 (lines 138-140). We believe that the form of our task corresponds exactly to the EM algorithm [1].\n\n### **Response to Weakness4 and Question3 about EM convergence:**\n\n> *How does the paper ensure that the EM algorithm converges through iterations?*\n\nEM is a convergent algorithm, as demonstrated in [2]. And, as illustrated by our experimental results, our approach has shown progressive improvement in annotation performance (Table 2) and generalization capabilities (Table 6) through iterations. \n\n[1] Dempster A P, Laird N M, Rubin D B. Maximum likelihood from incomplete data via the EM algorithm[J]. Journal of the royal statistical society: series B (methodological), 1977, 39(1): 1-22.\n\n[2] Wu C F J. On the convergence properties of the EM algorithm[J]. The Annals of statistics, 1983: 95-103.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jtdMKblo4E", "reviewer": "Reviewer_i3ot", "summary": "The paper proposes to augment hallucination annotation dataset and improve the performance of the hallucination annotator simultaneously in an iterative self-training framework. During each iteration, they use a Expectation-Maximization Algorithm for data annotation and hallucination annotator training. The trained hallucination annotator can be further used for downstream tasks such hallucination evaluation and hallucination mitigation. Extensive experiments were performed to show the superiority in hallucination detection and the effectiveness of the proposed approach.", "questions": "- Typo in ine 76-77, \"we reduce the hallucination of the final LLM generations from 25% to 37%.\"\n- Line 156, when you use the majority vote to select the the most common hallucination type, wouldn't it be dominanted by \"No Hallucination\"? As you are also showing in Table 7, the hallucination rate for InternLM2 is less than 20%. If you generate n candidates (depending on how big your n is) and use a majority votes, not many hallucination types will be selected.\n- How many iterations did you do to get the dataset in Table 1?\n- In Table 2, any discussions on GPT4 having much better RougeL and BertScore?", "limitations": "The limitations are discussed in Appendix which actually addresses some of the questions I was gonna ask, such as more benchmarks or more backbone models for the hallucination annotator.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is very easy and pleasant to read. Each section and their subsections are well connected and explained without adding much redundant content. Figure 2 is especially illustrative.\n- I like how they explain a rather practical work in a theoretical manner using all the math equations in Section 3. It makes the work more sound with less confusions.\n- Although the model name is ANAH-v2, it actually brings big novelty compared to the original ANAH model. The model is in a self-training manner and the training prompt is expanded to do three tasks.\n- Large amount of experiments show impressive performance on hallucination detection. The thorough ablation study also shows the importance of each component.", "weaknesses": "- My biggest concern for this paper is that the model essentially does three things: Factual Existence Judgment, Reference Information Extraction, and Hallucination-Type Judgment. Each step relies on the results from previous step. For example, if the extracted reference information is limited, it will greatly impact the hallucination judgement as well. The experiments mainly focus on the third step without discussions on the previous two steps. So I’m not certain how stable the model actually is.\n- The hallucination mitigation is a little weak, it basically generates multiple responses and select the response with the least portion of hallucination sentences. There are multiple other ways to mitigate hallucinations that are not compared in Table 8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comment. Following are our responses to each individual comment (which are highlighted in italics).\n\n### **Response to Weakness1 about stability of the annotation steps:**\n\n> *My biggest concern for this paper is that the model essentially does three things: Factual Existence Judgment, Reference Information Extraction, and Hallucination-Type Judgment. Each step relies on the results from previous step. For example, if the extracted reference information is limited, it will greatly impact the hallucination judgement as well.*\n\nWe acknowledge that there exists the possibility that such previous steps may have influenced the final judgement. Therefore, in addition to majority voting on the final step of data construction, we also perform consistency checks (lines 157-161) on the previous steps to ensure quality.\n\nMeanwhile, we utilize RougeL, Bertscore to evaluate the performance of Factual Existence Judgment (Step1) and Reference Information Extraction (Step2), and we find that these two metrics in Table 2 increase steadily as the iteration progresses. These evaluations can show the stability and effectiveness of previous steps. \n\nIn addition, Hallucination-Type Judgment (Step 3) relies on the results from the previous steps. Therefore, the results for Step 3 (F1 and ACC in Table 2) can also reflect the robustness and accuracy of the previous steps.\n\n### **Response to Weakness2 about mitigation:**\n\n> *The hallucination mitigation is a little weak, it basically generates multiple responses and select the response with the least portion of hallucination sentences. There are multiple other ways to mitigate hallucinations that are not compared in Table 8.*\n\nOur primary focus is on automatically scaling up the annotation dataset and building annotators. \nThe mitigation section aims to show the potential of our annotator for mitigation, rather than to achieve SOTA results. \n\nTherefore, we only used a simple re-ranking strategy to mitigate the hallucination of LLM, and the promising results in Table 8 proved that our annotator could be used for hallucination mitigation.\n\nIn the future, we will explore methods to apply our annotators in mitigation, such as RLAIF.\n\n### **Response to Question1 about typo error:**\n\n> *Typo in ine 76-77, \"we reduce the hallucination of the final LLM generations from 25% to 37%.\"*\n\nWe use \"reduce\" because the metric NLI described here is inversely related to the level of hallucinations. For clarification, we will change the presentation to \"we reduce hallucination, with the NLI metric increasing from 25% to 37% on HaluEval\" in the next version. \n\n### **Response to Question2 about balance:**\n\n> *Line 156, when you use the majority vote to select the the most common hallucination type, wouldn't it be dominanted by \"No Hallucination\"? As you are also showing in Table 7, the hallucination rate for InternLM2 is less than 20%. If you generate n candidates and use a majority votes, not many hallucination types will be selected.*\n\nTo clarify, the hallucination rate of InternLM2 less than 20% in Table 7 is achieved at settings that provide reference when generating response. \n\nIn the setting of \"w/o reference\", the hallucination rate of InternLM2 is ~80% and the responses both with and without reference are mixed together to construct the dataset. Therefore, although the majority voting may enhance such bias in the single setting, the overall dataset is balenced. For example, in the training data at Stage 2, the ratio of hallucinations to non-hallucinations is **52.53 : 47.47**.\n\nIn addition, the results in Table 3 show our majority-vote method improves accuracy.\n\nWe will add a corresponding discussion in the next version.\n\n### **Response to Question3 about iteration:**\n\n> *How many iterations did you do to get the dataset in Table 1?*\n\n3 iterations\n\n### **Response to Question4 about GPT4 performance:**\n\n> *In Table 2, any discussions on GPT4 having much better RougeL and BertScore?*\n\nAs we mentioned in footnote 2, GPT-4 pre-annotation in ANAH makes the RougeL and BERTScore higher than the zero-shot setting in this work. In order to analyze the causes of this phenomenon more clearly.\n\nDuring the construction of ANAH [1],  GPT-4 is used for the initial pre-annotation. Subsequently, humans refine these pre-annotations, and humans tend to not change the pre-annotations. This methodology inherently aligns the final 'golden' answers closely with the outputs by GPT-4. \n\nIn addition, we added an LLM-based Evaluation to exclude the similarity due to pre-annotations. Specifically, we use FactScore [4] to assess the consistency of generated reference points with the source document.\n\nBelow is Table 2 from our paper, which contains the newly added metric FactScore (column 3). The results of the FactScore indicates that the reliability of our model's generated reference points progressively improves and ultimately exceeds that of GPT4. This trend is consistent with F1 and ACC, reflecting the reliability of the FactScore.\n\n| Model          | F1    | ACC   | **FactScore** | RougeL | BertScore |\n|----------------|-------|-------|---------------|--------|-----------|\n| GPT-4          | 87.11 | 86.97 | **84.39**     | 86.32  | 96.21     |\n| ANAH-7B        | 78.69 | 79.92 | **80.60**     | 58.51  | 87.27     |\n| ANAH-20B       | 80.49 | 81.01 | **81.51**     | 58.82  | 88.44     |\n| ANAH-V2-Stage1 | 84.45 | 84.85 | **83.63**     | 60.10  | 88.43     |\n| ANAH-V2-Stage2 | 87.75 | 88.18 | **84.54**     | 67.28  | 90.80     |\n| ANAH-V2-Stage3 | 89.30 | 89.55 | **86.36**     | 69.44  | 91.43     |\n\nWe will add a corresponding discussion and evaluation results in the next version.\n\n[1] Ji Z, Gu Y, et al. ANAH: Analytical Annotation of Hallucinations in Large Language Models[J]. ACL, 2024.\n\n[2] Min S, Krishna K, Lyu X, et al. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation[J]. arXiv preprint arXiv:2305.14251, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the valuable feedback from the reviewers!

We are honored that our work can be reviewed as: 

- The paper is "well motivated" and addresses the critical issue (R-5wgD).  
- It provides a novel, effective and feasible solution (R-nXFU & 4rBJ & i3ot), and "obtains strong and thorough empirical results" (R-nXFU & 5wgD & i3ot). 
- The paper  is "well written" (R-nXFU & 5wgD & i3ot) and well explained by figures and equations (R-i3ot). 

For each question from all reviewers, we have provided a specific response in the relevant section below.

Any additional clarification and discussion suggested by the reviewers will be included in the revised version.

Meta-review: This paper proposes an iterative self-training method to scale up hallucination annotation with better accuracy and uses the model to mitigate hallucination of LLMs. The proposed framework is scalable and efficient and shows strong performance on hallucination detection benchmarks. All the reviewers consistently acknowledge the contributions of this paper. Thus, I recommend acceptance.
Decision: Accept (poster)