ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models

Authors: yuzhe  gu
ziwei  ji
wenwei  zhang
chengqi  lyu
dahua  lin
kai  chen
stronger  annotator
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 14:22:03

Abstract: Large language models (LLMs) exhibit hallucinations in long-form question-answering tasks across various domains and wide applications. Current hallucination detection and mitigation datasets are limited in domain and size, which struggle to scale due to prohibitive labor costs and insufficient reliability of existing hallucination annotators. To facilitate the scalable oversight of LLM hallucinations, this paper introduces an iterative self-training framework that simultaneously and progressively scales up the annotation dataset and improves the accuracy of the annotator. Based on the Expectation Maximization algorithm, in each iteration, the framework first applies an automatic hallucination annotation pipeline for a scaled dataset and then trains a more accurate annotator on the dataset.  This new annotator is adopted in the annotation pipeline for the next iteration. Extensive experimental results demonstrate that the finally obtained hallucination annotator with only 7B parameters surpasses GPT-4 and obtains new state-of-the-art hallucination detection results on HaluEval and HalluQA by zero-shot inference. Such an annotator can not only evaluate the hallucination levels of various LLMs on the large-scale dataset but also help to mitigate the hallucination of LLMs generations, with the Natural Language Inference metric increasing from 25% to 37% on HaluEval.

Main paper: 
 introduction: 
 Large Language Models (LLMs) have shown remarkable capabilities in various tasks [10, 11, 35, 51, 57] . However, they tend to produce hallucination, i.e., plausible-sounding but unfaithful or nonsensical information [5, 30] , that significantly hinders their real-world applications. Initial steps to address this issue involve the creation of datasets that can help to detect, annotate, and mitigate hallucinations [14, 29, 40] . Since the potential hallucinations of LLMs are in various fields, the spectrum of knowledge in the dataset is expected to be large-scale and comprehensive, covering various domains. Consequently, the size and diversity of datasets are critical for the oversight of LLM hallucinations.However, constructing and scaling-up hallucination annotation datasets face significant hurdles [8, 9, 29, 43] . One primary challenge is the prohibitively high costs and labor intensity required for their accurate assessment [43, 47] , since the fine-grained hallucination annotation requires intensives labor for reading long documents and annotating the hallucination details sentence by sentence. Moreover, due to the insufficiency of accurate human annotations, the reliability of existing hallucination annotators and detectors becomes another pressing concern [29] . These tools have been found to produce inaccurate results [9, 46, 61] , e.g., even GPT4 [1] , one of the most powerful LLMs, is not satisfactory and cannot achieve a compatible performance of humans [29] .Existing works [3, 25, 36, 38, 56, 58, 69] have explored strategies in data augmentation and selftraining to extend dataset size and boost the performance of models in the fields of image segmentation, multi-lingual translation, math reasoning, etc. However, how to scale the hallucination annotation datasets efficiently is under-explored in the community, which significantly hinders the in-depth analysis and further mitigation of LLMs hallucinations at a large scale. To address the research gap, this paper proposes an iterative self-training framework designed to scale up the hallucination annotation dataset and simultaneously increase the accuracy of annotators (Fig. 1 ). The iterative framework can be explained from the perspective of the Expectation Maximization (EM) algorithm. In the Expectation (E) step, we apply the existing best hallucination annotator to estimate the ground-truth hallucination annotations of the scaled dataset. We adopt an inference pipeline on top of the annotator with self-consistency strategy [63] to provide a more robust estimation of the annotations, which lays the groundwork for training a more precise annotator in the subsequent step. In the Maximization (M) step, we combine the existing annotations with the scaled data annotations derived from the previous E steps to train a new hallucination annotator. Training on more data leads to a more accurate annotator and a more robust annotation pipeline, setting the stage for the subsequent round of annotations.The iterative process consists of three stages of multi-dimensional data scaling as shown in Fig. 1 . Initially, we train a weak annotator on human annotations. In the second stage, we collect the hallucination responses from more open-source LLMs for the same questions in the dataset to improve the generalization ability of hallucination annotators to model responses. In the third stage, we expand the number of topics and questions in the dataset and collect hallucination annotations with the more robust annotator. This progressive scaling strategy stabilizes the annotator's performance when evaluating familiar and unfamiliar responses across diverse topics.Extensive experimental results show that our enhanced annotator significantly outperforms existing models, including the advanced GPT-4, in terms of accuracy. Our annotator not only performs best on the in-domain fine-grained hallucination annotation dataset ANAH (89.24%) but also obtains new state-of-the-art (SOTA) results on HaluEval (81.54%) and HalluQA (94.44%) under zero-shot setting. In addition, the annotator automates the hallucination evaluation on the dataset, offering a comprehensive benchmark for the research community to evaluate the hallucination levels of numerous open-source models, providing a practical reference for future hallucination mitigation of LLMs. Using a simple reranking strategy with the annotator, we reduce the hallucination of the final LLM generations on HaluEval, with the NLI metric increasing from 25% to 37%. 
related work: 
 Self-improvement of Large Models. As Large Language Models (LLMs) become more and more powerful, the community starts to explore different strategies to achieve the self-improvement of LLMs, i.e., to improve the LLMs using the supervision from LLMs [12, 24, 28, 50, 53] . For example, existing works have explored self-alignment using LLMs with ethical principles [3, 58, 69] . There are also methods [13, 25, 38, 56, 70] strengthen LLM's capabilities on tasks such as reasoning by training the LLMs on the high-quality responses from themselves on the same questions. In the field of computer vision, SAM [36] introduces manual and model-assisted labeling to expand the image segmentation dataset and enhance the performance of image segmentation models. However, the application of self-improvement is under-explored in fine-grained hallucination annotation. This field In the E-step, given unlabeled new data from the Data Growth Flywheel, the annotator predicts N candidate outputs y. Then the representative annotation y * is chosen via self-consistency. As a result, we construct a larger dataset by collecting the new annotations. In the M-step, we train an annotator on the larger dataset aligned to our training format. This annotation process consists of three phases: Factual Existence Judgment, Reference Information Extraction, and Hallucination Type Judgment. As a result, we gain a stronger annotator with higher accuracy. is challenging for automatic annotators due to its meticulous nature, which requires fine validation with long documents. It is also noteworthy that most self-improvement works require extra resources such as human labor or a supplementary model [36, 38] . In contrast, our pipeline is self-sufficient, relying solely on the annotator model and the initial dataset.Hallucination Annotation Dataset. The development of the hallucination annotation dataset is the cornerstone for detecting hallucinations in models' output. These datasets can be used to train a hallucination detector/annotator and evaluate the hallucination level via the detector/annotator. Early works [21, 22, 26, 37, 40, 44, 48, 60, 62, 67] in this domain tended to broadly classify entire responses as either hallucinatory or not, providing a coarse-grained analysis of hallucination occurrences. Recent works [29, 47] annotate hallucinations in a more fine-grained and meticulous way. Despite this progress, these datasets, especially those having fine-grained annotations, suffer from limitations in size and scalability due to the high costs associated with the usage of human annotators or commercial models like GPT4. In addition, the difficulty of this task and the limited human annotations result in unsatisfactory performance of the automatic hallucination annotator.Hallucination Mitigation. Considering the harm of hallucinations, researchers have explored various techniques for mitigating hallucinations. Techniques such as multi-task learning [23, 65] , model editing [18, 31] , and fine-grained RLHF [66] are proposed to suppress hallucination tendencies during training. Alternative strategies have been proposed that do not require further model training, including different decoding strategies [15, 41, 52, 55] , multi-agent methods [20] , and variants of the Chain-of-Thought approach involving verification or reflection [19, 32, 39, 64] . Our ANAH-v2 shows efficiency in hallucination mitigation as a re-ranker and has the potential to combine with the existing methods such as fine-grained RLHF. 
method: 
 This paper proposes an iterative self-training framework to simultaneously scale up the hallucination dataset and improve the accuracy of the hallucination annotator. We follow the analytical hallucination annotation ( § 3.1) to annotate the hallucination sentence-by-sentence. The multi-iteration framework is theoretically grounded in the EM algorithm ( § 3.2) and involves three stages to progressively scale the dataset in multiple dimensions ( § 3.3). We also reveal how the hallucination annotators can be applied for hallucination evaluation and mitigation ( § 3.4).The aim of a hallucination annotator is to identify hallucinations in the model responses. ANAH [29] developed a fine-grained annotation method that locates reference points in the document for each sentence and makes hallucination-type judgments, with the whole process completed in one turn of dialog. However, this hybrid task diverges from the human judgment processes and fails to clearly indicate the relationship between reference points and hallucination judgments, resulting in unsatisfactory annotation accuracy.Instead of using the original ANAH training prompts, we developed a more reliable training method tailored to the hallucination annotation process. As depicted in the lower right part of Fig. 2 , the process is outlined in three phases: (1) Factual Existence Judgment, where the annotator assesses whether the provided sentence contains verifiable facts. If no factual content is present, the sentence is categorized as 'No Fact' and requires no further annotation. (2) Reference Information Extraction, where the annotator extracts relevant reference points from the documents related to the question and answer. (3) Hallucination-Type Judgment, where the annotator determines the type of hallucination based on the extracted reference points. If the sentence aligns with the references, it is classified as 'No Hallucination'. If it contradicts the references, it is deemed a 'Contradictory Hallucination'. If it lacks supporting evidence and cannot be verified, it is labeled as 'Unverifiable Hallucination'. The above three phases will form a multi-turn dialogue in training data. Compared to the ANAH approach, which involves simultaneous judgments on multiple criteria, our phased process aligns more closely with human cognitive judgment processes. The detailed data format and prompts for our annotation process are in Appendix A.Simultaneously scaling up the dataset and improving the accuracy of the annotator can be formulated by the EM algorithm. For the input set X, we need to estimate two hidden variables simultaneously, the output set Y and the model parameters θ. Specifically, based on the task formulation in § 3.1, we define the input x from the input set X of the hallucination annotator consists of a question, a sentence to be annotated, and a reference document. The expected output y to be estimated in the data output set Y includes the factual information f , the key reference points r from the reference document, and the type of hallucination h. We maximize the log-likelihood estimation of Y by alternately performing the E-Step and the M-Step to update the model parameters θ:EQUATIONStep. A straightforward approach to estimating Y is to use a single model to predict annotations. However, this method lacks sufficient accuracy [45] . To improve the accuracy and stability of the estimation of Y , we introduce the self-consistency method [63] , which provides a more robust representation of the distribution of the Y . As shown in Fig. 2 . For each input x, we perform multiple samplings to yield K independent outputs y = {y 1 , • • • , y i , • • • , y K }, where the i-th output sample y i is composed of factual information (f i ), reference point (r i ) and hallucination type (h i ). We use a self-consistency metric to select the most representative sample y * among all outputs:EQUATIONDuring this selection process, we consider the hallucination type h, reference point r, and factual information f in turn. We determine the most common hallucination type h * by tallying a majority vote across all samples, denoted as h * = arg max h K i=1 I(h i = h). Then, we form the candidate reference set R by taking the corresponding r from the output containing the h * . We select the most "consistent" reference point r * by comparing the cosine similarities. For each r i in R, we first calculate its average cosine similarity with the other elements in R. After that, we select the reference point r * with the highest average cosine similarity: r * = arg max r i ∈R ( 1 n-1 n j=1,j̸ =i sim(r i , r j )). Finally, with (r * , h * ), we can uniquely select the corresponding f * .Step. Following the robust estimation in the E-step, the M-step updates the model parameters to maximize the likelihood of the selected output y * . Combining Eq. 1 and Eq. 2, we formulate the parameter update strategy at iteration t: Table 1 : The dataset size for ANAH-v2 in different stages, including the number of topics, model responses, and annotated sentences. EQUATIONGrounded in the EM algorithm, our framework operates in an iterative manner. This multi-iteration process acts as a data growth flywheel to progressively scale up the dataset in multiple dimensions, consisting of three stages:Stage 1: Seed Data and Basic Annotator. We utilize ANAH dataset [29] as our seed data, which includes over 700 topics and around 4,300 LLM-generated questions and responses. For each response, ANAH provides the hallucination type for every sentence, determined through a human-inthe-loop approach. We train an initial hallucination annotator, noted as ANAH-v2 Stage1, with this seed data using the annotation method described in § 3.1. Stage 3: Scaling up in Topic Dimension. We expand the topic coverage along four categories: location, person, event, and thing, paralleling ANAH's configuration. For each topic, we generate several questions based on the provided reference documents (more details in Appendix B). Then, we use the same method in Stage 2 to collect responses from multiple models and annotate the response following the same procedure as in Stage 2, using ANAH-v2 Stage2 annotator. The resulting dataset, combined with data from the previous stages, is used to train the ultimate annotator version.Overal Statistics. The final dataset encompasses both over ∼3k topics, ∼196k model responses, and ∼822k annotated sentences, in English and Chinese (Tab. 1). The topics cover celebrities, events, locations, and things, and span a wide array of domains, such as politics, health, and sports (Fig. 3 ).The statistics underscore the comprehensiveness and extensive scale of our dataset.Hallucination Evaluation. As the accuracy of the hallucination annotators becomes satisfactory, we can apply it to automate the process of evaluating the hallucination levels of existing open-source models. After categorizing sentences into four distinct types (introduced in § 3.1), we consider type Contradictory and Unverifiable Hallucination as sentences with hallucinations, and type No Fact and No Hallucination as sentences without hallucinations. This tool enables researchers to assess the reliability and accuracy of generated texts, ensuring models can be responsibly integrated into practical applications.Hallucination Mitigation. We further show a simple re-ranking strategy to mitigate the LLM's hallucinations with the annotator, whereas more advanced strategies can be explored in future research. Specifically, we adopt our annotator θ for response re-ranking. LLM first generates N candidate responses {G 1 , • • • , G N } by top-k sampling. Then we select the best response G * with the lowest hallucination rate over all the generated responses as below:EQUATIONwhere a θ,i,n is the generated annotation type by θ given the input x i,n including a question, the i-th sentence to be annotated from G n , and a reference document. A C and A U means sentence type Contradictory and Unverifiable Hallucination, respectively. L n is the sentence number of G n . 
experimental setup: 
 Implementation. In our experimental framework, we adopt the pre-trained InternLM2-7B [7] model to fine-tune the hallucination annotator. Further implementation details can be found in Appendix C.Evaluation. We use a subset of the ANAH [29] data as a test set, which is not used for training in stage 1. To assess the performance of the annotator in predicting hallucination types, we utilize F1 and Accuracy. We also employ RougeL [42] and BertScore [72] to compare the generated text with gold-standard human reference in terms of gram, continuity, order and semantics. Notably, the RougeL and BERTScore of GPT-4 are higher than ANAH-v2. Because GPT-4 is used for the initial pre-annotation during the construction of ANAH [29] . Subsequently, humans refine these pre-annotations, and humans tend to not change the pre-annotations. This methodology inherently aligns the final 'golden' answers closely with the outputs by GPT-4. Therefore, we tend to use "accuracy" as our primary metric because the type judgment is determinative of the annotation quality. For example, an annotation that wrongly judges type (low accuracy) but finds the correct reference fragment (high RougeL/BERTScore) remains completely unacceptable. In addition, we conduct an LLM-based evaluation to exclude the similarity due to pre-annotations in Appendix D.F1 ↑ ACC ↑ R ↑ BERT ↑ GPT-We also observe that ANAH-v2 already outperforms ANAH-20B at Stage 1 (84.85% v.s. 81.01% in accuracy) with only 7B parameters, when being trained on the same hallucination corpus. This superior performance is attributed to the innovative multi-turn dialogue training strategy ( § 3.1).Impact of Self-Consistency.To verify the effectiveness of self-consistency during inference in E-Step (introduced in § 3.2), we compare the performance of the annotator with different selfconsistency settings in Tab. 3. When the annotator model with the same training data at each data scaling stage, the inference strategy with self-consistency (w/ SC) consistently outperforms without self-consistency (w/o SC), where the annotator generates only once for each input. Therefore, self-consistency improves the accuracy and stability of the estimation of hallucination annotations. In M-Step, we train the model on data from the E-Step of the preceding iteration. We observe that when the annotator model with the same inference strategy, the model trained on self-consistently processed data (w/ SC) surpasses the performance with data generated through a single pass (w/o SC). This finding indicates that training data processed through self-consistency leads to a stronger annotator. This improvement can be attributed to the reduced distribution variance between the inferred labels and true labels.Impact of Progressive Data Scaling. To assess the impact of progressive data scaling (introduced in §3.3), we compare the performance of annotators with different types of data scaling in Tab. 4.In our progressive approach, the updated annotator from Stage 2 is employed to annotate the responses from additional topics, continuously enriching the training data. Conversely, in the nonprogressive approach, the basic annotator from Stage 1 is employed to generate annotations for the additional training data during Stage 3. With the same size of training data, the annotator trained on non-progressive data scaling underperforms that with our progressive data scaling, proving the effectiveness of our progressive data scaling.We also analyze different training strategies for annotators in different data scaling stages in Tab. 5. In our default training process, we mix the newly annotated data with old data to re-train an annotator. Alternatively, we only use the newly annotated data to further train the annotator model from the previous stage. The results demonstrate that our training strategy with mixed training data performs better than further training with new data. The integration of different data qualities across training stages improves the robustness of the annotator model.We further validate the effectiveness of ANAH-v2 on other hallucination detection datasets using two third-party datasets: HaluEval [40] for English and HalluQA [14] for Chinese. Each dataset provides four components: questions, reference documents, responses, and labels indicating whether the responses contain hallucination. For each question, we let ANAH-v2 judge the type of responses containing and not containing the hallucination separately. Note that in HaluEval we only use the QA The primary metric we use for evaluation is Accuracy in determining the type of response. We compare the zero-shot performance of ANAH-v2 with current SOTA results on HaluEval achieved by KnowHalu [71] and baseline results by GPT-4.The results in Tab. 6 reveal that our annotation model achieves notable accuracies on both HaluEval and HalluQA. Remarkably, ANAH-v2-Stage3 obtains new SOTA accuracy on HaluEval (81.54%) and HalluQA (94.44%) even under a zero-shot setting, underscoring the generalization capability of ANAH-v2. Moreover, we find that ANAH-v2-Stage3 outperforms the annotators from Stage1 and Stage2, further proving the data scaling strategy effectively stabilizes performance when dealing with unfamiliar responses.Hallucination Evaluation Benchmark.Our ANAH-v2 dataset and annotator can serve as a benchmark for the hallucination levels in generated texts by existing models. As shown in Tab. 7, we evaluate the performance of various LLMs, including InternLM2 [7] , Qwen1.5 [2] , Baichuan2 [4] , Mistral [33, 34] , DeepSeek-LLM [6] , and Llama2 [59] , spanning different model sizes. We also offer detailed evaluation results on different languages and categories of topics to deepen our understanding.We find that all models exhibit superior performance in English compared to Chinese, underscoring the need for further research to understand and mitigate language-dependent discrepancy. The performances of all models with reference documents are better than those without. Qwen1.5-14B achieves the lowest hallucination rate when using reference documents (5.33%) and Deepseek-67B achieves the lowest hallucination rate when reference documents are not provided (47.17%). Moreover, we find no clear trend in the performance distribution across four categories of topics. In addition, the results of different stages of annotators in Tab. A2, A3, and 7 show that there is a consistent trend and fixed biased ordering relationship between LLMs, thus confirming the reliability of our assessment method. More details are in Appendix E.Hallucination Mitigation. Besides being used to measure hallucination levels, ANAH-v2 can also be used to mitigate hallucinations. We use the QA samples from HaluEval, which comprises questions and correct answers from HotPotQA [68] . We use two models InternLm2-7B and LLaMA2-7B. For each model, we generate 36 candidate responses by top-k sampling (k=40), then re-rank the responses using our annotator. To quantify the hallucination degree, we employ RougeL, BertScore, NLI, and QuestionEval. These metrics measure the congruence between the generated responses with the golden responses and/or reference documents. Table 8 : Evaluation results for hallucination mitigation with LLaMA2-7B and InternLM2-7B on HaluEval. Here, "baseline" means the direct generation results, and "re-rank" means the results with our re-ranking mitigation method.Results in Tab. 8 show a clear reduction of hallucination levels after the re-ranking process via our annotator. For instance, the NLI metric for LLaMA2-7B shows a notable increase, rising from 25.00% to 37.01%. This suggests that the application of our annotative approach can significantly mitigate the issue of hallucinations in language model outputs. 
conclusion and future work: 
 In this paper, we aim to explore a scalable framework for the oversight of LLM hallucinations. Through iterative self-training, we progressively expand the diversity and scale of the dataset and improve the accuracy of the hallucination annotator. The finally obtained ANAH-v2, for the first time, outperforms GPT-4 in various hallucination detection benchmarks with only 7B parameters and obtains superior zero-shot performance on third-party hallucination detection benchmarks. ANAH-v2 not only provides an automatic hallucination evaluation benchmark with the scaled dataset, which paves the way for future research on hallucination mitigation but also exhibits potential in hallucination mitigation by the simple re-ranking strategy. We believe ANAH-v2 can also benefit more hallucination mitigation strategies such as fine-grained RLHF.With the large-scale dataset as seed data, future work can explore creating hallucination annotation data in other NLG tasks such as dialogue generation. Another direction is to improve the generalizability of the annotator across different languages, tasks, and topics. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We have discussed the broader impacts of our proposed method in the appendix G.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our method does not involve these risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have provided detailed citations for the datasets and other models used in our paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: As described in § 3.1, our annotation process consists of three phases: (1) Factual Existence Judgment via the prompt in Fig. A1 , (2) Reference Information Extraction via the prompt in Fig. A2 (3) Hallucination-Type Judgment via the prompt in Fig. A3 .As described in § 3.3, we collect model responses via Fig. A5 . The open-source models include InternLM2(7B&20B) [7] , Baichuan2 (7B&13B) [4] , LLama2 (7B&13B) [59] , Qwen1.5 (7B&14B&72B) [2] , Deepseek (7B&67B) [6] , and Mistral (7B&7×8B) [33, 34] .We automate the topic selection based on occurrence frequency via Google Ngram Viewer 4 and retrieve corresponding reference documents from pre-training databases [27] .We generate questions on each topic via Fig. A4 .In our experimental framework, we adopt the pre-trained InternLM2-7B [7] model to fine tuning the hallucination annotator.In E-Step, we generate responses by implementing sampling via the LMDeploy library [17] . During each iteration, we generate 32 candidate responses per input and apply a self-consistency quality control mechanism to them. The decoding strategy involves the top-k (k = 40) sampling with a temperature of 0.8.In M-Step, we train the annotator model with the following settings and hyper-parameters: the epoch is 1, the learning rate is 1e-5, and the AdamW optimizer is with a linear scheduler, the maximum sequence length is set to 32k. Additionally, following the configuration in ANAH [29] , we perform a multi-task setting where additional tasks such as dialogue generation from ShareGPT [49] and Dolly [16] are integrated with the fine-grained hallucination annotation. Our model is trained on 32 NVIDIA A100 GPUs. Table A1 : The score assessing the consistency of generated reference points with the source documents.To exclude the similarity due to the impact of preannotations described in § 4.2, we conduct an LLM-based evaluation to assess the consistency of generated reference points with the source document. Specifically, we followed the prompt in FactScore [46] , which aims to clarify whether the generated reference points are supported by the given source document, rather than simply calculating the similarity between them using metrics such as RougeL or BERTScore. We employ the InternLM2-7B-Chat [7] as the estimator.As shown in Tab. A1, the results indicate that the reliability of our model's generated reference points progressively improves and ultimately exceeds that of GPT4. This trend is consistent with F1 and ACC in Tab 2.To assess the reliability of our hallucination annotator, we measure the hallucination levels of the above LLMs using the annotator from different stages. Tab. A2, A3, and 7 show the results measured by annotator ANAH-v2 from Stage 1, 2, and 3, respectively. The trends in these three tables are consistent where Qwen1.5-14B achieves the lowest hallucination rate with reference documents and DeepseekLM-67B achieves the lowest hallucination rate without reference documents. This consistency and fixed biased ordering relationship between LLMs confirm the reliability of our assessment method.Although this study presents a novel multi-iteration self-training framework for the scalable oversight of LLM hallucinations and achieves significant improvements in hallucination annotation, there are some limitations.Despite the progressive scaling and increasing accuracy of the hallucination annotator, there may still exist a non-negligible margin of error in the annotations. This margin could affect the convergence of the model and the quality of the final hallucination annotator. Furthermore, the success of our framework is measured largely by its performance on our own dataset and other benchmarks such as HalluEval and HalluQA. However, these datasets might not encompass the full spectrum of real-world scenarios where hallucinations pose a problem. Lastly, this work primarily uses InternLM2-7B as the backbone of the hallucination annotator. Other different underlying models and different numbers of parameters are not explored.In addition, the EM algorithm, which is the theoretical foundation of our framework, may also introduce some problems. For example, the EM algorithm is sensitive to initial conditions, which would impact the convergence process. Although we employ many methods to ensure the stability of the training process, such as selecting a high-quality, human-labeled hallucination dataset as the seed and using a progressive scaling strategy, we cannot claim that we have eventually converged to a globally optimal solution. Moreover, the iterative EM algorithm requires computational effort.Our method uses 32 A100 GPUs to iteratively train the 7B model. It took approximately 100 hours for inference and training. Based on the price of the computing platform Lambda (1.29 USD per GPU per hour), it costs 4,128 USD. However, using the "manual + GPT4-assisted" annotation model, as described in ANAH [29] (0.9 USD and 20 minutes per annotation), it would take 177,237 USD and 65,643 hours to reach the size of the dataset in our work. So we believe our method is a better trade-off between computing resources and labour+API costs, which is acceptable.By exploring the hallucination annotation and mitigation in LLMs, this paper contributes to the development of more reliable and trustworthy AI technologies. Our innovative multi-iterative selftraining framework significantly reduces the reliance on expensive and time-consuming manual annotations by automating the hallucination detection process. Our hallucination annotator offers a benchmark for the research community evaluating the hallucination levels of existing open-source models. Additionally, we provide a large-scale and diverse dataset from which the broader research community can benefit, fostering further innovation and study in this domain.You will act as a fact checker, and I will provide you with a question and a corresponding partial answer. Your task is to determine whether the content of the answer contains verifiable facts. ## Judgment Criteria:-Verifiable Facts: Specific, objective points of information that can be verified through data, research results, or other reliable sources. Examples include statistical data, historical events, scientific laws, and specific case studies.-Non-factual Descriptions: Personal opinions, subjective judgments, or unverifiable statements. ## Task Process: 1. Carefully read the question, which is as follows: {question} 2. Carefully read the partial answer, which is as follows: {annotation} 3. Conduct the Analysis: Based on the above judgment criteria, determine if the answer contains verifiable facts.-If there are no verifiable facts in the answer, output "<No Facts>".-If there are verifiable facts in the answer, output "<Facts Present>".Chinese Prompt: English Prompt:你将作为一个事实判断器，我会给你提供一个问题和一个针对该问题的部分回答，你的任务是判断回 答中的内容是否存在可以判断的事实。 ## 判断标准： -可以判断的事实：具体的、客观的信息点，这些信息可以通过数据、研究结果或其他可靠来源进行 验证。例如，统计数据、历史事件、科学定律、具体案例等。 -非事实描述：个人意见、主观判断或 无法验证的声明。 ## 任务流程： 1. 仔细阅读问题，问题如下： {question} 2. 仔细阅读回答，部分回答如下： {annotation} 3. 进行分析：根据上述判断标准，判断回答中是否包含可以判断的事实。 -如果回答中不存在可以判断的事实，则输出"<无事实>"。 -如果回答中存在可以判断的事实，则 输出"<有事实>"。You will act as an information extractor. I will provide you with a question, a related reference document, and a partial answer to that question. Your task is to extract information from the reference document that is relevant to the question and answer. ## Operational Steps: 1. Carefully read the question, which is as follows: {question} 2. Carefully read the partial answer, which is as follows: {annotation} 3. Analyze the Reference Document: Identify information most relevant to the question and answer. This information may be completely the same, partially similar, or conflicting with the content of the answer. The reference document is as follows: {reference} 4. List the Relevant Information: List all the relevant information found in order, separated by <SEP> if there are multiple pieces of information. 5. Output When No Information Is Found: If no relevant information is found, output <No Reference Information>.Chinese Prompt: English Prompt: I would like you to act as a question generator. I will provide references and you will generate 10 questions about "{topic}" based on the reference. The specific requirements are as follows:你将作为一个信息提取器，我将给你提供一个问题、一份相关的参考文档，以及一个针对该问题的部 分回答，你的任务是从参考文档中提炼出与问题和回答相关的信息。 ## 操作步骤： 1. 仔细阅读问题，问题如下：{question} 2. 仔细阅读回答，部分回答如下：{annotation} 3. 分析参考文档： 找出与问题和回答最相关的信息，这些信息可能与回答内容完全相同、部分相同， 或存在冲突。参考文档如下：{reference} 4. 列出相关信息： 按顺序列出所有发现的相关信息，如果有多条信息的话以 <SEP> 作为分隔。 5. 无相关信息时输出： 如果没有找到相关信息，请输出<无参考信息>。1. the questions can be fully answered based only on the reference document, i.e. the answers to the questions are fully contained in the reference document. The questions should be objective and not too subjective or open-ended.2. the 10 questions should be of as many different types as possible, e.g. what, when, where, why. Questions can be asked from different perspectives, e.g. descriptions, explanations, reasons, etc. Ensure that the questions are of different types and cover all aspects of the information. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).The first three rows of data are from ANAH[29].The first six rows of data are from KnowHalu[71].https://books.google.com/ngrams/ 


Decision: This paper proposes an iterative self-training method to scale up hallucination annotation with better accuracy and uses the model to mitigate hallucination of LLMs. The proposed framework is scalable and efficient and shows strong performance on hallucination detection benchmarks. All the reviewers consistently acknowledge the contributions of this paper. Thus, I recommend acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes an iterative self-training framework that simultaneously and progressively scales up the hallucination annotation dataset and improves the accuracy of the hallucination annotator. The framework is based on the expectation maximization algorithm, alternately annotating a scaled dataset and training a more accurate hallucination annotator on the dataset. A 7 billion model trained by this framework can surpass GPT-4 and obtains state-of-the-art hallucination detection results on HaluEval and HalluQA by zero-shot inference.
strength_weakness: strength_weakness: The RougeL and BertScore may not be the most capable metrics for evaluating generated texts.
questions: questions: Is it possible to leverage large language model based evaluation for hallucination detection?
limitations: limitations: Appendix E describes the limitations.
paper_summary: paper_summary: The authors propose an innovative approach to tackle the persistent issue of hallucinations in large language models (LLMs) during long-form question-answering tasks. Current methods for detecting and mitigating these hallucinations are constrained by limited data and high labor costs. To address this, the paper introduces an iterative self-training framework that scales up the hallucination annotation dataset while simultaneously enhancing the accuracy of the annotators. Using the Expectation Maximization algorithm, the process involves multiple iterations where a pipeline annotates a growing dataset, and the improved annotator is used for the next cycle. This approach not only leads to a highly accurate annotator that surpasses GPT-4 but also effectively reduces hallucinations in LLM outputs. The results show significant improvements in key benchmarks, offering a scalable and efficient solution for managing LLM hallucinations.
strength_weakness: strength_weakness: 1. One of the big drawbacks of the EM algorithm is its sensitivity to initial conditions. If you don't start with the right parameters, the algorithm can easily get stuck in a local maximum instead of finding the best possible solution, which can be frustrating.

2.  The EM algorithm can be quite demanding in terms of computational resources. Each iteration involves a lot of number crunching, which means it can be slow and resource-intensive, especially when dealing with large datasets or complex models.

3. The authors could enhance their literature review by including several highly relevant papers on data annotation. The annotation task has been discussed in [1] [2] [3] [4] [5].

[1] https://arxiv.org/abs/2310.04668
[2] https://arxiv.org/abs/2303.15056
[3] https://arxiv.org/abs/2306.04349
[4] https://dl.acm.org/doi/pdf/10.1145/3613904.3642834
[5] https://dl.acm.org/doi/pdf/10.1145/3594536.3595161
questions: questions: See weaknesses.
limitations: limitations: No
paper_summary: paper_summary: This paper introduces an iterative self-training framework to address hallucinations in large language models (LLMs), enhancing the accuracy of annotators and scaling up hallucination detection datasets. The framework utilizes the Expectation Maximization algorithm to progressively improve the hallucination annotator's performance by annotating a scaled dataset and training a more accurate annotator in each iteration. Experimental results demonstrate that the enhanced annotator surpasses GPT-4 in hallucination detection, achieving state-of-the-art results on HaluEval and HalluQA.
strength_weakness: strength_weakness: 1. The rationale behind using the EM algorithm to solve this problem is not clearly articulated. What considerations led to this choice?
2. How does the paper ensure that the EM algorithm converges through iterations?
questions: questions: 1. The difficulty in automatically constructing high-quality hallucination datasets is not clearly explained. The relevant works in this area, the unresolved issues, and why these difficulties persist are not thoroughly discussed by the authors in the paper.
2. The rationale behind using the EM algorithm to solve this problem is not clearly articulated. What considerations led to this choice?
3. How does the paper ensure that the EM algorithm converges through iterations?
limitations: limitations: none
paper_summary: paper_summary: The paper proposes to augment hallucination annotation dataset and improve the performance of the hallucination annotator simultaneously in an iterative self-training framework. During each iteration, they use a Expectation-Maximization Algorithm for data annotation and hallucination annotator training. The trained hallucination annotator can be further used for downstream tasks such hallucination evaluation and hallucination mitigation. Extensive experiments were performed to show the superiority in hallucination detection and the effectiveness of the proposed approach.
strength_weakness: strength_weakness: - My biggest concern for this paper is that the model essentially does three things: Factual Existence Judgment, Reference Information Extraction, and Hallucination-Type Judgment. Each step relies on the results from previous step. For example, if the extracted reference information is limited, it will greatly impact the hallucination judgement as well. The experiments mainly focus on the third step without discussions on the previous two steps. So I’m not certain how stable the model actually is.
- The hallucination mitigation is a little weak, it basically generates multiple responses and select the response with the least portion of hallucination sentences. There are multiple other ways to mitigate hallucinations that are not compared in Table 8.
questions: questions: - Typo in ine 76-77, "we reduce the hallucination of the final LLM generations from 25% to 37%."
- Line 156, when you use the majority vote to select the the most common hallucination type, wouldn't it be dominanted by "No Hallucination"? As you are also showing in Table 7, the hallucination rate for InternLM2 is less than 20%. If you generate n candidates (depending on how big your n is) and use a majority votes, not many hallucination types will be selected.
- How many iterations did you do to get the dataset in Table 1?
- In Table 2, any discussions on GPT4 having much better RougeL and BertScore?
limitations: limitations: The limitations are discussed in Appendix which actually addresses some of the questions I was gonna ask, such as more benchmarks or more backbone models for the hallucination annotator.

Comments: title: Please let us know if your concerns have been addressed
comment: Dear Reviewer i3ot,

We would like to thank you for the thoughtful and constructive feedback and appreciate that you agree on the strengths of our paper. During the rebuttal, **we have provided more details and analysis to address your concerns.** As the discussion phase is nearing its end, we are warmly concerned whether our rebuttal addresses your concerns.

**It would be appreciated if you could raise your score on our paper if we address your concerns.** We thank you again for your effort in reviewing our paper.

Best regards,

Authors of Submission 10034.
Thank you for your response and for keeping the 8 (Strong Accept) rating. We are happy that our discussions and evaluation results about metrics have addressed your question. We will include these discussions in the final manuscript.
Thank you for your response. My question was addressed and I keep the rating.
Thank you for your response and for increasing the rating to 6 (Weak Accept). We are happy that our discussions on algorithm performance, computational resources, and related works are convincing. We will include these discussions in the final manuscript.
The authors have satisfactorily addressed most of my problems.  Most concerns has been addressed and some senarios may out of scope of this paper. I have raised my score. Once again, I want to express my gratitude for your hard work and commitment.