Title: ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Hallucination

Large Language Model

Fine-grained Annotation

Dataset
Abstract: Large language models (LLMs) exhibit hallucinations in long-form question-answering tasks across various domains and wide applications. Current hallucination detection and mitigation datasets are limited in domain and size, which struggle to scale due to prohibitive labor costs and insufficient reliability of existing hallucination annotators. To facilitate the scalable oversight of LLM hallucinations, this paper introduces an iterative self-training framework that simultaneously and progressively scales up the annotation dataset and improves the accuracy of the annotator. Based on the Expectation Maximization algorithm, in each iteration, the framework first applies an automatic hallucination annotation pipeline for a scaled dataset and then trains a more accurate annotator on the dataset.  This new annotator is adopted in the annotation pipeline for the next iteration. Extensive experimental results demonstrate that the finally obtained hallucination annotator with only 7B parameters surpasses GPT-4 and obtains new state-of-the-art hallucination detection results on HaluEval and HalluQA by zero-shot inference. Such an annotator can not only evaluate the hallucination levels of various LLMs on the large-scale dataset but also help to mitigate the hallucination of LLMs generations, with the Natural Language Inference metric increasing from 25% to 37% on HaluEval.

ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models

Abstract

Large language models (LLMs) exhibit hallucinations in long-form questionanswering tasks across various domains and wide applications. Current hallucination detection and mitigation datasets are limited in domain and size, which struggle to scale due to prohibitive labor costs and insufficient reliability of existing hallucination annotators. To facilitate the scalable oversight of LLM hallucinations, this paper introduces an iterative self-training framework that simultaneously and progressively scales up the annotation dataset and improves the accuracy of the annotator. Based on the Expectation Maximization algorithm, in each iteration, the framework first applies an automatic hallucination annotation pipeline for a scaled dataset and then trains a more accurate annotator on the dataset. This new annotator is adopted in the annotation pipeline for the next iteration. Extensive experimental results demonstrate that the finally obtained hallucination annotator with only 7B parameters surpasses GPT-4 and obtains new state-of-the-art hallucination detection results on HaluEval and HalluQA by zero-shot inference. Such an annotator can not only evaluate the hallucination levels of various LLMs on the large-scale dataset but also help to mitigate the hallucination of LLMs generations, with the Natural Language Inference metric increasing from 25% to 37% on HaluEval. 1

1 Introduction

Large Language Models (LLMs) have shown remarkable capabilities in various tasks [10, 11, 35, 51, 57]. However, they tend to produce hallucination , i.e. , plausible-sounding but unfaithful or nonsensical information [5, 30], that significantly hinders their real-world applications. Initial steps to address this issue involve the creation of datasets that can help to detect, annotate, and mitigate hallucinations [14, 29, 40]. Since the potential hallucinations of LLMs are in various fields, the spectrum of knowledge in the dataset is expected to be large-scale and comprehensive, covering various domains. Consequently, the size and diversity of datasets are critical for the oversight of LLM hallucinations.

However, constructing and scaling-up hallucination annotation datasets face significant hurdles [8, 9, 29, 43]. One primary challenge is the prohibitively high costs and labor intensity required for their accurate assessment [43, 47], since the fine-grained hallucination annotation requires intensives labor for reading long documents and annotating the hallucination details sentence by sentence. Moreover, due to the insufficiency of accurate human annotations, the reliability of existing hallucination annotators and detectors becomes another pressing concern [29]. These tools have been found to

produce inaccurate results [9, 46, 61], e.g. , even GPT4 [1], one of the most powerful LLMs, is not satisfactory and cannot achieve a compatible performance of humans [29].

Existing works [3, 25, 36, 38, 56, 58, 69] have explored strategies in data augmentation and selftraining to extend dataset size and boost the performance of models in the fields of image segmentation, multi-lingual translation, math reasoning, etc . However, how to scale the hallucination annotation datasets efficiently is under-explored in the community, which significantly hinders the in-depth analysis and further mitigation of LLMs hallucinations at a large scale.

To address the research gap, this paper proposes an iterative self-training framework designed to scale up the hallucination annotation dataset and simultaneously increase the accuracy of annotators (Fig. 1). The iterative framework can be explained from the perspective of the Expectation Maximization (EM) algorithm. In the Expectation (E) step, we apply the existing best hallucination annotator to estimate the ground-truth hallucination annotations of the scaled dataset. We adopt an inference pipeline on top of the annotator with self-consistency strategy [63] to provide a more robust estimation of the annotations, which lays the groundwork for training a more precise annotator in the subsequent step. In the Maximization (M) step, we combine the existing annotations with the scaled data annotations derived from the previous E steps to train a new hallucination annotator. Training on more

Dataset

Annotator

Size

89.55%

200K

88.18%

GPT4-

86.97%

More Responses

More Topics

12K

84.85%

ultra

Weak

Medium

Annotator

Annotator

Dataset

个

个

Small

Strong

Massive

Dataset

Annotator

Dataset

Stage1

Stage2

Stage3

data leads to a more accurate annotator and a more robust annotation pipeline, setting the stage for the subsequent round of annotations.

The iterative process consists of three stages of multi-dimensional data scaling as shown in Fig. 1. Initially, we train a weak annotator on human annotations. In the second stage, we collect the hallucination responses from more open-source LLMs for the same questions in the dataset to improve the generalization ability of hallucination annotators to model responses. In the third stage, we expand the number of topics and questions in the dataset and collect hallucination annotations with the more robust annotator. This progressive scaling strategy stabilizes the annotator's performance when evaluating familiar and unfamiliar responses across diverse topics.

Extensive experimental results show that our enhanced annotator significantly outperforms existing models, including the advanced GPT-4, in terms of accuracy. Our annotator not only performs best on the in-domain fine-grained hallucination annotation dataset ANAH (89.24%) but also obtains new state-of-the-art (SOTA) results on HaluEval (81.54%) and HalluQA (94.44%) under zero-shot setting. In addition, the annotator automates the hallucination evaluation on the dataset, offering a comprehensive benchmark for the research community to evaluate the hallucination levels of numerous open-source models, providing a practical reference for future hallucination mitigation of LLMs. Using a simple reranking strategy with the annotator, we reduce the hallucination of the final LLM generations on HaluEval, with the NLI metric increasing from 25% to 37%.

2 Related Work

Self-improvement of Large Models. As Large Language Models (LLMs) become more and more powerful, the community starts to explore different strategies to achieve the self-improvement of LLMs, i.e. , to improve the LLMs using the supervision from LLMs [12, 24, 28, 50, 53]. For example, existing works have explored self-alignment using LLMs with ethical principles [3, 58, 69]. There are also methods [13, 25, 38, 56, 70] strengthen LLM's capabilities on tasks such as reasoning by training the LLMs on the high-quality responses from themselves on the same questions. In the field of computer vision, SAM [36] introduces manual and model-assisted labeling to expand the image segmentation dataset and enhance the performance of image segmentation models. However, the application of self-improvement is under-explored in fine-grained hallucination annotation. This field

E-Step: Data Annotation

Iterative Optimization

Lager Dataset

Stronger Annotator

Input x

Annotator

From M-Step

Generate n

Candidate Output

Original Hallucination Dataset

Question:

contributions to mathematics?

Answer:

…

He wrote a book called 'Treatise on the

Circumference of a

Circle' …

Document:

His surviving mathematical works

…

Label:

<fact

<ref

point>

<hallu

type>

<Question>

<Answer>

<Document>


:

You will act as a fact checker.Your task

is

to determine whether the answer contains

verifiable

facts.

<answer>


:

<Facts Present>


:

You will act as an information extractor.

Your task is to extract information from

document that is relevant to the

question

and

answer.

<question>

<answer>

<document>


:

His surviving mathematical works include:

A commentary on the difficulties concerning

the postulates of Euclid's Elements

.


:

You will act as a Hallucination annotator.

You need to determine whether the

provided

answer contains any

hallucinatory content

and annotate the type of hallucination.


:

<Contradictory>

Align Data to Train Format

Factual

Existence

Judgment

Reference

Information

Extraction

Hallucination

Type Judgment

Output y*

<fact

<ref

point>

<hallu

type>

<fact

<ref

point>

<hallu

type>

Candidate y 1

Candidate y n

…

f* <fact info>

r* <ref point>

h* <hallu type>

Choosing

Representative

Output

Fine Tuning a Stronger Annotator

Construct a Larger Dataset

New Data

Unlabeled

Data Growth

Flywheel

New Data

Labeled

Self-

Consistency

is challenging for automatic annotators due to its meticulous nature, which requires fine validation with long documents. It is also noteworthy that most self-improvement works require extra resources such as human labor or a supplementary model [36, 38]. In contrast, our pipeline is self-sufficient, relying solely on the annotator model and the initial dataset.

Hallucination Annotation Dataset. The development of the hallucination annotation dataset is the cornerstone for detecting hallucinations in models' output. These datasets can be used to train a hallucination detector/annotator and evaluate the hallucination level via the detector/annotator. Early works [21, 22, 26, 37, 40, 44, 48, 60, 62, 67] in this domain tended to broadly classify entire responses as either hallucinatory or not, providing a coarse-grained analysis of hallucination occurrences. Recent works [29, 47] annotate hallucinations in a more fine-grained and meticulous way. Despite this progress, these datasets, especially those having fine-grained annotations, suffer from limitations in size and scalability due to the high costs associated with the usage of human annotators or commercial models like GPT4. In addition, the difficulty of this task and the limited human annotations result in unsatisfactory performance of the automatic hallucination annotator.

Hallucination Mitigation. Considering the harm of hallucinations, researchers have explored various techniques for mitigating hallucinations. Techniques such as multi-task learning [23, 65], model editing [18, 31], and fine-grained RLHF [66] are proposed to suppress hallucination tendencies during training. Alternative strategies have been proposed that do not require further model training, including different decoding strategies [15, 41, 52, 55], multi-agent methods [20], and variants of the Chain-of-Thought approach involving verification or reflection [19, 32, 39, 64]. Our ANAH-v2 shows efficiency in hallucination mitigation as a re-ranker and has the potential to combine with the existing methods such as fine-grained RLHF.

3 Method

This paper proposes an iterative self-training framework to simultaneously scale up the hallucination dataset and improve the accuracy of the hallucination annotator. We follow the analytical hallucination annotation (§ 3.1) to annotate the hallucination sentence-by-sentence. The multi-iteration framework is theoretically grounded in the EM algorithm (§ 3.2) and involves three stages to progressively scale the dataset in multiple dimensions (§ 3.3). We also reveal how the hallucination annotators can be applied for hallucination evaluation and mitigation (§ 3.4).

3.1 Analytical Hallucination Annotation

The aim of a hallucination annotator is to identify hallucinations in the model responses. ANAH [29] developed a fine-grained annotation method that locates reference points in the document for each sentence and makes hallucination-type judgments, with the whole process completed in one turn of dialog. However, this hybrid task diverges from the human judgment processes and fails to clearly indicate the relationship between reference points and hallucination judgments, resulting in unsatisfactory annotation accuracy.

Instead of using the original ANAH training prompts, we developed a more reliable training method tailored to the hallucination annotation process. As depicted in the lower right part of Fig. 2, the process is outlined in three phases: (1) Factual Existence Judgment , where the annotator assesses whether the provided sentence contains verifiable facts. If no factual content is present, the sentence is categorized as 'No Fact' and requires no further annotation. (2) Reference Information Extraction , where the annotator extracts relevant reference points from the documents related to the question and answer. (3) Hallucination-Type Judgment , where the annotator determines the type of hallucination based on the extracted reference points. If the sentence aligns with the references, it is classified as 'No Hallucination' . If it contradicts the references, it is deemed a 'Contradictory Hallucination' . If it lacks supporting evidence and cannot be verified, it is labeled as 'Unverifiable Hallucination' . The above three phases will form a multi-turn dialogue in training data. Compared to the ANAH approach, which involves simultaneous judgments on multiple criteria, our phased process aligns more closely with human cognitive judgment processes. The detailed data format and prompts for our annotation process are in Appendix A.

3.2 Expectation-Maximization Algorithm

Simultaneously scaling up the dataset and improving the accuracy of the annotator can be formulated by the EM algorithm. For the input set X , we need to estimate two hidden variables simultaneously, the output set Y and the model parameters θ . Specifically, based on the task formulation in § 3.1, we define the input x from the input set X of the hallucination annotator consists of a question, a sentence to be annotated, and a reference document. The expected output y to be estimated in the data output set Y includes the factual information f , the key reference points r from the reference document, and the type of hallucination h . We maximize the log-likelihood estimation of Y by alternately performing the E-Step and the M-Step to update the model parameters θ :

E-Step. A straightforward approach to estimating Y is to use a single model to predict annotations. However, this method lacks sufficient accuracy [45]. To improve the accuracy and stability of the estimation of Y , we introduce the self-consistency method [63], which provides a more robust representation of the distribution of the Y . As shown in Fig. 2. For each input x , we perform multiple samplings to yield K independent outputs y = { y 1 , · · · , y i , · · · , y K } , where the i -th output sample y i is composed of factual information ( f i ), reference point ( r i ) and hallucination type ( h i ). We use a self-consistency metric to select the most representative sample y ∗ among all outputs:

̸

During this selection process, we consider the hallucination type h , reference point r , and factual information f in turn. We determine the most common hallucination type h ∗ by tallying a majority vote across all samples, denoted as h ∗ = arg max h ∑ K i =1 I ( h i = h ) . Then, we form the candidate reference set R by taking the corresponding r from the output containing the h ∗ . We select the most 'consistent' reference point r ∗ by comparing the cosine similarities. For each r i in R , we first calculate its average cosine similarity with the other elements in R . After that, we select the reference point r ∗ with the highest average cosine similarity: r ∗ = arg max r i ∈ R ( 1 n -1 ∑ n j =1 ,j = i sim( r i , r j )) . Finally, with ( r ∗ , h ∗ ) , we can uniquely select the corresponding f ∗ .

M-Step. Following the robust estimation in the E-step, the M-step updates the model parameters to maximize the likelihood of the selected output y ∗ . Combining Eq. 1 and Eq. 2, we formulate the parameter update strategy at iteration t :

3.3 Multi-dimensional Data Scaling

Grounded in the EM algorithm, our framework operates in an iterative manner. This multi-iteration process acts as a data growth flywheel to progressively scale up the dataset in multiple dimensions, consisting of three stages:

Stage 1: Seed Data and Basic Annotator. We utilize ANAH dataset [29] as our seed data, which includes over 700 topics and around 4,300 LLM-generated questions and responses. For each response, ANAH provides the hallucination type for every sentence, determined through a human-inthe-loop approach. We train an initial hallucination annotator, noted as ANAH-v2 Stage1, with this seed data using the annotation method described in § 3.1.

Stage 2: Scaling up in Response Dimension. In Stage 1, for each question, ANAH provides responses that GPT-3.5 generates with the reference document, while InternLM-7B generates without any reference document. We first augment the dataset's model responses by collecting responses to the same existing questions from 13 additional open-source models of various sizes and series. For each model, responses were collected with and without knowledge of reference documents. The prompt details are in Appendix B. After filtering out similar model responses, these responses are annotated sentence by sentence using the self-consistency pipeline with ANAH-v2 Stage1. The newly annotated data, combined with the seed data, was used to train ANAH-v2 Stage2.

Stage 3: Scaling up in Topic Dimension. We expand the topic coverage along four categories: location, person, event, and thing, paralleling ANAH's configuration. For each topic, we generate several questions based on the provided reference documents (more details in Appendix B). Then, we use the same method in Stage 2 to collect responses from multiple models and annotate the response following the same procedure as in Stage 2, using ANAH-v2 Stage2 annotator. The resulting dataset, combined with data from the previous stages, is used to train the ultimate annotator version.

Overal Statistics. The final dataset encompasses both over ∼ 3k topics, ∼ 196k model responses, and ∼ 822k annotated sentences, in English and Chinese (Tab. 1). The topics cover celebrities, events, locations, and things, and span a wide array of domains, such as politics, health, and sports (Fig. 3). The statistics underscore the comprehensiveness and extensive scale of our dataset.

3.4 Applications

Hallucination Evaluation. As the accuracy of the hallucination annotators becomes satisfactory, we can apply it to automate the process of evaluating the hallucination levels of existing open-source models. After categorizing sentences into four distinct types (introduced in § 3.1), we consider type Contradictory and Unverifiable Hallucination as sentences with hallucinations, and type No Fact and No Hallucination as sentences without hallucinations. This tool enables researchers to assess the reliability and accuracy of generated texts, ensuring models can be responsibly integrated into practical applications.

Hallucination Mitigation. We further show a simple re-ranking strategy to mitigate the LLM's hallucinations with the annotator, whereas more advanced strategies can be explored in future research. Specifically, we adopt our annotator θ for response re-ranking. LLM first generates N candidate responses { G 1 , · · · , G N } by top-k sampling. Then we select the best response G ∗ with the lowest

Art:

Sports:

12.99%

2.41%

Nature:

Others:

1.12%

0.84%

Economy:

Medicine:

Sci.Technol.

4.49%

36.94%

Thing:

24.09%

Event:

3.83%

Philosophy:

13.54%

Person:18.29%

Location:

2.74%

Geography:

20.68%

3.41%

Jurisprudence:

Religion:

6.21%

3.95%

Literature:

Politics/Military:

7.18%

37.29%

hallucination rate over all the generated responses as below:

where a θ,i,n is the generated annotation type by θ given the input x i,n including a question, the i -th sentence to be annotated from G n , and a reference document. A C and A U means sentence type Contradictory and Unverifiable Hallucination , respectively. L n is the sentence number of G n .

4 Experiment

4.1 Experimental Setup

Implementation. In our experimental framework, we adopt the pre-trained InternLM2-7B [7] model to fine-tune the hallucination annotator. Further implementation details can be found in Appendix C.

Evaluation. We use a subset of the ANAH [29] data as a test set, which is not used for training in stage 1. To assess the performance of the annotator in predicting hallucination types, we utilize F1 and Accuracy . We also employ RougeL [42] and BertScore [72] to compare the generated text with gold-standard human reference in terms of gram, continuity, order and semantics.

4.2 Overall Results

The last 3 rows of Tab. 2 illustrate the performance of ANAH-v2 at each stage of Data Scaling in § 3.3. The performance progressively improves with the increasing dataset number (see in Tab. 1) in successive stages. This trend underscores the scalability and effectiveness of our hallucination annotation framework. Remarkably, ANAH-v2 surpasses GPT-4 with the F1 of 87.78% and the accuracy of 88.03% at Stage 2. Eventually, we achieve the F1 of 89.30% and the accuracy of 89.55% at Stage 3.

Notably, the RougeL and BERTScore of GPT-4 are higher than ANAH-v2. Because GPT-4 is used for the initial pre-annotation during the construction of ANAH [29]. Subsequently, humans refine these pre-annotations, and humans tend to not change the pre-annotations. This methodology inherently aligns the final 'golden' answers closely with the outputs by GPT-4. Therefore, we tend to use 'accuracy' as our primary metric because the type judgment is determinative of the annotation quality. For example, an annotation that wrongly judges type (low accuracy) but finds the correct reference fragment (high RougeL/BERTScore) remains completely unacceptable. In addition, we conduct an LLM-based evaluation to exclude the similarity due to pre-annotations in Appendix D.

We also observe that ANAH-v2 already outperforms ANAH-20B at Stage 1 (84.85% v.s. 81.01% in accuracy) with only 7B parameters, when being trained on the same hallucination corpus. This superior performance is attributed to the innovative multi-turn dialogue training strategy (§ 3.1).

4.3 Ablation Studies

Impact of Self-Consistency. To verify the effectiveness of self-consistency during inference in E-Step (introduced in § 3.2), we compare the performance of the annotator with different selfconsistency settings in Tab. 3. When the annotator model with the same training data at each data scaling stage, the inference strategy with self-consistency (w/ SC) consistently outperforms without self-consistency (w/o SC), where the annotator generates only once for each input. Therefore, self-consistency improves the accuracy and stability of the estimation of hallucination annotations.

In M-Step, we train the model on data from the E-Step of the preceding iteration. We observe that when the annotator model with the same inference strategy, the model trained on self-consistently processed data (w/ SC) surpasses the performance with data generated through a single pass (w/o SC). This finding indicates that training data processed through self-consistency leads to a stronger annotator. This improvement can be attributed to the reduced distribution variance between the inferred labels and true labels.

Impact of Progressive Data Scaling. To assess the impact of progressive data scaling (introduced in §3.3), we compare the performance of annotators with different types of data scaling in Tab. 4. In our progressive approach, the updated annotator from Stage 2 is employed to annotate the responses from additional topics, continuously enriching the training data. Conversely, in the nonprogressive approach, the basic annotator from Stage 1 is employed to generate annotations for the additional training data during Stage 3. With the same size of training data, the annotator trained on non-progressive data scaling underperforms that with our progressive data scaling, proving the effectiveness of our progressive data scaling.

Impact of Training Strategy. We also analyze different training strategies for annotators in different data scaling stages in Tab. 5. In our default training process, we mix the newly annotated data with old data to re-train an annotator. Alternatively, we only use the newly annotated data to further train the annotator model from the previous stage. The results demonstrate that our training strategy with mixed training data performs better than further training with new data. The integration of different data qualities across training stages improves the robustness of the annotator model.

4.4 Generalization Capability Analysis

We further validate the effectiveness of ANAH-v2 on other hallucination detection datasets using two third-party datasets: HaluEval [40] for English and HalluQA [14] for Chinese. Each dataset provides four components: questions, reference documents, responses, and labels indicating whether the responses contain hallucination. For each question, we let ANAH-v2 judge the type of responses containing and not containing the hallucination separately. Note that in HaluEval we only use the QA

samples and in HalluQA, we only use the samples that provide a textual reference document, which aligns with our annotator's designed setting.

The primary metric we use for evaluation is Accuracy in determining the type of response. We compare the zero-shot performance of ANAHv2 with current SOTA results on HaluEval achieved by KnowHalu [71] and baseline results by GPT-4.

The results in Tab. 6 reveal that our annotation model achieves notable accuracies on both HaluEval and HalluQA. Remarkably, ANAHv2-Stage3 obtains new SOTA accuracy on HaluEval (81.54%) and HalluQA (94.44%) even under a zero-shot setting, underscoring the generalization capability of ANAH-v2. Moreover, we find that ANAH-v2-Stage3 outperforms the annotators from Stage1 and Stage2, further proving the data scaling strategy effectively stabilizes performance when dealing with unfamiliar responses.

4.5 Application

Hallucination Evaluation Benchmark. Our ANAH-v2 dataset and annotator can serve as a benchmark for the hallucination levels in generated texts by existing models. As shown in Tab. 7, we evaluate the performance of various LLMs, including InternLM2 [7], Qwen1.5 [2], Baichuan2 [4], Mistral [33, 34], DeepSeek-LLM [6], and Llama2 [59], spanning different model sizes. We also offer detailed evaluation results on different languages and categories of topics to deepen our understanding.

We find that all models exhibit superior performance in English compared to Chinese, underscoring the need for further research to understand and mitigate language-dependent discrepancy. The performances of all models with reference documents are better than those without. Qwen1.5-14B achieves the lowest hallucination rate when using reference documents (5.33%) and Deepseek67B achieves the lowest hallucination rate when reference documents are not provided (47.17%). Moreover, we find no clear trend in the performance distribution across four categories of topics. In addition, the results of different stages of annotators in Tab. A2, A3, and 7 show that there is a consistent trend and fixed biased ordering relationship between LLMs, thus confirming the reliability of our assessment method. More details are in Appendix E.

Hallucination Mitigation. Besides being used to measure hallucination levels, ANAH-v2 can also be used to mitigate hallucinations. We use the QA samples from HaluEval, which comprises questions and correct answers from HotPotQA [68]. We use two models InternLm2-7B and LLaMA2-7B. For each model, we generate 36 candidate responses by top-k sampling (k=40), then re-rank the responses using our annotator. To quantify the hallucination degree, we employ RougeL, BertScore, NLI, and QuestionEval. These metrics measure the congruence between the generated responses with the golden responses and/or reference documents.

Results in Tab. 8 show a clear reduction of hallucination levels after the re-ranking process via our annotator. For instance, the NLI metric for LLaMA2-7B shows a notable increase, rising from 25.00% to 37.01%. This suggests that the application of our annotative approach can significantly mitigate the issue of hallucinations in language model outputs.

5 Conclusion and Future Work

In this paper, we aim to explore a scalable framework for the oversight of LLM hallucinations. Through iterative self-training, we progressively expand the diversity and scale of the dataset and improve the accuracy of the hallucination annotator. The finally obtained ANAH-v2, for the first time, outperforms GPT-4 in various hallucination detection benchmarks with only 7B parameters and obtains superior zero-shot performance on third-party hallucination detection benchmarks. ANAHv2 not only provides an automatic hallucination evaluation benchmark with the scaled dataset, which paves the way for future research on hallucination mitigation but also exhibits potential in

hallucination mitigation by the simple re-ranking strategy. We believe ANAH-v2 can also benefit more hallucination mitigation strategies such as fine-grained RLHF.

With the large-scale dataset as seed data, future work can explore creating hallucination annotation data in other NLG tasks such as dialogue generation. Another direction is to improve the generalizability of the annotator across different languages, tasks, and topics.

References

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

Bai, J., Bai, S., Chu, Y ., Cui, Z., Dang, K., Deng, X., Fan, Y ., Ge, W., Han, Y ., Huang, F., et al.: Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al.: Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 (2022)

Baichuan: Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305 (2023), https://arxiv.org/abs/2309.10305

Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., Do, Q.V., Xu, Y., Fung, P.: A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023)

Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al.: Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954 (2024)

Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., et al.: Internlm2 technical report. arXiv preprint arXiv:2403.17297 (2024)

Cao, Z., Yang, Y., Zhao, H.: Autohall: Automated hallucination dataset generation for large language models. arXiv preprint arXiv:2310.00259 (2023)

Chen, X., Song, D., Gui, H., Wang, C., Zhang, N., Yong, J., Huang, F., Lv, C., Zhang, D., Chen, H.: Factchd: Benchmarking fact-conflicting hallucination detection. IJCAI abs/2310.12086 (2023), https://api.semanticscholar.org/CorpusID:264289140

Chen, Z., Du, W., Zhang, W., Liu, K., Liu, J., Zheng, M., Zhuo, J., Zhang, S., Lin, D., Chen, K., Zhao, F.: T-eval: Evaluating the tool utilization capability step by step. CoRR abs/2312.14033 (2023). https://doi.org/10.48550/ARXIV.2312.14033, https://doi.org/10.48550/arXiv. 2312.14033

Chen, Z., Liu, K., Wang, Q., Zhang, W., Liu, J., Lin, D., Chen, K., Zhao, F.: Agent-flan: Designing data and methods of effective agent tuning for large language models. CoRR abs/2403.12881 (2024). https://doi.org/10.48550/ARXIV.2403.12881, https://doi.org/10.48550/arXiv. 2403.12881

Chen, Z., Mao, H., Wen, H., Han, H., Jin, W., Zhang, H., Liu, H., Tang, J.: Label-free node classification on graphs with large language models (llms). arXiv preprint arXiv:2310.04668 (2023)

Chen, Z., Deng, Y., Yuan, H., Ji, K., Gu, Q.: Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335 (2024)

Cheng, Q., Sun, T., Zhang, W., Wang, S., Liu, X., Zhang, M., He, J., Huang, M., Yin, Z., Chen, K., et al.: Evaluating hallucinations in chinese large language models. arXiv preprint arXiv:2310.03368 (2023)

Chuang, Y.S., Xie, Y., Luo, H., Kim, Y., Glass, J., He, P.: Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883 (2023)

Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., Xin, R.: Free dolly: Introducing the world's first truly open instruction-tuned llm (2023), https://www.databricks.com/blog/2023/04/12/ dolly-first-open-commercially-viable-instruction-tuned-llm

Contributors, L.: Lmdeploy: A toolkit for compressing, deploying, and serving llm. https: //github.com/InternLM/lmdeploy (2023)

Daheim, N., Dziri, N., Sachan, M., Gurevych, I., Ponti, E.M.: Elastic weight removal for faithful and abstractive dialogue generation. arXiv preprint arXiv:2303.17574 (2023)

Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., Weston, J.: Chain-ofverification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495 (2023)

Du, Y., Li, S., Torralba, A., Tenenbaum, J.B., Mordatch, I.: Improving factuality and reasoning in language models through multiagent debate (5 2023), http://arxiv.org/abs/2305.14325

Durmus, E., He, H., Diab, M.: Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 5055-5070 (2020)

Dziri, N., Kamalloo, E., Milton, S., Zaiane, O., Yu, M., Ponti, E.M., Reddy, S.: Faithdial: A faithful benchmark for information-seeking dialogue. Transactions of the Association for Computational Linguistics 10 , 1473-1490 (2022)

Garg, S., Peitz, S., Nallasamy, U., Paulik, M.: Jointly learning to align and translate with transformer models. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 4453-4462 (2019)

Gilardi, F., Alizadeh, M., Kubli, M.: Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences 120 (30), e2305016120 (2023)

Gulcehre, C., Paine, T.L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al.: Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998 (2023)

Gupta, P., Wu, C.S., Liu, W., Xiong, C.: Dialfact: A benchmark for fact-checking in dialogue. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 3785-3801 (2022)

He, C., Li, W., Jin, Z., Wang, B., Xu, C., Lin: Opendatalab: Empowering general artificialintelligence with open datasets. https://opendatalab.com (2022)

He, Z., Huang, C.Y., Ding, C.K.C., Rohatgi, S., Huang, T.H.K.: If in a crowdsourced data annotation pipeline, a gpt-4. In: Proceedings of the CHI Conference on Human Factors in Computing Systems. pp. 1-25 (2024)

Ji, Z., Gu, Y ., Zhang, W., Lyu, C., Lin, D., Chen, K.: Anah: Analytical annotation of hallucinations in large language models. arXiv preprint arXiv:2405.20315 (2024)

Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y ., Madotto, A., Fung, P.: Survey of hallucination in natural language generation. ACM Computing Surveys (2022)

Ji, Z., Liu, Z., Lee, N., Yu, T., Wilie, B., Zeng, M., Fung, P.: RHO: Reducing hallucination in open-domain dialogues with knowledge grounding. In: Rogers, A., BoydGraber, J., Okazaki, N. (eds.) Findings of the Association for Computational Linguistics: ACL 2023. pp. 4504-4522. Association for Computational Linguistics, Toronto, Canada (Jul 2023). https://doi.org/10.18653/v1/2023.findings-acl.275, https://aclanthology.org/ 2023.findings-acl.275

Ji, Z., Yu, T., Xu, Y., Lee, N., Ishii, E., Fung, P.: Towards mitigating hallucination in large language models via self-reflection. EMNLP Findings (2023)

Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al.: Mistral 7b. arXiv preprint arXiv:2310.06825 (2023)

Jiang, A.Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D.S., Casas, D.d.l., Hanna, E.B., Bressand, F., et al.: Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024)

Kamalloo, E., Dziri, N., Clarke, C.L.A., Rafiei, D.: Evaluating open-domain question answering in the era of large language models. In: Rogers, A., Boyd-Graber, J.L., Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023. pp. 5591-5606. Association for Computational Linguistics (2023). https://doi.org/10.18653/v1/2023.acl-long.307, https://doi.org/10.18653/v1/2023.acl-long.307

Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4015-4026 (2023)

Laban, P., Schnabel, T., Bennett, P.N., Hearst, M.A.: Summac: Re-visiting nli-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics 10 , 163-177 (2022)

Lee, N., Wattanawong, T., Kim, S., Mangalam, K., Shen, S., Anumanchipali, G., Mahoney, M.W., Keutzer, K., Gholami, A.: Llm2llm: Boosting llms with novel iterative data enhancement. arXiv preprint arXiv:2403.15042 (2024)

Lei, D., Li, Y., Wang, M., Yun, V., Ching, E., Kamal, E., et al.: Chain of natural language inference for reducing large language model ungrounded hallucinations. arXiv preprint arXiv:2310.03951 (2023)

Li, J., Cheng, X., Zhao, W.X., Nie, J.Y., Wen, J.R.: Halueval: A large-scale hallucination evaluation benchmark for large language models. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 6449-6464 (2023)

Li, K., Patel, O., Viégas, F., Pfister, H., Wattenberg, M.: Inference-time intervention: Eliciting truthful answers from a language model. arXiv preprint arXiv:2306.03341 (2023)

Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out. pp. 74-81 (2004)

Liu, J., Fu, Y., Xie, R., Xie, R., Sun, X., Lian, F., Kang, Z., Li, X.: Phd: A prompted visual hallucination evaluation dataset. arXiv preprint arXiv:2403.11116 (2024)

Liu, T., Zhang, Y., Brockett, C., Mao, Y., Sui, Z., Chen, W., Dolan, B.: A token-level reference-free hallucination detection benchmark for free-form text generation. arXiv preprint arXiv:2104.08704 (2021)

Mena, J., Pujol, O., Vitrià, J.: A survey on uncertainty estimation in deep learning classification systems from a bayesian perspective. ACM Computing Surveys (CSUR) 54 (9), 1-35 (2021)

Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.t., Koh, P.W., Iyyer, M., Zettlemoyer, L., Hajishirzi, H.: FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In: EMNLP (2023), https://arxiv.org/abs/2305.14251

Mishra, A., Asai, A., Balachandran, V., Wang, Y., Neubig, G., Tsvetkov, Y., Hajishirzi, H.: Fine-grained hallucination detection and editing for language models. arXiv preprint arXiv:2401.06855 (2024)

Muhlgay, D., Ram, O., Magar, I., Levine, Y., Ratner, N., Belinkov, Y., Abend, O., Leyton-Brown, K., Shashua, A., Shoham, Y.: Generating benchmarks for factuality evaluation of language models. arXiv preprint arXiv:2307.06908 (2023)

None: Sharegpt (2023), https://huggingface.co/datasets/RyokoAI/ShareGPT52K

Pei, X., Li, Y., Xu, C.: Gpt self-supervision for a better data annotator. arXiv preprint arXiv:2306.04349 (2023)

Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., et al.: Kilt: a benchmark for knowledge intensive language tasks. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 2523-2544 (2021)

Rebuffel, C., Roberti, M., Soulier, L., Scoutheeten, G., Cancelliere, R., Gallinari, P.: Controlling hallucinations at word level in data-to-text generation. Data Mining and Knowledge Discovery 36 (1), 318-354 (2022)

Savelka, J.: Unlocking practical applications in legal domain: Evaluation of gpt for zero-shot semantic annotation of legal texts. In: Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law. pp. 447-451 (2023)

Semnani, S., Yao, V., Zhang, H., Lam, M.: Wikichat: Stopping the hallucination of large language model chatbots by few-shot grounding on wikipedia. In: Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 2387-2413 (2023)

Shi, W., Han, X., Lewis, M., Tsvetkov, Y., Zettlemoyer, L., Yih, S.W.t.: Trusting your evidence: Hallucinate less with context-aware decoding. arXiv preprint arXiv:2305.14739 (2023)

Singh, A., Co-Reyes, J.D., Agarwal, R., Anand, A., Patil, P., Liu, P.J., Harrison, J., Lee, J., Xu, K., Parisi, A., et al.: Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585 (2023)

Sun, K., Xu, Y.E., Zha, H., Liu, Y., Dong, X.L.: Head-to-tail: How knowledgeable are large language models (llm)? aka will llms replace knowledge graphs? arXiv preprint arXiv:2308.10168 (2023)

Sun, Z., Shen, Y., Zhou, Q., Zhang, H., Chen, Z., Cox, D., Yang, Y., Gan, C.: Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems 36 (2024)

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

Varshney, N., Yao, W., Zhang, H., Chen, J., Yu, D.: A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation. arXiv preprint arXiv:2307.03987 (2023)

Vectara: vectara hallucination evaluation model. https://huggingface.co/vectara/ hallucination_evaluation_model

Wang, A., Cho, K., Lewis, M.: Asking and answering questions to evaluate the factual consistency of summaries. arXiv preprint arXiv:2004.04228 (2020)

Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)

Wang, Z., Mao, S., Wu, W., Ge, T., Wei, F., Ji, H.: Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona selfcollaboration. arXiv preprint arXiv:2307.05300 1 (2), 3 (2023)

Weng, R., Yu, H., Wei, X., Luo, W.: Towards enhancing faithfulness for neural machine translation. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 2675-2684 (2020)

Wu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P., Smith, N.A., Ostendorf, M., Hajishirzi, H.: Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01693 (2023)

Yang, S., Sun, R., Wan, X.: A new benchmark and reverse validation method for passage-level hallucination detection. arXiv preprint arXiv:2310.06498 (2023)

Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W.W., Salakhutdinov, R., Manning, C.D.: HotpotQA: A dataset for diverse, explainable multi-hop question answering. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) (2018)

Yuan, W., Pang, R.Y., Cho, K., Sukhbaatar, S., Xu, J., Weston, J.: Self-rewarding language models. arXiv preprint arXiv:2401.10020 (2024)

Yuan, Z., Yuan, H., Li, C., Dong, G., Tan, C., Zhou, C.: Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825 (2023)

Zhang, J., Xu, C., Gai, Y., Lecue, F., Song, D., Li, B.: Knowhalu: Hallucination detection via multi-form knowledge based factual checking. arXiv preprint arXiv:2404.02935 (2024)

Zhang*, T., Kishore*, V., Wu*, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with bert. In: International Conference on Learning Representations (2020), https: //openreview.net/forum?id=SkeHuCVFDr

A Training Prompt

As described in § 3.1, our annotation process consists of three phases: (1) Factual Existence Judgment via the prompt in Fig. A1, (2) Reference Information Extraction via the prompt in Fig. A2 (3) Hallucination-Type Judgment via the prompt in Fig. A3.

B Data Scaling Details

As described in § 3.3, we collect model responses via Fig. A5. The open-source models include InternLM2(7B&20B) [7], Baichuan2 (7B&13B) [4], LLama2 (7B&13B) [59], Qwen1.5 (7B&14B&72B) [2], Deepseek (7B&67B) [6], and Mistral (7B&7 × 8B) [33, 34].

We automate the topic selection based on occurrence frequency via Google Ngram Viewer 4 and retrieve corresponding reference documents from pre-training databases [27].

We generate questions on each topic via Fig. A4.

C Implementation Details

In our experimental framework, we adopt the pre-trained InternLM2-7B [7] model to fine tuning the hallucination annotator.

In E-Step , we generate responses by implementing sampling via the LMDeploy library [17]. During each iteration, we generate 32 candidate responses per input and apply a self-consistency quality control mechanism to them. The decoding strategy involves the top-k (k = 40) sampling with a temperature of 0.8.

In M-Step , we train the annotator model with the following settings and hyper-parameters: the epoch is 1, the learning rate is 1e-5, and the AdamW optimizer is with a linear scheduler, the maximum sequence length is set to 32k. Additionally, following the configuration in ANAH [29], we perform a multi-task setting where additional tasks such as dialogue generation from ShareGPT [49] and Dolly [16] are integrated with the fine-grained hallucination annotation. Our model is trained on 32 NVIDIA A100 GPUs.

D Addtional Evaluation for the Overall Results

To exclude the similarity due to the impact of preannotations described in § 4.2, we conduct an LLM-based evaluation to assess the consistency of generated reference points with the source document. Specifically, we followed the prompt in FactScore [46], which aims to clarify whether the generated reference points are supported by the given source document, rather than simply calculating the similarity between them using metrics such as RougeL or BERTScore. We employ the InternLM2-7B-Chat [7] as the estimator.

As shown in Tab. A1, the results indicate that the reliability of our model's generated reference points progressively improves and ultimately exceeds that of GPT4. This trend is consistent with F1 and ACC in Tab 2.

E Hallucination Evaluation

To assess the reliability of our hallucination annotator, we measure the hallucination levels of the above LLMs using the annotator from different stages. Tab. A2, A3, and 7 show the results measured by annotator ANAH-v2 from Stage 1, 2, and 3, respectively. The trends in these three tables are

Table A1: The score assessing the consistency of generated reference points with the source documents.

consistent where Qwen1.5-14B achieves the lowest hallucination rate with reference documents and DeepseekLM-67B achieves the lowest hallucination rate without reference documents. This consistency and fixed biased ordering relationship between LLMs confirm the reliability of our assessment method.

F Limitation

Although this study presents a novel multi-iteration self-training framework for the scalable oversight of LLM hallucinations and achieves significant improvements in hallucination annotation, there are some limitations.

Despite the progressive scaling and increasing accuracy of the hallucination annotator, there may still exist a non-negligible margin of error in the annotations. This margin could affect the convergence of the model and the quality of the final hallucination annotator. Furthermore, the success of our framework is measured largely by its performance on our own dataset and other benchmarks such as HalluEval and HalluQA. However, these datasets might not encompass the full spectrum of real-world scenarios where hallucinations pose a problem. Lastly, this work primarily uses InternLM2-7B as the backbone of the hallucination annotator. Other different underlying models and different numbers of parameters are not explored.

In addition, the EM algorithm, which is the theoretical foundation of our framework, may also introduce some problems. For example, the EM algorithm is sensitive to initial conditions, which would impact the convergence process. Although we employ many methods to ensure the stability of the training process, such as selecting a high-quality, human-labeled hallucination dataset as the seed and using a progressive scaling strategy, we cannot claim that we have eventually converged to a globally optimal solution. Moreover, the iterative EM algorithm requires computational effort. Our method uses 32 A100 GPUs to iteratively train the 7B model. It took approximately 100 hours for inference and training. Based on the price of the computing platform Lambda (1.29 USD per GPU per hour), it costs 4,128 USD. However, using the "manual + GPT4-assisted" annotation model, as described in ANAH [29] (0.9 USD and 20 minutes per annotation), it would take 177,237 USD and 65,643 hours to reach the size of the dataset in our work. So we believe our method is a better trade-off between computing resources and labour+API costs, which is acceptable.

G Broader Impacts

By exploring the hallucination annotation and mitigation in LLMs, this paper contributes to the development of more reliable and trustworthy AI technologies. Our innovative multi-iterative selftraining framework significantly reduces the reliance on expensive and time-consuming manual annotations by automating the hallucination detection process. Our hallucination annotator offers a benchmark for the research community evaluating the hallucination levels of existing open-source models. Additionally, we provide a large-scale and diverse dataset from which the broader research community can benefit, fostering further innovation and study in this domain.

English Prompt:

You will act as a fact checker, and I will provide you with a question and a corresponding partial answer. Your task is to determine whether the content of the answer contains verifiable facts. ## Judgment Criteria:

- Verifiable Facts: Specific, objective points of information that can be verified through data, research results, or other reliable sources. Examples include statistical data, historical events, scientific laws, and specific case studies.

Non-factual Descriptions: Personal opinions, subjective judgments, or unverifiable statements.

## Task Process:

Carefully read the question, which is as follows: { question }

Carefully read the partial answer, which is as follows: { annotation }

Conduct the Analysis: Based on the above judgment criteria, determine if the answer contains verifiable facts.

If there are no verifiable facts in the answer, output '<No Facts>' .

If there are verifiable facts in the answer, output '<Facts Present>' .

Chinese Prompt:

你 将 作 为一个事 实 判 断 器 ， 我 会 给 你 提 供 一个 问 题 和 一个 针 对 该 问 题 的 部 分 回 答 ， 你 的 任 务 是 判 断 回 答 中 的 内 容 是 否 存 在 可 以 判 断 的 事 实 。

## 判 断 标 准 ：

-可 以 判 断 的 事 实 ： 具 体 的 、 客 观 的 信 息 点 ， 这 些 信 息 可 以 通 过 数 据 、 研 究 结 果 或 其 他 可 靠 来 源 进 行 验 证 。 例 如 ， 统 计 数 据 、 历史 事件 、 科 学定 律 、 具 体 案 例 等 。 -非 事 实 描 述 ： 个人 意 见 、 主 观 判 断 或 无 法 验 证 的 声 明 。

## 任 务 流 程 ：

仔 细 阅 读 问 题 ， 问 题 如 下 ： { question }

仔 细 阅 读 回 答 ， 部 分 回 答 如 下 ： { annotation }

进 行 分 析 ： 根 据 上 述 判 断 标 准 ， 判 断 回 答 中 是 否 包 含 可 以 判 断 的 事 实 。

-如 果 回 答 中不 存 在 可 以 判 断 的 事 实 ， 则 输 出 ' < 无 事 实 > ' 。 -如 果 回 答 中 存 在 可 以 判 断 的 事 实 ， 则 输 出 ' < 有 事 实 > ' 。

English Prompt:

You will act as an information extractor. I will provide you with a question, a related reference document, and a partial answer to that question. Your task is to extract information from the reference document that is relevant to the question and answer.

## Operational Steps:

Carefully read the question, which is as follows: { question }

Carefully read the partial answer, which is as follows: { annotation }

Analyze the Reference Document: Identify information most relevant to the question and answer. This information may be completely the same, partially similar, or conflicting with the content of the answer. The reference document is as follows: { reference }

List the Relevant Information: List all the relevant information found in order, separated by <SEP> if there are multiple pieces of information.

Output When No Information Is Found: If no relevant information is found, output <No Reference Information> .

Chinese Prompt:

你 将 作 为一个 信 息 提 取 器 ， 我 将 给 你 提 供 一个 问 题 、 一份 相 关 的 参 考 文 档 ， 以 及 一个 针 对 该 问 题 的 部 分 回 答 ， 你 的 任 务 是 从 参 考 文 档 中 提 炼 出 与 问 题 和 回 答 相 关 的 信 息 。

## 操 作 步 骤 ：

仔 细 阅 读 问 题 ， 问 题 如 下 ： { question }

仔 细 阅 读 回 答 ， 部 分 回 答 如 下 ： { annotation }

分 析 参 考 文 档 ： 找 出 与 问 题 和 回 答 最 相 关 的 信 息 ， 这 些 信 息 可 能 与 回 答 内 容完 全 相 同 、 部 分 相 同 ， 或 存 在 冲 突 。 参 考 文 档 如 下 ： { reference }

列 出 相 关 信 息 ： 按 顺 序 列 出 所 有 发 现 的相 关 信 息 ， 如 果有 多 条 信 息 的 话 以 <SEP> 作 为 分 隔 。

无 相 关 信 息 时 输 出 ： 如 果 没 有 找 到 相 关 信 息 ， 请 输 出 < 无 参 考 信 息 > 。

English Prompt:

You will act as a 'Hallucination' annotator. I will provide you with a question, a partial answer to that question, and related reference points. You need to determine whether the provided answer contains any hallucinatory content and annotate the type of hallucination.

'Hallucination' refers to content that contradicts the reference points or is unsupported by them. ## Judgment Criteria:

1. No Hallucination: If the answer is completely consistent with the reference points and does not introduce any contradictory information, output: <No Hallucination> .

Contradiction: If the answer clearly contradicts the reference points, output: <Contradictory> .

3. Unverifiable: If the answer contains information not mentioned in the reference points and cannot be supported or verified by them, output: <Unverifiable> .

## Task Process:

Carefully read the question, which is as follows: { question }

Carefully read the partial answer, which is as follows: { annotation }

Carefully read the reference points, which are as follows: { reference }

Conduct the analysis: Based on the above judgment criteria, determine if the answer contains hallucinations and output the type of hallucination.

Chinese Prompt:

你 将 作 为一个 ' 幻 觉 ' 标 注 器 ， 我 将 会 给 你 提 供 一个一个 问 题 ， 一个 针 对 该 问 题 的 部 分 回 答 和 相 关 的 参 考 要 点 。 你 需 要 判 断 提 供 的 回 答 中 是 否含 有 幻 觉 性 内 容 ， 并 标 注 幻 觉 类 型 。

' 幻 觉 ' 指 的 是 与 参 考 要 点 相 矛 盾 或 在 参 考 要 点 中 没 有 依 据 的 内 容 。

## 判 断 准 则 ：

1. 无 幻 觉 ： 如 果 回 答 与 参 考 要 点 完 全 一 致 ， 且 没 有 引 入 与 参 考 要 点 相 矛 盾的 信 息 ， 请 输 出 ： < 无 幻 觉 > 。

矛 盾 ： 如 果 回 答 内 容 与 参 考 要 点 存 在 明显 矛 盾 ， 请 输 出 ： < 矛 盾 > 。

3. 无 法 验 证 ： 如 果 回 答 包 含 的 信 息 在 参 考 要 点 中 没 有 提 及 ， 且 无 法 从 参 考 要 点 中 得 到 支 持 或 验 证 ， 请 输 出 ： < 无 法 验 证 > 。

## 任 务 流 程 ：

仔 细 阅 读 问 题 ， 问 题 如 下 ： { question }

仔 细 阅 读 回 答 ， 部 分 回 答 如 下 ： { annotation }

仔 细 阅 读 参 考 要 点 ， 参 考 要 点 如 下 ： { reference }

进 行 分 析 ： 根 据 上 述 判 断 标 准 ， 判 断 回 答 中 是 否 包 含 幻 觉 ， 并 输 出 幻 觉 类 型 。

English Prompt:

I would like you to act as a question generator. I will provide references and you will generate 10 questions about "{topic}" based on the reference. The specific requirements are as follows:

1. the questions can be fully answered based only on the reference document, i.e. the answers to the questions are fully contained in the reference document. The questions should be objective and not too subjective or open-ended.

2. the 10 questions should be of as many different types as possible, e.g. what, when, where, why. Questions can be asked from different perspectives, e.g. descriptions, explanations, reasons, etc. Ensure that the questions are of different types and cover all aspects of the information.

3. 10 questions can cover different levels of knowledge, from general, basic knowledge to more specialized, complex subject knowledge or domain knowledge.

have only one question per item.

Reference: {reference document}

Please list the 10 questions directly based on the above reference without any explanation:

Chinese Prompt:

我 希 望 你 充 当 一个 问 题 生 成 器 。 我 将 提 供 参 考 资 料 ， 你 将 根 据 资 料 生 成 关 于 ' {topic} ' 的 10 个 问 题 。 具 体 要 求 如 下 ：

1. 只 根 据 参 考 资 料 ， 完 全 可 以 回 答 问 题 ， 即 问 题 的 答 案 完 全 包 含 在 参 考 资 料 中 。 问 题 要 客 观 ， 不 要 太 过 主 观 和 开 放 。

2. 10 个 问 题 尽 量 是 不 同 类 型 的 ， 比 如 ： 什么 、 何 时 、 何 地 、 为什么 。 问 题 可 以从不 同 的 角 度 出 发 ， 例 如 描 述 、 解 释 、 原 因 等 。 确 保 问 题 类 型 多 样 ， 覆 盖 资 料 的 各 个 方 面 。

3. 10 个 问 题 可 以 涉 及 不 同 层 次 的 知 识 ， 从 常 识 性 、 基 本 性 的 知 识 ， 到 更 专业 化 、 复 杂 化 的 学 科 知 识 或 领 域 知 识 。

每 条 只 有 一个 问 题 。

参 考 资 料 ： {reference document}

请 根 据 以上 参 考 资 料 ， 不 做 说 明 直 接 列 出 10 个 问 题 ：

English Prompt:

Reference document: {reference document}

Please answer the question based on the above reference: {question}

Chinese Prompt:

参 考 资 料 ： {reference document}

请 根 据 以上 参 考 资 料 ， 回 答 问 题 ： {question}

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and introduction, we have comprehensively summarized and presented our methods and contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations of our proposed method in the appendix F. Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided comprehensive experimental details in the appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Dataset, code, and model are released at https://github.com/ open-compass/ANAH .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided comprehensive experimental details in the appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We have validated the effectiveness of our method through a large number of experiments in § 4 Experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided comprehensive experimental details in the appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research complies with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the broader impacts of our proposed method in the appendix G.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our method does not involve these risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have provided detailed citations for the datasets and other models used in our paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes] ,

Justification: Dataset, code, and model are released at https://github.com/ open-compass/ANAH .

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our research does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our research does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to augment hallucination annotation dataset and improve the performance of the hallucination annotator simultaneously in an iterative self-training framework. During each iteration, they use a Expectation-Maximization Algorithm for data annotation and hallucination annotator training. The trained hallucination annotator can be further used for downstream tasks such hallucination evaluation and hallucination mitigation. Extensive experiments were performed to show the superiority in hallucination detection and the effectiveness of the proposed approach.\n\nSTRENGTHS:\n- The paper is very easy and pleasant to read. Each section and their subsections are well connected and explained without adding much redundant content. Figure 2 is especially illustrative.\n- I like how they explain a rather practical work in a theoretical manner using all the math equations in Section 3. It makes the work more sound with less confusions.\n- Although the model name is ANAH-v2, it actually brings big novelty compared to the original ANAH model. The model is in a self-training manner and the training prompt is expanded to do three tasks.\n- Large amount of experiments show impressive performance on hallucination detection. The thorough ablation study also shows the importance of each component.\n\nWEAKNESSES:\n- My biggest concern for this paper is that the model essentially does three things: Factual Existence Judgment, Reference Information Extraction, and Hallucination-Type Judgment. Each step relies on the results from previous step. For example, if the extracted reference information is limited, it will greatly impact the hallucination judgement as well. The experiments mainly focus on the third step without discussions on the previous two steps. So I’m not certain how stable the model actually is.\n- The hallucination mitigation is a little weak, it basically generates multiple responses and select the response with the least portion of hallucination sentences. There are multiple other ways to mitigate hallucinations that are not compared in Table 8.\n\nQUESTIONS:\n- Typo in ine 76-77, \"we reduce the hallucination of the final LLM generations from 25% to 37%.\"\n- Line 156, when you use the majority vote to select the the most common hallucination type, wouldn't it be dominanted by \"No Hallucination\"? As you are also showing in Table 7, the hallucination rate for InternLM2 is less than 20%. If you generate n candidates (depending on how big your n is) and use a majority votes, not many hallucination types will be selected.\n- How many iterations did you do to get the dataset in Table 1?\n- In Table 2, any discussions on GPT4 having much better RougeL and BertScore?\n\nLIMITATIONS:\nThe limitations are discussed in Appendix which actually addresses some of the questions I was gonna ask, such as more benchmarks or more backbone models for the hallucination annotator.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an iterative self-training framework to address hallucinations in large language models (LLMs), enhancing the accuracy of annotators and scaling up hallucination detection datasets. The framework utilizes the Expectation Maximization algorithm to progressively improve the hallucination annotator's performance by annotating a scaled dataset and training a more accurate annotator in each iteration. Experimental results demonstrate that the enhanced annotator surpasses GPT-4 in hallucination detection, achieving state-of-the-art results on HaluEval and HalluQA.\n\nSTRENGTHS:\n1. The paper addresses a unique perspective on the issue, as many current works focus on hallucination detection and mitigation. The focus on automatically constructing high-quality hallucination datasets is crucial for addressing hallucinations in large language models. This is an important and valuable contribution to the field.\n2. The structure of the paper is clear, and the writing is concise. The experimental section is detailed and thorough, enhancing the credibility of the results.\n\nWEAKNESSES:\n1. The rationale behind using the EM algorithm to solve this problem is not clearly articulated. What considerations led to this choice?\n2. How does the paper ensure that the EM algorithm converges through iterations?\n\nQUESTIONS:\n1. The difficulty in automatically constructing high-quality hallucination datasets is not clearly explained. The relevant works in this area, the unresolved issues, and why these difficulties persist are not thoroughly discussed by the authors in the paper.\n2. The rationale behind using the EM algorithm to solve this problem is not clearly articulated. What considerations led to this choice?\n3. How does the paper ensure that the EM algorithm converges through iterations?\n\nLIMITATIONS:\nnone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an innovative approach to tackle the persistent issue of hallucinations in large language models (LLMs) during long-form question-answering tasks. Current methods for detecting and mitigating these hallucinations are constrained by limited data and high labor costs. To address this, the paper introduces an iterative self-training framework that scales up the hallucination annotation dataset while simultaneously enhancing the accuracy of the annotators. Using the Expectation Maximization algorithm, the process involves multiple iterations where a pipeline annotates a growing dataset, and the improved annotator is used for the next cycle. This approach not only leads to a highly accurate annotator that surpasses GPT-4 but also effectively reduces hallucinations in LLM outputs. The results show significant improvements in key benchmarks, offering a scalable and efficient solution for managing LLM hallucinations.\n\nSTRENGTHS:\n1. One of the major strengths of this paper is its scalable framework. By automating the annotation process and iterating through larger datasets, the authors manage to overcome the usual limitations of manual data labeling, making the approach both cost-effective and efficient.\n\n2. The application of the Expectation Maximization algorithm to improve annotator accuracy through iterative training is a standout feature. This method not only refines the annotations with each cycle but also ensures that the annotator becomes progressively more reliable, which is a clever and effective use of existing statistical techniques.\n\nWEAKNESSES:\n1. One of the big drawbacks of the EM algorithm is its sensitivity to initial conditions. If you don't start with the right parameters, the algorithm can easily get stuck in a local maximum instead of finding the best possible solution, which can be frustrating.\n\n2.  The EM algorithm can be quite demanding in terms of computational resources. Each iteration involves a lot of number crunching, which means it can be slow and resource-intensive, especially when dealing with large datasets or complex models.\n\n3. The authors could enhance their literature review by including several highly relevant papers on data annotation. The annotation task has been discussed in [1] [2] [3] [4] [5].\n\n[1] https://arxiv.org/abs/2310.04668\n[2] https://arxiv.org/abs/2303.15056\n[3] https://arxiv.org/abs/2306.04349\n[4] https://dl.acm.org/doi/pdf/10.1145/3613904.3642834\n[5] https://dl.acm.org/doi/pdf/10.1145/3594536.3595161\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper proposes an iterative self-training framework that simultaneously and progressively scales up the hallucination annotation dataset and improves the accuracy of the hallucination annotator. The framework is based on the expectation maximization algorithm, alternately annotating a scaled dataset and training a more accurate hallucination annotator on the dataset. A 7 billion model trained by this framework can surpass GPT-4 and obtains state-of-the-art hallucination detection results on HaluEval and HalluQA by zero-shot inference.

STRENGTHS:
- The paper is well motivated by the fact that large language model hallucination significantly hinders applications but its annotation is difficult and very labor intensive.
- The solution based on self-training effectively and feasibly addresses the above challenge. (1) The paper defines a procedure of analytical hallucination annotation, that aligns with human cognitive processes. (2) Staged multi-dimensional data scaling, collecting synthetic data from more large language models and for more numbers of topics and questions, ensures the richness of the dataset. (3) Leveraging EM algorithm is suitable.
- Strong empirical results are shown for both in-domain hallucination detection and on existing benchmarks, HaluEval and HalluQA.
- Very well-written paper.

WEAKNESSES:
The RougeL and BertScore may not be the most capable metrics for evaluating generated texts.

QUESTIONS:
Is it possible to leverage large language model based evaluation for hallucination detection?

LIMITATIONS:
Appendix E describes the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-16