Venue: NAACL 2024

# Language Models Hallucinate, but May Excel at Fact Verification

Jian Guan\({}^{1+}\), Jesse Dodge\({}^{2}\), David Wadden\({}^{2}\), Minlie Huang\({}^{1\dagger}\), Hao Peng\({}^{3*}\)

\({}^{1}\)The CoAI group, DCST, Institute for Artificial Intelligence,

###### Abstract

Recent progress in natural language processing (NLP) owes much to remarkable advances in large language models (LLMs). Nevertheless, LLMs frequently "hallucinate," resulting in non-factual outputs. Our carefully-designed human evaluation substantiates the serious hallucination issue, revealing that even GPT-3.5 produces factual outputs less than 25% of the time. This underscores the importance of fact verifiers in order to measure and incentivize progress. Our systematic investigation affirms that LLMs can be repurposed as effective fact verifiers with strong correlations with human judgments. Surprisingly, FLAN-T5\({}_{\rm 11B}\), the least factual generator in our study, performs the best as a fact verifier, even outperforming more capable LLMs like GPT3.5 and ChatGPT. Delving deeper, we analyze the reliance of these LLMs on high-quality evidence, as well as their deficiencies in robustness and generalization ability. Our study presents insights for developing trustworthy generation models.

languageresourceLanguage Model Predictive Learning

**Language Models Hallucinate, but May Excel at Fact Verification**

+
Footnote †: Corresponding author.

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1090–1111

June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

LLMs have demonstrated remarkable performance across various natural language generation (NLG) tasks Brown et al. (2020); OpenAI (2023); Chowdhery et al. (2022). However, they persistently suffer from the _hallucination_ problem Bang et al. (2023), often generating non-factual and sometimes misleading outputs. This is quantitatively substantiated by the first part of this paper. In our carefully designed human evaluation of several current LLMs, GPT-3.5 only manages to produce factual outputs less than 25% of the time; other models perform even worse. Such underperformance is achieved on Wikipedia, a domain that they have been extensively trained in and intuitively "familiar with." Our findings highlight the serious challenge that the hallucination issue presents, and underscore the crucial importance of developing effective fact verification methods Vlachos and Riedel (2014). These methods are central to evaluating and incentivizing progress in improving LLMs' factuality.

In the second part of the paper, we explore the prospect of leveraging instruction-tuned LLMs for fact verification. We hypothesize that, despite struggling to generate factual outputs, they may still be able to judge whether a piece of text is factual--a task that intuitively appears easier, at least for sentence-level judgments. Our systematic investigation affirms this hypothesis, especially when LLMs are augmented with retrieval components. Specifically, given a statement to be verified, we retrieve evidence from an external corpus and reframe the statement and evidence into a prompt to instruct an LLM to judge the factuality. We then normalize the LLM's generation probabilities of pre-defined answers as the factuality score, which shows stronger correlations with human judgments than previous statistical and model-based methods. Extensive experiments further reveal that FLAN-T5\({}_{\rm 11B}\)Chung et al. (2022), the least factual generator in our study, even surprisingly outperforms GPT3.5 and ChatGPT for fact verification.

We further analyze the LLM-based fact verifiers from the following perspectives: **(1) Influence of given evidence:** ChatGPT is susceptible to irrelevant evidence but deals with relevant but counterfactual evidence better than FLAN-T5\({}_{\rm 11B}\). **(2) Robustness:** GPT variants are less robust to different prompts than FLAN-T5\({}_{\rm 11B}\); **(3) Generalization ability:** It is more difficult to evaluate sentences that are from larger generators, dependent on the context or involving numerals. Evaluating paragraphs is also challenging, and can be facilitated by aggregating judgments of individual, de-contextualized sentences rather than evaluating them directly. Our contributions are as follows:

**I.** Well-designed human evaluation affirms the seri

ous challenges that current LLMs frequently hallucinate, even in their familiar Wikipedia domain.

**II.** We explore the potential of LLMs to assess factuality on multiple domains and analyze their reliance on given evidence, robustness, and generalization ability. These findings may inspire the development of trustworthy generation models and fact verification methods in future research. 1. The evaluation suite also serves as a new comprehensive benchmark for hallucination evaluation2.

Footnote 1: The data and evaluation scripts are publicly available at [https://github.com/JianGuanTHU/LLMforFV](https://github.com/JianGuanTHU/LLMforFV).

Footnote 2: All the data and evaluation scripts will be made public.

**III.** Based on our study, we recommend the following practices for fact verification: minimizing irrelevant evidence, taking sentences as base units for long paragraph verification, and de-contextualizing context-dependent sentences before verification.

## 2 Related Work

HallucinationMany metrics have been proposed to measure hallucinations for directed generation tasks such as summarization, including statistical and model-based metrics. Statistical metrics focus on lexical input-output matching Dhingra et al. (2019); Wang et al. (2020); Shuster et al. (2021). Model-based ones further capture semantic-level variations, including unsupervised metrics based on information extraction (IE) Nan et al. (2021), question answering (QA) Wang et al. (2020) and natural language inference (NLI) Laban et al. (2022), and supervised or semi-supervised metrics trained on specific datasets of evaluation-related tasks Izacard and Grave (2021); Kryscinski et al. (2020). These metrics can potentially adapt to open-ended generation by measuring mismatching between outputs and retrieved evidence. One additional challenge lies in retrievers possibly producing noisy, redundant, or contradictory evidence.

Fact VerificationLots of datasets have been collected towards fact verification in various domains, e.g., politics Vlachos and Riedel (2014), encyclopedia Thorne et al. (2018, 2021); Eisenschlos et al. (2021), news Perez-Rosas et al. (2018), climate Diggelmann et al. (2020), science Wadden et al. (2020), and healthcare Kotonya and Toni (2020). Honovich et al. (2022) aggregated multiple datasets to assess the ability to measure input-output consistency. Statements in all above datasets usually contain only single sentences and are crafted by either crawling from dedicated websites Vlachos and Riedel (2014), manually mutating sentences from factual articles Thorne et al. (2018) or re-framing QA pairs Thorne et al. (2021). We further involve model-generated statements and paragraph-level evaluation in our study.

LLMs as EvaluatorsThere are many active efforts to use LLMs' generated answers or generation probabilities for NLG evaluation Yuan et al. (2021); Colombo et al. (2022); Ke et al. (2022). More recent studies show high correlations of ChatGPT with human judgments for evaluating summarization, story generation, etc. Wang et al. (2023); Luo et al. (2023); Li et al. (2023). SelfCheckGPT Manakul et al. (2023) judged the factuality of a model output based on its similarity with other sampled outputs from the same model, and does not apply to non-model-generated statements or model-agnostic generation. FActScoreMin et al. (2023) used LLMs to evaluate people's biography generation through the fraction of atomic facts supported by retrieved evidence. In contrast, we focus on open-ended generation around various entities and analyze LLMs' robustness and generalization ability.

## 3 Quantifying LLMs' Hallucination

Our first research question is

_To what extent do current LLMs hallucinate?_

We quantify this through human evaluation, with a specific focus on the Wikipedia domain, which serves as a reliable information source for annotators. While clean and credible resources exist for other domains such as science and finance, it remains challenging for individuals lacking the expertise to evaluate LLMs' outputs in these domains.

Generation ModelsWe consider four representative LLMs, including FLAN-T511b Chung et al. (2022), LLama\({}_{\text{30B}}\), LLama65B Touvron et al. (2023) and GPT3.53. These LLMs vary in model architectures, model sizes, accessibility and training manners. We expect to establish a clear relationship between these variables and final hallucination performance.

Footnote 3: In this work, InstructionGPT/GPT3.5/ChatGP7/ refers to the OpenAI’s API “text-davinci-002”/“text-davinci-003”/"grt- 



**Completion (SentCom):** completing a sentence following the first two tokens of a factual claim from the test set of FEVER Thorne et al. (2018), a fact verification dataset. The training set provides abundant factual and non-factual claims, enabling us to assess whether supervised verifiers can generalize to model outputs in SS4. **(2) Wikipedia Paragraph Generation (ParaGen)**: generating a paragraph of five sentences about a given entity from Wikipedia. Here the outputs are expected to be longer, and the setting focuses on more long-tailed topics than (1).

We generate 50 outputs for two tasks, respectively, using four generation models with greedy decoding4, leading to \(50\times 2\times 4=400\) statements totally. During generation, we provide five manually selected factual demonstrations to the models, as shown in Tab. 1.

Footnote 4: Aksitov et al. (2023) showed that lower temperatures generally lead to less variability and potentially higher factuality.

Human AnnotationWe collect human workers' fractuality judgments of LLMs' outputs through Amazon Mechanical Turk (AMT). Each HIT (human intelligence task) contains five statements with the same input--four generated and one gold. Three workers are hired to search for evidence from Wikipedia and annotate the factuality label of each sentence in the statements, including factual, unfractual and not sure, as shown in Fig. 15. We further instruct workers to choose reasons if they annotate not sure, and exclude sentences from our evaluation set that are labeled as subjective or hard to understand by at least one worker since such sentences can be ambiguous to determine the factuality Guo et al. (2022). We discard low-quality submissions using well-designed rules and ensure each sentence has three valid annotations. The Fleiss's kappa score Fleiss and Joseph (1971) is 0.91/0.74 for SentCom/ParaGen, indicating a substantial inter-annotator agreement. Finally, we use majority voting to obtain sentence-level human judgments. As for paragraph-level judgments for ParaGen, we first truncate each paragraph to ensure all sentences are not subjective or hard to understand, and then label the paragraph unfractal if _any_ of the sentences are labeled contextual, factual if all sentences are labeled factual, and not sure otherwise. A paragraph contains 4.3 sentences on average after truncation. Since the only valid reason for not sure is no evidence found, we call the label no evidence

Figure 1: The annotation interface for ParaGen; that for SentCom is similar.



onward. Appendix A.2 presents more details.

Tab. 2 summarizes the annotation results: **(1)** Larger models tend to generate more factual outputs. **(2)** GPT3.5, the best-performing generator in this study, yields factual paragraphs less than 25% of the time. **(3)** All models generate notably more factual outputs for SentCom than ParaGen. We conjecture it is because the average frequency of inputs for SentCom is \({\sim}335\) times more than that of input entities for ParaGen. We count frequency using the WebText corpus Radford et al. (2019). **(4)** Fig. 2 shows increasing percentages of No Evidence as the generation proceeds to later sentences, which may be caused by irrelevant or spurious information introduced into outputs by the error accumulation inherent in auto-regressive generation Zhang et al. (2023). Appendix C.6 shows the influence of the context on model generation and fact verification.

The above human evaluation confirms the LLMs' serious hallucination issue, emphasizing the urgent need for effective fact verifiers to measure and incentivize progress in LLMs' fractuality. This motivates us to explore LLMs' potential as fact verifiers.

## 4 Repurposing LLMs as Fact Verifiers

Our second inquiry lies around the question

_Can LLMs be repurposed as effective fact verifiers?_

We define fact verification as follows: given a statement \(s\) and its leading context \(c\), the verifier should give a probability \(p\) of \(s\) being factual6. The context may be absent, and the statement is a sentence or paragraph. Additionally, we retrieve an evidence set, denoted as \(E=\left\{e_{1},e_{2},\cdots,e_{M}\right\}\), from external corpora Lewis et al. (2020) with the concatenation of \(c\) and \(s\) as the query. Each piece of evidence is a passage. Next, we first describe the evaluation sets (SS4.1), our verification method (SS4.2), compared verifiers (SS4.3), and then present the experiment results (SS4.4).

Footnote 6: We formulate the fact verification task as regression instead of traditional classification because continuous scores are more informative to reflect the nuance of inputs and may give generators fine-grained feedback in future work.

### Evaluation Sets

We design three evaluation sets across multiple domains and sources:

* [noitemsep]
* **Model-Generated Statements (mgs):** It includes SentCom and ParaGen statements generated by four LLMs in SS3.
* **Wiki-Domain Statements (wks):** It aggregates three fact verification datasets in the Wikipedia domain including FEVER Petroni et al. (2021), BoolQ-FV Thorne et al. (2021) and FM2 Eisenschlos et al. (2021).
* **Domain-Specific Statements (dss):** It aggregates four domain-specific datasets including PubMedQA Jin et al. (2019), XsumFaith Maynez et al. (2020), SummEval Fabbri et al. (2021) and SciFact Wadden et al. (2020). Statements supported/refuted by golden evidence are labeled factual/unfactual, and we remove statements without golden evidence since the factuality is unknowable.

These evaluation sets span various domains, origination, and lengths. Tab. 3 summarizes the statistics of wks and dss, and Appendix B includes more details.

### Verification Method

We transform each input \(x=(E,c,s)\) into a prompt, as shown in Tab. 4. The LLM is expected to generate a judgment \(W=(w_{1},w_{2},\cdots,w_{T})\) about whether \(s\) is factual. We compute the factuality score \(p\) by normalizing the LLM's output probabilities of all valid answers Ke et al. (2022):

\[p=\frac{\sum_{w\in L_{A}}p_{\texttt{LLM}}\left(w\mid x,W_{<t}\right)}{\sum_{w \in L_{A}\cup L_{B}}p_{\texttt{LLM}}\left(w\mid x,W_{<t}\right)},\]

where \(p_{\texttt{LLM}}\) is the LLM's probability distribution over the vocabulary, \(L_{A}\)/\(L_{B}\) is the set of plausible answer words to indicate factual/non-factual, and \(t\) is the maximum time step that makes each token in \(W_{<t}\) not included in \(L_{A}\) or \(L_{B}\). We define \(L_{A}=\left\{\text{``A'', ``a'', ``Yes", ``yes", ``YES''}\right\}\) and \(L_{B}=\left\{\text{``B'', ``b'', ``No", ``no", `NO"}\right\}\). If \(W\) does

Figure 2: Distribution of different labels across sentence positions in ParaGen (Para).

 

not include any valid answer words, we set \(p\) to 0.5. In Appendix C.4, we compare other verification methods such as Chain-of-Thought prompting Wei et al. (2022) and Likert-scale rating.

### Compared Verifiers

We test the following instruction-tuned LLMs for fact verification using the method in SS4.2:

1. [label=**(0)]
2. FLAN-T5118:** It is instruction-tuned from T5118 Raffel et al. (2020) on 1.8K+ tasks.
3. GPT3.5: We approximate \(p_{\text{LLM}}\) by normalizing the probabilities calculated from top five logits returned by the API.
4. ChatGPT: We hard code the score as 1/0 if it returns "A"/"B" using the prompt in Tab. 4 since its output probabilities are unavailable.

For all models, the maximum length is set to 4,000 tokens tokenized by the GPT-series BPE tokenizer. Appendix C.3 presents the results of more LLMs.

We compare the LLMs to baselines widely used to measure hallucination:

1. [label=**(0)]
2. knowledgeF1 (KF1): It measures the average unigram overlap between the statement and each piece of evidence Shuster et al. (2021).
3. NLI: It is an unsupervised verifier computed as the entailment probability between the evidence and statement. We use the public T5118-based NLI model fine-tuned on a mixture of multiple NLI datasets Honovich et al. (2022).
4. FiD: It is a supervised verifier based on a finetuned binary factuality classifier Izacard and Grave (2021); Liu et al. (2022) with the statement and evidence as input. We build FiD based on FLAN-T5780M and fine-tune it on three wks training sets, respectively, to obtain corresponding verifiers.
5. FactScore (FAS): It first automatically splits a statement into multiple atomic facts and then computes the factual precision as the overall score, i.e., the percentage of atomic facts supported by evidence Min et al. (2023).

Regarding the retrieval components, we employ the Wikipedia dump from Izacard et al. (2022) as the external corpus for mgs and wks, with each piece of evidence being a passage of 100 words. Ten pieces of evidence are retrieved for each test sample using Contriever Izacard and Grave (2021). Appendix C.2 presents details about retrievers. For dss, we use the golden evidence provided by the original papers.

### Results on wks&dss

Taking human judgments of factual/unfactual statements as 1/0, we use the following metrics to evaluate fact verifiers (Appendix C.1 shows more details):

1. [label=**(0)]
2. Expected Calibration Error (ECE) Guo et al. (2017): It estimates to what extent the predicted score can indicate accuracy. Lower ECE mean better calibration.
3. Accuracy (ACC): It is the fraction of examples that are correctly predicted.
4. Area Under the ROC Curve (AUR): It measures the ability to discriminate factual statements from others.
5. Pearson's Correlation (r): It measures the

\begin{table}
\begin{tabular}{l l} \hline \hline (Task) & Answer the following question: \\ Input & Facts \\  & 1. \{e_{1}\} \\  & 2. \{e_{2}\} \\  & ... \\  & M. \{e_{M}\} \\  & Context: \{c\} \\  & Statement following the context: \{s\} \\ Question & Based on the given facts, is the statement correct? (A) Yes. (B) No. Please answer A or B: \\ \hline \hline \end{tabular}
\end{table}
Table 4: An example prompt used to adapt LLMs for fact verification. The prompt may be changed under different settings. For example, when \(c\) is empty, we delete “Context: \(\{c\}\)” and “following the context”.

 

correlation between prediction and human judgments.

As shown in Tab. 5, on wks, **when using retrieved evidence, (1)** KF1 hardly captures any factuality features; **(2)** NLI\({}_{\text{11B}}\) struggles to generalize to fact verification and significantly underperforms FLAN-T\({}_{\text{511B}}\); **(3)** Instruction-tuned LLMs achieve strong performance and are comparable to or better than supervised FiD models; And **(4)** FLAN-T\({}_{\text{511B}}\) outperforms GPT3.5 in both calibration and discrimination ability. **When not using any evidence**, ChatGPT performs the best, possibly attributed to its superiority in memorizing and utilizing knowledge. The overall inferior performance of not using evidence reveals the importance of retrieval components. Furthermore, the results on dss in Tab. 6 show that FLAN-T\({}_{\text{511B}}\) surpasses ChatGPT in terms of most metrics, which also indicates the potential of relatively small-scaled models for hallucination evaluation across various domains. Appendix C.3 shows a similar conclusion on a contemporary dataset FactPrompts Chern et al. (2023).

Results on mgsBesides metrics used on wks, we also report **Precision (P)**, **Recall (R)**, and **Area Under the Precision-Recall Curve (AUP)** on the factual category for SentCom and ParaGen (Sent), considering the label imbalance. We treat human judgments of factual sentences as 1 and others as 7, and calculate the human judgment of a paragraph as the faction of factual sentences. We use FiD trained on FEVER for the experiments.

Tab. 7 shows: **(1)** Despite FiD's best performance on FEVER, it underperforms FLAN-T\({}_{\text{511B}}\) on SentCom, indicating limited generalization of supervised models. **(2)** All verifiers exhibit inferior performance on ParaGen (Sent) compared to SentCom, potentially due to less prevalent entities and more contextual dependencies (e.g., coreference). **(3)** The high recall scores of GPT variants in contrast to FLAN-T\({}_{\text{511B}}\) show that they prefer to predict factual. This may account for the greater disparity in precision and AUP between GPT variants and FLAN-T\({}_{\text{511B}}\) on ParaGen that contains more non-factual statements than SentCom. **(4)** On ParaGen (Para), averaging sentence-level scores usually yields better correlations than direct verification. This is attributed to a stronger ability

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**PubMedQA**} & \multicolumn{4}{c}{**XsumFaith**} & \multicolumn{4}{c}{**SummaEval**} & \multicolumn{4}{c}{**Selfact**} \\ \cline{2-19}
**Verifiers** & **ECE** & **ACC** & **AUR** & \(r\) & **ECE** & **ACC** & **AUR** & \(r\) & **ECE** & **ACC** & **AUR** & \(r\) & **ECE** & **ACC** & **AUR** & \(r\) \\ \hline
**FLAN-T\({}_{\text{511B}}\)** & **12.6** & 78.4 & **84.6** & **60.0** & **20.1** & **78.2** & **70.8** & 20.0 & **3.5** & 92.4 & **93.2** & **62.0** & **11.1** & 83.2 & **95.3** & **77.9** \\
**ChatGPT** & 20.4 & **79.6** & 76.7 & 55.7 & 23.6 & 76.4 & 68.1 & **21.4** & 7.5 & **92.5** & 71.0 & 51.4 & 16.5 & **88.0** & 86.4 & 69.5 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Results on dss. Both models are under the zero-shot setting and provided with the golden evidence. **Takeaway:** FLAN-T\({}_{\text{511B}}\) surpasses ChatGPT in terms of most metrics on specific domains.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Verifiers**} & \multicolumn{4}{c}{**FEVER**} & \multicolumn{4}{c}{**BoolQ-FV**} & \multicolumn{4}{c}{**FM2**} \\ \cline{2-19}  & **ECE** & **ACC** & **AUR** & \(r\) & **ECE** & **ACC** & **AUR** & \(r\) & **ECE** & **ACC** & **AUR** & \(r\) \\ \hline
**Constant** & _48.3_ & _51.7_ & _50.0_ & _N/A_ & _29.4_ & _70.6_ & _50.0_ & _N/A_ & _50.6_ & _49.4_ & _50.0_ & _N/A_ \\ \hline \multicolumn{19}{c}{**Retrieving Evidence from External Corpora**} \\ \hline
**KF1** & 51.3 & 48.3 & 53.9 & 9.2 & 70.3 & 29.4 & 44.4 & -7.7 & 48.8 & 50.6 & 49.8 & -0.6 \\
**NLI\({}_{\text{11B}}\)** & 18.3 & 81.7 & 83.8 & 67.4 & 45.1 & 54.8 & 66.8 & 33.2 & 34.5 & 65.5 & 68.0 & 38.7 \\
**FiD\({}_{\text{780M}}\)** & **2.9** & **94.6** & **98.2** & **90.5** & 13.8 & 82.7 & 88.5 & 62.0 & 15.5 & 77.0 & 85.6 & 59.1 \\
**FLAN-T\({}_{\text{511B}}\)** & 3.1 & 93.8 & 98.2 & 90.2 & **10.2** & 85.5 & **94.7** & **75.3** & **8.2** & **82.0** & **89.5** & **68.8** \\
**GPTS3.5** & 7.6 & 91.7 & 96.6 & 84.7 & 17.9 & 81.7 & 87.4 & 61.1 & 21.0 & 77.5 & 82.8 & 55.9 \\
**ChatGPT** & 8.2 & 92.8 & 92.6 & 84.3 & 13.1 & **87.3** & 88.0 & 70.1 & 21.4 & 79.1 & 78.6 & 56.7 \\ \hline \multicolumn{19}{c}{**Not Using Any Evidence**} \\ \hline
**FiD\({}_{\text{780M}}\)** & **4.2** & 77.3 & 85.7 & 62.0 & **17.4** & 64.6 & 60.8 & 17.2 & 21.6 & 59.4 & 65.1 & 26.4 \\
**FLAN-T\({}_{\text{511B}}\)** & 11.1 & 74.4 & **87.0** & 62.6 & 28.6 & 56.0 & 65.6 & 23.6 & **15.7** & 59.5 & 66.3 & 28.2 \\
**GPTS3.5** & 25.6 & 73.8 & 78.3 & 52.7 & 32.8 & 65.6 & 67.9 & 23.2 & 41.4 & 57.3 & 61.5 & 19.5 \\
**ChatGPT** & 18.1 & **82.0** & 81.5 & **63.9** & 27.1 & **73.1** & **69.7** & **37.4** & 33.5 & **66.7** & **66.5** & **34.1** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results on wks. “Constant” always predicts a factuality score of 1. All instruction-tuned LLMs are under the zero-shot setting. We highlight the best result in **bold** and underline the second best. Note that the results of ChatGPT except for ACC may be underestimated because they are calculated based on generated answers instead of probabilities (details in Appendix C.3). **Takeaway:** In the Wikipedia domain, ChatGPT performs the best when not using any evidence, while FALN-T\({}_{\text{511B}}\) excels ChatGPT with retrieved evidence.

 

to capture subtle details within a paragraph, facilitated by independent retrieval and verification for every sentence. **(5)** FAS shows comparable or inferior performance in contrast to FLAN-T\({}_{5118}\), suggesting that noise introduced during the generation of atomic facts may impact the final performance.

Influence of Model SizesTab. 8 demonstrates a positive correlation between model sizes and the performance of FLAN-T5. Notably, on more challenging datasets, FM2 and ParaGen, there exists a relatively larger performance gap between FLAN-T\({}_{750\text{M}}\) and FLAN-T\({}_{53\text{N/11B}}\).

## 5 Analysis on LLM-based Verifiers

This section further analyzes the influence of given evidence on LLM-based verifiers (SS5.1), as well as their robustness (SS5.2) and generalization ability (SS5.3). Appendix C.8 also investigates how LLMs' memorization of inputs may influence their judgments. And Appendix C.9 further shows several representative cases that LLMs fail to judge, to provide more insights.

### Influence of Given Evidence

The aforementioned experiments reveal the importance of retrieving external knowledge for fact verification. We further assess how different types of evidence influence performance. Tab. 9 shows: **(1)** Golden evidence is better than retrieved one, both outperforming null and random evidence. **(2)** ChatGPT is more susceptible to irrelevant information in given evidence than FLAN-T\({}_{511\text{B}}\). For example, ChatGPT is superior with solely golden evidence but underperforms with mixed golden and random evidence. This can potentially elucidate ChatGPT's suboptimal performance when using retrieved evidence that may include irrelevant information. **(3)** ChatGPT performs much better than FLAN-T\({}_{511\text{B}}\) if given relevant but fake (Adv) or contradictory evidence (Adv+Gold), which may be prevalent cases when retrieving evidence from the Internet Shu et al. (2017). This reflects ChatGPT's more limited reliance on given evidence and stronger conviction in its internal knowledge. Nevertheless, the overall drop in accuracy compared to using golden evidence confirms the susceptibility of LLMs to false information Bian et al. (2023).

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Verifiers** & **ECE** & **P** & **R** & **AUR** & **AUP** & \(r\) \\ \hline
**Constant** & _33.2_ & _66.8_ & _100.0_ & _50.0_ & _66.8_ & _N/A_ \\
**FiD** & 11.0 & 79.0 & 96.1 & 88.6 & 93.7 & 64.2 \\
**FLAN-T\({}_{5118}\)** & **10.5** & **96.5** & 84.5 & **93.1** & **96.9** & **74.9** \\
**GPT3.5** & 17.7 & 85.2 & 89.2 & 87.8 & 90.8 & 60.8 \\
**ChatGPT** & 15.0 & 91.7 & 85.3 & 84.8 & 88.0 & 67.6 \\ \hline
**FAS\({}_{\texttt{FLAN-T5}}\)** & 11.7 & 91.7 & 77.5 & 89.8 & 93.7 & 69.3 \\
**FAS\({}_{\texttt{CHAT}}\)** & 18.5 & 80.4 & **92.2** & 78.8 & 83.6 & 55.0 \\ \hline  & \multicolumn{6}{c}{Sent} & \multicolumn{6}{c}{Para} \\ \cline{2-7}
**Verifiers** & **ECE** & **P** & **R** & **AUR** & **AUP** & \(r\) \\ \hline
**Constant** & 27.7 & _27.3_ & _100.0_ & _50.0_ & _27.3_ & _NA_ & _NA_ \\
**FID** & 38.9 & 35.9 & 92.9 & 85.9 & 76.7 & 46.5 & 39.1 \\
**FLAN-T\({}_{5118}\)** & **9.2** & 76.0 & 71.6 & **88.0** & **80.9** & **66.8** & 45.17 / 79.4 \\
**GPTS.5** & 41.4 & 38.5 & **95.5** & 87.7 & 70.6 & 39.5 & 48.67 / 59.7 \\
**ChaGTPT** & 25.6 & 51.7 & 90.3 & 79.7 & 49.6 & 52.6 & 52.6 / 68.1 \\
**FASP\({}_{\texttt{TLAN-T5}}\)** & **10.6** & **77.4** & 57.4 & 84.0 & 72.6 & 61.9 & **79.7** \\
**FAS\({}_{\texttt{CAVARY}}\)** & 45.5 & 35.7 & 85.2 & 70.7 & 42.4 & 29.4 & 45.3 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Results on mgs (**Top:** SentCom; **Bottom:** ParaGen) with retrieved evidence. Two scores **A/B** of LLMs on ParaGen (**Para**) mean, **A**: directly evaluating a whole paragraph, **B**: averaging corresponding sentence-level scores. FAS\({}_{\texttt{FLAN-T5/chart}}\) refers to FAS using FLAN-T\({}_{511\text{B}}\)/ChatGPT to judge the factuality of atomic facts. **Takeaway:** FLAN-T\({}_{511\text{B}}\) exhibits the overall best performance; Taking sentences as base units for verification, as opposed to paragraphs or potentially noisy atomic facts, yields better results.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Verifiers** & **FEVER** & **FM2** & **SentCom** & \multicolumn{2}{c}{**paraGen**} \\ \cline{2-5}  & **STnt** & **Para** \\ \hline
**FLAN-T\({}_{578\text{M}}\)** & 85.8 & 61.6 & 75.4 & 55.6 & 71.8 \\
**FLAN-T\({}_{53\text{N}}\)** & 88.6 & 66.6 & **75.8** & 65.8 & 78.6 \\
**FLAN-T\({}_{511\text{B}}\)** & **90.2** & **68.8** & 74.9 & **66.8** & **79.4** \\ \hline \hline \end{tabular}
\end{table}
Table 8: Pearson’s correlation scores of FLAN-T5 with different model sizes. We calculate factuality scores on ParaGen (**Para**) by averaging sentence-level scores here and below.

 



### Robustness

LLMs are widely observed to be sensitive to synonymous prompts Jiang et al. (2020). We design three additional prompts by changing the question in Tab.4 (details in AppendixC.5). Tab.10 shows the mean and standard deviation on mgs across four prompts. We find (1) FLAN-T5\({}_{\text{11h}}\) and Chat GPT are better under the zero-shot setting, while GPT3.5 is better under the few-shot setting. (2) FLAN-T5\({}_{\text{11h}}\) is more robust to different prompts than GPT3.5 and ChatGPT with lower variance.

### Generalization Ability

It is crucial for fact verifiers to deal with various inputs Garbacea et al. (2019). We focus on LLMs' generalization to statements generated by different models, depending on the context or not, or involving different types of named entities.

**Smaller models struggle to verify outputs from larger models.** Fig.3 shows FLAN-T5\({}_{\text{11h}}\) performs worse at verifying outputs from relatively large models (LLama6\({}_{\text{65h}}\) and GPT3.5). ChatGPT is more stable across different generation models.

**Verifying context-dependent statements is more challenging and can be enhanced through contextualization.** For each sentence in ParaGen (Sent), we manually annotate whether it refers to any entities in the context. Tab.11 shows: **(1)** Performance on context-dependent sentences will notably drop if the context is unseen, indicating the verifiers are indeed utilizing the given context to judge the factuality. De-contextualization by first performing coreference resolution (CR) can bring substantial improvement. **(2)** Performance on context-independent sentences becomes even better without context, suggesting the context may introduce noise. CR hardly further improves the performance. **(3)** De-contextualization at the sentence level also benefits paragraph-level verification. AppendixC.6 shows more experiment details.

**Numerical-involved statements are more difficult to verify.** The fractality of a statement mainly manifests in the entities involved and inter-entity relations Nan et al. (2021). We are curious whether the verifiers perform similarly at verifying state

Figure 4: Precision and recall on the unffactual category varying with entity types. “Non-NE” means non-named-entity common words (e.g., “computer”).

\begin{table}
\begin{tabular}{l c r r r r r} \hline \multirow{2}{*}{**Verifiers**} & \multicolumn{2}{c}{**Setting**} & \multicolumn{2}{c}{**SentCom**} & \multicolumn{2}{c}{**ParaGen**} \\ \cline{3-7}  & \multicolumn{1}{c}{**w/ Dependencies**} & \multicolumn{3}{c}{**sw/o Dependencies**} & \multicolumn{1}{r}{**sw/o Dependencies**} & \multicolumn{1}{r}{**para**} \\ \cline{3-8}  & **AUR** & **Avg** & **v** & **AUC** & **AU** & \(r\) & \(r\) \\ \hline \hline
**FLAN-T5\({}_{\text{11h}}\) (ZS)** & 87.4 & 72.6 & 59.9 & 88.6 & 84.7 & 69.1 & 79.4 \\
**w/o Context** & 81.0 & 58.5 & 49.5 & 90.2 & **85.6** & 72.3 & 82.9 \\
**w/ CR** & **88.9** & **79.0** & **68.4** & **90.9** & 85.2 & **72.8** & **86.1** \\ \hline
**ChatGPT** (ZS)** & 75.1 & 38.1 & 41.1 & 84.1 & 62.9 & 63.4 & 68.1 \\
**w/ Contex** & 72.2 & 35.0 & 36.5 & 84.8 & 65.9 & 66.0 & 74.4 \\
**w/ CR** & **79.6** & **45.7** & **52.0** & **85.1** & **66.4** & **67.1** & **78.1** \\ \hline \end{tabular}
\end{table}
Table 11: Generalization to sentences with or without dependencies on the context. **w/o Context:** not providing the context during verification. **w/ CR:** first performing coreference resolution using ChatGPT to eliminate potential dependencies in a sentence and then verifying it without context.

Figure 3: Pearson’s correlations when verifying outputs from different generation models. **Top**: SentCom; **Bottom**: ParaGen (Sent). We calculate the standard deviation (black bars) across four instructions in §5.2.



ments concerning different types of entities. We experiment on the FaVIQ dataset (Park et al., 2022), which consists of 188k sentences converted from information-seeking QA pairs. The original answer to the question is a word or phrase, and the resulting sentence is factual only if the answer is correct, enabling us to know which part of a sentence is unfactual. We categorize all sentences by the entity types8 of corresponding answers and randomly sample 100 factual and 100 unfactual sentences in each category to test the verifiers.

Footnote 8: We use spaCy Honnibal and Montani, 2017) for named entity recognition.

Fig. 4 shows: **(1) LLMs perform better at "Person," Work of Art," and "Building;" **(2)** The performance is poor at numeral-related types "Cardinal" and "Ordinal." Particularly, LLMs tend to misidentify factual sentences with cardinal numerals as unfactual. The reasons lie in the difficulty of reasoning over inter-numeral logical relations, and retrieving numerals-related evidence. For example, the evidence recall@10 score of the "Cardinal" type is \(\sim\)26% lower than that of "Person" (0.43 vs. 0.58). More details are in Appendix C.7.

## 6 Conclusion

We present a comprehensive study around two research questions: the extent to which current LLMs hallucinate; and their potential as effective fact verifiers. Firstly, we quantitatively affirm the significant hallucination issue of current LLMs through a well-designed human evaluation. This highlights the urgent need for powerful fact verifiers to measure and incentivize progress in LLMs' factuality. To this end, we repurpose the LLMs into fact verifiers, and examine their ability to judge the factuality of model-generated and human-crafted statements. Further analysis shows their heavy reliance on high-quality evidence and discusses their robustness and generalization ability. The evaluation suite and implemented verifiers in this paper can facilitate further research on fact verifiers and trustworthy generation models.

## Acknowledgements

This work was supported by the National Key Research and Development Program of China (No. 2021ZD0113304), the National Science Foundation for Distinguished Young Scholars (with No. 62125604), and the NSFC projects (Key project with No. 61936010).

### Limitations

We summarize our limitations as follows:

* Regarding the Quantification of Hallucinations
* We only focus on quantifying hallucinations for retrieval-free open-ended generation. Although Shuster et al. (2021) showed that retrieval could significantly reduce hallucinations, LLMs are observed to overly depend on retrieved texts for generation (e.g., directly copying from those texts) regardless of fluency and relatedness with input instructions Liu et al. (2023), which is out of our scope.
* The model outputs are generated under the few-shot setting to ensure that they are as objective and informative as possible, although most generation models are used under the zero-shot setting in reality.

### Regarding Evaluation Sets

* Examples in mgs are limited to the Wikipedia domain due to the difficulty of manually annotating statements involving much professional knowledge in other domains. To mitigate this issue, we include model-generated statements from the news domain (e.g., XsumFaith) in dss and conduct experiments on ChatGPT-generated statements in the question-answering domain (Appendix C.3). we expect to collect examples in more professional domains (e.g., finance) in future work.
* Statements in all three evaluation sets rarely require multi-hop reasoning for judging, considering that outputs of current LLMs are organized as chains of single-hop ones.
* Regarding Fact Verifiers
* Although we endeavor to evaluate LLMs' ability to deal with contradictory or fake evidence in SS5.1, the auto-constructed evidence does not occur naturally and may exhibit many artifacts. Contradictory or fake evidence hardly occurs when retrieved from Wikipedia, but it will be much more common if retrieving evidence from the Internet.
* We do not perfectly answer why FLAN-T5 is less susceptible to irrelevant information in retrieved evidence than ChatGPT due to the lack of knowledge about the implementation details of OpenAI's GPT series. We conjecture that the reasons possibly lie in that the 

instruction-tuning data of FLAN-T5 models have included lots of noisy inputs, leading to better robustness to irrelevant information. In contrast, ChatGPT's better knowledge memorization ability may make it less influenced by the relevant but fake information in the evidence. The causal factors underlying these results remain to be investigated in the future work.
* We only equip LLMs with single-step interaction with external corpora through the retriever. In future work, we expect to build autonomous reasoning agents that can verify any texts through multiple-step interaction with diverse knowledge environments Guan et al. (2024).

## Ethics Statements

Our experiment results are based on existing public resources (datasets, model checkpoints, and codes). We use widely adopted settings for model generation and evaluation, making our analysis easily replicated. We resorted to Amazon Mechanical Turk (AMT) for human evaluations of model-generated statements. We did not ask about personal privacy or collect any personal information of annotators in the annotation process. We pay each worker $2.5/$7.5 for each HIT task of SentCom/ParaGen, respectively, leading to an hourly wage of \(\sim\)$30, which is much higher than the minimum wage of $7.5/h in the U.S. We decided the payment according to the average length of data examples. We admit that there may still be unpredictable bias in mgs even though we have carefully reviewed all annotated results from an ethical perspective.

## References

* Aksitov et al. (2023) Renat Aksitov, Chung-Ching Chang, David Reitter, Siamak Shakeri, and Yunhsuan Sung. 2023. Characterizing attribution and fluency tradeoffs for retrieval-augmented large language models. _arXiv preprint arXiv:2302.05578_.
* Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Wilky Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. _arXiv preprint arXiv:2302.04023_.
* Bian et al. (2023) Ning Bian, Peilin Liu, Xianpei Han, Hongyu Lin, Yaojie Lu, Ben He, and Le Sun. 2023. A drop of ink may make a million think: The spread of false information in large language models. _arXiv preprint arXiv:2305.04812_.
* Bowman et al. (2015) Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 632-642.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Chern et al. (2023) I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. 2023. Factool: Fractuality detection in generative ai-a tool augmented framework for multi-task and multi-domain scenarios. _arXiv preprint arXiv:2307.13528_.
* Chowdhery et al. (2022) Akanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. _arXiv preprint arXiv:2204.02311_.
* Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barrett Zoph, Yi Tay, William Fedus, Eric Li, Xuexih Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. _arXiv preprint arXiv:2210.11416_.
* Colombo et al. (2022) Pierre Jean A Colombo, Choe Clavel, and Pablo Piantanida. 2022. Infolm: A new metric to evaluate summarization & data2text generation. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 36, pages 10554-10562.
* Dingra et al. (2019) Bhuwan Dingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4884-4895.
* Diggelmann et al. (2020) Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. Climate-fever: A dataset for verification of real-world climate claims. _arXiv preprint arXiv:2012.00614_.
* Eisenschlos et al. (2021) Julian Eisenschlos, Buwan Dhingra, Jannis Bullian, Benjamin Borscinger, and Jordan Boyd-Graber. 2021. Fool me twice: Entailment from wikipedia gamification. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 352-365 

Alexander R Fabbri, Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. _Transactions of the Association for Computational Linguistics_, 9:391-409.
* Fleiss and Joseph (1971) Fleiss and L. Joseph. 1971. Measuring nominal scale agreement among many raters. _Psychological Bulletin_, 76(5):378-382.
* Gao et al. (2023) Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human-like summarization evaluation with chatgpt. _arXiv preprint arXiv:2304.02554_.
* Garbacea et al. (2019) Cristina Garbacea, Samuel Carton, Shiyan Yan, and Qiaozhu Mei. 2019. Judge the judges: A large-scale evaluation study of neural language models for online review generation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3959-3972.
* Guan et al. (2024) Jian Guan, Wei Wu, Zujie Wen, Peng Xu, Hongning Wang, and Minlie Huang. 2024. Amor: A recipe for building adaptable modular knowledge agents through process feedback. _arXiv preprint arXiv:2402.01469_.
* Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In _International conference on machine learning_, pages 1321-1330. PMLR.
* Guo et al. (2022) Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. A survey on automated fact-checking. _Transactions of the Association for Computational Linguistics_, 10:178-206.
* Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
* Honovich et al. (2022) Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kulkiansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. True: Re-evaluating factual consistency evaluation. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3905-3920.
* Izacard et al. (2022) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning.
* Izacard and Grove (2021) Gautier Izacard and Edouard Grave. 2021a. Leveraging passage retrieval with generative models for open domain question answering. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 874-880.
* Izacard and Gravev (2021) Gautier Izacard and Edouard Grave. 2021b. Leveraging passage retrieval with generative models for open domain question answering. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 874-880, Online. Association for Computational Linguistics.
* Jiang et al. (2020) Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? _Transactions of the Association for Computational Linguistics_, 8:423-438.
* Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhenping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2567-2577.
* Ke et al. (2022) Pei Ke, Hao Zhou, Yankai Lin, Peng Li, Jie Zhou, Xiaoyan Zhu, and Minlie Huang. 2022. Ctrleval: An unsupervised reference-free metric for evaluating controlled text generation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2306-2319.
* Khot et al. (2018) Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 32.
* Kotonya and Toni (2020) Neema Kotonya and Francesca Toni. 2020. Explainable automated fact-checking for public health claims. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7740-7754, Online. Association for Computational Linguistics.
* Krycsinski et al. (2020) Wojciech Krycsinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9332-9346.
* Laban et al. (2022) Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2022. Summac: Re-visiting nl-based models for inconsistency detection in summarization. _Transactions of the Association for Computational Linguistics_, 10:163-177.
* Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. _Advances in Neural Information Processing Systems_, 33:9459-9474.
* Li et al. (2023)Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. Halueval: A large-scale hallucination evaluation benchmark for large language models. _ArXiv e-prints_, pages arXiv-2305.

 

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3214-3252.
* Liu et al. (2023a) Nelson F Liu, Tianyi Zhang, and Percy Liang. 2023a. Evaluating verifiability in generative search engines. _arXiv preprint arXiv:2304.09848_.
* Liu et al. (2022b) Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and William B Dolan. 2022b. A token-level reference-free hallucination detection benchmark for free-form text generation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 6723-6737.
* Liu et al. (2023b) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b. Gpteval: Nlg evaluation using gpt-4 with better human alignment. _arXiv preprint arXiv:2303.16634_.
* Luo et al. (2023) Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for abstractive text summarization. _arXiv preprint arXiv:2303.15621_.
* Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgrt: Zero-resource black-box hallucation detection for generative large language models. _arXiv preprint arXiv:2303.08896_.
* Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1906-1919.
* Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. _arXiv preprint arXiv:2305.14251_.
* Nan et al. (2021) Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen Mckeown, and Bing Xiang. 2021. Entity-level factual consistency of abstractive text summarization. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 2727-2733.
* OpenAI (2023) OpenAI. 2023. Gpt-4 technical report.
* Park et al. (2022) Jungsoo Park, Sewon Min, Jaewoo Kang, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022. Faviq: Fact verification from information-seeking questions. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 5154-5166.
* Perez-Rosas et al. (2018) Veronica Perez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. 2018. Automatic detection of fake news. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 3391-3401, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Petroni et al. (2021) Fabio Petroni, Aleksandra Pitkus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. 2021. Kilt: a benchmark for knowledge intensive language tasks. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2523-2544.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _The Journal of Machine Learning Research_, 21(1):5485-5551.
* Sarlin et al. (2020) Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. 2020. Superglue: Learning feature matching with graph neural networks. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 4938-4947.
* Sathe et al. (2020) Aalok Sathe, Salar Ahter, Tuan Manh Le, Nathan Perry, and Joonsuk Park. 2020. Automated fact-checking of claims from wikipedia. In _Proceedings of the 12th Language Resources and Evaluation Conference_, pages 6874-6882.
* Schuster et al. (2021) Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin c! robust fact verification with contrastive evidence. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 624-643.
* Schuster et al. (2019) Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3419-3425, Hong Kong, China. Association for Computational Linguistics.
* Shu et al. (2017) Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake news detection on social media: A data mining perspective. _ACM SIGKDD explorations newsletter_, 19(1):22-36.
* Shuster et al. (2021)urt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 3784-3803 

* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsuonri B. Hashimoto. 2023. Stanford alpaca: An instruction-following llamale model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Thorne et al. (2021) James Thorne, Max Glockner, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych. 2021. Evidence-based verification for real world information needs. _arXiv preprint arXiv:2104.00640_.
* Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gaurier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LIMA: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Vlachos and Riedel (2014) Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In _Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science_, pages 18-22, Baltimore, MD, USA. Association for Computational Linguistics.
* Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannane Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7534-7550, Online. Association for Computational Linguistics.
* Wang et al. (2020a) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020a. Asking and answering questions to evaluate the factual consistency of summaries. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5008-5020, Online. Association for Computational Linguistics.
* Wang et al. (2023) Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt a good nlg evaluator? a preliminary study. _arXiv preprint arXiv:2303.04048_.
* Wang et al. (2020b) Zhenyi Wang, Xiaoyang Wang, Bang An, Dong Yu, and Changyou Chen. 2020b. Towards faithful neural table-to-text generation with content-matching constraints. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1072-1086.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thoughrt prompting elicits reasoning in large language models. _Advances in Neural Information Processing Systems_, 35:24824-24837.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
* Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. _Advances in Neural Information Processing Systems_, 34:27263-27277.
* Zhang et al. (2023) Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023. How language model hallucinations can snowball. _arXiv preprint arXiv:2305.13534_.
* Zhang et al. (2019) Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1298-1308.
* Zhong et al. (2023) Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgtp understand too? a comparative study on chatgpt and fine-tuned bert. _arXiv preprint arXiv:2302.10198_.

## Appendix A Quantifying LLMs' Hallucination

### Generating Statements

The statements in mgs are generated under the few-shot setting with manually selected in-context demonstrations. For SentCom, the demonstrations are selected directly from the FEVER training set. And for ParaGen, the demonstrations are selected by sampling an entity from all Wikipedia titles as the input (not overlapping the inputs for generation) and taking the first five sentences of the corresponding introduction section as the golden output. Tab. 14 shows the demonstrations. We ensure that all words in each input entity of demonstrations and generation outputs for ParaGen appear more than 100 times in the WebText corpus to avoid rare words.

### Human Evaluation

Fig. 1 in the main paper shows the annotation interface. To control the annotation quality, we discard those submissions where workers (1) do not provide evidence for sentences annotated as factual or unfactual; (2) do not annotate sentences in golden statements as factual; (3) annotate more than 90% sentences as factual since manual inspection finds that the faction of factual sentences is much less 

than 90%. When there are sentences assigned with different labels by three workers in a statement, we collect two additional annotations for it and retain three annotations with the highest agreement for each sentence. We obtain the final human judgments by repeating the above steps until each sentence has three valid annotations.

We ask workers to provide evidence mainly in order to force workers to search for related information to help them judge the factuality. Tab. 12 shows that LLMs get better results using golden evidence than retrieved evidence on mgs, indicating the high quality of collected evidence.

### Human Judgements for No-Evidence Statements

There are three ground-truth labels for each statement in mgs, i.e., Factual, Unfactual, and No Evidence. To convert human judgments to numerical scores, we regard Factual as 1 and both Unfactual and No Evidence as 0, following prior work that converted the 3-way classification to the 2-way classification Sarlin et al. (2020); Sathe et al. (2020). We would still like to investigate whether LLMs perform differently to verify unfactual statements and no-evidence ones. As shown in Tab. 13, the conclusion is similar under both settings: FLAN-T5\({}_{\text{11h}}\) outperforms ChatGPT under both settings. This indicates that it is reasonable to merge unfactual and no-evidence statements as one category.

## Appendix B Collecting wks

wks aggregates seven existing fact verification datasets, including:

* **FEVER:** It contains crowd-sourced statements by altering a word or negating sentences from Wikipedia Thorne et al. (2018), thereby leading to strong artifacts Schuster et al. (2019). We use the KILT version of FEVER Petroni et al. (2021). Since the official test set is hidden, we sample 1K examples from the validation set for testing and use the rest for validation.
* **BoolQ-FV Thorne et al. (2021): It consists of more realistic claims than FEVER, which are derived from real-world information needs by rewriting users' search-engine queries and verifying them against evidence from Wikipedia.
* **FMZ**Eisenschlos et al. (2021): It is collected by gamification. Players write challenging statements supported or refuted by evidence from Wikipedia and spot refuted claims written by others.
* **PubMedQA**Jin et al. (2019): It is initially a question-answering dataset specifically designed for the biomedical domain based on the PubMed database9, which is a comprehensive collection of biomedical literature. Each PubMedQA example consists of a yes-no question and the abstract of the corresponding background paper. We prompt ChatGPT to convert the question into a declarative sentence as the statement to be judged with the abstract as the golden evidence. Footnote 9: [https://pubmed.ncbi.nlm.nih.gov/](https://pubmed.ncbi.nlm.nih.gov/)
* **XsumFaith**Maynez et al. (2020) and **SummEval**Fabbri et al. (2021): Each example consists of a summary and the corresponding source document. The summaries are generated by various models and paired with human-annotated binary faithfulness labels to the source documents. We regard the summary as a statement to be judged and the source document as the golden evidence.
* **SciFact**Wadden et al. (2020): It contains scientific claims against a corpus of scientific papers. The claims are re-formulated from

\begin{table}
\begin{tabular}{l c c c c c} \hline \multirow{2}{*}{**Verifiers**} & \multicolumn{4}{c}{SentCom} & \multicolumn{2}{c}{ParaGen (Sent)} \\ \cline{2-6}  & **AUR** & **AUP** & \(r\) & **AUR** & **AUP** & \(r\) \\ \hline \multicolumn{6}{c}{**Distinguishing Factual Stenertions from Unfactual Ones**} \\ \cline{2-6}
**FLAN-T5\({}_{\text{11h}}\)** & 93.0 & 97.7 & 70.7 & 90.2 & 88.3 & 71.4 \\
**CatGPT** & 83.3 & 90.5 & 62.2 & 82.8 & 66.4 & 63.6 \\ \hline \end{tabular} 
\begin{tabular}{l l c c c c} \hline \multirow{2}{*}{**SelfFact**} & \multicolumn{4}{c}{**Social Set 

naturally occurring citation sentences. We use the dataset to test the generalization of LLMs to more professional domains than Wikipedia.

We do not include FaVIQ used in SS5.3 in wks since (1) most statements of FaVIQ are transformed from information-seeking QA pairs, which is similar to BoolQ-FV; and (2) FaVIQ focuses on token-level factual errors, which have been covered by FEVER. To make wks less redundant, we do not include FaVIQ in wks.

## Appendix C Experiments

### Evaluation Metrics

We use multiple metrics to evaluate the verifiers, including ECE, ACC, AUR, AUP, and \(r\), etc. These metrics are focusing on different aspects. **(1)** ECE is the weighted average absolute difference between metric scores and accuracy in each bin of \([0,1]\) as follows:

\[\text{ECE}=\sum_{b=1}^{B}\frac{n_{b}}{N}|\text{acc}(b)-\text{conf}(b)|,\]

where acc(b) is the accuracy, conf(b) is the average metric scores (confidence), and \(n_{b}\) is the number of examples in \(b\)-th bin. We set \(B=20\) in our experiments. A well-calibrated verifier can serve as an estimation of probabilities of making mistakes, which is more informative than only predicting "factual" or not. Note that lower ECE does not mean better discrimination ability. Supposing that a model always predicts a score of 0.5 on a perfectly balanced dataset, ECE will be 0, while the model is useless. **(2)** ACC is the fraction of examples that are correctly predicted, which means both human judgments and metric scores are either larger or smaller than 0.5. **(3)** AUR is one of the commonest metrics to evaluate binary classifiers. Although both ACC and AUR test the discrimination ability, AUR does not assume a pre-defined

\begin{table}
\begin{tabular}{l} \hline
**SentCom demonstrations:** \\
1. **Swedish Empire** was ruled by Gustavus Adolphus from 1611 to 1632. \\
2. **The Boston Celtics play their home games at TD Garden. \\
3. **Chris Hensworth** appeared in A Perfect Getaway. \\
4. **History of art includes architecture, dance, sculpture, music, painting, poetry literature, theatre, narrative, film, photography and graphic arts. \\
5. **Nikolaj Coster-Waldau** worked with the Fox Broadcasting Company. \\ \end{tabular}
\end{table}
Table 14: Demonstrations for generating statements in mgs. The red words correspond to input in Tab. 1.

 

threshold, so it is more important on imbalanced datasets (e.g., BoolQ-FV and ParaGen). **(4)** AUP is also widely used for the evaluation of imbalanced datasets. For comparison, AUR/AUP is more sensitive to the discrimination ability of verifiers on the factual/non-factual category. Therefore, AUP is a better metric when most statements are non-factual (e.g., ParaGen). **(5)** All the above metrics require human judgments to be binary, while Pearson's correlation \(r\) can be used between two continuous variables (e.g., ParaGen (Para)). In summary, we recommend comprehensively considering different metrics from multiple perspectives in future research and realistic application.

### Retriever

We compare BM25 and Contriever Izacard et al. (2022) in terms of recall of golden evidence on wks, i.e., the ratio of tokens in golden evidence that also appear in the top 10 pieces of retrieved evidence. Tab. 15 shows the much higher recall scores of Contriever, so we use it in our experiments.

Furthermore, Tab. 16 shows the Pearson's correlation scores of FLAN-T511B and ChatGPT varying with the number of retrieved passages (each passage contains 150 tokens) on the FM2 dataset. We see that ChatGPT reaches saturation faster than FLAN-T511B with increased retrieved passages, although ChatGPT supports a maximum length of 4,096 tokens. This indicates the length extrapolation ability of FLAN-T5 to some extent10. We agree that the training-testing length bias may potentially impact the performance of FLAN-T5, although we have not explicitly observed such an impact. Actually, when the length of retrieved passages is much larger than 512 in our paper, FLAN-T511B always performs the best. We will further investigate the influence of the length bias in our future work.

Footnote 10: The encoders of FLAN-T5 models are trained with a maximum length of 512 tokens. Thanks to the relative positional encoding mechanism adopted by FLAN-T5 (all relative distances are truncated into 512), we can test the performance of FLAN-T5 with much longer inputs.

### Verifiers

Nli11bThe corresponding model card on HuggingFace is "google/t5_xxl_true_nli_mixture". It is trained on a mixture of SNLI Bowman et al. (2015), MNLI Williams et al. (2018), FEVER Thorne et al. (2018), Scitail Khot et al. (2018), PAWS Zhang et al. (2019) and VitaminC Schuster et al. (2021). Although FEVER is included in the training set of the model, we still regard this metric as unsupervised since they use golden evidence for training while we mainly use retrieved evidence for testing.

FiDTab. 17 shows the dataset transfer results of the supervised verifier FiD on wks, illustrating the poor generalization ability of supervised verifiers with a significant drop in accuracy when transferring it to a different dataset. On the other hand, when applying FiD on ParaGen (Sent), we do not provide the context during verification since FiD has not been trained to utilize the context.

Flan-T5FLNA-T5 series have been trained on SciFact with golden evidence. Therefore, the results of FLAN-T5 on SciFact in Tab. 6 may be over-estimated. We ensure that FLAN-T5 series have not been trained on the other three datasets of wks in the Wikipedia domain, including FEVER, Boolq-FV, and FM2.

 

soft coding (i.e., using the _generation probabilities_) based on FLAN-T5\({}_{11\text{h}}\), indicating that the performance of ChatGPT may be underestimated in terms of metrics that depend on probabilities, including ECE, AUR, AUP, and \(r\) in our experiments.

More LLMsIn addition to FLAN-T5 series, GPT3.5 and ChatGPT, we also test the performance of LLama7B, Alpaca7B [20] and InstructGPT on wks. Tab. 19 shows the results under different settings. We observe that:

**(1)**: InstructGPT is better calibrated than GPT3.5 with similar discrimination performance under all settings. Compared with CPT3.5, InstructGPT has not been trained through RLHF (reinforcement learning from human feedback), indicating that RLHF may impact calibration, which is accordant with GPT4's technical report [6].
**(2)**: LLama7B performs like random guessing on FM2 under all settings. When using retrieved evidence, its performance is also at the random level on FEVER, which can be improved using golden evidence, suggesting its poor robustness to noise in the given evidence. Surprisingly, when not using any evidence, the AUR score of LLama7B on FEVER instead increases compared with using golden evidence, despite the worse ECE and ACC scores. This means LLama7B is seriously biased toward a certain category when not using evidence, and providing golden evidence can alleviate the bias but impact the discrimination ability.
**(3)**: Alpaca7B is comparable to or better than LLama7B on the whole. One contrary phenomenon to LLama7B is that Alpaca7B is better calibrated with a higher ACC score when not using any evidence than using golden evidence on FEVER, indicating that the instruction tuning fashion of Alpaca7B increases the bias toward a certain category when using golden evidence, despite a better ability to utilize internal and external knowledge.

Results on FactPromptsThe dataset is released by a contemporary work [8], which comprises real-world prompts from various sources, such as Quora and TruthfulQA [14], along with corresponding responses generated by ChatGPT. We regard the response as the statement to be judged and retrieve 10 passages from C4 [12] using an off-the-shelf tool11 as the evidence. FactPrompts include 177 factual statements and 56 unfactual ones. All these statements are model-generated, so we believe that they should be unseen during the training stages of LLMs. Nevertheless, we acknowledge that texts with similar distributions to these statements might still be seen.

Footnote 11: [https://c4-search.apps.allenai.org/](https://c4-search.apps.allenai.org/)

As shown in Tab. 22, FLAN-T5\({}_{11\text{b}}\) outperforms ChatGPT with or without external evidence. ChatGPT without any evidence has a random performance on FactPrompts (AUR\(\approx\)50.0). This might be because the statements in the dataset are generated by ChatGPT itself, so ChatGPT easily regards them as factual statements.

### Verification Method

Besides using generation probabilities of certain answer words (e.g., "A," and "B") as factuality scores described in SS4.2, we also try several other methods widely used for NLG evaluation as follows:

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{3}{c|}{**FEVER**} & \multicolumn{3}{c}{**Boolo-FV**} & **FDAZ** \\ \cline{2-9}
**Dataset** & **ECE** & **AVR** & **ECE** & **AUR** & **\#** & **ECE** & **AUC** & \(r\) \\ \hline
**Soft** & **3.1** & **98.2** & **90.2** & **10.2** & **94.7** & **75.3** & **8.2** & **89.5** & **68.8** \\
**Hard** & 6.2 & 93.9 & 87.7 & 14.5 & 87.6 & 70.0 & 18.0 & 81.9 & 65.0 \\ \hline \end{tabular}
\end{table}
Table 18: Results of hard coding and soft coding based on FLAN-T5\({}_{11\text{h}}\).

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline \multirow{2}{*}{**Verifiers**} & \multicolumn{3}{c}{**FEVER**} & \multicolumn{3}{c}{**FM2**} \\ \cline{2-7}  & **ECE** & **ACC** & **ACU** & **ECE** & **ACC** & **ADR** \\ \hline
**Constant** & 48.3 & 51.7 & 50.0 & 50.6 & 49.4 & 50.0 \\ \hline \multicolumn{7}{c}{**Retrieving Evidence from External Corpora**} \\ \hline
**LLam7B** & 45.9 & 51.6 & 47.8 & 47.7 & 49.4 & 45.0 \\
**Alpaca7B** & 43.9 & 51.3 & 56.3 & 45.3 & 49.3 & 46.2 \\
**FLAN-T5\({}_{11\text{h}}\)** & **3.1** & **93.8** & **98.2** & **8.2** & **82.0** & **89.5** \\
**InstrectGPT** & 6.4 & 88.8 & 96.1 & 15.0 & 71.5 & 81.5 \\
**GTP3.5** & 7.6 & 91.7 & 96.6 & 21.0 & 77.5 & 82.8 \\
**ChatGPT** & 8.2 & 92.8 & 92.6 & 21.4 & 79.1 & 78.6 \\ \hline \multicolumn{7}{c}{**Using Golden Evidence**} \\ \hline
**LLama7B** & 6.3 & 59.6 & 64.8 & **6.4** & 54.6 & 55.8 \\
**Alpaca7B** & 43.8 & 52.1 & 86.4 & 46.7 & 49.2 & 73.9 \\
**FLAN-T5\({}_{11\text{h}}\)** & 5.4 & 93.3 & **98.6** & 9.7 & 87.3 & **96.1** \\
**instructGPT** & **3.9** & 93.9 & 98.0 & 8.0 & 85.4 & 92.6 \\
**GPT3.5** & 6.1 & 93.8 & 96.6 & 12.1 & 87.1 & 93.2 \\
**ChatGPT** & 5.8 & **94.4** & 94.3 & 11.6 & **88.5** & 88.4 \\ \hline \multicolumn{7}{c}{**Not Using any Evidence**} \\ \hline
**LLama7B** & 14.7 & 48.5 & 67.5 & **6.7** & 50.5 & 49.4 \\
**Alpaca7B** & **5.4** & 74.9 & 80.5 & 19.9 & 51.8 & 54.4 \\
**FLAN-T5\({}_{11\text{h}}\)** & 11.1 & 74.4 & **87.0** & 15.7 & 59.5 & 66.3 \\ \hline
**lnstructGPT** & 14.4 & 75.3 & 83.4 & 26.4 & 58.4 & 62.9 \\
**GPT3.5** & 25.6 & 73.8 & 78.3 & 41.4 & 57.3 & 61.5 \\
**ChatGPT** & 18.1 & **82.0** & 81.5 & 33.5 & **66.7** & **66.5** \\ \hline \hline \end{tabular}
\end{table}
Table 19: Performance of different instruction-tuned LLMs on FEVER and FM2. All models are under the zero-shot setting.

 

The task is to judge whether the statements are factually correct based on the corresponding facts. Two examples are as follows: Example 1. Facts:

0. Baseball Hall of Fame in 1988 \(\cdots\)

1. He was a carpenter and construction worker by trade. \(\cdots\)

2. players, and 599 of them loved Willie Stargell \(\cdots\)

Statement: Willie Stargell lived with his father after his mother and father separated.

Output: Willie Stargell lived with his mother in Florida after his parents' divorce, and later returned to live with her in California. There is no mention of him living with his father after the separation. So the statement is incorrect; the answer is No.

Example 2. Facts:

0. Alfredo Stroessner Alfredo Stroessner Matiauda \(\cdots\)

1. in most other Latin American countries \(\cdots\)

2. led to the liberalization of Paraguay \(\cdots\)

3. system. On 3 February 1989, Stroessner was \(\cdots\)

4. some of Stroessner's policies \(\cdots\)

Statement: After the overthrow of Stroessner, Paraguay had its first non-military head of state in four decades.

Output: Stroessner, a military officer, had been in power for 35 years, from 1954 to 1989, and had been overthrown in a military coup led by General Andres Rodriguez. After Stroessner's ousting, Rodiriquez orchestrated a political campaign with the Colorado Party and won the presidency in a multiparty election held in May 1989. This marked the first time in four decades that Paraguay had a non-military head of state. So the statement is correct; the answer is Yes.

Now please judge the following statement: The task is to rate the factuality level for the statement based on the fraction of details in the statement that can be supported by the facts. You should output only one single number from 1, 2, 3, 4, and 5, indicating the factuality score of the statement, where a higher score means that more details of the statement are factual.

1 - Almost all details in the statement are not supported by the facts.

2 - Some details in the statement are supported by the facts, but the majority are not.

3 - About half of the details in the statement are supported by the facts.

4 - Most of the details in the statement are supported by the facts.

5 - All details in the statement are supported by the facts.

Please output the score of the following statement:

Table 20: Input Prompt for CoT prompting (Top) and Likert-scale rating (Bottom).

* **Chain-of-Thought Prompting (CoT)** was proposed to elicit LLMs' reasoning ability Wei et al. (2022), and recently was also used for improving NLG evaluators Liu et al. (2023b).
* **Likert-Scale Rating** is the most common method for human and automatic evaluation Gao et al. (2023). We prompt LLMs to rate statements from 1 to 5 and re-scale the ratings into the range from 0 to 1 as the final factuality scores. Tab. 20 shows the prompt to instruct LLMs to judge statements using the above two verification methods. We compare different verification methods based on ChatGPT and show the results in Tab. 21:

**(1)** Directly predicting "A" or "B" is the simplest but best verification method.
**(2)** Using CoT prompting will lead to a significant performance drop. We attribute the drop to the impact of generated non-factual rationales, which misleads ChatGPT to mistake factual statements for non-factual ones, as indicated by the similar precision but lower recall than direct prediction.
**(3)** When using Likert-scale rating, LLMs tend to give higher scores than direct prediction, with a lower precision but higher recall. The reason may be that the ground-truth labels of the statements are collected more extremely than Likert-scale rating, which means only very minor factual errors can make a statement labeled unfactual. For example, a statement with a ground-truth label of 0 can be rated 4 (0.75 after re-scaling) under the Likert-scale rating. Therefore, Likert-scale rating may also be a good choice for fact verification if users

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Methods** & \multicolumn{6}{c}{**FEVER**} \\ \cline{2-7}  & **ACC** & **P** & **R** & \(r\) & **ACC** & **P** & **R** & \(r\) \\ \hline
**Direct** & **92.8** & 92.8 & 92.8 & **92.8** & **79.1** & 80.1 & 75.2 & **56.7** \\
**CoT** & 87.5 & **93.2** & 81.8 & 75.7 & 70.7 & **83.1** & 51.1 & 44.5 \\
**Likert-Scale** & 89.6 & 86.9 & **93.4** & 79.6 & 76.1 & 74.6 & **76.1** & 53.5 \\ \hline \hline \end{tabular}
\end{table}
Table 21: Performance of different verification methods. **Direct** means directly predicting “A” or “B” as described in §4.2.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Models** & **ACC** & **AUR** & \(r\) \\ \hline
**Retrieving** Evidence from External Corpora** \\ \hline
**FLAN-TS\({}_{118}\)** & 64.4 & **68.1** & **26.5** \\
**ChatGPT** & **71.7** & 59.4 & 19.5 \\ \hline \multicolumn{4}{c}{Not Using any Evidence} \\ \hline
**FLAN-TS\({}_{118}\)** & 63.5 & **67.6** & **26.0** \\
**ChatGPT** & **72.1** & 50.5 & 1.6 \\ \hline \hline \end{tabular}
\end{table}
Table 22: Results on FactPrompts.

 

expect the score to be less strict and more informative. In future research, we recommend focusing on standardizing Likert-scale protocols for factuality rating and designing efficient and effective methods for detecting fine-grained factual errors (e.g., at the token level).

### Prompts for Robustness Assessment

We design different prompts to assess the robustness of LLMs in SS5.2 by changing the "Question" part in Tab. 4 to:

* "Is the statement entailed by the given facts? (A) Yes. (B) No. Please answer A or B:"
* "Based on the given facts, judge whether the statement is factually correct. Please answer Yes or No:"
* "Can the given facts support the statement? Please answer Yes or No:"

### Influence of Non-Factual Context

Influence on Model GenerationIn Tab. 2, we observe the increase of no-evidence sentences as the generation proceeds to later sentences. We conjecture that it is because of the error accumulation during generation. A contemporary work Zhang et al. (2023) also highlights the phenomenon and empirically attributes it to error propagation. We show our finding in Tab. 23: There is a lower proportion of no-evidence sentences when the context is factual than non-factual. Here, factual context means the context of the sentence includes only factual sentences, and non-factual context means the opposite.

Intuitively, more errors in the context (unfactual or no-evidence) correlate to more no-evidence sentences. However, it may be difficult to affirm the causality between them. The potential insight is that when generating open-ended long texts using auto-regressive models, it is necessary to dynamically involve external feedback, e.g., from humans, retrievers, planners, etc., to avoid the impact of non-factual contexts.

Dependency AnnotationFor each sentence in ParaGen (Sent), we manually annotate whether it involves dependencies on the context, i.e., including references to any entities in the context. We label a sentence independent if

1. It is the first one in the paragraph (137 sentences totally).
2. It does not contain nouns or pronouns that refer specifically to entities in the context.
3. It contains nouns that refer to entities in the context but can be understood solely. For example, in the paragraph "Russian wine is produced in several regions. \(\cdots\) Russian wines have won numerous international awards.", the second "Russian wine" in the last sentence refers to the same entity as the context, but the last sentence is not dependent on the context. Otherwise, the sentence is labeled dependent on the context. We asked two graduates for annotation and found they annotate the same labels for more than 90% sentences. Finally, we obtain 189 context-dependent sentences (61 factual and 128 non-factual), and 279 context-independent ones (94 factual and 185 non-factual).

De-ContextualizationWe use ChatGPT for coreference resolution (CR) using the prompt in Tab. 24. To assess the accuracy of the CR method, we inspect twenty randomly sampled CR results and find ChatGPT can complete the task perfectly without any errors. This may be because all sentences in a paragraph of ParaGen are almost centered on the same entity so that it is easy to resolve coreferences.

Influence on Fact VerificationTab. 11 in the main paper has shown greater difficulty in verifying context-dependent statements than context-independent ones. We are also curious whether

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Proportion (\%)** & **Factual** & **Unfactual** & **No Evidence** \\ \hline
**Factual Context** & 45.0 & 45.5 & 9.5 \\
**Non-Factual Context** & 15.1 & 42.1 & 42.7 \\ \hline \hline \end{tabular}
\end{table}
Table 23: Label distribution of sentences in ParaGen (Sent) with factual or non-factual context. Here, factual context means the context of the sentence includes only factual sentences, and non-factual context means the opposite.

\begin{table}
\begin{tabular}{p{113.8pt} p{34.1pt}} \hline \hline Task: Given a statement following its context, find all coreferences to the context in the statement, and replace them with words that they refer to. Be careful not to change the original word order as much as possible. \\ \hline For example, if the context is “Mary lves pizzas.”, and the statement following the context is “She eats then every day.”, you should output “Mary eats pizzas every day.” Now finish the following task: \\ Context: [\(c\)] Statement following the context: \textbackslash{} \\ Output: \\ \hline \hline \end{tabular}
\end{table}
Table 24: Prompts to instruct ChatGPT to perform coreference resolution.

 

non-factual context will impact the performance of the fact verifiers. Tab. 25 answers the question in the affirmative. Performing CR will alleviate the disparity by improving the performance on statements with non-factual context.

### Entity Type

Tab. 26 shows details of entity types used in Fig. 4. We see that "Person," "Work of Art," and "Building" have relatively lower \(\frac{3}{4}\) percentile. This means these types of entities are distributed more sharply, which may account for better performance on the statements corresponding to these types than other types.

### Influence of LLMs' Memorization

We are curious about whether there is a degree of memorization when LLMs make judgments about factuality. For example, they may inherently tend to judge those texts that are more probable under them as hallucinatory regardless of the given evidence. To investigate this correlation, we take perplexity as a proxy to measure the memorization degree of an LLM for any one statement. Then, we compute Pearson's correlation between the LLM's perplexity and predicted/golden factuality score on the FEVER and FM2 test sets. It is intractable to compute the perplexity under GPT models since their APIs do not return the logits of any specified texts. Therefore, we only compute the correlation based on the FLAN-T5 models, and use its decoder to compute the perplexity. Tab. 27 shows the correlation results.

We see that the perplexity of FLAN-T5 correlates significantly with the predicted factuality score: When the LLM memorizes a statement better (indicated by a lower perplexity score), it is more likely to regard the statement as factual. However, the perplexity score does not correlate with the golden label. The results suggest that memorization can influence LLMs' judgments about factuality to some extent, but may not be the main contributor to the superior fact verification performance of LLMs.

### Case Study

Fig. 5 plots precision and recall to better understand the preference of different LLMs. On SciFact, ChatGPT surpasses FLAN-T5118 with higher precision and recall. In the Wikipedia domain, ChatGPT exhibits a preference for predicting factual in contrast to FLAN-T5118, resulting in higher recall on factual statements (Fig. 5 c). Such a preference also leads ChatGPT to make more mistakes when predicting factual (Fig. 5 a) and misidentify unfactual statements as factual ones (Fig. 5 d). Manual inspection reveals that ChatGPT easily builds spurious connections between related yet distinct concepts, as indicated in the following case study. On the other hand, Zhong et al. (2023) also emphasized the inadequacy of ChatGPT in assessing inter-sentence similarity compared with BERT-based models, which further supports our observation.

Tab. 29 shows a case that ChatGPT fails to judge while FLAN-T5118 does not, suggesting that ChatGPT may build spurious connections between related yet distinct concepts (e.g., "math teacher" and "study mathematics").

Through manual inspection of test examples that both ChatGPT and FLAN-T5118 make wrong judgments, we summarize several typical types of errors. To illustrate these error types, Tab. 28 shows several representative cases:

**(1) False Numerical and Logical Reasoning:** In Example 1 and 2, despite correct retrieval, ChatGPT mistakes "36 years old" for "40s", and uses "Continuity is not sufficient for dif

Figure 5: Precision and recall scores with retrieved evidence in wks (**Left:** Factual **Right:** Unfactual). **Takeaway:** ChatGPT prefers to predict factual.

 

ferentiability" to refute "differentiability is sufficient for continuity," indicating the weakness of LLMs in understanding numerical and logical relations.
2. **Misled Retriever:** In Example 3, the retriever is misled by the non-factual information (i.e., "the United States") in the statement, thus leading to useless evidence and wrong predictions. When giving correct evidence to LLMs (i.e., the Wikipedia page "Race to Berlin"), we find they can make the right prediction.
3. **Specious Inter-Entity Relations:** Statements generated by larger models (such as GPT3.5) tend to seem more fluent and factual with interrelated entities, although the relations between them may be non-factual. In Example 4, the evidence provides useful information about "IES," but it does not mention whether "IES" is linked to the government or its research is only limited to the British. ChatGPT ignores such specious relations and hence makes wrong predictions.

\begin{table}
\begin{tabular}{l l c r} \hline \hline
**Entity Types** & **spaCy Label** & **Description** & \(\frac{3}{4}\) **Percentile** \\ \hline
**Non-NE** & N/A & Common words that are not named entities & 9,526. 5 \\
**Person** & PERSON & People, including fictional & 420. 0 \\
**Work of Art** & WORK_OF\_ART & Titles of books, songs, etc. & 2,166. 0 \\
**Product** & PRODUCT & Objects, vehicles, foods, etc. (not services) & 3,456. 5 \\
**Event** & EVENT & Named hurricanes, battles, wars, sports events, etc. & 8,559. 0 \\
**Language** & LANGUAGE & Any named language & 245,337. 0 \\
**Building** & FAC & Buildings, airports, highways, bridges, etc. & 603. 0 \\
**Company** & ORG & Companies, agencies, institutions, etc. & 18,557. 5 \\
**Group** & NORP & Nationalities or religious or political groups & 21,983. 5 \\
**Country** & GPE & Countries, cities, states & 205,839. 0 \\
**Location** & LOC & Non-GPE locations, mountain ranges, bodies of water & 8,624.75 \\
**Date** & DATE & Absolute or relative dates or periods & 372,927. 0 \\
**Cardinal** & CARDINAL & Numerals that do not fall under another type & 3,572,566. 0 \\
**Ordinal** & ORDINAL & “first”, “second”, etc. & 50,076. 0 \\ \hline \hline \end{tabular}
\end{table}
Table 26: Entity Types and corresponding labels and descriptions from spaCy. \(\frac{3}{4}\) percentile means \(\frac{3}{4}\) of entities of some entity type appear in fewer Wikipedia passages than that number.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Models** & **FEVER** & **FM2** \\ \hline FLAN-T5\({}_{\text{7500h}}\) & -16.4* & -11.6** \\ FLAN-T5\({}_{\text{JB}}\) & -11.0** & -15.2** \\ FLAN-T5\({}_{\text{1IB}}\) & -10.3** & -12.2** \\ \hline \hline \multicolumn{3}{c}{**Correlation between Perplexity and Golden Factuality Score**} \\ \hline
**Models** & **FEVER** & **FM2** \\ \hline FLAN-T5\({}_{\text{7500h}}\) & -5.7** & 3.4** \\ FLAN-T5\({}_{\text{3B}}\) & -2.6** & 2.5** \\ FLAN-T5\({}_{\text{1IB}}\) & -3.6** & 3.7** \\ \hline \hline \end{tabular}
\end{table}
Table 27: Correlation scores between FLAN-T5’s perplexity and predicted/golden factuality scores on FEVER and FM2. ** means p-value\(<\)0.01.

 



\begin{table}
\begin{tabular}{l l c c c c c}
**ID** & **Statement** & **Retrieved Most-Related Evidence** & **Label** & **Pred** & **ChatGPT Explanation** \\
1 & Georges Bizet died in his early 40s. (_from FM2_) & ✓(Georges Bizert) Georges Bizet (25 October 1838 – 3 June 1875), registered at birth as Alexandre César Léopold Bizet, was a French composer of the Romantic era. & 0 & 1 & Georges Bizet was born in 1838 and died in 1875, which means he was 36 years old when he died. This is considered to be in his early 40s. & 1 & 0 & A function can be continuous at a point x0, then f must also be continuous at x0. In particular, any differentiable function must be continuous at every point in its domain. & 0 & 1 & The U.S. Army’s 2nd Armored Division was the first American division to reach the Elbe at Schonebeck on 11 April and Entered Berlin in July, making the statement correct. \\
3 & **The United States** was the allied nation that was the first to reach the German capital of Berlin. (_from FaVIQ_) & �(2nd Armored Division United States) After a rest in February, the division, now commanded by Major General Isaac D. White,roe on across the Rhine on 27 March, and was the first American division to reach the Elbe at Schonebeck on 11 April. & 0 & 1 & The U.S. Army’s 2nd Armored Division was the first American division to reach the Elbe at Schonebeck on 11 April and Entered Berlin in July, making the statement correct. \\
4 & Institute for Employment Studies is a **government-linked** think tank on **British** employment policy and practice. (_from ParaGen_) & �(Institute for Employment Studies) The Institute for Employment Studies is a British think tank (IES) is a British centre of research and consultancy on human resources and employment. & 0 & 1 & The Institute for Employment Studies is a British think tank that focuses on research and consultancy related to human resources and employment. \\  & & & & & that focuses on research and consultancy related to human resources and employment. \\  & & & & & resources \\ \end{tabular}
\end{table}
Table 29: An unfactual statement that ChatGPT fails to judge while FLAN-T511b succeeds. We obtain the explanation by prompting ChatGPT to explain its prediction. Red words are copied from the evidence, and cyan and blue words are conflicting.