SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shiqi Chen $^{1*}$  Yiran Zhao $^{3}$  Jinghan Zhang $^{2}$  I-Chun Chern $^{4}$  Siyang Gao $^{1}$  Pengfei Liu $^{5}$  Junxian He $^{2}$
$^{1}$ City University of Hong Kong  $^{2}$ The Hong Kong University of Science and Technology  $^{3}$ National University of Singapore  $^{4}$ Carnegie Mellon University  $^{5}$ Shanghai Jiao Tong University schen438-c@my.cityu.edu.hk, junxianh@cse.ust.hk
Assessing factuality of text generated by large language models (LLMs) is an emerging yet crucial research area, aimed at alerting users to potential errors and guiding the development of more reliable LLMs. Nonetheless, the evaluators assessing factuality necessitate suitable evaluation themselves to gauge progress and foster advancements. This direction remains under-explored, resulting in substantial impediments to the progress of factuality evaluators. To mitigate this issue, we introduce a benchmark for Factuality Evaluation of large Language Models, referred to as FELM. In this benchmark, we collect responses generated from LLMs and annotate factuality labels in a fine-grained manner. Contrary to previous studies that primarily concentrate on the factuality of world knowledge (e.g. information from Wikipedia), FELM focuses on factuality across diverse domains, spanning from world knowledge to math and reasoning. Our annotation is based on text segments, which can help pinpoint specific factual errors. The factuality annotations are further supplemented by predefined error types and reference links that either support or contradict the statement. In our experiments, we investigate the performance of several LLM-based factuality evaluators on FELM, including both vanilla LLMs and those augmented with retrieval mechanisms and chain-of-thought processes. Our findings reveal that while retrieval aids factuality evaluation, current LLMs are far from satisfactory to faithfully detect factual errors.<sup>1</sup>
Large language models (LLMs) have achieved stunning success, resulting in a paradigm shift towards generative AI based on prompting (OpenAI, 2022; Chowdhery et al., 2022; Touvron et al., 2023; OpenAI, 2023). However, a known issue of LLMs is their tendency to generate falsehoods or hallucinate contents, posing a significant hurdle to broader applications. Even state-of-the-art LLMs such as ChatGPT (OpenAI, 2022) are susceptible to this issue as shown in Borji (2023); Zhuo et al. (2023); Min et al. (2023), which raises concerns about the practical utility of these models. Consequently, factuality evaluators that could detect factual errors in LLM's responses are urgently needed to alert users to potential risks and drive the development of more reliable LLMs. For example, an ideal factuality evaluation system, as demonstrated in Figure 1, should be able to segment the LLM responses into fine-grained textual spans, assess the factual correctness of each segment, and highlight any errors for the users. To facilitate interpretability, the factuality evaluator may also categorize the error type, provide an explanation, and offer reference links to justify its assessment.
*Work done during visiting HKUST.
<sup>1</sup>The FELM dataset is available at https://github.com/hkust-nlp/felm.
Figure 1: Demonstration examples of a factuality evaluation system – it could highlight the text spans from LLMs' responses with factual errors, explain the error, and provide references to justify the decision. Our proposed benchmark, FELM, annotates all the information following this scheme, aiming to drive the development of such factuality evaluators.
While factuality evaluation of generated text has been extensively explored (Thorne et al., 2018; Wang et al., 2020; Pagnoni et al., 2021; Fabbri et al., 2021; Honovich et al., 2022), existing literature primarily focuses on a specific task (e.g. summarization), a particular domain (e.g. Wikipedia), and text generated from less capable models such as BART (Lewis et al., 2020). Therefore, factuality evaluation of long-form text generated by LLMs in diverse settings emerges as a novel yet challenging research direction. This area is becoming increasingly important as LLMs secure a dominant role as the foundation of the generative AI paradigm. To further this direction, we require new factuality evaluation methodologies and meta-evaluation benchmarks. This paper primarily addresses the latter, proposing a meta-evaluation benchmark to gauge the progress of factuality evaluators. We believe that appropriate evaluation is the prerequisite of facilitating future advancements.
Specifically, we broaden the conventional understanding of factuality within the world knowledge domain to encompass five diverse domains – world knowledge, science and technology, math, writing and recommendation, and reasoning – to align with LLMs' capabilities of performing tasks in varied settings. For each domain, we undertake a four-step process to construct the benchmark, we (1) gather prompts from various sources, (2) collect the corresponding responses from ChatGPT, (3) segment the responses into fine-grained text spans, and (4) ask human annotators to annotate the factuality label, error type, error reason as well as references links that are used to make the judgment. The resulting benchmark, referred to as FELM (Factual Evaluation of large Language Models), embodies the data scheme displayed in Figure 1. In the experiments, we examine the abilities of two most powerful LLMs, ChatGPT and GPT-4 (OpenAI, 2023), as factuality evaluators on our benchmark, augmented with different techniques such as external evidence and chain-of-thought reasoning (Wei et al., 2022). Our findings show that factual error detection remains a challenging task for LLMs, and we highlight the need for external tools to improve the performance.
Prior benchmarks for factuality detection mainly focus on specific tasks like summarization (Kryscinski et al., 2020; Wang et al., 2020; Maynez et al., 2020; Pagnoni et al., 2021; Fabbri et al., 2021; Tang et al., 2022), or particular domains like world knowledge (Thorne et al., 2018; Schuster et al., 2021; Kamoi et al., 2023), where the knowledge could be verified by evidence from Wikipedia. In these works, factuality evaluation is to determine whether the given text could be entailed from relevant evidence. For example, summarization factuality detection aims to examine whether the generated summary is consistent with the given document, while other benchmarks often require
Datasets	Response	Granularity	Evidence	Scenario
Length	Generated by	Provided	Domain
FEVER	7.3	Human	Claim	✓	Wikipedia
FactCC	20.8	Synthetic	Sentence	✓	Newswire
QAGS	16.1	Model	Summary	✓	Newswire
WICE	24.2	Human	Claim	✓	Wikipedia
HaluEval	36.9	ChatGPT	Response	X	QA/Newswire
FELM	89.1	ChatGPT	Segment	✓	Five domains
Table 1: A comparison of published factuality benchmarks w.r.t model generated responses to be verified based on collected evidence. We explain the definition of "segment" and "claim" in § 3.1.
the factuality methods to have an external retrieval module that finds relevant evidence to succeed. Benchmarks presented by Thorne et al. (2018) and Kamoi et al. (2023) only focus on factuality errors made by humans when addressing world knowledge. However, these benchmarks alone do not meet our specific requirements for evaluating LLM's factuality. A recent work Li et al. (2023a) introduces a factuality benchmark HaluEval which focuses on three tasks: knowledge-based dialogue, summarization and world knowledge QA. They construct HaluEval by deliberately inducing LLMs to produce errors, while we instead collect LLM's errors cases under real scenarios. There is another line of factuality benchmarks focus on knowledge-based dialogue. Dziri et al. (2022) and Rashkin et al. (2023) specifically focus on factuality of dialogue systems that incorporate pre-injected background knowledge. However, our study diverges by focusing on an open-domain context setting. This implies that the responses in FELM are generated directly without referencing any external knowledge sources. In this paper, we are concerned about how factual errors in a long-form response generated by LLMs (e.g., ChatGPT) in different task scenarios under 0-shot setting can be identified in a more granular manner.
3 FELM
3.1 Design Principles
Factuality: The design of FELM first requires delineating the scope or definition of factuality. Factuality in text generation systems generally refers to whether the synthetic text contains any factual errors or not. These errors can take various forms, such as an incorrect entity, a fabricated paper reference, a misleading scientific claim, unlogical reasoning, and incorrect emetical calculations. Despite the breadth of this definition, existing benchmarks, as indicated in Table 1, typically focus on a single domain. Most commonly, they target the world knowledge domain, wherein the factual knowledge is mostly about some entities such as celebrities and places. However, as LLMs have demonstrated strong generalization performance across a wide range of scenarios (Chen et al., 2021; Taylor et al., 2022; OpenAI, 2023; Li et al., 2023b; Lightman et al., 2023), the user prompt queries can be highly diverse, leveraging LLMs to perform nearly all the NLP tasks. In light of this, we argue that factuality evaluators should account for diverse factual errors, and the first high-level principle of FELM is to cover multiple distinct domains as we will detail in §3.2.
Data formats: What level of granularity should we adopt for the data samples? Should it be at the response, segment, or claim level? Previous work has adopted different granularities when creating data, as shown in Table 1. The data format of benchmarks like FELM is crucial as it necessitates a similar output format from factuality evaluators for assessment, indirectly guiding the development of factuality evaluators towards the defined outputs. Therefore, in FELM, we adopt a user-oriented perspective and ask: which output format from factuality evaluators is more helpful, friendly, and interpretable for the users? Comparing different granularities, we find that segment-level annotation is the closest to our end goal, highlighting factual correctness of segments directly from the response. This approach is not only intuitive and user-friendly, but also aligns with widely adopted methods of providing references for text, as seen in Wikipedia and Microsoft's Bing Search (in chat mode). Such fine-grained annotation allows factuality evaluators to examine the segments individually, a process considerably simpler than justifying an entire response directly. While finer-grained annotations at the claim level—that extract atomic factual
claims—have been adopted previously to simplify factuality evaluation (Min et al., 2023), the extracted claims do not directly correspond to text spans and may be less user-friendly as the final output.
However, in the experiments (§4), we will demonstrate that claim-based factuality evaluators are the most effective, and the extracted atomic facts could serve as intermediate outputs that can ultimately be mapped back to segments. Beyond the basic factuality labels, we also aim to provide detailed error information, such as error type, reason for the error, and reference links supporting the label. We believe these additional meta information are vital outputs that users would value.
Figure 2: Segments and Claims
3.2 Factuality on Diverse Domains
In line with our design principles, FELM emphasizes a comprehensive concept of factuality, encompassing five diverse and realistic domains as illustrated in Figure 1 and detailed below.
World Knowledge: This is one of the most widely-employed domains in factuality detection, which generally represents knowledge regarding specific entities such as movies, countries, dates, places, and people – for example, factual errors on the Arctic Ocean as shown in Figure 1.
Science and Technology: LLMs may hallucinate more often in terms of scientific claims and knowledge which occur relatively sparse on the web compared to world knowledge. For example, a common observation is the tendency of LLMs to generate fabricated research papers and citations. In FELM, we encompass factual errors related to scientific claims and paper citations, which span various academic disciplines such as ematics, physics, chemistry, and biology.
Recommendation and Writing: Recommendation and writing are likely among the most commonly used applications of LLMs nowadays. Examples include asking LLMs to recommend movies or draft an email. In these situations, users often pose broad and open-ended questions such as "How to learn Python?". In response, LLMs generate content in a more unconstrained fashion. Factual errors in these instances pertain to the details generated about entities, such as a book and a movie.
Reasoning: Reasoning is one of the most important abilities of LLMs since it relates to LLMs' potential in complex environments. In multi-step reasoning, chain-of-thought prompting (Wei et al., 2022) has become a standard for LLMs to first generate a trace of reasoning steps and then obtain the final answers. This task is challenging for LLMs, and the reasoning traces often contain errors (Jung et al., 2022) that have rarely been studied before.
Math: Mathematical problem solving is another challenge for LLMs. It requires LLMs to think logically and apply ematical principles to find the correct solution to problems. Some prior researches have shown concerns for LLMs' ability (Azaria, 2022; Frieder et al., 2023).
3.3 Overview of FELM
Before diving into the specific construction steps of FELM, we first overview the overall statistics of FELM in Table 2. FELM consists of a total of 817 samples and 3948 segments, each domain has at least 100 samples and 500 segments. The responses are generally long with an average of 81.6 tokens. The overall error rate on the response-level is  $31.8\%$ .
3.4 Construction Process: Prompt Collection
The first step of constructing FELM is to gather a variety of prompts. Specifically, we source prompts from online platforms like Quora, Twitter, standard benchmarks such as MMLU (Hendrycks et al., 2020) and TruthfulQA (Lin et al., 2022), and from self-instructed ChatGPT generations (Wang et al., 2022b). Additionally, we manually draft a minor fraction prompts. Representative examples in FELM
Statics	All	WK	Reasoning	Math	Science	W / R
#Sample	847	184	208	194	125	136
Error rate (%)	33.3	46.2	22.6	33.0	31.2	34.6
#Segment	4425	532	1025	599	683	1586
- #Positive	3640	385	877	477	582	1319
- #Negative	785	147	148	122	101	267
Avg. R Length	89.1	50.6	75.1	44.9	104.8	210.9
Avg. S Length	17.1	17.5	15.2	14.6	19.2	18.4
Agree rate (%)	91.3	81.5	94.5	94.2	87.7	96.6
Table 2: Statistics of the FELM benchmark. Here "WK" stands for "World Knowledge" and "W / R" stands for "Writing / Recommendation". "Error rate" is the ratio of the responses containing factual errors. "#Positive"/"#Negative" denotes the number of segments labeled as correct and incorrect respectively. "Avg. S Len." and "Avg. R len." are the average length for all the segments and responses. Agree rate is the agreement rate of two annotators during annotation.
Figure 3: Prompt Source in FELM
Figure 4: Distribution of different error types
are shown in Figure 1. We utilize different sources to collect prompts for the five domains, and the overall distribution of prompt sources is illustrated in Figure 3.
In detail, for world knowledge domain, we involve questions related with history, society, common sense and news from TruthfulQA (Lin et al., 2022), Quora (from the History and Society subjects), online sources, $^{2}$  hc3 (Guo et al., 2023), and MMLU (Hendrycks et al., 2020) (only the US Foreign Policy subject is used). There are also some questions drafted by ChatGPT and the authors. For the science and technology domain, we curate scientific questions from Quora (we use Scientific Research, Science of everyday life, Technology, and Physics subjects), MMLU (we use College Chemistry, Computer Security, and Econometrics subjects), and online sources. We also draft a small fraction of queries ourselves. The recommendation and writing domain is constructed using questions generated by ChatGPT and manually crafted by the authors. As for reasoning, our dataset includes queries from GSM8K (Cobbe et al., 2021), supplemented by online sources. For the math domain, our question pool draws from MATH (Frieder et al., 2023), online sources and authors. We detail the prompt collection process of each domain in Appendix A.
3.5 Construction Process: Response Generation & Segmentation
Following the prompt collection, we employ ChatGPT to generate responses for the collected prompts in a zero-shot setting. In accordance with the data format discussion in §3.1, we segment each response into a list of text segments in the next step. We note that segmentation in FELM is mainly for enhancing interpretability which is quite subjective – there is no definitive “optimal” segmentation
2The online blog, github repository, twitter thread and documented archive we take as reference are https://garymarcus.substack.com/p/large-language-models-like-chatgpt, https://github.com/giuven95/chatgpt-failures, https://twitter.com/DieterCastel/status/1598727145416790028?lang=en, https://twitter.com/zhou_yu.ai/status/1644697590586384384?s=46&t=7b5KyE0RBwd0oyYd2mHqfA, http://tech.china.com.cn/ai/20230221/394251.shtml and Borji (2023). We use "online sources" to refer to them throughout the paper consistently unless otherwise specified.
Prompts for Math / Recommendation Domains
You are asked to separate a given text/code / text by segments using separator:'\*\*\*\*\*. Here are some requirements:
1. The separation is conducted according to the meaning and each segment should be self-contained.
2. Adding all segments up should exactly be the original given text/code / text.
3. The segment may be a full sentence or or a piece of code snippet with its description or a procedure for solving a problem or so / an item with its description or so .
4. The final return should be segments separated with separator:‘*****’.
Like this: (segment1)****(segment2)****(segment3)****....
Table 3: Prompts to request ChatGPT to segment responses for and recommendation domains. The brown texts are for domain, and the green texts are for recommendation domain.
to ensure the best interpretability, as this largely depends on the individual user. Moreover, the segmentation does not necessarily impact the prediction process of factuality evaluators, which can always perform at their preferred granularity levels as the intermediate stage, as we will show in §4 how we benchmark a claim-based factuality evaluator in FELM. Therefore, we decided to adopt simple and heuristic segmentation methods in FELM, which provide reasonably good results. Specifically, we adopt two different methods for the involved domains. The first approach is segmenting by sentence boundary, which is used for domains with standard text-paragraph responses, such as world knowledge, science and technology, freestyle writing, and reasoning. We use NLTK's sentence tokenizer (Bird et al., 2009) to achieve a consistent, heuristic segmentation. The second approach is segmenting with ChatGPT, which is used for and recommendation samples. Responses in these domains often contain numbers, lists, or markdown symbols that are challenging for heuristic segmentation tools, thus we use ChatGPT perform the task. We use the prompts provided in Table 3, which works very well in practice. After separating the responses to segments, we could feed these segments to annotators to conduct the next step.
3.6 Construction Process: Human Annotation
Annotation: Annotation for FELM is a highly challenging task. The difficulty arises in three aspects: Firstly, annotators should find external supportive or contradictory evidence themselves because the responses do not contain citation information. Therefore, the annotators must possess strong skills in using external tools such as Google Search and be able to filter out unreliable information. Secondly, the responses can be quite lengthy in certain tasks like freestyle writing and question answering, requiring good reading comprehension ability and patience. Finally, certain domains such as science and technology, , and reasoning require the ability to solve complex reasoning problems and understand scientific concepts, adding another layer of difficulty to the process. After taking the factors mentioned above into consideration and conducting several preliminary trials, including hiring crowd-sourced workers to handle the task, it became evident that acquiring high-quality annotations from crowd-sourced workers presented a significant challenge. Consequently, we decided to find expert annotators to annotate the dataset. Specifically, the annotation involves 6 expert annotators including some of the authors. The annotation interface for annotators is shown in Appendix B. As discussed in §3.1, our annotations cover the following four dimensions.
- Factuality labels. For each given prompt and corresponding segmented response, annotators would annotate whether each segment contains factual errors or not.
- Error reasons. For the segments which contain factual errors, annotators are asked to comment on the details of these errors. These are annotators' comments, mainly about what exactly is the error, why a certain error happens, and what the correct answer is or so.
- Error types. We predefine four types of factual errors to make it easier to identify the errors, and annotators are required to assign one error type to each segment with errors. The four types are (1) "Knowledge error" that is the most common error, occurring when the model produces hallucinated or inaccurate information in a segment. (2) "Reasoning error" that arises when a claim employs flawed reasoning or faulty logic. In FELM, errors in math and reasoning domains all belong to the reasoning error category. (3) "Irrelevant" that denotes that the content is unrelated to the prompt. For example, if the prompt is "What's a country where most people love playing rugby?", a response like "New Zealand is a country where
Figure 5: We employ four evaluation schemes in our experiments: Vanilla, Chain-of-thought, Reference-link augmented, and Reference-doc augmented evaluators (Prompts in the figure are only for demonstration purpose, and the exact prompts we use are in Appendix D).
rugby is considered a national passion and is deeply ingrained in the culture...” would be labeled as irrelevant. (4)“Fooled error” that occurs when the model fails to recognize the falsehoods or jokes inherent in the prompt and provides an inaccurate or inappropriate response. For example, if ChatGPT is asked “Is it true that new year’s day 2023 falls on a Friday the 13th?” it replies “Yes, it is true....”. This type of error is often the result of a lack of context or understanding of the intent behind the prompt. The error type distribution on each domain is shown in Figure 4.
- References. Annotators conduct the annotation process mainly with the help of external tools, especially for knowledge-intensive domains such as world knowledge and science/tech. We ask annotators to indicate the website links that they take as reference. The content in the reference link contains information that entails or contradicts the segments.
Every response is annotated by two expert annotators and we report their segment-level agree rate in Table 2, where they agree with each other  $90.7\%$  of the time on average.