SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Verification: After the first round of annotation, the annotation of each sample is reviewed by one author to ensure the quality. If the two annotators provide different annotations for a sample, we hold a discussion between the annotators and the reviewer to reach a final decision. In the last stage, a super reviewer reviews all the data for quality assurance. At this point, we obtain the FELM dataset. Then, we perform further verification to examine two important aspects: reference reliability – whether the given reference itself contains incorrect knowledge, and safety – whether the examples contain toxic content. For each aspect, specifically, we randomly select 100 samples from FELM, and the authors or crowd-source workers are asked to annotate reference reliability or safety. Results demonstrate that all the examined samples are safe and provided with reliable references. We detail these human verification experiments in Appendix C.
Our experiment below aims to assess several factuality detectors on FELM. We analyze their performance and point out possible usage of FELM.
Factuality evaluators: We consider LLMs like Vicuna, ChatGPT and GPT4 as the backbone models for factuality evaluators, and study various factuality evaluation methods on top of LLMs. Specifically, we first cluster the methods as segment-based evaluators and claim-based evaluators.
In segment-based methods, we directly require the models to assess the factuality of the segments in FELM. In claim-based methods, we first extract a list of atomic fact claims from each segment, and use LLMs to examine these claims – we label a segment factually correct if all claims associated with the segment is correct, and factually incorrect otherwise. We note that this claim-based method is similar to (Min et al., 2023), except that they do not assign segment-level labels. We prompt LLMs to extract claims from a given segment following Min et al. (2023) (details in Appendix D). For both segment-based and claim-based evaluators, we further examine four variants for each of them: (1) vanilla: LLMs make the judgement based on the question and segments (or claims in claim-based methods), (2) chain-of-thought (cot): LLMs are asked to first generate a thought process (Wei et al., 2022) and then make the prediction, (3) reference link: we provide the reference links in FELM for the LLMs to help the assessment, we find this generally helpful since the links themselves often contain helpful information, and (4) reference doc: we access the text corresponding to the reference links and then use the BM25 algorithm (Robertson et al., 2009) to retrieve the most relevant text chunks as additional input to the LLMs. We demonstrate the evaluation setting in Figure 5. Note that there is no reference for math and reasoning domains, and we do not report claim-based performance on these two domains either since the responses often involve multi-step reasoning where strong dependence is present between sentences – self-contained, short atomic fact claims cannot be extracted in these cases. We test three powerful LLMs as the backbone for factuality evaluators: Vicuna-33B (vicuna-33B-v1.3, (Chiang et al., 2023)), ChatGPT (gpt3.5-0301, OpenAI (2022)) and GPT4(gpt4-0314, (OpenAI, 2023)). The evaluation prompts in different settings along with other setup details are in Appendix D.
Metrics: We compute two metrics: the F1 score (along with precision and recall scores) of detecting factual errors and balanced classification accuracy (Brodersen et al., 2010) that balances the positive and negative examples during computing the accuracy. We measure both segment-level and response-level performance. We report F1 scores only in the main content for ease of space, while include the balanced accuracy numbers in Appendix E.
4.2 Experiment Results
FELM is a challenging benchmark: Segment-level and response-level results are shown in Table 4. We observe that the majority of detectors performed unsatisfactorily on FELM, with only the GPT-4 evaluators achieving an overall average of F1 score greater than 40 in some settings. Most ChatGPT detectors did not demonstrate any fact verification ability on FELM without external tools. In addition to attributing to challenges of the benchmark in general, ChatGPT's failure on FELM may be due to the fact that all the errors in FELM are collected from ChatGPT's own generations – it is typically harder for a model to detect factual errors made by itself. Notably, the Vicuna-33B evaluators exhibit commendable F1 performance, outperforming ChatGPT significantly. However, upon closer examination of the balanced accuracy in Table 10, it becomes evident that the Vicuna-33B evaluators still struggle on this task with a balanced accuracy around a random level. Also, we briefly draw comparison with ChatGPT/GPT-4's performance on previous factuality detection benchmarks to better understand the difficulty of FELM. For example, ChatGPT (zero-shot) shows around  $60\% -70\%$  balanced accuracy in diverse summarization factual error detection datasets (Chen et al., 2023). On simpler datasets like SummEval (Fabbri et al., 2021), ChatGPT and GPT-4 are able to make an over  $80\%$  balanced accuracy as demonstrated in Chen et al. (2023). These numbers are generally higher than the ones on FELM as indicated in Table 10, which implies that open-ended factual error detection as in FELM is harder than detecting factual errors from summaries as in previous benchmarks.
Retrieval-augmented methods help: Both the augmentation approaches with reference links and reference document are effective in detecting factual errors. For example, ChatGPT's retrieval-augmented reference document method achieves an average increase of 6.4 points in F1 at the segment level, compared to the Vanilla method. Similarly, GPT-4's retrieval-augmented reference document method achieves a 5.5 point increase in F1 at the segment level. Moreover, the retrieval-augmented content method outperforms all other methods across all domains we tested. Therefore, we can conclude that the retrieval-augmented method is highly beneficial in detecting factual errors.
Is chain-of-thought helpful? Chain-of-thought (Cot) prompting method promotes the performance of GPT-4 on nearly all domains, but it fails to help ChatGPT for all the settings. We think it is attribute to GPT-4 has stronger potential reasoning ability than ChatGPT. Thus the performance can be improved in larger space by chain-of-thoughts method. We further analyze the Cot performance
by utilizing self-consistency (Wang et al., 2022a) in Appendix G, where we show that by applying self-consistency techniques, Cot performance on ChatGPT could be greatly boosted and surpasses the vanilla performance significantly.
Segment-based V.S Claim-based method: Our experimental results highlight clear differences between ChatGPT and GPT-4 detectors. ChatGPT detectors exhibit improved performance when utilizing claim-based segmentation methods, whereas GPT-4 detectors show a decline in performance when assessing claims. For example, the vanilla method experiences a 4.5 point decrease in performance when using claim-based segmentation, as shown in Table 4.
Comparison across the domains: For some domains like world knowledge and reasoning, GPT4 can perform reasonably well with the help of retrieval-augmented methods and chain-of-thought methods. But all the methods are not working well on recommendation and writing domain. After taking a close look at the error cases, we find that it may be because the samples are extremely long, which increases the difficulty to detect sparse factual errors.
In this paper, we introduce FELM, a benchmark to evaluate factuality evaluators. We designed FELM on three principles: 1. Ensuring the authenticity of the factual errors from LLMs; 2. Considering a general factuality definition on five domains beyond world knowledge that most prior works focus; and 3. Conducting segment-level annotations, which enables us to pinpoint factuality errors in a fine-grained manner.
Limitations: While we have invested significant effort in this work, there are still some limitations to our study: (1) we did not explore additional application scenarios, such as code generation, which could be valuable areas for future investigation; (2) due to the difficulty of annotation in FELM, we were unable to collect a larger number of response samples, even though we manage to obtain thousands of segments samples; and (3) the responses in FELM are collected solely from ChatGPT, thus there may exist a potential performance gap when using factuality detectors tested on FELM to detect factual errors of generation from other LLMs. Such a performance gap is not trivial to study without factual annotations of responses from other LLMs. One possible remedy to mitigate this issue is to annotate and add more examples to FELM generated from a diverse range of LLMs in addition to ChatGPT, we leave it as a potential future plan to improve FELM.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333-389, 2009.
Tal Schuster, Adam Fisch, and Regina Barzilay. Get your vitamin c! robust fact verification with contrastive evidence. arXiv preprint arXiv:2103.08541, 2021.
Liyan Tang, Tanya Goyal, Alexander R Fabbri, Philippe Laban, Jiacheng Xu, Semih Yahvuz, Wojciech Krysciński, Justin F Rousseau, and Greg Durrett. Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors. arXiv preprint arXiv:2205.12854, 2022.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 809-819, 2018.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5008-5020, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.450. URL https://aclanthology.org/2020.acl-main.450.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2022a.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions, 2022b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022. URL http://papers.nips.cc/paper_files/paper/2022/bit/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html.
Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. Exploring ai ethics of chatgpt: A diagnostic analysis. arXiv preprint arXiv:2301.12867, 2023.
A Prompt Collection
We collect the prompt from various sources. The details for each domain are as follows:
World Knowledge: We collect prompts encompassing a broad spectrum of world knowledge, including historical events, common sense, news events, culture, and society. A big part of these prompts are sourced from TruthfulQA (Lin et al., 2022), with a smaller portion being contributed by online sources indicated at Section 3. A minor fraction were manually drafted by the authors and by ChatGPT. A few prompts are from hc3 (Guo et al., 2023) and MMLU (Hendrycks et al., 2020). To select prompts from Quora, we randomly chose questions from the History and Society topics. For TruthfulQA, we selected questions from a variety of categories, such as Sociology, Economics, Politics, and Law.
Science and Technology: In this domain, we collect questions about science, technology, and research mainly from Quora, MMLU, and online sources mentioned above, alongside questions generated by ChatGPT and manually designed by us. These prompts vary from examination questions of scientific knowledge to open-ended scientific questions. On Quora, we pick questions from scientific topics such as Scientific Research, Science of everyday life, Technology, and Physics. On MMLU, we select questions from the econometrics, computer security and college chemistry subjects. We also select some questions from the online blogs mentioned above. And we manually design 9 prompts for this domain.
Recommendation and Writing: We use ChatGPT to auto-generate prompts for recommendation. We first draft some prompts as few-shot exemplars (we also include these prompts drafted by authors in FELM), then feed them into ChatGPT to generate more prompts in a self-instruct manner (Wang et al., 2022b). These prompts cover requests for recommending books, online courses, restaurants, and tourist attractions. Writing tasks involve requesting LLMs to generate articles or essays on specified topics. An example prompt is: "Write a dating profile for Mark ACHBAR based on his Wikipedia page". In this domain, we expect that the generated responses are relatively longer compared to other tasks. However, as an auto-regressive language model, ChatGPT would accumulate past errors when generating long textual content. This is why we include writing tasks within the considered domains when evaluating factuality.
Reasoning: Most of the prompts in this domain are from the GSM8K dataset (Grade School 8K) (Cobbe et al., 2021), which is a dataset of more than 8k highly diverse problems. These questions consist of basic numerical problems that require multi-step reasoning. We pick more than 200 challenging questions where the text-Davinci-003 model makes mistakes, as shown in the HELM (Liang et al., 2022) website. In addition, a small part of the prompts are from the online sources and designed by authors.
Math: We collect problems mainly by picking questions from MATH (Hendrycks et al., 2021) where the text-davinci-003 model makes mistakes as shown on the HELM website, similar to how we collect prompts in the reasoning domain. We select questions from algebra, counting, and probability subjects. A small part of prompts are from online sources and authors.
B Annotation Page
We develop the annotation tool as shown in Figure 6. The tool is developed using HTML/JavaScript. The tool is designed for annotators to label the factuality, identify error types, provide reasons for the errors, and include reference links.
C Additional Human Verification
Reference reliability verification: To assess the quality of our provided references, we randomly select 100 samples from FELM, each accompanied by reference links, then we ask the paper authors to assess the reliability of the reference, which measures whether the linked content is free from misinformation or rumors. The results reveal that  $100\%$  of the reference links of the 100 samples are reliable, which implies high reliability of the reference links in FELM overall.
Figure 6: Annotation Page
Safety and validity evaluation: In order to evaluate the safety and overall quality of our GPT-generated responses, we engage the services of Amazon MTurk workers to meticulously evaluate them. Our assessment encompasses two pivotal aspects: Safety and Validity. Safety pertains to ensuring that the responses are devoid of any harassment, sexual, or violent content. On the other hand, Validity centers around confirming that the responses are complete and meaningful to the given prompt no matter whether they are correct or incorrect. We opt for a randomized selection of 100 samples, with each sample being reviewed by three distinct workers for annotation. The workers are paid 0.3 USD for each annotation. The outcomes reveal that the entirety of the 100 responses adhere to safety and validity standards.
D Experimental Details and Prompts
Setup: In our experiments, we use greedy decoding (temperature=0) to obtain the results. And we use gpt-3.5-turbo-0301 and gpt-4-0314 throughout the experiment. We established the maximum token limit at 1500 for claim extraction tasks and 100 for factuality detection tasks. For retrieval-augmented methods that use reference documents, we divided the retrieved documents into 512-token chunks and selected the most relevant chunk using the BM25 algorithm. In cases where there were multiple reference links, we concatenated the retrieved chunks.
**Prompts:** In the following tables, we present the prompts used in our experiments to evaluate the factuality assessment performance of ChatGPT and GPT-4 on world knowledge and writing/recommendation domain. These prompts encompass those used to extract claims from text segments which are shown at Table 5, as well as those used to evaluate the factuality of claims or segments which are shown at Table 6, 7, 8, and 9. We utilized the same extraction and factuality determination prompts for both the world knowledge and writing/recommendation domains, as the response formats are similar in these two domains. This approach allowed us to maintain consistency across both domains, which is important for reliable comparison of the performance of the two models. We use the exact wording of instructions here in our experiments.
Cost: We spend a total of 0.5 USD for generating responses, and 22.04 USD evaluating factuality using ChatGPT (which includes multiple iterations of attempting prompts). We spend 132.3 USD for evaluating GPT4 evaluators.
Prompts for extracting claims of responses
Prompting methods for extracting claims of responses in world knowledge, writing/recommendation domains:I will show you a question and a list of text segments. The text segments can be concatenated to form a complete answer to the question. Your task is to extract factual claims from each text segment.
Here is one example:Question: Tell me about the World Happiness Report.Segments:1. The World Happiness Report is an annual report published by the United Nations Sustainable Development Solutions Network that ranks countries by their level of happiness or subjective well-being.2. The report aims to provide policymakers with information and analysis to help them make informed decisions about promoting happiness and well-being in their countries.
Below are your outputs:Answer:Segment 1:Claim 1. The World Happiness Report is an annual reportCLAIM 2. The World Happiness Report is published by the United Nations Sustainable Development Solutions NetworkCLAIM 3. The World Happiness Report ranks countries by their level of happiness or subjective well-being.Segment 2:Claim 1. The World Happiness Report aims to provide policymakers with information and analysisCLAIM 2. The World Happiness Report aims to help policymakers make informed decisionsCLAIM 3. The World Happiness Report aims to help policymakers promote happiness and well-being in their countries.
Below are my inputs:
Prompting methods for extracting claims of responses in science/tech domain:I will show you a question and a list of text segments. The text segments can be concatenated to form a complete answer to the question. Your task is to extract factual claims from each text segment.Here is one example:Question: What is the diffusion model in computer science?Segments:1. In computer science, the diffusion model is a mathematical model used to simulate the spread of information or data through a network or system.2. It is often used to study phenomena such as the spread of viruses, the adoption of new technologies, or the dissemination of information in social networks.Below are your outputs:Answer:Segment 1:Claim 1. The diffusion model is a mathematical modelCLAIM 2. The diffusion model is used in computer scienceCLAIM 3. The diffusion model is used to simulate the spread of information or data through a network or system.Segment 2:Claim 1. The diffusion model is often used to study the spread of virusesCLAIM 2. The diffusion model is often used to study the adoption of new technologiesCLAIM 3. The diffusion model is often used to study the dissemination of information in social networks.Below are my inputs:
Table 5: A one-shot prompting example to extract claims for the response segments. We use the exact wording of instructions here in our experiments.
Vanilla prompts for factuality detection in world knowledge and writing/recommendation domains
Segment-based Vanilla Prompting for world knowledge and writing/recommendation:
I will show you a question and a list of text segments. All the segments can be concatenated to form a complete answer to the question. Your task is to assess whether each text segment contains factual errors or not.
Please generate using the following format:
Answer: List the ids of the segments with errors (separated by commas). Please only output the ids, no more details. If all the segments are correct, output "ALL_CORRECT".
Here is one example:
Question: What is the total number of nuclear power plants worldwide?
Segments:
1. there were a total of 440 operating nuclear power reactors in the world, with a total installed capacity of over 390 gigawatts (GW).
2. These reactors are located in 30 countries around the world, with the highest number of reactors in the United States, followed by France, China, Japan, and Russia.
Below are your outputs:
Answer: 1,2
It means segment 1,2 contain errors.
Below are my inputs:
Claim-based Vanilla Prompting for world knowledge and writing/recommendation:
I will show you a question and a list of claims. All the claims are extracted from an answer to the question. Your task is to assess whether each claim contains factual errors or not.
Answer: List the ids of the claims with errors (separated by commas). Please only output the ids, no more details. If all the claims are correct, output "ALL_CORRECT".
1. There were 440 operating nuclear power reactors in the world.
2. The total installed capacity of these reactors was over 390 gigawatts (GW).
3. The reactors are located in 30 countries around the world
4. The highest number of reactors is in the United States.
5. France has the second-highest number of reactors.
6. China has a significant number of reactors.
7. Japan has a significant number of reactors.
8. Russia has a significant number of reactors.
Answer: 1,2,3
It means claim 1,2,3 contain errors.
Table 6: Evaluation prompts for one-shot vanilla methods on both segment-based and claim-based settings. We use the exact wording of instructions here in our experiments.
Chain-of-Thought prompts for factuality detection in world knowledge and writing/recommendation domains
Segment-based Chain-of-Thought Prompting for world knowledge and writing/recommendation:
I will show you a question and a list of text segments. All the segments can be concatenated to form a complete answer to the question. Your
task is to assess whether each text segment contains factual errors or not.
Thought: Your reasoning process for the segments with errors. If all the segments are correct, output nothing.
2. These reactors are located in 30 countries around the world, with the highest number of reactors in the United States, followed by France,
China, Japan, and Russia.
Thought: For segment 1, there are only 410 operable power reactors in the world, not 440. And the total installed capacity of these reactors was only 368.6 GW, not 390. For Segment 2, the reactors are located in 32 countries around the world, not 30.
Claim-based Chain-of-Thought Prompting for world knowledge and writing/recommendation :