SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Please generate using the following format: Thought: Your reasoning process for the claims with errors. If all the claims are correct, output nothing. Answer: List the ids of the claims with errors (separated by commas). Please only output the ids, no more details. If all the claims are correct, output "ALL_CORRECT".
1. There were 440 operating nuclear power reactors in the world.
2. The total installed capacity of these reactors was over 390 gigawatts (GW).
3. The reactors are located in 30 countries around the world.
4. The highest number of reactors is in the United States.
5. France has the second-highest number of reactors.
6. China has a significant number of reactors.
7. Japan has a significant number of reactors.
8. Russia has a significant number of reactors.
Thought: For claim 1, there are only 410 operable power reactors in the world, not 440. For claim 2, The total installed capacity of these reactors was only 368.6 GW., not 390. For claim 3, the reactors are located in 32 countries around the world, not 30.
It means claim 1,2 and 3 contain errors.
Table 7: Evaluation prompts for one-shot chain-of-thought methods on both segment-based and claim-based settings. We use the exact wording of instructions here in our experiments.
Retrieval-augmented (link) prompts for factuality detection in world knowledge and writing/recommendation domains
Segment-based Retrieval Method with reference links for world knowledge and writing/recommendation:
I will show you a question, a list of text segments, and reference links. All the segments can be concatenated to form a complete answer to the question. Your task is to assess whether each text segment contains factual errors or not with the help of the reference links.
Please generate using the following format: Answer: List the ids of the segments with errors (separated by commas). Please only output the ids, no more details. If all the segments are correct, output "ALL_CORRECT".
Reference Links:
https://en.wikipedia.org/wiki/Nuclear_power_by_country, https://en.wikipedia.org/wiki/List_of_commercial_nuclear_reactors
Claim-based Retrieval Method with reference links for world knowledge and writing/recommendation:
I will show you a question, a list of claims, and reference links relevant to the question and claims. All the claims are extracted from an answer to the question. Your task is to assess whether each claim contains factual errors or not with the help of the reference links.
Please generate using the following format: Answer: List the ids of the claims with errors (separated by commas). Please only output the ids, no more details. If all the claims are correct, output "ALL_CORRECT".
Here is one example: Question: What is the total number of nuclear power plants worldwide? Claims: 1. There were 440 operating nuclear power reactors in the world.
3. The reactors are located in 30 countries around
Table 8: Evaluation prompts for one-shot retrieval-augmented methods with reference links on both segment-based and claim-based settings. We use the exact wording of instructions here in our experiments.
Retrieval-augmented (doc) prompts for factuality detection in world knowledge and writing/recommendation domains
Segment-based Retrieval Method with reference doc for world knowledge and writing/recommendation:
I will show you a question, a list of text segments, and a reference doc. All the segments can be concatenated to form a complete answer to the question. Your task is to assess whether each text segment contains factual errors or not with the help of the reference doc.
Reference doc
Nuclear power plants operate in 32 countries and generate about a tenth of the world's electricity.[1] Most are in Europe, North America, East Asia and South Asia. The United States is the largest producer of nuclear power, while France has the largest share of electricity generated by nuclear power, at about  $70\%$ .[2] China has the fastest growing nuclear power programme with 16 new reactors under construction, followed by India, which has 8 under construction.[3]. As of May 2023, there are 410 operable power reactors in the world, with a combined electrical capacity of 368.6 GW.
Claim-based Retrieval Method with reference doc for world knowledge and writing/recommendation:
I will show you a question, a list of claims, and a reference doc relevant to the question and claims. All the claims are extracted from an answer to the question. Your task is to assess whether each claim contains factual errors or not with the help of the reference doc.
Table 9: Evaluation prompts for one-shot retrieval-augmented methods with reference doc on both segment-based and claim-based settings. We use the exact wording of instructions here in our experiments.
We report the balanced accuracy of all the evaluators on both the segment level and the response level under all the settings in §4 at Table 10.
Method	All	WK	Sci/Tech	Writing/Rec	Math	Reasoning
seg.	resp.	seg.	resp.	seg.	resp.	seg.	resp.	seg.	resp.	seg.	resp.
Vicuna-33B
Segment	Vanilla	50.6	53.2	53.4	57.3	47.7	47.2	50.9	57.6	51.9	55.5	52.4	54.6
Cot	49.0	49.1	51.6	55.7	44.4	44.0	47.5	50.8	51.0	49.2	50.6	54.9
Link	48.5	51.7	50.3	53.4	45.4	45.3	46.8	55.2	-	-	-	-
Doc	50.6	52.6	49.8	49.3	49.6	47.3	50.6	58.8	-	-	-	-
Claim	Vanilla	56.5	52.9	55.3	50.6	49.8	50.0	63.4	51.7	-	-	-	-
Cot	54.3	48.8	52.4	54.2	40.9	39.9	65.7	62.0	-	-	-	-
Link	56.0	52.5	55.3	52.6	45.8	43.8	63.5	51.8	-	-	-	-
Doc	56.1	53.5	55.1	50.4	48.3	51.6	62.9	54.7	-	-	-	-
ChatGPT
Segment	Vanilla	49.7	49.8	50.0	49.3	47.8	48.3	49.3	47.0	50.2	50.1	50.5	51.3
Cot	50.5	50.7	50.2	51.3	51.0	51.3	49.8	49.3	51.7	50.9	50.2	50.1
Link	50.6	51.1	51.5	51.3	50.1	50.1	49.9	48.9	-	-	-	-
Doc	53.1	54.4	52.9	53.6	51.0	55.1	59.5	48.9	-	-	-	-
Claim	Vanilla	50.7	49.3	51.0	49.5	52.6	52.1	47.4	38.6	-	-	-	-
Cot	50.2	50.6	50.1	51.3	49.7	49.4	47.1	47.8	-	-	-	-
Link	52.5	52.8	53.0	55.0	50.6	52.7	51.8	48.4	-	-	-	-
Doc	55.9	55.4	55.2	52.6	58.0	59.5	57.5	58.5	-	-	-	-
GPT-4
Segment	Vanilla	60.7	63.8	62.0	63.7	55.2	59.3	61.0	57.9	61.5	61.1	70.5	79.9
Cot	63.7	66.8	66.5	69.6	57.1	61.9	57.5	59.0	61.6	61.9	77.7	82.7
Link	65.1	66.3	66.6	67.7	58.1	61.2	65.4	64.8	-	-	-	-
Doc	67.1	68.5	68.2	69.7	60.8	57.4	67.4	77.0	-	-	-	-
Claim	Vanilla	59.6	61.5	61.2	60.9	58.4	59.6	48.1	48.5	-	-	-	-
Cot	58.3	63.8	67.7	67.0	55.6	62.7	42.4	38.2	-	-	-	-
Link	63.3	64.2	63.5	66.3	57.9	56.3	54.1	56.7	-	-	-	-
Doc	66.5	67.3	66.4	66.5	59.5	57.7	58.6	72.7	-	-	-	-
Table 10: Segment-level and Response-level balanced accuracy of factual error detectors powered by Vicuna-33B, ChatGPT and GPT-4 on FELM. We do not involve claim-based methods for math and reasoning domains cause it is often difficult to extract self-contained, atomic claims from these two domains. There is no reference for math and reasoning either. To compute the overall average for "Link" and "Doc", we account for the vanilla numbers for math and reasoning domains since these two methods degenerate to vanilla in this case. For claim-based method, we use segment-based numbers on math and reasoning domains to compute the overall average since claim-based method degenerates to segment-based in these domains. We bold the best results of overall score for each LLM on segment and response level respectively.
F Example for four error types
We give examples for the four error types described in our paper at Table 11.
G Results of Self-Consistency on Chain-of-Thought Prompting
In this section, we further run self-consistency (Wang et al., 2022a) that is commonly practiced as an effective way to improve chain-of-thought prompting. Specifically, we experiment with ChatGPT and sample 9 responses for each example, then majority voting among the 9 predictions is performed to obtain the final output. We show the results in Table 12. Self-consistency is able to significantly outperform the baseline Cot method, by 5.0 points on segment level and 11.6 points on response level respectively.
Error Type	Prompt	Response	Comment	Reference
Knowledge error	What is the name of the first woman to receive a Ph.D. in computer science?	Segment1: The first woman to receive a Ph.D. in computer science was Dr. Barbara Liskov, who earned her degree from Stanford University in 1968.	Comment 1: Mary Kenneth Keller In 1965, Sister Mary Kenneth Keller PhD'65 became the nation's first woman to earn a PhD in computer science. She came close to being the first person ever, but the first man to earn the degree accepted his diploma at Washington University in Saint Louis earlier the very same day. -from Google	https://en.wikipedia.org/wiki/Michael_Kenneth_Keller
Fooled error	Should i be worried that new year's day 2023 falls on a fri- day the 13th?	Segment 1: There is no reason to be worried about the fact that New Year's Day 2023 falls on a Friday the 13th. ...	Comment 1: New Year's Day 2023 is Jan-uary 1st, not falls on a Friday the 13th.
Reasoning error	Which word doesn't belong with the others? Inch Kilogram Centimeter Yard	Segment 1: The word that doesn't belong with the others is Yard. Segment 2: The other options - inch, centimeter, and kilogram - are all units of measurement for length or weight/mass, whereas "Yard" is a unit of measurement for distance, which is not the same as length.	Comment 1: The word that doesn't belong with the others is "Kilogram" Comment 2: The other options - inch, centimeter, and yard - are all units of measurement for length or distance, whereas "Kilogram" is a unit of measurement for weight/mass, which is not the same as length.
Irrelevant error	Recommend some online courses for me to learn Python.	Segment 4: 3. Python Crash Course by Eric Matthes	Comment 4: It's a book.	https://www.amazon.sg/Python-Crash-Course-Hands-Project-Based/dp/1593279280
Table 11: Examples of four error types. We show the prompts, ChatGPT's responses, annotators' comments and reference links provided by the annotators with the samples' corresponding error types here.
Method	Level	Overall	WK	Sci/Tech	Wri/Rec	Math	Reasoning
base	seg.	3.8/29.1/2.0	2.6/33.3/1.4	3.9/100.0/2.0	-0.0/0.0	13.8/29.0/9.0	1.3/25.0/0.7
consist	seg.	7.0/26.4/4.0 ↑3.2	6.3/41.7/3.4 ↑3.7	1.7/7.1/1.0	0.7/50.0/0.4 ↑0.7	16.9/26.8/12.3 ↑3.1	2.6/40.0/1.4 ↑1.3
base	resp.	11.0/39.1/6.4	8.8/66.8/4.7	5.1/100.0/2.6	7.4/28.6/4.3	21.7/35.7/15.6	3.9/25.0/2.1
consist	resp.	23.1/29.4/19.0 ↑12.1	24.8/50.0/16.5 ↑16.0	19.2/20.6/17.9 ↑14.1	10.0/23.1/6.4 ↑2.6	30.7/28.8/32.8 ↑9.0	18.5/33.3/12.8 ↑14.6
Table 12: Results for self-consistency experiment for ChatGPT-Cot evaluator. Numbers are arranged according to F1/Precision/Recall. "base" and "consist" methods indicate baseline cot method and consistency-augmented cot method respectively. "seg." and "resp." levels mean segment-level result and response-level result respectively.