Evaluating LLMs at Detecting Errors in LLM Responses
Authors: Ryo Kamoi
Sarkar Snigdha Sarathi Das
Renze Lou
Jihyun Janice Ahn
Yilun Zhao
Xiaoxin Lu
Nan Zhang
Yusen Zhang
Haoran Ranran Zhang
Sujeeth Reddy Vummanthala
Salika Dave
Shaobo Qin
Arman Cohan
Wenpeng Yin
Rui Zhang
Venue: COLM (2024)
Keywords: error detection,evaluation
TL;DR: We introduce the first error detection benchmark consisting of objective, realistic, and diverse errors made by GPT-4 and Llama 2 70B, and evaluate 11 LLMs on the error detection task.

Abstract: With Large Language Models (LLMs) being widely used across various tasks, detecting errors in their responses is increasingly crucial. However, little research has been conducted on error detection of LLM responses. Collecting error annotations on LLM responses is challenging due to the subjective nature of many NLP tasks, and thus previous research focuses on tasks of little practical value (e.g., word sorting) or limited error types (e.g., faithfulness in summarization). This work introduces ReaLMistake, the first error detection benchmark consisting of objective, realistic, and diverse errors made by LLMs. ReaLMistake contains three challenging and meaningful tasks that introduce objectively assessable errors in four categories (reasoning correctness, instruction-following, context-faithfulness, and parameterized knowledge), eliciting naturally observed and diverse errors in responses of GPT-4 and Llama 2 70B annotated by experts. We use ReaLMistake to evaluate error detectors based on 12 LLMs. Our findings show: 1) Top LLMs like GPT-4 and Claude 3 detect errors made by LLMs at very low recall, and all LLM-based error detectors perform much worse than humans. 2) Explanations by LLM-based error detectors lack reliability. 3) LLMs-based error detection is sensitive to small changes in prompts but remains challenging to improve. 4) Popular approaches to improving LLMs, including self-consistency and majority vote, do not improve the error detection performance. Our benchmark and code are provided at https://github.com/psunlpgroup/ReaLMistake.

Published as a conference paper at COLM 2024
Evaluating LLMs at Detecting Errors in LLM Responses
Ryo Kamoi', Sarkar Snigdha Sarathi Das', Renze Lou', Jihyun Janice Ahn Yilun Zhao?, Xiaoxin Lu', Nan Zhang', Yusen Zhang', Ranran Haoran Zhang' Sujeeth Reddy Vummanthala', Salika Dave', Shaobo Qin
Arman 2 Wenpeng Yin Rui Zhang'
Penn State University, Yale University, SSony Brook University, *Allen Institute for Al (ryokamoi, rmz5227 (155227@@sane @psuu.due.. edu
Abstract
With Large Language Models (LLMs) being widely used across various tasks, detecting errors in their responses is increasingly crucial. However, little research has been conducted on error detection of LLM responses. Collecting error annotations subjective nature of many NLP tasks, and thus previous research focuses on tasks of little practical value (e.g., word sorting) limited error types (e.g., faithfulness in summarization). This work introduces RisLMis the first error detection benchmark consisting of objective, realistic, and diverse errors made by LLMs. ReaLMistake contains three challenging and meaningful tasks that introduce objectively assessable errors in four cate- gories (reasoning correctness, instruction-foolloinng context- context-faithfulness. and parameterized knowledge), eliciting naturally observed and diverse errors in responses of GPT-4 and Llama 2 70B annotated by experts. We use Real.Mistake to evaluate error detectors based on 12 LLMs. Our findings show: 1) Top LLMs like GPT-4 and Claude 3 detect errors made by LLMs at very low recall, and all LLM-based error detectors perform much worse than humans. 2) Explanations by LLM-based error detectors lack reliabil- ity. 3) LLM-based error detection is sensitive to small changes in prompts but remains challenging to improve. 4) Popular approaches to improving LLMs, including self-consistency and majority vote, do not improve the error detection performance. Our benchmark and code are provided wwwpp//u..ggrrrooeeeeeti........
LLM responses is challenging due to the
https://githur cwtiaa...om/pptite
Introduction
As LLMs have been increasingly used in real-world applications, it is critical to develop methods for automatically detecting errors in responses from LLMs (Bommasani et al. 2021
LLM banchmarks otten incluce Subjective Tasks
LLM posed
ReaL.Mistake consists of Realistic and Objective Tasks
Eriy Desestor
basse Enrr Dellcttor
Input
Resporse
Binary Detector
Input
Response
Emro Detector
Compose an engaging travel
pleasure of vistings
recently had tho
math
compleses laps
blog post about recent trp
Hawal and quidkly became
Human Amnotators
lollwing reguieemens..
Human Ammtttions
Heweli...
one my tavorie
pacee.
not sure it suffi-
Not interioting but
understanding of relative speed diroular track
The problem recuites an
long, many slometeess
Marta
The quastion not invove reliive speed.
cerdy unangaging
The respanse includss erroet
(Task MT-Bench)
classity ns emor...
Cannot evaluate Error Detector...
Human annotators objective error label for evalueting error detectors!
Error Detector
cannet
wrron!!
objectve
Figure 1: Left: Tasks in existing LLM evaluation benchmarks are often subjective and not suitable for collecting errors made by LLMs for the purpose of evaluating binary error detection methods. Right: We introduce the ReaLMistake benchmark with realistic, objective, and diverse errors made by LLMs for evaluating error detection.
1

---

Published as a conference paper at COLM 2024
Math Word Problem Generation
Fine-gralined Fact Verification
Answerability Classification
Generato a moth word probicm that satiriccs the tot lowinng teep inmmerts
Widree Lantzohin En- gland, snd te cled aged IS...
Cleam: Adwre
Asseume you are on Jan 16, 2018 Your reasett ing shoule not includs any toctual oncrs. th queecion includess inaomeet infommtion.... ... dleary state that the queston unanswerasts.
Input LLMs
The sted quesdion shoulld bw solid and le-
Exiterpe:
soeratie
lin Misk dcter disce agedl its..
The problem requires on understa dinn relative
and Mics Adame was Great Britin intemaaion and lommer caplasn al Walreee..
Chuestann: Surnng wiich ars dil the composen: for whon the La crater ondierr in ramedt composer?
speed limw in croater
Requirements introduces
Factual Mistakes introduces Parameterized Knoweldge Tasks also ask LLMs to provide Explanations, which introduce errors in Reasoning
Context introduces
errors Instruction-Followw
errors FolteuttFalthfulnnsss
Response
Question: Maria complenes 32 laps around the track
Marta long,
Indeed born In Widnes, Engrand (Ine 0,14).
evidence confrrinng
Adams
The banwegian compose for whom the Greg ceher
Gring Greg
Howeres, the avidenaa daas not sroecl any iilo-
masion sbout where Adame died... Thersfore,
Butation:: Firat, need find oul....
not clear whether the claim supported not.
I Arawwen: The Romantic
Annotation by Experts
Dinary Error Labal:
Enor
Binary Error Label:
Error Context Falthtunness and Reasoning Correctness
Binary Enor Label:
Error
Error Categories:
Instruction- Inttrction-Folixiingg Following
Error Categories:
Error Categorins:
Parameterized Knowledge
The question Invoive relatvv speod.
The conckuzion should be "nat supporsuds
Grieg ciater
Figure 2: Examples of three tasks in ReaLMistake with four error categories. Each instance includes a binary error label, error categories, and annotator's 's explanations about errors on response from GPT-4-0613 Llama 2 70B. [xppendixxprovviess full details.
No Subjective Cnterra
Extor Categorias
Benchmark
Task
Medills
Daba
Label
Reasening Corretness
Instraction- Following
Canted- Fathifulness
Patamelettadd ledgl
MTT-Bench, P
LLMDval/2
Multipli NLP Tasks
Multiple LLMs
Ranking (Paiwsse)
Wikibo GPT-3 Manakui al. 2003
Whsspedia Generation
GPT-3
238
(Puint ise)
Laban
SuimmFiitts
Summarization
GPT-35 Turbo
6,348
Rirarry (Fointwise)
RI Bench Mistake Iyen al 2:21
(eg, word sorting)
Logical Taoks
PaLM2
2,778
Rirurr (Taintwise)
RosLMistake (Ous))
3
G2T- (0613))
Bisury
Different Domains
2
(Pointwise)
Table 1: Comparison between ReaLMistake and prior benchmarks for evaluating error detectors or evaluators for LLM responses. There is a deficiency in benchmarks for evaluat- ing pointwise evaluation, although there are diverse benchmarks for evaluating pairwise evaluation (e.g..ZZeng. et al. [2023 Dubois et al., 2023 Zhang et al.) 2023). ReaLMistake is the first benchmark that includes objective and realistic errors in LLM responses in diverse categories.
Srivastava al., 2022). However, there is a deficiency in an analysis specifically targeting error detection (Huang et al. 2024 Tyen et al. 2024).
An obstacle in studying error detection is the lack of benchmarks that include binary error annotations (i.e., whether the response contains errors or not) on objective, realistic, and diverse errors made by LLMs| An NLP task should satisfy the following criteria to be suitable for collecting errors made by LLMs for the purpose of benchmarking error detection. First, to provide objective error labels, tasks should not involve subjectivity or am ambiguity. As illustrated in Figure [ subjective tasks on which humans cannot provide reliable error annotation are not suitable for benchmarking error detection methods. For example, in text summarization, humans cannot objectively annotate binary error labels because the evaluation of "whether the generated summary properly includes important content in the source" (the "relevance" criterion) is subjective (Laban et al. 2023). It is difficult to evaluate error detectors if humans cannot determine whether LLM responses include errors. As shown in Table [ tasks in benchmarks for ranking- based LLM evaluators (Zheng et al., 22023] Wang et al. 2023a 2024) and -riteiion-specific error detectors (e.g., factuality evaluators)
(Laban al. 2023: Manakul et al. 2023) often involve subjectivity (Zeng et al. 2024). Second,
In this paper, "objective errors" represent errors in LLM responses that do not involve ambiguity or subjectivity in the detection process and can be detected in binary manner with a high inter-annotator agreement.
2

---

Published as a conference paper at COLM 2024
error detection benchmarks should be built tasks that make LLMs introduce errors similar to errors in real-world applications. Some prior studies use tasks such as word sorting (Tyen et al. 2024 and chess (Wu et al. 2024), which are objective but have different properties from practical uses of LLMs. Third, to provide a comprehensive testbed, the benchmarks need to include variety of tasks to provide errors in diverse categories. Previous research in detecting mistakes in LLM responses focuses on errors of limited types
(Laban et al. 2023 Manakul et al., 2023 Lightman et al., 2024), as shown in Table I Finally, tasks should be challenging for strong
LLMs to make them introduce errors.
To create tasks that satisfy these requirements, we propose an approach to design tasks so that they make LLMs introduce errors detected by objective, realistic, and diverse evalu- ation criteria. First, we identify four evaluation criteria (Reasoning Correctness, Instruction- Following, Context-Faithfulnesss and Parameterized Knowledgee which can be objectively evalu- ated by humans and cover diverse errors in LLM responses. Subsequently, we create three tasks (Math Word Problem Generation, Fine-grained Fact Verification, and Answerability Classifi- cafion) with the intention of making LLMs introduce errors detected by the four evaluation criteria, eliminating subjectivity from the error annotation process. This process ensures that the created tasks have desirable properties for collecting mistakes in LLM responses for error detection benchmarks. Using these three tasks, we introduce MiatMist the first benchmark for evaluating error detection methods for LLM responses, consisting of ob- jective, realistic, and diverse errors made by LLMs. As shown in Figure our benchmark includes error annotations (binary error label, error categories, and human explanation about errors) on responses from GPT-4 (OpenAl 2023) and Llama 70B (Touvron et al., 2023) on the three tasks. The annotation process requires careful checking of the entire LLM responses, and 14 expert annotators spent 90 hours to provide high-quality annotations.
By using ReaLMistake, we analyze the error detection performance of 12 LLMs (7 open- source and 5 closed-soue -source models). Our findings show: (a) In the experiments on LLM- based error detectors with zero-shot prompting (Kojima et al., 2022), top LLMs like GPT-4 and Claude 3 Opus detect errors made by LLMs at very low recall, and all models perform much worse than humans the error detection task in ReaLMistake, (b) The performances of the error detectors on the three tasks in MistMae have different trends, indicating that Mistake provides error detection tasks with diverse properties. (c) Our manual analysis shows that explanations provided by the LLM-based error detectors are unreliable, and open-source LLMs often provide wrong reasoning even when the binary predictions are correct. (d) Our analysis of four different types of prompts for LLM--asee error detection shows that error detection performance is sensitive to small differences in prompts. (e) Finally, to improve error detectors, we evaluate three methods motivated by popular approaches: self-consistency (Wang et al., 2023b), using multiple LLMs (Cohen et al. 2023 Chan et 2004), and providing evaluation steps in prompts (Liu et al. 2023a). However, we do not observe improvement in the error detection performance. Our analysis indicates that ReaLMistake provides challenging and diverse error detection tasks, and further research is needed to improve LLM-based error detectors for LLM responses.
2 Related Work
Ranking-based Evaluation. Evaluation of LLMs has been mainly studied in ranking- based (Chen et al. 2023c Dubois et al.) 2023 Zheng et al., 2023 Wang et al., 2023a 2024 Zeng et al. 2024) or Likert scale (Chiang & Lee 2023) since recent LLMS otten do not make obvious mistakes and many NLP tasks involve subjectivity. These evaluation approaches are useful for system-level comparison of LLMs but not suitable for detecting erroneous responses. Binary error detection is motivated by a real-world situation in which we want to automatically detect errors in LLM responses to reject improve bad outputs.
Self-Correction. Error detection is an important task in its own right, but it has been mainly studied in the context of improving LLM responses using feedback from LLMs (Bai et al. 2022 Madaan et al. 2023 Shinn et al. 2023 Kim et al.) 2022 Chen et al.) 2023a Gou et al. 2024 Pan et al. 2024 Kamoi et al. 2024) These studies often evaluate the performance
3

---

Published as a conference paper at COLM 2024
Resporse
Model
Average # tokens
Errors in Responses from GPT-4 Llama 70B [%]
Task
# Data
L.M Response
Reasaning Correctness
Instruction- Following
Contest- Faithfalness
Parameterized Knowledge
Total
Input
Eroor
GPT44
0613
Fine-grained Fact Verification
Math Word Problem Generation
140
523
252
151
83
25.0
25.7
57.1
45.0
62.1
62.9
Answerability Classification
140
119
75
22.1
8.6
40.7
62.1
Liama
Math Word Problem Generation
160
235
163
51.2
67.5
80.0
70B
Fine-grained Fact Verification
160
511
168
36.9
44.4
45.6
80.6
Answerability Classification
96
48.1
48.1
81.2
Table Statistics of our ReaL.Mistake benchmark.
of the refined outputs on downstream tasks and an analysis of the binary error detection step is lacking.
3 ReaLMistake
The MeatMistake benchmark includes 900 instances consisting of error annotations by experts (binary error label: error or no error, error categories, and human explanation) on responses from GPT-4-0613 (OpenAIl 2023) and Llama 2 70B (Touvron et al. (2023) on three tasks designed for providing objective error labels in diverse error categories, as shown in Table 2 Figure provides examples from the three tasks in our benchmark.
3.1 How to Collect Objective, Diverse, and Realistic Errors in LLM Responses?
The main challenge in creating evaluation benchmarks for error detection is in the process of collecting errors in LLM responses. There are various requirements for tasks used to collect errors in LLM responses for the purpose of evaluating error detection. First, human annotators should be able to provide objective error labels for LLM responses on the tasks to create a reliable testbed of error detection, as shown in Figure addition, the tasks should introduce diverse types of errors observed in real-world applications of LLMs. Finally, the tasks should be difficult even for recent LLMs make LLMs introduce mistakes. However, tasks should not be too difficult for humans to allow annotations and detailed analysis of error detection. We propose a strategy to construct tasks that satisfy these requirements.
(1) Create Tasks that Make LLMs Introduce Objective, Realistic, and Diverse Errors. Evaluation of NLP tasks often involves subjectivity and even humans cannot provide objective binary error labels. For example, the text summarization task involves subjectivity in the "relevance" criterion, which evaluates "whether important content in the source is properly selected" (Laban et al. 2023 .. To put it the other way around, humans can provide reliable error labels it tasks can be evaluated by objective criteria. To create tasks that make LLMs introduce objective, realistic, and diverse errors, we propose a bottom-up approach to design tasks so that they can be evaluated using diverse and objective evaluation criteria. We identify the following four criteria, which comprehensively cover objective criteria about errors in LLM responses that are caused by components of the inference process of LLMs: instructions and context inputs, reasoning by LLMs, and knowledge in parameters. By creating tasks so that they can be evaluated by the following criteria, LLM responses to the tasks include realistic and diverse errors that can be objectively detected by humans.
Reasoning in responses (e.g., chain-of-thought (Wei et al. 2022b) should be logically valid. This criterion is objective because logical correctness is well- defined. Prior studies also annotate logical mistakes in reasoning by LLMs
Reasoning Correctness:
binary manner (Ling et al. 2023 Miao et al., 2023 Tyen et al., 2024 Lightman et al. 2024).
Instructionn Responses should follow the requirements specified in inputs 2023a Zeng et al. 2024 Qin al. 2024 Instruction-followingg is an objective
Zhou al.
criterion if the requirements are objective (Zhou et al. 2023a).
AAlthough the term "instruction-folloiinng is used in various scopes Wei et al. 2022a Ouyang
al. 2022), we use definition that focuses the requirements explicitly mentioned in prompts.
4

---

Published as a conference paper at COLM 2024
Math Word Problem Generation
Fine-grained Fact Verification
Answerability Classification
Mam Word Problem
Cruin Aderre
Web Actice Winapedia
Indics of Bentanova
Walpedka based
Panagraphe
Dataset
Base
AQUA at the time hics starte walking around Dutaset the same rack. Maria completes Se lops
and Nic comp boee 12 lape
Dataset
VICE
bom Widnes,
Lancadditee
Lie O, 14
HotpotOA
Dataset
driog craae nemsal sod for coescoad dhing our
dring joones
Fivard Galeg
Artcka
Creation
Process
Dalaset
dendannding of relatvv speed
in
The problem requinss an un
The sotution imchee
Select
GPT4
procortes
Careraada
Propmott
Past sova the supporting senterocs randority onlected sententedd
Cicate
verticatioo
GPT4
information ina
the queetton
introuuce weong
the qussiiion
Fix
Proprngi:
Solect knoo
cut-oal delm eauth
That the prob em
unonswe able
may beecoone
chuckk all pa eous of information
the cla and
Ca warte mails ward problem trat sutiafiiu the Waowing requrnmentt
ocon whether each part of the casm supporeed
Ascome you are Jan 2018
Created Task
Ghem: Adiaroo was burn in Widhowe, Lantaodiie Englans, and ho died aged 65
Quaes During which wns did the Nurwastin composor. whom the Gring crater on rammed, composss?
The protdemn requi unoworterdinng of relative
spood and time a diouuaar trock.....
Eviclence::
Inn O: Mick/domm dies, aged 56...
Figure 3: Creation processes of three tasks in Miatais Details are in Appendix B
Context-Faithfulness: Responses should be faithful to the context provided in inputs and should not ignore any part of the context. If the input instructs not to use any resources other than the provided context, the response should not be affected by parameterized knowledge (Zhou et al. 2023b). This criterion is often used
objective "factuality" criterion for text summarization (Fabbri et al. 2021 Cao & Wang 2021 Laban et al.) 2023).
Parameterized Knowledge: Responses should be factually correct. In this work, we focus on parameterized knowledge without allowing LLMs to access external resources. Prior studies also annotate mistakes in parameterized knowledge in a binary manner (Manakul et al. 2023
Min et al. 2023), supporting the objectiveness of this criterion.
(2) Create Tasks Challenging for LLMs but Feasible for Humans. We also highlight the difficulty in creating challenging tasks for recent strong LLMs. Classic objective tasks, such as mathematical reasoning at the grade school level, are too easy for recent LLMs (OpenAI
2023 Anthropic 2024) and cannot collect mistakes. However, tasks that involve expert
know ledge (Malaviya et al. 2023 Rein et al. 2023) or very long context (Wu et al. 2021 Pang et al. 2022) are not suitable for our purpose because is too difficult for human annotators and users of this benchmark to conduct an analysis of error detection methods. Since it is
challenging to create difficult tasks, some datasets include artificially created negative cases created by prompting LLMs to intentionally make mistakes (Zeng et al. 2024 Li et al., 2023a Laban et al., 2023), but artificial mistakes cannot evaluate error detection in realistic settings.
To address these problems, we design tasks involving properties that are observed to be challenging for LLMs: (a) following requirements (Liu et al. 2024 Zhou et al., 2023a), (b) comparison of multiple texts (Laban et al.) 2023: Chen et al. 2023b), and (c) detection of small mistakes in text (Cohen et al. | 2023). As shown in Table tasks in our benchmark make GPT-4 introduce errors in more than 50% of cases, while their inputs consist of few hundred words that only involve high-school-level mathematics and knowledge in
Wikipedia. These tasks allow us to efficiently collect errors made by LLMs.
3.2 Tasks
Our benchmark includes three challenging, objective, and realistic tasks designed for col- lecting diverse categories of errors in LLM responses. As discussed in the previous section, these tasks are designed so that they can be evaluated by the four objective and diverse criteria, eliminating the subjectivity in the error annotation process while ensuring the diversity. In addition, the three tasks are selected to involve skills required in real-world applications of LLMs (e.g., arithmetic reasoning, fact verification, and factual knowledge), in contrast to the objective tasks in prior work that have properties different from those of real-world uses of LLMs, such as chess (Wu et al. 2024) and word sorting (Tyen et al. 2024). Figure inncuuess
includes an example from each task. Figure visualizes the creation processes.
MathGen Math Word Problem Generation is task of generating math word problems that follow multiple requirements. This task is designed to be evaluated on Reasoning Correctness and IndiructioonFollowiiy This task instructs models to generate question that
5

---

Published as a conference paper at COLM 2024
Category of Requirements in MailiGei Inputs
%
The problem requires an understanding of the concept of
54.2
The problem should include [price number of...
Rumbe
problem should include (specific number]
Noother
Whong Infermation
The problem should include the phrase
23.7
include any other type number
The problem should include [integers, fractions, etc] and should not
25.3
Pape how
Aptwerab's
The solution should involve [addition, linear equation, etc.]
60.0
The solution should include the phrase "..."
18.9
Informato
Insutiiiee
The solution should include the question"... "..".
15,8
The solution should include lintegers, fractions, etc.] and should not include any other type of number
63.2
Figure 4: Distribution of questions in Answerability Classification task. This task includes diverse types of unan- swerable questions.
Table 3: Math Word Problem Generation task consists of diverse requirements in 9 categories.
follows all requirements in input (Instruction-Follouinn) and also instructs to generate solution for the generated question (Reasoning Correctness). Creating valid mathematical questions is challenging for LLMs and allows us to collect mistakes by LLMs without using advanced mathematical concepts, which are too difficult for annotators and users. Creation Process: We generate the requirements in inputs by using questions in AQuA (Ling et al., 2017). Specifically, we use GPT-4 to generate nine properties (e.g,, topics, mathematical concepts, types of numbers; listed in Table 3 of each pair of question and solution in A QuA and select two to four properties as requirements task. To make the task objective, we only include objective requirements.
the math word problem generation
FgFactV Fine-grained Fact Verification is a task of checking whether each piece of information in claim sentence is supported by the provided evidence. Fine-grained fact verification that instructs to provide detailed explanations has not been explored and similar tasks have been reported to be challenging for LLMs (Fabbri et al. 2021). This task is designed to be evaluated on Reasoning Correctness and Confext-Faithful instructs the models provide reasoning (Reasoning Correctness) on whether each piece of information in the claim is supported by the evidence or not Context-Faithfulness sure that annotators can objectively evaluate LLM responses in a binary manner, we provide detailed instructions such "check all pieces of information in the claim and state reasoning on whether each part of the claim is supported". Creation Process: We use WiCE (Kamii al. 2023 as a base dataset, which includes factuality classification task. We reconstruct this task info fine-grained verification, which instructs models to explicitly check all pieces of information in the claim. Since WiCE includes very long web articles evidence, we retrieve part of them to make the model inputs in our task shorter.
This task
To make
AnsCls Answerability Classification is task of classifying the answerability of factual questions. Answerability classification is popular task (Rajpurkar et al.) 2018 K Kwiatkowski et al. 2019), but the performance of LLMs challenging cases unexpl lored. Our task instructs the models to answer the provided question, but they should classify the questions as unanswerable if the questions include factual errors. Unlike classic datasets that include entirely unanswerable erable questions, our task requires detecting small factual mistakes. This task is designed to be evaluated on Reasoning Correctness and Parameterized Knowledge. This task instructs models to reason whether the questions are answerable (Reasoning Correctness) by checking the factual correctness of questions (Parameterized Knowledge). Creation Process: We use HotpotQA (Yang et al. 2018) as base dataset. To create unanswerable questions, we (1) add a requirement to use knowledge before a specific date to make the questions time-sensitive (Chen et al., 2021) and (2) use GPT-4 to introduce factual mistakes in the numbers, nouns, and modifiers in the questions. By using multiple prompts to introduce mistakes, create unanswerable questions in diverse categories, shown in Figure

---

Published as a conference paper at COLM 2024
3.3 Annotation
14 faculty and graduate students participate in the annotation process. We train the anno- tators on multiple practice annotations. Each instance is annotated by one annotator and verified by another annotator. The tasks in our benchmark require annotators to carefully read the whole text in inputs and responses, and the annotations take about 90 hours in total (6 minutes for each case). Details are in Appendix [ Filtering: We remove pairs of input and output that involve errors detected by other criferia (e.g., grammar mistakes) or subjectivity in determining whether the response is erroneous or not. However, this process removes only 3.8% of cases, showing that our tasks are properly designed to be evaluated objectively by the four criteria. Human Agreement: To assess the human agreement and annotation quality, we evaluate the expert performance on 35 instances from each dataset (35 x 6 instances in total). The average performance of three annotators on the three tasks was 95.7 in F1, verifying that humans can annotate binary error labels with high agreement.
4 Error Detection Performance on ReaLMistake
We evaluate 12 LLMs on the error detection task in ReaLMistake. The questions addressed
in this section include: (1) How well can LLMs detect errors made by LLMs? (2) Can LLM-based error detectors provide reliable explanations for mistakes in LLM responses? (3) How can we improve LLM-based error detectors? and (4) Does ReaLMistake include challenging and diverse error detection tasks?
The LLM-based error detectors evaluated in our experiments use the same prompts of "classify whether the response includes errors" for all tasks, differing from criterion-specific detectors in prior work, such as factuality evaluators. As in Figure [ evaluation of the riterra-indepenneen detectors enabled by the tasks in ReaLMistake,, which do not involve subjectivity in the evaluation. In contrast, for example, the text summarization task cannot objectively evaluate criteria-inde indeeeddeinneeeddeett detectors because they involve the subjective "relevance" criterion, although it can objectively evaluate factuality evaluators.
LLMs for Detectors. Open: Gemma 7B (Google 2024), Llama 2 (13B and 70B) (Touvron
et al. 2023), Mistral 7B (Jiang et al., 2023), Mixtral 8x7B (Jiang et al. 2024), Qwen 1.5 (14B
and 72B) (Bai et al. 2023 Qwen leam 2024), Closed: Gemini 1.0 Pro Gemini Team Google
2023), Claude 3 Opus (Anthropic 2024), GPT-3.5 Turbo (Brown et al. 2020 Ouyang et al.
2022 and GPT-4 (2023-0613 and 2024-0125) (OpenAl 2023). Details are in Appendix H
Prompts. We use four zero-shot prompts (Kojima et al.) 2022 for error detection that have variations in two properties: different wordings and flipped order of the binary error label options (Section 4.3 Figure 77.. In Table 4 we the average performance of the four
prompts to alleviate the influence of biases caused by the prompt design.
4.1 Strong L.Mss Detect Errors in LLM Responses at Low Recall
Table 4 shows the error detection performance of 12 LLMs with zero-shot prompts on Rea Mistaaee It shows that LLM-based error detectors often perform worse than random baselines in F1, showing that the error detection task in ReaLMistake is difficult even for strong LLMs such as Claude 3 and GPT-4. The results also show that the three tasks in ReaLMistake exhibit different difficulties. GPT-4 achieves reasonable Fl scores on MathiGen but performs poorly on FgFaciV and AnsCls with very low recall. This result suggests that error detection on tasks with different properties has varying difficulties, and indicates that ReaLMistake provides error detection tasks on diverse tasks with different properties.
To analyze relationships between the performance of LLMS on error detection and other popular tasks, we compare the performance of 12 LLMs on ReaLMistake LMSYS Elo Rating (Zheng et al. 2023 3 Figure 5 shows that LLMs with higher LMSYS Elo Rating (stronger models) detect errors in LLM responses with higher precision but with lower recall, revealing the challenges LLM-based error detection: low recall of strong LLMs and
arena- arena--eederboard (March 16, 2024)
7

---

Published as a conference paper at COLM 2024
Gemma 7B
Llama 2
Mistral
Qwen 1.5
GPT35
Gemini
Claddes
GPT-4
Expert
Error Detector
Random
13B
70B
7B
8x7B
14B
72B
0125
10
Opus
0613
0125
Human
F1
Llama 2 GPT4
MathiGen
46.5
54.2
59.5
6.9
45.5
52.3
32.8
65.3
42.5
50.1
63.1
70.9
62.1
90.0
RgFactV
60.3
65.4
69.9
50.9
46.8
57.7
24.9
41.4
45.8
48.9
12.7
20.8
62.9
95.5
AnsCls
59.2
69.8
69.8
48.1
38.3
53.8
15.1
28.8
40.7
38.5
20.0
22.1
62.1
90.5
MathGGn
56.6
69.2
9.0
72.3
52.9
81.8
98.3
FigFactV
68.9
78.7
81.8
68.2
35.1
64.6
18.3
34,2
42.0
45.2
38.8
68.5
80.6
100.0
AnsCls
77.4
51.6
61.9
29.8 8
3.7
16.4
23.2
75.9
81.2
100.0
Precision
GPT4
MathGen
61.6
62.6
73.0
22.8
75.5
77.4
82.9
77.3
78.1
94.9
94.4
88.9
62.1 1
100.0
FgFactV
62.4
58.4
61.3
49.9
67.2
78.2
100.0
95.5
AnsCls
64.0
62.2
65.2
59.8
60.9
68.6
55.4
72.8
78.4
74.9
79.9
88.2
62.1
95.0
Llama 2
MathGen
82.6
79.5
88.6
41.8
89.0
96.2
94.5
86.4
90.0
95.0
97.7
95.2
80.0
100.0
FgFactV
83.5
81.9
82.4
80.0
96.3
83.2
73.7
98.7
85.7
99.3
85.4
92.6
80.6
100.0
AnsCls
82.5
77.3
83.8
86.3
78.3
100.0
100.0
Recall
Llama 2 GPT4
MathGen
30.0
52.3
75.3
4.3
35.1
49.7
23.3
64.1
41.7
35.9
48.0
59.5
62.1
81.8
FgFactV
60.5
73.0
83.2
45.2
44.3
36.9
39.2
38.6
95.5
AnsCls
57.2
81.3
79.3
45.4
29.6
54.0
8.9
19.3
31.6
26.4
11.5
12.6
62.1
86.4
MathGen
51.2
50.2
72.9
5.7
44.3
47.3
37.5
65.8
46.9
72.7
81.2
86.9
80.0
96.7
208
RgFactV
61.8
77.5
82.9
60.7
24.4
61.2
11.0
24.2
32.2
32.6
25.8
54.8
80.6
100.0
AnsCls
23.3
77.5
46.7
52.3
19.4
45.2
2.7
1.9
9.8
13.3
45.2
62.1
81.2
100.0
Table 4: Error detection performance of 12 LLMs with zero-shot prompts on ReatMis
This table shows the average performance on four prompts in Section 4.3 "Random" baseline predicts each instance as an error in the same probability as the frequency of the error labels for each dataset. Human performance is evaluated on 35 cases in each setting. Gray color represents the values worse than the random baseline.
Math Word Problem Generation
Fine Fine-grained Fact Verification
Answerability Classification
1.00
OPTSy
L138
Off4
0.75
0.50
Semg
1100
cm13
GeLo
Gel.0
Gausss
Hrr
LTED
GeLD 48470
0728
GIT2
0731
1000
1050
1100
1150
1200
1250
1000
1050
1100
1150
1200
1250
1000
1050
1200
1150
1200
1250
LMSYS Chathot Rating
LMSYS
Ele Rating
Chatbot
Rating
Figure 5: Relationship between LMSYS Elo Rating (Zheng et al. 2023) and error detection performance on GPT-4 responses in ReaLMistake made by GPT-4 with higher precision performance on MMLU (Hendrycks et al. 2021 in Appendix G.1 shows the same trends.
LLMs. Stronger LLMs detect errors but with lower recall Comparison to the
G.1shows
low precision of weak LLMs. Comparison to the performance on MMLU (Hendrycks al., which shows consistent results, and correlation coefficients are in Appendix G.T
2021
4.2
Explanations Generated by LLM-based Error Detectors are Not Reliable
Our prompts instruct LLM-based error detectors provide an explanation in addition to predicting the binary error label. We manually analyze the explanations provided by error detectors based on GPT-4-0125, Claude Opus, and three open-soure models that show relatively good binary error detection performance in Table 4
f the distribution of mistakes the explanations by the error detectors on responses from GPT-4-6113. We observe that GPT-4-0125 and Claude 3 Opus generate explanations of relatively good quality, but they still make mistakes, especially on MathiGn. Explanations by open-source models more often wrong even when the binary predic- tions are correct. They often make mistakes even for correct responses from GPT-4-0613, not just missing errors, resulting in low precision. To improve the error detection performance,
8

---

Published as a conference paper at COLM 2024
Malh Word Problem Generation
Fine-grained Fact Verification
Answerability Classification
0714 (01251
26
25
Spua
30
25
50
15
35
25
@wen1: 720
22
15
20
20
15
25
32
25
Lima 2 70b
50
Figure 6: Distribution (%) of mistakes in explanations by LLM-based error detectors (Type 1-A) on responses from GPT-4-0613 in ReaLMis Open-source models introduce more mistakes compared to GPT-4 and Claude 3. We manually classify outputs from error detectors for 20 cases in each dataset: correct binary label prediction with correct, insufficient (some requirements not evaluated), wrong reasoning, or O no reasoning; wrong binary label prediction with no reasoning, insufficient (missing errors), or wrong reasoning. Appendix provides example outputs from LLM-based error detectors.
Error Sensitive Prompt (Type 1)
Conservative Prompt [Type 2)
Wording Bias
Your task to detect errors the model response.
Your task to dassty whether the model response valid
Will get different rosults for promptt with differant wordings but in the same meaning?
Shared Definition
definition
resporse
requiemente
imput
Provice your explanation first and condiude your resporse with
Provide your explanation frrst and condlude your resporso with
Positional get diferrett results option order 1llpped?
contains or".. contalrs
resporioe valid.
contaiins
valid.
Type 1-A
Type 1-B
Type 2-A
Type 2-8
Figure 7: Prompts for LLM-based error detectors for LLM responses. We evaluate four zero-shot prompts: two different wordings (Type and 2) and the order of the error label options (Type A and B). We use a shared task definition in all prompts to make sure that they use the same definition of errors. The full prompts are provided in Appendix
this analysis suggests that we need to make strong LLMs more careful about the mistakes in the provided LLM responses and improve the reasoning capability of open-source models.
4.3 Error Detection Sensitive to Prompt Design, but Not Easy to Improve Performance
We analyze the influence of two small changes in prompts on error detection by comparing
the performances of LLM-based error detectors with four prompts (Figure 7.. Positional
Bias (Wang et al., 2023a Zheng al. 2023 Pezeshkpour & Hruschka 2023 We evaluate the influence of the order of the error label options in the prompts. In an average of 12 LLMs and tasks, putting the "error" option first (Type B in Figure has 16.0 + 21.7% (Type 1) and 27.2 + 23.9% (Type 2) higher recall, suggesting that LLM-based error detectors prefer the first option. Wording Bias: We compare two wordings in error detection that ask to "detect errors in the response" (Type 1) and "evaluate whether the response is valid" (Type 2) while providing the same definition to make sure that they are semantically identical. In an average of 12 LLMs and tasks, Type 1 has 16.9 20.3% higher recall These results show that recall of error detection is sensitive to small changes in prompts
However, we also observe that these biases cannot be easily used to improve error detectors. First, although low recall is a problem in error detection by strong LLMs, we observe that small changes prompts have a very small influence on the recall of strong models. In ad- dition, we do not observe consistent trends in precision, although low precision is a problem in error detection by weaker models. These results suggest that non-trivial improvements to prompts or frameworks are needed to improve the error detection performance.
4.4 Popular Techniques to Improve LLMs Do Not Improve LLM-based Error Detection
A possible approach to improve LLM-based error detectors is to apply techniques proposed for improving LLM-based evaluators for LLM responses (e.g., pairwise evaluation). We
We compare the average performance of Type 1-A and 1-B with that of Type 2-A and 2-B. Thhe numbers after "+" represent standard deviation. Detailed results are in Appendix G.2

---

Published as a conference paper at COLM 2024
apply three popular approaches to LLM-based error detection: self-consistency (Wagg et al. 2023b), integrating responses by multiple LLMs (Zhang et al. 2023 Li et al. 2023b
Cohen
et al. 2023
Chan et al. 2024), and providing evaluation steps in prompts (Liu et al. 2023a).
self-consistency with five samples, the majority vote of binary error predictions by multiple models, and providing human-writter instruction of evaluation steps in prompts. However, none of these methods in popular ap- proaches improves error detection performance (Appendix A). This observation indicates that applying existing techniques for LLM-based evaluators is not sufficient to improve LLM-based error detectors, and RiaLais provides challenging error detection tasks.
Specifically, we evaluate three simple methods:
Future Directions Evallation on Ambiguous Errors
In this work, we propose to evaluate error detectors on objective errors because this approach has multiple advantages, including smaller noises, the absence of subjectivity of annotators, binary annotations, and smaller annotation costs. However, since many NLP tasks are subjective, expect future research to evaluate whether error detectors have good correlation with human evaluators on ambiguous errors. A possible approach to collecting benchmarks for evaluating error detectors ambiguous errors is collect annotations by multiple annotators for each instance to get average error scores. However, difficulties in creating such benchmarks include (1) hiring annotators who represent the distribution of (possibly subjective) error annotations by general people, (2) evaluating binary predictions from L.L.-based error detectors using labels by multiple annotators, and (3) high annotation cost. We expect future work to address these challenges.
6 Conclusion
We introduce ReaLMistake, the first benchmark for evaluating error detection methods for LLM responses, consisting of objective, realistic, and diverse errors made by LLMs. Our experiments this benchmark with error detectors based on 12 LLMs show that detecting mistakes in LLMs (GPT-4 and Llama 2 70B) is challenging even for recent LLMs.
Reproducibility Statement
Experiments. We provide a GitHub repository that includes the ReaLMistake benchmark (also hosted at Hugging Face Datasets all outputs from LLM-based error detectors used in Section 4 and evaluation code.
Annotation.
Appendix provides annotation instructions we used for our benchmark.
Acknowledgment
This work was supported by Cisco Research Grant. We thank Greg Durrett for the valuable discussions and comments. We appreciate valuable suggestions from anonymous reviewers.
References
Anthropic. Introducing the next generation of claude, 2024. URL comm/newss/claude---faaiii
anthropic
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng
10

---

Published as a conference paper at COLM 2024
Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang. Sheng- guang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. Xiv preprint ar Xi::: 16609, 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Law
Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback. arv preprint ar arXiv:2212.08073, 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXio:2108.0720 072558 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL http::: hww.:/eeeeeddinnnn.nuuirr.i.pepe nwwrips....... httpsaaccaaaaleeeeeeoeee//ee pdf
lamper_fllee files/paper/
Shuyang Cao and Lu Wang. CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6633 6649, Online and Punta Cana, Dominican Association for Computational Linguistics. doi: 10.18653 /8653/v1/ org/ 2021. emnlp-main.55
Republic, November
2021.emnlp-main.e 532.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better LLM-based evaluators through multi-agent debate. In The Twelfth International Conference on Learning Representations, 2024. URL
https://openreever. wtt.::..........ommoeeeoooooaaoa
Wenhu Chen, Xinyi Wang, and William Yang Wang. A dataset for answering time- sensitive questions. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL httpss///ppeeereeeee http://ppnrrevieeeeeee/e/rrrroaa net/forum?
id=9-LSfSU74n-
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXtt:2304.05122 2023a.
Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. Exploring the use of large language models for reference free text quality evaluation: An empirical study, In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi (eds.), Findings of the Association for Computational Linguistics: IJCNLP- AACL 2023 (Findings), pp. 361-374, Nusa Dua, Bali, November 2023b. Association for Computational Linguistics. doi: 10.18653. wtt.///.2222////// wwt..finnn-njnnllppp.. URL https:// aclanthology.co org/
w02./ 2023. ff findings-ijenlp.35
11

---

Published as a conference paper at COLM 2024
Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. Exploring the use of large language models for reference-free text quality evaluation: An empirical study. In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi (eds.), Findings of the Association for Computational Linguistics: IJCNLPP AACL 2023 (Findings), pp. 361-374, Nusa Dua, Bali, November 2023c. Association for Computational Linguistics. doi: 111/86633/122222 htwp://655////022222nnnn//a//// findings-ijenlp.2. URL https:// aclanthology orgg/0200 findings-ijjelp..3
Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15607-15631, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18053/...1 2023. acl-long.. 870
870. URL https:://c.lanthooooyy org/
Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. LM vs LM: Detecting factual errors via cross examination. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12621-12640, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/vi 2023.emnlp-main. 778.
2203..222222.2222 emnlp-main. 778
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback. In Thirty-seventhh Conference on Neural Infor- mation Processing Systems, 2023. URL httl/tttt///peeeviiieeeee
Alexander R. Fabbri, Wojciech Kryšciñski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. SummEval: Re-evaluating Summarization Evaluation. Transactions of the Association for Computational Linguistics, 9:391-409, 04 2021. ISSN 2307-387X. doi: 10.1162/ tacl a 00373. UtLtttss:///ooi httit::::/doio.ogg/000/ wwt.ei1i1ccccc..a 303333
Gemini Team Google. Gemini: A family of highly capable multimodal models. Xiv preprint ar arXtt:2312.11105 2023.
Google. Gemma open models, 2024. URL URL httpt::ss:/aaoogeeeeeeeee devggeemaa
Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Nan Duan, and Weizhu Chen. CRITIC: Large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference on Learning Representations, 2024. URL https
//openreview.net/// wtt.://eeeeoooddddddeeeeeee
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learnngg Representations, 2021. URL //tperee//ppnrevieew www.:neeefftfiiememmmm
d7KBjjmI3Gm
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xiny- ing Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, 2024. URL
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anee Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. arXiv preprint arXiv:2310.0682 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang,
12

---

Published as a conference paper at COLM 2024
Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Tim- othée Lacroix, and William El Sayed. Mixtral of experts. arviv preprint ar arXio:2401.04088, 2024.
Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. WICE: Real-world entailment for claims in Wikipedia. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7561-7583, Singapore, December 2023, Association for Computational Linguistics. doi: 10. http:////222.../mmmma///
URL http:tp::/aacllaboooooo.ooogg/g/ org/ emnlp-main
470
Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can Ilms actually correct their own mistakes? critical survey of self-correction of llms. ar preprint arXiv:2406.01297 2024.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. Xiv preprint ar XrX2:2330..7741, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Philippe Laban, Wojciech Maciej Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caim- ing Xiong, Shafiq Joty, and Chien-Sheng Wu. Summedits: Measuring LLM ability at factual reasoning through the lens of summarization. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL PHtXqUNGUA
Junyi Li, Xiaoxue Cheng, Xin Zhao, Yun Nie, and Ji-Rong Wen. HaluBval: A large-scale hallucination evaluation benchmark for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6449 6464, Singapore, December 2023a. Association for Computational Linguistics. doi: 10.18653 /1133//v//// 222.2222mmmmmmain.....
URL https
//aclanthology.orge. org/2023. emnlp-main.30 397
Ruosen Li, Teerth Patel, and Xinya Du Prd: Peer rank and discussion improve large
language model based evaluations. ar Xiv preprint arXi::2307.0277 2023b.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, 1lya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL //openreview www.:neeeftnniiiiemm 20000000000010
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by ratio- nale generation: Learning to solve and explain algebraic word problems. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 158-167, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653 10.18653/v11P172// 1111771011 URL https://eclanthooogy org/P17-1015 1ttp:/aaoooooooooooo////11111110
Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. Deductive verification of shain-of-thoughh reasoning. In Teirth- Conference on Neural Information Processing Systems, 2023. URL bottps///ppereevies wtw.w/ww.ennnnonnr///e
13

---

Published as a conference paper at COLM 2024
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G- eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods Singapore, December 2023a. Association for Computational Linguistics. doi: 10.18653/v
in Natural Language Processing, pp.
URL https
//aclanthology cccclanthooogy.rrr222222 org/2023. emnlp-main 153
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G- eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Singapore, December 2023b. Association for Computational Linguistics. doi: 10.18653 /vl /v 20//2033..mllppmmmin1....... URL https
in Natural Language Processing, Pp.
//aclanthology wrggggy..orggggge org/2023. emnlp-main. 153
Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan. Benchmarking gener- ation and evaluation capabilities of large language models for instruction control- lable summarization. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Find- ings of the Association for Computational Linguistics: NAACL 2024, pp. 4481-4501, Mex- ico City, Mexico, June 2024. Association for Computational Linguistics. URL https: //aclanthology wwg.//g//.22... htt.://o..or//222...i.iiinieeee findings--aacll
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bod- hisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In A. Oh, T. Neu- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. 46534-46594. Curran Associates, Inc., 2023. URL lttps:: httpss:/pproceeddngg/ www.ine..ricccoo wwtpppperpcpppppppppeeeee faaapper_fieepppppeeeee /a1les//aperr222//51eee) 1le2f00722b11ba0000ffff666333 Conferree pdf
Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. Expertqa: Expert-curated questions and attributed answers. arXiv preprint ar arXiv:2309.07852, 2023.
Potsawee Manakul, Adian Liusie, and Mark Gales. SelfCheckGPT: Zero-resource black- box hallucination detection for generative large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9004-9017, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/vll //aclanthology. org/ 202//2223222 emnlp-main.n5..
URL https
Ning Miao, Yee Whye Teh, and Tom Rainforth. Selfcheck: Using Iims to zero-shot check their own step-by-step reasoning. arxiv preprint arXiv:2308.004 00446, 2023.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12076-122 122000, Singapore, December 2023. Association for Computational Linguistics. doi: 20.18653/v1n 00.18653/w112023.emmm /tt0////222..mmmmmmm..... //aclanthology. hwwpchhooogoooooooooooo
URL https
www.//2.........///. 2023. emnlp-main 741
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arxiv preprint arXio:2203.02155. 2022.
14

---

Published as a conference paper at COLM 2024
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically Correcting Large Language Models: Surveying the Landscape of Diverse Automated Correction Strategies. Transactions of the Association for Computational Linguisticss, 12:484 506, 05 2024. ISSN 2307-387X. doi: 10.1162/ tacl a 00660. URL https //doi. 1wt.:...11.aaaaala a_00660____000_______
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman.
QuALITY: Question answering with long input texts, yes! In Marine Carpuat, Marie- Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5336-5358, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/wl wt22022.maacmmmin.333. URL https //aclanthology. .og./ 202/2022......ooooo 2022. naacl mmal..minn....
Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions. Xiv preprint XXXx:2300811483 2023.
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models. Xiv preprint ar arXiv:2401.03601, 2024.
Qwen Team. Introducing qwen1.5, 2024 URL https: http:p/:///geeal....ii.bb.bb/b/a
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Compulational Linguistics (Volume 2: Short Papers), pp. 784-789, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653 1111665//11P188-1222 URL httppp///bcllaathooogooor/r/////
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google- proof q&a benchmark. arXiv preprint ar drXiv:2311.12022, 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL httpps://openrevien net/
Fooum?1d=vAAIIPPKkk
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. ar Xiv preprint ar Xii::22060046115 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arxiv preprint ar arX::23077.0928 2023.
Gladys Tyen, Hassan Mansoor, Victor Cärbune, Peter Chen, and Tony Mak. Lims cannot 2024.
find reasoning errors, but can correct them! ar Xio preprint ar
15

---

Published as a conference paper at COLM 2024
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. Xiv preprint (xXi::236.11926, 2023a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Repre- sentations, 2023b. URL attps://openrevier:
Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Wenjin Yao, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. PandaLM: An automatic evaluation benchmark for LLM instruction tuning optimization. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreever: net wwt/fooummmiii 5d-
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022a. URL htttp:://operrvval net/forumiid ddzggezGGoZddZZGd
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho Advances in Neural Information Processing Systems, 2022b. URLlattts://oppereey http/forumnidd__jjMMMaa
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arve preprint ar arXiv:2109.10862, 2021.
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North Ameri- can Chapler of the Association for Compulational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1819-1862, Mexico City, Mexico, June 2024. Association for org/2024. 2024. www.eanaccllaa -long. 102
Computational Linguistics. URL (RLL:ttps://aacathhooLL
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380, Brussels, Belgium, October-November 2018. As- sociation for Computational Linguistics. doi: 10.1863//1//11--1255
URL https
//aclanthology. www./o/----. 1259
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations, 2024.
troKiidPLCc
Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. Wider and deeper I1m networks are fairer Ilm evaluators. Xiv preprint arXiv:2308.0186 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
16

---

Published as a conference paper at COLM 2024
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXio preprint ar Xxi::2311.07911, 2023a.
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context-faithful prompting for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14544-14556, Singapore, December 2023b. Association for Computational Linguistics. doi: 10.18653/v1/2033 findings-emnlp.99 968.. URL WRRLhttps:///
htap:/22000./dddnngeeemmml..1 968
17

---

Published as a conference paper at COLM 2024
Table of Contents of Appendix
Performance of Advanced Error Detection Methods Mistae
19
A.1 Self Consistency
19
A.2 Majority Vote
19
A.3 Human-wrilten Instruction of Evaluation Steps
19
Dataset Creation Process
22
B, Math Word Problem Generation
22
B.2
Fine-grained Fact Verification
22
B,3
Answerability Classification
23
Annotation Instructions
25
Definition of Error Labels
25
Task-Specific Instructions
26
D Diversity of Tasks in ReaLMistake
28
Prompts for LLM-based Error Detection
29
Additional Results of Error Defection Performance
30
Additional Results for Error Detection Analysis
31
Additional Results for Comparison to Other Tasks
31
G.2
Additional Results for Bias Analysis
33
H Model Access Details
34
H.1
Closed Source LLMs
................................
34
H.2
Open Source LLMs
34
Data Example of ReaLMistake and Error Detection Outputs
35
Math Word Problem Generation (MathGen)
35
1.2
Fine-grained Fact Verification (FgFactV)
40
1.3
Answerability Classification (AnsCIs)
44
18

---

Published as a conference paper at COLM 2024
A Performance of Advanced Error Detection Methods on ReaLMistake
This section includes the results of the advanced error detection methods in Section 4.4 In our experiments, the three methods do not improve the error detection performance.
A.1 Self Consistency
Table5 the error detection performance of three LLM-based error detectors (prompt type I-A in Figure with self-consistency (Wang et al. 2023b) on MiatMis We use k = 5 and 1emperature=0.5 for all models. We do not observe improvement compared to the performance of a single output with temperature=0.0.
A.2 Majority Vote
Tsoos the performance of the majority vote on binary predictions from LLM-based detectors based on three open-source LLMs. We use four prompts in F and take the majority label of 12 outputs models prompts). We do not observe improvement in the approach of taking the majority vote on the binary predictions.
A.3 Human-written Instruction of Evaluation Steps
The idea of providing instructions on evaluation steps is motivated by G-Eval (Liu et al. 2023b), which provides LLM-generated evaluation steps to LLM-based evaluators. For simplicity, we manually create human-written evaluation steps and provide the following instructions in the error detection prompt to help error detectors provide better reasoning. We added the following instructions at the end of the Prompt 1-A (Appendix)
Human-written Instruction of Evaluation Steps
Follow the instructions below to check whether the model response contains an error:
1. Read the model input carefully.
2. Read the model response carefully.
3. Check whether the model response follows all instructions and requirements in the model
input. Provide your explanation on each requirement.
4. Select your answer from "Therefore, the model response contains error." or "Therefore, the
model response contains no error.
Table 7 includes the improvements made by providing human-written instruction evalua- tion steps in prompts. We do not observe any consistent improvement by this method.
19

---

Published as a conference paper at COLM 2024
Detector
Mixtral 8x7B
GPT-3-Turbo
GPT-4-0125
Temperature
T=0.0
T=0.5
T=0.0
T=0.5
T=0.0
T=0.5
# sample
k=1
k=5
k=1
k=5
k=1
k=5
F1
MathGen
63.0
64.2
81.2
82.4
74.5
83.0"
FgFactV
82.4
75.4
67.9
61.8
22.9
27.8
AnsCls
41.7
44.0
52.2
51.2
23.5
18.2
Precision
MathGen
68.0
70.8
70.0
71.8
86.4
91.7
FgFactV
75.7
68.4
76.0
70.8
100.0
AnsCls
50.0
50.0
66.7
73.3
66.7
60.0
Recall
MathGen 58.6
58.6
65.5
75.9
FgFactV 90.3
83.9
61.3
54.8
12.9
16.1
AnsCls 35.7
39.3
42.9
39.3
14.3
10.7
Table 5: Performance of LLM-based error detectors (prompt type 1-A) with self-consistency (k=5, temperature=0.5) on 50 cases from three datasets with GPT-4-0613 responses in ReaLMistake. Self-consistency does not provide consistent improvement in the error detec- tion task in Reatakis ": p < 0.1 on the pairwise bootstrap test with 1,000 samples.
Llama 2
Mixtral 8x7B
Qwen 72B
Majority Vote Three Models
70B
F1
GPT-4
MathGen
45.5
32.8
37,9
0813
FgFactV
46.8
24.9
39.1
AnsCls
69.8
38.3
15.1
26.8
Llama
MathGen
69.2
56.0
50.3
708
FgFactV
81.8
35.1
18.3
22.1
AnsCls
51.6
29.8
5.1
1.5
Precision
GPT4
MathGen
73.0
75.5
82.9
75.9
0613
TgFactV
62.4
61.3
67.1
62.5
AnsCls
65.2
60.9
55.4
60.0
Llama 2
MathGen
88.6
89.0
94.5
93.1
208
FgFactV
82.4
96.3
73.7
100.0
AnsCls
77.3
86.3
70.5
33.3
Recall
GPT-4
MathGen
75.3
35.1
23.3
25.3
0513
FgFactV
83.2
44.3
17.0
28.4
AnsCls
79.3
29.6
8.9
17.2
Llama
MathGen
72.9
44.3
37.5
42.2
708
EgFactV
82.9
24.4
11.0
12,4
AnsCls
46.7
19.4
2.7
0.8
Table 6: Performance of the majority vote of binary predictions from 3 LLM--ased error detectors (rightmost column). The performance of each model (left columns) is the average performance of the four prompts (Table 4 and the majority vote is taken over 12 outputs (3 models x four prompts). Yello color represents values better than the performances of majority vote predictions. We do not observe improvement by taking the majority vote of predictions from multiple models.
20

---

Published as a conference paper at COLM 2024
Gemma 7B
Llama
Mistral
Qwen 1.5
GPT35
Gemini
Claude3
GPT-4
Error Detector
138
70B
7B
8x7B
14B
72B
0125
1.0 Pro
Opus
0613
0125
F1
GPT-4
0613
MathGen
FgFactV
17.0
4.6
-28.4
-1.9
-0.2
-8.7
2.8
4.3
-6.0
-4.2
17.1
15.4
-10.4
-2.0
-15.0
-29.4
-29.5
14.3
-3.9
2.3
-1.0
3.3
AneCls
-8.5
Llama
MathGen
18.4
-30.9
0.7
-8.9
3.5
-4.1
3.6
-1.5
-24.7
4.2
-0.5
-0.1
708
FgFactV
1.9
-2.9
0.4
-23.3
13.4
-53.9
9.7
-39.3
-20.2
5.4
-5.8
-7.0
AnsCls
3.2
-11.0
4.7
-17.0
-19.0
-44.8
10.2
-5.6
-20.1
-3.9
-2.7
-0.8
Precision
GPT-4
MathGen
17.1
-13.7
0.2
17.2
5.6
5.6
-6.0
2.5
6.6
-6.9
1.9
6.1
0613
EgFactV
0.2
3.6
0.2
-1.8
-1.4
2.2
9.5
3.0
4.3
-6.7
0.0
0.8
AnsCls
-3.0
13.8
2.1
0.0
-10.5
-1.8
-5.0
-5.7
Liamm
MathGen
12.9
0.7
25.0
8.1
1.0
-1.1
3.3
12.0
1.3
0.8
1.4
703
FgFactV
-3.2
5.7
0.1
11.8
2.7
-13.2
1.2
1.3
8.7
1.7
4.8
+3.5
AnsCls
-4.1
-0.8
-3.6
-1.6
-3.0
-1.4
12.6
-2.8
1.0
0.0
1.5
1.1
Recall
GPT-4
MathGen
16.1
-36.8
-1.1
8.0
3.4
-19.5
31.0
-9.2
-47.1
19.5
2.3
-4.6
06.3
FgFacIV
-12.5
-1.1
-12.5
10.2
-17.0
19.3
-20.5
-25.0
-31.8
-2.3
2.3
AnsCls
11.5
-10.3
5.7
-10.3
-3.4
-20.7
0.0
-12.6
-32.2
-8.0
-3.4
4.6
Llama
MathGen
-1.6
708
FgFactV
7.0
-13.2
0.8
-36.4
15.5
-70.5
8.5
33.88
-26.4
5,4
-5.4
-7.8
AnsCls
-22.3
14.6
-23.8
-13.8
-65.4
6.2
3.11
-1.88
-3.8
-3.1
-1.5
Table 7: Improvement of LLM-based error detectors by using human-written instructions of evaluation steps. This table compares the performance of the original prompt type 1-A in Figure and the updated prompt type 1-A with human-written evaluation steps. The gray color represents that human-written instructions decrease performance. This approach does not provide improvement from the original simple prompt.
21

---

Published as a conference paper at COLM 2024
B Dataset Creation Process
We provide details of the dataset creation process (Figure 3 for tasks in RiaLMistake (Section 3.2 Please refer to Appendix I for full inputs of these tasks.
B.1 Math Word Problem Generation
Math Word Problem Generation is task to create math word problems that follow the provided requirements. We use math word problems and solutions in the AQuA dataset (Ling et al. 2017) to generate requirements. We generate 9 types of requirements to introduce diversity to this task (Table 3). First, to create requirements about the types of numbers (e-g., integers, fractions) in questions and solutions, we create data type detection script using regular expression operations. Second, We use the following prompt to generate 7 properties of the question in AQuA by using GPT-4-0613. We note that this prompt only asks GPT-4 to retrieve specified information from the questions, so we believe that there is no strong influence caused by the choice of LLMs in this process.
Your task is to generate properties of the provided math word question.
Each generated sentence should be one sentence long.
You do not need to generate formatting information such as "multiple options are provided as potential answers".
The generated properties should not include "like" or "such as".
You should only generate list of properties. not generate anything else.
The generated properties should be in the following format:
The first one should explain the overview of the question.
The second one should explain the instances included in the question.
The third should include short key phrase extracted from the question. The key phrase should not include more than one number.
The fourth one should include only one randomly selected number in the question. This number should not be included in the key phrase.
'The fifth one should explain mathematical operations (e.g., linear equation) included in the solution. When there are unknown variables in equations, specify what they represent.
The sixth one should include short key phrase extracted from the solution. The key phrase should not include any number. This should not include the answer to the question.
The seventh one should include a randomly selected equation or calculation in the solution. You need to insert spaces to improve readability if necessary. This should not include the answer to the question.
[few-shot examples]
B.2 Fine-grained Fact Verification
We use the WiCE dataset (Kamoi et al., 2023) as a base dataset. WICE includes fact- verification classification task and each instance of the dataset includes claim (sentence) in a Wikipedia article, web articles cited for the claim in Wikipedia, an entailment label (supported, partially supported, not supported), and indices of supporting sentences (a part the web articles supporting the claim).
Web articles in WiCE are too long for our benchmark, tasks. Specifically, we use the oracle chunks provided in WiCE, which include the supporting sentences in the web articles and randomly selected sentences from web articles
only use part of them in our
We note that changed the entailment classification task to binary classification of supported or not supported. In our task, the original partially supported cases are regarded
oracle, chunks
22

---

Published as a conference paper at COLM 2024
as not supported. We added instructions to perform fact verification for all pieces of information in the claim to reconstruct the task to fine-grained fact verification.
B.3 Answerability Classification
Answerability Classification a task to classify questions as unanswerable if they include factual errors any other problems. We use the HotpotQA dataset (Yang et al., 2018 base dataset. Each instance in HotpotQA includes multi-hop question about knowledge in Wikipedia articles and paragraphs from Wikipedia articles that include information required to answer the question. The original task in HotpotQA is mult-hop QA using the provided evidence, but do not provide the paragraphs and make a closed-book QA task.
First, since questions in HotpotQa are often grammatically unnatural, we use GPT-4-0613 to improve the questions without changing any properties.
Your task is improve the following question while keeping the original meaning of the question. You should only generate the rewritten question. Do not include anything else in your response.
[few-shot examples]
Second, three types of automatic processes create diverse unanswerable questions (Figure We use each of the following processes for 1/3 of the cases in our dataset.
Factual Mistakes in Numbers. The first type is mistakes in numbers. We provide the following prompt to GPT-4-0613 to generate (possibly) unanswerable questions that include factual mistakes in numbers.
We provide a question that can be answered by using Wikipedia articles. First, add correct additional information from related Wikipedia articles to the question that includes at least one number (e.g,, year, age, quantity). The added information should not change the answer to the question. Second, introduce a minor mistake in the number in the added information. Do not include anything else in your response.
[few-shot examples]
Other Factual Mistakes. The second type is mistakes other than numbers. We provide the following prompt GFT-4-0613. We instruct not make mistakes proper nouns because they make the answerability detection task easier. However, this requirement is not always satisfied as shown Figure
We provide a question that can be answered by using Wikipedia articles. First, add correct additional information from related Wikipedia articles the question. The added information should not change the answer to the question. Second, introduce factual mistake in the added information (nouns, adjectives, verbs, etc.). However, do not introduce mistakes in people's 's names, titles of books or movies, and locations (e.g., countries).
You should introduce factual errors, so the mistakes should not use synonyms paraphrases that do not change the factual correctness of the information.
Do not include anything else in your response.
shot examples]
23

---

Published as a conference paper at COLM 2024
Time Sensitive Questions. The third type is time sensitivity. All inputs in our Answer- ability Classification include time constraints format of:
Assume you on fdate) and questions that require knowledge after this date should classified as unanswerable.
For the questions involving the above two types of factual mistakes, we randomly select dates between 2022-09-01 and the date the first version of the Wikipedia article was created (we select the newest article from the ones provided for each question in HotpotQA). However, randomly selected dates often do not make the questions unanswerable.
To create more questions that are unanswerable due the time constraint, we use years in the paragraphs provided in HotpotQA as a knowledge resource for each question. Specifically, we randomly select years in the paragraphs provided for each question in HotpotQA and use the time constraint of one year before the selected year. This strategy makes least one event in the paragraphs unanswerable, although it is not guaranteed that this event is involved in the question-answering process.
24

---

Published as a conference paper at COLM 2024
C Annotation Instructions
Tasks in ReaLMistake include detailed task definitions in inputs (prompts for LLMs), and binary error detection on these tasks is well-defined without any additional instructions (please refer to Appendix [ for examples of full inputs of our tasks). However, provide annotation instructions to annotators to further improve the annotation quality and provide instructions error category annolations. We provide 9-page annotation instructions with multiple annotation examples.
To make sure that annotators understand these instructions, we provide five challenging cases for each dataset (3 x cases in total) as a practice annotation. We manually check their annotations and provide detailed feedback. they make major mistakes, we provide another five cases. We continue this process until all annotators clearly understand the annotation instructions. This phase took about two weeks in total to train all annotators.
This section provides key parts of the annotation instructions.
C.1 Definition of Error Labels
error
Select this option when the response includes least one error.
Provide a detailed explanation in the "feedback" column.
Responses with any kind of errors, not restricted to the provided error categories, should be categorized as errors (e-g,, grammar mistakes). When you do not select any of the four categories, please explain in the "feedback" column.
no.error
Select this option only when the response does not include any error
You do not need to provide feedback.
not-sure
Select this option when you cannot determine the binary error label.
For example, you should select this option when mistakes in the output too minor or the model input is invalid.
The cases with this label will be removed from the final dataset.
25

---

Published as a conference paper at COLM 2024
C.2 Task-Specific Instructions
For analysis purposes, we categorize errors in model responses into four categories. Since model inputs do not provide any information about error categories, we provide detailed instructions on error category annotations.
Math Word Problem Generation
In this task, each model output includes generated question and solution. We expect mistakes in this task be mainly in Instruction-Folloving and Reasoning Correctness. In this task, an output from the models includes generated question and solution.
When the generated question solution does not follow the constraints, select the Instruction-Folloving error category.
When the generated question is invalid (unanswerable), you also need to select the Instruction-Following error category because the instruction says "The generated question should be valid and answerable".
When the solution includes wrong reasoning, select the Reasoning Correctness category.
Unnecessary information
When the question includes unnecessary information, should be marked the Instruction- Followiin error because the instruction says not to include unnecessary information in the generated questions.
In solution:
When the solution provides two possible ways to answer the question verify the final answer, regarded as an Instruction-Folloving error because the instruction says not to include this.
When the solution includes unnatural and unnecessary steps other than the above cases, select the Reasoning Correctness error category.
[Multiple Annotation Examples]
26

---

Published as a conference paper at COLM 2024
Fine-grained Fact-Checking
Check whether all pieces of information in the claim are verified in the response. If any part of the claim is not verified in the response, mark it Context-Fathfullnss error even if the final answer is correct. Note that we regard the claim context as well as the evidence this task because this task to compare two texts (claim and evidence).
Check whether each piece of information has been verified correctly.
If the response misses or misinterprets any part of the evidence, mark it as a Context-Faiithullees error.
If the response provides incorrect reasoning from correctly retrieved information, mark it as a Reasoning Correctness error.
Check whether the final answer is faithful to the verification of each piece of infor- mation by the model. If not, mark it error of Reasoning Correctness.
If the response does not follow the formatting requirements, mark it as an Instruction- Following error.
[Multiple Annotation Examples]
Answerability Classification
In this task, we provide factual questions with time constraints that may include incor- rect information. LLMs need to classify the question as unanswerable if (1) it requires information after the specified date or (2) it includes incorrect information. Many questions in this task are unanswerable, but some questions may still be answerable. You need to check all the information in each question by using Wikipedia any other reliable resources.
the question includes mistakes but the model response does not point out any factual mistakes, this is Parameterized Knowledge error.
When there are multiple mistakes in question, the response only needs to point out one mistake. The response does not need to point out all mistakes.
We introduce an intentional error in the questions. However, the original question can already include mistakes.
If the response includes factual mistakes, they are also errors in Parameterized Knowledge.
The response can include factual mistakes even when the answerability classifica- tion is correct. This case is also Parameterized Knowledge error.
For example, the response can detect one mistake in the question correctly but include factual mistakes caused by another mistake in the question. The response only needs to detect one mistake to clarify it as unanswerable, but should not include factual mistakes.
Mistakes in reasoning that do not involve factual information should be marked as a Reasoning Correctness error.
If the response correctly points out the mistakes but does not classify the question unanswerable, classify it as a Reasoning Correctness error.
The question should be classified unanswerable if it requires any information after the timestamp in the instruction. If the response ignores this instruction, it should be marked as an instruction-Followin error.
If the response does not include any factual mistakes reasoning mistakes but ignores factual mistakes in the question, this is an error of Instruction-Following
Multippe Annotation Examples]
27

---

Published as a conference paper at COLM 2024
D Diversity of Tasks in ReaLMistake
Math Word Problem Generation. Table includes the distribution of 9 categories of requirements included in this task.
Fine-grained Fact Verification. Figure shows the distribution of topics of Wikipedia articles used in the Fine-grained Fact Verification task.
Human
Organization
Other
Music
Entertainment
Academic
Geography
Politics
Sports
Film Literature
Figure 8: Topics of Wikipedia articles in the Fine-grained Fact Verification task.
Answerability Classification Task. Figure shows the distribution of the reasons for the unanswerability of the questions in this task.
28

---

Published as a conference paper at COLM 2024
E Prompts for LLM-based Error Detection
In the error detection experiments in Section described Figure 7 The following is the full text of the prompts 1-A and 2-A. The green text represents different wordings (Type and 2), and red and blue texts represent the options for the binary error label prediction (Type A and B). 1-B and 2-B flip the order of the options at the end of the prompts 1-A and 2-B. The sentences in black color in the first paragraph explain the shared definition of the binary error labels.
four simple zero-shot prompts
When evaluate the binary classification of the LLM-based error detectors, we classify the responses as predicting "error" if they include phrase "contains an error" or "response is not valid", and classify as predicting "no arrorr if they include "contains no error" or "response is valid". Otherwise, the responses are classified as invalid. Therefore, we did not evaluate the responses provide their answer at the end of their responses, which is often not satisfied by weaker models.
Error Detection Prompt 1-A
We provide an input to language model (model input) and a response from the model (model response). Your task to detect errors in the model response. You need check whether the model response follows all instructions and requirements in the model input. You need to classify the model response as an error if the model response does not follow any of the instructions or requirements in the model input.
Model Input Begins ==
(input)
Model Input Ends
Model Response Begins
[response]
Model Response Ends
In your response, provide your explanation first and conclude your response with "Therefore, the model response contains error." or "Therefore, the model response contains error.
Error Detection Prompt 2-A
We provide an input to language model (model input) and a response from the model (model response). Your task is to evaluate the quality of the model response and classify whether it is valid or not. You need to check whether the model response follows all instructions and requirements in the model input. You need to classify the model response valid only if the model response follows all instructions and requirements in the model input.
===== Model Input Begins
[Omitted. Same as Prompt 1-A.]
Model Response Ends =====
In your response, provide your explanation first and conclude your response with "Therefore, the model response not valid." or "Therefore, the model response is valid."
29

---

Published as a conference paper at COLM 2024
F Additional Results of Error Detection Performance
This section provides additional results for the experiments in Section 4 Table shows the error detection performance Table in accuracy.
Error Detector
Gemma
Llama
Mistral 7B
Qwen
1.5
GPT35
Gemini 1.0 Pro
Cloade3 Opus
GPT-
Random
Expert
13B
70B
8x7B
14B
72B
0125
0613
0125
Human
Accuracy
GPT44
MathGen
43.4
47.3
58.2
36.1
50.7
48.2
63.6
52.3
58.4
70.0
52.9
88.2
FgFactV
50.9
54.1
58.2
42.1
49.5
51.1
42.0
48.2
48.4
53.9
41.4
44.1
53.3
94.3
AnaCIs
52.7
57.0
59.6
48.0
42.3
53.0
41.4
46.4
51.4
48.6
43.2
44.6
52.9
88.2
Llama 2
MathiGen
50.0
49.1
66.9
20.2
48.3
54.8
47.8
64.1
50.8
74.7
83.4
85.9
68.0
97.1
FgFFctV
67.2
71.7
54.7
37.3
28.0
38.6
40.3
45.3
37.0
60.2
100.0
AnsCls
33.1
68.1
50.6
51.7
29.2
47.5
20.6
20.0
24.8
29.5
54.4
68.3
69.5
100.0
Table 8: Error detection performance of LLMs with zero-shot prompts on Mistake in accuracy. This table includes the additional results of the experiment in Table
30

---

Published as a conference paper at COLM 2024
G
Additional Results for Error Detection Analysis
This section includes additional results for the analysis of error detection in Section
G.1 Additional Results for Comparison to Other Tasks
Figure the full version Figure which compares the error detection performance ReaLMistake with popular evaluation benchmarks: MMLU (five-shot) (Hendrycks et al.
2021) and LMSYS Chatbot Arena Elo Rating (Zheng et al., 2023). These results consistently show that strong LLMs MMLU and LMSYS Elo Rating detect mistakes made by LLMs at high precision but low recall (Section 4.1. Table 9 shows that precision and recall have positive and negative correlations with these metrics in both Pearson and Spearman correlation coefficients. We observe that performance on MathGen shows different trends in recall (weaker negative correlation for GPT-4 responses and positive correlation for Llama 70B responses) which again indicates that Rist.ki includes diverse error detection tasks with different properties.
Table 10 includes LMSYS Chatbot Arena Elo Rating and MMLU performance (five-shot) of 12 LLMs used in Figure 5 Figure 9
We accessed LMSYS Chatbot Arena on March 16, Sources of MMLU performances are provided the table.
2024
ReaL.Mistake
Precision
Recall
Response Model
Task
Pearson
Spearman
Pearson
Spearman
MathGen
0.77
0.90*
0.12
-0.03
GPT-4-0613
FgFactV
0.72*
0.78*
-0.66*
-0.77*
AnsCls
0.64*
0.58*
0.67*
-075**
MathGen
0.59*
0.69*
0.67*
0.48
Llama 2 70B
FgFactV
0.50
0.51
-0.47
0.53
AnsCls
0.63*
0.43
0.20
0.22
(a) LMSYS Chatbot Arena Elo Rating vs, Error Detection Performance on ReaLMistake
ReaLMistake
Precision
Recall
Response Model
Task
Pearson
Spearman
Pearson
Spearman
MathGen
0.78*
0.91*
-0.3
-0.32
GPT-4-0613
FgFactV
0.73*
0.69*
-0.70*
-0.77
AnsCls
0.54
0.47
-0.73*
-0.80*
MathGen
0.62*
0.77*
0.57
0.28
Llama 2 70B
FgFactV
0.33
0.51
0.68*
-0.44*
AnsCls
0.48
0.23
-0.52
0.61*
(b) Performance on MMLU Error Detection Performance on ReaLMistake
Table 9: Correlation between precision and recall on the error detection task in ReaLMistake and performance on MMLU (Hendrycks al.) (2021) or LMSYS Chatbot Arena Elo Rating (Zheng et al. 2023) on 12 LLMs. Precision and recall on ReaLMistake have positive and neg- ative correlations with these metrics both Pearson and Spearman correlation coefficients, especially in the error detection task responses from GPT-4-0613. *; p < 0.05.
31

---

Published as a conference paper at COLM 2024
Math Word Problem Generation
Fine-grained Fact Verification
Answerability Classification
1.00
GITa
1.00
1.00
6.75
bade Lie
0.75
0.50
Corc
0.50
0.50
0.25
0.25
0.25
76
80
70 Accurary MHLI (1S
80
Accurary WHLL (%]
Accurary MMLU
(a) Error Detection on GPT-4 Responses vs. MMLU
Math Word Problem Generation
Fine-grained Fact Verification
Answerabillty Classification
3.00
0.75
e orrs Madde
0738
Gff4e
0.75
00
one
0.75
0.50
0.50
0.50-
k0
010
0.25
20.70
0738
GIT2 Gak0
50
60
70
80
90
50
60
80
90
50
60
70
80
90
Accurasy on MHLI (%)
Accuracy on MMLU [%]
Accurwcy on MHLLI [%]
(b) Error Detection on Llama 2 70B Responses vs. MMLU
Math Word Problem Generation
Fine-graines Fact Verification tion
Answerablee aewerabllly Classification
1.00
1.00
OFTAG
1.00
6672
GaL. R
Couso
DeLo
0.50
0.50
0049
0.50
0.25
0.25
0.25
Hig
Arisa.
1000
3050
1300
1150
1200
1250
1000
1050
1100
1150
1200
1250
1000
3050
1100
1150
1200
1250
LMSYS Chalbot Arena Ele Ratin
LMSYS Chatbot Arena Bo Rating
LMSYS chotbot Arena Bo Ratin
(c) Error Detection on GPT-4 Responses vs. LMSYS Chatbot Arena Elo Rating
Math Word Problem Generation
Fine-grained Fact Verification
Answerabilily Classification
1.00
1.00
1.00-
6.75
6.75
(1
0.75
5433
Aomas
0222
0.50
0.50
0.50
0.25
0.25
0.25
1000
1050
1300
1150
1200
1250
1000
3050
1300
1350
1200
1250
1000
1050
1300
1150
1200
1250
LMSYS Chatbot Arena El Rating
LMSYS
AhetbotArens Be Rating
LMSYS Chatbot Arena Bo Rating
(d) Error Detection on Llama 70B Responses vs. LMSYS Chatbot Arena Elo Rating
Figure 9: Relationship between precision and recall on the error detection task in ReaLMis- take and performance on MMLU and LMSYS Chatbot Arena Elo Rating. Stronger LLMs on MMLU (Hendrycks et al. 2021) and LMSYS Chatbot Arena Elo Rating (Zheng et al. 2023 detect errors with higher precision but with lower recall. This the full version of Figure
LLMs
LMSYM Elo Rating
MMLU
Source of MMLU Performance
GPT-4-0125
1251
Claude 3 Opus
1247
86.8
Anthropic 2024
GPT-4-0613
1159
86.4
OpenAl 2023
Gemini 1.0 Pro
1118
71.8
Gemini leam Google
2023
GPT-3.5 Turbo
1096
70.0
(OpenAl
2023)
Qwe 72B
1146
77.5
Qwen Team
2024)
Mixtral 8x7B
1114
70.6
(Jiang et al. 2024)
Llama 70b
1082
68.9
Touvron al.
2023
Llama 2 13b
1043
54,8
louvron et al.
2023
Qwen1.5 5 14B
1032
67.6
Qwen Team
2024
Gemma 7B
1029
64.3
(Google 2024
Mistral 7B
1004
60.1
[Jiang et al. 2023
Table 10: LMSYM Elo Rating and MMLU performance of 12 LLMs. *: The performances of
previous versions of GPT-4 and GPT-3.5 in March 2023.
32

---

Published as a conference paper at COLM 2024
G.2 Additional Results for Bias Analysis
Figure 10 and 11 show the additional results the analysis of biases caused by small changes prompts on the error detection task in Miataes These figures show that all results are consistent with the analysis in Section 4.3 As discussed in Section 4.3 these results show that recall is strongly affected by small changes in prompts (wordings and the order of binary label options) but there is no consistent trend in precision. In addition, we observe that GP'T-4 (red markers) and Claude 3 (yellow star markers) less affected by small differences in prompts.
40
Goveoo
Dirt in Frectsion
20
Uame 20
50-
25
Ome33
Deerd 7M1
Gevii11.P9
Osuce Doen
MathGen FofactV AnsCSs
Mathien FgFectV AnsCa Liama /Ob
MethCen FgFaciV AnsCls GP1-4 (0613)
MathGen FgfectV AraCa
GPI-4 (0613)
Liama /0b
Figure 10: Wording bias: The values in the figures represent the recall (left) and precision (right) of prompt type 1 ("detect errors") minus type ("evaluate the response"). Prompt type decreases recall on most models, but there is no consistent trend in precision.
Genme
tiet Recall
75
Cut in frecision
20
Minri
Lame 215
Matral Ex73
Deed
-00
GFT33 Tutt
-25
MathiGien FgFartVV AnsCls
Mathitien FgFacrV/ AnsCi
-40
MathiCen FiFacctV AnsCis
Mathiien FgFactv AncC
UP14 28121
Uni 301nt
GPI-4 (0813)
Liama 2 /Ob
GP1-4 (0613)
Liama /0b
(a) Prompt Type 1
10
Diff in Pracision
in Recall
30
20
Marrl
Lame 700
25
Dewn 5
D
-25
FgFactV
MathiGan FoFactV Llama 2 70b
FoFactV
FaFactV Laaa 70b
GPT-4 (0613)
GPT-4 (0613)
(b) Prompt Type 2
Figure 11: Performance prompts with different option orders shows positional bias. The values in the figures represent the recall (left) and precision (right) of prompt type B ("no error" " option first) minus prompt type A ("error" option first). In most models, putting the "no error" option before the "error" option increases recall (more errors missed by error detection). However, there is no consistent trend in precision.
33

---

Published as a conference paper at COLM 2024
H
Model Access Details
The model responses included in ReaLMistake were collected between February 1 and 21, 2024. The model responses for evaluating the error detection performance in Section were collected between February 28 and March 20, 2024.
We use temperature 0.0 (do sample=False for open-source models). For generating re- sponses in Mist.e (GPT-4 and Llama 2 70B), we the maximum possible number of new tokens for each model. For LLM-based error detection, we use the maximum number of new tokens 4,096 for all models.
H.1 Closed Source LLMs
OpenAl's Models. We experiment with GPT-3.5 Turbo (Brown et al. et al. 2022) and GPT-4 (OpenAl 2023) through the OpenAI's API
2020 Ouyang Specifically, we use 061- for generating the initial responses in ReaLMistake. We also use 1pt-3.5-turbo--e
and gpt-4-0125-previa for error detection in Section
Claude Opus. We experiment with Claude 3 Opus (Anthropic 2024
through the Claude
API
A Specifically, we use
Gemini 1.0 Pro. We experiment with Gemini 1.0 Pro (Gemini Team Google 2023 Google's Vertex AT API Specifically,
through
comini--10-ppe
H.2 Open Source LLMs
We use open-source LLMs provided on Hugging Face Hub We generate prompts by using the using any system prompt. We evaluate the open-source models by using eight NVIDIA A100 SXM4 GPUs.
chat_templast function provided by the tokenizer for each model without
Llama 2.
We use Llama 70B (Touvron et al. 2023) for generating the initial responses in ReaLMistake. For error detection, also evaluate Llama 2 13B. Specifically, we use 1ata-1lama/Llama-21111 13b--chthhhhh and 2-0b--mmaa/
Gemma 7B.
We use Gemma 7b model [Google 2024
for evaluating error detection.
Specifically, we use google/gemma-7011 gemma-7b-it.
Mixtral 8x7B. We Mistral 7B (Jiagg et al. 2023 and Mixtral 8x7B (siagg et al. 2024 for evaluating error detection. Specifically, we use misrralai/Misttrall----- Innttuuuutttntttttttcotm and
Qwen 1.5. We use Qwen 1.5 (14B and 72B) (Bai et al. 2023 Qwen Team 2024 for evaluating error detection. Specifically, we use Twen/Quent 1--11.5-11 5- 4B- Chahat and Qwen/ Qwen1 5-72B-C7 -hatt.
10 Whttpe://platfoorm.ppeaa wwt.:/orrm..mmaii...oo com
httpe::/www.antrrooicccoo com
12 httpes/ccloodl.ooole....oovvvvee ai
htrpa///aggiingaae..cc/ootid
34

---

Published as a conference paper at COLM 2024
I Data Example of ReaLMistake and Error Detection Outputs
This section shows examples from ReaLMis in Figure 2 and outputs from LLM-based error detectors (Section 4). The manual analysis for responses from LLM-based error detectors categorizes mistakes made by detectors in Figure 6
1.1 Math Word Problem Generation (MathGen)
MathGen Input
Generate a math word problem that satisfies the following requirements. First, provide the generated question. Second, generate step-by-step solution for the generated question.
* When the requirements instruct only to use integers in the problem or solution, you may use integers with units (e.g., 15kg, 10%) but should not use fractions (e.g., 1/2) if not specified. This applies to all data types.
The term "fractions" in the requirements refers to the numbers expressed using integer numerator and denominator, and does not include decimals.
General Requirements:
The generated question should not include sub-problems.
* The generated question should be valid and answerable.
The generated question should not be ambiguous and the final answer the generated questions must be unique.
* The generated question should not include information that not necessary question.
the The solution should include only one solution without any unnecessary procedures. You should not include verification of the final answer after solving the problem.
* The solution should not include any approximation.
The question and solution should use reasonable data types and values for the domain of the question. For example, the question is about the number people, the answer should be positive integer.
Specific Requirements:
The problem requires an understanding of relative speed and time in a circular track scenario. * "Te problem statement should include the phrase "Marla completes 32 laps around the track per hour".
The solution involves division and rounding off the nearest whole number.
The solution should include [integers, numbers with one decimal place, numbers with 2 decimal places, fractions] and should not include any other type of number.
35

---

Published as a conference paper at COLM 2024
MathGen Response from GPT-4
Question:
Marla completes 32 laps around the track per hour. If the track is 400 meters long, how many kilometers does Marla run 2.5 hours?
Solution:
Step 1: First, we need to find out how many meters Marla runs in one hour. Since she completes 32 laps and each lap is 400 meters, we multiply these two numbers together. 32 laps/hour' 400 meters/ 12800 meters/hour
Step 2: To convert meters to kilometers, divide by 1000 (since there 1000 meters in a kilometer). 12800 meters/hour 1000 12.8 kilometers/hou
Step 3: Now, we need to find out how many kilometers Marla runs in hours. We multiply her speed by the time. 12.8 kilometers/ /hour 2.5 hours 32 kilometers
So, Marla runs 32 kilometers in 2.5 hours.
Human Expert Annotation on the Response from GPT-4
Error Label: error
Error Categories: Instruction-Folloiing
Explanation: The question does not involve relative speed and rounding off the nearest whole number, which are in the requirements. In addition, the solution does not include numbers with two decimal places.
Error Detector based GPT-4-0125 with Prompt Type 1-A
The model response violates the specific requirement that the solution include "[integers, numbers with decimal place, numbers with 2 decimal places, fractions]" and should not include any other type of number. The solution provided uses a time value "2.5 hours," " which is number with decimal place, aligning with the requirements. However, the specific requirements also mention that the solution involves division and rounding off to the nearest whole number, which was adhered to in the provided solution. The solution directly calculates the distance Marla runs in 2.5 hours without involving any rounding off to the nearest whole number instructed.
Therefore, model response contains an error.
Manual Analysis:
Correct binary label, insufficient reasoning
The response does not mention why the fourth specific requirement not satisfied. In addition, it does not mention the first requirement about "relative speed".
36

---

Published as a conference paper at COLM 2024
Error Detector based on Claude 3 Opus with Prompt Type 1-A
The model response follows all the general and specific requirements outlined in the model input. The generated question is valid, unambiguous, and does not include unnecessary information. The solution step-by-step, does not involve approximations, and includes only one solution without unnecessary procedures.
The specific requirements are also met:
The problem involves relative speed and time a circular track scenario.
The phrase "Marla completes 32 laps around the track per hour" is included in the problem statement.
The solution involves division and rounding off to the nearest whole number (32 kilometers in the final answer).
The solution includes integers (32 laps/hour, 400 meters/lap, 1000 meters/ kilometer) and numbers with one decimal place (12.8 kilometers /hour), as required. It does not include any other type of number.
Therefore, the model response contains no error.
Manual Analysis:
Wrong binary label, wrong reasoning
The detector misses some requirements (e.g,, data types) and makes mistake by saying "The problem involves relative speed".
Error Detector based on Qwen 1.5 72B with Prompt Type 1-A
The model response presents a math word problem involving relative speed and time on circular track, instructed. It includes the required phrase "Marla completes 32 laps around the track per hour" and provides a step-by-step solution that involves division and rounding to the nearest whole number. The solution uses integers, numbers with one decimal place, and numbers with two decimal places, as specified. There are no sub-problems, unnecessary information, ambiguous elements in the question, and the answer is unique.
Therefore, the model response contains no
Wrong binary label, wrong reasoning The detector makes mistake by saying "The solution uses ... numbers with two decimal places".
Manual Analysis:
Error Detector based on Mixtral 8x7B with Prompt Type 1-A
The model response contains no error. The question and solution meet the requirements specified in the model input. The question is clear, concise, and unambiguous, and the solution accurate and correctly follows the steps outlined. The solution uses only integers and decimal numbers specified, and the final answer is rounded off to the nearest whole number. The problem also correctly includes the phrase "Marla completes 32 laps around the track per hour" " required. Therefore, the model response contains no error.
Manual Analysis:
Wrong binary label, wrong reasoning
The detector makes mistake by saying" the final answer is rounded off to the nearest whole number".
37

---

Published as a conference paper at COLM 2024
Error Detector based on Llama 70B with Prompt Type 1-A
The model response contains
The error in step 3 of the solution, where the model divides the speed by the time to find the distance traveled. The correct calculation should be:
12.8 kilometers/hour" 2.5 hours 3200 meters
Then, to convert meters to kilometers, we divide by 1000:
3200 meters 1000 3.2 kilometers
Therefore, Marla runs 3.2 kilometers in 2.5 hours, not 32 kilometers the model sponse states.
The specific requirements of the model input instructed the model to use integers, numbers with one decimal place, numbers with decimal places, and fractions. The model response uses integers and numbers with one decimal place, but not fractions. This is not an error, but it is worth noting that the model could have used fractions to provide more precise answer.
In conclusion, the model response contains an error in the calculation of the distance traveled and should be corrected to 3.2 kilometers.
Manual Analysis:
Wrong binary label, wrong reasoning
The detector makes many mistakes.
38

---

Published as a conference paper at COLM 2024
MathGen Ressonse from Llama 2 70B
Question:
Marla competes in bicycle race circular track with diameter of 800 meters. She completes laps around the track per hour. she maintains constant speed throughout the race, how many minutes will it take her to complete one lap?
Solution:
Let's start by converting the diameter of the track from meters to kilometers: 800 0.8 km
Next, calculate Marla's speed in kilometers per hour:
Speed = Distance / Time
0.8 km 1 h
0.8 km/h
Now, we need to convert Marla's speed from kilometers per hour to meters per sec- ond:
Speed ()/s) Speed (km/h) 1000
0.8 km/h 1000
800 m/s
Since Marla completes 32 laps per hour, calculate the time it takes her to plete one lap:
Time Distance / Speed 0.8 km / 800 m/s 5
find out how many minutes it takes Marla to complete one lap, we can divide the time in seconds by 60: Minutes = Time (s)
=0.8 60
=0.1133 min
Rounding up to the nearest whole number, we get: Minutes = 1 min
Therefore, it will take Marla approximately minute complete one lap around the track.
Human Expert Annotation on the Response from Llama 70B
Error Label: error
Error Categories: Instruction-Followingg Reasoning Correctness
Explanation: The question does not involve relative speed. It also includes unnecessary infor-
mation (the length of the track is not necessary). The solution makes mistakes in calculating her speed.
39

---

Published as a conference paper at COLM 2024
I.2 Fine-grained Fact Verification (FgFactV)
FgFactV Input
We provide pair of claim and evidence. The claim is sentence in a Wikipedia article, and the evidence includes sentences retrieved from web articles.
Your task to check all pieces of information in the claim and state reasoning on whether each part of the claim is supported by the evidence not.
Your response should end with of the following sentences:
* "Therefore, the claim is supported." " Select this option if all information in the claim is supported by the evidence.
"Therefore, the claim is not supported." Select this option unless all information in the claim supported by the evidence. other words, select this option if any part of the claim is not mentioned in the evidence or is contradicted by the evidence.
"Therefore, it is not clear whether the claim supported not." Select this option you cannot conclude whether the claim is supported not because of ambiguity in the verification process.
In your reasoning, provide line numbers of the evidence sentences supporting each part of the claim.
You should not any external knowledge other than the provided evidence.
Your response should not include anything that is not necessary for verifying the claim.
Claim (a sentence in the section "Background." in the article "Mick Adams"): Adams was born in Widnes, Lancashire, England, and he died aged 65 in Tamworth, New South Wales, Australia.
Evidence (sentences retrieved from web articles): line (meta data) TITLE: Mick Adams: Ex-Widnes rugby league captain dies, aged 65 BBC Sport
line 2: ## Accessibility links
line 3: BBCSPORT All Sport All Sport Home Football Formula 1 Cricket Rugby U Tennis
Golf Athletics Cycling BBC Sport Home Football Formula Cricket Rugby Union Tennis Golf Athletics Cycling
line 6: ### Events
line 7: Winter Olympics Commonwealthh Games Men's World Cup Women's World Cup
line 9: England Scotland Wales Northern Ireland
line 10: ### World Sport
line 11: European Football Champions League African Football Sport Africa Rugby League
Scores Fixtures Tables All Teams Scores & Fixtures Tables All Teams Rugby League Home
line 12: # Mick Adams: Ex-Widnes rugby league captain dies, aged 65
line 14: Mick Adams was Great Britain international, and former captain at Widnes
line 17: He part of four Wembley triumphs, and scored 68 tries in more than 400 appearances culminating in 1992's addition to the Chemics' Hall of Fame.
line 18: The bowkerower, who passed away Thursday, also represented Great Britain and England during his career.
line 19: "The club would like to pass their sincere condolences to Mick's family and friends at this time," statement said.
line 20: Widnes will hold a minute's applause for Adams before Sunday's Super League fixture against Castleford.
line 21: Former Widnes winger Martin Offiah paid his respect Mick Adams
line 22: Former Widnes team-mate Eddie Cunningham gave this tribute to Adams
line 23: ## Find out more
line 24: #N# The Super League Show
40

---

Published as a conference paper at COLM 2024
FactV Response from GPT-4
The claim states that Mick Adams born in Widnes, Lancashire, England, and died aged in Tamworth, New South Wales, Australia. The evidence supports the first part of the claim, confirming that Adams was indeed born in Widnes, England (line 0, line 14). However, the evidence does not provide any information about where Adams died, specifically whether it was in Tamworth, New South Wales, Australia. Therefore, it not clear whether the claim is supported or not.
Human Expert Annotation on the Response from GPT-4
Error Label: error
Error Categories: cnttruction-Following Reasoning Correctness
Explanation: Line 14 does not mention that he was born in Widnes. The response does not mention his age when died. addition, the conclusion should be "not supported".
Error Detector based on GPT-4-0125 with Prompt Type 1-A
The model response correctly identifies that the evidence supports the claim that Mick Adams was born in Widnes, England, as indicated by lines 0 and 14. However, the model response incorrectly states that the evidence does not provide any information about where Adams died. The evidence does not explicitly mention Tamworth, New South Wales, Australia, as the place of death, which is crucial part of the claim that needs verification. Since the model response concludes that "it is not clear whether the claim is supported or not" without verifying the specific location of Adams' death, it fails to follow the instruction that requires clear conclusion based on the evidence provided. The correct conclusion, based on the instructions and the evidence provided, should have been that the claim is not supported, as there is no evidence to confirm the part of the claim regarding the place of death.
Therefore, the model response contains an error.
Manual Analysis:
Correct binary label, wrong reasoning
The detector makes mistake by saying "the evidence supports the claim that Mick Adams was born in Widnes, England" and "the model response incorrectly states that the evidence does not provide any information about where Adams died".
Error Detector based on Claude 3 Opus with Prompt Type 1-A
The model response correctly identifies that evidence supports Adams being born Widnes, England. It also correctly points out that the evidence does not mention where Adams died, and therefore it cannot be determined if the claim about his death location supported or not.
However, the model response concludes with "Therefore, it is not clear whether the claim is supported not." " This does not follow the instructions in the model input, which state that the response should end with either "Therefore, the claim supported.", "Therefore, the claim is not supported." ", or "Therefore, is not clear whether the claim supported not." The model should have concluded with "Therefore, the claim is not supported." since part of the claim (the death location) is not supported by the given evidence.
Therefore, the model response contains an error.
Manual Analysis:
Correct binary label, wrong reasoning
The detector makes a mistake by saying "The model response correctly identifies that the evidence supports Adams being born in Widnes, England".
41

---

Published as a conference paper at COLM 2024
Error Detector based on Qwen 1.5 72B with Prompt Type 1-A
The model response correctly identifies that the evidence supports the fact that Mick Adams born in Widnes, England. However, it also correctly acknowledges that there is no information in the evidence about where Adams died. The model response appropriately concludes with "Therefore, it not clear whether the claim is supported or not." This indicates that the model has followed the instructions and requirements the model input. Therefore, the model response contains no error.
Manual Analysis:
Wrong binary label, wrong reasoning
The detector makes mistake by saying "The model response correctly identifies that the evidence supports the fact that Mick Adams was born Widnes" and "The model response appropriately concludes with "Therefore, it is not clear whether the claim is supported not."" ""
Error Detector based Mixtral 7x8B with Prompt Type 1-A
The model response correctly identifies that the first part the claim is supported by the evidence. However, the response should not conclude that it is not clear whether the claim is supported or not just because the evidence does not mention where Adams died. The model should only make determination about the supported status of the claim based on the provided evidence. Since the evidence does not contradict the claim that Adams died in Tamworth, New South Wales, Australia, it is possible that the claim is true even if the evidence does not specifically support Therefore, the model response contains an error.
Manual Analysis:
Correct binary label, wrong reasoning
The detector makes mistake by saying "The model response correctly identifies that the first part of the claim is supported". In addition, reasoning of "Since the evidence does not contradict the claim..." " is wrong.
Error Detector based on Llama 2 70B with Prompt Type 1-A
The model response contains an error.
The eгror in the model response is the conclusion that is not clear whether the claim is supported not. The evidence provided in the model input does not mention anything about Mick Adams' death or where he died. Therefore, the model response should have concluded that the claim is not supported, there is no evidence to support the second part of the claim regarding Adams' death.
The correct response should be:
"Therefore, the claim is not supported."
The evidence only supports the first part of the claim, which states that Mick Adams was born in Widnes, Lancashire, England. The evidence does not provide any information about Adams' death where he died, therefore, it does not support the second part of the claim.
Manual Analysis:
Correct binary label, wrong reasoning
The detector makes mistake by saying "The evidence only supports the first part of the claim".
42

---

Published as a conference paper at COLM 2024
FgFactV Response from Llama 2 70B Therefore, is not clear whether the claim supported not.
Reasoning:
'The first part of claim, " "Adams was born Widnes, Lancashire, England," " is not directly supported by the provided evidence. None of the evidence sentences mention Adams' birthplace or date birth.
The second part of the claim, "and he died aged 65 lamworth, New South Wales, Australia," is also not directly supported by the provided evidence. While the evidence does mention Adams' death age (line 18) and location (line 19), it does not explicitly state his age the time of death as 65.
well-known rugby player, which could indirectly support the claim that he was born and died in the places mentioned. For example, line mentions Adams' role former captain Widnes, and line 17 highlights his achievements in the sport.
However, some evidence sentences do support the idea that Adams
Overall, while there is no direct evidence supporting every part of the claim, there is enough related information in the evidence to suggest that the claim might be true. Yet, due to the lack of explicit support, it is still unclear whether the claim is entirely accurate. Hence, the response is "Therefore, it not clear whether the claim is supported or not."
Human Expert Annotation on the Response from Llama 70B
Error Label: error
Error Categories: Instruction-Following, Reasoning Correctness
Explanation: The conclusion should be "not supported". In addition, the evidence does not include Eric's name. addition, misses hid death age in line Line 19 does not support the death place.
43

---

Published as a conference paper at COLM 2024
1.3 Answerability Classification (AnsCls)
AnsCls Input
Answer the following question. Assume you are on Jan 18, 2018 and questions that require knowledge after this date should be classified unanswerable. Provide reasoning first, then state your answer the end.
* Your reasoning should not include any factual errors.
If the question includes incorrect information or is invalid (e.g, inconsistent date assumption or ambiguous due to insufficient information), your answer the end of your response should clearly state that the question is unanswerable.
* future event asked the past tense, the question should be classified as unanswerable.
Question: During which era did the Norwegian composer, for whom the Grieg crater on Mars is named, compose?
AnsCls Response from GPT-4
The Norwegian composer for whom the Grieg crater Mars is named is Edvard Grieg. Edvard Grieg was composer during the Romantic era of music, which spanned from approximately 1800 to 1900. Grieg was born in 1843 and died in 1907, so his active years composer fell within this period. Therefore, the era during which Grieg composed was Romantic era.
Answer: The Romantic era.
Human Expert Annotation on the Response from GPT-4
Error Label: error
Error Categories: Parameterized Knowledge
Explanation: The crater is on Mercury.
Error Detector based on GPT-4-0125 with Prompt Type 1-A
The model response accurately identifies Edvard Grieg the Norwegian composer for whom the Grieg crater on Mars is named and correctly states that composed during the Romantic era, which is consistent with the historical period of approximately 1800 to 1900. The response provides a clear reasoning for the answer, mentioning Grieg's lifespan (1843-1907) and situating his active years within the Romantic era. The question did not include incorrect information, was not invalid, and did not ask about a future event in the past tense. Therefore, the model response follows all instructions and requirements in the model input.
Therefore, the model response contains no error.
Manual Analysis:
Wrong binary label, wrong reasoning
The detector makes mistake by saying "The model response accurately identifies Edvard Grieg as the Norwegian composer for whom the Grieg crater on Mars is named ...".
44

---

Published as a conference paper at COLM 2024
Error Detector based on Claude 3 Opus with Prompt Type 1-A
The model response correctly identifies Edvard Grieg as the Norwegian composer for whom the Grieg crater on Mars is named. also accurately states that Grieg composed during the Romantic era, which spanned from approximately 1800 1900, and that Grieg's life (1843-1907) fell within this period. The response provides clear reasoning and a concise answer to the question, without including any factual errors or inconsistencies with the given date assumption January 18, 2018.
Therefore, model response contains no error.
Manual Analysis:
Wrong binary label, wrong reasoning
The detector makes mistake by saying "The model response correctly identifies Edvard Grieg as the Norwegian composer for whom the Grieg crater on Mars is named".
Error Detector based on Qwen 1.5 72B with Prompt Type 1-A
The Norwegian composer Edvard Grieg, to whom the Grieg crater on Mars is dedicated, indeed lived and composed during the Romantic era, which lasted from around 1800 to 1900. Grieg's birth year was 1843, and he passed away in 1907, so his compositional career fell within this time frame. The model response accurately provides the reasoning and the answer. Therefore, the model response contains no error.
Manual Analysis:
Wrong binary label, wrong reasoning
The detector makes a mistake by saying "The Norwegian composer Edvard Grieg, to whom the Grieg crater Mars ..."
Error Detector based on Mixtral 7x8B with Prompt Type 1-A
The model response provides accurate information about Edvard Grieg and the Romantic era of music. However, the model input states that the answer should provided assuming the date is January 18, 2018. The response does not explicitly state that the answer is based on this assumption. Although the answer correct, the lack of explicit reference to the date assumption is minor issue that could potentially lead to confusion.
Therefore, the model response contains error.
Manual Analysis:
Correct binary label, wrong reasoning
It is true that the model response does not provide explicit reasoning the date assumption, but does not mean that the response is wrong.
45

---

Published as a conference paper at COLM 2024
Error Detector based on Llama 70B with Prompt Type 1-A
The model response contains
The eгror in the model response lies in the fact that it assumes that the question is ask- ing about the composer's 's active years composer, which is not explicitly stated in the question. The question only asks when the composer composed, without specifying whether it refers to their active years not. Therefore, the model response's assumption that the composer's 's active years fall within the Romantic era not necessarily true.
Additionally, the model response does not address the fact that the question is asking about future event (the composition of music) in the past tense. This inconsistency in tense should be addressed in the response, and the model should recognize that the question is unanswerable due to this ambiguity.
Therefore, the model response contains an error, and should be classified as such.
Manual Analysis:
Correct binary label, wrong reasoning
There are many mistakes.
AnsCls Response from Llama 70B
The Norwegian composer for whom the Grieg crater on Mars is named is Edvard Grieg, He was born in 1843 and died in 1907. Therefore, he composed during the Romantic era, specifically from the mid-19th century to the early 20th century.
However, it's important to note that the Grieg crater on Mars was not named until much later, in 1973, by the International Astronomical Union So while Edvard Grieg did compose music during his lifetime, it's not possible for him to have composed music during the era when the crater was named, as that would have required him to still alive and composing decades after his death.
Therefore, must classify this question unanswerable due its reliance impossible scenario.
Human Expert Annotation on the Response from Llama 2 70B
Error Label: error
Error Categories: Reasoning Correctness
Explanation:
The reasoning in the second paragraph of the response is invalid.
46


Reviews:
[{"review_id": "cmDIAoXrLB", "reviewer": "Reviewer_9NZk", "summary": "The paper introduces ReaLMistake, which the authors claim is the first error detection benchmarking suite consisting of objective, realistic and diverse errors made by LLMs. ReaLMistake contains three tasks that introduce objectively assessable errors in four categories (reasoning correctness, instruction-following, context-faithfulness, and parameterised knowledge), eliciting naturally observed and diverse errors in responses of GPT-3 and Llama 270B annotated by experts on three tasks three tasks (Math Word Problem Generation, Fine-grained Fact Verification, and Answerability Classification). The benchmark includes error annotations (binary error label, error categories, and human explanation about errors) on responses from GPT-4 and Llama 2 70B on the three tasks. The authors show that top LLMs like GPT-4 detect errors made LLMs at very low recall, and all LLM-based error detectors perform much worse than humans, that explanations by LLM-based error detectors lack reliability and LLM-based error detection is sensitive to small changes in prompts but remains challenging to improve, and finally, popular approaches to improve LLMs, including self-consistency and majority vote, do not improve the error detection performances. The authors promise to make the code and data available on acceptance.\n\nOverall the quality of the paper is quite high in terms of the detail of the description of the dataset, though this is somewhat in part due to the lengthy Appendices making very clear what the prompts and responses were. ReaLMistake undoubtedly could be part of a good resource for evaluation for LLM error detection, though it is limited in scope, as partially admitted to by the authors. The presentation overall is quite strong and clear, though see below for some queries about it which should be addressed. It seems more incremental research than highly novel, as the authors define the benchmarking dataset as an intersection of what has gone before, however the paper claims ReaLMistake is novel in focusing on errors that are unambiguously judged as such by human annotators (though see concerns on this claim below). The approach is quite principled scientifically, though I would dispute the claim of unambiguously annotated error data as being essential for good error detection benchmarking for LLMs, and perhaps if accepted, the authors can reduce the strength of that claim and describe alternatives where multiple annotator’s judgments are included (see below).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed comments and valuable suggestions!\n\n> My main reservation about the paper, which could be corrected if claims were slightly toned down [...]\n\nThank you for your suggestion. We agree that it is technically possible to study ambiguous errors by annotating each instance with multiple annotators, which can be a future work. We will discuss this point and tone down our claim.\n\nIn this work, we choose to use unambiguous errors because of the following advantages:\n\n(1) Not affected by annotators' subjectivity.\n\n(2) Reliable evaluation: Unambiguous errors can reliably evaluate error detectors because of the small noise.\n\n(3) Binary labels: We can collect binary labels, making the evaluation of binary predictions easy.\n\n(4) Annotation cost: We do not need to collect many annotations for each instance.\n\n> It is not clear whether the explanations were agreed upon between the annotators.\n\nThank you for your suggestion. We agree that this is a promising direction for further verifying that the errors in ReaLMistake are unambiguous.\n\n> [...] Table 1 suggests ‘MT-Bench, PandaLM, LLMEvalˆ2 Multiple NLP Tasks’ has more criteria and in multiple domains\n\n> As per the above, it is not clear how novel the approach is [...]\n\nWe would like to clarify that those benchmarks are for evaluating **pairwise (ranking)** evaluators, which compare multiple responses, while our benchmark is for evaluating **pointwise** evaluators. Pairwise benchmarks cannot be directly used to evaluate error detectors because they include relative evaluation between multiple responses.\n\nThis table intends to show that benchmarks for **pointwise** evaluation are lacking, although there are diverse **pairwise** benchmarks. We will improve it in future versions.\n\n> I would reduce the strength of the claims around ambiguity [...]\n\nThank you for these suggestions. We will definitely improve the use of the terms to describe the properties of our benchmark more precisely.\n\n> Please calculate and report Cohen’s kappa [...]\n\nWe report F1 because we evaluate human performance, not inter-annotator agreement. Each instance in ReaLMistake is annotated by one annotator and verified by another, and after the benchmark is finalized, human performance is evaluated using error classifications by other annotators.\n\nFor reference, when we consider the labels in ReaLMistake and human classification (for human performance) as two annotators, Cohen's kappa is 0.80 for GPT-4 and 0.96 for Llama 2 70B.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There is a useful data contribution of the 900 instances of error annotations by experts, including human explanations for the judgment of an error on responses from GPT-4-0613. This is potentially very useful for certain kinds of error detection benchmarking and is challenging, as shown by the authors who show that near state-of-the-art models such as GPT-4x exhibit a very high error rate in detecting errors in certain tasks with the prompts designed by the authors.\n\nThe statistics of the ReaLMistake benchmark are clear and show a good range of errors in the different dimensions defined by the authors.\n\nThe Appendices are very useful as a clear exemplification of prompts and further results, which when included, mean the reproducibility of the paper is quite high from the text alone.\n\nOverall the paper is well-written, and fairly easy for readers to understand the method and approach employed.", "reasons_to_reject": "My main reservation about the paper, which could be corrected if claims were slightly toned down, is in terms of the motivation of the form of the benchmark: the authors claim they are against human subjective ratings of errors in a response, or at least instances which are not universally agreed upon for their label (e.g. “We cannot evaluate error detectors if humans cannot determine whether LLM responses include errors”). However, this claim is too strong, and it is not clear how the inclusion of multiple (potentially differing) annotations would make things worse for a benchmark – they are certainly a challenge to be solved going forward, but rigorous inter-annotator agreement studies in creating these benchmark datasets could be a way to deal with ambiguous/non-unanimous labelling - in fact it could even be a good idea to include the differing annotations in the benchmark. The challenge would then shift away from limiting a dataset to unambiguous error instances only to developing systems that give similar confidence levels to the level of inter-annotator agreement. Even in some cases there are mathematics-based prompts, there is still room for ambiguity in the interpretation of verbal prompts.\n\nIt is not clear whether the explanations were agreed upon between the annotators.\n\nIt is not made clear how novel the benchmark is as Table 1 suggests ‘MT-Bench, PandaLM, LLMEvalˆ2 Multiple NLP Tasks’ has more criteria and in multiple domains.", "questions_to_authors": "I would reduce the strength of the claims around ambiguity and frame your dataset as collecting error judgements which are low in ambiguity (as illustrated by the 95% F1 score of agreement), rather than ‘objective’. This is scientifically too strong.\n\nThe claim of it being a diverse set of errors might also be a bit strong, as it is dependent on the diversity and difficulty of the original tasks – I would reduce this and discuss that there is more work to be done to enlarge the scope of error type.\n\nAs per the above, it is not clear how novel the approach is, given Table 1’s column ticks – please further clarify how it is going further than the top row’s system in the text, particularly if you lessen the claim that human inter-subjective disagreement is a bad thing for annotation overall.\n\nPlease calculate and report Cohen’s kappa for the agreement score, which is similar to F1 but more comparable to other inter-annotator agreement scores.\n\nThere are some errors in phrasing throughout which require a proof-read - e.g.\n-\tUse ‘LLM-based’ (rather than LLMs-based)\n-\tUse determiners in places like ‘We introduce ReaLMistake benchmark’ –> ‘We introduce the ReaLMistake benchmark’", "extra_scores": {}}, {"review_id": "ovzbXHHqBd", "reviewer": "Reviewer_3EKz", "summary": "The paper introduces a benchmark for detecting errors in LLM-generated text. The dataset is generated by modifying several existing datasets (e.g. AQuA, HotpotQA), by prompting an LLM to modify existing questions according to certain instructions. The LLM-generated responses are manually filtered for those that failed to correctly follow the instructions. After constructing this dataset, the paper evaluates a number of LLMs, and finds low recall in identifying errors. The paper also finds that, even when errors are correctly identified, the LLM-generated explanations of these errors are frequently incorrect.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful comments and valuable feedback! \n\n> It would be interesting to understand what factors make an error hard to identify. The results in Table 4 are fairly coarse grained. Does the dataset support more fine-grained evaluations?\n\nOur analysis shows that error detection on MathGen (which includes errors in instruction following) is relatively easy for strong models, while the other tasks are difficult for all models. It suggests that instruction-following errors are relatively easy for detectors, but other types of errors (especially context-faithfulness and knowledge) are challenging.\n\nReaLMistake includes information that can be used for more detailed analysis: error categories and human explanations about the errors. We will provide more analysis in future versions.\n\n> [...] it would be useful to evaluate prompting strategies more comprehensively, beyond what is described in Section 4.3.\n\nBeyond Section 4.3, in Section 4.4 and Appendix A.3, we also evaluate a prompting strategy that provides detailed instructions for evaluation steps in prompts, which is motivated by G-Eval. However, it does not improve performance.\n\nWe agree that more analysis of prompting strategies, such as decomposing responses, is a promising direction for future research. \n\n> It seems like it should be possible to improve recall at the cost of lower precision. How large is that cost?\n\nAs shown in Figures 9 and 10, we observe that the recall of strong models (e.g., GPT-4) is not sensitive to prompt engineering. Therefore, as mentioned in Section 4.3, we conclude that there is no easy way to improve the recall of strong models, even at the cost of lower precision.\n\n> The errors are generated from a limited number of templates on three tasks. It is uncertain how generalizable the results will be to other types of errors, and those that occur in the wild. The paper is nonetheless a good starting point.\n\nReaLMistake is suitable for evaluating error detectors because it includes unambiguous errors in LLM responses, collected using well-designed tasks. In addition, to enable our benchmark to provide generalizable analysis, we include three diverse tasks and four error categories widely observed in LLM responses.\n\nWe agree that creating more diverse benchmarks is a promising direction for future research. However, collecting unambiguous errors is difficult in tasks in the wild, which are often open-ended and subjective.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is addressing an important use case. It would be useful to have LLM-based error detection in many applications.\n\n- The method for constructing the errors is scalable and reliable. The errors are manually verified, and there is very high inter-rater agreement.\n\n- The experimental results are interesting and convincing. The paper evaluates an appropriate set of models, and finds strikingly poor performance in most cases.\n\n- The paper is clearly written.", "reasons_to_reject": "- It would be interesting to understand what factors make an error hard to identify. The results in Table 4 are fairly coarse grained. Does the dataset support more fine-grained evaluations?\n\n- This is not very important, but it would be useful to evaluate prompting strategies more comprehensively, beyond what is described in Section 4.3. It seems like it should be possible to improve recall at the cost of lower precision. How large is that cost?\n\n- The errors are generated from a limited number of templates on three tasks. It is uncertain how generalizable the results will be to other types of errors, and those that occur in the wild. The paper is nonetheless a good starting point.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FLiEJyutsA", "reviewer": "Reviewer_sN45", "summary": "This paper proposes. new benchmark, ReaLMistake, that evaluates how good LLMs are at detecting objective errors. The authors first constructed three sets of instructions (math word problem generation, fine-grained fact verification, and answerability classification) and prompted GPT-4/Llama-2-70B to generate responses. Then the authors manually annotated the errors in the model responses with rationales. The error types are categorized into (1) instruction following, (2) reasoning correctness, (3) context-faithfulness, and (4) parameterized knowledge. All four are common errors of LLMs and can be objectively evaluated. The human agreement rate is high (95.9 F1).\n\nThen the authors evaluated a number of open-source/closed-source LLMs on the benchmark -- this includes both the accuracy on detecting the errors and a manual evaluation on the rationales generated by the models. The experiment results are quite interesting: (1) even the best models struggle at accurately predicting the errors. More surprisingly, there seems to be a negative correlation between model capacity (measured by Chatbot arena score) and the error detection recall. (2) The rationales generated by models can be very wrong even if the prediction is correct, especially for open-source models. (3) Prompt engineering is very difficult for error detection and even more advanced prompting techniques like self-consistency does not help.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments and valuable feedback!\n\n> (1) The source tasks are limited to synthetic tasks like question generation and fact verification. It would be more interesting to see such a dataset constructed on more diverse instructions such as ShareGPT/Alpaca/OpenAssistant.\n\nWe design realistic and meaningful tasks in ReaLMistake to collect unambiguous errors in LLM responses, which are useful for evaluating error detectors.\n\nWe agree that creating error detection benchmarks with more diverse instructions is desirable. However, collecting unambiguous error annotations in general tasks is challenging, as they are often open-ended and subjective.\n\n> (2) The negative correlation between the LLM capacity and error detection recall, though interesting, is a bit unintuitive. The authors also did not offer a reasonable guess of why this happens -- which makes me wonder if this is due to suboptimal designs of the prompt.\n\nWe consider there is no easy way to improve the prompts since strong LLMs consistently show low recall on multiple prompt designs (Sections 4.3, 4.4). Here are possible explanations for the negative correlation.\n\n* Strong LLMs can correctly classify responses without mistakes as \"no error\", resulting in high precision. However, they are conservative about detecting mistakes, resulting in low recall.\n* Weak LLMs do not have strong error detection capabilities and may return almost random predictions for difficult cases. As a result, recall can become higher than the conservative strong LLMs, although it results in low precision.\n\nWe will improve the explanation in future versions of our paper.\n\n> Would it make the model perform better if they are prompted with the error type definitions and are prompted to provide the error type as well?\n\nReaLMistake is designed to evaluate general-purpose error detectors, which do not assume any specific target task domains or error types. This setting is motivated by a real-world situation in which providers of LLM-based services try to detect any type of mistake in LLM responses automatically.\n\nIn the situation of targeting specific tasks, we agree that error detectors would be improved when designed for specific error types (e.g., prompted with the error type definitions, as you suggested).\n\n> The experiments would be more insightful as well if a per-type analysis is provided.\n\nThank you for your suggestion. We will provide a more detailed per-type analysis in future versions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) This is a well-motivated and high-quality dataset that will provide high utility to the language model community. The dataset annotations are manually curated by the authors and have a high human agreement rate. The schemas (source datasets and error types) are intuitive and comprehensive. Not only is error detection an important applications of LLMs, but this dataset (ReaLMistake) can serve as a benchmark for evaluating general LM abilities as well.\n\n(2) The experiments are comprehensive and insightful, especially (a) the negative correlation between model capabilities and error detection recall, and (b) the prompt sensitively of the task. Our community will benefit from those insights for better understanding LLM capabilities and building better LLMs/error detection models.", "reasons_to_reject": "(1) The source tasks are limited to synthetic tasks like question generation and fact verification. It would be more interesting to see such a dataset constructed on more diverse instructions such as ShareGPT/Alpaca/OpenAssistant.\n\n(2) The negative correlation between the LLM capacity and error detection recall, though interesting, is a bit unintuitive. The authors also did not offer a reasonable guess of why this happens -- which makes me wonder if this is due to suboptimal designs of the prompt.", "questions_to_authors": "If I understand correctly, the annotation includes the types of the error, but the performance on Table 4 and the prompts used by models are simply binary error detection. Would it make the model perform better if they are prompted with the error type definitions and are prompted to provide the error type as well? The experiments would be more insightful as well if a per-type analysis is provided.", "extra_scores": {}}, {"review_id": "ptLKCgbo88", "reviewer": "Reviewer_BGkA", "summary": "In their paper, the authors introduce an error detection benchmark called ReaLMistake. This benchmark evaluates four distinct types of errors—reasoning correctness, instruction-following, context-faithfulness, and parameterized knowledge—made by large language models (LLMs) across three selective natural language processing (NLP) tasks. The results reveal several key findings:\n\n1) Top-performing LLMs, including GPT-4, exhibit low recall in error detection tasks.\n2) While LLMs can correctly identify errors, their explanations remain unreliable.\n3) The performance of error detection is notably sensitive to subtle variations in prompts.\n4) Previous methods aimed at enhancing error detectors have proven ineffective.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments and valuable feedback!\n\n> The LLMs employed to generate the response (GPT4 and Llama2) could impact the reliability of the benchmark.\n\nI would appreciate it if you could elaborate on this point. We select GPT-4 and Llama 2 70B because they are widely adopted, and the study of error detection on their responses is well-motivated.\n\n> The ReaLMistake benchmark contains a relatively small number of samples compared to other benchmarks with binary labels (e.g., SummEdits and BIG-Bench Mistake).\n\nWhile we agree that larger is better, we consider the size of our benchmark to be reasonable for the following reasons:\n\n(1) Unambiguity: ReaLMistake includes unambiguous errors with high annotation quality. We expect that a relatively small benchmark can reliably evaluate error detectors.\n\n(2) Diversity: Our benchmark includes diverse instances in three tasks and four error categories.\n\n(3) Annotation Cost: SummEdits and Big-Bench Mistake use synthetic data or relatively simple tasks, which make human annotation easier. Our work focuses on natural mistakes by LLMs, whose error annotation is much more difficult.\n\n> In Table 2, I notice that the dataset is biased [...]\n\nOur benchmark includes errors in LLM responses in the natural distribution. We consider that the label distribution is not an issue if a sufficient number of instances are included for each label. When a balanced dataset is required, we can sub-sample instances with the majority label or up-sample instances with the minority label.\n\nOur experiments suggest that our benchmark includes a reasonable number of instances for each label because (1) our experiments capture performance differences between different models and (2) the state-of-the-art models are still far from human performance.\n\n> In Table 4, is it suitable to use your defined \"Random\" recall score as the baseline? [...]\n \nWe select our random baseline to simulate a situation in which we know the estimated error frequency of LLM responses. For example, when providers of LLM-based services try to detect mistakes in LLM responses, they often have the estimated error frequency of their LLMs and use it as a baseline.\n\nWe also note that there is no universally reasonable random baseline. For example, a random baseline with 50% error probability is relatively strong for error detection on GPT-4 responses (with around 50% errors) but very weak for Llama 2 70B responses (with around 75% errors).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1） The main contribution of this paper is to introduce a novel benchmark, ReaLMistake, for error detection in LLM responses. The benchmark contains various, objective and realistic errors made by LLMs, which is valuable for future research in this area.\n2) It provides an extensive analysis of the performances of 11 different LLMs, offering valuable insights into the error detection capabilities of popular LLMs. \n3) The paper is well-organized, presenting sufficient experiments, ablations, and detailed analysis.", "reasons_to_reject": "The LLMs employed to generate the response (GPT4 and Llama2) could impact the reliability of the benchmark. Another concern is about the sample size. The ReaLMistake benchmark contains a relatively small number of samples compared to other benchmarks with binary labels (e.g., SummEdits and BIG-Bench Mistake).", "questions_to_authors": "1) In Table 2, I notice that the dataset is biased, especially the error percentage of the Llama2 response is super high (75.0, 78.8, 77.5). How different data distribution affects the error detection results? What is the reasonable range of the error percentage?\n\n2) A following question of 1)： In Table 4, is it suitable to use your defined \"Random\" recall score as the baseline? Your dataset with Llama 2 is super biased and to give a more extreme example, if the error percentage is 100%, no model can beat the recall baseline(100). However, many models work better than random guess (recall=50). In this case, I think your claim \"LLM-based error detectors often perform worse than random baselines\" is a bit unsound.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper contributes to the timly problem of evaluating LLMs when used as evaluators. The authors introduce a welcome new error detection benchmark dataset, ReaLMistake, comprising organic errors with granular annotations. Using this, they then provide new empirical insights into the strengths and weaknesses of LLMs when used as evaluators. 

Overall, this is a nice, timely contribution which will be of broad interest and use to the COLM community.
Decision: Accept