Evaluating LLMs at Detecting Errors in LLM Responses

Authors: ryo  kamoi
sarkar  snigdha
sarathi  das
renze  lou
jihyun janice ahn
yilun  zhao
xiaoxin  lu
nan  zhang
yusen  zhang
ranran haoran zhang
sujeeth  reddy vummanthala
salika  dave
shaobo  qin
arman  cohan
wenpeng  yin
rui  zhang
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-03 17:35:59

Abstract: With Large Language Models (LLMs) being widely used across various tasks, detecting errors in their responses is increasingly crucial. However, little research has been conducted on error detection of LLM responses. Collecting error annotations on LLM responses is challenging due to the subjective nature of many NLP tasks, and thus previous research focuses on tasks of little practical value (e.g., word sorting) or limited error types (e.g., faithfulness in summarization). This work introduces ReaLMistake, the first error detection benchmark consisting of objective, realistic, and diverse errors made by LLMs. ReaLMistake contains three challenging and meaningful tasks that introduce objectively assessable errors in four categories (reasoning correctness, instruction-following, context-faithfulness, and parameterized knowledge), eliciting naturally observed and diverse errors in responses of GPT-4 and Llama 2 70B annotated by experts. We use ReaLMistake to evaluate error detectors based on 12 LLMs. Our findings show: 1) Top LLMs like GPT-4 and Claude 3 detect errors made by LLMs at very low recall, and all LLM-based error detectors perform much worse than humans. 2) Explanations by LLM-based error detectors lack reliability. 3) LLMs-based error detection is sensitive to small changes in prompts but remains challenging to improve. 4) Popular approaches to improving LLMs, including self-consistency and majority vote, do not improve the error detection performance. Our benchmark and code are provided at https://github.com/psunlpgroup/ReaLMistake.

Main paper: 
 introduction: 
 As LLMs have been increasingly used in real-world applications, it is critical to develop methods for automatically detecting errors in responses from LLMs (Bommasani et al., 2021;  LLM benchmarks often include Subjective TasksGenerate a math word problem that satisfies the following requirements. ...not sure if it is sufficiently unengaging to classify it as error...There is no error in the response. Compose an engaging travel blog post about a recent trip to Table 1 : Comparison between ReaLMistake and prior benchmarks for evaluating error detectors or evaluators for LLM responses. There is a deficiency in benchmarks for evaluating pointwise evaluation, although there are diverse benchmarks for evaluating pairwise evaluation (e.g., Zheng et al., 2023; Dubois et al., 2023; Zhang et al., 2023) . ReaLMistake is the first benchmark that includes objective and realistic errors in LLM responses in diverse categories. Srivastava et al., 2022) . However, there is a deficiency in an analysis specifically targeting error detection (Huang et al., 2024; Tyen et al., 2024) .An obstacle in studying error detection is the lack of benchmarks that include binary error annotations (i.e., whether the response contains errors or not) on objective, realistic, and diverse errors made by LLMs. 1 An NLP task should satisfy the following criteria to be suitable for collecting errors made by LLMs for the purpose of benchmarking error detection. First, to provide objective error labels, tasks should not involve subjectivity or ambiguity.As illustrated in Figure 1 , subjective tasks on which humans cannot provide reliable error annotation are not suitable for benchmarking error detection methods. For example, in text summarization, humans cannot objectively annotate binary error labels because the evaluation of "whether the generated summary properly includes important content in the source" (the "relevance" criterion) is subjective (Laban et al., 2023) . It is difficult to evaluate error detectors if humans cannot determine whether LLM responses include errors. As shown in Table 1 , tasks in benchmarks for ranking-based LLM evaluators (Zheng et al., 2023; Wang et al., 2023a; 2024) and criterion-specific error detectors (e.g., factuality evaluators) (Laban et al., 2023; Manakul et al., 2023) often involve subjectivity (Zeng et al., 2024) . Second, error detection benchmarks should be built on tasks that make LLMs introduce errors similar to errors in real-world applications. Some prior studies use tasks such as word sorting (Tyen et al., 2024) and chess (Wu et al., 2024) , which are objective but have different properties from practical uses of LLMs. Third, to provide a comprehensive testbed, the benchmarks need to include a variety of tasks to provide errors in diverse categories.Previous research in detecting mistakes in LLM responses focuses on errors of limited types (Laban et al., 2023; Manakul et al., 2023; Lightman et al., 2024) , as shown in Table 1 . Finally, tasks should be challenging for strong LLMs to make them introduce errors.To create tasks that satisfy these requirements, we propose an approach to design tasks so that they make LLMs introduce errors detected by objective, realistic, and diverse evaluation criteria. First, we identify four evaluation criteria (Reasoning Correctness, Instruction-Following, Context-Faithfulness, and Parameterized Knowledge) which can be objectively evaluated by humans and cover diverse errors in LLM responses. Subsequently, we create three tasks (Math Word Problem Generation, Fine-grained Fact Verification, and Answerability Classification) with the intention of making LLMs introduce errors detected by the four evaluation criteria, eliminating subjectivity from the error annotation process. This process ensures that the created tasks have desirable properties for collecting mistakes in LLM responses for error detection benchmarks. Using these three tasks, we introduce ReaLMistake, the first benchmark for evaluating error detection methods for LLM responses, consisting of objective, realistic, and diverse errors made by LLMs. As shown in Figure 2 , our benchmark includes error annotations (binary error label, error categories, and human explanation about errors) on responses from GPT-4 (OpenAI, 2023) and Llama 2 70B (Touvron et al., 2023) on the three tasks. The annotation process requires careful checking of the entire LLM responses, and 14 expert annotators spent 90 hours to provide high-quality annotations. of the refined outputs on downstream tasks and an analysis of the binary error detection step is lacking. 
realmistake: 
 The ReaLMistake benchmark includes 900 instances consisting of error annotations by experts (binary error label: error or no error, error categories, and human explanation) on responses from GPT-4-0613 (OpenAI, 2023) and Llama 2 70B (Touvron et al., 2023) on three tasks designed for providing objective error labels in diverse error categories, as shown in Table 2 . Figure 2 provides examples from the three tasks in our benchmark.3.1 How to Collect Objective, Diverse, and Realistic Errors in LLM Responses?The main challenge in creating evaluation benchmarks for error detection is in the process of collecting errors in LLM responses. There are various requirements for tasks used to collect errors in LLM responses for the purpose of evaluating error detection. First, human annotators should be able to provide objective error labels for LLM responses on the tasks to create a reliable testbed of error detection, as shown in Figure 1 . In addition, the tasks should introduce diverse types of errors observed in real-world applications of LLMs. Finally, the tasks should be difficult even for recent LLMs to make LLMs introduce mistakes. However, tasks should not be too difficult for humans to allow annotations and detailed analysis of error detection. We propose a strategy to construct tasks that satisfy these requirements.(1) Create Tasks that Make LLMs Introduce Objective, Realistic, and Diverse Errors. Evaluation of NLP tasks often involves subjectivity and even humans cannot provide objective binary error labels. For example, the text summarization task involves subjectivity in the "relevance" criterion, which evaluates "whether important content in the source is properly selected" (Laban et al., 2023) . To put it the other way around, humans can provide reliable error labels if tasks can be evaluated by objective criteria. To create tasks that make LLMs introduce objective, realistic, and diverse errors, we propose a bottom-up approach to design tasks so that they can be evaluated using diverse and objective evaluation criteria.We identify the following four criteria, which comprehensively cover objective criteria about errors in LLM responses that are caused by components of the inference process of LLMs: instructions and context in inputs, reasoning by LLMs, and knowledge in parameters. By creating tasks so that they can be evaluated by the following criteria, LLM responses to the tasks include realistic and diverse errors that can be objectively detected by humans.• Reasoning Correctness: Reasoning in responses (e.g., chain-of-thought (Wei et al., 2022b) ) should be logically valid. This criterion is objective because logical correctness is welldefined. Prior studies also annotate logical mistakes in reasoning by LLMs in a binary manner (Ling et al., 2023; Miao et al., 2023; Tyen et al., 2024; Lightman et al., 2024) . • Instruction-Following: Responses should follow the requirements specified in inputs (Zhou et al., 2023a; Zeng et al., 2024; Qin et al., 2024 ).2 Instruction-following is an objective criterion if the requirements are objective (Zhou et al., 2023a) . Adams was born in Widnes, Lancashire, ...Select 2-4 propertiesLine 0, 14 Retrieve the supporting sentences and randomly selected sentences in the article Create well-defined task instructions for fine-grained fact verificationThe • Context-Faithfulness: Responses should be faithful to the context provided in inputs and should not ignore any part of the context. If the input instructs not to use any resources other than the provided context, the response should not be affected by parameterized knowledge (Zhou et al., 2023b) . This criterion is often used as an objective "factuality" criterion for text summarization (Fabbri et al., 2021; Cao & Wang, 2021; Laban et al., 2023) .• Parameterized Knowledge: Responses should be factually correct. In this work, we focus on parameterized knowledge without allowing LLMs to access external resources. Prior studies also annotate mistakes in parameterized knowledge in a binary manner (Manakul et al., 2023; Min et al., 2023) , supporting the objectiveness of this criterion.(2) Create Tasks Challenging for LLMs but Feasible for Humans. We also highlight the difficulty in creating challenging tasks for recent strong LLMs. Classic objective tasks, such as mathematical reasoning at the grade school level, are too easy for recent LLMs (OpenAI, 2023; Anthropic, 2024) and we cannot collect mistakes. However, tasks that involve expert knowledge (Malaviya et al., 2023; Rein et al., 2023) or very long context (Wu et al., 2021; Pang et al., 2022) are not suitable for our purpose because it is too difficult for human annotators and users of this benchmark to conduct an analysis of error detection methods. Since it is challenging to create difficult tasks, some datasets include artificially created negative cases created by prompting LLMs to intentionally make mistakes (Zeng et al., 2024; Li et al., 2023a; Laban et al., 2023) , but artificial mistakes cannot evaluate error detection in realistic settings.To address these problems, we design tasks involving properties that are observed to be challenging for LLMs: (a) following requirements (Liu et al., 2024; Zhou et al., 2023a) , (b) comparison of multiple texts (Laban et al., 2023; Chen et al., 2023b) , and (c) detection of small mistakes in text (Cohen et al., 2023) . As shown in Table 2 , tasks in our benchmark make GPT-4 introduce errors in more than 50% of cases, while their inputs consist of a few hundred words that only involve high-school-level mathematics and knowledge in Wikipedia. These tasks allow us to efficiently collect errors made by LLMs.Our benchmark includes three challenging, objective, and realistic tasks designed for collecting diverse categories of errors in LLM responses. As discussed in the previous section, these tasks are designed so that they can be evaluated by the four objective and diverse criteria, eliminating the subjectivity in the error annotation process while ensuring the diversity. In addition, the three tasks are selected to involve skills required in real-world applications of LLMs (e.g., arithmetic reasoning, fact verification, and factual knowledge), in contrast to the objective tasks in prior work that have properties different from those of real-world uses of LLMs, such as chess (Wu et al., 2024) and word sorting (Tyen et al., 2024) . Figure 2 includes an example from each task. Figure 3 visualizes the creation processes.MathGen -Math Word Problem Generation is a task of generating math word problems that follow multiple requirements. This task is designed to be evaluated on Reasoning Correctness and Instruction-Following. This task instructs models to generate a question that follows all requirements in input (Instruction-Following) and also instructs to generate a solution for the generated question (Reasoning Correctness). Creating valid mathematical questions is challenging for LLMs and allows us to collect mistakes by LLMs without using advanced mathematical concepts, which are too difficult for annotators and users. Creation Process: We generate the requirements in inputs by using questions in AQuA (Ling et al., 2017) . Specifically, we use GPT-4 to generate nine properties (e.g., topics, mathematical concepts, types of numbers; listed in Table 3 ) of each pair of a question and solution in AQuA and select two to four properties as requirements for the math word problem generation task. To make the task objective, we only include objective requirements.FgFactV -Fine-grained Fact Verification is a task of checking whether each piece of information in a claim sentence is supported by the provided evidence. Fine-grained fact verification that instructs to provide detailed explanations has not been explored and similar tasks have been reported to be challenging for LLMs (Fabbri et al., 2021) . This task is designed to be evaluated on Reasoning Correctness and Context-Faithfulness. This task instructs the models to provide reasoning (Reasoning Correctness) on whether each piece of information in the claim is supported by the evidence or not (Context-Faithfulness). To make sure that annotators can objectively evaluate LLM responses in a binary manner, we provide detailed instructions such as "check all pieces of information in the claim and state reasoning on whether each part of the claim is supported". Creation Process: We use WiCE (Kamoi et al., 2023) as a base dataset, which includes a factuality classification task. We reconstruct this task into fine-grained verification, which instructs models to explicitly check all pieces of information in the claim. Since WiCE includes very long web articles as evidence, we retrieve part of them to make the model inputs in our task shorter.AnsCls -Answerability Classification is a task of classifying the answerability of factual questions. Answerability classification is a popular task (Rajpurkar et al., 2018; Kwiatkowski et al., 2019) , but the performance of LLMs on challenging cases is unexplored. Our task instructs the models to answer the provided question, but they should classify the questions as unanswerable if the questions include factual errors. Unlike classic datasets that include entirely unanswerable questions, our task requires detecting small factual mistakes. This task is designed to be evaluated on Reasoning Correctness and Parameterized Knowledge. This task instructs models to reason whether the questions are answerable (Reasoning Correctness) by checking the factual correctness of questions (Parameterized Knowledge). Creation Process:We use HotpotQA (Yang et al., 2018) as a base dataset. To create unanswerable questions, we (1) add a requirement to use knowledge before a specific date to make the questions time-sensitive (Chen et al., 2021) and ( 2) use GPT-4 to introduce factual mistakes in the numbers, nouns, and modifiers in the questions. By using multiple prompts to introduce mistakes, we create unanswerable questions in diverse categories, as shown in Figure 4 .14 faculty and graduate students participate in the annotation process. We train the annotators on multiple practice annotations. Each instance is annotated by one annotator and verified by another annotator. The tasks in our benchmark require annotators to carefully read the whole text in inputs and responses, and the annotations take about 90 hours in total (6 minutes for each case). Details are in Appendix C. Filtering: We remove pairs of input and output that involve errors detected by other criteria (e.g., grammar mistakes) or subjectivity in determining whether the response is erroneous or not. However, this process removes only 3.8% of cases, showing that our tasks are properly designed to be evaluated objectively by the four criteria. Human Agreement: To assess the human agreement and annotation quality, we evaluate the expert performance on 35 instances from each dataset (35 × 6 instances in total). The average performance of three annotators on the three tasks was 95.7 in F1, verifying that humans can annotate binary error labels with high agreement. 
error detection performance on realmistake: 
 We evaluate 12 LLMs on the error detection task in ReaLMistake. The questions addressed in this section include: (1) How well can LLMs detect errors made by LLMs? (2) Can LLM-based error detectors provide reliable explanations for mistakes in LLM responses?(3) How can we improve LLM-based error detectors? and (4) Does ReaLMistake include challenging and diverse error detection tasks?The LLM-based error detectors evaluated in our experiments use the same prompts of "classify whether the response includes errors" for all tasks, differing from criterion-specific detectors in prior work, such as factuality evaluators. As in Figure 1 , evaluation of the criteria-independent detectors is enabled by the tasks in ReaLMistake, which do not involve subjectivity in the evaluation. In contrast, for example, the text summarization task cannot objectively evaluate criteria-independent detectors because they involve the subjective "relevance" criterion, although it can objectively evaluate factuality evaluators.LLMs for Detectors. Open: Gemma 7B (Google, 2024) , Llama 2 (13B and 70B) (Touvron et al., 2023) , Mistral 7B (Jiang et al., 2023) , Mixtral 8x7B (Jiang et al., 2024) , Qwen 1.5 (14B and 72B) (Bai et al., 2023; Qwen Team, 2024) , Closed: Gemini 1.0 Pro (Gemini Team Google, 2023) , Claude 3 Opus (Anthropic, 2024) , GPT-3.5 Turbo (Brown et al., 2020; Ouyang et al., 2022) , and GPT-4 (2023-0613 and 2024-0125) (OpenAI, 2023) . Details are in Appendix H.Prompts. We use four zero-shot prompts (Kojima et al., 2022) for error detection that have variations in two properties: different wordings and flipped order of the binary error label options (Section 4.3, Figure 7 ). In Table 4 , we use the average performance of the four prompts to alleviate the influence of biases caused by the prompt design.Table 4 shows the error detection performance of 12 LLMs with zero-shot prompts on ReaLMistake. It shows that LLM-based error detectors often perform worse than random baselines in F1, showing that the error detection task in ReaLMistake is difficult even for strong LLMs such as Claude 3 and GPT-4. The results also show that the three tasks in ReaLMistake exhibit different difficulties. GPT-4 achieves reasonable F1 scores on MathGen but performs poorly on FgFactV and AnsCls with very low recall. This result suggests that error detection on tasks with different properties has varying difficulties, and indicates that ReaLMistake provides error detection tasks on diverse tasks with different properties.To analyze relationships between the performance of LLMs on error detection and other popular tasks, we compare the performance of 12 LLMs on ReaLMistake to LMSYS Elo Rating (Zheng et al., 2023 Figure 5 : Relationship between LMSYS Elo Rating (Zheng et al., 2023) and error detection performance on GPT-4 responses in ReaLMistake of 12 LLMs. Stronger LLMs detect errors made by GPT-4 with higher precision but with lower recall . Comparison to the performance on MMLU (Hendrycks et al., 2021) in Appendix G.1 shows the same trends. low precision of weak LLMs. Comparison to the performance on MMLU (Hendrycks et al., 2021) , which shows consistent results, and correlation coefficients are in Appendix G.1.Our prompts instruct LLM-based error detectors to provide an explanation in addition to predicting the binary error label. We manually analyze the explanations provided by error detectors based on GPT-4-0125, Claude 3 Opus, and three open-source models that show relatively good binary error detection performance in Table 4 Figure 6 shows the distribution of mistakes in the explanations by the error detectors on responses from GPT-4-0613. We observe that GPT-4-0125 and Claude 3 Opus generate explanations of relatively good quality, but they still make mistakes, especially on MathGen. Explanations by open-source models are more often wrong even when the binary predictions are correct. They often make mistakes even for correct responses from GPT-4-0613, not just missing errors, resulting in low precision. To improve the error detection performance, A ) on responses from GPT-4-0613 in ReaLMistake. Open-source models introduce more mistakes compared to GPT-4 and Claude 3. We manually classify outputs from error detectors for 20 cases in each dataset: correct binary label prediction with correct, insufficient (some requirements not evaluated), wrong reasoning, or no reasoning; wrong binary label prediction with no reasoning, insufficient (missing errors), or wrong reasoning. Appendix I provides example outputs from LLM-based error detectors.Your task is to detect errors in the model response.Your task is to classify whether the model response is validError Sensitive Prompt (Type 1) Conservative Prompt (Type 2)You need to check whether the model response follows all instructions and requirements in the model input. Will we get different results when the option order is flipped?Provide your explanation first and conclude your response with this analysis suggests that we need to make strong LLMs more careful about the mistakes in the provided LLM responses and improve the reasoning capability of open-source models.We analyze the influence of two small changes in prompts on error detection by comparing the performances of LLM-based error detectors with four prompts (Figure 7 ). Positional Bias (Wang et al., 2023a; Zheng et al., 2023; Pezeshkpour & Hruschka, 2023) : We evaluate the influence of the order of the error label options in the prompts. In an average of 12 LLMs and 3 tasks, putting the "error" option first (Type B in Figure 7 ) has 16.0 ± 21.7% (Type 1) and 27.2 ± 23.9% (Type 2) higher recall, suggesting that LLM-based error detectors prefer the first option. Wording Bias: We compare two wordings in error detection that ask to "detect errors in the response" (Type 1) and "evaluate whether the response is valid" (Type 2) while providing the same definition to make sure that they are semantically identical. In an average of 12 LLMs and 3 tasks, Type 1 has 16.9 ± 20.3% higher recall. 4 These results show that recall of error detection is sensitive to small changes in prompts. 5However, we also observe that these biases cannot be easily used to improve error detectors. First, although low recall is a problem in error detection by strong LLMs, we observe that small changes in prompts have a very small influence on the recall of strong models. In addition, we do not observe consistent trends in precision, although low precision is a problem in error detection by weaker models. These results suggest that non-trivial improvements to prompts or frameworks are needed to improve the error detection performance.A possible approach to improve LLM-based error detectors is to apply techniques proposed for improving LLM-based evaluators for LLM responses (e.g., pairwise evaluation). We apply three popular approaches to LLM-based error detection: self-consistency (Wang et al., 2023b) , integrating responses by multiple LLMs (Zhang et al., 2023; Li et al., 2023b; Cohen et al., 2023; Chan et al., 2024) , and providing evaluation steps in prompts (Liu et al., 2023a) .Specifically, we evaluate three simple methods: self-consistency with five samples, the majority vote of binary error predictions by multiple models, and providing human-written instruction of evaluation steps in prompts. However, none of these methods in popular approaches improves error detection performance (Appendix A). This observation indicates that applying existing techniques for LLM-based evaluators is not sufficient to improve LLM-based error detectors, and ReaLMistake provides challenging error detection tasks. 
future directions -evaluation on ambiguous errors: 
 In this work, we propose to evaluate error detectors on objective errors because this approach has multiple advantages, including smaller noises, the absence of subjectivity of annotators, binary annotations, and smaller annotation costs. However, since many NLP tasks are subjective, we expect future research to evaluate whether error detectors have a good correlation with human evaluators on ambiguous errors. A possible approach to collecting benchmarks for evaluating error detectors on ambiguous errors is to collect annotations by multiple annotators for each instance to get average error scores. However, difficulties in creating such benchmarks include (1) hiring annotators who represent the distribution of (possibly subjective) error annotations by general people, (2) evaluating binary predictions from LLM-based error detectors using labels by multiple annotators, and (3) high annotation cost. We expect future work to address these challenges. 
conclusion: 
 We introduce ReaLMistake, the first benchmark for evaluating error detection methods for LLM responses, consisting of objective, realistic, and diverse errors made by LLMs. Our experiments on this benchmark with error detectors based on 12 LLMs show that detecting mistakes in LLMs (GPT-4 and Llama 2 70B) is challenging even for recent LLMs. 
 
 Appendix: Experiments. We provide a GitHub repository6 that includes the ReaLMistake benchmark (also hosted at Hugging Face Datasets7 ), all outputs from LLM-based error detectors used in Section 4, and evaluation code.Annotation. Appendix B provides annotation instructions we used for our benchmark. Table 7 : Improvement of LLM-based error detectors by using human-written instructions of evaluation steps. This table compares the performance of the original prompt type 1-A in Figure 7 and the updated prompt type 1-A with human-written evaluation steps. The gray color represents that human-written instructions decrease performance. This approach does not provide improvement from the original simple prompt.We use the WiCE dataset (Kamoi et al., 2023) as a base dataset. WiCE includes a factverification classification task and each instance of the dataset includes a claim (sentence) in a Wikipedia article, web articles cited for the claim in Wikipedia, an entailment label (supported, partially supported, or not supported), and indices of supporting sentences (a part of the web articles supporting the claim).Web articles in WiCE are too long for our benchmark, so we only use a part of them in our tasks. Specifically, we use the oracle chunks provided in WiCE, which include the supporting sentences in the web articles and randomly selected sentences from web articles.8 We note that we changed the entailment classification task to a binary classification of supported or not supported. In our task, the original partially supported cases are regarded as not supported. We added instructions to perform fact verification for all pieces of information in the claim to reconstruct the task to fine-grained fact verification.Answerability Classification is a task to classify questions as unanswerable if they include factual errors or any other problems. We use the HotpotQA dataset (Yang et al., 2018) as a base dataset. Each instance in HotpotQA includes a multi-hop question about knowledge in Wikipedia articles and paragraphs from Wikipedia articles that include information required to answer the question. The original task in HotpotQA is a mult-hop QA using the provided evidence, but we do not provide the paragraphs and make it a closed-book QA task.First, since questions in HotpotQa are often grammatically unnatural, we use GPT-4-0613 to improve the questions without changing any properties.Your task is to improve the following question while keeping the original meaning of the question. You should only generate the rewritten question. Do not include anything else in your response.Second, we use three types of automatic processes to create diverse unanswerable questions (Figure 4 ). We use each of the following processes for 1/3 of the cases in our dataset.Factual Mistakes in Numbers. The first type is mistakes in numbers. We provide the following prompt to GPT-4-0613 to generate (possibly) unanswerable questions that include factual mistakes in numbers.We provide a question that can be answered by using Wikipedia articles.First, add correct additional information from related Wikipedia articles to the question that includes at least one number (e.g., year, age, quantity). The added information should not change the answer to the question. Second, introduce a minor mistake in the number in the added information. Do not include anything else in your response.Other Factual Mistakes. The second type is mistakes other than numbers. We provide the following prompt to GPT-4-0613. We instruct not to make mistakes in proper nouns because they make the answerability detection task easier. However, this requirement is not always satisfied as shown in Figure 4 .We provide a question that can be answered by using Wikipedia articles.First, add correct additional information from related Wikipedia articles to the question. The added information should not change the answer to the question. Second, introduce a factual mistake in the added information (nouns, adjectives, verbs, etc.). However, do not introduce mistakes in people's names, titles of books or movies, and locations (e.g., countries). You should introduce factual errors, so the mistakes should not use synonyms or paraphrases that do not change the factual correctness of the information. Do not include anything else in your response.Time Sensitive Questions. The third type is time sensitivity. All inputs in our Answerability Classification include time constraints in a format of:Assume you are on {date} and questions that require knowledge after this date should be classified as unanswerable.For the questions involving the above two types of factual mistakes, we randomly select dates between 2022-09-01 and the date the first version of the Wikipedia article was created (we select the newest article from the ones provided for each question in HotpotQA). However, randomly selected dates often do not make the questions unanswerable.To create more questions that are unanswerable due to the time constraint, we use years in the paragraphs provided in HotpotQA as a knowledge resource for each question. Specifically, we randomly select years in the paragraphs provided for each question in HotpotQA and use the time constraint of one year before the selected year. This strategy makes at least one event in the paragraphs unanswerable, although it is not guaranteed that this event is involved in the question-answering process.Tasks in ReaLMistake include detailed task definitions in inputs (prompts for LLMs), and binary error detection on these tasks is well-defined without any additional instructions (please refer to Appendix I for examples of full inputs of our tasks). However, we provide annotation instructions to annotators to further improve the annotation quality and provide instructions on error category annotations. We provide 9-page annotation instructions with multiple annotation examples.To make sure that annotators understand these instructions, we provide five challenging cases for each dataset (3 × 5 cases in total) as a practice annotation. We manually check their annotations and provide detailed feedback. If they make major mistakes, we provide another five cases. We continue this process until all annotators clearly understand the annotation instructions. This phase took about two weeks in total to train all annotators.This section provides key parts of the annotation instructions.C.1 Definition of Error Labels• error -Select this option when the response includes at least one error.-Provide a detailed explanation in the "feedback" column.-Responses with any kind of errors, not restricted to the provided error categories, should be categorized as errors (e.g., grammar mistakes). When you do not select any of the four categories, please explain in the "feedback" column.• no error -Select this option only when the response does not include any error -You do not need to provide feedback. • not sure -Select this option when you cannot determine the binary error label.-For example, you should select this option when mistakes in the output are too minor or the model input is invalid. -The cases with this label will be removed from the final dataset.For analysis purposes, we categorize errors in model responses into four categories. Since model inputs do not provide any information about error categories, we provide detailed instructions on error category annotations.In this task, each model output includes a generated question and solution. We expect mistakes in this task to be mainly in Instruction-Following and Reasoning Correctness. In this task, an output from the models includes a generated question and solution.• When the generated question or solution does not follow the constraints, select the Instruction-Following error category.-When the generated question is invalid (unanswerable), you also need to select the Instruction-Following error category because the instruction says "The generated question should be valid and answerable".• When the solution includes wrong reasoning, select the Reasoning Correctness error category. • Unnecessary information -When the question includes unnecessary information, it should be marked as the Instruction-Following error because the instruction says not to include unnecessary information in the generated questions. -In solution:* When the solution provides two possible ways to answer the question or verify the final answer, it is regarded as an Instruction-Following error because the instruction says not to include this.* When the solution includes unnatural and unnecessary steps other than the above cases, select the Reasoning Correctness error category.Fine-grained Fact-Checking• Check whether all pieces of information in the claim are verified in the response. If any part of the claim is not verified in the response, mark it as a Context-Faithfulness error even if the final answer is correct. Note that we regard the claim as context as well as the evidence in this task because this is a task to compare two texts (claim and evidence). • Check whether each piece of information has been verified correctly.-If the response misses or misinterprets any part of the evidence, mark it as a Context-Faithfulness error. -If the response provides incorrect reasoning from correctly retrieved information, mark it as a Reasoning Correctness error.• Check whether the final answer is faithful to the verification of each piece of information by the model. If not, mark it as an error of Reasoning Correctness. • If the response does not follow the formatting requirements, mark it as an Instruction-Following error.In this task, we provide factual questions with time constraints that may include incorrect information. LLMs need to classify the question as unanswerable if (1) it requires information after the specified date or (2) it includes incorrect information. Many questions in this task are unanswerable, but some questions may still be answerable. You need to check all the information in each question by using Wikipedia or any other reliable resources.• If the question includes mistakes but the model response does not point out any factual mistakes, this is a Parameterized Knowledge error.-When there are multiple mistakes in a question, the response only needs to point out one mistake. The response does not need to point out all mistakes. -We introduce an intentional error in the questions. However, the original question can already include mistakes.• If the response includes factual mistakes, they are also errors in Parameterized Knowledge.-The response can include factual mistakes even when the answerability classification is correct. This case is also a Parameterized Knowledge error. -For example, the response can detect one mistake in the question correctly but include factual mistakes caused by another mistake in the question. The response only needs to detect one mistake to clarify it as unanswerable, but should not include factual mistakes.• Mistakes in reasoning that do not involve factual information should be marked as a Reasoning Correctness error. • If the response correctly points out the mistakes but does not classify the question as unanswerable, classify it as a Reasoning Correctness error. • The question should be classified as unanswerable if it requires any information after the timestamp in the instruction. If the response ignores this instruction, it should be marked as an Instruction-Following error. • If the response does not include any factual mistakes or reasoning mistakes but ignores factual mistakes in the question, this is an error of Instruction-Following.Math Word Problem Generation. Table 3 includes the distribution of 9 categories of requirements included in this task.Fine-grained Fact Verification. Answerability Classification Task. Figure 4 shows the distribution of the reasons for the unanswerability of the questions in this task.This section provides additional results for the experiments in Section 4. This section includes additional results for the analysis of error detection in Section 4.Figure 9 shows the full version of Figure 5 , which compares the error detection performance on ReaLMistake with popular evaluation benchmarks: MMLU (five-shot) (Hendrycks et al., 2021) and LMSYS Chatbot Arena Elo Rating (Zheng et al., 2023) . These results consistently show that strong LLMs on MMLU and LMSYS Elo Rating detect mistakes made by LLMs at high precision but low recall (Section 4.1). Table 9 shows that precision and recall have positive and negative correlations with these metrics in both Pearson and Spearman correlation coefficients. We observe that performance on MathGen shows different trends in recall (weaker negative correlation for GPT-4 responses and positive correlation for Llama 2 70B responses) which again indicates that ReaLMistake includes diverse error detection tasks with different properties. Table 9 : Correlation between precision and recall on the error detection task in ReaLMistake and performance on MMLU (Hendrycks et al., 2021) or LMSYS Chatbot Arena Elo Rating (Zheng et al., 2023) on 12 LLMs. Precision and recall on ReaLMistake have positive and negative correlations with these metrics both in Pearson and Spearman correlation coefficients, especially in the error detection task on responses from GPT-4-0613. * : p < 0.05.Figure 10 and 11 show the additional results of the analysis of biases caused by small changes in prompts on the error detection task in ReaLMistake. These figures show that all results are consistent with the analysis in Section 4.3. As discussed in Section 4.3, these results show that recall is strongly affected by small changes in prompts (wordings and the order of binary label options) but there is no consistent trend in precision. In addition, we observe that GPT-4 (red markers) and Claude 3 (yellow star markers) are less affected by small differences in prompts. error" option first) minus prompt type A ("error" option first). In most models, putting the "no error" option before the "error" option increases recall (more errors are missed by error detection). However, there is no consistent trend in precision.This section shows examples from ReaLMistake in Figure 2 and outputs from LLM-based error detectors (Section 4). The manual analysis for responses from LLM-based error detectors categorizes mistakes made by detectors as in Figure 6 .I.1 Math Word Problem Generation (MathGen)Generate a math word problem that satisfies the following requirements. First, provide the generated question. Second, generate a step-by-step solution for the generated question.Question: Marla completes 32 laps around the track per hour. If the track is 400 meters long, how many kilometers does Marla run in 2.5 hours? Solution:Step 1: First, we need to find out how many meters Marla runs in one hour. Since she completes 32 laps and each lap is 400 meters, we multiply these two numbers together. 32 laps/hour * 400 meters/lap = 12800 meters/hourStep 2: To convert meters to kilometers, we divide by 1000 (since there are 1000 meters in a kilometer). 12800 meters/hour ÷ 1000 = 12.8 kilometers/hourStep 3: Now, we need to find out how many kilometers Marla runs in 2.5 hours. We multiply her speed by the time.12.8 kilometers/hour * 2.5 hours = 32 kilometers So, Marla runs 32 kilometers in 2.5 hours.• Error Label: error • Error Categories: Instruction-Following • Explanation: The question does not involve relative speed and rounding off to the nearest whole number, which are in the requirements. In addition, the solution does not include numbers with two decimal places.The model response violates the specific requirement that the solution should include "[integers, numbers with one decimal place, numbers with 2 decimal places, fractions]" and should not include any other type of number. The solution provided uses a time value of "2.5 hours," which is a number with one decimal place, aligning with the requirements. However, the specific requirements also mention that the solution involves division and rounding off to the nearest whole number, which was not adhered to in the provided solution. The solution directly calculates the distance Marla runs in 2.5 hours without involving any rounding off to the nearest whole number as instructed.Therefore, the model response contains an error.Manual Analysis: Correct binary label, insufficient reasoningThe response does not mention why the fourth specific requirement is not satisfied. In addition, it does not mention the first requirement about "relative speed".In this paper, "objective errors" represent errors in LLM responses that do not involve ambiguity or subjectivity in the detection process and can be detected in a binary manner with a high inter-annotator agreement.Although the term "instruction-following" is used in various scopes(Wei et al., 2022a;Ouyang et al., 2022), we use a definition that focuses on the requirements explicitly mentioned in prompts.https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard(March 16, 2024)We compare the average performance of Type 1-A and 1-B with that of Type 2-A and 2-B.The numbers after "±" represent standard deviation. Detailed results are in Appendix G.2.https://github.com/psunlpgroup/ReaLMistakehttps://huggingface.co/datasets/ryokamoi/realmistakehttps://github.com/ryokamoi/wice/tree/main/code_and_resources/entailment_inputs/ oracle_chunkshttps://huggingface.co/spaces/lmsys/chatbot-arena-leaderboardhttps://platform.openai.comhttps://www.anthropic.com/apihttps://cloud.google.com/vertex-aihttps://huggingface.co/models 


Decision: This paper contributes to the timly problem of evaluating LLMs when used as evaluators. The authors introduce a welcome new error detection benchmark dataset, ReaLMistake, comprising organic errors with granular annotations. Using this, they then provide new empirical insights into the strengths and weaknesses of LLMs when used as evaluators. 

Overall, this is a nice, timely contribution which will be of broad interest and use to the COLM community.

Reviews: review: paper_summary: paper_summary: The paper introduces ReaLMistake, which the authors claim is the first error detection benchmarking suite consisting of objective, realistic and diverse errors made by LLMs. ReaLMistake contains three tasks that introduce objectively assessable errors in four categories (reasoning correctness, instruction-following, context-faithfulness, and parameterised knowledge), eliciting naturally observed and diverse errors in responses of GPT-3 and Llama 270B annotated by experts on three tasks three tasks (Math Word Problem Generation, Fine-grained Fact Verification, and Answerability Classification). The benchmark includes error annotations (binary error label, error categories, and human explanation about errors) on responses from GPT-4 and Llama 2 70B on the three tasks. The authors show that top LLMs like GPT-4 detect errors made LLMs at very low recall, and all LLM-based error detectors perform much worse than humans, that explanations by LLM-based error detectors lack reliability and LLM-based error detection is sensitive to small changes in prompts but remains challenging to improve, and finally, popular approaches to improve LLMs, including self-consistency and majority vote, do not improve the error detection performances. The authors promise to make the code and data available on acceptance.

Overall the quality of the paper is quite high in terms of the detail of the description of the dataset, though this is somewhat in part due to the lengthy Appendices making very clear what the prompts and responses were. ReaLMistake undoubtedly could be part of a good resource for evaluation for LLM error detection, though it is limited in scope, as partially admitted to by the authors. The presentation overall is quite strong and clear, though see below for some queries about it which should be addressed. It seems more incremental research than highly novel, as the authors define the benchmarking dataset as an intersection of what has gone before, however the paper claims ReaLMistake is novel in focusing on errors that are unambiguously judged as such by human annotators (though see concerns on this claim below). The approach is quite principled scientifically, though I would dispute the claim of unambiguously annotated error data as being essential for good error detection benchmarking for LLMs, and perhaps if accepted, the authors can reduce the strength of that claim and describe alternatives where multiple annotator’s judgments are included (see below).
strength_weakness: strength_weakness: My main reservation about the paper, which could be corrected if claims were slightly toned down, is in terms of the motivation of the form of the benchmark: the authors claim they are against human subjective ratings of errors in a response, or at least instances which are not universally agreed upon for their label (e.g. “We cannot evaluate error detectors if humans cannot determine whether LLM responses include errors”). However, this claim is too strong, and it is not clear how the inclusion of multiple (potentially differing) annotations would make things worse for a benchmark – they are certainly a challenge to be solved going forward, but rigorous inter-annotator agreement studies in creating these benchmark datasets could be a way to deal with ambiguous/non-unanimous labelling - in fact it could even be a good idea to include the differing annotations in the benchmark. The challenge would then shift away from limiting a dataset to unambiguous error instances only to developing systems that give similar confidence levels to the level of inter-annotator agreement. Even in some cases there are mathematics-based prompts, there is still room for ambiguity in the interpretation of verbal prompts.

It is not clear whether the explanations were agreed upon between the annotators.

It is not made clear how novel the benchmark is as Table 1 suggests ‘MT-Bench, PandaLM, LLMEvalˆ2 Multiple NLP Tasks’ has more criteria and in multiple domains.
questions: questions: I would reduce the strength of the claims around ambiguity and frame your dataset as collecting error judgements which are low in ambiguity (as illustrated by the 95% F1 score of agreement), rather than ‘objective’. This is scientifically too strong.

The claim of it being a diverse set of errors might also be a bit strong, as it is dependent on the diversity and difficulty of the original tasks – I would reduce this and discuss that there is more work to be done to enlarge the scope of error type.

As per the above, it is not clear how novel the approach is, given Table 1’s column ticks – please further clarify how it is going further than the top row’s system in the text, particularly if you lessen the claim that human inter-subjective disagreement is a bad thing for annotation overall.

Please calculate and report Cohen’s kappa for the agreement score, which is similar to F1 but more comparable to other inter-annotator agreement scores.

There are some errors in phrasing throughout which require a proof-read - e.g.
-	Use ‘LLM-based’ (rather than LLMs-based)
-	Use determiners in places like ‘We introduce ReaLMistake benchmark’ –> ‘We introduce the ReaLMistake benchmark’
paper_summary: paper_summary: The paper introduces a benchmark for detecting errors in LLM-generated text. The dataset is generated by modifying several existing datasets (e.g. AQuA, HotpotQA), by prompting an LLM to modify existing questions according to certain instructions. The LLM-generated responses are manually filtered for those that failed to correctly follow the instructions. After constructing this dataset, the paper evaluates a number of LLMs, and finds low recall in identifying errors. The paper also finds that, even when errors are correctly identified, the LLM-generated explanations of these errors are frequently incorrect.
strength_weakness: strength_weakness: - It would be interesting to understand what factors make an error hard to identify. The results in Table 4 are fairly coarse grained. Does the dataset support more fine-grained evaluations?

- This is not very important, but it would be useful to evaluate prompting strategies more comprehensively, beyond what is described in Section 4.3. It seems like it should be possible to improve recall at the cost of lower precision. How large is that cost?

- The errors are generated from a limited number of templates on three tasks. It is uncertain how generalizable the results will be to other types of errors, and those that occur in the wild. The paper is nonetheless a good starting point.
paper_summary: paper_summary: This paper proposes. new benchmark, ReaLMistake, that evaluates how good LLMs are at detecting objective errors. The authors first constructed three sets of instructions (math word problem generation, fine-grained fact verification, and answerability classification) and prompted GPT-4/Llama-2-70B to generate responses. Then the authors manually annotated the errors in the model responses with rationales. The error types are categorized into (1) instruction following, (2) reasoning correctness, (3) context-faithfulness, and (4) parameterized knowledge. All four are common errors of LLMs and can be objectively evaluated. The human agreement rate is high (95.9 F1).

Then the authors evaluated a number of open-source/closed-source LLMs on the benchmark -- this includes both the accuracy on detecting the errors and a manual evaluation on the rationales generated by the models. The experiment results are quite interesting: (1) even the best models struggle at accurately predicting the errors. More surprisingly, there seems to be a negative correlation between model capacity (measured by Chatbot arena score) and the error detection recall. (2) The rationales generated by models can be very wrong even if the prediction is correct, especially for open-source models. (3) Prompt engineering is very difficult for error detection and even more advanced prompting techniques like self-consistency does not help.
strength_weakness: strength_weakness: (1) The source tasks are limited to synthetic tasks like question generation and fact verification. It would be more interesting to see such a dataset constructed on more diverse instructions such as ShareGPT/Alpaca/OpenAssistant.

(2) The negative correlation between the LLM capacity and error detection recall, though interesting, is a bit unintuitive. The authors also did not offer a reasonable guess of why this happens -- which makes me wonder if this is due to suboptimal designs of the prompt.
questions: questions: If I understand correctly, the annotation includes the types of the error, but the performance on Table 4 and the prompts used by models are simply binary error detection. Would it make the model perform better if they are prompted with the error type definitions and are prompted to provide the error type as well? The experiments would be more insightful as well if a per-type analysis is provided.
paper_summary: paper_summary: In their paper, the authors introduce an error detection benchmark called ReaLMistake. This benchmark evaluates four distinct types of errors—reasoning correctness, instruction-following, context-faithfulness, and parameterized knowledge—made by large language models (LLMs) across three selective natural language processing (NLP) tasks. The results reveal several key findings:

1) Top-performing LLMs, including GPT-4, exhibit low recall in error detection tasks.
2) While LLMs can correctly identify errors, their explanations remain unreliable.
3) The performance of error detection is notably sensitive to subtle variations in prompts.
4) Previous methods aimed at enhancing error detectors have proven ineffective.
strength_weakness: strength_weakness: The LLMs employed to generate the response (GPT4 and Llama2) could impact the reliability of the benchmark. Another concern is about the sample size. The ReaLMistake benchmark contains a relatively small number of samples compared to other benchmarks with binary labels (e.g., SummEdits and BIG-Bench Mistake).
questions: questions: 1) In Table 2, I notice that the dataset is biased, especially the error percentage of the Llama2 response is super high (75.0, 78.8, 77.5). How different data distribution affects the error detection results? What is the reasonable range of the error percentage?

2) A following question of 1)： In Table 4, is it suitable to use your defined "Random" recall score as the baseline? Your dataset with Llama 2 is super biased and to give a more extreme example, if the error percentage is 100%, no model can beat the recall baseline(100). However, many models work better than random guess (recall=50). In this case, I think your claim "LLM-based error detectors often perform worse than random baselines" is a bit unsound.

Comments: title: Response to the rebuttal
Ack
comment: This is good the Cohen's kappa figures have been added and would be added to an accepted version.

The clarification on pair-wise benchmarks vs pointwise are welcome, and again would be good to include to discuss the limitations.

My reservation is that the claims must be toned down about the unambiguous nature of the data - no natural language data is fully unambiguous.
Thank you for answering my questions. I'll keep the score.
Thanks for answering my questions!
Thank you for your response.

>We agree that creating more diverse benchmarks is a promising direction for future research. However, collecting unambiguous errors is difficult in tasks in the wild, which are often open-ended and subjective.

I am sympathetic to the difficulty of collecting errors in the wild, and agree that this is a good place to start. Nonetheless, it is not clear how well the results will generalize.