Evaluating LLMs at Detecting Errors in LLM Responses
Ryo Kamoi1, Sarkar Snigdha Sarathi Das1, Renze Lou1, Jihyun Janice Ahn1,Paper & Dataset
2111
1Yilun Zhao , Xiaoxin Lu , Nan Zhang , Yusen Zhang , Ranran Haoran Zhang ,Sujeeth Reddy Vummanthala1, Salika Dave1, Shaobo Qin3,2,411
Arman Cohan , Wenpeng Yin , Rui Zhang
1Penn State University, 2Yale University, 3Stony Brook University, 4Allen Institute for AI{ryokamoi, rmz5227}@psu.edu

We introduce ReaLMistake, a benchmark for evaluating LLMs at detecting errors in LLM responses.
Our Experiments show that even strong LLMs, such as GPT-4 and Claude 3, detect errors made by
LLMs at very low recall and also explanations by LLM-based error detectors are unreliable.
Introduction and Motivation
As LLMs have been increasingly used inreal-world applications, it is critical to develop
methods for automatically detecting errors in
responses from LLMs. However, there is adeficiency in research specifically targetingerror detection of LLM responses.
An obstacle in studying error detection is thelack of benchmarks that include binary
error annotations (i.e., whether the responsecontains errors or not) on objective, realistic,
and diverse errors made by LLMs.
Specifically, to provide objective error labels,
tasks should not involve subjectivity orambiguity. In many NLP tasks, even humans
cannot objectively annotate binary error labels
because the tasks are often open-ended andevaluation involves ambiguity.

● Many NLP tasks are ambiguous and even humans cannot annotate errors in high agreement● We introduce the ReaLMistake benchmark with realistic, objective, and diverse errors made byLLMs (GPT-4 and Llama 2 70B) for evaluating error detection.

Dataset Creation
To create tasks that satisfy the requirements,
we propose an approach to design tasks sothat they make LLMs introduce errors
detected by objective, realistic, and diverse
evaluation criteria. We identify four criteriathat can be objectively evaluated by humans
and cover diverse errors in LLM responses:● Instruction-Following● Context-Faithfulness● Parameterized Knowledge● Reasoning
We create three tasks with the intention of
making LLMs introduce errors detected bythese four evaluation criteria, eliminating
subjectivity from the error annotationprocess.
We create the ReaLMistake dataset bycollecting error annotations on 900 responses
from GPT-4 and Llama 2 70B in the threetasks. The annotation process requires carefulchecking of the entire LLM responses, and 14
expert annotators spent 90 hours in total toprovide high-quality annotations.

Dataset Creation Process of ReaLMistake● We identify four categories of errors that can be objectively evaluated by humans● We design three tasks so that LLM responses only include these four objective errors

Experiments
We evaluate LLM-based error detectors withzero-shot prompts using 12 LLMs:● Zero-shot chain-of-thought (4 prompts)● Self-consistency● Majority vote by multiple LLMs
● G-Eval style human-written instruction
However, all detectors are much worse thanhuman performance and are often evenworse than random baselines.
In addition, our manual analysis shows that
explanations generated by LLM-based error
detectors are often wrong, even when the final
answer (error or no error) is corect.

GPT-40613
Llama 270B

MathGenFgFactV
AnsClsMathGenFgFactV
AnsCls

Gemma
7B
46.5
60.3
59.2
54.3
68.9
34.8

Error Detection Task in ReaLMistake

Precisions and recalls of 12 LLM-based error detectors
on ReaLMistake (errors in GPT-4 responses)● Better LLMs achieve better precision (red circles) but with lower recall (blue triangles)● Strong LLMs are conservative about detecting mistakes and miss many errors in LLM responses!

Llama 2
13B
70B
54.2
59.5
65.4
69.9
69.8
69.8
56.6
69.2
78.7
81.8
77.4
51.6

Mistral
7B
6.9
50.9
48.1
9.0
68.2
61.9

8x7B
45.5
46.8
38.3
56.0
35.1
29.8

Qwen 1.5
14B
72B
52.3
32.8
57.7
24.9
53.8
15.1
54.9
50.3
64.6
18.3
44.9
5.1

GPT-3.5
0125
65.3
41.4
28.8
72.3
34.2
3.7

Gemini
1.0 Pro
42.5
45.8
40.7
52.9
42.0
16.4

Claude 3
Opus
50.1
48.9
38.581.8
45.2
23.2

GPT-4
0613
63.1
12.7
20.0
88.7
38.8
61.6

0125
70.9
20.8
22.1
90.8
68.5
75.9

Random

Human

62.1
62.9
62.1
80.0
80.6
81.2

90.0
95.5
90.5
98.3
100.0
100.0

F1 scores of 12 LLM-based error detectors (zero-shot CoT) on ReaLMistake. Gray color represent values worse than the random baseline.