Title: HaluEval: A Large-Scale Hallucination Evaluation Benchmark
for Large Language Models
Benchmark: HaluEval
Summary: HaluEval introduces a large-scale benchmark to evaluate the hallucination behaviour of large language models (LLMs) across question answering, dialogue, summarization, and general queries. It seeks to understand what types of content and to which extent LLMs can generate hallucinated response. It combines 30,000 automatically generated and 5,000 human-annotated samples using a two-step “sampling-then-filtering” framework. Results show that LLMs, including ChatGPT, frequently produce hallucinations (~19.5%) and struggle to recognise them, but performance improves with external knowledge or reasoning steps.
Contribution: The paper provides a high-quality hallucination benchmark across multiple tasks and proposes a structured pipeline to generate plausible yet incorrect responses using LLMs themselves. It offers actionable insights into failure modes and mitigation strategies for hallucination detection.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Hallucination
Phenomenon defined by authors: Yes
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Given a test sample (e.g. QA, dialogue, summary), classify whether it contains hallucinated content or not.
Task item definition: One item consists of a prompt + response pair, annotated with hallucination presence/absence.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: For curation of the task-specific examples from three tasks i.e., question answering, knowledge-grounded dialogue, and text summarization, 30,000 instances are randomly sampled from the training set of HotpotQA (Yang et al., 2018), OpenDialKG (Moon et al., 2019), and CNN/Daily Mail (See et al., 2017), and their hallucinated examples are generated. For the general user queries, 52k instruction tuning dataset from Alpaca is adopted for human annotation. To screen out user queries where LLMs are most likely to generate hallucinations, they use ChatGPT to sample three responses for each query and only retain 5,000 queries with the lowest similarity among three responses. 
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 35,000
Dataset metadata available: Yes
Dataset metadata detail: Topic category, tasks (general queries, QA, dialogue, summarization)
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Response format detail: The provided dataset consists of two variants for each subset including, (1) content, response and a binary classification (yes/no) as a short free response if hallucination is detected or not in the response, (2) content, right free response, and hallucinated free response.
Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric aggregation: Simple Mean
Metric statistics: Mean accuracy is used to aggregate model performance across tasks and conditions
Metric subscores: Yes
Metric subscores detail: Per-task accuracy (QA, dialogue, summarization, general queries), per-hallucination pattern (comprehension, factualness, specificity, and inference)
Metric metascoring: majority@k (majority vote over k trials)
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: Yes
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Benchmark difficulty validated by poor model performance and successful human agreement (κ = 0.811)
Taxonomy: Language Modelling / Hallucination
New taxonomy: General Capability / Hallucination