Title: TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization
Benchmark: TOFUEVAL
Summary: Propose a summarization dataset generated by LLMs and human annotations of factual consistencies. Show that LLMs hallucinate and have diverse errors, and that non-LLM evaluators can capture these errors better than LLMs.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: text summarization
Phenomenon defined by authors: Yes
Phenomenon definition: (1) are LLMs up to the task of evaluating model outputs? (2) can LLMs generate factually consistent summaries without hallucinations for non-news domains?
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Topic-focused Dialogue summarization Evaluation of factual consistency
Task item definition: a document and a topic for summarization; summary and the corresponding document for evaluation
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 1,479 summaries split into 70%/30% development/test so the test should be 444 summaries
Dataset size notes: 1,479 summaries split into 70%/30% development/test so the dev should be 1035 summaries
Dataset metadata available: Yes
Dataset metadata detail: topic area
Sampling method: Random sample (creators defined a task space and sampled from it), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Validation
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Metric definition: Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: for different data sources
Metric access: Model access required (e.g. logits)
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: they conduct extensive human experiment
Taxonomy: NLP / Summarization
New taxonomy: General Application / Summarization