Title: Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trends
Summary: This work releases a span‑level benchmark that labels factual inconsistencies (“hallucinations”) in dialogue summaries produced by GPT‑4, Alpaca‑13B, and several fine‑tuned BART‑style models on SAMSum and DialogSum. It introduces a refined error taxonomy, most notably the new class Circumstantial Inference, and shows that existing automatic factuality metrics miss many of these subtle errors; two prompt‑based detectors they propose perform better.
Contribution: (1) New human‑annotated dataset of 2 × dialogue corpora + 3 × model summaries with span‑level error tags
(2) refined hallucination taxonomy
(3) new prompt/MoE detectors that beat prior QA/NLI metrics at binary and span detection, especially for Circumstantial Inference.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Faithfulness / hallucination in dialogue summarization
Phenomenon defined by authors: Yes
Phenomenon definition: The paper defines hallucination (the phenomenon of interest) as:
“statements in summaries that do not have direct evidence in the source material”.
Additionally, a specific subclass 'Circumstantial Inference' is introduced and defined as:
“statements that appear plausible based on circumstantial (but not direct) evidence in the dialogues”, and further: “When the language model draws inferences based on circumstantial but not direct evidence in the conversation, we label this as a circumstantial inference error” (summarized).
This framing reflects an expanded taxonomy of faithfulness violations, emphasising both factual absence and contextually unsupported inference.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Given a dialogue and its machine‑generated summary, identify whether the summary contains unsupported content and mark the non‑factual span(s).
Task item definition: A single row contains: dialogue ID, dialogue text, model name, summary text, list of human‑marked non‑factual spans, supporting evidence indices, binary factual label, and error type(s).
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Dialogues come from SAMSum (synthetic chit‑chat) and DialogSum (natural spoken dialogues); summaries are generated zero‑shot by GPT‑4 and Alpaca‑13B plus four fine‑tuned BART variants; error spans are crowd‑verified and linguist‑labeled by the authors.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 1,902
Dataset metadata available: Yes
Dataset metadata detail: fields include original corpus, model, span list, error taxonomy category, linguistic category.
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Mean, F1, balanced accuracy; 95% CIs via bootstrap.
Metric subscores: Yes
Metric subscores detail: Results are broken down by error category (Circumstantial Inference, Logical, etc.)
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Human baseline: No
Authors discuss validity: Yes
Taxonomy: NLP / Summarization
New taxonomy: General Application / Summarization