Title: SummEdits: Measuring LLM Ability at Factual Reasoning Through the Lens of Summarization
Benchmark: SummEdits
Summary: SUMMEDITS introduces a 10‑domain benchmark to test whether language models can detect factual inconsistencies in summaries. The authors create a low‑cost, highly reproducible protocol in which seed summaries are lightly edited by an LLM and then labeled by humans as factually consistent or not; most LLMs perform barely above chance, with GPT‑4 still 8 pp below human accuracy.
Contribution: - (1) A new editing‑based annotation protocol that yields inter‑annotator agreement ≈0.9 while costing ≈20× less than prior datasets.
- (2) The 6,348‑samples SUMMEDITS benchmark spanning news, legal, scientific, dialogue, and sales domains.
- (3) Extensive evaluation showing specialised factuality methods often beat most general LLMs, and even GPT‑4 trails humans.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Factual inconsistency detection in summaries
Phenomenon defined by authors: Yes
Phenomenon definition: A summary should either be labeled as inconsistent if any factual inconsistency is identified with the document or consistent otherwise, to improve label interpretability.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Given a document and an edited summary, predict whether any factual inconsistency exists (binary label).
Task item definition: A single row consist of: document text, summary text, gold label, plus edit‑metadata.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Seed summaries are partly GPT‑3.5‑generated; edits are made using GPT‑3.5-Turbo; and further humans filter and label samples.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 6,348
Dataset metadata available: Yes
Dataset metadata detail:  domain, edit‑type, seed‑source, annotator‑agreement
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Mean
Metric subscores: Yes
Metric subscores detail: by domain and by edit‑type
Metric access: Outputs alone
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: They compute Cohen’s κ ≥ 0.9 after removing borderline cases and show GPT‑4 oracle nearly closes the gap, implying task measures intended skill not noise.
Taxonomy: NLP / 
New taxonomy: General Capability / Factuality