Title: Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation
Benchmark: RoSE
Summary: Propose a modified summarization salience protocol, curate the Robust Summarization Evaluation (RoSE) benchmark, conduct a comparative study of human evaluation protocols. Evaluate 50 automatic metrics and their variants and demonstrate how the benchmark leads to more statistically stable and significant results.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Text summarization evaluation
Phenomenon defined by authors: Yes
Phenomenon definition: "We focus on a specific summarization meta-evaluation study on summary salience. Salience is a desired summary quality that requires the summary to include all and only important information of the input article. The human evaluation of summary salience can be conducted in either reference-free or reference-based manners…the latter requires the annotators to assess the information overlap between the system output and reference summary, under the assumption that the reference summary is the gold standard of summary salience…we focus on reference-based evaluation for our human judgment dataset collection"-p4142
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: "Specifically, the evaluation process is decomposed into two steps: (1) Atomic Content Unit (ACU) Writing – extracting facts from one text sequence, and (2) ACU Matching – checking for the pres- ence of the extracted facts in another sequence. We formulate the ACU protocol as a recall-based pro- tocol, such that the first step only needs to be per- formed once for the reference summary, allowing for reproducibility and reuse of these units when performing matching on new system outputs. "-p4142
Task item definition: Given a reference summary, a system summary, and a set of Atomic Content Units (ACU), annotators have to decide whether ACUs exist in the system summary
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 1.5k docs, 10.2k Atomic Content Units (ACU)-level annotations and around 14k summary-level annotations,
Dataset size notes: val: 1k docs, 11.6k Atomic Content Units (ACU), 8k summaries
Dataset metadata available: Yes
Dataset metadata detail: topic area
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric subscores: Yes
Metric subscores detail: topic area
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: the authors conduct human experiment
Taxonomy: NLP / Summarization
New taxonomy: General Application / Summarization