Title: Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articles
Benchmark: DIVERSESUMM
Summary: Present a text summarization dataset for articles of diverse opinions towards same events and schema to find them. Present LLM-based evaluation methods for this dataset. Show that LLMs can well summarize single documents but fail to do so for multiple.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Text summarization
Phenomenon defined by authors: Yes
Phenomenon definition: "In the realm of news reporting, each event is often chronicled by multiple sources, providing a rich tapestry of perspectives and insights…we propose the Multi-document Diversity Summarization (MDDS) task, aimed at faithfully illuminating the diverse information presented in multiple sources. Following Laban et al. (2022), we formalize di- verse information as questions and answers where numerous sources can answer the same question, and the corresponding answers extracted from dif- ferent news articles exhibit a variety of opinions or perspectives "-p570
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: generate a natural-language summary that effectively captures the diverse information presented within clusters of differently-opinionated news articles centered around the same news event
Task item definition: A cluster of news articles, and question for them
Task source: Real task examples (e.g. GitHub issues), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 235*10 news articles
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric definition detail: main results in the paper are evaluation results by humans, but the authors also propose a method to use LLM as judge.
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: authors conduct human experiment
Taxonomy: NLP / Summarization
New taxonomy: General Application / Summarization