Title: OpenAsp: A Benchmark for Multi-document Open Aspect-based Summarization
Benchmark: OPENASP
Summary: Present a dataset for multi-document open aspect-based summarization. Show the dataset is of high quality and it presents challenge to LLMs.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Text summarization
Phenomenon defined by authors: Yes
Phenomenon definition: "In query-focused summarization (QFS), a query is highly flexible and can target specific information within particular text. In contrast, aspect-based summarization (ABS) datasets traditionally predefined small sets of generic subtopics within a common topi- cal category on which aspect-based summaries are generated. Open-ABS (OABS; Tan et al., 2020), allows aspects to differ for each source text, yet still just as subtopics in the text. "-[first page, no page number in the paper]
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Given a set of documents on the same topic and an aspect, the task is to output a short aspect-based summary.
Task item definition: A cluster of documents and an aspect label
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 192 topics, 596 instances, 6,536 docs
Dataset size notes: Train: 145 topics, 476 instances, 4,878 docs; Valid: 82 topics, 238 instances, 2,168 docs
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: n-gram (BLEU, ROUGE, chrF)
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: the authors conduct human experiment for evaluation
Taxonomy: NLP / Summarization
New taxonomy: General Application / Summarization