Title: Unveiling the Essence of Poetry: Introducing a Comprehensive Dataset and Benchmark for Poem Summarization
Benchmark: PoemSum
Summary: The paper proposes the task of poem summarization for LLMs and presents the first benchmark, PoemSum, to evaluate such capability. SOTA summarization models are benchmarked and limitations of current models on the poem summarization task are discussed.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: poem summarization
Phenomenon defined by authors: Yes
Phenomenon definition: "In recent years, there has been notable research conducted on text summarization in the field of Natural Language Processing (NLP). However, to the best of our knowledge, no such work has been done in the domain of poem summarization yet. While the summarization process of poems seems quite similar to the generic text summarization, there are some major differences between the two... Summarizing literary work poses lots of challenges."
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: A poem is given and a summary must be generated.
Task item definition: Each sample is represented by the poem title, poet name, poem text, poem link, and poem summary.
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 301
Dataset size notes: Train: 2409; Validation: 301
Dataset metadata available: Yes
Dataset metadata detail: number of poets, max poem length, max summary length, avg poem length, avg summary length, avg # poems per poet
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Summarization
New taxonomy: General Application / Summarization