Title: STORYSUMM: Evaluating Faithfulness in Story Summarization
Benchmark: STORYSUMM
Summary: Propose a dataset, show that one human annotation protocol is likely to miss inconsistencies, and recent automatic metrics do not perform well either
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: "LLM summaries often contain subtle errors, particularly for narrative text which requires nu- anced interpretation...By focusing on faithfulness in narrative summarization and using real-world data from LLMs and Reddit, STORYSUMM poses a realistic but hard benchmark to push our methods forward." -p9989
Phenomenon defined by authors: Yes
Phenomenon definition: "We define a consistent summary as: The events and details described in the summary should not misrepresent details from the story or include de- tails that are unsupported by the story."-p9990
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: "Is the information in the summary consistent with the story?"-p9990
Task item definition: given a story and a summary, the model/human has to decide whether the summary is faithful to the story.
Task source: Real task examples (e.g. GitHub issues), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: original data is sourced from Reddit (-p9989)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 63 stories
Dataset size notes: val: 33 stories
Dataset metadata available: Yes
Dataset metadata detail: difficulty
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Validation
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Metric definition: Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric subscores: Yes
Metric subscores detail: difficulty
Metric access: Model access required (e.g. logits)
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: the authors conduct extensive human experiment
Taxonomy: NLP / Summarization
New taxonomy: General Application / Summarization