FABLES: Evaluating faithfulness and content selection in book-length summarization
Authors: Yekyung Kim
Yapei Chang
Marzena Karpinska
Aparna Garimella
Varun Manjunatha
Kyle Lo
Tanya Goyal
Mohit Iyyer
Venue: COLM (2024)
Keywords: Faithfulness, Content Selection, Book-length Summarization, Human Evaluation
TL;DR: In this paper, we conduct the first large-scale human evaluation of faithfulness and content selection on LLM-generated summaries of fictional books.

Abstract: While long-context large language models (LLMs) can technically summarize book-length documents (> 100K tokens), the length and complexity of the documents have so far prohibited evaluations of input-dependent aspects like faithfulness. In this paper, we conduct the first large-scale human evaluation of faithfulness and content selection on LLM-generated summaries of fictional books. Our study mitigates the issue of data contamination by focusing on summaries of books published in 2023 or 2024, and we hire annotators who have fully read each book prior to the annotation task to minimize cost and cognitive burden. We collect FABLES, a dataset of annotations on 3,158 claims made in LLM-generated summaries of 26 books, at a cost of $5.2K USD, which allows us to rank LLM summarizers based on faithfulness: CLAUDE-3-OPUS significantly outperforms all closedsource LLMs, while the open-source MIXTRAL is on par with GPT-3.5-TURBO. An analysis of the annotations reveals that most unfaithful claims relate to events and character states, and they generally require indirect reasoning over the narrative to invalidate. While LLM-based auto-raters have proven reliable for factuality and coherence in other settings, we implement several LLM raters of faithfulness and find that none correlates strongly with human annotations, especially with regard to detecting unfaithful claims. Our experiments suggest that detecting unfaithful claims is an important future direction not only for summarization evaluation but also as a testbed for long-context understanding. Finally, we move beyond faithfulness by exploring content selection errors in book-length summarization: we develop a typology of omission errors related to crucial narrative elements and also identify a systematic over-emphasis on events occurring towards the end of the book. We release FABLES to spur further research on the evaluation of book-length summarization.

Published as a conference paper at COLM 2024
FABLÉS: Evaluating faithfulness and content selection in book-length summarization
Yekyung Kim Yapei Chang Marzena Karpinska Aparna Garimella*
Varun Manjunatha*, Kyle Lo Tanya Goyal Mohit Iyyer UMass Amherst Adobe Allen Institute for A Princeton (yekyungkim, yapeichang, mkarpinska, mi miyyer]@umass. edu (garimel1, ymanjunay@@dobee.. com, fylel@allenai..o org, tanyagoyaleppinnceton. edu
Abstract
While long-context large language models (LLMs) can technically summa- rize book-length documents (>> 100K tokens), the length and complexity of the documents have so far prohibited evaluations of input-dependent aspects like faithfulness. In this paper, we conduct the first large-scale human evaluation of faithfulness and content selection on LLM-generated summaries of fictional books. Our study mitigates the issue of data contam- ination by focusing on summaries of books published in 2023 or 2024, and we hire annotators who have fully read each book prior to the annotation task to minimize cost and cognitive burden. We collect FABLES, a dataset of annotations books, at cost of $5.2K USD, which allows us to rank LLM summarizers based on faithfulness: CLAUDE-3- --ODUS significantly outperforms all closed- source LLMs, while the open-source MIXTRAL is on par with GPT--.5-TTRRR An analysis of the annotations reveals that most unfaithful claims relate events and character states, and they generally require indirect reasoning over the narrative to invalidate. While LLM-based auto-raters have proven reliable for factuality and coherence in other settings, we implement several LLM raters of faithfulness and find that none correlates strongly with hu- man annotations, especially with regard to detecting unfaithful claims. Our experiments suggest that detecting unfaithful claims is an important future direction not only for summarization evaluation but also testbed for long-context understanding. Finally, we move beyond faithfulness by ex- ploring content selection errors in book-length summarization: we develop a typology of omission errors related to crucial narrative elements and also identify a systematic over-emphasis on events occurring towards the end of the book. We release FABLES to spur further research on the evaluation of book-length summarization.
3,158 claims made in LLM-generated summaries of 26
https: wttpccco/mmnggg//FABBLEEL
Introduction
Advances in long-context language models have sparked interest in summarizing book- length documents (>100K tokens). Despite the importance of faithfulness and content relevance for summary quality, recent work in this regime focuses only on input-agnostic aspects like coherence (Chang et al., 2023b). This is due to the length and complexity of the input documents: hiring human annotators to read and understand them is expensive and time-consuming. Our work fills this gap by presenting the first large-scale human evaluation of faithfulness and other content selection errors book-length summarization.
We mitigate challenges associated with document complexity by hiring workers who have already read a book published in 2023 or 2024 (to avoid data contamination) for enjoyment prior to beginning the annotation task. We produce summaries for these books via five
1

---

Published as a conference paper at COLM 2024
claimy
dasing
datim
Stammmrr
daiin
GPT-4
claimy
(a) Summarization
(b) Claim Extraction
(c) Human Evaluation
Figure 1: Our pipeline for collecting faithfulness annotations in book-length summarization (FABLES). First, (a) we generate summaries through hierarchical merging. Next, (b) we prompt GPT-4 to extract decontextualized claims. Finally, (c) we conduct human evaluation of these claims, requiring annotators to validate each claim and provide their reasoning and evidence for the assigned label.
configurations of the hierarchical summarization methodology described in Chang et al. (2023b), each of which varies the base LLM and chunk size. Following prior work on faith- fulness and factuality evaluation, such as LongEval (Krishna et al., 2023) and FactScore (Min et al., 2023), we decompose each summary into list of claims which are then individually verified against the input document.
In total, our FABLES dataset (Faithfulness Annotations for Book- Summarization) contains 3,158 claim-level annotations of faithfulness across 26 narrative texts, along with evidence for each claim in the form of quotations from the book as well as free-form comments at both the claim and summary level (Figure 1). Overall, we observe that CLAUDE- 3-OPus is the most faithful book-length summarizer by significant margin, followed by GPT-4-TURBO. Beyond ranking LLMs,, our annotations also shed light on the following previously unexplored questions:
What kinds of faithfulness errors do LLM summarizers make? (53) A qualitative analysis of FABLES reveals that the majority of claims marked as unfaithful are related to evenís or states of characters and relationships. Furthermore, most of these claims invalidated via multi-hop reasoning over the evidence, highlighting the task's complexity and its difference from existing fact-verification settings (Min et al., 2023; Kamoi et al., 2023).
only be
Can faithfulness be evaluated automatically? (54) Collecting human annotations on 26 books cost us $5.2K, demonstrating the difficulty of scaling our workflow to new domains and datasets. We thus implement multiple LLMebased raters of faithfulness, following prior work such as BooookScore (Chang et al., 2023b) and FactScore (Min et al., 2023) that achieve high correlation with human judgments. However, all of our metric configurations struggle to reliably identify unfaithful claims. Our best-performing method operates similarly to seeedl-in-thh-haystee seee-haystckk""sst "-style evaluations (Kamradt, 2023; Gemini Team, 2024) by feeding as much of the book as possible into a long-context LLM along with a single claim to verify. We promote this claim-level verification task as both important for book-length summarization evaluation as well as challenging benchmark for long-context understanding.
What other errors, beyond faithfulness, do LLM summarizers make? ($5) By coding all of the summary-level free-form comments in FABLES, we find that annotators frequently point out omissions of critical information. We develop the first taxonomy of omission errors in book-length summarization and observe that key events, details, and themes are frequently omitted by all LLMs. We also observe other content selection errors: for example, even our strongest summarizers, CLAUDE-3-OPUt and GPT-4-TURBO, overemphaaiee content towards the end of books to the detriment of the beginning.
All prompts used in this paper can be found in ŞB.
While cannot release the book text due to copyright restrictions, publicly release all summaries and annotations.
2

---

Published as a conference paper at COLM 2024
Books
Annotations
Documents (n=26)
Summaries (n=130)
Claims (1=3,158)
Reasons (m=1,513)
Evidence (m=3,051)
Comments (n=130)
Mean 121,467
594.3
19.8
37.6
194.7
153
St. dev.
35,836
119.5
6.4
33.4
218.5
148.4
Max 243,965
798
58
281
2435
823
Min 49,762
172
5
6
Table 1: Number of tokens across books and FABLES annotations; based on tiktoken (https: / //github www.oppeai/tiiiiiicciceooooo.iee /ttnaiiiiiii hittai/iiiikttkn)nm/ tokenizer.
2 Collecting human annotations
In this section, we describe our pipeline for collecting FABLES, which consists of human annotations of both faithfulness and overall quality of LLM-generated book summaries.
Collecting a corpus of newly-published fictional books: It is infeasible, both terms of cost and time, to ask annotators to read long books (2 100K tokens) for the sole purpose of annotating LLM-generated summaries. While we can remove this burden by choosing famous books that many people have already read, such as those in BookSum (Kryscinski et al., 2022), LLMs have also likely seen these books and their summaries during pretrain- ing (Chang et al., 2023a), which can skew the evaluation generated claims. Instead, we use an annotator-driven workflow to sidestep these issues. We recruit pool of annotators via Upwork2 who self-report having read one or more English books published in 2023 or 2024. Our final annotator pool consists of 14 native English speakers, and we purchase electronic copies of 26 books listed by them.3 3 The mean length of books our dataset is 121K tokens (see Table 1 for statistics).
Prompting LLMs to generate book summaries: To summarize book-length documents, we adopt the hierarchical merging strategy from (Chang et al., 2023b); see Figure 1 for an illustration of the method. We use GPT-3.5-TURBO, GPT-4, GPT-4-TURBO (OpenAI, 2023), MIXTRAL (Jiang al., 2024), and CLAUDE-3-OPCS (Anthropic, 2023) the backbone models."
Decomposing summaries into claims: Following prior works on evaluating long-form summary faithfulness (Krishna et al., 2023; Min et al., 2023; Wei et al., 2024), we decompose our summaries into atomic claims enable fine-grained annotation. We prompt an LLM (GPT-4) with two primary instructions: (1) each atomic claim must be fully understandable on its own without requiring additional context from the summary (e-g., resolved pronouns), and (2) whenever possible, each claim should be situated within its relevant temporal, locational, and causal context. Human validation by the authors of random sample of 100 extracted claims demonstrated 100% precision (i.e., each claim can be traced to the summary without any extra or incorrect information). See Figure for example of summary and its extracted claims; see SB for exact prompt and 5G.4 for recall analysis.
Collecting human annotations: objectives:
The Upwork annotators were tasked with two primary
Claim-level: Assess the faithfulness of claims extracted from model- gedee--geerratdd summaries of their assigned book(s). Annotators reviewed claims made about their selected book(s) and determined their accuracy by choosing one of four options for each decomposed claim: (a) Faithful accurate reflection of the narrative, (b)
htt htttt:::/www.w.e hhttps://ww.upporrkk upwork. com
We convert epubs text files preserving all information including front and back matter. AAll summaries were generated in February 2024 using the following checkpoints: 5-turbo gpt-4-0613, gpt-4-011 1--122 25-preview, Bastrrl--x7B-nnttrrue and claude- moude-3-opus-222400
3

---

Published as a conference paper at COLM 2024
Summary
Extracted Claims
this eecerpt from Curiis Sittenfidd's unvel "Romantir Oomelyy.
Sally Milk is wrker for the comuddy skutch show The Night Owl (TNO).
Sally Mila, a viiteer for the comedy sketch show The Night Owls
2. Mustcian Neah knowerr hosted The Night Ouk pou8
(INO), develops romantc relationship with musician Noah
Two years after Noah's stint on TNO, initites contact with Sally via emtill
Kirmeter, who hosted the show 2018. Two years afte Nosh's stint
Salty and Neah begin correspondingg and quickt derelop clase bond.
on TNO, he readues out to sally via email They begin corresunding
disuusiig thisk lives, carsers, und personal philossphis.
und quiclly develop a close hond, diocussing their lites, eareets, and
Sully, who his been TNO for years, cousides leaving the slure parsue carees in screenwriting
passual philoosphiss Sully, who las boan al TNO for yeur, in
considering leasing to puntue screenwriting She drives from kanstne-
Sally drives from her home in Kanses City to visit Nosh his house in Los Angeles
7-
City to vist Noat his home in Los Angeles, where their
rnfationahip becomes physically intimate. Honewerr the budding
During her valt Low Augelss, Suily and Noatlss relat cosddip boccomes physoally intimate
romunnc faces challenges. Suily strugates with insouuibes about
dating a culbrity and the public senatiyy dut comee with it Ste
Sulty struzcles with insecurities about dating celebrity and the public senuliny that comes with it.
9
- brisfly retronti to hotd to son out her loclings Mesawhilic Sulli 8 4
stepdalter Jeny tnis ill with COVID 29 back in Kanses Noth
Sally iompururlyy relenas to botdl out bur fealings about bar relationchip with Noah.
11
peccompariss sor Sally to care for Jory, demoostreingg his commiment and willingues to suppoct her through diffizult times. This
Sally's stepfather, Jerry, fills ill with DOV11-19 Kansae City,
12
experience deepens their connection. Sally ultmaately decides to
13
10. Nush accumpanios Sally back Knmoow City carn for her sick stepfather, Jerry.
laave-No and move to Los Augils to be will Noall, They put
14
amariee in private ccremony in ooen Sally works on her first
15
12. The expariatioo of caring for Junry dangans the connection but morm Sally
feature film accipt whilg Noah comtinues his musec career, touring
and Noah.
16
Sifum passible, Jurry and his boogle Sugar move in with the couple
Sally decides leove I INO and move to Angeles to be with Neohi Sally and Nowh married in privste cnnemonry scor
The novel explones themes of finding love later in life, navggtingg the
18
challengee of fame, and balancng pemonal and professional
Afber script
Los Angeles, Sally begins warking on first Seature film
sepirationss It also tonshee on the impact of the D0V111-1 pandemii
bon zelatlonshipss and family dynamies, Thounhout the stoey, Salty
21
Nual continues liss music
and his beag'n, Sugar, Los Anaekss touring possibtt
gropples with her own insecuttees and leams to embraaee happiness uta love when thy come into ler life unospaciaally.
with sally and Nonh
The "Romantic Comedi" cuplores themes of findlngg later lifel
22
novel
challenges
Iolancing prrsenso
professional arpirations
impaac the COVID-19 pandenic relationshije and fumily
"Romantic Comedyy' by Curtis Sittenfeld
Summarry by Clande-g-Opus
Throughout the novel, grappees with her lsseccaties
happiness ausqueteslyy the uovdl Canedly"
they
Figure 2: Example summary generated by CLAUDE-3-OPUS and claims extracted by GPT-4.
Unfaithful misrepresentation of the narrative, (c) Partial Support - partially partially corrobo- rated by the narrative, or (d) Can't verify - indeterminable. They provided free-form textual justifications to support their selections, including evidence in the form of quotations from the book when relevant.
Summary-level: Provide free-form summary-level comments on the overall quality of the summaries. Annotators critiqued the claim set as a whole, identifying omissions, inaccuracies, disproportionate emphasis on trivial plot points, or other concerns.
The annotators used a customized interface, 5 which provided them full access to the book text for reference. Each annotator was assigned to annotate all five LLM-generated sum- maries for their assigned book, which were presented in randomized order. Annotators received $200 for this task, which took ~11 hours to complete (STD=6.34). In total, FABLES contains 3,158 annotated claims from 130 summaries across 26 books at cost of $5.2K USD. We assess the quality of our dataset using inter-annotator agreement and self-consistency metrics. More details can be found in SC.
3 Developing a taxonomy of faithfulness errors in FABLES
In this section, we present results from our statistical and qualitative analysis of the 3,158 claim-level faithfulness annotations in FABLES, which include both free-form comments and citation evidence to support refute these claims." Broadly, we observe that CLAUDE-3- OPus is the most faithful LLM summarizer, with 90% of its claims rated as faithful, followed by GPT-4 and GPT-4-TURHO at 78%, GPT-3.5-TURRO at 72%, and MIXTRAL at 70% (Table 2).
RReee to SC for the screenshots of the interface and the exact wording of the task. 6For 107 claims, the annotators were unable cite evidence either in favor or against the claim.
Note that GPT-4-TURBO, MIXTRAL, and GPT-4 are capable of processing sequences of up to 16K, 32K, and 8K tokens respectively. In this study we opted to use shorter context window to maintain consistency with the experimental setup described in Chang al. (2023b).
4

---

Published as a conference paper at COLM 2024
Model
Chunk size
Avg # ClaimssTD
Faithful
Unfaithful
Partial support
Can't verify
GPT-3.5-TURB0
2,048
23.23329
71.52
11.26
13.08
4.14
MIXTRAL
2,048
27.30
68.67
11.47
17.2
2.66
GPT-4"
2,048
26.23354
78.15
4.55
15.98
1.32
GPT-4-TURBO
100,000
21.652.35
77.62
7.64
12.08
2.66
CLAUDE-3-OrUS
180,000
22.85
90.89
2.1
0.35
Table 2: Percentage of claims extracted from LLM-generated summaries rated by humans as faithful, unfaithful, partial support can't verify. Chunk size denotes the token count per chunk used for summarization across models; we also include the mean and standard deviation of claim counts in generated summaries. Please note that the percentage of each label for CLAUDE-3-OPe is calculated from 24 out of 26 books. The model was unable to merge summaries for two books due to content discrepancies.
Analysis of unfaithful claims: To further study the nature unfaithful claims, we charac- terize all 205 such claims along two dimensions: CLAIM TYPE and REASONING TYPE (see Table 3 for taxonomy and frequency counts). 8 Most unfaithful claims are about specific events (31.5%) or the state of some character or relationship (38.6%). Crucially, a majority of unfaithful claims require indirect reasoning to refute (50.2%), making this more challenging faithfulness evaluation setting compared to prior work (Kamoi et al., 2023; Min et al., 2023). More details on this analysis can be found in SE.
LABEL
FREQ
EXAMPLE CLAIM
REASON FOR REJECTION
Claim Type
State
38.6
Roman Kitt under pressure from father join the family business.
Roman not under pressure, father bribes people he gets his dream job.
Event
31.5
Patricia Lit,, Athena's mother, discovers that Jame has sold Athena's manuscript and confronts her.
Patricia never confronts June.
Cause/effect
11.2
Lilly's abusive ex-boufriend, Alan Bushy, becomes suspect due the meticulous nature of ders.
He becomes suspect because was abusive Lilly:
High-level
11.2
The narrative non-linear and features flashbacks, switches between alternate worlds viewpoints, and present day corversations between Sally and Danny
The narrative largely linear.
Introspection
7.5 Juniper Song encounters Athena Liu at a literary event, triggering feelings of admiration, intintiaa- tion, and self doubt.
No part of the book shows that Juniper admires Athena.
Reasoning Type
Indirect
50.2
Dea stirs up tensions with palace server Fawn.
This encounter is merely Rennick being protective of Amelia, tension can't be in- ferred from the book.
Direct
36.8
The narnatine reveals that Maggie brief affair with doctor named Danny Bangkok while she being followed by unknown entities.
The book directly states that they are married, so it's not a brief affair.
Subjective
7.2
Forest is torn between desire to protect Iris and confronting Itis past actions.
don't think Forest makes any real effort to confront his past actions
Extra info
5.7
book "Wildfire" is the first the Toebreaker series.
It's not stated in the book, but this is ac- tually the second in the series.
Table 3: Taxonomy of faithfulness errors with respect to claim type and reasoning type in FABLES. For each label, we report its frequency and provide example claim-reason pair. More examples and the general labeling scheme can be found in Table 15.
8 There are actually 247 annotations with unfaithful claims, but for this analysis we leave out 42 unclear ones that require further clarification from the annotators. Note that since the claims sometimes contain multiple subclaims, we allow each annotation to have multiple labels.
5

---

Published as a conference paper at COLM 2024
4 Challenges with automatic faithfulness evaluation
While insightful, human annotation of faithfulness in book-length summarization is simply not scalable: our annotations cost $40 USD per summary for a total cost of S5.2K USD, which is prohibitively expensive for usage during model development and with bigger corpora. In this section, inspired by methods such as FactScore (Min et al., 2023) and BooookScore (Chang et al., 2023b), we develop LLM-powered automatic raters of faithful- ness that operate at the claim level. However, our best method, which relies on prompting CLAUDE-3-OPU with the entire book to verify single claim, is expensive and unreliable at detecting unfaithful claims in FABL ES, suggesting important directions for future work.
Automatic raters of faithfulness: We implement our automatic raters by prompting an LLM in zero-shot manner to verify a single claim given evidence from the book (Table 13), where the evidence can be one of the following:
None: As a lower bound, we evaluate the faithfulness of claims without any evidence from the book.
We can also use human-annotated evidence from FABLES obtained via the pipeline described in $2. This evidence is always related the claim, but it often takes the form of short, highly-contextual spans that may or may not be sufficient to support claim verification.
Human evidence:
BM25 retrieval: We employ BM25 (Robertson et al., 1995) to retrieve passages from the book using the claim as a query, We concatenate the most relevant passages to use as evidence for our evaluation prompt. We set = 5 and chunk passages up to 256 tokens. See SG.4 for performance changes when varying passage length.
Entire book (EB): Retrieval is especially challenging in our setting due to the complexity of both the query and document. Intuitively, long-context LLMs can bypass explicit retrieval by simply fitting the entire book into the context as evidence. seelleiin-th-hhassee
This setting resembles 2023; Levy et al., 2024), except that it tests much deeper understanding of the input document.
evaluations of prior work (Kamradt,
Dataset for experiments: Due to budget constraints associated with the "entire book" setting, we select seven books, each shorter than 125K tokens, to evaluate the performance of our auto-rater configurations. This results in 723 total claims, 69 of which are marked as Unfaithful and 654 Faithful by our human annotators. Note that we do not consider par- tially supported or unverifiable claims in our experiments due to the increased subjectivity associated with these labels. Detailed information regarding this dataset and experiment costs can be found in SG.
Results: We evaluate the performance of each auto-rater configuration by comparing its predictions to the ground-truth labels (Faithful and Unfaithfull from our human annotations. Due to the class imbalance, we report separate F1 scores for each label, split across claims generated by different LLMs, in Table 5.9 As a sanity check, the "no evidence" setting per- forms extremely poorly; more interestingly, human evidence underperforms both retrieval and the entire book setting, suggesting that the LLM requires more context to judge claim validity. The best performing auto-rater is CLAUDE-3-OrUS in the entire book setting, which significantly outperforms both GPT-4-TURBO in the same setting as well as BM25.
Conclusion: Despite it having the best performance in Table 5, CLAUDE-3-OPUS ultimately performs too poorly be reliable auto-rater (58.2 F1 when classifying Unfaithful claims). This comes as surprise as this pattern of bocompose-then-veriie correlate with human judgments in other settings, like Min al. (2023). Manual analysis of
has been shown to
We note that scores for Unfaithful claims per-model level should be taken with grain of salt due to the small sample size, particularly for CLAUDI-3-OPUS summaries.

---

Published as a conference paper at COLM 2024
The climax of the story the long- anticipated wedding of Americ and Arnold.
Amelia experiences a moral crisis upon learning of Rennick's s role Gilpin's murder.
While the weoding is the end goalipinn for occurs the protagomist. Janelle and her sister Americ have confronsazion.
FALSE:
TRUE:
she mad that Rewnick killed Gilpin. Then she is happy that he killed him for her Her crisis is that olay with
Gilpins dearh.
A
TRUE
TRUE
FALSE
FALSE
Justine eventually uccepts Price's explanation, leading to an improved relationship between them.
Martha and Patrick have been married eight years.
Justine doesn't accept Mrx Price explanation, Mrs Price attacks her and starts to strangle her, that
FALSE:
Martha is referving to the eight years and Patrick had heen married for: "The sofa became our stoch onewer to the question of how we after that, few variations, People always laughed..
TRUE:
Justine kills Mrs
Price with the special pen that her mother gave to
for eight yours,
AI TRUE
FALSE
TRUE
FALSE
Figure 3: Examples of mistakes in label prediction made by CLAUDE-3 3-PPUS and GPT-4- TURBO accompanied by annotator labels and reasoning. More examples can be found in Figure 11.
the errors reveals that CLAUDE-3-OPUS struggles most with claims involving non-narrative information (23.1%), assessments often based common sense reasoning (20.5%), and character confusions (12.8%), which often require a deep understanding of the entire book; confusion matrix in Figure and more details in $32. Qualitatively, from annotator comments (Table 4) the difficulty of this claim verification task as evidence may be difficult to localize (in "needle-in soadle-in-the-hastacee manner) and require full document reasoning.
can also gauge
COMMENTS
hardest part was some of the claims were very general about the text, such as describing overall character arcs, which made it hard to find specific textual support.
most difficult part for me was how general some of the sentences were. Because material was so broad, I felt that could use 20 or 30 quotations. For example, this book is about many stories of a private investigator in Africa (not exactly what it said, but close). I could recite the entire book.
O The most difficult part for me was finding supporting quotations for claims that were more abstract (e.g. "The book grapples with the scars of colonialism. "). Although was able to tell right away whether the claim was true or false, based on my own reading, specific quotation that best proved the clain. The themes were more often implicit the text, rather than explicit.
at times difficult find
O The most difficult part was to give citations for claims about writing style and intent. The reason was that these claims are usually based on the book as a whole, so an accurate citation would be the whole book.
Table 4: Annotator comments highlighting the challenges in evidence retrieval.
Diseussion: It is generally agreed that benchmarking the faithfulness of LLM-generated text is important. However, recent efforts have primarily focused on verifying entity-centric facts (Min et al., 2023). Our work, and others (Zhu et al., 2023; Tang et al., 2024; Mishra et al., 2024), show that these do not provide coverage in more challenging settings like book summarization. Moreover, the retrieve-then-verify framework that forms the backbone of most past evaluation techniques (Bohnet et al., 2022;
all types of LLM errors, especially
7

---

Published as a conference paper at COLM 2024
Gao et al., 2023) completely fails for our significantly more challenging setting. Given this evidence, we call for broadening the scope of error types and task settings (including our current task of book-length summarization) considered by current faithfulness evaluation benchmarks.
No-Context
Human Evidence
Entire
Entire Book CLAUDE-3-OPUS
Summirizal by
GPT-4-TURBO
GPT-4-TURBO
GPT-4-TURBE
GPT-I-TURBO
Failhful
Unfaithful
Failhful
Unfaithful
Failhful
Unfaith/ul
Failhful
Unfaithjul
Faithful
Unfaitiful
CPT-3.5-TUR80
0.396
0248
0.686
0.369
0.801
0.373
0.887
0.357
0.929
0.619
MIXTRAL
0.248
0.178
0.760
0.361
0807
0.312
0.966
0.440
0.922
0.645
GPT-4
0.337
0.146
0.657
0.229
0.739
0.162
0.909
0.230
0.999
0.600
GPT-4-TUABO
0
0,217
0.650
0.264
0.794
0.241
0
0.109
0.915
0.600
CLAUDE-3-OPUS
0.242
0,018
0,510
0.022
0.692
0.000
0.962
0.000
0.971
0.000
Overall
0.305
0.167
0.249
0,952
0.386
0.955
0.582
Table 5: Fl scores for Faithful and Unfaithful label across models with evaluators on 7 books. The best results of each label are in bold. Entire Book refers to the entire book method evaluating faithfulness from large (125k) chunks using either GPT-4-TURBO or CLAUDE-3- OPUS.
5 Beyond faithfulness: content selection errors in book summarization
As book-length summarization is still a nascent area, research into other error types beyond coherence (Chang et al., 2023b) and faithfulness ($3) is still lacking- In this section, we perform qualitative coding over all 130 free-form, summary-level comments from FABLES and present a taxonomy of content selection errors (e.g., omissions) that may prove more difficult to detect than faithfulness. 10
General issues with LLM-generated summaries: Table 6 summarizes the percentage of summaries affected by specific issues as per annotators' comments." 11 Our analysis shows that every LLM makes chronological errors, though these were less pronounced in models with extended context BLLAUDE-3-OPULE and GPT-4-TURBO). All models were also criticized for omitting important information, with CLAUDE-3-Ord O-UUS being the least affected (52%), compared to 80.8% and 84.6% for GPT-4-TURBO and GPT-3.5-TURBO, respectively. The least faithful models, GPT-3.5-TURE and MIXTRAL, also both have tendency to generate overly generic statements (38.5%). Finally, we look also at cases where the summary was explicitly praised for being good or comprehensive. CLAUDE-3-OPUS received the most praise (48% and 54% respectively), while OPT-3.5-TURBO received the least (11.5% and 15.4% respectively).
Exploring omission errors: As mentioned above, omission of key information plagues all LLM summarizers. To better understand the nature of the omission errors identified by our annotators, we categorize them into the following categories: characters, events, details, relationships, themes. 13 Figure shows heatmap of omission errors broken down by model. A large proportion of summaries (33.3% to 65.4%) lack mentions of key events, creating gaps in the overall narrative, and we also note omissions of significant details about the characters, events, or objects (16.7% to 38.5%). Furthermore, GPT-4-TURBO and MIXTRAL have a tendency to entirely omit mentions of crucial characters (23.1%).
Details of the annotation scheme used to analyze the comments are in Table 21 in the SF 11 II two cases, CLAUDE-3-OPL refused to merge two summaries, as they were affected by the extra information available in the front and back matter and did not constitute a logical story. We excluded these cases from this analysis.
Perercet of summaries where the annotator expressed specific concerns about the factuality of the entire claim set. See $D for the percentage of affected claims per summary. In short, most summaries contained factual inaccuracies with only five summaries receiving 100% of Faithful labels (indicating complete factual accuracy).
13Since annotators did not identify every specific omission, we focused on a binary classification: whether summary was impacted by given omission type, rather than counting the total number of omissions by type. See Table 22 in the 5F for more details.
8

---

Published as a conference paper at COLM 2024
CLAUDE-3-OPUS
GPT-4-TURBO
GPT-4
GPT-3.5-TUREO
MIXTRAL
Chronology
33.3
36.0
46.2
50.0
61.5
Omissions
52.0
80.8
65.4
84.6
65.4
Factuality12
58.3
69.2
84.6
Overemphasis
20.8
34.6
19.2
30.8
46.2
Underemphasis
12.5
23.1
19.2
38.5
34.6
Vague/Generic Generic
0.0
23.1
3.9
38.5
38.5
P Repetitive
0.0
11.5
0.0
7.7
3,9
Data-Influenced
19.2
19.2
34.6
Comprehensive
54.2
30.8
38.5
15.4
34.6
A Well-done
50.0
23.1
26.9
11.5
15.4
Table 6: Percentage of summaries per model identified with specific issues, based annotator general comments (not the claim-wise faithfulness ratings). The upper row, colored in purple, outlines categories of critique, whereas the lower row, in green, indicates categories where the models received compliments.
mijier chanoster: ghout of Com..."
4.17
23.08
7.89
7.69
23.08
Temnic'' muthor's
characters
death atso omiieed whec poter...
33.33
57.69
38.45
66.38
38:46
events
hage pult point Conser and Ruman belng the
soms penaas."
16.57
38.46
34.02
34.62
38
attributes
There eeension of
Cluue and Asss
8.33
11.54
11.54
11.54
15.
relatiinnsips
with
her scmullity 6-3 the mnjor thonces..
15.38
Claude-3-
GPT-4-Turbo
GPT'4
GPT-3.5-Tur
Mixtral
Figure 4: Percentage of summaries flagged by the annotators for one of five omission errors, characters, events, attributes, relationships, and themes, by model.
Long-context models overemphasize book endings: One interesting observation is that CLAUDE-3-OrUS and GPT-4-TURBO, which both have chunk sizes 100K, tend to place more emphasis on the endings of the books to the detriment of the beginning. Since these models were often provided with the entire book context during prompting, this suggests a potential issue in processing long inputs (Kamradt, 2023; Levy et al., 2024). This phenomenon especially prominent with CLAUDE-3-OPUS, where at least 20% of the generated summaries exhibit an overemphasis on the book's 's ending, compared to 7.7% for GPT-4-TURBO (see examples in Table 25 in the ŞIF). We also note that the back matter many books (e.g,, author's biography, dedications, etc.) often unduly impacts all LLMs during the summarization process. We observe conflation between characters in the narrative and names in the back matter, well entirely hallucinated narratives; CLAUDE-3 3-OPUS is the only model seemingly unaffected by this additional information; see SF for more analysis this phenomenon.

---

Published as a conference paper at COLM 2024
6 Related work
Narrative summarization: Our paper builds on prior work in narrative summarization, including short stories (Wang et al., 2022; Subbiah al., 2024), poetry (Mahbub et al., 2023), screenplays (Chen et al., 2022), among others. Wu et al. (2021) demonstrated how an LLM can overcome long context to summarize books, like those in the BookSum (Kryscinski et al., 2022) dataset. Closely related to our work is Chang et al. (2023b), but while they focus evaluating summary coherence (which requires only judging the model generation), we address faithfulness and content selection (which requires relating model generations back to the long source inputs).
Faithfulness and content selection in summarization: Our paper builds on prior work in evaluating hallucination and inconsistency in summarization (Maynez et al., 2020; Kryscin- ski et al., 2020; Ladhak, 2024) which are even challenging for humans (Daumé & Marcu, 2005). Pagnoni et al. (2021) introduce the FRANK dataset, where they use human annota- tions of generated summaries to produce a taxonomy of factual errors based on linguistic analysis, resembling the work of Goyal& Durrett (2020) and Goyal & Durrett (2021). Closest to our work, Krishna et al. (2023) perform human evaluation of faithfulness on summaries of short stories, whereas we study book-length inputs. Our exploration of omission errors rooted in prior research on content selection (Nenkova & Passonneau, 2004; Gillick & Liu, 2010; Ladhak et al., 2020).
Claim verification for evaluating summaries: Our paper also relates to prior work on claim verification, where claims verified given reference to some knowledge source (Thorne et al., 2018; Wadden et al., 2020; Schuster et al., 2021). Min et al. (2023) propose FActScore, an LLM-based metric of factual precision in biography generation, which was expanded upon in SAFE (Wei et al., 2024). Manakul et al. (2023) propose SelfCheckGPT, which uses LLMs to evaluate the faithfulness of GPT-3 generated texts dataset of Wikipedia-style passages about people.
Conclusion
We present FABLES, the first large-scale human evaluation faithfulness and content se- lection in book-length summarization. By recruiting annotators who had read recently- published books for enjoyment, we collect 3,158 claim-level faithfulness annotations from LLM-generated summaries of 26 narratives. This allows us to rank LLM summarizers based on faithfulness, revealing that CLAUDE-3-OPUS is the most faithful book-length summarizer, followed by GPT-4-TURBO. Next, experiment with using LLMs for automatic claim verifi- cation. Our results expose the limitations of both retrieval and long-context understanding: LLM auto-raters cannot reliably detect unfnithful claims, even when prompted with the full book text. Our analysis shows that unfaithful claims primarily pertain to states and events, often necessitating reasoning over extended contexts, which makes them complicated to detect for both humans and machines. Finally, we move beyond faithfulness to explore and characterize common content selection errors such as omissions of key events, attributes, and characters, as well as the over-emphasis of content from the end of the book.
Our work on FABLES suggests several promising directions for future work. With better auto-raters of faithfulness, we can perform fine-tuning or preference tuning on long-context language models by using the auto-raters as a scorer (Tian et al., 2023), which could improve their summarization capabilities by reducing hallucination (Cao et al., 2021). Additionally, FABLES can used as a dataset and protocol to meaningfully benchmark future work on novel long- long-context language model architectures and training objectives.
Ethical considerations
All annotators consented to the use and publication of their annotations. The dataset excludes copyrighted texts, containing only annotations done on model-generated sum-
10

---

Published as a conference paper at COLM 2024
mary claims. Additionally, ensured annotators received fair compensation for their contributions.
Acknowledgments
We extend special gratitude to the Upwork annotators for their hard work, and to members from the UMass NLP lab for their feedback. This project was partially supported by awards IIS-2202506 and IIS-2312949 from the National Science Foundation (NSF) well as award from Adobe.
References
Anthropic. Model Card: Claude 3. Technical report, Anthropic, 2023. URL
https:/ hool:://wwwweeee cdn wttp:/wwc...athro.....ccoo wtt..aapp...cmm//eebaaaaaaaaaaaa
cot.c///dd..aaooo.caaaaaaaaaaaaa (ccbbcchabf533336677666666 8857627/ /Model_ Card_Claude_33 pdf. Accessed: 2024-03-25.
Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, et al. Attributed question answering: Evaluation and modeling for attributed large language models. arviv preprint ar arv:22212.008033 2022.
Mengyao Cao, Yue Dong, and Jackie Chi Kit Cheung. Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization. In Annual Meeting of the Association for Computational Linguistics, 2021. URL httph: semanticscholar... org/ CorpusID:24490944.
Kent Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. Speak, memory: An archaeology of books known to PRaGGPT/GPT-4 In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Pp. 7312-7327, Singapore, December 2023a. Association for Computational Linguistics. doi: 11..6663/v1222..mmmmll 1e..emmll-maii45555 453n URL https:://clanthhology 2023.emnlp-mailn
Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit lyyer. Booookscore: A systematic exxtemmaticepploration of book-length summarization in the of lims. ArXiv, abs/ /2300.0088 2023b. URL
00785.
Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. Summscreen: A dataset for abstractive screenplay summarization, 2022.
Hal Daumé and D. Marcu. Bayesian summarization at duc and a suggestion for extrinsic evaluation. In Document understanding conference, 2005/// 2005.
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. Xiv preprint ar Xlte:2305.14677, 2023.
Google Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
Dan Gillick and Yang Liu. Non-expert evaluation of summarization systems is risky. In Chris Ballc and Mark Dredze (eds.), Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's 's Mechanical Turk, pp. 148- 151, Los Angeles, June 2010. Association for Computational Linguistics. URL https:
//aclanthology.org wwwpollgg.ogg//iie... wwwp://owww//////00///
Tanya Goyal and Greg Durrett. Evaluating factuality in generation with lependency-level entailment. Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3592-3603, Online, November 2020. Associ- ation for Computational Linguistics. doi: 10.18653/v https://acllnthhl..og wrgp/22000.../ htts//222...iniinsseseseee eeeiings-ennl.... 322.
URL
11

---

Published as a conference paper at COLM 2024
Tanya Goyal and Greg Durrett. Annotating and modeling fine-grained factuality in summa- rization, 2021.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. Wice: Real-world entailment for claims in wikipedia, 2023.
Greg Kamradt. Needle in a haystack. NeedleennHHasstak, 2023.
Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit lyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. LongEval: Guidelines for human evaluation of faithfulness in long-form summarization. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1650-1669, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10.18653/v / 2023.eacl-mmin.....
URL https: //ab:::://clamnee anthooloyyy org/ 2023. eecc-mallaaa 121.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9332-9346, Online, November 2020. Association for Computational Linguistics. doi: 10. 00..863//112222eemllppp-o org/2020 emnlp-main mampppmainn.. 750.
URL
Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. BOOKSUM: A collection of datasets for long-form narrative summarization. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 6536-6558, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653//v 1111865///2022.. findings-emnlp..te 488. URL httpt::t/cclllaaae /ttp::////ccllaholooooooo orapolloooooor///////ooo org/2022. findings-emnlp 488.
Faisal Ladhak. Faithfulness in abstractive summarization: Progress and challenges, 2024.
Faisal Ladhak, Bryan Li, Yaser Al-Onaizan, and Kathleen McKeown. Exploring content selection in summarization of novel chapters. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics, pp. 5043-5054, Online, July 2020. Association for Computational Linguistics. doi: 10.18653 vv2lvvl// a12/202.acllmaall 453. URL https: httpss///laaathhllgoooo orgoorgg/// 2020. acl-main.455 453.
Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact input length on the reasoning performance of large language models, 2024.
Ridwan Mahbub, Ifrad Khan, Samiha Anuva, Md Shihab Shahriar, Md Tahmid Rahman Laskar, and Sabbir Ahmed. Unveiling the essence of poetry: Introducing comprehensive dataset and benchmark for poem summarization. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14878-14886, Singapore, December 2023. Association for Computational Linguistics. doi: 01665//1/2023.emmppmmea 2023. emnlp-main. 920.
URL https::/aclanttoll.on .rg/
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. E.MNIP, 2023.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Dan Jurafsky, Joyce Chai, Natalie Schluter,
12

---

Published as a conference paper at COLM 2024
and Joel Tetreault (eds.), Proceedings of the 58th Annul Meeting of the Association for Com- putational Linguistics, pp. 1906-1919, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/ /1/0553///// h/////1112202 22t.2/20....lmiii.11... URL https://aallanthool.g org/
acl acl-main.172 1733.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 12076-12100. 00g3.023.
Association for Computational Linguistics, 2023. URL https://aclanthhon emnlp-main. 741.
Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. Fine-grained hallucination detection and editing for language models. Xiv preprint ar Xriv:2401.06833 2024.
Ani Nenkova and Rebecca Passonneau. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computationnl Linguistics: HLT-NAACL 2004, pp. 145-152, Boston, Massachusetts, USA, May 2 May 7 2004. Association for Computational Linguistics. URL
OpenAL GPT-4 technical report. CoRR, wbbb//33330087/44 2023. doi: 10.48550 ARXIV22// 08774. URL htths::/ddoii.. htt.://.rrr//1.../////ra////// 2303.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4812-4829, Online, June 2021. Association for Computational Linguistics. doi: naacl-main. 383.
URL https:
//aclan wec.aanthoooy.rrggo.ooo
Stephen E. Robertson, Steve Walker, Susan Jones, Micheline M. Hancock-Beaulieu, Mike Gatford, et al. Okapi at trec-3. NIST Special Publication SP, 109:109, 1995.
Tal Schuster, Adam Fisch, and Regina Barzilay. Get your vitamin C! robust fact verification with contrastive evidence. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 624-643, Online, June 2021. Association for Computational Linguistics. doi: 10. /12653//v/2222..aaall// 52. URL https::////at httpps://aclantholog/ or org/2021 mta.22211nnaanall.eeaaaaaaaaa aaaacc-mainn 52.
Melanie Subbiah, Sean Zhang, Lydia B. Chilton, and Kathleen McKeown. Reading subtext: Evaluating large language models on short story summarization with writers, 2024.
Liyan Tang, Igor Shalyminov, Amy Wing-mei Wong, Jon Burnsky, Jake W Vincent, Yu' an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, et al. Tofueval: Evaluating halluci- nations of Ilms on topic-focused dialogue summarization. arXiv preprint ar arXim:2402.13244) 2024.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large-scale dataset for fact extraction and VERification. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 809-819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 11185///1//111-- URL https: httptht://cclaattotttt/////
13

---

Published as a conference paper at COLM 2024
Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D. Manning, and Chelsea Finn. Fine-tuning language models for factuality. ArXiv, abs/2311.08401,, 2023. URL https: ceshnnt
265158181.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7534-7550, Online, November 2020. Association for Computational Linguistics. doi: 10.18653 / emnlp-main.60% URL https: http:://aaaaaaaaaccco
org/2020. www..eme....eeeemmmmmmmmm maml-pmmin... 609.
Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel R. Bowman. SQuALITY: Building a long-document summarization dataset the hard way. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 1139-1156, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/vi 2022.emnlp-mainn75 URL https: httts:://alaantoology www.lloggg.rggg///// or2/2022. emnlp-main. m 75.
Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le. Long-form factuality in large language models, 2024.
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback, 2021.
Rongxin Zhu, Jianzhong Qi, and Jey Han Lau. Annotating and detecting fine-grained factual errors for dialogue summarization. arvivv preprint arX1::2305.16548, 2023.
A Dataset
In this section, we include further details about FABLES. We list all the books used for summarization in Table 7, along with details about the authors, genre, length, publication date, and variety of English. We also detail the data preprocessing process in $A.1.
A.1 Data Preprocessing
Preprocessing books: In order to obtain the summaries via hierarchical merging, we first purchased books from amazon. com in epub format and converted them into text files, retaining all information intact (i.e., without removing front and back matter). We then used the Huggingface GPT-2 tokenizer to divide the books into chunks fitting the models' context window. During our chunking step, we checked for punctuation marks to ensure that all chunks end with complete sentence. This approach sometimes resulted in chunks being shorter than the specified size, leading to the final chunks of some books consisting only of brief snippets with meta information, which could influence the summaries. Ideally, a robust model would distinguish between supplementary information and the main storyline produce a coherent summary. However, we observed that some models were influenced by this extra information, leading them to fabricate aspects of the story.
Generating summaries: To summarize book-length documents, we adopt the hierarchical merging strategy which Chang et al. (2023b) found to outperform competing approaches in terms of summary coherence. We employ zero-shot prompting to summarize each chunk independently. Next, we form pairs of adjacent chunk-level summaries and again use zero- shot prompting to merge each pair, incorporating added context from previously-generatee merged summaries to ensure coherence and continuity (see Figure 1a). We generate five summaries for each book in this fashion using GPT-3.5-TURNO GPT-4, GPT-4-TURHI (Ope- nAI, 2023), MIXTRAL (Jiang al., 2024), and CLAUDE-3-OPUS (Anthropic, 2023). All sum- maries were generated in February 2024 using the following checkpoints: gpt-3. 3.5-urbbo,
h httttsss//uugggngaaaa....o/o////
14

---

Published as a conference paper at COLM 2024
gpt-4, got-4-turbo-prevve
and claude-3-opps--2224222
We use publicly-reeased code, prompts, and hyperparameters from Chang et al. (2023b) for summary generation. We further prompt GPT-4 model to extract decontextualized claims from the summaries. Examples of summaries along with extracted claims can be found in Table 8, Table 9, Table 10, Table 11, and Table 12.
TITLE
AUTHOR
GENDER
GENRE
LENGTH
PUBLICATION
LANG
A Haunting on the Hill Agency for Scamdal
Elizabeth Hand Laura Wood
F
horror, Gothic historical fiction, mystery, romance fantasy, romance, young adult romance
117,577
Oct 2023 Jan 5, 2023
AmF BrE
F
116,809
Divine Rivels
Rebecca Ross
F
137,616
Apr 4, 2023
AmE
Fairytale of York
Zoë Folbigg
F
134,369
Aug 28, 2023
BrF
Flawless
Elsie Silver
F
romance
119,580
Jun 24, 2022
CanE
Fourth Wing
Rebecca Yarros
fantasy
243,965
May 2, 2023
AmR
Modern Disination
Isabel Agajanian
urban fantasy
167,568
Jan 30, 2023
AmE
Only for the Week
Natasha Bisho
F
African American
87,056
May 11, 2023
AmE
romance
Pel
Catherine
F
thriller, mystery
124,679
July 13, 2023
NZE
Chidgey
Romantic Comedy
Curtis Sittenfeld
romance
116,560
Apr 4, 2023
AmE
Same Time Next Yoar
Tessa Bailey
romance
49,762
Dec 1, 2023
AmE
She is Haintiing
Trang Thanh Tran
F
romance
106,659
Feb 28, 2023
AmR
Six Scorcked Roses
Carissa Broadbent
fantasy romance
54,481
Mar 21, 2023
AmE
Sorrow and Bliss
Meg Mason
mental health
112,468
Sep 2, 2020
AusE
The Atonement Murders
Jenifer Ruff
F
mystery, thriller
105,493
Apr 14, 2023
AmB
The Guest
Emma Cline
thriller
89,977
May 16, 2023
AmE
The Marringe Act
John Marrs
M
thriller, mystery, dystopia
135,901
Jan 19, 2023
BrB
The Spy Const The Wager
Tess Gerritsen
mystery, thriller
128,918
Nov 2023
AmE AmE
David Grann
M
nonfiction,
156,022
Apr 18, 2023
historical
The White Lody
Jacqueline
historical fiction, mystery mystery, fantasy
126,051
21, 2023
BrE
Winspear
This Impossible Brightness
Jessica Bryant
F
119,115
Feb L, 2024
AmE
Klagmann
Viciously Yours
Jamie Applegate
F
fantasy, romance
86,774
Jan 23, 2024
AmE
Hunter
Weyward
Emilia Hart
F
historical fiction, fantasy
128,728
Feb 2023
AusE
Wildfire
Hannah Grace
F
romance, sport
140,060
3, 2023
Brü AmR AmE
Yellourface
R.F. Kuang
thriller
114,346
May 25, 2023
You, Again
Kate Goldbeck
F
romance
137,624
Sep 12, 2023
Table 7: Details of the 26 books used for summaries. Length of each book is provided in tokens as computed with tiktoken.
B Prompts
In this section, we included all prompts used for our experiments: (1) claim extraction, (2) automatic evaluation in Table 13,
C Human Annotations
In this section, we present details of our annotation task. Figure 5 displays the instructions provided to annotators for evaluating faithfulness. Figure 6 illustrates the interface used by annotators for this task. The list of claims displayed on the left side of the screen, with each claim on a separate line. Content of the book is presented on the right side. Annotators can navigate the book's s content using the scroll function and perform keyword searches to locate relevant information. When annotators hover over a claim, it becomes highlighted, and clicking on it triggers a popup window to appear (see Figure 7). Given that completing the annotation process takes considerable amount of time (approximately 1.5h-2.5h), we have implemented a feature that allows annotators to save their work at any point during the annotation process. Upon completing the annotations, the annotator is required to
15

---

Published as a conference paper at COLM 2024
CLAUDE-3-OPUS
Iris Winnow, an aspiring writer, works at the Oath Gazette newspaper the city of Outh, competing with her rival Roman Kitt for a columnist position. Iris's s brother gods Eura and Dacre. She hasn't heard from him in months and decides to become a war correspondent to find him. Iris and Roman develop feelings for each other through letters they exchange via enchanted typewriters, not realizing they are writing to each other. Roman, engaged to Elinor Little through an arranged marriage by his father, breaks off the engagement and follows Iris to the war front. In the town of Avalon Bluff near the front lines, Iris stays with Marisol, who runs a bed and breakfast, and befriends fellow correspondent Attie. Roman arrives and he and Iris fall in love, getting married just before an attack on the town by Dacre's forces. realizes is her brother Forest. Forest, who was supposed to have died from his wounds but was healed by Dacre, deserted and has been fighting for the enemy. In the chaos, Iris is separated from Roman, who is hadly woundedd. Iris and Forest return to Oath, where Iris learns the full story of Forest's s service to Dacre. She discovers letter from Roman in her room, confirming he knew her identity while writing to her. Meanwkilee, the god Dacre finds the dying Roman on the buttlefield and, sensing his strong spirit, decides to take him heal his wounds and have him fight for his army. The story ends with Dacre turning his sights towards finding Enoa in the east.
fighting in a war between the
Summary
forces. During the altack, Iris is saved by disguised soldier who she later
1. Iris Winnow works at the Oath Gazette newspaper in the city of Oath.
2. Iris Winnow is competing with Roman Kitt for a columnist position at the Oath Gazette.
3. Iris Winnow's brother, Forest, is involved in war between the gods Enva and Dacre.
4. Iris Winnow decides to become war correspondent after not hearing from her brother, Forest, for months.
5. Iris Winnow and Roman Kitt develop feelings for each other through letters they exchange via enchanted typewriters.
6. Iris Winnow and Roman Kitt do not initially realize they are writing letters to each other.
7. Roman Kitt was engaged Elinor Little through an arranged marriage by his father.
Extracted Claims
8. Roman Kitt breaks off his engagement Elinor Little and follows Iris Winnow to the war front.
9. Iris Winnow stays with Marisol at a bed and breakfast in the town of Avalon Bluff near the front lines.
10. Iris Winnow befriends fellow correspondent Attie in Avalon Bluff.
11. Roman Kitt arrives at Avalon Bluff and he and Iris Winnow fall in love.
12. Roman Kitt and Iris Winnow get married just before an attack on Avalon Bluff by Dacre's forces.
13. During the attack on Avalon Bluff, Iris Winnow is saved by soldier who she later realizes is her brother, Forest.
14. Forest was supposed to have died from his wounds but was healed by Dacre.
15. Forest deserted from his original side and has been fighting for Dacre.
16. Iris Winnow is separated from Roman Kitt, who is badly wounded during the attack on Avalon Bluff.
17. Iris Winnow and Forest return to Oath after the attack.
18. Iris Winnow learns the full story of Forest's s service to Dacre once they return Oath.
19. Iris Winnow finds a letter from Roman Kitt her room, which confirms he knew her identity
while writing her.
20. Dacre finds the dying Roman Kitt on the battlefield and decides to take him to heal his wounds.
21. Dacre plans for the healed Roman Kitt to fight for his army.
22. The story ends with Dacre planning to find Enva in the east.
Table 8: Example of a summary produced by CLAUDE-3-OrUs along with the extracted set of claims for "Divine Rivals," novel by Rebecca Ross. Examples by the other models can be found in Table 9, Table 10, Table 11 and Table 12.
provide a comment on the overall quality of the summary claims by clicking on general comments (see Figure 8).
How do annotators perceive the task? Annotators highlighted several challenges in assessing the summaries, particularly when dealing with broad claims about themes rather than specific plot points, making it difficult to find relevant supporting evidence within the text. Abstract concepts, like emotions or thematic claims, posed significant obstacles, with some annotators struggling to locate quotations that precisely supported or refuted these
16

---

Published as a conference paper at COLM 2024
claims. They also pointed out the difficulty of evaluating claims that were only partially true, which required more detailed support (see Table for actual comments).
1. Task Overview
Your job to validate the factuality of claims made within an Al- Al-geeerated book summary. will be given the text the book itself. Your task to decide whether not each claim supported by the book, and provide evidence from the book to justify your decisions. You should also comment broadly on major omissions, salience issues, with the chronology events within the of claims. finish the text, must (1) annotate every claim for factuality and (2) write broad comment about the provided list of claims.
claims derived from book summary along with
Make have closely read the entire book prior to starting the annotation process. White we do provide the text of the book along with a simple search functionality, our expectation that you will not need to re-read the book assess the factuality of most claims.
II. Accessing the interface
1. Go the link gave via Upwork message.
2. Look for the file labeled "upwork- [number to 5) [bookok_ [book_name]..ire name)..tmL.
will be given 5 different claims from same book. Let's start annotating III. Annotation Instructions
You will be working with the following interface:
Malant monnorot bocE
Mataet sran cramsoos
hun
uloce Ad serted tought the
tne
The List of claims is the left (one each line), and the book's content is right.
Your annotation activity will take place in the space provided on the left.
Figure 5: Instructions for annotation task described in $2
17

---

Published as a conference paper at COLM 2024
Clear
Search text..
Enttr
The protagonist of the story is young nomed Justine who lees an unspecifeed town.
mysef.
undress him. can't wesh him Moot of all pottops, the bad days con't keep explnining
Justine loses er gains sheomether, Mrs. Price.
get Limma ari' lis desggher and am not has wife neturned from the deadi Everyone agrees place.
Justinels life challongedlly by har suspicton towards Mrs. Price, her strupgle with opliepyy
Amy.
"The etory primerlyy taloss place 1984 when Justine a student Michselis,
"So handsome, attoough ihfs his card gan. The bundty is always mising clethee with thooe the other reaidertts, "You look like you're adyy the town, sayss "Gat good!
tips
inside cardigan-
Catholic school
Mrs. Price. known for her unconventional methods and mysterious past. Juustine and Arm'ss teacher Michael's 1984,
Lustindds futher
Passing Time Antiques, a shop frequsently visited by
Price
never cays. Maybe
Price grow closer uuatiee intiaelly eccresose wec eommetttt about Nel and Mins Phioes weddinng but inter
Justine's mother
many.
isughs. WWaughry
castno,
rights wowing
girbe You better watch out for this ons, Juzrtine
Justine suspeens Min. Price of steal ing scries of items that disappear from the school
will' say: dod"
Justine's mother hobit
only vistbie black
serving recurtin
sttry
She nods, pato my hand. He's lucky to have
The story often flashes back to Justine's childhood, providing contoot about her mather's
bake my eyee twe Surwly newe ton Sunely some part nemembers. Surie, basgge pinnedd her semntt funall tunie, One of those uniforme that's
reletionship Ary.
Justiie discovers hidden room in Mins Pricele houre filled with stalen neme from
look live normal clothea. restemment communikty pr
personal
approach, so the residents really feel they've come home. Thooe the stand-atone towrh
Justine's relationship with her tother Nell, becomes strained after he initialyy diamisses
they
her about Mine Percels thertss
bird, No replacement cats or blids can acquiied after the old onao dia, though and Dud farkinn
Amyy Justine's fraend, clee suiiicn,
inttaaly
premiumicon
Justine feds guilty about Ary a death and hounted by the idue Ainy mighttbbe
watching
Sonia.
Mr. Chusholm socal worker supports Juatine throughout the story.
Donis fanilk
simiificnn
want talk her she Mrs Price but what low
Figure 6: Screenshot of the interface for the annotation task described $2.
18

---

Published as a conference paper at COLM 2024
General comments
Submit annotations
Save draft
Restart annotations
is
early 1800s.
The novel follows the life of Deborah Samssn, young girl who becomes orphaned and is sent to live with Widow Thatcher.
Deborah Samson is later indentured to farmer until the age of eighteen.
indenture sparks rebellion within Deborah Samson become soldier and fight her freedom.
Deborah Samsoo lives with Thomas family servont.
Deborah forms close bond with Jeremiah, the youngest of the Thomas family.
[7] As the American Revolution unfolds, Deborah becomes intrigued by the idea of independence and self-governance.
Is the claim faithful to the book? Choose...
Provide quotations from the book to justify your choice. If the claim is faithful to the book, provide quotations that support the claim. Otherwise, provide quotations that contradict the claim describe the contradiction in the comment section). cases where a claim is inapplicable only partially supported, provide either quotations comments. Multiple quotations can be presented.
Quotation (copy text from the book here)
Comment (Optional)
Add evidence
Decorah forms close bonds with rellow soldiers, such as Phineas. She also deals with mutinies and conflicts within the army.
Figure 7: Pop-up window showing the interface where the annotators have to select the faithfulness label supplemented by free-form reasoning and evidence extracted from the book.
General comments
Submit annotations
Save draft
Restart annotations
After reading the summary, provide free free-form comments about omissions (e.g., important events thematic elements) that should have been included in the summary, salience issues with claims that do not feel unimportant enough be included summary, chronology errors with the order in which claims are presented (if they do not reflect the chronology of events in the book), and the overall factuality of the claims. For the latter, If you identify any factually incorrect claims, provide corrected versions of those claims here. Also please comment on any other the summary that you were able explain task.
annotation
Write your comments here
Save
Figure Pop-up window prompting the annotator to provide free-form comment on the quality of summary claims highlighting omissions, salience, chronology, and factuality issues.
19

---

Published as a conference paper at COLM 2024
Quality of Annotations We perform two additional analysis experiments that demonstrate the high quality of our dataset: (1) self-consistency of annotations (i.e., how often a single annotator assigns the same label to claims with the same semantic content generated by different models), and (2) inter-annotator agreement on a subset of claims where we had access to another annotator who also read the book.
Inter-annotator agreement: For two books in our dataset, we hired an additional annotator who had also read them to provide overlapping annotations. This re- sulted 115 claims with overlapping annotations, allowing us to evaluate the agreement rate between the original and new annotators. The new annotator is 91.30%, with Cohen's Kappa of 0.621 (p < .0001), indicating substantial agreement. Unfortunately, annotating the entire dataset with multiple annotators is unfeasible due to the difficulty and high cost of finding multiple individuals who have read the same book. Each annotation costs approximately $200 to $250 per book and requires around 10 hours of work.
Self-consistency: For each book, an annotator analyzed five summaries, each gen- erated by a different model. To assess self-consistency (intra-annotator agreement), we randomly selected five books and compared the annotations made on the first and last summaries (as per annotation order) for claims with the same semantic content. For example, "Aurora suffers emotional discomfort due to her father's 's disinterest and her parents' failed marriage" and "Aurora struggles with her father's lack of attention and affection" are semantically equivalent claims from summaries of Wildfire generated by GPT-4 and Claude3, respectively. By comparing the first and last summaries, we evaluated the annotators' consistency in handling claims after significant time intervals, during which they annotated three additional sum- maries. Consistency in labels for similar claims across these two summaries would indicate stable judgment and suggest that labels were not arbitrarily assigned. Out of 127 claims examined in the first summary, 46 had semantically equivalent claims in the last summary, and we found that all 46 of these claims were consistently labeled.
D Results of Human Evaluation
This section provides details on the number of Unfaithful and Partially Supported claims per summary. Figure 9 presents the percentage of problematic claims (either Unfaithful or Partially Supported) identified within each model's summaries. Notably, only four (4) out of 130 summaries were rated 100% Faithful (two by GPT-3.5-TURRO, one by GPT-4-TURBO, and one by MIXTRAL), The remaining summaries varied in accuracy, with some containing up to 66.67% incorrect or partially incorrect claims.
E Analysis of Faithfulness Annotations
In this section, we provide additional details on analysis of faithfulness annotations involving unfaithful claims. Refer to Table 15 for our general labeling scheme and examples for each category. Table 17 shows the reasoning type distribution for each claim type.
Evidence coverage and reasoning-claim relationship To investigate the quality of evi- dence provided by annotators, we analyze the coverage of evidence with respect to the annotators' reasoning. In addition, we also analyze the relationship between the claim and the annotators' reasoning. Results are summarized in Table 16. 51.6% of the time, annotators provide some evidence to justify every component their reasoning (i.e., complete coverage). in 56% of partial coverage (i.e., some part of the reasoning does not have corresponding evidence) cases and all cases of N/A coverage (i.e., no evidence is provided at all), the missing evidence is due to the annotator's 's inability to find any relevant information that either pairs, we
supports or refutes the claim. Qualitatively, for all matched reasoning-evidee find that the evidence often does not provide enough context that would allow someone
20

---

Published as a conference paper at COLM 2024
100
Haunting on the Hill
35.00
15.00
51.72
45.63
60.00
Agency for Scandal
18.18
3.57
12.90
10.00
0.00
Divine Rivals
26.09
4.35
8.70
9.52
52.17
Fairytale of New York
53.85
12.50
21.43
39.13
25.93
Flawless
14.81
12.50
Wing
Divination
the Week
11.54
20.00
37.50
20.00
46.43
Romantic Comedy
4.17
4.55
15.79
8.70
15.63
Same Time Next Year
0.00
8.33
55 56
35.71
60
She Haunting
18.18
19.23
27.59
22.22
29.03
Six Scorched Roses
42.86
6.67
3.45
66.67
25.00
Sorrow and Bllss
21.05
3.85
14.29
15.38
16.13
The Atonement Murders
13.64
3.23
7.69
22.22
29.63
The Guest
44.83
cansaay provlded
34.78
47. 62
57.14
The Marriage
16.00
3.57
23.08
34.48
Spy
Wager
The White
This impossible Brighmess
25.00
20.00
27.27
Viciously Yours
27.59
8.00
30.77
23.81
29.41
Weyward
35.29
4.35
30.43
9.52
30.77
Wildfiee
0.00
9.52
16.13
3.70
11.11
yellowface
39.13
15.00
26.92
33.33
25.09
You, Again
26.09
15.00
4.35
9.52
19.23
-
GPT-3. G.T-3.5-Turba Turba
Claude-3-oge 3-pus
GPT.4
GPT-4-uubo Turbo
Mixtral
Figure 9: Percentage of claims rated Unfailifull or Partially Supported across models, analyzed by book. Only four (4) out of 130 summaries were 100% Faithful. In two cases, CLAUDE- 3-OPUS declined to merge two summaries due to significant content discrepancies ("Same Time Next Year" and "The Guest").
21

---

Published as a conference paper at COLM 2024
who has not read the book to determine the faithfulness of the claim. As result of decon- textualization, claims always refer to people by name, but evidence often use pronouns instead. The annotator would need to quote much larger chunk from the book in order for the evidence to include names as well. An even trickier case is that when dealing with high-level claims like "X is the protagonist of the story" or "The themes of the story X, Y, and Z, one needs knowledge of the entire book, but citing the entire book as evidence is trivial. If annotators were collect self-contained and sufficient evidence for every claim, the task would become significantly more challenging, sometimes even impossible. This difficulty with evidence gathering sheds light on why automatic evaluation does not work so well for this task.
Model-wise analysis We report model-wise results reasoning type and reasoning-claim
relationship in Table 18 and Table 19.
Comment Analysis
In this section, we provide additional details regarding our analysis of the comments pro- vided by annotators on the summary claims. Table 20 features examples of such comments. These comments were further annotated based the criteria outlined in Table 21 and Table 22. The distribution of errors depicted in Figure 10 and Table 23.
Table 24 displays examples where the models' generation was influenced by information in the front and back matter. Table 25 highlights comments indicating that models may sometimes overly focus on the latter parts of the stories. Lastly, Table 4 shares annotators' feedback on the annotation task.
Impact of front and back matter on the summary quality Books frequently contain additional information beyond the main narrative, including the author's biography, table of contents, dedications, and more, positioned at the beginning the end of the book. Ideally, models should exclude this extraneous content, focusing solely on summarizing the core story. However, have noted that models sometimes unduly influenced by these elements, which can dominate a significant part of the summary and occasionally compromise its accuracy. Overall, between 19.23% (GPT-3.5-TURBO and GPT-4) and 34.62% (MIXTRAL) of summaries were affected by such content, either through focusing on this information," 15 confusing story characters with names found in the front and/ or back matter, 16 making up entire narrative based on single mention." 17 CLAUDE-3-OPUS was the only model seemingly unaffected by the additional information. However, when faced with two mummaries-whhee since it represented the final chunk chunk-the the model declined to perform the task. We regard this cautious approach preferable to introducing unfounded details or irrelevant content. Examples of such cases are shown in Table 24.
primarily summarized the content of the back matter,
G Details on Experimental Setup
In this section, we provide further details on our experimental setup complemented with further results.
15 This summary includes a description of who the author thanks the end of the book which is not important to the plot of the book." "lCCaiir is not character in this book. The comments are factual, but of Charlie not Clair."
17 "l..ciamms are very focused on the idea of themes of digital age and the story doesn't 't cover that at all. Its not even based on a modern world." author's s social media accounts are mentioned at the very end of the book.
22

---

Published as a conference paper at COLM 2024
Claude Opus
GPT-3. GPT-3.5-Turb 5-Rurbo
GPT-4- A-uubo
GPT-4
Mixtral
Chronology
33.33
36.00
46.15
50.00
61.54
80
Omissions
70
60
Factuality
58.33
69.23 23
80.77
69.23
84.62
50
Overemphasis
40
Underemphasis
12.50
23.08
19.23
38.46
34. 34.62
30
VaguerGeneric
0.00
23.08
3.85
38.46
38
20
Repetitive
0.00
11.54
0.00
7.69
3.85
10
Data Influenced
0.00
23.08
19.23
19.23
34.62
Figure 10: Percentage summaries affected by specific issue mentioned in comments by model.
23

---

Published as a conference paper at COLM 2024
G.1 Implementation details
For BM25-based evidence retrieval, we use the text of e e-books purchased from amazon. com, split into passages of up to 256 tokens each. The search is restricted to the book content, and we set k = 5 to retrieve the top 5 most relevant passages as evidence.
G.2 Additional Results
Results for each evidence extraction method broken down by summarizer can be found in Table 26. We also report book-wise precision and recall for each evidence extraction method: (1) No-Context (Table 27); (2) BM25 (Table 29); (3) Human evidence (Table 28); (4) Entire book (Table 30). Further results for the entire book (EB) prompting can be found in SG.3.
G.3 Prompting LLMs with the Entire Book (EB)
Prompting LLMs with large chunks (entire books) to evaluate the faithfulness each claim is prohibitively expensive (see 5G.5). Hence, for this experiment, we select 7 books based on: (1) token length (<125K), and (2) presence of at least one Unfaithful claim. This sub-dataset
includes: (1) "Yellowface, (2) "Only For The Week," (3) "Viciously Yours," (4) "Six Scorched
Roses," " (5) "Sorrow and Bliss," " (6) "She Is Haunting," " and (7) "Pet." " Table Table 31 shows the number of claims per label in the sub-dataset. Further details on each book can be found in Table 7.
Claim verification with the entire books We prompt CLAUDE-3-OPUT and GPT-4-TURBO models with the entire book content and each claim in order to obtain the Faithful/ Unfaaihfuul labels.
Table 32 presents confusion matrix broken down by claim source (i.e., the model that generated the claim) and prediction model (CI.AuDE-3-OLUS and GPT-4-TURBO). Figure 11 shows examples of misidentified labels by label-type and prediction model along with human labels and reasoning. Table 30 shows average precision (PR) and recall (RE) broken down by model and book.
G.4 Ablation study
Recall of the claim decomposition step we analyze the extracted claims on subset of 20 summaries (371 sentences, 450 total extracted claims). We manually evaluate the quality of the extracted claims against the content of each summary. Calculating recall proved challenging due to the ambiguity in granularity (e.g,, sentences, clauses, words). Notably, 3.8% of the 371 sentences in the 20 summaries were omitted in the extracted claims. Of these omissions, 85.7% were generic statements, and 14.3% were minor details. Additionally, we observed small percentage of omissions at the sub-sentential level (e-g., clauses), which did not impact the narrative. All These omissions can be broadly categorized into two types.
Generic statements lacking substantive content: For instance, "The narrative unfolds with intrigue, danger, and treacherous encounters" appears in the summary but is omitted in extracted claims. Note that this sentence only addresses things already covered by other extracted claims in a generic way, so omitting has few consequences.
Insignificant details that contribute little to the narrative: For instance, "Altha, 17-century woman, stands trial unjustly accused of witchcraft due to her remarkable healing abilities which are misunderstood by her village" appears in the summary, but "misunderstood by her village" is omitted the extracted claims. However, this is only minor detail with little impact on the narrative.
Importantly, confirmed that none of these discrepancies between the summaries and the extracted claims led to criticisms regarding omissions, chronological errors, or factual inaccuracies in the annotators' summary-level free-form comments.
24

---

Published as a conference paper at COLM 2024
Amelia carries the hidden burden of her identity throughout her life.
The story emphasizes the complex role of social media in creating meaningful connections and facilitating change.
FALSE: Amelia has no her identity uniil tells her and even then she doesn't fuily believe it. Amelia believes she Inman.
FALSE:
This set of summary claims are very focused on the idea of thomes of digiul age and the story doesn't bazed
az
world.
A
TRUE
TRUE
A
TRUE
TRUE
Lilith's struggles with her vampire nature and revelations about her family's past are mirrored in the personal crises and realizations
The relationship between Neil
?
Justine strained due to the emotional struggles Neil experiencing.
medern characters.
Justine's faiher having emotionnl struggles but he doasn't blame Justiee he blames kimself for not sensing what war going on, Justine's father axcures her that he is not ashamed of her, and shat Mry Angela Price should be ashameol.
FALSE:
FALSE: She doesn't struggle per se and nothing show mirrored personal crisis and modern characters realisation.
A
FALSE
TRUE
A
TRUE
FALSE
Justine successfully navigates her past experiences and gains a deep understanding of herself and her place in the world.
Fae traditions and politicking cause tensions in Reanick and Amelin's relationship.
TRUE:
Jussine is having difficully coping with the loss of dearest people
Beconse Amelia is "human" there lut of Beconce she has easily killed. No one has married human before. The politics of royal faee marriage combined with Amelia being "human" make it Rennick camnot tell anywho that Amelia his due to safety reasons, This is mojor tsaue through book.
TRUE:
successful life and she greatly helped in duing so by her hurbhand Dom. Justine vistiing father hume becouse of his demoniia, and she starts romembering the events from 1984, events shaped her personality
monages
uncertainties comes morrying royal
she has royals marry myals.
on her back
retirement
A
FALSE
FALSE
A
FALSE
FALSE
Martha's sister, Ingrid, is dealing with post-partum issues.
Martha has a fear of becoming
mother.
TRUE:
Ingrid struggingg after giving burth: "She did then, telling me shrough teurs that If' understod what was being woken up baby every two hours when it takes an hour and filty-mine minutes
Martha talking Robert, her psychiatrist: "I said don't i want baby. I would be a bad mother.". ".
TRUE:
might and
to feod
nipple with four lundred knives, then I would be like, do you know what? I think will just get my sister the liquorice specificulll liker."
like being stubled
A\
FALSE
TRUE
AI TRUE
FALSE
Figure 11: Examples of claims accompanied by annotator labels and reasoning, along with predictions made by CLAUDI-3-OPUS and GPT-4-TURHO.
25

---

Published as a conference paper at COLM 2024
Varying length of tokens used in BM25 As we increase the length of B.M25-rtrrieved pas- sages, the overall performance improves (Figure 12). However, this approach remains less effective for identifying unfaithful claims than our best performing method, i.e, prompting the model with the content of the entire book. This is likely due to the fact that even longer passages may not provide the entire context needed for verification of broader claims.
Reasoning type of false positive cases We analyzed failure cases in which our auto-rater experiment, conducted on seven books using CLAUDE-3-OPUS and GPT-4-TURBO incorrectly marked Unfaithful claim Faithful. We annotated the types of reasoning required to verify these claims, as presented in Table 33. The results indicate that approximately 75% of these failure cases necessitate multi-hop reasoning across the book. This is significantly higher than the overall distribution of 62.8% across the seven books, suggesting that our auto-raters struggle with multi-hop reasoning.
0.8
Faithful Unfaithful
0.7
0.6
F1 Score
0.4
0.3
0.2
50
100
150
200
250
Size of Chunk
Figure 12: score varying chunk size for BM25.
G.5
API costs
Generating book-length summaries amounted about $288 USD.
The total cost of summarization for all 130 summaries
Extracting claims The total cost of claim extraction for all 130 summaries amounted to about $8 USD, as the input and output sequence is relatively short.
Prompting with the entire book This experiment costed roughly $720 USD for GPT- 4-TURBO and $1070 USD for CLAUDE-3-OrUS (corresponding to the last two columns in Table 5).
18$64.6 for CLAUDI-3-OPUS, $169.4 for GPT-4, $47.5 for GPT-4-TURRO, $2.8 for GPT-3.5-TURRE and $3.4 for MIXTRAL. All in USD.
26

---

Published as a conference paper at COLM 2024
GPT-4-TURBO
In the epic narrative of "Divine Rivals," " we are introduced to Iris Wimnono tenacious young woman propelled into the role of war correspondent amidst a devastating conflict influenced by the ancient deities Enoa and Dacre. Iris's journey into the heart of battle is not just professional but deeply personal, fueled fueleed the disappearance of her beloved brother, Forest, who had enlisted to fight, and the anguish over her mother's 's demise. Her quest for truth and understanding is set against the backdrop of the war-forn Avalon Bluff, close to the war's s front lines, where she encounters the stark realities of conftict, from the silent hope of garden amidst destruction the horror of wounded soldiers and the terror of artillery fire. Roman Kitt, previously Iris's 's rival at the Oath Gazette, emerges as a significant figure in her narrative, joining her as a fellow correspondent. Despite their kistory of competitive antagonism, their relationship evolues, highlighted by Roman's 's mysterious decision to leave behind a promising career and a broken engagement to be by Iris's 's side, Their dynamic is further complicated and enriched by their shared experiences on the battlefield and Roman's unspokent feelings for Iris, culminating in a poignant moment of sacrifice during trench attack where Roman shields Iris with his body from a grenade explosion. Adding a magical element to the story, fris maintains a correspondence with Carver, mysterious figure, through enchanled typewriters. These letters offer Iris solace and metaphysical connection that transcends the chaos surrounding her, underscoring themes of hope and connection.. As the narrative unfolds, shocking revelations come to light. Forest, Iris's 's brother, believed to have been fighting for Enoa, is discovered to have been forcefully conscripted by Dacre after near-death experience. This discovery unfolds through a dramatic reurtion where Forest saves Iris from one of Dacre's deadly gas attacks, only to reveal his changed allegiance by deceiving her and leaving Roman behind, presumed dead captured. The heart of the story explores Iris and Roman's profound relationship, against the backdrop of war and divine manipulation. Moments from their past, including their secret wedding in a garden before devastating attack by Dacre's forces, and Roman's heartfelt letter confessing his love, serve as poignant reminders of what they fight for. This emotional core is juxlaposed with the chaos of the present, wherein Iris must confront the betrayal of her brother and the uncertainty of Roman's 's fate. "Dioine Rivals" " delves into themes of love, loyalty, and the harrouing choices faced by individuals eaught in the machinations of gods and war. Iris, within the turmoil of her personal and professional world, remains defermined to uncover the fruth, seeking to reclaim agency and hope amidst despair. Her journey is not just a search for her brother and Roman but an exploration of the resilience of the human spirit in the face of divine conflicts and the devastation of war. Through Iris's 's eyes, the story weaves a complex tapestry of human emotions, the indomitable will to survive, and the quest for truth in a reality marred by capricious gods and endless conflict.
Summary
1. Iris Winnow is the main character in the book "Divine Rivals."
2. Iris Winnow becomes a war correspondent during a conflict influenced by the deities Enva and Dacre.
3. Iris's brother, Forest, has disappeared after enlisting to fight the war. 4. Iris is devastated by her mother's 's death.
5. Iris's journey takes her to Avalon Bluff, a location close to the war's front lines.
6. Iris encounters the realities of war, including wounded soldiers and artillery fire.
7. Roman Kitt is a significant character in the narrative and joins Iris as a fellow war correspon- dent.
Extracted Claims
8. Roman Kitt was Iris's rival at the Oath Gazette before they became colleagues.
9. Their relationship evolves, despite history of competitive antagonism.
10. Roman Kitt leaves promising career and broken engagement to join Iris.
11. Roman Kitt shields Iris from grenade explosion during trench attack.
12. Iris maintains magical correspondence with mysterious figure named Carver through
enchanted typewriters.
13. The letters from Carver offer Iris solace and metaphysical connection.
14. Forest was believed to be fighting for deity Enva, but is revealed to have been forcefully
conscripted by deity Dacre
15. This revelation comes after Forest saves Iris from a gas attack by Dacre and then deceives her by leaving Roman behind.
16. Iris and Roman had a secret wedding in a garden before an attack by Dacre's 's forces.
17. Roman writes a letter confessing his love for Iris.
18. Iris must confront her brother's 's betrayal and uncertainty over Roman's 's fate.
19. Iris is determined to uncover the truth and reclaim agency and hope amidst despair.
20. Iris's 's journey is search for her brother and Roman, and an exploration of human resilience
in the face of conflict and divine manipulation.
21. The story of "Divine Rivals" explores themes of love, loyalty, and the difficult choices faced by
individuals in times of war.
Table 9: Example of a summary produced by GPT-4-TURBO along with the extracted set of claims for "Divine Rivals," " a novel by Rebecca Ross. Examples by the other models can be found in Table 8, Table 10, Table 11 and Table 12.
27

---

Published as a conference paper at COLM 2024
GPT-4
Divine Riculs is an intricute narratioe set in a dystopian world at war, invoked by two resarrected rioal deilies, Euza and Dacre. The plot centres around Wannow, a sensitive and tenacious employee the Oath Gazette newspaper in the city of Outh, and her professional rival Roman Kitt. Tris's uneventful life- spirass into chaos with her brother, Forest, joining the gods' war, leaving her with the responsibility of their alcoholic mother, Aster. Caught in a whirlpool of personal and professional rivatry, Iris's is bond with Roman deepens unexpected ways when Roman anorymousty receives and responds to Iris's emotionally now letters intended for Forest. Iris uses nagicaal vintage typewriter to send these letters, leading to an anonynous correspondence that amplifies their unspoken connection. Roman's ''s social standing is stark contrast to Iris's modest background; he affianced to a high society woman, Elinor Little, and faces the challenges of striking balauce between his family's i aspirations ions and his new role Oath Gazette. Following their mother's 's sudden passing, Iris impulsively resigns from her job at Oath Cazette. Determined to locate Forest and report ou the escalating gods' war, she accepts a position as a war correspondent with Inkridden Tribune necospaprr Iris embarks on a perilous / journey to Avalon Bluff, town near the war front, accompanied by another correspondent, Thea "Attie" Aftwood. Their hostess in the cerie town, Marisol Torres, warns the puir about the regulor threats from different mythical creatures such as hounds and cithrals that desolate their surroundings Roman's arried in Avalon Bluff reignites their convoluted relationship. He assists Iris documenting the harrouingg realities of the usar while managing the growing tension beteven them, As Roman becomes severvly injured during horrific encounter the fightts fights her way to ensure hiss sufity. This life- thradening cvent leads Tris to raugrise Roman as Caroer, her mysterimms pen pal, triggering au intense turmoil of guilt and suspicion over Roman's 's subterfuuee Amid her growing concern for Roman's 's fitt and her conflicted feelings about his deception, shocking swist reveals that Forest, previously presumed dend, has survided and fought for Dacre, the enemy deity. Further ensuing chaos sees Roman disappear, leaving Iris trapped between her obligations lowards her effectively traitorous brother and the desperate search for her missing love interest. "Divine Rivals" interwenves tris's personal story of resilience and growth against the backdrop of an escalating war. As she metanorphoses from correspondent to a key player in the war, Iris embodies the human spirit's it' ability to endure and adapt amidst adversity. The narrative encapsulates the strength of character, familial obligations, professional riculries, societal nurms, and profound transformations at the heart of the story. relationships and hardships, Iris pledges to remain by Forest's 's side and continue her search for Roman. The story of "Divine Rivais" is testament to human endurance, love, and loyalty amidst a surreal world at war.
Summary
warfront, Iris, who has grown deeply involoed in reporting the soldiers' personal narratives,
the cooloing
1. Divine Rivals a narrative set in dystopian world at war.
2. The war in the world of "Divine Rivals" is invoked by two resurrected rival deities, Enva and Dacre.
3. The story's plot focuses on Iris Winnow, an employee at the Oath Gazette newspaper in the city Oath.
Iris Winnow has professional rival named Roman Kitt.
5. Iris Winnow's 's brother, Forest, joins the god's 's war.
6. Iris Winnow is left with the responsibility of their alcoholic mother, Aster, after 7, Forest joins the war.
Extracted Claims
8. Iris Winnow's bond with Roman Kitt deepens when Roman anonymously receives and responds to Iris's 's letters intended for Forest.
9. Tris Winnow uses magical vintage typewriter to send letters to Roman Kitt.
10. Roman Kitt engaged to high society woman named Elinor Little.
11. Roman Kitt faces challenges balancing his family's 's aspirations and his role at the Oath Gazette.
12. Iris Winnow resigns from her job at the Oath Gazette after their mother's sudden death.
13. Iris Winnow accepts a position war correspondent with the Inkridden Tribune newepaper.
14. Iris Winnow journeys to Avalon Bluff, town near the war front, with another correspondent named Thea "Attie" Altwood.
15. In Avalon Blutf, Marisol Torres warns Iris Winnow and Thea Attwood about the regular
threats from mythical creatures.
16. Roman Kitt arrives in Avalon Bluff and assists Iris Winnow in documenting the war.
17. Roman Kitt is severely injured during an encounter at the warfront.
18. Iris Winnow realizes Koman Kitt is Carver, her mysterious pen pal, after he is severely injured.
19. Iris Winnow feels guilt and suspicion over Roman Kitt's deception.
20. It revealed that Forest, previously assumed dead, survived and fought for Dacre, the enemy deity.
21. Roman Kitt disappears, leaving Tris Winnow torn between her obligations towards her brother and the search for Roman.
22. Iris Winnow evolves from a correspondent to a key player in the war.
23. Tris Winnow pledges to stay by her brother's 's side and continue her search for Roman Kitt.
24. The story of "Divine Rivals" is a testament human endurance, love, and loyalty during a war.
Table 10: Example of a summary produced by GPT-4 along with the extracted set of claims for "Divine Rivals," " novel by Rebecca Ross. Examples by the other models can be found in Table 8, Table 9, Table 11 and Table 12.
28

---

Published as a conference paper at COLM 2024
GPT-3.5-TURBO
In the city of Oath, siblings Iris and Forest Winoww face separation as Forest departs for war, leaving Iris to care for their mother. As Iris grapples with loneliness and worry, a fierce riculry emerges at the Oath Gazette between Iris and Roman Kitt, two ambitious journalists. The backdrop of an impending war between the goddess Evva and god Dacre adds tension and intrigue to their lives. Iris, driven by desire prove herself in a male-dominated field, finds finds herself entangled in personal challenges while navigating her complicated relationship with Roman. As the war escalates and mysterious creatures surface, Iris and Roman become immersed in a world of gods, magic, and journalism, facing both personal and professional dilemmas. Iris decides to resign from her job to embark on a journey to find her brother Forest, who becomes entwined in the conflict. Roman, haunted by his past and familial obligations, struggles to reconcile his feelings for Iris while maintaining his journalistic integrity. Their paths eventually converge in Avalon Bluff. where tris becomes a war correspondent equipped with a mysterious typewriter with a sigrificant history. As Iris and Roman navigate the chaos of war, their inferactions revval doeper emotions and motivations, leading to a fateful meeting at high-end restaurant. Attie, a friend of Iris, discloses her talent for playing the violin amidst the turmoil of war, adding a layer of depth to the narratice. Roman's 's vulnerability and imner turmoil are brought to the forefront when he is saved by Iris during an air raid, further further strengthening their bond. As they confront personal and professional challenges amidst the impending battle, Iris reumites with Forest, only to disconer his involvement with the opposing forces. Forest, torn between protecting Iris and confronting his past actions, adds layer of complexity and tension to the story. Amidst the impending attack by Dacre, fris and Roman make a decision fo stay and assist in Avalon Bluff, preparing for the inevitable confrontation. The intertwining destinies of the characters unravel themes of sacrifice, resilience, and the search for identity a world plagued by war and loss. As truths unfold, loyalty tested, blurring the lines between good and evil. Characters like Dacre and Enoa reveal sinister plans, setting the stage for complex and emotional journey fraught with uncertainty and conflict. Through grief, rimalry, and love, tris and Roman confront their vulnerabilities and ambitions, ultimately reshaping their destinies amidst the backárop of war and its significant consequences.
Summary
1. Iris and Forest Winnow are siblings in the city of Oath.
2. Forest Winnow departs for war and Iris stays behind to care for their mother.
3. Iris struggles with feelings of loneliness and worry.
4. At the Oath Gazette, a fierce rivalry develops between Iris and Roman Kitt, both of whom are ambitious jourmalists.
5. An impending war between the goddess Enva and god Dacre adds tension and intrigue to Iris and Roman's 's lives.
6. Iris aims to prove herself in the male-dominated field of journalism.
7. Iris faces personal challenges while navigating her complicated relationship with Roman.
Extracted Claims
8. As the war intensifies, Iris and Roman encounter mysterious creatures.
9. Iris resigns from her job at the Oath Gazette to search for her brother Forest, who is involved
in the war.
10. Roman Kitt has a troubled past and family obligations that make it difficult for him to balance his feelings for Iris and his journalistic integrity.
11. Iris and Roman's 's paths intersect in Avalon Bluff, where Iris becomes war correspondent.
12. Iris uses typewriter with significant past while working as war correspondent.
13. Iris and Roman's interactions during the war reveal deeper emotions and motivations.
14. Iris and Roman have a significant meeting a high-end restaurant.
15. Attie, friend of Iris, talented at playing the violin.
16. Roman's vulnerability is revealed when Iris saves him during an air raid.
17. Iris reunites with Forest during the war, discovering his involvement with the opposing forces.
18. Forest torn between his desire to protect Iris and confronting his past actions.
19. Amidst the approaching attack by Dacre, Iris and Roman decide to stay and help in Avalon Bluff
20. The characters' intertwining destinies reveal themes of sacrifice, resilience, and identity search
in a war-torn world.
21. The story explores blurred lines between good and evil as loyalties are tested.
22. Characters like Dacre and Enva have sinister plans.
23. Iris and Roman confront their vulnerabilities and ambitions, leading to change in their destinies amid the backdrop of war.
Table 11: Example of summary produced by GPT-3.5-TURBO along with the extracted set of claims for "Divine Rivals," a novel by Rebecca Ross. Examples by the other models can be found in Table 8, Table 9, Table 10 and Table 12.
29

---

Published as a conference paper at COLM 2024
MIXTRAL
In a world torn by a deonstating war between two divine families the Skyasarss and the Underlings, "Divine Riculs" " follows the journey of Iris Wannon; au orphaned junior reporter the Oath Gazette in the city of Dath, as she strives to become renoumed journalist and uncover the truth about the war thut took her mother and brother, Forest, from her. The Skyasards and Uinderlins' conflic has left the realm of Canbria in chaos, where magic aud mythology intertwine with the mortal world, and gods walk among humans. Ivis' dream of becoming a great journalist is fueled by her love for her late brotker and her desire to understand why he joined the war. She faces rizal, Roman Kitt, talented storyteller hailing from wealthy family. While tris longs to write about significant issues, Roman, who shares a complicated background with Tris, excels in storytelling and is being pressured by his father to join the family business. Zeb Autry, the editor of the Oath Gazette, is reluctant to cover the war, much to tris' dismay. tris' family burdened with financial difficulties and personal struggles after losing her mother and brother the war. Driven by her determination to uncover the truth about the way and her late brother's irvolvement, iris finds solace in her resolve. The Inkridden Tribune, rival neuspaper, announces an opportunityy for war correspondents, enticing Iris to compete with Roman for the position. Iris's s journey is farther complicated by her mysterious correspondence with an unknown person who leaves her letters her wardrobe, forming a unique bond. As lris competes for the position, she reflects on her past and her friendship with Forest, who introduced her to the world of journalism. Their differing views on the war and the Skyword goddess, Erva, drove them apart. Iris wanted to highlight the braverr of soldiers, while Forest aimed to reveal Enes's 's munignilation of young minds to fuel the wsr. Despite the challenges, Iis remains focused on her goul and finally manages to land job war correspondent for the Inkridden Tribune. She emburks to cover the war, war,,aciin facing various dangers and hardships along the way. Iris continues to write letters to the mysterious strunger, who becomnes her confidannte and support system during her perilous journey. Meamollile, the town of Aoulon Bluff prepares for au impending battle against au unknown enemy- Dacce, the god the Underlings. Tris, Roman, and other companiios gather in Avalon to aid the tount's i defense, tensimm rise with the theft of energencc supply kits and Keegan's is distracted behavior. Amidat the chaos, Iris and Roman rekindle their counection, and Romau proposes to iris, offering a moment of hope in the face of devastation. As the group prepares for battle, Tris witnesses firsthand the horrors of war, including the loss innocent lives and the destruction of her childhood home, Clover Hill. Despite the pain, Iris continues to report on the war, using her words to botk document the bottles and search for Forest. "Divine Rivals" is a compelling narrative that explores themes of identity, courage, and the power of words in the face of war. Iris' smaonvering determination to make a difference by writing about the trath, seeking answers about her brother, and making her mark as a journalist resomates throughout the story. The author extends gratitude towards the indrviduals who helped her publish this captivating tale, acknowledging her literary agent, film agent, subrights team, editors, cover designer, copy editor, and fellow authors.
competition from her professional
Summary
dangerous journey
1. Divine Rivals is set in a world affected by war between two divine families, the Skywards and the Underlings. The protagonist of "Divine Rivals" is Iris Winnow, an orphan who works as junior reporter at the Oath Gazette in the city of Oath.
3. Iris Winnow's 's aspiration to become a renowned journalist is driven by her desire to uncover the truth about the war that resulled in the death of her mother and brother, Forest.
4. The conflict between the Skywards and the Underlings has caused chaos in the realm of Cambria, a place where magic and mythology interact with the mortal world.
5. Iris Winnow has professional rival named Roman Kitt, skilled storyteller from a wealthy family.
6. Roman Kitt is under pressure from his father to join the family business.
7. Zeb Autry, the editor of the Oath Gazette, does not want to cover the war, against Iris Winnow's 's wishes.
8. After the loss of her mother and brother, Iris Winnow's 's family is facing financial and personal struggles.
9. The Inkridden Tribune, competing newspaper, advertises a job for war correspondents, leading both Iris
Winnow and Roman Kitt to compete for the position.
10. Iris Winnow maintains mysterious correspondence with an unidentified individual who leaves her letters
Extracted Claims
in her wardrobe.
11. Iris Winnow's 's brother, Forest, introduced her to journalism.
12. Iris Winnow and her brother Forest had differing views on the war and the Skyward goddess, Enva.
13. Despite various challenges, Tris Winnow achieves her goal of becoming a war correspondent for the Inkridden Tribune.
14. Tris Winnow embarks on dangerous journey to cover the war.
15. Iris Winnow continues to write letters to the unknown person who becomes her confidante during her journey.
16. The town of Avalon Bluff is preparing for battle against unknown enemy.
17. Dacre, the god of the Underlings.
18. Amidst the chaos, Iris Winnow and Roman Kitt rekindle their connection, and Roman Kitt proposes to Iris Winnow.
19. Iris Winnow experiences the horrors of war firsthand, including the loss of innocent lives and the destruction of her childhood home, Clover Hill.
20. Despite the hardships, Iris Winnow continues to report on the war, using her writing document the battles and search for her brother, Forest.
21. Divine Rivals explores themes of identity, courage, and the power of words during war.
22. Iris Winnow's 's determination to make a difference by writing about the truth and seeking answens about her
brother is central theme in "Divine Rivals".
23. The author of "Divine Rivals" ac acknowledges her literary agent, film agent, subrights team, editors, cover designer, copy editur, and fellow authors for their assistance in publishing the book.
Table 12:
Example of a summary produced by MIXTRAL along with the extracted set of claims for "Divine Rivals," a novel by Rebecca Ross. Examples by the other models can be found in Table 8, Table 9, Table 10 and Table 11.
30

---

Published as a conference paper at COLM 2024
Claim Extraction Template
You are trying to verify the faithfulness of statements made in given summary of book against the actual text book. To do so, you first need to break the summary into a set of "atomic claims", each of which will then be passed to a human who will read the book and verify if the claim is true not. Each atomic claim must be fully understandable without any other context from the sunmary (e.g., all entities must be referred to by name, not pronoun), and they must be situated within relevant temporal, location, and causal context whenever possible. Try to keep each atomic claim maximum sentences. Each atomic claim separated with
Sunmary:
List of atomic claims:
Evaluation Template
You are provided with a context and a statement. Your task carefully read context and then determine whether the statement is true or false. Use the information given in the context to make your decision.
Context:
Statement:
Question: Based the context provided, is the above statement True or False?
Answer:
Table 13: Prompt templates used for CLAIM EXTRACTION and EVALUATION EXTRACTION.
Model
Faithful
Unfaithful
Partial Support
Can't verify
GPT-3.5-TURBO
432
68
79
25
GPT-4
31
108
MIXTRAL
491
83
122
19
GPT-4-TURRO
437
43
68
15
CLAUDE-3-OPUS
528
23
40
3
Table 14: Distribution of collected labels by model.
31

---

Published as a conference paper at COLM 2024
LABEL
DERINITION
EXAMPLE (CLAIM // REASON)
Claim type
Event
Concrete event where someone does something, something hap- pens to someone, etc.
Maggie reunites with her old friends and fellow retired spies. // Maggie does not reunite with these people.
Introspection Characters' thoughts, opin- ions, etc.
Justine feels guilty about Amy's 'sss death and is haunted by the idea that Amy might be watching her. If Justine doesn''t feel guilty.
Goals, motivation, purposes
Charlie Brown decides to return to New York to con- front Harry Taylor and pursue connection with Pete Makris after discovering Harry's infidelity. He is not there confront Harry.
eafee/
Causes or effects of events, actions, thoughts, etc.
The discovery of the love story sparks Jade's curiosity about the house and its past inhabitants. I/ Jade's curiosity is not sparked by the love story, but by dream she had.
Relationship between characters
Maggie reumites with her old friends and fellow retired spies. // Maggie does not reunite with these people.
State
Traits of character
The magic of royal fae in "Viciously Yours" manifests after twenty free years. It does not manifest after 25 years but becomes full strength at 25 years. They born with magic.
State of character, place, etc.
Phillip Hardwicke, wealthy businessman who was believed be dead, revealed to be alive the story. // Bella Hardwicke is revealed to be alive, not Phillip.
Characteristics of the narrative
The narrative style of the book is non-linear and features flashbacks and switches between alternate worlds or viewooonts. // The book is almost exclu- sively from Aurelia's s point of view and is linear.
High-level
General story selting
The narrative style of the book is non-linear and features flashbacks and switches between allernate worlds or viewpoints. // It's set in Adcova, Nyaxia is the name of the goddess.
Themes
The narrative of "The Guest" " explores themes of mem- ory, identity, and the pursuit of understanding within human relationships. If It's set in Adcova, Nyaxia is the name of the goddess.
Reasoning type
Direct
Reasoning requires only one hop
Alex attends guthering at Victor's 's house. I/ The book states that the gathering is in Helen's 's house.
Reasoning requires more than one
Alex and Jack bond over their shared experiences. II They don't have any shared experiences, Jack is from wealthy, privileged home, and while we aren't I told much about Alex's 's background, we know she doesn't live a cosseted life like him.
Indirect
hop
Annotator is arguing for a lack of support
Maggie is portrayed as skilled assassin in addition to being a former intelligence officer. // No information in the book really supports that.
Subjective
Requires subjective judgment
Forest is torn between his desire to protect Iris and confronting his past actions. // don't 'tt think Forest makes amy real effort to confront his past actions, his main motivatiinn is protecting tris.
Extra info
Requires extra/ extra/meta/io /eta information
The book "Wildfire" is the first in the Icebreaker series. I/ No evidence in the book, but this is the second in the series, after "Ioebreaker".
Table 15: General scheme for assigning labels in our faithfulness annotation analysis along with more examples. This table complements Table 3.
32

---

Published as a conference paper at COLM 2024
Evidence Coverage
Reasoning-Claim Rel.
Type
Freq
Type
Freq
Complete
56.1
Direct cont.
50.5
Partial
34.7
Indirect cont.
30.1
Irrel.
1.5
Lack of support
19.4
N/A
7.7
Table 16: Results from our analysis on evidence coverage and reasoning-clain relationship.
Claim type
Total
Direct evidence
Logical inference
Subjective interprelation
Requires info
Event
Thought
42.3
7.7
Cause/effect
2.8
State
5.5
6.3
High-level
16.7
11.1
Table 17: Distribution of reasoning type for each claim type. Apart from total count, all numbers are reported as percentage.
Model
Total count
Direct evidence
Logical inference
Subjective interpretation
Requires mela info
AUDE-3-OPUS
12
66.7
8.3
GPT-4
26
12
GPT-4-TURBO
27.5
62.7
GPT-3.5-TURRO
63
31.2
51.6
10.9
6.2
MIXTRAL
76
38.5
48.7
Table 18: Distribution of reasoning type for different models. Apart from total count, all numbers are reported percentage.
Model
Total count
Direct contradiction
Indirect contradiction
Lack of support
CLAUDE-3-OPUS
12
66.7
33.3
0
GPT-4
26
48.3
27.6
24.1
GPT-4-TURBO
48
24.5
34.7
40.8
GPT-3.5-TURB0
63
44.1
30.9
25
MIXTRAL
76
57.7
19.2
23.1
Table 19: Distribution of reasoning-clain relationship for different models. Apart from total count, all numbers are reported as percentage.
33

---

Published as a conference paper at COLM 2024
Model Leaning Positive
Leaning Negative
This is an excellently structured summary. It commumicatts the main plot of the book in a succinct, organised nanner, fouching on all the pivotal moments in a logical sequence. There is bulance between story and commentary.
Amelia's s last name isn't I very relevant to the summary. She is referved to by it once in the book. Taking out her last name would be more effective. Rennick's s mother's 's death is also omitted which is key point in the story. The other missing key point is that Amelia is believed to be a human. That has major implications and sets the stage for the events in the story. Overall this summary compared the first was much better summary of the story because it contains key points and not gentralizations.
GPT-4-TURBO CLAUDE-3-OPUS
Overall, I think this is a good summary. It hils the major points of the book and the three stories are each in order. However, the book jumps back and forth between the three stories and this summaary goes through each one separately. It also leaves out Violet drinking a tea that causes abortion which is an important part of her character's. s.
This is not a strong summary of the book. The entire section about visiting and living with Jerry is missing. There is no mention of the paparazzi pics being leaked. This summary seems to focus more on the message of the book rather than the plot. Many of the claims seems to be paraphrased versions of each other and keep repeating the same ideas of this book being about overcoming challenges, facing insecurities etc.
This summary is written in chronological order, it accuratefy categorizes the excerpt of Wreck the Halls and the claims mentioned here are either True or Partially True. It is easy to follow and is not repetitive. This is one of the stronger summaries. The claims were easy to verify and with the partially true claims- there were only minor issues. However, all the summaries including this one the importance of Sumner's 'ss family. It touches on view of his parents as a couple and his relationship with his grandmother. (...)
The majority of these claims were true, but the overall summary does poor job of following the plot. It skips over many important plot points and does poor job of describing the main characters. Similar to another summaries, this one makes lot of broad thematic claims but misses the overall storyline. If hadn't read the book, I would not understand this summary.
GPT-4
of the
aspect that is ignored
This is a good summary with most of the plot points covered. The chronological sequence is largely appropriate. Though would move the claim about Sally and Noah's 's marriage closer to the conclusion as if happens at the end of the book. In one claim, Jerry is listed as uncle, though he is her stepfather. think there should also a claim that Sally goes to live Noah's place during the pandemic, as they get closer from their email exchanges. Otherwise, the summary communicates the heart of the book.
This summary has lot of issues including chronology problems, irrelevant information, and factual errors. Multiple claims draw from the acknonoledgments and about the author section and these claims appear in the middle of the summaryy Factual errors include that it an assassin, not Diana that murders Gavin in Banghok, Declan and Ingrid not being involved in past crimes in Malta, and Luther and Callie not having connections to Maggie's 's past. This summary misses some important points including that Maggie's 's husband was killed as result of the Malta/ Cyrano mission.
CPT-3.5-TURBB
The summary focuses on most of the key points mentioned in the story. starts chronological order and focuses on the evoloement of their relationskip from business arrangement towards happily ever after. It describes the initial and final scenes well, including all relevant details. However, it misses certain events such as getting to know each other, the green card interview, first kiss (..))
This not a good summary and should not be used. there are too many false statements. Also, the jumping back and forth in the narrative makes it hard to follow.
MIXTRAL
Table 20: Examples of positive and negative comments submitted by the annotators for specific models
34

---

Published as a conference paper at COLM 2024
Issue
Definition
Example
CHRONOLOGY
Issues with the chronological ordering of claims.
(..)) though it has some chronology problems (Ari's propesal comes after run the city, which after josh and Radhyn open the pop-up) (...)
OMISSIONS
The annutator mentions any omissions of content that should have been included the summary.
Onrissions: Dr. Rob Valentine grooned Sum- mer from when she was teenager until she turned 18 and then started an infimate rela- tionship. He later leaves Summer and ries her sister. Forn after he married, he doesn't let Summer move on.
FACTUALITY
Issues with factuality explicitly men- tioned by the annotator. Note that this category correlates partially with an- notated factuality errors.
There were some serious issues with this sum- mary. The first being that the book is referred to as "The Retirement summary which the incorrect title.
twice the
OVEREMPHASIS
Too much emphasis put less signifi- cant events or characters.
Salience: Churles is not inpurtant char- acter, he is the manager of the guest house where she stays New York, and she only chats him couple of times.
UNDEREMPHASIS
Certain events or characters are men- tioned but little emphasis put on their importance for the story.
There not enough emphasis on the rela- tionship bettween Justine and Dom, later becomes her husband.
VAGUE/GENERIC
Vague generic claims included in the summary.
Most of the sentences at the end of the mary are genevalized and there are no sub- staniial facts.
REPETITIVE
Repetitive claims included in the sum- mary.
As noted in the annotations, clains 16 and 17 repetitions and necessry.
DATA-INFLUENCED
summary was influenced by front and/ or back matter.
[n/ a: judgment for this calegory was made one of the co-authors during analysis]
COMPREHENSIVE
The annotator praises the summary for being comprehensive.
Out of all the summary claims, this feels the most releomt and comprehensive of the key events that take place.
WELL-DONE
The annotator praises the summary for being well-done.
This was good, things were sequence, and main points covered.
Table 21: Categories used for the analysis of annotators' comments on the quality of the entire summary.
35

---

Published as a conference paper at COLM 2024
Omission Type
Definition
Example
CHARACTERS
Summary fails mention impor- tant characters.
This summary excluded lot of the main plot points (...) and the very important prin- cipal antagonists Alma, Thomas, and Mar- ion. Alma and Thomas are present seseettdyyy rep- resentations of colonization and investors in the project turn Nhà Hoa into a bed-and- breakfast. Summary fails to mention impor- tant events and/ turning points.
EVENTS
Some important events in book were omit- ted, such as the part where Alex follows group of young people to house and has sex with a girl's boyfriend, the part where she sneaks into club and pretends be little boy's nanny, and the part where she follows Margaret to her home. These events are filled with tension, showcase Alex's dar- ing exploits, add deeper layer of meaning the story, and ultimately propel the narra- tive, so they shoudd have been included. Alex almost drowns in the beginning of the novel, frightening incident that she mentions to Victor during the party.
ATTRIBUTES
Summary fails mention attributes details about the events, interac- tions, characters, etc.
(...) but the overall summary misses huge plot point of Caroer and Roman being the same person.
RELATIONS
Summary fails mention specific relationships between the charac- ters which are important to the plot.
There no mention of Clover and Amos the Desert King being mates. Nor Fawn and Dean the Garden King being mates.
THEMES
Summary fails to depict main/ important themes of the novel.
Water, pools, and beaches are recurring metaphors in the book, yet they aren't t men- tioned highlighted in any of the claims.
Table 22: Description of omission categories used for annotating comments provided by our evaluators. Omissions were annotated in two steps: (1) binary choice (either omissions were mentioned or not), and (2) categorization.
OMISSION TYPE
CLAUDE-3-Orus
GPT-4-TURBO
GPT-4
GPT-3.5-TURBO
MIXTRAL
Characters
4.17
23,08
7.69
7.69
23.08
Events
33.33
57.69
38.46
65.38
38.46
Attributes
16.67
38.46
34.62
34.62
38.46
Relations
8.33
11.54
11.54
11.54
15.38
Theme
12.50
15.38
3.85
15.38
19.23
Table 23: Percentage of summaries affected by specific type of omission error by model.
36

---

Published as a conference paper at COLM 2024
Model Comment
Summary
None of the claims is completely factually correct, or focused on the book they ought to be. The first two claims where the protagonist Alex is mentioned, but both claims are still wrong. The other claims are incomprehensible have absolutely noth- ing to do with the book, and are therefore inadmissible. In fact, the claims appear be focused on different topic entirely, not the book. Please note that the quotes in- cluded below don't 't' necessarity prove any of the claims wrong (or right, for that matter). (...)
apologize for the confusion, but the provided summaaries do not appear to be related to the same story. Summary I describes novel about young woman named Alex and her experiences in the Hamptons, while Summary 2 simply states that the provided excerpt is a table of contents or metadata without any actual story content. As there is no coherent narrative between the two summaries, it not possible to merge them into a single, comprchensice summary. In order to create a merged summary, I would need access to summaries that are all velated to the some story, with each summary focusing on different parts or aspects of that story. If you are able to provide a series of summaries that are all connected to the same narrative, would be happy to assist in combining them into single, cohesive summary that includes key events, backgrounds, settings, characters. objectives, and motivations, while also introducing major elements and organizing the summary in a chronological manner. Please let me know if you have any other summaries related to the story mentioned in Summary 1, as that would allow me to proceed with the merging process you have requested.
the only ones
CLAUDE-3-O 3-OUUS 3-OPUS
NOTE: The model declines to merge two summaries due to significant discrepan- cies in content.
This summarr largely factual and looks at the ovenall are of text, though it has some chronology problems (Ari's proposal comes after the run in the city, which comes after Josh and Radhya open the pop-up), and focuses disproportiontae conversation with another writer, Kate Robb, opposed the text itself. also doesn't 't discuss the nonlinear storytelling.
(..) Goldbeck's insightful writing and clever comedic timring elevate the narrative, making stand out and resomate with broad variety of readers. Reflecting upon shared preferences like tequiila and various NYC spots, Goldbeck successfully manages to craft relatable characters with everyday experi- ences, despite their complex emational journey. The author's 's creative process in the narvative, including decision to cut cruise
on the author's 's
GPT-4
subplot involving Ari working as comedian ship, adds a layer of interesting behind-the--st -he--esssss perspective. The narrative style, which pays homage to classic 2000s ro- mantic comedies, echoes elements of escapism, emphasizing the importance of literature during umprecedented times like the global pandemic. (..)
NOTE: The model includes back matter information. While accurate, it should not have been included in the summary.
This set of summary claims are very focused on the idea of themes of digital age and the story doesn't cover that at all. Its not even based a modern world. Chronology wasn't 't relevant as majority of the claims weren't 't cor- rect, applicable or veriftable. Factuality was on zero and the claims don't cover any of the key events that occurs except that there was themes of life and death.
(...) Parallel to Liltth and Vale's journey, the story deloes into the lives of modern city-dwellers: Alex, a budding In- stagram influencer; Jordan, TikTok content creator; Sam, Twitter-usiig usinn political activist; and Taylor, who leverages Facebook for business and family connections. Each character embodies the diverse ways social media influences the pursuit of fulfillment, showcasing the profound effects of these plat- forms on personal and collective experiences. (...) Themes of authenticity versus constructed online personas, the search for connectivity amidst digital isolation, and the double-ed -dged sword of viral fame are explered, reflecting on the profound impacts of social media in modern society. (...) In summary, this narrative offers a comprehensice exploration of human experiences, sponning the ethereal and the digital. By par- alleling the mystical journey of Lilith against the backdrop of a socicty deeply influenced by social media, the story in- vites readers into an inmersive world where the boundaries between life and death blur, and where the quest for under- standing and connection transcends the mortal coil and the digital divide. Through the intertwined liees of its chavacters, the story seroes as a reflection on the complexities of the digital age, hightighting the profound and varied impacts of social media on the contemporary human condition.
GPT-4-TURBO
NOTE: The model hallucinates informa- tion based on brief mentions of author's 's social media accounts.
Table 24: Examples of summaries influenced by front/ back matter information along with the annotators' comments. The CLAUDE-3-Ore example was excluded from the analysis because the model failed to generate a summary. Although not ideal, this behavior is arguably better than the model fabricating content.
37

---

Published as a conference paper at COLM 2024
Model
COMMENTS
CLAUDE-3-OPUS
also focuses extensioely on the last couple chapters of the book. This the only summary so far that has included claims about the very last chapter of the book that is from Dacre's 's point of view.
CLAUDE-3-OPUS
"(... and hits the main thematic elements of the text, though it disproportionately addresses the epilogue over other portions of the text (...)"
CLAUDE-3-OPUS
"This summary included a lot of true elements, but also included many irrelevant details not integral to the plot. This is especially true for the end of the book."
GPT-4-TURBO
"This summary focuses heavily on the end of the book and misses plot points that happen in the beginning of the book."
GPT-4
"(...) and focuses disproportionately on the author's conversation with another writer, Kate Robb, opposed the text itself." [the interview is included at the
end of the book]
MIXTRAL
"The summary puts emphasis Part of the book which is not in proportion to rest of the book" [Part last part]
Table 25: Comments from annotators on models' focus towards the book's end
No-Context
BM25
Human Evidence
Summarized by
Faithful
Unfaithful
Faithful
Unfaithful
Faithful
Unfaithful
F1 score against the human annotations
GPT-3.5-TURBO
0.727
0.261
0.835
0,476
0.712
0.430
MIXTRAL 0.643
0.183
0.837
0.244
0.784
0.406
GPT-4
0.687
0.130
0.794
0.088
0.721
0.207
GPT-4-TURBO
0.634
0.033
0.887
0.080
0.792
0.139
CLAUDE-3-OPUS
0.674
0.000
0.738
0.000
0.684
0.031
Overall
0.681
0.124
0.826
0.215
0.755
0.259
Token length the given evidence against prediction label
GPT-3.5-TURBO
0.0
1136.4
1131.0
292.4
126.5
MIXTRAL
0.0
1139.5
1132.9
211.7
153.4
GPT-4
0.0
1138.6
1132.5
241.7
160.8
GPT-4-TURRO
0.0
1141.9
1138.3
257.8
152.9
CLAUDE-3-OPUS
0.0
1134.5
1128.6
214.4
151.5
Average
0.0
1138.2
1132.6
243.6
149.0
Table 26: Comparison of automatic evaluation using GPT-4-TURBO based on different evidence extraction methods. We presents the score and token length of the extracted evidence for each summarizer. Overall mean values were calculated using all the claims across FABLES.
38

---

Published as a conference paper at COLM 2024
TITLE
PR-Faithful
RE-Taithful
PR-Unfaithful
RE-Unfaithiful
A Haunting the
0.821
0.329
0.230
0.650
Hill
Agency for Scandal
0.960
0.133
0.034
0.833
Divine Rivals
0,960
0.156
0.140
0.917
Fairytale of New York
1.000
0.123
0.174
1.000
Flawless
0.950
0.217
0.012
0.500
Fourth Wing
1.000
0.169
(1.112
1.000
Modern Divination
1.000
0.062
0.092
1.000
Only For The Week
0.893
0.186
0.056
0,500
Pet
0.871
0.151
0.121
0.881
Romantic Comedy
1.000
0.170
0.020
1.000
Same Time Next Year
0.667
0.161
0.204
0.700
She Is Haunting
0.917
0.220
0.065
0.667
Six Scorched Roses
0.750
0.179
0.228
0.701
Sorrow and Bliss
0.983
0.197
0.029
0.750
The Atonement Murders
1.000
0.067
0.069
1.000
The Guest
0.688
0.182
0.253
0.810
The Marriage Act
1.000
0.101
0.041
1,000
The Spy Coast
0.864
0.151
0.103
0.790
The Wager
1.000
0.495
0.085
1,000
The White Lady
0.750
0.045
0.151
0,938
This Impossible
1.000
0.147
0.044
1.000
Brightness
Viciously Yours
0.950
0.200
0.113
0.833
Weyward
0.947
0.504
0.161
0.750
Wildfire
1.000
0.229
0.036
1.000
You, Again
1.000
0.214
0.041
1,000
Yellowface
0.933
0.163
0.119
0.938
Table 27: Precision (PR) and Recall (RE) from LM evaluation using GPT-4-TURBO no context for each book.
TITLE
PR-Faitiful
RE-Failhful
PR-Unfaithful
RE-Usfaithful
A Haunting on the Hill
0.967
(15117
0.337
0.950
Agency for Scandal
1.000
0.570
0.063
1.000
Divine Rivals
0.980
0.427
0.190
0.917
Fairytale New York
1.000
0.230
0.195
1.000
Flawless
1.000
0.352
0.033
1.000
Fourth Wing
0.985
(16662
0.244
0.950
Modern Divination
1.000
01.409
0.138
1.000
Only For The Week
1.000
0.509
0.119
1.000
Pet
0.981
0.676
0.242
0.833
Romantic Comedy
1.000
0.204
0.024
1.000
Same Time Next Year
0.851
0.780
0,300
0.600
She Is a Haunting
0.985
0.641
0.186
0,750
Six Scorched Roses
0.907
0.521
0.334
0,759
Sorrow and Bliss
1.000
0.340
0.038
1.000
The Atonement Murders
1.000
0.333
0.093
1.000
The Guest
0.868
0.644
0.391
0,589
The Marriage Act
1.000
0.360
0.049
1.000
The Spy Coast
0.975
0.392
0.168
0,933
The Wager
1.000
0.810
0.220
1.000
The White Lady
1.000
0.311
0.201
1.000
This Impossible
1.000
(13788
0.049
1.000
Brightness
Viciously Yours
0
0.508
0.156
0.688
Weyward
1.000
0.598
0.157
1.000
Wildfire
1.000
0.387
0.042
1.000
You, Again
0)967
(1.469
0.048
0.667
Yellowface
0.935
01.461
0.153
0.813
Table 28: Results of average Precision (PR) and Recall (Re) estimated by human evidence and LM evaluation using GPT-4-TURBO for each book.
39

---

Published as a conference paper at COLM 2024
T
PR-Faitlful
RF-Faithful
PR-Unfaithful
RF-Unfaithfuul
A Haunting on the Hill
0.902
0.781
0.520
0.550
Agency for Scandal
0.974
0.721
0.067
0.333
Divine Rivals
0,907
0.541
0.209
0.556
Fairytale of New York Flawless
0.874
0,712
0,263
0.556
1.000
0.663
0.056
1.000
Fourth Wing
0,953
0.759
0.244
0.600
Modern Divination
0.950
0.644
0.142
0.700
Only For The Week
0,967
0.789
0.167
0.417
Pet
0.907
0.604
0,175
0.649
Romantic Comedy
1.000
0.675
0.044
1.000
Same Time Next Year
0.836
0.859
0.333
0.425
She Is Haunting
0,949
0.772
0.100
0.250
Six Scorched Roses
0.816
0.500
0.269
0.616
Sorrow and Bliss
1.000
01588
0.077
1.000
The Atonement Murders
1.000
0..62
0.115
1.000
The Guest
0.845
0.737
0.395
0.598
The Marriage Act
0.987
0.833
0.119
0.833
The Spy Coast
0.953
0.527
0.216
0.738
The Wager
0.958
(1,862
0.100
0.167
The White Lady
0.836
0.628
0.097
0.250
This Impossible
0.961
0.607
0.022
0.250
Brightness
Viciously Yours
0.929
0.725
0.172
0.521
Weyward
0.969
(1.774
0.300
0.583
Wildfire
0.980
0.664
0.043
0.750
You, Again
0.960
0.609
0.015
0.333
Yellowface
0.941
0.695
0.204
0.562
Table 29: Results of average Precision (PR) and Recall (RF) estimated by BM25 retriever and LM evaluation using GPT-4-TURBO for each book.
Evaluation
TITLE
PR-Faitiful
RE-Faitiful
PR-LInfaitiful
RE-Linfaitlful
Only For Week Pet
0.960
0.972
0.333
0.167
0,921
0,923
0.333
0.262
She Is Haunting
0.957
0.949
0.417
0.333
GPT-4- TURBO
Six Scorched Roses
0.794
0.958
0.625
0.288
Sorrow and Bliss Viciously Yours Yellowface
1.000
0.868
0.139
1.000
0.919
0.919
0.367
0.354
0.952
0.948
0.450
0.438
Only For The Week
0,980
0.971
0.571
U.667
0.920
0.910
0.429
0.462
She Haunting Six Scorched Roses Sorrow and Bliss Viciously Yours Yellowface
0,968
0.968
0.571
CLAUDE-3- OPus
0.919
0.958
0.800
0.667
1.000
0. 966
0.429
1.000
0.931
0.931
0.417
0.417
0,963
0.963
0,700
0.700
Table 30: Average Precision (PR) and Recall (Re) for the Entire Book (EB) approach (i.e., prompting the model with a claim and entire book as evidence) broken down by the rater models (GPT-4-TURBO and CLAUDE-3-OPUs), for each book.
Model
Faithful
Unfaithful
Partial support
Can't verify
GPT-3.5-TURBO
111
22
20
GPT-4
152
11
30
MIXTRAL
144
18
GPT-4-TURBO
107
17
25
CLAUDE-3-Orus
140
17
Table 31: Number of claims per label for each model in the sub-dataset of seven books.
40

---

Published as a conference paper at COLM 2024
GPT-4-TURBO
CLAUDE-3-OPUS
Human
Claim Source
Unfaithful
Faithful
Unfaithful
Faithful
Unfaxthful
CLAUDE-3-OPUS
GPT-4-TURBO
GPT-4
GPT-3.5-TURDO
14
13
MIXTRAL
CLAUDE-3-OPUS
10
130
133
Faithful
GPT-4-TURBO
107
103
GPT-4
20
132
10
142
GPT-3.5-TURBO
10
101
104
MIXTRAL
136
141
Table 32: Count of labels predicted by (LAUDE-3-Orut and GPT-4-TURBO contrasted with numan-annotated labels, segmented by the model that generated each claim.
Reasoning Type
CLAUDE-3-OPUs (28 examples)
GPT-4-TURBO (37 examples)
Both Models
(24 examples)
Indirect
75.0%
73.0%
75.0%
Direct
14.3%
12.5% 10.8%
Subjective
7.1%
8.3% 10.8%
Extra info
3.6%
5.4%
4.2%
Table 33: Reasoning type distribution for false positives case by each model
41


Reviews:
[{"review_id": "DbAby5bOi9", "reviewer": "Reviewer_FiuX", "summary": "This paper introduces a human evaluated dataset of LLM generated summaries of book-length documents (>100K tokens). Humans manually evaluated the faithfulness of summaries by checking the truthfulness of automatically extracted claims (using GPT4). Further, the evaluation covers typical issues of LLM generated summaries where the content of the summaries is analyzed, i.e. omission or chronological mistakes are evaluated. The paper introduces a taxonomy of such common mistakes. The dataset covers roughly 3K claim level annotations across 26 texts. The authors additionally implemented automatic rates by prompting an LLM in a zero-shot manner to verify a single claim given evidence from the book.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to the reviewer for the valuable comments and insightful feedback. We would like to address the concerns raised in the review below.\n\n``\"Intermediate steps for human evaluation are automatically performed, especially the claim extraction, which might also introduce errors.\"``\n\nGreat point! While our paper contains a discussion of the high precision of the claim extraction in Section 2, we did not include a discussion of recall. In response, we closely analyze the extracted claims on a subset of 20 summaries (371 sentences, 450 extracted claims), from which we conclude that __the claim extraction process is not responsible for the omissions that our annotators observe. We will include details about this experiment in the next version of our paper.__\n \n__Recall analysis:__ For each of the 20 summaries, we manually evaluate the quality of the extracted claims against the content of each summary. Calculating recall is difficult since it is unclear what granularity to calculate it against (e.g., sentences, clauses). 3.8% of the 371 sentences in the 20 summaries were omitted in extracted claims. Of these, 85.7% were generic statements, and 14.3% were minor details. We also qualitatively observe a small percentage of omitted details at the sub-sentential level (e.g., clauses), none of which impacts the narrative. These omissions can be broadly categorized into two types:\n\n1. Generic statements lacking substantive content. For instance, “The narrative unfolds with intrigue, danger, and treacherous encounters” appears in the summary but is omitted in extracted claims. Note that this sentence only addresses things already covered by other extracted claims in a generic way, so omitting it has few consequences.\n\n2. Insignificant details that contribute little to the narrative. For instance, “Altha, a 17-century woman, stands trial unjustly accused of witchcraft due to her remarkable healing abilities which are misunderstood by her village” appears in the summary, but “misunderstood by her village” is omitted in the claims. However, this is only a minor detail with little impact on the narrative.\n\nImportantly, __we confirm that none of these discrepancies between the content of the summaries and claims were reasons for criticism of omissions, chronological errors, and factuality issues in the annotators’ summary-level free-form comments.__ In the next version of the paper, we will run this qualitative check on all summaries and their extracted claims.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a useful study on current LLM capabilities regarding faithfulness. The presented results are useful information for the community and the provided dataset offers a usefull analysis, which will might improve future generations of LLMs. In general it is very valuable to be aware of current capabilities and common mistakes.", "reasons_to_reject": "Intermediate steps for human evaluation are automatically performed, especially the claim extraction, which might also introduce errors. The paper would benefit from an analysis of the correctness and completeness of claims extracted from the summary (I could not find that in the paper). Apart from that, there are no obvious reasons to reject this paper.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MfR3U0Iexj", "reviewer": "Reviewer_xhrf", "summary": "- This work presents a detailed evaluation of LLM-generated summaries of long texts (full books).\n- The evaluation covers a human annotation for faithfulness of 3000+ individual claims in the summaries.\n- The paper compares summaries of different LLMs, and contributes to a qualitative understanding of weaknesses and common error types.\n- It provides mixed/negative results for automatic (LLM-based) faithfulness ratings, and connects this to abstractiveness and missing context, highlighting limitations to this paradigm in summarisation evaluation.\n- The authors promise to release the faithfulness annotations (though not the books themselves), which could be a valuable resource for the community.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedback and address their concerns below.\n\n`Limited methodological innovation`\n\nWe believe evaluation is crucial and complements the development of new methods. The field lacks robust benchmarks for assessing long-context LLMs with extremely long inputs due to data contamination and the challenge of evaluating tasks requiring familiarity with entire books. Our work aims to fill this gap.\n\n`Future directions`\n\nThe reviewer is concerned about how to improve the auto-rater. Please see our response to Reviewer AyeL.\n\n`Annotation quality control`\n\nAs this concern is shared with Reviewer BatK, we kindly refer the reviewer to our response to Reviewer BatK due to space limit.\n\n`Recall of claim decomposition`\n\nIn Figure 2, the omitted part is a generic statement at the beginning of the summary, but the romantic relationship between the characters is indicated in claims 4, 7, 9, 12, 13, 14, and 18. For a detailed analysis of recall, please see our response to Reviewer FiuX.\n\n`Claim vs subclaim-level annotation`\n\nIn footnote 8, the phrase “claims sometimes contain multiple subclaims” refers to claims making multiple statements. During coding, we consider all subclaims, so a single claim may be assigned multiple labels to ensure each aspect is classified.\n\n`Correlation of Qualitative Observations with Faithfulness`\n\nThis question is relevant to Section 5, where we find that the least faithful models tend to be more generic. In addition, while all models omit important information according to annotators, Claude 3 Opus (most faithful) is noted for making the fewest omissions.\n\n`Different Prompting`\n\nUnfortunately, with book-length input few-shot prompting is impossible with the evaluated models. As per wording of the prompt, we conducted further experiments requesting (1) the answer-only, (2) answer and then explanation, and (3) explanation and then answer. We observe ~1% difference in accuracy for these prompts. \n\n`Difference Across the Books`\n\nWe measured the percentage of incorrect claims across books. Most variation we observed was due to the models; for example, Claude 3 Opus consistently performs well. There is also some variation across books, but it’s unclear how much of it is due to the difference in book content. Due to space constraints in the rebuttal, we are unable to include a detailed table of results by book. We will add it along with a relevant discussion in the next version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written, clearly structured, concise, makes well motivated methodological choices grounded in recent prior work (summarisation-related, faithfulness-related, claim-extraction related, ...), and presents a research journey with lots of qualitative insights and interesting detail.\n- The annotation is cleverly designed with a focus on books that annotators are already familiar with.\n- The error analysis, in particular in connection with the auto-rating is extensive and in-depth. \n- I enjoyed reading this manuscript.", "reasons_to_reject": "- There is little methodological innovation in this work - it fully focuses on evaluation.\n- While the paper paints a great picture of the status quo in terms of weaknesses, I miss a constructive path forward. The auto-rater path was explored, and its limits for this use case are made clear (which is great), but I miss concrete answers to the question \"How can this be improved?\"\n- Annotation quality is not cross-checked (from what I can tell) - e.g. via duplicate annotations and checking the agreement between annotators. I can see that this may not be straightforward since it requires multiple readers per book. But it would help shed some light on how reliable the faithfulness ratings are - without this annotation quality is unclear.", "questions_to_authors": "- Did you validate that the annotators have read the book, or thoughts on how that could be validated? It's unclear to me how reliable self-reporting is in this context.\n- Could you address recall / coverage of the extracted claims? (e.g. in the example of Fig 2, the summary contains a part about a romantic relationship, which isn't extracted as a claim).\n- Unclear: claim vs subclaim-level annotation / multiple annotations per claim. Can you clarify this?\n- Are there any qualitative observations on summaries of the different models that correlate with faithfulness?\n- 0-shot prompting LLMs for faithfulness classification may not work well out of the box, depend on the prompt, etc. Do you have a sense for how important that is, and whether that changes the relative performance? Otherwise, what can we readers take away from these potentially very prompt-dependent relative numbers?\n- Did you obtain duplicate annotations for some books / summaries? What is the agreement?\n- Did you observe systematic differences across books? I.e. how much of the variation in faithfulness comes from different models, and how much from different books?", "extra_scores": {}}, {"review_id": "a3yaVoW6Yp", "reviewer": "Reviewer_AyeL", "summary": "The paper presents a new dataset for evaluating faithfulness and content selection in book-length summarization. The annotations are collected from Upwork platform and are based on summaries generated by five different LLMs: GPT-3.5-Turbo, GPT-4, GPT-4-Turbo, Mixtral, Claude-3-Opus. The authors compare the human annotations results with several LLM-based auto-raters, and show that non of them correlates strongly with human annotations. The authors also develop a typology that includes error types besides unfaithful error, and make corresponding analysis.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "We appreciate the detailed feedback and would like to address the reviewer's concerns.\n\n`Choosing the books`\n\nWe did not select books ourselves; rather, we hired annotators who self-report books they have read that match our criteria (>2023 fiction), as noted in Section 2. That said, our dataset has multiple genres (see Table 7).\n\n`Annotation quality control`\n\nGreat point! __We perform two small-scale analyses to show the annotations’ high quality:__ (1) inter-annotator agreement, and (2) self-consistency. We will add them to the next version of our paper.\n\n__Inter-annotator agreement:__ For two books, we hired another annotator who had also read them, resulting in 115 claims with \noverlapping annotations. __The agreement rate between the two annotators is 91.30%, with Cohen’s Kappa of 0.621 (p < .0001), indicating substantial agreement__ (https://aclanthology.org/J08-4004.pdf). Unfortunately, we cannot do this for the entire dataset, as finding multiple workers who have read the same book is challenging and expensive, costing ~$250 per book and 10 annotation hours.\n\n__Self-consistency:__ We randomly selected __five books__ and compared the annotations made on two of the summaries for semantically similar claims. Consistent labels would suggest that labels were not arbitrarily assigned. Out of all 127 claims in the first set of summaries, 46 had semantically equivalent claims in the second set,  __all 46 claims were consistently labeled.__\n\n`Claim taxonomy`\n\nThe taxonomy was created via manual coding of collected annotations and aligns with prior work (https://arxiv.org/pdf/2303.01432). See Table 15 for details. In short, indirect reasoning is multi-hop reasoning over different parts of the book, subjective reasoning calls for subjective judgment (usually related to themes), and extra info requires meta information about the book (e.g., publication date). We will clarify Table 3 in the next version.\n\n`Future directions`\n\nWe thank the reviewer for pointing this out and will add it in the next version. One future direction is to address content omissions via planning (e.g., generate important arcs, characters, and relationships, and use them as a basis for the summary). Another is to build more reliable auto-raters, as our work shows that existing ones are insufficient. Once auto-raters are improved, we can also use their signals to fine-tune LLMs to become better summarizers (similar to factuality tuning in https://openreview.net/pdf?id=WPZ2yPag4K).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The dataset could be a good benchmark for faithful book-length summarization task.", "reasons_to_reject": "1. some important details are missing from the paper about the dataset \n- how to you choose the books? do you select ones from diverse domain?\n- how do you evaluate and control the quality of the annotations?\n\n2. In Table 3 in section3, how is the taxonomy created? what is the definition for each of the claim type and reasoning type? The difference between indirect reasoning, subjective reasoning and extra info reasoning is not very clear to me, at least from the examples presented.\n\n3. The paper presents a very useful benchmark, with some interesting findings. It will be more insightful if the authors could point out some research directions based on the findings.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VdPgw0Vmmk", "reviewer": "Reviewer_wcHx", "summary": "The paper addresses an important bottleneck in the current research of long-input undestanding and summarization: \nevaluation of faithfulness. The authors report a large scale human annotation experiment where summaries produced by a diverse set of SOTA models were evaluated with respect to truthfulness using established methodology (claim-level evaluation). The results \nshowed that a substaintial portion of LLM-generated summaries have unfaithful claims. A follow-up experiment to see whether \nLLM-powered methods could match human-level faithfulness detection indicated a major shortcoming of such methods: it appears that all automatic methods tend to always evaluate summaries as faithful. The authors observe that often unfaithfulness errors require multi-hop reasoning to identify.\nIn addition to the main results, the paper also provides examination of the types of content errors that automatic summaries contain. Specfically, one frequent error is omission of essential information.\nThe authors promise to release the dataset of annotations.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their insight feedback and address the concerns mentioned in the review below. The reviewer asks about why our automatic evaluation metric fails:\n\n`\"A more in-depth analysis of where and why the automatic evaluation methods fail would be welcome. One valuable insight is that unfaithful claims often require multi-hop reasoning to be detected: it would useful to estimate the percentage of cases where this is the case. The authors note themselves that the sample size of unfaithful claims is very small -- so it would probably be possible to even do the in-depth analysis manually.\"`\n\nThis is an insightful point. As we show in Table 3, 50.2% of unfaithful claims require multi-hop (or indirect) reasoning to be invalidated. Following the reviewer’s suggestion, we look at predictions from our auto-rater experiment (on seven books) for which Claude 3 and GPT-4 Turbo incorrectly mark a false claim as true, and we annotate the type of reasoning required to verify these claims. The results in the table below show that ~75% of the failure cases require multi-hop reasoning over the book, which is significantly higher than the data distribution of 62.8% with seven books and suggests that our auto-raters struggle with multi-hop reasoning. We will include a more in-depth discussion and analysis in the next version of our paper.\n\n| Reasoning Type | Claude 3 False Positives (28 examples) | GPT-4 Turbo False Positives (37 examples) | False Positives Common to Both Models (24 examples) |\n|----------------|---------------------------------------|-----------------------------------------|----------------------------------------------------|\n| Indirect       | 75.0                                  | 73.0                                    | 75.0                                               |\n| Direct         | 14.3                                  | 10.8                                    | 12.5                                               |\n| Subjective     | 7.1                                   | 10.8                                    | 8.3                                                |\n| Extra Info     | 3.6                                   | 5.4                                     | 4.2                                                |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Addresses a very pressing problem in the current research on long-input summarization.\n- Explores a diversity of methods for automatic evaluation.\n- Evaluates a good diversity of SOTA summarization model.\n- Dataset of human annotations, performed with a clean methodology.", "reasons_to_reject": "- A more in-depth analysis of where and why the automatic evaluation methods fail would be welcome. One valuable insight is that unfaithful claims often require multi-hop reasoning to be detected: it would useful to estimate the percentage of cases where this is the case. The authors note themselves that the sample size of unfaithful claims is very small -- so it would probably be possible to even do the in-depth analysis manually.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CypjjmUwpJ", "reviewer": "Reviewer_BatK", "summary": "This paper presents work on long-form summarization (of books), focusing on evaluating the quality of summaries generated by various LLMs. This is a well-written paper with interesting analysis. The experimental results show a lot of room for improvement in LLM summarization of books. The annotated data would also be released, providing a valuable resource for future work.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful comments and would like to address their concerns below. First, the reviewer wonders about the annotation quality:\n\n`Annotation quality control`\n\nGreat questions! The reviewer is correct that we only hire one annotator per book, as in practice it is difficult to find 2-3 people who have read the same book. That said, we believe that our annotators had not only read the books but also carefully conducted the task, as the process of gathering evidence and giving comments (in addition to the T/F label) forces them to frequently revisit the book. \n\n__We perform two additional small-scale analysis experiments that demonstrate the high quality of our dataset:__  (1) _inter-annotator agreement_ on a subset of claims where we had access to another annotator who also read the book, and (1) _self-consistency_ of annotations (i.e., how often a single annotator assigns the same label to claims with the same semantic content generated by different models). We will add a detailed description of these experiments to the next version of our paper.\n\n__Inter-annotator agreement:__ For two books in our dataset, we were able to hire another annotator who had also read those books and provided overlapping annotations. This process resulted in 115 re-annotated claims, and __we find the agreement rate between the original annotator and the new annotator is 91.30%, with Cohen’s Kappa of 0.621 (p < .0001), indicating substantial agreement.__ Unfortunately, re-annotating the entire dataset with multiple annotators is prohibitively expensive, costing around $200-$250 per book and requiring ~10 annotation hours.\n\n__Self-consistency:__ For each book, an annotator analyzed five summaries, each generated by a different model. To assess self-consistency (intra-annotator agreement), we randomly selected __five books__ and compared the annotations made on the first and last summaries (as per annotation order) for claims with the same semantic content. Consistent labels would suggest that they maintained a stable judgment throughout the process. Out of 127 claims from the first summary, 46 have semantically-equivalent claims in the last summary,  __all 46 of these claims were consistently labeled.__\n\n`Recall of claim decomposition`\n\nThe Reviewer wonders about the recall of the claim decomposition step. As this concern is shared with Reviewer FiuX, we kindly refer the Reviewer to our response to Reviewer FiuX.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- comprehensive evaluation and analysis\n- valuable dataset to be released to the community\n- provides pointers on where LLMs/long summary models could be improved", "reasons_to_reject": "- nothing obvious", "questions_to_authors": "- how sure are you that the annotators on UpWork had read the books carefully enough to be able to judge the quality of the summaries? \n- would you have any way to estimate the recall of the claim decomposition step? It seems that one of the main errors following is omission of details. Could that be related to recall of the claim extraction phase? \n- did I understand correctly that there is just one annotator per book? Did you consider multiple annotations per book to get a sense of agreement or estimate of the difficult of the task for humans?\n\nTypos\nThe color names in the heading of Table 6 don't match the colors (at least for me).", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper tackles the evaluation of faithfulness and content selection in LLM-generated summaries of books. It introduces a valuable human-annotated dataset, providing a benchmark for assessing LLM performance on this task. The paper offers detailed insights into the strengths and weaknesses of current LLMs, particularly concerning faithfulness and content errors. It also highlights the limitations of LLM-based auto-raters for evaluating faithfulness.

Reviewer Consensus:

Reviewers agree that this is a well-written and insightful paper, praising its clarity, structure, and in-depth analysis. The creation of the dataset is recognized as a significant contribution to the field.

Points of Discussion:

One reviewer questioned the methodological novelty, while others commended the well-motivated design choices. Concerns about annotation reliability were raised, but the authors addressed these by providing evidence of high inter-annotator agreement and self-consistency. The paper was also suggested to include concrete proposals for addressing identified weaknesses in LLM summarization.

Overall Assessment and Recommendation:

Despite minor disagreements, the paper is viewed favorably. It addresses a critical need in the field and provides a solid foundation for future research. The authors' detailed rebuttal effectively addresses concerns, further strengthening the paper's contributions.

Based on the assessments and responses, I recommend acceptance. This paper is a valuable contribution to the field and will likely stimulate further research on evaluating and improving long-form summarization models.
Decision: Accept