BooookScore: A systematic exploration of book-length summarization in the era of LLMs
Authors: Yapei Chang
Kyle Lo
Tanya Goyal
Mohit Iyyer
Venue: ICLR 2024 oral (2024)
Keywords: summarization
evaluation
long context
prompting
LLM
Primary area: datasets and benchmarks

Abstract: Summarizing book-length documents ($>$100K tokens)  that exceed the context window size of large language models (LLMs) requires first breaking the input document into smaller chunks and then prompting an LLM to merge, update, and compress chunk-level summaries. Despite the complexity and importance of this task, it has yet to be meaningfully studied due to the challenges of evaluation: existing book-length summarization datasets (e.g., BookSum) are in the pretraining data of most public LLMs, and existing evaluation methods struggle to capture errors made by modern LLM summarizers. In this paper, we present the first study of the coherence of LLM-based book-length summarizers implemented via two prompting workflows: (1) hierarchically merging chunk-level summaries, and (2) incrementally updating a running summary. We obtain 1193 fine-grained human annotations on GPT-4 generated summaries of 100 recently-published books and identify eight common types of coherence errors made by LLMs. Because human evaluation is expensive and time-consuming, we develop an automatic metric, BooookScore, that measures the proportion of sentences in a summary that do not contain any of the identified error types. BooookScore has high agreement with human annotations and allows us to systematically evaluate the impact of many other critical parameters (e.g., chunk size, base LLM) while saving \$15K USD and 500 hours in human evaluation costs. We find that closed-source LLMs such as GPT-4 and Claude 2 produce summaries with higher BooookScore than those generated by open-source models. While LLaMA 2 falls behind other models, Mixtral achieves performance on par with GPT-3.5-Turbo. Incremental updating yields lower BooookScore but higher level of detail than hierarchical merging, a trade-off sometimes preferred by annotators. We release code and annotations to spur more principled research on book-length summarization.

Published as a conference paper at ICLR 2024
BOOOOKSCORE:
A SYSTEMATIC EXPLORATION OF BOOK-LENGTH SUMMARIZATION IN THE ERA OF LLMS
Yapei Chang
Kyle Lo
University of Massachusetts Amherst yapeichang@ umass. edu
Allen Institute for Al kylel@allenai. org
Tanya Goyal
Mohit lyyer
Princeton University
University of Massachusetts Amherst miyyerêcs. umass edu
taryaapoyal@princetoo.co
ABSTRACT
Summarizing book-length documents (>100K tokens) that exceed the context window size of large language models (LLMs) requires first breaking the input document into smaller chunks and then prompting an LLM to merge, update, and compress chunk-level summaries. Despite the complexity and importance of this task, it has yet be meaningfully studied due to the challenges of evaluation: isting book-length summarization datasets (e.g.. BookSum) are in the pretraining data of most public LLMs, and existing evaluation methods struggle capture rors made by modern L.M summarizers. In this paper, we present the first study of the coherence of LLM-based book-length summarizers implemented via two prompting workflows: (1) hierarchically merging chunk-leve summaries, and (2) incrementally updating running summary. We obtain 1193 fine ane-grained human annotations on GPT-4 generated summaries of 100 recently-publisher books and identify eight common types coherence errors made by LLMs. Because hu- man evaluation is expensive and time-consuming, we develop automatic met- summary that
ric, BooOOkSCORE, that measures the proportion of sentences do not contain any of the identified error types. BOOOOKSCORE has high agree- ment with human annotations and allows us to systematically evaluate the impact of many other critical parameters (e.g., chunk size, base LLM) while saving $15K USD and 500 hours in human evaluation costs. We find that closed-source LLMs such as GPT-4 and Claude produce summaries with higher BOOO than those generated by open-source models. While LLaMA 2 falls behind other mod- els, Mixtral achieves performance on par with GPT-3.5-Turbo. Incremental up- dating yiclds lower BooOokSCORE but higher level of detail than hierarchical merging, trade-off sometimes preferred by annotators. We release code and annotations to spur more principled research on beok- summarization.
(kshabb.co//llaak//BoooocCC
INTRODUCTION
Just two years ago, caaamattialll-gennraaae summaries were riddled with artifacts such as grammar errors, repetition, and hallucination (Zhao et al. 20200 Fabbri et al.) 20200 Goyal & Durrett (021)) Nowadays, such artifacts have mostly disappeared; fact, Pu et al. (2023b) find that summaries generated by large language models (LLMs) are preferred over those written by humans, leading them to pronounce the death of summarization research. However, as with most prior work on summarization, the input documents in their study relatively short (<1 tokens). Widespread adoption of LLMs outside the research community has driven the development of a more ambitious task: summarizing book-length documents, which define to be texts longer than 100K tokens.

---

Published as a conference paper at ICLR 2024
As these documents exceed the context window limits of today's LLMs (e.g.. 8K tokens for GPT-4), summarizing them via prompt-base based approaches necessitates heuristics to chunk the input, process each chunk, and then combine and compress the outputs (Wu et al. 202T
Despite the promise that LLMs hold for long-context tasks, the research community still lacks a principled and systematic approach to evaluate their capabilities on book-length summarization. Our paper identifies three open challenges with evaluation: (1) data contamination, in which exist- ing benchmarks such BookSum (Kryscinski et al. 2022) are in the pretraining data of modern LLMs (Chang et al. 2023); (2) an unexplored error distribution, as most prior summarization re- search centers around short source documents and fails to capture coherence errors that are exacer- bated by the "chunk and combine" book-length summarization setting, and (3) a lack of any reliable automatic metric, which requires careful design and validation against human annotations.
A protocol for evaluating coherence in book-length summarization ($3). To mitigate the impact of data contamination, we design our evaluation framework around the use of newly-published books. We propose a reference-free protocol that leverages human annotation of the coherence of L.LM-generate summaries (i.e., their logical connectedness) under different prompt- ing strategies. Our protocol unifies and extends best- best-practices across disparate works in document understanding and evaluation research, including adoption of fine-grained annotation units, use of QA pairs to denote points of confusion, and a taxonomic breakdown of different coherence errors.
Contribution 1:
We validate our protocol by collecting 1 1193 span-level human annotations on GPT-4 generated summaries of a carefully curated set of 100 reccenty-publlisher books (costing $3K USD and 100 annotator hours) using two prompting strategies (hierarchical merging and incremental updating, shown in Figure [ In categorizing these annotations into eight frequent error types, we reveal an error distribution in GPT4 summaries that differs from that observed in prior studies short- document summarizers; notably, we identify new error types (causal omissions, salience errors) through our book-length summarization setting (Table
assess summary coherence 4$ Since our human evaluation is expensive, we follow recent work by developing an LLM-based evaluation metric called Boo0okScORE that identifies and explains instances of any of our eight established coherence errors in a given summary. Human validation shows that BooookScoRE'S annotations are almost as reliable as those of human annotators, which allows us automatically evaluate many other book-length summarization configurations. Because BOOOOKSCORE does not rely on gold summaries, can easily be used to evaluate new LLM summarizers on any collection of newly-publisher books, ensuring that the metric will remain meaningful for LLMs of the future.
Contribution 2: An automatic metric-BooookscoRE.
Contribution 3: A systematic evaluation of different LLMS using BooOOKSCORE (55) We use BooOok evaluate the impact of several critical design decisions on the coherence of generated summaries, including the choice of prompting strategy, base LLM, and chunk size, a study that altogether cost $10K (USD) in LLM API calls. Our findings include (1) hierarchical merging generally results in more coherent summaries but reduced level of detail compared to incremental updating: (2) GPT-4 and Claude 2 produce the most coherent summaries, while LLaMA 2 is sub- stantially worse and fails to follow instructions; (3) increasing the chunk size does not improve hier- archical merging but does substantially benefit Claude when using incremental updating; and (4) summary-level preference judgments highly subjective and do not correlate with BooOOKSCORE.
2 BACKGROUND: SUMMARIZING BOOK-LENGTH TEXTS WITH LLMS
Before discussing our evaluation protocol. we first outline two strategies hierarchical merging and incremental updating- for prompting an L.M to summarize book-length documents that exceed its maximum context size. In both strategies, the length of the input document necessitates first dividing it into smaller chunks and then repeatedly merging, updating, and/or compressing chunk-level partial summaries (Figure []. While neither strategy is well-explored by published research, hierarchical merging essentially adapts the strategy proposed by Wu et al. (2021) to zero-shot prompting, while incremental updating resembles chain-of-densiee
prompting proposed for short-document summa- rization (Adams et al. (2023). Both are implemented in widely-user open-source LLM libraries such as LangChain but the relative merits of each method remain unexplored.
I "hanCCaiin implements incremental updating via refine and hierarchical merging via map-reduce
2

---

Published as a conference paper at ICLR 2024
blerarchical mergingg
Incremental undading
compress summary
too
212
$3.4
update summary next
summaries
contaxt
summarize first chunk
$1.2
$1,2.3
summarize chuniss
divide book into chuniss
Figure I: perform book-length summarization, we first divide a book into smaller chunks that fit within the context window of an LLM. Then, we explore two strategies for summarization: (1) hierarchical merging, in which chunks are first summarized and then the corresponding summaries merged via separate prompts; and (2) incremental updating, in which global summary is updated and compressed as we step through the book chunk-by thunk-by-chunk.
More specifically, both strategies assume an LLM with context window size W is used to summa- rize an input document D whose length L > W. We thus split D into non-overlapping chunks C2 ... Crg1 &1 where C W is the length of each chunk
Hierarchical merging: Wu et al. (2021) propose a method in which an LLM (in their case, GPT- 3) is fine-tuned via reinforcement learning summarize each chunk and then hierarchically merge the chunk-level summaries until one summary is left of the entire input document. This method has since been simplified into zero-shot prompting strategy without further training, shown in Figure (left). Hierarchical merging requires three unique prompts for (1) summarizing an input chunk, (2) merging chunk-level summaries, and (3) merging summaries with added context from previously-generaan merged summaries. We ensure that the total length of each prompt and its associated inputs less than W Gi. where G1 hyperparameter controlling summary length that varies depending on the level I. Summaries are recursively merged until only one summary (of the full book) remains; see Appendix [.. for further details.
Incremental updating: It is possible that since hierarchical merging necessitates summarizing portions of the input document without complete context, it may introduce more coherence errors. For example, in the first level, chunks towards the end of the book will he summarized without knowledge of what came before. which can lead incoherent summaries especially for non-linear or multi-perspectie dating (Figure 0 right) that iterates through each chunk in order while continuously updating global summary with salient information. While this method may be better able to handle inter- chunk dependencies than hierarchical merging, requires more complicated prompts for (1) sum- marizing input chunk, (2) updating the global summary 81 $1,2,...i-11 with information from the current chunk 6, and (3) compressing the global summary when exceeds the maximum summary length G2, See Appendix A.2 for a full specification of incremental updating.
narratives. We thus explore an alternate prompting strategy incremennttte up-
3 EVALUATING COHERENCE OF BOOK SUMMARIES
In this section, we define our framework for human evaluation of coherence errors in book-length summarization. Our framework involves: (1) corpus collection focusing on newly-published books, (2) unification and extension of best-practices from prior document understanding and evaluation literature to guide data annotation, and (3) analysis of human annotations centered around emergent coherence error categories summaries generated by modern LLMss
We ensure each chunk ends sentence boundary.

---

Published as a conference paper at ICLR 2024
Collecting a corpus of newly-published books. The only existing public dataset for book-length summarization is BookSum (Kryscinski et al.) 2022). which contains famous books from the Project Gutenberg public-domain repository along with reference summaries scraped from popular websites such as CliffNotes and GradeSaver. Both the source books and reference summaries are in the pretraining data of existing L.LMs: Chang al. (2023) confirm that many books in the BookSum held-out split (c.g.. The Adventures of Huckleberry Finn, The Picture of Dorian Gray) are among the most-memorized books by GPT-4 and GPT-3.5-Turbo and we were able to auto-complete several reference BookSum summaries by prompting GPT-4 with short prefix of the summary.
To reduce the confounding impact of summary memorization, we manually collect 100 books pub- lished within the past year to form our dataset (see Table [3 for full list). Some of these books could still have appeared in the pretraining dataset of recent L.LMS such as Claude 2 and I. LMM2, although it is much less likely than in BookSum. However, summaries of these books do not pub- licly exist: we did not find summaries online for any books in our dataset, which significantly lowers the possibility of LLM memorization F The average length of the books in our dataset is 190K to- kens, compared to 112K tokens in BookSum. Due to copyright laws, we cannot publicly release this dataset; even if we could, we would still recommend that researchers collect their own datasets of nowly-published books to minimize contamination with LLMs of the future.
An evaluation framework for book-length summarization. Since we lack gold summaries, design our evaluation framework to be reference- frree which aids in scalability. To do this, our eval- uation framework synthesizes best-practices of prior document understanding and summarization evaluation research. Our evaluation employs: (1) fine fine-grained evaluation units as recommended by LongEval (Krishna et al.] 2023); (2) information-seell -eekiig questions to represent naturally-occurrinn points of confusion (Ko et al. 2020; Wu et al., 2023; Meng et al. [023] Newman et al. 2023); and (3) focus on summary coherence, which evaluates the logical structure and readability of the summary itself (Goyal et al. 2022a). We do not directly evaluate the faitlifulness of the summaries (i.e., how factually accurate they are at conveying information from the source text), as the length of the source texts poses considerable issues for any existing faithfulness evaluation. We qualitatively discuss faithfulness in Section 5 and leave further investigation for future work.
Annotation protocol. We implement our framework through source- and reference-free anno- tation protocol where (1) annotators read through an LLM--gnnerted summary, (2) highlight all confusing spans, and (3) ask question(s) for each marked span that highlight their confusion See Table I (third column) for examples of spans and questions produced by our annotators. We hired four annotators with extensive English proofreading experience tated 25 disjoint summaries. Each summary takes roughly 30 minutes to fully annotate with spans and questions, and we paid $15 USD per summary for total of S3K evaluate both prompting strategies. To generate the summaries, we set the base LLM to GPT-4 with a chunk size of 4096 and a maximum summary length G. = 1200; other hyperparameters are detailed in Section In total, the annotators mark 840 (incremental updating) and 353 (hierarchical merging) coherence errors for @eT-4-generated summaries; see Table [ (right) for the split across error types.
Upwork]] each of whom anno-
Validating the annotations: Typical measures of agreement are difficult to obtain in our setup, as measuring recall would require ground truth annotations with all possible coherence errors in the summaries; additionally, Goyal et al. (2022a) and Dou et al. (2022) observed low recall among an- notators when evaluating machine-genee generated text at fine- grained level. This motivates us to instead measure the precision of given error annotation (i.e., after reading the corresponding question, do you agree that the span is confusing?), as it is simpler and cheaper while still being an informative metric. Given span from summary marked as containing an error, along with questions highlight- ing the confusion, we ask annotators (1) whether they think the span is confusing; and (2) whether the corresponding questions highlight the central confusion. We use the same four annotators hired before for this task, but make them validate human and (and later GPT-4) annotations for 25 books that they did not annotate in the first task. Overall, we validated 1, 1,659 annotations for total cost of
'We roughly balance our dataset across the following genres: fiction, non-fiction, sci-fi, fantasy, historical, contemporary, and memoir. We also include both linear and non-lincar (multi-peree imusti-perspeetive and time-shifting) narratives in the dataset, and we purchase electronic copies of each of the 100 books in the dataset.
'However, we did find book reviews, which intentionally do not reveal major plot points other spoilers. s We also enabled forming relations between two spans in case multiple spans contributed to the same issue. http: / //tp:o//////////// com
4

---

Published as a conference paper at ICLR 2024
Table I: Definition of all coherence error types, an example annotation for each, and their prevalence (%) in generated summaries, which is calculated as the number of error occurrences in all summaries normalized by the total number of sentences all summaries.
Ervor Type
Definition
Example spans & questions
% errors per sentence inc / hier
Emity amission
object. place) meniioned in summary. but key comext details, milos- ing or unclear.
An entiyy
A mysterious introduces Proctor "Ariivalism. Quentoo: Who mysterious
7.71
Event omission
An even is mentioned in the sum-
Spon: During mission to find Cueli. Precior captured by watchmen while Thea escapes.
4.25 /2.27
but key details missing unclesr.
Question: happened Caeli?
Crasal amission
motication missing under explained.
Spon: Prector
Callista about the investigation Quertion: Why would Callista know something about the investigation?
2.75
Discominuity
An interruption in the Bow the nimalive such as subden jumps in time perspcctive.
Spon: In the new seitlement, Thea adjusts to her life, working hard and
2.23 1.56
linding soluce in nature.
Queition: Why the shift to Thes's persective?
Salience
Inclusion of details that do not con- muin plot.
Spon: His father.. flaes, resulting a chantic chase the significance incident?
1.42/ 1.03
Quertion:
Language
Spelling grammur besues; biguous wurding
Span: Despóse
Deborah hearbrokeen by Quertion: Why the preposition "Despite" used here when she is, fact, heatbroken hecouse of her love for him?
0.82/0.71
Incomiistency
discrepancy commudictio story's plot, velopment. or themess
Spon: In firewell, Procter murriss his bnother Malesim Cynthia and says goodbye his loved
0.97/ 1.03
Quertion: If Cynthia mother and Malcolm his brother, how can mother and sun muny? ?
Doplication
Redurdant repetition of similtr fonmation.
Span Proctor... deals with stidears and issues, coeking roof replacement.
2.12/ 1.18
from Callata
Spon Proctce's life continues is he.. deals with school fsmess, such as funding for a roof replacenenn the information appear twice?
Queztion;
$418.90 (USD) and discover that 79.7% % of annotated spans validated as legitimate through this task. More details on our validation can be found in Appendix[]
Categorizing coherence errors: After collecting spans and questions from the annotators, develop an error taxonomy consisting of the eight types detailed in Table [ which covers the vast majority of annotations, and manually code each annotation using this taxonomy. We intention- ally went through this process without relying on the SNaC taxonomy (Goyal et al., 2022a) so as to not be overly influenced by their error annotation schema which was tailor-made for fine-tuned sum- marization models. While we find considerable overlap in the two error schemas, we also discover two new instances of prominent eтrors not present in SNaC: causal omissions and salience issues. Our taxonomy also places less emphasis on language errors (e.g. coreference issues from SNaC) since modern L.LMs rarely make such mistakes (Goyal et al., 2022b)). Table shows that omission errors are the most common across both incremental and hierarchical prompting strategies, and also that hierarchical merging makes fewer errors of every type but inconsistencies.
BOOOOKSCORE: AN AUTOMATIC EVALUATION METRIC
Since human evaluation of summary coherence is not scalable due to the high financial and time cost, develop automatic metric- BoooOkSCORE: that prompts an LLM to identify instances of the eight error types we identified in Section 3 We validate BoOOOKSCORE via a human evaluation of its precision (following the annotation task discussed in the previous section) and show that its precision matches that of human annotators (78.2% vs. 79.7%). We then use BooookSCORF to evaluate many other book-length summarization configurations, saving S155K USD in evaluation costs and 500 hours in annotator time. We emphasize that incorporating definitions and examples from our error taxonomy into the prompt is critical to achieve high precision with BOoOOKSCORE
7 This cost includes validation of both human and BooooKSCORF annotations. 'In preliminary experiments without definitions and few-shot demonstrations, we qualitatively observe sig- nificantly reduced annotation precision.
5

---

Published as a conference paper at ICLR 2024
4.1 IMPLEMENTING BOOOOKSCORE
Motivated by prior successful efforts to evaluate LLM-gennrated text via LLMs, such as Al- pacaEval (Dubois al. 2023). FActScore (Min et al., 2023), and G-Eval (Liu al., 2023b), Boo0oKSCORE automatically measures the coherence of summaries generated by a book-length summarization system via few-shot prompting. BooookSCORE is both source-free and reference- free (i.c., it does not require access to the input book or reference summary), similar to the SNaC classifier built for fine-tuned summarizers by [ooya et al. 2022a).
Specification: Assume we have a summary consisting of sentences 81, $2........ We develop few-shot error-identffication prompt E that instructs the LLM to identify any instances of one of the eight specified error types in a given sentence 8, of the summary. Concretely, we iterate over each sentence 8; in the summary, feeding the prompt E, full summary S, and target sentence s; at each step. There are two acceptable outputs at each step: either (1) no error is found and the LLM outputs No con funfusion ion, or (2) an error(s) is identified and the LLM asked to generate a corresponding question and associated error type. We include two full summaries with 42 sentence- level annotations in the prompt as demonstrations
10 The BOOOOKSCORE single summary S
(Figure 2) is then computed as:
BOOOOKSCORE(S)- 1 Σ (LLM(E,,,s)) == No confusion] n 9,E8
When computing BoOOOKSCORE, we consider each sentence as a singular unit of confusion, rather than each of the questions associated with that sentence. This because both LLMs and human annotators occasionally ask multiple questions that essentially target the same issue within a given sentence Thus, our metric intuitively measures the propor- tion of sentences in the summary that contain no er- rors (i.e., higher is better). To obtain system-level score, we compute the mean BooOokSCORE across all summaries generated by that system.
Narrecic sumrray Any
Sunaoo"
What the digiftranne of the partyl
coardom
ens gen
S..uch nistowly avoiding trouble during saand costume
BooookSeore 92%6
GPT-4
(23/25 free sentences)
make plun her
Validating BoooOKSCORE: We validate BooOokSCORE annotations in the same way that we validate human annotations in Section 3 by hiring human annotators to judge whether they agree with an LLM -Lneegeeerted annotation (here, GPT-4). We observe that the precision of human annotations is 79.7% %o while the precision of BooookScoRE annotations is 78.2% % (details in Appendix D) Ad- ditionally, we compute BOOOOKSCORE using human annotations instead of LLn-ggeerated ones for both 4 configurations (i.e., replacing LLM(E, S, s:) in Equation with the human error annotation for s;) and observe extremely similar system- yields a BooOoKSCORE of 82.1 and 89.4 for
John Lenox
over...
4110 spece
Who John? (Fatity omiision)
Figure 2: BOOOOkSCORE measures the pro- portion of error-free sentences in sum- mary, where coherence errors are detected by prompting GPT-4.
scores. Using human annotations in Equation I
After iterating over the design in numerous preliminary experiments, we find that our prompt works most reliably at the sentence level, rather than at the full summary level. As such, sentence tokenization is required preprocessing step for Booook
Future work should focus on implementations the summary level, as would save many calls to the LLM; here, we need to prompt the model separately for each sentence.
10Tese examples contain a combination of sentences with and without confusion, all the while maintaining diverse range of error types. The full prompt can be found in M.4
"Foor example, the questions "Who is John? Is he Lia's husband?" both seek to establish John's identity. Counting the number of questions instead of highlighted sentences would inadvertently overstate the weight of certain errors found within the same sentence.
12 Recall that human annotators can (1) highlight multiples consecutive sentences as one span and (2) create relations between two spans, while GPT-4 can only highlight single sentences as spans. To adjust for this differ- ence, we treat both consecutive sentences and relations single sentences when computing BooookSCORF. for humans.

---

Published as a conference paper at ICLR 2024
Table 2: BooookScoRE for summaries generated under different configurations; higher scores in- dicate better coherence. We additionally report the average summary length in tokens based on tiktoken
com/ wwwnae/tikt tokenizer, the percentage of novel trigrams compared to the source, and percentage of repeated trigrams in the summary.
Model
Chunk size
BooookSCORE
Avg. length
% novel 3-grams
% rep. 3-grams
Summaries generated via hierarchical merging
GPT-4
2048
89.1
778.6
82,4
4.2
GPT-3.5-Turbo
2048
84.2
667.3
82.8
9.0
Claude 2
2048
91.1
522.6
88.4
1.3
Claude 2
88000
90.3
551.5
87.1
2.0
Mixtral-8x7B
2048
81.5
679.1
85.9
4.1
LLaMA2-7B-Inst
2048
72.4
684.9
76.4
36.1
Summaries generated via incremental updating
GPT-4
2048
82.5
805.4
84.1
3.4
GPT-3.5-Turbo
2048
67.0
484.5
68.2
3.5
Claude 2
2048
78.6
657.1
89.4
1.9
Claude 2
88000
90.9
493.7
84.7
1.9
Mixtral-8x7B
2048
64.5
558.7
82.3
3.5
GPT- summaries generated via incremental updating and hierarchical merging, respectively, while using LLM annotations yields BooookSCORE of 82.4 and 90.8. Figure 4 compares the error distributions from GPT-4 to those of human annotators and shows that GPT-4 is more sensitive to omission errors and less sensitive to duplication or language errors. Taken as whole, these results confirm that BooookScORE is reliable annotator of coherence for book-length summarization. While we implement BooOoKSCORE with GPT-4 for the remainder of this paper, implementing BOOOOKSCORE with open-source LLM annotators is an exciting future direction.
SYSTEMATIC EVALUATION OF LLMS
Armed with BOOOOKSCORE, we investigate the impact of several critical implementation deci- sions on summary coherence, including the choice of prompting strategy, base LLM, and chunk size. Overall, Claude produces the most coherent summaries measured by BOOOOKSCORE, followed closely by GPT-4 and distantly by GPT-3.5-Ture Turbo, Mixtral-8x7BB -877, and LLaMA2-7B-Ine GPT-4's summaries are significantly longer and more detailed than the others across both prompting strategies. The rest of this section drills down into finer-graines grained results.
however,
Experimental setup: Table 2 contains results for five instruction-tuned LLMs: GPT-4, GPT-3.5- and
Turbo, Claude 2, Mixtral-8x77 the chunk size to 2048, maximum summary length G, to 900, decoding temperature to 0.5 and p 1 for ancestral sampling!" [5| To avoid confounds, we use identical prompts for all models except LLaMA2-7B-BRE which only functions with a simpler prompt. LLM API costs for our experiments were $10K USD (Table more experimental details are in Appendix
[3] Unless otherwise specified, we set
Incremental summaries are almost always less coherent than their hierarchical counterparts. Hicrarchical summaries generally have higher BooOOKSCORE than incremental summaries, likely because the incremental updating task requires the base LLMs to follow more complex instructions
1BGPT-4 configurations in this table are not comparable the ones analyzed in Section since we had reduce chunk size and summary length due LLaMA2-7B-Inst and GPT-3.5-Turbo's smaller context size.
"Claude 2 is the only exception, as we use its default temperature of 1. We use temperature of for compression, which improves adherence the max summary length.

---

Published as a conference paper at ICLR 2024
(e.g., deciding what to include from the current book chunk, what to discard from the summary, whether to restructure the summary, etc.). While hierarchical summarization potentially drops long- range dependencies, its instructions are generally simpler (summarize merge).
Incremental summarization benefits from increased chunk size. The one exception to the above result is Claude 2 with a chunk size of 88K, whose incremental configuration produces slightly more coherent summaries than the hierarchical version (90.9 vs. 90.3 BOOOOKSCORE). In contrast, using Claude 2 for incremental summarization with a chunk size of 2048 results a Boo0OKSCORE of 78.6, so clearly the model benefits from fewer updating and compression steps. We do not ob- serve similar behavior with hierarchical summaries, which suggests that hierarchical book-length summarization preferred for smaller context models.
LLaMA 2 struggles on book-length summarization while Mixtral shows promising per- achieves by far the worst hierarchical BooookSCORE of any model. Its summaries also contain significant repetition (% of repeated tri- grams), which is a critical coherence error. Furthermore, we could not get the LLaMA-2-7B-nnsmn checkpoint to perform incremental updating at all, as it just copied text from the chunks until it reached the summary length limit, at which point it failed to follow the compression instruction. On the positive side, Mixtral-8x7B, another open-source LLM, outperforms LLaMA-2-7B-Instn by substantial margin, though it still trails behind most of the closed-soures models. Nonethe- less, it is encouraging to note that with performances closely matching that of GPT-3.5-Turbo on both summarization approaches, Mixtral-8xx 8x7B signals the narrowing gap between open- source and closed-source models.
formance, Table 2 shows that LLaMA-2-7B--nsst
High coherence does not necessarily correlate with human preferences. How well do coher- ence measurements from BOOOOkSCORE correlate with coarse-grained human preferences? We conduct another human evaluation study with the same four annotators in which we solicit prefer- ence judgments on pairs of GPT-4 generated incremental and hierarchical summaries T6 As shown in Table[4] incremental summaries are almost always preferred over hierarchical summaries in terms of level of detail (83% vs. 11%) However, hierarchical summaries are preferred for better structure (59% vs. 35%), logical consistency (53% vs 38%), and overall (54% vs. 44%). When forming their overall preference, some annotators preferred the higher level of detail of incremental summaries at the expense of coherence; thus, both strategies can be viable depending on the needs of the user.
Qualitative analysis: Appendix L contains summaries generated from Janika Oz's A History of Burning, which tells a multi-generrtionna story about an Indian family living in Uganda. We observe that both GPT-4 and GPT-3.5- T-rbo tend to generate oft-repetitive and vague sentences within their summaries (e.g., The story highlights the resilience and determination of the characters they nav- igate the complexities of life, love, and identity across generations and continents)). Such artifacts rarely produced by the 88K chunk size version of Claude 2, which instead omits key informa- tion present in the beginning or middle of the input (e.g.. the entire story of the first generation in the book) in favor of focusing on the end of the book, following the findings of Liu et al.] (2023a). All configurations make faithfulness errors: for example, in A History of Burning, the mother of the character Hari is incorrectly identified as Rajni by Claude 2, while GPT-4 does describe Hari's parentage correctly at one point in the summary but incorrectly at another. We show in Appendix I that automatic quality metrics such as BLANC (Vasilyev et al. 2020) and SUPERT (Gao et al., 2020) are inadequate for book-length summarization.
LIMITATIONS
Our error taxonomy is derived just from errors made by GPT-4, We decided to conduct our human evaluations in Section on summaries produced by GPT-4 for two reasons: (1) wanted our error taxonomy to focus on errors that are actually made by state-of-the-art LLMs (unlike e.g., fluency errors present in SNaC); and (2) human evaluation is very costly, so we could not evaluate many different LLMs on our annotation budget. Similarly, we implement BooookScoRE using GPT-4 as a base LLM, which may have some systematic biases that could be alleviated by using a pool of LLM annotators as in AlpacaEval (Dubois et al. 2023).
''Echh annotator compared 25 disjoint pairs of summaries, and we paid $15 per task for total of $1.5K. To prevent bias, we shuffle the ordering of incremental and hierarchical summaries for each summary pair, and conceal the summarization method of each summary.
8

---

Published as a conference paper at ICLR 2024
BooooKSCORE can be expensive to run. Since computing BOOOOkSCORE requires iterating through a summary sentence by sentence using GPT-4, it can be expensive and slow especially given that the annotation prompt is long (see Appendix M.4). We did experiment with approach that asked GPT-4 to annotate errors in the entire summary at once, but the generated annotations would often include too many trivial questions, and alignment with human judgments was low. That said, despite the API costs of GPT-4 and the relatively slow time to evaluate one summary, BoOOOKSCORE still significant cheaper and faster than performing human evaluations.
Boo00kScoRE does not account for the relative importance of different error types. Unlike similar evaluation frameworks such as MQM (Freitag al. 202I). we choose not assign severity weights to different error types. Nowadays, powerful LLMs rarely make errors related to grammar, which can be objectively defined. For other error types like those in taxonomy, the notion of assigning relative importance is ill-defined. Furthermore, prior work (Goyal et al. [022a]] Dou et al.) 2022) shows low recall between human annotations for NLG evaluation, which indicates that error type severity is subjective as annotators often do not highlight issues that others may find critical.
No validation of recall. Due to the expense, we do not collect overlapping annotations for each summary during human evaluation. Since the annotation task involves subjectivity, overlapping annotations can help ensure that all errors within summary can be captured. However, recent work (Krishna et al., (023) shows that a comprehensive annotation of all information units is not required to produce useful aggregate score that can be used to rank different models.
RELATED WORK
Book-length narrative summarization:
Most prior forg-form summarization work still focuses
on documents shorter than 10K tokens (Cohan et al. 2018 Kornilova & Eidelman 2019 Wang et al.]
2022). BookSum (Krycciskii et al., 2022) is the first published summarization dataset that includes book-level source text as part of their data, which encouraged modeling efforts in this direction (Wul
al. 2021 Xiong al. 2022 Pang et al. 2023 Cao & Wang] 2023 Pu et al. 2023a).
Fine-grained evaluation of generated text: Our work relates evaluation protocols within ma-
chine translation that annotate spans, eπror types, and error severities (Freitag et al., 2021 Fernandes 2023). which are more meaningful than output ranking and Likert ratings. Also related is ACU (Liu et al. 2023c), an annotation protocol for summary salience evaluation that breaks sum- maries down into fine- mne-grained content units, FactScore (Min et al. (023), which dissects machine- generated text into atomic facts before evaluating their factual consistency, LongEval (Krishna al.) 2023), which includes an in-depth analysis best practices for faithfulness evaluation in long-form summarization coherence evaluation, and SNaC (Goyal al. 2022a). coherence error taxonomy built for fine-tuned summarization models.
al.
Automatic evaluation with LLMs: LLM evaluators have recently emerged as cost-effective al- ternative to human evaluations, explored for both general conversational and instruction following capabilities (Dubois et al., 2023 Zheng al. (023)) and traditional NLG tasks like summariza- tion (Fu et al. 2023 Liu et al. [202bb] Wang et al. (023). These latter studies substantiate LLMs' potential as an NLG metric, but only for evaluating short input-output pairs. In our work, we use GPT-4 to evaluate book-length summaries, uniquely employing a fine- -nainnd automatic evaluation schema to set our work apart from existing research.
8 CONCLUSION
Our work presents the first systematic study of book-length summarization using LLMs. We es- tablish novel human evaluation protocol assess summary coherence on newly-published books. Then, we develop an LLM-based automatic metric called BooooKSCORE that relies on a coherence error taxonomy derived from our human annotations. Using BooookSCORE allows us evaluate various prompting strategies and model choices, revealing insights such as: hierarchical merging produces more coherent summaries but may lack detail compared to incremental updating; and in- creasing chunk size can significantly improve incremental updating. Interesting future directions include automatically evaluating faithfulness in the book-length summarization setting, benchmark- ing newer cong-context LLMs using BOOOOXSCORE. and expanding BoOOOkSCORE multilingual texts. We release our BooOoKSCORE metric and annotated summaries to enable meaningful progress in beok-length summarization.

---

Published as a conference paper at ICLR 2024
9 ACKNOWLEDGMENTS
We extend special gratitude to members from the UMass NLP lab for participating in the pilot study and offering valuable feedback, and the Upwork annotators for their hard work. This project was partially supported by awards IIS-2202506 and IIS-2046248 from the National Science Foundation (NSF) well as an award from Open Philanthropy. We also thank the NSF's CloudBank program for supporting the majority of our LLM API-based experiments.
REFERENCES
Griffin Adams, Alexander Fabbri, Faisal Ladhak, Eric Lehman, and Noémie Elhadad. From sparse to dense: Gpt-4 summarization with chain of density prompting, 2023.
Shuyang Cao and Lu Wang. Awesome: Gpu memory long document summarization using memory mechanism and global salient content, 2023.
Kent K. Chang. Mackenzie Cramer, Sandeep Soni, and David Bamman. Speak, memory: An ar- chaeology of books known to chatgpt/gpt-4, 2023.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Scokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-awe awaree attention model for abstractive summarization of long docu- ments. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Pupers), pp- 615-621, New Orleans, Louisiana, June 2018. Association for Computational I Linguistics. doi: 10.18653/v 1/N18-2977 URL http:: http:p::///aaaaaoooooooooo//////
Yao Dou, Maxwell Forbes, Rik Koncel-Kedzionss Noah A. Smith, and Yejin Choi. Is gpt-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text, 2022.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
Alexander Fabbri, Wojciech Kryšcinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. arXiv preprint arXiv:2007.12226, 2020.
Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André F. T. Martins, Graham Neubig, Ankush Garg, Jonathan H. Clark, Markus Freitag, and Orhan Firat. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation, 2023.
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. Experts. errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460-1474, 2021. URL httpttttt///claattttt
doi: 10. 10.1166//acl
wwt.::yw.rr..0.o....... tacl-1.81 :ttp/2221...all111....
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate you desire. arXiv preprint ar 30.iv:2302.044 2023.
Yang Gao, Wei Zhao, and Steffen Eger. SUPERT: Towards new frontiers unsupervised eval-
uation metries for multi-document summarization. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 1347-1354, Online, July 2020. As- URL https:
sociation for Computational Linguistics. doi: 10. 10.18653/1122220acll //aclanthologe org ptgg/220..c....oo acl-main. 124
Tanya Goyal and Greg Durrett. Annotating and modeling fine-grained factuality in summariza- tion. In Proceedings of the 2021 Conference of the North American Chapter the Association for Computational linguistics: Human Language Technologies, pp. 1449-1462, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/200
naacl-main. 114. URL
https://aclantho..oom wwwph/oooooooooooooooooooeooeere 2021
1aa22.naaacl.. naacl ma maiin. 1.n..111.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. summarization, 2022a.
Snac: Coherence error detection for narrative
10

---

Published as a conference paper at ICLR 2024
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News Summarization and Evaluation in the Era of GPT-3. arXiv preprint arXiv:2209. 12356, 2022b.
Wei-Jen Ko, Te-yuan Chen, Yiyan Huang, Greg Durrett, and Junyi Jessy Li. Inquisitive question generation for high level text comprehension. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 6544 6555, Online, November 2020. Association for Computational Linguistics. doi: 10. 56553/v1/222/ wwwp:.0..mpmmmii..55.. URL,
lttpss:/aaclantho log wrg.//2g//2 ee022.0emmmnn.. ammnp-ma inmaiin... 530
Anastassia Kornilova and Vladimir Eidelman. BillSum: A corpus for automatic summarization of US legislation. In Proceedings of the 2nd Workshop on New Frontiery in Summarization, pp- 48-56, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D1995444 U.L.httss://ae antho canthology.... wwggyorgg/1999500
Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit lyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. Longeval: Guidelines for human evaluation of faithfulness in long-form summarization. In European Chapter of the Association for Computational Linguistics, 2023.
Wojeiech Kryseinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. BOOKSUM: A collection of datasets for long-form narrative summarization. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pp. 6536-6558. Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. orgp:///ooooy.g/222200
doi: 10. 18653/v1/2022. http://22...nnnnngnnnn....... URL
Findinggs-emmlp..4 488
Chin- Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: http//acaaattoooooooooooo wrg./rgggg.ooo www./////////o00/0
Nelson F. Liu. Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023a.
arXiv:2307.03172.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023b.
Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien- Sheng Wu, Caiming Xiong, and Dragomir Radev. Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers). pp. 4140-4170, Toronto, Canada, July 2023c. Association for Computational Linguis- tics. doi: 10.18853/v 1/2633/112023aaclloooo acl-long.2 228
URL https://aclanthhlle org/2023.
Yan Meng, Liangming Pan, Yixin Cao, and Min- Yen Kan. FollowupQG: Towards information- secking follow-up question generation. In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi (eds.), Proceedings of the 13th International
Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pucific Chapter of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 252- 271, Nusa Dua, Bali, November 2023. Association for Computational Linguistics. URL https:
paclanthology www.oogg..rggg2200
jen 1p main..7
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation, 2023.
Benjamin Newman, Luca Soldaini, Raymond Fok, Arman Cohan, and Kyle Lo. A question an- swering framework for decontextualizing user-facing snippets from scientific documents. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing, pp. 3194-3212, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v www.0022..mmmmmamaiaaaa https://aci antth waccholooyy.oo ogy. ogy. emw.m.mmmllppmmmmmmmm 193
URL

---

Published as a conference paper at ICLR 2024
Bo Pang, Erik Nijkamp. Wojeiech Kryšcińssi, Silvio Savarese, Yingbo Zhou, and Caiming Xiong- Long document summarization with top-down and bottom-up inference. In Findings of the Asso- ciation for Computational Linguistics: EACL 2023, pp. 1237-1254, 2023.
Donggi Pu, Yifan Wang, and Vera Demberg. Incorporating distributions of discourse structure for long document abstractive summarization, 2023a.
Xiao Pu, Mingqi Gao, and Xiaojun Wan. Summarization is (almost) dead, 2023b.
Oleg Vasilyev, Vedant Dhamidharra,, and John Bohannon. Fill in the blanc: Human- quality estimation of document summaries, 2020.
Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel R. Bowman. SQuAL- ITY: Building long-document summarization dataset the hard way. arXiv preprint 2205. 1465, 2022.
Jiaan Wang. Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. chatgpt good nlg evaluator? a preliminary study, arXiv preprint arXiv:2303. 04048, 2023.
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Chris- tiano. Recursively summarizing books with human feedback, 2021.
Yating Wu, William Sheffield, Kyle Mahowald, and Junyi Jessy Li. Elaborative simplification as implicit questions under discussion, 2023.
Wenhan Xiong, Anchit Gupta, Shubham Toshniwal, Yashar Mehdad, and Wen tau Yih. Adapting pretrained text-to-text models for long text sequences, 2022.
Zheng Zhao, Shay B. Cohen, and Bonnie Webber. Reducing quantity hallucinations in abstrac- tive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2237-2249, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653//1/2020.ff http:///22..nnnnnnemem.......... Findings-emmlp..20
URL httph:t/taclllaaaee httt:s://acllanhhloogy.. org/2020.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with in-bencch and chatbot arena. arXiv preprint arXiv:2306. 05685, 2023.
12

---

Published as a conference paper at ICLR 2024
A DETAILS ON THE TWO PROMPTING STRATEGIES
Assume LLM with context window size W is used to summarize input document D whose length > W. We thus split D into non-overlapping chunks c1, ...... 9 where C' < W the length of each chunk.
A.1 HIERARCHICAL MERGING
Hierarchical merging works follows:
1. Obtain summaries at the base level = 0 by summarizing each chunk.
2. Obtain summaries for the first level by prompting the L.M to merge many consec- utive level-0 summaries 84, s/+1.... as possible such that the total length of the merging prompt, the selected summaries, and the prior context (if there exists preceding summary at the same level) is less than W G₁, where G; is hyperparameter controlling summary length that varies depending on the level
3. Repeat the previous step recursively until we are left with single summary for the book.
A.2 INCREMENTAL UPDATING
Incremental updating works follows:
1. Feed the summarization prompt into the LLM along with the first chunk to obtain summary of the first chunk, which initializes the global summary 91
2. Now, provide the LLM with the updating prompt, the next chunk c2, and the current global summary 91. The model prompted updating the global summary to ge with information from the current chunk.
3. Iterate through the remaining chunks. If g, exceeds the maximum summary length G call the compression prompt to compress yi to fit within the length limit 18 See Appendix A.2. for more details on compression.
A.2.1 COMPRESSION
The compression step is required for incremental updating. Through experimentation, we have observed that as the model processes a book through incremental updating, consistently adds more information to the running summary instead of removing things. Even with updating prompt, the summary often surpasses the target length as removing content from is not the model's ' natural inclination. Thus, separate prompt is needed for the model to condense the summary. However, in hierarchical summarization, condensing is not required. The merging step is less likely to run over the summary limit since does not have work with a pre-existing running summary. If the summaries generated during hierarchical merging over the summary limit, simply asking the model regenerate up to fixed number of times would suffice.
B DATASET DETAILS
B. TABLE OF ALL BOOKS IN THE DATASET
See T
Wu et al. 2202 suggest that since independent chunk-level summarization might miss vital context from earlier sections of the story, can mitigate this effect by joining many preceding summaries from the level as possible. We thus implement this approach our method.
As the final summary often contains artifacts like "in the current segment" or "in this section", especially with weaker models, we included additional post-precsssng prompt to clean up these artifacts. See Ap- pendix M.3
13

---

Published as a conference paper at ICLR 2024
Table 3: Title, author, genres, and publication date of all books in dataset, sorted alphabetically.
Nipe
Suppet
Horrou thmiigg
beeti
(12tel) Stay
Leijh 14 M.Maaka Alcaso
30398506
Aprul Vis
Dosat Royan
Euite (killo
Andi & Sacd Nor How
Jutio Carta Vila
CRAcepepr toki INEREO
Backk00 Clua
Carradee V Wilio
Care Kamuup
Chine Dhinediing All
Noma Kease Agu--tecyss
(S010) 1234.
loa Hee Awar
Crydl Far
Diiaa
Rest
Dincuug
What's Eaqul di Fasele
Calerta: Balinil
15305%
Ulbvn Muri
3623631
Cane Nigh Boo
Latt Alheno
3C15620
(iace Not Lave
Lex
Hogpy
Hovey
Warsee
3030120
Hovwy Dikked Thander
P.li Daals
Co cobeppay folee
[Mlnn-eee new Denition
Rolorrs Velkk
1x:8:12 2. of PNpace
T.1. 33
(REUN (314, (84
36554225
36385544
Lab Tar's Carcoo Monee
Live Sez
Low Wance
Vinor Lafaale
aud Round
Ling lan Haiing
Ca cxieepom)
Pxbequandi: Hinay Workla Eilla Flagus
Janoical
All Owe fark
30216311
Scorde Your
Teatl Duily
3C11077
Samt
1:34.
lra llanting
Trang Thank
(R10%) 1:34 prdt
2611025
Cleyy
Nigh
Harrs
The Achoebocci Amim
The 3u Gowlinday Thipgs
Auadial Maluuu
30125007
Th- Cymmae cemen of Ma
Abolhin Singlowe
The Fectras
Zudia Croi
(eluh Eote acE Felle
2G385722
The Call Whe Full Bo Sea
Addl Di
(aduh Edte cotickgs Aigs ailling
Thc 1611 Noom
Mar lich Sc.ak
3C116722
Thc 33 on
SC11425
The
The Mini
Tradkin
The So
Bally
Th: tapermers
(dec)
Vieaa Year
Day
tokm
36271331
The Waye: Stipercc. Mariy and Nander
Divid Craca
3C15435
Novea
Mag
The Nrah Komas
caiceperry
3013106
The Woray
tecke, myser
WWn 1341 My Mecher
Sauce D.Lall, leacy frkce
3C15420
Tamaroo unl Treeern Tes anve
3028335
Laaele
(entio Eiiee
Yea Woad's Hoai Adris fo: Maddom
Scro Se
cillupppe patgg Ealie myptary datlar
Yulking Prac Sua
Dollil 55
(aduos Esiea beres life of
Vau Hegaeo Nev.
Cart 2ne Scare Norees
cotict prary (amb)) 90hm,, Wiserry (ardury Eore, acE)
3611677
Madc Kig
Maro Vicr
3611530
Yeleder
Co Inepperry latke mysies Euiler
NF 15/2
C COARSE-GRAINED HUMAN EVALUATION
Table 4 shows detailed results of the coarse-grained human evaluation discussed in Section
14

---

Published as a conference paper at ICLR 2024
Table 4: Results from coarse-grained human evaluation. Annotators compare 100 pairs of GPT- 4 generated incremental and hierarchical summaries and judge (1) overall preference; (2) level of detail; (3) structure and pacing: (4) logic and understandability each summary.
Preference
Incremental
Hierarchical
Tie
Overall
44
54
2
Detail
83
11
6
Structure
35
59
Logic
38
53
9
D MORE EXPERIMENT DETAILS
We use the LLaMA-2-7B II-strcuct checkpoin fine-tuned long-contens summarization (Book- Sum). For Mixtral, we used the hosted by the Together API. For closed-source models, the gpt-4 2023-03-11 and 3. 3...b-urrr-o0.. 1 checkpoints on Microsoft Azure. Anthropic unfortunately does not dis- close checkpoint information, but our summaries were all obtained via their Claude 2 API in Septem- ber 2023.
Instruct -v0. checkpoint
In our experiments, LLaMA uses a context window size of 4096 tokens, while other models leverage their full context window. While generating LLaMA 2 hierarchical summaries, we have to truncate the results the final punctuation mark. not, the model would be stuck at the regeneration phase, as it does not follow the given word limit at all. In addition, for LLaMA 2 summaries, we do not apply post-processing as described Appendix [..]] because would significantly alter the structure of the LLaMA 2 summaries, thereby enhancing their coherence. Instead, post-process the summaries using a standard string matching approach to get rid of sentences copied from the prompt. Without this step, we cannot evaluate them with GPT-4, since GPT-4 would treat these prompt artifacts as instructions rather than sentences to annotate.
D.1 DATA PROCESSING DETAILS
In order to preserve all visual separators within the text, extract all text elements from the epub files without further automatic processing. As a result, sometimes content from non-narrative sec- tions would appear in the generated summaries. We apply simple post-processing by prompting GPT-4 to remove information coming from non-narrative sections of the book. The prompt can be found in Appendix M.3
E EXPERIMENTS WITH SQUALITY
To investigate the effect of incremental updating and hierarchical merging on summary coherence, we evaluated GPT-4 on the validation set of the SQuALITY dataset, which contains sci-fi stories that are 4000-6000 words long. 5able[5)show the ROUGE-L scores. The baseline setting is where the model summarizes the stories in one go, truncating the stories whenever they exceed the model's context window. Using incremental updating lowers ROUGE-L by small amount, indicating that baseline summaries have slightly more overlap with the provided reference summaries. We will update the BooookScoRE of these summaries in the next version of the paper.
Instruct
15

---

Published as a conference paper at ICLR 2024
Table 5: ROUGE-L of summaries generated by GPT-4 on the SQuALITY validation under two settings: baseline and incremental updating.
Chunk size
ROUGE-L
Boseline
14.9
Incremental updating
13.2
512
13.7
E. QUALITATIVE ANALYSIS
We show the baseline, incremental, and human summaries for Venus Is a Man's World from SQuAL- ITY in T The incremental summary was generated by GPT-4 with chunk size of 512.
The baseline summary provides the most coherent and comprehensive overview of the story's key events and themes. It clearly introduces the main characters Ford, Evelyn, and Butt and their backgrounds, while succinctly summarizing the plot including Ford's ' friendship with Butt, Evelyn's 's discovery and confrontation, and Butt's unconventional marriage proposal. The summary highlights central conflicts related to gender roles and norms. In contrast, the incremental summary, while touching on similar plot points and themes, does so in more disjointed, less cohesive manner. Details about the setting, dunging drug harvesting, and weaponry feel redundant. Finally, the human summary lacks clarity in many parts, with confusing references to the Male Desuffrage Act and inconsistent character details. It focuses heavily the early parts of the plot at the expense of later key events.
Overall, our finding that GPT-4 summaries can be better than human summaries aligns with results from recent work on LLM-based short-document summarization (Pu et al.) 2023b). The qualitative analysis shows that methods like incremental updating could introduce coherence errors into LLM- generated summaries.
F EFFECT OF SUMMARY LENGTH ON BOOOOKSCORE
0.9
0.8
BOODDKSCORE
0.6
0.5
550
600
650
700
750
800
850
900
Length (tokens)
Figure 3: BOoOOKSCORE vs. summary length.
16

---

Published as a conference paper at ICLR 2024
Table Summaries of William Tenn's Venus Is a Man's 's World.
Haseline
Incremental updating
Human
Venus Is Man's s World is a story about a young boy named Ferdinand, or "Ford," and his sister Evelyn, who traveling
In the stary. sabling Ford and Evelyn from an underwater city accompany women seoking huskonds on a journcy Vanus. Ford befriends stowaway, Butt Lee Brown, who has a trou- bled past. They bond over their life stories as educates Ford about Venusian culture and weaponry. Despite being outsiders, they travel
Ferrlinand is on a spaceship with his elder sis- ter. The ship jam-packed with females ing Verus scarch huskund and its the cabin while the women are still in their hammocks. The women at his times have all rights the Male Desuffrage Act, so the boy admitted to the ship on behalf of his sister. He explores empply ship
spaceslip called the Eleance Rousevelt to the planet Venus. Eve- lyn is seeking a hushand on Venus, as there is shortage of men Earth factors. Ford, curious and adventurous, ex- plores the ship and stumbles upon stowaway named Bult Lee Brown, Venusian farmer had gotten into trouble on Earth and now hid- ing in a lifeboar.
other
important things since
Evelyn convinces Ford attend geography lecture, where she becomes deeply eugagod in the discussion of illegal dunging drag harvesz- ing the Macro Continent. Ford's know lodge of dunging and the Venusian lerm "Paddllefoot" piques Evelyn's 's interest. Ford argaes that Vena- slans pioncers who need strong, indepen- dent men for building a civilization on the now planet, thus intriguing Evelyn.
portholes and after some hesitation enters a forbidden arca. There he looks at the stars und then tries open lock
Food Butt form hand, with teaching Ford about Venusian culture, songs, and even politics. Ford starts to admine Butt's 's strong. independent nature and hegins to question the Earth's is femnlo-dominatedd socicty. However. when Evelyn discovers Ford's is frsendship with Butt, she confronts the stowaway, accusing him of cornupting her brother's 's morals and breaking the law by stowing away on the ship.
lifebout. huge scary with a blaster and frighteningly cold gaze. Fer- dinand explains that he comes from Undersea. Farth, and tells his family story- his parents being one of the first married couples in Undersea and dying a while ago. leading to sister's decision stranger, Butt, tells about the lack of w women on Venus and his travel to Earth in search a wille without any iden "it's woman's world". So got in trouble with the and stowed mway on this ship. His many brothers were killed in rising and only
appears
Food takes Evelyn mest Butt Lee Brown. making her promise not to turn him Upon mecting, Evelyn confronts Butt about his ille- gal activities and moral delinguency. Bott dis- misses her concerns, sparking dehite hetween them about Earth and Venusian socicties. Dur- ing their conversation, Butt proposes to Eve- lyn, stating that they have similar goals in finding a spouse and that she possesses quali- ties he finds desirable. Evelyn is taken aback the proposal, courages her to accept.
migrute
Venas. The
Evelyn's accusations, pro- poses marriage her, stating that they are hoth on the same journey to find partace and that she has the qualities he is looking for in wife. Ford excitelly urges his sister accept Butt's proposal, but the story leaves the outcome of this unconventioaal proposal unresolved. The story highlights the clash of gender roles, so- cictal norms, and the challenges faced by in- dividuals sesking love and companionship rapidly changing world.
Butt, unfazed
left. From that day Fertlnannd keeps visiting stowaway bringing fruits and listening to stories about Venus. Bull tesches the boy to use the blaster without giving it hold and constuntly
her brother excitedly en-
about Evelyn, the sister. Oues, Ferdirand attends a poography lecture on the with his sister and lecturer about Venusian geogruphy, Evelyn starts eliciting where the boy learned that and the boy tells about real men working Venus. gets angry with those masculine ideas doess't helieve them to come from a little bay. Ferdisand trics to lie but Sis suppresses him into confession
The story continues explore the churacters' relationships
differing perspectives the new planct, as well as the potential conse- quenees of their actions and decisions.
leads her
She sells
Butt about all the lows he has broken while the least respords with an appeal to sense. Sud- denly, Butt simply proposes a mutually bene- ficial marriage to stop the debute.
To explore the impact of summary length on BOOOOKSCORE, we plot BOOOOKSCORE vs. length in Figure the 100 summaries generated by GPT-4 with incremental updating using a chunk size of 2048. The plot indicates that there is no discernible correlation between length and BoROOKScORF.
G BOOTSTRAPPING ANALYSIS OF BooookSCORE
To check the stability of the BooOOKSCORE metric, we ran a bootstrapping experiment. Given the BooookSCORE of 100 summaries generated by GPT-4 using incremental updating with a chunk size of 2048, randomly sample 1000 times with replacement using sample size of 100, then compute the mean of these 1000 samples. The standard deviation of these samples is 0.015, which suggests that SoOR is consistent and reliable across multiple random samples.
17

---

Published as a conference paper at ICLR 2024
H LABEL-WISE ALIGNMENT BETWEEN GPT-4 AND HUMAN ANNOTATIONS
Hierarchical Marging
Incremental Updating
Human GPT-
Human GPT-4
0.68
0.10
0.64
0.00
0.00
0.62
0.04
0.00
eveent preresionn
Blcrettnulty
Incomussteeess
dustiiatiinn
languagg
0.00
ereect seriiaacr
Caootal amionian
@icenttmaaity
tateeooce
Rnconsisterrcy
Muttliation
lanquaiinn
trstsiion
amomiian
Figure 4: Distribution of all error types for incremental and hierarchical summaries generated by GPT-4, normalized by the total number of sentences in the summaries.
In our approach to GPT-4 automatic evaluation, we incorporate error type prediction into the prompt to assist the model in making reasoned judgments. We analyze the label-wise correlation between GPT-4 and human annotations, presenting the results in Figure Despite the high agreement be- tween precision discussed in Section 4.1 GPT-4's error distributions vary significantly from those of human annotations. GPT-4 demonstrates a strong propensity for labeling omission errors, occa- sionally applying these labels to sentences that could be ferent error types, In this study, we have not delved deeper into the eтror types predicted by GPT-4 given the observed inaccuracy. Further analysis of GPT-4's capability to precisely predict error types remains a potential area for future research.
appropriately categorized under dif-
I EFFECTIVENESS OF EXISTING REFERENCE-FREI EVALUATION METRICS
Table 7: Results from BLANC and SUPERT on all hierarchical summaries.
BLANC
SUPERT
GPT-4 (4096)
0.0248
0.3627
GPT-4 (2048)
0.0221
0.3615
GPT-3.5-Turbo (2048)
0.0203
0,3658
Claude 2 (88000)
0.0171
0.349
Claude 2 (2048)
0.0167
0.3559
Mixtral-8x7B (2048)
0.0169
0,3836
ILLMA2-7BBInst (2048)
0.013
0.3446
We investigate how well existing reference-free evaluation metrics work for our setting where the source text is at book level. BLANC (Vasilyev et al., 2020) measures how helpful summary is to understanding the source document by testing if a pre-trained model can better fill in masked words given access to the summary. SUPERT (Gao et al. 2020) measures the semantic similarity between the a summary and some pseudo reference summaries generated from the source document. We compute both metrics for hierarchical summaries generated under all configurations, and show results in Table 7 Both BLANC and SUPERT differ by nearly negligible margins across models, which makes them less meaningful. Furthermore, both metries rate Claude 2 (88000) summaries as inferior to GPT-3.5-Turbo (2048) summaries, a finding that clearly contradicts results from our qualitative analysis.
18

---

Published as a conference paper at ICLR 2024
We do not compute ROUGE (Lin 2004 it has been shown to have significantly low human correlation when applied to summaries generated by GPT-3 (Goyal et al. 2022b).
J COMPUTING ANNOTATION PRECISION
In this human evaluation task, given summary and annotation (span-question pair) for the sum- mary, we ask annotators whether they agree, partially agree, or disagree with the annotation. We provide the following standards for determining agreement:
1. The span is confusing,
2. The questions are:
(a) Relevant to the span.
(b) Not answered anywhere the summary, explicitly or implicitly.
(c) Addressing issues that left unresolved, would make the summary incoherent or make it hard for readers understand the main storyline.
Agreement would mean both the span and the questions satisfy the standards. Disagreement would imply that the span is not at all confusing, regardless of the questions. Partial agreement would apply when the span is confusing, but the questions fail to meet one more of the standards.
We mix human and GPT-4 annotations the data we present to the annotators, concealing the origin of these annotations to prevent any potential bias. Each annotator responsible for the human and GPT-4 annotations of 25 books (different from the 25 books which they annotated summaries for). We paid $1.7 per summary and $0.15 per annotation. 100 summarics and 1659 annotations resulted in total cost of $418.85 (USD). Note that annotators were informed of this evaluation task weeks after they completed the first task (where they were asked to highlight spans and ask questions given book summaries). Thus, monetary reward for the second task could not have biased annotators to produce as many annotations as possible for the first task.
When computing precision, we count all cases of agreement and partial agreement.
K
API COSTS
We present estimation of the API costs of experiments in Table g
19

---

Published as a conference paper at ICLR 2024
Table 8: API cost estimates in USD.
Model
Chunk Size
Summarization
BoooOkSCORE Eval
Summaries generated via hierarchical merging
GPT-4
4096
580.3
566
GPT-4
2048
628.3
394.1
GPT-3.5-Turo
2048
29.5
379.8
Claude
2048
267.2
379.5
Claude 2
88000
227
378.9
Mixtral-8x7B
2048
460.5
LLaMA2-7B-Inst
2048
538.5
Summaries generated via incremental updating
GPT-4
4096
786.7
797.5
GPT-4
2048
873.2
536.8
GPT-3.5-Turbo
2048
69.7
318.4
Claude 2
2048
329.6
356.6
Claude 2
88000
229
518.7
Mixtral 8x7B
2048
395.7
Total
4020.5
6021.2
20

---

Published as a conference paper at ICLR 2024
L EXAMPLE SUMMARIES
Table 9: GPT-4 4096 summaries of Janika Oz's A History of Burning.
Stralogy
Summary
Hier
A History of Hurning follows the lives of an Indian family living in U ganna during the late 19th and early 2ith cennariss, experiencing political tammoil, displacement, and personal struggles The story begins with Pirblai, a young buy from Gujorat, Indis, who lured into indentured servitude and forced haild millways for the British East Africa. eventually marries Sonal, they settle Ugands, having named Vinod.
Vinud grows up fooeiig the pressure to provide for his family and takes a job at tea-colfoe trading mamin He mamriss Rajui, a woman from Kunschi, and they have rwo daughters named atkka and Mayori. I Liika grows in Uganda,
diserimination and violence during Amin's reign, whike Mayuri decides study medicine Bumbay, India, kaving the family behind. As the politiaal sitstion Uganda worsens, the fumily subjocted to the "cattle count" consus. Despite these hundships, the family remains resillent and determined to survive. Latika becomes involved in political activism, writing for a subversive newspuper called Jicho, and falls in love with Arun. charismatic stadent activist who boards with her family. However, Arun becomes disillasioned with his work. and their reluiiossipp becomes strained. husband Kariuki distribute her wrilings licho.
Latika colists the help Ieene, African wonan who woeks for Arun's family.
The two women form a boad based on their matual struggle for justice and cquality
Years laterr Vined, Rajm, and their family ar living in Toronto, Canada, after being forced to leave Uganda under kJi Amin's is rulc. They struggle adapt their lives, maintain connections their homeland. and navigate the complexities of their relationships Rajui works as nanny for a woman named and daughter Dahliz, while Vined woeks low fow-paying job. Their daughter Kiya has become distant and wilhimmwn, whille Mayuri is now married Kunal, a man she barely knows.
Mayuri returns to Taronto after years of living in Bombay, where she had studied medicine and formed a relationship with a woman named Ruhi. She has martied Kunal to be able and environmens compared to sister Kiya is strained, as Kiya is resentful of Mayuris ss absence and her sudden return.
her family. Upen arriving Canada, Mayuri struck by the differences in the weather She is surprised by how much family changed over the years. Mayun's is's relationship with
One day, Kiya receives letter from Adroa, her former lover, revealing that her long lost sister Latika may still be alive in Uganda, The letter brings back memones and emotions for Kiya, who struggles to process the information. around her. Kiya decides to share news her father, As the family tries to cope with the revelation. Vinod and Rajni decide to move into the spartment above the convenience store owned by their late friend Pulinhhai and his wife Nainahhen, leaving their house to their children. Hari, who has grown sullen and restless, is grappling with the family's
she contemplates the her family life continues devastated the revolation.
feelings ahandomennt and anger Rajai and Vined trying reach out Hari help him throngh his pain.
Despic the turmoil within the family, they continue to support each other, with
In the following years, Hari becomes involved in a procest against racism. which takes violent turn as the police artempt to disperse the crowd. Vinod's 's health deserioentes, het he finds solace in the thoughi of spending time with his family in their gunden, Kiya and Mayuri receive a phone call informing them that Hari miscing join their parents searching for him. they search, they reflect their experiences and sacrifices their family has made a better
The story then shifts to an older woman and a with separation and loss, and they find solace each other's is company. They speak of fires old and new, and the woman shares tale from her post.
man standing by lake, discussing their shared history and experiences. Roth have dealt
losses, they Throughour the story, charcters gepple with the challenges of living in a foreign land, maintaining their amhitions and careers, movigning the completities of their relationships with euch other and the larger political events happening around them. Despite the obstacles they face, they remain determined to carve out meuningtil lile for themselves and continue their light for better future. The novel explores themes family, sepuration, and the importance inderstrrding one's is history and lineage achieve greater understanding
recalls
logether, acknowladging the cyele life
importance knowing
history
personal
identity, The Family's history of burnig and rehuilding surves as a intaphor for thoir rovilionce and determimation to ercate a betler foture.
Inc
In 1898, Indian boy Pirbhal is tncked into working on a ratlway in Mombusa, Kenya He linds shelier with Sonal a family in Kisumu and het. starting family. Sonal gifled heaker. and their son Vinod strugples to balance his Indian betiigg with his friendshpps Inspired his uncle mama, Vinod stays Ugunda. joining a tea coffee trading company.
Pirbhai arranges Vinod's 's marriage to Rajni in Karachi. Rajui apprehensive about leaving her horne but her parents see as an opportunity to escape the violence of Partition. Rajni marries Vined and gives bitth to a daughter. The couple faces challenges together and grow closer, finding strength their growing family. When the third granshugheer is Nanu support despite initial tension. Pirbhais life neurs end, he shares memoties and wisdom with Latika, emphasizing resilience their family's is history.
visits, Beeing violence Kenya. They
The family takes in Arun, university student and paying guest. He grows closer to Latika, the aldest granddanghter. Arun offers her tour violent hur remaints active alongside Arun. They marry
the campus, exciring the household. Latika joins him in political mily which Arun work distribute writings
despite parental disapprowal and move to linja. Ugaand. Latila Mayuri, Latika's sister. encouraged to study abroad and attends a Navratri celebration where she finds a connection with girl who teaches her to dance with dandlyns, leading her to stdy medicine in Bomboy. Her younger sister, Kiyn. hecomes close to Adroa, a young man woeking shop. The family goes camp for Asions has an encounter with hamssing soldier. Adrca later reveals At Mayuri's is goodhye dinner, tensions arise between Kiya's is family and Adroa. Kiya's mother calls her a "veshya" causing Kiya to hreak down. Adraa leaves, and Kiya finds solace in her sistens' embrace. Vinod encounters soldier named Moses and is warned to get his kipande soon. Vinod stapped soldiers who force him and other Arun receives refugee card, giving them seventy- two hours to leave Uganda. discovers letter from the goverament ordering termina- tion of Latika's 's nowspoper licho, but decides not to confront her about it. Their time in Uganda runs out, and Latika stays behind to search for Armn who was taken
unity and resistance.
but forced restart line. Kiyn feels lonely Adroa appears distant and been asked to join Amin's is army, causing rift between them.
the ground,
soldier recognizes him
Good and him go.
by armed men. She entrists her child, Harital, to Rajni, na leave Lganna..
The family adjusts to life Toronto, coping with the cold weather and cultural differences. Rajai becomes friends with neighhor lane, while Mayuri returns to Taronto to reunite with her family. Harihal is placed in a special class for gifted students, forming ai friendshhp with a buy nomed Solomon. Kiyn hecomes concerned about their class and investigates the siturionn.
house and excited about the prospect of now beginnin.. accident occurs while Harilal helping Vinod point the house, Harilal falls olf ladder is rushted to hospical with a broken ankle. Vinad Rajai reflect on their roles as purents and the impretance fariily and faith.
Vined becomes interosted in buying house seeking stability. He visits
Rajni neesives a phone call from Kantabhen, Arun's 's mothser revealing that Latika is alive and living in London, Kantabben wants to give back the property that her hushand had transfereed to Arun. Rajai and Vinodl docide tell Hari the truth about his real porents. Hari angry and family straggles come to terms with the revolation
betrayed by their deccit.
confronts the challenges presents.
Hari spands his days playing basketbull with his friends. including Solommn, who is strugeling with his job and family iosucs. Hari reflects on his family's is past and The phone number that Ma had given him to contact Laiika. As he and his friends continue to play basketball,, they notice that the hoops have been removed from the court. symbolizing the loss of their family has made give
and the challenges they face.
considers the implications of reaching oul to Laiika and the sucrifioss t importance of family and the power of fargiveness. choosing to cubbrac the truth and the complexities of his relationships with his parents. sastens, and friends As the family continues to nav navigate the chillenges of thair past. present and future, they find strength in their connections and the resilience that has been passed down through generations. and Solomon confront Han'' 's relationship with his family grows stronger as he recognizes the thnead of trust, love, and time that has held them togother. Rajni's perspactive on her family's as history and strogglas evalves, and she begins to understand the importance of bringing the forward. The family stays strong. supporting another through life's dificuuliiss they continue to navigate the balance between their beriagge and their new Toranto.
chance a betier He realises
the
of their buketball sanctuary and rally together. deciding
they were
could only together.
21

---

Published as a conference paper at ICLR 2024
Table 10: GPT-4 2048 summaries of Janika Oz's A History of Burning.
Strategy
Summary
Hier
"A History of Burning" by Janika Oza follows an Indian family living in Ugunda through multiple generations as they navigate hardships, sacritices, and cultural identity struggles. The story begins with Pirhhai, 13-year-old boy from Gujarat, India, who travels to Africa in search of work during time of drought and hardship. He eventually marries Sonal, a young gidl living with her family in Keaya, and they move to Kampula, Uganda, where they have a son numed Vinod.
Vinod helps elevate his family's ' status in Uganda and marries Rajni, year-old girl from Karachi, amidst rising tensions between Hindus and Muslims. They raise two daughters. Latika and Mayuri, while facing challenges in their relationship and adapting to their new surroundings. Throughout the story, the characters face hardships, sacrifices, and the struggle to maintain connections to their homeland while navigating the challenges of colonialism, migration, and cultural identity.
Years later, Latika grows up and becomes involved in activism against colonial rule Africa, writing for a secret network called Jicho. She marries Arun, fellow activist. but struggles with her new life living with his family and the expectations placed upon her. Mcanwhile, her sister Mayuri navigates a newly integrated school in Uganda, facing racism discrimination while considering her future. Eventually, Mayuri studies medicine in India and later returns to her family in the U.S., accompanied by her partner Kunal.
The family laces challenges as political changes in Uganda, the Mau Mau rebellion in Kenya, and the lingering effects British colonialism impact their lives. Latika secretly writes for an anti-govemment newspaper. JICHO, and her busband Arun is kidnupped due to her involvement. Mayuri is suppised by the changes in her family, particulurly her parents' aging and her younger siblings, Kiya and Hari. She learss that Hari, the son of her deceased sister I Latika and her husband Arun, unaware of true parentage.
As the family memhers struggle to maintain connections to their homeland and each other, they experience loss, love, and resilience in the face of adversity. Throughout the story, the characters find strength and hope their shared experiences and love one unother, as they try build better life for themselves and their families. The story also explores the challenges faced by immigrants as they adapt to new environments, such as Mayun's 's struggle to practice medicine in Canada and the family's 's experiences with racism and violence.
In the end. the family must come together to face the destruction of their shop and the loss of their past, finding solace in euch other und the hope for better future. Through their shured experiences and love for one another, they find strength and hope to endure the challenges that come their way. The story highlights the resilience and determination the characters they navigate the complexities of life, love, and identity across generations and continents.
Inc
Indian immigrant Pirbhai moves his family from Kenyn to U Uganda, where his son Vinod marriss Rujni from Pakistan. They have three daughters, Latika, Mayuri, and Kiya, and face hardships under General ldi Amin's 's rule. Forced to leave Uganda, they settle Toronto, where Vinod and Rajni face cultural challenges. Mayuri marries Kunal but struggles to adupt and returns to her family when her medical degree is unrecognized. Kiya becomes unexpectedly prognant and faces shame and isolation. The story alternates between Kiya and Mayuri's perspectives in Toronto and Latika's 's life in London.
Vinod and Rajni bay house in Toronto, providing stahility for their family. They find sense of helonging and hope but still worry about their daughteess Latika struggles in London, gaining insight into her past through a chance encounter with woman from Ugunda. Encouraged by her mother-in-law, Latika considers contacting her family. Back in Toronto, Rajni learns that their soap factory in Uganda may be returned to them under reputriation scheme. She also discovers that Latika visited Uganda, shocking the family with the revelation that she alive. The family confronts the secrets they's ve kept over the years.
Rajni tells her brother Harilal the truth about Latika and their parents' past as political activists. The family dynamic becomes strained as Harilal reacts with anger and shock. Vinod and Rajni move into an apartment above a friend's convenience
give their children space. They find purpose and develop routine, while Harilal's attempts reconnect are met with resistance.
Mayuri. now living in her family's 's house, tries to maintain normaley but struggles to connect with Harilal after revealing the truth about Latika. She befriends Darlene, Trinidadian caretaker who shares her experiences of displacement and finding a new home. Through gurdening and connecting with neighbors, they find solace and sense of community. As the family confronts their past, Kiya tries to hold them together and find forgiveness. They find healing and un- derstanding by sharing stories, exploring themes of family bonds, secrets, and adaptation. Mayuri's 'ss son Hari struggles with the family's 's secrets and changes in their relationships. He finds solace in playing basketball with friends who understand his feelings of displacement.
Hari's basketball court taken away, symbolizing his struggle for belonging and justice. His family tries to reconnect with him, and he begins to understand their silence as a means of survival. Hari leurns about protest against racial injustice and police hrutality and decides to participate. The protest becomes chaotic, and Hari separated from fricads and blinded by tear gas.
As the chaos unfolds, Rajni and Vinod search for Hari The family reunites and witnesses their convenience store hurning down, hut they resolve to rebuild and support one another. They come terms with their past and find strength in their shared experiences. such as gardening and connecting with their community.
In the epilogue, Latika and her hushand reflect on their lives and the stories they' 've shared. They find solace in their connection and the knowledge that what the water takes. returss. The family's joumey is marked by resilience. forgiveness, and the power of community to heul and rebuild.
The novel also explores the importance of knowing one's x history and family lineage to achieve a greater understanding personal identity. Each family member faces moments where they must decide between complicity and resistance, and their family legacy influences their decisions. The novel emphasizes the power of storytelling and the intercon- nectedness of the characters as they navigate their relationships, secrets, and the challenges they face. Food plays a significant role the narrative, symbolizing community. belonging, and the characters' connection their cultural ronts. At the heart of the novel is the idea of community and collectivism, as well as the question of who is included and excluded in these spaces. The author hopes readers will reflect on these themes and the possibilities of solidarity. the charucters' fates are entangled and they rely on one another.
22

---

Published as a conference paper at ICLR 2024
Table 11: GPT-3.5-Turbo 2048 summaries of Janika Oz's A History of Burning.
Strategy
Summary
Hier
"A History of Burning" by Janika Oza is mull-generational story that follows the lives of Pirbhai, Sonal, and their children Vinod, Rajni, Mayuri, Kiya, Meetu, and Hari through displacement, cultural struggles, and personal tragedies. The novel is set during political unrest Kenya and Uganda in the mid- mid-twentieth century, where the family faces poverty and exploitation us migrant workers. Pirbbai leaves his family in India to work in Kenya, where he faces poverty and exploitation as a migrant worker.
haunted by the guilt of leaving family behind and is forced make difficult choices, including burning down cluster of ahandoned huts. Pirhhai marries Sonal, and they start family, facing financial struggles and the impact of colonialism.
The story explores themes poverty, desperation. and the harsh realities of migrant workers. The family faces political unrest and racial tensions in U Uganda during the time of the country's independence from British rule. They are forced to leave Uganda due to the political instubility caused by ldi Amin's 's regime. The story also delves into Mayun'' 's struggles medical professional facing discrimination Toronto, The characters are motivated by their desire protect their families and survive in dangerous euvironment, with themes of family, resilience, and the challenges faced by immigrants in udupting to new cultures.
The story follows their children, Vinod and Rajni, their striggles with identity, cultural expectations, and displace- ment. The family's 's relationships und dynamics are explored, and their motivations and fears are hinted at. The family lives in Uganda during the time of the country's 's independence from British rule, and the story explores themes of grief, family, and tradition. Pirbhai and Sonal celebrate the birth their third granddaughter, Kiya, and plan a picnic the Botanical Gardens in Entebbe. The family faces political unrest and racial tensions, with Pirbhai's 's grunddaughter Latika hecoming involved in resistance movement against tyranny. The story features non non-linear narratives and flashbacks, with Pirbhai reflecting The novel shifts hetween different characters' experiences, flashhacks, and tragedies, providing rich and layered narra- tive. Specifically, the characters' past traumas, including the disappearance of Latika, add complexity to their struggles and sense of loss, while moments of triumph and fear reveal the human capucity for resilience and transformation. Beyond the family's 's personal struggles, the novel delves into themes such as community inclusivity, belonging, and solidarity.
past and consequences his actions.
Despite the family's struggles, the story presents moments of hope and triumph. such as Vinod's dream of buying a house, and moments of uncertainty and fear, such as Har''s fall from a ladder. Ultimately, the story presents a nuanced and emotionally resonant portrait immigrant life and the human capacity for resilience and transformation. The characters experience moments of hope and trimphh such as Vinod's dream of buying a house, and moments of uncertainty and fear, such as Hari's fall from a ladder.
The setting is in Uganda, Kenya, and Canada, and the story highlights the importance of family, resilience. and the challenges faced by immigrants in adupting to cultures. The churacters are motivated by their desire to protect their families and survive in dangerous environment. The story presents a consistent and chromological narrative, despite featuring non-linear narrutives, flashbacks, and switches between altemate worlds or viewpoints.
In conclusion "A History of Burning" is a complex and emotionally resonant story that explores themes of family, resilience, displacement, and the challenges faced by immigrants in adapting to new cultures. The story follows the lives of Pirbhai, Sonal, and their children through political unrest, racial tensions, and personal tragedies. Despite the family's 's struggles, the story presents moments of hope and triumph, ultimately presenting nuanced and layered portrait of immigrant life and the human capacity for resilience and transformation.
Inc
A History of Burning follows the lives of nine family members over a century, exploring the importance of knowing one's 's history and family lineage to achieve greater understanding of personal identity. The novel emphasizes the significance of community and belonging. while also considering who is included and excluded in these spaces. Food plays a pivotal role placemaking and finding one's sss way into new understandings of home and belonging. The protagonist of the novel the family itself, and the choice of whose voice to enter for specific moments often intuitive.
Vinod's s health is declining. and he suffers a stroke. His family worries about him, and Rajni struggles with memories of the past, including the secret of her lost daughter, Latika. Mayuri and Kiya tell Hari and Meetu about their lost sister. and they all cat cake together. Hari confides in Sol about his boss uccusing him of stealing tips, and they start playing baskethall together.
While Hari marches in a protest with Solomon, Marcus, and Sami, the protest becomes violent, and Hari is blinded by scaring fog. He stumbles on, tring to feel his way unsecing over whose streets. Hari yells out the names of his family. have come and gone, and sees visions boat dimly bobbing across black waters. The chemical fog hangs like white sheet in the night sky, and Hari opens his eyes to see the destruction around him. He runs home, and his family is relieved to see him. Later, Vinod and Rajni meet with woman of people remote and unknowable as the land before the water. The woman has traveled far, and yet all the movement. a part of her feels restored. Encircled by her family, she knows that she hus come not just forward, but also back. When their conversation falters, they take tide crest over their feet and remember what the water has carried and what it has swallowed. The man slips a little on a rock slick with algae, and the woman rights him in a gesture unpracticed and natural. They recall losses that they cannot know. Beyond them, the tide continues. The water shifts the reassembles the shells, sucks the stones to pebbles. The waves break and mend, break and mend.
body huddled in a dank chumber, a tree blooming red as flame, and a
meky outcropping off the shore. They recognize each other and speak
stillness amund them, how waves make silence with sound. They let
23

---

Published as a conference paper at ICLR 2024
Table 12: Claude 2048 summaries of Janika Oz's A History of Burning.
Strategy
Summary
Hier
The novel follows an Indian tamily across multiple generations, beginning in late 19th century India when the patriarch Pirhhai heards ship Africa seeking work. Tricked into indentured labor in Africa, Pirbhai endures difficult condi- tions building the Uganda Railway. Years later in 1902, his granddaughter Sonal notices the strong ars of an Indian laborer, Pirbhai, who comes asking for work at her father's is shop in Nairobi. Sonal develops an interest in Pirbhai, who remains quiet about his traumatic past.
The story shifts to 1917, where Sonal is now married woman with two daughters, living Kampala, Uganda. Her husband Pibhai insists on sending money to his aging mother in India, while Sonal resents having another child amidst financial struggles. the 1940s, their daughter-ing law Rajni secretly poisons husband Vinod to make him ill for military conscription during World War II, afraid he will die in combar.
Years later, another descendant, Latiku marries local activist named Arun, against her purents' wishes. During political turmoil the 1960s, Arun is imprisoned sedition. Latika flees Uganda for London, abandoning their infant son. Decades later, Latika's sister Mayuri arranges her son Hari's marriage woman named Rajui. keeping secret that Rajni is actually Latika's duughter-in-law: In the 1970s, President Idi Amin expels Asians from Uganda, forcing the family to scatter across continents. They lose
home. honds, and history in the upheaval. In 1980s Canada, Rajni and Vinod struggle to assimilate while hiding the truth about Hari's purentuge. Huri feels disconnected from his family's 's silenced history. He grapples with fighting racism alongside friends versus fullilling his duties their shop and ill grandfather. When Hari finally learns about mother Latika, The family ruptures further when Latika my mysteriously returns after 30 years, devastating Rajni.
furious over the lies.
They continue navigating the legacy of I Latika's 's unexplained ahsence and the cultural erasure caused by their exile from Uganda, But glimmers of reconciliation emerge, as when Rajui and Vinod open a new shop providing community. Later, while caring for their hedridden grandfather, Hari realizes his family's 's perseverance outweighs the hidden past. Ultimately, Latika's 's return signifies the possibility of mending fissures left by displacement and secrecy. Spanning a tumultuous century, the epic novel conveys fumily's 's resilience and quest for home and belonging amidst upheaval. Rajui cared for her siling husband Vinod after his stroke. Their adopted son Hari moved back in to help, though Rajni still felt distant from him ufter revealing his birth mother Latiku was alive. Rajni heard warnings about potential violence at upcoming protest rally, let Hari decide to attend and support marginalized community.
Inc
Rajni reflected on her family's 's complex history and relationships. She considered her daughters Kiya and Mayuri, who had tried their own ways to heal family rifts. Rajui realized she had once been close with Hari and longed for that again. She thought of the connections between her family's 's struggles and those of other marginalized groups like Hari's friends, Rajni guined perspective on the importunce of open communication to bring people together across divides. Meanwhile, Vinod slowly recovered, regaining mohility and strength. He woke up determined to become more inde- pendent. Vinod and Rajui both acknowledged their growing frailty. but continued supporting each other. Vinod's daughters Kiya and Mayuri tended community garden, bringing together neighbors and friends. Though facing opposition, Kiya was inspired by her ancestor's vision of shared place welcoming to all. She hoped the garden could be purt of creating inclusive home.
I. Latika had told them the story of how their grandfather Vinay ripped up homes and displaced communities when building the Ugandan railway. Vinod now reflected on his father's sss complex legacy and motivations. Despite their troubled relationship. he emputhized with Vinay's desire for security and belonging. Vinod realized himself had fought hard to establish a home in Canada, sometimes neglecting to appreciate what he had. Now physically weakened but guining perspective, Vinod hoped find contentment.
Hari went to the protest rally downtown with his friend Solomon and others. As the peacefiul protest ended, tensions rose between protesters and police. Chaos broke out, with violence erppinn on both sides. Police barricaded the protesters, using tear gas and force to disperse them. In the confusion. Hari lost sight of Solomon and the others. Blinded by the teur gas, Hari desperately tried to find his way and reconnect with his friends during the fray.
Rajni and Vinod went searching for Hari during the chaotic protest. In their confusion amidst the clashes, they struggled to find their way home. Fortunately their daughter Meetu arrived and helped guide them back. reuniting with the rest of the family. They seturned home to find their shop had been destroyed in a fire during the fray. Though devastated, the family drew strength from one another. committed to rebuilding together.
24

---

Published as a conference paper at ICLR 2024
Table 13: Claude 88000 summaries of Janika Oz's A History of Burning.
Strategy Hier
Summary
In 1972 in Uganda, Latika und Arun, an Indian couple with baby son named Harilal, receive a mysterious letter delivered by a young Ugandan soldier. country under dictatorship Idi Amin, who has ordered all Asians to leave Uganda within 90 days. Latika hides the letter from Arun and reads alone, discovering is an order from Amin to inmeddateey stop publishing their underground new spaper Jicho, which has been printing anti-govrmmment articles. Latika has secretly kept the paper going against Arin's 's wishes, as he argues endangers their family while she believes justice is more important than their personal safety.
Latika hides the letter out of fear of Arun's 's reaction and thrill at the power it places in her hands, as the soldier assumed Arun was in charge,
realizing she runs Jicho, Her secrecy over the paper and letter signals a rupture trust between them, as Arun wants to protect his family but Latika is singleminded in her activism. sparking confliet in their marriage. With the expulsion order, this clash between family and justice becomes increasingly fraught.
The story then jumps ahead 1992 in Canada, where Rajui Vinod are searching night for their missing son Hari after a protest rally tumed violent. They find their daughters Mayuri, Kiya and granddaughter Meetu waiting. but Hari is still missing. Earlier that day, Hari went to the rully despite his family's 's concern.. His friend Solomon was also there. When police arrived, tear gas was released protesters fed as confirmtations hrnke out. Hari Salomon chuos.
Unable to find Hari, Rajni realizes they should go home, as home is where her family is, not any physical place. Just then, Mayuri spots them and brings them their shop. which has been set on fire during the unrest. As the family watches the burning shop, Hari appears, having made his back alone. Angry they are just standing there, he wants to try saving the shop, hut Rajni says calmly it's done; they know how to rebuild. Vinod agrees, saying now is the time rest.
The fire destmys the shop hut leaves the ground ready for something new. Hari realizes his family has persevered, almost whole. Though they will have to start over with the shop gone, they will insist on something better next time rather than accept injustice. Rajni reflects how they have migrated from India Africa to Canada, realizing any wnywhere can feel like home if her family is tngether. After so much loss, they made here united. While the shop is destroyed, their family remains strong.
summary, the story spans 20 years and families, exploring how broader politicul conflicts intertwine with and impuct family re relationships. In 1970s Uganda, rising political tensions struin the marriage of activists Latika and Arun as they clash over family versus justice. Two decades later in 1990s Canada, the family of Rajni and Vinod draw closer amidst violent unrest, losing their shop but keeping their bond intact. Despite different places and contexts, both families experience the complex dynamics between individual ideals, family ties, and turbalent political realities. Ultimately they share the resilient hope that as long as their family together, they have arrived home.
Inc
1972. Latika and her husband Arun live in Kampala, Uganda with theu baby son Harilal. Latik runs an underground new newspaper called Jicho with her friend Danicl, who has recently disappeared likely due to being critical of dictator Idi Amin's 's regime. On the momning of August 4th, Amin declares over the radio that Asians must leave Uganda within 90 days. Latika's family is away studying medicine in India. Latika's 'sk father Vinod goes to pray at the temple and sees a dead hody in the river, then finds a copy of Jicho which he destroys out of fear.
shock, trying to understand they will be impacted Ugandan citizens. Her sister Mayuri
The family spends days waiting endless lines try to get paperwork to leave, selling possessions, and being humili- ated hy soldiers. Latika's 's parents are unable to secure clearance to leave yet. One day, soldiers line and shoot Asians on the street. but Vinod is spared because he has red mark on his forehead from praying. He sees his childhood friend Moses, now soldier, who lets him gn.
Meanwhile, a soldier delivers a letter to Latika ordering her to cease publication of Jicho, but she hides it from Arun. As the deadline to leave looms, Latika reflects on her tense relationship with Arun over her activism and thinks about why she chose him. She resolves continue publishing the hanned paper secret, seeing it the purpose of her life. On the night of September 9th, the eve of the expulsion deadline, soldiers break into Latika and Arun's 's apartment. They beat and arrest Arun while Latika screams and clings to their baby Harilal. The next day, Latika's parents and sister Kiya come to collect Harilal so they can flee the country together. But despite their pleas, Latika sefuses to leave Kumpala without Arun, cntrusting them with caring for her son. Heartbroken. the family leaves Latika behind as they make their way to airport and eventually resettle as refugees in Canada.
Now alone in Kampala, Latika descends into despair searching for Arun. She publishes final defiant issue of Jicho before her supplies run out. An old friend named Adroa visits and is shocked to find Latika still there. Seeing the toll it has taken on her, he encourages Latika to leave Uganda and make new life elsewhere. Latika finally resolves to so, boarding plane to London with the other remaining Asian refugees.
25

---

Published as a conference paper at ICLR 2024
Table 14: Mixtral 2048 summaries of Janika Oz's A History of Burning.
Strategy Hier
Summary
" History of Burming" by Jania Oza is an intricate, muligennrutional saga that explores themes of intergenerational trauma, resilience, identity, colonial exploitation across time, spaces, and continents. The narrative spans from Jamnagar, Gujarar, to Kenya, Uganda, and Canada, weaving logether the lives of several characters whose experiences intersect und diverge amidst political uphenval, personal sacrifices, and cultural expectations.
The story begins in Jumnagar during a severe drought, where thirten-yyearoldd Pirbhai struggles to find work. Driven his mother's encouragement, he embarks on joumey Jamnagar and meets mysterious merchant who leads him to work on dhow sailing towards an unknown destination. Meanwhile, in Kenya, Sonal is a young Indian woman living with her family. When Pirbhai, now refugee Kenya, is employed they navigate their lives logether amidst political turmoil and personal sacrifices.
Sonal's father, the form a deep bond
Pirbhai's 's pust unveils his harrowing experiences as un indentured laborer in East African colonies and the guilt he feels over the circumstances brothers who perished during migration, and her struggle forge connections with her family and her new home. The nurrative intertwines with Rajni's is story during her migration from Uganda to Canada, following Idi Amin's expul- sion of Asians from the country. Rajni, a resilient woman, grapples with new
led him Kenya. Simultaneously, the narrative explores Sonal's losses, including
cold and foreign country while mouming the loss of her daughter, Latika. The lives of these churactery continue to intersect and divenge in Toronto, revealing a rich tapestry of experiences, hopes, and triumphs despite the adversities they encounter.
Other characters play significant roles shaping the narrative. Vinod. Indian immigrant, navigates cultural assimi- lation, identity, and fumily expectations in Canada. Kiya. Rajni's daughter, deuls with her resentment towards her sister Mayuri, who left the family during their time of need. Hari, the Patels' son, and his friend Solomon grapple with the consequences of their actions during protest against racism.
Through the characters' experiences, the narrative explores the intricate connections hetween historical events and their impact on the characters' lives. From the struggles of poverty in Jannagar, the brutal labor conditions in East African colonies, the racial tensions and political turmoil in Uganda, the story emphasizes the power of resilience, human connections, and the complex nature of identity forged amidst shifting political alliances and dislocations.
In Toronto, Kiya joins the army. and the family grieves after Latika's 's disappearance. Hari and Solomon confront the consequences of their actions during a protest against racism. The non-lineur storytelling features past and present flashhucks, providing rich context for the characters' experiences. Despite the non-lineur narrative, the story ultimately presents consistent and chronological account of the characters' lives and their struggles to forge a sense of belonging amidst adversity.
In conclusion, "A History of Burning" is a compelling exploration of characters' lives woven together throughout history, emphasizing the profound power resilience, human connections, and the complex nature of identity. By enguging with themes such colonial exploitation, community, survival, and identity, the story forges rich and imaginative world that touches on the lives of Pirbhai, Sonal, Rajni, Vinod, Kiya, Mayuri, and others, as they navigate across continents and navigate an intricate web of cultural expectations, personal sacrifices, and political upheaval.
Inc
The story follows the Pirbhai family, whose roots lie in India but have ties Kenyn, Ugunda, and now Canada. The family includes Rajni and Vinod, their sons Hari and the late Arun, daughter Mayuri, Rajni's 's uncle Nanu, reemerged sister Kiya, and her son Harilal. In 1991, Hari bonds with friends Marcus, Solomon, and Sami over shared experiences and basketball. He grapples with Arun's 's death and the revelation of long-lost sister. Latika. The family unites dis- cuss Latika and commemorate Arun. Hari investigates the Ugandan Asian expulsion hut disenvers limited information. 1992, Hari turns 19 and forms a close friendship with Marcus, Solomon. and Sami. Ignited by the Rodney King beating and the deaths of Raymond Lawrence and Michael Wade Lawson, he joins community protests, driven by his pursuit of justice and skepticism of official narratives. Together, they restore a vandalized local basketball court. Hari's reflections on family dynamics lead him to play basketbull with Solomon instead of completing a report.
Parallelly, Rajni navigates Vinod's 's deteriorating health while embracing his desire for privacy. Together, they face his physical decline. Rajni's 's past reemerges while she raises her concerns for Hari's safety during protests. She understands the importance of supporting their causes despite her fears and apprehensions.
s At the demonstration. Hari advocates for justice and unity. When lensions escalate, he gets separated from his friends. Rajai turns to prayer for Hari's i safety, prompted by reminiscences of her pust. Upon discovering Hari's disappearance. the family unites in searching for him.
After the protest, disoriented and lost, Hari reconnects with his friends, who assist him in returning home. Aware of Hari's safety, the family recenvenes to make sense of the day's events, solidifying their unity and commitment to fighting for justice.
The narrative intertwines Rajni Hari, and their friendships as they confront their histories, find comfort in unity, and sustuin their roots amid political strife. Acclaimed writers like Sharon Bala, Ngig) wa Thiong's o, and Guiutra Bahadur enrich the storyline with dedicated support, historical context, and scholarship. Mentors and writing programs bolstered Hari's belief in his writing abilities.
The Pirbhai family's iss story spans generations and continents, with each member experiencing their own trials and tri- umphs. They carry the weight their family legacy remains united, finding strength in their shared experiences and their commitment to fighting for justice and equality. The story emphasizes the importance of community, collectivism, and solidarity, inviting readers to reflect on who is included and excluded their own spheres.
history
ancestors. Through all, the family
26

---

Published as a conference paper at ICLR 2024
Table 15: LLaMA 2048 summaries of Janika Oz's A History of Burning.
Stratggy Fierr
Summary
In 1898, young boy namedl Pirbhai leaves his home in India in search of work. He meets merchant who offers him a job beat heading to an unknown destination. Along with several other men, Pirbiul emburks the jomne. In the lase nineseenth century, Pathaa leaves india to and work on a ship bound for Affica. Once abourd, he meets Jamcel. fellow countryman. and together they endure the hardshipe of life sea, Alter several weeks, the ship arrives Mannhass, Kosya and Pibliti and Junuel railway line through the jungle. Pirchui singgles adjust life of the nature his empleyment Despie these chal challenges he finds solace
those who disembark. They we takes to nauby nillway
work tuiikiing where he plagued by memories of his family back home and the guilt of being
where they are
company fellow workers and the simple pleasures
1956, Pibhlai Rakesh work coolies
under Briich
down shandened hus coustrysdz, 1956, day thidd granälaggher Kiya they lind pectaiii collected magpie droppings.
tisked
Publiii feels contlicted about participuting such
Elizabeth
They
way
local in
Mayuti, young Lgandan womann receives
search
in late
offers century. Pistiai India heading an unknowa destination. Along Affica Chce
emitakks the jouracy. Juncel. felkow countryman, logether they andure handshipe lamool are among these who disembark. They are taken nearty railony camp. where they are put Pirtbai strupples to adjust to in the camp, where he plagued by memocies his family back home and the pullt being unowure of the true nature of his employment. Despite these challenges, he finds solace the company of his fellow worker and the simple pleasures of life the camp. ia 1956, Pibitai and Rakesh work is coolies in India undar the Britsh Empac. They are tasked wath barning down abandoned huts is the coustrysde, and Pirhiii feels confliened about porticipering in sach vielenor. In 1956, the day their third grancliuggher Kign was been, they find a portriit of Queen Elirabeth vosteere in the street They were on their way to make donttion at the local Asion school, whose moi was rusted and collectid magnie droppings. Mayuri, a young Ugannaan woman, recetves call from her friend Kiya in 1990.
other
sea, After several weeks, the ship ship Mombxsa, Keuva,
meets building milleyy through jungle
Pualiu leaves ludia sceuch work in 1898. He meets merchant who offass him job: bost heading unknown destination. Along with several other men. Pufhhai emburks on the jouracy. the late nincteenth century. Pubhai leaves India to find wock ship bound for Affica. Once aboard, meets Jamool, fellew countrmmer, and together they andere the landships life
27

---

Published as a conference paper at ICLR 2024
M PROMPTS
M.1 INCREMENTAL UPDATING
Generate initial summary
Below is the beginning part of a story:
()
We are going over segments of story sequentially to gradually update one comprehensive summary of the entire plot. Write summary for the excerpt provided above, make sure to include vital information related key events, backgrounds, settings, characters, their objectives, and motivations. You must briefly introduce characters, places, and other major elements if they are being mentioned for the first time in the summary. The story may feature non-linear narratives, flashbacks, switches between alternate worlds or viewpoints, etc. Therefore, you should organize the summary so it presents a consistent and chronological narrative. Despite this step-by-step process of updating the summary, you need to create sumnary that seems as though it is written in one go. The summary should roughly contain () words and could include multiple paragraphs.
Summary (1) words):
Generate intermediate summaries
Below ow is a segment. from a story:
()
Be lowow is summary of the story up until this point:
are going over segments story sequentially to gradually update one comprehensive summary of the entire plot. You are required to update the summary to incorporate any new vital information in the current excerpt. This information may relate to key events, backgrounds, settings, characters, their objectives, and motivations. You must briefly introduce characters, places, and other major elements if they are being mentioned for the first time in the summary. The stery may feature non-linear narratives, flashbacks, switches between alternate worlds or viewpoints, etc. Therefore, you should organize the summary
28

---

Published as a conference paper at ICLR 2024
it presents consistent and chronological narrative. Despite this step-by-step process of updating the summary, you need to create a summary that seems as though it is written in one go. The updated summary should roughly contain {} words and could include multiple paragraphs.
Updated summary (i) words)
Compression
Below is a summary of part a story:
()
Currently, this summary contains () words. Your task is to condense it less than (} words. The condensed sunmary should remain clear, overarching, and fluid while being brief. Whenever feasible, maintain details about key events, backgrounds, settings, characters, their objectives, and motivations but express these elements more succinetly. Make sure to provide brief introduction to characters, places, and other major components during their first mention in the condensed summary, Remove insignificant details that do not add much to the overall story line. The story may feature non-linear narratives, flashbacks, switches between alternate worlds viewpoints, etc. Therefore, you should organize the summary so it presents consistent and chronological harrative.
Condensed summary (to be within {} words):
M.2 HIERARCHICAL MERGING
Generate lowest-level summaries
Below is part of a story:
()
are creating one comprehensive summary for the story by recursively merging summaries of its chunks. Now, write sunmary for the excerpt provided above, make sure to include vital information related to key events, backgrounds, settings, characters, their objectives, and motivations. You must briefly introduce characters, places, and other major elements if they are being mentioned for the first time in the summary, The story may feature non-linear narratives, flashbacks, switches between alternate worlds or viewpoints, etc. Therefore, you should organize the summary so it presents consistent and chronological narrative. Despite this recursive merging process, you need to create summary that seems though it is written in go.
29

---

Published as a conference paper at ICLR 2024
The summary must be within words and could include multiple paragraphs.
Summary:
Merge summaries
Below are several summaries consecutive parts of story:
()
We are creating one comprehensive summary for the story by recursively merging summaries of its chunks. Now, merge the given summaries into one single summary, make sure to include vital information related to key events, backgrounds, settings, and motivations. You must briefly
characters, their objectives, introduce characters, places, and other major elements if they are being mentioned for the first time in the summary. The story may feature non-linear narratives, flashbacks, switches between alternate worlds or viewpoints, etc. Therefore, you should organize the summary so it presents a consistent and chronological narrative. Despite this recursive merging process, you need to create a summary that seems as though it written in one go. The sumnary must be within words and could include multiple paragraphs.
Summary:
Merge summaries with prior context
Below is summary of the context preceding some parts of story:
O
Below are several summaries of consecutive parts of a story:
()
We are creating one comprehensive sunmary for the story by recursively merging summaries of its chunks. Now, merge the preceding context and the summaries into one single summary, make sure to include vital information related to key events, backgrounds, settings, characters, their objectives, and motivations, You must briefly introduce characters, places, and other major elements if they are being mentioned for the first time in the summary. The story may feature non-linear narratives, flashbacks, switches between alternate worlds viewpoints,
30

---

Published as a conference paper at ICLR 2024
etc. Therefore, you should organize the summary it presents consistent and chronological narrative. Despite this recursive merging process, you need to create a summary that seems though it is written in one go. The summary must be within () words and could include multiple paragraphs.
Summary:
M.2.1 LLAMA PROMPTS
Generate lowest-level summaries
Below is part of story:
()
Write a coherent and chronological summary for the excerpt provided above. Briefly introduce characters, places, and other major elements if they are being mentioned for the first time in the summary. The summary must be within words and could include multiple paragraphs.
Summary:
Merge summaries
Below are several summaries of consecutive parts of a story:
()
Merge the given summaries into one coherent and chronological summary. Briefly introduce characters, places, and other major elements if they are being mentioned for the first time in the summary. The summary must within (} words and could include multiple paragraphs.
Summary:
Merge summaries with prior context
Below is summary of the context preceding some parts of story:
()
Below are several summaries of consecutive parts of story:
31

---

Published as a conference paper at ICLR 2024
(
Merge the preceding context and the summaries into one coherent and chronological summary. Briefly introduce characters, places, and other major elements if they are being mentioned for the first time in the summazy. The surmary must within () words and could include multiple paragraphs.
Summarys
M.3 ARTIFACT REMOVAL
Below is summary of book:
Your task is edit the book summary by removing any phrases that indicate it was developed progressively. Delete terms such part of the story, " "in the
segment, " "in excerpt, "in the updated summary, and any similar phrases. The goal is to make the summary read as if was written all at once, not in stages. In addition, eliminate any elements taken from non-narrative sections like the table of contents, sekknowledgments.
as "in the
author's biography, author's note, information of the author's s other works, and so on. Apart from these adjustments, do not make any other changes the summary.
M.4 GPT-4 AUTOMATIC EVALUATION
We ellipsis here to keep this prompt concise. The complete version includes two full summaries and 42 sentence-level annotations, and will be made available in our codebase.
Given a book summary and a sentence from that summary, determine if that sentence causes any confusion. Types of confusion include the following:
Entity omission: an entity, real or abstract (person, object, place, concept, etc.) is mentioned, but key details are missing unclear
Event omission: an event is mentioned, but key details are missing or unclear
Causal omission: the reason or motivation for something missing or unclear
Salience: inclusion of trivial details that do not contribute to the main storyline
Discontinuity: an interruption in the flow of the narrative, including but not restricted to: sudden jumps between perspectives, time periods, or settings; poor transition between sentences paragraphs; sentences or paragraphs that seem out of place; illogical sentence order or summary structure Duplication: redundant repetition of similar information
32

---

Published as a conference paper at ICLR 2024
Inconsistency: two parts of the summary contain contradicting information
Language: grammar issues; confusing wording phrasing; etc. For something qualify confusion, it must meet these two conditions:
1. Without resolving the confusion, readers would struggle substantially to grasp the main narrative, or the summary would appear incoherent.
2. The confusion can't be resolved solely using the information provided in the summary.
If the given sentence involves confusion that meets these two qualifications, ask relevant clarifying questions and provide the confusion types. There be multiple questions, and keep in mind that sentence may involve multiple types of confusion. If the given sentence doesn't involve any confusion, simply say "no confusion". Some examples are provided below, followed by the summary and sentence that you need to annotate.
[Example summary 1]
In the small town of Elm Avenue, teenage twins Aida and Salma are inseparable. When Aida mysteriously disappears, Salma recounts their childhood and the search for her sister begins...
Throughout their time together, Sara and Juan explore their pasts, finding unique bond and sense confort in each other's company, as the story moves through its diverse and interconnected characters and settings.
[Example sentence] In the small town of Elm Avenue, teenage twins Aida and Salma are inseparable.
[Example response]
Questions: no confusion
Types: no confusion
[Example sentence]
When Aida mysteriously disappears, Salma recounts their childhood and the search for her sister begins.
[Example response]
Questions: no confusion
Types: ne confusion
[Example sentence]
Meanwhile, the focus shifts new character, Fausto, and his relationship with Paz, as they navigate life in hurricane-ravaged Miami neighborhood.
33

---

Published as a conference paper at ICLR 2024
[Example response] Questions: Why are we suddenly switching to new character? Does Aida have any connections with Fausto? Types: discontinuity
[Example summary ry 2]
[Example sentence]
Proctor Bennett, a ferryman in Prospera, assists people transitioning new lives upon retirement.
[Example response]
Questions: Where or what is Prospera? Types: entity omission
[Example sentence] Struggling with dreams and discontent, his wife Elise suggests a change in his work.
[Example response] Questions: no confusion Types: no confusion
[Example sentence]
Deciding to quit being a ferryman, Proctor teaches Caeli how to swim.
[Example response] Questions: How does quitting being a ferryman relate to teaching a girl how to swim? Types: discontinuity, causal omission
[Your summary]
()
[Your sentence]
[Your response] Determine if it the sentence above involves confusion that can't be clarified using information from any part
34

---

Published as a conference paper at ICLR 2024
of the given summary, and those which, if left unresolved, would make the summary highly incoherent or significantly hinder readers from understanding the main storyline. If you don't identify any confusion like that, simply say "no confusion" for both questions and types in your response.
35


Reviews:
[{"review_id": "OQrhjVRdDU", "reviewer": "Reviewer_rPVA", "summary": "This paper provides a systematic empirical survey of the coherence problem in book-length summarization with LLMs. The contributions include a novel evaluation protocol, an automatic metric for coherence assessment, and a systematic evaluation of different LLMs.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper presents a very systematic survey, with solid and extensive experiments. The writing is clear and the narrative is easy to follow, facilitating understanding of complex concepts. The creation of the BOOOOKSCORE metric is particularly innovative, offering a decent solution to the cost and time constraints of human evaluation.", "weaknesses": "A potential weakness lies in the design of the BOOOOKSCORE metric. The approach of weighing all sentences equally may overlook the varying importance of different parts of the text for overall coherence. Additionally, there is an absence of a consistency check for the evaluation metric.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P4R3sLDxbO", "reviewer": "Reviewer_hJ5Z", "summary": "This paper addresses the problem of book-length summarization evaluation, which refers to documents with 100k+ tokens. It identifies the following main challenges: 1) the problem of data contamination, that is, most books are already in the pre-training data of large language models; 2) the lack reliable automatic metrics. To solve the first issue, the authors collect a dataset set of 100 recently published books. To address the second issue, they generate summaries for each book with GPT-4, and then ask humans to identify and classify coherence errors such as \"entity omission\", \"causal omission\", and \"duplication\". Then, to automate this process, they prompt GPT-4 to classify sentences in summaries according to the same coherence error taxonomy, and they find the precision of GPT-4 is similar to the human annotators. The coherence annotation by GPT-4 is the basis of BooooKScore, a reference-free automatic metric counts the fraction of sentences in summaries that are free from coherence issues. Finally, the paper evaluates summaries generated by GPT-4, Claude 2, and LLama-2-7B using two techniques: hierarchical merging and incremental updating. They find that GPT-4 and Claude 2 summaries are more coherent and that hierarchical merging also results in more coherent summaries, but at a cost of detail.", "questions": "In the description of \"incremental updating\" in section 2, you state that \"One major issue with the hierarchical merging approach is that it necessitates summarizing portions of the input document without complete context ... which can lead to incoherent summaries especially for non-linear or multi-perspective narratives.\" However, your experimental findings show that hierarchical merging results in more coherent summaries. Do you have an explanation for this observation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper addresses an important problem in summarization, proposing solutions to scale the evaluation of very long documents that are longer (for now) than current context sizes of large language models. The effort applied in validating a reference-free automatic metric with newly collected books and human annotation are relevant contributions for future work in summarization.", "weaknesses": "One point I miss in the experiments is a baseline without hierarchical merging or incremental updating. The reason is that a significant fraction of books have length around 100k tokens or less (if we observe the statistics of BookSum, for instance), and it would be interesting to see if the hierarchical merging or incremental updating introduce (or not) a high quantity of coherence issues compared to a vanilla LLM approach. Even with some level of truncation, it should still be possible to assess coherence issues.\n\nMinor issue: you mention in section 3 that \"we did not find summaries online for any books in our dataset, which means LLM memorization of these summaries is impossible.\" Not finding results online by no means imply that memorization is impossible. In fact, we have no guarantee that closed-source models such as GPT-4 are trained just on publicly available data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eAnW0XkzJT", "reviewer": "Reviewer_SAFf", "summary": "This paper tackles the problem of summarization of books. This is an ambitious task (that would have been considered unattainable a few years ago), as well as very expensive (because any annotation or even evaluation is very cognitive intense and long). Very little research is done in that area so far, and most of it is based on automatically scraped material.", "questions": "How did you chose the number of `O` in `BOOOOKSCORE` ?", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "There is much to like in this paper. It is a great idea, and has been very, very well executed.\n\n- The authors obtain 100 recent books from the last year - while reviews of those books exist those are not summaries. I would be less certain that those books or their summaries are _impossible_ to be part of the training corpus. Modern LLM go well beyond only crawling Internet data and providers have deals with many different publishers. Still, this is arguably one of the most rigorous attempts to create a test set that avoids dataset contamination.\n\n- The evaluation focuses on precision. The standard approach of evaluation summaries is to rely on human-written summaries and the compare those with system-generated. This is very expensive, and creates lots of pitfalls as it depends on the similarity metric used to compare summaries (even without using n-gram overlap metrics, learnt embedding metrics like BERTScore have their own set of biases). Instead, this work focuses on assessing a given summary by asking questions about it.\n\n- It compares the two standard ways of summarizing very long documents: hierarchical (summarizing parts and combining those) and incremental (left to right) with interesting insights  (eg, incremental is preferred for level of detail, but not for other aspects)\n\n\nThis work will certainly be considered a fundamental paper in the coming months and years, and be a required reference and reading for summarization research. It jumps right into what is still difficult to do with modern LLMs", "weaknesses": "I have very little concerns about this paper. It is however unfortunate that the evaluation framework did not consider faithfulness. This could have been done by asking annotators to assess each fact and search for it inside the book to verify its factuality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jzs5kaQ0ZW", "reviewer": "Reviewer_vK8q", "summary": "The paper investigates book-length summarization. It proposes an automatic, LLM-based, reference- and source-free metric to evaluate the coherence of a summary. Using this metric, two different book-length summarization techniques are evaluated, and the evaluation is compared to human evaluation.", "questions": "- Table 2: Why is there no entry for LLaMA2 in the incremental update section?\n- Page 7, Section 5: Doesn't the choice of $p=1$ for nucleus sampling disable it completely? The choice of a large temperature also seems rather unusual for experimental evaluation, where a temperature of 0 is often chosen for reproducibility. Considering that Claude used a different temperature than other models, have you investigated how stable the results are when the temperature is varied?\n- Page 7, Section 5, \"Incremental...\" paragraph - shouldn't \"lower\" by \"higher\"?\n- Page 8, top paragraph: \"Highly extractive\" seems to contradict the large proportion of novel trigrams, which would be more indicative of an abstractive summary.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed metric is reference- and source-free, and thus has broad applicability. It also follows recently proposed best practices.\n- To reduce the likelihood that LLMs have seen this data before, the paper used an evaluation dataset based on recently published books.\n- The authors promise to release code and annotations.", "weaknesses": "- The evaluation of human annotations focuses only on precision and does not investigate recall. However, the authors are open about this limitation.\n- Some of the examples of types of errors shown in Table 1 seem to be of questionable quality; for example, the first example about the \"mysterious man\" might as well be unanswered in the book (although I do not know the contents of the book in question). This may be a side effect of offering a monetary reward based in part on the number of annotations (cf. Appendix G).\n- The sentence-level score may disproportionately favor summaries that contain a large number of (short) sentences. Furthermore, the evaluation of different models does not further investigate whether some of the score differences can be explained by the different length of the summaries (e.g., a shorter summary may be more prone to omissions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)