NarrativeXL: a Large-scale Dataset for Long-Term Memory Models
Authors: Arsenii Kirillovich Moskvichev
Ky-Vinh Mai
Venue: EMNLP 2023 Findings (2023)
Keywords: NLP
long-term memory
long term memory
reading comprehension
TL;DR: We propose a large-scale reading comprehension dataset based on full-length novels; having more training items and longer documents than existing alternatives makes our dataset well-suited for training long-term memory NLP models.

Abstract: We propose a new large-scale (nearly a million questions) ultra-long-context (more than 50,000 words average document length) reading comprehension dataset. Using GPT 3.5, we summarized each scene in 1,500 hand-curated fiction books from Project Gutenberg, which resulted in approximately 150 scene-level summaries per book. After that, we created a number of reading comprehension questions based on these summaries, including three types of multiple-choice scene recognition questions, as well as free-form narrative reconstruction questions. With 990,595 total questions, our dataset is an order of magnitude larger than the closest alternatives. Crucially, most questions have a known ``retention demand'', indicating how long-term of a memory is needed to answer them, which should aid long-term memory performance evaluation. We validate our data in four small-scale experiments: one with human labelers, and three with existing language models. We show that our questions 1) adequately represent the source material 2) can be used to diagnose a model's memory capacity 3) are not trivial for modern language models even when the memory demand does not exceed those models' context lengths. Lastly, we provide our code which can be used to further expand the dataset with minimal human labor.

NarrativeXL: a Large-scale Dataset for Long-Tere Term Memory Models
Arseny Moskvichev Santa Fe Institute Santa Fe, NM ar arseny. meny..osskiichee@@mail.... com
Ky-Vinh Mai University of California, Irvine / Irvine, CA
kyvinhmêuci. edu
Abstract
opposed to extrapolating from shorter sequences [Anil et al., 2022] which prone to generaliza- tion errors). In our work, we create such a dataset. We capitalize on recent results showing that for tasks that do not require long-term memory, LLMs rival human labelers [Gilardi et al., 2023]; we use this "local" competence to create massive datset (nearly a million questions in total) of ex- tremely long-term memory problems (average con- text lengths from 54,334 to 87,051 words for dif- ferent question types).
We propose a new large-scale (nearly a mil- lion questions) ultra-long-context (more than 50,000 words average document length) read- ing comprehension dataset. Using GPT 3.5, summarized each scene in 1,500 hand-curated fiction books from Project Gutenberg, which resulted in approximately 150 scene-level sum- maries per book. After that, we created a num- ber of reading comprehension questions based these summaries, including three types of multiple-choice scene recognition questions, well free-form narrative reconstruction questions. With 990,595 total questions, our dataset is an order of magnitude larger than the closest alternatives. Crucially, most questions have known "retention demand"", indicating how long-term of a memory is needed to an- swer them, which should aid long-term mem- ory performance evaluation. We validate our data in four small-scale experiments: one with human labelers, and three with existing lan- guage models. We show that our questions I) adequately represent the source material 2) can be used to diagnose model's memory capacity 3) are not trivial for modern language models even when the memory demand does not ceed those models' context lengths. Lastly, we provide our code which can used to further expand the dataset with minimal human labor.
2 Language Modeling is Not Enough
In theory, optimal next-word prediction requires perfect memory unlimited capacity, hence one might argue that supervised long-term memory datasets are unnecessary, given the abundance of unsupervised data. In practice, there are two consid- erations as to why Language Modeling alone might not be the best approach to train and test language models with extremely long context windows.
First, Language Modeling performance will likely see diminishing returns when the context window is increased. Many documents in popular unsupervised datasets are simply not long enough to benefit from contexts larger than ten thousand words. Additionally, for longer documents (e.g. fiction books), it is likely that remembering the last read chapter two is nearly equivalent, in terms of the next word prediction quality, to remembering all the chapters read so far. is possible that in some narratives, given character or item might reappear after a long absence, but such cases are likely to happen only few times per book, making the task extremely sparse and inefficient in training long-term memory models.
1 Introduction
Although on paper many modern Large Language Models (LLMs) have maximal context lengths mea- sured in tens of thousands of tokens, in practice, they often fail to access information plainly pre- sented within those contexts [Liu et al., 2023] and their performance generally deteriorates as inputs grow larger.'
We believe that this issue is consequence of the lack of supervised datasets that could be used to directly train extremelyy-ong-coonee LLMs (as
Second, language modeling does not offer di- rect way to interpretably measure long-term mem- ory capacity and performance. For example, we do often see improvement in perplexity when the ef- fective context window is increased (e.g. [Dai et al.,
'This pattern, only recently ohserved in the literature, repli- cates on our data (s.f. subsection A.8).

---

2019]), but it is still difficult to measure and under- stand where exactly the improvement comes from and what kind of information is retained. One sce- nario could be that longer context window helps a given model better understand lengthy philosoph- ical treatises present in the dataset, which, in turn, allows the model to extrapolate such arguments in consistent and sound ways, resulting in lower per- plexity. Alternatively, the model might simply be better at populating bibliography sections of such treatises, being able to copy the cited names from the main text into the bibliography using its long context.
In NarrativeQA, the authors employed crowd- source workers to create book and movie script understanding questions based on corresponding web-scraped summaries. While we highly resonate with the importance and motivation of their work, the dataset has few crucial disadvantages.
1) Since all questions in NarrativeQA are writ- ten based on summaries alone, by construction, the dataset can not evaluate any reading comprehen- sion that goes beyond knowing the summary. But, arguably, by reading a book, one gains detailed memories and understanding far exceeding what one could get by simply reading its summary. It seems highly desirable for any long-term reading comprehension dataset to reflect that.
We speculate, therefore, that in order for ex- tremely long-context models to thrive, it is nec- essary to develop specialized supervised datasets that would address the limitations above. Creating such dataset is the main contribution of our paper.
2) The size of the dataset is limited by the num- ber of pre-existing summaries available online. which restricts the dataset approximately 1500 documents. Of those, only 4000 are books, the rest being movie scripts, which are, generally, much shorter (hence the avarage document length Nar- rativeQA is around 50,000, compared to 87,000 in our work). Overall, the Narrative dataset con- tains ~45,000 questions, which is good for evalu- ation, but might not be enough for training long- term memory models. Our dataset offers order of magnitude more questions.
3 Existing datasets
Traditionally, long-term memory transformers were tested either on 1) artificial tasks (e.g. |Tay et al., 2020, Moskvichev and Liu, 2021, Weston et al., 2016]) 2) language modeling (e.g. [Dai al., 2019, Rae al., 2019, Bulatov et al., 2022)).
Training or evaluation on supervised naturalistic long-term datasets is relatively rare. Until recently, creating such datasets in a brute-force manner had been prohibitively expensive, as it required tens of thousands of hours of human labor. There have been, however, creative workarounds taking advan- tage of existing resources. Notable examples in- clude [Cohan al., 2018] which used scientific pa- pers from ArXiv and PubMed and their correspond- ing abstracts to create medium-term summariza- tion data, and BookSUM [Kryšciískii et al., 2021] which scraped WebArchive to find summaries for project Gutenberg books. The SCROLLS dataset [Shaham et al., 2022] aggregated and curated number of such datasets in order to create long- text understanding benchmark. A recent Zero- SCROLLS dataset [Shaham et al., 2023] (concur- rent with our work) expanded SCROLLS with two more sub-datasets tailored towards zero-shot evalu- ation.
3) NarrativeQA does not offer a natural learning progression. All questions are asked at the end of the book/movie script which offers no natural curriculum for the model (c.g. learning to han- dle short retention questions first). In contrast, in NarrativeXL, our 726,803 multiple-choice ques- tions cover a wide variety of memory demands from a few thousand tokens to more than a hundred thousand. Importantly, our dataset also decouples memory demand (how long ago the relevant infor- mation was presented) from total context length (how much context was given in general), which makes it better suited for diagnosing and evaluating memory performance.
We discuss additional, subtler differences be- tween Xa (our work) and NarrativeQA in subsection A.6. Overall, without diminishing the importance of NarrativeQA, we believe that the limitations above warrant the development of new
In this context, it is especially important to dis- cuss the NarrativeQA dataset [Kočiský et al., 2018] since this work is especially close to ours in its goals, scope, and structure?.
from NarrativeQA offer by far the longest memory reten- tion demands: the averuge number of words in NarrativeQA subtask documents is 49,384, while the second-longest subtask offers only 10,839 words per document. Average document length in our data is 87,541, higher than in NarrativeQA since we use full-length books only.
'NarrativeQA is included as one of the subtasks in both SCROLLS and ZeROLCOOL In those datascts, documents

---

Dataset
Avg. Words
Documents 405 books
Task Items
BookSum IKryšciński et al..
110,000
405 book summaries
2021]
NarrativeQA [Kočiský al.,
50,000
1.572 (783 books, 789
46,765 QA pairs
2018]
screenplays) 232 meetings
QMSum [Zhong et al., 2021]
10,839
1,808 query-summary pairs
QuALITY [Pang et al., 2022]
$ 6,000
762 articles
6,737 QA pairs
ASJ [Dangovski et al., 2021]
5,975
50,134 academic papers
50,134 press releases
Table Existing long-range dataset statistics. "Avg. Words" shows the average document length per task item.
Narrative XL question type
Avg. Words
Documents
Task Items
Read along
54,334*
1500 Books
726,803 (multiple choice)
Scene summary reconstruction
87,051
1500 Books
244,111 (freeform)
Hierarchical summary reconstruction
87,051
1500 Books 19,681 (freeform)
Table XarrativeXI. (our contribution) dataset statistics. *Read along questions are asked before the book fully read, reducing effective document length.
long-term reading comprehension dataset. To the best of our knowledge, among naturalis- tic supervised Natural Language datasets, Narra- tiveQA (Kočiský et al., 2018] is the only one com- ing close our work terms of document lengths and the number of training items (s.f. subsec- tion 7.2).
a given work has more than one volume, Project Gutenberg often has duplicate entries, storing both individual volumes and the same volumes merged into single file. Keeping both versions of such books would have led to various dataset contami- nation issucs. Overall, the goal of this stage was to select books that contain a single connected narra- tive and remove irelevam//contaminatnn infor- mation.
General statisties for number of related datasets are provided Table 1. When compared to Narra- tiveXL (see Table 2), can be clearly seen that we offer a combination of context length and dataset size not covered by existing alternatives.
4.2 Summary creation
To obtain scene summaries, books were split into -3,000-symbol chunks (with 300-symbol over- lap), and then each chunk was summarized us- ing the GPT-3.5 API (the code, including prompt details, is provided at https: thub. com/ r-ssny/NarrativexX.)
4 Methodology
Our general approach was to test long-term terg-ttemmmm reading comprehension through book scene reconstruction and recognition. Since we want to encourage flex- ible memory representations rather than verbatim text memorization, instead of complete raw scenes, we used scene summaries. Our overall pipeline is illustrated in Figure 1 and is described below.
Question types
5.1 Read-along questions (multiple-choice)
Most reading comprehension datasets assume that their questions will be asked after the entire docu- ment is processed. In contrast, real-life linguistic activities are more "on-line". For example, one's 's understanding long dialogue or book does not suddenly materialize after the end of the text, rather, one's 's understanding continuously develops as the reading/talking proceeds.
4.1 Data preparation
Raw books were downloaded from Project Guten- berg, with the boilerplate license information re- moved using a script. After that, we manually in- spected each book, to remove 1) books that do not have overarching narrative, such as short story collections, diaries, memoirs, published letters of prominent figures, and so author names, ti- tles, table of contents, dedication, preface, adden- dum, glossary, translator's notes, and similar non- narrative information, 3) duplicate books. When
To capture this property, have constructed large number of "read along" questions that be asked not at the end of the book, but rather at specific times as reading progresses. These ques- tions are multiple-choice, in the form of "In what

---

Hand-Curated Fiction Books
Summarization
Chunk
Summary
Hierarchicul Summary
Split each book into chunks
Chunk
Summary
Hicrarchical Summary
Books are downlosded
Chunk
Summary II
from Project Gisenbarg manually filtered and
Each chank +500 words.
Question Generation
Read-Along Questions
Scene Reconstruc- tion Questions
Figure 1: Data creation pipeline. Note that Read-Along Questions are generated only using summaries, while Scene Reconstruction Questions are generated using both summary and hierarchical summaries.
you've read so far, was there a scene where ...", af- ter which a number of scene summaries are given, along with "None of the above" option.
provide richer learning signal to the model. We, therefore, added "summary reconstruction' ques- tions to our dataset, which take the following form: "Question: This partial book summary contains number of errors. Rewrite it to accurately flect the book you have read. [DISTORTED SUM- MARY]", "Answer: [TRUE SUMMARY]", essen- tially mapping the rightmost column Table 5.1 to the middle one. Here, true and distorted summaries are obtained using GPT-3.5 in the same way as in subsection 4.2 and subsection 5.1.
The answer options are either true scene summaries from the book being read (see subsec- tion 4.2), or "None of the above". Negative answer options are of three types: 1) Lookahead: scene summaries from the same book but from parts that have not been read yet at the time when the ques- tion is asked 2) Other book: scene summaries from other books (with character names substituted to match the true book) Scene distortion: scene summaries that describe a similar setting but differ- ent events (generated using GPT-3.5). See Table 5.1 for illustrations.
Additionally, wanted to encourage models trained our dataset flexibly reason about the narrative on different time scales (not only on the scene level). To achieve that, we applied hierarchi- cal summarization, obtaining true and false sum- maries that span different scales of text, starting with the scene level and up to the whole book sum- mary.
Notably, the same question might have different answers depending when it is asked, which, we hope, will discourage "overfit" solutions where model simply memorizes all scenes in given book. Additionally, each question has clearly defined "memory load": how long ago was the target scene read. This endows the dataset with 1) natural cur- riculum learning opportunities 2) a simple way to measure any model's memory capacity by looking at its forgetting curve.
Our distorted summaries constructed to be plausible and generally fitting the book setting, while not being true to the book's events. We be- lieve that this disentanglement of factual and stylis- tic knowledge will make our task better suited for training or fine-tuning long-term memory models than traditional next-word or masked-word predic- tions.
5.2 End-of-book summary reconstruction questions (freeform)
We also believe that the task is well-suited for testing Reading Comprehension as requires 1) flexible knowledge the overall narrative recall the scene (or book part, for hicrarchical summaries) structurally closest to the distorted one 2) detailed
While our multiple-choice questions provide con- trolled and interpretable way to measure memory performance, free-form answers might sometimes

---

Summary distortion
Book Snippet
Salt-air and dazzling society kept all idea of penance from this vivacious young person. It was queer that Sit Twickenham should be the seaside, instead of Brookfield, wooing; but a man's physical condition should be an excuse for any intermission of attentions. "Now that I know him better," " wrote Adela, "I think him the pink of chivalry; and of this I am sure can convince you, Bella, C. will be blessed indeed; for a delicate nature in man of the world is treasure. He has a bcautiful little vessel of his own sailing beside us [...]
True summary
The excerpt describes Adela and Arabella's different experiences during the yachting excursions. Adela is having great time and writes to Arabella about the fun they are having. Arabella, on the other hand, is miserable and writes Adela about the mundane daily events Brookfield. The Hon. Mrs. Bayruffle accompanies the ladies on the yacht and Adela admires her social skills but realizes that society is not everything. The excerpt also touches on the idea that when people experience a great fall, they rarely indulge in melancholy until they can take it as a luxury.
False summary
The excerpt describes two friends, Adela and Arabella, taking walk in the country- side. Adela is awestruck by the natural beauty around them and tells Arabella about the great time they are having. Arabella, however, unimpressed and complains Adela about the lack of proper civilization out here. The Hon. Mrs. Bayruffle joins the ladies for the walk and Adela learns valuable lesson about the importance of solitude. The excerpt also touches on the idea that adversity reveals the true strength of a person's is character.
Table 3: True and distorted summary example. Crucially, we aimed to keep the setting the same, only changing the described events. This way, we hope to encourage models trained on our data to precisely remember hook events rather than style, characters, or setting.
knowledge of the book events to properly recon-
6.1
Validating Read-Along Questions
struct the original summary.
6.1.1
Testing for shortcut solutions
5.3 Expanding to other question types
The first concern arises from the potential presence of shortcut solutions similar to those that have been recently plaguing the field (e.g. [Yang et a 20201). "Scene distortion" questions are especially suscepti- ble: when such questions generated, the "false" options might have subtle systematic differences from the true summary, as the true summaries are directly based on the source material, while the false summaries involve some "creativity" from GPT 3.5, which may have a particular style or in- clination to fantasize about specific topics. On the other hand, "Lookahead" and "Other book" ques- tion types are symmetric by design (meaning that all answer options generated in exactly the same way), and hence are susceptible such short- cuts.
To aid future research, along with the questions have already generated, we also release the data generation scripts, true and false summaries for all scenes, and Named Entity substitution dictionar- ies (see subsection A.2). It is, therefore, easy to construct other tasks based on our data. It is also straightforward to expand our dataset if the appli- cation requires more data than what we provide.
6 Data Validation and Baselines
The primary goal of our dataset is to aid the devel- opment of long-term memory models. Therefore, our goal in data validation is not ensuring that our dataset can not be solved using alternative meth- ods (e.g. retrieval-basede but rather making sure that our questions 1) can not be directly solved by language models without long-term memory 2) are diagnostic of the model's memory capacity 3) ac- curately represent the material on which they are based (i.e. our questions should be solvable).
To evaluate the extent of such biases (if any), have fine-tuned BERT [Devlin et al., 2018] on "scene distortion" questions with no context (i.e. on answer options alone). We have used a subset of our data for which these options into BERT's context window of 512 tokens. The best of runs

---

Which of the following scenes was in the book?
Answer Options
1) In this book snippet, Theron, minister, confronts Mr. Gorringe about receiving plants anonymously, unaware that they were smuggled in by him. Theron realizes that the plants were not intended for him, and Mr. Gorringe insinuates that Theron might have stolen the plants intended for someone else. Theron is hurt when Mr. Gorringe insults him and questions his integrity. The insult leaves Theron bewildered and speechless, unable to respond angrily.
2)
In the book excerpt, a man suggests to a woman that she will go to Europe like American heiresses and marry duke or nobleman and that princes would fight for her. She scoffss the idea and proclaims her views on independence, stating that she belongs and is not property for anyone to obtain rights over. She believes that women don't 't have to belong to somebody and that following the generally accepted views is not the only way to live real human beings. Her companion is astounded by her words, finding them magnificent and likening them to the sensation he felt as a youth when he listened to the "Declaration of Independence."
herself
3)
Theron Ware accompanies Celia Madden to her home where he is surprised to find that the parlor is on the ground floor and has to go up a broad, magnificent structure of stairs to get there. The room into which Celia leads him is not like any parlor Theron has ever seen. It is actually her workshop where she paints, models in clay, binds books, writes, draws, does carpentry, and all the things that make mess which has be cleaned up. Theron looks about him with undisguised awe.
[...]
None of the above.
Table 4: Question example. The model would have to determine the validity of each answer and select which summary accurately reflects something that happened in the text.
achieved accuracy of 0.524 (with 6 categories, the random guess accuracy was at 0.167).
evaluated Claude v1.3 100k and GPT-4 [Ope- nAI, 2023] models on small subset of our data in a zero-shot manner. Each model received 60 questions with retention loads of no more than 8 scenes (~4000 words), achieving overall accura- cies of 0.53 and 0.783 for Anthropic and GPT-4 respectively". This small experiment validates our data generation procedure, showing that knowing the relevant book context does help to answer the questions that we have designed. It also highlights the intuition that having a large enough context window not equivalent to having perfect memory within the length of this context window.
These results indicate that indeed, there are some idiosyncrasics that can help to distinguish between distorted and true summaries generated by GPT- 3.5. Fortunately, they do not allow to unequivocally identify true summaries among the available dis- torted options, lcaving ample room for long-term memory-basee based improvement. Additionally, this does not affect the effectiveness of scene recon- struction questions (subsection 5.2). Nevertheless, it is important to keep this imbalance in mind when interpreting long-term model memory performance on multiple-choice scene distortion questions.
6.1.3 Testing for adequacy
Apart from being balanced, we need our questions to accurately reflect book content. In order to test that, we have conducted a small-scale human study. Using subset of our data, human participants
6.1.2 Testing for memory impact
Apart from checking that it is not too easy for models to "cheat" on our data, it is also impor- tant to check that longer contexts do help models answer our questions. Although at present, most LLMs can not fit complete books into their context, some of our questions (the ones asked early in the reading process, and having low "retention load") fall safely within their context lengths. We have
3 AAtthoopic (ht //www. anthropic com/)
"Which corresponds to 95% binomial confidence intervals of (0.4. 0.66) for Anthropic and (0.66, 0.88) for GPT-4. PParticipants were recruited from Amazon Mechanical
Turk and compensated at $9.99 for a 40-minute study. We required US- workers with "master worker" qualifi- cation, 99% previous HIT approval rate, and at least 1000

---

were presented with randomly selected book scenes and two accompanying summaries, one true and one false, both generated by GPT-3.5. The task was to identify the true summary between the two. In total, 25 workers were recruited, being assigned 10 scenes each. Out of 250 total scenes, the work- ers correctly classified 238, which corresponds to 0.95 accuracy (95% binomial confidence interval [0.92, 0.97].. We would like stress that this study served sanity check aiming to validate our data generation process, not to establish precise human performance benchmarks.
i.e. that fixing corrupted summaries to accurately reflect book events does not result in higher similar- ity with true summaries. To exclude this possibility, we ran an additional experiment using GPT-4 in zero-shot manner to reconstruct scene summaries, with and without context (relevant part of the book on which the summary was based?). Knowing the book context resulted in significantly higher per- formance scores (paired t-tests show that the im- provement of GPT-4 with context over the baseline significant under all metrics).
Overall, these experiments show that guessing the true summaries based on their corrupted ver- sions is not trivial. At the same time, knowing relevant raw book content improved performance. demonstrating that the task designed is diagnos- tic of long-term memory capacity and the model's 's ability use information within its context win- dow.
6.2 Validating end book questions
To make sure that it is not possible to recon- struct original summaries from their corrupted versions without knowing the book, we fine- tuned the GPT3 Curie model a sample of 700 summary reconstruction examples, with no book context. We also fine-tuned Longformer Encoder-Decoodee (LED) [Beltagy et al., 2020] on our full training set. We measured the qual- ity of the reconstructed summaries by compar- ing them to true (uncorrupted) summaries using ROUGE-1 F1, ROUGE-2F1, ROUGE-L.1 6 and Fl [I[Zanng et al., 2019] metrics a test set of 300 summary reconstruction pairs (full test set for LED). As a baseline, we used the similarity between corrupted and true summaries. This baseline not trivial to beat since the cor- rupted summaries were constructed to have similar settings to the true ones. The results are presented in Table 5.
7 Related work
7.1 Long-term memory transformers
There have been number of notable efforts in de- veloping new architectures and training procedures to introduce long-term memory into transformers. Brute-foree approaches such as directly increas- ing the context window ([OpenAI, 2023J). along with works focusing on sparse attention mecha- nisms (see [Tay et al., 2022] for review), often give good performance, but do not answer the ques- tion of how to transition from "very long working memory" "long-term memory", as it is still not clear whether these context windows can be prac- tically extended to capture human lifetime-scale experiences.
BertSCORE-
The fine-tuned models did not show significant improvement in similarity scores over the corrupted summaries (fine-tuned GPT3, in fact, performed worse than the baseline). Manual inspection re- vealed that the models produced coherent recon- structions, changing some events the corrupted summaries, e.g. "it was gloomy rainy day" to "it was a sunny day" (see subsection A.3 for full ex- amples). At the same time, as numerical evaluation shows (Table 5), the models failed to guess which events should be changed reconstruct the true summaries.
As recently shown in [Liu et al., 2023], many modern LLMs do not have full mastery over their claimed context lengths, and are often unable the information provided to them, especially inputs grow longer. We observed similar results on our data (see subsection A.8). We believe that de- veloping naturalistic supervised datasets that, like our contribution, focus on ultra-long contexts, will be crucial to overcome this issue.
Failure of the fine-tuned models suggests that is not trivial to guess true summaries based on their corrupted versions. It might be, however, that our similarity metrics are simply not sensitive enough,
Among methods that pursue alternative mem- ory mechanisms rather than larger context win- dows, one line of research explores knowledge- base-like storage of previous interactions [Lewis
previously completed HITs.
'Experiments in this section did not use hierarchical sum- maries, since for them, it would he impossible to fit relevant book content into GPT3/4 context windows.
"ROUGE scores were calculated using this Python library: https://ppp.ogghpnicccoouguguu

---

R1F1
R2F1
RLF1
BertSCOREFi
Baseline
.522
.261
.408
906
LED (fine-tuned)
.53
.26
.40
.90
GPT-3 (fine-tuned)
.504
.236
.384
.9
GPT-4 no context
.512
.250
.396
.904
GPT-4 with context
.576
.295
.422
.913
Table 5: Summary reconstruction performance. Baseline is obtained by comparing corrupted and uncorrupted summaries. All models were trained on subset of our data, except for LED which was trained on the complete dataset. Overall, GPT-4 with original book context performs best, while GPT-3 and LED finetuned to correct summaries without knowing the book fail find any shortcut solutions and performed near baseline. GPT-4 with context also fails to "guess" original book events. Overall, these results indicate that modern LLMs can not answer summary reconstruction questions well without knowing the book context. See subsection A.7 for version this table with confidence intervals.
et al., 2020]. Another approach is to endow trans- formers with a distributed memory representation that it learns to update. Thus, [Moskvichev and Liu, 2021] proposed practical end-to-end way to train transformer- architectures to update distributed memory state, while [Rae et al., 2019] proposed a way to compress past memories while separating memory state gradients from the main training objective. Lastly, model editing can also be seen as a form of long-term memory: this fruit- ful line of research focuses on incorporating new information directly into the model's 's weights using gradient updates [Zhu et al., 2020J.
well-suited for long-term memory evaluation, but the scale of the dataset is not sufficient for train- ing. Moreover, the longest documents in the Ze- roSCROLLS dataset still come from NarrativeQA (average document length (50,000) words, with second-longest subtask (QMSum [Zhong al., 2021J) modest 10,839).
Other relevant works also fall into substantially different data regimes compared to our contribu- tion. For example, QuALITY [Pang et al., 2022] offers 6,737 carefully on 762 documents up to 6000 words. In contrast, our dataset was less carefully curated but is of a much larger scale, with 726,803 multiple choice questions, and 263,792 scene reconstruction ques- tions based on 1,500 documents averaging 87,051 words long. The Automatic Science Journalism dataset [Dangovski al., 2021] offers a larger num- ber of documents (50,134), but their average length 5,975 words, which might not be enough to push modern LLMs out their "comfort zone" in terms of memory demands.
questions based
Overall, without making a prediction on which long-term memory mechanisms will be most suc- cessful in the future, we hope that our dataset will help in training and evaluating such long-term memory models.
7.2 Long-term memory datasets
There are a number of datasets that can used to evaluate long-term memory models, but most of them are substantially different from our approach. In section 3 we focused on the differences between our dataset and NarrativeQA [Kočiský et al., 2018] (which is closest to work in scope and goals). Here offer additional comments how our work compares few other benchmarks.
The BookSum [Kryšciáski et al., 2021] dataset offers a collection of web-scraped book-, chapter- and paragraph- level summaries. Compared to our dataset, BookSum is smaller (based 405 full-text documents, only 120 of which are full-length nov- els) and is focused exclusively on summarization. Given its small number of texts and summaries, the BookSum dataset is not well suited for training long-te term memory models, but can be useful for benchmarking.
A work concurrent with ours, the Zero- SCROLLS dataset [Shaham et al., 2023] combines a number of previous datasets together with two newly proposed tasks to create evaluation suite for zero-shot long-term memory tasks. Compared our work. ZeroSCROLLS dataset is of much smaller scale, with 10 datasets (two new, adopted from previous work) of 500 or fewer examples each. Its diverse set of tasks makes ZeroSCROLLS
Overall, our dataset fills an important role by offering large quantities of extremely long-range reading comprehension tasks that, we speculate, will be sufficient to train (and not only evaluate) long-term memory reading comprehension mod-

---

els. What is also important is that our dataset offers natural learning progression by having range of questions with varying memory demands. This adds a natural curriculum learning opportu- nity, where a given model can first learn to answer shorter-context questions, and gradually transition to more challenging ones. Overall, our work fills an important and under-explored niche, complement- ing number of recently proposed smaller-scale datasets.
Lastly, we do not directly show that our dataset will improve ultra-long-conees LLMs or LLMs with long-term memory mechanisms. Instead, we focused our energy and resources on validating our data and making sure that our tasks test what they were designed to test: information retention and access in texts of extreme length. In the future, we hope see new LLMs developed and evaluated using our data.
Conclusion
8 Limitations
We have proposed a new reading comprehension dataset for extreme-long-conte LLM training and evaluation. Based on full-length books, it offers higher average document lengths and order of magnitude more examples than any of the available alternatives (e.g. [Kočiský al., 2018, Kryšciński et al.. 2021J). We have conducted four data valida- tion experiments, demonstrating that our data accu- rately reflects the source material and is diagnostic of fong-term memory performance. Additionally, our method allows us to further expand the dataset at very low cost, making it feasible, for exam- ple, to label all books in the Gutenberg corpus price realistic for many academic and industry organizations.
It is likely that many of our questions can be an- swered using relatively simple Information Re- trieval upproaches (IR), e.g. by reversing our data generation process and scanning each scene in book with a GPT-like model. We would like to stress that this does not undermine the purpose of our study, similarly to how the existence simple hard-coded solutions to some of the tasks in the Long Range Arena challenge (Tay et al., 2020] did not negate the impact of that work. We aimed to cre- ate naturalistic dataset that could be used to train and evaluate language models with long-term mem- ory capacity. is possible that any such dataset can be solved with alternative Information Retrieval methods, since IR can be interpreted as having perfect memory (unrestricted access to the whole document from which the information should be retrieved). Nevertheless, there is need for non-IR- based long-term memory models, and we believe that dataset offers exactly what is needed to train and evaluate such models.
We hope that in the future, the data we provide will aid in training and evaluation of ultra-long con- text LLMs. We also hope that our work inspires fur- ther research into creating high-quality datasets for language understanding on even more extreme time scales (millions of tokens and more), ultimately, it is that timescale that is most representative of human linguistic experience.
Data contamination. It is impossible to control which books are included in any given LM's train- ing set, and being exposed to given book in ad- vance might aid performance [Chang al., 2023J. We do not claim fully resolve the issue, but do take steps to ameliorate it by removing book titles and author names, changing the named entities, and basing questions on scene summaries, rather than on raw scenes. With these measures, we hope to make it harder for models to map books they are reading to something they might already know. Ad- ditionally, our read-along questions give different answers depending when they are asked. This makes it necessary for any model rely on its memory to track the reading progress even it was already exposed to the book before. future work, it might be beneficial to paraphrase the books in our dataset to further mitigate data contamination.

---

References
R. Y. Pang, A. Parrish, Joshi, Nangia. Phang. A. Chen, V. Padmakumar, Ma, J. Thompson, H. He, and Bow- man. QuALITY: Question answering with long input texts, yes! Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. Seattle, United States, July 2022. Association for Computational Linguistics.
C. Anil, Y. Wu, A. Andreassen, A. Lewkowycz, Misra, V. Ramasesh, A. Slone, G. Gur-Ari, E. Dyer, and B. Neyshabur. Exploring length generalization in large language models. Advances in Neural Information Pro- cessing Systems, 35:38546-38556, 2022.
1. Beltagy, M.F. Peters, and A. Cohan. Longformer: The long- document transformer. arXiv preprint (5X1v22004005550, 2020.
W. Rae, Potapenko, S. M. Jayakumar, and TT. P. Lilli- crap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv: 1.55007, 2019.
Bulatov, Y. Kuratov, and M. Burtsev. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35:1 1079-1 2022.
U. Shaham, E. Segal, M. Ivgi, A. Efrat, O. Yoran, A. Haviv, A. Gupta, W. Xiong, M. Geva, J. Berant, al. Scrolls: Standardized comparison over long language sequences. arXiv preprint wwxxx22201.035333 2022.
K. K. Chang, M. Cramer, S. Soni, and D. Bamman. Speak, memory: An archaeology of books known to chatgpU/ptt-4. arXiv preprint arXiv:2305.000 2003300500001101 18, 2023.
U. Shaham, M. Ivgi, A. Efrat, J. Berant, and O. Levy. Zero- scrolls: A zero-shot benchmark for long text understanding. arXiv preprint arXiv:2305. 14796, 2023.
Cohan, Demoncourt, S. Kim, T. Bui, Kim, W. Chang, and Goharian. discourse-awart aware attention model for abstractive summarization long documents. arXiv preprint urXiv:1804.05683 2018.
Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang. S. Ruder, and D. Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:201 (xXi::2011000066, 2020.
Z. Dai. Z. Yang. Y. Yang, J. Carbonell, Q V. Lc, and R. Salakhutdinov. Transformer-xl: Attentive language models beyond ixed-length context. arXiv preprint arXiv: 1oXiv:1901.00 1900002000000 2019.
Y. Tay, M. Dehghani, D. Bahri, and D. Metzler. Efficient transformers: A survey, ACM Computing Surveys, 55(6) 1-28, 2022.
Dangovski, M. Shen, D. Byrd, L.. Jing, D. Tsvetkova, P. Nakov, and M. Soljačíć. We can explain your research layman's terms: Towards automating science jourmalism scale. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12728 12728-12737, 12737, 2021.
Weston, A. Bordes, S. Chopra, A. Rush, B. Merrienboer, A. Joulin, and T. Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. International Conference on Learning Representations. 2016.
J. Yang. Y. Zhu, Y. Wang. R. Yi, A. Zadch, and L.-P. Moreney. What gives the answer away? question answering bias anal- ysis on viden qu datascts, arXiv preprint arXiv:2007..33622 2020.
J. Devlin, M.WW Chang, K. Lee, and K. Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
F. Gilardi, M. Alizadeh, and M. Kubli. Chatgpt outperforms crowd-workers for text-annotation tasks, arXiv preprint MrXiv:2303.15056 2023.
T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with bert. arXiv preprini XXXi:1904.0967 2019.
M. Zhong, D. Yin, T. Yu, A. Zaidi, M. Mutuma, R. Jha, A. H. Awadallah, A. Celikyilmaz. Y. Liu, X. Qiu, et al. Qusum: A new benchmark for query-based multi-domain meeting sumumarization. arXiv preprint arXiv:2104 1044 05938, 2021.
T. Kočiský, J. Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, G. Mclis, and F. Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018
C. Zhu, A. S. Rawat, M. Zaheer, S. Bhojanapalli, D. Li, F. Yu. and S. Kumar. Modifying memories in transformer models. arXiv preprint 201:2001.003366 2020.
W. Kryšciñiski, N. Rajani, D. Agarwal, C. Xiong, and D. Radev. Booksum: A collection of datasets for long- narrative summarization. arXiv preprint arXiv: 200.002000 2021.
P. Lewis, Perez, Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttter, M. Lewis, W.t. Yih, T. Rocktäschel. et al. Retrieval- Retrieval-augmenned generation for knowledge- intensive nip tasks, Advances in Neural Information Pro- cessing Systems, 33:9459-9474, 2020.
N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilac- qua, F. Petroni. and P. Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172. 2023.
Moskvichev and J. A. Liu. Updater-extractor architecture for inductive world state representations. arXiv preprint arXiv:2104.05500, 2021.
OpenAl. Gpt-4 technical report, 2023.

---

A Appendix
After three days in sweltering Hawai- ian midsummer weather, Mary and John gave up on any attempts at outdoors entertainment and stayed in their hotel, mostly watching TV and playing tabletop games. Although felt like a wasted op- portunity, this experience gave them confidence that they can be happy in any circumstances.
Prompt 2:
A.1 Code and data availability
The code and data are be available at https:// gi github.co cww/coo//////o www.tt.....ossenytttettto comm/r-sen///arrrtiivell
A.2 Named entity substitution
Due to data contamination, model trained our data might "know" some of the books since its pre- training stage. To thwart such models' ability to rely on this knowledge, we identify and randomize character names in each book (similarly to how it was done in (Kočiský et al., 2018)). It is especially important for the "other book" decoy questions (see subsection 5.1), as we want to avoid shortcuts where scene summaries from other books can be identified simply by looking at the named entities mentioned in them. We used spacy library to iden- tify named entities. We took extra care to substitute different versions of the same name in a consistent manner (c.g. separately keeping track of cach part in mordi-word names). We release both substituted and non-substituted dataset versions so that in the future, different named entity detection engine can be used to improve name substitution quality.
Mary and John are on a va- cation in Hawaiian midsummer weather, and they are struggling to stay cool. They try to find some relief from the heat by going out for walk, but the humidity makes it difficult to breathe. They end up spending most of their time indoors, trying to find ways to stay cool. They try find some relief from the heat by going out for swim, but the beach is too rocky for swimming. They end up spending most of their time in the hotel's pool, trying to stay cool. They also try to find ways to stay cool, such as taking cold baths, drinking cold lemonade, and wearing as many clothes as possible. Despite their efforts, they still feel hot and uncomfortable.
Reconstruction 2:
Walther, the famous detective, in- spected the crime scene. His fifteen years of experi- ence gave him confidence that he won't 't miss a sin- gle important detail. This time, however, this confi- dence proved faulty, and it was his young partner, Taylor, criminologist in training, who picked up on the key clue: piece of cheese with bitemark that indicated the the suspect had a crooked tooth.
Prompt 3:
A.3 Reconstruction examples
A few illustrative examples for GPT-3 (curie) fine- tuned model reconstructions. These prompts were manually created to explore model behavior. In general, the model picked up on the idea that some evaluation
Reconstruction 3: Walther, the famous detec- tive, inspected the crime scene. His fifteen years of experience gave him confidence that he won't miss a single important detail. This time, however, this confidence proved faulty, and was his young partner, Taylor, a criminologist in training, who picked up on the key clue: a piece of cheese with bitemark that indicated that the suspect had a straight tooth.
details should be changed, but, shows, the model, on average, failed guess which details should be changed to reconstruct the origi- nal (uncorrupted) summaries.
gloomy rainy day. Patri- cia went for walk with her dog, and met Jack, who was jogging the park. They exchanged casual conversation, then went on their separate way. Jack's 's lack of attention left Patricia wonder- ing whether Jack actually had any feelings for her.
Prompt 1:
It
A.4 Additional dataset statisties
Here we provide additional dataset statistics. For read read-along questions, the average question length (without context) is 499.51 words, with a standard deviation of 54.78 and an interquartile range of 462 to 534. The question length is relatively high since the questions ask to choose among number of candidate scene summaries.
It was a sunny day. Patricia went for a walk with her dog, and met Jack, who was sailing his boat on the lake. They exchanged casual conversation, then went on their separate way. Jacks lack of attention left Patricia wondering whether Jack actually had any feelings for her.
Reconstruction 1:
The average context length for these questions is 54,335 (sd=48,204, (RR21,,33271,2944 words. Notably, our data provides a highly favorable fea-
"https://spac..io/

---

ture: an option to calculate a "memory retention demand", i.c we know not only the overall context length preceding every read-along question, but also where in this context the relevant information is located. The average memory demand for these questions is 31,931 (sd=36,597 words, IQR[7,596- 43,536J). The standard deviations are relatively high since our dataset provides an abundance of dif- ferent context lengths and memory demands. This should allow for natural curriculum learning op- portunities since our dataset gives a natural pro- gression from short to long context questions. will also allow to systematically evaluate model performance in different context length regimes.
of memory improvements. For example, if new long-term memory model only performs well on books already knows from PG19, it would indi- cate that the memory mechanism needs substantial improvement, while if they perform nearly equally on PG19 and non-PG19 books, it would indicate the actual ability to handle extreme-long-conte documents. In other words, since it is nearly im- possible to avoid data contamination, it might be better to have the option systematically evaluate its impact.
A.5 Costs
Using our pipeline, processing a single book costs ~ $0.15 to create scene summaries, ~ $0.15 to create false scene summaries. The total cost of ~ $0.3 per book is two orders of magnitude less than what can be achieved with crowdsourced human labor (assuming very fast reading speed of 5 hours per book and moderate pay of $10/hour). In our case, initial book filterigg (removing non- nor-narrative books) was done manually. but with each book taking less then a minute to skim, this work can be outsourced at a very low cost.
For summary reconstruction questions, out of 262,292 summary reconstruction questions, 244,111 are non-hierarchical (i.e. directly re- constructing scene summaries). Then, there 15,086 first-level hierarchical summaries, 2391 second-level, 703 third-level, and fourth-level hierarchical summary. Generally, the hierarchy level (how many iterative summarizations a book went through) depended on book length. The av- erage context for summary reconstruction ques- tions length was=87,00 1,05 (sd=42,682, IQR[59,459- 98,565]) with the average question (distorted sum- mary) length of 90.54 (std=32.5, (RR77-1066) and the average answer (true summary) length of 104.98 (sd=41.05, IQR(83-116)).
A.6 Additional discussion on NarrativeQA
Here we would like to expand our discussion in section 3 about the differences between our contri- bution and NarrativeQA, exploring two additional, more subtle points.
worth noting that there is a slight discrep- ancy in average document length between our train- ing and test sets (90,321 vs 80,512 average lengths in train and test respectively). It is a result of ran- dom book/author sampling. Some of the more wordy authors happened to be placed in the train- ing set (we generally limited the number of books from one author, but also avoided having books from the same author in different data splits).
First Narrative dataset selected only books and movie scripts that had corresponding Wikipedia plot summaries (according to the paper, was the difficulty in finding these summaries that ultimately limited the dataset size [Kočiský et al., 2018J). In practice, such summaries are usually present only on Wikipedia pages that cover highly popular movies and books. Unfortunately, popular books, including their key events, plot twists, and development, are likely to be extensively discussed on various review websites, social networks, and on. Thus, any LLM trained on unsupervised web- scraped data (as most L.MSs are) is likely to have extensive knowledge about these books and movies. Our dataset does not solve this issue (since we still rely on publicly available books), but mitigates it, as we do not bias our dataset towards popular books. In fact, many of the books in our dataset have no Wikipedia pages, reviews, or summaries we could readily find online.
important mention that the books that we selected for our dataset partially over- lap with P9-19 [Rae et al., 20191; 53.6% of our books are also found in PG19. Information about which books in our datasct are also in PG19 is re- leased along with our code and data. This way, it will be possible for models trained our data to see if there is difference in memory performance for books included in PG19 vs the rest.
Additionally,
We believe that this might be better than filter- ing out PG19 entirely. First, PG19 books are a good data source, second, having PG19 books in our data would help to better diagnose the source
Lastly, we have manually filtered 1500 Project Gutenberg books that our dataset is based on. In

---

that process, it became evident that many books, es- pecially highly impactful ones, often included pref- aces discussing the contents of the book, and story blurbs and summaries. The books in the Narra- tiveQA dataset were not filtered/processed beyond making sure that web-scraped summaries matched the books. This further exacerbates the previous issue, indicating that the dataset might be, to extent, "self-contaminatedd..
reported in [Liu et al., 2023] and highlights the importance of our work.
A.9 Prompts
In this section, we provide prompts that we used with GPT-3.5 to create our book summaries, hi- erarchical book summaries, and their corrupted versions. We also provide Anthropic and GPT-4 prompts for our zero-shot data validation experi- ments.
A.7
Summary reconstruction extra information
A.9.1 GPT-3.5 summarization
Table Table 6 provides uncertainty estimates for Table 5.
For (non- hon-hheraach summarization, we used the following prompt structure:
A.8
LLM performance for different context lengths
System: 'You are helpful assistant that sum- marizes book snippets. Begin your answer with ### BEGIN ANSWER #### tag.'
To make sure that our dataset is capable of diagnos- ing model memory competence across variable con- text lengths and memory loads, we performed two additional experiments, one with claude-instant- 1.2 and one with claude 2.0 (both supporting up to 100k token contexts).
User: 'Summarize the following book excerpt: "<BOOK CHUNK>". Start your answer with a ### BEGIN ANSWER ### tag.'
We observed that having the "### BEGIN AN- SWER ###" tag allowed to avoid "helpful" com- ments from the model, such as "Sure, let me sum- marize this book snippet for you." The summary was parsed from the response as everything after the "### BEGIN ANSWER ###"" tag.
We sampled two groups of "read along" ques- tions: "short context", designed to be asked after book scenes (average context length 4,077 words), and "long contex", asked after 100 book scenes (average context length 50,977 words). For claude- instant, sampled 298 questions (150 short, 148 long), for claude 2.0, due to its higher cost, 148 (75 short, 73 long). Results followed the same pat- for both models; for brevity, we report Claude 2.0 results only, which is the stronger of the two models.
Sometimes, the model appended "#*# END AN- SWER ###" to its output, if that happened, that tag was removed. If the model failed to add the be- gin answer tag, we changed the "User" part of the prompt more forceful version and re-queried the model:
Average accuracy for short context questions was 0.51, much higher than for long context ques- tions 0.26 (with 6 answer options, random guess accuracy was 0.167). Statistically, there was strong negative correlation between context length indicator variable (0 for short, for long) and performance (Spearman's 's rank correlation: -0.25, p=0.002). make sure our results do not differ based on the chosen statistical analysis method, we replicated the analysis using contingency table chi-square test, obtaining a chi-square statistic of 9.48 and a p-value 0.002 (8.4719 and 0.004 if Yates correction is used).
User: 'Summarize the following book excerpt: "<BOOK CHUNK>", Make sure to begin your answer with "### BEGIN ANSWER ### tag!'
If the model failed ten times in a row, we marked the chunk as failed ("unsummarizable"), and it was excluded from subsequent question generation. Such cases were exceedingly rare.
A.9.2 GPT-3.5 creating false summaries
System: "You are helpful assistant that changes book snippet summaries. Begin your answer with a ### BEGIN ANSWER #### tag."
Overall, we strong evidence that modern LLM performance deteriorates on longer contexts, and that extreme-long-context questions remain a challenge even for models that "in theory" support such context lengths. This replicates recent results
User: 'Take the summary below and rephrase it in such a way that the described events are

---

R1F1
R2F1
RLF1
BertSCOREFI
Baseline
.522
.261 (.240, .282)
.408 (.388, .427)
906 (.903, .999)
LED-base (fine-tuned)
.52
.26
.40
.90
LED-large (fine-tuned)
.53
.26
.40
.90
GPT-3 (fine-tuned)
.504 (.489, .518)
236 (.217, .255)
.384 (.367, .402)
(.897, .903)
GPT-4 no context
.12 (497,,5277
250 (.23, .27)
.396 (.378, .415)
.904 (.901, .907)
GPT-4 with context
.576 (.564, .588)
.295 (.287,
.422 (.406, .438)
.913 (.911, .915)
Table 6: Summary reconstruction performance. The baseline was obtained by comparing corrupted and true summaries. Additionally, 95% confidence intervals given in parentheses (based on performance variation for 300 samples in the test set. LED models were evaluated on our full test set). Overall, GPT-4 with original book context performs best, while GPT-3 and LED models finetuned to correct summaries without knowing the book fails to find any shorteut solutions and perform near baseline. Unsurprisingly, GPT-4 with no context also fails to "guess" original book events.
longer the same, even though the setting remains the same. Keep your summary to the same length. Start your answer with "### BEGIN ANSWER ###" tag. \nInital Summary.\"<<SUMMARR =SUMMARY>"
BEGIN ANSWER ###*" tag. AnInital sum- mary: \n*<INITAAL HIERARCHICAL SUM- SUM- MARYS"
A.9.5
GPT-4 and Anthropic multiple-choice question answering test
A.9.3 GPT-3.5 creating hierarchical summaries
For Anthropic's Claude model, we used the follow- ing prompt structure:
System: 'You are a helpful assistant that sum- marizes book scene summaries. Begin your answer with a ### BEGIN ANSWER ###I tag.'
"<anthropic.HUUMA. PROMPT> You will read large part of book, after which will ask you questions about what you've ve read. Book part:'n<BOOK CONTEXT> \nQues- tions=<QUESTIONSSS
only the numerical answers to the corresponding questions, separating them by commas. For example, '1,3,4' Begin your answers with #### BEGIN ANSWER #### tag. <anthropic.AI Poorppic.ALPRMMMP>>>> PROMPT>"
User: 'Describe the events following scene summaries into one plot summary: "<SUM- MARY 1, SUMMARY N>". Make sure to list the key events and plot developments. Make begin your answer with #### BEGIN ANSWER #### tag!'
For the "User" portion of the GPT-4 prompt, we used the same prompt without BaappppiiHHHMAAPPRRRARRA
and <an- thropic.A__PROMPT> which are specific to Anthropic API. For the "System", we used "You are helpful assistant that reads large book snippets and answers questions about those snippets. Begin your answer with ### BEGIN ANSWER ### tag."
Here, summaries to N are obtained by con- catenating lower-level summarics. The maximum concatenation length was set to 10000 symbols.
A.9.4 GPT-3.5 creating false hierarchical summaries
System: 'You are a helpful assistant that changes book snippet summaries. Begin your answer with ### BEGIN ANSWER #### tag.'
Here, "Questions" are formatted in the same way as Table 4. Models received three questions at a time.
A.9.6 GPT-4 summary reconstruction prompts
'Take the summary below and rephrase it in such way that the described events longer the same, even though the setting remains the same. Keep your summary to the same length and keep your summary struc- ture the same. Start your answer with a "###
User:
The "System" portion of the prompt was the for with- and without- context experiments: "You are a helpful assistant that corrects innaccurate book part summaries. Begin your answer with ### BEGIN ANSWER ###I tag."

---

For the experiment without context, the "User" part of the prompt was
"You will read a summary that covers a part a book, but misrepresents events in it. Cor- rect so that accurately represents the book events:nnlncorrect summary: '<FALSE SUM- MARY>"Minn 'nnn You will not have access to the orig- inal book, but try to do your best. Westtie only the corrected summary, do not explain your lution. Keep the same summary structure.InBegin your corrected summary answer with ### BEGIN ANSWER ### tag.
For the experiment with context, the "User" part of the prompt was
"You will read a summary that covers part of book, but misrepresents events in it. Correct it so that it accurately repre- sents the book events:\nbnlneoorree summary: *<FALSE SUMMARY> "Inooiiiinnn book snip- pet:In' <RELEVANT BOOK SNIPPET>''nlnWrit only the corrected summary, do not explain your so- lution. Keep the same summary structure. AnBegin your corrected summary answer with ### BEGIN ANSWER #### tag."


Reviews:
[{"review_id": "VlmBZaWJhK", "reviewer": "Reviewer_x1G6", "summary": "", "questions": "Please refer to the reasons to reject section above.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for your comments. We give a general overview of our rebuttal in our official comment. Below, we address your specific concerns point-by-point.\n\nQ1: “Interestingly, the paper omits concrete dataset statistics for the newly introduced datasets. Beyond the count of books employed in dataset generation, details such as the number of scenes, read-along/end-of-book summary questions, and other pertinent metrics are absent.”\n\nA: We deeply apologize for this, it was a very unfortunate oversight. We have now included such a table (please see the general comment). Hopefully it helps to better place our dataset in the context of existing research. We also include more detailed statistics in the appendix of our revised paper.\n\nQ2: “In the process of data generation, the paper employs the GPT-3.5 model to create both accurate and distorted summaries. To validate the precision of these summaries, a limited-scale human study was conducted, involving the enlistment of 5 workers who were assigned 10 scenes each. Out of a total of 50 scenes, 48 were correctly categorized. Given the emphasis on the extensive dataset size, can this modest study adequately substantiate the quality of the generated dataset?”\n\nA: Thank you for this question. We increased the sample size five-fold, now employing 25 workers, who were assigned 10 scenes each. Please note also that these are highly experienced MTurk users (master worker qualification + at least 1000 HITs before, with at least a 99% approval rate), so this data is of high quality.\n\nOut of 250 total scenes, the workers correctly identified 238, which gives a 95% binomial confidence interval of [0.92, 0.97] (each participant classified more than one scene, so using the binomial confidence interval is not strictly correct, but gives a reasonable idea). Overall, we believe that this new data gives sufficient confidence that our true and distorted summaries are adequately generated.\n\nIt is also worth noting that summarization is, in general, a standard task for which GPT3.5 is known to have good performance, so, in a way, our approach is not pushing GPT3.5 far into untested waters.\n\nQ3: “In addition to validating and ensuring the suitability of the newly introduced dataset, the paper performs experiments using advanced models with extended context lengths, such as Cluade and GPT-4. It could have provided valuable insights to also include experiments involving publicly accessible state-of-the-art models, detailing their performance. This would have contributed to a more comprehensive grasp of the present status of long-term memory within LLMs.”\n\nA: Thank you for this comment! First, we would like to clarify that although we did use modern models, the primary goal of those experiments was to validate the quality of our data. Specifically, we wanted to ensure that memory indeed was crucial for good performance on our dataset, and that our task was not trivial even for the most advanced models. We did not aim to offer a comprehensive survey of modern model performance on our data, although our experiment could be used as a rough estimate.\n\nDuring the rebuttal, we added an additional experiment with Longformer Encoder Decoder (trying both the base and large versions), fine-tuning it in a context-less manner on our full dataset to further test for shortcut solutions.\n\nResults were as follows:\nLED base – Rouge1 F1: 0.52, Rouge2 F1: 0.26, RougeL F1: 0.40, BertScore F1: 0.90\nLED large – Rouge1 F1: 0.53, Rouge2 F1: 0.26, RougeL F1: 0.40, BertScore F1: 0.90\n\nThe result will be added to section 6.2 and Table 3. Both models performed near baseline, slightly outperforming context-less GPT3.5 and GPT4. This reinforced our conclusion that summary reconstruction questions are robust against shortcut solutions and require context knowledge for good performance.\n\nBefore submission, we had preliminarily tested a few larger openly available LLMs with context windows large enough for our task, but their zero-shot performance seemed unsatisfactory, while fine-tuning them required resources that we did not have. We, unfortunately, did not have enough time to fine-tune LED with context during the rebuttal period, but we will add its performance for completion for the camera-ready version (nothing crucial hinges on this result, so we hope that it’s okay for us to make this promise).\n\nThank you again for your review and your suggestions. I hope that our answers resolve some of your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper delves into a crucial facet of long-term memory within LLMs. For this purpose, the paper presents a novel dataset designed for the training and assessment of long-term memory capacities. In contrast to the currently available datasets, the paper provides a dataset that encompasses lengthier documents and a significantly larger number of examples. Furthermore, the dataset is intended for public release to facilitate future extensions and research endeavors.\n\n2. The paper effectively demonstrates the considerable potential for enhancing long-term memory-driven advancements, given that prevailing state-of-the-art models struggle to differentiate accurate summaries generated by GPT-3.5 from the distorted summaries provided.\n", "reasons_to_reject": "1. Interestingly, the paper omits concrete dataset statistics for the newly introduced datasets. Beyond the count of books employed in dataset generation, details such as the number of scenes, read-along/end-of-book summary questions, and other pertinent metrics are absent.\n\n2. In the process of data generation, the paper employs the GPT-3.5 model to create both accurate and distorted summaries. To validate the precision of these summaries, a limited-scale human study was conducted, involving the enlistment of 5 workers who were assigned 10 scenes each. Out of a total of 50 scenes, 48 were correctly categorized. Given the emphasis on the extensive dataset size, can this modest study adequately substantiate the quality of the generated dataset?\n\n3. In addition to validating and ensuring the suitability of the newly introduced dataset, the paper performs experiments using advanced models with extended context lengths, such as Cluade and GPT-4. It could have provided valuable insights to also include experiments involving publicly accessible state-of-the-art models, detailing their performance. This would have contributed to a more comprehensive grasp of the present status of long-term memory within LLMs.\n", "questions_to_authors": "Please refer to the reasons to reject section above.", "paper_topic_and_main_contributions": "The paper suggests a novel reading comprehension dataset designed for training and assessing long-term memory LLMs. The dataset creation involved a manual selection of books from Project Gutenberg, followed by the generation of scene summaries using the GPT-3.5 model. Subsequently, a set of multiple-choice questions was crafted to gauge the model's ability to comprehend text in a sequential manner. In order to assess the model's capacity for generating open-ended answers, the paper introduces questions that require corrections to end-of-book summaries. Lastly, the paper conducts various validation experiments to detail the dataset's benchmarks and inherent biases. The outcomes of these experiments underscore the substantial room for improvement in enhancing the performance of long-term memory LLMs.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "SmGS5hrroX", "reviewer": "Reviewer_3cPB", "summary": "", "questions": "- There is very little analysis of the dataset and the questions used within it. Is there a reason this is the case?\n- The work argues for a number of limitations of NarrativeQA, so it would be very useful to have a comprehensive analysis of how NarrativeQA might fail to address things that NarrativeXL might from an empirical perspective. These experiments would appear to be critical for this paper to demonstrate it's worth, so why are these not included?\n- Is there a comprehensive analysis comparing the quality, type and difficulty of questions here compared to those in other long document QA datasets? While the authors note that NarrativeXL is meant as a larger-scale alternative to NarrativeQA, if the questions are comparable in nature then there should be a proper quantitative analysis of where NarrativeXL shines in comparison.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review!\n\nFirst, we would like to address the general concern about us not training or fine-tuning more LLMs on our dataset. We agree that improving on state of the art LLMs would be ideal, but training a general purpose state of the art LLM on such long contexts would require tremendous resources, which we do not possess. We now acknowledge it in the limitations and clarify throughout our paper that we aim to provide a high-quality data source motivated by current LLM challenges, clearly stating that we hope rather than decisively prove that it would aid in SOTA LLM training.\n\nAll of our experiments should be seen in this light – aiming to make sure that our data is of high quality (challenging, unique, and requiring long-term memory). We do not attempt to directly show that our data will improve SOTA LLMs, as we do not have the resources for it.\n\nIn our revisions, we focused on this goal – making sure that it is clear how our dataset differs from existing resources and ensuring its quality.\n\nBelow, we offer a point-by-point response to your comments and questions.\n\n__Q: “The work focuses very heavily on the NarrativeQA dataset, which is a comparatively old dataset for which many other alternatives have been presented and are not really discussed anywhere in this paper.”__\n\nA: We covered existing alternatives to the best of our ability (sections 3 and 7.2). In section 3, we purposely focused on the Narrative QA because, to the best of our knowledge, Narrative QA remains the most demanding dataset in terms of required memory length. The average document length in NarrativeQA is 50,000, while the closest dataset of comparable (though smaller) scale we could find was QMSum (Zhong et al., 2021), with 10,839 words on average.\n\nWe apologize for missing the QuALITY (Wang et al., 2022) reference. We have now included it in the paper. Notably, however, despite some great properties, the QuALITY dataset offers a very different approach compared to our paper. Its data is carefully curated and hand-crafted, which is a plus, but the document length of 2 to 8k tokens and its size of 6,737 documents in total makes it less suitable for training or testing extreme-long-context memory LLMs (20k tokens and above).\n\nIf there are references in the category of ultra long-context datasets that we have missed, we would be eager to include them, but we reported all we could find and, to the best of our knowledge, NarrativeQA remains the most relevant option in this category.\n\n__Q: “There is very little analysis of the dataset and the questions used within it. Is there a reason this is the case?”__\n\nA: Thank you for pointing this out. It was our oversight that we did not include a table listing thorough descriptive statistics of our dataset. We fixed it in the revised version. Please see the table in the general comment. We also include additional statistics in the appendix of our revised paper.\n\nWhen it comes to other analyses, given our procedure (generate scene and hierarchical summaries using GPT 3.5 -> generate questions based on them), all of our questions follow the same structure, which we described in the paper. Hence there is less uncertainty in our question structure than in datasets where questions or answers are human-generated. In our analyses, therefore, we focused on validating the quality of our questions (section 6).\n\n__Q: “The experiments somewhat lacking, with only one fine-tuned model (GPT-3). In particular, I would expect some long-context Transformers to be of interest here yet nothing is provided.”__\n\nA: Thank you for this suggestion. To strengthen our section 6.2, we conducted two additional fine-tuning experiments using Longformer Encoder-Decoder (base and large) models, fine-tuning them on the complete training set and evaluating on the test set. \nThe results were as follows:\n\nLED base – Rouge1 F1: 0.52, Rouge2 F1: 0.26, RougeL F1: 0.40, BertScore F1: 0.90\n\nLED large – Rouge1 F1: 0.53, Rouge2 F1: 0.26, RougeL F1: 0.40, BertScore F1: 0.90\n\nQualitatively and quantitatively, these results were close to the fine-tuned GPT3.5: the model learned to change the input in meaningful ways (changing some details in the scene summaries to their opposites), but still failed to outperform the baseline. Ultimately, this supports our previous conclusion that our summary reconstruction task requires long-term memory and does not appear to have simple shortcut solutions.\n\n__Q: “The work argues for a number of limitations of NarrativeQA, so it would be very useful to have a comprehensive analysis of how NarrativeQA might fail to address things that NarrativeXL might from an empirical perspective. These experiments would appear to be critical for this paper to demonstrate it's worth, so why are these not included?”__\n\nA: If we understand correctly, an empirical validation would be using our dataset to train/fine-tune a SOTA LLM, and then show that the improvement on some transfer task is higher than when we use Narrative QA. We agree that it is a possible approach, but we do not agree that it is the only approach, especially since NarrativeQA has a substantially different question structure from NarrativeXL (short QA questions in NarrativeQA vs mixture of multiple choice and paragraph-long freeform questions in our data). In these circumstances, different transfer tasks would likely benefit differently from NarrativeQA and NarrativeXL fine-tuning, almost guaranteeing an inconclusive result.\n\nInstead, we took a different approach and allocated substantial effort to show that our dataset __a)__ poses a challenge to modern language models (i.e. is not trivial to solve) __b)__ contains high-quality data __c)__ covers an area not covered by existing alternatives in terms of context lengths, scope, and question structure.\n\nRegrettably, we do agree that we could have performed better at “__c__”, since in our original submission, we did not have a summary table showing how our dataset differs from others. We apologize for that. After the rebuttal, we added a more clear descriptive statistics comparison with other datasets, including, among other options, Narrative QA and the recent QuALITY work by Wang et al. (see table in the general comment) showing that our dataset covers a gap in terms of dataset size and context lengths required to solve it.\n\nOverall, especially with this addition, we believe that we make a reasonably strong case that our dataset will be useful for the community.\n\nTo be clear, we agree that a paper that would both propose a dataset and use it to improve the present state of Language Modeling would be ideal. But training or even fine-tuning and comprehensively evaluating state of the art LLMs demands extreme amounts of resources that we do not have. We now mention this fact in the limitations section.\n\n__Q: “Is there a comprehensive analysis comparing the quality, type and difficulty of questions here compared to those in other long document QA datasets? While the authors note that NarrativeXL is meant as a larger-scale alternative to NarrativeQA, if the questions are comparable in nature then there should be a proper quantitative analysis of where NarrativeXL shines in comparison.”__\n\nA: We have included descriptive statistics, highlighting the differences in scope and document length between our dataset and Narrative QA (please see our general comment). At the same time, while NarrativeQA shares similar goals with our work (long-range language understanding), the format of its questions is substantially different. In particular, Narrative QA is a Question Answering dataset, with answers being words or small phrases. In our case, the questions are either multiple choice (where the right scene summary has to be selected) or freeform (where a summary needs to be reconstructed). Therefore, a direct “side-by-side” comparison of question difficulty or quality is not quite possible.\n\n__Q: “Typos, grammar, and style comments''__ & __\"The presentation of the paper appears very scattered and difficult to read. A great deal of the writing also appears to come from a somewhat speculative angle and do not seem necessary.”__\n\nA: Thank you very much for your paper organization suggestions! We have adjusted the paper to accommodate most of them. For example, we moved the last two points from section 3 to the appendix, as they are less central to the argument, and also reorganized Table 1 to be more readable. We have also moved the table with confidence intervals to the appendix. In the main paper, we now have a condensed version (no confidence intervals) and a reference to the full version in the appendix. \n\nAs for question examples - to clarify, all multiple-choice read-along questions follow the same format as in Table 2, the difference is in how the “wrong” answer options are obtained. We clarified it in the paper and in the table description. We also included an additional illustration in the appendix to illustrate how different decoy answers were generated.\n\nLastly, we went through the paper, aiming to scale down the speculative claims and to indicate more clearly when our statements are speculative.\n\nThank you again for your time reviewing our article, as well as for your criticisms and suggestions. We hope that our rebuttal helps to address some of your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The problem is interesting and somewhat under-explored as of now.\n- The benchmark could be useful for testing long-context LLMs.", "reasons_to_reject": "While I appreciate the effort the authors put into the paper, I feel that there are a number of things that are lacking in this paper\n\n- The work focuses very heavily on the NarrativeQA dataset, which is a comparatively old dataset for which many other alternatives have been presented and are not really discussed anywhere in this paper.\n- The experiments somewhat lacking, with only one fine-tuned model (GPT-3). In particular, I would expect some long-context Transformers to be of interest here yet nothing is provided.\n- The presentation of the paper appears very scattered and difficult to read. A great deal of the writing also appears to come from a somewhat speculative angle and do not seem necessary.", "questions_to_authors": "- There is very little analysis of the dataset and the questions used within it. Is there a reason this is the case?\n- The work argues for a number of limitations of NarrativeQA, so it would be very useful to have a comprehensive analysis of how NarrativeQA might fail to address things that NarrativeXL might from an empirical perspective. These experiments would appear to be critical for this paper to demonstrate it's worth, so why are these not included?\n- Is there a comprehensive analysis comparing the quality, type and difficulty of questions here compared to those in other long document QA datasets? While the authors note that NarrativeXL is meant as a larger-scale alternative to NarrativeQA, if the questions are comparable in nature then there should be a proper quantitative analysis of where NarrativeXL shines in comparison.", "paper_topic_and_main_contributions": "The authors create a long-narrative QA dataset, NarrativeXL, aimed to be large enough to train LLMs. It also aims to provide better ways to evaluate how models perform in terms of long-term information retention. The authors provide results on GPT-3/4 to show the challenging nature of the benchmark.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "One obviously lacking comparison that is lacking is QuALITY (Wang et al., 2022). While the benchmark shares some differences with NarrativeXL, both aim to ostensibly address the same issue, which it the lack of long-document QA benchmarks\n\n```\n@inproceedings{pang-etal-2022-quality,\n    title = \"{Q}u{ALITY}: Question Answering with Long Input Texts, Yes!\",\n    author = \"Pang, Richard Yuanzhe  and\n      Parrish, Alicia  and\n      Joshi, Nitish  and\n      Nangia, Nikita  and\n      Phang, Jason  and\n      Chen, Angelica  and\n      Padmakumar, Vishakh  and\n      Ma, Johnny  and\n      Thompson, Jana  and\n      He, He  and\n      Bowman, Samuel\",\n    booktitle = \"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies\",\n    month = jul,\n    year = \"2022\",\n    address = \"Seattle, United States\",\n    publisher = \"Association for Computational Linguistics\",\n}\n```", "Typos_Grammar_Style_and_Presentation_Improvements": "- Section 2 feels like it could be condensed as simply a single paragraphs as part of an analysis or background section. The paragraphs go into very subjective analysis which don't particularly add to the paper in any meaningful manner.\n- Section 3 could be much better organized. Furthermore it focuses solely on NarrativeQA in a way that feels somewhat excessive and redundant with other parts of the paper.\n- Section 5 would be much easier to understand with a figure or table which gives examples of the different question types. \n\n- Tables 1 and 2 are somewhat dense in text and therefore difficult to parse. \n- Table 3 is very difficult to read, especially the confidence intervals which make it very confusing to the reader what values are most important to compare."}}, {"review_id": "EmT8lZzlOD", "reviewer": "Reviewer_wT87", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your time reviewing our paper and for your comments. Please note that, along with this individualized rebuttal, we have also added an official comment, summarizing our rebuttal contributions to all reviewers. \n\nBelow we respond to each of your concerns and suggestions.\n\n__Q: \"The actual statistics of the dataset are missing from the paper. We need to see a table with mean and stddev of the length of the prompts, as well as the targets.\"__\n\nA: We deeply apologize for not including overall dataset statistics in our original submission. We have now added large-scale dataset information comparing our dataset to existing alternatives in terms of the total number of documents, questions, and so on (please see our official comment). Additionally, the following information (and more detailed descriptive statistics and distribution visualizations) will be included in the appendix section of our paper:\n\n| Question type                        | Context length                  | Question length (words)        | Answer length (words)               |\n|--------------------------------------|------------------------------------------|-------------------------------------|-------------------------------------|\n| Read-along (multiple choice)         | mean=54334 (sd=48204, IQR[21932, 71294])  | mean=499.5 (sd=54.8, IQR[462, 534]) | 1 (multiple choice)                 |\n| End of book (summary reconstruction, free-form) | mean=87051 (sd=42682, IQR[59459, 98565]) | mean=104.4 (sd=41.7, IQR[83, 116])  | mean=90.0 (std=33.01, IQR[76, 106]) |\n\n*The context length is shorter for read-along questions since they are asked before the book is fully read.\n\n__Q: \"While the authors claim that they are mitigating one of the problems with current LLMs which states that they are not trained on log-document data, the authors do not train an LLM on their dataset. While I understand that the resources to accomplish that would be very high, it remains a limitation given the current presentation of the paper.\"__\n\nA: We agree with this criticism. We now acknowledge it in the limitation section of our paper. We also re-worked the phrasing of our paper throughout, making it more clear that we speculate (rather then decisively prove) that our dataset should help mitigate long-term memory problems of modern LLMs.\n\nQ: Missing reference\n\nA: Thank you very much for pointing this work to us. We have included a short discussion of Dangovski, R. et al. (2021) to section 7.2, and we have also added summary statistics for this dataset to the table that now summarizes other work addressing similar issues as ours (please see our \"rebuttal summary\" official comment). As can be seen from the table, we believe that this dataset is highly relevant, but also substantially different from ours to be considered a complementary, rather than competing work.\n\nThank you again for your time reviewing our paper. We appreciate your positive feedback and we hope that the newly provided statistics will improve the overall impression that our paper makes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The type of dataset is novel, and it addresses some of the issues with previous long-document dataset (e.g. context corruption, training data corruption, etc.)\n\n2. The evaluation experiments are meaningful and support the addition of that dataset for evaluation.\n\n3. The methodology could be used by the community to be scaled to other datasets.", "reasons_to_reject": "1. The actual statistics of the dataset are missing from the paper. We need to see a table with mean and stddev of the length of the prompts, as well as the targets. \n\n2. While the authors claim that they are mitigating one of the problems with current LLMs which states that they are not trained on log-document data, the authors do not train an LLM on their dataset. While I understand that the resources to accomplish that would be very high, it remains a limitation given the current presentation of the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors address the problem of lack of training and evaluation data for long-context LLMs. The paper presents a novel long-document dataset that consists of well-curated books from Project Gutenberg, which are chunked and fed into GPT 3.5 to create questions and labels that accompany the story. Through the process of hierarchical summarization, the authors create questions and labels that correspond to the whole book. The authors study the merits of their work by training GPT3 and evaluating Claude and GPT4 on the task in a variety of experimental settings.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Dangovski, R., Shen, M., Byrd, D., Jing, L., Tsvetkova, D., Nakov, P., & Soljačić, M. (2021, May). We Can Explain Your Research in Layman's Terms: Towards Automating Science Journalism at Scale. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 14, pp. 12728-12737). -- dataset with long scientific documents and their long press releases for abstraction summarization (for section 7.2)"}}, {"review_id": "fFDqrLlv60", "reviewer": "Reviewer_ms7Q", "summary": "", "questions": "Question A: Is your corpus from PG is filtered with existed PG19 datasets? I concern that almost all of the pre-trained models are using PG19. However, using as the training corpus doesn't affect the evaluation for long-term capacity. But if some models give more pre-training corpus percentage on existing PG datasets, if may contribute to better results. \n\nQuestion B: Are the answers generated by GPT-3.5 good enough to be a dataset source? For example, some comparisons with hand-labeled methods may explain a lot. I don't doubt the quality of GPT series, but it can't be the best model 5 years later.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for your comments! Below we would like to address some of your concerns.\n\n__Q: “I'm not sure that using GPT series to generate answers is a fair method for comparisons with other models. It may cause that GPT is always better than other models even though there are great improvements in open-source community.”__\n\nA: This is a good point, and we now mention it in the limitations. We believe, however, that it’s not as critical as it might seem, for a few reasons:\n\n1. We used GPT3.5 only to generate local scene summaries and their distorted versions, and we then used human labelers to validate that the summaries accurately represent original text. Generally, since short-scene-summarization performance is essentially at human level for most modern LLMs, we hoped that it would lower the impact of  specific model choice for summary generation. In the preliminary stages of our study, we tried a number of open-source models for summary generation, but GPT3.5 offered the best quality to resource ratio.\n2. Given how our data was generated, the only potential advantage for the GPT-family of models might come from stylistic differences in how GPT3.5 writes its local summaries. To see whether GPT3.5 has an advantage because of it, during the rebuttal period, we conducted an additional experiment for section 6.2 (Table 3), fine-tuning two openly available Longformer-Encoder-Decoder models (base and large versions), like we did with GPT3.5. They performed slightly better than GPT3.5 and GPT4 with no context, but worse than GPT4 with context. This again suggests that it’s the memory performance and not difference in writing style that determines model performance on our data.\n3. We generated long-term memory questions based on GPT-generated local summaries, but GPT3.5 does not actually have the memory capacity to answer the resulting questions. Even if there is some advantage to GPT3.5 because it was involved in data generation, this specific model won’t be able to capitalize on that.\n4. It is not clear how similar the GPT-based family of models are, compared to other LLMs. That is, it’s not clear whether GPT4 is similar enough to GPT3.5 to get any advantage on our dataset, compared to, say Claude or any of the open-source models.\n\nMost importantly, we see the main impact of our work in that it can be used to train and diagnose long-term memory models. Each question has a known memory retention demand, showing how much context the model needs to remember to answer it. We find that there is no good dataset for this at present. Even if the evaluation was slightly biased towards GPT models overall because of, say, stylistic differences in summary writing, it should not affect the shape of the forgetting curve.\n\n__Q: “Is your corpus from PG is filtered with existed PG19 datasets? I concern that almost all of the pre-trained models are using PG19. However, using as the training corpus doesn't affect the evaluation for long-term capacity. But if some models give more pre-training corpus percentage on existing PG datasets, if may contribute to better results.”__\n\nA: Thank you for pointing this out! We have not filtered PG19 dataset out, but we did now calculate the overlap percentage with PG19. We find that 53.6% of our books are also found in PG19.\n\nWe will release information about which books in our dataset are also in PG19. This way, it will be possible for models trained on our data to see if there is a difference in memory performance for books included in PG19 vs the rest.\n\nWe believe that this might be better than filtering out PG19 entirely. First, PG19 books are a good data source, second, having PG19 books in our data would help to better diagnose the source of memory improvements. For example, if a new long-term memory model only performs well on books it already knows from PG19, it would indicate that the memory mechanism needs substantial improvement, while if they perform nearly equally on PG19 and non-PG19 books, it would indicate actual ability to handle extreme-long-context documents.\n\nIn other words, since it is nearly impossible to avoid data contamination, it might be better to have the option to systematically evaluate its impact.\n\n__Q: “Are the answers generated by GPT-3.5 good enough to be a dataset source? For example, some comparisons with hand-labeled methods may explain a lot. I don't doubt the quality of GPT series, but it can't be the best model 5 years later.”__\n\nA: This is a great question. We conducted extensive experiments to make sure that our method results in good questions. It is important to remember that GPT3.5 only generated local summaries and their distorted versions, not full questions and their answers. Therefore, we only need local summary performance to be good. As described in section 6.1.3, we conducted a human-participant study to evaluate the quality of these local summaries.\n\nDuring the rebuttal period, we expanded this study to 25 highly experienced MTurk participants and 250 total summary evaluations. Participants were given a book snippet and a pair of GPT3.5-generated summaries (one accurate and one misrepresenting the data). Participants were able to identify the correct summary in 238 out of 250 cases, which corresponds to 0.95 accuracy (95% binomial confidence interval [0.92, 0.97]). It indicates that our data generation process was adequate for the task.\n\nOverall, we only apply GPT3.5 to a local task at which it is already extremely good. This, we hope, would ensure the longevity of our data even as better overall models are developed. That being said, we would be happy to see our methodology extended in the future using other, more advanced models.\n\nThank you again for your review and your comments. We hope that our response mitigates some of your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The dataset's building pipeline is novel and adequate for their claims. The question design is a smart way to evaluate real long-term performance.", "reasons_to_reject": "I'm not sure that using GPT series to generate answers is a fair method for comparisons with other models. It may cause that GPT is always better than other models even though there are great improvements in open-source community.", "questions_to_authors": "Question A: Is your corpus from PG is filtered with existed PG19 datasets? I concern that almost all of the pre-trained models are using PG19. However, using as the training corpus doesn't affect the evaluation for long-term capacity. But if some models give more pre-training corpus percentage on existing PG datasets, if may contribute to better results. \n\nQuestion B: Are the answers generated by GPT-3.5 good enough to be a dataset source? For example, some comparisons with hand-labeled methods may explain a lot. I don't doubt the quality of GPT series, but it can't be the best model 5 years later.", "paper_topic_and_main_contributions": "The paper collects a new dataset which can evaluate language model's long-term memory capability. The corpus is from Project Gutenberg and manually inspected to ensure quality. Then, the paper proposes two question types, read-along questions and summary correction questions to evaluate model's memory capacity and global understanding. The answers are generated by GPT 3.5 API and the dataset is easy to construct other questions.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As the invlved discussion with reviewrs shows, this is interesting and thought provoking work, however, there is consensus that the evalution as presented does not fully alleviate all the concerns of the reviewers.  The authors have been generous with the reviewers and put in the time to dilligently address the concerns raised, including imporving and expanding the human evaluation and the set of models tested, listing of data sets and more meaningful evaluation of the significance of their results. 

I'd like to thank both the authors and reviwers for their engadement to improve this work.
Decision: Accept-Findings