SlowGuess/NAACL_2022_OCR huggingface dataset:
CHAPTERBREAK: A Challenge Dataset for Long-Range Language Models
Simeng Sun Katherine Thai Mohit Iyyer
{simengsun,kbthai,miyyer}@cs.umass.edu
While numerous architectures for long-range language models (LRLMs) have recently been proposed, a meaningful evaluation of their discourse-level language understanding capabilities has not yet followed. To this end, we introduce CHAPTERBREAK, a challenge dataset that provides an LRLM with a long segment from a narrative that ends at a chapter boundary and asks it to distinguish the beginning of the ground-truth next chapter from a set of negative segments sampled from the same narrative. A fine-grained human annotation reveals that our dataset contains many complex types of chapter transitions (e.g., parallel narratives, cliffhanger endings) that require processing global context to comprehend. Experiments on CHAPTERBREAK show that existing LRLMs fail to effectively leverage long-range context, substantially underperforming a segment-level model trained directly for this task. We publicly release our CHAPTERBREAK dataset to spur more principled future research into LRLMs.<sup>1</sup>
Research on long-range language models (LRLMs) aims to process extremely long input sequences by making the base Transformer architecture more efficient (e.g., through sparse attention, recurrence, or cached memory). These modifications are commonly validated by training LRLMs on PG19 (Rae et al., 2020), a long-document language modeling dataset, and demonstrating small perplexity decreases over shorter context models (Roy et al., 2021;?). However, recent analysis experiments (Sun et al., 2021; Press et al., 2021) show that modern LRLMs rely mostly on local context (i.e., the immediately preceding 1-2K tokens) and are insensitive to various perturbations applied to more distant context.
Figure 1: An illustrative example of our suffix identification task from Kurt Vonnegut's Slaughterhouse Five, in which an LRLM needs to make connective inferences across temporal and spatial shifts in a long prefix of the narrative to correctly disambiguate the  $(+)$  start of the next chapter from  $(-)$  negative examples.
(+) Billy Pilgrim could not sleep on his daughter's wedding night. He was forty-four... [ground-truth start of ch. 4]
(一) Billy Pilgrim says that the Universe does not look like a lot of bright little dots to the creatures from Tralfamadore...
(一) All the trains were slow. The coaches stunk of coal smoke and rationed tobacco and rationed booze and the farts of people eating wartime food.
In this paper, we move beyond token-level perplexity by evaluating LRLMs on a task that requires a rich understanding of long-range dependencies. Our task is an instance of suffix identification, in which a language model is given a long input sequence (or prefix) and asked to disambiguate the next  $n$ -token segment from a set of hard negatives sampled from the same narrative. To succeed at this task, an LRLM should assign high probability to the ground-truth next segment and low probability to the negatives. To specifically test long-range dependencies, we restrict our prefixes to end at chapter breaks of a longer cohesive narrative (e.g., a novel).
We construct a challenge dataset, CHAPTER-BRACK, by automatically detecting chapter bound
<sup>1</sup>We make our code and data public at https://github.com/SimengSun/ChapterBreak
Human Language Technologies, pages 3704 - 3714
aries within both held-out PG-19 documents (in-domain for pretrained LRLMs) and works of fan fiction published on the Archive of Our Own (out of domain). We perform a detailed analysis of the types of chapter transitions in our dataset and discover a high frequency of narrative shifts in point-of-view, location, and time, all of which require global narrative understanding over long input sequences. For example, Figure 1 contains a complex prefix in which the time-traveling Billy Pilgrim moves between World War II, 1960s suburban life, and an alien planet. Understanding the cliffhanger ending, in which the narrative abruptly switches from a wartime scene to a 1967 alien abduction, requires an LRLM to make connective inferences using details buried far back in the context (e.g., Billy's age in 1967).
We evaluate three LRLMs on CHAPTERBREAK, including BigBird (Zaheer et al., 2020), the Routing Transformer (Roy et al., 2021), and its local attention variant, all pretrained or fine-tuned on PG-19. Our experiments show that these LRLMs perform poorly at selecting the ground-truth suffix, regardless of the length of the input sequence. As an upper bound, we train a small RoBERTa-based segment-level language model on PG-19 and discover that it substantially outperforms all LRLMs on CHAPTERBREAK, which suggests that LRLMs have considerable room for improvement on this suffix identification task. Finally, we perform an analysis on the instances for which all models struggle to choose the correct suffix, which shows that shifts in location and events in focus are particularly challenging to disambiguate. Taken together, these results suggest that CHAPTERBREAK is a useful benchmark for future research into LRLMs.
2 The CHAPTERBREAK dataset
Authors often break long-form narratives into a sequence of discrete chapters to impose "an order and shape over events in time" (Stevick, 1970). Henry Fielding writes in his novel Joseph Andrews that the space between chapters is like "an Inn or Resting Place" for readers to reflect on the preceding chapter (Fielding, 1779). Chapters come in many flavors: for example, Murakami's Kafka on the Shore uses chapter breaks to alternate between parallel narratives focusing on the two protagonists, while cliffhanger endings such as the one in Figure 1 add suspense. Making sense of the complex
narrative shifts associated with chapter transitions (e.g., changes in point-of-view, time, location, and theme) requires a deep understanding of the entire text. To maintain global narrative coherence, Myers et al. (1994) show that human readers tend to reactivate memory about "backgrounded" information from the long-range context.
Task overview: Given that chapter transitions requires global context understanding, how can we turn this into a task to evaluate LRLMs? A simple approach is to evaluate the token-level perplexity of an LRLM only at chapter boundaries (i.e., on the first  $n$  tokens of each chapter); however, the vast majority of tokens can be predicted using just local context (Sun et al., 2021) under the teacher-forcing setup, which obscures an LRLM's usage of long-range context as we show in Section 3. We instead turn to the task of suffix identification, which closely resembles existing datasets such as SWAG (Zellers et al., 2018).
Each instance of our task is defined by a triplet  $(c, s^{+}, s_{i}^{-} \in \mathbf{N})$ , where  $c$  is a prefix sequence of up to 8K tokens that ends at a chapter break,  $s^{+}$  is the gold suffix of length 128 tokens (i.e., the beginning of the next chapter), and  $s_{i}^{-}$  is a negative 128-token-long suffix from a set  $\mathbf{N}$  of five<sup>3</sup> future chapter beginnings sampled from the same narrative.<sup>4</sup> All negatives are modified to begin with the same chapter index (e.g., if the gold suffix begins with "Chapter III", the chapter indices of all negatives is set to "Chapter III") to eliminate the effect found by Sun et al. (2021) of language models memorizing chapter indices in long contexts. We then evaluate whether an LRLM assigns higher probability to the gold suffix  $P(s^{+}|c)$  than to all negative suffixes  $P(s_{i}^{-}|c)$ .
Dataset overview: Where do we get these triplets from? We collect a dataset, CHAPTER-BREAK, with two splits: CHAPTERBREAKPG19, which contains 241 examples extracted from the PG-19 validation set (Rae et al., 2020), and CHAPTERBREAKAO3, which contains 7,355 ex
3We use a small number of negatives because it is time-consuming and resource-intensive to evaluate the probabilities of long sequences with LRLMs.
<sup>4</sup>In Appendix F, we show that in-book negatives are much harder than out-of-book negatives as they often contain the same named entities and rare tokens as the gold suffix. Thus, disambiguating the correct suffix requires a deep understanding of the context.
We only collect examples from validation set as two baseline models in the later sections are trained on PG-19.
$^{2}$ https://archiveofourown.org
Category	Definition	Pct.
Events	Previous event ends and new event starts	76%
Previous event continues into next chapter	24%
Actors	Change of perspective or character in focus	36%
No change in POV or main character	64%
Locations	Change of location	68%
No change in location	32%
Continuity	Discontinuous but chronological	29%
Continuous	62%
Analepsis	2%
Parallel	6%
samples extracted from an online dump<sup>6</sup> of fanfiction posted on Archive of Our Own (AO3). We apply filtering to remove fanfiction works that are too short or not rated for general audiences. Each work contains on average 42K words and 21.5 chapters.<sup>7</sup> Even though the CHAPTERBREAKPG19 split is small, we include it because many LRLMs are pretrained on PG-19; the much larger CHAPTERBREAK  $AO3$  split is out-of-distribution for all models that we evaluate. To extract chapters in PG-19, we match for lines beginning with the string "chapter", while AO3 stories already have chapter-level metadata.
What are the different types of transitions in CHAPTERBREAK and how often do they occur?
To get a better sense of our dataset, we perform a fine-grained annotation of 300 randomly-selected chapter transitions from CHAPTERBREAK  $AO3$ . For each transition, we annotate any changes in the following four aspects: events, actors (characters in focus), locations, and continuity. To annotate continuity, we follow a simplified version of the scheme proposed by Ireland (1986), which considers five categories: continuous (the next chapter occurs within a day of the previous chapter), discontinuous (the next chapter occurs more than a day after the previous chapter), analepsis (the next chapter is a "flashback" to an earlier point in the narrative), and parallel (the next chapter reverts to the time of a previous chapter, switching the
Table 1: Our human annotation on 300 chapter transitions randomly sampled from CHAPTERBREAK  ${}_{AO3}$  shows the diversity and complexity of the dataset.
#Params	Seq Len	PPLPG19	AccPG19	AccAO3
LT	516M	8K	76.8	25%	24%
RT	490M	8K	72.3	22%	24%
Bigbird	128M	4K	56.2	27%	26%
GPT-2	1.5B	1K	78.2	23%	24%
GPT-3	175B	2K	-	36%*	28%*
SuffixLM	87M	10K	-	52%	41%
Table 2: Summary of LRLMs (top), Transformer LMs (middle), and our SuffixLM (bottom). All models are trained or fine-tuned on PG-19 except for GPT-2. The third column shows the word-level perplexity of gold suffix in the PG-19 split. The last two columns show the suffix identification accuracy of each model on the two CHAPTERBREAK splits when evaluated at maximum input length. * indicates results are on a subset of CHAPTERBREAK.
Figure 2: Suffix identification accuracy on both splits (PG-19 and AO3) of CHAPTERBREAK is much lower for LRLMs than our SuffixLM upper bound.
character or event in focus). The results, shown in Table 1, demonstrate that CHAPTERBREAK covers a diverse array of transitions, including many that require global narrative understanding.
We evaluate three different long-range language models on CHAPTERBREAK and compare their results to those of standard Transformer language models as well as an upper bound directly trained for suffix prediction.
Language models: We evaluate three LRLMs pretrained on PG-19: the Local Transformer (Roy et al., 2021, LT), Routing Transformer (RT) (Roy et al., 2021, RT), and BigBird (Zaheer et al., 2020). The BigBird model is the decoder part of the released checkpoint fine-tuned with causal LM objective on 14k books of PG-19 for 100k steps. We also evaluate two standard Transformer language models, GPT-2 large (Radford et al., 2019) and GPT-3 (Brown et al., 2020).<sup>10</sup> We summarize these
$^{9}$ Appendix B contains more details about each category.  $^{10}$ Due to OpenAI's API costs for GPT-3, we only evaluate in total a subset of 200 examples instead of the full dataset.
https://archive.org/download/A03Story_dump Continuing
7More preprocessing details and statistics can be found in Appendix A.
To validate our continuity annotations, we also annotate every chapter in Pride and Prejudice and obtain almost the same proportion of continuous transitions (67%) as the number reported by the expert annotation of Ireland (1986) (72%).
models in Table 2, more details about each model are included in Appendix C.
An upper bound directly trained for suffix identification: As authors often write stories that are intended to surprise readers, it is possible that many examples in CHAPTERBREAK are ambiguous by nature (i.e., the upper bound for suffix identification accuracy may not be  $100\%$ ). To obtain a reasonable upper bound, we also train a model (SuffixLM) directly on the suffix identification task by scaling up the sentence-level language model proposed by Ippolito et al. (2020).<sup>11</sup> We divide an input sequence into multiple segments, each of which is embedded via the [CLS] vector of a small finetuned RoBERTa network (Liu et al., 2019). Our SuffixLM then performs "language modeling" atop the dense [CLS] vectors, predicting the next segment representation given the representations of previous segments via contrastive predictive coding (van den Oord et al., 2018).<sup>12</sup> Formally, our SuffixLM minimizes the following loss:
$$
\mathcal {L} _ {i} = - \log \frac {\exp (\hat {\mathbf {z}} _ {i} ^ {\top} \mathbf {z} _ {i} ^ {+})}{\sum_ {\mathbf {z} _ {i} \in \{\mathbf {z} _ {i} ^ {+} , \mathcal {Z} _ {i} ^ {-} \}} \exp (\hat {\mathbf {z}} _ {i} ^ {\top} \mathbf {z} _ {i})}
$$
where  $\hat{\mathbf{z}}_i$  is the predicted representation by SuffixLM,  $\mathbf{z}_i^+$  is the gold suffix representation obtained from a small encoder (RoBERTa), and  $\mathcal{Z}_i^-$  is the set of dense representations of the negatives. More details about our SuffixLM are included in Appendix D.
Overall, the results in Table 2 (rightmost two columns) confirm that all of the language models studied in this paper struggle on CHAPTERBREAK, especially when compared to the SuffixLM upper bound, which outperforms the best LM by  $\sim 25\%$  absolute accuracy when evaluated on the entire PG-19 split. We describe other interesting results and analysis below:
Accuracy increases with longer prefixes: Figure 2 shows that as prefix sequence length increases, some LRLMs (e.g., LT) barely improve, while others show modest improvements (e.g.,
Figure 3: Left: Prefixes ending at chapter breaks benefit more from long-range context than other types of discourse boundaries. Right: Word-level perplexity of the gold suffix does not correlate to accuracy (e.g., GPT-2 has high perplexity but outperforms RT on suffix identification).
GPT-3 and fine-tuned BigBird). However, all LRLMs significantly underperform our SuffixLM upper bound, even when the SuffixLM is given prefixes that are only 256 tokens long. Additionally, SuffixLM's accuracy increases far more than those of LRLMs when increasing the prefix length (from  $31\%$  at prefix length of 256 to  $46\%$  at 8K on the AO3 split<sup>13</sup>). This result suggests that the token-level LRLMs evaluated in our work are not taking full advantage of information in the long-range context to solve CHAPTERBREAK.
Perplexity does not always correlate with accuracy: Previous LRLM efforts use validation perplexity (e.g., on PG-19) to compare against other models. However, we show that perplexity is not by itself a predictor of suffix identification accuracy: As shown in Table 2, GPT-2 achieves higher accuracy than RT despite yielding a word-level perplexity of 78.2 on gold suffixes, compared to 72.3 for RT.[14] We advocate that future research on LRLMs includes evaluation on suffix identification tasks like CHAPTERBREAK, as perplexity alone does not reflect LRLMs' capabilities to model long-range dependencies.
Why chapter breaks over other discourse boundaries? Other discourse markers, including cause and dialogue, also often prompt human readers to reactivate memories of global context (Albrecht
11Our SuffixLM can process up to 10K tokens, while the model of Ippolito et al. (2020) supports only up to ten sentences.
<sup>12</sup>Our SuffixLM is closely related to the model in Ainslie et al. (2020), but differs crucially by predicting the representation of next segment instead of summaries.
13We collected 13,682 fan-fictions posted on AO3 and fine-tuned our SuffixLM on subset of this dataset to be the model SuffixLM $_{AO3}$ . More details about the filtered AO3 works are included in Appendix A
<sup>14</sup>As these models use different tokenizers, we normalize the subword-level perplexities to the word level as suggested by Rae et al. (2020). More details about this can be found in Appendix E.
and Myers, 1995). We create suffix identification datasets for these two discourse markers by string matching over corresponding cue phrases ('because', 'due to' for the cause subset and text within quotation marks for dialogue).[15] Figure 3 (left) shows that with prefixes of length 256 tokens, our SuffixLM is able to successfully disambiguate the correct suffixes for both discourse markers more than  $80\%$  of the time, while the accuracy is much lower at chapter boundaries. As the prefix length increases, accuracy only slightly increases for cause and dialogue, especially compared to the robust improvement at chapter boundaries.[16]
Short-context Transformers are comparable to LRLMs: Our results show that GPT-2, despite its high perplexity on gold suffixes and short maximum sequence length (1024 tokens), achieves comparable performance to RT and LT on both splits. Meanwhile, GPT-3 achieves much higher performance on both CHAPTERBREAK at a sequence length of 2,048 tokens, and the increasing GPT-3 curve in Figure 2 is promising for future work scaling LMs to longer sequence lengths.
Limitations of our work: While we have used the SuffixLM as an upper bound in this paper and demonstrated that it substantially outperforms LRLMs on CHAPTERBREAK, a more compelling comparison would include human performance on our task at varying prefix lengths, especially since some chapter transitions are specifically intended by their authors to be unpredictable. However, obtaining reliable human performance numbers is very difficult, as it requires in-depth comprehension of long narratives on the part of workers. Due to the time-consuming nature of this task and its high cognitive demand, it is not possible (within a reasonable budget) to use crowdsourcing, as ensuring that the annotators fully read the prefix instead of skimming or ignoring it is a major challenge. These issues also carry over to experiments performed with in-person subjects. As such, we leave a thorough human evaluation on CHAPTERBREAK to future work.
Our work depends heavily on recent advances in efficient Transformers (Tay et al., 2020) that pro
cess long sequences (Rae et al., 2020; Beltagy et al., 2020; Zaheer et al., 2020; Ainslie et al., 2020; Roy et al., 2021). Sparse attention (Child et al., 2019), relative position encoding (Shaw et al., 2018; Raffel et al., 2020; Guo et al., 2021), recurrence mechanism and memory (Dai et al., 2019; Weston et al., 2015; Hutchins et al., 2022; ?) and other tricks (Shen et al., 2020; Katharopoulos et al., 2020; Gupta and Berant, 2020; Stock et al., 2021; Yogatama et al., 2021; Borgeaud et al., 2021; Hawthorne et al., 2022) are commonly adopted by recent Transformer variants to make the operation on long sequences more time/memory efficient.
Besides perplexity, many downstream extrinsic tasks for evaluating long-range language models were developed recently, such as long-form QA (Fan et al., 2019; Pang et al., 2021), document-level summarization (Krysciński et al., 2021; Huang et al., 2021), and machine translation (Liu and Zhang, 2020). More recently, Shaham et al. (2022) introduce a new benchmark covering multiple domains and tasks, while Tay et al. (2021) propose multimodal long sequence tasks.
We introduce CHAPTERBREAK, a suffix identification dataset targeted at evaluating the discourse-level understanding of long-range language models. The dataset is extracted from long-form narratives and covers a variety of complex chapter transitions, such as shifts in location and events in focus. Experiments show that existing LRLMs perform poorly on CHAPTERBREAK and much worse than a SuffixLM trained as an upper bound on this task. We release the dataset to spur more principled development of future LRLMs.
We thank the anonymous reviewers and UMass NLP group for the thoughtful comments on the draft of this paper. We are grateful to AO3 Support Chair and volunteers for answering data related questions. This work was supported by awards IIS-1955567 and IIS-2046248 from the National Science Foundation (NSF).
CHAPTER BREAK is constructed from two sources: public domain books published prior to 1919 (from the held-out set of PG-19) and works of fanfiction extracted from an online dump of stories posted
15 Appendix A contains more details about data for these two discourse markers. 16 Appendix G shows similar trends on cause and dialogue with other models.
on Archive of Our Own (AO3). We refer readers to Rae et al. (2020) for more details about PG-19. For AO3, we apply multiple filters to obtain long fanfiction stories rated as suitable for "General Audiences". We refer readers to Appendix A for more preprocessing details. More generally, this work focuses on long-range language models, which could potentially be misused to generate offensive output. However, the main purpose of this paper is to present a dataset which provides a better evaluation of the discourse-level capabilities of such models.
Joshua Ainslie, Santiago Ontonon, Chris Alberti, Va-clav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. Etc: Encoding long and structured inputs in transformers.
Jason E. Albrecht and Jerome L. Myers. 1995. Role of context in accessing distant information during reading. Journal of experimental psychology. Learning, memory, and cognition, 21 6:1459-68.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2021. Improving language models by retrieving from trillions of tokens. arXiv preprint arXiv:2112.04426.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy. Association for Computational Linguistics.