SlowGuess/NAACL_2022_OCR huggingface dataset:
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. Eli5: Long form question answering.
Henry Fielding. 1779. The History of the Adventures of Joseph Andrews., volume 1. J. Fr. Valade.
Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2021. Longt5: Efficient text-to-text transformer for long sequences.
Ankit Gupta and Jonathan Berant. 2020. Gmat: Global memory augmentation for transformers. arXiv preprint arXiv:2006.03274.
Curtis Hawthorne, Andrew Jaegle, Catalina Cangea, Sebastian Borgeaud, Charlie Nash, Mateusz Malinowski, Sander Dieleman, Oriol Vinyals, Matthew Botvinick, Ian Simon, et al. 2022. General-purpose, long-context autoregressive modeling with perceiver ar.arXiv preprint arXiv:2202.07765.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419-1436, Online. Association for Computational Linguistics.
DeLesley S. Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. 2022. Block-recurrent transformers. ArXiv, abs/2203.07852.
Daphne Ippolito, David Grangier, Douglas Eck, and Chris Callison-Burch. 2020. Toward better storylines with sentence-level language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7472-7478, Online. Association for Computational Linguistics.
KR Ireland. 1986. Towards a grammar of narrative sequence: The model of the french lieutenant's woman. Poetics today, 7(3):397-420.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Francois Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning.
Wojciech Krysciński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2021. Booksum: A collection of datasets for long-form narrative summarization.
Siyou Liu and Xiaojun Zhang. 2020. Corpora for document-level neural machine translation. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3775-3781, Marseille, France. European Language Resources Association.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: A robustly optimized bert pretraining approach.
Jerome Myers, Edward O'Brien, Jason Albrecht, and Robert Mason. 1994. Maintaining global coherence during reading. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20:876-886.
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel R. Bowman. 2021. Quality: Question answering with long input texts, yes!
Ofir Press, Noah A. Smith, and Mike Lewis. 2021. Shortformer: Better language modeling using shorter inputs. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5493-5505, Online. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53-68.
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. 2022. **Scrolls: Standardized comparison over long language sequences.**
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464-468, New Orleans, Louisiana. Association for Computational Linguistics.
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2020. Q-bert: Hessian based ultra low precision quantization of bert. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8815-8821.
Philip Stevick. 1970. The Chapter in Fiction: Theories of Narrative Division. Syracuse, NY: Syracuse University Press, c1970.
Pierre Stock, Angela Fan, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. 2021. Training with quantization noise for extreme model compression. In International Conference on Learning Representations.
Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Iyyer. 2021. Do long-range language models actually use long-range context? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 807-822, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020. Efficient transformers: A survey.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. *ArXiv*, abs/1807.03748.
Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks.
Dani Yogatama, Cyprien de Masson d'Autume, and Lingpeng Kong. 2021. Adaptive semiparametric language models. Transactions of the Association for Computational Linguistics, 9:362-373.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Otanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93-104, Brussels, Belgium. Association for Computational Linguistics.
A Dataset statistics
We collected 13,682 fanfictions from an online dump of stories posted on Archive of Our Own (AO3) by filtering works written in English language, rated General Audience by the author and contains at least 10K words and more than 10 chapters. For each chapter, we remove the text within the range of “ $\star \star$ Notes for the Chapter:  $\star \star$ ”, “ $\star \star$ Summary for the Chapter:  $\star \star$ ” and “ $\star \star$ Author's Note:  $\star \star$ ”. The meta/comments inserted into the main text by the authors are not removed. The statistics of this long-fic dataset are included in Table 3. We do not apply other profanity filters to the fictions, therefore there may still be inappropriate content for general audience as the rating is self-labeled by each author. Besides chapter breaks introduced in the main text, we also collected two other discourse boundaries, cause and dialogue, as comparisons to the chapter boundary examples. We present the statistics each type of examples in Table 4.
- Cause: The beginning of the suffix contains words or phrases 'because', 'due to', 'owing to'. According to (Albrecht and Myers, 1995), human readers reactivate memory of global context for comprehending statements following causes or goals.
- Dialogue: The gold suffix in this category starts with a quotation mark. This often happens in dialogues where the continuation of one interlocutor depends heavily on the immediately preceding utterance. We conjecture this is the type where the prediction relies more on the local rather than the global context.
mean	min	max
#chapters	21.5	11	589
#words	41,513.2	10,000	636,468
B Annotation Scheme
We annotate each chapter transition from four aspects: events, actors (point-of-view or characters in focus), location, and continuity in timeline.
Table 3: Statistics of long fanfictions collected from AO3 story dump.
AO3	PG19
Suffix Type	#works	#examples	#works	#examples
cause	965	8,133	45	506
dialogue	979	8,724	46	3,165
chapter breaks	1202	7,355	17	241
Table 4: Data statistics of CHAPTERBREAK as well as another two discourse boundary examples.
Events We define two subcategories based on whether (1) previous event ends in the previous chapter and new event starts in the new chapter, (2) old event does not end and continues into the next chapter.
Actors We define two subcategories based on whether there is a shift in POV or main character in focus.
Location We define two subcategories based on whether the location described in the prefix and in the new chapter is different.
Continuity Following Ireland (1986)'s work, we categorize the chapter transition by timeline continuity into four subcategories:
- Discontinuous but chronological: Reusing the standard by Ireland (1986), discontinuous represents a gap in time forward for more than one night.
- Continuous: The time interval between chapters lasts for no more than one night.
- Analepsis: Analepsis represents retrospective evocation of an event, or "flashback" to an earlier point in the narrative.
- **Parallel:** This includes timeline reverting back to the time of any previous chapter, typically accompanied by switching character in focus or description of a separate set of events independent of the last chapter. This category is a collapse of "alternate phase", "parallel phase" and "simultaneous phase" introduced in (Ireland, 1986).
Bigbird (Zaheer et al., 2020) To reduce the quadratic complexity of self-attention in the standard Transformer, the Bigbird model employs a mixture of global, random and local attention mechanisms, which successfully reduce the complexity
to linear. The idea is to insert each sequence  $O(1)$  global tokens, which attend to all other tokens. The rest tokens attend to their neighbor tokens, random tokens in the sequence as well as the inserted global tokens. A very similar idea is developed concurrently in the Longformer (Beltagy et al., 2020). The Bigbird model we fine-tuned is the decoder part of the released checkpoint. We fine-tune the model with causal LM objective on 14K books of PG-19 with peak learning rate 0.0001 for 100K steps. We set attention type to be "original_full" instead of using "block_sparse" during fine-tuning. Training is completed on a single RTX8000 GPU for around 6 days.
Local Transformer Rather than implementing all three types of sparse attention in Bigbird, the Local Transformer relies only on the local attention, i.e., each token attends to neighbors within a local window. The maximum attainable sequence length scales linearly with the number of layers, e.g., with window size  $k$ , the token representation at layer  $l$  theoretically covers information in a range of  $k \times l$  tokens.
Routing Transformer (Roy et al., 2021) Different from previously described models which use position-based sparse attention, the Routing Transformer employs content-based sparse attention. Namely, each token are routed to clusters and the attention is performed only within each cluster. The clustering operation effectively reduces the quadratic complexity in length  $L$  to  $O(L^{1.5})$ . Both the RT and LT checkpoint we used were trained on PG-19 (Rae et al., 2020). For both RT and LT, we evaluate on single RTX8000 GPU.
GPT-2/3 The GPT models have a lot shorter maximum input length than the rest models we evaluated. While GPT-2 model does not use sparse attentions at all, GPT-3 model adopts alternated layers of sparse and dense self-attention. We use the GPT-2 large model, which was pre-trained on data scraped from the Internet. The GPT-3 model was pre-trained on a mixture of filtered CommonCrawl, WebText2, Books1, Books2, and Wikipedia.
D Finding the best SuffixLM
As there are no prior long-range segment-level LM architectures that we can borrow from, we experiment multiple design choices and report the result of only the best performing one in the main text. For all variants, we use RoBERTa-base (Liu et al.,
Figure 4: Performance of each SuffixLM variant. Detailed information about each variant is included in Appendix D.
2019) as the encoder to obtain the encoded segment representation. This is done by extracting the representation of the  $[CLS]$  token prepended at the beginning of each sequence. We describe five variants below.
- SuffixLM-A This variant contains a frozen RoBERTa-base encoder and a SuffixLM using a 6-layer Transformer as the base architecture.
- SuffixLM-B This variant contains a frozen RoBERTa-base encoder and a SuffixLM using a 6-layer average-attention Transformer as the backbone. The motivation of using uniform distribution for attention weights is to encourage the model to get more information from the distant context rather than rely too much on local context.
- SuffixLM-C This variant is essentially SuffixLM-A but during training we perform "segdrop" - stochastically dropping prefix segments with probability  $0.2^{17}$  when performing self-attention. When the local segments are dropped, the model has to predict the next segments with only the distant context, which also encourages learning better long-range prefix representations.
- SuffixLM-D Instead of freezing the encoder, this variant fine-tunes part of the encoder and the rest is the same as SuffixLM-A. Due to limited memory capacity, we only fine-tune the last two layers of the RoBERTa-base.
- SuffixLM-E This model is the same as SuffixLM-D except that we truncate the en
$^{17}$ Tried  $\{0.1, 0.2, 0.4\}$ ,  $0.2$  works the best.
Evaluation on PG19
Evaluation on AO3
Figure 5: Evaluation results on both CHAPTERBREAKPG19 and CHAPTERBREAKAO3.
coder to just the two tunable layers and train all parameters in the encoder including the embedding parameters.
All SuffixLMs with frozen encoders are trained with average sequence length of 10240 tokens for up to 60k steps, and the one with trainable encoder is trained for max 120k steps. The dimension of the model is 768, hidden dimension 2048, attention heads 8. The peak learning rate is 0.0001 with warm up steps 4000. We train SuffixLM on entire PG-19 dataset and evaluate the best checkpoint selected by dev loss. We use segment size 128 in all SuffixLMs we trained. Each segment starts from a new sentence, if not reaching 128 tokens, we pad with a special `<pad>` token. For very long sentences, the part exceeding 128 tokens overflows to the next segment. We plot the suffix identification accuracy of each variant on CHAPTERBREAK while feeding in prefixes of increasing length. As shown in Figure 4, SuffixLM-E outperforms all other variants across various prefix lengths. Therefore in the main text, all SuffixLM refers to the SuffixLM-E variant. Note that one limitation of SuffixLM is it exclusively models on segment-level, which prohibits it from performing token-by-token generation and thus impossible for us to evaluate perplexity.
E Suffix perplexity
Although the task of CHAPTERBREAK is to identify gold suffix from negatives, we also present the gold suffix perplexity of next-token prediction LMs. Note that all models were trained or fine-tuned on PG-19 except for GPT-2/3. As these models use different tokenizers, the 128-token suffix may cover different number of words, to make the results comparable, we convert the subword-level perplexity to word-level by multiplying a constant to the log probability value of each model. For RT/LT, we multiple by 1.248 as used in the official repository. We multiply the value by 1.30 for GPT-2, and 1.22 for Bigbird. These values are estimated via the subword/word ratio on validation set of PG-19. Our fine-tuned Bigbird model achieves the lowest perplexity on PG-19, even better than Routing Transformer or Local Transformer. This implies that context from long-range is not necessary for achieving low perplexity since the maximum input length of Bigbird is half that of RT/LT.
F In-book vs. Out-of-book
This section is better read after reading through  $\S 3$ . In this analysis experiment, we show why it is better that the negatives are from the same narrative as the gold suffix. We evaluate our upper bound
Figure 6: Left: In-book vs. out-of-book. Right: SuffixLM performance when evaluated with different suffix length. The variation in suffix length does not explain the large gap between SuffixLM and token-level LMs.
model SuffixLM on PG-19 set when the negatives are out-of-book suffixes, and plot the suffix identification accuracy in Figure 6. When evaluate against out-of-book negatives, this suffix identification task is almost solved by our SuffixLM, especially when the out-of-book examples are from another split in CHAPTERBREAK. The extremely high accuracy under out-of-book setup suggests the segment representation from different books are easy for SuffixLM to distinguish, thus we adopt a harder setup where the negatives are from the same book. Besides, in-book negatives may contain the same re-occurring named entities or rare words, which require solid understanding of the prefix to differentiate the gold from the distractors.
G Various Discourse Relationships
In addition to chapter breaks, we also evaluate the other two types of discourse boundary examples introduced in Appendix A. As shown in Figure 5, for all suffix types other than chapter breaks, the evaluated models stop improving as the sequence length grows to more than 2K tokens long. However, there is a significant increasing trend in chapter breaks for SuffixLM. For the rest models, the performance is either flat or not improving. On the AO3 split, the accuracy of SuffixLM improves for  $\sim 15\%$  as the sequence length increases from 256 to 8K, whereas the improvement of RT is only  $\sim 1.4\%$ . This is in contrast with SuffixLM's  $\sim 1.5\%$  and RT's  $\sim 0.3\%$  improvement for the 'cause' examples. We draw two conclusions from these observations: (1) the chapter breaks examples form a special case where longer prefix is preferred in order to pick the correct continuation. (2) By comparing the relative improvement, the token-level LMs fall far behind the SuffixLM, which is, besides the absolute performance gap, another evidence that current
LRLMs do not effectively leverage long-range context for sequence tasks requiring discourse-level understanding.
H Tackle difference in Tokenizers
As the models we evaluated use different tokenizers, there are small variations in term of suffix length, i.e., the 128-token suffix may cover different number of words. To understand how the difference in length impacts validity of evaluation, we evaluate SuffixLM with various suffix lengths. Figure 6 (right) indicates even though there are small variances when the suffixes are of different lengths, the large gap between SuffixLM and Routing Transformer still remains, thus the difference in suffix length does not explain the large performance gap.
I Error analysis
Models struggle with location and event shifts: Among the 300 examples we annotated in Section 2, 89 examples were wrongly predicted by all models we have evaluated. By breaking the incorrectly predicted examples into category as presented in Table 1, we find that models tend to make wrong prediction when there is a shift in location or event, and when plots are continuous in timeline.[18]
Category	Definition	Ratio
Events	Previous event ends and new event starts	0.74
Previous event continues into next chapter	0.26
Actors	Change of perspective or character in focus	0.43
No change in POV or main character	0.57
Locations	Change of location	0.64
No change in location	0.36
Continuity	Discontinuous but chronological	0.24
Continuous	0.62
Analepsis	0.03
Parallel	0.11