SlowGuess/NAACL_2022_OCR huggingface dataset:
While both are adopted as common practice in existing work (Zaheer et al., 2020; Beltagy et al., 2020), we study the isolated effect of each component in both pretraining and finetuning experiments. For the non-overlap variant, we increase the block size by a factor of 2 such that the amount of tokens each token attends to remains the same. We show the results in Table 4. Surprisingly, we see different stories in terms of MLM pretraining and downstream tasks. While both mechanisms are useful for achieving lower MLM perplexity, only the global-token mechanism seems important for downstream tasks. Note that in the document retrieval tasks, removing both mechanisms results in slightly better performance. Now the model is only able to use the first block of the whole document for retrieval. While this seems to suggest that this task is highly local and involves strong positional bias<sup>8</sup>, the gap might be too trivial to be conclusive. Additionally, we only use a single global token for this task, it is likely that assigning more global tokens, e.g., at passage boundaries, could bring additional improvements. Investigating the particular task further is beyond scope of this work. In terms of the effect of attention-window overlap, it is expected that this scheme is crucial for lower perplexity: it not only enables more distant dependencies but also reduces the number of "boundary tokens" which can only attend to one side of the context. However, it is counter-intuitive that the
The answer context appears at the beginning of the Wikipedia page.
Model	MLM PPL	TriviaQA	NQ Doc Retrieval	Hyperpartisan
Blockwise LW	3.39	68.1/72.9	85.0	95.0
- w/o overlap	3.52	68.4/73.2	86.3	96.5
- w/o overlap & global tokens	3.54	56.5/61.0	85.4	94.6
overlapping attention links between neighboring blocks, which adds more long-range information, result in worse downstream performance. Also, note that this observation is consistent for all the tasks we studied. There are two possible implications of this finding: 1) the tested tasks still highly depend on locality bias, i.e., most of the important information can be captured solely from the local bias, or 2) the overlapping scheme is not effective at capturing the long-range dependency in downstream tasks. To confirm either hypothesis, we conduct another set of experiments with models that have access to different sizes of context.
On Locality Bias. We take the non-overlapping variant and experiment with various block sizes to see whether longer context is important to studied tasks. We show the results in Table 5 and the pretraining curves in Figure 2. While the long-range connections brought by the attention overlap is not helpful for downstream results, we see that increasing the local block sizes does consistently improve both pretraining and downstream performance although the improvement becomes modest beyond block size 256. It is also interesting that the models with smaller block sizes converge faster at the early stage of pretraining. This suggests a staged pretraining process might be more efficient than directly training from long sequences, which aligns with Press et al. (2021)'s finding on unidirectional LMs. Overall, this set of experiments suggests that increasing model's capabilities to capture a longer context is generally helpful for both pretraining and downstream tasks. However, using overlapping attention windows is not an effective way to make use of more context. Thus, we hypothesize the MLM perplexity improvements of overlapping local attentions might mainly come from the reduction of the "boundary" tokens instead of the ability to capture long-range dependencies. For downstream tasks, the issue of "boundary" tokens is not that essential and the introduction of the overlapping attention windows might disrupt the effective modeling of local context, as the attention module needs to extract both local and distant information from the
Table 4: Ablation of the Blockwise LW Model.
BlockSize	Val PPL	TriviaQA Ans F1
64	4.16	68.9
128	3.74	70.7
256	3.52	73.2
512	3.39	73.5
Figure 2: Pretraining curves of the non-overlapping block attentions with various context windows.
same set of tokens.
Initializing from Existing Short Models. While we train all models from scratch for the sake of fair comparison, existing state-of-the-art long context models like Longformer (Beltagy et al., 2020) or BigBird (Zaheer et al., 2020) usually initialize their longer models from an extensively pretrained short model like RoBERTa (Liu et al., 2019). With simple techniques like positional embedding copying, a strong long-context encoder can be initialized without the need of pretraining from scratch. To test our findings from the above analysis in this setting, we follow the same scheme
Table 5: Pretraining and long-doc QA results of the non-overlapping blockwise attention.
BlockSize	Speed ↑	Ans EM/F1
Longformer (64k)	6.6k	73.1/77.8
Blockwise LW w/o overlap (64k)	14.8k	73.2/77.9
Table 6: Comparing with Longformer with TriviaQA when initializing the models from RoBERTa. Speed is measure by thousand word per second at pretraining.
As the depth of the model increase, the tokens' representation will be added information of more distant tokens.
but use the non-overlapping block attention as discussed in §4.3. We compare this model with Longformer (based on Sliding Window attention) as it uses the same long-doc corpus and pretrain-and-finetune pipeline (e.g., packages and downstream data processing) as our experiments. $^{10}$  Same as our setting in §4.2, here we control the batch size and number of training updates: we use a batch size of 64 and train the model for 64k steps. Note that as we drop the attention window overlaps, the model is  $2\mathrm{x}$  more efficient than Longformer: Given the same window/block size  $B$  and sequence length  $L$ , the complexity of the non-overlapping block attention is  $L \times B$  compared to Longformer's  $2L \times B$ . We show the TriviaQA results in Table 6, where the speed is measured by words per second during pretraining. With only half of the pretraining compute, our model with disjoint attention blocks achieves slightly better performance than Longformer. This confirms that our findings of the attention overlap from the above section are still valid when the models are not trained from scratch.
Various studies have investigated the effective usage of distant context in unidirectional language models. Khandelwal et al. (2018) look into the context usage of LSTM LMs and find that these models are only capable to make full use of the nearby 50 tokens and the longer range context is only roughly captured, i.e., excluding detailed information such as word orders. Similarly, O'Connor and Andreas (2021) studies the mid- and long-range context usage in transformer LMs, by manipulating the ordering and lexical information in the text. Their experiments show that while long-range context is usually helpful, most of the usable information is carried by local ordering statistics and non-function words instead of detailed content like sentence orders. These observations provide a possible explanation of our ablation experiments in §4.3 that adding overlaps to attention windows does not yield better downstream results, despite allowing the capture of more long-range interaction. Press et al. (2021) observe diminishing returns as they increase the context length when using sliding windows at infer
ence time. They propose a staged training paradigm that train LMs from smaller context to longer ones. This paradigm can more efficiently use the training compute and achieves lower perplexity compared to directly training with long sequences. Given that models with smaller attention windows converge faster at early training steps (Figure 2), the staged training might also benefit MLM pretraining but further investigation is required to validate whether it can also bring downstream improvements.
Other Long-Range Architectures. Instead of modifying the attention calculation, other work proposes to augment transformers with parametric long-term memories. Transformer-XL (Dai et al., 2019) maintains frozen activations of previous tokens in memory and uses them as additional inputs. To handle the shift of positional information of these activations, it also requires a relative position encoding mechanism which brings additional computation cost. The Compressive Transformer (Rae et al., 2020) takes a similar scheme but proposes to use compression modules to account for even further memories. Both methods cannot be directly applied to long-context understanding tasks. Under the scheme of kernel-based methods, Katharopoulos et al. (2020); Peng et al. (2021); Schlag et al. (2021) also attempt to linearize the softmax with kernel methods. The core ideas of these methods are similar to Performer and they only differ in the choice of kernel functions. Outside of the transformer families, a recent work (Lei, 2021) proposes to augment recurrent LMs with minimal attention blocks. It is more efficient while achieving stronger LM perplexity compared to Transformer-XL. However, it is still unknown whether this model scales as well as transformer architectures.
We present a systematic study of recent proposed efficient attention variants on real-world long-context NLP tasks. In contrast to existing work, we are the first to test these models with a set of unified and large-scale experiments. Our results reveal the gap between a widely used benchmark and practical downstream tasks after conducting largescale pretraining. Among all the studied attention methods, we find that the simplest local attentions outperform other complex attention paradigms on downstream tasks. We also show that existing local-attention models can be further simplified by removing the attention-window overlap, resulting in
<sup>10</sup>Note that while BigBird has a similar overlapping local attention and outperforms Longformer, it uses a larger corpus, more pretraining compute and different finetune pipelines, making a direct comparison difficult.
a faster model that achieves similar or better results. Importantly, our work calls for more careful and practical evaluation protocols while developing long-context NLP models.
Joshua Ainslie, Santiago Ontañón, Chris Alberti, Va-clav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. ETC: encoding long and structured inputs in transformers. In EMNLP (1), pages 268-284. Association for Computational Linguistics.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. abs/2004.05150.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. 2021. Rethinking attention with performers. In *ICLR*. OpenReview.net.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5156-5165. PMLR.
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 284-294, Melbourne, Australia. Association for Computational Linguistics.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In *ICLR*. OpenReview.net.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086-6096, Florence, Italy. Association for Computational Linguistics.
Benjamin Lefaudeau, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, and Susan Zhang. 2021. xformers: A modular and hackable transformer modelling library. https://github.com/facebookresearch/xformers.
Tao Lei. 2021. When attention meets fast recurrence: Training language models with reduced compute. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7633-7648, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Joe O'Connor and Jacob Andreas. 2021. What context features can transformer language models use? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 851-864, Online. Association for Computational Linguistics.
Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick S. H. Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Wen-tau Yih, Sonal Gupta, and Yashar Mehdad. 2021. Domain-matched pre-training tasks for dense retrieval. arXiv, abs/2107.13602.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. 2021. Random feature attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Ofir Press, Noah A. Smith, and Mike Lewis. 2021. Shortformer: Better language modeling using
shorter inputs. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5493-5505, Online. Association for Computational Linguistics.
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53-68.
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. 2021. Linear transformers are secretly fast weight programmers. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9355-9366. PMLR.
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. 2020. Sparse sinkhorn attention. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 9438-9447. PMLR.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021. Long range arena: A benchmark for efficient transformers. In ICLR. OpenReview.net.
Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. abs/1806.02847.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv, abs/2006.04768.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. 2020. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pages 10524-10533. PMLR.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. 2021. Nyströmformer: A nyström-based algorithm for approximating self-attention. In AAAI, pages 14138-14148. AAAI Press.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Transformers for longer sequences. In NeurIPS.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In NeurIPS, pages 9051-9062.
Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. 2021. Long-short transformer: Efficient transformers for language and vision. NeurIPS, abs/2107.02192.
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, pages 19-27. IEEE Computer Society.
However, the tested local attentions typical requires less GPU memory than all the other models.
Downstream Task	Hyperparameter Grid
TriviaQA	learning rate: 1e-5, 3e-5, 5e-6;
warmup ratio: 0%, 10% of total steps;
random seed: 42, 3, 4321;
batch size: 32;
max epochs: 10
NQ Doc Retrieval	learning rate: 1e-5, 5e-6, 3e-5;
random seed: 42, 3;
batch size: 8;
max epochs: 10
Hyperpartisan	learning rate: 1e-5, 3e-5;
random seed: 42, 3, 5, 1992;
batch size: 8;
max epochs: 40
Table 7: Hyperparameters of downstream finetuning.
TriviaQA
Average|P95%	Hyperpartisan
Average|P95%	NQ doc Retrieval
Average|P95%
769.8|2,067.0	3,333.9|11,444.3	6,732.9|17,493.4
Table 8: Document length statistics in the tested downstream datasets.
Downstream Task Details. On TriviaQA, there are usually multiple matched spans in the document, we train the model to maximize the marginalized probability of all matched spans. The prediction head in the classification task is defined as a 2-layer MLP with tanh activations. For the retrieval task, we follow existing passage retrieval methods and use in-batch documents as negative retrieval targets. The loss is simply a cross-entropy loss defined over the scores of all documents in the batch. All the models are finetuned using the Adam optimizer with linear decays. We conduct grid search for all the tested models. The hyperparameters for all the three tasks are shown in Table 7. In Table 8, we show the average and the  $95\%$  percentile of the document lengths in each dataset. As mentioned in the main text, we drop the tokens exceeding 4,096 tokens.
Pretraining Details. Our pretraining pipeline is implemented with fairseq<sup>11</sup>. We control the memory usage of each model by adjusting model-specific hyperparameters. The details in shown in Table 9. Due to different model designs, we are not able to exactly control the memory consumption.
11https://fairseq.readthedocs.io/en/ latest/
Model	Avg Memory Usage (GB)	Architecture Setting
Sinkhorn	14.2	block size: 128
LSH	18.2	num of hash functions: 4; chunk size: 16
Linformer	17.2	compression ratio: 8
Nystrom	16.3	num of landmarks: 256; convolution kernel size: 35;
Performer	14.2	random feature dimension: 256; kernel function: relu
Long-Short	16.3	block size: 128; num of landmarks: 32
Sliding Window	15.3	attention window size: 256
Blockwise LW	15.1	block size: 128; overlap: 64
Blockwise LW w/o global toks	14.7	block size: 128
Blockwise LW w/o overlap	13.4	block size: 256
Blockwise LW w/o overlap & global	13.2	block size: 256
Table 9: Model-specific architecture settings and each model's GPU memory usage when feeding in a single sequence of 4,096 tokens.