SlowGuess/NAACL_2022_OCR huggingface dataset:
For TriviaQA, we compare LongT5 with various top approaches on the leader board: BigBird-ETC (Zaheer et al., 2020a), Fusion-in-Decoder (Izacard and Grave, 2021), and ReadTwice (Zemlyanskiy et al., 2021). As shown in Table 3, TriviaQA inputs are quite long, therefore being able to scale up both in model size and to 16k input length helps LongT5 achieve state-of-the-art.
5.1 Input Length vs Speed
In order to evaluate the training speed and memory consumption of LongT5, compared to T5.1.1, we performed a series of training runs in the NQ data set starting at input length 512, and increasing
Figure 4: Speed versus Performance on NQ (short-answer F1), for T5, LongT5 with Local Attention and LongT5 with TGlobal attention, for different input sequence lengths. Input lengths start at 512, and go as far as possible before running out of memory. Measurements taken with batch size 128, on 4x8 TPUv3 slices.
the input length steadily until models ran out of memory on a 4x8 TPUv3 slice. Results are shown in Figure 3, which compares 6 different model configurations: T5.1.1 base, T5.1.1 large, LongT5 (base Local), LongT5 (large Local), LongT5 (base TGlobal), and LongT5 (large TGlobal). For each model configuration, we show a curve plotting the number of sequences per second processed during training (speed, in the vertical axis) for each input length (horizontal axis). Both axes are shown in logarithmic scale.
We can see that at shorter lengths (512), T5.1.1, LongT5 Local, LongT5 TGlobal have similar speeds, but as we increase the sequence length, LongT5 becomes significantly faster. For example at sequence length 2048, T5.1.1 base can only process 479 sequences per second, while LongT5 (base TGlobal) can process 765 and LongT5 (base Local) can process 860. The differences grow even larger as sequence length increases.
Another important fact that Figure 3 shows is that T5.1.1 models reach their out of memory point much earlier. For example, we could only scale up to 6k tokens for T5.1.1 base. On the other hand, LongT5 (base Local) can go up to 36k tokens in length, and LongT5 (base TGlobal) up to 12k. Large models show a similar picture with T5.1.1 large going only up to 3k, but the LongT5 variants going to 10k (large Local) and 6k (large TGlobal).
5.2 Input Length vs Performance
This section presents a similar analysis, but where we plotted model speed versus performance in NQ
(F1 score). Results are shown in Figure 4 for models with large size. Each point in the curves is annotated with the corresponding sequence length.
As Figure 4 shows, performance increases significantly as input length increases, highlighting the benefits of LongT5. Moreover, input length by itself is not enough to achieve good performance in all datasets, and in particular, in the NQ dataset (used in this figure), using Local Attention significantly hurts performance when compared with TGlobal or with T5.1.1. So, even at very long input lengths, LongT5 with Local Attention just matches T5.1.1 with input length of  $3\mathrm{k}$  in NQ. However, LongT5 with TGlobal attention outperforms T5.1.1. Moreover, note that although the plot shows a few irregularities (such as  $8\mathrm{k}$  length for LongT5 with Local Attention, or  $6\mathrm{k}$  length with TGlobal Attention), that is because the plot shows only the results of a single run, and hence there is some noise. However, trends can clearly be seen.
5.3 Principle Sentences Generation vs. Span Corruption
As mentioned in section 3.2, we use PEGASUS Principle Sentences Generation instead of default Span Corruption used in T5 as our pre-training objective. Table 5 shows our ablation study for fine-tuning on NQ and arXiv from a model pretrained using the default Span Corruption objective, a model pre-trained with Principle Sentences Generation, and a model pre-trained with both objectives. The comparison is done on the dev set of the tasks, and with TGlobal base models. Both pretraining and fine-tuning on the models mentioned above are done with input sequence length 4096. The table shows, even though Principle Sentences Generation was developed by Zhang et al. (2019a) as a pre-training strategy for summarization, it benefits both summarization and QA tasks, but using both objectives together perform worse than just using PSG.
Table 6 shows an additional ablation study with arXiv and PubMed, where we compare using regular T5.1.1 with Span Corruption compared to T5.1.1 pretrained with Principle Sentences Generation while using the same pre-training input sequence length of 512 (as was done in the original T5.1.1 pre-training task). As expected, Principle Sentences Generation helped the model achieve better results compared to Span Corruption when seeing the same amount of pre-training data. We
Objective	NQ	arXiv
EM	F1	R-1	R-2	R-3
PSG	62.21	66.94	44.95	18.74	40.99
SC	58.65	63.05	43.49	18.12	39.71
SC + PSG	59.74	64.54	44.85	18.79	40.90
Table 5: Ablation study on dev set for different pretraining strategies using span corruption (SC) vs. principle sentences generation (PSG) and the effects on NQ and arXiv fine-tuning tasks. The models are TGlobal base, and fine-tuning is done with input sequence length 4096.
arXiv
Objective	R-1	R-2	R-3
SC	44.59	18.34	40.65
PSG	45.78	18.94	41.53
LongT5 (4k)	45.66	19.22	41.49
LongT5 (16k)	48.21	21.7	44.03
PubMed
Objective	R-1	R-2	R-3
SC	47.86	22.14	44.39
PSG	48.74	23.42	45.24
LongT5 (4k)	48.47	23.38	45.01
LongT5 (16k)	50.12	24.78	46.56
Table 6: Ablation study on arXiv and PubMed for different pre-training strategies using span corruption (SC) vs. principle sentences generation (PSG) with T5.1.1 model along with LongT5 with TGlobal attention. Fine-tuning was done on large model size, with input sequence length of 4096 except where otherwise noted.
also compare this with dev scores from LongT5 with TGlobal attention at 4k and 16k input lengths, such that we can see having full attention will allow for better results, but being able to scale to longer input sequence lengths allows LongT5 to achieve its stronger results.
Language model pre-training followed by task specific fine-tuning has proven to be a powerful tool for numerous NLP tasks (Devlin et al., 2019; Liu et al., 2019; Zhang et al., 2019b; Radford et al., 2019; Raffel et al., 2019a; Lewis et al., 2020; Joshi et al., 2020). BERT (Devlin et al., 2019) introduced Mask Language Model (MLM), where a model predicts masked tokens given a sequence of text input. Fine-tuning a pre-trained BERT model has led to improved performance on various NLP tasks. However, MLM predictions are not made auto-regressively, which limits the capability of the
BERT family for generation tasks. Raffel et al. (2019a) introduced the span corruption task in T5 as the pre-training objective, where a model predicts the masked token span using an autoregressive model. It can handle the generation tasks as the pretraining is done in a generative way. BART (Lewis et al., 2020) is similar to T5 but used a slightly different pre-training objective, in which spans are masked from the input but the complete output is predicted. However, none of these works tried to investigate pre-training for very long sequence inputs. They often use a transformer (Vaswani et al., 2017) architecture as backbone, the complexity of which is quadratic to the input length, making them impractical to model very long sequence input.
Long text modeling An extensive amount of work has also been done for modeling long text like documents. The work from Roy et al. (2016); Chen (2017); Wu et al. (2018) obtained document embeddings from word-level embeddings. Another line of research tries to model long documents through hierarchical training. The work from Yang et al. (2016); Miculicich et al. (2018) employed Hierarchical Attention Networks for document classification and neural machine translation, and Guo et al. (2019) proposed using a hierarchy network to build document embeddings on top of sentence embeddings for parallel document mining.
More recent research has been focusing on improving the memory and computation efficiency of transformer models (Tay et al., 2020b, 2021) for handling long input. One type of such approaches is using non-full attention patterns to restrict the attention field range, so that it reduces the attention complexity from  $O(n^{2})$  to  $O(n\log n)$  or  $O(n)$ , including Sinkhorn (Tay et al., 2020a), Longformer (Beltagy et al., 2020), ETC (Ainslie et al., 2020), and BigBird (Zaheer et al., 2020a). Another type of approaches is leveraging the low-rank approximation of the attention matrix, such as Linformer (Wang et al., 2020), Performer (Choromanski et al., 2021), Random Feature Attention (Peng et al., 2021), and LUNA (Ma et al., 2021).
This paper presented a new Transformer-based neural model called LongT5, with which we have explored the effects of scaling both input length and model size at the same time. Specifically, the main differences of LongT5 with respect to T5.1.1 are (1) a new scalable attention mechanism called Tran
sient Global attention, which is a drop-in replacement to the standard T5 attention mechanism, and hence can be used without needing additional side-inputs to the model or modifications to the model inputs; and (2) using a PEGASUS-style Principle Sentences Generation pre-training objective.
Via experimentation in several challenging summarization and question answering datasets, we have explored the performance gains that can be achieved by scaling both input length and model size, resulting in state-of-the-art results on several datasets: arXiv, PubMed, BigPatent, MediaSum, and TriviaQA.
As part of our future work, we would like to pursue several directions such as studying efficient attention mechanisms in the decoder and decoder-to-encoder attention pieces of the model (both Local Attention and TGlobal attention are only applied to the encoder in LongT5 for now). Additionally, we would like to incorporate additional long-input transformer ideas into the LongT5 architecture, that could further improve model efficiency.
Joshua Ainslie, Santiago Otonanon, Chris Alberti, Va-clav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. Etc: Encoding long and structured data in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020).
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150.
Minmin Chen. 2017. Efficient vector representation for documents through corruption. 5th International Conference on Learning Representations.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. 2021. Rethinking attention with performers. In International Conference on Learning Representations.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 615-621, New Orleans, Louisiana. Association for Computational Linguistics.
Peng Cui and Le Hu. 2021. Topic-guided abstractive multi-document summarization.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-News: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074–1084, Florence, Italy. Association for Computational Linguistics.
Alexios Gidiotis and Grigorios Tsoumakas. 2020. A divide-and-conquer approach to the summarization of long documents. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:3029-3040.
Mandy Guo, Yinfei Yang, Keith Stevens, Daniel Cer, Heming Ge, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Hierarchical document encoder for parallel corpus mining. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 64-72, Florence, Italy. Association for Computational Linguistics.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada. Association for Computational Linguistics.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering
research. Transactions of the Association of Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. 2021. Luna: Linear unified nested attention. In Thirty-Fifth Conference on Neural Information Processing Systems.
Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. Document-level neural machine translation with hierarchical attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2947-2954, Brussels, Belgium. Association for Computational Linguistics.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. 2021. Random feature attention. In International Conference on Learning Representations.
Ofir Press, Noah A. Smith, and Mike Lewis. 2021. Train short, test long: Attention with linear biases enables input length extrapolation.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019a. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019b. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683.
Tobias Rohde, Xiaoxia Wu, and Yinhan Liu. 2021. Hierarchical learning for generation with long source sequences.
Dwaipayan Roy, Debasis Ganguly, Mandar Mitra, and Gareth J. F. Jones. 2016. Representing documents and queries as sets of word embedded vectors for information retrieval. CoRR, abs/1606.07869.
Eva Sharma, Chen Li, and Lu Wang. 2019. BIGPATENT: A large-scale dataset for abstractive and coherent summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2204-2213, Florence, Italy. Association for Computational Linguistics.
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. 2020a. Sparse sinkhorn attention. In ICML.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020b. Efficient transformers: A survey. ArXiv, abs/2009.06732.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity.
Lingfei Wu, Ian En-Hsu Yen, Kun Xu, Fangli Xu, Avinash Balakrishnan, Pin-Yu Chen, Pradeep Ravikumar, and Michael J. Witbrock. 2018. Word mover's embedding: From word2vec to document embedding. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4524-4534. Association for Computational Linguistics.
Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2021. PRIMER: Pyramid-based masked sentence pre-training for multi-document summarization.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480-1489, San Diego, California. Association for Computational Linguistics.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020a. Big bird: Transformers for longer sequences. CoRR, abs/2007.14062.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020b. Big Bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems, volume 33, pages 17283-17297. Curran Associates, Inc.
Yury Zemlyanskiy, Joshua Ainslie, Michiel de Jong, Philip Pham, Ilya Eckstein, and Fei Sha. 2021. Readtwice: Reading very large documents with memories.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019a. PEGASUS: pre-training with extracted gap-sentences for abstractive summarization. CoRR, abs/1912.08777.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019b. ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441-1451, Florence, Italy. Association for Computational Linguistics.
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. 2021. MediaSum: A large-scale media interview dataset for dialogue summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5927-5934, Online. Association for Computational Linguistics.
A Summarization Results
Table 8 shows the full set of results on the summarization datasets used in this paper. This includes both standard T5 model (using version T5.1.1), T5 with PEGASUS Principle Sentences Generation pre-training, and LongT5 model.
As can be seen, scaling up the input size for the models helps achieve better performance metrics. T5 models though struggle when scaling up to 4k for input, as the fine-tuning task can take many days even when using a large topology of TPUv3.
When comparing regular T5.1.1 model with a T5.1.1 model using PEGASUS Principle Sentences Generation pre-training, the latter was able to achieve better results, with the results also improving as the input size scaled up. This helps show that both using the latter pre-training objective along with scaling up allows us to get the best results from these models.
LongT5, despite having a reduced attention from using TGlobal attention, is able to get strong performance results due to both scaling up to larger inputs and leveraging the Gap Sentences Generation pre-training strategy.
B QA Results
Table 7 shows the full set of results comparing T5.1.1 and LongT5 models on the QA datasets used in this paper. For both NQ and TriviaQA in this comparison study, we use  $90\%$  of the official training set for training while using  $10\%$  as holdout dev set to fine-tune the hyperparameters and training epoch, and use the official dev set to report the numbers in this table. We run each model to the largest input length allowed before running out of memory on specific hardware configuration - base/large models on 4x8 TPUv3 with no model partitioning, and xl models on 8x16 TPUv3 with 8 partitions.
Approach	NQ	TriviaQA
EM	F1	EM	F1
base:
T5.1.1 (512)	50.93	52.54	48.91	52.89
T5.1.1 (6k)	56.73	56.73	59.09	63.31
large:
T5.1.1 (512)	57.29	60.68	53.26	57.01
T5.1.1 (3k)	60.09	64.17	60.15	64.15
xl:
T5.1.1 (4k)	60.75	64.07	65.33	69.43
base Local:
LongT5 (512)	54.39	58.24	-	-
LongT5 (1k)	54.60	57.88	-	-
LongT5 (2k)	56.48	60.56	-	-
LongT5 (4k)	56.10	60.52	-	-
LongT5 (8k)	55.90	59.98	-	-
LongT5 (16k)	56.41	60.46	-	-
LongT5 (32k)	55.84	59.59	-	-
LongT5 (36k)	55.77	59.66	-	-
base TGlobal:
LongT5 (512)	55.73	59.06	-	-
LongT5 (1k)	57.41	61.25	-	-
LongT5 (2k)	56.96	60.25	-	-
LongT5 (4k)	58.97	63.03	-	-
LongT5 (8k)	58.07	62.67	-	-
LongT5 (12k)	58.12	62.44	63.27	67.42
large Local:
LongT5 (512)	55.19	58.00	-	-
LongT5 (1k)	57.47	60.79	-	-
LongT5 (2k)	58.49	62.12	-	-
LongT5 (4k)	59.44	63.72	-	-
LongT5 (8k)	58.66	62.28	-	-
LongT5 (10k)	60.01	64.40	-	-
large TGlobal:
LongT5 (512)	57.55	61.53	-	-
LongT5 (1k)	59.69	63.91	-	-
LongT5 (4k)	60.77	65.38	-	-
LongT5 (6k)	59.17	63.38	63.76	67.82
xl TGlobal:
LongT5 (4k)	62.38	66.39	-	-
LongT5 (8k)	62.66	66.61	67.89	71.71
Table 7: QA results comparing T5.1.1 and LongT5 at different sequence lengths. Base and large models are trained on 4x8 TPUv3 with no model partitioning, and x1 models are trained on 8x16 TPUv3 with 8 partitions.