Adapting Pretrained Text-to-Text Models for Long Text Sequences
Authors: Wenhan Xiong
Anchit Gupta
Shubham Toshniwal
Yashar Mehdad
Scott Yih
Venue: EMNLP 2023 Findings (2023)
Keywords: long context
summarization

Abstract: We present an empirical study of adapting an existing pretrained text-to-text model for long-sequence inputs. Through a comprehensive study along three axes of the pretraining pipeline -- model architecture, optimization objective, and pretraining corpus, we propose an effective recipe to build long-context models from existing short-context models. Specifically, we replace the full attention in transformers with \textit{pooling-augmented blockwise attention}, and pretrain the model with a masked-span prediction task with spans of varying lengths. In terms of the pretraining corpus, we find that using randomly concatenated short-documents from a large open-domain corpus results in better performance than using existing long document corpora, which are typically limited in their domain coverage. With these findings, we build a long-context model that achieves competitive performance on long-text QA tasks and establishes the new state of the art on \emph{five} long-text summarization datasets, often outperforming previous methods with larger model sizes.

Adapting Pretrained Text-to-Text Models for Long Text Sequences
Wenhan Xiong', Anchit Gupta", Shubham Toshniwal, Yashar Mehdad, Wen-tau Yih
Meta Al
(xwhan, anchit, shtoshni, mehdad, (oottyih)@ff
Abstract
Extending the success of PLMs long texts nontrivial for the following reasons. First, the quadratic complexity of self-attention makes it pro- hibitive to directly apply full-attention to long se- quences. Any long-range architecture needs to be computationally efficient and at the same time cap- ture long-distance dependency. Second, the train- ing objectives used by existing PLMs have largely focused on short text and have not been well- studied for long-context scenarios. For instance, BART (Lewis et al., 2020) pretraining involves re- constructing the whole corrupted input sequence, which impractical for long sequences given the computational overhead of decoder-side attention. Additionally, while abundant short documents can be easily collected from web dumps to pretrain short-context models that work well across differ- domains, long documents much scarcer and are often collected from specific domains as books or movic scripts (Gao et al., 2021). unknown whether the existing corpora are more effective for pretraining a versatile long-context model com- pared to using artificially constructed long texts.
We present an empirical study of adapting an existing pretrained text-to-text model for long-sequence inputs. Through comprehen- sive study along three axes of the pretraining pipeline model architecture, optimization ob- jective. and pretraining corpus, propose an effective recipe to build long-context models from existing short-context models. Specifi- cally, we replace the full attention in transform- ers with pooling-augmenter blockwise atten- tion, and pretrain the model with masked- span prediction task with spans of varying lengths. In terms of the pretraining corpus, we find that using randomly concatenated short- documents from a large open-domain corpus results in better performance than using exist- ing long document corpora, which are typically limited in their domain coverage. With these findings, build long-context model that achieves competitive performance on long-text QA tasks and establishes the new of the art five long-text summarization datasets, often outperforming previous methods with larger model sizes.
In this work, we conduct a thorough experi- mental study to find a recipe for building high- performing long-context models. In contrast to a recent work (Guo et al., 2022) that pretrains a long- context model from scratch, we choose to adapt an existing short-text model for long texts with further pretraining. Our empirical results demonstrate the effectiveness of this strategy by achieving stronger
1 Introduction
NLP applications like summarization and question answering often require processing long text quences. While there have been tremendous em- pirical breakthroughs (Vaswani al., 2017: Devlin al., 2019) from large pretrained language models (PLMs), most of these successes have been con- fined to short-context tasks (Rajpurkar& et al., 2016; Wang et al., 2019). On long-context NL.P bench- marks (Kočiský et al., 2018; Zhong et al., 2021; Pang et al., 2022b), where the input sequences are often longer than 10,000 tokens, there is still sig- nificant gap between human performance and the state-of-the-an models.
performance on various downstream tasks, while saving on the high cost of pretraining from scratch. More specifically, we explore three axes of the pretraining pipeline, namely efficient long-range model architectures, long text corpora creation and
While there exists a long list of elficient attention vari- ants (Tay et al., 2020). their efficacy is only validated in syn- thetic or small-scale experiments and it is unknown whether these variants are scalable and suitable for large-scale pre- training for natural language (Xiong et al., 2022; Tay et al., 2022).
Equal Contribution.
Our code has heen released at https: //g/ thub. com/ f cebook reeeoookeeeeee resea research,.... whhchh//oaa /tbarrrlaa...

---

the choice of pretraining objectives. Our main find- ings are summarized as follows:
Add & Norm
1) Among long-range mechanisms, such as global tokens and sliding-window attention, we find a simple saginn-- blockwise atten- tion to be the most effective choice for various tasks.
Feed Forward
Add & Norm
2) For the pretraining corpus, we surprisingly find that using randomly concatenated documents from a large open-domain corpus (CommonCraww) performs better than using existing Long-document corpora such as book collections.
Multi-Head Attention
Pool Altention
Avg. Pooling
Linear
3) We experiment with various pretraining ob- jectives including standard masked-span predic- tion (Raffel al., 2020), primary sentence predic- tion (Zhang al., 2020), and a novel model- span prediction objective. While we find all of these objectives can bring gains over models that are not pretrained on long texts, we consider the masked-span prediction objective (using both short and long spans) remains as the best choice, thanks to its simplicity and balanced effectiveness on both short- and long-output tasks.
Block Attention
o
Linear
Figure 1: The pooling augmented self-attention layer. The pooling attention parameters marked separately are newly introduced and randomly initialized.
Using these findings, we build a strong long- context text-to-text model that establishes new state-of-the-art on five long-text summarization tasks (with= > 10% relative ROUGE-2 improve- ments on three of the datasets) and achieves com- petitive performance on long-text QA tasks despite its modest size.
tions in the encoder:
Global-token mechanism: Previous work (Guo et al., 2022; Zaheer et al., 2020; Beltagy al., 2020) has proposed augmenting block-sparse atten- tion with a small set of "global tokens" that attend to the entire sequence and hence enable long-range interactions in the encoder. Specifically, we mark the first 64 tokens in each attention block global tokens and share the projection matrices for both the global and regular tokens. This mechanism has proven effective in encoder-only models, espe- cially for question answering tasks as shown by the aforementioned methods.
2 Model and Data
2.1 Efficient Models for Long Sequences
Our model is based on a standard transformer with block-sparse self-attentions (Zaheer et al., 2020) on the encoder side. While various new architec- tures (Wang et al., 2020; Choromanski et al., 2021; Lei, 2021; Gu et al., 2021) have been proposed, we stick the simple architecture for the follow- ing reasons: 1) it makes it easy reuse existing pretraining pipelines, which are often highly op- timized specifically for vanilla transformers, e.g., learning rate schedules, normalization layers, opti- mizers; 2) using local attentions, where each token attends to only tokens in the local context, allows our model to reuse all the model parameters from existing PLMs, while other attention variants different parameterizations that prohibit inheriting the weights of an existing pretrained model.
2) Overlapping (strided) attention windows: Biding-attention with overlap a straightforward way to introduce long-range connections in local attention models. As we stack the layers in the encoder, the receptive field of each token would in- crease exponentially. For example, (Beltagy et al., 2020) use the stride of one token and each token attends
equal number of tokens from both sides. We develop simpler and faster block-wise version which makes the parallelization easier implement; namely, tokens in each block will at- tend to all the tokens inside the block, and half the tokens from its immediate left and right blocks.
In addition to block attention, we investigate three mechanisms that enable long-range connec-
3) Pooling layers: Recent work (Zhang et al.,
2021; Pang et al., 2022a) has explored using pool-

---

ing operations to reduce the number of key and value states in transformers. We implement sim- pler version that only requires standard average pooling operations. All illustration of the pooling- augmented attention layer is shown in Figure 1. Specifically, in the top layers of the transformer encoder, we add second attention module which takes input the hidden states output by the ith block self-attention layer X; RRLx,, where L is the sequence length and is the size of the hidden states. As in the vanilla attention layers, X; is first projected to create the key, query, value matrices Qº K2, V E ?xx We first average pool the K and V sequences, with a fixed kernel/stride size, into smaller lengths V K. < Lx where L < L. We then apply standard attention using Qº K. and V resulting in O(L x Σ) complexity. The output of the pooling layers is added with X; to form residual connection.
2.3 Prefraining Objectives
A variety of self-supervised pre-training objectives have been proposed for sequence-to-sequee mod- els (Lewis et al., 2020; Raffel et al., 2020; Guo et al., 2022). In the long document setting, we ideally seek an objective that promotes long-range reasoning ability in the model. We investigate the following different pretraining objectives and the effect of input length during pretraining,
1) T5 Span Denoising: Applying BART's de- noising objective to long sequences is computa- tionally expensive as it requires reconstructing the entire input and incurs significant computa- tion overhead on the decoder-side attention. More- over, reconstructing the entire input would be at odds with most downstream tasks such as question- answering and summarization, which require gener- ating shorter text. Thus, we adopt T5-style denois- ing for pretraining our model, i.e., we randomly pick set of spans in the input sequence as the de- coding target and mark them with special sentinel tokens. The model is then trained to generate the uncorrupted spans. This objective readily appli- cable to long documents as we can control both the length and the number of spans. We experiment with both fixed span lengths as in (Raffel et al.. 2020), and also mixed span lengths with both short and long spans, with which we hope the model is able perform well range of tasks requiring differing output lengths.
We compare these variants via the performance on downstream long-sequence tasks in Sec 3.2.
2.2 Pretraining Corpus
The choice of the corpus has a significant impact on the downstream results. We consider long documents from formal text domains, including Books3 (Gao et al., 2021), STORIES (Trinh and Le, 2018), RealNews (Zellers al., 2019); and long dialogues including MediaSum (Zhu et al., 2021) and OpenSubtitles (Tiedemann, 2016). While col- lecting a long-documes corpus seems to be a nat- ural choice for long-xequence downstream tasks, as they are more likely to include long-range de- pendencies than common short texts on the inter- net, pretraining only on these datasets also brings the risk of overfitting specific domains, instead of achieving consistent gains on range of tasks. Thus, we also consider general-domain corpus C4 as used by T5 (Raffel et al., 2020). Addi- tionally, instead of using randomly concatenated sequences, we also tried to concatenate semanti- cally similar C4 documents (using similarity metric learned by dense retrieval models) with the hope that the model can learn to capture more long-range dependencies across relevant documents. We dis- cuss the effects of these corpus variants in Sec 3.3,
2) Pegasus - Primary Sentence Prediction: Originally proposed for summarization pretrain- ing in (Zhang et al., 2020) and recently used for long documents by Guo al. (2022), this objective identifies and masks out a set of principle sentences, i.e., sentences with high ROUGE score with the rest of the document. The model is then trained to generate these principle sentences. The output length can be controlled by choosing the number of principle sentences to mask.
3) Model-based Denoising: Apart from ran- domly selecting the decoding targets, we also ex- plore a novel model-based objective. Here we use separate encoder-oniy model (with local atten- tion) to select decoding targets for the sequence- to-sequence model. This approach is inspired by ELECTRA (Clark et al., 2020) and we hope the prediction loss of the encoder-only model can be a good proxy to select spans that require long- range dependencies to predict. Specifically, we first mask a larger number of tokens (5, 120 tokens
"The projection layers to create these matrices are not used in existing pretrained models and will be randomly initialized hefore further pretraining

---

instead of 1,024) in the input sequence. We then apply an encoder-only masked language model to recover the masked spans. Based on the losses of the masked language model, we only keep the top 20% hard spans to train the text-to-text model. The encoder-only model can either be frozen or jointly trained with the sequence-to-sequene model.
ized. Table 1 shows the results on both QA and summarization lasks. For the global-token mecha- nism, we mark the first 64 tokens of each block as global tokens. We see that pooling layers produce the most consistent improvements even for Gov- Report, where the baseline already achieves strong numbers. Consistent with prior study on encoder- only models (Xiong et al., 2022), attention win- dow overlaps fail to produce further improvements over the disjoint block-attentie global tokens consistently helps on QA tasks but not on summarization tasks. We hypothesize that in pecoder-decoder models, the cross- cttos-nttention can offset the effect of global tokens, as each decoding position has access to all input tokens' representa- tions. When finetuning our final pretrained model, also try to combine global tokens with pooling layers for QA tasks, but did not observe further improvements.
3 Experiments
layers. Adding
3.1 Downstream Tasks& Finetuning Setup
We evaluate the models on six summarization datasets and four QA datasets. The summariza- tion datasets are from formal domains, includ- ing GovReport (Huang et al., 2021), ArXiv & PubMed (Cohan et al., 2018) and BookSum Chap- ters (Kryšciński et al., 2021) 1); or informal conver- sational domains, such as TVMeaSaite & Forever- Dreaming (Chen et al., 2022). For QA, we consider Qasper (Dasigi et al., 2021), which contains ques- tions over NLP papers; QMSum² (Zhong et al., 2021), longform QA over meeting scripts, and two QA datasets on books: QuALITY (Pang et al., 2022b) and NarrativeQA (Kočiský al., 2018). We finetune the model with a maximum of 16,384 tokens. For long-sequence QA tasks, we adopt the input format as used by the state-of-the- art open-domain QA system (Izacard and Grave, 2021). Specifically,
3.3 Effect of Pretraining Corpus
With the assumption that models should be exposed to as many long dependencies as possible at pre- training time, we initially tried to only pretrain the model with natural long documents that are collected from sources like books, news, and TV dialogues. However, we did not achieve consis- tent improvements with this corpus alone. Instead, we found it is important to include sufficient doc- uments from diverse domains, even if those docu- ments are mostly short sequences. We present our ablation analysis in Table 2. We reported results on small summarization datasets where the gaps are more visible. Note that the sizes of documeten corpora are usually smaller than open-domain cor- pus. To remove the size factor that affects model performance, we limit the pretraining steps such that the model does not see repeated examples from each corpus. The length statistics of document sources can be found in the Appendix.
repeat the question/query at the start of each attention block. We also utilize the robust finetuning technique proposed by Agha- janyan et al. (2021). We conduct a grid search over finetuning hyperparameters, such as learning rate and dropout rate, the details of which are presented in Table 8 in Appendix A. We report ROUGE* scores for summarization datasets. For QA, we report Exact Match (EM) scores for datasets with short answers and F1 scores for datasets with long answers.
3.2 Effect of Architectures
To study the effectiveness of different model choices with modest computation cost, we initialize a base-size block-attention model using BART's weights. We augment the model with three ad- ditional long-range mechanisms, as described in Sec 2. Note that only the pooling layers introduce additional parameters that will be randomly initial-
We see that pretraining on corpora that only have long documents, which are often from specific do- mains, hurts the downstream performance for most of the datasets, except for NarrativeQA, which is from very close domain. On the other hand, pre- training randomly concatenated C4 documents brings visible gains for most of the tasks. In addi- tion to directly using concatenations of random C4 documents, we tried assemble long sequences using semantically similar C4 documents, with the hope of creating more long-range connections in
*QMSum is proposed as a "query-based summarization" dataset. We consider as special case of QA as our model uses the same input format QMSum and other QA datasets, "https: : / / thub. com/ / trdy /f 1lleee2rra rouge

---

GovReport
ArXiv
QMSum
Qasper Ans
QuALITY
Models
R-1
R-L
R-1
R-L
R-1
R-L
Ans
block-atta baseline.
60.5
57.5
49.0
44.2
35.2
30.4
28.0
31.6
+ attn window overlaps
60.6
57.6
49.0
44.3
34.8
30.2
28.0
31.6
+ global tokens
60.3
57.3
49.1
44.3
35,4
30.7
29.8
32.5
+ pooling layers
61.0
58.1
49.1
44.3
35,9
31.2
30.6
32.9
Table I: Ablation of different long-range mechanisms using base-size models.
QMSum R-1
Qasper Ans F1
QuALITY Ans EM
NarrativeQA Ans F1
Models
non-pretrain
35.9
30.6
32.9
20.4
Long corpus
34.7
29.9
31.3
21.2
C4 C4-linked
36.3
32.8
32.8
21.6
35.7
32.1
32.8
21.3
Table Effects of pretraining corpus. Base size models pretrained for 20k steps to avoid repetitions. Long corpus: Books3 + RealNews STORIES + MediaSum + OpenSubiitees; C4: randomly concatenated documents to form long sequences.; C4-linked: concatenate related short documents using a retriever model.
QMSum R-1
Qasper Ans Fl
for the 8k length pretraining such that the input to- kens in each batch stays the same. We also increase the masking ratio for the 8k model to 1/8 that the decoding sequence length remains 1,024. Note that under this setting, pretraining with 8k-length batches is a bit slower compared the 16k batches due to the decoder-side eeff-attention
Models
no no-pretraining
35.9
30.6
TS avg span_len - 8k
36.7
32,9
+ T5 avg span len - 16k
37.0
34.6
+ T5 mixed span_len
37.0
35,4
pegasus
37.4
34.4
model-hased
37.0
32.5
Pretraining with longer sequences is useful. While prior work (Guo et al., 2022) pretrains their model with sequences shorter than downstream tasks, we find it is generally better to directly pre- train with longer sequences. In terms of conver- gence rate, we find pretraining with 8k and 16k sequences are similar (the loss curves can be found in Appendix A). For downstream results, we find that training with longer sequences lengths in- deed helpful for low-resource datasets - QMSum and Qasper are both small with few thousand ex- amples (T5 avg span_len 5 8k T5 avg span_ks span_len 5 16k). We find using range of short spans (mixed span_len) tends to give more gains on QA tasks.
Table 3: Ablation of different pre-training objectives
C4 corpus
the pretraining sequences. For each document, we use dense retrieval model (Izacard al., 2021) to find similar documents and concatenate them as long pretraining sequences. We denote this corpus as "C4-linked". However, this new corpus is either similar or worse compared to directly using C4. We conjecture that it is because the retrieved doc- uments may contain redundant information, mak- ing some of the masked spans trivial predict the training perplexity after 100k updates on "C4- linked" significantly lower than that on the origi- nal C4 corpus (10.5 vs 12.2).
3.4 Effect of Pretraining Objectives
A Alternative objectives works similar as random masking. While the Pegasus objective is effec- tive for summarization, we do not find it to be consistently better than T5 denoising. It also in- more data processing costs compared to T5's random masking. We also find that model-based denoising fails yield better performance than ran- dom denoising, even though it introduces a harder pretraining task, i.c., larger training losses. We conjecture that, while this objective might provide
We compare the effects of different pretraining ob- jectives in Table 3. The generation targets are usu- ally peragraph-length for QMSum, while Qasper expects the model to predict spans single sen- tences most of the time. All the models pre- trained for 100k updates on the C4 corpus. To in- vestigate the effect of pretraining sequence length, we compare the 16k model with a model pretrained with 8/ sequence length. We double the batch size

---

GovReport R-2
BookSum R-2
ArXiv
PubMed
Model
# Param
R-1
R-L
R-1
R-L
R-1
R-2
R-L
R-1
R-2
R-L
BigBird
S80M
31.8
6,5
14.2
46.6
19.0
41.8
46.3
20.7
42.3
LED
460M
59.4
26.5
56.6
32.8
7,5
14.6
46.6
19.6
41.8
47.0
20.2
42.9
PageSum
440M
59.9
27.2
57.1
49.7
21.1
44.7
48.2
21.1
44.3
BART-Hepps
440M
56.9
22.6
53.8
48.2
20.3
41.8
48.1
21.1
42.7
DYDE
525M
61.0
28.8
57.8
46.4
18.0
41.5
LonggTSlaagg'
750M
48.3
21.6
44.1
50.0
24.7
46.5
T5-xl
3B
48.4
21.9
44.3
50.2
24.8
46.7
Top-down (AvgP)
460M
37,9
9.1
18.01
48.7
20.7
43.9
48.3
21.4
44,2
Top-down (AdaP)
660M
38.3
9.2
18.1
51.0
21.9
45.6
51.1
23.3
46.5
BART-LS
440M
62.0
30.9
59.2
38.5
10.3
36.4
50.2
22.1
45.4
50.3
24.3
46.3
Table 4: Results on long-documen summarization. Pipelined approaches are highlighted in gray. LED's results on GovReport are from PageSum (Liu et al., 2022). ": The AdaPool version the Top-Down model requires an additional encoder model to predict the weights in its pooling layers. t; The baseline R-L scores on BookSum are taken from Pang et al. (2022a) and may not be rigorously comparable due to the unknown ROUGE script version used their paper.
more training signals that are related to long-range dependencies, it can also introduce noisy supervi- sion, which is harmful for the model to learn wide range of language understanding skills.
also consistently outperforms PageSum and DYLE which are specifically designed for summarization tasks. Note that PageSum proposes the idea of us- ing weighted combination of multiple decoder predictions (corresponding to taking the encodings of different parts of the input sequences as inputs). which could be orthogonal to our method.
3.5
Main Results
Best Model Configuration. Following the anal- ysis of base-size models, we pretrain large-size model with the best configuration, which consists of (a) block attention and pooling layer augmen- tations applied to the vanilla Transformer archi- tecture, (b) long-sequence training data batches formed by randomly concatenating the documents from C4 corpus, and (c) T5 denoising loss with mix of short and long spans the training loss. We pretrain the model for 100K steps. Our model is denoted as "BART-LS" in the following sections.
Compared to LongT5 (large and xl), our model achieves stronger performance ArXiv and is on-par on PubMed, even with much fewer param- eters. The recently proposed Top-Down Trans- former (Pang et al., 2022a) applies similar pool- ing operation at the finetuning stage. Our model architecture is similar to their "Average Pooling" variant but conceptually simpler. With the pro- posed pretraining method, our model outperforms "Top-down (AvgP)" on all tasks. Besides down (AvgP)", the authors also proposes advanced pooling layer that the token impor- tance predicted by another encoder-only model aggregate the hidden states for each pooling opera- tion, i.c., "Top-down (AdaP)". While this method should be orthogonal to our model during finetun- ing, we find the model- bbssdd adaptive pooling hard to replicate. Our model matches the performance of "Top-down (AdaP)" performance ArXiv and PubMed in terms of R-2/R-L,, and surpass their results on BookSum.
3.5.1 Summarization
Table shows results of formal long- dong-document summarization. We first compare our model with models that directly reuse existing models' weights without further pretraining and newly introduce parameters. Apart from BigBird and LED that simply use encoder-side local attention allow existing PLM take longer context, we also consider more recent baselines including Page- Sum (Liu et al., 2022) which investigates the local- ity bias on both encoder and decoder side; BART- Hepos (Huang et al., 2021) which applies head- wise cross- cross-attentions; DYLE (Mao et al., 2022) which combines context extractor with gen- erator that only takes short text as input, and uses a complex training pipeline provide supervision for the extractor. Our model outperforms BigBird, LED, and BARTHepos by a large margin. With simple sequence-to--squeeee finetuning, our model
In contrast to formal documents, dialogue texts, especially multi-person conversations, can be nois- ier, more unstructured, and cover more diverse top- ics within each document. We test our model on two summarization datasets collected from popu- lar TV series (Chen et al., 2022). As shown in Table 5, our model achieves even stronger relative gains compared to gains on formal-domail datasets.

---

TVMegaSite
FrraninDreaming
Model
R-1
R-2
R-L
R-1
R-2
R-L
BART-large
43,5
10.3
41.4
33.8
7.5
29.1
DialogLM
45.6
10.8
43.3
35.8
8.3
30,8
Top-down (AvgPool)
49.3
14.4
47.5
35.8
8,9
31.1
Top-down (AdaPool)
51.0
14.7
49.0
36.8
9.2
31.1
BART-LS w/o pretrain
50.9
14.5
48.9
37.1
9.6
32.5
BART-LS
51.8
17.2
50.0
39.1
10.7
33.5
Table 5: Results of on long dialogue (scripts from TV series) and narrative summarization.
QMSum R-2
ous methods that do not apply data augmentation. SecEnc (Vig al., 2022) is also block-attention version of BART distribute overlapped texts (in- stead of disjoint text blocks) into each self-attention window and reuses the position embeddings of the first 1,024 tokens. On long-document QA datasets (as shown in Table 7), our best model consis- tently better than our block-attention baseline and is aligned with LongT5 in terms of scaling effect our model's size is between the base and large versions of LongT5.
Model
R-1
R-I.
BaRT-large
32.2
8.0
27.7
DialogLM
34.0
9.2
30.0
DYLE
34.4
9.7
30.1
LED
34.2
10.3
30.0
SecEnne
37.1
13.0
32.6
SecEnc-W
37.8
13.4
33.4
Block-BART (ours)
36.6
12.1
32.4
BART-LS
12.1
33.1
Table 6: Results on query-based meeting summarization (QMSum). The highlighted row indicates additional data has been used for training.
3.6 Performance Analysis on Input Lengths
To further investigate the performance gains of our proposed model, we compare the performance of the proposed model against the base model as a function of source document length for two summarization datasets, namely SummScreen and TVMegaSite. To conduct our analysis, we divide the validation split of both the datasets into short and long documents. The cutoff length to separate the two groups chosen such that approximately 75% of the documents are classified as short doc- uments. Figure 2 presents the results this com- parison. For both the datasets: (a) there's ''s a perfor- mance drop for both the best and the base model for longer documents, and (b) the best model is better than the base model on all data splits. For Summ- Screen the performance gap between the best and the base model is bigger for long documents than for short documents relative ROUGE-L increase of 0.80% and 3.96% for short and long documents respectively. This suggests that the performance gains for the best model can be attributed to better long- context modeling. For TVMegaSite this trend of increasing performance gap between the best and the base model with increase in document length still holds true, though the increase in perfor- mance gap modest in comparison the increase observed for SummScreen relative ROUGE-I. increase of 2.43% and 2.75% for short and long documents respectively.
Qasper Fl
NarrativeQAA EM
QuALITY EM-T/H
Model
LongT5-hase
46.6
23.0
37.9/36.6
LongT5 larag
53.3
27.2
40.6/38.6
Long
53.1
29.3
46.0/42.1
Block-BART (dev)
38.1
24.1
35.7
BART-LS (dev)
40.6
25.4
37.6
BART-LS
48.7
26.2
37.8/34.0
Table 7: Test results on QA tasks. LongT5's num- bers are taken from the Scrolls benchmark (Shaham et al., 2022). We also compare our model with block- attention baseline that reuses BART's weights on the dev set, shown gray rows. Note that our model's size is in between Long base and LongT55 large.
Note that DialogLM (Zhong et al., 2022) is specif- ically designed for the dialog domain and further pretrained PLM checkpoint on dialog corpus. The large improvements their results again suggest the importance of open-domain pretraning corpus.
3.5.2 QA and Query-Base -Based Summarization As mentioned in Sec 3.1, we use the same input format for finetuning QA tasks and query- based summarization. As there are existing baselines of long models that reuse the weights of short- sequence models, we also report the performance our implementation of Block-attention BART. As shown Table 6, our model outperforms all previ-

---

BART-LS w/o pretrain
BART-LS
BART-LS pretrain
BART-LS
20
ROUGE-L
ROUGE-L
13
188
<. 9400
D 2 9400
1. <5
2 7500
Document Length
Document Longth
(a) SummSereen
(b) TVMegaSite
Figure 2: ROUGE-L scores as a function source document length for the base model and the best model for two dialogue summarization datasets.
4 Related Work
be inserted into a pretrained transformer at finetun- ing time and bring additional performance gains on summarization. Instead of relying on sin- gle model that directly processes the whole input, Mao al. (2022) proposes a two-stage extract-and- generate approach, where the extractor age the supervision signal learned by the generator. However, despite the complicated training recipe, it does not bring consistent gains and underperforms our non-pretrain baselines. The most relevant work to ours is LongT5 (Guo et al., 2022), which adopts both global tokens as well as local attention, and pretrains the model with 4k text sequences from C4. Compared to LongT5, augment local attentions with pooling layers and present a more compre- hensive study on pretraining strategies. Without pretraining from scratch, we achieve stronger sum- marization performance. Concurrent to our work, Phang et al. (2022) also present an empirical study on adapting short-text models for long document summarization. While their study mostly focuses architectures, we present additional analysis the choices of pretraining corpus and learning ob- jectives.
4.1 Efficient Transformer Architectures
A long list of works has been proposed to reduce the complexity of the attention layers of transform- The simplest paradigm is restrict each to- ken's attending context subset of the whole sequences, e.g., Reformer (Kitaev et al., 2020) and the Routing transformer (Roy et al., 2021) proposes hashing or clustering based attention, where each token only attends to tokens of a sin- gle bucket/cluster. Our model architecture is influ- enced by previous work like Longformer (Beltagy al., 2020), BigBird (Zaheer al., 2020) and ETC (Ainslie et al., 2020) that demonstrate strong downstream performance. These models assume strong locality bias in language data and restrict each token's attending context to nearby tokens. In contrast, we augment the block attention with pooling layers and study the effect of additional pretraining long sequences. Other approaches tackling the efficiency bottleneck includes kernel- based (Choromanski et al., 2021; Peng al., 2021) and low-rank approximation (Wang al., 2020) of the N attention matrix. However, in contrast to local attention transformers, the effectiveness of these approximation approaches is yet to be vali- dated in downstream tasks.
lever-
Conclusion
Through a comprehensive study on the effects of model architectures, training losses and pretraining dataset, we present an effective recipe to adapt existing pretrained text-to-text models for long- sequence NLP tasks. The resulting model sets new state-of oft-off-hhhhee f-tthee-t on five long sequenc summariza- tion tasks and achieves consistent gains QA over docal-attentio models that simply reuse BART's parameters. Apart from presenting stronger checkpoint for finetuning on downstream tasks, we hope our findings in the study can provide insights for future works that aim to develop stronger long- sequence models for downstream tasks.
4.2 Generation from Long Text Inputs
To apply pretrained models to long- onng-eeuennce tasks, early studies (Zaheer et al., 2020; Beltagy et al., 2020) reuse parameters from models pretrained on short sequences and replaces the encoder full atten- tion with sparse local attentions. While the mod- els are not exposed long sequences at pretrain- ing time, they demonstrates consistent improve- ments over previous models that can only take trun- cated inputs. Complementary to local attentions, Zhang et al. (2021) show that pooling layers can

---

6 Limitations
60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Lonng Papers), pages 8602-8615, Dublin, Ireland. Association for Compu- tational Linguistics.
Pretraining language models is costly endeavor, and even more so in the case of long-context PLMs. Because of computational budget constraints, only explored a limited space of the hyperparame- ter search space.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song. Andreea Gane, Tamás Sar- 1ós, Peter Hawkins, Jared Quincy Davis, Afroz Mo- hiuddin, Lukasz Kaiser, David Benjamin Belanger. Lucy J. Colwell, and Adrian Weller. 2021. Rethink- ing attention with performers. In ICLR. OpenRe- view.net.
We experiment with training on either just long document corpora or a pseudo long doc- ument corpora formed by concatenating short documents. Future work can investigate using combination of the two.
Kevin Clark, Minh- Than-Thang Luong, Quoe V. Lc, and Christopher D. Manning. 2020. ELECTRA: pre- training text encoders as discriminators rather than generators. In ICLR. OpenReview.net.
We have a surprising empirical finding that pretraining on pseudo long documents formed by concatenating random documents of a short-document corpora (C4) outperforms both: (a) pretraining on actual long documents from a long-document corpora, and (b) pre- training on pseudo long documents formed by concatenating related documents from the same short-document corpora. Future work can investigate more detail the reasons for these empirical gains, and also test these mod- els on their discourse understanding.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui. Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-awe aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Short Papers), pages 615-621 615-621, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-secte seeking questions and answers anchored in research papers. In NAACI.
Due to the human evaluation cost for long- context summarization tasks, we rely on au- tomatic metrics which can be unreliable as suggested by prior work (Kryscinski et al., 2019; Fabbri et al., 2021).
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
References
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. 2021. Better fine-tuning by reducing representational collapse. In 9th International Conference on Learn- ing Representations, ICIR 2021, Virtual Event, Aus- tria, May 3-7, 2021. OveeRevi...nee.
Alexander Fabbri, Wojeiech Kryšciński, Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev, 2021. SummEval: Re-evaluating summariza- tion evaluation. Transactions of the Association for Computational Linguistics, 9:391-409,
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Horace Hc, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The pile: An 800gb dataset of diverse text for language modeling. arXiv, abs/2101.00027.
Joshua Ainslie, Santiago Ontañión, Chris Alberti, Va- clav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Yang. 2020. ETC: encoding long and structured inputs in transformers. In EMNLP (I), pages 268-284, Asso- ciation for Computational Linguisties.
Albert Gu, Karan Goel, and Christopher Ré. 2021. Efi- ciently modeling long sequences with structured state spaces.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. abs/2004.051.0..
Mandy Guo, Joshua Ainslie, David Uthus, Santiago On- tanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022. LongT5: Efficient text-to-text transformer for long sequences. In Findings of the Association for
Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022. SummScreen: dataset for abstrac- tive screenplay summarization. In Proceedings of the

---

Computational Linguisties: NAACL 2022, pages 724- 736, Seattle, United States. Association for Compu- tational Linguistics.
pages 7871-7880, Online, Association for Computa- tional Linguistics.
Yixin Liu, Ansong Ni. Linyong Nan, Budhaditya Deb, Chenguang Zhu, Ahmed H Awadallah, and Dragomir Radev. 2022. Leveraging locality in abstractive text summarization. arXiv preprint arXiv:2205. 01122205,.22700 12476.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419-1436, Online.
Ziming Mao, Chen Henry Wu, Ansong Ni, Yusen Zhang, Rui Zhang, Tao Yu, Budhaditya Deb, Chenguang Zhu, Ahmed Awadallah, and Dragomir Radev. 2022. DYLE: Dynamic latent extraction for abstractive long inp-inupt summarization. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 1687-1698, Dublin, Ireland. Association for Compu- tational Linguistics.
Association for Computational Linguistics.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Towards unsupervised dense information retrieval with contrastive learning. arXiv, abs/21 I 12.091 18.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 874-880, Online. Association for Computa- tional Linguisties.
Bo Pang, Erik Nijkamp, Wojeiech Kryšcińskii Sil- vio Savarese, Yingbo Zhou, and Caiming Xiong- 2022a. Long document summarization with top- down and bottom-up inference. arXiv preprint urXiv:2203.07586.
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma. Jana Thompson, He He, and Samuel Bowman. 2022b. QUALITY: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5336-5358, Seattle, United States. Association for Computational Linguistics.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In ICLR. OpenReview.net.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Ed- ward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Trunsactions of the Asso- ciation for Computational Linguistics, 1:377-328.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. 2021. Random feature attention. In ICLR. OpenReview.net.
Wojciech Kryscinski, Nitish Shirish Keskar, Bryan Mc- Cann, Caiming Xiong. and Richard Socher. 2019. Neural text summarization: critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (FMNIPP-ICCNIP)) P, pages 540-551, Hong Kong, China. Association for Computational Linguis- tics.
Jason Phang, Yao Zhao, and Peter J Liu. 2022. Investigating efficiently extending transformers for long input summarization. arXiv preprint 22Xi:22008.03447
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020, Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 11:1:140:400::
Wojciech Kryšciñski, Nazneen Rajani, Divyansh Agar- wal, Caiming Xiong, and Dragomir Radev. 2021. Booksum: A collection of datasets for long-form narrative summarization.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Tao Lei. 2021. When attention meets fast recurrence: Training language models with reduced compute. In Proceedings of the 2021 Conference on Empir- ical Methods in Natural Language Processing, pages 7633-7648, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021. Efficient content-based sparse attention with routing transformers. Transactions of the A Association for Computational Linguistics, 9:53- 68.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequuee
pre-training for natural language generation, translation, and com- prehension. Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics,
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong. Mor Geva, Jonathan Berant, and Omer Levy. 2022.

---

Scrolls: Standardized comparison over long language sequences.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In NeurIPS. pages 9051-9062.
Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q. Tran, Dani Yogatama, and Donald Met- zler. 2022. Scaling laws vs model architectures: How does inductive bias influence scaling? CoRR, abs/2207. 10555
Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2021. Poolingformer: Long document modeling with pool- ing attention. ICML, volume 139 of Proceedings of Machine Learning Research, pages 12437-12446. PMLR.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Met- zler. 2020. Efficient transformers: A survey, arXiv, abs/2009.067322.
Jingging Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In Pro- ceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org.
Jörg Tiedemann. 2016. Finding alternative tramslations large corpus of movie subtitle. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC' 16), pages 3518- 3522, Portorož, Slovenia. European Language Re- sources Association (ELRA).
Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022. Dialoglm: Pre-trained model for long dialogue understanding and summa- rization. AAAI.
Trieu H. Trinh and Quoe Le. 2018. A simple method for commonsense reasoning. arXiv, abs//806.00847
Ming Zhong. Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev, 2021. QMSum: A new benchmark for query- based multi-domale meeting summarization. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905-5921, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Jesse Vig, Alexander Fabbri, Wojeiech Kryscinski, Chien- Sheng Wu, and Wenhao Liu. 2022. Exploring neural models for mucry-focused summarization. In Findings of the Association for Computational Lin- guisties: NAACL 2022, pages 1455-1468, Seattle, United States. Association for Computational Lin- guistics.
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. 2021. MediaSum: A large-scale media interview dataset for dialogue summarization. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5927-5934, Online. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman, 2019. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In ICLR (Poster). OpenReview.net.
A Additional experiment/data info
Build the linked C4 corpus We attempt to use text retrieval techniques to assemble long text se- quences with the hope that the model can learn more long-range dependencies from linked rele- vant documents. We first encode all the documents into dense vectors with the Contriver (Izacard al., 2021) encoder. For documents that have more than 512 tokens, we use primary sentences (Zhang et al., 2020) as the input to the encoder. Directly retriev- ing documents from the whole index (340M vec- tors) is prohibitive in terms of computation cost. We follow the idea inverted indices, we first k- means get 256 clusters of documents and then assemble long sequences within each cluster. Start- ing from each documents, we concatenate it with its top-k nearest neighbors until the length exceeds certain threshold. To avoid repeated documents,
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv, fbs/2006.04768.
Wenhan Xiong, Barlas Oguz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Scott Yih, and Yashar Mehdad. 2022. Simple local attentions remain com- petitive for long-context tasks. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, pages 1975-1986, Seattle, United States. Association for Computational Linguisties.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang. and Amr Ahmed. 2020. Big bird: Trans- formers for longer sequences. In NewrlPS.

---

Document length distribution of each corpus
17500
(Tokenized) document lengon
12500
10000
7500
2500
OpenSub
Stories
MediaSum
RealNows
Figure 3: Document length distribution of each source corpus. The sizes of each corpus (file sizes of tokenized texts) also shown in the x-axis. The median and mean lengths are denoted via the while line and the triangle. We did not show the statistics of the Books3 corpus (60G) here it has much longer documents with mean/medium over 100k tokens.
we enforce that each documents can appear in at most 2 sequences.
AEQUIRCC engo
30-
23-
13
2556
5005
10000 Trasninng updatte stops
15:0
325ce
isdee 37550
Figure 4: Training curves with 8k/16k sequence lengths. Pretraining with different sequence lengths shows simi- lar level of data efficiency.
Hyperparameters We use a fixed of hyperpa- rameters for pretraining: we set the learning rate to be 1e 4, the weight decay coefficient to be 0.01 and applies polynomial decay with 500 warm up steps; we use a batch size of 256 (16,384 tokens per sample) and fix the random seed to 12. The hyperparameter grids for the downstream tasks shown in Table 8.

---

Downstream Task
learning rate
batch size
epoch
dropout
warmup steps (polynomial decay)
arXiv
1e-4, 3e-4, 4e-4
128
8
0. 0.1
200
GovReport
5e-5, 3e-4, 4o-4
128
70
0, 0.1
200
PubMed, BookSum
3e-4, 4e-4
64
60
0, 0.1
SummScreen
5e-5, 3e-5, le-4
64
130
0.1
200, 500, 1000
Qasper, QMSum, Quality
le-4, Se-5, 3e-5
32, 64
150
0, 0.1
100, 200
NarrativeQA
5c-5, 3c-5
64
8
0, 0.1
200
Table 8: Hyperparamter grid for downstream task finetuning. We use Adam optimizer (3 = (0.9, 0.999), lc-6) for all tasks.
Downstream Task
generation parameters
arXiv
beam: 4, max max_Ien: 300, min Jen: 50, length_penalty: 5.0, no. nepectt nngrmt: 3
GovReport
beam: 4, max_len: 740, min_len: 50, length_penalty: 4.0,
PuhMed
beam: 4, max_lon: 400, min_
length length_penalty: 4.0, wwt.rreeengaaaaee
BookSum
beam: 4, max_ 550, min_len: 20, length_ 4.0, no no_repeat_ngram: 3
SummSSrcen-FFD FD
beam: 4, max max_len: 300, min_len: 50, length Longth_penalty:: 4.0, no wwereee__ggaaaane 3
SummScreen TVM
beam: 4, max_len: 640, min_len: 50, length_ 5.0, no_repeat_ngram:
Qasper
beam: 4, max_len: 80, Length_penalty: 1.0, no_repeat_ngram:
NarrativeQA
beam: 4, max_len: 20, length length_penalty: 3.0. mo_repeat_ngram:: 3
QMSum
beam: 4, max_len: 256, min_len: 40, Length_penalty 4.0, no_repeat_ngram:: 3
QUALITY
beam: 4, max 50, length__peaalye penalty: 3.0. no no_reppatt_ngram:
Table 9: Generation parameters for each task.


Reviews:
[{"review_id": "CvGMerslbG", "reviewer": "Reviewer_Z5kL", "summary": "", "questions": "Please see the reasons to reject section.\nA: Have you done any qualitative/human analysis of the generations from the models for the different pretraining corpora?\nB. Is it possible to apply any of these techniques to decoder only models for long context summarization/QA? Especially in the distractor setting in multi-hop QA where we have 10 context passages that the model has to parse through, would any of these methods help decoder only models where context window is a bottleneck?\nC: How do these methods compare to techniques such as RoPE and Alibi?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Addresses a very relevant topic to the NLP community and presents a comprehensive evaluation of various strategies that can be applied to many different models.\n* Clearly written and easy to follow.\n* Strong empirical performance improvements over considered baselines.\n", "reasons_to_reject": "* The result that randomly concatenating passages from an open-domain corpus gives better performance than natural long form text and semantically linked passages is counter-intuitive. I would like to understand if this conclusion is an artifact of the datasets for long form training or the tasks considered. I would like to see some qualitative analysis/human evaluations done on a small subset of the generations between the different models to make sure that the conclusion is valid.\n* It would have been nice to consider baselines such as Rope and Alibi relative positional embeddings to verify the performance improvement obtained by making the changes suggested in the paper.", "questions_to_authors": "Please see the reasons to reject section.\nA: Have you done any qualitative/human analysis of the generations from the models for the different pretraining corpora?\nB. Is it possible to apply any of these techniques to decoder only models for long context summarization/QA? Especially in the distractor setting in multi-hop QA where we have 10 context passages that the model has to parse through, would any of these methods help decoder only models where context window is a bottleneck?\nC: How do these methods compare to techniques such as RoPE and Alibi?", "paper_topic_and_main_contributions": "* The paper presents a comprehensive study on how to adapt existing pretrained language models for long sequence tasks. It considers techniques that fall under three different axes - model architecture, modeling objectives and pretraining corpus.\n* For model architecture - augmenting block wise attention with pooling layers results in best performance.\n* For the modeling objectives, T5 style demonising objective with a mix of short and long spans achieves the best result and for the pretraining corpus, the paper reports that using randomly concatenations passages from a large open-domain corpus gives better performance than using long document corpora like books.\n* Using these findings, a strong long context language model is trained that achieves strong results on long-text summarization and QA tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f4X68Kavkd", "reviewer": "Reviewer_eLL5", "summary": "", "questions": "Please address the concerns mentioned in \"Reasons to Reject\".", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a comprehensive study on adapting existing pretrained text-to-text models for long-sequence NLP tasks, offering valuable insights and findings for the research community. The authors explore the effects of model architectures, training losses, and pretraining datasets, providing an effective recipe for building high-performing long-context models.\n\n2. The proposed model achieves state-of-the-art performance on five long-sequence summarization tasks and consistently outperforms local-attention models that simply reuse BART's parameters. This demonstrates the effectiveness of the adapted model and its potential for improving downstream tasks such as question answering.\n\n3. The paper conducts performance analysis on input lengths, comparing the proposed model against the base model for short and long documents. The results show that the proposed model performs better than the base model on all data splits, with a larger performance gap for long documents. This suggests that the model's performance gains can be attributed to better long-context modeling, providing further evidence of its effectiveness.", "reasons_to_reject": "1. Lack of novelty or contribution: The paper may not present significant new findings, insights, or approaches that advance the field of natural language processing. If the paper's contribution is deemed incremental or does not sufficiently differentiate itself from existing work, it may be rejected.\n\n2. Methodological flaws or limitations: The paper may have methodological issues, such as flawed experimental design, insufficient evaluation metrics, or inadequate comparison with existing approaches. If the methodology is not rigorous or the results are not convincing, it may lead to rejection.\n\n3. Poor presentation or writing quality: The paper may have significant issues with clarity, organization, or writing quality. If the paper is difficult to understand, lacks coherence, or contains numerous grammatical or typographical errors, it may be rejected.", "questions_to_authors": "Please address the concerns mentioned in \"Reasons to Reject\".", "paper_topic_and_main_contributions": "This paper presents a study on adapting pretrained text-to-text models for long text sequences. The authors propose a recipe for building long-context models by replacing full attention with pooling-augmented blockwise attention and pretraining the model with a masked-span prediction task. They also compare the effectiveness of using randomly concatenated short documents versus existing long document corpora for pretraining. The findings show that the proposed approach achieves competitive performance on long-text QA tasks and sets a new state-of-the-art on long-text summarization datasets.", "excitement": 4, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "teBwfpdbsF", "reviewer": "Reviewer_kAsi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A valuable empirical study of methods for long-text modeling", "reasons_to_reject": "Although this paper explores some methods, I have a hard time understanding its most significant contributions. It explores many aspects (model, data, objective) but fails to give an in-depth analysis of each aspect. For example, its modeling exploration is more like an empirical study of existing methods. I noticed that this paper claims it has some interesting findings, but I don't think it shows sufficient insight that can help the community better understand or solve the long-text modeling challenge.\n\nAlso, the mainstream research focusing on long-text modeling is mostly about the GPT architecture now. While it makes no sense to follow the mainstream research to also study this problem on the GPT-style models, this paper should add more discussion about the recent related work because I think there are a lot of methods that can be shared regardless of enc-dec models or decoder-only models.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a method for adapting text-to-text models to handle longer text sequences. The authors conducted an empirical study of three aspects of the pretraining pipeline: the model architecture, the optimization objective, and the pretraining corpus. They proposed an approach to building long-context models from existing short-context models. The resulting model achieved competitive results on summarization tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a comprehensive study on adapting pre-trained text-to-text models for long text sequences. It explores three key aspects: model architecture, modeling objectives, and pretraining corpus. The authors recommend replacing full attention with pooling-augmented blockwise attention, using a T5-style denoising objective with short and long spans, and pretraining with randomly concatenated passages from the C4 corpus. These adjustments result in a strong long-context language model that excels in long-text summarization and QA tasks. The work requires further study on the claim that "randomly concatenating passages improves over long documents".
Decision: Accept-Findings