Investigating Efficiently Extending Transformers for Long Input Summarization
Authors: Jason Phang
Yao Zhao
Peter J Liu
Venue: EMNLP 2023 Main (2023)
Keywords: summarization
nlp
architectures
long context
TL;DR: We investigate methods for extending pretrained Transformers for long input summarization, and introduce PEGASUS-X based on our findings

Abstract: While large pretrained Transformer models have proven highly capable at tackling natural language tasks, handling long sequence inputs still poses a significant challenge. One such task is long input summarization, where inputs are longer than the maximum input context of most models. Through an extensive set of experiments, we investigate what model architectural changes and pretraining paradigms most efficiently adapt a pretrained Transformer for long input summarization. We find that a staggered, block-local Transformer with global encoder tokens strikes a good balance of performance and efficiency, and that an additional pretraining phase on long sequences meaningfully improves downstream summarization performance. Based on our findings, we introduce PEGASUS-X, an extension of the PEGASUS model with additional long input pretraining to handle inputs of up to 16K tokens, which achieves strong performance on long input summarization tasks comparable with much larger models.

Investigating Efficiently Extending Transformers for Long Input Summarization
Jason Phang 1+ Yao Zhao² Peter J. Liu²
NNw York University, Google Research, Brain Team
jasonphangenyu.ede { yaazhaayz, jeterrliu/@googlee
Abstract
PEGASUS-X
25.5
While large pretrained Transformer models have proven highly capable tackling natural language tasks. handling long sequence inputs poses significant challenge. One such task is long input summarization, where inputs are longer than the maximum input context of most models. Through an extensive set of ex- periments, investigate what model architec- tural changes and pretraining paradigms most efficiently adapt pretrained Transformer for long input summarization. We find that stag- gered, block-local Transformer with global en- coder tokens strikes good balance of perfor- mance and efficiency, and that an additional pretraining phase on long sequences mean- ingfully improves downstream summarization performance. Based on our findings, we in- troduce PEGASUS-X: X. an extension of the PE- GASUS model with additional long input pre- training to handle inputs of up to 16K tokens, which achieves strong performance on long input summarization tasks comparable with much larger models.
SOROLLS Summ. Score
PEGASUS X
LongT5
25.0
LongT5
24.5
24.0
BART
23.5
BAR SLED
22.5
203M
400M
800M
1600M
3200M
Params
Figure I: Performance SCROLLS (Shaham al., 2022) summarization tasks. All models evaluated on up to 16K input tokens. PEGASUS-X outperforms other models at comparable model sizes. Scores (as of 08/08/22) shown are the average of the geometric mean of ROUGE-1/2/L.
of such tasks are often significantly longer than the maximum input lengths of most standard Trans- former models, and hence warrant both architecture modifications as well as new training regimes. For instance, to avoid the quadratic growth in memory consumption of attention in Transformers, many memory-efficient Transformer variants have been proposed (Tay et al., 2020, 2021). However, the manner in which these changes are incorporated into models has been inconsistent and ad-hoc, and there few established best practices. For in- stance, some works directly fine-tune on long-input summarization tasks (Zaheer et al., 2020; Pang et al., 2022), while others first perform additional pretraining (Beltagy et al., 2020). Because the high cost of training these models, there has yet to be systematic study of how best to adapt models for long input sequences. Hence, it has been diffi cult establish which model and training changes necessary complementary.
1 Introduction
Large pretrained Transformer models have proven to be extremely capable at tackling natural lan- guage tasks (Devlin et al., 2018; Brown et al., 2020). However, handling long textual sequences continues be significant challenge for these models. Training models to handle long sequences is expensive in both computation and memory, and moreover requires training and evaluating on long sequence data, which rarer and more costly to collect. Given the broad success of Transformer models on short-sequence language tasks, our goal is to investigate the best way to extend these models to handle longer input sequences.
In this work, focus the task of long in- put summarization: summarizing long input doc- uments into shorter text sequences. The inputs
To answer these questions, we conduct an ex- tensive empirical investigation into the architec- tural changes, model configurations and pretraining
Work done while at Google.

---

schemes to identify better approaches training Transformer models for long imput summarization. We evaluate set of efficient Transformer variants, and propose a simple biiee-wise local Transformer architecture with staggered blocks and global to- kens that strikes a good balance of performance and memory efficiency. We show that given fixed to- ken budget, pretraining on short sequences and then pre-adapting the model to efficient Transformer architecture by training on longer sequences leads better performance than only long imput pretrain- ing or no adaptation at all. We also investigate model design choices such as position encoding schemes, encoder-decoder layer distributions, and the impact of discrepancies between pretraining and fine-tuning architecture hyperparameters.
2.1 Prefraining
We generally follow the recipe from PEGASUS (Zhang et al., 2020) for pretraining PEGASUSBase sized models. All experiments in our ablation study performed pretraining with C4 (Raffel al., 2020) for 500k steps with 512 input tokens and 256 output tokens and a masking ratio of 45%, unless other- wise stated. For long input pretraining we extend the input length to 4096 tokens, and adjust the masking ratio from 45% to 5.625% reducing the ratio by factor of 8 to account for the 8x increase in imput sequence length. We also filter for only documents longer than 10000 characters.
2.2 Fine-tuning
We evaluate models by fine-tuning on the arXiv (Cohan et al., 2018) and GovReport (Huang et al., 2021) long input summarization tasks. Where relevant, we also fine-tune on the shorter-context XSUM and CNN/DailyMail tasks. For each ex- periment, we report the best validation set scores based the geometrie average (RG) of ROUGE-I, 1, ROUGE-2 and ROUGE-L scores (Lin, 2004) based on the rouge score package." Fine-tuning hy- perparameters can be found in Appendix E. Unless otherwise stated, we directly switch to the efficient Transformer architectures hetween pretraining (on shorter context) and fine-tuning (on longer con- texts), with no adaptation phase in between.
Based on the findings from our empirical inves- tigation, we adapt the pretrained PEGASUS model (Zhang et al., 2020) to tackle long input summarization on up to 16K input tokens. The re- sulting model, which we call PEGASUS-X X, attains top scores on long summarization tasks, outper- forming much larger models like LongT5 (Guo al., 2021). Moreover, impact on short input sum- marization performance minimal. A smaller ver- sion which we call PEGASUS-X Base attains simi- lar scores with much fewer parameters. Beyond summarization, we believe that many of our find- ings will be useful to the community for efficiently adapting Transformer models to handle ever longer input sequences for other tasks.
Ablation Experiments
In summary, our contributions are:
3.1 Encoder architectures
1. We evaluate a series of proposed efficient Transformer architectures as well as other model modifications, and report their efficacy and computational trade-offs when applied to long input summarization tasks.
We first investigate whether using an efficient Transformer encoder allows models to incorpo- rate longer input sequences while consuming rea- sonable amounts device memory. We consider two encoder architectures that exemplify different approaches to efficient attention. Big Bird (Za- heer et al., 2020) uses sparse attention computation, combining biddngg-wnnow and random attention, and set of global-attention tokens. Conversely, Performer (Choromanski et al., 2021) factorizes attention matrices via orthogonal random features. Both model also performed well on the LRA tasks (Tay et al., 2021). For this experiment, we perform both pretraining and fine-tuning with the same en- coder architecture to avoid the issue of mismatch between pretraining and fine-tuning architectures.
2. Based for adapting a short-context, pretrained Trans- former encoder-decoder longer inputs, and apply it to PEGASUS to greatly improve its long-document summarization performance, with comparable short-input performance.
findings, we propose recipe
2 Experimental Setup
Similar to Zhang al. (2020), we perform the ma- jority of our experiments with a PEGASUSBaR. sized model, before applying our findings to PEGASUS, Lange Lagge--izeee -ized model.
google- http:tttubb.cccccooeetteeeetttte ctw.:///ggooolleeerrrreeeee

---

XSUM
CNNDD
arX
GovRsport
Encoder
RI / R2 / RL
RG
RI / R2
RL.
RG
RI / R2
RL
RG
RI
R2
RL
RG
Steps/s
Mem
Transtomner
16.0/6.99330
27.9
39.5/19.0/28. 28.6
27.8
BigBird
39.6/ 31.7
27.6
39.3/18.2/28.
27.2
16.6/996/28.0
29.5
60.5/28.5/
50.1
37.3
0.31
1.88
Performer
16.0//4.0//5 28.7
24.5
37.4/174/2699
26.0
39.07 23.2//388
23.1
55.8 (5.8/20.2/22 24.7
30.3
0.96
1.12
Local
18.5/15.7/30.6
26.4
39.0/18.4/28
27.2
46.3/19.7/21
29.5
60.2/28.3/
30.0
37.1
1.00
1.00
Globbll-Local
38.7/16.2/3122
19.0//8.6//8.
27.3
20.2/
28.5
30.1
61.4/29.3/
38.0
Table Comparison of different encoder architectures on short (XSUM, CNN/DM) and long (arXiv, GovReport) summarization tasks. Training steps per second and memory are computed based arXiv, and normalized to Local Transformer performance.
Stagger Local Blocks
Use Global
arXiv
GovReport
In Decoder
Encoder
R2
RG
R1
R2
RL
RG
Global-Local
48.1/20.31 28.5
30.3
60.5/28.8/ 30.5
37.6
Global-Local
47.0/19.5/279
29.5
60.9/28.9/30
37.6
Global-Local
47.7/20.4/2.6
30.3
/.3/29A/30.8
38.1
Global-Local
46.7 19.5/29
29.4
59.5/27.8/294
36.5
Local
46.8/ 19.7
29.6
59.2/2779/30 30.0
36.7
Local
46.5 /9.5
29.1
58.8/27.5/288
36.0
Table 2: Comparison of architectural tweaks to Local and Globall encoder. Staggering local blocks uses
different blocks boundaries for different layers in block-local attention. Global information is incorporated in the decoder via an additional cross-attention before cross-attention over the encoded input.
In addition, we also introduce two simple vari- ants of local attention Transformer encoders. First, we use a simple block-local Transformer (Local), where encoder input tokens are divided into non- overlapping blocks, and tokens can only attend to other tokens within the block. Second, we extend the local Transformer by adding set of global to- kens with learned embeddings, that can attend to and be attended from every encoder token (Global- Local). These components are similar to the slid- ing window attention and global token attention Big Bird, ETC (Ainslie et al., 2020) and Long- former (Beltagy et al., 2020). However, we opt for the simpler block-local attention rather than sliding window attention, and compensate for the lack of overlapping blocks by staggering the local atten- tion blocks, which elaborate on in Section 3.2. As show below, the performance is highly com- petitive despite its simplicity.
Performer has relatively low memory consumption and trains efficiently, it performs worst among the architectures tested by a noticeable margin.
On the other hand, Local and Global-Local en- coders strike good balance of performance and efficiency. The simple local attention encoder, which uses block-local attention, attains perfor- mance close to that of Big Bird while being much faster and using much less memory. Global-Local trades off small amount of speed and memory for better performance, outperforming Big Bird.
Local attention is a strong baseline, and adding global tokens significantly improves performance. Both models are resource-efficient.
Takeaways:
3.2 Local and Global-Local configurations
Given the good performance of both Local and Global-Local encoder variants, we next consider further architectural tweaks to these models.
Results on short and long summarization tasks are shown in Table I, with the relative training steps per second and memory consumed per device for fine-tuning arXiv shown in the right-man most columns. Among the short tasks, the full-attention Transformer performs best, followed by BigBird. On the long tasks, Big Bird and Global-Local mod- els perform best, but Big Bird consumes signifi- cantly more memory and trains much more slowly than the other architectures. Conversely, although
First, we introduce staggering of local attention blocks. In block-local attention, tokens can only attend to other tokens within the same block. If the input tokens are divided up into the same blocks in every layer, this means that no information is exchanged across blocks through the entire encoder. To address this pitfall, we stagger attention blocks by shifting the block boundaries by half a block every other layer. We show an example of this Figure 2. practice, we implement this by padding

---

Block-local attention
Block-local attention with staggered blocks
Figure 2:
In block-local attention (a), the same block boundaries are used across all layers, preventing information from being shared across blocks. Staggering the block boundaries (b) be shifting the boundaries every other layer allows for cross-block interactions with minimal additional computational cost or complexity.
the hidden representations either side by half block and masking accordingly.
resources allow, but is not high priority compared to other model improvements. For the remainder of the ablation experiments, we use block size of 64 and 32 global tokens for consistency.
Secondly, in the Global-Local model, the de- coder only attends to the encoded token representa- tions, and not the global token representations. We consider a variant where we supply the global to- ken representations to the decoder and introduce second cross-attention layer that attends only the global tokens. Our goal is to allow the decoder to incorporate global information before performing cross-attention over the encoded sequence.
Takeaways:
Larger block sizes and/or number of global tokens leads to improved performance, although the effect saturates.
3.4 Other Architecture Modifications
We further investigate of series architectural mod- ifications to the encoder-decoder model, including the position encoding scheme (Table 8), scaling the encoder and decoder layers (Table 10) and using cross-attention only a fraction the decoder lay- ers (Table 12). We find that the sinusoidal position encoding provide good balance of performance and efficiency, and that balanced encoder-decoder with full cross-attention generally performs the best. More details provided in Appendix B.
Results are shown in Table We find that staggering local blocks noticably improves perfor- mance in both Local and Global-Local models. Per- formance improves even with Global-Local mod- cls, which already allow for bross-block interac- tions via global tokens, indicating that both model improvements are complementary. Conversely, in- corporating global token information the decoder did not lead to much performance improvement, particularly once staggered local blocks were used.
3.5 Pretraining vs Fine-tuning Architectures
Staggering local attention blocks significantly improves performance, and is com- plementary to global tokens.
Takeaways:
Previous works using efficient Transformer en- coders have generally taken the parameters of a full-attention Transformer pretrained on a shorter sequences and adapted them to efficient architec- tures, either directly during fine-tuning (Zaheer et al., 2020) with an intermediate stage of ad- ditional pretraining (Beltagy al., 2020). In this section, investigate if such approach is opti- mal, if models benefit from being pretrained with efficient encoders from the beginning. Note that we still perform pretraining on a short sequences (51 12 tokens), even with an efficient encoder.
3.3
Global-Local: Block Size and Number of Global Tokens
Next, we vary the block size and number of global tokens for the Global-Local encoder, with results shown in Table 3.2 Broadly, we find that increas- ing either block size or global tokens leads to im- proved performance, with a corresponding increase in memory consumption and computation time. However, the effect size from going to larger block sizes is not large, and saturates with larger block sizes or number of global tokens. As such, in- creasing either of these hyperparameters is ideal
We consider both pretraining with Transformer and pretraining with the efficient architecture for both Local and Global-Local models. We also vary the block size, the main difference be- tween a Transformer and Local Transformer is the block size (aside from staggering, a Local model with block size 512 is equivalent to a dense
'Experiments with very small block sizes or number global tokens ran into memory issues, because TPUs pad small di- mensions of arrays to cortain minimum lengths, leading to larger than expected memory consumption.

---

arXiv
GovReport
Block Size
Global Tokens
R1 / R2
RL
RG
R1 / R2
RL
RG
Steps/s
Mem
16
32
47.1/20.0/ 28.3
29.9
59.7 29.2
36.5
0.92
1.15
64
46.8/19.7 28,0
29.6
60.8/28.6/
30.0
37.4
0.75
1.54
128
47.7/20.0/28,
30.0
60.7/28.8/ 30.2
37.5
0.58
1.70
64
47.7/20.3/
30.2
61.0/29.3/
30.8
38.0
0.47
1.07
47.4/2022/ 28.5
30.1
60.9/29.1 30.7
37.9
0.94
128
47.8/ 28.6
30.3
60.9/29.0/ 30.3
37.7
0.85
1.26
128
32
46.9/19.7/28
28.0
29.6
60.9/28.7/30,
37.5
1.00
1.00
64
47.4/20.2/ 28.4
30.1
60.9/28.9/30. 30.8
37.8
0.96
1.05
128
/7.1/20.0/ 28.3
29.9
61.0/28.9/30 30.6
37.8
0.90
1.15
256
32
47.3/20.2/28.1
30.0
61.6/29.4/3.77 30.7
38.2
0.92
1.11
64
47.2/20.2/28. 28,4
30.0
59,2/28.6/30, 30.5
37.2
0.88
1.16
128
48.1 /
30.4
61.7/29.3/ 30.8
38.2
0.83
1.26
Table 3: Varying the block size and number of global tokens in Global-Local encoders. Training steps per second and memory are computed based on arXiv, and normalized to the run with Block Size=128 and Global Tokens=32.
Transformer), and hence the difference in block size also corresponds to the extent which the model needs to adapt between architectures. When adapting from a pretrained Transformer encoder to a Global-Local architecture, because the Global- Local model relies on newly introduced global to- ken embeddings, we initialize them by randomly sampling tokens from the vocabulary embeddings.
different information from short documents, hence limiting training only long inputs mae reduce the diversity of training data. Different long con- text Transformers have taken different approaches to pretraining on long inputs. For instance, Long- former (Beltagy et al., 2020) performed several additional stages of increasingly songer-sequence pretraining to adapt the initial RoBERTa to long sequence inputs. On the other hand, Long T5 (Guo et al., 2021) is pretrained exclusively with long in- put sequences. Others (Zaheer et al., 2020; Ivgi et al., 2022) perform no long input pretraining at all. In this section, we investigate how the balance short and long pretraining impact downstream per- formance, and try to find the best trade-off between pretraining cost and downstream performance.
Results are shown in Table 11. For Local mod- cls, pretraining with local attention using small block sizes tends to hurt performance, but at mod- erate block sizes (c.g. 64) there is little differ- ence between the two approaches. In contrast, for Global-Local pretraining with the efficient archi- tecture tends to perform better. We hypothesize that this difference arises because of the learned global embedding tokens, which are randomly ini- tialized when adapting from a pretrained Trans- former and hence may benefit from pretraining and being jointly trained with the local attention.
We consider two setups for pretraining: short- input pretraining, with 512 input tokens and 256 output tokens, and long-input pretraining, with 4096 input tokens and 256 output tokens. We de- scribe the corresponding differences in data pre- processing in Section 2.1. We fix the number of input tokens seen during training, and vary config- urations subject to this constraint. This constraint roughly proxies for the amount of compute con- sumed and corresponds to the number of input to- kens seen during pretraining.
Takeaways: For moderate block sizes, either pre- training or adapting to Local encoder performs about equally well, but pretraining with a Global- Local encoder performs slightly better.
3.6 Pretraining Schemes
Up to this point, we have only considered pretrain- ing with short sequences. We might expect that pretraining with longer sequences ought to improve performance on downstream long input summariza- tion. However, pretraining only on long sequences is computationally expensive and requires a large collection long input documents, which are rel- atively rarer. Long documents may also contain
We set our total input token budget at 131 billion tokens, which corresponds to million steps with 512 input tokens, compared to the 500k steps in the above experiments. This larger budget ensures that when only do long-inpu pretraining, the model
'If we instead fixed the number of training steps, long- input pretraining would consume far more compute for the same number of steps.

---

XSUM
CNNIDM
arXiv
GovReport
Pretraining Scheme
Encoder
R2
RL
RG
RI / R2
RL
RG
RI
RL
RG
RI
R2 / RL
Short (50%)
Local
39/2/18.1/279
27.1
46.8/19.7/
29.6
60.1/283//9.8
Global-T.ocal
16.4/ 31.5
27.4
39.1/18.6/ 28.3
27.4
47.7/20.4/25 28.6
30.3
61.9/29.6/ 30.8
38.4
Short (100%)
Local
27.1
39.2/18.6/2.3
27.4
46.9/19.7/280
60.1/28.3/93
Global-Local
39.9/
31.9
27.9
19.8/18.6/2 28.3
27.6
48.1/20.5/28/ 28.7
30.5
61.9/29 30.8
38.4
Short (75%) Long (25%)
Local
38.8/15.91 30.7
39.1/18 28.0
27.1
47.5/20.1/28.2
30.0
60.6/2899/366
Global-Local
39.61 16.81 31.7
27.6
39.8/18.8
28.5
27.7
48.4/20.7/ 28.8
30.7
61.8//988/1.1
38.5
Short (50%) Long (50%)
Local
38.47 30.5
26.4
19.4/18.1/221
27.1
47.7/20.2/ 28.3
30.1
60.9/29.1 30.7
37.9
Global- Local
39.3/ 16.416643311 31.4
27.3
39,4/18.3/28
27.3
48.4/20.9/ 29.1
30.9
61.7 31.2
38.7
Long (1008)
Local
36.0/ 14.07 28.6
24.3
38.4/12.7/2744
26.5
46.7/14.5/277
27.7
29.3
59.8/28.0
29.5
36.7
Global-Local
36.41 14.3/ 28.9
24.7
38.5/17.8/275
26.6
47.3/199/28
28.1
29.8
61.1 30.7
37.9
Table 4: Comparison of different pretraining formats, given a input token budget of 131B tokens, which corre- sponds to IM steps with 512 input tokens. Short pretraining uses 512 input tokens, whereas long pretraining uses 4096 input tokens.
is still pretrained for reasonable number of steps. We consider four pretraining configurations:
4 PEGASUS-X
Based on our findings, we settle on the follow- ing recipe for adapting PEGASUS models (Zhang et al.. 2020) to long sequence summarization.
Short-input for 100% of tokens (IM steps)
Short-input for 75% of tokens (98.3B, 750k steps), then long-input for 25% of tokens (32.8B, 31.25k steps)
We use Global-Local architecture with block staggering, large number of global tokens, and large block sizes during pretraining.
Short-input for 50% of tokens (62.5B, 500k steps), then long-input for 50% of tokens (62.5B, 62.5k steps)
We perform additional long input pretraining on 4096 token inputs for 300k steps.
Long-input for 100% of tokens (125k steps)
We extend input sequences up to 16384 input tokens in fine-tuning, depending on the task.
We compare the performance of the different pretraining scehemes in Table 4. We also include short-input pretraining for 500k steps for compar- ison. First, comparing short-input pretraining for 500k and IM steps, we find that more pretraining still improves performance, indicating that our base models may still be undertrained at 500k steps. Sec- ond, long-input pretraining performs consistently worse than the other variants, which we attribute having fewer training steps, again highlighting the issue of potential undertraining. For the middle three configurations, on the long tasks, all three non-long-one only variants atttain similar scores, with more long-input pretraining having slightly better performance, particularly on the ROUGE-2 and ROUGE-L scores. While the small absolute differ- ences scores make it hard to draw strong conclu- sions, we lean towards the conclusion that adding a short phase of long input pretraining can improve performance on long input summarization tasks.
We experiment with two model sizes: PEGASUS-X (PEGASUS eXtended) based on PEGASUS Large and PEGASUS XBase based on newly trained PEGASUSBus model which we call PEGASUSBaR***
We initialize the weights of PEGASUS-X and PEGASUS-X with the pretrained weights PEGASUS and PEGASUSBase- respectively. Only two new sets of parameters are introduced: global token embeddings, and a new LayerNorm for the global input representations in each Trans- former layer. This is ~ IM more parameters for PEGASUS-Xpas and 2M more for PEGASUS-X. We initialize the global token embeddings by ran- domly sampling tokens from the input embeddings, and we initialize the LayerNorm weights with the regular input LayerNorm weights.
The task- and model-specific hyperparameters for fine-tuning can be found in Appendix 15. For this section, report ROUGE- -Lumm rather than ROUGE-L for consistency with the metrics re- ported in other papers and leaderboards.
Takeaways: Given fixed compute budget, allo- cating some training steps long-input training can improve performance, although the optimal al- location is difficult to determine. Exclusively long pretraining results in worse performance.
"See Appendix C. 'https: //gp::::///i wtt.ccc./////o/ whw...ccw//oggleeeeeerr...

---

PEGASUS -Base
PEGASUS-X
PEGASUS-X and BART-LS share many similari- ties, and we see the strong performance of BART- LS as confirmation of the efficacy of parts of our recipe for longer sequence models. PEGASUS- X also outperforms LongT5 on both arXiv and PubMed, despite both compared LongT5 models having more parameters. However, we find that LongT5 performs much better on BigPatent, which is largely extractive summarization task. We hy- pothesize that larger hidden size may improve extraction over very long sequences.
Parameters
272M
568M
Global Tokens
128
128
Block Size
512
512
Batch Size
512
1024
Additional
300K steps
300K steps
Pretraining
Table 5: Hyperparameters of PEGASUS-X Models
4.1 Results on Summarization Tasks
Long summarization tasks In Table 6, com- pare the performance of PEGASUS models those of PEGASUS-X X three long-imput summarization tasks: arXiv, Big Patent and PubMed. In all three tasks, we see significant improvements in perfor- mance of PEGASUS-X over PEGASUSBase+ and PEGASUS-X X over PEGASUSLagg- To iso- late the impact of additional long input pretraining compared to only switching the architecture dur- ing fine-tuning, also include evaluation on the PEGASUS models using the Global-Locke archi- tecture with no further pretraining, which we list in the table as PEGASUSBac* Global-Local.
Short summarization tasks We show in Ta- ble 14 the performance of PEGASUS and PEGASUS-X models on shorter summarization tasks, where there is a slight regression in perfor- mance of both PEGASUS-X models compared to their PEGASUS equivalents. We hypothesize that long input pretraining might negatively impact the performance shorter input tasks because of the data filtering for long documents, resulting in potentially less diverse training data distribution.
4.2 SCROLLS Summarization Tasks
We also compare to reported results of Big Bird-PEGASUSS (Zaheer et al., 2020), LED (Belt- agy al., 2020), Top-Down Transformer (Pang al., 2022) with both Average-Pool (AvgP) and Adaptive-Pood (AdaP) variants, BART-LS (Xiong and XL, and SLED
We report the performance of the PEGASUS- X models on the summarization tasks in the re- cently introduced SCROLLS benchmark in Table 7. This includes GoyReport (Huang et al., 2021), the ForeverDreaming subset of SummScreen (Chen et al.. 2022), and QMSum (Zhong et al., 2021). PEGASUS-X outperforms all other models GovReport, setting the state of the art the dataset.? It also performs comparably both LongT5 and Hop-Down-Avgh on Summ- Screen/FD, although it underperforms LongTS models and BART-LS on QMSum. Morcover, PEGASUS-XBas also performs competitively, out- performing both LongT5 models on GovReport, and only a small margin behind PEGASUS-X X on all three tasks. PEGASUS-X X also outperforms a larger model with similar 16K input length.
et al., 2022a), (Ivgi et al., 2022). LED, Top-Down and SLED are initialized with BART weights with no additional pretraining on long input sequences. BART-LS is concurrent work that also incorpo- rates staggered block-local attention and addition long-sequence pretraining, in addition to pooling layers and different pretraining data.
PEGASUS-X X outperforms Big Bird-PEGASUS on all tasks, and Aop-Down-AvgP on both com- pared tasks. Although Top-Down-Adal outper- forms PEGASUS-X, it uses much more complex fine-tuning setup, using importance tagger reference summaries to construct token pooling weights, whereas PEGASUS-X X only uses standard fine-tuning. Even so, PEGASUS-X still outper- forms Top-Down-AdaP on PubMed. PEGASUS-X outperforms BART-LS on PubMed and slightly underperforms on arXiv; as mentioned above,
5 Pertinent Related Work
Many works such as Zaheer et al. (2020), Beltagy et al. (2020), Tvgi et al. (2022) have investigated extending short input models to longer sequences using efficient attention mechanisms. In closely comparable work, Guo et al. (2021) pretrained T5 model on long sequences from scratch, incor-
madhtwo- Saavors-of-rougeee
"Big Bird-PEGASUS only has context of 3072 tokens, likely due to the larger memory consumption of Big Bird.
As of 08/08/2022

---

arXiv
Big Patent
PubMed
Model
#Params
RI
R2
RL.s
RG
R1
RLs
RG
R1
RTs
RG
PEGASUSBEL Buac
271M
34.8/10.2/22
20.0*
43.5/204/31.**
30.5*
40.0/15.2/25.2"
24.8*
PEGASUSpa USsee
271M
42.2/15.8/
37.3
29.2
51.2/32.6/
41.0
40.9
44.1/18.3/ 40.1
31.9
PEGASUS B Global-Local
272M
47.6/20.2/
42.4
34.4
58.1/39.57
47.2
47.7
47.3/214/ 43.0
35.2
PEGASUS-X
272M
49.4/2166/
44.0
36.1
61.3/42.6/
50.1
50.8
49.6/23.6/
45.2
37.5
PEGASUSEER
567M
44.7/17.2/2577
27.0*
53,4/32.9/42.1
42.0*
15.1/19.6/27.4*
28.9*
PEGASUS-X
568M
50.0/218// 44.6
36.5
64.8/47.5/ 54.3
55.1
F1.0/24.7/ 46.6
38.9
BART-LS
460M
50.2/22.17
45,4
36.9
50.3/24.3/ 46.3
38.4
Longformer Encoder-Decoder
464M
46.6/19.6/
41.8
33.7
Top-Down (AvgP)
464M
48.7/20.7/
43.9
35.4
48.3/214/
44.2
35.7
Top- Now-Down (AdaP)
464M
$1.0/219/
45.6
37.1
51.1/23.3/
46.5
38.1
Big Bird-Pegasus
567M
46.6/19.0/
41.8
33.3
60.6/42./
50.1
50.5
46.3/20.7/
42.3
34.4
Ttaage
770M
48.3/216//
70.4/56.//
62.7
63.1
50.0/24.77
46.5
38.6
L5x
3B
48.4/219/
44.3
36.1
76.9/66.1/
70.8
71.1
50.2/24.8/
46.7
38.7
Table 6: Comparison long summarization tasks (Test sets). Results for other models taken from their respective papers. *: PEGASUS (Zhang et al., 2020) only reports ROUGE-L and not ROUGE-LSum.
GovReport
SummScrcen/FD
QMSum
Model
#Params
RI / R2
RL
RG
RI / R2 RL
RG
RI
R2
RL
RG
PEGASUS
Base
272M
59.3/29.3/ 30.9
37.7
35.0/ 8.9 / 20.4
18.5
32.9/ 9.8 /21.4
PEGASUS-X
568M
60.3/30.0/ 31.5
38.5
35.7/
200.
33.2/
/21.6
BART SLED
406M
58.0/26.9/ 27.6
35.1
33.8/ 8.0 18.5
17.1
32.1/10.2/21.0
19.0
BART-IS
460M
59.4/29.8/30 30.8
37.9
37.7/ 10.2/
20.2
35.1/ 23.3
21.4
Top- Top-Down-Ave Down-AvgP
464M
35.8/ 8.9 /30.6*
21.4*
Adda
464M
-.-
36.8/ 9.2 /31.1*
21.9*
TSnge
770M
84
29.8
35.5
35.6/ 9.2 222
19.1
15.1/12.0/ 23.3
21.4
L5ngTSxx.
3B
/4.7/28.2/
30.2
36.0
35.8/
9.6
21.1
19.4
34.9/1188/2 23.5
21.3
UL.2
20B
53.6/26.1 28.8
34.3
32.9/ 7.8
19.4
17.1
31.1/ 8.5 2004
17.5
Table Comparison on SCROLLS benchmark (Summarization tasks, Test sets). Results SLED, BART-LS. LongT5 and UL2 models taken from the SCROLLS benchmark leaderboard. ": Top-Down Down (Pang al., 2022) reports much higher scores for ROUGE-L on SummScreen/FDD than any other model, and may have been computed with variant of ROUGE-L that involves splitting on sentences rather than newlines.
porating sliding window attention and global rep- resentations. However, pretraining only on long sequences significantly increases the pretraining time, and show in Section 3.6, pretraining first short inputs and then subsequently on long inputs is much more cost efficient.
Conclusion
In this work, we investigate a range of proposed improvements to Transformer models to effectively and economically handle long inputs in summariza- tion tasks. Through extensive ablation experiments, find simple but effective recipe for extend- ing short-input models to tackle long-input sum- marization. Based on our findings, we introduce PEGASUS-X, an extended version of PEGASUS with a modified architecture and additional long- sequence pretraining. We show that PEGASUS-X sets the state of the art on two long input summa- rization tasks (GovReport and PubMed) and per- despite
In concurrent work released shortly before this submission deadline, Xiong et al. (2022a) also in- vestigated extending short input Transformer mod- els for long input tasks. While they focus on BART rather than PEGASUS, they similarly find that global tokens, staggered block-local attention, and extended pretraining greatly improve performance, lending further support to findings. Their fi- nal model also incorporates pooling layers and is trained on different data.
forms competitively many others, being much smaller than some compared models. Our findings can be extended models in other do- mains beyond summarization, both for pretraining long input models from scratch as well as extending already pretrained short sequence models.
broader treatment of related work can found in Appendix A.

---

Limitations
Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. Etc: Encoding long and structured data in transformers. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP 2020)-
Challenges of Evaluating Long-Document Summarization Models
One limitation of our work is that evaluation of long-document summarization models is chal- lenging, and while we evaluate on the widely used benchmarks for long-document summariza- tion models, we highlight here the difficulties of measuring the capabilities of such models. In addi- tion to the widely accepted issues with automatic evaluation of model--geerateer summaries with met- rics such as ROUGE, long-document summariza- tion brings about new challenges. In particular, there are relatively fewer long-document summa- rization tasks available to evaluate models on, and many of them (e.g. arXiv, Pub Med, SummScreen) are constructed by repurposing existing data and proxies for summaries (e.g. abstracts) rather than explicitly written summaries. As such, the avail- able datasets for summarization reflect the data that is easy to repurpose into summarization rather than practical downstream summarization settings; in other words, the available evaluation datasets may not match the distribution of data or settings where such models are realistically used.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05//3
James Bradbury. Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018. JAX: composable transformations of Python+NumPy programs.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022. SummScreen: A dataset for ab- stractive screenplay summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8602-8615, Dublin, Ireland. Association for Computational Linguistics.
On scoring generations, human evaluation should idcally be conducted to mcasure the qual- ity of model-generated summaries. However, the much longer input texts also means that human evaluation of summaries becomes much more ex- pensive and onerous, as raters would need to read the whole input before judging the quality of the summary.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sar- 16s, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2021. Rethinking attention with per- formers. In International Conference on Learning Representations, ICLR 2021.
More discussion on the challenges of evaluat- ing long-da long-document summarization models can be found in Wang et al. (2022).
Findings May Not Generalize to Other Tasks
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Na- zli Goharian. 2018 A discourse-aweer aware attention model for abstractive summarization of long docu- ments. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Short Papers), pages 615-621, New Orleans, Louisiana. Association for Computa- tional Linguisties.
We have confined our study to summarization tasks, as it matches our goal of investigating the ability for models to process large input contexts, with less focus on generating long outputs. We acknowl- edge that our ablation studies and experiments focused solely on summarization tasks, and that our findings may not directly apply or extend to other long-input language tasks.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arxiv:1810.04805.
References
Joshua Ainslie, Santiago Ontañón, Chris Alberti, Va- clay Cvicek, Zachary Fisher, Philip Pham, Anirudh
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian

---

Chelba, and Colin Cherry, 2021. Scaling laws for neural machine translation. arXiv:2109.0774
Bo Pang, Erik Nijkamp, Wojeiech Kryšcinskki Silvio Savarese, Yingbo Zhou, and Caiming Xiong. 2022. Long document summarization with top-down and bottom-up inference.
arXiv preprint
Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2021. Longt5: Efficient text-to-text trans- former for long sequences.
Ofir Press, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In International Confer- ence on Learning Representations.
Curtis Hawthorne, Andrew Jacgle, Cätälina Cangea, Sebastian Borgeaud, Charlie Nash, Mateusz Mali- nowski, Sander Dieleman, Oriol Vinyals, Matthew Botvinick, Ian Simon, Hannah Sheahan, Neil Zeghi- dour, Jean- Jean-Baptist Baptiste Alayrac, João Carreira, and Jesse Engel. 2022. General-purpose, long-context autoregressive modeling with perceiver ar.
Colin Raffel. Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Re- search, 21(140):1-67.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Mar- vin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van Zee. 2020. Flax: A neural network library and ecosystem for JAX.
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. 2022. Scrolls: Standardized comparison over long lan- guage sequences.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Si- monyan, Erich Elsen, Jack W. Rae. Oriol Vinyals, and Laurent Sifre. 2022. Training compute-opting large language models. CoRR, abs/2203. 155566.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yun- feng Liu. 2021. Roformer: Enhanced transformer with rotary position embedding.
Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao. Sharan Narang. Vinh Q Tran, Dani Yogatama, and Donald Metzler. 2022. Scaling laws model architectures: How does inductive bias influence scaling?
Luyang Huang, Shuyang Cao. Nikolaus Parulian, Heng Ji. and Lu Wang. 2021. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. pages 1419-1436 On- line. Association for Computational Linguistics.
Yi Tay, Mostafa Dehghami, Samira Abuar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sehastian Ruder, and Donald Metzler. 2021. Long range arena A benchmark for efficient trans- formers. In International Conference on Learning Representations.
Yi Tay, Mostafa Dehghami, Dara Bahri, and Donald Metzler. 2020. Efficient transformers: A survey. CoRR, abs/2009.06732.
George Hudson and Noura AI Moubayed. 2022. Muld:
The multitask long document benchmark. In Pro- ceedings of the Language Resources and Evaluation Conference, pages 3675-3685, Marseille, France. European Language Resources Association.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Maor Ivgi, Uri Shaham, and Jonathan Berant. 2022. Ef- ficient long-text understanding with short-text mod- els.
Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Ja- son Phang, and Samuel R. Bowman. 2022. SQuAL- ITY: Building a long-document summarization dataset the hard way. arXiv preprint 2205.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess. Rewon Child, Scott Gray, Alee Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. CoRR, 01/22001.8361..
Wenhan Xiong, Anchit Gupta, Shubham Toshniwal. Yashar Mehdad, and Wen-tau Yih. 2022a. Adapt- ing pretrained text-to-text models for long text se- quenees.
Wojciech Kryšcińiski, Nazneen Rajani, Divyansh Agar- wal, Caiming Xiong, and Dragomir Radev. 2021. Booksum: A collection of datasets for long-form narrative summarization.
Wenhan Xiong. Barlas Oguz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Scott Yih, and Yashar Mehdad. 2022b. Simple local attentions re- main competitive for long-context tasks. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational
Chin-Yew Lin, 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- fion Branches Out pages 74-81, Barcelona, Spain. Association for Computational Linguistics.

---

Linguistics: Human Language Technologies, pages 1975-1986, Seattle, United States. Association for Computational Linguistics.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslic, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances Neural Information Processing Systems, 33.
Biao Zhang, Behrooz Ghorbani, Ankur Bapna, Yong Cheng. Xavier Garcia, Jonathan Shen, and Orhan Firat. 2022. Examining scaling and transfer of lan- guage model architectures for machine translation.
Jingging Zhang, Yao 7hao, Mohammad Saleh, and Peter Liu. 2020. PEGASUS: Pre-tral Pre-training with tracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference Machine Learning. volume 119 of Proceedings of Machine Learning Research, pages 11328 11339. PMLR.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, Dragomir Radev. 2021. QMSum: A new benchmark for query- based multi domain meeting summarization. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905-5921, Online, Association for Computational Linguistics.

---

A Full Related Work
tribution lies in our extensive ablation study that identifies architectural tweaks that improve and, just as importantly, do not improve downstream performance.
Long Document Summarization Several new long imput summarization datasets and benchmarks have been recently introduced, providing better measures of long imput summarization capabil- ity as well as prompting new interest in this re- search direction. The BookSum dataset (Krys- ciñski et al., 2021) consists of paragraph, chapter, and full summaries of books on Project Gutenberg based on web-scraped educational website. (Chen et al., 2022) consists of television show transcripts and episode summaries based web-scraped fan- written summaries. The SCROLLS benchmark (Shaham et al., 2022) and the MuLD benchmark (Hudson and Al Moubayed, 2022) consist of multi- plc natural language tasks with long inputs, includ- ing long input summarization. The SQuALITY dataset (Wang et al., 2022) consists of question- focused summaries of Project Gutenberg stories, where annotators write summaries based on dif- ferent questions that cover different aspects of the same story.
Among the listed model architectures for long input summarization, LongTS (Guo et al., 2021) is the most similar to PEGASUS-X, sharing similar encoder-decoder architecture, similar training ob- jective in generating masked sentences, and mix of local attention and global information sharing for the encoder. We briefly highlight the key dif- ferences between the two models. Firstly, LongT5 trains from scratch on long sequences, whereas we initialize our model weights with PEGASUS weights (which is trained on short sequences) be- fore doing additional pretraining long input quences. This significantly reduces the overall pre- training cost, as short sequence pretraining and be performed much more economically. LongT5 also uses the T5 relative position biases whereas PEGASUS- X uses sinusoidal position embeddings- as shown in Section B. T5 relative position biases perform slightly better but are significantly slower. The efficient encoder architecture between the two models is also different: LongT5 uses transient global representations based on pooling chunks of tokens, whereas PEGASUS-X - uses learned global token embeddings. LongTS also uses sliding win- dow local attention based on ETC (Ainslie et al., 2020), whereas we use a simpler block-local at- tention with staggered blocks. Lastly, the largest LongT5 model is 3B parameters, more than 5x the size of PEGASUS-X.
Efficient Transformers Many efficient Trans- former variants have been introduced in recent years (Tay et al., 2020), and we discuss here the works more relevant to this manuscript. (Beltagy et al., 2020) use global tokens well as sliding window local attention, implemented using custom CUDA kernels. The ETC model (Ainslie al., 2020) uses both global tokens and block-wise slid- ing window local attention, although the global attention is incorporated based on the first few to- kens of sequence, rather than separately learned global tokens. Zaheer et al. (2020) extend ETC by adding random attention blocks, but we found that this significantly increases code complexity and computational cost. Guo al. (2021) sim- ilarly extend ETC's block-wise sliding window attention, but computes transient "global token" representations by pooling over blocks of tokens. Pang et al. (2022) propose to augment the Long- former encoder-decoder with additional pooling layers to improve mng-seeuence summarization performance. Ivgi et al. (2022) propose an alter- native approach to sparse attention via encoding overlapping chunks and fusing information across chunks int he decoder. We highlight that while the final Global-Local model architecture that we set- tle on shares similarity with several other proposed efficient Transformer architectures, our key con-
More broadly, Tay al. (2021) compare vari- ety of efficient Transformer architectures on set of tasks designed to probe long-sequence processing capability, evaluating the different models on both performance as well as computation requirements. Tay et al. (2022) further evaluate the scaling proper- ties of novel Transformer architectures, finding that deviating from full attention tends to hurt down- stream performance. Xiong al. (2022b) showed that simple local attention variants can be highly competitive with more complex sparse attention schemes, consistent with our findings.
B Details of Architecture Modification Experiments
B.1 Position Encoding Schemes
New position encoding schemes encoding schemes such as RoPE (Su et al., 2021) and ALiBi (Press

---

XSUM
CNN/DM
arXiv
GovReport
Position Eicoddin
R1 / R2
RL
RG
R1 / R2
RL
Step/s
None
26.8
22.6
25.67 7.8
117.7
15.2
36.17 9.8
/22.0
19.8
38.3/13.2/ 18.7
21.1
0.96
Sinusoidal
39.8/16.9/31.8
27.8
40.0/18.6/28,
44.5/17.6/26.
27.6
10.//188/2.5
25.6
0.96
T5
40.1/17.1/32
28.0
398//888//8
27.8
44.9/
26.8
27.8
40.2/19.5/ 22.9
26.2
0.53
RoPE
19.8/16.9/31
31.8
27.8
39.2/18.7/28.5
27.5
43.5/17.2/26,
27.1
40.0/
19.1/22
25.8
0.85
Absolute
39.1/16.4/31
27.2
39.7/18.7/28.5
27.7
44.3/17.5/26.5
27.4
18.6/17.5/21.
24.2
1.00
Table 8: Comparison of position encodings schemes for a Transformer encoder-decoder. Training steps per sec- ond computed based on arXiv summarization. Absolute position embeddings are replicated longer input sequences, following Beltagy et al. (2020). Training steps per second is computed based on arXiv, and normalized to the run with absolute position embeddings.
arXiv
GovReport
Position Encoding
RI R2 / RL
RG
R2
RL
RG
Factor= 10000
48.1 22.1/20.4/22 28.6
30.4
38.0 61.4/29.5/ 30.9
60.9/29.3/ 30.8
Factor500000
48.1 /0.4/ 28.6
30.4
38.3
Table Comparison of different scaling constants in sinusoidal position encodings.
et al., 2022) have garnered recent attention, show- ing improved performance on downstream evalua- tions. As input sequence lengths have gotten much longer, and in particular longer than the dimensions of hidden representations, previous choices of posi- tion encoding may no longer be optimal. Moreover, relative position encodings such as RoPE, T5 and ALiBi may be better suited for adapting models to different input lengths between pretraining and fine-tuning. Hence, this is good opportunity to revisit the choice of positioning encoding schemes in encoder models.
ALiBi to cross-attention.
Our results are shown in Table 8. We find that al- though T5 performs the best, it is also almost twice as slow the other position encoding schemes, which is consistent with the findings of Press et al. (2022). Sinusoidal position encodings and RoPE perform only slightly worse than T5 with much bet- ter efficiency, making them more desirable choices. Given the much simpler implementation of sinu- soidal position encodings, we opt to stick with them for the remainder of the experiments.
Takeaways: Sinusoidal position encodings still remain good choice for long input Transformers.
Because of the more complex interaction be- tween local attention blocks and relative position encoding implementations, we conduct prelimi- nary investigation with a full-attention Transformer. We pretrain with an input length of 512, and fine- tune with an input length of 2048 for the long sequence tasks this experiment also tests the propensity for position encodings to be adapted longer sequences downstream. In addition to the sinusoidal position encoding used in PEGA- SUS and Vaswani al. (2017), we also consider the mucket-based relative position encoding scheme of T5, RoPE, absolute position embeddings, and no position encoding as baseline. For absolute position embeddings, we follow the recipe of Belt- agy et al. (2020) and duplicate the learned position embeddings to handle longer sequences before fine- tuning. The chosen position encoding scheme is applied to all parts of the model, including both the encoder and the decoder. We do not experiment with ALiBi, as we found no natural way to adapt
B.2 Scaling Encoder and Decoder Layers
Scaling laws (Kaplan et al., 2020; Ghorbani et al., 2021; Zhang et al., 2022) that describe the em- pirical relationship between model sizes and per- formance have proven surprisingly consistent and gotten significant attention in recent years. We present in this section small set of scaling experi- ments, exploring the distribution of layers between encoder and decoder.
Our results are shown in Table 10. In the top half, we fix the total number of layers to 24, and con- sider both mccoerrheavy and decoder-heavy distri- butions, for both Local and Global-Local models. We observe that impact of distribution of encoder and decoder layers on performance is relatively small. For Local models, see slight boost from decoder-heavy models. For Global-Local models, we observe that a balanced incoder-decoded outper- forms encoder- and decoder-heavy models, both of

---

XSUM
CNN/DM
arXiv
GovReport
Architecture
Enc
RI
R2
RG
RI / R2 / RL
RG
RI
R2
RL
RG
RG
Local
18
6
37.4/15.0/2.77
25.5
39.0/18.2/27. 27.9
27.0
46.0/14//266
29.1
58,91
4/
36.1
12
37.5/149//977
25.5
38.5/18.0/22 27.6
26.7
45.4/18.9/27.3
29.2/ 29.3
36.3
18
37.7/15.1/29.
25.7
38.5/18.1/22.
26.9
46.3/19.3/7.6
29.1
59.4/27.8/ 29.5
36.5
Global-Local
18
38.6/15.9/30.9
26.7
39.2/18.5/ 28.2
27.3
47.3/20.1/28,
60.2/28.7/
37.5
12
38.6/ /5.9/ 30.7
26.6
40.0/ 28.3
27.6
47.5/20.1/ 28.3
30.0
21.1/29.3/ 30.7
38.1
18
37.7/15.1/29.9
25.7
38.5/18.1/27.
26.9
46.4/1995/273 27.9
29.3
60.3/28.6/ 30.0
37.2
Global-Local
18
18.5/15.7/3066
26.4
38.7/18.4/
28.1
27.1
47.3/20.01 28.3
29.9
60.2/29.2/3100
37.9
12
18
38.6/ 15.81///0
26.5
38.6/18.31 28.0
27.0
47.5 20.3/ 28,5
30.2
60.9/29.0/
30.4
37.7
Table 10: Varying the distribution of encoder/decoder layers)
which perform about comparably.
the results of pretraining and fine-tuning Global- Local models with cross-attention only on specific layers on a variety of configurations. We find that reducing the number of cross-attention layers leads to a drop in performance, but the impact on per- formance is smaller than expected. For instance, with only aross-attntiion the first and sixth layer, the Global-Local model still outperforms Local model. The reduction of cross-attention layers also leads to a corresponding improvement in training step and reduction in memory consumption.
We also consider cases where we further increase the size of either the encoder or decoder to 18 lay- ers, shown in the second half of Table 10. We observe no improvement in performance over the 12/12-layes encoder-decoder, and suspect that other hyperparameters (e.g. hidden size) might be the bottleneck rather than the number of layers.
We highlight here that because of the asymmetry of the input and output lengths, there are different computational trade-offss to different balances of encoder and decoder layers. Encoder-heavy mod- els require more memory because of the long input sequences, whereas decoder-heavy models are rel- ative slower at inference because the autoregres- sive nature of decoding. Given the relatively small difference in the margin of performance, memory or computational constraints may outweigh the per- formance differences in practical scenarios.
Given the small drop in performance from using fewer decoder layers with cross-atis -attentttnnion, we con- sider the viability of dropping cross-attention layers after pretraining. In other words, we take a Global- Local model pretrained with full cross-attention, drop the cross-attention for subset of layers, and fine-tune directly. Our results are shown in Ta- ble 13. We find that dropping the cross-attention after pretraining again only leads to a small (ad- ditional) dip in performance. This indicates that dropping cross-attention may be viable strategy for further reducing memory requirements for an existing pretrained model with small performance trade-off, and pretraining a separate model from scratch is not necessary.
Takeaways: A balanced encoder-decoder per- forms best, but the difference in performance may be outweighed by other resource considerations.
B.3 Partial Cross Attention
Given the use of an efficient attention architec- ture, which has memory consumption scale lin- early rather than quadratically in input sequence length, another major memory bottleneck is the encoder-decoder cross-attention. Because each de- coder layer attends separately to the long encoder representations, and the attention dense, this is a large contiguous chunk of memory that we could seek to reduce.
Dropping cross-attention for frac- tion of decoder layers can reduce memory con- sumption at the cost of slight performance regres- sion. Cross-attention can be dropped after pretrain- ing, with an associated performance trade-off.
Takeaways:
B.4
Comparison short summarization tasks
Perceiver AR (Hawthorne et al., 2022) demon- strated strong performance by using only single cross-attention at the bottom layer of an autoregres- sive language model. Based on these results, we in- vestigate the impact only having cross-attention on a subset decoder layers. In Table 12, we show
C PEGASUSB... Base+
In a similar finding as Hoffmman al. (2022), we found that PEGASUS benefits from training on significantly more tokens. As such, we trained a PEGASUS Base for a much larger number of to-

---

arXiv
GovReport
Pretraining + Fine-tuning
Block Size
R1 / R2
RL
RG
R1 / R2
RL
RG
Transformer -> Local
16
46.4/196//7.9
29.4
$9.6/28.2/29.
36.9
46.5/19.3/27.8
29.3
59.5/28.0/29.6
36.7
256
46.8/19.7/2..0
29.6
59.8/28.0/2988
36.8
Local + Local
16
45.9/19.1/27.5
28.9
59.0/27.5/29.3
36.2
46.5/19.5/27.8
29.3
59.7/28.1/29,
36.8
256
47.1/19./28.1
29.8
59.7/28.5/ 30.3
37.2
Transformer - Lloaal-Local
16
16.0/19.2/2..5
29.0
60.3/28.2/29. 29.8
37.0
64
47.0/20.0/2.,2
29.8
60.8/28.7/30. 30.1
37.4
256
47.6/20.3/284
30.2
60.8/28.7/300 30,0
37.4
Global-Local - Gloaal-Local
16
47.1/20.0/28.1
29.9
59.7/27.8/29.2
36.5
64
$7.7/20.3/ 28.5
30.2
61.0/29.3/30
38.0
256
47.3/20.2/28. 28.3
30.0
61.6/29.4/ 30.7
38.2
Table Comparison of adapting models architectures between pretraining and fine-tuning.
XSUM
CNN/DM
arXiv
Gov
Cross- CtussAttietionn
R1 / R2
RL.
RG
R1
/ R2
RL.
RG
R1
/ R2
RI.
RG
R1 / R2
RL.
RG
Step/s
Mem
Full
38.8 38.8/16.0//14 31.0
26.8
19.5/18 6/ 28.4
27.5
47.7 22.7/20.4/26 28.6
30.3
61.3/29.4/
30.
38.1
1.00
1.00
Cr06s(0,2,46668,, 10]
38.3/15.6/00
48.1/20.4/
61.0/29.0/
Cro6s(0,3,6,91111
38.0/15.3/305
30.2
26.0
38.8/18.4/28 28.1
27.2
46.9/19.9282.
29.7
60.1/28.6/ 30.2
37.3
1.15
0.88
Cross(0.4,8,
37.8/1533/01
25.9
38.5/18.1/271
26.9
47.6/20.2/28.
30.1
60.9/28.9/00.
37.6
1.15
0.86
Cross[0,6,
37.4/148//29 29.7
25.4
188//881//79
27.0
46.9/19.7/28.
28.1
29.6
60.3/28.5/
30.2
37.3
1.18
0.87
Cross(0.6]
37.3/1499/9.77
25.5
38.3/18.0/274
27.8
26.8
47.1/19.8/2.1
29.7
60.4/28.1 29.7
36.9
1.21
0.85
Table 12: Comparison of models with cross-attention only in subset the 12 decoder layers. Training steps per second and memory are computed based on arXiv, and normalized the Cross[0.6] run.
kens (the same as PEGASUS which achieves much better performance than the previously re- leased PEGASUS Base model.
with JAX (Bradbury et al., 2018) and Flax (Heek et al., 2020). PEGASUS PEGASUS-EE..ann and PEGASUS- Xwere trained by converting the weights from the TensorFlow checkpoint to Flax checkpoint for- mat, and then continuing with long input training.
D Encoder Architecture Hyperparameters
For experiments in Section 3.1, BigBird, Local and Global-Local all use a block size 64. BigBird and Global-Local also use 32 global tokens. Per- former uses 256 random features.
E Fine-tuning Hyperparameters
For arXiv, fine-tune with input length of up 16384 tokens and 256 output tokens. while for GovReport we use an input length of 10240 in- put tokens and 1024 output tokens given the longer summaries for the task. For XSUM and CNN/Daily Mail, with use an input length of 512, and output lengths of 64 and 128 respectively, following PE- GASUS hyperparameters. The full set of hyper- parameters for fine-tuning models shown in Table 15.
F Engineering Details
The original PEGASUS model was trained using a codebase based on TensorFlow. The experiments in this paper were run using a new codebase written

---

arXiv
GovReport
Cross-Atiention
Model
RI
RL
R2
RL.
RG
Pretrained
Full
47.7/20.4/22 28.6
30.3
61.3
30.8
38.1
Cross{0.2,4466,8,00
48.1/20.4/ 28.6
30.4
61.0/29.0/ 30.7
37.9
Cross[0.6]
4 28.1
29.7
60.4/28.
1/2997
36.9
Converted
(r0.6,00.,44,668,000
46.4/ 28.1
29.5
60.2/ 30.3
37.4
Cross[0,6]
46.2/ 19.7 28.1
60.2/28.11 29.8
36.9
Table 13: Comparison models pretrained with cross- for subset of layers, and adapting a pretrained model by dropping cross-attention layers only during fine-tuning
CNNi/aillMail
XSum
Model
R1
R2
RLs
RG
R1 / R2
RLs
RG
PEGASUS
41.8/1888/
38,9
31.3
39.8/16.//
31.7
27.6
PEGASUS USBusse
42.5/
39.6
32.4
43.8/21.2/
36.0
32.2
PEGASUS -BXae
42.5/20.1/ 39.6
32.4
42.9/20.1/
35.0
31.2
PEGASUS
44.2/21.5/
41.1
33.9
47.2/24.6/
35.7
PRGASUS-X
43.4/21.2/
40.6
33.5
45.8/228//
37.6
34.0
Table 14: Comparison on short summarization tasks (Test sets)
Dataset
Batch Size
Learning Rate
Num Steps
Max Input
Max Output Tokens
Beam Size
Beam
Tokens
Alpha
PEGASUS MAAASUSSXBRS
XSum
64
8c-4
97.5K
1024
128
4
0.8
CNN//ailyMail
64
8e-4
410K
1024
128
4
0.8
arXiv
64
8c-4
92.5K
16384
256
Big Patent
64
8e-4
272.5K
16384
256
PubMed
64
8e-4
85K
8096
256
GovReport
64
8e-4
40K
12288
1024
SummScreen
64
8e-4
90K
16384
256
QMSum
8c-4
7.5K
16384
256
PEGASUS-X
XSum
8e-4
5k
1024
128
0.8
Maal
8e-4
7.5k
1024
128
0.8
arXiv
64
8c-4
85k
16384
256
1
Big Patent
8e-4
390k
12192
256
1
PubMed
64
8c-4
47.5k
12192
256
GovReport
64
8c-4
75K
12288
1024
SummScreen
64
8c-4
40K
12192
256
QMSum
64
8c-4
35K
12192
256
Table 15: Hyperparameters for fine-tuning models


Reviews:
[{"review_id": "8wOKZWoupy", "reviewer": "Reviewer_fXYW", "summary": "", "questions": "A. What is the impact of different choices of global token embeddings on the performance of PEGASUS-X for long-document summarization tasks?\nB. How does the performance of PEGASUS-X compare to other models for long-document summarization in terms of computational resources required for training and inference?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the kind feedback.\n\nA) We show in Section 3.3 / Table 3 that having more global tokens generally leads to better performance, albeit at a memory and computational cost.\nB) At inference time, the computational cost should be similar to other models with Global+Local layers (e.g. Longformer), with the exact performance depending on the optimizations used (e.g. Longformer uses custom CUDA kernels). For training, the primary cost comes from more long-context pre-training, but we feel our investigation (Section 3.6 / Table 4) provides a good answer to the trade-off in the short vs. long table.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper conducts extensive ablation experiments and empirical investigations to explore various improvements to Transformer models for long-document summarization tasks.\n-  PEGASUS-X achieves state-of-the-art performance on long-input summarization tasks in two datasets: GovReport and PubMed.\n- The paper addresses the computational efficiency of long-document summarization models. It proposes a block-wise local Transformer architecture with staggered blocks and global tokens, striking a balance between performance and memory efficiency.", "reasons_to_reject": "Nothing comes to my notice.", "questions_to_authors": "A. What is the impact of different choices of global token embeddings on the performance of PEGASUS-X for long-document summarization tasks?\nB. How does the performance of PEGASUS-X compare to other models for long-document summarization in terms of computational resources required for training and inference?", "paper_topic_and_main_contributions": "This paper addresses the problem of effectively and efficiently handling long inputs in summarization tasks. It investigates various improvements to Transformer models to tackle the challenges posed by long-document summarization. The paper proposes PEGASUS-X, an extended version of PEGASUS, with a modified architecture and additional long-sequence pretraining. The goal is to achieve state-of-the-art performance on long-input summarization tasks while being computationally efficient.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Nothing comes to my notice.", "Typos_Grammar_Style_and_Presentation_Improvements": "Nothing comes to my notice."}, "extra_text": {}}, {"review_id": "6fe9L5pQR2", "reviewer": "Reviewer_8ipF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the kind feedback.\n\nRegarding BART-LS: we mentioned in the related work that BART-LS is work concurrent to this manuscript. More broadly, BART-LS investigates a similar set of methods and settle on a similarly configuration, using a staggered block-local attention and extended pretraining for the best performing version. BART-LS is in between PEGASUS-X-base and PEGASUS-X-large in the number of parameters, and is also trained on a different set of data. Hence, while the numerical results are not directly comparable (in terms of extrapolating conclusions on architecture), we find that the good performance of BART-LS is consistent with the overall findings of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a strong long input Transformer model for summarization tasks. The architecture design choices are solid and well-studied.", "reasons_to_reject": "The empirical effectiveness seems to be insignificant to previous models like BART-LS which has even fewer parameters. For example, PEGASUS-X in Table 6 and 7 both show marginal (<1%) improvements over BART-LS. It is also unclear whether PEGASUS-X is faster than BART-LS or not.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper studies long input summarization tasks and introduces an extension model (called PEGASUS-X) to the pretrained PEGASUS model. PEGASUS-X has a staggered and block-local Transformer architecture with global encoder tokens and it provides a better trade-off between task performance and efficiency. PEGASUS-X supports input tokens with 16K and shows strong task performance compared to baseline models.\n\n\nThe paper has several contributions: (1) a systematic study of the model architectures for long input summarization, including efficient encoders, local and global block Transformers, and their combinations (2) a simple and effective architecture extension based on the findings from the study, strong evaluation results for both long and short summarization tasks.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DRoubcwYuZ", "reviewer": "Reviewer_5wpq", "summary": "", "questions": "- Why do you choose to only conduct the study on long document summarization? It seems to me that this study can be beneficial to more tasks, such as long-form QA.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the kind feedback.\n\nWe chose to study long-context models in the setting of summarization tasks as it is a well-studied task with established datasets used in other academic works. Focusing on a single category of tasks with well-established (if imperfect) metrics allowed us to experiment more extensively with architectural and training configurations, without needing to juggle too many different evaluation settings or choices (e.g. evaluation of generated responses for QA).\n\nWe agree that summarization is limited in scope compared to the broader range of long-context tasks, and that even among summarization tasks there are many sub-genres and domains of data. This work can certainly be extended to cover broader domains of data and tasks, where different model configurations may be advantaged or disadvantaged in different settings. We see this work as an attempt to carefully and meticulously study the model and training configurations within a fixed and well-established if slightly narrow scope.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- They extensively study the effect of important design choices in a block attention-based efficient Transformer on long document summarization. In my opinion, it is of huge significance to conduct such a solid ablation study. Though they don't have a novel model design and one can probably find many existing models similar to the model they experiment with, prior work that introduces new models rarely investigate the effect of their designs with apple-to-apple comparisons, thus failing to truly reveal the effectiveness of those model design. For example, the difference in the pre-training data is a common confounding factor. This work is doing a pretty good job revealing the effects of those design choices.\n- They also adapt the PEGASUS model to a long-context version, PEGASUS-X. It has the best performance on PubMed and GovReport.", "reasons_to_reject": "- The domains of data used in their study are limited. It would be great to cover more difficult tasks such as meeting summarization and narrative summarization. Though SummScreen and QMSum are tested in the final experiment comparing PEGASUS-X with existing pre-trained models, having one of those datasets in the ablation study would make it more extensive.", "questions_to_authors": "- Why do you choose to only conduct the study on long document summarization? It seems to me that this study can be beneficial to more tasks, such as long-form QA.", "paper_topic_and_main_contributions": "This paper optimizes the block attention-based efficient long-context Transformer model by extensively studying the effects of various design choices, such as the block size, the number of global tokens, the arrangement of blocks, the positional encoding, and the pre-training scheme. Part of their experimental results show that, in order to achieve better performance on long document summarization, it is crucial to have larger block sizes, more global tokens, staggered blocks. Based on their findings, they adapt the pre-trained encoder-decoder model PEGASUS to a long-context model named PEGASUS-X, which performs better on long document summarization datasets including GovReport and PubMed than existing pre-trained models. To sum up, their contributions are:\n- They carry out an extensive study of design choices in a block attention-based efficient Transformer.\n- They introduce a new long-context pre-trained Transformer, PEGASUS-X.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the challenge of effectively handling long inputs in summarization tasks by proposing PEGASUS-X, an extended version of PEGASUS, with a modified architecture and additional long-sequence pretraining. PEGASUS-X aims to perform state-of-the-art long-input summarization tasks while maintaining computational efficiency. It introduces a block-wise local Transformer architecture with staggered blocks and global tokens to balance performance and memory efficiency. PEGASUS-X demonstrates strong performance on datasets like GovReport and PubMed. However, the paper's limitation lies in its domain coverage, as it primarily focuses on specific datasets and broader tasks like meeting and narrative summarization tasks are not included in the study. Also,  the paper's empirical effectiveness compared to BART-LS shows marginal improvements.
Decision: Accept-Main