Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer
Authors: Qingru Zhang
Dhananjay Ram
Cole Hawkins
Sheng Zha
Tuo Zhao
Venue: EMNLP 2023 Findings (2023)
Keywords: Long-Range Transformer
Attention Mechanism
Mixed Attention Span
TL;DR: We propose MASFormer, a long-range transformer variant with mixed attention spans that achieves competitive performance to transformer with full attention while significantly reducing computational cost.

Abstract: Pretrained transformer models have demonstrated remarkable performance across various natural language processing tasks. These models leverage the attention mechanism to capture long- and short-range dependencies in the sequence. However, the (full) attention mechanism incurs high computational cost -- quadratic in the sequence length, which is not affordable in tasks with long sequences, e.g., inputs with 8k tokens. Although sparse attention can be used to improve computational efficiency, as suggested in existing work, it has limited modeling capacity and often fails to capture complicated dependencies in long sequences. To tackle this challenge, we propose MASFormer, an easy-to-implement transformer variant with mixed attention spans. Specifically, MASFormer is equipped with full attention to capture long-range dependencies, but only at a small number of layers. For the remaining layers, MASformer only employs sparse attention to capture short-range dependencies. Our experiments on natural language modeling and generation tasks show that a decoder-only MASFormer model of 1.3B parameters can achieve competitive performance to vanilla transformers with full attention while significantly reducing computational cost (up to 75\%). Additionally, we investigate the effectiveness of continual training with long sequence data and how sequence length impacts downstream generation performance, which may be of independent interest.

Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer
Qingru Zhangi; Dhananjay Ram", Cole Hawkins", Sheng Zha", Tuo Zhao
iGeorgia Institute of Technology Ammaonn Web Service (gingru zhang, tourzhao)@@ateech.euu {radhna, colehawk, wewaahanggaamazonnn...
Abstract
These models leverage the attention mechanism (Vaswani et al., 2017) to compute the dependency score for each pair tokens in an input sequence. Some practical tasks require these transformer models to handle long-sequence inputs like 8k to- kens. For example, chatbot systems gather long- term contexts of user interactions to generate infor- mative texts (Roller et al., 2021). Summarization for news, government reports, and academic papers request models take inputs of long sequences generate comprehensive summaries (Shaham al., 2022), otherwise models often miss important in- formation. Note that typical transformer models apply full attention to capture token dependencies pair-wise. It leads to a quadratic time and space complexity w.r.t. input length. However, such a complexity is prohibitive for long sequences. In particular, it incurs massive memory consumption during the back propagation. For example, trans- former model with 250M parameters consumes over 80G GPU memory when sequence length is Sk (Zuo et al., 2022).
Pretrained transformer models have demon- strated remarkable performance across vari- natural language processing tasks. These models leverage the attention mechanism to capture long- and short-range dependencies in the sequence. However, the (full) attention mechanism incurs high computational cost quadratic in the sequence length, which is not affordable in tasks with long sequences, c.g., inputs with tokens. Although sparse at- tention can be used to improve computational efficiency, as suggested in existing work, has limited modeling capacity and often fails to capture complicated dependencies in long sequences. To tackle this challenge, we pro- pose MASFormer, easy-to-implement trans- former variant with Mixed Attention Spans. Specifically, MASFormer equipped with full attention capture long-range dependencies, but only at small number of layers. For the remaining layers, MASformer only employs sparse attention capture short-range depen- dencies. Our experiments on natural language modeling and generation tasks show that decoder-only MASFormer model of I.3B pa- rameters can achieve competitive performance vanilla transformers with full attention while significantly reducing computational cost (up 75%). Additionally, we investigate fectiveness of continual training with long quence data and how sequence length impacts downstream generation performance, which may be of independent interest.
To address this scalability issuc, various ap- proaches have been proposed to reduce the com- plexity. One approach is sparse attention, which restricts each token to attend subset of tokens based on predefined sparsity patterns (Beltagy al.. 2020; Zaheer et al., 2020; Ainslie al., 2020). For instance, block sparse attention (Kitaev al., 2020; Ma et al., 2023) divides the input sequence into sev- eral blocks, and only intra-block attention is per- formed. Besides, sliding-winng window attention (Belt- agy et al., 2020; Zaheer et al., 2020; Ainslie et al., 2020) allows each token to attend to its neighboring tokens within a sliding window. These methods, though reducing the complexity of full attention, cannot sufficiently capture long-range dependen- cies. Other variants, such as kernel approximation (Peng et al., 2021) and low-rank approximation (Wang et al., 2020; Chen et al., 2021) methods, share the similar spirit and drawbacks. To com-
ef-
1 Introduction
Pre-trained transformer models have manifested superior performance in various natural language processing tasks such as natural language modeling (NLM) (Dai al., 2019; Radford et al., 2019), natu- ral language generation (NLG) (Brown et al., 2020) and natural language understanding (NLU) (De- vlin et al., 2019; Liu et al., 2019; He et al., 2021b).
Work was done during Qingru Zhang's 's internship at Amazon Web Service.

---

pensate for the lack of long-range dependencies. LongTSs (Guo et al., 2021) introduces global tokens that are obtained by average pooling on every block of tokens (Ainslie et al., 2020). However, the block pooling operations can weaken the signal of crucial tokens and prevent the long-range dependencies from being detected.
these signals.
To demonstrate the effectiveness of MASFormer, We conduct experiments on natural language mod- eling (ArXiv and PubMed Cohan et al. (2018)) and natural language generation (ArXiv, Cohan et al. (2018) and SCROLLS, Shaham et al. (2022)) tasks. Specifically, we compare the performance of MASFormer to other attention methods using pre- trained GPT-2 model (Radford et al., 2019) of 1.3 billion parameters. Our empirical results demon- strate that MASFormer consistently outperforms baseline methods across different attention cost (i.e. the total number of computed attention scores), In particular, MASFormer can achieve comparable performance to full attention while significantly re- ducing the computational cost. For example, with 27% of its attention cost, MASFormer achieves a close R2 score full attention QMSUM dataset.
In addition to these methods, state space mod- els (SSMs) prespecify global dependency patterns to capture the long-range dependencies only (Gu et al., 2020, 2021; Li et al., 2022; Zuo al., 2022; Ma al., 2023; Smith al.. 2023). These models can be regarded as linear recurrent neural networks with specifically designed fixed weights. As tai- lored for global dependencies, SSMs fail to effec- tively capture local dependencies, In order to com- bine both local and global dependencies, SPADE (Zuo et al., 2022) and MEGA (Ma et al., 2023) aug- ment SSM layers into transformer layers equipped with local attention. However, state space meth- ods require sophisticated implementation, and of- ten encounter computational instability during the back propagation, especially when scaling up to large model size (Gupta et al., 2022), SPADE and MEGA hence inherit these drawbacks.
We also make additional discoveries with MAS- Former, which are of independent interest, Firstly, investigate the effectiveness of continual train- ing for long sequence modeling. Many publicly available models are pre-trained with sequences shorter than 2048, and often fail to perform well on longer sequences (e.g. 8k/16k tokens). To bridge the gap, we explore the option of continual training to adapt these models to long sequences, thereby avoiding pre-training from the scratch. We discuss its effectiveness with MASFormer in Section 4.3. Secondly, we showease that increasing sequence length can yield more performance gains on down- stream tasks than NLM tasks evaluated by per- plexity. We are aware of the recent findings by Sun et al. (2021) that increasing context length exhibits limited impact on NLM perplexity. Nev- ertheless, when applying MASFormer to down- stream tasks like long-context summarization, we find that model performance benefits significantly from extending context length. Such a difference arises from the fact that predicting the next tokens in NLM primarily relies on locality of reference. Capturing infrequent long-range tokens can im- prove perplexity but not significantly. Therefore, we emphasize the necessity to evaluate model per- formance on downstream tasks that require long- range dependencies. Furthermore, our empirical evidence suggests that increasing the length can improve the performance only if models possess sufficient capability to handle additional long-range information. Local attention, as a counterexample, often fails to capture long-range signals and hence
Note that the aforementioned methods apply same attention mechanism for every layer. We chal- lenge this conventional wisdom and propose trans- former variant MASFormer (Mixed Attention Span transFormer). MASFormer utilizes full at- tention only at a subset of layers whereas employs sparse attention at the remaining layers. Our design is motivated by the phenomenon - that most con- texts in NLP data display a great deal of locality of reference (Zaheer et al., 2020; Beltagy et al.. 2020). That is, most of information about a token can be derived from its neighboring tokens. In contrast, long-range dependencies among tokens sparse and infrequent. Consider an academic paper as an example. Within paragraph, there exist numerous short-term dependencies. Neighboring tokens are closely connected to convey meaningful semantics. Across paragraphs, there can be small number of long-range dependencies. For example, tokens as- sociated to the primary theme of the paper exhibit rare and weak dependencies across a long span. Since long-range dependencies occur much less frequently, few layers of full attention are ade- quate to capture them. stark contrast, short-term dependencies are more frequent, necessitating local attention in the majority of layers to fully extract

---

benefits much less from long sequences.
local window of size w. In most of cases, block sparse attention exhibits similar performance as sliding-window attention (Zuo et al., 2022).
2 Background
2.1 Pretrained Language Models
Our Approach
Pre-trained transformer models (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020; Dosovitskiy et al., 2020; He et al., 202 bb,) have manifested superior performance in various NLP tasks. These models are often pre-trained on enormous amounts of unlabeled data in unsuperviseuussll-uuprnyy manner such that they can learn rich semantic knowledge. By further fine-tuning these pre-trained models, we can effectively transfer such knowledge to benefit downstream tasks (Zhang et al., 2023). Existing research on long-range transformers commonly requires pre- pre-training the proposed mod- els from scratch to accommodate new architectures and long inputs (Guo et al., 2021; Zuo et al., 2022). However, the significant training overheads raise barrier for the widespread utilization of these meth- ods across different language models. Motivated by this, we explore the possibility of leveraging existing pre-trained models and adapting them to long sequences though continual training.
We present method MASFormer, long- range transformer variant that mixes different tention spans across layers.
3.1 MASFormer: Mixed Attention Span
MASFormer leverages full attention exclusively at a subset of transformer layers, whereas it employs block sparse attention at the remaining layers. The structure of MASFormer is illustrated in Figure Ic. We choose full attention to encode long-range in- formation due to the following reasons: (i) full at- tention exhibits superior capability to capture long- range dependencies compared to sparse attention; (ii) full attention does not require sophisticated im- plementation and hence computationally stable compared to SSMs (Zuo et al., 2022; Gupta et al., 2022); (iii) full attention is compatible with exist- ing pre-tree pre-trained transformer models, enabling us to conduct continual training which we elaborate in Section 3.2. To mitigate the computational cost, we restrict the number of layers using full attention. MASFormer is motivated by empirical investiga- tions on performance comparison between models that apply the same attention span at every layer. Figure 2 presents the performance of block sparse attention and full attention on language modeling and summarization tasks. We find that, given long- sequence inputs, sparse attention is often insuffi- cient capture long-range dependencies beyond its attention span. As result, it shows unsatis- factory performance. To remedy it, one can either increase attention span or switch to full attention to improve model capability of capturing sophis- ticated dependencies. Though improving model performance, it incurs high computational cost. Confronting such trade-off between computa- challenge tional cost and model performance, the common practice that applies the same atten- tion span at every layer: MASFormer provides an alternative solution. Instead of increasing attention span evenly, MASFormer allocates large portion of attention computations to a subset of / layers by equipping them with full attention. Specifically, equipping bottom layers with full attention can yield the best performance suggested by our empirical analysis in Section 4.3'. At the remain-
2.2 Attention Mechanism
Suppose the input to the layer is X E Raxd where n is the input sequence length and dl is embedding dimension, then self-attention mechanism outputs
QK
Attn(X)= softmax
V
(1)
Vd
where Q = XW,, = XW, V VW, and W; WR. Wi, are learnable projection weights. Such full attention can simultancously evaluate the alignment between any pair of tokens in the se- quence. Specifically, denote the attention score A softmax(QK" /Vd), then Aij captures the alignment between tokens i and j. typical trans- former model applies the full attention at every layer. Denote the number of layers as L.. Then its attention cost is Lne..
Sparse attention variants are introduced to miti- gate the computational cost of full attention. Fig- ures la and lb illustrates the attention patterns of block sparse attention and diding-windoww atten- tion. For instance, block sparse attention divides tokens into blocks of size b and performs intra- block attention only, resulting in an attention cost bn. Sliding- window attention allows each token to attend its left/right neighboring tokens within a
Please see Section 4.3 for detailed explanations

---

Ouipusa
Tokans
Layer Full
Layer Full
Layer Block
Layer
Layer
Block
Block
(a) Block Attention
(b) Window Attention
(c) MASFormer
Figure I: Illsttaation of attention patterns of (a) block sparse attention with block size = 3; (b) sliding-window
attention with window size
(on each side); (c) MASFormer that integrates full and sparse attention.
14
13.19
Pempextry
Perpletity
10.19
25-
23.04
10.75
14:49
20
10
12:51
(a) NLM for ArXiv
(b) NLM for PubMed
(c) Sum. for ArXiv
(d) Sum. for GovReport
Figure (a.b): We evaluate the perplexity pre-trained GPT-2 model with block attention of differnet block after continual training, (c.d): We fine-tune GPT-2 model with block attention and compare the summarization
performance on ArXiv and GovReport under different block size. Here the input length is 8192.
ing layers, MASFormer utilizes block attention of small size m, resulting in controlled attention cost of (L 1)mn + in As mentioned in Sec- tion 1, such design is inspired by the phenomenon that most of contexts in NLP data exhibit a great deal of locality of reference. Long-range depen- dencies, in contrast, are less frequent. Therefore, is not necessary to enhance attention span at every layer. Instead, a few layers of full attention are sufficient to capture infrequent long-range signals. The majority of layers can maintain small attention spans adequately extract local dependencies and control the attention cost.
and pre-trained weights. Meanwhile, accelcration packages, such as FlashAttention (Dao et al., 2022) and xFormers (Lefaudeux et al., 2022), are applica- ble to further accelerate the computation of block attention and full attention in MASFormer.
3.2 Continual Training with Long Sequences
As mentioned, MASFormer can be implemented upon majority pre-trained transformers by mod- ifying their attention patterns. However, most publicly available models are pre-trained with se- quences shorter than 2048, and often exhibit subpar performance on longer sequences such as 8k/ 6k. To bridge this gap, we propose the continual train- ing to adapt the revised model on long sequences and new attention pattern. As such, we can preserve existing pre- traireened knowledge and circumvent the intensive overheads of pre-training from scratch. In particular, we first modify the attention pattern of the target model as proposed by MASFormer. If the pre-trained model uses absolute position em- beddings, we duplicate them to accommodate long sequences. Subsequently, model with long sequences (e.g.. 8k) from pre- training corpus like PILE (Gao al., 2020). Then we conduct continual pre-training using casual lan- guage modeling (CLM) objective. We discuss the effectiveness of continual training in Section 4.3.
Our empirical results demonstrate that, with the same attention cost, MASFormer significantly outperforms sparse attention. Remarkably, MAS- Former can achieve comparable performance to full attention while substantially reducing compu- tational cost. Therefore, by mixing different atten- tion spans, MASFormer strikes better balance be- tween computational cost and model performance. Moreover, MASFormer offers additional imple- mentation advantages. As using the same attention function, MASFormer is easy implement and compatible with existing pre-trained models. We can build MASFormer upon pre-trained transform- ers by changing their attention patterns, which does not involve modification on model architectures
provide the revised

---

4 Experiments
All sliding- attention is to apply sliding- window attention at every layer, which is another variant of sparse attention. shares the similar spir- its and often performs similarly as block attention. In the following experiments, we compare MAS- Former and the baseline methods across different attention cost C. That is, for all block sparse atten- tion, we set the block size b - C/ (Ln). For all Miding-winddow attention, we choose the window size as w = C/(2Ln). For MASFormer, we apply a small block size m = 1024 for its block atten- tion and set as (C L - mn). Then we observe how their performance evolves when enhancing the attention cost C or input length n. Experiment Overview. We briefly summarize the experimental contents as follows:
We evaluate the effectiveness and efficiency of MASFormer on natural language modeling (ArXiv and PubMed, Cohan et al. (2018)), natural language generation (ArXiv Cohan et al. (2018), QMSUM and GovReport Shaham et al. (2022)). We choose the GPT-3 XL model architecture (Brown et al., 2020) as our base model, which consists of 1.3 billion parameters and 24 layers and is pre-trained on PILE (Gao et al., 2020) for 300 billion tokens. GPT is a general purpose model that can be ap- plied to many tasks instead of tailoring them for specific tasks. As such, it makes easy control experiments and showease the difference among various methods.
Implementation Details. Our base model uses absolute positional embeddings with maximum length 1024. To accommodate longer inputs, we du- plicate its positional embeddings have the maxi- mum length as 8192 such that the model can handle sequences containing up to 8192 tokens. Then, we implement different attention methods by modify- ing the attention pattern of the base model. We implement all the models with PyTorch (Paszke et al., 2019). All the experiments are conducted on NVIDIA A100 GPUs.
Section 4.1 presents the perplexity evaluation of all the models on ArXiv and PubMed after contin- ual training.
Section 4.2 compares the summarization perfor- mance of the models on ArXiv, QMSUM, and Gov- Report after fine-tuning. Besides, we also discuss the difference between perplexity and downstream evaluation in reflecting model capacity to capture long-range dependencies.
Section 4.3 provides three crucial analyses: (i) we evaluate the benefits of increasing input length and discuss the requirements to attain these gains; (ii) we analyze the effectiveness of continual train- ing for long-sequence modeling; (iii) we conduct an ablation study to demonstrate that equipping bottom layers with full attention yields the most significant performance gains than other options. We further provide the explanations.
Continual Training Details. After changing the attention pattern, we conduct the continual training for MASFormer and baseline methods on PILE cor- pus (Gao et al., 2020) to adapt the revised models to new attention patterns and long-sequence inputs. We leverage the casual language modeling (CLM) objective train the model for 50,000 steps with a warmup of 2000 steps. We set the input length as 8192 and use a batch size of 128 such that the models are optimized with IM tokens per step. We use the constant learning 0.0001 for all methods. Baseline. We compare MASFormer with the fol- lowing methods:
4.1 Natural Language Modeling
4.1.1 Datasets and Evaluation Details
Datasets. We evaluate the perplexity of the up- dated GPT-2 for each attention method after con- tinual training. The evaluation is conducted on test sets of ArXiv and PubMed (Cohan et al., 2018). Table presents the statistics of these two datasets, Pubmed consists of scientific documents, with a document's content used as input and its corre- sponding abstract as the target summary. ArXiv is similar to PubMed, with documents from arXiv. Evaluation Details. We conduct the perplexity evaluation under two settings.
All full attention is to apply full attention at every layer. It has been adopted by most of existing transformer models as default. Although incurring the maximum attention cost, it achieves the best performance for most of our tasks. Hence, acts as an upper bound for other methods.
All block sparse attention is to upply block atten- tion at every layer, which is an effective method to reduce computational cost when modeling long se- quences. Block attention sets the attention span of each layer identical such that it evenly distributes the budget of attention computation across layers.
(i) We calculate the perplexity (ppl.) with all documents from test sets. Table presents the overall perplexity of dif- ferent models on two datasets. (ii) To showcase

---

Abenton
18
16
2
12.
MASFumend--::
Perplexity
30SSFomen))))
Perplexity
14
12-
8-
10
2000
4000
6000
8000
2000
4000
6000
8000
Example Length
Example Length
ArXiv ppl.
(b) PubMed ppl.
Figure 3: Perplexity evaluation on ArXiv and PubMed with examples of different length. Here z-axis is the maximum document length of each subset, ie., & x 1024 (A: 1,2.3.....)
the varying behaviors of models on documents of different length, we divide all documents into sev- eral subsets according to their length. Each sub- set consists of examples, whose length is within ((k-1) x 1024, A 1024] (k: 1,2,3,... Then, we evaluate the perplexity on each subset. Fig- ure presents the perplexity of models on different subsets of examples.
with 27% of its attention cost, MASFormer exhibits curve of ppl. v.s. length that closely resembles to that of full attention. This demonstrates the effec- tiveness and efficiency of MASFormer to capture long-range dependencies. In contrast, block sparse attention benefits much less from long contexts and underperforms both of them because of its incapa- bility to encode long-range signals. For example, when = 1024, block attention achieves similar perplexity on PubMed examples of different length.
4.1.2 Results
Table compares the overall perplexity on test sets of ArXiv and PubMed. The results suggest that, with = layers of full attention, MASFormer achieves comparable performance to all full atten- tion, while reducing 72% of its attention cost. With the similar attention cost C. MASFormer outper- forms all block attention that evenly distributes the budget of attention computation. For exam- ple, MASFormer with / achieves 8.75 ppl. on PubMed, which is 1.37 lower than that block attention of b = 2048.
4.2 Natural Language Generation
4.2.1 Datasets and Training Details
Datasets. We evaluate the downstream perfor- mance of models on several abstractive summariza- tion tasks compare their capability of handling long sequences in practice. Specifically, we fine- tune models on ArXiv (Cohan et al., 2018), QM- SUM and GovReport (from SCROLLS benchmark, Shaham et al. (2022)). Their statistics are summa- rized in Table 5. We mainly use ROUGE-2 (R2) score (Lin, 2004) as the evaluation metric, which is more important and sensitive than R1 and RL.
Methods
ArXiv
PubMed
Full attention
1,610M
8.63
7.63
Training Details. After continual training, we fine- tune each model and report R2 scores validation sets. Specifically, we fine-tune models for 3000 steps on QMSUM, 8000 steps on GovReport, and 12000 steps on ArXiv. We set the batch size as 64 for ArXiv and 32 for QMSUM and GovReport. We pick the learning rates from {1 x 10-5, 5 x 10-5, 1 x 10-4, 5 x 10-4), and choose the optimal ones to report the performance of each method. Moreover, the input length is fixed as 8192. We apply the greedy decoding for generation. Please see Appendix for more details.
Block (l=1024)
201M
13.19
12.19
Block (l=2048)
402M
10.75
10.13
MASFormer (l=2)
318M
10.25
8.75
MASFormer (l=4)
436M
9.31
8.25
MASFormer (1=8)
671M
9.63
8.25
Table 1: Perplexity evaluation on ArXiv and PubMed.
Figure 3 illustrates the perplexity variation of each method given examples of different length. We can tell that MASFormer and full attention show better performance longer documents, sug- gesting increasing context length can improve their prediction performance. Full attention, though in- curring the highest attention cost, always achieves the best performance due to its outstanding capabil- ity to handle sophisticated dependencies. Notably,
4.2.2 Results
In Table 2 and Figure 4, we present the fine-tuning results on QMSUM, ArXiv and GovReport across
Pllassc see Table in Appendix B for all ROUGE scores.

---

02
Block Almentionn
MASFormer
0.50
0.75
1.00
1.50
1,00
1:25
1.50
0.50
0.75
1.25
1.30
Attention Cos
Amtenion Cost
%10"
Attemton Cost
K30
(a) ArXiv
(b) QMSUM
(c) GovReport
Figure 4: Given input length as 8192, we compare summarization performance between MASFormer and block/full
attention when increasing the attention cost.
cal dependencies. Capturing infrequent long-ran aangrange tokens does not significantly improve perplexity. Thus, this discrepancy emphasizes the necessity evaluate long-range models on downstream tasks.
Methods
C
QMSUM
ArXiv GovReport
Full attention
1610M
8.00
19.32
28.83
110dow(w=1024)
402M
4.32
13.51
17.03
Block((=2048)
402M
5.03
9.61
12.31
4.3
Analysis
MASFormer((=4)
436M
6.59
14.91
18.82
4.3.1
Benefits of Increasing Sequence Length
Window(w=20488)
805M
5,05
15.21
22.79
In this section, we investigate the benefits of increasing input length for downstream perfor- mance. Specifically, we select the input length from {2048, 4096, 6144, 8192). and present the fine-tuning performance of full attention in Fig- ure 5. The results consistently demonstrate that the input length increases, the model's s performance improves. That is, downstream performance ben- efits significantly from long-sequence inputs. In contrast, increasing example length beyond 6k re- sults in marginal improvements in perplexity (See Figure 3), highlighting again the importance of downstream evaluation.
Block((=4096)
805M
5.15
14.50
23.64
MASFormer(==6)
553M
7.15
15.72
21.20
MASFormer((=8)
671M
7.46
17.00
24.42
MASFomerr(=12)
906M
8.70
18.58
26.26
Table 2: Summarization performance of models with different attention methods. The best results are shown in bold.
different attention cost. The results demonstrate that, with the similar attention cost, MASFormer significantly outperforms sparse attention variants, Furthermore, when enhancing attention cost, MAS- Former achieves greater performance gains than sparse attention methods. This is evident from the steeper slope of its R2 curve versus attention cost, in contrast to the baseline method. For exam- plc, when incrcasing C form 553M to 67 IM. the R2 score of MASFormer on QMSUM exhibits substantial improvement, reaching 8.70 from 7.46. Remarkably, this score surpasses even that of full attention. Therefore, MASFormer addresses the trade-off between computational cost and perfor- mance gains in more efficient and effective way.
In addition, when comparing the behaviors block attention in Figure 2c and 2d, we find that sparse attention often insufficiently capitalize on the benefits offered by longer inputs. For instance, given block size 4096, its performance on ArXiv remains nearly unchanged when increasing input length from 4096 (R2 15.52 in Figure 5a) 8192 (R2 14.49 in Figure 2c). This finding sug- gests that enhancing input length can only improve model performance the model possesses the suffi- cient capability to handle long-range dependencies.
Notice that, in order to achieve comparable sum- marization performance to full attention, MAS- Former needs at leaset / = 8 layers of full attention, and providing more can lead to more gains. This observation different from the findings in NLM (Figure 3) that increasing / beyond provides lim- ited improvement in perplexity. Their different capacity requirements arise from the fact that pre- dicting next tokens NLM primarily relies on lo-
4.3.2 Effectiveness of Continual Training
We analyze the effectiveness of continual training by comparing fine-tum inc-tuning performance of MAS- Former 8) under the following settings: (i) MASFormer without continual training (w.o. C.T.); (ii) MASFormer continually trained with short in- puts (C.T. (n=2048)); (iii) MASFormer continually trained with long inputs (C.T. (n=8 (192). Table 3

---

20
19.32
28.83
8.01
27.5
7.8-
16.94
7.65
23.54
7.6
22.5
14
20.0
7.39
13.09
Iopert length
Topui length
Input langh
(a) ArXiv
(b) GovReport
(c) QMSUM
Figure Fine-tuning performance of full attention under different input length.
presents fine-tuning performance of these models. We can tell that continual training with long inputs indeed facilitates the revised models to adapt to new structures and long-sequence inputs.
Position
QMSUM
GovReport
Every 3
28.26/6.94/25.03
26. 26.16/12.37/24.82
Top 8
20.89/4.52/18.3
-/-/-
Middle 8
27.27/5.99/24.06 20.80/9.01/99552
1= 8
QMSUM
GovReport
Bottom 8
30.91/7.45/27.02
54.33/24.42/51.87
w.o. C.T.
29.33/6.43/25.71 $3.28/23.61/5111
Every 2
35.34/16.04/33.68
C.T. (m=2048)
29.87/7.16/26.1
02.28/23.01/499
Bottom 12
32.53/8.70/28.75 56.98/26.26/54.46
C.T. (1=21122)
30.91/7.45/27.02 54.37/24.4255.887
Table 4: Performance comparison of MASFormers that apply full attention at different layers (# layers L=24).
Table 3: We report R1/R2/RL for the above results.
duce computational cost. However, as discussed in Section 4.3.3, this approach is neither effective nor efficient as MASFormer when handling long sequences. As shown in Table 4, applying full atten- tion every 2 or layers underperforms applying it bottom 12 8 layers. Therefore, alternating be- tween full and block attention results in additional computational cost and performance degradation. contrast, MASFormer presents an effective solution for efficient long-sequence modeling. It provides guidance on adapting existing pre-trained transformers to long inputs. Meanwhile, it pro- vides insights for designing large long-range mod- els, especially for deeper models. By equipping only subset of bottom layer with full attention. we can substantially mitigate computational cost. Additionaly, the computation of MASFormer can be further optimized by leveraging system-level acceleration techniques (e.g., Attenntn and xFormer) that support both block and full attention.
4.3.3
Where to use full attention
To answer where to apply full attention, we com- pare fine-tuning performance of MASFormers that apply full attention at (i) bottom layers; (ii) mid- dle layers; (iii) top layers; (iv) every L/t layers. The results in Table 4 demonstrate that equipping bottom layers with full attention yields the best performance. This because that long-range de- pendencies can be continually captured and rein- forced by bottom layers before propagated to upper layers. As such, these long-range signals can be effectively incorporated into the upper layers with local attention, facilitating their encoding of local information. In contrast, when equipping local at- tention at bottom layers, long-range tokens first aggregated with neighboring tokens by local atten- tion, thereby weakening their long-range signals. Morcover, alternating full and local attention ev- ery L/l layers, the long-range signals cannot be continually reinforced nor efficiently captured.
6 Conclusion
5 Discussion
We propose an efficient long- rangrrane transformer MASFormer that utilizes full attention at few of bottom layers and employs sparse attention at the remaining layers. Our empirical results on natural language modeling and generation tasks demon- strate that MASFormer can address the trade-off between computational cost and performance gains in a more efficient and effective way.
GPT-Neo (Black et al., 2021) introduces an atten- tion pattern that alternates full and window atten- tion. However, this models is not tailored for long the local window size 256 and has the maximum input length as 2048, unable to handle long sequences. Instead, this attention pattern is applied heuristically in an attempt to re-
sequences.

---

References
4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Joshua Ainslic, Santiago Ontanon, Chris Alberti, Va- clav Cvicek, 7achary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. Etc: Encoding long and structured inputs in transformers. arXiv preprint arXiv:2004, 08483.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint urXiv:2010.11929
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint (ox:::0004.055555
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint 0rXiv:2101.0000
Sid Black, Leo Gao, Phil Wang. Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large
Scale Autoregressive Language Modeling with Mesh- Tensorflow.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. 2020. Hippo: Recurrent mem- ory with optimal polynomial projections. Advances in neural information processing systems, 33:1474- 1487.
Albert Gu, Karan Goel, and Christopher Ré. 2021. Effi- ciently modeling long sequences with structured state spaces, arXiv preprint arXiv:2111.00 10399..
Mandy Guo, Joshua Ainslie, David Uthus, Santiago On- tanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2021. Longt5: Efficient text-to-text transformer for long sequences. arXiv preprint arXiv:2112.079)
Ankit Gupta, Albert Gu, and Jonathan Berant. 2022. Diagonal state spaces are effective structured state spaces. In Advances in Neural Information Processing Systems.
Beidi Chen, Tri Dao, Eric Winsor, 7hao Song, Atri Rudra, and Christopher Ré. 2021. Scatterbrain: Uni- fying sparse and low-rank attention approximation. arXiv preprint arXiv: 153433.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021a. Debertav3: Improving deberta using electra-style pre- training with diidient-disentangledd embedding shar- ing, arXiv preprint arXiv:21 11111111.9553.
Arman Cohan, Franck Demoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Téchnologies, Volume 2 (Short Papers), pages 615-621, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021b. Deberta: secoding-enhancee bert with disentangled attention. In International Conference on Learning Representations.
Nikita Kitaev, Lukasz Kaiser, and Auselm Levskaya. 2020. Reformer: The efficient transformer. Inter- national Conference on Learning Representations.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language mod- els beyond a fixed-length context. arXiv preprint 1:Xivv1901.02864
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong. Vittorio Caggiano. Sean Naren, Min Xu, Jieru Hu, Marta Tin- tore, Susan Zhang, Patrick Labatut, and Daniel Haziza. 2022. xformers: A modular and hack- able transformer modelling library. https: //github. www.aaaoooooororeehernccceo
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness.
Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadcepta Dey, 2022. What makes convolutional models great on long sequence modeling? arXiv preprint 2rX.9:2210.09298.
Jacob Devlin, Ming-Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.

---

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke 7ettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint 1.Kiv:997..1160
Sinong Wang. Belinda Z Li, Madian Khabsa, Han Fang. and Hao Ma. 2020. Linformer: self-attention with linear complexity. arXiv preprint arXiv: 2xXvv2006..04 04768.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances neural information processing systems, 53:17283-17297.
Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. 2023. Mega: Moving average equipped gated attention. The Eleventh Interna- tional Conference on Learning Representations.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng Hc. Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh In- termational Conference on Learning Representations.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary Veti,o Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Fang. Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style. perf-perrormacc deep learning li- brary. Advances Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurlPS 2019, De- cember 8-14, 2019, Vancouver, BC, Canada, pages 8024 8035.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Denis Charles Eren Manavoglu, Tuo Zhao, and Jianfeng Gao. 2022. Efficient long sequence modeling via state space augmented transformer. arXiv preprint arXiv:2212.08136.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. 2021. Random feature attention. In International Confer- ence on Learning Representations.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, lla Sutskever, et al. 2019. Language models unsupervised multitask learners. OpenAl blog, 188)9
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason We- 2021, Recipes for building chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 300-325, Online. Association for Computational Linguistics.
open-domain
Uri Shaham, Flad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, al. 2022. Scrolls: Stan- dardized comparison over long language sequences. arXiv preprint 1tXXi::2201.33533
Jimmy T.H. Smith, Andrew Warrington, and Scott Lin- derman. 2023, Simplified state space layers for se- quence modeling. In The Eleventh International Con- ference on Learning Representations.
Simeng Sun, Kalpesh Krishna, Andrew Mattarella- Micke, and Mohit lyyer. 2021. Do long-range lan- guage models actually use long-range context? arXiv preprint arXiv:2 109.09115..
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural procesinin processing systems, 30.

---

A Dataset Statisties
the following table, we provide the detailed statistics of datasets in our experiments, including example splits and length statistics.
Example Count
Input Length Median
Dataset
Train
Valid
Test
Average
90th percentile
ArXiv PubMed
203,037
6,436
6,440
10,720
8,519
20,170
119,924
6,633
6,658
4,748
3,883
8,883
QMSUM 1,257
272
281
9,497
14,197 8,841
27.761
GovReport 17,457
972
973
7,886
18,835
Table 5: Statistics of datasets. Input length measured in tokens using a SentencePiece Model.
B Natural Language Generation
B.1
The Results of All ROUGE Scores
Methods
C
QMSUM
ArXiv
GovReport
Full attention
1610M
31.50 8.00 27.81
46.13/ 19.32 41.89
60.53 / 28.83 57.88
Window (w=1024)
402M
23.31 4.32/ 20.62
35.90/ 13.51 32.19
49.82 17.03 47.42
Window (w=2048)
805M
26.73 5.05 23.40
38.74/ 15.21 34.87
56.14 22.79 53.50
Block (b=2048)
402M
26.24 5.03 23.13
21.85 /9.11 / 19.86
26.37 / 12.31 / 25.
Block (b=4096)
805M
26.96 5.15 / 23.85
35.95 14.50 32.37
49.83 / 23.64 47.50
MASFormer (l=4)
436M
29.86 / 6.59 25.87
38.85 14.91 / 34.98
46.67 18.82 / 44.39
MASFormer (I=6)
553M
30.83 7.15 27.12
36.29 15.72 32.96
49.26 21.20 46.89
MASFormer (I=8)
671M
30.91 8.00 27.81
43.31 17.00 39.12
54.37 24.42 / 51.87
MASFormer (I=12)
906M
32.53 8.70/ 28.75
45.19 18.58 40.72
56.98 26.26 54.46
Table 6: Finetuning performance of different attention methods.
B.2
Training Details
Methods
QMSUM
ArXiv
GovReport
Full attention
1x 10-3
1 1x 10-4
x 10 10-4
Window attention (w=1024)
x 10-4
5 10 5
x 10TH
Window attention (w=2048)
5 10-5
5 10-5
1 10-4
Block attention (w=2048)
1 1x 10
1x 10-5
1 10
Block attention (w=4096)
5 x 10-5
5 10-4
1 10-5
MASFormer (I=4)
5 10
1x 10
5X 10
MASFormer (I=6)
5 10
5x
5 10
MASFormer (l=8)
5 10
5x 10
5 10
MASFormer (1=12)
1 10-5
1x 10-4
1x 10-4
Table 7: The fine-tuning learning rate of each method on each dataset.
We conduct continual training for all attention methods with training date form PILE and input length as 8192. After continual training, we obtain the continually trained models for each method and fine-tune them on QMSUM, ArXiv and GovReport to compare their summarization performance. During the

---

fine-tuning, we the input length as 8192 for all datasets and all models. We apply the greedy decoding for generation and set the maximum oulput length as 256 for QMSUM, 1024 for GovReport, and 512 for ArXiv. Table 8 lists the details of these hyperparameters. Besides, we apply the linear learning rate schedule to fine-tune the models and the base learning rates are summarized in Table
Hyperparameter
QMSUM
ArXiv
GovReport
Training steps
3000
12000
8000
Batch size
32
32
64
Input length
8192
8192
8192
Maximum generation length
256
512
1024
Weight decay
0.001
0.001
0.001
Table 8: The other fine-tuning parameters for each dataset, which remain the same for every method.


Reviews:
[{"review_id": "vkETDSLhtl", "reviewer": "Reviewer_F2Z8", "summary": "", "questions": "None", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the helpful feedback and thank the reviewer for acknowledging the effectiveness and efficiency of our method. We would provide the following response to your comments. \n\n**Q:  The settings of parameters are underspecified or subjectively determined. The training/evaluation data are not widely available.** \n\nWe would kindly remind the reviewer that we provide the detailed training and evaluation hyperparameters in Section 4.1.1, Section 4.2.1, and Appendix B.2. We use the fixed learning rate 0.0001 for continual training and search for the best learning rate for each method during the fine-tuning. For the other hyperparameters, we set them as standard values following the prior work. Please see the mentioned sections for more details. For the datasets, we use the publicly available datasets (ArXiv, PubMed, SCROLLS) to evaluate model performance on NLM and NLG tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The new architecture MASFormer effectively combines self-attention and sparse attention mechanisms in a transformer network. By doing this the authors can significantly reduce the computation cost without impacting too much on the performance.", "reasons_to_reject": "None", "questions_to_authors": "None", "paper_topic_and_main_contributions": "The authors tried to lower the quadratic complexity of the self-attention mechanism by proposing MASFormer, which has self-attention in very few layers and sparse attention in the rest of the layers.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}, {"review_id": "3MISFFM8lD", "reviewer": "Reviewer_xgsE", "summary": "", "questions": "- Why haven’t you included an analysis of the time and space effectively required for each model?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thoughtful review and constructive feedback from the reviewer. We sincerely thank the reviewer for acknowledging the effectiveness and efficiency of our method. We provide the following response to the reviewer’s question.   \n\n**Q: The analysis of the time and space effectively required for each model.** \n\nThanks for your helpful suggestion. We kindly remind the reviewer that the time and space complexities are closely tied to the attention cost $\\mathcal{C}$, which is the practical cost of attention computations. In this regard, memory footprint and computational speed roughly align with the attention cost. We compare different methods under the same attention cost. While incurring similar time and space complexity, MASFormers consistently achieve much better performance than block/window attention baselines (please see Figure 4 and Table 2). \n\nNotice that, the practical training acceleration primarily depends on intricate implementation details. For example, we can employ a function of FlashAttention dedicated to block attention to accelerate the computation of block attention (please see FlashBlocksparseAttention in the official flash-attention repo). In contrast, window attention is not compatible with this function and hence exhibits less speedup.  In practice, when fixing attention cost as, e.g.,~430M and sequence length as 8k on summarization tasks, MASFormer and block attention show similar memory footprint and training speed which is 11% faster than the window attention. In the same setting, they can save 35% memory consumption and achieve ~45% training speedup compared to full attention.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is clear and well written.\n- The model proposed is quite simple and performs well when compared with the vanilla transformer while being more efficient. With similar attention costs, MASFormer performs better than transformers with block sparse attention or sliding window attention in all layers.\n- The method works well on top of a pre-trained model. Saving the time to pre-train the model from scratch.\n- The authors provide a detailed analysis, which shows that having the full-attention layers at the bottom of the transformer works best and that continuing the model training with the proposed attention mechanisms and higher sequence length improves the downstream performance.\n", "reasons_to_reject": "- The paper is missing a comparison between the time and space effectively required for each model.\n", "questions_to_authors": "- Why haven’t you included an analysis of the time and space effectively required for each model?\n", "paper_topic_and_main_contributions": "In this paper the authors propose a new efficient long-range transformer, the MASFormer (Mixed Attention Span Transformer). \n\nMotivated by the fact that most of the relevant information about a token can be derived from its neighboring tokens and that long-range dependencies are sparse and infrequent, the authors propose a transformer with some layers of full-attention and some layers with block sparse attention.\n\nAlso, instead of training the model, the authors simply continue gpt2’s training using bigger sequence lengths and the proposed attention.\n\nThe authors perform experiments on language modeling and summarization, which showcase the advantages provided by MASFormer.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rUCcPBR5QB", "reviewer": "Reviewer_THaz", "summary": "", "questions": "A. Table 1 : perplexity of sliding window is missing. Block atention cannot exchange info accross blocks - but allowing neighbouring blocks to communicate can alow info to propagate as we go up the layers. So sliding window ppl needs to be evaluated.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thoughtful review and constructive feedback from the reviewer. In response to the reviewer's comments, we provide the following clarifications and explanations. \n\n**Q1: The novelty of our method.** \n\nWe respectfully disagree with the reviewer’s view on the novelty of our approach. It is acknowledged that sparse attention and continual training have been respectively explored in the prior literature. Nevertheless, we are the first to apply the mixed attention span and continual training to long sequence modeling and combine them to resolve a new question. That is how to construct efficient long-range transformers based on well-established pre-trained models. This is our key novelty. The findings gained through MASFormer provide a deeper understanding of long-sequence modeling. Our method appears straightforward but is non-trivial. We value simplicity over unnecessary complexity: if the question can be effectively resolved by a simple approach, additional complexities may not be necessary. In this light, we would advocate our approach for its practical effectiveness, implementation friendliness, and compatibility with existing pre-training progress. \n\n**Q2: The perplexity results of sliding window attention.** \n\nThanks for your helpful suggestions. We completed the perplexity evaluation for sliding window attention. Similar as Figure 3, the following tables present the comparison among methods with close attention cost. \n\nTable 1: The perplexity evaluation on examples of PubMed. \n| Methods \\ Example length | 4096 | 5120 | 6144 | 7168 | 8192 | \n| :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | \n| Full attention | 8.26 | 7.47 | 7.00 | 6.70 | 6.41 | \n| Window ($w$=1024) | 8.06 | 7.67 | 7.89 | 8.21 | 8.17 | \n| Block ($b$=2048) | 9.72 | 11.29 | 9.65 | 11.54 | 10.09 | \n| MASFormer ($l$=4) | 8.49 | 7.69 | 7.23 | 6.94 | 6.65 | \n\nTable 2: The perplexity evaluation on examples of ArXiv.\n| Methods \\ Example length | 4096 | 5120 | 6144 | 7168 | 8192 | \n| :-----: | :-----: | :-----: | :-----: | :-----: | :-----: |\n| Full attention | 11.00 | 10.32 | 9.61 | 9.31 | 8.92 | \n| Window ($w$=1024) | 10.25 | 9.80 | 9.29 | 9.14 | 8.88 | \n| Block ($b$=2048) | 12.53 | 14.80 | 11.57 | 13.20 | 11.13 | \n| MASFormer ($l$=4) | 11.39 | 10.72 | 9.99 | 9.71 | 9.32 | \n\nFrom the results, we can tell that MASFormer surpasses both block and window attentions when handling long-sequence examples (>6k) on PubMed. On the ArXiv dataset, window attention exhibits the best performance. Generally, window attention achieves better performance than block attention due to the reason mentioned by the reviewer – the information can propagate among windows when going forward to upper layers. As a result, multi-layer window attention can capture longer local dependencies than block attention, resulting in better perplexity. \n\nAs discussed in Section 1(line 172), however, the perplexity performance primarily relies on capturing local dependencies and hence cannot practically reflect the downstream performance. When evaluated on downstream summarization tasks, window attention consistently underperforms MASFromer and full attention (as demonstrated in Table 2). Therefore, window attention is adept at capturing local dependencies but ineffective in long-range dependencies. \n\nWe kindly remind the reviewer that sliding window attention can also be integrated into our approach, replacing the block attention. The reason of selecting the block attention for MASFormer is that its better compatibility with FlashAttention provides additional acceleration: FlashAttention actually provides a function dedicated to the block attention, leading to a better speedup than the sliding window attention (please see FlashBlocksparseAttention in the official flash-attention repo). Note that, if computationally affordable, the sliding window attention can also be used in MASFormer, which has the potential to further improve the performance. However, it takes lots of time and requires substantial computational resources to re-train new MASFormers with the sliding window attention. Unfortunately, our access to computational resources has been constrained due to adjusted priorities after the EMNLP submission. We are committed to incorporating additional experiments with sliding window attention in our next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem is well motivated and the proposed trick is very simple, practical and can be used plug-and-play with the most commonly used LMs. Importantly, utilizing Flash attention, etc is also easy which is big plus.\n2. Experiments are sensible and results are fair - they do save compute and the degradations are not too significant.\n\nWhile the paper is not suitable for main conference, I do think that these findings are interesting enough for *Findings track*.", "reasons_to_reject": "There is no novelity - no new techniques are introduced.", "questions_to_authors": "A. Table 1 : perplexity of sliding window is missing. Block atention cannot exchange info accross blocks - but allowing neighbouring blocks to communicate can alow info to propagate as we go up the layers. So sliding window ppl needs to be evaluated.", "paper_topic_and_main_contributions": "Authors observe that for long-range language modeling and summarization tasks, it is more important to have layers that can capture long-range dependencies in the document early in the computation than later. Given a pre-trained Transformer LM, they propose to use chunked attention with non-overlapping chunks at all but bottom few layers where they retain full attention. They show that this saves compute while maintaining performance on long-range language modeling and summarization tasks. They show that sparsifying alternate layers, middle, bottom layers is not as effective as sparsifying the top layers.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 461: thought -> though"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary of the paper: The core idea presented in this paper is that for long-range language modeling and summarization, it is more critical to capture long-range dependencies early in the computation by using full attention in the bottom layers and sparse attention in the latter layers.

Pros:
- The simplicity of the idea makes it easy to replicate
- Their proposed setup works with pre-trained model

Cons:
- **Time and memory experiment** 
Efficient transformers are useful because they reduce the compute and time complexity of transformers from quadratic in the sequence length to something less than quadratic in the sequence length. Since the proposed method uses fully attention as the first few layer the computational and memory requirement is $O(c_1L^2 + c_2Lk)$ where  L is the sequence length, k is the local attention block size, $c_1 + c_2$ is the total  number of transformer layers. So theoretically the proposed method is quadratic in sequence length. 

Now, it is true that even if theoretically $L^2$ will dominate at large $L$ we could still think that the whole computation will be done much faster at lower $L$. First this is a drawback of the method not reflected in the experiments/analysis. Second, this needs to be demonstrated by measuring the computational time and memory required rather than “attention cost”. The reason is that compute time will be throttled by the lower layer, something that is not reflected in the “attention cost” metric. Moreover this tradeoff needs to be better analyzed in the paper.

- **Core idea of the paper and comparison to baselines**
Multiple Sparse Attention Transformers [1, 2, 3, 4, 5, 6] have been proposed over the past 4-5 years. The idea presented in the paper is combining a few fully attending layers at the bottom with local attention layers at the top. The closest idea to the paper is using few fully connected tokens (i.e. tokens that have full attention) in each layer. This has been explored  in *several* papers [4, 5, 6] along with an analysis of their compute and memory requirement [7]. Given the relationship of this paper with such previous work, the lack of comparison to efficient transformers, which are similar in principle, is another major concern for this paper.
Decision: Accept-Findings