CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling
Authors: Jun Zhang
Shuyang Jiang
Jiangtao Feng
Lin Zheng
Lingpeng Kong
Venue: Submitted to ICLR 2023 (2023)
Keywords: Long Sequence Modeling
Benchmark
Efficient Attention

Abstract: Transformer has achieved remarkable success in language, image, and speech processing. Recently, various efficient attention architectures have been proposed to improve transformer's efficiency while largely preserving its efficacy, especially in modeling long sequences. A widely-used benchmark to test these efficient methods' capability on long-range modeling is Long Range Arena (LRA). However, LRA only focuses on the standard bidirectional (or noncausal) self attention, and completely ignores cross attentions and unidirectional (or causal) attentions, which are equally important to downstream applications. Although designing cross and causal variants of an attention method is straightforward for vanilla attention, it is often challenging for efficient attentions with subquadratic time and memory complexity. In this paper, we propose Comprehensive Attention Benchmark (CAB) under a fine-grained attention taxonomy with four distinguishable attention patterns, namely, noncausal self, causal self, noncausal cross, and causal cross attentions. CAB collects seven real-world tasks from different research areas to evaluate efficient attentions under the four attention patterns. Among these tasks, CAB validates efficient attentions in eight backbone networks to show their generalization across neural architectures. We conduct exhaustive experiments to benchmark the performances of nine widely-used efficient attention architectures designed with different philosophies on CAB. Extensive experimental results also shed light on the fundamental problems of efficient attentions, such as efficiency length against vanilla attention, performance consistency across attention patterns, the benefit of attention mechanisms, and interpolation/extrapolation on long-context language modeling.

Under review as a conference paper at ICLR 2023
CAB: COMPREHENSIVE ATTENTION BENCHMARKING ON LONG SEQUENCE MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
Transformer has achieved remarkable success in language, image, and speech processing. Recently, various efficient attention architectures have been proposed to improve transformer's 's efficiency while largely preserving its efficacy, especially in modeling long sequences. A widely-used benchmark test these efficient meth- ods' capability on long-range modeling is Long Range Arena (LRA). However, LRA only focuses on the standard bidirectional (or noncausal) self attention, and completely ignores cross attentions and unidirectional (or causal) attentions, which are equally important to downstream applications. Although designing cross and causal variants of attention method is straightforward for vanilla attention, it is often challenging for efficient attentions with subquadratic time and memory complexity. In this paper, we propose Comprehensive Attention Benchmark (CAB) under a fine-grained attention taxonomy with four distinguishable attention patterns, namely, noncausal self, causal self, noncausal cross, and causal cross attentions. CAB collects seven real-world tasks from different research areas to evaluate effi- cient attentions under the four attention patterns. Among these tasks, CAB validates efficient attentions in eight backbone networks to show their generalization across neural architectures. We conduct exhaustive experiments to benchmark the perfor- mances of nine widely-used efficient attention architectures designed with different philosophies on CAB. Extensive experimental results also shed light on the funda- mental problems of efficient attentions, such as efficiency length against vanilla attention, performance consistency across attention patterns, the benefit of attention mechanisms, and interpolltion/eetaaapptttn on long-contex language modeling.
INTRODUCTION
Transformer has achieved great breakthroughs in natural language processing (Devlin al., 2019; Radford et al., 2019; Brown et al., 2020), computer vision (Dosovitskiy et al., 2020; Liu et al., 2021b), speech processing (Schneider et al., 2019; Ren et al., 2021), and biology (Jumper et al., 2021; Brandes et al., 2022). A major drawback of the transformer architecture is its quadratic complexity in both time and memory. The problem has been more evident with the ever-increasing need in applying transformers for longer sequence modeling in different domains.
Recent research on efficient attention mechanisms seeks to respond to this problem by improving attention efficiency while preserving efficacy (Wang et al., 2020; Kitaev al., 2019; Zaheer et al., 2020; Choromanski et al., 2020; Zheng et al., 2022). The commonly adopted test bed for benchmark- ing efficient transformers in the context of processing long sequences, is the Long Range Arena (Tay et al., 2020a; LRA), consisting of both synthetic probing tasks and real-world tasks. However, all of these tasks focus on the self attention setting, ignoring cross attention and causal attention, which are equally important and often more challenging. In other words, the transformer model is only used as a sequence encoder LRA, while in real applications, cross attention is essential for conditionality modeling tasks such as -oreence-----eeencee (Bahdanau et al., 2015), data-to-text (Dušek et al., 2020) and mnwweede-enhaanedd models (Liu et al., 202 and causal attention is critical for causality modeling tasks such as text generation (Vaswani et al., 2017; Zhang et al., 2020), language model- ing (Radford et al., 2019; Brown et al., 2020) and speech synthesis (Li et al., 2019). Another potential drawback of LRA as discovered by researchers recently, is that with proper tuning, the performance gap between different transformer variants in these tasks can be insignificant (Xiong et al., 2022; Ivgi al., 2022; Shaham et al., 2022), impairing its effectiveness standard benchmark.

---

Under review as a conference paper at ICLR 2023
To address these problems. propose comprehensive attention benchmark (CAB) for long se- quence modeling. Above all, we present a fine-graid aoeegraandd attention taxonomy, considering attentive functionality on conditionality and causality modeling. We present four patterns of attentions, namely, noncausal self, causal self, noncausal cross, and causal cross, representing the distinguishable atten- tive functionality to sequence modeling ($2). With that in mind, we then collect seven real-world tasks from diverse fields of computer vision, natural language processing, speech processing, and time series forecasting ($3). Among these tasks, CAB includes rich backbone architectures to evaluate the attention mechanisms, testing their performances and generalization abilities. Given four attention patterns defined by the attention taxonomy, we advocate a pattern- patternwiise comparison between attention mechanisms, evaluating their attentive functionality respectively.
We conduct exhaustive experiments on CAB, assessing nine widely-used efficient attentions designed with different philosophies (64). The experimental results reveal several insights into designing effi- cient attention architectures. First, we show that existing efficient transformers claiming comparable or even superior performances to vanilla attention, often achieve less competitive results in the causal cross scenario, indicating efficient attentions' modeling capability cannot always generalize across different attention patterns. Second, by quantifying the efficiency of attentions in long sequence con- texts using efficiency length (i.e., the minimum length for a quaddatiia efficient model to surpass vanilla attention in efficiency), attention methods in modeling relatively short sequences. Third, we investigate interpolation and extrapolation on a long-context language model, and find that it is promising for efficient attention, such as local attention and LongShort Transformer, to scale to long-context language modeling. We hope CAB and elaborated experimental efforts can shed light on the fundamental problems of efficient attentions, and inspire the design of advanced attention mechanisms. CAB and all the related codes will be released at https: / / g thub. com/ Anonymous.
disclose the underlying inefficiency problem of existing efficient
2 ATTENTION TAXONOMY
Attention methods are designed capture token-to-token dependency within and across sequences. Let Y (1h, V2...... Yn } and Y E Ig be target sequence and its feature matrix, and X {x1, P2.......... Fm and X Rmxd be a source sequence and its feature matrix, where and m are target and source sequence lengths respectively, and d denotes dimensionality. Note that X could be the same as in the case of self attention. The target and source feature matrices are transformed to Query-Key- Key-Value feature matrices as Q YWq. K = XWK, V = XWv with learnt parametric matrices Wq. WK. Wy. Vanilla transformer (Vaswani et al., 2017) learns to integrate key-value feature matrices K E m V E gmnd into a query one Q = {qi, . qn3 E gnnd with token-to-token attention:
Attn Q,K, V)- softmax (QK T /Va/ V
(1)
in which omit the multihead notation without loss of generality. Vanilla attention in Eq. advocates a token-to-token alignment matrix QK which results in quadratic complexity ((m) of computational time and memory usage. It is observed that Attn(-) in Eq. acts as an integration model, projecting query features Q by integrating key-value features without changing Q' 's shape.
Beyond the vanilla token-to-token attention mechanism, we extend Eq. a more general form of attention family as:
Attn Q, V) = f(Q: K, V): Rax + Raxd where Attn; (Can,,QQ K, V)- fi(q; Q.K, K, V),
(2)
(Q.K,
where Q treated as the main variable and K, V are conditional ones. Comparing with Eq. 1, Eq. covers efficient attention methods without explicitly modeling the token-to-token alignment. In usage, attentions are modified to fit generative models with underlying structures, such as conditional models and causal models, to achieve specific attentive functionality.
In this section, we put forward a fine-graing -ne-graided attention taxonomy, considering the conditionality and causality of attentions. We show their challenges and potential impact on real-world applications in modeling conditionality and causality. Under the taxonomy, we present four attention patterns with different attentive functionality shown in Figure 1.
2

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
(d)
Figure I: Computation diagrams for (a) noncausal self, (b) causal self, (c) noncausal cross, (d) causal cross attentions. Shaded blocks represent future tokens that are invisible to the current state. Blocks with red rims represent the current state token.
Self Attention & Cross Attention Self attention (Lin et al., 2017; Vaswani et al., 2017; Dosovitskiy et al., 2020) learns contextual features within a sequence whereas cross attention (Bahdanau et al., 2015) learns integrate features across sequences. Both self attention and eross attention highly effective in sequence learning, but designing cross attention is more challenging for efficient attentions without explicit token-to-token alignment. On the aspect of computation, Q IRa and K, V E mRx are of different shapes and are impossible to perform elementwise operations along length dimension, such cross-covariance used in XCIT (Ali al., 2021), 1). Besides, different from self attention, cross attention lacks implicit alignment between Q and K, V. such as neighborhood information. Cross attention learns to integrate conditional features K, V into the main features Q. and has been successfully applied to real-world applications, such as machine translation (Bahdanau et al., 2015; Vaswani et al., 2017), data-to-text generation (Dušek et al., 2020; Shen et al., 2020) and knowledge enhanced model (Jin et al., 2020; Liu et al., 2021a).
Noncausal Attention & Causal Attention While modeling a sequence, noncausal models have holistic view of the whole sequence whereas causal ones only access previous tokens up to now." This restriction forbids efficient attention like Nyströmformer (Xiong et al., 2021), which must be conditioned on the whole target sequence to achieve linearity, to be applied in causal models. Furthermore, when training causal model, efficient attentions such as RFA (Peng et al., 2021) and ABC (Peng et al., 2022) have to provide different sets of features from Y for each query token. This mechanism in these attentions, therefore, leads to multiplied memory consumption compared to their noncausal counterparts. We provide detailed analysis the increasing computation costs in Appendix B. In real-world applications, causal attentions bring up strong generation capability and is widely adopted in text generation (Bahdanau et al., 2015; Vaswani et al., 2017; Zhang et al., 2020), text-to-speech synthesis (Li et al., 2019) and language modeling (Radford al., 2019; Brown et al., 2020; Raffel et al., 2020; Chen et al., 2021a).
Attention Patterns As discussed above, conditionality and causality are two orthogonal as- pects of modeling sequences. Thus we obtain four attention patterns via Cartesian product as (noncausal, causal) (self, cross): noncausal self (NS), causal self (CS), noncausal cross (NC), and causal cross (CC). Following the general form of attentions in Eq. 2, we formulate the four attention patterns as below:
NS, K, V) & fig: Q,KK, 8.. Y =X
(3)
CS, K, V) & filg: Qu,, V 3.t =
(4)
NC, K, V) & fíqq: Q,K,V
(5)
CC K, V) - fi(g: V,
(6)
Figure illustrates the computation diagrams of these four patterns. These attention patterns construct typical sequence models with attention mechanisms. For instance, Transformer (Vaswani et al., 2017) adopt NS attention in encoder, and CS/CC ones in decoder; recently prevailing Autorertrerre Transformer (Gu et al., 2018) models adopt NS attention in the encoder, and NS/NC attention in the decoder. Previous efficient attention benchmark LRA only tests attentions with the NS pattern but
'hen generating a sequence Y = {yl, y2, ... autoregressively, neural models (Vaswani et al., 2017; Li Li al., 2019; Brown et al., 2020) usually adopt causal graph G(V, F)) with vertices V = {yi E Y} and edges F. {vi + y E [1, (1....,7,. n).j j [1,...,m] n), j).
3

---

Under review as a conference paper at ICLR 2023
Table 1: Task statistics of datasets, data lengths, evaluation metrics, backbone neural networks and required attention patterns. For encoder-decoder models, both source and target lengths are listed.
Noncausal Self
Causal Self
Noncausal Cross
Causal Cross
Task
Dataset
Length
Metrie
Model
559
FastSpeech 2
X
X
X
TTS
IJSpeech
100/141
MSD
Transfomer-TTS
V
X
V
Sum
Multi-News
4,096/400
ROUGE
Transformer
V
V
X
V
ETT
336/720
LSTF
ECT.
336/960
MSE MAE
Informer
X
V
X
Weather
720/720
PCC
PCN
1,536/3,584
Poinlr
X
X
F-Score
LM
8,192
GPT-2
X
X
X
MLM
PG-19
2,048
PPL
RoBERTa-base
V
X
X
X
SR
FFHQ He
16,384
PSNR SSMI
SR3
V
X
X
X
without the others, leading to an insufficient evaluation of attentive functionality. In the subsequent sections, we will introduce comprehensive attention benchmark CAB that evaluates attentions comprehensively on the four patterns.
3 COMPREHENSIVE ATTENTION BENCHMARK
We propose Comprehensive Attention Benchmark (CAB) that collects seven tasks covering four research fields, including computer vision, natural language processing, speech processing, and time series forecasting. We introduce these tasks with their datasets, data lengths, evaluation metrics, and backbone neural networks. Then we present a compositional index (CT) to combine individual task performance into compositional one for direct performance comparison across tasks and models.
3.1 LONG SEQUENCE MODELING TASKS
We collect seven real-world long-sequence modeling tasks in CAB with data lengths from 300 16,000 and include eight widely-used models as backbones in CAB to assess typical efficient attention mechanisms. Table summarizes the tasks' statistics, evaluation metrics, backbone neural networks and the required attention patterns ($2) are also checked.
Text-to-Speech Synthesis (TTS) This task requires models to convert input text sequences, de- scriptions, or narrations produced by a single speaker, to synthesized audio clips sharing the same timbre as the provided speaker. The CAB incorporates the LJSpeecch dataset (Ito, 2017) whose audio clips are sampled with 22,050 Hz. Under this set of relatively high sample rates, the average sequence length of processed audio clips is 559, We use non-autoregressive FastSpeech 2 (Ren et al., 2021) and autoregressive Transformer-TTT (Li et al., 2019) as backbone networks. We adopt Mean Cepstral Distortion (MCD) and Mel Spectral Distortion (MSD) for objective evaluation.
Summarization (Sum) This task is one that makes comprehensive summary of multiple input documents without losing important information. We consider multi-document summarization task and use Multi-News datasets (Fabbri et al., 2019) for long sequence modeling. To make the task more challenging, we set the maximum source text and target summary lengths to be 4,096 and 400, respectively. Transformer (Vaswani et al., 2017) implemented PARAGEN (Feng al., 2022) serves the evaluating backbone and ROUGE (R-N) (Lin, 2004) is used to evaluate each attention.
Long Sequence Time-series Forecasting (LSTF) Long sequence time-series forecasting is to
predict long-term future behavior based on past states. This task evaluates models on three datasets,
4

---

Under review as a conference paper at ICLR 2023
including Electricity Transformer Temperature (ETT), Electricity Consuming Load (ECL), and Weather (Zhou et al., 2021). Following (Zhou et al., 2021), we use Informer as backbone and conduct univariate and multivariate evaluations. We average their Mean Square Error (MSE) and Mean Absolute Error (MAE) to obtain the final scores.
Point Cloud Completion (PCC) This task is to complete a frame of point cloud data when providing partial points. CAB adopts PCN dataset (Griffiths & Boehm, 2019) which inputs 2,048 3D point coordinates and requires models to output the complete point cloud data comprising 14.336 points. PoinTr (Yu et al., 2021) is adopted here serving evaluating backbone. Chamfer Distance (CD) (Huang et al., 2020) and F-Score (Tatarchenko et al., 2019) are selected to evaluate each attention. For clearer comparison, CD metrics are enlarged by 1,000 times.
Language Modeling (LM) The language modeling task requires the language model to predict the next text based on the previous information. In this task, we consider long-context language modeling where the context length is prolonged to 8,192. We use PG-19 dataset (Rae et al., 2019) 1) and conduct evaluations based the backbone model GPT-2 (Radford et al., 2019) implemented on FAIRSEQ (Ott et al., 2019). Perplexity (PPL.) is used to serve the evaluation metric.
Masked Language Modeling (MLM) Different from the language modeling task, masked lan- guage modeling (Devlin et al., 2019) uses full context to predict masked tokens. Roberta-base (Liu et al., 2019) implemented on FAIRSEQ is adopted as the evaluation backbone. We use the same dataset and evaluation metric as LM task while the context length is set to 2,048.
Super-Resolution (SR) In this task, we aims to convert low-resolution (16 x 16) face images into high-resolution (128 128) images. Following Saharia et al. (2022), we train the backbone model SR3 on Flickr-Faces-HQ (FFHQ) dataset (Karras al., 2019) and conduct evaluation CelebA-HQ dataset (Karras al., 2018). Peak Signal-to-Noise Ratio (PSNR) and Structural SIMilar- ity (SSIM) (Wang al., 2004) selected to evaluate models from different aspects.
To test attentions under different patterns, we assign each pattern with tasks as suggested Table 1. We exclude LSTF and PCC tasks from noncausal self attention test for the limited contribution of attentions on them (see $4.2).
3.2 COMPOSITIONAL INDEXES
We compute a compositional index (CT) from seven tasks for performance comparison. CI is a normalized score balance the influence among evaluation metrics, and high CI represents excellence. It is computed as follows: a) we transform all evaluation metrics beforehand, so that a higher score indicates better performance; b) we then normalize each transformed metric with Z-score normalization; c) after normalization, the score of each evaluation metric is averaged within each task, and further averaged across tasks. We include details of CI calculation in Appendix C.
4 EXPERIMENTS
4.1 MAIN RESULTS
We conduct exhaustive experiments CAB by applying various efficient attentions to the backbone models and measure their performance under noncausal self, causal self, noncausal cross, and causal cross attention patterns. We extensively study the fundamental problems of efficient attentions, in terms of efficiency length against vanilla attention, performance consistency across attention patterns, long-context language modeling. The
the benefit of attention, and interpolationiextrapllttioo details of experimental settings for tasks are shown in Appendix D.
Compared Efficient Attentions We benchmark nine typical efficient attentions designed with different philosophies, including sparse attention (local attention (Luong al., 2015)), kemel- based linearized attention (Performer (Choromanski et al., 2020), cosFormer (Qin et al., 2021), LARA (Zheng al., 2022)), low-rank factorization (Nyströnformer (Xiong et al., 2021)), pro-
5

---

Under review as a conference paper at ICLR 2023
Table Efficient attentions with supported attention patterns. The complexity is based on sequence length n, omitting independent factors for simplicity. (*) denotes the attentions that perform differ- ently as the original paper claims. We clarify the reasons Appendix A.
Method
Attention Name
Complexity
Noncausal Self
Causal Self
Noncausal Cross
Causal Cross
Sparsity
local attention
O(n)
X
LARA
O(n)
X
Kernel
cosformer
O(n)
XX
X
Perforner
O(n)
X
Low-rank
Nyströmformer
((n)
X
X
Juctorization
ABC
O(n)
Prototype
ProbSparsc
O(nlogn)
X
X
Hybrid
LongShort
O(n)
X
X
State space
S4D
O(n)
X
X
totype (ABC² (Peng al., 2022), ProbSparse (Zhou et al., 2021)). hybrid attention (LongShort Transformer (Zhu et al., 2021)), and state space (S4D (Gu et al., 2022)) and evaluate these attention architectures in terms of different attention patterns described $2. We provide a brief overview of these attention architectures in Table 2 and leave detailed description in Appendix F.
The main results are shown in Table 3. observed that efficient attention achieves comparable performance with vanilla attention under noncausal self attention pattern, and some of them even surpass vanilla attention. It suggests that the current efficient attention indeed improves the ability to learn contextual features in long sequences of various modalities. Although the performance of efficient attention is remarkable in the noncausal self pattern, is less competitive in the other three patterns. As shown in Table 3d, causal cross attentions inferior to vanilla attention by significant margin. Due to the challenge of causality modeling, is notable that when compared with autoregressive models (i.e. Transformer and Transformer-TTI non-autoregressivee or parallel generation models (i.e. FastSpeech 2, SR3, and RoBERTa) are more likely to have their efficient variants without significant performance drops.
extremely hard design and
On the dimension of efficient attention, we find that local attention is a strongly competitive efficient model on most real-world tasks, which agrees with the previous studies (Xiong et al., 2022), although it is utterly defeated in LRA. It achieves the highest score of 0.978 under the noncausal self pattern and ranks second under the causal self pattern. In contrast, S4D, which achieves impressive results on LRA, appears less inspiring than expected on Sum and MLM tasks under the noncausal self attention pattern.
Efficiency Analysis We conduct a simulation efficiency experiment under noncausal self attention pattern. The experiment is performed on a single A100 GPU, where attention mechanisms are fed set of dummy sequences with lengths of (256, 512, 1024, 2048, 4096, 8192). Table 2 shows the time and memory consumption against sequence length by each attentions. We can find that the advantage of efficient attention gradually expands as the sequence length increases. Moreover, kernel- -based and low low-rank attentions are more efficient on time and space. Further, we report efficiency length to measure the utility of efficient attentions. The efficiency length is defined the intersection point of computational time and memory curves by efficient models and vanilla attention. The efficiency length represents the minimum length that a sub-quadratic efficient model surpasses vanilla attention in efficiency. Specifically, we perform regression on vanilla and efficient attentions by their theoretical complexity. Then we calculate the intersection of the regression line and the quadratic regression curve to get an efficiency length. Figure 2c shows that efficient attentions gain superiority over vanilla attention once the sequence contains thousands of tokens. For noncausal self attention, ABC and Performer have the shortest efficiency lengths on running time and memory usage, respectively. The details of efficiency lengths and experiment settings can be found Appendix G.
We implement and experiment ABC with its fast version as suggested by Peng al. (2022).

---

Under review as a conference paper at ICLR 2023
Table 3: Experimental results on CAB with (a) noncausal self, (b) causal self, (c) noncausal cross, and (d) causal cross attention pattern. (-) denotes that the method fails on the task. (f) denotes that the attention does not support fp16 training. The CI of each efficient attention shown in Appendix E.
TIS
Sum
SR
MLM
Method
FustSpeech 2
Transformer-TT ITS
Transformer
SR3
RoBERTa
CIT
MCDJ
MSD.
MCDJ
MSDJ
R-11
R-21
R-L1
PSNR
SSMI1
PPL,
vanilla
3.475
1.974
4.095
2.199
34.61
6.35
31.66
23.18
0.675
3.42
-0.024
local
3,419
1,970
4.015
2.164
38.50
10.54
35.39
23.33
0.682
4.18
0.978
cosFormer!
3.400
1.956
4.030
2.160
34.77
6.34
31.74
23.53
0.690
5.25
0.466
LongShort
3,436
1.996
3,913
2.136
34.35
6.41
31.55
23.28
0.681
3.38
0.269
LARA
3,463
2.012
4.116
2.209
34.03
31.23
23.35
-0.032
Performer
3,437
1.933
4.115
2.198
34.85
6.54
31.88
23.34
0.682
111.73
-0.285
Nystrünfommrr
3.557
2.036
4.274
2.276
34.45
6.30
31.56
23.20
0.679
6.32
-0.440
ProbSparse
3.363
1.946
4.034
2.161
34.62
6.36
31.64
22.98
0.667
186.35
-0.661
ABC
3.393
1.966
4.085
2.204
33.80
6.07
30.98
22.54
0.635
10.50
-0.686
S4D
3.303
1.905
4.017
2.195
23.35
0.682
25.34
(a)
ITS
Sum
LM
Method
Transforner
TTS
Transformer
GPT-2
CTT
MCDJ
MSD]
R-11
R-21
R-L
PPLL
vanilla
4.095
2.198
34.61
6.35
31.66
S4D
4.030
2.188
34,90
6.65
31.98
15.78
0.985
ShogShort
4.039
2.195
33.55
6.27
30.71
15.52
0.617
local
4.141
2.220
33.50
6,27
30.74
19.73
-0.239
ABC
4.058
2.189
30.17
5.48
27.92
29.13
-0.623
(b)
PCC
LSTF
PoinTr
Informer
Method
ETT
Weathrr
ECL
CIt
CD-f3
CD-fal
F-Scoref
MSEJ
MAEJ
MSEJ
MAEJ
MSEJ
MAEJ
vanilla
7.425
0.231
0.779
1.138
0.775
0.478
0.508
0.449
0.486
0.596
ABC
7.344
0.227
0.784
1.147
0.809
0.489
0.520
0.519
0.527
0.204
Performer
7.368
0.235
0.783
1.254
0.819
0.463
0.508
0.881
0.722
0.014
cosFormer
8.673
0.298
0.704
1.219
0.474
0.509
0.632
-0.815
TTS
Sum Transformer
Method
Transformer-TTI
CI
MCDJ
MSDJ
R-11
R-27
R-L1
vanilla
4.095
2.198
34.61
31.66
0.956
ABC
5.780
2.631
32.22
5.55
29.53
0.058
Performer
6.635
3.053
27.22
3.88
25.21
-1.014
(d)
4.2 DISCUSSIONS
Performance Consistency Across Attention Patterns We investigate whether efficient attentions perform consistently across different attention patterns. To achieve this, we compute Pearson correlation (Benesty et al., 2009) over CI between each pair of attention patterns against the same arrangement of efficient attentions for berra-bbnchark comparison. Note that we also include CI computed from LRA for inter-benchmark comparison between CAB's 's four attention patterns and LRA. In Figure 3a, CAB's NS attention pattern is highly correlated with LRA, reflecting that efficient NS
TThe order is [vanilla, local, ABC, LARA, cosFormer, ProhSparse, Performer, LongShort, Nyströmformer,
S4D]
7

---

Under review as a conference paper at ICLR 2023
40
vesill Portonnar
vadia
Ruaning Time (Kellatvv)
Memory Usager (Rcclative)
30
LongShon
LengShett
PrebSpaase
20
Former
SAD
Nytrömfoemer
40
10
ABC
20
LARA
Perfemer
250
512
1324
2008
4096
8192
1090
2000
3000
4030
5630
6693
Sequence Length
Sequence Length
Efficiency Length
(a)
(b)
Figure 2: Empirical running time (a) and memory (b) with sequence length. Relative measure- ments to vanilla attention are reported. (c) efficiency length of attention architectures compared to vanilla attention. The efficient attention order on the y-axis is monotonically sorted by efficiency length on memeory usage.
TTS
vanilla ottention altention
100
LongSbamt
ABC
0.5
Perplexity
N8-
CS-
0.0
NC
PCC
LSTF
CC
1024 2088 2955 aRR 1258819384 Context Length
(b)
(c)
Figure 3: (a) Pairwise attention pattern correlation where NS, CS, NC, and CC signify noncausal self, causal self, noncausal cross, and causal cross attention respectively; (b) Ablation study of removing attention. The gray part is the score achieved by the existing efficient attention family, where we select the most well-performed efficient attention for each task; (c) the performance of efficient attentions based on different context lengths in the I.M task during the test phase. S4D fails on the context with 16,384 tokens.
attentions developed on LRA do contribute to wide range of real-world applications. In contrast, cross attention patterns share non-positive correlations with self attention patterns, indicating their inconsistent performance on self and cross attention.* On the causality dimension, the performances of causal and noncausal models are highly related but not strictly consistent. We also include task-to-task correlation analysis in Appendix
Benefit of Attention Self attention is designed to capture contextual features while cross attention integrates non-homologous information. But how much do these features benefit the model? We conduct ablation study without the attention mechanism. In preliminary experiment, causal and cross patterns are indispensable to the backbone models as expected, while noncausal self attention only has limited contribution. Therefore, of noncausal self attention. Figure 3b depicts the CI of backbone models with and without attention on tasks. Detailed CI be found in Appendix H. Results show that attention mechanism improves performance on most tasks. Embarrassingly. after removing the attention mechanism, the PoinTr and Informer achieve improvement on PCC and LSTF tasks, respectively. A possible reason that there alternative contextualizing structures such as the convolutional neural network in the backbone models and the contribution of the attention mechanism weakened. Besides, we also notice that current efficient attention family has already surpassed vanilla attention in modeling noncausal self relationships on most tasks.
find that
focus on discussing the attentive functionality
*The negative correlation here is biased due to limited exploration of causal and efficient attentions.
8

---

Under review as a conference paper at ICLR 2023
Interpolation and Extrapolation Long-Context Language Modeling Language models with efficient attentions are more likely to face unexpected long context that exceeds predefined lengths, resulting in an extrapolation problem. It is critical for efficient attentions to scale to longer sequences while preserving the performance on short ones. Therefore, we conduct an experiment on the language modeling task, where we train the model with the input of 8,192 tokens and calculate the perplexity of contexts with 256-16, 384 tokens during the test. Figure 3c shows the results on both of interpolation and extrapolation. On one hand, as the size of context grows to 8,192, language models achieve decreasing perplexity. It means that longer contexts indeed improve language modeling, and efficient attentions do well in interpolation as expected. On the other hand, for sequences with more than 8,192 tokens, the perplexities of all these efficient attention -eperpppd- language models become higher. Notably, local attention and LongShort with local features have only slight drop in performance in extrapolation, while ABC with only global features fails to extrapolate to longer sequences. Although S4D performs well as causal attention, it has a large perplexity with sequence input longer than 8, 192, indicating that S4D has relatively poor extrapolation ability.
RELATED WORK
Long Sequence Modeling Most efficient attention architectures are designed for long sequence modeling. Previous works evaluate their proposed attention modules separately on Language Model (Choromanski al., 2020; Qin et al., 2021; Peng et al., 2022), Masked Language Model (Qin et al., 2021; Peng et al., 2022), Image Classification (Choromanski et al., 2020; Zheng et al., 2022), and Machine Translation (Peng et al., 2022; 2021). Without unified criteria for evaluating attention modules, researchers find it hard to distinguish between them and select an appropriate attention module for their research. Later, Tay et al. (2020a) proposed Long Range Arena (LRA), a benchmark that contains 5 different classification tasks across language and image domains their bench- mark only considers the noncausal self attention pattern and contains most synthetic tasks which are far from rerl-world applications. Besides, different attention modules perform nearly equally on LRA (Xiong et al., 2022), which causes LRA a limited benchmark. Shaham et al. (2022) proposed SCROLLS for long language sequences but ignores long sequence modeling tasks in other fields.
Efficient Attention The emergence and prevalence of Transformer models (Vaswani et al., 2017) as well as its strong capability for encoding and generation have imposed impressiveness on various applications. To improve the efficiency of attention in Transformer models, previous work explored efficient attention mechanisms, including sparse attention (Guo et al., 201 9a; Ainslie et al., 2020;
Beltagy et al., 2020), low-rank attention (Guo al., 2019b; Chen et al., 2020; Xiong al., 2021), kernel-based linearized attention (Choromanski et al., 2020; Katharopoulos et al., 2020), prototype attention (Vyas et al., 2020; Zhou al., 2021), prior attention (Zhang al., 2018; Ying al., 2021; Tay et al., 2021), and memory compress (Liu* et al., 2018; Lee et al., 2019; Wang et al., 2020). For interested readers, we refer them to (Tay al., 2020b) and (Lin al., 2021) for more detailed description of efficient attention.
6 CONCLUSION
In this paper, propose to evaluate efficient attentions under more fine-grained attention taxonomy with four distinguishable attention patterns, each of which reflects different attentive functionality. Under the taxonomy, we present Comprehensive Attention Benchmark (CAB), consisting of seven real-world tasks and eight backbone networks. We conduct exhaustive experiments to benchmark nine typical efficient attentions on CAB. The empirical results show that existing efficient attentions make significant progress in noncausal self attention, but are still struggling to catch up with vanilla attention in conditionality and causality modeling. The efficiency analysis shows that although existing efficient attentions enjoy lower computational complexity, most of them only start to achieve significant efficiency gain when applied to sequences with more than one thousand tokens, especially in the causal attention scenario. Moreover, we find that attention mechanisms, in the noncausal self case, are sometimes not helpful or even harmful to neural models that already contain feature mixing. Our study sheds light on long-contex: language modeling and shows that efficient attentions such as local attention and LongShort Transformer are promising for extrapolation problems in long-context language modeling.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek. Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. ETC: Encoding long and struc- tured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 268-284, Online, November 2020, Associ- ation for Computational Linguisties. doi: 10. 11113//1120222mmmmpmmm //aclanthology' org/ 2020. emnlp 1alp-mmiin main.
URL https:
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34:20014-20027,, 2021.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015. URL httt:::/aaaw...og////aaaaa
0473.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.055 -55500 2020.
Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. In Noise reduction in speech processing, pp. 1-4. Springer, 2009.
Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. Proteinbert: A universal deep-learning model of protein sequence and function. Bioinformatics, 38(8):2102-2100, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https: /ttp::://ppcceeeeeee http://pcoceeeig..cccc nwwrips....pp... /cc/ cccpaperr/// httpppppr/2///111222 (aeeleee1155000066 ccb 967 41 Bbfb8ac1 Paa.er..aappp..ott..
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021a.
Yifan Chen, Qi Zeng, Heng Ji, and Yun Yang. Skyformer: Remodel self-attention with gaussian kernel and nystr' "om method. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 2122-2135, Curran Associates, Inc., 2021b. URL htttp:://ppocceeiiiiiiii nenggs.neuris... cc/ paper/20211 htt///ilee//a/ wtt.aaeeee1////001010o DECC Papeeee.--apppppp...e.
Ziye Chen, Mingming Gong, Lingjuan Ge, and Bo Du. Compressed self-attention for deep metric learning with low-rank approximation. In Christian Bessiere (ed.), Proceedings of the Twenty-Ninth International Joins Conference on Artificial Intelligence, IJCAI-20, pp. 2058 2064. International Joint Conferences on Artificial Intelligence Organization, 7 2020. doi: 20224963/jcca..22002255. URL htap::://dooiooo /00.00000011 .29.2446666 249924493344 ijcai. 20200222 Main track.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreca Gane, Tamas Sarlos, Peter Hawkins, Jared Quiney Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4188 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19- 1423. URL https: //acl hn ogy. 111199-35 9-19--144 1423..
10

---

Under review as a conference paper at ICLR 2023
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. Evaluating the state-of-the-art of end-to- end natural language generation: The e2e nlg challenge. Computer Speech & Language, 59:123-156. 2020.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1074-1084, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 1/11865/11/11911111 102. URL https: / aaaaacaaaa www.nooootooo....ooooo wtg/pog--100 1102.
Jiangtao Feng, Zhou, Jun Zhang, Xian Qian, Liwei Wu, Zhexi Zhang, Yanming Liu, Mingx- uan Wang. Lei Li, and Hao Zhou. Paragen: A parallel generation toolkit. arXiv preprint arXiv:2210.03400. 2022.
David Griffiths and Jan Boehm. A review on deep learning techniques for 3d sensed data classification. Remote Sensing, 11(12): 2019.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in Neural Information Processing Systems, 33: 1474-1487, 2020.
Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models. arXiv preprint arXiv:2206.1 11022206.1199/. 1193, 2022.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. Non-autoregressive neural machine translation. In International Conference on Learning Representations, 2018.
Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. Star- transformer. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 1325, Minneapolis, Minnesota, June 2019a. Association for Computational Linguistics. doi: 10.18653/1/N19-11133 httpss::: http:///caaaattttaaoaee wwwptehhooooooooo/......oo org/Ni 1133.
Qipeng Guo, Xipeng Qiu, Xiangyang Xue, and Zheng Zhang. Low-rank and locality constrained self-attention for sequence modeling. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(12):2213-222 2019b. doi: N0.1109/AASLP..09922229
Zitian Huang, Yikuan Yu, Jiawen Xu, Feng Ni, and Xinyi Le. Pf-net: Point fractal network for 3d point cloud completion. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 7662-7670, 2020.
Keith Ito. The speech dataset. https: /
2017.
Maor Ivgi, Uri Shaham, and Jonathan Berant. Efficient long-text understanding with short-text models. arXiv preprint (tXXi:2208.00745 2022.
Hanqi Jin, Tianming Wang, and Xiaojun Wan. Semsum: Semantic dependency guided neural abstractive summarization. Proceedings of the AAAI Conference Artificial Intelligence, 34 (05):8026-8033 Apr. 2020. doi: 10. 10.1609%aaiivv34000 v34005..334 6312. URL https:////// org/ index, fodex.phpp wttp://pppAAAAAaaateaaaaa
6312.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunnyauuunakool,, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(78733:583-588)
2021.
Tero Karras, Timo Aila. Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
11

---

Under review as a conference paper at ICLR 2023
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition (CVPR), June 2019.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5156-5165. PMLR, 13-18 Jul 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2019.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based jermutation-invariant neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3744- 3753. PMLR, 09-15 Jun 2019. URL https: http///pecceeg...e/ee//////sssse html.
wwainngsmmmm.. mlr. wwwppppwsssseeeeeeeppeeppp 11//////e//ll
Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01): 6706-6713, Jul. 2019. doi: (0.16009aaai.v33300) v33..33331111176666 URL hwtph:tt////eectc /ttp:///jjss.o org/ index.phpt AAAI httppppptttt/ttaleeeeee//eeaeeee
Chin Lin. Rouge: A package for automatic evaluation of summaries. Text summarization branches out, pp. 74 81, 2004.
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. arXiv preprint ar arxiv:2106.... 04554,, 2021.
Zhouhan Lin, Minwei Feng. Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A STRUCTURED SELF-ATTENTIVE SENTENCE EMBEDDING. In International Conference Learning Representations, 2017. URL htttps://operevii/ net / (otoorummiddJ_______
Peter J. Liu*, Mohammad Saleh*, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. In International Confer- ence on Learning Representations, 2018. URL https:tp////.cc.. /ttps///eep/ppeeeeeeemm wwtp://nntfootett//eeeemmm ffr.mmidd- HygOvbWC-
Ye Liu, Yao Wan, Lifang He, Hao Peng, and Philip S. Yu. Kg-bart: Knowledge graph-augmented bart for generative commonsense reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(7):6418-647) May 2021a. doi: 10.1669/aaai.x355 v35i......37..17.999 URL https: //ojs. aaai. org/ index. php/ wAt./articee/veee/////aa.aaattat
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021b.
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention- based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412-1421, Lisbon, Portugal, September 2015, Association for Computational Linguistics. doi: 10. 10.18653/v1/115-11166 URL https: //acl antho oggy
org/D15--116
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)) pp. 48-53, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 36653/v1/N1994//0 URL CtI.tttss:/ //acl antholooyy...ooooo .orggy..orrooo org/N1/ 9- - 4009.
12

---

Under review as a conference paper at ICLR 2023
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. In International Conference on Learning Representations, 2021. URL
https: httpss//pppeeeveeeeeeee wwtp:/iwww..eeeeee..em..e...ee nnt/
Hao Peng, Jungo Kasai, Nikolaos Pappas, Dani Yogatama, Zhaofeng Wu, Lingpeng Kong, Roy Schwartz, and Noah A. Smith. ABC: Attention with bounded-memenn control. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 7469-7483, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 1226665//1/22222aallloooog 515.
htthttps:: http::sss///aaaaahhhooooooo
Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. In International Conference Learning Representations, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with unified text-to-text transformer. Mach. Learn. Res., 21(140):1-67, 2020.
M Kafaei Razavi, Asghar Kerayechian, Mortaza Gachpazan, and Stanford Shateyi. A new iterative method for finding approximate inverses of complex matrices. In Abstract and Applied Analysis, volume 2014. Hindawi, 2014.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations, 2021.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. In INTERSPEECH, 2019.
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. Scrolls: Standardized comparison over long language sequences. arXiv preprint arXiv:2201 005533033330000 2022.
Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min. Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, and Lawrence Carin. Baseline needs more love: On simple word-embedding- based models and associated pooling mechanisms. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 440-450, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 18.18653/v 1/P18- 1041. URL https: wwwaaaaannnnooooooooooo
/P1/ 1041.
Xiaoyu Shen, Ernie Chang, Hui Su, Cheng Niu, and Dietrich Klakow. Neural data-to-tex generation via jointly learning the segmentation and correspondence. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 7165- Online, July 2020. Association for Computational Linguistics. doi: 10. man.. URL https: org/ 2200.00000 secl-mainn 641.
Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. What do single-view 3d reconstruction networks learn? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3405-3414, 2019.
13

---

Under review as a conference paper at ICLR 2023
Yi Tay, Mostafa Dehghami, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: benchmark for efficient transformers. In International Conference Learning Representations, 2020a.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys (CSUR), 2020b.
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention for transformer models. In International conference on machine learning, pp. 10183-10192. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, tukasz Kaiser, and Iliia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Apoory Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. Advances in Neural Information Processing Systems, 116716466-26 2020.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04770 2020.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 1344)6600-612 2004.
Wenhan Xiong, Barlas Oguz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Scott Yih, and Yashar Mehdad. Simple local attentions remain competitive for long-context tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Asvociation for Computational Linguistics: Human Language Technologies. pp. 1975-1986, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. (0.11853// 1t/p222.2a2aaa/./ main. 144. URL https: http::s:///llaaahoooossso orgg 2022 nwacl-ma 144.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: nyström-based algorithm for approximating self-attention. Pro- ceedings of the AAAI Conference on Artificial Intelligence, (1116)14138-14148 May 2021. doi: 200009//aa.v35511 17664. URL https://ojss aaai. org/ index.dddxxe/// php/ AAAI/ article/view/17666
Chengxuan Ying, Guolin Ke, Di He, and Tie-Yan Liu. Lazyformer: Self attention with lazy update. arXiv preprint 11XX::21022127022 2021.
Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geor transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pp. 12498-12507, October 2021.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 1778333-17297, 2020.
Biao Zhang, Deyi Xiong, and Jinsong Su. Accelerating neural transformer via an average attention network. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1789-1798, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 12.18855/11/11-1111 org/P18-1166.
URL https: peattss//aatt //acl anthology.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. International Conference on Machine Learning, pp. 11328-11339, PMLR, 2020.
Lin Zheng, Chong Wang, and Lingpeng Kong. Linear complexity randomized self-attention mecha- nism. arXiv preprint arXiv:2204.04661 2022.
14

---

Under review as a conference paper at ICLR 2023
Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of the AAAI Conference Artificial Intelligence, (5(12):11106-1111) May 2021. UtLhttps:
//ojs. aaaj..aaai.... org/ hegg/inder index. php/ /
Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-shor transformer: Efficient transformers for language and vision. Advances in Neural Information Processing Systems, 34:17733-17766, 2021.
15

---

Under review as a conference paper at ICLR 2023
A EFFICIENT ATTENTIONS WITH PERFORMANCE DIFFERENCES
cosFormer requires cos-reweigtting mechanism to behave like vanilla softmax attention. How- ever, its reweighting mechanism needs the maximum length between the source and target sequences in Eq. 16. Under causal self pattern, this operation requires different L values for each token of Q and K, which ignored its released codebase repository". Therefore, the released code cannot model causality.
Performer uses random feature matrices to project Q and However, before transforming Q, K with the softmax kernel, its released codebase" uses the golden target sequence length to compute random feature maps under causal self pattern. However, this operation brings inconsistency between training and inference, because the golden target length is unavailable during inference. Thus, cannot use the released code to perform causal self tasks.
B INCREASING COMPUTATION COSTS FOR CAUSAL EFFICIENT ATTENTION
In this section, we use an efficient attention example RFA (Peng et al., 2021) to illustrate enforcing causality might bring increase in computation for efficient attention. In RFA, the noncausal version models the query q: as:
((q:) Σ, o(k;) @ us
RFA (b,),
( E, (kk)
<(q:) St
(8)
(qq)) Zz
where S, E R22xd and 2, E Rd and 2D is the dimensionality of kernel d. In its causal version, the hidden state S, and z are recurrently modified to maintain the causality:
- = + ((ks) X 2 Ze Z + ((ks)
(9)
We need to store S, and z, explicitly during training, which contains the historical information specific for each query As result, these recurrent operations add additional O(ndD) time and memory consumption.
C COMPOSITIONAL INDEX
We define st the of i-th attention model k-th metric in the j-th task, where 1,..., K and j = 1, I....,,.. M. To obtain Compositional index (CI), we first normalize s For the metric whose value is higher indicating high performance, st is normalized to sty by sty * of and of denote the mean and standard deviation of the models' scores on this metric. Otherwise, normalize sty by b - for the metric whose value lower indicating high performance. Then we average the normalized score of each metric to obtain a task score Ch further converted the final score CT₁ ΣM CTa
- where n
ΣK t4 which
M
D HYPERPARAMETERS
Hyperparameters for all tasks are shown in Table We also report the hyperparameters of efficient attentions in Table 5.
'ttpps: com
http:///hoooommma/ccoooomeee www.err.re.r...... py 06 Phttps://ge thub. com /ogpe//////oeeeeeeeeere///ogg/// reseeeseerrrrreoooooeeeeerrrrrr /bl obloasteerr
16

---

Under review as a conference paper at ICLR 2023
Table 4: Hyperparameters for the tasks in CAB.
Sumn
MLM
FFHQ
Data
LJSpeech
ETT
ECL
Weather
PCN
Multi-News
PG-19
CelebA HQ
Batch Number of Steps
20K
(epochs)
300 (epochs)
50K
125K
125K
IM
Warmup Steps
4K
4K
Peak Learning Rate
5e-4
le-4
5e-4
5e-4
5e-4
3e-4
le-4
Scheduler
Inverse
Sqrt
Exponential Decay
LamdaLR
Sqrt AdamW (0.9. 0.98)
Inverse
Inverse
Sqrt
Polynomial
Docay
Lincar
Optimizer
Adam'W (0.9. 0.98)
AdamW (0.9.0 0,999)
Adam (0.9, 0.98)
Adam (0.9, 0.999)
Adam (0.9. 0.98)
Adam
Adam
(0.9. 0,999)
Clip Norm
5.0
0.0
0.0
0.0
0.0
Attention Dropout
0.1
0.05
0.0
0.1
0.1
0.1
0.2
Weight Decay
0.01
Se
0.01
0.01
Tokens per
2
2
Iteration
Table 5: Hyperparameters for various efficient attentions under noncausal self attention pattern (a), causal self attention pattern (b), noncausal cross attention pattern (c), and causal cross attention pattern (d). (*) We find that the performance of LongShort with num aum_landmarks = 64 is significantly lower than its performance with num landmarks 16 Sum task under the noncausal self attention pattern. So we finally use LongShort with num landmarks 16.
Method
Hyperparameters
TTS
Sum
SR
MLM
Method
Hyperparameters
TTS
Sum
LM
local
local
wsize
15
15
LongShort
Jandmarks
64
ABC
num landmarks
64
64
16
16
wsize
LARA
num landmarks
64
16
16
ABC
nom landmarks
64
(b)
ProbSparse
factor
5
5
5
Performer
approx attn dim
64
64
16
16
Method
Hy Hyperparameters
PCC
LSTF
wsize
15
15
15
15
ABC
num landmarks
64
16
LongShort
num landmankss
64
16*
16
16
Performer
approx_attn.diin dim
64
16
kernel size
cosFormer
num landmarks
64
64
16
16
Nyströmformer
kernel.sizee
Method
Hyperparameters
TTS
Sum
S4D
d_state
64
64
16
16
ABC
num landmarks
64
cosFormer
Performer
approx_attn,ding alin.dim
64
(a)
E TASK SCORE FOR COMPARISON
We show the compositional index of each task in Table 6. For new efficient attention, can use our provided means and variances to obtain the normalized score. The means and standard deviations of metrics are shown in the Table Table 8, Table 9, and Table 10.
DETAILED DESCRIPTION OF EFFICIENT ARCHITECTURES
We here only present the noncausal self forms of these efficient architectures. the subsequent discussion, we denote n and m as the target and souree sequence lengths, respectively. Also, we omit the factor 1/ vd for simplicity.
Local Attention Local attention (Luong et al., 2015) intuitively forces each query token to attend to its neighborhood tokens within fixed window size reduce the computation costs. In the full attention map, the neighborhood tokens contribute a large proportion of the attention score for the querying token (Qin al., 2021). Therefore, local attention intuitively selects (w/2 for causal
17

---

Under review as a conference paper at ICLR 2023
Table 6: Compositional index.
NS
CS
NC
CC
Method
TTS
Sum
SR
MLM
TTS
Sum
LM
PCC
LSTF
TTS
Sum
vanilla
-0.301
-0.246
-0.077
0,527
-0.047
0.784
0,449
0.744
1.047
0.867
local
0.362
2.617
0,421
0.515
-1.361
0.337
0,305
ABC
0.043
-0.678
-2.525
0.414
0.707
-1.461
-1.117
0.573
-0.164
-0.112
0.228
LARA
-0.639
-0.522
0.554
cosFormer
0.516
-0.190
1.042
0.498
-1,496
-0.135
ProbSparse
0.705
-0.246
-0.697
-2.408
Performer
-0,282
-0,088
0,439
-1.211
0.473
-0.445
-0.935
-1,094
LongShort
0.572
-0.322
0.298
0,528
0.701
0.340
0.812
Nyströmformer
-2.011
-0.321
0.088
0,481
S4D
1.035
0.457
0.176
1.030
1.143
0.780
self attention pattern) tokens around querying token to serve K, and V.- Its formulation is as follows:
Local(g: K.,. V = softmax(q:
(10)
where K Var wRxd denotes the neighborhood tokens within w size around the query token qi in K and V. The complexity is of linear complexity (nnwd) when is predefined and independent
of sequence length n. Due to the locality inductive bias, local attention competent only on causal self and noncausal self attention patterns.
Nyströmformer Nyströmformer (Xiong et al., 2021) adopts the Nyström method to approximate the softmax attention matrix. It selects r landmarks for Q and K with Segment-means (Shen et al., 2018) and produces and K. Using iterative Moore-Penrose pseudoinverse (Razavi al., 2014), Nyströformer decomposes the softmax attention matrix into three sub-softmax matrices as follows:
Nyströmformer Q K, V) softmax QK (softmmax QK )) softmax(QK V
(11)
where (softmax T)+ is the Moore-Penrose pseudoinverse of softmax QK ). The complexity Moore-Penrose pseudoinverse is linear with respect to n when its iteration time constant. Therefore, the overall computation cost is O(nrd) when is predefined. Because it needs to select the same number of landmarks for Q and K and requires the full Q representation, it only adaptable in the noncausal self attention pattern.
Performer Performer (Choromanski et al., 2020) another linear transformer using random feature map o(.) Rd - R., (for some > 0) as a kernel function to reduce computational complexity. Formally, the random feature map is defined to approximate the exponential kernel as
exp(q, k,) = E, ((:)) (k2)),
(12)
where (xx) ex, | (1,1) E R"' and all elements of W, are independently drawn from standard Gaussian q(w) = N (w: 0, 1). Given the defined random feature map, the whole softmax attention for each query can then be approximated as
m exp(g, k,
o(qi) ΣT= (kz)), Performer(g (q, K, V) ((g)) Σラ [kg)
(13)
-y= Ex=eepp(q kT)
Thanks to the kernel approximation, Performer achieves ((rd) complexity in both noncausal self and noncausal cross attention patterns; however, for causal self and causal cross attentions. Performer has to store the cumulative summation for both Σ o )k and > o(kz), which requires dedicated CUDA operator to accelerate the computation.
LARA LARA (Zheng et al., 2022) further extends the random feature approximation from estimat- ing individual exponential kernels to directly estimating the whole softmax function. In particular, there exists distribution p that instantiates the random feature (Zheng et al., 2022) such as:
exp(q, k
m
((g)) T (TTT(k)) (k))
E,
(14)
1 exp(q, k))
3 Φ(g)) (k(())
18

---

Under review as a conference paper at ICLR 2023
Table 7: The means and standard deviations for noncausal self attention pattern evaluation.
TTS
Sum
SR
MLM
Method
FastSpoech 2
Transformer- TT-
Transformer
SR3
RoBERTa
MCD
MSD
MCD
MSD
R-1
R-2
R-L
PSNR
SSMI
PPL
μ
3.425
1.974
4.069
2.190
34.89
6.79
31.96
23.21 0,274
0.676 0.015
36.292
a
0.068
0.036
0.094
0.038
1.396
1.410
1.314
62.308
Table 8: The means and standard deviations for causal self attention pattern evaluation.
TTS
Sum
LM
Method
Transformer- TTS
Transformer
GPT-2
MCD
MSD
R-1
R-2
R-L
PPL
4,083
2.200
32.96
6.09
30.26
22.26
0.045
0.013
1.927
0.410
1.619
8.299
By viewing softmax attention as an expectation over a potentially complicated distribution p, Zheng et al. (2022) noted that Performer can be understood performing self-normalized importance sampling to approximate this expectation, with standard Gaussian as the proposal distribution. LARA generalizes this view by adopting multiple adaptive proposal distributions to further improve the approximation fidelity. shown that LARA takes the following form:
((gx)
A, A, プ (k))o) ((qm) A, -k
LARA (q K, V)
(15)
where A, is a query-specific diagonal matrix that ensures the computation overall still yields valid self-normalized importance sampling estimation: and the elements of sampled W, in Φ(x) exp(W,x || 1,) are allowed to be drawn from different distributions beyond standard Gaussian.
cosFormer Following the same method defined in Eq. 13, cosFormer (Qin et al., 2021) adopts another kernel function ReLU(.)) instead of random feature map. Due to the importance of the softmax operator that can ensure the non-negativeness of value weights and stabilize the training process, cosFormer designed new cos-based reweighting mechanism similar softmax. Denote qf Re: and kj = (kk), the re- re-weight formula is expressed as follows:
- x i i-j
s(g, kj) = q: k coss(
2 EL'
(16)
where L = max(n, m). Using this formula and Ptolemy's theorem, the final expression of cosFormer is as follows:
f== 9: cns (k UD) E Tit ((kain)Teg TTv))
cosformer K, V)
(17)
Th 2j=1 cos)
koon
> (q.in) ksin
where gf00 cos( Ti 2T, ), g. q sin( 2T, Ag00 ky cos( Tt ), Aajnn ky sin (E). Similar to Performer, can efficiently fulfill noncausal self and noncausal cross attentions with linear complexity O(nd?), but its causal attention relies on specialized CUDA operator achieve high efficiency.
LongShort Transformer LongShort Transformer (Zhu et al., 2021) models the long-range and short-term dependency of sequence with two different methods discussed above. For long-range modeling, it utilizes the low-rank property of the softmax function to convert key and value to more compact representation using a learnable matrix W E Rdxr, K = P K E Rexd and V E xd where = softmax (KW and r the compressed dimension. For short- term modeling, it borrows local attention output token representation fused with neighborhood features via segment-wise wise local attention. Let / be the segment length and s; = [i/l] be the segment id for i-th token, segment-wise local attention produces K., K (e.-11+++ (: (+.++1)1++
19

---

Under review as a conference paper at ICLR 2023
Table 9: The means and standard deviations for noncausal cross attention pattern evaluation.
PCC
LSTF
Method
Informer
PoinTr
ETT
Weather
ECL
CD-&
CD-E2
F-score
MSE
MAE
MSE
MAE
MSE
MAE
7.702
0.247
0.762
1,189
0.806
0.476
0.511
0.620 0.189
0.581
0.647
0.033
0.039
0.056
0.021
0.010
0.006
0.103
Table 10: The means and standard deviations for causal cross attention pattern evaluation.
TTS
Sum
Method
Transformer-TTS
Transformer
MCD
MSD
R-1
R-2
R-L
5,503
2.627
31.35
5,26
28.80
1.292
0.427
3.771
1,260
3.286
and V v (o Combined with these two methods, the output of LongShort Transformer is formalized as follows:
LongShort(44 q; K, V) softmax K.: K V V
(18)
where [ is concatenation operator at the first dimension. The complexity O + nwdd is also linear w.rf. the target length n. Due to its introduction of local attention, LongShort Transformer can only complete tasks under causal self and noncausal self attention patterns.
ProbSparse Attention ProbSparse (Zhou al., 2021) calculates attention scores for few (Q, K) dot-product pairs based on sparsity of self attention scores. Specifically, ProbSparse selects some "important" queries whose attention distributions on keys non-uniform. Then only need to calculate the attention scores between the "important" queries and all keys and output the weighted sum of V. For other trivial queries, ProbSparse takes the mean of V outputs.
To distinguish the "important" queries, ProbSparse first samples keys K, and then measures the sparsity of query gi by M (q: K) as follows:
q,k_
LR
q
M (q, K) max
(19)
Vd
LK
Vd
where LK denotes the number of keys. Finally, the Top-u queries with large M (q, K) are the "important" queries.
ABC
In this paper, we use ABC to denote ABCMLP (Peng et al., 2022). ABCMLP bounds the amount of memory slot for K and V to be r < m, independent of both and m. ABCMLP uses leamable matrix to fill each memory slot with reweighted key and value representations:
K
softmax (W K K,
softmax (WV V
where W W/ grxd a learnable matrix representing the bounded memory. Leveraging the & bounded representation K and V, ABCMLP executes following softmax attention:
ABC K. V) softmax(g.K*
(20)
ABCMLP is again of linear complexity ((nrd).
S4D S4D (Gu et al., 2022) is different from the above dieentionbbased methods, as relies on State Space Matrix called "HiPPO matrix" (Gu al., 2020) which enables the whole model to be viewed as
20

---

Under review as a conference paper at ICLR 2023
convolutional model. It does not require the introduction of Q. K, V it takes the whole modeling process as linear time-invariant (LTI) system u(t) + y(t) where u(t) is the input sequence. To parameterize the system, S4D designs a Convolution Kernel K(t) E Raxd and introduces a inner state dimension r during parameterization of (():
((1) Ce v(() (K ()
where A, B, C' are all initialized with certain values and trained with the whole model. During implementation, S4D processes and K in the frequency domain and transforms back the output to the time domain:
((w) = fft(K(t)) ( fft(u(t))
(21)
(22)
S4D(u) - -Lf((((()U(())
(23)
Therefore, S4D has the complexity of ((nr)).
G EFFICIENCY LENGTH
We show efficiency lengths under noncausal self attention pattern in Table 11, where we set hyper- parameters as emb dim 512, head = 8, batch s size 4, ws ze = 16, num landma r ks 16, approx_attn, aeprox_attt_dinn = 16, d_state 16, conv nonv_kernel_ss factor for ProbSparse introduced in Skyformer (Chen et al., 2021b) repository".
5. We use the adaptive
Table 11: The efficiency lengths of attention architectures compared to vanilla attention under noncausal self attention pattern.
Running Time
Memory Usage
Method
Efficiency Length
R²
Efficiency Length
H²
Performer
2,361
0.977
37
0.999
LARA
2,541
0.975
68
0,999
ABC
1,877
0.964
70
0,999
Nyströmformer
3,045
0.987
94
0.999
cosFormer
2,539
0.990
164
0.999
ProbSparse
3,450
0.989
281
0.994
LongShort
5,652
0.998
342
0,999
S4D
6,011
0.996
949
0.999
local
4,195
0.996
1,169
0.999
H BENEFIT OF ATTENTION
We report the CI of backbone models with and without attention in Table 12.
Table 12: The Cl of backbone models with and without attention on tasks. fails on the task.
denotes that the method
Method
TTS
Sum
PCC
SR
LSTF
MLM
efficient attention
1.034
0.715
0.720
1.041
0.591
0.528
vanilla attention
-0.300
-0.246
0.908
-0.076
-0.102
0.527
w/o attention
-2.155
-1.817
0.940
-0.058
0.027
21

---

Under review as a conference paper at ICLR 2023
I INTER-TASK CORRELATION
Figure 4 shows the task-to-task correlation. Except for the causal cross attention pattern, tasks within the other three patterns scarcely correlate positively with each other. This suggests that there does hardly exist single task that can comprehensively test the capability of some attention module under an attention pattern.
LRA
NS
CS
NC
CC
Pathfinder
Retrieval
ListOps
Text
Image
TTS
Sum
SR
.LMM
TTS
Sum
LM
PCC
LSTF
TTS
Sum
1.00
Text
LRA
ListOps
0.75
Retrieval
Pathfinder
0.50
Image
TTS
0.25
Sum
CC CC NC CS NS
SR
0.00
MLM
TTS
-0.25
Sum
LM
-0.50
PCC
LSTF
0.75
TTS
Sum
-1.00
Figure 4: Task-to-tast task correlation.
22


Reviews:
[{"review_id": "oZJ2t-PG7p", "reviewer": "Reviewer_kTjS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": " The authors propose Comprehensive Attention Benchmark (CAB) under a fine-grained attention taxonomy with four distinguishable attention patterns, namely, noncausal self, causal self, noncausal cross, and causal cross attentions. The benchmark collects seven real-world tasks from diverse fields of computer vision, natural language processing, speech processing, and time series forecasting. And an interesting finding is that some efficient Transformers often achieve less competitive results in the causal cross scenario.", "summary_of_the_review": "The benchmark is interesting and covers many tasks from different domains. The exploration of different attention mechanism is interesting. My major concern is that there has been benchmarks for NLP tasks. After all, I would like to see people working on this new benchmark.", "strength_and_weaknesses": "Strength:\n1. A benchmark to evaluate different types of attention patterns for the task with long sequences\n2. The benchmark covers a broad number of tasks.\n\nWeakness:\n1. The number of baselines in figures b,c,d are limited. It would be better to list more.\n2. LongT5, \"Efficient Text-To-Text Transformer for Long Sequences\", also works on the causal attention mechanism. All the datasets in LongT5 could also be a good benchmark. Language model is also a widely used task to evaluate causal attention. Thus, the benchmark may not be novel enough if only considering the NLP area.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ji2LSc99l8", "reviewer": "Reviewer_VyuY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents CAB, a new benchmark to test Transformers and other sequence models on long sequence data. The paper benchmarks a number of methods as a baseline.", "summary_of_the_review": "I think this paper could make a good contribution to the literature, but the paper has a few basic mistakes that should be corrected. If my concerns are addressed in the rebuttal, I'm happy to raise my score above the acceptance threshold.\n\nUpdate after rebuttal: my concerns have been addressed, so I am increasing my score to a 6.", "strength_and_weaknesses": "+ Fills in a missing gap in the literature beyond LRA\n+ Will be useful for researchers evaluating long-sequence models\n\n- There appear to be a few basic mistakes in the methods characterization\n- It would be nice to have a few more baselines\n- Paper could use some copy edits", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FXXSNzW03Q", "reviewer": "Reviewer_auuz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel benchmark, called CAB,  to compare the efficiency of different attention mechanisms in transformers. The current widely used benchmark LRA focusses only on noncausal self-attention. CAB seeks to evaluate four types of attention mechanisms: the combinations of causal/ non-causal and self/ cross attention. The benchmark consists of nine datasets for seven real-world tasks across vision, NLP, speech and time-series forecasting. CAB  is also used to evaluate 8 baseline models covering the four attention mechanisms using standard statistical performance metrics and a proposed composite index as well their efficiency – relative running times and memory usage as a function of the sequence length, highlighting the strengths and weakness of efficient approach across the four attention types suggesting where more research is needed.\n", "summary_of_the_review": "The paper introduces a new benchmark for comparing and evaluating efficient attention mechanisms. I expect the benchmark to be useful to the research community and is likely to help in guiding research into the design of novel efficient attention mechanisms, especially in the causal and cross-modal domains. There is an extensive benchmarking of exemplary baseline algorithms. The paper is mostly written well (though the writing, clarity and the discussions can be somewhat improved).\n", "strength_and_weaknesses": "**Strengths**\n\n- CAB is a novel benchmark with a finer granularity of attention taxonomy than any existing benchmark. LRA addresses only one of the 4 types of attention mechanisms.\n-  CAB comprises 7 tasks of interest to the research community on 9 datasets being used by the research community across 4 domains.\n- The paper also evaluates the performance of 8 popular models as baselines as a part of CAB for new approaches to be benchmarked against.\n- The paper further identifies the scenarios where efficient approaches are significantly lagging behind vanilla attention in performance as the unmet need for the research community to focus on.\n\n**Weaknesses**\n\n***Analysis of Results*** \n- The paper should provide more insights on why results from certain methods vary across various studies: local attention in (Xiong et al, 2022) vs LRA; S4D on LRA vs Sum, MLM tasks, etc.\n- (Figure 3) In the ‘self’ regime, why is it that performance holds across LRA and NS but drops (in correlation) for causal-self? \n- (Figure 3) There is a clear blockiness along the diagonals indicating high positive correlation in performance of approaches within the ‘self’ scenarios (LRA, non-causal and causal) and within the ‘cross’ scenarios (non-causal as well as causal). What may be the reasons for high negative correlation in the off-diagonal blocks? In other words, can you kindly elaborate on footnote 4? Also, what explorations need to be done beyond this benchmark?\n- It is not clear what ‘non-homologous information’ is. Kindly provide explanation and a reference for the term.\n\n***Clarity***\n- Figures showing the tradeoff between performance (accuracy) and cost (time, memory) are recommended to communicate the tradeoffs better.\n- Writing should be improved in Sections 4 and 5. The authors should avoid using words/ phrases like ‘utterly defeated’, ‘embarrasingly’,  etc. \n- Appropriate capitalization should be used in the ‘References’ section.\n- Some easily fixable typos.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5gF7j5i8Joi", "reviewer": "Reviewer_MeQL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose Comprehensive Attention Benchmark (CAB) under a fine-grained attention taxonomy with four distinguishable attention patterns, namely, noncausal self, causal self, noncausal cross, and causal cross attentions.\n", "summary_of_the_review": "This paper proposes to evaluate efficient attentions under a more fine-grained attention taxonomy\nwith four distinguishable attention patterns, each of which reflects different attentive functionality.", "strength_and_weaknesses": "Strength\n1. the paper is well written and feeling pleasure to read. It summarize many eixting of Transformer.\n\n2. The authors propose Comprehensive Attention Benchmark for ling sequence modeling.\n\nWeaknessness\n1. the novelty of this work is not enough for ICLR. It seems that this paper is more related to benchmark dataset.\n\n2. They claim that  related codes will be released at https://github.com/Anonymous. However, I cannot find it.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject