Long Range Arena : A Benchmark for Efficient Transformers 
Authors: Yi Tay
Mostafa Dehghani
Samira Abnar
Yikang Shen
Dara Bahri
Philip Pham
Jinfeng Rao
Liu Yang
Sebastian Ruder
Donald Metzler
Venue: ICLR 2021 Poster (2021)
Keywords: Transformers
Attention
Deep Learning

Abstract: Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models. To this date, there is no well-established consensus on how to evaluate this class of models. Moreover, inconsistent benchmarking on a wide spectrum of tasks and datasets makes it difficult to assess relative model quality amongst many models. This paper proposes a systematic and unified benchmark, Long Range Arena, specifically focused on evaluating model quality under long-context scenarios. Our benchmark is a suite of tasks consisting of sequences ranging from $1K$ to $16K$ tokens, encompassing a wide range of data types and modalities such as text, natural, synthetic images, and mathematical expressions requiring similarity, structural, and visual-spatial reasoning. We systematically evaluate ten well-established long-range Transformer models (Reformers, Linformers, Linear Transformers, Sinkhorn Transformers, Performers, Synthesizers, Sparse Transformers, and Longformers) on our newly proposed benchmark suite. Long Range Arena paves the way towards better understanding this class of efficient Transformer models, facilitates more research in this direction, and presents new challenging tasks to tackle.

Published as a conference paper at ICLR 2021
LONG RANGE ARENA: A BENCHMARK FOR EFFICIENT TRANSFORMERS
Yi Tay's ; Mostafa Dehghani' Samira Abnar', Yikang Shen', Dara Bahri', Philip Pham Jinfeng Rao', Liu Yang', Sebastian Ruder", Donald Metzler
I Google Research
Google DeepMind
{yitay, dehghani (oathann)@goooglle Jégoogle. com
ABSTRACT
Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of ef- ficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality vanilla Transformer models. To this date, there is no well-established consensus on how to evaluate this class of models. Moreover, inconsistent benchmarking on a wide spectrum of tasks and datasets makes it difficult to assess relative model quality amongst many models. This paper proposes systematic and unified benchmark, Long-Range Arena, specifically focused on evaluating model quality under long-context sce- narios. Our benchmark is a suite of tasks consisting of sequences ranging from 1K to 16K tokens, encompassing wide range of data types and modalities such as text, natural, synthetic images, and mathematical expressions requiring sim- ilarity, structural, and visual-spatial reasoning- We systematically evaluate ten well-established long-range Transformer models (Reformers, Linformers, Linear Transformers, Sinkhorn Transformers, Performers, Synthesizers, Sparse Trans- formers, and Longformers) on our newly proposed benchmark suite. Long-Range Arena paves the way towards better understanding this class of efficient Trans- former models, facilitates more research in this direction, and presents new chal- lenging tasks to tackle.
INTRODUCTION
Transformers Vaswani et al. 2017 are ubiquitously state-of-the-art across many modalities, from language (Devlin al., 2018 Raffel et al. 2019 Child et al.) (019)) to images (Tan & Bansal Lu et al.] 2019) to protein sequences (Rives et al., 2019). A common weakness of Transformers is their quadratic memory complexity within the self-attention mechanism that restricts their potential application to domains requiring longer sequence lengths. To date, dizzying number of efficient Transformer models ( ('xformers) ") have been proposed to tackle this problem (Liu et al. 2018 Kitaev
al. 2020 Wang et al. 2020 Tay et al.) 2020b Katharopoulos et al. 2020). Many of these models demonstrate comparable performance to the vanilla Transformer model while successfully reducing the memory complexity of the self-attention mechanism. An overview of this research area can be
found in (Ta et al. 2020c).
Comparing the evaluation and experimental setup of many of these papers, we can make the follow- ing observations. Firstly, there is no unifying consensus on what makes an acceptable test bed for benchmarking efficient Transformers. There also a large diversity in the types of tasks adopted- every single model is evaluated on different set of tasks and datasets, which makes comparison of different models as well as an assessment of their relative strengths and weaknesses difficult. Sec- ondly, the benchmarks used for evaluation are often arbitrarily chosen, without much consideration to whether the task is suitable for evaluating long-range modeling. Thirdly, many papers tend to conflate the effectiveness of the inductive bias with the benefits of pretraining (Ainslie et al. [2020] Zaheer et al. 2020 Wang et al. 2020), which tends to obfuscate the true value of the architecture.
First two authors contributed equally.

---

Published as a conference paper at ICLR 2021
Pretraining itself is computationally expensive endeavour and de-coupling inductive bias research from pretraining would make xformer research more accessible.
benchmark. Long-Range Arena (LRA), for the purpose of bench- marking sequence models under the long-context scenario. We design a benchmark suite comprised of both synthetic probing tasks and real-world tasks and provide relative comparisons for ten cently proposed efficient Transformer models including Sparse Transformers (Child et al. 2019),
In this paper, propose
Reformer (Kitaev al., 2020). Linformer (Wang et al., 2020), Longformer (Beltagy et 2020 Sinkhorn Transformers (Ta et al. 2020b), Performers ( (Choromanski et al. 2020), Synthesizers (Tay
al.) 2020a), Linear Transformers (Katharopoulos al., 2020), and BigBird (Zaheer al.) 2020). This is the most comprehensive and extensive side-by-side evaluation of this class of models.
While the focus of this benchmark the ability of these architectures reason in long-context sce- narios, we are also fundamentally interested in understanding the capabilities and properties of these xformer architectures when exposed to different types of data and conditions. Hence, our benchmark is purposefully designed to be capability probing, i.e, we select datasets and tasks with certain innate structure. For example, can these architectures model long sequences that are intrinsically hierar- chical or that contain some form of spatial structure? In general, we are especially interested in the relative performance of these xformer models across diverse circumstances. We hope that under- standing these better will inspire research more efficient architectures in the future. While the focus of this paper is on efficient Transformer models, our benchmark is also model agnostic and also serve benchmark for long-range sequence modeling.
Aside from comparing the quality of these models, we also conduct extensive efficiency and memory usage analysis of these models. We believe such side-by-side performance benchmark will be valuable to the community, providing deeper insight on the practical efficiency of these methods.
Overall, we propose a unified framework for enabling easy side-by-side comparisons of efficient Transformer models and broadly speaking, long-range sequence models in general. Our framework, which we plan to open source, written JAX/FLAX
2 LONG-RANGE ARENA (LRA)
This section introduces the Lang-Range Arena (LRA) benchmark (pronounced el-ra). We imple- ment our benchmark (which includes the task, evaluators, and models) Python 3 and Jax/Flax and plan to open open-source our code-making it easy to extend and to build on top of our work.
2.1 DESIDERATA
For creating the Long-Range Arena benchmark, we established set of desiderata:
1. Generality: All efficient Transformers models should be applicable to our tasks. For instance, given that not all xformer models are able to perform autoregressive decoding (Wang et al. 2020). we include tasks that only require encoding.
2. Simplicity: The tasks should have simple setup. All factors that make comparisons dif- ficult should be removed. This encourages simple models instead of cumbersome pipelined approaches. For instance, we avoid including any particular data augmentation and consider pretraining to be out of scope of this benchmark.
3. Challenging: The tasks should be difficult enough for current models to ensure there is room for improvement encourage future research in this direction.
4. Long inputs: The input sequence lengths should be reasonably long since assessing how dif- ferent models capture long-range dependencies is a core focus of LRA.
5. Probing diverse aspects: The set of tasks should assess different capabilities of models like their ability to model relations and hicrarchical/spatial structures, generaizzation capability, etc.
6. Non-resource intensive and accessible: The benchmarks should be deliberately designed to be lightweight so to be accessible to researchers without industry-grade computing resources.
2

---

Published as a conference paper at ICLR 2021
2.2 TASKS
This section describes the tasks in the LRA benchmark. Note that these tasks are specifically de- signed for the purpose of assessing different aspects of efficient Transformer models. Further details about each task can be found in the appendix.
2.2.1 LONG LISTOPS
In this task, we are interested in the capability of modeling hierarchically structured data in a long- context scenario. This benchmark task is longer variation of the standard ListOps task proposed in (Nangia & Bowman 2018). which was designed investigate the parsing ability of neural mod- els.
The dataset is comprised of sequences with a hierarchical structure and operators MAX, MEAN, MEDIAN and SUM_MOD that are enclosed by delimiters (brackets). An example (much shorter) sequence is as follows:
INPUT: [MAX 4 IMIN 3 1 [MEDIAN 1 5 9, 211
OUTPUT: 5
In our task we use version of ListOps of sequence lengths up to 2K test the ability to reason hierarchically while handling long contexts. Naturally, in the above example the model needs to access all tokens and model the logical structure the inputs in order to make a prediction. The task is ten-way classification task and is considerably challenging.
2.2.2 BYTE-LEVEL TEXT CLASSIFICATION
This task using real-world data represents common use case of efficient Transformers, which are often needed to process long documents. Text classification in particular is associated with many real-world applications such as spam, fraud, and bot detection and commercial document classification, among others (Howard & Ruder 2018).
This task also benchmarks the ability of the models to deal with compositionality as it required to compose characters into words into higher-level phrases. Compared to ListOps, boundaries are less well defined and need to be learned from the data, which challenging problem in its own right (Kawakami al. 2019).
We consider the byte/character-level setup of this task in order to simulate a longer input sequence, which also makes the task considerably more challenging ! Note that this setup differs significantly from character-level language modeling (char LM). In char LM, it would suffice to read nearby context to determine the next character, e.g., model very likely predict 'e" after having seen the prefix 'appl'. For byte-level text classification, the model needs to reason with compositional. unsegmented data in order to solve meaningful real-world task. We use the IMDb reviews (Maas) et al. 201T) dataset, which is a commonly used dataset to benchmark document classification. We use fixed max length of 4 4K for this task, which truncated or padded when necessary. This is binary classification task with accuracy as the metric.
2.2.3 BYTE-LEVEL. DOCUMENT RETRIEVAL.
We further evaluate a model's 's ability encode and store compressed representations that are useful for matching and retrieval. Learning the similarity score between two vectors is a common problem in machine learning and is useful for a wide array of applications (Guo al.) 2016). Hence, this task is mainly about modeling similarity score between two documents in 'two tower setup' in which compressed representations are concatenated and passed into a linear classifier. Note that deliberately prevent models from using cross attention. This task thus serves as a test of how well models are able to compress long sequences into representations suitable for similarity-basee matching.
We use the ACL Anthology Network (AAN; Radev et al.) 2013) dataset, which identifies two papers have citation link, common setup used in long- form document matching (Jiang et al.
OOn the IMDb word-level task, models without pre-training achieve accuracies in the high 80s while the models in the mid 60s the character-level task (Tay (0220b).
3

---

Published as conference paper at ICLR 2021
2019 Yang al. 2020). Similar to the text classification setup, byte/character level setup, which challenges the model to compose and aggregate information over longer contexts. We use sequence length of 4K for each document, which makes the total text length 8K for this task. This a binary classification task with accuracy as the metric.
2.2.4 IMAGE CLASSIFICATION ON SEQUENCES OF PIXELS
This task is an image classification task, where the inputs are sequences of pixels. In other words, N N image is flattened sequence of length N pixels. Similar how the previous tasks require capturing the hierarchical structure in the data, this task requires the model to learn the 2D spatial relations between input pixels, while presented as ID sequence of symbols. We focus on assessing Transformer models that are designed to process sequence of discrete symbols, we do allow extra modules such as CNN stem that emheds pixel-level inputs. To simplify the setup, we map the input images to a single gray-scale channel where each pixel is represented with an 8-bit pixel intensity (vocabulary size of 256). LRA, we use the CIFAR- dataset (Krizhevsky) 2009) for the image classification task.
2.2.5 PATHFINDER (LONG-RANGE SPATIAL DEPENDENCY)
The Pathfinder challenge (Linsley et al. 2018 Kim et al. 2020 was first introduced for learning long-range spatial dependen- cies. It is synthetic visual task motivated by cognitive psychol- ogy (Houtkamp & Roelfsemal 2010). The task requires a model to make binary decision whether two points represented as circles are connected by path consisting of dashes. We show a positive example of two connected points and negative example of two unconnected points in Figure
The dataset also contains distractor paths, which makes this setup challenging. We model this task by treating images as sequences of pixels. In this task, images are of dimensions (32 32), which make up sequence length of 1024.
(a) positive example.
2.2.6 PATHENDER-X X FLONG-RANGE
SPATIAL DEPENDENCIES WITH EXTREME LENGTHS)
Finally, we consider an extreme version of Pathfinder (Pathfinder- X) where examples consist of 16K pixels (i.e., images of 128 128). The key goal here is to observe if model would fail to solve the 16K extreme version even if it successfully learn the stan- dard version of 1024 tokens. This is an interesting litmus test to see the same algorithmic challenges bear a different extent of difli- culty when sequence lengths are much longer. We include this in our benchmark as Path-X.
(b) A negative example.
Figure 1: Samples of the Pathfinder task.
2.3 REQUIRED ATTENTION SPAN OF LRA TASKS
1.4
One of the main goals of the LRA benchmark is assess- ing the ability of different efficient Transformer models capture long-range dependencies. The tasks and se- tups are designed with this goal in mind. In order to have a quantitative estimate of the spatial extent needed to be considered by attention mechanism to encode the in- puts, we define required attention span.
Required Attention Span
1.2
1.0
0.8
0.6
Given trained attention-based model and a sequence of tokens as inputs, the required attention span of an atten- tion module is computed as the mean distance between the query token and the attended tokens, scaled by atten- tion weights. Here, compute the mean required atten-
Text
Retrieval
Image
Pathfincier
(L-2k)
[L-4k]
(L=4k)
(L-1k)
[L=1k)
Figure 2: Required attention span different tasks.

---

Published as a conference paper at ICLR 2021
Model
ListOps
Text
Retrieval
Image
Pathfinder
Path-X
Avg
Chance
10.00
50.00
50.00
10.00
50.00
50.00
44.00
Transformer
36.37
64.27
57.46
42.44
71.40
FAIL
54.39
Local Attention
15.82
52.98
53.39
41.46
66.63
FAIL
46.06
Sparse Trans.
17.07
63.58
59.59
44.24
71.71
FAIL
51.24
Longformer
35.63
62.85
56.89
42.22
69.71
FAIL
53.46
Linformer
35.70
53.94
52.27
38.56
76.34
FAIL
51.36
Reformer
37.27
56.10
53.40
38.07
68.50
FAIL
50.67
Sinkhorn Trans.
33.67
61.20
53.83
41.23
67.45
FAIL
51.39
Synthesizer
36.99
61.68
54.67
41.61
69.45
FAIL
52.88
BigBird
36.05
64.02
59.29
40.83
74.87
FAIL
55.01
Linear Trans.
16.13
65.90
53.09
42.34
75.30
FAIL
50.55
Performer
18.01
65.40
53.82
42.77
77.05
FAIL
51.41
Task Avg (Std)
29 (9.7)
61 (4.6)
55 (2.6)
41 (1.8)
(3.7)
FAIL
52 (2.4)
Table 1: Experimental results on Long-Range Arena benchmark. Best model is in boldface and sec- ond best underlined. All models do not learn anything on Path-X task, contrary to the Pathfinder task and this is denoted by FAIL. This shows that increasing the sequence length can cause seriously difficulties for model training. We leave Path-X on this benchmark for future challengers but do not include it on the Average score it has no impact relative performance. Important: These results always represent the results at the time of ICLR submission and are not modified for archival purposes. Please appendix in arxiv version of this paper for updated snapshots new runs of this leaderboard with new settings and use that for paper comparisons.
tion span over attention modules in our best vanilla Transformer model for each task, averaged over 1K random samples from the validation set. Figure 2 summarizes the required attention span for each task in LRA. For all the tasks in LRA the required attention span rather high. This shows, Transformer model needs to go beyond combining only local information, while in many other tasks and datasets, attention mechanism mostly need to combine information from neighboring po- sitions. Given the purpose of LRA, found required atlention span serves good proxy for how difficult a task is for Transformer-based models
3 EXPERIMENTAL RESULTS
3.1 MODELS
This section describes the models we evaluate on our LRA benchmark. We base our evaluation ten recently proposed efficient Transformer models. Aside from the standard vanilla Trans- former (Vaswani et al.) 2017) and a simple local attention baseline, we compare Sparse Trans- 22020), Linformers (Wang Reformers (K Kitaae et al. 2020), Sinkhorn Transformers (Tay et al. (2020b). Synthesizers (Tay et al. 2020ah, BigBird (Zaheer et 2020), Linear Transformers (Katharopoulos et al., 2020), and Per- formers (Choromanski et al.) (2220.. We believe these ten models to represent diverse cross-section of recent efficient Transformer models.
formers (Child et al. 2019). Longformers (Beltagy
20200)
3.2 PHILOSOPHY BEHIND THE BENCHMARK
We note that it is non-trivial and almost impossible to conduct perfectly fair evaluation of all models. The large search space motivates us to follow a set of fixed hyperparameters (number of layers, heads, embedding dimensions, etc.) for all models. The best performance and relative order of the models may change if we aggressively tune hyperparameters for all models. Hence, the results provided in this paper are not meant be final authoritative document on which xformer the
NNte that this metric mainly provides an indication of the required attention span for a task and the relative differences hetween tasks based reasonably strong model; a better model might only need to attend to
shorter ranges (Daniluk 2017 Rac & Razavi 2020

---

Published as a conference paper at ICLR 2021
best. Instead, provide starting point for future research and strive to be as fair possible. In order to do so, we plan to release the code with all the hyperparameters and implementation details. Additionally, we intend for our paper to be a living document and encourage researchers (authors and the broader community) to contribute and continue updating this paper (with rules and limitations described in the appendix). We also implemented all models to the best of our abilities. We often consulted with the original developers of the included models. For simplicity, the main text of the paper will always contain the initial set of results that we submitted to ICLR. Important: All future updates will, for consistency, be in the appendix of the paper with marked version identifiers so other researchers can always reference them in other papers. If you obtain better result on any of the following models we tried, please send an email and we can discuss making an update to the leaderboard.
3.3 QUANTITATIVE RESULTS
Based on our results, we observe that (1) all proposed tasks in LRA are considerably challenging and (2) there are meaningful differences in model performance across different xformer models.
Results on ListOps The ListOps task (10-way classification) has proven to be reasonably diffi- cult with the best models obtaining only 37%. The considerable gap to random chance shows that models are indeed learning the task. We notice that the inductive bias the xformer models plays a substantial role on this task in which approximately half the xformer models are able to get > 30% performance while the remainder of the models only get slightly above random chance. This may imply that certain efficiency-inspired inductive biases may be better at handling hierarchical data than others. For instance, the results from our experiments seem to suggest that kernel- mod- els (e.g.. Performer, Linear Transformers) are possibly not effective hierarchically structured data. We feel that ListOps may be useful in probing a model's capability in handling hierarchically structured data.
Results on Text Classification Byte-level classification is shown be difficult and challenging especially when no pretraining or contextual embeddings are used. The best model only obtains 65.90 accuracy. The Linear Transformer performs well on this task, along with the Performer model. Contrary to the ListOps task, it seems like fast kernel-based models do well on this task.
Results on Retrieval The scores of different models on this task are also rather low (average of 55%), indicating the difficulty of the task. The vanilla Transformer model only achieves 57.46% accuracy with some xformer variants scoring very close to random chance. The best performing model is the Sparse Transformer and the second best is BigBird. We find that models that follow fixed sparse patterns to do well on this task. Models that are based on low-rank factorization and kernels perform relatively worse.
Results on Image Classification On the image classification task, most models perform quite similarly (low variance amongst model performance). The best model this task is the Sparse Transformer, followed by the Performer. Linformer and Reformers do not do well on this task. On related note, we also observed most of models struggle generalizing the test even though they manage to overfit the training set. While we extensively tried different regularization techniques on every single model, there a rather large gap between their performance on train and test set (More details in Appendix).
Results on Pathfinder Path-X Results show that all models achieve reasonable performance on the Pathfinder task. The average performance is 72 and the best model Performer obtains 77.05% accuracy. The Local Attention model performs the worse out of all models. It seems that the best models on this spatial reasoning task are the kernel models (Performer and Linear Transformer).
All models failed to solve the Path-X task, achieving best 50%. We find this intriguing because this is essentially an identical task to the standard Pathfinder, albeit with much longer sequence lengths. Hence, we observe that the extreme length of the task significantly obstruct model from leaning anything meaningful. We leave Path-X in our benchmark suite, hoping to spur future progress in modeling sequences extreme lengths.

---

Published as a conference paper at ICLR 2021
Train Speed (Steps per second)
Peak
Memory
Usage
(GB)
Model
IK
2K
3K
4K
1K
2K
3K
4K
Transformer
8.1
4.9
2.3
1.4
0.85
2.65
5.51
9.48
Local Attention
9.2 (1.1x)
8.4 (1.7x)
7.4 (3.2x)
7.4 (5.3x)
0.42
0.76
1.06
1.37
Linformer
9.3 (1.2x)
9.1 (1.9x)
8.5 (3.7x)
7.7 (5.5x)
0.37
0.55
0.99
0.99
Reformer
4.4 (0.5x)
2.2 (0.4x)
1.5 (0.7x)
1.1 (0.8x)
0.48
0.99
1.53
2.28
Sinkhorn Trans
9.1 (1.1x)
7.9 (1.6x)
6.6 (2.9x)
5.3 (3.8x)
0.47
0.83
1.13
1.48
Synthesizer
8.7 (1.1x)
5.7 (1.2x)
6.6 (2.9x)
1.9 (1.4x)
0.65
1.98
4.09
6.99
BigBird
7.4 (0.9x)
3.9 (0.8x)
2.7 (1.2x)
1.5 (1.1x)
0.77
1.49
2.18
2.88
Linear Trans.
9.1 (1.1x)
9.3 (1.9x)
8.6 (3.7x)
7.8 (5.6x)
0.37
0.57
0.80
1.03
Performer
9.5 (1.2x)
9.4 (1.9x)
8.7 (3.8x)
8.0 (5.7x)
0.37
0.59
0.82
1.06
Table 2: Benchmark results of all Xformer models with a consistent batch size of 32 across all models. We report relative speed increase/decrease in comparison with the vanilla Transformer in brackets besides the steps per second. Memory usage refers to per device memory usage across each TPU device. Benchmarks are run on 4x4 TPU V3 Chips. We report inference speeds in the supplementary material.
3.4 EFFICIENCY BENCHIMRRKS
In this section, we report efficiency metrics of our runs. For simplicity, we use the byte-level text classification benchmark and report run times and memory consumption of the sequence lengths (1K,2K, 3K, 4K]. We use batch size of 32 (1 example per core) for all runs and conduct ex- periments on 4x4 TPU V3 Chips. We emphasize that these runs are again largely conditioned on hardware and implementation details. For speed (steps per second), this is based realistic train- ing speed (i.e., including overheads such as io, batching, pipeline etc.) More details can be found in the appendíx. Do note that the main point of these efficiency numbers is to provide relative comparison. For follow-up work, it would be ideal to always rerun these numbers on a comparable hardware/setup. We report inference speeds in the supplementary material.
Results on Train Speed Table reports our efficiency benchmarks on the xformer models. We note that low-rank and kernel- based models are generally the fastest. The overall fastest model is the Performer model (Choromanski et al., 2020), which is 5.7x faster than Transformers on the 4k sequence length. Linformer (Wang et al. 2020 and Linear Transformers (Katharopoulos et al., 2020) come in a close second and are almost as fast as Performers (at 5.5x to 5.6x faster). Local Attention is also considerably fast. Based on implementation, the slowest model is the Reformer model (Kitaev et al. 2020) that is about 80% the speed of vanilla Transformer 4K sequence lengths and half the speed at 1K sequence length.
Results on Memory Consumption The model with the smallest memory footprint in our bench- marks is the Linformer model, coming in at 0.99GB per TPU device as compared to 9,48GB per TPU device for the vanilla Transformers at N 4K. That is about a 10x reduction in memory footprint. Similar to speed, Performers and Linear Transformers are also relatively compact and are almost as compact as Linformers. Other models (Local Attention, Reformers, BigBird, Synthesiz- ers) are still less memory hungry compared to vanilla Transformers but are relatively less efficient (memory consumption wise) compared to Linformers, Performers, and Linear Transformers. We also notice that the memory consumption of models such as Linformer and Performer scales very well, with the memory usgae at 3K and 4K being approximately equal.
3.5 OVERALL RESULTS: No
Based our analysis, the best qualitative performance in terms of LRA score, i.e. integrated across all five tasks, is the BigBird model. While BigBird does not do extremely well on any individual task compared to other models, it has consistently good performance across all tasks. Performers and Linear Transformers have strong performance on some tasks but their average is lowered by the I ListOps task.

---

Published as a conference paper at ICLR 2021
Figure 3 shows the trade-off between qualitative performance (y-axis), model speed (x-axis), and memory footprint (size of the circles). While BigBird performs well, its speed almost similar to the vanilla Transformer. In fact, based slower at shorter sequences (i.e., 1K-2K). On the other hand, model like Local Attention is fast the cost of lower quantitative performance. Among these models, the kernel-based variants, i.e., Performer, Linformer, and linear Transformer seem to be able to make a better trade-off in terms of speed and performance, while having reasonable memory usage. Overall, the models that lie on the pareto-optima curve is BigBird and Performers.
efficiency benchmarks in Table 2. it is slightly
58
Big Bird
BigBird
Transformer
Transformer
54
54
Synthesizer
Synthesizer
LRA Score
52
Linformer
Performer
LRA Score
52
Performer
Sinkhorn
Reformer
Sinkhorn
Reformer
50
Linear Transformer
Linear Transformer
Local Attention
Local Attention
50
100
150
200
250
300
350
Speed (examples sec)
Memory Footprint (GB)
Figure 3: Trade-off between performance axis) and resources (x: axis). On the left, circle size corresponds to memory footprint. On the right, it corresponds to examples per second.
RELATED WORK
4.1 EFFICIENT TRANSFORMERS
The pervasiveness of Transformer models, along with its well-known trait of being memory inten- sive, has spurred on a large number of innovations on this front. Early work in this area has typically considered a fixed pattern (local window) approach (Liu et al.) 2018: Parmar et al. 2018). More
advanced models have been proposed recently. Early work in this area has typically considered combinations of fixed patterns (Child et al., 2019; (Ho et al. 2019 Beltagy et al. 2020: Zaheer al. 2020) that learn sparse attention by considering combinations of fixed strides or local win- dows. There has been also interesting work in striving to learn these patterns patterns (Kitaev al.
2020 Roy et al, 2020) using LSH hashing, clustering (Roy et al., (020)) and/or sorting (Tay et al., 2020b}. The latest models are largely based on kernels (Katharopoulos et al. [020 Choromanski et al.] (020) or low-rank approximations (Wang et al. (2020) which either treat the attention matrix low-rank (using low-rank projections Wang al. (2020)) or rewriting of the self-attention equa- tion (Katharopoulos et al. 2020). For the sake of brevity, we refer interested readers to (Tay et al.,
2020ch for detailed survey of this line of research.
4.2 EXISTING BENCHMARKS
Generative Modeling / Language Modeling This generative modeling task requires predicting the next character, word, or pixel and is staple in xformer evaluations (Roy et al. [2020] Kitaev al.) 2020). However, it has been debated how much long- range signal such tasks actually encode (Rae & Razavi 2020 L.TTM language models augmented with attention have been shown to rarely attend beyond seven preceding words of context (Daniluk et al. (017) and samples from LSTM language models are known to quickly devolve into generic text. On the other hand, recent models such as the Transformer-XL (Dai et al. (2019) have been observed to be sensitive to context of around 900 tokens and samples from large-scall scale models (Radford et al. 2019) maintain a consistent theme over much longer sequences. Even such recent models, however, can be improved by limiting the range of attention (Rae & Razavi 2020). In sum, while standard language modelling datasets contain some long-range signal, which is required to perform long-range coreference resolution, reasoning
8

---

Published as a conference paper at ICLR 2021
with events, discourse understanding, etc. (Ruder al. 2019) it seems to be overshadowed by the much stronger signal of short-term word co-occurrences and is thus difficult to evaluate
Question Answering Another commonly used evaluation task is question answering (QA: Zaheer al. 2020). Open-domain QA in partieular typically requires the model to answer questions based on long contexts such as entire Wikipedia documents (Joshi et al. 2017 Kwiatkowski et al. 2019) books (Kočiský al. (018)) Other datasets explicitly designed to require multiple 'hops' of reasoning (Welbl et al., 2018 Yang et al., 2018). Successful approaches are often highly engineered, computationally expensive systems that require pre-training and a separate retrieval model (Lee et al., 2019; Guu et al. 2020).
Natural Language Understanding / GLUE tasks Evaluation natural language understanding (NLU) tasks also common (Wang al., (020). Examples in most of these datasets such as MultiNLI (Williams al. (2118) and SST (Socher et al. 2013) consist of single sentences and less than 100 tokens on average.
5 CONCLUSION
We proposed Long Range Arena (LRA), new benchmark for evaluating progress efficient Trans- former research. Our new benchmark is challenging and prohes at model capabilities dealing with diverse data types and structures such text, mathematics, and visual data. Our benchmark comprises of tasks ranging from 1K to 16K tokens. For the first time, we conduct an extensive side-by-side comparison ten recently proposed efficient Transformer models. The experimental results show that these tasks are very challenging even for long-range Transformer models. The overall results show that there no sre-sizeefiis-alll solution and trade-off's have be made in terms of model quality and speed/memory. We plan to open source our code and benchmarks to facilitate future benchmarking, research and model development.
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Philip Pham, Anirudh Ravula, and Sumit Sanghai. Etc: Encoding long and structured data in transformers. arXiv preprint arXiv:2004.08488 2020.
Iz Beltagy, Matthw E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.00 05000005,0011 2020.
Rewon Child, Scott Gray, Alee Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Jared Davis, Tamas Sarlos, David Belanger, Lucy Colwell, and Adrian Weller. Masked language modeling for pro- teins via linearly scalable long-context transformers. arXiv preprint arXiv:2006.03555 2020.
Zihang Dai, Zhilin Yang, Yiming Yang, William W. Cohen, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer-XL. Attentive Language Models Beyond a Fixed-Length Context. In Proceedings of ACL 2019, 2019.
Michal Daniluk, Tim Rockt, Johannes Welbl, and Sebastian Riedel. Frustratingly Short Attention Spans in Neural Language Modeling. In Proceedings of ICLR 2017, 2017.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04808 2018.
Datasett such LAMBADA (Paperno et al. 2016) more explicitly test for context understanding but still restricted comparatively short contexts of hive sentences average.

---

Published as a conference paper at ICLR 2021
Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 55-64, 2016.
Kelvin Guu, Kenton Lee, Zora Tung, and Panupong Pasupat. REALM: Retrieval-Augeente Lan- guage Model Pre-Training In Proceedings of ICML. 2020, 2020.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- mensional transformers. arXiv preprini urXiv: 2019.
Sara Hooker. The hardware lottery. arXiv preprint crXiv:2009.06445) 2020.
R. Houtkamp and P. R. Roelfsema. Parallel and serial grouping of image elements in visual percep- tion. J Exp Psychol Hum Percept Perform,, 2010.
Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classifica- tion. In Proceedings of ACL 2018, 2018.
Jyun Yu Jiang, Mingyang Zhang, Cheng Li, Michael Bendersky, Nadav Golbandi, and Mare Najork. Semantic text matching for long-form documents. In The World Wide Web Conference, pp. 795- 806, 2019.
Mandar Joshi, Eunsol Choi, Daniel S Weld, Luke Zettlemoyer, and Paul G Allen. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. Proceedings of ACI. 2017.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. arXiv preprint arXiv:2006.1622 16236 2020.
Kazuya Kawakami, Chris Dyer, and Phil Blunsom. Learning to discover, ground and use words with segmental neural language models. In Proceedings of ACL 2019, pp. 6429-6441, 2019.
Junkyung Kim* Drew Linsley*, Kalpit Thakkar, and Thomas Serre. Disentangling neural mecha- nisms for perceptual grouping. In International Conference on Learning Representations, 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. International Conference on Learning Representations, 2020. URL.htttss://openeeeee
In
net//oozum?id-rkgNWaaa
Tomá' Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA Reading Comprehension Challenge. Transactions of the Association for Computational Linguistics, 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin Kenton Lee, Kristina Toutanova, Llion Jones Matthew Kelcey, Ming- Wei Chang, Andrew M Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural Questions: Benchmark for Question Answering Research. In Transactions of the ACL, 2019.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent Retrieval for Weakly Supervised Open Domain Question Answering. In Proceedings of ACL 2019, 2019.
Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learning long-range spatial dependencies with horizontal gated recurrent units. In Advances in neural information processing systems, pp. 152-164, 2018.
Peter Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. arXiv preprint arXiv: 1801. 10198, 2018.
10

---

Published as a conference paper at ICLR 2021
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision-and and-language tasks. In Advances in Neural Information Processing Systems, pp. 13-23, 2019.
Andrew L. Maas, Raymond E. Daly, Peter Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150,
Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http: 7/ 1/www.oo wwwwwb.ww........... org/ htt/a/aoooooyl--1110//
Nikita Nangia and Samuel R Bowman. Listops: A diagnostic dataset for latent tree learning, arXiv preprint arViv: 1804 06028, 2018.
Denis Paperno, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern. The LAMBADA dataset: Word prediction requiring broad discourse context. In Proceedings of ACL 2016, 2016.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. arXiv preprint arXiv:1802.0573) 2018.
Dragomir R Radev, Pradeep Muthukrishnan, Vahed Qazvinian, and Amjad Abu-Jbara. The acl anthology network corpus. Language Resources and Evaluation, 47(4):919-944, 2013.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. 2019.
Jack W Rae and Ali Razavi. Do Transformers Need Deep Long-Range Memory? In Proceedings of ACL 2020, pp- 7524 -5299, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Alexander Rives, Siddharth Goyal, Joshua Meier, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. bioRxiv, pp. 622803, 2019.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. arXiv preprint arXiv:2003.05997 2020.
Sebastian Ruder, Matthew E Peters, Swabha Swayamdipta, and Thomas Wolf. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pp. 15-18, 2019.
Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP 2013, pp. 1631 1631-1 Citeseer, 2013.
Hao Tan and Mohit Bansal. LXMERT: Learning Cross-Modality Encoder Representations from Transformers. In Proceedings of EMNLP 2019, 2019.
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention in transformer models. arXiv preprint 2rX1v:2005.00743, 2020a.
Yi Tay, Dara Bahri, Liu Yang. Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. arXiv preprint arXiv:2002. /296, 2020b.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, arXiv preprint arXiv:2009.06732 2020c.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
11

---

Published as a conference paper at ICLR 2021
Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint 2rXiv:2006.04766 2020.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing Datasets for Multi-hop Read- ing Comprehension Across Documents. In Transactions of the Association for Computational Linguistics, 2018.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A Broad- Cover-Ce Challenge Corpus for Sentence Understanding through Inference. Proceedings of NAACL-LLT 2018, 2018. URL
http://arxiv org. wwtp:///.rr/bb//11/////4 05426
Liu Yang. Mingyang Zhang, Cheng Li, Michael Bendersky, and Marc Najork. Beyond 512 tokens: Siamese multi-depth transformer-based hierarchical encoder for document matching. CoRR, abs/2004. 2020. URL/ttss://axxt...
wttporgg/abbaaaaa//////// 12297
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of EMNLP 2018, 2018.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint 100:0 2020.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.I
LRA TASKS
This section describes the details and hyperparameters of each task. We also plan to release the configuration files along with the implementation of the models and benchmarks, that can be used to reproduce the results reported in the paper.
A.1.1 LISTOPS
Following the generation steps in (Nangia & Bowman 2018). 8). we generate our own long version of this task. We use a sequence length 2k for this task. All our xformer models have an embedding dimension of 512, 8 heads, 6 layers and a feed-forward dimensions of 2048. We train all models for 5K steps. The [CLS] token is used and mapped into 10 class Softmax layer for classification.
A.1.2 BYTE-LEVEI. DOCUMENT CLASSIFICATION
We use the IMDb reviews dataset MMas et al. 2011 and sequence length of (1K, 2K,3K, 4K) tokens for all models. We pick the best results across these four sequence lengths. We use a [cls] token for prediction. All the [cls] tokens from xformer encoders are passed into a two layered MLP with ReLU activations. The MLP emits 2-class logits for binary classification. We optimize the softmax cross entropy loss function. All xformer models are parameterized by the same number of layers, heads and hidden dimensions, namely 8 heads, 512 hidden dimensions and d 2048 for positional FFN layers. We use 6 layers for all xformers. The learning rate is 0.05 with weight decay of 0.1. We use Adam with warmup. All models trained for 20K steps and batch size of 32.
A.1.3 BYTE-LEVEL DOCUMENT MATCHING
We use the ACI. anthology network for a related article matching task. We use a sequence length of 4K per document (8K tokens in total for two sequences). The two encoders share parameters. Similar to document classification, we use the [cls] token from xformer encoders. Let X₁ be the [cls] token embedding from document I and X₂ be the [cls] token embedding from document 2, the final score is computed via:
X2,
X2))
(I)
where MLP(.) is a two layered MLP with relu activation functions. In lieu of the much longer sequence length, of 512 and 4 layers. Model is trained with Adam for 5K steps with learning rate of 0.5.
batch size of 32, embedding dimension of 128, 4 heads, FFN dimension
A.2 IMAGE CLASSIFICATION
We use the gcay-s (single channel) CIFARIO as the image classification dataset, with 10 classes. The resolution of input images is 32 32 and after flattening the input images, we feed our xformer encoders with a sequence of 1024 pixels. Similar to our other classification tasks, there is classifier head on top of the xformer encoder, consisting of two-layer MLP with ReLU activation. Softmax cross-entropy has been used for optimizing the parameters of the models. We trained our models for 200 epochs and have done extensive sweeps over different hyper-parameet values leading to the best average performance across all xformers: 3 layers, 4 heads, 128 as the hidden dimensions of FFN blocks, 64 as the query/key/value hidden dimensions, and finally learning rate of 0.01.
and found the following
A.2.1 GENERALIZATION GAP
For the image classification benchmark, in Section mentioned that most of the models struggle generalizing to the test set. Table presents the train and test accuracy for different models and for almost all these models, the gap between the two scores is considerably high.
While this task can be simple to solve for convectional models (c.g.. accuracy of wide-resnet on gray-scale CIFAR10 10 with no data augmentation is 89.21) it is rather difficult for Transformer-based
13

---

Published as a conference paper at ICLR 2021
Model
test accuracy
train accuracy
Transformer
42.44
69.45
Local Attention
41.46
63.19
Sparse Trans.
44.24
66.74
Longformer
42.22
71.65
Linformer
38.56
97.23
Reformer
38.07
68.45
Sinkhorn Trans.
41.23
69.21
Synthesizer
41.61
97.31
BigBird
40.83
71.49
Linear Trans.
42.34
65.61
Performer
42.77
73.90
Table 3: Test and train accuracy different models on Image Classification task.
models with this setup. Naturally, one can find ways to improve the performance with a different setup. For instance, in our setup, models and consume them as independent symbols. We observed that learning embedding that reflects this property is rather hard for most of these models (Figure) ). If we simply replace the embedding layer with CNN stem, we see imitate boost in the performance (e.g. replacing the embedding layer of vanilla Transformer with a convectional stem, with 3 kernel, we get accuracy of 75.32).
informed about the oridinality of pixel intensities
Another modification that can lead to better performance is to incorporate spatial representation that are translation invariant in Transformer models (e.g., adding relative positional embedding to vanilla transformer, we get accuracy of 61.72). However, adding these sorts of changes make the setup digress from the original point this task in our benchmark.
A.2.2 VISUALIZATIONS OF LEANED EMBEDDING BY A VANILLA TRANSFORMER
Figure presents visualizations for the pixel intensity and positional embedding that a vanilla trans- former model learns for the image classification task, on the gray-scaled CIFARIO detest.
s larty of poattion embbediige
Red 1101001
250
Figure Left: The cosine similarity between the embedding learned for each pixel intensity. Right: Each tile shows the cosine similarity between the position embedding of the pixel with the indicated row and column and the position embeddings of all other pixels.
On the left, we can see the pairwise similarity of learned embeddings for pixel intensities. Although there is higher similarity for close pixel values, the patterns from these learned embeddings do not perfeetly reflect the ordinality the pixel intensities. On the right, we can see the pairwise similarity of positional embeddings for different input positions. We can that the lower the
14

---

Published as a conference paper at ICLR 2021
distance between two pixels is, the more similar their learned positional embeddings. However, the spatial closeness in y axis is more preserved in the learned embedding than the distances in the axis.
A.3 PATHFINDER
Pathfinder task probes the ability of models to detect long range spatial dependencies between input features. To solve the task, model requires to identify the target contour and trace from one end to the other. Although Pathfinder is visually simple task, it has been show that the clutter and variations in path shape makes the task difficult for CNN models (Linsley et al. 2018 Kim et al, 2020).
The Pathfinder task a binary classification task and resolution of input images 32 x 32. Similar to image classification task, we feed our xformer encoders with sequence of 1024 pixels after flattening the input images. The classifier head on top of the xformer encoder is also a two- layer MLP with ReLU activation and we use Softmax cross-entropy loss for the optimization. We trained our models for 200 epochs. The hyper-parameters used for the xformer model are as follow: layers, 8 heads, 128 as the hidden dimensions of FFN blocks, 128 as the query/key/value hidden dimensions, and the learning rate of 0.01.
A.3.1 VISUALIZATION OF THE ATTENTION MAPS FROM A VANILLA TRANSFORMER
Given that transformers have many units with global receptive field, they have better potential for solving the task, compared to models with local receptive fields. Figure the attention distri- butions for a set of examples given on token (CLS token) as the query. We can see that the attention module collects information from different positions in input to be able to trace the target path.
Figure 5: Attention map for different examples from the Pathfinder task. Each map presents the attention distribution, given the CLSS token at the final layer as the query, averaged across heads in vanilla Transformer model. Note that for visualization, use attention-rollon (Abnar & Zuidema 2020) for more precise input attribution.
We have also included Pathfinder- in LRA, which is similar to Pathfinder, but inputs are in higher resolutions, i.c. longer input sequences. On Pathfinder-X, we have tried setups for training models, first training models from scratch, second evaluating models that are trained on Pathfinder. In both cases, found out none of the models able to deal with/generalize to 16K input length.
15

---

Published as a conference paper at ICLR 2021
B MODELS AND IMPLEMENTATION
This section describes the details of our implementation. The code is primarily written in JAX and FLAX. In this section, we note specific details about certain implementations models. We plan to release hyperparameters in form of readme or script later.
B.1 SPECIAL CASES OF OUR IMPLEMENTATION
This section describes several special cases in our implementation details. The diverse suite of Transformers come with plethora hardware constraints and implementation details. To succeed, Transformer model needs to also 'win' the hardware lottery (Hooker) (020), i.e., having readily supported ops, kernels or accelerator support take advantage of its technical design. This section discusses some of the trade-off's and edge cases that make comparison of several models challenging. In the end, we argue that simplicity virtue and not requiring any special support positive thing for an efficient Transformer model.
On CUDA kernels CUDA kernels are cumbersome and are specific to GPU hardware, making difficult to implement or use TPU pods. Generally, these are considered to be undesirable and inconvenient in practical applications. Hence, Sparse Transformer and Longformer are implemented with equivalent implementations emulate for performance. This is by applying equivalent mask. For this reason, we do not benchmark Sparse Transformer and Longformer for speed.
Reformer's Implementation Having optimized ops support many of Reformer's functionality crucial. Hence, Reformer is implemented slightly differently from other Transformer models. Instead of computing tensors with batch size dimensions B and head dimensions H. (i.e., B H x N x d), we compute the attention function for tensors of N x d dimensions. After which, we parallelize this function via VMAP over the batch and head dimensions.
C INFERENCE SPEED BENCHMARKING
In section 3.4 presented the comparison between the speed of different models at the training time. This section provides the results similar comparison, but at the evaluation//nferenee time.
Model
1K
2K
3K
4K
Transformers
114
64
28
16
Local Attention
108 (0.47x)
110 (1.71x)
108 (3.86x)
111 (6.93x)
Reformer
54 (0.47x)
27 (0.42x)
18 (0.64x)
13 (0.81x)
Synthesizer
111 (0.97x)
106 (1.65x)
55 (1.96x)
31 (1.94x)
Sinkhorn Transformer
111 (0.97x)
108 (1.69x)
110 (3.92x)
100 (6.25x)
Linformer
112 (0.98x)
111 (1.73x)
109 (3.89x)
110 (6.88x)
BigBird
70 (0.61x)
34 (0.53x)
23 (0.82x)
17 (1.06x)
Linear Transformers
108 (0.95x)
111 (1.73x)
109 (3.89x)
111 (6.94x)
Performer
116 (1.02x)
110 (1.72x)
116 (4.14x)
110 (6.89x)
Table Speed (Steps per second) on running inference. Benchmarked on 4x4 TPU V3 chips with a batch size of 32 (1 example per core). Results are computed on Xformers with 4 layers, heads, 128 hidden size. Similar training, we include realistic evaluation time which includes pipeline ops, and batching into the overall time.
Inference results Overall, the relative results and trends inference is not too different from training (shown Table 2 of the main paper). Similarly, Performer, Linear Transformers, Linformer and Local Attention remains to be very strong in terms of inference speed. It is worth also note that all xformer variants are slower than the vanilla Transformer at 1K length. This is unlike training, where most xformers performing R 1.1x speed of vanilla Transformers.
16

---

Published as a conference paper at ICLR 2021
D CONVERGENCE ANALYSIS
We analyze the convergence quality of these xformer models. We report the steps to N% accuracy on the Image benchmark. Given that the final val accuracy is about 38% 40% for most models, we report the time for models take to reach 30% and 35% as an estimate of how fast these models converge
Model
N = 30%
N = 35%
Transformer
2048
3920
Local Attention
1425
3512
Sparse Transformer
1050
3325
Reformer
1720
6650
Linformer
452
875
Longformer
713
1575
Sinkhorn Transformer
875
1575
Synthesizer
2975
3325
Linear Transformer
2186
3500
BigBird
1225
2462
Performer
1400
2625
Table 5: Number of training steps to reach N% validation accuracy where N =300, 35..
Results The Linformer model converges the fastest, followed by Longformer, Sinkhorn Trans- formers and then BigBird. The vanilla Transformer is a little on the slow side of convergence but is still faster than Reformer, the slowest model converge. Notably, the convergence speed is not signal of how well it would finally perform as noted in the results in Table 1.
accuracy tends
17

---

Published as a conference paper at ICLR 2021
E ARCHIVAL SNAPSHOTS OF THE RUNS
This section reports the archival snapshots and change log of the leaderboard and comparisons of the different model. As we tune hyperparameters and fix certain issues, we will update the snapshot of the model comparison here. Future versions may also include other models that we benchmark and compare. For easy comparisons in future papers, please quote the version number of the leaderboard. We will continiously update this section in the appendix if there are any futhur updates the existing models.
E.1 VERSION 1: ICLR CAMERA READY.
Version active as of March 2021, The main changes is that we reran all models for the ListOps task. We train models for longer this time round for up to 10K steps.
Model
ListOps
Text
Retrieval
Image
Pathfinder
Path-X
Avg
Chance
10.00
50.00
50.00
10,00
50.00
50,00
44.00
Transformer
36.38
64.27
57.46
42.44
71.40
FAIL
54.39
Local Attention
15.95
52.98
53.39
41.46
66.63
FAIL
46.08
Sparse Trans.
35.78
63,58
59.59
44.24
71.71
FAIL
54.98
Longformer
36.03
62.85
56.89
42.22
69.71
FAIL
$3.54
Linformer
35.49
53,94
52.27
38.56
76.34
FAIL
51.32
Reformer
36.30
56.10
53.40
38.07
68.50
FAIL.
50.47
Sinkhorn Trans.
34.20
61.20
53.83
41.23
67.45
FAIL
52.78
Synthesizer
36.50
61.68
54.67
41.61
69.45
FAIL
52.78
BigBird
37.08
64.02
59.29
40.83
74.87
FAIL
55.22
Linear Trans.
17.15
65.90
53.09
42.34
75.30
FAIL
50.76
Performer
36.00
65.40
53.82
42.77
77.05
FAIL
55.01
Task Avg (Std)
32 (7.9)
61 (4.6)
55 (2.6)
41 (1.8)
72 (3.7)
FAIL
52 (2.4)
Table 6: Experimental results Long-Range Arena benchmark. Best model is in boldface and sec- ond best is underlined. All models do not learn anything on Path-X task, contrary to the Pathfinder task and this is denoted by FAIL. This shows that increasing the sequence length can cause seriously difficulties for model training. We leave Path-X on this benchmark for future challengers but do not include on the Average score has no impact relative performance.
E.1.1 NOTABLE CHANGES
The main outcome on the new leaderboard is as such:
The relative results on I Listops largely remain unchanged for most models. However, mod- els such Performer and Linformer managed get comparable results to the other models with more training steps.
Performer is the 2nd best model. BigBird is still the leading model.
E.2 VERSION ORIGINAL RESULTS TO ICLR
Version 0 from September 2020 March 15th 2021.
18

---

Published as a conference paper at ICLR 2021
Model
ListOps
Text
Retrieval
Image
Pathfinder
Path-X
Avg
Chance
10.00
50.00
50.00
10.00
50.00
50.00
44.00
Transformer
36.37
64.27
57.46
42.44
71.40
FAIL
54.39
Local Attention
15.82
52.98
53.39
41.46
66.63
FAIL
46.06
Sparse Trans.
17.07
63.58
59.59
44.24
71.71
FAIL
51.24
Longformer
35.63
62.85
56.89
42.22
69.71
FAIL
53.46
Linformer
35.70
53.94
52.27
38.56
76.34
FAIL
51.36
Reformer
37.27
56.10
53.40
38.07
68.50
FAIL
50.67
Sinkhorn Trans.
33.67
61.20
53.83
41.23
67.45
FAIL
$1.39
Synthesizer
36.99
61.68
54.67
41.61
69.45
FAIL.
52.88
BigBird
36.05
64.02
59.29
40.83
74.87
FAIL.
55.01
Linear Trans.
16.13
65.90
53.09
42.34
75.30
FAIL
50.55
Performer
18.01
65.40
53.82
42.77
77.05
FAIL
51.41
Task Avg (Std)
29 (9.7)
61 (4.6)
55 (2.6)
41 (1.8)
72 (3.7)
FAIL
52 (2.4)
Table Experimental results on Long-Range Arena benchmark. Best model is in boldface and sec- ond best is underlined. All models do not learn anything on Path-X task, contrary to the Pathfinder task and this is denoted by FAIL. This shows that increasing the sequence length can cause seriously difficulties for model training. We leave Path-X on this benchmark for future challengers but do not include on the Average score as has no impact on relative performance.
19


Reviews:
[{"review_id": "sTco47Y0ify", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary*: This paper proposes a new benchmarks for the host of recently-proposed transformer variants focused on efficiency and scaling to longer sequence lengths (xformers). The authors reimplement and study the performance of 10 xformers on their benchmark. Furthermore, the authors conduct a study of the memory consumption and speed of the models on their text classification benchmark.\n\n*Strengths*: Detailed and thoughtful comparison of 10 models aimed at more-or-less alleviating the same problems. The tasks span a wide range of sequential data modalities.\n\n*Weaknesses*: It’s not entirely clear to me that LRA is best-positioned as a benchmark, rather than an analysis tool; the authors themselves seem to also note this (“Hence, the results provided in this paper are not meant to be a final authoritative document on which xformer is the best”). This toolkit seems more useful to me as an analysis tool---the choice of tasks itself also reflects the benchmark’s values. For instance, a NLP researcher might care more about performance of ListOps, since it’s possible that these results have more relevance for the type of structure found in natural language.\n\nInstead of seeking to rank models (calling LRA a “benchmark” and having an “average” LRA score directly feeds into this), I’d like to see this toolkit shift toward being more customizable for individual user questions and values; similarly, this paper might take a more analytical approach to empirically studying the performance of these 10 models on this starter set of tasks, versus trying to rank them.\n\nDespite this lack of clarity of goals, I think that this toolkit and study offer useful contributions.\n\n*Recommendation*: 7 . While it remains unclear to me that this paper has a useful benchmark contribution, the analysis of existing models is valuable. Furthermore, the observation that inherent tradeoffs in performance and speed make no model the one-size-fits-all option is important; in light of this, I’d like to see the authors move towards making their toolkit better for determining what the “right” option is for a given user’s use case.\n\nComments and Questions:\n\nFigure 2: y axis label should be “Span”, not “Apan”\n\nSome abstraction is taken with respect to hyperparameters---a single set of hyperparameters is used across all models. Do you have a sense of how much this can potentially impact performance? For instance, for a single task, comparing results when you use this single set of hyperparameters vs individually tuning models for the task.\n\nDo you think that future developers of xformers should “hillcimb” on LRA?", "extra_scores": {}, "extra_text": {}}, {"review_id": "kfTSt2kwaEq", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents LongRangeArena (LRA), a new benchmark for evaluating models such as Transformers on tasks that require long-range processing. The authors present a suite of 6 tasks, each requiring access to tokens that are hundreds and even thousands of tokens apart from the target token. Their tasks share a set of important properties such as being challenging and yet simple (though not realistic, see below). The authors make a thorough comparison between 9 models designed to support efficient processing of long inputs, and show nice tradeoffs between performance, speed and memory usage. Overall, this paper makes an important contribution and despite some limitations should be accepted to ICLR.\n\nThe paper tackles an important problem: evaluating long-range transformers. There have been multiple efforts recently to design models that scale better than the quadratic time and space complexity of vanilla transformers to long sequences. Despite the large number of works, there hasn't been a standard benchmark to compare them against each other, which makes it harder to evaluate the quality of each of them. This paper addresses this problem, and presents a relatively good solution: a single, diverse benchmark that requires long-sequence processing. The new benchmark is likely to promote research on this important topic.\n\nMy main concern with this benchmark is that it doesn't involve any realistic task. The proposed tasks are either synthetic (ListOPs) or made artificially hard by forcing byte-level processing or flattening images to vectors. This means that there is no real value in performing well on these tasks, other than being a potential proxy to other tasks with similar characteristics, assuming such exist. I think relating this benchmark to real tasks would have made it much stronger. I realize that the authors made an attempt to control for different variables such as structure, but I am still not convinced that models that do well on this benchmark will be useful for anything real.\n\nOther comments:\n1. The paper would benefit from discussing the differences between the different models. A very minimal discussion is given on the last page, with a pointer to a survey paper. While I realize this is not the main focus of the paper, the different models are an important part of it, and without such even basic comparison, it is hard to appreciate the analysis presented in the results section (e.g., the last sentence on the Results on ListOps).\n2. The requirement that the tasks should allow for \"probing diverse aspects\" of the models is somewhat under-delivered. The authors discuss some probing aspects of the first 3 tasks, but do not come back to these later in the paper, which was a bit disappointing.\n\n\nMinor: \n1. Adding random/majority baselines to table 1 would make it easier to appreciate the results\n2. Results on ListOps: \"the inductive bias of the xformer *model* plays \" -> *models*", "extra_scores": {}, "extra_text": {}}, {"review_id": "jGDbOZ41kw1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes benchmark tasks for the new crop of efficient Transformer models, an evaluation suite which the authors dub the \"Long Range Arena.\" These tasks are selected to require hierarchical modeling of long-distance dependencies, and specifically to *exclude* pre-training or a need for pre-trained models (making many of the NLP tasks which motivate these papers unsuitable).  Many of the tasks have a somewhat artificial flavor: purely artificial (ListOps), treating images as sequences of pixels, or byte-level classification rather than word level or word piece level. Regardless, the paper shows interesting variation in the tasks across the different models, particularly on ListOps. BigBird seems to work the best out of the box, but the authors caution against reading too much into this, as they could not extensively tune hyperparameters on each of these models.\n\nI think this paper addresses a desperate need in the literature, which is a way of making sense of all these Transformer variants that have been proposed. In this sense, I like what it's doing. My main reservations concern the somewhat artificial nature of the tasks and the generalizability of the results.\n\nNATURE OF THE TASKS: This paper is probably wise to not conflate distinctions in architecture with distinctions in pre-training, and exploring both of these factors is too much and too complex for a single piece of work.  Papers such as the Longformer have primarily evaluated on NLP tasks like coreference and question answering; however, the \"long-range\" abilities on these datasets only add a few percentage points on the given metrics, and they rely on pre-training to be successful. However, I feel like the distribution of long-range effects in natural language is very different than the distribution of effects here: you have effects like entity coreference that aren't captured well by these tasks (these don't matter much for text classification and are absent from ListOps and the image tasks). So I'm not fully convinced that a model that does better on this benchmark suite would naturally do better on problems we really care about.\n\nThe tasks here do feature some nice effects: explicit hierarchy (ListOps), local composition (the byte-level tasks), and regular long-range correspondences (the image tasks). It's not clear to me how critical this last category is, as the 2D layout is an inductive bias that would almost be built into the architecture, but that's a separate discussion. However, the point remains that this benchmark does make choices about what is and isn't important, *particular* when you average across the datasets (ascribing each one equal weight), and I can imagine another benchmark being proposed focusing on a different set of effects. So I'm not sure how to judge this benchmark suite versus others that might exist.\n\nRESULTS: This leads to my second point, about the results. I do like how the paper couches the results, explains the findings, and avoids overclaiming. I agree with the authors that, with some tuning, many of these approaches could possibly substantially better at various of these tasks. We might also see that some of these approaches are fundamentally not well-suited to these tasks.  But again, given the somewhat artificial nature of the tasks, what are we likely to actually learn? I foresee a future with ~10 papers hillclimbing on this benchmark, and I'm not convinced that those architectures will be any better than these when applied to new settings, LM pre-training, etc.\n\nThat said, the differences on ListOps are pretty interesting.  I wonder if Path-X has fundamental identifiability issues due to its scale: for example, there are now more geometries the model can learn from the sequence, and possibly there are somehow just too many of these for learning to effectively figure out which is correct. (I don't have a more formal argument, but it feels like a case of exponential numbers of hypotheses outstripping the size of the dataset as things scale up.)\n\nAll this said, I tend to come down positive on this paper: I think it brings some clarity to this space and will be a very useful starting point for others in the literature. And I don't think this meta-benchmark is necessarily more flawed than others like GLUE.  But nor do I think it's perfect.", "extra_scores": {}, "extra_text": {}}, {"review_id": "dQQCrDPfVca", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a suite of long sequence processing tasks to benchmark efficient transformer variants in terms of their accuracy/speed tradeoff. The benchmark tasks are constructed following six desiderta: generality, simplicity, challenging, long inputs, probing diverse aspects, and accessibility, resulting in six tasks mostly in the form of sequence classification where the sequences range from 1k to 16k tokens. Ten existing efficient transformer models are evaluated on these tasks and their accuracy/speed tradeoffs are compared on a relatively fair basis.\n\nPros:\n1. This work is well motivated and timed. Given the attention efficient transformer variants recently attracted, it is important to have a common benchmark for comparability of results.\n2. The diversity of the proposed tasks (in terms of types of tasks and modalities covered) enables a full grasp of the performance among different types of tasks, especially exposing the weaknesses. For example, it is interesting to see that kernel-based methods fail on tasks requiring hierarchical structures.\n3. This benchmark is accessible for academia : from table 2 it takes at most 10G GPU memory for a normal transformer.\n\nCons:\n1. For generality this suite of tasks only considered transformer encoders, but not the autoregressive decoding process, so the tasks here are mostly sequence classification tasks only. I think it is a limitation of the benchmark here. For example, all efficient transformers fail on path-x, but that don't mean that path-x is not useful. It is the limitation of the efficient transformer method itself if it does not support efficient autoregressive decoding.\n2. While the benchmark proposed here considers image and text, it'd be interesting to add audio processing as well, such as hot word detection.\n3. Pro 3 might be a downside of this benchmark as well, the tasks here can be mostly solved by a normal transformer while one reason we want efficient transformer is to apply it to extreme situations where normal transformers are infeasible, and there seems to be more focus on inference speed but not training/inference memory.\n4. Related to Con 3, it might be nice to separate out training and inference (memory/speed).\n\nQuestion for the authors:\nfor speed and peak memory, do you only measure at inference time? Is there any evaluation of training statistics like memory and time to convergence?\n\nTypo:\nThis is the most comprehensive and extensives -> extensive\n\nOverall, I think this would become an important bechmark for comparing efficient transformer approaches, and I would recommend its acceptance if the issues I mentioned above can be mitigated.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper attempts at providing a general benchmark for evaluating/analysis of long range transformer models, consisting of a 6 evaluation tasks. The main goal of the paper is to remove conflating factors such as pretraining from model performance and keeping the benchmark accessible. All reviewers agreed that these are important positive aspects of the paper and the presented analysis/results are useful.  

While reviewers generally feel positive about the work, there are some critical concerns on how useful this benchmark is in practice, how generalizable are the results, and whether the benchmark is good at what is intended for. For example, the vanilla Transformer model performs very well on all the proposed tasks, making me question on what we can actually learn about long range dependencies through this benchmark. In addition, most tasks are synthetic and all models fail on 1 of the 6 proposed tasks. 

Therefore, I think LRA should be viewed more as a tool for analysis, or as authors nicely put in their response, it should be viewed as a means to "encourage hypothesis driven research instead of hillclimbing or SOTA chasing.". 
During discussion period with reviewers, while acknowledging the above-mentioned issues, this strength was highlighted as a valuable contribution. Therefore, given the general positive sentiment about the work, I'd recommend accept.
Decision: Accept (Poster)