Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Authors: Ofir Press
Noah Smith
Mike Lewis
Venue: ICLR 2022 Poster (2022)

Abstract: Since the introduction of the transformer model by Vaswani et al. (2017), a fundamental question has yet to be answered: how does a model achieve extrapolation at inference time for sequences that are longer than it saw during training? We first show that extrapolation can be enabled by simply changing the position representation method, though we find that current methods do not allow for efficient extrapolation. We therefore introduce a simpler and more efficient position method, Attention with Linear Biases (ALiBi). ALiBi does not add positional embeddings to word embeddings;  instead, it biases query-key attention scores with a penalty that is proportional to their distance. We show that this method trains a 1.3 billion parameter model on input sequences of length 1024 that extrapolates to input sequences of length 2048, achieving the same perplexity as a sinusoidal position embedding model trained on inputs of length 2048 but training 11% faster and using 11% less memory. ALiBi's inductive bias towards recency also leads it to outperform multiple strong position methods on the WikiText-103 benchmark.

Published as a conference paper at ICLR 2022
TRAIN SHORT, TEST LONG: ATTENTION WITH LINEAR BIASES ENABLES INPUT LENGTH EXTRAPOLATION
Ofir Press 1.2
Noah A. Smith! 1.3
Mike Lewis?
1 Paul G. Allen School of Computer Science & Engineering, University of Washington
Facebook AI Research
3 Allen Institute for AI of efirpecs. washington. edu
ABSTRACT
Since the introduction of the transformer model by Vaswani al. (2017), funda- mental question has yet to be answered: how does a model achieve extrapolation at inference time for sequences that are longer than it saw during training? We first show that extrapolation can be enabled by simply changing the position represen- tation method, though we find that current methods do not allow for efficient ex- trapolation. We therefore introduce a simpler and more efficient position method, Attention with Linear Biases (ALiBi). ALiBi does not add positional embeddings to word embeddings; instead, it biases query-key attention scores with penalty that is proportional to their distance. We show that this method trains 1.3 bil- lion parameter model on input sequences of length 1024 that extrapolates to input sequences length 2048, achieving the same perplexity as a sinusoidal position embedding model trained inputs of length 2048 but training 11% faster and using 11% less memory. ALiBi's inductive bias towards recency also leads to outperform multiple strong position methods on the WikiText-100 benchmark
INTRODUCTION
When constructing a transformer-based language model, major design decision is the length of training sequences, denoted herein, which has date been equivalent the length of inference sequences. More context, achieved by larger L, improves predictions at inference time. But longer sequences are more expensive to train on
Before transformers, RNN language models were trained on shorter- L sequences and assumed to generalize longer contexts inference time (Mikolov al., [010; Mikolov & Zweig 2012 Zaremba et al. 2014). Vaswani et al. (2017), introducing the transformer, speculated that it "may [...] extrapolate to sequence lengths longer than the ones encountered during training." " We define extrapolation as model's ability continue performing well the number of input tokens during validation increases beyond the number tokens on which the the model was trained. We find that transformer language models (LMs) that sinusoidal position embeddings have very weak extrapolation abilities; see Figure]
We demonstrate that this failure to extrapolate is caused by the position embedding method. As shown in Figure [ recent alternatives to the original sinusoidal position method (Su et al. 202T
Raffel al. 2020) have improved extrapolation. However, the better of these, the T5 bias, siderably slower than the sinusoidal approach and uses extra memory and parameters (Figure 2
We therefore introduce Attention with Linear Biases (ALiBi) to facilitate efficient extrapolation. ALiBi negatively biases attention scores with a linearly decreasing penalty proportional to the dis- tance between the relevant key and query. Our simple approach eliminates position embeddings.
'Code & models: https::////....... pettps://githti com/ of i wwtrrrpeeaattt attent lon_with___t 1 ineiinnrraa _biaeerr_basss FFigure in the appendix plots training speed, in words per second, against T..

---

Published as a conference paper at ICLR 2022
Extrapolation for Models Trained on 512 Tokens
Extrapolation for
Models Trained on 1024 Tokens
55
55
Sinusoidal
Rotary
Perplexity (-)
45
Perplexity (+)
45
Sinusoidal
155 Bias
Rotary
ALiBi
35
35
T5 Bias
ALiBi
25
25
15
512
4000 8000
12000
16000
1024
4000
8000
12000
16000
Inference Input Tokens
Inference Input Tokens
Figure 1: Extrapolation: as the (validation-set') input sequence gets longer (z-axis), current po- sition methods (sinusoidal, rotary, and T5) show degraded perplexity (y-axis, lower is better). but our method (33 does not. Models were trained on WikiTeext-O0, 103 with sequences L = 512 (left) or L 1,024 (right) tokens. '55 ran out of memory our 32GB GPU. For more detail on exact perplexities and runtimes, see Tables anddd in the appendix.
Compared to sinusoidal model trained on the same input length, our method requires no additional runtime or parameters and incurs negligible (0-0.7%) memory increase. ALiBi can be imple- mented by changing only a few lines of existing transformer code.
Using ALiBi, transformer LM can be trained short-L sequences and therefore at much lower cost, and it can still be reliably applied to long sequences at runtime. For example, a 1.3 billion parameter LM trained on L 1024 tokens with ALiBi achieves the same perplexity sinusoidal model trained on L = 2048 when both are tested on sequences of 2048 tokens, even though our model is 11% faster and uses 1 % less memory.
Though performance peaks at around two times the number of tokens that the model was trained on, ALiBi maintains strong performance even sequences of length 10,000. In recently explored set- tings where NLP training examples are given as context to an LM (Brown et al. 2220)). our approach will allow exposure to more examples. Additionally, it enables generation of longer outputs.
2 CURRENT APPROACHES Do NOT EXTRAPOLATE EFFICIENTLY
We show for the first time that the sinusoidal position method, which technically should be able to extrapolate, in practice has very limited extrapolation capabilities. Though the rotary position method improves over the sinusoidal one, it still does not achieve satisfying results. Holding every- thing else constant, we are the first to observe that the T5 bias method leads to better extrapolation than either of these, and we conclude that extrapolation ability depends heavily on the position embedding. Unfortunately, the T5 bias is computationally costly (Figure
2.1 BACKGROUND AND EXPERIMENTAL SETUP
transformer LM receives list of tokens and outputs probability distribution representing its prediction for the next token. We call the input list the current input subsequence since the inputs to language models are typically subsequences from (much longer) training or evaluation sequences. During both training and perplexity evaluation (i.e., scoring a fixed sequence), many predictions can be calculated at once: this done using a "causal mask" that ensures each position's 's prediction is influenced only by tokens to its left. Let L be the length of each input subsequence during training; it includes predictions, which on average have access to L+1 tokens of (left) context. To explore a model's 's extrapolation abilities, considered at evaluation time. When L differs between inference and training, we use to refer to the length of subsequences during training and Traati refer their length at validation.
2
interested in cases where sequences of length L > L are
2

---

Published as a conference paper at ICLR 2022
30k
Truining Speed
100k
Inference Speed
30
Training Memory
Sinnsoidal
20k
75k
20
Rokary 'T Blas
wPs
50k
10k-
25k
GB
10
ALiBi
0
Ok
512
1024
3072
1024
3072
512
1024 Input Length
3072
Input Length
Input Length
comparison of batched training, inference speed and memory use of the sinusoidal, rotary, T5 bias, and our ALiBi position methods. The speed differences between our method and the sinusoidal are within I % during training and 3% for inference, which insignificant on our hardware. ALiBi uses 100MB of extra memory when training on input lengths 1024 and 3072 in this setting. Memory usage is lower in all approaches when training on 3072 tokens (compared to 1024) since we break batches into multiple updates. See Table in the appendix for exact numbers.
Figure 2:
Nonoverlapping Inference To train evaluate sequence longer than L tokens, it is typical to segment the sequence into L-length subsequences and train on or evaluate them independently. Unless otherwise stated, use nonoverlapping inference to report perplexity scores.
Extrapolation During Inference Formally, the functions that define transformer layer are ag- nostic to input length they map from some arbitrary, unfixed number of input vectors to the same number of output vectors. When transformers are applied to data that is inherently sequential, like text, positional information injected into the inputs in various ways.
Vaswani et al. (217)) discussed two options for embedding positions into vectors to be added to word embeddings: learning embeddings for specific positions and unlcarned sinusoidal embeddings. They observed similar performance between these two but preferred the sinusoidal approach, which they argued might extrapolate to longer input sequences during inference. We find that this model cannot extrapolate to more than few dozen tokens beyond L
Experiment Setup We first test the extrapolation abilities of various position methods on the WikiText-103 corpus (Merity et al., 2016) using the transformer language model of Baevski & Auli 2018). We use this model because of its prominent role in recent language modeling develop- ments (Khandelwal et al. 2020, Press et al. 202T). ). The training set is about 103 million tokens from English Wikipedia (half gigabyte). The model has 16 transformer layers of dimension 1024, with 8 heads, and feedforward inner dimension of 4096. This model ties the word embedding and softmax matrices (Press& & Wolf 2017 Inan et al. 2017). In our experiments, other than varying the position method and training subsequence length, we modify no other hyperparameters, including the random seed and number of training epochs (205).
2.2 MEASURING EXTRAPOLATION
Sinusoidal Position Embeddings Sinusoidal position embeddings (Vaswani et al. 2017 $3.5) are constant, non-learned vectors that are added to token embeddings on input to the first layer of the transformer. They are frequently used in transformer language modeling (Baevski Auli 2018 Lewis et al., 202T] and machine translation (Vaswani et al. 2017 Out et 2018) models. We first consider the unmodified model of Baevski & Auli (2018). which uses sinusoidal position embeddings, and train it on L = 512 tokens; we then run inference with it on the validation set on L + k tokens, with l ranging from to 15,000. Figure (left) and the corresponding Table P (in the appendix) show that while the model improves perplexity up to h = 20, performance stops improving and stays steady from & = 20 to k = 50 and then begins degrading. Similar results are obtained for model trained with L 1024 tokens (Figure (right) and Table 3 in the appendix). That model improves for up to Lyatid = L + 50 tokens, after which performance declines.
'These include the embedding lookup, feedforward sublayer, and softmax layer, which act independently on vector inputs, well as the attention sublayers, whose parameters do not depend on input length (and which must handle variahle-length inputs, e.g., due to causal masking).
"The learned positional embedding approach does not have way to encode positions greater than L; it therefore has ability to extrapolate.
3

---

Published as a conference paper at ICLR 2022
Rotary Position Embeddings The rotary method was introduced by Su et al. (2021) and has recently been popularized by the open source GPT-3 (Brown et al. 2020) implementation GPT- J (Wang& & Komatsuzaki 202I). Instead of adding sinusoidal embeddings at the bottom of the transformer, they multiply the keys and queries of every attention layer by sinusoidal embeddings.
Unlike the sinusoidal or learned positional embedding approach, the rotary method injects position information into the model at every layer, not just at the initial one. In addition, it adds no position information to the values of the self-attention sublayer. The output of self-attention sublayer is a linearly transformed, weighted sum of the input value vectors; therefore, by not inserting position information into the values, the outputs of each transformer-layer contain no explicit position infor- mation. We suspect that this segregation of position information may be beneficial for extrapolation, and we draw inspiration from it in the design of our method ($3)
We apply the rotary position embedding method to our Baevski & Auli baseline + The perplexity results (Figure [ and Appendix Tables and
and (3) are better than the sinusoidal approach: the model with L = 512 (L 1024) improves perplexity with up to k = 200 (k 100) more tokens than it saw during training, but this comes at the cost of slower training and inference (Figure 2
T5 Bias Though most models use trained or sinusoidal position embeddings. the T5 model of Rat- fel et al. (2020) uses relative position method (Shaw et al. 2018: Huang et al., 2019) that adds no position information to word embeddings (as in the previous method). Instead, it modifies the way attention values are computed. We refer to this as the "T5 bias" method To compute attention values in the unmodified transformer, we compute the dot product of every query with every rele- vant key and then softmax these attention values. In this method, we compute the attention values as before, but then we add learned, shared bias to each query-key score that is dependent on just the distance between the query and key. Therefore, all query-key scores where the query and key distance are zero (i.e., the query and key represent the same token) get specific learned bias, all scores where the query and key are one word away get a different learned bias, and so on, up to a certain point, from where multiple different distances share the same learned bias (which might be beneficial for extrapolation). As in the rotary method, the T5 bias injects position information into the model at every layer and integrates no explicit position information into the self-attention value vectors.
Raffel al. 2020 propose that the T5 bias may allow extrapolation, but they did not report exper- iments testing this. Here, we show that the T5 bias does allow language models to extrapolate. We do this by again modifying the Baevski & Auli model, this time to insert the T5 bias into it
As Figure D shows, the T5 bias improves perplexity with longer sequences than the ones it was trained on, i.e., k- = 600 (A: = 800) extra tokens for a model trained on = 512 (L = 1024) input tokens. Unfortunately, this impressive performance comes at cost: training is at least twice as slow as with the sinusoidal model. Therefore, this model's extrapolation ability provides no efficiency advantage. For example, to do inference 1024 tokens, we could either train the sinusoidal model with L= = 1024 or train the T5 bias model on L = 512 tokens and extrapolate to 1024 for inference. However, the L 1024 sinusoidal model runs at 28.5k words per second (WPS), while the L 512 T5 bias model runs at 14.4k WPS (Appendix Table []. so there is no speedup when training on shorter sequences with this method 8
*Our rotary method implementation is based on the code in https:///ithubb www. Yom/JunnYu/ RoFormer-pytorch which is linked from the official repository of [Su al.] (202T): (https: //github.com/ ZhuiyiTechno
After we finished rumning our experiments with the rotary method, we were informed that the runtime of the code linked above could be optimized, making it only 2% slower than the sinusoidal approach. This optimization would not change extrapolation performance.
'This method is similar to the one used in Parikh et al. (2016] Equation 7).
"Our T5 bias implementation is based on the one used in HuggingFace Transformers Wolf et al. 2020
which in turn based on the official Mesh Tensorflow T5 code.
NNaaang et al. 2021 benchmarked the T5 bias as being just 8.7% slower than the sinusoidal approach: thus, while always incurring a runtime penalty, this method's 'ss runtime could be faster depending on the choice of hardware and software frameworks used. Narang et al. used the Tensorflow T5 library running TPUs, while used the PyTorch Fairseq library running on GPUs.
4

---

Published as a conference paper at ICLR 2022
+ -2 -1
m
Rc
-2
-2
Figure 3: When computing attention scores for each head, our linearly biased attention method, AL- iBi, adds constant bias (right) to each attention score (q kj. left). As in the unmodified attention sublayer, the softmax function then applied to these scores, and the rest of the computation is un- modified. m is a head-specific scalar that is set and not learned throughout training. We show that our method for setting m values generalizes to multiple text domains, models and training compute budgets. When using ALiBi, we do not add positional embeddings at the bottom of the network.
ATTENTION WITH LINEAR BIASES (ALiBI)
In the transformer model of Vaswani al. (017)) position embeddings added to the word embeddings at the bottom of the network. For an input subsequence of length L, the attention sublayer computes the attention scores for the ith query qi E R xd, (1 i L) in each head, given the first i keys K E Rixd, where d is the head dimension:
softmax (oftmax(qqKK' (q
These attention scores then multiplied by the values to return the output of the attention sublayer
When using ALiBi, we do not add position embeddings at any point in the network. The only modification apply is after the query-key dot product, where we add static, non-learned bias
softmax(q.K3 + m-
(- m- 1),..., ..., 0),
where scalar m is a head-specific slope fixed before training. Figure offers a visualization.
For models with heads, the slopes that used the geometric sequence: 2 22, 28- For models that require 16 heads, we interpolate those 8 slopes by geometrically averaging every consecutive pair, resulting in the geometric sequence that starts at v2 and has the ratio of v 20.3: ST: 225:.... ..... 2 In general, for heads, our set of slopes is the geometric sequence that starts at and uses that same value as its ratio.
In $4 we observe that this set of slopes works wide variety of text domains and model sizes. Therefore, we do not believe that it is necessary to tune these slope values every time a new model trained new dataset. This makes
method similar to the sinusoidal approach, where the hyperparameters (the start and end the geometric progression of wavelengths) were set once by Vaswani al. (2017) and then reused in different models of different sizes on different datasets.
ALiBi has an inductive bias towards recency; it penalizes attention scores between distant query-key pairs, with the penalty increasing as the distance between key and query grows. The different heads increase their penalties at different rates, depending on the slope magnitude.
We initially experimented with making the slopes trainable, but this did not yield strong extrapola- tion results A brief manual exploration of around ten slope sets led us to discover the set of slopes that we finally picked. Our main insight from this exploration is that the slope sets that work best are those with slopes in the (0, 1) range, with the slopes' density increasing also found our method to robust to slope choice. Even randomly sampling from the exponential distribution worked well in some cases (although that method had high variance).
we get closer to O. We
Since ALiBi is a relative position method, we add position information at every layer to the keys and queries but not to the values, as is done in the T5 bias and rotary methods. We hypothesize that these properties might be beneficial for extrapolation.
FFor simplicity omit the key, query, value and final output projections, dropout, and the scaling factor. 1he ALiBi hias is not multiplied by the Vdg scaling factor from Equation of Vaswani et al. 2017) In experiments, trainable slopes also slowed down the training speed by 3%.
5

---

Published as a conference paper at ICLR 2022
Implementation. ALiBi is easy to implement, with all changes accomplished in few lines of code. We implement it by modifying the mask matrix by adding the linear biases to it (in practice, when training transformer LM, query q, attends only to keys 1 to i; this is implemented by adding a mask matrix to the query-key dot product before the softmax operation is applied). This means that there is no runtime penalty when using our method since we add operations the network.
Compared to the sinusoidal model trained on the same input lengths, AliBi incurs a memory increase (up 100MB in some of our experiments); in the unmodified transformer, the mask is of size Lx L; when using ALiBi, the mask is a slightly larger n. x L x L (where n is the number of heads) since the linear biases added for each head uses a different slope. But, as we show, ALiBi enables training on much smaller sequences while still achieving (and occasionally surpassing) results obtained using sinusoidal embeddings on longer sequences, which saves multiple gigabytes of memory.
RESULTS
We first show that WikiText103 ALiBi is efficient and enables training models with short input subsequences that outperform strong baselines even when the ALiBi models extrapolate to more than six times the number of tokens that they were trained our method (the set of slopes) that worked on WikiText-103 and show that - with no modification they provide strong results on dataset in a very different domain: books. Finally, we show that a 1.3B parameter model trained with AliBi on a much larger (461 GB) dataset with much more compute provides a superior alternative to the sinusoidal method since achieves similar perplexity scores while running faster and using less memory (since it is trained on shorter inputs).
We then take the same hyperparameters for
While multiple alternatives to the position methods presented in Vaswani et al., (2017) have been proposed, few have been adopted in large (1B or more parameter) LMs since that setting is much more challenging than the smaller scale experiments. GPT-3 and Jurassic-1 (Lieber al.] (02T) use the learned position embedding method from Vaswani et al., and GPT-J uses the rotary method. Our results on the 1.3B parameter model show our method's ability to generalize to larger models, dataset sizes and training durations without retuning the hyperparameter.
4.1 RESULTS ON WIKITEXT-IO1 103 AND TORONTO BOOKCORPUS
ALiBi Extrapolating on WikiText-10
21.0
. ALiBi, L 512
20.5
Simusidal,, L= 512
20.0
-A- ALiBi, L = 1024
19.5
Simuseidal, L= 1024
Perlexity
19.0
ALiBi I. 1536
18.5
Simuscidal,
18.0
L
Simusodal, L 2048
17.5
ALiBi, 1. = 3072
17.0
5i2
1024
1536
2048
3072
Simssiddal, L= 3072
Validation Input Length (Lowtt))
Figure 4: ALiBi models trained and evaluated on varying sequence lengths on the WikiText-103 validation set and the sinusoidal baseline (not evaluated longer sequences). All of our models outperform the sinusoidal ones even when trained on fewer tokens. Appendix Table 5 has exact perplexities, more ALiBi models (trained on fewer tokens), and results for rotary and T5 bias models.
We first develop our method on the WikiText- 103 corpus (Merity et al. 2016), replacing the sinu- soidal position embeddings in the language model of Baevski & Auli 2018) with ALiBi.
Figure (and the corresponding Appendix T) show our results for models trained with varying numbers of input subsequence tokens (L), extrapolating to longer subsequence lengths on the valida- tion dataset. Our first observation is that, without extrapolation, for every L, our models outperform those using the sinusoidal method, sometimes by significant amount. For example, the Baevski & Auli model achieves 18.67+0.24 (std. dev.) perplexity when trained with L = 3072 imput tokens, but our I. 3072 model achieves 17.60 perplexity (when both models evaluate with Tawalis 3072).

---

Published as a conference paper at ICLR 2022
Our second observation is that of our models can extrapolate, and they obtain improved perplexity scores when handling more tokens than they observed during training. For example, our model trained on 512 tokens (which achieves 19.73 perplexity when evaluating subsequences of length 512 in the development set) achieves a perplexity score of 18.40 on the development set when extrapolating to subsequences of length 3072. Surprisingly, this surpasses the score that the I. = 3072 sinusoidal model obtains on the development set by a statistically significant margin. Note that all our models trained on L 512 L 2048 outperform the sinusoidal baseline trained on L = 3072 when extrapolating to Liaiid = 3072 even though those models all take much less time to train since they train on shorter subsequences (Appendix Figure X compares training speed to perplexity for these models)! The L 512 model is 1.84 times faster to train and yet still outperforms the L 3072 sinusoidal model when extrapolating to Limiid = 3072. addition, training the I. 3072 sinusoidal model requires GPU with more than 16 GB of memory to fit the large attention matrices, which our L = 512 outperforms even though it can be trained on a GPU with much less memory due to much smaller attention matrices.
Additionally, Table 5 (in the appendix) also shows that, for Ls of 1024 and 3072, our method per- forms better than the rotary and T5 bias models even when Lialid L (i.e., no extrapolation is occurring). Figure (and the corresponding Appendix Tables 2 and 33] more broadly explore our method vs. the other position methods. They show that the T5 bias (the best of the baselines) im- proves perplexity until Lyaii is around 21. but on the WikiTexx- 103 dataset our method continually improves perplexity until least around 3L, with the L 512 model improving perplexity even when Lialid exceeds 12k tokens. Even when unable to improve perplexity given longer sequences, ALiBi always maintains strong performance as more tokens are added.
Appendix T shows that our results on the validation set also transfer to the test set of WikiText- 103. Currently, almost all models that present results on WikiText- 103 use sliding window evalu- ation (defined in $B) to compute perplexities. We apply that method to our (and to the sinusoidal, rotary and T5 bias) models in Appendix Table 7 We find that our I. = 3072 model surpasses the performance of Transformer-XI. (Dai et al. 2019), the Sandwich (Press et al., 2020), and Short- former (Press et al., 2021) models. Our results are similar to the ones obtained with staged train- ing (Press et al. 2021) but fall short of results obtained by Routing Transformer (Roy et al. (0200) and KNN-LM whhadeelwal et al. 2020). The methods used in those models are orthogonal to ours, and we hypothesize that combining them with ours might lead to even larger performance increases.
After developing our method on WikiText- 103, in Appendix Section A.3] we run one set of experi- ments different domain (books) using a similar model architecture and without modifying any of the ALiBi hyperparameters (the slopes) and show that our results fully transfer to this new do- main. Our models are able to both surpass the sinusoidal baseline when not extrapolating while also outperforming it when extrapolating to longer sequences.
4.2 RESULTS ON THE CC100+ROBERTA CORPUS
Our final set of experiments investigates whether ALiBi transfers to a larger model trained with larger computational budget on a larger dataset than the ones we previously used. We show that our method achieves strong results in this more challenging setting, obtaining similar performance the sinusoidal baseline while using significantly less memory, since we train on shorter subsequences.
The dataset we choose is a combination of the datasets used train the RoBERTa (Liu et al., 2019) implementation of BERT (Devlin et al.] 2019) and the English part of the CC-100 corpus intro- duced Conneau et al. (2200) for a total of 461 GB. The RoBERTa training corpus-i.e.. the Toronto Book Corpus (Zhu et al. 2015). English Wikipedia, CC-News (Nagel) 2016). OpenWeb- Text (Gokaslan & Cohen 2019) and Stories (Trinh& & Le 2018)) is 161 gigabytes, and the English part of the CC-100 corpus is 300 gigabytes. The validation set contains 649K tokens.
Our models for this dataset have 25 transformer layers with 16 heads and a dimension of 2048, with an 8192 hidden dimension of the feedforward sublayers. These models have 1.3B parameters. We train our models for one epoch, which is 50k updates on 128 V100 GPUs.
In Figure 5 (left), we compare the validation perplexity for Lvalid 1024 throughout the training process for an ALiBi model trained with L = 512 compared to the sinusoidal model trained with 7. 1024. Since our model is trained on shorter sequences, it is 7% faster and uses 1.6 GB less
7

---

Published as a conference paper at ICLR 2022
Validation Parplexity Through Training
Validation Perplexity Through Training
with Liaid = 1024
with Lyndid 2048
11.0
11.0
10.5
10.5
Perplexity (-)
10.0
Perplexity
10.0
9.5
Simusoidal
9.5
Sinusoidal
L= 1024
L= 2048
9.0
ALiB;
9.0-
ALiBi
L 512
1024
8.5
8.5
0
1000 2000 3000 4000 5000 6000
1000 2000
3000 4000 5000 6000
Training Time (GPU Hours)
Training Time (GPU Hours)
ALiBi model trained on 512 (1024) and evaluated on 1024 (2048) tokens during training, compared to the sinusoidal baseline trained on 1024 (2048) to- kens. The ALiBi models obtain strong results even though they use 6%- 11% less memory since they train on shorter sequences. Appendix Table [] shows memory use and end-of-training perplexities.
Figure 5: On the left (right), a
memory. We halt training of the sinusoidal baseline when our model reaches the end of its training (one epoch). At that time, our model is just 0.06 perplexity away from the baseline even though was trained sequences that half the length of those the baseline used and requires less memory.
In Figure 5 (right), results become even more impressive, showing that our model trained on L = 1024 outperforms by 0.09 perplexity the sinusoidal model trained L. 2048 (when evaluating with Lvalt = 2048) even though our model uses 3.1 GB less memory. Our model maintains a lead in perplexity over the sinusoidal model during the entire training process. By sampling five evenly distributed points across the training process, we compute that our L = 1024 model reaches a given perplexity value, on average, 11% faster than the sinusoidal model does.
Since our models in these comparisons use much less memory, they allow for stacking more layers, which would further improve performance (with negligible, if any, runtime cost). To keep our experiments as straightforward as possible, however, we do not add layers to our models.
Appendix Table 12 presents additional results comparing our models to the sinusoidal baseline when both are trained on the same L, showing that ALiBi performs similarly to the sinusoidal baseline when not extrapolating. This contrasts with the results presented on the smaller datasets, where ALiBi consistently outperforms other position methods even when not extrapolating, suggesting that ALiBi's inductive bias provides additional benefits for lower-resource language modeling.
Extrapolation, L 512 on CC100+ RoBERT
Extrapolation L 1024 10.2
on CCI00+RoBERTa
10.2
Sinusoidal
Perplexity (-)
9.8
Perplexity (+)
9.8
ALiBi
9.4
9.4
9.0
Sinusoidal
ALiBi
8.6
8.6
512
2500
5000
7500
10000
1024
2500
5000
7500
10000
Validation Input Length (Lowiia)
Validation Input Length (Iauts)
Figure 6: The ALiBi and sinusoidal models (with both == and 1024) trained for 50k updates (1 epoch) the CC100+RoBERTA corpus, extrapolating on the validation set. ALiBi achieves the best results at around 2T. but maintains strong performance even up to 10000 tokens in these experiments.
Figure 6 shows that models trained L 512 and L 1024 achieve the best results when extrapolating to about double the tokens that they were trained on. Specifically, the L = 512 model (that obtains 9.79 perplexity when T 512) achieves its best score (9.3) when extrapolating to
8

---

Published as a conference paper at ICLR 2022
1012 tokens, and the L = 1024 model (that obtains 9.16 perplexity when Lvalid 1024) achieves its best score (8.9) when extrapolating to 2024 tokens.
One possible explanation is that the subsequences the model observes during training up to L tokens long. When performing inference on subsequences of length 2L, half of the subsequences the model consumes long the examples seen during training. When inference is performed on subsequences length 2L + or longer, less than half of the predictions the model makes are on subsequences of lengths seen during training, and that might degrade performance.
The sinusoidal model cannot extrapolate at all in this setting, with its performance degrading for both the I, 512 and 1024 models as soon one token more than L is added during evaluation.
In Appendix B we find that ALiBi's edge over sinusoidal embeddings is largely explained by its improved avoidance of the early token curse. We posit that future work building on ALiBi might achieve further gains by more efficiently exploiting longer histories.
5 RELATED WORK
In parallel with work, Wennberg & Henter (2021) introduce relative position method that, like our method, adds a bias to attention scores that is a function of the distance between the key and query elements. Unlike ALiBi method, which uses non-learned linear function, their method uses a radial-basis function, with multiple trainable parameters (in our experiments, this led to a slight decrease in runtime). In addition, they present experiments on text classification, not on language modeling. They do not explore extrapolation. The Distance Aware Transformer (Wu al., 2021) multiplies attention scores by a bias that a function of the distance between the key and query. This function uses different, learned parameter in every head. They show results only on text classification. In our experiments (not presented), multiplying attention scores by the bias (instead of adding, in ALiBi) degraded performance.
Transformer-X XI. (Dai al. 2019) presented a language model that uses a cache and can attend to more tokens during inference than it was trained on (by increasing the length of the cache). However, this work presents results only where output length is limited to the L (the training length). and their relative position method is very slow (Press al. 2021). The Longformer (Beltagy et al.) 2020) adapts models trained on shorter sequences to document-level tasks. However, to achieve this they had to partially train their models on longer sequences. Our ALiBi method enables extrapolation without any additional training on longer sequences.
To our knowledge, extrapolation has not been previously explored in transformer language model- ing, but it has been investigated previously and concurrently with transformers on other tasks, such as machine translation (Rosendahl et 2019 Neishi & Yoshinagal 2019 Newman et al. 2020 Kiyono et al. 2021), sequence-tor perueecce-oosequenne to-seueence models trained on an artificial dataset (Hupkes et al. 2020). pretrained sequence-to to--euueceee models tested on arithmetic tasks (Nogueira et al.) [202]] Appendix C), models trained with reinforcement learning (Lampinen et al. (2021), image, speech recognition, and machine translation models (Likhomanenko al. 2021). and protein structure prediction (Jumper et al. 2021 Appendix 1.5).
6 CONCLUSION
We showed that the sinusoidal position embedding approach does not enable transformers extrap- olate to inputs longer than the ones they were trained on. We then established that extrapolation in transformers can be enabled by just changing the position method. We showed that our ALiBi method offers an extremely simple replacement for existing position approaches and allow models to extrapolate. addition, when not extrapolating, our method achieves either better perplexity than the sinusoidal method (in models smaller than IB parameters, trained on less data) similar perplexity (in larger, billion parameter models trained on much more data). ALiBi is simple to im- plement and does not slow down runtime require extra parameters (but does occasionally require a negligible amount of extra memory). Using our method, we sped up the training of a 1.3 billion parameter model evaluated the same input sequence length GPT-3 (2048).

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We thank Tim Dettmers, Gabriel Ilharco, Jungo Kasai, Hao Peng. Sewon Min, Sofia Serrano, Sam Shleifer, Luke Zettlemoyer, Julian Michael, Nikolaos Pappas, Yizhong Wang, and the anonymous reviewers for their valuable feedback and fruitful discussions.
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. CoRR, abs/1809. 10853, 2018. URL.tt::// http://axxxv...g/bbb.assse
www.:/oggg/aaaaa..ee... absp/80s/1110/// 10853
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv:2004.05150, 2020.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. doi: 10. /125://///220.allmmaaaiiia
URL http:// doi. aed...orooo 20tpg/1...163/111222222 acl ma in. 747
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer-XL XL: Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 2978 2988,, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10. 1..1866/vvll 1/P19- 1285. URL
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies. Volume (Long and Short Papers), pp. 4171-4186, Minncapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/1//N19-14233 URL https:
7/ /www. www.wacllee www.ab..l..b..... org. wwtp://.......aoooooooo//1//339 1423
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. hatp:://8kyllo000.gghhhht OpenWebbeextCorpus 2019.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam M. Shazeer, Andrew M. Dai, M. Hoffman, M. Dinculescu, and D. Eck. Music transformer: Generat- ing music with long-term structure. In ICLR, 2019.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795, April 2020.
doi: 10.1613/jair...111644 URL https: h
11674
Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In ICLR, 2017. URL htttps:/ .net/
forum? d=rla
J. Jumper, Richard Evans, A. Pritzel, Tim Green, Michael Figurnov, O. Ronneberger, Kathryn Tun- yasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, A. Bridgland, Clemens Meyer, Simon A A Kohl, Andy Ballard, A. Cowie, B. Romera-Paredes. Stanislav Nikolov, Rishub Jain, J. Adler, T. Back, Stig Petersen, D. Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, S. Bodenstein, D. Silver, Oriol Vinyals, A. Senior, K. Kavukcuoglu, P. Kohli, and D. Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596:583 589, 2021.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through Memorization: Nearest Neighbor Language Models. International Conference on Learning Representations (ICLR), 2020.
11

---

Published as a conference paper at ICLR 2022
Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, and Kentaro Inui. Shape: Shifted absolute position embedding for transformers. ArXiv, abs/2109.056444, 2021.
Andrew Kyle Lampinen, Stephanie C. Y. Chan, Andrea Banino, and Felix Hill. Towards mental time travel: hierarchical memory for reinforcement learning agents. CoRR, abs/2105.14039, 2021. URL http::://arxivvcco aawpr/g//abaaaee
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers:
Simplifying training of large, sparse models, 2021.
Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. Jurassic-I; Technical details and evalua- tion. Technical report, A121 Labs, August 2021.
Tatiana Likhomanenko, Qiantong Xu, Ronan Collobert, Gabriel Synnaeve, and Alex Rogozhnikov. CAPE: encoding relative positions with continuous augmented positional embeddings. CoRR, 06. 031 43
abs/2106.031433 2021. URL http:p:s//axvxvo//
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Tomas Mikolov and G. Zweig. Context dependent recurrent neural network language model. 2012 IEEE Spoken Language Technology Workshop (SL.7), pp. 234-239, 2012.
Tomas Mikolov, M. Karafiát, L. Burget, J. Cernocky, and S. Khudanpur. Recurrent neural network based language model. In INTERSPEECH, 2010.
Sebastian Nagel. news www.satatatt.eea avail attiiabiillbbeee/oom 2016.
Cc-news.
https://commoorrza..r.rr////2222
Sharan Narang, Hyung Won Chung. Yi Tay, William Fedus, Thibault Fevry, Michael Matena, Kar- ishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Marcus, Adam Roberts, and Colin Raffel. Do transformer modifications transfer across implementations and applications?, 2021.
Masato Neishi and Naoki Yoshinaga. On the relation between position information and sen- tence length in neural machine translation. In Proceedings of the 23rd Conference on Com- putational Natural Language Learning (CoNLL), pp. 328-338, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10.18653/v1/K19-1000 ptttps://aclanthole
URL
K1 K19-1031
Benjamin Newman, John Hewitt, Percy Liang, and Christopher D. Manning. The eos decision and length extrapolation. In BlackBoXVLP@ EMNLP, 2020. URL
pdf
Rodrigo Nogueira, Zhiying Jiang, and Jimmy Li. Investigating the limitations of the transformers with simple arithmetic tasks. ArXiv,
2021.
Myle Ott, Sergey Edunov, David Grangier. and Michael Auli. Scaling neural machine translation.
In Proceedings of the Third Conference on Machine Translation (WMT), 2018
Ankur Parikh, Oscar Täckström, Dipanjan Das. and Jakob Uszkoreit. A decomposable atten- tion model for natural language inference. In Proceedings of the 2016 Conference on Em- pirical Methods Natural Language Processing, pp. 2255, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 10.18653/v1/DI6-11 1244. URL https: 7/ wac.antholooy..or//1.....
Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proceedings of the 15th Conference the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 157-163, Valencia, Spain, April 2017, Association for Computational Linguisties. URL https: www.. acl www..e.oor//chhooogg//eo/ooo/
12

---

Published as a conference paper at ICLR 2022
Ofir Press, Noah A. Smith, and Omer Levy. Improving transformer models by reordering their sublayers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2996-3005, Online, July 2020. Association for Computational Linguistics. doi: aaaaannmm...mm URL httph:t//////m. //wp:///wwww./ ac www.ww.wwww....................w wueb.org/anthor http///nnttoolo/22000000 ogyy/222..
acl-main. 270
Ofir Press, Noah A. Smith, and Mike Lewis. Shortformer: Better language modeling using shorter imputs. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the Ith International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pp. 5493-5505, Online, August 2021. Association for Computational Linguisties. URL pettps:/acllathosn
org/ 2021 aal-acclllllll 427
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lilli- crap. Compressive transformers for long-range sequence modelling. In International Confer- ence on Learning Representations, 2020. URL
net/ forum?id
SylKikSYDH
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1667, 2020. URL http:
7/ //jmlr. wwtp://ppppppp.s..1..... 20002200002 074 html
Jan Rosendahl, Viet Anh Khoa Tran, Weiyue Wang, and Hermann Ney. Analysis of positional encodings for neural machine translation. In International Workshop on Spoken Language Trans- lation, Hong Kong, China, November 2019.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers, 2020.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp-
464-468, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v /ttp/////11-220444 URL wwt.thttt///// / www ac web. http://..wwwb......ttttoooooooee wwtg/attoooogg//Il........ 2074
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021.
Trieu H. Trinh and Quoc Le. A simple method for commonsense reasoning, 2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems. volume 30. Curran Asso- ciates, Inc., 2017. URL https: attps::/prooeedings.u wtw.:44477de11110000eeeeeeeee.e
cc/ htt.ipss.eppppee///eeee httpp/apppr/1////// Paper. .pdf
Ben Wang and Aran Komatsuzaki. GPT-1-6B: A 6 Billion Parameter Autoregressive Language
Model. https: www. httm/kinnoffooll. ht////eshhttt www.ssttrrnooorooeeeeee jax May 2021.
Ulme Wennberg and Gustav Eje Henter. The case for translation-invarian self-attention in transformer-based language models, 2021.
Thomas Wolf, Lysandre Debut, Vietor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. As- sociation for Computational Linguistics. URL https: www.. a aclweb. aarg/anthology/
2020 emn 1 1p demoe.
13

---

Published as a conference paper at ICLR 2022
Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. DA-transformer: Distance- aware transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies. pp. 2059-2068, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18833/v//D0 1/2021 maicl-main.I. 166. URL. htwp://ntntoooooooooo2222222222
htttps://acllamthoooo.... org/2021 wwa.caacll-mm...o -ma In. 1 166
Wojeiech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization, 2014.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference computer vision, pp. 19-27, 2015.
14

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1
INTRODUCTION
The training speed of transformer LMs gets slower the input subsequence length increases. Figure 7 visualizes this.
Training Speed
40000
35000
Sinusoidal
30000
ALiBi
Words Per Second
25000
20000
15000
10000
5000
64
128
256
512
1024
1836
2048
3072
Input Tokens
Figure 7: Training speed of our model and the sinusoidal baseline trained on different amounts of input subsequence tokens L.
Table I contains the runtimes and memory statistics for models using the various position meth- ods discussed in this work.
Table 1:
The speed (during training and evaluation, in words second) and memory usage (during training) of the rotary, T5 bias, and ALiBi models compared to the sinusoidal baseline on WikiText- 103. Training and inference are batched, and speeds are shown for one V100 GPU.
Speed (t)
Position Method
Train Length
Memory (L)
Train
Eval.
512
28.5k
82.
15.3 GB
Sinusoidal
1024
26.0k
77.8k
19.2 GB
3072
15.3k
42.4k
15.1 GB
512
20.0k
43.4k
17.8 GB
Rotary
1024
17.7k
39.4k
22.8 GB
3072
11.5k
29.5k
17.8 GB
512
14.4k
21.8k
16.9 GB
T5 Bias
1024
13.0k
20.2k
20.9 GB
3072
4.3k
4.9k
15.9 GB
512
28.3k
85.8k
15.3 GB
ALiBi
1024
25.8k
76.4k
19.3 GB
3072
15.5k
42.2k
15.2 GB
Tables 3 and 4 show the perplexity and runtime of models using the sinusoidal, rotary T5 bias, and ALiBi position methods when extrapolating to sequences longer than the ones they were trained on. The models used in these tables were trained on L. 512, 1024 and 3072 tokens.
15

---

Published as a conference paper at ICLR 2022
Table 2: The sinusoidal, rotary, T5 bias and ALiBi models trained I, 512 Wiki'Text1033 and evaluated with different values of Tvalid on the validation set. Bold shows the best score for each model. Inference speeds (in words per second) are from inference GPU with batch size of one.
Sinusoidal
Rotary
TS Bias
ALiBi
Inputs
PPL (4)
WPS (t)
PPL (4)
WPS (t)
PPL (4)
WPS (t)
PPL (J)
WPS (t)
512
20.05
15046
20.07
10839
19.65
11724
19.73
14726
513
19.98
14925
20.01
10806
19.57
10491
19.62
14965
522
19.93
15116
20.02
11295
19.57
9970
19.64
15316
532
19.91
15358
19.98
10854
19.53
10382
19.61
15383
542
15076
19.94
10795
12270
15301
552
19.91
16394
19.93
12267
19.47
13000
19.54
16540
562
19.91
16646
19.87
12481
19.39
12201
19.49
16385
572
19.95
16934
19.83
12668
19.36
12851
19.46
16881
582
20.13
16961
12594
19.41
13904
17064
592
20.18
17243
13007
19.36
13706
19.43
17289
602
20.40
17502
19.81
12788
19.33
14102
19.38
17141
612
20.59
17637
19.81
12601
19.27
14573
19.38
1 17661
712
24.86
15614
19.79
12676
19.10
13818
19.14
15637
812
17151
13954
18.94
14377
17210
912
37.42
17200
20.73
13887
18.86
15345
18.88
17619
1012
43.54
16304
21.37
13759
18.79
14240
18.73
16059
1112
50.36
16424
22.01
13891
18.77
14014
18.68
16659
1212
58.01
17294
23.02
15245
18.87
14589
18.67
17372
1312
15314
13698
13138
15698
1412
70.75
15663
24.81
13928
18.87
12857
18.59
1 15860
1512
76.23
15812
25.99
14248
18.91
13752
18.52
1 16225
2512
132.41
15254
31.58
13456
20.41
9948
18.41
15204
3512
178.97
13293
35.54
11850
22.91
7847
18.40
13329
4512
209.37
11767
39.15
10485
25.91
6146
18.41
11738
5512
240.44
10168
43.14
9020
29.54
5309
18.36
9986
6512
271.40
9052
47.81
8108
34.48
4680
18.35
9022
7512
293.02
8315
51.12
7483
39.29
4102
18.33
8324
8512
305.65
7259
54.98
6718
43.08
3660
18.34
7366
9512
336.02
6672
$7.85
6211
48.90
3370
18.34
6555
10512
341.53
6126
60.77
5575
$2.95
3010
18.32
6030
11512
362.74
5994
66.62
5445
61.38
2873
18.32
$882
12512
373.17
5421
69.70
4988
64.94
2602
18.31
$287
13512
382.91
5174
73.27
4692
OOM
18.31
4962
14512
399.98
4351
75.52
4103
0OM
18.31
4352
15512
406.01
4291
79.25
3969
0OM
18.31
4289
16

---

Published as a conference paper at ICLR 2022
Table 3: The sinusoidal, rotary, T5 bias and ALiBi models trained I. 1024 WikiText-103 and evaluated with different values of Lvalid on the validation set. Bold shows the best score for each model. Inference speeds (in words per second) from inference GPU with batch size of one.
Sinusoidal
Rotary
TS Bias
ALiBi
Inputs
PPL (L)
WPS (t)
PPL (4)
WPS (t)
PPL (4)
WPS (t)
PPL (J)
WPS (t)
1024
19.34
17002
19.33
14690
18.80
14973
18.66
16951
1025
19.33
16630
19.34
14423
18.82
14635
18.67
16690
1034
19.27
16589
19.28
14351
18.74
14435
18.60
16707
1044
19.26
16760
19.27
14491
18.72
14644
18.60
16667
1054
19.23
16747
19.26
14503
14800
16833
1064
19.21
16676
19.22
14623
18.70
14498
18.55
16941
1074
19.19
16879
19.19
14464
18.65
14670
18.49
16936
1084
19.22
16942
19.23
14650
18.70
14607
18.56
17090
1094
19.24
16771
19.22
14629
18.69
14517
18.54
16880
1104
19.28
16870
19.27
14837
18.69
14635
17009
1114
19.29
16795
19.27
14879
18.69
14540
18.52
17050
1124
19.26
17312
19.18
15121
18.62
14480
18.46
17571
1224
20.54
17901
19.38
15584
18.58
14956
18.40
18013
1324
23.13
16308
14386
18.52
13726
18.33
16422
1424
26.45
16217
21.27
14385
18.48
13516
18.28
16121
1524
29.82
16377
22.59
14693
18.42
13587
18.22
1 16659
1624
34.27
15928
24.34
14228
18.40
12979
18.17
16053
1724
38.24
16640
25.66
14686
18.35
12976
18.15
16607
1824
42.23
16840
14918
13071
1924
46.46
15071
29.64
13452
18.31
11843
18.08
15118
2024
51.09
15591
31.17
13706
18.34
11906
18.05
15557
3024
96.46
13639
35.67
12256
18.62
8480
17.92
13668
4024
144.00
12441
11203
7443
12402
5024
182.31
11431
48.31
10324
20.47
6384
17.92
11394
6024
214.02
10238
54.78
9117
21.76
5577
18.01
10119
7024
261.86
8785
62.83
7950
23.64
4867
17.93
8779
8024
284.88
8132
64.91
7355
25.79
4377
17.96
8086
9024
310.04
7045
71.91
6380
27.54
3787
17.98
7001
10024
337.48
6633
77.70
6016
29.54
3582
17.97
6583
1024
358.43
5722
81.15
5219
31.94
3170
18.02
5641
12024
375.95
5560
87.51
5072
33.35
2940
18.01
$294
13024
393.57
4691
94.74
4383
OOM
17.98
4621
14024
403.52
4905
96.10
4546
OOM
18.01
4827
15024
431.66
4518
99.78
4170
0OM
17.96
4447
16024
453.32
4239
106.99
3878
0OM
17.98
4153
17

---

Published as a conference paper at ICLR 2022
Table 4: The sinusoidal. rotary, T5 bias and ALiBi models trained on L 3072 on WikiText-100 and evaluated with different values of Lvati on the validation set. Bold shows the best score for each model. Inference speeds (in words per second) are from inference on GPU with batch size of one.
Sinusoidal
Rotary
T5 Bias
ALiBi
Inputs
PPL (4)
WPS (t)
PPL (4)
WPS (t)
PPL (4)
WPS (1)
PPL (J)
WPS (t)
3072
18.67
13380
18.57
12548
18.01
8828
17.60
13866
3073
18.67
13773
18.57
12474
18.01
8483
17.59
13793
3082
18.62
13741
18.54
12388
17.95
8698
17.59
13778
3092
13742
12458
8361
13783
3102
18.65
13701
18.52
12365
17.94
8764
17.59
13747
3112
18.64
13809
18.51
12449
17.96
8665
17.59
13827
3122
18.68
13722
18.52
12432
17.98
8437
17.58
13795
3132
18.67
13825
18.54
12490
17.97
8653
17.58
13784
3142
18.69
13543
12230
17.97
8282
17.61
13572
3152
18.66
13520
18.56
12240
17.98
8608
17.59
13523
3162
18.71
13501
18.56
12253
18.04
8589
17.62
13598
3172
18.72
13563
18.55
12297
17.99
8583
17.59
13625
3272
13453
18.55
12148
8144
13482
3372
19.46
13533
18.50
12254
17.88
8442
17.52
13565
3472
20.55
13047
18.52
11868
17.95
7857
17.54
13107
3572
21.84
13128
18.50
11882
17.86
7814
17.50
13170
3672
23.04
13106
18.49
11859
17.87
7719
17.48
13196
3772
13287
18.54
11942
7579
17.49
13312
3872
25.85
12621
18.40
11272
17.82
7581
17.41
12566
3972
27.21
12379
18.48
11151
17.84
7483
17.41
12324
4072
28.59
12178
18.59
11019
17.88
6974
17.48
12212
5072
45.53
11076
18.80
9887
17.76
6230
10938
6072
65.01
10114
19.50
9049
17.68
5554
17.26
10133
7072
85.96
8647
20.60
7861
17.83
4820
17.22
8670
8072
102.74
7755
21.60
6991
18.06
4281
17.30
7729
9072
125.99
6953
22.14
6360
18.12
3823
17.26
6939
10072
133.68
6646
23.21
6068
18.37
3579
17.28
6597
11072
161.29
5663
24.39
5158
18.64
3119
17.26
$585
12072
169.55
5567
26.70
5111
18.93
2920
17.24
5397
13072
189.43
5044
29.33
4658
19.10
2735
17.15
4809
14072
203.86
4915
32.21
4616
OOM
17.22
4866
15072
221.14
4561
33.47
4292
OOM
17.23
4491
16072
231.29
4382
34.51
4099
0OM
17.22
4312
18

---

Published as a conference paper at ICLR 2022
A.2
ALIBI RESULTS ON WIKITEXT-101
Training Speed vs. Valid Perplexity with Lidiid = 3072
19.0
Sinusoidal
Perplexity (
L. 3072
ALiBi
18.5
X L= 512
18.0
X L= 1024
3072
X L= 1536
X
X L-= 2048
17.5
15000
20000
25000
30000
Training Words Per Second (+)
Figure 8: The training speed and validation perplexity (with L = 3072) for ALiBi models and the sinusoidal model trained with L = 3072. All our models trained on 512 or more tokens achieve better perplexity than the sinusoidal model even though all of them (except the L = 3072) require less time and memory to train.
Figure g depicts cross section of Figure 4 showing our models with different train lengths and the sinusoidal baseline, all evaluated on Leaid 3072 tokens. We observe that all our models with 512 < L < 3072 are faster to train than the sinusoidal model with L = 3072, but they all achieve greater perplexity scores on the validation set. Our model with 3072 trains just as fast as the sinusoidal one but bests its score by more than one perplexity point; (the standard deviation for the the sinusoidal model with = 3072 is 0.24).
Table 5 shows the perplexity values obtained when 8 different ALiBi models, trained on L values between 64 and 3072, extrapolating to Lyyiid values longer than the ones they were trained on. In addition, we present results for the sinusoidal, rotary and T5 bias models, with T T..
Table 5: Perplexity when ALiBi extrapolates the WikiText-103 development set. *For results present for the sinusoidal, rotary and T5 bias models, L = Lvatid (so we do not test the extrapolation abilities of those baselines here).
ALiBi
Evaluation Length
Train Length
64
128
256
512
1024
1536
2048
3072
28.46
24.70
22.88
22.09
21.73
21.63
21.59
21.53
128
23.98
21.70
20.67
20.36
20.29
20.31
20.28
256
21.29
19.89
19.29
19.13
19.10
19.03
512
19.73
18.81
18.50
18.48
18.40
1024
18.66
18.20
18.05
17.96
1536
18.12
17.90
17.72
2048
17.91
17.64
3072
17.60
Sinusoidal" Rotary*
28.03
23.81
21.45
20.05
19.34
19.05
18.87
18.67
20.07
19.33
18.57
T5 Bias*
19.65
18.80
18.01
Table 6 compares ALiBi to the sinusoidal, rotary and T5 bias baselines on the test set of WikiText- 103, and Table compares ALiBi to the current state of the models that test set.
19

---

Published as a conference paper at ICLR 2022
Table Test perplexity and runtime on WikiText-103 for of our ALiBi models and models that use the sinusoidal, rotary and T5 bias methods.
Train
Inference
Model
Param.
Speedf
Speed f
Valid +
Test 4
Sinusoidal, == 3072
247M
15.3k
13.6k
18.67
19.38
Rotary, L 3072
247M
11.5k
12.2k
18.57
19.28
TS Bias, L 3072
247M
4.3k
7.3k
18.01
18.73
ALiBi 512, Livatdd = 3072
247M
28.3k
13.6k
18.40
19.08
L = =300,, Lealid = 3072
247M
15.5k
13.6k
17.60
18.30
Table Valid and test perplexity scores WikiText-109 for of ALiBi models and models that use the sinusoidal, rotary and TS bias methods with sliding window evaluation ($B and S=512 following (Baevski & Auli 2018; Khandelwal et 2020 Press 202T)). The sinusoidal model presents our results from training and inference with the model of Baevski & Auli.
Model
Param.
Valid
Test 4
Adaptive Inputs (Baevski & Auli 2018
247M
17.97
18.70
Transformer-X XL (Dai et al. 2019
257M
18.3
Shortformer (Press et al. 2021)
247M
17.47
18.15
Sandwich Transtormer Press et al.
2020
247M
17.96
Staged Training (Press et al. 2021
247M
17.56
Compressive Transtormer (Rae
2020
329M
17.1
Routing Transformer (Roy et al. 2020)
15.8
KNN-LM (Khandelwal al. 2020)
247M
15.81
15.79
Sinusoidal, 3072
247M
17.95
18.67
Rotary, L = 3072
247M
17.98
18.72
T5 Bias, L = 3072
247M
17.37
18.12
= 512, Leatid = 3072 == 3072, Lvalid = 3072
247M
18.30
19.01
247M
16.97
17.66
A.3 RESULTS ON THE TORONTO BOOK CORPUS
To ensure that our results are not specifie the WikiText- 103 corpus, we next apply our model and the baselines to different domain while using similar model architecture and the same ALiBi slopes as those used the previous subsection.
We emphasize that our set of slopes was chosen by running experiments on the WikiText- 103 corpus, and here we apply that set slopes the entire process of developing this method, we ran only one set of experiments this domain using the previously selected set of slopes.
model trained on very different domain. Throughout
Specifically, we use the Toronto BooksCorpus (huu et al. 2015). which has been used to train BERT (Devlin et al. 2019) (in conjuction with the English Wikipedia). The corpus about 700M tokens (2.9 GB).
We use the same train/validation/eestt split as Khandelwal et al. (2020) and their tokenization, which uses BERT's vocabulary of 29K byte-pair encodings. Since the vocabulary is much smaller than WikiText-103's we replace the adaptive word embedding and softmax of Baevski & Auli (2018)
with tied word embedding and softmax matrix (Press & Wolf 2017 Inan et al. 2017).
Our results in Figure 9 (and Table g replicate our success on the WikiText- 103 dataset. Our model surpasses the sinusoidal baseline when trained on the same amount of input tokens (T.) and, in
20

---

Published as a conference paper at ICLR 2022
ALiBi Extrapolating on Toronto BookCorpus
15.0
. L 512
Sinusoidal, L 512
Perplexity (-)
14.5
1024
Sinusoidal, L 1024
14.0
3072
Sinusoidal, L 3072
13.5
x
13.0
512
1024
3072
Validation Input Length (Lyiid))
Figure 9: ALiBi-enabled models evaluated on different input lengths on the Toronto BookCorpus. Our models extrapolate to longer sequence lengths and outperform the sinusoidal baseline even when trained on much shorter sequences.
addition, our model able to extrapolate to longer sequences at inference. This occurs even though of slopes not tuned this dataset. This result establishes the generality of ALiBi and the particular set of slopes we found and suggests that they may be used on different text domains without further hyperparameter tuning.
Tables 9 and 10 present the perplexities for our ALiBi models, the baselines, and the current state of the on the Toronto BookCorpus validation and test sets. Our results here mirror our results on WikiText-1100 we improve over the sinusoidal baseline even when AliBi is trained on fewer tokens.
Table 8: ALiBi models extrapolating the Toronto BookCorpus development set. *For the results of the sinusoidal models, Livatd (so we do not test the extrapolation abilities of those models here).
Evaluation Length
Train Length
512
1024
3072
512
14.29
13.64
13.55
1024
13.86
13.52
3072
13.15
Sinusoidal"
14.80
14.73
14.46
Table 9: Validation and test perplexities on the Toronto Book Corpus dataset.
Model
Param.
Valid +
Test J
Sinusoidal, I. 3072
247M
14.46
11.67
ALiBi
Ttrain 512, Togatid 3072
247M
13.55
10.98
Tarain = =072,, T.vatid = 3072
247M
13.15
10.73
A.4 RESULTS ON THE CC100+ROBERTA CORPUS
Table compares our 1.3 billion parameter ALiBi models when extrapolating to two times the number of tokens that they were trained on. We the sinusoidal model baseline, and train for the same amount of time as we train the ALiBi model that we compare it to (and so since our ALiBi models run faster in this setting, the sinusoidal models complete less updates).
21

---

Published as a conference paper at ICLR 2022
Table 10: Validation and test perplexities on the Toronto Book Corpus dataset with sliding window
(s.). Following (Baevski & Auli 2018: Khandelwal et al. 2020 Press et al. 2020 2021
we set
the sliding window stride S=512.
Model
Param. +
Valid +
Test +
KNN-LM (Khandelwal et al. 2020
247M
14.20
10.89
Shortformer (Press et al. 2021)
247M
13.40
10.88
Sandwich (Press et al. 2020)
247M
10.83
Staged Traiinn (Pesss et al.
2021
247M
12.80
10.48
Sinusoidal, == 3072
247M
14.06
11.40
ALiBi
L =12,, Loalid 3072
247M
13.76
11.11
L=3072 Liatid = 3072
247M
12.70
10.40
Table 11: Perplexity, memory, and train time on the CCIOO+RoBERTa corpus for our ALiBi models
and the sinusoidal baseline. We run our L = 512 (1024) model and the sinusoidal model with I = 1024 (2048) for the same amount of time. We show that our models achieve strong results even though they use 6-11% less memory.
Training
Valid PPL +
Memory 4
Updates
Hours
Diaatd 1024
Loviid 2048
Sinusoidal, Lirain 1024
26.2 GB
46.7k
5.5k
9.24
ALiBi, Ltrain = 512
24.6 GB
50.0k
5.5k
9.30
Sinusoidal, Lirain = 2048
29.3 GB
44.2k
5.9k
9.01
ALiBi, Ltrain 1024
26.2 GB
50.0k
5.9k
8.92
Table 12 compares our 1.3 billion parameter ALiBi models to the sinusoidal baselines, with and without extrapolation, with all models completing 50,000 updates.
Table 12: Perplexity, train time and memory of the sinusoidal and ALiBi models the CC100+RoBERTa corpus when all models are trained with 50k updates.
Training
Valid PPL
Memory
Updates
Hours
Liatt = 512 Linit 1024
Liati = 2048
Sinusoidal, Terain 512
24.6 GB
50.0k
5.5k
9.71
37.05
105.42
ALiBi, Lavain 512
24.6 GB
50.0k
5.5k
9.79
9.30
9.54
Sinusoidal, TArain 1024
26.2 GB
50.0k
5.9k
9.15
48.85
ALiBi, Terain 1024
26.2 GB
50.0k
5.9k
9.16
8.92
Sinusoidal, Ltrain 2048
29.3 GB
50.0k
6.7k
8.83
ALiBi, Lerain 2048
29.4 GB
50.0k
6.7k
8.84
B ANALYSIS
In this section we investigate why ALiBi works so effectively. We find that ALiBi's decrease in perplexity when given longer sequences is largely explained by its improved avoidance of the early token curse. We hypothesize that future work building on ALiBi might achieve further gains by more efficiently exploiting longer histories.
22

---

Published as a conference paper at ICLR 2022
B.1 DEFINING SLIDING WINDOW EVALUATION AND THE EARLY TOKEN CURSE
The big gray cat sat on the mat
Figure 10: Sliding window evaluation (top; blue) compared to nonoverlapping evaluation (bottom; red) on sequence words using a model with Lvalid 4. Nonoverlapping evaluation is much faster since it requires just two inference passes (as opposed to the five passes required by the siding window approach). But the sliding window approach provides more context for each prediction.
Sliding Window Inference As mentioned in Section nonoverlapping inference is commonly used to evaluate sequences longer than (the number of tokens in each training subsequence). An alternative is to use sliding window during evaluation (Baevski & Auli 2018).
stride picked between and L 1, and the window is advanced by S tokens after each forward pass This means that L tokens from the previous subsequence re-encoded, and only 5 new tokens are output. The advantage is that all outputs each subsequence after the first have least L --5 S previous tokens condition on. However, since tokens must be re-encoded multiple times, this approach is much slower than the nonoverlapping one. When S = 1, we output one token every inference pass, each using the maximal context window that the model can handle; however, this is the slowest approach. Figure 10 is visualization of the nonoverlapping and sliding window evaluation approaches.
We use sliding window inference as tool to analyze our models, but we note that it is normally prohibitively slow in practice (Press al. 2021).
Early Token Curse Splitting an evaluation set into subsequences means that predictions occuring early in each subsequence cannot access many previous context tokens (appearing the end of the previous subsequence). The result, referred to as the early token curse (Press et al. 2021]. increases (i.e., degrades) perplexity scores. A workaround is evaluate the model using a sliding window, giving each prediction more context. This solution slow since it requires many more forward passes of the model.
B.2 EXTRAPOLATION REDUCES THE FARLY TOKEN CURSE
We presented results showing that our ALiBi method (and, to lesser extent, the T5 bias) allows LMs extrapolate during inference. Two reasons could explain why these methods enable LMSs to achieve better perplexity given longer input subsequences:
1. Performance improves because the models can use longer contexts to make more accurate predictions. For example, the average article length in the WikiText-103 corpus is about 3600 tokens;
therefore, if model trained on I, 512 tokens extrapolates to Lowell 3072 tokens during inference and achieves better results, that might be because it can spot patterns occurring across more than 512 tokens.
2. Performance improves because longer input sequences mean the early token curse re- duced. For example, during nonoverlapping evaluation on sequences of length Lvatis = 1000, 10% predictions have 100 tokens of context or less. If we rerun nonoverlapping evaluation on that model with Luati 2000 tokens, now only 5% of predictions have 100
Nonoverlapping inference be viewed sliding window inference with stride T..
23

---

Published as a conference paper at ICLR 2022
tokens of context or less. So, by simply being able to handle longer seguences, model can substantially reduce the early token curse and improve performance
To better understand what might be occurring, we re-evaluate the development with our models and the sinusoidal baseline with L, = 512, 1024, 3072. However, this time we use sliding window evaluation with a stride of S 1, meaning that we move the sliding window just one token after every inference pass, giving each prediction the maximum number of context tokens that the model can use.
WikiText-
ALiBi and Sinusoidal Evaluation (w/ Sliding Window) on WikiText-103
19.5
Simusoidal, L= 512
19.0
Sinusoidal, 1024
Perplexity (-)
Sinusoidal, L 3072
18.5
ALiBi, = 512
18.0
A
ALiBi, L= 1024
17.5
X
ALiBi, L 3072
17.0
x
16.5
512
1024
1536
2048
3072
Validation Input Length (Iraits)
Figure 11: ALiBi models evaluated on different input lengths on WikiText-103 with sliding window evaluation (with stride S = 1). Unlike results shown in Figure [ where performance improves in each our models as we increase the validation sequence length, here performance stays relatively flat as we increase Lyais- This might mean that ALiBi increases performance when Lialid > L not because it uses longer contexts, but because fewer tokens suffer from the early token curse. Note that as in 2 the perplexity of the sinusoidal model explodes when Lvalid L even when using sliding window evaluation.
The results shown in Figure I1 and in the corresponding Tables 13 (sinusoidal) and 15 (ALiBi).
Unsurprisingyy, for the sinusoidal model, as in $2 increasing Liaaid causes an explosion in perplexity when using sliding window evaluation. Our ALiBi models cannot improve perplexity when looking at longer sequences in this setting, but they keep perplexity flat when Lyalid increases.
This leads us to believe that our perplexity improvement when increasing Lyalid and using nonover- lapping evaluation is caused by explanation 2, not explanation 1. Because sliding window evaluation provides long context windows for every prediction made, it curtails the early token curse. In this setting, ALiBB's performance remains flat when Lvalid increases, leading us to hypothesize that the gains seen while increasing Lialid were the result of larger Livlid values mitigating the early token curse.
Our ALiBi results mirror what occurs in the model using the T5 bias: when using sliding window evaluation, perplexity remains relatively flat when evaluating longer sequences (see Table 14).
Our analysis reveals that when Lvaltt > L, ALiBi might not be using contexts longer than the ones it was trained on. This highlights research direction that could be pursued in future work.
These findings do not lessen the value of ALiBi. When Liatts L. ALiBi achieves either superior or similar results to the sinusoidal method and other alternatives even though is simpler and requires no learned parameters. When evaluating Lyaid > L tokens, even if ALiBi does not attend more than I. tokens, it yields better results than the other alternatives that can be used in this case, i.e., standard nonoverlapping inference (which is cheap, but does not perform as well) and the more accurate sliding window approach (which is very slow).
100 tokens is arbitrary small number used here represent short history context, i.e., one in which making predictions for the next output token would be harder.
24

---

Published as a conference paper at ICLR 2022
Table 13: Perplexities of the sinusoidal models evaluated with sliding window evaluation with stride on the WikiText-103 validation dataset.
Evaluation Length (S 1)
Train Length
512
1024
1536
2048
3072
512
18.35
204.42
264.74
306.19
360.12
1024
18.05
206.55
302.6
393.71
3072
18.03
Table 14: Perplexities of the T5 bias models evaluated with sliding window evaluation with stride S on the WikiText-109 103 validation dataset.
Evaluation
Length (S = 1)
Train Length
512
1024
1536
2048
3072
512
17.92
18.51
20.36
22.62
30.77
1024
17.65
17.87
18.51
20.66
3072
17.41
Table 15: Perplexities of the ALiBi models evaluated with sliding window evaluation with stride = 1 on the WikiText-103 validation dataset.
Evaluation Length (S 1)
Train Length
512
1024
1536
2048
3072
512
17.98
17.92
18.2
18.28
18.3
1024
17.46
17.47
17.62
17.92
3072
16.96
25


Reviews:
[{"review_id": "pPd6EU_o60C", "reviewer": "Reviewer_xt6W", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- Injecting temporal bias to attention is a neat idea for the language model extrapolation problems.\n- This paper presents comprehensive experiments on comparing the proposed method with existing positional encoding approaches.\n- The paper is well written and easy to understand.\n\nCons:\n- It would be helpful to discuss the potential applications of the proposed method other than language modeling.\n- I am curious about comparing the transformer+ALiBi with LSTM in extrapolation tasks. The idea of adding temporal bias to attention is similar to the forget gate in LSTMs. Therefore, adding LSTM as a reference will make the paper stronger.\n\n**Update:** The additional results are convincing. I raised my rating to acceptance.", "summary_of_the_paper": "This paper investigates the extrapolation capability of transformer-based language models. The authors observed that existing positional encoding methods (e.g., sinusoidal embedding, relative positional embedding) fail to generalize to longer sequences in language modeling tasks. Therefore, they introduce a new positional encoding method called ALiBi, which adds temporal bias to the multi-head attention to penalize attention score proportional to token distances. Experimental results show that ALiBi has significantly stronger extrapolation capability compared to other positional encoding methods.", "summary_of_the_review": "This paper proposes an interesting and novel idea for enhancing the extrapolation capability of transformer-based language models. A few additional experiments and discussions will make the paper stronger.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QeCzYq5SPAg", "reviewer": "Reviewer_ZAja", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "At a very high level, I did enjoy the paper as the method is simple and it indeed helps a pretrained transformer-based models to extrapolate to much longer sequences. Some of my concerns were addressed in the authors' response, and the others do require extensive exploration. Therefore, I would like to see this submittion at ICLR2022.\n\n\n====end of the update====\n\n1. When the dimension of a transformer module is roughly the same as or significantly larger than the number of tokens, the dimension becomes the main contributing factor to the time complexity, which explains why, with the linear bias, the model only achieved ~10% speedup.\n\n2. I was wondering if we could directly manipulate the probability after the softmax layer, it probably would achieve a similar effect. For example, one can multiply the probability map with a matrix with 1s in the diagonal terms and with linearly decaying off-diagonal terms towards 0, which also effectively biased the model to learn from nearby tokens. \n\nMy point here is that the submission could have been more generalised in a way that, say, as long as the bias terms are fixed before training and they have an impact on the attention scores or the probability maps, the model will extrapolate to very long sequences. This would've been a stronger and more generalised message.\n\n3. The title and the intro gave me the impression that it was designed for transformers, but I was wondering whether it would hinder transformers' capability in modelling images or biological sequences where tokens that are far from the current one would still play an important role.\n\nFor images, the current approach of serialising an image is either at pixel-level or at patch-level, which means that tokens surrounding the current one in 2-dimensional space will be the context, however, the proposed approach would potentially worsen the situation.  \n", "summary_of_the_paper": "The submission proposed an effective approach to allow pre-trained transformer-based language models to extrapolate beyond the maximum length used in training, which potentially reduces the training time as extrapolation is empirically guaranteed. The proposed method adds fixed biases to the dot-product values between queries and keys that linearly decays w.r.t. the gap between two positions. Empirically, the proposed method indeed successfully allows pre-trained models to be evaluated on sequences that are multiple times longer than the training ones without significant loss.", "summary_of_the_review": "The submission proposed a simple yet effective method that helps pre-trained language models to extrapolate beyond the sequence length used in the training, but I think the paper could've delivered a stronger message. I am open to discussions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F6ECvzSYDXu", "reviewer": "Reviewer_6wKQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method is simple and quite effective. The paper addresses an important research problem of input length extrapolation. ALiBi developed on Wikitext-103 generalizes to 1.3B parameter model. ALiBi’s inductive bias also improves the accuracy.\n\nPrevious works did not rigorously evaluate the extrapolation of a transformer and simply assumed the possibility of extrapolation. On the other hand, this paper carefully measured extrapolation compared with other works (Rotary and T5 Bias) and devised their own method to overcome the limitations of previous works. The method itself might look less novel or incremental because previous works inspire its many parts. It would be much better to provide theoretical explanations more than empirical proof on why ALiBi enables better extrapolation and higher final accuracy.\n\nALiBi is only evaluated on language modeling in this paper. A transformer is a widely used neural architecture for many different tasks and domains. They also mentioned in the related work section that other works studied extrapolation on other tasks. I wonder about the authors’ thoughts whether their ALiBi could be helpful to other tasks as well. Of course, the importance of the longer context and extrapolation ability may vary depending on the task.\n\nOne minor question is that the dot products of queries and keys are usually divided by the square root of the dimension, and it is maybe abbreviated in the equation. I am curious this division is performed after or before adding a bias.\n\nEach head has a different slope for the linear bias, so I expect that heads learn different patterns. An analysis of that would be interesting. The authors argue that the method is robust to slope choice, but they found that other alternatives underperform, such as learning these slopes. Because many other design choices are possible, I am curious how they found the final solution and what they tested.\n\nALiBi was tested on two different model sizes. According to their results, extrapolation on a billion language model (improving until ~2x) is relatively inferior to that on a Wikitext-103 scale language model (improving until ~6x). I worry whether extrapolation ability reduces as the model becomes bigger (or with more training data).", "summary_of_the_paper": "The paper addresses the extrapolation problem where a test sequence longer than training sequences is given and proposes Attention with Linear Biases (ALiBi) that adds a penalty linear to the distance between a query and a key to the attention scores. ALiBi shows remarkable input length extrapolation ability while computationally efficient with almost marginal overhead compared to the standard transformer. Moreover, ALiBi does not induce any additional parameters and generalizes well to a billion scale language model.", "summary_of_the_review": "The paper is well written and easy to follow. The contribution is concrete and practically useful since a transformer is a building block of many machine learning models. More importantly, the size of language models becomes bigger, so their training cost is prohibitive. ALiBi improves the efficiency of language model (or transformers in general) training. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EFkVYJ2_Ex_", "reviewer": "Reviewer_dz7S", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- To my knowledge, the paper is the first to study length extrapolation in Transformer language models. This is an important open problem for language modeling.\n- The proposed ALiBi mechanism is simple to implement and computationally efficient.\n- Experiments confirm that the proposed method enables length extrapolation for language modeling.\n- The paper is well-written and easy to follow.\n\nWeaknesses:\n- Experiments can be expanded. I am curious if the findings also apply to other tasks, such as text classification, sequence labeling, and sequence-to-sequence generation. The proposed method is simple to implement, so I imagine it would not be hard to add a few more tasks.\n\nMissing related work: \n- Xu et al., 2021. How neural networks extrapolate: from feedforward to graph neural networks. This paper studies a similar kind of input length/size extrapolation for graph neural networks.", "summary_of_the_paper": "This paper studies input length extrapolation for Transformer language models; i.e., how Transformer LMs perform on test sequences that are longer than training sequences. The paper finds that how positions are encoded plays a crucial role for input length extrapolation. Models with sinusoidal and rotary position embeddings do not extrapolate well, while T5’s position-dependent attention mechanism (dubbed T5 bias) enables better extrapolation. The paper then proposes ALiBi, another attention mechanism that also allows extrapolation while being computationally more efficient than T5 bias. These results are empirically confirmed on two datasets.", "summary_of_the_review": "The paper studies a novel problem, input length extrapolation in language modeling, and proposes a simple solution with good empirical results. The paper is also well-written. One way to further improve the paper is to add experiments on other tasks. Overall, I recommend acceptance of this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes a simple, efficient, and effective position representation method for the Transformer architecture called ALiBi. ALiBi enables better extrapolation and performance (in terms of efficiency and task performance). The submission also includes careful analysis and extensive experiments, and notably suggests that the gains of ALiBi may be less pronounced in more scaled-up settings. All reviewers agreed the paper should be accepted. I think it's reasonably likely that ALiBi will become a common choice in future Transformer models, or at the very least that this work will prompt further work on developing improved position representations for Transformer models. I therefore recommend acceptance.
Decision: Accept (Poster)