Dissecting Transformer Length Extrapolation via The Lens of Receptive Field Analysis
Year: 2023

Abstract:
Length extrapolation permits training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences. A relative positional embedding design, ALiBi, has had the widest usage to date. We dissect ALiBi via the lens of receptive field analysis empowered by a novel cumulative normalized gradient tool. The concept of receptive field further allows us to modify the vanilla Sinusoidal positional embedding to create Sandwich, the first parameter-free relative positional embedding design that truly length information uses longer than the training sequence. Sandwich shares with KERPLE and T5 the same logarithmic decaying temporal bias pattern with learnable relative positional embeddings; these elucidate future extrapolatable positional embedding design.

heading: 1 Introduction
text: The length of input sequences is an important hyperparameter choice for pretraining a transformer language model. A vanilla transformer language model has a quadratic training cost w.r.t Ltr, the training sequence length. As the value of Ltr increases, cost becomes impractical. However, we can use the model for substantially longer evaluation sequence lengths Lex Ltr as gradients no longer need to be recorded. The discrepancy between Ltr and Lex motivates the task of length extrapolation (Press et al., 2022): Can a transformer language model maintain equally good, if not better, perplexities when longer sequences are used in the testing stage?
Several extrapolatable transformer language models have been proposed including ALiBi (Press et al., 2022) and KERPLE (Chi et al., 2022), of which the relative positional embedding design is hypothesized to be critical to success. Empirically, they extrapolate to Lex Ltr much better than other absolute and relative positional embeddings
including Sinusoidal (Vaswani et al., 2017), Rotary (Su et al., 2021), and T5 (Raffel et al., 2020), resulting in the adoption of ALiBi for the recently released Bloom (Scao et al., 2022) model. Despite the significant empirical success of ALiBi, there is still a lack of fundamental understanding of why it works.1
Figure 1 shows the implementation of ALiBi. We hereinafter refer to the coefficient 1
2h as slope.
Intuitively, ALiBi encourages a token to focus on neighbors based on its temporal biases matrix. When two tokens are distant, ALiBi becomes highly similar to windowed attention, shown in Figure 2. Experiments in §4 will further establish the
1https://github.com/ofirpress/attention_with_ linear_biases#why-do-you-think-alibi-works
13522
connection between the two. Windowed attention allows the easy derivation of a theoretical (maximum) receptive field: wR for an R layer transformer model with windowed attention size w. A windowed attention model can extrapolate if Ltr > wR because 1) wR is fully covered by Ltr during the training stage, and 2) it simply ignores the additional Lex−wR tokens during the testing stage. Surprisingly, a model can still extrapolate when Ltr < wR which we show in §4. This calls for the need for empirical receptive field measurement and motivates our model-agnostic cumulative normalized gradient tool. The tool we develop can be applied back on ALiBi to show that Ltr covers most of its empirical receptive field.
Our analysis tool also provides critical context for explaining the length extrapolation failure (Press et al., 2022; Chi et al., 2022) of Sinusoidal (Vaswani et al., 2017) and Rotary (Su et al., 2021) by showing their violation of the empirical receptive field coverage principle. Sinusoidal can be fixed by dropping the intermediate terms and keeping only the decay-with-distance biases; this leads to the creation of Sandwich, the first parameter-free relative positional embedding that uses information beyond Ltr. Sandwich shares a similar temporal bias pattern with trainable positional embeddings such as KERPLE (Chi et al., 2022) and T5 (Raffel et al., 2020), and they jointly suggest the future design of extrapolatable transformer positional embeddings.

heading: 2 Related Work

heading: 2.1 Length Extrapolation
text: In the context of language modeling, we expect token-level perplexities to remain at least the same, if not lower (i.e. better), when Lex Ltr sequences are provided. Recurrent neural networks (Mikolov et al., 2010; Mikolov and Zweig, 2012; Zaremba et al., 2014) can easily perform length extrapolation. But this is not an easy task for transformer language models, among which only those equipped with special relative positional embeddings (Press et al., 2022; Chi et al., 2022) are length extrapolatable.

heading: 2.2 Positional Embeddings
text: It is widely believed that the design of positional embeddings is the key to successful length extrapolation of transformer language models (Press et al., 2022; Chi et al., 2022). We can roughly catego-
rize existing positional embeddings into absolute (APE) (Vaswani et al., 2017) and relative (RPE) (Su et al., 2021; Raffel et al., 2020; Press et al., 2022; Chi et al., 2022) variants. APE often assigns one positional embedding per token and combines them directly with input embeddings. In contrast, RPE adds temporal bias terms to the self-attention matrix to encode the relative distance between token pairs. For example, the right triangular matrix in Figure 1 shows the set of temporal bias terms. It is challenging for APE to extrapolate well without any further fine-tuning since either the beyond L positional embeddings do not exist, or the model needs to process unseen positional embeddings (e.g. unseen sinusoidal embeddings). (Press et al., 2022; Chi et al., 2022). In contrast, RPE usually performs better length extrapolation since it is easier to construct the additional temporal bias terms.

heading: 2.3 Windowed and Sparse Attention
text: We will see later that ALiBi can be viewed as imposing a windowed attention mask on the selfattention matrix, similar to previous transformer models with sparse attention (Beltagy et al., 2020; Zaheer et al., 2020; Ainslie et al., 2020; Gupta and Berant, 2020). Interpreting ALiBi from the perspective of windowed attention allows us to easily calculate the theoretical receptive field of a model.

heading: 2.4 Receptive Field
text: A model’s receptive field is defined as the size of the input region that contributes the most to model outputs. It is often measured in the context of convolution neural networks (Luo et al., 2016; Dai et al., 2017; Araujo et al., 2019; Raghu et al., 2021; Dosovitskiy et al., 2021) and their dilated variants (Oord et al., 2016; Yu and Koltun, 2016; Chang et al., 2017; Beltagy et al., 2020) with the ultimate goal of receptive field size maximization. Even though we focus on transformer language models, we borrow the idea to show that the empirical receptive field coverage of a model is crucial to its length extrapolation performance.

heading: 3 Background and Notations

heading: 3.1 Transformer Language Model
text: Given a sequence of L ∈ {Ltr, Lex} input embeddings {em}Lm=1 in Rd, an R layer transformer language model with H attention heads converts each em into its corresponding query, key, and value
vectors in R d H at each layer:
qm = Wqem, km = Wkem, vm = Wvem,
where Wq, Wk, Wv ∈ R d H ×d are learnable matrices. The resulting vectors are processed by the self-attention module for pre-Softmax logits:
lmn = { 〈qm,kn〉, if m ≥ n − inf, otherwise
followed by the scaled softmax normalization:
am,n = exp(lm,n/
√ d/H)
∑L i=1 exp(lm,i/ √ d/H)
(1)
To be precise, the matrices (W (h)q , W (h) k , W (h) v ), vectors (q(h)m , k (h) m , v (h) m , o (h) m ), and scalars (l (h) mn, a (h) mn) are associated with a head number h. For notation simplicity, we only show the dependency on h when we need it. For example, the output vector o(h)m at position m for head h is:
o(h)m = L∑
n=1
a(h)m,nv (h) n
All the H output vectors are concatenated, denoted by ⊕, and transformed by Wo ∈ Rd×d to obtain om ∈ Rd:
om = Wo(o (1) m ⊕ o(2)m ⊕ · · · ⊕ o(H)m )
A layer normalization (Ba et al., 2016) on om, i.e. LayerNorm(om), gives the input embedding to the next layer. After R layers of propagation, the last om is transformed by V ∈ Rv×d and normalized by Softmax to get the distribution p ∈ Rv over vocabulary size v:
p = Softmax(V om) (2)
We set R = 12, H = 12, d = 768, and Ltr = 512 for all experiments reported in this paper.

heading: 3.2 ALiBi
text: ALiBi modifies lm,n to be:
lmn = { 〈qm,kn〉 − 12h (m− n), if m ≥ n − inf, otherwise
(3) The range of h is n · 8H , where n = {1 . . . H}.

heading: 3.3 Windowed Attention
text: If the windowed attention has a size w, then:
lmn = { 〈qm,kn〉, if n+ w > m ≥ n − inf, otherwise

heading: 3.4 Evaluation of Length Extrapolation
text: We prepare N = 1000 text segments of length Lex > Ltr from the evaluation dataset. For each segment, we alter the number of previous tokens ranging from 1 to Lex−1 of the last token and only calculate its perplexity:
PPL = exp
( 1
N
N∑
i=1
− log pi ) ,
where pi is the predicted probability from Eq. (2) of the last (Lex-th) token in the i-th segment. This ensures that the same set of tokens is always used for perplexity calculation and only their number of previous tokens is varied, see Figure 3.2

heading: 4 ALiBi and Windowed Attention
text: Here, we alter the slope ( 1 2h ) of ALiBi to check if the length extrapolation property persists and reveal the connection between ALiBi and windowed attention. We present three experiments on two datasets, ArXiv and OpenWebText2 (Appendix A), to ensure that the observations are consistent across different text domains, shown in Table 1 and 4.
2There exists another evaluation protocol named nonoverlapping subsequences adopted in the main experiment tables of ALiBi (Press et al., 2022). It is not the most suitable protocol for length extrapolation evaluation as it suffers from the “early token” curse. Please refer to Appendix B of ALiBi (Press et al., 2022) for details.

heading: 4.1 Slope Shift (Shift all h by ∆)
text: We first investigated whether slope diversity (each attention head has one slope) is the key to length extrapolation. We shift h by a fixed amount ∆ and find that the model, unfortunately, fails to extrapolate beyond a certain quantity. This implies that diversity itself might not be the deciding factor, but that the actual slope value is more important.

heading: 4.2 Slope Equalization (Same h for all heads)
text: To identify the slope magnitude that enables length extrapolation, we set all slopes to be the same instead of the original geometric sequence. We then steadily increase the slope value from 0 to 8 and find that only large slopes ( 1
2h ), or equivalently
small h, allow a model to extrapolate well. Large slopes implicitly enforce a narrow windowed bias on the self-attention matrix such that distant tokens cannot interact with each other.

heading: 4.3 Windowed Attention (Size w)
text: We make the implicit window effect explicit as shown by Eq. (3), which is also adopted by Longformer (Beltagy et al., 2020). We define the windowed attention size to be w. The model underperforms at small w and diverges on long Lex at large w. The same trend holds in the first two experiments when h is too small or large.

heading: 4.4 Other Observations
text: First, ALiBi does not in fact extrapolate since its perplexities all increase instead of staying the same when Lex > Ltr. In contrast, windowed attention models are extrapolatable up to w = 100. Second, we can clearly see that once Lex passes a certain threshold, the perplexity either remains the same or explodes. This suggests that the model is either ignoring tokens beyond a certain length (same)3 or not using it properly (explosion). In the next
3A limited but similar observation was made in Appendix B.2 of ALiBi (Press et al., 2022).
section, we will use the concept of receptive field to explain these observations.

heading: 5 Receptive Field Measurement
text: Following the definition of windowed attention size w, an R layer transformer has a theoretical receptive field (TRF) of wR, which is the maximum number of tokens that contribute to the prediction of the next token. In practice, a neural model often uses a subset of TRF, named empirical receptive field (ERF). While previous work (Luo et al., 2016; Dai et al., 2017; Araujo et al., 2019; Raghu et al., 2021; Dosovitskiy et al., 2021; Beltagy et al., 2020) aims to increase ERF to match TRF, we show that decreasing ERF could serve as one feasible approach to enable successful length extrapolation.
Consider the case where TRF≤ Ltr: This model can extrapolate easily because its TRF is fully covered and trained. Concretely, if we set R = 12, Ltr = 512 in Table 1 and 4, we know that as long as w < 42.6 = 512/12, TRF will be fully covered by Ltr. Surprisingly, the model is still able to extrapolate up to w = 100, leading to a TRF of 100 ∗ 12 = 1200 512. This can be explained by the ERF and TRF discrepancy discussed above; this calls for the need to quantify ERF.

heading: 5.1 Quantifying Empirical Receptive Field
text: We first calculate the normalized gradient (Luo et al., 2016) of each input token w.r.t the prediction of the next token:
sm = ‖gm‖2∑Lex n=1 ‖gn‖2 ,
where gm is the gradient vector of the input embedding em. We then calculate the cumulative sum as:
cm =
Lex∑
n=m
sn, 0 ≤ cm ≤ 1,
Visualizations of cm for the slope shift and windowed attention experiments are shown in Fig-
0 512 1024 1536 2048 Position
0.0
0.2
0.4
0.6
0.8
1.0
=-3 =0 =2 =4 =6 =8
Figure 5: Cumulative normalized gradient on ArXiv when predicting the next (2048-th) token.
ures 4 and 5. We define the ERF of a model as:
ERF = min{m | cm > 0.99}.
Figure 4 demonstrates how we derive the model’s ERF when it is predicting the 2048-th token. For models with w ∈ [40, 80, 100], the most recent Lex = Ltr = 512 (1536-th to 2047-th) covers more than 99% of the total (1.0) normalized gradient, so their ERF is smaller than 512. In contrast, models with w ∈ [120, 160, 320] have ERF = 768, 1024, and 1536 tokens, respectively. Since Ltr = 512 does not fully cover their ERFs, they fail to extrapolate well.
We next focus on the more complex Figure 5, in which neither of the configurations reaches 0.99 within the most recent Ltr = 512 tokens. Generally, this explains why the perplexity often bumps up when Lex goes from 512 to 1024: Models cannot perfectly process more tokens than they were trained on. If we take a closer look, the ∆ = −3 model has the strongest windowing effect and the smallest ERF=768 tokens, therefore its perplexity plateaus the soonest at Lex = 1024 in Table 1. The remaining models all need ERF=2048 tokens to reach cm = 0.99, which explains why their perplexities become stable only after Lex = 2048 (Table 1). For ∆ ∈ [6, 8] models specifically, the difference between Ltr and ERF is too large to be handled, resulting in exploded perplexities.

heading: 5.2 Fixing Failed Cases
text: We fix the failed cases in Table 1 section 1 (varying ∆) and section 3 (varying w) by increasing Ltr to cover their ERFs. We increase Ltr to 1024 for
windowed attention with w = 160; For shifted ALiBi with ∆ = 6, we need Ltr = 2048 tokens. Table 2 shows that both are now able to maintain stable perplexities.

heading: 5.3 Analyses of Sinusoidal and Rotary
text: Sinusoidal (Vaswani et al., 2017) constructs the positional embedding at position m and ∀i ∈ [1, d/2] as:
pm,2i = sin ( m
100002i/d
) ,
pm,2i+1 = cos ( m
100002i/d
) (5)
They will be added with the input embeddings {em}Lm=1 followed by the query and key transformations as shown in Eq. (4). Unlike addition, Rotary (Su et al., 2021) multiplies each token embedding em with a position-specific rotation matrix Rmem.
What could cm tell us when it is applied to the non-extrapolatable Sinusoidal and Rotary positional embeddings? As we can see in Figure 6 and 7, they both fail to focus on the most recent Ltr
0 512 1024 1536 2048 Position
0.0
0.2
0.4
0.6
0.8
1.0 128 512
Figure 7: Cumulative normalized gradient of Sinusoidal on ArXiv when predicting the last (2048-th) token with Ltr ∈ [128, 512].
(Wq(em + pm)) >(Wk(en + pn)) = (4) e>mW > q Wke
> n︸ ︷︷ ︸
semantic info.
+ e>mW > q Wkpn + p > mW > q Wken + p > mW > q Wkpn︸ ︷︷ ︸
mixture of semantic and positional info.
≈ e>mW>q Wke>n︸ ︷︷ ︸ semantic info. + p>mpn︸ ︷︷ ︸ positional info.
tokens because neither of their formulations guarantees a Ltr-bounded receptive field. Figure 7 tells additional stories: To predict the last token (2048- th), Sinusoidal focuses on the 512-th token when Ltr = 512 and the 128-th token when Ltr = 128 as indicated by the sudden jump on their normalized gradient plots. This is because the model has only seen at most Ltr positional embeddings and overfitted on them, which provides explicit evidence to the Sinusoidal, or APE in general, overfitting hypothesis made by the author of ALiBi4. It also explains why RPE is a better choice for length extrapolatable transformers: They cannot overfit on the positional embeddings.

heading: 6 A New RPE for Length Extrapolation

heading: 6.1 Introduction to Sandwich
text: We fix the overfitting issue of Sinusoidal by transforming it into a new RPE, Sandwich, shown in Eq. (4). Specifically, we drop the cross terms and keep only the inner product of two positional embeddings5 at m and n. Now p>mpn with m,n ∈
4https://twitter.com/OfirPress/status/ 1435690039925567489
5We set pm,n to 2d as doing so gives better empirical performance; it only needs to be computed once before training.
[1, L] become the temporal bias terms of Sandwich:
p>mpn = d̄/2∑
i=1
sin
( m
100002i/d̄
) sin ( n
100002i/d̄
) +
cos
( m
100002i/d̄
) cos ( n
100002i/d̄
)
=
d̄/2∑
i=1
cos
( m− n
100002i/d̄
)
A similar observation was previously made in a context different from length extrapolation (Yan et al., 2019).
The largest value of p>mpn happens at the point where m−n = 0, which gives the maximum value of d̄/2. To align Ltr with the ERF of Sandwich, we need to further check that p>mpn demonstrates a similar windowed attention effect as ALiBi. This can be done by subtracting all p>mpn by d̄/2 and further dividing them by a set of predefined compression ratios. for the sake of simplicity, we set the compression ratios to be the same as ALiBi’s h = n · 8H with n ∈ {1 . . . H}:
p>mpn − d̄/2 h
(6)
Eq. (6) is added after the scaled softmax is done in Eq. (1). Figures 8 and 9 show a visualization of
0 2000 4000 6000 8000 10
8
6
4
2
0
Figure 9: We plot the last row in Figure 8. The red curve is the least-squared fitted log function: y = −0.825 · log(|m− n|) + 1)− 0.8 with m = 8192 in this example.
Sandwich when h = 8. Sandwich indeed has the same decay-with-distance pattern as ALiBi.6
Note that we deliberately decouple this d̄ from d in Eq. (5) since we treat d̄ as a hyperparameter that controls the shape of Sandwich. A larger d̄ leads to a stronger windowed attention effect as shown in Figure 10. We set d̄ = 128 in this work for all the experiments. We also experiment with smaller and larger d̄ and only find worse performance. Finally, readers can find the reference Python implementation in Appendix E.

heading: 6.2 Experiments and Discussion
text: To verify the performance of Sandwich, we train a transformer language model following previous
6Fun fact: We imagine different compression ratios as the ways we eat sandwiches: For a huge sandwich, we have to squeeze it more to fit in our mouths!
work (Press et al., 2022; Chi et al., 2022). Table 3 presents the results; the left part contains all models without learnable parameters, and the right part contains models with learnable parameters. These numbers should not be compared across sections.
In general, models on the right achieve lower perplexities across the three datasets. This is expected as they can adapt to individual datasets more easily thanks to the additional learnable parameters. However, there is no free lunch: They often consume more GPU memory and run much slower. For example, T5 is 10% slower than Sandwich during the training stage. Note that Sandwich can also be equipped with learnable parameters such as learnable compression ratios h; this is left to future work. We now shift our focus to the left section. When Lex = Ltr = 512, Sandwich is comparable to other models except that Rotary performs a bit better on OpenWebText2. Once we increase Lex, Sandwich begins to reveal its advantages: On ArXiv and GitHub, it is consistently better than all the baselines but only marginally worse than ALiBi when Lex ≥ 4096 on OpenWebText2.
It is worth mentioning that Sandwich is the first parameter-free RPE that truly makes use of distant token information beyond Ltr = 512. To see this, notice that lower (better) perplexities occur at Lex > Ltr = 512. The gradient analysis tool in §5.1 further corroborates this in Figure 11, which reveals a receptive field pattern distinct from that of ALiBi and windowed attention. Even though Sandwich allocates about 60% of the total cumulative gradient on the most recent Ltr = 512 tokens, distant tokens beyond Ltr still contribute substantially
to the model prediction. Why do ALiBi and windowed attention need to have their ERFs covered by Ltr while Sandwich does not? To answer this question, we revisit Figure 9 and approximate (least-squared) the original temporal bias pattern using a log curve, which gives a snug fit7: y = −0.825 · log (1 + |m− n|)− 0.8. Table 3 shows its language modeling performance under the “smoothed” column. Pictorially, the log curve decays relatively fast when two tokens are nearby and plateaus when the distance between them increases. In other words, tokens that are far away from the last one (m = 8192) share similar temporal biases, possibly leading to beneficial averaging and denoising effects. Note that the averaging effect does not come out of thin air during the extrapolation stage: The almost linear segment ranging from 1536 to 1792 suggests that Sandwich was trained to perform averaging within Ltr; it just needs to average over more historical tokens when
7In the actual implementation, we fit the curve using the most recent 50 points of Sandwich. The reason is because the most recent tokens are more important, and we want them to be closer to the original Sandwich.
it extrapolates to longer Lex. In contrast, ALiBi’s linear bias lacks the middle ground to learn the averaging behavior: It either decays so fast that distant tokens are masked out or so slow that the ERF becomes much greater than Ltr. The averaging hypothesis also explains why Sandwich, KERPLE, and T5’s perplexities go up in Table 3 instead of continuing to decrease after some Lex (4096 on ArXiv for example): While averaging and denoising improve performance, doing so over too many historical tokens (very large Lex) will reintroduce noises.

heading: 6.3 Connection to KERPLE and T5
text: KERPLE (Chi et al., 2022) has the formulation of c − r1 · log (1 + r2|m− n|). The −0.8 in our fitted log curve term can be absorbed by c, as Softmax is shift-invariant, and if we set r1 = 0.825 and r2 = 1, Sandwich becomes a special case of KERPLE. T5 (Raffel et al., 2020) adopts the logbinning strategy that assigns distinct bins to nearby tokens whereas distant tokens all share the same bin. In spirit, T5 treats distant tokens similarly to Sandwich. Figure 11 verifies that all three of them
share a similar empirical receptive field pattern.

heading: 7 Conclusion
text: In this paper, we first establish the connection between ALiBi and windowed attention through their constructions and language modeling performance. We then develop a cumulative normalized gradient tool to measure the empirical receptive field. It shows that length extrapolation of ALiBi and windowed attention is possible when the training sequence length covers the empirical receptive field. It also reveals the models’ limitation of not utilizing information beyond the training sequence length. Fortunately, this is overcome by our new relative positional embedding, Sandwich, which is simplified from the earliest proposed Sinusoidal positional embedding. Finally, Sandwich demonstrates a log-decaying temporal bias pattern similar to that previously seen in the design of KERPLE and T5, and such pattern is likely to be the secret to successful length extrapolation. Together these findings supports more effective design of future extrapolatable transformer language models.
Limitations
Although Sandwich, KERPLE, and T5 use information beyond training sequence length, their receptive fields still highly favor the most recent tokens. While this recency bias is beneficial to the modeling of human-written text, it is problematic in other scenarios.
Let us consider the task of parity prediction: A model needs to predict whether a bit string has
an even or odd number of ones. For example, the parity of [1, 1, 0, 1] is odd (or 1) and the parity of [1, 0, 1, 0] is even (or 0). Unlike human-written text, every single bit is equally important. Transformer language models with current RPEs still struggle on this simple task (Anil et al., 2022). Its difficulty can be explained by the recency bias effect that we described. Devising a new positional embedding or transformer model architecture that solves this problem is a promising direction for future work.
Ethics Statement
Our work advances the understanding of positional embeddings adopted in almost all transformer models. In addition, our proposed new positional embedding significantly reduces energy consumption and training cost thanks to its length extrapolation property. Finally, our work lays the groundwork for developing future transformers that are greener and more cost-efficient enabled by improved length extrapolation. Inappropriate usage of our technique might have negative societal impacts. These include the ethical challenges of improper text generation and privacy issues inherent in the data collection process. These implications apply to any natural language processing research and are not unique to this specific work.

heading: Acknowledgment
text: The authors acknowledge the support from Boeing (2019-STU-PA-259), Amazon (CC ADV 00474341 2021 TR), NSF MRI Award 1919452, and Princeton Research Computing.

heading: A Results on OpenWebText2
text: Table 4 includes the three experiments conducted in §4 on OpenWebText2. Their corresponding receptive field plots are shown in Figure 12 and 13.

heading: B Efficient Inference
text: Although ALiBi might not be using token information further than Ltr, it has the nice property of efficient inference (Press, 2022). Tables 1 and 4 show that ALiBi perplexities stay constant when Lex ≥ 2048. This suggests a cache window size w̄ = 2048 for inference. The generation of the first w̄ tokens remains the same, and we can still cache all qm, km, and vm vectors for m ∈ [1, 2048]. When it comes to generating the w̄ + 1-th token, we simply discard the first cached q1, k1, and v1 and use the rest of w̄ − 1 tokens along with the newly added token to perform self-attention. If we want to generate a length Lex text snippet, the complexity is O(w̄×Lex) instead ofO(L2ex). This complexity is also better than that of an APE model, which isO(w̄2×Lex) since an APE model needs to completely re-encode the previous w̄ vectors when generating new tokens following the first w̄ ones.
We implement the process discussed above to verify that ALiBi indeed allows for efficient inference. The results, along with ones for Sandwich, are presented in Table 5. Both ALiBi and Sandwich permit efficient inference by setting w̄ = 2048. It is worth pointing out that the performance of Sandwich at Lex = 4096 becomes a bit worse compared to that in Table 3. This is more evidence that Sandwich is using longer than Ltr token information.

heading: C Scientific Artifacts
text: We use the gpt-neox library (Andonian et al., 2021) under Apache-2.0 license and the datasets (Gao et al., 2020) released by the authors of gpt-neox. The codebase and datasets (Table 6) are publicly released for research purposes. The steps taken to protect the privacy and anonymization are discussed in Gao et al. (2020) section 6 and 7. Finally, Gao et al. (2020) section 5 also discusses the distribution and statistics of the datasets used in this work.
D Implementation Details
The configurations and hyperparameters are outlined in Table 7. The pretraining takes 5 hours on
a single NVIDIA A-100 GPU. We do not tune any hyperparameters and just use the default ones.
0 500 1024 1536 2048
0.2
0.4
0.6
0.8
1.0 0.94
-3 0 2 4 6 8
Figure 13: Cumulative normalized gradient on OpenWebText2 when predicting the last (2048-th) token. Shift all h by ∆ =
Lex OpenWebText2 Arxiv GitHub Sandwich ALiBi Sandwich ALiBi Sandwich ALiBi

heading: 16384 24.1 23.5 5.35 5.59 2.81 3.01

heading: 8192 24.1 23.5 5.35 5.59 2.81 3.01

heading: 4096 23.9 23.5 5.31 5.59 2.79 3.01

heading: E Python Implementation of Sandwich
text: import numpy as np
base = 1e4 heads = 12 seq_len = 8192 positions = np.arange(seq_len)[..., None] bar_d = 128 # This is the hyperparameter of Sandwich i = np.arange(bar_d // 2)
pos_embs = np.concatenate([np.sin(positions / base ** (2 * i / bar_d)), np.cos(positions / base ** (2 * i / bar_d))], axis=-1)
sandwich = np.matmul(pos_embs, pos_embs.T) compression_ratio = np.arange(1, heads + 1) * 8 / heads multi_head_sandwich = sandwich[None, ...] / compression_ratio[..., None, None]
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations section

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: abstract and section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 6

heading: 3 B1. Did you cite the creators of artifacts you used?
text: appendix C

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: appendix C

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? appendix C

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? appendix C

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? appendix C
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. appendix C
C 3 Did you run computational experiments? section 6

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? section 6 and appendix D
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? section 6 and appendix D

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 6
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Together these findings supports more effective design of future extrapolatable transformer language models. Limitations
Although Sandwich, KERPLE, and T5 use information beyond training sequence length, their receptive fields still highly favor the most recent tokens. While this recency bias is beneficial to the modeling of human-written text, it is problematic in other scenarios. Let us consider the task of parity prediction: A model needs to predict whether a bit string has
an even or odd number of ones. For example, the parity of [1, 1, 0, 1] is odd (or 1) and the parity of [1, 0, 1, 0] is even (or 0). Unlike human-written text, every single bit is equally important. Transformer language models with current RPEs still struggle on this simple task (Anil et al. , 2022). Its difficulty can be explained by the recency bias effect that we described. Devising a new positional embedding or transformer model architecture that solves this problem is a promising direction for future work