Title: The Impact of Positional Encoding on Length Generalization in Transformers
Venue: NeurIPS 2023
Accepted: yes
Keywords: Transformers

Positional Encoding

Length Generalization
Abstract: Length generalization, the ability to generalize from small training context sizes to larger ones, is a critical challenge in the development of Transformer-based language models. Positional encoding (PE) has been identified as a major factor influencing length generalization, but the exact impact of different PE schemes on extrapolation in downstream tasks remains unclear. In this paper, we conduct a systematic empirical study comparing the length generalization performance of decoder-only Transformers with five different position encoding approaches including Absolute Position Embedding (APE), T5's Relative PE, ALiBi, and Rotary, in addition to Transformers without positional encoding (NoPE). Our evaluation encompasses a battery of reasoning and mathematical tasks. Our findings reveal that the most commonly used positional encoding methods, such as ALiBi, Rotary, and APE, are not well suited for length generalization in downstream tasks. More importantly, NoPE outperforms other explicit positional encoding methods while requiring no additional computation. We theoretically demonstrate that NoPE can represent both absolute and relative PEs, but when trained with SGD, it mostly resembles T5's relative PE attention patterns. Finally, we find that scratchpad is not always helpful to solve length generalization and its format highly impacts the model's performance. Overall, our work suggests that explicit position embeddings are not essential for decoder-only Transformers to generalize well to longer sequences.

The Impact of Positional Encoding on Length Generalization in Transformers

Abstract

Length generalization, the ability to generalize from small training context sizes to larger ones, is a critical challenge in the development of Transformer-based language models. Positional encoding (PE) has been identified as a major factor influencing length generalization, but the exact impact of different PE schemes on extrapolation in downstream tasks remains unclear. In this paper, we conduct a systematic empirical study comparing the length generalization performance of decoder-only Transformers with five different position encoding approaches including Absolute Position Embedding (APE), T5's Relative PE, ALiBi, and Rotary, in addition to Transformers without positional encoding (NoPE). Our evaluation encompasses a battery of reasoning and mathematical tasks. Our findings reveal that the most commonly used positional encoding methods, such as ALiBi, Rotary, and APE, are not well suited for length generalization in downstream tasks. More importantly, NoPE outperforms other explicit positional encoding methods while requiring no additional computation. We theoretically demonstrate that NoPE can represent both absolute and relative PEs, but when trained with SGD, it mostly resembles T5's Relative PE attention patterns. Finally, we find that scratchpad is not always helpful to solve length generalization and its format highly impacts the model's performance. Overall, our work suggests that explicit position encodings are not essential for decoder-only Transformers to generalize well to longer sequences.

1 Introduction

The ability to generalize from smaller training context sizes to larger ones, commonly known as length generalization, is a major challenge for Transformer-based language models (Vaswani et al., 2017; Deletang et al., 2023; Zhang et al., 2023). Even with larger Transformers, this issue persists (Brown et al., 2020; Furrer et al., 2020). With larger context sizes, a model can benefit from more in-context-learning examples, higher numbers of reasoning steps, or longer text generation. However, training a Transformer with a larger context size can be excessively slow and memory-intensive. This is even more pronounced in the recent paradigm of model finetuning on instruction-following datasets (Wei et al., 2022a; Chung et al., 2022; Ouyang et al., 2022). It is not only infeasible to train the model on all possible context lengths, but also the number of training examples drops dramatically as the sequence length increases requiring the model to generalize from finite and shorter-length training examples. In this work, we focus on the effect of positional encoding on length generalization in the ' decoder-only " Transformers on various tasks trained from scratch. Figure 1 summarizes our finding that using no positional encoding is better than using explicit positional encodings.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Mean Reciprocal Rank (Higher is better

)

Rotary

ALiBi

T5's Relative PE

0.22

0.33

0.50

0.55

0.69

Positional Encodings

used in GPT3-style

LLMs:

BLOOM (2022a),

PaLM (2022),

and LLaMA (2023).

→

Positional encoding (PE) seems to be a major factor in the length generalization of Transformers as the model has to systematically encode tokens in all possible positions. The original Transformer architecture (Vaswani et al., 2017) used non-parametric periodic functions to represent absolute position embeddings (APE) in a systematic manner, but further studies have shown that these functions are inadequate for length generalization (Ontanon et al., 2022). The prevailing belief is that relative PEs (Shaw et al., 2018; Raffel et al., 2020) are more effective in length generalization than APE variants (Ontanon et al., 2022; Csordás et al., 2021). However, Press et al. (2022) has shown that even Transformers with relative PEs, such as Rotary (Su et al., 2021), can be poor at length generalization. But the evaluation of PEs often relies on language modeling perplexity as a key metric (Haviv et al., 2022; Press et al., 2022) which does not always align with the performance on downstream tasks (Tay et al., 2022). This raises important questions: what exactly is the influence of positional encoding on length generalization at downstream tasks ? Moreover, early empirical evidence shows that decoderonly Transformers without explicit position information (Tsai et al., 2019; Haviv et al., 2022) can perform as well as existing PEs in in-distribution settings, but its effects on length generalization and downstream performance are unclear.

Recently, asking models to emit intermediate computation steps into a scratchpad, also referred to as chain-of-thought , has been adopted to improve the length extrapolation in Transformers (Nye et al., 2021; Wei et al., 2022b). These techniques are architecture-independent and can be used with any PE method. However, it remains an open question whether these techniques, at least in regard to length generalization, render the choice of PE irrelevant, especially given that model performance is highly sensitive to the scratchpad format (Bueno et al., 2022; Akyurek and Akyurek, 2022).

To answer these questions, we conduct a systematic study on the length generalization of decoderonly Transformers, popularized by the GPT-family of models (Radford et al., 2019), with the most commonly used positional encoding schemes, both with and without scratchpad. Specifically, we evaluate APE (Vaswani et al., 2017), T5's Relative PE (Raffel et al., 2020), ALiBi (Press et al., 2022), Rotary (Su et al., 2021) and no positional encoding (NoPE) on a battery of reasoning and mathematical tasks. Our results show that:

Most commonly used positional encoding methods, including ALiBi, Rotary, and APE, are ill-suited for length generalization in downstream tasks and are outperformed by T5's Relative PE.

Transformers without positional encoding (NoPE) outperform all explicit positional encoding schemes. They achieve this without computing additional terms in the attention mechanism (in contrast to explicit PEs).

We show that NoPE is theoretically capable of representing both absolute and relative PEs. But empirically, it is closer to the relative encoding scheme similar to T5's Relative PE.

Scratchpad is not always helpful for length generalization and its format highly impacts the performance. The attention distributions reveal that NoPE and T5's Relative PE encourage attending to both long and short-range positions, ALiBi to recent positions, and Rotary and APE to no particular positions.

2 Background: Positional Encoding in Transformers

Transformers, in contrast to sequential models such as RNNs, are parallel architectures that employ positional encoding to help encode word order. The most common choices for positional encoding are either absolute , where each absolute position (e.g. 1, 2, 3, ...) is directly represented, or relative , where the distance between tokens is used as positional information. In this section, we briefly review the popular encoding methods used in Transformers (Refer to Appendix B for more formal details).

Absolute Position Embedding (APE), embeds each absolute position i into position vector p i and adds word embeddings to their corresponding p i before feeding them to the model. The non-parametric variant of APE uses periodic functions such as sine and cosine to generate embeddings for any position i (Vaswani et al., 2017). On the other hand, a learned version of APE, used in GPT3 (Brown et al., 2020) and OPT (Zhang et al., 2022), trains the position embeddings along with the model parameters, and it cannot generate a position embedding for unseen positions, so the context window is set to a fixed length.

T5's Relative bias , first maps the relative distance ( i -j ) between tokens at positions i and j to a scalar bias value b = f ( i -j ) , where f is a lookup table. The relative bias b (learned during training) then is added to the dot product of the query and key in the self-attention mechanism. The lookup table maps distances larger than a threshold to the same parameter to enable generalization to unseen distances.

Rotary , used in PaLM (Chowdhery et al., 2022) and LLaMA (Touvron et al., 2023), rotates the query and key representations with an angle proportional to their absolute positions before applying the dot product attention. As a result of this rotation, the attention dot product will only depend on the relative distance between tokens, effectively making it a relative positional encoding (Su et al., 2021).

ALiBi , used in BLOOM (Scao et al., 2022a), is similar to T5's Relative Bias but instead subtracts a scalar bias from the attention score. This bias grows linearly with the distance between the query and key tokens. This, in effect, creates a preference toward recent tokens (recency bias).

Note that encoder-only Transformers, such as BERT, become bag-of-words models in the absence of positional encoding. However, decoder-only Transformers with causal attention mask are not permutation invariant and can model sequences even without explicit position information (Tsai et al., 2019). But it is unclear if these models encode position information implicitly or generalize to unseen lengths. We demystify this in Section 5.

3 Model Evaluation

Length Generalization Setup Following Anil et al. (2022), we focus on algorithmic tasks such as copying, addition, etc. For each task, we train on a finite number of examples of up to a certain length and test them on both seen and unseen lengths at inference. We present these problems as sequence-to-sequence tasks, where the input sequence is the problem instance and the output sequence is the solution. Formally, let D = { ( x i , y i ) } denote a dataset of such task where x i is the input and y i is the output sequence. For each task a function λ : D → N can be defined that returns the length bucket of a task instance d ∈ D . This can be the number of tokens or any general notion of length/depth of reasoning. Using this function and a threshold L , we employ samples where λ ≤ L for learning the task and samples where λ > L for evaluating generalization. The performance on each instance is reported as the exact-match accuracy of its answer with the ground truth.

Architecture We use a conventional decoder-only Transformer architecture as a base for all experiments and consider different approaches for encoding positions: Absolute Position Embedding (APE) , ALiBi , Rotary and T5's Relative Bias . We also consider removing the positional encoding ( NoPE ) to better understand its role in length generalization. Note that we use APE with sinusoidal functions (Vaswani et al., 2017) as the learnable variant cannot produce embeddings for unseen positions. Given the absence of publicly available Transformer-based LM trained with aforementioned PEs on the same pretraining data, we opt to train our models from scratch for each task on its training data with the autoregressive language modeling objective log p θ ( y | x ) = ∑ T t =1 log p θ ( y t | x , y 1: t -1 ) . We use the same hyperparameters for all PEs and employ the ' base ' model size configuration, popular in HuggingFace library (Wolf et al., 2020), resulting in ∼ 107M trainable weights (List of all hyperparameters in Appendix D.2).

NoPE

T5

ALiBi

Rotary

APE

0

0.25

0.50

0.75

Mean Rank (

→

)

Primitive Tasks

NoPE

T5

ALiBi

Rotary

APE

Mathematical & Reasoning

NoPE

T5

ALiBi

Rotary

APE

Positional Encoding

Figure 2: Aggregate ranking of positional encoding methods on length extrapolation across three different groups of tasks. No PE and T5's Relative Bias outperform other encoding methods in these categories.

Tasks Our study of length generalization is concentrated on downstream tasks. Particularly, we evaluate the models on three categories (Table 1) of synthetic tasks that have been widely used in the literature to investigate length generalization: (1) Primitive tasks such as Copying and Reversing (Ontanon et al., 2022), (2) Mathematical and reasoning tasks such as Addition (Nye et al., 2021), Polynomial Evaluation, Sorting, Summation (Saxton et al., 2019), Parity (Anil et al., 2022), LEGO (Zhang et al., 2023) and (3) Classical length generalization datasets such as SCAN (Lake and Baroni, 2018) and PCFG (Hupkes et al., 2020). These tasks provide us with complete control over the train-test distribution, while also requiring reasoning and compositionality skills, which serve as fundamental building blocks for more complex tasks. For the first two categories, we generate the corresponding datasets. Specifically, we first sample the length of the task instance from the uniform distribution U (1 , L ) , and then, according to the task's generative process, we sample the input and output sequences. For the test set, we follow the same procedure but sample length from U (1 , 2 L ) to include both seen and unseen lengths. Throughout the paper, unless otherwise stated, we use L = 20 . For the third category of tasks, we use length generalization splits from the corresponding datasets. Table 1 provides an example of each task (More examples in Appendix D.1).

We report the results of our empirical evaluation over ten tasks and three seeds per dataset-PE pair.

4 What Is The Effect of Positional Encoding?

In this section we provide comparative results of positional encodings at length generalization. To provide a holistic view, following Liang et al. (2022), we report the mean ranking of various models in Figures 1 and 2 when compared against each other for all tasks and scenarios. Furthermore, we showcase the accuracy of models evaluated on examples of various lengths in Figure 3. (Detailed results for each task and scenario can be found in Appendix E).

First, we observe that in most tasks, models achieve a perfect or near-perfect accuracy (Figure 3) on the I.I.D. lengths, which indicates that models have no problem fitting to the training data. However, the differences among positional encoding methods become more apparent when we evaluate on lengths that are larger than seen during training. In most extrapolation scenarios, T5's Relative Bias

15

20

25

30

35

40

0

0.2

0.4

0.6

0.8

1

Copy

Positional Encoding

NoPE

T5's Relative PE

ALiBi

Rotary

Absolute Position Embedding

8

10

12

14

16

Addition

20

30

40

SCAN

15

20

25

30

35

40

0

0.2

0.4

0.6

0.8

1

Reverse

8

10

12

14

16

Summation

10

20

30

Problem Length

PCFG

outperforms other explicit positional encodings. ALiBi positions itself in the middle of the pack, while APE and Rotary show poor generalization performance.

Although Rotary is often considered a relative encoding method (Ontanon et al., 2022), our results show that it performs more similarly to APE than to other relative schemes. Moreover, ALiBi, despite its promise for length generalization, underperforms with respect to T5's Relative Bias in most cases. This result aligns with Taylor et al. (2022) who found no significant improvement from ALiBi.

Surprisingly, the NoPE model, which is just a decoder-only Transformer without any positional encoding, performs on par with or even better than the best-performing explicit PE, T5's Relative Bias. NoPE achieves the same level of generalization without any computational overhead since it does not compute any additional term in the attention mechanism. This property has a direct impact on the runtime and memory footprint of the model. For instance, Press et al. (2022) reported that the additional computation incurred by T5's Relative Bias can make the training and inference time of the model almost two times slower than the Transformer with APE.

5 How Does NoPE Represent Positions?

The surprising performance of NoPE model suggests that it capture useful positional information that can also generalize. But, how it does so is the primary question. In the next two sections, we provide theoretical and empirical analysis towards answering this question.

5.1 NoPE can theoretically represent both absolute and relative PEs

Let f θ be a NoPE decoder-only Transformer model, where θ denotes the model parameters. f θ processes the input sequence x = [ <bos> , x 1 , . . . , x T ] by applying a series of layers. Note that since f θ does not have any PE, the input x is not augmented with positional information (e.g. [1 , 2 , . . . , T ] ). Each layer l , consisting of self-attention heads and a feed-forward sub-layer, reads the previous hidden state H ( l -1) and produces the hidden state at layer l : H l . Each head is parameterized by a query W Q , key W K , value W V , and output W O matrices, where W Q , W K , W V ∈ R h × d and W O ∈ R d × h . d and h are the model's hidden state size and attention dimension, respectively. W 1 , W 2 ∈ R d × k.d are the weight matrices of the feed-forward sub-layer.

2

1.5

1

0.5

0

2

Layer #05

Layer #06

Layer #07

Layer #08

0

0.5

1

1.5

D

(

NoPE

,

M

)

Layer #01

Layer #02

0

10

20

30

40

50

0

0.5

1

1.5

Layer #03

0

10

20

30

40

50

Example Length

Layer #04

0

10

20

30

40

50

Example Length

0

0.25

0.50

0.75

1

1.25

1.50

Average

D

(

NoPE

,

M

)

D

(

NoPE

,

M

)

Positional Encoding (

M

)

NoPE'

T5's Relative PE

ALiBi

Rotary

Absolute Position Embedding

0

0.5

1

1.5

0

10

20

30

40

50

0

0.5

1

1.5

2

Layer #09

0

10

20

30

40

50

Layer #10

0

10

20

30

40

50

Layer #11

0

10

20

30

40

50

Example Length

Layer #12

Theorem 1 ( Absolute Encoding ) . Let x be an input sequence of length T +1 to the model. Then, the first layer of f θ can recover absolute positions [1 , . . . , T + 1] in the hidden state H (1) . That is, there exist W Q , W K , W V , W O , W 1 , and W 2 such that the self-attention and feedforward operations in the first layer compute absolute positions and write it to the next hidden state.

We refer to Appendix C.1 for the complete proof of Theorem 1. This theorem shows that stochastic gradient descent (SGD) can potentially learn to recover absolute positions in NoPE Transformers. Next, we demonstrate how relative PE can be implemented in subsequent layers:

Theorem 2 ( Relative Encoding ) . Suppose that the hidden state H (1) contains absolute positional information, as stated in Theorem 1, and assume that it is not overwritten by any subsequent layers. Then, the self-attention in all subsequent layers can implement a relative positional encoding: there exists a parameterization of f θ such that, for l ≥ 2 , the attention dot product between query q n and key k m at positions n and m can be expressed as:

where f cnt is a function of their content, and f rel is a function of their relative distance.

Appendix C.2 provides the complete proof of Theorem 2. Our theoretical results suggest that SGD can choose between relative and absolute encoding in NoPE Transformers. But, what mechanism SGD learns in practice is not clear. We next investigate this question empirically.

5.2 NoPE learns to use relative PE in practice

In order to explore the mechanisms that NoPE employs in practice, we conduct a quantitative analysis by comparing its attention pattern to models trained with different positional encoding techniques. The hypothesis is that if NoPE utilizes a similar algorithm to other PEs, then the attention patterns of these models should be quite similar.

To this end, we feed the same input to both models and, at layer l , we compute the minimum distance between the attention distribution of any heads in the first model and any head in the second model. Formally, let P t = p ( k | q t ) be a probability distribution produced by a causal selfattention head for query at position t , over the keys k ∈ [ k 1 , . . . k t ] in a given transformer layer. Over a sequence of length T , we define the similarity between two heads P and Q as D AT (P , Q) = 1 T ∑ T t =1 D JSD (P t || Q t ) which averages the Jensen-Shannon divergence (JSD) between the two heads over all positions. For the distance of two models A and B at layer l , we take the minimum distance

between all pairs of attention heads in the corresponding layer:

where A l and B l are the attention heads in layer l of models A and B respectively. We empirically measure the distance between NoPE and other positional encoding schemes after training. Specifically, we sample examples from each length bucket and feed them (the concatenation gold input and output) to compute the attention maps and the distance using Equation (2). We also consider the distance between different seeds of NoPE as a baseline. Figure 4 shows the distance per layer for the first four layers. (later layers show similar trends Figure F.7). We find that NoPE's attention patterns are most similar to that of T5's Relative PE, and least similar to APE and Rotary. The same trend can be observed across all layers and length buckets, and even when averaged across all layers. These results potentially suggest that a Transformer model without positional encoding, trained with stochastic gradient descent learns to represent positions in a way similar to T5's Relative PE, which is a relative positional encoding scheme.

6 Does Scratchpad Render The Choice of Positional Encoding Irrelevant?

In scratchpad/CoT prompting, the model generates intermediate computations required to reach the final answer as explicit parts of the output. Such mechanisms, in effect, provide a direct decomposition and storage for intermediate values, which has been shown to improve the length generalization of Transformers even at small scale (Nye et al., 2021). Since scratchpad only modifies the model's input and output (not the architecture), it is unclear and unexplored how architectural choices such as positional encoding affect the length generalization in the presence of scratchpad. To answer this question, we train all PEs with and without scratchpad on the mathematical and reasoning group of tasks, and compare their performance.

Moreover, the decision of how to represent the intermediate computations in the scratchpad, i.e. the scratchpad format, is an important design choice that has a non-trivial impact on the model's performance (Bueno et al., 2022).

To account for those, we consider five components in each step of scratchpad: <input> , <computation> , <output> , <variable_update> , and <remaining_input> (Figure 5). In our experiments, we create different variations of scratchpad format by enabling or disabling each component, which allows us to systematically study their impact. 1 Figure 6 summarizes our results. Similar to the remarks made by (Nye et al., 2021; Anil et al., 2022) we observe that across all PEs and regardless of the format, scratchpad is beneficial solely for the

Input (

x

)

Compute 5 3

7 2 6 + 1 9 1

7 =

Output (

s

;

y

)

<scratch>

I

For digits 6

and 7,

C

We have (

6 + 7 + carry )

% 10 =

13 % 10 = 13 % 10

O

Which is equal to 3 .

V

We update carry to 13 //

10 = 1.

R

So, the remaining

input is

5 3 7 2

+ 1 9 1

...

</scratch>

The answer is

5 5 6 4 3.

addition task. Additionally, our findings indicate that having a positional encoding with robust length generalization is crucial since scratchpad/CoT alone may not enhance the generalization.

6.1 Which part of the sequence is attended to?

The scratchpad format that is often used (Nye et al., 2021), similar to Figure 5, contains redundant information. One such example is the repetition of the remaining portion of an input ( R ) in each step of the scratchpad. But, the attention can attend to this information directly from the main input. So, it remains unclear which specific part of the scratchpad different PEs rely on to solve the task.

To address this question, we take the models trained with full Format on addition, the case in which scratchpad is helpful across all PEs, and examine their attentions. Specifically, for tokens in

Addition

LEGO

Parity

Polynomial Eval.

Sort

Summation

0

0.25

0.50

0.75

1

Mean Reciprocal Rank (

→

)

No Scratchpad

Full

Full - 'Step Input'

Full - 'Step Computation'

Full - 'Step Output'

Full - 'Intermediate Vars'

Full - 'Remaining Parts'

Minimal (Only Compute+Output)

the output sequence, we calculate the distance d between the current query q t and the attended key k n as ( t -n +1) and subsequently normalize it based on the length of the sequence at the present step. The normalized value is denoted as ¯ d .

Figure 7 depicts the distribution of ¯ d . Values of ¯ d close to 0 indicate attention to tokens near the current position (e.g. current scratchpad step), while values close to 1 signify attention to distant tokens (e.g. the input). NoPE and T5's Relative PE resemble each other and exhibit a bimodal distribution, reflecting both short-range and long-range attention. Conversely, ALiBi (due to its recency bias) strongly favors short-range attention. Rotary, on the other hand, produces a distribution resembling APE, which is more uniformly distributed. Notably, NoPE and T5's RPE are the top-performing PEs in this setup, which suggest the bimodal distribution to be more optimal.

7 Discussion

Practitioners have to make important choices about the nuances of the Transformer architecture like positional encoding before undertaking the costly pretraining process. In the I.I.D evaluation of PEs, we demonstrate similar performance across different PEs, in line with observations of Haviv et al. (2022) and Scao et al. (2022b), which makes the choice of optimal positional encoding challenging.

In our paper, we utilize length generalization in downstream tasks as a mean to assess the expressivity of positional encodings. Our setup, in contrast to the I.I.D.

0

35000

70000

NoPE

0

35000

70000

T5's Relative PE

0

35000

70000

ALiBi

0

35000

70000

Rotary

0

0.25

0.50

0.75

1

Normalized Attended Distance (

¯

d

)

0

35000

70000

Absolute Position Embedding (APE)

evaluation, reveals a clear distinction among approaches of encoding positions. We find that NoPE outperforms explicit PEs, and within explicit PEs, commonly used methods lag behind T5's Relative PE. In fact, the recent release of LLMs (Touvron et al., 2023; Chowdhery et al., 2022) suggests a shift towards adopting Rotary as a replacement for APE in the Transformer architecture. However, our result in Section 4 clearly demonstrates that Rotary marginally outperforms APE at length generalization. Furthermore, it exhibits similar behavior to APE, as shown in Section 6.1, indicating potential susceptibility to the same limitations.

The disadvantages of explicit PEs over NoPE in length extrapolation contribute to the growing evidence that positional encodings pose challenges for Transformers (Sinha et al., 2022; Luo et al., 2021). Our empirical results and theoretical analysis suggest that removing positional encoding holds promise as a modification to the widely used decoder-only Transformer architecture.

Scaling up to 1B models In order to study the behavior of position embeddings at scale, we trained three 1B variants post-submission - ALiBi, Rotary and NoPE - with context length of 1024 tokens on a subset of StarCoder training set (Li et al., 2023). For more details, refer to Appendix F. Our results on language modeling show that at I.I.D all variants have similar perplexity, but at length generalization, Rotary fails to generalize as its perplexity explodes. NoPE and ALiBi generalize similarly to larger context sizes up to almost twice their training context size, and for larger contexts ALiBi is relatively more stable than NoPE (see the discussion on perplexity vs. downstream performance ). Preliminary exploration of fine-tuning the pretrained models, on datasets in Section 3, yielded to identical performance among PE variants as the training context size of the 1.3B models is much larger than instance lengths in our datasets. A comprehensive downstream evaluation of these models remains a subject for future research.

Perplexity vs. downstream Performance Due to human cognitive constraints (Gibson, 1998; Kiyono et al., 2021), language modeling data might encompasses short-range dependencies. The combination of this naturally occurring structure (which can be abundant in internet-based corpora) with the Recency Bias inherent in positional encodings like ALiBi could portray an unrealistic representation of models' length generalization performance. In fact, Chi et al. (2023) recently demonstrated that ALiBi's length generalization performance could be replicated using a window attention mask, where tokens beyond a window size w are masked out. Interestingly, we also observe that T5's Relative PE, which can be regarded as trainable version of ALiBi, learns to attend both large and short range dependencies (Figure F.3). This is in line with Tay et al. (2022) observation and underscores the importance of evaluation setups on downstream tasks as compared to solely relying on perplexity.

8 Related Work

Length Generalization Failure In Transformers The length generalization problem has been a topic of interest in the study of neural sequence models for a long time (Graves et al., 2016; Kaiser and Sutskever, 2016; Lake and Baroni, 2018; Hupkes et al., 2020; Yehudai et al., 2021). Transformers, being state-of-the-art sequence models, have been no exception. A group of studies showed the generalization failure of conventional Transformers with APE on specific datasets such as PCFG (Hupkes et al., 2020), LEGO (Zhang et al., 2023), or CLUTRR (Sinha et al., 2019; Gontier et al., 2020). The length generalization problem has been reported even in pretrained Transformers such as T5 (Furrer et al., 2020) and LaMDA (Anil et al., 2022). Csordás et al. (2021) and Ontanon et al. (2022) study the effect of positional encoding on length generalization but mainly focus on showing relative PE outperforms APEs. Press et al. (2022), on the other hand, propose a new encoding method, ALiBi, and demonstrate that it outperforms popular PEs on extrapolation but only in the context of human language modeling. Most relevant is Deletang et al. (2023)'s recent study on length generalization in various neural sequence models (including RNNs and Stacked-RNNs) for tasks from Chomsky hierarchy. However, they do not focus on the difference among positional encoding or on autoregressive models. Unlike these studies, our work extensively compares length generalization in popular PEs for a wide range of tasks, specifically focusing on autoregressive models, which represent many contemporary LLMs.

Positional Encoding A core component of Transformers is the positional encoding mechanism, which helps the model represent the order of the input sequence. Self-attention mechanism in the encoder of Transformers is order-invariant and requires PE to avoid becoming a bag-of-word model. Many methods have been proposed for this purpose. Originally, Vaswani et al. (2017) introduced absolute positional encoding sinusoidal functions (a learned variant popularized by Devlin et al. (2019)). Relative approach for encoding positional information was further introduced by Shaw et al. (2018), which gave rise to a number of pre-trained LM with relative encodings such as TransformerXL (Dai et al., 2019) and T5 (Raffel et al., 2020) that perform well in length generalization. More recently, Su et al. (2021) takes the concept of sinusoidal functions and suggests a new way of encoding positional information by rotating the hidden representations before applying self-attention. This method, referred to as Rotary , has become a popular choice in the recent LLMs. Press et al. (2022) simplify the T5's Relative encoding and introduced a more efficient variant called ALiBi, while keeping the same or improving extrapolation performance. Decoder-only Transformers, due to their causal attention mask, are not order-agnostic and can operate without explicit positional information.

This was observed early on by Shen et al. (2018) and later explained by Tsai et al. (2019). The observation that Transformers without positional encoding can perform on par with explicit PE has been made in various domains such as machine translation (Yang et al., 2019), language modelling (Irie et al., 2019; Haviv et al., 2022), and even other domains like vision or speech (Likhomanenko et al., 2021). In our work, not only we demonstrate that Transformers can operate without explicit position information, but also we present an important setup where they outperform explicit PEs. Furthermore, we theoretically show how they are capable of learning both absolute and relative encodings.

9 Conclusion

We studied the robustness of different positional encodings, in decoder-only Transformers, at length generalization on various downstream mathematical and reasoning tasks. Our extensive empirical study shows the effectiveness of NoPE, and further demonstrates that widely used explicit PEs are not suited for length generalization. We also prove that NoPE can implicitly learn both absolute and relative positions, but uses the latter in practice. Finally, we find the effectiveness of scratchpad is task-dependent, and is not a reliable solution for length generalization.

Limitations

Our work primarily focuses on positional encodings as a design choice in the Transformers decoder architecture. We could not study how large-scale pretraining affects different PEs because there are no publicly available large language models trained with various PEs under similar conditions. We leave this for future work due to our limited compute budget.

References

Ekin Akyurek and Afra Feyza Akyurek. 2022. Notes on teaching gpt-3 adding numbers.

Ekin Akyurek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2023. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations .

Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, et al. 2023. Santacoder: don't reach for the stars! ArXiv , abs/2301.03988.

Cem Anil, Yuhuai Wu, Anders Johan Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Venkatesh Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. 2022. Exploring length generalization in large language models. In Advances in Neural Information Processing Systems .

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. ArXiv , abs/1607.06450.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya

Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual .

Mirelle Candida Bueno, Carlos Gemmell, Jeff Dalton, Roberto Lotufo, and Rodrigo Nogueira. 2022. Induced natural language rationales and interleaved markup tokens enable extrapolation in large language models. In Proceedings of the 1st Workshop on Mathematical Natural Language Processing (MathNLP) , pages 17-24, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, and Peter Ramadge. 2023. Dissecting transformer length extrapolation via the lens of receptive field analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1352213537, Toronto, Canada. Association for Computational Linguistics.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. ArXiv , abs/2204.02311.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. ArXiv , abs/2210.11416.

Róbert Csordás, Kazuki Irie, and Juergen Schmidhuber. 2021. The devil is in the detail: Simple tricks improve systematic generalization of transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 619-634, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2978-2988, Florence, Italy. Association for Computational Linguistics.

Gregoire Deletang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A Ortega. 2023. Neural networks and the chomsky hierarchy. In International Conference on Learning Representations .

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, et al. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread .

Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Schärli. 2020. Compositional generalization in semantic parsing: Pre-training vs. specialized architectures. ArXiv , abs/2007.08970.

Edward Gibson. 1998. Linguistic complexity: locality of syntactic dependencies. Cognition , 68(1):176.

Nicolas Gontier, Koustuv Sinha, Siva Reddy, and Christopher Pal. 2020. Measuring systematic generalization in neural proof generation with transformers. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual .

Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka GrabskaBarwi´ nska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. 2016. Hybrid computing using a neural network with dynamic external memory. Nature , 538(7626):471-476.

Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. 2022. Transformer language models without positional encodings still learn positional information. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 1382-1390, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Dan Hendrycks and Kevin Gimpel. 2020. Gaussian error linear units (gelus). ArXiv , abs/1606.08415.

Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2020. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research , 67:757-795.

Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019. Language modeling with deep transformers. In Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019 , pages 3905-3909.

Lukasz Kaiser and Ilya Sutskever. 2016. Neural gpus learn algorithms. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings .

Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, and Kentaro Inui. 2021. SHAPE: Shifted absolute position embedding for transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3309-3321, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Brenden M. Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 2879-2888. PMLR.

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, et al. 2023. Starcoder: may the source be with you! ArXiv , abs/2305.06161.

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. ArXiv , abs/2211.09110.

Tatiana Likhomanenko, Qiantong Xu, Gabriel Synnaeve, Ronan Collobert, and Alex Rogozhnikov. 2021. CAPE: Encoding relative positions with continuous augmented positional embeddings. In Advances in Neural Information Processing Systems .

David Lindner, János Kramár, Matthew Rahtz, Thomas McGrath, and Vladimir Mikulik. 2023. Tracr: Compiled transformers as a laboratory for interpretability. ArXiv , abs/2301.05062.

Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: Designing data and methods for effective instruction tuning. ArXiv , abs/2301.13688.

Ziyang Luo, Artur Kulmizev, and Xiaoxi Mao. 2021. Positional artefacts propagate through masked language model embeddings. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 5312-5327, Online. Association for Computational Linguistics.

Maxwell Nye, Anders Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show your work: Scratchpads for intermediate computation with language models. ArXiv , abs/2112.00114.

Santiago Ontanon, Joshua Ainslie, Zachary Fisher, and Vaclav Cvicek. 2022. Making transformers solve compositional tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3591-3607, Dublin, Ireland. Association for Computational Linguistics.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv , abs/2203.02155.

Sejun Park, Chulhee Yun, Jaeho Lee, and Jinwoo Shin. 2020. Minimum width for universal approximation. ArXiv .

Ofir Press, Noah A. Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21:140:1-140:67.

David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical reasoning abilities of neural models. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022a. Bloom: A 176b-parameter open-access multilingual language model. ArXiv , abs/2211.05100.

Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Jason Phang, Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Launay, and Iz Beltagy. 2022b. What language model to train if you have one million GPU hours? In Challenges & Perspectives in Creating Large Language Models .

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) , pages 464-468, New Orleans, Louisiana. Association for Computational Linguistics.

Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018. Disan: Directional self-attention network for rnn/cnn-free language understanding. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 , pages 5446-5455. AAAI Press.

Koustuv Sinha, Amirhossein Kazemnejad, Siva Reddy, Joelle Pineau, Dieuwke Hupkes, and Adina Williams. 2022. The curious case of absolute position embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 4449-4472, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. 2019. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 4506-4515, Hong Kong, China. Association for Computational Linguistics.

Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. Roformer: Enhanced transformer with rotary position embedding. ArXiv , abs/2104.09864.

Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. 2022. Scale efficiently: Insights from pretraining and finetuning transformers. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net.

Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. ArXiv , abs/2211.09085.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models. ArXiv , abs/2302.13971.

Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 4344-4353, Hong Kong, China. Association for Computational Linguistics.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998-6008.

Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. 2022. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proc. of EMNLP , pages 5085-5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022a. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022b. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems .

Gail Weiss, Yoav Goldberg, and Eran Yahav. 2021. Thinking like transformers. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 11080-11090. PMLR.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online. Association for Computational Linguistics.

Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S. Chao, and Zhaopeng Tu. 2019. Assessing the ability of self-attention networks to learn word order. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 3635-3644, Florence, Italy. Association for Computational Linguistics.

Gilad Yehudai, Ethan Fetaya, Eli A. Meirom, Gal Chechik, and Haggai Maron. 2021. From local structures to size generalization in graph neural networks. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 11975-11986. PMLR.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models. ArXiv , abs/2205.01068.

Yi Zhang, Arturs Backurs, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, and Tal Wagner. 2023. Unveiling transformers with lego: a synthetic reasoning task. ArXiv , abs/2206.04301.

0

1000

2000

3000

4000

0

15000

30000

45000

60000

Count

FLAN (CoT)

0

1000

2000

3000

4000

Example Length

Super-Natural Instructions

A Number of instances decreases rapidly as sequence length grows

The recent trend of SFT-RLHF pipeline (Ouyang et al., 2022) relies on finetuning LLMs on the instruction following tasks. However, the training data of these datasets is often skewed towards shorter sequences. Figure A.1 shows the distribution of instruction lengths in two instruction finetuning datasets: FLAN (CoT subset) (Longpre et al., 2023) and Super Natural Instructions (Wang et al., 2022). The median length of instructions in these datasets is quite short compared to the maximum length that exists. Such distribution shape highlights the importance of length generalization in these tasks. In fact, the models are supposed to learn from short instructions and generalize to ones during inference that might be much longer.

B Background

B.1 Preliminaries

In this section, we lay the groundwork and introduce the notation we use throughout the paper. We will refer to this in Appendices C.1 and C.2.

Let f θ be a decoder-only Transformer model, where θ denotes the full set of model parameters. f θ processes the input sequence x = [ x 0 , x 1 , . . . , x T ] and maps it to the output sequence y = [ y 0 , y 1 , . . . , y T ] by applying a sequence of Transformer layers. Note that being decoder-only means the attention mechanism in each layer is causal, i.e. the attention weights are computed based on the previous positions only.

The layer TLayer ( l ) ( H ( l -1) ; θ l ) , consisting of self-attention heads and a feed-forward sub-layer, reads the previous hidden state H ( l -1) and produces the hidden state at layer l : H l , where l is the layer index, and θ l is the set of parameters of the l -th layer. Each hidden state H ( l ) ∈ R d × ( T +1) is matrix where column t , denoted as h ( l ) t , is the hidden state at position t .

Alayer l is parameterized by a set of parameters θ l = { ( W m Q , W m K , W m V , W m O ) m , W 1 , W 2 } , where W m Q , W m K , W m V ∈ R h × d and W m O ∈ R d × h are the query, key, value, and output matrices of the m -th head, respectively. W 1 , W 2 ∈ R d × k.d are the weight matrices of the feed-forward sub-layer. d denotes the model's hidden state size, h is the attention dimension (where h = d # heads ), and k is a multiplier of the hidden state size in the feed-forward sub-layer (it is usually set to 4 in common implementations of the Transformer). Note that we drop the layer index l and the attention head index m where it is clear from the context.

The Transformer layer TLayer ( l ) processes each column of H ( l -1) independently and in parallel to produce the output. The computation of the t -th column of H ( l ) is as follows:

where FF is the feed-forward sub-layer, λ is layer normalization, and a t ∈ R d is the output of the multi-head self-attention sub-layer at position t . Specifically, a t is computed as:

where Attn ( m ) is the m -th attention head. Let o t ∈ R d denote the output of an attention head at position t . Then, o t is computed as:

where ˆ α = softmax( α ) ∈ R ( t +1) , and α is the attention weight vector such that:

where q t = W Q h ( l -1) t ∈ R h , k i = W K h ( l -1) i ∈ R h , and v i = W V h ( l -1) i ∈ R h . ⟨· , ·⟩ denotes the dot product operation.

The feed-forward sub-layer FF( · ) ∈ R d is a two-layer MLP:

where σ is a non-linear activation function (usually ReLU or GeLU (Hendrycks and Gimpel, 2020)). Additionally, λ ( · ) ∈ R d is layer normalization (Ba et al., 2016). Note that we take the additive (Elhage et al., 2021) view of attention heads in Equation (4) instead of concatenate and multiple view (Vaswani et al., 2017) as it is easier to understand and analyze. But, they are mathematically equivalent (Elhage et al., 2021).

The hidden state is initialized with a learned embedding of the input sequence H (0) = W E X , where W E ∈ R d × V is the embedding matrix and X ∈ R V × ( T +1) is the one-hot encoded input sequence. V is the vocabulary size.

B.2 Positional Encoding

Almost all positional encoding methods can be explained and formulated as how they implement the dot product operation in Equation (6). So, in this section, we explain how the dot product ⟨ q t , k i ⟩ is implemented in different positional encoding schemes.

Absolute Positional Encoding (APE) The process of Absolute Positional Encoding (APE) involves assigning a position vector p i to each absolute position i and combining them with word embeddings before inputting them into the model. So, APE first modifies how the hidden state is initialized:

where W P ∈ R d × T is the positional embedding matrix and P ∈ R V p × ( T +1) is the one-hot encoded absolute position sequence. V p is the maximum absolute position. Therefore, the hidden state at column j is:

where e j ∈ R d is the word embedding of token x j and p j ∈ R d is the positional embedding for position j . Then, the dot product for the first layer in Equation (6) is computed as:

In the learned variant of APE, p j ∈ R d is learned during training. In the sinusoidal variant, p j is calculated using a non-parametric function. Specifically, p j is computed as:

where ω i = 1 10000 2 i/d .

T5's Relative PE The Relative bias in T5 is a type of relative positional encoding that initially calculates the relative distance ( t -i ) between tokens at positions t and i . This distance is then transformed into a scalar bias value b and is incorporated into the dot product between the query and key. b is learned during training. Thus, the dot product in every layer can be written as:

where

This function maps the relative distance d to a bucket index, which will be used to look up the weight corresponding to that bucket. B is the number of buckets, D is the maximum distance. It assigns half of the buckets to distances smaller than D 2 with linear spacing and the other half to distances larger than D 2 with logarithmic spacing. The weight for distances larger than D is the same. This is to facilitate generalization to unseen distances. In the original implementation of T5, B = 32 and D = 128 . Following shows an example of the bucket function with B = 5 and D = 6 :





























ALiBi Similar to T5's Relative PE, ALiBi subtracts a scalar bias from the attention score. As the distance between the query and key tokens increases, the bias grows linearly. Specifically, the dot product in every layer can be written as:

where m is head index and C is a constant defined as:

For example, if the number of heads is 8, then we have 1 2 , 1 2 2 , . . . , 1 2 8 (Press et al., 2022).

Rotary The Rotary is a relative PE that applies a rotation to the query and key representations based on their absolute positions before dot product attention. Due to this rotation, the attention dot product relies solely on the relative distance between tokens.

First, we formulate Rotary for model dimension d = 2 . Rotary positional encoding defines the dot product as:

C Proofs

In this section, we provide proof of why NoPE can implicitly learn both absolute and relative positions. We refer the readers to Appendix B.1 for the notation and definitions used in this section.

C.1 Absolute Positional Encoding in NoPE

This section discusses how NoPE can recover absolute positions in the hidden state. Our proof is inspired by Weiss et al. (2021); Lindner et al. (2023) and relies on the causal attention mask in the decoder-only Transformer and the softmax function to recover absolute positions.

Theorem 1 ( Absolute Encoding ) . Let x = [ <bos> , x 1 , . . . , x T ] be an input sequence of length T +1 to the model. Then, the first layer of f θ can recover absolute positions [1 , . . . , T +1] in the hidden state H (1) . That is, there exist W Q , W K , W V , W O , W 1 , and W 2 such that the self-attention and feedforward operations in the first layer compute absolute positions and write it to the next hidden state.

Proof.

Our proof only specifies the weights of a single attention head in the first layer (and additionally the parameterization of feedforward sub-layer). In this parameterization, we only require the first three dimensions of the hidden states. The rest of the heads, as long as they do not override the first three dimensions, can be arbitrary. This does not impose any challenges as Transformers used in practice usually have a very large model dimension d . In the rest, we provide the construction of the weights and then verify that they can recover absolute positions.

First, we construct the word embedding matrix W E ∈ R d × V , where each column is the embedding of a token in the vocabulary. We construct W E such that it always sets the first dimension of every embedding vector to be 1. Additionally, it sets the second dimension to 1 if and only if the token is <bos> . Otherwise, it sets it to zero. The third dimension of all embedding vectors is set to zero. Other dimensions can take any arbitrary values. Without loss of generality, assume <bos> is the first token in the vocabulary, i.e. The first column. Then, we have:

where R tθ is a rotation matrix that rotates x by tθ radians:

for d > 2 , Rotary applies the same approach on every two consecutive dimensions of q t and k i , but with different θ angles. Refer to Su et al. (2021) for the exact formulation.

NoPE NoPE does not explicitly encode positional encodings. So, the dot product in every layer can be written as:

where e d,i ∈ R .

Secondly, for head dimensions h ≥ 1 , we construct the weights W Q , W K , W V , W O of the first attention head in the first layer. Specifically,

W K reads from the first dimension of the hidden state, which is initialized with 1 using the embedding matrix. Since all word embeddings have one in their first dimension, this parameterization will result all key vectors to be the same. Moreover, W V reads from the second dimension of the hidden state, which is initialized with 1 if the token is <bos> . So, the value vector will have 1 in its first dimension only if the corresponding token is <bos> .

W Q can be any arbitrary matrix. W O will write the result of the attention to the third dimension of the hidden state and can be constructed as:

Now, we verify that for any input sequence x = [ <bos> , x 1 , . . . , x T ] , the first layer can recover absolute positions [1 , . . . , T +1] in the hidden state H (1) . We verify this for column t of H (1) . That is, we show that absolute position information is available in the third dimension of h (1) t .

First, we use the word embedding matrix W E to compute the embedding H (0) :

We now provide the attention computation at position 1 ≤ t ≤ T +1 . First, we use W Q to compute the query vector q t by applying q t = W Q h (0) t :

Recall that W Q can be any arbitrary matrix. So, q j ∈ R can take any arbitrary value. Next, we compute the key vectors by applying k i = W K h (0) i :

Note that all key vectors are the same and we only need to compute them up to position t as the attention mask is causal, i.e query can only look at positions ≤ t . Next, we compute the attention weight vectors α :

where α ∗ = q 1 + q 2 + . . . + q h . Next, we apply softmax to compute the attention probabilities. Since all α i 's are the same, we have:

Now, we compute the value vectors by applying v i = W V h (0) i :

Finally, we compute the output of the attention head by applying W O :

Thus, the output of our constructed attention head recovers the absolute position information and writes it to the third dimension of output.

We used the decoder-only property of Transformer implicitly in Equation (23), which helped us to only attend to position ≤ t . So, the lengths of the attended sequence are always t . Moreover, the presence of <bos> token in the input sequence helped us to anchor the absolute position information. This is not a problem as in practice models are often prompted with some instructions which can act as <bos> token.

With this information available to the rest of the network, the feedforward sub-layer, with sufficient hidden width, can recover the absolute positions [1 , 2 , . . . , T +1] from the third dimension of attention output. This is because the feedforward sub-layer is MLP with ReLU activation. So, it can learn any arbitrary function (Park et al., 2020). Note that the layer-norm operation can be bypassed as explained by Akyurek et al. (2023).

C.2 Relative Positional Encoding in NoPE

In this section, we show if the hidden state contains absolute positional information as explained in the previous section, then the attention mechanism in all subsequent layers can implement a relative positional encoding. We refer the readers to Appendices B.1 and C.1 for the notation and definitions used in this section.

Theorem 2 ( Relative Encoding ) . Suppose that the hidden state H (1) contains absolute positional information, as stated in Theorem 1, and assume that it is not overwritten by any subsequent layers. Then, the self-attention in all subsequent layers can implement a relative positional encoding: there exists a parameterization of f θ such that, for l ≥ 2 , the attention dot product between query q t and key k i at positions t and i ( t ≥ i ) can be expressed as:

where f cnt is a function of their content, and f rel is a function of their relative distance.

Proof.

Our proof only specifies a few entries of weight matrices for attention heads in layers l ≥ 2 , which does not impose any challenges for Transformers used in practice as they usually have a very large model dimension d . Moreover, we require to have absolute positions in the third dimension of the hidden state as explained in Theorem 1. To show NoPE can implement relative encoding, we only need to prove that its attention dot product depends on the relative distance between tokens (See

Appendix B.1 for an overview of relative encoding methods). In the rest, we provide the construction of the weights and then verify that they can implement relative position encoding.

For head dimension h ≥ 2 , we construct the weights W Q , W K of the attention heads in the second layers and above. Specifically,

where w i,j , w ′ i,j ∈ R can take any arbitrary value. Their corresponding W V and W O can take any arbitrary values as long as they do not override the first three dimensions of the residual stream.

Now we verify that for any input sequence x = [ <bos> , x 1 , . . . , x T ] , the attention dot product between query q t and key k i at positions t and i ( t ≥ i ) will depend the relative distance between tokens.

First, assume that absolute positions are computed in the hidden state H ( l ) for l ≥ 1 , as stated in Theorem 1. Specifically,

where h i,j ∈ R can be any arbitrary value as the first three dimensions of the hidden state are reserved for PE computation. The rest of the dimensions can take any arbitrary values as in regular computation of Transformers.

We now present the attention computations at position 1 ≤ t ≤ T +1 . We use W Q to compute the query vector q t by applying q t = W Q h ( l ) t :

where q j ∈ R can take any arbitrary value. Next, we compute the key vectors by applying k i = W K h ( l ) i :

where k ( · , · ) ∈ R can have any arbitrary value. So, for k i we have:

Next, we let us present the attention dot product between q t and k i :

Thus, the dot product between q t and k i depends on the relative distance between tokens (assuming the rest of the terms do not cancel out which can be easily avoided by setting the respective weights in Equations (28) and (29)). Note that our proof uses the linear spacing between tokens, but the MLP the first layer can write any arbitrary function of absolute positions to the third dimension of the hidden state, which enables more complex relative encoding schemes.

D Experimental Details

D.1 Tasks

Here we provide the details and more examples of the tasks and datasets we used in our evaluation. For each task, we sample 100K examples for the training set and 10K for the test. Also, we use 15% of the train as the validation set.

Addition The addition task (Nye et al., 2021) asks the model to compute the sum of two numbers. Each number is represented as a sequence of digits that are separated by space. So, the model has access to the exact digits.

Input

we create each length bucket based on the number of digits in each number, e.g. 6-by-3, 6-by-4, etc. For the training set, we use buckets where one of the numbers has at most L digits. For the test set, we use buckets where any of the numbers have at most L digits. The model is evaluated on the correctness of its predicted result.

Polynomial Evaluation The polynomial evaluation task (Nye et al., 2021) asks the model to evaluate a polynomial expression at a given value of x . The polynomial terms and digits are separated to make just the tokenizer does not glue symbols together.

The length bucket is created based on the number of terms in the polynomial expression. We sample x from U ( -2 , 2) , the degree of each term from U (0 , 3) , and the coefficient of each term from U ( -3 , 3) . We take the modulo of the result by 10 to make the task easier for the model and make sure we only measure the generalization of the length of the problem instance not the value of the polynomial. The model is evaluated on the correctness of its predicted result.

Sorting The sorting task (Saxton et al., 2019) asks the model to sort a sequence of input numbers. We use this task in two variants: Single Token and Multi Digit. In the Single Token variant, we create an alphabet of 50 tokens from the model's vocabulary and fix some canonical ordering among them

through task. Each instance is a sequence of tokens from the alphabet in a random order, and the model is asked to sort them in the canonical order.

In the Multi Digit variant, we simply present a sequence of multi digit/tokens numbers to the model, and ask it to sort them in ascending order. Each number is represented by its digits and they are separated by a space.

In this case, we sample each number from U (0 , 10000) . In both cases, the length bucket is created based on the length of the input sequence. The model is evaluated on the correctness of its predicted result.

Summation In this task (Saxton et al., 2019), we ask the model to compute the sum of a sequence of input numbers modulo 10 as we want to specifically measure how the model generalizes to longer sequences not the value of summation result:

Each digit is randomly sampled from U (1 , 9) . The length bucket is created based on the length of the input sequence. The model is evaluated on the correctness of its predicted result.

Parity In the parity task (Anil et al., 2022), we ask the model to compute the parity of a binary sequence.

LEGO In the LEGO task (Zhang et al., 2023), the model is provided with a simple computation graph (DAG), where each node represents a variable, and variables are connected by simple operations which created the edges in the computation graph. We refer to Zhang et al. (2023) for a detailed description.

To sample each example, we first sample the list of variables based on the length of the example, and then we uniformly sample the value of each variable to make sure all variables are represented with both -1 and +1. Finally, given the value of variables, we deterministically compute the operation on each edge. For each example, we query all variables from the middle of the computation graph to the end. The model is evaluated on the correctness of its predicted result.

Copy The copy task is straightforward. The model has to repeat the input sequence in the output.

We create multiple variants of this task to better understand the models' generalization behavior. In the first variant, the input tokens are the same, so the model has to basically count the number of input tokens. In the second variant, the model has to replace the input tokens with another token sampled from the vocabulary. In the third variant, we sample the input tokens from the model's vocabulary, and the model has to predict them in the same order. We also create 2x versions of variants 1 and 3 to make the tasks more challenging.

Reverse In this task the model, the model has to reverse the order of input tokens in its output.

As in the copy task, we create multiple variants of this task. In the first variant, the model has to reverse the order of input tokens, where the tokens are randomly sampled from the model's vocabulary. In the second variant, the model has to reverse the order of input tokens, as in the first variant, but also it has to reverse it one more time, recreating the original input.

D.2 Hyperparameters

Table 2 shows the hyperparameters we used in our experiments. We use the same hyperparameters for all models and positional encoding schemes. In our initial experiment, we tried a few more hyperparameters such as lr ∈ { 0 . 00001 , 0 . 00003 , 0 . 00005 } and WeightDecay ∈ { 0 , 0 . 05 , 0 . 1 } , but we did not observe any significant difference in the results. So, we decided to use the same hyperparameters throughout our experiments.

D.3 Compute

In our experiments, we used single-GPU training setup for the models. Specifically, we ran our experiments on a mix of NVIDIA V100 32G, NVIDIA RTX8000 48G, NVIDIA A100 40G, and NVIDIA A100 80G GPUs. Depending on the GPU type and the positional encoding, each of our training runs took 6 to 15 hours, per each seed, on average to complete. Considering all the datasets, and positional encoding schemes, in addition to the scratchpad experiments, and three seeds, we ran about 870 individual training runs for the results in this paper.

D.4 Reproducibility

In this study, all experiments employed open-source libraries, specifically HuggingFace (Wolf et al., 2020) from which we utilized their implementation as a foundation for the training loop, optimizer, and the Transformer architecture. To ensure reproducibility, we will also release a singularity binary with all dependencies and libraries to enable running our experiments on any machine with NVIDIA GPUs and at any time in the future. Moreover, every reported number in this paper is linked to the source code package that deterministically (up to GPU stochasticity) reproduces the results, which we release publicly on GitHub at https://github.com/McGill-NLP/length-generalization .

E Full Results

E.1 Detailed Model Accuracy

We report the detailed results of our experiments in Figures F.4 to F.6. We refer the readers to Appendix D.1 for the description of each task.

E.2 Detailed Head Distance

Figure F.7 shows the layer-wise distance of No PE's attention patterns with other positional encoding schemes measured across instances of the SCAN dataset. We refer the readers to Section 5.2 for the details and analysis of these results.

E.3 Detailed Model Accuracy On Various Scratchpad Formats

Figure F.8 shows the generalization of various scratchpad formats for each model aggregated across all datasets. Figures F.9 to F.15 show the generalization of various scratchpad formats for each model on each dataset. We refer the readers to Section 6 for the details and analysis of these results.

F Pretraining at 1.3B Scale

In this section, we elucidate our preliminary assessment of length generalization on a pretrained LLM scaled to 1.3B. Ensuring a fair evaluation, we pretrain models across varied positional encodings, all on identical data with consistent parameters. Given the intrinsic significance of element positions to the semantics in code data, our choice leaned towards code-based pretraining.

F.1 Model Architecture

Our approach remains consistent with the original codebase, merely increasing the model dimensions. Specifically, we adopt a decoder-only Transformer structure encompassing 24 layers. The configuration details are as follows: d model = 1024 , d kv = 128 , d ff = 16 , 384 , and 32 attention heads. Consequently, this amounts to a Transformer model with 1.3B parameters. We train the models with a context size of 1024 tokens. For the rest of training hyperparameters, we follow Allal et al. (2023). Specifially, we use a global batch size of 256 and AdamW optimizer with β 1 = 0 . 9 , β 2 = 0 . 95 , ϵ = 10 -8 , and weight decay 0 . 1 . Also, the learning rate of 2 × 10 -4 with cosine decay and 2% warm-up is used.

F.2 Dataset Selection

We utilize a subset of 30M documents from the StarCoder dataset (Li et al., 2023), constituting a blend of 40% Python, 25% Java, 25% JavaScript, 5% GitHub issues, and 5% GitHub commits. Processed through the StarCoder tokenizer with a vocab size of 49 , 152 , the dataset results in 30B tokens. We have trained the model for a single epoch. For comprehensive details on the remaining hyperparameters, we direct readers to Allal et al. (2023).

F.3 Generalization Evaluation

Our evaluation aims to assess how the model's performance in predicting a given token varies with changes in context size, especially when the context size exceeds the training context size. Ideally, we expect the perplexity to generally improve as the context size increases since the model has access to more information. In this experiment, we select a sample of 1500 validation documents, each representing a single source code file. For each document, we calculate the perplexity of the last tokens, considering various context sizes ranging from 512 to 2560 (Note the model itself is pretrained with a context size of 1024 ) This approach allows us to evaluate the model's perplexity under both in-distribution and out-of-distribution conditions. To keep the computation and number of inferences tractable, we measure the perplexity on a maximum of last 200 tokens in the document and ensure exactly the same set of tokens are evaluated per each context size.

To obtain a comprehensive understanding of the model's performance, we also categorize documents into buckets based on their lengths and calculate the average perplexity within each bucket. This categorization enables us to examine the nature of dependencies within each bucket, recognizing that longer documents do not always imply longer dependencies.

We present the results in Figure F.2 and Tables 3 and 4. First, it is observed that all models benefit from a larger context up to the training size of 1024 . In the I.I.D. case (context length ≤ 1024 ), no significant differences among the models are discernible across all length buckets. Conversely, in the O.O.D. case (context length > 1024 ), the model trained with Rotary fails to generalize as its perplexity explodes in all cases, aligning with our findings in Section 4. NoPE and ALiBi effectively generalize to larger context sizes up to 1800 . However, as context sizes grow beyond that, ALiBi exhibits relative stability compared to NoPE, though both models show a pattern of increasing perplexity, potentially indicating inefficiencies to capture longer dependencies.

550

600

1

2

3

4

[750, 1000]

Positional Encoding

NoPE

ALiBi

Rotary

500

600

700

800

1

2

3

4

5

[1000, 1250]

600

800

1000

1200

0

2

4

6

[1250, 1500]

500

750

1000

1250

0

200

400

[1500, 1750]

500

750

1000

1250

1500

-

10000

-

5000

0

5000

10000

[1750, 2000]

500

1000

1500

0

500

1000

1500

[2000, 2250]

500

1000

1500

0

500

1000

[2250, 2500]

500

1000

1500

2000

0

500

1000

1500

[2500, 2750]

500

1000

1500

2000

0

500

1000

1500

[2750, 3000]

500

1000

1500

2000

0

1000

2000

[3000, 3250]

500

1000

1500

2000

0

500

1000

1500

[3250, 3500]

500

1000

1500

2000

0

500

1000

1500

[3500, 3750]

500

1000

1500

2000

0

1000

2000

[3750, 4000]

500

1000

1500

2000

-

4000

-

2000

0

2000

4000

6000

[4000, 4250]

500

1000

1500

2000

Context Size

0

500

1000

1500

[4250, 4500]

Dimension 1

Dimension 2

Dimension 3

Dimension 4

30

F0

25

5

Embedding Value

Embedding value

Embedding Value

20

15

15

10

20

5

30

2

0

35

0

5

10

15

20

25

30

D

5

10

15

20

25

OE

-5

0

5

10

20

25

30

0

10

15

20

25

30

Index

Index

Index

Index

Dimension5

Dimension6

Dimension7

Dimension8

30

Embedding Value

Embedding Value

-5

Embedding Value

anen buippaq

-5

-10

10

-10

15

15

10

15 -

-20

20

20

5

oE-

25

25

0

30

0

5

10

15

20

25

30

5

10

15

20

25

30

0

5

10

15

20

25

30

0

5

10

15

20

25

30

Index

Index

Index

Index

Dimension9

Dimension10

Dimension 11

Dimension 12

30

0

25

ot-

Embedding Value

5

Embedding Value

-5

Embeding value

-10

10

-20

15

15-

-30

20

20

-40

25

25

5

0

5

10

15

20

25

30

0

5

10

15

20

25

30

0

5

10

15

20

25

30

0

5

10

15

20

25

OE

Index

Index

Index

Index

Embedding Value

0

10

20

30

40

0

0.2

0.4

0.6

0.8

1

Copy (

Variant 1

)

Positional Encoding

NoPE

T5's Relative PE

ALiBi

Rotary

Absolute Position Embedding

0

10

20

30

40

0

0.2

0.4

0.6

0.8

1

Copy (

Variant 1 (2x)

)

0

10

20

30

40

0

0.2

0.4

0.6

0.8

1

Copy (

)

0

10

20

30

40

0

0.2

0.4

0.6

0.8

1

Copy (

)

0

10

20

30

40

0

0.2

0.4

0.6

0.8

1

Copy (

)

0

10

20

30

40

0

0.2

0.4

0.6

0.8

1

Reverse (

Variant 1

)

0

10

20

30

40

0

0.2

0.4

0.6

0.8

1

Reverse (

Variant 1 (2x)

)

0

10

20

30

40

Problem Length

0

0.2

0.4

0.6

0.8

1

Reverse (

)

4

8

12

16

0

0.2

0.4

0.6

0.8

1

Addition

Positional Encoding

NoPE

T5's Relative PE

ALiBi

Rotary

Absolute Position Embedding

4

8

12

16

0

0.2

0.4

0.6

0.8

1

Summation

4

8

12

16

0

0.2

0.4

0.6

0.8

1

Parity

4

8

12

16

0

0.2

0.4

0.6

0.8

1

Sorting (

Single Token

)

4

8

12

16

0

0.2

0.4

0.6

0.8

1

Sorting (

Multi Token

)

4

8

12

16

0

0.2

0.4

0.6

0.8

1

Polynomial Evaluation

4

8

12

16

Problem Length

0

0.2

0.4

0.6

0.8

1

LEGO

0

10

20

30

40

50

0

0.2

0.4

0.6

0.8

1

SCAN

Positional Encoding

NoPE

T5's Relative PE

ALiBi

Rotary

Absolute Position Embedding

0

10

20

30

Problem Length

0

0.2

0.4

0.6

0.8

1

PCFG

0

0.5

1

1.5

2

D

(

NoPE

,

M

)

Layer #01

Positional Encoding (

M

)

NoPE'

T5's Relative PE

ALiBi

Rotary

Absolute Position Embedding

Layer #02

Layer #03

Layer #04

0

0.5

1

1.5

2

Layer #05

Layer #06

Layer #07

Layer #08

0

10

20

30

40

50

0

0.5

1

1.5

2

Layer #09

0

10

20

30

40

50

Layer #10

0

10

20

30

40

50

Layer #11

0

10

20

30

40

50

Example Length

Layer #12

No PE

T5's Rel.

ALiBi

Rotary

APE

0

0.25

0.50

0.75

Mean Reciprocal Rank (

→

)

No Scratchpad

Full

Full - 'Step Input'

Full - 'Step Computation'

Full - 'Step Output'

Full - 'Intermediate Vars'

Full - 'Remaining Parts'

Minimal (Only Compute+Output)

0

0.05

0.10

0.15

0.20

Accuracy (Avg. over all OOD lengths)

No PE

0

0.05

0.10

0.15

0.20

T5's Relative PE

0

0.05

0.10

0.15

0.20

ALiBi

0

0.05

0.10

0.15

0.20

Rotary

No Scratchpad

Minimal (Only Compute+Output)

Full

Full - 'Step Input'

Full - 'Step Computation'

Full - 'Step Output'

Full - 'Intermediate Vars'

Full - 'Remaining Parts'

0

0.05

0.10

0.15

0.20

Absolute Position Embedding

0

0.1

0.2

0.3

0.4

Accuracy (Avg. over all OOD lengths)

No PE

0

0.1

0.2

0.3

0.4

T5's Relative PE

0

0.1

0.2

0.3

0.4

ALiBi

0

0.1

0.2

0.3

0.4

Rotary

No Scratchpad

Minimal (Only Compute+Output)

Full

Full - 'Step Input'

Full - 'Step Computation'

Full - 'Step Output'

Full - 'Intermediate Vars'

Full - 'Remaining Parts'

0

0.1

0.2

0.3

0.4

Absolute Position Embedding

0

0.2

0.4

0.6

0.8

Accuracy (Avg. over all OOD lengths)

No PE

0

0.2

0.4

0.6

0.8

T5's Relative PE

0

0.2

0.4

0.6

0.8

ALiBi

0

0.2

0.4

0.6

0.8

Rotary

No Scratchpad

Minimal (Only Compute+Output)

Full

Full - 'Step Input'

Full - 'Step Computation'

Full - 'Step Output'

Full - 'Intermediate Vars'

Full - 'Remaining Parts'

0

0.2

0.4

0.6

0.8

Absolute Position Embedding

0

0.25

0.50

0.75

1

Accuracy (Avg. over all OOD lengths)

No PE

0

0.25

0.50

0.75

1

T5's Relative PE

0

0.25

0.50

0.75

1

ALiBi

0

0.25

0.50

0.75

1

Rotary

No Scratchpad

Minimal (Only Compute+Output)

Full

Full - 'Step Input'

Full - 'Step Output'

Full - 'Remaining Parts'

0

0.25

0.50

0.75

1

Absolute Position Embedding

0

0.25

0.50

0.75

Accuracy (Avg. over all OOD lengths)

No PE

0

0.25

0.50

0.75

T5's Relative PE

0

0.25

0.50

0.75

ALiBi

0

0.25

0.50

0.75

Rotary

No Scratchpad

Minimal (Only Compute+Output)

Full

Full - 'Step Input'

Full - 'Step Output'

Full - 'Remaining Parts'

0

0.25

0.50

0.75

Absolute Position Embedding

0

0.2

0.4

0.6

Accuracy (Avg. over all OOD lengths)

No PE

0

0.2

0.4

0.6

T5's Relative PE

0

0.2

0.4

0.6

ALiBi

0

0.2

0.4

0.6

Rotary

No Scratchpad

Minimal (Only Compute+Output)

Full

Full - 'Step Input'

Full - 'Step Computation'

Full - 'Step Output'

Full - 'Intermediate Vars'

Full - 'Remaining Parts'

0

0.2

0.4

0.6

Absolute Position Embedding

0

0.2

0.4

0.6

0.8

Accuracy (Avg. over all OOD lengths)

No PE

0

0.2

0.4

0.6

0.8

T5's Relative PE

0

0.2

0.4

0.6

0.8

ALiBi

0

0.2

0.4

0.6

0.8

Rotary

No Scratchpad

Minimal (Only Compute+Output)

Full

Full - 'Step Input'

Full - 'Step Output'

Full - 'Remaining Parts'

0

0.2

0.4

0.6

0.8

Absolute Position Embedding

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBeyond model training, sequence length is one of the crucial aspect impacting the fluency, accuracy and completeness of LM outputs. For transformer based model's positional embeddings (PE) are used to explicitly represent the location of tokens. Over the past few years several works have attempted to propose a better PE approach, particularly aiming to improve over the original absolute PE of the Transformer. \n\nThis work, proposes a no positional encoding (noPE) model training for decoder only Transformer, to better generalize on generation length. The proposed noPE LM is evaluated on set of maths and reasoning tasks, using a small size (~100M parameter) LM. Findings show that noPE based LM performs similarly with relative PE based model (from the T5 model), and outperforms models with alternative PE variants such as Rotary, AliBi, and absolute PE for length generalization. Along with the evaluation results, detailed empirical analysis and theoretical arguments are provided for why noPE LM can intrinsically support an absolute and relative token position representations.\n\nSTRENGTHS:\nThis work, provided an in-depth analysis of Transformer LM training and inference without using PE. Although, the focus is to address the question of LM length generalization, the findings inform future possibility of an increasingly simplified transformer based LM's. In addition to the surprising results where noPE works in a similar way as the best performing relative PE (T5's PE), the work, delivered a thorough discussion as to why noPE works.\n\nWEAKNESSES:\nDespite the finding not using PE (noPE) can generalize for sequence length, the following points remain unanswered or not fully investigated:\n- Tasks selected for modeling and evaluation does not cover challenging length generalization scenarios. For instance, the noPE approach could have been testing using a summarization task which is highly impacted by the length of the summary.\n- Regardless of the provided results on length generalization, detailed and task specific performance metrics comparing LM's with PE's and noPE LM is missing. The absence of these comparison leaves open question on task specific performance comparison.\n\nQUESTIONS:\n- In addition to the mean rank, can authors add more task specific evaluation metrics, or could you please provide more motivation on why not to report task specific performance metric?\n- This can be a hypothetical one, have you considered a fine-tuning approach for LM's, for instance removing the PE when fine-tuning for down-stream tasks?\n- Can authors expand more on the selection of the LM size (100M parameters)? can we expect similar findings to hold if model size changes (i.e. larger LM's with more diverse and large scale train data)?\n\nLIMITATIONS:\nLimitation is provided, specifically explaining why the proposed noPE is not applied for large scale LM training. The limitation, not being able to access publicly available LM can be further expanded, for instance why not use 500M? If there is any identified societal impact its encouraged to include it.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work follows a long line of work analysing the effectiveness of various types of positional encodings and, in particular, a line of work exploring the effectiveness of causal masking without any positional encodings [1 - 8]. To that end, they evaluate the length generalisation performance of Transformers trained with various types of positional encoding schemes on a diverse set of reasoning and algorithmic tasks and find that Transformers without positional encoding schemes generalise better to higher lengths than other encoding schemes.\n \n \nThey consider 3 three types of tasks -- primitive (copying or reversing strings), algorithmic (addition, parity, etc.), and prior length generalisation benchmarks (e.g. SCAN). They evaluate five different encoding schemes, including Alibi and rotary encodings. They use autoregressive decoder-only Transformers of sizes of the order of 100M. For each task, Transformers are first trained on a set of examples using a language modelling objective and are then evaluated on examples of higher lengths.\n \nAcross the three types of tasks, they find that Transformers without positional encoding (but with masking) and T5's relative bias outperform other encoding schemes. Additionally, they show via a construction that Transformers without positional encodings can simulate both absolute and relative encoding schemes. They compare the attention patterns of Transformers trained without positional encoding schemes with other schemes and find that the attention patterns are most similar to relative positional encodings. Lastly, they find that when trained with a scratchpad, T5's relative positional encoding scheme and no positional encodings seem more effective than others.\n\nSTRENGTHS:\n(S1) **Interesting results.** Overall, I believe there are several interesting findings in the paper. I believe the contributions in the paper take a step forward in understanding the effectiveness of Transformers with just causal masking. The analysis with the scratchpad setup demonstrating the importance of PE for scratchpad and the effect of scratchpad for length generalisation is interesting as well.\n \n \n(S2) **Sound experimental setup and well-motivated.** The question of understanding and identifying the most effective positional encoding scheme is an important problem, and this work takes a step forward in that direction. The choice of tasks seems interesting, and the evaluation with Transformers trained with language modelling objectives is more aligned with practice than prior works that have explored this direction.\n \n \n(S3) **Well written.** The paper is well-written, the motivations are clearly outlined, and the arguments are well-presented.\n\nWEAKNESSES:\n(W1) **Novelty of results is slightly overclaimed.** Based on my limited knowledge, the use of Transformers without positional embeddings and with only causal masking was first studied in [1]. It was later analysed empirically [2,3] and theoretically [3, 5] in further works. These works mostly suggested that Transformers without positional encodings are competitive with standard encoding schemes on machine translation and language modelling tasks. On synthetic formal language tasks, [4] and more recently [6] have noted that using no positional encoding leads to better length generalisation performance. In more practical settings, recently [7, 8] have shown that large language models trained with only causal masking are competitive with models with positional encoding.\n \nThe fact that Transformers with only causal masking can encode positional information and generalise well to higher lengths is not entirely new. To be fair to the authors, I do not think that they explicitly claim this. I think the paper does a good job of establishing that just causal masking is better at length generalisation than various other schemes in a more systematic way. However, given the prior work exploring Transformers with only causal masking, the framing of the introduction seems a bit odd, and some of the papers mentioned above [1-8] are mentioned passively in the paper. It could be helpful to readers unfamiliar with the area if the prior work was presented in a more comprehensive manner.\n \n \n(W2) **Effect of scale.** Since the tasks in the paper are confined to synthetic tasks and models with ~100M parameters, it is still unclear if training LLMs without positional encodings will be better for length generalisation in practice. Modifications to Transformers at this scale for effectiveness on such tasks have often not scaled to LLMs in practice. As mentioned in the paper, that could require a larger computational budget which is understandable to some extent. But at the same time, comparing Transformers trained for language modelling on text data on higher lengths would have been useful. Based on the results in [7, 8], it seems like positional encoding schemes help improve perplexity in language modelling, but it may be interesting to see if that changes for length generalisation tasks.\n \n \n \n \n   \n   \n[1] DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding. 2017  \n[2] Assessing the Ability of Self-Attention Networks to Learn Word Order. 2019  \n[3] Transformer Dissection: An Unified Understanding for Transformer's Attention via the Lens of Kernel. 2019  \n[4] On the Ability and Limitations of Transformers to Recognise Formal Languages. 2020  \n[5] On the Computational Power of Transformers and its Implications in Sequence Modeling. 2020  \n[6] Transformers Learn Shortcuts to Automata. 2022  \n[7] Transformer Language Models without Positional Encodings Still Learn Positional Information. 2022  \n[8] Bloom: A parameter open-access multilingual language model. 2022\n\nQUESTIONS:\n(Q1) Regarding Section 5.2, if the models are trained with different positional encoding schemes from scratch, how can similarity between attention patterns help us determine how the model is computing positional information? Is it not possible that the models have very different learning trajectories? What about cases when Transformer without positional encoding performs better than relative encodings, wouldn't the underlying mechanism be different in that scenario? I think it is interesting the attention patterns without encodings are relatively much closer to ones with relative encodings (close to the baseline with different initialisations) than other ones, and that does provide some evidence that it is closer to relative encodings. But it is not clear to me why comparing the attention patterns should help us determine which type of positional information is being used.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLength generalization is one of hot topics for transformers. In this paper authors investigate how positional embedding influences this generalization for **decoder-only** models on **reasoning tasks**. Authors compare for variety of tasks (e.g. copy, summation, sorting, etc.) sinusoidal positional embedding, T5, AliBi, Rotary and no positional embedding (NoPE). \nFirst, authors proof by construction that NoPE for decoder-only models is able to learn positional information due to causal masking. Then authors show that none of the positional embeddings generalizes to longer sequences, while NoPE in some cases performs even better than others (though performance is still bad compared to seen lengths). Empirically by comparing attention patterns authors show that NoPE learns attention similar to T5 models. Finally authors also play with scratchpad and show that it is not always helping generalization for longer sequences.\n\nSTRENGTHS:\n- first time (at least I didn't see in prior works) formal proofs that decoder-only models are capable of positional embedding reconstruction (I went through the proof, no issue found with the math)\n- emergency for other positional embeddings development or investigation of transformers failure for the longer sequence generalization for reasoning tasks\n- interesting empirical results for reasoning tasks showing that mainly all positional embeddings currently widely used do not work for reasoning task on longer sequences while no positional embedding (NoPE) works similarly or even better sometimes\n- interesting insights on the NoPE (learnt attention pattern is more similar to models trained with T5 bias) and results on scratchpad\n\nWEAKNESSES:\n- overstatement in the test about positional embeddings for the decoder-only models: here it should be clarified about the type of the tasks where we apply, as for LMs we know that e.g. AliBi is working.\n- small scale experiments and limited length generalization evaluation.\n- no in-depth comparison of the root for the different behaviour e.g. for AliBi between the reasoning tasks and LM task.\n\nQUESTIONS:\n- from the Abstract and introduction I expected to see NoPE working well for GPT-models, while main results are provided for small-scale models and reasoning tasks rather than language modeling itself. I guess could be better to state about reasoning tasks in the abstract in addition to the restriction to the decoder-only models to be considered\n- From results (Appendix plots) I can see that actually no any positional embedding (including NoPE) generalized to long sequences for the reasoning tasks, so the statement \"Overall, our work suggests that explicit position embeddings are not essential for decoder-only Transformers to generalize well to longer sequences.\" is not supported as NoPE also doesn't work (works only for very couple cases + sequence length is not very large compared to training one).\n- For RoPE did authors try to insert it in every layer similar to T5 or AliBi which are used in every transformer layer? Otherwise could be not fair comparison in the end if the positional embedding saturates over the layers.\n- Eq. (2) why do we take min? Also do we consider all possible pairs of attention heads or we do pairing head1 from A and head1 from B? For other part of evaluation I found smart to use another seed with NoPE to measure how it changes to see that it actually like T5.\n- line 40-49: even for relative positional embedding it was shown that it could fail to generalize, see \"CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings\", NeurIPS 2021. In the same paper as well as e.g. in \"An image is worth 16x16 words: Transformers for image recognition at scale\", ICLR 2021 \"Conditional Positional Encodings for Vision Transformers\", ICLR 2023 there were reported that NoPE can works for the encoder-based models.\n- line 301-305: yes, authors consider LLMs style models (which is just decoders) but the data / tasks are different, so that we cannot just generalize from the observations to the LM data as in LM data we saw opposite behaviour. I found confusing the whole paragraph in this context.\n- Limitations stated as \"not study how large-scale pretraining affects\" - AliBi did study on WikiText103 which not large-scale + model was also rather small <1B params. I believe the issue is the data / task type (see below comments).  \n- Notation of multiplication as dot is weird. Prefer to see normal multiplication in $k.d$ -> $k\\cdot d$\n- Eq. 29 $W_V$ -> $W_K$\n- Fig. E2 for Copy Variant 1: did authors check if bias in T5 is learnt to be negative or positive? As in AliBi it emulate smoothed window attention and far away positions are mainly ignored, while for the task if copy you need to look at the whole sequence indeed. So could be that design of the positional embeddings are not designed for specific tasks where another reasoning on the positions is done (e.g. for LM we just need some summary context, but not exact values for every positions as we don't do e.g. copy task). Wonder to see what T5 bias then learnt to be consistent or disprove this hypothesis. For now the hypothesis seems to be consistent along all plots shown in the Appendix: e.g. for all copy tasks and reverse AliBi doens't work); for sorting Rotary is the best - which can be explained as we can solve sort by just comparing neighbors and reordering them, so strong relative position bias is needed; for other more or less different positional embeddings behave similarly.\n- Why do authors consider only 2L sequence generalization and not longer as in AliBi? I see that in the end we are not able to generalize even from 20 positions to 40 positions for simple reasoning tasks, while in LM (e.g. AliBi) they generalize to longer contexts, e.g. 16k tokens. I think in the main text it is overstated that some positional embeddings do not work - there should be clearly expressed \"let's consider reasoning tasks different from LM and investigate how there length generalization happens.\" I find that it would be interesting to have discussion on the difference between LM task and considered in the paper, as right now for me it is not clear why LM generalization with AliBI e.g. works nicely for even 16k tokens while in the simpler reasoning tasks this is not the case. Is it model? data? task itself? I presume we have more complex dependence on the positions and sequence length than in standard LM where simple summarization of far away context could be enough for good generation longer sequences while in reasoning task it is crucially to have proper positions distinguishing.\n\nLIMITATIONS:\nLimitations in the main text, however main restriction on considering non-LM tasks is not clearly delivered in the paper.\n\n**Update: Originally score was 3 for contribution and overall score 6-weak accept. After rebuttal and discussion they are raised to 4 for contribution and 7-accept.**", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper presents a comprehensive study on the role of positional encodings in Transformer models, focusing on their ability to generalize over sequence lengths. The authors introduce a novel positional encoding method, which does not use explicit positional information.

STRENGTHS:
1. The paper introduces a novel positional encoding method, which is a significant contribution to the field. 
2. The paper is well-written and easy to follow.

WEAKNESSES:
The authors did not study how large-scale pretraining affects different PEs due to the lack of publicly available large language models trained with various PEs under similar conditions. This leaves a significant aspect of the problem unexplored.

QUESTIONS:
1. How would NoPE perform in a broader range of tasks beyond the mathematical and reasoning tasks evaluated in this study?
2. Could the authors elaborate on the potential implications of their findings for the design of future Transformer models?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10