SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Amirhossein Kazemnejad<sup>1</sup>, Inkit Padhi<sup>2</sup>  
Karthikeyan Natesan Ramamurthy<sup>2</sup>, Payel Das<sup>2</sup>, Siva Reddy<sup>1,3,4</sup>  
<sup>1</sup>Mila, McGill University; <sup>2</sup>IBM Research; <sup>3</sup>Facebook CIFAR AI Chair; <sup>4</sup>ServiceNow Research  
{amirhossein.kazemnejad,siva.reddy}@mila.quebec  
inkpad@ibm.com,{knatesa,daspa}@us.ibm.com
Length generalization, the ability to generalize from small training context sizes to larger ones, is a critical challenge in the development of Transformer-based language models. Positional encoding (PE) has been identified as a major factor influencing length generalization, but the exact impact of different PE schemes on extrapolation in downstream tasks remains unclear. In this paper, we conduct a systematic empirical study comparing the length generalization performance of decoder-only Transformers with five different position encoding approaches including Absolute Position Embedding (APE), T5's Relative PE, ALiBi, and Rotary, in addition to Transformers without positional encoding (NoPE). Our evaluation encompasses a battery of reasoning and mathematical tasks. Our findings reveal that the most commonly used positional encoding methods, such as ALiBi, Rotary, and APE, are not well suited for length generalization in downstream tasks. More importantly, NoPE outperforms other explicit positional encoding methods while requiring no additional computation. We theoretically demonstrate that NoPE can represent both absolute and relative PEs, but when trained with SGD, it mostly resembles T5's Relative PE attention patterns. Finally, we find that scratchpad is not always helpful to solve length generalization and its format highly impacts the model's performance. Overall, our work suggests that explicit position encodings are not essential for decoder-only Transformers to generalize well to longer sequences.
The ability to generalize from smaller training context sizes to larger ones, commonly known as length generalization, is a major challenge for Transformer-based language models (Vaswani et al., 2017; Deletang et al., 2023; Zhang et al., 2023). Even with larger Transformers, this issue persists (Brown et al., 2020; Furrer et al., 2020). With larger context sizes, a model can benefit from more in-context-learning examples, higher numbers of reasoning steps, or longer text generation. However, training a Transformer with a larger context size can be excessively slow and memory-intensive. This is even more pronounced in the recent paradigm of model finetuning on instruction-following datasets (Wei et al., 2022a; Chung et al., 2022; Ouyang et al., 2022). It is not only infeasible to train the model on all possible context lengths, but also the number of training examples drops dramatically as the sequence length increases requiring the model to generalize from finite and shorter-length training examples. In this work, we focus on the effect of positional encoding on length generalization in the "decoder-only" Transformers on various tasks trained from scratch. Figure 1 summarizes our finding that using no positional encoding is better than using explicit positional encodings.
Figure 1: No positional encoding (NoPE) outperforms all other positional encodings at length generalization of decoder-only Transformers (GPT-style) trained from scratch and evaluated on a battery of reasoning-like downstream tasks. This figure shows aggregate ranking of positional encoding methods across 10 tasks.
Positional encoding (PE) seems to be a major factor in the length generalization of Transformers as the model has to systematically encode tokens in all possible positions. The original Transformer architecture (Vaswani et al., 2017) used non-parametric periodic functions to represent absolute position embeddings (APE) in a systematic manner, but further studies have shown that these functions are inadequate for length generalization (Ontanon et al., 2022). The prevailing belief is that relative PEs (Shaw et al., 2018; Raffel et al., 2020) are more effective in length generalization than APE variants (Ontanon et al., 2022; Csordás et al., 2021). However, Press et al. (2022) has shown that even Transformers with relative PEs, such as Rotary (Su et al., 2021), can be poor at length generalization. But the evaluation of PEs often relies on language modeling perplexity as a key metric (Haviv et al., 2022; Press et al., 2022) which does not always align with the performance on downstream tasks (Tay et al., 2022). This raises important questions: what exactly is the influence of positional encoding on length generalization at downstream tasks? Moreover, early empirical evidence shows that decoder-only Transformers without explicit position information (Tsai et al., 2019; Haviv et al., 2022) can perform as well as existing PEs in in-distribution settings, but its effects on length generalization and downstream performance are unclear.
Recently, asking models to emit intermediate computation steps into a scratchpad, also referred to as chain-of-thought, has been adopted to improve the length extrapolation in Transformers (Nye et al., 2021; Wei et al., 2022b). These techniques are architecture-independent and can be used with any PE method. However, it remains an open question whether these techniques, at least in regard to length generalization, render the choice of PE irrelevant, especially given that model performance is highly sensitive to the scratchpad format (Bueno et al., 2022; Akyurek and Akyurek, 2022).
To answer these questions, we conduct a systematic study on the length generalization of decoder-only Transformers, popularized by the GPT-family of models (Radford et al., 2019), with the most commonly used positional encoding schemes, both with and without scratchpad. Specifically, we evaluate APE (Vaswani et al., 2017), T5's Relative PE (Raffel et al., 2020), ALiBi (Press et al., 2022), Rotary (Su et al., 2021) and no positional encoding (NoPE) on a battery of reasoning and mathematical tasks. Our results show that:
- Most commonly used positional encoding methods, including ALiBi, Rotary, and APE, are ill-suited for length generalization in downstream tasks and are outperformed by T5's Relative PE.
- Transformers without positional encoding (NoPE) outperform all explicit positional encoding schemes. They achieve this without computing additional terms in the attention mechanism (in contrast to explicit PEs).
- We show that NoPE is theoretically capable of representing both absolute and relative PEs. But empirically, it is closer to the relative encoding scheme similar to T5's Relative PE.
- Scratchpad is not always helpful for length generalization and its format highly impacts the performance. The attention distributions reveal that NoPE and T5's Relative PE encourage attending to both long and short-range positions, ALiBi to recent positions, and Rotary and APE to no particular positions.
2 Background: Positional Encoding in Transformers
Transformers, in contrast to sequential models such as RNNs, are parallel architectures that employ positional encoding to help encode word order. The most common choices for positional encoding are either absolute, where each absolute position (e.g. 1, 2, 3, ...) is directly represented, or relative, where the distance between tokens is used as positional information. In this section, we briefly review the popular encoding methods used in Transformers (Refer to Appendix B for more formal details).
Absolute Position Embedding (APE), embeds each absolute position  $i$  into position vector  $\pmb{p}_i$  and adds word embeddings to their corresponding  $\pmb{p}_i$  before feeding them to the model. The non-parametric variant of APE uses periodic functions such as sine and cosine to generate embeddings for any position  $i$  (Vaswani et al., 2017). On the other hand, a learned version of APE, used in GPT3 (Brown et al., 2020) and OPT (Zhang et al., 2022), trains the position embeddings along with the model parameters, and it cannot generate a position embedding for unseen positions, so the context window is set to a fixed length.
T5's Relative bias, first maps the relative distance  $(i - j)$  between tokens at positions  $i$  and  $j$  to a scalar bias value  $b = f(i - j)$ , where  $f$  is a lookup table. The relative bias  $b$  (learned during training) then is added to the dot product of the query and key in the self-attention mechanism. The lookup table maps distances larger than a threshold to the same parameter to enable generalization to unseen distances.
Rotary, used in PaLM (Chowdhery et al., 2022) and LLaMA (Touvron et al., 2023), rotates the query and key representations with an angle proportional to their absolute positions before applying the dot product attention. As a result of this rotation, the attention dot product will only depend on the relative distance between tokens, effectively making it a relative positional encoding (Su et al., 2021).
ALiBi, used in BLOOM (Scao et al., 2022a), is similar to T5's Relative Bias but instead subtracts a scalar bias from the attention score. This bias grows linearly with the distance between the query and key tokens. This, in effect, creates a preference toward recent tokens (recency bias).
Note that encoder-only Transformers, such as BERT, become bag-of-words models in the absence of positional encoding. However, decoder-only Transformers with causal attention mask are not permutation invariant and can model sequences even without explicit position information (Tsai et al., 2019). But it is unclear if these models encode position information implicitly or generalize to unseen lengths. We demystify this in Section 5.
3 Model Evaluation
Length Generalization Setup Following Anil et al. (2022), we focus on algorithmic tasks such as copying, addition, etc. For each task, we train on a finite number of examples of up to a certain length and test them on both seen and unseen lengths at inference. We present these problems as sequence-to-sequence tasks, where the input sequence is the problem instance and the output sequence is the solution. Formally, let  $\mathcal{D} = \{(x_i,y_i)\}$  denote a dataset of such task where  $x_{i}$  is the input and  $y_{i}$  is the output sequence. For each task a function  $\lambda :\mathcal{D}\to \mathbb{N}$  can be defined that returns the length bucket of a task instance  $d\in \mathcal{D}$ . This can be the number of tokens or any general notion of length/depth of reasoning. Using this function and a threshold  $L$ , we employ samples where  $\lambda \leq L$  for learning the task and samples where  $\lambda >L$  for evaluating generalization. The performance on each instance is reported as the exact-match accuracy of its answer with the ground truth.
Architecture We use a conventional decoder-only Transformer architecture as a base for all experiments and consider different approaches for encoding positions: Absolute Position Embedding (APE), ALiBi, Rotary and T5's Relative Bias. We also consider removing the positional encoding (NoPE) to better understand its role in length generalization. Note that we use APE with sinusoidal functions (Vaswani et al., 2017) as the learnable variant cannot produce embeddings for unseen positions. Given the absence of publicly available Transformer-based LM trained with aforementioned PEs on the same pretraining data, we opt to train our models from scratch for each task on its training data with the autoregressive language modeling objective  $\log p_{\theta}(\pmb{y}|\pmb{x}) = \sum_{t=1}^{T}\log p_{\theta}(y_t|\pmb{x},\pmb{y}_{1:t-1})$ . We use the same hyperparameters for all PEs and employ the "base" model size configuration, popular in HuggingFace library (Wolf et al., 2020), resulting in  $\sim 107M$  trainable weights (List of all hyperparameters in Appendix D.2).
Table 1: Examples of the input and output of the tasks.
Task	Input Example	Output Example
Primitive Tasks
Copy	Copy the following words: <w1><w2><w3><w4><w5>	<w1><w2><w3><w4><w5>
Reverse	Reverse the following words: <w1><w2><w3><w4><w5>	<w5><w4><w3><w2><w1>
Mathematical and Algebraic Tasks
Addition	Compute: 5 3 7 2 6 + 1 9 1 7?	The answer is 5 5 6 4 3.
Polynomial Eval.	Evaluate x = 3 in (3 x ** 0 + 1 x ** 1 + 1 x ** 2) % 10?	The answer is 5.
Sorting	Sort the following numbers: 3 1 4 1 5?	The answer is 1 1 3 4 5.
Summation	Compute: (1 + 2 + 3 + 4 + 7) % 10?	The answer is 7.
Parity	Is the number of 1's even in [1 0 0 1 1]?	The answer is No.
LEGO	If a = -1; b = -a; c = +b; d = +c. Then what is c?	The answer is +1.
Classical Length Generalization Datasets
SCAN	jump twice and run left	JUMP JUMP TURN_LEFT RUN
PCFG	shift preprocess K10 R1 K12, E12 F16	F16 K10 R1 K12 E12
Figure 2: Aggregate ranking of positional encoding methods on length extrapolation across three different groups of tasks. No PE and T5's Relative Bias outperform other encoding methods in these categories.
Tasks Our study of length generalization is concentrated on downstream tasks. Particularly, we evaluate the models on three categories (Table 1) of synthetic tasks that have been widely used in the literature to investigate length generalization: (1) Primitive tasks such as Copying and Reversing (Ontonon et al., 2022), (2) Mathematical and reasoning tasks such as Addition (Nye et al., 2021), Polynomial Evaluation, Sorting, Summation (Saxton et al., 2019), Parity (Anil et al., 2022), LEGO (Zhang et al., 2023) and (3) Classical length generalization datasets such as SCAN (Lake and Baroni, 2018) and PCFG (Hupkes et al., 2020). These tasks provide us with complete control over the train-test distribution, while also requiring reasoning and compositionality skills, which serve as fundamental building blocks for more complex tasks. For the first two categories, we generate the corresponding datasets. Specifically, we first sample the length of the task instance from the uniform distribution  $\mathcal{U}(1,L)$ , and then, according to the task's generative process, we sample the input and output sequences. For the test set, we follow the same procedure but sample length from  $\mathcal{U}(1,2L)$  to include both seen and unseen lengths. Throughout the paper, unless otherwise stated, we use  $L = 20$ . For the third category of tasks, we use length generalization splits from the corresponding datasets. Table 1 provides an example of each task (More examples in Appendix D.1).
We report the results of our empirical evaluation over ten tasks and three seeds per dataset-PE pair.
4 What Is The Effect of Positional Encoding?
In this section we provide comparative results of positional encodings at length generalization. To provide a holistic view, following Liang et al. (2022), we report the mean ranking of various models in Figures 1 and 2 when compared against each other for all tasks and scenarios. Furthermore, we showcase the accuracy of models evaluated on examples of various lengths in Figure 3. (Detailed results for each task and scenario can be found in Appendix E).
First, we observe that in most tasks, models achieve a perfect or near-perfect accuracy (Figure 3) on the I.I.D. lengths, which indicates that models have no problem fitting to the training data. However, the differences among positional encoding methods become more apparent when we evaluate on lengths that are larger than seen during training. In most extrapolation scenarios, T5's Relative Bias
NoPE
Figure 3: Showcasing the generalization behavior of different positional encodings on 6 datasets. The shaded area represents evaluation examples with I.I.D. lengths (i.e. seen during training). Since all models perform perfectly, or close to it, on the I.I.D. lengths (measured on unseen examples), for improved readability, we only show a subset of them in the figure. Refer to Appendix E for more detailed plots.
T5's Relative PE
ALiBi
Absolute Position Embedding
outperforms other explicit positional encodings. ALiBi positions itself in the middle of the pack, while APE and Rotary show poor generalization performance.
Although Rotary is often considered a relative encoding method (Ontonon et al., 2022), our results show that it performs more similarly to APE than to other relative schemes. Moreover, ALiBi, despite its promise for length generalization, underperforms with respect to T5's Relative Bias in most cases. This result aligns with Taylor et al. (2022) who found no significant improvement from ALiBi.
Surprisingly, the NoPE model, which is just a decoder-only Transformer without any positional encoding, performs on par with or even better than the best-performing explicit PE, T5's Relative Bias. NoPE achieves the same level of generalization without any computational overhead since it does not compute any additional term in the attention mechanism. This property has a direct impact on the runtime and memory footprint of the model. For instance, Press et al. (2022) reported that the additional computation incurred by T5's Relative Bias can make the training and inference time of the model almost two times slower than the Transformer with APE.
5 How Does NoPE Represent Positions?
The surprising performance of NoPE model suggests that it captures useful positional information that can also generalize. But, how it does so is the primary question. In the next two sections, we provide theoretical and empirical analysis towards answering this question.
5.1 NoPE can theoretically represent both absolute and relative PEs
Let  $f_{\theta}$  be a NoPE decoder-only Transformer model, where  $\theta$  denotes the model parameters.  $f_{\theta}$  processes the input sequence  $\pmb{x} = [<\mathsf{bos}> , x_1 , \dots , x_T]$  by applying a series of layers. Note that since  $f_{\theta}$  does not have any PE, the input  $\pmb{x}$  is not augmented with positional information (e.g.  $[1,2,\dots,T]$ ). Each layer  $l$ , consisting of self-attention heads and a feed-forward sub-layer, reads the previous hidden state  $H^{(l - 1)}$  and produces the hidden state at layer  $l$ :  $H^l$ . Each head is parameterized by a query  $W_{Q}$ , key  $W_{K}$ , value  $W_{V}$ , and output  $W_{O}$  matrices, where  $W_{Q}, W_{K}, W_{V} \in \mathbb{R}^{h\times d}$  and  $W_{O} \in \mathbb{R}^{d\times h}$ .  $d$  and  $h$  are the model's hidden state size and attention dimension, respectively.  $W_{1}, W_{2} \in \mathbb{R}^{d\times k.d}$  are the weight matrices of the feed-forward sub-layer.
ute Position Embedding
Positional Encoding  $(M)$
NoPE
AI jBi
Figure 4: Distance of NoPE attention patterns with other positional encoding schemes measured across instances of SCAN dataset. The left figure shows the distance per layer, and the right figure shows the average distance across all layers. NoPE' denotes NoPE trained with a different seed.
otary
Absol
Theorem 1 (Absolute Encoding). Let  $\mathbf{x}$  be an input sequence of length  $T + 1$  to the model. Then, the first layer of  $f_{\theta}$  can recover absolute positions  $[1,\dots,T + 1]$  in the hidden state  $H^{(1)}$ . That is, there exist  $W_{Q}, W_{K}, W_{V}, W_{O}, W_{1}$ , and  $W_{2}$  such that the self-attention and feedforward operations in the first layer compute absolute positions and write it to the next hidden state.
We refer to Appendix C.1 for the complete proof of Theorem 1. This theorem shows that stochastic gradient descent (SGD) can potentially learn to recover absolute positions in NoPE Transformers. Next, we demonstrate how relative PE can be implemented in subsequent layers:
Theorem 2 (Relative Encoding). Suppose that the hidden state  $H^{(1)}$  contains absolute positional information, as stated in Theorem 1, and assume that it is not overwritten by any subsequent layers. Then, the self-attention in all subsequent layers can implement a relative positional encoding: there exists a parameterization of  $f_{\theta}$  such that, for  $l \geq 2$ , the attention dot product between query  $\mathbf{q}_n$  and key  $\mathbf{k}_m$  at positions  $n$  and  $m$  can be expressed as:
$$
\langle \boldsymbol {q} _ {n}, \boldsymbol {k} _ {m} \rangle = f _ {\text {c n t}} (\boldsymbol {q}, \boldsymbol {k}) + f _ {\text {r e l}} (n - m) \tag {1}
$$
where  $f_{\mathrm{cnt}}$  is a function of their content, and  $f_{\mathrm{rel}}$  is a function of their relative distance.
Appendix C.2 provides the complete proof of Theorem 2. Our theoretical results suggest that SGD can choose between relative and absolute encoding in NoPE Transformers. But, what mechanism SGD learns in practice is not clear. We next investigate this question empirically.
5.2 NoPE learns to use relative PE in practice
In order to explore the mechanisms that NoPE employs in practice, we conduct a quantitative analysis by comparing its attention pattern to models trained with different positional encoding techniques. The hypothesis is that if NoPE utilizes a similar algorithm to other PEs, then the attention patterns of these models should be quite similar.
To this end, we feed the same input to both models and, at layer  $l$ , we compute the minimum distance between the attention distribution of any heads in the first model and any head in the second model. Formally, let  $\mathrm{P}_t = p(\pmb{k}|\pmb{q}_t)$  be a probability distribution produced by a causal self-attention head for query at position  $t$ , over the keys  $\pmb{k} \in [k_1,\dots k_t]$  in a given transformer layer. Over a sequence of length  $T$ , we define the similarity between two heads  $\mathrm{P}$  and  $\mathrm{Q}$  as  $D_{\mathrm{AT}}(\mathrm{P},\mathrm{Q}) = \frac{1}{T}\sum_{t=1}^{T}D_{\mathrm{JSD}}(\mathrm{P}_t||\mathrm{Q}_t)$  which averages the Jensen-Shannon divergence (JSD) between the two heads over all positions. For the distance of two models  $A$  and  $B$  at layer  $l$ , we take the minimum distance
between all pairs of attention heads in the corresponding layer:
$$
D ^ {(l)} (A, B) = \min  _ {(\mathrm {P}, \mathrm {Q}) \in A _ {l} \times B _ {l}} D _ {\mathrm {A T}} (\mathrm {P}, \mathrm {Q}) \tag {2}
$$
where  $A_{l}$  and  $B_{l}$  are the attention heads in layer  $l$  of models  $A$  and  $B$  respectively. We empirically measure the distance between NoPE and other positional encoding schemes after training. Specifically, we sample examples from each length bucket and feed them (the concatenation gold input and output) to compute the attention maps and the distance using Equation (2). We also consider the distance between different seeds of NoPE as a baseline. Figure 4 shows the distance per layer for the first four layers. (later layers show similar trends Figure F.7). We find that NoPE's attention patterns are most similar to that of T5's Relative PE, and least similar to APE and Rotary. The same trend can be observed across all layers and length buckets, and even when averaged across all layers. These results potentially suggest that a Transformer model without positional encoding, trained with stochastic gradient descent learns to represent positions in a way similar to T5's Relative PE, which is a relative positional encoding scheme.
6 Does Scratchpad Render The Choice of Positional Encoding Irrelevant?
In scratchpad/CoT prompting, the model generates intermediate computations required to reach the final answer as explicit parts of the output. Such mechanisms, in effect, provide a direct decomposition and storage for intermediate values, which has been shown to improve the length generalization of Transformers even at small scale (Nye et al., 2021). Since scratchpad only modifies the model's input and output (not the architecture), it is unclear and unexplored how architectural choices such as positional encoding affect the length generalization in the presence of scratchpad. To answer this question, we train all PEs with and without scratchpad on the mathematical and reasoning group of tasks, and compare their performance.
Moreover, the decision of how to represent the intermediate computations in the scratchpad, i.e. the scratchpad format, is an important design choice that has a non-trivial impact on the model's performance (Bueno et al., 2022).