Length Generalization of Causal Transformers without Position Encoding (2024)

Abstract: Generalizing to longer sentences is important for recent Transformer-based language models. Besides algorithms manipulating explicit position features, the success of Transformers without position encodings (NoPE) provides a new way to overcome the challenge. In this paper, we study the length generalization property of NoPE. We find that although NoPE can extend to longer sequences than the commonly used explicit position encodings, it still has a limited context length. We identify a connection between the failure of NoPE’s generalization and the distraction of attention distributions. We propose a parameterefficient tuning for searching attention heads’ best temperature hyper-parameters, which substantially expands NoPE’s context size. Experiments on long sequence language modeling, the synthetic passkey retrieval task and real-world long context tasks show that NoPE can achieve competitive performances with state-of-the-art length generalization algorithms. The source code is publicly accessible1.

heading: 1 Introduction
text: Causal Transformer has been widely applied in modern language models. To help models recognize the correct ordering of words, it is common to configure Transformers with explicit position encodings (e.g., the sinusoidal embeddings in the original development of Transformer (Vaswani et al., 2017), the relative position encoding in T5 (Raffel et al., 2020), and the rotary position encoding in GPT series (Su et al., 2021)). The setup of position features provides flexibility to include prior knowledge structure on describing distance, but it also brings the problem of length generalization: language models trained with in-domain position features can not handle longer sentences (i.e., those with out-of-domain position features) in testing
* Equal contribution. 1 https://github.com/AntNLP/nope_head_scale
time. Generalizing to unseen sentence length is crucial in many language model applications like retrieval augmented language models (Izacard et al., 2023), personalized language models (Wang et al., 2023), language-model-based agents (Park et al., 2023).
Departing from the standard ways of encoding positions, one may ask (following the principle of parsimony) that are the explicit position features necessary? The answer is no. Both empirically (Haviv et al., 2022) and theoretically (Chi et al., 2023; Kazemnejad et al., 2023), the casually masked Transformers are shown to be able to successfully model languages without any prior position encoding (NoPE). The finding calls for a deeper understanding of implicit position information in Transformer-based language models, and also inspires a new direction for length generalization: without explicit position features, can NoPE generalize?
In this paper, we study the length generalization property of NoPE. Our main findings are,
• When extending to unseen sentence length, NoPE has less performance loss. However, beyond a certain range, NoPE also fails to extend, with
1 14024
no substantial difference observed when compared to explicit position encodings. For example, NoPE can effectively extend the training length by 20% (from 2K to 2.4K, Figure 1) without a significant increase in perplexity. In contrast, the rotary position encoding (RoPE) is only capable of extending by 10%.
• We analyze the failure cases of NoPE’s generalization and find that they always co-occur with the distraction of attention distributions: the attention heads begin to allocate their weights to tokens evenly when NoPE’s extension performance begins to collapse. The connection between NoPE’s generalization and concentration of attention heads suggests controlling the behaviors of attention heads during length extension.
• We show that by simply searching one temperature hyper-parameter, NoPE’s length generalization can be significantly improved. For example, by scaling the attention score by a factor of 1.2, NoPE can immediately generalize to over 4K tokens (Figure 1).
• Moreover, we developed an advanced version of this strategy by searching temperature parameters for each head, in the light that different layers and heads exhibit varied behaviors. The procedure resembles a parameter-efficient fine-tuning, with an extremely small number of tunable parameters (704 delta parameters over 1B model parameters). We show that the proposed method can help NoPE to generalize further (Figure 4).
We conduct length generalization experiments on long sequence language modeling, synthetic tasks (passkey retrieval), and LongBench. The results show that NoPE enjoys a competitive extension performances to state-of-the-art length gener-
alization methods for explicit position encodings (e.g., PI (Chen et al., 2023), YaRN (Peng et al., 2024)).

heading: 2 Length Generalization of NoPE

heading: 2.1 Language Modeling with NoPE
text: Before diving into the length generalization problem, we first briefly describe the NoPE models used in this paper. 2 Our default NoPE has 1.1B parameters. It is trained from the TinyLlama (Zhang et al., 2024b) code base 3, with training sequence length L = 2048 and 50K steps (≈ 100B tokens). More details can be found in Section 4.1.
We also include the original TinyLlama model which uses rotary position encoding (RoPE) for comparison. By default, both models are trained with identical settings.

heading: 2.2 Length Generalization
text: Given a language model (LM) with pre-trained maximal sequence length L, the goal of length generalization is to expand it to length L′ > L. Length generalization can be tested in a zero-shot manner (“train short, test long”) or with some finetuning.
Figure 1 depicts language modeling performances of NoPE (and RoPE). We can observe that, within the pre-training length (L = 2048), NoPE has a similar performance as RoPE, which agrees with existing works: casual masking can implicitly encode the positions of a sequence (Haviv et al., 2022; Chi et al., 2023).
When the testing sequence length exceeds the training length, we see that 1) NoPE’s length gen-
2For simplicity, we refer NoPE to both the implicit way of encoding positions and the language model trained without position encoding.
3https://github.com/jzhang38/TinyLlama
2
eralization error (light blue dashed line, measured with log-perplexity) is lower than RoPE (light red dashed line). 2) vanilla NoPE still has an increased perplexity than in-domain tests. Therefore, though it is not a perfect solution, removing explicit position encoding can effectively reduce the length generalization error. Next, we will try to find the reason for the failure of NoPE’s length generalization, and also develop algorithms for improving it.

heading: 2.3 Extension? Attention!
text: To analyze NoPE’s generalization failure, we first see that since explicit position encodings have been dropped, the casual Transformer block is only left with three core modules, the embedding layer, feedforward layers, and self-attention layers. The outputs of the former two modules are independent of their inputs’ position in sequence (i.e., no matter which position, they always have the same output). Therefore, multi-head attention layers become our main target.
We visualize the attention pattern of NoPE at different lengths. Specifically, given a validation set with a size n and a target position i, we define the average attention entropy Hi at position i, as
Hi = 1 n×m ∑
x,h
H(h)i (x) (1)
H(h)i (x) = − i∑
j=1
α (h) ij (x) · logα (h) ij (x) (2)
where x is a sample, α(h)ij (x) is the attention probability of token i focusing on token j in the h-th attention head (h ∈ {1, 2, ...,m}), H(h)i (x) is the entropy of the attention distribution α(h)ij (x) evaluated at position i.
The light solid lines in Figure 1 show the average entropy for NoPE (light blue) and RoPE (light red). We can observe that, the inflection point of Hi is highly consistent with the inflection point of perplexity. It implies that failed length generalization of NoPE (and RoPE) might be connected to the distraction of attention: attention heads begin to allocate attention to more tokens. To further verify the connection, we also draw a successful extension algorithm for RoPE (RoPE-NTK (bloc97, 2023b) which interpolates out-of-domain encodings to indomain encodings). Its length generalization loss curve is flat, while its entropy curve also has no steeply increasing point.
Unlike explicit position encodings, NoPE has no clear target objects to manipulate, thus it is quite challenging to perform length generalization without fine-tuning on longer sequences. However, the strong correlation between length extension and attention pattern transition suggests such an object, the entropy of attention heads.

heading: 2.4 Uniform Attention Scale
text: We write the general scaled dot-product attention as
α (h) ij =
eλq (h) i ·k (h) j
∑ k e λq (h) i ·k (h) k
(3)
where the scaling factor λ is the temperature hyperparameter of the SoftMax operator. The prevalent setting is λ = 1√
d .
Based on observations in Section 2.3, we know that NoPE’s failure of length generalization might be correlated with distracted attention, hence we can try to gradually increase the scale factor λ to reconcentrate attention, and see whether the generalization error can be reduced. Figure 2 visualizes the average entropy under different scale values and the corresponding perplexity curves.
We first find that when increasing the scale factor during length generalization evaluation (e.g., the pre-training scale λ= 1√
d is increased to λ= 1.2√ d ),
the inflection points of entropy curves are shifted to longer lengths, at the same time, NoPE all generalize to further positions (L=2k → L′=4k). That is, with all NoPE’s parameters frozen and only uniformly increasing the softmax’s temperature, NoPE can successfully generalize to unseen lengths.
The same conclusion doesn’t hold for RoPE (Figure 2 Right): no matter what value the scale takes (from λ=0.8 to λ=1.4), the inflection points of entropy curves remain almost unchanged, meaning that it fails to generalize to longer lengths. On the other side, successful RoPE extension algorithms (e.g., RoPE-NTK in Figure 1) can control the distraction of entropy by explicitly manipulate position encodings. Therefore, though attention scaling has been used for RoPE (Su, 2021; Chiang and Cholak, 2022), it may contribute marginally to RoPE’s generation.
We also find that extending NoPE to more distant positions generally requires a larger scale (i.e., a more concentrated attention distribution). As the position becomes further, the number of tokens involved in the attention calculation increases, the attention is more easily scattered, and therefore, a
3
larger scaling factor is needed to concentrate the attention. In particular, for our NoPE model, generalizing to twice the pre-training length requires about 1.2 times the scale, four times the length requires about 1.5 times the scale, and eight times the length requires about 1.8 times the scale. Appendix B reports the fitted function of the scaling factor with respect to the generalization length L′.
Finally, we note remark that the attention scaling factor in this section takes the same value for all positions, including the pre-training length (uniform scaling). We experimented with a piecewise function which use the original scale within the pre-training positions, and a more concentrated attention scale for the extrapolated positions. We also try position-dependent functions, where the scale increases with position. However, none of these methods could further improve generalization. We speculate that if the attention at earlier positions is not highly concentrated, the learned token representations may hinder the concentration of attention at latter positions. We leave a deeper discussion and analysis of this observation in future work.

heading: 3 Head-based Attention Scale
text: After verifying that the attention scaling can help NoPE generalizing, we delved deeper into the multi-head attention mechanism and posed a new question, “Does each attention head require a unique scaling factor?”
In this section, we first visualize the average entropy curves for each head and find that they have different attention patterns. Hence we propose to replace the uniform scaling with head-based scaling (from one factor to 22× 32 = 704 factors). To address the issue of an exploding search space, we efficiently determine the values of scaling factors
through automated hyperparameter search, considering both parameter efficiency and data efficiency. As a result, head-based scaling generalizes better than uniform scaling. Moreover, correlation analysis shows that within each layer, the smaller the converged entropy (i.e., the more concentrated attention), the larger the required scaling factor to maintain that concentration.

heading: 3.1 Visual Analysis
text: The entropy values span a broad spectrum, with each attention head demonstrating a distinct attention pattern. In Figure 3, certain attention heads show a highly concentrated pattern, with entropy values converging to ≈ 1, while others exhibit a highly dispersed pattern, with entropy values converging to ≈ 10. The full head visualization of Figure 3 is located in Appendix D.
This phenomenon casts doubt on uniform scaling — how can a single scaling factor cater to diverse attention heads? Inspired by this, we further propose a head-based scale method.

heading: 3.2 Head-based Scale
text: We reformulate the uniform attention scale as headbase attention scales
α (h) ij =
eλ (h)q (h) i ·k (h) j
∑ k e λ(h)q (h) i ·k (h) k
(4)
where λ(h) is a unique attention scaling factor for each head, totaling 704. Compared to a uniform attention scale, 704 head-based scales make it difficult to determine the optimal values by grid search. Similar to AutoML (He et al., 2021), we model the scales’ optimal search as a parameter-efficient fine-tuning task. Given a NoPE model M and a set
4
of head-based scales {λ(1), λ(2), . . . , λ(m)}, we fix the model M and define the head-based scales as trainable parameters θ = {λ(1), λ(2), . . . , λ(m)}. We aim to find an optimal set of values θ∗ = {λ∗(1), λ∗(2), . . . , λ∗(m)}, that allows the model M(θ∗) to successfully extend to the target length L′. To this end, we optimize the language modeling loss function LLM on the pre-training dataset D with length L′ and size n′, n′ ≪ n.
θ∗ = minimize x∈D LLM (M(θ, x)) (5)
The search process is highly efficient. (1) The number of tunable parameters is extremely small, only 704 delta parameters over 1B model parameters; 2) The amount of training tokens for fine-tuning is extremely small too, only 0.03% of the pre-training data.
In addition, to ensure that the attention is reconcentrated instead of distracted by the scaling factors, we apply a focus constraint during the optimization of Equation 5
λ(∗) ≥ 1√ d
(6)
Initializing HeadScale In practice, we found that the initial value of head-based scales has a significant impact on the search of θ∗. An obvious approach is to use the default value λ(∗)= 1√
d from
the pre-training phase. However, its length generalization results are quite unstable, with most being subpar, as the optimal scale often deviates significantly from the default value. We propose another approach to utilize the best uniform scale from the grid search as the initial value. The ablation study for the initialization approach is in Section 4.5.
Figure 4 compares the two generalization methods of NoPE, uniform scale versus head-based scales. Head-based scale exhibits better generalization than the uniform scale, achieving a lower log-PPL by 0.2 at 4K positions (2×L) and by 0.8 at 8K positions (4×L). The average entropy Hi of the head-based scale is higher than that of the uniform scale, suggesting that the uniform scale method over-concentrates attention, particularly for some heads that inherently have more distracted patterns.
Figure 5 shows the correlation between the converged entropy and the searched scale. To save space, we uniformly sampled 7 layers and all their respective heads. We observed that the correlation is layer-dependent, within each layer, heads with more concentrated attention (i.e., lower entropy) searched for larger scales, while heads with more dispersed attention (i.e., higher entropy) searched for smaller scales. The result is as expected, the more concentrated the attention pattern, the larger the scaling factor needed to maintain its focus. Furthermore, we observed that attention heads in lower layers are generally more dispersed, whereas heads in higher layers are generally more concentrated (note that this is not strictly observed).

heading: 4 Experiment
text: We train a NoPE base model from scratch and investigate its capability in length generalization. We conduct length generalization experiments on long sequence language modeling, synthetic tasks (passkey retrieval), and real-world long context tasks (LongBench). Detailed experiment setup can be found in Appendix A.

heading: 4.1 NoPE pre-trained model
text: For a fair comparison with RoPE, we train a NoPE model with 1.1B parameters from the TinyLlama (Zhang et al., 2024b) code base4. The NoPE model has 22 layers of Transformer blocks, 32 attention heads per layer, 2048 embedding size. The model is trained on Slimpajama (Soboleva et al., 2023)
4https://github.com/jzhang38/TinyLlama
5
joint with Starcoderdata (Li et al., 2023) by 50K steps (≈ 100B tokens) with sequence length L = 2048.
All settings are kept identical to those of TinyLlama, including the model architecture, training data, training procedure, and hyper-parameters, except that the rotary position embedding (RoPE) in TinyLlama is removed, making it a NoPE model, and the learning rate is set to 3.5× 10−4.
Following TinyLlama, we evaluate the commonsense reasoning ability of the NoPE model and report acc_norm in Table 1. We compare with the TinyLlama checkpoint that is trained on 100B tokens. The purpose of this experiment is to prove the NoPE base model performs on par with RoPE.

heading: 4.2 Long Sequence Language Modeling
text: Success on long sequence language modeling tasks is essential for length generalization. A method that does not perform well in language modeling probably won’t handle real-world long-context tasks.
Settings. To evaluate the long sequence language modeling performances, we test our NoPE-based methods and RoPE-based baselines on PG19 (Rae et al., 2020) and proof-pile (Azerbayev et al., 2022) datasets. For each dataset, we sample a subset of the test set and evaluate on 2M tokens using sliding window evaluation (S = 256) suggested by Press et al. (2022). We report the perplexity (PPL) of the models in Table 2.
Main results. Firstly, by comparing the original language models, NoPE’s perplexity (PPL) is comparable to RoPE’s for lengths within the train-
6
ing distribution, confirming the findings of Haviv et al. (2022); Chi et al. (2023). However, all LMs, including ALiBi models, fail to generalize out-ofthe-distribution, indicating that explicit positional encoding is not the main reason for their failure in generalization. Current work on length generalization still focuses mainly on manipulating positional encoding. Therefore, the length generalization issue within causal Transformer networks warrants a reanalysis and reinterpretation.
Secondly, by comparing the two generalization methods for NoPE proposed in this paper, the uniform scale method has significant limitations. Although using a larger scale can reduce the PPL at greater positions, it significantly affects the PPL at closer ranges. For instance, with a scale value of 1.8, the PPL on 2K@PG19 rises from 14.6 to 30.4, and on 2K@Proof-pile, it rises from 3.5 to 5.1. On the contrary, the head-based scale method not only successfully extrapolates to 16k but also has minimal impact on the PPL at closer distances (for 18K, increases only +3.7 on 2K@PG19, +0.5 on 2K@Proof-pile), proving that attention heads with different patterns indeed require distinct scale values.
Third, a full comparison with RoPE LM’s generalization method. Comparing the zero-shot generalization methods, the head-based scale has better generalization than NTK, but weaker than YaRN. In a fair comparison with the RoPE generalization methods which require fine-tuning, the head-based
scale method is competitive with these RoPE baselines, especially the Proof-pile dataset. However RoPE baselines (PI, YaRN) still benefit from more training tokens, and the head-based scale on NoPE reaches its upper limit.
In summary, the head-based scale generalization method for NoPE slightly outperforms RoPE’s early generalization method NTK, but still lags behind the recently introduced YaRN, particularly in near-distance PPL performance. Considering the significant challenge of generalizing NoPE compared to RoPE (due to the lack of explicit positional encoding to manipulate), this work, as the first to tackle length generalization for NoPE, has achieved its set goals.
The observed gap may imply that constraining the NoPE model to focus on fewer tokens could detrimentally affect its efficacy. Future efforts will be directed at enhancing the head-based scaling method to regain the level of performance seen in pretraining.

heading: 4.3 Synthetic Long Context Tasks
text: A synthetic task is constructed in Landmark Attention (Mohtashami and Jaggi, 2023b) called "Passkey Retrieval". It aims to test the effective context window size of the model. The task is to retrieve a randomly placed passkey from a long sequence of tokens, where the passkey is a randomly sampled number of 5 digits and the sequence is built by concatenating irrelevant sentences.
7
Settings. We evaluate the performance of passkey retrieval across various context lengths. For each specified context length, we conduct tests on 10 distinct passkey depths, each associated with 10 randomly selected passkeys. We report the retrieval accuracy in this task.
It is observed in Figure 6 that both the NoPE base model and head-based scale perform well even when evaluating on 2× the pretraining or fine-tuning context window, while RoPE strictly operates within the pre-trained sequence length and immediately fails outside of it. The result indicates that NoPE possesses significant potential for generalization.

heading: 4.4 Real-World Long Context Tasks
text: LongBench (Bai et al., 2023) is a comprehensive assessment of the long context understanding capabilities of large language models. We test all models using beam search decoding with beam size 5. The evaluation context size is set to the model context window accordingly in order to test the model’s capability to utilize a longer context. We only include raw PI and YaRN as the baseline in this task.
We find that the performance of the NoPE base
model is better than its RoPE counterpart. Concluding better information utilization in the original length. Moreover, the head-based scale at a 4k extension length performs the best among all baselines. We attribute it to the capability of the NoPE base model and the successful length generalization of the head-based attention scale method. While the head-based model still suffers from performance degradation when extending to a longer context, as it is stated in Section 4.2.

heading: 4.5 Ablation Study
text: We have introduced two key components of HeadScale in Section 3.2, a concentration constraint and an initializing technique. The ablation study in Table 4 depicts that although occasionally perform better in language modeling, the two variants are less preferment in passkey retrieval and LongBench, indicating their inability to utilize long context information.
Detailed results of the passkey retrieval task can be found in Figure 9 in the Appendix C. They are completely unable to answer the passkey except when it is at the beginning of the context window.

heading: 5 Related Work
text: Transformers without position encoding Haviv et al. (2022) was the first to discover that causal Transformer networks could perform language modeling tasks successfully even without explicit PE. Chi et al. (2023) provided a theoretical explanation for NoPE, demonstrating that for an initialized NoPE LM, the variance of the hidden representations in each layer is position-dependent, with variance decreasing for larger positions. Both works demonstrate that the NoPE hidden layer rep-
8
resentation implies positional information through the probing task. Kazemnejad et al. (2023) proved through constructive methods that NoPE can learn absolute PE from the first layer and relative PE from the second layer. They also showed that NoPE has an extremely weak length generalization ability (train ∼20, test ∼40), but is slightly better than LM with explicit PE. This paper first proposes length generalization methods for NoPE with uniform scale and head-based scale. For the first time verifies the effectiveness of NoPE generalization in real LLM settings.
Length generalization Due to high computational and memory requirements, LLM training is usually limited to short inputs. Directly applying LLMs to long inputs faces the challenge of outof-distribution (OOD) issues. Research to enable LLMs to process long inputs has been extensive (Huang et al., 2023; Dong et al., 2023). The earliest methods involved designing new relative PE mechanisms during pre-training (Press et al., 2021; Sun et al., 2023). Subsequent studies focused primarily on the widely used RoPE (Su et al., 2024) and proposed length extension by mitigating RoPE’s OOD issues through interpolated positions (Chen et al., 2023; kaiokendev, 2023; Peng et al., 2023; emozilla, 2023; bloc97, 2023b,a). Other works employed sliding window attention mechanisms to prevent relative positions from exceeding the maximum distance seen in pre-training (Mohtashami and Jaggi, 2023a; Han et al., 2023; Xiao et al., 2023; Jin et al., 2024; Zhang et al., 2024a). However, these models ignore information from distant tokens, thus failing to capture long-distance context dependencies. All existing methods rely on specific explicit PEs. However, the NoPE architecture is more streamlined and more aligned to the form of human language modeling. Exploring NoPE’s length generalization is therefore more intriguing and attractive.

heading: 6 Discussion
text: We studied the length generalization of Casual Transformer without explicit position encoding. We developed a parameter-efficient tuning algorithm which aims to search for the best temperature hyper-parameters for attention heads. Through empirical evaluation, we saw that NoPE can achieve competitive length generalization and might be a promising alternative for long-context language modeling.
NoPE provides a new perspective to understanding the role of positional information by isolating and eliminating the effects of explicit positional encoding. Our work demonstrates the correlation between length generation failures and distraction of attention in NoPE models, thus the proposed method concentrates the attention by adjusting the scaling factor. While current works on length generalization mainly focus on manipulating positional encoding, our work suggests a new key component to generalization.
Limitation
The length generalization algorithms discussed in this paper exhibit competitive performances, but the NoPE model itself still underperforms with state-of-the-art RoPE models, which makes the results over long sequence language modeling tasks and LongBench tasks are less competitive. NoPE still faces the challenges of considerable memory usage and computational complexity due to the quadratic nature of attention computation when processing extremely long contexts. Hardware limitations are likely to become a constraining factor for length generalization soon. We plan to further improve the NoPE’s performances for a fairer comparison. This paper is also most an empirical one, which requires a deeper theoretical understanding of NoPE’s length generalization in the future.

heading: Acknowledgement
text: The authors wish to thank all reviewers for their helpful comments and suggestions. The corresponding authors are Tao Ji, Yuanbin Wu and Xiaoling Wang. This research was (partially) supported by NSFC(62076097), National Key R&D Program of China (2021YFC3340700), the Open Research Fund of Key Laboratory of Advanced Theory and Application in Statistics and Data Science (East China Normal University), Ministry of Education.

heading: A Experiment Setup
text: Searching scales. We approach the search for optimal head-based scales λ(h) by parameter-efficient fine-tuning. We use a large learning rate (LR, =0.05 or =0.1) for fine-tuning, as λ spans a wide range, (e.g., [ 1√
d , 3√ d ], shown in Figure 5). The fine-tuning
data comes from the pretraining dataset (Slimpajama (Soboleva et al., 2023) and Starcoderdata (Li et al., 2023)) with a different data fetching seed from the pretraining. We set the batch size to 8 and set the optimizer to the AdamW (β1 = 0.9,
11
β2 = 0.95) without weight decay (Loshchilov and Hutter, 2017). We use a cosine LR decay from LR to 0.1LR for 200 fine-tuning steps and a linear warmup for the first 20 steps. We found that the head-based scale searching on 16K suffers from a minor PPL degradation at the end of the context window. We simply expanded the length L′ to 18K and then solved it.
Length generalization baselines. To compare with mainstream length generalization research, we reproduced three generalization baselines on RoPE, including:
• NTK (2023c), zero-shot generalization;
• PI (2023), efficiently train long, test long;
• YaRN (2024), supports both settings 5.
For the zero-shot setting, we grid-searched the baseline hyper-parameters and reported their best results. For the baselines that need fine-tuning, we propose two settings, one for a fair comparison, with the same number of fine-tuned tokens (0.3‰ of pre-trained data) as the head-based scales searching, and the other follows their original paper, which is 1.3‰ of pre-trained data. Specifically, we fine-tune the RoPE model for 200 steps in the “fair” version, and 1000 steps for the “raw” version.
In addition, we incorporate open-source ALiBi models (Press et al., 2022) into our baselines, which include BLOOM 1.1B (BigScience Workshop, 2022) and MPT 7B Base (MosaicML NLP Team, 2023), both of which are trained on a context length of 2K. We test a zero-shot generalization of the ALiBi models following the original paper (Press et al., 2022).

heading: B Fitted Function of the Uniform Scale
text: In the study depicted in Figure 7, a hyper-parameter search was conducted for the uniform scale λ with an interval of 0.01√
d . This search was applied to two
checkpoints of the pre-trained NoPE model, to fit the optimal λ at the extension length. We note remark that the scaling factor takes the same value for all positions during a single test. The output of a single test is the perplexity across all positions. We run multiple tests with different scales and find the best one for each position.
5The YaRN paper also proposes a “train short, test long” setting with lower training costs. However, for a fair comparison, we relax this setting to “train long, test long” which generalizes better.
Based on the search results, we guess a function form that best fits the data points. We then fit this function over the range i ∈ [2048, 16384]. The fitted function, along with its corresponding coefficient of determination, is presented below:
• For NoPE at 10k steps, the coefficient of determination R2 = 0.9954. The fitted function is
λ = 1 + 0.3010 ln s√
d
• For NoPE at 50k steps, the coefficient of determination R2 = 0.9773. The fitted function is
λ = 1 + 0.3973 ln s√
d
In these functions, s is defined as iL for each position i, representing the model’s extension ratio relative to its pre-training length.
Furthermore, it is also found by Peng et al. (2024) that the YaRN method benefits from a similar uniform scale on LLaMA2 (Touvron et al., 2023), although the scale does not have a direct impact on the RoPE extension capability (refer to Figure 2). The scale proposed by the YaRN method can be formulated as follows, which is quite similar to our result.
λ = (1 + 0.1 ln s)2√
d
In conclusion, the optimal uniform scale varies across different models. It is also observed from Figure 7 that uniform scale, despite being optimal, cannot flatten the NoPE model’s perplexity within a large context window. This finding underscores the importance of employing a head-based scaling method for managing model perplexity effectively across larger context windows, thereby enhancing the model’s performance.

heading: C Additional Passkey Results
text: In Section 4.2, we note that the ALiBi baselines do not exhibit competitive performance in terms of perplexity when applied to longer contexts. We also conduct Passkey Retrieval tests on these models, with the results depicted in Figure 8. These models yield expected results within their pre-trained sequence length, but they are unable to complete the task when it exceeds this length.
In Section 4.5, we conducted an ablation study on HeadScale. Figure 9 shows the passkey retrieval task of the two variations of HeadScale.
12

heading: D Entropy Visualization of All Heads
text: Figures 10 to 12 show attention entropy across all layers and all heads of the 8k extension head-based scale method, UniformScale and the original NoPE. An additional theoretical upper bound of entropy is also plotted in the figures. We note that for each position i, the maximum entropy is achieved when ∀j, α(h)ij = 1i is satisfied in Equation 2. The maximum value is then given by H(h)i = log i.
It is observed in Figure 10 that the lower layers have high entropy, closely approaching the upper bound. Most heads exhibit constant entropy for all positions. And the attention values span a broad spectrum, ranging from 0 to theoretical upper-bound.
13
14
λ(h) 8K
15
λ = 1.6√ d
16
NoPE
17

heading: Future Work
text: This paper is also most an empirical one, which requires a deeper theoretical understanding of NoPE’s length generalization in the future