Extending the Context of Pretrained LLMs by Dropping Their Positional Embedding
Authors: Yoav Gelberg
Koshi Eguchi
Takuya Akiba
Edoardo Cetin
Venue: ICLR 2026 Poster (2026)
Keywords: LMs
Long Context
Positional Embeddings
Architecture
Primary area: foundation or frontier models, including LLMs
TL;DR: We extend the context of pretrained LMs by dropping their positional embeddings after training.

Abstract: So far, expensive finetuning beyond the pretraining sequence length has been a prerequisite to effectively extend the context of language models (LM). In this work, we break this key bottleneck by ***Dro**pping the **P**ositional **E**mbeddings of LMs after training (DroPE)*. Our simple method is motivated by three key theoretical and empirical observations. First, positional embeddings serve a crucial role during pretraining, providing an important inductive bias that significantly facilitates convergence. Second, over-reliance on this explicit positional information is also precisely what prevents test-time generalization to sequences of unseen length. Third, positional embeddings are not an inherent requirement of effective language modeling and can be safely *removed after pretraining* following a short recalibration phase. Empirically, DroPE yields seamless *zero-shot* context extension *without any long-context finetuning*, quickly adapting pretrained LMs without compromising their capabilities in the original training context. Our findings hold across different models and dataset sizes, far outperforming previous specialized architectures and established rotary position embedding scaling methods.

Published as a conference paper at ICLR 2026
EXTENDING THE CONTEXT OF PRETRAINED LLMs BY DROPPING THEIR POSITIONAL EMBEDDINGS
Yoav Gelberg"
Koshi Eguchi Sakana Al
Takuya Akiba Sakana Al
Edoardo Cetin Sakana Al edo@sakana a
University of Oxford yoavêrobote ac..ac.uk..
ABSTRACT
So far, expensive finetuning beyond the pretraining sequence length has been a prerequisite to effectively extend the context of language models (LM). In this work, we break this key bottleneck by Dropping the Positional Embeddings of LMs after training (DroPE). Our simple method is motivated by three key the- oretical and empirical observations. First, positional embeddings serve a crucial role during pretraining, providing an important inductive bias that significantly facilitates convergence. Second, over-reliance on this explicit positional informa- tion is also precisely what prevents test-time generalization to sequences of unseen length. Third, positional embeddings are not an inherent requirement of effective language modeling and can be safely removed after pretraining following short recalibration phase. Empirically, DroPE yields seamless zero-shof context exten- sion without any long-context finetuning. quickly adapting pretrained LMs without compromising their capabilities in the original training context. Our findings hold across different models and dataset sizes, far outperforming previous specialized architectures and established rotary positional embedding scaling methods.
INTRODUCTION
Transformers established themselves as the predominant architecture for training foundation models at unprecedented scale in language and beyond (Brown et al.,
100%
Accuracy (%)
75%
2020; Team et al., 2023; Jumper et al.,
50%
2021; Dosovitskiy et al., 2020). The defin- ing feature of transformers is abandoning explicit architectural biases such as convo- lutions and recurrences in favor of highly general self-attention layers (Vaswani et al., 2017), while injecting positional information about the sequence through positional embeddings (PF) and causal masking. However, despite significant efforts to scale attention to long sequences on modern hardware (Dao et al., 2022; Liu et al., 2023a; Liu & Abbeel, 2023), this powerful layer is inherently bottlenecked by quadratic token-token interactions, which makes pretraining at long sequence lengths computationally intractable scale. As result, enabling models to contexts beyond their pretraining length without additional long-conter context fune-tuning ("zero-shot context extension") has emerged as a central challenge for the next generation of foundation models (Press al., 2021: Chi al., 2023).
25%
RoPE YaRN DroPE
0%
1k
2k
3k
Distance to needle (tokens)
Figure 1: DroPE generalizes zero-shot to long se- quences. feedle-in---hhaystack retrieval accuracy sequences at 2x the original context length with no long context training (zero-shot context extension).
When inference sequence lengths exceed the pretraining context, the performance of modern transformer-batt LMs degrades sharply. This directly caused by their use of explicit PEs such as the ubiquitous rotary positional embeddings (RoPE) (Su et al., 2024), which become out-of diitr--iiittiittionn unseen sequence lengths. To address this issue, careful scaling techniques
"This work conducted during internship Sakana AI.

---

Published as a conference paper at ICLR 2026
RoPE NoPE DroPE
RoPE
NoPE
24
Perplexity
22
3.8
20
3.6
3.4
18
16-
0K
2K
4K
6K
8K
10K
12K
14K
16K
RoPE
NoPE
DroPE
Step
Figure 2: DroPE matches RoPE's in-context perplexity. We compare three training recipes: (1) RoPE transformer trained for 16K steps (16B tokens), (2) a NoPE transformer trained for 16K steps, and (c) DroPE transformer obtained by training the 14K-step RoPE checkpoint for 2K additional steps. The DroPE recipe matches the RoPE loss within 2K steps and achieves lower final perplexity than the NoPE-from-scratch baseline under the same budget.
that adapt RoPE frequencies on longer sequences were introduced (Chen et al., 2023; bloc97, 2023; Peng et al., 2023; Ding et al., 2024). However, despite their popularity, these methods still rely on an expensive, long-context finetuning phase to meaningfully use tokens beyond the original sequence length, failing to generalize out of the box (Iu et al., 2024a). Beyond RoPE transformers, alternative architectures and positional embedding schemes have shown early promise reducing costs by attenuating the underlying quadratic burden Choromanski et al. (2020); Wang et al. (2020); Xiong et al. (2021); Zaheer et al. (2020) or maintaining better out-of-context generalization (Kazemnejad al., 2023; Yang et al., 2025b; Puvvada et al., 2025). Yet, these parallel efforts still far from challenging established pipelines, introducing notable performance and stability trade-offs that prevent wide adoption.
In this work, we challenge the conventional role of RoPE language modeling, and propose to overcome this inherent trade-off by Dropping the Positional Embeddings (DroPE) of LMs after pretraining. Our method is based on three key theoretical and empirical observations. First, explicit positional embeddings significantly facilitate pretraining convergence by baking in an important inductive bias that is difficult to recover from data alone. Second, over-reliance on positional embeddings precisely what prevents test-time generalization to sequences of unseen length, with RoPE-based context extension methods focusing on recent tokens to retain perplexity. Third, explicit PE not an inherent requirement for effective language modeling and can be removed after pretraining, following short recalibration phase at the original context length.
Empirically, DroPE models generalize zero-shot to sequences far beyond their training context, marking sharp contrast traditional positional scaling techniques. Moreover, we show that adapting RoPE models with DroPE does not compromise their original in-context capabilities, preserving both perplexity and downstream task performance. Our findings hold across LMs with different architectures pretrained with up to hundreds of billions of tokens, establishing a new standard for developing robust and scalable long-context transformers.
Contributions. In summary, our main contributions are as follows:
(1) In Section 3, we provide empirical and theoretical analysis of the role of positional embed- dings in LM training, showing their importance significantly accelerating convergence.
(2) In Section 4, we discuss why RoPE-scaling methods fail to reliably attend across far-away tokens when evaluated zero-shot on long sequences, showing that these approaches in- evitably shift attention weights, hindering the model's test-time behavior.
(3) In Section 5, introduce DroPE, new method that challenges the conventional role of positional embeddings in transformers, motivated by our empirical and theoretical analyses of its role transient but critical training inductive bias.
(4) We demonstrate that DroPE enables zero-shot generalization pretrained RoPE trans- formers far beyond their original sequence length, without any long-context finetuning. DroPE can be incorporated at no extra cost into established training pipelines, and can be used to inexpensively empower arbitrary pretrained L.LMs in the wild.
2

---

Published as a conference paper at ICLR 2026
We share our full code with this submission to facilitate future work and extensions toward devel- oping foundation models capable of handling orders-of orders-of-mageen -f-mannitdde longer contexts.
PRELIMINARIES
Self-attention. Let ( by E TRd be the representations fed into the I-th multi-head attention block. Queries qi keys k, and values v are then computed by projecting the inputs h; via linear layers Wo. WK. and Wv. The attention operation then computes T x 'I' matrices of attention scores Siy and weights azj between all pairs of sequence positions, and reweighs value vectors:
Sij
- ki, Ozj = softmax(5: ,.1 Ba)s. vd,
Σ 55
Bij
where de is the head dimension. A multi-head attention block computes multiple attention outputs ...... concatenates them, and projects to the model dimension: = Wolx .....,?"."."" 2, (*)
Language and positional embeddings. State-of-the-art autoregressive transformer LMs use information about sequence positions provided both implicitly via causal masking of the attention scores', and explicitly with positional embeddings (PEs). In particular, the modern literature has settled on the Rotary PE (RoPE) scheme (Su et al., 2024), providing relative positional information to each attention head by rotating q: and kj in 2D chunks before the inner product in Equation I:
(R'g.) (R)) = Jdz 9 R Ri kg: $ij = 15--(RR Jd Jdz
R = block-diag (R(w)). R(way/2)) (2)
Here, each R(wm) E R222 is planar rotation of angle win b- 2m/11//2 acting on the (2m, 2m + 1) subspace of qi and ky. The base b is commonly taken to be 1
Context extension for RoPE. Given the rapidly growing costs of self-attention, adapting LMs for longer sequences than those seen during training has been longstanding open problem. To this end, prior context-extension methods introduce targeted rescaling of the RoPE frequencies in Equation 2 to avoid incurring unseen rotations for new sequence positions. Formally, let the training and inference context lengths be Ctrain < Crest and define the extension factor $ = Crest/ / /Ctrain- Context extension methods such as PI (Chen et al., 2023), RoPE-NTK (bloc97, 2023), and the popular YaRN (Peng et al., 2023) define new RoPE frequencies w/m = Vm/m with scaling factors:
PI
NTK
and YaRN = (1 11--Kyg Km Kyn 7m
m
where K. [0, 1] interpolates between 0 and 1 as the base frequency Wm grows (see Appendix A). These methods, referred to as RoPE scaling, still require additional finetuning on long sequences, and don't 't generalize to long-context downstream tasks out of the box (Lu et al., 2024b).
NoPE transformers. In a parallel line of work, there have been efforts to train transformers without positional embeddings, commonly referred to as NoPE architectures Haviv et al. (2022); Kazem- nejad et al. (2023) to avoid the need for rescaling RoPE frequencies. While NoPE was shown to be a viable LM architecture, it has failed to gain traction due to degraded performance (Haviv al., 2022; Yang et al., 2025b). For an in-depth introduction to the above concepts, see Appendix A.
3 EXPLICIT POSITIONAL EMBEDDINGS ARE BENEFICIAL FOR TRAINING
While NoPE transformers were shown to be expressive enough for effective sequence model- ing (Haviv et al., 2022; Kazemnejad et al., 2023), find that they consistently underperform state-of-the-art RoPE architectures throughout our experiments. As illustrated in Figure 3, NoPE transformers maintain visibly worse perplexity throughout training. These empirical results are consistent with past literature (Haviv et al., 2022: Yang et al., 2025b), yet the reasons why positional embeddings are key for effective language model training have never been fully understood.
From a purely mechanistic perspective, even without explicit positional embeddings, NoPE transformers can exploit the causal mask to encode positional information, maintaining the same expressivity as their RoPE counterparts (Haviv et al., 2022; Kazemnejad et al., 2023). Specifically,
Note the softmax Equation taken the first i tokens, implementing causal mask.
3

---

Published as a conference paper at ICLR 2026
Kazemnejad et al. (2023) prove that the first attention layer in NoPE transformer perfectly reconstruct sequence positions, and subsequent layers can emulate the effects of relative or absolute positional embeddings. As detailed in Section 3.1, rather than looking theoretical expressivity, we investigate this empirical performance diserepancy from training perspective, providing theoretical analysis of the positional bias of NoPE transformers during optimization. The theoretical and empirical analysis in this section can be summarized in the following observation.
Observation 1. Positional information and attention non-uniformity which are crucial for sequence modeling, develop at bounded rate in NoPE transformers. In contrast, explicit PE methods, such as RoPE, provide a strong bias from the outset and facilitate fast propagation of positional information, resulting in faster training.
At high level, analysis focuses the rate which NoPE and RoPE transformers can de- velop positional bias in their self-attention heads, which captures their non-uniformity. We quantify attention positional bias linear functional on the attention map:
Definition 3.1 (Attention positional bias). Given centered positional weights cý E R with Ejgr Cij O, the positional bias of the attention weights aij
T
A (a)
= T
Σ i= JS
cyaaj.
Attention heads with a strong positional bias would maximize the average value of A across input sequences. For example, "diagonal" attention head, focusing mass on the current token, is exactly the maximizer of A", with Cij having 1s on the diagonal and otherwise.
To validate the theory behind Observation 1, empirically compare the gradients of the attention positional bias functional in atten- tion heads of RoPE and NoPE transformers. Specifically, we measure the average gradient norm at initialization common language modeling patterns: diagonal attention heads, placing mass on the current token, and off-diagonal heads, capturing immediate previous token context. As illus- trated in Figure 4, the gradient magnitudes of NoPE transformers are far lower than those of RoPE transformers, with the gap between the two growing in deeper layers. This reflects NoPE's scalability challenges and difficulty in recovering positional information, which we theoretically analyze in the next section.
RoPE NoPE
the direction of two
4K
6K
8K 10K Step
12K 14K 16K
Figure RoPE outperforms NoPE. Training loss curves for RoPE and NoPE transformers on 16B fineweb tokens. RoPE outperforms NoPE throughout training.
3.1 THEORETICAL ANALYSIS
We detail our findings, summarized in Observation 1, with series of formal results, bounding the rate which positional bias can develop early in training. We provide full proofs and an extended analysis of these results in Appendix B. Throughout this section, we study the sensitivity of the attention positional bias A to the transformer's parameters and interpret ||V,A"]] as bounding the rate at which non-uniform attention patterns emerge during training.
Warm-up: NoPE transformers break on constant sequences. Before moving to the main theoretical result, we consider motivating example that illustrates NoPE transformers' training difficulties. Because attention forms a convex combination of value vectors, an attention head applied sequence of identical tokens position. Moreover, since normalization layers, MLP blocks, and residual connections act pointwise tokens, this uniformity propagates through the network. In NoPE transformer, this means
zy produces identical outputs at every
4

---

Published as a conference paper at ICLR 2026
NoPE
Avg. Grad Norm
Grad Norm
0.9
RoPE
0.6
0.6
0.4
15
10 Layer Index
Layer Index
(a) Diagonal head bias.
(b) Off-diagonal head bias.
Figure 4: RoPE transformers have higher positional bias gradients at initialization. We com- pare the average norm A across layers, for RoPE and NoPE transformers. In 4a we plot gradient norms of positional bias towards diagonal head, and in 4b, we take bias towards previous token attention, off-diagonal head. In both cases, the gradient norm is consistently higher for RoPE across layers, meaning that RoPE heads can learn these patterns faster.
the attention logits are constant over all S hence the post-softmax attention probabilities uniform. Consequently, the model cannot induce any positional preference and 0 for any positional weights
Proposition 3.2. Let M be NoPE transformer. If the input sequence = ( FT) is comprised of identical tokens 21 =... 2r. then (1) all attention heads are uniform: asj } (2) query and key gradients vanish: @C/OWQ @C/OWK = O, (3) for all heads and any positional weights A O, A = 0 and (4) ouiput constant: M(x) = M(x)
The explicit positional information injected into attention heads in RoPE transformers circumvents this issue. Enabling non-zero A gradients even on constant sequences.
Proposition 3.3. For non-trivial RoPE attention head, even if the input sequence is constant, there are positional weights c, for which A > O, and V A > 0.
NoPE transformers propagate embedding uniformity. embedding matrix are drawn i.idd. from a distribution with a fixed small variance (commonly, x2 0.02). Therefore, the token embeddings are close uniform at the beginning training. The next theorem shows that for NoPE transformers, this uniformity persists throughout the network, and bounds the attention positional bias A and its gradients.
initialization, the entries of the
Theorem 3.4. Define the he prefix-spread of the hidden states at layer
1 1 = max
|| - -h where
R = Σ
1S1SIT
jzt
For NoPE transformers, there exists > 0 and constants C₁, C2. and C3 such that if the initial embeddings ( < then for all layers < L:
< CGE,
||A (oAs//Wo// - /oW S Cos,
with high probability over the initialization distribution. The constants only depend on the num- ber of layers and heads, and not the sequence length.
The main idea in the proof Theorem 3.4 is that uniformity in the embeddings causes uniformity in the attention maps, so 0ij 1/i. Uniform mixing of tokens cannot increase the prefix spread; thus, uniformity persists throughout the network. This result explains the discrepancy between RoPE and NoPE transformers illustrated in Figure
5

---

Published as a conference paper at ICLR 2026
7
Training content
100%
Training contort
Log perple plexittyy
6
Accuracy (95)
75%
50%
25%
3
0%
1K
2K
3K
4K
0
1K
2K
3K
Context length (tokens)
Distance to needle (tokens)
RoPE
RoPE YARN
RoPE + Cropped to 2048
Figure 5: YaRN crops effective retrieval context. We compare RoPE's and YaRN's perplexity and NIAH performance at up-to 2x the original context length against baseline that crops the input sequence to the training context length. Both YaRN and the cropped baseline can maintain perplexity on sequences exceeding the training context length, but are unable to retrieve information placed far away from the query.
In summary, we demonstrate that while NoPE attention can learn positional bias, attention non- uniformity develops slowly early in training due bounded A gradients initialization.
4 ROPE PREVENTS EFFECTIVE ZERO-SHOT CONTEXT EXTENSION
State-of-the- ahr- RoPE scaling methods fail to effectively generalize to sequences longer than those seen in training without additional long-context finetuning. While YaRN and other popular frequency scaling techniques do avoid perplexity degradation long-context sequence (bloc97, 2023; Peng et al., 2023), they exhibit sharp performance drops downstream tasks whenever important infor- mation is present deep in the sequence, beyond the training context (Lu al., 2024b; Liu al., 2023b). We empirically demonstrate this phenomenon, comparing the perplexity and needle-in-a-haystuck (NIAH) (Kam- radt, 2023; Hsieh et al., 2024) performance of RoPE transformer scaled with YaRN and cropped context baseline. As illustrated in Figure 5, YaRN's zero-shot behavior closely matches that of simply cropping the sequence length to the pretraining context, maintaining constant perplexity but ignoring information present outside the cropped window.
1.0
Vm (scaling factor)
Pl
ROPE-NTK
YaRN
0.8
0.7
0.5
10-4
10-
10-2
10
10
Wm (original RoPE frequency)
Figure RoPE frequency scaling under PI, NTK-aware scaling (RoPE-NTTK) and YaRN, with scaling factor 2.
The cause of this limitation lies the way context extension methods scale different RoPE frequencies. As detailed in Section 2, elaborated on in Appendix A, and illustrated in Figure 6, the scaling factors of PI (Chen et al., 2023), RoPE-NTK (bloc97, 2023), and YaRN (Peng et al., 2023) have strong effect on low frequencies. In Section 4.1, discuss why this scaling leads to the observed failures, yielding our second observation.
Observation 2. RoPE--caling methods must compress low frequencies keep positional phases in-distribution. This, turn, shifts semantic attention heads at large relative distances, causing the observed failures on downstream tasks, preventing zero-shot context extension.
4.1 WITY EXTRAPOLATION FAILURE IS INEVITABLE
Effect of RoPE scaling. RoPE scaling methods modify the frequencies at inference time to evaluate sequences that longer than those seen during pretraining. In each (2m, 2m++1) subspace, the RoPE phase at relative distance is (A) = A, so scaling the frequency to a Nm is equivalent to using phase d',, () Tmmim A. As illustrated in Figure 6, most scaling methods

---

Published as a conference paper at ICLR 2026
RoPE
504
505
506
507
508
509
510
521
512
513
514
515
516
517
518
519
YaRN
508
521
Key position
Figure 8: RoPE scaling shifts semantic attention mass. Attention weights of the last token (query) with tokens from a retrieval target (keys) in a semantic head evaluated NIAH probe. Since the head uses low frequencies and the relative distance is non-trivial, the impact of YaRN is substantial, shifting attention mass between tokens.
leave high frequencies nearly unchanged (rin 1) but all of them compress the low frequencies (Tm = 1/ss. As demonstrated both theoretically and empirically in Barbero et al. (2024), high RoPE frequencies are primarily used by positional heads, with attention patterns based on relative token positions (c.g., diagonal or previous-toker heads), whereas low frequencies are predominantly used by semantic heads that attend based on query/key content. Consequently, positional heads are largely unaffected by scaling, but semantic attention is shifted. Moreover, the effect on low- www-rrqueecy dominated semantic heads is exacerbated for distant tokens, since the relative phase &m (A) larger, and thus the 1/s scaling factor has a greater effect. In other words, scaling warps low-freqe cow-frequency phases, shifting long-range attention in precisely the subspaces most used for semantic matching.
In Figure 7 and Figure 8, illustrate this be- havior in practice. We start by selecting a posi- tional attention head in pretrained QWEN2.5- 0.5B model by examining its average atten- tion positional bias (Definition 3.1) across lay-
RoPE
YaRN
w
Query position
*
80
ers. In Figure 7, we show the average atten- tion weights in this positional head under YaRN scaling with s = 2. Because high frequencies, which are least affected by YaRN, dominate po- sitional heads, the average attention profiles are similar. In Figure 8, we then contrast this be- havior with that of semantic head for long needle-in inelee-i----iaie aon-a-haystack sequence, plotting the average attention of the last token (query) with tokens around the needle (keys). YaRN's aggressive scaling of low frequencies substantially shifts attention mass across tokens, reflecting the impact of frequency compression at longer ranges.
10
10
6
12 16
6 10 12
Key position
Key position
Figure 7: RoPE scaling preserves average at- tention in positional heads.
Why this is inevitable. In standard RoPE setup, fow-frequency phases never make full cycle over the original context length: @m (Ctrain) = Wm. Ctrain < 2 for small w. E.g. for a standard RoPE base 10a transformer with head dimension dę 64, will have at least five low fre- quencies for which @m (Ctrain) < 2m, even at a training context of Ctrain = 32,000. If we leave Wm unchanged at an extended length Crest > Ctrain- the new maximal relative phase dm (Crest) is pushed outside the training regime and becomes out of distribution for the head. Therefore, to con- strain phases to remain in range, any scaling method must choose Tm I Crrain = J. which becomes increasingly small as the extension factor grows. In other words, when applying a RoPE trans- former sequences longer than those seen in training, any post-hoc scaling method must compress the low frequencies. But this compression, in turn, shifts attention weights at long relative distances.
C.ast
DROPE: DROPPING POSITIONAL EMBEDDINGS AFTER PRETRAINING
Taken together, Observations 1 and 2 imply that providing explicit positional information with PE key component for effective LM training, but is also fundamental barrier long-context generalization. This raises natural question: is it possible to harness the inductive bias from positional embeddings exclusively during pretraining? We answer the affirmative. In this section, we empirically show that it is possible to drop all positional embeddings from pretrained transformer and quickly recover the model's in-context capabilities with a short recalibration phase. Most notably, this simple new procedure (DroPE) unlocks strong zero-shot long context generalization to unseen sequence lengths, far beyond highly-tuned RoPE extensions and prior alternative architectures.
7

---

Published as a conference paper at ICLR 2026
ARC-E
ARC-C
HellaSwag
OpenBookQA
PIQA
Winogrande
Average
60
Score
50
40
30
20
Dropt (3087) Cooge (s0m tronk (2xa7) Small (50597)
Drope (Seam)
Grarg (sam Grong (12087) Smallu (60001))
Drset (seam)
Croit 1e0m trarg (20887)
Smallal (66u8))
Dnoees (resm) Croit deam Drove Smalllt (occary
Cmmee (Surr) Drift reasy) trere (20897) Smellla (botar)
Grarg (seam) Drait (d099)) Grart (1 SmolLM (oovar)
Crowt (3007)
Drawn eearr) troft (22097
Smollk (600877)
Figure 9: DroPE matches base model in-context performance. Comparison of base SMOLLM with SMOL.M-DROPE on standard LM benchmarks, using three recalibration recipes.
Positional embeddings can be removed after pretraining, allowing LMs to generalize zero-shot to unseen sequence lengths without compromising their in-context perfor- mance after short recalibration on fraction of the training tokens at the original context size.
Observation 3.
5.1 LARGE-SCALE EMPIRICAL EVALUATION
We extensively validate DroPE with two different LMs and dataset scales, showing it outperforms prior approaches both as a zero cost integration into pretraining recipes and as an inexpensive way to adapt any LM in the wild already pretrained on hundreds of billions of tokens. For all experiments this paper, we provide full implementation details of each evaluated architecture and optimization phase, including comprchensive hyperparameter lists in Appendix C.
Integrating DroPE at no extra cost. For our first set of experiments, we train from scratch different LMs with half a billion parameters on 16B fineweb tokens (Penedo al., 2024), over twice the chinchilla-optimal rate (Hoffmann et al., 2022). We repeat this recipe for RoPE and NoPE trans- formers, well as RNoPE-SWA model Yang et al. (2025b), alternative architecture specif- ically aimed at long-context capabilities. We implement DroPE by taking the 14B tokens RoPE transformer checkpoint, removing positional embeddings from every layer, and resuming training for the final 2B tokens. Despite only recalibrating at the very end of training, at no extra cost, DroPE matches the final in-context validation perplexity of RoPE trained on the full 16B tokens, showing a clear edge over the NoPE baseline trained without positional embedding all the way (Figure 2).
To evaluate the long-context generaliza- tion of each method, we select three tasks from the RULER benchmark (Hsieh et al., 2024): (1) multi-query: retrieve needles for several listed keys, (2) multi-key: retrieve the needle for one specified key, and (3) multi-value: retrieve all needles for one key with single query. For the base RoPE transformer, we consider three context extension strategies: PI (Chen et al., 2023), NTK-RoPE (bloc97, 2023), and the popular YaRN (Peng et al., 2023) described in Section 2 and Appendix A.
Table 1: Zero-shot NIAH at 2x training context. Re- sults are reported as a success rate over 500 trials.
Multi- Query
Multi- Key
Multi-
Method
Value
RoPE transformer
0.0
0.0
0.0
RoPE transformer Pl
0.0
0.0
0.0
RoPE transformer RoPE-NTK
21.1
19.4
16.5
RoPE transformer YaRN
17.8
0.5
14.6
ALiBi transformer
5.2
0.0
1.1
NoPE transformer
9.2
36.2
21.4
RNoPF-SWA transformer
5.2
25.6
20.6
DroPE transformer
28.0
41.6
23.3
For our DroPE and NoPE transformers, we follow Wang et al. (2024) and scale the softmax temperature at test time; we report the exact scaling scheme in Appendix C. 'In Table 1, we report the success rate on each task at 2x the training context length. Our DroPE transformer substantially outperforms all of our baselines in each setting. While RoPE-NTK and YaRN also yield improvements to the original RoPE transformer, they consistently trail DroPE, as most evident on the multi-key task. In contrast, specialized architectures such RNoPE-SWA (Yang et al., 2025b), ALiBi (Press et al., 2021), and NoPE Kazemnejad et al. (2023) underperform on multi-query tasks, which are the logic-intensive setting where strong base models excel. We believe these results provide compelling evidence toward validating DroPE's potential to be integrated as a standard component in the training pipeline of future generations of L.Ms.
8

---

Published as a conference paper at ICLR 2026
Table 2: DroPE outperforms RoPE-scaling methods on long context-tasks. We evaluate SMOLLM- --DoPPE and the base SMOLLM model, extended with different RoPE scaling methods, on four long context language modeling tasks from Bai al. (2023) and seedle-in-a-hasttare
Method
MultiFieldQA
MuSiQue
GovReport
LCC
NIAH
Avg.
SMOLLM
4.03
0.4
4.48
5.99
0.0
2.98
SMOLLM +
13.68
2.45
5.67
11.52
0.0
6.66
SMOLLM RoPE-NTK
18.87
4.89
23.71
8.26
29.84
17.11
SMOLLM + YaRN
20.78
4.77
15.03
10.87
48.25
19.94
SMOLLM MMOOLE DROPE
29.33
7.93
21.87
18.56
74.92
30.52
Extending the context of LMs in the wild with DroPE. For our second set of experiments, we directly apply DroPE to recent language model from the SMOLLM family (Allal al., 2024) that was pretrained on over 600B tokens. We perform DroPE's recalibration for this model with continued pretraining using the same context length, data, and hyperparameters as reported by Allal et al. (2024). We consider three different recalibration budgets of 30, 60, and 120 billion tokens, adjusting the learning rate schedule accordingly. order support training at a high learning rate as perscribed by Allal et al. (2024), after applying DroPE we add QKNorm (Henry et al., 2020). We note that this addition not change the capacity of the our model, as analyzed in Appendix
We start by analyzing how quickly SMOLLM-DROPE can recover SMOLLM's in-context perfor- mance on six different LM reasoning benchmarks (Clark et al., 2018; Zellers et al., 2019; Mihaylov et al., 2018; Bisk al., 2020; Sakaguchi et al., 2021). As shown in Figures 9 and 10 well as Table 6, our shortest training schedule almost matches SMOLLM on every task while our longest schedule even manages exceed its original performance. Furthermore, inspecting every checkpoint throughout training, we find that DroPE quickly recovers over 95% of SMOLLM's performance after less than 5B tokens, representing minuscule 0.8% of SMOLLMs original budget.
model
We then evaluate SMOLLL--DOPE'' zero-shot length four different tasks from Long- Bench (Bai et al., 2023), a challenging benchmark even for closed-source LMs. including knowledge--xttrattoon problems longer than 80 times SMOLLM's pretraining context (2048 tokens). We compare our method with the base SMOLLM and three RoPE extensions: PI, RoPE- NTK, and YaRN. As shown in Table 2, despite a signif- icant difficulty spike compared our prior evaluations, DroPE still displays clear edge over prior approaches, improving the base SMOLLM's average score by over 10 times. These gains are far beyond all prior zero-shot RoPE extensions currently used across modern LMs. Ad- or needle-in-
54
generalization
Avg. evalution score
50-
48
46-
608 tokam DM
Original model
08
208
408
608
80B
1009
1208
Training tokens (B)
Figure 10: SMOLLM-DROPE recali- bration. We compare three recipes, us- ing 30B, 60B, and 120B training tokens.
ditionally, we evaluated SMOLLM-DROPE a-haystack tasks of size up 8x SMOLLM's original context length.
Table 3: DroPE outperforms RoPE-scaling methods on long needle- sesellein---hayaenn in-a-haystack tasks.
Method
2x original context
4x original context
8x original context
SMOLLM + ROPE-NTK
29.84
14.37
7.19
SMOLLLM + YaRN
48.25
25.62
12.18
SMOLLMM +
BOALLMM+LLoggoPPH LongRoPF2
44.20
26.20
16.45
SMoI.LM-ROOPE
74.92
555.00
52.20
Overall, our comprehensive in-context and out-of-context results provide a clear demonstration of new, efficient, and effective long-context adaptation, which we believe can have concrete im- plications for reducing training costs and tackling the canonical context scalability challenges of transformers.

---

Published as a conference paper at ICLR 2026
Table Length generalization results on larger models. We evaluate DroPE on SMOLLM-1.7B and LLAMA2-7B, and compare it against different RoPE scaling methods, on long context language modeling tasks from Bai et al. (2023).
Model
Method
MultiFieldQA
MuSiQue
GovReport
Avg.
Base
4.12
0.50
4.70
3.11
RoPE-NTK
27.58
3.37
24.65
18.53
SMOLLM-1.7B
YaRN
27.60
3.90
17.19
16.23
DroPE
32.18
7.53
24.77
21.49
Basc
17.26
10.43
32.41
20.03
RoPE-NTK
21.81
10.91
32.91
21.88
LLAMA2-7B
YaRN
23.13
7.65
26.65
19.14
DroPE
25.90
12.88
39.47
26.08
Scaling to larger models. To test DroPE's ability to scale to larger LMs in the wild we additionally apply DroPE SMOLLM-1.77 (Allal et al., 2024) and LLAMA2-7B (Touvron et al., 2023). For both of these models, we perform recalibration on 20B tokens. For SMOLLM-1.77 sents 2% of the pretraining budget and for LLAMA2-7B the recalibration represents only 0.5% of the pretraining budget. As demonstrated in Table 4, even with this small relative recalibration bud- get, SMOLLM-1 1.....--.B---E... 1.7B-DROPE and SAAMA---B-DRRREE outperform state-of-the-art RoPE-scaling and summarization. For additional experimental re-
this repre-
methods on long-conter sults, including results on the entire LongBench benchmark, and performance by query length breakdown, see Appendix D.
question-answert
RELATED WORK
Recent improvements to RoPE include variants based on Fourier and wavelet transforms (Hua et al., 2025; Oka al., 2025) and methods such as p-RoPE (Barbero al., 2025), NRoPE-SWA (Yang et al., 2025b), and SWAN-GPT (Puvvada et al., 2025) which occupy a middle ground between RoPE and NoPE, Our approach represents fundamentally different paradigm, replacing RoPE with NoPE at different stages of training. These directions are complementary to ours and can be used in place RoPE within the DroPE framework. parallel direction seeks length generalization while retaining dedicated positional vector yet modifying its indexing or adaptivity (zican Dong et al., 2024; Wu al., 2024; Zheng al., 2024). Another line of related works considers post-training architectural modification, e.g., for efficient inference (Ji et al., 2025).
DISCUSSION AND EXTENSIONS
Our findings support a reinterpretation of positional embeddings in transformer LMs as a useful inductive bias that is essential for efficient training (Observation 1), but inherently constrains zero-shot context extension (Observation 2). Based on these findings, we propose DroPE, a new method rethinking the conventional role of PEs a temporary scaffold that can and should be removed after serving their training-tioe time purpose (Observation 3). We empirically validate DroPE across different models and data scales, showing its effectiveness and potential be integrated new core component of future state-of-the-ann training pipelines. Integrating DroPE in state-of-the-art training pipelines for autoregressive or diffusion-based I.Ms is an interesting direction for future work. More broadly, our work demonstrates that canonical trade-offs in LM design can be reconciled by employing different architectural choices for training and inference, which we hope will inspire further research toward challenging established bottlenecks in AI.
ETHICS STATEMENT
DroPE aims to further advance the field of long context language modeling by removing positional embeddings after pretraining. Our intended use to advance robust, efficient, context reason- ing in open research environments and practical systems. Our experiments training corpora and benchmarks. Given this, we foresee no issues regarding fairness, privacy, or security, any other harmful societal ethical implications in general.
publicly available
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We provide our full source code the supplementary material and provide all details of our experi- mental setup- includinng datasets, model specification, training regime, evaluation protocol, and the full set of hyperparameters- - in Appendix C. Our base models and datasets are all publicly available.
ACKNOWLEDGMENTS
YG is supported by the UKRI Engineering and Physical Sciences Research Council (EPSRC) CDT in Autonomous and Intelligent Machines and Systems (grant reference EP/S024050/1).
REFERENCES
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. Smollm blazingly fast and remarkably powerful, 2024.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXi::2388. 14508 2023. doi: 10.488500arXiiv2208.45555
URL httpp::/arxiv, org/ abs/2308 14508.
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round go! what makes rotary positional encodings useful? arXiv preprint arXiv:2410.06200. 2024.
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference Learning Representations, 2025. URL https:
//openreview. http://fooummmdttttttoeeeoeeeeee
Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Smollm-corpus, 2024. URL https:/ htts:///hgggggacceee.. co/datasets
HuggingFaceTTs smol 1m- corpus.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
bloc97. Ntk-aware scaled rope allows llama models have extended (8k+) context size with- out any fine-tuning and minimal perplexity degradation. Reddit post on L/Lo June 2023. URL https::/tt//ttte teckaarre_scaed__roo aaloow watrs__lammmmddeee tt have/. Accessed: 2025-09- 19.
reddit........ http:/tttww..dd....c//e/../.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15550 2023.
Ta-Chung Chi, Ting-Han Fan, and Alexander I Rudnicky. Attention alignment and flexible positional embeddings improve transformer length extrapolation. arXiv preprint arXiv:2311. 1.00684, 2023,
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. /4794, 2020.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 0803 05457, 2018

---

Published as a conference paper at ICLR 2026
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344- 16359, 2022.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending Ilm context window beyond 2 million tokens. arXiv preprint urXiv:2402.13753. 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1192: 2020.
Nathan Habib, Clémentine Fourrier, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall. Lighte- val: A lightweight framework for Ilm evaluation, 2023. URL https::/gitte com/
huggingface/ lighteval.
Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. arXiv preprint arXiv:2203. /6634, 2022.
Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers. arXiv preprint arXiv:2010.042455 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute- -pttiaal large language models. arXiv preprint arX3v:2203.155550 2022.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's 's the real context size of your long-context language models? arXiv preprint arXiv:2404.0666 2024. doi: 10.48550laxXi.2444..666. URL https:// //ar:v//rxvvvv/// org/ abs 2404 0664.
Ermo Hua, Che Jiang, Xingtai Lv, Kaiyan Zhang, Youbang Sun, Yuchen Fan, Xuckai Zhu, Biqing Qi, Ning Ding, and Bowen Zhou. Fourier position embedding: Enhancing attention's 's periodic extension for length generalization. In Forty-second International Conference on Machine Learn- ing, 2025. URL https:://peeeeeeeeeeee
Tao Ji, Bin Guo, Yuanbin Wu, Qipeng Guo, Lixing Shen, Zhan Chen, Xipeng Qiu, Qi Zhang, and Tao Gui. Towards economical inference: Enabling deepseek's 's multi-head latent attention in any transformer-bas lims. arXiv preprint arXiv:2502.14877 2025.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583-5555 58933,5594400 2021.
Greg Kamradt. Needle in haystack - pressure testing 1lms. https: http//gtttth/.ee/////// com/ 2023. GitHub repository.
EkkarradtLLLMeest.. htw.reett_eedllleaaaaaaaatttt.tt
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva
Reddy. The impact of positional encoding length generalization in transformers. Advances in Neural Information Processing Systems, 36:24892-224922 2023.
Hao Liu and Pieter Abbeel. Blockwise parallel transformers for large context models. Advances in neural information processing systems, (8:8828-8844,, 2023.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near- infinite context. arXiv preprint arXiv:2310.01889, 2023a.
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172. 2023b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17110 2017.
12

---

Published as a conference paper at ICLR 2026
Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M Rush. A controlled study on long context extension and generalization in Ilms. arXiv preprint arXiv:2499.12181, 2024a.
Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T. Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M. Rush. A controlled study on long context extension and generalization in Ilms, 2024b. URL hitps: : http//axvvvv.////// org/ abs
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering, arXiv preprint arXiv:1809.022 02789, 2018.
Yurii Nesterov. Introductory lectures convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013.
Yui Oka, Taku Hasegawa, Kyosuke Nishida, and Kuniko Saito. Wavelet-based positional represen- tation for long context. In The Thirteenth International Conference on Learning Representations, 2025. URL htwpt:tttth/...tt
http///ppeeeeeeeeeeeeeeeeee /
net/ fo Furum? dd=0hhuMUNSTT..
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501. 00656, 2024.
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances Neural Information Processing Systems, 37:3081 1-30849, 2024.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint (=xxxxx339.00071 2023.
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2 12409, 2021.
Krishna C Puvvada, Faisal Ladhak. Santiago Akle Serrano, Cheng-Ping Hsich, Shantanu Acharya, Somshubra Majumdar, Fei Jia, Samuel Kriman, Simeng Sun, Dima Rekesh, et al. Swan- gpt: An efficient and scalable approach for long-context language modeling. arXiv preprint arXiv:2504.087/9, 2025.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, dean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018
Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang, and Xiaoling Wang. Length generalization causal transformers without position encoding. arXiv preprint 2404 /2224, 2024.
13

---

Published as a conference paper at ICLR 2026
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang. and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04776 2020.
Tong Wu, Yanpeng Zhao, and Zilong Zheng. An efficient recipe for long context extension via ficuue- positional encoding. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:/ wwwpopeeeeeew..mmm net/ ftrrmmormmm/ ?id= aNHEQFMSON.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty. Mingxing Tan, Glenn Fung, Yin Li, and Víkas Singh. Nystrômformer: A nyström- based algorithm for approximating self-attention. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 14138-14144 2021.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report, 2024. URL https://arxiv. org/abs/2407.1067 1, 7:8, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
Bowen Yang, Bharat Venkitesh, Dwarak Talupuru, Hangyu Lin, David Cairuz, Phil Blunsom, and Acyr Locatelli. Rope to nope and back again: A new hybrid attention strategy. arXiv preprint 1rX1v:2501.18795, 2025b.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences, Advances in neural information processing systems, 33:17283-17297 2020.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprini 1905vv:905.007
2019.
Chuanyang Zheng, Yihang Gao, Han Shi, Minbin Huang, Jingyao Li, Jing Xiong, Xiaozhe Ren, Michael Ng. Xin Jiang, Zhenguo Li, and Yu Li. DAPE: Data-adaptive positional encoding for length extrapolation. In The eiihty- Annual Conference on Neural Information Processing Systems, 2024. URL https: http:////ppeeeeeeeeeeeeee net http:/eent/ouo/ie/////// forum?rrummmmmmmmm id=rnUEUbRxV..
zican Dong. Junyi Li, Xin Men, Xin Zhao, Bingning Wang, Zhen Tian, weipeng chen, and Ji-Rong Wen. Exploring context window of large language models via decomposed positional vectors. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL
https: /ttps///ppppeeiiee weeiiiwwweet..... .nt/ fest//formmiii wew.uummddzzexyyxgeeee
A EXTENDED PRELIMINARIES
Attention. Throughout this section, we consider a pre-norm, decoder-only transformer with L layers, H attention heads per layer, model dimension d dmodel, and head dimension dg-
(1) E Rd denote the representations fed into the 1-th multi-head attention block. For head h in layer 1, queries, keys, and values are computed by
(1)
q: (1h) = Q
(,)))
(1,b) W. (0)
(L,)) w.(l n
(4)
K
The attention scores and weights are then computed by
(L.N)
(a dd
(L.)
(L.)) (
(1,8) aij
= softmax(s
(1,A)
(1.A)
(5)
(L.b) are referred to as attention weights or probabilities. Note that the softmax is taken over j < i, implementing a causal mask. The output of the multi-head attention block is
(1.A) are referred to as attention logits or scores and a Sij
(L.) (L.))
o (
w (L.I
(L.I)
(LH))
where l....., represents concatenation along the feature dimension. When clear from context, we omit layer and head indices.
14

---

Published as a conference paper at ICLR 2026
Positional embeddings in transformers. The attention mechanism does not directly encode rel- ative distances between queries and keys. Therefore, attention is invariant to prefix permutations: for any permutation a Sp of the first p input tokens, attn(z0-<(1)+ (1)............................. ........................ ( pp...... 2r) attn(21,. ... ,T), for every address this, Vaswani et al. (2017) introduced absolute positional embeddings, adding position in- formation to the token embeddings before the first transformer block. More recently, many architec- tures replace absolute embeddings with relative schemes that inject pairwise positional information directly into the attention mechanism. The most widely used approach is Rotary Position Embed- ding (RoPE) (Su et al., 2024). RoPE modifies the attention scores in Equation by rotating queries and keys before taking their inner product:
p. In other words, pure attention is blind to token positions. To
RoPE
vd 41 R 'kj,,
RoPE = softmax (8.1 b)
RoPE
RoPE
where, R O(2)) a block-diagonal orthogonal matrix composed out of 2 rotation blocks:
(cos(w) (in((w) cos(w)
sin(ws)
R= block-diag (R(w)),.... ((wa,))).....,,,22) R(w)
(8)
2 with b 10,000.
In the standard RoPE parameterization, Win
Language model context extension. Generalizing to contexts longer than those seen during train- ing is a key challenge for transformer-bused language models. The key issue is that when applying a transformer on a longer context, the attention mechanism must operate over more tokens than was trained to handle. This issue is exacerbated with RoPE: applying RoPE to sequences beyond the training length introduces larger position deltas, and thus larger rotations, pushing attention logits out of the training distribution. RoPE context-extension methods address this by rescaling the RoPE frequencies when the inference context length exceeds the training context length. Let Crrain be training context and Cust > Ctrain the target context with extension factor Ctest /Ctrain- Such methods define new frequencies
w'm = 7m
m
using scaling factors T'm Vm (s). E.g. Position Interpolation (PI) (Chen et al., 2023), uses uniform scaling of
PI 7m
(9)
NTK-RoPE (bloc97, 2023) uses
7im
NTK
2%
2m
(10)
so that low frequencies (m ss d2/2) are scaled similarly to PI and for high frequencies Ym 2 1. YaRN (Peng al., 2023) uses
Wm < p
YaRN 7m
(1 kim) + Kmn
Fimm
Wm >
(11)
Wm S g.
g-p
with tunable p and q parameters, originally chosen as as == 1, q = 32, See Figure for a comparison between these different RoPE scaling methods with 2, 3, and 4.
B THEORETICAL RESULTS AND PROOFS
In this section, we analyze the behavior of positional bias, or attention non-uniformity. in NoPE transformers and RoPE transformers early in training. We provide formal statements and proofs for all the results from Section 3, starting with Propositions 3.2 and 3.3, followed by Theorem 3.4. The notation of this section follows that of Appendix A.
15

---

Published as a conference paper at ICLR 2026
1.0
Vm (scaling factor)
0.9
0.8
0.7
0.6
0.5
0.4
0.3
10
10
10
10-1
10
Wm (original RoPE frequency)
Pl (s= 3,
RoPE-NTK (5= 2, 2.
YaRN (s 2, 2,3,4) 4)
Figure 11: RoPE frequency scaling under PI, NTK-aware scaling (RoPE-NTK), and YaRN, with scaling factors s 2, 3, 4.
B. PROOF OF PROPOSITION 3.2
Proposition 3.2. Let be NoPE transformer. If the inpul sequence x (x1,..., is ZT. then (1) all attention heads are uniform: ai (2)
comprised of identical tokens 21 query and key gradients vanish: aC/oWq = 0C/0WK O, f3r all heads and any positional weights A 0, A 0, and (4) the output is constant: M(x)1
M(x)T.
Proof. Let 21...., be constant input sequence, ... Iy, and let M be NoPE trans- former, i.c. a transformer with no positional encodings and causal self attention. The order of the proof is (4) = (1) = (2 3).
(4) Layer outputs, and thus model outputs, are constant. At the first layer, inputs are identical ( A = h. This means that for every attention head and every 1 < S T Wvh.
fy
Therefore, the output of the attention head is
Σ JSL
O5 =
Oij
1.
JSL
independent of Concatenating heads and applying Wo preserves equality across positions. Resid- ual connections, LayerNorm, and the MLP are positionwise (the same function is applied indepen- dently at each position), so identical inputs produce identical outputs at every position. Thus the layer output remains constant. By repeating this argument layer-by-layer, every subsequent layer receives identical inputs and outputs identical states, so the end
M(x)
M(x)..
(1) Uniform causal attention. Using (4), we know that for every layer S
1 -...- 112 h.
Therefore, for every attention head and every ST
= = Woh, k, = = Wxh,
= =::== Wyh.
Thus, for each of or j). Hence
< < < T. the attention scores $ij = q k/ /dly = are constant (independent
Os softmax ()) = (i
entries
16

---

Published as a conference paper at ICLR 2026
(2 3) Vanishing Wq. Wx gradients. Since, the inputs for every layer are constant, know from (1) that every attention head has aij = 1/i, independant of Wq and WK. Therefore 0azy/ / = Daag/WWs k = O. Since the attention bias A depends the parameters 9 only through az and the loss & depends on We and W only through Orij. all these gradients vanish, More formally, using the chain rule,
0A
1
Σ
daij O, Cif
00
00
ISiSIST
0C aw
OL daij 0,
0C
OL doyj dayj
E
Σ
daiz
dW
OWk
dWk K
ISiSIT
IsisiTT
Additionally, since the heads are uniform the attention bias is zero begin with
T
A T
Σ ISSSTT
Σ 15
=
T
T 1m
Remark B.1. Note that part (4) of the proposition holds for RoPE transformers as well. Parts (1), (2) and (3) do not. The relative rotations break attention uniformity and thus changing the magnitude of ||Woll |I and ||Wx|] can affect the attention weights. This formally demonstrated the next section.
B.2 PROOF OF PROPOSITION 3.3
Proposition 3.3. For a non-trivial RoPE attention head, even if the input sequence is constant, there positional weights c, for which A O, and || 0.
Proof. Let T1 Rd xd be the query and key projection parameters. Since the projection maps shared across tokens, the queries and keys are constant as well:
FT
Rdd be the inputs to RoPE attention head, and let Wo, WK
= Wo Wox = 4
hy = WK = WK* k.
Set the positional bias weights to be
Cj 0
Since Ejst aij = 1. we have Ejss Cig = 0 as required. The positional bias A is
T
A
Ja fais)
-
i=1 iS
1=1
By Cauchy-Schwarz,
1 =
Σ Oii 1
E1)=
= Σ a
jSi
with equality only when a:1 = atf Therefore,
Σ
7
JS
with equality iff oi = 1/i is uniform. Therefore, following lemma asserts that this is not the case
0 unless ai uniform for all The
Lemma B.2. For any non den-eegeee RoPE head and input embeddings T = 2, = x, there exists i such that 8. 8ü and Ofly.... Ois are not uniform.
17

---

Published as a conference paper at ICLR 2026
The proof of Lemma B.2 is at the end of this subsection. As for V rewrite A as
A
Σ
so the dependence in the parameters 0 is entirely through
F. =
Σ
JSP
From the definition of RoPE, we have
1
oij softmax(si1..... 845)s
8ij
ddz. 9 R
Consider scaling by scalar À >> 0: q Aq. For fixed prefix i, define
edsis
(A) = Σ
ais(A)
B(A) = Σ aai
Za(2)'
JSi
85i
Then
Z((2 Z;(A)
FA(A)
log F.(A)
R E -2 22 Baaa (2)
dx
((22
A((A)
where A,(A) == log (A) is the log see-partition function. The second derivative of the log partition function is the logit variance
A((A) (A) (2((A)) (Z((A))2
Ejsi 2,(2)
Eiss sijedass 7₂(A)
(a)
therefore AÇ(A) (AI since from Lemma B.2 sij not all equal and aij(A) 0. Thus, A((A) is strictly increasing in X. Hence, for any with non-constant logits,
F((A) = F.(A) 2((((22) A(OA > O.
dA
and in particular at A = 1,
P((A)
>0. d=1
d
By the chain rule for + Ag.
= VFF((O) q. x=1
F((A)
dA
Thus v2D(() * 0 (otherwise the dot product with & couldn't be strictly positive). Finally, since q = Wo..
V Fi Fg
and with + we get [oF.
Vwa
F, 0.
Therefore
T
A
V
T
has strictly positive norm sum of nonzero matrices sharing the same nonzero right factor cannot be the zero matrix unless all left factors vanish, which they don't for i 2 2).
To conclude this section, we now prove Lemma B.2.
Lemma B.2. For any non-degenerate RoPE head and input embeddings T = 2, = ε, there
exists i such that s.1,.-.; 8ü and 041,.... Ois are not uniform.
18

---

Published as a conference paper at ICLR 2026
Proof. RoPE acts independent 2 rotations disjoint coordinate pairs. Thus
M
R R(Awm), M dx/2
with pairwise distinct frequencies Wm E (0, 2T).. Decompose (q)...... 9m), k (ki...... kM),
am bm E IR?,
so 81j = ff where
S(A) 1
T Um RR( wm) Nim
dk m=1
Let
0os ①
sin Φ
J= (1
0 -1 )
R(6)
sin Ø
cos
For any u, E R.?
R( v = (u v) cos d (u Ju) sin db
Define Am = ki and Bm Una Jbm Then
M
M
f(A) =
Σ dR m=l
(Am cos(Awm) + Bn sin Awi)) = R
Σ
Cm idwm
m=1
1
Cm jdwn + Ome -imwm
2
m=1
where
Cm = Am iBm
vdk
Assume ((A constant A for A 0,....21 2M Then we have
... = dg. and denote the constant value by Co
Σ
Cmeesum
were C-m := C.. and w-m Wmm Since {
,...,, ,...,...
iws e ,..., ow } e
are all distinct,
by Vandermonde's 's identity this means Cm Cm 0 for 1,..., M, = Am Bm 0 for m 1,..., .......M M. Now Am = Bm = means
9m 1 Km and 4m Jkm
If Am * 0, then (km; Jkm) spans R.2, forcing 4m = 0. Thus for every block m, either qn
km O, which results in degenerate RoPE head, contradicting the assumption. Therefore, for i 2 dy + 1 the attention logits stj are not constant, and thus the attention weight az are not constant.
B.3 PROOF OF THEOREM 3.4
In this section, we prove Theorem 3.4. To do so, we first need to prove sequence of Propositions and Lemmas. First, restate the theorem here.
Theorem 3.4. Define the he prefix-spread of the hidden states at layer as
A (
11 where F
Σ
max
ISiSiST
jši
For NoPE transformers, there exists = > 0 and constants C₁, C2 and C3 such that if the initial (1)
embeddings A b < s, then for all layers < L:
Cie,
A S C2e,
|aaA//wa 00A /oWx S C3e,
with high probability over the initialization distribution. The constants only depend on the number of layers and heads, and not the sequence length.
19

---

Published as a conference paper at ICLR 2026
Since all weight matrices are drawn from Gaussian distribution with fixed variance, there exists a constant B, depending only on the architecture, such that with high probability the operator norms of W.. WK. Wv. and Wo, as well as the Lipschitz constants of the MLPs and normalization layers are all bounded by B. To see this use, e.g. Theorem 4.4.5 from Vershynin (2018) and the fact that for two layer MLP f. it's Lipschitz constnat is bounded by Lip(f) S W |l [W2] Lip(g). Let I. be the number of layers, and H be the number of attention heads per layer. For any vector sequence a, R2 we denote by a; 1 Eigs Oj the prefix sum of ag. For real sequences with two indices "i E IR we denote 02 = (a.1,-.. asa E TR. and a, = + o4g. Proposition B.3. Fix a row i in an atlention head at the 1-th layer.
Sij 5 BBVHA
max 55
Proof. Notice that
1
1
$1
8
4i Ki
+
Σ
9i Ter
9: (ke ks).
vd
dk
vdk
Psi
Therefore, by Cauchy-Swartz
1
|| ||k - Rill
Sij 82
Vdl.
By the linearity of WK we get Iky ---k ku]l [[x(h,; b.)ll < || |
- ا.ll < | 4 S
BA (1) As for l [Whhll |l. recall that h, are the output of a normalization layer, and therefore (at initialization) |[hal| = vd. Thus, || S Bvd. Putting it all together gives
Si
5
S B²
V
j - = BBVAA
d
dk
To finish the proof, take maximum over j S i.
To bound the effect on the attention probabilities, we need the following Lemma. Lemma B.4. For any b E R
||softmax(a + -ootmmax((()] S |16ll
Proof. A C2 convex function f R + IR satisfies [V]((x) - vf(i),i S ||x - yllo (I- smoothness) if d V2 S |dd| for all x, d E R" (see Theorem 2.1.6 in Nesterov (2013)). Take fz) = log Σm= e² ). f is C2, convex and ff(x)= softmax(r). Therefore, all we need to show that for all x, R
d Vsoftmax(s)d= dV ||d]2
and indeed,
d (softmax(z)d = df diag(softmax(z))d (ssftm/))))))))))((())))))))))))
< d diag(softmaxxt))d
< |d] [|softmax(I)]
= ||]]
as required.
Using Lemma B.4, we can bound the uniformity of ouj and the prefix spread the head outputs. Proposition B.5. Let u; = f1 E R. In any layer I
||a: 4ull1 - (A !)
(12)
and,
|| 1 (1))
(13)
20

---

Published as a conference paper at ICLR 2026
Proof. To get Equation 12, let be the constant vector (3 (......,;;)) 5.) IR' and let b 8, By Lemma B.4
||;; 4ell ||softmax(a b) softmax(a)ll S |16||
Now, notice that [16] maxjsi s Stj 3.}. therefore Proposition B.3 gives the desired inequality. For Equation 13 notice that,
(ay - (v, 2- U3 Σ
i5i
hence
|| 0/ll $ max ||: 35
0e ||a uill s BA B = =BVH(4
We now bound the next layer's spread in terms of the current one. Denote by A max; maxys' llzy - ælll the prefix spread of an attention head's output. First, we'll give bound for ! and then use this bound prove the entire propagation result. Before, we need short lemma.
Lemma B.6. For any sequence (xz) and j S i.
| - =z S max [r, 2zll max J,, Zz|l Ssj
Proof. - 2, = E
(a, (a,
Ty)) and triangle inequality.
Proposition B.7, For any layer 1 L L, (1)
A s 2
(A
Proof. Fix i and j S i. Write 2j (ūg 0.) + (zj 0.) -(z 0;), so
|| - 2 S 10 - ūz|l + || Ty | 0e
(a)
[(c)
By Lemma B.6,
(a) ===1: 1|0 0/|l
< max rS1
-,,-- 0.1 < [Wrll AB
By Proposition B.5
(b) S B'
As for (c), Notice that,
Σ
2 0.))
1 Σ ((z, (,) + (ū, -))),
therefore by the triangle inequality, Proposition B.5, and Lemma B.6,
(c)
Σ 1| 1. +
1 Σ
||, - d.ll < B3 (A))
max eek Uz ES:
rsi
= B A 2
max ||ui - Tell kSt
< (A 2 BA (1)
To finish the proof, take the maximum over and j i. Proposition B.8 (Full Transformer block recursion). There exist constants A Ag depending only on B, and H. such that
A ((+1) s A (2)
21

---

Published as a conference paper at ICLR 2026
Proof. From Proposition B.7, the single-head spread is bounded by linear term 2BA, plus quadratic term VB. Concatenation and Wo multiply by at most |WWll (up to fixed constant depending on number of heads). Adding the residual preserves a linear contribution in 4 The positionwise LayerNorm/MLPP being B-Lipschitz, scales the spread by at most B. Collecting the constants into A and, A2 gives the desired result.
We can now proof the full propagation result.
Theorem B.9. For any finite depth L, there exists 0 (depending B, L, and Hi) such that if A ( S e, then for all S I.,
4 CA (2) S Ce,
with C = C, L, H).
Proof. By Proposition B.8, Ah I A1 + ) Choose S (ii (A1/A2)3 so that
(1+1)
A (1) < A.. Then A (1+1) < 2A
(1) Induction yields A/ (2A1) S CA (1) for
s L with C' (2A,)--1
This conclude the first part of the proof, regarding uniformity propagation across depth. Note that the bounds in the proof do not depend on the number of tokens the input sequence.
A bound. Recall that,
CijCij
T
i=1
where Cij centered positional weights, i.e. Ejs Ctj-
= 0. For any such Cij have
T
T
T
T
A
OsjCis
= T
(ag - t)ss +
Cij
T
i=1
131
15
1
aay Elleyl
max 1sisiST
layl
|a
T
1=1
SCB O(s).
Q/K gradient bounds. Let gij 0A / 0A//0s.)
We have
gij (as -.])
where oe¹ = =Epsi Crppipp-
Lemma B.10. For every i, Ejg: 95z = O, and therefor for any vectors aj
Σgga, Egr(a, a.).
i5i
Proof. First notice that
Σ 9h Σ
Σg
auy(es - -it 7
Isnaa (ess - Epra. [cipl] = 0.
For the second part, observe that
Σ -a -0,) -,)= Σ
gija, a. Σ 9ij Σ
gijaj-
JSi
jái
151
22

---

Published as a conference paper at ICLR 2026
Now, from direct computation and an application of Lemma B.10, we have
a
1
1
giskk
h4
g (Kg k.)h
vdk
vdk
i=1
jSi
i=1
0A
1
T
ΣΣ
T
g - -h,)
aWK
dR
= vd i=1 55
Let's analyse the norm:
0A
(igg:(h -ha) TIa
S
T
ggy
|14: 11gg - h44
aw
K
V
Vdk i=
1=1
J5t
5
T
S BVHA
gij
t=1 Jكر
BVHA (1)
T
T
(ayy +)(() - GI) +
Σ Σ
Hay - +1)
T
1=1 iS
s BVHA (1)
B BVVHA ( C + C) =O(E),
where C = maxiss/isst
lag
ol S maxi SisiST csyl An analogous result holds for Wo.
0A @WQ
T
- k.)AT v d
|
R,|l Whell
Vd
BVHA (
Σ
194l
SSJSKT
BVHA (1)
T
(aig - (c1j-- +
Σ Hcy-
T
< BVHA (1)
(B' c++ = 0(s). OO(E).
This concludes the proof of Theorem 3.4.
C EXPERIMENTAL DETAILS
C.1 TRAINING
DroPE from RoPE transformer trained from scratch. For the first part of our experi- mental evaluation, we train a small RoPE transformer with almost half a billion parameters on Wibe (Penedo et al., 2024) for over 16B tokens with sequence length of 1024. We note this is well over 2 times the chinchilla optimal number of tokens from Hoffmann et al. (2022). We use QWEN2 (Yang al., 2024) tokenizer and follow the specifications (number of layers/hidden di- mensions) from the 0.5B model from the same family. We implemented all our baselines on top of this architecture, pretraining them for the same large number of tokens. We use the AdamW W opti- mizer Loshchilov & Hutter (2017) with a small warmup phase of 520 steps, a batch size of 1024, peak learning rate of 3.0 10-4, and cosine decay thereafter. For DroPE we followed a similar optimization setup, but only training for 2B total tokens using a shorter warmup of 70 steps and a slightly larger learning rate of 1.0 x 10-3 to compensate for the shorter training budget. We provide a full list of hyperparameters and training specifications for this setting in the left column of Table 5.
DroPE from a pretrained SMOLLM. For the second part of our experimental evaluation, we use SMOLLM (Allal et al., 2024) with around 362 million parameters already extensively pretrained on the SmolLM corpus (Ben Allal et al., 2024) for over 600B tokens with a sequence length of 2048 almost 100 times the chinchilla optimal number. This model used GPT2 (Radford et al., 2019)
23

---

Published as a conference paper at ICLR 2026
Pretraining and DroPE Hyperparameter
RoPE transformer
SMOLLM
Model architectures
Model parameters
494M
362M
Model parameters w/o embeddings
358M
315M
Hidden size
896
960
Hidden MLP size
4864
2560
Hidden activation
SiLU
SILU
Number of hidden layers
24
32
Number of attention heads
14
15
Number of key-value heads
Head dimension
64
64
Attention bias
false
false
Attention dropout
0.0
0.0
Initializer range
0.02
0.02
RoPE A
1, 1,000,000
10,000
Tied word embeddings
truc
truc
Output router logits
truc bfloat16 QWEN2
truc bfloat16
Computation dtype
Tokenizer
GPT2
Pretraining setup
Optimizer
Adam 3.0 x 10 0.1 (0.9, 0.95, 1x10=* Cosine decay 520
Adam x 10-3
Learning rate
Weight decay
0.1 (0.9, 0.95, 1 10 Cosine decay N/A
Adam parameters (81, (2
Learning rate scheduler
Warmup steps
Maximum sequence length
1024
2048
Global train batch size (sequences)
1024
512
Tokens per training step
1,048,576 16.8B
1,048,576
Total tokens
600B
Dataset
fineweb
smollm-corpus
DroPE setup
QK-norm
False Adam W 1.0 x 10-3
True Adam W 1.0 x 10-3
Optimizer
Learning rate
Weight decay
0.1 (0.9, 0.95, 10 Cosine decay 70
0.1 (0.9, 0.95, x 10 Cosine decay 490
Adam parameters (81, e)
Learning rate scheduler
Warmup steps
Maximum sequence length
1024
2048
Global train batch size (sequences)
1024
512
Tokens per training step
1,048,576
1,048,576
Total tokens Dataset
2.B
31.46B/66.9B/1155.88
fineweh
fineweb-edu
Table 5: Architectures, optimization, and other training setup hyperparameters for pretraining our RoPE transformer, SMOLLM, and our two new DroPE phases.
tokenizer and its architecture was designed be similar to models of the LLAMA2 family (Touvron et al., 2023). While not all training details have been disclosed, Allal et al. (2024) explicitly mentions using the AdamW W optimizer Loshchilov & Hutter (2017), batch size of 512, peak learning rate of 3.0 x 10-3 and a cosine decay thereafter. For DroPE we again tried to follow a similar optimization setup, across our different 30B/60B/1 /0BB training regimes, introducing a short warmup of 490 steps and a slightly lower learning rate of 1.0 X 10 we found their reported 3.0 10-3 led to instabilities from the small batch size. Given the more extended training period, we used a simple QKNomm (Henry al., 2020) after dropping the positional embeddings, which we found beneficial to mitigate sporadic instabilities from large gradients. We note that preliminary experiments showed that normalizing only the queries led even faster learning and also successfully stabilized long
24

---

Published as a conference paper at ICLR 2026
training. We believe further exploration of this new Q-norm method could be exciting direction for future work to train transformers without positional embeddings at even larger scales. We provide a full list of hyperparameters and training specifications for this setting in the right column of Table 5.
C.2 EVALUATION
Weedle-in-a-hasttee We evaluate long- mongect retrieval using the sanelle-i---haasttan (NIAH) setup, which places a short "needle" inside a long distractor "haystack." Following prior work (Kam- radt, 2023), haystack is random excerpt from Paul Graham's essays, and each needle is seven-digit "magic number" paired with short key/descriptor. We study three variants:
(Standard NIAH) We insert a single needle and prompt the model to retrieve it.
Multi-Query NIAH: We insert multiple (key, value) pairs and prompt the model return many values possible for given list of keys. For ample: The special magic numbers for whispering-worte workhorse and elite-butterfly mentiored in the provided text are:.
(Multi-Key NIAH) We insert multiple (key, value) pairs but query for single key, e.g., The special magic number for elite-butterfly mentioned in the provided text is:
(Multi-Value NIAH) We associate multiple values with one key and ask for all of them without pointing to specific positions, e.g., What are all the special magic numbers for cloistered-colonizer provided text?
mentioned the
Inserted needles and example targets are formatted in natural language, e.g., One of the special magic numbers for whispering-workhnee
is: 1019173 and One of the special magic numbers for elite-butterfly is: 4132801. For standard NIAH variant, we report the average success rate over all possible needle depths. For the multiple needles NIAH variants, we always insert four (key, value) needle pairs, placed at ran- dom sequence locations. Unless otherwise noted, we use greedy decoding (logit temperature 0) for reproducibility.
Long-context evaluations. We use standard implementations of PI, RoPE-NTK, and YaRN. For tasks that require fixed maximum context length (e.g., NIAH at 2x the training context), we set the extension factor manually. For settings that require reasoning across multiple context lengths and extended generations, we employ a dynamic scaling schedule that adjusts 7 as a function of the generation length as detailed in Peng et al. (2023).
For DroPE, we follow Wang et al. (2024) and apply softmax temperature scaling when evaluating on longer sequences. In practice, we tune single scalar logit scale (equivalently, the inverse temper- ature) on held-out set at the target length. Analogous to (Peng et al., 2023), we fit this coefficient by minimizing perplexity obtain the optimal scaling. For the DroPE model trained from scratch, the best-performing scale is
8 + 0.4)2
and for SMOLLM--RRPE the optimal scale is
+ 0. 103 In(s),
Where Ctest /Craiin the context extension factor. Unless otherwise specified, all other decod- ing settings are held fixed across lengths.
Language modeling benchmarks. We evaluate SMOLLM and SMOLLM-DROPE six stan- dard multiple-choice benchmarks using the LIGITEVAL harness (Habib et al., 2023); ARC-E/C: grade-school science QA split into Easy and Challenge sets, the latter defined by questions that defeat simple IR and co-occurrence baselines (Clark et al., 2018); HellaSwag: adversarially fil- tered commonsense sentence completion that is easy for humans but challenging for LMs (Zellers et al., 2019); OpenBookQA: combining a small "open book" of science facts with broad common- sense answer 6K questions (Mihaylov et al., 2018); PIQA: two-choice physical commonsense
25

---

Published as a conference paper at ICLR 2026
Table 6: DroPE matches base model in-context performance. Comparison of the pretrained SMOLLM model with SMOLLM-DROPE evaluated on variety of LM benchmarks.
Model
ARC-E
ARC-C
HellaSwag
OpenBookQA
PIQA
Winogrande
Avg.
SmolLM
65.6
36.0
53.8
37.2
72.0
53.7
53.1
SmolLM-Dras M-DroPE
67.3
37.6
53.9
38.0
71.5
52.3
53.4
Table Validation perplexity for 500M parammeteet transformer trained on 16B tokens, when drop- ping positional encodings different stages of pretraining.
DroPE @ OK (NoPE)
DroPE @ 8K
DroPE @ 14K
DroPE @ 16K (RoPE)
Validation perplexity
23.77
22.42
21.73
21.72
reasoning (Bisk et al., 2020); and WinoGrande: large-scale, adversarial Winograd-style corefer- ence/commonsense benchmark (Sakaguchi et al., 2021). We follow the harness defaults for prompt formatting, decoding, and scoring, and do not perform any task-specific fine-tuning or data adapta- tion.
D ADDITIONAL EXPERIMENTAL RESULTS
When should start recalibration? In this setup, train 500M-parameter transformer 16B tokens and remove its PEs during training. We vary the training step at which recalibration is activated. We consider four recipes:
Dropping PEs from step 0 (NoPE transformer),
Dropping PEs at step 8K,
Dropping PEs at step 14K,
Dropping PEs at step 16K (RoPE transformer, i.e., no dropping during training).
Table 7 reports the final validation perplexity for each setting.
We find that this ablation further strengthens theoretical observation that DroPE should be inte- grated later in training. Our analysis Section 3 suggests that NoPE transformers struggle to train efficiently, whereas retaining RoPE for most of training benefits optimization. Consistent with this, we observe that dropping the positional encoding only at the very end of pretraining (DroPE @ 16K) yields the best validation perplexity, while earlier dropping steadily degrades performance.
Finally, we emphasize that in this setup DroPE does not incur additional training cost: the total number of optimization steps unchanged, and once the positional encoding removed, training becomes slightly faster due to skipping the RoPE rotation operations in attention.
Average LongBench scores and tasks breakdowns.
Table 8: Average performance over all LongBench tasks for different RoPE scaling methods.
Method
Avg. LongBench score
SMOLLM SMOLLM + PT
2.59
2.48
SMOLLM RoPE-NTK
12.21
SMOLLM + YaRN
13.07
SMOLLM-DROPE
13.81
Effect of QKNorm and loss spike. Here we clarify the role of QKNorm Henry et al. (2020) in our architecture and verify that it is not confounding factor for the effectiveness of DroPE. QKNomm was introduced strictly as an optimization-stabilis mechanism to enable training with higher learn- ing rates, following recent practices in large-scale model training such OLMo2 (OLMo et al.,
26

---

Published as a conference paper at ICLR 2026
Table MultiFieldQA performance across context length buckets for SMOLLM variants.
Model
0-4K (0-2x ctx)
4-8K (2-4x ctx)
8-16K (4-8x ctx)
SmolLM-Droo
32.82
24.73
30.07
SmoILM-NTK
34.25 SmolLM- YARN
22.30
21.63
33,96
22.91
20.08
Table 10: MuSiQue performance across context length buckets for SMOLLM variants.
Model
0-4K (0-2x ctx)
4-8k (2-4x ctx)
8-16K (4-8x ctx)
16-32K (8-16x ctx)
SmolL MrDrroRR
50.00
6.11
8.05
16.67
SmoILM-NTK
0.00
4.36
3.36
0.00
SmolLM- YaRN
0.00
19.68
3.13
7.14
2024) and Qwen3 (Yang et al., 2025a), where normalization is used primarily to stabilize gradients and mitigate loss spikes rather than increase modeling capacity.
To assess the interaction between QK Norm and DroPE, we conducted controlled ablation study on the SmoLLM-360M model using six configurations: three learning rates (3 10 5, 3.33 10 10-3), each trained with and without QK Norm.
The results, summarized in Table 11, yield two main observations:
Lower learning rates (3 10-5, 5, 10="). DroPE works effectively without QKNorm.
At the lowest learning rate (3 x 10 10-5), the model without QK Norm achieves a slightly better final loss (2.713 3. 102). Together with the 3 10-4 setting (2.530 vs.2.555), this indicates that QK Norm does not consistently improve performance in low-volatility regimes and
the source of our gains.
High learning rate (10-3). At the highest learning rate, the model without QKNomm becomes unstable (loss spikes, gradient explosions), leading to poor convergence (final loss 6.334). In contrast, adding QKNorm stabilizes training and allows higher learning rate to achieve the best overall performance (final loss 2.496).
leverage the
Figure 12 shows the corresponding training curves with and without QK Norm, highlighting the presence of loss spikes higher learning rates, in line with observations reported in ?. These results empirically demonstrate that the primary role of QK Norm is to act as a stabilizer that enables the use of more aggressive, compute-efficien learning rate. Importantly, DroPE can still be applied without QK Norm by using a moderate learning rate (e.g., (3 x 10 10=4), which is our default setting for all experiments except the longer SmoILMM3660M recalibration phases.
Table 11: Ablation study Mmo6L.M3360M recalibration with and without QK Norm across differ- ent learning rates.
Learning Rate
With QK Norm
Without QK Norm
Status
10-3 (High) 3 x 10-4 (Mid)
2.496
6.334
Unstable without Norm
2.555
2.530
Stable Comparable
3 10 10-5 (Low)
3.102
2.713
Stable Comparable
E LARGE LANGUAGE MODEL (LLM) USAGE
We used large language models to assist with refining the wording in some sections of this paper. Their role was limited to improving clarity of technical explanations, grammar, style, and overall readability. All research ideas, methodology, experiments, analysis, and conclusions are entirely our own; the models were employed solely writing aids.
27

---

Published as a conference paper at ICLR 2026
train/loss
208 tokens, Ir-3--5, weo Norm
208 tokens, Ir-3e4, Norm
20B tokens, Ir-3e-5, QK Norm
208 tokens, Ir-1e-3, QK
tokens, (r-3e4,
208 tokens, I=-e-33, my
7
4
10k
Figure 12: QKNorm allows for recalibration at higher learning rate.
28


Reviews:
[{"review_id": "8CaWqM3nS1", "reviewer": "Reviewer_Zi8P", "summary": "This paper introduces DroPE, a method that removes RoPE after pretraining to address its limitations. The authors demonstrate that while RoPE provides crucial inductive bias for rapid convergence during training, its explicit positional encoding hinders zero-shot generalization to longer sequences. They analyze why existing RoPE-scaling techniques fail in zero-shot scenarios by showing how they distort low-frequency attention heads essential for long-range semantic understanding. The core innovation is DroPE, which eliminates RoPE and employs a brief recalibration phase, compelling the model to rely on implicit positional cues from the causal mask and data patterns. In experiments, this approach achieves robust zero-shot extrapolation when extending the context length to twice the training length, significantly outperforming complex RoPE-scaling methods.", "questions": "1. Can NoPE extrapolation methods such as [2] be integrated with DropRoPE?\n2. A 2× extrapolation advantage is likely of limited practical significance. Can you evaluate generalization over a wider range of lengths (for example, from 2× up to 8×, as in [2])?\n3. Why were only four LongBench subsets selected rather than the full LongBench suite?\n4. Can the method be validated on widely used LLMs (e.g., LLaMA, Qwen)?\n\n[2] Length Generalization of Causal Transformers without Position Encoding (https://arxiv.org/pdf/2404.12224)", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. The paper shows that RoPE trained from scratch outperforms NoPE: NoPE has higher perplexity and exhibits gradient-vanishing issues during training.\n2. The authors propose a new way to obtain a NoPE base model by “dropping” RoPE from a well-trained RoPE model (Drop RoPE).\n3. They evaluate on selected datasets (NIAH and four LongBench subsets) and empirically demonstrate that RoPE→NoPE bases achieve length-generalization benefits for limited extrapolation ranges (mainly around 2×).", "weaknesses": "1. To me, the manuscript reads more like a blog post than a formal paper. The central claim—that DropRoPE provides a generalization advantage—feels weak across several experimental dimensions: the variety of base models tested, the extrapolation distances considered, the number and comprehensiveness of benchmarks, and the choice of baselines.\n2. The observation that performance can be restored by continued training after dropping positional encodings has been reported elsewhere. For example, Table 5 in [1] shows smollm-135M can recover performance via continued training even when positional encodings are fully or partially removed.\n\n[1] Towards Economical Inference: Enabling DeepSeek’s Multi-Head Latent Attention in Any Transformer-based LLMs (https://arxiv.org/pdf/2502.14837v1)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bc76Y6K1W6", "reviewer": "Reviewer_iW8i", "summary": "This paper proposes a novel method to extend the context length of pretrained language models by removing RoPE after pretraining and conducting a short recalibration phase. The authors argue that positional embeddings are crucial for training convergence but harms zero-shot generalization to longer contexts.", "questions": "1. Why was YaRN not able to extrapolate to 2x context length? This might be a mistake since Qwen3 uses YaRN without finetuning and has a perfect score in the needle-in-a-haystack benchmark.\n2. Do you know if DroPE is better than YaRN because it has more training time during the recalibration phase? Can you compare it against YaRN but with the same recalibration phase?\n3. Do you think that gradually removing the RoPE during the recalibration phase be better than outright removing it in a single step? It's highly likely that DroPE is equivalent to setting a RoPE Linear scaling to infinity.\n4. What happens with DroPE's performance at higher context scaling lengths? 4x, 8x, etc?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Novel and counterintuitive hypothesis where positional embeddings might not be necesssary throughout a model's lifecycle.\n2. Treating PEs (Positional Embeddings) as \"training scaffolds\" that can be removed is elegant and very good for downstream use cases such as long context fine-tuning and kv-cached inference performance.\n3. Good theoretical contributions on why PEs are necessary during training and why they can be removed after.\n4. Strong potential for practical impact because of how simple this method is. Furthermore, reduces the inductive biases of LLMs after removing RoPE.", "weaknesses": "1. Experiments on small models might not scale well to larger models. Could be possible to take existing large LMs and recalibrate them with DroPE.\n2. No ablations or thorough experimentation on the recalibration phase. Naively removing PEs might work but given the theory there should be better ways. Furthermore, recalibration cost is not explored either.\n3. Dubious claim that YaRN cannot extrapolate to longer contexes, directly contradicting the original paper and results from the industry (DeepSeek R1, Qwen3, GPT-OSS).\n4. Lack of comparisons against more recent length generalization work (LongRope 2, sparse attention, etc.)\n5. Only test at 2k context lengths, most methods now test at 128k+.\n6. Specific solution to autoregressive LLMs, does not generalize to diffusion models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lxEQvX9gHN", "reviewer": "Reviewer_PvVC", "summary": "This paper introduces DroPE, a simple and effective method for extending the context length of pretrained language models without expensive long-context finetuning. The core idea is to remove the positional embeddings (PEs) from a fully trained model and then perform a short \"recalibration\" training phase. The authors argue that PEs are a crucial scaffold for efficient pretraining but inherently limit generalization to longer sequences. By dropping them post-training, DroPE achieves significant zero-shot context extension. Empirical results on a 0.5B parameter model and the SmolLM model show that DroPE outperforms established RoPE scaling methods on long-context tasks while preserving performance within the original context window.", "questions": "1. How do you expect DroPE to perform when scaling to much larger models (e.g., 7B+) and extending the context by a much larger factor (e.g., from 4k to 128k)? Do you anticipate any new optimization challenges?\n\n2. Have you evaluated DroPE on tasks that are highly sensitive to precise token positions (e.g., Passage reranking in HELMET)? Is there a trade-off where long-context generalization comes at the cost of fine-grained positional awareness?\n\n[1] HELMET: How to Evaluate Long-context Language Models Effectively and Thoroughly\n\n3. What is the intuition for what the model learns during recalibration? Is it primarily learning to infer relative positions from the causal mask, and have you observed corresponding changes in attention patterns?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The central idea of dropping positional embeddings to achieve context length extrapolation is interesting. It reframes PEs as a temporary training aid rather than a permanent architectural component.\n\n2. The proposed DroPE method is simple to implement and demonstrates strong empirical performance, simply but effectively alleviating the OOD issue of RoPE.", "weaknesses": "1. The paper's central claims are not validated at a scale that reflects the current state of long-context LLMs. The experiments are limited to relatively small models (under 0.5B parameters) and a modest 2x context extension (e.g., 2048 to 4096 tokens). This is a significant limitation, as the most pressing need for context extension exists in much larger models (7B+) and for vastly longer sequences (e.g., 32k, 128k, and beyond). It is unclear if the method's effectiveness and training stability would hold when extending context by 10x or 100x, where challenges like attention dilution and loss of positional signal become far more severe. The strong claims of the paper require evidence at a more demanding scale to be fully convincing.\n\n2. While the paper provides extensive analysis, a notable portion of it lacks novelty and feels disconnected from the proposed method. The analysis in Section 4, which details the failure of RoPE-scaling methods due to the compression of low frequencies, largely reiterates well-established findings from the original papers on YaRN, NTK-RoPE, and LongRoPE. Furthermore, these theoretical insights do not directly inform the specific design of the DroPE recalibration process, such as the required duration or optimal hyperparameters. The theory explains why a problem exists but offers little guidance on how to best implement the proposed solution.\n\n3. The theoretical justifications in Section 3, intended to prove the necessity of PEs during training, rely on arguments that feel trivial. For example, Proposition 3.2 proves that a NoPE transformer's gradients vanish on an artificial sequence of identical tokens. This is an unrealistic edge case that has little bearing on training with diverse, real-world data. Spending significant space on such formalisms seems to over-justify a widely accepted premise (that PEs help training) and detracts from what could have been a more focused empirical investigation into the DroPE method itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lz47auZM7a", "reviewer": "Reviewer_fTZx", "summary": "The paper challenges the conventional methodology by bravely removing positional encoding at serving time (NoPE), but the model had been initially trained using RoPE and recalibrated with continuous pretraining in a later phase without positional encoding. \n\nFor the background, it is well-known that NoPE\n1. can learn positions of tokens due to causal mechanisms in attention;\n2. but underperforms in terms of negative log-likelihood when comparing with most existing positional encodings.\nAlthough NoPE is known to have good generalization properties in terms of context length, it is generally dismissed as the widespreading intuition is that the positional information is \"hard to learn\" or even \"lost\". \n\nThe paper provides an explanation of why NoPE has difficulties in learning positional information. On the other hand, it also provides some observations to explain why RoPE, though learning the positional information fast, has trouble with length generalizations.", "questions": "It's possible that I miss certain things already in the appendix, but below are the questions I want to understand better.\n\n- I don't understand Figure 9 and Table 4: are you comparing the original SmolLM with SmolLM + extra 30/60/120B token's finetuning? Or did you match an equi-token setup and further finetune SmolLM for the same amount of tokens? If the former, it is perhaps not entirely fair and it still bears the question as to how much the original model would trend higher given the additional training. Also, do you have certain measurement of confidence interval on those eval accuracies? \n- In Table 2, what are the exact setup for SmolLM+PI/NTK/YaRN, and which token budget for the DroPE did you use?\n- In Appendix C.1, you mentioned a \"simple QKNorm\". Does that mean you modified the architecture slightly after dropping the positional encoding? What happened without? Are we able to see a detailed analysis on ablating all these variables?\n\nI will read more later and write more questions if something arises.\n\nI am tentatively giving it a 6 as I think it is very intriguing work and the authors have very good math foundations and taste. But experimental rigorousness is crucial. **I may revise the score higher or even lower** as I understand more about the details of the work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. Simplicity.\n2. The idea is fairly novel.\n3. Results are fairly compelling with good theoretic analysis to back them up.\n4. Potential impact. I personally think this inspires the search of better positional encoding by not only looking at architecture manipulations, but also the training process.", "weaknesses": "- One of the biggest concern to widely adopt this in LLM training is a rigorous study to confirm potential risks. There are two aspects that I'm particular concerned, despite the good numbers on academic benchmarks\n  1. The loss spike and cooling down, does it hurt further learning dynamics? What does the spike do to other layers' parameters, parameter/grad norms, etc. Would it make further SFT different?\n  2. This is less important as to a \"weakness\", but how does it fare on other more up-to-date LLM evaluations? This might be relevant if we want to understand if it is genuinely without regression on all tasks, or whether there are some issues that we need to understand.\n- There seems to be a lack of experimental rigorousness. For this I still very much want to understand all these comparison setups, what are held as control variables and how the confounders are ruled out. Also there seems to be a sneaky modification of architecture in Appendix C.1, even though they \"look minor\", introducing unwanted variable makes experiments shaky. See my questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)