Frayed RoPE and Long Inputs: A Geometric Perspective
Authors: Davis Wertheimer
Aozhong Zhang
Derrick Liu
Penghang Yin
Naigang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: RoPE
context length extension
sink tokens
clustering
attention
long context
transformer
language model
Primary area: generative models
TL;DR: We advance a geometric understanding of latent attention dynamics, use it to explain RoPE's failure to generalize to long contexts, and mitigate that failure with a straightforward architectural adjustment

Abstract: Rotary Positional Embedding (RoPE) is a widely adopted technique for encoding position in language models, which, while effective, causes performance breakdown when input length exceeds training length. Prior analyses assert (rightly) that long inputs cause channels to rotate “out of distribution,” but it is not clear how extra rotation relates to or causes pathological behavior. Through empirical and
theoretical analysis we advance a unified geometric understanding of attention behavior with RoPE. We find that attention induces tight clustering of separated key and query latent point clouds, allowing for creation of sink tokens: placeholders that allow attention heads to avoid token mixing when not required. RoPE applied to longer inputs damages this key/query cluster separation, producing pathological
behavior by inhibiting sink token functionality. From this geometric perspective, we propose RoPE-ID (In Distribution), a straightforward modification that allows attention layers to generalize to longer inputs out of the box: apply RoPE with high frequency to a subset of channels. We demonstrate the effectiveness of RoPE-ID for extended inputs using 1B and 3B parameter Transformers on the LongBench
and RULER information retrieval benchmarks.

Published as a conference paper at ICLR 2026
FRAYED ROPE AND LONG INPUTS: A GEOMETRIC PERSPECTIVE
Davis Wertheimer, Aozhong Zhang*, Derrick Liu, Penghang Yin*, Naigang Wang IBM Research; *University at Albany, SUNY {davis wertheimer, dliuje @imm. com, nwangeus. ibm. com *aazhang3, ea pyinnealbany.....mmmmm.... banybedu edu
ABSTRACT
Rotary Positional Embedding (RoPE) a widely adopted technique for encoding position in language models, which, while effective, causes performance break- down when input length exceeds training length. Prior analyses assert (rightly) that long inputs cause channels to rotate "out of distribution." but is not clear how extra rotation relates to or causes pathological behavior. Through empirical and theoretical analysis we advance unified geometric understanding of attention be- havior with RoPE. We find that attention induces tight clustering of separated key and query latent point clouds, allowing for creation of sink tokens: placeholders that allow attention heads to avoid token mixing when not required. RoPE applied to longer inputs damages this key/query cluster separation, producing pathological behavior by inhibiting sink token functionality. From this geometric perspective, we propose RoPE-ID (In Distribution), straightforward modification that allows attention layers to generalize to longer inputs out of the box: apply RoPE with high frequency to subset of channels. We demonstrate the effectiveness of RoPF-ID for extended inputs using IB and 3B parameter Transformers on the LongBench and RULER information retrieval benchmarks.
INTRODUCTION
Transformer models form the backbone of modern large language models (LLMs), enabling them to capture complex dependencies across long sequences. The attention mechanism in transformers maps inputs into queries, keys, and values: queries and keys determine token relevance through sim- ilarity scores, while values provide the content to be aggregated. This separation allows the model to learn both where to attend and what information to extract, producing context-aware representations that drive the success of transformers in natural language processing and beyond.
To enhance the interaction between queries and keys, positional encoding is used to distinguish token order, constituting a fundamental component of transformer design. Rotary Positional Embedding (RoPE) (Su et al. 2023) has emerged the predominant approach and is now implemented in most sfate-of-thc-aa LLMs, including LLaMA (Grattafiori et al. 2024), GPT (OpenAI et al. 2025). and DeepSeck (DeepSeek-Al
al. 2025). However, key limitation of RoPE is performance degradation when imput length exceeds training context, Most attempts to analyze and address this issue attribute the failure to channels rotating "out of distribution." leading to frequency rescaling as
a workaround (Chen et al. 2023 bloc97 2023b Peng et al. 2023 Ding et al. 2024).
Another important phenomenon in transformers is the attention sink, which has been shown in- fluence long-context generalization (Xiao et al. (2023). The attention sink, typically the first input token, possesses little semantic meaning but consistently large attention scores. Its presence is con- sidered crucial to prevent over-mixing of information, and empirical evidence shows that attention sinks must preserved when extending context lengths (Xiao et al. 2023 Han et al. 2024).
The relationship between attention, RoPE, and attention sinks three seemingly disconnected con- cepts - is the focus of this paper. We propose a unified geometric perspective, based on analysis of popular LLM families including LLaMA, Gemma (Team et al. 2024). and Olmo (Groeneveld al., 2024). We find that, contrary to the common intuition of attention as a soft nearest-neighbor lookup, queries and keys form tight clusters with minimal overlap, while the sink token resides the

---

Published as a conference paper at ICLR 2026
origin (Fig. 1 left). Within the training context length, this separation ensures that the sink token, with its small norm, naturally absorbs the majority of attention weight. Beyond the training context, RoPE disperses and overlaps the query/key clusters. This geometric disruption prevents the sink token from functioning, leading to long-context performance breakdowns (Fig. right).
Based this analysis, we identify two nec- essary criteria for length generalization with RoPE: 1) lower bound cluster overlap, and 2) attaining this bound within the training length. The former ensures continued sink to- ken functionality in the limit, while the latter prevents out-o--distribution errors. We propose RoPE-ID (RoPE In Distribution) as simple yet effective example for how fulfill these crite- ria. A plug-in replacement for RoPE, RoPE- ID applies high-frequency rotation to a fraction of channels in each head, leaving the rest un- changed. RoPE-free channels maintain sepa- rated key/query clusters across length (criteria 1), while high-frequency rotation ensures that in subspaces where clusters merge, they do so within the training length (criteria 2), facilitat- ing length generalization. We validate RoPE- ID on LongBench and RULER using trained IB- and 1B-parameter decoders, demonstrating strong context length generalization and improvements over prior tuning-free methods.
Keys
Kays
Siik
sint
Queries
Queries
Figure 1: A 2D diagram of our observed latent geometry. Left: Keys/queries cluster tightly into opposing point clouds with negative dot products. The sink token has low norm and thus the great- est dot product. Matched key/query pair k align orthogonally, letting their product approach and exceed the sink's. Right: RoPE on long inputs makes keys/queries disperse and overlap, causing spurious alignment. Sink token stops functioning.
2 BACKGROUND AND RELATED WORK
Position Encoding techniques can be broadly divided into two categories: absolute position embed- dings (APE) and relative position embeddings (RPE). APE directly injects position information into latent representations using token indices, in fixed sinusoidal (Vaswani al. (023) learnable form (Devlin et al. 2019). APE exhibits limited generalization beyond the training context length, which RPE addresses by encoding distances between token pairs rather than their absolute indices.
Notable RPEs include T5' s relative bias (Raffel et al. 2023) and Alibi (Press et al. 2022), which add distance-based biases attention logits, and RoPE (Rotary Position Embedding) (Su et al. 2023), which encodes relative distance as latent angular displacement, now the de facto standard for L.MMs.
The key insight behind RoPE that relative position, through properties of rotation, decomposes into independent key and query transformations. RoPE encodes relative position via angular dis- placement proportional to token distance, interposing a block block-diagonal matrix of 2 2 rotations into the key/query dot product. Each submatrix has a constant frequency 0 scaling token distance m:
cos m0; - sin m0, sinm00 cos m0,
[oos m0 sin m0
R@ Diag
sin miim200/2 cos m00s/2
In practice, this decomposes into independent rotations on query and key k, at positions i,j:
RoPB((u,,;) qi R2 kjj = Ro'k q. Rig (ke, Rb) = (r(a, 1), rk,,
where r(. i) represents rotation by R Understanding the impact of this progressive rotation on latent keys and queries is crucial to understanding out-of-distribution failures on long contexts.
Context Length Extension: As L.MMs are increasingly applied to long-context tasks, substantial re- search has focused on extending their usable sequence lengths without retraining. While RPEs such as RoPE are designed to improve long-context generalization, performance still degrades beyond the training length. To address this, Position Interpolation (PI) (Chen et al.) 2023) linearly inter- polates position indices within the pre-trained sequence length. NTK-by-pare (bloc97 2023b) and NTK-aware (bloc97 (023aa) introduce nonlinear interpolation schemes inspired by Neural Tangent Kernel dynamics, scaling RoPE frequencies in groups and outperforming simple PI. YaRN (Yet An- other RoPE Extension) (Peng et al. 2023) further integrates previous NTK-based approaches with
2

---

Published as a conference paper at ICLR 2026
temperature scaling attention logits. More recently, LongRoPE (Ding et al. 2024) employs lutionary search to optimize the frequency rescale factors for each dimension. LongRoPE extends context window to beyond 2 million tokens, albeit with multi-step long-context fine-tuning. Here we focus on tuning-free generalization and leave tuning to future work.
Some studies find that RoPE's low-frequency components induce high-norm semantic bands, which become unstable in long contexts (Barbero et al.] 2024b) or hinder the encoding of semantic in- formation (Chen al. 2024). They propose limiting RoPE subset of channels, finding this improves performance. Our analysis provides novel perspective and caveats for this technique.
Sink Tokens, or attention sinks, refer tokens with disproportionately high attention despite lack of semantic meaning (Xiao et al.) 2023). This phenomenon is widely observed in LLMs and plays
critical role in preserving model behavior, especially long contexts (Xiao et [023 Han al.)
2024 Yang et al. 2024). Usually the first token of a sequence (Xiao et al.] [023]] Cancedda 2024),
attention sinks have also been linked to massive activations observed across LLMS (Sun et al. 2024).
Gu et al. (2025) investigate when and how attention sinks emerge during pretraining, and Barbero jet al. (2025) provide theoretical and empirical evidence that sink tokens prevent representation col- lapse from information over-mixing (Barbero et al. 2022a)). Here we relate sink tokens to RoPE and attention geometry, pinpointing the sink token as the failure mechanism in long contexts.
3 ANALYSIS
We perform a geometric analysis of attention with RoPF, showing that keys and queries cluster tightly in opposing directions, while RoPE inhibits this behavior, with clusters dispersing and over- lapping over time. Alongside small sink token E2 norm, these separated clusters produce a learned bias toward the sink. However, as RoPE disperses and overlaps key and query clusters, this mecha- nism becomes tenuous. We claim that the breakdown of transformers in long contexts is a breakdown of the sink token: past the training length, models begin attending to the wrong token(s) by default.
Analysis is conducted on Mlama--8-IInstuue with additional trials on Olmo-7b and Gemma-7b for verification. When not otherwise specified, the relevant model is Llama3. Input text drawn from the Wikitex12 dataset (Merity et al. 2016). Further details are provided in A.5
3.1 KRY/QUERY CLUSTERING
An intuitive understanding of the attention operation is that it functions as a soft nearest-neighbor lookup. query oriented in latent space to align with one or more contextually relevant key vectors, and the degree of alignment defines the mixing ratios for corresponding values. The curse of dimensionality ensures that random IID latent points are orthogonal by default, so directional alignment in high-dimensional space is difficult. Thus we can imagine that keys and queries form overlapping point clouds around the origin. Key/query matching is accomplished by high directional alignment: activated pairs should have large, positive dot products. Keys and non- non-matching queries, meanwhile, should be orthogonal, with small dot products, to keep retrieval discriminative.
This model of attention, while intuitive, is also wrong, at least for RoPE models. Instead of overlap- ping clouds on the origin, keys and queries form tight clusters away from the origin, with minimal overlap. Further, such clusters are generally unaligned directionally, with the origin sitting between the clusters. Fig. [2] shows the mean intra- and inter-cluster pairwise cosine distances (f2-normalized dot products) for keys and queries, averaged over layers and heads. Before RoPE, intra-cluster dis- tances (key-key and query-query), bounded to 11, are generally close to 1. Key and query point clouds are situated within a tight are, in small clusters displaced from the origin. At the same time, these clusters are largely aligned against each other, with negative mean key-query dot product in Fig. 2 (right). This paints an entirely different picture of attention: instead of overlapped point clouds, envision keys and queries in opposing quadrants. Queries avoid attending to most keys via negative dot products. If a query and key land on aligned quadrant boundaries, though, the resulting zero dot product exceeds the baseline and yields a large attention weight. Softmax shift-invariance ensures that this arrangement (negative baseline and orthogonal "aligned" pairs) produces identical mixing behavior to the original conception (orthogonal baseline and positive products for aligned pairs). Fig. (left) illustrates the proposed geometry. In practice, key/query clusters are not directly across the origin (dot products in Fig. 2 (right) are negative but small), but the intuition holds.
3

---

Published as a conference paper at ICLR 2026
Liama wro RoPE
Lama3 RoPE
Gemma w/ RoPE
Gemma w/ RoPE
Olmo wio RoPE
Oino RoPE
1.0
0.1
Cosine DiCaNToR
0.6
0.6
-3.1
-0.2
0.2
-0.3-
0.0
0.0
3k
6k
16k
4k
32k
64k
1k
0k:
16k.
Que Query-Query Cosine Distance
Key-Key Cosine Distance
Query<Key Cosine Distonce
Figure 2: Effect of RoPE across context length on pairwise angular distances within heads for
Llama3-8B, Gemma-7B and OLMo-7B.
3.1.1 THE IMPACT OF RoPE CLUSTERING
RoPE sends points through a predetermined trajectory of orbits around the origin, so any tightly- clustered point cloud, displaced from the origin, should inevitably disperse. This is indeed the case for transformers: Fig. shows that RoPE decreases intra-cluster aligmment, with further decrease time points spiral further away. The model compensates for this by positioning key/query clusters such that RoPE misaligns them further: fhemfurther:
key-query dot product also decreases, and does not rise until after the training length (2k for Olmo, 8k Llama/Gemma), Meanwhile, the clusters without RoPE mostly maintain their behavior across context lengths. It thus appears that RoPE weakens the clustering, but does not eliminate it until the training context length is exceeded. We corroborate this finding via additional distance/clustering metrics in A.3
Fig. 3 illustrates this behavior, by taking a PCA "snapshot" of the point clouds without RoPE at time 4096, and applying the same projection with RoPE and at time / 65536 (more plots are available iT]. Prior observations are confirmed visually: the first and third views, points form tight clusters displaced from the origin, and key and query clusters are located across from each other (with four queries per key cloud due to GQA (Ainslie et al. (023)). RoPE causes clusters within training length to disperse slightly, but at length 64k eliminates cluster separation entirely (in this projection). The exact impact this overlap in key/query clouds discussed in $3.2 but is clear that query-key "alignment" works differently length 64k compared to the other scenarios.
0.10 IN
0.10
0.10
01
0,30.20
-15
0 0.30M
0.3
-30
-20
-10
-20
-10
10
20
-00
-20
-20
-20
-20
RoFE
G4K
RpPE
RoPE
Figure 3: 2D PCA projections of Llama3 representations under different context lengths and RoPE settings (3rd key head layer 21 and its queries). RoPE at long contexts destroys cluster separation, and increases stable rank (in parentheses) with sequence length, consistent with Theorem
3.1.2 A SINGULAR VALUE PERSPECTIVE ON CLUSTERING
While the visual analysis striking, it only captures 2D projection of a 128-dimensional latent space. Similarly, Fig. reports pairwise relationships, an incomplete picture global behavior. We therefore corroborate our findings with a holistic and rigorous analysis based on singular values. In an attention head, the set of key or query vectors forms matrix X E Raad where is the inference sequence length and d is head width. X's singular values correspond to the principal components of the point cloud, an ordered set of directions maximizing variance along the earliest directions up to orthogonality constraints. When singular values are equal, variance is constant in all directions, and the point cloud forms a ball around the origin. Unequal values indicate uneven spread. In practice, the first singular value (FSV) of key and query clusters before RoPE is large, accounting for over 75% of total cluster variance on average for Llama3. Fig. 4 (left) plots the distribution across individual heads and layers, and the degree of variance covered by the first principal component
4

---

Published as a conference paper at ICLR 2026
ranges from about half to nearly all of it. Thus the majority of spread around the origin occurs in one direction: either the cluster a long thin needle, or it is tight cloud displaced along this direction. Given the intra-cluster dot products in Fig. 2 the latter must be the case.
1.00
Stable Pank
1S Retio
0.75
0.25
0.6 0.7
0.4
512
DC
32K
64K
1KK
B4C
Percent Varlance
Input
Length
Figure 4: Left: Histogram across layers and heads showing the percentage of variance (relative to origin) explained by the first principal component of latent key/query clusters. Middle: Ratio of first singular value (FSV) after and before RoPE. Blue lines plot individual key/query heads, red plots the average trend. RoPE shrinks the FSV, but accelerates beyond the training length, producing cluster dispersal. Right: Mean stable rank of key clusters by input length, showing monotonic increase with RoPE, indicating cluster dispersal. Error bars denote standard deviation.
When RoPE is applied, we expect principal components to skew more evenly: RoPE throws channel pairs through decorrelated rotations (ensured by irrational frequency ratios), so in the limit, point cloud under RoPE maps to a shell IID points orbiting the origin. We can formalize and justify this intuition: FSV (i.c., the spectral norm) ratio under RoPE, [|(x)] shrinks as the sequence length n increases, where R denotes the RoPE action. The following informal result provides an asymptotic estimate of this ratio; the rigorous statement and proof are deferred to A.
X
Lemma 1. Assume the keyqueery matrix X = uv where E RR. v E RR with || = 1. Then as the inference sequence length n - oc, under mild conditions on u, applying RoPE to X shrinks its spectral norm (or FSV) by at least a factor of V2, and by up to a factor of Vd.
Fig. (left) shows that keylquery matrices are close to rank-1 in practice, which motivates our rank- assumption on X in Lemma Under this assumption, the jth key/query projection takes the form uyv E Rdd We expect similar result to hold when this assumption is relaxed to the approximately rank-1 1 case, which we leave for future work. The exact reduction factor depends on the energy distribution across the channels of the underlying learned representation as detailed in A.
Fig. 4 (middle) plots the ratio of FSV under RoPE in practice. In all cases, RoPE decreases the FSV as expected, but the decrease is limited within the training context, falling smoothly up to length 4k. The decrease more aggressive for inputs above the 8k training length. This confirms that cluster means drift toward the origin as RoPE is applied over longer sequences.
At the same time, RoPE preserves the sum of X's squared singular values, or equivalently, preserv- ing the Frobenius norm, |X|] F (Golub & Van Loan 2013):
Lemma 2. Applying RoPE preserves the Frobenius norm of any X. i.e. |[(X)]] = ||||
Therefore, as the FSV falls due to RoPE, other singular values (representing other principal compo- nents) must grow to compensate. This suggests that RoPE induces expansion and dispersion of clusters while pulling them toward the origin. We formalize this intuition and characterize cluster dispersion using the notion of stable rank (Rudelson Vershynin) 2007| Sanyal et al.) 2019). continuous analogue of standard matrix rank. For a nonzero matrix X E Raxe with 2 d, its stable rank is defined as
X
srank(X) (X)
XI
of
where 01 ... Oa are X's singular values, with the following properties: (i) 1 S srank(X) < d; (ii) srank(X) 1 + rank ()) = 1; (iii) srank(X) = d o, 's are all equal.
rank X)
JA
Theorem J which follows directly from Lemmas T andd 2 shows that applying RoPE increases the stable rank when the input length n. becomes large, demonstrated by Fig. 4 (right). This increase in stable rank causes the point cloud to spread out around the origin, as illustrated in the last panel of Fig. 3 where the corresponding stable rank values are reported.
Theorem 1. Under the same assumptions as in Lemma 7 applying RoPE to X increases its stable rank by at least a factor of 2, and by up a factor of d, as the sequence length - co.
5

---

Published as a conference paper at ICLR 2026
Similar to Lemma 1 the precise increase factor depends the energy distribution of across the rotation planes: concentration on single plane yields a factor of 2, whereas uniform distribution across all planes yields factor of d.
We conclude that our prior observations accurately describe high-dimensional clustering behavior.
3.2 SINK TOKENS
Latent keys and queries cluster tightly into unaligned point clouds, and RoPE causes those clouds to disperse and overlap over time, particularly when input length exceeds training length. But how does this produce di-oo--dittibbuton behavior in the attention mechanism itself, and for transformer model as whole? We claim that the effect of clustering is mediated by the sink token.
Prior work establishes that transformers attend heavily the first token, regardless of input (iiao et al. 2023). Prevailing wisdom is that this prevents over-mixing: information retrieval is not always useful, so attention heads must formulate a null operation (Barbero et al., (0055). Softmax normalizes attention scores to sum to 1, so heads cannot avoid attending. Instead, they "sink" attention into a placeholder key conveying no information in practice first token, typically beginning- of-sequence indicator. When attention head does perform meaningful information retrieval, borrows weight from the sink token and reallocates it to the chosen key, as shown A.2
Under the intuitive overlapping-clouds model of latent keys/queries, sink token behavior is hard to reconcile. How can a single embedding align to all directions by default? Why not default to the current token where keys and queries. projections of the same input, are easy align
as in Mamba and other linear attention layers? (Gu & Dao 2023; Katharopoulos et al. 2020) Our observations help to explain this behavior. Same-token key-query self-alignment is difficult to consistently impose when keys and queries are distant. It easier to assign sink, and place its key near the origin, granting it a near-zero dot product with all queries. When average key-query product is negative, the sink becomes the most-attended by default. This is borne out in practice: key vectors for the first input token have unusually small Ez norm, as shown Fig. 5 (left). Smaller norm produces larger dot products, shown in Fig. 5 (right), which displays the average dot product of each key across subsequent queries, normalized by the largest product in the set. this case, that's consistently the sink token, with recency bias responsible for the later upward trend.
Layer
Layer 20
Query Product
0
Layer
Layer
Layer 25 Layer
L2 Norm
15
10
Layer
Layer
Layer 5
Layer 25
Layer
Layer
10
100
1k
10
100
Sequence Length
Sequence Length
Figure 5: Left: Key & norm function of position in sequence. Sink token is consistently small. Right: Keys have low dot product against subsequent queries in expectation, except for the first and most recent tokens. Scores are normalized by the highest value, in this case always the sink.
3.2.1 THE IMPACT OF RoPE ON SINK TOKENS
Within the training context length, key and query clusters are separated to the point that a sink token with small key norm can absorb the majority of attention weight. Beyond the training length, however, RoPE causes clusters to disperse and overlap. When this happens, key and query points begin to obtain positive dot products. This stops the sink token from functioning, and we claim this is the cause out-of--iitributiinn behavior when transformers are exposed to long inputs.
Fig. 6 illustrates this behavior in Llama3. The left-hand plot captures the attention weight allocated to the sink token as a function of input length. The share attention weight (with RoPE applied) varies widely but stably within the training length of 8k, but then falls sharply, decaying zero time as clusters progressively overlap and highly-aligner point pairs accumulate. Meanwhile, the activation of the sink token without RoPE stays roughly constant. We hypothesize that the activation

---

Published as a conference paper at ICLR 2026
starts off lower within the training length due to the observation in Fig. (right): key/query clouds are more opposed after applying RoPE, lowering the average dot product relative to the sink. In any is clear that RoPE causes catastrophic collapse of sink token weight for long inputs.
Fig. (right) confirms that the decay in sink token attention weight a function of key/query cluster overlap. As point clouds approach the origin and disperse, the chance for high directional alignment between point pairs increases, and so the maximum dot pairwise product should increase well. In practice, the maximum key/query dot product across all keys, per-query, does rise steadily over sequence length when RoPE is applied. Without RoPE, cluster behavior stable over time, and so too, therefore, the maximum degree of alignment between key and query points.
3.2.2 A UNIFIED UNDERSTANDING OF ROPE ATTENTION
We now establish unified geometric understanding of attention, RoPE, and sink tokens. The shift- invariance of softmax induces keys and queries to gather into opposing clouds across the origin. Sink tokens implemented by positioning the first key near the origin, making that key's s dot product with any query small. Because the clusters are opposed, average keylquery dot product is negative, defaulting attention the sink and mixing tokens only in the case of particularly aligned pairs.
RoPE complicates this arrangement by spinning points around and across the origin. Some channel pairs rotate much faster than others, but over time fully from their original locations. Eventually all channels shift into orbit, transforming previously well-separated key and query clusters into dispersed, overlapping balls. This produces positive dot products between keys and queries, overwhelming the small, but previously dominant sink token logit. Transformers with RoPE fail on long inputs because they effectively lose access to the sink token, broadcasting an excess of information from the wrong tokens forward through time.
and more channel pairs drift meaning-
Attn Score of Sink Token
1.00
w/ RoPE w/o RoPE
Max of QK product
RoPE
150
w/o RoPE
0.75
0.50
100
50
0.00
10k
20k
30k
40k
50k
60k
1k
2k
4k
8k
16k
32k
64k
Input Length
Input Length
Figure 6: Left: Sink token attention weight vs. input length. With RoPE, sink-token attention decays to zero beyond the training length; without RoPE remains stable. Right: Maximum Query- dot product vs. input length. The max QK product increases with length only when RoPE is active.
4 METHOD
Insight into RoPE's cil-of--istrbbuttonn behavior informs mitigation techniques. Based on our anal- ysis, two criteria must be met preserve sink token functionality across longer sequences with RoPE. First, key and query clusters must maintain some degree separation in the limit, in order prevent spurious positive dot products between keys and queries that overwhelm the sink token. Second, this lower bound on cluster separation must be attained within the training length; other- wise, keys and queries will drift closer than seen during training, again opening the door spurious positive dot products. Many scaling techniques exist that (perhaps unintentionally) fulfill these cri- teria. PI (Chen al.) 2023) and YaRN (Peng al., 2023). for example, both limit the degree of drift in low-frequency channels to that seen during training. This maintains the separation of key and query clusters those channels over extended contexts, while avoiding ouiti--disti drift.
Training a new model from scratch offers greater design freedom. Barbero et al.) (2024b) suggest that information is better preserved when RoPE is limited to fraction of channels, as positional and contextual information can be embedded in the RoPE subspace, while long-term semantic content can be allocated to the stable RoPE-fres subspace. However, we still expect this approach to fail on extended contexts, it only fulfills the first criterion (cluster stability) without the second (lower- bound attainment). Low-frequency channels still exist, reproducing the issues observed in $ Fig (left) illustrates this behavior by repeating the singular value analysis from Fig. 4
synthetic
7

---

Published as a conference paper at ICLR 2026
point cloud. Standard RoPE (blue) fulfills neither criteria, decaying FSV all the way to zero, and not within the training length. RoPE applied to half the channels (green) successfully imposes a non-trivial lower bound on FSV (and thus cluster overlap). but does not attain it within 4k steps. Thus we can still expect dut-of-distribution behavior when FSV falls below levels seen in training.
Birt Seguls value Rabo
1.00
0.8
0.8
0.50
0.2
0.09
0.0
-0.25
255
30C
64K
250K
11
40
8K
12K
20C
Inpur Longt
Impur Longth
Figure/ 7: Expected RoPE behavior for our proposed method and three baselines. Dotted line indi- cates hypothetical training length of and head dimension 128. Left: Repeats the Fig. singular value ratio before/after RoPE, for synthetic point cloud of ones vectors. Right: Long-term RoPE decay for the same techniques, showing similar behaviors.
An alternative solution is eliminate the low frequencies of RoPE altogether: raise the lowest value enough to complete a full cycle within the training length. This ensures that clusters "finish" drifting the origin, as they devolve into uncorrelated rotations overlapping shells around the origin- - that the model can expect to persist indefinitely. Channel pairs with RoPE cannot rotate "out of distribution" if the entire rotation are is covered. However, this again only fulfills one of the necessary criteria for length generalization: as shown in Fig. 7 high frequency RoPE (orange) with base frequency f = 652 (the lowest frequency to complete a cycle in 4k steps) approaches its FSV lower bound in 4k steps but this lower bound is trivial. Key and query clusters are decaying entirely, and thus failing to preserve information across longer contexts. Liu et al.] (2024) analyze this approach, but limit their evaluation perplexity. While perplexity is improved by the newfound stability over long contexts, does not capture the loss of distant information through several cycles of uncorrelated rotation. Long-context information retrieval is likely still difficult in this setting.
We hypothesize that combining high RoPE frequency with partial application fulfills both criteria, yielding natural generalization long contexts. Both changes required for stable, diseriminative behavior: key/query clusters fully merge in the RoPE subspace (satisfying criteria 2), but preserve sink token functionality via continued separation in RoPE-free subspace (criteria 1). This approach, which we name RoPE-ID (In Distribution). in Fig. gives the best of both worlds: FSV decay (red) non- non-trivially
and this bound is attained in 4k steps, maintaining cluster separation and sink token functionality by construction, while avoiding out- -fdtt---iittibttion behavior.
4.1 IMPLEMENTATION
We apply RoPE half the channels of each attention head, and adjust RoPE frequencies to attain desired behaviors. RoPE frequencies interpolate exponentially between and 1 ä where is the base frequency hyperparameter. We adjust the low end of this scale two full rotations per training length, as one rotation may still preserve correlation between low frequency channels (i.e. some decay still occurs after the purple line in the orange high frequency curve of Fig. (left)). Maximum rotation speed is set to one cycle every 32 tokens, to better preserve information over short windows. Taking a softmax over many IID logits increases the denominator but not the numerator, resulting in the mixture distribution becoming artificially smooth over time. We address this via input length based temperature scaling, borrowing from (Peng et al. (023). We evaluate both with and without temperature scale. Ablations in A.6 including comparisons to prior work of (Chen et al. 2024), indicate that these simple heuristics represent safe middle-of-the-rooe choices, with model perfor- mance gradually improving as high frequencies eliminated, until hitting sharp falloff. Details, code and further discussion for RoPE-ID can be found in A.4
To evaluate RoPE-ID, we pretrain example 1B- and 3B 3B-parameter decoders. Models use the Llama3 tokenizer and Dolma v.7 dataset (Soldaini et al.) 2024), reweighted per (Chu et al. (024). Training proceeds over 21 billion tokens, with hyperparameter and architecture details provided in A.5
8

---

Published as a conference paper at ICLR 2026
5 RESULTS
We compare RoPE-ID against four baselines. First is vanilla decoder with RoPE base frequency 500k, which we expect fail beyond the training length. Second and third are approaches from $ 4 increase RoPE frequency so that all channel pairs complete rotation, or apply the original RoPE on half the channels per head. The former should yield stable, but poor, performance over context lengths, while the latter should mitigate but not prevent performance degradation. The strongest baseline is tuning-free extension of the vanilla model using YaRN (Peng et al.) (023). an inference-tine time NTK method (bloc97, 2023a), with default hyperparameters. Note that YaRN provided oracle access to the final sequence length, when that length exceeds the default 4k. Across several benchmarks, our method is comparable to outperforms all baselines, while generalizing gracefully out of the box, requiring no model adjustment or oracle access to sequence length.
Table 1:
Average RULER benchmark accuracy scores by sequence length. Highest average score for each sequence length and model size is in bold; runner-up up is underlined.
Llama-1B
Llama-3B
Method
8k
16k
4k
8k
16k
RoPE
39.72
0.01
0.03
46.19
0.14
0.01
High frequeney
16.04
7.60
2.37
28.02
14.31
5.14
HalfRoPE
43.07
0.14
0
51.28
0.4
0.03
YaRN
40.24
35.55
30.25
43.90
45,09
40.14
RoPE-ID
39.15
29.71
14.29
44.86
37.51
21.95
RoPE-ID (scaling)
39.15
35.64
30.83
44.86
43.39
42.0
RULER (Hsieh et al. 2024) measures long context performance on synthetic tasks, such as needle- in-a-haystack retrieval and word counting. shows that baselines perform expected: RoPE and HalfRoPE perform well at 4k training length (with HalfRoPE even delivering a boost from its improved semantic encoding), but immediately fall to near-zero performance beyond that. High frequency RoPE degrades less in comparison, but also starts from a much lower score at 4k, as RoPE scrambling latent information aggressively. YaRN delivers robust extrapolation lengths 8k and 16k, without major penalty to baseline performance at 4k. Meanwhile, RoPE-ID with temperature scaling comparable YaRN with slight gains for longer sequences. It is marginally the strongest evaluated approach overall, but clearly surpasses RoPE-ID without temperature scaling, so we omit the non-scaled version from further analysis. A full breakdown of scores by task provided in A.7
LongBench (Bai et al. 2024) corroborates our RULER results, with reported averages over five task categories: single document question-answering, multi-document question-answerrigg few-shot learning, code completion and summarization. We exclude non-Enn non-English tasks as our models are trained on English data. A full breakdown of scores provided in A.8 Results in Table mirror Table I RoPE and HalfRoPE drop immediately after 4k, with HaliRoPE delivering a small boost within 4k. High frequency performs stably but poorly, while YaRN is able bring up performance for long inputs. Our method trails YaRN at 3B scale, but is superior at 1B. We conclude that RoPE- ID successfully generalizes to longer inputs out of the box.
Table LongBench accuracy scores, averaged over 14 English tasks, by sequence length. Highest average score for each sequence length and model size is in bold; runner-up is underlined.
Llama-1B
Llama-3B
Method
4k
8k
16k
4k
8k
16k
RoPE
14.61
8.23
8.73
18.62
11.36
10.42
High frequency
11.8
11.44
11.04
14.19
13.82
13.78
HaltRoPE
15.38
8.73
8.86
19.42
10.7
10.62
YaRN
14.84
14.54
14.09
15.87
19.29
19.63
RoPE-ID (scaling)
15.83
15.83
15.80
15.92
17.13
17.94
Commonsense Reasoning tasks act sanity check in Table Model scores all similar: RoPE frequency and number of channels have little impact on expressivity within the training length. To the degree that scores differ, RoPE-ID is in the top-3 for all tasks and settings.

---

Published as a conference paper at ICLR 2026
Table 3: Standard evaluation (accuracy) common sense reasoning tasks
Llama-IB
Llama-3B
Method
ARC-C
HellaSwag
PIQA
ARC-C
HellaSwag
PIQA
Avg.
RoPR
25.77
44.00
69.26
46.34
29.18
53.95
72.74
51.96
High frequency
25.17
42.99
69.26
45.81
29.61
53.32
71.87
51.60
HalfRoPE:
26.45
44,00
68.77
46.41
32.17
53.87
72.03
52.69
YaRN 25.60
41.89
68.61
45.37
29.10
52.46
72.20
51.25
RoPE-ID
25.60
43.58
68.88
46.02
30.03
53.95
72.25
52.08
Extended Context Length in above evaluations is limited to four times the training length, as tuning-free free extension techniques generally cannot avoid performance degradation beyond this point. To evaluate RoPE-ID at greater context lengths, we fine-tune our 1b checkpoints to 128k sequence length, applying YaRN scaling to the vanilla 1b model. Fig. X shows that RoPE-ID and YaRN trade off RULER performance over different context lengths, with RoPE-ID offering improvement on longer inputs. Surprisingly, applying YaRN-style scaling to the RoPE-ID checkpoint results tuned model outperforming both techniques at all input lengths. Further discussion and implemen- tation details are in A.9 and we leave exploration possible synergies to future work.
Accuracy
YaRN RafE
RoPE-D
FineiTuning Context Length
Figure 8: Average RULER accuracy scores for fine-tuned 1b models. RoPE-ID and YaRN trade off performance over length, but demonstrate surprising synergy when applied jointly.
Reapplying Analysis from $ 3.] confirms that our trained models obey our geometric framework. We recreate Fig. and 3 for baseline and RoPE-ID models in A010 Our IB example decoder mirrors the behavior of established LLMs, while RoPE-ID behaves as expected, delivering stable, and stably separated, key and query clusters over time.
DISCUSSION
From empirical analysis we develop a unified understanding of attention geometry and long-context failure modes. Keys and queries form tight clusters in opposing directions, allowing sink tokens to absorb attention weight by default via small l2 norm. RoPE inhibits this behavior by merging and dispersing the point clouds, particularly beyond the training length. Overlapped point clouds inflate key/query alignment, preventing the sink token from functioning. From this understanding, derive two natural criteria for length generalization with RoPE: lower-bounded cluster separation in the limit, and attainment of that limit during training. We fulfill these criteria and produce stable model behavior by applying RoPE with high frequency to fraction of channels, and demonstrate strong long-conted performance on downstream tasks out of the box.
Beyond RoPE-ID, other approaches based on this analysis are possible (e.g.. applying high fre- quency RoPE to fraction of heads, or manually injected sink tokens as in Hymba (Dong et al.) 2025 Additionally, our proposed criteria are necessary for generalization, but not sufficient. [Wang al., 2024), for example, demonstrate that high frequencies distribution radially, a failure mode not covered by RoPE-ID. Combining such techniques, as in RoPE-ID itself, could produce with inference- time model adjustments. We leave this, as well as further exploration of long-context fine-tuning of RoPE-ID models, to future work.
still drift slowly out of
more robust models. It is also possible to combine RoPE-ID
Reproducibility: Code for RoPE-ID is provided in A.4 with training details
A.5 Empirical
analysis techniques are straightforward, and evaluation standard benchmarks.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We are grateful to Selcuk Gurses (UAlbany) for valuable discussions on the theoretical analysis. This work was in part supported by NSF grant DMS-2208 126, the SUNY-IBM AT Research Alliance grant, the CEAIS grant, and the IBM PhD Fellowship Award.
REFERENCES
Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305.132255 2023.
Yushi Bai, Xin Lv, Jiajie Zhang. Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: bilingual, mul- titask benchmark for long context understanding, 2024. URL https: wawaiaiiiivvoooo http::////xxiioorraabb/////// 2308.
Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João G. M. Araújo, Alex Vitvitskyi, Razvan Pascanu, and Petar Veličković. Transformers need glasses! information over-squashing in language tasks, 2024a. URL
abs 2206.4466...0 04267
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? arXiv preprint arXiv:2410.06205, 2024b.
Federico Barbero, Alvaro Arroo,, Xiangming Gu, Christos Perivolaropoulos. Michael Bronstein, Petar Veličković, and Razvan Pascanu. Why do iims attend to the first token? arXiv preprint arXiv:2504.02732. 2025.
bloc97. Ntk-aware scaled rope allows Ilama models to have extended (8k+) con- text size without any fine-tuming and minimal perplexity degradation., 2023a. URL https: http:://wwwww..d..t....
reddit http:///////aaaaaaaaaaaaataaaaaa /tt.///et///1111///////aaaaaaaaa
sccaleedr wwwpaaaloosslaaaa_aee______ea modelo to_have/eeeeeeeeeeeeeee
bloc97. Add ntk-aware interpolation "by parts" correction, 2023b. URL https://g
com jquesngeeeseeeee. e/ sts/scaless led 1/1
Nicola Cancedda. Spectral filters, dark signals, and attention sinks, 2024. URL httht:t/pp:/xii>
bzq/abs/2402. 09221
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window
of large language models via positional interpolation, 2023. URL https://arxiv.ore/
abs/ 2206 15595
Yuhan Chen, Ang Lv, Jian Luan, Bin Wang, and Wei Liu. Hope: A novel positional encoding without long-term decay for enhanced context awareness and extrapolation, 2024. URL https:
//arxiv.org/ abs/ 2410. 21216
Linsong Chu, Divya Kumari, Tri Dao, Albert Gu, Raghu Ganti, Dakshi Agrawal, Mudhakar Srivatsa, Davis Wertheimer, Yu Chin Fabian Lim, Antoni Viros, Nelson Gonzalez, Tuan
HoangTrong, Ofir Arviv, Yotam Perlitz, Michal Shmueli, Haochen Shen, Minjia Zhang, Gabe Goodhart, Naigang Wang, Nick Hill, Joshua Rosenkranz, Chi-Chun Liu, Adnan Hoque, Chih- Chieh Yang. Sukriti Sharma, Anh Uong. Jay Gala, Syed Zawad, and Ryan Gordon. Bamba: Inference-efficient hybrid mamba2 model. https: http:////hhggggacccccceeooaao bamba-inferente ber 2024. Accessed: 2025-05-06.
-eteeenttbbbit hhybrid-mmmaa22.. -asoaaaammmdddee - 9070889F8908000 Decem-
DeepSeck-AL, Aixin Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai. Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao. Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui
Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. I. Cai, Jian Liang, Jianzhong Guo, Jiaqi
11

---

Published as a conference paper at ICLR 2026
Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kai ge Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua 7hang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang. Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shut- ing Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang. Xiao Bi, Xiaodong Liu, Xiaohan Wang. Xiaojin Shen, Xiaokang Chen, Xi- aokang Zhang. Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu. Xingkai Yu, Xinnan Song. Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang. Y. X. Wei, Y. X. Zhu, Yang Zhang. Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao. Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yonggiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang. Zhen Zhang, Zhenda Xie. Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025. URL
httpss//aarxvv.orrr//////2111
https://arxiv. wwt.:////g///aaaa.o.ooooo.ss 2t.bo//241..111 19437
Jacob Devlin, Ming- Hiig-Weiiee Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL htttp:://arxiv.oo
abs/ /ees/1810..m 04805
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending Ilm context window beyond 2 million tokens. arXiv preprint 2rXiv:2402.13777 2024.
Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, ZIJIA CHEN, Ameya Sunil Mahabalesh- warkar, Shin-Yang Liu, Matthijs Van keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Celine Lin, Jan Kautz, and Pavlo Molchanov. Hymba: A hybrid-head head architecture for small language models. In The Thirteenth International Conference on Learning Representations, 2025. URL
ew net/ ew.
Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Ko- renev, Arthur Hinsvark. Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind That- tai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jong- soo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia. Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani. Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren
12

---

Published as a conference paper at ICLR 2026
Rantala- Yeara, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh. Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Ku- mar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoy- chev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasie, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ra- mon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Ro- hit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa. Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Fdunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende. Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mi- haylov, Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Gold- schlag, Yashesh Gaur, Yasmine Babaei. Yi Wen. Yiwen Song, Yuchen Zhang. Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, An- drew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, An- nie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leon- hardi, Bernie Huang. Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovie, Brian Gamido, Britt Mon- talvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang. Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpemn Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harri- son Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jen- nifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings. Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Jun- jie Wang. Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena. Keqian Li, Kiran Jagadeesh. Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager. Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Ro- driguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin
13

---

Published as a conference paper at ICLR 2026
Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan. Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ra- maswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang 7hang, Sinong Wang, Sneha Agarwal, Soji Sajuyighe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk. Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Ku- mar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang. Wenwen Jiang. Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiao- jian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi. Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhao- duo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https:: /ttp/aaxivvvv/. wtg. alss 407.21770
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXhv:2402.00838, 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint 2:X1v:2312.00752, 2023.
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view, 2025.
Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm- infinite: Zero-shot extreme length generalization for large language models, 2024. URLhttps.
//axxiv.org/abs/2233
100.308..1113 16137
Cheng-Ping Hsieh, Simeng Sun. Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models?, 2024. URL htt.tttts///aaxiy.. wttp://wwwww///////////oo/ee 2204. 06654
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference ma- chine learning, pp. 5156-5165. PMIR, 2020.
Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of roPE-based extrapolation. In The Twelfth International Conference Learning Representations, 2024. URL
htttps://openreye www.ee/wwww.eeetooumteeee.. wwt.ooooummddooooooooo
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 1rXiv:1609.07780 2016.
OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko. Timur Garipov, Kristian Georgiev, Mia Glaese, Tarun Gogineni, Adam Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh. Saachi Jain, Zach Johnson, Chris Koch, Irina Kofiman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Lc, Guillaume Leclere, James Park Lennon, Scott Lessans, Mario Lezcano-Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily, Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCal- lum, Josh McGrath, Scott McKinney, Aidan McLaughlin, Song Mei, Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ash- ley Pantuliano, Giambattista Parascandolo, Jongsoo Park, Leher Pathak, Carolina Paz, Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Prochl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev, Max Schwarzer, D. Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal, Yang Song.
14

---

Published as a conference paper at ICLR 2026
Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wallace. Xin Wang. Miles Wang, Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whitney, Hannah Wong, Lin Yang, Yu Yang, Michihiro Yasunaga, Kristen Ying, Wojeiech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao. gpt-oss- 120b & gpt-oss-20b model card, 2025. URL htt.pss//aaxxx..orrr//b///////// 1
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yam: Efficient context window extension of large language models, 2023. URLhttps: http://aww.vv..///////..00000
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022. URL/httss:///zz....or///bbb/
12409
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL htttps://aradv..rrgabb////// 1961/ 10683.
Mark Rudelson and Roman Vershynin. Sampling from large matrices: An approach through geo- metric functional analysis. Journal of the ACM (JACM), 54(4):21 2007.
Amartya Sanyal, Philip HS Torr, and Puneet K Dokania. Stable rank normalization for improved generalization in neural networks and gans, arXiv preprint (tt.:I99000445550 2019.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint 2rX00:240..00159 2024.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding, 2023. URL 09864
Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large language models, 2024. URL http:: /ttp:::////xvv../// org/ /bwp///2.2.776665
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL httpt://///axxi
brg/aba/170 03762
Suyuchen Wang, Ivan Kobyzev, Peng Lu, Mehdi Rezagholizadeh, and Bang Liu. Resonance RoPE: Improving context length generalization of large language models. In Lun- Wei Ku, Andre Mar- tins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 586-598, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 103655//112224.fnmmlm findings--cl..333
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 174533 2023.
Shuai Yang. Yuying Ge, Yang Li, Yukang Chen, Yixiao Gc, Ying Shan, and Yingcong Chen. Seed- story: Multimodal long story generation with large language model, 2024. URL https:// arxiv. hww.vorrg/aao abs 2407 08683
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 THEORETICAL ANALYSIS OF CLUSTER DISPERSION UNDER ROPE
Let first fix the notations: R(X) denotes the application of RoPE to keyqquery matrix
X I1, In E Raad Specifically, RXX) = [R R. E Raxd where
R Diag R R2,00;2) E Rddd with Ry = [co((002) - sin()0,) sin(j0k) cos
E R2x and
0. = B 2(k- 1)/d for all 1 S k I d/2. For any a E Rd, x (8) := [ F26l E R denotes
the subvector the 4t rotation plane/channel. | 1|2 denotes the Euclidean or &2 norm of a vector
or denotes the spectral norm of matrix, and || || denotes the Frobenius matrix norm. The stable |
rank of a matrix X is defined as srank (X) 1
Lemma 1 (formal), Suppose X = wo E Rud where u E R" E R2 and ||ll2 = 1. If Vj. u, (1), and the sequence {u,] 1ÿ=1 has sublinear growth of total variation the sequence length i.e. - |už+1
u²l1 o(n),, then increases, we have
| x)
1
max Og + o(1)
X 1/2
V2
ISSdd/2
where or := 1|00000112 = / + U22u satisfying maxk as 2/d,d, 1].
Remark 1. The assumption that Σラ 1už+1 u o(n) imposes that the sequence {u]] must exhibit a certain degree of monotonicity. Indeed, if {u}} is strictly monotonic, then Σ}= n1 1
uiil (1). In contrast, if {u3} highly oscillatory, then ΣØ=1 |už+1 - 23 (n)) which violates the assumption.
Proof of Lemma
7 The pre- me-RRoPE spectral norm
|X Amax X X)
||e||2 Amax
||u/|||
|||l = ||u||2 @(Vn),
where Amax( (.) denotes the largest eigenvalue.
In what follows, estimate the growth of post-RoPE spectral norm, i.e., Vmmax (R())R((X Since ||o[((]|| || oz, due to rational invariance, without loss of generality, we assume ((k)
[ay,,0) 0) for simplicity.
Consider the Gram matrix G ((X))((X)))) Raxd since R(X) = |u Ruu, Un R Iaxdd we have
(R(v)
G R(X) R(X) R(()
jur Rio, ... Un Un
Σ
(R))
un (
j=1
Rio,oo v11)
[(R,0 ... (Ry00 0(4/21)T]
j=1
Rijears (d/22)
Diagonal Blocks of G. For < < d/2, the 4th diagonal block of G is given by
G
Σ 1 R R ul (R, o()) 22x2
Recall that
Ry = [cos(j(a) - ssin(j
sin(j(k) cos(j0z)
16

---

Published as a conference paper at ICLR 2026
have R3 ((4) ork (c)sss sin(())) and thus
cos² (j0k)
cos()0))sin((0))) (j0n)
Gk₂
2
cos(j(x) sin(j01)
j=1
k
2
1 + cos c00s(2j0k)
(2n02)
2 j=1
Uj
sin())0x) 1 (2)0k)
ag
|
1/2 In + E
2
where
Es.k
n 04
Σ
ug (os((2)0k)
sin(2)0x) sin(220x)) (2)00.)
2
j=1
Off-Diagonal Blocks of G. Similarly, for & I, the (k, 1) block is
Ex Gb.1
ú o 0 w (8)) (R (0))
n
(ik( cos cjs() cos( (oxx) sin()(tt) sin(j(x) cos() sin(j0n, sin (j0i)
j=1
0
Σ
n
ug
(O0 + 0:)) + cos(j(O> - 0z)), sin(j(0s + 01)) - sin(i - -0:)
(0n + 0:)) + sin(j(0x 01)), cos([j(xx 01)) cos(j(0s. 01))
2
j=1
Therefore,
| 2
Diag 12 ,02/212) + E.
Bounding Amax (G). We want to show that E is subleading term and is entry-wise o(n). Note that Cr S 1, for any entry of Es,k. Vk, its magnitude upper bounded by
2
Σ uz cos (2j0k)
Σ uz sin(2)0c)
2004
Denoting Sj = Eim1 2002 and using Abel's summation formula, have
n
n-1 (u - u2)ss,
u 2502
2
j=1
n-1
S u |aal + max
|S
Σ
| - u²}b
15jšn-1
1=1
where |S,| |
1- 25
$ 1 2- sinn( = O(1 for all since O2 1
2-
g: 1] is an irrational multiple of т. Moreover, we have assumed Σ/=1 |u3 23 = o(n), so
Σ 250 o(n).
Similarly, for any entry of the off-diagonal block Ex,i, we can also show that its magnitude is upper bounded by
Σ 23c
+ Σ
018 = =(())
since (0s + 0:)/2 are still irrational multiples of
17

---

Published as a conference paper at ICLR 2026
By Weyl's inequality, have
| 2
max of
EEl2 I Xmax (G) -
max of + |
2
2
Note that since is fixed,
E 2
|E = o(dn) = o(n), and thus
| 22
Amax (G)
max 0i o(n).
2
Finally, since ||u||2 @(vn), we have
[
X )|
Amax G))
max ax + o(1).
o(1).
X
|u/12
v2 1SKSd/2
Lemma 2. Applying RoPE preserves the Frobenius norm of any X, i.e., |R(X))|p = |X|
Proof of Lemma 2
Since ||
ا and |R(X) Σラ= RR it suffices to
show that for all j. ||,, R 12 Since every diagonal block of R3 is a x 2 rotation matrix, R2 also rotation matrix and thus norm preserving, which completes the proof.
Theorem (formal). Suppose = uo T E Raxd where u E R", v E Rd and ||1|2 = 1. Under the same assumptions on as in Lemma we have
Drank(R(X))
2
lim 500
E [2,
srank X)
max1sk5a/2 of
where = -110>(()) following the definition in Lemmal !
Proof of Theorem
Using Lemmas and 2
we have
srank(((X))
[|R(X)]l
2
||I2
2
2
srank(X)
X
(
112
maxigk5d/2 ax + o())
Since Σ d/2 1|e112
k= ot
have max; of [2/d, 1]. Taking n - oo completes the proof.
A.2 ATTENTION MAP AND SINK TOKEN VISUALIZATION
mevtlee
022
2:20
talo
180 Laper Head
3el Layer Heed
Layyy
Hood
Figure 9: Attention patterns of three heads in LLaMA3-8B, Sink token behavior is clearly observed, when performing non-trivial token mixing.
18

---

Published as a conference paper at ICLR 2026
3 w/o RoPE
Uaam33 w/ RoPE
Gemma W/o RoPE
Gemma W/ RoPE
Olmo wio RoPE
Olno RoPE
230
600
Det Product
200
400
-40
200
50
-00
0-
8k
8c
64k
4k
8k
64k
4k
8k
64k
length
Inpurt Kay Key Dat Product
Inpul
Query-Query Dot Product
Query Key Dot Product
Figure 10: Mcan of the pairwise dot product of query and key values across all heads for Llama3- 8B, Gemma-7B and OLMo-7B which showcases the effect of RoPE across various context lengths showing similar trend Fig.
Silhouette
Davies Bouldin
1.0
3.0
wlo RnPE
w/l RoP
0.9
RoPE
Devies Davics-Douldin Douldinn Index
2.5
RoPE
Siloouete Score
2.0
0.5
1.5
0.4
1.0
0.2
0.5
0.0
0.0
1k
2k
4<
8k
16k
32k
64k
1k
2k
4k
8c
15k
32k
64k
Input Length
Input Length
Figure 11: Silhouette Score (left) and Davies- Index (right) showcasing the effect of RoPE on Internal Representation Clustering for Llama3- Caamma--BB-Inttiiee SB-Instruct. Lower Silhouette Score and higher Davies-Bouldin Index represents more overlap.
A.3 ADDITIONAL CLUSTERING ANALYSIS
Here we repeat the analysis performed in Fig. 2 for other distance and clustering metrics. Fig. TO shows inter- and intra-cluster alignment measured by dot product rather than cosine distance, better reflecting the actual attention logits. This introduces noise, as embedding norms can shift over time without affecting clustering behavior, and cosine distance norm-invariant whereas dot products are not. This also introduces large differences between different models. Nevertheless, overall trends are roughly the same.
Fig. II directly quantifies the degree of clustering, using Silhouette Score (left) and -Buuld-BBoulddn Index (right) in Llama3-8B. Results again mirror Fig. 2 clustering consistent across sequence lengths prior to RoPE (and even increases beyond the training length), but falls over time once RoPE is applied (Silhouette Score decreases, while Davies-Bouldin Index increases as sequences become long). We conclude that clusters are behaving as described in the main paper.
A.4 IMPLEMENTATION DETAILS
In our approach we apply RoPE to half the channels of each attention head, and adjust the RoPE frequencies to attain desired behaviors. Standard RoPE frequencies interpolate exponentially be- tween 1 and 1. where is the base frequency hyperparameter, typically 10k or 500k. We adjust both endpoints of this interpolated scale. First, we must ensure that all frequencies are high enough to complete at least one rotation within our training length of 4k tokens. We set two full rotations as the minimum, as one rotation may not be sufficient to fully eliminate correlation between low frequency channels (i.e. some decay still occurs in the high frequency curve of Fig. 7 (Left), even after the slowest channel pair finishes full rotation at the purple line). We therefore update the min- imum frequency scale value from to 4096 Second, we also pull the maximum frequency scale of 1 toward more conservative value. Since apply RoPE only a fraction of available channels, it is important that the channels discriminative. RoPE's s fastest channel pair completes a cycle in 2x tokens, after which information is effectively lost, becomes impossible disentangle
4
19

---

Published as a conference paper at ICLR 2026
relative position modulo 2x from content. An effective 6-token window is very aggressive, pull back the max frequency to 2m
completing cycle in 32 tokens.
32
While applying high-frequeeny RoPE to fraction of channels ensures stable clustering and sink token behavior for long inputs, out-of-diittibuttion behavior can still occur via the softmax activation. Key/query dot products stable over time by construction, taking softmax over an increasing number of IID key/query pairs will increase the softmax denominator, without a corresponding increase the numerator. The result the mixture distribution becoming smoother than expected over time. We account for this by introducing temperature scaling based on input length, borrowing from (Peng et al. )023). The adjustment is (1+001+ 0.11+I(mmim) (minn4096, (4096, n)))² where 4096 is the training length and is the given input length.
Example code for this approach in HuggingFace Transformers is provided below.
Codeblock 1: Modification of scaling factor within the intieniioninterface
arc/transfoorereae s/ htt.alammmmmmmaaaa ing lama.pp
class LlamaAttention (nn Module) (nnMMoulee))))
def forward(...).
attn_output, attn_weights attention_interface ( sclf, query_states,
key_states, value_states,
11
ittention_mask,
dropout=0.0 if not self.trainee else self. attention iottrpoott welling=self.scalln
15
(0.1 math. log (max (eurrentt ccaucrerettpooiiioonn 4096) / 4096) + 1) +2, **kwargs,
16
Codeblock 2: Modification of modeling rope utils with our method
ROPE_INIT_FUNCTION
"ourmethod"; _ommptee_oormmttbd__aaaaattttttr
def _wc.puteeeouue otttooouuouuoooeeeeeeeeeeeo mathod__paaaeeeeeteeeeeee parameters(
config, device, sen = Nene, **rope_kwargs
11
if config not None and (en(rop__wwargs) 0: raise ValueError
12.
13
if len (rope_kvargs) > 0:
base - rrop_kwwagee []base"] aim rope_kwargs "dim"]
elif contig is not None:
17
base = config. rope theta & partial_rotary_facte
18
\ if hasattr (config, "partial_rotarr__fcttt""" else 1.0 head_dim getattr (config, "head_dim", None) or configghidden....... hidden_iize // dim int (head_diin pnttial_ottary_factor
-
20
21
22
attenticn_factor 1.0 Unused in this type of RoPE
25
26
logstart = math. log (2 math.pi / base) $ 1 cycle in ratio steps logend math. (4 math.pi / 4096) $ 2 cycles in 4k steps pos torch. aranhe (0, dim // 2, devi deveicc)) / (dir // 2 1) logfreq pos (logend logstart) logstart
27
20

---

Published as a conference paper at ICLR 2026
inv freq logfreq. return inv_freq, attention_facter
31
Codeblock 3:
Applying RoPE to only half the channels. Due to how HuggingFace implements
complex-valued RoPE, the first and third quarter of real-valued channels correspond to the first half of complex-valued channels.
1iama.py
def app_y_rotary_pos_coo (g, k, sin, iesttionniss-None unsqueeze_dimer cos unsqueeze (unsqueezee_ii) sin sin.unsqueeze (unsqueeze_dim)
quartille size = q.shape // 4
ql, c2, q3, q4 torch.split (g, split_ size or_ ou kquartile_size k. shapee[-1]
size, dim=-1)
11
k1, k2, k3, k4 torch.
dim=-1)
rot terch. (ql, q3), dim=-1)
15
rot terch.cat (k1, k3), dim--1)
16
roro_embed = rq cos) + trotate_hale ret) sin)
rot__emeed_ (k_rot cos) (rotate half rot) sin)
20
ql up_updd q3_updated torch. torch.split(@_rro__emeed \ dim=-1)
21
k1 updated, k3 k3_ updated = split split_size wtw.sresettttsseeeeeeeeeeeeeeeee ott.aaaee....tttttoooo....aaaaad
(aapcch.splitt_o__eeeeeeeeee rot___beeeee,,,e \
dim=-1)
24
25
embed torch. cat (ql_updated, q2, q3 qd_pddae q4), dim=-1) cmmbd = torch. cat (ki_updated,, k2, k3 u3_updated, k4), dim=-1)
26
29
return e_embed k_embed
A.5 TRAINING, MODEL, AND EVALUATION DETAILS
During evaluation, we take care avoid inducing out-of-distribution behavior not related to extended context length. In particular, we report point cloud behaviors "with" and "with- out" RoPE. Both cases drawn from the same single forward pass from given model, with the model unaltered and RoPE applied. Strictly speaking, these point clouds come from after and be- fore the application of RoPE, respectively. This keeps observations within-distribution. discussing un-RoPEd point clouds inside of a RoPE-using model. Performing the actual attention without RoPE would cascade errors through the model.
Evaluation:
when
Model training proceeds over 21 billion tokens, with context length of 4096 and half million tokens per minibatch. All models are trained across 16 NVIDIA A100s in parallel. We employ learning rate of 3e - 4, with warmup over 2k steps and cosine decay. Optimizer is AdamW with X (.9, (99, and weight decay 0.1. Model architecture follows Llama3, with details provided in Table
Training:
21

---

Published as a conference paper at ICLR 2026
Table 4: Model architectures used for pretraining and evaluation
Parameters
1B
3B
Vocab
128256
128256
Width
1280
2048
Layers
32
48
Heads
16
16
KV heads
4
Head dim
80
128
Inner dim
4096
7168
A.6 RoPE-ID HYPERPARAMETER ABLATIONS
We perform additional ablations on the hyperparameters of RoPE-ID, namely the high and low frequencies, the number of channels with RoPE applied, and the degree of temperature scaling. We train several additional IB demo models using the same training procedure, and report average scores from the RULER benchmark.
We observe a consistent general trend: reducing the number of high- figgufrreuency channels improves performance, especially shorter contexts. However, the model eventually reaches threshold beyond which length generalization drops sharply. We hypothesize that without enough high- frequency channels, the model instead learns encode position based on learned patterns of latent drift, which do not generalize to longer contexts. This explains the patterns observed in Table 5 and Fig. 12] where decreasing the number of RoPE channels, and increasing the wavelength of the highest frequency, gradually improves performance, until triggering catastrophic collapse at longer contexts. We conclude that our hyperparameter choices for channel fraction and highest frequency (50% of channels, shortest wavelength 32) represent safe middle-gne middle-ground.
Halving the wavelength of the lowest frequency highly beneficial, showing that one period is in- deed not sufficient to decorrelate all rotating channels. This aligns with Fig. [ where the orange curve (RoPE with high enough frequency to complete one period) still performs some FSV de- cay beyond the training length. Meanwhile, RoPE-ID, the red curve, holds the FSV ratio constant beyond the training length.
The impact of temperature scaling, shown in Table 5 is minimal. Here we raise and lower the exponent of the YaRN temperature scaling formula, and find that the default value of 2 works fine.
4K
Fractionn Channels
Wavelangth (Highest
Wavelength (Lowest
(Exponent)
40
-0
ACC. rayy
30
42
10
178
1,4
3/8
1/2
5/8
3/1
7/5
24
16
54
2055
4056
1.5
2.0
2.5
Fraction.oo of Cha vnels
Wawelength llighest Treg)
Whove engtn fLowest Freq]
Temperature Scale (Exponent
Figure 12: Llama IB ablation studies with average RULER scores on the y-axis, The x-axis of the 4 plots cover fraction of channels, wavelength (high frequency), wavelength (low frequency) and temperature scaling, respectively. The color coded legend represents context lengths 4k, 8k and 16k.
These results provide further insight when comparing RoPE-ID prior partial-RoPE methods, such as HoPE (Chen et al., 2024). While HoPE and RoPE-ID are algorithmically similar, the differing justifications lead to better-informed design choices for RoPE-ID. In particular, HoPE corresponds to several suboptimal settings in our ablations: no temperature scaling, a high frequency wavelength of 2x, and low frequency wavelength of 4k, all of which damage retrieval performance varying
22

---

Published as a conference paper at ICLR 2026
degrees. The number of RoPE channels in HoPE coincidentally matches setting (50%) for the lb model, but climbs to 80% for 3B, which we find to also be suboptimal.
Table 5: RULER scores for LlaMA IB ablations as function of fraction of channels, wavelength, and temperature scaling
Llama-1B Models
Fraction of Channels
4K
8K
16K
1/8
46.30
35.12
2.18
1/4
41.59
38.75
36.41
3/8
39.11
32.83
27.12
39.15
35.64
30.83
5/8
36.52
31.52
17.72
3/4
33.17
27.40
21.63
7/8
34.23
16.11
9.05
Wavelength (Highest Freq)
4K
8K
16K
2n
37.37
35.66
27.77
16
38.46
36.52
26.76
39.15
35.64
30.84
64
41.62
27.16
9.46
Wavelength (Lowest Freq)
4K
8K
16K
2048
39.15
35.64
30.83
4096
36.11
20.66
16.07
Temperature Scale (Exponent)
4K
8K
16K
39.15
34.56
29.51
2.0
39.15
35,64
30.83
2.5
39.15
36.91
31.15
3.0
39.15
37.58
29.61
23

---

Published as a conference paper at ICLR 2026
A.7 RULER BENCHMARK
Table 6: Performance of different methods on the RULER benchmark. Highest average score for each sequence length and model size is in bold; runner-up is underlined.
Method
Seq.
N-S1
N-S2
N-83
N-MKI
N.MK2
N-MKS
N.MV
N-MQ
VT
CWE
FWE
QA-1
QA-2
Avg.
IB Models
23.45
RoPE
0.13
0.01
16k
0
0
0
0
0.4
0.03
4k
42
44.8
35
57.4
0.2
3.6
7.05
4.85
0.1
16.55
15.6
16.04
High Frequeccy
15.6
1.2
0.04
2.87
7,60
16k
100
96
76.4
22
17.2
12:2
10.95
3.58
48.72
27.73
22,42
22.4
43.07
HaIRROE
8k
0
0
0
0.07
1,6
0.2
0.14
16k
23.45
92.4
16k
99.8
85.8
68
44.2
1.6
1.4
20.35
17.7
18.24
0.34
12
11.07
12.8
30.25
62
35.4
RoPE-ID
100
33.4
40.4
0.5
21.8
20.55
11.44
2.44
29.33
10.28
15.8
29.71
16k
96.4
19.1
1.2
14.2
0
9.65
2.1
2.36
20.33
8.73
11.4
14.29
4k
100
100
85.6
62
2.6
29.1
30.85
0.52
12.12
35.4
27.52
19.6
99.15
RoPE-ID (Scaling)
100
70.6
50.8
26.3
34.55
12.8
6.16
33.13
15.4
35.64
3B Models
4k
100
100
93.4
56
7.2
3.8
39.55
53.55
12.96
41.28
36.07
30.25
26.4
46.19
RoPE
0.07
0.14
16k
0.13
0.01
High Prequency
27.8
24.4
25.6
23.8
1.8
4.8
17.45
15.4
10.12
10.6
14.31
16k
8.8
9.2
5.2
7.8
0.4
7.65
2,05
1.58
5.53
7.27
11.4
5.14
100
99.5
85
58.9
55.7
15.8
44
38.4
33.4
25.6
$1.28
HalfRoPE
16k.
0.13
0.03
98.2
91.5
57.2
4.8
64.
38.6
20.88
14.66
21.27
28
21.2
43,90
YaRN
8k
100
100
98.5
71
4.4
42.22
60.25
20.32
13.94
29.8
17.18
26.6
45.09
16k
99.8
63.4
34.55
.6.
11.76
8.66
18.8
15.82
20.2
10.14
4k
100
100
97.8
77.4
9.4
$.4
28.2
37,25
12.84
13.18
42.27
30.58
25.6
11.86
RoPE-ID
8k
100
99
88.6
45.4
23.7
36.5
13.4
6.54
36.4
14.28
21.4
37,51
16k
97.8
77.4
28.2
37.25
12.84
13.18
25.6
RoPE-ID (Scaling)
8k
100
100
96
$5.8
6.6
3.4
43.4
48.9
20.6
12.36
15.62
24.4
43.39
16k
90.8
99.2
86
$9.6
7.2
0.4
48
42.4
36.56
10.24
31,07
17.07
17.4
42.0
24

---

Published as a conference paper at ICLR 2026
A.8 LONGBENCH BENCHMARK
14 English Tasks for Different Categories used for evaluation:
Single Document QA: NarrativeQA, Qasper, and MultiFieldQQAenn Multi-Documeer QA: 2WikiMultrhopOA, HotpotQA, MuSiQue Summarization: GovReport, MultiNews, QMSum Few-shot Learning: SAMSum, TREC, TriviaQA Code Completion: LCC, RepoBench-P
Table 7: Performance of different methods LongBench, averaged by task type. Highest total average score for each sequence length and model size in bold; runner-up is underlined.
Methods
Seq.
Single-Doc QA
Multi-Doc QA
Summarization
Few-Shot Learning
Code
Avg.
1B Models
4k
6.65
4.32
14.22
27.27
23.55
14.61
RaPF
8k
4.53
1.69
11.48
7.55
19.75
8.23
16k
4.84
2,08
12.74
8.19
19.33
8.73
4k
5.26
4.22
12.97
19.3
19.96
11.8
High Frequency
81:
5.31
4.22
12.61
17.61
20.47
11.44
5.2
3.92
12.49
16.71
19:77
11.04
4k
6.94
4.72
16.07
28.73
22.99
15.38
HalfRoPE
1.94
15.89
6.78
16.77
8.73
16k
5.25
2
16.17
6.5
17.17
8.86
YaRN
8k
6.83
4.77
15.29
26.76
21.33
14.54
6.71
4.80
14.61
31.68
24.11
RoPE-ID (scaling)
8k
7,18
5.23
14.69
31.39
23.11
15.83
3B Models
7.32
4.81
14.79
35.13
37.28
18.62
8k
5.19
2.57
13
11.03
31.84
11.36
5.12
2.51
11.67
11.22
27.15
10.42
4k
5.42
4.52
12.61
25.85
26.69 69
14.19
High Frequency
S1.
5,82
4.6
12.64
23.06
27.58
13.82
16k
5.93
4.51
12.51
22.7
28.02
13.78
4k
7.75
4.81
17.14
40.68
30.4
19.42
HallRoPE:
Sk
5,13
2.23
16.81
9.84
23.89
10.7
16k
4.99
2.23
16.11
10.87
23.03
10.62
6.5
15.2
YaRN
8k
7.96
5.46
15.57
40.30
31.12
19.29
16k
8
5.94
16.45
28.81
19.63
7.71
5,23
RoPE-ID (scaling)
8k
8.52
5.65
14.55
36.37
22.24
17.15
25

---

Published as a conference paper at ICLR 2026
A.9 LONG-CONTEXT FINE- TUNING WITH ROPE-ID
While our analysis focuses on RoPE-ID as tuning-free approach to length generalization, we can also combine it with fine-tuning further extend the effective context length. Here take trained Llama IB models and tune them to 128k context length in stages: first, we load the 4k model checkpoint and apply any relevant RoPE frequency scaling. Then, we continue pretraining for 5k steps, with sequence length increased to 32k. The total tokens per batch is held constant at 500k, and learning rate warms up over 250 steps until reaches the final LR of the previous checkpoint (3e 5), where it is held constant. We then repeat the process for another 5k steps, going from 32k sequence length to 128k.
We extend three models in this fashion: first, we apply YaRN scaling to the baseline RoPE model during each jump sequence length. Second, we tune the RoPE-ID model with no adjustment to RoPE frequencies. Third, tune the RoPE-ID model, but with YaRN-style scaling also applied. YaRN cannot be applied directly to RoPE-ID models as the default YaRN hyperparameters do not work for such high frequencies. We therefore set the scaling thresholds to the highest and lowest fre- quencies and interpolate in-between (the highest frequency unchanged, and the lowest frequency scales up by L' L'/L, where L', L are the new and old sequence lengths, respectively).
Results for RULER are given in Table 8 and Fig. 8 The tuned RoPE-ID model exhibits better length generalization at context length 64k and above. Notably, the combination of RoPE-ID and YaRN-style scaling achieves superior performance at all context lengths, compared to either method alone. This suggests that YaRN and RoPE-ID do address the same out-o--distribution behaviors, indicating possible synergy between these approaches. We leave such exploration for future work.
Table 8: RULER scores for LlaMA IB models up to 128k context length, where experiments cover tuning with YaRN, RoPE-ID, and RoPE-ID YaRN.
Llama-1B Models
Fine- TunnnT
YaRN
RoPE-ID
RoPE-ID
YaRN
4K
43.29
39.49
47.09
8K
39.27
35.81
41.94
16K
34.55
30.63
38.59
32K
32.74
30.66
36.16
64K
21.90
27.34
31.48
128K
12.25
19.78
29.23
26

---

Published as a conference paper at ICLR 2026
A.10 REPEATED CLUSTERING ANALYSIS FOR TRAINED MODELS
We repeat our original analysis on our trained IB models (baseline and RoPE-ID), Our baseline model exhibits the behavior observed in state of the LLMs, while our RoPE-ID model exhibits the desired stable behavior and consistent clustering across sequence lengths.
RoPE-ID w/o RoPE
RoPE-ID w/ RoPE
Base w/o RoPE
Base w/ RoPE
1.0
10
022
088
08
Cosiie Distance
0.6
0.6
0.0
0.4
0.4
0.2
0.2
-0.2
0.0
0.0
-0.3
1k
24
48
8kc
16k
12k
1k
22
42
B2
16k
70k
14
4k
R2
16k
77%
Length
Imput Length
Imput Length
Query-Query Cosine Distance
Key-Key Cosine
Query-Key Key Distance
Figure 13: Pairwise angular distances within- and between-clusters for our trained models. Base model matches prior LLM observations, including inflection point for key-query product with RoPE at the training length (4k). RoPE-ID maintains stable behavior over time.
Usma
-15
-20
-10
-1
-10
10
13
-10
-10
9
10
(3)
Usia 18 ROPE-I
Layer 1: Head
15
15
15-
10-
-10
RoPE
RoFE
RoFE
64X
RoPE
Figure 14: Clustering behavior for our trained models. Vanilla transformer (top) matches prior LLM observations; RoPE-ID (bottom) maintains stable cluster separation.
27

---

Published as a conference paper at ICLR 2026
A.11 ADDITIONAL CLUSTER VISUALIZATIONS
We repeat Fig. 3 for additional layers and heads. The same general trend can be observed, where separated clusters disperse and overlap when RoPE is applied longer contexts. We randomly sample 16 Key heads and their corresponding Query heads from Llama3-8B-Instruc and Query- Key head pairs from Gemma-7b and Olmo-7b.
10
0,4
-10
-10
-10
-10
-20
-10
10
30
-30
-20
-10
-00
-2
-20
10
30
[8]
(b)
(c)
[d)
Llama-3-00 ---tt------------ instruct Layer Head
0-
-10
01
-20
-20
20
-20
20
-20
-20
-0
-20
-20
-10
20
1a1 4K W/o RoPE
(b) 4K RoPE
(c) 64K wdo RoFE
[d) 64K ROPE
Lama
Head
RLA
10-
0,8
-10
-10
-10
-16
-30
-10
:0
-30
-20
-10
-20
-30
-20
10
30
(c)
(d)
Llama-2 3 Instruct Laver Head
Q
-20
-20
-10
20
-20
-20
-10
10
20
-20
-10
-20
-20
-10
10
(al 4K w/o RoPE
(b) 4K w/ RoPE
(c) 64K wlo RoPE
[d) 64K
ROPE
Lama Instruct
Head
10
10
-10
-10
-10
-10
0L1
0,1
-10
-30
ROPE
28

---

Published as a conference paper at ICLR 2026
10
0
-10
-10
-10
-10
-20
-10
10
-0
-20
-10
20
-30
-20
-20
10
30
[8]
(c)
Lioma
16
-10
-10
-10
-16
-30
-10
10
-30
-20
-10
-30
-20
-30
-20
10
30
(c)
Llema-s-BU--matrue Layer Head
10
10
30-
-10
-10
-10
-30
0.1
0.1
RoPE
ROPE
Llama-d3 5B instruc: Layer 20 Head 3
10-
10
J0-
-10
-10
-10-
-30
-20
-10
RoPE
4K RoPE
64K
RoPE
64K ROPE
Libma 88 Instruct Layer 20 Head
10-
10-
10-
10-
03
-10
-10
-10
-30
-20
-10
4K RoPE
(6) 4K
RoPE
(c) 64K w/o RoPE
(d) 64K RoPE
Lioma 3 88 Instruct Layer 20 Head
10
10-
03
-30
03
-36
0.1
-20
-20
(8) 4K w/o RoPE
(6) 4K
RePE
(c) 64K w/o RoPE
(d) 64K ROPE
29

---

Published as a conference paper at ICLR 2026
Llama-2
10
0.4
-10
-10
-10
-16
-30
-20
-10
10
-30
-20
-10
20
-30
-20
20
10
30
[8]
(c)
16
-10
-10
-10
-16
-30
-10
-30
-20
-10
-30
-20
-30
-20
-0
30
(c)
Gormema /b Head
10
10-
30-
-10
-10
-10-
-30
RoPE
ROPE
Gemmaa- Laver Head
19-
10-
10-
-10
-10
-10-
-30
-20
-00
RoPE
4K RoPE
64K
RoPE
ROPE
Gemmia 7b Laver 12 Head 16
10-
10-
10-
-10
-10
-10
-30
-20
-20
4K RoPE
(6) 4K RoPE
(c) 64K
RoPE
(d) 64K ROPE
Gomma T6 Layer 20 Head
10
10-
-30
-10
-30
-20
-20
-20
-20
(8) 4K w/o RoPE
(6) 4K RoPE
(c) 64K w/o RoPE
(8) 64K ROPE
30

---

Published as a conference paper at ICLR 2026
-10
-10
-10
-10
-30
-10
10
-30
-20
-10
20
-30
-20
10
-20
10
30
(c)
16
-10
-10
-10
-16
9
-30
-10
-30
-20
-10
-30
-20
-30
-20
-0
30
(c)
Liemme-/B Laver 28
10
10
30-
-10
-10
-15-
-30
RoPE
ROPE
OLMo-7B Layer Head
19-
10-
J0-
-10
-10
-10-
-30
-20
-10
RoPE
4K RoPE
(c) 64K
RoPE
ROPE
OUMo 7b Layar's 5 Head
10
10-
10-
10-
-10
-10
-10
-30
-20
4K RoPE
(6) 4K
RoPE
(c) 64K RoPE
(d) 64K ROPE
Oute 7b Layer 14 Head
10-
10
-30
-30
-20
-20
-20
(6) 4K w/o RoPE
(6) 4K RePE
(c) 64K w/o RoPE
(d) 64K ROPE
31

---

Published as a conference paper at ICLR 2026
14
10
10
-0
10
10-
10-
Layer
0
-10
-20
-30
-20
-10
10
20
30
-20
-30
-20
-10
10
20
-30
-20
-00
10
20
-30
-20
-10
10
20
30
[a] 4K w/o RoFE
(b)
4K
RoPE
(c)
64K w/o RePE
(d) 64K
RoPE
ULMo-bb Layer 22 Head 21
-10
-10
-20
20
20
-20
-00
-0
-30
20
151 4K W/o RoPE
(b) 4K RoPE
(c) 64K w/o RoFE
(d) 64K
RRPE
-10
-10
-10
-10
-20
-10
10
-30
-20
-10
-30
-20
-20
10
30
(c)
(d)
OLMo-7b Layer 31 Head 29
20
20
0-
-20
20
10
10
-20
20
10
10
20
30
10
10
20
20
10
30
101
4K
w/o ROPE
(b) 4K w/ noPE
(c) 64K wio RoPE
(d) 64K WT ROPE
32


Reviews:
[{"review_id": "DaFaXem7sd", "reviewer": "Reviewer_QNK8", "summary": "This paper presents a geometric analysis of Rotary Positional Embeddings (RoPE) to explain long-context failure modes in Transformers. The authors find that RoPE induces tightly clustered query/key point clouds within training length, supporting the formation of sink tokens that prevent over-mixing. However, when input length exceeds training context, RoPE disperses and overlaps these clusters, disabling sink token functionality and causing degraded attention behavior.\nBuilding on this insight, the paper proposes RoPE-ID (In-Distribution), a modification that applies RoPE with high frequency to only a subset of channels, maintaining cluster separation while preserving position information. Experiments on LongBench and RULER with 1B- and 3B-parameter Transformers show that RoPE-ID achieves comparable or superior long-context generalization to tuning-free baselines such as YaRN, without fine-tuning.", "questions": "In Figure 7, could the authors clarify the comparison between the configuration with $\\theta = 500k$ and the RoPE-ID variant? The plot seems to indicate that the $\\theta = 500k$ baseline achieves a higher FSV ratio, implying stronger cluster preservation. Since a smaller $\\theta$ corresponds to higher-frequency rotations, why does the high-frequency (orange) curve yield the lowest FSV ratio? This appears to contradict the authors’ claim that higher frequencies should maintain cluster stability.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. The geometric interpretation of RoPE and attention behavior is original, intuitive, and interesting, linking positional encoding, cluster geometry, and sink token dynamics into a unified framework.\n\n2. The paper validates hypotheses through detailed analyses (PCA projections, singular value decomposition, attention maps) and replicates findings across multiple LLM families (LLaMA, Gemma, Olmo).\n\n3. Figures (e.g., cluster diagrams and singular-value ratios) effectively convey geometric intuition and support claims.\n\n4. Addresses a critical bottleneck in scaling LLMs to longer contexts, thus of broad interest to both researchers and practitioners.", "weaknesses": "1. While the geometric intuition is appealing, the paper lacks a mathematical analysis of how RoPE’s rotation frequencies affect cluster stability. The mechanism by which RoPE transforms i.i.d. token embeddings into clustered structures and subsequently causes cluster dispersion under out-of-distribution (OOD) conditions remains insufficiently explained.\n\n2. Figure 2 analyzes cosine similarities, yet attention operates on dot products. This mismatch raises concerns about whether the reported geometric observations truly reflect the behavior of the attention mechanism.\n\n3. The so-called “Unified Theory of RoPE Attention” Section 3.2.2 primarily summarizes prior empirical findings rather than providing a formal theoretical analysis. The term “theory” may therefore be overstated.\n\n4. The proposed RoPE-ID method, which applies RoPE to only half of the channels using higher frequencies, does not convincingly demonstrate that it can effectively mitigate the identified issues. The paper does not clearly explain why this configuration mitigates key/query dispersion, and the presented results suggest the phenomenon is not fully resolved. Moreover, the authors should further analyze the trade-off that higher frequencies blur local positional distinctions.\n\n5. The authors should include ablation studies on the proportion of channels to which RoPE is applied and on the chosen frequency range parameters, to better justify these design choices.\n\n6. Figure 5 lacks proper x-axis labeling, and the notion of “time” used in the caption is ambiguous. Additionally, many figures appear visually unpolished and would benefit from clearer annotations, consistent axis scaling, and improved aesthetics to enhance interpretability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RHMG2SsIkL", "reviewer": "Reviewer_AmPV", "summary": "Transformers with RoPE fail on inputs longer than training length. The paper observes that keys and queries form separated clusters, enabling \"sink tokens\" to absorb default attention. RoPE's low-frequency channels rotate slowly, barely moving during training. Beyond training length, they reach \"out of distribution\" rotation angles, causing clusters to overlap and breaking sink token functionality. RoPE-ID applies high-frequency RoPE to half the channels (completing full rotations during training), keeping the other half stable. This maintains cluster separation while staying in-distribution, enabling 16× context generalization without fine-tuning.", "questions": "- In Figure 3, could you align the scaling of the x and y axis within the 4k and the 64k lengths? \n- Could you add to the tables what the numbers are? Maybe the unit?\n- Why alternating quarters? I understood that 1 and 3 are rotated, leaves 2 and 4 stable. Why not just first half and second half? Or more finegrained interleaving? Or could simpler patterns work just as well?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Overall, thorough analysis, then simple solution. Awesome! Great Science!\n- Reveals that attention uses separated key/query clusters (opposite of conventional wisdom), connects RoPE mechanics, attention geometry, and sink tokens into one elegant explanation, slow-rotating channels reach unseen angles beyond training length, destroying cluster separation\n- Zero-shot method that matches or exceeds YARN. RoPE-ID is basically \"use high frequencies on half the channels\", trained on 4k, works on 64k with no tuning, conceptually cleaner than previous method\n- Super great presentation. Clear and meaningful figures, great writing. Makes complex geometry intuitive", "weaknesses": "- The last paragraph of the intro is hard to parse, though easy to understand after reading the paper. \n- The tables could emphasize more that RoPE-ID is zero shot.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KxL7j6IsbC", "reviewer": "Reviewer_v9An", "summary": "This paper analyzes why Rotary Positional Embedding (RoPE) fails beyond its training context and attributes the issue to geometric degradation in attention. The authors show that key and query vectors form tight clusters and rely on a “sink” token for stability, but RoPE distorts this structure at long ranges. They propose RoPE-ID, which applies high-frequency RoPE to only part of the channels to preserve cluster geometry without fine-tuning. Results on 1B and 3B models show improved long-context performance on LongBench and RULER benchmarks compared to standard RoPE baselines.", "questions": "an you define a quantitative “cluster overlap” metric (e.g., silhouette score, Davies-Bouldin index) and show its correlation with sink attention and downstream scores?\n\nWhat is the empirical sensitivity to the RoPE channel ratio (e.g., 25%, 37.5%, 62.5%, 75%) and how does the trade-off between positional fidelity and cluster preservation behave?\n\nPlease provide a direct empirical comparison to Hope under identical experimental settings to clarify the regimes where RoPE-ID’s training-free advantage holds.\n\n\n*(Optional) Does RoPE-ID scale to 7B+ models and much longer contexts (100k–1M tokens)? How do key/query cluster properties change with model scale?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Introduces a clear geometric explanation for RoPE’s long-context failures, supported by multiple complementary analyses.\n\nProposes a simple, low-cost remedy (RoPE-ID) that requires no fine-tuning, facilitating rapid deployment.\n\nDemonstrates robust empirical evaluation across tasks and baselines and provides reproducibility details.", "weaknesses": "Theoretical analysis stops short of giving formal bounds that relate rotation frequency to cluster overlap or performance.\n\nExperimental scope is limited to 1B/3B models and up to 16k context; applicability to 7B+ models or 100k+ contexts is not shown.\n\nKey hyperparameter choices (channel ratio, cycle length, temperature coefficient) lack comprehensive ablation or principled justification.\n\nBaseline comparisons are incomplete—direct controlled comparisons to Hope[1], and other recent methods under identical settings are missing.\n\n[1] Chen, Y., Lv, A., Luan, J., Wang, B., & Liu, W. (2024). *HoPE: A novel positional encoding without long-term decay for enhanced context awareness and extrapolation*. arXiv preprint arXiv:2410.21216.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x933FFKdNb", "reviewer": "Reviewer_9pc3", "summary": "The authors propose RoPE-ID, a rotary positional embedding (RoPE) based positional embedding method that scales a proportion of head-specific channels to ensure that long-context inputs are more in-distribution relative to the training data. The authors show that this method is competitive with existing long-context extension techniques on numerous benchmarks on models of multiple sizes.", "questions": "- While the methodology differs, I believe that the underlying phenomena that the authors discuss relates significantly with [1]. In particular, ensuring that the low frequency dimensions complete a rotation is quite related to ensuring that longer relative distances does not appear OOD relative to what was learned during training.\n\n- Based on my understanding of the pseudocode in Appendix A.3, the scaled dimensions are hard-coded as the first quarter and third quarter of the dimensions. Why is this the case and wouldn't it make more sense to apply a more adaptive scaling method that looks at all the dimensions instead and choose a proportion of the dimensions to scale based on the frequency values?\n\n- While I'm not the largest advocate for scaling trends, given the inconsistency in the results the method would be more convincing if either results on larger models or data sizes were provided and showed a more clear trend of the advantages of RoPE-ID relative to existing methods.\n\n[1] Resonance RoPE: Improving Context Length Generalization of Large Language Models", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The authors provide a strong motivation for the method through the analysis of attention sinks and singular values. The experiments are also comprehensive and show some cases of RoPE-ID being beneficial compared to alternative methods.", "weaknesses": "- The contribution itself remains somewhat trivial; relative to the changes with standard RoPE, it appears that the primary difference is only that low frequency RoPE dimensions are individually scaled such that they can complete the rotation within the provided training length. \n\n- There is a rather significant lack of clarity regarding the effects of the hyper-parameter choices; for example using half the dimensions in a fixed manner and/or the maximum/minimum frequencies allowed.\n\n- The experimental results are rather unconvincing of the benefits of the method due to some inconsistencies. For example, on RULER, improvements in the 1B model do not appear statistically significant compared to YaRN. Meanwhile, on the 3B model, performance on the 8K context is much better in YaRN but then decreases for 16K. For LongBench, YaRN outperforms RoPE-ID on the 3B model but not the 1B. Thus I cannot gauge exactly what the trend will be for either larger models or more data.\n\n- These models all require training from scratch; this is a rather strong limitation. Additional baselines should include NTK-aware scaling methods, which the authors mention in the related work but do not compare with on downstream tasks. Given the additional complexity of pre-training from scratch, it is further important to compare against methods that calibrate or adjust scaling factors rather than require pre-training new models from scratch as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)