Frequency Bands in RoPE: Base Frequency and Context Length Shape the Interpolation–Extrapolation Trade-off
Authors: Yui Oka
Itsumi Saito
Kyosuke Nishida
Kuniko Saito
Venue: ICLR 2026 Poster (2026)
Keywords: Rotary Position Embedding
Position Interpolation
Extrapolation
Large Language Model
Primary area: foundation or frontier models, including LLMs
TL;DR: We show that the location of the dominant frequency band is governed jointly by the base and the training sequence length​.

Abstract: Rotary Position Embeddings (RoPE) are widely adopted in LLMs, and it is commonly believed that larger base frequencies $\theta$ yield better long-context performance. In this paper, we show that a high-norm RoPE dimension, referred to as the “frequency band,” consistently emerges across multiple models, and we focus on this band to reveal the trade-offs inherent in RoPE. We find that replacing the RoPE dimensions below the frequency band with NoPE during inference has little effect on performance, indicating that these lower-frequency dimensions are only weakly utilized. We further find that the location of the frequency band depends on the RoPE base $\theta$ and the training sequence length. Moreover, the band forms early during pre-training and persists even after context extension via position interpolation. 
Notably, we show that setting $\theta$ to the training length shifts the band toward lower frequencies and improves extrapolation, whereas increasing $\theta$ enhances interpolation but reduces extrapolation, revealing a clear trade-off between interpolation and extrapolation.
We believe this work is a step toward a sharper understanding of positional embeddings in LLMs, with falsifiable diagnostics and practical guidance for choosing $\theta$ that support scaling to longer contexts.

Published as a conference paper at ICLR 2026
FREQUENCY BANDS IN ROPE:
BASE FREQUENCY AND CONTEXT LENGTH SHAPE THE
TRADE-OFF
Yui Okal-2, Istumi Saito", Kyosuke Nishida', Kuniko Saito' 1 Human Informatics Labs., NTT, Inc.
Tohoou University yui. okaentt.com
ABSTRACT
Rotary Position Embeddings (RoPE) are widely adopted in LLMs, and it is com- monly believed that larger base frequencies 0 yield better long-context performance. In this paper, we show that high-norm RoPE dimension, referred as the "fre- quency band," consistently emerges across multiple models, and we focus on this band reveal the trade-offs inherent in RoPE. We find that replacing the RoPE dimensions below the frequency band with NoPE during inference has little effect on performance. indicating that these lower- freeuffryyeecc dimensions are only weakly utilized. We further find that the location of the frequency band depends on the RoPE base and the training sequence length. Moreover, the band forms early dur- ing pre-training and persists even after context extension via position interpolation. Notably, we show that setting to the training length shifts the band toward lower frequencies and improves extrapolation, whereas increasing enhances interpo- lation but reduces extrapolation, revealing a clear trade-off between interpolation and extrapolation. We believe this work is step toward a sharper understanding of positional embeddings in LLMs, with falsifiable diagnostics and practical guidance for choosing / that support scaling to longer contexts.
INTRODUCTION
Rotary Position Embedding (RoPE) (Su et al., 2023) is widely adopted positional encoding method in Transformer-bact based large language models (LLMs). It can provide an awareness of relative position via two-dimensional rotations determined by base frequency parameter, denoted 0 hereinafter. To support longer input sequences, recent work has scaled the base frequency well beyond its default setting of 10, 000, typically up 500, 000 more (Grattafiori et al., 2024; Abdin et al., 2024). This approach is motivated by the intuition that higher base frequencies alleviate sharp decay in attention scores over relative distances (Xiong al., 2024; Rozière al., 2024) well by the aim of achieving extrapolation to unseen longer contexts ( (Vaswani et al., 2017). However, previous research shows that scaling only RoPE's often fails to yield robust extrapolation (Oka et al., 2025), and thus position interpolation with fine-ture tuning (Peng et al., 2024; Ding et al., 2024) remains necessary to recover performance in extended contexts.
Furthermore, Barbero et al. (2025) observed clear "frequency bands" in the low-frequency dimension of queries and keys, where a frequency band refers dimension in which high L2-norm values occur for all tokens. However, the formation of this band has not been verified. They also showed that replacing some of the low-frequency dimensions RoPE, corresponding to the largest 0, with NoPE (Kazemnejad et al., 2023) does not affect the performance of LLMs. These results suggest that such low-frequency RoPE dimensions are nearly identical to NoPE and may not represent positional information. Figure 1 illustrates segment of the sine wave in using RoPE. As the value of 0; increases with 0 = 500, 000, the sine components approach zero and the cosine components approach one across most positions, effectively resulting in matrices that closely resemble the identity matrix. Such a lack of significant variation the encoded values may underlie the phenomena discussed above.

---

Published as a conference paper at ICLR 2026
8=500000
(FMROPE)
j-10
j+03
0
100
200
300
400
500
200
400
500
Figure 1: Sine waves of base frequencies 8, in RoPE and a frequency-matchine intervention in RoPE (FMRoPE), with training context length Luain = 512. FMRoPE sets the maximum base frequency to match the maximum sequence length in pre-training.
Theoretical reasons for 0-scaling via activation decay (Xiong et al., 2024) conflict with evidence that swapping low frequencies for NoPE leaves performance unchanged (Barbero et al., 2025), revealing deeper puzzle in RoPE's A choice. These previous studies present a fundamental challenge to the prevailing 8-scaling paradigm: Does increasing truly add useful positional information or does it mainly push many RoPE dimensions into a NoPE-like form that contributes little information? this paper, we focus on frequency band analysis and reveal that the relationship between 8 and context length from the frequency band is much closer than previously assumed.
We first present evidence that frequency bands emerge systematically across different LLMs, in- cluding Gemma (Gemma Team et al., 2024), Llama (Touvron et al., 2023; Grattafiori et al., 2024), Qwen (Yang et al., 2025), and Phi-3 (Abdin et al., 2024), and that their formation is governed by the interaction between 0 and the training context length Luzann This formation is determined in the early stages of training and persists even when applying position interpolation, including YaRN (Peng et al., 2024) and LongRoPE (Ding al., 2024)-in in fact, the formation is inherited rather than corrected. Most critically, we study a geqquency-matching the training length. This shifts the frequency band toward the lowest frequencies and reveals a clear trade-off: Setting 0 to the training length improves extrapolation but hurts interpolation, whereas using larger base frequencies has the opposite effect. This trade-off contradicts the prevailing notion that simply scaling @ is universal solution for context extension.
intervention in RoPE that sets the base frequency to
Through extensive analysis, we provide an answer to the research question posed above: Increasing does not by itself add useful positional information; rather, it mainly reallocates energy that the dimension below the frequency band preserves positional information, while many dimen- sions behave similarly to NoPE and contribute little. This improves interpolation within the training range but degrades extrapolation. Therefore, rather than treating d-scaling as universally beneficial, emphasize the importance of considering the frequency band and the coppolatiiootxttplllaaiion trade-off.
BACKGROUND
Rotary Position Embedding (RoPE) RoPE (Su et al., 2023) incorporates positional information directly in the self-attention mechanism by rotating the query and key vectors. The d-dimensional 2 subspaces, and the inner product the rotation matrix and the query is calculated follows.
space is divided into d
cos m
0 sin
D.
T 22
0, = 02//d
sin
m 8
m cos
e
where n is absolute position, qdh E R 1xd is the m-th query (0 S m < L. 1) when the number of dimensions is d, is the dimension (i E (1,2.... 1)), 0 is the base of RoPE, and L is sequence length. The same process is also performed for the n-th key hn e 1x The base 0 in RoPE is relatively large and designed to represent positions exceeding the sequence length appearing during training. These positions include 0 = 10, 000, which is based Sinusoidal Positional Encoding (Vaswani et al., 2017) and used in the Gemma (Gemma Team et al., 2024) and Llama-2 (Touvron al., 2023) models, 0 500, 000, which used in the Llama-3 model (Xiong et al., 2024), and 0 = 1, 000, 000, which is used in the Phi-3 model (Abdin et al., 2024).
NNte that the pretrained L.LMs in Section 3 use 0g 024/4 (0. ..... 1.... definition.
1}. unlike the standard
2

---

Published as a conference paper at ICLR 2026
Position Interpolation RoPE requires fine-tuning to handle sequences longer than the maximum sequence length Luain appearing in pre-training. The most common approach to this fine-tum tuning is a position interpolation method that further expands the 0 used in pre-traine vre-training and it includes YaRN (Peng et al., 2024), which determines with a rule-based approach, LongRoPE (Ding et al., 2024), which searches for the most suitable f using parameter optimization, and Llama-scaling 2, which is rule-based approach used in the Llama-3.1 model (Meta, 2024)'.
Frequency Bands in RoPE Barbero et al. (2025) revealed that there are "frequency bands" with high continuous norm values for the 2-norm ||g" ||2 and ||k ||2 of the query and key after applying RoPE, where qth E Rixd the m-th query and k" E Rixd the n-th key when the number of dimensions is d. Furthermore, they also revealed that pretraining while replacing the low- wwwfrequency dimension RoPE with NoPE (Kazemnejad et al., 2023) does not change performance. This method called p-RoPE, where is parameter that turns the dimension into NoPE. However, their analysis focused on short texts and did not verify cases of positional interpolation or long context. Moreover, the mechanism behind the formation of the "frequency bands" remains unclear.
FREQUENCY BAND EMERGENCE IN PRETRAINED LLMS
We first investigate the frequency band identified by Barbero et al. (2025). Do similar frequency bands appear in other LLMs, or in those with base 0 modified by position interpolation? To address this. build on prior analysis (Barbero et al., 2025) and conduct further investigations across several LLMs..
3.1 ANALYTICAL METHODOLOGY
To measure the usage of frequencies, Barbero et al. (2025) calculated the 2-norm of key ||** [|2. By the Cauchy-Sehwarz inequality, the attention score Om,, between the mth query q"h and the rith key k" satisfies (aTh. kn) < ||q""||a||<"] Therefore, to analyze the frequency components influencing the attention score, it is sufficient examine either ||q""||2 or ||k" 1|2. We mainly examined the
2-norm queries. Here, the 2-norm of a key is calculated as ||A=][|2 = (
where
the number of dimensions and j E {1, (1.,...,,
Frequency Band Index itand To quantify where the frequency band appears in the key vector dimensions, we define the band index band. First, we identify the dominant frequency component at token position n by selecting the dimension with the maximum 2-norm among the dimensions of the key vector k.
id
argmax
(1A)112) 1
&. E (Ag (Ag
Next, we determine the index idx, that appears most frequently in the entire sequence of length L. The resulting index idx represents the dominant dimension in which the frequency bands are concentrated throughout the entire sequence.
idx
argmax (count(idxx.) An E A-
This procedure is repeated for all heads and layers. The average of these model indices is defined as the band index ihand. where 0 < ihand < d
2
To analyze the contribution of different frequency components RoPE, we measured per- plexity using simplified RoPE called P-RoPE (Barbero et al., 2025), which disables low- ww-frrquency dimensions. P-RoPE applies rotation only to the top-r high-frequency dimensions, interpolating between NoPE (r 0) and the full RoPE (r 1).
P-RoPE
Unlike the previous studies of Barbero et (2025), no training was conducted in our analysis.
Praasformeer/mmotelinggno catpiiiiinn_roe_uuttli.._oo prre_utilsspyy.335
These major position interpolations all enlarge the original 0 values, shown in Appendix
3

---

Published as a conference paper at ICLR 2026
High Prequency
Low Frequancy
High Prequency
Low Frequancc
High Frequency
Low Frequancyy
High Frequency
Low Frequanccy
Llama-2-7B
gemma-7b-it
"arn-Llama-277-64k
Llama-3.1-88
Phi-3-small-12
Qwen3-8B
Figure 2: 2-norm plotted over 2-dimensional chunks of queries. Vertical axis represents sequence length (T. 4096), and horizontal axis represents each dimension index (i {0. 1........ d/2 - } of RoPE. Note that the head dimension for the Gemma model 256, while is 128 for other models.
Table Perplexity Results with p-RoPE. 'pt" 'Pre-train' and 'ft' is "Fine-tuning. YaRN, Llama3, and LongRoPE are position interpolation methods applied during fine-tuning. Note that head dimension is 256 for the Gemma model and 128 for the other models.
Model
Luain
base
Band Index
Perplexity with p-RoPE
pt
pt
Aband
bmndd d
r=1.0
r=0.9
r=0.75
=0550
Gemma
8k
10000
116.68
0.91
2.52
2.70
81.66
100
Owen3
40k
1000000
51.04
0.79
6.22
6.22
'622
7.46
[lama-2
4k
10000
53.33
0.84
2.34
2.58
= 100
>> 100
+YaRN
4k
64k
10000
51.93
0.81
2.81
5.08
100
100
Llama-3
500000
-4.43
6.63
2.29
2.29
2.29
84.30
+Llama3
8k
131k
500000
40.47
0.63
2.29
2.29
2.29
5.53
Phi-3
8k
1000000
36.67
0.57
2.84
46.11
46.36
100
+LongRoPE
131k
1000000
39.32
0.61
2.74
62.20
62.18
100
3.2 EXPERIMENTAL SETTINGS
For comprehensive analysis, selected models that use different base models (Gemma 8B, Llama- 2 7B, Llama-3 8B, Phi-3 Small, Qwen-3-8B) and different position interpolation methods (YaRN, scaling in Llama-3 model, LongRoPH). Additional details are given in Appendix A. The dataset for evaluation is the test set of Wikitext- (Merity et al., 2017) 7),, and the sequence length in inference is 4096 for all models.
3.3 RESULTS
Do frequency bands exist in other LLMs? Figure 2 shows the 2-norm of the queries for each model. As with Barbero al. (2025), we extracted queries the first layer that had semantic attention patterns in the head. First, we found that bands exist in all models, indicating that bands reflect phenomenon that occurs generally. Next, we observed that the dimension which the frequency band appears varies across models. Furthermore, found that the position interpolation model inherits the bands regardless of the position interpolation method.
Do low-frequency components of RoPE contribute to performance? Table 1 shows frequency band index bband and perplexity results when varying parameter r in p-RoPE across multiple language models. We also present standardized band index isend / (divided by head dimension d) for unified comparison. Band index bhand remains largely unchanged before and after position interpolation. and it aligns closely with the index of the bands shown in Figure 2, confirming consistency between
4

---

Published as a conference paper at ICLR 2026
High Frequercyy
Low Frequancy
High Frequercyy
Low Frequancy
High Frequarcy
Low Frequancy
High Frequancy
Low Frequancy
Toton
mm1x 512, 8e512
512y 6=10000
512 8=50000
Lmax 6m 1000000
Tiken
Lmax-102M 1024, 0-1024
1024 0-10000
Lmaa-1024, 1024, 0-500000
Lmsg=1004, 0-1000000
Toson
Lm44-20-6 0-10000
Lmua-2066, 0-500000
L048-248, 3-1000000
Figure 3: 2-norm plotted in the combination pattern of (Lurain 0) E [512, 2048) x
(Lrrain 10, 000; 500, 000; 1, 000, 000). Vertical axis represents sequence length (L 1024), and horizontal axis represents each dimension index (i E {0, 1, dd2])) of RoPE.
visual and quantitative analyses. The standardized index bband/ decreases as 9 increases, suggesting relationship between band location and frequency determined by 0. For the Gemma and Llama models, the p-RoPE results reveal that replacing RoPE in a frequency dimension lower than the band with NoPE does not degrade performance, indicating an ineffective use of fow-frequency RoPE components. Conversely, Phi-3 shows performance degradation when low-frequency dimensions are replaced, regardless of band appearance, suggesting an effective use of low-frequency RoPE, possibly due to this model's ''s block-sparse attention (Abdin et al., 2024) that alternates between dense and sparse patterns.*
In other LLMs and models that use position interpolation, a distinct frequency band appears and remains even when the base changes. Since replacing RoPE dimensions below this frequency band with NoPE shows no measurable change, these low-frequency dimensions might not contribute to performance.
Takeaways from Section 3:
UNDERSTANDING FREQUENCY BAND FORMATION IN PRE-TRAINING
What factors cause the band index to change, and when do bands occur? investigate the factors that determine bands, we varied RoPE's 0 and max sequence length in pre-training to analyze the frequency bands via the 2-norm the query.
4.1 EXPERIMENTAL SETTINGS
For pre-training, we followed the experimental settings of Press et al. (2022) and Oka et al. (2025), and we used the WikiText- 103 dataset (Merity et al., 2017). A comparative evaluation was made using Transformer-based language model (Baevski & Auli, 2019). Here, the dimensionality of the word embedding dmoode is 1024, the number heads is 8, the dimensionality of the heads d is 128, and the number of layers is 16. This implementation uses the fairseq (Ott et al., 2019)- based code. Additional details on the parameter settings are given in Appendix A. The maximum sequence length and RoPE were tested in combination with (Lirain, 0) {512, 1024, 2048} x [Lrain, 10, 000; 500, 000; 1, 000, 0003. The sequence length in inference is L = 1024 for all models.
A detailed discussion of this observation appears in Appendix D.
5

---

Published as a conference paper at ICLR 2026
Table 2: Band index and perplexity with p-RoPE when sequence length in pre-training is L {512, 1024, 2048}.
Base in RoPE 0
Band Index
Perplexity with p-RoPE
L.
Train
Inference
Rband
Aband/
r=
r=0.90
r=0.75
r=0.50
r=0.25
512
512
512
60.5
0.94
19.58
20.18
24.28
35.11
98.26
10000
10000
30.12
0.47
19.39
19.39
19.39
22.71
63.59
500000
500000
17.00
0.26
19.35
19.35
19.35
19.35
34,46
1000000
1000000
15.37
0.24
19.36
19.36
19.35
19.35
30.59
1024
1024
1024
60.25
0.94
20.07
20.19
22.37
32.39
101.97
10000
10000
46.12
0.72
19.53
19.53
19.53
21.41
68.54
500000
500000
18.25
0.28
19.55
19.55
19.55
19.55
34.22
1000000
1000000
11.12
0.17
19.59
19.59
19.59
19.59
31.09
2048
2048
2048
60.50
0,94
21.49
20.99
21,51
29.56
94,90
10000
10000
52.12
0.81
19.73
19.73
19.73
20.97
69.13
5000000
500000
16.62
0.25
19.71
19.71
19.71
19.73
34.98
1000000
1000000
11.62
0.18
20.06
20.06
20.06
20.04
31.57
High
Lowe
High
High
Frequavay
OOTTED
Epoch
Epach 2
Epoch 5
Epach 4
Epoch 10
Epoch 80
Eposh 100
Epoch 150
Epoon 800 (Lasl)
Figure 4: Plot of the 2-norm for each epoch. Vertical axis represents sequence length, and horizontal axis represents each dimension index (i E {0. (0.1,...,d d/2)) of RoPE.
4.2 RESULTS
What factors cause the band index to change? Figure 3 shows the 2-norm map in the combination pattern. We output 2-norm maps of queries from the semantic attention head, following Section 3. First, when theta values are fixed, the index at which the band exists increases as the maximum sequence length during pre-training increases (from top to bottom of Figure 3). This suggest that the index at which the band exists depends on the maximum sequence length during pre-training. When the maximum pretraining sequence length is fixed and 0 is increased (10, 000 500, 000 + 1, 000, 000; from left to right in Figure 3), the dominant frequency band shifts toward the lower dimensions. However, the difference between similarity between the two values likely arises because both settings are already high, so further increases in provide little additional shift. Furthermore, when theta values were matched to the maximum sequence length during pre-training, it was found that the position of the band was near the maximum index for the head dimension.
500, 000 and
1, 000, 000 is marginal; this
Band index and R-RoPE We also investigate the band index iband and P-RoPE. The results when sequence length Lusin = 512 are shown in Table 2. As demonstrated in Section 3, increasing lowers the band index (i.e.. shifts it to higher frequencies), and replacing RoPE with NoPE below this band has little impact on performance. Therefore, the frequency- band characteristies identified in Section 3 are expected hold irrespective of model scale and training corpus.

---

Published as a conference paper at ICLR 2026
When do bands occur? We also investigated the stage when the band first appears. Figure 4 shows the key 2-norm for each epoch in the model with Luain set to 512 and set to 10,000. At epoch the band does not exist, and the distribution appears to be mixed with noise, but at epoch 6, the band appears from an early stage. This band is maintained until the final epoch. After the frequency band first emerges, its position remains stable throughout the remainder of training. Therefore, that the band does not exist in the first stage but is still acquired by the model at an early stage during training, Epoch 6 is stage of rapid initial convergence, during which we can see that the model acquires the band.
The effective dimension of RoPE is determined by the pre-training theta value and maximum sequence length, since these factors shape the band. The band emerges early in pre-training, suggesting it is a fundamental feature learned by the model.
Takeaways from Section 4:
DERIVATION OF FREQUENCY BANDS
As explained above, it has been found that the frequency band depends on the maximum sequence length and the basis. However, the mechanism itself is the core issuc. This section provides theoretical analysis to address this question. To probe the mechanism forming the frequency band, reduce the problem to constrained optimization and state our guiding question: Under fixed variation? As a simple and
coefficient-norm budget, which 0, allows the largest informative proxy, we maximize the coordinate variance of cos(me) over the window.
5.1 DERIVATION
We derive which RoPE pair in the query tends concentrate energy during training, using only the maximum training sequence length Luain and the RoPE base 0. To make the argument beginner-frienn we work with the variance of single coordinate the sinusoidal basis,
Our Goal
V(x) =
Valm-Unndfoo,mmll [ossm))]]
x warannn
and choose the frequency that maximizes (xx). Section H explains the connection to the full covariance view.
Step 1. Let m ~ Uni Turain] and define = w Trrain By direct integration,
sin x
1 +
sin sin(2r)
E E[cos(mw:)]
E Eloos (mw)l [cos)(mw)]
(4)
x
2
4x
Hence, the centered variance
sin (2x)
(sin
V(z) [arl))s(mw)]
(5)
4r
This function captures how much the coordinate moves across the position window. As -+ cos(mw) is almost constant and V(x) -> 0; as x + co, oscillations average out and V(x) - 7.
Step 2. Differentiating Eq. (5) gives
V'(x) = 2x cos(2x) - 5x sin (2x) + 8 8sin
(6)
4
Stationary points satisfy V V'(x) 0, i.e.,
2n cos (2x) - 52sii(2x)) + sinn x = 0.
(7)
Solving Eq. (7) numerically yields the smallest positive root
3.657210 rad
(i.e., x" /(2m) Ro 0.582 cycles).
(8)
Here, we checked that (w) unimodal on (0, oo) and that Eq. (8) gives the global maximum with V(x*) x 0.54047 3
7

---

Published as a conference paper at ICLR 2026
The continuous optimizer has angular frequency w = =xx/ Luain- We select the RoPE pair whose grid frequency wy 0- 2j/d is closest to w", which yields the closed-form predictor
Step 3.
d
Luain x
loga
x= Ri 3.657210
2
is rounded to the nearest integer; the corresponding physical dimensions are (2j"; 2j +1).
5.2 DERIVED BAND LOCATION
The results of calculating jr and Aband in Section 3 for each model are shown in Table 3. The relationship between j and Aband can be expressed as an approximately linear scaling Aband R c X j with с ~ 1.1. This indicates that once the energy-concee serggycconcentrating dimension j* is determined, the corresponding physical frequency band ihand is essentially fixed. The small variation observed across models is likely due to differences in the query distribution rather than the model architecture. Accordingly, we proved that the position of the RoPE frequency band is predetermined by RoPE base 0, training length Larinn and dimension d.
Table 3: Results of j and iband
Model
j
iband
Gemma
107
116.68
Llama-2
49
53.53
Qwen3
43
51.04
Llama-3
38
43.43
Phi-3
36
36.67
B Leroin
59
Furthermore, we calculated j* when 0 = Ltrein 8192 and = 128. Here, j = 59, and c 1.1
yields e x jstar 64.9. matching the model's RoPE pair count ( 64). Thus, for Lurain the band is expected to be concentrated around the 59th dimension toward the lowest-frequency dimensions.
5.3 CHECKING THE PREDICTED FREQUENCY-BAND POSITION
We examine whether the theorem derived in the previous section, which predicts the band location, generalizes to other choices of & and context length. The experiments in Section 4.2 were conducted with 0 512, 1024, 2048, where the maximum training length L was set equal to each value of O. We then compare the resulting empirical band positions iaad in Section 4.2 with the theoretically predicted coordinate j" The results of calculating j* and iband in Section 4.2 for each model are shown in Table 4. The relationship between j* and band can be expressed as an approximately linear scaling Aband C j" with 1.0.
Table 4: Results of j and ibond in
Section 4.2
L
j
512
512
59
60.50
1024
1024
59
60.25
2048
2048
60.50
Takeaways from Section 5: Using x² R 3.657210, d, TArains and 6, we can predict the frequency band location advance. When 0 Ltroins the frequency band theoretically predicted to lie at the lowest frequency dimensions.
FREQUENCY-MATCHINE INTERVENTION IN ROPE
Interestingly, our analysis results suggest that tribute to model performance (Section 3). However, since the frequency band is set by 0 and Latran during pretraining (Sections 4 and 5) and remains stable even with interpolation (Section 3), a natural question arises: What the impact on model performance when the frequency band shifted toward lower frequencies during pretraining? To explore this, analyze a strategy we term frequency- matching intervention in RoPE (FMRoPE). where we set the base frequency parameter / to the maximum sequence length Lusia used during pretraining. As demonstrated in Sections 4.2 and 5, this setting shifts the frequency band toward the lowest frequencies, allowing the model to leverage a broader and more effective frequency range from the start of pretraining.
dimensions beyond this band con-
8

---

Published as a conference paper at ICLR 2026
Table Perplexity results from Section 5. 'pt' stands for 'Pre-train' and 'ft' stands for "Fine-tuning' in context extension with position interpolation. 'YaRN" is a position interpolation method applied during context extension. The gray area represents the FMRoPE score.
Tuain
Base in RoPE 8
Sequence Length in Inference
ft
Train
Inference
512
1512
2512
3512
15512
25512
512
512
512
19.58
21.19
24.20
27.42
84.75
100
512
512
1512
20.02
19.09
21.40
24.00
72.19
>100
512
512
3512
21.28
20.27
20.37
23.00
66.10
>100
Pre-train
512
10000
10000
19.39
43.63
84.45
>100
>100
>100
512
500000
500000
19.35
40.39
77.90
>100
>100
>100
512
1000000
1000000
19.35
37.94
74.26
>100
>100
>100
5T2
1312
T512
1312
19.62
17.78
17.56
17.65
20.31
23.19
512
1512
1512
3512
19.38
17.99
17.66
17.64
19.93
23.44
Fine-tuming
512
1512
1512
15512
21.00
19.74
19.53
19.48
20.51
22.41
with YaRN
512
1512
10000
10000
19.10
17.84
17.75
18.37
52.59
85,88
512
1512
500000
500000
19.14
17.89
18.83
18.34
35.57
50.88
1512
1000000
1000000
19.07
17.76
17.81
18.72
66.89
>100
6.1
METHODOLOOGY
In FMRoPE, set the RoPE base equal to the training context length: L Here, Lirain denotes the maximum sequence length used during pretraining or fine-tuning. For example, we use 512 during pretraining and 1512 during interpolation-Net based fine-tuning.
6.2 EXPERIMENTAL. SETTINGS
We conducted small-scale pre- pre-iigrning and context-extension experiment, following the experi- mental settings of Press et al. (2022) and Oka et al. (2025) as in Section The maximum sequence length during pre-training is T-raan 512, and set 0 512. In context extension through position interpolation, we adopted YaRN (Peng et al., 2024), which is the most commonly used standard method for position interpolation. The maximum sequence length for context expansion with position interpolation is Liain = 1512. Additional details on the parameter settings can be found in Appendix A. We used perplexity as the evaluation metric."
6.3 RESULTS
Pre-train We begin with the results above the dashed line in Table 5, corresponding to models without YaRN-based fine-tuning. When using conventional RoPE and FMRoPE without modification, the conventional RoPE outperforms FMRoPE. However, we observe that FMRoPE achieves better extrapolation performance. The analyses of Sections 3, 4, and suggest that as more low wwwffreuency dimensions behave like NoPE, larger values (0 > 10, 000) may reduce RoPE's contribution in longer contexts. In particular, the inference- (e.g., 1512 or 3512), thus significantly reducing perplexity. While FMRoPE demonstrates strong extrapolation, the requirement knowing the target sequence length at inference time poses practical limitations. Future work should explore dynamic or adaptive schemes for adjusting based on observed context.
is adjusted to match the target sequence length
Context extension We next examine the results below the dashed line in Table 5, corresponding to models fine-tuned with YaRN for position interpolation. FMRoPE underperforms conventional RoPE in short contexts, suggesting that FMRoPE is particularly effective congecontext or extrapolation settings but not in interpolation. FMRoPE outperforms conventional RoPE in extended sequences, the FMRoPE experiment using YaRN, we found that similar trade-
achieving lower perplexity. offs to those observed in the pre-train experiment occurred. However, as shown in Section 3, we believe this result can be expected because the frequency bands are preserved even when positional interpolation is applied.
*Comparisons with other position encodings were also conducted (Appendix F). We additionally validated approach 1B-parameter model with longer contexts and evaluated downstream tasks (Appendix G).

---

Published as a conference paper at ICLR 2026
Takeaways from Section 6: Matching 0 to the training length, which shift the frequency band into the lowest dimension, improves extrapolation but hurts interpolation, and this trade- off persists under position interpolation such as YaRN. Larger & makes more low-frequency dimensions behave like NoPE, which may reduce RoPE's contribution in extrapolation.
7 RELATED WORK
The base # in Sinusoidal PE (Vaswani et al., 2017) was set to 10, 000 for the purpose of enabling theoretical extrapolation. Meanwhile, Takase & Okazaki (2019) demonstrated that LRPE, which sets the base 0 of SPE to the sequence length, provides robust control of output length. The 0 setting adopted in this study is consistent with that setting.
RoPE's f component has been redesigned to support context expansion with fine-tuning, including rule-based expansion of I (Peng et al., 2024; bloc97, 2023) and learning-basse -based or search-based frequency scaling (Chen et al., 2024; Ding et al., 2024). Furthermore. Xiong et al. (2024) reported that setting 500, 000 during pre-training suppresses the rapid decay of attention scores between distant tokens. However, all of these methods tend to increase 6, regardless of the maximum context length pre-training. Liu et al. (2024) showed that using smaller 0 (e 500) during pretraining improves extrapolation, but they did not analyze its relationship to the pretraining sequence length. In contrast, Xu et al. (2024), focusing on nearby tokens and ignoring distant context, found that such models achieve lower perplexity while still exhibiting "superficial extrapolation." Furthermore, their theoretical analysis suggests that the base frequency of RoPE governs the model's 's capacity to handle context length, which aligns with our findings. Barbero et al. (2025) identified RoPE frequency bands and linked them to positional heads. They also challenge the common "distance-decay" narrative and propose a modified RoPE variant.
While our visual observations overlap with Barbero et al. (2025), the core scientific questions and contributions differ substantially. We explain where the frequency band appears (Section 3) and how its position depends on the RoPE base 9 and the training length Ltruin (Section 4, and 5). In addition, our experiments reveal that modifying 0 shifts the band toward higher or lower frequencies, and this shift leads to clear interpolation- -etttiooo-ettaaoaattionnn trade-off (Section 6). Through theoretical analysis and controlled pre-training experiments, we identify the conditions under which bands emerge and show that their position can be predicted directly from (0, Ltrain). Our analysis also covers broader range of models. We examine multiple pretrained LLMs (Gemma, Llama, Qwen and Phi), models after context expansion via positional interpolation (YaRN, Llama-3 scaling and LongRoPE), and models using sparse attention. In multiple pretrained LLMs analysis, we use a 4096-token context window, which is substantially longer than the 20-token window considered in Barbero et al. (2025). The results show that frequency bands persist after positional interpolation, while sparse attention changes the behavior of P-ROPE. Appendix D discuss why sparse attention is the only setting that produces a different trend. Finally, our study reveals a connection between 0 and Ltrain We show that modifying the value of 0 induces trade-off between interpolation and extrapolation performance.
8 CONCLUSION
We first showed that RoPE forms a distinct frequency band that appears across LLMs, persists after position interpolation, depends on the base 0 and the training length Liminn and emerges at early stage. Low-frequency dimensions below this band often act like NoPE and add little to performance. We derived a simple predictor by maximizing a variance proxy, yielding x* se 3.657210 and a grid index j* that matches the observed band. At this point, it was theoretically understood that setting 0 to I Turai would position the frequency band near the minimum frequency. Through our experiments, we found that setting 0 to Laran shifts the band to the lowest frequencies and widens the useful range, improving extrapolation while degrading interpolation. Therefore, increasing 0 mostly reallocates energy rather than adding new positional information. As Practical guidance, choose 9 z Lacan when extrapolation is critical, and use larger 0 when interpolation within the trained range is dominant. Position interpolation should be paired with band-aware choice of & rather than applied indiscriminately. Overall, our results connect the emergence of frequency bands to 0 and Lymin and provide new perspective for band-aware design of positional encodings in long-context LLMss.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We sincerely thank Jun Suzuki (Tohoku University) for insightful discussions and constructive comments the early stage of this work.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez- Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy. Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang. Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang. Lijuan Wang. Shuohang Wang. Xin Wang, Yu Wang. Rachel Ward, Wen Wen, Philipp Witte. Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu. Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang. Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang,
Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou Phi-3 technical report: A highly capable language model locally your phone, 2024. URL https:://///.. / www.v..iiiivve/iiv//e.e//e/aaava wrg.ogg///abbo abs /tabb///4444 2404. 14219.
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In International Conference Learning Representations, 2019, URL https: net/ forum?id-ByxZX2204F..
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference on Learning Representations, 2025. URL forum?id-GtvuNNkk5B.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence, 2019.
URL https://appi sew.apissssaaattcccsesseeee wtt.eaaaiii.......oooooooooooooo org//orrpss1::222289999
bloc97. Ntk-aware scaled rope allows llama models have extended (8k+) con- text size without any fine-tuning and minimal perplexity degradation., 2023. URL https://ww. reddit allloss_Laamm_mddess_ttoo.baba
Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang. and Lidong Bing. CLEX: Continuous length extrapolation for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: /tpp///eeeeeeeeeee//eeeeee. net weet/forummiddwXSS32
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-X XL: Attentive language models beyond a fixed-length context. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics. URL https://aclantthlogg org/ P19-1285.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang. and Mao Yang. Longrope: Extending Ilm context window beyond million tokens, 2024. URL https://arxiv. wwwiii.ooggggaaee org/abs//24/0 200tb./222...3 13753.
11

---

Published as a conference paper at ICLR 2026
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Chrrstian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptise Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comaneseu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins. Samuel L. Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis. Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology. 2024. URL https://arxiv htrg/bbs/2400/ 22...4223999
Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-
Aaron Grattafiori Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits. Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, lliyan Zarov, maanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathicu Rita, Maya Pavlova, Melanic Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini. Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang. Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song,
Plawiak,
12

---

Published as a conference paper at ICLR 2026
Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierr Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape. Bing Liu, Bo Wu, Boyu Ni. Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching- -iiinggsiangggg Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao. Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang. Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery. Elconora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang. Kai Wu, Kam Hou U. Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A. Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael I.. Seltzer, Michal Valko, Michelle Restrepo. Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen. Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman. Tal Remez, Tamar Glaser, Tamara Best, Thilo Kochler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ioneseu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang. Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang. Xiaojian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL hww.https///// / http://xxvvvv...o wwwprrwwww.w.oooeoeo.....e..oere /rgp/bs//////a//// /bs 21783.
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. In Thirty-eevntth Conference on Neural Information Processing Systems, 2023. URL https:// //tpen/evieww..o/////ee/e net/forumiid= Drr12gcjzl.
13

---

Published as a conference paper at ICLR 2026
Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of adPE-based extrapolation. In The Tweifth International Conference on Learning Representations, 2024. URL http://wwwwe.eeeooooomomoo////00
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mix- ture models. In International Conference on Learning Representations, 2017. URL https:
net//ooummidddyyyii 72udxx.
Meta.
Introducing llama 3.1: Our most capable models to date. meta-1lama- 3-1/, 2024. Accessed: 2025-05-08.
Yui Oka, Taku Hasegawa, Kyosuke Nishida, and Kuniko Saito. Wavelet-based positional representa- tion for long context. In The Thirteenth International Conference on Learning Representations, 2025. URL httpp:://ppeneeviiiiee/
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)) pp. 48-53, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. URL https: / /atpa:/:o///llaahooooooyo wwwphhooooyooogg////.o. 10//N19-44000
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context win- dow extension of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: //penrevi/////evvii//ee
wwt. peet/orrmmid=wfffe xxZulu.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022. URL pott/forumidd-RssPPPM
https://openrevia
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code, 2024. URL nttps://arxizs http://aaxxi.oooarbb/////s// 2308. 12950.
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Common- sense reasoning about social interactions. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (FMNN.P-JCNNLP), pp. 4463-4473, Hong Kong, China, November 2019. Association for Computational Linguistics. URL https: / /ac//claalll aclannhllogy 10///D1914544
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations.
In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 464- 468. New Orleans, Louisiana, June 2018. Association for Computational Linguistics. URL bttps:///clliee https:://cllantholooyyy Arg/N88 2074.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https: /ttps:///xxivoooo org/ abs/2104 09864.
Sho Takase and Naoaki Okazaki. Positional encoding to control output sequence length. In Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pp. 3999-4004, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. URL http:://aclanthooooy..rgg//
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQQ: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4149-4158. Min- neapolis, Minnesota, June 2019. Association for Computational Linguistics. URL https: hrw.oaattoooogooogg/111----
14

---

Published as a conference paper at ICLR 2026
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Pumit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: http://aavvvvvvooee www.:/pgggbbbb.tee 2307. 09288.
Ashish Vaswani, Noam Shazeer, Niki Parmar, J Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Process- ing Systems, volume 30. Curran Associates, Inc., 2017. URL https: / http://epoceeee..ee/eeee/oe/s.se htwp:/eeeeeeein.eeeeeeeeeeeeeeee neurips. http://lee//ppee//////////
fbd053c1 Paper pdf.
Wenhan Xiong, Jingyu Liu, Igor Molybog. Hejia Zhang. Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang. and Hao Ma. Effective longcontext scaling of foundation model. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 1643-4663, Mexico City, Mexico, June 2024, Association for Computational Linguistics. URL https: http:::///caaahhllgoooo org/2024 www.aaaacllaaaa 1onl-lloogg... 260
Mingyu Xu, Xin Men, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and weipeng chen. Base of roPE bounds context length. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. URL https: / //pp://e/eeeeeeeee/e/eeeeeeeeeee met//forrumdd-Eiiehh2tt...
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujic Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL //arts:///rxivvvv org abs/ 2505. 09388.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pp. 4791 48000 Florence, Italy, July 2019. Association for Computational Linguistics. URL https: // aalaatooooaaa.o.o..o.mmom orgg
15

---

Published as a conference paper at ICLR 2026
A DETAILS OF EXPERIMENTAL SETTINGS
A. FREQUENCY BAND EMERGENCE IN PRETRAINED LLMS
The detailed experimental settings are described Section 3.2. For comprehensive analysis, we used the following models:
google/gemma-7b
meta-Llama/LLlmma227777
NourReeeerrt//aamm--LL----2----
meta-LLLamm/hetttt-LLLmma-3333
meta--llama/llamm33-1--888
picrrssoftt/hhi-333maa11--------
We selected models that use different base models (Gemma, Llama, Phi-3) and different position interpolation methods (YaRN, Llama-scaling, LongRoPE). Here, the head dimension for the Gemma model is 256, and that for the other models is 128. The dataset for evaluation is the test set of Wikitext-103 (Merity et al., 2017) 6, and we used the subset of dataset collection of over 100 million tokens extracted from set of articles verified Good and Featured on Wikipedia. The subset of Milkiteet-13--eee
This
has 4358 sentences as test set. In our analysis, we concatenated all sentences in the dataset create long context for measuring perplexity. The sequence length in inference is L = 4096 for all models.
A.2 UNDERSTANDING FREQUENCY BAND FORMATION IN PRE-TRAINING
We described the detailed experimental settings in Section 4.1, For pre-training, we used the WikiText- 103 dataset (Merity et al., 2017), which consists of over 103 million tokens of English Wikipedia articles. We performed a comparative evaluation using Transformer- based language model (Baevski & Auli, 2019). The dimensionality of the word embedding dmmodet is 1024, the number of heads N is 8, the dimensionality of the heads d is 128, and the number of layers is 16. This implementation used the fairseq (Ott al., 2019)-based code provided in previous work(Press et al., 2022), and all hyperparameters were set to the same values as those in the literature(Press et al., 2022). The number of training epochs is 205, and the batch size is 9216. The learning rate was set 1.0, and the learning process was updated by 1e-7 every 16,000 steps. The maximum sequence length and RoPE were tested combination with (Terain, 0) (512, 1024, 2048} (Itrainy 10, 000; 500, 000; 1, 000, 000).
A.3 FREQUENCY MATCHING ROTARY POSITION EMBEDDING
The detailed experimental settings described in Section 6.2. We conducted small-scale pre- learning and context-extension experiment. In pre-training, we used the WikiText-1003 103 dataset (Merity al., 2017). Furthermore, performed comparative evaluation using Transformer-based language model (Baevski & Auli, 2019). Other parameter settings are the same as in Section 4.3. The maximum sequence length during pre-training is Lumin 512, and we set 512. In context extension achieved through position interpolation, we adopted YaRN (Peng et al., 2024), which is the most standard method for position interpolation. The maximum sequence length for context expansion with position interpolation is Tarain 1512, so we used 0 = 1512. Perplexity was used as the evaluation metric.
16

---

Published as a conference paper at ICLR 2026
B LAYER-WISE VISUALIZATION OF A SINGLE ATTENTION HEAD
To verify whether the frequency-band pattern identified in Figure and discussed in Section is consistent across layer, we visualize the query structure for single attention head (Head 19) across all 32 layers of Llama-3-8B. For each layer, we compute the 2-norm of the query matrix over the head dimensions and arrange the results into a two-dimensional map, following the same procedure as in Figure 2.
Figure 5 provides the full visualization. This layer-wise visualization provides a detailed view of how the characteristic frequency band emerges, shifts, or dissipates across the model. The visualizations reveal that frequency band appears every layer. Notably, the band becomes increasingly pronounced in deeper layers, indicating that the model progressively emphasizes fixed set frequencies as depth increases. In contrast, the early layers exhibit a more heterogeneous pattern, suggesting that a wider range of frequencies contributes to the representation before the model consolidates onto a narrower band. Although the exact coordinate at which the band appears varies across layers, find that in most cases the strongest frequency concentration occurs near the dimension predicted in Section 3. Importantly, no frequency band is observed at higher frequencies beyond those identified in Section 3.
C HEAD-WISE VISUALIZATION OF A SINGLE ATTENTION LAYER
To verify whether the frequency-band pattern identified in Figure and diseussed in Section 3 is consistent across head, we visualize the query structure for single attention layer (each Layer 0 and 31) across all 32 heads of Llama-3-8B, For each head, we compute the 2-norm of the query matrix over the head dimensions and arrange the results into a two-dimensional map, following the same procedure as in Figure 2.
Figure 6 provides the full visualization in Layer O. Figure 7 provides the full visualization in Layer 31. As reported in Section B, we observe a clear difference in the visibility of frequency bands between the shallow and deep layers. In layer 0, well-defined band appears at almost the same coordinate across all heads. In layer 31, the band is still present, but its position shifts slightly. Even so, a faint band remains in a consistent region, which suggests that the deeper layers still inherit the influence of the original frequency band.
D STRUCTURAL FACTORS BEHIND THE PHI-3 ANOMALY
In Section 3, only the Phi-3 model showed a tendency for P-RoPE results to differ. This section examines the reasons for this. A key difference is that Phi-3's block-sparse attention already removes a subset of long-range attention interactions by construction. This means Phi-3 relies more heavily on the positional channels that remain available. In particular, the cow-frequency RoPE dimensions that encode long-range relational structure. In Ilaaa and Gemma multiple query heads share the same key-value projections, which creates redundancy across heads. As result, if a subset of RoPE dimensions is replaced, other heads may still access similar positional information through shared KV projections. However, Phi-3 allocates each head distinct block-sparse pattern. These patterns eliminate many token-por pair interactions, leaving fewer pathways through which long-range positional signals can be recovered. When low- www-requency RoPE dimensions are removed, Phi-3 loses one of the only mechanisms that supports long-distance reasoning under its sparsity constraints, leading to the sharp degradation observed in Table 1.
17

---

Published as a conference paper at ICLR 2026
Layer 0
Layer 1
Layer
Layer
Layer
Layer 5
Layer
211
23
Layer
Layer
Layer
Layer
Layer 12
Layer 13
Layer 14
Layer 15
Layer
Layer 17
Layer 18
Layer
Layer
Layer 21
Layer 22
Layer
Layer 24
Layer 25
Layer 26
Layer 27
222
Layer
Layer 29
Layer
Layer
Figure 5: Layer-wise 2-norm maps of the query matrix for attention head 19 in Llama-3-8B. Each subplot shows the 2-norm plotted over 2 2-dimensional chunks of the query vectors, following the same visualization procedure as Figure 2. The vertical axis corresponds sequence length (L 4096). and the horizontal axis corresponds to RoPE dimension index (i E 0, 0,1, 1...., d/2 1) with = 128 for Llama-3-88.
18

---

Published as a conference paper at ICLR 2026
Head 0
Head
Head
Head
Head 4
Head
Head
Head
27
23
Head
Hcad
Hcad
Head
Head 12
Head 13
Head 14
Head 15
Head 16
Head 17
Head 18
Head 19
20
Head
Head 24
Head 25
Head 26
Head 27
Head
Head
28
Figure 6: Head-wise 2-norm maps of the query matrix for attention layer 0 in Llama-3-8B. Each subplot shows the 2-norm plotted over 2-dimensional chunks of the query vectors, following the same visualization procedure as Figure 2. The vertical axis corresponds sequence length 4096). and the horizontal axis corresponds to RoPE dimension index (i E 0, 1,..., ......d/2 1) with = 128 for Llama-3-8B.
19

---

Published as a conference paper at ICLR 2026
Head 0
Head 1
Head
Head
Head 4
Head 5
Head
Head
Head
Head
Head
88
Head 12
Head 13
Head 14
Head 15
Head 16
Head 17
Head 18
Head 19
Head 20
Head 21
Head 22
Head
Head 24
Head 25
Head 26
Head 27
Head
29
Head
20
Figure 7: Head-wise 2-norm maps of the query matrix for attention layer 31 in Llama-3-8B. Each subplot shows the 2-norm plotted over 2-dimensional chunks of the query vectors, following the same visualization procedure as Figure 2. The vertical axis corresponds sequence length 4096). and the horizontal axis corresponds to RoPE dimension index (i E 0, 1,..., d/2 1) with = 128 for Llama-3-8B.
20

---

Published as a conference paper at ICLR 2026
E How POSITION HEADS BEHAVE UNDER FMROPE
We examined the "position heads" described by (Barbero et al., 2025) under both standard RoPE and FMRoPE in Section 6. Figure 8, 9 and 10 shows 20-token attention maps for clarity. First, confirmed that position heads exist in both RoPE and FMRoPE. Second, changing the inference-time value of 0 in FMRoPE did not substantially affect these heads. Consistent with Barbero al. (2025), these heads appear to ignore semantic content and attend purely based on relative offsets. Our results indicate that FMRoPE retains the same class of Slaaiie-pooition-riiveee heads standard RoPE.
Figure 8: Position Head with RoPE (Training 0 10000, Inference 0 10000)
Figure 9: Position Head with RoPE (Training 512, Inference 0 512)
Figure 10: Position Head with RoPE (Training 512, Inference 0 1024)
21

---

Published as a conference paper at ICLR 2026
Table Perplexity results from Section 5. Here, 'pt' stands for PPre-train' and 'ft' stands for 'Fine- tuning' in context extension with position interpolation. 'YaRN' is a position-interpolation method applied during context extension.
Fimis
base 0
Sequence Length L
Train
Inference
512
1.12
2512
3512
15512
25512
SPE
512
20.02
77.30
>100
>100
>100
>100
Transformer-XL
512
19.98
18.88
19.02
19.53
0OM
0OM
RPE
512
21.20
21.89
34.77
74.55
0OM
0OM
WaveleiRPF:
512
19.20
17.99
18.00
18.21
0OM
0OM
ALiBi
512
19.69
18.53
18:40
18.43
18.39
18.39
10000
10000
19.39
43.63
84.45
>100
>100
>100
512
500000
500000
19.35
40.39
77.90
>100
>100
>100
512
1000000
1000000
19.35
37.94
74,26
>100
>100
>000
RoPE
512
512
512
19.58
21.19
24.20
27.42
84.75
512
512
1512
20.02
19.09
21.40
24.00
72.19
>100
512
512
15512
25.83
26.90
28.46
30.08
60.44
91.35
512
1512
15124YRRN
1512+YaRN
19.62.
17.78
17.56
17.65
20.51
23.19
512
1512+YaRN
3512-YaRN
19.38
17.99
17.66
17.64
19.93
23.44
512
1512
1512+YaRN
15512+YaRN
21.00
19.74
19.53
19:48
20.51
22.41
RoPE+YaRN
512
1512
DORO-YaRN
10000-YaRN
19.10
17.84
17.75
18.37
52.59
85.88
1512
500000+ YaRN
Y0RG
19.14
17.89
18.83
18.34
35.57
50.88
512
1512
100000+Y::RN
YORR
19.07
17.76
17.81
18.72
66.89
>100
F COMPARISON WITH OTHER POSITION-ENCODIO METHODS
F.1 EXPERIMENTAL SETTINGS
In addition to experiment in Section 6, we also compared our method with the following position- encoding methods.
NoPE (Kazemnejad al., 2023)
Sinusoidal PE (SPH) (Vaswani et al., 2017)
Transformer- XL PE (Dai et al., 2019)
Relative Position Representation (RPE) (Shaw et al., 2018) with clipping size 32
Attention with Linear Biases (ALiBi) (Press et al., 2022)
Wavelet PE (Oka et al., 2025)
For pre-training, we used the WikiText- 103 dataset (Merity et al., 2017), which consists of over 103 million tokens of English Wikipedia articles. We performed comparative evaluation using Transformer-basee language model (Baevski & Auli, 2019). The experimental setup is identical to that used in Section 6. Please refer to Appendíx A.3 for details.
F.2 PERPLEXITY RESULTS
Figure 6 presents the perplexity scores for each method. We first confirmed the effectiveness of ALiBi and WaveletPF, both of which are known for their strong extrapolation capabilities. However, methods based relative position encoding (RPE), such as RPE itself, WaveletPE, and Transformer- XL, showed out-of-memory (OOM) errors as the sequence length increased, and these methods were unable to generate results. In contrast, ALiBi consistently maintained strong extrapolation performance even at longer sequence lengths. RoPE, on the other hand, generally exhibits lower extrapolation performance compared to other positional encoding methods. Even FMRoPE, an enhanced variant of RoPE, did not surpass the original RoPE in extrapolation ability. Nevertheless, fine-tuned accordingly, both
when the context length expanded to L 1512 and the models FMRoPE and RoPE showed improved performance relative to coappolation-oriennee PE methods. Notably, beyond
1512, FMRoPE outperformed not only RoPE but also the other PE methods.
22

---

Published as a conference paper at ICLR 2026
G DOWNSTREAM TASK
Beyond the analyses in Section 6, we fürther examined FMRoPE under extended context lengths and larger model scales. In addition, assessed performance not only in terms of perplexity but also across a suite of downstream tasks.
G.1 EXPERIMENTAL SETUP
We trained a decoder-only Transformer with RoPE and FlashAttention. The model has 11.2 parameters with hidden size - Maayerr 1he and an MLP expansion ratio of 8. We use RMSNorm without biases. Dropout is disabled throughout (residual (residual__dropottee 0, eetention_dropoutee 0, drw.oddotnndrroooototoo 0). The maximum training context length is 1024 tokens. Vocabulary size is 50,280 using the GPT-NeoX/OLMo Dolma v1.5 tokenizer with right-side truncation/paadding: pos_token_id= 0. pad_token_ide _i= 1. We use AdamW with (21,8))=(0.)) 0.95), 6=10- weight decay 0.1 (applied to embeddings and LayerNorm scales; The peak learning rate is 6x 10 with cosine schedule and 10,000 warmup steps; the final LR decays to 0.1x the peak. We use AMP 6 training with gradient clipping at 1.0. Training uses distributed data parallelism with gradient synchronization the batch boundary. The global batch size 512 sequences; per-device microbatch size is We enable pinned memory, prefetching, and persistent dataloader workers for throughput. Checkpointing saves unsharded states every 5,000 steps; evaluation train with fash_attentinn ion=true. Distributed training uses find unused_paa sereed_prraaae params=aalse gradi- synchronization mode is set batch. We log metrics every 10 steps and monitor throughput with moving window of 20 steps. Pretraining uses the English C4 corpus (high-quality web text) preprocessed NumPy shards. Unless otherwise noted, we train for one epoch.
every 1,000 steps. We
G.2 EVALUATION METRIC
We report validation perplexity C4 using fixed-length chunks to probe length generalization: (256, 512, 1024, 2048, 4096, 8192) tokens. Batch size is 64. Beyond perplexity, we evaluate zero- shot performance (unless specified) on standard commonsense and QA benchmarks: PIQA (Bisk et al., 2019), HellaSwag (Zellers et al., 2019), CommonsenseQA (Talmor et al., 2019), and Social IQa (Sap al., 2019). We additionally report Basic Arithmetic perplexity. For more realistic long-context generation setting, we also evaluated the model on the Needle-in --tss--- task 7
G.3 RESULTS
G.3.1 PERPLEXITY
Table 7 shows the perplexity results. When the inference length does not exceed the training length (L S 1024), all settings achieve comparable perplexity around 20. The lowest perplexity is 19.77 when training and inference both use 0 10,000. Differences appear once the inference length exceeds the pre-training context. The baseline configuration with 0 = 1024 shows sharp perplexity increase to 42.36 L 2048 and diverges beyond 4096. In contrast, FMRoPE enlarges the inference base to 2048 or 8192 while keeping training at 1024, and this substantially improves extrapolation. These results show that simply enlarging the inference base frequency effectively extends the usable context without additional training. A model trained and inferred with 0 = 10,000 maintains competitive perplexity up
1024 but degrades rapidly beyond that point, reaching 46.61 at L = 2048 and 57.83 at L = 4096. This observation confirms that training with an excessively high base does not guarantee long-context generalization.
G.3.2 DOWNSTREAM TASK
Table 8 shows the downstream task results. Across all tasks, the differences among configurations are small, showing that changing the RoPE base for inference has little negative impact on general language understanding. When training and inference both use 0 1024, the model achieves strong overall accuracy with 43.96 on SocialIQA, 69.58 on PIQA, 33.66 on Commonsense@AA, 44.80 on
hhttps: htttpp:://gtthhe htttts::/ggthuu.u..mggaarrrrrrre wttp///aaaaattt//........ e Inaaytyst
23

---

Published as a conference paper at ICLR 2026
Table Perplexity results from Section G. "pt" stands for 'Pre-train' and 'ft' stands for "Fine-tuning' in context extension with position interpolation. The gray area represents the FMRoPE score.
Lusin
Base in RoPE
Sequence Length in Inference L
pt
ft
Train
Inference
256
512
1024
2048
4096
8192
Pre-train
1024
1024
1024
23.08
21.02
19.88
42.36
>100
>100
1024
1024
2048
23.10
21.05
19.90
19.33
>100
>100
1024
1024
8192
23.98
22.07
21.08
19.85
19.58
22.86
1024
10000
10000
23.01
20.94
19.77
46.61
57.83
>100
Table 8: Downstream task results from Section G.
Base in RoPE 0
Downstream Task
Train
Inference
SocialIQA
PIQA
CommonsenseQA
HellaSwag
Arithmetic
1024
1024
43.96
69.58
33.66
44.80
24.90
1024
2048
43.85
70.07
33.98
45.10
24.36
1024
8192
44.16
68.71
32.92
44.91
24.06
10000
10000
43.90
70.78
32.35
45.00
24.86
HellaSwag, and 24.90 on Arithmetic. Using FMRoPE with an inference base of 2048 maintains or slightly improves performance. The model reaches 70.07 PIQA, 33.98 on Commonsense and 45.10 on HellaSwag, which are the best or nearly the best among all settings, while keeping SocialIQA and Arithmetic close to the baseline. When the inference base further increased 8192, performance remains stable with 44.16 on SocialIQA and 44.91 on HellaSwag, indicating that large inference base does not harm downstream accuracy. model trained and inferred with 10,000 achieves the highest PIQA accuracy of 70.78, although Commonsense(AA drops to 32.35.
These results show that frequency matching during inference preserves or slightly enhances down- stream task performance while providing the long-context benefits demonstrated in perplexity eval- uation. The findings confirm that decoupling the training and inference RoPE bases does not compromise the model's ability to perform common natural language understanding tasks.
G.3.3 NEEDLE IN A HAYSTACK
We evaluate four settings of RoPE on the Beedle-n---HHassterl task with context lengths from 512 8192 tokens. The training context length is 1024 tokens. Figure 11, 12, 13 and 14 shows Needle-in-a-Hayse Haytsakk results. In the figures, green indicates high scores and red indicates failure.
First, the standard RoPE with base 10000 shows a clear boundary stay green for contexts up to 1024 tokens, which means the model interpolates well inside the training range. Beyond 1024 tokens, the scores turn red, and the model loses the ability to retrieve the needle. This behavior shows that standard RoPE does not extrapolate. Next, the model trained with base = 1024 and inferenced with the same base shows almost the same pattern. The model keeps green scores inside 1024 tokens and turns red beyond it. However, the scores near the beginning of the sequence drop to red. When we train with base 0 = 1024 but inference with base = 2048, the behavior changes. The model keeps green scores up 2048 tokens. This result shows that the larger inference base pushes the usable range beyond the training length. A small degradation remains near the beginning, but the overall pattern suggests partial extrapolation. Finally, inference with base 0 = 8192 expands the effective range even further. The model keeps green scores up to 2048 tokens and shows mostly green scores around 4096 tokens. Between 4096 and 7168 tokens, some regions still appear green. These results indicate that long-range retrieval becomes possible ble in parts of this extended range. Overall, the model trained with =1024 exhibits clear extrapolation once the inference base is increased, demonstrating that inference-time base scaling expand the effective context length without additional training.
training length. Scores
24

---

Published as a conference paper at ICLR 2026
05
66
2012
6144
8016
7148
7000
Limit
Figure 11: Veedle-in-a-Haystade performance with RoPE. The maximum training context length is 1024 tokens.
64
65
6544
0014
7166
7000
Limit
Figure 12: Needle-in-a-Haye Haystack performance with FMRoPE (0 1024). The maximum training context length is 1024 tokens.
03
66
62
2000
2012
301
6144
8038
baken Limit
Figure 13: Needle-in-al Haystack performance with FMRoPE (0 2048). The maximum training context length is 1024 tokens.
05
66
Limit
Figure 14: Needle-in-a Haystacc performance with FMRoPE (6 8192). The maximum training context length is 1024 tokens.
25

---

Published as a conference paper at ICLR 2026
G.4 COMPUTATIONAL CONSIDERATIONS
To address computational concerns under different 0 settings (RoPE vs FMRoPE), we track four metrics during training: training loss, validation loss, peak GPU memory, and throughput. As shown in Figure 15, all 0 values lead to smooth and stable optimization, with no divergence in either loss curve. Peak memory usage is nearly identical across settings, and throughput varies only within normal noise levels. These results confirm that has no material effect on training stability, memory footprint, or efficiency. RoPE's computational cost does not depend on 0 Changing only modifies the numerical values of the cos and sin rotations applied to each query/key pair, but the number of operations stays exactly the same. The dominant costs in inference, namely the self-attention and feed-forward matrix multiplications, remain unchanged. Therefore, inference speed and memory usage remain identical across different settings.
taan/Crosstntopppiss
Step
33k
24
10c
15k
20c
Sysbem/Peak GPU Nemory
phroughhuhdeeicceeatttttth second
FHROPPE
20000
3000
2000
0.55
2000
1900
0.31
S3op
Stou
0
36
30c
15k
22k
14
32k
15k
20k
Figure 15: Training stability and computational efficiency across different base frequencies 0. All plots use training steps as the x-axis. (a) Training cross-entropy loss curves. (b) Validation cross- entropy loss curves. (c) Peak GPU memory usage during training. (d) Throughput measured in batches per second.
26

---

Published as a conference paper at ICLR 2026
H CONNECTION TO COVARIANCE VIEW (WHY THIS PROXY WORKS)
The full 2 x 2 covariance the basis
[cos(mi)) (sin(mw))
)
Σ11
E12
E(w)
Cov
E2i
has explicit entries (with = (irrrin))
sin(2r)
sin(2x)
E1 2 +
sin
E22
--coosa
(10)
4x
4x
1 - cos( clsoossn 2x)
sin r
1 COS
12
Σ2 E12.
(11)
4x:
The variance maximized is exactly the (1, entry: V(x) (x). If, instead, optimizes over all linear combinations A cos(mw) B (mw) under coefficient-norm budget, the centered variance is It² Amax (mmax(((w)) (E(w)) by the Rayleigh-Ritz theorem. Here, Amax represents indicator of the maximum variance along the principal component direction of the covariance matrix and is used as more general optimization criterion. This value can be computed via the eigenvalue decomposition of the matrix.
We additionally computed the optimal point by maximizing the largest eigenvalue Xmax of the full 2 x 2 RoPE matrix. This full covariance analysis yields an alternative maximizer x R 4.493409, whereas the simplified proxy gives 3. 3.657210 in Section 4. While the numerical values differ, this discrepancy has only minor effect the predicted band index iband in Table 3 and 4. because the theoretical predictor depends on æ only through the logarithmic relationship Eq (9). Substituting æ into this expression shifts j by at most 1-2 dimensions across all values of used in the experiments, due to the strong dampening effect of the logarithm. As a consequence, the empirical relation between predicted j and the predicted band index Aaand (Table 3 and 4), including the near-linear fit with slope e R 1.0 1.1, remains essentially unchanged.
I DISTRIBUTION OF 0g IN ROPE
Figure 16 shows the distribution of 0; when position interpolation is applied at positions 10,000, 500,000, and 1,000,000. We examined several interpolation methods, including YaRN, Llama-scaling, and RoPgRoPE. Overall, position interpolation tends to increase the proportion of low-frequency 0; components.
8=10000
107
[m 10000
8=1000000
10f
Llama-Graling
B- 1000030
101
200000-LongRoPE
8=10000
10f
187
101
10a
24 Head Dimention Index
Figure 16: Distribution of f, values across dimensions i when position interpolation is applied at positions 10,000, 500,000, and 1,000,000. The x-axis represents the dimension index i, and the y-axis shows the corresponding 0, values.
27

---

Published as a conference paper at ICLR 2026
J ANALYSIS OF LONG-TERM DECAY
To better understand interpolation and extrapolation trade-off, we next investigate the long-term decay of RoPE.
J.1 LONG-TERM DECAY OF QUERY AND KEY
Figure 17 plots the attention logit (query-key dot product) for the first query vector in the final decoder layer across relative positions; all heads show the same trend, so we report just the first head for brevity. For large base frequencies (0 > 10, 000), the logit decays almost monotonically with distance, whereas with 8 = 512, no such decrease activation is observed.
J.2 LONG-TERM DECAY OF ROPE
To isolate the effect of 0, we follow prior work (Su et al., 2023; Xiong al., 2024) and visualize RoPE activation when both the query and key vectors are filled with ones (Figure 18, left). The original activation grows with 6, confirming that larger base frequencies inject more energy into low-frequency dimensions.
Here, we hypothesize that RoPE components frequencies higher than the band index are NoPE. To isolate the effect of the active components, we visualize the activation using only the dimensions higher than the band index in the right part of Figure 18. Surprisingly, we found that RoPE activation was reduced when theta was large. In contrast, when 0 matches the sequence length, most dimensions fit within the band, resulting in relatively high activation. When the relative distance is within the maximum sequence length used during pre-training, the activation tends to be low. In contrast, for distances beyond the pre-training range, the activation becomes relatively higher. We speculate that this pattern is the reason why activation does not decrease in extrapolation in the actual activation shown in Figure 17.
Long-torm Query
0-1552
200
600
1003
2000
Detabuu poittion
Figure 17: Attention logits (query- key dot product) for the first query vector, plotted across relative positions.
Long term decay of RaPE before pre: training
Long torm decay of RoPE after pre-training
C- 30000
--100000100000000 200000000
9-10000
De 1000000
0.75
5-500000
D-022
5050000
5-512
0.6
Advvtton
0.50
0.6
0.2
0.50
0.0
-02
200
602
800
2000
1200
3400
400
000
1000
1200
1400
Picutivc
Figure 18: RoPE activation when both query and key vectors are filled with ones. Gray area indicates relative positions beyond the maximum sequence length Lurain = 512 used during pre-training.
28

---

Published as a conference paper at ICLR 2026
K
LIMITATION
potential limitation of FMRoPE is that the optimal may differ across pretraining, finetuning, and inference. While our analysis suggests that such cross-stage differences correspond to the same linear frequency rescaling in RoPE, and our experiments did not observe degradation from pretrain tinferece mismatches (Table 5), more systematic study larger model scales remains an important direction for future work.
Our analysis focuses on long-context extrapolation, and did not study multi-step reasoning tasks. Our evaluation does not fully cover larger model sizes wide range of long-context benchmarks. remains an open question how FMRoPE interacts with main-of-thoouhh reasoning and other forms of multi-step problem solving. We leave this direction for future work.
29


Reviews:
[{"review_id": "4BwipFQI5j", "reviewer": "Reviewer_aPpJ", "summary": "This paper tries to analyze which frequencies are chosen by using LLM's in their inference. Their definition is to understand which keys and values have the highest norm and observing that certain dimensions consistently become large when they apply to Rope based positional embeddings. They call this frequency band. They demonstrate several claims about this frequency band:\n\n1) The band's location is a function of thte base frequency $\\theta$.\n2) It forms early and stays throughout the training and even models that do position interpolation like Yarn.\n3) They find that this is the key property and that if we use p-ROPE which uses Nope for lowest frequencies, there is a sharp threshold. Once the p-ROPE cuts off the frequency band, the model perplexity drops precipitously.\n\n4) Pretraining Experiments - They show that where the band forms is a function of the pretraining length $L_t$ and $\\theta$. For a fixed training length, an increase in  $\\theta$  leads to a higher frequency. For a fixed $\\theta$, as the training length increases the band shifts to the lower frequencies. \n\n5) The paper then relates a theoretical model that proposes what base frequency $\\theta$ would maximize the variance of a random position. They find that the best base frequency is a certain constant $x^*$  divided by the average training length. They find that this matches the empirical distribution found in a number of different models. \n\n6) From the above, they posit a new frequency-matching Rope that improves extrapolation without too much loss in interpolation by setting the training length appropriately and forcing the model to use the entire spectrum usefully.", "questions": "1) Can you explain why Phi-3s performance drops so quickly. In particular, what about block sparse matrix makes this phenomenon go away.\n\n2) What is the key innovation with respect to the work of Barbero et al?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper shows that frequency bands appear and uses p-Rope to show that these bands are crucial for length extrapolation.\nThe paper produces new ideas and proposes a new mechanism FMRope to set the train length as a function of the base frequency. \nIt validates this hypothesis using both theory and experiments and results in a strong conclusion. It also shows why long context can be so difficult.", "weaknesses": "The p-ROPE experiment, which is the entire basis for the \"weak utilization\" claim, produced a critical anomaly: the Phi-3 model (Table 1).1 Unlike Llama or Gemma, replacing the low-frequency dimensions in Phi-3 immediately and severely degraded performance (PPL 2.84 $\\to$ 46.11). The paper's explanation for this is a single line attributing it to Phi-3's \"block-sparse attention\". I would like to understand this result more. \n\nA number of results seem similar to the paper  Barbero et al. They also show that frequency bands appear and introduce p-Rope. I would like to understand the differences and the key innovations with respect to the above paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YQ3r6DFrqV", "reviewer": "Reviewer_jmkZ", "summary": "This paper investigates the formation of the frequency band, i.e., RoPE dimensions with high norms. Through both empirical and theoretical analysis, the authors show that the training length and $\\theta$ both affect the band. The authors confirm that removing RoPE for dimensions below the band has little impact on model performance, and find that the band does not change much after finetuning. Next, the authors show that by setting $\\theta$ to the training length $L$, the band shifts to a lower frequency and improves extrapolation, but not interpolation.", "questions": "- As this paper only focused on semantic heads, I wonder how the other 'position heads', as described in [1], change with FMRoPE. \n- The authors find that the band does not change much after position interpolation. Could this be because the compute used on finetuning is less than pretraining? Will the band gradually shift with finetuning on more tokens? \n- In Section 5.2, why is there a scaling constant c? What is the actual i_band in the last row of Table 3 (from your pretrained model later)?\n- In the 1B experiments, for downstream tasks, what is the evaluation context length? Are there any evaluations on the performance of generation tasks like GSM8K? \n- Please insert PDFs as figures. Many figures (e.g., Figures 1 and 2) look blurry.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**: Although the idea of 'frequency band' is not entirely new, the authors conduct a comprehensive investigation of the band across models and under many settings. It provides unique value and largely extends prior work. \n\n**Quality**: The analysis methodology is convincing, with many insights about the formation of the frequency band. The experiments are carefully controlled, and the conclusions are solid. \n\n**Significance**: The proposed method is supported by both theoretical analysis and empirical evidence. The final remark on the tradeoff between interpolation and extrapolation is intriguing.  \n\n**Clarity**: The organisation of the paper is clear and well motivated, starting with existing models, to empirical and theoretical analysis, to the proposed method and results. The takeaway messages are clear and easy to understand.", "weaknesses": "- The lack of real long-context evaluation is a weakness. Prior work shows that perplexity is not a good measure of long-context task performance [1]. The authors could show some real long-context tasks (e.g. NIAH, RULER) that involve generation to compare the performance. \n\n- The practical implications of the proposed method are still unclear. Most frontier models go through context extension to a large final context length via finetuning, and during inference time, limit the context length to the max. As the authors do not show concrete evidence on the actual task performance of extrapolation, I doubt the applicability of the method in real long context tasks. \n\n- A potential issue with the FMRoPE method is the mismatch between pretraining, finetuning, and inference, which can all have different context lengths. In Table 4's Finetuning, it seems that we probably need different $\\theta$s for pretraining, finetuning, and inference to obtain the best performance. If the values of $theta$ vary largely across these stages, will there be performance issues? The paper does not systematically study cross-stage mismatches, so it is unclear how severe these issues would be at scale.\n\n- Some details seem problematic and require further checking: \n  - Equation (2): Should be argmax instead of 'max'. \n  - Line 138 is confusing: why is it selecting only from the first d/2 dimensions? How about the second d/2 dimensions?  \n  - Table 7 has a wrong caption. \n\n\n[1] Fang et al. (2024) What is Wrong with Perplexity for Long-context Language Modeling?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZO7Kb99Ekg", "reviewer": "Reviewer_26SL", "summary": "This paper investigates the frequency-band phenomenon in RoPE: a small subset of positional dimensions whose norms dominate. It shows that the RoPE base frequency $\\theta$ and the pre-training context length jointly determine where this band lies, and argues that most lower-frequency dimensions behave almost like NoPE and contribute little. \n\nEmpirically, the band emerges early in training, persists under position-interpolation methods (e.g., YaRN, LongRoPE), and shifts predictably with $\\theta$ and the training length. The authors also derive a simple closed-form predictor for the band location by maximizing a variance-based proxy, and propose FMRoPE, which sets $\\theta$ to training length to improve length extrapolation at the cost of interpolation.", "questions": "# Questions:\n\n\n1. About Figure 2. Does the figure report averages over heads and layers, or only the first layer (as suggested around line 189)? What is the variance across heads? Do layers beyond the first exhibit the same phenomenon, or is it concentrated in early layers?\n\n2. The takeaways from Section 5 are interesting. Could you please (i) include a figure analogous to Figure 2 (or stratified by layer) to illustrate stability across heads/layers? (2) report your predictor and empirical results when theta is set to other values, to validate the closed-form estimate beyond the main setting?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "# Strengths\n\n1. The band phenomenon is demonstrated on multiple families (Gemma, Llama, Qwen, Phi-3) and persists under different interpolation schemes (YaRN, Llama-scaling, LongRoPE). The evaluations use Wikitext-103 with long concatenated contexts and fixed inference length (L=4096), which provides a consistent testbed.\n\n2. The band index ablation cleanly probes which frequency ranges matter. Below-band dimensions can be swapped to NoPE with little perplexity change in several models, supporting the NoPE-like claim.\n\n3. The single-coordinate variance proxy yields $x^{\\star}$ and a closed-form predictor for j.\n\n4. Aligning $\\theta$ with training length L, which pushes the band to the lowest frequencies and improves length extrapolation (but hurts interpolation), shows when to pick small vs. large $\\theta$ rather than advocating a one-size-fits-all choice.", "weaknesses": "# Weaknesses\n\n1. While the study is careful and informative, its scope is limited. Much of the contribution reads as an in-depth follow-up to Barbero et al. (2024). The proposed method, **FMRoPE**, also feels underdeveloped for practice: its real-world applicability and deployment conditions (e.g., when $\\theta$ can be set or adapted) are not clearly demonstrated. I would like to list the minor weaknesses in the following.\n\n**Minor points**\n\n2. The phrase “aligning $\\theta$ with the training length” is confusing on first read. Because setting $\\theta$ equal to the training context window is uncommon, please make this explicit in the abstract/introduction (e.g., “we set ( \\theta \\approx L_{\\text{train}} )”) and briefly motivate why this choice helps extrapolation.\n\n3. The figure should be explained in greater detail (axes/units, how norms are aggregated across heads/layers, selection criteria), see my question in the next section.\n\n4. In Section 4.2 and the Takeaways results are shown primarily for (L_{\\text{train}}=512), yet the setup states combinations ({512, 1024, 2048}) were tested. Please include representative results for 1024/2048; this evidence is important to support the takeaway that “the effective RoPE dimension is determined by the pre-training $\\theta$ and maximum sequence length.”\n\n--\n\nReference:\n\nBarbero, F., Vitvitskyi, A., Perivolaropoulos, C., Pascanu, R., & Veličković, P. (2024). *Round and Round We Go! What Makes Rotary Positional Encodings Useful?*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cSp4liIogr", "reviewer": "Reviewer_uqNo", "summary": "This paper provides a novel analysis of RoPE, revealing a predictable \"frequency band\" whose location depends on the base frequency `θ` and training length `L_train`. The core finding is a crucial trade-off: large `θ` aids interpolation but harms extrapolation, while setting `θ ≈ L_train` improves extrapolation. The authors support this with a concise theoretical model and extensive experiments, challenging the common practice of simply scaling `θ`.", "questions": "1.  How does FMRoPE perform in extrapolation if the inference `θ` remains fixed at its training value?\n2.  Could you elaborate on the hypothesis linking Phi-3's distinct `p-RoPE` results to its block-sparse attention?\n3.  How much would the theoretical optimum `x*` change if derived from the full covariance matrix's largest eigenvalue (`λ_max`) instead of the simplified proxy?\n4.  How stable is the frequency band's position throughout the entire training process *after* its initial formation?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The discovery of the interpolation-extrapolation trade-off provides clear, practical guidance for RoPE design. The work combines a predictive theoretical model with rigorous, well-controlled experiments across multiple LLMs.", "weaknesses": "* The proposed FMRoPE intervention requires an impractical adaptive inference scheme.\n* Core from-scratch training experiments are conducted on a relatively small scale.\n* The main body's analysis is heavily focused on perplexity, with less exploration of the trade-off's impact on specific downstream tasks.\n* The theoretical model, while elegant, uses strong simplifications (e.g., single coordinate variance) and abstracts away complex query dynamics.\n* The analysis is confined to RoPE, limiting the generalizability of the frequency-band mechanism to other positional encoding families like ALiBi.\n* FMRoPE improves extrapolation at the cost of interpolation performance, presenting a trade-off rather than a universally superior solution.\n* Empirical constants (the `c ≈ 1.1` factor) and model-specific anomalies (e.g., Phi-3) are noted but not fully explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rSps7pw19X", "reviewer": "Reviewer_K7WU", "summary": "This paper investigates the impact of the base frequency parameter θ in Rotary Position Embeddings (RoPE) on long-context performance in large language models. The authors identify concentrated high-norm dimensions in RoPE, referred to as frequency bands, and show that this property is consistent across model families such as Gemma, LLaMA, Qwen, and Phi-3, as well as position interpolation strategies. The paper provides a closed-form predictor for the location of these bands based on θ and the training sequence length L_train. Building on these findings, the authors propose Frequency-Matching RoPE (FMRoPE), which selects θ to align with the target context length in order to improve long-context extrapolation. While the empirical analysis is systematic and the findings are practically relevant, the contributions are incremental relative to prior studies, and the proposed method has limited applicability in realistic deployment settings.", "questions": "How does the emergence of the frequency band affect attention head behavior? Does it alter cross-token interactions, particularly for long-range dependencies?\n\nWhat is the theoretical explanation for the shift of the frequency band toward lower frequencies when $\\theta$ aligns with training length? How does this affect positional information density?\n\nWhat practical heuristics can practitioners use to balance interpolation and extrapolation performance when selecting $\\theta$ under limited compute?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Empirical analysis across multiple model families reveals a consistent frequency band phenomenon in RoPE.\n\nThe predictor for band locations is mathematically grounded and aligns well with observed distributions.\n\nDiscussion of the interpolation–extrapolation trade-off provides useful guidance for model tuning.\n\nExperimental setup and training configurations are sufficiently documented for reproducibility.", "weaknesses": "1. The argument that weight decay drives the emergence of frequency bands is not empirically validated, and no ablation is provided to support this claim.\n2. Similar observations about RoPE frequency structure have appeared in prior work such as [1], reducing novelty.\n3.The proposed method requires advance knowledge of the target context length, which limits applicability for variable-length real-world inputs.\n4. Evaluation focuses primarily on perplexity using relatively small models trained on WikiText-103, leaving uncertainty about performance on long-context reasoning tasks.\n5. Computational considerations such as training stability, memory usage, or inference efficiency under different θ settings are not discussed.\n\n[1] Barbero, F., Desmaison, A., & Storkey, A. (2024). Spectral analysis of positional encodings in large language models. arXiv preprint arXiv:2402.12345.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)