Probing Rotary Position Embeddings through Frequency Entropy
Authors: Yui Oka
Kentaro Hanafusa
Taku Hasegawa
Kyosuke Nishida
Kuniko Saito
Venue: ICLR 2026 Poster (2026)
Keywords: Rotary Position Embedding
Frequency Entropy
Large Language Model
Primary area: foundation or frontier models, including LLMs
TL;DR: Frequency Entropy enables analysis of RoPE on a rotational pair basis, allowing measurement of RoPE's periodicity and bands.

Abstract: Rotary Position Embeddings (RoPE) are widely used in Transformers to encode positional information in token representations, yet the internal frequency structure of RoPE remains poorly understood. Previous studies have reported conflicting findings on the roles of high- and low-frequency dimensions, offering empirical observations but no unifying explanation. In this paper, we present a systematic framework that bridges these disparate results. We introduce Frequency Entropy (FE), a metric that quantifies the effective utilization of each RoPE frequency dimension, and we provide an analysis of how RoPE’s sinusoidal components contribute to model representations on a per-dimension basis. Based on an analysis of the Llama-4 model, which incorporates both RoPE and NoPE layers, we find that the periodicity captured by FE appears in RoPE layers but not in NoPE layers. Furthermore, FE identifies dimensions in which energy concentrates under RoPE. These characteristics are observed across the spectrum rather than being confined to specific dimensions. Moreover, attenuating extreme-entropy dimensions at inference yields downstream accuracy that is statistically indistinguishable from the baseline, with modest perplexity improvements on average, suggesting that such dimensions are often redundant. Overall, FE provides a simple, general diagnostic for RoPE with implications for analysis and design.

Published as a conference paper at ICLR 2026
PROBING ROTARY POSITION EMBEDDINGS THROUGH FREQUENCY ENTROPY
Yui Oka"; Kentaro Hanafusa 2+1 Taku Hasegawa', Kyosuke Nishida', Kuniko Saito
2 Human Informatics Labs., NTT, Inc. Bhime University yui. www.ntt..coommmm
ABSTRACT
Rotary Position Embeddings (RoPE) are widely used in Transformers encode positional information in token representations, yet the internal frequency structure RoPE remains poorly understood. Previous studies have reported conflicting findings on the roles of high- and low-frequency dimensions, offering empirical observations but no unifying explanation. In this paper, we present systematic framework that bridges these disparate results. We introduce Frequency Entropy (FE), a metric that quantifies the effective utilization of each RoPE frequency dimension, and provide an analysis of how RoPE's sinusoidal components contribute to model representations on a per-dimension basis. Based on an analysis of the Llama-4 model, which incorporates both RoPE and NoPE layers, we find that the periodicity captured by FE appears in RoPE layers but not NoPE layers. Furthermore, FE identifies dimensions in which energy concentrates under RoPE, These characteristics are observed across the spectrum rather than being confined specific dimensions. Moreover, attenuating extreme-entropy dimensions inference yields downstream accuracy that is statistically indistinguishable from the baseline, with modest perplexity improvements on average, suggesting that such dimensions are often redundant. Overall, FE provides a simple, general diagnostic for RoPE with implications for analysis and design.
INTRODUCTION
Position representations in Transformers (Vaswani et al., 2017) are crucial factor determining their ability to handle long-range dependencies. Among these representations, Rotary Position Embeddings (RoPE) (Su et al., 2023) have become standard in many of the latest large language models, including Llama (Touvron et al., 2023; Grattafiori et al., 2024). Qwen (Qwen et al., 2025; Yang et al., 2025), and Gemma (Gemma Team et al., 2024a), and have contributed improved performances long-text processing. However, the design of RoPE was introduced empirically, and the role of each frequency dimension and how they are utilized within the model remain unclear.
In recent years, several analyses at the RoPE dimension level have been reported. For example, Barbero et al. (2025) observed that high-frequency components contribute to positional pattern formation, while low-frequency components contribute to semantic information. They also demon- strated that replacing part of the low-frequency components with NoPE (Kazemnejad et al., 2023) does not significantly affect model performance. On the other hand, Chiang & Yogatama (2025) showed that high-frequency components are scarcely utilized and can be removed without impacting performance. Furthermore, Hong al. (2024) pointed out that the low-frequency component is essential for modeling long-range dependencies in specific attention heads. Previous analyses of RoPE have largely relied on visual inspection of heatmaps and coarse division into high- versus low-frequency components. The Llama-4 (Meta, 2025) model introduces iRoPE, which combines interleaved NoPE layers with frequency scaling to extend context length, though its internal frequency dynamics remain largely unexplored. These examples highlight the conflicting reports regarding the
"These authors contributed equally.
Work done during internship NTT, Inc.

---

Published as a conference paper at ICLR 2026
Spectrum Frequency Entropy
Spectrum Frequency Entropy highlights RoPE bands
Entropy
Query L2-Norm
Token
Dimens
Sequence Frequency Entropy
Entropy
Dimension
Sequence Frequency Entropy
reveals periodic dimensions
Dimension
Figure 1: Example of our Frequency Entropy. Left: Frequency Entropy (FE) per rotary pair, where the horizontal axis the entropy value and the vertical axis the pair index j E (O...., d/2 Right: Query fn-ommms across rotary pairs, where the horizontal axis is sequence length (L = 4096) and the vertical axis is the pair index.
role of each RoPE dimension, and analysis remains observation-based lacking unified theoretical and empirical understanding.
In this paper, we mathematically formalize the contribution of each frequency dimension and introduce Frequency Entropy (FE), classical quantitative measurement framework. It comprises two complementary metrics, Spectrum Frequency Entropy (Spectrum FE) and Sequence Frequency Entropy (Sequence FE), and makes RoPE's spectral behavior measurable on a per-dimension basis. Figure illustrates the method. We compute these entropies Llama-4 model with RoPE and NoPE layers and find that Spectrum FE quantifies power concentration the Fourier domain and reveals
rotation pairs with sustained energy, while Sequence FE measures token-wise regularity and detects persistent oscillations of rotation pairs induced by RoPE. Furthermore, these signatures absent in layers with NoPE.
To probe the functional relevance without fine-tuning, we conduct targeted attenuation study. For rotation pairs whose FE falls below threshold, reduce their contribution attention during inference by multiplying the corresponding query and key channels by a constant a < 1, keeping all other parameters fixed. The experimental results demonstrate that suppressing Low-Sequence- FE dimensions leaves perplexity and downstream accuracy unchanged, whereas suppressing low- Spectrum dimensions worsens perplexity. In addition, attenuating high-Spe Sighh-Seettum--a outliers has negligible effect on the perplexity and the downstream accuracy. These results indicate that persistent oscillations induced by RoPE are largely redundant, while band-limited components carry a task-relevant signal.
Across experiments, we observe that oscillatory and spectrum rather than only specific frequencies extremes. Consequently, the conventional high- versus low-frequency dichotomy is insufficient. FE provides a spectrum-aware, model-agnostic lens that reconciles prior mixed observations and informs pruning, reweighting, and the design of future positional schemes.
behaviors occur throughout the
BACKGROUND AND RELATED WORK
ROTARY POSITION EMBEDDING (RoPE)
RoPE (Su et al., 2023) has become the de facto standard positional embedding method in many of the latest large language models, such as Llama (Touvron et al., 2023; Grattafiori et al., 2024) Qwen (Qwen al., 2025; Yang al., 2025), and Gemma (Gemma Team et al., 2024a). RoPE
2

---

Published as a conference paper at ICLR 2026
introduces position information by applying rotation to the query and key vectors in the self- attention mechanism. This property allows RoPE to encode relative positional information while preserving compatibility with an absolute token index.
Am, (gn) (Bm. 9n Rm - n,9 Km
The rotation matrix R E Raxd the block-diagonal rotation matrix, defined as follows:
cos(n(o)) sin(n0))
sin(1(lo)
cos( (106))
O sin (n1)) cos(n
0
0
0
cos(11) sin(n0.)
0
0
0
Rn,6
:
0
cos(nBa/2
sin(n
0
sin (n0a/2-1)
cos(n0
(2)
where A E RLXL
In E R2xd is the 1-th query when the number of dimensions is d and is the 1) (0 5 m - 1). Typically, the rotation angles are chosen as the base of RoPE 0, = 10000 2j/d (j=0.... - In practice. the base values 0 in RoPE are typically set to be quite large. For example, 0 10,000 is adopted in the Gemma model (Gemma Team al., 2024a) and Llama-2 (Touvron et al., 2023), = 500,000 is used in Llama-3 (Xiong et al., 2024), and 0 = 1 1,000,00 000 is employed in Qwen-3 (Yang et al., 2025). RoPE provides complex phase concept of relative offsets for attention across all layers and all heads while remaining parameter-fre free and sordware--riendl]
absolute position (0 S n & - 1) when the sequence length is L, and km E IR xd is the m-th key
2.2 UTILIZATION OF INFORMATION WITHIN RoPE
The distribution and utilization of information within RoPE the dimensional level remain only partially understood. An early perspective at the attention- head level emphasized the low-frequency structure as necessary for long-distaned modeling. Hong et al. (2024) identified "positional heads" whose activations align with token distance. Ablating these heads dominated by lower-frequency (slower-varying) RoPE dimensions substantially degrades the long-contex performance. second line of empirical evidence dissected pretrained models to associate roles with high and low frequency. (Barbero et al., 2025) reported that high-frequency components drive distinctive off-diagonal "posi- tional" attention patterns, whereas low-frequency components correlate more with semantic content. Further, replacing parts of the low-frequency spectrum with NoPE-like (Kazemnejad et al., 2023) variants leaves performance largely intact or even improved in mall--o--mi--salle settings. These results suggest redundancy within portions of the low-frequency subspace. A third line of work has argued almost the converse for the high-frequency end: Chiang Yogatama (2025) measured the per-dimension utilization and showed that dimensions with larger rotation angles (i.e., higher frequencies) are weakly used. This points to an over-allocation representational capacity to rapid positional oscillations that downstream layers seldom exploit.
Methodologically, existing analyses have tended to employ broad classifications such as high- ffrjueecyylow-feeuueeeeyy lacking a unified metric that transcends dimensions. Taken together, these findings create a tension: (i) particular heads critically depend on low-frequency structure for long- range dependencies (Hong et al., 2024); (ii) parts of the frequeeccccy spectrum appear semantically entangled and sometimes dispensable (Barbero et al., 2025); yet (iii) high-frequency dimensions look under-utilized and safely removable (Chiang & Yogatama, 2025).
2.3 IROPE
Llama-4 (Meta, 2025) introduces iRoPE, which augments RoPE with two key changes. First, it utilizes an interleaved layer design where rotary position embeddings are applied only to alternating attention layers, while the others operate without explicit positional signals (NoPE; Kazemnejad et al., 2023), encouraging content-basse angles by factor < 1, slowing phase growth and extending the effective positional range well beyond the training context. Despite these advances, the internal behavior iRoPE remains largely unexplored.
long-rang reasoning. Second, iRoPE scales the RoPE rotation
3

---

Published as a conference paper at ICLR 2026
2.4 FREQUENCY ENTROPY
Frequency entropy (also called spectral entropy) quantifies the uncertainty of signal's 's frequency distribution by applying Shannon's 's entropy (Shannon, 1948) to its power spectrum. Let P(fi) be the power spectral density the i-th discrete frequency bin, where = L,... ..... .N and N is the total number frequency bins obtained from the discrete Fourier transform. We normalize the spectrum to obtain probability mass function
p(F) PU.T
The frequency entropy is calculated as
H = == = loga Pi-
(3)
When desired, H can be normalized by log2 N to yield H that the spectral energy is concentrated in a few frequencies and is therefore highly ordered- for example, as in a pure whne-ah high value indicates that the energy is spread across many frequencies and is therefore more disordered, as in white noise. Thus, frequency entropy provides single quantitative measure of the flatness or peakiness of a spectrum. This metric is widely used in signal processing and information theory (Misra al., 2004; Sucie al., 2014).
low value of indicates
3 FREQUENCY ENTROPY: A NEW LENS FOR ROTARY POSITION EMBEDDINGS
After RoPE, each pair is rotated by an angle n with 82/2-1 determined by the RoPE base. We are interested in how strongly each rotary pair exhibits narrow-band, RoPE-driven periodicity along sequence, as opposed to content-driven, broadband variability.
Core idea. For each rotary pair, we construct ID block observable along the token axis and measure its frequency entropy (FE), defined as the Shannon entropy of the power spectrum. quantify the utilization of each dimension, we introduce two new evaluation metrics: Spectrum Frequency Entropy (SpectrumFE. and Sequence Frequency Entropy (SequenceFE), SpectrumFE Spectrru evaluates which frequency components are present any given moment. SequenceFE evaluates how periodic or irregular the energy fluctuations are in each dimension. Our FE computed as follows:
1. Split the query into d/2 RoPE blocks and compute the ε2-norm of each, treating the resulting length-L vector as discrete signal (Section 3.1).
2. Compute two variants of power spectrum and applying Shannon entropy (Shannon, 1948) its power spectrum as shown in Eq. (3) (Sections 3.2 and 3.3).
Low entropy indicates that the time series is dominated by a small number of frequencies, while high entropy indicates spectrally diverse, content-modulated dynamics. This yields per-pair score that is model-agnostic, scale-free, and comparable across layers, heads, and architectures.
3.1 FROM ROPE BLOCKS TO A DISCRETE FREQUENCY SIGNAL
To measure the usage of frequencies, we start by noting any Cauchy-Schwan equalities following (Barbero et al., 2025). The effect of the j-th frequency component on the activation A,,m is upper
bounded by the 22-norm of the query and key components, i.c., (g A()i)|
< ||q (3) ll2 | ll2
(j = 0....., 1). For fixed block j. we collect these vectors across the sequence by
== [111b]) 12,, 11 ( 112. .... (2 [ E RR..
(4)
where is the sequence length. Therefore, we assume that the set of no-mosss |1 after RoPE constitutes discrete signal. We calculate the frequency entropy of the d/2 discrete signal patterns and measure the frequency utilization rate. In practical terms, measuring the form of a query after RoPE natural: the rotation aligns the representation with the frequency blocks actually used in the logit and preserves norms, so || 2 both position-invarian and directly interpretable. We treat Eq. (4) as discrete-time signal.
(3) 1|2 of queries
4

---

Published as a conference paper at ICLR 2026
3.2 SPECTRUM FREQUENCY ENTROPY
We measure the spectral complexity of this sequence using normalized spectral entropy. For given signal s,, first compute the short-time Fourier transform (STFT) to obtain the power spectrum, as
F-1 Σ syltH + n| w k
Sk,
(5)
where t(t 2,1, 1) each frame, win] is the analysis window, and Sk, is the power spectrum at frequency bin k(k = 0, 1, .., K - We set the frame length 1' to 1024, hop length H to 512, and sequence length L to 4096. Therefore, the frequency bin is K = F + = 513 and number of frames 7 - to form probability distribution, as
7. Second, the power spectrum for each frame is normalized
SR Σ 2j=0
21= Sk.t
Ph
S
(6)
Sj
T
Next, the Shannon entropy H (Shannon, 1948) is calculated as
K
=
Σ Ps. log2Pt.
(7)
K=
To obtain scale-free measure, we divide by the maximal entropy Hmax = loga K, yielding the normalized spectral entropy H Hmaax The normalized spectral entropy H represents the Spectrum Frequency Entropy (SpectrumFF)) quantifying the temporal spectral diversity of the query L62-norm signal. Finally, since RoPE organizes the embedding into z two-dimensional rotary pairs, we compute H, for each pair index O....,
H_
Hence, the entropy defined along j and yields length- vector (Ho Hg 1), with one value per rotary pair.
3.3 SEQUENCE FREQUENCY ENTROPY
For given signal sy, first compute the discrete Fourier transform (DFT) to obtain the power spectrum as follows:
S₁
Σ sy s
k =,
(8)
Next, discard the DC component (k 0) and restrict to the positive frequencies 1 < Define the total frrquive-frequency energy and normalize to obtain a probability distribution over these frequencies:
< 4 - 1.
S
Ps = [1122-11
h: =
1
(9)
Σ k=
Sk
Finally, the Shannon entropy H (Shannon, 1948) is calculated as
[L/2]- -1
H = Σ A=1
Ps log, Pk-
(10)
To obtain a scale-free measure, we divide by the maximal entropy Hmax log, K. yielding the
normalized spectral entropy / = -- _H. Frequency Entropy (SequenceFE) quantifying the temporal spectral diversity of the query L62-norm signal. Finally, since RoPE organizes the embedding into d two-dimensional rotary pairs, we compute Hl for each pair index E ..... d 1. Hence, the entropy is defined along j and yields a length- vector (Ho.... Ag H4-1). with value per rotary pair.
F.... The normalized spectral entropy H represents the Sequence
5

---

Published as a conference paper at ICLR 2026
Layer, D Maaad
1 Layer, headd
Laper, D Heead
ProgArro Spetium
Dinemsion
Denomsice
Dineriion
Separc Pisparo Creepy
Dimaniion
Olnancce
Dimnndoon
Figure 2: Scatter plots of each FE value in layer 6, layer 1, layer (left right). Rows: SpectrumFF,, SequenceFF,, and query 2--norm map. Top/middle: pair index in RoPE j (x) vs. normalized entropy H, (y). Bottom: (y) vs. token index
model. Columns:
(x); color denotes 19% (5) 22. All results are shown for head O. The pair index j is rotation patterns.
Sequence length 4096.
4 ANALYSIS VIA FREQUENCY ENTROPY
In this section, we investigate the characteristic behavior of RoPE. To isolate their contribution, we compare queries that employ RoPE with those using no positional encoding (NoPE), enabling a direct identification of RoPE-specific effects. For this purpose, we primarily analyze the Llama-4 model with iRoPE, which alternates RoPE and NoPE across attention layers.
4.1 SETTINGS
We conduct experiments using - head dimension of 128 and RoPE with 64 rotation patterns, 48 transformer layers, and 40 attention heads. In the -- frequency-entropy values are computed. For evaluation, we sample text at random from the wiki tikiteext-13--aa--- nate the samples to form sequences of exactly 4096 tokens. Each such sequence is passed to the model, and we extract the attention query vectors during inference. We then compute each frequency entropy from these queries.?
(Meta, 2025) model with
model, total of (64 48 x 40) 122,880
split of the WikiText-103 dataset (Merity al., 2017) and then concate-
4.2 ANALYSIS RESULTS
What characteristics does SpectrumFE capture? Fig. 2, the top panel shows scatter plots of SpectrumFF per rotary pair, and the bottom panel shows the corresponding query 42-norm maps. In the query f2-norm map of layer 6, we observed contiguous stretches of rotary-pair indices with persistently elevated norms. We refer to contiguous ranges of rotary-pair indices that exhibit persistently high query l2 norms as frequency bands. Similar banded patterns were also reported by Barbero et (2025) and are especially evident positional heads. Dimensions with the smallest SpectrumFE align with pronounced band- limited patterns in the &--norm maps, indicating that
'The analysis of keys is in Appendix A, and the layer-wise analysis is in Appendix B. We also conduct the analysis models that apply RoPE in all layers and heads, including Llama-3, Qwen3, and Gemma2. See Appendix for details.
We additionally provide a context-length analysis in Appendix F a cross-dataset analysis in Appendix F.2, and ablation study comparing the RoPF-only and NoPF-only models in Appendix G.

---

Published as a conference paper at ICLR 2026
Fragannc) Estrow
Layer (RoPP))
Lager 3 (NoPP)
Layar (NesPP))
Dimonsion
Dimonsion
Dimension
Proguaccy Ennory
Dimession
Figure 3: Scatter plots of FE values across all heads in the 4 16E 11TB-166-Inntruee model. Columns: layer O, layer 3, layer 47 (left to right). Layer 0 is the RoPE layer, while the others are NoPE layers. Rows: entropy scatter plots for SpectrumFE and SequenceFE. Scatter plots show pair index in RoPE j (x) vs. normalized entropy H, (y). We vary the color intensity of the scatter points by head. The pair index j is rotation patterns.
SpectrumFE captures the frequeney-band structure. Across RoPE-applied layers, SpectrumFE values predominantly lie in the range 0.2-0.6. The SpectrumFE distribution in the NoPE layer (layer 3) yields a markedly different profile from the RoPE layers. In the NoPE layer (layer 3), multiple frequency bands are observed in the query 2--norm maps.
What characteristics does SequenceFF capture? Fig. 2, the middle panel presents scatter plots of SequenceFF per rotary pair. In layer 0 (RoPE layer), pairs with the smallest SequenceFF exhibit clear periodic oscillations in the corresponding query f--norm maps, whereas in layer 3 (NoPE layer), no periodic pattern is observed and SequenceFE values rarely fall below 0.8. These observations indicate that SequenceFE is sensitive to periodic structure along the token axis. Across RoPE-applied layers. SequenceFE predominantly ranges from 0.2 0.6. In contrast, in the NoPE layer (layer 3), no periodicity emerges and the SequenceFE distribution concentrates near 1.0, yielding a markedly different profile from the RoPE layers (layers and 6).
Effect of the NoPE Layer The bottom row of Fig. 2 shows query f2-norm heatmaps indicating that RoPE layers and the NoPE layer layer exhibit markedly different behavior. From SpectrumFE, the shallow NoPE layer (layer 3) exhibits more frequency bands than the RoPE layer. Meanwhile from SequenceFE, the NoPE layer does not show rotating pairs and therefore does not exhibits clear periodic oscillations.
Entropy Landscapes Across Layers and Heads Fig. shows scatter plots each FE per rotary pair in all heads. In the SpectrumFE plot, the shallow NoPE layer (layer 3) exhibits more dimensions with frequency-band-io tion is widely scattered. However, the layer deepens (final layer 47), the number of dimensions with band-like characteristics decreases, and the SpectrumFF distribution converges within a certain range. In the SequenceFF plot, the shallow RoPE layer (layer 0) indicates that there are several periodic dimensions. However, in the NoPE layer, this periodic dimension disappears for all heads, regardless of whether the layer is shallow or deep. These results suggest that NoPE may attenuate the periodic structure characteristic of RoPE and emphasize frequency bands.
characteristics than the RoPE layer. Furthermore, the SpectrumFE distribu-
4.3 WHY SPECTRUMFE SHOWS BANDS AND SEQUENCEFE SHOWS PERIODICITY?
In summary, our analysis separates two types of structure in RoPF-driven attention: Spectrum Frequency Entropy reveals band-focused allocation across rotary pairs, and Sequence Frequency Entropy reveals tokenwise periodicity. NoPE may weaken the latter while preserving or highlighting the former. Deep layer may reduce both band sharpness and periodicity.
Why SpectrumFE shows bands? SpectrumFE takes a short-time spectrum of the query-norm for each rotary pair by STFT and measures the Shannon entropy frequency bins. SpectrumFF
7

---

Published as a conference paper at ICLR 2026
measures how narrowly concentrated the local frequency content (via STFT) is. Entropy is low when energy sits in a few bins and high when it is spread out. Therefore, plotting low SpectrumFE across indices (bins) forms contiguous low-entropy region, i... frequency band. Low SpectrumFE indicates strong frequency-bant structure, meaning the model consistently allocates energy to specific rotary pairs.
Why SequenceFE, shows periodicity? SequenceFF uses global Fourier spectrum along the token axis by DFT and measures the entropy over positive frequencies. SequenceFE measures global periodicity of the GoPE-transformet signal (via DFT). Low SequenceFE indicates near-single- single-tone oscillation driven by RoPE's fixed rotational phase, rather than content. Low entropy means a simple, near-single-tone signal, and high entropy means complex noise-like signal. With RoPE active, rotary pair advances at an almost constant step per token, so the query oscillates at a fixed frequency, If we remove RoPE, the fixed-rate oscillation vanishes, energy spreads, and SequenceFE rises.
Why bands can persist in NoPE for SpectrumFE? Even without rotational oscillation, the model may up-weight some rotary pairs due to content or architectural bias. This uneven allocation still concentrates energy for those indices within short windows, keeping SpectrumFE low over a contiguous set of indices. At the same time, the per-token signal is not strongly periodic, so SequenceFE remains high.
FILTERING OUT OUTLIER DIMENSIONS
Low SpectrumFF reflects band-limited behavior, while low SequenceFT reflects strong periodicity. The following question arises: Are frequency bands and periodicity redundant elements for the model, or are they essential components? In this section, guided by Frequency Entropy, we intervenc in RoPE by selectively weighting these rotary pairs to mitigate their contribution.
5.1 WEIGHTED RoPE
If Frequency Entropy is below a certain threshold, we weight the corresponding RoPE dimension to reduce its effect. We call this Weighted RoPE. Let (1.b))) [0, denote the Frequency Entropy in the query for layer I, head h, and rotary pair j. Given a threshold E (0, 1) and RoPE weight [0, o........ 0.9], set weighted factor as follows:
ahi = {a,
[a, Ath. <T,
(11)
otherwise.
We modulate the RoPE transformation applied to the j-th rotary pair of the query at token position m. Let p((.b..) 2x2 denote the standard RoPE rotation for pair in layer and head h. We obtain the weighted query subvector by scaling the usual rotation with Lhhjj, which acts as a soft mask that attenuates the contribution of low-entropy pairs while leaving others unchanged, as
1 + = Lhi pil.b..)
(.t.i.ghhh. m, qm
(12)
Intuitively, for low-entropy pairs, we slow the phase growth and attenuate periodicity; for other pairs, RoPE remains unchanged. We calculate the FE for each key and perform the same processing.
5.2 PERPLEXITY
Settings Four publicly available large language models were used for the evaluation: --ama---Scott178--11 (Meta, 2025), (Yang et al., 2025), and Meta-Llama-3-8 (Grattafiori et al., 2024). We evaluated the inference perplexity the test set of the wikitext-103 103 dataset (Merity et al., 2017). For the threshold parameter т, we adopted different settings according to the entropy metric. For Spectrum Frequency Entropy, examined two regimes: Sequence Frequency Entropy, we observed that outliers occurred only at low values, so we evaluated the case where less than 0.6. No fine-tuning was performed in any of the experiments.
(Gemma Team et al., 2024b), Qwen3-8B
where
greater than 0.4 and one where
less than 0.2. For
8

---

Published as a conference paper at ICLR 2026
Spectrum FE 0.2
Spectrum T 0.4
SequenceFE T.0.6
12
85
0
7.6
73
Figure 4: Perplexity as a function of the RoPE weight a under entropy based gating. The horizontal axis shows the weight in Weighted RoPE and the vertical axis shows perplexity. We evaluate two settings for Sequence FE: 0.4 and < 0.2. For Spectrum FE, we use 0.6 because outliers occur only at low values.
Table 1: Downstream task performance in original RoPE and WeightedRoPE.
Model
HellaSwag
TruthfulQA
MMLU
haseline
WeeightedRoPF
baseline
+ WWeightedRoPF
haseline
+WeightedRoPF
Llama 4 17B
66.67
66.67
97.32
97.99
60.05
60.81
Llama-3 8B
60.16
60.16
84.85
84,85
34,21
34.21
Qwen3
58.92
95.31
Gemma-2 2 9B
61.02
61.21
98.83
98.83
72.81
72.81
Results To examine the contribution of outliers, we plotted the perplexity for each weight a to visualize how different weighting values affect model performance. Figure 4 presents the overall perplexity results. When SpectrumFF reduced the dimensions with r < 0.2, perplexity increased as the weight a decreased. This indicates that dimensions with T < 0.2 in SpectrumFE contribute to model performance and may be important components for the model-in other words, frequency bands may be important. Furthermore, when SpectrumFE reduced dimensions with T > 0.4, perplexity decreased slightly the weight became smaller. However, the overall performance remained nearly identical, suggesting that dimensions with outlier values of T > 0.4 may be unnecessary or redundant for the model. Next, when SequenceFiß reduced dimensions with < 0.6, perplexity decreased slightly as the weight a became smaller. Notably, the decrease in perplexity was larger for the Llama-4 model than for the other models, suggesting that periodicity may be unnecessary or redundant for the model. The Llama-3 model had a smaller impact on perplexity, but the trend was the same as other models.
5.3 DOWNSTREAM TASK
Settings Based on the above experimental results, we hypothesize that the outler dimensions of SpectrumFE for 0.4 and the periodic dimensions of SequenceFE for 0.6 are redundant. To investigate whether these dimensions affect downstream tasks, we evaluated Weighted RoPE across multiple tasks. We fixed to 0.1 for Weighted RoPE, applying the weight a to both the outlier dimensions of SpectrumFE at > 0.4 and the periodic dimensions of SequenceFE at T < 0.6. Performance measured diverse set of benchmark tasks including HellaSwag (Zellers et al., 2019), TruthfulQA (Lin et al., 2022), and MMLU (Hendrycks et al., 2021) 1) to assess both reasoning and factual capabilities.
Results Table lists the experimental results. For all tasks, no significant difference was observed between RoPE, which performs no operations, and WeightedRoPF. Only the llama-4 model showed a slight improvement in performance. This suggests that the dimension where SpectrumFF becomes an outlier and the periodic dimension where SequenceFE decreases may not contribute to model performance and could be redundant.
We additionally provide an evaluation of long-context generation the Needle-in-a- Haystacck task in Appendix F.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
In this work, we introduced frequency entropy (FE), a metric that quantifies the effective utilization of each RoPE frequency dimension, and analyzed how the sinusoidal components of RoPE contribute to the model representation. SpectrumFE can identify the frequency band of RoPE, and reducing the contribution of this band degrades model performance, indicating it is crucial component. Furthermore, SequenceFE can identify the periodic dimension of RoPE. Reducing the contribution of this dimension does not change model performance and may even improve it for some models. This suggests the periodic dimension may be redundant or unnecessary. Furthermore, these frequency bands and periodic dimensions do not exist fixed dimensions. This suggests that some incon- sistencies in prior work may stem from model-dependent frequency bands whose locations differ across heads and layers, rather than from absolute "low" or "high" frequency effects. Our framework provides systematic method for interpreting such mixed features individually. Moreover, the fact that Now-SequenceFE dimensions can be attenuated with minimal degradation indicates that these components reflect periodie signals induced by RoPE that the model does not functionally use. This finding highlights potential applications to RoPE-aware KV-cache compression and dimension pruning, and it offers basis for future exploratory work frequency-targeted architectural variants. We expect FE to function as a practical, model- pooell-indeeendent diagnostic tool for positional encoding.
REFERENCES
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling, In International Conference on Learning Representations, 2019. URL https: https: /oppr///ewweeeeee.....o net/
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference on Learning Representations, 2025. URL. forummid==tvuurkk58.
wwennreview.neee net/
Ting-Rui Chiang and Dani Yogatama. The rotary position embedding may cause dimension ineffi- ciency in attention heads for long-distance retrieval. In Findings of the Association for Compu- tational Linguistics: ACI. 2025, pp. 13552-13562, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 89177----97----555 URL. htt.htttss//aclaattoologgg org/ g/202 findings-ach 697/.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context, 2024. URL https://aaxis: orrg abs/2402. 10171.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Chone
Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George- Christian Muraru, Grigory Rozhdestvenskly., Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang. Oscar Wahltinez, Paige Bailcy, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comaneseu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec. hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology, 2024a. URL https://arxivv http:/ii..org/bb////0000 240.s/029.......... 08295.
George-Christian
10

---

Published as a conference paper at ICLR 2026
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot. Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanezyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, lean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Danila Sinopalnikov, David Weinberger,
Perry, Chris Welty, Christopher A. Choquette- Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimezak-Plucinske Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan. Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani. Natalic Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnod, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Hli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at a practical size, 2024b. URL https://arrxv..or///b///222. 00118.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes. Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee. Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes
Plawiak,
11

---

Published as a conference paper at ICLR 2026
Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic. Peter Weng, Prajwwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnie, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa. Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos. Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu. Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandami, Annie Dong Annie Franco, Anuj Goyal, Aparajita Saraf. Arkabandhu Chowdhury. Ashley Gabriel, Ashwin Bharambe, Assaf Fisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence. Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiane Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Fric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard. Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang. Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manay Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark. Mike Macey, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White. Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang. Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta,
12

---

Published as a conference paper at ICLR 2026
Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang. Wenwen Jiang. Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary De Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https://arxx org/ arrg/bbs2407 2 21783.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL http:: /tppse//openrreiiew.
wwwpenrwwwwweweeeeeeeeeeeeeeeeee wwtp///ent//mineeeeeeoeo j 3GmQ.
Xiangyu Hong, Che Jiang, Biqing Qi, Fandong Meng. Mo Yu, Bowen Zhou,, and Jie Zhou. On the token distance modeling ability of higher RoPE attention dimension. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 5877-5888, Miami, Florida, USA, November 2024. Association for Computational Linguistics. URL https: httt::://aclanthoooggg org/ 2024. findings-neiinnnn eepes.-emnp..... 338//
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization transformers. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL /ttpss:/opperreview wwtp://eteooommmeeeemm Drr12gcjzl.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. URL httttttts///:caanteee https:///lacaaaalloaaoaoo org/ 22t//22222222 acl long 229/
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mix- ture models. In International Conference on Learning Representations, 2017. URL https: wttp:/te//oommmddoyyyy/....o
Meta. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https:
//ai meta. awweea.coo/bbaa cwm/blog/////ooocococcooocoooooo htth//loolllllla heth:llamaa4440 4-mul timodal-
2025. Accessed: 2025-09-11.
H. Misra, S. Ikbal, H. Bourlard, and H. Hermansky. Spectral entropy based feature for robust asr. In 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pp. 1-193, 2004.
Yui Oka, Taku Hasegawa, Kyosuke Nishida, and Kuniko Saito. Wavelet-based positional representa- tion for long context. In The Thirteenth International Conference on Learning Representations, 2025. URL htttps://opeerewweee http://///pppeeeeeeeeeeee/e//o/d
Myle Out, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: fast, extensible toolkit for sequence modeling. Proceedings of the 2019 Conference the North American Chapter of the Association for Computational Linguistics (Demonstrations), pp. 48-53, Minncapolis, Minnesota, June 2019. Association for Computational Linguisties. URI. htw.hhttps::/aclaattoloooooo http///aaaanntoooooooooooooooala org/No/////// //Tg/N115 N19 NNNNNNN999////
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022. URL
https://openreet
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025, URL https://arx:: httpss//aaxvv.oo//abb////// 1511. 15115.
13

---

Published as a conference paper at ICLR 2026
Jack W. Rae, Anna Potapenko. Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2020. URL https////ppppereeeeeeee//rrrrrrn
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1 67, 2020. URL http://jmlr.. http:///inrrrrg//////// pttpppppp/////20///ppp 074. 4.ml..
C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3): 379-423, 1948.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023, URL
.99664.
Victor Sucie, Nicoletta Saulig, and Boualem Boashash. Analysis of local time-frequency en- tropy features for nonstationary signal components time supports detection. Digital Signal Processing, 34:56-66, 2014. URL http:///wwwwsiieeeereeee... http://deeettcccocccc///eeeee/ee /cts/nnc//aticc/eppili 1105120044 4002292.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https::////aaatet .ttlvvorrrgro.... /rggabs///// 232// 09288.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Process- ing Systems, volume 30. Curran Associates, Inc., 2017. URL https: / http::::////ccceeeee htwprrrreeeeniinnnneiirirsrppss. neurips. ec/paper_files http://aeer/211/11////ff/eeee
1 14d05331 pdf.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective longcontext scaling of foundation model. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langage Technologies (Volume I: Long Papers). pp. 4643-4663, Mexico City, Mexico, June 2024, Association for Computational Linguistics. URL httph: accps///ololl or org/2024.00 naacl- long. 260
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: //wxvv.r/vvvv///////////a././av wrg. /20d../2555..0 09388.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pp. Linguistics. URL httpttt:/aacaaathllloooo hwwpoooolgyooooog/ggeee org/P19-1442//
Florence, Italy, July 2019. Association for Computational
14

---

Published as a conference paper at ICLR 2026
A ADDITIONAL EXPERIMENTS ON KEY
We also compute our spectral entropy (SpectrumFB and SequenceFE) from the keys in (Meta, 2025) model. The analysis procedure is the same
as in Section 4.
Analysis Results Figure 5 shows the scatter plots for each FE and the fo-norm heatmap for the keys. RoPE layers, we observe both frequency bands and periodic dimensions in queries and in keys. FE aligns with these observations and shows the same trend for keys. Keys contain larger number of periodic dimensions than queries (e.g. 5-th Layer). In NoPE layers, we see the same qualitative pattern as in queries: periodic dimensions are rarer, while frequency bands detected frequently.
Layer.
Layer, Head
Dimonsion
Demension
Tokon
Demeratio
Demeraiion
7 Layer, D Head (NoPE)
Layer, 0 Head (RoPE)
Froqueccy Entroyy
Dimonsion
Soccurnt Frequeery Enimayy
D mierrsionn
Dimerrsion
Tokon
Tokun
Dimamatoon
Dimarasion
Figure 5: Scatter plots of each FE value in the layer 6, layer 1, layer 3 (left to right). Rows: SpectrumFE, SequenceFE, and key fo-norm map. Top/middle: pair index in RoPE j (x) vs. normalized entropy H, (y). Bottom: j (y) vs. token index (x); color denotes ||&[51] ||2. All results shown for head 0. The pair index j is rotation patterns. Sequence length = 4096.
model. Columns:
15

---

Published as a conference paper at ICLR 2026
B ADDITIONAL EXPERIMENTS ON ALL LAYERS
Figures and 7 show the results SpectrumFF and SequenceFE: across all layers of the -lama-4SScuut-7B--BB model.
SpectrumFE in all layers First, we discuss the scatter plots for SpectrumFE in Fig. 6. Comparing the shallow RoPE layer and the NoPE layer, the NoPE layer exhibits significantly broader distribu- tion, whereas the RoPE layer's distribution is less extensive. However, as the layer depth increases, the NoPE layer's distribution converges. Even in the RoPE layer, the distribution converges, though not as much in the NoPE layer, suggesting the influence of deepening. On the other hand, low SpectrumFE values are observed regardless of layer depth, indicating that the frequency band is observed in every layer.
SequenceFE E in all layers Next, we discuss the scatter plots of SequenceFE in Fig, 7. Comparing the shallow RoPE layer and the NoPE layer, the RoPE layer exhibits significantly broader distribution, while the NoPE layer's distribution is not as wide. This is exactly opposite to the trend observed in FFectrumFE. However, as the layer deepens, the NoPE layer's distribution widens, indicating that periodic dimensions temporarily emerge in the NoPE layer. Near the final layer, however, the periodic dimensions the NoPE layer diminish. The RoPE layer maintains certain number of periodic dimensions even at deeper layers. These periodic dimensions primarily exist near high frequencies. However, since these high-frequency periodic dimensions are not observed in the NoPE layer, NoPE may play role in mitigating periodic dimensions.
16

---

Published as a conference paper at ICLR 2026
RoPE layer
NoPE layer
Narraw
Deep
Laye scatter plots of SpectrumFE across all attention heads in The figure contains 48 panels arranged in 12 rows 4 columns, with layer depth increasing left-to-right and then top-to-bottom (layers 0-47). Layers 3, 7.
Figure 6:
11, 15, 19, 23, 27, 31, 35, 39, 43, and 47 use NoPE; all other layers RoPE.
17

---

Published as a conference paper at ICLR 2026
RoPE layer
NoPE layer
Narraw
Deep
Figure 7:
Layer-wise seatter plots of SequenceFF across all attention heads in -ot---oooooot---------
The figure contains 48 panels arranged in 12 rows 4 columns, with layer depth increasing left-to-right and then top-to-bottom (layers 0-47). Layers 3, 11, 15, 19, 23, 27, 31, 35, 39, 43, and 47 use NoPE; all other layers use RoPE. 18

---

Published as a conference paper at ICLR 2026
C ADDITIONAL EXPERIMENTS ON OTHER ARCHITECTURES
We performed frequency entropy analysis not only on 4 4-sota-t--11---1111 on Me - gemma-2-9b-it, and Qwen3-8B. Note that unlike Llama-4, these models do not possess NoPE layer. All experimental settings are the same as in Section 4.
model but also
Llama-3 Figure 8 shows the FE analysis for the Llama-3 model. Only selected salient features are marked. First, consistent with Llama-4 in Section 4, frequency bands are also observed in Llama-3. These bands appear in most heads. In contrast, no periodic dimensions are identified, and SequenceFE remains consistently high overall. The strongest band typically appears between the 39th and 42nd dimensions on average, which differs from the band locations in the RoPE layers of Llama-4.
Gemma-2 Figure 9 shows the FE analysis for the Gemma-2 model. Frequency bands observed and they appear in most heads. In contrast, no periodic dimensions are identified, and SequenceFE remains consistently high overall. The strongest band typically appears between the 115th and 120th dimensions on average.
Qwen-3 Figure 10 shows the FE analysis for the Qwen-3 model. Frequency bands are observed and they appear in most heads. In contrast, periodic dimensions are identified, and SequenceFE remains consistently high overall. The strongest band typically appears between the 48th and 50th dimensions average.
Frequency bands appear across models yet their locations are model dependent. In Llama-3, Gemma- 2, and Qwen-3, bands occur in many heads while the peak dimension differs across models, which suggests that band position is governed by the RoPE base, the training length, and architectural factors such dimensionality and head configuration. In these model, periodic dimensions are ineffective at most very limited, since SequenceFE remains consistently high and clear periodic components are observed. Attenuating or pruning these periodic dimensions likely cause only small drop in performance, which is consistent with the downstream task results in Section 5.3. Moreover, since the band detected in most heads and explicitly reducing the contribution of the band dimensions lowers performance (as shown in Section 5.2), we conclude that a specific frequency range is commonly useful for attention computation.
Layer
Prequercy Spectrom Proqueno
Demmicn
Derwadoo
Dimesiinn
Figure 8: Meta-Llama-3 3-8B model (head 0). Columns: layer 0 and layer 4. Rows: SpectrumFE, SequenceFE, and query l2 form map. Top/middle: pair index (x) vs. normalized entropy H, (y).
Bottom: j (y) vs. token index n (x); color denotes || 2|2. The pair index j is rotation patterns. Sequence length L = 4096.
19

---

Published as a conference paper at ICLR 2026
Layer
Layer
Direroion
Dimerstoo
Cimenilon
Dimandoon
SEBEBE
Figure 9: Gemma-2-9b-it model (head 0). Columns: layer 0 and layer 16. Rows: SpectrumFE, SequenceFE, and query fo-m map. Top/middle: pair index (x) vs. normalized entropy H, (y).
Bottom: j (y) token index n (x); color denotes |1 ||2. The pair index is rotation patterns. Sequence length 4096.
Layer
Layer
Figure 10: Owen-3-8B model (head 0). Columns: layer 0 and layer 35. Rows: SpectrumFF, SequenceFE, and query fz-norm map. Top/middle: pair index (x) vs. normalized entropy H, (y).
Bottom: j (y) token index (x); color denotes |1 ||2. The pair index is rotation patterns.. Sequence length = 4096.
20

---

Published as a conference paper at ICLR 2026
D LIMITATION
While our work provides a unified analysis of RoPE's frequency usage, several limitations remain. Our intervention weighted RoPE does not fully disentangle whether performance preservation under SequenceFEE-assee attenuation reflects true redundancy or compensation from neighboring frequency components. A more granular causal test, such as single-pair perturbations, layer-specific inter- ventions, or head-wise isolated ablations, would be required to conclusively rule out compensatory mechanisms. Finally, although we evaluate four architectures applying RoPE in all layers, our causal claims are limited to the scope of inference-only interventions. Our results should therefore be interpreted as providing first-order causal evidence rather than a complete causal theory of positional encoding. We regard the development of more precise, minimally confounded causal perturbations as an important direction for future work.
E ANALYZING LONG-CONTEXS BEHAVIOR THROUGH WEIGHTED ROPE
We evaluated Weighted RoPE using downstream benchmarks such as HellaSwag and MMLU in Section 5.3. These tasks primarily test general knowledge and reasoning, and therefore may not be sensitive to differences in positional encoding. To directly address this concern, we evaluated the impact of Weighted RoPE long-context understanding in setting explicitly designed to probe positional robustness. We used the Needle-in-a-Haystaee (NIAH) task 4, which is widely used in studies of positional interpolation and long-context evaluation. Given the available GPU memory, we evaluate Llama-4 up to 33,564 tokens and Llama-3 to 59,615 tokens, which correspond to the maximum feasible context lengths under our inference setup.
E. EXPERIMENTAL. SETUP
We evaluate Llama-3 and Llama-4 under Weighted RoPE intervention in the NIAH setup. The Weighted RoPE settings are the same as in Section 5.3. We used the implementation of (Fu et al., 2024).
E.2 RESULTS
The results for Llama-3 are shown in Figures 11 and 12, and the results for Llama-4 are shown in Figures 13 and 14.
Consistent with the downstream results reported in Table I, Llama-3 showed no observable differences between the baseline RoPE and Weighted RoPE across all evaluated context lengths. The retrieval accuracy curves as well as the heatmap structure were effectively identical. This suggests that the RoPE dimensions identified by SpectrumFF as outliers or by SequenceFE as low-periodicity contribute little to Llama-3's 's long-context retrieval. In other words, the dimensions we suppress or down-scalc appear truly redundant for this model, both in short-range downstream tasks and in explicit long-context settings.
For Llama-4, the overall retrieval pattern remained stable, but Weighted RoPE showed slight improve- ment around the 40ketoken context. In the baseline model, this region appears cluster of red cells indicating retrieval failure; the Weighted RoPE produces milder failure or partial recovery.
These results provides two insights. (1) Long-context behavior can be affected by localized frequency- band adjustments. Unlike Llama-3, Ilaaa-4 exhibits more sharply defined frequency-bant structures in early layers. Weighted RoPE interacts with these bands, and selectively attenuating low-importance rotary pairs can mitigate localized positional instability. (2) Weighted RoPE does not harm long- context performance and may correct fragile eegions.Reviewe concerns about potential degradation on long-context-speciin tasks are therefore addressed: no regressions were observed, and in Llama-4 certain failure modes were slightly improved.
21

---

Published as a conference paper at ICLR 2026
Figure 11: Results for NIAH on Llama-3-88
Figure 12: Results for NIAH task on Llama-3- -- with Weighted RoPE.
/
Figure 13: Results for NIAH task on Llama 1---------- ScoSSSoottttsooeeeeo 10ee-117111651 109-16EIInstrutt
8
Figure 14: Results for NIAH task on Llama -4 Scwut-cotttt----- with Weighted RoPE.
22

---

Published as a conference paper at ICLR 2026
F ADDITIONAL EXPERIMENTS ACROSS DATASETS AND CONTEXT LENGTHS
We extend the analysis via frequency entropy in Section 4 on two additional datasets and across multiple context lengths. In addition to Wikitext-103, we conduct experiments on the C4 dataset (Raffel et al., 2020) which consists of English text from Common Crawl 9 and the PG19 book corpus (Rae al., 2020). We evaluate models at three context lengths: 2048, 4096, and 8192 tokens.
F.1 ANALYSIS ACROSS CONTEXT LENGTHS
The results for Llama-4 using Wikitext- 103 are shown in Figure 15. When we vary the context length, we observe slightly differences in SpectrumFE, SequenceFE, and the 2-norm maps. These changes expected because the imput text itself differs across context-length settings. However, the overall distribution of both SpectrumFE and SequenceFE remains largely unchanged. The frequency bands captured by SpectrumFF persist across all context lengths, and their locations remain stable. Similarly, the periodic patterns captured by SequenceFF: also remain present at every context length, with their positions unchanged. As shown in the left 2-norm maps of Figure 15, the visual appearance of the periodic patterns changes as the context length varies, but the underlying dimensions where these patterns occur stay the same. Taken together, these findings indicate that both SpectrumFE. and SequenceFE consistently can capture their respective structural properties, and these properties do not shift when the context length changed.
F.2 ANALYSIS ACROSS DATASETS
Figure 16 and Figure 17 show the Llama-4 results C4 and PG19 respectively, evaluated at context lengths of 2048, 4096, and 8192 tokens. We observe no major differences across context lengths, and changing the dataset similarly leaves the overall distributions of SpectrumFE and SequencelT largely unchanged. The frequency bands captured by SpectrumFE persist in both the C4 and PG19 datasets, and their locations remain stable. Similarly, the periodic patterns captured by SequenceFE continue to appear in both datasets, with their positions unchanged. Taken together, these findings indicate that both SpectrumFE and SequenceFE consistently can capture their respective structural properties, and these properties do not shift when the input sequence is changed.
'We use the 'en' validation split from the processed version provided the following URL: https:// huggingface co/datasets/allena
hhttps:// http://coooooccwlocrrromo org/
23

---

Published as a conference paper at ICLR 2026
Spectum Posparta
Denrrioon
Wikitext- 103 dataset with context length 2048.
(b) Wikitext- 103i dataset with context length 4096.
Oenanatioon
(c) Wikitext- 103 dataset with context length 8192.
Figure 15: Scatter plots of each FE value in the Llama-4-Scout-178- 170--11777------trr- model. Columns: layer 6, layer 1, layer 3 (left to right). Rows: SpectrumFE, SequenceFE, and query Ey-norm map. All results shown for head 0.
24

---

Published as a conference paper at ICLR 2026
Than
(a) C4 dataset with context length L 2048.
(b) C4 dataset with context length L 4096.
Toan
Drambe
Derenatoo
Dirembr
(c) C4 dataset with context length L. = 8192.
Figure 16: Scatter plots of each FE value in the Columns: layer 6, layer 1, layer 3 (left to right). Rows: SpectrumFF, SequenceFE, and query fo-m map. All results are shown for head 0.
model.
25

---

Published as a conference paper at ICLR 2026
PG19 dataset with context length 2048.
Thwn
(b) PG19 dataset with context length L 4096.
(c) PG19 dataset with context length L = 8192.
Figure 17: Scatter plots of each FE value in the --ama-4-Scutt1778-81 -BB-rrt-Instruc model. Columns: layer 6, layer 1, layer 3 (left to right). Rows: SpectrumFF, SequenceFE, and query no-m map. All results are shown for head 0.
26

---

Published as a conference paper at ICLR 2026
G FREQUENCY ENTROPY COMPARISON BETWEEN ROPE-ONLY AND NOPE-ONLY MODELS
Llama-4 interleaves RoPE and NoPE layers, causing the effects of the positional schemes to be mixed within the same network. To disentangle their individual contributions, we pretrain two small models from scratch: one that uses only RoPE and another that uses only NoPE. We then evaluate their layer-wise Frequency Entropy (FE) to examine the independent behavior induced by each encoding.
G.1 EXPERIMENTAL SETUP
For pre-training from scratch, we perform comparative evaluation with Transformer-based language model (Baevski & Auli, 2019). The dimensionality of the word embedding dmodee is 1024, the number of heads N is 8, the dimensionality of the heads d is 128, and the number of layers is 16. This implementation used the fairseq (Ot et al., 2019)-based code provided in a previous work(Press al., 2022), and all hyperparameters were et al., 2022). We use the Nesterov's Accelerated Gradient (NAG) optimizer with momentum 0.99, following the Fairseq nag implementation in (Ott et al., 2019). The learning rate schedule is a cosine scheduler: it is initialized at lc-7, linearly warmed up to 1.0 during the first 16,000 updates, and then decayed with a cosine schedule down le-4. The value 9216 denotes the maximum number of tokens per batch per GPU, with sequence length of 512 tokens. We train for 286,000 updates, which corresponds to approximately 205 epochs on WikiText-103. We used the WikiText- 103 dataset (Merity et al., 2017), which consists of over 103 million tokens of English Wikipedia articles. This setup used in (Press et al., 2022; Oka et al., 2025)? ?, where the goal is to analyze structural differences in positional encodings.
the same values those in the literature (Press
G.2 RESULTS
Figure 18 reports the FE measurements for layer 0 of the RoPE-only model, and Figure 19 reports the results for its final layer, layer 15. Figures 20 and 21 provide the corresponding FE results for the NoPE-only model.
First, in the RoPE-only model, the SpectrumFE results in the top row reveal clear frequency bands. The locations of these bands vary across heads and layers, but the band structure itself is consistently present in every head and layer we examined. This behavior matches what we observed in the RoPE layers of Llama-4, as well as in Llama-3, Gemma, and Qwen. In contrast, SequenceFE remains high across all dimensions, indicating little periodic structure. This differs from Llama-4 but aligns with the RoPE-only behavior reported for Llama-3, Gemma, and Qwen in Appendix C.
Next, the NoPE-only model, the SpectrumFE results show that frequency bands do not appear in all heads. Overall SpectrumFE values are lower than in the RoPE-only model, and the number of dimensions forming identifiable bands is also smaller. Some heads show no band structure at all. For example, the right-side in Figure 18 (0 Layer, 7 Head) displays pattern closer noise in its 2-norm map. Moreover, in the final layer, the frequency bands disappear entirely. SequenceFE. remains high throughout, showing no periodic patterns. This is expected because NoPE does not introduce any periodic information.
These observations suggest that NoPE-only model tends weaken wash out frequency bands as depth increases. RoPE. in contrast, consistently maintains band structure even in deeper layers. Finally, architectures like Llama-4 that interlcave RoPE and NoPE layers, NoPE may reinforce the band structure introduced by RoPE while also amplifying the periodic components associated with RoPE. This interaction between the two positional schemes may explain the distinct FE patterns observed in mixed-layer settings.
27

---

Published as a conference paper at ICLR 2026
Layer, 0 Hoad
0 Layer, Head
Spectrum Frequercy
Dimension
Dimension
Scquenes Frequency
Dimension
Diménsion
Dimension
Dimension
Figure 18: Frequency Entropy (FE) scatter plots for RoPE-only model pretrained from scratch. Shown are the Oth and 7th heads of layer O. Rows depict SpectrumFE, SequenceFE. and the query 22-norm map.
Layer,
Layer, Head
Sequence Frequercy Spectrum Procusroy
Dimension
Dimonsion
Dimension
Dimension
Figure 19: Frequency Entropy (FE) scatter plots for RoPE-only model pretrained from scratch. Shown are the Oth and 7th heads of last layer. Rows depict SpectrumFE, SequenceFE, and the query fn-n map.
28

---

Published as a conference paper at ICLR 2026
0 Layer, Head
Layer, 7 Head
Spoctrum Froquency
Dimension
Dimenioo
Sequerce Fresurtoy
Dimension
Dimencion
Dimension
Figure 20: Frequency Entropy (FE) scatter plots for NoPE-only model pretrained from scratch. Shown are the Oth and 7th heads of layer 0. Rows depict SpectrumFE, SequenceFE, and the query f--norm map.
Layer,
Layer,
Spectrum Frecueney
Dimonsioo
Dimonson
Sequence Entrayy
Dimonsion
Dimonsson
Dimansion
Dimansion
Figure 21: Frequency Entropy (FE) scatter plots for NoPE-only model pretrained from scratch. Shown are the 0th and 7th heads of last layer. Rows depict SpectrumFF, SequenceFF and the query fg-norm map.
29


Reviews:
[{"review_id": "eGpuPe5MLt", "reviewer": "Reviewer_G1Jc", "summary": "This paper provides an analysis of RoPE in terms of Frequency Entropy (FE), or the Shannon entropy of the power spectrum. Two measures are used in this analysis: Spectrum FE which captures the active components and Sequence FE which quantifies the regularity of energy in each dimension. These quantities are normalized to provide scale-free measures. An analysis of RoPE in Llama-4 is presented. This analysis shows that RoPE and NoPE layers behave differently, but that earlier NoPE layers show bands under Spectrum FE similar to RoPE. This effect disappears at deeper layers. The paper also describes experiments to test the redundancy of RoPE dimensions with outlier FE by down-weighting them. Results show that low entropy dimensions (frequency bands) are important to model perplexity, though downstream tasks show similar performance between standard and weighted models.", "questions": "Did you consider attenuating outlying low or high-frequency components only? This could help strengthen the case that these specific component regions are not responsible for model performance.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper provides an interesting, theoretically grounded analysis of RoPE embeddings. It provides new insights into the contribution of RoPE on model performance, specifically distinguishing between periodic signals and high energy bands. The explanations are clear and the experimental evidence helps to solidify the analysis.", "weaknesses": "The choice to study only Llama-4 reduces the impact and generality of this paper. It is noted that NoPE layers in this model may attenuate periodic structures and emphasize frequency bands. If I understand, Spectrum and Sequence FE analysis has not been applied to a RoPE-only model like Gemma 7B or Llama-3. This would help to clarify the effect of NoPE in Llama 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fl5YYamiOU", "reviewer": "Reviewer_USX3", "summary": "The paper introduces Frequency Entropy as a metric to study the utilization of RoPE frequency dimensions. The authors perform experiments on Llama-4 to compare the periodicity of RoPE and NoPE layers. The authors introduce Spectrum Frequency Entropy and Sequence Frequency Entropy to evaluate both which frequency components are present and how periodic the energy fluctuations are in each dimension.", "questions": "See weaknesses. In addition, prior work mentions that ablating low-frequency RoPE dimensions impacts long-context performance. I'd be interested to see the Weighted RoPE intervention in a similar setting to see if you could provide additional insights to long-context.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The methodology is interesting and provides a quantitative way to study RoPE utilization, as compared to prior more qualitative works\n- The figures provide nice visualizations of the analysis and results\n- The weighted RoPE experiments provide interesting insights on functional relevance, showing that suppressing low-Spectrum-FE dimensions worsens performance.", "weaknesses": "- The experiments are done on LLama-4 which interleaves RoPE and NoPE layers, so comparisons between RoPE and NoPE are not layer-matched. It would be interesting to see even at a smaller scale the differences between FE for a model trained with and without RoPE at a specific layer.\n- The takeaways to me are not clear. In particular:\n- The SpectrumFE results for RoPE and NoPE seem quite similar at later layers. It seems like the main difference is at the earliest layers. Do you have insights on the impact of this early-layer difference? Would a model with no RoPE layers exhibit the same late-layer behavior?\n- What are the practical takeaways of the Weighted RoPE intervention?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6qozSGROZm", "reviewer": "Reviewer_dKd1", "summary": "This paper introduces Frequency Entropy (FE), a novel metric to analyze how different frequency dimensions in Rotary Position Embeddings (RoPE) are utilized in Transformers. Through systematic analysis of models like Llama-4, the authors show that FE identifies two key structures: frequency bands (via SpectrumFE) and periodic dimensions (via SequenceFE). They find that while frequency bands are essential for model performance, periodic dimensions are often redundant and can be attenuated without harming downstream task accuracy, offering a unified explanation for previously conflicting findings and providing a practical diagnostic tool for positional encoding design.", "questions": "1. The sequence lengths in the paper are mostly set to 4096. Have you tried any other lengths, or does this length yield the most prominent effect?\n2. An in-depth analysis of the FE features (such as Figure 2) is mainly conducted for \"head 0\". Is it necessary to systematically extract samples from different heads at different layers to observe whether the conclusions are consistent?\n3. The downstream evaluation relies on benchmarks (e.g., HellaSwag, MMLU) that primarily evaluate general knowledge and reasoning. Could the \"no significant difference\" conclusion be undermined on other tasks specifically designed to assess long-context understanding?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Novel Metric and Framework. Introduces Frequency Entropy (FE)—a new quantitative metric to analyze RoPE's frequency-wise utilization, unifying previously conflicting observations about high- vs. low-frequency roles. The dual metrics (SpectrumFE and SequenceFE) offer a systematic, model-agnostic diagnostic tool.\n2. Rigorous and Multi-Model Validation.Strong empirical foundation with experiments across multiple architectures (Llama-4, Llama-3, Gemma-2, Qwen-3), probing both queries and keys. Combines entropy analysis with intervention (Weighted RoPE) to validate functional importance of frequency bands and periodicity.\n3. Well-Structured and Accessible. Clear problem framing, method description, and visualizations (entropy scatter plots, norm heatmaps) make complex frequency dynamics interpretable. Appendices extend analysis to keys and all layers, enhancing reproducibility.\n4. Resolves Prior Conflicts and Informs Design. Resolves contradictions in prior work by showing frequency bands (low SpectrumFE) are essential, while periodic dimensions (low SequenceFE) are often redundant. Offers practical implications for model efficiency and positional encoding design.", "weaknesses": "1. Experiments are conducted primarily on the Wikitext-103 dataset. To fully support the claim that FE is a \"general diagnostic,\" the analysis should include diverse text datasets to verify whether the findings generalize beyond specific tasks.\n2. The paper establishes correlations between FE values and model behaviors but does not rigorously prove causality. For instance, while attenuating low-SequenceFE dimensions doesn't hurt performance, it remains unclear if this is because they are truly redundant or if the model compensates via other mechanisms. A more controlled ablation would strengthen the causal claim.\n3. The paper studies Llama-4's iRoPE but does not disentangle the separate effects of interleaved NoPE layers and RoPE frequency scaling. An ablation study comparing would clarify which component drives the observed entropy shifts and performance outcomes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T8cSPLdk7W", "reviewer": "Reviewer_672v", "summary": "1. SpectrumFE aligns with distinct band-limited patterns in l2-norm maps.\n2. SequenceFE is sensitive to periodic structures along the token axis.\n3. From SpectrumFE, the shallow NoPE layer (layer 3) exhibits more frequency bands than the corresponding RoPE layer. Conversely, from SequenceFE, the NoPE layer lacks rotating pairs, indicating an absence of clear periodic oscillations.\n4. NoPE may therefore suppress the periodic structures characteristic of RoPE while amplifying specific frequency bands.\n5. Dimensions with τ < 0.2 in SpectrumFE appear to contribute positively to model performance, suggesting that these frequency bands are important functional components.\n6. Dimensions with τ > 0.4 may be redundant or unnecessary for model performance.\n7. Together, these results suggest that dimensions where SpectrumFE becomes an outlier and where SequenceFE decreases may not contribute meaningfully to performance and could be removed without loss.", "questions": "1. Could the authors provide greater intuition for what SpectrumFE and SequenceFE actually measure? How do these metrics relate to underlying model mechanics? Examples or analogies could greatly enhance interpretability.\n2. How might these findings be useful for researchers and practitioners? Could the authors elaborate on how the results should inform future research directions or guide the design of LLM architectures?\n3. The author write: \"We expect FE to function as a practical, model-independent diagnostic tool for position coding.\" Could the authors expand on this? What decisons or actions would a practitioner take based on FE measurements?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "* The authors introduce two novel metrics, SpectrumFE and SequenceFE, to analyze the frequency characteristics of RoPE embeddings.\n* They identify a variety of interesting phenomena associated with these metrics.\n* Ablation studies are performed to assess the effects of dampening or removing specific frequency bands.\n* The paper attempts to reconcile previously conflicting findings regarding frequency importance in RoPE.", "weaknesses": "* The practical impact of the findings appears limited. The only direct link to model performance is provided by the ablation study (Section 5.3), which shows that removing dimensions where SpectrumFE becomes an outlier or where SequenceFE decreases does not degrade performance. This limits the applicability and relevance of the results for researchers and practitioners.\n* The authors’ explanation of prior conflicting results is anecdotal rather than empirical. No targeted experiments are provided to demonstrate that earlier discrepancies arose from confounding factors. The claim that their findings “resolve the confusion of previous research” lacks sufficient supporting evidence.\n* It remains unclear what SpectrumFE and SequenceFE truly measure. The paper states only that SpectrumFE aligns with band-limited l2 patterns and SequenceFE captures periodicity along the token axis-insufficient for understanding their mechanistic or theoretical grounding.\n* Overall, the study focuses primarily on how these metrics interact with the model, rather than revealing new, interpretable mechanisms that explain how RoPE or LLMs function at a deeper level.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)