Fourier Position Embedding: Enhancing Attention’s Periodic Extension for Length Generalization
Authors: Ermo Hua
Che Jiang
Xingtai Lv
Kaiyan Zhang
Youbang Sun
Yuchen Fan
Xuekai Zhu
Biqing Qi
Ning Ding
Bowen Zhou
Venue: ICML 2025 poster (2025)
Keywords: Position Embedding
Length Generalization
Length Extrapolation
Fourier Transform
Primary area: deep_learning->large_language_models

Abstract: Extending the context length of Language Models (LMs) by improving Rotary Position Embedding (RoPE) has become a trend.
While prior works mainly address RoPE's limitations within attention, this paper uncovers the adverse effects on length generalization from nearly all parts of LMs.
Using *Discrete Signal Processing* theory, we show that RoPE enables periodic attention by implicitly achieving *Non-Uniform Discrete Fourier Transform*.
However, this periodicity is undermined by the spectrum damage caused by: 1) linear layers and activation functions outside of attention; 2) insufficiently trained frequency components brought by time-domain truncation. 
Building on our observations, we propose ***Fourier Position Embedding (FoPE)***, which enhances attention's frequency-domain properties to improve both its periodic extension and length generalization. 
FoPE constructs *Fourier Series* and zero-outs the destructive frequency components, increasing model robustness against the spectrum damage.
Experiments across various model scales and benchmarks show that, within varying context windows, FoPE maintains a more stable performance compared to other baselines.
Several analyses and ablations bring further support to our method and theoretical modeling.

Fourier Position Embedding:
Enhancing Attention's Periodic Extension for Length Generalization
Ermo Hua Che Jiang'
Xingtai Lv
Kaiyan Zhang'
Youbang Sun 1.2
Yuchen Fan 13 Xuekai Zhu 14 Biqing Qi Ning Ding!
Bowen Zhou" 13
https:/ httpp:://gitee //tph/ggtttt/tt// cww httm//iinnguaC31/orreiee. Posit tion- Embedding
Abstract
Extending the context length of Language Models (LMss) by improving Rotary Position Embedding (RoPE) has become a trend. While prior works mainly address RoPE's limitations within atten- tion, this paper uncovers the adverse effects on length generalization from nearly all parts of LMs. Using Discrete Signal Processing theory, we show that RoPE enables periodic attention by implicitly achieving Non-Une Hon-Uniform Discrete Fourier Trans- form. However, this periodicity is undermined by the spectrum damage caused by: 1) linear layers and activation functions; 2) insufficiently trained frequency components brought by time-domain truncation. Building on our observations, we pro- pose Fourier Position Embedding (FoPE), which enhances attention's frequency-donen domain properties to improve both its periodic extension and length generalization. FoPE constructs Fourier Series and zero-outs the destructive frequency compo- nents, increasing model robustness against the spectrum damage. Experiments across various model scales and benchmarks show that, within varying context windows, FoPE maintains more stable performance compared to other baselines. Several analyses and ablations bring further sup- port to our method and theoretical modeling.
20
Sezusnoc
(a) Accuracy on Passkey Retrieval (higher is better)
120
NoPE
100
ALiBi
KERFLE
Porplosity
RoPE
FoPE
512
1024
2040
4096
8192
Sequence Length
(b) Perplexity on C4 (lower better)
Figure 1. Comparison of 1.2B models with different position beddings (trained with a maximum sequence length 512) on a needle-in- needle-in-haystack task and pre- traininng perplexity. FoPE yields significantly more stable results than RoPE and other baselines.
typically trained fixed context window (Vaswani, 2017: Touvron et al., 2023; Groeneveld et al., 2024) and tends to overfit to the specific context length.
1. Introduction
Generation based on the information from long contexts is crucial for Language Models (LMs). However, LMs are
Many studies consider the absolute position embedding (Vaswani, 2017) as key contributor to overfitting. As miti- gation, several relative position embedding methods have been proposed (Press al., 2021; Chi et al., 2022; Peng et al., 2023; Li et al., 2024; Jin al., 2024; Su al., 2024) to improve LMs' long-distance dependency. Among these, ALiBi (Press et al., 2021) introduced a position-biase at- tention mask, which linearly declines the attention weights based on distance. ALiBi achieves stable perplexity dur- ing pre-training, but fails to retain information from distant
'Tsinghua University Northeastern University SShanghai Artificial Intelligence Laboratory 'Shanghai Jiaotong University. Correspondence Biqing Qi <qibiqing cuibiiing@pilab.rgg pjlaaborg.cn) Ning Ding <dingning @ Seieeiinn@siigguuaa.m tsiiggua.cdu.ccn>. Bowen Zhou <zhoubowen @wwi@@@ttsasiuu.....
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Enhancing Attention's Periodic Extension for Length Generalization
(a)
Spectral Damage
Spectrum Leakage
Spectrum Distortion
Information of multiple frequencles mixed single dimension
H
(n) (Hay H EEH eloyp
Treating each dim as single-freq function
Dim
RoPE
(T)))) (#2, H 2 eloru
y Jt
o (H) (H))
X
H
hy(n)= = elapp H eloaa (IHt / elaon) on
Linear
Activation
Astatta
Treating each dim as multi-freq function
FoPE
6 w
j w 6
(4( H mn + Hb, elmps + (IHls elann)
(b)
Undertrained Frequency Components
H
Keep Periodic Extension on length extrapolation
h h R
Pre-training token length
0OD on length extrapolation detertorates Periodic Extension
00g Adequately trained frequency
j (2) 60k
RoPE
ht
() Floor frequency adequately trained
Zero-out Undertrained frequency components
() Undertrained frequency
n (n) H
H elonn
Figure 2. The reasons why RoPE's periodic extension deteriorates and how FoPE addresses these issues improve length generalization. (a) As signals pass through linear and nonlinear transformations, this causes spectral leakage and distortion, mixing multiple frequencies into single dimension. Under RoPE, each dimension treated dimension as a Fourier series of different frequency components, thereby separating information more effectively and mitigating spectral damage. (b) FoPE eliminates inadequately trained frequency components, which are harmful for periodic extension. By preserving only the fere-frcy component, FoPE safeguards periodic extension and delivers more robust length generalization.
component. By contrast, FoPE models each
tokens, leading to poor performance on long-context down- stream tasks. Another method, RoPE (Su et al., 2024), uses the phase of complex numbers to store the position infor- mation. Combined with continual pre-training and other interpolation- based methods (Peng et al., 2023; Xiong et al., 2024; Chen et al., 2024a; Jin et al., 2024), RoPE provides better access long-disen distanee information, making one of the most widely used position embedding. However, RoPE- based LMs still struggle with length generalization without supplementary methods like YARN (Peng et al., 2023).
periodic extension for better length generalization. Com- pared to ROPE, FoPE introduces two main improvements: I) While RoPE treats each dimension as single-frequency function, FoPE models each dimension as a Fourier Se- ries, consisting of a dominate frequency component and several harmonic components. This approach better mirrors the actual spectrum LMs and helps attention separate information across different wavelengths, mitigating the negative effects of Spectral Damage. 2) FoPE clips inad- equately trained frequency components that is harmful to length generalization. To keep the passing of long wave- length information, we substitute these components with zero, as the zero- cero-frequency component corresponds to the longest wavelength.
In this paper, we take closer look at RoPE in the frequency- domain with tools from Discrete Signal Processing (DSP)
theory. Our modeling reveals that RoPE implicitly per- forms Non-Unifore Discrete Fourier Transform (NUDFT)
on the hidden states, enabling periodic attention based on the frequency-domitl encoding. However, we find that the periodicity is hindered by the spectral damage caused by: 1) linear layers and activation functions outside attention; 2) inadequately-trann -rrineed frequency components within attention (See Fig 2). This explains why RoPE fails to achieve length generalization without assistance from other methods.
We summarize our contribution follows:
1. Based on DSP, we provide requency-domall analysis to reveal the negative influence from nearly all parts from LMs. We find that the length generalization is hindered by the Spectrum Damage arised from: 1) linear layers and activation functions; 2) undertrained frequency components.
2. We propose FoPE improve attention's robustness on the Spectrum Damage. FoPE construct Fourier Series to extract multi-frequency information in each dimension, and
Building on our observations above, propose Fourier Po- sition Embedding (FoPE) to further improve the attention's
2

---

Enhancing Attention's Periodic Extension for Length Generalization
clip the frequency of destructive components to zero. Thus, FoPE delivers better periodic extension of attention, thus bringing better length generalizaion.
where n na ni and Hm = Qm Km Finally, the overall attention weight between different tokens becomes:
M-1 Σ
M-1 Σ
3. We conduct experiments across several model scales and datasets. The perplexity in pre-training and the accu- racy in needle-in-haysstck demonstrate FoPE's superiority over other baselines on length generalization. Evaluation on more complex tasks (i.e. summarization and few-shot question-answering brings further support to method and theoretical modeling.
h(m) hmm (n)
H e
(4)
Comparing it with Eq (1), it can be observed that RoPE implicitly achieves a token-leve Inverse NUDFT with fre- quency components {wm)
Based on NUDFT, RoPE models the interactions between different tokens as functions compose of several periodic components, which brings periodic extension in each di- mension
2. Preliminaries
2.1. Non-Uniform Discrete Fourier Transform
Given a finite sequence of { == Io, T1,,,,, equally-sampled from continuous function z, Discrete Fourier Transform (DFT) converts them into equally- spaced components [Xm]
Nym (n + Nus...
hmm (n)
(5)
2x is this component's period. This property where Num dim can generalize LMS longer context.
Xu, ...,,,,................. . in frequency-domain, the original samples can be recovered
by Inverse DFT (IDFT):
3. Spectrum Damage Confine the Length Generalization
N-1 Xm = Σ Tne 12m Fm In n=
M-1 Σ m==
M7
Xmelt
e2
22 f n
Ideally, RoPE-based Attention achieves periodic extension in any length scenario. However, this extension is confined as a key ideal property that is not guaranteed in LMs.
(1) As giun coswn + isin wn is periodic in the original domain, DFT implicitly transforms the original function into a linear combination of periodic waves with frequency Wm 2n m function, which is lossless only if the original function ex- actly composes of these specific periodic components.
3.1. Negative Inftuence of Spectrum Damage
M Thus, DFT is an estimation of the original
The ideal coefficients and frequencies of NUDFT have one- to-one correspondence. The coefficient of each frequency represents the influence of each token on others propagated at a specific wavelength.
To achieve a more accurate approximation, the sampled frequencies Wm == wo (1,, W1 can follow any arbi- trary distribution tailored the data characteristics, subject only to the constraint Wm E [0, 2T). This generalized form of DFT is known as the Non-Uniform Discrete Fourier Transform (NUDFT).
However, the periodic extension hindered, if the coeffi- cient also contains the information from another frequency component w. with coefficient H a Huw called the Spectrum Damage.
If we define the damaged function as W/m = Hwm [(1 o )e + oe we find:
2.2. RoPE implicitly achieves Periodic Attention based on NUDFT
Ny, (n + Num.. + (n)
(6)
Given a M-dimension query Q and key K for tokens a and b, RoPE rotates different dimensions m to different phases:
as NN.. is not the period of lww, In other words, the infor- mation from each component transmitted through waves with mismatched wavelengths, leading to inaccurate esti- mation of the influence propagated within each wavelength. (See Fig 3) When multiple frequency components are mixed within dimension, the periodicity of information variation with token distance changes (e.g., Ground Truth in the top and middle aub-figure), no longer matching the periodicity expected by RoPE for that dimension. As a result, the pe- riodic extension and length generalization of attention are adversely affected.
Im (na) Qm twe Kym (1op) K owmm Uo
where Wmn 1/0 (M1) and 0 is the pre-defined parame- ters in RoPE. Then, the attention weight (n) in each dimension will be calculated as:
Iym (n) Te kim (ns) = H kwmn Kimn
3

---

Enhancing Attention's ' Periodic Extension for Length Generalization
with frequencies as linear combinations of w1 and war
RoPE
FoPE
Ground Truth
Attention Score
0.25
g(æ(n)) = Σ E 0j Cos(fos kwz)n
(8)
JEN
0.25
0
50
100
150
200
250
which can be generalized to any multi-freqqennc function <(n) = Ea sin wn + bu cos wm))
Attention Score
As the hidden states are transformed into multi multi-frequency functions by Linear Layer, passing them across Activation Functions introduces additional harmonic components, lead- ing to serious Spectrum Distortion.
50
100
150
200
250
Attention Score
These two types of Spectrum Damage undermine the peri- odic extension of attention (as shown in Eq.(5)(6)), hinder- ing the model's length generalization property.
150
250
Taken Distance
3.3. Spectrum Damage Inside Attention
Figure 3. An illustration of the negative impact of Spectrum Dam- age length generalization. The figure show the contributions of two frequency components to the final attention score (top & middle), followed by the sum of their contributions (bottom). We consider two dimensions hidden states with two frequency com- ponents. Because of Spectrum Damage, each dimension actually contains multi- fuuuunefr information (referencing Ground Truth). multi- multi--freqency position embedding introduced by FoPE pro- duces attention scores that are closer to the actual information compared to RoPE.
Besides the spectrum damage outside attention, the under- trained components of attention within extremely low fre- quencies (Wm < 2a also bring spectrum damage.
Given tangle-frequency function I'm (n) e e (ect(n) truncated by square wave:
{1 n S N
rect(n)
(9)
n N
Based on the results of DFT, the spectrum of x(n) is!:
3.2. Spectrum Damage Outside Attention
X(w) a6(wm)) sin((N a (w wm))
(10)
The LMs' linear layers and activation functions outside attention bring two types of spectrum damage, destroying the one-to-one correspondence between coefficients and frequencies.
W Wm
N and N,, 2m where = N
In the frequency domain, time-domain truncation introduces noisy components via the latter sub-function. When the pe- riod of the primary frequency component exceeds the trun- cation length, its amplitude is significantly weakened. Con- sequently, noisy components dominate these dimensions, impairing the periodic extension (as defined by (5))5) In contrast, high-frequency components are minimally af- fected because their coefficients a dominate over the noisy components.
Linear Layer uses weights W MMM map M dimension hidden state X E RM to another hidden state e M Thus, each dimension of Y will be a linear combination of different components of X:
M
Ym
W VrmmXx
Intuitively, when sampling sinusoidal functions based on token positions, these low-frequency components cannot cover complete cycle. Therefore, for positions exceeding the pre-training sequence length, these dimensions may sample outside the training domain, leading to difficulties in generalization. Although previous works (Peng et al., 2023) have identified this issue, we are the first to model from a Fourier perspective and provide a theoretical explanation.
This results in Spectrum Leakage, different frequency components exhibit interplay.
Activation Function has non-linearte lineaiiyy in the time domain, generating harmonic frequencies as described by the follow- ing Lemma:
Lemma 3.1. Given a double-frequency sinusoid function I(n) = cos win + cos w2n and any time-indepe ine-indepenneet non- linear function g. The effect of g on æ(n) yields functions
From (Oppenheim al., 1982)

---

Enhancing Attention's 's Periodic Extension for Length Generalization
FoPE-512
FoPE-512- YARN-112
FoPE-612+FOPE-104
RoPE-512
RoPE-S12-YARN-10Z
RoPE-S12 FOPE-100
100
100
100
80
60
60
40
0
1024
2040
6192
20382
2048
20304
522
3024
2048
4090
0192
15386
Lenpth
Length
(a) Accuracy on Passkey Retricval (higher better)
120
50
1.00
40-
20
Length
Sequence
(h) Perplexity C4 (lower is better)
Figure 4, Effectiveness of FoPE in length extrapolation. Starting point models trained with 512 context length are extrapolated using YARN and FoPE on corpus with a maximum sequence length of 1024. The generalization ability FoPE not only surpasses that RoPE as base position embedding, but FoPE also demonstrate capability an extrapolation method applied to pre pre-trained models.
4. Fourier Position Embedding
are sampled separately in implementation, which can also use the same coefficient. The D and are kept as hyper-parameteer to be adjusted.
To mitigate the negative affect of the non-ideal frequency- domain properties in LMs, propose Fourier Position Embedding (FoPE) modify frequency-domain properties of attention:
Zero-out Under-trained Frequencies. As analyzed in Sec 3.3, the inadequate training of extremely-low frequencies Wm impairs the frequency-domaii properties of atten- tion. Thus, we define the floor frequency wi 27 and clip the frequencies under the floor frequency zero.
Treating Each Dimension as Multi-Frequency. Although Linear Layers and Activation Functions bring serious Spec- trum Leakage and Spectrum Distortion, they are crucial for enhancing expressive capacity. Therefore, keep these modules unchanged but focus modifying how attention processes information within each dimension.
We choose zero as the substitute because the zero-frequency component can has the shortest and the longest wavelength at the same time, which presents both long-term and short- term dependency. Also, zero-frequency brings zero average position embedding and will not bring positional bias, which benifits length generalization.
To achieve this, we replace the single frequency in each dimension with Fourier Series:
Tim (n) Hm (n) + Σ Eane
(11)
Overall function of FoPE can be formalized as:
where < because wm is the dominant frequency. This allows attention modules to capture multi-frequency infor- mation in each dimension.
hym (n) H (
(12)
: Wm < wil
We initialize vector {wm} as same RoPE, and initialize vector {w} and matrix } based on the analysis Sec 3.2: For {w} R make sure M < D so that {wm) } C {w}. and the other frequencies can be sampled within [0, T] in any distribution. For {aw} E D we initialize it with NO or) based on the hypothesis that the Spectrum Damage obeys the similar distribution as the Linear Layers. The coefficients for the real and imaginary part of the frequency
f(wm) eiumm
+ Σ e
, Wm 2 wil
(13)
which treats each dimension either as a Fourier Series or as fre-ffreyeency component.
FoPE can be easily achieved with a weight matrix F PM Mo), where Mp is the number of zero- froeurreuencc components each head (details in Appendix B)
5

---

Enhancing Attention's Periodic Extension for Length Generalization
5. Experiments
300
FoPE 60M
To demonstrate the effectiveness of FoPE as both a posi- tion embedding and an extrapolation method, we conduct experiments during pre-training (Sec. 5.2), continual pre- training (Sec. 5.3) and fine-tuning (Sec. 5.4). Additionally, we perform ablations to analyze the impact of hyperparam- eters on FoPE (Sec. 5.5), and conduct empirical analysis to showcase the working machenism of FoPE (Sec. 5.6).
275
RoPE 60M
FoPE 180M
RoPE 180M
225
180M
Porpicsity
FoPE 1.28
200
RoPE 1.28
175
150
Experimental details and results on more downstream tasks are shown Appendix A.4.
125
100
5.1. Basic Settings
512
1024
2048
4096
8192
Sequence Length
Pre-training and Continual Pre-training: In these two settings, we conduct experiments with the OLMo (Groen- eveld al., 2024) framework and consider different scale models having 60M, 180M, 1.2B parameters. We consider two metrics: perplexity for validation set and accuracy Passkey Retrieval (Mohtashami & Jaggi, 2023). Passkey Retrieval measures the models' ability in retrieving a short passkey (i.e., a five- number) from a large context full of meaningless text. We conduct this evaluation based on the implementation from (Peng et al., 2023). During evaluation, the passkey is randomly positioned at uniformly distributed locations within the context. For each context length, we test for 1000 trials to ensure the positions sampled are suffi ciently dispersed.
Figure 5. Training with max seq leg_tegt---5122 on Gutenberg Books and evaluating on a validation set of C4, FoPE also demonstrates its ability generalize across different data distributions.
training set, which can further evaluate the generalization ability of different methods.
Results of Perplexity. (See Fig 1.b & 5) In both settings, FoPE shows significant advantage over RoPE. But FoPE is slightly worse than ALiBi, as there is an issue when AL- iBi meets this training corpus, which is also mentioned in other papers (Peng et al., 2023; Chen et al., 2024a). On the one hand, the corpus in C4 and Books mainly have short- distance dependency. thus the information from a short con- text window is enough for the prediction of almost all tokens. On the other hand, AliBi uses linear declined attention to eliminate long-distance information, and only pays attention to doot--istance dependency. Based on these two reasons, ALiBi does not have any decline in perplexity as the context length increases.
Fine-tuning: To assess the effectiveness of our method on more complex downstream tasks, we select SmolLMM-1.70 (Allal et al., 2024), a capable open-source model with the same architecture as Llama (Touvron et al., 2023), our base model. We then fine-tune it with the same training recipes as SmolLM-1, 1.7B--Inssteee setting, we evaluate two types of downstream tasks, summa- rization and few-shot question-answen answering. For summariza- tion, we use GovReport (Huang et al., 2021) and MultiNews -anwwern... we
(Allal et al., 2024). In this
Results of Passkey. (See Fig Laa FoPE demonstrates a significant advantage all baselines in this task. RoPE's accuracy drops sharply to zero at twice the training length and remains at zero for longer sequences. ALiBi shows linear decline in accuracy, further illustrating that its linearly declining attention is unable to capture information from long distances. contrast, FoPE maintains stable retrieval accuracy any position, demonstrating strong ability to extract subtle information from long sequences.
(Fabbri et al.. 2019). For few-shot TREC (Li & Roth, 2002), TriviaQA (Joshi al., 2017), and SAMSum (Gliwa et al., 2019). All these evaluations are conducted under the same setup in (Bai et al., 2023).
5.2. Length Generalization after Pre- Pre-Training
We consider two settings to evaluate both the intra-domain and out-of-domail generalization:
5.3. Length Generalization after Continual Pre- Pre-Training
Setting 1: We train models with a 10B-tokens subset of C4 (Raffel et al., 2020) and evaluate the perplexity in a validation set from C4.
Beyond the use of positional embeddings during the origi- nal pre-training phase, several extrapolation methods (Peng et al., 2023; Chen et al., 2024a) have been proven critical for enhancing length generalization. Thus, we investigate two key aspects of FoPE: 1) whether existing extrapolation methods are also effective for FoPE; 2) whether FoPE can enable extrapolation on RoPE-based models, thereby allow- ing seamless integration with existing open-source models.
Setting 2: We train models with ~5B tokens from Guten- berg Books (Hart, 2007) and evaluate them in the same validation set as Setting 1. In this setting, the language distribution is different between the validation set and the
6

---

Enhancing Attention's Periodic Extension for Length Generalization
TaP2 03M
lorghhe52
langth
FORE CON
larghhe2024
FORE CON with
2012hh-2082
PJF2 CON wiy
-#- FoFt couble Fand, SWh
7oFt couble others
12
2013
Coaftcinn
PPL Ratio for different sub-methods
(b) PPL Ratio for different
(c) Passkey Ace for different D
Figure 6. Ablation studies on several hyperparameters. (a) and (b) evaluate PPL Ratio = PPLes/PPLasse while (c) evaluates accuracy on Passkey Retrieval as we found slightly influence the perplexities of training. See the detailed analysis See 5.5.
5.4. Length Generalization after Fine- Mone-Tuning
Table Evaluation on Summarization and Few-shot QA tasks. On most benchmarks, FoPE delivers better length generalization compared to RoPE. The only exception is TriviaQA, where FoPE performs slightly worse in shorter contexts. However, FoPE re- mains stable up to 8k+ and outperforms RoPE significantly in 8k+.
We further evaluate the effectiveness of FoPE within the widely used LLaMA architecture, focusing on more chal- lenging tasks such as summarization and few-shot ques- tion answering. For this purpose, select SmolLM-1.77B which is pre-trained with context length of 2048. We then fine-tune SmolLM to a context length of 4096 using both RoPE and FoPE, and compare their performance on samples of varying sequence lengths from selected datasets, categorized into 0-4k, 4-8k, and 8k+.
Benchmark
Length
RoPE
FoPE
0-4k
13.02
13.27 f 0.25
GovReport
4-8k
11.35
12.50 f 1.15
8k+
12.02
12.38 f 0.36
0-4k
12.71
12.92 f 0.21
MultiNews
4-8k
11.11
12.98 f 1.87
Results. (See Table On nearly all benchmarks, FoPE not only improves inner-domann performance for samples with 0-4k length, but also demonstrates more stable gener- alization capabilities for longer context lengths. The only exception occurs in TriviaQA, where FoPE performs slightly worse in shorter contexts. However, FoPE's performance re- mains stable up to 8k+ and significantly outperforms RoPE.
8k+
10.85
12.23 f 1.38
0-4k
37.00
41.00 f 4.00
TREC
4-8k
42.00
56.00 1 14.0
8k+
36.00
51.00
1 15.0
0-4k
36.50
33.26
|. 3.24
TriviaQA
4-8k
36.12
33.53
2.49
8k+
25.02
33.87
1 8.85
5.5. Ablation Studies
0-4k
10.27
19.77 1 9.50
Considering the consistent performance of FoPE across different parameter scales and datasets, we conduct ablation studies only on the 60M models trained on 5B tokens from Gutenberg Books. Except for the primary ablation variable, all other prpemperraeeers are kept identical to the main experimental settings.
SAMSum
4-8k
6.37
15.85 1 9.48
8k+
8.49
17.26 1 8.87
In this sub-experiment, select representative extrapo- lation method, YARN (Peng et al., 2023), our baseline. We fine-tune the last checkpoint from pre-traiee pee-rraiing for ~ IB tokens in this setting.
Both sub-methods of FoPE are useful (See Fig 6a). FoPE is constitutive of two parts, called Fourier Series (FS) and Clip Floor Zero (CF). Although these two sub-methods are both useful for length generalization. combining them together brings more significant improvement. On one hand, FS contributes more to length generalization, which demonstrates that the Spectrum Damage have significant influence on length generalization. On the other hand, CF contributes more to fitting the current dataset and sequence length, which implies rero-frequency component the most informative and indispensable component.
Results. (See Fig 4) In comparison with RoPE+YARN, FoPE+YARN achieves significantly better length general- ization performance, as demonstrated by lower perplexity on the C4 dataset and higher accuracy in the Passkey Re- trieval task. Moreover, FoPE outperforms YARN in length extrapolation for both RoPE-based and FoPE-based models. These findings underscore the effectiveness and practical utility of FoPE, which holds the potential to enhance all RoPE-based open-source source models.

---

Enhancing Attention's ' Periodic Extension for Length Generalization
Increasing the dimension of attention heads is more ben- eficial than increasing the number of attention heads or layers (See Fig 6a). More dimensions introduce more frequency components, making attention more robust to Spectral Damage. In contrast, adding more attention heads and layers aggravates Spectrum Damage, which diminishes the benefits of expanding the parameter scale.
62
100
104
105
(a) Q vectors' mean activation across heads
Variance a of {aw] (See Fig 6b). We keep D 16 to only evaluate o's influence. By grid searching from 0 to 0.5, we find that setting = 0.3 for 60M model obtain the best perplexity, especially for longer context. The best o implies the estimated strength of Spectrum Damage of the 60M model, and the estimation may become larger as the models' parameter scale increases.
Layer
100
164
108
112
Head Dimonsion
Number D of {wi.. We keep 0.3 to only evaluate D's influence. By grid searching from 16 to 128, we find that D does not significantly influence the perplexity, but it is important for Passkey Retrieval. Setting D = 64 can obtain the best accuracy for Passkey Retrieval. The best D the estimated number of strong enough noisy components of each model, and this number may become larger as the parameter scale increases. The harmonic frequencies tend to be weaker than the base frequencies, and this phenomenon is more significant to the higher-order harmonics. Thus, there are limited noisy frequency components that have enough intensity to disturb the passing of the base wave, paying attention to not important components hinders the effectiveness of the model.
(b) vectors' mean activation across heads
Figure 7. The activation values across different dimensions of g, vectors various layers, averaged over the attention heads Llama2-7B. We found that trained dimensions holds higher absolute values across all layers of the model.
Modeling each dimension as Fourier Series. We use a toy model consisting of a single-layer MLP with an activa- tion function, to validate the necessity of multi- felliiiereuuecyy representation in single dimension by simulating attention scores' calculation. For simplicity. we model the interaction of token with only two dimensions (frequency compo- nents) hidden states, setting both Query and Key matri- ces as identity matrices during attention score computation. We track the transformations of each dimension through the MLP and activation function, matching frequency com- ponents with corresponding sinusoidal functions obtain crequency-matcherr attention scores, termed Ground Truth (See Fig 3). Comparing Ground Truth with RoPE and FoPE attention scores, we find that FoPE more accurately captures multi- fullii-freuency periodicity, yielding attention scores that better align with information transfer.
Although the choice of hyper-parameters affects the effec- tiveness of FOPE, the ablation study shows that FoPE consis- tently outperforms RoPE across all hyperparameter settings, demonstrating its robustness to hyperparameter selection.
Table 2. The loss of 20M toy models trained sequence length of 512. We consider three types of position embeddings. among which only RoPE has frequencies that cannot complete full cycles. For RoPE-A, all frequencies are adjusted from RoPE to the nearest values that exactly complete full cycles.
The negative effect of under-trained dimensions. By vi- sualizing the numerical expectations of the q and k vectors in each dimension (details in Appendix A. observe that the absolute values of the dimensions corresponding to tader-trained frequencies holds higher absolute values across all layers of the model. This positional bias may adversely affect robustness to out-of-domain rotation matrix values during length generalization. To further verify this hypothesis, we train several toy models (See Table 2). For models with QK _orom, we normalize the q.1 vectors (en- forcing mean of 0 and variance of 1) before applying rota- tion matrix to eliminate positional bias. Such normalization delivers positive impacts only on under-trained frequency components, but not on frequency components complete full cycles. These experimental results validate our hypothesis.
Sequence Length
512
1024
2048
4096
8192
RoPE
5.50
6.01
6.58
6.99
7.16
RoPE + Norm
5.46
5.56
5.89
6.32
6.66
RoPE-A
5.72
5.86
6.18
6.46
6.67
RoPE-A + Qomm
5.69
5.89
6.27
6.59
6.81
NoPE
5.65
6.03
6.60
6.81
6.99
NoPE + QK_Norm
5.59
6.18
6.87
7.10
7.43
5.6. Empirically Validation of FoPE's Mechanism
To further validate the theoretical insights of FoPE analyzed before, we conduct empirical experiments on its two core modification of RoPE
8

---

Enhancing Attention's Periodic Extension for Length Generalization
6. Related Work
pared baselines. Our ablation studies brings analyze on the influence of each sub-method and hyperparameter on FOPE. Several empirical toy experiments also provide fur- ther support for our method and theoretical modeling.
Seqquency-Domaile Embedding. Discrete Fourier Trans- form (DFT) (Oppenheim et al., 1982) has been widely used in various areas having periodic signals (Edfors al., 2000; Sanchez, 2010). In machine learning, (Uteuliyeva et al., 2020; Lin et al., 2024b; Tancik et al., 2020; Tamkin et al., 2020; Thorphorp et al., 2022; Gillman et al., 2024) em- ployed Fourier features into neural networks to enhance performance on NLP or CV tasks. S4 (Gu et al., 2021) also leveraged FFT and IFFT to shift its core computation into the frequency-domatt delivering more efficient com- putation. (Wang al., 2019; Su et al., 2024) improved the attention mechanism by defining position embedding with complex number, while "phase" used in these methods is a typical concept in frequency-domain.
Impact Statement
Our DSP-based modeling in the frequency domain provides novel perspective for LMs to enhance length general- ization and explore broader applications. These include aligning frequency-domait representations for better model collaboration, optimizing kv-cache compression via spec- tral analysis, and improving semantic communication with learnable frequency-domain embeddings.
A Acknowledgement
Length Generalization. Due to resource constraints, LMs are trained on limited-length corpus chunks and struggle with longer contexts (Voita et al., 2023; Dong et al., 2024; Hong al., 2024; Qi al., 2024). While absolute position embeddings (Vaswani, 2017) restrict the general use of po- sitional information, methods as (Shaw et al., 2018; Yang, 2019) directly adjust the attention mechanism, another intu- itive method is to redesign the position embedding (Press et al., 2021; Chi et al., 2022; Li et al., 2024; Kazemnejad al., 2024; Su et al., 2024: Wang et al., 2024; Choroman- ski al., 2024; Barbero et al., 2024; Chen al., 2024b). Among these, RoPE (Su et al., 2024) encodes positional in- formation using the phase of complex numbers, leveraging their periodicity to enhance access to long-distance depen- dencies. Several training-free free or fine-tuning-i based methods can also improve the LM's length generalization by refining RoPE (Peng et al., 2023; Chen et al.. 2024a; Jin al., 2024; Lin et al., 2024a). However, these works mainly address the drawbacks of RoPE in attention mechanism, neglecting the influence of other components in LMs.
This work is supported by the National Science and Tech- nology Major Project (20232D0121403), the Beijing Nat- ural Science Foundation (IS23059), the Young Elite Sci- entists Sponsorship Program by CAST (2033NNRC001) and the National Natural Science Foundation of China (No. 62406165). We further extend our gratitude to Yihao Liu, Yiming Shi, Yizhou Jiang, Hang Deng and Chuxuan Shan, for their insightful discussion with us.
References
Allal, L. B., Lozhkov, A., Bakouch, E., von Werra, L., and Wolf, T. Smollm - blazingly fast and remarkably powerful, 2024.
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. Longbench: A bilingual, multitask benchmark for long context under- standing. arXiv preprint arXiv:2308. 14508, 2023.
Barbero, F., Vitvitskyi, A., Perivolaropoulos, C., Pascanu, R., and Veličković, P. Round and round we go! what makes rotary positional encodings useful? arXiv preprint arXiv:2410.06200 2024.
7. Conclusion
In this paper, we analyze RoPE-based attention by modeling in the frequency domain using Discrete Signal Process- ing (DSP) theory. Our analysis reveals that RoPE achieves periodic attention by implicitly performing Non-Uniforn Discrete Fourier Transform (NUDFT). However, this peri- odicity is corrupted by the non-ideal spectrum properties introduced by several parts in LMs. We propose Fourier Position Embedding (FoPE) to enhances attention's peri- odic extension and length generalization. FoPE models each dimension as Fourier Series and zero-out inadequately- trained frequency components. Experiments are conducted on models with various architectures and scales. Based on the evaluation on diverse well-known datasets including summarization and question-answering tasks, FoPE demon- strates significantly improvement length generalization com-
Chen, G., Li, X.. Meng, Z., Liang, S., and Bing, L. CLEX: Continuous length extrapolation for large lan- guage models. In The Twelfth International Confer- ence on Learning Representations, 2024a. URL https: //openreview.ct net/ wwwpmmmidddwpppppppp.
Chen, Y., Lv, A., Luan, J., Wang, B., and Liu, W. Hope: A novel positional encoding without long-term decay for enhanced context awareness and extrapolation. arXiv preprint arXiv:2410.21211 2024b.
Chi, T.-C., Fan, T.-H., Ramadge, P.J., and Rudnicky, A. Ker- ple: Kernelized relative positional embedding for length

---

Enhancing Attention's Periodic Extension for Length Generalization
extrapolation. Advances in Neural Information Process- ing Systems, 35:8386-8399, 2022.
Jin, H., Han, X., Yang, J., Jiang, Z., Liu, Z.,, Chang, C.- Y., Chen, H., and Hu, X. Lim maybe longlm: Self- extend I1m context window without tuning, arXiv preprint arXiv:2401.01322 2024.
Choromanski, K., Li, S., Likhosherstov, V., Dubey, K. A., Luo, S., He, D., Yang, Y., Sarlos, T., Weingarten, T., and Weller, A. Learning a fourier transform for linear relative positional encodings in transformers. In International Conference on Artificial Intelligence and Statistics, pp- 2278-2286. PMLR, 2024.
Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint urXiv:1705.033 03751, 2017.
Dong, Z., Li, J., Men, X., Zhao. W. X., Wang. B., Tian, Z., Chen, W., and Wen, J. Exploring context window of large language models via decomposed positional vectors. arXiv preprint ar arXiv:2405. 18009, 2024.
Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., and Reddy, S. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024.
Edfors, O., Sandell, M., Van De Beek, J.-J., Wilson, S. K., and Börjesson, P. O. Analysis of dft-based channel esti- mators for ofdm. Wireless Personal Communications, 12: 55-70, 2000.
Lee- Thorp. Lee-Thorp. J., Ainslie, J., Eckstein, I., and Ontanon, S. Fnet: Mixing tokens with fourier transforms. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.
2022.
Fabbri, A. R., Li, I., She, T., Li, S., and Radev, D. R. Multi-news: A large-scale multi-document summariza- tion dataset and abstractive hierarchical model. arXiv preprint arXiv:1906.0 01000...1..9,... 2019.
Li, S., You, C., Guruganesh, G., Ainslie, J., Ontanon, S., Zaheer, M., Sanghai, S., Yang, Y., Kumar, S., and Bho- janapalli. Functional interpolation for relative positions improves long context transformers. In The Twelfth Inter- national Conference Learning Representations, 2024.
Gillman, N., Aggarwal, D., Freeman, M., Singh, S., and Sun, C. Fourier head: Helping large language models learn complex probability distributions. arXiv preprint arXiv:2410.22269, 2024.
Li, X. and Roth, D. Learning question classifiers. In COL ING 2002: The 19th International Conference on Com- putational Linguistics, 2002.
Gliwa, B., Mochol, I., Biesek, M., and Wawer, A. Samsum corpus: A human-annotated dialogue dataset for abstrac- tive summarization. arXiv preprint 11X.v:1111.12237, 2019.
Lin, H., Lv, A., Chen, Y., Zhu, C., Song, Y., Zhu, H., and Yan, R. Mixture of in-context experts enhance Ilms' long context awareness. arXiv preprint arXiv:2406. 19598. 2024a.
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al. Olmo: Accelerating the science of lan- guage models. arXiv preprint arXiv:2402.00888, 2024.
Lin, S., Lin, W., Hu, X., Wu, W., Mo. R., and Zhong, H. Cyclenet: enhancing time series forecasting through mod- eling periodic patterns. arXiv preprint arXiv:2409.18473 2024b.
Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00300 2021.
Mohtashami, A. and Jaggi, M. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305. 16300, 2023.
Hart, M. S. Project gutenberg. https: www. gutenberg.oom org. 2007. Accessed by OLMo Team on Unknown Date.
Oppenheim, A. V., Willsky, A. S., and Nawab, S. Signals and Systems. Prentice Hall, 1982.
Hong, X., Jiang, C., Qi, B., Meng, F., Yu, M., Zhou, B., and Zhou, J. On the token distance modeling ability of higher rope attention dimension. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 5877- 5888, 2024.
Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Effi- cient context window extension of large language models. In The Twelfth International Conference Learning Representations, 2023.
Press, O., Smith, N., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapo- lation. In International Conference on Learning Repre- sentations, 2021.
Huang, L., Cao, S., Parulian, N., Ji, H., and Wang, L. Effi cient attentions for long document summarization. arXiv preprint arXiv:2 104.0// /2, 2021.
10

---

Enhancing Attention's Periodic Extension for Length Generalization
Qi. B., Gao, J., Zhang, K., Li, D., Liu, J., Wu, L., and Zhou, B. Smr: State memory replay for long sequence modeling. arXiv preprint arXiv:2405.
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P.,
Hou, R., Martin, L.,, Rungta, R., Sankararaman, K. A.,
2024.
Oguz, B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S., Malik, K., Fan. A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H. Effective long-context scaling of foundation models. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Pupers), pp. 4643-4663, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.18665lv1120224mmamiiooooo
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Sanchez, J. Cluster expansion and the configurational theory of alloys. Physical Review C-Condensed Matter and Materials Physics, 81(22)222420, 2010.
URL https:// aclanthology. hww.:thhhooo...oog//... wtw.:ooo/2222..ac.111ooooo.. 260.
Shaw, P., Uszkoreit, J., and Vaswani, A. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155, 2018.
Yang, Z. Xlnet: Generalized autoregressive pretrain- ing for language understanding. arXiv preprint 1aXiv:9906002377,, 2019.
Su. J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Tamkin, A., Jurafsky, D., and Goodman, N. Language through a prism: A spectral approach for multiscale lan- guage representations. Advances Neural Information Processing Systems, 33:5492-5504, 2020.
Tancik, M., Srinivasan, P.. Mildenhall, B., Fridovich-Keil. S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., and Ng, R. Fourier features let networks learn high fre- quency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 13971, 2023.
Uteuliyeva, M., Zhumekenov, A., Takhanov, R., Assylbekov, Z., Castro, A. J., and Kabdolov, O. Fourier neural net- works: A comparative study. Intelligent Data Analysis, 24(5):1107-11200 2020.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Voita, E., Ferrando, J.. and Nalmpantis, C. Neurons in large language models: Dead, n-gram, positional. arXiv preprint ar ar.iv:2309. 04827, 2023.
Wang, B., Zhao, D., Lioma, C., Li, Q.. Zhang, P., and Simon- sen, G. Encoding word order in complex embeddings. arXiv preprint arXiv: 12333, 2019.
Wang, J., Ji, T. Wu, Y., Yan, H., Gui, T., Zhang, Q Huang, X., and Wang, X. Length generalization of causal transformers without position encoding. arXiv preprint arXiv:2404. /2224, 2024.

---

Enhancing Attention's Periodic Extension for Length Generalization
A. More Experimental Results
A.I. Visualization of g vectors before applying RoPE
We conduct visualization experiments using the Llama2-7B model, which features an attention module with 32 heads, cach comprising 128 dimensions, and a pretraining sequence length of 4096 tokens. The number of cycles sampled by each sinusoidal function during pretraining is calculated as ry Eariin where denotes the dimension index. Based on this, 2 determined that the dimensions corresponding incomplete cycles fall within the ranges [45, 64] U [109, 128].
We randomly sample 1000 tokens and compute the average activation values across heads for each dimension. The average activation values for every dimension of each layer are then plotted in Fig 7. The absolute activation values are significantly higher for dimensions corresponding to undertrained frequencies compared to others. This indicates that the RoPE rotation matrix pattern during pretraining has a notable impact on the distribution q, vector activations.
A.2. Influence of undertrained components in time-domain
We conduct visualization and ablations to further investigate the influence of undertrained components in RoPE.
N(u=0, a=1.0)
-1
Ni a=1.0)
Attention Weight
/o halp
20
40
60
80
100
120
N/u=1.0, d=1.0)
Ideal RoPE's Contribution
Actual RoPE's Contribution
0.6
0.4
20
40
60
80
100
120
Distance
Figure 8. The statistical average contribution of RoPE attention scores. We sample Ik Q and K vectors from Gaussian distributions with mean of 0, 0.1, and 1.0. The long-distance decay effect weakens as the mean decreases, disappearing entirely when the mean is
In Fig 8, we visualize the time-domain pattern of RoPE. To get the "Ideal RoPE's Contribution", we suppose the Query and Key vectors are equal to constantly (as in the original paper of RoPE (Su et al., 2024). To get the "Actual RoPE's Contribution", we suppose the Query and Key vectors obey the Gaussian distribution and sample 1000 times to get the mathematical expectation the "Actual RoPE's Contribution".. It can be seen that RoPE brings decay attention score in its naive setting, which brought by the undertrained components in RoPE. We hypothesis that this decay brings positional bias that may adversely affect robustness to out-of-domain rotation matrix values during length generalization.
As the positional bias can be eased if the mean of Query and Key vectors is set to zero, we conduct further ablation studies to normalize the Query and Key vectors before attention. Based on the results in 2, only if the position embedding contains components that cannot complete full cycle, the normalization brings better length generalization. Thus, these components are partially proved to deliver negative affect by positional bias. This experiment also demonstrate that the positional decay does not have beneficial influence on length generalization.
12

---

Enhancing Attention's Periodic Extension for Length Generalization
A.3. Setups of main experiments
Our main experiments are conducted with 4 cards NVIDIA A6000 (maximum GPU memory=48GB).
Pre-Training Settings. The pre-training of 60M/180M/1.22 OLMo in 10B tokens lasts for 10/20/100 hours, respectively. The time-consuming has a linear relation with the number of tokens in other settings. For all model scales and experimental settings, we select 6e-4 as the learning rate and warm-up for 10000 steps with cosine scheduler. While the mini-batchsize on each device is different for each model, we accumulate gradients until the global batchsize reaches 1024 in all experiments.
Fine-T Fine-Tuning Settings. Our fine-tuning procedure totally aligns with the official SmoILM-I.7B-Inttree recipe (Allal et al., 2024). We fine-tune SmolLM-I 1.7B with approximately 350k samples for one epoch, using the Adam W optimizer with learning of 3e-4 and cosine scheduler with warmup ratio of 0.1. The entire training process takes 4 hours machines.
Table 3. The hyper-parameter of different model scales. The upper part is the parameter shared by all experiments, while the lower part is parameter specific to FoPE. We use different mini-batch sizes depending on the max seq length: the first corresponds max seq 1ength=512, and the second corresponds to max seq 1024th=1024.
Model Scale
60M
180M
1.2B
Num N
8
8
16
Num Layers
8
8
16
MLP Ratio
8
8
8
Head_ Dim
64
128
128
Mini Batchsize
64/32
32/16
8/4
Var Freq
0.3
0.4
0.6
Num D
64
128
128
Evaluation Settings. For pre-training, evaluations are conducted on the checkpoint from the last step. For fine-tuning, we save checkpoints every 100 steps and report the best result for each method. This is partly due to YARN (Peng et al., 2023) being prone to overfitting with excessive fine-tuning steps, a limitation not observed FoPE.
A.4. Experimental results on more benchmarks
In addition to evaluating the long-context capabilities of FoPE, we also assessed its performance on a variety other tasks (see Tables 5, and 6). The results show that FoPE performs comparably to, even better than, other baselines, demonstrating that FoPE can enhance long-context capabilities without compromising performance on other tasks.
Table 4. Accuracy of 1.2B models with different positional embeddings several downstream tasks.
Method
basic arithmetic
social iqu
winogrande
openbook qa
sciq
hellaswag
piqa
commonsense qa
NoPE
25.67
43.71
51.86
29.80
76.70
41.83
68.83
31.61
51.40
42. 14
ALiBi
24.97
42.53
53.12
31.40
77.70
43.06
69.42
33.42
53.68
42.93
KERPLE
25.03
43.81
54.07
32,40
78.20
43.65
69.64
32.92
52.46
43.22
FIRE
25.60
42.63
49.88
33.40
77.00
42.75
69.31
32.92
50.35
42.38
RoPE
24.60
43.45
51.54
33.60
77.10
43.36
70.13
33.01
52.98
42,98
FoPE
26.17
44.12
53.20
32,20
77.80
43.83
70.08
32.92
53.33
43.37
Table 5, Cross-Entropy Loss of 1.2B models with different positional embeddings several downstrearn tasks.
Method natural qs open trivia qa wiki
easy
avg.
NoPE
1.4334
1.6129
1.3077
1.4513
ALiBi
1.3879
1.6057
1.2789
1.4242
KERPLE
1.4149
1.5878
1.2825
1.4284
FIRE
1.4655
1.6258
1.3118
1.4580
RoPE
1.4114
1.5973
1.2588
1.4225
PoPE
1.3818
1.5736
1.2272
1.3941
13

---

Enhancing Attention's Periodic Extension for Length Generalization
Table 6. Evaluation of 1.2B models with different positional embeddings mmlu.
Method
stem
social science
humanity
other
avg.
NoPE
25.47
25.38
28.01
24.39
25.81
ALiBi
25.93
27.20
26.44
24.39
25.99
KERPLE
26.02
25.97
27.82
24.82
26.16
FIRE
26.34
27.41
26.25
24.16
26.04
RoPE
27.01
27.26
27.87
24.53
26.68
FoPE
27.30
27.31
29.89
25.79
27.57
B. Implementation Details
Implementation of FoPE can be easily achieved with weight matrix W E MDD where Mo is the number of zero-frequency components in each head. This matrix maps the coefficients of all frequencies to Fourier Series for each dimension. Since the zero-freqeenyy sinusoidal function does not affect the original hidden states, the output dimension less than the dimension of each head. To introduce more diversity and better simulate the randomness of the Spectrum Damage, we assign separate weights for different heads, as well as for the cosine and sine functions. In our implementation, gradients are required for these matrices, so FoPE adds negligible memory and computation overhead compared RoPE.
Pseudo-code of FoPE is shown in the final pages.
C. Theoretical Details
C.1. Derivation Lemma 3.1
Given non-linear function g(x), be rewritten as a power series by Taylor expansion:
((x)
Σ
apop
(14)
PEN
Suppose the input is a foubleefrrquenccies function:
x<()) = coS win + cos
(15)
the output becomes:
((x(n)) Σ
ap ( w2s
(16)
PEN
Considering the product conversion formula of sinusoid function:
cos cos
|cos(a 0) + cos(a + 0)]
(17)
2
Thus, each sub-function of (16) can generate harmonic functions. For example, when 2: (cos win n + coswyn)² 2 (cos n2 + (coSw2n)² + 2 cosw cos
1 cos 2wzn + cos(w11 w2)n + cos(wl + wa)n
(18)
= + 2
cos 2w1 n + 2
C.2. Derivation Equal (10)
Before we begin the derivation, let's familiarize with two functions in time domain: Rectangular Pulse Function and Single-Frequency Function.
Given Rectangular Pulse function:
r(t) = { 1.
M < T
(19)
0, t > T
14

---

Enhancing Attention's Periodic Extension for Length Generalization
whose Fourier Transform is Sa Function:
T {" R(w) e-Jut dl 2 sin(wT)
(20)
-T
Given Single-Frequency function:
f(t) edimm
(21)
whose Fourier Transform is:
F(w) f 1.00
@ -jt dt
(22)
which is equal to an Impulse Function in frequency domain:
((mm) = {
1,
w = Wm
(23)
0, w + Wm
Then, Eq (10) can be easily derivated, as the context truncated with length T can be seen as:
x(t) f(t) r(t)
(24)
Its Fourier Transform in discrete case Eq. (10).
C.3. Low-Pass Property of RoPE
As the frequency of RoPE obey wm = 1/ 1/0(2m/M), only samples frequency less than and samples more densely near 0.
According to the following Theorems and Corollary, RoPE only filters the low-frequency components from hidden states for attention, thus achieving low-pass filter in attention.
Theorem C.1. For any continuous function, to reconstruct the original function using its discrete samples without distortion, the relation between the sampling frequency fa and the highest frequency component fr. of the function must obey: fa 2 2
Theorem C.2. For any discrete function, the sampling frequency Wa constant: W. = 2T.
Corollary C.3. The frequency of any discrete function obey: S
Comparing with high band w E (T/2, m) lower frequency components delivers a longer term influence on other tokens in time domain, letting attention have stronger bias to grab long- long-distance information.
C.4. The Relationship Between the Theoretical Modeling and Practical Implementation of RoPE
Let's define the query and key in the complex domain as = ||g|le and IP=, in the matrix domain as दे [uz: 4y] and k = [Kz, J.y]. According to the geometrical meaning of complex number, the relationships between the variations above are ||/|| 02 og and 0g = arctan 4x
+ kv, with similar expressions holding for ||||| and 0z.
Thus, the implementation based on the rotary matrix reflects the geometric interpretation of phase rotation the complex domain, as illustrated by the following derivation:
15

---

Enhancing Attention's 's Periodic Extension for Length Generalization
(fg (Xm: m), fu n!)
(25)
(qeimo, keino)
(26)
Relak' cl(m-ni00 Relak'eelmnnin
(27)
RR|[|all [|llell - - (mm
(28)
||ul|||&|| cos(O, 02 + (m - n)]0
(29)
|| [cos(0,- - 02) cos(mm n)0 sin(0, 0z) sin r2]0]
(30)
|| [[cos By cos 0% + sin 0g (in cos(m n)0 (sin 0g cos 02 cos 0g sin 0k) sin(m n))
(31)
(gutis + 4y cos(mm m n (gykis (okyy sin n)
(32)
cos(mm n)0
sin(m - n)0
Rizz
[vl
(33)
sin(m n00
cos(m n)0
kiy
cos mB
sin m8)
cos n0 sin no
sin no
kg
1
[vl
(34)
sin m0
cos cosmo
cos no
kgg
(Rg, 9)" r (Ro, k)
(35)
q Rø k
(36)
16

---

Enhancing Attention's Periodic Extension for Length Generalization
class FouricrEnbedding (RotaryEmbeddinng) def _init_ (self, config):
super ()__ntt____it__ (config)
self. siff.inuttddim sel f. inw__reeeee..._oe size((1) (-))
self. salfooutput_diigg = self.input_die din. if seit. selt.input_iim <= httthhead_dim// else self
sead_iim//4
self.sin_coef nn. (
forch.randn www.f.configg....oo.oooooo config.n_heads, self. self.inputddim, self falf.on out.foutput_dim)
requires_grad-Falsee
self.cos_con Pn.Parameter(
tarch wtt.eee....o..gnn.ee.eeeeeee self.input_dee self.output_dim) requires_grad-Falec
otteea/w.....t.eeee..ooooooolold gain=self.config:
rope_foorieer_nntttoommnggiinn
torch.nn.inin ttt. cos__ sain-self.conffl
self.sin coef torch.eye (self input_iitiit dim, device=self. coef.devize
20
self.cos_coef += torch.eye inppttiniittttiii dim, device=self.com cos cww_scoe.ooooooooooooooooooooooe device)
21
def apply_rotary__o___mbddllllll possin, pos_cos, t): fourier_sin torch wor.h. ("ataaemmm"bbbi....Ddbddbbabt pos_sin, self.sin self.
(mn.coeef.uum((iir222.. keepdim=True))
24
fourier_cos = torch.einsum ("D_
pos_cos, self.cos_coef self.
sts_ccoff.ummddieeeee (din==222 keepdin=True)
fourier_sin E.padd(nnuttffuurressss pad= (0, hal..beeacdaa/2-forrrr........
(-1)), mode="constant value=1)
27
fourier_cos -
pad- felf.head 2e1F
(-1)), mode=" "constant"," value-1)
29
fourier _sin torch.cat (ffuurier sin, (urree__iin) dim=-1)
30
fourier_cos = tarch.cat ((fourier_cos, fourier_cos), dim=-1)
return ((t fourier_cos)
fourier_sin)) .to (t.dtype)
34
class RotaryEnbeddiing (nn Modulc):
35
def _iiit_ (self, config):
super () (nnerit)t_i_____ init_()
self.config config
39
self.dim = wewf.config....o cont.g.. // self col.f.ooffi..a.eeet
self.inv_free self wnw.:eeeeeiitt_eeeeeeeeeeee
def get_inv_fre (self, dim): inv_freq 1.0 / (
43
44
self.con forch.floate dim)
** Shorch.aranee (0, dim, 2, device-device, dtype-
46
if celf.config. is True:
47
inv_freqlinv_freg 2xtorch.pi/sol. s. wonfll.max.equunneo.ooooo inv_freq = hnw_frrrlinvvffeee != 0.0]
48
inv_freq
17

---

Enhancing Attention's 's Periodic Extension for Length Generalization
def get_rotary_embeddingg (self, seq_len):
with tcrch. acrchaautocast (device. type, enabled-False): seq torch arange (seq_len, device=device, dtype=torch. float)
freqs torch. htdd",, seq, self.inv_frep
if self.config fope is True: positions freqs.unee unsquezze (0)
else:
positions torch.cat (freqs, freqs), dim=-1) unsqueeze (0)
10
return positions.sil sin (),, positions.cos ()
def rotate_half x):
B, nh, I, hs s.size()
15
x = x.view nh, т, 2, hs // 2)
x1, x2 u.unbind
19
return forch.cat ( (-x2, x1), dim=-1)
30
def apply_rotary_pos_embbbe (self, pes_sin, pos_cos, t):
((t pos_cos) (atl pos_sin)) to (t..pteee)
24
def fsrward(self, x, len)
25
with torch.autocast (x.device.type wxee
enabled-False)
x_len shapeee[
pos_sin, pos_cos self self..et_rotrr__mmddddno. (al1
28
poa_sin pos_sin.type_as (x_) poa_cos = pos_cos.type_as(xx]
29
x_ self. selff.aply_roarr__oss.ooo
pos_sinli, i, all_len x_len all_len, il,
33
pos_cosl;, ;, all_ x len, :), x
34
return x_ ty.eype_as (x)
18


Reviews:
[{"review_id": "qjqfqNTfEc", "reviewer": "Reviewer_FqQv", "summary": "## update after rebuttal\n\nI read the latest clarification by the authors, and understand q and k in Re[qk*e^{i{m-n}\\theta}] are not exactly the 2-dim vector [q_x, q_y]^T but an implicit complex number. I raised my score back.\n******\n\nThis paper points out that in RoPE, different dimensions correspond to different frequencies,\nand suggests viewing the interaction between queries and keys from the perspective of non-Uniform DFT (Eq.2 - Eq.4).\n(However, there is an unclear point here: while RoPE is motivated by complex number rotation, its implementation is actually based on vector rotation--—they are not equivalent. The authors' analysis is based on the former, but the actual analysis object is the latter, and the authors' implementation is also the latter. Therefore, at minimum, they should explain this gap.)\n\nBased on this **deconstructive analysis**,\nthe authors identify two issues:\n1. If we consistently view RoPE from a signal perspective, we cannot ignore the spectral leakage caused by activation functions and linear layers;\n2. Considering that low frequencies correspond to stable components in long contexts, while the text used in training is not very long, these low-freq components are insufficiently trained.\n\nThe authors propose two improvements to RoPE:\n1. Allow feature dimensions that previously corresponded to a *single* frequency to now correspond to *multiple* frequencies, while still maintaining a *primary* frequency, enabling more flexible adjustment of the disturbed spectral information.\n2. Set the insufficiently trained dimensions directly to 1.\n\nIn experiments, the authors primarily validate the benefits of FoPE for length extrapolation capabilities. Experimental scenarios include pre-training, continual pre-training, and fine-tuning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful comments from the reviewer, we are going to make clarifications for the concerns above.\n\n# Clarification for the \"heuristic approach\" and \"practical approach\" of RoPE/FoPE\nIt seems this is the major concern of the reviewer, we acknowledge the importance of clarifying this point:\n\n1. **The Rotary Matrix implies the real part of the complex number in Matrix Space, which is also the main difference between the \"heuristic approach\" and \"practical approach\" of RoPE.** The derivation is in Sec 3.4 of Roformer's original paper [1], we also provide a derivation under our understanding: \nIn the 2D case of RoPE, the vectors $q$ and $k$ can be viewed either as 2D plane vectors ($\\mathbf{q}=(q_x,q_y)^T$) or as vectors in the complex plane ($\\mathbf{q}=\\|\\mathbf{q}\\|e^{i\\theta_q}$, $\\|q\\|=\\sqrt{q_x^2+q_y^2}$, $\\theta_q = \\arctan{\\frac{q_x}{q_y}}+k\\pi$ ). The inner product of the two vectors is equal to $\\langle\\mathbf{q},\\mathbf{k}\\rangle=Re[\\mathbf{qk}^*]$ (the proof is at the end of this reply).\n\n2. **As taking the real part does not change the frequency-domain property of a vector, thus: ① RoPE's actual implementation still achieves Eq.2; ② we can still write a strict Fourier form equation.**\n\n3. **As a result, our theoretical analysis and implementation are well-aligned.** To provide better clarity, we will explicitly include a derivation in the future revisions.\n\n4. **Our further empirical results also demonstrate that the actual implementation of RoPE properly achieves its theoretical motivation, also well-aligning with our frequency-domain analysis (See https://anonymous.4open.science/r/FoPE-Supplementry-Material/Empirical_Analysis_on_Spectrum_Damage.pdf).** In this experiment, we force every token only has one frequency component in the first layer. Then, we compare the spectrum of the first and second layer with DFT on attention map (aligned with the derivation in Sec 2.2).\n\n# Relationship between fourier/signal modeling and multiple stacked blocks\n\n- Firstly, we want to clarify that, **as the Fourier Transform is a linear transform [2], it does not change the main properties of models.** Thus, the multiple stacked blocks are still in charge of modeling more diverse/complex and higher-order functions/signals, which improves the expression ability of models.\n- Secondly, based on our modeling from a signal/frequency-domain perspective, **different layers actually process signals of different frequency distribution.** This is useful for improving the expression ability of models. However, **RoPE wrongly regards the frequency distribution as same in different layers, leading to drawbacks in length generalization**.\n- Lastly, our modeling in frequency-domain can analyze the periodicity of attention mechanism, providing insights into the drawback of RoPE in length generalization.\n\n# Typical range of values for $\\omega_m$\nWe have shown these statistics in Appendix D.3 (line 727-728). The frequencies of RoPE are between (0, 1] and samples more densely near 0.\n\n# Comparison with more baselines on more benchmarks \nWe also conduct more supplementary experiments, please check our reply to Reviewer LYp1 for detailed results.\n\n---\n## Reference\n[1] Jianlin Su, et al. RoFormer: Enhanced Transformer with Rotary Position Embedding.\n\n[2] Oppenheim, et al. Signals and Systems.\n\n---\n## Additional Proof\n$$\\langle f_q(\\mathbf{x_m},m),f_k(\\mathbf{x_n},n)\\rangle$$\n$$=\\langle \\mathbf{q}e^{im\\theta},\\mathbf{k}e^{in\\theta}\\rangle$$\n$$=\\text{Re}[\\mathbf{qk}^*e^{i(m-n)\\theta}]$$\n$$=\\text{Re}[\\|q\\|\\|k\\|e^{i[\\theta_q-\\theta_k+(m-n)\\theta]}]$$\n$$=\\|q\\|\\|k\\|\\cos(\\theta_q-\\theta_k+(m-n)\\theta)$$\n$$=\\|q\\|\\|k\\|\\left[ \\cos(\\theta_q-\\theta_k)\\cos(m-n)\\theta - \\sin(\\theta_q-\\theta_k)\\sin(m-n)\\theta\\right]$$\n$$=\\|q\\|\\|k\\|\\left[(\\cos\\theta_q\\cos\\theta_k+\\sin\\theta_q\\sin\\theta_k)\\cos(m-n)\\theta-(\\sin\\theta_q\\cos\\theta_k-\\cos\\theta_q\\sin\\theta_k)\\sin(m-n)\\theta \\right]$$\n$$=(q_xk_x+q_yk_y)\\cos (m-n)\\theta - (q_yk_x-q_xk_y)\\sin (m-n)\\theta$$\n$$=[q_x, q_y][\\cos (m-n)\\theta, -\\sin (m-n)\\theta; \\sin (m-n)\\theta, \\cos (m-n)\\theta][k_x, k_y]^T$$\n$$=[q_x, q_y][\\cos m\\theta, \\sin m\\theta; -\\sin m\\theta, \\cos m\\theta][\\cos n\\theta, -\\sin n\\theta; \\sin n\\theta, \\cos n\\theta][k_x, k_y]^T$$\n$$=(\\mathbf{R_{\\theta,m}q})^T(\\mathbf{R_{\\theta,n}k})$$\n$$=\\mathbf{q^T R_{\\theta,m-n}k}$$", "claims_and_evidence": "I have concerns about the following claim:\n\nThe authors view the interaction between queries and keys from the perspective of non-Uniform DFT (Eq.2 - Eq.4),\nand they hope this perspective can be maintained throughout the model, thus proposing two improvements.\nThe starting point of these approaches is understanding RoPE's formula as Eq.2.\nHowever, **it should be noted that RoPE's actual implementation is not Eq.2.**\nEq.2 can be considered as the heuristic starting point of RoPE,\nwhile the actual implementation rotates each pair of dimensions by a certain angle, which is not equivalent to Eq.2,\ntherefore, in fact **we cannot write a strict Fourier form equation**.\n\nIt can be said that:\n**RoPE has both a \"heuristic approach\" and a \"practical approach\",\nand the authors' analysis is based on the former \"heuristic approach\" but improves upon the \"practical approach\".**\n\nI would like to hear other reviewers' and AC's opinions on this point.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There are no specific theoretical claims, but the paper's analysis has certain theoretical aspects. Please see my concerns in (Claims And Evidence).", "experimental_designs_or_analyses": "Yes. \n\nPS: I have not personally conducted similar experiments and have insufficient understanding of the experimental details.", "relation_to_broader_scientific_literature": "This paper provides insights into understanding Transformer architecture and solving long-range dependency problems from a signal analysis perspective.", "essential_references_not_discussed": "Not sure.", "other_strengths_and_weaknesses": "Other Strengths:\n1. Provides a deconstructive analysis of existing models；\n2. The proposed method is lightweight and effective；\n3. Although I am not very familiar with the experimental approaches in this direction, I find the authors' writing of the experimental section to be very fluent.\n\nOther Weaknesses:\n- I still have concerns about the signal-based explanation. First, there are the concerns raised in (Claims And Evidence). Additionally, I am concerned about: in the authors' analysis, attention with RoPE becomes an inverse Fourier transform form, which essentially treats the input as a frequency domain response. However, if this is the case, how can we analyze multiple stacked blocks from a signal perspective?", "other_comments_or_suggestions": "What is the typical range of values for $\\omega_m = 1/{\\theta}^{(2m/M)}$ in Eq.2? This should be explicitly stated.", "questions_for_authors": "Please refer mainly to the (Claims And Evidence) section,\nand also see the (Other Weaknesses) and (Other Comments Or Suggestions) sections", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kvdxBmyz69", "reviewer": "Reviewer_LYp1", "summary": "This paper analyses the limitations of Rotary Position Embedding (RoPE) in extending language model context length using Discrete Signal Processing theory. It identifies spectral damage from linear layers, activation functions, and insufficient frequency training as key issues affecting RoPE’s periodicity. To address this, the authors propose Fourier Position Embedding (FoPE), which constructs a Fourier Series and removes harmful frequency components to enhance length generalisation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed comments from the reviewer, we would like to make several clarification below.\n\n# Computation cost of FoPE is similar to RoPE\nWe agree with the reviewer that the efficiency and computation cost are essential for Position Embedding, thus:\n- FoPE keeps the rotary matrix having the similar shape as RoPE.\n- FoPE keeps the similar pipeline as RoPE: ① pre-compute the rotary matrix and save the matrix in cache before training; ② take out the matrix for rotation during training. \n\nTherefore, FoPE is as efficient as RoPE, independent of the model scale. (Reviewer FqQv also acknowledges that FoPE is lightweight and effective in \"Other Strengths And Weaknesses\")\n\n# Evaluation on more tasks and baselines\nIt is a nice suggestion to evaluate FoPE on more diverse tasks, thus we supplement the following experiments:\n- We validate the effectiveness of FoPE by evaluating on 10+ more benchmarks and comparing with more 3 more baselines. These supplementary experiments are conducted on OLMo-1.2B training on C4 datasets, keeping all of the settings similar to the experiments in Sec 5.2 (line 293-329). We do not show the results on GSM8K as all methods achieve nearly 0 acc, which may be caused by the lack of math data in C4.\n\n| methods | avg acc | basic_arithmetic | social_iqa | winogrande | openbook_qa | sciq  | hellaswag | piqa | commonsense_qa | arc_easy |\n|-|-|-|-|-|-|-|-|-|-|-|\n| NoPE | 42.14 | 25.67 | 43.71 | 51.86 | 29.80 | 76.70 | 41.83 | 68.83 | 31.61 | 51.40 |\n| ALiBi | 42.93 | 24.97 | 42.53 | 53.12 | 31.40 | 77.70 | 43.06 | 69.42 | **33.42** | **53.68** |\n| KERPLE | 43.22 | 25.03 | 43.81 | **54.07** | 32.40 | **78.20** | 43.65 | 69.64 | 32.92 | 52.46 |\n| FIRE | 42.38 | 25.60 | 42.63 | 49.88 | 33.40 | 77.00 | 42.75 | 69.31 | 32.92 | 50.35|\n| RoPE | 42.98 | 24.60 | 43.45 | 51.54 | **33.60** | 77.10 | 43.36 | **70.13** | 33.01 | 52.98 |\n| FoPE | **43.37** | **26.17** | **44.12** | 53.20 | 32.20 | 77.80 | **43.83** | 70.08 | 32.92 | 53.33 |\n\n| methods | avg acc | mmlu_stem | mmlu_social_sciences | mmlu_humanities | mmlu_other |\n|-|-|-|-|-|-|\n| NoPE | 25.81 | 25.47 | 25.38 | 28.01 | 24.39 |\n| ALiBi | 25.99 | 25.93 | 27.20 | 26.44 | 24.39 |\n| KERPLE | 26.16 | 26.02 | 25.97 | 27.82 | 24.82 |\n| FIRE | 26.04 | 26.34 | **27.41** | 26.25 | 24.16 |\n| RoPE | 26.68 | 27.01 | 27.26 | 27.87 | 24.53 |\n| FoPE | **27.57** | **27.30** | 27.31 | **29.89** | **25.79** |\n\n| methods | avg ce loss | natural_qs_open | trivia_qa_wiki | arc_easy |\n|-|-|-|-|-|\n| NoPE | 1.4513 |1.4334 | 1.6129 | 1.3077 |\n| ALiBi | 1.4242 |1.3879 | 1.6057 | 1.2789 |\n| KERPLE | 1.4284 | 1.4149 | 1.5878 | 1.2825 |\n| FIRE | 1.4580 | 1.4365 | 1.6258 | 1.3118 |\n| RoPE | 1.4225 | 1.4114 | 1.5973 | 1.2588 |\n| FoPE | **1.3941** | **1.3818** | **1.5736** | **1.2272** |\n\n- **Although FoPE is primarily designed for length generalization and long-context tasks, it also performs comparably or even surpasses all baseline methods**. But on MMLU, all methods have weak performance, perhaps caused by the training data. In the future, we may conduct more experiments on datasets containing reasoning and math data.\n- We also conduct evaluation similar to those in Sec 5.2 (See https://anonymous.4open.science/r/FoPE-Supplementry-Material/Supplementary_Main_Experiments.pdf). **FoPE still demonstrates the best accuracy on passkey retrieval and the second-best ppl on C4 (only slightly behind ALiBi).**", "claims_and_evidence": "The claims are generally supported  and shows performance (length generalisation) gains across benchmarks. However, it does not address the computational cost of FoPE, which is important for understanding the trade-off between performance and efficiency.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate. FoPE enhances length generalisation by improving position encoding, addressing spectral issues in long-context attention. The evaluation on diverse benchmarks (e.g., GovReport, MultiNews, TREC) effectively tests performance across varying context lengths. However, including more complex reasoning tasks would provide a more comprehensive assessment.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "While the paper demonstrates performance improvements on long contexts, it doesn't provide insights into how the Fourier Position Embedding method impacts computation costs of training and inference.", "relation_to_broader_scientific_literature": "This paper extends Rotary Position Embedding (RoPE) by analyzing its frequency-domain properties using Discrete Signal Processing (DSP) and identifying spectral distortions that hinder long-context generalization. Building on prior work in positional embeddings (Su et al., 2021; Press et al., 2022) it introduces Fourier Position Embedding (FoPE), which filters harmful frequency components to improve attention’s periodicity.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n1) Frequency-domain analysis of RoPE using Discrete Signal Processing (DSP), offering new insights into spectral distortions affecting long-context generalization.  \n2) Practical impact with Fourier Position Embedding (FoPE), demonstrating strong improvements in context extension across model scales and tasks.  \n\nWeaknesses:\n\n1) Computational cost of FoPE is unclear, especially for large-scale models.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1) Could the authors include benchmarks on reasoning tasks like MMLU and GSM8K to demonstrate greater task diversity?\n\n2) The authors are asked to provide a comparison of computational analysis.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jILJLHuBRr", "reviewer": "Reviewer_LcUb", "summary": "The paper introduces the Fourier positional embedding (FoPE) based on Fourier series. The authors begin by analyzing the rotary positional embedding (RoPE) method in the frequency domain. Further analysis of the feed-forward network yields additional information that linear layers produce spectrum leakage (mix of frequencies) and the activation functions lead to spectrum distortion due to their harmonics generation. These two types of spectrum effects are causes for spectrum damage for context length generalization, hindering RoPE’s effectiveness. Then, the attention mechanism is shown to produce spectrum damage on low frequencies for *undertrained* components. Following this analysis, FoPE is proposed as a multi-frequency representation for each dimension based on Fourier Series, instead of a single frequency like RoPE. The proposed method also includes zeroing the low frequency components to control for undertrained frequencies. Next the authors compare FoPE against RoPE and ALiBi on pre-training, and against YARN extension on continual pre-training of OLMo, and against RoPE on fine-tuning SmolLM-1.7B. They measure perplexity and passkey retrieval to test the context length generalization of the resulting models. In most cases, FoPE shows better generalization when increasing the context length at test time. Further, ablation analysis is performed to test both the zeroing of frequency and without the Fourier series, and various analysis related to other aspects of the language model.\n\n## update after rebuttal\nThanks to the authors for clarifying the various points.  After a thoughtful consideration, primarily given the additional comparisons, I've decided to update my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the elaborate comments, we will address them according to their proposed order.\n\n# Clarification for \"undertrained components\" and the impact of poor threshold\n- **We have jointly defined the \"undertrained components\" and \"floor frequency\" in Fig 2, Sec 3.3 and Sec 4, using both visualization and formula.**\n- Also, **these two concepts are well-known in the area of length generalization**, mentioned by many papers [1, 2].\n- As for the impact of poor threshold, we add ablations on 60M OLMo trained with 512 context length (similar setting as Sec 5.2). **Our findings suggest that selecting a floor frequency no-less than $2\\pi/L$ is necessary, but a higher threshold does not have a significant influence.**\n|threshold (f=$2\\pi/L$)|0|0.5f|0.75f|f|1.25f|1.5f|\n|-|-|-|-|-|-|-|\n|**ce loss on 8192 length**|6.86|6.54|5.97|5.84|5.86|5.85|\n\n# Empirical analysis of the Spectrum Damage on well-trained models\n- In Sec 5.4, our experiments are based on SmolLM-1.7B, a famous model well-trained by HuggingFace for academic purposes.\n- In Sec 5.6, Fig 3 and Fig 7, we have a detailed empirical analysis of the influence of Spectrum Damage on LLaMA-2-7B.\n- To further illustrate this phenomenon, we conduct another empirical analysis on LLaMA-2-7B (See https://anonymous.4open.science/r/FoPE-Supplementry-Material/Empirical_Analysis_on_Spectrum_Damage.pdf). We force every token only has one frequency component in the 1st layer. Then, we compare the spectrum of the 1st and 2nd layer after DFT on the attention map. **Many other frequency components appear in the 2nd layer, which demonstrates the happening of Spectrum Damage.**\n\n# Advantage of FoPE over ALiBi and the well-known negative influence of linear declined attention\n- **Our results in Sec 5.2 clearly show FoPE's advantages compared to ALiBi, it is unclear why the reviewer got the opposite conclusion.**\n- We agree that \"the mutual information between tokens behaves as a power-law distribution\", and we appreciate this perspective.\n- However, this property is only useful for short context modeling, but not for long-context modeling. For example, **the \"linear declined attention\" hinders the retrieval of long-distance information, thereby lacking the crucial ability for long-context modeling.** \n- **Not only do our results in Fig 1 demonstrate this phenomenon, but many well-known work also have the similar viewpoint [2, 3, 4]** that \"linear declined attention\" and ALiBi is fall short in long-context applications.\n\n# Clarification for Sec 5.4 (the maximum sequence length and base model performance)\n- We have provided the maximum sequence length in Sec 5.4 (line 328 and 373).\n- The performance of SmolLM-1.7B-base is as follows, which do not influence our conclusions:\n|length|gov_report|multi_news|trec|triviaqa|samsum|\n|-|-|-|-|-|-|\n|0-4k|9.10|5.06|42.0|68.53|17.97|\n|4-8k|9.50|7.06|51.0|69.21|16.70|\n|8k+|8.02|6.01|38.0|65.81|20.17|\n\n# Advantage and clarification of \"smaller models' performance decay faster than larger ones\"\n- Firstly, this phenomenon implies FoPE is a scalable method, which should be considered as an advantage.\n- Secondly, it is not odd that larger models have better generalization.\n\n# FoPE is valuable for many reasons, although current LLMs achieve 100k+ context length\n- It is well-known that current Transformer-based LLMs achieve 100k+ context length using extrapolation as YARN [1]. **But in Fig 4, we have shown FoPE can also be used for extrapolation and has better performance than YARN.**\n- With FoPE, models achieve better length generalization trained with a much shorter context length. **Thus, FoPE can significantly improve the training efficiency of models (the longer length, the much slower training).** This is valuable for saving time and money.\n\n# Comparison with more baselines on more benchmarks\nPlease check our reply to Reviewer LYp1 for detailed results.\n\n# Influence of residual connections, LayerNorm and specific activation functions\n- We have modeled the influence of activation functions in Sec 3.2.\n- Considering the residual connections and LayerNorm only deliver linear transform in frequency domain, we did not include their analysis in our paper. But for clarity, we will contain them in future revisions.\n\n# Generalization ability of FoPE on non-training data\n- **We have evaluated the OOD generalization in Sec 5.2.** (trained on Gutenberg Books and evaluated the ppl on C4)\n- Our evaluations on Sec 5.4 include datasets from SCROLLS (See Sec 5.4).\n\n# Clarification for Figure 6\nThe description is clearly presented just under Fig 6 in Sec 5.5.\n\n---\n\n[1] YaRN: Efficient Context Window Extension of Large Language Models. ICLR 2024.\n\n[2] FIRE: Functional Interpolation for Relative Positions Improves Long Context Transformers. ICLR 2024.\n\n[3] CLEX: Continuous Length Extrapolation for Large Language Models. ICLR 2024.\n\n[4] Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs. NeurIPS 2024.", "claims_and_evidence": "The claims made in the submission are mostly supported. A formal definition of *undertrained* components of attention is missing, so it is unclear when does spectrum damage based on this definition matters. An empirical analysis of the potential Spectrum Damage on a well-trained model is lacking.", "methods_and_evaluation_criteria": "The derived methods seem sound. The selection of the low frequency threshold is not analyzed. What is the impact on the method if an incorrect or poor threshold selected? The evaluation criteria for the method seems correct.  The datasets could be selected to test generalization further (e.g., perplexity on SCROLLS).", "theoretical_claims": "The derivations in the paper seem correct. Please, see comments above about undertrained components.", "experimental_designs_or_analyses": "The proposed method is evaluated on a pre-training on C4 and Books datasets. The results show that ALiBi performs better than FoPE. In the text, the authors suggest that there is an “issue” with how ALiBi considers the linear declined attention, putting a considerable effort to learn the short-distance information. However, it is well-known that the distribution of mutual information between tokens behaves as a power-law distribution [1]. Namely, this is a property of the data, and the model exploits such prior knowledge (and not the other way around). In addition, pre-trained language models have better perplexity than the ones showed in Section 5.2, and the models are much bigger in size today (using more train tokens) than those used in the paper. \n\nThe passkey retrieval results in section 5.2 show that the bigger the model, the better the retrieval results. If the 3 FoPE model sizes have the same embeddings and trained with the same context length, then why are the smaller models unable to generalize in the same way as the biggest model? Their curves decay much faster. Also, existing models can solve tasks beyond 8k context length today. It is unclear if they can generalize similarly as FoPE or even better. This suggests that there may be an issue when training the models or with the experiment itself.\n\nLast, it is unclear what is the maximum sequence length for section 5.4? Also, what is the performance of the baseline model *before* fine-tuning?\n\nOther positional embedding methods have not been compared. \n\n[1] “Critical Behavior in Physics and Probabilistic Formal Languages”, Lin and Tegmark, 2017", "relation_to_broader_scientific_literature": "The contribution of FoPE and proposes that context length generalization is of some value to the community. Current LLMs have context lengths that are very large (100k+ tokens), thus the significance is low. The theoretical analysis is well executed but limited (i.e., how do residual connections influence the frequencies? How does the LayerNorm or the specific activation functions?).", "essential_references_not_discussed": "Not aware of such references.", "other_strengths_and_weaknesses": "* Theory and derivations are valuable in the paper. \n* The experiments and results are not conclusive. It’s hard to really conclude the value of FoPE versus RoPE or other not-compared embedding methods.", "other_comments_or_suggestions": "The descriptions in the text of the experiments related to Figure 6 look disconnected to the Figure itself.", "questions_for_authors": "See questions in sections above.\n* Can you learn the threshold value for the low frequencies?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YhCxrdG5sx", "reviewer": "Reviewer_FbaF", "summary": "The paper analyzes how RoPE enables periodic attention patterns and then analyzes the limitation of RoPE in that regard.\nThe authors argue that the limitation arises from spectral damage prevalent when RoPE is used with typical DL architectures.\nThe authors propose FoPE, which is based on RoPE, but while RoPE treats each dimension as a single-frequency function, FoPE models each dimension as a Fourier Series, consisting of a dominant frequency component and several harmonic components.\nAdditionally, FoPE clips low frequencies because the authors argue these are undertrained.\nIn experiments, the authors show that FoPE maintains a better performance for increased context lengths.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments from the reviewer, we want to deliver further explanations accordingly.\n# Comparison with more baselines on more benchmarks\nIt is a nice suggestion to compare FoPE with more baseline methods, thus we supplement the following experiments:\n- We validate the effectiveness of FoPE by evaluating it on 10+ additional benchmarks and comparing it with 3 more baselines. These supplementary experiments are conducted on OLMo-1.2B training on C4 datasets, with all experimental settings kept consistent with Sec 5.2 (line 293-329).\n\n|methods|avg acc|basic_arithmetic|social_iqa|winogrande|openbook_qa|sciq|hellaswag|piqa|commonsense_qa|arc_easy|\n|-|-|-|-|-|-|-|-|-|-|-|\n|NoPE|42.14|25.67|43.71|51.86|29.80|76.70|41.83|68.83|31.61|51.40|\n|ALiBi|42.93|24.97|42.53|53.12|31.40|77.70|43.06|69.42|**33.42**|**53.68**|\n|KERPLE|43.22|25.03|43.81|**54.07**|32.40|**78.20**|43.65|69.64|32.92|52.46|\n|FIRE|42.38|25.60|42.63|49.88|33.40|77.00|42.75|69.31|32.92|50.35|\n|RoPE|42.98|24.60|43.45|51.54|**33.60**|77.10|43.36|**70.13**|33.01|52.98|\n|FoPE|**43.37**|**26.17**|**44.12**|53.20|32.20|77.80|**43.83**|70.08|32.92|53.33|\n\n|methods|avg acc|mmlu_stem|mmlu_social_sciences|mmlu_humanities|mmlu_other|\n|-|-|-|-|-|-|\n|NoPE|25.81|25.47|25.38|28.01|24.39|\n|ALiBi|25.99|25.93|27.20|26.44|24.39|\n|KERPLE|26.16|26.02|25.97|27.82|24.82|\n|FIRE|26.04|26.34|**27.41**|26.25|24.16|\n|RoPE|26.68|27.01|27.26|27.87|24.53|\n|FoPE|**27.57**|**27.30**|27.31|**29.89**|**25.79**|\n\n|methods|avg ce loss|natural_qs_open|trivia_qa_wiki|arc_easy|\n|-|-|-|-|-|\n|NoPE|1.4513|1.4334|1.6129|1.3077|\n|ALiBi|1.4242|1.3879|1.6057|1.2789|\n|KERPLE|1.4284|1.4149|1.5878|1.2825|\n|FIRE|1.4580|1.4365|1.6258|1.3118|\n|RoPE|1.4225|1.4114|1.5973|1.2588|\n|FoPE|**1.3941**|**1.3818**|**1.5736**|**1.2272**|\n\n- **While FoPE is primarily designed for length generalization and long-context tasks, it also performs comparably or even surpasses all baseline methods**. But on MMLU, all methods have weak performance, perhaps caused by the training data. In the future work, we may conduct more experiments on datasets containing math data.\n- We also conduct evaluation similar as Sec 5.2 (See https://anonymous.4open.science/r/FoPE-Supplementry-Material/Supplementary_Main_Experiments.pdf). **FoPE still demonstrate the best accuracy on passkey retrieval  and the second-best ppl on C4 (only slightly behind ALiBi).**\n- As time is limited, we have not conducted experiments on RWKV, xLSTM and Mamba. But it is really a nice suggestion to consider the length generalization capabilities of models with different architectures. We would like to cite these papers in related work in our future revisions.\n\n# Clarification of the hyper-parameters' influence on FoPE\nWe agree with the reviewer that hyper-parameters optimization is crucial for the large-scale training, but we would like to make some clarifications:\n1.**In a wide range of parameters we tested, FoPE consistently demonstrates a significantly better performance than RoPE**. \n2. **The performance of FoPE is not highly sensitive to hyper-parameters** (see ablation in Fig 6 and Sec 5.5), although an elaborate hyper-parameter selection of FoPE may lead to a better performance.\n\nThus, **even without a careful selection of hyper-parameters, FoPE is still competent for replacing RoPE in Transformers**. \n\nAdditionally, we emphasize that hyper-parameter tuning is an inevitable procedure before pre-training nowadays, many other hyper-parameters also need to be selected (i.e. hidden_dim, num_heads, ...). As for FoPE, the ablation studies in Fig 6 and Sec 5.5 suggested that:\n- The optimal $\\sigma$ for FoPE increases as hidden_dim and num_layers grow, which is expected, as larger models suffer more from spectral damage.\n- The best $D$ tends to be slightly bigger than the head_dim of each attention head. This is because too few frequency components cannot represent the spectrum damage well, , while too many exceed the model’s representational capacity.\n\nBut it is a nice suggestion to have a better selection strategy, we would like to continually research this problem.\n\n# Influence of clipping low frequencies\nIt is an insightful suggestion to consider the negative influence of clipping low frequencies, we want to make some clarification below:\n- \"Clipping low frequencies\" is quite similar to the \"interpolation\" used by many extrapolation methods like YARN[1]. As this operation is mainstream solution for LLMs' length generalization, we suppose \"Clipping low frequencies\" does not have significant negative influence in most tasks.\n- **Also, FoPE has a consistent performance on diverse benchmarks, .** Thus, the answer to this problem is still unclear. But we would like to continually consider this issue.\n\n---\n\n[1] YaRN: Efficient Context Window Extension of Large Language Models. ICLR 2024.", "claims_and_evidence": "The primary claim—that FoPE improves length extrapolation—is well-supported by empirical analysis. The experiments effectively showcase FoPE’s advantages over RoPE, and the inclusion of comparisons with YARN strengthens the argument.", "methods_and_evaluation_criteria": "The evaluation methods and criteria appear appropriate. The experiments test models up to 1.2B parameters, a reasonable scale for assessing generalization. The comparison with YARN adds credibility.", "theoretical_claims": "I briefly reviewed the theoretical claims presented in the main paper and did not identify any obvious issues.", "experimental_designs_or_analyses": "The experimental design seems to be valid and sound. Models up to 1.2B parameters are tested, which is already a reasonable size to support generalization to practical settings. The comparison optionally includes YARN, which strengthens the argument.\nAs I am not deeply familiar with related approaches, I don't know if a comparison to additional methods would be appropriate.\nA comparison to non-transformer architectures that provide favourable length extrapolation capabilities would be interesting in addition (e.g. see Figure~7 in [1]).\nThat said I have one concern regarding the hyperparameters D and sigma. The authors rightly conduct a sensitive analysis which shows that the parameters seem to be important. However, there is no clear guidance on how to select this parameter a-priori.\n\n[1] xlstm: Extended long short-term memory, M Beck et al. Advances in Neural Information Processing Systems 37", "relation_to_broader_scientific_literature": "The paper is closely related to research on positional embeddings for transformers and the broader literature on length extrapolation. The study builds on RoPE, a widely used technique in modern deep learning models.", "essential_references_not_discussed": "I am not aware of any essential references that are missing.", "other_strengths_and_weaknesses": "Strengths:\n- Clear motivation with according analysis of the approach (spectrum damage) \n- Evalution includes comparision to length extrapolation technique (yarn)\n\nWeakness:\n- No guidance for hyperparameter selection (D and sigma) although the methods seem to be sensitive to these parameters.\n- No comparison to non-transformer LLM architectures that provide more favorable length generalization properties like RWKV, xLSTM or Mamba (e.g. see Figure~7 in [1])\n\n[1] xlstm: Extended long short-term memory, M Beck et al. Advances in Neural Information Processing Systems 37", "other_comments_or_suggestions": "-", "questions_for_authors": "- How would you select the sigma and D hyperparameter for a new large-scale models where a hyperparameter search is not feasible because of the scale/cost of the model training? (The sensitivity of these parameters could affect the real-world usability of FoPE)\n- Could clipping low frequencies have unintended consequences in certain applications?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper uses insights from discrete signal processing to analyze how RoPE enables periodic attention patterns and its limitations in that regard, identified as a form of "spectral damage". The paper proposes FoPE as a solution, which is differs from RoPE in which it models each dimension not as a single frequency but as a Fourier Series, consisting of a dominant frequency component and several harmonic components. Experiments show that FoPE maintains a better performance for increased context lengths.

This paper provides a valuable contribution which analyzes weaknesses of current models and uses insights from discrete signal processing to propose a solution. Reviewers point out as weaknesses concerns about the signal-based explanation and the computational cost (both of which were clarified in the rebuttal), as well as somewhat inconclusive experiments and results, which does not completely demonstrate the value of FoPE versus RoPE or other embedding methods. The authors addresses most of the concerns. I am leaning towards acceptance.
Decision: Accept (poster)