Wavelet-based Positional Representation for Long Context
Authors: Yui Oka
Taku Hasegawa
Kyosuke Nishida
Kuniko Saito
Venue: ICLR 2025 Poster (2025)
Keywords: Positional Encoding
Extrapolation
Wavelet Transform
Transformers
RoPE
ALiBi
NLP
Primary area: foundation or frontier models, including LLMs
TL;DR: We found that RoPE can be interpreted as a restricted wavelet transform. And we propose a new position representation method that captures window sizes by leveraging wavelet transforms without limiting the model's attention field.

Abstract: In the realm of large-scale language models, a significant challenge arises when extrapolating sequences beyond the maximum allowable length. 
This is because the model's position embedding mechanisms are limited to positions encountered during training, thus preventing effective representation of positions in longer sequences.
We analyzed conventional position encoding methods for long contexts and found the following characteristics.
(1) When the representation dimension is regarded as the time axis, Rotary Position Embedding (RoPE) can be interpreted as a restricted wavelet transform using Haar-like wavelets. 
However, because it uses only a fixed scale parameter, it does not fully exploit the advantages of wavelet transforms, which capture the fine movements of non-stationary signals using multiple scales (window sizes). 
This limitation could explain why RoPE performs poorly in extrapolation.
(2)
Previous research as well as our own analysis indicates that Attention with Linear Biases (ALiBi) functions similarly to windowed attention, using windows of varying sizes.
However, it has limitations in capturing deep dependencies because it restricts the receptive field of the model.
From these insights, we propose a new position representation method that captures multiple scales (i.e., window sizes) by leveraging wavelet transforms without limiting the model's attention field.
Experimental results show that this new method improves the performance of the model in both short and long contexts. 
In particular, our method allows extrapolation of position information without limiting the model's attention field.

Published as a conference paper at ICLR 2025
WAVELET-BASED POSITIONAL REPRESENTATION FOR LONG CONTEXT
Yui Oka, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito NTT Human Informatics Laboratorics, NTT Corporation yui. wui.oka@ntt.com
ABSTRACT
In the realm of large-scale language models, a significant challenge arises when extrapolating sequences beyond the maximum allowable length. This because the model's position embedding mechanisms are limited to positions encountered during training, thus preventing effective representation of positions in longer se- quences. We analyzed conventional position encoding methods for long contexts and found the following characteristics. (1) When the representation dimension regarded the time axis, Rotary Position Embedding (RoPE) can be interpreted as a restricted wavelet transform using Haar-like wavelets. However, because it uses only fixed scale parameter, it does not fully exploit the advantages of wavelet transforms, which capture the fine movements of non-stationary signals using multiple scales (window sizes). This limitation could explain why RoPE performs poorly in extrapolation. (2) Previous research as well as our own anal- ysis indicates that Attention with Linear Biases (ALiBi) functions similarly windowed attention, using windows of varying sizes. However, it has limitations capturing deep dependencies because restricts the receptive field of the model. From these insights, we propose a new position representation method that cap- tures multiple scales (i.e., window sizes) by leveraging wavelet transforms with- out limiting the model's attention field. Experimental results show that this new method improves the performance of the model both short and long contexts. particular, our method allows extrapolation of position information without limit- ing the model's 's attention field.
INTRODUCTION
Several pre-trained large language models based on Transformer architecture /Vaswani et al. 2017) have demonstrated robust capabilities in various generative tasks (Devlin et al., [019)] Raffel et al. 2020 Brown et al., 2020 TTovron et al., 2023a Jiang et al. 2023). However, limitations on the input sequence length arise due to the computational resource constraints encountered during the pre-training phase. Such constraints necessitate a determination of the maximum allowable length of sequences, hereinafter Tarains prior the pre-training process, thus hindering the model's per- formance in processing sequences longer than those encountered during training. This weakness is primarily attributed to the positional encoding's ineffectiveness in handling sequences that exceed
the length of those encountered during the model's training phase (Devlin et al. 2019 Press et al. 2022).
Rotary Position Embedding (RoPE) (Su et al. 2021 has become a common approach in many language models that handle long contexts, and it employs rotation matrix to encode positional information and facilitate the processing of long sequences. To manage sequences longer than those encountered during training, various scaling strategies (Chen et al., 2023; bloc97 2023) Peng al.) 2024 Liu et al.) 2024) have been applied to RoPE, although these often require additional fine- tuning and incur further learning costs addition to those of pre-training. with Linear Biases (ALiBi) (Press et al. 2022) is able to sequence length estimation beyond the limits of pre-training without requiring additional fine-tuning. However, ALiBi limits the attention's receptive field (Chi et al. 2023) in the manner of windowed attention (Beltagy al. 2020). For this reason, a model using ALiBi may not be able to obtain information that is in a distant dependency relationship. In this paper, we analyze conventional positional encoding methods for long contexts,
contrast, Attention

---

Published as a conference paper at ICLR 2025
We
90
Pe.o Pa. Pru Pr.
Pays
Px.L
In axd hob
Shift
andlyodd q1
In
1.
1-1
gm
1nd-1
Scale
Wavelet Vi
Query
Relative Position Representation
Overview of Wavelet-based Relative Positional Representation As in RPE (Shaw al. 2018), our method computes a relative positional representation (Pm,n)) to the query 9m and the key kn Instead of learnable embedding RPE, the position computed based on the wavelet function. Different wavelet functions Va,b are used for each dimension of the head d. Furthermore, the scale parameter a and the shift parameter b change depending on the dimension of the head
Figure 1:
and we propose a novel positional representation that permits extrapolation without constraining the attention mechanism's 's receptive field. First, mathematically show that RoPE performs pro- cess similar to a wavelet transformation- considered the gold standard of time-frequent freeereeeuucc analysis methodology. We interpreted the position of each token in the sequence frequency analysis. However, RoPE does not perform a transformation in accordance with the order of positions but rather in accordance with the number of dimensions, and it does not capture the dynamic change in signal over time. Furthermore, the values corresponding to the wavelet scale (i.e., window size) are constant, so RoPE does not make good use of the key characteristic of wavelet transforms, which is the ability to analyze signals on multiple scales. In other words, RoPE may fail to capture the dynamic change signal over time, such as what occurs in natural language. this study, we also show that ALiBi provides different window sizes for each head.
time point in time-
Based on these insights, we propose a wavelet transform- -baee method, using multiple window sizes, offer robust and flexible approach to positional encoding. By performing a wavelet trans- form along the order of positions and introducing various scale parameters, our method can capture the dynamic changes in a sequence over positions in the manner of the original feature of wavelet transformation, i.c., time-ffree teee-rrreueccy analysis. Following the methodology of Relative Position Rep- resentation (RPH) (Shaw al. 2018), we implement method with relative ease.
From our experiments on extrapolation capabilities using the wikitext- 103 dataset (Merity et al, 2017). the results demonstrate that our method surpasses traditional positional encoding methods in perplexity. We also report that our method has lower perplexity than RoPE in experiments with long contexts using the Llama-2 model (Touvron et al. 2023b) and the CodeParrot dataset.
2 BACKGROUND
2.1 POSITIONAL REPRESENTATION
Within the Transformer architecture, positional encoding is employed to accurately represent the sequential position of each token. Positional encoding can be divided into main types: absolute position, which expresses the position of token from the static beginning of the sequence, and relative position, which expresses the position of each token in relation to the other tokens within the sequence. RoPE (Su et al. 2022)) which adopts type of absolute position, uses a rotation matrix to compute the position and then multiplies it by the query and key to represent the position. RPE (Shaw et al. 2018), based on type of relative position, uses learnable embedding that represents the position of distances of up to 16 or 32 tokens by clipping. Two other variations include T5 Bias (Raffel et al. 2020), which has enlarged RPE window size, and Transformer-Xi (Dai al., 2019) which uses a sine wave for position representation instead of learnable embedding.
Position encoding plays a critical role in enabling models to effectively handle long context quences, and it allows for extrapolation. Relative position is not a position expression that depends on the length of the sequence, so it is effective in extrapolation. ALiBi (Press et al. 2022 is an
2

---

Published as a conference paper at ICLR 2025
effective position representation method for extrapolation: It uses the relative position bias of all tokens by adding a linear bias to each head's attention score, rather than using position embed- ding. However, ALiBi is unable to obtain information in distant dependency relationship due to its constraints on the self-attention mechanism's receptive field(Chi et al. 2023). On the other hand, absolute position is unsuitable for extrapolation because it expresses the position of all words in the sequence. For this reason, many methods have been proposed for fine-tuning RoPE by interpolating positions in using absolute position (Chen al. 2023 bloc97 2023
Peng al. 2024).
2.2 FREQUENCY ANALYSIS AND TIME-FREQUENCY ANALYSIS
Frequency analysis in signal processing involves analyzing the frequency components of a signal to understand its behavior. The Fourier transform (FT) (Bracewell & Bracewell 1986) is a key method for frequency analysis, converting signal from the time domain to the frequency domain, thus providing global view of its frequency content. However, the FT does not provide any infor- mation about when specific frequencies occur. To address this limitation, time-frequened analysis techniques have been applied. The wavelet transform (WT) (Grossmann & Morlet 1984, Mallat 1989) offers more flexible approach by analyzing the signal at multiple scales or resolutions. The WT adaptively provides high time resolution for high-frequency components and high frequency resolution for low- now-frequency components, making it well-suited for analyzing signals with non- stationary or transient features. This adaptability allows the wavelet transform to capture both time and frequency information with varying degrees of precision.
3
ROPE AND WAVELET TRANSFORM
3.1 PRELIMINARY
Wavelet Transform A wavelet is wave that decays quickly and locally it approaches zero. A function & defined on a real IR is called a wavelet function if it belongs to the space (R) of square integrable functions and satisfies the following conditions:
(x) 1 dx < co.
The wavelet function defined as follows.
1
b
(1,
In this case, b is the shift and > 0 is the scale parameter. The scale parameter a simultancously changes the range over which the wavelet is localized as well the wavelet's amplitude. Typical wavelets include the Haar wavelet (Haar) 1910), Ricker wavelet (Ricker) 1944), and Morlet wavelet (Bernardino & Santos- Victor 2005). Suppose that we sample T values regular intervals from continuous signal. Wavelet transform (WT) (Grossmaan & Morlet, 1984) is the process of trans- forming signal xx()) into the frequency domain and time domain by computing the inner product of the wavelet function (t) and signal z(t).
ba, = Σ ba (t))(t)((t). t=0
In some cases, the term "Discrete Wavelet Transform" or "Wavelet Transform" is used to refer to multi-resolution analysis (Mallat 1989), but in this paper follow the original definition. We can see that the FT only converts to the frequency domain, whereas the WT converts to two domains: scale and shift b. For example, consider the case of conversion
scales and four shifts. When a E [2, 4) and & E [0, 1, 2, 3). the wavelet transform can be expressed in terms of determinants as follows:
W2 0)7
v2)
22) (1)
42 1)
(x(0)
(4
(0)
(1)
(2)
V4,o 1)
((1)
122
(0)
(1)
(2)
PT 1)
xx()
144,1)
(0)
(1)
(2)
(T -1)
(4)
[ 1)
3)4
(0)
(1)
(l.s(2)
VT
3

---

Published as a conference paper at ICLR 2025
Furthermore, since (ta,(tt) 2aa.((t b) from Eq 2 v of the wavelet transform in Eq as follows.
expressed
W(2.0)
(2)
(2.0
22.
22 - 1)
=(0)
((4.0)
(0)
(1)
(2)
V.olT
x(1)
1)2,1)
122,0(-1
(0)
(2)
x=(2)
1(4.1)
49,.0
(0))(0)
(1)
4.,0
(5)
:
Lo(T
34)
(-3,
(
(4,0(-1)
0..0 3)
Due to the characteristics of the scale parameter a, the values of the wavelet matrix become 0 or approach 0 outside certain range that depends on the specific wavelet function.
RoPE RoPE incorporates positional information directly into the self-attention mechanism by ro- tating the query and key vectors in complex space. When divided into even and odd dimensions, the following calculations are performed for the m-th query in each sequence. In even dimensions, RoPE is expressed as follows.
40
cos m0
sin ml
0
qu
Q2
0
cos m02
sin m02
9i
9d
0
cos m00a2
-in sinm0aay2
ld-2
where 9thh E R is the m-th query when the number of dimensions is d and 0, 10000-2(1-1)44 E [1,2...., d/2). For RoPE in odd dimensions, see Appendix A.1 The same process is also performed for the n-th key knd Rxxd
3.2 THEORETICAL ANALYSIS
First, we show the wavelet transform using the following two Haar-like wavelets Haar 1910).
cos f(t)
0
sin f() 0 0st,,
(t))
sin f(t)
1 1<t
& (t)
cos f(t) 1 1
0
otherwise.
0
otherwise.
Here, ʃ + R is a function that satisfies - (b( di = 0 and Eq. Assuming that when
a0t)(0 < t < d 1) is signal with d elements, the wavelet v is used and wavelet transform is performed at each scale a We define the shift parameter as by j j- -j-6(j)
0, 2., d 2). Here, 5(t) is function such that 0 < / S d 1 and 0 < 8(1) < When the wavelet function is Haar-like wavelet (1) Eq. ( and = and
[by, b2 .., da-2], the wavelet matrix &b in the wavelet transform the can be expressed in terms of determinants as follows.
xx0)
(1,, bo) (1,,b2)
cos Φo
sin Φ1
0
0
s<1)
cas d2
sin Ø3
0
0
x(d 2) Lx(d 1)
(11, ba-))
0
COS Ød-2
sin Pa-1
(8)
To simplify the notation in the matrix representation above, we write Φj for j = 0, d 1, where Φ; f(1 + 8(j)) j is odd, and Φ; = f(8(j)) otherwise. Let r be the query gdh and define such that o, = dj+1 = m8 22227 for j = 0. d 2, where 8; 10000 -000002(i--)) 1// and i E [1.2, 2. ... d/.. Under this definition, the transformation matrix of Eq. becomes identical to that of Eq. 6 in RoPF In other words, RoPE can be viewed wavelet transform using Haar- like wavelets that change amplitude on fixed scale. Furthermore, the same result RoPE odd
The proof of the existence of f(t) that satisfies this condition is provided in Appendix A.2
4

---

Published as a conference paper at ICLR 2025
slope= slope= 1/8
slope=1/32 slope=1/122
Figure 2: Heatmap of scaled attention scores via softmax normalization in ALiBi without non- overlapping inference. The vertical axis represents the query, while the horizontal axis corresponds to the key in the attention map. For clarity, values of 0.001 or more are mapped to black, while values below that mapped to yellow. The maximum allowable length of sequences Lirain 512, and the inference length is 1012.
dimensions can be obtained when using v6 for wavelet transformation. This wavelet transform in RoPE is performed across the number of query head dimensions d. Therefore, RoPE can be considered a wavelet transformation along the head dimension using a wavelet with fixed scale of 2
WINDOW SIZE VARIABILITY IN ALIBI
ALiBi has restricted receptive field and behaves in the manner of windowed attention (Chi al. Beltagy et al.] 2020). A receptive field refers to the specific region of the input space that significantly influences the model's output, typically representing the area where the most relevant features captured. ALiBi is expressed as
2023
softmax (g. KTT slope (- ...,. -2,-
where the slope is a head-specific slope fixed before training and KT E gm is the first keys. In this section, we analyzed the window size in ALiBi using the attention map.
4.1 INSIGHTS FROM ATTENTION MAP ANALYSIS
heatmap of scaled attention scores obtained through softmax normalization is shown in Figure 1 The number of heads N is 8, and the slope of ALiBi is 7 8 To+ 32 87 T2g 256 In extrapola- tion, sequences are often divided, but this section the sequences are not divided. The experimental setting was set to the same as that in Section 6. The perplexity results are shown in Table !
1
The attention map shows that ALiBi uses multiple window sizes corresponding to relative positions and that the window size increases the slope decreases. Morcover, previous research (Chi et al. 2023) shows that constraining the window size (slope) to a single value leads to increased perplexity. Consequently, one of the reasons ALiBi is effective, compared fixed window sizes in T5 Bias (Raffel et al., 2020), is its ability to accommodate multiple window sizes. ALiBi does perform calculations like those in Eq. (3), so does not exactly match the wavelet transform. However, having windows of various sizes is similar to the role of the scale parameter used in wavelet transforms.
previous relative position using
WAVELET-BASED POSITIONAL REPRESENTATION
Wavelet transform (WT) is method of analyzing signals using variable-scale wavelets, and it is possible to adjust the scale of the window. This scalability allows both broad and fine signal features be efficiently extracted by shifting the wavelet while changing the window size. In partieular, this is suitable for investigating non-stationary signals. For this reason, we believe that the wavelet
Additionnlly, when sin m0, 2, the shift 4,. d/2).
m0,, the Haar wavelet matrix and RoPE the same when the scale Refer to Appendix A.3 for the detailed proof.
From previous research Tancik et al. 22020), we also hypothesized that this could be equivalent to Fourier transform. However, this hypothesis does not hold (refer Appendix A.4 details).
5

---

Published as a conference paper at ICLR 2025
transform approach is effective for capturing the dynamic fluctuations of signals that change time, and it is also effective for the fluid nature of natural language, which is not constrained by periodicity. Furthermore, when extrapolating, is important to be able to respond flexibly to changes in context and information. For this reason, we believe that the wavelet transform is also an effective method for extrapolation.
When applying wavelet transforms to positional encoding, a key question arises: Which features should be leveraged for handling long-context dependencies? Notably, RoPE shares conceptual similarities with the wavelet transform (Section :): however, RoPE depends on absolute positional information, which limits its effective context window to the training length (Luraiu) and restricts its extrapolation capabilities. In contrast, ALiBi offers extrapolation capabilities by using relative position, and it supports varying window sizes (Section { However, ALiBi's linear bias constrains its receptive field, making it insufficient for capturing long-range dependencies. According to Press al. (2022), conventional relative positional encoding (RPE) methods (Shaw al., [0188] Raf- fel et al. 2020), which rely on fixed window size, are similarly ineffective for extrapolation. In conclusion, we adopt relative position with flexible window sizes to handle long-context and extrap- olation.
Accordingly, we propose positional representation based on wavelet transform with the following characteristics:
1. Position-based Transformation: RoPE predominantly relies on independent transforma- tion based on the 'head' dimensions. ALiBi employs multiple windows based on the rela- tive position of the sentence, rather than the dimension of the head, which may contribute to its performance. Therefore, we apply a wavelet transform based on the relative position of the sentence.
2. Type of Wavelet: RoPE can be thought of as a wavelet transform using the Haar wavelet, which is the simplest wavelet. However, Haar wavelets might fall short in capturing the intricacies of natural languages. Transitioning toward the use of more sophisticated wavelet functions could enhance our approach to distilling and representing a broader spectrum of features inherent in natural languages.
3. Diversification of Window Sizes (Scale Parameters): From our amalysis of ALiBi, we found that having multiple windows is effective for long contexts. The original version of RoPE works with single fixed scale. To address this limitation, we introduce variety of scale and shift parameters.
5.1 METHODOLOGY
Incorporating Wavelet Transform into PE Due to the wavelet shift feature, we adopt relative position representation using ALiBi because is more suitable than absolute position representation. In a transformer model (Vaswani al. 2017), the self-attention mechanism operates by projecting the input sequence into three distinct representations queries (Q), keys (K), and values - learnable weight matrices. Self-attention sublayers employ N attention heads. In self-attention sublayers, em, is the attention score for each query, and then the key is calculated. RPEeSSha (et al. 2018) expresses position by calculating the inner product of the query and the relative position embedding. We incorporate the wavelet function into RPE as follows.
qm kT + 9m (Pmm,))
Em.n
Vd.
(10)
where 4m is the mith query (fm Bixd S m S L) of a sentence of length L, kn is the rith key
(kn IR 1xd 1 S S L) for 9m. and d is the number of dimensions of each head. Here, Pm, is
the relative position from the m-th query to the n-th key. RPE (Shaw et al., 2018) uses learnable embedding for Pm,, Rdd and a fixed scale by clipping. However, instead of using learnabee embeddings to represent Pm,n. we use d-pattern wavelet functions with multiple scales to calculate the position. In our method, there no clipping, and the distance of the position expression is fixed regardless of the length of the sentence.
'We also considered incorporating wavelet transforms into RoPF, but decided not to do this hecause it would make the computational cost even higher. A discussion this included in Appendix A.5

---

Published as a conference paper at ICLR 2025
Wavelet Function In conventional wavelets, such as in Eq. 2 the amplitude also varies depend- ing on the scale parameter a, In the proposed method, all amplitudes are the same.
(1) - -b (²(
(11)
The variable is assigned the relative position, which is m n. We used the Ricker wavelet Ricker 1944) as a base wavelet, which is formulated as follows.
(() = (1 (1- (2) exp
(12)
2
Shift and scale parameters for the shift parameter 6.
We use distinct patterns for the scale parameter and å patterns
(a, b) E 22",2), x 1,,1,2,3.....
1}.
(13)
The scale parameter is power of derived from the principles of the discrete wavelet transform. By combining the e --pattern shift parameters b with the s-pattern scale parameters we generate distinct wavelets. In this way, our method can set the s-pattern context window size using the scale parameter and the d-pattern context window using both the scale parameter and the shift parameter b. For instance, with a head dimension of = 128, we use s = 8 scale variants (a E {20, 2 ....27) and 16 shift variants (b E {0. 1, 2, .... 15)), resulting in 8 x 16 128 unique wavelets. Finally, Pm,, is computed as follows
Pm.. (I
m n by 2 exp -
m n by 3
(14)
a
SHORT-CONTEXT EXPERIMENT
6.1 EXPERIMENTAL. SETTINGS
First, we conducted a small-scale experiment to compare our approach with various position encod- ings. We used the WikiText-I01 dataset (Merity et al. 2017). which consists of over 103 million tokens of English Wikipedia articles. We performed a comparative evaluation using a Transformer- based language model (Baevski & Auli 2019). The dimensionality of the word embedding dmodel is 1024, the number of heads N is 8, the dimensionality of the heads d is 128, and the number of layers is 16. The implementation was based on the fairseq (Ott et al., 2019)-based code pro- vided in previous work(Press et al.] (022), and all hyperparameters were set the same values as those in the literature (Press et al. 2022 The maximum allowable lengths of sequences were set to Ltrain 512 and Lirain 1024.
Compared Methods Although 0 = 10, 000 is usually used for RoPE, it has been found that ex- tending 0 to 500,000 is effective for long contexts (Xiong et al., 2024). Therefore, we compared 0 = 10, 000 with 0 = 500, 000. In addition to ALiBi and RoPE, the following position representa- tions were also compared: NoPE (Kazemnejad et al. (023), in which position information is given, and TransXI. (Dai et al. 2019), which is a relative positional representation that uses sine waves.
We use perplexity as our evaluation metric. Following previous research (Press et al. 2022), evaluated the validation set. To evaluate sequences longer than Ltrain tokens, it is common to divide the sequence into -lrain- sub-sequences, evaluate each independently, and report the average score. However, methods that relative positions express wide range, such as ALiBi, Trans-XI, and the proposed method, are able to consider a wider range of contexts than Lersin. For this reason, in this paper, we report not only the perplexity of non-overlapping ference but also the normal perplexity when the sequence is not divided into partial sequences. Note
Evaluation Metric
'Implementation tips for reducing the memory and computational efficiency of the proposed method included in Appendix A.6
ftt / / thub. cwm/co/o//////o of wwressssssee.. ptpoess/aatent ion_with linar __aaees See Appendix A.7 for more details of hyperparameters.
7

---

Published as a conference paper at ICLR 2025
Table 1: Perplexity of validation set in extrapolation experiments using Wikitext-103. Maximum
allowable lengths of sequences in pre-training are Lerain = 512 and Ltaain 1024.
Sequence Length
Tirain
Tirniu
1024
pos
128
256
512
1012
1512
2512
1024
1524
3024
5024
Perplexity in Non-overlapping Inference with Lerain
NoPE Kavemniide all
0023
26.38
23.23
21.53
21.52
21.53
21.53
20.81
21.52
21.49
21.45
RoPE al 201
abs
23.82
20.98
19.39
19.35
19.39
19.38
18.42
19.51
19.52
19.48
RoPE
M024
23.81
20.95
19.35
19.32
19.35
19.33
18.50
19.53
19.54
19.50
Trans XT. (Du
2019
rel
24.16
21.53
19.96
19.92
19.93
19.96
18.67
19.75
19.74
19.70
at
24.18
21.32
19.69
19.64
19.64
18.66
19.64
19.65
19.62
Wavelei[Rioka
rel.
23.64
20.82
19.19
15.15
T9.17
19.20
18.26
19.30
19.59
19,26
Perplexity without Non-overlapping Inference
NoPE Kazemociadet
RnPF Sa al
2011
abss
23.82
20.98
19.39
23.25
44.38
93.94
18.42
18.29
33.20
122.52
RoPE
Xoog a al.
3024
abs
23.81
20.95
19.35
23.70
40.39
77.90
18.50
18.30
29.25
83.45
Trans-XL XL
2019
24.16
21.53
19.96
19.09
18.92
19.05
18.67
18.25
18.17
18.76
ALiBi
Press et
rel 24.18
21.32
19.69
18.71
18.42
18.41
18.66
18.14
17.86
17,88
Wavele
18.23
Haar (Fixed scale)
rel 24.98
22.07
20.49
51.61
1 166.87
299.26
rel 23.73
20.89
19.27
18.34
18.11
18.17
Moslet
24.15
21.28
19.65
19.02
20.46
26.56
Gaussian
rel 23.77
20.90
19.30
18.31
18.02
17.88
that when the sequence length is less than Ltraiu- the scores for the perplexity of non-overlapping inference and the normal perplexity without division into partial sequences are the same. Of course, when perplexity is considered without division into partial sequences, the performance of RoPE is expected to decrease greatly because unknown values are used for RoPE when processing a sequence longer than the length encountered during training.
6.2
MAIN RESULTS
The experimental results shown in Table The results of perplexity in inference without overlap show that the proposed method using wavelets achieved the lowest perplexity and was also effective for extrapolation. In RoPE, the values used during training are also used in inference without over- lap, so the perplexity remains low even when the sequence length exceeds Ltrain At the same time, however, perplexity higher for ALiBi and Trans- than for RoPE, which is attributed the lim- ited context range of the position representation' 's applicability due to the division of the sequence into sub-sequences. contrast, the proposed method maintains low perplexity even in the case of division into sub-sequences, suggesting that the wavelet position representation is highly effective.
On the other hand, perplexity without non-overlapping inference showed the opposite results. First, since RoPE uses absolute positions, is necessary to perplexity increased significantly. However, the case = 500, 000, the increase in perplexity was relatively small. On the contrary, Trans-XL and ALiBi, which use relative positions, able to handle longer contexts, and perplexity decreased as the range of position representations expanded. In the proposed method, perplexity also decreased and the best score achieved. Trans-XL uses position representation based on a periodic sine wave function, but the proposed method, which uses wavelets, could further decrease perplexity. This result supports our claim (section[5) that an approach like wavelet transformation is more effective than periodic functions in capturing the fluid nature of natural language, which is not constrained by periodicity.
values for unknown positions, and thus
6.3 ANALYSIS
6.3.1
HOW EFFECTIVE ARE THE OTHER WAVELET TYPES?
We also conducted experiments to see whether the same effect could be obtained with other wavelets. The wavelets tested were the Gaussian-based wavelet (b() czpp(-?),, the Morlet- based wavelety((t) = ezp(- ccop(-) and the Haar-bace based wavelet. Note that when (tt/a) exists in our Morlet wavelet, the frequency of this cosine wavelet is not affected by the scale parameter a.
8

---

Published as a conference paper at ICLR 2025
AUBII
RoPE
Wavelet
Figure 3: Heatmap of scaled attention scores via softmax normalization in 4th head after softmax operation without non-overlapping inference. The vertical axis represents the query, while the hor- izontal axis corresponds to the key. For clarity, values of 0.001 or more are mapped to black, while values below that are mapped to yellow. The maximum allowable length of sequences in pre-training is Lerain 512 and the inference length is 1012. See Appendix A.12 for other heads.
We used the following formula for the Haar wavelet.
1
-0.5 S t<0,
(1()=
- ss 0.5,
(15)
0
otherwise.
We kept the shift and scale parameters constant, only changing the wavelet function. We also tested the Haar wavelet when set E {2 20; 2220 ........ Consequently, this restricted Haar wavelet had the same scale parameter setting as the RoPE demonstrated in Section 3.2 [l
The graphs of these wavelet functions are shown in Appendix A.10 (Fig. (). Extrapolation experiments were conducted
under the same conditions as the experimental setup in Section 6
with Lurain 512 during training.
As shown in Table 1 the Ricker-, Haar- and Gaussian-based wavelets had lower perplexity than the Morlet wavelet. One possibility is that complex wavelets with multiplied cosine waves, such as Morlet wavelets, are not suitable for relative positional representation. On the other hand, wavelets with all positive values, such Gaussian-based wavelets, are expected represent positions within a narrower distance than the window specified by the scale parameter due to softmax normaliza- tion. This suggests that wavelets with a specific range of negative values are suitable, like Ricker wavelet, for positional representation. Although the Haar wavelet is simple, it is such wavelet with negative values within a specific range. Therefore, it is considered effective, although not as much as Ricker wavelet. However, when the scale parameter is restricted ( (a E (2"...., the perplexity increases. This demonstrates the importance of having multiple scales, or in this case, window sizes. We also performed ablation studies for each shift and scale parameter (Appendix A.13) and for discrete wavelets as well as continuous wavelets (Appendix A.14).
..... 20)). as in RoPE,
6.3.2 CAN IT HANDLE TOKENS WITH LONG-RANGE DEPENDENCIES?
Figure shows the attention map of scaled attention scores obtained through softmax normalization for the proposed method. The inference length is L = 1012 without non-overlgpping inference. The most notable feature of the proposed method is that is always able to attend to specific tokens. The words that always receive attention are those that are important in the sentence, such the special token, the first token, and the subject of the sequence. On the other hand, ALiBi has a restricted receptive field for attention, making it unable to capture long-dis dintteist dependencies. Similar to the proposed method, RoPE emphasizes important and special words but struggles to capture those that farther apart. Moreover, as the sentence lengthens, it loses the ability attend to the initial word. This tendency was also seen in sentences shorter than Turain- Accordingly, the proposed method has demonstrated its superiority at capturing long dependencies without restricting the receptive field of attention.
'Nommally the wave is localized when > 0 in the Haar wavelet, but in the decoder model, only the range f < 0 is used. Therefore, we transformed the Haar wavelet into a form that reflects the original function f(x) across the y-axis.

---

Published as a conference paper at ICLR 2025
Table 2: Perplexity in Non-overlapping Inference with Luaain 4096.
Sequence
Length
k
8
16
32
ReFE: Kiong Wavelet
2024
9.45
9.33
9.12
8.90
9.00
9.01
8.83
8.60
7 LONG CONTEXT
7.1 EXPERIMENTAL SETTINGS
Next, conducted large-scale experiment using a Llama- model (Touvron al. 2023b We pre-trained the Llama-2-7B model from scratch. For pre-training, we used the RedPajama dataset (Computer) 2023), which selects 1B-token sample of all samples. The maximum allowable length of sequences in pre-training was set to Liaii = 4096. For the same reason as given in Section 6.1 set 0 500, 000 for RoPE. Furthermore, when the scale parameter is a E {2", 2 ..... ..., the range within which the wavelet is localized becomes narrow. Therefore, in our method, we changed the scale parameter (ouvvon et al. 2023b). We used CodeParrot [ for evaluation, which is good for long-distanee testing because requires an understanding of patterns and contextualization of information over long distances. [T
22. ..... 2 The other parameters are the same as those used
for the Llama-2-7B
7.2
MAIN RESULTS
The experimental results are shown in Table Regardless of whether interpolation or extrapolation was applied, the perplexity of method was lower than RoPE. Therefore, even with large-scale models and long contexts, our method was found to be effective. Moreover, the results Section 6.2 show that not dividing the sequence further reduces perplexity. Therefore, our method might also be able to further reduce perplexity. We investigated the use of LongBench(Bai et al. 2024). with the results given in Appendix AA.
In addition, position interpolation methods (Chen et al. 2023 bloc97 2023 Peng et al. 2024 Ding al. (024) have been proposed adapt RoPE for longer contexts. We believe these methods can be integrated into our approach for the following reasons. First, the parameter 0 in RoPE corresponds our method, implying compatibility between the two frameworks. Both 0 and refer to the upper limit of the number of positions be expressed. Second, the LongRoPE paper (Ding et al.) 2024) reveals that performance improves when extrapolation avoided for the initial positions, which likely aligns with the shift parameter b in our method. Thus, it is highly likely that existing position interpolation methods will integrate seamlessly with our approach.
to the scale parameter
CONCLUSION
In this paper, we demonstrated that RoPE can be interpreted as a wavelet transform, and we intro- duced a novel positional representation method that leverages the wavelet transform's advantages, effectively capturing positional information across various window sizes. Our experimental results demonstrate the proposed method's superior performance in extrapolation tasks when compared to traditional positional representation techniques. Importantly, our approach offers the advantage of not constraining the receptive field, which allows more flexible and comprehensive analysis of po- sitions. Calculating relative positions is known require more resources than calculating absolute positions, so we show methods for reducing memory consumption in Appendix AA However, the computational overhead of calculating relative positions may still impose bottleneck, and thus reducing is an important direction for future work.
ww..ccccooo....oooooo. ng we..fac..... co/ /tt///atassstsssss http//aaasstt//oeeerroooo httpppppo.oooooooooooooooo......
See Appendix A.8 for more details of the hyperparameters.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In International Conference on Learning Representations, 2019. URL httLhttttss//peeeeeeet
Gertmo ietorummiddssee ddmmmnddddx2220000 QqFQ
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilin- gual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 3119-33 1139,1 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 00.18653/v1/0024 172. URL https://aclaant antho oogy. wrw.oorg//224/. waw.:022.ccllll. -on..o.g..................
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. 05X55 2020.
Alexandre Bernardino and José Santos-Victor. A real-time gabor primal sketch for visual attention. In Jorge S. Marques, Nicolás Pérez de la Blanca, and Pedro Pina (eds.), Pattern Recognition and Image Analysis, pp. 335-342, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg. ISBN
978-3-540-32237-5.
bloc97. Ntk-aware scaled rope allows Ilama models to have extended (8k+) con-
text size without any fine-tuning and minimal perplexity degradation., 2023. URL
https: http//www.... reddit. http::::/wwww.eddt..cco /ow.o//////aaLLMMa htwpaaaaaaaaaacott////////////// scal ledd_rop_____ lows_ 11 1amaa aama_modell sttt__tt_____________ hawee//
Ronald Newbold Bracewell and Ronald Bracewell. The Fourier transform and its applications, volume 31999. McGraw-Hill New York, 1986.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Vosse Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL pttpp:://procceedngg.. www.::/sw.......i............... http:://ccccaag....cc.ccccccoepp fi E5le//457ccddofdd444711188888888 htt.:eeeffeb444499988888088
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023. URL https://arxiv.cr
abs/ / 2306 15595
Ta- Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, and Peter Ramadge. Dissecting transformer length extrapolation via the lens of receptive field analysis. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 13522- 13537, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 18653/ vl/2023.acl-long.777 URL https: httppsss//aaaaaatolloo..ooo org/2023..3.2233 wttp:://222....1111oo..... acl-lonllg..o. 756
Together Computer. Redpajama: An open source recipe to reproduce llama training dataset, 2023.
URL http:://g..hhbbooooooooogoeeeerr www.aanwwwtteeeeeeaeaaeeeeeeeeee -amaaeaaaaaaaaaaaaaaaaaaaaa
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.
Transformer-XL: Attentive language models beyond a fixed-length context. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedingy of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 2978-2988 Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10. 18653v/ 1/P19- 1285. URL http:://ccaattolllss...
lorg/P19-122
Ingrid Daubechies. Ten lectures wavelets. Society for industrial and applied mathematics, 1992.
11

---

Published as a conference paper at ICLR 2025
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-418 Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10. 10.18653/11/119-44222
URL
N19 1442
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending IIm context window beyond 2 million tokens, 2024.
URL httph:tt http://axxvvvvooooo org/ 2 13753
W. M. Gentleman and G. Sande. Fast fourier transforms: for fun and profit. In Proceedings of the November 7-10, 1966, Fall Joint Computer Conference, AFIPS '66 (Fall), pp. 563-578, New York, NY. USA, 1966. Association for Computing Machinery. ISBN 9781450378932.
doi: 10.
1145/1464291, 1444444422
wwt.oo/g//o0.. 111.:0.1111111411 11t.:/44442911114455...5 1464352
A. Grossmann and J. Morlet. Decomposition of hardy functions into square integrable wavelets of constant shape. SIAM Journal Mathematical Analysis, 15(4):723-736, 1984. doi: 10. 1137/ 0515056.
A. Haar. Zur theorie der orthogonalen funktioneensysteme. (erste mitteilung). Mathematische An- nalen, 69:331-371, 1910.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Seao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
06810
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization transformers. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. 24892-24928. Curran Associates, Inc., 2023. URL htttps://proceeddngggg/o// www.::is.nnnnuuu.oo.oo cc htt.://errr...ccppppppeeeeeee/aa pssppp/pppssssse//ppe///23333 File/
Gregory R. Lee, Ralf Gommers, Filip Waselewski, Kai Wohlfahrt, and Aaron 08217:Leary. Py- wavelets: A python package for wavelet analysis. Journal of Open Source Software, 4(36):1237, 2019. doi: (0211005/oos.002233
URL. fttttttt:::/aad ottp:/0orrg11.. 21101...111/11111 01237
Yang Li, Si Si, Gang Li, Cho-Jui Hsich, and Samy Bengio. Learnable fourier features for multi- dimensional spatial positional encoding. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:// openreview
Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of roPe-based extrapolation. In The Twelfth International Conference on Learning Representations, 2024. URL
https::/openrevian net wttp//oorummooooooe 1o-rummid--000000355
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL UBLL::pps://openeeeeee.eee net.:/iwwwwweeeeeeeeeeeeeeeeeeee fortt//frmm?aa
Bkg6RiCqY7
S.G. Mallat, A theory for multiresolution signal decomposition: the wavelet representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(7):674-693, 1989. doi: 10. 109/ 34.192463.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mix- ture models. In International Conference on Learning Representations, 2017. URL https:
//openrevi ew. dttprrmmmmddByyyyyuyy
12

---

Published as a conference paper at ICLR 2025
Nhat Khang Ngo, Truong Son Hy, and Risi Kondor. Multiresolution graph transformers and wavelet positional encoding for learning long-range and hierarchical structures. The Journal of Chemical Physics. 159(3):034109, 07 2023a. ISSN 0021-9606. doi: 10.1063/5.0152833
URL https:
//doi.org//1 1063/5..... 0158333
Nhat Khang Ngo, Truong Son Hy, and Risi Kondor. Multiresolution graph transformers and wavelet positional encoding for learning hierarchical structures. arXiv preprini arXiv:2302.086477 2023b.
Myle Ott, Sergey Edunov, Alexci Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context win- dow extension of large language models. In The Twelfih International Conference on Learning Representations, 2024. URL
net/ forum?i fxh
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference Learning Representations, 2022. URL
htttps://ppenreviiet net/ forum? 2oorummiidd--s@PpGc
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
7/jmlr httpll/.orr/ppppe////2//// 0744 html
Norman Ricker. Wavelet functions and their polynomials. Geophysics, (()): 07 1944. ISSN 0016-8033. doi: 10.1 1445082. URL/irps:s//ae...0//10....
Ohad Rubin and Jonathan Berant. Retrieval transformer: with self-retrieval, 2024. URL
htw.://eerrracttaaaaaaaanrrrreea Long language modeling
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position represen- tations. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 2 (Short Papers), pp. 464-468, New Orleans, Louisiana, June 2018 Association for Computational Linguistics. doi: 11/665//1//N8-20771 URL https: / /aaaaaaannttoooooo org/ N1NNNNN8--000
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. 2021.
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaud- hary, Xia Song, and Furu Wei. A length-extrapplatabbe transformer. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki (eds.), Proceedings of the 6l Annual Meeting of the Associa- tion for Computational Linguistics /Volume 1: Long Papers), pp. 14590-14604, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 12216665////2223aalllloo http://c.....oooloooooo///////..
URL
wtt. acw-33a.cll1oo aclnooo 816
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Kei,, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let net- works Iearn high frequency functions in low dimensional domains. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc ISBN 9781713829546.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena : A benchmark for efficient transformers. In International Conference on Learning Representations, 2021. URL https: //openrvvie ew. net. frrrmoorrmm? wtw.:rmmmddeeeeoomooooommm d=qVyew-grCRk
R. Tian, Z. Wu, Q. Dai, H. Hu, Y. Qiao, and Y. Jiang. Resformer: Scaling vits with multi- resolution training. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22721-22731, Los Alamitos, CA, USA, jun 2023. IEEE Computer Society. doi: 00.11099CVRS522292223.001111
URL https: http://l....eeccptttttrrrreeoe.r 10. 10.1109//VPR527220 202152222292222...0 02176
13

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation lan- guage models. ArXiv, abs/2302. 13971, 2023a. URL https: //aps:://ap... semant cscholax #TTp//oorpus1:22522135
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fermandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL pattps://arxiin org/ abs 2307 09288
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL Eic233
Guyon, U. Von
cctp:per...p..eeeeoee fi /ess/pperr//011///////
Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simon- sen. Encoding word order in complex embeddings. In International Conference on Learning Representations, 2020. /RRp:attt////eeett...
net/ htttt/foummiiie Hke WTVtwr
Yuhuai Wu, Markus Norman Rabe, DeL.ssley Hutchins, and Christian Szegedy. Memorizing transformers. In International Conference on Learning Representations, 2022. URL https:
//openreview. wwt. wwtp:/eeeeettooomdddeeeoroo
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Mar- tin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective longcontext scaling of foundation model. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume Long Papers), pp. 6643-4663, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10018633311/2224naaceeeeee https://acl antho www.nnhhhoo.......oooo wwwp.orgg22444 naacl-long. 260
URL
Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Soaring from 4k to 400k: Extending Ilm's context with activation beacon, 2024. URL https: arxiv.
orq/ ht5//bb/24401.. 03462
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ROTARY POSITION EMBEDDING
RoPE incorporates positional information directly into the self-attention mechanism by rotating the query and key vectors in the complex space. When divided into even and odd dimensions, the following calculations are performed for the m-th query each sequence. In even dimensions, RoPE is expressed as follows.
q0
cos m0n
sin mb:
0
90
m
Q2
0
m0,
sinm
Ad
0
cos m0a/2
sin m00a2
(16)
In odds dimensions, RoPE is expressed as follows.
4i
sinO,
cos0i
0
90
43
0
0
sinm0,
cosm
Qi
(17)
sinmma/2 cosm
4d-2 9d
where 9thh E Rixd is the m-th query when the number of dimensions is d and 0g 10000-2(--)/d i [1, 2,,,,...
222....d///. d/2]. The same process also performed for the n-th key km E Rixd
A.2 PROOF OF THE EXISTENCE OF f(t)
We prove the existence of f(t) as described in 3.2 such that Φ, = 0j+1 = m0 where 0,
20000-2 and i E [1, ..., d/2]. Here, we restrict our proof to (b(1) in Eq. but similar argument be applied to v (t), following analogous steps establish its validity.
First, we revisit the definition of (t)::
cos f(t)
0st<1,
(t) sin f(t)
1st<2
(18)
otherwise.
Here, f: R is monotonous function that satisfies 1 (b) dt = 0 and Eq. Assuming that when (0 S S d - 1) is signal with d elements, the wavelet z! is used and wavelet transform performed each scale a We define the shift parameter by Here, ö(L) is monotonous function such that 0 s < d - 1 and 0 < d(1) <
8()) 0, 2, .., 2).
r(0)
W(1.b)) bo) 6(),b))
coo do sin d'1
0
0
x(1)
dbe
O3
w(1,))
cos d
xG 1)
008 0a-2
-in da-1
x(d 2)
Lord 1)_
(19)
To simplify the notation in the matrix representation above, we write Φj for j = 0, 0......, 1, where Ø = f1 + 8(f)) if is odd, and Φ, f(((j)) otherwise. We let be the query qhh The function ft) is defined such that 0 < (t) I 2kT for 0 < t < and < f(t < 2k for S < 2, where l: is the smallest natural number satisfying m 2km
15

---

Published as a conference paper at ICLR 2025
Do Haar-like wavelets satisfy the necessary conditions of wavelet? Here, f(t) must be function such that tt() satisfies the conditions of a wavelet. For Eq. | it is evident that holds for any f satisfying 0 < f(t) < 2kT. Next, we consider the zero-mean property. As example, consider f(1) defined as f(t) = 2krt for 0 - < 1 and f(t) 2k 1) for & t < 2. we set
0 = (t),, have:
(t)ddt = fa
2kx
2km
cos 0d0 +
sin 0d0 0.
(20)
Since this satisfies the zero-mean property, we conclude that there exists an ft) such that (t) is wavelet.
Furthermore, we observe that there exists 8(() satisfying 0, f(o(i))) Φ5+1(= (=++(((()((()) f(1+8()))
2k) = m0 - for j 0, 2, .............................. 2. In other words, we can simply choose function d (
m0 2
that satisfies 8(5)
for j = 2, ,d - 2.
2k
A.3 HAAR WAVELET
Here, we explain wavelet transform using the Haar wavelet, which the simplest wavelet. The definition of the Haar wavelet is as follows.
0 t<12,
1/25|1, ø(t) = f1 0 0
(t)
(21)
otherwise.
otherwise.
Haar wavelets defined not only wavelet function but also by a scaling function &
The method of analyzing signals by performing a discrete wavelet transform using these two func- tions is called multi-resolution analysis. When the scale is fixed 2 and the shift b 2, d/2), the wavelet transform using the wavelet function and scaling function is expressed as follows.
(0)
(1))(1)
(2))
(3)
VT 2)
<T 1)
d2,o(0)
(2)
(2)
22 2)
d. 1)
x(0)
02.0(-2
va.0
(z))(0)
(1)
=(1)
(2.0(-2)
(2)
(2))(0)
d2,0(1)
02.0 4)
02.0(T 3)
x<2)
v2.0
v2.0 1)
22.0 2)
02 4 +3)
v2.0(0)
(2.0
a - 2)
602.0
02.0
1--2--1 +
62,0l
+
22
+3)
02.0
d2,((1)
11
(22)
From Eq. 21
2200 and 22,0 are follows.
1/V2
0 0
x20((1) = 1/v2
0 s<t
(2)
1/V2
1 1st2,
(23)
otherwise.
otherwise.
Therefore, the Haar wavelet transform a x 2 block matrix.
#22. 0) 1/V2
-1/22
0
2(0)
6(2.0)
1/V2
1/v2
0
2(1
(22 ((2,2)
0
0
1/v2
-1//2
0
1/v3
1/v2
2(3)
(24)
(22, 2)
1/V2
-1/V2
(02, T 2)
0
1/V2
1/V2
This matrix is the Haar forward transform using matrix multiplication for a T element signal. This matches the RoPE matrix with mf = π 4/4.
16

---

Published as a conference paper at ICLR 2025
A.4 ISN'T ROPE FOURIER TRANSFORM?
We also hypothesized that this could be equivalent to a Fourier transform. However, this hypothesis does not hold. When signal ær()) that changes time is Fourier transformed, its spectrum F(k) is obtained. The process of converting an actual discrete signal æ(()) into a spectrum F(k) as follows.
T
F(F) = Σ a(t)w
(25)
t=
The Fourier transform can be expressed as a matrix formula as follows.
w-00
w-1
w0-2
(1-1-1)7
F F(L)
w-k-a
1.1.1
w1.2
w
x(0)
2-1
w
x=1)
F(2)
(=2)
(26)
on 1--D
Here, f E is the wave number, T E R is the number of samples, and i is the imaginary unit. exp( 2x1) - is called the Twiddle Factor (Gentleman & Sande 1966). which complex number expressed in polar form using Euler's formula e cos isinf. In the complex plane, wf represents point on the unit circle with an argument of the complex number _ From this formula, we can see that the Fourier transform calculates the inner product of all signals and sine waves. However, RoPE, the inner product with sine waves is calculated only within each block.
Next, when calculating the attention score with RoPE, does the Fourier transform hold? Attention scores of the m-th query tth and the n-th key ka with RoPE are calculated as follows.
R
4/2
R (Q.) R (12311
( ) R
R Ki
(27)
R4/2
K-d/2
where 4/2 the query divided into every two dimensions, and R4/2 is the rotation matrix.
Q4/2
0m dd-1
K:d/2
Rd-11
n
R4/2
[oosmBa/2 sinm0a/2
-inmm0a22 cosm0
4m
Rd
Aligning with the Fourier transform, as illustrated Equation 26 requires a process involving the inner product between frequency tensor of dimensions fxT and signal tensor of dimensions' Tx1 (such as the query vector). However, RoPE operates on independent 2x2 blocks, where each block is processed separately. Consequently, RoPE's block-wise operations do not conform to the structure required by the Fourier transform. Moreover, we focus solely on the RoPE and key operations in Equation 27] they may appear to align with the structure of a Fourier transform. However, since the final step involves taking the inner product with the query, the overall operation deviates from the path of becoming a perfect match with the Fourier transform. Furthermore, the rotation factor represents a rotation in the complex plane, and even if is expressed as Eq. using a rotation matrix, it does not completely match a rotation matrix that represents a rotation in the Euclidean plane.
Therefore, RoPE cannot be equated with the Fourier transform. Furthermore, even if it were the same as the Fourier transform, it would be unsuitable for processing non-stationary signals and thus unsuitable for processing natural language, which is non-stationary flow.
17

---

Published as a conference paper at ICLR 2025
A.5 CONSIDERATION OF WAVELET TRANSFORMATION BASED ON ROPE
In this paper, we explore the incorporation of wavelet transforms into RoPE (Relative Positional Encoding) following our previous discussion on RPE (Relative Position Encoding). In this regard, integrating wavelet transforms into RoPE presents challenges for controlling computational and memory costs. In Sections 3 and 4, we highlighted the potential effectiveness of employing multiple scales for extrapolation. With this in mind, we present a simplified formula for applying various scales and wavelet transforms to RoPE, which we refer here as a RoPE-based Wavelet.
96"
cos m0 sin m0
sin m0
9
qi"
cos m0
qi
cos m02
sin m02
m0
sin mll
Q2
q3
sin m0,
cos m0,
m0,
m0,
93
4d-1
oos m0aa/2
sin m0aa2
COs m0
sin m0
cos rela/2
9d-2
[d-2
sin m0aa2
Cos m0a/2
sin mday2
sin mm
sin m0a/2
cos m00/2
g8-1
(28)
where 9thh 10000
Rlxd is the m-th query when the number of dimensions is d and 10000-2(1-1)/4 20/d1//d 21, ..., d/2]. The same process also performed for the n-th key k" Bixdd
Conversely, the method introduced in Section 5 is here called RPE-based Wavelet. The key differ- ences between RoPE-based Wavelet and aas-d Wavelet are follows:
Number of Scale Parameters: In RPE-based Wavelet, the scale parameters can be selected up to the maximum sequence length. However, in RoPE-based Wavelet, the selection is limited to a maximum of d.
Memory Usage: RoPE-based Wavelet requires a wavelet matrix that corresponds to the number of absolute positions m. Consequently, the memory usage is significantly higher. Unlike RoPE-basee Wavelet, RPE-based Wavelet does not necessitate wavelet matrix that matches m values, allowing the use of Tip 2 from Appendix A.6 A which improves memory efficiency.
When applying wavelet transforms using RoPE-based Wavelet, it is necessary to use absolute positions. In contrast, RPE-based Wavelet can use relative positions. which enhances extrapolation.
Absolute and Relatíve Positions:
Computational Cost: Implementing wavelet transforms via RoPE-based Wavelet requires processing both the query and the key, necessitating two calculations. RPE-based Wavelet, discussed in Section 5 only requires computation, since processes only the query.
Additionally, we conducted an experiment with RoPE-based Wavelet. Unfortunately, we had to halt the learning process because it took over five times longer than anticipated. Considering the learning costs associated with large-scale language models in recent years, we believe the RoPE- based Wavelet approach is not feasible.
18

---

Published as a conference paper at ICLR 2025
A.6 IMPLEMENTATION TIPS FOR WAVELET POSITION REPRESENTATION
Tip 1
Similar to RPE(Shaw et al. 2018), we used Eq. (T0) as
aij softmar (9: R:KTT + q:(pis)))
vdg
By transforming it in this way, it is possible to reduce the computational complexity to Obbatch x n x length d + length? x d), where batch is the batch size, n is the number of heads, length the number of tokens, and d is the number of dimensions of each head. The experiments in Section are implemented based on the methodology introduced in this section.
When dealing with long contexts of over 4 k with large model, the memory efficiency of (d, length, length) of the wavelet position becomes a bottleneck. Therefore, we further reduce the memory usage (d, length) by using torch. scatter scatter the wavelet position represen- tation to the attention mask. In the relative position representation in the decoder, only the position information of the token before the current token is required, for example, 0, -1, -2, elc. There- fore, we pre-compute the information up to 0, -1, -2, ..nenghh and reduce the memory usage by using torch. scatter distribute it. Specifically, we prepare a (d, length) wavelet tensor and calculate the 2D inner product with the query, which has been transposed to (length x batch, d). The tensor after the calculation becomes (length x batch, length), which then scattered us- ing torch. S sca tter so that it becomes relative position in the attention mask. This reduces the amount of memory used from (d, length, length) to (d, length), and the calculation can be performed using calculations between 2D tensors. The experiments in Section 7 are implemented based on the methodology introduced in this section.
Típ 2
A.7 EXPERIMENTAL SETTINGS IN SHORT- CONTE-T EXPERIMENT
The parameter settings used in the extrapolation experiments were the same those the original ALiBi paper. The dimensionality of the word embedding dmndel is 1024, the number of heads N is 8, the dimensionality of the heads is 128, and the number of layers is 16. The implementation was based on the fairseq (Ott et al. 20199-basee code [2] provided in previous work(Press et al. 2022), and all hyperparameters were set the same values as those the literature(Press al. 2022). The number of training epochs is 205, and the batch size is 9216. The learning rate was set to 1.0, and the learning process was updated by 1c-7 every 16,000 steps.
A.8 EXPERIMENTAL SETTINGS IN LONG-CONTENS EXPERIMENT
The dimensionality of the word embedding dinodet is 4096, the number of heads N is 32, the di- mensionality of the heads d is 128, and the number of layers is 32. The number of training steps is 30,000, and the batch size is The learning rate was set to 0.0003. We used Adam AdamW(Loshee ((oshhhhllyy & Hutter 2019) as the optimizer, with (3,, B2) (0.9. 0.95). In accordance with previous research (Rubin& Berant 2024; Wu et al. 2022] Zhang et al. 2024), we then used 100 sampled sequences in the training set for evaluation. In this experiment, due to the large model size and long sequence length, we report perplexity only for non-overlapping inference using Lerain, since memory capacity is exceeded.
near 1inar
ases
19

---

Published as a conference paper at ICLR 2025
A.9
RICKER WAVELET
Figures and show the Ricker wavelets with multiple scale a.
1.00
a=1
0.75
a-2
a-4
0.25
a-16
0.00
-0.50
-80
-60
-40
-20
40
60
80
Figure 4: Graph of compared Ricker wavelet functions with a = [2 21, 22, 23
2
0.8
a-64
0.4
a1:12
0.0
-02
-3000
-2000
-1000
1000
2000
3000
Figure 5: Graph of compared Ricker wavelet functions with = 27, 29.22
A. WAVELET TYPE
Figure 6
shows graphs of the wavelets compared in Section 6.3.1 It can be seen that the simplest is the Haar wavelet, while the most complex the Morlet wavelet.
nicker
Morlet Gaussian
0.50
0.25
0.00
-0225
-50
-40
-20
20
40
60
Figure 6: Graph of compared wavelet functions. The case with scale parameter 24 and shift parameter = is shown.
20

---

Published as a conference paper at ICLR 2025
A.11 EXAMPLE OF HEAT MAP AND TEXT CORRESPONDENCE
Figure 7 shows the attention map after softmax operation for the proposed method. First, the notable feature the proposed method is that is always able pay attention specific tokens. The words that always receive attention are those that are important in the sentence, such as the <
> token, the first token, and words that are the subject of the sequence. such as 'he.' Moreover, as with ALiBi, the proposed method has a different scope of attention for cach head.
n=1
n=
n=4
2015 leadership contast to succeed Milihand Fit is a suppoomer.t of a Beesti repubic. Detroit Red Winga of the National Hockey League. 
 </8> = Filmogrspty <lls agents Frank Black Lance Herriksen) ) and Emma Holls Klea Scott travel to Travelers Rest,
dts Radcliffe is a fan of the </8 
 Awards and nominstions = <iB> </t> </t- Invesligation by the
Sei
olso
etj after softmax operation for the proposed method. The maximum sequence length is Immaz = 512, and the sequence length at inference is L, = 1012. From left to right, n 1, 2, 4th heads are shown. Scores above 0.01 mapped in black and the rest in yellow. Words that were always given attention in all heads are shown in red, and words that were frequently given attention only in the n 2nd head are shown in blue. Sentences are omitted in the middle because they are long with 1012 tokens.
Figure 7: Heatmap of attention
21

---

Published as a conference paper at ICLR 2025
A.12 CAN IT HANDLE TOKENS WITH LONG-RANGE DEPENDENCIES?
ALIBI
RoPE
Wavelet(OUrs)
1st
2nd
3th
5th
6th
7th
REA mh
8th
Figure 8: Heatmap of scaled attention scores via softmax normalization in 1-3rd and 5-8th head after softmax operation for ALiBi, RoPE, and our method. For clarity, values of 0.001 or more are mapped black, while values below that are mapped yellow.
22

---

Published as a conference paper at ICLR 2025
Table 3: Perplexity of validation set in extrapolation experiments using Wikitext- 1033. Maximum allowable length of sequences in pre-training is Ltrain = 512.
Sequence Length
shift
128
Perplexily without Nom overlapping Inference
Ricker
(2" 21
27)
{0, 1, 15)
23.64
20.82
19.19
18.23
18.00
17.99
Ricker
[2' 12',2
{0, 1, 2.... 15]
23.77
20.89
19.25
18.23
17.97
18.02
Ricker
f22.2a
{0, 1, 2.. 15}
23.92
21.03
19.40
18.41
18.14
18.07
Ricker
(2".21, 22.2
(0, 20.,,
23.96
Ricker
(20.
1,
24.49
21 .0
19.95
20.90
32.01
70.80
Ricker
(2 (2" 2"., 2
(0, 1,21,
23.74
20 88
19.24
18.22
17.96
17.84
Ricker
120 231
1,,1,2.3)
23.75
20.86
19.26
18.24
17.96
17.84
Ricker
120 ..,
(U.1)
23.75
20.88
19.30
18.31
18.04
18.02
Ricker
(2", 2
(0)
Ricker
(27
10.1,2...., 127]
24.35
21.45
19.80
20.68
20.87
21.31
Gaussian
[2 (2 (2
27
[0,1,2.... 15)
23.97
20.90
19.30
T8.31
18.02
1 17.88
Gaussian
10 15)
23.92
21.02
19.41
18.41
18.15
18:01
Gaussian
20, 15)
23.98
21.09
19:46
18.43
18.13
17.93
Gaussian
12",2".0 22.2
20. 311
23.83
Gaussian
12".21
1. 1..
24.28
21.35
19.70
18.96
19.63
23.14
Gaussian
(2 2
(0, 1,2...2....7
23.72
20.86
19.24
18.24
17.95
17.77
Gaussian
(20 2 ...2211
(0. 1,3,3
23.78
20.92
19.29
30
18.01
17.85
Gaussian
(2 ,2 (2
1)
23.86
20.98
19.37
18.20
18.10
Gaussian
2
(0)
24.21
19.68
18.45
18.45
Gaussian
(27
1..1.2 1277
127]
24.48
21.62
20.05
19.53
22.63
35.23
Haar
24.98
22.07
20.49
51.61
166.87
299.26
Haar
(2
27
1. 15)
23.73
20.89
19.27
18.34
18.11
18.17
Morlet
(20.2) 2
[0, 15}
24.15
A.13 ABLATION STUDY SCALE AND SHIFT PARAMETERS
In this section, we present the findings from our ablation study focusing on the shift and scale param- eters of the Ricker and Gaussian wavelets. As indicated Table both wavelet types demonstrate substantial effectiveness in our method. To further evaluate their performance, we explored the con- tributions of two parameters, i.c., the scale parameter a and the shift parameter b, while keeping all other settings consistent with those outlined in Section
The results of our experiments are summarized in Table 3 Both the Ricker and Gaussian wavelets exhibit similar trends regarding the influence of the scale and shift parameters on extrap- olation performance. Initially, we observed that increasing the scale parameter value while hold- ing the shift parameter b ((20, 21, 2") X {0, 2, 15). {2' 22, ..., 28} x {0, 1, 2, ... 15) and (22.2) 23 ...,. 2º) 1,2, 15)) constant maintained the performance of extrapolation, albeit with some fluctuations. Conversely, when we increased the number of shift parameters while decreasing the number of scale parameters (f20, 22,,, 22, 24) (0, 1, 2,. and (2 2 x f0, 1,... 63)),, there was a noticeable decline performance. These findings underscore the significance of the scale parameters in extrapolation. Moreover, we found that increasing the number of scale pa- rameters while decreasing the number of shift parameters led to performance improvements in some instances (120, 21 .2.,....25) 255
Results
3)).. However, {0, 1, 2.., 7} and (2., } x {0, 2,
when the shift parameters were reduced to two or entirely eliminated ((20) 21 263 x (0, 1} and (2 21, 21,,,,,..,...... 2127) ), relying solely the scale parameters resulted in deterioration of extrapolation performance, Moreover, even when the scale parameter was fixed and only the shift parameter was used ((2) x {0, importance of shift parameters as well.
127)), the extrapolation performance decreased. This suggests the potential
In conclusion, our analysis highlights the critical roles of both shift and scale parameters in the effectiveness of our wavelet-based method.
23

---

Published as a conference paper at ICLR 2025
A.14 ABLATION STUDY OF WAVELET TYPES
In this section, we also explored a variety of wavelet types beyond those previously discussed. In Section 6.3.1] our focus was primarily on wavelets that could be computed directly from mathemat- ical formulas. However, in this section, we expand our inquiry to include wavelets with varying numbers of vanishing moments well discrete wavelet transformations. Additionally, drawing from previous research (Wang et al. 2020), we considered the necessity for a distinct approach when incorporating complex numbers into positional encoding. Consequently, our study did not encompass wavelets that incorporate complex numbers.
Wavelet types The specific wavelets under consideration in our investigation are outlined as fol- lows:
Daubechies (db) (Dabbechie] 1992) Compactly supported orthonormal wavelets
Symlets (sym) Wavelets with minimum asymmetry
Coiflets (coiE) The scaling and wavelet functions have the same number of vanishing moments
Meyer (dmey)- Wavelets defined in the frequency domain
Biorthogonal Spline (bior)- - Two wavelets are used: one for decomposition, and the other for reconstruction
Reverse biorthogonal Spline (rbio)
In addition, the graphs of these wavelets are shown in Snguresyandd 10 As the number of vanishing moments increases, the wave oscillation becomes larger. Therefore, we also conducted a survey by vanishing point moment. The name of a wavelet is derived from the number of vanishing mo- ments. For example, db6 is Daubechies wavelet with vanishing moments, and sym3 is a Symlet wavelet with 3 vanishing moments. In the case of Coiflet wavelets, coif3 3 is Coiflet wavelet with 6 vanishing moments. The names of bior and rbio wavelets are derived from the number of vanishing moments possessed by the decomposition and reconstruction wavelets, respectively. For example, bior3. 5 is Biorthogonal wavelet that has 3 vanishing moments for the decomposi- tion wavelet and 5 vanishing moments for the reconstruction wavelet. Biorthogonal wavelets and Reverse- Bintrss-Biorthogonal wavelets can calculate the approximate values of decomposition wavelets and reconstruction wavelets, but in this case, we only used the values of decomposition wavelets.
Experimental Settings We used Pywavelet (Lee et al. 2019 to calculate the approximate val- ues of these wavelets. In addition, this experiment, we calculated the approximate values by specifying 8 levels of 21,2,... ..., 8} instead of the 8- 8-pattern scale parameters (2", 2 2,,,.,,,?? 27). We used the shift parameter 10, .... 15]. The other experimental settings are the same as those in Section
6
The experimental results are summarized in Table 4 Overall, the performance observed was suboptimal for extrapolation. However, it is important note that since the parameters were fixed at levels 21.2.... ..., 8}. we believe that performance may be enhanced with adjustments to these levels. Notably, the rbiol.1 wavelet demonstrated promising extrapolation capabilities, suggesting significant potential for future improvements. In contrast, the coif and dmey wavelets exhibited lim- ited performance, even with shorter sequences, indicating their potential unsuitability for position encoding tasks. Conversely, while the extrapolation performance (> 512) of other wavelets was generally low, their interpolation performance (S 512) remained consistently stable, highlighting another avenue for enhancement. Furthermore, the performance of the db, bior, and rbio wavelets showed a positive correlation with an increasing number of vanishing points. This finding under- scores the importance of vanishing points a critical factor influencing performance. In conclusion, our analysis indicates that both the shape of the wavelet and the number of vanishing points play significant roles in determining extrapolation performance. Future work should explore these rela- tionships further to identify optimal configurations for improved performance outcomes.
Results
https::/pywaaell httptts..eeeeeecccooeo///////aee
24

---

Published as a conference paper at ICLR 2025
Table 4: Perplexity without Non-overlapping Inference. We evaluated the validation set in extrapo- lation experiments using Wikitext-103. The maximum allowable length of sequences in pre-training Liaain 512.
Sequence Length
Wavelet type
128
256
512
1012
1512
2512
Continuous Waveler Familios
Ricker Gaussian
23.64
20.82
19.19
18.23
18.00
17.99
23.77
20.90
19.30
18.31
18.02
17.88
Morlet
24.15
21.28
19.65
19.02
20.46
26.56
Familliss
Huar
23.73
20.89
19.27
18.34
18.11
18.17
0b2
25.22
22.76
20.64
30.50
60.27
130.93
db4
25.22
22.47
21.37
41.78
51.75
56.18
db1 6
25.23
22.43
21.24
21.15
22. 16
46.65
db32
25.12
22.35
21.14
21.20
22.40
38.00
aym2
25.11
22.21
20,68
31.25
61.00
126.32
8ym4
25.27
22.56
21.98
24.70
26.81
42.8
sym3
23.97
31.47
92.36
coifl
31.24
28.00
26.24
64.62
71.06
97.60
coif4
49.91
45.15
42.42
41.07
56.08
10.27
coif8
25.15
22.39
21.26
21.31
22.26
35.73
colf16
126.38
117.88
113.42
132.14
166.77
230.95
dmey
30.38
27.12
25.45
25.88
46.35
131.48
biorl.3
88.66
Eior2.2
25.28
22.51
21.59
29.71
29.43
$0.25
blor2.6
bior3.1
26.92
24.02
22.38
59.30
113.81
205.54
bior3.5
25.17
22.49
21.65
27.41
27.19
$3.99
25.24
22.48
21.51
21.89
23.86
50.14
bior1.1
25.52
22.72
21.64
21.67
24.42
$1.46
bior5.5
bior6.8
25.14
22.39
21.21
21.10
22.31
46.97
rbio1.1
tbio1.3
25.28
22.50
21.39
$2.06
47.78
59.94
rbio2.2
25.92
23.08
21.98
68.57
86. 12
93.90
rbio2.6
25.29
22.68
21.60
24.54
24.47
44.57
25

---

Published as a conference paper at ICLR 2025
db2
db4
db8
50-
-45
db16
db32
-04
sym2
sym4
sym8
40
114
coif1
Coif2
Coif4
63-
30-
coif8
coif16
dmey
525
-04
-22
Figure 9: Graph of compared wavelets with level=10. Pywavelet (eee al. 2019 used to calculate wavelets.
26

---

Published as a conference paper at ICLR 2025
bior1.3 3.3
bior2.2
bior2.6
55
-00
-15
-18
bior3.1
bior3.5
bior3.9
200
23
bior4.4
bior5.5
bior6.8
13
18-
-030-
-0.9
rbio1.1
rbio1.3
600
600
1228
-229
-007
rbio2.2
rblo2.6
3.83
675
638
676
038
0.0
Figure 10: Graph of compared wavelets with level=10. Pywavelet (Lee et al. 2019 used to calculate wavelets.
27

---

Published as a conference paper at ICLR 2025
A.15 EVALUATION ON LONGBENCH
The models pre-trained in Section were evaluated on LongBench (Bai et al. 2024). This evaluation was conducted using a dataset that contained relatively long sentences, Furthermore, the multi- document QA task and single-document QA task were evaluated on all datasets. Since pre-training was conducted using an English dataset, evaluation was conducted using only the English dataset.
The results are shown in Figure II In some datasets, the performance of the model that adopted RoPE was good (Qasper, MuSiQue, and QMSum). In NarrativeQA, the models attained almost the same score. However, in the remaining tasks, the proposed method was more effective. Note that this is an evaluation of model that was pre- pre-trained on small dataset (redpajama-Il) 1B). As future work, it will be necessary to pre-train the model with a larger dataset and conduct evaluations with other models that are effective for long sentences, such as LongRange (Tay et al. 2021
3.5
-0.5
-1
NarratigegQQA
Qasper
MultFietdQAAeen
HotporQA
2WIkIMQA
Musique
TriviaQA
SAMSum
QMSum
Figure 11: Evaluation results using LongBench(Bas (BaichhhBBiia et al. 2024 We evaluated the model pretrained Section The scores of the difference between the model using the proposed method, which uses wavelet-based position representation, and the model using RoPE are shown. The tasks were evaluated using the dataset, which contains relatively long sentences.
Table 5: Overview of the dataset statistics in LongBench (Bai et al. 2024). Avg len (average length) computed using the number words in the English.
Dataset
Avg lon
Metric
Samples
NarrativeQA
18,409
F1
200
Qasper
3,619
F1
MuldheeldQAA-en
4,559
F1
150
HotporQA
2WikiMQA
4,887
Fl
200
MuSiQue
11.214
Fl
TrivisQA
8.209
Fl
200
SAMSum
6258
Rouge-L
200
QMSum
10614
Rougo-L
28


Reviews:
[{"review_id": "E6duFTc7ts", "reviewer": "Reviewer_QwSG", "summary": "In this work, the authors proposed wavelet-based positional encoding for the length extrapolation problem of language models. The motivation is based on an observation that the widely used Rope is related to wavelet transform via simple Haar wavelet functions with a fixed scale. By further investigating the properties of ALiBi, the authors modified the form of relative positional encoding to use wavelet transform based approaches. Several experiments are conducted to demonstrate the performance of the proposed positional encoding.", "questions": "See the Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is easy-to-follow. \n2. The length extrapolation problem is important for language models.", "weaknesses": "1. **There exist gaps from the motivation to the proposed approach**. In Section 3 and 4, the authors provide analysis on the relationship between RoPE and wavelet transform, and the properties of ALiBi like positional encodings. The ability of ALiBi to accommodate multiple window sizes is concluded as the key point for better length extrapolation performance, while RoPE is claimed to be worse. Based on these statements, a natural question is, what is the advantage of using wavelet transform? The authors do not provide enough supportive quantitative evidence for it, but directly modify the relative positional encoding by using wavelet transform. If it has superiorities, why don't we follow the form of RoPE to use wavelet transform instead of RPE? These points should be better clarified to make the motivation of the proposed approach more convincing.\n\n2. **The design choice for the proposed approach should be better supported**. In Section 5, the authors propose to use Ricker Wavelet as the base function of wavelet transform, and set the shift and amplititude parameters with predefined strategies. However, the reasons to choose these designs are not well clarified (except lines 294-303, which are rather superficial).\n\n3. **The empirical improvement is marginal**. From Table 2 and 3, we can see that the proposed PE does not show significant improvement compared to compared approaches such as ALiBi, which is proposed in 2022 and is not the state-of-the-art nowadays. The evaluations are also limited in task types and scales. Overall, the empirical results are not convincing to show the superiority of the proposed approach.\n\nOverall, I hope the authors can well address the above concerns, which I think are important for the quality of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZkIXSYKMqT", "reviewer": "Reviewer_fCa9", "summary": "the paper presents a novel positional encoding method leveraging wavelet transforms to address the challenges of limited receptive field and  extrapolation in LLMs. \n\nThe proposed wavelet-based approach aims to overcome these limitations by introducing a multi-scale analysis that captures the fluidity of natural language and enhances the model's ability to extrapolate beyond its training context length.\n\nIn extrapolation experiments, model that used Ricker-based wavelet positional embedding had the lowest PPL trained on the WikiText-103 dataset, compared with RoPE, ALibi, Transformer-XL.", "questions": "- How did the “shift and scale parameters” been decided? Is there any ablation study or results for this particular setting?\n- Did you compare the length extrapolation results with other RoPE-based methods? (eg. relative works you mentioned in L47-L48)\n- Both RoPE and Wavelet methods aim to model distances at different scales by placing them in different dimensions [1]. In Figure 3, the diagonal lines in the RoPE plot appear because the model can focus on the features of specific token distances. However, Wavelet functions do not have this stable frequency inductive bias, which is why they do not produce diagonal patterns in the attention scores. Do you think this is a desirable property?\n- In the experiment represented by Figure 3, what kind of text is input into the model? Why is the initial word particularly important? Which specific tokens does the model attend to? Is this phenomenon common?\n\n[1] Hong, Xiangyu, et al. \"On the token distance modeling ability of higher RoPE attention dimension.\" arXiv preprint arXiv:2410.08703 (2024).", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The authors provide a solid theoretical foundation by drawing parallels between RoPE and wavelet transforms, and by extending this analogy to propose their method.\n- The proposed method has the potential to be widely applicable to various transformer-based models\n- The paper is easy to read and positions itself clearly with respect to related work.", "weaknesses": "- The paper primarily evaluates the method on language modeling tasks. It would be valuable to see how the approach generalizes to other NLP tasks such as question answering or text summarization.\n- The paper does not provide sufficient experimental evidence to support the authors' claim that Wavelet Transform can capture the dynamic changes in a sequence over positions.（L84-85）\n- See my questions/suggestions below", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rmom89YIBT", "reviewer": "Reviewer_DFcs", "summary": "The paper proposes a wavelet transform-based positional representation for transformer models. It begins with highlighting the properties of existing position representation techniques such as relative position bias, RoPE and ALiBi. The authors then show how RoPE can be interpreted as a single scale wavelet transform with a Haar-like wavelet. Then the authors present a position embedding based on wavelet transform. This multi-scale embedding can be viewed as a generalization of RoPE and possess attractive \"multi-window\" properties of ALiBi. Experiment results have been reported on short and longer-context scenarios showing improved perplexity over existing methods and better extrapolation properties.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written barring some typographical errors. It motivates the problem well, starts with a well-defined goal, describes existing methods clearly, and presents the proposed method in a manner which is easy to appreciate.\n- The paper tackles a critical problem of context length extrapolation which often arises in practical settings. The method holds significance, not just for the language modeling community, but also other domains such as time series forecasting where context length extrapolation may improve the performance of models on high-frequency data (see [1]). \n- The proposed method is novel to the best of my knowledge. It creatively relates RoPE to time-frequency analysis and proposes a promising method based on wavelet transforms. \n- The method outperforms alternatives and scales gracefully with extrapolated context lengths. \n\n[1] Ansari, Abdul Fatir, et al. \"Chronos: Learning the language of time series.\" arXiv preprint arXiv:2403.07815 (2024).", "weaknesses": "- While the discussion of related methods is generally well done, discussion of RoPE scaling techniques (linear, NTK-aware) is missing. A discussion and comparison with these techniques would significantly improve the positioning of this work.\n- The results reported in sections 6 and 7 are excellent proofs of concept but they lack comprehensiveness. Particularly, in section 7, evaluations beyond the CodeParrot dataset would be needed to thoroughly appreciate the proposed method. Furthermore, the paper only evaluates the model in term of the perplexity. For a stronger evaluation, this needs to be augmented with task-based evaluations. Apart from common language tasks, you might also want to consider associative recall tasks [2] and the long range arena [3] since the focus of this work is context length extrapolation. To further demonstrate the robustness of the proposed method, domains beyond natural language may also be considered, for example, DNA modeling [4], audio generation [4] and time series forecasting [1]. \n\nI am willing to raise my score if the evaluation is strengthened. \n\n[1] Ansari, Abdul Fatir, et al. \"Chronos: Learning the language of time series.\" arXiv preprint arXiv:2403.07815 (2024).    \n[2] Arora, Simran, et al. \"Zoology: Measuring and improving recall in efficient language models.\" arXiv preprint arXiv:2312.04927 (2023).    \n[3] Tay, Yi, et al. \"Long range arena: A benchmark for efficient transformers.\" arXiv preprint arXiv:2011.04006 (2020).    \n[4] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yM57GeN1C3", "reviewer": "Reviewer_kKUx", "summary": "This paper conducts a unified analysis of positional encoding methods in large language models and proposes a novel wavelet-based approach. The work begins by demonstrating that RoPE can be interpreted as a restricted wavelet transform using Haar-like filters operating at a fixed scale. The authors then analyze ALiBi, revealing that it functions similarly to windowed attention with varying window sizes but is limited by constraints on the attention mechanism's receptive field. Building on these insights, the paper introduces a new wavelet-based positional representation method that leverages multiple scales through wavelet transforms. The method is designed to capture both local and long-range dependencies without restricting the model's receptive field. The authors validate their approach through experiments on short and long contexts.", "questions": "1. How does your approach fundamentally differ from MGT's wavelet position encoding?\n2. Can you provide formal verification of the wavelet admissibility conditions for the proposed \"Haar-like wavelets\"? Or what else condition you might need to add to make it a wavelet?\n3. Why were these specific wavelet families chosen? Have you explored other wavelet families, particularly discrete wavelets like Daubechies or biorthogonal wavelets?\n4. What guided your choice of scale parameter ranges? Have you conducted sensitivity analyses on these choices?\n5. What is the theoretical justification for removing the $\\frac{1}{\\sqrt{a}}$ amplitude term?\n\n[1] Ngo, Nhat Khang, Truong Son Hy, and Risi Kondor. \"Multiresolution graph transformers and wavelet positional encoding for learning long-range and hierarchical structures.\" The Journal of Chemical Physics 159.3 (2023).", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation and connection are clear as the authors provide a unified analysis of the positional encoding methods.\n2. The paper introduces a new wavelet-based positional representation method that leverages multiple scales through wavelet functions.\n3. The proposed wavelet positional embedding shows improvements across tasks.", "weaknesses": "1. Novelty Concerns: \n- The core idea of wavelet-based positional encoding has been explored in the previous work GMT (Ngo et al., 2023a,b), which uses graph wavelets to generate node positional representations that can capture the structural information of a center node on the graph at different resolutions, though in a different domain. However, there is insufficient discussion of how this approach differs fundamentally from or improves upon prior works.\n2. Mathematical Rigor:\n- The unified comparison for different positional encodings is not rigorous as the wavelet property needs stronger mathematical justification. The proposed \"Haar-like wavelets\" in Equation (7) require verification of wavelet admissibility conditions beyond just square integrability.\n3. Experimental Design: \n- Limited exploration of wavelet families (only four tested).\n- Insufficient analysis of scale parameter choices (only two tested).\n4. Minor issues:\n- The paper lacks of clear notation definition and problem setup affects readability.\n- Inconsistent terminology between abstract (\"simple Haar wavelets\" in Line 18) and technical content (\"Haar-like wavelets\" in Sec 3.2).\n- The phrase 'single window size' (in Line 19) should be replaced with 'fixed scale parameter' to align with standard wavelet terminology, as it specifically refers to the dilation factor in wavelet transforms.\n- The space definition L⊭(R) in Line 135 appears to have a typographical error - it should be L²(R), the space of square-integrable functions.\n- Some mathematical notation is not explicitly defined. For example, the superscript T in equation (3) seems to represent the length of a discrete sequence, but this isn't explicitly defined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)