Round and Round We Go! What makes Rotary Positional Encodings useful?
Authors: Federico Barbero
Alex Vitvitskyi
Christos Perivolaropoulos
Razvan Pascanu
Petar Veličković
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Transformers
Positional Encodings
Rotary Positional Encodings
Primary area: foundation or frontier models, including LLMs

Abstract: Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust `positional' attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments to verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling LLMs to large sizes and context lengths.

Published as a conference paper at ICLR 2025
ROUND AND ROUND WEGo! e WHAT MAKES ROTARY POSITIONAL ENCODINGS USEFUL?
Federico Barbero University of Oxford
Alex Vitvitskyi Google DeepMind
Christos Perivolaropoulos Google DeepMind
Razvan Pascanu
Petar Veličković Google DeepMind
Google DeepMind
ABSTRACT
Positional Encodings (PEs) are a critical component of Transformer-bast Large Language Models (LLMs). providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to RoPE to construct robust 'positional' attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling L.LMs to large sizes and context lengths.
INTRODUCTION
It is common to provide positional information to the attention mechanism in Transformers through the use of absolute positional encodings (Vaswani et al., 2017), relative positional encodings (Su et al., 2024), or by introducing a bias directly to the activations (Press et al., 2021). One of the currently most widely adopted encodings, especially in Large Language Models (LLMs), are Rotary Positional Encodings (RoPE) (Su et al., 2024), being used in popular models such as LLama 3 (Dubey et al., 2024) and Gemma (Gemma Team et al., 2024). RoPE acts on the queries and keys by splitting them in 2-dimensional chunks and rotating each chunk at different frequency. The method can be implemented efficiently and provides an interesting geometric approach to positional encodings.
Despite the significant adoption of RoPE, the specific reasons why this method is useful to Transformer models remains poorly understood. One of the main arguments in favour of RoPE made by Su et al. (2024) is that the method helps to decay attention coefficients as the relative distance grows. Most such claims, however, rely on the queries and keys being constant vectors which is uncommon in practice. In fact, in this work we find that there are many situations in which this decay does not occur and that this is exploited at times by attention heads in Gemma 7B (Gemma Team et al., 2024).
Further, there are open intriguing questions relating to how exactly the different frequencies in RoPE are useful. In the standard parameterisation of RoPE, the fastest frequencies rotate at radian per token, whilst the slowest are several orders of magnitude slower at * 1/10,000 radians per token. As dot product attention directly depends on the angle between the queries and keys, the highest frequencies are extremely sensitive to small token rearrangements, making them poor carriers of
Work performed while the author Google DeepMind.

---

Published as a conference paper at ICLR 2025
4: p(g)
y
q = p(g)
9 2 p(g)ap
pig) k
p(a
1
k;
plo) Mkia
k.
kj-1 16
ki 15
ki 3 40
1/
Figure 1: Depiction of our construction which allows Transformers to obtain positional attention heads using RoPE - zooming in on a single RoPE frequency for clarity. On the left we depict key and query vectors for each position where keys are all identical, and queries are just rotated version of the key, in way that matches one of RoPE's highest frequencies. The center depicts how keys get rotated by RoPE, making the key all other keys will lead to a smaller attention weight. On the right we show the resulting attention weights, resulting in this case in an off-diagonal positional attention. See Section for more details.
perfectly align with the query. Due the high frequency of the rotation,
Consequently, we find it interesting and important to understand how exactly the different rotation frequencies are used in LLMs.
information.
Contributions. In this work, we study in-depth empirically and theoretically how Transformers and in particular auto-regressive LLMs benefit from RoPE. We rely on the pretrained and open-source Gemma 7B (Gemma Team et al., 2024) model for our empirical analysis and show consistent results with 1 8B (Dubey et al., 2024) in the Appendix (Section C). We summarise our findings:
In Section 3, we argue against the common claim that RoPE is useful because encourages the decay of attention coefficients with distance. We provide theoretical and empirical evidence to support our claim.
Section 4, we propose new way to understand the usage of different frequencies in the queries and keys. We find that Gemma 7B largely prefers to the low frequencies of RoPE. The first and last layers instead show the most use of the high frequencies.
In Section 5, we show that the highest frequencies RoPE are cleverly used by Gemma 7B to construct special 'positional' attention heads (see Figure 1). We mathematically prove the 'robustness' of the construction.
In Section 6, we study how the low frequencies are used. We observe distinct 'bands' in the low frequencies of the queries and keys. We conjecture that Gemma 7B is using them as 'information channels'. We prove that these channels cannot be robust over long context.
In Section 6. propose
technique called p-RoPE that removes the lowest fre- quencies of RoPE to create robust semantic channels. We show not only that removing a percentage of the frequencies maintains the performance, but also improves it on 2 billion parameter models. We believe that our work explains why increasing the maximum RoPE wavelength helps with long-con mongcontext, e.g. as shown in Llama (Dubey et al., 2024).
BACKGROUND
We denote by x; E Rd the d-dimensional token embedding the i-th token. Query and key vectors take the form qi Wex, and k, Wxx, respectively, given query and key matrices W Q. Wx [dd The attention mechanism' performs the following computation:
exp (ai.) with aij (qi, kj)
Oij
Eie exp (a.))
(I)
We ignore here the d scaling factor introduced Vaswani al. (2017) for of notation.
2

---

Published as a conference paper at ICLR 2025
where & a kernel function which in our work takes the form of either simple dot product, i.c. No Positional Encoding (NoPE) (Haviv et al., 2022; Kazemnejad et al., 2024), or RoPE. We focus on the case in which the attention mechanism is causal, the most common type of attention used in LLMs today. In causal mechanism, we have that as = 0 when j > i and Zjjjss Qi.j = 1. We also note that we assume az, to be finite, as it is always the case in practice, such that 0 < ass < 1, when j < i. We call a the "activation' or 'logit', while we call as the "attention coefficient' between i and j. It is sometimes useful to view the attention coefficients in matrix-form, which in our specific case results in a lower triangular row-stochastic matrix.
We highlight an important special case for k which we call KNOPE, i.e. no positional encoding: ANOPE (q, kj) = q. kj, where q, denotes the tranpose of q. other words, NoPE the kernel function computes simply the dot product, providing no positional information to the Transformer. It has been shown that Transformers can still perform well, especially out of distribution, with NoPE (Kazemnejad et al., 2024). In particular, Kazemnejad et al. (2024) prove that the Transformers could in principle recover absolute positionall information through the causal mask; however, the proof relies on the universal approximation theorem, which we believe is practical limitation.
2.1 ROTARY POSITIONAL ENCODINGS (RoPE)
For simplicity of notation in this work assume that query are key vectors are d-dimensional, with d > 2 being an even number. We decompose queries and keys into 2-dimensional chunks
qi # q [k,++1] @k-1.. 9; (k) where 4 denotes direct sum (concatenation). In other
words, we denote by q, using analogous notation for the key vectors.
(b+ IR2 the k-th 2-dimensional chunk of the query vector of the i-th token,
RoPE considers a sequence of angles G = (9x= = 8- 2(-2((11))/)) 1)/d 1, ,.......,//]] d/2)². where 91 is the fastest rotating component at 1 radian per token and 9d/2 (d-22)/ x 0-1 the slowest rotating component at approximately 1/0 rotations per token. The parameter 0 is called the base wavelength, which by default is 10,000 (Su et al., 2024), although works have explored increasing it to, for instance, 500,000 (Xiong et al., 2023; Roziere et al., 2023; Dubey et al., 2024). We denote by p(gx) the matrix form of gu:
cos(gk) sin(gx)
p(gk)= =
(2)
sin(gN gk)
cos(gk) )
highlighting that p(gh)) is a 2-dimensional orthogonal transformation (rotation). One can view p(gg) as a 'unit rotation' by 9k radians. The RoPE technique amounts to the construction of a block- diagonal matrix R + P(gx)' E R where each 2 block the diagonal is rotation by a different frequency of RoPE. The R denotes in fact matrix exponentiation by an integer which is the position of x; 3. We can exploit nice property of rotation matrices, i.e. that p(g))* = p(gg)) to avoid the computation of the matrix power. As this matrix is block diagonal, computing R,q, means that the rotations only on 2-dimensional chunks of the query (or key), i.e.
R.q. DR-1.../2 )q This leads to the final formulation of KROPE:
KROPE (q:. ky) = (R'q:) (R'k,) = q. R Rock-
Σ (a(tt) plan)'kkp
(3)
k- k-l...d2
where we use the fact that (p(g)))) p(gx)' = p(9x)= = )- We highlight how the block diagonal structure of R allows to decompose the dot product into the sum of dot products of 2-dimensional chunks, with each key vector chunk rotated at a frequency dictated by gk-
2.2 RELATED WORKS
number of works have investigated how different modifications RoPE affect its generalisation. well-known method involves increasing the parameter 0 from the originally proposed 10.000 to
We denote the angles Ia instead 0g as we opt for more of group theoretic perspective of RoPE For clarity, and have nothing to do with /-1, but instead denote the positions i and j of the tokens in the sequence.
3

---

Published as a conference paper at ICLR 2025
Constant queries and keys
Gaussian queries and keys
1.0
1.0
0.5
0.5
Activation
Activation
0.0
0.0
-0.5
-0.5
-1.0
-1.0
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Relative Distance
Relative Distance
(a) RoPE applicd to constant queries and keys.
(b) RoPE applied to Gaussian queries and keys.
Figure 2: RoPE applied either (a) constant 'all-ones' queries and keys (b) queries and keys with entries sampled IID from a Gaussian. The decay of the activations is present when the queries and keys are constant all-ones vectors, but not when they are Gaussian random vectors.
larger value, such as 500,000 (Xiong et al., 2023; Roziere et al., 2023). Notably, this is also used in the recently released LLama 3 class of models (Dubey et al., 2024). The justification provided for this modification is that higher base wavelength means that the attention decay induced by RoPE will be slower, allowing for more robust learning over larger context. In this work, we challenge this common assumption and investigate why RoPE and modifications such as these are helpful.
In a different direction, works have pointed out that NoPE shows strong performance in out-of- distribution (OOD) settings when compared to RoPF, arguing that the causal mechanism is sufficient to learn positional information (Haviv et al., 2022: Kazemnejad et al., 2024)*. In this work, instead argue that NoPE and RoPE have complementary strengths and weaknesses and that for instance NoPE is unable to learn certain types of attention matrices present in Gemma 7B. Ruoss et al. (2023) propose to provide randomised positional information, showing that this helps boost OOD performance, claiming that this helps the model to learn over a longer range of relative distances. In our work (Appendix, Section B.3), we provide different explanation as to why this kind of process might be helpful from the point of view of the type of invariance it encourages.
Overall, we believe this work provides a different and perhaps more nuanced perspective on different works that tackle positional encodings. In spirit, this work similar works that aim understand LLMs from mechanistic perspective (Elhage et al., 2021; Olsson et al., 2022; Wang et al., 2022; Hanna et al., 2024) and from the representations they produce (Barbero et al., 2024; Veličković et al., 2024). In the Appendix (Section B), we provide additional discussions on related works.
3 DOES ROPE DECAY ACTIVATIONS WITH DISTANCE?
In this section, we argue against the common claim that RoPE is helpful because it helps to decay activations as the relative distance between tokens increases. Such claims often work under the assumption that queries and keys are for instance a vector with all entries equal to each other, which we believe an unrealistic oversimplification") Importantly. this claim was originally provided by the authors of RoPE (Su et al., 2024) as justification for the chosen structure of the encoding. Follow up works have used this claim justify modifications, such as increasing the base wavelength 0 to 500.000. We therefore find it important to point out cases in which this decay does not in fact occur.
We start by showing, in Proposition 3.1 that given any key, we can find a query such that RoPE is maximal for any chosen relative distance. This highlights the fact that RoPE provides Transformers
'Some works have argued that instead NoPE's extrapolation ability is still limited (Dong al., 2024; Wang al., 2024).
$ 'Intuitively, Su et al. (2024) ask the question of what happens, as we vary the relative distance, to the dot product between already aligned queries and keys. This indeed will lead to 'decay" with relative distance. However, this perspective ignores what happens to originally misaligned queries and keys, whose dot product increase with relative distance.
4

---

Published as a conference paper at ICLR 2025
with robust ways to attend to specific relative distances. In fact, will show, in Section 4, that this mechanism is what Gemma 7B uses to construct heads that attend to specific positions. We provide the proof in the Appendix (Section A.1).
Proposition 3.1 (RoPE can be maximal at arbitrary distance). Given any query and any relative distance Z, we can find a key k such that the softmax value is largest at distance with RoPE.
Next, in Proposition 3.2, we show that given queries and keys sampled independently from standard multivariate Gaussian, the expected value of the activations is Morcover, this is independent of the relative distance of the queries and keys implying that the expected value of the activations is independent of the relative distance when the queries and keys are sampled from a Gaussian. We provide the proof in the Appendix (Section A.
Proposition 3.2 (Gaussian queries and keys do not decay.). Let q.k N(O, I). Then, for any relative distance E Z, we have that:
['RR 0. q.k (O.T)
E
We showcase this by constructing a synthetic experiment in which we either set queries and keys as all-ones vectors, e.g. as done by Su et al. (2024); Xiong al. (2023), sample entries independently from a Gaussian, accounting for appropriate normalisations. Figure 2 shows the results. While there seems to be some form of decay of the activations as relative distance increases when the queries and keys are all-ones vectors (a) up to appropriate normalisation, this is clearly not the case when the queries and keys are instead random Gaussian vectors (b).
Summary of the Section: While RoPE helps to decay activations with relative distance in very specific conditions, this does not have happen. In fact, we will see the next section that this is something that Gemma 7B exploits to create specific attention patterns.
4 How ARE DIFFERENT FREQUENCIES USED?
In this section, we explore how different frequencies of RoPE are used. RoPE relies on a set of frequencies that take values 1,.... 0 (d-2)/d with the highest frequency varying by 1 radian per token, while the lowest being much more stable, varying at s 1/0 per token. As the angle between vectors affects the dot product, the contribution from the highest frequencies should behave similarly to random noise, i.e. small perturbation in the token sequence will give largely different activation contribution. A natural question is whether these frequencies are being used and, if so, how exactly they helpful?
To measure the usage of frequencies, we start by noting that by Gauchy-Schwarr the effect of the k-th frequency component on the activation a.j is upper bounded by the 2-norm of the query and key
components, i.c. Kg (k) k()) & lاq (k) Ilkk] (8) J. It is therefore natural to look at the mean 2- 2-norm
for each k* in Gemma 7B over long sequences. For Gemma 7B, note that k: I,..., 128, i.e. the hidden dimension is 128 x 2 = 256.
Figure 3 shows the results. We plot the average 2-norm each layer of each rope 'chunk' over a number of sequences, ordering them by frequency, with the intuition that the norm will be an upper bound for how much that frequency will impact the activation dot product. We emphasize that the mean is taken over all of the 16 heads at each layer. It is clear that learning has assigned much higher norm on average the lowest frequencies, meaning that they will likely influence the dot product the most. This seems to be true at each layer. Interestingly, there seems to be some high frequency usage present especially at the very first and last layers.
We believe this to be remarkable evidence showcasing how Gemma adapts to RoPE by preferring to use the lowest frequencies when computing attention activations. 6 the Appendix (Section E.2, Figure 14), for completeness, vectors. This highlights that this behaviour is a consequence of RoPE, where learning discovers that these medium to high frequency 'chunks' are not useful and hence pushes their norm to 0 so that
show that this type of distribution does not occur for the value
't remains under debate whether high attention scores imply meaningful preference (Bihal al., 2022).
5

---

Published as a conference paper at ICLR 2025
Query Irequency usegee per layor
frequency usage layer
Loos Frequticks
Mean norm
Mean norm
(a) Mean query norm distribution at each layer.
(b) Mcan key norm distribution at each layer.
Figure 2-norm plotted over 2-dimensional chunks of queries (a) and keys (b) for each layer in Gemma 7B, corresponding to different RoPE frequencies. A mean is taken over 10 different Shakespeare quotes and the 16 attention heads at each layer.
Query frequency usage (Layer
Key Erequency usage (Layer
Frspuantios
14
-2
Mean nomm
Moa0 borm
20
Lown Exaigenncios
-10
9 10 11 12 13 Attention Hoad
15
2 9 11 13 15
Attention Head
(a) Mean query norms at each attention head.
(b) Mean key norms at each attention head.
Figure 4: 2-norm plotted over 2-dimensional chunks of queries (a) and keys (b) for each attention head of the first layer in Gemma 7B, corresponding to different RoPE frequencies. A mean taken over 10 different Shakespeare quotes. We explain Section the high frequency behaviour Head 5 and Head 8.
their impact on the dot product minimal. Corresponding entries in value vectors, not being rotated, do suffer from the same pressure have their norm 0.
In Figure 4, we show the frequency usage of the 16 attention heads in the first layer. The heads that stand using the high frequencies, especially for the keys, Heads 5 and 8. We will show in the next section that these heads correspond to positional attention heads. We also highlight the sparse nature of the frequency usage, with the presence of 'high norm' bands, especially the lower frequencies. We highlight that this kind of pattern seems consistent with the observation that feed- forward layers act as sparse dictionary lookup tables (Geva al., 2021). In this context, believe that these bands are used to perform some kind of sparse query and key semantic matching.
Summary of the Section: We empirically showed that most of the RoPE usage in Gemma 7B occurs at the low frequencies. We also identified 'high frequency' heads and high norm bands.
HIGH FREQUENCIES: POSITIONAL ATTENTION
The highest frequencies in RoPE are interesting to study as their usefulness not immediately obvious. In particular, in the previous section, showed that Gemma 7B seems to largely avoid

---

Published as a conference paper at ICLR 2025
Attention pattern (Layer Head
Attention pattern (Layer Head
C6
Attention
10
Aztention
13
12
13
13
14
14
18
18
19
-0.0
19
-00
Tokea
Token
(a) diagonal head Gemma 7B.
(b) A previous-token head in Gemma
Figure 5: Examples of purely positional heads occurring in Gemma 7B, showcasing a diagonal head at the last layer (a) and a previous-toker tokenn head at the first layer (b).
Query frequency usage (Layer 27 Head 8)
Key trequency usage (Layer 27 Head 8)
High Foequnciis
100
NorTh
Nonta
Lan sesaanan
Luw Thoquetioo
Query frequency usage for Layer 27 Head head
(b) Key frequency usage for Layer 27 Head
Figure 6: Query and key frequency usage for diagonal head present the final layer, shown in Figure (a). The head mostly relies on the high frequencies.
them. In this section, we will study the cases in which certain attention heads mostly use the very highest frequencies, showing that these heads tend to display purely positional attention patterns. We will prove by construction that RoPE allows for arbitrarily sharp attention patterns of this type construction that Gemma 7B seems to be closely learning in practice. We will also prove that such heads cannot be constructed with NoPE. We believe the ability construct such heads is important when trying optimise for auto-regressive generation. These heads have also shown to be useful for structure generalisation (see, c.g., Figure in Ruoss al. (2023)).
In Figure 5, we show certain heads in Gemma 7B that specialise into purely positional attention patterns with Figure 5 (a) showing 'diagonal' attention pattern and 5 (b) 'previous-token' attention pattern. These heads seem to attend based purely on relative position, with no regards for semantics. is immediately clear how purely diagonal attention patterns are helpful, as these in principle behave like a residual connection. The reason why Gemma learns to construct such pattern outside the scope of this work, constitutes interesting learnt behaviour which requires further analysis perhaps training 'bug'. Figure 6, shows that for the diagonal head, Gemma seems to rely mostly on the very highest frequencies. We provide in the Appendix (Section E.2) more examples showing that this seems to be a repeated pattern throughout these positional heads. In fact, we found it remarkably consistent to identify positional heads, by looking at the usage of the highest frequencies. The heads and previously pointed in Figure in fact correspond to positional heads with attention head being purely diagonal and attention head previous-token head.

---

Published as a conference paper at ICLR 2025
Constructing robust positional heads. We now study theoretically the mechanism through which Gemma 7B learns to robustly construct positional attention patterns. We start by defining more formally diagonal and previous-token attention patterns. Importantly, our definition relies on being able to learn patterns to an arbitrary precision 6, given a maximum size N.
Definition 5.1. An attention head can learn a diagonal attention pattern if given an >> O and a fixed Similarly, for the previous-
size N. for all & N, we have 04,i > 1 6, implying that Lzzz-s token head, we must have Qi,i-1 1 for all > 0, implying that L 0i, E.
We prove in Proposition 5.2 that construction exists for NoPE that is able to learn such attention patterns. The proof relies on a counterexample involving token repetition and may be found in the Appendix (Section A.2).
Proposition 5.2. An attention head with NoPE cannot learn diagonal or off-diagonal pattern.
Instead, in Theorem 5.3, we show that an attention head with RoPE is able to learn such patterns. The construction relies on setting queries and keys equal to each-other and for the off-diagonal case, inverse-rotating them by a unit rotation for each RoPE chunk. The proof is provided in the Appendix (Section A.2).
Theorem 5.3. An atlention head with RoPE can learn diagonal or off-diagonal pattern.
In fact, Gemma 7B seems be learning a construction that is strikingly similar to our construction we provide evidence in the Appendix (Section E.1). We also find why the highest frequency rotations are most similar, as they are the ones which will affect the dot-product most rapidly. For the diagonal case, setting queries and keys equal means that the dot product (q. R R> will be maximal when j = and strictly smaller when j * The decrease in the dot product will depend on the angle between the rotated queries and keys. This angle will become dissimilar the fastest when using the highest frequency rotations. If ga/2 1/10000 for instance, then 9272 will misalign the queries and keys by a radian at distance of 10,000 tokens, while the highest frequency 91 will only need single token.
Summary of the Section: High frequencies in RoPE provide mechanism construct 'posi- tional' atlention patterns. NoPE is instead provably not able to construct such patterns.
Low FREQUENCIES: SEMANTIC ATTENTION
We now study how the low frequencies are being used. As observed in Section 4, the majority of the allocated norm (within the key and query embeddings) in Gemma 7B seems to be towards the lower frequencies. Furthermore, there seems to be a presence of distinct 'bands' in attention heads, especially in these low frequencies. In this section, we argue that the low frequencies are most useful to detect information related to token 'semantics' as they are the most invariant to token relative distance. The rotations are however not perfectly invariant over long relative distance, leading eventually to misalignment of the vectors also at the lowest frequencies.
In other words, the lowest frequencies useful as they are precisely the frequencies for which dot product is the least affected by the relative distance. In particular, we helieve this is why works such as LLama 3 have found it useful increase the base wavelength 500,000, meaning that the lowest frequencies rotate at roughly 1 1/500,000 radians per token. With a context length of 128k in LLama the standard wavelength of 10,000 will in fact complete 2.04 rotations, which might lead to poor generalisation due to tokens being erroneously misaligned simply due to the large context. We therefore deduce that with larger and larger contexts, the 9 base wavelength of RoPE will have also be increased accordingly
relationship that has been studied in depth (Xu et al., 2024).
In Figure 7 (a), we identify an interesting semantic head that makes tokens appearing after an apostrophe token attend to the apostrophe". We point out the qualitative difference when compared to the frequency usage in the positional heads from Section 5. This head interesting being it is semantic (detecting apostrophes), but also positional, only detecting apostrophe at the previous
For instance, for the following 4 tokens [BOS, I, m]. the attention head token m will attend to the apostrophe token'. '. For all other tokens will attend the ROS token.
8

---

Published as a conference paper at ICLR 2025
Attention pattern (Layer Heed 10)
Irequency usege (Layer Head
Elip: Breeuestces
14
-0.8
-0.8
Attention
10
Low Frequueedes
14
-0.2
-0.0
Tokenn
Token
(a) Semantic "apostrophc' attention head.
(b) Key frequency usage for the apostrophe head.
Figure 7: (a) Semantic attention head making tokens appearing after an apostrophe attend to it (b) Key frequency usage of same attention head, showing bands around low frequencies. The apostrophe tokens are positions 3, 9, and 15.
token. We see how Gemma 7B is using the highest frequencies, visibly active in Figure (b), to detect 'previous token' apostrophes. The low frequency dark band instead we believe is being used to attend robustly the BOS token when there is no apostrophe present at the previous token see the Appendix (Section E. 1) for details.
We now however show that the 'semantic bands' cannot be robust over large context. In particular, irrational rotations are 'dense' meaning that given long enough context, they rotate any arbitrary value. We use this to show that there exists relative distance for which the attention activation will be erroneously large. We prove the 2-dimensional case, i.e. single RoPE frequency, for simplicity. We believe that this still an informative case as we observe that these bands are often very distinct and focused single frequency. We provide the proof in the Appendix (Section A.3) and point out that the observation that NoPE is instead able to build this type of attention pattern is immediate.
Theorem 6.1. Let the hidden dimension be d = i.e. there is only single RoPE frequency gi- Consider sequence Xboa, .1..... XN with target token at some position Xa. Then for large enough N, the attention head with RoPE is unable to attend such token such that ai, for any <> O and any & & N.
6.1 TRUNCATING THE LOWEST FREQUENCIES RoPE
In the previous section and more formally in Theorem 6.1, claimed that RoPE lacks robust semantic channels. To adapt to this, as shown in Figure 3, Gemma 7B seems to prefer to use the lowest frequencies to construct high norm semantic bands. The key insight is that unfortunately these high norm bands are not robust when the context becomes very large. We believe that this is why works such Llama found helpful to increase the base wavelength of RoPE due to the larger context.
In this section, to investigate this further, we carry out an ablation in which we truncate the very lowest frequencies of RoPE, The intuition is that if newly found intuition correct, truncating the lowest frequencies should not harm performance. In fact, this allows RoPE to provide robust semantic channels that are distance agnostic. We call this modification RoROP with 0 & p being the fraction of RoPE 'kept'. We highlight two special edge cases: while the case p with RoPE. As such, the quantity p behaves like interpolant between the two, with higher values of being closer to RoPE and lower to NoPE. We provide additional details in the Appendix (Section D) and summarise the properties the three types encodings in Table 1.
coincides with NoPE,
We would like to note that R-RoPE is in spirit similar to the idea of increasing the wavelength of RoPE from 10,000 to 500,000, first proposed by Roziere et al. (2023) and later employed by LLama 3 (Dubey et al., 2024). The effect that increasing the wavelength has is that of increasing the amount of tokens required to destroy semantic information, leading to improved long-range performance. For wavelength that is large enough compared the context, the very lowest frequencies act

---

Published as a conference paper at ICLR 2025
Table 2: Validation perplexity comparison between NoPE, RoPE and p- RoRP.
Table I: A summary of the discussed theoreti- cally properties of RoPE, NoPE and p-RoPE.
Encoding
Wiki
FlanV2
NoPE
4.8594
6.6429
Encoding
NoPE
RoPE
p-RoPE
RoPE, 10k
4.4627
6.4429
RoPEg 500k
4.4485
6.4593
Positional
x
4.4592
6.4683
Semantic
x
/
0.75-RoPE Epartial
4.4537
6.4562
0.25-RoPE 0.75-RoPE
4.5302
6,5111
4.4414
6.4422
semantic channel, the position will have very little impact on the dot product. A summary of the properties between NoPE, RoPE, and p-RoPE is shown in Table 1.
We train Gemma 2B models from scratch on the Wiki and FlanV2 training datasets (see Section D for details), We benchmark the use of RoPE, NoPE, and p-RoPE. We set the base wavelength 0 to 10,000 and train and evaluate using the standard Gemma 8k token context. In Table 2, we show the perplexity on the validation set. We can see how truncating the lowest frequencies not only maintains the same performance, but even seems to improve the validation perplexity, supporting our claims regarding the low frequencies acting as non-robust semantic channels in standard RoPE. Truncating even more of the frequencies (0.25-RoPF) harms performance, but with results still significantly better than with NoPE. We find that p-RoPE also outperforms simply increasing the wavelength (0 = 500k) and beats removing the highest frequencies (denoted 0.75- 775-RoPE - validating our intuition that the lowest frequencies the ones that should be removed. We further show that p- RoPE seems to outperform RoPEpartial (see Appendix Section D for discussion).
Of course 8k context is not very large and improvements due to wavelength increase typically were observed on larger context of 32k (Xiong et al., 2023). We lack the resources to validate this larger context lengths, but believe this be very interesting future direction. We in fact expect that truncating some small percentage of the lowest frequency tokens to help with long-context generalisation by providing robust semantic channels that are independent of relative distance and can therefore generalise automatically to any context length. We comment in the Appendix (Section B.5) further on this.
Summary of the Section: Low frequencies in RoPE tend to be used by semantic attention heads, the rotation based on relative distance between tokens has minimal impact the dot product. Removing the rotation for a certain percent of low frequencies improves performance on Gemma 2B models.
7 CONCLUSION
In this work, we explored the behaviour of RoPE. We started by arguing that the use of RoPE does ensure that attention weights will decay with relative distance. This is provably the case if assume, for instance, that the keys and queries are Gaussian random vectors. Further, we provided construction that ensure the attention weight is maximal at any provided relative distance, regardless of magnitude, with these observations going against the typical intuitions used to justify RoPE.
We provided evidence that figuueerque bands are used build circuitry that leads to positional attention. We provided construction that ensures this behaviour and identified strikingly similar construction within the attention heads Gemma 7B. Furthermore we showed that NoPE is unable to do similar positional attention.
Finally we showed that low-frequencies are used for semantic attention, and argued that removing the RoPE rotation for these low- fwwffreuesccis can potentially improve the ability to attend semantically and in particular to generalize such type of attention to longer contexts. We verified this through an ablation study with Gemma 2B. We overall see work providing more nuanced understanding of RoPE and hope that it can lead to an improved understanding of how LLMs use positional encodings, ultimately leading performance gains especially over long contexts.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like to thank Anian Ruoss (Google DeepMind), Simon Osindero (Google Deep- Mind), and Constantin Kogler (University of Oxford) for their valuable comments and suggestions on this work.
REFERENCES
Joshua Ainslie, James Lee- Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint 2rXiv:2305..3333 13245, 2023.
Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João GM Araújo, Alex Vitvitskyi, Razvan Pascanu, and Petar Veličković. Transformers need glasses! information over- squashing in language tasks. arXiv preprint 24X::24400004666 2024.
Adrien Bibal, Rémi Cardon, David Alfter, Rodrigo Wilkens, Xiaoou Wang. Thomas François, and Patrick Watrin. Is attention explanation? an introduction to the debate. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3889-3900, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10. 1061166///0022..cc 269. URL ht / /ataa/aaannttooooooooooooo org 2022 acl- 269.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06744 067424 2022.
George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of control, signals and systems, 2(4))303-31 -34-314, 1989.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309. 16588, 2023.
Zican Dong, Junyi Li, Xin Men, Wayne Xin Zhao, Binghing Wang, Zhen Tian, Weipeng Chen, and Ji-Rong Wen. Exploring context window of large language models via decomposed positional vectors. arXiv preprint arXiv:2405. 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama herd of models. arXiv preprint wxXiv:2407.21788 2024.
Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, and Tanmoy Chakraborty. How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning. arXiv preprint arXiv:2402.18312 2024.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for 1lms. arXiv preprint arX1v:2310.01801, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprini arXiv:2403.08299 2024.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-fom forward layers are key-value memories. pp. 5184-54449999 5495, 01 2021. doi: 10. 1188653/11202111 1/28633/ www.lp-main.....m
Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, interpretable attention heads in the wild. arXiv preprint arXiv:23/2.09230, 2023.

---

Published as a conference paper at ICLR 2025
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than': Inter- preting mathematical abilities in a pre- vee-trained language model. Advances in Neural Information Processing Systems, 36, 2024.
Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. arXiv preprint vrXiv:2203.666:3 2022.
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024.
Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. Lower perplexity is not always human-like. arXiv preprint arXiv:2 106.0/299 2021.
Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. arXiv preprint arXiv:2307.098
2023.
Aixin Liu, Bei Feng, Bin Wang. Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: strong, economical, and efficient mixture-of- experts language model. arXiv preprint arXiv:2405.0m 04434, 2024.
Introducing Llama. 3.1: Our most capable models date. Meta, 2024.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 20.2230113300 13688, 2023.
Howard Masur and Serge Tabachnikov. Rational billiards and flat structures. In Handbook of dynamical systems, volume 1, pp. 1015-1089, Elsevier, 2002.
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit component reuse across tasks in transformer language models. arXiv preprint arXiv:2310.08744, 2023.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5()):00024-001, 2020.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint ar arXiv:2209. 1895,, 2022.
Olir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2 12409, 2021.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. Randomized positional encodings boost length generalization of transformers. arXiv preprint 23.Xiv:335./6844 2023.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you (nips, 2017. arXiv preprint 17X1v:1706.03362, 10:5011455251160013533 2017.
Petar Veličković, Christos Perivolaropoulos, Federico Barbero, and Razvan Pascanu. softmax is not enough (for sharp out-of-distributiin)) arXiv preprint arXiv:2410.00 104,0/00,000 2024.
Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang, and Xiaoling Wang. Length generalization of causal transformers without position encoding. arXiv preprint arXiv:2404. /2224, 2024.
12

---

Published as a conference paper at ICLR 2025
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593 2022.
Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality. arXiv preprint arXiv:2404. 15574, 2024.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang. Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint (tt1:2309./603330 2023.
Mingyu Xu, Xin Men, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, et al. Base of rope bounds context length. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
A PROOFS
In this section we provide the omitted proofs for the theoretical statements in the paper. We also provide more detailed discussions on the constructions and ideas behind the proofs. In Section A.1 we provide proofs for Section 3, in Section A.2 we provide proofs for Section A.2, and in Section 6 we provide proofs for Section A.3.
A.1 DOES ROPE DECAY WITH DISTANCE?
In some of our proofs, use known results irrational rotations. We point to the literature for detailed overview of these concepts (Masur & Tabachnikov, 2002), although the results we leverage in our work are elementary. The first statement is about the "uniqueness" of irrational rotations.
Lemma A.1. Consider y E Q with g + 0 and n & Then, ng 0 (mod 2xx) only when n 0. In particular, this also holds if g is algebraic.
We now prove that given a non-zero query q and relative distance r, we can find key k such that the activation is maximal at distance r. We note that the non zero query assumption is minor as zero query would imply with RoPE perfectly uniform attention as all activations would be equal to each other. The proof is by construction and involves rotating each (8) chunk by p(gx)= We then apply Lemma A.1 to show that the relative distance r will give the maximum activation.
Given any (non-zery query q and any relative distance Z, we can find a key such that the softmax value is largest at distance with RoPE.
Proposition 3.1.
Proof. Consider distance r, a query q = v/ that is non zero by assumption, and key such that (k) p(gg))" (bb(() or equivalently k Rrab. Assume that the query is at position i and the key at position & i, we compute:
9. R ky 26 (R(-))+
Σ
(20(e))) T p(ox) (i-))r-
k- k-1..d/2
/2scal//2 cos ((j + r)gh)
Σ k=1..d/22
We now note that the maximum will only be achieved, by Lemma A. when j j- i= = -r, such that all cos ((j i + r)ga) cos(O) 1, completing the proof. We note the minus sign coming from our convention that j & i, implying that i & O.
We will later show that Gemma 7B seems to be learning a construction similar to the one above in practice in Section F. We next show that given queries and keys sampled from standard
13

---

Published as a conference paper at ICLR 2025
multivariate Gaussian, their expectation will be 0. This follows from the fact that the standard multivariate Gaussian is invariant to rotations (isotropic).
Proposition 3.2. Let q, k (O I). Then, for any relative distance E E, have that:
IE q.-~N(O.))
q R'K O.
Proof. We will use the well-known fact that if X [M, Σ) (multivariate) Gaussian random variable, then the linear transformation Y AX + b gives another Gaussian random variable such that Y (Au A,AAA We start by computing:
Σ (q p(gx) k(())
[q RRk]
q.k-V(0.1)
V.k.V(
Let (*) ((9)))(4) As k()) 1O, we have that E[K p(g)) 0 0 and V[k p(gk) I as p(gs) is orthogonal. This implies that also (k) is standard Gaussian random variable, i.c. (M) ~ (O, We proceed with the change of variables:
Σ (q())
)(())
p(gx (k(})
Σ
T [g)()))
k(())
E q.-.((0.1)
E k.o.)
E
q T
q No.1)
Σ
[4:]
E
ki
0.
# q;,1(0,11
k,,110
A.2 POSITIONAL ATTENTION PATTERNS.
We focus on the proofs regarding positional attention. We start by showing that NoPE cannot learn diagonal or off- off-diagonal attention. The proof is by counter-example, using a sequence that has repeated tokens.
Proposition 5.2. An attention head with NoPE cannot learn diagonal or off-diagonal pattern.
Proof. We start by proving the diagonal head claim. Recall that a diagonal attention head satisfies Og,i for any 0.
Consider the sequence [Xtos, X1, x1] that has a repeated token. We have that as,3 <qs, ks) (qs, k2) = 23,2. We compute the attention coefficient 033,3
exp(a3,3)
0r3,3
< 2'
exp(as,1) + 2 exp(as.3)
exp(as.1 a3.3) +
which contradicts the requirement that 03.3 > for any >> 0.
We now focus on the second claim. Recall that an off diagonal head satisfies 04,1-1 e for any 0. The same example contradicts this condition, as we have that 03,2 J.
We instead show that RoPE is able to leam such attention patterns robustly. The construction is related to the construction used in the proof of Proposition 3.1.
Proposition 5.3. An aftention head with RoPE learn diagonal off-diagonal pattern.
14

---

Published as a conference paper at ICLR 2025
Proof. We start by proving the diagonal case. The construction involves setting all queries and keys equal, ie. q; = k; = & with &A not zero. To simplity notation and book-keeping, we focus on the case in which the hidden dimension is only d 2, i.. only acts through a single rotation. We will then explain how to generalise to 2. We call the rotation ! and its matrix representation p(g).
We now note without RoPE that:
(qi ky) ||qill [|yl| cos((..))
||
with R., the angle between qi ky which is 0 as we have by construction that qi k, = v. When using RoPE, the dot product becomes:
a = (o(g)' qi p(g))k))
= (l(()) 'ql)) [|((g)) k5]l cos ((j - i) g + 0z,))
| cos ((j 9)
where we use the fact that ||(]))(x|| ||x|| as p(g)* is isometry. Due to Lemma A.I, have that () g = 0 (mod 2r) only when j = i. This implies that a = =| and a.j < ag. We now compute axsss
exp(ar.)
Ciji Ekes (sp(ar,)) + exp(as.:)
exp ||061
Eeer exp |10b11 cos ((k- - 1) 9) exp (11b611)
1 + Σxer exp || cos ((k i) g) 118b113)
+ Exer exp [| (cos ((k i) g) -1)
We observe that cos ((k- - 1) g) < 1 for * i, implying that || (cos ((k: - 1) - 1) < 0. In particular, we have that:
sup + 00 1 + Ek exp (12511 (cos ((1s - i) g) 11)
1,
meaning that indeed our construction satisfies og > - e for any e > O, with function of ||/b//2
We now focus the off-diagonal case. Here set all queries qi s/s and all keys k; p(g)ub Φb for 2b non-zero. We now observe:
a (plo)' qi, (()))
(plg)' p(g) 1-1 -l
<p(g)' p(g)'v)
||o())' cos ((i i) g)
1|
The same reasoning for the diagonal case then follows. For the high-dimensional case, it is sufficient (k))
take k; R&p equivalently k.{a)
15

---

Published as a conference paper at ICLR 2025
In practice, Gemma 7B learns construction very similar to that used in Theorem 5.3, show in Section E. Further, as softmax tends to be rather 'sharp', , the norm does not in practice have to become very large for the construction to be robust. This can be seen for example in Figure 6, where the chunk norms are not above 10, but the attention matrix allocates robustly attention on the diagonal.
will
Can NoPE still learn positional patterns? We clarify that our results prove that NoPE cannot learn diagonal or previous-token patterns given a head operating in isolation. This is interesting because we found for instance that Gemma 7B learnt such patterns already at the first layer. Our proof shows that this would be impossible to learn robustly with NoPE. It is interesting to study the expressivity of an attention head operating in isolation because it aims to capture the "expressive efficiency' of single attention head. Of course, here make no claims that a sequence of attention heads cannot learn such patterns. In fact, Kazemnejad et al. (2024), prove that this should be possible, with application of the Universal Approximation Theorem (Cybenko, 1989).
A.3 SEMANTIC ATTENTION.
The next statement is a useful result on irrational rotations, namely that they are dense in [0, ] Intuitively, this means that given an irrational rotation g. we can find an integer such that ng (mod 2x) is arbitrarily close to any g' [0, 2x]. The statement is commonly shown via pigeon- hole argument.
Lemma A.2. The subset {ng (mod 2m) Z} = [0, 2m] is dense in [0, 2n
We now show simple, but useful Lemma regarding a required condition for "sharpness', namely that the activation entering the softmax has to be the largest to have an attention coefficient larger than 1/2.
Lemma A.3. Let E Ih be sequence of activations entering softmax such that a softmax (a) IR", Denote by ai, og the i-th activation and attention coefficient, respectively. If a; aj for some j. then 04 & 2-
Proof. We have that a; exp(as) 2 with Z Σx exp (ag). We first note that if a; aj, then 0;- ajj as Z is constant and exp monotonically increasing. Now assume by contradiction that o; 1. we then have that 03 > a; > 1. implying that a; + at > 1. This violates the sum-to-one constraint of the softmax function, implying that a; < if < aj for some j.
We are now ready to show the result regarding the instability of RoPE. The intuition of the proof is that depending on the position in the sequence, the rotation can make the dot product positive or negative. This means that we can always find permutations, given a long enough sequence, that will make the activation of the desired element smaller than another element.
Let the hidden dimension be d 2, i.e. there is only a single RoPE frequency 91- Consider sequence Xins, X1...., Xx with target loken at some position X.. Then for large enough N, the attention head with RoPE is unable to attend to such a token such that Ci,n > - é for any <> O and any 1 E n E.
Theorem 6.1.
Proof. Assume the desired element is at position n., that the head is correctly attending to it such that Oi,n 1 e for all e and > n, and that N is large enough. By Lemma A.3, the fact that ain > 1 - e implies that a > at.j for all j. As a has to be the largest element, we have two cases. Either a & 0 is non- positive, implying that a 0 for all j # kt or a > is positive.
We start with the first case, where a < O. Consider an element at position k * n. We have that:
a = ||q ||k/| cos (0a. + 91 (R- i)):
where 0g.2 the angle between qu and kg. As a < O, we must have that 0g,2 + gi (k - i) (mod 2x) (g, 3x) (making the cosine negative). is sufficient to swap the element position with some element k' such that 0g + 91 (K' i) (mod 2T) e (0, 7) o (3m 2nr), which is always possible for large enough N by Lemma A.2. This swap would make a O. which by Lemma A.3, implies that now Oijk & 3-
16

---

Published as a conference paper at ICLR 2025
In other words, above we have shown that if an attention head can attend successfully at certain position on a sequence that is long enough, certain swaps of the tokens in the sequence guarantee that this attention head would attend to the wrong clement. In particular, if all activations negative, it is sufficient to perform a swap that makes a non-target element positive.
We now focus the case a 0. The principle is the same, but need apply two swaps. The first swaps n with & * n, such that for the target token now at position k we have that a < 0 after the swap. If after the swap a is not the largest element anymore, we done. If not, we need to perform second swap with k' * k" şe k, which swaps an element at position k' with k" such that now a 0. This forces the target element to not be the largest anymore. We can then apply again Lemma A.3 to complete the proof.
In other words, if some activations are positive, we need perform two swaps. First we swap the target activation such that it becomes negative. Then, we are either done or we need to perform another swap making a non-target activation positive. These two cases imply the desired statement.
B ADDITIONAL DISCUSSIONS
In this section, we provide additional discussions omitted from the main part of the manuscript. We start by providing, in Section B.1 a more thorough discussion on related works. In Section B.2, we comment on how believe work and findings relate to positional encodings other than RoPE. In Section B.3, we comment on the work of Ruoss et al. (2023), which proposes a process that randomises the positions of tokens.
B. ADDITIONAL RELATED WORKS
Our work is related to the field of mechanistic interpretability (Olah et al., 2020; Elhage et al., 2021) whose focus is that of 'reverse engineering' models and more specifically in our case, LLMs. A number of works have studied and identified interesting behaviours of attention heads. Notable examples are induction heads (Olsson al., 2022), chain-of-thought heads (Dutta et al., 2024), indirect object identification heads (Wang et al., 2022), multiple-choice heads (Lieberum et al., 2023), content- gatherer heads (Merullo et al., 2023), successor heads (Gould et al., 2023), attention sinks (Darcet al., 2023), comparator heads (Hanna et al., 2024), and retrieval heads (Wu et al., 2024). Works have also exploited known patterns to speed-up inference time (Ge et al., 2023). In our work, we identify an "apostrophe head' and 'positional heads' as far as we aware we the first identify the apostrophe head, while the diagonal and previous-token heads are instead likely to be already known.
While the identification of circuits and different types of attention heads constitutes interesting direction, our work is conceptually different as it focuses on answering the question how does RoPE help to construct such heads? We believe this to be rather new type of mechanistic interpretability approach as it is concerned with a 'lower level' of interpretability, in which one studies how the Transformer implements specific heads. In other words, we are more interested in the efficiency of the implementation rather than only limiting ourselves to understanding the high-level functionality. We believe that studies such as this one are important to better our understanding of the contributions of different components of LLMs, to ultimately improve them.
B.2 OTHER POSITIONAL ENCODINGS
In our work, we focused on RoPE as we believe it is one of the most relevant positional encodings used today, being used for example by Gemma and Ilama 3 models. We also find RoPE to have particularly interesting geometric structure, that ties nicely with the dot product. These two reasons motivated our in-depth study of the method.
Regardless, we believe that our approach and findings can be transferred more broadly. First of all, we would like to point out the strong similarity between RoPE and the sinusoidal absolute encodings (APE) proposed by Vaswani et al. (2017), with both encoding position through range of frequencies of sines and cosines. The fact that APE is added to the queries and keys and not applied as a linear map is significant difference and would be interesting to explore in more detail. We suspect
17

---

Published as a conference paper at ICLR 2025
Constant queries and keys
Gaussian queries and keys
1.0
0.5
0.5
Activation
Activation
0.0
0.0
-0.5
-0.5
-1.0
-1.0
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Relative Distance
Relative Distance
RoPE
Random RoPE 20=)
RoPE
Random RoPE (L=20la)
Random RoPT (L=10k)
Random RoPE (L=50k)
Rendom RoPE (L=10k)
Random RoPE (L=50k)
(a) RoPE and Random RoPE applied to constant queries and keys.
(b) RoPE and Random RoPE applied to Gaussian queries and keys.
Figure 8: RoPE and Random RoPE (Ruoss et al., 2023) applied to either (a) constant 'all-ones' queries and keys or (b) queries and keys with entries sampled IID from Gaussian. We vary the L parameter, showing that in (a), this tends to increase the decay rate.
however that our findings relating to the frequency usage be somewhat transferable, with the lower frequencies being used to carry semantics and the high frequencies providing method to encode positional attention patterns.
A popular method that is less similar to RoPE is Alibi (Press et al., 2021). We suspect that prior to this work, the common belief would be that Alibi and RoPE behaved in an analogous way, by decaying attention coefficients with distance. This is clearly true with Alibi. We however showed in this work that this not the case with RoPE. Analysing Alibi in a way similar to this work would therefore be an interesting future direction of research to better understanding what kind of patterns these different positional encodings encourage. Overall, we hope that the understanding gained through our work will help the community to better classify different positional encodings.
B.3 RANDOMISED POSITIONAL ENCODINGS
A recent work by Ruoss et al. (2023) proposes to 'randomise' the positions of RoPE, showing that this process helps to boost long-context generalisation performance. In particular, they propose given some maximum training context N. to choose some L > N, and to then sample without replacement from 1... N elements, ordering them from smallest to largest, providing them as the positions during training and iteratively remmanl The motivation is that this process would allow the model to train using larger positions, helping with diitrr-distiiittionn generalisation to long sequences. We believe instead that this technique is instead helpful for more subtle reason.
Providing during training random positions invites the model to learn some soft form of invariance to relative distance. The model has to fact learn produce the same output given different sampled positions when optimising the training loss. While pure invariance with RoPE cannot be achieved, the best the model can do when using RoPE is rely on the very lowest frequencies to minimise the training loss. In other words, randomising the positions according to Ruoss et al. (2023) pushes the model use the lowest frequencies. Testing this is laborious and outside the scope of our work, but we believe this to be an interesting future direction.
Furthermore, this randomised process has another side-effect, namely it increases the average relative distance. The effect of this increase in relative distance is that it will increase the decay of the activations. Figure (a), we show the effect increasing the sampling upper bound on a constant vector of all-ones (up to normalisation). We see how increasing L makes the decay occur faster, but that all curves seem to finally converge around 0. Instead in (b), we show that this does not affect the activation patterns when sampling from a Gaussian. This serves counter the original claim that this process helps as the model will see larger relative distances. In fact, we see that as we increase the relative distance, the effect the activations becomes and more indistinguishable.
18

---

Published as a conference paper at ICLR 2025
B.4 ADDITIONAL RESULTS ON SYNTHETIC ACTIVATION DECAY
We provide in Figure 9 further ablation where we show that two random Gaussian random vectors do not show decay properties. The difference between the plots from Figure 2 (b) is that here we do not sample a different Gaussian RV for each position in the sequence. but keep it constant. This provides supporting evidence for the fact that the decay visible Figure (a) is not only for constant queries and keys, but also that the queries and keys need to be 'perfectly aligned'.
Constant Gaussian queries and keys 1.0
0.5
Activation
0.0
-0.5
-1.0
1000
2000
3000
4000
5000
Relative Distance
RoPE (10k wavelength)
RoPE (500k wavelength)
Figure 9: RoPE applied to 'constant' Gaussian random vectors. We sample a single Gaussian RV as key and a single Gaussian RV query. We then repeat these vectors form sequence of length n. There no clear evidence of decay.
B.5 LIMITATIONS OF OUR WORK
As with other mechanistic interpretations. it is not clear that our description covers the entire spectrum of how LLMs use the RoPE encoding. While, the magnitude of the key entries corresponding to middle band of frequencies is considerably smaller average, it is not O, and therefore it could still play a role. Our ablation removing the low frequencies, while promising and helping to significantly validate our claims, could be improved with larger scale experimentation to asses the improvements specifically on large contexts which we believe to an important application of this work. Of course, such experimentation would be much more resource intensive.
Our current evaluation of R-RoPE involves the training of 2B parameter Gemma models from scratch and checking the perplexity a validation set. Of course, lower perplexity does not necessarily mean better downstream performance (Kuribayashi et al., 2021). The goal of our work was to better understand RoPE. We found that our findings naturally lead p-RoPE and we were happy to see that it showed strong initial signal. We leave to future work the more careful evaluation of p- RoPP as we believe this to be outside the scope of this paper.
C LLAMA3.1 8B
In this section, we demonstrate that similar patterns also occur in the released LLama3. 8B model (Dubey et al., 2024; Llama, 2024). The Gemma and Llama models we study of course similar architecturally. but have significantly different design choices. For instance, Llama 8B uses Grouped Query Attention (GQA) (Ainslie al., 2023) and uses higher wavelength parameter of 500k. Further, the Llama3.1 models are trained with a much larger context of 128k tokens.
Figure 10 shows the results of the frequency analysis we report in Figure We see that the queries (a) and keys (b) both seem to have higher activation towards the slower frequencies. Interestingly, we find that the start of the high norm bands occurs at approximately 500,000 -0/664 x 0.0001, which is roughly close where they appear in the Gemma model with the much lower wavelength
19

---

Published as a conference paper at ICLR 2025
LLama3.1 BB Query frequency usage per layer
LLama3.1 Key frequency usage per layer
Higgh Progpatiooss
3.5
High Proquancios
Moan norhh
Mean norm
-2.0
LIM Prequenties
Frong anogaandiis
-1.0
-0.
Layer
Mean query distribution each layer.
(b) Mean key norm distribution each layer.
LLama3.1 8D Value frequency layer
Proqucnides
0.7
Lox Ercquentes
-0.3
-0.3
-0.1
15 17 Tayer
25
29
(c) Value key norm distribution at cach layer.
Figure 10: Norm plotted 2-dimensional chunks of Queries (a), keys (b), and (c) Values for each layer in Llama 3.1 8B, corresponding to different RoPE frequencies. The same process of frequency display does not show any clear pattern each layer RoPE applied the values. This supports the claim that the patterns shown in the queries and keys are due to RoPE. The queries and keys seem prefer use the lower frequencies.
parameter at 10,000-0.5 x 0.0006. This suggests that the effect of increasing this wavelength is to effectively provide to the model more 'slow enough' frequencies that seem to useful for the model to use. As in the Gemma model, the value vectors do not have such distribution of norms also in the Llama model.
We further provide evidence that the diagonal head pattern also occurs in Llama3. 1 8B. In Figure 11, we show the frequency usage of 4 attention heads located at the 32nd (last) layer in the 4th head group. We highlight the 3rd head of this group that seems to use the high frequencies much more prominently. This head is fact a diagonal attention head.
D SUPPLEMENTARY P-ROPE EXPERIMENTAL DETAILS AND RESULTS
We start by highlighting that mechanisms similar to R-RoPE have been discovered although we were unaware of this at the time of writing this work. An example is this GitHub Issue that suggests partial rotary embeddings :: https: htpp:///giii htt///gttbbbt///////////tttthttt com/ 1 lucl. idda n x-trans formers//// issues/ Other works have also found RoPEpartal to be useful (Black et al., 2022; Liu et al., 2024).
We believe RoPEparial and p-RoPE to be similar nature as they provide parts of the queries and keys that void of rotations. However, given study, believe p-RoPE be better motivated.
20

---

Published as a conference paper at ICLR 2025
LLama3.1 Query frequency usage (Layer 32, Head Group
Hight Frequencies
12
10
Mean morm
88
Low Frequencces
2
Attention Head
Figure 11: Frequency usage the query vectors LLama3.1 1 8B. We focus on the 32nd (last) layer and the 4th head group. Attention head 3, which has clearly more dominant high frequency usage, is a diagonal attention head.
Importantly, our study also helps to provide some insights into why a method such as partial-RoPE might be useful, which might of independent interest regardless. We found in our experiments p-RoPE to outperform RoPEpariia
In our additional experiments we train Gemma 2B from scratch separately Wikipedia (Wiki) and F1anV2.
datasets: English
Wiki is dataset based on English Wikipedia articles, built from the Wikipedia dump. Each sample contains the contents of a full Wikipedia article, with processing done to strip markdown and unwanted sections. The dataset available at: nttps: : /www. tensor flow. org/ / da t sets catalog / wik wikipedia There are 6.672,479 documents and the total size is of 19.88 GiB. We held out 10% of the data randomly for the validation split.
F2 was introduced by Longpre et al. (2023). It is dataset for instruction tuning which combines collections from FLAN, P3/TO, and Natural Instructions with dialog, program synthesis, and complex reasoning tasks. The dataset contains 15,000,000 samples.
In all experiments we train for 10,000 steps using a batch size of 512 and a sequence length of 8,192. We fix the wavelength to the standard value of 10,000. We use the standard Gemma 2B architecture (Gemma Team al., 2024), For P-RoPE, we use the algorithm apply_p hoppll___rrpee described below.
def apply_p_rope
inputs: jax.Array, positions: jax Array, head_dim: int,
# [B, L] $ [B, L]
max nax_wavelength: int MAX WAVELENGTH,
rope_percentage: float 1.0, jax.Array:
"""Applies R-RoPE. ""*
rope_angles int prope_percentage head dim // nope_angles head dim // 2 rope_angles
fraction 2. arange (0, rope_angles) / head_dim
timescale wax_wavele wtrpenngtthfaaaoooooooo.ooo fraction
timescale inp. ( max_ fox_wavelengthh- f
(0, nope nope_ saoe__angee),
mode=' constant',
constant_values= (0, jnp.ing
21

---

Published as a conference paper at ICLR 2025
sinusoid inp
positions [..., imp.newaxist / timescale newaxis, jnp. newaxis, :]
sinusoid inp sinusoid_inp(.... newaxis,, :]
sin inp. sin sin(sinusoid_inp)
cos = jnp. cos (simusoid_inp)
(inputs, 2, axis=-1) first_part first_half cos second_half sin second_part second_half cos + first_half sin out = inp. ((frrettttte(ittttaieeeeee return out astype (inputs dtype)
first_half, second_half
second_part], axis--1)
E SUPPLEMENTARY GEMMA 7B RESULTS
We provide supplementary results for our analysis of Gemma 7B. We start by providing, in Section E.I, evidence supporting our claims regarding the types of constructions being learnt by Gemma We end by providing in Section E.2, the query, key, and attention pattern plots for all of the attention heads we study in the main part the manuscript.
E.1 CONSTRUCTIONS LEARNT BY GEMMA 7B
We explore the various constructions discussed in the main part of the manuscript. As small aside for some geometric intuition, we note that the queries and keys in Gemma 7B are 256-dimensional. If they were randomly sampled from Gaussian, their dot product should be very close to O. This follows from the fact that high-dimensional Gaussian random vectors are close to orthogonal.
Diagonal heads. We start by focusing diagonal attention heads. In particular, focus the diagonal attention heads from Figure 14 and Figure 15. The hypothesis we want to verify is that the attention heads learning construction from Theorem 5.3.
queries and keys approximately equal each-other, following the
To measure how close the queries and keys are, we leverage the Schwarz inequality, In particular, we know that:
9 R
R--K,
| [|y/|| cos (Os.) + gi.)) &
| |kk]|,
Vd
meaning that the quantity ||q: [|k]] [| serves as an upper-bound, achieved when the queries and keys are perfectly aligned. We highlight that is important to introduce the normalisation factor 1/vd to get meaningful bound. We then compare the activations (logits) to this upper bound. If the queries and keys are indeed aligned, the logits should be close to the upper bound. In Table 3 and Table 4, we showcase that this seems to be the case for the diagonal attention heads occurring at Layer 1 (Head 5) and Layer 27 (Head 8). The previous-token activations instead in comparison decay significantly in these heads.
Previous-token head. We perform a similar measurement for the previous token head (Figure 16). An analogous situation occurs in which the previous-token activations seem as seen in Table 5. We find that in comparison to the
much closer to
the upper bound given by Cauchy-Schwarl diagonal heads, the gap to the upper bound larger. This expected, as the head has to learn to model R. which is more complicated block-diagonal rotation matrix. Nevertheless, some tokens in particular are surprisingly aligned, supporting our claims.
Apostrophe head. We now analyse the construction of the 'apostrophe attention head' see Section 6. In particular, we identified this head as one that attends to a previous setkoouu--okenn apostrophe it exists and to the BOS token, otherwise. We are particularly interested in understanding what the
22

---

Published as a conference paper at ICLR 2025
Token
BOS
3
4
7
Upper Bound
3.74
182.34
203.65
483.97
141.48
130.71
168.52
122.31
Diagonal
3.25
119.60
156.93
383.00
98.47
92.15
137.10
90.71
Previous-token
N/A
-8.45
76.67
82.13
29.01
45.33
73.04
55.25
Table 3: Activations for diagonal head (Layer Head 5). The upper bound is given by Cauchy- Schwarz. The diagonal activations are close to the upper-bounde meaning that the head setting queries and keys to be very aligned to each other. The previous-token activations instead in comparison decay significantly.
Token
BOS
2
4
5
6
Upper Bound
7.91
64.42
112.43
51.97
52.69
67.10
116.41
67.37
Diagonal
6.05
55.84
96.52
45.75
44.00
53.83
105.47
57.78
Previous-token
N/A
-9.63
32.51
29.49
25.96
20.08
19.95
6.07
Table 4: Activations for a diagonal head (Layer 27 Head 8). The upper bound is given by Cauchy- Schwarz. The diagonal activations are close to the upper- upper-bound, meaning that the head setting queries and keys to be very aligned to each other.
Token
BOS
2
3
4
5
6
7
Upper Bound
3,85
76.54
79.13
102.41
61.34
54.54
56,62
57.2
Diagonal
-1.58
3.29
5.21
-1.46
4.23
4.61
2.47
3.08
Previous-token
N/A
31.04
35.31
18.91
19.72
18.96
17.46
13.67
Table 5: Activations for previous-token head (Layer Head 8). The upper bound is given by Cauchy-Schward While the diagonal activations seem to be close to 0, the previous-token activations significantly larger and closer the upper-bound.
23

---

Published as a conference paper at ICLR 2025
low frequency band is being used for and how it acts semantic channel. We will show in this section that this semantic channel is being used to provide the BOS attention part of the pattern. In particular, this band is very positive when the token being attended to is the BOS token, and very negative otherwise. corresponding the BOS token has activations of value 2, while the apostrophe tokens the off- diagonal token have attention values of R 7. The other values instead range from 2 to the lowest of -22 with all values being negative. These activations show how the head operates. It will default to attending to the BOS token, but will instead be dominated by an apostrophe token if it appears on the off off-diagonal as it will have a more positive activation.
Figure 12, we show the raw activations of the attention head. The column
We now study the contribution from the low frequency, clearly visible in Figure 17 in the queries and keys. The semantic channel occurs at f119 = 10,000 2011--(1111111111 x 0.0002 - one of the lowest frequencies
of the 128 frequencies (d 256) in Gemma 7B. As the context length of 8k, we get that the maximal rotation available is of 8000g119 & 1.64 2 radians, meaning that this channel is relatively stable amounting to quarter of full rotation, even the largest context lengths. By Gauchy-Schwarg as the norms corresponding to these frequencies are very large on average compared to the other frequencies, we expect this to 'dominate' the dot-product.
The queries corresponding to this frequency for the non BOS tokens are all set approximately
the same vector Anotsos R [-4.1, 11.3]T and similarly the keys k(119) [11.2. -3.5
(119)
Instead, we have that 9sos
(119) 10.9.- and (119) [-,, 1.3 We then note that
(119)) 24.9. other words, this channel con- tributes very negative activation for tokens attending non-BOS tokens, and very positive acti- vations to tokens attending to the BOS token. As the the underlying frequencies are very low, this mechanism will be stable given the limited context length of Gemma 7B. We however expect this mechanism to break over long enough sequences, in accordance with Theorem 6.1. This provides evidence supporting claims that the lowest frequencies used semantic channels, this kind of behaviour will be most robust exactly for those channels that provide the lowest frequencies.
(111) ((119) x -5.5, while (RROBOS
Activation pattern (Layer Head 10)
2
3.0
2.5
5
Activation
Token
2.5
10
12
5500
77.5
15
-10.0
-1.55
11
12
13
14
15
16
17
18
19
Token
Figure 12: Activation pattern of the 'apostrophe head' in Gemma 7B. The upper triangular mask is artificially set to the value of - 10 for clarity of the visualisation, but in reality takes a value of & 1000 Token is the BOS token. The 3 darkest activations on the off-diagonal correspond to tokens after apostrophe token attending to the apostrophe token.
E.2 ANALYSIS OF PATTERNS IN DIFFERENT ATTENTION HEADS
24

---

Published as a conference paper at ICLR 2025
Query frequency usage per layer
Key frequency usage per layer
High Faquuecios
High Prequencies
Moan nortn
Moan norm
Lawe Froeg woogannciio
Lawe Preegaancioo
25
13
25
27
Tayer
Tayer
(a) Mean query norm distribution at each layer.
(b) Mean key norm distribution each layer.
Value frequency usage
14
-0.8
Mean
0.66
-0.4
11
13
15 Layer
17
19
21
23
25
27
(c) Value key norm distribution at each layer.
Figure 13: Norm plotted over 2-dimensional chunks of Queries (a), keys (b), and (c) Values for each layer in Gemma 7B, corresponding to different RoPE frequencies. The same process of frequency display does not show any clear pattern at each layer as RoPE not applied to the values. This supports the claim that the patterns shown in the queries and keys are due to RoPE.
25

---

Published as a conference paper at ICLR 2025
Query frequency usage (Layer Head
Key frequency usage (Layer
High Projuancios
High Frogpimiios
-44
14
100
Norm
-1
Noom
Lawe Peequionies
Lone Pooquamiin
-6
66
-2
Token
(b)
Attention pattern (Layer Head 5)
-08
-0.2
Figure 14: Queries (a), keys (b), and attention pattern (c) for diagonal head present at the first layer
in Gemma 7B. The head implements 'residual connection'.
26

---

Published as a conference paper at ICLR 2025
Query frequency usage (Layer
Key frequency usage (Layer Head
High Projuanncis
10
High Frespommitis
Norm
Norm
Lawe Peequioniis
Lowe Pooquammtio
Token
(b)
Attention pattern (Layer 27 Head B)
008
-0.2
-00
Figure 15: Queries (a), keys (b), and attention pattern (c) for a diagonal head present at the last layer in Gemma 7B. The head implements 'residual connection'.
27

---

Published as a conference paper at ICLR 2025
Query frequency usage (Layer Head
Key frequency usage (Layer
High Projucsciis
14
High Progpimiios
12
10
Norrh
Norm
88
Lawe Proquconiess
-6
Lowe Pooquamtinn
-6
-4
Token
(b)
Attention pattern (Layer Head 8)
-08
-0.2
Figure 16: Queries (a), keys (b), and attention pattern (c) for the previous-token head in Gemma 7B. The head makes tokens attend tokens appearing immediately before them.
28

---

Published as a conference paper at ICLR 2025
Query frequency usage (Layer
Key frequency usage (Layer
High Projucscios
High Fregpemiiis
12
-0
Norr
Norm
-8
Lawe Proquionies
Lowe Pooquamtio
66
-4
Token
(b)
Attention pattern (Laper Head 10)
1.0
008
Allestion
17
18
-00
Tokon
Figure 17: Queries (a), keys (b), and attention pattern (c) for the apostrophe head in Gemma 7B. The attention head learns make tokens appearing after an apostrophe token attend to the apostrophe.
29

---

Published as a conference paper at ICLR 2025
Query frequency usage (Layer Head 13)
Key frequency usage (Layer Head
High Peaqueniees
18
High Eriquencies
6
-15
12
Norm
Loto Frosgueme loxk
Proquamccess
14
15
12
Attention pattern (Layer Head 13)
1.0
008
-066
(c)
Figure 18: Queries (a), keys (b), and attention pattern (c) for a general 'semantic' attention head in
Gemma 7B, The queries in particular display high norm bands on the low frequencies.
30

---

Published as a conference paper at ICLR 2025
Gomma Query frequeney usage (Italian)
Gemma Query froquency usage (Chinese)
Eigh Froquicctes
6
Figh Proqueccia
8
Moan norn
Moan norm
Low Eroqueeces
1ow Pioquenciss
Layor
Layer
(b)
Gemma 7B Query trequency usage (Code)
Gemma 7B Query trequency usage (Arithmetic)
High Proquenccss
High Praguencios
-
5
Moan norm
Moan norm
Lovo Prorganncios
Praggamecioo
19
Layer
(c)
(d)
Figure 19:
Query frequency usage for Gemma 7B over different input domains. (a) Shows the usage for Italian prompt, (b) for Chinese prompt, (c) for Python Code, and (c) for Arithmetic additions between two large integers. The frequency usage patterns are very similar across all domains, with very slight differences.
31


Reviews:
[{"review_id": "raH0YWgspE", "reviewer": "Reviewer_T6HH", "summary": "This paper delves into the role of Position Embedding (PE) in LLMs, challenging the traditional view that RoPE primarily attenuates attention weights as the relative distance between words increases. It proposes a new hypothesis that RoPE constructs position-attention patterns (e.g., diagonal or previous-token focus) using high-frequency components while leveraging low frequencies to convey semantic information. Several case studies and theoretical analyses support this hypothesis.\n\nOverall, the paper provides insightful findings and hypotheses on a key component of LLMs: position encoding. However, the arguments mainly rely on case studies with gemma-7B, and the experiments lack diversity in the foundation models (e.g., missing LLaMA series) and tasks (e.g., language model vs. QA vs. code...). Additionally, some parts of the proofs contain errors.", "questions": "1. In practice, do the phenomena observed on gemma-7b apply to other LLM backbones, such as llama-2 or llama-3?\n2. The phenomena observed in this paper are based on what scale and type of data? Could the findings be validated across various tasks, such as language modeling, code, or QA?\n3. In Line 190, the authors claim that NoPE has strong OOD capabilities, but Kazemnejad et al., 2024 only validated this for sequences of length up to 50, where NoPE performed slightly better than other PEs, without exhibiting exceptionally strong OOD performance (refer to their Figure 3). The following works explored NoPE's extrapolation, and it can be seen that the generalization ability of the NoPE baseline is limited:\n> Length Generalization of Causal Transformers without Position Encoding (Wang et al., Findings 2024)\n> \n> [Neurips24 spotlight] Exploring Context Window of Large Language Models via Decomposed Positional Vectors\n4. In Line 497, the authors state, \"p-RoPE is in spirit similar to the idea of increasing the wavelength of RoPE from 10,000 to 500,000,\" so I suggest that Table 2 should include RoPE with a base of 500,000.\n5. How do the observations in Section 4 change across different RoPE variations, such as 0.25-RoPE, 0.75-RoPE, RoPE_10000, and RoPE_500000? For example, in Figure 3, how does the norm distribution of low frequencies vary across these models?\n6. In Line 726, this proof only applies to the first layer of NoPE’s attention heads. Starting from the second layer, the assumption \n$a_{3,3}=<q_3, k_3>=<q_3, k_2>=a_{3,2}$ \nno longer holds.\n7. Attention heads with specific patterns have been widely studied. Besides the patterns discussed in this paper (e.g., diagonal or previous-token focus), could the authors observe and analyze other representative attention patterns, such as special token focus, punctuation focus, and locality focus? Please refer to typical patterns in the following paper:\n> [ICLR24] Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper successfully challenges, both theoretically and empirically, the traditional view that RoPE attenuates attention weights as relative distance between tokens increases. \n2. The authors' hypothesis about the roles of the high-frequency and low-frequency components of RoPE is novel and insightful.", "weaknesses": "1. The experimental validation lacks diversity in both foundation models and datasets.\n2. Some perspectives and proofs regarding NoPE contain errors, while they don't affect the main conclusion, they may mislead readers.\n3. The discussion of related work is insufficiently thorough, and few papers are cited (only about one page).\n4. Throughout Section 4, the authors conceal a core assumption: that attention scores are interpretable or meaningful. Higher attention scores for certain tokens imply a meaningful preference in the model, giving special significance to the larger norms in Equation at Line 257. I want to point out that this assumption is still being debated, and I suggest that the authors make it explicit.\n> Is Attention Explanation? An Introduction to the Debate (Bibal et al., ACL 2022)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bra9GES0eY", "reviewer": "Reviewer_eLgp", "summary": "This paper investigates the role of Rotary Positional Encodings (RoPE) in Transformer-based Large Language Models (LLMs). The authors challenge the common belief that RoPE's usefulness comes from its ability to decay token dependency with increasing relative distance. Instead, they explore how different frequencies in RoPE are utilized, particularly within the Gemma 7B model. The paper provides both theoretical and empirical analyses, proposes a modified RoPE, and highlights the importance of understanding positional encodings for scaling LLMs.", "questions": "1. I have some doubts about the display of Figure 2. Normally, for the same qk, different relative distances should have a gradually decreasing effect. But does different qk introduce different variables, making the comparison unfair?\n2. Is there any empirical experiment on how many frequencies are cut off for the best effect?\n3. Will removing low-frequency RoPE make the effect on short text worse?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a fresh perspective on RoPE, questioning existing assumptions and offering new explanations for its effectiveness.\n2. The authors present mathematical proofs to support their claims, enhancing the credibility of their findings.\n3. The use of the Gemma 7B model for empirical analysis adds practical relevance to the theoretical insights.", "weaknesses": "1. Although the observed phenomena and mathematical proofs can support the paper's point of view, the experimental performance does not seem good enough. The paper hopes to adapt to any context length, but the actual experimental results only have one result on 8K. And the evaluation of PPL is not comprehensive enough.\n2. At the semantic level, the results of the models in Table 2 should be compared on the general benchmark or other tasks that are more representative of semantics, which will be more convincing.\n3. At the positional level, it should be compared with similar experiments such as needle in a haystack or Ruler on long contexts to prove its long context expansion ability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xIZr2627Lb", "reviewer": "Reviewer_rv9e", "summary": "Authors explore how transformer models use RoPE frequencies to learn semantic and positional information. Authors provide a theoretical proof that RoPE does not force the decay in attention coefficient with distance, but instead can create specific patterns. In their experimental study of the Gemma 7B model, authors show that transformers mostly rely on low frequencies, while some heads display high frequency bands, mostly in 1st and last layers of the model. Authors further show that high-frequencies in RoPE provide a mechanism to encode positional information. Low frequencies are used as information channels that are not robust over long context. Finally, authors propose p-RoPE encoding that cuts low frequencies and can help to improve model's performance.", "questions": "1. Pre-training model from scratch is expensive and not always feasible. I was wondering if you can instead continuously train other (Gemma-7B, Llama-3 1/3/8B, etc) models for fewer steps but with p-RoPE approach? Do you think it would work or no and why?\n\n2. Please, correct me if I'm wrong, but in the proof of Proposition 3.1:\nfor k>1, g_k=theta^{-2(k-1)/d} is not necessarily rational, but algebraic. Therefore, Lemma A.1 should be stated not for rational g, but for algebraic g, and it should use the fact that pi is transcendental.\nline 680, cos(j-i-r) -> cos(j-i+r)\nline 680, the comma in the displayed equation should be a period.\nline 684, cos(j-i-r) -> cos(j-i+r)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The authors conduct a novel theoretical and empirical study of RoPE encodings in transformer models.\n- They provide detailed proofs of their main claims.\n- The paper is clear and well-written.\n- This study can help researchers better understand the underlying mechanisms of popular transformer architectures and encourage research into alternative improved solutions.", "weaknesses": "- The empirical study is limited to a single Gemma architecture. While this is unlikely, some results may be artifacts of the specific model selected.\n- In Section 3 authors train 2B model and show improvements on validation perplexity. While these results are positive, perplexity improvements do not always results in overall improvements in model's abilities. Authors could provide evaluation results on popular benchmarks* to build more convincing picture.\n\n\n* see, for ex, Section 2.3 in https://arxiv.org/pdf/2307.09288", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "stJiuAMm0w", "reviewer": "Reviewer_7eRV", "summary": "The paper provides practical insights into positional encoding for decoder-only models. However, further investigation is needed to establish the effectiveness and reliability of the theories presented.", "questions": "Why did the authors provide an ablation study only for the lower frequencies rather than for both low and high frequencies?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "Pros:\n\n1. The paper offers a fresh discovery of Rotary Positional Encoding (RoPE), challenging the belief that it primarily helps decay attention coefficients with distance.\n\n2. The paper gives an analysis of high and low RoPE frequencies, their roles in positional and semantic attention, and an innovative RoPE modification (p-RoPE) that demonstrates improvements in some cases.\n\n3. Every section has a summary part which makes the whole paper clear and ready to read.", "weaknesses": "Cons:\n\n1. The figures in the paper are somewhat confusing. For instance, while the paper emphasizes frequency aspects, Figure 1 illustrates vectors with identical frequency differences, which does not fully align with the paper’s focus. Additionally, the results are primarily presented using heat maps, which may appear monotonous and lack of expressiveness. Given that this is a language modeling task, presenting some results in natural language format would enhance readability and interpretability.\n\n2. (MAIN LIMITATION) The experiments are limited to decoder-only models, making it unclear whether the findings can generalize to other transformer architectures. Whether the observed improvements stem from specific model structures (e.g., encoder-only or encoder-decoder models) rather than from a comprehensive enhancement applicable to diverse positional encodings of transformer. Additionally, the authors did not consider the impact of long-range dependencies or how the choice of positional encoding influences masked attention performance of decoder-only model. It would be valuable if the authors could report comparative results using unmasked attention (e.g., with encoder-only transformers) to examine these effects.\n\n3. (MAIN LIMITATION) The proposed modification p-RoPE appears to primarily integrate elements of RoPE and NoPE, which may lack novelty. While the study advances understanding of RoPE, its applicability to other positional encoding methods, such as Alibi, remains limited, potentially constraining its relevance for models with alternative encoding schemes. The authors should consider incorporating other positional encoding methods and propose more innovative improvements to enhance the usage of lower and higher frequencies.\n\n4. The paper states that higher frequencies correspond to positional attention, while lower frequencies correspond to semantic attention. However, the authors provide an ablation study only for the lower frequencies, without addressing both low and high frequencies. This aspect is insufficiently explained in the ablation experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R38mxyPD8S", "reviewer": "Reviewer_DtVE", "summary": "This paper studied the inner workings of rotary positional embedding. The authors started by challenging the common belief that RoPE decays with distance. Then, the authors showed in Gemma 7B that most RoPE usages appear in low frequencies. The authors explained that high frequencies are for positional information while low frequencies are for semantic information. Finally, the authors observed that the low-frequency components are not robust. Based on this observation, the authors proposed p-RoPE to remove (1-p)*100% of the low-frequency component and showed that this improved the performance of Gemma 2B models.", "questions": "* Is your p-RoPE having a similar flavor as the partial rotary?\n  - Partial rotary means the whole dimension is divided into rotary part and non-rotary part. The rotary part will be treated by RoPE while the non-rotary part is treated by NoPE.\n  - Partial rotary embedding has been known by the community for a while (see https://github.com/lucidrains/x-transformers/issues/40). It is known that the partial rotary is slightly better than rotary positional embedding.\n  - Because the proposed p-RoPE is interpolating between NoPE (p=0) and RoPE (p=1), it is possible that p-RoPE is similar to partial rotary.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written and the key points are delivered clearly.\n* The authors made a good point that RoPE doesn't necessarily decay with distance.\n* The figures clearly showed that most attentions happen in low frequencies, and some high-frequency heads and frequency bands exist.", "weaknesses": "* The claim that high-frequency components are for positional information is unclear.\n  - The reasoning seems to be: (a) RoPE can learn a diagonal or off-diagonal pattern. (b) NoPE cannot learn a diagonal or off-diagonal pattern. So RoPE can learn positional information. However, it doesn't necessarily mean the high-frequency components contribute to the diagonal/off-diagonal pattern. So the function of high-frequency components remains unclear.\n  - Some prior works provide evidence that NoPE can still encode positional information [1][2]. So the fact that NoPE cannot learn a diagonal or off-diagonal pattern may not imply it cannot encode positions.\n* The authors experimentally show that truncating the lowest frequencies can help RoPE learn better. However, it is based on a claim that \"RoPE lacks robust semantic channels\". \n  - This claim comes from an analysis on a 2-dimensional case. However, it is possible that the 2D case is too restrictive and a higher dimensional case could have a different result.\n\n[1] Haviv et. al. \"Transformer language models without positional encodings still learn positional information,\" EMNLP 2022 Findings.\n[2] Chi et. al. \"Latent positional information is in the self-attention variance of transformer language models without positional embeddings,\" ACL 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)