Round and Round We Go! What makes Rotary Positional Encodings useful?

## Abstract

Abstract Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust ‘positional’ attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments to verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling LLMs to large sizes and context lengths.

## 1 Introduction

It is common to provide positional information to the attention mechanism in Transformers through the use of absolute positional encodings (Vaswani et al., ), relative positional encodings (Su et al., ), or by introducing a bias directly to the activations (Press et al., ). One of the currently most widely adopted encodings, especially in Large Language Models (LLMs), are Rotary Positional Encodings (RoPE) (Su et al., ), being used in popular models such as LLama 3 (Dubey et al., ) and Gemma (Gemma Team et al., ). RoPE acts on the queries and keys by splitting them in $2$-dimensional chunks and rotating each chunk at a different frequency. The method can be implemented efficiently and provides an interesting geometric approach to positional encodings.

Despite the significant adoption of RoPE, the specific reasons why this method is useful to Transformer models remains poorly understood. One of the main arguments in favour of RoPE made by Su et al. () is that the method helps to decay attention coefficients as the relative distance grows. Most such claims, however, rely on the queries and keys being *constant* vectors – which is uncommon in practice. In fact, in this work we find that there are many situations in which this decay does *not* occur and that this is exploited at times by attention heads in Gemma 7B (Gemma Team et al., ).

Further, there are open intriguing questions relating to how exactly the different frequencies in RoPE are useful. In the standard parameterisation of RoPE, the fastest frequencies rotate at $1$ radian per token, whilst the slowest are several orders of magnitude slower at $\approx 1/10{,}000$ radians per token. As dot product attention directly depends on the angle between the queries and keys, the highest frequencies are extremely sensitive to small token rearrangements, making them poor carriers of information. Consequently, we find it interesting and important to understand how exactly the different rotation frequencies are used in LLMs.

##### Contributions.

In this work, we study in-depth empirically and theoretically how Transformers and in particular LLMs benefit from RoPE. We rely on the pretrained and open-source Gemma 7B (Gemma Team et al., ) model for our empirical analysis. We summarise our findings:

- •
In Section 3, we argue against the common claim that RoPE is useful because it encourages the decay of attention coefficients with distance. We provide theoretical and empirical evidence to support our claim.
- •
In Section 4, we propose a new way to understand the usage of different frequencies in the queries and keys. We find that Gemma 7B largely prefers to use the low frequencies of RoPE. The first and last layers instead show the most use of the high frequencies.
- •
In Section 5, we show that the highest frequencies in RoPE are cleverly used by Gemma 7B to construct special ‘positional’ attention heads (see Figure 1). We mathematically prove the ‘robustness’ of the construction.
- •
In Section 6, we study how the low frequencies are used. We observe distinct ‘bands’ in the low frequencies of the queries and keys. We conjecture that Gemma 7B is using them as ‘information channels’. We prove that these channels cannot be robust over long context.
- •
In Section 6.1, we propose a new technique called $p$-RoPE that removes the lowest frequencies of RoPE to create robust semantic channels. We show not only that removing a percentage of the frequencies maintains the performance, but also *improves it* on 2 billion parameter models. We believe that our work explains why increasing the maximum RoPE wavelength helps with long-context, e.g. as shown in Llama 3 (Dubey et al., ).

Figure: Figure 1: Depiction of our construction which allows Transformers to obtain positional attention heads using RoPE. On the left we depict key and query vectors for each position $i$, where keys are all identical, and queries are just a rotated version of the key, in a way that matches one of RoPE’s highest frequencies. The center depicts how keys get rotated by RoPE, making the key at $i-1$ perfectly align with the query. Due to the high frequency of the rotation, all other keys will lead to a smaller attention weight. On the right we show the resulting attention weights, resulting in this case in an off-diagonal positional attention. See Section 5 for more details.
Refer to caption: x1.png

## 2 Background

We denote by $\mathbf{x}_{i}\in\mathbb{R}^{d}$ the $d$-dimensional token embedding of the $i$-th token. Query and key vectors take the form $\mathbf{q}_{i}=\mathbf{W}_{Q}\mathbf{x}_{i}$ and $\mathbf{k}_{i}=\mathbf{W}_{K}\mathbf{x}_{i}$ respectively, given query and key matrices $\mathbf{W}_{Q},\mathbf{W}_{K}\in\mathbb{R}^{d\times d}$. The attention mechanism(^1^11We ignore here the $1/\sqrt{d}$ scaling factor introduced by Vaswani et al. () for ease of notation.) performs the following computation:

$$ $\alpha_{i,j}=\frac{\exp\left(\mathbf{a}_{i,j}\right)}{\sum_{\ell\leq i}\exp \left(\mathbf{a}_{i,\ell}\right)},\text{with }\mathbf{a}_{i,j}=k\left(\mathbf{ q}_{i},\mathbf{k}_{j}\right)$ (1) $$

where $k$ is a kernel function which in our work takes the form of either a simple dot product, i.e. No Positional Encoding (NoPE) (Haviv et al., ; Kazemnejad et al., ), or RoPE. We focus on the case in which the attention mechanism is *causal*, the most common type of attention used in LLMs today. In a causal mechanism, we have that $\alpha_{i,j}=0$ when $j>i$ and $\sum_{j:j\leq i}\alpha_{i,j}=1$. We also note that we assume $\mathbf{a}_{i,j}$ to be finite, as it is always the case in practice, such that $0<\alpha_{i,j}<1$, when $j<i$. We call $\mathbf{a}_{i,j}$ the ‘activation’ or ‘logit’, while we call $\alpha_{i,j}$ the ‘attention coefficient’ between $i$ and $j$. It is sometimes useful to view the attention coefficients in matrix-form, which in our specific case results in a lower triangular row-stochastic matrix. We highlight an important special case for $k$ which we call $k_{\text{NoPE}}$, i.e. no positional encoding:

$$ $k_{\text{NoPE}}\left(\mathbf{q}_{i},\mathbf{k}_{j}\right)=\mathbf{q}_{i}^{\top }\mathbf{k}_{j},$ (2) $$

where $\mathbf{q}_{i}^{\top}$ denotes the tranpose of $\mathbf{q}_{i}$. In other words, in NoPE the kernel function computes simply the dot product, providing no positional information to the Transformer. It has been shown that Transformers can still perform well, especially out of distribution, with NoPE (Kazemnejad et al., ). In particular, Kazemnejad et al. () prove that the Transformers could in principle recover absolute positional information through the causal mask; however, the proof relies on the universal approximation theorem, which we believe is a practical limitation.

### 2.1 Rotary Positional Encodings (RoPE)

For simplicity of notation in this work we assume that query are key vectors are $d$-dimensional, with $d\geq 2$ being an even number. We decompose queries and keys into $2$-dimensional chunks $\mathbf{q}_{i}=\bigoplus_{k=1\dots d/2}\mathbf{q}_{i}^{[k,k+1]}=\bigoplus_{k=1
\dots d/2}\mathbf{q}_{i}^{(k)}$, where $\bigoplus$ denotes direct sum (concatenation). In other words, we denote by $\mathbf{q}_{i}^{(k)}\in\mathbb{R}^{2}$ the $k$-th $2$-dimensional chunk of the query vector of the $i$-th token, using analogous notation for the key vectors.

RoPE considers a sequence of angles $G=\left(g_{k}=\theta^{-2(k-1)/d}:k=1,\dots,d/2\right)$(^2^22We denote the angles $g_{k}$ instead of $\theta_{k}$ as we opt for more of a group theoretic perspective of RoPE.), where $g_{1}=1$ is the fastest rotating component at $1$ radian per token and $g_{d/2}=\theta^{-(d-2)/d}\approx\theta^{-1}$ the slowest rotating component at approximately $1/\theta$ rotations per token. The parameter $\theta$ is called the base wavelength, which by default is $10{,}000$ (Su et al., ), although works have explored increasing it to, for instance, $500{,}000$ (Xiong et al., ; Roziere et al., ; Dubey et al., ). We denote by $\rho(g_{k})$ the matrix form of $g_{k}$:

$$ $\rho(g_{k})=\begin{bmatrix}\cos(g_{k})&-\sin(g_{k})\\ \sin(g_{k})&\cos(g_{k})\end{bmatrix},$ (3) $$

highlighting that $\rho(g_{k})$ is a $2$-dimensional orthogonal transformation (rotation). One can view $\rho(g_{k})$ as a ‘unit rotation’ by $g_{k}$ radians. The RoPE technique amounts to the construction of a block-diagonal matrix $\mathbf{R}^{i}=\bigoplus_{k=1\dots d/2}\rho(g_{k})^{i}\in\mathbb{R}^{d\times d}$, where each $2\times 2$ block on the diagonal is a rotation by a different frequency of RoPE. The $\mathbf{R}^{i}$ denotes in fact matrix exponentiation by an integer $i$ which is the position of $\mathbf{x}_{i}$ (^3^33We emphasize that $i$ and $j$ have nothing to do with $\sqrt{-1}$, but instead denote the position $i$ and $j$ of the token in the sequence.). We can exploit a nice property of rotation matrices, i.e. that $\rho(g_{k})^{i}=\rho(ig_{k})$ to avoid the computation of the matrix power. As this matrix is block diagonal, computing $\mathbf{R}_{i}\mathbf{q}_{i}$ means that the rotations act only on $2$-dimensional chunks of the query (or key), i.e. $\mathbf{R}_{i}\mathbf{q}_{i}=\bigoplus_{k=1\dots d/2}\rho(ig_{k})\mathbf{q}_{i
}^{(k)}$. This leads to the final formulation of $k_{\text{RoPE}}$:

$$ $k_{\text{RoPE}}\left(\mathbf{q}_{i},\mathbf{k}_{j}\right)=\left(\mathbf{R}^{i} \mathbf{q}_{i}\right)^{\top}\left(\mathbf{R}^{j}\mathbf{k}_{j}\right)=\mathbf{ q}_{i}^{\top}\mathbf{R}^{j-i}\mathbf{k}_{j}=\sum_{k=1\dots d/2}\left(\mathbf{q }_{i}^{(k)}\right)^{\top}\rho(g_{k})^{j-i}\mathbf{k}_{j}^{(k)},$ (4) $$

where we use the fact that $\left(\rho(g_{k})^{i}\right)^{\top}\rho(g_{k})^{j}=\rho(g_{k})^{-i}\rho(g_{k})
^{j}=\rho(g_{k})^{j-i}$. We highlight how the block diagonal structure of $\mathbf{R}$ allows one to decompose the dot product into the sum of dot products of $2$-dimensional chunks, with each key vector chunk rotated at a frequency dictated by $g_{k}$.

### 2.2 Related works

A number of works have investigated how different modifications of RoPE affect its generalisation. A well-known method involves increasing the parameter $\theta$ from the originally proposed $10{,}000$ to a larger value, such as $500{,}000$ (Xiong et al., ; Roziere et al., ). Notably, this is also used in the recently released LLama 3 class of models (Dubey et al., ). The justification provided for this modification is that a higher base wavelength means that the attention decay induced by RoPE will be slower, allowing for more robust learning over a larger context. In this work, we challenge this common assumption and investigate why RoPE and modifications such as these are helpful.

In a different direction, works have pointed out that NoPE shows strong performance in *out-of-distribution* (OOD) settings when compared to RoPE, arguing that the causal mechanism is sufficient to learn positional information (Haviv et al., ; Kazemnejad et al., ). In this work, we instead argue that NoPE and RoPE have *complementary* strengths and weaknesses and that for instance NoPE is unable to learn certain types of attention matrices present in Gemma 7B. Ruoss et al. () propose to provide *randomised* positional information, showing that this helps boost OOD performance, claiming that this helps the model to learn over a longer range of relative distances. In our work (Appendix, Section B.2), we provide a different explanation as to why this kind of process might be helpful from the point of view of the type of invariance it encourages.

Overall, we believe this work provides a different and perhaps more nuanced perspective on different works that tackle positional encodings. In spirit, this work is similar to works that aim to understand LLMs from a mechanistic perspective (Elhage et al., ; Olsson et al., ; Wang et al., ; Hanna et al., ) and from the representations they produce (Barbero et al., ; Veličković et al., ).

## 3 Does RoPE decay activations with distance?

In this section, we argue against the common claim that RoPE is helpful because it helps to decay activations as the relative distance between tokens increases. Such claims often work under the assumption that queries and keys are for instance a vector with all entries equal to each other, which we believe is an unrealistic oversimplification(^4^44Intuitively, Su et al. () ask the question of what happens, as we vary the relative distance, to the dot product between already aligned queries and keys. This indeed will lead to a ‘decay’ with relative distance. However, this perspective ignores what happens to originally misaligned queries and keys, whose dot product can *increase* with relative distance.). Importantly, this claim was originally provided by the authors of RoPE (Su et al., ) as a justification for the chosen structure of the encoding. Follow up works have used this claim to justify modifications, such as increasing the base wavelength $\theta$ to $500{,}000$. We therefore find it important to point out cases in which this decay does not in fact occur.

Figure: (a) RoPE applied to constant queries and keys.
Refer to caption: extracted/5911145/figures/constant-qks.png

We start by showing, in Proposition 3.1 that given any key, we can find a query such that RoPE is maximal for any chosen relative distance. This highlights the fact that RoPE provides Transformers with *robust* ways to attend to specific relative distances. In fact, we will show, in Section 4, that this mechanism is what Gemma 7B uses to construct heads that attend to specific positions. We provide the proof in the Appendix (Section A.1).

###### Proposition 3.1 (RoPE can be maximal at arbitrary distance) .

Given any query $\mathbf{q}$ and any relative distance $r\in\mathbb{Z}$, we can find a key $\mathbf{k}$ such that the softmax value is largest at distance $r$ with RoPE.

Next, in Proposition 3.2, we show that given queries and keys sampled independently from a standard multivariate Gaussian, the expected value of the activations is 0 0. Moreover, this is independent of the relative distance of the queries and keys – implying that the expected value of the activations is independent of the relative distance when the queries and keys are sampled from a Gaussian. We provide the proof in the Appendix (Section A.1).

###### Proposition 3.2 (Gaussian queries and keys do not decay.) .

Let $\mathbf{q},\mathbf{k}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. Then, for any relative distance $r\in\mathbb{Z}$, we have that:

$$ $\mathop{\mathbb{E}}_{\mathbf{q},\mathbf{k}\sim\mathcal{N}(\mathbf{0},\mathbf{I })}\left[\mathbf{q}^{\top}\mathbf{R}^{r}\mathbf{k}\right]=0.$ $$

We showcase this by constructing a synthetic experiment in which we either set queries and keys as all-ones vectors, e.g. as done by Su et al. (); Xiong et al. (), or sample entries independently from a Gaussian, accounting for appropriate normalisations. Figure 2 shows the results. While there seems to be some form of decay of the activations as relative distance increases when the queries and keys are all-ones vectors (a) – up to appropriate normalisation, this is clearly not the case when the queries and keys are instead random Gaussian vectors (b).

###### Proposition 3.1 (RoPE can be maximal at arbitrary distance) .

###### Proposition 3.2 (Gaussian queries and keys do not decay.) .

## 4 How are different frequencies used?

In this section, we explore how different frequencies of RoPE are used. RoPE relies on a set of frequencies $G$ that take values $1,\dots,\theta^{-(d-2)/d}$, with the highest frequency varying by $1$ radian per token, while the lowest being much more stable, varying at $\approx 1/\theta$ per token. As the angle between vectors affects the dot product, the contribution from the highest frequencies should behave similarly to random noise, i.e. a small perturbation in the token sequence will give a largely different activation contribution. A natural question is whether these frequencies are being used and, if so, how exactly are they helpful?

To measure the usage of frequencies, we start by noting that by Cauchy-Schwarz, the effect of the $k$-th frequency component on the activation $\mathbf{a}_{i,j}$ is upper bounded by the $2$-norm of the query and key components, i.e. $\lvert\langle\mathbf{q}_{i}^{(k)},\mathbf{k}_{j}^{(k)}\rangle\rvert\leq\lVert
\mathbf{q}_{i}^{(k)}\rVert\lVert\mathbf{k}_{j}^{(k)}\rVert$. It is therefore natural to look at the mean $2$-norm for each $\mathbf{k}^{k}$ in Gemma 7B over long sequences. For Gemma 7B, we note that $k=1,\dots,128$, i.e. the hidden dimension is $128\times 2=256$.

Figure 3 shows the results. We plot the average $2$-norm at each layer of each rope ‘chunk’ over a number of sequences, ordering them by frequency, with the intuition that the norm will be an upper bound for how much that frequency will impact the activation dot product. We emphasize that the mean is taken over all of the $16$ heads at each layer. It is clear that learning has assigned much higher norm on average to the lowest frequencies, meaning that they will likely influence the dot product the most. This seems to be true at each layer. Interestingly, there seems to be some high frequency usage present especially at the very first and last layers.

We believe this to be remarkable evidence showcasing how Gemma adapts to RoPE by preferring to use the lowest frequencies when computing attention activations. In the Appendix (Section D.2, Figure 11), for completeness, we show that this type of distribution does not occur for the value vectors. This highlights that this behaviour is a consequence of RoPE, where learning discovers that these medium to high frequency ‘chunks’ are not useful and hence pushes their norm to 0 0 so that their impact on the dot product is minimal. Corresponding entries in value vectors, not being rotated, do not suffer from the same pressure to have their norm 0 0.

In Figure 4, we show the frequency usage of the $16$ attention heads in the first layer. The heads that stand out as using the high frequencies, especially for the keys, are Heads 5 and 8. We will show in the next section that these heads correspond to *positional attention heads*. We also highlight the sparse nature of the frequency usage, with the presence of ‘high norm’ bands, especially at the lower frequencies. We highlight that this kind of pattern seems consistent with the observation that feed-forward layers act as sparse dictionary lookup tables (Geva et al., ). In this context, we believe that these bands are used to perform some kind of sparse query and key semantic matching.

Figure: (a) Mean query norm distribution at each layer.
Refer to caption: extracted/5911145/figures/query-chunk-distribution.png

Figure: (a) Mean query norms at each attention head.
Refer to caption: extracted/5911145/figures/first-layer-queries-frequency.png

## 5 High frequencies: Positional attention

The highest frequencies in RoPE are interesting to study as their usefulness is not immediately obvious. In particular, in the previous section, we showed that Gemma 7B seems to largely avoid them. In this section, we will study the cases in which certain attention heads mostly use the very highest frequencies, showing that these heads tend to display purely positional attention patterns. We will prove by construction that RoPE allows for *arbitrarily sharp* attention patterns of this type – a construction that Gemma 7B seems to be closely learning in practice. We will also prove that such heads cannot be constructed with NoPE. *We believe the ability to construct such heads is important when trying to optimise for auto-regressive generation.* These heads have also shown to be useful for structure generalisation (see, e.g., Figure 6 in Ruoss et al. ()).

Figure: (a) A diagonal head in Gemma 7B.
Refer to caption: extracted/5911145/figures/gemma-diagonal-lastlayer-attention.png

Figure: (a) Query frequency usage for Layer 27 Head 8. head
Refer to caption: extracted/5911145/figures/gemma-q-layer27-head8.png

In Figure 5, we show certain heads in Gemma 7B that specialise into purely positional attention patterns – with Figure 5 (a) showing a ‘diagonal’ attention pattern and 5 (b) a ‘previous-token’ attention pattern. These heads seem to attend based purely on relative position, with no regards for semantics. It is not immediately clear how purely diagonal attention patterns are helpful, as these in principle behave like a residual connection. The reason *why* Gemma learns to construct such a pattern is outside the scope of this work, but constitutes interesting learnt behaviour which requires further analysis – perhaps a training ‘bug’.

Figure 6, shows that for the diagonal head, Gemma seems to rely mostly on the very highest frequencies. We provide in the Appendix (Section D.2) more examples showing that this seems to be a repeated pattern throughout these positional heads. In fact, we found it remarkably consistent to identify positional heads, by looking at the usage of the highest frequencies. The heads 5 and 8 previously pointed in Figure 4 in fact correspond to positional heads with attention head 5 being purely diagonal and attention head 8 a previous-token head.

##### Constructing robust positional heads.

We now study theoretically the mechanism through which Gemma 7B learns to robustly construct positional attention patterns. We start by defining more formally diagonal and previous-token attention patterns. Importantly, our definition relies on being able to learn patterns to an arbitrary precision $\epsilon$, given a maximum size $N$.

###### Definition 5.1 .

An attention head can learn a diagonal attention pattern if given an $\epsilon>0$ and a fixed size $N$, for all $i\leq N$, we have $\alpha_{i,i}>1-\epsilon$, implying that $\sum_{j:j<i}=\epsilon$. Similarly, for the previous-token head, we must have $\alpha_{i,i-1}>1-\epsilon$ for all $\epsilon>0$, implying that $\sum_{j:j\neq i-1}\alpha_{i,j}=\epsilon$.

We prove in Proposition 5.2 that no construction exists for NoPE that is able to learn such attention patterns. The proof relies on a counterexample involving token repetition and may be found in the Appendix (Section A.2).

###### Proposition 5.2 .

An attention head with NoPE cannot learn a diagonal or off-diagonal pattern.

Instead, in Theorem 5.3, we show that an attention head with RoPE is able to learn such patterns. The construction relies on setting queries and keys equal to each-other and for the off-diagonal case, inverse-rotating them by a unit rotation for each RoPE chunk. The proof is provided in the Appendix (Section A.2).

###### Theorem 5.3 .

An attention head with RoPE can learn a diagonal or off-diagonal pattern.

In fact, Gemma 7B seems to be learning a construction that is strikingly similar to our construction – we provide evidence in the Appendix (Section D.1). We also find why the highest frequency rotations are most similar, as they are the ones which will affect the dot-product most rapidly. For the diagonal case, setting queries and keys equal means that the dot product $\langle\mathbf{q}_{i},\mathbf{R}^{j-i}\mathbf{k}_{j}\rangle$ will be maximal when $j=i$ and strictly smaller when $j\neq i$. The decrease in the dot product will depend on the angle between the rotated queries and keys. This angle will become dissimilar the fastest when using the highest frequency rotations. If $g_{d/2}\approx 1/10000$ for instance, then $g_{d/2}^{j-i}$ will misalign the queries and keys by a radian at a distance of $\approx 10{,}000$ tokens, while the highest frequency $g_{1}$ will only need a single token.

###### Definition 5.1 .

###### Proposition 5.2 .

###### Theorem 5.3 .

## 6 Low frequencies: Semantic attention

We now study how the low frequencies are being used. As observed in Section 4, the majority of the allocated norm (within the key and query embeddings) in Gemma 7B seems to be towards the lower frequencies. Furthermore, there seems to be a presence of distinct ‘bands’ in attention heads, especially in these low frequencies. In this section, we argue that the low frequencies are most useful to detect information related to token ‘semantics’
as they are the most invariant to token relative distance. The rotations are however not perfectly invariant over long relative distance,
leading eventually to misalignment of the vectors also at the lowest frequencies.

In other words, the lowest frequencies are useful *as they are precisely the frequencies for which dot product is the least affected by the relative distance*. In particular, we believe this is why works such as LLama 3 have found it useful to increase the base wavelength to $500{,}000$, meaning that the lowest frequencies rotate at roughly $1/500{,}000$ radians per token. With a context length of $128$k in LLama 3, the standard wavelength of $10{,}000$ will in fact complete $\approx 2.04$ rotations, which might lead in certain situations to poor generalisation due to tokens being erroneously misaligned simply due to the large context. We therefore suspect that with larger and larger contexts, the $\theta$ base wavelength of RoPE will have to also be increased accordingly.

In Figure 7 (a), we identify an interesting semantic head that makes tokens appearing after an apostrophe token attend to the apostrophe(^5^55For instance, for the following $4$ tokens [BOS, I, ’, m], the attention head at token m will attend to the apostrophe token ’. For all other tokens it will attend to the BOS token.). We point out the qualitative difference when compared to the frequency usage in the positional heads from Section 5. This head is interesting being it is semantic (detecting apostrophes), but also positional, only detecting an apostrophe at the *previous token*. We see how Gemma 7B is using the highest frequencies, visibly active in Figure 7 (b), to detect ‘previous token’ apostrophes. The low frequency dark band instead we believe is being used to attend robustly to the BOS token when there is no apostrophe present at the previous token – see the Appendix (Section D.1) for details.

Figure: (a) Semantic ‘apostrophe’ attention head.
Refer to caption: extracted/5911145/figures/gemma-apostrophe-attention.png

We now however show that the ‘semantic bands’ cannot be robust over large context. In particular, irrational rotations are ‘dense’ meaning that given a long enough context, they can rotate to any arbitrary value. We use this to show that there exists a relative distance for which the attention activation will be erroneously large. We prove the $2$-dimensional case, i.e. a single RoPE frequency, for simplicity. We believe that this is still an informative case as we observe that these bands are often very distinct and focused on a single frequency. We provide the proof in the Appendix (Section A.3) and point out that the observation that NoPE is instead able to build this type of attention pattern is immediate.

###### Theorem 6.1 .

Let the hidden dimension be $d=2$, i.e. there is only a single RoPE frequency $g_{1}$. Consider a sequence $\mathbf{x}_{bos},\mathbf{x}_{1},\dots,\mathbf{x}_{N}$ with a target token at some position $\mathbf{x}_{n}$. Then for large enough $N$, the attention head with RoPE is unable to attend to such a token such that $\alpha_{i,n}>1-\epsilon$ for any $\epsilon>0$ and any $1\leq n\leq N$.

###### Theorem 6.1 .

### 6.1 Truncating the lowest frequencies of RoPE

In the previous section and more formally in Theorem 6.1, we claimed that RoPE lacks robust semantic channels. To adapt to this, as shown in Figure 3, Gemma 7B seems to prefer to use the lowest frequencies to construct high norm semantic bands. The key insight is that unfortunately these high norm bands are not robust when the context becomes very large. We believe that this is why works such as Llama 3 found it helpful to increase the base wavelength of RoPE due to the larger context.

In this section, to investigate this further, we carry out an ablation in which we truncate the very lowest frequencies of RoPE. The intuition is that if our newly found intuition is correct, truncating the lowest frequencies *should not harm performance*. In fact, this allows RoPE to provide robust semantic channels that are distance agnostic. We call this modification $p$-RoPE, with $0\leq p\leq 1$ being the fraction of RoPE ‘kept’. We highlight two special edge cases: $p=0$ coincides with NoPE, while the case $p=1$ with RoPE. As such, the quantity $p$ behaves like an interpolant between the two, with higher values of $p$ being closer to RoPE and lower to NoPE. We provide additional details in the Appendix (Section C) and summarise the properties of the three types of encodings in Table 1.

**Table 1: A summary of the discussed properties of RoPE, NoPE and $p$-RoPE.**
| Encoding | NoPE | RoPE | $p$-RoPE |
| --- | --- | --- | --- |
| Positional | ✗ | ✓ | ✓ |
| Semantic | ✓ | ✗ | ✓ |

We would like to note that $p$-RoPE is in spirit similar to the idea of increasing the wavelength of RoPE from $10{,}000$ to $500{,}000$, first proposed by Roziere et al. () and later employed by LLama 3 (Dubey et al., ). The effect that increasing the wavelength has is that of increasing the amount of tokens required to destroy semantic information, leading to improved long-range performance. For a wavelength that is large enough compared to the context, the very lowest frequencies can act as a semantic channel, as the position will have very little impact on the dot product. A summary of the properties between NoPE, RoPE, and $p$-RoPE is shown in Table 1.

We train Gemma 2B models from scratch on the Wiki and FlanV2 training datasets (see Section C for details). We benchmark the use of RoPE, NoPE, and $p$-RoPE. We set the base wavelength $\theta$ to $10{,}000$ and train and evaluate using the standard Gemma $8$k token context. In Table 2, we show the perplexity on the validation set. We can see how truncating the lowest frequencies *not only maintains the same performance, but even seems to improve the validation perplexity*, supporting our claims regarding the low frequencies acting as non-robust semantic channels in standard RoPE. Truncating even more of the frequencies ($0.25$-RoPE) harms performance, but with results still significantly better than with NoPE. Of course an $8$k context is not very large and improvements due to wavelength increase typically were observed on a larger context of $32$k (Xiong et al., ). We lack the resources to validate this on larger context lengths, but believe this to be a very interesting future direction. We in fact expect that truncating some small percentage of the lowest frequency tokens to help with long-context generalisation by providing robust semantic channels that are independent of relative distance and can therefore generalise automatically to any context length.

## 7 Conclusion

In this work, we explored the behaviour of RoPE. We started by arguing that the use of RoPE does not ensure that attention weights will decay with relative distance. This is provably the case if we assume, for instance, that the keys and queries are Gaussian random vectors. Further, we provided a construction that ensure the attention weight is maximal at any provided relative distance, regardless of magnitude, with these observations going against the typical intuitions used to justify RoPE.

We provided evidence that high-frequencies bands are used to build circuitry that leads to positional attention. We provided a construction that ensures this behaviour and identified a strikingly similar construction within the attention heads of Gemma 7B. Furthermore we showed that NoPE is unable to do similar positional attention.

Finally we showed that low-frequencies are used for semantic attention, and argued that removing the RoPE rotation for these low-frequencies can potentially improve the ability to attend semantically and in particular to generalize such type of attention to longer contexts. We verified this through an ablation study with Gemma 2B. We overall see our work as providing a more nuanced understanding of RoPE and hope that it can lead to an improved understanding of how LLMs use positional encodings, ultimately leading to performance gains – especially over long contexts.

#### Acknowledgments

The authors would like to thank Anian Ruoss (Google DeepMind) and Simon Osindero (Google
DeepMind) for their valuable comments and suggestions on this work.

## Appendix A Proofs

In this section we provide the omitted proofs for the theoretical statements in the paper. We also provide more detailed discussions on the constructions and ideas behind the proofs. In Section A.1 we provide proofs for Section 3, in Section A.2 we provide proofs for Section A.2, and in Section 6 we provide proofs for Section A.3.

### A.1 Does RoPE decay with distance?

In some of our proofs, we use known results on irrational rotations. We point to the literature for a detailed overview of these concepts (Masur & Tabachnikov, ), although the results we leverage in our work are elementary. The first statement is about the ‘uniqueness’ of irrational rotations.

###### Lemma A.1 .

Consider $g\in\mathbb{Q}$ with $g\neq 0$ and $n\in\mathbb{Z}$. Then, $ng\equiv 0\pmod{2\pi}$ only when $n=0$.

We now prove that given a non-zero query $\mathbf{q}$ and a relative distance $r$, we can find a key $\mathbf{k}$ such that the activation is maximal at distance $r$. We note that the non-zero query assumption is minor as a zero query would imply with RoPE perfectly uniform attention as all activations would be equal to each other. The proof is by construction and involves rotating each $\mathbf{q}^{(k)}$ chunk by $\rho(g_{k})^{-r}$. We then apply Lemma A.1 to show that the relative distance $r$ will give the maximum activation.

Proposition 3.1. *Given any (non-zero) query $\mathbf{q}$ and any relative distance $r\in\mathbb{Z}$, we can find a key $\mathbf{k}$ such that the softmax value is largest at distance $r$ with RoPE.*

###### Proof.

Consider a distance $r$, a query $\mathbf{q}=\bm{\psi}$ that is non zero by assumption, and a key such that $\mathbf{k}^{(k)}=\rho(g_{k})^{r}\bm{\psi}^{(k)}$ or equivalently $\mathbf{k}=\mathbf{R}^{r}\bm{\psi}$. Assume that the query is at position $i$ and the key at position $j\leq i$, we compute:

$$ $\displaystyle\mathbf{q}_{i}^{\top}\mathbf{R}^{j-i}\mathbf{k}_{j}$ $\displaystyle=\bm{\psi}^{\top}\mathbf{R}^{(j-i)+r}\bm{\psi}$ $\displaystyle=\sum_{k=1\dots d/2}\left(\bm{\psi}^{(k)}\right)^{\top}\rho(g_{k} )^{(j-i)+r}\bm{\psi}^{(k)}$ $\displaystyle=\sum_{k=1\dots d/2}\left\lVert\bm{\psi}^{(k)}\right\rVert^{2} \cos\left((j-i-r)g_{k}\right),$ $$

We now note that the maximum will only be achieved, by Lemma A.1, when $j-i=-r$, such that all $\cos\left((j-i-r)g_{k}\right)=\cos(0)=1$, completing the proof. We note the minus sign coming from our convention that $j\leq i$, implying that $j-i\leq 0$.
∎

We will later show that Gemma $7B$ seems to be learning a construction similar to the one above in practice in Section D.1. We next show that given queries and keys sampled from a standard multivariate Gaussian, their expectation will be 0 0. This follows from the fact that the standard multivariate Gaussian is invariant to rotations (isotropic).

Proposition 3.2. *Let $\mathbf{q},\mathbf{k}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. Then, for any relative distance $r\in\mathbb{Z}$, we have that:*

$$ $\mathop{\mathbb{E}}_{\mathbf{q},\mathbf{k}\sim\mathcal{N}(\mathbf{0},\mathbf{I })}\left[\mathbf{q}^{\top}\mathbf{R}^{r}\mathbf{k}\right]=0.$ $$

###### Proof.

We will use the well-known fact that if $X\sim\mathcal{N}(\mathbf{\bm{\mu}},\mathbf{\Sigma})$ is a (multivariate) Gaussian random variable, then the linear transformation $Y=\mathbf{A}X+\mathbf{b}$ gives another Gaussian random variable such that $Y\sim\mathcal{N}(\mathbf{A}\bm{\mu}+\mathbf{b},\mathbf{A}\mathbf{\Sigma}
\mathbf{A}^{\top})$. We start by computing:

$$ $\displaystyle\mathop{\mathbb{E}}_{\mathbf{q},\mathbf{k}\sim\mathcal{N}(\mathbf {0},\mathbf{I})}\left[\mathbf{q}^{\top}\mathbf{R}^{r}\mathbf{k}\right]$ $\displaystyle=\mathop{\mathbb{E}}_{\mathbf{q},\mathbf{k}\sim\mathcal{N}( \mathbf{0},\mathbf{I})}\left[\sum_{k}\left(\mathbf{q}^{(k)}\right)^{\top}\rho( g_{k})\mathbf{k}^{(k)}\right].$ $$

Let $\tilde{\mathbf{k}}^{(k)}=\rho(g_{k})\mathbf{k}^{(k)}$. As $\mathbf{k}^{(k)}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, we have that $\mathop{\mathbb{E}}[\tilde{\mathbf{k}}^{(k)}]=\rho(g_{k})\mathbf{0}=\mathbf{0}$ and $\mathbb{V}[\tilde{\mathbf{k}}^{(k)}]=\rho(g_{k})\rho(g_{k})^{\top}=\mathbf{I}$ as $\rho(g_{k})$ is orthogonal. This implies that also $\tilde{\mathbf{k}}^{(k)}$ is a standard Gaussian random variable, i.e. $\tilde{\mathbf{k}}^{(k)}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. We proceed with the change of variables:

$$ $\displaystyle\mathop{\mathbb{E}}_{\mathbf{q},\mathbf{k}\sim\mathcal{N}(\mathbf {0},\mathbf{I})}\left[\sum_{k}\left(\mathbf{q}^{(k)}\right)^{\top}\rho(g_{k}) \mathbf{k}^{(k)}\right]$ $\displaystyle=\mathop{\mathbb{E}}_{\mathbf{q},\tilde{\mathbf{k}}\sim\mathcal{N }(\mathbf{0},\mathbf{I})}\left[\sum_{k}\left(\mathbf{q}^{(k)}\right)^{\top} \tilde{\mathbf{k}}^{(k)}\right]$ $\displaystyle=\mathop{\mathbb{E}}_{\mathbf{q},\tilde{\mathbf{k}}\sim\mathcal{N }(\mathbf{0},\mathbf{I})}\left[\mathbf{q}^{\top}\tilde{\mathbf{k}}\right]$ $\displaystyle=\sum_{k}\mathop{\mathbb{E}}_{\mathbf{q}_{i}\sim\mathcal{N}(0,1)} \left[\mathbf{q}_{i}\right]\mathop{\mathbb{E}}_{\mathbf{k}_{i}\sim\mathcal{N}( 0,1)}\left[\mathbf{k}_{i}\right]=0.$ $$

∎

###### Lemma A.1 .

###### Proof.

###### Proof.

### A.2 Positional attention patterns.

We focus on the proofs regarding positional attention. We start by showing that NoPE cannot learn diagonal or off-diagonal attention. The proof is by counter-example, using a sequence that has repeated tokens.

Proposition 5.2. *An attention head with NoPE cannot learn a diagonal or off-diagonal pattern.*

###### Proof.

We start by proving the diagonal head claim. Recall that a diagonal attention head satisfies $\alpha_{i,i}>1-\epsilon$ for any $\epsilon>0$.

Consider the sequence $[\mathbf{x}_{bos},\mathbf{x}_{1},\mathbf{x}_{1}]$ that has a repeated token. We have that $\mathbf{a}_{3,3}=\langle\mathbf{q}_{3},\mathbf{k}_{3}\rangle=\langle\mathbf{q}
_{3},\mathbf{k}_{2}\rangle=\mathbf{a}_{3,2}$. We compute the attention coefficient $\alpha_{3,3}:$

$$ $\alpha_{3,3}=\frac{\exp(\mathbf{a}_{3,3})}{\exp(\mathbf{a}_{3,1})+2\exp( \mathbf{a}_{3,3})}=\frac{1}{\exp(\mathbf{a}_{3,1}-\mathbf{a}_{3,3})+2}<\frac{1 }{2},$ $$

which contradicts the requirement that $\alpha_{3,3}>1-\epsilon$ for any $\epsilon>0$.

We now focus on the second claim. Recall that an off diagonal head satisfies $\alpha_{i,i-1}>1-\epsilon$ for any $\epsilon>0$. The same example contradicts this condition, as we have that $\alpha_{3,2}<\frac{1}{2}$.
∎

We instead show that RoPE is able to learn such attention patterns robustly. The construction is related to the construction used in the proof of Proposition 3.1.

Proposition 5.3. *An attention head with RoPE can learn a diagonal or off-diagonal pattern.*

###### Proof.

We start by proving the diagonal case. The construction involves setting all queries and keys equal, i.e. $\mathbf{q}_{i}=\mathbf{k}_{j}=\bm{\psi}$ with $\bm{\psi}$ not zero. To simplify notation and book-keeping, we focus on the case in which the hidden dimension is only $d=2$, i.e. only acts through a single rotation. We will then explain how to generalise to $d>2$. We call the rotation $g$ and its matrix representation $\rho(g)$.

We now note without RoPE that:

$$ $\langle\mathbf{q}_{i},\mathbf{k}_{j}\rangle=\left\lVert\mathbf{q}_{i}\right \rVert\left\lVert\mathbf{k}_{j}\right\rVert\cos(\theta_{i,j})=\left\lVert\bm{ \psi}\right\rVert^{2},$ $$

with $\theta_{i,j}$ the angle between $\mathbf{q}_{i}$, $\mathbf{k}_{j}$ which is 0 0 as we have by construction that $\mathbf{q}_{i}=\mathbf{k}_{j}=\bm{\psi}$. When using RoPE, the dot product becomes:

$$ $\displaystyle\mathbf{a}_{i,j}$ $\displaystyle=\langle\rho(g)^{i}\mathbf{q}_{i},\rho(g)^{j}\mathbf{k}_{j}\rangle$ $\displaystyle=\left\lVert\rho(g)^{i}\mathbf{q}_{i}\right\rVert\left\lVert\rho( g)^{j}\mathbf{k}_{j}\right\rVert\cos\left(\left(j-i\right)g+\theta_{i,j}\right)$ $\displaystyle=\left\lVert\bm{\psi}\right\rVert^{2}\cos\left(\left(j-i\right)g \right),$ $$

where we use the fact that $\left\lVert\rho(g)^{i}\mathbf{x}\right\rVert=\left\lVert\mathbf{x}\right\rVert$ as $\rho(g)^{i}$ is an isometry. Due to Lemma A.1, we have that $\left(j-i\right)g\equiv 0\pmod{2\pi}$ only when $j=i$. This implies that $\mathbf{a}_{ii}=\left\lVert\bm{\psi}\right\rVert^{2}$ and $\mathbf{a}_{ij}<\mathbf{a}_{ii}$. We now compute $\alpha_{i,i}$:

$$ $\displaystyle\alpha_{i,i}$ $\displaystyle=\frac{\exp(\mathbf{a}_{i,i})}{\sum_{k<i}\exp(\mathbf{a}_{i,k})+ \exp(\mathbf{a}_{i,i})}$ $\displaystyle=\frac{\exp\left(\left\lVert\bm{\psi}\right\rVert^{2}\right)}{ \sum_{k<i}\exp\left(\left\lVert\bm{\psi}\right\rVert^{2}\cos\left(\left(k-i \right)g\right)\right)+\exp\left(\left\lVert\bm{\psi}\right\rVert^{2}\right)}$ $\displaystyle=\frac{1}{1+\sum_{k<i}\exp\left(\left\lVert\bm{\psi}\right\rVert^ {2}\cos\left(\left(k-i\right)g\right)-\left\lVert\bm{\psi}\right\rVert^{2} \right)}$ $\displaystyle=\frac{1}{1+\sum_{k<i}\exp\left(\left\lVert\bm{\psi}\right\rVert^ {2}\left(\cos\left(\left(k-i\right)g\right)-1\right)\right)}.$ $$

We observe that $\cos\left(\left(k-i\right)g\right)<1$ for $k\neq i$, implying that $\left\lVert\bm{\psi}\right\rVert^{2}\left(\cos\left(\left(k-i\right)g\right)-1
\right)<0$. In particular, we have that:

$$ $\sup_{\left\lVert\bm{\psi}\right\rVert^{2}\to\infty}\frac{1}{1+\sum_{k<i}\exp \left(\left\lVert\bm{\psi}\right\rVert^{2}\left(\cos\left(\left(k-i\right)g \right)-1\right)\right)}=1,$ $$

meaning that indeed our construction satisfies $\alpha_{i,i}>1-\epsilon$ for any $\epsilon>0$, with $\epsilon$ a function of $\left\lVert\bm{\psi}\right\rVert^{2}$.

We now focus on the off-diagonal case. Here we set all queries $\mathbf{q}_{i}=\bm{\psi}$ and all keys $\mathbf{k}_{i}=\rho(g)\bm{\psi}=\bm{\phi}$, for $\bm{\psi}$ non-zero. We now observe:

$$ $\displaystyle\mathbf{a}_{i,i-1}$ $\displaystyle=\left\langle\rho(g)^{i}\mathbf{q}_{i},\rho(g)^{i-1}\mathbf{k}_{i }\right\rangle$ $\displaystyle=\left\langle\rho(g)^{i}\bm{\psi},\rho(g)^{i-1}\rho(g)\bm{\psi}\right\rangle$ $\displaystyle=\left\langle\rho(g)^{i}\bm{\psi},\rho(g)^{i}\bm{\psi}\right\rangle$ $\displaystyle=\left\lVert\rho(g)^{i}\bm{\psi}\right\rVert^{2}\cos\left(\left(i -i\right)g\right)$ $\displaystyle=\left\lVert\bm{\psi}\right\rVert^{2}.$ $$

The same reasoning for the diagonal case then follows. For the high-dimensional case, it is sufficient to take $\mathbf{k}_{i}=\mathbf{R}\bm{\psi}$ or equivalently $\mathbf{k}_{i}^{(k)}=\rho(g_{k})\bm{\psi}^{(k)}$.
∎

In practice, Gemma 7B learns a construction very similar to that used in Theorem 5.3, as we will show in Section D. Further, as softmax tends to be rather ‘sharp’, the norm does not in practice have to become very large for the construction to be robust. This can be seen for example in Figure 6, where the chunk norms are not above $10$, but the attention matrix allocates robustly attention on the diagonal.

###### Proof.

###### Proof.

### A.3 Semantic attention.

The next statement is a useful result on irrational rotations, namely that they are *dense* in $[0,2\pi]$. Intuitively, this means that given an irrational rotation $g$, we can find an integer $n$ such that $ng\pmod{2\pi}$ is arbitrarily close to any $g^{\prime}\in[0,2\pi]$. The statement is commonly shown via a pigeon-hole argument.

###### Lemma A.2 .

The subset $\{ng\pmod{2\pi}:n\in\mathbb{Z}\}\subset[0,2\pi]$ is dense in $[0,2\pi]$.

We now show a simple, but useful Lemma regarding a required condition for ‘sharpness’, namely that the activation entering the softmax has to be the largest to have an attention coefficient larger than $1/2$.

###### Lemma A.3 .

Let $\mathbf{a}\in\mathbb{R}^{n}$ be a sequence of $n$ activations entering a softmax such that $\alpha=\operatorname{softmax}\left(\mathbf{a}\right)\in\mathbb{R}^{n}$. Denote by $\mathbf{a}_{i},\alpha_{i}$ the $i$-th activation and attention coefficient, respectively. If $\mathbf{a}_{i}<\mathbf{a}_{j}$ for some $j$, then $\alpha_{i}\leq\frac{1}{2}$.

###### Proof.

We have that $\alpha_{i}=\frac{\exp\left(\mathbf{a}_{i}\right)}{Z}$ with $Z=\sum_{k}\exp\left(\mathbf{a}_{k}\right)$. We first note that if $\mathbf{a}_{i}<\mathbf{a}_{j}$, then $\alpha_{i}<\alpha_{j}$ as $Z$ is constant and $\exp$ is monotonically increasing. Now assume by contradiction that $\alpha_{i}>\frac{1}{2}$, we then have that $\alpha_{j}>\alpha_{i}>\frac{1}{2}$, implying that $\alpha_{i}+\alpha_{j}>1$. This violates the sum-to-one constraint of the softmax function, implying that $\alpha_{i}\leq\frac{1}{2}$ if $\mathbf{a}_{i}<\mathbf{a}_{j}$ for some $j$.
∎

We are now ready to show the result regarding the instability of RoPE. The intuition of the proof is that depending on the position in the sequence, the rotation can make the dot product positive or negative. This means that we can always find permutations, given a long enough sequence, that will make the activation of the desired element smaller than another element.

Theorem 6.1. *Let the hidden dimension be $d=2$, i.e. there is only a single RoPE frequency $g_{1}$. Consider a sequence $\mathbf{x}_{bos},\mathbf{x}_{1},\dots,\mathbf{x}_{N}$ with a target token at some position $\mathbf{x}_{n}$. Then for large enough $N$, the attention head with RoPE is unable to attend to such a token such that $\alpha_{i,n}>1-\epsilon$ for any $\epsilon>0$ and any $1\leq n\leq N$.*

###### Proof.

Assume the desired element is at position $n$, that the head is correctly attending to it such that $\alpha_{i,n}>1-\epsilon$ for all $\epsilon>0$ and $i\geq n$, and that $N$ is large enough. By Lemma A.3, the fact that $\alpha_{i,n}>1-\epsilon$ implies that $\mathbf{a}_{i,n}>\mathbf{a}_{i,j}$ for all $j$. As $\mathbf{a}_{i,n}$ has to be the largest element, we have two cases. Either $\mathbf{a}_{i,n}\leq 0$ is non-positive, implying that $\mathbf{a}_{i,j}<0$ for all $j\neq k$, or $\mathbf{a}_{i,n}>0$ is positive.

We start with the first case, where $\mathbf{a}_{i,n}\leq 0$. Consider an element at position $k\neq n$. We have that:

$$ $\mathbf{a}_{i,k}=\left\lVert\mathbf{q}_{i}\right\rVert\left\lVert\mathbf{k}_{k }\right\rVert\cos\left(\theta_{i,k}+g_{1}\left(k-i\right)\right),$ $$

where $\theta_{i,k}$ is the angle between $\mathbf{q}_{i}$ and $\mathbf{k}_{k}$. As $\mathbf{a}_{i,k}<0$, we must have that $\theta_{i,k}+g_{1}\left(k-i\right)\pmod{2\pi}\in\left(\frac{\pi}{2},\frac{3\pi
}{2}\right)$ (making the cosine negative). It is sufficient to *swap* the element at position $k$ with some element $k^{\prime}$ such that $\theta_{i,k}+g_{1}\left(k^{\prime}-i\right)\pmod{2\pi}\in\left(0,\frac{\pi}{2}
\right)\cup\left(\frac{3\pi}{2},2\pi\right)$, which is always possible for large enough $N$ by Lemma A.2. This swap would make $\mathbf{a}_{i,k^{\prime}}>0$, which by Lemma A.3, implies that now $\alpha_{i,k}\leq\frac{1}{2}$.

In other words, above we have shown that if an attention head can attend successfully at a certain position on a sequence that is long enough, certain swaps of the tokens in the sequence guarantee that this attention head would attend to the wrong element. In particular, if all activations are negative, it is sufficient to perform a swap that makes a non-target element positive.

We now focus on the case $\mathbf{a}_{i,n}>0$. The principle is the same, but we need to apply two swaps. The first swaps $n$ with $k\neq n$, such that for the target token now at position $k$ we have that $\mathbf{a}_{i,k}<0$ after the swap. If after the swap $\mathbf{a}_{i,k}$ is not the largest element anymore, we are done. If not, we need to perform a second swap with $k^{\prime}\neq k^{\prime\prime}\neq k$, which swaps an element at position $k^{\prime}$ with $k^{\prime\prime}$ such that now $\mathbf{a}_{i,k^{\prime}}>0$. This forces the target element to not be the largest anymore. We can then apply again Lemma A.3 to complete the proof.

In other words, if some activations are positive, we need to perform two swaps. First we swap the target activation such that it becomes negative. Then, we are either done or we need to perform another swap making a non-target activation positive. These two cases imply the desired statement.
∎

###### Lemma A.2 .

###### Lemma A.3 .

###### Proof.

###### Proof.

## Appendix B Additional discussions

In this section, we provide additional discussions omitted from the main part of the manuscript. In Section B.1, we comment on how we believe our work and findings relate to positional encodings other than RoPE. In Section B.2, we comment on the work of Ruoss et al. (), which proposes a process that randomises the positions of tokens.

### B.1 Other positional encodings

In our work, we focused on RoPE as we believe it is one of the most relevant positional encodings used today, being used for example by Gemma and Llama 3 models. We also find RoPE to have a particularly interesting geometric structure, that ties nicely with the dot product. These two reasons motivated our in-depth study of the method.

Regardless, we believe that our approach and findings can be transferred more broadly. First of all, we would like to point out the strong similarity between RoPE and the sinusoidal absolute encodings (APE) proposed by Vaswani et al. (), with both encoding position through a range of frequencies of sines and cosines. The fact that APE is added to the queries and keys and not applied as a linear map is a significant difference and would be interesting to explore in more detail. We suspect however that our findings relating to the frequency usage to be somewhat transferable, with the lower frequencies being used to carry semantics and the high frequencies providing a method to encode positional attention patterns.

A popular method that is less similar to RoPE is Alibi (Press et al., ). We suspect that prior to this work, the common belief would be that Alibi and RoPE behaved in an analogous way, by decaying attention coefficients with distance. This is clearly true with Alibi. We however showed in this work that this is not the case with RoPE. Analysing Alibi in a way similar to this work would therefore be an interesting future direction of research to better understanding what kind of patterns these different positional encodings encourage. Overall, we hope that the understanding gained through our work will help the community to better classify different positional encodings.

### B.2 Randomised Positional Encodings

A recent work by Ruoss et al. () proposes to ‘randomise’ the positions of RoPE, showing that this process helps to boost long-context generalisation performance. In particular, they propose given some maximum training context $N$, to choose some $L>N$, and to then sample without replacement from $1\dots L$ $N$ elements, ordering them from smallest to largest, providing them as the positions during training and iteratively re-sampling. The motivation is that this process would allow the model to train using larger positions, helping with out-of-distribution generalisation to long sequences. We believe instead that this technique is instead helpful for a more subtle reason.

Providing during training random positions invites the model to learn some soft form of invariance to relative distance. The model has to in fact learn to produce the same output given different sampled positions when optimising the training loss. While pure invariance with RoPE cannot be achieved, the best the model can do when using RoPE is to rely on the very lowest frequencies to minimise the training loss. In other words, randomising the positions according to Ruoss et al. () pushes the model to use the lowest frequencies. Testing this is laborious and outside the scope of our work, but we believe this to be an interesting future direction.

Furthermore, this randomised process has another side-effect, namely it increases the average relative distance. The effect of this increase in relative distance is that it will increase the decay of the activations. In Figure 8 (a), we show the effect of increasing the sampling upper bound $L$ on a constant vector of all-ones (up to normalisation). We see how increasing $L$ makes the decay occur faster, but that all curves seem to finally converge around 0 0. Instead in (b), we show that this does not affect the activation patterns when sampling from a Gaussian. This serves to counter the original claim that this process helps as the model will see larger relative distances. In fact, we see that as we increase the relative distance, the effect on the activations becomes more and more indistinguishable.

Figure: (a) RoPE and Random RoPE applied to constant queries and keys.
Refer to caption: extracted/5911145/figures/constant-qks-ablation.png

### B.3 Limitations of our work

As with other mechanistic interpretations, is not clear that our description covers the entire spectrum of how LLMs use the RoPE encoding. While, the magnitude of the key entries corresponding to middle band of frequencies is considerably smaller on average, it is not 0 0, and therefore it could still play a role. Our ablation of removing the low frequencies, while promising and helping to significantly validate our claims, could be improved with larger scale experimentation to asses the improvements specifically on large contexts – which we believe to be an important application of this work. Of course, such experimentation would be much more resource intensive.

## Appendix C Supplementary p 𝑝 p italic_p -RoPE Experimental Details and Results

In our additional experiments we train Gemma 2B from scratch separately on two datasets: English Wikipedia (Wiki) and FlanV2.

Wiki is a dataset based on English Wikipedia articles, built from the Wikipedia dump. Each sample contains the contents of a full Wikipedia article, with processing done to strip markdown and unwanted sections. The dataset is available at: https://www.tensorflow.org/datasets/catalog/wikipedia. There are $6{,}672{,}479$ documents and the total size is of $19.88$ GiB. We held out $10\%$ of the data randomly for the validation split.

FlanV2 was introduced by Longpre et al. (). It is a dataset for instruction tuning which combines collections from FLAN, P3/T0, and Natural Instructions with dialog, program synthesis, and complex reasoning tasks. The dataset contains $15{,}000{,}000$ samples.

In all experiments we train for $10{,}000$ steps using a batch size of $512$ and a sequence length of $8{,}192$. We fix the wavelength to the standard value of $10{,}000$. We use the standard Gemma 2B architecture (Gemma Team et al., ). For $p$-RoPE, we use the algorithm apply_p_rope described below.

[commandchars=
{}]
def apply_p_rope(
inputs: jax.Array, # [B, L]
positions: jax.Array, # [B, L]
head_dim: int,
max_wavelength: int = _MAX_WAVELENGTH,
rope_percentage: float = 1.0,
) -> jax.Array:
"""Applies p-RoPE."""
rope_angles = int(rope_percentage * head_dim // 2)
nope_angles = head_dim // 2 - rope_angles

fraction = 2. * jnp.arange(0, rope_angles) / head_dim
timescale = max_wavelength**fraction
timescale = jnp.pad(
max_wavelength**fraction,
(0, nope_angles),
mode=’constant’,
constant_values=(0, jnp.inf)
)

sinusoid_inp = (
positions[…, jnp.newaxis] / timescale[jnp.newaxis, jnp.newaxis, :]
)
sinusoid_inp = sinusoid_inp[…, jnp.newaxis, :]
sin = jnp.sin(sinusoid_inp)
cos = jnp.cos(sinusoid_inp)

first_half, second_half = jnp.split(inputs, 2, axis=-1)
first_part = first_half * cos - second_half * sin
second_part = second_half * cos + first_half * sin
out = jnp.concatenate([first_part, second_part], axis=-1)
return out.astype(inputs.dtype)

## Appendix D Supplementary Gemma 7B Results

We provide supplementary results for our analysis of Gemma 7B. We start by providing, in Section D.1, evidence supporting our claims regarding the types of constructions being learnt by Gemma 7B. We end by providing in Section D.2, the query, key, and attention pattern plots for all of the attention heads we study in the main part of the manuscript.

### D.1 Constructions learnt by Gemma 7B

We explore the various constructions discussed in the main part of the manuscript. As a small aside for some geometric intuition, we note that the queries and keys in Gemma 7B are $256$-dimensional. If they were randomly sampled from a Gaussian, their dot product should be very close to 0 0. This follows from the fact that high-dimensional Gaussian random vectors are close to orthogonal.

##### Diagonal heads.

We start by focusing on diagonal attention heads. In particular, we focus on the diagonal attention heads from Figure 11 and Figure 12. The hypothesis we want to verify is that the attention heads are learning to set queries and keys approximately equal to each-other, following the construction from Theorem 5.3.

To measure how close the queries and keys are, we leverage the Cauchy-Schwarz inequality. In particular, we know that:

$$ $\frac{1}{\sqrt{d}}\mathbf{q}_{i}^{\top}\mathbf{R}^{j-i}\mathbf{k}_{j}=\frac{1} {\sqrt{d}}\left\lVert\mathbf{q}_{i}\right\rVert\left\lVert\mathbf{k}_{j}\right \rVert\cos\left(\theta_{i,j}+g_{i,j}\right)\leq\frac{1}{\sqrt{d}}\left\lVert \mathbf{q}_{i}\right\rVert\left\lVert\mathbf{k}_{j}\right\rVert,$ $$

meaning that the quantity $\left\lVert\mathbf{q}_{i}\right\rVert\left\lVert\mathbf{k}_{j}\right\rVert$ serves as an upper-bound, achieved when the queries and keys are perfectly aligned. We highlight that it is important to introduce the normalisation factor $1/\sqrt{d}$ to get a meaningful bound. We then compare the activations (logits) to this upper bound. If the queries and keys are indeed aligned, the logits should be close to the upper bound. In Table 3 and Table 4, we showcase that this seems to be the case for the diagonal attention heads occurring at Layer 1 (Head 5) and Layer 27 (Head 8). The previous-token activations instead in comparison decay significantly in these heads.

**Table 3: Activations for a diagonal head (Layer 1 Head 5). The upper bound is given by Cauchy-Schwarz. The diagonal activations are close to the upper-bound, meaning that the head is setting queries and keys to be very aligned to each other. The previous-token activations instead in comparison decay significantly.**
| Token | BOS | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Upper Bound | 3.74 | 182.34 | 203.65 | 483.97 | 141.48 | 130.71 | 168.52 | 122.31 |
| Diagonal | 3.25 | 119.60 | 156.93 | 383.00 | 98.47 | 92.15 | 137.10 | 90.71 |
| Previous-token | N/A | -8.45 | 76.67 | 82.13 | 29.01 | 45.33 | 73.04 | 55.25 |

**Table 4: Activations for a diagonal head (Layer 27 Head 8). The upper bound is given by Cauchy-Schwarz. The diagonal activations are close to the upper-bound, meaning that the head is setting queries and keys to be very aligned to each other.**
| Token | BOS | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Upper Bound | 7.91 | 64.42 | 112.43 | 51.97 | 52.69 | 67.10 | 116.41 | 67.37 |
| Diagonal | 6.05 | 55.84 | 96.52 | 45.75 | 44.00 | 53.83 | 105.47 | 57.78 |
| Previous-token | N/A | -9.63 | 32.51 | 29.49 | 25.96 | 20.08 | 19.95 | 6.07 |

##### Previous-token head.

We perform a similar measurement for the previous-token head (Figure 13). An analogous situation occurs in which the previous-token activations seem to be much closer to the upper bound given by Cauchy-Schwarz, as seen in Table 5. We find that in comparison to the diagonal heads, the gap to the upper bound is larger. This is expected, as the head has to learn to model $\mathbf{R}$, which is a more complicated block-diagonal rotation matrix. Nevertheless, some tokens in particular are surprisingly aligned, supporting our claims.

**Table 5: Activations for a previous-token head (Layer 1 Head 8). The upper bound is given by Cauchy-Schwarz. While the diagonal activations seem to be close to 0 0, the previous-token activations are significantly larger and closer to the upper-bound.**
| Token | BOS | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Upper Bound | 3.85 | 76.54 | 79.13 | 102.41 | 61.34 | 54.54 | 56.62 | 57.2 |
| Diagonal | -1.58 | 3.29 | 5.21 | -1.46 | 4.23 | 4.61 | 2.47 | 3.08 |
| Previous-token | N/A | 31.04 | 35.31 | 18.91 | 19.72 | 18.96 | 17.46 | 13.67 |

##### Apostrophe head.

We now analyse the construction of the ‘apostrophe attention head’ – see Section 6. In particular, we identified this head as one that attends to a previous-token apostrophe if it exists and to the BOS token, otherwise. We are particularly interested in understanding what the low frequency band is being used for and how it acts as a semantic channel. We will show in this section that this semantic channel is being used to provide the BOS attention part of the pattern. In particular, this band is very positive when the token being attended to is the BOS token, and very negative otherwise. In Figure 9, we show the raw activations of the attention head. The column corresponding to the BOS token has activations of value $\approx 2$, while the apostrophe tokens on the off-diagonal token have attention values of $\approx 7$. The other values instead range from $\approx 0$ to the lowest of $\approx-12$, with all values being negative. These activations show how the head operates. It will default to attending to the BOS token, but will instead be dominated by an apostrophe token if it appears on the off-diagonal as it will have a more positive activation.

We now study the contribution from the low frequency, clearly visible in Figure 14 in the queries and keys. The semantic channel occurs at $g_{119}=10{,}000^{-2(119-1)/256}\approx 0.0002$ – one of the lowest frequencies out of the $128$ frequencies ($d=256$) in Gemma 7B. As the context length is of $8k$, we get that the maximal rotation available is of $8000g_{119}\approx 1.64\approx\frac{\pi}{2}$ radians, meaning that this channel is relatively stable amounting to a quarter of a full rotation, even at the largest context lengths. By Cauchy-Schwarz, as the norms corresponding to these frequencies are very large on average compared to the other frequencies, we expect this to ‘dominate’ the dot-product.

The queries corresponding to this frequency for the non BOS tokens are all set approximately to the same vector $\mathbf{q}^{(119)}_{not\texttt{BOS}}\approx[-4.1,11.3]^{\top}$ and similarly the keys $\mathbf{k}^{(119)}_{not\texttt{BOS}}\approx[11.2,-3.5]^{\top}$. Instead, for we have that $\mathbf{q}^{(119)}_{\texttt{BOS}}\approx[0.7,-1.9]^{\top}$, and $\mathbf{k}^{(119)}_{\texttt{BOS}}\approx[-2.5,1.3]^{\top}$. We then note that $\left(\mathbf{q}^{(119)}_{not\texttt{BOS}}\right)^{\top}\mathbf{k}^{(119)}_{
not\texttt{BOS}}\approx-85.5$, while $\left(\mathbf{q}^{(119)}_{not\texttt{BOS}}\right)^{\top}\mathbf{k}^{(119)}_{
\texttt{BOS}}\approx+24.9$. In other words, this channel contributes a very negative activation for tokens attending to non-BOS tokens, and very positive activations to tokens attending to the BOS token. As the the underlying frequencies are very low, this mechanism will be stable given the limited context length of Gemma 7B. We however expect this mechanism to break over long enough sequences, in accordance with Theorem 6.1. This provides evidence supporting our claims that the lowest frequencies are used as semantic channels, as this kind of behaviour will be most robust exactly for those channels that provide the lowest frequencies.

Figure: Figure 9: Activation pattern of the ‘apostrophe head’ in Gemma 7B. The upper triangular mask is artificially set to the value of $-10$ for clarity of the visualisation, but in reality takes a value of $\approx-10^{30}$. Token 0 0 is the BOS token. The $3$ darkest activations on the off-diagonal correspond to tokens after an apostrophe token attending to the apostrophe token.
Refer to caption: extracted/5911145/figures/activation-pattern-apostrophe.png

### D.2 Analysis of patterns in different attention heads

Figure: (a) Mean query norm distribution at each layer.
Refer to caption: extracted/5911145/figures/query-chunk-distribution.png

Figure: (a)
Refer to caption: extracted/5911145/figures/gemma-q-layer1-head5.png

Figure: (a)
Refer to caption: extracted/5911145/figures/gemma-q-layer27-head8.png

Figure: (a)
Refer to caption: extracted/5911145/figures/gemma-q-layer1-head8.png

Figure: (a)
Refer to caption: extracted/5911145/figures/gemma-q-layer1-head10.png

Figure: (a)
Refer to caption: extracted/5911145/figures/gemma-q-layer1-head13.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. In practice, do the phenomena observed on gemma-7b apply to other LLM backbones, such as llama-2 or llama-3?
2. The phenomena observed in this paper are based on what scale and type of data? Could the findings be validated across various tasks, such as language modeling, code, or QA?
3. In Line 190, the authors claim that NoPE has strong OOD capabilities, but Kazemnejad et al., 2024 only validated this for sequences of length up to 50, where NoPE performed slightly better than other PEs, without exhibiting exceptionally strong OOD performance (refer to their Figure 3). The following works explored NoPE's extrapolation, and it can be seen that the generalization ability of the NoPE baseline is limited:
> Length Generalization of Causal Transformers without Position Encoding (Wang et al., Findings 2024)
> 
> [Neurips24 spotlight] Exploring Context Window of Large Language Models via Decomposed Positional Vectors
4. In Line 497, the authors state, "p-RoPE is in spirit similar to the idea of increasing the wavelength of RoPE from 10,000 to 500,000," so I suggest that Table 2 should include RoPE with a base of 500,000.
5. How do the observations in Section 4 change across different RoPE variations, such as 0.25-RoPE, 0.75-RoPE, RoPE_10000, and RoPE_500000? For example, in Figure 3, how does the norm distribution of low frequencies vary across these models?
6. In Line 726, this proof only applies to the first layer of NoPE’s attention heads. Starting from the second layer, the assumption 
$a_{3,3}=<q_3, k_3>=<q_3, k_2>=a_{3,2}$ 
no longer holds.
7. Attention heads with specific patterns have been widely studied. Besides the patterns discussed in this paper (e.g., diagonal or previous-token focus), could the authors observe and analyze other representative attention patterns, such as special token focus, punctuation focus, and locality focus? Please refer to typical patterns in the following paper:
> [ICLR24] Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
soundness: 3
strengths: 1. This paper successfully challenges, both theoretically and empirically, the traditional view that RoPE attenuates attention weights as relative distance between tokens increases. 
2. The authors' hypothesis about the roles of the high-frequency and low-frequency components of RoPE is novel and insightful.
summary: This paper delves into the role of Position Embedding (PE) in LLMs, challenging the traditional view that RoPE primarily attenuates attention weights as the relative distance between words increases. It proposes a new hypothesis that RoPE constructs position-attention patterns (e.g., diagonal or previous-token focus) using high-frequency components while leveraging low frequencies to convey semantic information. Several case studies and theoretical analyses support this hypothesis.

Overall, the paper provides insightful findings and hypotheses on a key component of LLMs: position encoding. However, the arguments mainly rely on case studies with gemma-7B, and the experiments lack diversity in the foundation models (e.g., missing LLaMA series) and tasks (e.g., language model vs. QA vs. code...). Additionally, some parts of the proofs contain errors.
weaknesses: 1. The experimental validation lacks diversity in both foundation models and datasets.
2. Some perspectives and proofs regarding NoPE contain errors, while they don't affect the main conclusion, they may mislead readers.
3. The discussion of related work is insufficiently thorough, and few papers are cited (only about one page).
4. Throughout Section 4, the authors conceal a core assumption: that attention scores are interpretable or meaningful. Higher attention scores for certain tokens imply a meaningful preference in the model, giving special significance to the larger norms in Equation at Line 257. I want to point out that this assumption is still being debated, and I suggest that the authors make it explicit.
> Is Attention Explanation? An Introduction to the Debate (Bibal et al., ACL 2022)

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. I have some doubts about the display of Figure 2. Normally, for the same qk, different relative distances should have a gradually decreasing effect. But does different qk introduce different variables, making the comparison unfair?
2. Is there any empirical experiment on how many frequencies are cut off for the best effect?
3. Will removing low-frequency RoPE make the effect on short text worse?
soundness: 2
strengths: 1. The paper provides a fresh perspective on RoPE, questioning existing assumptions and offering new explanations for its effectiveness.
2. The authors present mathematical proofs to support their claims, enhancing the credibility of their findings.
3. The use of the Gemma 7B model for empirical analysis adds practical relevance to the theoretical insights.
summary: This paper investigates the role of Rotary Positional Encodings (RoPE) in Transformer-based Large Language Models (LLMs). The authors challenge the common belief that RoPE's usefulness comes from its ability to decay token dependency with increasing relative distance. Instead, they explore how different frequencies in RoPE are utilized, particularly within the Gemma 7B model. The paper provides both theoretical and empirical analyses, proposes a modified RoPE, and highlights the importance of understanding positional encodings for scaling LLMs.
weaknesses: 1. Although the observed phenomena and mathematical proofs can support the paper's point of view, the experimental performance does not seem good enough. The paper hopes to adapt to any context length, but the actual experimental results only have one result on 8K. And the evaluation of PPL is not comprehensive enough.
2. At the semantic level, the results of the models in Table 2 should be compared on the general benchmark or other tasks that are more representative of semantics, which will be more convincing.
3. At the positional level, it should be compared with similar experiments such as needle in a haystack or Ruler on long contexts to prove its long context expansion ability.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: 1. Pre-training model from scratch is expensive and not always feasible. I was wondering if you can instead continuously train other (Gemma-7B, Llama-3 1/3/8B, etc) models for fewer steps but with p-RoPE approach? Do you think it would work or no and why?

2. Please, correct me if I'm wrong, but in the proof of Proposition 3.1:
for k>1, g_k=theta^{-2(k-1)/d} is not necessarily rational, but algebraic. Therefore, Lemma A.1 should be stated not for rational g, but for algebraic g, and it should use the fact that pi is transcendental.
line 680, cos(j-i-r) -> cos(j-i+r)
line 680, the comma in the displayed equation should be a period.
line 684, cos(j-i-r) -> cos(j-i+r)
soundness: 3
strengths: - The authors conduct a novel theoretical and empirical study of RoPE encodings in transformer models.
- They provide detailed proofs of their main claims.
- The paper is clear and well-written.
- This study can help researchers better understand the underlying mechanisms of popular transformer architectures and encourage research into alternative improved solutions.
summary: Authors explore how transformer models use RoPE frequencies to learn semantic and positional information. Authors provide a theoretical proof that RoPE does not force the decay in attention coefficient with distance, but instead can create specific patterns. In their experimental study of the Gemma 7B model, authors show that transformers mostly rely on low frequencies, while some heads display high frequency bands, mostly in 1st and last layers of the model. Authors further show that high-frequencies in RoPE provide a mechanism to encode positional information. Low frequencies are used as information channels that are not robust over long context. Finally, authors propose p-RoPE encoding that cuts low frequencies and can help to improve model's performance.
weaknesses: - The empirical study is limited to a single Gemma architecture. While this is unlikely, some results may be artifacts of the specific model selected.
- In Section 3 authors train 2B model and show improvements on validation perplexity. While these results are positive, perplexity improvements do not always results in overall improvements in model's abilities. Authors could provide evaluation results on popular benchmarks* to build more convincing picture.


* see, for ex, Section 2.3 in https://arxiv.org/pdf/2307.09288

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: Why did the authors provide an ablation study only for the lower frequencies rather than for both low and high frequencies?
soundness: 2
strengths: Pros:

1. The paper offers a fresh discovery of Rotary Positional Encoding (RoPE), challenging the belief that it primarily helps decay attention coefficients with distance.

2. The paper gives an analysis of high and low RoPE frequencies, their roles in positional and semantic attention, and an innovative RoPE modification (p-RoPE) that demonstrates improvements in some cases.

3. Every section has a summary part which makes the whole paper clear and ready to read.
summary: The paper provides practical insights into positional encoding for decoder-only models. However, further investigation is needed to establish the effectiveness and reliability of the theories presented.
weaknesses: Cons:

1. The figures in the paper are somewhat confusing. For instance, while the paper emphasizes frequency aspects, Figure 1 illustrates vectors with identical frequency differences, which does not fully align with the paper’s focus. Additionally, the results are primarily presented using heat maps, which may appear monotonous and lack of expressiveness. Given that this is a language modeling task, presenting some results in natural language format would enhance readability and interpretability.

2. (MAIN LIMITATION) The experiments are limited to decoder-only models, making it unclear whether the findings can generalize to other transformer architectures. Whether the observed improvements stem from specific model structures (e.g., encoder-only or encoder-decoder models) rather than from a comprehensive enhancement applicable to diverse positional encodings of transformer. Additionally, the authors did not consider the impact of long-range dependencies or how the choice of positional encoding influences masked attention performance of decoder-only model. It would be valuable if the authors could report comparative results using unmasked attention (e.g., with encoder-only transformers) to examine these effects.

3. (MAIN LIMITATION) The proposed modification p-RoPE appears to primarily integrate elements of RoPE and NoPE, which may lack novelty. While the study advances understanding of RoPE, its applicability to other positional encoding methods, such as Alibi, remains limited, potentially constraining its relevance for models with alternative encoding schemes. The authors should consider incorporating other positional encoding methods and propose more innovative improvements to enhance the usage of lower and higher frequencies.

4. The paper states that higher frequencies correspond to positional attention, while lower frequencies correspond to semantic attention. However, the authors provide an ablation study only for the lower frequencies, without addressing both low and high frequencies. This aspect is insufficiently explained in the ablation experiments.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: * Is your p-RoPE having a similar flavor as the partial rotary?
  - Partial rotary means the whole dimension is divided into rotary part and non-rotary part. The rotary part will be treated by RoPE while the non-rotary part is treated by NoPE.
  - Partial rotary embedding has been known by the community for a while (see https://github.com/lucidrains/x-transformers/issues/40). It is known that the partial rotary is slightly better than rotary positional embedding.
  - Because the proposed p-RoPE is interpolating between NoPE (p=0) and RoPE (p=1), it is possible that p-RoPE is similar to partial rotary.
soundness: 3
strengths: * The paper is well-written and the key points are delivered clearly.
* The authors made a good point that RoPE doesn't necessarily decay with distance.
* The figures clearly showed that most attentions happen in low frequencies, and some high-frequency heads and frequency bands exist.
summary: This paper studied the inner workings of rotary positional embedding. The authors started by challenging the common belief that RoPE decays with distance. Then, the authors showed in Gemma 7B that most RoPE usages appear in low frequencies. The authors explained that high frequencies are for positional information while low frequencies are for semantic information. Finally, the authors observed that the low-frequency components are not robust. Based on this observation, the authors proposed p-RoPE to remove (1-p)*100% of the low-frequency component and showed that this improved the performance of Gemma 2B models.
weaknesses: * The claim that high-frequency components are for positional information is unclear.
  - The reasoning seems to be: (a) RoPE can learn a diagonal or off-diagonal pattern. (b) NoPE cannot learn a diagonal or off-diagonal pattern. So RoPE can learn positional information. However, it doesn't necessarily mean the high-frequency components contribute to the diagonal/off-diagonal pattern. So the function of high-frequency components remains unclear.
  - Some prior works provide evidence that NoPE can still encode positional information [1][2]. So the fact that NoPE cannot learn a diagonal or off-diagonal pattern may not imply it cannot encode positions.
* The authors experimentally show that truncating the lowest frequencies can help RoPE learn better. However, it is based on a claim that "RoPE lacks robust semantic channels". 
  - This claim comes from an analysis on a 2-dimensional case. However, it is possible that the 2D case is too restrictive and a higher dimensional case could have a different result.

[1] Haviv et. al. "Transformer language models without positional encodings still learn positional information," EMNLP 2022 Findings.
[2] Chi et. al. "Latent positional information is in the self-attention variance of transformer language models without positional embeddings," ACL 2023.

Meta-review: **Summary:** This paper provides a theoretical and empirical study of Rotary Positional Encodings (RoPE) in transformer-based LLMs. The authors challenge the conventional belief that RoPE facilitates token dependency decay with increasing distance. Instead, the work hypothesizes that RoPE's utility lies in its ability to construct robust positional attention patterns using high frequencies, while low frequencies encode semantic information. The paper validates these hypotheses through mathematical proofs and extensive experiments, including a novel modification of RoPE, named p-RoPE, which demonstrates performance improvements in specific settings. The study advances the community's understanding of positional encodings and proposes actionable insights for future model design.

**Decision:** The paper makes a significant theoretical and empirical contribution by deepening our understanding of RoPE and proposing meaningful modifications. Despite some concerns regarding the experimental breadth (e.g., limited evaluation on downstream tasks and reliance on perplexity as the primary metric), the added ablations and discussions during the rebuttal phase strengthened the case for the paper’s claims. Reviewer 7eRV raised concerns regarding novelty and limited to decoder-only models. However, based on the prevailing techniques in this field, I do not consider these to be weaknesses of this work. 

Overall, the contributions of this paper outweigh its shortcomings, and I recommend its acceptance. I encourage the authors to incorporate the reviewers' feedback and the additional content provided in the rebuttal into the final version to further enhance the quality of the paper.

Decision: Accept (Poster)