Round and Round We Go! What makes Rotary Positional Encodings useful?

Authors: Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, Petar Veličković
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Transformers, Positional Encodings, Rotary Positional Encodings

Abstract:
Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust `positional' attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments to verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling LLMs to large sizes and context lengths.

Decision: accept (poster)
Reviewer scores: 8, 5, 8, 5, 5

Reviews and discussion:
<thread>
creation date:
1729051707977

thread_type:
official_review

<thread_object>
creator: reviewer_dtve

<content>
summary:
This paper studied the inner workings of rotary positional embedding. The authors started by challenging the common belief that RoPE decays with distance. Then, the authors showed in Gemma 7B that most RoPE usages appear in low frequencies. The authors explained that high frequencies are for positional information while low frequencies are for semantic information. Finally, the authors observed that the low-frequency components are not robust. Based on this observation, the authors proposed p-RoPE to remove (1-p)*100% of the low-frequency component and showed that this improved the performance of Gemma 2B models.

strengths:
* The paper is well-written and the key points are delivered clearly.
* The authors made a good point that RoPE doesn't necessarily decay with distance.
* The figures clearly showed that most attentions happen in low frequencies, and some high-frequency heads and frequency bands exist.

weaknesses:
* The claim that high-frequency components are for positional information is unclear.
  - The reasoning seems to be: (a) RoPE can learn a diagonal or off-diagonal pattern. (b) NoPE cannot learn a diagonal or off-diagonal pattern. So RoPE can learn positional information. However, it doesn't necessarily mean the high-frequency components contribute to the diagonal/off-diagonal pattern. So the function of high-frequency components remains unclear.
  - Some prior works provide evidence that NoPE can still encode positional information [1][2]. So the fact that NoPE cannot learn a diagonal or off-diagonal pattern may not imply it cannot encode positions.
* The authors experimentally show that truncating the lowest frequencies can help RoPE learn better. However, it is based on a claim that "RoPE lacks robust semantic channels". 
  - This claim comes from an analysis on a 2-dimensional case. However, it is possible that the 2D case is too restrictive and a higher dimensional case could have a different result.

[1] Haviv et. al. "Transformer language models without positional encodings still learn positional information," EMNLP 2022 Findings.
[2] Chi et. al. "Latent positional information is in the self-attention variance of transformer language models without positional embeddings," ACL 2023.

questions:
* Is your p-RoPE having a similar flavor as the partial rotary?
  - Partial rotary means the whole dimension is divided into rotary part and non-rotary part. The rotary part will be treated by RoPE while the non-rotary part is treated by NoPE.
  - Partial rotary embedding has been known by the community for a while (see https://github.com/lucidrains/x-transformers/issues/40). It is known that the partial rotary is slightly better than rotary positional embedding.
  - Because the proposed p-RoPE is interpolating between NoPE (p=0) and RoPE (p=1), it is possible that p-RoPE is similar to partial rotary.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730382713946

thread_type:
official_review

<thread_object>
creator: reviewer_7erv

<content>
summary:
The paper provides practical insights into positional encoding for decoder-only models. However, further investigation is needed to establish the effectiveness and reliability of the theories presented.

strengths:
Pros:

1. The paper offers a fresh discovery of Rotary Positional Encoding (RoPE), challenging the belief that it primarily helps decay attention coefficients with distance.

2. The paper gives an analysis of high and low RoPE frequencies, their roles in positional and semantic attention, and an innovative RoPE modification (p-RoPE) that demonstrates improvements in some cases.

3. Every section has a summary part which makes the whole paper clear and ready to read.

weaknesses:
Cons:

1. The figures in the paper are somewhat confusing. For instance, while the paper emphasizes frequency aspects, Figure 1 illustrates vectors with identical frequency differences, which does not fully align with the paper’s focus. Additionally, the results are primarily presented using heat maps, which may appear monotonous and lack of expressiveness. Given that this is a language modeling task, presenting some results in natural language format would enhance readability and interpretability.

2. (MAIN LIMITATION) The experiments are limited to decoder-only models, making it unclear whether the findings can generalize to other transformer architectures. Whether the observed improvements stem from specific model structures (e.g., encoder-only or encoder-decoder models) rather than from a comprehensive enhancement applicable to diverse positional encodings of transformer. Additionally, the authors did not consider the impact of long-range dependencies or how the choice of positional encoding influences masked attention performance of decoder-only model. It would be valuable if the authors could report comparative results using unmasked attention (e.g., with encoder-only transformers) to examine these effects.

3. (MAIN LIMITATION) The proposed modification p-RoPE appears to primarily integrate elements of RoPE and NoPE, which may lack novelty. While the study advances understanding of RoPE, its applicability to other positional encoding methods, such as Alibi, remains limited, potentially constraining its relevance for models with alternative encoding schemes. The authors should consider incorporating other positional encoding methods and propose more innovative improvements to enhance the usage of lower and higher frequencies.

4. The paper states that higher frequencies correspond to positional attention, while lower frequencies correspond to semantic attention. However, the authors provide an ablation study only for the lower frequencies, without addressing both low and high frequencies. This aspect is insufficiently explained in the ablation experiments.

questions:
Why did the authors provide an ablation study only for the lower frequencies rather than for both low and high frequencies?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730495354355

thread_type:
official_review

<thread_object>
creator: reviewer_rv9e

<content>
summary:
Authors explore how transformer models use RoPE frequencies to learn semantic and positional information. Authors provide a theoretical proof that RoPE does not force the decay in attention coefficient with distance, but instead can create specific patterns. In their experimental study of the Gemma 7B model, authors show that transformers mostly rely on low frequencies, while some heads display high frequency bands, mostly in 1st and last layers of the model. Authors further show that high-frequencies in RoPE provide a mechanism to encode positional information. Low frequencies are used as information channels that are not robust over long context. Finally, authors propose p-RoPE encoding that cuts low frequencies and can help to improve model's performance.

strengths:
- The authors conduct a novel theoretical and empirical study of RoPE encodings in transformer models.
- They provide detailed proofs of their main claims.
- The paper is clear and well-written.
- This study can help researchers better understand the underlying mechanisms of popular transformer architectures and encourage research into alternative improved solutions.

weaknesses:
- The empirical study is limited to a single Gemma architecture. While this is unlikely, some results may be artifacts of the specific model selected.
- In Section 3 authors train 2B model and show improvements on validation perplexity. While these results are positive, perplexity improvements do not always results in overall improvements in model's abilities. Authors could provide evaluation results on popular benchmarks* to build more convincing picture.


* see, for ex, Section 2.3 in https://arxiv.org/pdf/2307.09288

questions:
1. Pre-training model from scratch is expensive and not always feasible. I was wondering if you can instead continuously train other (Gemma-7B, Llama-3 1/3/8B, etc) models for fewer steps but with p-RoPE approach? Do you think it would work or no and why?

2. Please, correct me if I'm wrong, but in the proof of Proposition 3.1:
for k>1, g_k=theta^{-2(k-1)/d} is not necessarily rational, but algebraic. Therefore, Lemma A.1 should be stated not for rational g, but for algebraic g, and it should use the fact that pi is transcendental.
line 680, cos(j-i-r) -> cos(j-i+r)
line 680, the comma in the displayed equation should be a period.
line 684, cos(j-i-r) -> cos(j-i+r)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730795016660

thread_type:
official_review

<thread_object>
creator: reviewer_elgp

<content>
summary:
This paper investigates the role of Rotary Positional Encodings (RoPE) in Transformer-based Large Language Models (LLMs). The authors challenge the common belief that RoPE's usefulness comes from its ability to decay token dependency with increasing relative distance. Instead, they explore how different frequencies in RoPE are utilized, particularly within the Gemma 7B model. The paper provides both theoretical and empirical analyses, proposes a modified RoPE, and highlights the importance of understanding positional encodings for scaling LLMs.

strengths:
1. The paper provides a fresh perspective on RoPE, questioning existing assumptions and offering new explanations for its effectiveness.
2. The authors present mathematical proofs to support their claims, enhancing the credibility of their findings.
3. The use of the Gemma 7B model for empirical analysis adds practical relevance to the theoretical insights.

weaknesses:
1. Although the observed phenomena and mathematical proofs can support the paper's point of view, the experimental performance does not seem good enough. The paper hopes to adapt to any context length, but the actual experimental results only have one result on 8K. And the evaluation of PPL is not comprehensive enough.
2. At the semantic level, the results of the models in Table 2 should be compared on the general benchmark or other tasks that are more representative of semantics, which will be more convincing.
3. At the positional level, it should be compared with similar experiments such as needle in a haystack or Ruler on long contexts to prove its long context expansion ability.

questions:
1. I have some doubts about the display of Figure 2. Normally, for the same qk, different relative distances should have a gradually decreasing effect. But does different qk introduce different variables, making the comparison unfair?
2. Is there any empirical experiment on how many frequencies are cut off for the best effect?
3. Will removing low-frequency RoPE make the effect on short text worse?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730813005106

thread_type:
official_review

<thread_object>
creator: reviewer_t6hh

<content>
summary:
This paper delves into the role of Position Embedding (PE) in LLMs, challenging the traditional view that RoPE primarily attenuates attention weights as the relative distance between words increases. It proposes a new hypothesis that RoPE constructs position-attention patterns (e.g., diagonal or previous-token focus) using high-frequency components while leveraging low frequencies to convey semantic information. Several case studies and theoretical analyses support this hypothesis.

Overall, the paper provides insightful findings and hypotheses on a key component of LLMs: position encoding. However, the arguments mainly rely on case studies with gemma-7B, and the experiments lack diversity in the foundation models (e.g., missing LLaMA series) and tasks (e.g., language model vs. QA vs. code...). Additionally, some parts of the proofs contain errors.

strengths:
1. This paper successfully challenges, both theoretically and empirically, the traditional view that RoPE attenuates attention weights as relative distance between tokens increases. 
2. The authors' hypothesis about the roles of the high-frequency and low-frequency components of RoPE is novel and insightful.

weaknesses:
1. The experimental validation lacks diversity in both foundation models and datasets.
2. Some perspectives and proofs regarding NoPE contain errors, while they don't affect the main conclusion, they may mislead readers.
3. The discussion of related work is insufficiently thorough, and few papers are cited (only about one page).
4. Throughout Section 4, the authors conceal a core assumption: that attention scores are interpretable or meaningful. Higher attention scores for certain tokens imply a meaningful preference in the model, giving special significance to the larger norms in Equation at Line 257. I want to point out that this assumption is still being debated, and I suggest that the authors make it explicit.
> Is Attention Explanation? An Introduction to the Debate (Bibal et al., ACL 2022)

questions:
1. In practice, do the phenomena observed on gemma-7b apply to other LLM backbones, such as llama-2 or llama-3?
2. The phenomena observed in this paper are based on what scale and type of data? Could the findings be validated across various tasks, such as language modeling, code, or QA?
3. In Line 190, the authors claim that NoPE has strong OOD capabilities, but Kazemnejad et al., 2024 only validated this for sequences of length up to 50, where NoPE performed slightly better than other PEs, without exhibiting exceptionally strong OOD performance (refer to their Figure 3). The following works explored NoPE's extrapolation, and it can be seen that the generalization ability of the NoPE baseline is limited:
> Length Generalization of Causal Transformers without Position Encoding (Wang et al., Findings 2024)
> 
> [Neurips24 spotlight] Exploring Context Window of Large Language Models via Decomposed Positional Vectors
4. In Line 497, the authors state, "p-RoPE is in spirit similar to the idea of increasing the wavelength of RoPE from 10,000 to 500,000," so I suggest that Table 2 should include RoPE with a base of 500,000.
5. How do the observations in Section 4 change across different RoPE variations, such as 0.25-RoPE, 0.75-RoPE, RoPE_10000, and RoPE_500000? For example, in Figure 3, how does the norm distribution of low frequencies vary across these models?
6. In Line 726, this proof only applies to the first layer of NoPE’s attention heads. Starting from the second layer, the assumption 
$a_{3,3}=<q_3, k_3>=<q_3, k_2>=a_{3,2}$ 
no longer holds.
7. Attention heads with specific patterns have been widely studied. Besides the patterns discussed in this paper (e.g., diagonal or previous-token focus), could the authors observe and analyze other representative attention patterns, such as special token focus, punctuation focus, and locality focus? Please refer to typical patterns in the following paper:
> [ICLR24] Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
</content>
</thread_object>

</thread>
<thread>
creation date:
1731971191484

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comment

comment:
We would like to thank the reviewers for their efforts in assessing our work and the valuable feedback. We have accordingly made significant improvements following the suggestions and comments. For convenience, *we have marked in the revised PDF in blue* the added sections and the captions of newly added figures. 

Below we summarise our changes: 

- Llama results: Reviewers T6HH and rv9e requested inclusion of an additional model. We have **added results with Llama 3.1 8B** in the 
Appendix (Section C) showcasing similar patterns to Gemma, which we believe greatly increases the value and generality of our claims.
- Added in Figure 19 an ablation of the usage of RoPE frequencies in different domains (Italian, Chinese, Code, and Arithmetic) to show that our results generalise over different types of domains.

- Added **additional experimental ablation** comparing p-RoPE to increasing the max wavelength parameter in Table 2 as requested by Reviewer T6HH. We are happy to report that p-RoPE outperforms this baseline.
- Added **additional experimental ablation** comparing p-RoPE to removing the *highest* frequency instead of the lowest ones as suggested in p-RoPE in Table 2, as requested by Reviewer 7eRV. We are happy to report that p-RoPE outperforms this baseline.
- Added **additional experimental ablation** comparing p-RoPE to partial-RoPE in Table 2, as requested by Reviewer 7eRV. We are happy to report that p-RoPE outperforms this baseline.

- Added a more **detailed analysis of the “apostrophe head”** in the Appendix (Section E.1)  – reverse engineering how RoPE is used to construct this particular head. 
- **Additional synthetic result** showing that RoPE does not decay over a constant sequence of *repeated* queries and keys sampled *once* from a Gaussian as requested by Reviewer eLgp in Appendix B.4.  
- **Improved related works**: Following the advice from Reviewer T6HH, we have added a more detailed discussion of related works in the Appendix (Section B). 

We hope that our revisions have strengthened our contributions and would like to thank the reviewers for their valuable suggestions.  We look forward to productive rebuttal discussions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732616399401

thread_type:
public_comment

<thread_object>
creator: ~mingyu_xu1

<content>
title:
An informal comment

comment:
Hello, I'm glad to find this article on ICLR. 

In line 423, "We therefore suspect that with larger and larger contexts, the $\theta$ base wavelength of RoPE will have to also be increased accordingly." 

I think the following paper may get you,  "there is an absolute lower bound for the base value to obtain certain context length capability".[1]

Good luck!

[1]Base of RoPE Bounds Context Length https://openreview.net/forum?id=EiIelh2t7S
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary:** This paper provides a theoretical and empirical study of Rotary Positional Encodings (RoPE) in transformer-based LLMs. The authors challenge the conventional belief that RoPE facilitates token dependency decay with increasing distance. Instead, the work hypothesizes that RoPE's utility lies in its ability to construct robust positional attention patterns using high frequencies, while low frequencies encode semantic information. The paper validates these hypotheses through mathematical proofs and extensive experiments, including a novel modification of RoPE, named p-RoPE, which demonstrates performance improvements in specific settings. The study advances the community's understanding of positional encodings and proposes actionable insights for future model design.

**Decision:** The paper makes a significant theoretical and empirical contribution by deepening our understanding of RoPE and proposing meaningful modifications. Despite some concerns regarding the experimental breadth (e.g., limited evaluation on downstream tasks and reliance on perplexity as the primary metric), the added ablations and discussions during the rebuttal phase strengthened the case for the paper’s claims. Reviewer 7eRV raised concerns regarding novelty and limited to decoder-only models. However, based on the prevailing techniques in this field, I do not consider these to be weaknesses of this work. 

Overall, the contributions of this paper outweigh its shortcomings, and I recommend its acceptance. I encourage the authors to incorporate the reviewers' feedback and the additional content provided in the rebuttal into the final version to further enhance the quality of the paper.