Base of RoPE Bounds Context Length

Authors: mingyu  xu
xin  men
bingning  wang
qingyu  zhang
hongyu  lin
yaojie  lu
xianpei  han
weipeng  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 06:45:39

Abstract: Position embedding is a core component of current Large Language Models (LLMs). Rotary position embedding (RoPE), a technique that encodes the position information with a rotation matrix, has been the de facto choice for position embedding in many LLMs, such as the Llama series. RoPE has been further utilized to extend long context capability, which is roughly based on adjusting the \textit{base} parameter of RoPE to mitigate out-of-distribution (OOD) problems in position embedding. However, in this paper, we find that LLMs may obtain a superficial long-context ability based on the OOD theory. We revisit the role of RoPE in LLMs and propose a novel property of long-term decay, we derive that the \textit{base of RoPE bounds context length}: there is an absolute lower bound for the base value to obtain certain context length capability. Our work reveals the relationship between context length and RoPE base both theoretically and empirically, which may shed light on future long context training.

Main paper: 
 introduction: 
 In the past few years, large language models have demonstrated surprising capabilities and undergone rapid development. By now, LLMs have been widely applied across various domains, including chatbots, intelligent agents, and code assistants [2, 3] . The Transformer [4] , based on the attention mechanism, has been the most popular backbone of LLMs due to its good performance and scaling properties [5] . One of the key component modules in the Transformer is position embedding, which is introduced to embed positional information that is vital for processing sequential data. Rotary position embedding (RoPE), which encodes relative distance information in the form of absolute position embedding [6] , has been a popular choice and applied in many LLMs [7, 8, 9] .RoPE introduces no training parameters and shows improvement in language modeling and many other tasks [6, 10] . One reason that RoPE is widely used is its ability for context length extrapolation [11, 12] , which extends the context length of a trained LLM without expensive retraining. In practice, many works [7, 13, 14] have successfully extended the window length by simply increasing base value, the only one hyper-parameter in RoPE, and fine-tuning on long texts.The reasons behind the success of these long context extensions are often explained as avoiding out-of-distribution (OOD) rotation angles [15, 16] in RoPE, meaning the extended context length (OOD) can be mapped to the in-distribution context length that has been properly trained. Based on the OOD theory, a recent study [15] finds that a smaller base can mitigate OOD and is beneficial for the model's ability to process long contexts, which inspires us to further study the relationship between the base of RoPE and the length of context the model can process.In this paper, we find that the model may show superficial long context capability with an inappropriate RoPE base value, in which case the model can only preserve low perplexity but loses the ability to retrieve long context information. We also show that the out-of-distribution (OOD) theory in position embedding, which motivates most length extrapolation works [11, 12, 15] , is insufficient to fully reflect the model's ability to process long contexts. Therefore, we revisit the role of RoPE in LLMs and derive a novel property of long-term decay in RoPE: the ability to pay more attention to similar tokens than random tokens decays as the relative distance increases. While previous long context works often focus on the relative scale of the RoPE base, based on our theory, we derive an absolute lower bound for the base value of RoPE to obtain a certain context length ability, as shown in Figure 1 . To verify our theory, we conducted thorough experiments on various LLMs such as Llama2-7B [17] , Baichuan2-7B [8] and a 2-billion model we trained from scratch, demonstrating that this lower bound holds not only in the fine-tuning stage but also in the pre-training stage.We summarize the contributions of the paper as follows:• Theoretical perspective: we derive a novel property of long-term decay in RoPE, indicating the model's ability to attend more to similar tokens than random tokens, which is a new perspective to study the long context capability of the LLMs.• Lower Bound of RoPE's Base: to achieve the expected context length capability, we derive an absolute lower bound for RoPE's base according to our theory. In short, the base of RoPE bounds context length.• Superficial Capability: we reveal that if the RoPE's base is smaller than a lower bound, the model may obtain superficial long context capability, which can preserve low perplexity but lose the ability to retrieve information from long context. 
background: 
 In this section, we first introduce the Transformer and RoPE, which are most commonly used in current LLMs. Then we discuss long context methods based on the OOD of rotation angle theory. ), we can observe that the RoPE's rotation angle of extended positions is in-distribution.The LLMs in current are primarily based on the Transformer [4] . The core component of it is the calculation of the attention mechanism. The naive attention can be written as:EQUATIONwhereA ∈ R L×L q, k, v ∈ R d .Position embedding is introduced to use the order of the sequence in attention.RoPE [6] implements relative position embedding through absolute position embedding, which applies rotation matrix into the calculation of the attention score in Eq. 1, which can be written as:EQUATIONwhere m = j -i is the relative distance of i and j, R m,θ is a rotation matrix denoted as:EQUATIONGenerally, the selection of rotation angles satisfies θ i = base -2i/d , the typical base value for current LLMs is 10,000.Based on RoPE, researchers have proposed various methods to extend the long context ability of LLMs, among which representatives are PI [12] and NTK-series (NTK-aware [18] , YaRN [11] , and Dynamical-NTK [19] ). Those methods depend on the relative scale s = T new /T origin , where T origin is the training length of the original pre-trained model and T new is the training length in long-context fine-tuning.PI PI directly interpolates the position embedding, and the calculation of A ij becomes:EQUATIONIn other words, the position embedding of the token at position i in pre-training becomes i/s in finetuning, ensuring the position embedding range of the longer context remains the same as before.The idea is that neural networks are difficult to learn high-frequency features, and direct interpolation can affect the high-frequency parts. Therefore, the NTK-aware method achieves high-frequency extrapolation and low-frequency interpolation by modifying the base value of RoPE. Specifically, it modifies the base b of the RoPE to:EQUATION)The derivation of this expression is derived fromT new b -d-2 d new = T origin b -d-2 dto ensure that the lowest frequency part being interpolated.A recent study [15] proposes to set a much smaller base (e.g. 500), in which case θ i = base -2i d is small enough and typical training length (say 4,096) fully covers the period of cos(t -s)θ i , so the model can obtain longer context capabilities.One perspective to explain current extrapolation methods is the OOD of rotation angle [15, 16] . If all possible values of cos(t -s)θ i have been fitted during the pre-training stage, OOD would be avoided when processing longer context. Figure 2 demonstrates how these methods avoid OOD of RoPE. Token Limit Figure 3 : The superficial long context capability of avoiding OOD by the smaller base. Following the recent work [15] , we fine-tune Llama2-7B with a small base (500) to a context length of 32k. 
motivation: 
 Recent advancements in long-context language models have seen widespread adoption of NTKbased methods [7, 13, 14] . However, a curious trend has emerged: practitioners often employ significantly larger base values than those originally suggested by NTK-aware approaches. This discrepancy raises critical questions about the efficacy of current theoretical frameworks. Why do practitioners deviate from the recommendations of NTK-based methods? Is the out-of-distribution (OOD) theory underlying these methods insufficient to unlock long-context capabilities fully?On the other hand, recent research [15] , driven by OOD theory, proposes using a much smaller base for RoPE to extend context length. However, our findings, as illustrated in Figure 3 , suggest that this approach may only provide superficial long-context capability [22] . While achieving low perplexity even at 128k context length (explicable by OOD theory), the model fails to retrieve relevant information for context lengths as short as 1kwell below its pre-trained length. The observation suggests that the small base determined by OOD theory can't unlock true long-context capability.These phenomena motivate us to delve deeper into the relationship between RoPE's base and context length. To address the gap between OOD theory and our observations, we conduct a theoretical exploration in the next section, aiming to uncover the underlying mechanisms of effective longcontext modeling. 
theory perspective: 
 For attention mechanism in language modeling, we have the following desiderata:Desiderata 1 The closer token gets more attention: the current token tends to pay more attention to the token that has a smaller relative distance. The ability to attend more to similar tokens than random tokens.The similar token gets more attention: the token tends to pay more attention to the token whose key value is more similar to the query value of the current token.Then we examine the desiderata when we apply RoPE to the attention mechanism in LLMs.For Desiderata 1, the property of RoPE makes the model attend more to closer tokens. This kind of long-term decay has been thoroughly discussed in previous work [6, 23] . It comes from the upper bound of attention score calculation, which can be written as:EQUATIONwhere h l = q T i [2l : l2 + 1]k j [2l : 2l + 1]. Equation 7 indicates that the upper bound of the attention score |A ij | decays as the relative distance increases. Figure 4 shows the long-term decay curve of this upper bound, which is in accordance with previous findings [6, 23] .In addition to the attention score's upper bound, we also find there exists another long-term decay property in RoPE: the ability to attend more to similar tokens than random tokens decays as the relative distance increases. We define the ability to attend more to similar tokens than random tokens as:EQUATIONwhere q ∈ R d is the query vector for the current token, k * = q + ϵ is the key value of a similar token, where ϵ is a small random variable, k ∈ R d is the key vector of a random token, R m,θ is the rotation matrix in RoPE. The first term in Eq. 8 is the attention score of q and a similar token k * , the second term in Eq. 8 is the attention score of q and random token k. Then we derive the following theorem:Theorem 1 Assuming that the components of query q ∈ R d and key k ∈ R d are independent and identically distributed, their standard deviations are denoted as σ ∈ R. The key k * = q + ϵ is a token similar to the query, where ϵ is a random variable with a mean of 0. Then we have:EQUATIONThe proof is shown in Appendix A. We denote∑ d/2-1 i=0cos(mθ i ) as B m,θ , and according to Theorem 1, B m,θ measures the ability to give more attention to similar tokens than random tokens, which decreases as the relative distance m increases, as shown in Figure 5 . For a very small base value, we can observe that the B m,θ is even below zero at a certain distance, meaning the random tokens have larger attention scores than the similar tokens, which may be problematic for long context modeling. To satisfy the Desiderata 2, we will getE q,k * [ q T R m,θ k * ] ≥ E q,k [ q T R m,θ k ]. According to Theorem 1, B m,θ needs to be larger than zero. Given the θ in RoPE, the context length L θ that can be truly obtained satisfies:EQUATION)In other word, if we follow the setting that θ i = base -2i/d , in order to get the expected context length L, there is a lower bound of the base value base L :base L = inf{base|B m,θ ≥ 0, ∀m ∈ [0, 1, ..., L]} (11)In summary, the RoPE's base determines the upper bound of context length the model can truly obtain. Although there exists the absolute lower bound, Eq. 9 and Eq. 11 are hard to get the closedform solution since B m,θ is a summation of many cosine functions. Therefore, in this paper, we get the numerical solution. Table 1 shows this lower bound for context length ranging from 1,000 to one million. In Figure 1 , we plot the context length and corresponding lower bound, we can observe that as the context length increases, the required base also increases.Note: this boundary is not very strict because the stacking of layers in LLMs allows the model to extract information beyond the single layers' range, which may increase the context length in Eq. 10 and decrease the base in Eq. 11. Notwithstanding, in Section 5 we find that the derived bound approximates the real context length in practice.Long-term decay from different perspectives. The long-term decay in section 4.1 and section 4.2 are from different perspectives. The former refers to the long-term decay of the attention score as the relative distance increases. This ensures that current tokens tend to pay more attention to the tokens closer to them. The latter indicates that with the introduction of the rotation matrix in attention, the ability to discriminate the relevant tokens from irrelevant tokens decreases as the relative distance increases. Therefore, a large B m,θ , corresponding to a large base value, is important to keep the model's discrimination ability in long context modeling. 
experiment: 
 In this section, we conduct thorough experiments. The empirical result can be summarized in Table 2 , the details are in the following sections.Table 2 : In Section 5, we aim to answer the following questions.Questions Answers Q: Does RoPE's base bounds the context length during the fine-tuning stage?Yes. When the base is small, it is difficult to get extrapolation for specific context length. Q: Does RoPE's base bounds the context length during the pre-training stage? Yes. Our proposed lower bound for RoPE's base also applies to pre-training. If we train a model from scratch with a small base but the context length is large (larger than the bounded length), the resulting model has very limited context length capabilities, meaning some of the context in pre-training is wasted.Q: What happened when base is set smaller than the lower bound?The model will get the superficial long context capability. The model can keep perplexity low, but can't retrieve useful information from long context.For fine-tuning, we utilized Llama2-7B [7] and Baichuan2-7B [8] , both of which are popular opensource models employing RoPE with a base of 1e4. We utilized a fixed learning rate of 2e-5 and a global batch size of 128 and fine-tuning for 1000 steps. For pre-training, we trained a Llama-like 2B model from scratch for a total of 1 trillion tokens. We set the learning rate to 1e-4 and adopted a cosine decay schedule, with models trained on a total of 1T tokens. The dataset we used is a subset of RedPajama [24] . More details of the experimental setup are provided in Appendix B.Our evaluation focused on two aspects: (1) Perplexity: we use PG19 dataset [25] which are often used in long context evaluation; (2) Retrieval: in addition to perplexity, we also adopt retrieval since it represents the real long-context understanding ability of LLMs. We choose a) Long-eval benchmark from [20] and b) Needle in a haystack (NIH) [21] . The Long-eval benchmark generates numerous random similar sentences and asks the model to answer questions based on a specific sentence within the context, while the NIH requires the model to retrieve information from various positions in the long context.According to Eq. 11, there is a lower bound of RoPE's base determined by expected context length. We fine-tune Llama2-7b-Base on 32k context with varying bases. As depicted in Figure 6 , although the difference in perplexity between different bases is negligible, the accuracy of Long-eval varies significantly. In Figure 6b , the dotted line denotes the lower bound derived from Eq. 11, below which the Long-eval accuracy declines significantly. Additional results are provided in Appendix C. Notably, this empirically observed lower bound closely aligns with our theoretical derivation. On the other hand, we can see that base = 2e5 achieves the best perplexity, but the accuracy of Longeval is very low, which indicates the limitations of perplexity in evaluating long context capabilities. We also provide the more comprehensive RULER [26] benchmark results in Appendix G.According to Theorem 1 and Eq. 11, these constraints could also apply to the pre-training stage. To validate this, we trained a 2B model from scratch with RoPE base=100. The results, depicted in the first row of Figure 7 , indicate that even though the model was trained with a context length of 4,096 tokens, it was capable of retrieving information from only the most recent approximately 500 tokens. This demonstrates that the base parameter bounds the context length during the pre-training stage as well. We define the context length from which the model can effectively retrieve information as the effective context length.According to our theory, the effective context length can be extended as the RoPE's base increases. Token Limit Token Limit Token Limit than 32k. Further, when we increase the base to 1e6 and fine-tune the base 2B model on 32K (the third row in Figure 7 ), the model could obtain a larger context length than base=1e4, which is in accordance with our theory.To further remove the influence of model size, we also fine-tuned a larger 7B model on a 32k context length with a RoPE base set to 1e4 and observed an effective context length nearly identical to that of the 2B model with the same RoPE base (see Appendix D). This is empirical proof that the effective context length is determined by RoPE's base.Based on our theory and empirical observations, it is easy to explain what happens in Figure 3 .Better Extrapolation (Perplexity)? Due to the small base, B m,θ can be smaller than zero as m increases, which is shown in Figure 5 . The model can't attend more to similar tokens than random tokens with a large relative distance, so the model tends to focus more on nearby tokens, this will lead to a smaller empirical receptive field, even smaller than the training length. In this case, the model has a strong ability to maintain perplexity stability [27] .Worse Ability (Long-eval and NIH)! According to our previous analysis, RoPE's base bounds the context length, and the context length bounded by 500 is much lower than that bound by 10,000. Therefore, when the base is set to 500, the effective context length drops sharply, even after training on 32k context length.Section 3 mentions that methods based on the OOD theory of rotation angles may not fully reflect the long context capability. In this section, we conduct further experiments to substantiate and explain this observation. We present two methods to extend the context length of Llama2 from 4k to 32k. Both of them are devoid of OOD angles. These methods are delineated mathematically as follows:• Method 1: We can see from Table 3 that these two methods exhibit significantly different long context capabilities. Under the perspective of OOD rotation angle, both methods avoid OOD rotation angle, suggesting effective extrapolation. However, despite being trained on a context length of 32k, "method 2" struggles in completing the retrieval task at a context length of 32k. This phenomenon is beyond the scope which the OOD theory can explain.θ i = (5e6) -2i/d ,Under our perspective, "method 2" is severely violating B m,θ ≥ 0 when m ∈ [15k, 30k], thereby impeding its ability to achieve long-context discrimination. We speculate that the model may achieve better extrapolation in the fine-tuning stage if the base is sufficiently large to surpass a lower bound and avoid OOD of rotation angles. 
related work: 
 Position embedding. Since its introduction, Transformer [4] has achieved remarkable results in the field of natural language processing. To make full use of the order of sequence, researchers have introduced position embedding. The earliest position embedding was based on sinusoidal functions [4] for absolute positions, learnable absolute position embedding [28] and many variants [29, 30] were proposed. Nevertheless, absolute position embedding has difficulties in extending directly to texts longer than the training length. Subsequently, researchers proposed relative position embedding methods [31, 32] . With the development of large language models, rotary position embedding and its variants [6, 23] has become widely used, such as Llama2 [7] , Baichuan2 [8] , Mistral-7B- [33] .A recent study reveals that no position embedding is also potential [34] .Long context learning. Implementing models with longer or even infinitely long contexts has always been an important goal in the field of natural language processing. Due to the squared complexity of the transformer model over time, a significant portion of the work focuses on improving the model structure [35, 35, 36, 37] . However, most of the work is still based on the transformer architecture. The other part of the work is aimed at reducing the computational complexity of attention itself, such as sparse attention [38] and group query attention [39] . In addition, there are also some optimizations in engineering efficiency, such as flash attention [40] and ring attention [41] .In the model inference stage, to save time and space, there are also some methods for accelerating long context, such as KV cache compression [42] , etc. And the position embedding is important in extrapolation. In the process of fine-tuning, methods such as PI [12] , NTK, and YARN [11] are used to change the original position embedding information. FoT [43] assigns the position information of the tokens outside the local context as the first token in the local context. 
limitation: 
 In this work, we investigate the relationship between the base of RoPE and context length. Although we have derived that there exists a lower bound for the base of RoPE determined by context length, the existence of the upper bound for RoPE's base remains an open question that warrants further exploration. In addition, because of the lack of effective benchmarks for assessing long-context capabilities, the scope of long-context capabilities discussed in this paper may be limited. 
conclusion: 
 Our work presents a comprehensive study on the role of RoPE in LLMs for effectively modeling long context. Our main contribution lies in uncovering a novel property of RoPE through theoretical analysis, demonstrating that as the relative distance between tokens increases, the model's ability to attend more to similar tokens decreases. According to our theory, we derive a lower bound for RoPE's base in accommodating to expected context lengths. Our experimental results validate that the base of RoPE bounds context length for not only fine-tuning but also the pre-training stage.Our theory offers a new perspective on understanding the functionality of RoPE in long-context modeling. By shedding light on the relationship between context length and position embedding, we hope our work could provide insights for enhancing the long context capability of LLMs. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?Answer: [NA] Justification: In this work, we don't release any data or model.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer:[Yes] Justification: We cite existing papers and url.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 Lower bound of RoPE's base y = 0.0424x 1.628 Figure 1 : Context length and its corresponding lower bound of RoPE's base value. 
 
 Appendix: Assuming that the components of query q ∈ R d and key k ∈ R d are independent, their standard deviations are denoted as σ ∈ R d and the means are donated as µ ∈ R d . The key k * similar to q is q + ϵ, where ϵ is a random variable with a mean of 0. Then, we have:EQUATIONThen we can get:EQUATION)And when all σ are equal, we can get:d/2-1 ∑ i=0 cos(mθ i ) = 1 2σ 2 (E q,k * q T R m k * -E q,k q T R m k) (14)For training, we mainly conducted experiments on Llama2-7B [7] and Baichuan2-7B [8] . In addition, we also trained a 2B model from scratch, whose structure is the same as Baichuan2-7B-Base but with a smaller hidden size = 2048. Both training and testing are accelerated by FlashAttention-2 [40] and Megatron-LM [44] . The dataset of both fine-tuning and training from scratch is a subset of RedPajama [24] . The hyperparameters of training are listed in Appendix 4. All experiments are conducted on a cluster of 16 machines with 128 NVIDIA A100 80G. For evaluation, we test the long context capabilities comprehensively, the benchmarks are listed below: perplexity on PG19 [25] test split. We evaluate the perplexity of each sample and get the mean value across samples.Long-eval [20] . This test generates massive random similar sentences and asks the model to answer questions according to a specific sentence in the context. Because the long context consists of many similar patterns, it's more difficult to get the right answer. We find this test is harder than other long context evaluations such as Perplexity, Passkey Retrieval [45] , Needle in Haystack [21] . A test sample is list in Figure 8 needle in haystack(NIH) [21] . NIH tests the long context capability not only under different context lengths but also at different positions where the correct answer is located in the context, which provides a more detailed view of the long context capability. Lower bound (b) Long-eval 32kFigure 9 : Fine-tuning Baichuan2-7B-Base on 32k context length with varying RoPE's base. Although the perplexity remains low with varying bases, the Long-eval accuracy reveals a discernible bound for the base value, below which the Long-eval accuracy declines significantly. the dotted line denotes the lower bound derived from Eq. 11. The Desiderata 2 introduced in Section 4 is intuitively plausible, but its empirical validity requires verification. To investigate this, we conducted a detailed empirical analysis. The similarity between tokens is measured by the cosine similarity (denoted as A) of their corresponding hidden states, while the attention allocation between tokens is governed by the attention score (denoted as B). The desiderata "similar tokens receive more attention" implies that a higher value of A should lead to a higher value of B.To test this desiderata, we performed experiments using Llama1-7B, Llama2-7B, and Llama3-8B models. We selected 200 segments from the PG19 dataset, each containing 1024 tokens, and computed Spearmans rank correlation coefficient between (A) and (B). A positive correlation coefficient would indicate that as token similarity (A) increases, the corresponding attention score (B) also increases. The magnitude of the coefficient reflects the strength of this correlation.The results, presented in Figure 15 confirm that Spearmans rank correlation coefficient is positive, validating the desiderata that "similar tokens receive more attention". Furthermore, we observe that this positive correlation is more pronounced in the Llama3-8B model compared to Llama2-7B and Llama1-7B, suggesting that larger and more advanced models are better at capturing this relationship. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Our paper focuses on the impact of key parameter settings in the model on its capability. To our knowledge, this does not involve any social impact. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper investigates the role of Rotary Position Embedding (RoPE) in Large Language Models (LLMs), focusing on the relationship between the RoPE base and the model's long-context ability. The study introduces a theoretical lower bound for the RoPE base, validated through empirical experiments. The findings provide actionable insights for practitioners looking to optimize RoPE-based LLMs for longer contexts without superficial gains. The paper highlights potential pitfalls in using small base values, which can lead to a superficial long-context capability.

The reviewers found the paper to be technically sound and impactful, with a strong recommendation for acceptance after addressing the noted weaknesses. The insights provided by this work are expected to benefit the development and optimization of long-context LLMs.

Reviews: review: paper_summary: paper_summary: Hi Area Chair, I am not qualified for the review of this paper, this is out of my knowledge scope.
strength_weakness: strength_weakness: n/a
questions: questions: n/a
limitations: limitations: n/a
paper_summary: paper_summary: This paper investigates the role of Rotary Position Embedding (RoPE) in Large Language Models (LLMs), with a focus on the relationship between RoPE's base and the model's long context ability. 
The study looks into the long-context abilities and limitations of current methods that rely on smaller RoPE bases. With a lower base, LLM models may exhibit superficial long-context abilities, achieving low perplexity but failing to retrieve relevant information in extended contexts.
Theoretically, The paper establishes two desiderata for the attention mechanism in language modeling: 1.Closer tokens receive more attention. 2.Similar tokens receive more attention. It examines these when applying RoPE to LLMs, revealing a long-term decay in attention score and the ability to differentiate similar from random tokens. This leads to a theorem indicating that RoPE’s base sets an absolute lower bound for achieving specific context lengths.
strength_weakness: strength_weakness: Overall, I like this paper. I would like to suggest that the authors strengthen their work by considering the following points:

1. Extensively test the model using benchmarks such as RULER [1].

2. Provide more empirical observations on the relationship between the base of RoPE and model performance on those challenging benchmarks.


--

[1] https://github.com/hsiehjackson/RULER
questions: questions: For the suggestions, please refer to the weaknesses section.
limitations: limitations: N/A
paper_summary: paper_summary: ROPE is wildily employed in popular LLMs which encodes positional information with a rotation matrix. Although RoPE is used to enhance long-context capabilities by adjusting its base parameter to address OOD issues, this paper finds that this may result in only superficial long-context abilities. Authors re-evaluate RoPE's role and introduce a novel property of long-term decay, showing that the base of RoPE limits context length, with an absolute lower bound required for certain capabilities. This work clarifies the theoretical and empirical relationship between context length and the RoPE base, offering insights for future long-context training.
strength_weakness: strength_weakness: There is no obvious weaknesses in my opinion. I just have a few questions:

- Regarding the "Desiderata 2 The similar token gets more attention", recently StreamLLM [1] shows that there exists "attention sink" in popular LLMs. Namely most of tokens attend to the first few tokens. This somehow contradicts with the principle that "the similar token gets more attention". Could you provide your thoughts on this statement?

- Anthropic's blogs reveal that different heads may have different functionality in in-context learning. How may this interplay with the Rope base? Do you think different heads may have different optimal rope base?

[1] Efficient Streaming Language Models with Attention Sinks
[2] In-context Learning and Induction Heads
questions: questions: See Weaknesses
limitations: limitations: See Weaknesses
paper_summary: paper_summary: The paper investigates the role of RoPE in long-context LLMs. It highlights that the base of RoPE crucially affects the model's ability to handle long contexts. This paper derives a theoretical (and empirical) lower bound for the base value required to maintain long-context capabilities and validates this through empirical experiments with models like Llama2-7B and Baichuan2-7B, and a 2B model trained from scratch. This work offers insights into the base of RoPE for long-context processing in LLMs, which is inspiring for the development and design of long-context LLMs.
strength_weakness: strength_weakness: 1. The Desiderata 2. The similar token gets more attention in Section 4 seems intuitively correct but may not be empirically correct. As this desideratum is the fundamental motivation of Theorem 1, a thorough empirical verification is a must-have.
2. The motivation section is not well-written or organized. It confuses me while I read this part. These are just some previous empirical observations and are not deeply discussed. As the motivation part is very important, I would like to see a clear and well-organized motivation.
3. For the Caption of Figure 6, it would be better to show how to derive the value of the dotted line. I would also like to see a detailed derivation here.
4. Will the author plan to release the models (including the fine-tuned Llama2, Baichun2 with varying lengths, and the 2b model series)? This would benefit future work.
5. I wonder about the negativeness and positiveness of the $q^{T}R_{m,\theta}k$ in equations (8) and (9). If the values are negative, say -2 and -1, which one indicates more attention?
6. For section 5.3, I would like to regard this as a conjecture rather than an explanation.
7. Moinries
    - The upper case and lower case of the title are not consistent, such as the title of Section 4.1 and Section 2.2
    - Line 225 “method2” -> “Method 2”
questions: questions: See **Weaknesses**
limitations: limitations: See **Weaknesses**

Comments: comment: Thank you for the detailed explanation regarding the data and configuration. I would encourage you to incorporate our discussion into the final version of your paper. Solid work overall! 

I have updated my score accordingly. Good luck!
Thank you for your feedback. We appreciate the opportunity to clarify and address your concerns.

1. **Data**: As mentioned in Section 5.1 and Appendix B, our training data is a subset of RedPajama. We upsampled long data during fine-tuning, with ~50% of tokens from documents ≥4k in length. This approach ensures a balance of shorter and longer data, differing from Longlora[1] which used only 8k to 32k data when fine-tuning LLaMA-2-7B to 32k. While data proportion isn't our paper's primary focus, we believe using a sampling method similar to Longlora would yield comparable conclusions.

2. **Training Configuration**:
   - As stated in Section 5.1 and Appendix B, we used a fixed learning rate of 2e-5, global batch size of 128, and fine-tuned for 1000 steps. We've reviewed our configuration and can provide additional details:

   | Parameter | Value |
   |-----------|-------|
   |Global Batch Size| 128 |
   | Steps | 1000 |
   | Tensor Parallelism (Tp) | 4 |
   | Pipeline Parallelism (Pp) | 2 |
   | Precision | bf16 |
   | Learning Rate (Lr) | 2e-5 |
   | Weight Decay (Wd) | 0 |
   | Adam Beta1 | 0.9 |
   | Adam Beta2 | 0.98 |
   | Gradient Clip | 1.0 |

Perhaps you can tell us the training configuration you have conducted before and the number you observed, and we can check together what caused the difference from your previous observations.

Furthermore, as mentioned in our response to Reviewer XWTB, we plan to open-source our trained models. You can download them and compare them with your own trained models at that time.

We look forward to your response and the opportunity for further discussion.

[1] LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
Thank you for your response. Could you please clarify the setup used for fine-tuning Llama2-7b to a sequence length of 32k? Specifically, I’m interested in understanding which data was used and the detailed training configuration. The number you reported appears to be *higher* than what I observed in my previous experiments.
Thank you for your clarification. I am satisfied with the response.

Good Luck!