Base of RoPE Bounds Context Length
Authors: Mingyu Xu
Xin Men
Bingning Wang
Qingyu Zhang
Hongyu Lin
Xianpei Han
weipeng chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Long context
Rotary position embedding
Primary area: natural_language_processing

Abstract: Position embedding is a core component of current Large Language Models (LLMs). Rotary position embedding (RoPE), a technique that encodes the position information with a rotation matrix, has been the de facto choice for position embedding in many LLMs, such as the Llama series. RoPE has been further utilized to extend long context capability, which is roughly based on adjusting the \textit{base} parameter of RoPE to mitigate out-of-distribution (OOD) problems in position embedding. However, in this paper, we find that LLMs may obtain a superficial long-context ability based on the OOD theory. We revisit the role of RoPE in LLMs and propose a novel property of long-term decay, we derive that the \textit{base of RoPE bounds context length}: there is an absolute lower bound for the base value to obtain certain context length capability. Our work reveals the relationship between context length and RoPE base both theoretically and empirically, which may shed light on future long context training.

Base of RoPE Bounds Context Length
Mingyu Xul; Xin Men', Bingning Wang', [ Qingyu Zhang? Hongyu Lin?, Yaojie Lu", Xianpei Han and Weipeng Chen
Baichuan Inc.
2 Chinese Information Processing Laboratory Institute Software, Chinese Academy Sciences xumingyu, (zhanggingyu2024, hongyu, ) ac. cn
com
Abstract
Position embedding is core component of current Large Language Models (LLMs). Rotary position embedding (RoPE), technique that encodes the po- sition information with a rotation matrix, has been the de facto choice for position embedding in many LLMs, such the Llama series. RoPE has been further utilized to extend long context capability, which roughly based on adjusting the base parameter of RoPE to mitigate out-of-distribution (OOD) problems in position embedding. However, in this paper, we find that L.LMs may obtain a superficial long-context ability based on the OOD theory. We revisit the role of RoPE LLMS and propose a novel property long-term decay, deriving that the base of RoPE bounds context length: there is absolute lower bound for the base value to obtain certain context length capability, Our work reveals the relationship between context length and RoPE base both theoretically and empirically, which may shed light on future long context training.
y 0.0424x 1.688
Lower bound RoPE's base
10
107
106
10a
10
103
104
105
106
Context length
Figure 1: Context length and its corresponding lower bound of RoPE's base value.
'Equal contribution. Order determined by swapping one [
Cooresponning author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
In the past few years, large language models have demonstrated surprising capabilities and under- gone rapid development. By now, LLMs have been widely applied across various domains, in- cluding chatbots, intelligent agents, and code assistants [2 GJ. The Transformer [8], based on the attention mechanism, has been the most popular backbone of LLMs due to its good performance and scaling properties [S]. One of the key component modules in the Transformer is position em- bedding, which is introduced to embed positional information that is vital for processing sequential data. Rotary position embedding (RoPE), which encodes relative distance information in the form of absolute position embedding [6]]. has been a popular choice and applied in many LLMs [Z N. M).
RoPE introduces no training parameters and shows improvement in language modeling and many other tasks [K) I[]] One reason that RoPE is widely used is its ability for context length extrapolation [[I, [2] which extends the context length of a trained LLM without expensive retraining, In practice, many works [77
D [ have successfully extended the window length by simply increasing base value, the only one hyper-parameter in RoPE, and fine-tuning on long texts.
The reasons behind the success of these long context extensions are often explained avoiding out-of-distribution (OOD) rotation angles [TS.. [6] in RoPE, meaning the extended context length (OOD) can be mapped the in-distribution context length that has been properly trained. Based on the OOD theory, a recent study [[5]] finds that a smaller base can mitigate OOD and is beneficial for the model's ability process long contexts, which inspires between the base of RoPE and the length context the model can process.
further study the relationship
In this paper, we find that the model may show superficial long context capability with inappro- priate RoPE base value, which case the model can only preserve low perplexity but loses the ability to retrieve long context information. We also show that the out-of-distribution (OOD) theory in position embedding, which motivates most length extrapolation works [[TI, D2, LS], is insufficient to fully reflect the model's ability to process long contexts. Therefore, in LLMs and derive a novel property of long-teri decay in RoPE: the ability to pay more attention to similar tokens than random tokens decays as the relative distance increases. While previous long context works often focus on the relative scale of the RoPE base, based on our theory, we derive an absolute lower bound for the base value of RoPE obtain certain context length ability, as shown in Figure To verify our theory, we conducted thorough experiments on various LLMs such as Llama2-7B [LZ], Baichuan2-7B [8] and a 2-billion model we trained from scratch, demonstrating that this lower bound holds not only in the fine-tuning stage but also in the pre-training stage.
revisit the role of RoPE
We summarize the contributions of the paper follows:
Theoretical perspective: derive a novel property of long-tern decay in RoPE, indicat- ing the model's ability to attend more to similar tokens than random tokens, which is a new perspective study the long context capability of the LLMs.
Lower Bound of RoPE's Base: to achieve the expected context length capability, we derive an absolute lower bound for RoPE's base according to our theory. In short, the base of RoPE bounds context length.
Superficial Capability: we reveal that if the RoPE's base is smaller than a lower bound, the model may obtain superficial long context capability, which can preserve low perplexity but lose the ability to retrieve information from long context.
2 Background
In this section, we first introduce the Transformer and RoPE, which are most commonly used in current LLMs. Then discuss long context methods based the OOD of rotation angle theory.

---

1.0
0.98
0.5
01.5
0.96
RaPE
0.0
Extenced
0.0
Extendel cuntent.
0..9
-0.5
-0.5
0.92
-10
-1.0
10000
20000
30000
10000
200000
300000
100000
200001
30000)
Contectoéiee
Contacciniiee
Contesi-ion
(a) base=le4
base=500
(c) basc=b-
An illustration of OOD in RoPE when extend context length from 4k to 32k, and two solutions to avoid the OOD. We show the last dimension as the lowest frequency part of RoPE, which suffers OOD mostly in extrapolation. (a) For a 4k context-length model with base value as 1e4, when we extend the context length to 32k without changing the base value, the context length from 4k to 32k is OOD for RoPE (red area in the figure). (b) OOD can be avoided with small base value like 500 [ since the full period has been fitted during fine-tuning stage. (c) We set base as b 8-g=z from NTK [LT] The blue line denotes the pre-training stage (base=le4) and the red dashed line denotes the fine-tune tne-tuning stage (base=b- SZ-x), we can observe that the RoPE's rotation angle of extended positions is in-distribution.
Figure 2:
2.1 Attention and RoPE
The LLMs in current are primarily based on the Transformer [[]]_ The core component of it is the calculation of the attention mechanism. The naive attention can written as:
Aij = q." ki
(1)
ATTN(X) softmax (A/vxxAA))a))))) v,
(2)
where RLXL q, k, Rd. Position embedding is introduced the order of the sequence in attention.
[Q] implements relative position embedding through absolute position embedding, which ap- plies rotation matrix into the calculation of the attention score in Eq. which can be written as:
RoPE
Aij (R: (Ry.ok,) q. R q Rm,.kkj
where m =j - i is the relative distance i and j, Rm, a rotation matrix denoted as:
cos(m)g) win(m)))
g) cos(m10)
con(m (me)))
s(((m) cos(m0
(4)
cos(m0a//211 sin(m0 sin(mmaa 1)
cos(m00/2
1)
Generally, the selection of rotation angles satisfies 0, = basee 24/d the typical base value for current LLMs 10,000.
2.2 OOD theory of relative rotation angle
Based on RoPE, researchers have proposed various methods to extend the long context ability of LLMs, among which representatives are Pl [I22 and NTK-series (NTK-aware [[8], YaRN [[T]] and Dynamical-NTK [[TI]). Those methods depend on the relative scale s = Tnew Tonginn where Torgin
the training length of the original pre- trained model and T Tney is the training length in long-context fine-tuning.
PI PI directly interpolates the position embedding, and the calculation of Aij becomes:
A = (Ra,s4: (aoi:4)T((R,RR) (R = of R = of Rm/skj,
(5)
In other words, the position embedding of the token at position in pre-training becomes i/s in fine- tuning, ensuring the position embedding range of the longer context remains the same as before.

---

NTK-series The idea is that neural networks are difficult to learn high-frequener features, and direct interpolation can affect the high-frequency parts. Therefore, the NTK-aware method achieves high-frequency extrapolation and wwwfrequency interpolation by modifying the base value of RoPE. Specifically, it modifies the base of the RoPE to:
Unew b
4-2
The derivation of this expression is derived from T Dnew frequency part being interpolated.
T Torigin
to ensure that the lowest
A recent study [[S]] proposes to set much smaller base (e.g. 500), in which case 0; = base 3 is small enough and typical training length (say 4,096) fully covers the period of cos(L s)0;, so the model can obtain longer context capabilities.
One perspective to explain current extrapolation methods is the OOD of rotation angle [6] If all possible values of cos(t - s)0, have been fitted during the pre-training stage, OOD would be avoided when processing longer context. Figure Ø demonstrates how these methods avoid OOD of RoPE.
3 Motivation
110
Justaz Cklleeeb
Linent Handee
Doylat
Chasse kegl
Wather Srredrradeddie
1001
eighs
Tales
(a) Perplexity
(b) Long-eval [00]
(c) Needle in Haystack [
Figure 3: The superficial long context capability of avoiding OOD by the smaller base. Following the recent work [[55], fine-tune Llama2-7B with small base (500) context length of 32k.
Recent advancements in long-context language models have seen widespread adoption of NTK- based methods M TR [.]. However, curious trend has emerged: practitioners often employ significantly larger base values than those originally suggested by NTK-aware approaches. This discrepancy raises critical questions about the efficacy current theoretical frameworks. Why do practitioners deviate from the recommendations of NTKKbbsed methods? Is the cat-of-diitribution (OOD) theory underlying these methods insufficient unlock long-context capabilities fully?
On the other hand, recent research [[S]], driven by OOD theory, proposes using a much smaller base for RoPE extend context length. However, our findings, as illustrated in Figure 3, suggest that this approach may only provide superficial long-context capability [22]. While achieving low perplexity even 128k context length (explicable by OOD theory), the model fails retrieve relevant infor- mation for context lengths as short as lkwell below its pre-trained length. The observation suggests that the small base determined by OOD theory can't 't unlock long-context context capability.
These phenomena motivate us to delve deeper into the relationship between RoPE's s base and context length. To address the gap between OOD theory and observations, we conduct theoretical exploration the next section, aiming to uncover the underlying mechanisms of effective long- context modeling.
Theory Perspective
For attention mechanism in language modeling, we have the following desiderata:
Desiderata 1 The closer token gets more attention: the current token tends to pay more attention to the token that has smaller relative distance.

---

15.0
base
baselto
Exsecled
0.6
Iaancled Iec
Tsoan Lad.
12.0
0.4
0.4
Tonner
10.0
2.5
0.0
2.5
0.2
5020
Relattre detance
Beletive datanne
Bidallow
Figure 4: The upper bound of attention with respect to the relative distance.
Figure 5:
The ability to attend more similar tokens than random tokens.
Desiderata 2
The similar token gets more attention: the token tends pay more atlention to the token whose key value is more similar to the query value of the current token.
Then examine the desiderata when apply RoPE to the attention mechanism in LLMs.
4.1 Long-term Decay of Upper Bound of Attention Score
For Desiderata the property of RoPE makes the model attend more to closer tokens. This kind of long-term decay has been thoroughly discussed in previous work [6, 233.. It comes from the upper bound of attention score calculation, which can be written as:
4/2
| 1u6 Rmk S max((1h h(t1l)
Sn
d/2
n-1
max h4 - h241l)
Σ
e (1-))0vv11
(7)
1=0
where hú = 0. [21 12 + 1]4,(21: 2/ 1]. Equation / indicates that the upper bound of the attention score Aij decays as the relative distance increases. Figure g shows the long-term decay curve of this upper bound, which accordance with previous findings [K] 233.
4.2 Long-term Decay of the Ability to Attend More to Similar Tokens than Random Tokens
In addition to the attention score's upper bound, we also find there exists another long-term decay property in RoPE: the ability attend more similar tokens than random tokens decays as the relative distance increases. We define the ability to attend more to similar tokens than random tokens as:
Eg [q_ Rmm.k*]] Ey n [a_ Rmm,ek].
(8)
where g Rd is the query vector for the current token, A q is the key value of similar token, where is a small random variable, & E Rd is the key vector of random token, Rm, o is the rotation matrix in RoPE. The first term in Eq. & is the attention score of q and similar token k.,,, the second term in Eq. H is the attention score of and random token Then we derive the following theorem:
Theorem Assuming that the components of query q E Re and key k E Re are independent and identically distributed, their standard deviations are denoted as a E R. The key k* = q + e is token similar to the query, where is random variable with mean of Then we have:
d/2-1
202 (Eg,kr [8 Rm,ek*] Eyy [9" Rm.,k ok))
Σ 1=0
cos(mo,)
The proof is shown in Appendix A We denote d/2-1 cos(me,) as Bm... and according to Theo- rem (I. Bm,e measures the ability to give more attention to similar tokens than random tokens, which decreases as the relative distance increases, as shown in Figure EL For very small base value, we can observe that the Bm,g even below zero at a certain distance, meaning the random tokens have larger attention scores than the similar tokens, which may be problematic for long context modeling.
5

---

Table 1: Context length and its corresponding lower bound of RoPE's base.
Context Len.
1k
2k
4k
8k
16k
32k
64k
128k
256k
512k
1M
Lower Bound
4.3e3
1.6c4
2.7c4
8.4e4
3.1e5
6.4e5
2.1c6
7,8e6
3.6e7
6.4e7
5.1e8
4.3
Base of RoPE Bounds the Context Length
To satisfy the Desiderata we will get Eg. [a" Rm. 2 Ey [q" Rm.ek]. According The- orem M. 8m,9 needs to be larger than zero. Given the @ in RoPE, the context length Lg that can be truly obtained satisfies:
Lg = su] 2 0, Vm E 10,1,...111 L}}
(10)
In other word, we follow the setting that 0; = basj-2 in order to get the expected context length L, there is a lower bound of the base value baser.
basel = inf (rase|Bm,e 2 O, vm E [0,
(11)
In summary, the RoPE's base determines the upper bound of context length the model can truly obtain. Although there exists the absolute lower bound, Eq. 9 and Eq. M are hard to get the closed- form solution since 0 is summation of many cosine functions. Therefore, in this paper, we get the numerical solution. Table II shows this lower bound for context length ranging from 1,000 to one million. In Figure II we plot the context length and corresponding lower bound, we can observe that as the context length increases, the required base also increases.
Note:
this boundary is not very strict because the stacking of layers in LLMs allows the model to extract information beyond the single layers' range, which may increase the context length in Eq. 70 and decrease the base in Eq. T Notwithstanding, in Section B we find that the derived bound approximates the real context length in practice.
Long-term decay from different perspectives. from different perspectives. The former refers the long-term decay of the attention score the relative distance increases. This ensures that current tokens tend to pay more attention to the tokens closer to them. The latter indicates that with the introduction of the rotation matrix in attention, the ability to discriminate the relevant tokens from irrelevant tokens decreases as the relative distance increases. Therefore, a large Bm,,, corresponding to a large base value, is important to keep the model's discrimination ability in long context modeling.
The long-term decay in section A and section ?
5 Experiment
In this section, we conduct thorough experiments. The empirical result can be summarized in Table 2 the details are in the following sections.
Table 2: In Section S we aim to answer the following questions.
Questions
Answers
Q: Does RoPE's base bounds the context length during the fine-tuning stage?
Yes. When the base is small, it is diflicult to get extrapolation for specific context length.
Yes. Our proposed lower bound for RoPE's base also applies to pre-training. If we train model from scratch with a small base but the context length is large (larger than the bounded length), the resulting model has very limited context length capabilities, meaning some of the context in pre-training is wasted. The model will get the superficial long context capability. The model can keep perplexity low, but can't 't retrieve useful information from long context.
Q: Does RoPE's base bounds the context length during the pre- pre-training stage?
Q: What happened when base is set smaller than the lower bound?
5.1 Experiments Setup
For fine-tuning, we utilized Llama2-7B [0] and Baichuan2-7B [8], both of which are popular open- source models employing RoPE with base of 1e4. We utilized fixed learning rate of 2c-5 and

---

32K-base: Ind
18
32K-base/2e
16
32K huse:965.
Accuracy 32K
0.3
bases5ef
Perplexity
14
32KK-bsee1ed
0.2
32K- bese:1e22
12
4K- Baseline
10
0.1
0.0
Lower bound
5000
10000
15000
20000
25000
30000
1e1
2e5 5e6
1e9
1e12
Context
Base value
(a) Perplexity
Long-oval 32k
Figure 6: Fine-tuning Llama2-7B-Base 32k context length with varying RoPE's base. Although the perplexity remains low with varying bases, the Long-eval accuracy reveals a discernible bound for the base value, below which the Long-eval accuracy declines significantly. 'he dotted line denotes the lower bound derived from Eq. I and code is provided in Appendix
global batch size of 128 and fine-tuning for 1000 steps. For pre-training, we trained Llama-like 2B model from scratch for total of trillion tokens. We the learning to le-4 and adopted cosine decay schedule, with models trained on a total of IT tokens. The dataset we used is a subset
of RedPajama [24]]. More details of the experimental setup are provided in Appendix [
Our evaluation focused on two aspects: (1) Perplexity: we use PG19 dataset [25]] which are often used in long context evaluation; (2) Retrieval: in addition to perplexity, we also adopt retrieval since it represents the real long-context understanding ability of LLMs. We choose a) Long-eval benchmark from [20] and b) Needle in haystack (NIH) [2I]]. The Long-eval benchmark generates numerous random similar sentences and asks the model to answer questions based on a specific sentence within the context, while the NIH requires the model to retrieve information from various positions in the long context.
5.2 Base of RoPE bounds context length in fine-tuning stages
According to Eq. III. there is a lower bound of RoPE's base determined by expected context length. We fine-tune Llama2-7b-Bann the difference in perplexity between different bases is negligible, the accuracy of Long-eval varies significantly. In Figure 16, the dotted line denotes the lower bound derived from Eq. I. below which the Long-eval accuracy declines significantly. Additional results are provided in Appendix Q Notably, this empirically observed lower bound closely aligns with our theoretical derivation. the other hand, we can see that base eval very low, which indicates the limitations of perplexity evaluating long context capabilities. We also provide the more comprehensive RULER [2f]benchmark results in Appendix G
on 32k context with varying bases. As depicted Figure & although
= 2e5 achieves the best perplexity, but the accuracy of Long-
5.3 The Base RoPE bounds context length in pre-training stages
According to Theorem II and Eq. II these constraints could also apply to the pre-training stage. To validate this, we trained 2B model from scratch with RoPE 100e=100. The results, depicted in the first row of Figure M indicate that even though the model was trained with context length of 4,096 tokens, it was capable retrieving information from only the most recent approximately 500 tokens. This demonstrates that the base parameter bounds the context length during the pre-training stage as well. We define the context length from which the model can effectively retrieve information as the effective context length.
According to our theory, the effective context length can be extended as the RoPE's base increases. To validate this, we further fine-tune this 2B model on 32k context length, with RoPE's base set to 1e4, as shown in the second row Figure While the effective context length increased, remains significantly below 32k since the effective context length bounded by lase=1e4 is much smaller
7

---

0.
1000
2900
2006
1000
Oetret angie
1000
1300
2000
1030
1000
2009
1100
Figure 7: The first row: the results of a 2B model training from scratch with base=1e2. The second row: The results of fine-tuning the 2B model with base=le4. The third row: The results of fine- tuning the 2B model with base=le6.
than 32k. Further, when increase the base to lеб and fine-tune the base 2B model 32K (the third row in Figure M. the model could obtain a larger context length than base=le4, which is in accordance with our theory.
To further remove the influence model size, we also fine-tuned larger 7B model on a 32k context length with RoPE base set to 144 and observed an effective context length nearly identical to that of the 2B model with the same RoPE base (see Appendix DI). This is empirical proof that the effective context length is determined by RoPE's base.
5.4 Interpretation for the superficial long context capability for small base
Based theory and empirical observations, it is easy to explain what happens in Figure R
Better Extrapolation (Perplexity)? Due the small base, Bn, can smaller than zero as m increases, which is shown in Figure E The model can't 't attend more to similar tokens than random tokens with a large relative distance, the model tends to focus more on nearby tokens, this will lead to a smaller empirical receptive field, even smaller than the training length. In this case, the model a strong ability to maintain perplexity stability [27]].
Worse Ability (Long-eval and NIH)! According to our previous analysis, RoPE's base bounds the context length, and the context length bounded by 500 is much lower than that bound by 10,000. Therefore, when the base is set to 500, the effective context length drops sharply, even after training on 32k context length.
5.5 OOD theory is insufficient to reveal long context capability
Section mentions that methods based on the OOD theory of rotation angles may fully reflect the long context capability. In this section, we conduct further experiments to substantiate and explain this observation. We present two methods to extend the context length of Llama2 from 4k to 32k. Both of them are devoid of OOD angles. These methods are delineated mathematically as follows:
Method 1: 0, (5e6)-24) 24/d

---

Table 3: The comparison of "Method 1" and "Method 2". These methods are designed carefully. They both are no OOD, but they are very different under our theory.
Lvng-eval 15k
numbers of m whose Bm,6 S 0
Method
OOD
30k
15k
30k
Method I Method 2
0.33
0.27
0
x
0.40
0.00
97
2554
Method 2: (le4) (le4)-21/128// -2//128/8
i 2 44
(le4 8128/88) 1 25/1228
i 44.
We can see from Table & that these two methods exhibit significantly different long context capabil- ities. Under the perspective of OOD rotation angle, both methods avoid OOD rotation angle, sug- gesting effective extrapolation. However, despite being trained on a context length of 32k, "method 2" struggles in completing the retrieval task context length of 32k. This phenomenon is beyond the scope which the OOD theory can explain.
Under our perspective, "method 2" is severely violating Bm.. 2 0 when impeding its ability to achieve long-context discrimination. We speculate that the model may achieve better extrapolation in the fine-tuning stage if the base is sufficiently large to surpass a lower bound and avoid OOD of rotation angles.
[15k;30kg thereby
6 Related Work
Position embedding. Since its introduction, Transformer [[]] has achieved remarkable results in the field of natural language processing. To make full use of the order of sequence, researchers have introduced position embedding. The earliest position embedding was based on sinusoidal functions [ for absolute positions, learnable absolute position embedding [2X] and many variants [29 [D] were proposed. Nevertheless, absolute position embedding has difficulties in extending directly to texts longer than the training length. Subsequently, researchers proposed relative position embed- ding methods [3T] 32) With the development of large language models, rotary position embedding and its variants [6. 2 has become widely used, such as Llama2 [[Z], Baichuan2 [8], Mistral-7B-
A recent study reveals that no position embedding is also potential [G4]].
Long context learning. Implementing models with longer or even infinitely long contexts has always been an important goal in the field of natural language processing. Due the squared com- plexity of the transformer model over time, significant portion of the work focuses improving the model structure [GS] RS
5G. 37). However, most of the work is still based on the transformer architecture. The other part of the work is aimed at reducing the computational complexity of atten- tion itself, such as sparse attention [6*] and group query attention [39]] In addition, there are also some optimizations in engineering efficiency, such as flash attention [+0]] and ring attention [[]]] In the model inference stage, to save time and space, there are also some methods for accelerating long context, such as KV cache compression [ا etc. And the position embedding is important in extrapolation. the process of fine-tuning, methods such as PI [12]), NTK, and YARN [[I] are used to change the original position embedding information. FoT [X assigns the position information of the tokens outside the local context as the first token in the local context.
7 Limitation
In this work, we investigate the relationship between the base of RoPE and context length. Although we have derived that there exists a lower bound for the base of RoPE determined by context length, the existence of the upper bound for RoPE's base remains open question that warrants further exploration. In addition, because of the lack of effective benchmarks for assessing long-context capabilities, the scope of long-context capabilities discussed in this paper may be limited.

---

Conclusion
Our work presents comprehensive study on the role of RoPE in LLMs for effectively modeling long context. Our main contribution lies in uncovering novel property of RoPE through theoretical analysis, demonstrating that as the relative distance between tokens increases, the model's ability to attend more to similar tokens decreases. According to our theory, we derive lower bound for RoPE's base in accommodating to expected context lengths. Our experimental results validate that the base of RoPE bounds context length for not only fine-tuning but also the pre-training stage. Our theory offers new perspective on understanding the functionality of RoPE long-context modeling. By shedding light on the relationship between context length and position embedding, we hope our work could provide insights for enhancing the long context capability of LLMs.
10

---

References
[1] Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and Weipeng Chen. Base of rope bounds context length. arXiv preprint arXiv:2405.14591, 2024.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arX1v:2303.08774, 2023.
[3] Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. Impact of code language models on automated program repair. In 2023 IEEF/ACM 45th International Conference on Software Engineering (ICSE), pages 1430-1442, IEEE, 2023.
[4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[5] Yi Tay, Mostafa Dehghani. Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws model architectures: How does inductive bias influence scaling? arXiv preprint (xXiv:2207.1055:1 2022.
[6] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiigg 568:127063, 2024.
[7] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
[8] Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10300 2023.
[9] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint 2309::2300.1609 2023.
[10] Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. arXiv preprint arXiv:2403.1333 2024.
[11] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yar:: Efficient context win- dow extension of large language models. In The Twelfth International Conference on Learning Representations, 2023.
[12] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context win- dow of large language models via positional interpolation. arXiv preprint arXiv:2306.15595 2023.
[13] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-lenghh video and language with ringattention. arXiv preprint arXiv:2402.08268 2024.
[14] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403 04652, 2024.
[15] Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws roPe-based extrapolation. In The Twelfth International Conference Learning Representations, 2024.
[16] Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm- infinite: Simple on-the-fly length generalization for large language models. arXiv preprint 161337, 2023.

---

[17] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale. Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernan- des, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[18] bloc97. Ntk-aware scaled rope allows Ilama models to have extended (8k+) con- text size without any fine-tuning and minimal perplexity degradation. ittps:
/ //www. reddit. http////LooaLLMMMM comments/ conmentss/41//////rtt/ 1412/ scaled rope
allows 1lama models to have/, 2023.
[19] emozilla. Dynamically scaled rope further increases performance of long context llama with zero fine-tuning. dynamically scaled rope further increases/, 2023.
[20] Dacheng Li*, Rulin Shao", Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can open-source ilms truly promise context length?, June 2023.
[21] Kamradt G. Needle in haystack pressure testing llms. LLMTest Needle NeedLesmAHaastack, 2023.
[22] Yutong Hu, Quzhe Huang. Mingxu Tao, Chen Zhang, and Yansong Feng. Can perplexity reflect large language model's ability in long text understanding? In The Second Tiny Papers Truck ICLR 2024, 2024.
[23] Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapoolttbbe transformer. arXiv preprint arXiv:221.100554 2022.
[24] Together Computer. April 2023.
Redpajama: An open source recipe to reproduce llama training dataset,
[25] Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for Long-range sequence modelling. In International Conference on Learning Representationss 2019.
[26] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What's 's's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024.
[27] Ta-Chung Chi, Man Fan, Alexander Rudnicky, and Peter Ramadge. Dissecting trans- former length extrapolation via the lens of receptive field analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),
pages 13522-13537, 2023.
[28] Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.04805. 2018.
[29] Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, and Kentaro Inui. Shape: Shifted absolute position embedding for transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3309-3321, 2021.
12

---

[30] Hailiang Li, YC Adele, Yang Liu, Du Tang, Zhibin Lei, and Wenye Li. An augmented trans- former architecture for natural language generation tasks. In 2019 International Conference on Data Mining Workshops (ICDMW), pages 1-7. IEEE, 2019.
[31] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position rep- resentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Papers), pages 464 0664466,777 2018
[32] Guolin Ke, Di He, and Tie- Yan Liu. Rethinking positional encoding in language pre-training. In International Conference on Learning Representations, 2020.
[33] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Scao, Thibaut Lavril Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[34] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024.
[35] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint /2.0052,, 2023.
[36] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derezynski, al. Rwkv: Reinventing rns for the transformer era. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14048-14077, 2023.
sequence modeling. Advances in Neural Information Processing Systems, 36, 2024. [37] Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for
[38] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long- document trans- former. arXiv preprint arXiv:2004.00 150, 2020.
[39] Joshua Ainslie, James Lee-Thope Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[40] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Process- ing Systems, 35:16344 16359, 2022.
[41] Hao Liu, Matei Zaharia, and Picter Abbeel. Ring attention with blockwise transformers for near-infinite context. In NeurlPS 2023 Foundation Models for Decision Making Workshop. 2023.
[42] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length Ilm inference with kv cache quantization. arXiv preprint arXiv:240I. 18079, 2024.
[43] Szymon Tworkowski, Konrad Staniszewski. Mikolaj Pacek, Yubuai Wu, Henryk Michalewski, and Piotr Miloš. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Processing Systems, 36, 2024.
[44] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model par- allelism, 2020.
[45] Amirkeivan Mohtashami and Martin Jaggi. Random-acces infinite context length for trans- formers. Advances in Neural Information Processing Systems, 36, 2024.
13

---

A The proof of Theorem
Assuming that the components of query q E Rd and key R are independent, their standard deviations are denoted E It and the means are donated as E R The key k" similar to is q + e, where random variable with a mean O. Then, we have:
E Ey q" Rmnk" - Eq Rink
-ngg" Rmq Eg, Rm Eg q." Rank
d/2-1
=Bg
Σ
(12s cos(m);) - (222241sin((88) + (2:+192sin(m88) + G5441 cos(m8;)) + Egg" E.
1=0
d/2-1
Eggk
Σ
(92rky, cos()),))
921k2441sin(m6)) + 92s+1 + 92s+ 92441h2141 cos(m);))
d/2
Σ
E(92x)
cos(m),)) ( Maspus=1sin((mm3) + 1) m.
1=0
d/2-1
Σ
(ngs cos(m);) PasMas-sin((m.)) + pepes--sin(mmB) + 12241 cos(ml:))
1=0
4/2-1
Σ
(Elge) +1211) - 12 - 12x+2) cos(mo))
i=
4/2-1
Σ i=U
(a) o241) cos(m):)
(12)
Then we can get:
d/2-1
Σ
(oz + = cos(mo:) = Eyg,. Bonk" - Eyg Rank
(13)
Om0
And when all are equal, we can get:
4/2-1
1 (y q" Rmnk" Ey Rmm
cos(m),)
(14)
20
B The detail setting of experiment
For training, we mainly conducted experiments on Llama2-7B [ and mnichuan2-7B [8]. In addi- tion, we also trained 2B model from scratch, whose structure the same as Baichuan2-7B-Ban but with smaller hidden size 2048. Both training and testing are accelerated by FlashAttention-2 [H]] and Megatron-L.M [_ The dataset of both fine-tuning and training from scratch is subset of RedPajama [ The hyperparameters of training listed in Appendix !L All experiments conducted on a cluster machines with 128 NVIDIA A100 80G.
Table 4: Training hyper- parameers in our experiments
Model
Training length
Training tokens
Batchsize
Base LR
LR decay
Weight decay
Llama2-7B-Bann
32K
4B
128
2e5
constant
0
Baichuan2-7B- Base
32K
4B
128
2e5
constant
0
Our-2B-Base
4K
1T
1024
2c4
cosine
0.1
14

---

Below record lines you remember. Each line begins with 'line <line index>' and contains '<REGISTER CONTENT> at the as numerical each line index, memorize its corresponding <REGISTER CONTENT>, the end of the record, will ask you to retrieve the corresponding <REGISTER CONTENT>> of a certain line index, Now the record start:
Question:
line swift-baby: REGISTER CONTENT is <12821>
line dangerous- brrgerous-breast:: REGISTER CONTENT <28051> line bad- sculsculptt REGISTER CONTENT <32916>
line flashy-college REGISTER CONTENT <34027>
line voiceless lorochure: REGISTER CONTENT <8964>
line fast- fast-peony: REGISTER CONTENT <5218>
Now the record is over. Tell me what is the <REGISTER_CONTENT: in line dangerous cangerous-breast? breast? need the number. Answer:
Figure 8: Long-eval sample prompt
For evaluation, test the long context capabilities comprehensively, the benchmarks listed below: perplexity on PG19 [25]] test split. We evaluate the perplexity of each sample and get the mean value across samples.
Long-eval [20]. This test generates massive random similar sentences and asks the model to answer questions according to specific sentence in the context. Because the long context consists of many similar patterns, it's more difficult to get the right answer. We find this test harder than other long context evaluations such Perplexity, Passkey Retrieval [AS], Needle in Haystack [RT]]. A test sample is list in Figure g
needle in haystack(NIH) [2I]]. NIH tests the long context capability not only under different context lengths but also at different positions where the correct answer located in the context, which provides more detailed view of the long context capability.
C Baichuan2-7B:. 2-B-Base: Lower bound Base of RoPE
20
32K-base:1ed 1e4
0.6
18
base:2s
16
32K hhso::e5.
Accuracy 32K
0.5
32K base:5e6
Perplexity
14
33K bxsse:e9
0.4
12
32K-base 1e12
0.3
Buseline
10
0.2
0.1
0.0
Lower bound
164
2659e5
le12
Context
Base value
(a) Perplexity
(b) Long-eval 32k
Figure 9: fine-tuning Baichuan2-7B-Bacn on 32k context length with varying RoPE's base. Al- though the perplexity remains low with varying bases, the Long-eval accuracy reveals discernible bound for the base value, below which the Long-eval accuracy declines significantly. the dotted line denotes the lower bound derived from Eq. I
D Long Context Test Results Various LLMs
15

---

Carcod
Coldect Deggt
10a0 Lse
Figure 10: Llama2-7B-Ban with base=le4 fine-tuned on 32k context (original context=4096)
Garood
Figure 11: Llama2-7B- BB-Base with base=2e5 fine-tuned on 32k context (original context=4096)
155
10:00
3000
2030
Ouclod keaz
Figure 12: Baichuan2-7B-Baan with base=le4 fine-tuned on 32k context (original context=4096)
Canroc 60
Figure 13: Baichuan2-P7 BB-aase with base=2e5 fine-tuned on 32k context (original context=4096)
Deplott
Contect
Figure 14: Wwenl.5-7B.B.... 5--B-Base [9] with base=le4 fine-tuned on 32k context (original context=4096)
16

---

E The python code for calculating the low bound base for a context length of 32k
""" the python code for calculate the
low bound base for context length of 32k""H import torch
import numpy as np def get BMtheta eetpetta_exeetaatinn (base, context_size=22t1_ dim=128)
realdim dim / 2 d- torch arange (0, realdim, theta base dist= torch outer (torch.. co thhta) cos
(-2+d/dim)
10
search_base dumt.aum (dim==1)) / realdim sum
return dist
[]
for range (3,10);
13
for i vange(1,10); (1,10);
14
for j in range (10):
15
search_bast basch_base.. append ((1+)/10) (10**x))
for base in search_base:
17
ans get_BNtheta__eye expectaateitiionnn (base)
18
True not (ans <0):
19
prinnd"FidlBaase" break
base)
idx up. argmax (ans 0)
22
print ('base', base, 'first zero position', idx)
F A empirical verification of Desiderata 2
The Desiderata 2 introduced in Section H is intuitively plausible, but its empirical validity requires verification. investigate this, we conducted a detailed empirical analysis. The similarity between tokens is measured by the cosine similarity (denoted as A) of their corresponding hidden states, while the attention allocation between tokens governed by the attention score (denoted as B). The desiderata "similar tokens receive more attention" implies that higher value of A should lead to higher value B.
To test this desiderata, we performed experiments using Llamal-7B, Llama2-7B. and Llama3-8B models. We selected 200 segments from the PG19 dataset, each containing 1024 tokens, and com- puted Spearmans rank correlation coefficient between (A) and (B). A positive correlation coefficient would indicate that as token similarity (A) increases, the corresponding attention score (B) also increases. The magnitude of the coefficient reflects the strength of this correlation.
The results, presented in Figure II confirm that Spearmans rank correlation coefficient is positive, validating the desiderata that "similar tokens receive more attention". Furthermore, we observe that this positive correlation is pronounced in the Llama3-8B model compared Llama2- 7B and Llamal-7B, suggesting that larger and more advanced models are better at capturing this relationship.
G Evaluation results on RULER
17

---

Brrad
Figure 15: Spearmans rank correlation coefficient between the similarity and the attention score of two tokens. The thick line represents the mean calculated from different samples. The upper and lower boundaries of the line are μ + a and μ a, respectively, where the standard deviation of different samples.
Table 5: Evaluation results on RULER. We finetune Llama2-7b to 32k context length (the low bound base is 6e5) using different RoPE's bases. NS is short for NIAH-single and NM is short for NIAH- Multikey.
Sab tasks
NS-1
NS-2
NS-3
NM-1
NM-2
NM-3
NIAH. Multivalue
NIAH Multiqeryy
VT
CWE
FWE
QAI
QA2
4k
23.0
31.0
26.0
31,0
13.0
11.0
73.0
75.0
2.0
59.5
53.7
51.0
20.0
36.78
Sk.
15.0
18.0
11.0
14.0
2.0
3.0
30.0
37.0
1.0
46.3
46.0
18.0
21.0
21.72
16k
5.0
8.0
5.0
9.0
2.0
1.0
28.0
33.0
0.0
23.0
40.7
23.0
27.0
15.74
32k
1.0
1.0
2.0
4,0
1.0
1.0
10.0
12.0
0.0
1.5
22.7
16.0
24.0
7,40
4k
99.0
100
96.0
91.0
$5.0
65.0
66.0
99.0
90.0
34.1
77.33
86.0
41.0
71.88
1e4
8k.
53.0
55.0
58.0
59.0
54.0
4.0
49.0
$4.0
1.0
33.7
27.67
30.0
20.0
39.90
16k
21.0
24.0
25.0
36.0
17.0
3.0
72.0
75.0
0.0
49.3
9.67
10.0
25.0
28.38
32k
8.0
11.0
13.0
7.0
0.0
38.0
0.0
17.1
1.33
19.0
14.19
4k
100
100
100
97.0
07.0
77.0
99.0
99.0
100
79.6
86.0
45.0
45.0
86.51
8k.
100
100
100
100
96.0
48.0
97.0
100
42.9
65.00
44.0
40.0
79.46
205
16k
100
23.0
20.7
8.33
68.23
6.5
66.33
78
74.16
12.0
96.
48.4
79.27
16k
74.45
18.5
17.26
86.40
Sk
100
100
97.0
68.0
92.0
47.6
70.3
81.38
16k
100
100
100
100
91.0
90.0
55.0
$6.0
100
100
28.0
53.7
35.0
79.00
32k
100
100
100
97.0
66.0
33.0
$1.0
100.0
9.7
64.0
20.0
39.0
67.67
4k
100
100
100
95.0
96.0
72.0
100
67.0
63.8
77.7
41.9
29.0
80.11
8k
100
100
100
96.0
90.0
$4.0
95.0
100
85.0
35.0
60.0
25.0
35.0
75.46
16k
100
100
100
96.0
77.0
43.0
83.0
100
72.0
23.7
$1.3
27.0
38.0
69.85
32k
100
100
100
93.0
69.0
23.0
58.0
92.0
94.0
18.1
58.7
17.0
35.0
65.75
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claims the paper's contributions and scope in the abstract and the last part the introduction in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Section A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:[Yes
19

---

We provide the assumptions of Theorem M| in itself and provide the proof in Appendix A
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the experimental details and hype-parameters in Section b.ll and Appendix R
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or the contribution it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate the research performed.
specific model and empirical evaluation,
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
20

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Due some privacy policies and the complexity of training large language models, we are unable to provide the data and code. But we believe that based on the open-source LLMs and open-source code, our results be reproduced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips. httpp/sccc/publc/uuuueeeee wtweredeeeeCooSeemieeeeeeeee
ssionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer:
[Yes]
Justification: We provide the experimental details and hype-parameters in Section b.l and Appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the high computational cost of large language models, it is difficult for us to perform the same experiment multiple times to get the error bar. However, we provide the results of different models and their performance under various evaluation metrics to support our perspective.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the enror bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative еπτог rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information in Section B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and the research conducted in our paper conform, in every respect, with the NeurlPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper focuses on the impact of key parameter settings in the model on its capability. To our knowledge, this does involve any social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e-g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: In this work, we don't release any data or model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions to access the model or implementing safery filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:[Yes)
Justification:
We cite existing papers and url.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode com/datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection. cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "blofoqDfaQ", "reviewer": "Reviewer_mP48", "summary": "Hi Area Chair, I am not qualified for the review of this paper, this is out of my knowledge scope.", "questions": "n/a", "limitations": "n/a", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "n/a", "weaknesses": "n/a", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8EoAOiri8S", "reviewer": "Reviewer_3a2j", "summary": "This paper investigates the role of Rotary Position Embedding (RoPE) in Large Language Models (LLMs), with a focus on the relationship between RoPE's base and the model's long context ability. \nThe study looks into the long-context abilities and limitations of current methods that rely on smaller RoPE bases. With a lower base, LLM models may exhibit superficial long-context abilities, achieving low perplexity but failing to retrieve relevant information in extended contexts.\nTheoretically, The paper establishes two desiderata for the attention mechanism in language modeling: 1.Closer tokens receive more attention. 2.Similar tokens receive more attention. It examines these when applying RoPE to LLMs, revealing a long-term decay in attention score and the ability to differentiate similar from random tokens. This leads to a theorem indicating that RoPE’s base sets an absolute lower bound for achieving specific context lengths.", "questions": "For the suggestions, please refer to the weaknesses section.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clarity and Technical Correctness: The paper is clear and technically sound, with theoretical and empirical analyses.\n\n2. Experimental Rigor and Reproducibility: It includes extensive experiments to back up the theoretical findings, along with detailed setup and results.\n\n3. Novel Findings: This paper presents a critical study of whether we should use smaller bases for continuous training, as suggested by previous work. Furthermore, this paper presents a novel perspective on long-term decay, as well as an absolute lower bound on the RoPE base parameter required for specific context lengths. This adds new knowledge to the field and improves our understanding of position embedding in LLMs.", "weaknesses": "Overall, I like this paper. I would like to suggest that the authors strengthen their work by considering the following points:\n\n1. Extensively test the model using benchmarks such as RULER [1].\n\n2. Provide more empirical observations on the relationship between the base of RoPE and model performance on those challenging benchmarks.\n\n\n--\n\n[1] https://github.com/hsiehjackson/RULER", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q:Extensively test the model using benchmarks such as RULER. Provide more empirical observations on the relationship between the base of RoPE and model performance on those challenging benchmarks.**\n\nA: We greatly appreciate your suggestion. We evaluated  Llama2-7b on RULER, and the evaluation results are shown in Table 1. We can observe that when the base value exceeds the lower bound 6e5 given in our paper for 32k context, the model can achieve better performance on the RULER. And when base is greater than 6e5, the improvement is slight. The detailed evaluation results on various sub tasks are presented in Table 2. \nWe will include these results in the revisions of our paper.\n\n**Table1. Comparison on Ruler, when fine-tuning Llama2-7b to a length of 32k ( the low bound base is 6e5) under different settings of RoPE's base**\n| Base || Length |  | |\n| - | - | - | - | - | \n| |4k|8k|16k|32k|\n| 500   | 36.78   | 21.72|15.74|7.40|\n| 1e4  | 77.88   | 39.80  | 28.38  | 14.19   |\n| 2e5   | 86.51 | 79.45  | 68.23  |  61.08 |\n| 6e5   | **87.12**   | 78.36  | 74.16  | 65.63  |\n| 9e5   | 86.76   | **79.87**  | **74.64**  | **66.65**  |\n| 5e6   | 86.91 | 79.27 | 74.45  | 66.24|\n| 1e9   | 80.04  | 75.46 | 69.85  |65.75 |\n\n\n**Table2. The detail results on sub-tasks of RULER. Niah_single is short for ns. Niah_multikey is short for nm.**\n|base|||ns1|ns2|ns3|nm1|nm2|nm3|niah_multivalue|niah_multiquery|vt|cwe|fwe|qa_1|qa_2\n| - | - | - | - | - |  - | - | - | - | - | - | - | - | - | - |  - | \n|500| length|Average Score|  |  |   |  |  |  |  |  |  |  |  |  |   | \n| | 2k |46.04| 57.0 |67.0| 48.0  |45.0  | 17.0 | 8.0 | 83.0 | 89.0 | 6.0 | 44.5 |45.0  |57.0  | 32.0  | \n| | 3k |34.04| 36.0 | 31.0 | 19.0  | 31.0 | 13.0 |3.0  | 74.0 | 67.0 | 3.0 |20.8  | 57.7 | 57.0 |  30.0 | \n| | 4k |36.78| 23.0 | 31.0 | 26.0  |31.0  | 13.0 | 11.0 | 73.0 | 75.0 | 2.0 | 59.5 | 53.7 | 51.0 |  29.0 | \n| | 8k | 21.72|15.0|18.0|11.0|14.0|2.0|3.0|50.0|37.0|1.0|46.3|46.0|18.0|21.0|\n| |16k|15.74|5.0|8.0|5.0|9.0|2.0|1.0|28.0|33.0|0.0|23.0|40.7|23.0|27.0|\n| |32k|7.40|1.0|1.0|2.0|4.0|1.0|1.0|10.0|12.0|0.0|1.5|22.7|16.0|24.0|\n|1e4| length|Average Score|  |  |   |  |  |  |  |  |  |  |  |  |   | \n| | 4k |77.88| 99.0 | 100 | 96.0  | 91.0 | 85.0 | 65.0 | 66.0 | 99.0| 90.0|  34.1|77.33  | 66.0 | 44.0  | \n| | 8k |39.80| 53.0 | 55.0 | 58.0  |59.0  |34.0  | 4.0 | 49.0 | 84.0 | 1.0 | 33.7 |27.67  | 30.0 | 29.0  | \n| | 16k |28.38| 21.0 | 24.0 | 28.0 | 36.0 | 17.0|3.0 | 72.0| 75.0| 0.0| 49.3| 8.67| 10.0|25.0 | \n| | 32k |14.19|5.0 | 8.0| 11.0| 13.0|7.0|0.0|38.0|39.0|0.0| 17.1|1.33|19.0|26.0| \n|2e5| length|Average Score|  |  |   |  |  |  |  |  |  |  |  |  |   | \n| | 4k |86.51| 100 | 100 | 100  | 97.0 |97.0  | 77.0 | 99.0 |99.0  |100  |79.6  | 86.0 | 45.0 | 45.0  | \n| | 8k |79.46| 100 |100  | 100  |100  | 96.0 | 48.0 | 97.0 | 100 | 100 | 42.9 | 65.00 |44.0  | 40.0  | \n| | 16k |68.23| 100 | 100 | 100  |97.0  | 74.0 | 23.0 | 92.0 | 100 |97.0 | 20.7 | 8.33 | 38.0 | 37.0  | \n| | 32k |61.08|99.0  |100.0  | 95.0  | 95.0 | 32.0 |9.0  | 62.0 | 87.0 | 82.0 |27.0  | 39.0 |29.0  |38.0   | \n|6e5| length|Average Score|  |  |   |  |  |  |  |  |  |  |  |  |   | \n| | 4k |87.12|100  |100  |100   |97.0  | 96.0|65.0| 99.0|100|100|84.6|90.0| 52.0|49.0 | \n| | 8k |78.36| 100| 100| 100 | 99.0 |96.0  |40.0  | 93.0 |100| 100|43.4 |66.33|34.0|47.0 | \n| | 16k |74.16|100|100| 100| 95.0| 74.0|37.0| 93.0|99.0|98.0|27.4| 62.67|37.0| 41.0 | \n| | 32k |65.63|100| 100| 94.0 |96.0 | 47.0 | 12.0 |70.0  |89.0  |97.0  | 20.5 | 63.67 | 25.0 |39.0   | \n|9e5| length|Average Score|  |  |   |  |  |  |  |  |  |  |  |  |   | \n| | 4k |86.91| 100 | 100 | 99.7  |97.0  |95.1  |71.0  |99.0  |99.7  |100  | 83.6 |88.5  |49.3  |46.9| \n| | 8k |79.27| 100 |100  |100   |98.4  |96.3  |48.4  |92.7  | 100 | 100 |44.66  |67.53  |35.8  |46.7| \n| | 16k |74.45| 100 |100  |100   |93.8  |78.8  |42.4  |90.9  |99.3  |98.6  |27.58  |59.97  |36.4  |40.1| \n| | 32k |66.24| 100|100  |95.8   |96.3  |52.7  |18.3  |64.3  |89.6  |97.9  | 17.26 | 63.77 |26.2  |39.0| \n|5e6| length|Average Score|  |  |   |  |  |  |  |  |  |  |  |  |   | \n| | 4k |86.40|100  |100  |99.0|97.0|93.0|85.0|99.0|99.0|100.0|81.2|85.0|43.0|42.0|\n| | 8k |81.38|100  |100  | 100  | 97.0|97.0|68.0|92.0|100|100|47.6|70.3|40.0|46.0|\n| | 16k |75.13|100  |100  | 100  |100|91.0|90.0|55.0|86.0|100|100|28.0|53.7|35.0|38.0|\n| | 32k |67.67| 100 |100  | 100  |97.0|66.0|33.0|51.0|91.0|100.0|9.7|64.0|29.0|39.0|\n|1e9| length|Average Score|  |  |   |  |  |  |  |  |  |  |  |  |   | \n| | 4k |80.04| 100  |100  |100   | 95.0|96.0  | 72.0|100  |99.0  |67.0  |63.8  |77.7  |41.9  |29.0   | \n| | 8k |75.46| 100  |100  | 100  | 96.0|90.0  |54.0  |95.0  |100  |88.0  |35.0  |60.0  |   28.0| 35.0|\n| | 16k |69.85| 100  |100  | 100  |96.0|77.0|43.0 |83.0  |100  | 72.0 |23.7 |51.3 | 27.0 |35.0 | \n| | 32k |65.75|  100 |100  | 100  |93.0|69.0|23.0 |58.0 |92.0|94.0| 18.1|55.7| 17.0| 35.0 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F3Li7nT1Rt", "reviewer": "Reviewer_XTUy", "summary": "ROPE is wildily employed in popular LLMs which encodes positional information with a rotation matrix. Although RoPE is used to enhance long-context capabilities by adjusting its base parameter to address OOD issues, this paper finds that this may result in only superficial long-context abilities. Authors re-evaluate RoPE's role and introduce a novel property of long-term decay, showing that the base of RoPE limits context length, with an absolute lower bound required for certain capabilities. This work clarifies the theoretical and empirical relationship between context length and the RoPE base, offering insights for future long-context training.", "questions": "See Weaknesses", "limitations": "See Weaknesses", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The two provided desideratas are highly logical and well-aligned with language modeling. The assumption made in this paper is quite reasonable and closely aligned with practical scenarios.\n\n- Insightful analysis. The theoretical results presented in this paper are easy to understand. To the best of my knowledge, the final bound for the ROPE base is novel and first introduced in this paper", "weaknesses": "There is no obvious weaknesses in my opinion. I just have a few questions:\n\n- Regarding the \"Desiderata 2 The similar token gets more attention\", recently StreamLLM [1] shows that there exists \"attention sink\" in popular LLMs. Namely most of tokens attend to the first few tokens. This somehow contradicts with the principle that \"the similar token gets more attention\". Could you provide your thoughts on this statement?\n\n- Anthropic's blogs reveal that different heads may have different functionality in in-context learning. How may this interplay with the Rope base? Do you think different heads may have different optimal rope base?\n\n[1] Efficient Streaming Language Models with Attention Sinks\n[2] In-context Learning and Induction Heads", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:Regarding the \"Desiderata 2 The similar token gets more attention\", recently StreamLLM  shows that there exists \"attention sink\" in popular LLMs. Namely most of tokens attend to the first few tokens. This somehow contradicts with the principle that \"the similar token gets more attention\". Could you provide your thoughts on this statement?**\n\nA1: \nFirstly, we think that they are not in conflict in phenomenon. We think that the sink token can be considered as a form of regularization to focus on and serves as an anchor point for positional information.  “most of tokens attend to the first few tokens” and  \"the similar token gets more attention\" are two distinct properties refer to different aspects. The former means \"current token tends to pay more attention to the first few tokens than the tokens in other positions\",  the latter means \"At the same relative position, current token tends to pay more attention to the similar token than random tokens\". They can actually both be satisfied at the same time. For example, when similar token exists, the current token give a weight of 0.8 to first token and 0.2 to similar token. When similar token does not exist, the current token give a weight of 1.0 to first token and 0 to other tokens. \n\nMore importantly, as StreamLLM say \"While StreamingLLM improves the efficiency of LLMs in streaming contexts, it does not extend the models’ context window or enhance their long-term memory capabilities. \"  \"most of tokens attend to the first few tokens\" may lead to low perplexity, but can't extend the context window.  While based on \"the similar token gets more attention than random ones\", we can adjust RoPE's base to achieve good performance on more challenge benchmarks rather than only low perplexity. \n\n**Q2: Anthropic's blogs reveal that different heads may have different functionality in in-context learning. How may this interplay with the Rope base? Do you think different heads may have different optimal rope base?**\n\nA1:  According to our research, for example, in the induction heads mechanisms proposed in Anthropic's blogs,  the head that retrieves long-distance information may require a larger base, while a copy head that simply copies the information of the previous token and only focuses on nearby tokens may only require a small base. So we believe that different heads require different optimal base. However, setting different bases for  different heads is a highly challenging task, perhaps a search method similar to LongRoPE[1] can be used. \n\n[1] LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OgAFWTVQtT", "reviewer": "Reviewer_aPSb", "summary": "The paper investigates the role of RoPE in long-context LLMs. It highlights that the base of RoPE crucially affects the model's ability to handle long contexts. This paper derives a theoretical (and empirical) lower bound for the base value required to maintain long-context capabilities and validates this through empirical experiments with models like Llama2-7B and Baichuan2-7B, and a 2B model trained from scratch. This work offers insights into the base of RoPE for long-context processing in LLMs, which is inspiring for the development and design of long-context LLMs.", "questions": "See **Weaknesses**", "limitations": "See **Weaknesses**", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The lack of long-context ability is still an under-explored question and a very important one. Thus, the research topic of this paper is both theoretically important and practical.\n2. The relationship between the RoPE base and long-context ability is important, and the proposed lower bound is interesting and inspiring.\n3. The experiments in this paper are comprehensive and well support the claims made in the paper.\n4. The presentation of this paper is overall very good and easy to understand and follow.\n5. The claims of this paper are inspiring for the development of long-context LLM.\n6. I am also very interested in RoPE-based selection in LLM and like this paper.", "weaknesses": "1. The Desiderata 2. The similar token gets more attention in Section 4 seems intuitively correct but may not be empirically correct. As this desideratum is the fundamental motivation of Theorem 1, a thorough empirical verification is a must-have.\n2. The motivation section is not well-written or organized. It confuses me while I read this part. These are just some previous empirical observations and are not deeply discussed. As the motivation part is very important, I would like to see a clear and well-organized motivation.\n3. For the Caption of Figure 6, it would be better to show how to derive the value of the dotted line. I would also like to see a detailed derivation here.\n4. Will the author plan to release the models (including the fine-tuned Llama2, Baichun2 with varying lengths, and the 2b model series)? This would benefit future work.\n5. I wonder about the negativeness and positiveness of the $q^{T}R_{m,\\theta}k$ in equations (8) and (9). If the values are negative, say -2 and -1, which one indicates more attention?\n6. For section 5.3, I would like to regard this as a conjecture rather than an explanation.\n7. Moinries\n    - The upper case and lower case of the title are not consistent, such as the title of Section 4.1 and Section 2.2\n    - Line 225 “method2” -> “Method 2”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:The Desiderata 2. The similar token gets more attention in Section 4 seems intuitively correct but may not be empirically correct. A thorough empirical verification is a must-have.**\n\nA1: In our paper, the similarity between tokens measured by the cosine similarity(A) of their corresponding hidden-states,  and the amount of attention is determined by the attention score(B). \"Similar token gets more attention\" means that a larger (A) leads to a larger (B).\n\nTo validate this desiderata, we conducted experiments on Llama1-7B, Llama2-7B, and Llama3-8B. We extracted 200 segments from PG19, each consisting of 1024 tokens, and calculated the Spearman’s rank correlation coefficient between (A) and (B). A positive Spearman’s rank correlation coefficient  indicates that as (A) increases, so does (B), and the higher the absolute value of the coefficient, the stronger the positive correlation.\n\nThe results are shown in  Figure 1 in PDF of Global Response.  The positive  Spearman’s rank correlation coefficient  validates the property of \"similar token gets more attention\". And we observe that this positive correlation is stronger in Llama3-8B compared to Llama2-7B and Llama1-7B, suggesting that more powerful models may better learn this positive correlation.\n\n**Q2:The motivation section is not well-written or organized. It confuses me while I read this part. These are just some previous empirical observations and are not deeply discussed.**\n\nA2: Thank you for your suggestions. We will reorganize the motivation in the revised version as follows:\n\nRecent advancements in long-context language models have seen widespread adoption of NTK-based methods [6, 12, 13]. However, a curious trend has emerged: practitioners often employ significantly larger base values than those originally suggested by NTK-aware approaches. This discrepancy raises critical questions about the efficacy of current theoretical frameworks. Why do practitioners deviate from the recommendations of NTK-based methods? Is the out-of-distribution (OOD) theory underlying these methods insufficient to fully unlock long-context capabilities?\n\nOn the other hand, recent research [14], driven by OOD theory, proposes using a much smaller base for RoPE to extend context length. However, our findings, as illustrated in Figure 3, suggest that this approach may only provide superficial long-context capability[21]. While achieving low perplexity even at 128k context length (explicable by OOD theory), the model fails to retrieve relevant information for context lengths as short as 1k—well below its pre-trained length. The observation suggests that the small base determined by OOD theory can't unlock true long-context capability.\n\nThese phenomena motivate us to delve deeper into the relationship between RoPE's base and context length. To address the gap between OOD theory and our observations, we conduct a theoretical exploration in the next section, aiming to uncover the underlying mechanisms of effective long-context modeling. \n\n**Q3: For the Caption of Figure 6, it would be better to show how to derive the value of the dotted line.**\n\nA3: Thank you for pointing out this. We apologize for not providing specific procedures and pseudocode for calculating the values in the paper. We will provide detailed information in the revised version. \n\nIn our paper, we adopted an imprecision search. Specifically, we traversed the base values of $a.b\\times 10^x$, where a, b, and x are all Arabic numerals, e.g. $2.3\\times 10^6$. We then observed whether $B_{\\theta, m} $ can always be non negative within the window length and select the minimum base that meets the conditions. The python code is in the in PDF of Global Response.\n\n**Q4:Will the author plan to release the models? This would benefit future work.**\n\nA4: We are happy to open source these models and hope that they can benefit future work. However, due to the requirements of rebuttal, we are unable to provide a download link here. After the decision is made on the paper, we will release all the models used in the paper. \n\n**Q5: I wonder about the negativeness and positiveness of the in equations (8) and (9). If the values are negative, say -2 and -1, which one indicates more attention?**\n\nA5: In equations (8) and (9), -1 indicates more attention.\n\n**Q6: For section 5.3, I would like to regard this as a conjecture rather than an explanation.**\n\nA6: It seems that we didn't provide much explanation in section 5.3. In section 5.3, we train a 2B model with small base from scratch and find that it has a pool long context capability, which is consistent with our theoretical perspective.\n\nPerhaps you are referring to section 5.4, so we answer according to the content of section 5.4 below. \n\nThank you for pointing it out. We agree that this is not a strict explanation. There are many different perspectives on positional encoding, and we are providing an interpretation for the strange phenomenon \"superficial long context capability for small base\" from the theory perspective proposed in our paper. This isn't a strict proof, and further research is needed on the performance of models and positional encoding.\n\n**Q7:Moinries: The upper case and lower case of the title are not consistent, such as the title of Section 4.1 and Section 2.2; Line 225 \"method2\" -> \"Method 2\"**\n\nA7: Thank you for carefully reviewing our paper. We will address the formatting and grammatical errors in revisions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers, Area Chairs, and Program Chairs:

We would like to express our gratitude to all reviewers for taking their valuable time to review our paper. We sincerely appreciate all reviewers for their positive comments on our theoretical analysis, technique soundness and contribution. 

Meanwhile, we appreciate the reviewers for pointing out the weaknesses. Your valuable comments help us improve the paper. We try to address each comment as satisfactorily as possible. In the response, we:

a)  compare on a comprehensive benchmarks RULER with 13 sub tasks

b)  discuss some related work such as StreamLLM and induction heads 

c)  verify the Desiderata 2 "The similar token gets more attention" empirically on Llama series.

Please find the responses to each reviewer’s comments below. Due to the page limitations, some replies may not be able to provide a detailed description, we welcome any further discussion with the reviewers.

Best regards,

Paper14302 Authors

Meta-review: The paper investigates the role of Rotary Position Embedding (RoPE) in Large Language Models (LLMs), focusing on the relationship between the RoPE base and the model's long-context ability. The study introduces a theoretical lower bound for the RoPE base, validated through empirical experiments. The findings provide actionable insights for practitioners looking to optimize RoPE-based LLMs for longer contexts without superficial gains. The paper highlights potential pitfalls in using small base values, which can lead to a superficial long-context capability.

The reviewers found the paper to be technically sound and impactful, with a strong recommendation for acceptance after addressing the noted weaknesses. The insights provided by this work are expected to benefit the development and optimization of long-context LLMs.
Decision: Accept (poster)