Title: Base of RoPE Bounds Context Length
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large Language Model

Long context

Rotary position embedding
Abstract: Position embedding is a core component of current Large Language Models (LLMs). Rotary position embedding (RoPE), a technique that encodes the position information with a rotation matrix, has been the de facto choice for position embedding in many LLMs, such as the Llama series. RoPE has been further utilized to extend long context capability, which is roughly based on adjusting the \textit{base} parameter of RoPE to mitigate out-of-distribution (OOD) problems in position embedding. However, in this paper, we find that LLMs may obtain a superficial long-context ability based on the OOD theory. We revisit the role of RoPE in LLMs and propose a novel property of long-term decay, we derive that the \textit{base of RoPE bounds context length}: there is an absolute lower bound for the base value to obtain certain context length capability. Our work reveals the relationship between context length and RoPE base both theoretically and empirically, which may shed light on future long context training.

Base of RoPE Bounds Context Length

Abstract

Position embedding is a core component of current Large Language Models (LLMs). Rotary position embedding (RoPE), a technique that encodes the position information with a rotation matrix, has been the de facto choice for position embedding in many LLMs, such as the Llama series. RoPE has been further utilized to extend long context capability, which is roughly based on adjusting the base parameter of RoPE to mitigate out-of-distribution (OOD) problems in position embedding. However, in this paper, we find that LLMs may obtain a superficial long-context ability based on the OOD theory. We revisit the role of RoPE in LLMs and propose a novel property of long-term decay, deriving that the base of RoPE bounds context length : there is an absolute lower bound for the base value to obtain certain context length capability. Our work reveals the relationship between context length and RoPE base both theoretically and empirically, which may shed light on future long context training.

10

3

10

4

10

5

10

6

Context length

10

4

10

5

10

6

10

7

10

8

Lower bound of RoPE's base

y

= 0

.

0424

x

1

.

628

1 Introduction

In the past few years, large language models have demonstrated surprising capabilities and undergone rapid development. By now, LLMs have been widely applied across various domains, including chatbots, intelligent agents, and code assistants [2, 3]. The Transformer [4], based on the attention mechanism, has been the most popular backbone of LLMs due to its good performance and scaling properties [5]. One of the key component modules in the Transformer is position embedding, which is introduced to embed positional information that is vital for processing sequential data. Rotary position embedding (RoPE), which encodes relative distance information in the form of absolute position embedding [6], has been a popular choice and applied in many LLMs [7, 8, 9].

RoPE introduces no training parameters and shows improvement in language modeling and many other tasks [6, 10]. One reason that RoPE is widely used is its ability for context length extrapolation [11, 12], which extends the context length of a trained LLM without expensive retraining. In practice, many works [7, 13, 14] have successfully extended the window length by simply increasing base value, the only one hyper-parameter in RoPE, and fine-tuning on long texts.

The reasons behind the success of these long context extensions are often explained as avoiding out-of-distribution (OOD) rotation angles [15, 16] in RoPE, meaning the extended context length (OOD) can be mapped to the in-distribution context length that has been properly trained. Based on the OOD theory, a recent study [15] finds that a smaller base can mitigate OOD and is beneficial for the model's ability to process long contexts, which inspires us to further study the relationship between the base of RoPE and the length of context the model can process.

In this paper, we find that the model may show superficial long context capability with an inappropriate RoPE base value, in which case the model can only preserve low perplexity but loses the ability to retrieve long context information. We also show that the out-of-distribution (OOD) theory in position embedding, which motivates most length extrapolation works [11, 12, 15], is insufficient to fully reflect the model's ability to process long contexts. Therefore, we revisit the role of RoPE in LLMs and derive a novel property of long-term decay in RoPE: the ability to pay more attention to similar tokens than random tokens decays as the relative distance increases. While previous long context works often focus on the relative scale of the RoPE base, based on our theory, we derive an absolute lower bound for the base value of RoPE to obtain a certain context length ability, as shown in Figure 1. To verify our theory, we conducted thorough experiments on various LLMs such as Llama2-7B [17], Baichuan2-7B [8] and a 2-billion model we trained from scratch, demonstrating that this lower bound holds not only in the fine-tuning stage but also in the pre-training stage.

We summarize the contributions of the paper as follows:

Theoretical perspective : we derive a novel property of long-term decay in RoPE, indicating the model's ability to attend more to similar tokens than random tokens, which is a new perspective to study the long context capability of the LLMs.

Lower Bound of RoPE's Base : to achieve the expected context length capability, we derive an absolute lower bound for RoPE's base according to our theory. In short, the base of RoPE bounds context length.

Superficial Capability : we reveal that if the RoPE's base is smaller than a lower bound, the model may obtain superficial long context capability, which can preserve low perplexity but lose the ability to retrieve information from long context.

2 Background

In this section, we first introduce the Transformer and RoPE, which are most commonly used in current LLMs. Then we discuss long context methods based on the OOD of rotation angle theory.

9

4

9

6

E

§

glyph[unionsq]

glyph[ceilingright]

\

glyph[ceilingleft]

glyph[floorright]

glyph[wreathproduct]

\

glyph[unionsq]

glyph[ceilingright]

§

0

10000

20000

30000

Context-size

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

RoPE

Extended context

(a) base=1e4

0

10000

20000

30000

Context-size

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

Extended context

(b) base=500

0

10000

20000

30000

Cglyph[wreathproduct]\glyph[unionsq]glyph[ceilingright]§↖∫〉‡glyph[ceilingright]

0

:

90

0

:

92

0

:

0

:

0

:

98

1

:

00

(c) base=

b

s

d

d

-

2

·

2.1 Attention and RoPE

The LLMs in current are primarily based on the Transformer [4]. The core component of it is the calculation of the attention mechanism. The naive attention can be written as:

where A ∈ R L × L q, k, v ∈ R d . Position embedding is introduced to use the order of the sequence in attention.

RoPE [6] implements relative position embedding through absolute position embedding, which applies rotation matrix into the calculation of the attention score in Eq. 1, which can be written as:

where m = j -i is the relative distance of i and j , R m,θ is a rotation matrix denoted as:

Generally, the selection of rotation angles satisfies θ i = base -2 i/d , the typical base value for current LLMs is 10,000.

2.2 OODtheory of relative rotation angle

Based on RoPE, researchers have proposed various methods to extend the long context ability of LLMs, among which representatives are PI [12] and NTK-series (NTK-aware [18], YaRN [11], and Dynamical-NTK [19]). Those methods depend on the relative scale s = T new /T origin, where T origin is the training length of the original pre-trained model and T new is the training length in long-context fine-tuning.

PI PI directly interpolates the position embedding, and the calculation of A ij becomes:

In other words, the position embedding of the token at position i in pre-training becomes i/s in finetuning, ensuring the position embedding range of the longer context remains the same as before.

NTK-series The idea is that neural networks are difficult to learn high-frequency features, and direct interpolation can affect the high-frequency parts. Therefore, the NTK-aware method achieves high-frequency extrapolation and low-frequency interpolation by modifying the base value of RoPE. Specifically, it modifies the base b of the RoPE to:

The derivation of this expression is derived from T new b -d -2 d new = T origin b -d -2 d to ensure that the lowest frequency part being interpolated.

A recent study [15] proposes to set a much smaller base (e.g. 500), in which case θ i = base -2 i d is small enough and typical training length (say 4,096) fully covers the period of cos( t -s ) θ i , so the model can obtain longer context capabilities.

One perspective to explain current extrapolation methods is the OOD of rotation angle [15, 16]. If all possible values of cos( t -s ) θ i have been fitted during the pre-training stage, OOD would be avoided when processing longer context. Figure 2 demonstrates how these methods avoid OOD of RoPE.

3 Motivation

0

25000

50000

75000

100000

125000

Context size

20

40

60

80

100

Perplexity

finetune on 32k(base=500)

Llama2-7B-Baseline

(a) Perplexity

1000

2000

3000

4000

5000

Context length

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

finetune on 32k(base=500)

Llama2-7B-Baseline

(b) Long-eval [20]

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

1400

1500

1600

1700

1800

1900

2000

2100

2200

2300

2400

2500

2600

2700

2800

2900

3000

3100

3200

3300

3400

3500

3600

3700

3800

3900

4000

Token Limit

0.0

11.0

22.0

33.0

44.0

56.0

67.0

78.0

89.0

100.0

Context length

0

2

4

6

8

10

(c) Needle in Haystack [21]

Score

Recent advancements in long-context language models have seen widespread adoption of NTKbased methods [7, 13, 14]. However, a curious trend has emerged: practitioners often employ significantly larger base values than those originally suggested by NTK-aware approaches. This discrepancy raises critical questions about the efficacy of current theoretical frameworks. Why do practitioners deviate from the recommendations of NTK-based methods? Is the out-of-distribution (OOD) theory underlying these methods insufficient to unlock long-context capabilities fully?

On the other hand, recent research [15], driven by OOD theory, proposes using a much smaller base for RoPE to extend context length. However, our findings, as illustrated in Figure 3, suggest that this approach may only provide superficial long-context capability[22]. While achieving low perplexity even at 128k context length (explicable by OOD theory), the model fails to retrieve relevant information for context lengths as short as 1kwell below its pre-trained length. The observation suggests that the small base determined by OOD theory can't unlock true long-context capability.

These phenomena motivate us to delve deeper into the relationship between RoPE's base and context length. To address the gap between OOD theory and our observations, we conduct a theoretical exploration in the next section, aiming to uncover the underlying mechanisms of effective longcontext modeling.

4 Theory Perspective

For attention mechanism in language modeling, we have the following desiderata:

Desiderata 1 The closer token gets more attention : the current token tends to pay more attention to the token that has a smaller relative distance.

5

7

¬

0

2000

4000

Relative distance

2

0

10

0

12

15

0

Relative upper bound

base:1e4

base:1e2

base:1e3

0

10000

20000

30000

Relative distance

10

15

base:1e4

base:1e5

base:1e6

5

0

2000

4000

Relative distance

0

0

0

2

0

0

base:1e4

base:1e2

base:1e3

0

10000

20000

30000

Relative distance

GLYPH<0>

0

2

0

0

0

2

0

0

base:1e4

base:1e5

base:1e6

¬

Desiderata 2 The similar token gets more attention : the token tends to pay more attention to the token whose key value is more similar to the query value of the current token.

Then we examine the desiderata when we apply RoPE to the attention mechanism in LLMs.

4.1 Long-term Decay of Upper Bound of Attention Score

For Desiderata 1, the property of RoPE makes the model attend more to closer tokens. This kind of long-term decay has been thoroughly discussed in previous work [6, 23]. It comes from the upper bound of attention score calculation, which can be written as:

where h l = q T i [2 l : l 2 + 1] k j [2 l : 2 l +1] . Equation 7 indicates that the upper bound of the attention score | A ij | decays as the relative distance increases. Figure 4 shows the long-term decay curve of this upper bound, which is in accordance with previous findings [6, 23].

4.2 Long-term Decay of the Ability to Attend More to Similar Tokens than Random Tokens

In addition to the attention score's upper bound, we also find there exists another long-term decay property in RoPE: the ability to attend more to similar tokens than random tokens decays as the relative distance increases. We define the ability to attend more to similar tokens than random tokens as:

where q ∈ R d is the query vector for the current token, k ∗ = q + ϵ is the key value of a similar token, where ϵ is a small random variable, k ∈ R d is the key vector of a random token, R m,θ is the rotation matrix in RoPE. The first term in Eq. 8 is the attention score of q and a similar token k ∗ , the second term in Eq. 8 is the attention score of q and random token k . Then we derive the following theorem:

Theorem 1 Assuming that the components of query q ∈ R d and key k ∈ R d are independent and identically distributed, their standard deviations are denoted as σ ∈ R . The key k ∗ = q + ϵ is a token similar to the query, where ϵ is a random variable with a mean of 0. Then we have:

The proof is shown in Appendix A. We denote ∑ d/ 2 -1 i =0 cos( mθ i ) as B m,θ , and according to Theorem 1, B m,θ measures the ability to give more attention to similar tokens than random tokens, which decreases as the relative distance m increases, as shown in Figure 5. For a very small base value, we can observe that the B m,θ is even below zero at a certain distance, meaning the random tokens have larger attention scores than the similar tokens, which may be problematic for long context modeling.

¬

¬

¬

¬

¬

5

5

5

glyph[arrowdblbothv]

∅

⊆

B

¬

¬

¬

¬

4

6

¬

¬

¬

¬

4

6

4.3 Base of RoPE Bounds the Context Length

To satisfy the Desiderata 2, we will get E q,k ∗ [ q T R m,θ k ∗ ] ≥ E q,k [ q T R m,θ k ] . According to Theorem 1, B m,θ needs to be larger than zero. Given the θ in RoPE, the context length L θ that can be truly obtained satisfies:

In other word, if we follow the setting that θ i = base -2 i/d , in order to get the expected context length L , there is a lower bound of the base value base L :

In summary, the RoPE's base determines the upper bound of context length the model can truly obtain. Although there exists the absolute lower bound, Eq. 9 and Eq. 11 are hard to get the closedform solution since B m,θ is a summation of many cosine functions. Therefore, in this paper, we get the numerical solution. Table 1 shows this lower bound for context length ranging from 1,000 to one million. In Figure 1, we plot the context length and corresponding lower bound, we can observe that as the context length increases, the required base also increases.

Note: this boundary is not very strict because the stacking of layers in LLMs allows the model to extract information beyond the single layers' range, which may increase the context length in Eq. 10 and decrease the base in Eq. 11 . Notwithstanding, in Section 5 we find that the derived bound approximates the real context length in practice.

Long-term decay from different perspectives. The long-term decay in section 4.1 and section 4.2 are from different perspectives. The former refers to the long-term decay of the attention score as the relative distance increases. This ensures that current tokens tend to pay more attention to the tokens closer to them. The latter indicates that with the introduction of the rotation matrix in attention, the ability to discriminate the relevant tokens from irrelevant tokens decreases as the relative distance increases. Therefore, a large B m,θ , corresponding to a large base value, is important to keep the model's discrimination ability in long context modeling.

5 Experiment

In this section, we conduct thorough experiments. The empirical result can be summarized in Table 2, the details are in the following sections.

5.1 Experiments Setup

For fine-tuning, we utilized Llama2-7B [7] and Baichuan2-7B [8], both of which are popular opensource models employing RoPE with a base of 1 e 4 . We utilized a fixed learning rate of 2e-5 and a

5000

10000

15000

20000

25000

30000

Context

6

8

10

12

14

16

18

20

Perplexity

32K-base:1e4

32K-base:2e5

32K-base:9e5

32K-base:5e6

32K-base:1e9

32K-base:1e12

4K-Baseline

(a) Perplexity

1e4

2e59e5 5e6

1e9

1e12

Base value

0

.

0

0

.

1

0

.

2

0

.

3

Accuracy on 32K

Lower bound

(b) Long-eval 32k

global batch size of 128 and fine-tuning for 1000 steps. For pre-training, we trained a Llama-like 2B model from scratch for a total of 1 trillion tokens. We set the learning rate to 1e-4 and adopted a cosine decay schedule, with models trained on a total of 1T tokens. The dataset we used is a subset of RedPajama [24]. More details of the experimental setup are provided in Appendix B.

Our evaluation focused on two aspects: (1) Perplexity : we use PG19 dataset [25] which are often used in long context evaluation; (2) Retrieval : in addition to perplexity, we also adopt retrieval since it represents the real long-context understanding ability of LLMs. We choose a) Long-eval benchmark from [20] and b) Needle in a haystack (NIH) [21]. The Long-eval benchmark generates numerous random similar sentences and asks the model to answer questions based on a specific sentence within the context, while the NIH requires the model to retrieve information from various positions in the long context.

5.2 Base of RoPE bounds context length in fine-tuning stages

According to Eq. 11, there is a lower bound of RoPE's base determined by expected context length. We fine-tune Llama2-7b-Base on 32k context with varying bases. As depicted in Figure 6, although the difference in perplexity between different bases is negligible, the accuracy of Long-eval varies significantly. In Figure 6b, the dotted line denotes the lower bound derived from Eq. 11, below which the Long-eval accuracy declines significantly. Additional results are provided in Appendix C. Notably, this empirically observed lower bound closely aligns with our theoretical derivation. On the other hand, we can see that base = 2 e 5 achieves the best perplexity, but the accuracy of Longeval is very low, which indicates the limitations of perplexity in evaluating long context capabilities. We also provide the more comprehensive RULER [26]benchmark results in Appendix G.

5.3 The Base of RoPE bounds context length in pre-training stages

According to Theorem 1 and Eq. 11 , these constraints could also apply to the pre-training stage. To validate this, we trained a 2B model from scratch with RoPE base=100. The results, depicted in the first row of Figure 7, indicate that even though the model was trained with a context length of 4,096 tokens, it was capable of retrieving information from only the most recent approximately 500 tokens. This demonstrates that the base parameter bounds the context length during the pre-training stage as well. We define the context length from which the model can effectively retrieve information as the effective context length.

According to our theory, the effective context length can be extended as the RoPE's base increases. To validate this, we further fine-tune this 2B model on 32k context length, with RoPE's base set to 1e4, as shown in the second row of Figure 7. While the effective context length increased, it remains significantly below 32k since the effective context length bounded by base=1e4 is much smaller

¬

¬

¬

¬

¬

¬

¬

0

1000

2000

3000

4000

Context

10

15

20

25

30

Perplexity

1000

2000

3000

4000

5000

Context length

0

0

0

1

0

2

0

0

0

0

102

204

306

408

510

612

714

816

918

1020

1122

1224

1327

1429

1531

1633

1735

1837

1939

2041

2143

2245

2347

2449

2551

2653

2755

2857

2959

3061

3163

3265

3367

3469

3571

3673

3776

3878

3980

4082

4184

4286

4388

4490

4592

4694

4796

4898

5000

Token Limit

0.0

5.0

11.0

16.0

21.0

26.0

32.0

37.0

42.0

47.0

53.0

58.0

63.0

68.0

74.0

79.0

84.0

89.0

95.0

100.0

Context length

0

2

4

6

8

10

0

1000

2000

3000

4000

Context

10

15

20

25

30

Perplexity

0

1000

2000

3000

4000

Context

10

15

20

25

30

Perplexity

0

102

204

306

408

510

612

714

816

918

1020

1122

1224

1327

1429

1531

1633

1735

1837

1939

2041

2143

2245

2347

2449

2551

2653

2755

2857

2959

3061

3163

3265

3367

3469

3571

3673

3776

3878

3980

4082

4184

4286

4388

4490

4592

4694

4796

4898

5000

Token Limit

0.0

5.0

11.0

16.0

21.0

26.0

32.0

37.0

42.0

47.0

53.0

58.0

63.0

68.0

74.0

79.0

84.0

89.0

95.0

100.0

Context length

0

2

4

6

8

10

0

1000

2000

3000

4000

Context

10

15

20

25

30

Perplexity

1000

2000

3000

4000

5000

Context length

0

0

0

1

0

2

0

0

0

0

102

204

306

408

510

612

714

816

918

1020

1122

1224

1327

1429

1531

1633

1735

1837

1939

2041

2143

2245

2347

2449

2551

2653

2755

2857

2959

3061

3163

3265

3367

3469

3571

3673

3776

3878

3980

4082

4184

4286

4388

4490

4592

4694

4796

4898

5000

Token Limit

0.0

5.0

11.0

16.0

21.0

26.0

32.0

37.0

42.0

47.0

53.0

58.0

63.0

68.0

74.0

79.0

84.0

89.0

95.0

100.0

Context length

0

2

4

6

8

10

than 32k. Further, when we increase the base to 1e6 and fine-tune the base 2B model on 32K (the third row in Figure 7), the model could obtain a larger context length than base=1e4, which is in accordance with our theory.

To further remove the influence of model size, we also fine-tuned a larger 7B model on a 32k context length with a RoPE base set to 1e4 and observed an effective context length nearly identical to that of the 2B model with the same RoPE base (see Appendix D). This is empirical proof that the effective context length is determined by RoPE's base.

5.4 Interpretation for the superficial long context capability for small base

Based on our theory and empirical observations, it is easy to explain what happens in Figure 3.

Better Extrapolation (Perplexity)? Due to the small base, B m,θ can be smaller than zero as m increases, which is shown in Figure 5. The model can't attend more to similar tokens than random tokens with a large relative distance, so the model tends to focus more on nearby tokens, this will lead to a smaller empirical receptive field, even smaller than the training length. In this case, the model has a strong ability to maintain perplexity stability [27].

Worse Ability (Long-eval and NIH)! According to our previous analysis, RoPE's base bounds the context length, and the context length bounded by 500 is much lower than that bound by 10,000. Therefore, when the base is set to 500, the effective context length drops sharply, even after training on 32k context length.

5.5 OODtheory is insufficient to reveal long context capability

Section 3 mentions that methods based on the OOD theory of rotation angles may not fully reflect the long context capability. In this section, we conduct further experiments to substantiate and explain this observation. We present two methods to extend the context length of Llama2 from 4k to 32k. Both of them are devoid of OOD angles. These methods are delineated mathematically as follows:

Method 1: θ i = (5 e 6) -2 i/d ,

¬

¬

¬

¬

¬

3

4

5

3

4

5

Score

Score

Score

We can see from Table 3 that these two methods exhibit significantly different long context capabilities. Under the perspective of OOD rotation angle, both methods avoid OOD rotation angle, suggesting effective extrapolation. However, despite being trained on a context length of 32k, "method 2" struggles in completing the retrieval task at a context length of 32k. This phenomenon is beyond the scope which the OOD theory can explain.

Under our perspective, "method 2" is severely violating B m,θ ≥ 0 when m ∈ [15 k, 30 k ] , thereby impeding its ability to achieve long-context discrimination. We speculate that the model may achieve better extrapolation in the fine-tuning stage if the base is sufficiently large to surpass a lower bound and avoid OOD of rotation angles.

6 Related Work

Position embedding. Since its introduction, Transformer [4] has achieved remarkable results in the field of natural language processing. To make full use of the order of sequence, researchers have introduced position embedding. The earliest position embedding was based on sinusoidal functions [4] for absolute positions, learnable absolute position embedding [28] and many variants [29, 30] were proposed. Nevertheless, absolute position embedding has difficulties in extending directly to texts longer than the training length. Subsequently, researchers proposed relative position embedding methods [31, 32]. With the development of large language models, rotary position embedding and its variants [6, 23] has become widely used, such as Llama2 [7], Baichuan2 [8], Mistral-7B-[33]. A recent study reveals that no position embedding is also potential [34].

Long context learning. Implementing models with longer or even infinitely long contexts has always been an important goal in the field of natural language processing. Due to the squared complexity of the transformer model over time, a significant portion of the work focuses on improving the model structure [35, 35, 36, 37]. However, most of the work is still based on the transformer architecture. The other part of the work is aimed at reducing the computational complexity of attention itself, such as sparse attention [38] and group query attention [39]. In addition, there are also some optimizations in engineering efficiency, such as flash attention [40] and ring attention [41]. In the model inference stage, to save time and space, there are also some methods for accelerating long context, such as KV cache compression [42], etc. And the position embedding is important in extrapolation. In the process of fine-tuning, methods such as PI [12], NTK, and YARN [11] are used to change the original position embedding information. FoT [43] assigns the position information of the tokens outside the local context as the first token in the local context.

7 Limitation

In this work, we investigate the relationship between the base of RoPE and context length. Although we have derived that there exists a lower bound for the base of RoPE determined by context length, the existence of the upper bound for RoPE's base remains an open question that warrants further exploration. In addition, because of the lack of effective benchmarks for assessing long-context capabilities, the scope of long-context capabilities discussed in this paper may be limited.

8 Conclusion

Our work presents a comprehensive study on the role of RoPE in LLMs for effectively modeling long context. Our main contribution lies in uncovering a novel property of RoPE through theoretical analysis, demonstrating that as the relative distance between tokens increases, the model's ability to attend more to similar tokens decreases. According to our theory, we derive a lower bound for RoPE's base in accommodating to expected context lengths. Our experimental results validate that the base of RoPE bounds context length for not only fine-tuning but also the pre-training stage. Our theory offers a new perspective on understanding the functionality of RoPE in long-context modeling. By shedding light on the relationship between context length and position embedding, we hope our work could provide insights for enhancing the long context capability of LLMs.

References

Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and Weipeng Chen. Base of rope bounds context length. arXiv preprint arXiv:2405.14591 , 2024.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. Impact of code language models on automated program repair. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) , pages 1430-1442. IEEE, 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.10551 , 2022.

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 568:127063, 2024.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.

Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305 , 2023.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 , 2023.

Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. arXiv preprint arXiv:2403.13298 , 2024.

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations , 2023.

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595 , 2023.

Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08268 , 2024.

Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652 , 2024.

Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of roPE-based extrapolation. In The Twelfth International Conference on Learning Representations , 2024.

Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lminfinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137 , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.

bloc97. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation. https: //www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_ allows_llama_models_to_have/ , 2023.

emozilla. Dynamically scaled rope further increases performance of long context llama with zero fine-tuning. https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/ dynamically_scaled_rope_further_increases/ , 2023.

Dacheng Li*, Rulin Shao*, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can open-source llms truly promise on context length?, June 2023.

Kamradt G. Needle in a haystack - pressure testing llms. https://github.com/gkamradt/ LLMTest_NeedleInAHaystack , 2023.

Yutong Hu, Quzhe Huang, Mingxu Tao, Chen Zhang, and Yansong Feng. Can perplexity reflect large language model's ability in long text understanding? In The Second Tiny Papers Track at ICLR 2024 , 2024.

Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatable transformer. arXiv preprint arXiv:2212.10554 , 2022.

Together Computer. Redpajama: An open source recipe to reproduce llama training dataset, April 2023.

Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations , 2019.

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654 , 2024.

Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, and Peter Ramadge. Dissecting transformer length extrapolation via the lens of receptive field analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 13522-13537, 2023.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, and Kentaro Inui. Shape: Shifted absolute position embedding for transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3309-3321, 2021.

Hailiang Li, YC Adele, Yang Liu, Du Tang, Zhibin Lei, and Wenye Li. An augmented transformer architecture for natural language generation tasks. In 2019 International Conference on Data Mining Workshops (ICDMW) , pages 1-7. IEEE, 2019.

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) , pages 464-468, 2018.

Guolin Ke, Di He, and Tie-Yan Liu. Rethinking positional encoding in language pre-training. In International Conference on Learning Representations , 2020.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.

Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems , 36, 2024.

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 , 2023.

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, et al. Rwkv: Reinventing rnns for the transformer era. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 14048-14077, 2023.

Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling. Advances in Neural Information Processing Systems , 36, 2024.

Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 , 2020.

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In The 2023 Conference on Empirical Methods in Natural Language Processing , 2023.

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems , 35:16344-16359, 2022.

Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. In NeurIPS 2023 Foundation Models for Decision Making Workshop , 2023.

Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quantization. arXiv preprint arXiv:2401.18079 , 2024.

Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miło´ s. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Processing Systems , 36, 2024.

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.

Amirkeivan Mohtashami and Martin Jaggi. Random-access infinite context length for transformers. Advances in Neural Information Processing Systems , 36, 2024.

A The proof of Theorem 1

Assuming that the components of query q ∈ R d and key k ∈ R d are independent, their standard deviations are denoted as σ ∈ R d and the means are donated as µ ∈ R d . The key k ∗ similar to q is q + ϵ , where ϵ is a random variable with a mean of 0. Then, we have:

Then we can get:

And when all σ are equal, we can get:

B The detail setting of experiment

For training, we mainly conducted experiments on Llama2-7B [7] and Baichuan2-7B [8]. In addition, we also trained a 2B model from scratch, whose structure is the same as Baichuan2-7B-Base but with a smaller hidden size = 2048. Both training and testing are accelerated by FlashAttention-2 [40] and Megatron-LM [44]. The dataset of both fine-tuning and training from scratch is a subset of RedPajama [24]. The hyperparameters of training are listed in Appendix 4. All experiments are conducted on a cluster of 16 machines with 128 NVIDIA A100 80G.

Question : Below is a record of lines I want you to remember. Each line begins with 'line <line index>' and contains a '<REGISTER_CONTENT>' at the end of the line as a numerical value. For each line index, memorize its corresponding <REGISTER_CONTENT>. At the end of the record, I will ask you to retrieve the corresponding <REGISTER_CONTENT> of a certain line index. Now the record start:

…

line swift-baby: REGISTER_CONTENT is <12821> line dangerous-breast: REGISTER_CONTENT is <28051> line bad-sculptural: REGISTER_CONTENT is <32916> line flashy-college: REGISTER_CONTENT is <34027> line voiceless-brochure: REGISTER_CONTENT is <8964> line fast-peony: REGISTER_CONTENT is <5218>

…

Now the record is over. Tell me what is the <REGISTER_CONTENT> in line dangerous-breast? I need the number. Answer :

For evaluation, we test the long context capabilities comprehensively, the benchmarks are listed below: perplexity on PG19 [25] test split. We evaluate the perplexity of each sample and get the mean value across samples.

Long-eval [20]. This test generates massive random similar sentences and asks the model to answer questions according to a specific sentence in the context. Because the long context consists of many similar patterns, it's more difficult to get the right answer. We find this test is harder than other long context evaluations such as Perplexity, Passkey Retrieval [45], Needle in Haystack [21]. A test sample is list in Figure 8

needle in haystack(NIH) [21]. NIH tests the long context capability not only under different context lengths but also at different positions where the correct answer is located in the context, which provides a more detailed view of the long context capability.

C Baichuan2-7B-Base: Lower bound Base of RoPE

5000

10000

15000

20000

25000

30000

Context

6

8

10

12

14

16

18

20

Perplexity

32K-base:1e4

32K-base:2e5

32K-base:9e5

32K-base:5e6

32K-base:1e9

32K-base:1e12

4K-Baseline

(a) Perplexity

1e4

2e59e5 5e6

1e9

1e12

Base value

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Accuracy on 32K

Lower bound

(b) Long-eval 32k

D Long Context Test Results on Various LLMs

Score

0

10000

20000

30000

Context size

6

8

10

12

14

Perplexity

0

2500

5000

7500

10000

12500

Context length

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

11000

12000

13000

14000

15000

16000

17000

18000

19000

20000

21000

22000

23000

24000

25000

26000

27000

28000

29000

30000

31000

32000

Token Limit

0.0

5.0

11.0

16.0

21.0

26.0

32.0

37.0

42.0

47.0

53.0

58.0

63.0

68.0

74.0

79.0

84.0

89.0

95.0

100.0

Context length

0

2

4

6

8

10

0

10000

20000

30000

Context size

6

8

10

12

14

Perplexity

0

2500

5000

7500

10000

12500

Context length

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

11000

12000

13000

14000

15000

16000

17000

18000

19000

20000

21000

22000

23000

24000

25000

26000

27000

28000

29000

30000

31000

32000

Token Limit

0.0

4.0

8.0

12.0

16.0

20.0

24.0

29.0

33.0

37.0

41.0

45.0

49.0

53.0

57.0

61.0

65.0

69.0

73.0

78.0

82.0

86.0

90.0

94.0

98.0

Context length

0

2

4

6

8

10

Score

0

10000

20000

30000

Context

8

10

12

14

Perplexity

0

2500

5000

7500

10000

12500

Context length

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

11000

12000

13000

14000

15000

16000

17000

18000

19000

20000

21000

22000

23000

24000

25000

26000

27000

28000

29000

30000

31000

32000

Token Limit

0.0

4.0

8.0

12.0

16.0

20.0

24.0

29.0

33.0

37.0

41.0

45.0

49.0

53.0

57.0

61.0

65.0

69.0

73.0

78.0

82.0

86.0

90.0

94.0

98.0

Context length

0

2

4

6

8

10

Score

0

10000

20000

30000

Context size

6

8

10

12

14

Perplexity

0

2500

5000

7500

10000

12500

Context length

0

.

2

0

.

4

0

.

6

0

.

8

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

11000

12000

13000

14000

15000

16000

17000

18000

19000

20000

21000

22000

23000

24000

25000

26000

27000

28000

29000

30000

31000

32000

Token Limit

0.0

5.0

11.0

16.0

21.0

26.0

32.0

37.0

42.0

47.0

53.0

58.0

63.0

68.0

74.0

79.0

84.0

89.0

95.0

100.0

Context length

0

2

4

6

8

10

Score

0

10000

20000

30000

Context

8

10

12

14

Perplexity

0

2000

4000

6000

Context length

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

11000

12000

13000

14000

15000

16000

17000

18000

19000

20000

21000

22000

23000

24000

25000

26000

27000

28000

29000

30000

31000

32000

Token Limit

0.0

5.0

11.0

16.0

21.0

26.0

32.0

37.0

42.0

47.0

53.0

58.0

63.0

68.0

74.0

79.0

84.0

89.0

95.0

100.0

Context length

0

2

4

6

8

10

Score

E The python code for calculating the low bound base for a context length of 32k

F A empirical verification of Desiderata 2

The Desiderata 2 introduced in Section 4 is intuitively plausible, but its empirical validity requires verification. To investigate this, we conducted a detailed empirical analysis. The similarity between tokens is measured by the cosine similarity (denoted as A) of their corresponding hidden states, while the attention allocation between tokens is governed by the attention score (denoted as B). The desiderata "similar tokens receive more attention" implies that a higher value of A should lead to a higher value of B .

To test this desiderata, we performed experiments using Llama1-7B, Llama2-7B, and Llama3-8B models. We selected 200 segments from the PG19 dataset, each containing 1024 tokens, and computed Spearmans rank correlation coefficient between (A) and (B). A positive correlation coefficient would indicate that as token similarity (A) increases, the corresponding attention score (B) also increases. The magnitude of the coefficient reflects the strength of this correlation.

The results, presented in Figure 15 confirm that Spearmans rank correlation coefficient is positive, validating the desiderata that "similar tokens receive more attention". Furthermore, we observe that this positive correlation is more pronounced in the Llama3-8B model compared to Llama27B and Llama1-7B, suggesting that larger and more advanced models are better at capturing this relationship.

G Evaluation results on RULER

0

5

10

15

20

25

30

Layer

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

correlation coefficient

llama1-7b

llama2-7b

llama3-8b

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claims the paper's contributions and scope in the abstract and the last part of the introduction in Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the work in Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer:[Yes]

Justification: We provide the assumptions of Theorem 1 in itself and provide the proof in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the experimental details and hype-parameters in Section 5.1 and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Due to some privacy policies and the complexity of training large language models, we are unable to provide the data and code. But we believe that based on the open-source LLMs and open-source code, our results can be reproduced.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the experimental details and hype-parameters in Section 5.1 and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Due to the high computational cost of large language models, it is difficult for us to perform the same experiment multiple times to get the error bar. However, we provide the results of different models and their performance under various evaluation metrics to support our perspective.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the information in Section B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the NeurIPS Code of Ethics and the research conducted in our paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our paper focuses on the impact of key parameter settings in the model on its capability. To our knowledge, this does not involve any social impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: In this work, we don't release any data or model.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer:[Yes]

Justification: We cite existing papers and url.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the role of RoPE in long-context LLMs. It highlights that the base of RoPE crucially affects the model's ability to handle long contexts. This paper derives a theoretical (and empirical) lower bound for the base value required to maintain long-context capabilities and validates this through empirical experiments with models like Llama2-7B and Baichuan2-7B, and a 2B model trained from scratch. This work offers insights into the base of RoPE for long-context processing in LLMs, which is inspiring for the development and design of long-context LLMs.\n\nSTRENGTHS:\n1. The lack of long-context ability is still an under-explored question and a very important one. Thus, the research topic of this paper is both theoretically important and practical.\n2. The relationship between the RoPE base and long-context ability is important, and the proposed lower bound is interesting and inspiring.\n3. The experiments in this paper are comprehensive and well support the claims made in the paper.\n4. The presentation of this paper is overall very good and easy to understand and follow.\n5. The claims of this paper are inspiring for the development of long-context LLM.\n6. I am also very interested in RoPE-based selection in LLM and like this paper.\n\nWEAKNESSES:\n1. The Desiderata 2. The similar token gets more attention in Section 4 seems intuitively correct but may not be empirically correct. As this desideratum is the fundamental motivation of Theorem 1, a thorough empirical verification is a must-have.\n2. The motivation section is not well-written or organized. It confuses me while I read this part. These are just some previous empirical observations and are not deeply discussed. As the motivation part is very important, I would like to see a clear and well-organized motivation.\n3. For the Caption of Figure 6, it would be better to show how to derive the value of the dotted line. I would also like to see a detailed derivation here.\n4. Will the author plan to release the models (including the fine-tuned Llama2, Baichun2 with varying lengths, and the 2b model series)? This would benefit future work.\n5. I wonder about the negativeness and positiveness of the $q^{T}R_{m,\\theta}k$ in equations (8) and (9). If the values are negative, say -2 and -1, which one indicates more attention?\n6. For section 5.3, I would like to regard this as a conjecture rather than an explanation.\n7. Moinries\n    - The upper case and lower case of the title are not consistent, such as the title of Section 4.1 and Section 2.2\n    - Line 225 “method2” -> “Method 2”\n\nQUESTIONS:\nSee **Weaknesses**\n\nLIMITATIONS:\nSee **Weaknesses**", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nROPE is wildily employed in popular LLMs which encodes positional information with a rotation matrix. Although RoPE is used to enhance long-context capabilities by adjusting its base parameter to address OOD issues, this paper finds that this may result in only superficial long-context abilities. Authors re-evaluate RoPE's role and introduce a novel property of long-term decay, showing that the base of RoPE limits context length, with an absolute lower bound required for certain capabilities. This work clarifies the theoretical and empirical relationship between context length and the RoPE base, offering insights for future long-context training.\n\nSTRENGTHS:\n- The two provided desideratas are highly logical and well-aligned with language modeling. The assumption made in this paper is quite reasonable and closely aligned with practical scenarios.\n\n- Insightful analysis. The theoretical results presented in this paper are easy to understand. To the best of my knowledge, the final bound for the ROPE base is novel and first introduced in this paper\n\nWEAKNESSES:\nThere is no obvious weaknesses in my opinion. I just have a few questions:\n\n- Regarding the \"Desiderata 2 The similar token gets more attention\", recently StreamLLM [1] shows that there exists \"attention sink\" in popular LLMs. Namely most of tokens attend to the first few tokens. This somehow contradicts with the principle that \"the similar token gets more attention\". Could you provide your thoughts on this statement?\n\n- Anthropic's blogs reveal that different heads may have different functionality in in-context learning. How may this interplay with the Rope base? Do you think different heads may have different optimal rope base?\n\n[1] Efficient Streaming Language Models with Attention Sinks\n[2] In-context Learning and Induction Heads\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the role of Rotary Position Embedding (RoPE) in Large Language Models (LLMs), with a focus on the relationship between RoPE's base and the model's long context ability. \nThe study looks into the long-context abilities and limitations of current methods that rely on smaller RoPE bases. With a lower base, LLM models may exhibit superficial long-context abilities, achieving low perplexity but failing to retrieve relevant information in extended contexts.\nTheoretically, The paper establishes two desiderata for the attention mechanism in language modeling: 1.Closer tokens receive more attention. 2.Similar tokens receive more attention. It examines these when applying RoPE to LLMs, revealing a long-term decay in attention score and the ability to differentiate similar from random tokens. This leads to a theorem indicating that RoPE’s base sets an absolute lower bound for achieving specific context lengths.\n\nSTRENGTHS:\n1. Clarity and Technical Correctness: The paper is clear and technically sound, with theoretical and empirical analyses.\n\n2. Experimental Rigor and Reproducibility: It includes extensive experiments to back up the theoretical findings, along with detailed setup and results.\n\n3. Novel Findings: This paper presents a critical study of whether we should use smaller bases for continuous training, as suggested by previous work. Furthermore, this paper presents a novel perspective on long-term decay, as well as an absolute lower bound on the RoPE base parameter required for specific context lengths. This adds new knowledge to the field and improves our understanding of position embedding in LLMs.\n\nWEAKNESSES:\nOverall, I like this paper. I would like to suggest that the authors strengthen their work by considering the following points:\n\n1. Extensively test the model using benchmarks such as RULER [1].\n\n2. Provide more empirical observations on the relationship between the base of RoPE and model performance on those challenging benchmarks.\n\n\n--\n\n[1] https://github.com/hsiehjackson/RULER\n\nQUESTIONS:\nFor the suggestions, please refer to the weaknesses section.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
Hi Area Chair, I am not qualified for the review of this paper, this is out of my knowledge scope.

STRENGTHS:
n/a

WEAKNESSES:
n/a

QUESTIONS:
n/a

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15