Scaling Laws of RoPE-based Extrapolation
Authors: Xiaoran Liu
Hang Yan
Chenxin An
Xipeng Qiu
Dahua Lin
Venue: ICLR 2024 poster (2024)
Keywords: Position Embeddin
Length Extrapolation
Large Language Model
Natural Language Processing
Primary area: general machine learning (i.e., none of the above)
TL;DR: In this work, we propose a unified framework from the prospective of period, to explain the mechanism of RoPE-based extrapolation by whether increasing or decreasing the rotary base.

Abstract: The extrapolation capability of Large Language Models (LLMs) based on Rotary Position Embedding \citep{su2021roformer} is currently a topic of considerable interest. The mainstream approach to addressing extrapolation with LLMs involves modifying RoPE by replacing 10000, the rotary base of $\theta_n={10000}^{-2n/d}$ in the original RoPE, with a larger value and providing longer fine-tuning text. In this work, we first observe that fine-tuning a RoPE-based LLM with either a smaller or larger base in pre-training context length could significantly enhance its extrapolation performance. After that, we propose \textbf{\textit{Scaling Laws of RoPE-based Extrapolation}}, a unified framework from the periodic perspective, to describe the relationship between the extrapolation performance and base value as well as tuning context length. In this process, we also explain the origin of the RoPE-based extrapolation issue by \textbf{\textit{critical dimension for extrapolation}}. Besides these observations and analyses, we achieve extrapolation up to 1 million context length within only 16K training length on LLaMA2 7B and 13B \citep{touvron2023llama2}.

Published as a conference paper at ICLR 2024
SCALING LAWS OF ROPE-BASED EXTRAPOLATION
Xiaoran Liu Hang Yan' Chenxin An², Xipeng Qiu Dahua Lin'
'Shnnghai AI Lab
Schooo of Computer Science, Fudan University
1iuxr22 @m. fudan. edu. en, fyanhang, Lindahua)@pilab. org en, {cxan20, xpqiuje (ogiu)efucee
ABSTRACT
The extrapolation capability of Large Language Models (L.LMs) based on Rotary Position Embedding (Su et al. 2021) currently a topic of considerable inter- est. The mainstream approach to addressing extrapolation with LLMs involves modifying RoPE by replacing 10000, the rotary base of 8, 10000 2n// in the original RoPE, with a larger value and providing longer fine-tuning text. In this work, first observe that fine-tuning RoPE-basee LLM with either smaller or larger base in pre-training context length could significantly enhance its ex- trapolation performance. After that, we propose Scaling Laws of RoPE-based Extrapolation, a unified framework from the periodic perspective, to describe the relationship between the extrapolation performance and base value as well tun- ing context length. In this process, we also explain the origin of the KoPE-based extrapolation issue by critical dimension for extrapolation. Besides these ob- servations and analyses, we achieve extrapolation up to I million context length within only 16K training length on LLaMA2 7B and 13B (Touvron et al. 2023b).
INTRODUCTION
Large Language Models (LLMs) have become the dominant architecture in a variety of natural lan- guage processing tasks(OpenA] 2023] Touvron et al. 2023a b). while Transformers (Vaswani et al.] 2017) based on Rotary Position Emhedding (RoPE) (Su et al. 2021) have become the dominant backbone in wide range L.M design (Chowdhery et al. 2022] Nijkamp et al.) 2023) Touvron al. 2023a b). While RoPE can theoretically represent sequences through trigonometric functions, as detailed in Appendix [A] its performance drops when the input sequence or context length sur- This extrapolation
passes the training length((Pess (Peess et al. 2022 Chen al. 2023). seen in Figure T
problem [Pesss et al. 2022 limits tasks like long text modeling and summarization An et al. 2023
PLAN
Compal Lempl
Courd
(a) Results on LLaMA2 7B
(b) Results on LLaMA2 13B
Figure I: Perplexity on Books3 Presser 2020) with different extrapolation methods, including Dynamic NTK (locc9 2023b). RoPE fine-tuned with a smaller or larger base on the original training length of 4K or much longer context of 16K, could outperform other extrapolation strategics and extrapolate to 100K context length.
Equal contribution.
Corresponding author: xpqiu Rogiu@fuudn.comm fww.a..du.aacm

---

Published as a conference paper at ICLR 2024
Concerning the extrapolation issue with RoPE, different works have provided various interpretations and corresponding solving attempts. These works could divided into two schools of thought. One
limits the scope of self-attention (Ratner et al. 2022 Han et al.
(023) given the fact that RoPE-based self-attention fails to keep stable beyond training context and exhibits attention score explosion as
well as monotonous entropy increase (Chen et al.
2023 Han et al.
2023). The other aims to capture
longer contexts with smaller rotation angles and longer fine-tuning context (Chen et al.) 2023 Peng al., 2023). Current popular methods, such as Dynamic NTK (bloc97 (023a) and Code LLaMA (Rozière et al., 2023). mainly come from the second approach. Both approaches adapt RoPE to longer contexts with a larger rotary base. Specifically, Dynamic NTK (bloc97 2023a) adjusts the base with a coefficient increasing with the length of inference, allowing RoPE-based LLMs to adapt simultaneously to longer context, while Code LLaMA (Rozière et al., 2023) directly sets the base at 1000000 and gets further trained on 16K context length, yielding context beyond 100K.
While recent studies have shown promising results, they have primarily focused on specific base values and tuning context lengths. This leaves notable gap in understanding how base value, tuning length, and extrapolation performance relate, For instance, while larger bases improve extrapolation in models like LLaMA2 (Touvron et al., 2023a), surprisingly, we also find that fine-tuning with smaller bases with the original training length is also conducive to the extrapolation capability of LLaMA2, which is also demonstrated in Figure [ Furthermore, when trained in longer context, RoPE with a smaller base can match or even surpass those with a larger one. At the same time, fine- tuning with base of 1000000 on the original training length also achieves extrapolation up 100K. These findings pose several questions. QI: Is 10000 the worst base value for extrapolation in the fine-tuning phase? Q2: Is there a mathematical relationship between rotary base, training context length, and extrapolation limit? Q3: If so, can we achieve unbound extrapolation accordingly?
In this paper, we conduct further experiments on increasing and decreasing the rotary base in Section and subsequently discover that adjusting the rotary base in both directions can contribute to the ex- trapolation of RoPF-bbased L.M.M. Building upon these observations, we provide a comprehensive explanation for the seemingly counter-intuitive phenomenon from a periodic perspective. Mean- while, we establish a unified theoretical framework for RoPE-based extrapolation known as the Scal- ing Laws of RoPE-based Extrapolation" We pinpoint specific changes during base reduction that lead to a significant boost in extrapolation in Section 3.1 and identify the upper bound of extrapola- tion for larger bases in Section 3.33 This clarifies how Code LLaMA (Rozière et al. 2023) manages
100K extrapolation with only 16K training context. Furthermore, we validate our theories in Section 3.4 and Appendix C. shedding light on both the core principles of Dynamic NTK (bloc97 2023a)
and pinpointing instability sources self-attention computations in RoPE-based extrapolation (Han al. (003).. Finally. we present the contributions and guiding significance of this work for other methods that achieve extrapolation during the inference phase. In summary, our contributions are as follows, and codes are available httpt:: htt//githhhbb/tte
We first highlight a surprisingly strange phenomenon 10000 is the worst base value for
RoPE-based extrapolation in the fine-tuning phase. Remarkably, fine-tuning with either a larger or smaller base within the training context length greatly enhances extrapolation, which provides new vision to the extrapolation research of RoPE (Su et al. 2021).
Then we introduce unified theoretical framework for RoPE-based extrapolation from a periodic perspective, known as the Scaling Laws of RoPE-based Extrapolation, which not only clarifies the aforementioned observations and addresses unanswered questions in ex- isting research (Rozière et al., 2023), but also discover the Critical Dimension for RoPE- based Extrapolation, revealing the underlying reasons for the extrapolation issue of RoPE.
Finally, for extrapolation within a defined context, we present the suggested fine-tuning base value determined by the context limit and extend the context of LLaMA2 7B and 13B (Touvron et al. 2023b) to surpass 100K tokens by tuning RoPE with base 1 1000000 and 4K tuning length. For unpredictable extrapolation, we propose RoPE with smaller base, such as 500, and achieve an almost IM token context with a mere 16K tuning length.
It important to note that the scaling laws proposed in this work are irrelevant with well-know scaling laws (Kaplan et al. 2020). In this paper, scale refers to the adjustment of RoPE's rotary angles (bloc97 2023a). rather than the change of the model size. Unlike the scaling laws (Kaplan et al. 2020) that are empirically derived, our scaling laws define mathematical relations between context window size and rotary base, supported by experiments.
2

---

Published as a conference paper at ICLR 2024
Calodd Longt
Ouarae Longt
Clocae Longl
Coutect Longl
(a) LLaMA2 7B
(b) LLaMA2 13B
(a) LLaMA2
(b) LLaMA2 13B
Figure 2: Perplexity of larger bases Books3
Figure 3: Perplexity of smaller bases on Books3 Presser 2020) shows better extrapolation.
Presser 2020) shows better extrapolation.
2 OBSERVATION
2.1 LARGER BASES PROMISE BETTER EXTRAPOLATION
We first conduct the extrapolation experiments with larger bases, based on the experimental setup in Appendix B.I It is evident that tuning with larger bases could significantly improve the extrapola- tion performance of RoPE as shown in Figure Besides, there are several noteworthy points.
First, Larger bases allow LLaMA2 Touvron al. 2023b to extrapolate beyond its training context length, aligning with findings from Rozière et al. (2023). Secondly, the extrapolation with larger bases has clear limit where language modeling perplexity stays consistent. Beyond this limit, the extrapolation performance declines significantly. Furthermore, as the base value rises, LJ A.M22 can extrapolate to a longer context. Finally, compared to Dynamic NTK (bloc97 2023a). RoPE tuned with larger bases degrades much quicker beyond its extrapolation upper bound. Therefore, for fine- tuning with larger bases, the performance beyond the upper bound could be consistently overtaken by Dynamic NTK. Nevertheless, within the upper bound, this approach still outperforms Dynamic NTK by a considerable margin, leading to a context beyond 100K with only 4K tuning length, when the base is set over 600000.
2.2 SMALLER BASES ALSO PROMISE BETTER EXTRAPOLATION
We then conduct the extrapolation experiments with smaller bases, using the same setup as for larger bases. Interestingly, even though this goes against common research findings (bloc97 2023a Rozière et al. 2023), fine-tuning RoPE with smaller bases on the original context length still boosts extrapolation, shown in Figure 3 It also extends the context window beyond the training length. Yet, there are distinct differences when comparing RoPE with smaller bases to larger ones.
Firstly, RoPE with smaller bases does not have a distinct upper bound of extrapolation. While perplexity worsens as context length grows, this decline gentler with smaller bases. Secondly, RoPE with smaller bases is not uniform. Between a base of the enhancement in extrapolation performance exhibits tiny improvement. Then between a base 10000 and 8000, the extrapolation of 8000 and 2608, improvement is moderate. After that, from 2608 to 1304 and further 652, the improvement becomes more pronounced. Finally, when the rotary base is 500, the extrapolation curve becomes sufficiently smooth thus resulting in strong extrapolation over 48K context length and superior performance over Dynamic NTK (bloc97 2023a).
Combining these two results, we observe surprisingly strange phenomenon in RoPE-based extrap- olation as depicted in Figure Specifically, base 10000 yields the worst extrapolation performance when fine-tuned, thus answering QI in the Introduction. As the base either decreases or increases, performance notably improves. Interestingly, the improvements differ between the two directions. For larger bases, although the performance steadily improves, there exists a clear extrapolation upper bound. In contrast, for smaller bases, while the improvement is not uniform, the resulting extrapola- tion curve does not have an obvious breaking point. In the heatmap in Figure 4 there is a clear and continuous boundary for larger bases well as distinct transition phase for smaller bases.
3

---

Published as a conference paper at ICLR 2024
Tratndl Leng
Unatedl Faidk
(a) Results on LLaMA2 7B
(b) Results on LLaMA2 13B
Figure 4: Perplexity and accuracy on the validation data from Books3 dataset Presser 2020) of LLaMA2 7B and 13B (Touvron et al. 2023b). In (a) and (b), the first row shows perplexity while the second row shows accuracy, both measured cumulatively. The first column shows the change of extrapolation performance w.r.t rotary base at different context windows with line plot, while the second column visualizes it with a heat map. In the heat map of perplexity, the value is log-scaled.
EXPLANATION
3.1 SCALING LAW FOR SMALLER BASES
To understand how RoPE with smaller bases achieves impressive extrapolation within a constrained tuning context, it is crucial to explore the impact of reducing the base. As outlined in Appendix A smaller base amplifies the rotary angle 6, = 10000-2n/4 This shortens Tn, the periods of sin (t - s)0n or cos (t s)6, that RoPE uses to represent relative positions. Figure 5 shows the cosine waves for different dimensions to represent position information. In Figure 5a it is evident that for smaller bases like 500, any period of cos (t - s)On, is confined to 4096, i.e., the training length of LLaMA2 (Touvron et al. 2023b). In contrast, larger bases like 10000, extend the periods for several dimensions beyond the training length, as detailed in Section 3.2
Hence, smaller bases lead to a broader range of cos or sin inputs during pre-training or fine-tuning, which ensures every dimension of qe and k, gets well-trained representation. Additionally, as the base decreases, three pivotal points emerge, T n/2, म, and 2n.. Only when the cos or sin inputs in every dimension span from 0 to я r/2 during training does the RoPe-based LLM recognize the negative values of cos and the non-monotonicit nature of sin. Similarly, the LLM becomes aware of the non-monotonicity pf cos and negative sin values once inputs hit т. The RoPE-based LLM fully grasps the entire range of cos and sin only when the inputs surpass 2x, potentially embracing the periodicity of position embedding in every dimension. Then we propose Theorem as follows.
Theorem 1. (Scaling Law of Smaller Bases) For RoPE-based LLMs pre-trained with context length Trais if we adjust the base to 33 < 10000 and conduct fine-tuning still with context length Train the extrapolation performance of RoPE-based LLMs will get improved. If / is reduced to 31, 33, f3 as calculated below, the cos or sin inputs in every dimension will span from oo π 77/2, π, 2T respectively, resulting in more significant improvement.
2Tuaiin
Tirain
Timin 2T
3
B2
53
(1)
Particularly, for LLaMA2 Touvron et al. 2023b), where the context length Tiain is 4096, we have 31 = 2608, 32- = 1304, 83 = 652. It is worth noting that these three bases align with the pivotal points where improvement speeds up, previously discussed during base reduction. Theorem tells the improving paradigm of base reduction. Since it does not set an explicit extrapolation upper bound, RoPE with much smaller bases can potentially realize extrapolation to infinite context.
4

---

Published as a conference paper at ICLR 2024
df on
wha Unomeen Iodocc
io topo
(a) RoPE base=500
(b) RoPE basc=10000
(c) RoPE basc=1000000
Figure 5: The visualized relationship among the period, training Length, and extrapolation. Consider RoPE- based LLM with head dimension size of 32. namely 16 rotary angles 0,, across various dimensions. Within each illustration, planes. These are contrasted against the 4096 training context length, shown as deep blue box. (a) For RoPE tuned with base 500, all periods of cos (1 - s)O, are limited within the training context. (b) For vanilla RoPE with base 10000, the periods of dimensions past the critical dimension (in red) stretch beyond the training context. (e) For RoPE tuned with base 1000000, although some updated periods colored (in sky blue) might surpass the training context, only dimensions past the critical dimension fail extrapolate.
visually represent the periods of cos (t s)0,, for these dimensions using parallel purple
3.2 CRITICAL DIMENSION FOR EXTRAPOLATION
In comparison much smaller bases like 500, where each period of cos (t s)0, fits within the training context, the default base in RoPE (Su et al. (021). which is 10000, causes periods of certain dimensions to extend beyond the training context, as visualized in Figure Therefore, for RoPE-based LLMs, there exists specific feature dimension, dextrs For dimensions before deatm- the periods of corresponding On remain shorter than Trrain while for those after dextaa the periods stretch beyond Tuain. In other words, essentially, dext is the number of dimensions where cos (L - s)0,, and sin (4 0) can cycle through their values within one period during pre-training or fine-tuning.
Consequently, for dimensions beyond dexu= when RoPF-based I.MMs extrapolate beyond Tuain the absolute position information newly added tokens and the relative positional information in rela- tion to previous tokens become out-of-distribution (OOD). This misalignment means the attention scores related to these dimensions, as illustrated in Equation 2 deviate from their expected dis- tribution, causing noticeable dittr--distributionn in overall attention scores, thus leading to the extrapolation issue. We refer to this key dimension as the Critical Dimension for RoPE-based extrapolation, which formally defined and calculated shown in Lemma 1.
d/2-1
Ar,a = Re
Σ n=U
s(t-5)0
attention scores
deam
4/2-1
Re
(n) [(() 4:
((t-a)0n
+
Σ
qi k
(n) s)--
n== decem
reliable for extrapolation
0OD part for extrapolation
Lemma 1. (Definition of Critical Dimension) For RoPE-based LLMs pre-trained with context length Tuain assuming that the size of self-attention head is d, there are at most the preceding desm dimensions that perceive complete periodic information thus receiving sufficient training for extrap- olation, which is formally described as follows:
2T
T,
= 2 10000 28 < Tunin- for = =0, -.... dextra/ /2 1,
0n
(3)
2T
T, 0
2n 10000 28 Tu for deama 2.... d/2 1.
5

---

Published as a conference paper at ICLR 2024
242006
caust fhreet 5200
coest Pheeers 24200248000
(a) Results Smaller Bases
(b) Results Larger Bases
Figure 6: The relation between attention scores in first 92 and last 36 dimensions with the extrapola- tion performance in LLaMA 7B (Touvron et 2023b) fine-tuned with larger or smaller bases. The first row shows how average attention score in the first 92 and last 36 dimensions and the perplexity changes in 100K context length. The second row highlights the changes in the first 16K tokens. The third row visualizes that the period of critical dimension determines the attention score explosion and extrapolation issue. The black lines stand for training length or max context size.
Then we define dexta the critical dimension for RoPE-based extrapolation and calculate it given
d
T Train
dextra 2
log 1000000
2
2x
For LLaMA2 Touvron et al. 2023b), the critical dimension deam is 92. This implies that only the first 92 dimensions of the q: k, vectors of LLaMA2 have seen the complete positional information during the pre- pre-training phase and are adequately trained. In other words, the last 36 dimensions lack sufficient training. contributing
the extrapolation challenges seen in RoPE-based LLMs (Chen et al. 2023 Han et al., 2023). The critical dimension plays a key role in enhancing extrapolation. the critical dimension is presented
Section 3.4 Here, we examine the attention score changes in the initial 92 versus the final 36 dimensions in relation to relative positions when the base reduced. As is shown in Figure 6al the attention scores of RoPE with smaller bases have effectively captured the oscillations from sin and cos within training, mitigating OOD concerns during extrapolation. Moreover, as the base becomes smaller, the perception becomes more comprehensive, resulting in improved extrapolation performance.
further discussion
3.3 SCALING LAW FOR LARGER BASES
Based on the concept of the critical dimension, we can clarify the extrapolation results when fine- tuning RoPE with larger bases the original context length. For LLaMA2 (Touvron al. 2023b), since the periods of the first 92 dimensions fit within the training length, these feature dimensions start with a strong foundation for fine-tuning, adjusting to the new periodic shifts of positional embedding for extended contexts. Therefore, when RoPE is fine-tuned with larger base value like 1000000, even though the tuning length is shorter than the extended periods corresponding to larger bases, these dimensions can still represent positional information correctly as is shown in Figure 5c
However, for the last 36 dimensions, the absence of a full understanding of periodicity leads to over- fitting. Furthermore, when the base is expanded, extending the period. these dimensions still fail to capture the entire positional information within the context length. So these dimensions are reliable only when the value of 0,, s) is previously observed. Therefore, can the updated period

---

Published as a conference paper at ICLR 2024
Samercal Upp. Road
Seadica Road
Kday Ja
Culid Le
(a) Results on LLaMA2 7B
(b) Results on LLaMA2 13B
(c) Validation on LLaMA2 70B
Figure 7: The comparison in 100K context length and 7B/13B model between max practical context length and extrapolation upper bound predicted by Theorem 2. for RoPE tuned with larger bases on pre-training context. We also validate our theory in 32K context length and 70B size and find that the tuning points of the perplexity curve match the theoretical upper bounds of length extrapolation.
of the critical dimension an upper bound for extrapolation in RoPE-based LLM. As result, we obtain Theorem 2., the scaling law for RoPE-based extrapolation with larger bases, which tells the relation between base value and extrapolation upper bound and thus answers Q2 in the Introduction.
Theorem 2. (Scaling Law of Larger Bases) For RoPE-based LLMs pre-trained with context length Train. if we adjust the base to 3 > 10000 and conduct fine-tuning still with context length GoPE-based LLMs will get improved. The extrapolation
Trainn the extrapolation performance upper bound of RoPE-basee LLM with larger bases, Teura, is calculated as follows:
Textra 2n I à = 2T 10g10000
Fizan
(5)
Inversely, if there is an expected extrapolation upper bound Texta, then the smallest capable base Po is calculated follows.
log Timis
Sn 10000
(6)
For extrapolation within a limited context, we can derive the suggested 30 based on the expected context length. Bg in Equation [ is referred as critical hase for extrapolation and discussed in detail in Appendix [ To support the claims of Theorem 2, as illustrated in Figure [bb] we examined the attention scores of the first 92 and final 36 dimensions under various larger bases. Notably, while the attention scores of the first 92 dimensions remain relatively stable regardless of relative positions, the last 36 dimensions show significant variations. Reviewing the perplexity increase curves, it is evident that once the context length surpasses the extrapolation upper bound, the last 36 dimensions encounter unfamiliar positional information, leading to OOD attention scores and sharp rise in perplexity. To further validate Theorem lengths for different bases with the extrapolation upper bound derived from Theorem 2 in Figure 7 Impressively, there remarkable alignment between empirical results and theoretical predictions.
compare the max supported context
3.4 FURTHER VALIDATION FOR EXTRAPOLATION
To further illustrate the causal relationship between critical dimension and extrapolation, we under- take the following three experiments. First, we max index of 4096 for the position embedding of the last 36 dimensions of qu, k2 in LLaMA2 7B (Touvron et al., 2023b). We find that apart from an outlier in one attention head, the attention score variations are considrably reduced, leading to better extrapolation. Second, we visualize the attention score of the first 92 and the last 36 dimen- sions during extrapolation with Dynamic NTK (bloc97 2023a) Figure Sa Consistent with our theory, the attention scores of the last 36 dimensions display enhanced consistency compared with straightforward extrapolation. Lastly, we remove the final 36 dimensions qr, k. in LLaMA2 7B Touvron et al. 2023b) and fine-tuned using the setup in Appendix B. Remarkably, post-trimming fine-tuning substantially surpassed direct fine- tin-ttuning allowing extrapolation beyond 20K tokens, as illustrated in Figure 8b) This offers compelling evidence for the correlation between the critical dimension, attention score shifts, and extrapolation upper bound. It confirms that interpreting and enhancing RoPE-based L.M extrapolation from periodic viewpoint is both valid and effective.
7

---

Published as a conference paper at ICLR 2024
128K
256K
512K
1M
base=500
9.49
12.41
23.99
51.28
base=500 log-scaled
9.13
10.01
12.07
19.07
base=1000000
7.07
76.82
1755.87
4914.81
Table 1: Perplexity Books3 of LLaMA2 7B with base 500 and 1000000 tuned on 16K context.
In summary, we establish a comprehensive framework from periodic perspective and identify the fundamental factors, the critical dimension for RoPE-based extrapolation. This not only unveils the root causes of attention score explosion highlighted in Chen et al. (2023) and Han et al. (2023), but also intuitively demonstrates how adjusting the base during fine-tuning can elevate the extrapolation capabilities of RoPE-basee LLMs. Furthermore, we answer Q3 in the Introduction and introduce the Extended Scaling Law of RoPE-based Extrapolation in Appendix C the combination of Theorem 1., Lemma 1. and Theorem 2., for tuning in a longer context. Additionally, we also present the instructive value of our theory for other works focused
achieving longer context in the testing phase in Appendix [ Finally, as depicted in Figure [ tuning RoPE with a base of 500 or 1000000 can both outperform Linear PI (Chen et 2023) and NTK method (bloc97 2023b a). Besides, we compare the performance of LLaMA2 7B Touvron et al. (2023b) with base 500 and 1000000 in 1M context length in Table [
and find that for extrapolating to unpredictable length, RoPE with base 500, has remarkable advantages, especially when combined with log-scaled attention (Su 2023b).
4 RELATED WORK
Long sequence modeling is important issue in LLM applications. It faces the following three challenges: the quadratic computational complexity, the difficulty of parallelism for sequence length, and the collapse of the performance beyond the training length. First of all, due to the quadratic complexity of the self-attention mechanism in Transformers, long sequence modeling re- quires a large computational overhead. Early efficient Transformers reduced computational over- head through methods such as recalculation like ReFormer Kitaev et al., 2020) and sparse atten- tion such as LongBormeyeellagy et al. (2020) and BigBird Zaheer al. 2020 After that, Lin-
Year KrILaNNS T0
zanoof the Ne oecoannic HIX.
Yeand for haring
W
(a) Results on Pre-trained Model
(b) Results on Fine-tuned Model
Figure& 8: The relation between attention scores in first 92 and last 36 dimensions with the extrapola- tion performance in LLaMA 7B (Touvron et al., 2023b) evaluated or fine-tuned with original bases 10000. The meaning of each is the same that in Figure 6
8

---

Published as a conference paper at ICLR 2024
Formen Wang et al., 2020) linearizes the self-attention complexity through low-rank approximation, and Performer(Choromanne
et al. 2021) expands the softmax calculation through a kernel func- tion. Compared with these efficient attention work based on approximation, FlashAttention eet al., 2022 Dao 2023) reduces HBM reading and writing and turns self-attention into a form that can be iteratively calculated in loop, which greatly improves the utilization of computing resources and has become the standard for long-sequence modeling.
The second challenge is parallelism for sequence length, that Sequence Parallelism (SP) (Li al, 2023c). Since the memory and calculation of a single device are limited, AI-Ss (Li et al. 2023cp first proposes the segmentation and parallelism for the sequence dimension in addition to tensor parallelism for the hidden dimension and pipeline parallelism for model depth. On this basis, Ring Attention et al., 2023) uses blockwise self- eeff-attention to split long sequences into different devices and overlap the communication of key-value blocks. Besides, LightSeq(Li et al., 2023b) fur- ther improves the efficiency of long sequence modeling through load balancing for causal language modelings and a re- --materializationawarrr checkpointing strategy. Although the above sequence parallelism can achieve infinitely long sequence modeling, they ignore the compatibility with the existing efficient self-attention mechanisms such as AtashAAtenntion(Dao to achieve input with almost infinity. On the contrary, Megatron Swroon-SSPKKKe Korthiiiat et al. (2022] only
al., 2022 Dao 2023)
uses sequence parallelism during Dropout and Layernorm operations, thereby reducing activation redundancy. In addition, lective communication gathering weight for attention computation when segmenting the sequence, avoiding communication overhead that originally increases with length.
et al.) (023) uses an alternative all-to-all col-
Recently, there have been a great deal of efforts devoted to solving the collapse of the performance beyond the training length and expanding the context length of RoPE-based LLMs to 16K, 32K, or even 100K (Chen et al.] 2023] Su 2023c; bloc97, 2023b). Considering that LLMs, such as LLaMA2 (Touvron et al.) 2023b). have already acquired sufficient knowledge in the pre-training stage and demonstrated excellence in short-context tasks, the emerging extrapolation improvements have primarily focused on the fine-tuning and the testing phase. Initially, the context window is a well-discussed strategy for any LLM without further training. Effective as it is, it prohibits tokens of their deserved global attention (Han et al., 2023) or destroys the order between token chunks (Rat- ner et al., 2022] Su, 2023a), which is unsupported for long document summarization (Sul 2023a). Contrastively. Chen et al. (2023) extends the context window to 16K by linearly reducing the rotary angles with À = Textm/" T Iainn to align the input position index within the original context size. Be- sides, is worth noting the remarkable effect of the Neural Tangents Kernel (NTK) method (bloc97, 2023b), especially the dynamic version (bloc97 2023a simply decreases the rotary angle expo- nentially with scaling coefficient a as function w.r.t. the inference length, which discussed in detail in Appendix( Dynamic NTK enables LLaMA2(Touvron et | (202bb) to extrapolate directly and keeps good performance within 16K context length. However, Dynamic NTK still presents an obvious degradation in performance around 32K context size as shown in Figure Inspired by NTK methods, Rozière et al. (2023) achieves 100K context window for code by increasing the rotary base to 1000000 and further training with 16K context length. Besides, Pal et al. (2023) also extends the context window to 32K by a new truncation strategy for refining the down-sampling method, which even sets some rotary angles to zero. Additionally, Peng et al. (023) proposes scaling function, fine-tunes LLaMA2 with 64K context, and finally achieves the 128K context length. Effective as they are, these methods still face disastrous collapse after unpredictable extrapolation bound.
CONCLUSION
In summary, we initially highlight intriguing observation: fine-tuning RoPE (Su al. 2021 with either a larger or smaller base using the original pre-training context can boost the length extrapolation of RoPE-based LLMs. We then elucidate this observation through a unified theoretical lens rooted in a periodic view, Scaling Laws of RoPE-based Extrapolation. This framework clarifies the score of the RoPE-basee extrapolation challenge and offers insights into how base modifications in fine-tuning or inference can enhance RoPe-based extrapolation. Finally, we provide the strategy for both extrapolating LLaMA2 (Touvron et al. 20236) a limited context and unpredictable inputs.
PI this work, follow the form of scaling function in QWenI.M (Alibaba 2023

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work is supported by Shanghai Artificial Intelligence Laboratory. Special thanks to Qipeng Guo, Shuo Zhang, and Kai Lv, whose invaluable suggestions have greatly contributed to this work.
REFERENCES
Alibaba. Qwen technical report. Technical report, 2023. URL https::/gianwen---tee oss beijing: aliyancs www. TECNNNNCCCC SECCCTTCCCCCCEEEECEERECCCC BEI_I__EPPPR.. p
Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. CoRR,
abs/2307.11088, 2023. doi: 10448550/ARXIV.230.. 1088. URL https//dol.ccr/11
48550/arXiv.200 11088
Iz Bellagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. CoRR, abs/2004.05150, 2020, URL https: //arxi//avvv///aa
http://aa.w....gab////////ssssss www.oogg/////.oooooo./.../o...st 2004. 05150
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning
about physical commonsense in natural language. In The Thirty- AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intel-
ligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances Artifi-
cial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7432-74433 AAAI Press, 2020, doi: 10.1609/AAALL33405..... v341 05. 6239
URLhttps: /tnnttp:://doi..og//11../ 110.0////./.100///aa...
bloc97. Dynamically scaled rope further http://wwwwdddttddotoo
of long context Ilama with zero fine- LLLMA LLaMA/ LLMMAccomeetttttts
tuning, July 2023a. URL httph:tt////ettt
wwarggg/dynamicooo sca htt.aaahew...aaaa...oo...e.eeeer
bloc97. Nik-aware scaled rope allows Ilama models to have extended (8k+) context
size without any fine-tuning and minimal perplexity degradation, June 2023b. URL
https::/www.w. www.itt.ww..d.oooo////t.t.... wtt.e///e//cccaaaeeeee
scaled rope allows 11ama models Saveeaaeee
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window
of large language models via positional interpolation. CoRR, abs/2306. 15522200..1555,,,. 2023. doi: 10. 48550/ARX1V.2306.1 15595. URL httpp:://dii.org///// 011110.4855000 48550 arxiv. 2306. 15595
David Chiang and Peter Cholak. Overcoming a theoretical limitation of self-attention. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 7654-7664. Association for Computational Linguistics, 2022. doi: 10.
ftttttts::// /tttpss://oi..or/11..BB88
2022. 2022.ac1-lons 527.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane. Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with per- formers. In 9th International Conference Learning Representattinss ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URRLhttps:///ppereeeee id=Ua6zuk OWRH
net / forum?
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick,
10

---

Published as a conference paper at ICLR 2024
Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Er- ica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstenn Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling doi: 1044850/ARXIV2200..22
with pathways. CoRR, abs/2204.02311,, 2022.
URL
220. 22033.1122 02311
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the A12 reasoning challenge. CoRR, abs/ 100.0054555 2018. URL httt:htpp///xiiee httpa/pwww......a/aaaa//aaaaaaaa
05457
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. URL
https: /ttp:://aaxvvvvee org/ wwwp/r/g///.ooooooo/tt.//t...art
Tri Dao. Flashattention-22 Faster attention with better parallelism and work partitioning. CoRR, abs/2307.08691, 2023. doi: 10.45550ARKXI22237006600 URL ltttps://di.ozz////
48 48550/arXiv 2307 08691
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL
nips. cttpspsccppppppppppee files/ http://eeesseeaaaee////////////e 20sh2/ahhhhha//// html
67
Hugging Face. Open Ilm leaderboard 2023. URL http::///uugghggfccccc//////////
Hluginggaceei//gggee 1ateea1mmmmmeeaaaaaa leaderboard
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse for language modeling. CoRR, abs/2101.00027,, 2021. URL
htttss:/arxxv..rgeg wwwpr///////// wtt.a/////.......
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. CoRR, htss23308.16331) 2023. doi: 10. 48550/ARRXIV2308..666 16137. BtLhttts:/// /tt::///ii.../ org. 48540/ ar..00/axxv..... 2308 22081..1337.1.
16137
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- net
view.net, 2021. URL/ittss://openeeee wwwp://weweeettteeeeeeeee www.ffiruiiiiooeeeoro jm. 3mm
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. CoRR, abs/2309. 14509, 2023. doi: 10. 48550/ARXIV.2309. 14509. URI. htt.:https:///oot wtt.:/dooo..oo....oo wttp:///....../......oooooo.o 2309.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pp. 1601-1611. Association for Computational Linguistics, 2017. doi: 10. 10.18655/V1/117-1114 URL https: //doi. 1ttp:/10..653/11111
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, 00///0001006611 2020. URL htt.::tpp///// /tabpp://arxiie http:///rxiv..oor///////////// www..orgg/bbb//0//// 08361

---

Published as a conference paper at ICLR 2024
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net, 2020. https::/apeneeeieee. htwp://wwwwwe.nee/orrrr.e.. forummi rkgNKkHtvR
Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. CoRR, abs/2205.05 2022. doi: 10. 10.48550ARRX1.22255.09999
htttps////oo:
10. 40.48850/aa aa..sv/ar/ivvvo 2205. 051 98
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Biig-WWe&&o Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: benchmark for question answering research. Trans. Assoc. Comput. Lingwistics, 7:452 466, 2019. doi: 10.1162/TACI) A) 00276. URL https: /
10. 16..1122EEE 10122tacl__0 00276
Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can open-source Ilms truly promise on context length? Technical report, 2023a. httphtts://mmm///
org httg///lgg///// 001-0230006-29-1gchheee
Dacheng Li, Rulin Shao, Anze Xie, Eric Xing, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. Lightseq: Sequence level parallelism for distributed training of long context transformers. CoRR, abs/2310.03294, 2023b. doi: 10.4850/AXXV2230002222 48550 2310. 03294
http:t:s:
Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from system perspective. In Anna Rogers, Jordan L.. Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 2391-2404. Association for Computational Linguistics, 2023c. doi: 10. 10.18653/V1/2023 ACL-LONG. 134. htRhsttps:/ //toi///oo..oo .or.lorgg/io 10. 1 (188665/v/ v/ /2023.22.2222 acl-aclllllo long 134
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic hu- man falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 3214-3252, Asso- ciation for Computational Linguistics, 2022. doi: -ong 229
URL
https://dol //t//oioorg//.../ 1865186634100 htt////w/22222222
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near- infinite context. CoRR, abs/2310.01888/ 2023. doi: 1048550ARXIV.230000105 URLhttps
//doi.org//0. 0B5.S.885505aaa. www.iii..311..... 2tt.::/.........3..52........... 01889
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. www.:/ett///////. d Bkg6RicqY7
OpenReview.net, 2019.
Kai Lv, Shuo Zhang, Tianle Gu, Shuhao Xing, Jiawei Hong. Keyu Chen, Xiaoran Liu, Yuqing Yang, Honglin Guo, Tengxiao Liu, Yu Sun, Qipeng Guo, Hang Yan, and Xipeng Qiu. Col- lie: Collaborative training of large language models in efficient way. In Yansong Feng and Els Lefever (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2023 System Demonstrations, Singapore, December 6-10, 2023, pp. 527-542. Association for Computational Linguistics, 2023. URL https:://aclaathahtttt org/2023 www 48
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.). Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium, October 31 November 4. 2018, pp. 2381-2391 Association for Computational Linguistics,, 2018 doi: 18.18//3/V 1/D18- 1260. http:/11111113311111111113
URL https:
12

---

Published as a conference paper at ICLR 2024
Erik Nijkamp. Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang. Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL /tapps://oee /ttt:s:://ppeerevieww net. pdf SaffiddiaYcJJKPY/
OpenAl. Gpt-4 technical report. Technical report, 2023.
Arka Pal, Deep Karkhamis, Manley Roberts, Samuel Dooley, Arvind Sundararajan, and Siddartha Naidu. Giraffe: Adventures in expanding context lengths in llms. CoRR, abs/2308. 10882, 2023. doi: 10.48550/RXXVV23008 10882. URL hattps://don org wttp///....0///..11.1.... 2308.
10882
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma. Jana Thompson, He He, and Samuel R. Bowman. Quality: Question answering with long input texts, yes! In Marine Carpuat. Marie-Catherinn de Marn- effe, and Iván Vladimir Meza Ruíz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties; Human Language Tech- nologies, NAACL. 2022, Seattle, WA, United States, July 10-15, 2022, pp. 5336-5358. Associ- ation for Computational Linguistics, 2022. doi: https://doi.org/ http:/.0..1153/1111 www.:2222nnaaclaanee www.aaaaclmm..n...0
URL
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedinggs of the 54th Annual Meeting of the Association for Computational I Linguistics, ACI. 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V// P16 1144, URLhttps /oii org
18611886533338 11t./1....63//1/111111 1144
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. CoRR, abs/2309.0007/. 2023. doi: 10,48550/ARXIV.23099 00071. URL httpt::/ wtt.://///g....///arttc.cc.. 20..
org 10. w8550/arX111.
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, long: Attention with linear bi- ases enables input length extrapolation. In The Tenth International Conference on Learning Representattinss ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL
https::/openrest
net/ meet//oooumlddRBSPPPP
Shawn Presser. Books3, 2020.
URL.
32028214932997888
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations toward training trillion parameter models. In Christine Cuicchi, Irene Qualters, and William T.
Kramer (eds.), Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, pp- 20. IEEE/ACM, 2020. doi: 2..1099/$441405.220.00000 7/doi aoroi.orr//000
https:
2020. 00024
Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown and Yoav Shoham. Parallel context windows improve in-context learning of large language models. CoRR, abs/2212. 10947, 2022. doi: 10.48550ARRXI.2212.1000 URL htttt::t///dat www./orggg.... 48550/ 2212.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code. CoRR, abs/2308. 12950, 2023. doi: 0..4550/ARXIV.330... 12950.. URL https:: doi, org. 48550/ arwii.iiiiv. 2308. 12950
13

---

Published as a conference paper at ICLR 2024
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An ad- versarial winograd schema challenge at scale. In The Thirty-Four Fourth AAAI Conference on Artifi- cial Intelligence, AAAI 2020, The Shirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelli- gence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp- 8732 8740. AAAI Press, 2020. doi: 10.1609/AAALV334000....
URL http::://.i..oo///....o 1111 v34105.
6399
Jianlin Su. Nbce: Naive bayes-based context extension, May 2023a. URL SRLpsttts://///
fm/
archives/ 9611
Jianlin Su. Improving transformer: Length extrapolation ability and position robustness. 2023b.
URL htttt::/sppacesess ac. en/ cn/archives/9449
Jianlin Su. Rerope: Rectified rotary position embeddings, July 2023c. URL htttpt:::///tttub
com/ bo wwajooonerrrrr rerope
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer
with rotary position embedding. CoRR, hass/2044.99644, 2021. URL Btllhttts://arxiv.com
09864
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaud- hary, Xia Song, and Furu Wei. A length-extrapolatable transformer. In Anna Rogers, Jordan L., Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Pupers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 14590-14660 14604. Association for Computational Linguistics, 2023. doi: 10.18653/ htt.tpp://o...oo///110..
V1/202.ACCLLLNGG. 816
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, 12ss/232..1971, 2023a. doi: 10.48550/ARXIV.2302111 pettp:://oo.orq//
URL
48550 www.vvarxvvvvv. 2302 13971
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferres Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, Anthony Hartshorn Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lec, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie. Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine- tuned chat models. CoRR, abs/2307 00288098888888 2023b. doi: 10.4850/ARX1122307.0009 209000098..7..888.. URL https://doi.oron http://ii.ogg...//////o 0ttp://1..855.../1 48540/ aat.0//axiivv... 2307 09288
Bo-Hsiang Tseng, Sheng-syun Shen, Hung-yi Lee, and Lin-Shan Lec. Towards machine compre- hension of spoken content: Initial TOEEL listening comprehension test by machine. In Nelson Morgan (ed.), Interspeech 2016, 17th Annual Conference of the International Speech Commu- nication Association, San Francisco, CA, USA, September 8-12, 2016, pp. 2731 ISCA, 2016. doi: 1021457/INTERSSEECHH..00 Interspeech.co catrrpeecch.2166-800 2016-876
URL https: /top::://do..oo htt.:/oo.o.orr/////o 21437/
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp-
14

---

Published as a conference paper at ICLR 2024
5998 6008, 2017. URL http://pps//ppoeeeo........e/o/p http::/ww.....aaoo.coocpor/r//// T
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Anmual Conference
Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC. Canada, pp. 3261-3275, 2019. URL http:tttt/ssoccediiineee neuripp....euusss.. oc/ppaper/2019/hatii Ats html
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. CoRR, abs/2006.04768, 2020. URL htt.::s://ex.....rr////o///
0706.04760
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Trans- formers for longer sequences. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, De- cember 6- 12, 2020, virtual, 2020. URL http::tst////oceeeetteeeeeeeeete
33:
neur pttppppccppppppec cc/paper/ htmm_
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 4791- 4800. Association for Computational Linguistics, 2019. doi: 10.18653/V 1472. URL htttp::/ddo..rg//220 http:///.............6//////////
A PRELIMINARY
A. ROPE FROM SEQUENCE DOMAIN
Transformer models require the integration explicit positional information through positional embeddings to effectively discern the order of input sequences (Vaswani et al. 2017). In this work, we direct our attention to the specific instance of positional encoding known as Rotary Position Embedding (RoPE) (Su et al.] 2021), prominently featured in the architecture of the LLaMA
model Touvron et al. 2023a b). Given a query vector
ge .... (d-1) E Rdd at position
k6 [dd Rd at position RoPE first splits k2 into pairs on the direction of feature dimensions, with every two dimensions forming a complex number, or vector in the complex plane as follows:
t and a key vector k, k.
[
(d/2-1) 1)]
gl") qe iq.
(2n) (2n+1)
q
q
k. (0)
(d/22-1
i.(m)) ((2m) ik! (2)+1)
After that, RoPE injects the position information by element wise multiplication between the pre- processed dr. k. and list of 0- D-parameterized rotary vectors in the complex plane. When attention
15

---

Published as a conference paper at ICLR 2024
is calculated, relative position information t is acquired through cos and sin.
A Re
(qe O e0
(ks e
d/2-1
d/2-1
Re
Σ
(n) elt0 c(
Re
Σ m=
qi
(a)
(()) (((
(8)
n=U
d/2-1
(2m)
+ Ut
(2n+1) (22m++1)
5)0x
Σ n=0
(2n) 1(2n+1) qt (2n + ((2m)
qi
sin (t 8 s)nn
While RoPE can theoretically convey the relative information at any context length, RoPE still fails to extrapolate practically. is worth noting that rotary angles 0 in Equationnpplay an important role. In the vanilla design of RoPE, 0 is defined Equation 9 Different angles correspond to different features and that is the starting point of most RoPE-based extrapolation methods (bloc97 2023b
Rozière al. 2023
Pal al. 2023 shown in Table
[oo.
0a/2-1]
On = 10000-2n/44 2n/d
For example, the NTK method (bloc97 its value form, thus enabling RoPE-based LLM to adapt to position information in longer context without further training.
2023b a reduces the rotary angle under the maintenance of
A.2 ROPE FROM FREQUENCY DOMAIN
From a frequency domain perspective, the rotation operation of RoPE can be viewed as the Inverse Fourier Transform from the frequency domain to the time domain.
d/2
Ar.s Re
Σ
Qt
(n) (n)
m00
(10)
Re
F
(n) (n)= =
At this juncture, the learning objective for RoPE-bases LLM is essentially to understand features in terms of frequency. Depending on the value of 0,, the higher dimensions correspond to the longer period as shown in Equation 11 well the lower-frequency features reflecting the longer contextual semantic relationship (Chen et al. 2023 Rozière et al. 2023).
2T T,
1000020/4 for n 0... d 2 -
(11)
0n
However, as illustrated in Figure 3 the trigonometric functions of lower-frequency features do not complete a full period within the training context. As a result, RoPE-based LLMs might not fully recognize the periodic nature of sin and cos waves, leading inadequate training. Consequently, these lower-frequency features are more susceptible to under-fitting or over-fitting. Hence, the num- ber of well-trained dimensions is essential and this is the critical dimension raised in Section 3.2
B APPENDIX
B. EXPERIMENT SETUP
We conduct experiments on the pre-trained 7B and 13B LLaMA2 models Touvron al. 2023b)
For fine-tuning 7B and 13B models, we use 32 A 1000 GPUs and adopt ZeRO3 strategies (Rajbhandari al. 2020). We use AdamW (Loshchilov & Hutter 2019) with 81 0.9 and B2 0.999. We set the learning rate to 2 x 10 10-5 with no warmup. We set the max gradient norm to 2.5 for 7B and for 13B respectively. We the weight decay to zero.
16

---

Published as a conference paper at ICLR 2024
Applying Phase
Training Length
Context Window
Auxiliary Window
Name
Ratio
Note
xPos
Sun al.
2023
Pre-training
1K
4K
x4
BCA
Chen et
inear Pr
2023)
Fine-tuning
16K
16K
x1
X
Giraffe
Pal al.
2023
Fine-tuning
4K
32K
X8
truncated
Code L[ MA
(Rozière al.
2023)
Fine-tuning
16K
100K+
X6
YaRN
PPng et al.
2023
Fine-tuning
64K
128K
x2
window ppl
Fixed TK
(bloc97
2023b)
Inference
4K
16K
X4
Dynamic
(bloc97
2023a
NTK
Inference
4K
16K+
X4
PCW
(Rttner et
2022
Inference
2K
6K
x3
for ICL
NBCE
Suu
2023a
Inference
2K
10K
X5
for ICL
LM- Infinite
HHan et al. 2023
Inference
4K
32K+
X8
A-shaped
Scaling RoPE
4K
100K+
X55
base=1000000
Fine-tuning
(Ours)
16K
1M+
X64
base=500
Table 2: RoPE-based Extrapolation Strategies. For those applied fine-tuning, the max training length is the tuming context length. For PCW and NBCE, the context size is the max training length times the optimal chunk number. It is worth noting that Scaling RoPE, namely tuning RoPE with smaller or larger bases discussed in this work, realizes the max context window and extension ratio.
For fine-tuning RoPE with different bases, we set the global batch size to 128, tuning the context length 4K, the same the training length, and the evaluating context length to 100K. We fine- tune the models for 1K steps using the next token prediction objective with training data from the Pile (Gao et al. 2021) and compare the tuning performance on the validation set Books3 subset (Presser] 22020) from the Pile. We fine-tune LLaMA2 with CoLLiE(Lv al. 2023), collaborative toolbox for tuning large language models an efficient way, and conduct evaluation discussed below
with OpenCompas Both training and testing accelerated by FlashAttention (Doo 2023)
We compare results with mainstream extrapolating strategies, such as Linear Position Interpolation (Chen al. 2023) and NTK method (bloc97 2023b a). For fine-tuning with Linear Pl, the global batch size to 64 and tuning length to 8K, which follows Chen et al. (023). For fine-tuning with 16K context length,
follows[Chen
the global batch size 32. When fine-tune LLaMA2 7B with the
last 36 dimensions being cut off, discussed in Section 3.4, set the softmax scale /92, the square
root of updated dimension size of q:, k2, and keep other setups the same. Except for the position embedding, do not modify the architecture of I.LaMA2 (Touvron et al. 2023b).
B.2 SHORT CONTEXT VALIDATION
Next, we validate whether Scaling RoPE, namely the fine-tuning RoPE with different bases on original context length, has side effect on LLM. use short context tasks including those listed in the Hugging Face Open LLM Leaderboard 2023). such 0-shot Lambuda(Papemo ct al. 2016), Zellers et al. 2019), NOiKwiatowwkki et al. 2019), TriviaQA Joshi et al. 2017 MBQylMi et al. 2018 PIQA et al. (020)) TruthfulQA
al. 2022). Wino- grande (akaguuhi et al. 2020), SuperGLUE. Wang et al. 2019). IRC-easy/challengel Clark et al, 2018) and 5-shot MMLU HMMUU(Hnnryycke
al. 2021), to the performance of RoPE-based LLM after the fine-tuning process. The results obtained are shown in the Table 3 and Table The results
hhtps://ppencomppssoor..c
17

---

Published as a conference paper at ICLR 2024
Lambada
SuperGLUE
Hellaswag
PIQA
Winogrande
OBQA
LLaMA2 7B
73,30
50.43
69.45
76.66
61 225
32.00
NTK fixed a 8
67.53
51.13
68.43
76.28
61.48
29.40
Linear PI À
72.64
50.71
69.75
76.01
61.56
40.60
Linear PI À
71.82
52.34
69.48
76.33
61.56
30.40
base=500
69.51
48.34
66.46
74.97
59.67
31.00
basc=652
68.78
46.86
64.00
75.35
60.22
28,80
1354=1304 1304
69.24
47.22
65.51
76,06
60.30
25,00
hase=2608
69.38
48.04
65.61
75.24
60.93
28.00
fase==1000
72.62
51.49
69.60
75.95
61.40
38.00
b0se=40000
72.77
$1.02
69.46
76.06
61.33
36,20
Dase=80000
72.68
$1.20
69.44
76.01
60.69
37.40
base= 160000
72.54
50.24
69.07
76.01
60.77
36.80
basc=400000
72.25
51.08
68.86
75.68
60.85
36.00
base=600000
72.09
51.02
68.90
75.63
61.33
34,80
base= 1000000
71.80
50.84
68.82
75.63
60.77
35.40
base=500, 16K
68.62
48.20
62.35
73.67
58.33
25.00
basc=10000,
72.17
50.60
67.68
75.79
60.22
28.80
base=40000, 16K
70.77
48.03
67.69
76.33
61.33
28.20
base=80000, 16K
72.48
$1.29
69.54
76.33
61.09
32.60
base=12000k 16K
72.66
50.34
69.54
76.22
61.96
32.20
base=1000000, 16K
71.14
49.80
68.45
75.90
61.88
29.80
LLaMA2 13B
76.48
52.35
74.22
79.54
63.85
42.60
NTK fixed 8
72.56
52.64
73.56
79.71
62.67
37.00
Linear PI À =
75.18
63.15
74.93
78.94
63.69
51.40
Linear PI À
74.29
61.64
74.81
79.38
63.77
50.60
base=500
72.56
$2.81
71.38
76.88
64.17
35,20
base=652
73.78
52.54
71.62
77.48
62.83
35.20
base=130k
74.21
52.19
72.38
77.37
64.01
36,80
base=2608
74.54
52.54
73.63
78.02
63.22
43.40
base=10000
75.12
53.19
75.19
79.00
62.98
55.00
4000=
74.87
53.11
74.76
79.60
62.98
52.60
basc=80000
74,89
52.37
74.76
79.27
62.90
53.80
160000
$2.63
74.50
79.05
62.75
54.20
base=400000
74.23
52.47
74.50
78.89
62.19
54.40
base=600000
74.19
$4.14
74.24
79.00
63.77
$3.60
100000
74.58
50.82
74.36
78.67
62.98
51.80
base=500, 16K
69.34
49.28
67.23
76,50
58.48
29.60
mase==0000. 16K
74.00
53.44
73.16
78.89
63.93
35.00
base=40000, 16K
75.06
$4.08
74.33
78.78
62.51
38.00
basc=80000,
74.77
53.28
75.11
79.05
63.61
38.20
base= 12000, 16K
75.06
53.91
74.85
78.94
63.61
43.60
base=1000000, 16K
74.71
$1.95
74.68
79.11
63.30
40.80
Table 3: Short context validation results of scaling RoPE with different base and other extrapolating
methods LLaMA2 (Touvron et al. 2023b). The first box is the result of 7B size and the second box is those of 13B version. I.LaMA2 7B or 13B corresponds the original result well as that of Dynamic NTK (bloc97 2023a) since Dynamic NTK does not affect the model within the training context. The final score of SuperrLUE is the average score of all corresponding subtasks.
of Scaling-Rool RoPE are comparable with that of the pre-trained directly fine-tuned LLaMA2 (Tou- vron et al., 20236). This indicates our approach not only enhances the extrapolation performance but also maintains the inherent knowledge of L.M.
18

---

Published as a conference paper at ICLR 2024
NQ
TriviaQA
TruthfulQA
ARC-e
ARC-c
MMLU
LLaMA2 7B
19.06
54.09
35.09
50.26
36.61
46.78
NTK fixed a 8
19.89
53.16
35.09
48.68
34.58
39.36
Linear
18.64
52.57
34.80
56.44
35.93
47.27
Linear PI
19.78
52.23
35.09
55.91
36.95
46.80
base=500
20.36
52.43
35.09
53.09
38.64
38.70
basc=652
17.12
48.90
34.36
46.74
33.90
33.43
Dase=13044
17.70
51.27
35.09
49.38
37.63
33,46
hasc=2608
16.01
51.21
34.06
48.15
34.58
37.07
basc=10000
20.14
55.14
35.53
54.32
38.31
47.73
base=40000
17.34
55.27
35.23
55.03
36.95
47.34
base=80000
17.81
54.72
35.23
53.09
37.63
46.87
16se=160000
18.98
55.28
35.09
53.26
38.64
46.24
basc=400000
20.39
54.91
35.23
52.03
38.64
46.01
base=600000
19.20
54.75
35.67
53.26
37.97
46.39
mase=1000000
19.50
54.76
35.82
52,20
37.29
46,29
base=500, 16K
11.58
37.95
34.36
46.21
36.95
28.79
basc=10000, 16K
15.29
50.31
34.50
49.91
36.61
42.71
base=40000, 16K
15.32
49.16
34.94
49.38
31.86
45.44
base=80000, 16K
16.51
51.16
34.36
52.20
35.25
45.08
base=120000, 16K
16.15
51.84
34.21
51.85
34.24
43.75
h0sc=1000000, 16K
16.68
53.14
34.80
50.79
34.92
41.83
LLaMA2 13B
24.82
60.69
34.65
59.44
40.00
55.77
NTK fixed 8
22.77
60.81
34.80
58.38
40.34
52.04
Linear PI 2
25.96
59.21
34.94
63.49
40.00
57.20
Linear PI
26.07
59.34
35.67
64.20
39.66
56.08
base=500
11.50
57.54
33.77
57.67
38.98
47.98
base=652
15.98
58.67
35.23
58.73
38.31
50.49
Dasc=1304
13.60
59.12
35.23
58.73
41.36
53.59
basc=2608
20.00
59.62
34.80
60.67
42.71
55.06
base=10000
26.68
61.57
35.09
62.79
42.03
55.78
hase=40000
26.87
60.80
35.53
60.49
42.37
55.26
basc=80000
26.76
61.25
35.38
61.20
41.36
55,32
base=160000
27.06
61.22
35.96
60.14
42.03
54.35
base=400000
27.53
60.94
35.82
62.96
42.37
54.01
base=600000
26.81
61.28
35.67
61.20
42.03
53.75
base= 100000
26.51
59.22
35.53
60.85
42.37
52.94
base=500, 16K
11.86
44.74
35.38
49.21
33.22
38.34
1000= 16K
19.14
51.22
34.65
58.20
41.02
51.11
base=40000, 16K
23.68
57.32
35.53
59.61
41.69
53.68
8000= 16K
24.90
58.74
34.94
61.38
38.98
52.86
base=1200Mk 16K
25.37
58.16
34.65
61.90
41.02
52.93
b0se=1000000, 16K
24.35
57.26
35.53
59.44
39.66
51.61
Table 4: Continued table of Table 3
The final score of MMLU is the average score of all subtasks.
B.3 LONG CONTEXT VALIDATION
Besides, validate that our proposed framework can improve the long-range seal-world tasks, we also compare the close-ended subtasks included in L-Eval benchmark An et al. 2023) within 16K context length, including the 16-shot GSMSK Cobbe et al. 2021), TOFFL-QA Tsegg et al. 2016). GSMSK/Cobbe
TopicRetrieval(Li al.) 2023a), QuALITY (Png et al. (2022), and Coursera An al. (023). Our method is still capable of these tasks, as shown in Table For RoPE-based LLMs fine- tuned on pre- training context, namely 4K tokens, they still have certain effects these tasks with 16K context
19

---

Published as a conference paper at ICLR 2024
size, which proves of the claims of work, that tuning with the original pre-training context can also boost the length extrapolation. This proves that we extrapolate the RoPE-basee -eased LLM to longer context without the disturbance of domain distribution in pre-trained data concerning the change of training context length (Peng et al., 2023). We also report the evaluation results of RoPE- based LLM fine-tuned with longer context, such 16K tokens, in Table Table Table and the length extrapolation improvement will be further analyzed in Appendix
GSM
TOFFL.
TopicRet
QuALITY
Coursera
LLaMA2 7B
0.00
34.94
0.00
3.96
NTK fixed
5.00
34.20
31.33
34.16
21.51
NTK dynamic
9.00
37.17
24.67
30.20
22.09
base=500
4.00
25.65
0.00
18.32
19.19
base=500, log-scaled
3,00
25.65
0.00
18.32
19.77
hase=652
6.00
16.73
0.00
13.37
11.63
base=652, log-scaled
6,00
16.73
0.00
16.83
11.63
basc=10000
0.00
18.59
0.00
2.97
0.00
b0se=40000
13.00
34.20
14.67
22.77
11.63
base=80000
11.00
36.43
29.33
23.76
15.70
basc=160000
10.00
40.15
38.00
22.28
16.28
base=400000
8.00
41.64
36.67
22.77
15.70
basc=600000
12.00
38.66
34.00
25,25
16,86
base=1000000
9.00
31.97
34.00
21.29
16.28
basc=2000000
7.00
30.86
31.33
22.77
15.70
basc=500, 16K
2.00
25.65
0.00
20.35
basc=1000k 16K
5.00
42.38
0.00
19.80
13.95
base=40000, 16K
11.00
43.49
20.67
28.22
19.19
base=80000, 16K
12.00
43.49
35.33
30.20
21.51
basc=120000, 16K
14.00
43.12
40.00
26.24
20.93
basc=1000000, 16K
13.00
44.24
41.33
25.74
21.51
aLaMA2 13B
0.00
59.48
0.00
6.44
0.00
NTK fixed
8.00
58.74
36.00
35.64
23.26
NTK dynamic
22.00
59.85
30.00
35.15
28.49
base=500
9.00
37.17
0.00
30.20
25.58
base=500, log-scaled
11.00
37.17
0.00
28.71
25.58
basc=652
12.00
37.17
0.00
30.69
25.58
base=652, log-scaled
9.00
37.17
0.00
31.19
25,00
h0se=10000
0.00
56.13
0.00
7.43
0.00
b0se=40000
18.00
64.68
20.67
49.01
16.86
basc=80000
20.00
63.94
30.00
52.97
24.42
base=160000
23.00
62.83
39.33
51.49
24.42
base=400000
17.00
59.48
34.67
$1.49
29.07
basc=600000
14.00
63.94
36.00
47.52
29.65
base= some=100000 1000000
10.00
64.68
28.67
45.05
19.19
basc=500, 16K
6.00
20.07
0.00
21.29
26.74
base=10000, 16K
13.00
55,39
0.00
38,61
25.00
basc=40000,
25.00
63.94
19.33
49.50
18.60
basc=80000, 16K
21.00
63.20
34.67
40.10
15.12
basc=120000, 16K
22.00
63.94
40.67
39.11
19.19
base=1000000, 16K
9.00
63.57
33.33
48.02
16.86
Table 5: Long context validation results of scaling RoPE with different base and other extrapolating methods LLaMA2 (Touvron et al. 2023b). The first box is the result of 7B size and the second box is those of 13B version.
20

---

Published as a conference paper at ICLR 2024
10:00
Peebohy
Cantac Lengl
(a) Results on LLaMA2 7B
(b) Results on LLaMA2 13B
Figure 9: Perplexity of RoPE fine-tuned with 16K context length and smaller or larger bases on the validation data of Books3 (Presser (220). Surprisingly, as the base increases from 500 to 1000000, the extrapolation capability first becomes weaker and then becomes stronger.
C EXTENSION
Concerning that the current extrapolation methods in the fine-tuning phase dominantly apply longer tuning contexts, we propose an extended version of the scaling law of RoPE-based extrapolation.
Theorem 3. (Extended Scaling Law of RoPE-based Extrapolation) For RoPE-based LLMs pre- trained with context length Train and critical dimension deuta. if we adjust the base to 3 and then conduct fine-tuning with context length Timee 2 Timin. the extrapolation performance of RoPE-based LLMs will get improved. Importantly, there exists critical base Po decided by Tume and Twain-
log Timis 2
Bo 10000
(12)
If B > So. the extrapolation upper bound is decided by 3 and deara as follows:
Textra 2n .ghom à = 2T 3 l0g10000
(13)
Otherwise, the extrapolation upper bound is Trunc and the critical dimension is updated satisfying
[d_ log
Tume
d 2
T Tirain
dextra 2
log10000
dextra-
(14)
2π
2T
However, the extrapolation beyond Tume will acquire further enhancement if B gets smaller. Partic- ularly, when 3 is smaller than B1, ,B, S3 as follows, the enhancement will be more significant.
2T
B2 Tume π
Tume 2
31
B3
(15)
π
Theorem 3. serves as both a combination and generalization of Theorem 1., Lemma 1. and Theorem
2. Here, the critical base is the worst base for extrapolation as well the smallest base forcing RoPE to extrapolate depending on the feature dimensions within the critical dimension. Specifically, when
Tume = Tirain the critical base, 30- 10000, relates to the intriguing observation outlined in Section
2 Equation 3 corresponds to tuning RoPE with larger bases discussed in Section 3.3 Equation
corresponds the definition of critical dimension
Section 3.2 and Equation 15 corresponss
to tuning RoPE with smaller bases discussed in Section [.1] If tune > Tuain- RoPE-based LLMs
can accommodate a broader context window. As illustrated in Figure D when fine-tuning LLaMA2
7B and 13B (Touvron et al.) 2023b) within 16K context, the max context length exceeds or equals 16K, surpassing the original LLMMA2 7B and 13B respectively, regardless of the base value.
From periodic perspective, since both base and training length have changed, it prompts us to consider whether additional positional information has been integrated during the fine-tuning phase for feature dimensions beyond the critical dimension, namely the 92nd dimension for LLaMA2 (Tourron et al. 2023b), as indicated in Equation According to the definition of the critical dimen- sion, based the current base B, we calculate how many dimensions the RoPE- LLM has where sin and cos complete period within the tuning length. If base B > Bo, then deaa. the
21

---

Published as a conference paper at ICLR 2024
number of dimensions that cover period during fine-tuning have already been able traverse complete period during pre pre-training, given that
d log
Tume 2T
d
Tune 2T
d
T tune 2x
dexara =
2
2
log,3
2
2
log
lox
Tyee
10000
T Tume
d log Fun 2x
Tinin 2n 2 log 10000 2n
10g10000
2
loog Fuain 2e
TiN
2T
d
Tusin
log10000
dextra
2
2n
Therefore, the critical dimension remains unchanged. Referring to Theorem 2 Section 3.3 we can calculate the extrapolation upper bound based on the updated base and the original critical dimension as Equation [3] exactly the same as Equation []] For LLaMA2 (Touvron et al. 2023b) fine-tuned with 16K context, illustrated in Figure the critical base is 7 71738, given Equation [2] For bases greater than 71738, such as 80000, 120000, and 1000000, their extrapolation upper bounds surpass 16K and the larger base corresponds
longer context, corroborating our theoretical framework.
If base 3 S Su. then during the fine-tuning phase, the number of dimensions able to complete a period surpasses the original critical dimension, so the critical dimension is updated as Equation [4] Besides, since this dimension depends on the fine-tuning length, the extrapolation upper bound is still constrained within the fine-tuning length T Tune- However, if [ is so small that the input of every cos (t- s),,, sin (t- - s)6, can span values from 0 to π 2 ग, or 2 within the fine-tuning length Twne. as indicated by Equation [15] similar to Theorem in Section [3.]] then the extrapolation per- formance will get further improved, marked by more stable perplexity growth curve. For LLaMA2 TTuuroo et al.) 20236) fine-tuned with 16K context, as shown in Figure D for bases smaller than the critical base 71738, such as 60000, 20000, 10000, and 500, the performance curves become progressively more stable. Among them, although B 10000 performs poorly in fine-tuning at the original context length, the performance gets significantly improved this time because the inputs of cos or sin have traversed to 2 within the 16K context length. When B = 500. LLaMA2 achieved 8 1000000, namely the design of Code LLaMA (Rozière al.
similar terrific performance 2023). a context length with at least 100K tokens, breaking the curse of entropy explosion mentioned in Han al. (2023). Since there exists an upper bound for extrapolation based on B 1000000, RoPE tuned with base 500 on 16K context length has the potential to extrapolate to an infinite context, thus answering Q3 in the Introduction.
Similarly, we also use scatter plots to visualize the fluctuation of attention scores for different bases after fine-tuning with a 16K context as illustrated in Figure 10 For base 500, given its exposure to enough fluctuations during the training phase, the perplexity curve remains notably stable. For base 10000, it clear that the fluctuation of attention scores in the last 36 dimensions somewhat limited, leading to a noticeable improvement in extrapolation performance given Equation 155 For base 40000, the position information acquired in the fine-tuning phase shrinks further as the base increases. For base 120000, the critical dimension goes back to 92 dimensions, and the extrapolation performance is governed by the first 92 dimensions. Remarkably, the extrapolation upper bound given Equation 13 matches the maximum supported context length. For base 1000000, the period of the first 92 dimensions is further extended, corresponding to a context length expanding beyond 100K. Eventually, based the above interpretation, we validate the correctness of Theorem 3. and provide a unified framework to explain the extrapolation of RoPF--based LLM with arbitrary base and fine-tuning context length.
D DISCUSSION
Besides, we discuss the instructive value of our theory for other extrapolation strategies focused on achieving longer context during the testing phase. These methods are still necessary given two facts. On one hand, the performance of RoPE with a smaller base is still left behind compared with RoPE with much larger bases, such 1000000, as shown in Figure I On the other hand, for RoPE with a base that is not large enough, it still can not extrapolate to a context of 100K or longer as shown in Figure 2 In order to further enhance RoPE's adaptability longer context, whatever the base
22

---

Published as a conference paper at ICLR 2024
Figure 10: The relation between attention scores in first 92 and last 36 dimensions with the extrap- olation performance in LLaMA 7B (Touvron et al., 2023b) evaluated or fine-tuned with different bases 16K context length. The meaning of each row is the same as that in eecurep that the second row highlights the changes in the first 32K tokens instead of the first 16K tokens.
value is, we discuss the effect of two complementary methods in the inference phase, log-scaled attention Suu 2023b) and dynamic-scaled RoPE (bloc97 2023a on RoPE with different bases.
Log-scaled Attention
as shown in Equation 16 is classic technique originally raised in Chiang
& Cholak (022) and eurrently applied in based extrapolation (Su 2023b Alibaba 2023 It
involves multiplying the original attention matrix by the logarithm of the current inference length
Traditionally, the base of the logarithm is training length Timin- However, given Equation 13
Theorem 3., the attention score within the max supported context length Texia is reliable. So we take Texra as the logarithm base and set the lower limit for the logarithmic correction value as meaning that no additional log scale is required within the extrapolation upper bound.
d/2-1
A = Re
Σ
4:
(n) c(t-
Pr
(16)
Pr = max (1,10g
Besides Log-scaled attention, window method, such sliding window and its variant, is also widely accepted strategy for extrapolation, used in inference or evaluation (Press et al. [022]] Sun al. 2023). Compared with the above strict window-based method, we follow the xPos method proposed in Sun et al. (2023). shown in Equation 17 originally used in the pre-training phase. In this work, we regard this method as a soft sliding window used in the inference phase as use it as further complement to the log-scaled method. Still, we do little modification besides using the Textr as the denominator instead of the original denominator Tirain
d/2-1
A Re
Σ
Ht
(n) (n) Toom c(t-=))n SR
(17)
+ 2n/d = 0...
d
G. 7
1, 7 0.4 2
Dynamic-scaled RoPE namely Dynamic NTK (bloc97 2023a is a widely used extrapolation. Here, we only do two little modifications. One is to change the base 10000 in vanilla RoPE (Su et al. (02T) with the base scaled the fine-tuning phase, 3. The other is still to replace the T Timin in Equation ?? with Texta we derive given Equation 13 in Theorem 3.
for Ar.., On = (3 (8.))) 2n/d
where or max (1,2 (oog T.l 1+1
T -1).
(18)
23

---

Published as a conference paper at ICLR 2024
ConafE
1000
1000
1000 adoe
1000
Cankod Lengs
Concen Longh
Lenghh
(a) Results on LLaMA2 7B with base 10000
Coan Card Lomiltule
Ph di-
Cratier Tanget
Chatinc Eregla
Canted Enagle
(b) Results on LLaMA2 7B with smaller bases
ba02-40030
beae- 160003
$ssc04000000
500000
laa qpem (TEeena
2006
Cascrioogh
2001
4000
(c) Results on I.L.MA2 7B with larger bases
Figure 11: Perplexity on the validation data from Books3 dataset (Presser 2020 of LLaMA 7B
Touvron al. 2023bb based on RoPE with different bases enhanced with log- attention, xPos and cenamiccssallds RoPE and their combination. Here, wrong means calculating a₂ in Dynamic NTK (bloc97 2023a) pt in (Sul (023b) with training length Timin, while correct means using extrapolation upper bound Textra instead of minn-
We experiment with these two methods on LLaMA2 7B (Tovrron et al. 2023b) and get the results as shown in Figure [1] Figure (hasswss the results on LLaMA2 based on RoPE with base 10000. It is clear that both log-scaled attention almost does nothing for pre-trained and fine-tuned LLaMA2,, but show great improvement for fine-tuning LLaMA2 with cutting the last 36 dimensions in q: k, off. This phenomenon further proves that the fluctuation coming from the dimensions of q;, kg beyond the critical dimension is the root cause of the extrapolation problem of RoPE-based LLM.
shows
Figure shows the results on LLaMA2 based on RoPE with bases smaller than 10000. There is clear trend that with the reduction of the base value, the improvement obtained from the log-scaled attention more dominant while the effect of Dynamic NTK shrinks gradually. For RoPE with base 500, the perplexity curve of log-scaled RoPE is flat enough, indicating the extrapolation capability to support 100K context length. On the contrary, Dynamic NTK shows a clear side effect. Hence, the position information learned in the training phase is reliable enough for LLM to extrapolate further, corresponding to Theorem 1. in Section 3.1
Figure 1lc shows the results on LLaMA2 based on RoPE with bases larger than 10000, such as 40000, 160000, 400000, and 600000. We do not test the performance of two methods on RoPE with base 1000000, since it already achieved the context length of 100K. Here, enable RoPE with bases larger than 10000 and smaller than 1000000 to extrapolate beyond the context length of 100K in the inference phase. For RoPE with larger bases, the improvement of extrapolation performance
24

---

Published as a conference paper at ICLR 2024
obtained from Dynamic NTK is remarkable. The working principle of Dynamic NTK has been visualized Figure and discussed in Section 3.4
Besides, replacing the Tiain with dextra becomes significantly important for RoPE with larger bases. For example, Dynamic NTK is carried out based on Tirain the improvement will be limited and destroyed when the base is large enough like 400000 and 600000. This phenomenon proves the guidance value this work for other extrapolation works. In conclusion, for a base smaller than 33 defined in Equation [ every dimension has learned complete positional information. Then the log-scaled method is sufficient to enhance extrapolation. For a base larger than 3p. namely 10000 for tuning on the original context, Dynamic NTK in the correct way is a good helper for extrapolation to much longer context.
E TEXT CONTINUATION
Finally, we execute a text continuation as our case study. For LLaMA2 7B(Touvron et al. 2023b) fine-tuned with various base values and fine-tuning lengths, we provide a context of 32K tokens from 3ookss3(rresser (2200) and prompt the model to continue writing. The resulting texts, after cleaning the special characters, are shown in Figure 12 Remarkably, whether the base is set 500 or 1000000, the generated text remains coherent, grammatically accurate, and logically consistent. For base 500, extended tuning length or incorporating log-scaled attention Suj (2023b) in the testing phase yields improved continuation results. Given the strict extrapolation upper bound for base 1000000, there a compelling case that a model fine-tuned with base 500 possesses an infinite extrapolation potential.
LIMITATION
In this work, primary objective is to elucidate the mechanisms by which RoPE, using either smaller or larger bases, enhances extrapolation performance. Additionally, there remains a need to find whether there exist upper and lower bounds for adjusting the base. Besides, needs additional efforts to integrate the performance of RoPE with smaller bases in the pre-training stage. Finally, since the base value and pre-training context length are keys extrapolation, there might be an optimal pair of base and training length for RoPE-basee LLM pre-training to be discovered.
25

---

Published as a conference paper at ICLR 2024
THIS BOOK IS FOR INFORMATIONAL PURPOSES ONLY. IT IS NOT IN- TENDED TO BE A SUBSTITUTE FOR CONSULTATION WITH A PROFESSIONAL. HEALTHCARE PRACTITIONER.
Context:
They invited airplane pilots with an average age of fifty-two who were interested in participating and evaluated performance in flight simula- tions. Every three minutes, the pilots were peppered with complex commands and codes that had to be remembered and implemented while under stressful flying conditions. After seven simulations, the study subjects were given Aricept for Reference: month and then retested under equally demanding but new simulations. The results showed that they performed better in the later round of evaluations, and the authors stated that Aricept "appears to have beneficial effects on retention of training on complex aviation tasks in nondemented older adults," and these findings were ultimately published in Neurology. There is already major pharmacologic market for physical performance- enhancing drugs. With an aging population, interest in medications for cognitive enhance- ment will likely grow.
Output (LLaMA2 7B): 000000000000000000000000000
Output (base=500); the purpose of improving their cognitive performance. Neuro-Geek: Alzheimer's s Drugs for the Healthy? While most adults will not develop Alzheimer's ss many adults will have mild cognitive impairments that affect how they perform in daily life and work. In the sector of the scientific community focused developing drugs to improve the symptoms associated with dementia, there is an understanding that those same drugs may be used by healthy people with the goal of maintaining cognitive acuity or even, in certain circumstances, improving mental agility above their established baseline. Output (base=1000000):
month. The results were clear: the drug improved their per- formance. The study was published in 2015, and the authors concluded that "the results suggest that the drug may be able improve cognitive performance in healthy individuals." The study was small, and the results are not yet conclusive. But it is a start. Prescription Antidepressants The first antidepressants were developed in the 1950s, and they have been used to treat depression ever since. But in the 1990s, scientists began to wonder if they might also be useful for improving cognitive functioning.
Output (base=500 16K): a week. The drug is cholinesterase inhibitor, which means it blocks the breakdown of acetylcholine, a neurotransmitter that helps regulate memory and learning. The results were striking. The pilots who took the drug performed better than those who didn't. They were able to remember more of the complex commands and codes, and they were able to implement them more quickly. The study authors concluded that the drug "may be useful in improving cognitive performance in healthy individuals." But there's s catch. The drug is not approved for use in healthy people. It's only approved for people with Alzheimer's disease.
a month, then switched to placebo for another month. The results were striking. The pilots who took the drug improved their performance by 20 percent, while those who took the placebo got worse. The study was small, and the results were not replicated in larger study. But the idea that healthy people might be able to improve their cognitive performance with drugs designed to treat dementia is intriguing. Prescription Antidepressants The most widely prescribed class of antidepressants,, selective serotonin reuptake inhibitors (SSRIs), are also used to treat anxiety, obsessive-compulsive disorder, and panic disorder.
Output (base=1000000 16K):
Output (base=500 16K log-scaled): a week. The drug is a cholinesterase inhibitor, which means blocks the breakdown of acetylcholine, neurotransmitter that critical for mem- ory and learning. The results were striking. The pilots who took the drug performed better than those who didn't. They were able to remember more of the complex commands and codes and to implement them more quickly. But the study also had major flaw. The pilots were not randomly assigned to take the drug or not. They were all given the drug because they were already healthy. The study's 's authors concluded that the drug improved perfor- mance in pilots who were already healthy. But that's not what the study actually showed.
Figure 12: Text continuation after 32n-he context from Books3 Prrsser 2020) of LLaMA2
7 Touvron et al. 2023b) tuned with base 500 or 1000000 and context length 4K or 16K. shows that our methods could still produce cohesive and fluid output text in much longer context.
26


Reviews:
[{"review_id": "KiesQeTgMs", "reviewer": "Reviewer_Gdv1", "summary": "Rotary Position Embeddings have played a significant role in improving the scalability of LLMs. However, RoPE faces problems when trying to extrapolate beyond the length the model was trained on.\n\nVarious approaches have been taken to solve the issue with extrapolation, and this paper goes in depth describing why these issues exist and derives pretty good solutions as well. Apart from this, it presents a good understanding of why some of the existing methods work and why a base of 10,000 is not a good value for RoPE's base.\n\nAll the claims are very well substantiated with theoretical as well as empirical analysis.", "questions": "None.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Pretty useful theorems that shows why and how RoPE's beta impacts extrapolation.\n- Results on critical base, d_{extra}, and T_{extra} and their relationship is a very useful contribution in this area.\n- Both experimental and theoretical analysis are strong.", "weaknesses": "A bit more clarification around the calculation of critical beta, and why extrapolation capabilities exist even when beta values are large could be added since we will see OOD embeddings as soon as T is larger than T_{train} during inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sLFqPnR5Ji", "reviewer": "Reviewer_jipE", "summary": "This paper focuses on the extrapolation capability of Rotary Position Embedding (RoPE) in large language models (LLMs). The authors first observe that pre-training RoPE with smaller or larger bases can significantly enhance its extrapolation performance. Then, they propose the Scaling Laws of RoPE-based Extrapolation, a unified framework that describes the relationship between extrapolation performance and base value as well as tuning context length from a periodic perspective. Furthermore, they explain the origin of the RoPE-based extrapolation issue by the critical dimension for extrapolation. Finally, they achieve extrapolation up to 1 million context length within only 16K training length on LLaMA2 7B and 13B.", "questions": "- Only 7B and 13B models are verified. Have you experimented with the proposed methods on larger or smaller models? So that the \"scaling law\" saying can be more convincing.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Observing that pre-training RoPE with smaller or larger bases can significantly enhance its extrapolation performance, providing a new perspective for extrapolation research.\n- Proposing the Scaling Laws of RoPE-based Extrapolation, offering a unified theoretical framework for RoPE extrapolation.\n- Explaining the origin of the RoPE-based extrapolation issue through the critical dimension for extrapolation.\n- Achieving extrapolation up to 1 million context length within only 16K training length on LLaMA2 7B and 13B, demonstrating the potential of RoPE extrapolation.\n- Providing guidance for other extrapolation methods, such as dynamic NTK and Code LLaMA.", "weaknesses": "- In fact, the analysis results of this article are basically available in the YaRN[4] paper, which had pointed out that certain dimensions have high frequency and enough turns are made during training, therefore, the training of these dimensions is sufficient and should remain unchanged during extrapolation; for those dimensions that are less than one circle, only interpolation (PI) can be used to ensure numerical stability. This is obviously the equivalent expression of the proposed concept \"critical dimension\" in this work.\n- Most experimental comparisons only report ppl, which does not represent real long context ability. The authors should conduct experiments on some practical benchmarks and tasks. Eg. PASSKEY RETRIEVAL[2], the Hugging Face Open LLM Leaderboard[3] (ARC, HellaSwag, TruthfulQA, etc).\n- The RoPE extrapolation methods discussed in the paper may have certain limitations in practical applications, it is just verified over perplexity, but not enough practical application scene.\n- Compared to baselines, it lacks some comparison results over computational resources and memory usage.\n- Missing reference. The work should compare results with ReRoPE[1].\n\n[1] J. Su. Rectified rotary position embeddings. https://github.com/bojone/rerope, 2023.\n\n[2] A. Mohtashami and M. Jaggi. Landmark attention: Random-access infinite context length for transformers, 2023. arXiv: 2305.16300.\n\n[3] Hugging Face. Open LLM Leaderboard, 2023. URL https://huggingface.co/spaces/ HuggingFaceH4/open_llm_leaderboard.\n\n[4] YaRN: Efficient Context Window Extension of Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CJnUl1K5P6", "reviewer": "Reviewer_Dcc1", "summary": "This paper studies how shall we scale the transformer context length up via modifying RoPE. Different from popular approaches that increase the rotary base, authors found both increasing and decreasing the rotary base could significantly enhance its extrapolation performance.", "questions": "See weakness above.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1) This paper studies a very interesting problem: How shall we modify the 10^4 rotary base? \n2) Well-written. \n3) Some reasoning and insights are very cool. Such as the Q1 and Q2 in the introduction.\n4) The theoretical analysis of the benefits of a smaller rotary base would be helpful to the long seq community.", "weaknesses": "1) I have one strong concern about the evaluation of this paper. This paper only evaluates the scaled model on the long-context perplexity and short-context well-established benchmarks. Low ppl on long context language modeling cannot really reflect the good results on long-range reasoning. The short-context results can only show the proposed approach is not that harmful on short tasks. If so, there is no evidence showing that the proposed framework can really improve the long-range real-world tasks, which is exactly what we espect in the long-range modeling research.\n\n2) There is no related work section. It is \"okay\" for expert readers, but it would be very helpful to make a broader impact if authors can write one. I think authors should at least include:\n- scaling seq len with position embedding. The authors discussed some of these approaches, but it would be better to summarize them in a thread.\n- scaling seq len with efficient implementation, such as Flash attention 1&2 [1]\n- scaling seq len with a distributed system, such as Sequence Parallelism [2][3]\n \n\n[1] https://github.com/Dao-AILab/flash-attention\n\n[2] Li, Shenggui, et al. \"Sequence parallelism: Long sequence training from system perspective.\" arXiv preprint arXiv:2105.13120 (2021).\n\n[3] Jacobs, Sam Ade, et al. \"DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models.\" arXiv preprint arXiv:2309.14509 (2023).\n\n3) It would also be very interesting to study the filling-in-the-middle pattern after applying the proposed trick. Is the filling-in-the-middle related to the position embedding? This is just a bonus question. It is an okay paper without this study, but I suggest authors to study this to deliver more insights.\n\nI give a relatively negative score at this stage because of the lack of real-world task evaluation, but the insights of this paper are interesting to me. Authors need to further justify them. I can improve my score if the authors can add the real-world long-context evaluation into the draft.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)