LongRoPE2: Near-Lossless LLM Context Window Scaling
Authors: Ning Shang
Li Lyna Zhang
Siyuan Wang
Gaokai Zhang
Gilsinia Lopez
Fan Yang
Weizhu Chen
Mao Yang
Venue: ICML 2025 poster (2025)
Keywords: Long-Context LLM
context window extension
RoPE
Primary area: deep_learning->large_language_models
TL;DR: We introduce a method for near-lossless LLM context window extension.

Abstract: LongRoPE2 is a novel approach that extends the effective context window of pre-trained large language models (LLMs) to the target length, while preserving the performance on the original shorter context window. 
This is achieved by three contributions: (1) a hypothesis that insufficient training in higher RoPE dimensions contributes to the persistent out-of-distribution (OOD) issues observed in existing methods;  (2) an effective RoPE rescaling algorithm that adopts evolutionary search guided by "needle-driven" perplexity to address the insufficient training problem;  (3) a mixed context window training approach that fine-tunes model weights to adopt rescaled RoPE for long-context sequences while preserving the short-context performance with the original RoPE. 
Extensive experiments on LLaMA3-8B and Phi3-mini-3.8B across various benchmarks validate the hypothesis and demonstrate the effectiveness of LongRoPE2. Remarkably, LongRoPE2 extends LLaMA3-8B to achieve a 128K effective context length while retaining  over 98.5% of short-context performance, using only 10B tokens -- 80x fewer than Meta's approach, which fails to reach the target effective context length.

LongRoPE2: Near-Lossless
LLM Context Window Scaling
Ning Shang"
Li Lyna Zhang 1I Fan Yang'
Siyuan Wang? 21 Gaokai Zhang
Gilsinia Lopez
Weizhu Chen
Mao Yang'
Abstract
(RoPE) (Su al., 2021) are rescaled to fit the expanded context. The model weights are then fine-tuned using long- sequence data to adapt to the rescaled RoPE.
LongRoPE2 is novel approach that extends the effective context window of pre-trained large lan- guage models (LLMs) to the target length, while preserving the performance on the original shorter context window. This is achieved by three con- tributions: (1) a hypothesis that insufficient train- ing in higher RoPE dimensions contributes to the persistent out-of-distribtiion (OOD) issues observed in existing methods; (2) effective RoPE rescaling algorithm that adopts evolution- ary search guided by "needle-driven" perplexity to address the insufficient training problem; (3) a mixed context window training approach that fine-tunes model weights to adopt rescaled RoPE for long-context sequences while preserving the short-context performance with the original RoPE. Extensive experiments on LLaMA3-8B and Phi3- mini-3.8B across various benchmarks validate the hypothesis and demonstrate the effectiveness of LongRoPE2. Remarkably, LongRoPE2 extends LLaMA3-8B to achieve a 128K effective context length while retaining over 98.5% of coort- performance, using only 10B tokens 80x fewer than Meta's 's approach, which fails to reach the target effective context length.
Extending the context window of a pre-trained LLM re- quires addressing the out-of-distribution (OOD) issue in rotary positional embeddings (RoPE). In RoPE, higher- dimensional RoPE embeddings produce OOD values extended token positions due to incomplete rotation periods within the original context window (Liu et al., 2023; Han et al., 2023; Men et al., 2024a). To mitigate this, RoPE rescaling remaps these OOD values into the in-distribution range learned during pre-training. Various methods, such as YaRN (Peng et al., 2023), NTK (LocalI.LaMA, 2023), and LongRoPE (Ding al., 2024), have been proposed to determine appropriate rescaling factors.
Despite attempts to mitigate the OOD issue with RoPE rescaling, context window extension still encounters two major challenges. First, rescaling factors derived from previ- ous methods often fall short of achieving the effective target context length. For example, LLaMA3.1 adopts YaRN to ex- tend its context window 128k; however, its performance RULER (Hsich et al., 2024). benchmark designed to evaluate L.MS' long-context processing capability, deterio- rates significantly when going beyond 64k (Fig. 1). Second, existing approaches to extending an LLM's context window usually lead to a noticeable performance degradation on tasks for the original short context window. As shown in Fig. 2(c), extending Phi3-mini (Abdin et al., 2024) to 128k results in MMLU score drops of 7.56, 4.34, and 3.52 points for YaRN, NTK, and Rong respectively. Restoring short-context performance typically requires costly mid- training strategies, such as multi-stage progressive exten- sion (Dubey et al., 2024) and pre-training data replay (Hu et al., 2024b), which increase both training costs (e.g.. 800B tokens for LLaMA3.1) and system complexity.
1 Introduction
A long context window has become an essential feature of Large Language Models (LLMs) (Achiam et al., 2023;
Dubey al., 2024; Abdin et al., 2024; Zhu al., 2024; Team, 2024). For instance, a 128k context window is now standard in recent LLMs like GPT-4o and LLaMA3.1. Con- text window extension is achieved through mid-training after pre-training, where the rotary positional embeddings
This paper introduces LongRoPE2, novel approach for context extension that enables LLMs to achieve an effec- tive long context window while preserving short-context performance. Our analysis reveals that lower RoPE dimen- sions are sufficiently trained, whereas higher dimensions critical for long-context processing receive inadequate training. This results in shorter effective RoPE rotation
'Equal contribution 'Microsoft "Shanghai Jiao Tong University 'Zhejiang University; Siyuan Wang and Gaokai Zhang did this work during the internship MSRA. Correspondence to: Lyna Zhang <lzhani @ mimm.s
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LongRoPE2: Near-Lossless LLM Context Window Scaling
strate the effectiveness of LongRoPE2. For Phi3-mini- 3.8B and L.LMM338BB, our rescaling factors shift the the- oretical critical dimension from 31 to 25 and from 35 to 30, respectively. By fully resolving RoPE OOD issues, LongRoPF2--etenddel Phi3-mini-3.80 and LLaMA3-8B achieve an effective 128k context window, significantly outperforming baselines on both synthetic and real-world long-context benchmarks. Moreover, with mixed context window training, LongRoPE2 is the only RoPE rescaling method that can retain over 97% of the original short-context performance on standard tasks. Remarkably, LongRoPE2- extended LLaMA3-8B- 128k surpasses Meta's LLaMA3.1- 12-k in long-context performance while maintaining comparable short-context accuracy, all achieved with just 10B training tokens-800 80x fewer than Meta's 800B tokens.
Accuracy on RULER (%)
80
60
LLaMAA-88- R2
Gwen2.5-7B
- Mistral Nemo Base 12B
40
M2257LLM..5-7B
-LLaMA3.1-BB-Mel
LLaMAA-88-GGddimmm
128k Figure 1. LongRooE2-extenddee LLaMA3-8B achieves the best performance at a 128k context length among 10B models.
4k
16k
32k
ranges within the pre-trained context length. We hypoth- esize that this undertraining in higher dimensions is the root cause of their extended rotation periods longer than their theoretical predictions. Consequently, the critical di- mensions shift earlier, leaving existing rescaling methods unable to fully address OOD issues across all dimensions. This hypothesis also explains the empirical observations showing that RoPE requires scaling factors larger than ana- lytically derived values in the higher dimensions for better long-context performance (Gao et al., 2024; Meta, 2024).
2 Context Window Extension and Challenges
2.1 Preliminary
Rotary Position Embedding (RoPE). Transformer models require explicit positional information, often in the form of position embedding, represent the order of input tokens. Our work builds on the Rotary Position Embedding (Su et al., 2021), which is widely used in modern LLMs. Let m E [0, c) be a position index and X1. .... XL E R a sequence vectors, where d is the attention head dimension. Using RoPE, the self-attention first incorporates position information the word embeddings and transforms them into query and key representations:
Building this hypothesis, LongRoPE2 adopts simple yet effective RoPE rescaling algorithm to fully address the OOD issues across all RoPE dimensions. It leverages evolu- tionary search to identify the true critical RoPE dimensions and optimal rescaling factors, guided by more effective "needle-driven" perplexity (PPL) evaluation. Unlike conven- tional PPI., which averages across all tokens, LongRoPE2 focuses exclusively on "needles" specific answer tokens within long documents that require deep contextual under- standing. This ensures accurate evaluation of long-context performance. The search determines the true critical di- mensions and rescaling factors for higher OOD dimensions, while NTK scaling is applied to the well- well-trained lower di- mensions. The rescaling factors yielding the lowest PPL. are selected as the final solution.
qm (x m); fa(mms m) =e.me :mm W ym
(I)
k, = fk (x., n); fx(Xn,, = ine Sem@WWx KX
(2)
where i VT is the imaginary unit. W WA R are projection matrices. Attention weights computed as:
4m Kn
sof softmaa
(3)
To preserve the original short-context performance, Lon- gRoPE2 incorporates mixed context window training, which simultaneously trains a pre-trained context window with the original RoPE and a long-context window with rescaled RoPE. The long-context window is trained by adapting model weights to the rescaled RoPE for long documents packed to the target length. Coneurrently, the short-context window is trained on short documents, also packed to the same target length, using an attention mask to prevent cross- document attention. At inference, original RoPE is used the input is within the short context; otherwise, rescaled RoPE is applied. This method optimizes long-context per- formance without sacrificing short-context performance.
where qm. k, are column vectors, and q kn is their Fu- clidean inner product. Let Re[-] denote the real part of a complex number, the inner product q becomes:
q k.. Re (W,x..)
(W (Wax (
(4)
where (W*x,)* is the complex conjugate of (Wex.). With RoPE, attention becomes a function only dependent on the relative position m n between tokens, rather than their absolute positions. By applying Euler's formula, in can be expressed as trigonometric functions. Then, RoPE encodings can be further written as a block diagonal matrix with entries of the form:
Extensive experiments across various LLM sizes and chal- lenging benchmarks validate our hypothesis and demon-
fg,((n), /cosnO, -sinne,)
sinnd, cosno, ) :0, Abase -25/d (5)
2

---

LongRoPE2: Near-Lossless LLM Context Window Scaling
Real Critical Dimension
LongRoPE
Theoretical Critical Dimenslon
YaRN
NTK
RULER 128k [long context)
MMLU (short regular tasks)
0.50
Scaling factor (log (op-ccale))
0.00
0.2 W
82
10²
Scale Factorm84
Thecredkal O-ikall Dimectioon
0.23
PhiB-Mini
70.78
0.00
775
YaRN
39.37
63.22
Ciiiial CiriialCCiaae
4098
10
NTK
49.37
66.43
Controd Lingih
LongRoPE
53.71
67.26
2018
10
10
25
RoPE dimension index
(c) P Performance of Phi3 m--nii extended to 128k using different extension methods
(a) RoPE OOD
(b) RoPE per-dimensional scaling factor
Figure 2. (a) RoPE OOD (red area) when extending context length from 2k to 4k. (b) Per-dimensional RoPE rescaling factor from
different approaches for extending Phi3-mini from 2k to 128k, all aligning with RoPE OOD theory. (c) Performance of Phi3-mini-1228 after fine-tuning. Existing methods fail achieve an effective 128k context length and show noticeable short-context performance drop.
where 0g is the per-dimensional rotation angle for i = 0,1,... d/2 - 1. Obase is a predefined RoPE base value, typically set to 10000 in pre-training-
When directly extrapolated to 4k, the cosine values between 2k and 4k fall outside the pre-trained range, becoming OOD RoPE values (highlighted in red).
RoPE Per-Dimensional Period. Due the periodicity of cosine and sine functions, RoPE is a periodic function. Specifically, for the 7th RoPE dimension, the corresponding period length T; can be calculated as follows:
Theoretical Critical RoPE dimension. In contrast higher RoPE dimensions, lower dimensions (e.g.. 8th and 16th dimension in Fig. 2(a)) have seen many full periods during pretraining. As a result, there exists a theoretical critical di- mension (TCD) deed that divides RoPE dimensions into two groups: one with multiple full periods within the pre- pre-trained length Tuain (i.e., T₁ < Turain < duca) and another with in- complete periods (i.e., T, 2 Lurins 2 drea). Following (Liu et al., 2023), the critical dimension can be computed as:
T, 2T 0:
(6)
The period length of each dimension is directly determined by its rotary angle 0,. As shown in Fig. 2(a), with a fixed Obose = 10000, 0; decreases as the dimensional index increases, leading to longer periods in higher RoPE dimen- sions. In typical cases, the periods in higher RoPE dimen- sions often exceeds the pre-trained context window size, leading to incomplete periods. For instance, in Phi3-mini, the pre-trained context window size is 2048, while the pe- riod length of the highest dimension (i.e., the 48th cosine dimension) is 51861, covering less than 4% of a full period.
d dea = 2 logo
Lirain
2x
As shown in Fig.2(a),, for Phi3 thii--iin(AAbdin al., 2024) with d=96, a base 100000 and Larin = 2048, the critical di- mension is 62, corresponding to the 31s cosine dimension. Unless otherwise specified, we focus on the cosine dimen- sions of RoPE (i.c., = 0, I...., d/2 1) for simplicity.
2.2 RoPE Rescaling Theory
RoPE 0OD theory. To address the RoPE OOD issue in long-context extension, a straightforward approach to rescale the per-dimensional rotation angle and ensure higher RoPE-OOD dimensions remain within the pretrained RoPE range. This forms the widely accepted RoPE OOD theory (Liu et al., 2023; Chen et al., 2023; Men et al., 2024a).
Despite its effectiveness, RoPE, like other position encod- ings, faces challenges in context length extrapolation. In particular, when input sequence length exceeds the prede- fined context window, the perplexity can shoot up to levels comparable to completely untrained models (i.e., > 103).
RoPE OOD. Direct length extrapolation fails because longer sequences introduce untrained token positions, lead- ing to out-of-distribution (OOD) positional values in RoPE. As shown in Fig. 2(a), the periods in high RoPE dimen- sions exceed the original context window size Luain- Con- sequently, for these dimensions, the model does not see a full rotation period during pre-training, resulting in new untrained RoPE values at extended token positions. For instance, in Fig. 2(a), the 40th cosine dimension does not complete full period within the pre- pre-tr length T =2k.
Formally, let the target context window size be L and A be the rescaling factor for the 2hh RoPE dimension. The rescaled per-dimensional rotation angle is then given by:
0, de x Obase 2
(8)
To avoid OOD, the new rescaled periods of higher RoPE dimensions (f., > dea) must remain within the pretrained
3

---

LongRoPE2: Near-Lossless LLM Context Window Scaling
range, leading to the following constraint:
theoretical analysis, LongRoPE employs a PPL-guided evo- lutionary search to find the per-dimensional scale factor To leverage NTK theory, it enforces a monotonically non non-decreasing scaling factor constraint during the search.
L
Lirain
LÓ,
Lirin
for i i2 died
x;
T;
T;
2T
2T
L
2
for
i i2 dhed
(10)
2.4 Challenges
Luain
Taain is the context window extension ratio. The RoPE OOD theory establishes this ratio as the lower bound for scaling factors in higher RoPE dimensions beyond dred-
Specifically, L
RoPE OOD theory are insufficient. Fig. 2(b) compares scale factor distributions for extending Phi3-mini from 2k to 128k. NTK, YaRN and LongRoPE all align the RoPE OOD with X 2 64 for i > dxed but yielding varied performance (Fig. 2(c)). NTK and LongRoPE outperforms YaRN on both short- and long-context tasks. We highlight two observa- tions: (1) The theoretical lower bound, Luan is often subop- timal. Beyond dimension deed = 31. YaRN strictly adheres to this bound (T (64) but NTK and LongRoPE use larger values achieve much better performance. (2) Beyond deed, larger scale factors don't 't always improve long-contex, performance. For example, in dimensions 31-48, NTK uses much larger scale factors than LongRoPE, yet LongRoPE achieves better performance. These findings align with prior works (Meta, 2024; Men et al., 2024a; Wang et al., 2024), where marginally larger scale factors than the extension ratio empirically improve performance.
2.3 Review of Prior RoPE Rescaling Approaches
Building on the RoPE OOD theory, various RoPE rescal- ing methods have been proposed for LLM context win- dow extension (Chen et al., 2023; Han et al., 2023; Men et al., 2024b; Yang et al., 2024b). Prominent approaches, including PI, NTK, YaRN and LongRoPE, have been widely adopted to enable long context in open-source LLMs (Yang et al., 2024a; Dubey et al., 2024; Abdin et al., 2024).
L
PI introduces linear positional interpolation, where all the RoPE dimensions use the same scale factor of X Toean Despite its simplicity, this uniform scaling "crowds" the positional information, making it difficult for the model to distinguish closely positioned tokens.
This raises the fundamental question: In RoPE OOD theory, if RoPE PRoPEperiods beyond critical dimension can address OOD with X; Inga'' to better performance?
NTK Scaling approaches RoPE from an information encoding perspective, applying the Neural Tangle Kernel (NTK) theory (Jacot et al., 2018; Tancik et al., 2020). The core idea is that neural networks are difficult learn high- frequency features (low RoPE dimensions), and large scal- ing factor can affect these high-frequency positional infor- mation, leading to the loss of crucial details needed to dif- ferentiate similar closely positioned tokens.
I., why do slightly larger scaling factors lead
Short performance drop. A persistent challenge in long context extension is performance degradation on original short window, which poses significant obstacle in practical LLM development. A common solution is progressively extension using large-scale training data (Dubey et al., 2024; Hu et al., 2024b). For example, I. 1.aMA3.1 (Dubey et al., 2024) adopts a SIX-stage extension process requiring 800B tokens to extend from 8k to 128k, greatly increasing training complexity and costs. Though LongRoPE introduces a training-free short scaling factor, it fails to fully address the performance drop (Figure 2(c)). As result, bridging this gap remains an unresolved challenge.
As a result, -bs-d methods suggest increasing the original RoPE base value Obase to a larger base 0yck. Several methods (Lo,a 2023; Men et al., 2024b; Liu et al., 2023) have been proposed to determine this new base value. However, some fail to align with RoPE OOD theory. For instance, (LocaILLaMA, 2023) use X; = (d/(22)) leading to insufficient interpolation and increased PPL before the target length. The approach in (Liuu et al., 2023), which calculates met based on the theoretical critical dimension, the most widely adopted -TK-based method. Specifically, Onck 2 0 log Lyuia yielding A; 2 The > drea). Unless stated otherwise, "NTK" in this work refers to this approach.
3 LongRoPE2 Methodology
3.1 New RoPE OOD Hypothesis
The empirical RoPE periods in higher dimensions are longer than theoretical values, limiting current methods to fully address RoPE OOD. In Sec. 2, we observe that RoPE scale factors slightly exceeding the theoretical lower bound beyond the critical dimension died yield improved long-context performance. We attribute this to insufficient training in higher dimensions, which extends rotation pe- riods and reduces the critical dimension index (Fig. 2(a)) relative to the theoretical expectations.
YaRN divides RoPE dimensions into three groups as shown in Fig. 2(b). For lower dimensions with high frequencies, YaRN proposes interpolation, setting X; = 1 to better preserve high-frequency positional information compared to NTK. For high dimensions, YaRN adopt Pl and set X;- = L. For dimensions that fall in-between use a linearly Dmm
increasing scale factor.
LongRoPE. Unlike other extension methods relying on

---

LongRoPE2: Near-Lossless LLM Context Window Scaling
Training S ample:
Algorithm 1 Initialization with theoretical periods
0
24
Input: theta base Ouasc; RoPE dim d. pre prettrined context window
size Luain. target length L; theoretical critical dimension diod
Full Perio-2/0
Effective RoPE range
1: Pu = [0] * 22/d
Horl
0.99
1 {Compute the dim with theoreti- 2: dea== loga,.ce 2xx10
0.95
Thooretical Period
cal 10 periods.)
-1
0.97
Real Effective Pertod
finclude smaller indices as candidate drea}
10
20
30
0
1000
1500
2000
4: s=randint( 424 2x Traai
3: for int drod=dle 10 to died do
(a) 8 Dim
48 Dim
Tupii
Figure 3. Sequence length required to span the theoretical period during Phi3-mini pre-training for different RoPE dimensions. In- sufficient training in higher RoPE dimensions leads to shorter effective RoPF ranges and longer actual periods.
5: Addred
6: Oa1o
ax0 ax0 (2x
7: A/0: : drea]= compute rescaling factors using NTK died
8: add into Po:
As illustrated in Fig. 3(a), lower RoPE dimensions (with shorter periods) receive repeated full- full-period training cycles within a single corpus. For example, in Phi3-mini, the 8th di- mension has a short period of 24, requiring only m-n = 24 tokens for a full cycle. A 2048-token training sample thus covers this dimension thousands of times, ensuring suffi- cient training. In contrast, higher RoPE dimensions, with period exceeding the pre-trained context window, receive far less training. For example, the 48th dimension spans only 44 of its cosine period within 2048-token sequence (Fig. 3(b)), resulting in the theoretical incomplete period being covered just once.
9: end for
10: Return Po:
the search, we introduce two key innovations.
Synthetic needle data to guide the search. Naively using PPL.-guided search can easily result in suboptimal rescaling factors. First, long sequences often contain irrelevant or low-dependency tokens, reducing the effective maximum token dependency. For instance, predicting the final token in a 128k-token book may not require the context of the first token. Second, standard PPL, by averaging over all token equally. fails to effectively capture the long-context abilities (Hu et al., 2024a; Fang et al., 2024) and can be dominated by irrelevant tokens, obscuring key answer to- kens. As a result, the rescaling factors that minimize PPL often fail to achieve the target context window size.
A deeper challenge arises after self-attention: these incom- plete RoPE periods in high dimensions exhibit reduced ef- fective ranges (Fig. 3(b)), stretching practical period beyond theoretical values. As shown in Eq. 3, RoPE positional in- formation is incorporated via self-attention, where the max relative token distance determines the practical RoPE range. As real-world data rarely contains long-range dependencies (e.g., distances of 2048 tokens), higher RoPE dimensions tend to be under-trained, amplifying period discrepancies.
To address this, we introduce a deedle-driven PPL evalua- tion. Instead of using real-world long documents, we synthe- size long data with controlled token dependency distances. Inspired by needle retrieval benchmarks for long-context evaluation (Hsieh et al., 2024; Li et al., 2024a), we randomly sample 10 books from the PG19 validation set. At the start of each sample, we insert a "needle" (a specific piece of text as shown in Appendix B). and at the end, we ask the model to retrieve this needle. We then compute the perplexity of only the retrieved needle tokens. The needle-bar based PPL evaluates how well the model, with the rescaled RoPE, can understand the entire context and retrieve the distant needle.
This trreingg in higher RoPE dimensions explains why larger scaling factors improve long-context perfor- mance. We formalize this insight as:
Hypothesis. Insufficient training in higher RoPE dimen- sions extends empirical rotation periods beyond the theoret- ical 2n This discrepancy necessitates larger scale factors to mitigate RoPE OOD and lowering the critical dimension index drod below its theoretical daed-
0.
Critical dimension-aware scale factor search. With the synthetic needle-driven PPL evaluation, we run a simple evolutionary search to identify the real critical dimension drod and the optimal rescaling factors. For search efficiency, we restrict the search to dimensions i 2 dred, while ap- plying NTK-aaare scaling lower dimensions (i < dred) using the adjusted base value derived from dred-
3.2 RoPE Rescaling Factor Search
Since the theoretical RoPE OOD theory cannot fully address OOD issues, we use a search-basee approach to identify the practical true critical dimension and optimal rescaled RoPE Inspired by LongRoPE, we search for scaling factors, apply them to the pre-trained L.L via rescaled RoPE, and com- pute perplexity (PPL) fixed samples at target context length (e.g., 128k). The factors that minimize PPL are cho- sen for best preserving pre-trained RoPE information while addressing OOD. Given that the approach relies entirely on
The search begins by initializing dred and rescaling factors, as detailed in Algorithm 1. Based on our hypothesis, smaller indices are considered potential dred- with candidates rang- ing from dio where the theoretical RoPE period spans 10 periods in the pre-training window, and deed. For each can-
5

---

LongRoPE2: Near-Lossless LLM Context Window Scaling
Algorithm 2 Critical dimension aware mutation
Original RoPE
Rescaled RoP E with long factor
Short data concatenoted to
Long data concalendiestuuncaeedd to
Input: population mutation probability pt synthetic long data X
1: Tap-k= = Update_ _Topk ( P);
2: SP=[, Iume. 2x L. 1ssassssshh space}
Tua.
0
for A in Top-k do
4: Aright= A(drad --11
5: Braak:nnutatioowwiii wwtp://whhhhhoooo.oommmm mmono_constraint (Arights P. SP) {mutate
scale factors beyond (d
O
5 6
4
3
2
0
6: A[drad 1| Apight
8
5
4
3
2
0
Odred Aright (0) (0)x 22x2dr/aa/ {update theta base Odved-)
8
7 6
5
4
0
8
Position ids of
Posittin ids of
2/0 : 1/= compute rescaling factors using NTK Odred {update
(a) Short context window training
(b) Long context window training
dims before d
Figure 5. Mixed context window training to improve both short and long context capabilities.
9:
Compute _PPL (LLM, X, X); add X into P:
10:
end for
11: Update with Top-k; Return the latest population P:
Table I. Mid-training data mix.
didate, rescaling factors above L are randomly sampled for dimension i 2 dred to address RoPF OOD value, while NTK scaling is applied to dimensions < dyed-
Tran
Short Context Window
Long Context Window
< Lerain
Livain 110k
100k-200k
Tokens
3B
3B
4B
We iteratively sample and mutate rescaling factors until reaching a population size N. Using the needle-driven syn- thesis method, we generate L-token documents and compute PPL for each candidate by applying the rescaling factors to the LLM and evaluating the input X.
model weights have not been trained with the rescaled RoPE, leading to poor performance on real-world long-context tasks. Second, extending context window size often de- grades performance on original short-context tasks (Ding et al., 2024; Hu et al., 2024b), making it challenging to balance long- and short-context capabilities.
The population is updated through standard evolution search. Algorithm 2 shows the mutation process. For each sampled scaling factor, we split RoPE dimensions at dyed. The higher group (i 2 dred) performs mutation with probability p under the monotonic non-decreasing constraint: A; S A The theta base for dred is updated after mutation, and NTK scaling is applied to rescale factors in the lower group.
To address these challenges, we introduce a novel mixed con- text window training approach that achieve both long- and short-context superior performance without adding system- level training complexity. Specifically, short-context train- ing reuses the original RoPE and fine-tunes on short se- quences, preserving pre-trained performance. Long-context training applies the rescaled RoPE and fine-tunes on long sequences, enabling effective long-context understanding.
Fig. 4 shows the final scaling factors identified by Lon- gRoPE2 for Phi3-mini and LLaMA3-8B under a 128k con- text. The practical critical dimensions (drea) are shifted ear- lier to 25 and 30, compared the theoretical values died of 31 and 35, respectively. The scaling factors for RoPE OOD dimensions slightly larger than PoP and notably smaller than NTK.
Fig. 5 illustrates this process. For target context win- dow size of L=128k, we sample short sequences (S Limin) and long sequences (8k-200k), chunked into 128k seg- ments with BOS and EOS tokens. For segments labeled as short windows, the original RoPE is used with attention masks to prevent self-attention across different documents as shown in Fig. 5(a). For long-context segments, we apply the rescaled RoPE for full attention within the 128k seg- ments (Fig. 5(b)). This design contrasts with prior "mixed training" methods such as LLaMA-3.1 (Dubey et al., 2024) and LongAlign (Bai et al., 2024), which use a single long RoPE scaling factor and disallow cross- cross-document attention. In contrast, our approach employs dual RoPE scaling and allows cross-document attention for long contexts. More details can be found in Appendix A.
YaRN
NTK
LongRoPE
Long
Scaling tactor (leg scaled
10+
1
Thesreciical CD
16
Theorociical CD
10'
10'
Real CD
Real
10
10"
0
29 31
3035
63
RoPE dimension index
RoPE dimension index
[a] P Phi3-mini (2k-=128k)
(b)LLaMA3-8B (8k->128k)
Figure 4. Scale factors across different RoPE rescaling approaches.
4 Experiments
4.1 Setup
3.3 Mixed Context Window Training
Evaluation LLMs and Tasks. We apply LongRoPE2 to I.AMM3-8B and Phi3-mini (3.8B). Phi3-mini, with its lim- ited capabilities, serves as a rigorous testbed for evaluating
We then apply the optimal rescaling factors to RoPE on the pre-trained LLM, but two critical challenges remains for ef- fective long-context LLM deployment. First, the pre-trained
6

---

LongRoPE2: Near-Lossless LLM Context Window Scaling
Phi3.mi (3 88) NTK.120K
Fhi3.min minimini
Lo
Table Comparison with prior SOTA RoPE rescaling methods on
0.8
RULER Benchmark. We report the average score across 13 tasks.
59
64
05
Method
4k
8k
16k
32k
64k
128k
Base
Model:
Phi3-mini
(3.8B)
70
02
YaRN
85.74
78.68
75.97
65.22
52.16
39.37
NTK
91.34
87.02
80.57
72.81
61.91
49.37
LLaMA3 OB 12tk (YaRN)
LLaMA3-58 Lon2RRPP2-1284
1.2
LongRoPE
88.40
83,23
79.46
71.20
64,63
53.71
Depth Perceet (%)
LongRoPE2
90.41
87.22
83,33
76.51
65.37
58.81
688
Basc Model:
LLaMA338B
YaRN
91.86
87.87
84.67
68 80
62.51
49.39
07
NTK
94,38
92.64
91.93
87,33
79.26
73.19
130
0.0
LongRoPE
94.60
92.70
91.01
86.60
81.23
73,40
107x
LongRoPE2
94.61
93.68
92.31
90.49
85.62
82.03
Context Lonoth
Figure 6. LongRoPE2 (right) delivers near-perfect lossless perfor- mance in the "Needle in a Haystack" pressure test.
RoPE rescaling methods. Performance is evaluated across three dimensions: (1) long-context stress tests, including RULER (Hsieh et al., 2024) and Needle in Haystack (Kam- radt, 2023); (2) real-world long-context benchmarks includ- ing LOFT (Lee et al., 2024a), InfiniteBench (Zhang et al., 2024a), and LongBench (Bai et al., 2023). Notably, since our method extends a pre-trained LLM without post-training, we prioritize sub-tasks aligned with completion-basee tasks and QA tasks with few-shot examples; (3) standard bench- marks within a 4096-token context.
icantly at longer contexts. For example, LongRoPE, the prior best, drops from 81.23 (64k) to 73.40 at 128k. For Phi3-mini-3.8BD LongRoPE2 shows even greater advan- tages, overcoming the challenges of the smaller model's weaker capabilities. NTK performs well below 32k and declines sharply beyond, while LongRoPE underperforms at shorter contexts. In contrast, LongRoPE2 consistently enhances performance across all lengths. Notably, the 128k average score of 58.81 is skewed by tasks with low scores on smaller LLMs, such as CWE, which achieves only 1 1% ac- curacy. Detailed per-task score is available in Appendix A.
Mid-training. Our method can potentially support million- level context length, but due to resources constraint, we extend the two models to 128k context window and mid- train on 64 A100 GPUs using a 10B-token dataset. Fol- lowing the per-source upsampling from (Fu et al., 2024), sample 4.5B, 2.5B, and 2B tokens from RedPajama- v1 (Computer, 2023), RedPajama-v2 (Weber et al., 2024), and StarCoder (Li et al., 2023), covering 8k-200k sequence lengths. For short context windows, we sample 1B tokens from Fineweb-Edu (Lozhkov et al., 2024). Table shows the token distribution by sequence length. We train for epoch with a global batch size of 64. The initial learning rate of 2 with cosine learning rate scheduler.
Needle in a Haystack pressure tests. We evaluate Lon- gRoPE2 using the popular long-contex: pressure test, Nee- dle in a Haystack, which measures a model's ability to retrieve "needles" from long documents at varying depths. We run 10 times at the same depth and length. As shown in Fig. 6, LongRoPE2 achieves hear-perfect accuracy across all evaluation lengths within the 128k context window. In contrast, methods like NTK often fail at longer contexts, and LLaMA3. 1-B extended by YaRN, despite being tine-tuned on 800B tokens, fails beyond 100k. These results highlight LongRoPE2's ' robust long-context modeling capabilities.
We compare with state-of-the-art RoPE rescal- ing methods, including YaRN, NTK, and LongRoPE. All baselines use the same mid -oa-iirining procedure for fairness.
Baselines.
Long-context performance on real-world benchmarks. Beyond synthetic tasks, we evaluate real-world benchmarks: LOFT (7 retrieval tasks including argumentative retrieval, fact-checking, web search, multi-hop reasoning QA, etc), InfiniteBench (key-value retrieval and multi-choice QA), and LongBench (in-context learning and code completion). Note that our models are evaluated without post-training so scores are lower than post-training results. As shown in Table 3, LongRoPE2 consistently improves performance across all benchmarks, demonstrating strong generalization to practical scenarios. In contrast, YaRN and NTK perform notably worse, particularly on the small Phi3-mini-3.8B.
4.2 Main Results
We present the main results of LongRoPEE2-extndde Phi3- mini-3.8B-1288. and LLaMA3-8B- 128k, comparing them with models using other STOA RoPE rescaling methods.
Long-context performance on RULER benchmark. Ta- ble 2 compares performance on RULER, which consists of 13 synthetic tasks. Across Phi3-mini-3..m and L.LMM3-8B,, LongRoPE2 consistently outperforms prior methods, achiev- ing superior results across all evaluation lengths within the 128k window. On LLaMA3-SB, LongRoPE2 achieves effective 128k context window, maintaining a strong score of 82.03 at 128k, while previous methods degrade signif-
Standard benchmarks at original context window. RoPE- based context extension typically sacrifices short-context performance. As Table 4 shows, prior methods like YaRN, NTK, and LongRoPE exhibit notable degradation. For ex-
7

---

LongRoPE2: Near-Lossless LLM Context Window Scaling
Table 3. Long context performance comparison under different extension methods on real-world benchmarks
Method
LOFT
InfiniteBench- LongBeach
MS MACRO
KV retrieval
Avg.
ArguAna
FEVER HotPorQA
NQ
Quera
SciPact
Avg.
En.MC
TrivinQA
TREC
LCC
RepoBench-P
Base model:
Phi3-mini (1.8B)
YaRN
5.86
4.0
4.0
0
8.0
12.0
1.0
12.0
50.96
5.8
31.44
61.00
63.98
59.23
NTK
7:57
0
21.0
0
6.0
13.0
4,0
9.0
52.31
5.1
37.55
84.01
65.00
62.36
59.82
LongRoPE
21.14
5.0
64.0
3.0
35.0
8.0
16.0
50.67
5.6
86.47
62.50
55.25
58.43
LangRoPF2
23.00
5.0
70.0
4.0
19.0
39.0
10.0
14.0
55.23
12.0
42.36
87.27
67.00
62.67
60.10
Base model:
LLaMA338B
YaRN
26.14
7.0
62.0
15.0
21.0
43.0
23.0
12.0
51.81
2.2
30.57
88.97
73.50
65.40
62.21
NTK
67.14
22.0
96.0
53.0
75.0
89.0
71.0
64.0
67.98
66.0
42.79
90.87
74.00
68.67
65.55
LongRoPE
60.85
22.0
96.0
25.0
57.0
90.0
74.0
62.0
70.39
74.0
45.85
89.99
76.00
69.13
67.38
LongRoPE2
74.28
28.0
96.0
70.0
80.0
94,0
79.0
73.0
73.37
88.0
46.72
91.13
76.50
70.47
67.39
Table Comparison of long-cone long-context LLMs with original Phi3- mini and MA3-88 on regular short benchmarks.
Table Ablation study on needle-PPL guided search.
Search Metric
8k
16k
32k
64k
128k
(a) Phi3-mini (3.8B) with 128k context window
Base Model: Phi3-mini
(3.8B)
PG19-128K PPL
91.16
87.93
83.05
75.27
62.72
50.23
Model
MMLU MMLU-Pro HellsSwag TruthfuQQA GSMSK
NGed-Needle 128k (ours)
87.22
76.51
65.37
58.81
Original Phi3-mini (2k)
63.2
70.78
41.17
77.96
47.82
78.54
YaRN
53.6
63.22
30.95
75.27
42.19
57.39
Base Model:
1.1 aL
NTK
36.09
13.34
PG19-128K PPL
94.46
93.36
91.67
90.28
84.55
78.68
LongRoPE
58.5
67.26
36.28
75.73
46.26
67.17
PG19-Needle 128k PPL (ours)
94.61
93.68
92.31
90.49
85.62
82.03
LongRoPE2
61.7
70.04
40.30
77.07
47.61
73,62
M.L.
128k context window
Tahle 7. Ablation study on mixed context window training.
Original LLaMA3-8B
11aMA3,1-8
$7.2
66.33
35.87
36,79
81.71
45.17
56.18
Method
MMLU
MMLU
GSMBK
4k
8k
16k
32k
64k
128k
YaRN
$2.1
62.25
31 888
81.25
42.61
42.45
Pio
NTK*
54.0
63.84
34.14
82.11
43.45
46.92
Base Midel:
Phi3 June
LosgRoPE
33.74
43.65
LongRoPF22
70.07
40.30
73.62
90.41 86.87 83.33
76.51
68.37
58.81
LongRoPF:2
55.7
65.01
34.61
81.69
46.17
50.80
LongRoPD2/
66.56
34,86
64.67
90.55 85.77 81.08
73.31
63.75
56.22
Basc Model: LLaMA3-88
Table 5. Ablation study on real critical dimension.
LongRoPE2
34.61
50,80
94.61
93.68 92.31 90.49
85.62 82.03
LongRoPD2/ wo. 64.57
33,83
48.37
94.67 93.15 91.24 89.38 83.53 80.18
Regular short tasks
RULER
Method
MMLU
NTK-red variants (see Fig. 9 in Appendix A). As shown in Table 5, correcting droa improves long-context performance for both methods, revealing the inadequacy of theoretical critical dimensions in fully addressing RoPE OOD issues. However, correcting the critical dimension alone does not ensure optimal results. By further optimizing scaling fac- tors, LongRoPE2 consistently outperforms YaRN-red and
MMLU
GSM8K
4k
Sk
16k
32k
64k
128k
Pro
Model: Phi3
(3.8B)
LongRoPE
70.07
40.30
73.62
90.41
87.22
8333
76.51
65.37
58.81
YaRN
63.22
30.95
57.39
85.74
78.68
75.97
65.22
52.16
39.37
YaRN-red
62.30
30.24
56.48
86.56
77.66
74.48
67.73
52,73
44.39
NTK
66.43
36.09
63,99
91.34
87.02
80.57
72,81
61.91
49.37
NTK-red
65.31
35.09
59.29
90.51
85.32
81.50
73,89
63.59
54.42
Base Model:
L.MMM3-8B
LongRoPE
65.01
34.61
50.80
94.61
93.68
92.31 90.49
85.62
82.03
NTK- on both short- and long-context benchmarks.
YaRN
31.88
42.45
91.86
87.87
84.67
68.80
49.39
YaRN-red
64.30
33.17
50.34
94,22
92.02
89.20
82.56
76.37
71.46
The effectiveness of need-PPI. guided search. Lon- gRoPE2 identifies the true critical dimension and scaling factors through needle-PPL. PPL-guided evolutionary search, which minimizes interference from irrelevant tokens to effec- tively capture the rescaled RoPE's long-context capabilities. To validate its effectiveness, we use 10 pure PG19 docu- ments as a baseline, identical to those used for generating our needle-data, applying the same search and mid-training process. Table compares the RULER scores for Phi3-mini- 3.8B-128k and LLLMA3-8B- using scaling factors from two PPL-guided searches. The results show that naive PPL- search fails to ensure effective rescaling factors, as it struggles to identify the correct critical dimension and tends to yield slightly smaller scaling factors.
NTK
63.84
34.14
46.92
94.38
92.64
91.93
87.33
79.26
73.19
NTK-xed
64.70
34.23
45.87
94,39
92.35 91.43 88.82
83.22
77.25
ample. YaRN and NTK show performance drop of -152%
and .9.3% oh Phi3-mini, with declines of -21.15 and -1.555 absolute points on GSM8K. In contrast, LongRoPE2 re- tains 97.6% and 98.6% o0f the pre-trained performance on Phi3-mini-3.88 and LLaMA3-SB, establishing it as the first lossless extension method that preserves core capabilities.
4.3 Ablation Study
The effectiveness of real critical dimension dred- A key factor in LongRoPE2's superior long-come mongccontxtt performance is its full resolution of RoPE OOD values across all dimen- sions. To validate this, we extend our experiments beyond LongRoPE2 by applying our identified practical critical di- mension drad to YaRN and NTK, yielding YaRN-red and
The effectiveness of mixed context window training. To ablate its effectiveness, we disable mixed context window training in LongRoPE2 and instead follow conventional mid-
8

---

LongRoPE2: Near-Lossless LLM Context Window Scaling
training with single rescaled RoPE. As shown in Table 7, removing mixed context window training results in a signif- icant drop in performance on regular short-context tasks, as expected. Interestingly, mixed context window training not only preserves short performance but also improves long- context performance (8k-128k). This may be attributed to the preservation of pre-trained RoPE for shorter contexts, allowing long-context training to focus more effectively on adapting the new introduced token positions.
ing of higher RoPE dimensions key limitation in han- dling OOD positional values- Longesooo uses evolution- ary search-guided rescaling and mixed context window train- ing to achieve 128k effective context length with just 10B tokens, retaining 97.6% of the original short-context per- formance. Extensive experiments on on LLaMA3-8D and Phi3-mini-3.83 demonstrates the superiority over prior art approaches. Future work will explore scaling LongRoPE2 toward fully lossless and infinite context window extension.
5 Related Works
Impact Statement
In addition to methods based on RoPE rescaling, this section discusses related works of other approaches.
This work advances the field of Machine Learning by en- abling LLMs to process longer contexts effectively. Lon- gRoPE2 enhances LLM capabilities for tasks like document summarization and scientific research. There are many po- tential societal consequences of our work, none of which we feel must be specifically highlighted here.
RAG and Agent-based extension. Retrieval- -uggevel--- Generation (RAG) approaches incorporate an external mem- ory module to store and manage long past context, coupled with dynamic retrieval mechanisms to fetch task-relevant documents during inference (Jeong et al., 2024; Chan et al., 2024; Dong et al., 2024; Gutiérrez et al., 2025; Luo et al., 2024). Agent-based methods, meanwhile, decompose long- context processing into iterative planning, summarization, and retrieval tasks, often employing multi-agent workflows: individual agents extract information from text segments, which are aggregated to bypass fixed context limits (Zhang et al., 2024b; Li et al., 2024b; Lee et al., 2024b), while others integrate specialized architectures (e.g.. hierarchical attention) for direct long-text handling (Gur et al., 2024). Both directions rerlyyin on external modules or multi-step decomposition-are complementary to our method.
References
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan,
A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M., Cai, Q., Chaudhary, V., Chen, D., Chen, D., Chen, W., Chen, Y.-C., Chen, Y.-L., Cheng, H., Chopra, P., Dai, X., Dixon, M., Eldan, R., Fragoso, V., Gao, J., Gao, M., Gao, M., Garg. A., Giorno, A. D., Goswami, A., Gunasekar, S., Haider, E., Hao, J., Hewett, R. J., Hu, W., Huynh, Iter, D., Jacoss,, S. A., Javaheripi. M., Jin, X., Karampatziakis, N., Kauffimann, P., Khademi, M., Kim, D., Kim, Y. J., Kurilenko, L.., Lee, R., Lee, Y. T., Li, Y., Li, Y., Liang, C., Liden, L., Lin, X., Lin, Z., Liu, C., Liu, L.,, Liu, M., Liu, W., Liu, X., Luo, C., Madan, P., Mahmoudzadeh, A., Majercak, D., Mazzola, M. Mendes, C. C. T., Mitra, A., Modi, H., Nguyen, A., Norick, B., Patra, B., Perez- Becker. D., Portet, T., Pryzant, R., Qin, H.,, Radmilac, M., Ren, L., de Rosa, G., Rosset, C., Roy, S., Ruwase, O., Saarikivi, O., Saied, A., Salim, A., Santacroce, M., Shah, S., Shang, N., Sharma, H., Shen, Y. Shukla, S., Song, X., Tanaka, M., Tupini, A., Vaddamanu, P., Wang, C., Wang, G., Wang. L., Wang, S., Wang, X., Wang, Y., Ward, R., Wen, W. Witte, P., Wu, H., Wu, X., Wyatt, M., Xiao, B., Xu, C., Xu, J., Xu, W., Xue, J., Yadav, S., Yang, F., Yang. J., Yang, Y., Yang, Z., Yu, D., Yuan, L.., Zhang, C., Zhang, C., Zhang, J., Zhang, L. L.. Zhang, Y., Zhang, Y., Zhang, Y., and Zhou, X. Phi-3 technical report: highly capable language model locally on your phone, 2024. URL https: htttps://arxii absg 2404../2244..40 14219.
Efficient long-context modeling. Attention computation and memory costs grow quadratically with context length, prompting research into reducing these challenges through improved attention mechanisms and innovative model struc- tures. Many methods leverage the sparsity of standard at- tention, reducing computation by focusing on local and auxiliary regions (Child et al., 2019; Beltagy et al., 2020; Zaheer et al., 2020; Guo al., 2022), while others extend context length using fine- fine-graines grained sparsity (Ding al., 2023) or chunked attention (An et al., 2024). Linear attention ap- proaches further lower complexity while achieving compara- ble performance, with additional optimization for hardware efficiency (Katharopoulos et al., 2020; Yang et al., 2024c). State-space models (SSMs) offer linear complexity for se- quence modeling (Gu & Dao, 2024; Yu et al., 2024), and hybrid tramsformer-SSM architectures enhance foundational model capabilities (Lieber et al., 2024; Ren et al., 2024). Most of these approaches build upon RoPE, making them complementary to our approach.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report, 2023.
6 Conclusion
We present Lon2RoPF22, method for near-lossless L.LM context window extension. By addressing insufficient train-
An, C., Huang, F., Zhang, J., Gong. S., Qiu, X., Zhou, C., and Kong, L. Training-free long-context scaling

---

LongRoPE2: Near-Lossiess LLM Context Window Scaling
of large language models, 2024. URL https:///axxi org/ hg/abs/2202. 17463.
Fang, L.,, Wang, Y., Liu, Z., Zhang, C., Jegelka, S., Gao, J., Ding, B., and Wang, Y. What is wrong with perplexity for long-context language modeling? arXiv preprint arXiv:2410.3377, 2024.
Bai, Y. Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z.,, Liu, X., Zeng, A., Hou, L., et al. Longbench: A bilingual, multitask benchmark for long context under- standing. arXiv preprint arXiv:2308. 14508, 2023.
Fu, Y., Panda, R., Niu, X., Yue, X., Hajishirzi, H., Kim, Y., and Peng, H. Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402.10177 2024.
Bai, Y., Lv, X., Zhang, J., He. Y., Qi. J., Hou, L.,, Tang, J., Dong, Y., and Li, Longalign: A recipe for long con- text alignment large language models. arXiv preprint arXiv:2401. 18058, 2024.
Gao, T., Wettig, A., Yen, H., and Chen, D. How to train long- context language models (effectively). arXiv preprint arXiv:2410.02660, 2024.
Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer, 2020. URL https//ss///ak org/abs/2004 05150.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL https://arxiv.
org/abs/22 htg//abs/331... 00752.
Chan, C.-M., Xu, C., Yuan, R., Luo, H., Xue, W., Guo, Y., and Fu, J. Rq-rag: Learning to refine queries for retrieval augmented generation, 2024. URL https:// arxiv.org/abe hbrrgabb/24044 00610.
Guo, M., Ainslie, J., Uthus, D., Ontanon, S., Ni, J., Sung, Y.-H., and Yang, Y. Longt5: Efficient text-to-text trans- former for long sequences, 2022. URL https://arxiv org/ /bs//////// 07916.
Chen, S., Wong, S., Chen, L., and Tian, Y. Extending context window large language models via positional interpolation. arXiv preprint 2tti::200..1599 2023.
Gur, I., Furuta, H., Huang, A., Safdari, M., Matsuo, Y,, Eck, D., and Faust, A. real-world webagent with planning, long context understanding, and program synthesis, 2024.
Child, R., Gray, S., Radford, A., and Sutskever, I. Generat- ing long sequences with sparse transformers, 2019. URL https:/ ar ar www.wriwww...................oor orggabs/////ooooooo/oooo.e 904. 10509.
URL https:://aaae org/ org/abs///2 ab///bb/..... 12856.
Gutiérrez, B. J., Shu, Y., Gu, Y., Yasunaga, M., and Su, Y. Hipporag: Neurobiologically inspired long-term memory for large language models, 2025. URL https://arxiv org/ abs 14831.
Computer, T. Redpajama: An open source recipe to reproduce llama training dataset, 2023. URL. https: //gi //github.ccmm/tgeee httpeettheecomppre//RRrPPaa. Data.
Han, C., Wang. Q., Xiong, W., Chen, Y., Ji, H., and Wang, S. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308. 16137, 2023.
Dao, FlashAttention-2. 2: Faster attention with better paral- lelism and work partitioning. 2023.
Ding, J., Ma, S., Dong, L.,, Zhang, X., Huang, S., Wang, W., Zheng, N., and Wei, Longnet: Scaling transformers to 1,000,000,000 tokens, 2023. URL https::/arxxi..orr 02486.
Hsich, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg. B. Ruler: What's the real context size of your long-context language models? 2024.
Ding, Y., Zhang, L. I.., Zhang, C., Xu, Y., Shang, N., Xu, J. Yang, F, and Yang, M. Longrope: Extending 11m context window beyond 2 million tokens. arXiv preprint arXiv:2402. 13753, 2024.
Hu, Y., Huang, Q., Tao, M., Zhang, C., and Feng, Y. Can perplexity reflect large language model's s ability in long text understanding? arXiv preprint arXiv:2405. 06155 2024a.
Dong, K., Deik, D. G. X., Lee, Y. Q., Zhang, H., Li, X., Zhang, C., and Liu, Y. Multi-view content-aware indexing for long document retrieval, 2024. URL https: https://arxiv. org/abs/ 2404. 15103.......
Hu, Z., Liu, Y., Zhao, J., Wang, S., Wang, Y., Shen, W., Gu, Q., Luu, A. T., Ng. S.-K., Jiang, Z., et al. Longrecipe: Recipe for efficient long context generalization in large language models. arXiv preprint arXiv:2409.00500 2024b.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. 2024. URL https://arxixs argg/bs//400) 21783.
Jacot, A., Gabriel, F,, and Hongler, C. Neural tangent ker- nel: Convergence and generalization in neural networks. Advances neural information processing systems, 31, 2018.
10

---

LongRoPE2: Near-Lossless LLM Context Window Scaling
Jeong, S., Baek, J., Cho, S., Hwang, S. J., and Park, C. Adaptive-rag: Learning adapt rerrieval-augmentedd large language models through question complexity, 2024. URL https://arxiv. htrg/bbs/24033 14403.
LocaILLaMA.
Ntk-aware scaled rope allows Ilama models have extended (8k+) context size without any fine-tuning and minimal perplexity degration, 2023, URL https: htts:://www.. wtt.ddtttccmm////oaaaaaaa comments/ (11.2//5/ntkawareessalllee llama_models
Kamradt, G. Needle in haystack pressure testing llms, 2023. //ti:/ggittttb/t ctw/ homm/gaamrada www.aaaaa/taaatn adtLLLMeest NeedleInAHaystack.
Lozhkov, A., Ben Allal, L., von Werra, L., and Wolf, T. Fineweb-edu: the finest collection of educational con- tent, 2024. URL /Rts::tts://huuggngg
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, Transformers are rnns: Fast autoregressive transformers with linear attention, 2020. URL htttt:///axxiv abs/2006. 6236.
co/datases tasets/
Luo, K., Liu, Z., Xiao, S., and Liu, K. Bge landmark embed- ding: A chunking-fiee frree embedding method for retrieval augmented long-context large language models, 2024. URL https: /arxiv org/ /et../222..1122 11573.
Lee, J., Chen, A., Dai, Z.,, Dua, D., Sachan, D. S., Boratko, M.. Luan, Y., Arnold, S. M., Perot, V., Dalmia, S., al. Can long-context language models subsume retrieval, rag, sql. and more? arXiv preprint arXivv2406.13121 2024a.
Men, X., Xu, M., Wang, B., Zhang, Q., Lin, H., Han, X., and Chen, W. Base of rope bounds context length, 2024a. URL https: //arxiv org/ h/g/abs/200 abs52405. 14591.
Lee, K.-H., Chen, X., Furuta, H., Canny, J., and Fischer, A human-inspinn reading agent with gist memory of very long contexts, 2024b. URL https:///axiis orrg/ abs/2402 09727.
Men, X., Xu, M., Wang, B., Zhang, Q. Lin, H., Han, X., and Chen, W. Base of rope bounds context length. arXiv preprint arXiv:2405. 14591, 2024b.
Li, M., Zhang, S., Liu, Y., and Chen, K. Needlebench: Can lims do retrieval and reasoning in million context window?, 2024a. URL https://arxiv, wrg. abs/ 11963.
Meta.
Llama3.2: Revolutionizing edge ai and vision with open, customizable models, URL https://all meta. com/blog/
2024.
1lama- 2-ma-332-connects
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. Starcoder: may the source be with you! arXiv preprint 1:Xiv:2305.06161, 2023.
Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2339.00077/ 2023.
Li, S., He, Y., Guo, H., Bu, X., Bai, G., Liu, J., Liu, J., Qu, X., Li, Y., Ouyang, W., Su, W., and Zheng, B. Graphreader: Building graph-based agent enhance long-context abilities of large language models, 2024b. URL https://arxit org/ abs/ /2t.../226..2 114406..4555 14550.
Ren, L., Liu, Y., Lu, Y., Shen, Y., Liang, C., and Chen, W. Samba: Simple hybrid state space models for efficient unlimited context language modeling, 2024. URL https: //arxiv. org/ abs/ abs/2406.... .7552..
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104. 000004000994...0 2021.
Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedi- gos, I., Safahi, E., Meirom, S., Belinkov, Y., Shalev- Shwartz, S., Abend, O., Alon, R., Asida, T., Bergman, A., Glozman, R., Gokhman, M., Manevich, A., Ratner, N., Rozen, N., Shwartz, E., Zusman, M., and Shoham, Y. Jamba: A hybrid transformer-mamba language model, org/ absaaaas/ 1.403.198800 19887.
Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil.. S., Raghavan, N., Singhal, U., Ramamoorthi, R., Bar- ron, J., and Ng, R. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33: 7537-7547, 2020.
2024. URL
Lin, Z., Miao, Y., Zhang. Q. Yang, F., Zhu, Y., Li, C., Maleki, S., Cao, X., Shang, N., Yang, Y., Xu, W., Yang, M., Zhang, L., and Zhou, L. nnscaler: Constraint-guice parallelization plan generation for deep learning training. In 18th USENIX Symposium on Operating Systems De- sign and Implementation (OSDI 24), pp. 347-363, 2024.
Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024. URL https://qwenlnn ittth....oblgg. qwen2.5%.
Wang, H., Liu, Q., Du, C., Zhu, T., Du, C., Kawaguchi, K., and Pang. T. When precision meets position: Bf breaks down rope in long-conters training, arXiv preprint arXhv:2411.1347.,1 2024.
Liu, X., Yan, H., Zhang, S., An, C., Qiu, X., and Lin, D. Scaling laws of rope-based based extrapolation. arXiv preprint arXiv.2310.05209, 2023.

---

LongRoPE2: Near-Lossless LLM Context Window Scaling
Weber, M., Fu, D. Y., Anthony, Q. Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., Athiwaratkun, B., Chalamala, R., Chen, K., Ryabinin, M.. Dao, T., Liang. P., Ré, C.. Rish, I., and Zhang, C. Redpajama: an open dataset for training large language models. NeurIPS Datasets and Benchmarks Track, 2024.
long sequences via approximate diagonalization. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https://opene https://openreview.nett wtwprreewwii.neeeouun..... id=DjeQ390oLQ.
Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q. Yang. L.,, and Ahmed, A. Big bird: Trans- formers for longer sequences. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 17283-17297. Curran As- sociates, Inc., 2020. URL https: nttps::/prooeeddings. neurips. httpc///pppper.illlepppppppepeee c851 c5512214422888877777758778221110
Yang, A., Yang, B., Hui, B.. Zheng, B.. Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang. F., Dong, G., Wei, H., Lin, H., Tang, J., Wang. J., Yang, J., Tu, J., Zhang, J., Ma, J., Yang. J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J. Dang, K., Lu. K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Liu, X., Fan, Y. Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang, Z., Guo, Z. and Fan, Z. Qwen2 technical report, 2024a. URL https://arxiv wttp//////00...... 10671.
pdf.
Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M. K., Han, X., Thai, Z. L., Wang. S., Liu, Z., et al. oobench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402.13711 2024a.
Yang, L., Xu, S., and Xiong. D. Deis: Efficient length ex- trapolation of iims via divide-and and--andd---ee scaling factor search. arXiv preprint arXiv:24 /2. 1881 2024b.
Zhang. Y., Sun, R., Chen, Y., Pfister, T., Zhang, R., and Arik, O. Chain of agents: Large language models collaborating on long-context tasks, 2024b. URL https:
Yang, S., Wang, B., Shen, Y., Panda, R., and Kim, Y. Gated linear attention transformers with hardware- efficient training, 2024c. URL https://arxiv.or abs/2312.06633
/ //arxiv. org/ http:/bs/24440 02244.266.......
Zhu, Q. Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R., Wu, Y., Li, Y., Gao, H., Ma, S., et al. Deepseek-coder v2: Breaking the barrier of closed-source models code intelligence. arXiv preprint arXiv:2406.1193/ 2024.
Yu, A., Nigmetov, A., Morozov, D., Mahoney, M. W., and Erichson, N. B. Robustifying saatesppace models for
12

---

LongRoPE2: Near-Lossless LLM Context Window Scaling
A Additional Experiments and Analysis
Additional details. For the rescaling factor search, we set a population size of P = 64, evolution iterations of 40, and a mutation probability = 0.3. The searched rescaling factors are then applied with mixed context window training.
To accelerate training and inference, we use F (Dao, 2023), which requires no modifications for mixed context window training or factor-switch waaoo--wwthh-bbeeette inference (as illustrated in Fig. 10). Given that GPU memory and computation time increase exponentially with sequence length, fine-tuning long-context models presents significant challenges. To address this, utilize nnScaler (Lin et al., 2024), an efficient distributed training system for long-context LLMs, to reduce training costs. 10B tokens take approximately 39 hours for Phi3-mini and 54 hours for LLaMA3-8B on 64 A100 GPUs. During inference, the switch between rescaled and original RoPE is triggered when the combined length of the input context and generated tokens exceeds the pre-trained context window. Switching to rescaled RoPE for long-context inference requires a one-time recalculation the KV cache, potential limitation we leave for future work.
Additional results on RULER and extended LLMs on the RULER benchmark. Figures and 8 provide comprehensive results for the needle-in-a--eee -naae--yt-- tests. As observed, the YaRN method frequently fails to retrieve needles across Phi3-mini-3.80 LLaMA3-8B, Meta-LLaMAA31-81) and Meta-LLaMAA,1-88 111.1----B--Irue
Tables 8 and 9 show the detailed per-task accuracy of our
Table 8. LongRoPE2-extendde Phi3-mini (3.8B)-128k per-task performance on RULER.
NIAH singlel
NIAH
NIAH
NIAH multikeyl
NIAH
NIAH
NIAH
NIAH
single-hop QA
multi-hop
Length
CWE
FEW
Avg.
single2
single3
multikey3 multikey2
multivalue
multiquery
QA
4096
100
100
99
91
96
97
97.75
97.75
85.8
93.7
85.33
82
50
90.41
8192
100
100
100
93
97
89.5
93.75
84
87.2
86
68
47
87.34
16384
82
55.4
32768
100
100
86
57
78
76.8
33.2
91.67
56
44
76.51
65536
100
100
99
85
71
32
67.75
69.25
66.8
0.4
71.67
50
37
65.37
131072
100
98
95
92
40
18
56.75
59
35.2
0.3
89.33
47
34
58.81
Table 9. LongRoPE2-cxtended LLaMA3-8B-1288 per-task performance on RULER.
NIAH singlel
NIAH
NIAH
NIAH
NIAH multikey2
NIAH multikey3
NIAH multivalue
NIAH
single-hop QA
multi-hop
Length
VT
CWE
FEW
Avg.
single2
single3
multikeyl
maltiquery
QA
4096
100
100
99
100
100
100
99
99.75
98.8
98.5
96.33
79
60
94.61
8192
100
100
100
100
100
100
99
99.75
99.8
95.9
91.33
74
58
93.68
16384
100
100
100
100
98
98.25
99.6
86,8
96.33
69
92.31
32768
100
100
100
100
96.25
98.6
63.9
95.67
72
90.49
65536
100
100
100
98
98
95
95.75
99.75
98.6
33.6
80.33
62
52
85.62
131072
100
100
99
96
91
94
96.5
97
92.6
9
85.33
56
50
82.03
Table 10, Ablation study on the number of scarched dimensions.
Regular short tasks
RULER
Method
MMLU
MMLU Pro
GSMBK
4k
8k
16k
32k
64k
128k
Base Model: Phi3-mini (3.8B)
LongRoPE2 (dred and higher dims)
70.07
40.30
73.62
90.41
87.22
83.33
76.51
65.37
58.81
LongRoPE2 (all dims)
69.96
39.84
74.83
90.02
87.21
82.42
74.86
63.95
57.34
Base Model: LLaMA3-8B
LongRoPE2 (dyod and higher dims)
65.01
34.61
50.80
94.61
93.68
92.31
90.49
85.62
82.03
LongRoPE2 (all dims)
64.34
33.83
51.55
93.92
92.61
91.41
89.30
83.11
78.07
The ablation study on search algorithm. our work, we focus on searching for the real critical dimension and the scaling factors of higher dimensions beyond it. For the lower dimensions before the critical dimension, we directly apply NTK scaling without further optimization. To evaluate this design, conduct additional ablation study. For comparison, also allowed the search to include lower dimensions. As shown in Table 10, while searching across all dimensions yields competitive results, it underperforms compared to our proposed method. A possible reason is that limiting the search to higher dimensions significantly reduces the search space, enabling more effective discovery of the optimal solution.
Discussions on long-short RoPE factor switch. When transitioning from the short context window (which uses the short factor, i.e., the original RoPE) to the long context window (which uses the rescaled long factor), KV cache recomputation
13

---

LongRoPE2: Near-Lossless LLM Context Window Scaling
Phi3-mini (3.8B) YaRN-128k
Thi3-mini (3.8B) NTK-128K
1.0
Depth Percent (%)
11
22
0.8
22
0.8
33
Depth Percen
33
44
44
0.6
58
67
0,4
67
0.4
78
78
89
100
100
1K
10k
19k
28k
3/A
40k
55k
64K
74k
83K
92K
101k
110k
119K
128K
1K
10k
10K
28k
37K
46K
55K
64K
74k
83K
92k
101K
110K
119K
128K
0.0
Context Length
Context Length
Phi3-mini L...P
Phi3-mini (3.8B) LongRoPE2-128K
0-
1.0
1.0
Depth Percent (%)
11
11
22
22
0.8
Depin Percent
33
44
0.6
44
0.6
67
56
0.4
56
67
0.4
78
0.2
78
0.2
89
100
1K
10K
19K
28k
37K
46K
55N
64k
74/
89K
92k
101K
110K
119K
128K
0.0
100
KK
10k
19k
28k
37K
48K
55k
64K
74K
83K
92K
101K
110K
119K
128K
0.0
Context Length
Context Length
Figure 7. Needle in Haystack full results for Phi3-mini (3.8 (3.8B) 128k.
LLaMA3-8B-Yaree -BA---8-------88----
LLaMA3. 1-88-18812888 128k
1.0
Depth Percent (%)
11
0.8
11
0.8
22
22
33
33
44
0.6
Depih Percar
44
0.6
56
67
0.10
56
67
0.4
78
0.2
78
89
89
100
1K
10K
19K
28k
37K
48k
58K
64k
74K
83K
92K
101k
110K
119K
128k
0.0
1K
10k
19k
28k
37K
48K
586
64K
74/
83K
92K
101k
110k
119K
128k
0.0
Context Length
Context Length
LLaMA3. 1-8B-Instruct-1288
LLaMA2-8B-LongRoPE2-111 128k
0-
1.0
Depth Percent (%)
11
11
0.8
0.8
22
22
33
Depth Parcant
33
44
0.6
44
0.6
56
0.40
56
67
67
78
78
89
0.2
89
0.2
0.0
0.0
1K
10K
19k
28k
37K
40A
55K
64K
74K
83K
92K
101K
110K
119K
128K
1K
10K
10K
28k
37K
40K
55k
64k
74k
83K
92K
101K
110K
119K
128K
Context Length
Context Length
Figure 8 Needle in Haystack full results for LLaMA3-8B-1228
14

---

LongRoPE2: Near-Lossiess LLM Context Window Scaling
Phi3-mini-
Phi3-mini-128k
LLaMA3-8B-128K
LLaMA3-8B-128k
2500
60
YaRN
15
YaRN
NTK
YaRN-rod
NTK-rod
YaRN-rod
NTK-rod
2000
Sccale Factor
1500
1000
20
5
500
25
3035
RoPE dimension Index
RoPE dimension Index
RoPE dimension Index
RoPE dimension index
Figure The RoPE rescaling factor distributions NTK/YaRN adjusted based on the critical dimension (i.c., YaRN-red, NTK-red).
Data:
# flash attention from Wash_attn import flash_ attn_func func
pre-training
deiginnii__ogge rotpi//oroppppooonnnoo ids, base, rope_ (ope_dim):
inv freq = (base**"rrh,Tensorr(. 2, ... (ope_dim)))
attn_ output = flash attn funci
query. states,
key_ states, value states, dropout p=dropout; causal=True)
return cos, sin
def longropeepooiiioon ids, base, rope_ dim, original max position embeddings, long_ factor, short factor):
# flash attention api call in mixed context window training from flash, attn import flash attn varien_ func
inv freq =1.0 (base**torch. Tensorrr..Tesor00...
..., rope dim]i)
attn, output flash attn varten funce
switch rescaled RoPE E the seq length exceeds the original context window if ttarch.max(posttion ids) + 1) original max_ pociikooostoonnoo embeddings:
query_ stales,
statess
ext factors =long_oooeeeeeeee Tactor
value_star states,
dropout dropout_p=drropor p=dropout
else:
causal=True,
factors short, factor use original RoPE
inv freq = ori inv freq / ext, factors
cu segiens q=cu_p seq_u_ee_________e lens a.
mäx_ seglen q=max lnggth q. max seqien k=max length
suglens k=çu seq
return cos, sin
Figure The pseudocode mixed context window training inference.
15

---

LongRoPE2: Near-Lossless LLM Context Window Scaling
required. However, this recomputation does not during every inference. It triggered only when the input length remains within the short context window, but the total sequence length (including both input and generated tokens) exceeds it for the first time. After this one-time recomputation, no further recomputation is needed for the remainder of the generation. In practice, this situation is relatively uncommon, as most inference cases either remain within the short context window or begin directly in long-context mode.
To quantify the computational overhead, we measured the KV cache recomputation time on 4x80GB A A100 GPU (using vLLM 0.7.3) for both Phi-3-mini and LLaMA3-8B, and compared it against normal decoding latency. As shown in Table 11, the additional recomputation cost is equivalent generating only 15 (phi3-mini) and 25(llama-8b) tokens, which negligible in the context of long-context generation.
Table I.. Prefil and decoding time costs. The numbers in () indicate the amount of decoded tokens corresponding to the time spent KV
cache recomputation.
Model
Prefil time (KV recompute)
Decode time across different generation length
512
1k
2k
4k
8k
16k
Phi3-mini (prefill 2k) LLaMA3-SB (prefill 8k)
124.1ms 613.9 ms
7.6 ms (16.2) 7.66 ms (16.2) 7.7 ms (16.1) 7.8 ms (I5.9) 14.3 ms (8.7) 23.3 ms (5.3)
24.1 ms (25.5) 24.2 ms (25.3) 24.1 ms (25.5) 24.2 ms (25.4) 23.5 ms (26. /) 23.6ms (26.0)
B
Synthetic data sample
Synthetic search data based PG19 book sample
A special magic number is hidden within the following text. Make sure to memorize it. will quiz you about number af terwarrs..
One of the special magic numbers for numerous-kite is: 6716897. The Old Testament of the King James Version of the Bible The First Book of Moses: Called Cenesis 1:1 In the beginning God created the heaven and the earth. And the earth without form, and void; and darkness upon the face of the deep. And the Spirit of God moved upon the face of the waters.
it be for a witness between me and thee. 31:45 And Jacob took stone, and set it up for pillar. 31:46 And Jacob said unto his brethren, Gather stones; and they took stones, made an heap: and they eat there upon the heap. 31:47 And Laban called it sahadtthh: but Jacob called it Galeed.
3:39 Also in the fifteenth day of the seventh month, when ye have gathered in the fruit of the land, ye shall keep a feast unto the LORD seven days: on the first day shall be a sabbath, and on the eighth day shall be a sabbath. 23:48 And ye shall take you the boughs of thick trees, and wi llllows of the brook; and shall rejoice before the LORD your God seven days. What is the special magic number for numerous-kar mentioned in the provided text? The special magic number for numerous-kite mentioned in provided text is
first day boughs
goodly trees, branches
palm trees,
16


Reviews:
[{"review_id": "KpynhegvSK", "reviewer": "Reviewer_Ebnv", "summary": "Maintaining the performance on both long and short benchmarks are a critical challenge for existing long context extension methods. LongRoPE2 is a new approach that extends the effective context window of pre-trained large language models to the target length, while preserving the performance on the original shorter context window.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response**: Thank you for your valuable feedback and for recognizing the strengths of our work. We appreciate the opportunity to address your concerns.\n\n1) **Affordable evolutionary search computational cost**: we acknowledge that evolutionary search introduces additional costs. To further clarify its feasibility, we conduct additional experiments to evaluate the search cost when scaling from 128k (current context window length) to 1024k. Using vllm0.7.3 as the inference engine and running on an 8*A100(80GB) server setup, we measured the total search time. As shown in the table below, even when scaling to 1M tokens, the total search time remains manageable at 240 hours (10 days). Moreover, this is a one-time *offline* process, and the search time can be linearly reduced by increasing the number of GPUs, due to the nature of evolutionary search. Therefore, it is practical for LLM pretraining teams. \n\n||128k|512k| 1024k|\n|:--:   |:--:   |:--:   |:--:   |\n|total search time on 8*80GB a100| 7.5h | 68h | 240h |\n   \n2) **KV cache recalculation occurs only in specific cases and has minimal overhead**: We acknowledge that KV cache recomputation is required when transitioning from the short context window (using the short factor, i.e., original RoPE) to the long context window (using the long factor). However, this recomputation does not occur in every inference. It happens only when the input length is within the short context window, but the total length (input+generated tokens) exceeds it **for the first time**. After this one-time recomputation, no further recomputation is needed for the rest of the generation. \n\nIn most general inference scenarios, this situation is relatively uncommon, as prompts and completions typically either remain within the short context window or start in the long context mode from the beginning. To quantify the cost, we measured KV recomputation time on a  4x80GB A100 GPU (with vllm 0.7.3) for Phi-3-mini and LLaMA3-8B, comparing it against normal decoding time:\n\n||prefill (kv recompute)|decode-output 512| decode-output 1k| decodetime-output 2k| decodetime-output 4k| decodetime-output 8k | decodetime 16k|\n|:--:   |:--:   |:--:   |:--:   |:--:   |:--:   |:--:   |:--:   |\n|Phi3-mini (prefill 2k)| 124.1ms| 7.63ms (**16.2**)| 7.66ms (**16.2**)| 7.71ms (**16.1**)| 7.78ms (**15.9**)| 14.29ms (**8.7**)| 23.3ms (**5.3**) |\n|LLaMA3-8B (prefill 8k)|613.9ms | 24.11ms (**25.5**)| 24.22ms (**25.3**)| 24.05ms (**25.5**)| 24.18ms (**25.4**) | 23.5ms (**26.1**) | 23.58ms (**26.0**) |\n\nThe numbers in () indicate the amount of decoded tokens corresponding to the time spent on KV cache recomputation. These results indicate that the additional recomputation cost is equivalent to generating only ~15 (phi3-mini) and ~25(llama3-8b)  tokens, which is negligible in the context of long-context generation.\n\n\n>Q2: Does mixed training cause interference between short/long contexts?\n\n**Response**: Thank you for your insightful question. While it’s true that mixed context window training applies two RoPE scaling factors simultaneously during mid-training, which could introduce interference, our empirical results suggest that this \"interference\" plays a constructive role and hence does not degrade performance. In fact, it not only recovers short-context performance but also enhances long-context performance. To better illustrate this, we refer to Table 7 from our original paper.\n\n||MMLU-(short)|MMLUPro-(short)|GSM8K-(short)| RULER-4k| RULER-8k| RULER-16k | RULER-32k|RULER-64k|RULER-128k|\n|:--:   |:--:   |:--:   |:--:   |:--:   |:--:   |:--:   |:--:   |:--:   |:--:   |\n|Phi3-mini (**with mixed context window training**)| **70.07**|**40.30**|**73.62**|90.41|**86.87**|**83.33**|**76.51**|**65.37**|**58.81**|\n|Phi3-mini (no mixed context window training)|66.56|34.86|64.67|**90.55**|85.77|81.08|73.31|63.75|56.22|\n||||||||||\n|LLaMA3-8B (**with mixed context window training**)|**65.01**|**34.61**|**50.80**|94.61|**93.68**|**92.31**|**90.49**|**85.62**|**82.03**|\n|LLaMA3-8B (no mixed context window training)| 64.57| 33.83| 48.37| **94.67**|93.15|91.24|89.38|83.53|80.18|\n\nA possible explanation for this surprising improvement is that the so-called “interference” actually plays a constructive role in learning. Specifically, the short-context window helps preserve position modeling for non-interpolated positions (e.g., LLaMA3’s native positions 0, 1, 2, ..., 8191), while the long-context window primarily facilitates the adaptation for newly interpolated positions (e.g., LLaMA3’s new positions like 1/16, 2/16, ..., 17/16). This training strategy effectively constrains the model’s adaptation to interpolated positions while maintaining consistency with the original position modeling - a concept similar to the KL divergence constraint in PPO, which prevents large deviation from the original policy model.\n\nWe appreciate this insightful question, which has prompted further reflection and discussions.", "claims_and_evidence": "Claims: LongRoPE2 extends context windows to 128k while retaining >97% short-context performance. The key contributions are (1) higher RoPE dimensions are undertrained, (2) evolutionary search for rescaling factors guided by needle-driven perplexity, (3) mixed training with original/rescaled RoPE.\n\nEvidence: Achieves strong results on RULER, and real-world benchmarks (LOFT, LongBench). Outperforms YaRN, NTK, and LongRoPE with much fewer tokens.", "methods_and_evaluation_criteria": "Methods: Evolutionary search for critical dimensions and scaling factors, mixed training (original RoPE for short contexts, rescaled RoPE for long).\n\nEvaluation: Benchmarked on RULER, Needle-in-a-Haystack (retrieval), LOFT/InfiniteBench (real-world), and MMLU/GSM8K (short-context).", "theoretical_claims": "Challenges prior RoPE OOD theory: insufficient training in higher dimensions extends empirical periods, requiring larger scaling factors than theoretical bounds. \n\nTo be honest I do not carefully check the correctness of all theoretical claims of this paper.", "experimental_designs_or_analyses": "Ablations confirm needle-PPL’s superiority over standard PPL and mixed training’s necessity.\nAdjusted baselines (YaRN-red/NTK-red) show improved but suboptimal performance.", "relation_to_broader_scientific_literature": "Builds on RoPE rescaling (NTK, YaRN) and evolutionary optimization. Different from RAG/agent-based methods, positioning LongRoPE2 as complementary methods.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths: Efficient scaling (10B tokens), minimal short-context degradation.\n\nWeaknesses: Evolutionary search computational cost; inference requires KV cache recalculation.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. How does evolutionary search scale to million-token contexts?\n2. Does mixed training cause interference between short/long contexts?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YBGS9h8WSG", "reviewer": "Reviewer_gztC", "summary": "This paper proposed LongRoPE2, a RoPE scaling method to extend the context window of LLMs. The primary extension compared to LongRoPE1 is that LongRoPE2 utilizes a needle-based search rather than perplexity-based one for various rope dimension scaling. The experimental results demonstrate the superior performance of LongRoPE2 compared to other RoPE scaling methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: Clarification on LLaMA3.1-8B long-context evaluation numbers, and the \"overclaim\" comments\n\n**Response**: We appreciate your feedback and would like to clarify the following points:\n\n1. **65.1 is the En.MC score of the instruct version, not LLaMA3.1-8B.**:  As noted in Table 2 of the LLaMA3.1 tech report, the 65.1 score is for the instruct-tuned version (a detail that can be overlooked). Compared to the fair baseline, LLaMA3.1-8B, our model achieves a higher score (**46.72** vs. **45.85**) on En. MC. Moreover, our model consistently outperforms LLaMA3.1-8B across several long-context benchmarks. Here are additional results:\n\n>InfiniteBench and LongBench:\n\n||avg.|En.MC|En.Sum| KV retrieval | TriviaQA | TREC | LCC | RepoBench-P|\n|:--:   |:--:   |:--:   |:--:   |:--:   |:--:   |:--:   |:--:   |:--:   |\n|LLaMA3.1-8B|54.28| 45.85| 15.27|16.20| **91.13**| 73.50| 70.24| **67.83**|\n|LongRoPE2-LLaMA3-8B|**65.20**|**46.72**| **16.20**|**88.0** |**91.13**| **76.50**| **70.47**| 67.39 |\n\n>LOFT:\n\n||avg.| ArguAna | FEVER | HotPotQA | MS MARCO | NQ | Quora | SciFact|\n| :--:  | :--:  | :--:   | :--:   |:--:   |:--:   |:--:   |:--:   |:--:   |\n|LLaMA3.1-8B| 53.14|19.0| 90.0| 12.0| 69.0| 78.0| 61.0|43.0|\n|LongRoPE2-LLaMA3-8B| **74.28**|**28.0** | **96.0** | **70.0** | **80.0**| **94.0**| **79.0** | **73.0**|\n\n\n2. **Our RoPE scaling method is designed to improve broad long-context capabilities, not to optimize for any specific benchmark like RULER.** The use of needle data for search is **not** designed to fit RULER but to **better control long-range token dependency distances** in long documents.  E.g., We used only the simplest number needle synthesis method. Our extensive experiments proved the superiority over other methods (e.g., NTK, YaRN) across diverse benchmarks.\n\n>Q2: What are the criteria for you to choose the evaluation tasks? \n\n**Response**: Our selection follows two key principles:\n1) **Effectiveness for evaluating a pre-trained LLM rather than a chat LLM**. Since our method extends a pre-trained LLM without post-training, we prioritize tasks aligned with this setup: (i) completion-based tasks, such as few-shot learning and code completion and En.MC in InfiniteBench. (ii) QA tasks with few-shot examples, such as various text-retrieval QA tasks in LOFT. \n\n2) **Comprehensive long-context evaluation**. To evaluate multiple aspects of long-context performance, we include tasks covering RULER, needle-in-a-haystack retrieval, real-world text QA, high-difficulty KV retrieval, multi-choice QA, few-shot learning, and code completion, as detailed in our evaluation section. We believe this selection fairly reflects the strengths of our method and provides a well-rounded assessment.\n\n*Additional results on chat-based sub-tasks*.  For your reference, we provide additional results on chat-based LongBench tasks. As shown below, We achieve the highest average score, even surpassing LLaMA3.1-8B.\n\n||Avg.|narrativeqa| Qasper | multifiledQA | hotpotqa | 2wikimqa | musique | gov_report | qmsum| samsum|\n| :--: | :--:  | :--:  | :--:   | :--:   |:--:   |:--:   |:--:   |:--:   |:--:   |:--:   |\n|LLaMA3.1-8B|22.60| 20.90|12.50|32.72|11.95| 13.98| **8.62**| 29.95|**25.53**|**47.23**| \n|NTK-LLaMA3-8B|20.32|21.14|11.93|29.02|11.91|14.71|7.81|21.50|22.09|42.70|\n|LongRoPE2-LLaMA3-8B|**24.31**|**21.79**|**18.13**|**36.25**|**13.85**|**19.42**|8.03|**30.12**|25.41|45.80|\n\n>Q3: The clarifications on the main contributions and claims:\n\n**Response**: We are grateful for your questions and would like to clarify that our main contribution is not the discovery of insufficient training in high-frequency RoPE, but rather the introduction of **a new RoPE OOD hypothesis**. This hypothesis explains why existing RoPE rescaling methods, such as NTK and YaRN, often result in suboptimal long-context performance. This contribution has been acknowledged by the other two reviewers.\n\nRegarding the mixed context window training, we would like to emphasize that the key difference between our approach and those used in LLaMA3.1 and LongAlign is the use of two distinct RoPE scaling factors: a short factor for short contexts and a long factor for long contexts. This **dual-factor** approach is essential in significantly recovering short-context performance, which we have shown through extensive experiments.  \n\nHere, we perform an additional comparison with LLaMA-3.1’s mixed training. As shown below, we significantly improve short-text performance.\n\n||(short)-MMLU|(short)-MMLU pro| (short)-GSM8k | Ruler-128k| \n| :--: | :--:  | :--:  | :--:   | :--:   |\n|LLaMA3-8B (**Our mixed context windows training**)|**65.01**|**34.61**|**50.80**|**82.03**|\n|LLaMA3-8B (mixed training in LLaMA3.1)| 64.18|32.95 |46.25 |71.83 | \n\nWe hope these responses address your concerns and clarify any confusion, and we will incorporate them in the revisions. Thank you again for your valuable feedback and suggestions, and we kindly ask you to consider re-evaluating our work.", "claims_and_evidence": "1. The most overclaim is  \"LongRoPE2-extended LLaMA3-8B-128k surpasses Meta’s LLaMA3.1-8B-128k in long-context performance with 80x fewer training tokens\". This claim is supported by the RULER results in Fig 1. However, LongRoPE adopts a needle-based search for RoPE scaling, which may (over)fit the synthetic tasks in RULER benchmark, hence achieving better results. For other general tasks such as En. MC in InfiniteBench, **LongRoPE2-LLaMA3-8B achieved a score of 46.72 but LLaMA-3.1-8B achieved 65.1**. This means LongRoPE2-LLaMA3-8B may still have a large gap to LLaMA-3.1-8B involving far more training tokens. Note that it's not necessary for LongRoPE2-LLaMA3-8B to surpass LLaMA-3.1-8B, but should fix the claim for clearness.\n\n2.  The mixed context window training is adopted in [1] and LLaMA-3.1 (as well as a common practise in long-context LLM community) to maintain the short-context performance, but the authors claims they propose such a \"novel\" strategy.\n\n[1]LongAlign: A Recipe for Long Context Alignment of Large Language Models", "methods_and_evaluation_criteria": "The evaluation benchmarks are popular in the long-context understanding field.", "theoretical_claims": "There are no proofs for theoretical claims.", "experimental_designs_or_analyses": "I have gone through the ablation studies which have demonstrated the effectiveness of LongRoPE2's designs.", "relation_to_broader_scientific_literature": "This paper is a direct extension of LongRoPE[1].\n\n[1] LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens", "essential_references_not_discussed": "This paper considers a mix context window training strategy as one of its primary contributions. However, such strategies are widely-used in some previous works such as LongAlign[1], which is missing in references.\n\n[1]LongAlign: A Recipe for Long Context Alignment of Large Language Models", "other_strengths_and_weaknesses": "Strength:\n1. Generally the experiments covering long and short contexts are well-designed and can demonstrate the effectiveness of the proposed method.\n2. The intuition regarding the insufficient training of high-frequencies RoPE makes sense.\n3. I believe the needle-driven PPL search is a better choice for LongRoPE as the pure PPL on normal documents may be orthogonal to long-context performance.\n\nWeaknesses:\n1. I feel most designs in the work have been proposed/adopted in previous works. For example, the intuition that high-frequencies RoPE  may be insufficiently trained has been introduced in a popular blog (https://spaces.ac.cn/archives/9706) regarding RoPE scaling. The mixed context window training is adopted in [1] and LLaMA-3.1 to maintain the short-context performance. It may be improper to regard these points as this work's contributions and should give some credit to the related works.\n\n[1]LongAlign: A Recipe for Long Context Alignment of Large Language Models", "other_comments_or_suggestions": "The related work section is better placed in the main body of the paper to make it self-contained. (This is a suggestion and wouldn't affect my rating.)", "questions_for_authors": "1. Do you think it's better to list the results of LLaMA-3.1-8B in the main table? Since you claimed the proposed method on 10B tokens can surpass LLaMA-3.1-8B's continual training of 800B tokens.\n\n2. What are the criteria for you to choose the evaluation tasks? It seems there are some challenging tasks such as En.QA, En. Sum, etc. in the InfiniteBench and some other tasks of various categories in LongBench. The selected tasks seem to be irregular.\n\n3. The claim regarding the intuition of insufficient high-frequencies RoPE training and the mix context window training should be better to fix.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GscOWQBEYB", "reviewer": "Reviewer_ak4t", "summary": "This paper mainly introduces LongRoPE2, aiming to achieve an effective long context window while preserving short-context performance by context extension. Based on LongRoPE, LongRoPE2 introduces a new needle-PPL guided evolutionary search method for settling the rescaling factors, and proves it to be more effective than the naive PPL-guided one by experiments. For retaining preformance on short contexts, LongRoPE2 proposes a novel mixed context window training method. Compared to YaRN, NTK and LongRoPE,  LongRoPE2 achieves better performance on long contexts while retaining over 98.5% of short-context performance.\n\n---\n\n## update after rebuttal\n\nThanks to the authors for their response, and I acknowledge that LongRoPE2 is a strong and valuable work. However, there are still some unclear aspects in the paper, such as the lack of a detailed explanation regarding how Figure 3(a) was derived. Due to these unresolved concerns, I have decided not to adjust our initial score. In our view, a score of 3 (borderline accept) remains reasonable and justified.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and for recognizing our contributions. We greatly appreciate your acknowledgment of our New RoPE OOD Hypothesis and the role of needle-PPL-guided search in validating this hypothesis through empirical results. We are also glad that you found our extensive experiments in Sections 4.2 and 4.3 valuable in demonstrating the effectiveness of LongRoPE2 and our key design choices, such as the real critical dimension and mixed context window training. Please let us know if there are any specific aspects where we can provide more details. Thank you again for your time and constructive evaluation!", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "YES", "theoretical_claims": "yes\nThis paper proposes a New RoPE OOD Hypothesis that the empirical RoPE periods in higher dimensions are longer than theoretical values, limiting current methods to fully address RoPE OOD. This implies that the actual optimal rescaling factors may be greater than the theoretical one. Then, by applying needle-PPL-guided search, LongRoPE2 does get rescaling factors larger than the theoretical one, and performs better, which from this point of view can test this hypothesis.", "experimental_designs_or_analyses": "Yes.\nIn 4.2, this paper presents reults on RULER, NIAH, LOFT of LongRoPE2-extened, comparing with other SOTA RoPE rescaling methods(YaRN, NTK, LongRoPE) and shows the effectiveness.\nIn 4.3,  to validate the effectiveness of real critical dimension, one experiment applies d_{rcd} to YaRN and NTK, which is also get improved.  To validate the effectiveness of needle-PPL guided search, one experiment compares it with the naïve one with the same training process on the same test dataset. Finally, the effectiveness of mixed context window training is also validated here.", "relation_to_broader_scientific_literature": "Some scaling methods before LongRoPE(PI, YarN, NTK) ignore the actual errors caused by different parameters of models after training. This issue was preliminarily solved in LongRoPE. LongRoPE2 is based on LongRoPE, and more effective.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "This paper is clearly demonstrated. The most enlightening contribution may be the New RoPE OOD hypothesis (in 3.1), as it gives directions for optimization of other methods  not limited to this paper.", "other_comments_or_suggestions": "There is one typo on page 8, subtitle: need-PPL  should be needle-PPL", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The study presents LongRoPE2, a method designed to significantly extend the context window of pre-trained large language models, such as LLaMA3-8B and Phi3-mini-3.8B, without compromising the performance on shorter context windows. By addressing the out-of-distribution (OOD) issues with existing methods, LongRoPE2 employs a hypothesis-driven evolutionary search algorithm to optimize the RoPE (Random Projections) dimensions, and introduces a mixed training approach. combines original and rescaled RoPE to ensure high performance across various benchmarks. The paper highlights extensive experiments that validate the effectiveness of LongRoPE2, demonstrating its ability to extend the context window to 128k while retaining over 98.5% of the short-context performance using only 10B tokens, a substantial improvement over previous approaches.
Reviewers raised concerns including affordable evolutionary search computational cost, KV cache recalculation, needle-PPL and mixed training necessity，evaluation tasks, effectiveness of the method, etc.  The authors addressed these concerns with clarifies and additional experimental results.
The method has the potential to significantly impact the field of large language models by enabling longer context windows, which is crucial for many real-world applications of LLMs.
Decision: Accept (poster)