A Controlled Study on Long Context Extension and Generalization in LLMs

## Abstract

Abstract Broad textual understanding and in-context learning require language models that utilize full document contexts. Due to the implementation challenges associated with directly training long-context models,
many methods have been proposed for extending models to handle long contexts. However, owing to differences in data and model classes,
it has been challenging to compare these approaches, leading to uncertainty as to how to evaluate long-context performance and whether it differs from standard evaluation. We implement a controlled protocol for extension methods with a standardized evaluation, utilizing consistent base models and extension data. Our study yields several insights into long-context behavior. First, we reaffirm the critical role of perplexity as a general-purpose performance indicator even in longer-context tasks. Second, we find that current approximate attention methods systematically underperform across long-context tasks. Finally, we confirm that exact fine-tuning based methods are generally effective within their extension range, whereas extrapolation remains challenging. All codebases, models, and checkpoints are made available open-source via https://github.com/Leooyii/LCEG , promoting transparency and facilitating further research in this critical area of AI development.

## 1 Introduction

The pretraining data scale of large language models (LLMs) has expanded greatly in recent years with open models trained up to 15T tokens (AI@Meta, ). Implementation challenges make it difficult to fully train models with longer context windows during pretraining (Liu et al., ).
Still, long-context windows are considered central, as they enable LLMs to perform tasks that require more extensive textual understanding, such as utilizing information from textbooks (Tanzer et al., ), summarizing novels (Kryściński et al., ), and engaging in many-shot learning (Bertsch et al., ; Li et al., ).

As a trade-off, researchers have proposed context extension, where an LLM initially pretrained on standard sequences is adapted for significantly longer context lengths (Chen et al., ; Peng et al., ; Han et al., ; bloc97, ). These methods differ in the type of attention used and in post-training adaptation techniques. They vary in complexity, training requirements, and qualitatively exhibit significantly different performance profiles.

Unfortunately, there is a relatively poor understanding of the quantitative rankings of these different methodologies. Owing to the perceived challenges of evaluation, several new metrics, such as long context perplexity (Chen et al., , ; Han et al., ), and retrieval accuracy (Mohtashami and Jaggi, ; gkamradt, ) have been introduced (Bai et al., ; An et al., ). However, the differences in long-context extension procedures make it hard to calibrate these metrics while controlling for other factors.

In this work, we implement a controlled protocol for context extension. The aim is to compare context extension while removing spurious factors that impact LLM ability.

Modeling: We standardize on the same base model for all experiments. Different base models behave significantly differently, making it challenging to draw general conclusions. For instance, past work evaluates LM-Infinite (Han et al., ) on LongBench (Bai et al., ) using different base models (Xiao et al., ; Lu et al., ).

Extensions: We implement a range of context extension methods within the same framework. We use a standardized recipe to eliminate potential gains from tailored hyperparameters. We additionally fix the post-training data for each method, utilizing an identical and open-sourced training corpus (Fu et al., ; Chen et al., ).

Metrics: We look at both intrinsic metrics, such as perplexity, and extrinsic properties, such as downstream task performance (Hsieh et al., ; gkamradt, ; Bai et al., ). We consider metrics within the extension length as well as an extrapolation to longer contexts.

Our study identifies several takeaways for future research. First, contrary to some of the arguments for needing new metrics, our findings indicate a strong correlation between perplexity and downstream task performance for exact fine-tuned methods in controlled studies. While some approximate attention methods are unbalanced between perplexity and downstream task performance, there is a strong correlation between the two for most extension tasks.

Second, we find relatively poor results for approximate attention methods. While they can handle longer length contexts, there generally is a trade-off in terms of accuracy for most of our benchmarks. Exact frozen methods also tend to degrade model performance, showing high sensitivity to hyperparameters and often failing with a general training recipe.

Finally, continual fine-tuning with exact attention generally works well, particularly within the extended context length. Specifically, Dynamic NTK (emozilla, ) works best among these methods. Extrapolation to longer lengths remains a challenging problem.
Our training code, models, and checkpoints will be open-sourced to support further research in this area.

## 2 Related Work

##### Long Context Methods

We divide extension methods into three broad classes: exact attention, approximate attention, and context compression.
Exact attention methods augment the parameterization of attention. Position interpolation (PI) (Chen et al., ), NTK-aware (bloc97, ), Dynamic NTK (emozilla, ), YaRN (Peng et al., ), and CLEX (Chen et al., ), all based on RoPE (Su et al., ), design position embeddings for length extension.
These methods may be applied with fine-tuning or to frozen models.
Other exact attention methods focus on training-time improvements, such as contrastive training (Tworkowski et al., ) .
Approximate attention methods introduce structured attention approximations that minimize the computational cost of length extension.
Chen et al. () introduced the use of LoRA (Hu et al., ) and a specialized local attention mechanism to reduce further the computational overhead of further fine-tuning with long context.
Other approaches break the text into chunks and utilize a well-designed "chunk representation" to retrieve relevant chunks for attention (Mohtashami and Jaggi, ; Xiao et al., ; Lu et al., ).
LM-Infinite and StreamLLM (Han et al., ; Xiao et al., ) retain only a few tokens from the beginning of the text and a local window to keep the attention window within the pretrained length.
Xu et al. () focuses on using retrievers to retrieve relevant blocks from long documents.
Finally, context compression methods, which we do not explore in this work, reduce length extension to length compression via a summarization step
(Jiang et al., ; Li et al., ).

##### Long Context Evaluation Benchmarks

The Long Range Arena (LRA) (Tay et al., ) is an early efforts in evaluating the proficiency of processing long contexts in different modalities. Since then, a growing number of benchmarks have emerged, including LongBench (Bai et al., ), LEval (An et al., ), and LooGLE (Li et al., ). These benchmarks are a mixture of diverse downstream tasks explicitly tailored to assess the capabilities of LLMs in understanding and generating lengthy contexts. Among these benchmarks, LongBench stands out for its inclusion of diverse sequences with varying lengths, distributions, patterns, languages, and domains, enabling a comprehensive, nuanced evaluation. In addition to evaluating LLMs’ performance on downstream NLP tasks, there is another line of benchmarks that specifically focuses on assessing particular aspects of long context processing ability (Liu et al., ; Hsieh et al., ). For instance, Mohtashami and Jaggi () propose the passkey retrieval task to challenge a language model to accurately locate and retrieve a simple passkey (a five-digit random number) in a long context sequence. Similarly, the Needle in a Haystack (gkamradt, ) test requires the model to accurately recite the information from a specified sentence(the "needle") from a long document.
However, most existing works mainly focus on evaluating mainstream commercial models (e.g. GPT-4 and Claude), open-source base models, or just perform individual evaluations of a few long context methods. There is a lack of comprehensive, yet controlled evaluation on long-context extension techniques themselves.

## 3 Context Extension Methods

### 3.1 Background: Attention and RoPE

The bottleneck in long context modeling in Transformers is attention. Attention is defined over
$C$ embeddings $\mathbf{X}=[\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{C}]^{\top}\in
\mathbb{R}^{C\times d}$ where $d$ is the model dimension. Learned weight matrices $\mathbf{W}_{v}\in\mathbb{R}^{d\times d_{k}}$, $\mathbf{W}_{q}\in\mathbb{R}^{d\times d_{k}}$, and $\mathbf{W}_{k}\in\mathbb{R}^{d\times d_{k}}$ are used to transform these inputs where $d_{k}$ is the projected hidden dimension. The attention mechanism itself computes the attention matrix and applies it to produce a weighted sum of the value vectors:

$$ $\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathbf{A}\mathbf{V}=\text{ softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d_{k}}}\right)\mathbf{V}.$ (1) $$

Basic attention was originally defined with: $\mathbf{Q}=\mathbf{X}\mathbf{W}_{q},\mathbf{K}=\mathbf{X}\mathbf{W}_{k},
\mathbf{V}=\mathbf{X}\mathbf{W}_{v}$. However, this approach does not directly encode the relative position of keys and values.

Rotary Position Embeddings (RoPE) (Su et al., ) encode positional information by applying a phase rotation to each element of the embedding vectors. Formally, we define a transformation $\mathbf{f}$:

$$ $\mathbf{f}_{\mathbf{W}}(\mathbf{x}_{i},\bm{\theta})=\mathbf{R}(\bm{\theta},i) \mathbf{W}^{\top}\mathbf{x}_{i}$ (2) $$

Here $\mathbf{x}_{i}\in\mathbb{R}^{d_{k}}$ is an embedding for position $i$, $\mathbf{W}$ is a projection matrix, and $\bm{\theta}\in\mathbb{R}^{d_{k}/2}$ is a frequency basis. The function is defined based on the rotary position matrix:

$$ $\mathbf{R}(\bm{\theta},i)=\begin{pmatrix}\cos i\theta_{1}&-\sin i\theta_{1}& \cdots&0&0\\ \sin i\theta_{1}&\cos i\theta_{1}&\cdots&0&0\\ \vdots\\ 0&0&\cdots&\cos i\theta_{\frac{d_{k}}{2}}&-\sin i\theta_{\frac{d_{k}}{2}}\\ 0&0&\cdots&\sin i\theta_{\frac{d_{k}}{2}}&\cos i\theta_{\frac{d_{k}}{2}}\\ \end{pmatrix}$ (3) $$

Due to the arrangement of frequencies, this matrix has the property that $\mathbf{R}(\bm{\theta},n-m)=\mathbf{R}(\bm{\theta},m)^{\top}\mathbf{R}(\bm{
\theta},n)$ by Ptolemy’s identity. We redefine the query-key product between two positions $m$ and $n$ as,

$$ $\displaystyle\mathbf{q}^{\top}_{m}{\mathbf{k}_{n}}$ $\displaystyle=\mathbf{f}_{\mathbf{W}_{q}}(\mathbf{x}_{m},\bm{\theta})^{\top} \mathbf{f}_{\mathbf{W}_{k}}(\mathbf{x}_{n},\bm{\theta})$ (4) $\displaystyle=\left(\mathbf{R}({\bm{\theta},m})\mathbf{W}^{\top}_{q}\mathbf{x} _{m}\right)^{\top}\left(\mathbf{R}(\bm{\theta},n)\mathbf{W}^{\top}_{k}\mathbf{ x}_{n}\right)$ (5) $\displaystyle=\mathbf{x}_{m}^{\top}\mathbf{W}_{q}\mathbf{R}(\bm{\theta},n-m) \mathbf{W}_{k}^{\top}\mathbf{x}_{n}$ (6) $$

In this way, the relative positional information $n-m$
is implicitly injected into the query and key product, thus the attention score.

The standard RoPE transformation, $f_{\mathbf{W}}(x_{i},\bm{\theta})$, sets the components
${\theta}_{j}=b^{-\frac{2j}{d_{k}}}$ with base $b=10000$.

### 3.2 Adjusting the Frequency of RoPE for Long Context Extension

We consider four methods for performing length extension on RoPE embeddings: Position Interpolation (PI) (Chen et al., ), NTK-RoPE (emozilla, ), YaRN (Peng et al., ) and CLEX (Chen et al., ). In this section our goal is to extend a method trained to extend position embeddings for context length $C$ to length $C^{\prime}>>C$. The methods in this section perform this extension by scaling the frequencies with the base scaling vector $\bm{\alpha}\in\mathbb{R}^{\frac{d_{k}}{2}}$:

$$ $\mathbf{f}_{\mathbf{W}}(x_{i})=\mathbf{f}(x_{i},\bm{\alpha}\odot\bm{\theta}).$ (7) $$

Linear Position Interpolation (PI) decreases the frequencies of the basis functions so that more tokens fit within each period.
PI is implemented by setting the components of the base scaling vector to

$$ $\alpha_{j}^{\text{PI}}=\frac{C}{C^{\prime}}=\frac{1}{t}.$ (8) $$

where $t=\frac{C^{\prime}}{C}$ is target length ratio. PI has been integrated into many open-source models such as LLaMA2-7B-32K (Together.AI, ), Vicuna-7B-v1.5 (Chiang et al., ), and LongAlpaca (Chen et al., ).

Neural Tangent Kernel Interpolation RoPE (NTK-RoPE) builds on linear position interpolation by introducing a per-dimension scaling factor. Inspired by findings from the NTK literature that show that high-frequency features are difficult for MLPs to learn, NTK-RoPE preserves high-frequency features while extending the period of low-frequency features. This is accomplished via a dimension-dependent base scaling vector $\bm{\alpha}$:

$$ $\alpha_{j}^{\text{NTK-RoPE}}=\kappa^{-\frac{2j}{d_{k}}},$ (9) $$

where $\kappa=\left(t\right)^{\frac{d_{k}}{d_{k}-2}}$ so that the lowest frequency is scaled to match PI and the highest frequency remains the same as in RoPE.

An extension to this approach, Dynamic NTK-RoPE suggests that instead of fixing scaling based on a set ratio $s$ for all examples during inference, the formula should adapt to the current context length for a specific example. We followed the set up of Fu et al. () for Dynamic NTK-RoPE. More details can be found in the Appendix 9.2.

YaRN, another RoPE extension method, uses “NTK-by-parts" interpolation strategies across different dimensions of the embedding space and introduces a temperature factor to adjust the attention distribution for long inputs.

$$ $\alpha_{j}^{\text{YaRN}}=(\left(1-{\gamma}_{j}\right)\frac{1}{t}+{\gamma}_{j}) /\sqrt{T}$ (10) $$

We use a ramp vector $\bm{\gamma}$ to determine the interpolation between the $\frac{1}{t}$ and the original frequency base. The interpolation gating is set based on the frequency for the dimension $j$.

$$ $\gamma_{j}=\begin{cases}0,&\text{if }{\theta}_{j}<p,\\ 1,&\text{if }{\theta}_{j}>q,\\ \frac{{\theta}_{j}-p}{q-p},&\text{otherwise}.\end{cases}$ (11) $$

The values of $p,q,T$ can be tuned as needed.

Other methods such as CLEX Chen et al. () models the scaling vectors as a dynamical system, intending to learn target-length dependent scaling vectors.

### 3.3 Approximate Attention

An alternative approach is to modify the attention function itself. Instead of exactly computing each longer attention, these methods select a subset of positions to attend to. We consider four well-known methods based on three different attention mechanisms: sparse attention, sliding window attention, and retrieval attention.

LongLoRA (Chen et al., ) avoids computing attention ranges over $C^{\prime}$ by only computing the block-diagonl part of attention. Formally, given a sequence length of $C^{\prime}$, LongLoRA divides it into $M$ blocks of size $B$, resulting in a sparse attention matrix $\mathbf{A}\in\mathbb{R}^{C^{\prime}\times C^{\prime}}$ with a block-diagonal structure:

$$ $\mathbf{A}=\begin{bmatrix}\mathbf{A}_{1}&\mathbf{0}&\cdots&\mathbf{0}\\ \mathbf{0}&\mathbf{A}_{2}&\cdots&\mathbf{0}\\ \vdots&\vdots&\ddots&\vdots\\ \mathbf{0}&\mathbf{0}&\cdots&\mathbf{A}_{M}\end{bmatrix}$ (12) $$

where $\mathbf{A}_{i}\in\mathbb{R}^{B\times B}$ is the attention matrix for the $i$-th block. In addition, they shift the blocks for half of the heads enabling the information flow between groups via shifting. Notably, while they employ local attention during the fine-tuning phase, full attention is still adopted during the inference stage.

Landmark Attention (Mohtashami and Jaggi, ) addresses the challenge of attending over long sequences by breaking the input sequence into chunks and using trainable “landmark" tokens to summarize these chunks. The attention process is carried out in two stages. Given a sequence of $C^{\prime}$ embeddings, divided into $M$ chunks, each of length $B$, the first step is to compute global attention between the query vectors $\mathbf{Q}\in\mathbb{R}^{C^{\prime}\times d_{k}}$ (corresponding to all input embedding) and the landmark vectors $\mathbf{L}\in\mathbb{R}^{M\times d_{k}}$ (which represent the chunks). From this global attention, a set of $n$-most attended-to chunks is selected for further processing. Next, a local attention mechanism is applied within each of the selected chunks. For the $n$-th selected chunk, the key matrix for the chunk is denoted as $\mathbf{K}_{n}\in\mathbb{R}^{B\times d_{k}}$ and $\mathbf{Q}_{n}\in\mathbb{R}^{B\times d_{k}}$. The attention matrices are then computed as follows:

$$ $\displaystyle\mathbf{A}^{1}$ $\displaystyle=\text{softmax}\left(\frac{\mathbf{Q}\mathbf{L}^{T}}{\sqrt{d_{k}} }\right)\in\mathbb{R}^{C^{\prime}\times M},$ (13) $\displaystyle\mathbf{A}^{2,n}$ $\displaystyle=\text{softmax}\left(\frac{\mathbf{Q}_{n}\mathbf{K}_{n}^{T}}{ \sqrt{d_{k}}}\right)\in\mathbb{R}^{B\times B},$ (14) $$

The final attention for each embedding is a combination of these two attention. This method efficiently scales attention mechanisms for long sequences by focusing on landmark tokens that summarize large parts of the sequence, followed by local attention within the relevant chunks.

LM-Infinite (Han et al., ) (a.k.a., Sliding Window Attention) maintains a sliding local window of size $M$ along with a fixed global memory of $G$ positions at the starting point of the given embedding. Given $C^{\prime}$ embeddings, attention is computed over the $M$ embeddings in its local window and $G$ embeddings in global memory. LM-Infinite replaces relative positional information $n-m$ with $\min(n-m,C)$ while computing the query and key product in Eq 4. Altogether, LM-Infinite reduces the complexity from $\mathcal{O}((C^{\prime})^{2})$ to $\mathcal{O}(C^{\prime}(M+G))$ without the need to scale positional encoding.

Self-Extend (Jin et al., ) maps the unseen positions in extended context length $C^{\prime}$ to positions within the pretraining context length $C$ to avoid training.
For each embeddings, Self-Extend chooses closest $M$ embeddings and any embeddings beyond are divided into multiple groups. Each group contains $N$ embeddings. When performing query-key product between two positions $m$ and $n$ in Equation 4, the relative positional information $n-m$ is replaced by $r$ which is computed by scaling $n-m$ w.r.t $M$ and $N$:

$$ $r=\begin{cases}n-m,&n-m\leq M,\\ M+\left\lfloor\frac{n-m}{N}\right\rfloor-\left\lfloor\frac{M}{N}\right\rfloor, &n-m>M.\\ \end{cases}$ (15) $$

where $\lfloor\cdot\rfloor$ denotes the floor division. The maximum extended context length $C^{\prime}$ is $(C-M)\cdot N+M$.

## 4 Long-Context Extension Protocol

Base Model
All models start from an identical LLaMA2-7B base checkpoint (Touvron et al., ). This approach removes potential biases from the checkpoint basis. Note that to study the influence of base model, particularly the size of base model, we also use Phi-2-base checkpoint (Javaheripi et al., ) for context extension, to verify whether the trends and analyses we observed are consistent across different base models, thereby avoiding potential over-generalization. The results of Phi-2-base are reported in Appendix 9.1.

Fine-Tuning
We sample 1B tokens from a long-context data mixture from Fu et al. () to achieve state-of-the-art context extension performance. The details of the data are reported in Appendix 9.3. We focus on extending the context window from 4k to 32k, as most downstream tasks require contexts under 32k.

We maintain a fixed training recipe to ensure consistency across all models (Chen et al., ). We follow existing practices by keeping an exponential moving average (EMA) of model weights with a constant decay. Most training hyperparameters are based on (Fu et al., ), with the learning rate set to $2\times 10^{-5}$. We use a linear learning rate warm-up and zero weight decay, utilizing 8 NVIDIA A100 GPUs.

For LongLora, we fine-tune the weights of the LoRA adapter with trainable embedding and normalization, then merge these trainable weights with the LLaMA2 base model for evaluation. For Landmark Attention, the training context length is set to 512, with a block size of 64. For CLEX, we set the max scale factor to 32 and use the SiLU activation function.

We reuse the original scale factor to maintain consistency for NTK, YaRN, and Position Interpolation methods. However, this base factor significantly degrades continual fine-tuned models, particularly causing performance deterioration in shorter sequences. Therefore, we conduct a grid search to determine a better scale factor for different input lengths for NTK-RoPE method. Based on our findings, we follow and improve upon Fu et al. () to set the scale factor for NTK-RoPE method. The scale factor and its relationship with perplexity are reported in the Appendix 9.5. Please refer to the Appendix 9.2 for detailed hyperparameter setups.

Metrics
We consider two sets of intrinsic metrics. The first is based on perplexity.
We use the book corpus PG19  (Rae et al., ) and the Proof-pile dataset (Azerbayev et al., ) to evaluate the long sequence language modeling performances. Following Press et al. (), all perplexity evaluations are calculated using a sliding window with a window size of 256.

The second is based on retrieval. We focus on the needle in the haystack task (gkamradt, )(NIAH). NIAH involves identifying a specific, relevant piece of information (the "needle") within a large set of irrelevant data (the "haystack"). This task is commonly used to test the precision and recall capabilities of LLMs in scenarios where the relevant data is sparse and surrounded by a significant amount of noise. Additionally, we evaluate with RULER (Hsieh et al., ). RULER enhances the standard NIAH test by incorporating variations with different types and quantities of needles. Additionally, it introduces new task categories, such as multi-hop tracing and aggregation, to evaluate behaviors beyond simple context-based searching.

For extrinsic metrics, we consider a collection of tasks. LongBench (Bai et al., ) is a family of bilingual, multitask evaluations for long-context understanding widely used in measuring the long-context abilities of LLMs (Jin et al., ; Xiao et al., ; Lu et al., ).
LongBench includes single-document question answering, multi-document QA, summarization, few-shot learning, and code completion.
We follow Bai et al. () to evaluate the models on 32k context window sizes by truncating the prompt from the middle when the task length exceeds a designated context window size. We also consider the ManyShots tasks, where the long-context model will be given several examples as prompts. We use the Trec News (Kontonis et al., ) dataset for this task.

## 5 Experimental Results

### 5.1 Result Overview

**Table 1: Overview of results across different extension types.**
| Attention Mechanisms | Model | PPL | Needle | Mshots | LongB | RULER |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Exact<br>Attention | Frozen | NTK-F | 14.52 | 18.8 | 64.5 | 25.54 | 00.72 |
| Fine-Tuned | PI | 5.85 | 42.1 | 75.5 | 33.48 | 57.66 |  |
| YaRN | 5.85 | 46.7 | 75.0 | 33.45 | 36.95 |  |  |
| CLEX | 5.82 | 71.1 | 74.0 | 33.48 | 52.17 |  |  |
| NTK-32K | 5.79 | 83.7 | 71.0 | 35.32 | 59.42 |  |  |
|  |  | NTK-64K | 5.93 | 69.1 | 73.0 | 34.30 | 60.03 |
| Approxi.<br>Attention | Frozen | LM-Infinite | 6.71 | 23.9 | 61.5 | 25.84 | 12.34 |
|  | Self-Extend | 6.11 | 25.8 | 72.0 | 33.62 | 29.50 |  |
| Fine-tuned | LongLora | 9.89 | 20.3 | 55.5 | 23.30 | 03.53 |  |
|  |  | Landmark | 8.13 | 50.9 | 50.0 | 28.19 | 13.56 |

**Table 2: Perplexity results of different methods on PG 19 and Proof-file. NTK-32K and NTK-64K refer to NTK-Dynamic, which requires finetuning on longer text. Len refers to the longest-length examples seen at training or fine-tuning. Ex refers to the exact attention. All results are produced by our experiments.**
| Model Details | Eval Length |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Len | Ex | Methods | 2k | 4k | 8k | 16k | 32k | 64k |
| PG19 |  |  |  |  |  |  |  |  |  |
| Frozen | 4k | ✓ | LLaMA2 | 6.61 | 6.30 | - | - | - | - |
| 4k |  | LM-Infinite | 6.61 | 6.30 | 6.25 | 6.45 | 6.71 | 8.49 |  |
| 4k | ✓ | NTK-Frozen | 6.61 | 6.30 | 6.82 | 7.94 | 14.52 | - |  |
|  | 4k |  | Self-Extend | 6.61 | 6.32 | 6.15 | 6.07 | 6.11 | 7.15 |
| Finetuned | 32k | ✓ | PI | 6.88 | 6.52 | 6.27 | 6.08 | 5.95 | - |
| 32k | $\checkmark$ | NTK-32K | 6.63 | 6.32 | 6.09 | 5.92 | 5.79 | 5.76 |  |
| 32k | $\checkmark$ | YaRN | 6.70 | 6.39 | 6.16 | 6.01 | 5.93 | - |  |
| 32k | $\checkmark$ | CLEX | 6.85 | 6.62 | 6.14 | 5.93 | 5.82 | 5.79 |  |
| 32k |  | LongLora | 12.80 | 11.52 | 10.70 | 10.18 | 9.89 | - |  |
| 32k |  | Landmark | 8.15 | 8.14 | 8.14 | 8.11 | 8.13 | 8.15 |  |
|  | 64k | ✓ | NTK-64K | 6.83 | 6.49 | 6.25 | 6.07 | 5.93 | 5.85 |
| Proof-file |  |  |  |  |  |  |  |  |  |
| Frozen | 4k | $\checkmark$ | LLaMA2 | 3.34 | 3.04 | - | - | - | - |
| 4k |  | LM-Infinite | 3.34 | 3.04 | 2.94 | 3.02 | 3.11 | 3.12 |  |
| 4k | ✓ | NTK-Frozen | 3.34 | 3.04 | 2.91 | 3.09 | 4.06 | 12.65 |  |
|  | 4k |  | Self-Extend | 3.35 | 3.06 | 2.88 | 2.78 | 2.75 | 2.90 |
| Finetuned | 32k | ✓ | PI | 3.34 | 3.03 | 2.83 | 2.68 | 2.58 | - |
| 32k | ✓ | NTK-32K | 3.27 | 2.98 | 2.78 | 2.64 | 2.54 | 2.48 |  |
| 32k | ✓ | YaRN | 3.29 | 3.00 | 2.81 | 2.68 | 2.59 | 106.38 |  |
| 32k | $\checkmark$ | CLEX | 3.37 | 3.10 | 2.80 | 2.65 | 2.55 | 2.48 |  |
| 32k |  | LongLora | 5.97 | 5.10 | 4.58 | 4.27 | 4.13 | - |  |
| 32k |  | Landmark | 4.51 | 4.50 | 4.48 | 4.49 | 4.49 | 4.49 |  |
|  | 64k | ✓ | NTK-64K | 3.33 | 3.03 | 2.83 | 2.69 | 2.58 | 2.51 |

Table 1 overviews the results across both types of evaluation. The main result demonstrate that fine-tuned exact attention methods for long contexts, such as NTK-32K and YARN, consistently outperform approximate attention methods and frozen methods by a significant margin. This suggests that trading accuracy for speed in approximate attention methods can result in a loss of important reasoning capabilities, particularly for retrieval-based tasks. The performance disparity highlights the importance of exact attention mechanisms in maintaining high accuracy over extended contexts, emphasizing the need for careful consideration of attention mechanisms in model design for long-context tasks. We now consider each type of result in more detail.

### 5.2 Intrinsic tasks

##### Perplexity

Table 2 shows perplexity scores across length. We see that continuous fine-tuning methods like PI, YaRN, and LongLora effectively keep low perplexity scores within the pre-training context length. However, when the context length exceeds perplexity scores escalate once the context surpasses the pre-trained window. Only NTK and CLEX can generalize to unseen sequence length in both pretraining and continual finetuning. Additionally, we find that exact attention maintains better perplexity than LoRA, which may reduce LongLora’s ability. We also note that results on both PG19 and Proof-file gave nearly consistent conclusions.

Figure: Figure 1: Needle in a Haystack evaluation. Green squares indicates a high retrieval success rate, the white dashed line denotes the longest length examples seen at training or finetuning, and the Y-axis represents the distance to the retrieved target.
Refer to caption: x1.png

**Table 3: RULER evaluation. Performance of models evaluated at length from 4k to 64k. Each score is computed by averaging the accuracy of 13 tasks. Train Len refers to the longest-length examples seen at continuous finetuning.**
|  | Models | Train<br>Len | 4k | 8k | 16k | 32k | 64k |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Frozen | LLaMA2 | 4k | 80.94 | - | - | - | - |
| LM-Infinite | 4k | 81.05 | 30.01 | 18.02 | 12.34 | 10.56 |  |
| NTK-Frozen | 4k | 81.14 | 44.45 | 14.79 | 0.72 | 0.91 |  |
|  | Self-Extend | 4k | 65.03 | 50.73 | 44.02 | 29.50 | 9.34 |
| Finetuned | PI | 32k | 84.56 | 76.04 | 69.64 | 57.66 | 0.00 |
| NTK-32K | 32k | 86.58 | 77.75 | 70.01 | 59.42 | 46.26 |  |
| YaRN | 32k | 79.12 | 65.60 | 54.21 | 36.95 | 0.00 |  |
| CLEX | 32k | 50.18 | 63.93 | 64.35 | 52.17 | 30.61 |  |
| LongLora | 32k | 10.58 | 6.37 | 3.67 | 3.53 | 0.00 |  |
| Landmark | 32k | 22.37 | 17.52 | 16.31 | 13.56 | 14.15 |  |
|  | NTK-64K | 64k | 86.60 | 76.34 | 69.56 | 60.03 | 49.31 |

##### Needle-in-the-haystack

NIAH results are shown in Figure 1.
Continuous finetuning approaches such as NTK, PI, and YaRN have successfully retrieved the "needle" within the pretraining length. Yet, only the NTK and CLEX method can retrieve the needle beyond the pretraining length, aligning with the perplexity results. The performance of the Exact Attention Method generally surpasses that of the Approximate Attention Methods. LM-Infinite and Landmark Excel are only within the local window, and they struggle to retrieve the intermediate text accurately. Regarding the Dynamic NTK method, NTK-F exhibits weak generalization when not trained. When trained on the same amount of data(1B), NTK-32K outperforms NTK-64K. When trained on 2B tokens, NTK-64K demonstrated a significant performance improvement, details are in Appendix 9.4.

**Table 4: LongBench results. N-32 and N-64 refer to NTK finetuned on 32K and 64K context lengths respectively. SE refers to Self-Extend. YN refers to YaRN. CX refers to CLEX. LLR refers to LongLora. Len refers to average length of the datasets. Train Len refers to the longest length examples seen at training or finetuning. Eval Len refers to the maximum length of the input prompt. ✓refers to whether the method is exact attention.**
|  |  | Frozen | Finetuned |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Len | Base | N-F | SE | PI | N-32 | YN | CX | LLR | Land | N-64 |
|  |  | ✓ | ✓ |  | ✓ | ✓ | ✓ |  |  | ✓ |  |
| Train Len | 4k | 4k | 4K | 32k | 32k | 32k | 32k | 32k | 32k | 64k |  |
| Eval Len | 4k | 32k | 32K | 32k | 32k | 32k | 32k | 32k | 32k | 32k |  |
| NQA 18k | 21.09 | 3.88 | 23.49 | 23.02 | 23.73 | 19.82 | 24.19 | 12.07 | 12.47 | 24.31 |  |
| QAP 4k | 26.94 | 26.79 | 28.75 | 25.85 | 27.50 | 26.98 | 23.36 | 20.15 | 19.06 | 24.97 |  |
| MQA 5k | 32.42 | 29.82 | 32.66 | 35.10 | 38.22 | 37.11 | 40.83 | 24.50 | 21.86 | 40.60 |  |
| HQA 9k | 31.23 | 32.10 | 37.63 | 36.98 | 41.56 | 38.60 | 35.59 | 27.41 | 33.66 | 41.47 |  |
| WQA 5k | 25.75 | 22.34 | 30.70 | 29.38 | 31.58 | 30.63 | 28.24 | 21.46 | 24.94 | 28.62 |  |
| MSQ 11k | 10.55 | 8.84 | 15.73 | 16.80 | 17.41 | 22.08 | 17.12 | 11.46 | 11.41 | 18.24 |  |
| GR 9k | 17.32 | 17.87 | 13.15 | 25.61 | 28.27 | 20.98 | 24.68 | 24.05 | 17.20 | 24.37 |  |
| QSM 11k | 21.28 | 15.35 | 20.20 | 21.19 | 21.52 | 20.66 | 21.55 | 17.66 | 18.83 | 21.65 |  |
| MWS 2k | 3.44 | 9.30 | 1.50 | 10.55 | 22.13 | 8.91 | 16.96 | 21.19 | 19.43 | 25.02 |  |
| TRE 5k | 66.00 | 67.50 | 69.00 | 71.00 | 69.00 | 69.00 | 67.50 | 50.00 | 49.00 | 69.00 |  |
| TQA 8k | 87.89 | 18.69 | 88.44 | 88.55 | 88.86 | 89.63 | 89.36 | 12.28 | 74.75 | 88.65 |  |
| SMS 6k | 41.70 | 32.46 | 43.76 | 43.35 | 42.21 | 44.25 | 43.02 | 13.45 | 40.38 | 41.59 |  |
| PSC 11k | 2.10 | 2.67 | 0.00 | 1.50 | 2.68 | 1.05 | 2.50 | 4.57 | 0.64 | 2.09 |  |
| PSR 9k | 9.00 | 3.77 | 4.50 | 4.50 | 4.62 | 3.79 | 8.50 | 3.50 | 2.50 | 6.50 |  |
| LCC 1k | 68.22 | 63.64 | 68.47 | 55.05 | 56.78 | 54.06 | 49.45 | 57.12 | 56.70 | 52.04 |  |
| REP 4k | 61.73 | 53.69 | 59.99 | 47.26 | 49.09 | 47.60 | 42.84 | 51.92 | 48.23 | 39.68 |  |
| Avg. 7k | 32.92 | 25.54 | 33.62 | 33.48 | 35.32 | 33.45 | 33.48 | 23.30 | 28.19 | 34.30 |  |

##### RULER

We test all models on 13 diverse tasks from the four RULER categories (Hsieh et al., ). Each model is evaluated with 500 examples for lengths of 4k, 8k, 16k, 32k, and 64k. Results are compared with the Llama2-7B baseline in Table 3. We observed a similar trend as in the NIHK task, where NTK family methods performed the best. NTK-32k maintained relatively good performance compared to other methods finetuned with a length cap of 32k. Performance of models on different length and breakdown by 13 subtasks can be found in Appendix 9.8.

### 5.3 Extrinsic tasks

##### LongBench

The evaluation results of most methods on LongBench are presented in Table 4, and results on all methods are presented in Appendix 9.7. Both LM-Infinite and Landmark Attention exhibit significant performance degradation compared to the base model. In contrast, the NTK, PI, and YaRN methods have successfully maintained their performance at 32k, demonstrating comparable results among these methods. This suggests that PI and YaRN perform similarly in downstream tasks, while the NTK family of models remains stable.

Notably, the LongLoRA method, which utilizes LoRA, also experiences a performance decline relative to the base checkpoint, LLaMA2. We argue that this may be due to the sensitivity of the training procedures for LongLoRA, and we acknowledge this in our limitation discussion section.

Furthermore, the overall performance on LongBench has not shown significant improvement over LLaMA2.We hypothesize that this is due to the average length of LongBench test data (approximately 7.5k) being considerably shorter than the 32k context window of the long-context methods.

Figure: Figure 2: Many-shot in-context learning accuracy on TREC News.
Refer to caption: x2.png

##### Many-shot In-Context Learning with Trec News

We evaluate TREC News (Li and Roth, ) with 1 to 1000 in-context examples. Performance improves with more examples in Figure 2. Exact Attention methods show significant gains from 10 to 50 examples (+44.0%) and 100 to 1000 examples (+25.9%), with slower growth from 50 to 100 examples (+5.7%). Approximate Attention methods consistently underperform. Performance gains align with model perplexity in longer contexts; NTK-Frozen excels with fewer examples but underperforms with more.

## 6 Analysis

##### Perplexity and Downstream Tasks

Several existing studies (Sun et al., ; An et al., ) suggest that perplexity may not consistently correlate with performance on long-range tasks. In Figure 4, we plot the perplexity of models from our evaluated benchmarks against their performance. The figure shows a general correlation between perplexity and model performance across various tasks for exact attention methods. However, approximate attention methods, LongLora, and Landmark on RULER, exhibit slight deviations from this trend but still roughly fit into the linear relationship. We hypothesize that previous studies mentioned this disconnection due to the lack of controlled studies and the presence of noisy data.

Figure: Figure 3: Averaged negative log-likelihood of different models broken down by context position.
Refer to caption: extracted/5873643/figures/ppl_scores_plot.png

As depicted in Figure 1, although LM-Infinite achieves a good perplexity score at 32k, it fails to generalize beyond 4k. This limitation arises because the LM-Infinite method focuses on a context window of only 4k, resulting in poor retrieval ability for longer contexts. This suggests that we should consider downstream tasks at different lengths when evaluating perplexity for approximate attention methods.

Figure: Figure 4: Perplexity and averaged downstream task accuracy for Needle in a haystack, LongBench and RULER.
Refer to caption: x3.png

##### Context extension hurts in the short term and gains in the long term

While context extension seems to improve perplexity, in Table 4, we do not notice a significant gain in performance.

We hypothesize that while this dataset contains long tasks, the average length is much shorter than 32K. These methods seem to improve the ability to model language over the long term but hurt in the short term. To understand this better we compute the averaged negative likelihood of each position of YaRN, LLaMa2, and NTK-32K per position (with LLaMa2 seeing just tokens every 4k chunks) in Figure 3.

##### NTK Generalizes Beyond 32k

In Figure 1, we observe that NTK-32K successfully generalizes to unseen sequence lengths beyond 32k in both Needle-in-the-Haystack and RULER tasks, performing on par with NTK-64K. In contrast, NTK-F demonstrates generalization up to 8k but fails to extend further. This suggests that while NTK methods may possess the capability to generalize to longer unseen sequences, their effectiveness is contingent upon conditions such as continual fine-tuning.

We find that up until 4K they all improve as expected with LLaMa2 having the best NLL. After 4K they all fluctuate in average, but we see a clear separation with Yarn and NTK taking into account the long context. At extremely long context NTK remains a strong model whereas Yarn becomes reverts to a similar performance as LLaMa2.

## 7 Limitations and Broader Impacts

Limitations As we are limited by computing budget, we only experiment with Llama-2-7B as our base model. The findings in this paper may not generalize to other, possibly larger, base models. We only fine-tuned models to context sizes of 32k, and generalization behaviors to longer contexts may differ when training contexts are longer. We also acknowledge that the standardized training recipe with fixed hyperparameters may bias some models more than the other models.

Broader Impacts
This paper provides an in-depth evaluation of various methods for extending the context lengths of LLMs, offering insights into their performance in long-range settings. By standardizing evaluations with a consistent base model and extension data, this work provides clearer benchmarks for future work. We also open-source code and models to promote transparency. Our models share similar risks as standard LLMs, where they may be used to generate harmful content and misinformation.

## 8 Conclusion

In this paper, we used a standardized approach to assess the performance of various long-context methods in LLMs.
Our study underscores the role of perplexity as a crucial, performance indicator at length and highlights the trade-offs inherent in different attention mechanisms.
We shed light on the strengths and weaknesses of various approaches, providing valuable insights for future research.
As part of our commitment to open science, all our resources, including codebases, models, and checkpoints, will be open-sourced upon acceptance, fostering future advancements in this pivotal area of AI research.

## 9 Acknowledgements

We thank Yao Fu, Yue Yu, Tianyu Gao, Celine Lee, Woojeong Kim, Jack Morris, Junxiong Wang, and Oscar Yin for their suggestions and feedback. This work was supported by NSF CAREER #2037519 and NSF #1901030.

## Appendix

### 9.1 Phi-2 for Context Extension

We use another open-weight model, Phi-2-base Javaheripi et al. [] as the base point for context extension, to verify whether the trends and analyses we observed are consistent across different base models.
Using an identical training recipe, we re-train and re-evaluate seven models with Phi-2-base.

#### 9.1.1 Perplexity on Proof-file of Phi-2-base

We evaluate the perplexity score of Phi-2-base on seven models in Table 5. Consistent with our observations in the original submission, continuous fine-tuning methods like PI, YaRN, and LongLora effectively maintain low perplexity scores within the pre-training context length. However, perplexity scores escalate once the context length exceeds the pre-trained window. Notably, only NTK could generalize to unseen sequence lengths during both pre-training and continual fine-tuning.

**Table 5: Perplexity results of different methods on Proof-file with Phi-2-base. NTK-32K and NTK-64K refer to NTK-Dynamic, which requires finetuning on longer text. Len refers to the longest-length examples seen at training or fine-tuning. Ex refers to the exact attention. All results are produced by our experiments.**
| Model Details | Eval Length |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Len | Ex | Methods | 2k | 4k | 8k | 16k | 32k | 64k |
| Frozen | 2k | $\checkmark$ | Phi-2-base | 4.02 | 25.72 | 175.05 | - | - | - |
| 2k | ✓ | NTK-Frozen | 4.02 | 3.73 | 4.07 | 5.49 | 12.58 | 36.68 |  |
| 2k |  | Self-Extend | 4.08 | 3.70 | 3.48 | 3.42 | 3.48 | 3.73 |  |
| Finetuned | 32k | ✓ | PI | 7.53 | 6.75 | 6.25 | 5.97 | 5.83 | 45.00 |
| 32k | ✓ | NTK-32K | 4.24 | 3.81 | 3.51 | 3.32 | 3.18 | 3.20 |  |
| 32k | $\checkmark$ | CLEX | 5.53 | 4.32 | 3.78 | 3.51 | 3.42 | 3.60 |  |
|  | 64k | ✓ | NTK-64K | 4.63 | 4.14 | 3.82 | 3.61 | 3.47 | 3.38 |

#### 9.1.2 RULER of Phi-2-base

We test all models on 12 diverse tasks (except QA-2) from the four Ruler Hsieh et al. [] categories in Table 6. Consistently, NTK-32k maintains relatively strong performance compared to other methods fine-tuned with a length cap of 32k. The only exception was CLEX and NTK-32k at 64k, showing a slight drop in performance.

**Table 6: RULER evaluation on seven methods with Phi-2-base. Performance of models evaluated at length from 2k to 64k. Each score is computed by averaging the accuracy of 12 tasks. Train Len refers to the longest-length examples seen at continuous finetuning.**
|  | Models | Train<br>Len | 2k | 4k | 8k | 16k | 32k | 64k |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Frozen | Phi-2-base | 2k | 83.73 | - | - | - | - | - |
| NTK-Frozen | 2k | 83.98 | 52.95 | 18.09 | 4.07 | 0.06 | 0.00 |  |
| Self-Extend | 2k | 68.55 | 50.82 | 36.65 | 22.00 | 7.83 | 2.32 |  |
| Finetuned | PI | 32k | 25.51 | 23.19 | 16.88 | 14.99 | 4.78 | 0.00 |
| NTK-32K | 32k | 81.18 | 66.90 | 52.57 | 46.53 | 32.06 | 12.84 |  |
| CLEX | 32k | 75.33 | 72.66 | 53.56 | 46.23 | 25.46 | 13.03 |  |
|  | NTK-64K | 64k | 78.73 | 59.87 | 47.56 | 41.87 | 25.66 | 17.69 |

### 9.2 Implementation Details

##### Training

To maintain consistency across all models, we use a fixed training protocol [Chen et al., ]. We adopt standard practices by applying an exponential moving average (EMA) to the model weights with a constant decay rate. The majority of our training hyperparameters are derived from [Chen et al., ], including a learning rate of $2\times 10^{-5}$. We implement a linear warm-up for the learning rate and set the weight decay to zero, utilizing 8 NVIDIA A100 GPUs.

For LongLora, we fine-tune the LoRA adapter weights along with trainable embeddings and normalization, subsequently integrating these trained weights into the LLaMA2 base model for evaluation. For Landmark Attention, the training context length is 512, with a block size of 64. For CLEX, we set the max scale factor to 32 and use the SiLU activation function.
We present the hyperparameter settings for different methods during the training stage in Table 7.

##### Inference

We list the scale factors used for different length ranges during inference in Table 8.
For NTK-RoPE, given the maximum observed length during training or inference, $C_{\text{test}}$, and the scaling hyperparameter $s$, we follow Fu et al. [] in replacing $t$ with $s\cdot\frac{\max(C^{\prime},C_{\text{test}})}{C}-(s-1)$, and set the hyperparameter $s$ to $\frac{C^{\prime}}{2C}$ during both training and inference.
For LM-infinite, we set the global memory $G=10$ and the local window $M=4096$.
For Landmark Attention, the training context length is set to 512, with a block size of 64.
For Self-Extend, we set the local window size $M$ for neighbor tokens to 1024 and the group size $N$ to 64.

**Table 7: Hyperparameters for Different Long Sequence Methods in Training.**
| Models | Train<br>Len | Train<br>Tokens | $\alpha$ | bsz | lr |
| --- | --- | --- | --- | --- | --- |
| PI | 32k | 1B | 8.0 | 32 | 2e-5 |
| NTK-32K | 32k | 1B | 29.0 | 32 | 2e-5 |
| YaRN | 32k | 1B | 8.0 | 32 | 2e-5 |
| LongLora | 32k | 1B | 8.0 | 32 | 2e-5 |
| Landmark | 32k | 1B | - | 32 | 2e-5 |
| NTK-64K | 64k | 1B | 57.0 | 32 | 2e-5 |

**Table 8: Hyperparameters for the Scale Factor Different Long-context Methods in Inference.**
| Models | 4k | 8k | 16k | 32k | 64k |
| --- | --- | --- | --- | --- | --- |
| Llama2 | - | - | - | - | - |
| LM-Infinite | - | - | - | - | - |
| NTK-Frozen | 1.0 | 3.0 | 7.0 | 15.0 | 31.0 |
| PI | 8.0 | 8.0 | 8.0 | 8.0 | 8.0 |
| NTK-32K | 29.0 | 29.0 | 29.0 | 29.0 | 61.0 |
| YaRN | 8.0 | 8.0 | 8.0 | 8.0 | 8.0 |
| LongLora | 8.0 | 8.0 | 8.0 | 8.0 | 8.0 |
| Landmark | - | - | - | - | - |
| NTK-64K | 57.0 | 57.0 | 57.0 | 57.0 | 57.0 |

### 9.3 Training Data Construction

We sample 1B tokens from a long-context data mixture following Fu et al. [].
We use the SlimPajama [Soboleva et al., ] dataset for continuous finetuning.
This dataset serves as an open-source replication of the LLaMA [Touvron et al., ] pretraining data mixture. It comprises 82% web data (sourced 67% from CommonCrawl and 15% from C4), 4.5% code data (Github), 4.5% Wikipedia content, 4.5% books, 2.5% Arxiv papers, and 2.0% StackExchange content.
We use per-source length-upsampling to sample 1B tokens from the datasets, which increases the portion of long sequences while keeping the domain mixture the same.
We packed all sampled data into chunks of the corresponding training length, regardless of document boundaries, following common practiceTouvron et al. [], Fu et al. [].

### 9.4 Longer Model Needs more Training Tokens

We observe that the performance of NTK-64K is not as good as NTK-32K. Consequently, we further sample 2B tokens from a long-context data mixture from Fu et al. [] for training and evaluate the model on the "Needle in A Haystack" task, as shown in Figure 5. Our NTK-64K model demonstrates a significant performance improvement when trained with more tokens, indicating that longer models require more tokens for effective training.

Figure: Figure 5: Needle in a Haystack evaluation. “NTK-64-2B” represents the NTK-64K model trained with 2B tokens. Green squares indicates a high retrieval success rate, the white dashed line denotes the longest length examples seen at training or finetuning, and the Y-axis represents the distance to the retrieved target.
Refer to caption: x4.png

### 9.5 RoPE Scale Factor for Dynamic NTK

We observe that the scale factor significantly degrades NTK-Dynamic models, particularly causing performance deterioration in shorter sequences. Therefore, we conduct a grid search to determine a better scale factor for different input lengths.
The scale factor and its relationship with perplexity on PG19 are reported in Table 9.

**Table 9: The scale factor and its relationship with perplexity on PG19. We only use the first 2 documents of PG19 to calculate the perplexity.**
| Models | Scale Factor | 4k | 8k | 16k | 32k | 64k |
| --- | --- | --- | --- | --- | --- | --- |
| NTK-Frozen | 1 | 7.65 | 118.82 | NaN | NaN | NaN |
| 3 | 8.19 | 7.99 | 57.15 | 386.02 | NaN |  |
| 7 | 9.39 | 9.26 | 9.61 | 72.62 | 486.13 |  |
| 15 | 11.53 | 12.04 | 12.98 | 20.15 | 180.59 |  |
| 31 | 16.18 | 20.66 | 26.67 | 40.06 | 69.01 |  |
| 63 | 30.22 | 48.78 | 69.89 | 89.75 | 118.59 |  |
| NTK-32K | 1 | 12.64 | NaN | NaN | NaN | NaN |
| 5 | 7.84 | 7.638 | 10.36 | NaN | NaN |  |
| 13 | 7.686 | 7.459 | 7.25 | 8.35 | NaN |  |
| 29 | 7.689 | 7.457 | 7.24 | 6.82 | 9.11 |  |
| 61 | 7.8 | 7.565 | 7.34 | 6.91 | 6.63 |  |
| 125 | 7.99 | 7.774 | 7.57 | 7.13 | 6.83 |  |
| NTK-64K | 1 | 19.16 | NaN | NaN | NaN | NaN |
| 9 | 8.02 | 7.79 | 7.63 | 22.6 | NaN |  |
| 25 | 7.89 | 7.65 | 7.443 | 7.04 | 14.02 |  |
| 57 | 7.922 | 7.67 | 7.44 | 7.01 | 6.75 |  |
| 121 | 8.016 | 7.75 | 7.51 | 7.06 | 6.77 |  |

### 9.6 LongLora Validation

To validate our LongLora Chen et al. [] implementation, we reproduced their Llama-2-7b-longlora-32k model following LongLora’s training data and training recipe. We evaluated the perplexity for the corresponding length on PG19 and Proof-file in Table 10.

**Table 10: Perplexity results of LongLora reported and our reproduction on PG 19 and Proof-file.**
| Method | 2k | 4k | 8k | 16k | 32k |
| --- | --- | --- | --- | --- | --- |
| PG19 |  |  |  |  |  |
| Llama-2-7b-longlora-32k | 8.29 | 7.83 | 7.54 | 7.35 | 7.22 |
| Our Reproduction | 8.10 | 7.69 | 7.43 | 7.28 | 7.32 |
| Proof-file |  |  |  |  |  |
| Llama-2-7b-longlora-32k | 3.35 | 3.01 | 2.78 | 2.61 | 2.50 |
| Our Reproduction | 3.33 | 3.01 | 2.80 | 2.67 | 2.61 |

### 9.7 LongBench Results

The evaluation results of all methods on LongBench are presented in Table 11.

**Table 11: LongBench results. N-32 and N-64 refer to NTK finetuned on 32K and 64K context lengths respectively. Inf refers to LM-Infinite. SE refers to Self-Extend. LLR refers to LongLora. AvgLen refers to average length of the datasets. Train Len refers to the longest length examples seen at training or finetuning. Eval Len refers to the maximum length of the input prompt. ✓refers to whether the method is exact attention.**
| Exact | AvgLen | Frozen | Finetuned |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Base | Inf | N-F | SE | PI | N-32 | YaRN | CLEX | LLR | Land | N-64 |
|  |  | ✓ |  | ✓ |  | ✓ | ✓ | ✓ |  |  | ✓ |  |
| Train Len | 4k | 4k | 4k | 4K | 32k | 32k | 32k | 32k | 32k | 32k | 64k |  |
| Eval Len | 4k | 32k | 32k | 32K | 32k | 32k | 32k | 32k | 32k | 32k | 32k |  |
| NQA | 18,409 | 21.09 | 10.39 | 3.88 | 23.49 | 23.02 | 23.73 | 19.82 | 24.19 | 12.07 | 12.47 | 24.31 |
| QAPR | 3,619 | 26.94 | 22.58 | 26.79 | 28.75 | 25.85 | 27.50 | 26.98 | 23.36 | 20.15 | 19.06 | 24.97 |
| MFQA | 4,559 | 32.42 | 26.19 | 29.82 | 32.66 | 35.10 | 38.22 | 37.11 | 40.83 | 24.50 | 21.86 | 40.60 |
| HPQA | 9,151 | 31.23 | 16.13 | 32.10 | 37.63 | 36.98 | 41.56 | 38.60 | 35.59 | 27.41 | 33.66 | 41.47 |
| WMQA | 4,887 | 25.75 | 20.64 | 22.34 | 30.70 | 29.38 | 31.58 | 30.63 | 28.24 | 21.46 | 24.94 | 28.62 |
| MSQ | 11,214 | 10.55 | 5.26 | 8.84 | 15.73 | 16.80 | 17.41 | 22.08 | 17.12 | 11.46 | 11.41 | 18.24 |
| GR | 8,734 | 17.32 | 13.43 | 17.87 | 13.15 | 25.61 | 28.27 | 20.98 | 24.68 | 24.05 | 17.20 | 24.37 |
| QMSM | 10,614 | 21.28 | 6.10 | 15.35 | 20.20 | 21.19 | 21.52 | 20.66 | 21.55 | 17.66 | 18.83 | 21.65 |
| MNWS | 2,113 | 3.44 | 3.63 | 9.30 | 1.50 | 10.55 | 22.13 | 8.91 | 16.96 | 21.19 | 19.43 | 25.02 |
| TREC | 5,177 | 66.00 | 61.00 | 67.50 | 69.00 | 71.00 | 69.00 | 69.00 | 67.50 | 50.00 | 49.00 | 69.00 |
| TRVQA | 8,209 | 87.89 | 81.40 | 18.69 | 88.44 | 88.55 | 88.86 | 89.63 | 89.36 | 12.28 | 74.75 | 88.65 |
| SMSM | 6,258 | 41.70 | 15.07 | 32.46 | 43.76 | 43.35 | 42.21 | 44.25 | 43.02 | 13.45 | 40.38 | 41.59 |
| PSC | 11,141 | 2.10 | 1.62 | 2.67 | 0.00 | 1.50 | 2.68 | 1.05 | 2.50 | 4.57 | 0.64 | 2.09 |
| PSR | 9,289 | 9.00 | 4.00 | 3.77 | 4.50 | 4.50 | 4.62 | 3.79 | 8.50 | 3.50 | 2.50 | 6.50 |
| LCC | 1,235 | 68.22 | 67.68 | 63.64 | 68.47 | 55.05 | 56.78 | 54.06 | 49.45 | 57.12 | 56.70 | 52.04 |
| REPO | 4,206 | 61.73 | 58.27 | 53.69 | 59.99 | 47.26 | 49.09 | 47.60 | 42.84 | 51.92 | 48.23 | 39.68 |
| Average | 7,425 | 32.92 | 25.84 | 25.54 | 33.62 | 33.48 | 35.32 | 33.45 | 33.48 | 23.30 | 28.19 | 34.30 |

### 9.8 RULER Subtasks Result

The performance of models on different lengths and breakdowns by 13 subtasks are reported in Table 12(RULER on 4k), Table 13(RULER on 8k), Table 14(RULER on 16k), Table 15(RULER on 32k) and Table 16(RULER on 64k).

**Table 12: Ruler results on 4k context length. N-32 and N-64 refer to NTK finetuned on 32K and 64K context lengths respectively. Inf refers to LM-Infinite. SE refers to Self-Extend. LLR refers to LongLora. Train Len refers to the longest length examples seen at training or finetuning. Eval Len refers to the maximum length of the input prompt. ✓refers to whether the method is exact attention.**
| Exact | Frozen | Finetuned |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Base | Inf | N-F | SE | PI | N-32 | YaRN | CLEX | LLR | Land | N-64 |
|  | ✓ |  | ✓ |  | ✓ | ✓ | ✓ | ✓ |  |  | ✓ |
| Train Len | 4k | 4k | 4k | 4k | 32k | 32k | 32k | 32k | 32k | 32k | 64k |
| Eval Len | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k |
| NIAH_S1 | 100.00 | 100.00 | 100.00 | 100.00 | 98.00 | 100.00 | 99.60 | 100.00 | 0.00 | 49.00 | 100.00 |
| NIAH_S2 | 100.00 | 100.00 | 100.00 | 100.00 | 99.80 | 100.00 | 88.60 | 100.00 | 0.00 | 20.60 | 100.00 |
| NIAH_S3 | 99.20 | 95.80 | 98.80 | 89.80 | 99.80 | 94.20 | 53.00 | 89.60 | 0.00 | 10.00 | 97.20 |
| NIAH_M1 | 99.20 | 98.80 | 99.20 | 79.00 | 99.20 | 99.20 | 62.60 | 95.80 | 0.00 | 10.60 | 98.00 |
| NIAH_M2 | 88.00 | 88.00 | 88.20 | 26.00 | 95.40 | 97.40 | 14.00 | 83.20 | 0.00 | 6.80 | 97.00 |
| NIAH_M3 | 61.40 | 62.00 | 61.60 | 14.40 | 78.00 | 68.20 | 8.20 | 53.80 | 0.00 | 1.20 | 84.80 |
| NIAH_MV | 83.55 | 90.45 | 86.60 | 82.10 | 95.45 | 96.40 | 50.25 | 95.10 | 0.05 | 10.80 | 96.15 |
| NIAH_MQ | 95.45 | 96.15 | 96.00 | 90.70 | 96.95 | 97.00 | 62.00 | 96.20 | 0.00 | 5.35 | 98.25 |
| VT | 57.72 | 58.56 | 56.48 | 8.92 | 96.64 | 98.16 | 25.68 | 85.72 | 0.00 | 2.92 | 97.00 |
| CWE | 78.20 | 75.90 | 78.20 | 73.56 | 81.38 | 80.86 | 58.78 | 82.60 | 64.70 | 23.16 | 74.26 |
| FWE | 84.33 | 84.20 | 84.93 | 80.07 | 58.40 | 85.53 | 26.20 | 52.60 | 18.53 | 84.93 | 81.40 |
| QA_1 | 62.20 | 60.40 | 62.40 | 60.60 | 57.80 | 62.40 | 60.20 | 55.80 | 26.20 | 37.20 | 55.80 |
| QA_2 | 43.00 | 43.40 | 42.40 | 40.20 | 42.40 | 46.20 | 43.20 | 38.20 | 28.00 | 28.20 | 46.00 |
| Avg. | 80.94 | 81.05 | 81.14 | 65.03 | 84.56 | 86.58 | 50.18 | 79.12 | 10.58 | 22.37 | 86.60 |

**Table 13: Ruler results on 8k context length.**
| Exact | Frozen | Finetuned |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Base | Inf | N-F | SE | PI | N-32 | YaRN | CLEX | LLR | Land | N-64 |
|  | ✓ |  | ✓ |  | ✓ | ✓ | ✓ | ✓ |  |  | ✓ |
| Train Len | 4k | 4k | 4k | 4k | 32k | 32k | 32k | 32k | 32k | 32k | 64k |
| Eval Len | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k |
| NIAH_S1 | - | 46.00 | 61.60 | 100.00 | 99.00 | 99.80 | 100.00 | 100.00 | 0.00 | 46.00 | 100.00 |
| NIAH_S2 | - | 36.60 | 59.40 | 98.80 | 100.00 | 100.00 | 99.40 | 100.00 | 0.00 | 7.20 | 100.00 |
| NIAH_S3 | - | 20.80 | 51.00 | 88.60 | 99.20 | 94.20 | 96.00 | 97.80 | 0.00 | 3.80 | 99.20 |
| NIAH_M1 | - | 27.80 | 46.00 | 69.40 | 98.00 | 94.20 | 86.60 | 90.20 | 0.00 | 7.60 | 95.20 |
| NIAH_M2 | - | 4.40 | 11.00 | 8.20 | 91.60 | 86.20 | 60.60 | 66.00 | 0.00 | 1.60 | 86.60 |
| NIAH_M3 | - | 2.60 | 4.00 | 3.20 | 48.40 | 52.20 | 34.60 | 11.80 | 0.00 | 0.00 | 47.40 |
| NIAH_MV | - | 30.35 | 41.35 | 52.95 | 65.50 | 85.95 | 70.40 | 61.25 | 0.00 | 6.25 | 84.75 |
| NIAH_MQ | - | 30.15 | 50.40 | 78.70 | 93.25 | 95.20 | 92.45 | 86.95 | 0.00 | 3.35 | 94.95 |
| VT | - | 4.88 | 69.88 | 1.48 | 91.20 | 96.16 | 77.52 | 48.16 | 0.00 | 3.08 | 94.36 |
| CWE | - | 65.08 | 40.30 | 30.82 | 45.66 | 45.76 | 44.72 | 32.72 | 18.92 | 22.08 | 40.80 |
| FWE | - | 56.73 | 64.87 | 59.00 | 65.07 | 70.13 | 10.53 | 45.40 | 16.73 | 76.60 | 54.13 |
| QA_1 | - | 35.80 | 44.40 | 31.00 | 50.80 | 49.20 | 43.20 | 48.20 | 22.80 | 25.00 | 50.20 |
| QA_2 | - | 29.00 | 33.60 | 37.40 | 40.80 | 41.80 | 36.80 | 42.60 | 24.40 | 25.20 | 44.80 |
| Avg. | - | 30.01 | 44.45 | 50.73 | 76.04 | 77.75 | 65.60 | 63.93 | 6.37 | 17.52 | 76.34 |

**Table 14: Ruler results on 16k context length.**
| Exact | Frozen | Finetuned |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Base | Inf | N-F | SE | PI | N-32 | YaRN | CLEX | LLR | Land | N-64 |
|  | ✓ |  | ✓ |  | ✓ | ✓ | ✓ | ✓ |  |  | ✓ |
| Train Len | 4k | 4k | 4k | 4k | 32k | 32k | 32k | 32k | 32k | 32k | 64k |
| Eval Len | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k |
| NIAH_S1 | - | 21.00 | 14.20 | 99.80 | 97.20 | 99.40 | 100.00 | 99.80 | 0.00 | 42.40 | 99.80 |
| NIAH_S2 | - | 17.00 | 17.40 | 93.40 | 100.00 | 100.00 | 99.20 | 100.00 | 0.20 | 6.80 | 100.00 |
| NIAH_S3 | - | 11.60 | 8.20 | 77.00 | 99.60 | 98.60 | 89.60 | 99.60 | 0.00 | 3.60 | 100.00 |
| NIAH_M1 | - | 15.80 | 9.20 | 60.00 | 97.80 | 93.20 | 83.40 | 89.40 | 0.00 | 5.60 | 90.80 |
| NIAH_M2 | - | 0.00 | 0.60 | 3.80 | 82.80 | 79.80 | 19.60 | 72.00 | 0.00 | 0.80 | 67.60 |
| NIAH_M3 | - | 1.00 | 0.00 | 1.80 | 34.20 | 18.20 | 7.40 | 15.00 | 0.00 | 0.00 | 29.60 |
| NIAH_MV | - | 8.40 | 6.90 | 38.85 | 77.55 | 81.95 | 58.75 | 62.40 | 0.00 | 4.80 | 83.50 |
| NIAH_MQ | - | 8.85 | 7.95 | 59.30 | 90.95 | 86.20 | 85.15 | 81.60 | 0.00 | 2.75 | 90.35 |
| VT | - | 6.56 | 11.28 | 1.16 | 68.84 | 83.56 | 47.12 | 48.16 | 0.00 | 2.52 | 88.68 |
| CWE | - | 19.94 | 28.36 | 17.80 | 27.26 | 26.32 | 23.72 | 28.60 | 0.62 | 11.90 | 21.20 |
| FWE | - | 77.13 | 25.80 | 59.80 | 47.93 | 61.73 | 10.13 | 57.33 | 12.93 | 81.60 | 51.73 |
| QA_1 | - | 22.80 | 36.40 | 28.00 | 46.00 | 45.20 | 43.20 | 49.20 | 13.20 | 23.00 | 45.00 |
| QA_2 | - | 24.20 | 26.00 | 31.60 | 35.20 | 36.00 | 37.40 | 33.40 | 20.80 | 26.20 | 36.00 |
| Avg. | - | 18.02 | 14.79 | 44.02 | 69.64 | 70.01 | 54.21 | 64.35 | 3.67 | 16.31 | 69.56 |

**Table 15: Ruler results on 32k context length.**
| Exact | Frozen | Finetuned |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Base | Inf | N-F | SE | PI | N-32 | YaRN | CLEX | LLR | Land | N-64 |
|  | ✓ |  | ✓ |  | ✓ | ✓ | ✓ | ✓ |  |  | ✓ |
| Train Len | 4k | 4k | 4k | 4k | 32k | 32k | 32k | 32k | 32k | 32k | 64k |
| Eval Len | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k |
| NIAH_S1 | - | 7.80 | 0.00 | 83.00 | 97.20 | 99.00 | 85.80 | 85.60 | 0.00 | 33.80 | 100.00 |
| NIAH_S2 | - | 7.00 | 0.00 | 68.40 | 99.00 | 100.00 | 81.00 | 94.00 | 0.00 | 3.20 | 99.20 |
| NIAH_S3 | - | 6.40 | 0.00 | 42.80 | 97.00 | 99.40 | 62.40 | 97.20 | 0.00 | 2.60 | 96.40 |
| NIAH_M1 | - | 8.80 | 0.00 | 29.40 | 93.40 | 90.80 | 63.20 | 78.40 | 0.00 | 5.40 | 82.60 |
| NIAH_M2 | - | 0.00 | 0.00 | 2.40 | 48.80 | 39.40 | 6.40 | 40.40 | 0.00 | 0.20 | 36.60 |
| NIAH_M3 | - | 0.00 | 0.00 | 1.40 | 5.80 | 8.60 | 1.20 | 8.00 | 0.00 | 0.00 | 7.20 |
| NIAH_MV | - | 3.75 | 0.00 | 24.65 | 61.30 | 68.20 | 37.95 | 60.75 | 0.00 | 2.80 | 82.20 |
| NIAH_MQ | - | 2.05 | 0.00 | 20.35 | 68.65 | 78.25 | 46.95 | 67.80 | 0.05 | 2.35 | 85.80 |
| VT | - | 2.08 | 0.00 | 2.32 | 56.68 | 43.28 | 22.00 | 30.08 | 0.00 | 2.52 | 71.28 |
| CWE | - | 4.48 | 0.02 | 17.46 | 26.72 | 11.78 | 11.38 | 22.70 | 13.26 | 3.68 | 7.34 |
| FWE | - | 72.67 | 2.93 | 46.73 | 31.00 | 64.53 | 15.13 | 34.67 | 13.93 | 72.47 | 51.53 |
| QA_1 | - | 20.20 | 5.20 | 20.60 | 33.40 | 34.40 | 23.00 | 28.00 | 6.00 | 22.60 | 27.00 |
| QA_2 | - | 25.20 | 1.20 | 24.00 | 30.60 | 34.80 | 24.00 | 30.60 | 12.60 | 24.60 | 33.20 |
| Avg. | - | 12.34 | 0.72 | 29.50 | 57.66 | 59.42 | 36.95 | 52.17 | 3.53 | 13.56 | 60.03 |

**Table 16: Ruler results on 64k context length.**
| Exact | Frozen | Finetuned |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Base | Inf | N-F | SE | PI | N-32 | YaRN | CLEX | LLR | Land | N-64 |
|  | ✓ |  | ✓ |  | ✓ | ✓ | ✓ | ✓ |  |  | ✓ |
| Train Len | 4k | 4k | 4k | 4k | 32k | 32k | 32k | 32k | 32k | 32k | 64k |
| Eval Len | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k | 4k |
| NIAH_S1 | - | 3.20 | 0.00 | 71.80 | 0.00 | 83.60 | 0.00 | 40.60 | 0.00 | 40.00 | 98.00 |
| NIAH_S2 | - | 3.80 | 0.00 | 0.20 | 0.00 | 95.60 | 0.00 | 68.80 | 0.00 | 3.00 | 98.00 |
| NIAH_S3 | - | 5.40 | 0.00 | 0.00 | 0.00 | 95.40 | 0.00 | 70.40 | 0.00 | 3.00 | 95.80 |
| NIAH_M1 | - | 5.40 | 0.00 | 0.00 | 0.00 | 76.80 | 0.00 | 55.40 | 0.00 | 5.20 | 67.20 |
| NIAH_M2 | - | 0.00 | 0.00 | 2.60 | 0.00 | 15.20 | 0.00 | 15.80 | 0.00 | 0.00 | 25.80 |
| NIAH_M3 | - | 0.00 | 0.00 | 0.20 | 0.00 | 1.20 | 0.00 | 1.00 | 0.00 | 0.00 | 4.00 |
| NIAH_MV | - | 4.45 | 0.00 | 0.20 | 0.00 | 51.70 | 0.00 | 36.40 | 0.00 | 3.70 | 51.20 |
| NIAH_MQ | - | 4.45 | 0.00 | 0.05 | 0.00 | 56.60 | 0.00 | 43.50 | 0.00 | 2.45 | 65.40 |
| VT | - | 1.28 | 0.00 | 12.20 | 0.00 | 34.28 | 0.00 | 0.00 | 0.00 | 2.40 | 41.48 |
| CWE | - | 0.76 | 0.00 | 6.85 | 0.00 | 6.58 | 0.00 | 9.72 | 0.00 | 1.70 | 7.88 |
| FWE | - | 72.20 | 11.47 | 26.47 | 0.00 | 25.27 | 0.00 | 11.73 | 0.00 | 82.67 | 27.73 |
| QA_1 | - | 16.20 | 0.20 | 0.80 | 0.00 | 30.80 | 0.00 | 25.60 | 0.00 | 19.60 | 29.20 |
| QA_2 | - | 20.20 | 0.20 | 0.00 | 0.00 | 28.40 | 0.00 | 19.00 | 0.00 | 20.20 | 29.40 |
| Avg. | - | 10.56 | 0.91 | 9.34 | 0.00 | 46.26 | 0.00 | 30.61 | 0.00 | 14.15 | 49.31 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: please see weaknesses
soundness: 3
strengths: 1. This work is the first to conduct a fair and comprehensive comparison of different long context extension methods, resulting in several useful conclusions.
summary: Using consistent base models and extension data, the study yielded several insights into long-context behavior. First, it reaffirmed the critical role of perplexity as a general-purpose performance indicator. Second, current approximate attention methods systematically underperform in long-context tasks. Finally, it confirmed that exact fine-tuning based methods are generally effective within their extension range, whereas extrapolation remains challenging.
weaknesses: 1. Based on my experience, increasing the model size in long-context downstream tasks such as LongBench yields some interesting conclusions that are inconsistent with the 7B model. I hope you can add some simple experimental results from the 13B model to discuss this point.

2. While enhancing the model's long-context capabilities, we also care about the impact of different methods on short-text performance and the extent of knowledge forgetting. I hope you can add some results from the common Open LLM Leaderboard to discuss this point.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: As the (1)-(3) in weakness.
soundness: 1
strengths: 1. The paper evaluates many methods under a controlled setting.
2. The paper proposes several might-be-useful conclusions based on their experiments.
summary: The paper evaluates several freeze/fine-tuning long-context methods on several public LLMs. Based on its experiments, it argues that (1) PPL is still an important metric in long-context scenarios, (2) "approximate attention" methods show poor performance (3) "exact fine-tuning based methods" are effective within trained range but cannot further process longer texts well.
weaknesses: 1. More evaluation of short texts is required to check whether these methods perform well in keeping the original performance. (for example, NTK by parts may behave better on this than NTK-rope.)
2. The so-called "approximate attention" method includes many quite different methods and can be misleading. It isn't very clear to include the streaming-LLM methods / longlora / landmark into one class. And MinInference-style work shows promising performance now, the conclusion that 'approximate attention is poor' might be misleading.
3. self-extend seems to be more like NTK-F than 'approximate attention'.
4. The contribution of the work might be limited if the arguments were not sufficiently robust.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: - What is the performance of new models, such as Qwen2.5, in this experiment?
- What are your thoughts on the generalization behavior of these methods for contexts longer than 32k? Are there any preliminary findings or conjectures regarding this?
soundness: 3
strengths: - The paper introduces a novel controlled protocol for evaluating long-context extension methods, addressing a significant gap in the literature regarding the comparison of such techniques.
- The study is comprehensive, utilizing a variety of metrics and tasks to assess model performance. The use of standardized base models and extension data enhances the quality of the comparative analysis.
summary: This paper systematically evaluates various methods for extending the context length of LLMs, aiming to provide insights into the behavior of long-context models and establish a standardized evaluation framework. It designs a controlled protocol for comparing context extension methods using consistent base models and extension data. The study includes an examination of the performance of different attention mechanisms in long-context tasks, confirms that perplexity remains a relevant performance indicator in longer-context scenarios, and presents findings indicating that exact fine-tuning methods are effective within their extension range, while approximation methods tend to underperform. Additionally, the paper emphasizes the open-sourcing of codebases, models, and checkpoints to promote transparency and facilitate further research.
weaknesses: - The study is limited to three base models, which may not accurately represent the performance of other, potentially larger models. Expanding the analysis to include a more diverse set of base models could strengthen the conclusions.
- While the paper acknowledges limitations due to fixed hyperparameters, a more in-depth exploration of how different hyperparameter settings might affect the results would be beneficial.
- The generalization of findings to longer contexts beyond 32k (e.g., 128k and 1m) is not addressed, which is a significant limitation given the focus on long-context models.
- The insights provided in this paper have been discussed in previous studies, and I did not gain any new takeaways from it.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: * Figure 1. I am surprised by the LM-Infinite result as the authors of the LM-Infinite reported an pass rate of about 80\% on the task of Passkey Retrieval, which is very similar to the needle-in-the-haystack evaluation conducted in Figure 1. However, LM-Infinite fails most cases on NIAH as reported in Figure 1. Could you help explain what are potential aspects that lead to this gap?
* Line 484 "Perplexity and downstream tasks". I'm not fully convinced by this argument and Figure 4. It seems that the linear trend in Figure 4 is highly dependent on LongLora and Landmark, in a sense that the linear trend is likely to disappear without these two compared methods. 
  * It seems that when perplexity is below certain level, e.g., below 6, the perplexity differences between models are small but the downstream performance differences are large. Thus I'm concerned with the claim that perplexity is a "general-purpose performance indicator" as suggested in the abstract, it can only indicate well within a certain region from my understanding.
  * Could you please consider adding metrics such as rank correlation to further strengthen the claim?
* Line 505 "Context extension hurts in the short term and gains in the long term". I'm not quite sure what this title means here. What are "short term" and "long term" referring to here?
* Is there any inference speed trade-off between these compared methods? e.g., Are attention approximation methods faster? By how much? For some applications the inference speed may be critical. Providing such information will help users make informed decisions.

Others:
* Line 194: What is the CLEX method here? Currently there is little introduction of it.
* Line 221: "key matrix", are you referring to "key and query matrices"?
soundness: 3
strengths: 1. The topic of this paper is timely and important. While there is a lot of recent work on long-context extension, the experiments were conducted with different base models and different training data, hence there is a lack of fair comparison. This work aims to answer this important question.
2. The background section (section 3) is mostly well-written and provides a unified view of these various context extension methods.
3. The authors conducted extensive experiments and ensured the comparison is conducted in a fair manner.
summary: This paper presents a comprehensive empirical study on long-context extension methods for language models, including 4 methods that adapt the RoPE positional embedding, and 4 methods that approximates the attention operations. The paper ensures a fair comparison of these methods and conclude with three key takeaways: (1) contrary to claims in previous works, perplexity and downstream task performance has high correlations; (2) RoPE-adaptation methods in general work better than attention approximation methods; (3) Dynamic NTK works best among compared methods.
weaknesses: 1. I'm not fully convinced by some of the experiment results and takeaways. See questions below.
2. The paper mentions lack of "quantitative rankings of different methodologies" as a motivation of this work. While this paper finds NTK-Dynamic to work best in general, this paper does not provide a full ranking.
3. The presentation and organization of the paper can be improved in various aspects. e.g., using more visualizations instead of tables to summarize the results and highlight main findings; having a table to summary the key characteristics of the 8 compared methods.

Meta-review: The paper presents a controlled study evaluating various methods for extending the context length that Large Language Models (LLMs) can handle during inference. The authors argue that existing evaluations are often inconsistent due to variations in base models, datasets, and methodologies. The paper's primary strength is its attempt to conduct a controlled comparison of different long-context extension methods. While the paper addresses generalization to some extent, its exploration of very long contexts (e.g., 128k tokens and beyond) is limited. This is a significant aspect of long-context modeling. As Reviewer 3BXk notes, some of the findings have been touched upon in previous work. While the controlled study adds value, the paper could have done a better job of highlighting truly novel insights. Reviewer j6WF points out that some conclusions, particularly regarding perplexity, might be overstated or need more nuanced presentation. I also listed some issues raised by the reviewers, which were discussed during the rebuttal phase. While the paper has merits for controlled experiments, the reviewers raised the concerns regarding the novelty of the findings, the generalizability of the conclusions, and certain aspects of the presentation and analysis. The authors are encourged to revise a stronger version by considering the comments of the reviewers.

Decision: Reject