A Controlled Study on Long Context  Extension and Generalization in LLMs

Authors: Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, Alexander M Rush
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Controlled Study, Long Context, Extension, Benchmark, Analysis

Abstract:
Broad textual understanding and in-context learning require language models that utilize full document contexts. Due to the implementation challenges associated with directly training long-context models, many methods have been proposed for extending models to handle long contexts. However, owing to differences in data and model classes, it has been challenging to compare these approaches, leading to uncertainty as to how to evaluate long-context performance and whether it differs from standard evaluation. We implement a controlled protocol for extension methods with a standardized evaluation, utilizing consistent base models and extension data. Our study yields several insights into long-context behavior. First, we reaffirm the critical role of perplexity as a general-purpose performance indicator even in longer-context tasks. Second, we find that current approximate attention methods systematically underperform across long-context tasks. Finally, we confirm that exact fine-tuning based methods are generally effective within their extension range, whereas extrapolation remains challenging. All codebases, models, and checkpoints will be made available open-source, promoting transparency and facilitating further research in this critical area of AI development.

Decision: reject
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730506510248

thread_type:
official_review

<thread_object>
creator: reviewer_j6wf

<content>
summary:
This paper presents a comprehensive empirical study on long-context extension methods for language models, including 4 methods that adapt the RoPE positional embedding, and 4 methods that approximates the attention operations. The paper ensures a fair comparison of these methods and conclude with three key takeaways: (1) contrary to claims in previous works, perplexity and downstream task performance has high correlations; (2) RoPE-adaptation methods in general work better than attention approximation methods; (3) Dynamic NTK works best among compared methods.

strengths:
1. The topic of this paper is timely and important. While there is a lot of recent work on long-context extension, the experiments were conducted with different base models and different training data, hence there is a lack of fair comparison. This work aims to answer this important question.
2. The background section (section 3) is mostly well-written and provides a unified view of these various context extension methods.
3. The authors conducted extensive experiments and ensured the comparison is conducted in a fair manner.

weaknesses:
1. I'm not fully convinced by some of the experiment results and takeaways. See questions below.
2. The paper mentions lack of "quantitative rankings of different methodologies" as a motivation of this work. While this paper finds NTK-Dynamic to work best in general, this paper does not provide a full ranking.
3. The presentation and organization of the paper can be improved in various aspects. e.g., using more visualizations instead of tables to summarize the results and highlight main findings; having a table to summary the key characteristics of the 8 compared methods.

questions:
* Figure 1. I am surprised by the LM-Infinite result as the authors of the LM-Infinite reported an pass rate of about 80\% on the task of Passkey Retrieval, which is very similar to the needle-in-the-haystack evaluation conducted in Figure 1. However, LM-Infinite fails most cases on NIAH as reported in Figure 1. Could you help explain what are potential aspects that lead to this gap?
* Line 484 "Perplexity and downstream tasks". I'm not fully convinced by this argument and Figure 4. It seems that the linear trend in Figure 4 is highly dependent on LongLora and Landmark, in a sense that the linear trend is likely to disappear without these two compared methods. 
  * It seems that when perplexity is below certain level, e.g., below 6, the perplexity differences between models are small but the downstream performance differences are large. Thus I'm concerned with the claim that perplexity is a "general-purpose performance indicator" as suggested in the abstract, it can only indicate well within a certain region from my understanding.
  * Could you please consider adding metrics such as rank correlation to further strengthen the claim?
* Line 505 "Context extension hurts in the short term and gains in the long term". I'm not quite sure what this title means here. What are "short term" and "long term" referring to here?
* Is there any inference speed trade-off between these compared methods? e.g., Are attention approximation methods faster? By how much? For some applications the inference speed may be critical. Providing such information will help users make informed decisions.

Others:
* Line 194: What is the CLEX method here? Currently there is little introduction of it.
* Line 221: "key matrix", are you referring to "key and query matrices"?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730629280318

thread_type:
official_review

<thread_object>
creator: reviewer_3bxk

<content>
summary:
This paper systematically evaluates various methods for extending the context length of LLMs, aiming to provide insights into the behavior of long-context models and establish a standardized evaluation framework. It designs a controlled protocol for comparing context extension methods using consistent base models and extension data. The study includes an examination of the performance of different attention mechanisms in long-context tasks, confirms that perplexity remains a relevant performance indicator in longer-context scenarios, and presents findings indicating that exact fine-tuning methods are effective within their extension range, while approximation methods tend to underperform. Additionally, the paper emphasizes the open-sourcing of codebases, models, and checkpoints to promote transparency and facilitate further research.

strengths:
- The paper introduces a novel controlled protocol for evaluating long-context extension methods, addressing a significant gap in the literature regarding the comparison of such techniques.
- The study is comprehensive, utilizing a variety of metrics and tasks to assess model performance. The use of standardized base models and extension data enhances the quality of the comparative analysis.

weaknesses:
- The study is limited to three base models, which may not accurately represent the performance of other, potentially larger models. Expanding the analysis to include a more diverse set of base models could strengthen the conclusions.
- While the paper acknowledges limitations due to fixed hyperparameters, a more in-depth exploration of how different hyperparameter settings might affect the results would be beneficial.
- The generalization of findings to longer contexts beyond 32k (e.g., 128k and 1m) is not addressed, which is a significant limitation given the focus on long-context models.
- The insights provided in this paper have been discussed in previous studies, and I did not gain any new takeaways from it.

questions:
- What is the performance of new models, such as Qwen2.5, in this experiment?
- What are your thoughts on the generalization behavior of these methods for contexts longer than 32k? Are there any preliminary findings or conjectures regarding this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711673193

thread_type:
official_review

<thread_object>
creator: reviewer_7saq

<content>
summary:
The paper evaluates several freeze/fine-tuning long-context methods on several public LLMs. Based on its experiments, it argues that (1) PPL is still an important metric in long-context scenarios, (2) "approximate attention" methods show poor performance (3) "exact fine-tuning based methods" are effective within trained range but cannot further process longer texts well.

strengths:
1. The paper evaluates many methods under a controlled setting.
2. The paper proposes several might-be-useful conclusions based on their experiments.

weaknesses:
1. More evaluation of short texts is required to check whether these methods perform well in keeping the original performance. (for example, NTK by parts may behave better on this than NTK-rope.)
2. The so-called "approximate attention" method includes many quite different methods and can be misleading. It isn't very clear to include the streaming-LLM methods / longlora / landmark into one class. And MinInference-style work shows promising performance now, the conclusion that 'approximate attention is poor' might be misleading.
3. self-extend seems to be more like NTK-F than 'approximate attention'.
4. The contribution of the work might be limited if the arguments were not sufficiently robust.

questions:
As the (1)-(3) in weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716514405

thread_type:
official_review

<thread_object>
creator: reviewer_zktb

<content>
summary:
Using consistent base models and extension data, the study yielded several insights into long-context behavior. First, it reaffirmed the critical role of perplexity as a general-purpose performance indicator. Second, current approximate attention methods systematically underperform in long-context tasks. Finally, it confirmed that exact fine-tuning based methods are generally effective within their extension range, whereas extrapolation remains challenging.

strengths:
1. This work is the first to conduct a fair and comprehensive comparison of different long context extension methods, resulting in several useful conclusions.

weaknesses:
1. Based on my experience, increasing the model size in long-context downstream tasks such as LongBench yields some interesting conclusions that are inconsistent with the 7B model. I hope you can add some simple experimental results from the 13B model to discuss this point.

2. While enhancing the model's long-context capabilities, we also care about the impact of different methods on short-text performance and the extent of knowledge forgetting. I hope you can add some results from the common Open LLM Leaderboard to discuss this point.

questions:
please see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732174225619

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear Reviewers,

We want to express our sincere gratitude for the constructive feedback provided by the reviewers for Submission8416. We have carefully addressed each of your concerns to enhance the quality and robustness of our paper. Details can be found in our response to your reviews. 

Below is a summary of the key actions we have undertaken in response to the reviewers' comments:

1. **Expansion to Larger and More Diverse Models:** (Reviewer ZKTb, Reviewer 3BXk)

   - In response to concerns about the limited number of base models, we have conducted experiments using the LLaMA-70B, LLaMA-13B model as base models and Qwen-2.5-7B model. These additional evaluations confirm that our findings are consistent across larger and more diverse LLMs, thereby strengthening the generalizability of our conclusions.

2. **Evaluation of Short-Text Performance:**  (Reviewer ZKTb, Reviewer 7saq)

   - To address the impact of long-context extension methods on short-text performance, we evaluated our models on several benchmarks from the [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard). The results indicate that while most long-context methods exhibit slight performance degradation on short-text tasks, whereas approximate attention methods are more robust in shorter tasks. 


3. **Inference Speed Trade-Off:** (Reviewer j6WF)

   - We addressed this concern about inference speed trade-offs by conducting inference speed comparisons under controlled conditions using the same hardware setup. Our experiments show that approximate attention methods achieve a speedup of approximately 1.5x to 2x compared to LLaMA when the context length is short. However, we didn't see a significant margin when the context length was longer. 


4. **Hyperparameter Sensitivity Analysis:** (Reviewer 3BXk)

   - We conducted a comprehensive hyperparameter sweep to analyze the effects on both training efficiency and inference performance. Our findings reveal that approximate attention methods, such as LongLoRA, are highly sensitive to hyperparameter changes, requiring meticulous tuning. In contrast, continual fine-tuning methods like NTK and YaRN demonstrated robustness across various hyperparameter configurations, resulting in more predictable training times and consistent inference performance.


5. **Generalization to Longer Contexts Beyond 32k:** (Reviewer 3BXk)

   - Although our initial study focused on context lengths up to 32k token in most cases, and 64k for two particular tasks, we have extended our evaluation to include 64k and 128k tokens. The results show that exact fine-tuning methods perform well within their trained context range but encounter challenges as context length increases beyond 32k tokens. Approximate attention methods show potential for handling longer contexts but often at the expense of reduced accuracy.


6. **Clarification and Reclassification of Methods:** (Reviewer 7saq, Reviewer j6WF)

   - Method Categorization: We will refine the categorization of context extension methods to avoid ambiguity. Specifically, we will reclassify methods like Self-Extend under RoPE-based methods instead of grouping them with approximate attention methods. Additionally, we will incorporate discussions on recent developments in MinInference-style methods, acknowledging their promising performance and clarifying their orthogonal relationship to our study.

   - PPL Correlation: We will update the manuscript to include a ranking-based correlation measure between perplexity and downstream task performance, addressing the reviewer's concerns and strengthening our claims.

7. **LM-Infinite Results:** (Reviewer j6WF)

   - We evaluated LM-Infinite on the Passkey Retrieval task. We observed that while it performs well at shorter context lengths, its performance significantly degrades as context length increases beyond its trained length. This aligns with findings from concurrent studies (e.g., InfLLM), suggesting that LM-Infinite effectively attends to closer tokens but struggles with longer contexts.


By implementing these revisions, we believe our submission offers a more thorough and balanced analysis of long-context extension methods in a controlled setting. The enhancements address the initial concerns raised by the reviewers and contribute valuable insights to the field.

Thank you for taking the time to read our rebuttal. We are confident that the improvements made have significantly strengthened the quality and impact of our work.

Best regards,

Authors of submission 8416
</content>
</thread_object>

</thread>
<thread>
creation date:
1732781920605

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
revised submission

comment:
Dear Reviewers and Area Chair,

Thank you for your thoughtful feedback on our manuscript. We have uploaded a revised version that incorporates your valuable suggestions. The revision includes expanded analysis with additional base model results,  short-context task results, a detailed discussion of generalization capabilities, revised analysis of perplexity and downstream tasks, hyperparameter search, and enhanced limitations. For your convenience, all modifications are highlighted in green within the manuscript.

We welcome your further comments and suggestions to strengthen our work. The manuscript remains open for additional improvements based on your expertise and insights.

Thank you for your continued guidance in improving this research manuscript.

Best regards,

Authors of Submission 8416
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a controlled study evaluating various methods for extending the context length that Large Language Models (LLMs) can handle during inference. The authors argue that existing evaluations are often inconsistent due to variations in base models, datasets, and methodologies. The paper's primary strength is its attempt to conduct a controlled comparison of different long-context extension methods. While the paper addresses generalization to some extent, its exploration of very long contexts (e.g., 128k tokens and beyond) is limited. This is a significant aspect of long-context modeling. As Reviewer 3BXk notes, some of the findings have been touched upon in previous work. While the controlled study adds value, the paper could have done a better job of highlighting truly novel insights. Reviewer j6WF points out that some conclusions, particularly regarding perplexity, might be overstated or need more nuanced presentation. I also listed some issues raised by the reviewers, which were discussed during the rebuttal phase. While the paper has merits for controlled experiments, the reviewers raised the concerns regarding the novelty of the findings, the generalizability of the conclusions, and certain aspects of the presentation and analysis. The authors are encourged to revise a stronger version by considering the comments of the reviewers.