Why Does the Effective Context Length of LLMs Fall Short?

Authors: Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, Lingpeng Kong
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Model, Long-Context Modeling, Position Embedding

Abstract:
Advancements in distributed training and efficient attention mechanisms have significantly expanded the context window sizes of large language models (LLMs). However, recent work reveals that the effective context lengths of open-source LLMs often fall short, typically not exceeding half of their training lengths. In this work, we attribute this limitation to the left-skewed frequency distribution of relative positions formed in LLMs pretraining and post-training stages, which impedes their ability to effectively gather distant information.
To address this challenge, we introduce Shifted Rotray Position Embedding (STRING).  STRING  shifts well-trained positions to overwrite the original ineffective positions during inference, enhancing performance within their existing training lengths. 
Experimental results show that without additional training, STRING dramatically improves the performance of the latest large-scale models, such as Llama3.1 70B and Qwen2 72B, by over 10 points on popular long-context benchmarks RULER and InfiniteBench, establishing new state-of-the-art results for open-source LLMs. Compared to commercial models, Llama 3.1 70B with STRING  even achieves better performance than GPT-4-128K and clearly surpasses Claude 2 and Kimi-chat.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730516623159

thread_type:
official_review

<thread_object>
creator: reviewer_qjga

<content>
summary:
This paper discusses how current LLMs use only a portion of their maximum context length due to issues in handling distant information. The authors identify the root cause as a left-skewed distribution in the model's relative positions during pretraining and post-training. 

To address this, they introduce a new method called Shifted Rotary Position Embedding (STRING), which adjusts well-trained positions during inference to improve performance without extra training. 
Experimental results show that STRING enhances the context length usage of open-source models like Llama 3.1 70B and Qwen2 72B by over 10 points on benchmarks like RULER and InfiniteBench. STRING's effectiveness even surpasses commercial models like GPT-4-128K, Claude 2, and Kimi-chat.

strengths:
The paper is well-structured, with clear organization, understandable tables, and figures, effectively addressing an important topic in LLM development.

The empirical analysis of position frequency on the training data is thorough (However, I recommend providing additional explanation regarding the meaning of the plot in Figure 1 in the Appendix for further clarity) .

The method has been evaluated on/against quite a few LLMs.

weaknesses:
1. The code is missing; the provided GitHub link leads to a 404 error.

2. The analysis in Section 3 is intriguing, but the results are limited to small-sized LLMs. It remains uncertain whether the same findings apply to medium- and large-sized LLMs.

questions:
1. Where can we find your implementation of STRING?

2. Are error bars available for the tables in the experiments? If space is limited, could they be included in the Appendix to give readers a better understanding of the variability in the results?

3. Regarding the improved results shown in the experiments, could you provide more insights into why and how STRING performs effectively? Additionally, are there any scenarios where STRING might not perform as well?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730517903590

thread_type:
official_review

<thread_object>
creator: reviewer_rthl

<content>
summary:
This paper suggests that large language models (LLMs) struggle to handle dependencies over long distances due to the skewed occurrence of relative distance towards shorter ranges while training. The authors propose STRING, a simple method that re-uses more frequently occuring positions for distant tokens, avoiding the use of under-utilized long-distance positions. Through extensive evaluations on long-context benchmarks, they demonstrate that STRING can improve the performance of SOTA models within their pre-trained context windows.

strengths:
1. The paper is clearly written and easy to follow.
2. The proposed method is simple and effective. In particular, it improves the performance of existing LLMs within their pre-trained context window.
3. The proposed method can be easily deployed. STRING does not require any additional training and can therefore be easily applied to existing LLMs. The authors also provide a way to make the method compatible with FlashAttention, enabling fast and efficient inference.

weaknesses:
1. Observation (1) in Section 3 (line 216) is overstated, as training with longer context involves larger computation demands due to the quadratic nature of self-attention. For fair comparison, training efficiency should be compared with the same amount of computation, instead of the number of consumed tokens.
2. STRING looks more like an empirical workaround to avoid using less-frequent positions, rather than a fundamental solution for the problem.

questions:
1. Can this method also be useful for context extrapolation? I understand that the primary focus of the paper is to improve performance within the pre-trained context window, but I believe this is also a large understated benefit of your method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730626296354

thread_type:
official_review

<thread_object>
creator: reviewer_v29a

<content>
summary:
This paper reveals the possible reasons why current LLM can't maintain good performance with long-context task even the training has involved such corpus. According to the analysis on the left-skewed frequency-distribution, the paper proposes a training-free method (STRING) to mitigate the position bias. By replacing infrequent long-range position index to frequently trained positions during inference, their method enhances the long-context capability without requiring additional training. The experiment results on multiple long-context benchmarks demonstrate the effectiveness of STRING.

strengths:
1. The exploratory analysis about the left-skewed frequency-distribution of position index is insightful, and can help the community understand the potential reasons why the long-context problem exists in current methods.

2. The proposed method is intuitive and simple to implement, which seems effective simultaneously.

3. The presentation of the paper is very clear,  and the analysis together with the method description are easy to follow,

weaknesses:
1. The parameter S and W needs to be set manually, which maybe improved by an automatic algorithm to obtain a better performance.

questions:
It is shown that the most distant position S indexes can be ignored during inference to obtain a better performance, have you conducted investigation whether it is the same case for training? Can decreasing the longest context window in training directly yields a better performance in inference?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731234973302

thread_type:
official_review

<thread_object>
creator: reviewer_swnh

<content>
summary:
The authors present STRING (Shifted Rotary position embedding) which proposes to shift the position relevance of tokens during inference. With STRING, the authors are able to demonstrate improved performance of 10+ points on RULER, and IB for LLAMA 70B. The authors further train a tiny-llama 1.3B model to demonstrate the effectiveness and compare STRING with SOTA models. The authors also discuss the implementation of STRING with modification to flash attention.

strengths:
Thank you for submitting your paper to ICLR. I thoroughly enjoyed reading your paper, especially it is very well written. 
1. The paper observes "left-skewed position frequency distribution" - a bias in the position indices required to answer a relevant question being towards the shorter end. For example, on SlimPajama, the authors observe that the frequency of position indices used to model relationships between distant tokens (distances
≥1024) is less than 20%, and for even longer distances (≥1536), it drops below 5%.
2. The authors also demonstrate interesting findings, such as "larger training context window consumes fewer tokens to achieve the same context length". Although it's only been tested on tinyllama (1.3B) so weather scaling laws hold is an open problem.

weaknesses:
1. The authors acknowledge in part (3) of 4.1 that "Applying Eq. 3 disrupts the model’s ability to capture
local relationships because it alters the relative positions between neighboring tokens" but I'm not fully convinced if the heuristic of "W" is sufficient to allevate this concern. While emperically the experiments suggest so, I'd have liked to see a slightly more detailed study on the impact of altering the relative positions between neighboring tokens.

questions:
The results in the paper for the 1.3B Tiny-llama and the commercial models are impressive on the long-context benchmarks. However, assuming the "system" to mean the LLM and STRING combination will be used together, I'd also like to see benchmarks for short-context lengths - especially around coding (MBPP, Humaneval), tool-use (BFCL), and Reasoning (GSM-8K, MATH). If not for all, atleast for 1/2 models to reason if there are any trade-offs. I'm especially curious for coding and tooling benchmarks, since they need to adhere to syntax, which might be compromised by the re-positioning. 

Since you have implemented STRING, what is the latency overheads (and impact on throughput) vis-a-vis vanilla RoPE?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732299884571

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear reviewers,

We are deeply grateful for your insightful feedback and valuable suggestions. Your comprehensive reviews have provided us with essential guidance to enhance our work

We also wish to express our appreciation for your recognition of the strengths of our work, including:

* Insightful analysis on the left-skewed frequency distribution of position indices which potentially hinders the long-context ability of current LLMs (`SwNh`, `v29A`, `rtHL`, `QJGa`)
* The proposed positional encoding STRING is effective with extensive  evaluation (`v29A`, `rtHL`, `QJGa`)
* The paper has a clear organization. The analysis together with the method STRING are easy to follow (`v29A`,` rtHL`, `QJGa`)

Below is a summary of the key updates made to the manuscript:

- Enhanced explanations (Appendix A.1):
    - How we utilize $W$ to preserve the relationships between neighboring tokens.
    - Why STRING produces results consistent with RoPE in short-context settings.
- Additional results: 
    - Results of STRING in a training setting (Table 2).
    - Results of STRING in a context extrapolation setting (Appendix Table 4,5).
    - Error bars to the tables in our Experiments section (Appendix Table 6)
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a simple and effective method that re-uses more frequently occuring positions for distant tokens via analyzing skewed distribution of LMs.

All reviewers agree the contributions of this paper such as interesting findings, in-depth analsysi, effective method, and promising results. Despite some minor concerns such as lack of experiments at scale, this paper can contribute to LLM community in terms of context length and position embedding. 

So, AC recommends accepting this paper.