Why Does the Effective Context Length of LLMs Fall Short?
Authors: name: Chenxin An; name: Jun Zhang; name: Ming Zhong; name: Lei Li; name: Shansan Gong; name: Yao Luo; name: Jingjing Xu; name: Lingpeng Kong
Published: 2024-10-24 13:51:50

Abstract: Advancements in distributed training and efficient attention mechanisms have
significantly expanded the context window sizes of large language models
(LLMs). However, recent work reveals that the effective context lengths of
open-source LLMs often fall short, typically not exceeding half of their
training lengths. In this work, we attribute this limitation to the left-skewed
frequency distribution of relative positions formed in LLMs pretraining and
post-training stages, which impedes their ability to effectively gather distant
information. To address this challenge, we introduce ShifTed Rotray position
embeddING (STRING). STRING shifts well-trained positions to overwrite the
original ineffective positions during inference, enhancing performance within
their existing training lengths. Experimental results show that without
additional training, STRING dramatically improves the performance of the latest
large-scale models, such as Llama3.1 70B and Qwen2 72B, by over 10 points on
popular long-context benchmarks RULER and InfiniteBench, establishing new
state-of-the-art results for open-source LLMs. Compared to commercial models,
Llama 3.1 70B with \method even achieves better performance than GPT-4-128K and
clearly surpasses Claude 2 and Kimi-chat.