Title: On the Measurement and Efficient Mitigation of Length Generalization Gaps in Large Language Models
Primary area: interpretability and explainable AI
Keywords: large language models, length generalization

Abstract: Large Language Models (LLMs) typically train on short text due to the quadratic complexity of their self-attention architectures. 
As a result, their performance suffers drastically on inputs longer than those encountered during training, substantially limiting their applications in real-world tasks involving long contexts.
In this paper, we rigorously establish an upper bound on length generalization in the measurement space and identify two length-related factors that limit performance. 
Our theory explains two recent observations: **_(i)_** out-of-distribution positions in longer contexts reduce length generalization, and **_(ii)_** fine-tuning on entire sequences is not necessary. 
Motivated by these insights, we propose _Virtual-context Learning_ (_VCL_), a flexible method that requires minimal modifications to most fine-tuning approaches.
Experiments on various tasks show that _VCL_ allows LLMs to generalize to 4 $\times$ context windows while retaining perplexity and improving performance on downstream tasks such as Passkey Retrieval and LongBench. 
_VCL_ brings substantial efficiency improvements, reducing decoding time and memory usage by up to 50\% compared with fine-tuning baselines.

TL;DR: Theoretical analysis, implications and inspired solution to length generalization