A Controlled Study on Long Context Extension and Generalization in LLMs
Authors: name: Yi Lu; name: Jing Nathan Yan; name: Songlin Yang; name: Justin T. Chiu; name: Siyu Ren; name: Fei Yuan; name: Wenting Zhao; name: Zhiyong Wu; name: Alexander M. Rush
Published: 2024-09-18 17:53:17

Abstract: Broad textual understanding and in-context learning require language models
that utilize full document contexts. Due to the implementation challenges
associated with directly training long-context models, many methods have been
proposed for extending models to handle long contexts. However, owing to
differences in data and model classes, it has been challenging to compare these
approaches, leading to uncertainty as to how to evaluate long-context
performance and whether it differs from standard evaluation. We implement a
controlled protocol for extension methods with a standardized evaluation,
utilizing consistent base models and extension data. Our study yields several
insights into long-context behavior. First, we reaffirm the critical role of
perplexity as a general-purpose performance indicator even in longer-context
tasks. Second, we find that current approximate attention methods
systematically underperform across long-context tasks. Finally, we confirm that
exact fine-tuning based methods are generally effective within the range of
their extension, whereas extrapolation remains challenging. All codebases,
models, and checkpoints will be made available open-source, promoting
transparency and facilitating further research in this critical area of AI
development.