A Comprehensive Survey on Long Context Language Modeling
Authors: name: Jiaheng Liu, name: Dawei Zhu, name: Zhiqi Bai, name: Yancheng He, name: Huanxuan Liao, name: Haoran Que, name: Zekun Wang, name: Chenchen Zhang, name: Ge Zhang, name: Jiebin Zhang, name: Yuanxing Zhang, name: Zhuo Chen, name: Hangyu Guo, name: Shilong Li, name: Ziqiang Liu, name: Yong Shan, name: Yifan Song, name: Jiayi Tian, name: Wenhao Wu, name: Zhejian Zhou, name: Ruijie Zhu, name: Junlan Feng, name: Yang Gao, name: Shizhu He, name: Zhoujun Li, name: Tianyu Liu, name: Fanyu Meng, name: Wenbo Su, name: Yingshui Tan, name: Zili Wang, name: Jian Yang, name: Wei Ye, name: Bo Zheng, name: Wangchunshu Zhou, name: Wenhao Huang, name: Sujian Li, name: Zhaoxiang Zhang

Efficient processing of long contexts has been a persistent pursuit in
Natural Language Processing. With the growing number of long documents,
dialogues, and other textual data, it is important to develop Long Context
Language Models (LCLMs) that can process and analyze extensive inputs in an
effective and efficient way. In this paper, we present a comprehensive survey
on recent advances in long-context modeling for large language models. Our
survey is structured around three key aspects: how to obtain effective and
efficient LCLMs, how to train and deploy LCLMs efficiently, and how to evaluate
and analyze LCLMs comprehensively. For the first aspect, we discuss data
strategies, architectural designs, and workflow approaches oriented with long
context processing. For the second aspect, we provide a detailed examination of
the infrastructure required for LCLM training and inference. For the third
aspect, we present evaluation paradigms for long-context comprehension and
long-form generation, as well as behavioral analysis and mechanism
interpretability of LCLMs. Beyond these three key aspects, we thoroughly
explore the diverse application scenarios where existing LCLMs have been
deployed and outline promising future development directions. This survey
provides an up-to-date review of the literature on long-context LLMs, which we
wish to serve as a valuable resource for both researchers and engineers. An
associated GitHub repository collecting the latest papers and repos is
available at:
https://github.com/LCLM-Horizon/A-Comprehensive-Survey-For-Long-Context-Language-Modeling{\color[RGB]{175,36,67}{LCLM-Horizon}}.