LONGCODEU: Benchmarking Long-Context Language Models on Long Code
  Understanding
Authors: name: Jia Li, name: Xuyuan Guo, name: Lei Li, name: Kechi Zhang, name: Ge Li, name: Jia Li, name: Zhengwei Tao, name: Fang Liu, name: Chongyang Tao, name: Yuqi Zhu, name: Zhi Jin

Current advanced long-context language models offer great potential for
real-world software engineering applications. However, progress in this
critical domain remains hampered by a fundamental limitation: the absence of a
rigorous evaluation framework for long code understanding. To gap this
obstacle, we propose a long code understanding benchmark LONGCODEU from four
aspects (8 tasks) to evaluate LCLMs' long code understanding ability required
for practical applications, including code unit perception, intra-code unit
understanding, inter-code unit relation understanding, and long code
documentation understanding. We evaluate 9 popular LCLMs on LONGCODEU (i.e., 6
general models and 3 code models). Our experimental results reveal key
limitations in current LCLMs' capabilities for long code understanding.
Particularly, the performance of LCLMs drops dramatically when the long code
length is greater than 32K, falling far short of their claimed 128K-1M context
windows. In the four aspects, inter-code unit relation understanding is the
most challenging for LCLMs. Our study provides valuable insights for optimizing
LCLMs and driving advancements in software engineering.