LoCoBench: A Benchmark for Long-Context Large Language Models in Complex
  Software Engineering
Authors: name: Jielin Qiu; name: Zuxin Liu; name: Zhiwei Liu; name: Rithesh Murthy; name: Jianguo Zhang; name: Haolin Chen; name: Shiyu Wang; name: Ming Zhu; name: Liangwei Yang; name: Juntao Tan; name: Zhepeng Cen; name: Cheng Qian; name: Shelby Heinecke; name: Weiran Yao; name: Silvio Savarese; name: Caiming Xiong; name: Huan Wang
Published: 2025-09-11 16:55:04

Abstract: The emergence of long-context language models with context windows extending
to millions of tokens has created new opportunities for sophisticated code
understanding and software development evaluation. We propose LoCoBench, a
comprehensive benchmark specifically designed to evaluate long-context LLMs in
realistic, complex software development scenarios. Unlike existing code
evaluation benchmarks that focus on single-function completion or short-context
tasks, LoCoBench addresses the critical evaluation gap for long-context
capabilities that require understanding entire codebases, reasoning across
multiple files, and maintaining architectural consistency across large-scale
software systems. Our benchmark provides 8,000 evaluation scenarios
systematically generated across 10 programming languages, with context lengths
spanning 10K to 1M tokens, a 100x variation that enables precise assessment of
long-context performance degradation in realistic software development
settings. LoCoBench introduces 8 task categories that capture essential
long-context capabilities: architectural understanding, cross-file refactoring,
multi-session development, bug investigation, feature implementation, code
comprehension, integration testing, and security analysis. Through a 5-phase
pipeline, we create diverse, high-quality scenarios that challenge LLMs to
reason about complex codebases at unprecedented scale. We introduce a
comprehensive evaluation framework with 17 metrics across 4 dimensions,
including 8 new evaluation metrics, combined in a LoCoBench Score (LCBS). Our
evaluation of state-of-the-art long-context models reveals substantial
performance gaps, demonstrating that long-context understanding in complex
software development represents a significant unsolved challenge that demands
more attention. LoCoBench is released at:
https://github.com/SalesforceAIResearch/LoCoBench.