LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering
Authors: name: Jielin Qiu; name: Zuxin Liu; name: Zhiwei Liu; name: Rithesh Murthy; name: Jianguo Zhang; name: Haolin Chen; name: Shiyu Wang; name: Ming Zhu; name: Liangwei Yang; name: Juntao Tan; name: Roshan Ram; name: Akshara Prabhakar; name: Tulika Awalgaonkar; name: Zixiang Chen; name: Zhepeng Cen; name: Cheng Qian; name: Shelby Heinecke; name: Weiran Yao; name: Silvio Savarese; name: Caiming Xiong; name: Huan Wang
Published: 2025-11-17 23:57:24

Abstract: As large language models (LLMs) evolve into sophisticated autonomous agents capable of complex software development tasks, evaluating their real-world capabilities becomes critical. While existing benchmarks like LoCoBench~qiu2025locobench assess long-context code understanding, they focus on single-turn evaluation and cannot capture the multi-turn interactive nature, tool usage patterns, and adaptive reasoning required by real-world coding agents. We introduce LoCoBench-Agent, a comprehensive evaluation framework specifically designed to assess LLM agents in realistic, long-context software engineering workflows. Our framework extends LoCoBench's 8,000 scenarios into interactive agent environments, enabling systematic evaluation of multi-turn conversations, tool usage efficiency, error recovery, and architectural consistency across extended development sessions. We also introduce an evaluation methodology with 9 metrics across comprehension and efficiency dimensions. Our framework provides agents with 8 specialized tools (file operations, search, code analysis) and evaluates them across context lengths ranging from 10K to 1M tokens, enabling precise assessment of long-context performance. Through systematic evaluation of state-of-the-art models, we reveal several key findings: (1) agents exhibit remarkable long-context robustness; (2) comprehension-efficiency trade-off exists with negative correlation, where thorough exploration increases comprehension but reduces efficiency; and (3) conversation efficiency varies dramatically across models, with strategic tool usage patterns differentiating high-performing agents. As the first long-context LLM agent benchmark for software engineering, LoCoBench-Agent establishes a rigorous foundation for measuring agent capabilities, identifying performance gaps, and advancing autonomous software development at scale.