NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents
Authors: name: Jingzhe Ding; name: Shengda Long; name: Changxin Pu; name: Huan Zhou; name: Hongwan Gao; name: Xiang Gao; name: Chao He; name: Yue Hou; name: Fei Hu; name: Zhaojian Li; name: Weiran Shi; name: Zaiyuan Wang; name: Daoguang Zan; name: Chenchen Zhang; name: Xiaoxu Zhang; name: Qizhi Chen; name: Xianfu Cheng; name: Bo Deng; name: Qingshui Gu; name: Kai Hua; name: Juntao Lin; name: Pai Liu; name: Mingchen Li; name: Xuanguang Pan; name: Zifan Peng; name: Yujia Qin; name: Yong Shan; name: Zhewen Tan; name: Weihao Xie; name: Zihan Wang; name: Yishuo Yuan; name: Jiayu Zhang; name: Enduo Zhao; name: Yunfei Zhao; name: He Zhu; name: Chenyang Zou; name: Ming Ding; name: Jianpeng Jiao; name: Jiaheng Liu; name: Minghao Liu; name: Qian Liu; name: Chongyao Tao; name: Jian Yang; name: Tong Yang; name: Zhaoxiang Zhang; name: Xinjie Chen; name: Wenhao Huang; name: Ge Zhang
Published: 2025-12-14 15:12:13

Abstract: Recent advances in coding agents suggest rapid progress toward autonomous software development, yet existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems. Most prior evaluations focus on localized code generation, scaffolded completion, or short-term repair tasks, leaving open the question of whether agents can sustain coherent reasoning, planning, and execution over the extended horizons demanded by real-world repository construction. To address this gap, we present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents. Given only a single natural-language requirements document and an empty workspace, agents must autonomously design the architecture, manage dependencies, implement multi-module logic, and produce a fully installable Python library. Our experiments across state-of-the-art open- and closed-source models reveal that long-horizon repository generation remains largely unsolved: even the strongest agents achieve below 40% average test pass rates and rarely complete an entire repository correctly. Detailed analysis uncovers fundamental long-horizon failure modes, including premature termination, loss of global coherence, fragile cross-file dependencies, and inadequate planning over hundreds of interaction steps. NL2Repo Bench establishes a rigorous, verifiable testbed for measuring sustained agentic competence and highlights long-horizon reasoning as a central bottleneck for the next generation of autonomous coding agents.