NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents
Authors: name: Jingzhe Ding, name: Shengda Long, name: Changxin Pu, name: Huan Zhou, name: Hongwan Gao, name: Xiang Gao, name: Chao He, name: Yue Hou, name: Fei Hu, name: Zhaojian Li, name: Weiran Shi, name: Zaiyuan Wang, name: Daoguang Zan, name: Chenchen Zhang, name: Xiaoxu Zhang, name: Qizhi Chen, name: Xianfu Cheng, name: Bo Deng, name: Qingshui Gu, name: Kai Hua, name: Juntao Lin, name: Pai Liu, name: Mingchen Li, name: Xuanguang Pan, name: Zifan Peng, name: Yujia Qin, name: Yong Shan, name: Zhewen Tan, name: Weihao Xie, name: Zihan Wang, name: Yishuo Yuan, name: Jiayu Zhang, name: Enduo Zhao, name: Yunfei Zhao, name: He Zhu, name: Chenyang Zou, name: Ming Ding, name: Jianpeng Jiao, name: Jiaheng Liu, name: Minghao Liu, name: Qian Liu, name: Chongyao Tao, name: Jian Yang, name: Tong Yang, name: Zhaoxiang Zhang, name: Xinjie Chen, name: Wenhao Huang, name: Ge Zhang

Recent advances in coding agents suggest rapid progress toward autonomous software development, yet existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems. Most prior evaluations focus on localized code generation, scaffolded completion, or short-term repair tasks, leaving open the question of whether agents can sustain coherent reasoning, planning, and execution over the extended horizons demanded by real-world repository construction. To address this gap, we present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents. Given only a single natural-language requirements document and an empty workspace, agents must autonomously design the architecture, manage dependencies, implement multi-module logic, and produce a fully installable Python library. Our experiments across state-of-the-art open- and closed-source models reveal that long-horizon repository generation remains largely unsolved: even the strongest agents achieve below 40% average test pass rates and rarely complete an entire repository correctly. Detailed analysis uncovers fundamental long-horizon failure modes, including premature termination, loss of global coherence, fragile cross-file dependencies, and inadequate planning over hundreds of interaction steps. NL2Repo Bench establishes a rigorous, verifiable testbed for measuring sustained agentic competence and highlights long-horizon reasoning as a central bottleneck for the next generation of autonomous coding agents.