BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?
Authors: name: Guoxin Chen, name: Fanzhe Meng, name: Jiale Zhao, name: Minghao Li, name: Daixuan Cheng, name: Huatong Song, name: Jie Chen, name: Yuzhi Lin, name: Hui Chen, name: Xin Zhao, name: Ruihua Song, name: Chang Liu, name: Cheng Chen, name: Kai Jia, name: Ji-Rong Wen
Organization: AweAI Team

Current benchmarks for code agents primarily assess narrow, repository-specific fixes, overlooking critical real-world challenges such as cross-repository reasoning, domain-specialized problem solving, dependency-driven migration, and full-repository generation. To address this gap, we introduce BeyondSWE, a comprehensive benchmark that broadens existing evaluations along two axes - resolution scope and knowledge scope - using 500 real-world instances across four distinct settings. Experimental results reveal a significant capability gap: even frontier models plateau below 45% success, and no single model performs consistently across task types. To systematically investigate the role of external knowledge, we develop SearchSWE, a framework that integrates deep search with coding abilities. Our experiments show that search augmentation yields inconsistent gains and can in some cases degrade performance, highlighting the difficulty of emulating developer-like workflows that interleave search and reasoning during coding tasks. This work offers both a realistic, challenging evaluation benchmark and a flexible framework to advance research toward more capable code agents.