SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering
  Tasks?
Authors: name: Xiang Deng; name: Jeff Da; name: Edwin Pan; name: Yannis Yiming He; name: Charles Ide; name: Kanak Garg; name: Niklas Lauffer; name: Andrew Park; name: Nitin Pasari; name: Chetan Rane; name: Karmini Sampath; name: Maya Krishnan; name: Srivatsa Kundurthy; name: Sean Hendryx; name: Zifan Wang; name: Chen Bo Calvin Zhang; name: Noah Jacobson; name: Bing Liu; name: Brad Kenstler
Published: 2025-09-21 06:28:17

Abstract: We introduce SWE-Bench Pro, a substantially more challenging benchmark that
builds upon the best practices of SWE-BENCH [25], but is explicitly designed to
capture realistic, complex, enterprise-level problems beyond the scope of
SWE-BENCH. SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of
41 actively maintained repositories spanning business applications, B2B
services, and developer tools. The benchmark is partitioned into a public set
with open access to problems sourced from 11 repositories, a held-out set of 12
repositories and a commercial set of 18 proprietary repositories where we have
formal partnership agreements with early-stage startups. Problems in the
held-out and the commercial set are not publicly accessible, but we release
results on the commercial set. Our benchmark features long-horizon tasks that
may require hours to days for a professional software engineer to complete,
often involving patches across multiple files and substantial code
modifications. All tasks are human-verified and augmented with sufficient
context to ensure resolvability. In our evaluation of widely used coding
models, under a unified scaffold, we observe that their performance on
SWE-Bench PRO remains below 25% (Pass@1), with GPT-5 achieving the highest
score to date at 23.3%. To better understand these limitations, we cluster the
failure modes observed in the collected agent trajectories for a clearer
characterization of the error patterns exhibited by current models. Overall,
SWE-BENCH PRO provides a contamination-resistant testbed that more faithfully
captures the complexity and diversity of real-world software development,
advancing the pursuit of truly autonomous software engineering agents at a
professional level.