Title: Auto-SWE-Bench: A Framework for the Scalable Generation of Software Engineering Benchmark from Open-Source Repositories
Primary area: datasets and benchmarks
Keywords: Benchmark, SWE Bench, Evaluation of Large Language Model, Coding Benchmarks, Coding Agent

Abstract: Benchmarks like SWE-bench have shaped the evaluation of Large Language Models (LLMs) on complex software engineering tasks, although such efforts remain limited due to manual curation, static datasets, and narrow coverage of issue types. We introduce SWE-Bench Atlas, a fully automated framework for generating high-fidelity, repository-level coding tasks from open-source GitHub projects. Unlike synthetic benchmarks such as SWE-Smith, SWE-Bench Atlas draws directly from SOTA-model-breaking pull requests and encompasses diverse issue types, capturing a broader spectrum of real-world software engineering demands. The framework is structured as a pipeline of six automated components: (1) a sourcing module that broadly identifies pull requests with high test coverage and complexity; (2) an agentic Dockerization system that guarantees reproducibility of historically accurate software environment for every task; (3) a “state-based” test classification with automatic log parsing for more granular test feedback; (4) a curation module to repair or augment tasks with low test alignment; (5) an annotation module to guarantee issue clarity and test-to-issue alignment; and (6) a trajectory curation system leveraging human-in-the-loop refinement to resolve SOTA-model-breaking issues. Our initial private  benchmark consists of 5,909 instances from 3,154 repositories across 7 languages. On a subset of 488 instances of this benchmark, today’s strongest models perform as follows: gpt-5-2025-08-07 (24.34% pass@1), claude-opus-4-20250514 (18.46%), gemini/gemini-2.5-pro (9.53%), and gpt-4o (3.65%). Supplementing our findings, a public release of the 488-task subset is soon to follow. We also demonstrate proof-of-value of our data by showing fine-tuning based improvements on the SWE-bench multilingual benchmark. By producing dynamic, diverse challenging tasks across programming languages and repositories, SWE-Bench Atlas enables scalable evaluation and advancement of AI coding and reasoning abilities of next-generation AI systems.