Training Software Engineering Agents and Verifiers with SWE-Gym
Authors: name: Jiayi Pan; name: Xingyao Wang; name: Graham Neubig; name: Navdeep Jaitly; name: Heng Ji; name: Alane Suhr; name: Yizhe Zhang
Published: 2024-12-30 18:15:39

Abstract: We present SWE-Gym, the first environment for training real-world software
engineering (SWE) agents. SWE-Gym contains 2,438 real-world Python task
instances, each comprising a codebase with an executable runtime environment,
unit tests, and a task specified in natural language. We use SWE-Gym to train
language model based SWE agents , achieving up to 19% absolute gains in resolve
rate on the popular SWE-Bench Verified and Lite test sets. We also experiment
with inference-time scaling through verifiers trained on agent trajectories
sampled from SWE-Gym. When combined with our fine-tuned SWE agents, we achieve
32.0% and 26.0% on SWE-Bench Verified and Lite, respectively, reflecting a new
state-of-the-art for open-weight SWE agents. To facilitate further research, we
publicly release SWE-Gym, models, and agent trajectories.