Training Software Engineering Agents and Verifiers with SWE-Gym
Authors: Jiayi Pan
Xingyao Wang
Graham Neubig
Navdeep Jaitly
Heng Ji
Alane Suhr
Yizhe Zhang
Venue: ICML 2025 poster (2025)
Keywords: Agents
Software Engineering Agents
Post-training
Primary area: deep_learning->large_language_models

Abstract: We present SWE-Gym, the first environment for training real-world software engineering (SWE) agents. SWE-Gym contains 2,438 real-world Python task instances, each comprising a codebase with an executable runtime environment, unit tests, and a task specified in natural language. We use SWE-Gym to train language model based SWE agents, achieving up to 19% absolute gains in resolve rate on the popular SWE-Bench Verified and Lite test sets. We also experiment with inference-time scaling through verifiers trained on agent trajectories sampled from SWE-Gym. When combined with our fine-tuned SWE agents, we achieve 32.0% and 26.0% on SWE-Bench Verified and Lite, respectively, reflecting a new state-of-the-art for open-weight SWE agents. To facilitate further research, we publicly release SWE-Gym, models, and agent trajectories.

Training Software Engineering Agents and Verifiers with SWE-Gym
Jiayi Pan 11
Xingyao Wang*?
Graham Neubig
Navdeep Jaitly
Heng Ji
Alane Suhr
Yizhe Zhang
Abstract
AWE-Bench Verifie Resolved (%)
Training Time Scaling
We present SWE-Gym, the first environment for training software engineering (SWE) agents. SWE-Gym contains 2,438 real-world task in- stances, each comprising a Python codebase with executable runtime environment, unit tests, and task specified in natural language. We use SWE- Gym to train language model based SWE agents, and achieve up to 19% absolute gains in resolu- tion rate on the popular SWE-Bench Verified and Lite test sets. We also experiment with inference- time scaling through verifiers trained on agent trajectories sampled from SWF-Gyym. When com- bined with our fine-tuned SWE agents, achieve 32.0% and 26.0% on SWE- WWE-Bench Verified and Lite, respectively, reflecting new state-of-the-art for open-weight SWE agents. To facilitate further research, we publicly release SWE-Gym, models,
20
18
14
12
100
200
300
400
500
Number Training Trajectories
SWE-Bench Verified Resolved (9)
Inference Time Scaling
t-0
30
28
26
24
22
20
and agent trajectories.
Number of Agent Rollouts
Figure 1: SWE-Gym enables scalable improvements for software engineering agents. Top: Scaling the amount of training data shows consistent performance improvements as we obtain more training trajectories, with no signs of saturation at 491 trajectories. We use temperature = - for evaluation. Bottom: For inference time scaling, we generate a number of candidate trajectories per task and select the best using verifier trained on SWE-Gym. This approach demonstrates roughly log-linear gains with the number of sampled solutions. t 0 (excluded from regression) is used as the first hypothesis to be consistent with the top figure; later rollouts use / = 0.5.
1. Introduction
Language models (LMs) have remarkable promise in tomating software engineering (SWE) tasks, most clearly measured by recent progress on benchmarks like SWE- Bench (Jimenez et al., 2024) and CommitO (Zhao et al., 2024). While L.M-based SWE agents have shown significant performance gains through improving agent-compute in- terfaces (Yang et al., 2024) and prompting strategies (Wang al., 2024c), advances in SWE agents have been limited by reliance proprietary models, with limited research to improve the underlying LM itself.
ing (Shao et al., 2024; Yuan et al., 2024), and web naviga- tion (Pan et al., 2024), software engineering currently lacks suitable training environments, and creating environments is uniquely challenging. Real-world software engineering requires interaction with an executable runtime that has been prepared with the appropriate software dependencies and reproducible test suites, among other requirements. These challenges are reflected in the existing resources (Tab. 1). For example, the -WE-Bench (Jimenez et al., 2024) training split contains only solutions (git patches that solve the task), missing the step-by-step actions taken by the developer to create each solution, and executable environments and re-
Unlike other domains where supervised fine-tuning and reinforcement learning have significantly improved LM ca- pabilitics, such as chat (Ouyang et al., 2022), math reason-
Equal contribution. 1 Equal supervision. 'UC Berkeley 'IUC CMU 'Apple. Correspondence to: Jiayi Pan <jjayi- pan @ ber.eeeyy.duu>. Xingyao Wang <xingyao6@illinn @illinois.edu Alane Suhr <suhr @ @beerklly.eddx, Yizhe Zhang <yiz- zhang@appler @arpgpaappp.........
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Training Software Engineering Agents and Verifiers with SWE-Gym
ward signals. R2E (Jain et al., 2024) uses synthetic tasks that very far from real-world problems, while datasets such as APPS (Hendrycks et al., 2021a) focus only on isolated tasks rather than realistic repository-level coding problems.
2. Related Work
Agents that solve GitHub issues. We focus on soft- ware engineering agents designed to automatically resolve GitHub issues within the SWE-Bench framework (Jimenez et al., 2024). These agents take GitHub issue and its associated code repository as input and generate a valid code modification (i.e., a git diff patch) to address the is- sue. The correctness of these modifications is verified us- ing a human-written test suite. Existing agent designs are categorized by the extent of human priors integrated into their workflows: Specialized workflows (Xia al., 2024; Örwall, 2024; Zhang et al., 2024b; Chen et al., 2024) in- volve human-defined stages (c.g., localization, code edit- ing, patch re-ranking), where a LM is iteratively prompted for each stage to produce the final result. This approach reduces the task horizon and minimizes the need for long- term planning. However, specialized workflows require significant human engineering. may not generalize to novel issue types, and can fail if intermediate steps encounter problems. In contrast, general-purpose prompting ((Yang et al., 2024; Wang et al., 2024c)) rely on LM's ability to plan over long horizons and generate actions based on a history of interactions without heavily pre-defined work- flows. While more flexible, general approaches demand higher capabilities from the underlying LM and can be com- putationally expensive due multiple interaction rounds. The most successful existing SWE agents are built on pro- prietary language models like GPT-4 or Claude and utilize specialized workflows to overcome these models' limita- tions. This contrasts with other sequential decision-making domains (Silver et al., 2017; Akkaya et al., 2019), where learning-baad based approaches, such as reinforcement learning, drive success by enabling systems to learn from interactions and rewards to develop task competence. A key barrier in the SWE agent domain is the lack of appropriate training environments. Our experiments show that SWE-Gym can be used to build strong learning-based agents, accelerating research in this area.
To bridge this gap, we present SWE-Gym, the first training environment combining real-world software engineering tasks from GitHub issues with pre-installed dependencies and executable test verification. SWE-Gym contains 2,438 Python tasks sourced from 11 popular open-source reposi- tories (Tab. 2), providing useful environments for training LMs as agents and verifiers.
SWE-Gym supports training state-of-the-art open- weight SWE agents. Based on the OpenHands (Wang et al., 2024c) agent scaffold for general-purposs software devel- opment ($2), we fine-tune a 32B Qwen-2.5 coder model (Hui et al., 2024b) using only 491 agent-environment inter- action trajectories sampled using SWE-Gym, and achieve substantial absolute improvements of + +12.3% (to 15.3%) and + +13.6% (to 20.6%) in resolution rate on SWE-Bench Lite and SWE-Bench Verified respectively ($4.2).
SWE-Gym is effective across agent scaffolds. In an- other agent scaffold based on a specialized workflow (Moat- lessTools; Örwall 2024; $2), we experiment with self- improvement, where the LM interacts with SWE-Gym, re- ceives reward from it, and learns to improve itself through re- jection sampling fine-tuning. This self-improvement boosts performance up to 19.7% on SWE-Bench Lite.
SWE-Gym supports training verifier models to enable inference-time scaling. We use test suites included in SWE- Gym to determine whether sampled agent trajectories are successful or not. Given these samples, we train a verifier model (i.e., an outcome-suppervised reward model; Cobbe et al., 2021) that estimates a trajectory's 's probability of suc- cess. This enables inference-time scaling, where we sample multiple agent trajectories and select the one with the high- est estimated reward according to the verifier. This further improves the resolution rate to 32.0% (+11.44 (.11.4% absolute improvement) on SWE-Bench Verified 1:; Fig. 1 bot- tom) and 26.0% on SWE-Bench Lite ($5. 1.2), establishing a new state-of-the-aa art among systems with publicly accessible weights (Tab. 9).
Environments for training software agents. There is no existing dataset suitable for training software engineering agents. SWE-Bench (Jimenez et al., 2024) is widely used for evaluating software engineering performance, but its train- ing split lacks executable environments and success signals present in the evaluation split, making it useful only for imi- tation learning approaches. HumanEval (Chen al., 2021) is designed for standalone code generation tasks, akin to coding competitions. Therefore, it falls short of addressing the complex challenges inherent in real-world, repository- level software engineering tasks, which involve thousands of files, millions of lines of code, and tasks such as bug fixing, feature development, and system optimization. Sim- ilarly, R2E (Jain et al., 2024) is small evaluation dataset
Our baseline training and inference-time scaling meth- ods on SWE-Gym yield continuously improved results with increasing compute (Fig. 1). In the training phase, performance scales with the number of sampled trajectories up to our current limit of 491 trajectories, suggesting that performance is currently limited by the compute budget for sampling rather than the number of tasks in SWE-Gym. Similarly, using the agent and verifier trained by SWE-Gym, the bottom panel shows that using more compute during inference time steadily improves the performance.
2

---

Training Software Engineering Agents and Verifiers with SWE-Gym
Table I: SWE-Gym is the first publicly available training environment combining real-world SWE tasks from GitHub issues with pre-installed dependencies and executable test verification. Repository-level, whether each task is situated in a sophisticated repository; Executable Environment: whether each task instance comes with an executable environment with all relevant dependencies pre-installed; Real task: whether task instruction is collected from human developers.
Datasel (split)
Repository-Leve
Executable Environment
Real task
# Instances (total)
# Instances (train)
CodeFoodback (Zheng et al., 2024b)
66,383
66,383
APPS (Hendrycks al., 2021a)
10,000
5,000
HumanEval (Chen et al,, 2021)
164
MBPP (Tao et al,, 2024)
X
/
974
374
R2E (Jain et al., 2024)
/
246
SWE-Bench (train) (Jimenez ot al., 2024)
x
19,008
19,008
SWE-Ciym Raw
X
64,689
64,689
SWE-Benee (test) (Jimenez et al., 2024)
/
2,294
SWF-Gym
2,438
2,438
with 246 instances and, due to its synthetic nature, lacks the realism and complexity in real-world software engineering scenario. Our proposed SWE- -ym instead uses real-wont world GitHub issues as task, and associated executable unit tests for evaluation. This results in realistic and complex task formulations, aligning closely with real-world challenges.
3. SWE-Gym Environment
SWE-Gym comprises 2,438 real-world software engineer- ing tasks sourced from pull requests in 11 popular Python repositories, with pre-configured executable environments and expert-validated test cases, constructed in close align- ment with SWE-Bench (Jimenez et al., 2024). These reposi- tories are separate from those used in SWE-Bench avoid contamination. These tasks require SWE agents develop test-passing solutions for real-world GitHub issues using provided codebases and executable environments. Such agents must map from natural language descriptions of the issue, as well as the initial state of the repository, to a pull request represented as a git patch.
Post-training: From chatbots and reasoners agents. Post-training, which fine-tunes pre-trained language models using supervised or reinforcement learning, significantly improves model performance across various domains. Tech- niques like RLHF (Ouyang et al., 2022) have become stan- dard for adapting language models into chatbots, improving both performance and alignment (Qwen Team, 2024). In math reasoning, datasets such as MATH (Hendrycks al., 2021b) and GSM-8K (Cobbe et al., 2021) facilitate the train- ing and evaluation of policy and verifier models (Cobbe et al., 2021; Wang et al., 2024a). Earlier works (Wang et al., 2024b; Chen et al., 2023; Zeng et al., 2023; Wu et al., 2024) demonstrate that distilling agent trajectories from stronger models improve weaker models. Recent studies (Xi al., 2024; Zhai al., 2024; Bai al., 2024) explore self- improving methods, showing that reinforcement learning or rejection sampling fine-tuning guided by reward enables LMs to enhance themselves without more capable teachers.
We also identify a subset of 230 tasks, SWE-Gym Lite, which contains generally casier and more self-contained tasks that are suitable for rapid prototyping, in alignment with SWE-Bench Lite (Jimenez et al., 2024). To support future research in SWE agent development and automatic dataset synthesis, we also release SWE-Gym Raw, large set of Python GitHub issues without executable environ- ments (64.689 instances spanning 358 Python repositories).
3.1. Dataset Construction
Identify Repositories. We first use SEART GitHub search ! to filter list of initial repositories. Unlike SWE-Bench, which focuses on the top 5k most downloaded PyPI libraries (Jimenez et al., 2024), we select Python repositories that were created before July 1, 2022 and have more than 500 stars, with at least 300 lines of code, more than 500 pull requests (PRs) and 100 contributors. This results in 358 repositories.
However, post-training typically depends on expert demon- stration data or training environments with reliable reward signals, which are largely absent in the software engineer- ing domain. This has led reliance on prompting-ben based methods with proprietary language models. Our work ad- dresses this gap with SWE-Gym, training environment based on real-world software engineering tasks that uses expert-writted tests as reward signals. Our experiments demonstrate that SWE-Gym can build strong SWE agents without prompt engineering.
Extracting Training Instances from Repositories. We use SWE-Bench's instance extraction script to convert these repositories into task instances, each corresponding to
hnttps:/sseart
di.usi... ch/
3

---

Training Software Engineering Agents and Verifiers with SWE-Gym
Category
Metric
SWE-Gym
SWE-Gym Lite
MONAL (374)
pariOk (731)
Size
# Instances
2,438 (2,294)
230 (300)
SWE-Gym (2438)
# Repos
11 (12)
11 (12)
moto (343)
Issue Text
Length by Words
239.8 (195.1)
186.2 (175.9)
pysteni
Non-test Files
971.2 (2944.2)
818.8 (2988.5)
mysy (25))
(225)
Sack (145)
moxiin (101)
Codebase
# Non-test Lines
340675.0 (363728.4)
340626.2 (377562.4)
mypy (40)
Lines edited
69.8 (32.8)
10.6 (10.1)
mot
Gold Patch
# Files edited
2.5 (1.7)
1.0 (1.0)
# Func. edited
4.1 (3.0)
1.4 (1.34)
(36)
SWE Gym Lite (230)
# Fail to Pass
10.0 (9.0)
2.04 (3,5) 99.9 (85.2)
Tests
# Total
760.8 (132.5)
MONAL 227)
Spdtiii (24)
dask (1-)
conan (12)
Table 2: Statistics comparing SWE-Gym with the SWE-Bench test split (in parenthesis). Except for size metrics, we report the average value across instances.
Figure 2: Repository distribu- tion of SWE-Gym instances.
GitHub issue including the natural language description of the issue, a snapshot of the repository in which the issue was created, and a set of unit tests. Over the 358 repositories, we extract 64,689 task instances. We refer to this dataset SWE--Gym Raw, which is over three times larger than the 19k instances gathered in previous work (Jimenez et al., 2024) and includes nearly ten times many repositories.
execution-based validation script to ensure that the gold patch (the human-submitted code diff) passes more unit tests than the original code. This process required approxi- mately 200 human annotation hours² and 10,000 CPU core hours. After validation and filtering out failed instances, we obtained 2,438 uni--tett-vlidaate instances from 11 reposi- tories. For full reproducibility, we publicly release pre-built Docker images for each instance, totaling 6 TB.
While SWE-Gym Raw instances contain code, issue de- scriptions, and the solution, they do not contain executable environments or a guarantee that its unit tests are effective in evaluating the correctness of a solution. Thus, we focus on 11 repositories with numerous instances and semi-manually create executable environments for them.
3.2. SWE-Gym Lite
Solving software engineering tasks is computationally in- tensive, costing usually $1 or more per task with frontier models (Wang et al., 2024c). To improve research effi- ciency via faster agent evaluation, Jimenez et al. (2024) introduce SWE-Beend Lite, canonical subset of 300 in- stances from SWE-Bench. Following the SWE-Bench Lite filtering pipeline, 3 we delineate the SWE-Gym Lite split, comprising 230 instances. Similar to SWE-Bench Lite, this subset excludes tasks that require editing more than one file, tasks with poorly descrihed problem statements, those with excessively complex ground-truth code diff', and tests focused on error message validation.
Version Training Instances. Associating instances with their respective version numbers (e.g. 1.2.3) and set- ting up environments cersion-by-versionn makes the envi- ronment collection process more practical by avoiding re- dundant setup work. We generalize SWE-Bench's ver- sioning script to support versioning via script execution, and semi-automatically collect versions for each instance based on information available in the repository (e.g., pyproject. toml, git tag, etc).
3.3. Dataset Statistics
Setup Executable Environments and Verify Instances. Creating executable environments with pre-installed de- pendencies is crucial for developing software engineering agents, as it mirrors deployment settings and allows for in- cremental unit test feedback. Configuring dependencies for specific codebase versions is challenging due to the lack of universal Python package installation method and backward compatibility issues, especially for older GitHub issues. Ig- noring these environments could introduce distribution bias, diminishing SWF--yyms utility. To address this, we man- ually configure dependencies for each task instance using relevant configuration files (e.g., requi rements.... txt),, CI scripts, or documentation from the repository snapshot the time of issue creation. We then use SWE-Bench's
Fig. illustrates that the task distribution across repositories exhibits a long-tail pattern. Notably, tasks associated with pandas comprise nearly one-third of the total, whereas tasks related to bokeh represent mere one percent.
Our analysis suggests that tasks in SWE-Gym are on av- erage harder than those included in SWE-Bench Tab. 2 shows that SWE-Gym has statistics similar to SWE--Bnch, with several key differences. Codebases in SWE-Gym, on average, have relatively fewer files than SWE-Bendh but a
Annotaatoons are done by a subset of the authors. Forr details on its construction process, see tps: : //ww. s swebench. con/ /itte. html.

---

Training Software Engineering Agents and Verifiers with SWE-Gym
similar number of total lines of code. However, gold patches in SWE-Gym have significantly more lines and files edited when compared to SWE-Bench's gold patches. Addition- ally, we find models have consistently lower performance on SWE-Gym compared to SWE-Bench." 4 Beyond models and scaffolds overlitting to SWE-Bench, the decreased per- formance on SWE-Gym may also be due to our inclusion of sophisticated repositories like pandas and MONAI.
2023). In contrast to specialized-workia wogtrsssswttttsssssentttstt ($2), it relies the LMM to generate actions and do planning. It equips the base LM with bash terminal and a file editor. We disable the browser feature of OpenHands in this work.
Trajectory Collection. By rejection sampling, we obtain 491 successful trajectories from SWE-Gym, These tra- jectories are sampled from gpt-44-2024-088 1-aude-355-sonaee -atemett222112222 with different tem- perature settings. Each successful trajectory, on average, has roughly 19 turns and approximately 19, 19,000tokens." Al- though SWE-Gym offers many more tasks and allows re- peated sampling, our 491 trajectories are limited primarily by computational budget.
and
4. Training LMs as Agents with SWE-Gym
We experiment with training language model agents using SWE-Gym. We use two agent scaffolds (OpenHands, Wang et al. 2024c, $4.2; Moatless Tools, Örwall 2024, $4.3).
Training on SWE-Gym trajectories turns LM into ef- fective agents to fix issues. As shown in Tab. 3, the pre- trained base model achieves resolution rates of 3.0% and 7.0% on SWE-Bench Lite and Verified, respectively. After
4.1. Setting
Agent Scaffolds. Recent LM-based SWE agents comprise a base language model, and a set of tools and prompts this base model has access to. This set of tools and prompting strategies is referred to as an agent scaffold, and recent work has developed numerous scaffolds for different purposes (refer to $2 for examples). We experiment with two types of agent scaffolds: one for general- purpose prompting (Open- Hands CodeAct; Wang et al. 2024c) and one for specialized workflows MoatlessToools Örwall 2024), which allows us to measure the efficacy of SWE-Gym across diverse deploy- ment settings.
fine-tuning on 491 trajectories 6 it improves by up to 12.3% (3.0% + 15.3%) and 13.6% (7.0% + 20.6%)
Training reduces stuck-in-loop behavior. For agent tasks, open-weight LMs often get stuck in loops, where the model perpetually generates the same action for multiple turns, especially when prompted with general-purpose prompts ($2). Thus, we report Stuck in Loop ( (%), the percentage of trajectories where the agent repeats the same action three times consecutively. As shown in Tab. 3, zero-shot pre- trained models often get stuck in loops; even the largest 32B model is trapped in 29.4% of SWE-Bench Verified tasks, Fine-tuning on trajectories from SWE-Gyym consistently reduces the stuck-in-loop rate by 4.6-18.6% across both SWE-Bench Lite and Verified tasks, except for the 32B model on SWE-Bench Lite, which increases by 1.5% due to its already low loop rate. This coincides with a decrease in the empty patch rate, likely enabling the agent to perform more code edits.
Policy Improvement Algorithm. We use SWE-Gym to improve the underlying LMM for given SWE agent. As a baseline, we employ a simple policy improvement algo- rithm: rejection sampling fine-tuning (a.k.a. filtered behav- ior cloning), where we fine-tune the base LM on success trajectories sampled from SwE-Gym.
Evaluation Metrics. We use the standard SWE agent bench- marks SWE-Bench Lite and Verified (Jimenez et al., 2024) for evaluation. We report (1) resolution rate ( (%), the pro- portion of resolved task instances, and (2) Empty Patch ( (%), the proportion of trajectories where none of the code in the repository is edited. We use OpenHands remote run- time (Neubig& Wang, 2024) to parallelize evaluation (e.g.. execute unit tests).
Performance scales with model size. Rather unsurpris- ingly, larger base models consistently improve the resolution rate, empty patch rate, and stuck-in-loop rate (Tab. 3).
Self-improvement remains ineffective. In addition to fine- tuning on trajectories sampled from strong teacher models, we also experiment with fine-tuning on trajectories sampled directly from the policy being updated. We use the fine- tuned 32B model to sample 6 trajectories per SWE-Gym instance (using temperature / 0.5), obtaining 868 suc- cessful trajectories (i.e., on-policy trajectories). We further fine-tune the base 32B model on mixture of 868 on-policy trajectories and the previously collected 491 off off-policy tra- jectories. When evaluating this fine-tuned model on SWE- Bench Lite, we observe the resolution rate drop from 15.3
Technical Details. For base LMs, we use Owen-22.5-Coder--1 I-Coder-Instrunn (Hui et al., 2024a) 7B, 14B, and 32B. $B.2 contains training run details.
4.2. Training General-Purposs Prompting Agents
In this section, we use OpenHands (version CodeActAgent 2. Wang et al. 2024b;c) as our agent scaffold, which is based on general-purpose purpose ReAct-style prompting (Yao et al.,
Tabb, contains more statistics of the sampled trajectories.
4 $B contains details of these experiments.
We use sampling temperature of 0 unless otherwise specified.
5

---

Training Software Engineering Agents and Verifiers with SWE-Gym
Table 3: Model performance (fine-tuned on 491 SWE-Gym-sampled trajectories) on SWE-Bench (Jimenez et al., 2024)
using OpenHands (Wang et al., 2024c) as agent scaffold. We use Owen-2. 2eeen-2.5-codde Ce---oodeIIssttee as the base model.
Model Size
Empty Patch (%, 4)
Stuck in Loop (%e, J)
Avg. Turn(s)
Resolve Rate (%, f)
zero-shot
fine-tuned
A
zero-shot
fine-tuned
A
zero-shot
tune-
A
zero-shot
fine-tuned
A
SWE-Bench Lite (300 instances)
7B
40.3
29.7
-10.7
47.0
31.0
-16.0
20.3
22.2
+1,9
(+ 1.0)
10.0 (+ 2.4)
+9.0
14B
49.7
18.1
-31.6
31.7
27.1
-.6
23.2
21.4
-188
2.7 (1 1.9)
12.7 (1 2.3)
+10.0
32B
27.0
18.1
-8.9
16.7
18.1
+1.5
15.5
29.3
+13.9
3.0 (+ 1.4)
15.3 (+ 2.5)
+12.3
SWE-Bench Verified (500 instances)
7B
45.8
33.8
-12.0
39.6
21.0
-18.6
21.9
35.3
+13.4
18 1.1)
10.6 (# 2.1)
+8.8
14B
44.9
14.5
-30.4
32.1
21.3
-10.7
25.5
30.1
+4.6
4.0 1.6)
16.4 (+: 2.0)
+12.4
32B
9.5
13.8
+4.3
29.4
23.8
-5.6
24.6
31.6
+7.0
7.0 (+: 1.3)
20.6 2.1)
+13.6
to 8.7%, suggesting that self-improvement is not yet work- ing. We hypothesize that we could achieve improved results using more advanced policy optimization methods, such as proximal policy optimization (PPO) (Schulman et al., 2017), or with a stronger base model. These directions remain promising avenues for future investigation.
Table 4: resolution rate (RR) and Empty patch rate (EP)
on SWE-Bench Lite with the MoatlessToon Scaffold after online rejection sampling fine-tuning (temperature = 0).
7B Model
32B Model
Setting
EP(%, J)
RR(%, f)
EP(%, J)
RR(%, f)
Zero-Shot
56.3%
7.0%
24.3%
19.0%
4.3. Self-Improvement with Specialized Workflow
Iteration 1
29.0%
9.0%
18.3%
19.7% %
Iteration 2
23.3%
10.0%
9.7%
19.7%
Unlike OpenHands, which offers freedom in long-horizon planning, MoatlessTools constrains the language model's action space to pre-defined specialized workflows, reducing task horizons. Specialized workflows outperform general- purpose prompting for open-weight LMs. In Tab. 3 and Tab. 4, the 7B and 32B LM achieve zero-shot resolution rates of 7% and 19% with MoatlessTools, compared to 1.0% and 3.0% with OpenHands on SWE-Bench Lite.
this balances dataset bias and size. A low cap reduces dataset size and performance ($5.2), while a high cap skews the distribution toward casier tasks. Empirically, a threshold of achieves a good balance, slightly outperforming the full dataset and improving training speed (Tab. 6). We rank trajectories by the number of model response rounds required, preferring fewer.
Given MoatlessTools' improved zero-shot performance and shorter task horizon, we hypothesize that self-improvement without strong teacher is achievable using this scaffold and training on SWE-GGmm With a limited compute budget, we conduct this experiment with only 7B and 32B models, us- ing LoRA (Hu et al., 2022) for the 32B model for improved efficiency. We use the 7B model for ablation experiments.
Results. Results. After two policy improvement iterations (Tab. 4), the 7B model's resolution rate increased from 7.0% to 9.0% after the first iteration and to 10.0% after the second. In contrast, the 32B model improved from 19.0% to 19.7% after the first iteration with no further gains. We attribute the limited gains in the 32B model to the scaffold's restricted action space and the rejection sampling fine-tuning method.
We use iterative rejection sampling fine-tuning for policy improvement. Each iteration involves (a) performing 30 tigh-temperature (1.0) rollouts per task on SWE-Gym-Lin -Ltee and adding successful trajectories to the fine-tuning dataset, and (b) fine-tuning the policy on these filtered trajectories. After two iterations, further improvements are negligible.
5. Scaling Agent Performance with SWE-Gym
We explore two scaling directions enabled by SWE-Gym to enhance agent performance: inference-ting time scaling ($5.1) and training-time data scaling ($5.2).
Data Bias Impacts Performance. Repeated sampling, as in Brown al. (2024), shows that task success probability follows long-tail distribution (Fig. 6). where more samples increase solved instances. While broader task coverage benefits training, it introduces a bias toward easier tasks, making it suboptimal to train on all successful trajectories, as first observed in math reasoning Tong et al. (2024).
5.1. Inference- Scaling with Verifiers
Trajectories sampled from SWE-Gym can be used not only for training a policy, but also for training verifier (i.e., reward) model. We train an outcome-supervise model (ORM) (Cobbe et al., 2021) that, given the relevant context of the task execution (including the problem state- ment, agent trajectory, and current git diff), generates score that estimates the probability that the agent has solved the problem. We experiment with using this model to rerank
reward
Mitigating Bias with Per-Instance Capping. We introduce per-instance capping- method that limits the maximum number of selected samples per task. As illustrated in Fig. 6,
6

---

Training Software Engineering Agents and Verifiers with SwE-Gym
candidate trajectories sampled from SWE agent policy, and show that such learned verifiers enable effective inference- time scaling for further performance improvement.
Pan Besbök
40
SWE- Verifted Resalve
35
22,09
5. VERIFIER FOR GENERAI. PURPOSE PROMPTING
30
28.8
For OpenHands agents ( (Wang et al., 2024b;c) with general- purpose prompting ($2). we train a verifier (ORM) that takes as input the trajectory T = [01: 01; 02, 02,.... : On: Onl, represented an interleaved sequence of observations and actions, and generates scalar reward r [o, 1]. Obser- vations o& include the task problem statement, command execution output, error messages, etc; action aş can be bash command or file operations (e.g., edit, view) from the agent.
24.0%
25
20
Number of Agent Rollouts
Figure 3: Increasing inference-time compute improves performance SWE-Bench Verified with a learnt verifier. Both the agent and the verifier are Qwen2. --e.es--oorr--tteeeeeeeee on the corresponding dataset ($5.1.1). OpenHands is used as the agent scaffold.
Training and Inference.
We fine-tune 32B label trajectories
model fine-tumed
Qwen2. owen2.5-Car 1eee2s5--Coer-Instrun as successful or unsuccessful using output tokens <YES> and <NO> respectively.' For training data, re-use two sets of trajectories we sampled on SWE-Gym for agent training in $4.2: (1) off-policy trajectories which contain 443 successful trajectories; (2) on-policy trajectories which contain 875 successful trajectories sampled from the fine-tuned Owen.2.
lect the best trajectory, demonstrates more modest but steady progress, improving from a resolution rate of 20.6@) 1 to 29.8@8, and up to 32.0@ 16. The gap between Pass @ k and @ due to the imperfect performance of our trained verifier, indicates there is room for improvements in reward modeling for coding agents. Surprisingly, we found that fine-tuning the verifier model using LoRA (Hu et al., 2022) (29.8 10,864 @ 8) with Unsloth (Unsloth Team, 2024) performs bet- ter than full uull-parmmeter fine-tuning (27.2@8), (), potentially due regularization. Furthermore, as shown in Fig. 1 (bottom), the Best &s curve exhibits strong linearity on a logarithmic scale, indicating a promising scaling behavior.
We combine both on-policy and off-policy trajectories, randomly sample the same amount of unsuccessful trajec- tories from each subset (1,318 each), and combine them as our dataset for verifier training (total 2,636 trajectories). We fine-tune the model to predict <YES> for successful trajectories and <NO> for unsuccessful ones.
At inference time, conditioned on the prompt and the agent trajectory т, we use SGLang (Zheng al., 2024a) to ob- tain the log probability of the next token being <YES> (ly) or <NO> (I,). We then calculate the reward as the probability of success by normalizing the log probability: r r= exp((y)((xxp(()) + exp(lx)).
Training data matters for verifier. We experiment with variations the choice of training data for our verifier model. Using full-parameter fine-tuning on Qwen 2. So-Cdde--oo Instruct-322 we use different mixtures of on- and off-policy trajectories, as well as differ- ent distributions of successful and unsuccessful trajectories. As shown in Fig. 8, our ablation study demonstrates that the choice of training data can significantly impact verifier performance. Training with mixture of off-policy and on-policy data yields the best results (our default setting), reaching a resolution rate of 27@8. In contrast, using only on-policy data from the fine-tuned model shows moderate but limited improvement, while training exclusively on off- policy data from Claude and GPT leads to early performance plateaus around 22% resolution rate. Our findings indicate that verifier training benefits most from diverse dataset combining both off-policy and on-policy examples.
Metrics. We report two metrics: (1) @ass@ kt the propor- tion of tasks with at least one successful solution among k samples, and (2) Best@k k. the success rate of the highest- reward trajectories selected by the verifier from & samples per task. Pass@1 @ k measures solution discovery (upper bound for @est@ k); Best@k k evaluates verifier accuracy. Mean and variance calculation are detailed in $B.1, following Lightman et al. (2023).
Results. Fig. shows how k and Best@ scale with the number of sampled agent trajectories using the fine- tuned 32B model as the agent model. Pass @ k demonstrates strong improvement, rising from 20.6 to 37.8% resolution rate as l increases from to 8, and up to 42.6@1116.... The Best@) l: metric, which relies on our verifier's ability to se-
1.1.2. VERIFIER FOR SPECIALIZED WORKFLOW
7 BB6 includes the verifier prompt template. We keep only trajectories within 32k-token length for training. which may reduce their number compared to Section 4.2.
For MoatlessTools agents with specialized workflows, given that doesn't 't have a turn-taking netion-observation trajec- tory like OpenHands CodeActAgent, we prepare verifier
7

---

Training Software Engineering Agents and Verifiers with SWE-Gym
34
Pass
32.15%
32.9%
9%.
33.3%
SWE SWE-Bend -reE-Benchh Verified Resolved (%)
Scaling number of training trajectories
32
Bestgk (328 GRM)
Be (RM)
29.6%
31.0%
20
32B
7B
Resolve (%)
27.3%
28
26.0%
15
26
24.4%
24.3%
24.9%
25.4%
23.5%
24
10
22
20
18
1.33%
10.1%
19.9%
20.3%
20.9%
21.1%
21.2%
21.1%
21.0%
25%
50%
100%
Percentage of Training Data Used
Number of Agent Rollouts
Figure 5: Scaling effects of increasing the number of ran- domly sampled trajectories for training.
Figure 4: Scaling inference-time compute for MoatlessTools Agents (32B) with learned verifiers SWE-Bench Lite. Temperature f 0.5.
Setup. Using OpenHands (Wang et al., 2024c) and the fine- tuning approach described in $4.2, we evaluate these scaling approaches on SWE-Bench Verified: scaling the number of trajectories, by subsampling from the full trajectory dataset from $4.2 (at most 491 trajectories); unique instance scal- ing on these trajectories deduplicated by instance ID (at most 294 trajectories), and repository-based scaling where sort repositories alphabetically and include all trajecto- ries from each repository in order (e.g., first 25% contains complete trajectories from the first N repositories). We com- pare models trained on 25%, 50%, and 100% of the full dataset for each approach, sampling training subsets using the methods described above for each scaling approach. 10
inputs through a parsing process adopted from Zhang al. (2024a), which combines task descriptions, relevant agent context, and generated patches." 2 We train the verifier to map from this input to single token indicating task success.
Following the training procedure described in $5.1.1, we train 7B and 32B verifiers using on-policy trajectories from the last (2nd round of sampling, applying LoRA (Hu al., 2022). To address the easy-data bias in the training dataset, cap the number of successful trajectories per instance at two and balance the data by subsampling failure cases to match the same number of successful ones.
Results. We evaluate the verifiers by sampling from an agent policy with k: = at temperature 0.5. As shown in Fig. and Fig. 7, these verifiers enable effective scaling across verifier and policy sizes: the 7B verifier improves from 10 to 13.3% resolution rate on SWE-Bench Lite when paired with 7B policy, while the 32B verifier improves from 19.7 to 26.3% when paired with 32B policy. The 7B verifier plateaus after = 4 samples when ranking trajectories from both 7B and 32B agents. In contrast, the 32B verifier continues improving even k suggesting that verifier size significantly affects scaling behavior.
Scaling trends suggest instance and repository diversity is not yet a bottleneck. Fig. 5 demonstrates substantial scaling behavior, with consistent improvements in resolu- tion rate as the number of training trajectories randomly increases, particularly for the 32B model. These results suggest that SWE-Gym's 's current size and repository diver- sity are likely not performance bottlenecks further im- provements could likely be achieved by allocating more computing resources sampling more training trajectories.
Fig. 9 reveals comparable overall performance between dif- ferent scaling approaches up to where deduplication takes effect. While Random Scaling (No Dedup.) achieves higher final performance, this is likely due to having more tra- jectories (491 vs 294) rather than better scaling efficiency, Among deduplicated approaches, Repository Scaling shows stronger initial performance at 25% data, suggesting that complete repository coverage may provide more coherent learning signals early in training. These results suggest that the repository and instance diversity of SWE-Gym is not yet a bottleneck further improvements could likely be achieved by simply sampling more agent trajectory data for traning, regardless of duplication or repository distribution.
5.2. Training- Time Scaling with Data
We then examine how scaling the amount of training data affects agent performance using 491 sampled trajectories from $4.2. We simulate three scaling methods through subsampling: (1) Scaling trajectories. where trajectories are randomly dropped (Fig. 5); (2) Scaling unique task instances, where only one successful trajectory per task instance is selected (Fig. 9); and (3) Scaling repositories, which sequentially includes all instances from each reposi- tory to assess repository-level diversity.
We provide the prompt template in $B.5.
1ab. contains detailed statistics of these datasets.
8

---

Training Software Engineering Agents and Verifiers with SWE-Gym
6. Conclusions, Limitations, and Future Work
could potentially be misused to automate the creation of ma- licious code. We encourage future work to further explore frameworks for responsible deployment of software engi- neering agents, including considerations around security, safety, and economic impacts.
In this paper, we introduce SWE-Gym, the first training environment that addresses critical gaps in enabling scal- able learning for software engineering agents. By combin- ing real-world Python tasks with repository-level context, pre-configured execution environments, and test verifica- tions, SWE-Gym will be a foundation for advancing LM agent training research. Through extensive experiments, we demonstrate that SWE-Gym enables both agent and verifier models to achieve significant improvements resolving complex software tasks. Our findings highlight the scalabil- ity of these approaches, revealing potential for continuous performance gains with increased compute.
A Acknowledgments
We thank John Yang and Ofir Press for helpful discussions, and John Yang for assistance in reproducing data analysis results from SWE-Bench. We thank Modal Labs"! for the GPU compute support through its Academic Credits Pro- gram. XW and HJ are partially supported by U.S. DARPA ITM Program No. FA8650-23-C-7316. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the offi- cial policies, either expressed implied, of DARPA, the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
We see many research directions that we are excited to explore in the future:
1. Automatic Environment Synthesis SWE-Gym, while effective, is limited by its environment diversity, in- cluding the number of repositories, types of tasks, and programming languages. We view environment syn- thesis-via automated environment creation, test-case generation, task generation critical next step.
References
Akkaya, I., Andrychowicz, M.. Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., al. Solving rubik's cube with a robot hand. arXiv preprint 1rXiv:1910.07113, 2019.
2. Self-Imprrovemett with Reinforcement Learning Despite notable progress, our inffovmproeeentt results are modest. Training language model agents with large- scale online reinforcement learning is promising di- rection for further improvements.
Badertdinov, I., Trofimova, M., Anapolskiy, Y., Abramov, S., Zainullina, K., Golubev, A., Polezhaev, S., Litvintseva, D., Karasik, S., Fisin, F., Skvortsov, S., Nekrashevich, M., Shevtsov, A., and Yangel, B. Scaling data collection for training software engineering agents. Nebius blog, 2024.
3. Human-Agent Interaction Current SWE settings fo- cus solely on task completion, neglecting human-in -n- the-loop collaboration, which essential for real-world software engineering. Methods like user simulation or learning from offline human-agent interaction data might offer ways for developing collaborative agents that align with human.
Bai, H. Zhou, Y., Cemri, M., Pan, J., Suhr, A., Levine, S., and Kumar, A. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. ArXiv, abs/2406.11886, 2024. URL
Impact Statement
org/ /ttC//oopaDD::005220000
This work presents SWE-Gym, environment for training software engineering agents, with strong empirical results on its effectiveness. We discuss few important societal implications to consider. First, improving automated soft- ware engineering capabilities could increase developer's s productivity and accessibilityy across industries. Although current models are primarily research artifacts and not yet production-ready, they can support critical open-source in- frastructure and potentially make software development more accessible. Secondly, as these agents become more capable, they may impact software engineering jobs and require careful consideration around code ownership, li- censing, and attribution. Additionally, while we focus on legitimate software engineering tasks, similar techniques
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., R'e, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling. ArXiv, abs/2407.21787, 2024. URL https: www..esmmttiicccchoooooe wwwp:aaa..ggoooooaaooeeeee.peeee 271571035.
Chen, B., Shu, C., Shareghi, E., Collier, N., Narasimhan, K., and Yao. S. Fireact: Toward language agent fine-tuning, ArXiv, abx/2310.05915,, 2023. URL https: //api. www.ssseennnsccoaar...ooom Corpppu/ 263829338.
" "nttps://nodal.cr
9

---

Training Software Engineering Agents and Verifiers with SWE-Gym
Chen, D., Lin, S., Zeng. M., Zan, D., Wang, J.-G., Cheshkov,
2022, Viriual Event, A5-29 2022. Open
A., Sun, J., Yu, H., Dong, G., Aliev, A., Wang, J., Cheng, X., Liang, G., Ma, Y., Bian, P., Xie, T., and Wang, Q. Coder: Issue resolving with multi- and task graphs.
2022. URL hww.::ttps///etttm /ttp:://///ppeeeeeeeeeee net/ forum? id-nZeVKeeFYEg,
CoRR in ArXiv, 0hh/2400.001344 2024.
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L.,, Liu, T., Zhang, J., Yu, B., Dang, K., al. Qwen2. 5- coder technical report. arXiv preprint arXiv:2409. 12186, 2024a.
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pondé, H., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L.,, Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D. W., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Babuschkin, I., Balaji, S., Jain, S., Carr, A., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M. M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large lan- guage models trained on code. ArXiv, 0oos/107.003777 2021. URL https:// /atp//aapiiaaa/ wemanticsccoooooooooeeie.. org/ /ttp//oouuii:33555:50
Hui, B., Yang. J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Dang, K., al. Qwen2. coder technical report. arXiv preprint arXiv:2409. 220.224999111. 2024b.
Jain, N., Shetty, M., Zhang, T., Han, K., Sen, K., and Stoica, 1. R2E: turning any github repository into a programming agent environment. In Forty-first International Confer- ence Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: //openreview net KXttruunyyia-KKxxxyyye
Jimenez, C. Yang, J.. Wettig, A., Yao, Pei, K., Press, O., and Narasimhan, K. R. Swe-hench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024. URL httph:ttp///cc htt.tttt///peeeeeeeee/ wwtp//wwnweee..eeeeeeeee. net/ forum?id=Vin 8YN0M66.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. ArXiv, abs/2110.14168, http::///a.....sisssesseeeeee
2021. URL http://oooutt:::39980.0
Golubev, A., Polezhaev, S., Zainullina, K., Trofimova, M., Badertdinov, I., Anapolskiy, Y., Litvintseva, D., Karasik, S., Fisin, F., Skvortsov, S., Nekrashevich, M., Shevtsov, A., Abramov, S., and Yangel, B. Leveraging training and search for better software engincering agents. Nebius and- blog, 2024. http::///bbbuusscooooooo/s//s/// agents.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. ArXiv, abs/2305 20050,, 2023. URL https: //api www.sseeeattttiteeeeeeeeeeee.eee owt.loorrrr//ooputttt 258987659.
Ma, Y., Cao, R., Cao, Y., Zhang, Y., Chen, J., Liu, Y., Liu, Y., Li, B., Huang, F., and Li, Y. Lingma swe-gpt: An open development-proon prapeentttrrree-eenttiii language model for automated software improvement. arXiv preprint arXiv:2411.00622 2024.
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He. H., Song, D., and Steinhardt, J. Measuring coding challenge competence with APPS. In Vanschoren, J. and Yeung, S. (eds.). Pro- ceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021a.
Modal. Modal: High- Al infrastructure. https:/ ptops://modie /odaa comm, 2024. Accessed: 2024-12-
Hendryeks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E,, Song, D. X., and Stein- hardt, Measuring mathematical problem solv- ing with the math dataset. ArXiv, abs/2103.03874, 2021b. URL https: /ttpps:///pp...e Ct:p:CoopSII::::21::00:: ID: http:////aa..ssaaiisssssseee sew.ntisschhh.l.................
18.
Neubig, G. and Wang, X. Evaluation of LLMs as Coding Agents on SWE-Bench (at 30x Speed!). All Hands Al blog, 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-2777 2022.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adapta- tion of large language models. In The Tenth Interna- tional Conference on Learning Representations, ICLR
10

---

Training Software Engineering Agents and Verifiers with SWE-Gym
Pan, J., Zhang, Y., Tomlin, N., Zhou, Y., Levine, S., and Suhr, A. Autonomous evaluation and finement of digital agents. ArXiv, abs/2404.06474, 2024. URL https:/ http//pp.ssaaiicccchooooos org/CorrussD:2290055
Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., and Ji, H. Executable code actions elicit better LLM agents. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, OpenReview.net, 2024b. URL https: / //openreview wwweeee//oouee www.moormmmmm did
PyTorch Team. torchtune: PyTorch native post-
training library. https://githani com/pytorch/ torchtune, 2024.
Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J.. Song, Y., Li, B., Singh, J., Tran, H. H., Li, F., Ma, R., Zheng, M., Qian, B., Shao, Y., Muennighoff, N., Zhang, Y., Hui, B., Lin, J., Brennan, R., Peng, H., Ji, H., and Neubig, G. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. CoRR in ArXiv, abs/2407.167411 2024c.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL httttps://wwelm.giiiii. giith 10/blog/qwen2.55
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. ArXiv, abs/1707.06347 2017. URL https:/ttttt //api. semanticscholie htt.:aaa.....eooooaaaaa222220000
Wu, Z., Bai, H., Zhang, A., Gu, J., Vinod Vydiswaran, V., Jaitly, N., and Zhang, Y. Divide-or-conquer? which part should you distill your Ilm? ArXiv, 2024.
Shao, 7.,, Wang, P., Zhu, Q., Xu, R., Song, J. Bi, X., Zhang, H. Zhang, M., Li, Y., Wu, Y. et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033300 2024.
Xi, Z., Ding, Y., Chen, W., Hong, B., Guo, H., Wang, J., Yang, D., Liao, C., Guo, X., He, W., Gao, S., Chen, L.. Zheng, R., Zou, Y., Gui, T., Zhang, Q.. Qiu, X., Huang, X., Wu, Z., and Jiang, Y.-G. Agent- gym: Evolving large language model-based agents across diverse environments. ArXiv, abs/2406.04151, 2024. URL https://api http:///aap.ssaaaiiiieesm www.steesace.oooeeeeeeee org/ /tt///oorD:::22285550
Silver, D., Hubert, T., Schritwwieser. J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L.., Kumaran, D., Grae- pel, T., Lillicrap, T. P., Simonyan, K., and Hassabis, D. Mastering chess and shogi by self-play with general re- inforcement learning algorithm. ArXiv, abs/1712.01815, 2017. URL https: http:://app.paa...aaaeeee/ weeeeettiisciiie org/Corppln I 080008118....
Xia, C. S., Deng, Y., Dunn, S., and Zhang, L. Agentless: Demystifying lim-based software engineering agents, CoRR, abs/2407. 0.9449.,,.,,,,, 2024. doi: 10.48550/ARXIV. 2407.01489. URL https: ptstts://dol htt:://oi..ooo http:///bii.ooo..8333333333 arXiv, 21A.v.2407.0144
Tao, N., Ventresque, A., Nallur, V., and Saber, T. Enhanc- ing program synthesis with large language models using many-objective grammar-guided genetic programming. Algorithms, 17(7)2887, 2024. doi: 10.3390/A17070287. URL httpt: www.dooooooooooooooooooooooooooe 3390 a17072287
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. Swe-agent: Agent- computer interfaces enable automated software engineer- ing. CORR, 010p222445.15533, 2024. doi: 10.48550/ARXIV, 2405. 15793, URL https: /ttp///oiio/a////////oooo 485504450/00 a arxi 2405. 15793.
Tong, Y., Zhang. X., Wang, R., Wu, R. M., and He, J. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving. ArXiv, abs/2407.13690, 2024. URL hattps://api... wtt.app...saaatisccclooooooeoeoe oxg/CoozpusID:2217705
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The Eleventh Interna- tional Conference
Unsloth Team. Easily finetune and train LLMss. Get faster with unsloth. https: : h/tp://uuns/ohhoo/eee ai/, 2024.
Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: http::///ppeeeeeaeeeeeeee opeprevieneeeeeeeeeeeee net /oormm? id=WE_ve DETEEvvlle vIUUUI-X.
Wang, P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-shepherd: Ver- ify and reinforce LLMs step-by-step without human an- notations, In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 9426-9439, Bangkok, Thailand, August 2024a. Association for Computational Linguis- URL https: 510.
Yuan, L., Cui, G., Wang, H., Ding, N., Wang, X., Deng, J., Shan, B., Chen, H., Xie, R., Lin, Y., Liu, Z., Zhou, B., Peng, H., Liu, Z., and Sun, M. Advancing LLM reasoning generalists with preference trees. CoRR, Abs/2404.02078 2024. doi: 10.48550/ARXIV22444.2222 //doi. /ttp//.....45/5//// 48550 2404 02078.
tics. doi: 10. 10.18655/1/222222dlllooggg s/aclanthon ogy. www../hwww.oo....././2////
URL https:

---

Training Software Engincering Agents and Verifiers with SWE-Gym
Zeng, A., Liu, M., Lu, R., Wang, B., Liu, X., Dong. Y., and Tang, J. Agenttuning: Enabling gener- alized agent abilities for Ilms. In Annual Meet- ing of the Association for Computational Linguistics, 2023. URL https: /appiapp.../// http:///.....smaane.e....oe. www.stiisccooool......... org/ h htg/CorpusII:2555 2543:II::54401600
Zhai, Y., Bai, H., Lin, Z., Pan, J., Tong, S., Zhou, Y., Suhr, A., Xie, S., LeCun, Y., Ma, Y., and Levine, S. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. ArXiv, abs/2405.10292. 2024. URL https: http//pp...aaaiccccco.ooooo atp/CorppsSD::2579977
Zhang, K., Yao, W. Liu, Z. Feng, Y., Liu, Z., Murthy, R., Lan, T., Li, L.. Lou, R., Xu, J., Pang. B., Zhou, Y., Heinecke, S., Savarese, S., Wang, H., and Xiong, Di- versity empowers intelligence: Integrating expertise of software engineering agents. ArXiv, abs/2408.0700 07060, 2024a. URL https: swtppiisemmattiiiiiiiieeo.o
Zhang, Y., Ruan, H., Fan, Z., and Roychoudhury, Au- tocoderover: Autonomous program improvement. In ISSTA, 2024b.
Zhao, W.. Jiang, N., Lee, C., Chiu, T., Cardie, C., Gallé, M., and Rush, A. M. Commitû: Library generation from scratch, 2024. URL attps://arke org/ /bs/ 2412.01769.
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Bar- rett, C., and Sheng, Y. Sglang: Efficient execution of structured language model programs, 2024a. URL https: //tp://aaaiii
071004.
Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W., and Yue, X. Open- codeinterpreter: Integrating code generation with ecution and refinement. ArXiv, 146524 2024b. URL https: http://ppp....a.aaaaaaemmmmamaaa orr//o::pusD::25778225
Örwall, A. Moatless Tool. https: //gthhbb. htmm//rrwal/mmaalessttto 2024. Accessed: 2024-10-22.
12

---

Training Software Engincering Agents and Verifiers with SWE-Gym
A. Comparison with Concurrent Works
Ma et al. (2024) trains an LM agent, Lingma SWE-GPT, using a method similar to our rejection sampling fine-tuning baseline, with dataset comparable to our SWE-Gym Raw splits. Without executable unit test feedback, they rely on manually defined heuristics to filter out low-quality trajectories. such as comparing similarity between submitted patches and edit locations with gold patches. The model weights are publicly accessible but not the training pipeline the dataset.
Most relevant to our work are two consecutive blog posts by Golubev et al. (2024) and Badertdinov et al. (2024), who also construct an executable training environment with real-world tasks from GitHub. Instead of manual configuration, they employ general environment setup script and simply discard instances that fail the setup process. This approach leads to key differences in dataset size and distribution: while it biases the environment away from tasks with complex dependencies, they successfully collect 6,415 instances, about 1.5 times larger than our dataset. In Golubev et al. (2024), they also study training agents and verifiers with the environment. Additionally, they explore lookahead setting where a trained verifier ranks and selects the best next action. With substantially large collection of agent trajectories (80,036 compared to thousands in our experiments) and model size (72B compared to 32B), Their best system achieves 40% accuracy on SWE-Bench Verified. While their dataset and agent trajectories are publicly accessible, the model is not.
In comparison, with a comparable dataset size, our SWE-Gym has executable feedback, avoids potential dataset bias through manual configuration of environments, while providing comprehensive analysis of agent and verifier training, their scaling behaviors, and positive results on agent self- ceffimmproveeet.. Our system achieves competitive results with significantly lower compute and smaller model size (32B vs 72B). Lastly, we open source all artifacts of the project, including dataset, model weights, agent trajectory data and the training pipeline.
Model
SWE-Bench
Openness
Name, Model Size
Lite
Verified
Model Environment
Ma et al. (2024), 72B
22.0
30.2
Golubev et al. (2024) Agent and Verifier, 72B
40.6
Our SWE-Gym Agent and Verifier, 32B
26.0
32.0
Table 5: Comparison of model performance on SWE-Bench benchmark and the model weights and environments are publically accessible (openness).
Cap
Traj
Empty Patch (%,
resolution rate (%, f)
(Zero-shot)
0
56.3
7.0
36
37.3
9.0
2
62
29.0
9.7
82
43.7
7.7
Cap (All)
172
30.7
9.3
Table 6: resolution rate and empty patch rate on SWE-Bench Lite with a 7B model trained using different instance capping strategies (Cap).
B. Experiment Details
B.1. Mean and Variance for @ass@N N and Best@N.
We mostly follow (Lightman et al., 2023) for obtaining the mean and variance for the Pass@N and Best@N curve. Given total of M rounds of rollouts, for N < M, we calculate the mean and variance across 100 randomly selected sub-samples of size N from the M rollouts. For the OpenHands CodeActAgent inference-time scaling curve at $3, we exclude this calculation for N=1 as we use a temperature of 0 for the first attempt.
B.2. OpenHands Agent Experiments
During training, we use OpenHands's remote runtime (Neubig & Wang, 2024) feature to execute agent trajectories in parallel on SWE-Gym. We use torchtune (PyTorch Team, 2024) for full parameter fine-tuning with a learning rate of 1e-4, maximum epochs, global batch size 8. max context length of 32 7 68. We fine-tuned both 7B, 14B, and 32B variant of the model, and experiments were performed with 2-8x NVIDIA H100 80G GPU on modal (Modal, 2024). The
13

---

Training Software Engincering Agents and Verifiers with SWE-Gym
Original
Dedup.
Sorted by Random (Dedup.)
Sorted by Repo (Dedup.)
First 25%
First 50%
First 25%
First 50%
getmoto/moto
155
72
12
33
46
Projet-MONA//MNNA
95
53
17
25
53
53
pandas-dev/pandas
70
61
14
30
0
python/mypy
46
27
7
12
0
0
dask/dask
45
29
8
17
6
29
iterative/dve
36
24
8
12
conan-io/conan
20
12
7
12
12
pydantic/pydantic
11
2
4
facebookkeexarcchhyyrrz
7
2
5
bokeh/bokeh
3
1
1
modin-project//modle
3
1
1
0
Total
491
294
73
147
73
147
Table 7: Distribution success trajectories used in training-time scaling experiments ($5.2). Dedup. denotes that the trajectories are deduplicated by randomly select ONE success trajectory per instance ID; Sorted by random (repo) X % (Dedup.) denotes a subset trajectories taken from the first X% from dedup. instances that are sorted randomly (by repository name).
Percentiles
Resolved
Count
Mean
Std
Min
Max
5%
10%
25%
50%
75%
90%
95%
Num. of Messages
5, 555.7.
39.2
31.9
7.0
101.0
9.0
9.0
9.0
20.0
61.0
100.0
101.0
491.0
T'Tokens
x 5, 555.07
17. 218.3
17. 761.6
1. 615.0
837.
1,330.0
1.907.0
268.0
105.05.0
26, 131.0
182.2
51. 780.6
491.0
18, 578.5
11,361.4
560.0
81,245.0
5,813.0
8,357.0
11, 559.5
15,909.0
24.55
31, 632.0
39, 512.5
Table 8: Statistics of SWE-Gym-sampled trajectories. We use the tokenizer from Qwen-2, estimate the number of tokens.
2.. 5-
to
14

---

Training Software Engincering Agents and Verifiers with SWE-Gym
20
Cap
# Successfu Trajectories
15
10
5
10
15
20
25
30
35
Instance ID (Sorted By # Successful Trajectories)
Figure 6: Success distribution over 30 rounds on SWE-Gym Lite with 7B model in zero-shot. The distribution is naturally biased toward easy tasks. Per instance capping reduces this bias but lowers the total trajectory count for training. We set temperature during sampling.
only exception the main experiment $5.1.1, where we use LoRA (Hu et al., 2022) (29.8% @8) via Unsloth library (Unsloth Team, 2024) to train verifier for max 2 epochs, while other hyper-parameter stays the same.
Inference during evaluation bounded by either 100 interaction turns the base LM's 32k context window length, whichever is reached first.
B.3. MoatlessTools Agent Experiments
All MoatlessTools models are trained with a context window of 10240. For experiments with the 7B model, we use torchtune train the policy model with full-finetur full-finetuning using H100 GPUs. We set batch size 8, learning rate 2 10-5 and train for 5 epochs.
For the 32B model, we use Unsloth (Unsloth Team, 2024) with a single H100 GPU for LoRA fine-tuning. We set the number of epochs to 5, batch size to 8, LoRA rank to 64, and learning rate to x 10-4, We use the same configuration for verifier training.
For AoatlessAgent experiments, we serve the agent with FP8 quantization for improved throughput, which we found have minimal effects model performance.
B.4. Details of OpenHands Trajectory Sampling
As detailed in Tab. 10, we collect a few sets of trajectories for fine-tuning experiments. We collect dataset Do by sample gpt-4o-22 2024-88--2
on SWE-Gym Lite with temperature and collected 19 trajectories that eventually solve the task (evaluated by unit test in SWE-Gym). We then varied the temperatures (setting t= 2, 0.3, 0,4, 0.5, 0. 8)) and sample on SWE-Gym Lite. Combining these instances with Do. we get 106 trajectories that solve the given problem (D1). We set the maximum number of turns to be 30 for both Do and Dr. To experiment on the (19 resolved out of 230)
effect max turn, we set max number of turns to 50 and sample gpt-4o-2024-08 (67 resolved out of 230) with temperature 0 on SWE-Gym Lite, and sample gpt-40-2024-00 06 (temperature t=(0, 1}) on SWE-Gym full set (in total 299 resolved out of 4876 instances).
and www.wdeee----- --ieee------ssn--t-----------
15

---

Training Software Engineering Agents and Verifiers with SWE-Gym
Agent
Model
Model Size
Training Data
Resolved (%)
SWE-Bench Verified (500 instunces)
SWE-Llama Uimencz al., 2024)
instancees
RAG
SWE-LLama (Fimenez et al,, 2024)
13B
10K instances
1.2
I Lingna Agent (Ma al., 2024)
Lingma SWE-GPT (v0925)
90K PRs from 4K repos
18.2
Lingma Agent 2024)
Lingma SWE-GPT (v0925)
72B
repos
OpenlHunds (Wang 2024c) (Ours)
fine tuned Qwen2 Coder Instruct
32B
491 ageni trajectories from repos
20.6
Openitands w/ Verilier (Wang et al,, 2024c) (Ours)
fine-tuned Qwen2.55 5 Coder- Instruct
32B (Ageni & Verilier)
491 ageni trajectories from 11 repos for agent 1318 x succassfallure agent trajectories verifier
32.0
Table 9: Performance comparison with SWE-Bench (Jimenez al., 2024) baselines with publicly accessible weights. Data
source: https: /ww. swebench. com/, Accessed on Dec 21, 2024.
Trajectory
Sampled from Model
Sampled on Dataset
Temperature
Max Turns
Success trajectories
Do
gpt 10lo-2224-000
SWE-Gym Lite
30
19 (8.26%)
(Cumulative) Total Do
19
SWE-Giym Lite
0.2
30
11 (4.78%)
2ot=44-2024-08-11
SWE-Gym Lite
0.3
30
17 (7.39%)
Dj \ Do
gpt-4o-2000 20244-0-0022
SWE-GGym Lite
0).4
(9.13%)
gpt-40-2024-08-00
SWE-Gym Lite
0.5
30
18 (7.83%)
gpt 9P:-10-2024-08-06
SWE--Gym Lite
0..8
30
(8.70%)
(Cumulative) Total D₁
106
gpt 40 2pt-44-2024-180 2004-00-06
SWE-Gym Lite
0
50
19 (8.26%)
SWE-Gyy Lite
0
50
67 (29. 1%))
D2 \
gpt-4o-2024-085 gpt gpt-40-2024-08-00
WWE-Gym Full
"111 (4.55%)
SWE-Giym Full
50
188 (7.71% (1)
(Cumulative) Total D2
491
Run into infrasenectuue-eellaeee ernor where somee instunces failed to complete, this number might be under estimate of actual mumber of suocess trijentoiiss
Table 10: Summary of trajectories sampled from SWE-Gym.
This gives us in total 106 19 299 491 success trajectories, which forms our final training trajectories D2-
B.5. MoatlessTools ORM Prompt
The following a pseudo-code that generates prompt for MoatlessTools Verifier (ORM), which is modified from (Zhang al., 2024a). Unlike (Zhang et al., 2024a), which relies on proprietary models like Claude-3.5-Sonne' for context extraction, obtain context directly from the agent's trajectory being evaluated.
SYSTEM_MESSAGE """You are expert in python for software engineering and code review. Your responsibility is to review the patches generated by language models to fix some issues and provide feedback on the quality of their code. в*"
MESSAGG """I want you evaluate an LLM -MM-genertted candidate patch that tries to resclve an issue in codebase.
To assist you in this task, you are provided with the following information:
You are given an issue text on a github repository (wrapped with
- wtt.ddrrppciooossssseededdeeeeee
You are also given some identified code spans that are relevant to the issue.
Each code span is wrapped with <code_span file Sile_path=FILE_PP
apan_id=SPAN ID></code span> tags, where FILE PATH is the path to the file containing the code span, and SPAN_ID is the unique identifier for the code span.
16

---

Training Software Engincering Agents and Verifiers with SWE-Gym
Each code span also comes with the line numbers for you better understand the context. It's possible that the code span not sufficient to fix the issue, adjust your score accordingly,
You are given the candidate patch that tries to resolve the target issue. For your convenience, you are given the hunks of original code and the code 4 after applying the patch.
The code before the patch is wrapped with deefore_ptcchx///ffooeeedddddd and
4 the code after the patch is wrapped with cafte__arcc><</ftteett
Note that the file names in before_patch starts with "a/" and the file names in after_patch starts with 'b/'.
dassue_descriptin
(issue_text]
s/issue_description
<before_patchh
(before_patch)
</before_patch>
<after_patch
(after_patch)
</after_patch>
(code_spans)
Response in "True" or "False" for whether the patch has resolved the issue.
B.6. OpenHands ORM Prompt
The following pseudo-code that generates a prompt for OpenHands Verifier (ORM).
SYSTEM
" You are an expert judge evaluating assistant interactions. Your task is to determine if the assistant successfully resolved the user's
request.
Key evaluation criteria:
1. Did the assistant complete the main task requested by the user?
2. Did the assistant handle all edge cases and requirements specified?
3. Were there any errors or issues in the final solution?
4. pid the assistant verify the solution works intended?
Respond only with "<juddgmenttxXIss//jjgeeeett or "KjudeenentbxOO4//ddddeeeaa""""
USER_MESSACE IPpleas evaluate the following interaction between an AI
+ assistant and a user:
INTERACTION LOG + trai_str ..
=== END INTERACTION ===
Based on the above interaction, did the assistant successfully resolve the user's + initial request? Respond with YES or 11
17

---

Training Software Engincering Agents and Verifiers with SWE-Gym
messages [
('role': 'system', 'content': : SYSTEM (ESSTEMMMESSGEE
role': luser', 'content': USER_MESSAGE),
('role': 'assistant', 'content': '<judgement>" + ("YES" if resolved else "NO") + '</judgement>>)
]
The last assistant messages that contains judgement is only provided during training time. At inference time, the trained verifier is responsible predicting the probability of 'Yes' and 'No'.
18

---

Training Software Engincering Agents and Verifiers with SWE-Gym
33.3%
32.9%
34
32.1%
Pass@k
SWE-Bench Lite Resolve Rate (%)
Best@k (32B Verifier)
31.0%
32
Best@k (7B Verifier)
29.6%
30
27.5%
28
26.0%
25.4%
24.9%
26
24.4%
24.3%
23.5%
24
22.6%
21.6%
22
19.1%
20
20.9%
21.1%
21.2%
21.1%
21.0%
20.5%
19.9%
18 1..3%
2
4
7
Number of 32B Agent Rollouts (k)
23.3%
24
Z2.5%
Pass@k
SWE-Bench Lite Resolve Rate (%)
21.4%
22
-
Best@k (32B Verifier)
Best@k (7B Verifier)
20.0%
20
18.3%
18
16.4%
16
14.6%
14.7%
15.0%
15.0%
14.0%
13.5%
3.2%
14
11.6%
12
13.1%
13.5%
13.7%
13.7%
13.5%
13.3%
9.4%
.2%
%
10
8
5.4%
1
2
3
4
5
6
7
8
Number of 7B Agent Rollouts (k)
Figure 7: Scaling inference-time compute for MoatlessTools Agents (7B and 32B) with their corresponding learned verifiers. Temperature & = 0.5.
19

---

Training Software Engincering Agents and Verifiers with SWE-Gym
mixture of off-policy & on-policy data Best@k
SWE-Bench Verified Resolve (%)
on-policy data (from fine-tuned model) Best@k
26
off-policy data (from claude & GPT) Best@k
off-policy 2x negative examples (from claude GPT) Best@k
24
22
20
18
1
2
3
4
5
6
7
8
Number of Agent Rollouts (k)
Figure 8: Ablation study for verifier training ($5.1.1). Performances are evaluated on SWE-Bench Verified. Both the agent and the verifier are Owen2. 5-Coder-IN Insterrtttttttt 32B model fine-tuned on the corresponding dataset. OpenHands (Wang et al., 2024c) is used as the agent scaffold.
20

---

Training Software Engineering Agents and Verifiers with SWE-Gym
Different strategies of scaling training trajectories
SWE-Bench Verified Resolved (%)
20
32B No Dedup.
32B Dedup. Sorted by Repo
18
32B Dedup. Sorted by Random
16
14
12
25%
50%
100%
Percentage of Training Data Used
Figure 9: Comparison of three data sampling approaches using 32B LM: scaling trajectories (dedup.). scaling unique task instances, and scaling repositories ($5.2).
21


Reviews:
[{"review_id": "Q0N51KoeWq", "reviewer": "Reviewer_aRo7", "summary": "The paper proposes SWE-Gym, which is a training environment for coding agents tasked to resolve GitHub issues.\nThey provide a collection of 2438 python-based SWE tasks.\nThey used filtered fine-tuning and showed improvement by fine-tuning LLMs in their training environment.\nFinally, to show effectiveness authors compared resolved rates on swebench verified and lite benchmarks.\nThey also trained verifiers which will be also beneficial for training via RL.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer aRo7 for their insightful feedback. Below, we address the primary concerns and suggestions provided.\n\n> I think Novelty is limited given that swe-bench training data already provide training data\n\nTo clarify, SWE-Bench doesn’t include unit tests or executable environments for their training data. Its training data is insufficient for effective agent training because it lacks both executable environments and test case for those instances. Thus, we put in the work to create a suitable training environment. We decided to focus on our own set of repositories, rather than the ones included in SWE-bench, to avoid contamination concerns.\n\nOur work is one of the first to study *training* SWE agents on real-world software engineering tasks, which is a significant departure from existing work that focuses on evaluation or prompting-based agents. We also achieve state-of-the-art open-model results on SWE-Bench, with log-linear training and test-time scaling results.\n\nThese distinctions make SWE-Gym a novel and valuable contribution to the field of software engineering agents.\n\n> Precision and Recall Analysis:\n\nWe appreciate the suggestion to include precision and recall metrics. While our evaluation follows established protocols from prior work (Cobbe et al., 2021), we agree that precision-recall curves would provide more comprehensive insights into our verifiers' performance. We are currently working on these curves and will soon follow up with the updated results.\n\nRegarding intermediate step evaluation, developing effective process rewards for software engineering agents is still an open problem. We would welcome the reviewer's insights on potential approaches to this problem, and would love to incorporate these in the future work.\n\n> Clarification Regarding pass@k: \n\nWe clarify that pass@k (k≥2) solely serves as a reference point to compare our learned verifiers against an oracle verifier that always selects the optimal solution. This metric was not used for our primary results or for comparisons with other methods.\n\n> Is this training data verified or not, i.e., does it contain sufficient information for the issues to be resolved? It may be the case that LLM learns to hallucinate if trained on this since there may be training data which doesn't have sufficient information\n\nOur data is as verified as in the original SWE-Bench paper. As described in Lines 152-186, following SWE-Bench’s task construction pipeline, we only keep around issues that have a gold-standard PR, which indicates that the issue was able to be resolved by a human programmer based on the information provided. Also, as shown in Table 9 in Appendix, claude 3.5 sonnet without any SWE-Gym specific training achieves the reasonable performance of 29.1% on SWE-Gym Lite within 50 turns, suggesting that a significant proportion of the task in SWE-Gym is solvable.\nFurthermore, our ultimate evaluation of our trained model is on SWE-Bench, not SWE-Gym, which is quite out of distribution from SWE-Gym, so any repo-specific hallucination learned through SWE-Gym wouldn't explain our improved SWE-Bench performance.\nWe’d be happy to add follow-up experiments if the reviewer has any thoughts on validating this hypothesis. \n\n> In Table 3, how many runs are used for showing the confidence intervals? Would be good if confidence intervals could be added to other results or comment on them; given the stochasticity of LLMs, its difficult to conclude from a single-digit number.\n\nTo mitigate the stochasticity of LLMs, we apply a consistent random seed and sampling temperature of 0 across all experiments in Table 3. We use a bootstrap test to estimate the standard deviation by sampling 1000 different subsets of the evaluation result.  Regardless of confidence estimation, we believe the performance gap before/after fine-tuning is significant.\nFor experiments with higher stochasticity, we plot error regions in Figures 3 and 4 to represent confidence intervals, as described in lines 357-365. Following Lightman et al. (2023), we estimate the uncertainty as detailed in Appendix B.2.\n\n> In the abstract, can you add the model name and the number of parameters along with improvement numbers?\n\nWe thank the reviewer for the suggestion. We will update the abstract to include the model name and the number of parameters.\n\n> only~2.5K training samples \n\nWe kindly request the reviewer to refer to our first response to Reviewer BE56, where we discuss how our dataset is already effective for model training and achieves state-of-the-art results. Importantly, our experiments in Section 5.2 directly show that the task diversity of SWE-Gym is not a limiting factor in further model improvements.", "claims_and_evidence": "Yeah, claims are generally clear and backed with experiments.", "methods_and_evaluation_criteria": "They use Resolve Rate (%), Empty Patch Rate (%), stuck in the loop rate, pass@k and best@k as criteria for improvement which makes sense. I don't think pass@k with k>=2 is of much use though.\n\nHowever, to better demonstrate the effectiveness, authors can also add precision and recall which can demonstrate the effectiveness of intermediate steps as well. This will also be convincing to demonstrate effectiveness due to the verifier's rewards.", "theoretical_claims": "Experimental paper; No theoretical claims.", "experimental_designs_or_analyses": "Yes.\nTraining with SWE-gym and Scaling agent performance, both seems reasonable.", "relation_to_broader_scientific_literature": "I see two addition as compared to boarded literature i) Addition of verifiers in fine-tuning of agent and ii) adding unit tests for part of training data", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n- The paper is clearly written and easy to parse\n- Enhancing agent's performance via verifier.\n- Scaling experiments\n\nWeakness:\n- I think Novelty is limited given that swe-bench training data already provide training data (if unit tests are created for swe-bench, then I don't see what's the difference)\n- Doesn't include precision and recall which are essential to understand the trajectories.\n- Missing analysis on how untrained LLMs will perform on the training data? Is this training data verified or not, i.e., does it contain sufficient information for the issues to be resolved? It may be the case that LLM learns to hallucinate if trained on this since there may be training data which doesn't have sufficient information\n- only ~2.5K training samples which I think are small for RL-type training", "other_comments_or_suggestions": "In the abstract, can you add the model name and the number of parameters along with improvement numbers?", "questions_for_authors": "In Table 3, how many runs are used for showing the confidence intervals?\n\nWould be good if confidence intervals could be added to other results or comment on them; given the stochasticity of LLMs, its difficult to conclude from a single-digit number.\n\n###Update after rebuttal:\n\nAll the weaknesses are still intact and the author's response does not address them constructively.\n- Especially, they agree that the difference to the SWE-bench is limited as was pointed out in the weakness initially that if unit tests are included in the swe-bench then there is no difference. \n- I think the response to stochasticity (or confidence interval) should be \"evidence with data\" as compared to saying \"We believe the gap is already enough\". I would appreciate if the authors were scientifically rigorous.\n- Also, authors agree that they need significant work, especially in coming up with metrics like precision/recall in their data set before it is useful (or better than existing benchmarks).\n\nIn response to this, I suggest rejecting the paper. It's also beneficial for authors to submit a complete and usable work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jePls7sjLi", "reviewer": "Reviewer_Be56", "summary": "This paper introduces SWE-Gym, the environment for training software engineering (SWE) agents. SWE-Gym contains 2,438 real-world Python tasks from 11 popular GitHub repositories, each equipped with pre-installed dependencies, executable runtime environments, unit tests, and natural language task descriptions. The authors demonstrate SWE-Gym's effectiveness by using it to train language model-based agents through rejection sampling fine-tuning, achieving significant improvements in resolve rate on the SWE-Bench Verified and Lite test sets. The paper also explores inference-time scaling through verifiers trained on agent trajectories sampled from SWE-Gym, showing that when combined with fine-tuned SWE agents, they achieve state-of-the-art performance of 32.0% and 26.0% on SWE-Bench Verified and Lite respectively. The authors publicly release SWE-Gym, the trained models, and agent trajectories to facilitate further research.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">  computational budget constraints limited the number of training trajectories to 491, which may affect the generalizability of some findings.\n\nWe would like to emphasize that our results represent the state-of-the-art open-model results at the time of submission, and used a substantial compute budget exceeding $30K. The challenge of collecting more training trajectories stems from the inherent computational intensity of agent training research, particularly when working with real-world software tasks that require full execution environments.\n\nImportantly, our scaling experiments in Sections 5.1 and 5.2 demonstrate consistent log-linear scaling behavior, which provides strong evidence for the generalizability of our findings beyond the current dataset size. Our experiments in Section 5.2 suggest that the task diversity of SWE-Gym is not a limiting factor in further model improvements.We believe these results offer valuable insights that will motivate and guide future open research in this direction.\n\nAdditionally, we note that these 491 trajectories are positive ones used for agent training. In fact, we used over 1000 trajectories (both positive and negative) for training the verifiers.\n\n> Limited exploration of alternative fine-tuning methods\n\nWe appreciate the reviewer's insightful suggestion regarding alternative fine-tuning methods. We would like to clarify that the primary contribution of our work is the SWE-Gym dataset itself, which addresses a critical gap in the field by providing executable environments for real-world software engineering tasks. While we demonstrate the dataset's effectiveness through recently-proposed approaches for model improvement via agent training and test-time scaling, these implementations serve as solid baselines rather than exhaustive explorations of optimal training techniques.\n\nAs one of the first papers to study SWE agent training on real-world tasks, we deliberately established strong baseline models using well-understood training approaches (Zelikman 2022, Singh 2023, Pan 2024) to provide clear evidence of SWE-Gym's effectiveness. Our results show substantial performance improvements (up to 19% absolute gains in resolve rate) using these straightforward methods, which we believe validates SWE-Gym's value as a training resource.\n\nWe agree that exploring more sophisticated training paradigms represents a promising direction for future research. By releasing our dataset, trained models, and agent trajectories publicly, we aim to facilitate such explorations by the broader research community.\n\n> How SWE-Gym could be extended to other programming languages beyond Python in future work.\n\nWe thank the reviewer for this valuable suggestion. In our future work section, we will add a comprehensive discussion on ways to extend SWE-Gym beyond Python. We envision two promising approaches: (1) establishing a collaborative community effort to systematically collect and curate datasets across multiple programming languages; or (2) developing specialized environment-setup language model agents that can automatically analyze repositories, identify dependencies, and construct executable environments for diverse programming languages.\n\n> I see two addition as compared to boarded literature i) Addition of verifiers in fine-tuning of agent and ii) adding unit tests for part of training data\n\nWe thank the reviewer for pointing out these contributions. We will update the related works section to include a more detailed comparison of our results with these literature.", "claims_and_evidence": "The claims made in the paper are generally well-supported by evidence.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate and well-designed for the problem:\n\n- Dataset construction: The authors detail a rigorous process for creating SWE-Gym, including repository selection criteria, versioning, and environment setup. The validation of instances using execution-based verification ensures high-quality training data.\n\n- Evaluation metrics: The use of standard SWE-Bench metrics (resolve rate, empty patch rate) provides consistency with prior work. Additional metrics like \"stuck in loop\" percentage offer valuable insights into agent behavior improvements.", "theoretical_claims": "The paper does not make formal theoretical claims requiring proof verification. The claims are empirical in nature, focusing on the effectiveness of SWE-Gym for training agents and verifiers.", "experimental_designs_or_analyses": "- Agent training experiments: The authors clearly specify model sizes, training procedures, and hyperparameters. The comparison between different agent scaffolds (OpenHands vs. MoatlessTools) provides valuable insights into the effectiveness of SWE-Gym across different agent architectures.\n\n- Verifier experiments: The authors explore different training data compositions for verifiers, showing how mixing on-policy and off-policy trajectories affects performance.\n\n- Scaling experiments: The three different scaling approaches (trajectory, instance, and repository scaling) are well-designed to isolate the impact of different aspects of training data.\n\n- Statistical rigor: The authors report standard deviations for key metrics (Table 3), enabling assessment of result reliability.\n\n- **One minor concern** is that the computational budget constraints limited the number of training trajectories to 491, which may affect the generalizability of some findings.", "relation_to_broader_scientific_literature": "The authors appropriately cite relevant prior work and clearly articulate how SWE-Gym addresses a critical gap in the literature.\n- SWE agent development: The authors contextualize their work within recent advances in SWE agents, highlighting the limitations of current approaches due to lack of suitable training environments.\n\n- Agent scaffolds: The paper discusses different agent design philosophies (specialized workflows vs. general-purpose prompting) and evaluates SWE-Gym's effectiveness across both paradigms.\n\n- Post-training methods: The authors connect their work to broader trends in LLM fine-tuning techniques, including trajectory filtering approaches.\n\n- Verifier models: The paper builds on outcome-supervised reward modeling and applies it to the software engineering domain.", "essential_references_not_discussed": "The paper covers most essential references in the field.", "other_strengths_and_weaknesses": "Strengths:\n- Practical contribution: SWE-Gym addresses a critical need in the field by providing a standardized, reproducible environment for training SWE agents.\n\n- Comprehensive experimentation: The paper explores multiple dimensions (model size, agent scaffold, training data composition).\n\n- Scaling analysis: The clear demonstration of scaling behaviors with both training data and inference compute.\n\nWeaknesses:\n\n- Limited exploration of alternative fine-tuning methods: While rejection sampling is effective, comparing with other approaches like PPO or DPO would strengthen the paper.\n\n- Computational constraints: The limited number of training trajectories (491) may not fully reveal the potential of SWE-Gym, though the authors acknowledge this limitation.\n\n- Task diversity analysis: While the paper mentions task distribution across repositories (Fig. 2), deeper analysis of how different task types benefit from training could provide additional insights.", "other_comments_or_suggestions": "Consider expanding the discussion on how SWE-Gym could be extended to other programming languages beyond Python in future work.", "questions_for_authors": "See \"Other Strengths And Weaknesses\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gq7V4mKTmi", "reviewer": "Reviewer_LJa6", "summary": "The paper introduces SWE-Gym, a novel training environment specifically designed for developing software engineering agents. The environment comprises 2,438 real-world Python task instances extracted from GitHub issues; each instance includes a codebase, an executable runtime environment with pre-installed dependencies, and a set of unit tests for verification. The authors leverage SWE-Gym to train language model (LM) based agents and verifiers, demonstrating significant improvements in task resolution rates on standard benchmarks such as SWE-Bench Lite and SWE-Bench Verified. Key contributions include:\n\n• A large-scale, realistic dataset that bridges the gap between existing synthetic or limited real-world benchmarks, enabling end-to-end training of agents that can handle complex repository-level software tasks.\n\n• The development and evaluation of two agent scaffolds—one based on general-purpose prompting (OpenHands) and another on a specialized workflow (MoatlessTools). Using these, the authors fine-tune LM agents via rejection sampling fine-tuning, which leads to substantial improvements (up to 19% absolute gains in resolve rate).\n\n• A novel approach to inference-time scaling where a verifier model is trained to estimate the success probability of candidate agent trajectories. By sampling multiple solutions and selecting the best one according to the verifier, they further boost performance to new state-of-the-art levels (achieving 32.0% and 26.0% on SWE-Bench Verified and Lite, respectively).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Could you clarify why a larger dataset (SWE-Gym) with executable environments is necessary, given that SWE-Bench already offers multiple, human-verified versions?\n\nWe’d like to clarify that SWE-Bench doesn’t include executable environments or unit tests for its training split. This makes it impossible to use for learning algorithms that train models through real-world action execution and observation. And training on the test split will introduce data \ncontamination and invalidate our results. \n\nSWE-Gym addresses this fundamental gap by providing a separate, complementary dataset with fully executable environments that allows us to train agents on real-world software engineering tasks while maintaining the integrity of SWE-Bench as a clean evaluation benchmark. This separation is crucial for accurately measuring progress of software engineering agents.\n\n> Can you provide comparisons against existing methods evaluated on SWE-Bench to isolate the unique contributions of your training approach?\n\nBecause our training approaches are focused on training models in real-world software engineering tasks, this requires execution environments and test cases for training task instances. SWE-Bench only includes environments and test cases for its test set, not its training set. Thus, we did not conduct these ablation studies as training on any subset of SWE-Bench’s test set would be problematic.\n\n\n\n> How do you justify the chosen evaluation criteria in light of the human verification already present in SWE-Bench, and what additional benefits does SWE-Gym provide?\n\nWe want to clarify that throughout our paper, we exclusively use SWE-Bench (both Verified and Lite versions) as our evaluation framework. This choice is deliberate as SWE-Bench provides human-verified test cases that serve as a reliable, standardized benchmark for measuring agent performance. SWE-Gym complements this by uniquely providing an effective training dataset with executable environments that enable end-to-end agent training without contaminating our evaluation data. \n\n\n\n> Essential References Not Discussed and Comparisons against existing methods evaluated on SWE-Bench \nWe appreciate the reviewer pointing out these references. We would like to clarify that we have discussed both references in our paper: SWE-agent [1] in line 102 and Agentless [2] in line 093.\n\nHowever, we acknowledge that our comparison with these works could be more comprehensive. In the next version of our paper, we will expand our analysis to include a more detailed comparison of our results with these frameworks. Essentially, although these approaches demonstrate that better prompts and agent scaffolds can enhance performance, our work shows that horizontally, end-to-end training—without relying on manual prompt design—can yield even greater improvements. \nWe will also present their performance on SWE-Bench to better contextualize our contributions. \n\nAdditionally, for concurrent works on SWE Agent training, we include a detailed comparison in appendix section A.", "claims_and_evidence": "While many experimental claims, such as improved agent performance through fine-tuning and inference-time scaling, are well-supported, the submission’s claim that SWE-Gym is a significant contribution primarily due to its larger scale compared to SWE-Bench is not clearly justified. The paper does not provide sufficient evidence or ablation studies to demonstrate that a larger dataset—and the inclusion of executable environments—is necessary or offers unique benefits over existing datasets, leaving that claim problematic.", "methods_and_evaluation_criteria": "The choice of SWE-Gym over SWE-Bench as a primary training dataset is not convincingly justified. SWE-Bench already includes multiple verified versions and has undergone human validation, ensuring high-quality benchmarks for evaluating agent performance.", "theoretical_claims": "The paper do not have theoretical claims.", "experimental_designs_or_analyses": "The evaluation provides evidence that the training method is effective to some extent, yet there are experimental design concerns. Specifically, while the results suggest improvements using SWE-Gym, the study lacks direct comparisons with existing methods evaluated on SWE-Bench and does not explore the impact of training on subsets of SWE-Bench itself. This omission makes it difficult to isolate the unique contribution of SWE-Gym and assess whether similar gains could be achieved using parts of the established benchmark dataset.", "relation_to_broader_scientific_literature": "The paper is realted SWE-bench, SWE-agent and so on.", "essential_references_not_discussed": "The paper do not discuss related works which also evaluated on swe-bench:\n\n[1] Yang J, Jimenez C E, Wettig A, et al. Swe-agent: Agent-computer interfaces enable automated software engineering[C]//The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024.\n\n[2] Xia C S, Deng Y, Dunn S, et al. Agentless: Demystifying llm-based software engineering agents[J]. arXiv preprint arXiv:2407.01489, 2024.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Could you clarify why a larger dataset (SWE-Gym) with executable environments is necessary, given that SWE-Bench already offers multiple, human-verified versions?\n2. Have you conducted ablation studies or direct comparisons to evaluate the benefit of training on SWE-Gym versus using subsets of SWE-Bench?\n3. Can you provide comparisons against existing methods evaluated on SWE-Bench to isolate the unique contributions of your training approach?\n4. How do you justify the chosen evaluation criteria in light of the human verification already present in SWE-Bench, and what additional benefits does SWE-Gym provide?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces SWE-Gym, a novel environment featuring executable setups and unit tests for training software engineering agents on real-world Python tasks. The authors demonstrate its utility by fine-tuning language model agents and training verifiers, achieving state-of-the-art results for open-weight models on the established SWE-Bench benchmark. Strengths highlighted by reviewers include the practical contribution of a much-needed training resource (Be56), comprehensive experiments with scaling analysis (Be56), and clear writing (aRo7). Key concerns revolved around the perceived novelty and necessity compared to SWE-Bench (aRo7, LJa6), the dataset size (aRo7, Be56), and the handling of evaluation metrics and stochasticity (aRo7).

Addressing the primary concerns, the authors effectively clarified that the SWE-Bench training split lacks the executable environments and tests required for agent training via environment interaction, making SWE-Gym a necessary contribution distinct from the SWE-Bench evaluation set. Using the SWE-Bench test set for training would constitute data contamination. While the dataset size and number of training trajectories collected are limited by significant computational costs, the authors demonstrate strong empirical results, including state-of-the-art performance and positive scaling trends, validating the environment's effectiveness with the current data (Be56). Concerns about specific metrics like precision/recall (aRo7) or alternative fine-tuning methods (Be56) point towards valuable future work but do not invalidate the current contributions evaluated using standard metrics.

Overall, SWE-Gym represents a significant step forward by providing the first large-scale, executable environment for training agents on realistic software engineering tasks, complementing existing evaluation benchmarks like SWE-Bench. The demonstrated state-of-the-art results achieved through training within SWE-Gym underscore its value. Despite limitations inherent in costly agent-environment interaction research, such as dataset scale, the work presents a solid contribution, releasing valuable resources (environment, models, trajectories) to the community. The critiques regarding novelty relative to SWE-Bench appear based on a misunderstanding of benchmark limitations, and other concerns represent avenues for future exploration rather than fundamental flaws. The paper meets the bar for acceptance at ICML.
Decision: Accept (poster)