R2E-Gym: Procedural Environment Generation and Hybrid Verifiers for Scaling Open-Weights SWE Agents
Authors: Naman Jain
Jaskirat Singh
Manish Shetty
Tianjun Zhang
Liang Zheng
Koushik Sen
Ion Stoica
Venue: COLM 2025 (2025)
Keywords: SWE Agents ; Inference Time Scaling ; Test Generation
TL;DR: R2E-Gym: Procedural Environment Generation for Scaling Open-Weights Software Engineering Agents

Abstract: Improving open-source models on real-world SWE tasks (solving GITHUB issues) faces two key challenges: 1) scalable curation of execution environments to train these models, and, 2) optimal scaling of test-time compute. We introduce R2EGym, the largest procedurally-curated executable gym environment for training real-world SWE-agents, consisting of more than 8.1K tasks. R2EGym is powered by two main contributions: 1) SWEGEN: a synthetic data curation recipe that enables scalable curation of executable environments using test-generation and back-translation directly from commits, thereby reducing reliance on human-written issues or unit tests. We show that this enables more scalable training leading to pass@1 performance of 34.4% on SWE-Bench Verified benchmark with our 32B model. 2) Hybrid Test-time Scaling: we provide an in-depth analysis of two test-time scaling axes; execution-based and execution-free verifiers, demonstrating that they exhibit complementary strengths and limitations. Test-based verifiers suffer from low distinguishability, while execution-free verifiers are biased and often rely on stylistic features. Surprisingly, we find that while each approach individually saturates around 42-43%, significantly higher gains can be obtained by leveraging their complementary strengths. Overall, our approach achieves 51% on the SWE-Bench Verified benchmark, reflecting a new state-of-the-art for open-weight SWE-agents and for the first time showing competitive performance with proprietary models such as o1, o1-preview and sonnet-3.5-v2 (with tools). We will open-source our environments, models, and agent trajectories.

Published as a conference paper at COLM 2025
R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents
Naman Jain'* Liang Zheng
Jaskirat Singh**
Manish Shetty
Tianjun Zhang
Koushik Sen'
Ion Stoica
ICC Berkeley
Australian National University
(namanjjainbberklleyy.ek)
jaskirat. siw.retu.iiingeeeueedu.u. au)
Abstract
Improving open-source models on real-world SWE tasks (solving GITHUB issues) faces two key challenges: 1) scalable curation of execution environ- ments to train these models, and 2) optimal scaling of test-time compute. We introduce R2B-Gym, the largest procedurally-curater executable gym environment for training real-world SWE-agents, consisting of more than 8.1K tasks. R2E-Gym powered by two main contributions: 1) SWEGEN: a synthetic data curation recipe that enables scalable curation of executable environments using test-generation and fack-translation directly from com- mits, thereby reducing reliance on human-writler issues or unit tests. We show that this enables more scalable training leading to PASS@1 of 34.4% on SWEBENCH-V VERIFID benchmark with our 32B model. 2) Hybrid Test- time Scaling: we next provide an in-depth analysis two test-time scaling axes; execution-based and execution-free verifiers, demonstrating that they exhibit complementary strengths and limitations. Test-based verifiers suf- fer from low distinguishability, while execution-free verifiers are biased and often rely on stylistic features. Surprisingly, we find that while each approach individually saturates around 42-43%, significantly higher gains can be obtained by leveraging their complementary strengths. Overall, our approach achieves 51% on the SWEBENCH- BEICc---VeRREE VERIFIED benchmark, reflecting a new state-of-the-a for open-weight SWE agents and for first time being competitive with proprietary systems such as o1 or sonnet w/ tools.
1 Introduction
Autonomous software engineering (SWE), aiming to solve real-world software engineering problems such as GITHUB issues, has made significant progress in recent times (Wang et al., 2024; Yang et al., 2024b). While LLM-based SWE-Agents have demonstrated remarkable improvements, state-of-the-art performance is largely driven by proprietary models (An- thropic, 2025; Jaech et al., 2024) with open-models lagging behind (Xie et al., 2025).
Addressing this gap requires solving two fundamental challenges: First, scalable curation of high-quality execution environments train these models; and second, developing efficient aggregation strategies to maximize test-time performance. While several bench- marks for evaluating SWF-agents on GITHUB issues exist (Jimenez et al., 2023; Zhao al., 2024), scalable curation of high-quality training environments remains a challenging prob- lem. For instance, while the training split from SWE-Bench (Jimenez et al., 2023) contains output patches, it lacks executable environments. Pan et al. (2024) collect executable test environments, but rely on human-written issues and test cases restricting sample-size.
In this paper, we introduce R2E-GyM, the largest procedurally curated environment for training real-world SWE-agents - consisting of more than 8.1K problems, with executable
Equal Contribution.
Project Page: https: httass:///eggui/ gym. thub.ib.

---

Published as a conference paper at COLM 2025
SWE-Bench Verified % (Bestgg
Itumen File
SWOGEN
Exsecttion Free
Passgh1
Execution Hybrdd Scaling
Somnes-3.5 OpenHands
53.0%
Commorced
Soncer-3 Tools
45%
Executable Training
Environments
SWE: EVEE-Benc Verttied (%6
+8.2%
+7.3%
01
48%
2,5X
+16 16.66%
O1 preview
41.3%
Ours
51.0%
51.0%
Openn
SweRL
41%
42.8%
43.7%
SweGymm
32%
34.4%
SwePixer
30.2%
(a) Synthetic Data
(b) Hybrid Test-time Scaling (c) Open-weights SOTA Performance
Figure 1: Overview. In this paper, we introduce R2B-Gym, the largest gym environment and training framework for training open-weight SWE agents. R2E-Gym is powered by two main contributions: (a) SWEGEN: a synthetic data curation recipe for curating executable training environments w/o relying on human tests and issues ($2). (b) Hybrid Inference Time Scaling: showing that while both execution-basee and execution-fres verifiers elicit inference-time gains; significantly better performance can be achieved by leveraging the strengths of both ($4). (c) Overall, the final approach reflects SOTA performance for open- weight SWE-agents, while also being competitive with some proprietary model baselines'.
gym environments, unit tests, and natural-language task descriptions ($2). R2E-GYM addresses both key challenges through two primary contributions (Figures la and 1b):
Synthetic Data Enables More Scalable Training. We propose SWEGEN novel synthetic data curation recipe that enables collection of a large number of executable training envi- ronments without reliance on human-written pull requests (PRs) or unit tests. We show that instead of using human-written PRs, good-quality execution environments can directly be curated from commits through backtranslation (Li et al., 2023; Wei et al., 2023) and test collection or generation ($2). Compared to PR-based data collection (Pan et al., 2024), this approach enables more scalable data curation (Figure 1a) and agent-training, resulting PASS@1 performance of 34.4% on the challenging SWEBENCH-VERIIE benchmark.
Hybrid Inference Time Scaling. We next leverage R2E-GyM to investigate two comple- mentary axes for scaling test-time compute (54): 1) Execution-based verifiers that evaluate patches through test cases (Xia et al., 2024b), and 2) Execution-free verifiers that assess trajectories through learned models (Pan et al., 2024). While prior works have studied these approaches in isolation, they lack a comprehensive analysis their relative strengths and weaknesses. We first present unique and in-depth analysis of their working mech- anisms, demonstrating that execution-free and execution-based methods actually exhibit complementary strengths and weaknesses. We find two key insights (studied in $4.2): a) Execution-based methods provide direct signals for patch correctness but struggle with dis- criminating between solutions, and b) Execution-free verifiers provide better discrimination but can be biased by other heuristics (e.g., agent thoughts) over the final patch. Based on the above insights, we propose a hybrid scaling approach leveraging the strengths of both methods. Surprisingly, while the performance of both execution-based and execution methods plateaus around 42-43%, , the hybrid approach yields significantly higher gains, achieving final performance of 51% SWEBENCH- VERIFIED (Figure 1b and $4.3).
The key contributions of this paper are: 1) We introduce R2E-GYM, the largest procedu- rally curated environment for training real-world SWE-agents, increasing the number of executable environments by over 3 times. 2) We provide an in-depth analysis demonstrating that execution-based and execution-free axes for scaling test-time compute exhibit comple- mentary strengths and weaknesses. 3) Based on the above insights, we propose a hybrid scaling approach that leverages the strengths of both methods, significantly improving test-time performance. 4) Finally, we release an open-weights 32B model that achieves 51% on SWEBENCH- VERIFIED, reflecting new state-of-the-art for open-weight SWE-agents, while also for the first time demonstrating competitive or better performance compared to commercial models (Fig. 1c), e.g. ol (Jaech al., 2024) and sonnet-3.5-v2 (Anthropic, 2024).
Results with all wpen-weight models are reported with test-time scaling.
2

---

Published as a conference paper at COLM 2025
2 R2E-GYM: Procedural Synthetic Data Generation
SCT scraby pereenid
Dataset (split)
Repu?
Executable?
# Instances
piloo
APPS (Hendryeks et al, 2021)
10,000
temado
13.5%
R2E (Jain et al,, 2024b)
246
alohttp
Benchitrain) (limenez et 2023)
19,008
SWE-Gym Raw (Pan et al,, 2024)
x
66,894
coveragepy datalad
SWP- ByF-BBecch (tesi) (Jimenes et al,, 2023)
2,294
R2E Subset
SWE Cym (Pan et al, 2024)
2,438
(4578)
R2E-Cymn Subset (Ours) R2E-Gym
3
3
4,578
31.5%
pandas
87.3%
numpy
Table 1: Dataset Statistics. Comparing statistics across different datasets curating executable train- ing environments for SWE-agent training. R2F- Gym refers to our full dataset, and R2E-Gym- Subset refers to filtered subset of tasks, with non- overlapping repositories with SWE-Bench.
Table 2: Repo distribution for R2E- Gym subset (no overlap with SWE- Bench) used for training (refer $3).
Overview. SWE task collection methods (Jimenez et al., 2023) rely on human-written issues and unit tests for problem statements and evaluation functions. However, this presents challenge for scaling data curation as size is limited by limitation, we propose SWEGEN -a synthetic data curation recipe using backtranslation and test generation. We procedurally generate environments using only commits from GITHUB repositories, reducing reliance on both human-written issues and test cases.
PRs. To overcome this
Repository and Commit Curation. We use SEART GITHUB search² to identify PYTHON repositories with large number of commits. Next, we extract commit history and associated code changes for each repository. We filter relevant commits using a combination of rule- based and LL.M-bssed heuristics, identifying interesting code changes. For each relevant commit, we next collect build scripts by semi-manually searching across dependency pins. We expand our set of heuristics and installation procedure further in the Appendix A.
Test-Valides Validation and Generation for Environment Collection. Following Jimenez et al. (2023), we use the existing test cases in the curated commits to identify Fail-PPss (F2P) test cases, iee test cases that fail in the original buggy commit and pass in the fixed commit. In cases where the curated commits do not have associated tests, limiting the ability to use them for training environments, we supplement such commits with automatically generated Fail-Passs test-cases. Appendix A expands our test generation approach.
Non-reliance GITHUB Issues. Using the above steps, collect large number of commits, associated build environments and F2P (Fail->Pass) test cases. Now, we need to collect the problem statements associated with the commits. Prior works (Jimenez et al., 2023; Pan et al., 2024) use human-written GITHUB issues as problem statements. This inevitably cannot use the entire commit history since human-written issues are not available for all commits. Here, following Li et al. (2023); Wei et al. (2023) we propose backtranslation approach to collect the problem statements associated with the commits.
Backtranslation:
However, naively kack-translating code changes is quite noisy as models often generate generic problem statements that do not capture the essence of the code changes. Instead, we identify that human-writter issues often contain failing tests and execution traces as part of bug reports. We use this observation collect high-quality problem statements by using the F2P test-cases as part of the backtranslation prompt. Similar to existing works (Jain et al., 2024b; Zhuo et al., 2024), we find that using test execution information allows generating precise and directed problem statements. Please find prompts and examples in Appendix.
We collect over 8.1K problem statements using this approach (referred to as R2E-Gym). We decontaminate this set by removing repositories overlapping with SWE-Bench test-set repositories, obtaining 4578 problems (referred to as R2E-Gym-Subsetl and use that across all experiments unless specified otherwise. Table shows the statistics of different datasets, and Figure and Figure 9 show the distribution of the repositories in R2F-Gym-Suun Gyys-Su and
2nttps://seare us/.cc//
3

---

Published as a conference paper at COLM 2025
Table 3: Resolve Rate (%) Comparison on SWEDENCH-VERIFIED and SWEBENCH-LiTE bench- marks. We observe that synthetic data curation (SwEGEN): allows our approach to scale better across different model sizes, All experiments use the Qwen- 2. 5-Coder as base-models.
Model Size
SWEBENCH-LITE
SWEBENCH-VVRIPEED SWE-Gym
Base-model
SWE-Gym
Ours
A
Base-model
Ours
A
7B
1.0 (+1.0)
10.0 (+2.4) 11.0 (+0.8)
+1.0
1.8 (+1.3)
10.6 (+2.1) 19.0 (+1.0)
+8.4
14B
2.7 (+1.9)
12.7 (+2.3)
20.67 (+0.7)
+7.97
4.0 (+1.6)
16.4 (+2.0) 26.8 (+1.4)
+10.4
32B
(#1.4)
15.3 (-5)) 23.77 (=0.8)
+8.47
7.0 (#1.3)
20.6 (+2.1) 344 (11.2) +13.8
R2E-Gym respectively. Notably, using our SWEGEN approach, we can collect over 2.5 times more problems than relying on the data collection relying on GITHUB issues (Figure la).
3 Training SWE-Agents using R2E-GyM Environments
Agent Scaffolding. We design a minimal scaffold on top of OPENHANDS (Wang et al., 2024) e
to experiment with agents for diverse SWE tasks. It uses a traditional REACT framework (Yao et ata without any specialized workflow; equipping the LLM with only a bash terminal, file editor, and search tool. Figure 16 depicts an example code editing trajectory.
Trajectory Collection and SFT Training. We next collect SFT trajectories using from R2E- Gym environments. To avoid contamination, we only use a subset of R2E-Gym consisting of repos with no overlap with the SWE-Bench dataset. The resulting subset (R2E-Gym- Subset) consists of 4578 executable environments across 10 repositories (Figure 2). For each task environment, we use SONNET-3.5-V2 with our agent scaffold and collect the successful agent trajectories. Through this process, we collect 3321 trajectories from 2048 unique task environments. We then use these trajectories to train our agent via supervised fine-tuning on agent thoughts and actions. For training, we use LLaMA-Factors and Qwen-2.5-Coder models (7B, 14B, 32B) our base models. For detailed experiment configuration and hyperparameters, please refer to Appendix B.
(Zheng et al., 2024)
3.1 Results and Analysis
Comparison to weenht SWE-Agents across Model Scales. We report PASS@1 of R2F- Gym trained models on the SWEBINCH- VERIFIED and SWEBENCH-L.VEE benchmarks Table 3. We also report comparisons with recently proposed SWE-Gym (Pan et al., 2024), which is most closest to our work. As seen in Table 3, we find that our approach enables better scaling for training SWE-agents across all model sizes. For instance, on SWEBENCH-VERIIG VERIFFED, for the same base-model type and scale, our 32B model significantly improves the PASS@1 performance by 14%; pushing the final performance from 20.6 (SWE-Gym) to 34.4%.
Scaling with Number of Trajectories. We investigate the relationship between training samplesize (number of trajectories) and agent performance in Figure 2. We evaluate 14B and 32B models trained with trajectory counts ranging from 100 to 200.. Our findings indicate that performance improves with increasing trajectory count, though with diminishing returns for both models. Notably, the 14B model begins to saturate at approximately 800 samples, while the 32B model still shows improvements, likely due its larger capacity. These results extend the findings of Pan et al. (2024), who studied dataset scaling up to ~ 500 samples. Our analysis demonstrates that while performance does improve with increasing samplesize, the rate of improvement diminishes or even plateaus for smaller models.
Real vs Synthetic Problem Statements. The R2F-Gym approach enables us generate problem statements without relying on human-written descriptions and test cases, offering greater scalability. We compare the performance of models trained on real GitHub issues versus our synthetic problem statements (collecting 400 trajectories from both sets). Remark- ably, models trained on synthetic data achieve nearly identical performance (27.8% PAsS@1) to those trained on real data (28.0%). This finding validates the efficacy of our synthetic data generation methodology, demonstrating that procedurally generated environments can match the training value of real-world examples while providing scalability.
4

---

Published as a conference paper at COLM 2025
Model Performance vs. Data Size
Ablation
Config
PASS@1 (%)
35.0
With
34.4
Adding Thoughts Without
30.4
30.0
27.5
Real
28.0
Real vs. Synthetic Synthetic 27.8
22.5
20.0
148 Model
Mccea
Figure 3: Top. Using thoughts in REACT agent trajectories leads to significant per- formance improvements. Bottom. Using SWEGEN synthetic generated issues and test cases achieves similar performance as real-world issues (400 trajectories for both real & synthetic in above) while providing better scalability during data collection.
326 Model
15.0
100
200
400
800
1600
3200
Data Size
Figure PASS@1 scaling curve with in- creasing number of training samples. Per- formance improvement with more training samples, enabled by SWEGEN approach.
Explicit Thought Traces are Important. During SFT we use both the agent's thought processes and actions training targets. Models trained with thought demonstrations achieve significantly better performance compared to those trained without (34.2% vs 30.4% in Table 3). This suggests that exposing the model step-by-step reasoning processes is necessary for reliable problem-solving in complex environments.
4 Efficient Inference Time Scaling With Hybrid Verifiers
We utilize R2E-Gym (62) for inference-time scaling experiments with coding agents. In $4.1, we explore different axes for scaling test-time compute, focusing on two distinct approaches: 1) Execution-based Verifiers and 2) Execution-free Verifiers. We analyze the relative strengths and weaknesses of each approach, demonstrating their complementary nature (64.2). Based on this insight, we propose hybrid approach that leverages the strengths of both, significantly improving test-time performance (54.3). Finally, we provide detailed ablations and analysis, examining critical design choices for our approach (54.4).
4.1 Exploring Different Axes for Training Verifiers
Given an input task description D, a set of agent trajectories (Ti) and candidate patch 1 to rank
outputs {Pi)k Piji-11 1. our objective is to build a verifier that assigns scores S = {
the outputs. To this end, we investigate two types of verifiers:
Verifiers. We train a specialized esstng-agent that generates reproduction test cases to determine whether candidate patch resolves the issue (i.e., whether the patch passes the generated test suite). Additionally, following Xia et al. (2024b), we leverage existing regression tests to filter out patches that fail to maintain backward compatibility. Our execution-based (EB) verifier thus comprises two components: 1) festing-agent that generates targeted tests evaluate bug fixes, and regression test filter that eliminates patches that compromise existing functionality. Specifically, we train the aeeting-agent (using OWEN-CODER-33 as base-model) to generate a comprehensive test script containing M = 10 diverse tests that cover various inputs, corner cases, etc.. See Appendix D for example generated tests. The execution-based score sEB for each each patch Ps is then computed as,
Execution-
TestScores, if RS; = max RSj, jF[L.X]
EB %t
where TestScore; Σ Pass(PPr Test:) (1)
0,
otherwise,
where RS₁ refers to the regression test score for the th patch and helps selelct the patches with the highest regression test scores (Xia et al., 2024b). TestScore, is simply the sum of the number of passing tests for each patch Px- Please refer to Appendix SC for further details.
Notably, unlike zero-shot test generation with Agentless (Xia et al., 2024b), our testing agent interacts with the environment to examine existing test cases and generates new

---

Published as a conference paper at COLM 2025
Best@K with Editing Agent Rollouts
Best@K with Test Agent Rollouts
52
50
20
Best@k
45
41.6
42.0
42.7
43.7
Best@x
220
42.6
80
Editing Agent Rollouts
Veriflers Hybrid
#11 Agent Rollouts
#16 Agent Rallouts
Execut on Based Ventier
Agent Rtalouts
320
Execution Free Verfler
#20 Agent Rollouts
# Agent Rollouts (K)
# Test Agent Rollouts (K)
Figure 4: Left. BEST@K with increasing number of aditing-agent rollouts. Inference-tine scaling improves final performance for both execution-base and execution verifiers. Hybrid Verifier combining execution-based and execution-free verifiers provides signifi- cantly superious scaling. Right. BEST@K with increasing number of testing-agen -ggett rollouts. Increasing test-agent rollouts also improves final performance and can provide more com- pute efficient scaling than naively increasing only -aget rollouts.
Table 4: Performance of various models models/mettine methods on SWE-Bench Verified.
Method
Model
Type
Verified
Proprietary Models
Agentless-1.5 1.5 (Xia et al,, 2024b)
GPT-4o
Pipeline
34.0
Agentless (Xia et al,, 2024b)
01
Pipeline
48.0
Claude Tools
Claude-3.6-Sonné
Agent
49.0
Agentless-1.5 (Xia et al., 2024b)
Claude-3.6-Soma
Pipeline
30.8
OpenHands (Wang et al., 2024)
Claude-3.6-Somee
Agent
53.0
Claude Tools
Claude-3.7-Somie
Agent
62.3
Claude Tools (Best@Any)
Claude-3.7-Sonne
Agent
70.3
Open-source Models
SWE-SynInfer (Ma el al,, 2024)
Lingma-SWE-GP1-728
Agent
30.2
SWE-Fixer (Xie et al,, 2025)
SWE-Fixer-72B
Pipeline
30.2
SWF-Gym (BEsT@16 w/ Verifier) (Pan al., 2024)
SWF-Gym-328
Agent
32.0
SWE-RL (BeSt0500 w/ Tests) (Wei et al., 2025)
SWE-RL-70B
Pipeline
41.0
Agentless (Xia al,, 2024b)
DeepSeek-RI
Pipeline
49.2
R2E-Gym (Ours) (PAss@1)
R2F-Gym-32B
Agent
34.4
R2E-Gym (Ours) (BEsT@16 w Hybrid)
R2E-Gym-32B
Agent
49.4
R2E-Gym (Ours) (Bust@26 w Hybrid)
R2E-Cym-32B
Agent
51.0
tests informed by these examples with execution feedback. We demonstrate that this environment-aware approach provides additional benefits over zero-shot methods in $4.4.
Execution-free Verifiers. We next train execution-free (EF) verifiers for selecting the best trajectory from set of sampled trajectories from the code-editing agent (53). In particular, following (Pan et al., 2024), given task description D, agent-trajectory T (sequence of thought, action, and observations) and output patch P, we finetune Qwen2. 5- model to predict YES and NO tokens to determine correctness of a trajectory using SFT on correct and incorrect trajectories. The execution-fres score is then computed by normalizing the relative probability of YES token as sLF P(YES)/(P(YEES) + P(NO)), where P(YES) and P(NO) are estimated through log-probabilities of corresponding token predictions.
4.2 Comparative Analysis of Execution-Based and Execution-Free Verifiers
Experimental Methodology. We evaluate verifier performance using the BEST@K metric, which quantifies each verifier's ability identify correct patches from multiple candidates. Specifically, given K trajectories, the BEST@K metric represents the percentage of problems where the verifier successfully selects the correct patch using its scoring mechanism. For our experiments, sample trajectory at temperature T 0 and 25 trajectories at temperatures = 0.8 and T = 0.9 from the R2E-Gym-320 model on SWEBENCH- VERIFIED problems. These trajectories achieve PAss@26 =64.4% (Figure 14). Next, sample 7 tests using our testing

---

Published as a conference paper at COLM 2025
agent at temperature T = 0.8. When generating tests, the test agent is provided a fixed in-context example (from Django) showing sample starter code and format for writing test cases, We empirically find that use of an incontext example is useful for improving output formatting and lacking domain knowledge in the base LM; improving test generation for 2% problems. Please see Listing C.1 for further details and incontext starter code.
Both verifiers elicit inference time gains. Figure 4 illustrates the BEST@K performance of both verifier types on the SWEBENII-VERIFIE benchmark as a function of number of editing agent rollouts. Both execution-based and execution-free verifiers demonstrate substantial performance improvements with increased number of rollouts. However, BEST@K rate quickly plateaus for both methods, converging similarly to 43.7% and 42.8% respectively.
Limited Distinguishability in Execution- Verifiers. Recall that these verifiers out- put scores based on test pass counts and thus cannot differentiate between patches with identical test pass-rates, limiting their discriminative capacity. We study this discriminative capability from tests generated by our 32B testing agent, prompted SONNET-3.5-y8 model, and Agentless-1.5 reprod uction tests (Xia et al., 2024b)² on subset of SWEBxXCH- problems. Figure 5 (left) presents the problem density distribution for distinguishabiiity rate, i.e., the proportion of tests that successfully differentiate between top-ranked correct and incorrect patches. The results demonstrate that for the majority of problems, less than 20% of tests provide discriminative signal, constraining the re-ranking. Figure 6 additionally depicts that most generated tests either do not reproduce the bug (high Pass +Pass values in 6-1eft) do not pass ground truth patches (high Fail + Fail values in 6-middle) primarily due to bugs or exceptions in the generated test cases.
Vulnerability to Test Toxicity. Following (Chen et al., 2022), we examine the prevalence of toxic tests, i.e., tests that pass incorrect patches but fail correct patches. Figure 5 (right) illustrates the distribution of toxic test rates across different test generation approaches. While toxic tests are generally rare, we find that for a small but significant subset of problems, testing agents generate toxic tests (up to 10% of total tests) that can erroneously rank incorrect patches above correct ones, undermining the reliability of execution verification.
Probability Density for Distinguishing Rates
Probability Density for Texicity Rates
Probability Density
Models
Probability Density
0.30
Models
0.08
Ours (328)
0.25
Ours (328)
Sonnet
Sonnet
0.06
Agentless 1.5 (gpt
Agentiess (gpt-40)
0.04
0.15
0.10
0.05
0.00
0.00
0.0
0.2
0.4
0.5
0.9
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Distinguishing Rates
Toxicity Rates
Figure 5: Analyzing limitations of execution-based verifiers. Left: Problem Probability Distributions for distinguishability rates depicting weak discrimination capabilities of tests. We observe that for the majority of problems, less than 20% of tests provide discriminative signal, constraining the re-ranking ability of test-based agent. Right: Distributions for toxicity rates showing (rare) generation of toxic tests. We find that execution- verifiers are also vulnerable to (rare) generation of toxic tests (tests that pass incorrect patches but fail correct patches); which can undermine the reliability of execution-based verifiers.
Execution-Free Verifiers can rely on heuristics. We next study the workings and limitations of execution-free verifiers. In particular, we first perform quantitative ablation studies, studying the impact of different trajectory components (e-g., output patch, agent thoughts) to verifier performance. To this end, we train multiple execution-free verifiers (54.1) excluding different trajectory components while training the verifier. Results are shown in Figure 7-a. We find that agent thoughts play a considerable role in determining the verifier performance. Surprisingly, the final BEST@26 drops from 42.8% to 37. 97.6% when we remove the trajectory from the verifier input (i.e., only use the final patches). This means that while patch alone is
We utilize test cases from the official artifacts repository (Xia al., 2024a).
7

---

Published as a conference paper at COLM 2025
Probability Density for Pass - Pass
Probability Density for Fall Hall-=Fall Fail
Probability Density for Fall Pass
0.0
Ours 1328)
Rates
Probabiilty Censity
Ours 1328)
Races
9.04
Curs (328)
Rates
004 Agoit ess (apt-4)
Sonnec
Sornec
Agenticss-1 15 (2pt-40)
3.33
Agentless 1. (ppt-40)
Sornec
Frobsbiilty
0.33
602
601
0,0
3.0
02
6.5
Pass
Pass
Fall Fall
Fall
Pass
Figure 6: Problem Probability Distributions for P +Pss, Fail->Fail, and Fail-+Par generated test fractions for various approaches. We identify large fraction of generated tests either do not reproduce the bug (left) or do not even pass the correct solution (middle).
Accuracy (%)
Best@26 (%)
Succeessuuly reproduced the issue Implemenned [-]]
Final Patch
Traj.
71.82
42.8
Ersured edge cases handled
Oalc
68.01
37.6
5. Maintained backward compatibility [..]
chunction-finishy submit /function? [-]
Traj.
Thoughts
68.77
41.4
(a) Impact of Patch Thoughts on execution-free verifier. Patch alone reduces performance, indicating that model relies on other heuristics (e.g., agent thoughts) for reranking; which can be misleading (see part-b: right).
Great works. see
issue:
I. We identified that the original code was failing becruse it was trying to use the inverse() method directly permutations, which [-]]
(b) Top two attention windows while predicting YES for an incorrect trajectory. We find that focusing on heuristics (agent thoughts) can mislead the verifier.
Figure 7: Quantitative and qualitative analysis on limitations of execution-fres verifiers. We perform two experiments: a) Quantitative ablations on the impact of output patch on verifier performance; showing that execution-based verifiers rely on other heuristics (e-g., agent thoughts) over the final patch. b) Qualitative visualization analyzing top k sliding windows with highest mean attention score while predicting output token YES (64.2) for an incorrect agent trajectory (yympyy.ympy--24444 24443: SWE-Bench (Yang et al., 2024b)). Focusing on heuristics (egg, agent thoughts) can be misleading, and the verifier predicts the trajectory as correct. Visualizations are condensed for space. Please refer to the Appendix for further visualizations and results.
responsible for determining the correctness, execution-free verifiers heavily rely on trajectory features, such agent thoughts, to make predictions.
To further investigate this phenomenon, we also perform an attention analysis trying to visualize parts of the input trajectory which are most relevant while predicting the output success with execution-free verifiers. In particular, we perform sliding window search over the input trajectory, and compute the mean attention score over the tokens in the window when predicting the final output token (YES: correct, NO: incorrect). Figure 7 (right) illustrates the top two windows receiving the highest attention scores, demonstrating that verifiers disproportionately attend to agent thoughts. This can be misleading since the verifier can use these sentiment signals these thoughts as proxies for correctness rather than evaluating the technical merits of the solution (i.e. the output patch).
4.3 Hybrid Inference Time Scaling
Combining the verifier strengths. Given the analysis from $4.2, we can summarize two key insights: 1) Execution-based approach provides direct signal for patch correctness through execution but suffers from lack of distinguishing tests 2) Execution-free approach offers better distinguishability between patches through a continuous reward score sEE but can be biased to pay more attention to heuristics (e.g,, agent thoughts) over final output patch.
Given the above insights, we thus propose hybrid verifier that leverages the strengths of both approaches. Particularly, we define the hybrid verifier with score sf! as,
if sf FB where Top,, (5 = -o0 otherwise.
FF
F. is among the top n scores, (2)
H = -Top,, (sFF) +s $
8

---

Published as a conference paper at COLM 2025
where s provides execution-feedback, & FF provides distinguishability in case of a tie with execution-based test scores (as s provides continuous score between 0 and 1), and Top, restricts hybrid verifier to only consider the top verifier ranked patches. In practice, we perform regression filtering after the top-n filtering to ensure non-zero scores.
Main Results. Results are shown in Tab. 4 and Fig. 4. While both execution-based and execution-free methods rapidly reach performance plateaus with increasing agent rollouts (saturating at ~ 43%),, our hybrid approach demonstrates substantially superior scaling properties, yielding significant performance improvements (additional 7-8%); achieving a BEST@26 performance of 51% % the challenging SWEBENCH-VERIRIE benchmark.
Comparison to Open Systems. The proposed approach significantly outperforms other wpen-weight alternatives; reflecting a new state-of-the-an in this domain. Among other generalist-agent methods, SWE-Gym (Pan et al., 2024) recently achieves a BEST@ 16 per- formance of 32.0%. Similarly, concurrent work (Wei et al., 2025) recently achieved 41.0% using RL and BEST@500 (using Agentless). In contrast, despite mainly relying on supervised fine-tuning for training, our proposed approach achieves PAss@1 itself of 34.4% with BEST@26 performance of 51.0% - achieving strong performance improvements through simply more scalable data curation ($2) and better test-time scaling (Figure 4).
4.4 Ablation Studies on Hybrid Verification Design
Figure 8: Ablation Study on Hybrid Verifier. We find three key insights: 1) While both execution-based and execution- free verifiers saturate around 42-43%, the hybrid approach yields significantly higher test-time gains (51%). 2) Regres- sion tests alone are insufficient for hybrid scaling
Esocatao Free Recil bro penerited teran
frit IN Agazshen Tyyrinl (0mm))
$1.0%
Desuth Westodd
11.1%
achiev-
41
ing only 47.4% aggregation performance. 3) Agentic vs Agentless: training a specialized testing agent is important improving the performance from 48.8% 51%.
Abtacion Stbdy her Hyfond Veetleer
Variation with Test-Agent Rollouts. As in 4.2, execution-based test generation can suffer from a lack of distinguishing tests. One approach to address this, is to sample more test- agent rollouts. We quantify this effect in Figure 4 (right). We observe that increasing number of test-agent rollouts consistently helps improve performance with our hybrid approach.
Compute-Efficient Rollouts. Figure 4 (right) illustrates the BEST@K performance as function of both test-agent and code-editing agent rollout counts. Interestingly, we find that sampling more test-agent rollouts can provide more compute optimized inference-scaling over naively sampling more editing-agent rollouts. For instance, increasing the number of editing-agent rollouts from 16 21 improves the BEST@K performance from 47.6% to 48.4%. In contrast, simply sampling more test-rollouts can yield better gains (BEST@K 49.3%).
Regression Tests Alone are Insufficient. Our execution-based verification framework inte- grates both regression and generated reproduction tests. Figure 5 (right) isolates the impact of regression tests alone on the final performance. While regression tests alone improve performance from 42.9% to 47.4%, using generated tests further enhances performance to 51.0%, demonstrating that both test types provide essential and complimentary signals.
Agentic vs Agentless Tests. A distinguishing feature of our approach is to train a specialized agent for test-generation; instead of the zero-shot approach from Xia et al. (2024b). To evalu- ate this design choice, we conducted a controlled comparison using official Agentless tests from their released artifact (Xia et al., 2024a) within our hybrid verification framework the DEBENCHI-VERIFEE benchmark. Figure 5 (right) demonstrates that while Agentless tests provide meaningful performance improvements, our agent-generated tests yield superior results (51.0% versus 48.8%), validating our agent-based approach to test generation.
Note that test-agent rollouts also usually considerably cheaper than editing-agent rollouts.

---

Published as a conference paper at COLM 2025
Role of To We evaluate the impact of the Top, filtering mechanism introduced in Equa- tion (2). Figure 5 (right) shows that this selective aepeeiivaaplicaattiiin application strategy improves performance from 49.8% to 51.0%. This improvement likely stems from mitigating the impact of toxic tests (64.2) by restricting their application to higher-quality patches (identified via execution-free reward scores s thereby enhancing the reliability of the verification process.
Related Work
Programming Agents. Recent work on GITHUB issue resolution includes SWE-agent (Yang et al., 2024b), Autocoderover (Zhang et al., 2024b), OpenHands (Wang et al., 2024), Agent- Less (Xia et al., 2024b), Moatless Orwall (2024). All of them rely on proprietary models due to a lack of datasets and open-weight models a gap our work addresses.
Agent Training Environments. Existing SWE agent environments have key limitations: SWE-Bench (Jimenez et al., 2023) lacks executable training environments, R2E (Jain al., 2024b) offers only 246 instances with function completion. SWE-Gym (Pan et al., 2024) collects executable GITHUB environments similar to us but rely on human-written issues and test cases. Synthetic data generation has been studied in various domains but our work is the first apply it for executable GITHUB environment collection. We use back- translation (Li et al., 2024) and test-generation in SWEGEN approach. Please see Long et al. (2024) for a comprehensive survey on synthetic data generation methods.
SWE-Agent Training. Ma et al. (2024) and Xie et al. (2025) train on synthetic code editing tasks. Pan et al. (2024) study SFT on agent trajectories and inference scaling similar to our work. Wei et al. (2025) explores reinforcement learning large scale data collected from real-world GITHUB issues without execution feedback.
Verifiers for SWE-Coding Tasks. Various works have explored of verifiers for SWE tasks. AgentLess (Xia et al., 2024b) used majority voting to select the best patch from multiple agents. Agentless-1.55 relied reproduction and regression tests to verify the correctness of generated patches. Zhang et al. (2024a) proposed multi-agent commitee-review (LLM judge) to select the best patch from multiple agents. Pan et al. (2024) proposed trajectory verifiers to re-rank the generated patches based on LLM score.
Verifiers for General Coding Tasks. Various works have explored the use of verifiers for general coding tasks on isolated puzzles (HumanEval (Chen et al., 2021)), interviews (Jain et al., 2024a), and competition or olympiad problems (Hendrycks et al., 2021; Li et al., 2022) Gu et al. (2024) showed that LLM judges perform poorly on checking correctness of generated code. Chen et al. (2022); Ridnik et al. (2024); Key et al. (2022); Zhang et al. (2023a) study how test generation can be used to re-rank the generated code samples. Inala et al. (2022); Zhang et al. (2023b); Ni et al. (2023) employ neural code re-ranker models.
In this work, we extend these lines of work by first presenting novel insights challenges and opportunities for both execution-based and execution-free approaches in SWE- Coding. Using these insights, we also propose novel hybrid approach that effectively combines their strengths to achieve better performance (51.0% on WNNEBENCH-VERIFIED
Conclusion
In this paper, we introduce R2E-Gym, the largest gym environment and training framework for scaling open-weight SWE agents. We share two key insights: 1) Synthetic data curation can enable more scalable training on SWE tasks. 2) Hybrid-test time scaling: different axis for test-time scaling (execution-based testing agents and execution-free verifiers) exhibit complementary strengths; which can be leveraged to achieve significantly higher test-time gains. Overall, our final approach achieves 51% on SWE-Bench Verified, reflecting a new state-of-the-art for weigtt SWE agents, while also for first-time showing competitive performance with some proprietary models. We hope that our work can offer unique insights for scaling open-source SWF-agents real-world applications.
10

---

Published as a conference paper at COLM 2025
Acknowledgement
N. Jain and M. Shetty are supported by NSF grants CCF:1900988, CCF:1908870, and by SKY Lab industrial sponsors and affiliates. This work is additionally supported by the R2E OpenPhilanthropy grant.
11

---

Published as a conference paper at COLM 2025
References
Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. Swe-bench+: Enhanced coding benchmark for llms. arXio preprint ar arXio:2410.06992, 2024. 19
Anthoopic. Raising the bar on SWE-bench Verified with Claude 3.5 Sonnet. https://www. anthropic. wwt////eaa/c/////////e/ee/ swwrearrrhssse---e-e---ossseee 2024.
Anthoopic. Claude 3.7 sonnet. https://www/
February 2025.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. arXiv preprint arXiv:220..1039, 2022. 7, 10
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Eval- uating large language models trained on code. arXiv preprint orXio:2107.03374, 2021. 10
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers to solve math word problems. Xrv preprint ar Xiv:22100. 14164 2021. 21
Alex Gu, Wen-Ding Li, Naman Jain, Theo X Olausson, Celine Lee, Koushik Sen, and Armando Solar-Lezama. The counterfeit conundrum: Can code language models grasp the nuances of their incorrect generations? arviv preprint ar 412:22402.19475, 2024. 10
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NeurlPS, 2021. 3, 10
Jeevana Priya Inala, Chenglong Wang, Mei Yang, Andres Codas, Mark Encarnación, Shu- vendu Lahiri, Madanlal Musuvathi, and Jianfeng Gao. Fault-aware neural code rankers. Advances in Neural Information Processing Systems, 35:13419-13492, 2022. 10
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXtv:2412.16720 2024. 1,2
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar- Koushik Sen, and lon Stoica. Livecodebench: Holistic and contami- nation free evaluation of large language models for code. arXiv preprint ar arXio:2403.07974, 2024a. 10
Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica. R2e: Turning any github repository into programming agent environment. In ICML 2024, 2024b. 3, 10
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? ar arXiv preprint ar arXt::3310.06770, 2023. 1,3, 10
Darren Key, Wen-Ding Li, and Kevin Ellis. I speak, you verify: Toward trustworthy neural
program synthesis. ar Xiu preprint ar xxXi:2210.008848, 2022. 10
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason We- ston, and Mike Lewis. Self-alignment with instruction backtranslation. arx preprint Stf::2308.062583 2023. 2,3
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. In The Twelfth International Conference on Learning Representations, 2024. URL https:/ http:://///eeeeee//e/mmme n
forum?id=1oi JHJBRS.. 10
12

---

Published as a conference paper at COLM 2025
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 37866624; 2022. 10
Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint ar Xrio:2406.15126 2024. 10
Yingwei Ma, Rongyu Cao, Yongchang Cao, Yue Zhang, Jue Chen, Yibo Liu, Yuchen Liu, Binhua Li, Fei Huang, and Yongbin Li. Lingma swe-gpt: An open development- process-centric language model for automated software improvement. ar preprint ar drXim:2411.00622, 2024. 6, 10
Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. Lever: Learning verify language-to-code generation with execution. In International Conference on Machine Learning, pp. 26106-266 26128. PMLR, 2023. 10
A. Orwall. Moatless tool. https: //tp::///ihhuun httptttt...////aaaaa/aaaasssseee 2024-10-22. 10, 19
2024. Accessed:
Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym, 2024. URL
https://arxiv.org//bl www..//gg/bbbb//...// 21139. 1, 3, 4, 6, 9, 10,
Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint ar arXio:2401.08500, 2024. 10
Xingyao Wang, Boxuan Li, Yufan Song, Frank Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. arv preprint ar Xit:2407.167711, 2024. 1, 4, 6, 10, 19
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. Xiv preprint ar arXiv:2312.02120, 2023. 2,
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. Swe-rl: Advancing IIm reasoning via reinforcement learning on open software evolution. arXiv preprin! drXie:2502.18449, 2025. 6, 9, 10
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: De- mystifying Ilm-based software engineering agents. https: Agentless, 2024a. 7,9
comm/ppenutoooder/
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang Agentless: Demysti- fying llmeb software engineering agents. arv preprint ar AxXiv:2407.00449, 2024b. 5,5, 6, 1,,,10
Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen. Swe-fixer: Training open-sourc 1lms for effective and efficient github issue resolution. arXiv preprint arXiv:2501.05040, 2025. 1, 6, 10
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. ar preprint ar 200711407.10671 2024a. 21
13

---

Published as a conference paper at COLM 2025
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXio:2405.15793, 2024b. 8,8, 10
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ar Xiv preprint drXio:2210.03629, 2022. 19
Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. Algo: Synthesiz- ing algorithmic programs with generated oracle verifiers. ar Xio preprint ar xrXo::3005.1451, 2023a. 10
Kexun Zhang, Weiran Yao, Zuxin Liu, Yihao Feng, Zhiwei Liu, Rithesh RN, Tian Lan, Lei Li, Renze Lou, Jiacheng Xu, et al. Diversity empowers intelligence: Integrating expertise of software engineering agents. In The Thirteenth International Conference on Learning Representations, 2024a. 10
Tianyi Zhang, Tao Yu, Tatsunori Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida Wang. Coder reviewer reranking for code generation. In International Conference on Machine Learning, pp. 41832-41846. PMLR, 2023b.
Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury, Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 1592-1604, 2024b. 10
Wenting Zhao, Nan Jiang, Celine Lee, Justin T Chiu, Claire Cardie, Matthias Gallé, and Alexander M Rush. Commit0: Library generation from scratch. arXiv preprint ar arXi::2412.01166, 2024. 1
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) Bangkok, Thailand, 2024. Association for Computational Linguistics. URL http://arxiv wwwp:iii.ooroaaeeoooo wtgp/r////////////// 13372. 19, 20, 21
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. ar Xiv preprint ar 246::2200611877,, 2024. 3
14

---

Published as a conference paper at COLM 2025
A
Dataset Details
CLIN
R2E-Gym (8135)
peltoo
Figure Repo distribution for our complete R2E-Gym dataset consisting 8135 instances.
Commit Filtering Heuristics. Our commit filtering approach employs multiple heuristics to identify high-quality bug fixes and improvements suitable for training data. We particularly filter for small scoped changes, prioritizing non-documentation updates, and correlated code and test matches. We perform this filter at both line and AST entity level. To ensure consistency and quality, we employ specific thresholds in our filtering process:
Maximum of 5 non-test files modified in a single commit
Maximum of 100 edited lines across all non-test files
Maximum patch length of 2000 characters to ensure focused changes
No more than deleted entity in non-test files
Maximum of 3 added entities in non-test files
Maximum of 3 edited entities in non-test files
No more than 10 statement-level changes to maintain tractability
Additionally, we use LLM as judge filter to further refine our dataset.
Repository Installation. Installing historical commits from GitHub repositories presents significant challenges due to evolving dependency requirements and API changes. We use a Docker-based approach with a search-based dependency resolution strategy to create reproducible environments for each commit. Our installation process follows these steps:
1. Extract dependency information from requi requntssnns.. txt, setup. py, etc
2. Iteratively identify potential version conflicts and compatibility issues
3. Generate multiple candidate dependency configurations
4. Test each configuration until working environment is found
This process is semi-manual and challenging to scale and we aim to rely more on LLMs in the future. Example installation scripts test multiple dependency combinations sequentially, exiting on the first successful build:
build_and muild_aad__chec__paaee wwwcckpandass(( local python_version=s$:
local numpy_version=s1:
local setuptoolss.erssion3333
15

---

Published as a conference paper at COLM 2025
Attempt with first configuration if chaak__c_ck_aaaaaa "3.7" "1.17.*" "<0.39 30 "62.** "0.23"; then
echo
exit fi
# Attempt with second configuration if caailddand_chec_ppaae "3.8" "1.20.* "00.30" *62.*" "0.23"; then
echo secood wew.eecooooocccooossccceeeeeeeee succeeded .Exitiing............... exit fi
Attempt with third configuration
_at_l__and_chek_padddaa "3.10" 26.x" "===3.0.. 3.0.5" "62.*" "0.23"; then
echo "[INFO]_Third_.coml succeeded suucceede...Exitinnn exit fi
Listing 1: Example installation script excerpt
This approach allows create working environments for historical commits, enabling execution-based validation of our dataset.
Test Generation. We use an Agentless-like reproduction test generation approach. A key difference is that we use the ground truth patch as context when generating the tests.
Issue Generation. As discussed in the main paper, we use backtranslation generate synthetic issues for commits that lack human-written GitHub issues. Our approach lever- ages both the code changes in the commit and the test execution results create realistic, informative issue descriptions. The issue generation process follows these steps:
1. Extract failing test functions from the execution results
2. Analyze test outputs to identify error messages and expected behaviors
3. Provide the LLM with commit message, code patch, and test execution results
4. Guide the LLM to generate concise, informative issue that describes the bug without revealing the solution
For each commit, we extract and utilize specific components:
Commit metadata: Hash and commit message provide context about the change
Code patches: We separate non-test file changes (showing what fixed) from test file changes (showing how to verify the fix)
Test execution: We include both old (failing) and new (passing) executions
Test functions: We extract relevant test functions that demonstrate the bug
Assertion failures: We extract and format the failing assertions from the old commit to show error details
The prompt construction carefully organizes these components to give the LLM sufficient context while focusing attention the most relevant information for issue generation. We carefully design our prompting strategy to ensure the generated issues resemble human- written ones, focusing on clarity, naturalness, and providing sufficient information for understanding the bug.
Build the complete prompt with components def get_prompt (commit, execution_result. issues=None):
# Include commit hash and message Include commit patch (non-test files)
Include test file changes
16

---

Published as a conference paper at COLM 2025
Include execution results from old and commits
Include improved test functions
Include test function code
#
Include assertion failures
# Include example issues and instructions
Listing 2: Issue generation code structure
The template below shows our prompt guidelines:
As you are trying generate synthetic issues, you will follow these guidelines:
1. Keep the issue concise and informative.
2. Describe the failing test, including the input that causes the failure the nature of the failure, and the expected behavior. Do NOT mention test functions or files directly.
3. Do not reveal the solution to the problem in the issue. Only describe the bug and the expected behavior.
4. If there are multiple failing tests, focus the most informative or a subset that best describes the general nature of the failure.
5. Describe the expected output of the failing test:
For errors, describe the error message.
For failing tests, mention what is supposed happen.
6. Write the issue as human would, using simple language without excessive formatting.
7. Use concrete terms to describe the nature of the failure. Avoid vague terms like "specific output" or "certain data".
8. INCLUDE test code describe the bug but keep it brief and relevant. Truncate or simplify tests longer than 5-6 lines.
9.
Do not mention external files unless absolutely necessary.
10. Format code snippets using triple backticks.
The issue should include:
1. clear and concise title
2. description of the problem with detailed example buggy code
3. Expected behavior
4. Actual behavior error message
Listing 3: Issue generation template
This approach enables to generate high-quality synthetic issues that provide clear problem statements for our training data, even for commits that lack human-written issues. Below are examples of synthetic issues generated using our approach:
**Tille:R Calling "load()) Before "draft()" Causes "draft ()* to Fail for JPEG Images
** **Decripttion:**
When generating a thumbnail for a JPEG image using the tthubbnail method, the method calls "load()) before "draft() This sequence results in the d) method returning "None", which prevents the thumbnail from being properly optimized.
**Example : "pythho
from PIL import Image
with Image open ("Tests//iages//oppelll jpg") as im:
im. thumbnail ((64, 64))
**Expected Behavior
17

---

ValueError The recent changes were intended to allow route names to Published as a conference paper at COLM 2025
The "thumbnail() ()" method should utilize the () method optimize the image size before loading, ensuring that the thumbnail is resized correctly and efficiently.
**Actual Behavior:*x
The ")raft()" method returns "None" because () is invoked before it This prevents the thumbnail from being optimized, potentially leading to incorrect thumbnail sizes or unnecessary memory usage.
Listing 4: Example synthetic issue for PIL. image thumbnail bug
**Title:** Unable Register Route with Names Containing Both Dots and Colons
**Description:**
After merging branch '0.18', attempting to register route with name that includes both dots ('.) and colons (" results in
be sequence of Python identifiers separated by dots or colons, but this combination is still causing issues.
**Example : python
from aichttp. web import Ur1Dispatcher, PlainRoute
def handler handler(reques) (request):
return 'Hello"
router Ur1Dispatcher()
Attempting to register a route with both dots and colons in the name route PlainRoute GET",, handler, 'test. test:test', //handler/to/pattl ") router. reuiserrreisteerooutee (ooutruoututeee
**Expected Behavior
Registering route with name like 'test. test: test should succeed without errors, the follows the updated rules allowing multiple identifiers separated by dots or colons.
**Actual Behavior:*x
VValueError" is raised with the message:
ValueError: Incorrect route name value, Route name should be sequence of python identifiers separated by dot or column
This prevents the registration of route names that include both dots and colons, contrary to the intended flexibility introduced in the recent commit.
Listing 5: Example synthetic issue for a route name validation bug
Patch Minimization. We identify that the ground-truth patches often contain irrelavant code changes that are not required to fix the bug, often making modifications to style and structure of the programs. We implement a patch-minimization approach to identify the minimal set of code changes required to fix the bug by iteratively removing the code changes and checking whether the tests still pass. This allows us to collect fine-grained signal for evaluating localization capabilities of LLMs.
B
SFT Training
Agent Details.
18

---

Published as a conference paper at COLM 2025
Editing Agent
Trajectory
Issue Code
Patch
Figure 10: Code-editing agent architecture: The agent takes an issue description and code- base as input and produces a patch that fixes the issue.
We use R2E-Gym to train general-purpose prompting agent. In particular, we train our code-editing agent on tasks from R2E-Gym, where given an executable environment & and problem description D, the agent is asked to solve the provided issue using any means necessary. Particularly, unlike (Orwall, 2024), we do not rely on the use of specialized workflows. The agent is tasked to solve the entire task end-to-end, including writing its own reproduction scripts, finding the bug, proposing a fix and then testing its correctness. Similar to (Wang et al., 2024), the agent also provided with finish tool, allowing it to submit a solution if it thinks it has completed the task.
Agent and Tools. Similar to (Aleithan et al., 2024; Wang et al., 2024), we adopt the traditional REACT format (Yao et al., 2022) for agent-design. For AGENTHUB, we use a minimalistic set of four tools to enable the agent to perform diverse SWE tasks; 1) file editor :or: for viewing and editing files, 2) search tool: for searching relevant term in a given file or folder, 3) execute _bash: allowing execution of non-interactive bash commands (e-g., for running test scripts), 4) submit: : for ending the current trajectory while returning expected outputs.No internet or browser access is provided to the agent during the training process.
Data Curation. For training, we use supervised finetuning with rejection sampling using trajectories from sonnet-3.5 model for supervision. To avoid contamination, we only use a subset of R2E-Gym consisting of repos with no overlap with the SWE-Benchdatasett
The resulting subset (R2E-Gym-lite) consists of 4538 executable environments across 10 repositories (Figure 2). Overall, we collect a total of 3321 successful trajectories from 2048 unique test environments. For rejection sampling we use the unit tests from R2F-Gym environments (both synthetic and existing). For each trajectory, N 40 steps. Also, we limit the number of tokens per-trajectory to 32K max tokens. Finally, also use maximum timeout of 10-min for the overall trajectory and 90 seconds for each action execution, in order to avoid cases where the agent launches a long-running background process. We collect all training data using a temperature of 0.2.
use maximum of
Training Setup and Hyperparameters. For training, we use the Qwen-2. Qwen-2.5-Codee 5-Coder 7B, 14B and 32B series the base model for training SWF-agents R2F-Gym. For training perform full SFT using the above collected trajectories using L.LMMA-Fcctory (Zheng et al., 2024). We train the overall model for total of 2 epochs, batch size as while using learning rate of 15- The warmup ratio for training was constraints, a maximum context length of 20K was used for training the agent. In future, the use of context-pane monext-paralllelim can enable us to further push the performance when training SWF-agents on more complex tasks requiring larger-context lengths.
0.1. Due to computational
C Inference Time Scaling
C.1 Execution-Based Testing Agents
Agent Details. We train a specialized testing-agent that generates reproduction test cases to determine whether candidate patch resolves the issue (i.e., whether the patch passes the generated test suite). Specifically, we train the testing-agent (using QWEN-CODER-32D as base-model) to generate comprehensive test script containing M 10 diverse tests that cover various inputs, corner cases, etc. We use the same agent scaffold from Sec. for training the testing agent.
19

---

Published as a conference paper at COLM 2025
Testing Agent
Trajectory
Issue Code
Test Patch
Figure 11: Testing agent architecture: The agent generates comprehensive test cases to verify if candidate patch resolves the issue.
Data Curation. For training, we use supervised finetuning using trajectories from sonnet-3. 5 model for supervision. Overall, we collect total of 2203 test-generation trajec- tories from sonnet (both positive and negative trajectories with minimal rejection sampling). For each trajectory, we use maximum N = 40 steps. Also, we limit the number of tokens per-trajectory to 20K max tokens. Finally, also
maximum timeout of 5-min for the overall trajectory and 60 seconds for each action execution, order to avoid cases where the agent launches long-running background process.
Training Setup and Hyperparameters. For training, we use the QWEN-CODER-326 model as the base model. We then the above collected training SFT trajectories to perform full finetuning with the QWEN-CODER-0 train the overall model for total of epochs, batch size as 8 while using a learning rate of le - 5. A maximum context length of 20K was used for training the agent. The warmup ratio for training was set 0.1.
model using LLaMA-Factory (Zheng et al., 2024). We
In-Context Starter Code Demonstration. We provide the following in-context starter-code demonstration (from the Django repository) the testing agent.
import
import django
from django. conf import settings
from django. db import models
from django. test import TestCase
from django. uest. import setup_test_eeviiommenn
Configure Django settings before setup os environ. (atpaaatttDDANGGSSSTTTTTI
'tests. test_sqlite. ')
Override settings
settings. configure DATABASES=(
"default"; {
"ENGINE": "django. db. backends. sqlite3", "NAME": "test.db",
"TEST": [
"NAME": "test. db".
)
}.
SOLTTTTNNMOOULESSSE testss: None), # Disable migrations for the tests app
Setup Django (etup) setup_.eett_eev.rrmmentt(
Define test models class ExampleModel (models.Modee): example_char = models. dee.e (max_eength=2555)
example_int models. IntegerField()
20

---

Published as a conference paper at COLM 2025
class Meta: app_label 'tesss' Set the app_label to 'tests"
# Create the database tables
from django. core. management import call_comman call_command ('migrate', run_syncdb=True)
def add_test_datat
"** Create test instances of the model """
ExampleModel objects cewampeeeecaarreeteeeeeeeeeeeeee
example_int=1)
ExampleModel objects create (example_chate " "esst_2", example_int=2)
Add test data add_ add__test_data()
Listing 6: Incontext Demonstration for Testing Agent
C.2 Execution-Free Verifiers
Trajectory (Issue React- Loop + Patch)
Trajectory Verifier
YES/NO
Figure 12:
Execution-free verifier architecture: The verifier predicts whether a patch is correct based on the full trajectory without executing the code.
Verifier Details. In addition the execution-based "testing agents", we also explore the
execution-free outcome-supervised reward models (a.k.a verifiers) (Cobbe et al., 2021). In particular, given a problem statement D, agent-trajectory
{ 42, 02, o2, 0n) and output patch O from the code-editing agent on the R2E-Gym environments, we train Owen2.5-Coder-111 model (Yang et al., 2024a) to output scalar score value s E [0,1] predicting the probability output patch being correct. Specifically, following (Pan et al., 2024) we output the correctness of each patch through output tokens YES (correct) and NO (incorrect). The overall reward score is then computed by normalizing the relative proba- bility of YES token through the log-probaiilities of the corresponding token predictions.
P(YES)/ (P((S)) P(NO)), where P(YES) and P(NO) are estimated
Training Data. We first use the trajectories collected for code-editing agent training $3 in order to obtain a collection of positive and negative samples for verifier training. Following the best configuration from (Pan et al., 2024), we also generate on-policy trajectories using our trained 32B model. We then filter the collected samples to have an equal number of positive and negative samples. The overall dataset consists of 5700 total trajectories including both positive and negative samples. For training, we follow the template from (Pan et al., 2024), asking the L.L.M model to predict the output as YES for positive and NO for negative trajectories.
Training Setup and Hyperparameters. For training, we use the QWEN-CODER-14B model as the base model. We then use the above collected training SFT trajectories to perform finetuning using LLaMA-Factory (Zheng et al., 2024). Similar to (Pan et al., 2024), we perform LORA finetuning using rank of 64. We train the overall model for total of epochs, batch size of while using a learning rate of le - A maximum context length of 32K used for training the agent. The warmup ratio for training set to 0.1.
21

---

Published as a conference paper at COLM 2025
C.3 Execution-Based Analysis
In our analysis of execution-based testing agents, we focus on two key metrics: distinguisha- bility and toxicity of generated tests. These metrics help understand the effectiveness and limitations of execution-based verification.
Distinguishability Rate. The distinguishability rate measures test's ability to differentiate between correct and incorrect patches. A test is considered " "distinguishing" if it behaves differently when applied correct patches versus incorrect patches. practical terms, this means the test can help us identify which patches are correct and which are not.
For example, consider a test that passes for all correct patches but fails for all incorrect patches-this test has perfect distinguishability: Conversely, test that passes (or fails) for both correct and incorrect patches provides no useful signal for distinguishing between them. Mathematically, for given test and a set of patches P divided into correct patches Pe and incorrect patches P, we compute distinguishability metric as:
Distinguish(1) 1 max (p, * max Pass (p, LPER:
pef.
where (p,)( indicates whether patch p passes test I, and [[ is the indicator function. This formula checks whether the best-performing incorrect patch behaves differently on the test compared to the best-performing correct patch. The distinguishability rate for a set of tests T is then the average distinguishability across all tests:
DistinguishRate(T) = [T]
Σ Distinguish(t) ICT
In our analysis, we found that most generated tests have low distinguishability typis-typ less than 20% of tests can effectively differentiate between correct and incorrect patches. This limitation significantly impacts the ability of execution-based verifi- cation to identify the best patches, especially as the number of candidate patches increases.
Toxicity Rate. We define toxic tests as those that incorrectly favor incorrect patches over correct ones. The toxicity rate the proportion of tests that exhibit this behavior. Mathemat- ically:
Toxic(!)
max Pass(p,t) max Pass( p)) pPPP
(5)
PCP.
The toxicity rate for set of tests T is:
ToxicityRate(
Σ Toxic(f) FCT
(6)
[T]
While toxic tests are generally rare, they can significantly impact verification reliability when present, with toxicity rates reaching up to 10% for some problems. These findings highlight the importance of generating diverse, high-quality tests and the value combining execution-based verification with other approaches, such execution-free verifiers, to achieve more robust results.
C.4 Execution-Free Analysis
Figure 13 shows the limitations of the execution-free verifier.
D Example Testing Agent Outputs
This section provides examples of test cases generated by our approach.
22

---

Published as a conference paper at COLM 2025
Since
[ASSISTANT]
1. Successfully reproduced the issue
Great! The fix works. Let's analyze what we did:
2. Implenented fix
3. Verified the fix works
4. Ensured edge cases are handled
1. We identified the original code was falling because
5. Maintained backward compatibility
was using inverse()) nethod directly which
believe
submit this the
solution:
doesn't exist far permutations.
2.
issue by:
<function=finishh
Using "**-1" for inverses, which works for both
sparameter sutt / aparameteraresulta the bhack_homomorphiie function...
Permutations and FresGroupElements
[ASSISTANT]
[USER] Exit code: Execution autput [execute_bash [STDOUT]
the changes look good. I've made the following impravements:
1. Simplified logic to check both the generator and inverse
Used the "**=1" and FreeGroupÉlement Maintained robust handling of inverted generators
for Permutation
Testing with DihedralGroup (3):
SUCCESS: Hamonorphism created successfully
Testing with SymmetricGroup(3)) (3): SUCCESS: Homomorphism created successfully
Let's the reproduce script again:
sfunctionFexecutr_beo
searameter=cnd-pythee reproduce_issee s/ parameter>
</function> E/ASSISTANT
Figure 13: Analyzing limitations of execution-free verifiers. Qualitative visualization analyzing top k = 4 sliding windows over agent trajectory (refer $4.2) with highest mean attention score (sympy--sympy 24443: SWE-Bench). We observe that the RM can be biased by the agent thought / actions, instead of relying on the final output patch.
Best@K with Editing Agent Rollouts
Best@K with Editing Agent Rollouts
PasegK
65
PassiK
60
60
Passik
55
Best@k
477
653
42.5
42.
44,7
45
43
40,1
41.5
Hybrid Verifler
Verfflers
401
41.3
Hybrid Verifler
Veriflers
Execuiter dhaned
Exse inn-iased Variflr
Executior Free Veriferr
32.9
Execution
57
6
9
0
5.5
6
Agent Rollouts (K)
Agent Rollouts (K)
Figure 14: Pass@K plot for our agent and using Agentless tests respectively.
D.1 Example SymPy Relational Parsing Tests
The following example shows truncated test suite for validating relational parsing in SymPy, demonstrating our approach's 's ability to generate multiple test cases. This test was generated to address the issue SymPy PR #24661, which fixes relational parsing the SymPy library.
from sympy import Lt, Gt, Le, Ge, Eq, Ne
def test_relational_ parrsingn_ppriing (): # Test case 1: Basic less than operation
try:
result parse_expr(e ('1 2', evaluate=False)
expected = (1 2, evaluate=false)
if str(result) == str (expecteed;
print Case 1: Issue resolved")
else:
23

---

Published as a conference paper at COLM 2025
print("Test Case Issue reproduced") except Exception e: Case Other issues")
# Test case 2: Greater than operation try:
result parse_expor ('3 > 2', evaluate=False) expected (3, 2, evaluate=False)
if str(result) == str (expected):
("int Case 2: Issue resolved")
else:
print Case 2: Issue reproduced")
except Exception as e: print Case 2: Other issues")
... more test cases omitted brevity]
# Test case 9: Chained comparisons
try:
result pxrse_expr( ('1 2', evaluate=False) if isinstance (result, bool): ("int("Test Case 9: Issue reproduced") else:
print("Tess Case Issue resolved")
except Exception as e: print Case 9: Other issues")
Listing 7: Test cases for SymPy relational parsing (truncated). Successfully detects incorrect code from correct code.
D.2 Example 2: Django Model Choice Field Validation Tests
The following truncated example demonstrates tests for Django's ModelChoiceField valida- tion with proper setup and teardown code. This test was generated to address the issue in Django PR #13933, which improves error messages in ModelChoiceField validation.
from django. forms import ModelchoiceFiild, ModelMultipp tippehiiliipllhhhcceellelee from django. db import models
from django. core. exceptions import ValidationError
from django. test import TestCase from django. db import connection
# Create a simple test model class TestModel (models Model):
name models. CharField (max_1ength=100)
def _sstr_ return self name
class Meta: app_label 'test_app"
def ketup_test_ee test_environment
Create the table
with connection. achema__ditore () as schema_ede editor:
cchema_ddttor..cratt__ddddd create_modell (TestModel)
Create test data TestModel cestModel.objcts.. create (name="Option 1")
TestModel objects create (name="Option 2")
TestModel. wwwoots....eaeeeeeee (name=" Option
"namenn 3")
def cleanup_test_enve ()
24

---

Published as a conference paper at COLM 2025
Drop the test table with connection schema schema_editorg as siinema_editor: schema_editon settmmaeeiioor.eleeeeeeeoooo delete_
(TestModel)
def dest_meeachotce..alltttt.
try:
Test Case 1: Basic invalid choice for ModelChoiceFielo field ModelChoicefield (cerssseeedctyyystteeeetteeeee objects all ())
try:
Rleld.clean (999) Non-existent ID print( Case Issue resolved")
except ValidationError as e: if "999" not in str(e):
Value should be in error message
("int Case 1: Issue reproduced")
else:
print ("Test Case Issue resolved")
# [4 more test cases omitted for brevity]
Test Case Valid choice obj TestModel. objects. first() try:
result field. clean (obj id)
if result obj: ("int("Test Case Issue resolved")
else:
print Case 6: Issue reproduced") except validationError:
print("Test Case Issue reproduced")
except Exception as e:
error: (e)")
Listing Test cases for Django ModelChoicePield validation (truncated). Most test cases error due to unhandled exceptions and do not distinguish.
E Agent Trajectory Example
This section provides visual example of an agent's trajectory while solving software engineering task. The sequence shows the step-by-step process from problem statement to solution, demonstrating how our agent approaches and solves real-world programming issues.
This trajectory example illustrates several key aspects of our agent's problem-solving ap- proach:
Systematic Exploration: The agent methodically explores the codebase to under- stand the context and locate the issue.
Root Cause Analysis: Rather than addressing symptoms, the agent identifies the underlying cause of the problem using test_issue.pr .py.
Solution Development:
The agent formulates a clear plan before implementing
changes.
These capabilities enable agent effectively tackle complex software engineering tasks that require deep understanding of code structure, programming language semantics, and software design principles.
25

---

Published as a conference paper at COLM 2025
Problem Statement
PolyElerent. as_espr()"" accepting symbols The method PolyEllement.as_coppr
httpp://g3thhbb.co https://sttaa.cc////y/sssssrs// http://wwww......././///./////ta
you
you wrong
of symbols, and get an error nessage, or you pass the right number of symbols, and it ignores then, using ring.siin. instead:
"python
>.0 fros synpy import ring,
syrbols
soo R, x, y, == ring("".y.,", 221
3exem2ay xxx U, V, M synbots("u,ww,")
f.ss_expr(U, 24y xoy42 762243
Figure 15: Problem statement presented the agent, describing the issue that needs to be resolved.
26

---

Published as a conference paper at COLM 2025
Thoggh
Proutieett ausetin sptartt. Jlls
spe opporde
whantiee
Searstar
Croution (nmrel
RCy's te Li on the Fil:: www.eeeeewwteeraraaeaoo
113
and petta No ON
mume tirt, nymdao
(a) Step 1: Initial analysis and exploration
(b) Step 2: Detailed Exploration
only bear Whyy valata
weheelee rlaclach abou gor aniter
and
de drsgnal
(c) Step 3: Reproducing the issue
(d) Step 4: Running reproduction tests
ooer denking urey caber seeets
set yeaato on This canr
meder au yoo teatt Haaott)
Har
willump yeids.
soder syaltt
pored
(e) Step Implementing the
(f Step 6: Verifying the fix
Figure 16: Short successful agent trajectory (using 32B model) showing the step-by-step process of solving a software engineering task. The agent analyzes the problem, identifies the root cause, implements solution, tests it, and verifies that it resolves the issue.
27


Reviews:
[{"review_id": "W5fxTHI6Rv", "reviewer": "Reviewer_fpRC", "summary": "The paper introduces AgentGym, the largest procedurally curated environment, consisting of 8.7k+ problems (>3x increase over prior benchmarks), with executable gym environments, unit tests, and task descriptions, for training SWE-agents. The paper proposes SynGen: a synthetic data curation recipe for curating executable training environments w/o relying on human tests and issues -- execution environments can directly be curated from commits through backtranslation. The paper also investigates \"Hybrid inference time scaling\": 1) execution-based verifiers that evaluate patches through test cases. 2) execution-free verifiers that assess trajectories through learned models. Empirically shows that a hybrid approach performs better than either alone. The paper is well written and easy to follow.\n\nOverall, the paper makes a significant contribution by scaling up the synthetic SWE task generation and showing strong empirical results of different verifier strategies. I recommend an accept.", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Empiricism, Data, and Evaluation\n\nThe paper provides extensive compelling empirical results to support their claims. The paper shows competitive open-weight performance (BEST@26 w/ hybrid 51% on SWEBench-Verified), closing the gap with proprietary systems (Claude-3.7 Sonnet at 70.3% BEST@Any).  \n\n* Technological Impact\n\nThe paper claims to open source their environments (more than 8.7k tasks), models, and agent trajectories. \n\n* Understanding Depth, Principled Approach\n\nThe paper provides insights in the complementary merits of execution-based verifiers vs execution-free verifiers. The paper provides thorough analysis on test-time scaling approaches.\n\n* Clarity, Honesty, and Trust\n\nThe paper is well written and mostly easy to follow.", "reasons_to_reject": "* Understanding Depth, Principled Approach\n\nI'd be curious to learn how to evaluate the quality of the curated problems--a qualitative evaluation of the curated benchmark would be helpful. \n\nThe draft offers few insights into common failure modes of the proposed approach (e.g. types of bugs still missed, qualitative examples).\n\nThe reasoning behind choosing Top­n in the hybrid verifier (Eqn. 2) and its sensitivity is not deeply examined.\n\n* Clarity, Honesty, and Trust\n\nConsider describe the backtranslation approach in the paper. \n\nSome acronyms are introduced before the definition (4.1, eq. 1).", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rum6tCYFX6", "reviewer": "Reviewer_2kAH", "summary": "This paper proposes AgentGym, consisting of the largest yet synthetic dataset SynGen of annotated software engineering (SWE) tasks of the SWE-Bench type, and a hybrid verification method that leads to SOTA performance on SWEBench-Verified. The paper is solid --- very well written, with the right level of details in the main paper and a great choice of experiments and articulation of findings. This should serve as a valuable resource for work on building open agents. Further, the insight that execution-based vs. execution-free verifiers offer complementary benefits is super useful in this line of work.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Well-written paper, was enjoyable to read.\n\n* Very timely and useful resource for several groups trying to build strong agents.\n\n* Valuable insight that execution-based and execution-free verifiers offer complementary benefits and that this can be leveraged to create a more powerful agent.\n\n* Strong performance of the resulting trained model on SWEBench-Verified.", "reasons_to_reject": "(**Important but not a reason to reject**) Please note ``AgentGym'' already exists as a related but apparently different resource for agents. See https://agentgym.github.io/ and https://arxiv.org/abs/2406.04151", "questions_to_authors": "N/A", "extra_scores": {}}, {"review_id": "h6uFzM7SGO", "reviewer": "Reviewer_DvQh", "summary": "This work introduces AgentGym, a large set of procedurally generated executable environments (8.7K tasks) for training SWE agents. This is done using test and instruction generation from the repository commits directly, skipping the reliance on human-written GitHub Issues and PRs.\n\nIn particular, the first GitHub Python repositories that can be built successfully. They then filter the commits with some heuristics. And just based on that, they produce an entire SWE task+evaluation with an executable environment. The tests are often LLM-generated, and the task instruction is generated from the commit changes.\n\nThey show that training a model on this data leads to substantial improvements in SWEBENCH-VERIFIED.\n\nAdditionally, in this environment, they explore two test-time scaling strategies: execution-based (passing tests) and execution-free (styling, process-based), and find their complementary strengths that can be combined to obtain even better results.\n\nThis work also achieves SOTA of 51% on the SWE-Bench Verified using open models.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "One of the main bottlenecks for training Swe agents is curating large-scale executable environments. This work introduces a pipeline and approach to scaleably generate these, and also demonstrates its success by developing a SOTA swe agent.\n\nThe experiments and contribution for demonstrating the complementarity of the two test-time scaling approaches are also a novel and interesting finding in themselves.", "reasons_to_reject": "This is an excellent and much-needed work! I don't see any strong reason to reject this paper. A couple of minor points:\n\nGiven LLMs are used to generate the data. It'd be good to have a sense of the quality and solvability of these tasks. E.g., to what extent can humans solve these auto-generated tasks? I understand, though, this is for training, so the bar is not necessarily as high, and downstream performance improvement already is a good signal of utility.\n\nIt is limited to only Python. But the entire pipeline can, in the future, perhaps be extended to other languages.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6gpf0erOGh", "reviewer": "Reviewer_gmVU", "summary": "This paper presents AgentGym, a resource that is the largest of its kind and breaks new ground in terms of proposing two complementary metrics (execution based and execution free) that when combined are argued to achieve comparable performance on SWE-Bench with proprietary models.  Many of the technical details are in the appendix.  A promise is made that the code and resources will be opensourced when the paper is accepted.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper includes a large amount of experimentation to locate where the techniques fall in the landscape.  The experimentation is rigorous, and SOTA models are used.  The claims to fame are the size of the resource and the performance of the hybrid approach being in league with that of proprietary models.  As I was reading, I felt that the paper was vague -- for example, when backtranslation is first mentioned, what isn't mentioned is what is being translated to what.  So the assertion is not really understandable until a little later when the idea of backtranslation for this application is unpacked.  The appendix make a lot more clear, so then I was less concerned about this.", "reasons_to_reject": "I am not totally convinced of the claim that the opensource approach is comparable to the best proprietary models.  Figure 1c seems to make that case, but Table 3 (which is never referenced in the paper) paints a different picture.  Here the best result from the opensource approach is comparable to middling proprietary models, but not the best ones. \n\nApart from the good performance numbers, the large number of experiments, and the bigness of the resources provided, the work is not that creative, and it's not clear that it really pushes forward the science.  I'd like to hear more about what we understand better from this work and not just what makes the performance numbers go up.", "questions_to_authors": "I'd like to hear an explanation for how to reconcile these two things.  I'd also like to see it acknowledged that though the results of this paper seem to indicate value in integrating execution-based and execution-free metrics, that past studies cast doubt of the validity of execution-free metrics.  Here's a reference to corroborate that:\n\nXie et al. CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks (on arxiv)\n\nCan you shed some light on how it's possible that integrating a low-validity metric improves performance?  What kinds of cases does it help with over-and-above just the execution-based metric?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents two ideas for improving Software Engineering (SWE) agents: 1) a scalable method for synthetic curation of execution environments, and 2) a way for combining execution-based and execution-free verifiers to improve agent performance. The experimental results clearly demonstrate the utility of the methods in improving performance on the SWE-bench tasks. The paper is well written overall and the experiments are well executed. 

The reviewers have provided areas for improvement including details on certain aspects of the methodology, additional assessment of the quality of the curated dataset, and some error analysis. The authors are encouraged to address these in the camera ready. Their response does a good job of this.
Decision: Accept