SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
Authors: Yuxiang Wei
Olivier Duchenne
Jade Copet
Quentin Carbonneaux
LINGMING ZHANG
Daniel Fried
Gabriel Synnaeve
Rishabh Singh
Sida Wang
Venue: NeurIPS 2025 poster ()
Keywords: Software Engineering
Reinforcement Learning
Primary area: deep_learning

Abstract: The recent DeepSeek-R1 release has demonstrated the immense potential of reinforcement learning (RL) in enhancing the general reasoning capabilities of large language models (LLMs). While DeepSeek-R1 and other follow-up work primarily focus on applying RL to competitive coding and math problems, this paper introduces SWE-RL, the first approach to scale RL-based LLM reasoning for real-world software engineering. Leveraging a lightweight rule-based reward (e.g., the similarity score between ground-truth and LLM-generated solutions), SWE-RL enables LLMs to autonomously recover a developer's reasoning processes and solutions by learning from extensive open-source software evolution data -- the record of a software's entire lifecycle, including its code snapshots, code changes, and events such as issues and pull requests. Trained on top of Llama 3, our resulting reasoning model, Llama3-SWE-RL-70B, achieves a 41.0% solve rate on SWE-bench Verified -- a human-verified collection of real-world GitHub issues. To our knowledge, this is the best performance reported for medium-sized (<100B) LLMs to date, even comparable to leading proprietary LLMs like GPT-4o. Surprisingly, despite performing RL solely on software evolution data, Llama3-SWE-RL has even emerged with generalized reasoning skills. For example, it shows improved results on five out-of-domain tasks, namely, function coding, library use, code reasoning, mathematics, and general language understanding, whereas a supervised-finetuning baseline even leads to performance degradation on average. Overall, SWE-RL opens up a new direction to improve the reasoning capabilities of LLMs through reinforcement learning on massive software engineering data.

SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
Yuxiang Wei
Olivier Duchenne'
Jade Copet'
Quentin Carbonneaux'
Lingming Zhang² Daniel Fried
Gabriel Synnaeve'
Rishabh Singh'
Sida I. Wang'
Meta AI
Unniersiiyy of Illinois Urbana- Chaaaa--h
Carnegie Mellon University
yuei400111inois.co edu
M sidadmeta.com
Abstract
The recent DeepSeek-R1 release has demonstrated the immense potential of rein- forcement learning (RI. in enhancing the general reasoning capabilities of large language models (LLMs). While DeepSeek-RI and other follow-up work primarily focus on applying RL to competitive coding and math problems, this paper intro- duces SWE-RL, the first approach to scale RL-based LLM reasoning for real-worle software engineering. Leveraging a lightweight ity score between ground -truth and -egeeeertdeedd solutions), SWE-RL enables LLMs to autonomously recover a developer's s reasoning processes and solutions by learning from extensive open-source software evolution data-the record entire software development cycles, including code snapshots, code changes, and events such as issues and pull requests. Trained on top Llama 3, our resulting reasoning model, bench Verified a human- collection of world GitHub issues. To our knowledge, this is the best performance reported for medium-sized (<100B) LLMs to date, even comparable to leading proprietary LLMs like GPT-4o. Surprisingly, despite performing RL solely software evolution data, Llama3-SWE.. has even emerged with generalized reasoning skills. For example, it shows improved results five out-of-domain tasks, namely, function coding, library use, code reasoning, mathematics, and general language understanding. whereas supervised- finetuning baseline even leads to performance degradation average. Overall, SWE-RI opens a new direction to improve the reasoning capabilities of LLMs through reinforcement learning on massive software engineering data.
reward (e.g.. the similar-
achieves a 41.0% % solve rate on SWE-
Introduction
The application of large language models (LLMs) to software engineering (SE) tasks has received significant attention, with researchers exploring their potential to automate various complex SE tasks, such as library-level and complex code generation (Zhuo et al., 2024; Zhao et al., 2024), real-world bug/issue resolution (Xia and Zhang, 2022; Jimenez et al., 2023; Yang et al., 2024c), and software testing (Deng et al., 2023; Jain et al., 2024a). Among these tasks, SWE-bench (Jimenez et al., 2023) benchmark for solving real-ve real-world software issues has emerged and researchers have proposed various agentic (Yang et al., 2024b; Wang et al., 2024b; Gauthier, 2024) and pipeline- based (Xia et al., 2024; Örwall, 2024; Zhang et al., 2024) methods to push LLMs' real-world issue solving capability. However, most current techniques rely on powerful proprietary LLMs like GPT-4o (OpenAl, 2024a) Claude-3.5-Sonnet (Anthropic, 2024a), where advancements are driven more by enhanced prompting strategies than by improvements in the underlying LLM.
focal point of research efforts,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Curate
Select
GitHub
Raw PR dataset
Seed RL dataset
compare( Reward
GRPO
Issue
Code
LM
Reasoning
Predicted patch
Oracle
patch
-1
Wrong format
Update weights
Figure Overview of SWE-RL. We create seed RI. dataset from GitHub PRs, including issue descriptions, code context, and oracle patches. A policy LLM generates code edits via reasoning. Rewards are based on the similarity to oracle patches, with penalties for formatting errors. We optimize the policy using GRPO.
With the release of DeepSeek-R.. R1 (DeepSeck-AI, 2025), reinforcement learning (RL) using rule- based rewards has become crucial technique for enhancing the reasoning capabilities of LLMs across various downstream tasks, including coding (Zeng et al., 2025) and mathematics (Yeo et al., 2025). However, their effectiveness in SE tasks remains limited (DeepSeek-Ar 2025), and their substantial total parameter size (671 (671B) poses challenges for researchers attempting to train them. For mathematics, the reward is generally defined whether the answer predicted by the LLM match the ground truth (DeepSeek-N) AI, 2025). While coding, existing RL research typically utilizes execution feedback (DeepSeek-Al, 2025; Gehring al., 2025) as the reward signal and is limited to competitive programming tasks (Li et al., 2022; Jain et al., 2024b), where code is self-contained apply to real-world SE tasks due to the execution cost
exactly
and easily executable. This challenging and lack of executable environments (Pan et al., 2024). Mcanwhile, previous research (Ma et al., 2024; Pan et al., 2024; Xie et al., 2025) relied on proprietary teacher models and focused primarily on supervised fine-tuning (SFT), which, as we show in the paper, is less effective and less generalizable.
To address these limitations, we propose SWE-RL, the first RL method to improve LLMs SE tasks by directly using rule-based rewards and software evolution data-thh the record entire software lifecycle, including all code snapshots, changes, and events like PRs and issues. As shown in Figure 1, we begin by curating a comprehensive dataset of GitHub pull requests (PRs), which is then transformed into the seed dataset for RL. Each data item includes issue, the corresponding code context, and the oracle patch merged by the PR. During RL, the policy LLM tasked with solving a given issue through reasoning and producing the code changes. The code changes are then converted into a consistent patch format for reward calculation. If the response is incorrectly formatted, the reward will be -1; otherwise, the reward similarity score (between and 1) of the predicted and the oracle patch calculated by Python's 's difflib. Sww.eneeueeeeeeeeeeeeeeeee ceMatcher (Ratcliff and Metzener, 1988). Notably, we provide the complete content of each file the input prompt, which implicitly teaches the model to reason about the precise fault locations before suggesting repair edits.
1ai---...-70--nttrruee (Dubey al., 2024), our model Llama3-SWE-RL-17 solves 41.0% % of the issues in SWE-bench Verified (OpenAI, 2024), human-verified subset of SWE-bbnch, with Agentless Mini, our pipeline-based scaffold built upon Agentless (Xia et al., 2024), featuring simplifications to match our RI. process and enhancements for scaling. This performance is comparable leading proprictary LLMs like GPT-4o (OpenAI, 2024a) and state-of-the-un among medium-sized LLMs with less than 100B total parameters. Our ablation studies demonstrate that S--------E----70-- significantly outperforms its Llama baseline. Additionally, we developed competitive supervised fine-tuning (SFT) model from Llama-3.3-700 3.3- using synthetic data generated in the Magicoder (Wei et al., 2024) style enhance the of (Wei al., 2022) process, employing the same seed as SWE-RL. We show that SWE-RL solely for solving issues, not only surpasses the SFT model in SWE-bench but also excels in other out-of-domain (OOD) tasks, including function-level coding (Chen et al., 2021; Liu et al., 2023), practical code generation with library use (Zhuo et al., 2024), code reasoning (Gu et al., 2024), mathematics (Hendrycks et al., 2021c), and general language understanding (Hendrycks et al., 2021b).
Applying SWE-RL
trained with

---

In these OOD tasks, Llama3-SWE-RL-7000 even outperforms Llama-3.3-70B Insttuutt whereas the SFT model results in decreased performance.
In summary, our contributions are as follows:
We introduce SWE-RL, the first RL approach specifically designed to enhance LLMs for SE tasks using software evolution data (e.g., PRs) and rule-based rewards.
We develop Llama3-SWE-RL-700 trained with SWE-RL on Llama-3.3-708 -1-7tB-Innttuct. It achieves 41.0% on SWE-bench Verified, the best performance among medium-sized language models (<100B) and even comparable to leading proprietary models like GPT-4o.
We show for the first time that applying RL solely to real-world SE tasks, such as issue solving, can already enhance an LLM's general reasoning abilities, enabling it to improve on out-of-domain tasks like math, code generation, and general language understanding.
2 SWE-RL
To prepare the initial dataset for RL, we extract 273k high-quality PR seeds from the raw PR dataset we collected from GitHub. These seeds are selected based on specific heuristics. For example, a PR instance should include at least one linked issue. the issue should describe a bug-fixing request, and the code changes should involve programming files. For each seed, we extract the issue descriptions and code context, including all changed files and some relevant but unchanged files. These are converted to input prompts the policy LLM. We also take the oracle patch from each seed, which will be used for reward calculation. More details are explained in Appendix A.
2.1 Reward modeling
[System] A user will ask you solve task. You should first draft your thinking process (inner monologue). Then, generate the solution. Your response format must follow the template below:
<think>
Your thoughts or/and draft, like working through exercise scratch paper. Be casual and as long you wont until you are confident generate correct solution.
</think>
<solution>
Final solution presented to the user. </solution>
[User] here is the issue text: (issue)
...here is the code context: (context)
...geeraate search/replace edits to fix the issue...
[Assistant]
Figure 2: Prompt template used to train Llama3-SWE-RL with SWE-RL. Given an issue description and corresponding code context, the policy LLM needs to generate search/replace edits (Xia et al., 2024) to fix this issue through reasoning. This is the only subtask we incorporate in the RL training. During inference, the I.LM can generalize to tasks outside the training domain (e-g, file-level localization and test generation). For conciseness, exclude certain prompt details, with the complete prompt template available in Appendix
We bootstrap the policy LLM with the prompt template shown in Figure 2. Assuming that the LLM has generated a rollout given an issue and its code context, the reward function is defined as follows:
if has wrong format, patchgt), otherwise.
R(o) compare
(1)
(patchpred, compare(patchhree
Here, patchpred the patch extracted from the L.LM generation o if correctly formatted, and patchgt means the oracle patch for this issuc.

---

In the implementation, we use Python's difflib. wew.bbseeeennnneatcheeeee as the compare function, which returns a floating point between 0 and 1 indicating the sequence similarity, and adopt Group Relative Policy Optimization (GRPO) (Shao et al., 2024) for policy optimization.
Given seed RL dataset seed where each item contains (1) issue, the issue description, (2) ctx, the code context required to solve this issue, including both files to repair and relevant files not intended for editing, and (3) patchgt, which is the oracle patch. The input prompt for each data item is formed by instantiating the prompt template in Figure 2 with the issue description and code context, which we denote as q = form-proc form-prompt (isspt ctx). The policy LLM me tries to solve the issue by generating code changes through reasoning, and produces multiple outputs for each input prompt 9 given the group size G. Then, the policy LLM aims to maximize the following GRPO objective:
(or 9)) Touia (oi 9) Au, clip
To(ox g) 1 1.l1,,,1+ c.1+c a.) -DDL( (1 To (or (4)
I(0) = E
mim
G
where (issue, ctx, patchet) ~ Desed. q form-prompt(issue, ctx), and for ~ Tooal (g).
(2)
e and 3 are hyperparameters, and Taaaaa and Tyee are the old and reference policy. The rewards [=-mean(tt....rc) are calculated using the normalized rewards rs = R(os)) and advantages A; = std(Fi within each group following GRPO. DKL denotes the estimated KL-divergence (Schulman, 2020).
Our training approach conditions the model the complete context of each file, implicitly forcing it to identify detailed fault locations before generating repair edits. This process inherently teaches the model both bug diagnosis and repair generation. However, during the evaluation on SWE-hench, Agentless Mini requires capabilities beyond generating repair edits, such as file-level fault localization, reproduction test generation, and regression test selection. Remarkably, even without explicit training on these subtasks, Llama3-SWE-Re can generalize to them through the RL process.
2.2 Aha moments and generalized reasoning capabilities
Gittilu solving (in
Simple function implementation -oooodomaannn
Mathematics (out-of-domain)
issue of preserving single quoces 1000 comments cocurs... Toool zew class, the description is being excracted... ...........
need to ideaiity
this peobien, that taklss of numbers and dellmiter as isput...
function
solve this pootiem, first need find
x-mtercepes the two given ellipee...
We can start by checking if the Input list is empty...
For the first elliper.. For the secend ellipee...
But walt, the Tobol tan constrocct, there's this line:
If the list is not empty..
desor iutionn Leggzriagte, sosellinher).
Now: let's find Insersection points.. let's formula...
Howeverr the witee being ssing
However, more Pythonte way to solve problem
descripticn Tedeabr(*
world oar liet compechereioo..
strengtomcrrtin Insarsantial tune): Jluu than's not the isue here.
The length of side All is.. The length side BC is... The length of sidee CD is., The length of side DA is...
But before that, let's think about the edge cases...
the sasconal
of
Alturnatively.. the zip function..
list compretacmiin with
ABCI have same kngth...
segnee all sides
sem, codellsntee ter descraption. Inceadr StringDomourian soon. Invar santial tura)...
But the most straighttooward way solre this problem use simple
Squaring boch sides.. Expanding and simplifynn.... Countiing like ter moc.. Muiljdyying bosh sinles 36.. Adding 42 to both sides..
The tutaal bosu is lilly due to.., And Lhen lite I've gut
Self reflection
Exploring approaches
Divide-and-conee
Figure 3: Reasoning skills emerged from LJama3-SWE-RI-- following the application of SWE-RL. RL helps the model develop reasoning skills like self-reflection, exploring alternatives, and divide-and-conquee strategies, for both in-domain (e.g., issue-solving) and out-of-domain tasks (e.g., function implementation and mathematics).
"Aha moments" on software engineering. With the application of SWE-RL, we observe "aha moments" (DeepSeck-AL, 2025) where Llama3-SWE-RL exhibits emergent reasoning skills. To our knowledge, this is the first study demonstrating the existence of such aha moments in the realm of real-world software engineering tasks, confirming the findings of DeepSeck-RI (DeepSeek-AL 2025), which mainly focuses on competition coding and math. As shown in Figure 3, through RL, Llama3-SW2 SEE----E---RRRB--- can allocate more thinking time reflect its initial assumptions during the issuengolving process. This behavior emerges naturally from the model's 's interaction with RL, rather than being explicitly programmed.
General reasoning capabilities. Surprisingly, we have identified additional aha moments where Llama3-SWE- acquires general reasoning abilities that are transferralee to various out-of-domain

---

tasks, such as function-level code generation and mathematics, although RL is applied exclusively to software issue solving. Figure demonstrates that Llama3-SWE SW--RL capable of reasoning through self-reflection, exploring alternative approaches, and solving complex problems by breaking them down into smaller subtasks. In $3.5, we demonstrate that Llama3-SWFF. improves over even more out-of-domain tasks, including library use, code reasoning, and general language understanding.
Evaluation
3.1 Experimental setup
Training configs. Llama3-SWE-RL-700 is trained on top of Llama-3.3-1 1.ee--..-770--Istiiii (Dubey et al., 2024) using SWE-RL for 1,600 steps with a 16k context window. We use a global batch size of 512, sampling 16 rollouts from each of the 32 problems in every batch. For every global step, a single optimization step is performed using Adam (Kingma and Ba, 2017). We train our models on 512 NVIDIA H100 GPUs; training run takes approximately 32 wall-time hours.
Scaffolding. We have developed Agentless Mini on top of Agentless (Xia et al., 2024) as the underlying scaffold. Different from Agentless's multi-step localization, Agentless Mini focuses solely on file-level localization, delegating detailed reasoning to the repair step by providing the entire file contents in the input. This enables -llam3-SSFFat to do more reasoning during SWE-RL and simplifies the RL process to focus on only one issue-solving task. Despite this simplification. Llama3-SWE- can still seamlessly complete other pipeline steps, benefiting from out-of-domain generalization through RL. Furthermore, while Agentless only employs one reproduction test per issue for reranking, Agentless Mini can multiple reproduction tests, which has proven effective in our scaling analysis ($3.4). More details about Agentless Mini can be found in Appendix B.
Evaluation setup. We conduct evaluation on SWE-bench Verified (OpenAI, 2024). subset of SWE-bench with 500 human-verified problems that can more reliably evaluate models' capability in solving real-world software issues. In the main evaluation ($3.2), we generate 500 patches for each problem using 1. 1.0 temperature, and use the top 30 reproduction tests for execution and reranking. Ultimately, only the highest-ranked patch for each issue will be submitted for SWE-bench evaluation to calculate the pass @ score.
SFT baseline. To understand the advantages of SWE-RL, we also trained an SFT baseline, named for experiments in $3.3, $3.4, and $3.5. It is trained on top of Llama-3.3. Instruct (Dubey et al., 2024) using a mixture of synthetic code editing data, Llama 3 (Dubey al., 2024) coding SFT data, and Llama 3 general SFT data. The synthetic data is generated using an approach inspired by Magicoder (Wei et al., 2024), where high-quality PR data serves as the seeds for creating chain-of-thought and subsequent editing, as well as serving as the oracle for filtering. Llama-3.3-7088 Ins-truutttt is employed for this generation process. In contrast to our RL model, which only requires a seed PR dataset to trigger the RL loop, the SFT baseline needs synthetic data generation for chain-of-thoughts and additional data mix ensure the dataset diversity and model generalizability. More details are explained in Appendix C.
3.2 Main results
Table presents the pass@1 results on SWE--bench Verified for models that utilize open-source scaffolds, with models categorized by their size. From the table, we observe that Llama3-SWE- RL-70B achieves state-of ffte-offthhe-rt results among small and medium-sized language models (<100B) by resolving 41.0% of the issues. Additionally, all other open-source baselines we compare, such as Lingma-SWWe SWE-GPT (Ma et al., 2024), SWE-Gym (Pan et al., 2024), and SWE-Fixer (Xie et al., 2025), include distilled outputs from GPT-4o or (laude-3.5-Sonnel
the training data. contrast, Llama3-SWE-RR is trained solely with publicly available data through our reinforcement learning technique SWE-RL, without relying on any proprietary LLMs in the pipeline. Llama3-SWE-RL also sets a new record for Llama-based methods on SWE-bench. Additionally, we show that our RL-ed model significantly better than the SFT baseline in this end-to-end setup, indicating that SWE-RI. enhances the general reasoning ability of LLMs to solve issues.
5

---

Table 1: Main results on SWE-bench Verified. We include representative methods with open-source
scaffolds. The scores are either collected from the SWE-bench Leaderboard (Jimenez et al., 2024) or from the corresponding reference.
Model
Scaffold
SWE-bench Verified Reference
Model closed-source size
100B
GPT-4o
SWE-agent
23.2
Yang al. (2024b)
Mlaude3.5-SSoneet
SWE-agent
33.6
Yang et al. (2024b)
GPT-4o
Agentless
38.8
Xia et al. (2024)
ol-preview
Agentless
41.3
OpenAl (2024b)
DeepSeek-V31
Agentless
42.0
DeepSeek-Al (2024)
Claude-3.5-Somme
AutoCodeeRovr--22..
46.2
Zhang et al. (2024)
Claude-3.5-Somm
Tools
49.0
Anthropic (2024b)
DeepSeek-R1*
Agentless
49.2
DeepSeek-At (2025)
Claude-3.. Maaude-3.5-omeee 5-Sonnet
Agentless
50.8
Xia et al. (2024)
Claude-3.5 3.50e-35--oor S-onnet
OpenHands
53.0
Wang et al. (2024b)
Model size 100B
SWE-Llama-I3B 13B
RAG
1.2
Jimenez al. (2023)
SWE-Llama-7B
RAG
1,4
Jimenez et al. (2023)
LingaaSWWE-PPT-25
SWE-SynInfer
18.2
Ma et al. (2024)
Linga-SWWE-GPT-22
SWE-SynInfer
28.8
Ma et al. (2024)
SWE-Gym-32B
OpenHands
32.0
Pan et al. (2024)
SWE-Fixer-72B
SWE-Fixer
32.8
Xie et al. (2025)
Laam3-SWWE-SFT-700
Agentless Mini
36.2
This paper
Llama--WEE-RL-708
Agentless Mini
41.0
This paper
OOpen-souree Mixture-of- of-Experts model with 671B total and 37B active parameters
Table 2: Baseline comparison on SWE-bench Verified. In this experiment, we compare the repair-only performance of baseline LLMs by providing oracle localized files in the input context, without doing test generation and execution. We greedy decoding by default, but for 6ama--3337783 7.---tutnntt we include a 20-sample majority voting result at a temperature of 0.6 to improve formatting accuracy.
Model
Setting
Correct format Repair performance (oracle)
Llama-3.3 -nsttuutttt
Greedy decoding
12.2%
5.4
Llama-3.3- 3.0ra.3.--70-IIsttru
Majority voting
44.6%
16.6
(lama3-SWESSFT-7
Greedy decoding
96.2%
29.6
Llama3-SWE-RL-708
Greedy decoding
95.6%
34.8
3.3 Baseline comparison
To understand how much SWE-RL improves LLMs in solving sofware issues, compare Llama3- SWE-RI. with the corresponding Llama-3 and SFT baseline Table 2, using Agentless Mini as the underlying scaffold. In this experiment, we also evaluate on SWE-bench Verified but focus on the models' repair ability. To achieve this, we provide oracle files in the context and let the model generate a single repair edit using greedy decoding, without incorporating additional pipeline steps such as localization and test generation. The table reveals that the base Llama-3.3 model struggles to produce correctly formatted code edits, even when using 20- sample majority voting approach, where outputs with incorrect formats are pre-filtered. With SFT, most code edits generated by the language model are correctly formatted, and the repair performance shows significant improvement. However, Llama3-SWE-RI-
demonstrates even greater enhancement in repair capabilities, although its format accuracy is slightly lower than that of the SFT version. This indicates that SWE-RL. aids the LLM in better reasoning about issue solving and code editing.

---

3.4 Scaling analysis with more samples
Agentless Mini supports scaling both the number of repair samples and the number of generated reproduction tests. The difference in the number of samples may affect the reranking accuracy. In this section, we evaluate how the final pass@ @ performance on -WE-bench Verified scales with the two factors.
SWE- SWE-bench FF--bench Verified score
42
42
40
40.6
41.0
40.6
41.0
41.0
39.8
38
38
38.8
37.0
35.8
34
34
33.6
40
80
160
320
500
10
20
30
Number of repair samples
Number of test samples
Figure 4: Scaling analysis with more repair samples and more reproduction tests. The figure on the left illustrates the resolve rate on SWE-bbech Verified relation to the number of repair samples, while maintaining constant 30 test samples. Conversely, the figure on the right depicts the resolution rate as varies with the number of reproduction test samples, with a fixed 500 repair samples.
Figure shows that increasing both the number of repair samples test samples enhances per- formance SWE-bench. Notably, for repair samples, there is significant score increase from 33.6 to 40.0 when the sample size expanded from 20 to 160. However, beyond 160 samples, the improvement trend begins platcau, with scores only rising slightly from 40.0 to 41.0 the sample size increases to 320 and 500. Although the impact of adding more reproduction test samples is less obvious, there is still gradual score improvement from 38.8 41.0 the number of test samples increases up to 20. There no difference between 20 and 30 test samples, suggesting a performance saturation point has been reached.
3.5 Generalizability of RL
Table 3: Generalizability of Llama3-SWE-RL-TO beyond SWE-bench. This table compares Llama-3.3-70B-Insin the SFT variant, and the RL model on five out-of-domain tasks, highlighting RL improvements and SFT declines. All experiments are done in a consistent setting using zero-shot greedy decoding. We report the macro average category accuracy for MMLU and pass @ for the others. In MATH, we use simple-evals's "Answer: only the RL model consistently follows the format requirements, to relax the constraint to include "Aboxed...."
prompt format (OpenAI, 2024). However,
also report MATH (lenient)
Category
Benchmark
1lama-3.3-700 Instruct
Saima--SWE-SFT---EE
Llam3-SWEE-RL-708
Function coding
HumanEval+
76.2
73.2
79.9
Library use
BigCodeBench- (I)
28.4
25.7
28.4
BigCodeBench-Hoae (C)
29.1
24.3
29.1
Code reasoning
CRUXEval-I
60.5
68.4
71.6
CRUXEval-O
61.9
75.1
75.5
Math
MATH (strict)
63.2
54.0
73.7
MATH (lenient)
70,9
71.7
73.7
General
MMLU
86.49
85.26
86.82

---

Llama3-SWE-RL is only trained with SWE-RL on issue-solving data. This raises a natural ques- tion whether such domain-spest -ppecffc training harms the performance on other tasks. To address this, we conduct an experiment in Table 3, evaluating the L.LMs on five out-of-domain benchmarks, i.e. HumanEval+ (Chen et al., 2021; ; Liu et al., 2023) for function-Ievel code generation, Big- CodeBench (Zhuo et al., 2024) for practical code generation with library use, CRUXEval (Gu et al., 2024) for code execution reasoning. MATH (Hendrycks et al., 2021c) for mathematical reasoning, and MMLU (Hendrycks al., 202 1b) for general language understanding. We also include the SFT baseline, which is finetuned on the same 1lama-.33-70B-Imm
model using issue-solving data,
combined with general coding and dialog data.
From the table, it evident that model and the SFT baseline. There are notable improvements in CRUXEval and MATH, where significant reasoning efforts are required to arrive at the final answer. Through SWE-RI,, the model enhances its reasoning skills and dedicates more thinking effort to solving problems compared to other baselines. Although trained on single task, SWE-RL enables the model to generalize reasoning capabilities across various domains. In contrast, the SFT version, average, underperforms relative to the original model.
trained with RL. outperforms both its base
Overall, our results suggest for the first time that reinforcement learning on real-world software data like PRs enables the model to acquire generalized reasoning skills, whereas supervised finetuning steers the language model towards a specific task distribution, leading to performance declines on tasks with lower emphasis, even when a meticulously curated data mix is used.
Statistical significance analysis. While small absolute gains (1-2 percentage points) on HumanEval CRUXEval are, by themselves, unlikely to be statistically significant, we evaluate across multiple benchmarks and aggregate evidence, observing improvements that consistently favor RL. With Eval Arena (Wang al., 2024a), we indicate that improvements of > 0.8 percentage points on MMLU, 3 points on CRUXEval, and > 3 points on the full MATH dataset are significant on their own; taken together, these results achieve significance at the 0.05 level.
3.6 Reward ablation
According Equation (1), the reward design of SWE-RL allows different instantiations of the compare function. Throughout the paper, we adopt the sequence similarity between the predicted and the oracle patch, which is a continuous value from 0 to 1. We denote this type of reward as continuous. It is natural to compare this continuous reward with discrete reward, where the compare function outputs if the predicted and oracle patches exactly match each other, and otherwise. We trained a variant of Llama3-SW SEE--SEEERRR RL with the discrete reward function, using the same training setup in the continuous reward case.
0.0
Reward type Correct format
Repair (oracle)
Reward
Discrete 94.2%
29.0
Continuous 95.6% %
34.8
Continuous
Discrete
1.0
Training steps
Figure 5: Ablation on SWE-RL's reward functions and their training dynamics. We compare SWE-RL using the default continuous reward function against discrete reward. Repair (oracle) evaluates the repair-only performance using greedy decoding, with oracle files in the input context.
As shown in Figure 5, while the discrete and continuous reward functions lead similar format accuracy, the continuous reward is more effective in enhancing the repair performance. From the training dynamics, we can see that discrete rewards grow slower than continuous rewards. Additionally, the average discrete reward remains approximately zero upon the completion of training, meaning it struggles to obtain patches exactly matching the oracles. This is because real-world patches are highly diverse and often cannot be easily matched. The continuous reward function
8

---

better captures partial correctness and incremental improvements, allowing the model to learn more nuanced and effective repair strategies.
Related work
4.1 Language models for software engineering
Large language models (LLMs), trained with billions to trillions of code tokens, have demonstrated outstanding performance in wide range of coding tasks, including code generation (Chen et al., 2021; Li al., 2023; Rozière et al., 2023; Guo et al., 2024; Wei et al., 2024; Lozhkov et al., 2024; DeepSeek- Al et al., 2024), code optimization (Cummins al., 2024; Liu al., 2024a), program repair (Xia and Zhang, 2022; Xia et al., 2023b; Jiang et al., 2023; Wei et al., 2023), and software testing (Xia al,, 2023a; Deng al., 2023; Yuan et al., 2023; Schäfer et al., 2023; Lemieux al., 2023). Initially, researchers primarily focused on single-shot code generation tasks, such as function-level (Chen et al., 2021; Austin et al., 2021: Hendrycks al., 2021a; Li et al., 2022; Jain et al., 2024b), class-level (Du et al., 2023), and repository-level code completion (Ding et al., 2023; Liu et al., 2024b; Zhang et al., 2023). However, with the rapid development of LLMs, the performance on many popular single-shot code generation benchmarks like HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and EvalPlus (Liu al., 2023) has become saturated. Since the development of SWE-bench (Jimenez et al., 2023). which requires solving real-world GitHub issues, researchers start to work on improving L.MM'' real-world ussue-solving capability and have designed various scaffolds for SWE-bench. Two general types are (1) agentic scaffolds (Yang et al., 2024b; Wang et al., 2024b; Gauthier, 2024), where an LLM drives the decision- making process based on its past actions and observations through tool-based interaction with the environment; and (2) pipeline-based scaffolds (Xia et al., 2024; Örwall, 2024; Zhang et al., 2024), where an LLM goes through -deanndeeined stages to solve given issue. Generally, agentic methods are more general but require strong instruction-folloiin and capable LLMs to drive the autonomous process, and can computationally intensive due to multi-round interactions. In contrast, appline-bbsee approaches are more specialized but efficient, with focus on LLMs' pure code editing capability. Therefore, we designed our minimalist ppeeine-based scaffold, Agentless Mini, to focus the enhancements of
core code editing capabiltiy.
4.2 Training software agents
While existing scaffolds have successfully leveraged proprietary language models to tackle world software enginecring tasks, open models typically yield subpar results in these settings. Morcover, the most effective approach to enhancing real-world software engineering capabilities through training remains unclear. Recently, researchers have begun exploring the possibility of training open LLMs specifically for software engineering tasks, aiming to improve performance on benchmarks such as SWE-bench. For instance, Lingma-SWE-GPP (Ma et al., 2024) introduces 7B and 72B model variants that build on top of Qwen2.5-Coder-77 (Hui et al., 2024) and Qwen2.5-722 --2.255-72-Intttrre (Yang approach. SWE-Gym (Pan al.,
al., 2024a), using an iterative eyvelopment-proces--ceee 2024) presents the first open training environment for software engineering agents, significantly improving the performance of Qwen2.5-Coded 7B and 32B variants on SWE-bench. More recently, SWE--Fier (Xie et al., 2025) finetunes the Qwen2.5 base series, resulting in a 7B code retriever and 72B code editor focused on efficient issue resolution, achieving notable best@) improvements. Notably, all these works incorporate distilled samples from either G4T-4o or Claude-3.5 3.5-one- in their training data and are built upon Qwen2.5 models. Their training objectives are all based on supervised finetuning. On the contrary, -llma3-SWE-BR is based on Llama 3 (Dubey et al., 2024) and trained through reinforcement learning (RL) using SWE-RL. The seed dataset for RL is sourced exclusively from publicly available repositories, allowing Alama3-SWE-RR to self self-immron improve its issue-solving capabilities through the RL inscentive. Remarkably, Llama3-SWE-RI achieves the best performance among these models with 41.0% solve rate on SWE-bench Verified (OpenAI, 2024), demonstrating for the first time that L.MSs can already effectively address real-world issues through RL on real-world software artifacts.

---

5 Conclusion
We introduce SWE-RL, the first reinforcement learning (RL) approach to improve language models (LLMs) on software engineering tasks using software evolution data (e.g., PRs) and rule-based re- wards. The resulting model, -lama--SWE-RL-7000 human-verified
achieves 41.0 % solve rate on SWE-bench Ver- ified, human verified collection of high- nigh-quality GitHub issues. This performance is state-of-the-ne among medium-
models and comparable to many proprietary LLMs such as GPT-4o. While SWE-RL is specifically applied to the issue-solving task, Llama3-SWF-- has developed generalized reasoning skills through RI., demonstrating improved performance on out-of-domain tasks such as code reasoning, mathematics, and general language understanding. Overall, SWE-RL opens new direction for enhancing the software engineering capabilities of L.Ms through RL..
Limitations. Despite the promising results, our approach has several limitations. First, reward implementation compares the sequence similarity between the predicted and oracle patch rather than their semantic equivalence. This may prevent the policy LLM from exploring alternative, functional equivalent solutions. Additionally, in Agentless Mini, the localization process is simplified to mapping repository structures file paths, which lacks comprehensive context. Moreover, approach, Agentless Mini divides all steps into distinct inference stages. This "external structure" prevents the model from learning through interaction feedback and hinders its ability to consider the entire problem holistically.
pipeline- based
Acknowledgement
We thank Chris Waterson, Chris Cummins, and Volker Seeker for their assistance in debugging and testing Agentless Mini; Kunhao Zheng, David Zhang, and Taco Cohen, Jonas Gehring, Vegard Mella for their assistance in the online RL infra; Pierre Chambon, Mihir Sanjay Kale, Parth Thakkar, Michael Jiang, Sinong Wang, and Jingyue Shen for their involvement in the PR dataset discussion; Jannik Kossen for his helpful discussions in model training; Kamila Benzina, Rachel Kim, Megan Miller, Hannah Schubloom, and Madeline Hepler for their support in the review process; Steven Xia and Yinlin Deng for their aid in setting up Agentless; Jiawei Liu, Yifeng Ding, Terry Yue Zhuo, and Naman Jain for their valuable discussions; Albert Örwall for their help in resolving issues with Moatless EvalTools.
References
Anthropic. Claude 3.5 sonnet model card addendum. Claude-3.5 Model Card, 2024a.
Anthoopic. Raising the bar SWE-bench Verified with Claude 3.5 Sonnet. https://www. anthropic cetchropi.com/reesete wwt.c///eeeeccseeeeeeeeeeeeceeee bence benhs--eee-s--eee,, sonnet, 2024b.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. attps://arzir org/ abs/2 07732..
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. The Eleventh International Conference on Learning Representations, 2023. https : /pwp//veeeeeeeeeee///ee/ee
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph. Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021.
10

---

Chris Cummins, Volker Seeker, Dejan Grubisic, Baptiste Roziere, Jonas Gehring, Gabriel Syn- naeve, and Hugh Leather. Meta large language model compiler: Foundation models of compiler optimization, 2024. https://arxiv.. Strrg/bbs/44000 .02.00002524.
DeepSeek-AI. Deepseek-v3 technical report, 2024.
/tt////xii.orr/aa///2/223 19437.
DeepScek-AL. Deepscek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, aawpaaa////111/1
2025. pettps:/arrivg.
DeepSeek-Al, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang. Peiyi Wang. Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang. Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, and Wenfeng Liang. Deepseck-coder-v2: Breaking the barrier of closed-source models in code intelligence, 2024. https: http///xxvvv..rr/eo aba/ aba/2406.... 114406.11300 11931.
Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models, 2023.
Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Kr- ishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. Crosscodee- val: A diverse and multilingual benchmark for cross-file code completion. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
https://openreviiew wtt..eee.oorrro//dddoooor.....
Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. Classeval: A manually-crafted benchmark for evaluating Ilms class-level code generation. arXiv preprint arXiv:2308. 06811, 2023.
Meta: Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, and Angela Fan et al. The Ilama 3 herd of models, 2024. https: // /tr//rrivv..// orgg/ abs a2407 21783.
Paul Gauthier. Aider is ai pair programming your terminal. https://aider: chat/, 2024
Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and Gabriel Synnaeve. Rlef: Grounding code lims in execution feedback with reinforcement leaming, 2025. attps://arxiv org/ prg//bs//a 2410. 02089.
GitHub. Github rest api documentation. https: //wp////oo.ooeeee/////c www.:iggttub.cccmeoeee wtt.ubb..coomaarrts/aaaaaasrtrrr 2022-11-28, 2022. Accessed: 2025-02-24.
GitHub. Github. https: http::///tthbccoommm 2025.
Ilya Grigorik. Gh archive. https: : http://wwwwwg..hhh........... orgp: 2025. Accessed: 2025-02-23.
Alex Gu, Baptiste Roziere, Hugh James Leather, Armando Solar- Gabriel Synnaeve, and Sida Wang. CRUXEval: A benchmark for code reasoning, understanding and execution. In Proceedings of the 41st ICML, volume 235 of Proceedings of Machine Learning Research, pages 16568-16622 PMLR, 21-27 Jul 2024. https: htt:://///oceeeees/// mlr. press/v235 htee//u24ccctccc html.
Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseck-coder; When the large language model meets programming - the rise of code intelligence, 2024.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps, 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 202lb. attps://openree net/fooum?idd-d7K3
11

---

Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021c. https: http///ppeerevveeeeeee wwtpaarrviee.eee//oorn/iieee.e.e
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang. Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang. An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical report, 2024. https:/ arxxiv. org/ abs/2409. 12186.
Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. Testgeneval: A real world unit test generation and test completion benchmark, 2024a. https://arxii org/ abs/2410. 00752.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024b.
Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. Impact of code language models on automated program repair, 2023.
Carlos E. Jimenez, John Yang, Alexander Wettig. Shunyu Yao. Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2023.
Carlos E. Jimenez, John Yang. Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench leaderboard. https: :/wp:://www... hwtp://wwwenc....ommm com, 2024. Accessed: 2025-02- 04.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization, 2017. https:
//arxiv. org/ abbs/1122 6980.
Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In 2023 JEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 919-931. IEEE, 2023.
Raymond Li, Loubna Ben Allal, Yangtian Zi Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu Benjamin Lipkin, Muhtasham Oblokulov. Zhiruo Wang. Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf. Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023.
Yujia Li, David Choi, Junyoung Chung. Nate Kushman, Julian Schrittieeerr, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-leve code generation with alphacode. Science, 37 1092- December 2022. ISSN 1095 92033. doi: 10. 2e11112/ieeneaaaee abq 158. http://dx. doi org/ 10. ab
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and I Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. htttp:://openreviee net//torumidd=112260000
Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. Evaluating language models for efficient code generation. In First Conference on Language Modeling, 2024a.
htttps://openreviii htwpaa/wwwwe..eeeeooororraaaaaaa
12

---

Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-Ieve code auto-completion systems. In The Twelfth International Conference on Learning Representations. 2024b. https://openreeiert
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu. Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generation, 2024.
Yingwei Ma, Rongyu Cao, Yongchang Cao, Yue Zhang, Jue Chen, Yibo Liu, Yuchen Liu, Binhua Li, Fei Huang, and Yongbin Li. Lingma swe-gpt: An open deweloppment-processsceoo centric language model for automated software improvement, 2024. https://arxiv.orr//aaa /221/00002221122.......
OpenAl. Gpt-4o system card, 2024a.
OpenAI. Openai ol system card, 2024b. https///arxx//.or//////2222222
OpenAl. simple-evals.
2024. Accessed: 2025-
02-23.
OpenAl. Introducing swe-bench verified.
https://opemai.com/iinn
2024. Accessed: 2025-02-04.
Jiay Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym, 2024. https://arxiv org/
abs/2412.22133
John W. Ratcliff and David E. Metzener. Pattern Matching: The Gestalt Approach. Dr. Dobb's Journal, page 46, July 1988. https://www.drdee dwwobbs..bb..... com/database/ pattern- sentern-matching-ther sts.eeeesssatttappprccaaaa11111
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2023.
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. An empirical evaluation of using large language models for automated unit test generation. JEEE Transactions Software Engineering, 2023.
John Schulman. Approximating kl divergence. 2020. Accessed: 2025-02-22.
appprro html,
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024.
org/ 03300.
Sida I. Wang, Alex Gu, Lovish Madaan, Dieuwke Hupkes, Jiawei Liu, Yuxiang Wei, Naman Jain, Yuhang Lai, Sten Sootla, Ofir Press, Baptiste Rozière, and Gabriel Synnaeve. Eval-Arena: noise and errors on I1m evaluations.
com/crux wwwux--vallevvllll arena, 2024a.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for software developers generalist agents, 2024b. https: /////aaxvvv/v org/ abs/2407 16741.
13

---

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi. Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems; 2022. https:/ http://pppeeeeeeeeeeeeeeeeeeeeee wwt./eeeuniidd_aaaeee _
Yuxiang Wei, Chunqiu Steven Xia, and Lingming Zhang. Copiloting the copilots: Fusing large language models with completion engines for automated program repair, 2023.
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering code generation with OSS-instrucc Proceedings of the 4 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 52632-52657. PMLR, 21-27 Jul 2024. htttss:/prrceeediee
mlr press/v235/ //a//////2hh.... html.
Chunqiu Steven Xia and Lingming Zhang. Less training, more repairing please: Revisiting automated program repair via zero-shot learning, 2022.
Chanqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. Universal fuzzing via large language models, 2023a.
Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated program repair in the era of large pre-trained language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1482-1494, 2023b. doi: 10. 11100/CCSE44661120030000011
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents, 2024. https: wtw.loog////aaaaaeeeeeee awwp/b//./////////
Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen. Swe-fier: Training open-source lims for effective and efficient github issue resolution, 2025.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2024a.
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In The Thirly-eighth Annua Annual Conference on Neural Information Processing Systems, 2024b. https: // //opp/neevew.nnee/// wwwp://wwwwewttuummmeeeeee dit.ummmtddxxppttt11
John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang. Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. Swe-bench multimodal: Do ai systems generalize to visual software domains?, 2024c.
https://arrs atgg/abb///11 038999
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long thain-of-thought reasoning in Ilms, 2025. https: arxiv. org/ atse//bs/220.... 03373.
Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. No more manual tests? evaluating and improving chatgpt for unit test generation. arXiv preprint arXiv:2305.04207. 2023.
Huaye Zeng. Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder rl via automated test-case synthesis. ArXiv, 2502.01718, 2025.
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. Repocoder: Repository-Ievel code completion through iterative retrieval and generation, 2023.
Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement, 2024. https:/ //apsi////xxvvv// arxii org/ abss/404, 05427.
14

---

Wenting Zhao, Nan Jiang. Celine Lee, Justin T Chiu, Claire Cardie, Matthias Gallé, and Alexander M Rush. CommitO: Library generation from scratch, 2024. https://arxiv.ot wtwpp//...g/b/b///ee/eeee/e htt//bb//24122 01769.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877 2024.
Albert Örwall. Moatless tools. https: hnte:://iithuu github. hotp////oooaa////oteeeessooooo.o 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We listed the core contributions and key results in the last paragraph of the introduction section ($1).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations of work in $5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those
discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't tt acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification: Our work studies improving LLMs' reasoning ability through reinforcement learning on open software data. Therefore, theoretical results are not applicable here. Instead, we performed comprehensive set of evaluations ($3) in an empirical fashion.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We detailed our data curation (Appendix A), technique ($2), and experimental configurations ($3). We also included the reward implementation and evaluation pipeline in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or way reproduce the model (e.g., with an open-souree dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
17

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We included the reward implementation and evaluation code in the supplemen- tal material, along with detailed instructions for using the artifact to ensure transparency and reproducibility of our results. At this time, we cannot share the training data training pipeline due to privacy and proprietary considerations. However, once the necessary reviews are completed and associated risks addressed, we will make these resources available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/ for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We detailed the configurations and rationales for model training $3.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We acknowledge that we did not include error bars for all evaluations on SWE- bench. This is because SWE--benh comprises real-worke software engineering problems, and each instance is very expensive to run. Also, we tried to follow the design of SWE-bench where all accepted submissions are evaluated using single attempt per example.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the emror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, one should state The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We reported our compute configurations in $3.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:///eerriiipeee httpp/pp....pppppeccccccceetiiie
Answer: [Yes]
Justification: We have read the NeurlPS Code of Ethics and believe our work does not violate the terms.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethies.
The authors should make sure to preserve anonymity (e.g., g. if there is special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our technique neutral in implying clear positive or negative impacts on society.
19

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is needed point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve the release of models datasets that pose high risk of misuse, so safeguards are not applicable in this context.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited the datasets and models for training, synthetic data generation, and evaluations, and specified their versions to our best efforts. We also licensed our code in the supplementary material.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
20

---

The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi th thoode. com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged reach out the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We clearly described the license of our asset and respected the license of all derived assets in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: LLMs are used to fix grammar mistakes and awkward phrasing in writing.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: / wt////eerippp.//ee cc/Conferrecces//000 2wtp:/2222222222222 /LL)) for what should or should not described.
22

---

A
Raw pull request data curation
GitHub
clones
Download
Aggregate and decontaminate
Prodict but unchanged files
Ei
Filter
GH Archive
GitHub events
Self-contained PRs
with relevant files
Raw PR dataset
Figure 6: Overview of SWE-RL's 's raw pull request data curation process. The collected git clones and GitHub events are transformed into self-contained PR instances via decontamination, aggregation, relevant files prediction, and filtering.
Figure 6 provides a high-level overview of our process for curating the raw PR dataset for Llama3- SWE-RL. In the following paragraphs, we detail each step of the curation process. During data processing. we exclude all the repositories used by SWE-bench (Jimenez et al., 2023) to prevent data contamination.
GitHub events and clones. The goal of this stage is to recover all pull request details that human developers can inspect on GitHub. To achieve this, we need two sources of information: (I) all events that occur within a PR and (2) the source code of a repo before the changes introduced by the PR merged. We derive all GitHub (GitHub, 2025) events from GHArchive (Grigorik, 2025), which contains all activity events data from GitHub. Our collection includes all GitHub events from Jan 1, 2015 to Aug 31, 2024.
To obtain source code, since pull requests often occur at different commit stages of a repository, we opt use git clone to retrieve the entire repository with its commit history, rather than relying the GitHub API (GitHub, 2022) to download specific code snapshots. Eventually, we successfully cloned and processed 4.6M repositories.
PR data aggregation. The collected events and git clones are disparate entities that require further processing before they can be used for training. At this stage, we focus on each PR individually and aggregate all pertinent information associated with it. This includes mentioned issues, user discussions, review comments, initial code contents, and subsequent commits and code changes.
First, we keep only merged PRs and gather all related conversational events for each PR, sorting them in chronological order. Next, using the base_ _omeiittottttt and _ead_commis hashes of a PR, we retrieve the contents of all modified files indicated by its patch at the merge base of the two commits. The reason is that many PRs aim to merge back into the main branch, which may have undergone changes since the PR was created. By considering the merge base as the actual starting point for a developer working on the PR, we can more accurately understand the context of the changes. We save all intermediate commits and code changes made between the merge base and the head commit. Additionally, we extract the complete patch representing cumulative changes from start to finish. Finally, we scan each aggregated PR to identify patterns that resemble issues, and associate the matched issues with the corresponding PR. In the end, we have 24M aggregated PR instances.
Relevant files prediction. Currently, each pull request includes only the code files that have been modified. In our earlier experiments, we noticed that this approach let LLMs learn a bias: the model consistently generated edits for every code file presented and was unable to handle noisy files presented in the context. This issue was also mentioned in the finetuning experiment discussed in the SWE-bench paper (Jimenez et al., 2023). To mitigate this problem, we prompt Llama-3.1 70B-Instruct (Dubey et al., 2024) to generate a list of relevant but unmodified files given each PR description and the changed files, and include the contents of these files our final dataset.
Data filtering. GitHub PRs can be quite noisy, so we implemented various filtering strategies to climinate potentially harmful PRs. In designing these filtering rules, our goal is maximize the recall of high-quality PRs while permitting a certain level of noise. First, we remove the bot-generated PRs whose title, description, or username contains keywords "[bot]", "dependabot", "renovate", "bump". or "automerge". Also, we remove PRs with empty changes or with extremely large number of changes (e.g., in some PRs, the developer uploaded directory of data files by mistake). Additionally, we implemented a more tne-grained set of filters used in CodeLlama (Rozière et al., 2023) to examine each code change hunk. We then removed any PRs where these filters flagged all code changes. For example, this can exclude PRs with only lock file changes or version updates. Finally, this gives us around M unique PR instances. Before applying reinforcement learning, we select 273k high-quality PRs from these raw PRs, as discussed in $2.
23

---

B Agentless Mini
Localization and repair
Localize
Repair
Issue/ repo structure
Relevant files
Patches
Reproduction tests generation and selection
2 +
Predict test
Generate tests
Execute filter
Issue repo structure
Relevant test file
Test samples
Repraduction tests
Reranking
Regression selection
Execute & select
passing tests
Execute
1L
Rerank
Issue codebase
Regressioon
Execution
Selected patch
Figure 7: The Agentless Mini scaffold. The design emphasizes easy decomposition, parallelization, and scalability.
In addition to a model proficient in code editing, effectively tackling software engineering tasks, such those found in SWE-bench (Jimenez et al., 2023), also requires robust scaffold. Agentless (Xia et al., 2024) is one of the state-of-the-ars scaffolds for SWE-bench at the time of writing. Building upon Agentless with various simplifications and enhancements, we developed Agentless Mini, framework that prioritizes straightforward component decomposition, parallelization, and scalability, With Agentless Mini, each step's inference execution compute can be independently scaled to enhance SWE-bench performance. In Figure 7, we present detailed illustration of Agentless Mini's working principles. The following paragraphs will elaborate on each step and highlight the differences from Agentless.
Localization and repair. For localization, we employ prompting- -based approach that enables the model to predict relevant file paths based on a given issue and the repository's structure. Unlike Agentless, which involves two additional detailed steps to identify related elements and files, as well as separate embedding model, Agentless Mini simplifies the process. It generates multiple samples of potentially problematic files from the model and consolidates them into unique sets for repair.
During the repair phase, the LLM is conditioned on the full content of the files to predict search/replace edits. We generate multiple repair samples from different location sets, ensuring comprehensive exploration of the patch search space.
Reproduction tests generation and selection. Agentless samples reproduction tests for patch selection. Initially, multiple reproduction tests are generated based on an issue description, and one majority sample is selected after filtering. These tests must have distinct logic to output "Issue reproduced" " and "Issue resolved, when the issue is reproduced or resolved, respectively. They filtered based on whether they correctly output "Issue reproduced" when executed in the original codebase. Agentless Mini enhances this pipeline with two key improvements. First, instead of relying solely the issue description, the model retrieves relevant test file guide test generation. Additionally, rather than selecting just one majority sample, Agentless Mini allows for the selection of multiple top test samples based on voting results. In our evaluation, using more test samples has proven beneficial for reranking ($3.4).
Regression tests selection. We select regression tests in the same manner Agentless. Initially, we gather all passing tests for each issue by executing the code before any modifications. This step does not require model inference and needs to be performed only once. Subsequently, additional, optional inference step is conducted to filter out passing tests that are supposed to fail after the issue is fixed. The rest of the tests will be marked as regression tests.
Agentless Mini utilizes both regression and reproduction tests for reranking. For each issue, every applied patch is executed against the regression tests. The patches that result in the mini- mum number of existing test failures are selected. For each issue, we choose the top-N reproduction tests and run each patch against these tests. If the execution outputs "Issue resolved", we mark the patch passing this test. We then adopt the dual execution agreement objective from CodeT (Chen et al., 2023). Specifically, patches P that pass the same set of reproduction tests T are denoted as
Reranking,
24

---

consensus group. Each consensus group is scored using the formula [PI x ITI.. With this objective, consensus groups with patches passing more tests receive higher scores. Additionally, groups where more patches pass the tests are scored higher, although passing more tests is prioritized over having more patches. Finally, we identify the consensus group with the highest score and select the best patch from this group using majority voting.
C Synthesizing supervised-finetunt data
Select high-quality PRs
Synthesize localization code-editing data
E SFT samples
uilter ground truth
PRs
SFT seeds
SFT dataset
Figure 8: Synthetic data pipeline for constructing SFT data. We start by collecting high-quality seed PRs using heuristics, then generate synthetic localization and code-editing samples, and finally use the ground-truth edited files and patches to filter out incorrect samples.
Figure 8 shows method of generating synthetic supperiseedfinettninng (SFT) data. The data generation pipeline inspired by Magicoder (Wei et al., 2024), where the OSS-Instruct technique generates high-quality code instruction data from open-source seed snippets. We apply similar methodology to generate both fault localization and code editing data using high-quality PR seeds.
Collecting high-quality seed PRs. To begin with, we extract high-quality PR seeds from the raw dataset we collected, as detailed Appendix A. These seeds are chosen based on specific heuristics. For example, PR instance should include at least one linked issue, the issue should describe a bug fix request, and the code changes should involve programming files.
Localization and editing data synthesis. We adopt Llama-3.3-700 I-Isrnttt (Dubey et al., 2024) for data synthesis. For localization data, we prompt the model with the issue description, repository structure. and the paths of edited and relevant files as hints. We then ask the model to identify the relevant files for modification or review by generating a thought process, followed by a prioritized list of file paths. During filtering, we ensure that the model's 's response includes all files that are genuinely edited in the PR, and these files should be prioritized the ranking.
Similarly, in terms of code editing, we synthesize code edits for given issue, in search/replace format (Xia al., 2024), by providing the ground-truth PR and patch guidance Llama-3.3-70B- Instruct. During the filtering process, we ensure that all search/replace blocks adhere to the correct format and that all search paths and blocks can be accurately matched within the input files' context.
SFT baseline. As discussed in $3.1, we meticulously constructed Llama3-SWEL SWE-SFT-70B as a strong SFT baseline. This SFT baseline is trained 16k context window for 2B tokens, where the training data consists of mix of the aforementioned synthetic localization and editing data, as well as coding and general SFT datasets from Llama (Dubey al., 2024).
D Complete prompt
The following is a complete version of Figure 2:
PROMPT_TEMPLATE """<(begin_of text st idd///ys////e/ end_header_idd >
user will ask you to solve a task. should first draft your thinking process (inner monologue). Then, generate the solution.
Your response format must follow the template below: <think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct solution. </think>
<solution>
25

---

Final solution presented to the user. </solution><
id hea
We are currently solving the following issue within our repository. Here the issue text:
BEGIN ISSUE
iprrolem_statemeee END ISSUE
Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
BEGIN FILE
{content)
END FILE
Please first localize the bug based on the issue statement, and then generate #SEARCH/REPLACE+ edits to fix the issue.
Every #SEARCH/REPLACE+ edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. contiguous chunk of lines search for the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE
Here is example:
python ### stccheeb/llas/aap.ppp
<<<<<<< SEARCH
from flask import Flask
import math
from flask import Flask
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> REPLACE
Please note that the SSEACCH/REPLCCE: edit REQUIRES PROPER INDENTATION.
you would like to add the line that out, with Wrap each #SEARCH/REPLACS
print(x)', you must fully write those spaces before the code!
edit in code block shown in the example above. If you have multiple #SEARCH/REPLACE* edits, use a separate code block for each heec//eot_id/>>> start_haaeerr_t___t >assistant<] end_header_idd>
26


Reviews:
[{"review_id": "cMlmrpLgxP", "reviewer": "Reviewer_n4Dr", "summary": "This paper proposes SWE-RL, a novel reinforcement learning (RL) approach that trains large language models (LLMs) to solve real-world software engineering (SE) tasks using open-source software evolution data, i.e., GitHub pull requests (PRs). Unlike previous RL methods that focused on competitive programming or math, SWE-RL targets real-world issue resolution, leveraging rule-based rewards derived from patch similarity between model-generated and human-written code. The authors train Llama3-SWE-RL-70B using this approach and evaluate it on SWE-bench Verified, where it achieves a 41.0% solve rate, surpassing all open-source LLMs under 100B parameters and rivaling proprietary models like GPT-4o. Surprisingly, this RL training also improves general reasoning ability across out-of-domain tasks (e.g., math, MMLU), even outperforming supervised fine-tuning (SFT) baselines trained on more diverse datasets", "questions": "(1) Have the authors considered semantic diffing or static analysis tools (e.g., AST comparison or test outcome delta) for more robust rewards?\n\n(2) Could test failures or runtime logs be incorporated as feedback in future RL training?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n(1) Novel direction for RL training: SWE-RL extends RL training to software evolution data—a previously untapped but richly structured resource. It moves beyond execution-based or synthetic RL setups.\n\n(2) Well-motivated reward function: The reward uses difflib-based patch similarity, allowing for partial credit and encouraging incremental improvements, which better reflects real-world issue resolution.\n\n(3) Good performance and comprehensive experiments: Achieves the best performance among medium-sized open models (41.0%), without using proprietary model outputs for supervision. The paper conducts comprehensive experiments including rigorous baselines (SFT and original model), scaling studies, reward ablation, and cross-domain generalizability testing.\n\nWeaknesses: \n\n(1) Reward function limitation: As the author said in \"Limitations\", the reward is based on string-level patch similarity, not semantic equivalence. This penalizes functionally correct but syntactically different fixes, potentially limiting solution diversity.\n\n(2) Limited interactivity: The pipeline is not interactive or agentic—it lacks tool use, exploration, or test-driven feedback during training. This may limit the model’s ability to generalize to more autonomous agent settings.\n\n(3) Single-task RL training: The model is only RL-trained on one task (issue resolution). While generalization is observed, multi-task RL training might further enhance its capabilities.\n\n(4) The training cost is too high, which requires 512 H100 GPUs for ~ 32 wall-time hours, which may limit accessibility and reproducibility despite using open-source models.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer n4Dr, we deeply appreciate your insightful feedback and suggestions for our work. In our responses below, we address each primary question (denoted as Q) and comment (denoted as C). Additionally, we will revise our paper to incorporate editorial suggestions. Should there be any misunderstandings of the questions, please kindly let us know; we are eager to communicate with you throughout the discussion period.\n\n> Q1: Have the authors considered semantic diffing or static analysis tools (e.g., AST comparison or test outcome delta) for more robust rewards?\n\nGreat point. We applied basic patch normalization (e.g., header and whitespace removal) before comparison, but not at the AST level. Regarding test outcomes, we conducted an additional experiment incorporating unit-test results into reward calculation. As detailed in Q2, we also enabled the model to interact with the environment using tools such as Bash and an editor. Due to time constraints, this experiment was performed with Llama-3.1-8B-Instruct. The table below shows pass@1 on SWE-bench Verified comparing three settings: (1) the baseline Llama-3.1-8B-Instruct, (2) RL with a binary execution-only reward (fail = 0, pass = 1), and (3) RL with execution reward combined with the SWE-RL reward. All models were trained for 400 RL steps on the SWE-Gym [1] dataset.\n\n|Setting|Pass@1|\n|-|-|\n|Baseline|0.8%|\n|Execution-only|11.0%|\n|Execution + SWE-RL reward|**14.2%**|\n\nBoth reward signals significantly improve over the baseline, and the SWE-RL reward complements execution-based rewards, aligning with our insight that software issues often require contextualized fixes, consistent with the Plastic Surgery Hypothesis [2] in program repair. While execution-based methods require substantial data collection in executable environments, the SWE-RL reward scales to large, open software corpora.\n\n> Q2: Could test failures or runtime logs be incorporated as feedback in future RL training?\n\nAbsolutely! In the Q1 experiments, we allowed the model to issue Bash commands freely when solving issues. Although not strictly enforced, the model could execute tests and use them as feedback to improve its solutions. In the table below, we divide trajectories generated during RL by whether they observed and fixed test errors, and we report their pass@1.\n\n|Trajectory Pattern|Pass@1|\n|-|-|\n|No error fixing|22.4%|\n|Fixing test errors|**25.8%**|\n\nA concrete example of the LLM leveraging runtime logs for better issue solving:\n\n```\n...\n[assistant]\nNow that we've updated the import statement, we should be able to run the test again.\n\nLet's try running the test again to see if the import issue is resolved.\n\n<tool: bash>\npython -m unittest /testbed/tests/test_cognitoidp_models.py\n</tool>\n</bash>\n\n[user]\nTraceback (most recent call last):\n  File \"<frozen runpy>\", line 198, in _run_module_as_main\n  File \"<frozen runpy>\", line 88, in _run_code\n  File \"/opt/miniconda3/envs/testbed/lib/python3.12/unittest/__main__.py\", line 18, in <module>\n    main(module=None)\n  File \"/opt/miniconda3/envs/testbed/lib/python3.12/unittest/main.py\", line 104, in __init__\n    self.parseArgs(argv)\n  File \"/opt/miniconda3/envs/testbed/lib/python3.12/unittest/main.py\", line 153, in parseArgs\n    self.createTests()\n  File \"/opt/miniconda3/envs/testbed/lib/python3.12/unittest/main.py\", line 164, in createTests\n    self.test = self.testLoader.loadTestsFromNames(self.testNames,\n                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/miniconda3/envs/testbed/lib/python3.12/unittest/loader.py\", line 207, in loadTestsFromNames\n    suites = [self.loadTestsFromName(name, module) for name in names]\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/miniconda3/envs/testbed/lib/python3.12/unittest/loader.py\", line 137, in loadTestsFromName\n    module = __import__(module_name)\n             ^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/testbed/tests/test_cognitoidp_models.py\", line 5, in <module>\n    class TestCognitoResourceServer(unittest.TestCase):\n  File \"/testbed/tests/test_cognitoidp_models.py\", line 6, in TestCognitoResourceServer\n    @mock_cognito_idp\n     ^^^^^^^^^^^^^^^^\nNameError: name 'mock_cognito_idp' is not defined. Did you mean: 'mock_cognitoidp'?\n\n[assistant]\nThe error message indicates that `mock_cognito_idp` is not defined, even though we've already updated the import statement to `mock_cognitoidp`.\n\nIt looks like there's another issue with the usage of `mock_cognito_idp` in the `TestCognitoResourceServer` class. We should actually use `@mock_cognitoidp` instead of `@mock_cognito_idp`.\n\nLet's update the test file to use `@mock_cognitoidp` instead.\n\n<tool: edit>\n/testbed/tests/test_cognitoidp_models.py\n<<<<<<< SEARCH\n    @mock_cognito_idp\n=======\n    @mock_cognitoidp\n>>>>>>> REPLACE\n</tool>\n\n[user]\nFile /testbed/tests/test_cognitoidp_models.py has been successfully updated at line 6 and now contains 22 lines.
\n\n[assistant]\nNow that we've updated the decorator to use `mock_cognitoidp` correctly, we should be able to run the test again.\n\nLet's try running the test again to see if the usage issue is resolved.\n\n<tool: bash>\npython -m unittest /testbed/tests/test_cognitoidp_models.py\n</tool>\n</bash>\n\n[user]\n.\n----------------------------------------------------------------------\nRan 1 test in 0.038s\n\nOK\n\n...\n```\n\nThese results indicate that incorporating tests improves issue solving, and we will include this valuable idea in future iterations.\n\n> C1: Reward function limitation: ...the reward is based on string-level patch similarity, not semantic equivalence. This penalizes functionally correct but syntactically different fixes, potentially limiting solution diversity\n\nWe want to kindly highlight that the main novelty of SWE-RL lies in the insight that real-world software bugs, unlike open-ended code generation, often follows constrained, localized patterns, aligning with the plastic surgery hypothesis [2] in program repair. The difflib-based reward can efficiently capture partial correctness of code changes. Crucially, this reward is what makes large-scale training possible; execution-based rewards require expensive environment setup and heavy data curation [1], limiting scalability. SWE-RL unlocks scalable, generalizable RL on real-world software data for the first time.\n\nIn the meantime, test-based reward is never perfect. In certain cases, it can be worse than similarity metrics where incorrect patches pass all the tests but miss the true intention of the problem. This claim is well supported by the literature. For example, EvalPlus [4] shows that the insufficient tests in the original HumanEval and MBPP benchmarks lead to many erroneous code solutions being accepted. Furthermore, [5] shows that the insufficient tests in SWE-bench impact 40.9% of SWE-Bench Lite and 24.4% of SWE-Bench Verified submissions. Below is an example illustrated in the paper, where the original issue requires a function that computes polynomial fits for data, to handle missing data in the inputs x and y. However, the original test case only considers scenarios where both x and y have missing data. The generated patch, while passing the test, fails to handle additional cases where only one input has missing data.\n\n```\n# Insufficient test\ndef test_missing_data(self, df):\n  groupby = GroupBy([\"group\"])\n  df.iloc[5:10] = np.nan\n  res1 = PolyFit()(df[[\"x\", \"y\"]], groupby, \"x\", {})\n  res2 = PolyFit()(df[[\"x\", \"y\" ]].dropna(), groupby , \"x\", {})\n  assert_frame_equal(res1, res2)\n\n# Incorrect patch that passes the test\ndef _fit_predict(self, data):\n  y = data[\"y\"].dropna()\n  x = data[\"x\"].dropna()\n  if x.shape[0] != y.shape[0]:\n    raise ValueError (\"x and y must have the same number of non - missing values \")\n  if x.nunique() <= self.order:\n    # TODO warn ?\n    xx = yy = []\n```\n\nOur insights are also backed by the results we showed in Q1 and Q2, where combining SWE-RL reward can surpass execution-only signal in a multi-turn RL setup.\n\n> C2: Limited interactivity: The pipeline is not interactive or agentic—it lacks tool use, exploration, or test-driven feedback during training...\n\nThank you for the thoughtful point. Our design choice was to prioritize scalability: SWE‑RL trains on large, real‑world repos without execution environments. The reward is orthogonal to interactivity; as Q1–Q2 show, adding it to agentic, tool-using RL improves over execution‑only signals. We view the practical recipe as hybrid, first training at scale with SWE‑RL, then doing agentic RL on executable subsets with test‑driven feedback. We’ll clarify this trade‑off and our roadmap in the paper.\n\n> C3: Single-task RL training: ...While generalization is observed, multi-task RL training might further enhance its capabilities.\n\nGreat suggestion! We plan to explore multi-task RL training in future work to further understand the capability transfer and generalization.\n\n> C4: The training cost is too high, which requires 512 H100 GPUs for ~ 32 wall-time hours, which may limit accessibility and reproducibility despite using open-source models.\n\nWe acknowledge the relatively high training cost. However, this cost is expected for doing RL at scale on software engineering tasks and remains affordable for industry labs (e.g., agentic systems like Kimi K2 [3] operate with over 10k parallel containers during training, substantially larger GPU and CPU cost). To ensure reproducibility, we have described our training hyperparameters clearly in the paper. We’ll also open source the pipeline, model, data, and also all evaluation results after careful privacy reviews.\n\n[1] Pan et al. Training Software Engineering Agents and Verifiers with SWE-Gym.\n\n[2] Barr et al. The Plastic Surgery Hypothesis.\n\n[3] Kimi Team. Kimi K2: Open Agentic Intelligence.\n\n[4] Liu et al. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation.\n\n[5] Yu et al. UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench."}, {"review_id": "GGMuGW1Pgp", "reviewer": "Reviewer_ZGKJ", "summary": "This paper proposes SWE-RL, a reinforcement learning framework to fine-tune Large Language Models (LLMs) on GitHub issue–patch data. The authors train a 70B Llama3-based model using a character-level difflib reward and the GRPO algorithm, achieving 41.0% on the SWE-bench Verified benchmark—a new state-of-the-art for open-source models of this size. The authors further claim that this training method improves the model’s general reasoning ability, as evidenced by improvements on out-of-domain tasks including MATH, HumanEval+, and MMLU.", "questions": "1. Your central claim is that reinforcement learning on software repair tasks leads to improved general reasoning. However, this remains an observational finding. Could you clarify what specific mechanism or training dynamics enable skill transfer from patch generation to tasks like MATH or HumanEval+?\n\n2. The RL-trained model is evaluated using a detailed chain-of-thought (CoT) prompt that could itself induce improved behavior. To attribute improvements to RL rather than prompting, did you evaluate the SFT baseline using the exact same prompt as used during RL training and evaluation? If not, how can you isolate the effect of RL from the prompt’s contribution?\n\n\n3. The difflib-based reward function operates on surface-level textual similarity and is agnostic to semantic correctness. Have you encountered examples where high-reward patches are semantically incorrect or logically invalid? How do such cases affect the interpretation of “improved reasoning”?\n\n4. Your RL formulation treats patch generation as a single-step decoding task, without intermediate actions or sequential planning. How does this setting support the emergence of non-trivial reasoning strategies, as opposed to shallow pattern recognition?\n\n5. The observed gains in task accuracy are not sufficient to support the claim that RL improves reasoning. To substantiate this, please provide concrete behavioral evidence that the model acquired new reasoning capabilities. such as generating longer or more structured solutions, using intermediate steps more effectively, or demonstrating improved error correction. Alternatively, comparative analysis of model behavior (e.g., reasoning trace complexity or error type distribution) before and after RL would strengthen the claim.\n\nIf the authors can provide more details to address my comments, I would be inclined to raise my score.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The paper presents a strong empirical result, achieving a 41.0% solve rate on SWE-bench Verified, which demonstrates that reinforcement learning can be effectively applied to large-scale, realistic software data.\n\n2. The experimental setup is clean and consistent—both the SFT and RL models are trained on the same dataset with aligned prompt templates, allowing for a meaningful attribution of performance gains to the training method itself.\n\n3. The paper is well-written, clearly organized, and easy to follow. The structure facilitates understanding of both the implementation details and the high-level motivations.\n\n4. The observed improvements on out-of-domain tasks like mathematics and general code reasoning, though small, are intriguing. These results could inspire future research into how specialized training in one domain affects generalization capabilities in others.\n\n## Weaknesses\n\n1. The paper makes limited methodological contributions. The problem formulation follows existing benchmarks like SWE-bench, and the approach is a direct combination of known components: GRPO for optimization, a difflib-based reward signal, and chain-of-thought prompting. There is no new algorithmic mechanism, inductive bias, or novel architecture introduced. As such, the contribution lies more in implementation scale than in conceptual innovation.\n\n2. The central claim that RL improves general reasoning ability is weakly supported. The reward function is based purely on token-level similarity, which does not capture semantic correctness or reasoning structure. The authors do not provide a theoretical or causal argument explaining why optimizing patch similarity should lead to improvements on tasks like math or logical reasoning, nor do they offer any behavioral or process-level evidence to validate such a connection.\n\n3. The RL formulation is shallow relative to the complexity of the reasoning tasks it claims to improve. The model operates in a single-step generation setting, with no planning, intermediate decision-making, or multi-turn interaction. The reward signal cannot distinguish between semantically different outputs if they are syntactically similar, further limiting the model’s capacity to learn deeper problem-solving strategies. This makes it likely that the model is learning format-aware mimicry rather than genuine reasoning.\n\n4. The evaluation design does not adequately isolate the effects of RL. While the SFT and RL models are trained on the same dataset, they are evaluated under different prompting conditions, and there is no ablation study to determine how much of the observed performance gain stems from RL versus prompting. Without a controlled comparison using the same prompt template, the attribution of improvement to RL is not justified.\n\n5. The reported cross-task improvements are small and not convincingly explained. Although the paper highlights slight gains on unrelated tasks such as mathematics, it does not analyze whether these improvements result from behavioral changes in the model or merely reflect incidental correlations. No trajectory analysis, failure case study, or reasoning trace inspection is provided to strengthen the generalization claim.", "was_revised": true, "final_justification": "The following points summarize my reasoning:\n1. **Generalization Evidence**   The authors provide concrete behavioral evidence (e.g., longer reasoning traces, increased output length) and controlled prompt experiments that support the claim that RL training improves general reasoning ability.\n\n2. **Reward Design Justification**   Although the difflib-based reward is surface-level, the authors justify its scalability and show that it complements execution-based rewards, enabling stable and large-scale RL training.\n\n3. **Empirical Strength**   The performance on SWE-bench Verified and out-of-domain tasks is competitive, with consistent improvements over SFT baselines trained on the same data.\n\n4. **Remaining Limitations**   The work lacks deeper theoretical explanation and more thorough ablations (e.g., KL penalty, group size), but these are not deal-breakers given the strong empirical results.\n\nOverall, I think the contribution is shareable with the community.", "rebuttal": "Dear Reviewer ZGKJ, we deeply appreciate your insightful feedback and suggestions for our work. In our responses below, we address each primary question (denoted as Q) and comment (denoted as C). Should there be any misunderstandings of the questions, please kindly let us know; we are eager to communicate with you throughout the discussion period.\n\n> Q1: …what specific mechanism or training dynamics enable skill transfer from patch generation to tasks like MATH or HumanEval+?\n\nGreat question. In SWE-RL training, the model needs to generate a correct patch in one step conditioned on the relevant code context that is typically long. To maximize the reward, it needs to perform extensive and accurate reasoning to find the correct edit location and to produce a correct patch. This process involves significant reasoning and helps the model bootstrap its general reasoning capabilities that are transferable to other tasks. In the table below, we can observe that the maximum thinking length (measured in character count) continuously increases during training:\n\n|Training steps|Max thinking length|\n|-|-|\n|400|**15546**|\n|800|**17704**|\n|1600|**20203**|\n\nSimilarly, in the following table, we measured the average output length of the original, SFT, and SWE-RL checkpoints on HumanEval+ and MATH. After RL, the reasoning is longer, supporting our assumption that the model acquires general reasoning ability through improved software issue solving.\n\n|Benchmark \\ Setting|Baseline|SFT|RL|\n|-|-|-|-|\n|HumanEval+|746|657|**1622**|\n|MATH|1876|1643|**2552**|\n\n> Q2: …did you evaluate the SFT baseline using the exact same prompt as used during RL training and evaluation?…\n\nWe use the most appropriate prompts tailored to each model checkpoint during evaluation. For SWE-bench, both the SFT and RL checkpoints are evaluated using identical system and user prompts. For out-of-domain benchmarks like HumanEval+ and MATH, we keep the user prompts the same across checkpoints, but apply the system prompt (shown in Figure 2) only to the RL-trained model. This is because the SFT checkpoint's training data includes system prompts for code editing tasks, but not for general coding tasks. As a result, applying a system prompt in those general tasks would be out-of-distribution for the SFT model. In contrast, the RL checkpoint is expected to generalize its reasoning under the system prompt, making its use appropriate.\n\nTo confirm that this difference in prompting doesn't unfairly benefit the RL model, we ran an additional experiment applying the system prompt to the SFT checkpoint on HumanEval+ and MATH as well. This allowed us to test whether the system prompt itself influences performance on general tasks for SFT.\n\n|Benchmark|SFT (prompt in paper)|SFT (RL prompt)|RL (RL prompt)|\n|-|-|-|-|\n|HumanEval+|73.2|70.1|**79.9**|\n|MATH|71.7|70.6|**73.7**|\n\nFrom the result, we can see that the SFT checkpoint is still worse than the RL one despite the prompt alignment. It’s also worse than the prompt we used for evaluation in the paper. It matches our claim that RL leads to generalized reasoning while SFT is more about memorizing the data patterns.\n\n> Q3: The difflib-based reward function operates on surface-level textual similarity and is agnostic to semantic correctness…\n\nWe want to kindly highlight that the main novelty of SWE-RL lies in the insight that real-world software bugs, unlike open-ended code generation, often follow constrained, localized patterns, aligning with the plastic surgery hypothesis [1] in program repair. The difflib-based reward can efficiently capture partial correctness of code changes.\nWhile we do observe some high-reward patches that fail the tests, these episodes can still improve reasoning because the difflib signal implicitly incentivizes accurate bug localization and minimal, targeted edits, which are necessary intermediate steps toward a correct fix. With more training steps, the model can learn to refine these patches further and produce semantically correct solutions.\nCrucially, SWE-RL reward is what makes large-scale training possible; execution-based rewards require expensive environment setup and heavy data curation, limiting scalability. For example, SWE-Gym [2] requires 200 human annotation hours and 10,000 CPU core hours to produce 2438 trainable instances, while we collect 273k instances for training fully automatically.\n\nIn the meantime, test-based reward is never perfect. In certain cases, it can be worse than similarity metrics where incorrect patches pass all the tests but miss the true intention of the problem. This claim is well supported by the literature. For example, [3] shows that the insufficient tests in SWE-bench impacts 40.9% of SWE-Bench Lite and 24.4% of SWE-Bench Verified leaderboard entries. Below is an example illustrated in the paper, where the original issue requires a function that computes polynomial fits for data, to handle missing data in the inputs x and y. However, the original test case only considers scenarios where both x and y have missing data. The generated patch, while passing the test, fails to handle additional cases where only one input has missing data.\n\n```\n# Insufficient test\ndef test_missing_data(self, df):\n  groupby = GroupBy([\"group\"])\n  df.iloc[5:10] = np.nan\n  res1 = PolyFit()(df[[\"x\", \"y\"]], groupby, \"x\", {})\n  res2 = PolyFit()(df[[\"x\", \"y\" ]].dropna(), groupby , \"x\", {})\n  assert_frame_equal(res1, res2)\n\n# Incorrect patch that passes the test\ndef _fit_predict(self, data):\n  y = data[\"y\"].dropna()\n  x = data[\"x\"].dropna()\n  if x.shape[0] != y.shape[0]:\n    raise ValueError (\"x and y must have the same number of non - missing values \")\n  if x.nunique() <= self.order:\n    # TODO warn ?\n    xx = yy = []\n```\n\nTo further support our point, we did an additional experiment showing that combining SWE-RL reward with execution is superior to applying execution reward alone. The results are in the table below. Please kindly refer to Q1 from Reviewer n4Dr for more details:\n\n|Setting|Pass@1|\n|-|-|\n|Baseline|0.8%|\n|Execution-only|11.0%|\n|Execution + SWE-RL reward|**14.2%**|\n\n> Q4: …How does this setting support the emergence of non-trivial reasoning strategies, as opposed to shallow pattern recognition?\n\nAs we explain in Q3, SWE-RL requires non-trivial reasoning to localize errors and generate patches correctly to solve the software issues. During RL, such reasoning patterns emerge, involving planning and backtracking. We show one example below. Please also kindly refer to Figure 3 in the paper for more examples.\n\n```\nWe need to identify where the issue of not preserving single quotes in TODO comments occurs… In the ToDoItem class, the description is being extracted…\nBut wait, in the ToDoItem constructor, there's this line: _type = description.Substring(0, idxDelimiter);. However, the idxDelimiter is being found …but that's not the issue here.\nBut then I saw it: in the second constructor of ToDoItem…\nThe actual issue is likely due to… And then it hits me… I think I've got it now.\n```\n\n> Q5: …please provide concrete behavioral evidence that the model acquired new reasoning capabilities…\n\nGreat point. As we showed in Q1 and Q2, the reasoning length of the model increases after SWE-RL in both training and evaluation. According to Figure 3 in the paper and the example we showed in Q4, SWE-RL triggers the model’s emergent reasoning skills such as planning, self-reflection, and divide-and-conquer. Below, we also show a concrete example on MATH where the model doesn’t do much reasoning before SWE-RL, but reasons deeply afterward:\n\nA math problem whose ground-truth answer is $\\frac{\\sqrt{21}}{5}$:\n\nBefore SWE-RL\n```\nWe have that $\\sin R = \\frac{2}{5} = \\frac{RS}{RT}$, so $RS = 2$ and $RT = 5$.  By the Pythagorean theorem… Then $\\sin T = \\frac{RS}{RT} = \\frac{2}{5}$.\n\nAnswer: $\\frac{2}{5}$\n```\n\nAfter SWE-RL:\n```\nTo find $\\sin{T}$, we first need to understand the given information. We're dealing with a right-angled triangle $\\triangle{RST}$, and it's given that $\\sin{R}=\\frac{2}{5}$…\nFrom the diagram, we see that the length of the hypotenuse…\nNext, to find $\\sin{T}$, we need to determine the length of the side opposite…\nHowever…\nGiven $RS = 2$ and $RT = 5$…\nBut, we actually need to use $ST$ and $RT$ …\nThus, $\\sin{T} = \\frac{ST}{RT} = \\frac{\\sqrt{21}}{5}$.\n</think>\n<solution>\nAnswer: $\\frac{\\sqrt{21}}{5}$ \n</solution>\n```\n\n> C1: The paper makes limited methodological contributions…\n\nAs we explained in Q3, the main novelty of SWE-RL lies in the insight that real-world software bugs, unlike open-ended code generation, often follow constrained, localized patterns, aligning with the plastic surgery hypothesis [1] in program repair. Furthermore, SWE-RL is the first to show that the simple difflib-based reward signal can already enable scalable and effective RL on massive real-world software data. This finding will impact lots of future work in this critical application domain. \n\n> C2: The reported cross-task improvements are small and not convincingly explained…\n\nIndeed, 1-2% performance gains on HumanEval or CRUXEval are not likely to be significant by themselves (at the 0.05 level). That is why we reported on multiple evaluations where all results are consistently in favor of RL, thus giving increased significance (for example, via Fisher’s combined probability test). To support this, we conducted additional statistical testing (link omitted due to rebuttal policy), showing that >0.8% on MMLU (14k examples), 3% on CRUXEval, and >3% on the full MATH are already significant. So the combination of our results will reach significance at the 0.05 level. We will include this statistical analysis in the revision of our paper.\n\n> C3/C4/C5 are covered in the previous responses.\n\n[1] Barr et al. The Plastic Surgery Hypothesis.\n\n[2] Pan et al. Training Software Engineering Agents and Verifiers with SWE-Gym.\n\n[3] Yu et al. UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench."}, {"review_id": "7hbT2VBRxY", "reviewer": "Reviewer_QFgA", "summary": "The paper proposes SWE-RL to train LLMs on Github PR data with GRPO algorithm for software engineering tasks. Specifically, Agentless Mini, a revised version of Agentless is adopted as the scaffold for both rollout and evaluation. Reward is calculated as the sequence similarity between the generated and the golden patch. Results demonstrate that SWE-RL achieves SOTA performance among all models under 100B. It not only surpasses the corresponding SFT baseline, but also generalizes to other tasks beyond SWE-Bench.", "questions": "* It is unexpected to see degradation of SFT model on general tasks in Table 3 with respect to the base model, given that coding and general SFT data are also included in training. Do you have any insights?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The proposed RL framework is novel in software engineering domain.\n* The simple reward of sequence similarity is easily scalable and demonstrated effective.\n* SWE-RL not only improves performance in the targeted domain but also generalizes to other tasks where reasoning is helpful.\n* The paper is written clearly.\n\nWeaknesses: \n* Did you employ any test-time scaling during RL training by generating multiple patches and selecting only one for each rollout? If not, what's the reason for reporting the evaluation metrics with test-time scaling, rather than just let the model generate one patch and submit for evaluation? It can be more intuitive to have training and evaluation under the same setting.\n* The comparison between the RL and the SFT model on the original SWE-Bench verified task (instead of the repair task) is critical but not included.\n* Have you tried SFT with synthetic code editing data only? While this may degrade performance on general benchmarks, such an experiment can provide an upper bound of SFT performance under the assumption that the downstream task is fixed and known. It would be great to see SWE-RL is better even under this setting.", "was_revised": true, "final_justification": "The rebuttal mostly addressed my concerns on the evaluation part. The primary reason for not giving a higher score is that edit similarity as a reward in SWE tasks is now overshadowed by execution feedback which is more precise and widely adopted, limiting the practical impact of this work.", "rebuttal": "Dear Reviewer QFgA, we deeply appreciate your insightful feedback and suggestions for our work. In our responses below, we address each primary question (denoted as Q) and comment (denoted as C). Additionally, we will revise our paper to incorporate editorial suggestions. Should there be any misunderstandings of the questions, please kindly let us know; we are eager to communicate with you throughout the discussion period.\n\n\n> Q1: It is unexpected to see degradation of SFT model on general tasks in Table 3 with respect to the base model, given that coding and general SFT data are also included in training. Do you have any insights?\n\nGreat question. We believe that the underlying reason is that RL teaches the model generalized reasoning while SFT only lets the model memorize the data it is trained on, as supported by a recent paper [1]. SWE-RL, while being a single RL task, requires the model to do complex reasoning on the long context input comprising hundreds to thousands of lines of code to correctly localize the bug and generate the patch. This process incentivizes the model to learn generalized reasoning.\n\n> C1: Did you employ any test-time scaling during RL training...\n\nThank you for the thoughtful question. No, we did not use test‑time scaling during RL training: each rollout produces a single patch and is rewarded via patch similarity, and we do not train any separate selector or reranker. We still report test‑time–scaled evaluation to follow existing work like Agentless [2], also validating the generalizability of SWE-RL where the reasoning learned through SWE-RL transfers to other subtasks. As a result, in training, we only need to guide the model to generate the best patch for each rollout, keeping the RL scalable and execution‑free. For a matched setting, our pass@1 (no scaling) results are reported in Table 2 and show consistent gains over SFT.\n\n\n> C2: The comparison between the RL and the SFT model on the original SWE-Bench verified task (instead of the repair task) is critical but not included.\n\nGreat point. The main reason why we evaluate the SFT and RL model on the repair task is to focus on comparing their bug fixing and code editing capabilities, isolating the effect of localization used in the Agentless framework. To have a better understanding of their respective performance on the full SWE-bench Verified task, we conducted an additional experiment below to show the SFT pass@1 with the end to end pipeline. From the table, we can see that SFT is worse than RL by 4.8 points. According to our statistical analysis, a 4-point difference is statistically significant enough to show that RL is better than SFT on SWE-bench Verified.\n\n|Setting|Pass@1|\n|-|-|\n|SFT|36.2%|\n|RL|**41.0%**|\n\n\n> C3: Have you tried SFT with synthetic code editing data only?...\n\nGood question. We mixed different SFT data to make sure the model can follow different kinds of instructions and can successfully complete all the steps in Agentless-Mini. In the table below, we also conducted an SFT experiment with only code editing data. It has a similar score to the mixed SFT model in the repair only setting. This checkpoint cannot be evaluated in the other Agentless steps because it cannot follow instructions well and suffers from many format errors in the other subtasks such as localization and test generation.\n\n|Setting|Repair Pass@1|\n|-|-|\n|SFT|29.6%|\n|SFT (editing only)|29.8%|\n|RL|**34.8%**|\n\n\n[1] Chu et al. SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training.\n\n[2] Xia et al. Agentless: Demystifying LLM-Based Software Engineering Agents"}, {"review_id": "ntD9uXH1nw", "reviewer": "Reviewer_2vNV", "summary": "This paper introduces a novel framework, SWE-RL, which uses reinforcement learning (RL) designed to enhance large language model (LLM) reasoning for software engineering (SE) tasks by leveraging software evolution data (GitHub pull requests). The authors fine-tune LLaMA-3-70B-Instruct using 'rule-based' rewards (patch similarity computed via Python library \"difflib.SequenceMatcher\"), with the resulting Model known as: __Llama3-SWE-RL-70B__. The proposed approach shows substantial performance by achieving a 41.0% solve rate on SWE-bench Verified, which outperforms medium-sized open models and rivals proprietary systems like GPT-4o. In addition, the derived model reveals emergent inference to out-of-domain (OOD) tasks (e.g., math, code reasoning) despite being trained exclusively on SE data.", "questions": "1. Have you validated whether the `difflib`-based reward encourages shallow formatting mimicry rather than semantic code transformations? I would recommend that the authors provide any adversarial examples or robustness tests.\n\n2. What empirical evidence links SWE-RL training to improved general reasoning, beyond slightly higher pass\\@1 scores on MATH and MMLU? Were alternative prompts or reasoning diagnostics used?\n\n3. Why didn’t the authors include an LLaMA-3 SFT baseline using the same PR dataset to isolate the contribution of RL?\n\n4. Can the authors report confidence intervals or run their experiments with different seeds to show variability and robustness?\n\n5. What was the observed variance in the reward signal during RL training? Were instabilities observed?", "limitations": "The authors partially acknowledge limitations (e.g., lack of statistical testing), but omit critical discussion on the brittleness of their reward signal, the risk of overfitting to diffs, and the unexplained generalization claim. More transparency about the model’s weaknesses in semantic reasoning, reasoning trace depth, or vulnerability to reward hacking would strengthen the work.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "This manuscript presents a rich set of valuable results that could potentially enhance the practice of SWE research and practice. However, some drawbacks hinder the validity of these contributions, which I believe the authors must address. \n\n __Strengths__\n\n On the one hand, this paper: \n\n- Addresses a timely and under-explored area in LLM fine-tuning for code-based tasks that enhances general reasoning through SWE tasks. Specifically on static metrics like PR. \n\n- Presents an actual use of PR-based 'diff' similarity as a reward signal for RL training on real-world code edits.\n\n- Illustrates competitive performance on SWE-bench Verified (41.0%), which suggests that the fine-tuned model improves at long-context issue resolution.\n\n- Claims generalization to OOD domains (MATH, HumanEval, MMLU), even though the model was trained only on SWE (PR-related) tasks.\n\n-- Reuses available (open) tools and datasets (e.g., SWE-bench), paving the way for future research and reproducibility in principle.\n\n\n\n __Weaknesses__\n\nOn the other hand, four major areas of weakness pose serious issues for the authors to address:\n\n1 **Quality:**\n\n1.1. The reward function represented in Eq. (1), Pg. 3 is a bit shallow, based on the fact that it relied mainly on the string-level similarity (`difflib`), thus, prone to exploitation via formatting tricks or lexical mimicking, rather than genuine semantic improvements. \n\n1.2. The policy LLM represented in Eq. (2), Pg. 4, interestingly, uses a KL Penalty for Regularization that penalizes the updated policy from drifting too far from the reference model, in this case, vanilla LLaMA-3, which preserves general capabilities while specializing in the SWE fine-tuning tasks. However, the KL Divergence Target Is Not Fully Defined: 𝜋_ref is introduced, but we do not know if it's frozen, periodically updated, or fixed to 𝜋_old\n\n1.3. Group Size 𝐺 Sensitivity specifies the number of rollouts per prompt, which is quite critical but under-specified by the authors.\n\n1.4. I would suggest that the authors include an ablation study on group size (𝐺) clipping (𝜖), and KL penalty 𝛽. Then, report the learning stability over time, particularly for low-reward categories.\n\n1.5. No ablation is provided comparing SWE-RL to supervised finetuning (SFT) on the same PR data, leaving it unclear whether RL contributes meaningfully. The authors should be intentional with their empirical setup upfront to guide a clear understanding of the reported results. \n\n1.6. The size or quality of the PR task is not considered as would be in a typical SWE setting. It's not clear to me if similar results would be observed with change request data, since PR is typical of the GitHub-related tool. Besides, on Pg. 5, the authors reported 500  patches and a temperature of 1.0. Why 500? Motivate this number and justify when the temperature was set to 1.0\n\n1.7. The generalization claim to OOD domains is not adequately supported: improvements are marginal (1–2%) and lack statistical testing or multiple runs.\n\n1.8. In Section 3, the authors seem to have omitted statistical testing (significance), as the authors claim, due to greedy decoding, which is incorrect. Variance still exists in task difficulty, decoding determinism, and sampling.\n\n1.9. The SWE-bench results (Table 1) are overinterpreted: outperforming DeepSeekCoder-33B and WizardCoder-15B is expected given the model’s size and data alignment. However, claiming “comparable to GPT-4o” without GPT-4o’s SWE-bench Verified number is speculative. The authors should turn down their tune on reported observations. \n\n1.10. Surprisingly, no analysis of reward variance, RL stability, or convergence is reported, despite the complexity of RL optimization.\n\n2 **Clarity.**\n\n__The authors should pay special attention to how scientific papers are written, specifically in words and the choice of color.__\n2.1. Footnote formatting is inconsistent; footnotes in Table 1 clutter interpretation.\n\n2.2. Equations and notation (e.g., Reward function R(o)) are inconsistently rendered.\n\n2.3. Typos such as “sofware” in Section 3.3 suggest rushed proofreading.\n\n2.4. Key terms like “aha moments” should be discouraged. They are used loosely and should be replaced with formal terminology.\n\n2.5 The authors should properly define and cite rule-based rewards on first mention, before using them.\n\n2.6. I was expecting to see more SWE papers in the Introduction when announcing SE tasks/bugs/Issues, and software testing.\n\n2.7. In Figure 1. IS there any dependency between these disconnected processes?\nFrom GitHub to Seed RL dataset and Issues/Code to GRPO. Can the authors explain how these two are related diagrammatically?\n\n2.8. ‘SWE tasks‘ is ambiguously defined in this paper. Can the authors properly define an SWE task, such as PR? To elucidate the process and explain why PR was selected for this study and not other SWE tasks? Besides, PR is typical of GitHub technology. What other/similar processes exist with other workflows?\n\n2.9. Pg.2 Line 48-50: \"we propose SWE-RL, the first RL method to improve LLMs on SE tasks by directly using rule-based rewards and software evolution data—the record of entire software lifecycle, including all code snapshots, changes, and events like PRs and issues.\" This statement is too assuming. I would advise the authors to familiarize themselves with SWE conferences on KLLM4Code and AIWare papers \n\n2.10. Line 50: Producing the code change, how? Explain what is code change is and how the LLM produces it at this stage\n\n2.11. Line 55-57: Add a visual to support this description, showing files with code change set patch sets. What similarity measures did you use? Cosine?\nLines 58-59: Given that fault localization is a difficult problem in SWE, the author should explain how this process was done.\n\nWhat does the Python ‘difflib’ do?\n\n3 **Significance:**\n\n3.1. This work aims to advance SWE domain expertise by developing models fine-tuned or trained with SWE-specific data, which could have a broader impact. Still, the lack of deeper reward design and validation makes the current framework fragile.\n\n3.2 If general reasoning improvement from domain-specialized RL could be proven, the implication would be important. However, this remains unsubstantiated. Contrary to this, how could we be certain that a vanilla LLM would not generalize to OOD tasks as such? I would suggest that the authors experiment in a control environment before concluding on OOD\n\n4 **Originality:**\n\n4.1. SWE-RL appears novel in combining GitHub PR diffs and RL, but builds directly on SWE-bench and DeepSeek-style setups. The reward shaping and update rule (GRPO) is inherited without critical innovation.\n\n\n\n-------\nI would suggest that the authors reference SWE papers from among these top-tier venues: ICSE/ICSME/MSR/SANER + TSE/TOSEM/EMSE/JSS Papers", "was_revised": true, "final_justification": "The authors have meaningfully improved the clarity of the paper and addressed several of my main concerns, particularly by:\nClarifying the reward design. Authors have provided a rationale for using the difflib-based similarity metric, linking it to the plastic surgery hypothesis in program repair and emphasizing its scalability advantages over execution-based rewards. Additional experiments showed that combining similarity with execution-based reward outperforms execution alone.\nProviding evidence for reasoning improvements. The authors supplied qualitative reasoning traces and quantitative measurements (longer “thinking length” and output length), suggesting that SWE-RL encourages multi-step reasoning behaviors.\nEstablishing stronger baselines. An SFT baseline trained on the same PR dataset was included, and end-to-end evaluations confirmed that RL yields consistent gains over SFT.\nStatistical significance and stability. The rebuttal added statistical testing details (e.g., typical standard error on SWE-bench Verified, significance thresholds) and confirmed that training reward trends are stable with low variance, supported by concrete reward progression data.\nReproducibility commitments. The authors have pledged to open-source their work, and clarified hyperparameters to support replication.\nThese additions strengthen the technical soundness of the work and address many practical concerns regarding robustness and reproducibility. \n\nHowever, some significant limitations remain:\nReward signal brittleness: While the authors argue for the efficiency of difflib, it still does not reason about control/data flow, side effects, or deeper semantic correctness, leaving potential for over-optimization on superficial similarity.\nLimited ablation coverage: The lack of sensitivity studies for KL penalty, group size, and clipping remains an empirical gap. While the computational constraints are acknowledged, even a partial or theoretical exploration of these effects would strengthen the framework’s methodological grounding.\nOOD generalization claims: Gains on MATH, HumanEval+, and CRUXEval are small; while the authors’ statistical arguments improve confidence, further targeted diagnostics (e.g., reasoning trace analysis on OOD tasks) would make the case more convincing.\nOverall, SWE-RL is a timely and relevant contribution to the intersection of RL and LLMs for software engineering. \n\nThe authors have adequately improved my confidence in the reported results. However, I strongly suggest future work to address enhancing semantic depth in the reward function and expanding robustness analyses. \n\nThat said, I maintain a borderline accept rating, leaning toward acceptance given the strengthened empirical support, clearer framing, and practical impact potential.", "rebuttal": "Dear Reviewer 2vNV, we deeply appreciate your insightful feedback and suggestions for our work. In our responses below, we address each primary question (denoted as Q) and comment (denoted as C). Should there be any misunderstandings of the questions, please kindly let us know; we are eager to communicate with you throughout the discussion period.\n\n> Q1: …difflib-based reward encourages shallow formatting mimicry rather than semantic…\n\nWe want to kindly highlight that the main novelty of SWE-RL lies in the insight that real-world software bugs often follow contextualized patterns, aligning with the plastic surgery hypothesis [1] in program repair. The difflib-based reward can efficiently capture partial correctness of code changes and makes large-scale training possible; execution-based rewards require expensive environment setup and heavy data curation [2], limiting scalability.\n\nAs a result, SWE-RL requires non-trivial reasoning to localize errors and generate patches correctly to solve the software issues. During RL, such reasoning patterns emerge, involving planning and backtracking. We show one example below. Please also kindly refer to Figure 3 in the paper for more examples.\n```\nWe need to identify where the issue of not preserving…\nBut wait, in the ToDoItem constructor, there's this line: _type = description…\nbut that's not the issue…\nBut then I saw it: in the second constructor…\nThe actual issue is likely…\nAnd then it hits me…\nI think I've got it…\n```\n\nMeanwhile, test-based reward is never perfect. Sometimes, it can be worse than similarity metrics where incorrect patches pass all the tests but miss the true intention. For example, [3] shows that the insufficient tests in SWE-bench impacts 40.9% of SWE-Bench Lite and 24.4% of SWE-Bench Verified leaderboard entries.\n\nIn the table below, we also did an additional experiment showing that combining SWE-RL reward with execution is superior to applying execution-only reward alone. Please kindly refer to Q1 from Reviewer n4Dr for more details.\n\n|Setting|Pass@1|\n|-|-|\n|Baseline|0.8%|\n|Execution-only|11.0%|\n|Execution + SWE-RL reward|**14.2%**|\n\n\n> Q2: What empirical evidence links SWE-RL training to improved general reasoning…\n\nGreat question. In the training, the model needs to generate a correct patch in one step conditioned on the relevant code context that is typically long. To maximize the reward, it performs extensive and accurate reasoning to find the correct edit location and to produce a correct patch. This process helps the model bootstrap its general reasoning capabilities that are transferable to other tasks. In the table below, we can observe that the maximum thinking length (measured in character count) continuously increases during training:\n\n|Training steps|Max thinking length|\n|-|-|\n|400|**15546**|\n|800|**17704**|\n|1600|**20203**|\n\nSimilarly, in the following table, we measured the average output length of the original, SFT, and SWE-RL checkpoints on HumanEval+ and MATH. After RL, the reasoning is longer, supporting our assumption that the model acquires general reasoning ability:\n\n|Benchmark \\ Setting|Baseline|SFT|RL|\n|-|-|-|-|\n|HumanEval+|746|657|**1622**|\n|MATH|1876|1643|**2552**|\n\nAlso, we can observe qualitative examples like the one shared in Q1 and those in Figure 3, indicating that SWE-RL incentivizes the model to develop new reasoning strategies.\n\nWe use the most appropriate prompts during evaluation and did an additional experiments to align the prompts. As shown below, RL performs the best regardless of the prompt change. Please kindly refer to Q2 from Reviewer ZGKJ for more details.\n\n|Benchmark|SFT (prompt in paper)|SFT (RL prompt)|RL (RL prompt)|\n|-|-|-|-|\n|HumanEval+|73.2|70.1|**79.9**|\n|MATH|71.7|70.6|**73.7**|\n\n> Q3: Why didn’t the authors include an LLaMA-3 SFT baseline…\n\nPlease kindly note that we already included the SFT baseline in Table 2 for repair performance evaluation. The SFT checkpoint uses the same PR dataset as in RL. We also did an additional end to end evaluation below, showing that RL is superior to SFT.\n\n|Setting|Pass@1|\n|-|-|\n|SFT|36.2%|\n|RL|**41.0%**|\n\n\n> Q4: Can the authors report confidence intervals…\n\nAbsolutely. For SWE-bench Verified, the typical paired standard error std(A-B) is 2% for models A and B. Thus two models typically need to differ by 1.96 * 2% (~4%) to be significantly different at the 0.05 level. Coincidentally, std(A) is also around 2% for SWE-bench Verified.\n\n> Q5: What was the observed variance in the reward signal…\n\nGood point. As shown in Figure 5, the training reward steadily increased during training. It’s overall stable and the variance is low.\n\n> C1: …the KL Divergence Target Is Not Fully Defined…\n\nDuring RL, $\\pi_{\\rm{ref}}$ is always set to be the original Llama 3 and kept frozen. Our objective formula and KL definition follows the standard of existing work [4]. We will clarify this in the revision.\n\n> C2: Group Size… under-specified by the authors.\n\nPlease kindly refer to Section 3.1 of our paper. We described that we sampled \"16 rollouts from each of the 32 problems in every batch\", so the group size is 16. We will make it more clear in the revision.\n\n> C3: …include an ablation study on group size, clipping, and KL penalty…\n\nThanks for the suggestion. Our hyperparameter choices largely follow the existing best practice in the literature [4]. Unfortunately, we do not have the compute budget to ablate all combinations of the hyperparameters, but this is a valuable suggestion and we will include more ablations in the future when we have sufficient computing resources.\n\n> C4: The size or quality of the PR task is not considered… Why 500…\n\nWe did rigorous filtering to ensure the quality of the PRs used in the RL training. Please kindly refer to Appendix A for more details regarding the data curation process. Regarding the 500 patches, we demonstrated in Figure 4 in the paper that 500 patches enabled continuous test time scaling of the issue solve rate on SWE-bench Verified.\n\n> C5: …claim to OOD domains is not adequately supported…\n\n1-2% performance gains on HumanEval or CRUXEval are not likely to be significant by themselves (at the 0.05 level). That is why we reported on multiple evaluations where all results are consistently in favor of RL, thus giving increased significance (for example, via Fisher’s combined probability test). To support this, we conducted additional statistical testing (link omitted due to rebuttal policy), showing that >0.8% on MMLU (14k examples), 3% on CRUXEval, and >3% on the full MATH are already significant. So the combination of our results will reach significance at the 0.05 level. We will include this statistical analysis in the revision of our paper.\n\n> C6: …Variance still exists…\n\nIndeed greedy decoding does not remove variance, and we make no such claims. Most papers and leaderboards do not report the confidence intervals, likely to avoid cluttered tables that do not convey sufficiently useful information. Fortunately, the confidence intervals are mostly a property of the evaluation task. For example, on SWE-bench verified, we measured that a 4% difference is significant for all 90+ results on the leaderboard.\n\n> C7: …outperforming DeepSeekCoder-33B and WizardCoder-15B …without GPT-4o’s SWE-bench Verified number…\n\nPlease kindly note that we did not compare DeepSeekCoder or WizardCoder in the paper because they are not widely used for SWE-bench related tasks. Meanwhile, we did include GPT-4o's score in Table 1 for both SWE-agent (23.2%) and Agentless (38.8%), and SWE-RL is superior (41.0%).\n\n> C8: …no analysis of reward variance…\n\nPlease kindly refer to Figure 5 in the paper. We show that the training reward steadily increases during training. It’s stable and the variance is low.\n\n> C9: The authors should pay special attention to… words and the choice of color\n\nWe appreciate the suggestions. We will improve the clarity of the paper in the revision.\n\n> C10: What does the Python difflib do?\n\nAs we explained in the paper, `difflib` computes sequence alignments to measure textual similarity; we use it (e.g., via `SequenceMatcher` on unified diffs) to score overlap between predicted and target patches, yielding a dense reward for localized edits.\n\n> C11: …lack of deeper reward design and validation…\n\nWe acknowledge the reward is intentionally simple, but training was stable with low variance (C8) and we observe emergent, multi-step reasoning behaviors in traces (Q1/Q2). Combining SWE-RL reward is also better than using execution-only reward alone (Q1).\n\n> C12: …general reasoning improvement… remains unsubstantiated…\n\nWe compare an SFT baseline trained on the same PR data (Q3) and also present behavioral diagnostics (Q2), both indicating that RL drives the out-of-domain gains while SFT leads to memorization. We will clarify this point in the revision.\n\n> C13: SWE-RL appears novel… without critical innovation.\n\nAs we explained in Q1, the main novelty of SWE-RL lies in the insight that real-world software bugs, unlike open-ended code generation, often follow constrained, localized patterns, aligning with the plastic surgery hypothesis [1] in program repair. The insight itself does not seem novel as it has been known for a decade, but SWE-RL is the first to show that the simple difflib-based reward signal can already enable scalable RL on massive real-world software data. This finding will impact lots of future work in this critical application domain.\n\n> C14: …reference SWE papers from among these top-tier venues…\n\nGreat suggestion. We’ll expand Related Work to cite representative papers from top-tier software engineering venues in the revision.\n\n[1] Barr et al. The Plastic Surgery Hypothesis\n\n[2] Pan et al. Training Software Engineering Agents and Verifiers with SWE-Gym\n\n[3] Yu et al. UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench\n\n[4] DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning"}]

Author rebuttal:

Meta-review: This paper proposes SWE-RL, a novel reinforcement learning framework that fine-tunes large language models (LLMs) on real-world software evolution data from GitHub pull requests. The core scientific claim is that this training method, which uses a simple difflib-based patch similarity as a reward signal, not only achieves state-of-the-art performance for open-source models on the SWE-bench Verified benchmark (41.0% solve rate) but also enhances the model's general reasoning abilities, leading to improved performance on out-of-domain tasks like MATH and HumanEval+. The paper's main strengths are its novel application of RL to this domain at scale, the scalability and effectiveness of its reward function, and its comprehensive experimental evaluation. The primary weaknesses noted by reviewers were the shallow nature of the difflib reward, which lacks semantic understanding, and the initial lack of convincing evidence for the generalization claim. The high computational cost and missing ablations were also cited as limitations. 

The authors' response addressed some of the reviewers' concerns. The authors conducted several new experiments. They demonstrated that combining the SWE-RL reward with execution-based rewards is superior to using execution-only rewards, justifying the use of their simple signal. They also ran a controlled experiment to show that the RL model's performance gains are not due to prompting but are a result of the RL training itself. To support their generalization claim, the authors provided quantitative evidence of increased "thinking length" and output length after RL training, along with qualitative examples of new reasoning patterns. Finally, they provided a direct end-to-end comparison, showing that the RL model significantly outperforms the SFT baseline on SWE-bench Verified. As a result of the discussion, all reviewers, including the highly confident reviewer 2vNV, updated their final justifications to be more positive, acknowledging that their concerns were "meaningfully improved" and their confidence in the results was "adequately improved." In my final decision, I weighed the authors' successful efforts to address the most critical empirical and validity concerns very highly, while recognizing that some remaining limitations—such as the lack of full hyperparameter ablations—are acceptable given the strong empirical results and the high computational cost.
Decision: Accept (poster)