Title: SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
Venue: NeurIPS 2025
Accepted: yes
Keywords: Software Engineering

Reinforcement Learning
Abstract: The recent DeepSeek-R1 release has demonstrated the immense potential of reinforcement learning (RL) in enhancing the general reasoning capabilities of large language models (LLMs). While DeepSeek-R1 and other follow-up work primarily focus on applying RL to competitive coding and math problems, this paper introduces SWE-RL, the first approach to scale RL-based LLM reasoning for real-world software engineering. Leveraging a lightweight rule-based reward (e.g., the similarity score between ground-truth and LLM-generated solutions), SWE-RL enables LLMs to autonomously recover a developer's reasoning processes and solutions by learning from extensive open-source software evolution data -- the record of a software's entire lifecycle, including its code snapshots, code changes, and events such as issues and pull requests. Trained on top of Llama 3, our resulting reasoning model, Llama3-SWE-RL-70B, achieves a 41.0% solve rate on SWE-bench Verified -- a human-verified collection of real-world GitHub issues. To our knowledge, this is the best performance reported for medium-sized (<100B) LLMs to date, even comparable to leading proprietary LLMs like GPT-4o. Surprisingly, despite performing RL solely on software evolution data, Llama3-SWE-RL has even emerged with generalized reasoning skills. For example, it shows improved results on five out-of-domain tasks, namely, function coding, library use, code reasoning, mathematics, and general language understanding, whereas a supervised-finetuning baseline even leads to performance degradation on average. Overall, SWE-RL opens up a new direction to improve the reasoning capabilities of LLMs through reinforcement learning on massive software engineering data.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework, SWE-RL, which uses reinforcement learning (RL) designed to enhance large language model (LLM) reasoning for software engineering (SE) tasks by leveraging software evolution data (GitHub pull requests). The authors fine-tune LLaMA-3-70B-Instruct using 'rule-based' rewards (patch similarity computed via Python library \"difflib.SequenceMatcher\"), with the resulting Model known as: __Llama3-SWE-RL-70B__. The proposed approach shows substantial performance by achieving a 41.0% solve rate on SWE-bench Verified, which outperforms medium-sized open models and rivals proprietary systems like GPT-4o. In addition, the derived model reveals emergent inference to out-of-domain (OOD) tasks (e.g., math, code reasoning) despite being trained exclusively on SE data.\n\nSTRENGTHS AND WEAKNESSES:\nThis manuscript presents a rich set of valuable results that could potentially enhance the practice of SWE research and practice. However, some drawbacks hinder the validity of these contributions, which I believe the authors must address. \n\n __Strengths__\n\n On the one hand, this paper: \n\n- Addresses a timely and under-explored area in LLM fine-tuning for code-based tasks that enhances general reasoning through SWE tasks. Specifically on static metrics like PR. \n\n- Presents an actual use of PR-based 'diff' similarity as a reward signal for RL training on real-world code edits.\n\n- Illustrates competitive performance on SWE-bench Verified (41.0%), which suggests that the fine-tuned model improves at long-context issue resolution.\n\n- Claims generalization to OOD domains (MATH, HumanEval, MMLU), even though the model was trained only on SWE (PR-related) tasks.\n\n-- Reuses available (open) tools and datasets (e.g., SWE-bench), paving the way for future research and reproducibility in principle.\n\n\n\n __Weaknesses__\n\nOn the other hand, four major areas of weakness pose serious issues for the authors to address:\n\n1 **Quality:**\n\n1.1. The reward function represented in Eq. (1), Pg. 3 is a bit shallow, based on the fact that it relied mainly on the string-level similarity (`difflib`), thus, prone to exploitation via formatting tricks or lexical mimicking, rather than genuine semantic improvements. \n\n1.2. The policy LLM represented in Eq. (2), Pg. 4, interestingly, uses a KL Penalty for Regularization that penalizes the updated policy from drifting too far from the reference model, in this case, vanilla LLaMA-3, which preserves general capabilities while specializing in the SWE fine-tuning tasks. However, the KL Divergence Target Is Not Fully Defined: 𝜋_ref is introduced, but we do not know if it's frozen, periodically updated, or fixed to 𝜋_old\n\n1.3. Group Size 𝐺 Sensitivity specifies the number of rollouts per prompt, which is quite critical but under-specified by the authors.\n\n1.4. I would suggest that the authors include an ablation study on group size (𝐺) clipping (𝜖), and KL penalty 𝛽. Then, report the learning stability over time, particularly for low-reward categories.\n\n1.5. No ablation is provided comparing SWE-RL to supervised finetuning (SFT) on the same PR data, leaving it unclear whether RL contributes meaningfully. The authors should be intentional with their empirical setup upfront to guide a clear understanding of the reported results. \n\n1.6. The size or quality of the PR task is not considered as would be in a typical SWE setting. It's not clear to me if similar results would be observed with change request data, since PR is typical of the GitHub-related tool. Besides, on Pg. 5, the authors reported 500  patches and a temperature of 1.0. Why 500? Motivate this number and justify when the temperature was set to 1.0\n\n1.7. The generalization claim to OOD domains is not adequately supported: improvements are marginal (1–2%) and lack statistical testing or multiple runs.\n\n1.8. In Section 3, the authors seem to have omitted statistical testing (significance), as the authors claim, due to greedy decoding, which is incorrect. Variance still exists in task difficulty, decoding determinism, and sampling.\n\n1.9. The SWE-bench results (Table 1) are overinterpreted: outperforming DeepSeekCoder-33B and WizardCoder-15B is expected given the model’s size and data alignment. However, claiming “comparable to GPT-4o” without GPT-4o’s SWE-bench Verified number is speculative. The authors should turn down their tune on reported observations. \n\n1.10. Surprisingly, no analysis of reward variance, RL stability, or convergence is reported, despite the complexity of RL optimization.\n\n2 **Clarity.**\n\n__The authors should pay special attention to how scientific papers are written, specifically in words and the choice of color.__\n2.1. Footnote formatting is inconsistent; footnotes in Table 1 clutter interpretation.\n\n2.2. Equations and notation (e.g., Reward function R(o)) are inconsistently rendered.\n\n2.3. Typos such as “sofware” in Section 3.3 suggest rushed proofreading.\n\n2.4. Key terms like “aha moments” should be discouraged. They are used loosely and should be replaced with formal terminology.\n\n2.5 The authors should properly define and cite rule-based rewards on first mention, before using them.\n\n2.6. I was expecting to see more SWE papers in the Introduction when announcing SE tasks/bugs/Issues, and software testing.\n\n2.7. In Figure 1. IS there any dependency between these disconnected processes?\nFrom GitHub to Seed RL dataset and Issues/Code to GRPO. Can the authors explain how these two are related diagrammatically?\n\n2.8. ‘SWE tasks‘ is ambiguously defined in this paper. Can the authors properly define an SWE task, such as PR? To elucidate the process and explain why PR was selected for this study and not other SWE tasks? Besides, PR is typical of GitHub technology. What other/similar processes exist with other workflows?\n\n2.9. Pg.2 Line 48-50: \"we propose SWE-RL, the first RL method to improve LLMs on SE tasks by directly using rule-based rewards and software evolution data—the record of entire software lifecycle, including all code snapshots, changes, and events like PRs and issues.\" This statement is too assuming. I would advise the authors to familiarize themselves with SWE conferences on KLLM4Code and AIWare papers \n\n2.10. Line 50: Producing the code change, how? Explain what is code change is and how the LLM produces it at this stage\n\n2.11. Line 55-57: Add a visual to support this description, showing files with code change set patch sets. What similarity measures did you use? Cosine?\nLines 58-59: Given that fault localization is a difficult problem in SWE, the author should explain how this process was done.\n\nWhat does the Python ‘difflib’ do?\n\n3 **Significance:**\n\n3.1. This work aims to advance SWE domain expertise by developing models fine-tuned or trained with SWE-specific data, which could have a broader impact. Still, the lack of deeper reward design and validation makes the current framework fragile.\n\n3.2 If general reasoning improvement from domain-specialized RL could be proven, the implication would be important. However, this remains unsubstantiated. Contrary to this, how could we be certain that a vanilla LLM would not generalize to OOD tasks as such? I would suggest that the authors experiment in a control environment before concluding on OOD\n\n4 **Originality:**\n\n4.1. SWE-RL appears novel in combining GitHub PR diffs and RL, but builds directly on SWE-bench and DeepSeek-style setups. The reward shaping and update rule (GRPO) is inherited without critical innovation.\n\n\n\n-------\nI would suggest that the authors reference SWE papers from among these top-tier venues: ICSE/ICSME/MSR/SANER + TSE/TOSEM/EMSE/JSS Papers\n\nQUESTIONS:\n1. Have you validated whether the `difflib`-based reward encourages shallow formatting mimicry rather than semantic code transformations? I would recommend that the authors provide any adversarial examples or robustness tests.\n\n2. What empirical evidence links SWE-RL training to improved general reasoning, beyond slightly higher pass\\@1 scores on MATH and MMLU? Were alternative prompts or reasoning diagnostics used?\n\n3. Why didn’t the authors include an LLaMA-3 SFT baseline using the same PR dataset to isolate the contribution of RL?\n\n4. Can the authors report confidence intervals or run their experiments with different seeds to show variability and robustness?\n\n5. What was the observed variance in the reward signal during RL training? Were instabilities observed?\n\nLIMITATIONS:\nThe authors partially acknowledge limitations (e.g., lack of statistical testing), but omit critical discussion on the brittleness of their reward signal, the risk of overfitting to diffs, and the unexplained generalization claim. More transparency about the model’s weaknesses in semantic reasoning, reasoning trace depth, or vulnerability to reward hacking would strengthen the work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes SWE-RL to train LLMs on Github PR data with GRPO algorithm for software engineering tasks. Specifically, Agentless Mini, a revised version of Agentless is adopted as the scaffold for both rollout and evaluation. Reward is calculated as the sequence similarity between the generated and the golden patch. Results demonstrate that SWE-RL achieves SOTA performance among all models under 100B. It not only surpasses the corresponding SFT baseline, but also generalizes to other tasks beyond SWE-Bench.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The proposed RL framework is novel in software engineering domain.\n* The simple reward of sequence similarity is easily scalable and demonstrated effective.\n* SWE-RL not only improves performance in the targeted domain but also generalizes to other tasks where reasoning is helpful.\n* The paper is written clearly.\n\nWeaknesses: \n* Did you employ any test-time scaling during RL training by generating multiple patches and selecting only one for each rollout? If not, what's the reason for reporting the evaluation metrics with test-time scaling, rather than just let the model generate one patch and submit for evaluation? It can be more intuitive to have training and evaluation under the same setting.\n* The comparison between the RL and the SFT model on the original SWE-Bench verified task (instead of the repair task) is critical but not included.\n* Have you tried SFT with synthetic code editing data only? While this may degrade performance on general benchmarks, such an experiment can provide an upper bound of SFT performance under the assumption that the downstream task is fixed and known. It would be great to see SWE-RL is better even under this setting.\n\nQUESTIONS:\n* It is unexpected to see degradation of SFT model on general tasks in Table 3 with respect to the base model, given that coding and general SFT data are also included in training. Do you have any insights?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SWE-RL, a reinforcement learning framework to fine-tune Large Language Models (LLMs) on GitHub issue–patch data. The authors train a 70B Llama3-based model using a character-level difflib reward and the GRPO algorithm, achieving 41.0% on the SWE-bench Verified benchmark—a new state-of-the-art for open-source models of this size. The authors further claim that this training method improves the model’s general reasoning ability, as evidenced by improvements on out-of-domain tasks including MATH, HumanEval+, and MMLU.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The paper presents a strong empirical result, achieving a 41.0% solve rate on SWE-bench Verified, which demonstrates that reinforcement learning can be effectively applied to large-scale, realistic software data.\n\n2. The experimental setup is clean and consistent—both the SFT and RL models are trained on the same dataset with aligned prompt templates, allowing for a meaningful attribution of performance gains to the training method itself.\n\n3. The paper is well-written, clearly organized, and easy to follow. The structure facilitates understanding of both the implementation details and the high-level motivations.\n\n4. The observed improvements on out-of-domain tasks like mathematics and general code reasoning, though small, are intriguing. These results could inspire future research into how specialized training in one domain affects generalization capabilities in others.\n\n## Weaknesses\n\n1. The paper makes limited methodological contributions. The problem formulation follows existing benchmarks like SWE-bench, and the approach is a direct combination of known components: GRPO for optimization, a difflib-based reward signal, and chain-of-thought prompting. There is no new algorithmic mechanism, inductive bias, or novel architecture introduced. As such, the contribution lies more in implementation scale than in conceptual innovation.\n\n2. The central claim that RL improves general reasoning ability is weakly supported. The reward function is based purely on token-level similarity, which does not capture semantic correctness or reasoning structure. The authors do not provide a theoretical or causal argument explaining why optimizing patch similarity should lead to improvements on tasks like math or logical reasoning, nor do they offer any behavioral or process-level evidence to validate such a connection.\n\n3. The RL formulation is shallow relative to the complexity of the reasoning tasks it claims to improve. The model operates in a single-step generation setting, with no planning, intermediate decision-making, or multi-turn interaction. The reward signal cannot distinguish between semantically different outputs if they are syntactically similar, further limiting the model’s capacity to learn deeper problem-solving strategies. This makes it likely that the model is learning format-aware mimicry rather than genuine reasoning.\n\n4. The evaluation design does not adequately isolate the effects of RL. While the SFT and RL models are trained on the same dataset, they are evaluated under different prompting conditions, and there is no ablation study to determine how much of the observed performance gain stems from RL versus prompting. Without a controlled comparison using the same prompt template, the attribution of improvement to RL is not justified.\n\n5. The reported cross-task improvements are small and not convincingly explained. Although the paper highlights slight gains on unrelated tasks such as mathematics, it does not analyze whether these improvements result from behavioral changes in the model or merely reflect incidental correlations. No trajectory analysis, failure case study, or reasoning trace inspection is provided to strengthen the generalization claim.\n\nQUESTIONS:\n1. Your central claim is that reinforcement learning on software repair tasks leads to improved general reasoning. However, this remains an observational finding. Could you clarify what specific mechanism or training dynamics enable skill transfer from patch generation to tasks like MATH or HumanEval+?\n\n2. The RL-trained model is evaluated using a detailed chain-of-thought (CoT) prompt that could itself induce improved behavior. To attribute improvements to RL rather than prompting, did you evaluate the SFT baseline using the exact same prompt as used during RL training and evaluation? If not, how can you isolate the effect of RL from the prompt’s contribution?\n\n\n3. The difflib-based reward function operates on surface-level textual similarity and is agnostic to semantic correctness. Have you encountered examples where high-reward patches are semantically incorrect or logically invalid? How do such cases affect the interpretation of “improved reasoning”?\n\n4. Your RL formulation treats patch generation as a single-step decoding task, without intermediate actions or sequential planning. How does this setting support the emergence of non-trivial reasoning strategies, as opposed to shallow pattern recognition?\n\n5. The observed gains in task accuracy are not sufficient to support the claim that RL improves reasoning. To substantiate this, please provide concrete behavioral evidence that the model acquired new reasoning capabilities. such as generating longer or more structured solutions, using intermediate steps more effectively, or demonstrating improved error correction. Alternatively, comparative analysis of model behavior (e.g., reasoning trace complexity or error type distribution) before and after RL would strengthen the claim.\n\nIf the authors can provide more details to address my comments, I would be inclined to raise my score.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SWE-RL, a novel reinforcement learning (RL) approach that trains large language models (LLMs) to solve real-world software engineering (SE) tasks using open-source software evolution data, i.e., GitHub pull requests (PRs). Unlike previous RL methods that focused on competitive programming or math, SWE-RL targets real-world issue resolution, leveraging rule-based rewards derived from patch similarity between model-generated and human-written code. The authors train Llama3-SWE-RL-70B using this approach and evaluate it on SWE-bench Verified, where it achieves a 41.0% solve rate, surpassing all open-source LLMs under 100B parameters and rivaling proprietary models like GPT-4o. Surprisingly, this RL training also improves general reasoning ability across out-of-domain tasks (e.g., math, MMLU), even outperforming supervised fine-tuning (SFT) baselines trained on more diverse datasets\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n(1) Novel direction for RL training: SWE-RL extends RL training to software evolution data—a previously untapped but richly structured resource. It moves beyond execution-based or synthetic RL setups.\n\n(2) Well-motivated reward function: The reward uses difflib-based patch similarity, allowing for partial credit and encouraging incremental improvements, which better reflects real-world issue resolution.\n\n(3) Good performance and comprehensive experiments: Achieves the best performance among medium-sized open models (41.0%), without using proprietary model outputs for supervision. The paper conducts comprehensive experiments including rigorous baselines (SFT and original model), scaling studies, reward ablation, and cross-domain generalizability testing.\n\nWeaknesses: \n\n(1) Reward function limitation: As the author said in \"Limitations\", the reward is based on string-level patch similarity, not semantic equivalence. This penalizes functionally correct but syntactically different fixes, potentially limiting solution diversity.\n\n(2) Limited interactivity: The pipeline is not interactive or agentic—it lacks tool use, exploration, or test-driven feedback during training. This may limit the model’s ability to generalize to more autonomous agent settings.\n\n(3) Single-task RL training: The model is only RL-trained on one task (issue resolution). While generalization is observed, multi-task RL training might further enhance its capabilities.\n\n(4) The training cost is too high, which requires 512 H100 GPUs for ~ 32 wall-time hours, which may limit accessibility and reproducibility despite using open-source models.\n\nQUESTIONS:\n(1) Have the authors considered semantic diffing or static analysis tools (e.g., AST comparison or test outcome delta) for more robust rewards?\n\n(2) Could test failures or runtime logs be incorporated as feedback in future RL training?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}