PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification
Authors: Hongwei Li
Yuheng Tang
Shiqi Wang
Wenbo Guo
Venue: ICML 2025 poster (2025)
Keywords: large language model
automatic program repair
autonomous software improvement
autonomous software engineering
Primary area: applications->everything_else
TL;DR: In this paper, we propose PatchPilot, an agentic patcher that strikes a balance between patching efficacy, stability, and cost-efficiency.

Abstract: Recent research builds various patching agents that combine large language models (LLMs) with non-ML tools and achieve promising results on the state-of-the-art (SOTA) software patching benchmark, SWE-bench. 
Based on how to determine the patching workflows, existing patching agents can be categorized as agent-based planning methods, which rely on LLMs for planning, and rule-based planning methods, which follow a pre-defined workflow.
At a high level, agent-based planning methods achieve high patching performance but with a high cost and limited stability. 
Rule-based planning methods, on the other hand, are more stable and efficient but have key workflow limitations that compromise their patching performance.
In this paper, we propose PatchPilot, an agentic patcher that strikes a balance between patching efficacy, stability, and cost-efficiency. 
PatchPilot proposes a novel rule-based planning workflow with five components: reproduction, localization, generation, validation, and refinement (where refinement is unique to PatchPilot).
We introduce novel and customized designs to each component to optimize their effectiveness and efficiency. 
Through extensive experiments on the SWE-bench benchmarks, PatchPilot shows a superior performance than existing open-source methods while maintaining low cost (less than 1\$ per instance) and ensuring higher stability.
We also conduct a detailed ablation study to validate the key designs in each component.
Our code is available at https://github.com/ucsb-mlsec/PatchPilot.

PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification
Hongwei Li!
Yuheng Tang'
Shiqi Wang² Wenbo Guo
Abstract
1. Introduction
Automatic patching of issues and vulnerabilities has long been challenging task in software engineering and secu- rity (Jiang et al., 2021; Le Goues et al., 2021; Monperrus, 2018; Gazzola et al., 2018). Before the emergence of gen- crative AI, automated code generation primarily relied on program synthesis (Feng al., 2018; Huang al., 2019), which requires human-written specifications and cannot be applied to complex programs due to the constraints of SMT solvers. With the recent success of LLMs in various gen- erative tasks (Peng et al., 2023; Lian et al., 2023; Ghosal et al., 2023; Huang et al., 2024), particularly in code genera- tion (Zhu et al., 2024; Anthropic, 2024; Achiam et al., 2023; Team al., 2023; Roziere et al., 2023), researchers recently started exploring their applications in automatically fixing software vulnerabilities. They build LLM-based agents that automatically analyze and fix issues in real-world code- bases (Wang et al., 2024b; Liu et al., 2024; Ruan et al., 2024; Zhang et al., 2024b; Yang et al., 2024a).
Recent research builds various patching agents that combine large language models (LLMs) with non-MI. tools and achieve promising results on the state-of-the-ee thrt (SOTA) software patching benchmark, SWE-bench. Based on how to de- termine the patching workflows, existing patching agents can be categorized as agent-based based planning methods, which rely on LLMs for planning, and rule-based planning methods, which follow pre- defined workflow. At high level, agent-based planning methods achieve high patching perfor- mance but with high cost and limited stability. Rule-based planning methods, on the other hand, are more stable and efficient but have key work- flow limitations that compromise their patching performance. In this paper, we propose PatchPilot, agentic patcher that strikes balance between patching efficacy, stability, and cost-efficiency. PatchPilot proposes novel rule-based planning workflow with five components: reproduction, localization, generation, validation, and refine- ment (where refinement is unique to PatchPilot). We introduce novel and customized designs to each component to optimize their effectiveness and efficiency. Through extensive experiments on the SWE-bbench benchmarks, PatchPilot shows a superior performance than existing open-source methods while maintaining low cost (less than I$ per instance) and ensuring higher stability. We also conduct a detailed ablation study vali- date the key designs in each component. Our code is available at https: httpg/ttttbbccoc.ooo thub. Besb-mlsec/Patern
Technically speaking, existing patching agents consist of three main components: localization, generation, and val- idation. The localization component identifies the code snippets responsible for the issue that need be fixed. The generation component produces patch candidates, while the validation component selects the final patch from candidates.
There are two ways to schedule these components: agent- based planning(Yea) Yang et al., 2024a: moatless, 2024; Zhang et al., 2024b; IBM, 2024; Liu al., 2024; CodeR, 2024; Pedregosa et al., 2011; Ma et al., 2024; Wang et al., 2024b; Amazon, 2024; Zhang et al., 2024a), which utilizes LLMs to dynamically determine the patching workflow for different issues; and rule-based planning(Xia et al., 2024; Ouyang et al., 2024) that follows a fixed, predefined workflow for all issues, as specified by humans. Although achieving high patching performance, agent- based planning methods suf- fer a high cost and are not stable, which significantly limits their applicability in the real world. In contrast, existing rule- based planning methods are more stable and cost-efficient but have limited patching performance due to limitations in their planning workflows.
'Department of Computer Science, University of California, Santa Barbara, USA MMeta, New York, USA. Correspondence to: Wenbo Guo thenrygwb@ucsb.coo
In this paper, we present PatchPilot, a novel patching frame- work that balances the patching efficucy, stability, and cost- efficiency. At a high level, PatchPilot designs rule-based
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

PatchPilot: A Cost-Efficient Software Engincering Agent with Early Attempts on Formal Verification
planning workflow composed of five components: repro- duction, localization, generation, validation, and refinement. Given an issue as input, PatchPilot first reproduces the is- sue and retrieves related testing cases and finds the root cause (code snippets causing the issue) through localiza- tion. Its generation and validation components then generate patch candidates and validate whether they fix the issues while preserving the benign functionalities. Unlike exist- ing rule-based patchers(Xia et al., 2024), which regenerate patches from scratch whenever validation fails, PatchPilot introduces a novel refinement component that iteratively improves the current patch based on validation feedback. This refinement continues until the patch passes validation (i.e., becomes "qualified") or the iteration limit is reached. This design is aligned with the human patching workflow, which requires multiple rounds of trials and errors.
Early attempts on verifiable patching. As detailed in section 5, we further add a formal verification component where we leverage LLM to generate specifications and use the Z3 solver to provide a formal guarantee on patch correct- ness. Although we only verify eleven patches in SWF:-hench Lite, this marks an early exploration of verifiable patching agents.
2. Existing Patching Agent and Limitations
At high level, existing patching agents mainly have three components: localization, generation, and validation. The localization component pinpoints the code snippets that cause the issue and need to be fixed (denoted as "root cause"). the generation produces patch candidates, and the validation tries to find a final patch in the candidates. Although they have similar components, based plan- ning strategies, existing patching agents can be categorized into agent- ggent-based planning and rule-based planning. Agent- based planning leverages LLMs to determine the patching workflow (i.c., deciding when and which components to call), which can be different from different issues. On the contrary, rule-based planning follows a fixed workflow for all issues pre-specified by humans.
As specified in Section 3, each component has its own tech- nical challenges, and we introduce specified designs to ad- dress them. Specifically, first, we not only reproduce the issue but also find related benign tests, which later are crit- ical for determining whether the generated patches break normal functionalities during the validation. Second, we design our localization to provide not only the root cause but also the related context that is necessary for patching, and design additional tools for localization to retrieve neces- sary information from the codebase. Third, for generation, we break it down into patch plamning, which produces a multi-step patching plan, and patch generation, which gen- erates patches following the plan. This design is inspired reasoning (Wei et al., 2022; Yao et al., 2024; Yang et al., 2024b). Having detailed plan can explicitly prompt the LLM to think deeper and reason about the issue and give more comprehensive patching solutions, which is more effective than directly prompting the model to generate the whole patch.
A Agent-based planning. Most existing patching agents fol- low agent-based planning. However, most of them are closed-source: Marscode Agent (Liu et al., 2024), Com- posio SWE-Kit (Composio, 2024), CodeR (CodeR, 2024), Lingma (Ma et al., 2024), Amazon Q (Amazon, 2024), IBM Research SWE-1.0 (IBM, 2024), devlo (devlo, 2024), Gru (gru, 2024), and Globant Code Fixer Agent (Globant, 2024). Here, we focus on the open-source approaches.
by the recent emergence of inference-
A notable early method is SWE-Agent (Yang et al., 2024a), which has only localization and generation and leverages an LLM planner to drive the patching process. To as- sist the planner in calling functions within each compo- nent, SWE-Agent provides an Agent-Computer Interface (ACI), which grants LLMs the ability to execute bash commands and handle file operations (e.g., fi file_open and func_ sn)__ Follow-up works improve SWE- Agent by either improving its current components (Au- toCodeRover (Zhang et al., 2024b)) or incorporating ad- ditional components (Moatless (moatless, 2024; Antoniades et al., 2024) and SpecRover (Ruan et al., 2024)). Notably, Moatless and SpecRover add validation component. This component first lets L.M generate an input that can trigger the issue (denoted as "Proof-of-Concee (PoC)") and then runs the PoC against the generated patches to decide if they fix the issue.
Through extensive evaluations, we first show that PatchPilot outperforms all SOTA open-source methods on the SWE- bench Lite and SWE-bench Verified benchmark (Jimenez et al., 2023). Besides, we show that PatchPilot achieves the lowest cost among both top open-source and closed-source methods, validating its balance in patching accuracy and cost-efficiency. Second, we demonstrate that PatchPilot is more stable than the SOTA aaent- planning method OpenHands (Wang et al., 2024b), validating the advantage of rule-based planning in terms of stability. Finally, we validate the key designs discussed above through detailed ablation study and demonstrate that PatchPilot is compati- ble with multiple SOTA LLMs. Although PatchPilot is not on top of the SWE-bench, to the best of our knowledge, it achieves the best balance between patching performance, stability, and cost-efficiency. These are critical for practical- ity, making PatchPilot promising candidate for deployment in real-world scenarios.
So far, the SOTA open-source tool in this category is Open- Hands (Wang et al., 2024b), which is inspired by the Code- Act Agent (Wang et al., 2024a). OpenHands has three com-
2

---

PatchPilot: A Cost-Efficient Software Engincering Agent with Early Attempts on Formal Verification
localization, generation, validation. Its validation follows a similar idea as SpecRover, i.e., reproducing and executing PoC to decide if the issue is fixed. Similar to the SWE-agent, OpenHands also designs an ACI for the agent.
ponents:
all functionalities in FB; while preserving the behaviors of the unaffected functionalities Fs -F FB,- Our main goals are twofold. First, we aim to resolve many issues as possible across different issues and diverse repositories. Second, we also aim to maximize the stability and reduce the cost of our patching framework. We believe stability and cost-efficiency are critical for real-world applications of a patching tool. An unstable tool that produces only one correct patch across multiple runs significantly hinders its applicability for critical bugs. Furthermore, if the tool is too costly to use, it limits its usage by ordinary users.
Limitations. Agent-based planning approaches inherently suffer from two critical limitations. First, as probabilistic models, LLMs intrinsically have randomness. The random- ness is aggregated and amplified when the model is making all critical decisions during the patching. This will signifi- cantly jeopardize the stability and reliability of the patching agents, hindering their real-world usage. Second, to reduce randommess, existing approaches conduct multiple samples and trials, and ensemble them to obtain the LLMs' decisions. Moreover, L.MMs often need multiple trials to obtain a cor- rect decision. All these extra samples and trials significantly raise computational costs well financial costs they need to use commercial models.
Rationale behind PatchPilot. Recall from Section 2 that we discussed the advantages and disadvantages of rule- based versus agent-based planning. In general, agent-based planning is more expensive and less stable than rule-based planning. However, it may give a higher optimal issue re- solved rate than rule-based planning, as the L.M planner can explore more tailored workflows for different issues. In contrast, rule-based planning relies on a uniform workflow across different scenarios, which may not be effective in cer- tain instances. As such, if the primary goal to maximize the resolved rate on certain benchmarks, agent-based plan- ning should be the preferred strategy. Indeed, most existing tools follow this approach, especially in industry settings with much more resources than academia. However, as men- tioned above, a high resolved rate is not our sole goal, nor is it the only metric for evaluating a good patching agent. Stability and cost-efficiency are equally important as the resolved rate given that we are developing tool that can be used in real work rather than just exploring the boundary of LLM agents. As such, we choose to follow rule-based planning in our patching agent.
Rule-based planning, Agentless (Xia et al., 2024) is the SOTA method following rule-based planning. Agentless strictly follows a pre-defined sequential workflow, compris- ing localization, generation, and validation. Specifically, for localization, Agentless designs three-step procedure (file, function, line), where LLM is used to pinpoint the root cause at each step. It directly queries LLM without leveraging the rich information in the code structure. Agent- less's generation feeds the root cause and issue description to LLM and lets the model generate patch candidates. It simply stacks the input information together without using advanced prompting strategies. Its validation is similar to the agent-based planning methods introduced above. Re- poGraph (Ouyang et al., 2024) improves the localization by providing repository-level graph but without changing other components. Having a pre-specified workflow makes these methods more stable than agent-based planning meth- ods. It also allows the agent to integrate human knowledge.
Figure 1 illustrates the workflow of PatchPilot. It consists of five phases: reproduction, localization, patch generation, validation, and patch refinement. As discussed in Section 2, localization and generation are commonly included in ex- isting approaches. We add three additional components to improve the overall patching effectiveness and efficiency. The reproduction and validation components are crucial for determining patch quality and selecting the correct patch candidates for deployment. Some advanced patching agents also include these components; in Section 3.2, we will spec- ify how we designed ours to be more accurate and stable. Refinement is unique component in PatchPilot, as we observe that improving a partially correct patch based on validation feedback is often more effective and efficient than generating a new patch from scratch. This aligns with hu- man experience, as a correct patch often requires multiple rounds of testing and refinement.
Limitations, Agentless's 's sequential workflow is overly re- strictive. The agent cannot refine the root cause, generated patches, and PoCs if the patch candidates cannot pass the validation. It has to start over again, which is less efficient. In addition, as discussed above, the individual components of Agentless and RepoGraph also have flaws.
3. Methodology of PatchPilot
3.1. Technical Overview
Problem definition. Given buggy code repository written in Python, denoted as R. which contains set of func- tionalities F = f1, f fn written in different files. The repository may have one or more issues, where each issue 3, has an issue description written in text, denoted as D;- The issue 33 affects a subset of functionalities, denoted as FB, S F. A successful patch, denoted as p, should fix
Workflow of PatchPilot. As shown in Figure 1, given the input of the codebase R and the description D, of the tar- get issue B, PatchPilot first calls reproduction to recover
3

---

PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification
Localization
Generation
Input
PoG Coverago and Outjul
Plan
dles cription
Reproduction
PaG
Root 25 Cause
Benign
Potendal Cause
LLM
Plan
Penee
Lo bartioiion
Reviow
Paaming
Oornonatio
B
Pusch
Refinement
Update
Validation
Localization
Output
Valida
tian
Yen
No
Feodbaak for Panus
Benign
PuC Test
Generation
Quellfed
No
C1
Gest
Gurrent
Rank
Figure 1. Overview of PatchPilot. The system processes input through reproduction, localization, generation, validation, and refinement to obtain a final patch. Both localization and generation have two phases. The validation considers both PoC and functionality tests. Finally, the iterative refinement involves two conditions: CI checks if the patch passes all tests, if yes, the patch will be outputted: if no, C2 then checks if the current patch passes new test. compared to the previous round.
a set of testing cases, including PoC (a test that can trig- ger the issue) and benign functionality tests. PatchPilot runs the PoC and obtains the files it covered and the out- puts. Then, the localization component takes as input R, D,, and information related to the PoC and outputs the root cause (specific lines causing the issue). Similarly to Agentless (Xia et al., 2024), our localization also follows a hierarchical workflow but with additional tools to better extract and leverage the program structures. After identify- ing the root cause, the generation component generates N patch candidates at once. As discussed in Section 3.4, the key novelty here is separating planning and generation and leveraging multiple prompting strategies encourage patch diversity. The generated patch candidates are then fed to the validation component, which ranks the candidates based on their results of running the PoC and functionality tests. If the validation cannot find a qualified patch that passes all available tests, the refinement component will be called to refine the top-ranked patch candidate refine the localiza- tion based on the validation results. PatchPilot iteratively performs refinement and validation until it either identifies a qualified patch or reaches the maximum allowed number of generated patches (Nmux).
which is similar to the Reflexion mechanism designed for language agents (Shinn et al., 2024). During the process, we let LLM iteratively generate and refine the generated PoC for certain iterations. We carefully construct our prompts to guide the LLM focus on checking and correcting 1) whether any key dependencies and configurations are missing; and 2) whether the PoC actually reproduces the target issue. If the reproduction fails to generate a valid PoC within the maximum iterations, we proceed without a PoC. Second, different from existing works that only use the generated PoCs, we extract a more complete set of information based PoCs. This includes files covered by running the PoC, stack traces and outputs. As we will discuss later, this extra information helps localization and refinement. Third, we utilize LLM to identify three functionality test files from R that are most relevant to the target issue (cach file may con- tain multiple testing cases). These functionality tests enable the validation component to decide if the patch candidates preserve the functionalities of Fss important metric for a successful patch. More details about additional information retrieved based on PoCs are discussed in Appendix A.
Validation. The simple validation strategy utilized in ex- isting works (Tao et al., 2024; Globant, 2024; Ma et al., 2024) is just to feed the patch candidates and the related information an LLM and let it select the most qualified one. A more advanced strategy (Liu et al., 2024; Wang et al., 2024b; Arora et al., 2024) is to run the generated PoC and let an LLM decide whether the patches fix the issue based on their outputs. As mentioned above, ensuring the correctness of the original functionalities is as important as fixing the issue. As such, we include the functionality tests recovered by our reproduction in the validation. Specifically, we first run our PoC on the patch candidates and use LLM as judge for its evaluation. Since no assertions are available for bug fixing, this serves as the only feasible solution. We then
3.2. Reproduction and validation
Reproduction. We introduce three improvements over ex- isting work (Xia et al., 2024). First, reproduction in existing patching agents directly provides an LLM with R and D; and prompts it to generate PoC. However, D; often in- cludes only short code snippets related to the issue without specifying necessary dependencies and configurations (c.g.. the issue descriptions of D angoo typically do not have en- vironment setups). Without such information, the generated PoCs often fail to run successfully. To address this chal- lenge, we propose a self-reffection-buuee PoC reproduction,
4

---

PatchPilot: A Cost-Efficient Software Engincering Agent with Early Attempts on Formal Verification
also run the functionality tests and decide whether they pass based on their given assertions. Finally, we rank the patches based on the tests they pass. As specified in Appendix A, we prioritize patches that pass PoC tests over functionality tests during ranking.
details. Appendix A has more details on the tools we inte- grate. Third, as shown in Figure 1, we add review step that lets an LLM retrieve code snippets related to the current root cause. As mentioned above, localization oftentimes returns overly precise root causes that fail to include necessary con- text or even do not fully cover all root causes. Identifying more contexts is important to generate correct and complete patches. Note that we still constrain the maximum length of the final root cause to make sure not to overwhelm the generation with excessive context.
3.3. Localization
Key challenges. Some existing localization directly query an LLM to identify the root cause at line level (Yang et al., 2024a; Arora et al., 2024; Zhang et al., 2024b). Although they provide the LLM with tools to retrieve information from the codebase and allow it to refine its results, is still difficult for L.MMs to directly perform localizations at the line level. Besides, most agent-based tools incur high costs because they need to maintain the LLM agent's 's context his- tory during localization. Agentless designs a hierarchical workflow, which first identifies the issue-related files, then the functions, and lastly the lines. This method gradually zooms into and makes the task easier at the line level as it filters out the majority of the non-related functions in the earlier steps. At each step, Agentless lets the LLM make decisions only based on the issue description. This approach has three critical limitations. First, the information in issue descriptions is diverse and not all of them have useful in- formation for localization. For example, some descriptions only specify error messages and PoC-related information that is not helpful for localization. Second, this method lacks direct mechanism for retrieving details directly from the codebase. Third, in most cases, the localization returns only the root cause it is confident about as few lines of code. While this information is accurate, it is often insuffi- cient for writing a correct patch due to the lack of necessary context.
3.4. Patch Generation
Key challenge. Most existing patch generation components simply stack the related information and feed them to LLM for patch generation. Such a simple solution has two critical challenges. First, LLMs typically give incomplete patches. This is because fixing an issue often requires modifications across multiple locations or involves multiple steps, mak- ing it difficult to generate complete patch in one shot. In addition, the incomplete root causes also lead this is- sue. Second, being able to generate diverse patches is also crucial to increasing the likelihood of finding successful patch within certain trials. Moreover, we find that simply increasing the temperature still results in similar patches. We need other strategies to increase patch diversity, enabling the agent to search for more potential solutions.
Our design. First, shown in Figure 1, rather than directly generating the patch, PatchPilot breaks down the generation process into planning and generation. The planning phase first queries the LLM to generate a patch plan with multi- ple steps. The generation phase then generates the patch following the plan. After finishing each step in the plan, we also include a lightweight in in-generation validation with lint and syntax checks, and reconduct this step if the check fails. This design is motivated by the prompting strategy (Wei et al., 2022). That is, having plan explicitly forces the LLM to break down the patch gener- ation into multiple steps. This helps the model to better reason about the patch task, encouraging it to provide more complete patches. Besides the in neneeeratiion validation can identify and fix errors at an early stage, improving the patch efficiency. Second, to enhance the diversity of the gener- ated patch candidates, we design three types of prompts for plan generation. These prompts explicitly guide the LLM to produce patching plans with different focuses: a com- prehensive and extensive patch designed to prevent similar issues, a minimal patch with the smallest possible modifica- tions, or a standard patch without any specific instructions. Appendix D contains more details on the prompts that we use. As demonstrated Figure 1, we will generate N plans following the pre-specified prompts and thus produce N patch candidates in each batch.
Our design. We follow the three-step procedure in Agent- less given it is more stable and efficient than letting LLM directly do line level localization. First, to address the lim- itation of inconsistent issue descriptions, we provide the LLM with the PoC code and information after running it (i.e., files it covered, stack trace, and running outputs). This enables the LLM to access more comprehensive informa- tion, such as key functions or classes invoked in the PoC and the stack trace, which is particularly useful for cases where only the code to reproduce the issue is provided in the issue description. For example, the files covered by PoC can help filter out some files irrelevant to the target issue, re- ducing the search space, especially for codebases with many files. Second, to enable the LLM to extract and leverage more information from the codebase, we add set of tools to the localization component. These tools allow the LLM to search for class definitions and function definitions, or perform fuzzy string matching to locate and return relevant files. These tools provide precise search capabilities and can handle both class/function level information and line level
5

---

PatchPilot: A Cost-Efficient Software Engincering Agent with Early Attempts on Formal Verification
Table 1. Comparison of PatchPilot and five baselines on the two benchmarks. -Agent-hasedl and "Rule-based" refer agent-hased
planning and rule-based planning, respectively. "-" means not available. Note that Globant does not report the results on SWE-
Verified, and CodeStory does not report their result on SWE-benel Lite. They both do not disclose the LLM model(s) in their agents.
Patching
Open-
SWE-bench Lite
SWE-bench Verified
agent
source
LLM
Resolved%
Cost (S)
L.M
Resolved %
Cost
AutoCodeRover
GPT-40
30.67% (92)
0.65
Claude-3.5 3.5
51.80% (259)
4.50
Agent- based
OpenHands
V
-laude-3.5--0 S.5net
41.67% (126)
1.33
Claude-3,5 3.5nno
53.00% (265)
0.78
Globant
X
48.33% (145)
1.00
CodeStory
X
62.20% (311)
20.00
Rule-
Agentless
Claude-3.5 Sonnet
40.67% (123)
1.12
Claude -,nnentet
50.80% (254)
1.19
based
PatchPilot
Haude-3.5-Sonne
45.33% (136)
0.97
Claude-3.5 Sonnet
53.60% (268)
0.99
3.5. Patch Refinement
ing model 03-mini (OpenAI, 2024b). We failed to integrate DeepSeek-rl (DeepSeek, 2025) due to the problems with their APIs (See Appendix C.3).
Recall that refinement is unique component in PatchPilot that existing works do not have. The motivation for adding this component is to better leverage the validation feedback and the current partially correct patches. As shown in Sec- tion 4.3, refining existing parties based on validation results is more effective and efficient than re-generating patches from scratch. More specifically, as demonstrated in Fig- ure 1, PatchPilot focuses refining the top-ranked patch in the current batch. It feeds the current batch and its val- idation result back to the generation component and asks it to generate a new batch of patches. The generation still follows the planning and generation workflow. Here, when generating the plans, we design the prompt to guide the model to correct the failed testing cases of the current patch. This process continues until qualified patch that passes all validations is generated, or the total number of generated patches reaches the predefined limit of Nax. Note that if the patches generated in a whole batch do not pass any new tests, we rerun the localization with the validation results to ob- tain a new root cause. This additional step gives PatchPilot the opportunity to leverage information from later compo- nents to correct localization errors and ultimately succeed in generating qualified patches.
4.1. PatchPilot vs. Baselines on SWE-bench
Setup and design. We utilize the SWE-bench (Jimenez et al., 2023) benchmark, where each instance corresponds to an issue in a GitHub repository written in Python. Specif- ically, we consider two subsets: SWE-bench Lite (SWE- Bench, 2023a), consisting of 300 instances, and SWE-bench Verified (SWE-Bench 2023b), comprising 500 instances that have been verified by humans to be resolvable.
We mainly compare PatchPilot with three SOTA open- source methods: two agent-based planning methods Open- Hands (Wang et al., 2024b) and AutoCodeRover (Zhang et al., 2024b), and rule-based planning method Agent- less (Xia et al., 2024). We also compare it with two closed- source methods that have cost reported: Globant Code Fixer Agent (Globant, 2024) (Globant for short) and CodeStory Midwit Agent (CodeStory, 2024) (CodeStory for short). In Appendix C.I, we include a more comprehensive com- parison of PatchPilot against 29 other tools, showing our positions on the SWE-bench leaderboard. Given our goal of addressing stability and cost together with the resolved rate, comparing closed-source methods that have higher resolved rate but without cost is not our focus. Most of these methods follow based planning that may cost way more than ours. For example, CodeStory mentions that it costs them $10,000 to achieve 62.2% on the SWE-bench Verified benchmark (SWE-Bench, 2023b), whereas PatchPi- lot achieves a 53.60% with less than $500 (20x cheaper). In addition, as shown in Section 4.2, agent-based planning is less stable than rule-based planning.
4. Evaluation
We evaluate PatchPilot from the following aspects: First, we perform a large-scale comparison of PatchPilot with both SOTA open-source and closed-source methods the SWE-bench Lite and SWE-bench Verified patching bench- mark (Jimenez et al., 2023), showcasing PatchPilot's 's ability to balance patching accuracy and cost-efficiency. Second, we conduct a stability analysis on PatchPilot and Open- Hands, demonstrating PatchPilot's rule-based planning is more stable than the SOTA agent-based planning. Third, conduct an ablation study to quantify the contribution of each component to PatchPilot's overall performance. Finally, we show PatchPilot's compatibility and perfor- mance on different models, including GPT-4o (OpenAI, 2024a), Claude-3.5. Sandee3.55-Sone 3.5-Sonnet (Anthropic, 2023), and a reason-
To align with most methods, we use the Claude-3.5-Sone model as the LLM in PatchPilot. Appendix B shows our implementation details. We report two metries Resolved Rate (%): the percentage of resolved instances in the bench- mark,' and Average Cost ($): the average model API cost
'An instance/issue is resolved means the patch fixes the issuc while passing all hidden functionality tests.
6

---

PatchPilot: A Cost-Efficient Software Engincering Agent with Early Attempts on Formal Verification
Comparison of Resolved Rates and Costs
50
80
Resolved Rate (%)
45.33%
Resolved Rate (%)
60
41.67%
Cost ($)
40
35
32.67%
2
PatchPilot
PatchPilot 196)
PatchPllst
OpenHands Resolved Rate 196
OpenHands Cost (5)
Figure 3. Ablation study results on the SWE-bench Lite bench- mark. 0~0 refers to Base Local+ Gen, Our Local+ Gen, Our Gon and Our Local+ Val, respectively.
Figure PatchPilot vs. OpenHands in the resolved rate (hars) and the total cost (lines) on 45 instances from SWE-bench Lite.
of running the tool on each instance. For the baselines, we retrieve their performance from their submission logs on the SWE-bench and their papers and official blogs.
and OpenHands on these instances three times with GPT-4o model and different Pyt hython random seeds. We report and compare their resolved rate and total cost in each run.
Results. Table shows the performance of PatchPilot and selected baselines on two subsets of the SWE-bench bench- mark. Although, both benchmarks, the closed-source methods achieve the highest performance, their internal de- sign and methodology are not publicly available and we cannot assess their stability. Notably, the cost of CodeStory x0x higher than PatchPilot. The cost of Globant more comparable to PatchPilot on SWE-bench Lite, but we cannot assess their performance and cost on SWE-bench Verified. Among open-source methods, OpenHands achieves higher resolved rates than the rule-based planning tool, Agent- less, on both benchmarks. However, OpenHands has higher cost than Agentless on SWE-bench Lite when us- ing the same Slaude-3.5-Sonntt This result validates our discussion in Section 3.1, rule-based planning is more cost- efficient than agent-basse based planning, and agent-based plan- ning has the potential achieve higher optimal resolved rates.
Results. Figure 2 shows the resolved rate and costs of PatchPilot and OpenHands across three runs. As shown in the figure, PatchPilot consistently resolved more instances, achieving 30, 32, and 35 resolved instances in the three runs, with standard deviation of 2.52. In comparison, OpenHands resolved only 15, 20, and 21 instances, with a higher standard deviation of 3.21. The lower standard deviation of PatchPilot demonstrates its stability, which fur- ther validates our discussion about rule-based planning vs. agent-based planning in Section 3.1. Additionally, Patch- Pilot demonstrated clear advantage in terms of cost ef- ficiency, with costs of 78.72. $14.81, and $14.42 for the three runs, resulting in an average of $12.65 per run. This substantially lower than OpenHands, which incurred costs of $32.78. $33.31, and $34.97, with an average of $33.69 per run. These results further highlight PatchPilot's ability to achieve higher resolved rates with greater stability and lower cost.
In comparison, PatchPilot demonstrates a clear advantage in balancing resolved rate and cost. On SWE-bench Lite, it resolves 45.33% (136/300) of the issues, outperforming all open-source methods with a low cost of $0.97 per instance. Similarly, on SWE-bench Verified, PatchPilot achieves resolved rate of 53.60% (268/500), surpassing all open- source methods while maintaining the same cost efficiency of $0.99 per instance. These results highlight the efficacy and cost-efficiency of PatchPilot.
4.3. Ablation Studies
Setup and design. We conduct detailed ablation study to investigate the efficacy of key designs in PatchPilot. We use the full SWE-bench Lite benchmark and the Claude- 3.5-Sonnet model for all variations of our method. Specif- ically, we consider the following four variations: Base Local+Gen: We combine simple localization without pro- viding the LLM with tools or review step. along with sim- ple generation without the two-phase design (Figure 1). We choose the final patch by majority voting. OOur Local Gen: We combine PatchPilot's localization and generation compo- nents together with majority voting for final patch selection. Comparing with can assess the effectiveness of our proposed techniques for localization and generation. @Our Local+ Gen+ PoC: We further add our validation component
4.2. PatchPilot vs OpenHands in Stability
Setup and design. We compare the stability of PatchPi- lot and OpenHands, the SOTA open-source based planning tool. We find 102 common instances resolved by PatchPilot and OaenHands in the SWE-bench Lite bench- mark and randomly select a subset of 45. We run PatchPilot

---

PatchPilot: A Cost-Efficient Software Engincering Agent with Early Attempts on Formal Verification
to @ but with only the PoC tests (the validation strategy employed by most existing tools). Comparing @ with @ can assess the effectiveness of having PoC validation instead of simple majority voting. @Our Local+ Val: We add the full validation component, comparing @ with O can assess the efficacy of having functionality tests in valida- tion. Finally, comparing with PatchPilot can assess the importance of having an additional refinement component.
the SWE-bench Lite benchmark; all these 100 instances have been successfully resolved by at least one method ranked Top-10 on the SWE-bench leaderboard. We run PatchPilot with the selected models on these instances and report the final resolved rate. We keep all other components the same and only change the model to show the impacts of the different models.
Results. The resolved rate of PatchPilot with different mod- els are: 4P 19.00%; Slaude-3.--Soneee 39.00%, and 03-mini: 43.00%. 03-mini achieves the highest resolved rate, indicating having inference-phant phase reasoning capabili- ties is helpful not only for general math and coding tasks but also for the specialized patching task. Note that although we cannot directly compare with the results reported from offi- cial reports (anthropic, 2024; OpenAI, 2024c; Under, 2024), as they conduct their testing on the SWE-bench Verified benchmark. However, they follow the same trend: 03-mini > Claude-3.5-Sonne > GPT-4o. It is also worth noting that PatchPilot with Claude-3.5-Sonnee on the SWE-bench Verified benchmark reports a higher resolved rate than the of- ficial report from -.aude-3.5-Somme and OpenAl-O1 model. Although the full 03 reports a resolved rate of 71.7%, do not disclose any details about the system design, cost, and stability. Overall, this experiment demonstrates the com- patibility of PatchPilot to different models as well as the efficacy of having reasoning model in PatchPilot.
Results. Figure 3 shows the resolved rates across different variations and our final method. By incrementally building upon the core functionalities of PatchPilot, we evaluate the contributions of individual components to the overall patching performance.
Localization and generation. First, we can observe that with the simple localization and generation only get re- solved rate of 32.7% (98/300). In contrast, @ with our improved localization and generation increases the resolved rate to 38.7% (116/300). This result first confirms the chal- lenges of simple localization and generation designs dis- cussed in Section 3.3 and Section 3.4, as they prevent O from achieving a better performance. More importantly, it validates the effectiveness of our designs in adding tools and review step in localization and the two-step procedure (i.e., planning and generation) in the generation.
PoC and functionality validation. & with our localization and generation as well as PoC validation unexpectedly low- ers the resolved rate to 37.00% (111/300). This result sug- gests that relying solely on PoC validation may resolve the targeted issue while introducing new functional issues. As such, when functionality tests are added, @ significantly improves the resolved rate to 41.67% (125/300). This result shows that functionality tests play a crucial role in identi- fying and filtering out the patches that fix the target issues but break the original functionalities of the codebase. As mentioned above, a patch must pass all hidden functionality tests to be marked success; having functionality tests is important to filter out false positives.
5. Additional Formal Verification Component
To strengthen the reliability of our automatically generated patches, introduce formal-verification stage built on CrossHair (Schanely, 2022). CrossHair takes as input a function together with a logical contract, consisting of pre- conditions and post-conditions. Here, the pre-conditions capture assumptions that must hold when the function is invoked, and the post-conditions describe the properties ex- pected to hold when the function returns. CrossHair works by symbolically executing the function under the given pre- conditions and using the Z3 SMT solver to check whether the post-condition holds for all valid inputs. If an input is found that violates the post-condition, CrossHair reports it as a counter-exane exampe.
Refinement. Finally, adding our refinement component on top of ! improves the resolved rate from 41.67% to 45.33%. The result demonstrates the effectiveness of our refinement design. It also justifies our claim in Section 3.5 that gener- ating new patches from scratch when the current trial fails is less effective than refining the partially correct patches based on the validation feedback.
In our workflow, the verification stage for each candidate patch includes the following three steps:
Contract Generation. We prompt the LLM with the issue description and the original versions of the functions modi- fied by the patch. For each function, we ask the LLM to gen- crate type annotations for the function signature and a logi- cal contract containing pre-conditions and post-conditions.
4.4. PatchPilot on Different Models
Setup and design. To demonstrate the compatibility of PatchPilot to different LLMs, we conduct experiment that integrates PatchPilot with three SOTA LLMs: two general models GPT-4o and Saude-3.5-Sonnee and one reasoning model: o3-mini. We select a subset of 100 instances from
Contract Validation. We perform an initial validation step to filter out clearly incorrect logical contracts generated by the LLM. Specifically, for each function modified by the
8

---

PatchPilot: A Cost-Efficient Software Engincering Agent with Early Attempts on Formal Verification
patch, we insert the generated type annotations and logi- cal contract into the original version of the function and run CrossHair. If CrossHair identifies at least one counter- example triggering an Assert ionttror in any of the original functions, the set of contracts for the patch is con- sidered plausible. If no counter-example is found for any function, we regenerate the logical contracts and type an- notations for all functions, repeating this process up to maximum of Nregen iterations. If plausible contracts still cannot be generated, we prompt the LLM again with the issue description and original functions, and allow it to re- move branches and logic in the original functions that are irrelevant to the cost-conditions, and keep only the key logic. plausible contracts still cannot be generated after this step, we mark the instance as unverified.
not cause crashes. LLMs can better understand the issue and make decisions based PoC outputs. We also tried to use CodeQL to infer patch-related locations that needed to be changed together with the current patch, but it failed due to CodeQL's 's limited performance.
Complex prompting strategy. We tried Tree of Thoughts (ToT) (Yao et al., 2024) in generation, i.e., generating multi- plc candidates for each step of plan. While significantly increasing costs, this approach does not improve the re- solved rate given the LLMs cannot generate candidates with enough diversity for specific patching steps.
7. Conclusion and Future Works
We present PatchPilot, a stable and cost-efficient patching agent driven by rule-based planning workflow. We de- sign five important components and each element has its own customized designs. Our experiment demonstrates the effectiveness of PatchPilot and its individual compo- nents. This work points to a few promising directions for future work. First, we can explore combining agent-based planning and rule-based planning, which potentially can reach higher resolved rate with reasonable stability and cost-efficiency. Second, as discussed in Appendix C.4, number of failure instances are caused by the fact that LLMs tend to give overly simple patches. It is worth investigating fine-tuning specialized patching models from SOTA general LLMs which can better understand and reason about the issues and give more comprehensive patches.
Formal Verification. We insert the logical contracts and type annotations into each patched function and run CrossHair. If CrossHair cannot find any counter-example that satisfies the pre-conditions but violates the post- conditions for any patched function in the instance, we consider the instance verified.
We evaluate this procedure on the SWE-bench Lite with the o3 model with with Nregen 3, and formally verify eleven patches. This modest number is due to the current limitations of Python-basee formal verifiers, which struggle with intricate data structures and deeply nested or dynamic function calls.
6. Discussion
Resolved rates vs. stability and cost. As rule-based planning method, PatchPilot achieves a well-balanced trade- off between resolved rates, stability, and cost-efficiency. We believe that emphasizing stability and cost-efficiency is essential for a patching agent to be practical in real-world applications. Although PatchPilot is not on the top of the leaderboard, it has been shown be a stable and affordable method, confirming its practicality. Furthermore, PatchPilot outperforms all open-source methods on the leaderboard, establishing reasonably good performance.
A Acknowledgements
This work was supported in part by ARL Grant W91 INF. 23-2-0137. We gratefully acknowledge the support of FAR AI, OpenAI, and Berkeley RDI.
Impact Statement
PatchPilot advances automated software patching by intro- ducing stable and cost-efficient framework that balances performance with real-world practicality. While our method may achieve lower resolved rates compared to some closed- source solutions, we believe this tradeoff is justified by significantly reduced costs and improved stability critical factors for widespread adoption in production environments. Our rule-based planning approach helps prevent the unpre- dictability and high costs associated with fully agent-based systems while maintaining competitive patch generation ca- pabilities. However, we acknowledge that further research is needed to address the complex program logic understand- ing challenges. The framework's 's open-source source nature and emphasis on practical deployment considerations contribute to making automated patching more accessible and reliable
Static analysis vs. LLMs. We tried multiple static pro- gram analysis approaches in different components which were not effective and cannot outperform LLMs. First, we added function summaries for the functions in the root cause to the generation component. It improves the per- formance of PatchPilot with GPT-4o. However, it is not helpful when using more advanced models (o3-mini and Claude-3.5 3.5-Sonnet),, indicating that advanced models may be able to infer function behaviors. In validation, we tried to apply rule-based criteria to the PoC outputs to decide whether an issue is fixed. This is worse than using LLM as judge, given that many issues are "logical bugs" that do
9

---

PatchPilot: A Cost-Efficient Software Engincering Agent with Early Attempts on Formal Verification
for real-world software maintenance.
devlo. devlo, 2024. URL httttt:////ee///
devlo.
Feng, Y., Martins, R., Bastani, O., and Dillig, I. Program synthesis using conflict-driven learning, ACM SIGPLAN Notices, (3(4):420-435 2018.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. I.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT-4 Technical Report. arXiv preprint wrXiv:2333.08774 2023.
fireworks.ai. The fastest and most efficient inference en- gine to build production-ready, compound Al systems. https: wwr.worrewwweeeeer .ai 2024.
Ahmed, T. and Devanbu, P. Better patching using I1m prompting, via self-consisteney. In 2023 38th IEEE/ACM International Conference Automated Software Engi- neering (ASE), pp. 1742-1746, IEEE, 2023.
Gazzola, L., Micucci, D., and Mariani, L. Automatic soft- ware repair: A survey. Proceedings of the 40th Inter- national Conference on Software Engineering, pp. 1219- 1219, 2018
Aider. Aider, 2024.
URL https: https://aides /aie:e//aadee..eee htt.//aaierrre......
Aider.
Ghosal, D., Majumder, N., Mehrish, A., and Poria, S. Text- to-audio generation using instruction-tuned Ilm and latent diffusion model. arXiv preprint arXiv:2304.1377 2023.
Amazon. Amazon 9, 2024, URL https://aws
amazon. wwwooncccooooooooooooooooooooooo wtt.c//o//////eeeeeeeeeeeeee develloerr.... Amazon Q
Globant. Globant code fixer agent, 2024. URL https: //ai.globarc www./ossseii Globant.
Anthropic. Claude family, 2023. URL https://
claude claude model.
gru. gru, 2024. URL httttts:///gru.aii gru.
anthropic. Raising the bar on swe-bench ver- ified with claude 3.5 sonnet, 2024. URL https: //wp:://wwww anthropicc com research/ swe- ses--bench--omme sonnet. Claude_SWE
Huang, R., Li, M., Yang, D., Shi, J., Chang, X., Ye, Z.,
Wu, Y., Hong, Z.,, Huang, Liu, J., et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 23802-23806 23804, 2024.
Anthropic. Introducing the next generation of Claude. https:/ www.aaaithhhi..o. wttp:iiccco///es.....e 1laude-3-fame ly, 2024.
Antoniades, A., Örwall, A., Zhang, K., Xie, Y., Goyal, A., and Wang, W. Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement. arXiv preprint :XXY:24400220885 2024.
Huang, Z., Lie, D., Tan, G., and Jaeger, T. Using safety properties to generate vulnerability patches. In 20191EEB symposium on security and privacy (SP), pp. 539-554. IEEE, 2019.
Arora, D., Sonwane, A., Wadhwa, N., Mehrotra, A., Ut- pala, S., Bairi, R.. Kanade, A., and Natarajan, N. Masai:
IBM. Ibm research swe-1.0, 2024. URL https://
research. i bmm com ccom/blan hom/blll/// immmimmmso Com-swe-agents.
Modular architecture for software-engineerin ai agents. arXiv preprint ar arXiv:2406.1 /638, 2024.
IBM Research SWE-1.0.
Jiang, N., Lutellier, T., and Tan, L. Cure: Code-aware neural machine translation for automatic program repair. In 2021 IEEF/AC 43rd International Conference on Software Engineering (ICSE), pp. 1161-1173. IEEE, 2021.
CodeR. Coder, 2024. URL https: /tthtttthhbt....eg com/ NL2Code/CodeR. CodeR.
CodeStory. Codestory midwit agent, 2024. URL https: //aide. dev CodeStory.
Jimenez, E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language mod- els resolve real-world github issues? arXiv preprint ar arXiv:2310.06770, 2023.
Composio. Composio swe-kit, 2024. URL https:
ceflocs.compsiion. iilev/intre intro/overview Composio.
Le Goues, C., Pradel, M., Roychoudhury, A., and Chandra, S. Automatic program repair. IEEE Software, 38(4): 22-27, 2021.
deepinfra. Fast ML Inference, Simple API. https:// deepinfra.. ctm/ 2024.
DeepSeek. Deepseek-rl release, 2025. URL https:
Lian, L., Shi, B., Yala, A., Darrell, T., and Li, B. Lim-grounded video diffusion models. arXiv preprint arXiv:2309./74 /7444, 2023.
api-docs deepseek.com. wtwp://ccccc//ee//////s55555 DeepSeek-R1 published.
10

---

PatchPilot: A Cast-Efficient Software Engincering Agent with Early Attempts on Formal Verification
Liu, Y., Gao, P., Wang, X., Liu, J., Shi, Y., Zhang, Z., and Peng, C. Marscode agent: Ai-native automated bug fixing. arXiv preprint arXiv:2409. 00899, 2024.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal rein- forcement learning, Advances in Neural Information Processing Systems, 36, 2024.
Ma, Y. Cao, R., Cao, Y., Zhang, Y., Chen, J., Liu, Y., Liu, Y., Li, B., Huang, F., and Li, Y. Lingma swe-gpt: An open development-process, cente language model for automated software improvement. arXiv preprint arXiv:2411.00622 2024.
SWE-Bench. Swe-bench-lit leaderboard, 2023a. URL https:/ /tww//wwwww... swwbench. www.e/ccchcccccccoooommmo.eeeeee #1 lite. SWE- Bench-Lite.
SWE-Bench, Swe-bench-verrffed leaderboard, 2023b. URL https: /wwww///wwww/....o//./// swebench. com/ #verified. SWE-Bench-Veriffed.
moatless. moatless, 2024. URL ptttps//githiin com/ htmp//orwall///e httzwaalmmoatteesss tools. moatless.
Monperrus, M. Automatic software repair: A bibliography. ACM Computing Surveys (CSUR), 51(1):1-24, 2018.
Tao, W., Zhou, Y., Wang, Y., Zhang, W., Zhang, H., and Cheng, Magis: LIm-based multi-agent framework for github issue resolution. arXiv preprint arXiv:2403. 17927, 2024.
OpenAI. Hello gpt-4o, 2024a. URL https: /openai. com/index/helll gpt - 4o/. GPT-4o published.
Team, G., Anil, R., Borgeaud, S., Wu, Y., All.Ally,,JJaa J.-B., Yu,
OpenAL.
Announcement of openai o3, 2024b.
J., Soricut, R., Schalkwyk,, J., Dai, A. M., Hauth, A., et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
URL https::/ccomuuatty.opnnn c/dap=-12-off-shhhhmmmseeerrr-rp
10618187page=2. GPT-o3 announcement.
together.ai. The AI Acceleration AccelerationCloud. https: //w:://wwwww... together.e...... atciherrai//220 2024.
OpenAL.
Learning to reason with https://openas com/ /ndexx Learning-to-resson
LLMs.
Under, C. D. OpenAI's o3 ol: The Dawn of Hyper-Intelligent / //medium.com/////e// imedium..ca//cooniioownneee https:
1athhhllmms 11ms/. 2024c.
AI.
Ouyang, S., Yu, W., Ma. K., Xiao, Z., Zhang, Z., Jia, M., Han, J., Zhang, H., and Yu, D. Repograph: Enhancing ai software engineering with repository-level code graph. arXiv preprint arXiv:2410. 21.224410.4444... 2024.
incelligetttar-effffe22tubtt---e
Will 1a116//Honnnnnngg2222rreee222555 2024.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-D2 222822-28830, 2011.
Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., and Ji, Executable code actions elicit better Ilm agents. arXiv preprint arXiv:2402.00 2402.01030 2024a.
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNeeatian, N., Costa, B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. A study of generative large language model for medical research and healthcare. NPJ digital medicine, 6(1):210, 2023.
Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge M., Pan, J., Song, Y., Li, B., Singh, J., et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint 1XVV:24077.16411 2024b.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of- th-uuughhh prompting elicits reasoning in large language models. Advances neural information processing systems, 35:24824 24837, 2022.
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. Code lamaa: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
Ruan, H., Zhang, Y., and Roychoudhury, A. Specrover: Code intent extraction via llms. arXiv preprint
Xia, C. S., Deng, Y., Dunn, S., and Zhang, L. Agentless: De- mystifying llm-based software engineering agents. arXiv preprint arXiv:2407.0/ 00444048..49.. 2024.
arXiv:2408.02232. 2024.
Schanely, P. CrossHair: An analysis tool for python that blurs the line between testing and type systems, 2022. URL https: //g/tt/tb///////// com/ hann///ccanely///// CrossHair.
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. Swe-agent: Agent- computer interfaces enable automated software engineer- ing. arXiv preprint orXiv:2405. 15793, 2024a.

---

PatchPilot: A Cost-Efficient Software Engincering Agent with Early Attempts on Formal Verification
Yang, L.., Yu, Z.., Zhang, T., Cao, S., Xu, M., Zhang, W., Gonzalez, J. E.,, and Cui, B. Buffer of thoughts: Thought- augmented reasoning with large language models. arXiv preprint arXiv:2406.04211 2024b.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Zhang, K., Yao, W., Liu, Z., Feng, Y., Liu, Z., RN, R., Lan, T., Li, L.,, Lou, R., Xu, al. Diversity empowers in- telligence: Integrating expertise of software engineering agents. In The Thiirtennth International Conference on Learning Representations, 2024a.
Zhang, Y., Ruan, H., Fan, Z., and Roychoudhury, A. Au- tocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 1592- 1604, 2024b.
Zhu, Q.. Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R., Wu, Y., Li, Y., Gao, H., Ma, S., et al. Deepseek-coder- v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint w 2024.
12

---

PatchPilot: A Cost-Efficient Software Engincering Agent with Early Attempts on Formal Verification
A. Additional Technical Details
Tools provided in localization. For file-level localization, we provide LLM with three tools, which are search_fun___ search_class class _less_ee,,ddde and search_string. sarrch_fune, def and search class d_e take the function or class name as input and output the file containing the corresponding function or class definition based on exact matching, search_string takes a string as input and returns the file that contains this string the most times. If no file is found, we perform a fuzzy match with decreasing similarity threshold until a match is found or timeout. This is because the string searched by the LLM, often an error message in D, is typically a formatted string in the code and may not exactly match the one in D.. We chose to provide these three tools because they could cover most of the information provided in the issue descriptions.
Note that the use of these tools is predefined and incorporated into the file-level prompt at the start of the localization process, rather than being determined dynamically by the LLM. This ensures that our approach remains firmly rooted in rule-based planning.
Additional PoC information. We also attempt to identify the introduirg commit by finding the first historical commit where the PoC triggers the issue and its preceding commit. This is achieved by using a binary search to find the latest commit whose PoC output matches the current commit. Then, we prompt LLM to analyze the PoC output to determine the preceding commit produces expected results or triggers unrelated errors. If it is the former case, we consider we have found the isseoinnrouuuiig commit. We extract the code difference between the issue-introducing commit and its previous commit. This code difference provides valuable insights into how the bug was introduced, which is highly beneficial for both localization and generation. In Section 4.1, PatchPilot successfully identified the issue-introducing commit in 18 out of 300 instances on the SWE-Benehs Lite benchmark, among which 13 were resolved, achieving a resolved rate of 72.22%. In the SWE- WWE-Bench Verified benchmark, the issue- introducun commit was found in 39 instances, with 24 successfully resolved, achieving a resolved rate of 61.53% These high resolved rates further demonstrate the effectiveness of the issue-introducing commit.
Ranking criteria. We design our ranking criteria to prioritize the PoC test over benign functionality tests. The rank of a patch p is defined as
RK = 1 (PoC_failed) + WWle funelldddd_____eeee test
(I)
7wW70 eeeccutee__fuc__euee func_Jest
where RKp is the rank of the patch p. ffiledd is an indicator function determined by whether the PoC fails, namMraleed fundileed__fnc_eee lest the number of failed functionality tests, and SenPeeseeuedd_fuc__eeee is the number of executed functionality tests. We rank the patches based on the reverse order of RK₁, (i.e., the lower the RKp. the higher the ranking). If multiple patches have the same rank, we leverage an LLM to determine which one is the best.
B. Implementation
Reproduction. For reproduction, we set the iteration limit as 7 for the PoC generation. We generate at most one PoC for each instance. If a PoC is successfully obtained, we leverage the Python coverage package to get the files that are covered during the execution of the PoC. We provide LLM with R's directory tree structure and the issue description D;, directly prompting L.M retrieve three existing test files as benign functionality tests.
Localization. In localization, we perform a hierarchical workflow, including file level, class and function level and line level localization, followed by review step. At the file level, PatchPilot constructs a tree-structured repository representation, filtered based on the PoC coverage, retaining only the files executed during PoC execution. The issue description, the repository representation, and the set of tools described in Appendix A are then provided to the LLM,, prompting it to return the top 5 files most relevant to the issue. To achieve self-consistency (Ahmed & Devanbu, 2023), we perform fele- localization 4 times and perform majority voting to get the top 5 files. These files are inputs of the class and function-leved localization. At the class and function level, the LLM is provided with the signatures and comments of classes and functions extracted from the retrieved files and is prompted to identify functions and classes likely related to the issue. The number of functions and classes returned by LLM is not limited. At the line level, we provide the complete source code of the identified classes and functions to the LLM and prompt it to localize to specific lines. The class and function-level localization and the live- localization are both performed 4 times, with the results of each step merged. The localized lines, along with the surrounding code within a +1 +15-line range, are considered the root cause that will be provided to generation. At the end, we prompt L.M to perform review, If the root cause is less than 150 lines of code, we prompt L.MM to retrieve more code
13

---

PatchPilot: A Cost-Efficient Software Engincering Agent with Early Attempts on Formal Verification
Table 2. SWF:-Bech Verified leaderhoard excerpts from tools with more than 50% resolved rate
Tool
LLM
Resolved
Blackbox Ai Agent
NA
31.(62.80%)
CodeStory Midwit Agent swe-search
NA
(2.
Learn-by-interace
NA
301(60.20%)
devlo
NA
251.58.20%
Emergent
El
NA
286(57.20%)
Gru
Claude 3.5-Sonnet
285(57.00%)
HPAM AI/Run Developer Agent
NA
277(55.40%)
Amazon Q Developer Agent
NA
275(55.00%)
PatchPilot
Claude-3.5- 3.5--nnnet
268 (53.60%)
OpenHands CodeAct v2.
Claude-3.5-Sond
265(53.00%
Google Jules
Gemini 2.0 F
261(52.20%)
Engine Labs
NA
259(51.80%)
Agentless
Claude 3.5-ennen
104(50.80%)
Solver
NA
100(50.00%)
Bytedance MarsCode Agent
NA
100(50.00%)
snippets related to the root cause. Otherwise, we prompt LLM to check whether the current root cause is correct and fix it if LLM determines that it is incorrect.
Generation. For the experiments in Section 4, we set Nmax as 12, and N as 4. Thus, PatchPilot generates 4 plans and 4 patches in a single batch and can generate up to 12 patches for one instance. We constrain the maximum number of steps in plan to 3. For the first batch, we use the prompts as specified in Appendix D. generating one patch for the comprehensive-patcch prompt, one for the minimal-patch prompt and two patches for the standard-patel prompt. For the following batches, we only use the standard-pate -atth prompt. This is done to facilitate caching and reduce costs. Following previous work (Aider, 2024), we prompt LLM to generate patch in Search/Replace edit format and transfer them to git diff format by ourselves. For each edit generated for a step of plan, we utilize Python's 's ast library for syntax check and Flake8 for lint check. If LLM cannot generate an edit that passes both the syntax check and the lint check, we skip generation for the current step.
Validation. For the PoC test, we run the PoC, collect the outputs and provide it to LLM, along with D; and the PoC code. and prompt LLM judge whether the issue is fixed. For functionality tests, we implement parser that extracts the function names of the failed test from the test output. We assign each patch rank specified in Appendix A.
Refinement. For the refinement, if the current batch of patch pass no new functionality tests, we rerun localization in the next batch. For the file level localization, we force the LLM to return one file that is not within the previous localized files. For the function and class level localization and the line level localization, we force LLM to return code that is not within the previous localization results. If there is still no patch that passes any new test in the next batch, we discard the newly localized results and restore the original localization results before rerunning localization in the following batches.
C. Additional Experiments and Results
C.1. More Comparison on SWE-Bench.
Table 3 and Table 2 are about the current results on the SWE-Bench Lite/Verified, which show that PatchPilot has the highest performance among open-source tools and is competitive with closed-source tools in terms of resolved rate and cost.
12
C.2. More Analysis on Stablility Test.
Figure 4, Venn of resolved instances in rounds
Next, we go deep to analyze the resolved instances in each run in Sec- tion 4.2. Figure provides a Venn diagram comparison of the inter-
14

---

PatchPilot: A Cast-Efficient Software Engincering Agent with Early Attempts on Formal Verification
Table 3. SWE-Bench Lite leaderhoard excerpts
Tool
LLM
%Resolved
SAvg. Cost
Blackbox Ai Agent
NA
147 (49.00%)
Gru
NA
146 (48.67%)
Globant Code Fixer Agent
NA
145 (48.33%)
<$1.00
devlo
NA
142 (47.33%)
PatchPilot
-laude-3.--oonet
136 (45.33%
$0.97
Kodu-vl
134 (44.67%)
OpenHands+CodeAAee v2.1
Claude-3.5-Sonec
126 (41.67%)
$1.33
Composio SWE-Kit
NA
124 (41.00%)
Agentless
Claude-3.5 3.5nonet
123 (40.67%)
GPT-4o
96(32.00 %)
S0.70
Bytedance MarsCode Moatless
NA
118 (39.33%)
Claude-3.55 3.5-onee
115 (38.33%)
S0.30
GPT-4o
74(24.67 %)
$0.13
Honeycomb
NA
115 (38.33%)
AppMap Navie v2
NA
108 (36.00%)
Isoform
NA
105 (35.00%)
SuperCoder2.0
NA
102 (34.00%
Alibaba Lingma Agent
Claudc-3.5-Somee + Claude-3.5-Sonnet
99 (33.00%)
TodeShellTester
GPT-4o
94(31.33 %)
Amazon Q Developer-v2
NA
89 (29.67%)
SpecRover
GPT-4o + Claude-3.5-Sone 3.5-Sennet
93 (31.00%)
$0.65
CodeR
GPT-4
85 (28.33%)
$3.34
SIMA
GPT-4o
83 (27.67%)
S0.82
MASAI
NA
82 (27.33 %)
IBM Research A0e
NA
80(26.67 %
Aider
GPT-4o + Claude-3,m 3.adee-35-Somee
79(26.33 %)
IBM Al Agent SWE-1.00
NA
71(23.67 %
Amazon Q Developer
NA
61 (20.33%)
AutoCodeRover-v2
GPT-4o
92 (30.67%)
RepoGraph
GPT-4o
89 (29.67%)
Openhands+CodeAAct v1.8
Claude Claude-3.5-Somee 3.5-Sonnet
80(26.67 %)
$1.14
SWE-agent
Claude-3.5-Soma
69(23.00 %)
$1.62
GPT-4o
55 (18.33%)
$2.53
GPT-4
(18.00%)
$2.51
section and union of resolved instances across the three runs for both tools. PatchPilot achieved total intersection of 25 instances and union 38 instances, yielding higher intersection-to-uniee total intersection of instances and union of 25 instances, with lower intersection-to-uniie ratio of 0.48. The larger and more stable intersection instances for PatchPilot indicate its higher consistency in resolving instances across different runs. This stability is further emphasized by the Venn diagram, which visually demonstrates PatchPilot's ability to consistently resolve overlapping instances, underscoring its reliability and effectiveness.
ratio of 0.66. contrast, OpenHands had a
C.3. Failed Attempts with DeepSeek-r1.
During experiments with four platforms offering DeepSeek-rl model services (DeepSeek (DeepSeek, 2025), to- getherai (together.ai, 2024). firework.at (fireworks.ai, 2024), and deepinfra (deepinfra, 2024)), we encountered multiple barriers that prevented us from completing the necessary tests. Specifically, DeepSeek's 's own platform was affected by a DDoS attack, causing repeated service disruptions. Meanwhile, both together.ai and firework.ai exhibited severe re- sponse delays that rendered our experiments infeasible. In addition, deepinfra restricted the permissible context length for DeepSeek-rl to 16k tokens, falling short of our PatchPilot pipeline's requirements. Consequently, despite our attempts, we
15

---

PatchPilot: A Cost-Efficient Software Engincering Agent with Early Attempts on Formal Verification
unable to obtain experimental results for DeepSeek-rl under these conditions.
C.4. Case Study
Listing Diff of the issue-introducing commit of django-11284.
if not getattr (cls, sel f.a.ttame None)
setattr (cls, self.attname, self.descri www.dcccpptoooooooo patescrittor_cssssss (self))
self. caoices not None:
4
setattr 'get_ seret__s__disp___ dispLay' 8 self.name,
5
partialmethod(css. get FIELD display, field=selš))
if not hasattr 'get_ 8 di_plaay $ self.name): setattr(
8
cls, 'get_ 8s ias_ss___pplyy__ & self.name,
9
10
p gww.ggee_FFF___eeeee _isplay,, field-seli),
Listing 2. Our best patch in the first batch.
1
if not getattr (cls, sel f.attnmm,, None) setattr(cls self.attname, self. www.cdeecpppee wwwsrripoor_ccassseeeee (a81
2
if self.choices
None:
not hasattr(cis, 'get_ dissisyyay' & self.name): setattr(
5
cis, 'get_as_display' $ self.name,
7
pattialmsthllcccs.eee_Prr dislly,, field=self), setattr(
10
11
cis, _get_ss_displad self.name, htr.lalmethadlcls...ettt.... display, field=self),
14
Listing Our final patch for django-12284.
if getattr(cls, sel f.attname, None) cetattr(cls: self. if self.choices is not None:
self www.rdeeorr.c....cc.o class (selE))
3
method_name (cls, "get_ 8s display' & self.rame):
if not hasattr
self.name
if method_nane setattr(
cls. dict_
7
8
cls, "get_as_display" $ self.name, method_name, battiallettodlcls..ee..e.e. display, field=sele),
12
Listing 4. Golden patch for 122ngo-12284.
if not getattr self.attname, None): setattr (cls, self.attnane, self www.ddeestrrrceeeeeeeeeo class (selE)) if self.choices is not Nore:
3
4
if not hasattr (cls, 'get_ 8s_ display' $ self.
if 'got_es_display $ self.name not in cls. dict_ setattr(
7
cls, "get_As_display" self.name, partialmethod let Sree__FFII__diplaa field=self),
8
16

---

PatchPilot: A Cost-Efficient Software Engincering Agent with Early Attempts on Formal Verification
Successful Cases. Listing 1-4 4 shows example issue that was resolved by PatchPilot but rarely resolved by other methods in Table 3. Among the top 10 methods, only Blackbox Al Agent successfully solved this issue besides our approach.
Issue details: In Django, when model field defines choices, the get_xxx_ xxx_display method returns the label for field's value. However, if a subclass overrides the choices, the ger_xxx_ display method may not work correctly for the new choices. This happens because Django reuses the method from the parent class instead of creating a new one for the subelass, leading to incorrect results for the updated choices. The patch solution should ensure that Django generates a new get_xxx get_xxx_display method specifically for the subclass properly handle the overridden choices.
Findings: In our study of processing this issue, we discovered that PatchPilot 's two features critical to resolving the issue: (1) pinpointing the issue-intruducing commit to accurately localize the root cause and gain insight into the critical logic modifications directly related to the issue. and (2) leveraging feedback from failed testy to iteratively refine the patch until all tests are passed.
Listing 1 shows the diff of the issue- introduiuo commit, which closely matches the code area modified by the golden patch in Listing and clearly shows the logical changes that led to the issue.
In the first batch of patching, the best patch generated by PatchPilot is shown in Listing It patched the buggy code by directly reversely applying the diff in the Listing 1. This patch successfully fixes the buggy behavior; however, original functionalities are also affected, leading to failed functionality test.
In the second batch, PatchPilot retrieves the output of the failed functionalityy test and its code, performs refinement based on the patch in Listing 2. PatchPilot successfully fixes the broken functionality and generates a qualified patch that passes all tests, as illustrated in Listing 3, which is semantically equivalent to the golden patch shown in Listing 4.
This case study demonstrates the effectiveness of our proposed techniques in Section 3.
Failed Cases. Listing 5 shows an vulnerable code chunk in the widely used scikit-learn package (Pedregosa et al., 2011).
Listing 5. Example a vulnerable code chuck. best_indices = argarg (scores, axis=1)
2
if self.multis www.timtlllaaaaaa == 'ovr':
ap..pamma [ccoefs_ppae paths [i, best Dest_indicessii [i], for i range (Len axis=0)
Lines 3 and raise an IndexError because best mest_indicee [ [i] may exceed the size of the second or third dimension of coefs pattss..hh.___ After feeding this chuck with exact vulnerable line numbers and the correct bug description into the SOTA GPT-4o and Claude-3.5. Sonnet models, both suggested plan to constrain the indexing on Lines 3 and 6 using the modulo operations to pre- vent IndexError. Following this plan, the models generated patches that change best_indices [i] in Line 3 and to fess__nndice ces [i] Scoefs sacoefs__aths___ee shape [1] and best sest_iindiessseeee [i] Scoefs acoefsspathss shape [2]. Al- though this patch avoids the IndexError, it also breaks the original computation logic of the program. This example demonstrates that SOTA LLMs still fall short in understanding program logic and reasoning about vulnerabilities. Without deep understanding, these models tend propose simple patches that either cannot resolve the problem harm the functionalities.
else:
6
- np..mmen ( ppooefs_paths::: i, Dast_indicest :] for i in range (len (folds)) axis-0)
17

---

PatchPilot: A Cost-Efficient Software Engincering Agent with Early Attempts on Formal Verification
D. Prompts
D.I. Standard-Patie Patch Prompt
System Prompt:
You are experienced software maintainer responsible for analyzing and fixing repository issues. Your role is to:
1. Thoroughly analyze bugs to identify underlying root causes beyond surface-level symptoms.
2. Provide clear, actionable repair plans with precise code modifications.
Format your repair plans using:
<STEP> and </STEP> tags for each modification step.
<Actions to be Taken> and </Actions to be Taken> tags for specific actions.
Maximum 3 steps, with each step containing exactly code modification.
Only include steps that require code changes.
the issue text includes a recommended fix, not apply directly. You should explicitly reason whether it can fix the issue. Output the reason that the recommended can or cannot fix the issue. You should explicitly reason whether the recommended fix keeps the same code style and adapt align with the codebase's style and standards. Ensure that the patch considers interactions across different code sections, including nested structures, function calls, and data dependencies.
The patch should maintain overall structural integrity, addressing the issue without unintended effects on other parts. Propose solutions that are resilient to structural changes or future extensions.
User Prompt:
You are required to propose plan fix issue.
Follow these guidelines:
Number of Steps:
The number of steps to fix the issue should be most 3. Modification: Each step should perform exactly one modification at exactly one location in the code.
Necessity: Do not modify the code unless it is necessary to fix the issue.
Your plan should outline only the steps that involve code modifications. a step does not require a code change, do not include the plan.
not write any code in the plan.
[format_em examplee
Here is the issue text: BEGIN ISSUE-
(problem stt.sttttttttttt
END ISSUE-
Below some code segments, each from relevant file. One more of these files may contain bugs.
BEGIN FILE
(content) }
END FILE
D.2. Minimal-Patel Prompt
The following prompt was used to generate plan that fixes the issue with minimal modification. The additional prompt, compared the standard-patch prompt in Appendix D.1, is highlighted blue.
18

---

PatchPilot: A Cost-Efficient Software Engincering Agent with Early Attempts on Formal Verification
System Prompt:
You are an experienced software maintainer responsible for analyzing and fixing repository issues. Your role is Thoroughly analyze bugs identify underlying root causes beyond surface-level symptoms.
2. Provide clear, actionable repair plans with precise code modifications.
3. In fixing the bug, your focus should be making minimal modifications that only target the bug- sug-triggering scenario without affecting other parts of the code or functionality.
Make sure that other inputs or conditions are not impacted. Modify only the specific behavior causing the bug, and not make any broad changes unless absolutely necessary. Format your repair plans using:
<STEP> and </STEP> tags for each modification step.
<Actions to be Taken> and </Actions to be Taken> tags for specific actions.
Maximum 3 steps, with each step containing exactly one code modification.
Only include steps that require code changes.
the issue text includes a recommended fix, not apply directly. You should explicitly reason whether it can fix the issue. Output the reason that the recommended can or cannot the issue. You should explicitly reason whether the recommended fix keeps the same code style and adapt align with the codebase's style and standards. Ensure that the patch considers interactions across different code sections, including nested structures, function calls, and data dependencies.
The patch should maintain overall structural integrity, addressing the issue without unintended effects on other parts. Propose solutions that are resilient to structural changes or future extensions.
User Prompt:
You are required to propose plan fix issue.
Follow these guidelines:
Number of Steps:
The number of steps to fix the issue should be most 3.
Modification: Each step should perform exactly one modification at exactly one location in the code.
Necessity: Do not modify the code unless it is necessary to fix the issue.
One File Only: Choose one file to modify, and ensure all changes are limited to that file.
Concentration on Input: If the issue mentions a specific input or argument that triggers the bug, ensure your solution only fixes the behavior for that input.
Your plan should outline only the steps that involve code modifications. step does not require code change, do not include it in the plan.
Do not write any code in the plan.
( (format_exxx _xammpleee
Here the issue text:
BEGIN ISSUE-
(problem st statement)
END ISSUE-
Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
BEGIN FILE
(content) }
END FILE
D.3. Comprehensive-Patch Prompt
The following prompt was used to generate a plan that comprehensively fixes the issue. The additional prompt, compared to the stancard-patch prompt in Appendix D.1, highlighted in red.
19

---

PatchPilot: A Cost-Efficient Software Engincering Agent with Early Attempts on Formal Verification
System Prompt:
You are an experienced software maintainer responsible for analyzing and fixing repository issues. Your role is Thoroughly analyze bugs identify underlying root causes beyond surface-level symptoms.
2. Provide clear, actionable repair plans with precise code modifications.
3. The plan should ensure that the solution is general, preventing any similar bugs from occurring in other contexts or input variations.
The plan will be used to generate a comprehensive and extensive patch that addresses the issue thoroughly, modifies related areas to ensure the bug is fully resolved, and enhances the overall robustness and completeness of the solution.
Format your repair plans using:
<STEP> and </STEP> tags for each modification step.
<Actions to be Taken> and </Actions to be Taken> tags for specific actions.
Maximum 3 steps, with each step containing exactly one code modification.
Only include steps that require code changes.
issue text includes a recommended fix, not apply directly. You should explicitly reason whether can fix the issue. Output the reason that the recommended fix can or cannot fix the issue. You should explicitly reason whether the recommended fix keeps the same code style and adapt align with the codebase's style and standards. Ensure that the patch considers interactions across different code sections, including nested structures, function calls, and data dependencies.
The patch should maintain overall structural integrity, addressing the issue without unintended effects on other parts. Propose solutions that are resilient to structural changes or future extensions.
User Prompt:
You required to propose plan to fix issue.
Follow these guidelines:
Number of Steps:
The number of steps to fix the issue should be at most 3. Modification: Each step should perform exactly one modification at exactly one location in the code.
Necessity: Do not modify the code unless is necessary to fix the issue.
Broad Solution: Your solution should aim to resolve the issue broadly and comprehensively, covering edge cases
and general patterns.
No assumptions about input: Avoid making assumptions based solely on the example given in the issue description. If the issue description mentions specific cases (e.g., particular input argument), the fix should be applicable to all possible cases, not just the ones listed.
( (lormat_exaape
Here the issue text:
BEGIN ISSUE-
(problem_ st statement)
END ISSUE-
Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
BEGIN FILE
content)) }
END FILE
20


Reviews:
[{"review_id": "HUX516RC3h", "reviewer": "Reviewer_Lckx", "summary": "In this paper, the authors describe PatchPilot, a novel human-based planning workflow for solving Github issues. The innovations include generating reproduction tests to help locate the root cause; a planning and generation task division for patch generation, and a refinement loop to iteratively improve a patch. Empirical evaluations on SWE-Bench showcases PatchPilot’s efficiency and stability in resolving Github issues compared to baseline methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive and constructive comments. \n\n## 1. Stability Comparison\n\nWe chose GPT-4o because of our budget limits before. We reran the stability comparison. First, we changed GPT-4o to Claude-3.5-Sonnet. Second, in response to Reviewer HrpY’s concern about small sample size, we increased the testing cases from 45 to 100 from SWE-bench-Lite. These cases were selected from three categories based on these methods’ results in one previous run: 30 are commonly resolved by all tools, 15 are solved independently by each of the three tools, and 55 are commonly unresolved cases (hard ones). The cases in each category are randomly selected. Third, we added Agentless based on Reviewer vPVX’s suggestion. We calculated the standard deviation (STD) of the resolved rate and cost. To show the statistical significance, we conducted a Bartlett’s test to compare the variance between PatchPilot and the comparison baselines. Our Null hypothesis $H0: \\sigma^2_p = \\sigma^2_b$,  $\\sigma^2_p$ and $\\sigma^2_b$ are the variance of PatchPilot and baseline. The results are as follows (the first number in p-value is the comparison of resolved rate, and the other is the cost). \n\n|      | Round 1  | Round 2  | Round 3  | Average Time | STD  | $p$-value |\n|------------|------------------|------------------|------------------|--------------|-------------|-----------|\n| PatchPilot | 38/100 ($98.23)  | 38/100 ($99.60)  | 39/100 ($97.88)  | 87 min  | 0.58($0.91) | -  |\n| OpenHands  | 31/100 ($156.58) | 35/100 ($160.38) | 39/100 ($169.58) | 90 min | 4.00($6.68) | 0.044/0.040 |\n| Agentless  | 35/100 ($117.75) | 33/100 ($120.31) | 34/100 ($117.02) | 102 min  | 1.00($1.73) | 0.50/0.43 |\n\nThe result confirms that PatchPilot has a much lower STD than OpenHands in the resolved rate. Besides, the $p$-value is smaller than 0.05, meaning we can reject the null hypothesis. Given that the STD of PatchPilot is smaller than OpenHands, this shows that PatchPilot is more stable than OpenHands with statistical significance. The resolved rates of all methods are lower than those in Table 1 because we include more hard cases in the 100 cases. The cost shows a similar trend. OpenHands is relatively more cost-effective when using Claude-3.5 compared to using GPT-4o in Section 4.2, as a stronger model can reduce the number of iterations of OpenHands. \n\nThe result also shows the stability of Agentless, validating our claim that human-based planning is more stable than agent-based planning. Finally, PatchPilot consistently achieves lower cost and higher performance than Agentless.\n\n## 2. Illustrative Example of Patch Planning and Generation\n\nWe will include more examples. Here, we show one example of django__django-11133. Django is a Python web framework. In this issue, the user reported that HttpResponse failed to handle memoryview objects.\n\n**PatchPilot generated a patch plan with one step:**\n\n<STEP> Add an explicit memoryview check in the content setter </STEP> \n<Actions> In the file django/http/response.py, locate the content setter of HttpResponse. Modify the code so that it first checks if the provided value is an instance of memoryview. If it is, immediately return bytes(value). Otherwise, proceed with the existing check for iterables. </Actions>\n\n**Following the plan, PatchPilot generates a correct patch:**\n```python\n         # - Python attempts str conversion first\n         # - when self._charset != 'utf-8' it re-encodes the content\n-        if isinstance(value, bytes):\n+        if isinstance(value, (bytes, memoryview)):\n             return bytes(value)\n```\n## 3.PoC Validation Beyond LLM Checking\n\nWe first convert the issue description into a structured format, extracting the expected behavior and wrong behavior. We also include the following rule-based checks. \n\n**Rule 1:** Discard PoCs that cover fewer than 3 files, as LLMs often create fake buggy functions inside the PoC instead of invoking real project code.\n\n\n**Rule 2:** If a specific exception is mentioned in the issue, we check if the same exception appears in stderr during PoC execution, confirming it triggers the described error.\n\nTo evaluate PoC quality, we manually reviewed 100 cases. Our validation filtered out 27 PoCs (20 by LLM, 5 by Rule 1, and 2 by Rule 2). Of the remaining 73 PoCs, 92% are correct. These results confirm the effectiveness of our patch validation and the high quality of generated PoCs.\n\n## 4. Majority Voting Strategy in Ablation Study\n\nIn the first two variants, we use majority voting since patch validation is not feasible without PoCs or functionality tests. We generate multiple patches per case and proceed as follows: Normalize each patch by stripping whitespace, line breaks, and comments; count the frequency of each normalized patch; and select the most frequent one. If there is a tie, we prompt an LLM to choose the best patch.\n\n## 5. Novelty of Refinement in PatchPilot\n\nWe discuss the novelty of our refinement in **response-1 to reviewer HrpY.**", "claims_and_evidence": "The claims are well-supported by clear evidence. The authors provided overall performance evaluations on SWE-Bench Lite and Verified set. Table 1 shows that PatchPilot strikes a balance between resolution rate and cost. Section 4.3 provided useful information on each component.", "methods_and_evaluation_criteria": "SWE-Bench is a standard benchmark dataset for measuring software agents so it makes sense the authors chose it to validate their work.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Most experimental designs are sound. I have one comment for Section 4.2, why did the authors present the PatchPilot vs OpenHands comparison results with GPT-4o instead of Claude 3.5 Sonnet? Considering both methods are reported with Claude 3.5 Sonnet in Table 1, this mismatch should have some explanation.", "relation_to_broader_scientific_literature": "Key novelties are: using reproduction tests to help localization; planning and generation separation for patch generation, and patch refinement. Some versions of the first two are present in other agent based works such as OpenHands. One could argue OpenHands' iterative approach has the refinement element too as a subset of history is retained in prompts. So the significance of these novelties are not super strong.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See sections above.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Can the authors provide an example plan generated for the patch generation step? And a corresponding example with LLM following the generated plan.\n\n2. As the PoC is used to help locate the root cause, its validity is crucial. Besides including instructions in the prompt, did the authors carry out other mechanisms to verify their validity?\n\n3. Majority voting is mentioned in Section 4.3, can the authors provide more information on how the majority voting is carried out?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IA1A24SZWm", "reviewer": "Reviewer_HrpY", "summary": "This paper proposes PatchPilot, an agentic framework for autonomous software patching. It relies on human-based planning and consists of five workflow components: reproduction, localization, generation, validation, and refinement. The overall workflow as well as each component (except for the final refinement step) closely resembles prior work Agentless. The key difference being:\n\n- During generation, PatchPilot explicitly prompts the model to output a multi-step patching plan.\n- Instead of regenerating patches, PatchPilot refines existing ones based on validation feedback.\n- Prompt engineering: The model is guided to generate diverse patches by explicitly prompting for both simple and complex solutions.\n\nExperiments conducted on SWE-bench (Lite and Verified) show that PatchPilot outperforms SOTA open-source methods while having the lowest cost. Results also show that PatchPilot is more stable than the SOTA agent-based planning method OpenHands.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comments. \n\n## 1. Novelty and Differences from Existing Tools\n\nAs discussed in Section 2, [a-g] are all agent-based, differing from our workflow (we included [b-g] and will add [a]).\n- Search tools: We acknowledge that AutoCodeRover and CodeR also have search tools, and we are inspired by their designs. The key differences are in the search mechanisms, where we use fuzzy search instead of exact search. As the searched strings may differ from the original code segments, the exact search misses some matches.\n- Coverage: AutoCodeRover's SBFL method requires post-patching test cases, which are unavailable in SWE-bench problem settings. PatchPilot generates coverage using only PoCs. \n- Separating planning and generation: We respectfully point out that AppMap Navie is a commercial product without research papers. Although it mentions having generation plans, the technical details are not available. Our separation design was inspired by our early observations of shallow patches. Besides, the detailed design can be very different. For example, we proposed diversity prompts and step-by-step generation. We believe it is reasonable to claim the novelty and originality of our design of the generation component. \n- Patch validation vs. Agentless: Agentless generates assertions for testing cases and validates based on assertions. It is hard to generate correct and comprehensive assertions. PatchPilot generates only PoCs and uses LLMs to validate based on PoC outputs. We applied both methods to 100 patches (50 from PatchPilot, 50 from Agentless). Our method identified 43 correct patches; Agentless identified 40.\n- PatchPilot differs from Agentless in the workflow (**refinement**) and applies optimizations to each component. Although the components' high-level ideas are similar, the specific designs can vary a lot and affect final performance, as shown in the comparison with Agentless.\n- About refinement: Agent-based tools refine the latest patch by prompting the LLM with prior conversation history. This simple method has a few limitations and cannot be applied to PatchPilot: 1) It generates only one patch in the first round and iteratively refines it, which lacks diversity. If the initial patch contains errors, it may be hard to fix. This is partially validated in our hyper-parameter experiments with a batch size of 1, showing a performance drop. 2) It cannot revert to earlier patches if a refinement worsens the result, while PatchPilot enables such a capability. 3) By including the refinement history in prompts, it risks context overflow and potential LLM confusion from prior bad patches. PatchPilot instead uses only the best previous patch and its validation results. Given these major differences, we believe it is reasonable to claim novelty for our refinement.\n\nWhile existing tools may share similar high-level ideas, our design is fundamentally different and more advanced, giving better performance and stability. We believe this brings essential novelty and contribution.\n\n## 2. Extra Ablation Studies for Hyperparameters\n\nWe conducted experiments to evaluate hyper-parameter sensitivity. PatchPilot includes the following hyper-parameters (default values in brackets): number of retrieved files (5), total patches generated (12), batch size per round (4), diversity prompts (3), and generation temperature (0.8). Each experiment varies one hyper-parameter while keeping others. We used 60 cases randomly selected from SWE-bench-Lite and Claude-3.5-Sonnet.\n- Num. of retrieved files in localization: We changed it from 5 to 3 and 7. Varying it introduces subtle differences, but retrieving more files will have a higher cost as LLM will have more conversations and longer contexts. \n- Total num. of patches: Changing it from 12 to 1, 4, 8, 16. A large one slightly improves performance but raises costs. 12 strikes a balance between performance and cost.\n- Batch Size: Changing the batch size from 4 to 2/6 introduces performance drops. For 2, the diversity is not enough; for 6, the refinement round (2) is not enough. A moderate batch size balances the refinement rounds and diversity.\n- Diversity Prompts: Instead of using three prompts together, we ran the experiment with each prompt separately. Mixing them gives the best performance as it enables the highest diversity.\n- We varied the temperature ($\\tau$) from 0.8 to 1 and 0.6. The results are similar. We set it high to encourage diversity.\n\n| Variants|Number|Resolved|Cost ($)|\n|---------|---------|-----------|----------|\n| Default| - |28/60|58.27|\n|Retrieved files| 3|24/60|50.38|\n|   | 7|27/60|64.23|\n|#. patches|1|20/60|30.96|\n| | 4|26/60|39.95|\n| |8|27/60|51.02|\n| |16|28/60|62.88|\n| Batch| 2|24/60|62.93|\n| |6|25/60|55.77|\n|Prompt|Standard|26/60|57.67|\n| | Big| 25/60| 61.17|\n| | Small| 22/60| 55.32|\n|$\\tau$| 0.6|27/60| 58.21|\n| | 1.0|28/60|58.02|\n\n## 3. Stability Evaluation\n\nOur stability experiment is in **response-1 to reviewer Lckx.**", "claims_and_evidence": "The paper presents an incremental improvement rather than a novel contribution, integrating existing tools and iterative refinement workflows from prior agentic work into the Agentless workflow without adequately acknowledging previous research.\n\n**Overall Design**\n\nThe approach closely follows the Agentless workflow, with the first four stages - reproduction, localization, generation, and validation - remaining largely unchanged. The primary modifications, such as searching tools, are also borrowed from prior work and are not novel. \n\nThe paper claims the Refinement is a unique component to PatchPilot, but this is not the case. Early LLM-based automated program repair work, such as ChatRepair [a], has already demonstrated that test execution feedback can refine previous patches more efficiently than regenerating them from scratch. Multi-agent frameworks like MarsCode [b] and SpecRover [c] similarly refine the generated patch based on test execution results. Also, existing agentic tools like OpenHands [d] enable LLMs to autonomously refine the PoC reproduction scripts and patch during their interactions - an approach commonly seen in their repair trajectories.\n\n\n**Other Technical Designs Inspired by Prior Work**\n\nThe other technical designs also closely resemble prior work from the same research area; however, the authors do not acknowledge these contributions and instead present them as if they were their own novel designs.\n\n- Search tools: Similar tools have been proposed in AutoCodeRover [e] and CodeR [f], yet the authors did not acknowledge these prior contributions and present them as if they were their own novel design.\n\n- Coverage information: AutoCodeRover [e] optionally leverages coverage analysis, such as Spectrum-based Fault Localization, for context retrieval.\n\n- Separating planning and generation: The separation of planning has been explored in prior work, such as AppMap Navie [g].\n\n- Patch validation strategy: The strategy is nearly identical to Agentless, except for prioritizing patches that pass PoC tests over functionality tests, whereas Agentless prioritizes patches that pass functionality tests over PoC tests. However, the authors do not clarify that this strategy is a modification of existing work.\n\n\n[a] Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. (ISSTA 2024)\n\n[b] MarsCode Agent: AI-native Automated Bug Fixing. (ArXiv 2024)\n\n[c] SpecRover: Code Intent Extraction via LLMs. (ICSE 2025)\n\n[d] OpenHands: An Open Platform for AI Software Developers as Generalist Agents. (ArXiv 2024)\n\n[e] AutoCodeRover: Autonomous Program Improvement. (ISSTA 2024)\n\n[f] CodeR: Issue resolving with multi-agent and task graphs. (ArXiv 2024)\n\n[g] AppMap Navie: https://appmap.io/blog/2024/06/20/appmap-navie-swe-bench-leader/", "methods_and_evaluation_criteria": "There are many hyperparameters that have not been examined through ablation studies, such as the iteration limit for refinement, the number of retrieved files, the number of plans and patches for a single instance. It is also unclear how sensitive the technique's performance and cost are to these parameters.\n\nMoreover, some key hyperparameters, such as the sampling temperature, are not even clearly specified.\n\nUpdate after rebuttal:\n\n* Ablation study shows that prompt engineering like the diverse-prompting design may be a main driver of improvement. I am not fully convinced by the main claim of this paper.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "In Section 4.2, the authors compare PatchPilot with OpenHands on a subset of 45 problems, repeating the experiments three times and reporting standard deviation. However, I wonder if this is statistically significant enough to support a strong conclusion.", "relation_to_broader_scientific_literature": "Overall, the contribution is insignificant compared to prior literature:\n\n- The proposed technique mainly integrates agentic designs (such as tool use and iterative refine loops) into an agentless framework, offering little novelty.\n\n- Observations on the cost-effectiveness and stability of human-based planning versus agent-based planning have been made before, as have insights on the effectiveness of refining existing patches compared to generating new ones.", "essential_references_not_discussed": "Essential related works are cited but not adequately discussed in the paper. (see above)", "other_strengths_and_weaknesses": "Other strengths:\n\n- Outperforms SOTA open-source methods while having the lowest cost", "other_comments_or_suggestions": "- “SWE-Bench” should be “SWE-bench”", "questions_for_authors": "1. How much improvement does prompt engineering contribute (specifically, by using three different prompts to generate simple, complex, and standard fixes) ? The ablation study did not isolate this design.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oS41uStqbi", "reviewer": "Reviewer_vPVX", "summary": "The paper proposes PatchPilot, an agentic patching framework designed to address the trade-offs among patching efficacy, stability. It introduces a novel human-based planning workflow, incorporating 6 components, with special emphasis on refinement as a unique contribution.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive and positive comments. \n\n## 1. Better experiment analysis \n\nWe will follow the reviewer’s suggestion and include more visualizations to show the comparison between our method and baselines in effectiveness, stability, and cost. To better demonstrate the advantage of PatchPilot over Agentless, we reran these two methods on 100 cases three times (the selection of these cases is stated in response-1 to reviewer Lckx) and reported their resolved rate, average cost, and run time. \n|   | Round 1          | Round 2          | Round 3          | Average resolved | Average cost         | Average time |\n|--------|------------------|------------------|------------------|--------------|-------------|-----------|\n| PatchPilot | 38/100 ($98.23)  | 38/100 ($99.60)  | 39/100 ($97.88)  |    38.33   | 98.57 | 87 min   |\n| Agentless  | 35/100 ($117.75) | 33/100 ($120.31) | 34/100 ($117.02) |   34   | 118.36 | 102 min |\n\nThe results show that PatchPilot consistently outperforms Agentless in all three metrics.\n\n\n## 2. Mitigating Potential Instability Introduced by Iterative Refinement\n\nThanks for pointing this out. We incorporate two strategies to improve stability during refinement.\n\n**(1) Batch-based Refinement with Selection Mechanism:** Instead of generating a single patch per iteration by directly modifying the previous one, PatchPilot generates multiple patch candidates in each batch based on the previous best patch. It then evaluates all candidates—together with the previous best patch—using both the PoC and functionality tests and selects the best-performing one. If all current candidates underperform compared to the previous best patch, the previous best patch is resolved. This mechanism significantly reduces the risk of accumulated generation noise and protects against overwriting correct earlier solutions (validated by our hyper-parameter experiment when we reduce the batch size from 4 to 1, the overall performance drops notably).\n\n**(2) Early Stopping:** As soon as a patch passes all tests, iterative refinement is terminated immediately, and the patch is returned. This prevents unnecessary iterations without useful feedback, avoiding potential degradation from further blind refinement.\n\n## 3. Failure Cases of PatchPilot\n\nWe will include more analysis of failure cases. In summary, some of the failure cases are because the issue descriptions are of low quality, such as incomplete or incorrect descriptions of the issues. To resolve these cases, we need to rewrite high-quality issue descriptions. Some other cases are shallow patches, where PatchPilot only patches the superficial logic instead of addressing the root cause and does not consider the corner case of the issue, or over-patching, where patches inadvertently affect more scenarios than necessary, causing unexpected behavior. These cases may be further resolved by fine-tuning specific models that can better understand the program logic of the target repos.", "claims_and_evidence": "Their claims, supported by extensive empirical evidence and rigorous evaluation on SWE-Bench performance.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Not much theoretical claims.", "experimental_designs_or_analyses": "Authors could enhance the analysis by including more intuitive visualizations, such as detailed plots illustrating trade-offs in stability and cost. Why it surpassed agentless in a clear way.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "Should add more related work.", "other_strengths_and_weaknesses": "Strengths: See above\n\nWeaknesses: \n\nThe performance is good, but the authors should provide more evidence to support their claims, as their method uses fewer tokens yet achieves good results. However, the experimental analysis does not fully substantiate their claims and lacks several important insights.", "other_comments_or_suggestions": "Could you elaborate on how PatchPilot specifically addresses or mitigates the potential instability caused by iterative refinement?\n\nThe manuscript could provide a deeper discussion of the scenarios where PatchPilot may fail or perform sub-optimally.", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2v5Pzr4NCG", "reviewer": "Reviewer_Sjbd", "summary": "This paper presents an improvement to the \"Agentless\" approach to solving SWE-bench tasks. They manager to solve 3-5% more problems on SWE-bench Lite/Verified while using up to 20% *less* money, with Claude 3.5. They provide detailed analysis and ablations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive and positive comments. \n\n## 1. Extra Clarification on Terminology: \"Human-Based Planning\"\n\nWe thank the reviewer for pointing this out. We agree that the term \"human-based planning\" may be misleading, as it could imply a human-in-the-loop system. To avoid this confusion, we will revise it to \"Rule-based planning\", which more accurately reflects our intent—i.e., that the plan is rules pre-specified by developers.\n\n## 2. Wall-Time Comparison with Agentless\n\nWe followed the reviewer’s suggestions and conducted an experiment to compare the wall-time of our system with Agentless and OpenHands on 100 cases (the selection of these cases is stated in the response-1 to reviewer Lckx). Since all tools support multiprocessing, we controlled the number of processes to be the same for all methods (8) to ensure a fair comparison. We measured the average time per instance and ran the experiment three times. The results are as follows. \n\n| Stability test | Round 1 |  | Round 2 |  | Round 3 |  | Average Time |\n|----------------|---------|--------|---------|--------|---------|--------|--------------|\n| PatchPilot | 38/100 | 85min | 38/100 | 86min | 39/100 | 89min | 87 min |\n| Openhands | 31/100 | 86min | 35/100 | 85min | 39/100 | 99min | 90 min |\n| Agentless | 35/100 | 102min | 33/100 | 103min | 34/100 | 100min | 102 min |\n\nThe results show that PatchPilot is faster than agentless and OpenHands. It also shows that PatchPilot is more effective than the baseline approaches.", "claims_and_evidence": "Yes, there is a wide array of experiments that were run in this work to support the claims.", "methods_and_evaluation_criteria": "Yes SWE-bench is the leading method to test software engineering agents, and obtaining *higher* SWE-bench scores at *lower* cost is a sign of a substantial improvement.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "Yes, as detailed in the SWE-bench guide, the only major issues that can happen with SWE-bench evaluations is either submissions using the 'hints' column which should not be used, or submissions using knowledge about fail2pass or pass2pass tests before submitting. I saw no evidence for either of these things being done in this work.", "relation_to_broader_scientific_literature": "There's two lines of work on how to solve swe-bench-style tasks: SWE-agent-like models and Agentless-like models. The first usually get higher scores but are very expensive, the Agentless models usually do slightly worse but are very cheap. This paper is a clear contribution to the second type of models, which are very important.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "I think this paper is very strong overall: great results, easy to read, good analysis, very interesting discussion section.", "other_comments_or_suggestions": "I would maybe not call it 'human-based planning'. Kinda makes it seem like you have a human-in-the-loop \"Devin\" like system where the human is planning and the agent is executing. Can you find a phrase for this that doesn't have the word 'human' maybe?", "questions_for_authors": "Is your system's wall-time better than Agentless?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary:** The paper proposes PathPilot with a human-based planning workflow composed of five components of reproduction, localization, generation, validation and refinement. The generation component also has a planning stage and the refinement component is designed to be iterative and used together with feedback from the validation component. Distinct from existing methods, the proposed approach focuses on reproducing the issues and retrieves relevant tests to be used in validation. The proposed approach aims to balance efficacy, stability and cost. Evaluations using SWE-bench demonstrate this balance.

**Strengths:**
- Reviewers acknowledge that the proposed method outperforms SOTA and reduces cost. This is in general accepted to be an important milestone and will be of interest to the ICML community. 
- Reviewers note that the evaluations use the standard benchmark for this domain, namely SWE-Bench. Having said that, there are mixed signals by reviewers around the design of experiments in terms of supporting the claims of the paper, with three out of four reviewers supporting the experiment design and commenting on the rigor of the evaluations.
- The paper is generally well written, organized and easy to read.

**Weaknesses:**
- There are outstanding concerns around the broad significance of the paper compared to the literature. Reviewers commented on the incremental nature of contributions compared to existing on Agentless and OpenHands and noted that novelty is limited.

**Recommendation:**
- Three out of four reviewers recommended acceptance of the paper with scores of 3,3 and 4. The remaining reviewer recommended a weak rejection (with a score of 2).
- I believe that the paper’s strengths on strong performance on SWE-bench through a combination of human-based and agent based planning approaches and iterative refinement with an emphasis on validation outweigh the weaknesses on limited novelty. I also believe this paper will be of interest to the ICML community as it is a timely contribution advancing agentic approaches for software engineering applications. Therefore, I recommend acceptance of this paper.
Decision: Accept (poster)