MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution
Authors: Wei Tao
Yucheng Zhou
Yanlin Wang
Wenqiang Zhang
Hongyu Zhang
Yu Cheng
Venue: NeurIPS 2024 poster (2024)
Keywords: Code Change
LLM Agent
Software Evolution
Primary area: other

Abstract: In software development, resolving the emergent issues within GitHub repositories is a complex challenge that involves not only the incorporation of new code but also the maintenance of existing code.
Large Language Models (LLMs) have shown promise in code generation but face difficulties in resolving Github issues, particularly at the repository level. 
To overcome this challenge, we empirically study the reason why LLMs fail to resolve GitHub issues and analyze the major factors. 
Motivated by the empirical findings, we propose a novel LLM-based **M**ulti-**A**gent framework for **G**itHub **I**ssue re**S**olution, **MAGIS**, consisting of four agents customized for software evolution: Manager, Repository Custodian, Developer, and Quality Assurance Engineer agents. 
This framework leverages the collaboration of various agents in the planning and coding process to unlock the potential of LLMs to resolve GitHub issues. 
In experiments, we employ the SWE-bench benchmark to compare MAGIS with popular LLMs, including GPT-3.5, GPT-4, and Claude-2. 
MAGIS can resolve **13.94%** GitHub issues, significantly outperforming the baselines.
Specifically, MAGIS achieves an eight-fold increase in resolved ratio over the direct application of GPT-4, the advanced LLM.

-
MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue ReSolution
Wei Tao Fudan University
Yucheng Zhou
University of Macau
wtao1 udn edu. cn
yucheng zhou@connect. um edu.mo
Yanlin Wang
Wenqiang Zhang
Hongyu Zhang
Sun Yat-sen University
Fudan University
Chongqing University hyzhang@cqu.cm
mangylin360mai.ssyr sysu. edu. cn
wqzhangêfudan. edu. cn
edu. edu. cn
Yu Cheng
The Chinese University of Hong Kong chengyučcse. cuhk, edu. hk
Abstract
In software development, resolving the emergent issues within GitHub repositories is a complex challenge that involves not only the incorporation of new code but also the maintenance of existing code. Large Language Models (LLMs) have shown promise code generation but face difficulties in resolving Github issues, particularly at the repository level. To overcome this challenge, we empirically study the reason why LLMs fail to resolve GitHub issues and analyze the major factors. Motivated by the empirical findings, propose novel LLM-based Multi- Agent framework for GitHub Issue reSolution, MAGIS, consisting of four agents customized for software evolution: Manager, Repository Custodian, Developer, and Quality Assurance Engineer agents. This framework leverages the collaboration various agents in the planning and coding process unlock the potential of LLMs to resolve GitHub issues. In experiments, we employ the SWE-bench benchmark compare MAGIS with popular L.Mss, including GPT-3.5, GPT-4, and Claude- 2. MAGIS can resolve 13.94% GitHub issues, significantly outperforming the baselines. Specifically, MAGIS achieves eight-fold increase in resolved ratio over the direct application of GPT-4, the advanced LLM.
Introduction
In real-world software development, the code repository for a project is rarely set in stone. High- quality and popular software always evolves address emergent bugs platforms such as GitHub [21]. issues typically signify the requirement for software evolution. How- ever, addressing these issues poses significant challenges,
requirements. On
requires implementing the code change across the entire repository and maintaining the existing functionality while integrating new capabili- ties. For example, dj ango, framework for over 1.6M projects has 34K issues [19]. Consequently, resolving GitHub issues remains a significant challenge across academia and industry 2
Large language models (LLMs) have demonstrated remarkable capabilities across variety of tasks 8) including code generation and code understanding 64 47]. Specifically, LLMs excel in
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

generating function-level code, as evidenced by their performance on numerous benchmark datasets such as MBPP 2 and HumanEval [2] Despite their success, LLMs remain challenged in tasks that
require advanced code generation capabilities, such as class-level code generation [14]. Moreover, L.MMs exhibit limitations processing excessively long context inputs and are subject to constraints regarding their input context length
[3]] This limitation is particularly evident in repository-level coding tasks, such as solving GitHub issues, where the context comprises the entire repository, thus imposing constraints directly using the full repository input to LLMs.
To harness the full potential of LLMs, many LLM-based multi-agent systems are designed 23 These methods have significantly improved LLMs' efficacy in code generation, enabling these systems to construct code repositories based on LLM. While these methods address the process of transitioning code repositories from inception to establishment, they rarely consider the handling of software evolution, e.g., resolving GitHub issues. For GitHub repositories, especially the popular ones, a large number of commits are pushed every day. These commits derive from spectrum of evolutionary requirements that span bug fixes, feature additions, performance enhancements, etc [49]. For open-source software, new requirements frequently emerge as issues in the project's repository.
Recently, Jimenez et al. 227 developed a benchmark, namely SWE-bench, investigate the capability of popular LLMs in addressing GitHub issues. Their study reveals that LLMs fail to resolve over 95% of instances, even when file paths that require modifications are provided. This significantly low rate underscores the importance of understanding the reasons behind their suboptimal performance.
In this study, analyze the factors impacting the effectiveness of LLMSs in resolving GitHub issues. Furthermore, our empirical analysis has concluded a correlation between locating files/lines to be modified and the performance of resolving GitHub issues. Based on these insights, we propose novel LLM-based multi-agent framework, termed MAGIS, comprising four types of agents: Manager, Repository Custodian, Developer, and Quality Assurance (QA) Engineer. Our approach facilitates the resolution of GitHub issues through collaboration among agents, each fulfilling a unique role: the Manager coordinates the entire process, the Repository Custodian enhances locating files, the Developer performs code changes after locating lines, and the QA Engineer reviews the code change.
In our experiment, we evaluate our framework SWE-bench and compare its performance against existing popular L.LMs, such as CharGPT-335 (37. GPT-4 [38. and Claude-2 [. The results demonstrate that our framework, utilizing GPT-4 as its base model, significantly outperforms baselines and achieves an eight-fold performance gain compared to the direct application of GPT-4. Further analysis reveals that additional factors, i.e., the planning of code change, locating lines within the code file, and code review process, can significantly influence the resolution rate.
Our main contributions are summarized as follows:
We conduct an empirical analysis of LLMs in resolving GitHub issues and explore the correlation between locating code file/line, complexity of the code change, and the success rate in resolution.
We propose a novel LLM--based multi-agent framework, MAGIS, to alleviate the limitations of existing LLMs on GitHub issue resolution. Both our designed four-type agents and their collaboration for planning and coding unlock LLMs' potential on the repository-level coding task.
We compare our framework and other strong LLM competitors (i.e., GPT-3.5, GPT-4, and Claude-2) the SWE-bench dataset. The results show MAGIS significantly outperforms these competitors. Further analysis confirms the effectiveness and necessity of our framework design.
2 Empirical Study
SWE-bench 27 reveals the challenges LLMs face in addressing GitHub issue resolution. For example, in their evaluation, GPT-4 can only resolve less than 2% issues of the test set. Conversely, in tasks like function-level code generation, LLMs exhibit superior performance (e.g., GPT-4 gets the score of 67.0 on HumanEval (36)). Given the complexity of GitHub issue resolution akin to repository-Ievel coding, we aim to investigate Why the Performance of Directly Using LLMs to Resolve GitHub Issue is Limited? (RQ 1). We answer this RQ from the following three aspects:
Locating the Files to be Modified. GitHub issue resolution is a repository-level coding task, distinguishing it from file-level coding tasks primarily in the challenge of locating the files requiring
2

---

modification. Jimenez et al. (27 employ the BM25 method 445 to retrieve relevant code files that are subsequently utilized imput to the LLM. After employing retrieval methods, it is necessary to select the top-K files or truncate the content based on the maximum context length of the LLM. Incorporating more files can enhance recall scores. However, it also imposes significant demands on the capabilities of LLMs. As demonstrated by the study [27], Claude-2 exhibits a decrease in the resolved ratio (from 1.96% to 1.22%) as recall scores increase (from 29.58 to 51.06). This decline may be attributed to the inclusion of irrelevant files or the limited capacity of LLMs to process longer contexts effectively. Consequently, optimizing the performance of LLMs can be better achieved by striving for higher recall scores with minimized set of files, thus suggesting strategic balance between recall optimization and the number of chosen files.
Locating the Lines to be Modified. Beyond the impact of file locating, we delve into the generation of failed instances when the correct modified files were provided. A typical code change consists of multiple hunks, each specifying the line numbers targeted for modification and detailing the changes made at these locations. To quantitatively analyze the accuracy of line localization, we use the line numbers' range of the modified content in the reference code change the basis assuming that the correct modification location of the code change is uniquely determined in most cases. By calculating the coverage ratio of the line number ranges of the generated and reference, we can estimate the accuracy of line localization in the generation process, i.e.,
Ei=o Σj= 16s es [s), Ei= 8 11
Coverage Ratio-
where the numerator the length of the intersection of modified lines between the reference divided into n hunks and the generation divided into m hunks, and the denominator is the number of modified lines in the reference. More details about Equation I can be found Appendix A.
For 574 instances in the SWE-bench that ex- periments GPT-4 277 the distribution of the coverage ratio between the results generated by three LLMs and the reference shown Fig. [ From this, we observe that the performance of LLMs in generating the code change is proba-
Caude
6
Frequanccy anc
bly related to their ability to locate code lines accurately (Detailed explanation can be found in Appendix
Furthermore, we assess the relationship between the coverage ratio and the issue resolution by cal- culating their correlation coefficient. Given that the distribution of these variables exhibits skew- ness, and the resolution result is binary (resolved or not), logistic regression is employed for the analysis across three L.M.s. However, due to the limited number of successfully generated instances on GPT-4 and GPT-3.5,, a statistically significant relationship is only detected in the result generated by Claude-2. The result, i.e.. P-value 0.05, shows statistical significance. Specifically, with coefficient, 0.5997, on Claude-2, there is a substantial and positive relation between improvements in the coverage ratio and the probability of successfully resolving issues, which demonstrates that locating lines is a key factor for GitHub issue resolution.
Line Locating Coverage Rato
Figure 1:
The comparison of line locating coverage ratio between three LLMs. The vertical axis representing the frequency of the range of line locating coverage ratio for cach group, and the horizontal axis representing the coverage ratio.
Complexity of the Code Changes. The complexity of the code change is reflected in various indices: the number of modified files, functions, hunks, and lines added or deleted. Firstly, we quantitatively assess the complexity by calculating the value of various indices corresponding the reference code change. Secondly, the coefficient is calculated between the numbers each index and the issue resolution. Tab. shows the correlation scores under the logistic regression.
As shown in Tab. D. all three LLMS demonstrate statistically significant correlation with the issue resolution across several indices. The correlation scores for the number of files and functions modified
3

---

Table 1: Correlation between the complexity indices and the issue resolution.
LLM
Files
# Functions
Hunks
# Added LoC
Deleted LoC
Changed LoC
GPT-3.5
-7.577'
-17.57'
-0.06"
-0.02
-0.03
-0.53"
GPT-4
25.15
15.
-0.06
-0.10
-0.04
-0.21
Claude-2
-1.47
- -1.77"
-0.11*
0.099
-0.07"
-0.44
The correlation between the index and the issue resolution is significant (P-value < 0.05).
Project Manager
Repository Custodian
Review
6stHub Issue
Code
</>
12
Husann
Humzon
build Teae
Neeting
Developer
Quallty
Assurance
</>
Repository
Planning
Coding
Now Reposttory
Figure 2: Overview of our framework, MAGIS. The detailed version can be found in Fig. 14
are notably negative for all models, indicating that an increase in these indices is associated with a decreasing likelihood of issue resolution. This suggests that the more complex the code change, as indicated by higher number of files and functions modified, may hinder the issue resolution. More analysis can be found Appendix [.33 The analysis reveals a relationship between the complexity, as measured by several indices, and whether to successfully resolve the issues in software evolution.
The negative correlations suggest that increased complexity, particularly in terms of the number of
files and functions changed, tends to hinder issue resolution.
3 Methodology
Based on the empirical study identifying key factors affecting LLMs' issue resolution, we design the framework illustrated in Fig. 2 This framework aims to mitigate negative impacts by transforming the complex task of GitHub issue resolution into a collaborative effort. incorporates four key roles for LLM-based agents working collaboratively in the workflow: @ Manager: this role tasks with team assembly, meeting organization, and plan formulation. @ Repository Custodian: it responsible for locating the relevant files in the repository acording to the GitHub issue and recording the change of the repository. @ Developer: this role participates in planning discussions and completes tasks from the Manager. @ Quality Assurance (QA) Engineer: it reviews the code change from Developers to ensure the quality the whole repository.
The collaborative process involves planning and coding. In the planning, an issue is assigned to the Manager and the Repository Custodian. The custodian identifies candidate files relevant to the issue for modification. With the issue description and a list of candidate files, the Manager defines tasks and assembles team, where each member is a Developer specifically designed for the defined task. The Manager holds kick-off meeting with Developers and devises plan. During coding, Developers undertake their assigned tasks from the Manager, and the QA Engineer reviews each code change. If a change fails to meet quality standards, the QA Engineer provides feedback, prompting further revisions until the QA Engineer approves or a set iteration limit is reached. More details can be found in our GitHub repository
htttp:://ggtaabb.oomcoooooleeee

---

3.1 Agent Role Design
Our workflow draws inspiration from the GitHub Flow 22 an effective human workflow paradigm adopted by many software teams. Both the human workflow and our LLM-based agent framework prioritize collaboration among individuals with diverse skills. While the underlying principles are similar, there are notable differences. Accordingly, we have tailored the roles as follows:
Manager. The Manager's role is pivotal in planning. In conventional setups, managers decompose the issue into tasks according to the pre-formed team and allocate these tasks for members with different skills. In contrast, our Manager agent can first decompose the issue into tasks and then design Developer agents to form a team. This setup improves team flexibility and adaptability, enabling the formation of teams that can meet various issues efficiently.
Repusitory Custodian. Considering extensive files in a repository. the custodian agent's 's task is to locate files relevant to the issue. Unlike humans, who can browse through the entire repository, the LLM- agent faces challenges in browsing. Although LLMs have extended context limits, their application is constrained in two aspects. First, it is a high computational cost to query each file in an entire repository for each update, particularly when some repositories update frequently. Second, the performance of LLMs degrades when the context input is long [I B3 67].
Developer. Compared to human developers, the Developer agent can work continuously and efficiently. Therefore, scheduling the agent to work in parallel is easier than scheduling humans who require considering factors beyond the task. Additionally, although numerous developer agents are capable of generating code [3] 45.. their ability to modify existing code is not equally proficient. To address this issue, our framework decomposes the code modification process into sub-operations including code generation. This approach enables Developers to leverage the benefits of automatic code generation thereby producing applicable code changes.
QA Engineer. In software evolution, QA Engineers play a crucial role in maintaining software quality through code review [34, 30). Despite their importance, code review practices are often undervalued even overlooked 4.. Such neglect can hinder software development, illustrated by instances where developers may experience delays of up to 96 hours awaiting code review feedback (6). To address this problem, our framework pairs each Developer agent with a QA Engineer agent, designed to offer task-specific, timely feedback. This personalized QA approach aims to boost the review process thereby better ensuring the software quality.
Algorithm Locating.
3.2
Collaborative Process
1: Input: repository: R. including files
3.2.1
Planning
{fi), GitHub issue: qa. I.M.: L
2: Config: filter top width: k. prompts: P. find the latest previous version of the file and its summary: findd
Three types of role agents engage in the planning: Repos- itory Custodian, Manager, and Developer. This process comprises three phases: locating code files, team building. and kfck-off meeting.
3: Output: candidate files: C² + 0, repos-
itory evolution memory: M t 0
4: R, BM25(R,, q.)
5:
C + Ral:k]
Locating Code Files. Firstly, the Repository Custodian
6:
for fa C do
employs the BM25 algorithm 45 to rank the files the repository based on the GitHub issue description. Subse-
7:
fn. 8h find (f., M)
8:
if 3 fr. and len(si) len(f.) then
quently, the top & files are selected as potential candidates
9:
if h is then
for further coding. However, as described in this sim-
10:
8: + Sh. else
ple retrieval method can introduce irrelevant files, increas-
11:
ing the cost and reducing the effectiveness of subsequent coding process. Therefore, we filter these files based on relevance to minimize their number. While is feasible
12:
Ad + diff( fa)
13:
m L(Ad, Pt)
14:
8: + 8/ Um
to directly assess the relevance between each file and the issue by LLMs, queries the LLM may contain the same code snippets as previous ones, leading to unnecessary
15:
end if
16:
else
17:
St +- L(S, P2)
18:
end
computational costs. Considering that applying the code change often modifies a specific part of the file rather than the entire file, we propose a memory mechanism to reuse the previously queried information.
19:
M f- MMMuudate((fF : s{))
20:
if L((s), qz), Ps) is false then
21:
C²¹ ch ch. -
22:
end
23:
end for
5

---

Algorithm I outlines the process of locating files with our designed memory M. If a file fo is compared for the first time with an issue 9r. the LLM L with prompt P2 compresses it into the summary 8;, where denotes the file's version. This summary is shorter than the code content in the file and stored in memory for future reuse. If the file Si has been previously compared, the latest previous version (h) of the file fy can be found by the script f'nd. Since fi can be represented as the combination of fr and the difference between them (Ad that be obtained via the "git diff" command), LLMs can understand f: by using fh. and Ad. If the difference is small and the file fi is long, it is valuable to reuse the previous summary 8). stored in memory rather than the content of ft. Specifically, if the length of 8 is less than that of ft. C with prompt Pi can summarize the code changes Ad as a "commit message" m. The combination of 8)2 and m forms the description of the newer version f. enabling the LLM L with prompt P3 to determine whether it is relevant to the issue in fewer context length. Based on their relevance, the custodian agent filters irrelevant files, allowing the Manager agent to define tasks with remaining relevant files.
Algorithm 2 Making the plan.
Team Building. In this process, the Manager agent has the flexibility to "recruit" team members as the issue needs. Firstly, upon receiving the located files, the Manager begins with analyzing the GitHub issue for the repository and breaks them into detailed file- level tasks. Specifically, for each code file fi in the candidate set C, the Manager leverages the LLM L with the prompt Pa and the issue description dz to define the corresponding file-level task t, One issue can be converted to multiple tasks. These tasks, along with the associated code file, are stored in task set T. Once a task is clarified, the Manager defines the personality role r; of the Developer by invoking LLM L with the prompt Ps and the task 4. By iterating through these candidate code files, the Manager agent ultimately designs a collection of Developer agent role descriptions D., thus forming the development team. The details of the team building are shown in Algorithm [] This approach simplifies the task for LLMs because each team member only needs to handle a sub-task rather than resolving the entire complex issue.
Input: candidate files: C₁, issue: Qz. LLM: L
Config: prompts: P
3: Output: tasks: T, + 0, Developer agents'
role description: D. + 0, plan: Cmain
4: for Js C.¹ do
L t- [(( (z),..9)... P4)
6:
T,e c T U(f 4))
7:
r. + L(t,, qo). Ps)
8:
+ + D. Ur.
end for
10: recording = _ick_off meeting(D))
11: Dr + [((D), recording), Po)
12: Cmain t- L'vecording, Pr)
Kick-off Meeting. After building the team, the Manager organizes a kifk-off meeting. This meeting serves two purposes: @ To confirm whether the tasks assigned by the Manager are reasonable and ensure that all Developers in the team can collaboratively resolve the issue qu. @ To determine which Developers' tasks can be executed concurrently and which tasks have dependencies need to be sorted. The meeting takes the form of circular speech: the Manager is responsible for opening the speech, guiding the discussion and summarizing the results, and the Developers provide their opinions based on previous discussions in turn. One example of the meeting can be found in Appendix [B] After the meeting, Developers adjust their role descriptions D based on the discussion recording, and the Manager, leveraging the LLM L and the prompt Pr, generates main work plan Cmain. This plan is presented as code, and embedded into the program for execution. The meeting makes collaboration among Developers more efficient and avoids potential conflicts.
3.2.2 Coding
Based on the empirical study on line locating and the complexity ($2), we transform the code change generation into the multi-step coding process that is designed to leverage the strengths of LLMs in code generation while mitigating their weaknesses in code change generation. Two types of agents participate the coding process: Developers and QA Engineers. As outlined in Algorithm 3 for each task I; and its associated code file fi in T.k. the Developer agent generates the role description of the QA Engineer o; by the L.LM L with the prompt Ps. Subsequently, Developers collaborate with their QA Engineers to execute the coding tasks. During each execution of the Developer, the range of lines of code that need to be modified is firstly determined as a set of intervals { e(]} where s; represents the starting line number in the i-th hunk, and eç is the ending line number. The determination is generated by analyzing the task content 1, and file content f; using L with the prompt Pg. These intervals split the original code file f, into parts to be modified part) and

---

parts to be retained. Developers then gener- ate new code snippets. new new_part, by L with the prompt Pio- The code snippets replace old_ _arr, resulting in new version of the code file J. Utilizing Git tools, the code change
Algorithm 3 Coding task execution.
Input: task pairs set: T.K. LLM: C
2: Config: prompts: P, the max of iteration: 7lmax
3: Output: code changes: D
4: for ft, 4 E T," do
Ad; for this file f. is generated. With the code change Ad,, QA Engineer produce re- diew_comment and review decision, by the LLM L with the prompt P11- If the deci- sion, review decision, is negative (i.e.. false),
[((f(, 4.), Ps)
for j E 0, Thmax ) do
7:
if > 0 then
8:
1 === (l., review_commeent)
9:
end if
the feedback, review pewpweeeeee,,,,eee prompts De- velopers to revise the code in the next attempt.
10:
{[o), t L(lfett)) Pa)
11:
fi, ood_pprrtttaapppooooeeeeadaaaadd +- split( c()))
This iterative process continues until the code change meets the quality standards (i.e., re- view new_decision is true) reaches predefined maximum number of iterations. After the it- eration, the final version of the code change,
12:
new_part + L((ft, ta old_ Pio)
13:
f. replace([f [[A, ei]] new parr)
14:
Dd, <- diff( fi)
15:
review comment = C((l,, Ad:), Pi1)
16:
review decision = [((evieww_ [((review_comment)) Pn.)
17:
if review_decistee is true then
Ad, is fixed, which is the ultimate modification
18:
break end if end
result on each file. All generated final-version code changes during this process are merged
19:
20:
into the repository- code change D as the issue solution.
21:
Ad 4- Liffif/. fi
22:
++ D Ad
23:
end for
4
Experiments and Analysis
4.1 Setup
In the experiments, we employ the SWE-bench dataset as the evaluation benchmark because it is the latest dataset specifically designed for evaluating the performance of the GitHub issue resolution. SWEE-bench comprises 2, 294 issues extracted from 12 popular Python repositories, representing real software evolution requirements. Given the observation that experimental outcomes the 25% subset of SWE--bench align with those obtained from the entire dataset (27), we opt for the same 25% subset previously utilized in experiments for GPT-4 according to their materials [13. Moreover, the experimental scores for the five LLMs, have been made available by them (28).
Our framework is flexible to integrate various LLMs. To compare with the scores reported by SWE-bench, GPT-4 is selected as the base LLM. Another reason for the selection is that GPT-4 shows remarkable performance code generation and understanding demonstrated on benchmarks such as MBPP [2) and HumanEval [2]. Claude-2 is not chosen due to the unavailability of API access.
Following SWE-bench 27 the applied and resolved ratio is used to evaluate the performance under the setting with the files requiring modification provided. The applied ratio indicates the proportion of instances where the code change is successfully generated and can be applied to the code repository by Git. The resolved ratio refers to the proportion of instances where the code change is successfully applied and passes a series of tests. Additional elaboration is provided in Appendix C
4.2 How Effective is Our Framework? (RQ 2)
The comparative performance analysis between our framework and other L.MSs
Table 2:
The comparison of overall performance between MAGIS baselines on SWE-hench.
on the same dataset presented in Tab. 2 The results indicate that our framework significantly outperforms other LLMs. Notably, with a resolved ratio of 13.94%, our framework's effectiveness is eight- fold that of the base LLM. GPT-4 This substantial increase underscores our framework's 's capability to harness the po- tential of L.LMs more effectively. Fur- thermore, when contrasted with the pre-
Method
% Applied
% Resolved
GPT-3.5
11.67
0.84
Claude-2
49.36
4.88
GPT-4
13.24
1.74
SWE-Llama 7b
51.56
2.12
SWE-Ilama 13h
49.13
4.36
MAGIS
97.39
13,94
MAGIS (w/o QA)
92.71
10.63
MAGIS (w/o hints)
94.25
10.28
MAGIS (w/o hints, w/o QA)
91.99
8.71
7

---

0.70
0.69
Reseived
0.68
0.67
Nurrber
30
0.66
0.65
5.
22
Number
Comeltion Level
Figure 3: Comparison of recall scores between Ours Figure 4: Distribution of the correlation score between the generated task description and the reference code change.
and BM25.
vious state-of-the-art LLM, Claude-2, our framework's resolved ratio exceeds that benchmark by more than two-fold. This superior performance unequivocally establishes the advance of our method.
The ablation study is designed to simulate two scenarios: @ Without QA (w/o QA): Considering the QA Engineer agent optional within our framework, we directly evaluate the code changes generated by the Developer agent, bypassing the QA process. This scenario aims to investigate the effectiveness and necessity of QA Engineer review. @ Without hints (w/o hints): Hints refer to the textual content found in the comments section of pull requests, which are typically created before the first commit of the pull request. This setting means framework operates without any clarifications except for the issue, despite such information being available on GitHub before the issue resolution process begins. This analysis aims to explore if the participation of humans could potentially improve the success rate of issue resolution.
Our framework shows significant improvement in issue resolution, even without QA or hints. It achieves a resolved ratio of 8.71%, which five times higher than that of the base LLM. This increase underscores the contribution of other agents in MAGIS to its overall performance. Furthermore, integrating cooperation with QA or hints separately can further elevate the resolved ratio by 1.92% or 1.57%, respectively. These findings underscore the value of QA Engineers and the participation of humans, as demonstrated by the resolved rates achieved through their integration.
For instance, to resolve the issue [1 from the repository Django [[5], the developer modifies four hunks in two files [16. as shown in Fig. 15 Despite the availability of two provided files, our method opts for modifications in only one file, as illustrated in Figure 16 Remarkably, this simpler code change enables the repository to pass all requisite test cases.
Additional comparison can be found in Appendix D and E. and detailed case study is shown in Appendix H
Furthermore, the statistics the generated code changes be found in Appendix
4.3 How Effective is Our Planning Process? (RQ 3)
To investigate the effectiveness of the planning process, we analyze the Repository Custodian and Manager agent. The performance of the Repository Custodian agent is observed in the recall score versus the file number curve, shown in Fig. 3 This curve demonstrates that our method consistently outperforms the BM25 baseline across varying numbers of selected files, indicating that our approach can identify the maximum number of relevant code files with the minimum selection.
For the Manager agent, we examined the aligmment of its generated task descriptions with the reference code change by LLM. Following the study [63]. we select GPT-4 as an evaluator to score the correlation between the reference code change and the generated task description. The correlation scores are determined based on set of criteria defined in Tab. 6 A higher correlation score indicates a better alignment and thus, a more accurate and effective planning direction. The distribution of these correlation scores is presented in Fig. 4 Notably, most of the scores are 3 or above, implying that the majority of task descriptions are the right direction concerning planning. Furthermore, higher scores correlate with higher probability of issue resolution, indicated by a larger proportion of "edsolved" outcomes in scores and 5. This signifies that when the generated task description closely aligns with the reference, there is a higher possibility of resolving the issue. The analysis
8

---

Our
GPT-4
Cumulative Frequency
1.0
0.175
0.150
Raiio
0.125
10.6
0.100
0.075
0.050
0.0
02
0.4
0.6
0.8
1.0
0.000
0.2
0.4
0.6
0.8
1.0
0.0
Locating Coverage Ruti
Line Locating Coverage
Figure Comparison of line locating coverage Figure 6: Resolved ratio in different line locating coverage between MAGIS (Ours) and baselines.
intervals.
above demonstrates the effectiveness of both the Repository Custodian and the Manager agent in the planning process of our framework.
4.4 How Effective is Our Coding Process? (RQ 4)
To evaluate the effectiveness of the coding process in our framework, we analyze the performance of Developers in locating code lines and resolving issues of different complexity.
Fig. 5 jillustrates the distribution of the line locating coverage ratio MAGIS and the baselines. This visualization reveals that our Developer agent frequently attains a line locating coverage ratio nearing Compared with baselines, the Developer agent demonstrates pronounced preference for higher distribution values close to 1, and conversely, reduced preference for lower distribution values near Such a distribution validates the superior performance of MAGIS in locating code lines.
Further analysis is provided in Fig. |illtuttagin the relationship between the line locating coverage ratio and the issue resolved ratio within those coverages. As shown in Fig. 6 the right four bars higher than the five left, which indicates that the resolved ratio can increase with the line locating coverage. This observation also suggests that locating lines accurately important for issue resolution. The cumulative frequency curve, shown in orange, provides an additional analysis, indicating the cumulative proportion of issues resolved ratio up each point along the line locating coverage. A steady increase in cumulative frequency accompanies the increase in line locating coverage, reinforeing the idea that resolving issues is more successful in areas of high coverage. The slope of the curve's 's left half is lower than that of the right half, indicating that the benefits of increasing the coverage ratio are less pronounced lower coverage ratios than at higher ones. Therefore, the Developer agent should prioritize improving its capability of locating code lines.
Moreover, shown in Tab. 3 present logistic regression analysis that quantifies the correlation between several complexity indices and issue resolution. The results show that GPT-4 has significant negative correlations across the number of files and functions, suggesting that as these indices increase, the likelihood of issue resolution decreases. Conversely, the negative correlations are less pronounced with our model, MAGIS, particularly the number files and functions, suggesting mitigation of challenges corresponding to these complexity indices.
Table 3: Correlation between the complexity indices and the issue resolution.
Method
# Files
# Functions
Hunks
# Added LoC
# Deleted LoC
# Changed LoC
GPT-4
-55.15
-5
-0.06
-0.10
-0.04
-0.21
MAGIS
-1.55"
-1.5
-0.12"
-0.04"
-0.06"
-0.57"
The correlation between the index and the issue resolution is significant (P-value < 0.05).
To evaluate the performance of the QA Engineer, the ablation experiment is conducted and the results are shown in Tab, 2 Engineer can increase the resolved ratio by 1.57% and 3.31%, respectively. This overall enhancement
the table shows, in settings with and without hints, the presence of the QA

---

substantiates the QA Engineer's contribution to improving outcomes. Furthermore, case detailed in Appendix[ underscores the QA Engineer's effectiveness.
5 Related Work
Researchers have developed LLM- based multi-agent systems, enabling more complex task com- pletion. For instance. MetaGPT [23] 24] simulates a programming team'ss Standardized Operating Procedures (SOPs) and achieves leading scores on benchmarks like HumanEval [12] and MBPP [2]. Similarly, ChatDev 4 functions as virtual development company, decomposing requirements into atomic tasks and utilizing mutual communication and self-reflection to mitigate L.LM hallucinations. While these systems excel transforming requirements into code, they often overlook the challenges of code change generation during software evolution [25]. GitHub issues include different types of requirements and most of them belong to bug fixing. Previous researchers have proposed methods to localize the bugs 65
42
and some researchers explored various methods to automatic program
repair 57
59 53
The full version of related work can found Appendix
Conclusion
This paper illuminates the potential of LLMs in software development, particularly in resolving GitHub issues. Our empirical study identifies the challenges direct LLM application. To address the challenges, we propose novel LLM-based multi-agent framework, MAGIS, enhancing issue resolution through well-designed agents' collaboration. The superiority of MAGIS on the SWE- bench against popular LLMs highlights its effectiveness, pointing towards promising direction for integrating LLMs into software evolution workflows.
References
[1] Anthropic. Claude httpes//wwaaarrrppii..c////rreee
2023.
[2] Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. arXiv Preprint, abs/2108.07732, 2021. URL https
//arxiv.org/abte abs/ /11.:21010007777
[3] Thomas H. Austin, Thomas Schmitz, and Cormac Flanagan. Multiple facets for dynamic information flow with exceptions. ACM Trans. Program. Lang. Syst. 39:3):00:1110:66, 2017. doi: 10. 15//3024088. /ttpttttss//oo.orrrrer
[4] Tobias Baum, Olga Liskin, Kai Niklas, and Kurt Schneider. Factors influencing code review processes industry. In Thomas Zimmermann, Jane Cleland- Cleland-Huang, and Zhendong Su, editors, Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, Seattle, WA, USA, November 13-18, 2016, pages 85-96. ACM, 2016. 2950323
doi: 10. 22.114//99502002950322
/ttpshtts:///oi.ooo
[5] Tegawendé F. Bissyandé, David Lo, Lingxiao Jiang, Laurent Réveillère, Jacques Klein, and Yves Le Traon. Got issues? who cares about it? A large scale investigation of issue trackers from github. In JEEE 24th International Symposium on Software Reliability Engineering, ISSRE 2013, Pasadena, CA, USA, November 4-7, 2013, pages 188-197. IEEE Computer Society, 2013. doi: 10.1 10.1109//SSSE.22 201.5366R...011.6999119 URL /ttp:tt:://doio.r//00000000 (ww.:///9ISSSEE221.....
[6] Amiangshu Bosu and Jeffrey C. Carver. Impact of developer reputation on code review outcomes in OSS projects: an empirical investigation. In Maurizio Morisio, Tore Dybå, and Marco Torchiano, editors, 2014 ACM-IEFE International Symposium on Empirical Software Engincering and Measurement, ESEM '14, Torino, Italy, September 18-19, 2014, pages 33:1 1- 33: 10. ACM, 2014. doi: 10. 10.114572555224225555555 URL https://doo.org//0...t
0855524.26525555
[7] Islem Bouzenia, Premkumar's T. Devanbu, and Michael Pradel. Repairagent: An autonomous, Ilm-based agent for program repair, arXiv Preprini, http//243..111344 2024. doi: 10.48550/ ARXIV. 22..2. 17134. URL
10

---

[8] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi. Marco Túlio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4, arXiv Preprint, abs/2303. 12712, 2023. doi: 10.448550AARXI12200.. 2303 URL.
https://doi.ors
(a.8555///xiivi 22... 12712
[9] Jiayi Geng Carlos E. Jimenez, John Yang. Swe-bench lite. lite.html 2024.
[10] Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better Ilm-based evaluators through multi-agen aaget debate. arXiv Preprint, abs/2308.0720 2023. doi: 10.4850//RXIV 2w...2000002.11. URL htttps://do..o 10.488000 wwt.ar.ww.......2000
[11] Lichang Chen, Jiuhai Chen, Heng Huang, and Minhao Cheng. PTP: boosting stability and performance of prompt tuning with perturbation-bast regularizer. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 13512- .org/
13525. Association for Computational Linguistics, 2023. htt.httpss//aclanttiie
2023. 2023.emnlp- www.memnllmmm. wwtelllpmmi.... www..ai......... 833
[12] Mark Chen, Jerry Tworek. Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Rudford, Matthew Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. arXiv Preprint, 10t.020777033344 2021. URL.UURLLttptt.. http//ttcsss//zxx..rr///b///////
[13] Google Drive. Swe-bench_appigeenennid 1EnrKzGAnsb_Nee htA.KK/AaSSS..MiZZZYy/E000m2/2/.
2024.
[14] Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. Classeval: A manually-crafted benchmark for evaluating Ilms on class-level code generation, 2023.
[15] Django Software Foundation. Django.
2024.
[16] Django Software Foundation. Fixed -ixe admindocs emrors rendeing docstrin without leading newlines. 2024.
[17] Django Software Foundation. #30255 (docutils reports an error rendering view docstring when the first line is not empty). http:///ghhaannononnnnn////////p
wtt.c/////iceeee//555555/55 2024.
[18) Django Software Foundation. #30664 (sqlite3 migrations fail when used quoted db table.). 2024.
[19] Django Software Foundation. Custom query django. httpp://ccdeecjaaggpooeee..tot
query May 11, 2024.
[20] Xinyang Geng and Hao Liu. Openllama: An open reproduction of Ilama, May 2023. URL http:s///gttuuuu.cmoooopppeeeres
[21] Inc. GitHub. Github. luttps///github.com 2024.
[22] Inc. GitHub. Github flow. htwp://gg./ooogggtttt..////////a fichub-flog 2024.

---

[23] Sirui Hong, Mingehen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang. Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for a multi-agent collaborative framework, 2023. URL htt.:/ttt://oi.rrg//////// 011111..4888888
40w.440050000222505 att. 2308.00333000033333 00352
[24]
Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Danyang Li. Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Li Zhang, Lingyao Zhang, Min Yang, Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Wenyi Wang, Xiangru Tang, Xiangtao Lu, Xiawu Zheng, Xinbing Liang, Yaying Fei, Yuheng Cheng, Zongze Xu, and Chenglin Wu. Data interpreter: An L.M agent for data science. arXiv Preprint, abs/2402. 2024. doi: 10.48550/ARXIV 22... 18679. URL
https://áoi.orgg
[25] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang. Kailong Wang, L. Li, Xiapu Luo, David Lo, John C. Grundy, and Haoyu Wang. Large language models for software engineering: A system- atic literature review. arXiv Preprint, abs/2308. 10620, 2023. doi: 10.48550/ARXIV. 2308. 10620. orgo wttp://i..r.11...8..... wtt.///.00...123...10022222
bttpps:////diooooo
[26] Xing Hu, Xin Xia, David Lo, Zhiyuan Wan, Qiuyuan Chen, and Thomas Zimmermann. Practitioners' expectations on automated code comment generation. In 44th IEFF/ACM 44th International Conference Software Engineering, ICSE 2022, Pittsburgh, PA, USA, doi: 10. 10.1145/35100033351000
May 25-27, 2022, pages 1693-1705. ACM, 2022.
URL
[27] Carlos E. Jimenez, John Yang, Alexander Wettig. Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,
2024. OpenReview.net, 2024. URL httpp//s//pppeeeeeeeeeeeeeerrrrr
(28) Carlos E. Jímenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Official comments to reviewer bfzn on swe-hench: Can language models resolve real-world github issues? httpp:///aaaeeeeieeeeeee//////// 1fJF38VxJr] 2024.
[29] Thomas Johnsson. Attribute grammars as a functional programming paradigm. In Gilles Kahn, editor, Functional Programming Languages and Computer Architecture, Portland, Oregon, USA, September 14-16, 1987, Proceedings, volume 274 of Lecture Notes in Computer Science, pages 173. Springer, 1987. doi: 10.1007/3-5400 18317-5 10. httpss/s:///oi. org/
[30] Oleksii Kononenko, Olga Baysal, Latifa Guerrouj, Yaxin Cao, and Michael W. Godfrey. In- vestigating code review quality: Do people and participation matter? In Rainer Koschke, Jens Krinke, and Martin P. Robillard, editors, 2015 IEEE International Conference on Software Maintemance and Evolution, ICSME 2015, Bremen, Germany, September 29 - October 1, 2015, pages 111-120. IEEE Computer Society, 2015. doi: 10.109//CSM.2015.333222 https:://di..or wtw.:////...1110.....0000.62.2..
URL
[31] Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context lims strug- gle with long in-context learning. arXiv Preprint, abs/2404.02060, 2024. doi: 10.48550/ IRXIV.2404.00
URL
http::://oo.iii .rrg 1..1000000/1. 201.:500rxxiv.244..002
[32] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December wwwpcccccpppeereeeee files/
10 16, 2023, 2023. URL. http:: 43e9d647ccd3e4D777 13996677ccee1777555a3b3333333333 haw.://eeeebbbttteteeeeooeoeeeee
walles/paper/e htpp:/apprr//// (emppr//022///2 222///ash//
[33] Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long con- texts. arXiv Preprint, abs/2307.03172, 2023. doi: 1.445500/ARXXV.220...00 URL
https://doi.c 305//10.485500 /ttp1//.8850///111.13...0
12

---

[34] Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. The impact of code review coverage and code review participation on software quality: a case study of the qt. vtk, and ITK projects. In Premkumar T. Devanbu, Sung Kim, and Martin Pinzger, editors, 11th Working Conference on Mining Software Repositories, MSR 2014, Proceedings, May 31 June doi: 10. 10.1145/2597073.25999
1, 2014, Hyderabad, India, pages 192-201. ACM, 2014.
URL
/
[35] Fangwen Mu, Xiao Chen, Lin Shi, Song Wang, and Qing Wang. Developer-intent driven code comment generation. In 45th IEEF/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pages 768-780) IEHE, 2023, doi: 10. 109/ URL
(CSF44611.2003.000))
http:///...rr1111100000666661110 00073
[36] OpenAL. GPT- technical report. Arxiv Preprint, abs/2303.08774, 2023. doi: 10.48550/ TRXIV.2303.08777 URL httt:https://deee httppp///0b..orr/00.666500////1
[37] OpenAL. Gpt-3.5 turbo fine-tuning and api updates. http:://ppenaa.ccm/blg//g--
turbo- ferbbo--ii--ttinn-----a www.iinnnaddd---------aa 2023.
[38] OpenAl. Gpt-4, http:://ppenal.com/reeerrdh/pp/p
2023.
[39] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. scikitlearn. http:s///githh...o//ssltttt---- htt../////cciiiteeeenneeeeeeeeee
learn 2024.
[40] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Duchesnay. [mrg] add seeds when n_jobs=l and use seed as random_state https://gi http://gghuuu.coo.o wtt.aub..ooocciiit.............. leaan/ecikiiiiee learn/ htlppppllll//9 2024.
[41] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Kmeans gives slightly different result for n jobs=1 vs. n_jobs 1.
https:/ gi www.tthoooocoeeeeet. wwitat//ciittteiee weattteeeeaan/iiiit-teeee
hht.b..ww.....kktttaaaae/cciteee www.eeerraaiiieaeeeeeeeeeee 9978 2024.
[42] Binhang Qi, Hailong Sun, Wei Yuan, Hongyu Zhang, and Xiangxin Meng. Dreamloc: A deep relevance muaching-based framework for bug localization. IEEE Trans. Reliab,, 71(1):235- 249, 2022. doi: 10. N..1109/TR.222 20222021111400 3300722..
[43] Chen Qian, Xin Cong, Wei Liu, Cheng Yang, Weize Chen, Yusheng Su, Yufan Dang, Jiahao Li, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv Preprint, 2023.
[44] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre- vre-training, 2018.
[45] Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu and Mike Gatford. Okapi at TREC-3, In Donna K. Harman, editor, Proceedings of The Third Text REtrieval Conference, TREC 1994, Gaithersburg, Maryland, USA, November 2-4, 1994, volume 500-225 of NIST Special Publication, pages 109-126. National Institute of Standards and Technology (NIST), 1994. URL
[46] Jessica Shich. Best practices for prompt engineering with openai api. OpenAl, Febru- ary https://help.. openai. openai-api, 2023.
[47] Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, Qipeng Guo, Xipeng Qiu, Pengcheng Yin, Xiaoli Li, Fei Yuan, Lingpeng Kong. Xiang Li, and Zhiyong Wu. A survey of neural code intelligence: Paradigms, advances and beyond, 2024.
[48] Yashar Talebirad and Amirhossein Nadiri. Multi-agent collaboration: Harnessing the power of intelligent LLM agents. arXiv Preprini, 033423300003314, 2023. doi: 10.48550/ ARXIV 220.2223033331 URL htttts://t/oiiii. org wrtg///.///.../. www.xxivi606 221.0233300000114
13

---

[49] Wei Tao, Yucheng Zhou, Yanlin Wang, Hongyu Zhang. Haofen Wang, and Wenqiang Zhang Kadel: Knowledge-awee aware denoising learning for commit message generation. ACM Trans. Softw. Eng. Methodol. jan 2024. ISSN 1049-331X.
doi: 10.1 10.1145/3643675,
URL https
0tbbi.ogg10..14536645333
[50] The Cogmition Team. Swe-bench technical report. 2024. URL https://www.conitt...
1 labs.com/post stt/:/ww-bbee wwwpss/sweeeecch-teeeeeeeeee www.ecchtthcinccl--rrr..o.
[51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv Preprint, abs/2302.13971 2023. doi: 10.48550/ARXIV 2302.13971. https///:///o (0w.0..555//////. arxiv 230.....3301777
[52] Michele Tufano, Anisha Agarwal, Jinu Jang, Roshanak Zilouchian Moghaddam, and Neel Sundaresan. Autodev: Automated ai-driven development, 2024. URL htttp://aoi..rrg
10.48550/ arxiv. ww0.v.24....0m wttpar.i.......999900
[53] Weishi Wang, Yue Wang. Shafiq Joty, and Steven C. H. Hoi. Rap-gen: Retrieval-augmentee augmented patch generation with codet5 for automatic program repair. In Satish Chandra, Kelly Blincoe. and Paolo Tonella, editors, Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposiun on the Foundations of Software Engineering. ESEC/FSE 2023, San Francisco, CA, USA, December 3-9, 2023, pages 10. 114./ 361 111116666366655
ACM, 2023. doi: 10.1
3611643.3616255 URL httpt:ttttt.oooooo
[54] Yue Wang. Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. Codet5: Identifier-aware unified pre-trained encoerrddecoder models for code understanding and generation. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pages 8696-8708. Association for Computational Linguistics, 2021. doi: 10.18653/v 1/202111/2221../ wtt.022..mmmppnaen emnlp-main.685.
URL htttt:: wtt.:/i..ogg1..111881111 wtw.18653/v1/2022 www.eeeemnlmmimeeememmmm
[55] Chu-Pan Wong, Priscila Santiesteban, Christian Kästner, and Claire Goues. Varfx: balancing edit expressiveness and search effectiveness in automated program repair. In Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta, editors, ESEC/FSE '21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, Augusi 23-28, 2021, pages 354-366. ACM, 2021. doi: 10. 011144/346224444666666 URL http:tts: //ttps://doiis /ttg///0...1155/6622333388000000
[56] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversation framework. arXiv Preprint, abs/2308 08 155, 2023. doi: 10.48550/ ARXIV SEXXV..3008.88 08155. URL //tttt::///oi...... 10tgg/10.e0 200.5500/XXiv..0...8888
[57] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated program repair in the era of large pre-trained language models. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pages 1482- 1494. IEEE, 2023. doi: 10.1109/CSE448699.0000 202..000229099
ICSE48619.2023.000
[58] John Yang. Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent computer interfaces enable software engi- neering language models, 2024.
[59] He Ye and Martin Monperrus. ITER: iterative neural repair for multi-location patches. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pages 10:1-10:13, ACM, 2024. doi: 10.1145/ 0597503.3623333 URL http:tttt///oooorreeee 0tt.:/...gg/111111/33555555....3
[60] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Au- tonomous program improvement. arXiv Preprint, abs/2404.054277, 2024. doi: 10.48550/ ARXIV 050.0427.7 URL
URLhttpss: http::tt:: :ttptps://do..oo org/ 20w.5555////21244444.4 054277
14

---

[61] Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jianguo Li, and Rui Wang. Unifying the perspectives of nlp and software engineering: A survey on language models for code, 2024. URL uttpstss:::
wttp.orr/aabb//..00/9099 1.09989
[62] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models. arXiv Preprint, abs/2303. 2023. doi: ://oo./org////////oooooooooooo 101.1//55/////1..2233....
10.48550/ARXIV.20 2303. 18223. URL
[63] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, edi- tors, Advances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http::/ppaper..a... wwtp:/ssiisssc.ccpppppepeppppppe wwwpp/s.ccppppppppeeeeee haappappppleeeeeeeepeepppppeeeee http:pppppr//2//// 225.8/18887770008008602266060010 Aat.a://11-1111ttattta------a--- Datasets and Lan__Bencharrkee htmll
[64] Zibin Zheng, Kaiwen Ning, Jiachi Chen, Yanlin Wang, Wenqing Chen, Lianghong Guo, and Weicheng Wang. Towards an understanding of large language models in software engineering tasks. arXiv Preprint, abs/2308. 1 11396, 2023. doi: 10.48550/ARXIV.2255 23088 1139. URL https: wtt..oo////...///r//////o./ arxiv 113966
[65] Jian Zhou, Hongyu Zhang, and David Lo. Where should the bugs be fixed? more accurate information retrieval-basee bug localization based bug reports. In Martin Glinz, Gail C. Murphy, and Mauro Pezzè, editors, 34th International Conference on Software Engineering. ICSE 2012, June 2-9, 2012, Zurich, Switzerland, pages 14-24. IEEE Computer Society, 2012. doi: 20.1199/
[66] Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chao Ji, Wenhai Li, and Dan Ding. Fault analysis and debugging of microservice systems: Industrial survey, benchmark system, and empirical study. IEEE Trans. Software Eng.. 47(2)243322)) 260. 2021. doi: 10. 10.110900 1,.11099/SE.2111 201....0088.888888888888 URL httpp::ss////oi.ooo http:///..11119/1111 201. 288.338884
[67] Yucheng Zhou, Xiubo Geng, Tao Shen, Chongyang Tao, Guodong Long, Jian Jian-Guang Lou, and Jianbing Shen. Thread of thought unraveling chaotic contexts. arXiv Preprint, abs/2311.08734, 2023,
doi: 10.48550/ARXIV 131.0844.
URL htttss///doioon
00.48550/arXiv.255 1.88734
[68] Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. Llama-moe: Building mixture-ofeexperts from llama with continual pre-training. arXiv preprint 2xXv:2446611554 2024. URL http:://arrii..og/a////0...0
15

---

A Detailed Explanation in Empirical Study
A.1 Coverage Ratio
The formula for calculating the coverage ratio is Equation 1 As it shows, for each instance of GitHub issue resolution, the range of code change (in terms of the number of lines) in the reference is represented as set of intervals D, = {[so. eol:... ... [Sn: ex]). while the line ranges of the generated code change g is g
{[56- % ..., [s em],. where and respectively represent the starting and ending line number of each modification hunk in the file, with hunks in the reference code change and m hunks in the generated one.
A.2 Observation on Fig. 1
As shown in Fig. I. we observe that: @ The distribution near the coverage ratio 0 (left side of the figure) is the highest for all three LLMs, indicating that in most cases, the content generated by these models has a very low coverage ratio with the reference in terms of locating code lines. This means that these LLMs are most likely not able accurately locate code lines that need be modified in the process of generating the code change. @ In the distribution near the line locating coverage of (right side of the figure), the three models show consistent ranking (i.e., Claude-2 GPT-4 > GPT-3355) and this ranking is also consistent with the proportion of instances solved by the three models. This phenomenon suggests that the performance of LLMs in generating the code change is probably related to their ability to locate code lines accurately.
A.3 Analysis on Complexity of the Code Change
As shown in Fig. [] compared with GPT-3.5 and GPT-4, Claude-2 exhibits different pattern, with much lower negative correlations for the number of files and functions, which indicates that it is more efficient approach generate the code change for GitHub issue resolution. However, it also shows significant negative correlations across other indices such as the number of hunks, added lines of code (LoC), deleted LoC, and changed LoC.
B Kick-off Meeting Example
Figure 7 illustrates kick-off team meeting, In this meeting, three participants are present: the Manager agent, Oliver CodeLead, and two Developer agents, Django Database Specialist and Alex Rossini. They discuss a specific issue") assigned tasks, and determine the workflow sequence.
C
Metrics
The applied ratio indicates the proportion of instances where the code change successfully generated and can be applied to the existing code repository using Git tools, i.e.,
Applied Ratio
(2)
[
where D represents the set of instances in the generated code change set that could be applied the original code repository using the "git apply" operation, and is the set of all instances in the test set. The resolved ratio refers the proportion of instances in which the code change is successfully applied and passed a series of tests, i.e.,
((o((Toa(((i)) nT (d.)))
Resolved Ratio
(3)
where Told denotes all the test cases that the old version of the code repository could pass, Taear represents all the test cases designed for new requirements, and d, denotes the code change generated to resolve the issue in the i-th instance. Furthermore, T(d) True means that the code change can pass all the test cases in T.
https::/ccode..e httppp///codd/aagggoooooooo/oo// http://aggoaatttto//////a//////a
16

---

Modig Stated R
tocay's Retasla rane, grsts are bottsdd: tirst, warre the the tratractorn sen tue nate an recaiee are clar sdicurts to dllaetivey this sasa, fecand, in
soving,
oprctional
dating sigutiess dublig cacced
Qler..
and leas auliving prasie and the selution strategy es we is place. 2'11 thas spen the flor for lopas, edecifically booking Lierrll ary gasiing pleces in feondircias marg talk, let's
cantic
is you'n ve sentioned, the sperationn an nor that are (acing is due to directly quta tabla sans is the "Clssall"" sption uian working sim Squibal, hass juss ariies Secate and eattyy tw direccty caree cadee can poeeher, cherge ow rendke ceste setres in the betesandit sceiittterr class hecls tho tasle lends correctly, stcredics regrenaom pisantere fordig by ralationanda
forsigo
dxcenet 12l chirgo and oplats aprase tae nettrrs transareyy faturs this prosien.
comsderdis
Bust you girap excanse tas iacistist for you incigntial caw vine Your olee sueac calorele eieeees deettty autresun the tetriil toe foo the diceow cide anderatand forsigh lays.
tabl
sigresson inobiing
tus,, exble-aueted the are ciresty casted, will ths refacter an hotgrod soecife attrahctes sett SOLiee
quese
conrectly, essning
surt fro toa a wey critival part of an alle the coutias tat was the ioea as tod the calurie, This is essertaal coald this proiles from affective
welleatig
asse-teell))
any parce you
my twve berrsoted or speciiica that we nea to condinate. deservdee, I melleve are all set to Mice off or bab.
the stretepti lggattan scians with quitee talle aanks cpang dica sacakase greaalit,, you rele will ineaive adjatting the autaarcittaaiiii eine, you' reponaale for
yuu
the
the
dreer
dvensiertes efivets
tatt tike clecty retated tweflt Avee
cioss coer portin,ion, asoecially crasdering the stared the sabla nuchet the wad for coprassuie tarting.
waet the ced coees Aleal can come taske to den sale with perladic dedicin to grec oair coriooment is fixd tha secunce is requires,
in hodggnng
bree hase clarptted thiy astundise 10e2lw are sheipptets for
exsre that overeet they high their vent. there proutly.
Taas for tesights, oliver. resausss your quettin, the orear caerations trese fius, thoud Ldantily accurescis of "eltatte" outedd
Oueng
twe reeite tordie qucie di ralle sases care corvectiy. the se hase taise tas cros contlcted ane wrified, or askiy to meclecs dialing with iisteeent gresatior.
ipenced
it tsta stape, Alas voste be ands to start westing the tarts, Tais sesenee coald slas poaar as free woing to swrtte turt 2n adfor, twe acconsdate charges sice to precescing plarsat.
fey cheen that clanges are cacalie squlte operations and rat assoercnnly change the brover for coker ditamoss.
soy've
destingg
Festgrettt other detabese bachems, svetd this, steeld enure
ligending reeded remaroos, I'n confortacde woth the proviced sevelieer scaes aerdaadoo ax eodiable docuettation. If anytting sim cones 13, 1'12 race sure to
We Alec Fraxisi
selianttteee smatititi tietie and it's tiile" anrettionn, cat ieveablly fellow to with the fest cenition Till alse mort our sebrrss bosss affect oteer 4e your fances by
apparces, contorratte sdts the persiaters develmmnnt stece priddees also thee it welld te tenelicted heve repter shed-drrs to wdlditee sigess travsprrety.
matause spciilistt
soced ii cxscssionn in apelins woe clean slln ane sgrect con ceperce al ticks sangg tstroce tecialist will stat by savoiiai gatal la saate" eueps wer reallve the liue ailver effectine oreer deassee tatkews.
bewvirr,
sairtaie nasercal and aranee to critics issas srite, lat's schssis reglaa dicc-inn. These aill ure as axprrisition to systresias our pagess, accress any that
exectoe dfffeenn stmess
the first cootonintt will to after the sestial oejectectss to mostooncewwiite iiee' are coplottd by tjang sutanse sacialie. at that pairt, on will rovire tie
dangus everysiing
af there dejections further poirts cascess, 1'1 atmee schadale thes clecksiits presits evesss sits upciced workizww ctart. After tuc, takks darre cxtruuee specialits an vis goer lign light and sles, slase prica for test cenolipent
adjeano mettnn scrkiing dn staydig for upretes brittad thes excetter presimm
mnttrane, rech
sbjeccives
you your
discusias,
Tust all fin par coste touttooss to con bastem willl praaes fullert ninge satatosr seccialist it will di-st lsextify and exciiy the inctaros of direcily qutted robest ane das affect power oatacate backena. sill iegleners replar cwedi-a to surchrolie our effarts and address any lasas proppay. Mich ban par in place and acjea
Mexting
Figure 7: Kick-off meeting to resolve the issue 18
17

---

The recall score versus file number curve is used to measure the effectiveness of locating code files to be modified. The recall score refers the proportion of files that are successfully located out of all the files that require modification. The formula for calculating the file locating recall score for the i-th instance is follows:
[ R [R.]
Recall
X 100%,
where Gi = Pig represents the set of file paths located by our framework, with each file path in the set denoted as gi.j and the total number of files as nt; R, = -i= Ti, denotes the paths of the files that need to be modified, with each reference file path denoted as Tt,k and the total file number as m. In this curve, "file number" refers to the average number of files that need to be processed across all instances to achieve the given recall score. Specifically, it illustrates how many files averagely need to be located by our framework before reaching the recall score denoted by the curve at any point. This metric represents both the effectiveness and efficiency of file locating.
D Comparison Result on SWE-bench Lite
Recently, some contemporaneous works, e.g., AutoCodeRover (60) and SWE-Agent 158 have been proposed for this task. These methods are evaluated using SWE-bench lite, canonical subset of SWE-bench, which is recommended for evaluation [9]. Considering budget constraints, we conducted experiments on SWE-bench lite compare with them on the same issues' resolution.
The experimental results are shown in Tab. 4. MAGIS achieves the highest resolved ratio, 25.33%, than other baselines. The performance of MAGIS slightly decreased when evaluated without QA, reaching 23.33%, and dropped under the other two ablation settings. This comparative study un- derscores the robustness of MAGIS, particularly when provided with comprehensive inputs, and highlights the impact of QA and hints on its performance. The results indicate that while new methods like AutoCodeRover and SWE-Agent show promise, MAGIS remains effective method for GitHub issue resolution.
Table 4: The comparison of overall performance between MAGIS and baselines on SWE-bench lite.
MAGIS
Method
AutoCodeRover
SWE-Agent
Full
w/o QA
w/o hints
w/o (hints, QA)
Resolved 16.11% (22.33%*) 18.00%
25.33%
23.33%
16.67%
16.00%
Note that 16.11 the average scores among runs while 22.33 is under the union of from the 3 runs.
E
Comparison with Devin
Devin is a novel agent for software development [50]. and its performance has also been assessed using the SWE-bench. However, the evaluation dataset employed by Devin differs from the subset used for experiments with GPT-4 reported by the paper of SWE-bench (27). An analysis of the repository name and pull request IDD of each instance reveals that only 140 instances coverage between the two datasets.
Within the shared pool of 140 instances, our framework successfully resolves 21 (15%) issues, surpassing Devin's resolution of 18 (12.86%) issues This comparison, however, may not be entirely equitable. Devin's 'ss possible underlying LLM is unknown, and it possesses the capability to integrate feedback from the environment. Moreover, Devin's 's reported scores are under the setting given the entire repository, and operates with "common developer tools including the shell, code editor, and browser", and "agents with internet access could potentially find external information through other methods" as detailed at the report In contrast, our approach solely relies on the shell, without the need of any additional external tools.
pass
18

---

For running time, 72% of instances resolved by Devin require greater than 10 minutes to complete. In contrast, our framework finalizes each resolved issue within approximately minutes. On average. our framework completes the processing of each instance in under 5 minutes, demonstrating its capability to assist in resolving GitHub issues with minimal time expenditure.
F Statistics on the Generated Code Changes
This section provides statistics on code changes corresponding to resolved issues and those applicable but unresolved using our framework.
The statistics the code change for instances with resolved issues presented in Tab. 5 Overall, the statistical information of the generated code changes for these instances, such as the average number of code files, functions, hunks, and deleted lines, all differ slightly (not exceeding 0.3) from the reference solutions written by humans. This indicates that for these instances, the complexity of the code change generated by our framework is similar that of humans. Furthermore, the maximum values observed in the table reveal that our framework can implement code modifications involving two files, four hunks, and 655 lines modification, with single modifications reaching up 190 lines. Results demonstrate the effectiveness of our method in resolving complex issues that need to modify the code file on multiple locations and with long context.
Specifically, the distribution of the number of modified lines for the resolving instances is shown in Fig. X We observe that the distribution of the number of modified lines in framework for the solved instances exceeds that of the reference solution, especially in terms of the number of added lines being significantly higher than the reference. Upon manual inspection, we found that the generation results provided by our framework often contained more comment information, which led to an increase in the total number of modified lines. For example, Fig. TO displays the generation result of our framework. Lines 365, 368, 371, 374, 383 in the new version file correspond to the comment for the added code. These natural language descriptions are valuable in actual software human-written written solution lacking such explanatory comments, which might disadvantage software maintainers in reading and understanding.
evolution 26 35]. In contrast, Fig. 12 shows a human-
The statistics on the code change for instances without resolved issues are shown in Tab. From the table, our framework can generate applicable code changes including up 13 files and 28 hunks, and the location of the modifications can be as far as line 7, 150, with a single modification reaching up to 3,367 lines. These results suggest that our method has strong adaptability in generating applicable code changes. However, considering that these code changes have not passed all the potential test cases they could pass, which indicates that there is still room for improvement.
To further analyze the reasons behind the failure of test cases in these instances, we have quantified the distribution of the lengths of code changes in the unresolved instances, shown in Fig. g From the figure, we observe that for unresolved instances, the framework tends to delete a larger number of lines while adding fewer lines, in contrast to the distribution of human-writted changes. This discrepancy may point to different repair strategies or attitudes towards problem-solving where the framework presented herein might prefer to reduce errors by removing potentially problematic code, whereas human developers may lean towards adding new code to address issues.
35
25
27
20
425
15
15
300
40
17
10
6
150
Add
Batt
Figure 8: Distribution of the LoC in the resolved in- stances.
Figure 9: Distribution of the LoC in the applied hut not resolved instances.
19

---

lafle
Ma Toaee', TROT viteel, par
is que Uo val, lenied if just stare ott
21.
bert derie recu Voia, kine, Me
544
54.
118
clarance,
23
130 esl. .usaane pome
214
210
The resitt apried
Homeel less
Lobal, tar certar
114
317
155
redee yosre po 11 10 Voe0 1er1
callve matts from
MiLreedi mewaaatii John wntodtinn eecoee 411 deiik,
-12 13 +SEL.S KAmENts Calaiters
Latto's
10
rAwii (er))
W0
(werk reters, X Lites stggteaah))
th Suaii locill splee soriides woitdngg
test lalts neyel)
test Snartta terrta
nerter-cnee
M0
veygan
-001 wirck ael Lusenit, E.Sttee,, strll geight RCW, 2000 Lommine
IiIl gllner
1P
"Eulesten (E): Nosallyy daptiutt wittt.
caniege saloniing stallatee i)
Convrgeroientte, steckleeeil)
teen
lit oodeen, bist Tarts, aedt owers,, knet the
13
vitan IecLaa dens, Lnte, teit, twr Eeruuiter
47
11
slas
Figure 10: Case from scikit-learn (ours, after re- view) for the issue
Figure 1 11: Case from scikit-learn (ours, before review) for the issue (4].
Moreover, a comparison between the resolved instances and not resolved ones shown in Tab. 5 reveals that the latter contains higher overall number of files, hunks, and changed lines of code. These instances, involving more modification locations, correspond to more complex scenarios. This phenomenon suggests that the performance of our framework in resolving such complex issues requires further enhancement.
Furthermore, the variability in difficulty across different software repositories may influence the effectiveness of code changes. To this end, we compile statistics on the resolved ratios in various software repositories, as shown in Fig. [3] From the figure, we observe that there is a significant variation the resolved ratios across different repositories our framework. Some repositories have a resolved ratio as high as 40%, while others are close to 0%. This suggests that the differences among various software such as code structure and coding style can impact the generation and application of the code change.
20

---

offrred Migertin the Tacter, "Aet!
10
230
M4
for stggle fork
Jost
Sextine
Ces op
(teead)
19
Laneis, leurita, sedeen, fan.. Saggir
04
modts Nealieto
silayt/ommnn Jiniaa,, 1961 Lig L.aame,.
and
Figure 13: The number of applied and resolved in- stances in different repositories.
Figure 12: Case from scikit-learn (gold) 40
Table The statistical analysis of our framework on resolved and applicd but not resolved instances.
Resolved Instances
Applied but Not Resolved Instances
MAGIS
Gold
MAGIS
Gold
Min
Max
Min
Max
Min
Max
Min
Max
# Code Files
1
2
1.02
1
1.04
1
1.50
1
# Functions
1
2
1.02
1
2
1.04
1.50
1
1.61
# Hunks
1
4
1.45
1
6
1.66
1
2.52
1
3.72
# Added Lines
146
9.75
0
4.34
920
40.38
0
3,050
28.27
8 Deleted Lines
0
77
5.27
0
115
5.16
0
9,160
327.27
0
2,975
14.51
Change
1,655
1,657
4,568
Change End Index
1,665
301 66.8
0
1,666
315.05
9
7,150
513.13
0
6,658
728.96
# Changed Lines
2
190
15.02
115
9.50
9.367
367.65
6,025
42.79
G Evaluation on Task Description
Since there is no ground truth for the task descriptions generated by the Manager, we utilize GPT-4 to simulate human evaluation and score each task description based on its corresponding reference code change. Table 6 illustrates the standards used by GPT-4 assess the correlation between the task description and the code change. The score given by GPT-4 is considered the performance metric for the task description.
Table 6: The meaning of scores in GPT-4 evaluation on the correlation between the generated task description and the reference code change.
Score
Mcaning
1
The code changes are unrelated to the task description.
2
The code changes address a minor part the task but are largely irrelevant.
3
The code changes partially meet task requirements but lack completeness or accuracy.
The code changes relevant and mostly complete, with minor discrepancies from the task description.
4
The code changes perfectly align with the task description, fully addressing all specified requirements with high accuracy and completeness.
5
21

---

H
Case Study
Fig. iiilusstrates the detailed process of our framework used to resolve the issue from the Django repository [15] as described in the following ticket 6 To address this issue, two candidate files were identified for modification. Based on the issue description and the candidate files, the Manager defined two file-level tasks. For these tasks, two Developers were assigned: Django Database Specialist (Developer I) and Alex Rossini (Developer II). Following a kick-off meeting attended by both Developers and Managers, the Django Database Specialist commenced work first, followed by Alex Rossini. During the coding phase, Developer identified the code lines be modified and generated the new code to replace them. The initial code changes made by Developer were approved by the QA Engineer. Developer II made three attempts during the coding process. The QA Engineer provided review comments on the first two attempis. Ultimately, both Developers completed their coding tasks, and the merged results from their code changes passed all necessary tests.
Fig. 15 shows a reference issue resolution result, which resolves the issue from the repository Django 15], the human developer modifies four hunks in two files [T6]. Despite the presence of modifications in two files, our method focuses on changes in only one file, as shown in Figure I6 Notably, this simpler modification allows the repository to pass all necessary test cases.
I The performance of the QA Engineer Agent
Fig. 12 shows an issue 41 from the repository scikit-learn 33 and the reference code change 40). During the flow of our framework, the Developer firstly modifies the code as shown in Fig. 11 but the parameterrandom_sste
(Line 371 in the new-version code) of the function not assigned the right number in seeds. After the erroneous modification was made, the QA Engineer identified the mistake and provided feedback. Their commentary highlighted the issue: "This code change modifies the implementation of K-means algorithm and doesn't seem entirely correct". They further elaborated, "Running the algorithm just one time could lead to worse results, compared to running it multiple times (n_init times) and choosing the best result, as was originally done". This critique specifically targets the flaw associated with the iterative process ("running times"). With the help the QA Engineer, the Developer further revise the code, and the final code change is shown in Fig. TO All of the necessary test cases passed after applying this code change.
treans_singl_
J Related Work (Detailed)
J.1 Large Language Models
Large Language Models (LLMs) refer to the pre-trained language models that contain a large number of parameters (62]. The parameter counts of these models typically range in the tens or hundreds of billions. Popular LLMs include the Generative Pre-trained Transformer (GPT) series, such as GPT- 3 [44). GPT-4 (38) and the open-source LLaMA [51] which publicly shares its weight information. The first version of the open- model LLaMA has parameters ranging from billion to 65 billion. Many researchers 688 20 have built upon the foundation of L.LaMA, implementing enhancements to forge new L.MMs. These L.MSs have demonstrated formidable natural language generation capabilities in general scenarios, with GPT-4, in particular, standing out [32] 63) It has consistently maintained
the top position in several rankings, including code generation, reflecting its significant potential in tasks related software engineering (25).
J.2 LLM-Based Multi-Agent System
With the powerful text generation capabilitics of LLMs, many researchers 23
48 10 56
43
52 61] have explored the construction of LLM-based Multi-Agent Systems, enabling them to accomplish tasks beyond the capabilities of the LLMs themselves. For example, MetaGPT (23] which simulates the Standardized Operating Procedures (SOPs) of programming team, completing tasks including definition, design, planning, coding, and testing through constructed roles (e.g.,
httts::/ccode http////gooooocccccooeteeeeeeeee https::/cooe.ddjagoo www.:oojnnooojjjjoo.o..o htt.ppooojec.coomccateett//////a
22

---

Project Manager
Repository
Custodian
Reváen
6itHub Issue
</>
Code
</>
Recuest
Human
kick-oft Peating
guild
Developer
Quality
Assurence encee
</>
</>
Repository
Planning
Coding
New Ropository
Nenory
</>
Code Cheraga
CONTLL NESSON
<
Repestorry
Credubtt File
Code Filos
Pla Planning
Squana wha
wdta proticted braeinege Eirgp
lako oneine de 4rB
specifiiclly
Divelture
dje www.eeteeeeeecccilleeeooooooo
Build
Team
</>
Chergg segi
srpaamen Wiirinn arening engroctie and senooreee ey, speciiiiully
Mert Flan
cap to Bitck cnt
</>
Jimmen www.eee mal wa
Tast
Stang datzzaes Specialist
Alax soutte
Develceer
Develver
Outa
Leons
10C
Coding
Eastondg serss terpeted
seem to aeitih from Parcgrdd
deve fity cesirnmiig naw socel and craating we tatte, This change tieiiia tiral
oae ctanges appar to the StEal Ricestivv catiing vanc Paas
chald to aeleoi paperiyy
ball cipofiiartly
caneess baceed enther
the cable syeter ao gas
rernod cirg ceea
votgget
Paolgnng ship
actasi ralo, a crtatine racine
tnts boom. to teite,
Wo changes mecessary
Regairso
Eequires mntor
All BaceEcsry Teats
Figure 14: Detailed overview of framework, MAGIS (Kick-off meeting refers Fig.
23

---

Suport
porsee Modeerorsetinng
froe Irspect Importt cleasto
free django magger Julper Iapart _aar__"__aappp
del "rite wwwitrringgeesttiingg
Oiiforlly teating Lin duestrings.
Boxol teww wwt.:/on///r///pe/////oo 4257//handiinng dowstring Lareenttttt
docviring not decatring. st-i
return
Hass
Inieett Line LevtLine, (striat)) for tiee lises if lire (strapp)) tereaned [Tinejj] Jennttd3] [Uaee[lddatt). for Thee Maae(so]]
parts docstrlog. Retum (ritle, baty, cetatits).
not decvirting
entturn
dbcatrinng cleandec (ecaarriin
parta[0] Tendpoens) on
+1,6 29
importilb teport taport, Eaport_ module
froe drapect Aaport cleands
Tree pathiltb Path
froe djangn.apes. feport apps
-266,7 x267,7 del pettooeteetttattteel thounge):
cortings
357
258
verbone docc....ooc....
verbas verbous destrtaggeesse))
((Iodde::)) oper.wecel_rom
utiia..pars, nett conndet "eada2", mededo opts arteilddd__aaa rane)
348
201
anganests".
Figure 15: Case from Django (gold) 16
34,7 134,8 def trin (occrrrnnggdecsrrnngg::
35
Convent tabs spaces and split Into lines
36
30
linus ducstring. expandtaabx)) (amamdtabx()sslitllaie sppittiinn))))
wint vin(len()ing Lee()ine,)str Istrip()) for Hines if Hiee (xtrlpp)))())
the rinima indentation (fiest doesn't
indent xin(lan(Line) Le((Line,striigg for line in lines[3]] if Tinw.Istrie()
18
telened [1fnes[]) return www.jjjn(nriimdd.stecc..
[nsennt]] for Hine in Hies[s:]]
Figure 16: Case from Django (ours) for issue 17

---

product managers, architects, project managers, etc.). This framework has achieved leading scores on the HumanEval [22 and MBPP (2). outperforming many LLMs, and researchers show its ability to complete a software establishment (e-g., code repository to play Gomoku game), indicating that a multi-agent framework can better leverage the capabilities of L.LMs in code generation tasks. Moreover, Qian et al. [3] designed ChatDev, virtual development company simulating a human development team, which decomposes requirements into atomic tasks assigned to the developer agents. Developers mitigate the hallucination that may arise with the LLM through mutual communication and self-reflection mechanisms. Experimental results show that ChatDev can complete the establishment of some small projects (averaging more than 5 files per project) in relatively short time (less than 7 minutes on average). However, these works focus on the transformation from the requirements to code and overlook the code change generation during software evolution, which requires not only understanding the requirement but also dealing with the large repository.
J.3 Automatic Bug Fixing
GitHub issue resolution is fundamental aspect of software evolution, with bug fixing being one of the most common scenarios. Fixing bugs involves both bug localization and repair. Previous researchers [65| 42 have developed methods to localize bugs before modifying the code. DreamLoc, proposed by Qi et al. [42), effectively models the characteristics
bug reports and source code files. For automatic program repair, Wong et al. [55] explored a retrieval-basee method, while Ye and Monperrus (59] proposed ITER, a generation- based method for handling fault localization re-execution. Additionally, some researchers generation models. Recently, Xia et al. [57] demonstrated that directly applying popular LLMs significantly outperforms existing APR methods, showcasing their potential for generating diverse and effective patches. Bouzenia et al. 77 introduced RepairAgent, an autonomous LLM-based agent that plans and executes bug fixes by dynamically interacting with various tools.
(33 54] have combined retrieval techniques with
K Limitation
The design of prompt words may impact the performance of LLMs, thereby affecting the validity and fairness of the results [T]. While this paper focuses on innovative aspects of the proposed framework design and relies on practical guidelines for the design of prompt word templates 46) to reduce the emergence of design biases, the complete elimination of the prompt bias is extremely difficult due to the inherent biases in the dataset instances and the limitations of API resources.
Prompt
Dataset The dataset contains a limited variety of software types. The evaluating dataset, SWE- bench, encompasses 12 repositories, which cover the Python programming language. However, this quantity remains insufficient compared to the diverse software projects available on GitHub. The code style, architectural design, and implementation techniques of these selected repositories, while representative, cannot fully reflect the diversity of all code repositories. In particular, the current dataset may fail encompass some specialized fields or different programming paradigms, such as microservice architecture (66) and functional programming [29]. This limitation implies that, although our framework is designed to be independent of any specific software, the validation of its effectiveness and general applicability might be affected by this limited sample scope. Therefore, applying the findings of this paper other code repositories may require further validation.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract (line 5 16) and introduction (line 46
68).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitation can be found in Appendix K
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The details about our framework are described Section experimental can be found in Section 4.1
3
The setup of the
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
Justification: The data will be made public in GitHub repository: co- pooolve-lab/magee
https:// /gg thub.com/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpss//niis.cc//
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccm wttpcccccullclgliii. wwwpiiigguies/e/eee www.iieeeeooeebubiiiice wwabbbmsssssiiii aasionolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental setting/details can be found in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Statistical significance the experiments can be found Tab. Tab. 3 etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
28

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: The experiments are conducted through LLMs' API rather than local compute resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
/ httppp///errrseccccaaaa//////lal
Answer: [Yes]
Justification:
The paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Positive societal impacts can found Section I
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper (27 that produced the dataset, SWE-bench.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, theodd, wwteccode..comddaiioo dattsatsssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
30

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "jrG77gdv9R", "reviewer": "Reviewer_15zT", "summary": "This paper proposes MAGIS, a multi-agent coding framework for solving patch generation tasks. The roles consist of Manager, Repository Custodian, Developer and QA engineer, with the task of planning, file location, file editing, and review, respectively. Experiments on the SWE-bench benchmark show that performance is improved over baselines.", "questions": "- SWE-bench Lite results (appendix D) should be in the body of the paper\n- Is ablation of the kickoff meeting possible? This is an interesting mechanism but it's hard to understand whether it is effective or not\n- Can you explain/show more about the prompts and exemplars (if exemplars are used) for each role?", "limitations": "yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed multiagent framework is simple and effective\n- The proposed model is effective with and without using the hints provided in SWE-bench\n- Ablations demonstrate the effectiveness of the QA engineer role", "weaknesses": "My concerns are along the evaluation and comparisons:\n- The main experiments are limited, as only GPT-4 is measured\n- Baselines such as SWE-agent are missing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the valuable time to review our manuscript and thanks for your positive comments (i.e., the simplicity and effectiveness of our method, and the effectiveness in various settings).\nWe are sorry for the confusion and unclear expression in the previous version. \nWe have addressed each of the comments and suggestions. \nPlease refer to our responses below for details.\n\n> W1: The main experiments are limited, as only GPT-4 is measured\n\nThanks for your comments. To further validate the effectiveness of our method, we conducted more experiments on other base LLMs. \nThe new experiments use DeepSeek (DeepSeek-V2-0628) [1*] and Llama 3.1 (405B) [2*] in addition to GPT-4 as the base model.\nThe corresponding results are shown below,\n\n| Base LLM   | Directly Use | MAGIS  | MAGIS (w/o hints, w/o QA) |\n|------------|--------------|--------|---------------------------|\n| DeepSeek   | 0.33%        | 12.67% | 11.00%                    |\n| Llama 3.1  | 1.33%        | 16.67% | 11.00%                    |\n\nPlease note that all prompts are identical to those we experimented with on GPT-4. In the \"Directly Use\" setting, the prompts are sourced from the SWE-bench, while prompts for other settings are designed by us.\nThe above table shows that our method has achieved a 38-fold performance improvement (to DeepSeek) and a 12-fold improvement (to Llama 3.1) compared to directly using these base LLMs.\nThe improvement in effectiveness validates that our method is general and can still unlock the potential of other LLMs in solving GitHub issues.\n\n\n---\n\n> W2: Baselines such as SWE-agent are missing\n\nThanks for your comments. The comparison with SWE-agent [3*] is discussed in Appendix D as SWE-agent is a contemporaneous work (the paper is publicly available at arXiv in May 2024). As shown in Table 4, our method achieved the resolved ratio of 25.33% on SWE-bench Lite, which is higher than the 18.00% reported by SWE-agent.\n\n---\n\n> Q1: SWE-bench Lite results (appendix D) should be in the body of the paper\n\nThanks for your advice. We will move the Lite results to the body in the revision.\n\n---\n\n> Q2: Is ablation of the kickoff meeting possible? This is an interesting mechanism but it's hard to understand whether it is effective or not\n\nThanks for your comments. The kickoff meeting serves as a transitional link. The minutes of this meeting are converted into a code representing the order of work (sequential or parallel) for each Developer agent moving forward. Therefore, the ablation of the kickoff meeting is not possible in our method. To make the mechanism clearer, one detailed example is provided in Figure 7 (Appendix B) on Page 17 and Figure 14 (Appendix H) on Page 23.\n\n---\n\n> Q3: Can you explain/show more about the prompts and exemplars (if exemplars are used) for each role?\n\nThanks for your advice. We will add the specific prompt content in the revision (one of the prompt templates for the Manager agent as shown below due to the context limit).\n\n```python\n# Prompt <em>P</em> (Line 5 in Algorithm 2)\nsystem_prompt = (\"You are a software development manager.\"\n                \"Your responsibility is to provide clear guidance and instructions to a developer regarding modifications or improvements needed in a specific code file. \"\n                \"This guidance should be based on the details provided in the issue description and the existing content of the code file.\")\nuser_prompt = (\"Review the issue description and the content of the code file, then provide specific instructions for the developer on the actions they need to take to address the issue with these files.\\n\"\n              f\"# Issue Description:\\n{issue_description}\\n# Code File:\\n{file_content}\\n\"\n               \"Respond concisely and clearly, focusing on key actions to resolve the issue. Limit your answer to no more than 100 tokens.\") \n```\n\n---\n\n*References* \\\n[1*] DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. *arXiv preprint* (2024).\\\n[2*] Dubey, Abhimanyu, et al. The Llama 3 Herd of Models. *arXiv preprint* (2024).\\\n[3*] Yang, John, et al. Swe-agent: Agent-computer interfaces enable automated software engineering. *arXiv preprint* (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xZbAgj6WmF", "reviewer": "Reviewer_gcvU", "summary": "This paper introduces MAGIS, a novel Large Language Model (LLM)-based multi-agent framework designed to address the challenge of resolving GitHub issues in software development. The authors conduct an empirical study to identify key factors affecting LLMs' performance in this task, including file and line localization accuracy and code change complexity. Based on these insights, they propose a collaborative framework consisting of four specialized agents: Manager, Repository Custodian, Developer, and Quality Assurance Engineer. These agents work together through planning and coding phases to generate appropriate code changes. The framework is evaluated on the SWE-bench dataset, demonstrating significant improvements over existing LLMs in resolving GitHub issues. Specifically, MAGIS achieves a resolved ratio of 13.94%, which is an eight-fold increase compared to the direct application of GPT-4. The paper also provides detailed analyses of the framework's components and discusses its effectiveness in different scenarios.", "questions": "1. Your study focuses on the SWE-bench dataset, which is limited to 12 Python repositories. Have you considered how MAGIS might perform on a more diverse set of programming languages and project types? What steps could be taken to validate the framework's generalizability beyond Python projects? \n\n2. The paper doesn't discuss the computational requirements of MAGIS. Can you provide information on the runtime and resource requirements of MAGIS compared to simpler approaches? How does this impact its practical applicability in real-world software development environments? \n\n3. How have you ensured that MAGIS is not overfitting to the specific characteristics of the SWE-bench dataset? Have you tested the framework on any GitHub issues outside of this dataset to validate its real-world applicability? \n\n4. While you describe the roles of different agents, there's limited analysis of how their interactions contribute to overall performance. Have you conducted any ablation studies on different interaction patterns between agents? This could provide insights into which aspects of the multi-agent approach are most crucial for success. \n\n5. While you mention using prompts, there's limited discussion on prompt design. Could you provide more details on your prompt engineering strategies and how they impact the framework's performance?", "limitations": "1. The authors acknowledge in Appendix K that the SWE-bench dataset, while representative, may not fully reflect the diversity of all code repositories, particularly in specialized fields or different programming paradigms. However, SWE-bench is known as one of the most challenging benchmarks, this might be ok. \n\n2. In Appendix K, the authors mention the potential impact of prompt design on LLM performance and the difficulty in eliminating prompt bias completely.\n\n3. The authors note in Appendix K that applying their findings to other code repositories may require further validation due to the limited sample scope.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality:\n1. The paper presents a in-depth empirical analysis of LLMs' performance in resolving GitHub issues, providing unique insights into the challenges of applying AI to complex software engineering tasks.\n2. The study leverage software engineering metrics, such as the line locating coverage ratio, to quantify LLM performance in code change generation.\n\nQuality:\n1. The empirical study demonstrates rigorous methodology, examining multiple factors affecting LLM performance including file localization, line localization, and code change complexity.\n2. The analysis employs statistical techniques to establish correlations between various complexity indices and issue resolution success, adding depth and reliability to the findings.\n\nClarity:\n1. The paper clearly articulates the gap between LLMs' performance on function-level tasks versus repository-level tasks, providing a strong motivation for the study.\n2. The authors present their empirical findings with well-designed visualizations and tables, making complex data easily interpretable.\n\nSignificance:\n1. The empirical analysis provides crucial insights that bridge theoretical understanding of LLMs with practical software engineering challenges.\n2. The findings directly inform the design of more effective AI-assisted software development tools, as demonstrated by the MAGIS framework.\n3. The study lays a foundation for future research in applying AI to software engineering tasks, potentially influencing the direction of both AI and software engineering fields.", "weaknesses": "1. Limited dataset diversity: The study relies solely on the SWE-bench dataset, which, while comprehensive, is limited to 12 Python repositories. This may not fully represent the diversity of real-world software projects.\n\n2. Computational resources and efficiency: The paper doesn't discuss the computational requirements or efficiency of MAGIS compared to simpler approaches.\n\n3. Potential overfitting to SWE-bench: The framework might be tailored too specifically to perform well on SWE-bench, potentially limiting its generalizability.\n \n4. Lack of user study: The paper doesn't include feedback from actual software developers on the usefulness and quality of MAGIS's solutions.\n\n5. Limited discussion on prompt engineering: While the paper mentions using prompts, it doesn't delve into the specifics of prompt design or its impact on performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for reviewing our manuscript and thanks for your positive comments (i.e., the in-depth empirical analysis, rigorous methodology, and the well-designed visualizations).\nWe are sorry for the confusion and unclear expression in the previous version. \nWe have addressed each of the comments and suggestions below.\n\n> Q1: Your study focuses on the SWE-bench dataset, which is limited to 12 Python repositories. Have you considered how MAGIS might perform on a more diverse set of programming languages and project types? What steps could be taken to validate the framework's generalizability beyond Python projects?\n\nThanks for your comments.\nWe acknowledge that the SWE-bench has limited diversity, which we discuss in Appendix K. While we recognize the need to evaluate MAGIS on a broader range of programming languages (PLs) and project types, we found that there are currently no available datasets in other programming languages.\nTo evaluate our framework's generalizability beyond Python projects, we plan to construct a dataset with various programming languages and projects, then validate MAGIS by collecting pull requests, issues, and test cases from popular repositories, setting up the testing environments, and executing MAGIS to assess issue resolution. This process aims to demonstrate MAGIS's effectiveness across diverse programming languages and project types.\n\n---\n\n> Q2: The paper doesn't discuss the computational requirements of MAGIS. Can you provide information on the runtime and resource requirements of MAGIS compared to simpler approaches? How does this impact its practical applicability in real-world software development environments?\n\nThanks for your comments. We will include the computational requirements of MAGIS in the revision.\nSpecifically, our framework resolves each issue in approximately 3 minutes, with an average processing time of under 5 minutes per instance (as noted in Appendix E). Our method utilizes GPT-4, so any machine capable of accessing the OpenAI API can run the experiments.\nWhile LLM-based multi-agent systems [1-3*], including ours, require more computational resources and time compared to simpler approaches, this investment is justified. Our method significantly increases the issue resolution rate from 1.74% to 13.94% (as shown in Table 2), allowing us to tackle problems that were previously unresolvable.\nIn summary, this paper emphasizes how to leverage LLMs effectively to enhance issue resolution and improve overall performance.\n\n---\n\n> Q3: How have you ensured that MAGIS is not overfitting to the specific characteristics of the SWE-bench dataset? Have you tested the framework on any GitHub issues outside of this dataset to validate its real-world applicability?\n\nThanks for your comments. \nFirst, our method operates as an LLM-based multi-agent system, which means it does not require training on a specific dataset. This characteristic helps prevent overfitting to the SWE-bench.\nWhile we have not yet validated our method on other GitHub issues, we acknowledge the importance of this step and plan to address it in future work. We understand that broader testing will enhance the generalizability of our results.\nFinally, the SWE-bench dataset comprises various repositories and real-world GitHub issues, making it a reasonable basis for evaluating our method.\n\n---\n\n> Q4: While you describe the roles of different agents, there's limited analysis of how their interactions contribute to overall performance. Have you conducted any ablation studies on different interaction patterns between agents? This could provide insights into which aspects of the multi-agent approach are most crucial for success.\n\nThanks for your comments.\nWe recognize the importance of analyzing the contributions of different agents in our system. In Section 4.2, we conducted an ablation study for the QA agent, and the corresponding results are shown in Table 2.\nWhile we have not yet performed ablation studies specifically on different interaction patterns between agents, we analyze the contributions of each phase—planning and coding—in Sections 4.3 and 4.4. These sections provide insights into how these phases impact overall performance.\nWe appreciate your feedback and will explore different interaction patterns in future work.\n\n---\n\n> Q5: While you mention using prompts, there's limited discussion on prompt design. Could you provide more details on your prompt engineering strategies and how they impact the framework's performance?\n\nWe appreciate your interest in our prompt design. In the revision, we will include a detailed discussion of our prompt engineering strategies and provide specific examples of the prompts used.\nOur prompt engineering strategies adhere to established best practices [4*]. Specifically, we construct our prompt templates in markdown format, ensuring that each prompt covers all relevant information required for the communication.\n\n---\n\n*References* \\\n[1*] Wang, Xingyao, et al. OpenDevin: An Open Platform for AI Software Developers as Generalist Agents. *arXiv preprint* (2024).\\\n[2*] Yang, John, et al. Swe-agent: Agent-computer interfaces enable automated software engineering. *arXiv preprint* (2024).\\\n[3*] Zhang, Yuntong, et al. Autocoderover: Autonomous program improvement. *arXiv preprint* (2024).\\\n[4*] Jessica Shieh. Best practices for prompt engineering with openai api. *OpenAI, https://help.openai.com/en/articles/6654000* (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1h2v3stETS", "reviewer": "Reviewer_L511", "summary": "The paper studies the reasons behind LLMs' failures in resolving GitHub issues, identifying key factors such as locating code files and lines, and the complexity of code changes. The authors propose a novel multi-agent framework, MAGIS, comprising four specialized agents: *Manager* and *Repository Custodian* for planning, *Developer* for coding, and *Quality Assurance Engineer* for code review. Experimental results show that MAGIS significantly outperforms GPT-4 and Claude-2, solving 13.94% on the 25% subset of the SWE-Bench dataset. Ablation studies confirm the effectiveness of each agent's role in the framework.", "questions": "1. In Section 2, I'm a bit confused about the calculation of coverage ratio. How is $[s_i, e_i]$ defined for pure 'deleting' and 'adding' operations? If many code adjustments occur in a single file, does it ensure that each adjustment contributes independently to the coverage? I mean, if model generated code is different from the reference code, the position behind is not aligned.)\n \n2. In Section 4, can you kindly provide more key implementing details, especially for the model comparison experiment in Table 2? Some helpful information may be the core instruction/input for each model, the number of interaction rounds, and whether code interpreter is allowed. Those are all related to the performance of the model.\n\n3. Do you submit (or plan to submit) the results to SWE-Bench official leaderboard? How do this method differ from other agent-based methods?", "limitations": "Yes, the authors adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles the challenging problem of using LLMs to resolve GitHub issues. The proposed method is intuitive and effective, achieving an eight-fold performance gain, compared to baseline GPT-4.\n\n- The paper conducts massive experiments on both the reason why LLMs fail in the process of resolving GitHub issues, and the role of each agent, providing insights for the research community.", "weaknesses": "- The reproducibility of experiments for baseline models like GPT-4 and Claude is unclear. The code and data are temporarily unavailable, hindering follow-up and comparison by other researchers. \n\n- Typos: There are minor typos, such as \"multi-agennt\" instead of \"multi-agent\" on page 2, line 63, and some missing \"%\" signs in the resolved ratio on page 7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the valuable time to review our manuscript and thanks for your positive comments (i.e., intuitive and effective method, massive experiments, and insights for the research community).\nWe are sorry for the confusion and unclear expression in the previous version. \nWe have addressed each of the comments and suggestions. \nPlease refer to our responses below for details.\n\n\n> W1: The reproducibility of experiments for baseline models like GPT-4 and Claude is unclear. The code and data are temporarily unavailable, hindering follow-up and comparison by other researchers.\n\nThanks for your comments. We will make the code and data publicly available. \n\n---\n\n> W2: Typos: There are minor typos, such as \"multi-agennt\" instead of \"multi-agent\" on page 2, line 63, and some missing \"%\" signs in the resolved ratio on page 7.\n\nThank you for carefully spotting the typo. All these typos will be corrected in the revision.\n\n---\n\n> Q1: In Section 2, I'm a bit confused about the calculation of coverage ratio. How is $[s_i, e_i]$ defined for pure 'deleting' and 'adding' operations? If many code adjustments occur in a single file, does it ensure that each adjustment contributes independently to the coverage? I mean, if the model generated code is different from the reference code, the position behind is not aligned.)\n\nWe are sorry for the unclear presentation. \n${[s_i, e_i]}$ indicates the range of lines in the file that have been modified, and the line numbers within this range are based on the file before the modifications are made.\nTherefore, ${[s_i, e_i]}$ should represent the whole file (i.e., ${[1, LastLineNumber]}$) for pure 'deleting' while it should be none (i.e., $\\emptyset$) for pure 'adding'.\nFor cases where a single file has many modifications, the line numbers are aligned based on `Git`, which uses the Myers diff algorithm [1*] as default.\nThe empirical study [2*] demonstrates that the Myers diff algorithm can ensure that each adjustment contributes independently to the coverage in most cases.\n\n---\n\n> Q2: In Section 4, can you kindly provide more key implementing details, especially for the model comparison experiment in Table 2? Some helpful information may be the core instruction/input for each model, the number of interaction rounds, and whether code interpreter is allowed. Those are all related to the performance of the model.\n\nThanks for your suggestions. We will add the specific prompt content and make the implementation clearer in the revision (part of it is shown below due to the context limit).\n\nSpecifically in Table 2, the prompts for directly using the LLMs are sourced from SWE-bench[3*] while other prompts for MAGIS in different settings are designed by us.\nFor each GitHub issue, our method interacts with the issue only once, which means the framework only generates one final result for an issue.\nMoreover, the code interpreter is not allowed to be used in our method.\n\n```python\n# Prompt <em>P</em> (Line 5 in Algorithm 2)\nsystem_prompt = (\"You are a software development manager.\"\n                \"Your responsibility is to provide clear guidance and instructions to a developer regarding modifications or improvements needed in a specific code file. \"\n                \"This guidance should be based on the details provided in the issue description and the existing content of the code file.\")\nuser_prompt = (\"Review the issue description and the content of the code file, then provide specific instructions for the developer on the actions they need to take to address the issue with these files.\\n\"\n            f\"# Issue Description:\\n{issue_description}\\n# Code File:\\n{file_content}\\n\"\n            \"Respond concisely and clearly, focusing on key actions to resolve the issue. Limit your answer to no more than 100 tokens.\") \n```\n\n---\n\n> Q3: Do you submit (or plan to submit) the results to SWE-Bench official leaderboard? How does this method differ from other agent-based methods?\n\nThanks for your comments. We plan to submit the results after the anonymous review period ends.\nThe main difference between our method from others is that MAGIS does not use external tools such as web browsers [4*].\nOur paper focuses on how to unlock the potential of LLMs for GitHub issue resolution and we conduct many empirical studies on analyzing the limitations of LLMs.\n\n---\n\n*References* \\\n[1*] Myers, Eugene W. An O (ND) difference algorithm and its variations. *Algorithmica* 1.1 (1986).\\\n[2*] Nugroho, Yusuf Sulistyo, et al. How different are different diff algorithms in Git?. *EMSE* 25 (2020).\\\n[3*] Yang, John, et al. Swe-agent: Agent-computer interfaces enable automated software engineering. *arXiv preprint* (2024).\\\n[4*] Wang, Xingyao, et al. OpenDevin: An Open Platform for AI Software Developers as Generalist Agents. *arXiv preprint* (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VwCAmidqYF", "reviewer": "Reviewer_1wgB", "summary": "This paper designs a Multi-Agent framework, MAGIS. Following the form of team problem solving, it designs four agents: Manager, Repository Custodian, Developer, and QA Engineer, which are used to decompose tasks, locate problem codes, generate codes, and review codes, respectively. They help the LLM to better locate problems and generate correct codes in a cooperative way. This framework can solve 13.94% of problems on the SWE-bench benchmark.", "questions": "See the questions in Weaknesses.", "limitations": "The authors explain the limitations of their approach in the paper.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper draws on the cooperation method of human team, assigns different roles to agents, and utilizes the powerful understanding ability of the big model for natural language, as well as the professional ability in specific fields stimulated by being assigned specific roles. Compared with simply calling the big model, the effect is improved.\n\n2. This paper draws on the software engineering idea very well. For example, the Kick-off Meeting method is adopted, which can help multiple developers clarify independent tasks, ensure that there is no conflict, and can determine the order of modification.", "weaknesses": "1. The paper uses 11 prompts in total, and the specific prompt content cannot be found in the main text, which makes people worry whether the experimental results will strongly depend on the quality of the designed prompts. After replacing other models, the effect may be uncontrollable.\n\n\n2. The framework is relatively complex and requires a large number of interactions with LLMs. In the end, compared with other baseline scores, whether these additional computing resources are necessary is not further analyzed in the paper.\n\n\n3. **SWE-bench clearly states that the hints field should not be used in the submission list, but from this paper, it is found that hints seem to be the key to MAGIS improvement.** Is this unfair to other methods?\n\n4. I noticed that there are a lot of new submissions on SWE-bench (Lite) recently. Do you need to do a new comparison? (Of course, I think it is reasonable not to compare, after all, they are working in the same period.)\n\n\n### Reference\n- [1] Opendevin: Code less, make more. https://github.com/OpenDevin/OpenDevin/, 2024.\n\n- [2] Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, et al. Coder: Issue resolving with multi-agent and task graphs. arXiv preprint arXiv:2406.01304, 2024.\n\n- [3] Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. How to understand whole software repository? arXiv preprint arXiv:2406.01422, 2024.\n\n- [4] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the valuable time to review our manuscript and thanks for your positive comments (i.e., drawing on the software engineering idea well, drawing on the cooperation method of the human team, and the improved performance).\nWe are sorry for the confusion and unclear expression in the previous version. \nWe have addressed each of the comments and suggestions. \nPlease refer to our responses below for details.\n\n> Q1: The paper uses 11 prompts in total, and the specific prompt content cannot be found in the main text, which makes people worry whether the experimental results will strongly depend on the quality of the designed prompts. After replacing other models, the effect may be uncontrollable.\n\nThanks for your comments. \n- **Prompt content:** We will provide the full set of prompt content in the revision (part of it is shown below due to the length limit). \n\n```python\n# Prompt <em>P</em> (Line 5 in Algorithm 2)\nsystem_prompt = (\"You are a software development manager.\"\n                 \"Your responsibility is to provide clear guidance and instructions to a developer regarding modifications or improvements needed in a specific code file. \"\n                 \"This guidance should be based on the details provided in the issue description and the existing content of the code file.\")\nuser_prompt = (\"Review the issue description and the content of the code file, then provide specific instructions for the developer on the actions they need to take to address the issue with these files.\\n\"\n              f\"# Issue Description:\\n{issue_description}\\n# Code File:\\n{file_content}\\n\"\n               \"Respond concisely and clearly, focusing on key actions to resolve the issue. Limit your answer to no more than 100 tokens.\") \n```\n\n- **More base LLMs:** Moreover, we replaced GPT-4 with other two LLMs (i.e., DeepSeek [1*] and Llama-3.1-405B [2*]), and the experimental results on SWE-bench Lite are shown below. \nPlease note that all prompts are identical to those we experimented with on GPT-4. In the \"Directly Use\" setting, the prompts are sourced from SWE-bench, while prompts for other settings are designed by us.\n\n| Base LLM | Directly Use | MAGIS  | MAGIS (w/o hints, w/o QA) |\n|---|---|---|---|\n| DeepSeek | 0.33% | 12.67% | 11.00% |\n| Llama 3.1 | 1.33% | 16.67% | 11.00% |\n\nThe above table shows that our method has achieved a 38-fold performance improvement (DeepSeek) and a 12-fold improvement (Llama 3.1) compared to directly using these base LLMs.\nThe performance improvement validates that our method is general and can still unlock the potential of other LLMs in solving GitHub issues.\n\n---\n\n> Q2: The framework is relatively complex and requires a large number of interactions with LLMs. In the end, compared with other baseline scores, whether these additional computing resources are necessary is not further analyzed in the paper.\n\nThanks for your comments. Compared with direct usage, LLM-based multi-agent systems [3-5*] including ours can be more complex and need many interactions. However, these additional computing resources are considered to be necessary and worthwhile because (1) The issue-resolving task is not a task that needs immediate resolution (in contrast with coding tasks such as IDE code completion) (2) The additional computing enables the model to resolve issues that were previously unresolvable (with the resolved rate increasing from 1.74% to 13.94%, as shown in Table 2).\nAs direct usage of LLM can resolve few GitHub issues, this paper focuses on how to better use LLM to resolve issues and improve performance.\n\n---\n\n> Q3: SWE-bench clearly states that the hints field should not be used in the submission list, but from this paper, it is found that hints seem to be the key to MAGIS improvement. Is this unfair to other methods?\n\nThanks for your comments. We reported the score (10.28%) under the setting without hints in Table 2.\nThis score shows that our framework performs approximately six times better than the base LLM, GPT-4, thus validating the effectiveness of our method even in the absence of hints.\nThe fairness of comparisons with other methods depends on whether these methods can access external information. For instance, in the design of OpenDevin [3*], there is a web browser (Browsing Agent), which allows it to obtain content from the web and these hints are available in GitHub before the issue resolution process begins.\n\n---\n\n> Q4: I noticed that there are a lot of new submissions on SWE-bench (Lite) recently. Do you need to do a new comparison? (Of course, I think it is reasonable not to compare, after all, they are working in the same period.)\n\nThanks for your comments and understanding! we are aware of the recent submissions on SWE-bench (Lite) and we compared the methods (Swe-Agent [4*], Autocoderover [5*]) that submitted their results before the submission deadline of NeurIPS 24 in Appendix D.\nThe experimental results are shown in Table 4 on Page 18. The main difference between our method from others is that MAGIS does not use external tools such as web browsers [3*]. Our paper focuses on how to unlock the potential of LLMs for GitHub issue resolution and we conduct many empirical studies on analyzing the limitations of LLMs.\nMoreover, thanks for your references, we will incorporate them [3,5-7*] into discussion.\n\n---\n\n*References* \\\n[1*] DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. 2024.\\\n[2*] Dubey, et al. The Llama 3 Herd of Models. 2024.\\\n[3*] Wang, et al. OpenDevin: An Open Platform for AI Software Developers as Generalist Agents. 2024.\\\n[4*] Yang, et al. Swe-agent: Agent-computer interfaces enable automated software engineering. 2024.\\\n[5*] Zhang, et al. Autocoderover: Autonomous program improvement. 2024.\\\n[6*] Chen, et al. CodeR: Issue Resolving with Multi-Agent and Task Graphs. 2024.\\\n[7*] Ma, et al. How to Understand Whole Software Repository?. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

**Thank you very much for your valuable time in providing these constructive comments and suggestions. We have addressed each of the comments and suggestions by adding more experiments or explanations:**

---

- **Experiments with Different LLMs (15zT, 1wgB)**: We added experiments with MAGIS using other LLMs, such as DeepSeek [1*] and Llama 3.1 [2*], in addition to GPT-4 as the base model. The results validate that our method is general and can still unlock the potential of other LLMs in solving GitHub issues.
- **Specific Prompt Content (15zT, L511, 1wgB)**: We added the specific prompt content to make the implementation clearer.
- **Generalizability (gcvU)**: We added steps to validate the framework's generalizability beyond Python projects in the limitation section in Appendix K.
- **Computing Resources (gcvU, 1wgB)**: We added more discussions about the computing resources, clarifying the necessity and worthiness of the additional cost.
- **Evaluation and Ablation Analysis (gcvU)**: We included more explanation about the evaluation and the ablation analysis.
- **Typo Corrections (L511)**: We corrected all identified typos and checked the paper.
- **Coverage Ratio Calculation (L511)**: We added more explanation about the calculation of the coverage ratio and cited the relevant diff algorithm [3-4*] for clarity.
- **Comparison with Contemporaneous Works (L511, 1wgB)**: We added the discussion about the difference between our method and other contemporaneous works [5-9*] in Appendix D and E.


---

We hope these updates address the reviewers' concerns. We remain open to further discussion and revisions.

---

*References* \
[1*] DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. *arXiv preprint* (2024). \
[2*] Dubey, Abhimanyu, et al. The Llama 3 Herd of Models. *arXiv preprint* (2024).\
[3*] Myers, Eugene W. An O (ND) difference algorithm and its variations. *Algorithmica* 1.1 (1986).\
[4*] Nugroho, Yusuf Sulistyo, et al. How different are different diff algorithms in Git?. *EMSE* 25 (2020).\
[5*] Wang, Xingyao, et al. OpenDevin: An Open Platform for AI Software Developers as Generalist Agents. *arXiv preprint* (2024).\
[6*] Yang, John, et al. Swe-agent: Agent-computer interfaces enable automated software engineering. *arXiv preprint* (2024).\
[7*] Zhang, Yuntong, et al. Autocoderover: Autonomous program improvement. *arXiv preprint* (2024).\
[8*] Chen, Dong, et al. CodeR: Issue Resolving with Multi-Agent and Task Graphs. *arXiv preprint* (2024).\
[9*] Ma, Yingwei, et al. How to Understand Whole Software Repository?. *arXiv preprint* (2024).

Meta-review: This work proposes a multi-agent approach (Sec. 3) to producing pull requests that solve issues in the SWE-Bench benchmark, inspired by an empirical analysis of common failure modes (Sec. 2). The primary novelty lies in the alignment of these agents with roles that may be found in a SE environment, which is implemented via a series of prompts. As most reviewers noted, it is important that these prompts are made publicly available to facilitate future work, which the authors have promised to do in the rebuttal. Reviewers generally appreciated the methodology and the positive results, leading to the decision to accept the work.

One of the main topics of discussion among the reviewers was the use of "hints" in the results. While these are not necessarily off-limits, their use by default when other work often does not use them can paint an incorrect picture of the results. The authors are strongly encouraged to report results without hints in the main tables, and move the results with hints to a different table. This should not subtract from the value of the work given the positive results reported with hints in Tab. 2.
Decision: Accept (poster)