RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph
Authors: Siru Ouyang
Wenhao Yu
Kaixin Ma
Zilin Xiao
Zhihan Zhang
Mengzhao Jia
Jiawei Han
Hongming Zhang
Dong Yu
Venue: ICLR 2025 Poster (2025)
Keywords: AI Software Engineering
Large Language Models
Code Intelligence
Graph Structure
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Large Language Models (LLMs) excel in code generation yet struggle with modern AI software engineering tasks. 
Unlike traditional function-level or file-level coding tasks, AI software engineering requires not only basic coding proficiency but also advanced skills in managing and interacting with code repositories. However, existing methods often overlook the need for repository-level code understanding, which is crucial for accurately grasping the broader context and developing effective solutions. On this basis, we present RepoGraph, a plug-in module that manages a repository-level structure for modern AI software engineering solutions. RepoGraph offers the desired guidance and serves as a repository-wide navigation for AI software engineers. We evaluate RepoGraph on the SWE-bench by plugging it into four different methods of two lines of approaches, where RepoGraph substantially boosts the performance of all systems, leading to a new state-of-the-art among open-source frameworks. Our analyses also demonstrate the extensibility and flexibility of RepoGraph by testing on another repo-level coding benchmark, CrossCodeEval. Our code is available at https://github.com/ozyyshr/RepoGraph.

Published as a conference paper at ICLR 2025
REPOGRAPH: ENHANCING AI SOFTWARE ENGINEER- ING WITH REPOSITORY-LEVEL CODE GRAPH
Siru Ouyang', Wenhao Yu², Kaixin Ma², Zilin Xiao ; Zhihan Zhang', Mengzhao Jia",
Jiawei Han', Hongming Zhang", Dong Yu²
University of Illinois Jibana-Champaige 2 Tencent Al Seattle Lab
Rice University, University of Notre Dame
si wrruuuilll1 8. edu
ABSTRACT
Large Language Models (LLMs) excel in code generation yet struggle with mod- ern Al software engineering tasks. Unlike traditional function-level or file-level coding tasks, Al software engineering requires not only basic coding proficiency but also advanced skills in managing and interacting with code repositories. How- ever, existing methods often overlook the need for repository-Iews code under- standing, which is crucial for accurately grasping the broader context and devel- oping effective solutions. On this basis, we present REPOGRAPE APH, plug-in mod- ule that manages repository-level structure for modern AI software engineering solutions. REPOGRAPI offers the desired guidance and serves as a repository- the
wide navigation for Al software engineers. We evaluate REPOGRAPH SWE-bench by plugging it into four different methods of two lines of approaches, where REPOGRAPH substantially boosts the performance of all systems, lead- ing to a new state-of-the-arl among open-source frameworks. Our analyses also demonstrate the extensibility and flexibility of REPOGRAPH by testing on other repo-level coding benchmark, CrossCodeEval. Our code is available at https: htt///gtttuu/e com/ wwwyoorryyeeeeeeeeeeeeeee reyyshr/RepoGrahh
INTRODUCTION
Recent advancements in large language models (LLMs) have showeased their powerful capabilities across various natural language processing tasks (OpenAI, 2023; Anil et al., 2023; Dubey et al., 2024), and now, coding-specific L.MSs are emerging to tackle complex software engineering chal- lenges (Hou et al., 2023; Fan et al., 2023), such as Code-Llama (Rozière al., 2023) and Star- Coder (Li et al., 2023a). These coding-specinn software engineering tasks, even achieving human-level performance in many function-level coding tasks, such as program synthesis (Chen et al., 2021; Austin et al., 2021), 1), code annotation (Yao et al., 2019), bug fixing (Tufano et al., 2019), and code translation (Rozière et al., 2020).
LLMs are capable of assisting users with various
Real-world software engineering often extends beyond single function or self-contained code files. Applications are typically built repositories containing multiple interdependent files, modules, and libraries (Bairi et al., 2024). These complex structures require a holistic understanding of the entire codebase to perform tasks such as code completion (Shrivastava et al., 2023; Ding et al., 2023), feature addition (Liang et al., 2024), or issue resolving (Jimenez et al., 2024). Recent benchmarks like SWE--Bncc (Jimenez al., 2024) have been proposed evaluate LLMs on real-world GitHub issues. It requires LLMs to modify the repository to resolve the issue, either by fixing a bug or intro- ducing a new feature. This task is particularly challenging because it requires navigating complex code bases, understanding intricate dependencies between code files, and ensuring that changes in- tegrate seamlessly without introducing new issues, which highlights the difficulties in scaling from function-level to repository-leve understanding, expounded in Figure 1.
A key step in addressing repository-level tasks is to understand the structure of a repository and iden- tify related code. To achieve this, retrieval-augmeente generation (RAG) and its variants (Xiao et al.,
Work done during internship Tencent AI Seattle Lab.

---

Published as a conference paper at ICLR 2025
def first repeated chanlssrry for index,c in enumerate(stri): If stsrr]iiddx-11.coo c).unnt)() > 1:return
(a) Function-level Coding Problem
Input text: Write a python function find the first repested character given string.
return "None"
(b) Repository-level Coding Problem
@ Understand intricate dopendencies
Unit test:
Issue: modeling's "separability matrix"
Generated patch:
test, cooodd manri
X
does not compute separability correctly for CompoundModdels..
diff -git s/strooyy/modelngg separable
test, colot
lest_ariln_.g Lipper
Code repository:
desttroyymmodellng sseperallleeepp py ++* blastropy/modal wwwprroppmoddllgggappaooo ingaspprrablee py
astropy care.py
fitting.py earth.py
(i)) Navigate complex
25 242,7 +242,7 @8 def _.stackdjeft.
(iil) Resolve without introducing new
coordinates
codebases
right)...
Figure The illustration of (a) a finction-level coding problem from HumanEval (Chen et al., 2021) and (b) repository-level coding problem from SWE-Bench (Jimenez et al., 2024).
2023; Zhang et al., 2023; Phan et al., 2024; Wu et al., 2024) have been leveraged, in a procedural manner, to retrieve relevant code files across the repository first, providing context for LLMs for fur- ther edition. However, indexing at file-level can only identify semantically similar but not genuinely related code snippets. Instead of using RAG, recent approaches like Agentless (Xia et al., 2024) construct a skeletal format for each file, and directly prompt LLMs to identify relevant files and code lines. However, this method still treats code repositories as flat documents (Zhang et al., 2024), which suffers from limitations of repository structure such as the intricate inter-dependencies across files. An alternative approach to design agent frameworks (Yang et al., 2024; Wang et al., 2024), which enables LLMs to interact with repositories using actions. While LLM agents can freely deter- mine the next action based on current observations, without the grasp of global repository structures, they tend focus narrowly specific files, resulting in local optimums. Addressing these limita- tions requires going beyond semantic matching and developing techniques that enable a deeper un- derstanding of the codebase structure. This will allow LLMs leverage fine- ane-ggrined context across multiple files and function calls, facilitating more informed, repository-wide decision-making for coding tasks.
Motivated by this, we propose REPOGRAPH, a plug-in module designed to help LLMs-based AI programmers leverage the code structure of an entire repository. REPOGRAPH a graph structure and operates at the line level, offering a more fine-grained approach compared to previous file- level browsing methods. Each node in the graph represents a line of code, and edges represent the dependencies of code definitions and references. REPOGRAPH is constructed via code line parsing and encodes the structured representation of the entire repository. Sub-graph retrieval algorithms then used extract ego-graphs, which represent the relationships of central node (in our case, specific keywords). These ego-graphs can be smoothly integrated with both procedural and agent comprehensive context for LLMs to solve real-
frameworks, offering key clues that provide world software engineering problems.
To assess REPOGRAPHH effectiveness and versatility plug-in module, integrate it with four existing software engineering frameworks and evaluate its performance using SWE--hechh a recent benchmark for AI software engineering. Experiment results show that REPOGRAPH boosts the suc- cess rate of existing methods for both agent and procedural frameworks by achieving an average relative improvement of 32.8%. We also test REPOGRAPI on CrossCodeEvval to verify its trans- ferability to general coding tasks that require repository- code understanding. Additionally, we systematically analyze different sub-graph retrieval algorithms and integration methods. Together with error analyses, we hope to shed light on future works targeting modern Al software engineering.
RELATED WORKS
2.1 LLM-BASED METHODS FOR AI SOFTWARE ENGINEERING
Recently, there has been significant increase in research focused Al-driven software engineer- ing, which can be broadly categorized into two primary approaches: (i) LLM agent-based frame- works and (ii) SWE-featured procedural frameworks. While this field has advanced rapidly, with most methods being released as proprietary solutions for industry applications (Cognition, 2024), related work section will concentrate specifically on open-source frameworks.
2

---

Published as a conference paper at ICLR 2025
LLM agent-based framework equips large language models (LLMs) with set of predefined tools, allowing agents to iteratively and autonomously perform actions, observe feedback, and plan future steps (Yang et al., 2024; Zhang et al., 2024; Wang et al., 2024; Cognition, 2024; Ouyang al., 2024; Tang et al., 2025). While the exact set of tools may vary across different agent frameworks, they typically include capabilities such opening, writing, or creating files, searching for code lines, running tests, and executing shell commands. To solve a problem, agent-based approaches involve multiple actions, with each subsequent turn depending on the actions taken in previous ones and the feedback received from the environment. For example, SWEE-agent (Yang et al., 2024) facilitates interactions with the execution environment by designing a special agent-computer interface (ACI). There are various actions, including "search and navigation", "file viewer and editor", and "con- text management". Another work, AutoCodeRover (Zhang et al., 2024), further offers fine-grained searching methods for LLM agents in better contexts without an execution process. Specifically, it supports class and function-level code search. OpenDevin (Wang et al., 2024), initiated after Devin (Cognition, 2024), is community-driven platform that integrates widely used agent systems and benchmarks. The action space design of OpenDevin is highly flexible, requiring LLM agents to generate code the fly.
SWE-fcatured procedural frameworks typically follow a two-step Localize/Searc--Edi ap- proach, as seen in existing literature (Zhang al., 2023; Wu et al., 2024; Liang et al., 2024; Xia et al., 2024). The localize step focuses on identifying relevant code snippets, while the edit step involves completing or revising the code. Some works introduce additional steps to further enhance performance, such as the Seurch-Expund--PP 2020) is a popular technique used for localization, allowing models to search for relevant code snip- pets from large repositories by treating issue descriptions queries and code snippets as indexed data. Some approaches use a sliding window to ensure completeness (Zhang et al., 2023). Besides, Agentless (Xia al., 2024) is recently developed method that uses LLMs to directly identify relevant elements for editing within code repositories. structure to generate a format that aligns files and folders vertically, with indents for sub-directories. This structure and the issue description are then input into the LLM, which performs a hierarchical search to identify the top-ranked suspicious files requiring further inspection or modification.
approach (Phan et al., 2024). Retrieval (Lewis et al.,
first recursively traverses the repository
2.2 REPOSITORY-LEVEL CODING CAPABILITY
The evaluation of coding capabilities in AI sys- tems has traditionally focused on function-level or line-level assessments (Lu et al., 2021; Chen et al., 2021; Austin et al., 2021), where indi- vidual code snippets or isolated functions are the primary units of analysis. Unlike previ- ous studies, SWE-bench (Jimenez et al., 2024) highlights the trend of repository- level coding, driven by recent advances of coding- -pding-specific ILMS (Guo et al., 2024; Li et al., 2023b). It reflects the growing user demand to understand and contribute to entire projects rather than iso- lated functions (Ouyang et al., 2023), as well as solving real-world problems in an end-to-end and automatic manner.
Table 1: Comparison between our approach RE- POGRAPH and cxisting methods for representing the repository on various aspects. "depoUnder- stander (Ma et al., 2024) and CodexGraph (Liu et al., 2024) are concurrent works to ours.
Model
Live-level File-level Repo-level
DraCo
X
Aider
x
RepoUnderssander**
CodexGraph*
X
/
REPOGRAPH
Long before the LLM era, the software engineering community has been studying the navigation of code repositories, including using eye-tracking data (Busjahn al., 2015) and exploring intercon- nections (Begel et al., 2010). Then pre-trained code LLMs incorporate repository-Ievel information such as file dependencies. But tasks at the repository level often involve more intricate call relation- ships within their context. Recent works like RepoCoder (Zhang et al., 2023) and RepoFuse (Liang et al., 2024) have started integrating RAG modules to harness additional information from reposito- ries. Building on this, subsequent research has focused on embedding repository-level context into their methodologies. For instance, DraCo (Cheng et al., 2024) introduces importing relationships between files, CodePlan maintains the code changes by LLMs with incremental dependency anal- ysis, while Aider (Gauthier, 2024) employs PageRank (Page, 1999) to identify the most significant contextual elements. RepoUnderstander (Ma et al., 2024) and CodexGraph (Liu al., 2024) model
3

---

additional Published as a conference paper at ICLR 2025
Description
Code Repo
Mode ing "aparability matrix dowe not compule supurablly yy conecilly nested CompoundMocelss Consider the following model:
entioppy
modeling
core py
hrbingPy
Integration with Prucedural F Frameworks
notecss
earth py
search torm soarch moogniah Sipponiiry mooM
locolization
core
Py
logeerpy
"Model
edition
Code snippet
genee patch
and sode les Llanssndd
REPOGRAPE
repographigammn
space for agant
seff, defuult del propare ingimbbll,
A
Node information
Amitiaizztion
ippu e solf, valitiio ixput usisg Spult iosti--
Observation
del hndommerrdd..... -...............................
fname: oy
kind: def
Planning
hame: Mudha fruemee eatsegehooia wwwpprooiiiiioooaggoo. outegerg: clam...
aategory: class
Fnd the depardenntas for "sppaalility. matti
refernncs node inwckes
2 clas
line: methods: ppupure."puul....
oepograp gaaarnt
definiton nods
contains
3 function
RepoGraph Construction
(c) Integration with Agent Frameworks
An in-depth illustration of (a) the construction, (b) the integration with procedural frame- works, and (c) the integration with agent frameworks of REPOGRAPH. Given a code repository, we first utilize AST to construct G = {V.E), where G consists of "reference" " and "definition" node, & includes "invoke" and "contain" relations (files and code lines shown in corresponding color). The constructed REPOGRAPII are then used in procedural frameworks by adding sub- retrieval results into each step, and agent frameworks by adding graph retrieval action "search repograph". A simplified version can be found in Figure 10.
Figure 2:
code files as knowledge graph. Despite similarities representation, methods vary how they re- trieve information from these structures and utilize it for downstream tasks. Table 1 summarizes the differences between these methods and REPOGRAPIL REPOGRAPE surpasses previous approaches by effectively integrating context at the line, file, and repository levels.
REPOGRAPH
This section introduces REPOGRAPIL, novel plug-in module that can be seamlessly integrated into existing research workflows for both agent-based and procedural frameworks. The primary goal of REPOGRAPH is to provide structured way analyze and interact with complex codebases, enabling detailed tracing of code dependencies, execution flow, and structural relationships across the repository. In the following sections, will provide detailed description of REPOGR AH''s construction, its underlying representation, and its utility across various scenarios. The overall ar- chitecture is depicted in Figure 2, highlighting its key components and operational flow.
3.1 CONSTRUCTION
Given a repository-level coding task, the first step is carefully examine the repository structure so that the necessary information can be collected. The input for REPOGRAPH construction is a repository, i.e., a collection of its folders and files, while the output a structured graph, where each node is code line, and each edge represents the dependencies in between. REPOGRAPII enables tracing back to the root cause of the current issue and gathering dependent code context help solve the problem. The construction process of REPOGRAPH could be divided into three key steps.
Step 1: Code line parsing. We first traverse the entire repository using a top-down approach to identify all code files candidates for next-step parsing. This is accomplished by filtering based on file extensions, retaining only those with relevant code file suffixes (c.g., -py) while excluding other file types (e.g., .git or requirements.tx)) which might be noisy and irrelevant for coding tasks. For each code file, we utilize tree-sitter to parse the code, leveraging its Abstract Syntax Tree (AST) framework. The AST provides a tree-based representation of the abstract syntactic structure of the source code, enabling the identification of key elements such as functions, classes, variables, types, and other definitions. While recognizing these definitions is crucial, tracing their usage and refer- ences throughout the code is equally important. Trec-sitter facilitates this by capturing the definitions
'https://pype httptp/:///ppi.orr////jee/////// ree- sitter- bitteer-anguuges, languages/
4

---

Published as a conference paper at ICLR 2025
and tracking where they are utilized or referenced within the codebase. For example, in figure 2, we not only identify definitions like class Model and its inherent methods but also references like self._ inpuuttuttniptt After processing each line of code with a tree-sitter, we selectively retain lines that involve function calls and dependency relations, discarding extrancous information. Our focus is primarily on the functions and classes, as these represent the core structural components of these elements and their interrelationships, REPOGRAPH optimizes
the code. By concentrating the analysis process by excluding less significant details, such as individual variables, which tend to be redundant and less relevant for further processing.
Step 2: Project-dependent relation filtering, After the previous parsing step, we obtain code lines with calling and dependency relations. However, not all relations are useful for fixing issues. Specif- ically, many default and built-in function/class calls could distract from the project-related ones. Therefore, we additionally introduce a filtering process that excludes the repository-independeee lations. Two types of such relations exist: (i) global relation refers to Python standard and built-in functions and classes. (ii) local relation are introduced by third-party libraries, which are specific to the current code file. For global relations, we maintain a comprehensive list of methods from standard and built-in libraries, excluding any identified relations from this list. The list is empir- ically constructed by gathering methods of the builtins library and default methods such as "list" and "tuple". For example, in figure 2, line inputs len(input) is excluded since "len" is default method. For local relations, we parse import statements in the code to identify third-party methods that are included, and exclude them accordingly. A detailed illustration of this step can be found in Figure 11.
re-
Step 3: Graph organization. At this stage, we construct REPOGRAPH using code lines as the fundamental building units. The graph can be represented set of nodes, with each node corresponding to a line of code, and & represents the set of edges, capturing the relationships between these code lines. Each node in V contains attributes to represent its meta-informe informatinn such as classify each code line either "definition" (def) or a "reference" (ref) to a particular module. A "def" node corresponds to the line where a function, class, or variable is initially defined, while "ref" node indicates code line where this entity is referenced invoked elsewhere in the code. Similar to soft links to "def" nodes, "ref" nodes also represent other variations of invoking methods. For example, in Figure 2, the class definition "class Model" would be a "def" node, while any subsequent usages of Model would be "ref" nodes. Each "def" node may have multiple "ref" nodes associated with it, as single function or class can be referenced in various places throughout the code. We define two types of edges: Eirvoke and Econtain. The triple Econtein V2) denotes that Vi (e.g., function definition) contains another module 1/2 (e.g., an internal function or class). The edge Econtain typically connects "def" node to its internal components. In contrast, Exmroke represents an invocation relationship, usually connecting a "def" node to a "ref" node, where the reference node includes dependency on the definition.
g {V,E), where V represents the
file _aame, directory, etc. Additionally, we
3.2 UTILITY
The constructed REPOGRAPH serves as a structured representation of the current repository and facilitates better-related information collection and aggregation. For information collection based on REPOGRAPII, specifically, we use one search term each time for subgraph retrieval. Search terms are the key functions or classes that are determined by current states. For example, "separability _matrix" the initial search term Figure 2(c). We retrieve the k-hop ego-graphs (Hu et al., 2024) with the search term in the centric. The ego-graph is crucial for solving the problem because it focuses on the immediate relationships (Jin et al., 2024) around the search term, capturing the relevant dependencies and interactions within the repository, which is key to understanding the functional context. Additionally, the retrieved content explicitly contains information at both the method and line levels and implicitly expresses the grouping the file level.
This process is abstracted via search sauchhrepographl as illustrated in the middle of Figure 2. The retrieved k- k-hop ego-graph will be flattened for further processing. We also tried other variants for integration later in Section 5.2 and their performance Table 4. We narrate how REPOGRAPH could be plugged in with existing representative research lines in the following.
5

---

Published as a conference paper at ICLR 2025
Integration with procedural framework. In a procedural framework, LLMs are usually prompted in "localization" and "editing" stages with the given repository context and issue description. In this case, use search repograph() before both stages, leveraging our REPOGRAPH to assist in making more informed decisions at each step. For example, in Figure 2, we first include the subgraph of "separability matrrix" for localization, and then use the localized result "Model" search in the edition stage. To implement the strategy, we flatten the context of retrieved ego-graphs and append as part of the prompt. As result, the LLM generation is conditioned on both retrieved ego-graphs and the context provided by baseline methods, helping the model preserve nuances.
Integration with agent framework. A significant difference in existing agent frameworks is the action space design, as expounded in Section 2. To leverage the power of REPOGRAPII, we put search.repograph() as an additional action in the action space. The agent decides when and where to use this action. The search term is also determined by the agent accordingly. The returned subgraph will be flattened and used as an observation for the next state.
EXPERIMENTS
4.1 SETUP
We evaluated REPOGRAPH as a plug-in component, i.e., integrated into existing baseline models of the two aforementioned research lines assess its performance. We use the same baseline settings and configurations when incorporating REPOGRAPH to ensure a fair comparison.
Dataset. We test REPOGRAPII in SWE-bench-Lite Each problem in the dataset requires submit- ting a patch to solve the underlying issue described in the input issue description. The goal is to generate a patch that accurately revises the relevant portions of the codebase within the repository, ensuring that all test scripts included in the dataset are successfully executed.
Baselines. We integrate REPOGRAPH with representative methods from both aforementioned re- search lines. (i) For procedural frameworks, we evaluate the widely used traditional method, RAG (Lewis al., 2020), as well as Agentless (Xia al., 2024), an open-source state-of-the ap- proach in this direction. For RAG, we follow its initial setting and use BM25 for file-level retrieval. After that, we append the context of REPOGRAPIL after the code files as part of the prompt. Agent- less first performs hierarchical localization in terms of "fleecllssfuncctioneiits repair based on localization. The context of REPOGRAPH is inserted in every step of Agentless. (ii) For agent frameworks,
and then conducts
consider SWE-agent (Yang et al., 2024) and AutoCodeRover (Zhang et al., 2024). For both frameworks, we add an additional action "search repograph" for the LLM agent as described in Section 3. All the choices in the two research lines incorporate GPT-4 and GPT-4o-based methods to ensure generalization. Detailed implementations and prompts used can be found in Appendix A.
We evaluate all methods across two key dimensions: Accuracy and Average Cost. () For accuracy, we report the resolve rate and patch application rate. The resolve rate rep- resents the percentage of issues successfully resolved across all data points. An issue is considered resolved if the submitted patch passes all test scripts. To assess the patch application rate, we attempt to apply the generated patches to the repository using the patch program, counting only successful applications toward this metric. (ii) To evaluate cost efficiency, we report two metrics: average cost and average tokens, which refer to the inference cost and the number of input/output tokens used when querying the LLMs, respectively.
Evaluation metrics.
Configurations. We use the same GPT version as in the baselines in the experiments. We used GPT- 4o (2024-05-13) and GPT-4-Turbo (gpt-4-14 10111612ze0 (-6---iiieww from OpenAl for evaluation and analyses in experiments. All evaluation processes are performed in containerized Docker environment 3 ensuring stability and reproducibilityy made possible through contributions from the open-source community. Plug-ins with procedural frameworks usually take around 2-3 hours to finish. For agent frameworks, the inference time is larger, up to around 10 hours.
CCurentt leaderboard could be found at bttps://www.xebbedd... potttps:/giiie wttp:ttt......oooooeeeaeeetaaaee hwwpaal///wwweeehhhheeee bewch-dochhhrrr Shttps: thub. com/ swe- http:///wweeeeeeeeeeeeeeee/////s
n luat on ite

---

Published as a conference paper at ICLR 2025
Table 2: Results of REPOGRAPH with open-source baselines in two research lines, including pro- cedural and agent frameworks. Numbers of accuracy-related metrics are directly taken from the leaderboard, while the cost-related ones are computed from the corresponding trajectories".
Accuracy # samples
Avg. Cost
Methods
LLM
resolve
putch apply
cost
# tokens
Procedural franewokks
RAG
GPT-4
2.67
8
29.33
$0.13
11,736
+REPOGRAPI
GPT-4
5,33
1618
47.67118.33
S0.17
15,439
Agentless
GPT-4o
27.33
82
97.33
S0.34
42,376
+REPOGRAPH
GPT-4o
29.67
89
18.00
S0.39
47,323
Agent frameworks
AutoCodeRover
GPT-4
19.00
57
83,00
$0.45
38,663
+REPOGRAPH
GPT-4
21.33
64
86.67 13.67
50.58
45.112
SWE-agent
GPT-4o
18.33
55
87,00
$2.53
498,346
+REPOGRAPH
GPT-4o
20.33
61
93.33
$2.69
518,792
4.2 EXPERIMENT RESULTS
Table 2 presents the main evaluation results of all baseline methods and the correspond- ing performance with REPOGRAPH (#REPoGRAPH) ) as a plug-in in the SWF-bench-L.it test set. We also report the number of correct samples for each method. The performance in- crease is marked by tmmm. We also tested other (open-source) LLMs using Claude-3.5- (claude-33 3- ---------e---------------------- 20040-204462000 6020) (Anthropic, 2024) from Anthropic in Appendix C. Based on the results, we have the following key observations:
(i) REPOGRAPH brings consistent performance gain for all combinations of frameworks and LLM model bases. Specifically, REPOGRAPH achieves an absolute improvement of +2.66 and +2.34 in terms of the resolve rate for RAG and Agentless, respectively, which is 99.63% and 8.56% of relative improvement. The notable improvement demonstrates the effectiveness of REPO- GRAPH in adapting to various scenarios by inducing relevant code context and performing precise code editings. Additionally, our best performance far by plugging in Agentless, 29.67, achieves the state-of-the-an performance on the benchmark" among all open-source methods.
(ii) Performance gain brought by REPOGRAPH is slightly larger on procedural frameworks than agent ones. With procedural frameworks, REPOGRAPH correctly fixes more issues than agent ones. This could be due to two primary reasons. Firstly, we observed that mature procedural frame- works tend to achieve better baseline performance than agent-based frameworks on SWE-hench. The initial definitive nature of procedural frameworks, with their well-defined running flow and structure, allows them to leverage plug-in plug-ins more effectively. Another reason is that this determinis- tic behavior reduces the complexity that arises from dynamic secision-making, a key characteristic of agent-based systems, thereby enabling a smoother integration of performance improvements.
(iii) Performance gain brought by REPOGRAPH does not rely on more costs. We also compute and report each method's average cost and token consumption. By introducing REPOGRAPH, manage to reduce the costs associated with managing the entire repository while achieving compa- rable superior performance. As shown in Table 2, the additional token cost introduced by RepoGraph is justified given the significant performance improvement it enables, particularly when compared the substantially higher costs incurred by previous methods like SWE-agent. This indicates REPOGRAPI 's performance gains are not mainly due to increased token usage.
(iv) Average costs are generally larger on agent frameworks with REPOGRAPH. This phe- nomenon is especially obvious with SWE-ageet, as it allows the agent to freely determine the next action based on the current observation. We also found that the integration with agent frameworks usually leads to larger cost increases, as exemplified by +0.13$ and +0.18$ with AutoCodeRover and SWE-agent, respectively. The reason lies the large exploration space in agent frameworks. The agents might call the search_repographe action many times, which leads to the explosion of
'ttpps: / //wwww/w/ /wwwwwwwo swwbench.h...................... com/ We kindly request that reviewers refrain from intentionally check- ing submitter information the leaderboard maintain the integrity of the double-blind review process.

---

Published as a conference paper at ICLR 2025
prompt contexts. We encourage users be mindful of cost and to adopt more granular approach to cost control when integrating REPOGRAPH into the agent framework in the future.
5 ANALYSIS
This section presents detailed analysis to demonstrate that the additional context provided by REPOGRAPH is beneficial for the task. We begin by analyzing localization accuracy in comparison the gold-standard patch. Next, we explore various REPOGRAPH configurations, focusing on how the additional context can be effectively integrated into the existing system. Finally, we perform an in-depth error analysis, highlighting aspects where REPOGRAPH can be further improved. For more analyses including resolve rate in various aspects and action distributions of agent frameworks, please refer to Appendices C.
5.1 LOCALIZATION COVERAGE
A crucial step in issue resolution is accurately identifying the correct locations within the code that re- quire modification. Proper local- ization is essential, it forms the foundation for generating an ef- fective and accurate patch. With- out this step, the quality of the fix may be compromised, lead- ing to incomplete or incorrect so- lutions. We compute the per- centage of problems where the edit locations match the ground truth patch in three granularity. Namely, file-level, function-level, and line-level. We report that patch contains the correct location it edits a superser of all locations in the ground truth patch.
Table 3: Percentage of problems for accurate edition localiza- tions with respect to file, function, and line levels. All the num- bers are computed from the corresponding generated patches.
Methods
LLM
file
function
line
RAG
GPT-4
47.3
23.3
12.7
+REPOGRAPH
GPT-4
51.7 14.4
25.3
14.3
Agentless
GPT-4o
68.7
51.0
34.3
+REPOGRAPHI
GPT-4o
75.3
54.0
36.7+2.4
Agent frameworks
AutoCodeRover
GPT-4
62.3
42.3
29.0
+REPOGRAPH
GPT-4
69.0
45.7
31.7
SWE-agent
GPT-4o
61.7
46.3
32.3
+REPOGRAPH
GPT-4o
67.3
49.3
35.0127
Table 3 presents the results of our analysis. We observed that integrating REPOGRAPH with all baseline methods significantly improves file-level accuracy, whereas the enhancement of accuracy in line-level is comparatively modest. This result aligns with our expectations, ization is the most coarse- -ganiee, making it inherently easier to improve. In contrast, line-level localization, being the most fine-grained, poses a greater challenge due its need for more precise identification of code segments. Additionally, we found that although line-level accuracy improve- ments are more pronounced for agent frameworks, their overall resolve rate lower than that of procedural frameworks, as shown in Table 2. This discrepancy can be attributed to the fact that localization, while necessary for generating final patch, is insufficient. The success of the final revision still heavily relies on the underlying capabilities of LLMs. Agent frameworks, designed to operate in trier-and-error fashion, are particularly susceptible to error accumulation. As these frameworks iteratively refine their patches, small inaccuracies in earlier localization steps prop- agate and magnify throughout the process, ultimately reducing the overall resolve rate. Procedural frameworks, on the other hand, follow more structured and deterministic approach to localization and patch generation. They typically localize and fix issues in a single, more direct step, which can help mitigate the compounding of errors.
file-level local-
5.2 INVESTIGATION OF REPOGRAPH VARIANTS
In this section, we investigate the efficacy of various combinations sub-graph retrieval and inte- gration techniques as outlined in Section 3. We explore two sub-graph retrieval variants and two integration methods. Specifically, for sub-graph retrieval, we index k-hop ego-graphs where l is set to 1 and 2. We limit our exploration to & values up due to the extensive context required for integration and the potential introduction noise or irrelevant nodes for k 2 3. For the integration methods, we employ two distinct approaches: (i) directly flattening the textual sub-graph by explic-
8

---

Published as a conference paper at ICLR 2025
Table 4: The number of nodes, edges, and tokens of REPOGRAPH and its variants. For different retrieval and integration variants, we report the average number on the test set. "summ." refers to the summarized version by LLMs of the retrieved ego-graph.
Metries
REPOGRAPH 1-hop flatten
1-hop + summm.
2-hop + flatten 2-hop summ.
Nodes
1419.3
11.6
11.6
54.5
54.5
Edges
26392.1
37.1
37.1
89.9
89.9
# tokens
2310.7
717.5
10505.3
1229.2
resolve rate
29.67
28.33
26.00
28.67
itly detailing the relationships between the search term and its neighboring nodes, and (ii) leveraging an LLM first to summarize the sub- -uubgrapphhn in terms of the core modules and salient dependencies, be- fore proceeding with further processing. Detailed implementations of these variants and prompts used can be found in Appendix B.1.
We begin by presenting some statistics for REPOGRAPH and its various configurations. Perfor- mance evaluations are conducted on Agentless with REPOGRAPH integrated as a plug-in module. Table 4 reports the number nodes and edges within the (sub))graphs. Notably, the average number of nodes and edges for REPOGRAPH across the SWE-bench dataset is quite substantial, featuring over 1,000 nodes and 25,000 edges. This highlights the comprehensive nature of the constructed structure. For the different variants, when & 1, the information within REPOGRAPH is concen- trated around the search term, resulting in an average of 11.6 nodes and 37.1 edges. As & increases to 2, the retrieved ego-graph expands exponentially, reaching an average of 54.5 nodes and 89.9 edges. Moreover, directly flattening the retrieved ego-graph often significantly increases the token count, frequently reaching several thousand tokens. However, utilizing an LLM as an additional summarizer greatly reduces the token count, typically around a thousand.
Table 4 presents the resolve rates for four variants of method. Notably, while the 2-hop vari- ant incorporates additional information, directly flattening this information results in the poorest performance, with resolve rate of 26.00%. even lower than the original baseline. In contrast, incorporating summarization via the LLM significantly alleviates context length constraints and enhances information organization, thereby improving performance. For the 1-hop variant, how- we observed that adding summarization actually degrades performance, reducing the resolving rate to 38%3.. We hypothesize that this occurs because the flat 1-hop ago-graph already contains comprehensive information that fits within the LLM's context window; thus, summarization may introduce inevitable information loss.
5.3 TRANSFERIBILITY TEST
To demonstrate the representational power of REPOGRAPH for repositories and its trans- ferability to tasks requiring an understand- conducted
Table 5: Results on the subset of CrossCodeE- val with GPT-4o and Deepseek-Codes Cooeeee-Cooer-V---Liiie V2-Lite- Instruct as the backbone LLMs.
ing of repository structures, experiments using the CrossCodeFVvl bench- mark (Ding et al., 2023). CrossCodeEval multilingual code completion benchmark that derived from real-world repositories. It is designed to emphasize numerous cross- file dependencies. We focus on problems using Python as the evaluation programming lan- guage, leading to 2,665 samples. For evaluation metrics, we follow the settings in the original paper and measure performance with code match and identifier match metrics, assessing accuracy with exact match (EM), edit similarity (ES), and Fl scores. our experiments, the search terms are determined to be the function within which the current line is to be completed.
Code Match
Identifier Match
Methods
EM
ES
EM
Fi
Deepseek-Coder
10.2
57.3
16.6
49.1
+REPOGRAPH
19.7
67.8
29.3
58.9
GPT-4o
10.5
59.6
16.8
47.9
+REPOGRAPH
28.7
68.9
36.0
61.3
Table 5 demonstrates the results on ErossCodeEval using GPT-4o and Deepseck-Codee Coder as the back- bone language model. The original LLMs struggle with repository-level tasks, evidenced by an EM score of only 10.8 for code matching and 16.7 7 for identifier matching. These results indicate significant limitations in handling code structure and variable usage in broader repository context.

---

Published as a conference paper at ICLR 2025
(e) Procedural framework
Agent framework
Agentloss
RopoGraph
SWE-agent
RopoGraph
5
77
12
16
39
22
18
Incorrect localization
Regressive fix
Contextual misalignment
Figure 3: Venn diagram of REpoGRaph and baselines on (a) procedural framework and (b) agent framework on SWE-Bench-Lite. We also plot the error distribution of failing cases against counter- parts, e.g., detailed error distribution of 12 cases REPOGRAPII succeeds while Agentless fails.
However, with the integration of the REPOGRAPH method, there is substantial improvement across all metries with both LLMss. Particularly, we find that the improvement brought by REPOGRAPH more pronounced when integrated with GPT-4o. A potential reason is that GPT-4o is better at inter- preting the additional context information provided for better code reasoning. These improvements suggest that incorporating repository-leve knowledge, as facilitated by REPOGRAPH, greatly en- hances the model's ability to understand and generate more contextually accurate and semantically consistent code.
5.4 ERROR ANALYSIS
We want to compare REPOGRAPH with the corresponding baselines to see the distribution of re- solved cases and analyze the error reasons for unresolved cases. We plot Venn diagram for repre- sentative methods in both procedural and agent frameworks in Figure 3, respectively. We manually examined all the unique error cases and defined three error categories.
(i) Incorrect localization refers to the failure in accurately identifying code snippets, (ii) contextual misalignment happens when the generated patch fails to align with the broader context of the codebase, and (iii) regressive fix introduces new issues in resolving the original issues. More examples are in Appendix D.
We found that the improvement agent frameworks more complementary than procedural frame- works, with larger uniquely resolved cases of 22 compared with 12 in procedural frameworks. To- gether, they make even larger performance of 31.33% and 22.33%. The reason could also be at- tributed to the determinism of procedural frameworks. As plug-in module, REPOGRAPH tends to make modifications on existing deterministic processes, resulting in larger overlaps in resolved issues compared with baselines. Agent frameworks, on the other hand, have quite different action distributions with REPOGRAPH as a plug-in (please refer to Appendix A.2). Therefore, the uniquely resolved cases are more compared with procedural frameworks. For error distributions, contextual misalignment is the most prevalent error type, followed by incorrect localization and regressive fixes for all methods, suggesting that while localization is often correct, the applied solutions may regress or fail to integrate contextually. The phenomenon also echoes the conclusion obtained in Section 5.1. This is intuitive as all the existing methods focus on providing a more comprehensive and desired context for LLMs to solve the task, which fundamentally depends on the power of backbone LLMs. We also found that when integrated with REPOGRAPIL the proportion of all three error types de- creases (specifically for "incorrect localization"). indicating that REPOGRAPHI is specifically useful in aggregating the related contexts of the current to-be-fixed issue.
CONCLUSION AND DISCUSSION
This paper introduces REPOGRAPH. a plug-in module for modern AI software engineering. REPO- GRAPH operates at the code-line level and offers desired navigation through code bases by incorpo- rating and aggregating information at the line level, file level, and repository level through subgraph real-world SWE tasks show that REPOGRAPE
retrieval of ego-graphs. Extensive experiments significantly boosts the performance both procedural and agent frameworks. In addition, REPO- GRAPH is proved to be smoothly and effectively transferred to other tasks that require repository- level understanding. Future work could further investigate the generalizability of REPOGRAPH across different programming languages, frameworks, and developer environments. Real-time ecution and feedback could also be explored to further enhance REPOGRAPH towards dynamic version for better AI software engineering abilities.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, lean-Baptiste Alayrac. Jiahui Yu, Radu Soricut, Jo- han Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Tim- othy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Dani- helka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and al. Gemini: A fam- ily of highly capable multimodal models. CoRR, 2tbs//312.111000 2023. doi: 10.48550/RRXIV doii org/ /1/ 48550 2tt.5//axxxv..21....2.
2312.1 URL https::////// //doi.
Anthropic. Introducing claude 3.5 sonnet.
claude- 3- sonnet, 2024. Accessed: 2024-10-01.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, et al. Program synthesis with large language models. arXiv preprint 108.:322...7733, 2021.
Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Arun Iyer, Suresh Parthasarathy, Sriram Raja- mani, B Ashok, and Shashank Shet. Codeplan: Repository-Ievel coding using Ilms and planning. Proceedings of the ACM on Software Engineering, 1(FSE):675-693 698, 2024.
Andrew Begel, Yit Phang Khoo, and Thomas Zimmermann. Codebook: discovering and exploiting relationships in software repositories. In Proceedings of the 32nd ACM/IEEE International Con- ference on Software Engineering- Volume 1, ICSE 10, pp. 125-134, New York, NY, USA, 2010. Association for Computing Machinery. ISBN 978 97816055 6055871 166. doi: 10. 10.1 10.114//806799 1806821. URL wwt.tttps/// / /doii www.rrgg/.0.. 11 1145/1 80 67886667799 99....9999... 6821..
Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha E. Crosby, James H. Paterson, Carsten Schulte, Bonita Sharif, and Sascha Tamm. Eye movements in code reading: relaxing the linear order. In Andrea De Lucia, Christian Bird, and Rocco Oliveto (eds.), Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension, ICPC 2015, Florence/Firenze, Italy, May 16-24, 2015, pp. 255 265. IEEE Computer Society, 2015. doi: 10. 1991ICC..222.....6... URL https:t//////. / doii org.org////ooo 110. 10. 1109 ICPC/ 2015.366666 36.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. 2021.
Wei Cheng, Yuhan Wu, and Wei Hu. shafloo-guidee retrieval augmentation for repository-level code completion. In ACL, 2024.
Cognition. Devin, 2024.
URL https://ww..comiiiion..////
introducing devin.
Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Kr- ishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. Crosscodee- val: A diverse and multilingual benchmark for fross-file code completion. In Seieth- Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: // /wpp/reeeeeieeeeee/eee/eeeeeeeee ew net forum? w fs

---

Published as a conference paper at ICLR 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Amirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits. David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Ko- revaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadcokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lec, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR, abs/2407.21783, 2024. doi: 1044550/AXXV..407222272 URL https: / doi.oiiiooo 1tt.1..44000/////////
48550/5500400 arxiv. 2407 21783.
Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. Large language models for software engineering: Survey and open problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engi- neering (CCEE-OSE), pp. 31-53. IEEE, 2023.
Paul Gauthier. Aider is ai pair programming your terminal. https: chat/, 2024.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y.K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming the rise of code intelligence, 2024. URL https: / //arxiv.crgg///e/.......ooooao.a http://aas/2401111.1
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large language models for software engineering: A systematic litera- ture review. arXiv preprint arXiv:2308. 10620, 2023.
Yuntong Hu, Zhihan Lei, Zheng Zhang, Bo Pan, Chen Ling, and Liang Zhao. GRAG: graph retrieval-augmented generation. CoRR, fbs/2405.16506 2024. doi: 10.48550/ARXIV.2405. 16506. URL https: /tw. /tttp::s///o..o//11/..o
aa..///////iv///a////aa/ 2440..65000 16566.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https: h/tp::////ppeerveee. pet/foounm V=FFFF 8yNQM66.
Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Zheng Li, Ruirui Li, Xi- anfeng Tang, Suhang Wang, Yu Meng, and Jiawei Han. Graph chain-of-thought: Augmenting large language models by reasoning on graphs. In Lun-Wa Ku, Andre Martins, and Vivek Sriku- mar (eds.). Findings of the Association for Computational Linguistics ACL 2024, pp. 163- 184, Bangkok. Thailand and virtual meeting, August 2024. Association for Computational Linguis- tics. doi: 10.1 11653811122224 1 findings-acl.11. URL https: http://aaaaaaataaaeeeca wontheoogg..oo org/2024.
findi winning-aee al..acl... 11.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yib, Tim Rocktäschel, et al. Retrieval-auggmen genera- tion for mmowledge-intensvvy nlp tasks. Advances in Neural Information Processing Systems, 33: 1459-9474, 2020.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozh- skii, Terry Yue Zhuo Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee,
12

---

Published as a conference paper at ICLR 2025
Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin. Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy V. Jason T. Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yaeine Jernite, Carlos Muñoz Fer- randis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! Trans. Mach. Learn. Res., 2023, 2023a. URL https://openrewe neeteeewweeeeeeeeeeeeeeeeeeeeeee www.foormmfioo dddooooogd--Ko 4haE.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprini arXiv:2305. 06/6/, 2023b.
Ming Liang, Xiaoheng Xie, Gehao Zhang, Xunjin Zheng, Peng Di, Hongwei Chen, Chengpeng Wang, Gang Fan, et al. Repofuse: Repository-level code completion with fused dual context. arXiv preprint arXiv:2402.1 14323, 2024.
Xiangyan Liu, Bo Lan, Zhiyuan Hu, Yang Liu, Zhicheng Zhang, Wenmeng Zhou, Fei Wang, and Michael Shieh. Codexgraph: Bridging large language models and code repositories via code graph databases. arXiv preprint arXiv:2408.00 003100000, 2024.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. Codexglue: A machine learning benchmark dataset for code understanding and generation. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Bench- marks I, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL htttss://attaat /dtaasttssssssss.e ceesss-ennchmaraeeee hash/c1 1aaah/cc6655200a1555555933333333
-eeee-eeesaaaaa--ereeeedeee peeessppoooeeddneeeeeeee ier.rineriis. cc ttt.cc//ppppeeee paper/2021, wwweeeftttttttttttttaaaaa- roundl. html.
Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. How to under- stand whole software reposttory? arXiv preprint arXiv:2406.01422 2024.
OpenAl. technical report. CoRR, abs/2303.087774 2023 doi: 1048550ARRXIV.303.00777 htt.::ppe//oolllooooo.....6
Siru Ouyang, Shuohang Wang. Yang Liu, Ming Zhong. Yizhu Jiao, Dan Iter, Reid Pryzant, Chen- guang Zhu, Heng Ji, and Jiawei Han. The shifted and the overlooked: A task-oriented investiga- tion of user-GPT interactions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2375-2393, Singapore, December 2023. Association for Computational Linguistics. doi: 11216633/112223..3 emnlp- main. 146. URL https: : / att/aaaaaanttooooo
/rg/2023. wwwle-mmmmmme main. 146.
Siru Ouyang, Zhuosheng Zhang, Bing Yan, Xuan Liu, Yejin Choi, Jiawei Han, and Lianhui Qin. Structured chemistry reasoning with large language models. In Forty-first International Con- ference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
L Page. The page-rank citation ranking: Bringing order to the web, 1999.
Huy N Phan, Hoang N Phan, Tien N Nguyen, and Nghi DQ Bui. Repohyper: Better context retrieval is all you need for repository-level code completion. arXiv preprint arXiv:2403.06000 060900 2024.
Baptiste Rozière, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Had- sell, Stari--Florina Balcan, and Hsuan-Tiee Lin (eds.), Advances in Neural Information Process- ing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: tttptttpppppp/oeeeeeeeeooo /ttp///eeceeeeccpeeeeepppee neurips.co/ paper/2020/ waahhhaaee e22.3 fbf f8 (od.aa5558887ee44444444eeeee hhtm. html.
13

---

Published as a conference paper at ICLR 2025
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llaaa: Open foundation models for code. CoRR, abs/2308. 12950, 2023. doi: 10448350/ARXIV23300. 12950. URL 12950.
Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. Repository-level prompt generation for large language models of code. In International Conference on Machine Learning, pp. 31693- 31715. PMLR, 2023.
Xiangru Tang, Tianyu Hu, Muyang Ye, Yanjun Shao, Xunjian Yin, Siru Ouyang, Wangchunshu Zhou, Pan Lu, Zhuosheng Zhang, Yilun Zhao, Arman Cohan, and Mark Gerstein. Chemagent: Self-updating library in large language models improves chemical reasoning. arXiv preprint arXiv: 2501. 06590, 2025.
Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. An empirical study on learning bug-fixing patches in the wild via neural machine translation. ACM Transactions on Software Engineering and Methodology (TOSEM), 28(4):1-29, 2019.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Opendevin: An open platform for ai software developers generalist agents. arXiv preprint (idi:2207..6744: 2024.
Di Wu, Wasi Uddin Ahmad, Dejiao Zhang, Murali Krishna Ramanathan, and Xiaofei Ma. Repoformer: Selective retrieval for repository-level code completion. arXiv preprint arXiv:2403.10059, 2024.
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: llm-based software engineering agents. arXiv preprint, 2024.
Demystifying
Zilin Xiao, Ming Gong, Jie Wu, Xingyao Zhang, Linjun Shou, and Daxin Jiang. Instructed language models with retrievers are powerful entity linkers. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing. pp. 2267-2282, Singapore, December 2023. Association for Computational Linguistics.
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprini arXiv:2405. 15793, 2024.
Ziyu Yao, Jayavardhan Reddy Peddamail, and Huan Sun. Coacor: Code annotation for code retrieval 2019.
with reinforcement learning. In The world wide web conference, pp.
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. Repocoder: Repository-level code completion through iterative retrieval and generation. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: /ttp://opppeeiiieeeee net/ forummid=quu wwt.::iiiigooiiiieeeee
Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement, 2024.
14

---

Published as a conference paper at ICLR 2025
Contents of Appendix
A Detailed implementations for each baseline method
16
A.1 Procedural frameworks
16
A.2 Agent frameworks..
16
B Detailed implementations for REPOGRAPH
18
B.1 REPOGRAPH variant
18
B.2 REPOGRAPII components
18
Additional Results
18
C.I Results other LLMs
18
C.2 Results on localization for both pass and fail cases
19 ................................
C,3 Resolve rate by repository
20
C,4 Resolve rate by time
20
D Examples for error analyses
21
Limitations and Future Work
Impact Statement
22
15

---

Published as a conference paper at ICLR 2025
A DETAILED IMPLEMENTATIONS FOR EACH BASELINE METHOD
This section details the implementation of all four methods in procedural and agent research lines mentioned in Section 4.1.
A.1 PROCEDURAL FRAMEWORKS
Figure and Figure 5 illustrate the instructions we used for procedural frameworks, including local- ization and edition, respectively.
Instruction
Please review the following GitHub problem description and relevant files, and provide a set locations that need to be edited to fix the issue. You will also be given list of function/class dependencies to help you understand how functions/classes in relevant files fit into the rest of the codebase. The locations can be specified as class names, function or method names, or exact line numbers that require madification.
### GitHub Problem Description #### groblem_statement
### Related Files ### (filecontents)
Template
### Function/Class Dependencies ###
grap_)
###
Please provide the class name, function or method name, or the exact line numbers that need to be edited.
#### Examples:
Demonstrations
full pallppaa///file..py
full_path3/fiile..p function: my_function 24
function: MyClas2.my.mmethod
line: 10
line:
class: MyClass1
line: 51
Return just the location(s)
Figure 4: Instructions used in the procedural framework to localize to detailed files and code lines of edition.
We flattened the context of the retrieved ego-graph from REPOGRAPH into the template of the instructions. Specifically, in both "localization" and "edition" stages, REPOGRAPH is flattened in the part of Function/Class Dependencies so that the LLMs could better understand its context.
A.2 AGENT FRAMEWORKS
We list all the instructions in every step of agent frameworks. The overall and system instruction is shown in Figure 6.
The system instruction defines the task setting and the template for each response. We add "search repograph" as new action for the agent to use in the command docs, with its signature listed Figure 7.
We also plot the frequency for action invoked for both SWE-agent and SWE-agent with REPO- GRAPH Figure 8. We can see that with REPOGRAPI, the maximum turn of finishing the task is reduced from 38 to 35. Also, we computed the average turn to finish the task, which demonstrates similar trend of 21.47 to 19.12, a significant improvement in efficiency while maintaining effective- We also observe that the action "search repooraheppograhh" is invoked mostly in the first 15 rounds of conversation with LLMs. It is more precise than the original action of "search dir". "search file", and "find file".
16

---

Published as a conference paper at ICLR 2025
We currently solving the following issue within repository. Here the issue text: pproblem_statemenn
Instruction
Below code segments, each from relevant file. One of these files may contain bugs. (content)
help you better understand the contexts of the code segments, provide set of dependencies the code segments. The dependencies reflect how the functions/classes the code segments referenced the codebase. (dependencies)
Ploase first localize the bug based the issue statement, and then generate "SEARCH/REPLACE+ edits fix the issue.
Template
Every *SEARCH/REPLCCE" edit must this format:
The file path
The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines search for the existing source code
The dividing line:
The lines replace into the code
The end of the replace block: >>>>>>> REPLACE
Here is an example:
Demonstrations
### mathweb//lask/app.ppp
Please note that the "SEARCH/REPLACE edit REQUIRES PROPER INDENTATION. you would like to add the line print[x', you must fully write that out, with all those spaces before the code!
<<<<<<< SEARCH
flask import Flask
import math
Wrap the "SEARCH/REPLACE" edit blocks "python...""
from flask import Flask
>>>>>>> REPLACE
Figure 5: Instruction used for fixing an issue based on the identified locations in certain template.
SETTING:
autoromous programmar, and you 're working directly command special The special interface consists of file editor that shows you (WINDOW) lines of file at time. addition typical bash commands, you can also use following commands help you navigate files.
COMMANDS:
(command docs)
Please note that THE EDIT COMMAND REQUIRES PROPER INDENTATION.
you'd like add lina" printtx) you write that with all those spaces before the codal Indentation important and code that is not indented correctly will fail and requira fixing before it can be run.
RESPONSE FORMAT:
Your shell prompt formatted fallaws: (Open file: <path>; <owd>
You need format output using fields; discussion and command.
Your output should always includo _ene_ discussion and _one__ command Field EXACTLY as fallowing example:
DISCUSSION
First by using what files the directory. Then maybe wa look relevant see they look like.
Is
should only include *SINGLE* command command saciion
wait rasponse
shell continuing
with more discussion and commands, Everything include in the DISCUSSION section will be saved for future reference.
you'd like issue two commends at once, PLEASE NOT DO THAT! Please insteed first submit just first cammand, and the second commanc.
then after receiving response you'll be able You're re free use eny other bas commands you want (e.g. find, grep, cat. Is, odi in addition to the special commands listed above. However, the environment does NOT support interactive session commands (e.g. python, vim), please do invoke them.
Figure 6: The signature of new action "search repograph" for agent frameworks.
17

---

Published as a conference paper at ICLR 2025
search_repose repograph
docstring: searches in the current repository with specific function class, returns def relations for search term. signature; searc"_repo <seerch_term>
arguments: search__tee (string) |required]; function class look for in the repository.
for agent frameworks. Figure 7: The signature of our new action
300
300
75
75
33
SWE-SWWEm
RepoGraph
findi File
search
open pylest
edit
find
python
search_ File
scroll clown
goto
pyint
scrol
submit
exit_coo cost
search repograph
Figure 8: The frequency with which actions are invoked at each turn by (a) SWE-agent and (b)
SWE-agent w/R m/RREOGRRPHH
B DETAILED IMPLEMENTATIONS FOR REPOGRAPH
B. REPOGRAPH VARIANT
In this section, we provide the implementations for variants of REPOGRAPH mentioned in Sec- tion 5.2.
In addition to directly flattening the retrieved ego-graph, we propose leveraging large language mod- els (LLMs) to first summarize the context. The full prompt, along with sample input and output, is provided in Figure 9.
B.2 REPOGRAPH COMPONENTS
We first provide simplified version of Figure in Figure 10, which helps readers quickly catch the overall pipeline of integration with REPOGRAPH.
We also provide an illustration Figure to explain in detail how we exclude project-indeppndent dependencies mentioned in Section 3.1 Step 2.
C ADDITIONAL RESULTS
C.1 RESULTS ON OTHER LLMS
We conduct additional experiments on SWE-Bench-Lite with Claude-3.5-Sonee, and the results are shown in Table 6. We found that with Claude-3.5 3.tnone,,,,,,n,,,,,,, the results of baselines are improved little. This is intuitive as Claude-3.5- grating REPOGRAPH with the baseline methods can still boost the performance by large margin, which further prove the effectiveness and extendability of REPOGRAPH on different foundational models.
is designed for better coding tasks. Additionally. inte-
18

---

Published as a conference paper at ICLR 2025
You will given of code lines and their data, please summarbze them the salient function/classes and the dependiencies, including invocation and intreritariee.
Instruction
For each instance, please keep the following cutput format:
ss Key Classes and Functions:*
1. Class/Function name]:
[Dascription of the class/function]
MCroee Deppendencies:""
[Clnse/Function name]:
[Description of the dependency)
location: astropy/madeling/coo ey lines 1765 1865 name: render
contents: inclass Modellmetaclase ModelMetta)nn renderfzell cut-None, coards None).'n at fixed positions, respecting the 'bounding
Evaluate a model The key difference relstive to avaluating the mocel directly is thatkn this mathod lim ted to hounding hox the hoddehounding hoxin optionalto
box.nnnn bouuddng_boxx)ie
attribute set.Ankn
Paramatensin out numpy.ndaray,
location: phropy/modeliggcore. py lines 4537 - 4633 name: render_modee
contents: vndel render modellmodel, arr-None, Nooejk-NonejAn V"V""'v bounding the Modelbouudmng box attribute set.inin Mociolin Modal to be evalusted.In arr sumpy.ndarray.. optionalin mapping to such tharvn
Evaluates model an input arrey. Evaluation Parameterein
limited
model:
Input
Array on which the modal evalusted coords:
array-ike, optionaltn Coordinata
arrcoordss -- arr.nnnn Returns....
location: attroylnnddsta(util..y incless Cutout2D:n def init I'n
name: _iii__
contents:
sell, data, position, size, wor-Norie, mode=""uint", fill valuemmp.rren getattridat,, 1 w'sss" Nonejnnnn i lsinstancelpasition, raise ValuaEmont "wes must be Input position SeyCoord""")
colynFalze'n Win if Noneen
SkyCoord]:nn five Nonartn
position
skycoord__s to sivel(position wos, mode-V"ally # (s. s/lvinn
up in
size np.reppeilpiie 2)
location: hirooy/modeling/soee
lines 3672 3986 name: render
contents: 'mccaas aompoonddddeelooeeeeee (yin def renderiself, out None, coords-Nenelinn \"""\"mn
Evaluete model at this
fixed positions, respecting the bounding box.nnin The koy diffarance ro biative evaluating the model directly thatin mothod
to bounding
Model bounding boxin
set...
Key Classes and Functions:
Model [Class):
Location: estropy/modeling/ee py (lie 1765-1365)
Function: This the base class for models in the astropy library. It containe rendel method, which evalustes the modell
fixed positions while respecting the bounding boxaatribbue. This methed ensures that the evalustion defined region,
outout arrays. Invocation: The Model class uses a custom mataclass Mode Meta The render method works with numpy arrays and he per mechods like extract arey and add arey for array puletion.
support vanoos inputs like coordinates
Cutour2D (Class):
Location: astopyy/ndata//ule
(ine 546-554)
cutout from larger 2D array (lie image) centered around specifi position. init mathod handles coordinate trandformations, shape adjustments, and manages Coordinate System (WCS) integration for astronomical deta.
Function: util ty class that
Output
Invocation: .....
render__madl (Function):
Core Dependencies:
extract_array
Extracts portion of 2D 3D array based on the proviced coordinates and shape, a allowing model evaluations occur withir specific subo-regions, used by Modelrenness Compounddoodelrendd, render, model.
skycoordi_to_ pope
Astronomica cal uttlities for converting between colestial coordinatas and pixel positions, used in Cutous222
WCS:
Figure Instruction used for summarizing the flattened ego-graph.
Methodology Overview
Code fine Parsing
RepoGraph
Agont Frameworks
Code Rapo
2. Projact-depanduen
Issue Description
Genarated Patch
refation filtering
3. Graph organization
Procedural Frameworks
Figure 10: A conceptual overview of our proposed REPOGRAPH
C.2 RESULTS ON LOCALIZATION FOR BOTH PASS AND FAIL CASES
In order to highlight the improvement in error localization and correction capability of REPOGRAPII, we provide a thorough analysis of both pass and fail cases. Results are shown in Table 7. Interest- ingly, we found that the localization accuracy for failure cases is largely lower than the overall average results. This indicates that REPOGRAPH fixes the issues by correctly identifying the editing locations, without which the issue is unlikely be resolved.
19

---

Published as a conference paper at ICLR 2025
Globel reltiomn
Local mnlationn::
import nampy AP
+ Imports
Pre-delined
pp puamme ope
import operator
asnooyyttts Import Maguhit, Quansity
Projech-eppeneet relation Sitering
Saction 3.1, Step
Class ModelBounding@ox:
def Jeit D::
Main Body
ModelBoundinggoxx
supert. intt___ inoo__ Ignored, order) interval =Q
Supart Amodel, sanool-inooodddee
if intenaais in 0 and intervals to
self velivoeeiefinonnlll order-order def booding boxD:
Intorval in
solf valicaaotnttaasss orderworder)
if intirvlf_ttovval
bouding, boxt!
return tuple/exxtthrrfefff.. htnpr//trrrotii_nnnaaavvvvveeeee inteaval.vvvlleee
Aionjjalt intenvali
also:
order self, self_ ord
inputs salf mods.
ordie BWV.-gWC orderforde))
Yrrder
inpuus model.npus
inputs inpuca(.-
bbox aoreaaaaaaaa if lunfbbod -- bbox bbox(0)
www.proofuuiideeallttt..ommmmm for input in inpusa)
wwwp-/-ooo------eeeeeeeee wwwp:///---eeeeooeee vitvdd
Figure 11: A detailed illustration of how we filter the project-dependent relations given code snippet. Code line colors are illustrated with respect to global and local relations.
Table 6: Results REPOGRAPH on SWE-Bench-Lic test set with Claude-3, Sruude-3.5-Somee ---Sonnet two re- search lines, including procedural and agent frameworks, We also report the corresponding cost of each method.
Accuracy samples
Avg. Cost tokens
Methods
LLM
resolve
patch apply
$ cost
Procedural framwworks
Agentless
GPT-4o
27.33
82
97.33
S0.34
42,376
+REPOGRAPU
GPT-4o
29.67+2.34
89
98.00+0.67
$0.39
47,323
Agentless
Claude-3.5 Sonnett
27.67
94.33
$0.28
40,984
+REPOGRAPH
30.33
91
98.67
$ 0.32
46,238
Agent frameworks
SWE-agent
GPT-4o
18.33
55
87.00
$2.53
498,346
-REPoGRAPH
GPT-4o
12.00
66
13.33
$2.69
518.792
SWE-agent
Claude-3.5- 3o5noneet
23.00
69
86.67
$1.62
521,208
RREPOGRAPH
Claude-3.5- 3.nnet
25.33
76117
90.67
$ 1.70
539,942
C.3 RESOLVE RATE REPOSITORY
We plot the results of the resolve rate in terms of repository distribution in Figure 12. From the fig- ure, it is clear that the resolution of issues varies significantly across different repositories. Notably, the Django and Sympy repositories have the most unresolved issues, with 75 and 61 unresolved issues, respectively. This may indicate a higher level of complexity in the issues or larger backlog compared to the other repositories. On the other hand, Django has the highest number of resolved issues, with 39 cases. This highlights a strong effort to address issues, even though the unresolved count is still high. Sympy follows closely with 16 resolved issues, suggesting similar trend. Other repositories like Scikit-learn, Sphinx, and Matplotlib have comparatively fewer issues overall, but their resolve rates are more balanced. For instance, Sphinx shows ratio of 13 resolved to 3 unre- solved issues, reflecting more consistent effort in issue resolution.
C.4 RESOLVE RATE BY TIME
We plot the results of the resolve rate in terms of the distribution of releasing time for repositories in Figure 13. Most of the issues were observed in recent years, starting from 2018, with a substantial increase in the total number of issues identified after 2018. In the early years (2012-2016), the number of issues remained relatively low, with both resolved and unresolved counts being minimal. Starting from 2017, there is noticeable increase unresolved issues, with 13 unresolved and only 3 resolved issues. In 2019, 19, the number of resolved and unresolved issues significantly increased,
20

---

Published as a conference paper at ICLR 2025
Table 7: Percentage of problems for accurate edition localizations for Agentess+RRPPGGRAPP with respect to file, function, and line levels. All the numbers are computed from the corresponding generated patches. We also report the accuracy for all pass and failure cases, respectively.
Methods
LLM
file
function
line
Agentless REPOGRAPH
GPT-4o
74.3
54.0
36.7
Pass cases
GPT-4o
91.0
79.8
65.2
Failure cases
GPT--4o
67.3
43.1
24.6
120
Resolved
Unresolved
Number of issues
60
18
django
pylint
scikit-laarn
sympy
flask
metplatlib
xarray
seabom
pytest
requestt
astropy
Figure 12: Distribution issues resolved by Agentless +REPOGRAPH plotted in terms of different repositories.
with 19 resolved out of 59 issues. This trend continued to rise until 2020, where 47 issues remained unresolved, and only 17 were resolved, marking the year with the highest number of unresolved issues in the dataset. By 2021 and 2022, the number of unresolved issues slightly decreased, while the resolve rate increased compared to 2020. This suggests an improvement in the system's ability address issues in these years. 2023, although the total number of issues dropped to the proportion of resolved issues remained strong, with 8 resolved out of 22 issues.
D EXAMPLES FOR ERROR ANALYSES
To help better understand the error category listed in Section 5.4, we provide one example for each category Figure 14, Figure 15, and Figure 16.
E LIMITATIONS AND FUTURE WORK
(i We only explored proprictary LLMs, i.c., GPT-4 series. Due to the poor performance of open- source models on this challenging task, we opted for proprictary models that have demonstrated superior results in code-related tasks. However, comprehensive evaluation of open-source models such as Llama (Dubey et al., 2024) could be valuable direction for future work, particularly as these models continue to improve.
(ii) Experiments were only conducted on the Lite set due to the high cost of running large-scale experiments with proprietary models. Exploring more efficient model deployment strategies and alternative cost-effective options for running experiments on larger datasets will be essential for broader applicability.
(iii) Although REPOGRAPI could be adapted to support other programming languages by adjusting the parsing schemes in the implementation, we only explored Python in our main experiments. Future work could extend this approach to other widely used programming languages, such as JavaScript, Java, or C++, to evaluate the generalizability our methodology across different pro- gramming paradigms.
21

---

Published as a conference paper at ICLR 2025
Resolved
Unresolved
56
Number of Issues
42
20
28
19
17
20
2012
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
Figure 13: Distribution of issues resolved by Agentless +REPOGRRPPH plotted terms of release time for repositories.
Correct expected format invalid FurationFiels arror message
Description
If you enter cluration "14:00" duration field, it transiates to "00:14:00" which minutes. The current message invalidl DurationField says that actual behaviour, should be: "[DD] [[HH:MM]]ess [es optional
format durations: *[DD] [HH[MM]]js]( uouuad) ouoouu"" because seconds are mendstory, minutes are optional, and provided. This Also the duration fields could have a default help_text text with the requested format, because the syntax is not zelf-expl gelf-xxplnaaeee anatoory
accorclng
Issue
mistake Django versions that suppori the Durationtield.
diff -git hefjang//tomm//illddd/.. py bidjanggoffomm//illd...
diff -git al//pnng//t/mmmddeeiieeaee _iniid/___ii__....p brdjengo/ds/moddls//ie jint_yy
index 9350Me..... bbb33d2 100644 a/djanga/Rorms/helca +++ rrdjango/fomm//ildss
77ba99.. Troa9Pe.de7d04 100644
a/django/db/ s/fielcss// /tcee//////eeeeecccc//////////// init
29 468,9 -468,
class DateTiimaFFoed@BBaaeTTmmooorr
+++ b/django/daa/ hww.gg//////mmmeeeee/ooo http///mll///llsss/// Siaald//_iin______ _ini___.yy
09 1587,7 +1587,7 80 class Duration
class DurationField(Fildd;
wtringg.tinnss... _alowaaloddddddoo False
default_error_ souut__aror__mmsaaee mesoamessagsss...
default_ wirkkkerrrr__mem arnorrmessagessss {
Inwaldd: ['Enter a valid duration:",,
'invalid': [" "Kyaauusss value has an invalid format.
('Entr valid duration in the format (DD)
must be
[HH]]MM]j][ (usuur)]
[[D] [HH[MM [|sa ummenn) formar.
'overflow': : __Thhe number of days must be between
[DD] [HH+]]MM mumun format.)
(min days) and (max, days).)
description _! Duration")
default help text ('Enter duration in the format [DD]
[HHH]MMM]jd [sd mummua)))
de' prepara valuefself, value):
tisinstancovalue, cenattme.ttimddeee
(a) RepoGraph
(b) Groundtruth
dumping (modll durationi TestSarillzation)
test loading (model. fielkk curstionnfeldd TestSerializatio))
test formfield (model, fie ds dest, duraticnfiole, Tessf Ter.eelle.eoofoooiiiieeioeo ornFieeld
v
test asact (modal Realsstest durationf eld. TeutCherying)
/
irvalld strrng y "mode: danattii TextVal detion)
Tests
gi fields durationfield TerrCuerying)
lest_create_empty Imodel_flekee _ieldddddsss dest_dduatiooneeeld (estSaveLoac) http:dddurttooeeeedeeeeeeeeedodd test, seconds seconds (mocial, fields test, durotionfiakd TesdSaveLcad
test, simple rounctrio (model Relds test duretionfield TestSevelooc)
Figure 14: An example of incorrect localization. The correct patch modifies the "DurationField" in httpo//b/mmodel//eed/...ii init...py. This is the correct place to handle the error message formatting for the "DurationField" used in Django models. REPOGRAPH, however, modifies django/forms/fieldsspp This file handles Django form fields, not the model fields. While both model and form fields have overlapping behavior, in this case, the correction required for the model field (DurationField in py), not the form field.
F IMPACT STATEMENT
The impact of this paper lies in its substantial contribution to enhancing the capability of AI-driven software engineering, particularly with respect repository-level code understanding. The introduc-
22

---

Published as a conference paper at ICLR 2025
Trailing whitospace DatasatGroupBly taxt represantation
When displaying a DatasetGroupBy interactive Python session, the first cutput contains trailing whitespace. first example in the documentztion demonstrate this:
import karray 2s x;, numpy as mp >00 de ar. Datasesl
(Moco ": 01'x'. "y"), np..andom.rend.
coords-("x"-1100 20, 30, 40), "lettars"; "nttars"("""" ("x', >a> di.groupbyl "letters"]
laat"aaa list"aabba
DatasettSroupBy, groupad 'lotters' 2 groups with labels 'el, 'b..
Thara trailing whitespace the first line output which "DatasetGroupBy, grouped 'lettars' This clearly by converting the object to a string (note the whitespace before 'In':
strich groupby!" letters") ')
Issue
DatasetGrouppy grouped over lletters \n2 groups with labels 'at, 'b'.'
While
problem itself, causes an issue us bocause we use correctly formatted and wa also have doctasts that rely on DetasatGrouuByy textual represantationn Flake8 raports violation docstrings If we remove the trailing wh tespeces, our doctests fail becouse the expected
continuous integration to verily our
the treiling whitessacces output doesn't match actual output, So vi have conflieting constraints coming from our which both seam reasenable. Trailing whitespaces are forbidden by ilake8 because, smong other reasons, they lead to no sy git difls. Doctest want the expected outout
exactly the We could configure flaka8 to ignore particular anion for files which wa hava these doctests, but may cause other trailing whitespaces to cresp our code, which we don't want. Unfortunately it's not possible to just add 't NoQA commants get flake8 Iggnore the violation only for specific lina because that sctual output from doctest point view. Flake6 doesn't allow to disable checks for blacks of code either.
the actual output and considers trailing whitespace to
gnificant difference.
diffarence between expected and
there reeson for having
riniin whitespece DatasetGroupBy representation? Whould be to remove so
please let me know and can make pull request.
diff -git w/karey/core/gro.ppei. py b/karray/core//eee //varsy/core/groppooyyy index b163803. 7/9s25e 100644
diff -giit a/sarraylore//rruppyyy py bikarnyloore/rroppyyyy index b463803 9aac7fc 100644
a/varry//ooeirruubbyyyu +++ Exkaraylcore/rroupkyy
atkaraay/ore//rroooyyyy. +++ b/kartsy/core//reoo
by
ee 1011,5 1011 as class DataetGrroupB@GGrroppyy
Go 436,7 +436,7 GB class Grous@yySupporttArmmmeee
ImplamanssDasseeRddiuc
return ziptse f_unique__cord seif_iter__geeee _errugpgggg
return sell limppp/mbaa de: ds "kwerrrr
def rapr (self):
+dof rapp...pp..eee [salf):
return 'B. grouped over (r) Infh groups with labels
Original representation with trailing whitespace removed
0. format(
groups ", " "joinlstrkd for in self grouss)
return '0. grouped over (HAndD groups with lebels
return "DetosetGroupBy grouped over
0. forrati
_eelf__rroup.n www.grrupp.mmmmm. grcuu.nameli//jnneeeli groupal) groups with labels
salf. class name
(groupsl'
_ell_unniqueec coore
ops pps.injcctreduue cerettrrdu____eeeeeeee Dethsodd/eettsttrrrruuuy.y
self. jttb_uuiqqe__cord..co
ops inject binary ops(DataaenrrouuupB)
injetject binery, spei/atteettrroupp))
(a) RepoGraph
(b) Groundtruth
sarray/estt/eest_ggroaa gycposggrroouooyrogroyy py:test wtt.mtsstmmii_jimmm.. wttp:/muut_iideeoeoo_eeeee groupbbborupyuuuummmm
hamsy..essshestt.. grw.phyyyyyyyy.m...m..m... test grw.pgggpyyyy...oommmmmmm map, shrink www.sshi.gggrrieee groupstobjjj
map_shrink
Ea
xar serrayyeest/eest_ggooo gro.pby.yyyyy.......... wwwpgdd_bggygoooooeo hap.yyymaauuaaaa.a func_args
sammay/esststtttt penett groupby reprobb[0-0)
Tests
xarmyy/tssts/test groupby groupby
beckppbyyyyyruooeeee reck ce. ce_dimension error
xai wtwp://eee/ttsttooooorryo///eeee http:/upbbbyppppptteeeeeee
wrreyyteetiieersrrrouoopppppsspe grwuppy..ooooooooogg.go gettem
An example of contextual misalignment. The model-generated generated patch places the rererr... method in the wrong class DatasetGroupBy, whereas the correct patch modifies it in the GroupBy class. The -repr_. method should be implemented in the GroupBy class because it deals with the general group-by functionality. "DatasetGroupBy" is a subclass and doesn't 't require a new represen- tation method if "GroupBy" already has one. Additionally, when fixing white space trailing, while functionally similar, REPOGRAPH's generation doesn't 't exactly match the style or intention of the correct patch.
Figure 15:
tion of REPOGRAPH not only significantly improves Large Language Models (LLMs) in navigating and comprehending entire codebases. but also showcases the potential of integrating repository -wide structures into Al workflows. By extending the scope from function-level tasks to holistic repository management, REPOGRAPH pushes the boundaries of Al's utility in modern software engineering.
23

---

Published as a conference paper at ICLR 2025
expr anprratomo) should return objects with no angs instead subclesses of Atom expr "xprratomet" with arguments returns subclesses of 'Attm in expr. But the correct definition of leaf node should be that .arge.
Issue
This should easy fix, but needs check that this doesn't affect the performance.
-giit a/spmpy/ccoe//seeicc by bykymppy/crr//bbsiccpp
diff -gitt a/smmpy/ wsymmp//cor//bess/i/ core/besiccccceoo bessicooyy
b/sympy//cre//besiccpp
index 97eböta Alazfze 100644
97el01a f1 100644
alsyppyloor/bbasi.py b/symoyicon//basic
brsymoy/oonerbaaiccppp
340 500,15 -500,11 @0 class Resdsneettdaaee--aaaaaaPPPeeeeee
-3077 +507,7 Basiclmetaclass Badedmaaclass-naaggdPPoo Manasaaooooooaaaaaiass types [Atom.]
MM,,, 2"sinly !pill
types:
Types tuplo!
reult - sent)
isinstancces type) typeit) types])
for expr preeoderr_tweerss/// traverseljplll)
nooo types:
isinstancee/xpp, types):
result lexpr for preorder_ traversallee) if exprarggd
tinstanceeeepp,, sypes and not expr.args:
else:
result aduleaa
types (Atorn.)
return result
setū
expr prearder, traversallsellf:
if Iisinstanneeeexp,, types):
aediett
types - tuploft if linstancalt, type) eise typeft for in types)
result texpr
presrder_traveessaee traversalbei) is nstance/axprr
types]]
retum result
RepoGraph
(b) Groundtruth
tas" literall evalf. number, is zem, is companable
tast_subs_ sanicodd_yybblle
Bott_fre__ymbool_gegg
test, wwtttppreeeerrreneee traversal
V
Test acomac
x
Tests
Los__canonicllvaree varisbles
test_siteof
CommmaDporator
Figure 16: An example of regressive fix. The model- patch successfully resolves the issue which requires the return objects with no args instead subclasses Atom by adding the key code line if not exprargs. However, the fix introduces other new issues such the size of return objects, as exemplified in the unit tests.
This advancement opens new opportunities for using LLMs in complex engineering tasks such as automated debugging, repository maintenance, and large-scale refactoring. Purthermore, by high- lighting the importance of repository-level context for accurate code generation and maintenance, the paper sets a new trajectory for future research Al and software engineering. It encourages deeper exploration of Al's ability not only write code but also understand and manage large-scale software projects more efficiently. We foresee minimal risks or negative societal impacts from this work. All datasets and benchmarks used in the evaluation are publicly available, and we adhered to their respective licenses. Additionally, REPOGRAPH has been open-sourced, making it accessi- ble to the research community, particularly groups with limited access to extensive computing resources, thus fostering broader adoption and further development.
24


Reviews:
[{"review_id": "JdWiA4T7Oo", "reviewer": "Reviewer_cQhV", "summary": "This paper proposes RepoGraph, a plug-in module designed to extract repository-level definition and invocation information, offering graphical navigation to improve AI software engineering. RepoGraph is compatible with popular agent-based and procedural based LLM frameworks. The authors conduct experiments with GPT-4 and GPT-4o on SWE-bench, yielding a promising result.", "questions": "The authors state that in the error analysis 'when integrated with REPOGRAPH, the proportion of error types “incorrect localization” and “contextual misalignment” largely decreases'. However, this conclusion appears limited in significance due to the small number of passed samples involved. Could the authors provide further clarification on how this conclusion was reached? Have the authors considered the implications of having too few passed samples in their analysis?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- RepoGraph can be easily built and integrated into existing methods and its construction process is straightforward and well-defined.\n  \n- The analysis presented in this paper is clear and detailed. The authors offering several key observations and insightful explanations, which could be inspiring for future research and broader adoption.\n  \n- The author conducted additional transferibility experiments to better explain the effectiveness of RepoGraph.\n  \n- The authors give comparative analysis between procedural and agent-based methods.\n  \n- A thorough classification analysis of error cases is provided, with specific examples for each category.", "weaknesses": "- The primary experimental results are presented solely on SWE-bench-Lite, and the improvement in accuracy resolution are not particularly significant. Additionally, the paper mainly focuses on the resolved numbers and accuracy as evaluation criterion. In cases with a limited number of passes, the quantitative comparison and analysis is not convincing enough and may exhibit considerable variance. It would be beneficial for the authors to provide a thorough analysis of both pass and fail cases, as examining failed cases could also highlight improvements in error localization and correction capability of RepoGraph.\n\n- As the authors note, the paper only evaluates based on GPT-4 and GPT-4o. Given the increasing availability of various General LLMs and Code LLMs, such as Claude and other open-source models, it would be better for the authors to include results from these additional LLMs. Previous studies indicate that different LLMs have varying performance when applied to new methods or tasks. It would be valuable to see how RepoGraph performs across a broader range of LLMs.\n\n- A presentation issue in Figure 10: some numbers overlap, and certain white numbers blend into the background.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3mhB4uW8gp", "reviewer": "Reviewer_H4KK", "summary": "This paper presents an approach to navigating software repos and\noffer the desired navigation by incorporating\ninformation at the line level, file level, and repository level. Toward that end, the authors\nproposed a graph-learning-based framework with nodes corresponding to code-line level and edges corresponding\nto connections of the nodes at the repo level.", "questions": "According to Figure 3, RepGraph was not able to find all error cases when added as a plug-in with Agentless and SWE-Agent. \nAs RepoGrpah works at line-level, file-level, and repo-level and was added as a plug-in, I expected RepoGrapgh to cover all error cases of Agentless and SWE-Agent.\nDo the authors have any intuition or hypothesis of the observation of Figure 3?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- A plug-in-based framework to easily integrate with other AI-based tools\n- Code and docker provided for reproducibility", "weaknesses": "- No human evolution studies\n- Missing prior relevant literature\n\nThe authors could verify their assessment on SWE-bench Verified, released as a part of the benchmark.\n\nAlthough the proposed work seems promising, the article lacks relevant literature citations. How to navigate the software repository has been studied by software engineering researchers before the LLM era. The\napproaches include the navigation behavior of developers using eye-tracking data, during code summarization tasks.\nI encourage the authors to consult the works of Bonita Sharif and follow the references.\n\nReferences\n1. Begel, A., Khoo, Y.P. and Zimmermann, T., 2010, May. Codebook: discovering and exploiting relationships in software repositories. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1 (pp. 125-134).\n2. Bairi, R., Sonwane, A., Kanade, A., Iyer, A., Parthasarathy, S., Rajamani, S., Ashok, B. and Shet, S., 2024. Codeplan: Repository-level coding using llms and planning. Proceedings of the ACM on Software Engineering, 1(FSE), pp.675-698.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4VT3u9zWbT", "reviewer": "Reviewer_v4kH", "summary": "This paper introduces RepoGraph, a plug-in designed to enhance the capabilities of LLMs in understanding and navigating code repositories. RepoGraph constructs a graph-based representation of the entire codebase using abstract syntax tree parsing. The system is evaluated using the SWE-bench Lite and CrossCodeEval benchmarks, demonstrating its effectiveness in issue resolution and fault localization. The authors also conduct various analyses, including complementarity and failure cause analysis.", "questions": "- Can the authors provide more information about the computational overhead involved in constructing and using the repository-level graphs? Specifically, how does the time complexity compare to other methods?\n\n- Given the small performance improvements, could the authors discuss potential areas for further optimization or enhancement of RepoGraph that could lead to more significant gains?\n\n- Could the authors offer guidance on the reproducibility, specifically:\n\n    - What are the exact environment requirements for running the tool?\n    - How should the necessary keys for LLM configuration be set up?\n    - What steps should users follow to accurately evaluate the results generated by RepoGraph?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Originality: RepoGraph introduces a novel approach by leveraging repository-level graph-based representations, which provides a more comprehensive understanding of the codebase. \n\n2. Quality: The paper is well-written, with a clear explanation of how RepoGraph work. It provides a robust experimental setup that evaluates the tool's performance across multiple dimensions. \n\n3. Clarity: The distinctions between RepoGraph and other repository understanding techniques are clearly articulated, making it easy to understand the motivation and design of the tool. \n\n4. Significance: Its contributions could lead to improvements in the effectiveness of automated issue resolution and code navigation tools.", "weaknesses": "1. Performance Gains: While RepoGraph shows some improvement in issue resolution and fault localization, the gains are relatively small. \n\n2. Cost Assessment: The paper does not adequately address the potential computational costs associated with graph construction and search operations. A thorough analysis of these costs, including time overhead, would strengthen the paper by providing a better understanding of RepoGraph’s practical feasibility.\n\n3. Reproducibility: The open-source code provided in the repository is not entirely reproducible based on the available documentation. Specific issues include: \n\n\t(1) Environment Setup: The steps for setting up the execution environment are incomplete. It is unclear whether specific configurations or environments are needed for the tool to function properly. \n\n\t(2) Configuration: The LLMs require additional keys for configuration, but the paper lacks guidance on how to set these up.\n\n\t(3) Evaluation Steps: There is no clear documentation on how to evaluate the generated results effectively. Detailed instructions on the \tevaluation process would help ensure that users can reproduce the experimental findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uaTVGgBZL6", "reviewer": "Reviewer_wovV", "summary": "This paper introduces a framework called *RepoGraph* for fine-grained modeling of code repositories to address the current challenges in repository-level code tasks. *RepoGraph* leverages tree-sitter to model code repositories at the level of code behavior, constructing a code structure graph based on the invocation and dependency relationships between functions and classes. By using this graph, *RepoGraph* provides more accurate code context for given retrieval terms, thereby improving the accuracy of code task completion. The authors tested *RepoGraph* on the **SWE-bench** benchmark, demonstrating that it can be seamlessly integrated with both advanced agent-based and non-agent frameworks, significantly enhancing the accuracy of issue-fixing tasks. Furthermore, they tested it on the **CrossCodeEval** benchmark, verifying that *RepoGraph* can also be applied to repository-level code completion tasks.", "questions": "1.\tAlthough the authors mentioned that due to resource limitations, only 500 samples were used for experiments on **CrossCodeEval**, the cost of conducting a full test on **CrossCodeEval** does not seem to be significantly higher compared to testing on **SWE-Bench**. This is because the cross-file context retrieved should not be very long. Could the authors report the costs of each test?\n2.\tWhile GPT-4o is also capable of code completion, **CrossCodeEval** primarily evaluates the completion abilities of base code models. Could the authors include results from some open-source code models, such as `Deepseek-Coder-V2` and `Qwen2.5-Coder`? These models should be relatively easy to run locally for inference.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.\tConducting high-quality, fine-grained modeling of code repositories is extremely challenging due to the numerous corner cases that need to be considered. Implementing such a detailed model for code repositories is no easy task, and I truly appreciate the authors’ efforts.\n2.\t*RepoGraph* can be seamlessly integrated with both agent-based and non-agent frameworks, significantly improving the ability to fix issues on the **SWE-bench**. This will be highly beneficial for future agent-driven code development.\n3.\t*RepoGraph* can also be applied to repository-level code completion tasks, which will be very useful for code completion in daily development.\n4.\tThe authors conducted extensive experiments to validate the effectiveness of *RepoGraph*.\n5.\tCompared to other papers I have reviewed, this paper is well-written, with a clear structure and a balanced level of detail.", "weaknesses": "This paper is well-written and does not have any major weaknesses, but there are a few minor typos:\n\n•\t`Line 228`: There is a repetition of `such as`.\n\n•\t`Line 306`: The sentence `To assess the patch application rate, we attempt to apply the generated patches to the repository using the patch program; successful applications contribute to this metric.` has a somewhat disjointed structure and logic. The semicolon is typically used to connect two independent but related clauses, but in this case, the first part of the sentence is not a complete clause, even though there is a logical connection between the two parts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kbvc0WFo9S", "reviewer": "Reviewer_btxE", "summary": "The authors introduce RepoGraph, a new tool that turns a codebase into a graph structure. The authors demonstrate how this tool, which can be integrated into both agent-ic and procedural AI software engineering frameworks, leads to improvements across the board for existing methods. The repository is constructed by filtering out unnecessary file types and repository-dependent code, identifying object oriented programmatic entities, and constructing the remaining pieces of the codebase into a graph. The experimental setup follows existing work. The authors demonstrate improvements for a number of existing AI SWE solutions when RepoGraph is integrated into them and perform some analyses around RepoGraph parameters, localization statistics, and transferability.", "questions": "- [Introduction, Line 75] - “they tend to focus narrowly on specific files, resulting in local optimums”. I’m not sure what this means? Why do agent approaches have this tendency?\n- [Intro, Line 85] - What is an “ego-graph”? I see the citation in Line 256, but may be good to define this explicitly as a preliminary somewhere earlier.\n- [Section 3.1, Line 205] - how is “non-essential” determined? I could imagine that editing requirements.txt might be necessary to import a new library for carrying out a fix.\n- [Section 3.1, Line 216] - What happens in step 1 if the codebase is not written in an object oriented manner? For instance, what if you’re dealing with JavaScript code primarily written in a functional programming style, which may not have classes? Or if you are editing a bash script which may not have these entities?\n- [Section 3.2, Line 267] - What is an “edition” stage? Perhaps this is a type? (“Editing”?) Repeated again on Line 350\n- [Table 2] - What is # samples?\n- [Table 4] - The number of nodes is a bit confusing. In SWE-bench, each codebase has an average of 438k lines. If each node corresponds to a line, why are there only 1419 nodes on average?\n- [Figure 3] - Is this for SWE-bench Lite? Or CrossCodeEval?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "RepoGraph examines the question of how codebases can be represented effectively. While other works have tried more traditional approaches for localization, such as retrieval and tool-augmented LMs, RepoGraph demonstrates that reshaping the codebase into a graph that is more traversable and useful for a language model can elicit stronger performance on the downstream editing subtask. The authors discuss a pipeline for carrying this out with Python codebases, showing how simple heuristics for removing noise and preprocessing the codebase in a more friendly way can lead to improvements that are method-agnostic. \n\n- [Section 3.1, Step 2] - This step seems quite interesting and novel. I think a visualization, something similar to Figure 2 but a bit more grounded in Step 3’s construction, might be helpful for readers to understand this construction more easily.", "weaknesses": "- Figures 1, 2 - I appreciate the effort that went into these figures, they are clearly drawn quite thoroughly. However, I think it is a bit too complex, making it hard to understand. I’d recommend simplifying the diagrams to make the takeaways a bit more obvious. For example, it took me a while to understand the meaning of the colors in Figure 2, (a).\n- The design of RepoGraph seems to be written in a fairly Python-centric manner. This is understandable to some degree, as the main evaluation (SWE-bench) is for Python based repositories. However, the contribution would be more impactful if somehow it was demonstrated that this approach scales to other kinds of codebases (with existing work or a new benchmark?). In Section 5.3, the authors mention how for CrossCodeEval, the evaluation is limited to Python. The authors discuss this in the appendix, but I think the generalizability of RepoGraph is a question that should be answered instead of being deferred to future work.\n- [Section 3.1, Step 2] - “We maintain a comprehensive list of methods” ⇒ This sounds like it’s hard-coded for Python. Would I have to manually re-perform this filtering for other languages? Is there an automatic filtering criteria I can apply to come up with a comparable list for another language?\n- [Section 5.2, Line 361] - This claim seems somewhat handwavy to me. The cost metric seems to go up with every method due to the inclusion of RepoGraph. However, this takeaway seems to suggest that the performance increase is proportional or lower than the increased cost - why is this true? What numbers are being compared for this proportion? (e.g. 2.66% increase vs. 0.04 cost increase? Why is this lower?)\n- [Section 5.1, Lines 413 - 420] - The discussion on the differences between procedural and agentic approaches is interesting, but I think it’s not quite clear to me why this discussion is included or how it relates to RepoGraph. Is the suggestion that procedural approaches take more advantage of RepoGraph?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)