RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph

## Abstract

Abstract Large Language Models (LLMs) excel in code generation yet struggle with modern AI software engineering tasks.
Unlike traditional function-level or file-level coding tasks, AI software engineering requires not only basic coding proficiency but also advanced skills in managing and interacting with code repositories.
However, existing methods often overlook the need for repository-level code understanding, which is crucial for accurately grasping the broader context and developing effective solutions.
On this basis, we present RepoGraph , a plug-in module that manages a repository-level structure for modern AI software engineering solutions. RepoGraph offers the desired guidance and serves as a repository-wide navigation for AI software engineers.
We evaluate RepoGraph on the SWE-bench by plugging it into four different methods of two lines of approaches, where RepoGraph substantially boosts the performance of all systems, leading to a new state-of-the-art among open-source frameworks.
Our analyses also demonstrate the extensibility and flexibility of RepoGraph by testing on another repo-level coding benchmark, CrossCodeEval.
Our code is available at https://github.com/ozyyshr/RepoGraph .

## 1 Introduction

Recent advancements in large language models (LLMs) have showcased their powerful capabilities across various natural language processing tasks (OpenAI, ; Anil et al., ; Dubey et al., ), and now, coding-specific LLMs are emerging to tackle complex software engineering challenges (Hou et al., ; Fan et al., ), such as Code-Llama (Rozière et al., ) and StarCoder (Li et al., ).
These coding-specific LLMs are capable of assisting users with various software engineering tasks, even achieving human-level performance in many function-level coding tasks, such as program synthesis (Chen et al., ; Austin et al., ), code annotation (Yao et al., ), bug fixing (Tufano et al., ), and code translation (Rozière et al., ).

Real-world software engineering often extends beyond single function or self-contained code files. Applications are typically built as repositories containing multiple interdependent files, modules, and libraries (Bairi et al., ). These complex structures require a holistic understanding of the entire codebase to perform tasks such as code completion (Shrivastava et al., ; Ding et al., ), feature addition (Liang et al., ), or issue resolving (Jimenez et al., ).
Recent benchmarks like SWE-Bench (Jimenez et al., ) have been proposed to evaluate LLMs on real-world GitHub issues. It requires LLMs to modify the repository to resolve the issue, either by fixing a bug or introducing a new feature.
This task is particularly challenging because it requires navigating complex code bases, understanding intricate dependencies between code files, and ensuring that changes integrate seamlessly without introducing new issues, which highlights the difficulties in scaling from function-level to repository-level understanding, as expounded in Figure 1.

Figure: Figure 1: The illustration of (a) a function-level coding problem from HumanEval (Chen et al., ) and (b) a repository-level coding problem from SWE-Bench (Jimenez et al., ).
Refer to caption: x1.png

A key step in addressing repository-level tasks is to understand the structure of a repository and identify related code. To achieve this, retrieval-augmented generation (RAG) and its variants (Zhang et al., ; Phan et al., ; Wu et al., ) have been leveraged, in a procedural manner, to retrieve relevant code files across the repository first, providing context for LLMs for further edition.
However, indexing at file-level can only identify semantically similar but not genuinely related code snippets.
Instead of using RAG, recent approaches like Agentless (Xia et al., ) construct a skeletal format for each file, and directly prompt LLMs to identify relevant files and code lines. However, this method still treats code repositories as flat documents (Zhang et al., ), which suffers from limitations of repository structure such as the intricate inter-dependencies cross files.
An alternative approach is to design agent frameworks (Yang et al., ; Wang et al., ), which enables LLMs to interact with repositories using actions. While LLM agents can freely determine the next action based on current observations, without the grasp of global repository structures, they tend to focus narrowly on specific files, resulting in local optimums.
Addressing these limitations requires going beyond semantic matching and developing techniques that enable a deeper understanding of the codebase structure. This will allow LLMs to leverage fine-grained context across multiple files and function calls, facilitating more informed, repository-wide decision-making for coding tasks.

Motivated by this, we propose RepoGraph, a plug-in module designed to help LLMs-based AI programmers leverage the code structure of an entire repository. RepoGraph is a graph structure and operates at the line level, offering a more fine-grained approach compared to previous file-level browsing methods. Each node in the graph represents a line of code, and edges represent the dependencies of code definitions and references. RepoGraph is constructed via code line parsing and encodes the structured representation of the entire repository. Sub-graph retrieval algorithms are then used to extract ego-graphs centered around specific keywords. These ego-graphs can be smoothly integrated with both procedural and agent frameworks, offering key clues that provide a more comprehensive context for LLMs to solve real-world software engineering problems.

To assess RepoGraph’s effectiveness and versatility as a plug-in module, we integrate it with four existing software engineering frameworks and evaluate its performance using SWE-bench, a recent benchmark for AI software engineering.
Experiment results show that RepoGraph boosts the success rate of existing methods for both agent and procedural frameworks by achieving an average relative improvement of $32.8\%$. We also test RepoGraph on CrossCodeEval to verify its transferibility to general coding tasks that require repository-level code understanding.
Additionally, we systematically analyze different sub-graph retrieval algorithms and integration methods. Together with error analyses, we hope to shed light on future works targeting modern AI software engineering.

## 2 Related Works

### 2.1 LLM-based methods for AI software engineering

Recently, there has been a significant increase in research focused on AI-driven software engineering, which can be broadly categorized into two primary approaches: (i) LLM agent-based frameworks and (ii) SWE-featured procedural frameworks. While this field has advanced rapidly, with most methods being released as proprietary solutions for industry applications (Cognition, ), our related work section will concentrate specifically on open-source frameworks.

LLM agent-based framework equips large language models (LLMs) with a set of predefined tools, allowing agents to iteratively and autonomously perform actions, observe feedback, and plan future steps (Yang et al., ; Zhang et al., ; Wang et al., ; Cognition, ).
While the exact set of tools may vary across different agent frameworks, they typically include capabilities such as opening, writing, or creating files, searching for code lines, running tests, and executing shell commands. To solve a problem, agent-based approaches involve multiple actions, with each subsequent turn depending on the actions taken in previous ones and the feedback received from the environment.
For example, SWE-agent (Yang et al., ) facilitates interactions with the execution environment by designing a special agent-computer interface (ACI). There are various actions, including “search and navigation”, “file viewer and editor”, and “context management”.
Another work, AutoCodeRover (Zhang et al., ), further offers fine-grained searching methods for LLM agents in better contexts without an execution process. Specifically, it supports class and function-level code search. OpenDevin (Wang et al., ), initiated after Devin (Cognition, ), is a community-driven platform that integrates widely used agent systems. The action space design of OpenDevin is highly flexible, requiring LLM agents to generate code on the fly.

SWE-featured procedural frameworks typically follow a two-step Localize/Search-Edit approach, as seen in existing literature (Zhang et al., ; Wu et al., ; Liang et al., ; Xia et al., ). The localize step focuses on identifying relevant code snippets, while the edit step involves completing or revising the code. Some works introduce additional steps to further enhance performance, such as the Search-Expand-Edit approach (Phan et al., ).
Retrieval (Lewis et al., ) is a popular technique used for localization, allowing models to search for relevant code snippets from large repositories by treating issue descriptions as queries and code snippets as indexed data. Some approaches use a sliding window to ensure completeness (Zhang et al., ).
Besides, Agentless (Xia et al., ) is a recently developed method that uses LLMs to directly identify relevant elements for editing within code repositories. It first recursively traverses the repository structure to generate a format that aligns files and folders vertically, with indents for sub-directories. This structure and the issue description are then input into the LLM, which performs a hierarchical search to identify the top-ranked suspicious files requiring further inspection or modification.

### 2.2 Repository-level Coding Capability

**Table 1: Comparison between our approach RepoGraph and existing methods for representing the repository on various aspects. *RepoUnderstander (Ma et al., ) and CodexGraph (Liu et al., ) are concurrent works to ours.**
| Model | Line-level | File-level | Repo-level |
| --- | --- | --- | --- |
| DraCo | ✗ | ✓ | ✗ |
| Aider | ✓ | ✗ | ✗ |
| RepoUnderstander* | ✗ | ✓ | ✓ |
| CodexGraph* | ✗ | ✓ | ✓ |
| RepoGraph | ✓ | ✓ | ✓ |

The evaluation of coding capabilities in AI systems has traditionally focused on function-level or line-level assessments (Lu et al., ; Chen et al., ; Austin et al., ), where individual code snippets or isolated functions are the primary units of analysis. Unlike previous studies, SWE-bench (Jimenez et al., ) highlights the trend of repository-level coding, driven by recent advances of coding-specific LLMs (Guo et al., ; Li et al., ). It reflects the growing user demand to understand and contribute to entire projects rather than isolated functions (Ouyang et al., ), as well as solving real-world problems in an end-to-end and automatic manner.

While the pre-trained code LLMs mentioned earlier incorporate repository-level information such as file dependencies, tasks at the repository level often involve more intricate call relationships within their context. Recent works like RepoCoder (Zhang et al., ) and RepoFuse (Liang et al., ) have started integrating Retrieval-Augmented Generation (RAG) modules to harness additional information from repositories. Building on this, subsequent research has focused on embedding repository-level context into their methodologies. For instance, DraCo (Cheng et al., ) introduces importing relationships between files, while Aider (Gauthier, ) employs PageRank (Page, ) to identify the most significant contextual elements. RepoUnderstander (Ma et al., ) and CodexGraph (Liu et al., ) model code files as a knowledge graph. Despite similarities in representation, methods vary in how they retrieve information from these structures and utilize it for downstream tasks. Table 1 summarizes the differences between these methods and RepoGraph. RepoGraph surpasses previous approaches by effectively integrating context at the line, file, and repository levels.

## 3 RepoGraph

This section introduces RepoGraph, a novel plug-in module that can be seamlessly integrated into existing research workflows for both agent-based and procedural frameworks.
The primary goal of RepoGraph is to provide a structured way to analyze and interact with complex codebases, enabling detailed tracing of code dependencies, execution flow, and structural relationships across the repository.
In the following sections, we will provide a detailed description of RepoGraph’s construction, its underlying representation, and its utility across various scenarios. The overall architecture is depicted in Figure 2, highlighting its key components and operational flow.

Figure: Figure 2: An in-depth illustration of (a) the construction, (b) the integration with procedural frameworks, and (c) the integration with agent frameworks of RepoGraph. Given a code repository, we first utilize AST to construct $\mathcal{G}=\{\mathcal{V},\mathcal{E}\}$, where $\mathcal{G}$ consists of “reference” and “definition” node, $\mathcal{E}$ includes “invoke” and “contain” relations. The constructed RepoGraph are then used in procedural frameworks by adding sub-retrieval results into each step, and agent frameworks by adding graph retrieval as an additional action “search_repograph”.
Refer to caption: x2.png

### 3.1 Construction

Given a repository-level coding task, the first step is to carefully examine the repository structure so that the necessary information can be collected.
The input for RepoGraph construction is a repository, i.e., a collection of its folders and files, while the output is a structured graph, where each node is a code line, and each edge represents the dependencies in between. RepoGraph enables tracing back to the root cause of the current issue and gathering dependent code context to help solve the problem.
The construction process of RepoGraph could be divided into three key steps.

Step 1: Code line parsing.
We first traverse the entire repository using a top-down approach to identify all code files as candidates for next-step parsing.
This is accomplished by filtering based on file extensions, retaining only those with relevant code file suffixes (e.g., .py)
while excluding non-essential file types (e.g., .git or requirements.txt), which are noisy and irrelevant for coding tasks.
For each code file, we utilize tree-sitter (^1^11[https://pypi.org/project/tree-sitter-languages/](https://pypi.org/project/tree-sitter-languages/)) to parse the code, leveraging its Abstract Syntax Tree (AST) framework. The AST provides a tree-based representation of the abstract syntactic structure of the source code, enabling the identification of key elements such as functions, classes, variables, types, and other definitions. While recognizing these definitions is crucial, tracing their usage and references throughout the code is equally important. Tree-sitter facilitates this by capturing the definitions and tracking where they are utilized or referenced within the codebase.
For example, in figure 2, we not only identify definitions like class Model and its inherent methods but also references like self._validate_input_units().
After processing each line of code with a tree-sitter, we selectively retain lines that involve function calls and dependency relations, discarding extraneous information. Our focus is primarily on the functions and classes, as these represent the core structural components of the code. By concentrating on these elements and their interrelationships, RepoGraph optimizes the analysis process by excluding less significant details, such as individual variables, which tend to be redundant and less relevant for further processing.

Step 2: Project-dependent relation filtering.
After the previous parsing step, we obtain code lines with calling and dependency relations. However, not all relations are useful for fixing issues. Specifically, many default and built-in function/class calls could distract from the project-related ones. Therefore, we additionally introduce a filtering process that excludes the repository-independent relations. Two types of such relations exist: (i) global relation refers to Python standard and built-in functions and classes. (ii) local relation are introduced by third-party libraries, which are specific to the current code file. For global relations, we maintain a comprehensive list of methods from standard and built-in libraries, excluding any identified relations from this list. The list is empirically constructed by gathering methods of the builtins library and default methods such as such as “list” and “tuple”. For example, in figure 2, line inputs = len(input) is excluded since “len” is a default method. For local relations, we parse import statements in the code to identify third-party methods that are included, and exclude them accordingly.

Step 3: Graph organization.
At this stage, we construct RepoGraph using code lines as the fundamental building units. The graph can be represented as $\mathcal{G}=\{\mathcal{V},\mathcal{E}\}$, where $\mathcal{V}$ represents the set of nodes, with each node corresponding to a line of code, and $\mathcal{E}$ represents the set of edges, capturing the relationships between these code lines. Each node in $\mathcal{V}$ contains attributes to represent its meta-information, such as line_number, file_name, directory, etc. Additionally, we classify each code line as either a “definition" (def) or a “reference" (ref) to a particular module. A “def" node corresponds to the line where a function, class, or variable is initially defined, while a “ref" node indicates a code line where this entity is referenced or invoked elsewhere in the code. Similar to soft links to “def” nodes, “ref” nodes also represent other variations of invoking methods.
For example, in Figure 2, the class definition “class Model” would be a “def" node, while any subsequent usages of Model would be “ref" nodes. Each “def" node may have multiple “ref" nodes associated with it, as a single function or class can be referenced in various places throughout the code.
We define two types of edges: $\mathcal{E}_{invoke}$ and $\mathcal{E}_{contain}$. The triple $(\mathcal{V}1,\mathcal{E}_{contain},\mathcal{V}_{2})$ denotes that $\mathcal{V}_{1}$ (e.g., a function definition) contains another module $\mathcal{V}_{2}$ (e.g., an internal function or class). The edge $\mathcal{E}_{contain}$ typically connects a “def" node to its internal components. In contrast, $\mathcal{E}_{invoke}$ represents an invocation relationship, usually connecting a “def" node to a “ref" node, where the reference node includes a dependency on the definition.

### 3.2 Utility

The constructed RepoGraph serves as a structured representation of the current repository and facilitates better-related information collection and aggregation. For information collection based on RepoGraph, specifically, we use one search term each time for subgraph retrieval. Search terms are the key functions or classes that are determined by current states. For example, “separability_matrix” is the initial search term in Figure 2(c). We retrieve the $k$-hop ego-graphs (Hu et al., ) with the search term in the centric.
The ego-graph is crucial for solving the problem because it focuses on the immediate relationships (Jin et al., ) around the search term, capturing the relevant dependencies and interactions within the repository, which is key to understanding the functional context.
Additionally, the retrieved content explicitly contains information at both the method and line levels and implicitly expresses the grouping at the file level.

This process is abstracted via search_repograph() as illustrated in the middle of Figure 2.
The retrieved $k$-hop ego-graph will be flattened for further processing. We also tried other variants for integration later in Section 5.2 and their performance in Table 4.
We narrate how RepoGraph could be plugged in with existing representative research lines in the following.

Integration with procedural framework.
In a procedural framework, LLMs are usually prompted in “localization” and “edition” stages with the given repository context and issue description.
In this case, we use search_repograph() before both stages, leveraging our RepoGraph to assist in making more informed decisions at each step.
For example, in Figure 2, we first include the subgraph of “separability_matrix” for localization, and then use the localized result “Model” to search in the edition stage.
To implement the strategy, we flatten the context of retrieved ego-graphs and append it as part of the prompt. As a result, the LLM generation is conditioned on both retrieved ego-graphs and the context provided by baseline methods, helping the model preserve nuances.

Integration with agent framework.
A significant difference in existing agent frameworks is the action space design, as expounded in Section 2. To leverage the power of RepoGraph, we put search_repograph() as an additional action in the action space. The agent decides when and where to use this action. The search term is also determined by the agent accordingly. The returned subgraph will be flattened and used as an observation for the next state.

## 4 Experiments

### 4.1 Setup

We evaluated RepoGraph as a plug-in component, i.e., integrated into existing baseline models of the two aforementioned research lines to assess its performance. We use the same baseline settings and configurations when incorporating RepoGraph to ensure a fair comparison.

Dataset.
We test RepoGraph in SWE-bench-Lite(^2^22Current leaderboard could be found at [https://www.swebench.com/](https://www.swebench.com/)).
Each problem in the dataset requires submitting a patch to solve the underlying issue described in the input issue description.
The goal is to generate a patch that accurately revises the relevant portions of the codebase within the repository, ensuring that all test scripts included in the dataset are successfully executed.

Baselines. We integrate RepoGraph with representative methods from both aforementioned research lines. (i) For procedural frameworks, we evaluate the widely used traditional method, RAG (Lewis et al., ), as well as Agentless (Xia et al., ), an open-source state-of-the-art approach in this direction.
For RAG, we follow its initial setting and use BM25 for file-level retrieval. After that, we append the context of RepoGraph after the code files as part of the prompt. Agentless first performs a hierarchical localization in terms of “file-class/function-edits” and then conducts repair based on localization. The context of RepoGraph is inserted in every step of Agentless.
(ii) For agent frameworks, we consider SWE-agent (Yang et al., ) and AutoCodeRover (Zhang et al., ). For both frameworks, we add an additional action “search_repograph” for the LLM agent as described in Section 3.
All the choices in the two research lines incorporate GPT-4 and GPT-4o based methods to ensure the generalization.
Detailed implementations and prompts used can be found in Appendix A.

Evaluation metrics. We evaluate all methods across two key dimensions: Accuracy and Average Cost. (i) For accuracy, we report the resolve rate and patch application rate. The resolve rate represents the percentage of issues successfully resolved across all data points. An issue is considered resolved if the submitted patch passes all test scripts. To assess the patch application rate, we attempt to apply the generated patches to the repository using the patch program; successful applications contribute to this metric.
(ii) To evaluate cost efficiency, we report two metrics: average cost and average tokens, which refer to the inference cost and the number of input/output tokens used when querying the LLMs, respectively.

Configurations.
We use the same GPT version as in the baselines in the experiments. We used GPT-4o (2024-05-13) and GPT-4-Turbo (gpt-4-1106-preview) from OpenAI for evaluation and analyses in our experiments.
All evaluation processes are performed in a containerized Docker environment (^3^33[https://github.com/aorwall/SWE-bench-docker](https://github.com/aorwall/SWE-bench-docker)), ensuring stability and reproducibility, made possible through contributions from the open-source community.
For plug-in with procedural frameworks, it usually takes around $2$-$3$ hours to finish. For agent frameworks, the inference time is larger, up to around $10$ hours.

**Table 2: Results of RepoGraph with open-source baselines in two research lines, including procedural and agent frameworks. Numbers of accuracy-related metrics are directly taken from the leaderboard, while the cost-related ones are computed from the corresponding trajectories(^5^55[https://github.com/swe-bench/experiments/tree/main/evaluation/lite](https://github.com/swe-bench/experiments/tree/main/evaluation/lite)).**
| Methods | LLM | Accuracy | Avg. Cost |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| resolve | # samples | patch apply | $ cost | # tokens |  |  |
| Procedural frameworks |  |  |  |  |  |  |
| RAG | GPT-4 | 2.67 | 8 | 29.33 | $0.13 | 11,736 |
| +RepoGraph | GPT-4 | 5.33 $\uparrow$2.66 | 16 $\uparrow$8 | 47.67 $\uparrow$18.34 | $0.17 | 15,439 |
| Agentless | GPT-4o | 27.33 | 82 | 97.33 | $0.34 | 42,376 |
| +RepoGraph | GPT-4o | 29.67 $\uparrow$2.34 | 89 $\uparrow$7 | 98.00 $\uparrow$0.67 | $0.39 | 47,323 |
| Agent frameworks |  |  |  |  |  |  |
| AutoCodeRover | GPT-4 | 19.00 | 57 | 83.00 | $0.45 | 38,663 |
| +RepoGraph | GPT-4 | 21.33 $\uparrow$2.33 | 64 $\uparrow$7 | 86.67 $\uparrow$3.67 | $0.58 | 45,112 |
| SWE-agent | GPT-4o | 18.33 | 55 | 87.00 | $2.51 | 245,008 |
| +RepoGraph | GPT-4o | 20.33 $\uparrow$2.00 | 61 $\uparrow$6 | 90.33 $\uparrow$3.33 | $2.69 | 262,512 |

### 4.2 Experiment results

Table 5 presents the main evaluation results of all baseline methods and the corresponding performance with RepoGraph ( +RepoGraph) as a plug-in in the SWE-bench-Lite test set. We also report the number of correct samples for each method. The performance increase is marked by $\uparrow$num. Based on the results, we have the following key observations:

(i) RepoGraph brings consistent performance gain for all combinations of frameworks and LLM model bases.
Specifically, RepoGraph achieves an absolute improvement of $+2.66$ and $+2.34$ in terms of the resolve rate for RAG and Agentless, respectively, which is $99.63\%$ and $8.56\%$ of relative improvement. The notable improvement demonstrates the effectiveness of our RepoGraph in adapting to various scenarios by inducing relevant code context and performing precise code editions. Additionally, our best performance so far by plugging in Agentless, $29.67$, achieves the state-of-the-art performance on the benchmark(^6^66[https://www.swebench.com/](https://www.swebench.com/).) among all open-source methods.

(ii) Performance gain brought by RepoGraph is slightly larger on procedural frameworks than agent ones. With procedural frameworks, RepoGraph correctly fixes more issues than agent ones. This could be due to two primary reasons. Firstly, we observed that mature procedural frameworks tend to achieve better baseline performance than agent-based frameworks on SWE-bench. The initial definitive nature of procedural frameworks, with their well-defined running flow and structure, allows them to leverage plug-ins more effectively. Another reason is that this deterministic behavior reduces the complexity that arises from dynamic decision-making, a key characteristic of agent-based systems, thereby enabling a smoother integration of performance improvements.

(iii) Performance gain brought by RepoGraph does not rely on more costs.
We also compute and report each method’s average cost and token consumption. By introducing RepoGraph, we manage to reduce the costs associated with managing the entire repository while achieving comparable or even superior performance. As shown in Table 5, the performance improvements achieved by RepoGraph are generally proportional to, or slightly lower than, the corresponding increase in cost. This indicates RepoGraph ’s performance gains are not mainly due to increased token usage.

(iv) Average costs are generally larger on agent frameworks with RepoGraph.
This phenomenon is especially obvious with SWE-agent, as it allows the agent to freely determine the next action based on the current observation. We also found that the integration with agent frameworks usually leads to larger cost increases, as exemplified by $+0.13\$$ and $+0.18\$$ with AutoCodeRover and SWE-agent, respectively. The reason lies in the large exploration space in agent frameworks. The agents might call the search_repograph() action many times, which leads to the explosion of prompt contexts. We encourage users to be mindful of cost and to adopt a more granular approach to cost control when integrating RepoGraph into the agent framework in the future.

## 5 Analysis

This section presents a detailed analysis to demonstrate that the additional context provided by RepoGraph is beneficial for the task. We begin by analyzing localization accuracy in comparison to the gold-standard patch. Next, we explore various RepoGraph configurations, focusing on how the additional context can be effectively integrated into the existing system. Finally, we perform an in-depth error analysis, highlighting aspects where RepoGraph can be further improved. For more analyses including resolve rate in various aspects and action distributions of agent frameworks, please refer to Appendices C.

### 5.1 Localization Coverage

**Table 3: Percentage of problems for accurate edition localizations with respect to file, function, and line levels. All the numbers are computed from the corresponding generated patches.**
| Methods | LLM | file | function | line |
| --- | --- | --- | --- | --- |
| RAG | GPT-4 | 47.3 | 23.3 | 12.7 |
| +RepoGraph | GPT-4 | 51.7 $\uparrow$4.4 | 25.3 $\uparrow$2.0 | 14.3 $\uparrow$1.6 |
| Agentless | GPT-4o | 68.7 | 51.0 | 34.3 |
| +RepoGraph | GPT-4o | 74.3 $\uparrow$5.6 | 54.0 $\uparrow$3.0 | 36.7 $\uparrow$2.4 |
| Agent frameworks |  |  |  |  |
| AutoCodeRover | GPT-4 | 62.3 | 42.3 | 29.0 |
| +RepoGraph | GPT-4 | 69.0 $\uparrow$4.7 | 45.7 $\uparrow$3.4 | 31.7 $\uparrow$2.7 |
| SWE-agent | GPT-4o | 61.7 | 46.3 | 32.3 |
| +RepoGraph | GPT-4o | 67.3 $\uparrow$4.6 | 49.3 $\uparrow$3.0 | 35.0 $\uparrow$2.7 |

A crucial step in issue resolution is accurately identifying the correct locations within the code that require modification. Proper localization is essential, as it forms the foundation for generating an effective and accurate patch. Without this step, the quality of the fix may be compromised, leading to incomplete or incorrect solutions.
In three granularity, we compute the percentage of problems where the edit locations match the ground truth patch. Namely, file-level, function-level, and line-level. We report that a patch contains the correct location if it edits a superset of all locations in the ground truth patch.

Table 3 presents the results of our analysis. We observed that integrating RepoGraph with all baseline methods significantly improves file-level accuracy, whereas the enhancement of accuracy in line-level is comparatively modest.
This result aligns with our expectations, as file-level localization is the most coarse-grained, making it inherently easier to improve. In contrast, line-level localization, being the most fine-grained, poses a greater challenge due to its need for more precise identification of code segments.
Additionally, we found that although line-level accuracy improvements are more pronounced for agent frameworks, their overall resolve rate is lower than that of procedural frameworks, as shown in Table 5.
This discrepancy can be attributed to the fact that localization, while necessary for generating a final patch, is insufficient. The success of the final revision still heavily relies on the underlying capabilities of LLMs.
Agent frameworks, designed to operate in a trial-and-error fashion, are particularly susceptible to error accumulation. As these frameworks iteratively refine their patches, small inaccuracies in earlier localization steps can propagate and magnify throughout the process, ultimately reducing the overall resolve rate. Procedural frameworks, on the other hand, follow a more structured and deterministic approach to localization and patch generation. They typically localize and fix issues in a single, more direct step, which can help mitigate the compounding of errors.

### 5.2 Investigation of RepoGraph Variants

**Table 4: The number of nodes, edges, and tokens of RepoGraph and its variants. For different retrieval and integration variants, we report the average number on the test set. “summ.” refers to the summarized version by LLMs of the retrieved ego-graph.**
| Metrics | RepoGraph | $1$-hop + flatten | $1$-hop + summ. | $2$-hop + flatten | $2$-hop + summ. |
| --- | --- | --- | --- | --- | --- |
| # Nodes | 1419.3 | 11.6 | 11.6 | 54.5 | 54.5 |
| # Edges | 26392.1 | 37.1 | 37.1 | 89.9 | 89.9 |
| # tokens | - | 2310.7 | 717.5 | 10505.3 | 1229.2 |
| resolve rate | - | 29.67 | 28.33 | 26.00 | 28.67 |

In this section, we investigate the efficacy of various combinations of sub-graph retrieval and integration techniques as outlined in Section 3.
We explore two sub-graph retrieval variants and two integration methods. Specifically, for sub-graph retrieval, we index $k$-hop ego-graphs where $k$ is set to $1$ and $2$. We limit our exploration to $k$ values up to 2 due to the extensive context required for integration and the potential introduction of noise or irrelevant nodes for $k\geq 3$. For the integration methods, we employ two distinct approaches: (i) directly flattening the textual sub-graph by explicitly detailing the relationships between the search term and its neighboring nodes, and (ii) leveraging an LLM first to summarize the sub-graph in terms of the core modules and salient dependencies, before proceeding with further processing. Detailed implementations of these variants and prompts used can be found in Appendix B.

We begin by presenting some statistics for RepoGraph and its various configurations. Performance evaluations are conducted on Agentless with RepoGraph integrated as a plug-in module. Table 4 reports the number of nodes and edges within the (sub)-graphs. Notably, the average number of nodes and edges for RepoGraph across the SWE-bench dataset is quite substantial, featuring over 1,000 nodes and 25,000 edges. This highlights the comprehensive nature of the constructed structure. For the different variants, when $k=1$, the information within RepoGraph is concentrated around the search term, resulting in an average of 11.6 nodes and 37.1 edges. As $k$ increases to 2, the retrieved ego-graph expands exponentially, reaching an average of $54.5$ nodes and $89.9$ edges. Moreover, directly flattening the retrieved ego-graph often significantly increases the token count, frequently reaching several thousand tokens. However, utilizing an LLM as an additional summarizer greatly reduces the token count, typically around a thousand.

Table 4 presents the resolve rates for four variants of our method. Notably, while the $2$-hop variant incorporates additional information, directly flattening this information results in the poorest performance, with a resolve rate of $26.00\%$, even lower than the original baseline. In contrast, incorporating summarization via the LLM significantly alleviates context length constraints and enhances information organization, thereby improving performance. For the $1$-hop variant, however, we observed that adding summarization actually degrades performance, reducing the resolve rate to $28.33\%$. We hypothesize that this occurs because the flat $1$-hop ego-graph already contains comprehensive information that fits within the LLM’s context window; thus, summarization may introduce inevitable information loss.

### 5.3 Transferibility Test

**Table 5: Results on subset of CrossCodeEval with GPT-4o as the backbone LLM.**
| Methods | Code Match | Identifier Match |  |  |
| --- | --- | --- | --- | --- |
| EM | ES | EM | F1 |  |
| GPT-4o | 10.8 | 59.4 | 16.7 | 48.2 |
| +RepoGraph | 28.5 | 68.3 | 36.1 | 61.9 |

To demonstrate the representational power of RepoGraph for repositories and its transferability to tasks requiring an understanding of repository structures, we conducted experiments using the CrossCodeEval benchmark (Ding et al., ). CrossCodeEval is a code completion benchmark designed to emphasize numerous cross-file dependencies. The original dataset consists of 2,665 samples derived from real-world repositories. Due to resource constraints, we randomly selected 500 samples from CrossCodeEval, focusing on problems using Python as the evaluation programming language.
For evaluation metrics, we follow the settings in the original paper and measure performance with code match and identifier match metrics, assessing accuracy with exact match (EM), edit similarity (ES), and F1 scores. In our experiments, the search terms are determined to be the function within which the current line is to be completed.

Table 5 demonstrates the results on CrossCodeEval using GPT-4o as the backbone language model. The original GPT-4o model struggles with repository-level tasks, evidenced by an EM score of only $10.8$ for code matching and $16.7$ for identifier matching. These results indicate significant limitations in handling code structure and variable usage in a broader repository context.
However, with the integration of the RepoGraph method, there is a substantial improvement across all metrics. Code Match EM improves dramatically to $28.5$, while identifier match EM more than doubles to $36.1$. Similarly, the F1 score for identifier matching jumps from $48.2$ to $61.9$, and the ES for code increases from $59.4$ to $68.3$. These improvements suggest that incorporating repository-level knowledge, as facilitated by RepoGraph, greatly enhances the model’s ability to understand and generate more contextually accurate and semantically consistent code.

### 5.4 Error Analysis

Figure: Figure 3: Venn diagram of RepoGraph and baseline methods on (a) procedural framework and (b) agent framework. We also plot the error distribution of failing cases against counterparts, e.g., detailed error distribution of $12$ cases RepoGraph succeeds while Agentless fails.
Refer to caption: extracted/5897883/figures/error_analysis.png

We want to compare RepoGraph with the corresponding baselines to see the distribution of resolved cases and analyze the error reasons for unresolved cases. We plot a Venn diagram for representative methods in both procedural and agent frameworks in Figure 3, respectively. We manually examined all the unique error cases and defined three error categories. (i) Incorrect localization refers to the failure in accurately identifying code snippets, (ii) contextual misalignment happens when the generated patch fails to align with the broader context of the codebase, and (iii) regressive fix introduces new issues in resolving the original issues. More examples are in Appendix D.

We found that the improvement in agent frameworks is more complementary than procedural frameworks, with larger uniquely resolved cases of $22$ compared with $12$ in procedural frameworks. Together, they make even larger performance of $31.33\%$ and $22.33\%$.
The reason could also be attributed to the determinism of procedural frameworks. As a plug-in module, RepoGraph tends to make modifications on existing deterministic processes, resulting in larger overlaps in resolved issues compared with baselines. Agent frameworks, on the other hand, have quite different action distributions with RepoGraph as a plug-in (please refer to Appendix A.2). Therefore, the uniquely resolved cases are more compared with procedural frameworks.
For error distributions, contextual misalignment is the most prevalent error type, followed by incorrect localization and regressive fixes for all methods, suggesting that while localization is often correct, the applied solutions may regress or fail to integrate contextually. The phenomenon also echoes with conclusion obtained in Section 5.1. This is intuitive as all the existing methods focus on providing a more comprehensive and desired context for LLMs to solve the task, which fundamentally depends on the power of backbone LLMs. We also found that when integrated with RepoGraph, the proportion of error types “incorrect localization” and “contextual misalignment” largely decreases, indicating that RepoGraph is specifically useful in aggregating the related contexts of the current to-be-fixed issue.

## Appendix A Detailed implementations for each baseline method

This section details the implementation of all four methods in procedural and agent research lines mentioned in Section 4.1.

### A.1 Procedural frameworks

Figure 4 and Figure 5 illustrate the instructions we used for procedural frameworks, including localization and edition, respectively.

Figure: Figure 4: Instructions used in the procedural framework to localize to detailed files and code lines of edition.
Refer to caption: extracted/5897883/figures/localization_instruction.jpg

Figure: Figure 5: Instruction used for fixing an issue based on the identified locations in certain template.
Refer to caption: extracted/5897883/figures/edit_instruction.png

We flattened the context of the retrieved ego-graph from RepoGraph into the template of the instructions. Specifically, in both “localization” and “edition” stages, RepoGraph is flattened in the part of Function/Class Dependencies so that the LLMs could better understand its context.

### A.2 Agent frameworks

LLM agents have a wide range of applications (Wang et al., ; Yuan et al., ).
We list all the instructions in every step of agent frameworks. The overall and system instruction is shown in Figure 6.

Figure: Figure 6: The signature of our new action “search_repograph” for agent frameworks.
Refer to caption: extracted/5897883/figures/system.png

The system instruction defines the task setting and the template for each response. We add “search_repograph” as a new action for the agent to use in the command_docs, with its signature listed in Figure 7.

We also plot the frequency for action invoked for both SWE-agent and SWE-agent with RepoGraph in Figure 8. We can see that with RepoGraph, the maximum turn of finishing the task is reduced from $38$ to $35$. Also, we computed the average turn to finish the task, which demonstrates a similar trend of $21.47$ to $19.12$, a significant improvement in efficiency while maintaining effectiveness. We also observe that the action “search_repograph” is invoked mostly in the first $15$ rounds of conversation with LLMs. It is more precise than the original action of “search_dir”, “search_file”, and “find_file”.

Figure: Figure 7: The signature of our new action “search_repograph” for agent frameworks.
Refer to caption: extracted/5897883/figures/search_repo.png

Figure: Figure 8: The frequency with which actions are invoked at each turn by (a) SWE-agent and (b) SWE-agent w/RepoGraph.
Refer to caption: extracted/5897883/figures/agent_action.png

## Appendix B Detailed implementations for each RepoGraph variant

In this section, we provide the implementations for variants of RepoGraph mentioned in Section 5.2.

Figure: Figure 9: Instruction used for summarizing the flattened ego-graph.
Refer to caption: x3.png

In addition to directly flattening the retrieved ego-graph, we propose leveraging large language models (LLMs) to first summarize the context. The full prompt, along with sample input and output, is provided in Figure 9.

## Appendix C Additional Results

### C.1 Resolve rate by repository

We plot the results of the resolve rate in terms of repository distribution in Figure 10. From the figure, it is clear that the resolution of issues varies significantly across different repositories. Notably, the Django and Sympy repositories have the most unresolved issues, with 75 and 61 unresolved issues, respectively. This may indicate a higher level of complexity in the issues or a larger backlog compared to the other repositories.
On the other hand, Django has the highest number of resolved issues, with 39 cases. This highlights a strong effort to address issues, even though the unresolved count is still high. Sympy follows closely with 16 resolved issues, suggesting a similar trend.
Other repositories like Scikit-learn, Sphinx, and Matplotlib have comparatively fewer issues overall, but their resolve rates are more balanced. For instance, Sphinx shows a ratio of 13 resolved to 3 unresolved issues, reflecting a more consistent effort in issue resolution.

Figure: Figure 10: Distribution of issues resolved by Agentless +RepoGraph plotted in terms of different repositories.
Refer to caption: extracted/5897883/figures/resolve_repo.png

### C.2 Resolve rate by time

We plot the results of the resolve rate in terms of the distribution of releasing time for repositories in Figure 11. Most of the issues were observed in recent years, starting from 2018, with a substantial increase in the total number of issues identified after 2018.
In the early years (2012-2016), the number of issues remained relatively low, with both resolved and unresolved counts being minimal. Starting in 2017, there has been a noticeable increase in unresolved issues, with 13 unresolved and only 3 resolved issues.
In 2019, the number of resolved and unresolved issues significantly increased, with 19 resolved out of 59 issues. This trend continued to rise until 2020, where 47 issues remained unresolved, and only 17 were resolved, marking the year with the highest number of unresolved issues in the dataset.
By 2021 and 2022, the number of unresolved issues slightly decreased, while the resolve rate increased compared to 2020. This suggests an improvement in the system’s ability to address issues in these years. In 2023, although the total number of issues dropped to 30, the proportion of resolved issues remained strong, with 8 resolved out of 22 issues.

Figure: Figure 11: Distribution of issues resolved by Agentless +RepoGraph plotted in terms of release time.
Refer to caption: extracted/5897883/figures/resolve_time.png

## Appendix D Examples for error analyses

To help better understand the error category listed in Section 5.4, we provide one example for each category in Figure 12, Figure 13, and Figure 14.

Figure: Figure 12: An example of incorrect localization. The correct patch modifies the “DurationField” in django/db/models/fields/__init__.py. This is the correct place to handle the error message formatting for the “DurationField” used in Django models. RepoGraph, however, modifies django/forms/fields.py. This file handles Django form fields, not the model fields. While both model and form fields have overlapping behavior, in this case, the correction is required for the model field (DurationField in __init__.py), not the form field.
Refer to caption: x4.png

Figure: Figure 13: An example of contextual misalignment. The model-generated patch places the __repr__ method in the wrong class DatasetGroupBy, whereas the correct patch modifies it in the GroupBy class. The __repr__ method should be implemented in the GroupBy class because it deals with the general group-by functionality. “DatasetGroupBy” is a subclass and doesn’t require a new representation method if “GroupBy” already has one. Additionally, when fixing white space trailing, while functionally similar, RepoGraph’s generation doesn’t exactly match the style or intention of the correct patch.
Refer to caption: x5.png

Figure: Figure 14: An example of regressive fix. The model-generated patch successfully resolves the issue which requires the return of objects with no args instead of subclasses of Atom by adding the key code line if not expr.args. However, the fix introduces other new issues such as the size of return objects, as exemplified in the unit tests.
Refer to caption: x6.png

## Appendix E Limitations and Future Work

(i) We only explored proprietary LLMs, i.e., GPT-4 series. Due to the poor performance of open-source models on this challenging task, we opted for proprietary models that have demonstrated superior results in code-related tasks. However, a comprehensive evaluation of open-source models such as Llama (Dubey et al., ) could be a valuable direction for future work, particularly as these models continue to improve.

(ii) Experiments were only conducted on the Lite set due to the high cost of running large-scale experiments with proprietary models. Exploring more efficient model deployment strategies and alternative cost-effective options for running experiments on larger datasets will be essential for broader applicability.

(iii) Although RepoGraph could be adapted to support other programming languages by adjusting the parsing schemes in the implementation, we only explored Python in our main experiments. Future work could extend this approach to other widely used programming languages, such as JavaScript, Java, or C++, to evaluate the generalizability of our methodology across different programming paradigms.

## Appendix F Impact Statement

The impact of this paper lies in its substantial contribution to enhancing the capability of AI-driven software engineering, particularly with respect to repository-level code understanding. The introduction of RepoGraph not only significantly improves Large Language Models (LLMs) in navigating and comprehending entire codebases, but also showcases the potential of integrating repository-wide structures into AI workflows. By extending the scope from function-level tasks to holistic repository management, RepoGraph pushes the boundaries of AI’s utility in modern software engineering.
This advancement opens new opportunities for using LLMs in complex engineering tasks such as automated debugging, repository maintenance, and large-scale refactoring. Furthermore, by highlighting the importance of repository-level context for accurate code generation and maintenance, the paper sets a new trajectory for future research in AI and software engineering. It encourages deeper exploration of AI’s ability to not only write code but also understand and manage large-scale software projects more efficiently.
We foresee minimal risks or negative societal impacts from this work. All datasets and benchmarks used in the evaluation are publicly available, and we adhered to their respective licenses. Additionally, RepoGraph has been open-sourced, making it accessible to the research community, particularly to groups with limited access to extensive computing resources, thus fostering broader adoption and further development.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: The authors state that in the error analysis 'when integrated with REPOGRAPH, the proportion of error types “incorrect localization” and “contextual misalignment” largely decreases'. However, this conclusion appears limited in significance due to the small number of passed samples involved. Could the authors provide further clarification on how this conclusion was reached? Have the authors considered the implications of having too few passed samples in their analysis?
soundness: 3
strengths: - RepoGraph can be easily built and integrated into existing methods and its construction process is straightforward and well-defined.
  
- The analysis presented in this paper is clear and detailed. The authors offering several key observations and insightful explanations, which could be inspiring for future research and broader adoption.
  
- The author conducted additional transferibility experiments to better explain the effectiveness of RepoGraph.
  
- The authors give comparative analysis between procedural and agent-based methods.
  
- A thorough classification analysis of error cases is provided, with specific examples for each category.
summary: This paper proposes RepoGraph, a plug-in module designed to extract repository-level definition and invocation information, offering graphical navigation to improve AI software engineering. RepoGraph is compatible with popular agent-based and procedural based LLM frameworks. The authors conduct experiments with GPT-4 and GPT-4o on SWE-bench, yielding a promising result.
weaknesses: - The primary experimental results are presented solely on SWE-bench-Lite, and the improvement in accuracy resolution are not particularly significant. Additionally, the paper mainly focuses on the resolved numbers and accuracy as evaluation criterion. In cases with a limited number of passes, the quantitative comparison and analysis is not convincing enough and may exhibit considerable variance. It would be beneficial for the authors to provide a thorough analysis of both pass and fail cases, as examining failed cases could also highlight improvements in error localization and correction capability of RepoGraph.

- As the authors note, the paper only evaluates based on GPT-4 and GPT-4o. Given the increasing availability of various General LLMs and Code LLMs, such as Claude and other open-source models, it would be better for the authors to include results from these additional LLMs. Previous studies indicate that different LLMs have varying performance when applied to new methods or tasks. It would be valuable to see how RepoGraph performs across a broader range of LLMs.

- A presentation issue in Figure 10: some numbers overlap, and certain white numbers blend into the background.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: According to Figure 3, RepGraph was not able to find all error cases when added as a plug-in with Agentless and SWE-Agent. 
As RepoGrpah works at line-level, file-level, and repo-level and was added as a plug-in, I expected RepoGrapgh to cover all error cases of Agentless and SWE-Agent.
Do the authors have any intuition or hypothesis of the observation of Figure 3?
soundness: 3
strengths: - A plug-in-based framework to easily integrate with other AI-based tools
- Code and docker provided for reproducibility
summary: This paper presents an approach to navigating software repos and
offer the desired navigation by incorporating
information at the line level, file level, and repository level. Toward that end, the authors
proposed a graph-learning-based framework with nodes corresponding to code-line level and edges corresponding
to connections of the nodes at the repo level.
weaknesses: - No human evolution studies
- Missing prior relevant literature

The authors could verify their assessment on SWE-bench Verified, released as a part of the benchmark.

Although the proposed work seems promising, the article lacks relevant literature citations. How to navigate the software repository has been studied by software engineering researchers before the LLM era. The
approaches include the navigation behavior of developers using eye-tracking data, during code summarization tasks.
I encourage the authors to consult the works of Bonita Sharif and follow the references.

References
1. Begel, A., Khoo, Y.P. and Zimmermann, T., 2010, May. Codebook: discovering and exploiting relationships in software repositories. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1 (pp. 125-134).
2. Bairi, R., Sonwane, A., Kanade, A., Iyer, A., Parthasarathy, S., Rajamani, S., Ashok, B. and Shet, S., 2024. Codeplan: Repository-level coding using llms and planning. Proceedings of the ACM on Software Engineering, 1(FSE), pp.675-698.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - Can the authors provide more information about the computational overhead involved in constructing and using the repository-level graphs? Specifically, how does the time complexity compare to other methods?

- Given the small performance improvements, could the authors discuss potential areas for further optimization or enhancement of RepoGraph that could lead to more significant gains?

- Could the authors offer guidance on the reproducibility, specifically:

    - What are the exact environment requirements for running the tool?
    - How should the necessary keys for LLM configuration be set up?
    - What steps should users follow to accurately evaluate the results generated by RepoGraph?
soundness: 2
strengths: 1. Originality: RepoGraph introduces a novel approach by leveraging repository-level graph-based representations, which provides a more comprehensive understanding of the codebase. 

2. Quality: The paper is well-written, with a clear explanation of how RepoGraph work. It provides a robust experimental setup that evaluates the tool's performance across multiple dimensions. 

3. Clarity: The distinctions between RepoGraph and other repository understanding techniques are clearly articulated, making it easy to understand the motivation and design of the tool. 

4. Significance: Its contributions could lead to improvements in the effectiveness of automated issue resolution and code navigation tools.
summary: This paper introduces RepoGraph, a plug-in designed to enhance the capabilities of LLMs in understanding and navigating code repositories. RepoGraph constructs a graph-based representation of the entire codebase using abstract syntax tree parsing. The system is evaluated using the SWE-bench Lite and CrossCodeEval benchmarks, demonstrating its effectiveness in issue resolution and fault localization. The authors also conduct various analyses, including complementarity and failure cause analysis.
weaknesses: 1. Performance Gains: While RepoGraph shows some improvement in issue resolution and fault localization, the gains are relatively small. 

2. Cost Assessment: The paper does not adequately address the potential computational costs associated with graph construction and search operations. A thorough analysis of these costs, including time overhead, would strengthen the paper by providing a better understanding of RepoGraph’s practical feasibility.

3. Reproducibility: The open-source code provided in the repository is not entirely reproducible based on the available documentation. Specific issues include: 

	(1) Environment Setup: The steps for setting up the execution environment are incomplete. It is unclear whether specific configurations or environments are needed for the tool to function properly. 

	(2) Configuration: The LLMs require additional keys for configuration, but the paper lacks guidance on how to set these up.

	(3) Evaluation Steps: There is no clear documentation on how to evaluate the generated results effectively. Detailed instructions on the 	evaluation process would help ensure that users can reproduce the experimental findings.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: 1.	Although the authors mentioned that due to resource limitations, only 500 samples were used for experiments on **CrossCodeEval**, the cost of conducting a full test on **CrossCodeEval** does not seem to be significantly higher compared to testing on **SWE-Bench**. This is because the cross-file context retrieved should not be very long. Could the authors report the costs of each test?
2.	While GPT-4o is also capable of code completion, **CrossCodeEval** primarily evaluates the completion abilities of base code models. Could the authors include results from some open-source code models, such as `Deepseek-Coder-V2` and `Qwen2.5-Coder`? These models should be relatively easy to run locally for inference.
soundness: 4
strengths: 1.	Conducting high-quality, fine-grained modeling of code repositories is extremely challenging due to the numerous corner cases that need to be considered. Implementing such a detailed model for code repositories is no easy task, and I truly appreciate the authors’ efforts.
2.	*RepoGraph* can be seamlessly integrated with both agent-based and non-agent frameworks, significantly improving the ability to fix issues on the **SWE-bench**. This will be highly beneficial for future agent-driven code development.
3.	*RepoGraph* can also be applied to repository-level code completion tasks, which will be very useful for code completion in daily development.
4.	The authors conducted extensive experiments to validate the effectiveness of *RepoGraph*.
5.	Compared to other papers I have reviewed, this paper is well-written, with a clear structure and a balanced level of detail.
summary: This paper introduces a framework called *RepoGraph* for fine-grained modeling of code repositories to address the current challenges in repository-level code tasks. *RepoGraph* leverages tree-sitter to model code repositories at the level of code behavior, constructing a code structure graph based on the invocation and dependency relationships between functions and classes. By using this graph, *RepoGraph* provides more accurate code context for given retrieval terms, thereby improving the accuracy of code task completion. The authors tested *RepoGraph* on the **SWE-bench** benchmark, demonstrating that it can be seamlessly integrated with both advanced agent-based and non-agent frameworks, significantly enhancing the accuracy of issue-fixing tasks. Furthermore, they tested it on the **CrossCodeEval** benchmark, verifying that *RepoGraph* can also be applied to repository-level code completion tasks.
weaknesses: This paper is well-written and does not have any major weaknesses, but there are a few minor typos:

•	`Line 228`: There is a repetition of `such as`.

•	`Line 306`: The sentence `To assess the patch application rate, we attempt to apply the generated patches to the repository using the patch program; successful applications contribute to this metric.` has a somewhat disjointed structure and logic. The semicolon is typically used to connect two independent but related clauses, but in this case, the first part of the sentence is not a complete clause, even though there is a logical connection between the two parts.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - [Introduction, Line 75] - “they tend to focus narrowly on specific files, resulting in local optimums”. I’m not sure what this means? Why do agent approaches have this tendency?
- [Intro, Line 85] - What is an “ego-graph”? I see the citation in Line 256, but may be good to define this explicitly as a preliminary somewhere earlier.
- [Section 3.1, Line 205] - how is “non-essential” determined? I could imagine that editing requirements.txt might be necessary to import a new library for carrying out a fix.
- [Section 3.1, Line 216] - What happens in step 1 if the codebase is not written in an object oriented manner? For instance, what if you’re dealing with JavaScript code primarily written in a functional programming style, which may not have classes? Or if you are editing a bash script which may not have these entities?
- [Section 3.2, Line 267] - What is an “edition” stage? Perhaps this is a type? (“Editing”?) Repeated again on Line 350
- [Table 2] - What is # samples?
- [Table 4] - The number of nodes is a bit confusing. In SWE-bench, each codebase has an average of 438k lines. If each node corresponds to a line, why are there only 1419 nodes on average?
- [Figure 3] - Is this for SWE-bench Lite? Or CrossCodeEval?
soundness: 2
strengths: RepoGraph examines the question of how codebases can be represented effectively. While other works have tried more traditional approaches for localization, such as retrieval and tool-augmented LMs, RepoGraph demonstrates that reshaping the codebase into a graph that is more traversable and useful for a language model can elicit stronger performance on the downstream editing subtask. The authors discuss a pipeline for carrying this out with Python codebases, showing how simple heuristics for removing noise and preprocessing the codebase in a more friendly way can lead to improvements that are method-agnostic. 

- [Section 3.1, Step 2] - This step seems quite interesting and novel. I think a visualization, something similar to Figure 2 but a bit more grounded in Step 3’s construction, might be helpful for readers to understand this construction more easily.
summary: The authors introduce RepoGraph, a new tool that turns a codebase into a graph structure. The authors demonstrate how this tool, which can be integrated into both agent-ic and procedural AI software engineering frameworks, leads to improvements across the board for existing methods. The repository is constructed by filtering out unnecessary file types and repository-dependent code, identifying object oriented programmatic entities, and constructing the remaining pieces of the codebase into a graph. The experimental setup follows existing work. The authors demonstrate improvements for a number of existing AI SWE solutions when RepoGraph is integrated into them and perform some analyses around RepoGraph parameters, localization statistics, and transferability.
weaknesses: - Figures 1, 2 - I appreciate the effort that went into these figures, they are clearly drawn quite thoroughly. However, I think it is a bit too complex, making it hard to understand. I’d recommend simplifying the diagrams to make the takeaways a bit more obvious. For example, it took me a while to understand the meaning of the colors in Figure 2, (a).
- The design of RepoGraph seems to be written in a fairly Python-centric manner. This is understandable to some degree, as the main evaluation (SWE-bench) is for Python based repositories. However, the contribution would be more impactful if somehow it was demonstrated that this approach scales to other kinds of codebases (with existing work or a new benchmark?). In Section 5.3, the authors mention how for CrossCodeEval, the evaluation is limited to Python. The authors discuss this in the appendix, but I think the generalizability of RepoGraph is a question that should be answered instead of being deferred to future work.
- [Section 3.1, Step 2] - “We maintain a comprehensive list of methods” ⇒ This sounds like it’s hard-coded for Python. Would I have to manually re-perform this filtering for other languages? Is there an automatic filtering criteria I can apply to come up with a comparable list for another language?
- [Section 5.2, Line 361] - This claim seems somewhat handwavy to me. The cost metric seems to go up with every method due to the inclusion of RepoGraph. However, this takeaway seems to suggest that the performance increase is proportional or lower than the increased cost - why is this true? What numbers are being compared for this proportion? (e.g. 2.66% increase vs. 0.04 cost increase? Why is this lower?)
- [Section 5.1, Lines 413 - 420] - The discussion on the differences between procedural and agentic approaches is interesting, but I think it’s not quite clear to me why this discussion is included or how it relates to RepoGraph. Is the suggestion that procedural approaches take more advantage of RepoGraph?

Meta-review: This paper introduces a new framework that can provide guidance and serves as a repository-wide navigation for AI software engineers. A graph-learning-based algorithm is provided, where nodes correspond to the code-line level and edges correspond to connections of the nodes at the repo level. Extensive experiments are conducted to demonstrate that the framework can be applied to realistic repository-level code completion tasks. 

The motivation of this work is clear with an easy-to-implement method. The presentation and organization are overall good. Experiments and following discussions are also comprehensive. The potential drawback of this work is that it is limited to Python and experiments can be enhanced by introducing human evolution studies. In general, this work makes valuable contributions to AI software engineering, and can inspire future work. Therefore, AC recommends accepting it.

Decision: Accept (Poster)