Title: RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for  Complex Task Solving
Venue: NeurIPS 2025
Accepted: yes
Keywords: Repository Comprehension

GitHub Integration

Efficient Repository Exploration

Autonomous Agents
Abstract: The ultimate goal of code agents is to solve complex tasks autonomously. 
Although large language models (LLMs) have made substantial progress in code generation, real-world tasks typically demand full-fledged code repositories rather than simple scripts.  Building such repositories from scratch remains a major challenge.  Fortunately, GitHub hosts a vast, evolving collection of open-source repositories, which developers frequently reuse as modular components for complex tasks. Yet, existing frameworks like OpenHands and SWE-Agent still struggle to effectively leverage these valuable resources.
Relying solely on README files provides insufficient guidance, and deeper exploration reveals two core obstacles: overwhelming information and tangled dependencies of repositories, both constrained by the limited context windows of current LLMs.
To tackle these issues, we propose RepoMaster, an autonomous agent framework designed to explore and reuse GitHub repositories for solving complex tasks. 
For efficient understanding, RepoMaster constructs function-call graphs, module-dependency graphs, and hierarchical code trees to identify essential components, providing only identified core elements to the LLMs rather than the entire repository. 
During autonomous execution, it progressively explores related components using our exploration tools and prunes information to optimize context usage.
Evaluated on the adjusted MLE-bench, RepoMaster achieves a 110\% relative boost in valid submissions over the strongest baseline OpenHands. 
On our newly released GitTaskBench, RepoMaster lifts the task-pass rate from 40.7% to 62.9% while reducing token usage by 95%. 
Our code and demonstration materials are publicly available at https://github.com/QuantaAlpha/RepoMaster.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel agent, RepoMaster, that is used to solve code tasks like the ones found in MLE-Bench. RepoMaster is able to search through many GitHub repositories to use published code in its solutions. It also incorporates clever strategies to help the LLM navigate code repositories, thereby improving its performance and saving tokens. Additionally, the paper proposes a new benchmark, GitTaskBench, comprised of tasks including super resolution, style transfer, source separation and more.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nI think that the premise of the paper, i.e., how can we leverage published code to solve general machine learning tasks, is interesting. I also think that efficient agents, like RepoMaster, are of interest to the community. \nThe components of RepoMaster, Hybrid Hierarchical Analysis, Code Exploration, and Information Selection, make a lot of sense, and they seem to work in practice, both by improving performance and saving tokens. \nThe paper is clear and well-structured.\n\nWeaknesses: \n1. How was the subset use for MLE-R chosen? Why was a subset taken and not the entire MLE-Bench?\n2. As for other baselines, why wasn't AIDE [1] used? It holds the current state of the art on MLE-Bench. \n3. I would have liked to see a more comprehensive description of GitTaskBench in the paper. Looking through the Appendix/repo, I found the tasks to be interesting, and I think a comprehensive description in the paper is warranted. I'm interested in the tasks in GitTaskBench, and would like to know more. For example, in some of the tasks, it seems like there could be several correct answers (image coloring, style transfer). How are correct solutions determined?\n\nLess significant, but still noteworthy:\n\nIn general I like the style of the figures, but I think they're too information-dense. They would be easier to understand if they were simpler in my opinion. \n\nThe descriptions sometimes lack detail. In Table 3's description, it should specify that the ablation study was done on RepoMaster + 4o on GitTaskBench.\n\nI think that seeing RepoMaster results on SWE-Bench would be very interesting. Many of the repos that make up SWE-Bench contain many files and lines of code, and it seems like an agent that uses Hybrid Hierarchical Analysis + Code Exploration could achieve high performance.\n\n\n[1] Jiang, Zhengyao, et al. \"Aide: Ai-driven exploration in the space of code.\" arXiv preprint arXiv:2502.13138 (2025).\n\nQUESTIONS:\n1. How was the subset use for MLE-R chosen? Why was a subset taken and not the entire MLE-Bench?\n2. As for other baselines, why wasn't AIDE [1] used? It holds the current state of the art on MLE-Bench.\n3. I'm interested in the tasks in GitTaskBench, and would like to know more. For example, in some of the tasks, it seems like there could be several correct answers (image coloring, style transfer). How are correct solutions determined?\n4. In terms of GitTaskBench -- how would the results change if the underlying models were changed to o4-mini or Gemini 2.5 Pro? These are competitive cost-wise with the models used in the paper, and would probably yield  scores of 70-75% on GitTaskBench, given that the current state of the art gets nearly 63%. In this case, GitTaskBench would be very close to being solved, already on release. How would you address this? I really like the premise of GitTaskBench, but practically speaking, it seems close to being solved.\n\n\n[1] Jiang, Zhengyao, et al. \"Aide: Ai-driven exploration in the space of code.\" arXiv preprint arXiv:2502.13138 (2025).\n\nLIMITATIONS:\nThough limitations weren't explicitly discussed, I do not think they're of particular relevance here.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RepoMaster, a novel automated framework designed to effectively leverage code repositories for solving complex real-world tasks end-to-end. RepoMaster aims to comprehend code in a goal-oriented, human-like manner by integrating multiple advanced techniques. Specifically, it combines hybrid structural hierarchy modeling, core component identification, and context-aware code exploration with efficient information selection strategies to navigate and analyze large codebases autonomously.\n\nRepoMaster operates in three main stages: (1) Repository Search to identify relevant repositories based on user intent, (2) Hierarchical Repository Analysis to construct a detailed structural map of the code, and (3) Autonomous Exploration & Execution to dynamically interact with the repository, perform task-specific actions, and refine its understanding through iterative exploration.\n\nThe paper demonstrates the effectiveness and efficiency of RepoMaster by conducting experiments on diverse, complex tasks drawn from the MLE-R and GitTaskBench datasets. In comparison with existing frameworks such as OpenHands and SWE-Agent, RepoMaster achieves superior performance in automating complex repository exploration and execution, validating its potential to address real-world software development challenges.\n\nSTRENGTHS AND WEAKNESSES:\nThe framework is inspired from human programmer behaviour to explore the unfamiliar codebases. The integration of context-aware code exploration tools with interactive execution and log filtering is a practical step for working within LLM context constraints while mimicking real developer behaviour.\n\nThe use of ASTs and graphs (HCT, FCG, MDG) provides a principled structure-aware foundation to the framework.\n\nHowever their are some weaknesses:\n\n(1) No mention of how the model adapts or generalizes to diverse languages or non-Python repositories. \n(2) The importance scoring seems entirely heuristic-based—lacking learning-based validation or ablation\n(3) How does RepoMaster deal with stale or broken dependencies found during exploration?\n(4) What are the assumptions on the quality of README or internal documentation? What happens when the README is missing or incorrect?\n\nQUESTIONS:\nQ1. Why restrict analysis to .py files only? Many practical projects involve configurations (.yaml, .json), scripts (.sh), or compiled extensions (.cpp). Does this limit applicability?\n\nQ2. Were the weights in the importance scoring scheme manually set or learned? How are the weights (and the corresponding contribution) determined for each feature?\n\nQ3. Did the authors consider using GNNs or unsupervised learning for structural importance prediction? Are there any negative insights or learnings from such experiments?\n\nQ4. What is the fallback strategy when errors persist (e.g., missing dependencies, ambiguous import paths)?\n\nLIMITATIONS:\nI do not find an explicit discussion on the limitations of the framework. Highly encourage the authors to present a discussion on the limitations as it would help the future works to understand the challenges effectively. Answering some of the questions I have asked earlier could give you a discussion on the limitations of the framework.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes RepoMaster, an automated agent framework that solves complex real-world tasks by reusing GitHub repositories. It combines structural hierarchy modeling, core component identification, and context-aware code exploration to help LLMs operate efficiently under context limits. This papers also introduce a new benchmark, GitTaskBench, and show that RepoMaster outperforms strong baselines like OpenHands and SWE-Agent on both GitTaskBench and MLE-R.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. This paper presents a comprehensive static analysis pipeline that models repositories via hierarchical code trees, function-call graphs, and module dependency graphs. It further ranks modules and classes using multiple features (e.g., centrality, complexity, usage, doc quality) to identify core components, allowing the agent to focus its limited LLM context on the most relevant parts of the codebase.\n2.  RepoMaster achieves much higher success rates than OpenHands and SWE-Agent on multiple tasks and LLMs. \n3.  The authors introduce GitTaskBench, a suite of diverse tasks each tied to a concrete GitHub repo, and evaluate on multiple LLMs and agents.\n\n**Weaknesses:**\n\n1.  Many of RepoMaster’s ideas e.g.repository graphs and context selection overlap with recent work. For example, the concurrent RepoGraph [1] system also builds a repository-level graph structure to guide LLM code agents. The CGM [2] approach similarly integrates a code graph into an LLM’s attention. RepoMaster’s novelty seems to lie mostly in the engineering of combining known techniques (AST traversal, call/dependency graphs, heuristic scoring) rather than introducing fundamentally new algorithms. As a result, its contribution feels somewhat incremental relative to these related methods.\n2. The experiments only compare with OpenHands and SWE-Agent, but do not include newer repository-level methods like RepoGraph [1] or other Agents works, such as Agentless [3]. Adding such baselines would better position RepoMaster within the current literature.\n3.  The proposed GitTaskBench contains only 18 repositories and 54 tasks (line 230-231). While it covers diverse domains, the relatively small size may limit the generality of the conclusions. A larger benchmark would better demonstrate the robustness of the method.\n\n\n[1] Ouyang, S., Yu, W., Ma, K., Xiao, Z., Zhang, Z., Jia, M., ... & Yu, D. (2024). RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph. arXiv preprint arXiv:2410.14684.\n\n[2] Tao, Hongyuan, et al. \"Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks.\" arXiv preprint arXiv:2505.16901 (2025).\n\n[3] Xia, Chunqiu Steven, et al. \"Agentless: Demystifying llm-based software engineering agents.\" arXiv preprint arXiv:2407.01489 (2024).\n\nQUESTIONS:\n1. The implementation filters to Python files. Would the approach extend easily to, e.g., multi-language projects (C++, Java) on GitHub?\n2. Will the authors release the GitTaskBench tasks and code for RepoMaster? \n3. How were the thresholds (e.g. top-20 modules, top-10 classes, context window sizes) chosen? Is performance sensitive to these?\n\nLIMITATIONS:\nThe proposed GitTaskBench contains only 18 repositories and 54 tasks (line 230-231). While it covers diverse domains, the relatively small size may limit the generality of the conclusions. A larger benchmark would better demonstrate the robustness of the method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes RepoMaster, an autonomous agentic framework designed to solve complex coding tasks by exploring and reusing existing GitHub repositories. Instead of generating repository-level code from scratch, the framework focuses on a more practical \"reuse and adapt\" approach. It addresses this via three-stages: (a) Repository Search: Identifies relevant GitHub repositories based on the user's task description (intent). (b) Hierarchical Repository Analysis: Before interacting with the LLM, the framework performs a static analysis of the chosen repository and constructs a Hierarchical Code Tree (HCT), a Module Dependency Graph (MDG), and a Function Call Graph (FCG) to map the codebase's structure. It then uses a scoring system to identify the most critical \"core components\" (files and classes).\n(c) Autonomous Exploration & Execution: The agent uses the pre-analyzed structural information to guide its exploration. It iteratively interacts with the codebase using specialized tools for viewing code, analyzing dependencies, and searching. To manage the LLM's context, it employs an information selection strategy that prunes code, documents, and logs to retain only the most essential information.  \n\nThe authors evaluate RepoMaster on two benchmarks: an adjusted version of MLE-Bench called MLE-R and a newly introduced benchmark named GitTaskBench. The results show that RepoMaster significantly outperforms strong baselines like OpenHands and SWE-Agent.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\na) The concept of performing a static, structural analysis before dynamic exploration is innovative. The creation of hybrid structural maps (HCT, MDG, FCG) provides the agent with a foundational understanding.\n\nb) The introduction of GitTaskBench is a good contribution to the research community. \n\nc) The authors conduct a robust evaluation using multiple state-of-the-art LLMs (GPT-4o, Claude 3.5, DeepSeek V3) and two strong baselines. The comprehensive ablation study effectively isolates the contribution of each of RepoMaster's core components, confirming that the performance gains are directly attributable to the proposed mechanisms. The case study is a plus. \n\nWeaknesses:\n\na) The success of the entire workflow is highly dependent on the initial \"Repository Search\" stage. This stage relies on analyzing user intent, README files, and star counts. The paper could benefit from a deeper discussion of the limitations of this search phase and how the agent might recover if an unsuitable repository is initially selected.\n\nb) The initial \"Hierarchical Repository Analysis\" involves parsing all source files to build ASTs, dependency graphs, and call graphs. While effective, the paper does not discuss the computational cost or time required for this preprocessing step.\n\nc) The module-level importance scoring aggregates six features using equal weights. This seems somewhat arbitrary. The paper would be strengthened by a sensitivity analysis on these weights or a more principled method for determining them.\n\nQUESTIONS:\nSee the weaknesses\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}