RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for  Complex Task Solving
Authors: Huacan Wang
Ziyi Ni
Shuo Zhang
Shuo Lu
Sen Hu
Ziyang He
Chen Hu
Jiaye Lin
Yifu Guo
Yuntao Du
Pin Lyu
Venue: NeurIPS 2025 spotlight ()
Keywords: Repository Comprehension
GitHub Integration
Efficient Repository Exploration
Autonomous Agents
Primary area: applications
TL;DR: A novel approach; Very practical

Abstract: The ultimate goal of code agents is to solve complex tasks autonomously. 
Although large language models (LLMs) have made substantial progress in code generation, real-world tasks typically demand full-fledged code repositories rather than simple scripts.  Building such repositories from scratch remains a major challenge.  Fortunately, GitHub hosts a vast, evolving collection of open-source repositories, which developers frequently reuse as modular components for complex tasks. Yet, existing frameworks like OpenHands and SWE-Agent still struggle to effectively leverage these valuable resources.
Relying solely on README files provides insufficient guidance, and deeper exploration reveals two core obstacles: overwhelming information and tangled dependencies of repositories, both constrained by the limited context windows of current LLMs.
To tackle these issues, we propose RepoMaster, an autonomous agent framework designed to explore and reuse GitHub repositories for solving complex tasks. 
For efficient understanding, RepoMaster constructs function-call graphs, module-dependency graphs, and hierarchical code trees to identify essential components, providing only identified core elements to the LLMs rather than the entire repository. 
During autonomous execution, it progressively explores related components using our exploration tools and prunes information to optimize context usage.
Evaluated on the adjusted MLE-bench, RepoMaster achieves a 110\% relative boost in valid submissions over the strongest baseline OpenHands. 
On our newly released GitTaskBench, RepoMaster lifts the task-pass rate from 40.7% to 62.9% while reducing token usage by 95%. 
Our code and demonstration materials are publicly available at https://github.com/QuantaAlpha/RepoMaster.

RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving
Huacan Wang Midea Group wanghe 141@m4441..c.om
Ziyi Ni
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems School of Artificial Intelligence, University Chinese Academy of Science Institute of Automation, Chinese Academy of Sciences niziy12021@ia.an
Shuo Zhang
Shuo Lu Institute of Automation, Chinese Academy of Sciences lushuo20240a..aco cn
School of Artificial Intelligence, Beijing University of Posts and Telccommunications shuozēbupt. edu.
Sen Hu Peking University husen@pku. edu.
Ziyang He
Chen Hu StepFun hatcher@sttepfun.com
National University of Singapore e e1553228@u. nus edu
Jiaye Lin Tsinghua University
Yifu Guo
University of Aberdeen u08yg22Gabdn, ac
lin-jy220mails. tsinghua, edu. en
Yuntao Du
Pin Lyu' Institute of Automation, Chinese Academy of Sciences pin. www.ivviia.mm ac.
C-FAIR, Shandong University
State Key Lab. for Novel Software Technology, NJU yuntaoduesdu. edu. en
MLE-Benchh Revissed Performance Comparison
Made Submission
Valid S Submission
Above Median
Medal
95.5%
95.5%
RepoMaster
45.5%
45.5%
OpenHlands
9.1%
54.5%
50.0%
SWE-Agent
4.5%
Equual contribution. "orresppondigg authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Abstract
The ultimate goal of code agents is to solve complex tasks autonomously. Although large language models (LLMs) have made substantial progress in code generation, real-world tasks typically demand full-fledged code repositories rather than sim- ple scripts. Building such repositories from scratch remains a major challenge. Fortunately, GitHub hosts vast, evolving collection of open-source repositories, which developers frequently reuse as modular components for complex tasks. Yet, existing frameworks like OpenHands and SWE-Agent still struggle to effectively leverage these valuable resources. Relying solely on README files provides insufficient guidance, and deeper exploration reveals two core obstacles: over- whelming information and tangled dependencies of repositories, both constrained by the limited context windows of current LLMs. To tackle these issues, we propose RepoMaster, an autonomous agent framework designed to explore and reuse GitHub repositories for solving complex tasks. For efficient understanding, RepoMaster constructs function-call graphs, module-dependenn graphs, and hi- erarchical code trees to identify essential components, providing only identified core elements to the LLMs rather than the entire repository. During autonomous execution, progressively explores related components using our exploration tools and prunes information to optimize context usage. Evaluated on the adjusted MIF--bench, RepoMaster achieves a 110% relative boost in valid submissions over the strongest baseline OpenHands. On our newly released GinTaskBenchh RepoMaster lifts the task-pass rate from 40.7% to 62.9% while reducing token usage by 95% Our code and demonstration materials are publicly available at
Introduction
In recent years, the integration of toolchains 1
3 4 and iterative reasoning 55
6 7 8
has
significantly enhanced large language models (LLMs) in code-related tasks 9
10
11]. These
debug errors (14), and even address complex machine learning problems [15 [6]. However, when confronted with real-world challenges that necessitate task-driven code repositories [17), they struggle. At present, tackling such tasks remains largely manual and time-consuming due to the complexity and scale of the required code, which makes purely generative approaches impractical [I] 18] 19]. To overcome this, we propose a paradigm shift: reuse and adapt existing repositories as modular components tailored specific tasks. This approach not only mitigates the challenges associated with repository- code generation but also supports the broader goal of enabling agents to autonomously address sophisticated tasks using simple natural language instructions 20 21].
advancements have enabled LLMs to proficiently complete code snippets [3] [3).
To facilitate this approach, leveraging platforms like GitHub becomes crucial. With over 28 million public repositories out of 190 million total projects, GitHub offers an extensive library of ready- made solutions for code agents [17 [8] (2 (3). Developers frequently reuse these repositories to tackle complex problems, yet LLM-based systems still falter in fully automating this process. Although frameworks like OpenHands (24] and SWE-Agent [14] demonstrate strong general capabilities, they often stumble real-world codebases. In practice, simply following README instructions seldom works: READMES can be vague, incomplete, or even erroneous, and repositories are not guaranteed to match task's requirements out of the bommans may need parameter changes, and key files can be misplaced. Consequently, when agents fail to locate or execute the necessary code, they must adapt by modifying existing components or generating new code to bridge the gap.
To achieve it, agents need to understand the repository in a task-driven way. However, GitHub repositories often have two key properties that make this hard: (1) intricate structural complexity, with many interconnected files, classes, and functions, and (2) information density that exceeds the context limits of most LLMs. Existing frameworks [14 [5] 24 25] do not provide mechanisms for grasping repository structures, tracking detailed dependencies, or strategically managing information within these constraints, ultimately resulting in suboptimal performance and higher token cost.
In this paper, we introduce RepoMaster, an end-to-end agent framework designed for automating the use of code repositories to tackle complex tasks. To address these challenges, RepoMaster draws

---

inspiration from human programmers, who rarely read every line of code or error log when exploring unfamiliar codebases. Instead, they first map a project's structure, start viewing a key file, then jump to its relevant files based on signals like error traces, and filter out irrelevant details.
Following this intuition, RepoMaster first performs hierarchical structure analysis, builds dependency and call graphs, and identifies core components as the initial context. Navigated by these connections, progressively explores the repository and applies information selection when viewing files and execution feedback to keep each interaction concise. By iteratively applying these steps, RepoMaster mimics human prioritization and makes efficient use of limited context windows. When evaluated on both MLE-R- revised version of MLE-Bench-Le
[]6] and our newly constructed GitTaskBende (26), RepoMaster achieves significantly higher completion and success rates than OpenHands and SWE-Agent, while using far fewer tokens. Our contributions are summarized as follows:
(I) We propose novel automated framework, RepoMaster, that can effectively leverage code repositories to solve the complex real-world tasks end-to-end. (2) To efficiently comprehend code in a goal-oriented, human-like manner, we integrate hybrid structural hierarchy modeling with core component identification, context-aware code exploration, and efficient information selection. (3) We validate RepoMaster's effectiveness and efficiency against Openhands and SWE-agent through experiments on diverse complex tasks from the MLE-R and GifTaskBench.
2
Related Work
2.1 Code Generation
LLMs have made substantial progress in code generation 12
13
27
exemplified by closed-
source models [29] 30 31 and the open-source series [3
33
34
35
Beyond basic code com-
pletion [36]. modern LLMS now support advanced tasks such as semantic code editing [23] (37] debugging (38), and generating machine learning pipelines (e.g.. AIDE [25] and MLAB [5] for
Kaggle competitions). However, fully automating the creation of complex woal-world codebases from
scratch remains a critical challenge for AI agents [16 19 22
2.2 LLM-based Agents for Tool Use
External tools are essential for extending the capabilities of LLM agents [5] 39 40) Relying on executable code 9 scripts to import inherent libraries, or call APIs, functionalized tools has become a mainstream paradigm. Current works mainly focus on "tool learning" 1 I0 but the more essential aspect of where to find the right tools is relatively overlooked [41]. Benchmarks, such as API-Bank practical; platforms such as RapidAPI 44 host real services but are closed-source and hard to extend. Standards such as FastAPI 45 or MCP 446,, which unify interfaces for tool use via function calling mechanisms, have emerged. However, GitHub a rich and dynamic ecosystem for automatically creating
and ToolEyes 45), synthesize function libraries but are not realistic or
underutilized in this context. Although GitAgent [17] first explored GitHub repositories tool extension, it is limited by simplistic repository search and understanding, and lacks validation in diverse real-world scenarios.
2.3 Repository Utilization
Using GitHub repositories to solve complex real-world tasks presents significant challenges. Re- poAgent 47 produces high-level documentation but fails include realistic, task-oriented usage examples. ML-Bench-A [18 focuses on setting up the environment rather than understanding the repository. OpenHands 24 and SWE-Agent [14] are strong general agents that use step-by-step prompting to break down tasks and write code, but they lack methods to deeply understand the repository structure or build a clear hierarchy of its components. Aider 48 can track file dependen- cies but misses detailed function-level connections and cannot autonomously explore the codebase. projects but struggle in large-scale repository contexts due to limited dependency awareness.
Interactive assistants like Copilot [49] and Cursor (50) are effective for
3

---

(1) Repository Search
Hierarchical Repository Analysis
user's intent
Extract
want seratches from this old image."
#key entities
Search Select
Hierarchical Code Tree (HCT)
Module D Dependency Graph (MDG)
Function Call Graph (FOG)
RapoMaster
README
preparing
Star number
Core components ffile, class)
3 Autonomous Exploration & Execution
"I want remove scratches from this image.""
Granular Code View
Dependeacy Analysis
Explortiio
Agent
Initill components
Search
produce action
Exploration tools
Explore the Repio
RepoMaster
Search
Actions > Observations
Analyse se Feedback & Optimize
Iteratively optimize
Expand viewed scope
Figure I: Overview of RepoMaster, consisting of Repository Search, Hierarchical Repository Analysis and Autonomous Exploration & Execution.
Method
Most current frameworks follow the CodeAct paradigm 14 offering basic file-editing and exploration commands (e.g.. OpenHands' AgentSkills [24] and SWE-Agent command set But relying on README-based mappings and simple find/edit operations misses many core components and cannot perform deeper, autonomous exploration within limited LLM contexts. In contrast, Rapterster mimics human programmers by performing a static, structure-aware analysis to locate critical components, then dynamically selecting only the essential snippets snippets-skippingg skipping irrelevant information and focusing the LLM's limited context on what matters. The full end-to-end RepoMaster framework consists of three stages: (1) Repository Search: Identifying repositories relevant to the task. (2) Hierarchical Repository Analysis: Preparing the structures for exploration.
(3) Autonomous Exploration & Execution: Iteratively interact with the repository and adjust exploration actions based on execution feedback. An overview of the framework provided in Figure)
3.1 Repository Search
To address complex online tasks expressed in natural language, we develop a deep-search method to locate the GitHub repositories most relevant to the task. We begin by analyzing the user's intent and extracting key entities to target the suitable repositories. We examine their README file and star count assess their relevance and potential, and provide brief description. Then, select them by content quality and practical utility. Finally, we validate the top three candidates and deliver the results as structured JSON. An example of the deep-searching log is shown in Appendix B.
3.2 Hierarchical Repository Analysis
3.2.1 Hybrid Structural Repository Mapping
essential prerequisite for task-oriented repository automation is comprehensive structural model of the codebase. We sanitize the repository by removing all non-source files. For each retained

---

file, we perform a single Abstract Syntax Tree (AST) walk [T] to recursively harvest both the meta-information and the raw source snipper of every module, class, and function. These atomic units provide the basis for understanding the repository's structure.
Let the target repository be denoted R (M, C, F, where M {m.,.... [MM is the set of modules (one per -py file), C fci..... the set of classes, F = {fi,.... (fii! the set of functions/methods, and I Ç M x M the explicit import relations captured from source files. On this foundation, we construct three complementary artefacts:
Hierarchical Code Tree (HCT). T, nested package - module + class + function con- tainment map annotated with line counts and doestring snippets.
Function Call Graph (FCG). G3 (V, F, Ey, wy), where edge (fe E E exists if fa invokes fj: the weight wy encodes call frequency.
Module Dependency Graph (MDG), G.. = (Vm = M, Em: Wm,), which (m,, my) Em if my explicitly depends on Mij Wm measures coupling strength.
We thus obtain the tuple ( (M, C, F,I, Gy, Gm, T providing the agent with deterministic, loss- minimal structural synopsis of the entire repository before any task-specific exploration.
3.2.2 Core Component Identification
Having obtained a fine-grained yet verbose structural synopsis of the repository, now need to compress this information into concise context that preserves only the most influential code entities- small enough for multiple interaction turns within the LLM's window, yet rich enough preserve global semantics. To this end, we specify an importance scoring scheme that operates first at the module level and then propagates classes.
Module-level scoring. Each module m M receives score [(m) E [0, 10] by linearly aggregating six orthogonal features,
s [Dependency, Complexity, Usage, Semantic, Doc, Git].
(mm) min Ei wa (m)). 10). w,
(2)
where Dependency captures centrality in MDG using the personalized PageRank [52) algorithm, Complexity approximates cyclomatic complexity, Usage measures import and call frequency, Semantic flags high-value keywords (e.g., main, core), Doc quantifies docstring richness, and Git reflects commit volume and recency. Detailed formulas for each feature are deferred to Appendix F. The equal weighting scheme (w; 1) was empirically validated through sensitivity analysis (see Appendix G.4 for details).
Class-level refinement. Module scores serve as priors for class importance. For every class c located in module j(c), compute
Calls(Fe) max Fel max. Calls(Fe)*
Fel
J(c) = I(((e) +
+
(3)
where F, denotes the method set of class c. The second term rewards class richness in functionality; the third term captures how often its methods are actually invoked in the repository. Classes are ranked according to [c), and the top-k classes are selected as the repository's core components.
3.2.3 Repository Context Initialization
Building on the identified core components, we construct an initial repository context in four distinct blocks. First, we include the complete README md file, which provides high-level descriptions and detailed usage guidance authored by human developers. Second, we append series of concise natural-langen natural-language summaries for the highest-priority modules, giving the LLM brief overview of each critical script's 's purpose. Third, we provide the source code of core components (i.e., the classes scored and selected Section 3.2.2p as fine-grained semantic anchors. Finally, for all other top-ranked modules, we provide flat, directory-groupe list of their file paths for easy on-demand lookup. Figure 2 illustraees the initial context, and Appendix D provides complete example of this initial repository context construction.
5

---

Initial Context
Confext-aware Code Exploration
To view efficiendy.
Information Selection
REAUMEE.M
Codes
Logs
Enprplict
Medile Stmmmrry
4. Add into Contest
Exploration Calling
7.
Code Satgees
(3,)
AST:
Segement Match
Search Key Ensiiess
Dependency
Interactive Feedback based Execution
Prompt:
Herrove this Iraug''s scraicio
Eeput pabi
Ciez compenara
Ho ducatce sky....
warch the kyensets,
Enecirr keb paryre
outpott Corpartion image
Clock
Hage purpurt
Nead checepsitt sst. foudl
Output +130011
directory
Orginaal Image
Output trange
Figure 2: Overview of RepoMaster's autonomous exploration-exxcuios loop and example demonstration. The agent begins by analyzing the initial context (Step 1) and specifies a file to inspect (Step 2). For efficient viewing, it extracts only the key information from that file (Step 3) and appends to the context (Step 4). the next exploration- execution iteration (Step 6-2,, Step 7-+3), the agent uses exploration tools identify additional relevant files and repeats context-aware code exploration. Once has gathered enough information. RepoMaster alternates between writing and running ppy" scripts, handling errors, and debugging based on feedback until the task is completed.
cnteraatooneeccttioon
This structured context serves as the agent's "launchpad" for dynamic exploration, allowing it to prioritize high-impact modules, trace dependencies, formulate targeted code queries and select relevant classes or functions, bridging static analysis with dynamic reasoning and task execution.
3.3
Autonomous Exploration & Execution
3.3.1
Context-aware Code Exploration
Once the agent has internalized the repository's functionality and overall structure, it immediately transitions to dynamic analysis, performing an autonomous, hierarchical and graph-based traversal of the codebase. To support in-depth comprehension and effective utilization of the repository, offer a suite of fine-grained exploration tools organized into three categories: Granular Code View, Dependency Analysis, and Search.
Granular Code View. This tool enables the agent to inspect the implementation details of files, classes, and functions using the HCT. It also retrieves and exposes the repository' directory hierarchy, facilitating swift orientation within the codebase.
Dependency Analysis. This tool traces call chains and dependency paths by analyzing the FCG and MDG, respectively. It uncovers complex invocation and dependency relationships among code entities, thereby deepening the agent's comprehension of module interactions and overall code structure.
Search. This tool equips the agent with robust search capabilities, facilitating rapid location of specific code segments within large and intricate codebases. It employs keyword matching to ensure efficient retrieval of relevant entities.
Together, these tools empower Al agents to proactively and autonomously navigate and examine code repositories, achieving a level of comprehension and flexibility comparable to human developers. Empirically, we observe that complex repositories typically require detailed dependency analysis using FCG and MDG, whereas simpler repositories often allow agents to effectively rely on HCT.

---

3.3.2 Interactive Feedback-baset Execution
Task execution is grounded in the agent's evolving understanding of the repository. Once the agent has identified the hybrid structural elements described in Section and core components described in Section 3.2.2 relevant to given task, it begins to perform task-oriented operations.
Crucially, execution and exploration form continuous, interleaved loop rather than linear sequence. The agent can fluidly alternate between writing code and locating files, viewing content and reading logs, or tracing dependencies, all driven by the task context across different interaction turns, and powered by the exploration tools described in Section 3.3.J] This flexible loop allows the agent to iteratively refine its behavior by retrieving just-in-tima information from the codebase. Figure illustrates the execution and exploration pipeline of RepoMaster.
3.3.3 Context-aware Information Selection For Efficient Viewing
The agent must juggle source code, documentation, execution results and logs within tight LLM token window for multiple turns, making it difficult to maintain a globally coherent view of the repository and severely limiting its applicability to large projects. To mitigate this issue, we propose a multi-level content reduction strategy that retains only the most critical information.
Viewing Code. At the code level, the agent parses source files into Abstract Syntax Trees (ASTs), extracts semantically and structuralyy meaningful subtrees, and uses these extracted subtrees as inputs.
Viewing Documents. For large or unstructured artifacts (c.g., txt or csv files), the agent divides each file into fixed-length chunks of Le tokens, It then generates retrieval prompts tailored to the current subtask, ranks the chunks by relevance, and retains the top n. most relevant segments.
Viewing Feedback Logs. At the log level, we apply human-like debugging heuristic that retains only the opening and closing segments of the log (where command invocations, exception traces, and diagnostic results cluster) and discards verbose intermediate output.
Multi-level reduction strategies activate only when the combined size of all candidate inputs exceeds the per-interaction token limit L, preserving global coherence by focusing on high-impact information and ensuring each exccution-loop step relies on a compact, relevant context.
4 Experiments
4.1 Benchmarks and Metrics
To validate the effectiveness of RepoMaster, we evaluate it using two benchmarks.
The original MLE-Bench [16] derives from Kaggle competitions, designed to evaluate LLM agents' capabilities in end-to-end machine learning engineering tasks. To construct MLE-R, we select 22 MLF-Bench tasks (covering nearly all MI.E-Bench-lite cases) and apply the search procedure described in Section 3.1 to retrieve suitable GitHub repositories for each task, ensuring fair comparison } the tasks' requirements are set to be completed based on their chosen repository rather than generating code from scratch.
MLE-R.
Performance in MLE-R is evaluated using a medal-based system, the same as the original MLE- Bench, where solutions are assessed based on official Kaggle thresholds for gold, silver, and bronze medals. Metrics include the achieved score, medal thresholds, and medal qualification, providing a clear indication of the model's 's proficiency in competitive ML engineering tasks.
Benchs In contrast MLE-R, which emphasizes standard machine learning tasks (e.g.. image classification), our new proposed GenTaskBench [26 5 benchmark evaluates I.IM agents on more practical real-world problems-commit leveraging existing repositories, such as photo restoration. The benchmark consists of 18 repositories and 54 tasks, all described in natural language and designed to be completed using the provided repositories across a wide range of domains, such as image processing, video analysis, speech, physiological signals, office automation, and security and privacy.
tasks whose complexity or format largely demands
Ennsre a fair comparison, as other general agent frameworks do not support automatic repository retrieval. "The specific thresholds for gold, silver, and bronze medals are provided in Appendix
MMor detailed descriptions can be found in Appendix A.
7

---

Table 1: Performance comparison of different frameworks and LLMs on MLE-R. The best perfor- mance is bolded, and the second-best underlined; the same is below.
Made Submission (%)
Valid
Above Median (%)
Bronze (%)
Silver
Gold (%)
Any Medal (%)
Framework
LLM
Submission (%)
(%)
GPT
72.73
54.55
0.00
0.00
0.00
0.00
0.00
SWE-Ageet
Claude 3.5
54.55
50.00
13.64
0.00
0.00
4.55
4,55
DeepSeck V3
54.55
36.36
4.55
0.00
0.00
4.55
4.55
GPT-4o
50.00
45.45
0.00
0.00
0.00
0.00
0.00
OpenHands
Claudo
50.00
45.45
9.09
0.00
0.00
4.55
DeepSeek V3
36.36
GPT-4o
86.36
86.36
36.36
4.55
0.00
13.64
18.18
RepoMaster
Clande 3.5
95.45
95.45
45.45
4.55
0.00
22.73
27.27
DeepSeck V3
95.45
86.36
36.36
4.55
4.55
13.64
22.73
GirTaskBench evaluates two key aspects: Execution Completion Rate (measuring the model's ability leverage the repository for output) and Task Pass Rate (assessing whether the output meets task-specific evaluation criteria). Given the diversity of tasks, evaluation metrics are predefined and tailored within the benchmark, ensuring a comprehensive assessment. Note that total tokens include both input and output tokens.
4.2 Evaluation Setup
We evaluate our approach against two baseline frameworks and compare the performance across three state-of-the-ard LLMs. The evaluation setup is as detailed below.
Baseline Frameworks.
We evaluate two baseline frameworks: OpenHands (24 and SWE-agent [14]. OpenHands provides sandboxed environments for code execution and API interactions, while SWE-agent focuses on autonomous GitHub issue resolution.
Large Language Models. We evaluate multiple leading LLMs, including the closed-source GPT- 4o-2024-08-06 [53] and Claude-3-5-sommee-222241000
[54] well the open-source DeepSeek V3-0324 [55] This setup enables a comprehensive assessment of both agent architectures and LLM capabilities on solving real-world tasks with repository utilization.
Implementation Details. Our proposed solution RepoMaster built on a multi-agent dialog platform few key hyperparameters. Specifically, set the maximum token length per interaction L to 8000 tokens. For initial context construction, we generate concise summaries for the top 20 modules by importance score and extract k 10 key classes. During the feedback phase, unstructured text files are split into chunks of Le 1000 tokens, retaining the n. most relevant segments.
AutoGen (1 To ensure agent performance,
4.3 Comparison with SOTA
On the MLE-R benchmark, RepoMaster with Claude 3.5 attains 95.45% valid submission rate and 27.27% medal acquisition rate (including 22.73% gold medals), representing more than five-fold improvement over the best open-source Agent baseline. RepoMaster with GPT-4o also achieves strong 86.36% valid submission rate and 18. 18% medal rate, further confirming its robust performance advantage under varied settings.
RepoMaster's significant performance improvement stems primarily from its effective identification and utilization of core components within open-souree repositories, such as neural network architec- ture designs, optimized hyperparameter configurations, and data preprocessing pipelines. In contrast, baseline methods like OpenHands and SWE-Agent often struggle to pinpoint critical modules during repository exploration, filling limited context windows with excessive irrelevant code, resulting in insufficient understanding of model architectures and training logic.
In the GinTaskBench evaluation, RepoMaster significantly outperforms existing open-source frame- works SWE-Agent OpenHands. Based on Claude 3.5, RepoMaster achieves a 75.92% execution completion rate and 62.969 96% task pass rate, surpassing OpenHands (53,70%, 40.74%) and SWE-Agent

---

(41.67%, 22.23%). Similarly, RepoMaster maintains significant advantages GPT-4o and DeepSeek V3, demonstrating that RepoMaster's inherent capabilities have good universality across underlying models. More importantly, RepoMaster substantially reduces computational overhead, with token consumption when using Claude 3.5 approximately 95% lower than OpenHands (154k vs 2883k tokens/task), proving the effectiveness of our hybrid hierarchical structure analysis and information pruning strategies.
Table 2: Performance comparison of different frameworks and LLMs on GirTaskBench.
Framework
LLM
Execution Completion Rate (%)
Task Pass Rate (%)
#Total Tokens
GPT-4o
17.58
10.19
276k
SWE-Agent
Claude 3,5
41.67
22.23
456k
DeepSeek V3
18.52
12.04
414k
GPT-4o
21,30
OpenHands
53.70
40.74
2883k
DeepSeek V3
45.37
26.85
4750k
GPT-4o
48.14
40.74
250%
RepoMaster
Chude 3,5
75.92
62,96
154k
DeepSeek V3
61.11
44.44
255k
4.4 Insightful analysis
Ablation Study To quantitatively assess the contribution of each component in RepoMaster, conduct a comprehensive ablation study on the GirTaskBench benchmark using GPT-4o as the underlying model. By systematically removing key mechanisms, we measure their impact three metrics of effectiveness and efficiency: execution completion rate, task pass rate, and token usage. The results shown in Table
Table 3: Ablation study on the impact of core mechanisms in RepoMaster with GPT-4o on the GeeTasBBench Benchmark. The "w/o All" refers to RepoMaster without any of the three mechanisms. "Exec. Comp." " stands for "Execution Completion."
Configuration
Exec. Comp. Rate (%)
Task Pass Rate (%)
#Tottl Tokens
RepoMaster (Full)
48.14
40.74
250k
w/o Hybrid Hierarchical Analysis
46.29 (-1.85%)
38.88 (-1 8666))
274k
w/o Code Exploration
44.44 (-3.7%)
35.18 (-5.56%)
284k
w/o Information Selection
46.29 (-1.85%)
38.88 (-1.86%)
265k
w/o All
29.62 (-18.52%)
24.07 (-16.67%)
112k
Removing this component causes slight decreases in execution completion and task pass rates, with other components partially compensating. Token usage increases by 9.67% showing that without structural repository understanding, more exploration is required.
Hybrid Hierarchical Analysis:
Context-aware Code Exploration: This component provides a suite exploration tools enabling agents to autonomously navigate codebases according task requirements, inspect file contents, and trace call chains and dependencies. The experiments prove this component to be particularly critical, with its removal causing significant decreases in execution and pass rates. More notably, token usage increases substantially, validating our hypothesis that strategic exploration effectively reduces repository exploration needs, allowing the agent to precisely target task-relevant components.
Information Selection: This component applies multi-level content reduction strategies to extract key information from source code, documents, and execution logs, preventing the context window from being filled with irrelevant content. Removing it decreases performance similarly to removing hybrid hierarchical analysis, with token usage increasing by only 6.00%. Its main value is maintaining a high signal-to- to-noise ratio rather than reducing token consumption.
The most revealing comparison is between the full RepoMaster system and the base code agent without any of our proposed components. The baseline achieves only 29.62% execution completion and 24.07 % task pass rates rates-decreaae decreases of 18.52% and 16.67%. Interestingly, the baseline's 's token

---

"Estinate 1D loman videe's 10 Las polats using prosided VidesPoss50 repoiluees
RepoMaster
Goal unonaattdd
Openhands
Limiteed reposileey understaddngg Highty staplable
SWE-Agent
Code modificotoons
Highly exploratory
Nul auildingg
Dasnd SEADNE Londenttnn
12 esbiles lnaked 30 classes loeded, 6O "anctices Inake..
Fout let cocille enskened sccounilyy on the ve deterpee
Pracoding
Now les and La REAUXE an iiffon FacdFadd spuuday
hylanch seu Ensrablation fatled.
?/
Argaurts: ('pats" /NsdePPaadd
Seagsaked toe) Bist www.rrrrryyyaapooooonooeo seperdency sean L
Albeeyiing to po
pracee prodicay' ssta...
Alinsiia fis prours lapucia big de catan of ipa tus fi.....
4/3
medidalatiinndkrrr moddde
300y
</
Ang 1t2. ('T2A pats
(
seyt ['posstiom__2 "wetadete),, (scalore).
Sko pidrap lnnclatic
tepat
Sedic to Iamly cay txploeeetrie
Hoityey Tadt M, Sue to you to fie somti doraly de ierect
Tresesack (soat recast cal: bert):
My fiis coceante: Teapeet ongeness 'erypaiene' antoeats
Fres liki peotee 28 tara) trataaal jamal.
Preceseed kappolrtta shape: (3M), 2)
4/>
kmtll cocelaatic: castlice with rea ay fooch lagie
powe powo_241:r.55 [5] luopesnts
Alerast with inqsenck.
bwikig 10 beype des ln tleugh dir
How bosic leck fact Ri- factees raaue mxrect pou dit (owetty bee loysing iopptts exedd bo
Success octract 20 crandirates using from (116, 12)
Let's check Its strvetare,
[Eroo 2] to nut file or strectory. "/ENNcee ppeetrrreteeettteeeee....
24it the file again to tantle
Chockeg
cencerycens Deesker feoel Creinng
kayonnar reselved.
This aor ctrurt corsertly...
ciract parch applaed to fetca() to
4/
13 weee sood to /wdeett-o oqe
view repactrory. stracture & crarca
aecomabrr are dora foret
(100,
lask suscessfully completed. code analysis done
failed
failed
17 turns
140 turns
20 turns
Figure 3: RepoMaster vs. OpenHands and SWE-Agent: A Comparative Case Study,
usage is significantly lower, but this reflects failure case rather than efficiency: the agent simply gives up earlier without the necessary tools to effectively explore and utilize the repository.
Further analysis of the failure modes in ablated systems reveals: Without hybrid hierarchical analysis, the agent struggles to locate key repository components, often getting lost in non-essential files; without context-aware exploration, the agent frequently explores irrelevant parts of the repository, resulting in context fragmentation and redundant exploration; without information selection, the agent's 's context window becomes cluttered with low-value information, causing it to miss important details in error messages and execution traces.
4.5
Case Study
For the case study, we evaluated Repo against OpenHands and SWE-Agent on a challenging 3D pose estimation task from GirTaskBench. As shown in Figure B neither baseline completed the task due to different failure modes. OpenHands ran extensive trial-and-error iterations 1~140 attempts, >> 10x others) and consumed higher tokens without success. SWF--gent, although quicker, lacked task- repository understanding- -reatnnnnnngttiigggggg each error as a standalone fix and defaulting to a coarse 3D pose method that strayed from the core algorithm, causing task degradation. In contrast, RepoMaster leveraged structured repository analysis to efficiently focus on key components, achieving successful task completion with fewer attempts (<20 iterations).
5 Conclusion
We introduce RepoMaster, an end-to-end autonomous agent framework designed for automating the use of code repositories to tackle complex tasks. By combining static structural analysis of the repository with autonomous exploration, RepoMaster outperforms OpenHands and SWE-Agent in two challenging benchmarks. These results demonstrate that treating open-source repositories as modular, composable tools-rather than burdens to be regenerated from scratch-forme a powerful paradigm for solving complex real-world tasks. Beyond performance gains, RepoMaster promotes a more sustainable and collaborative Al-for-code ecosystem. Its capacity to reuse and adapt existing repositories lays the groundwork for large-scal orchestration of multiple projects within a single workflow, automated propagation of bug fixes and security patches upstream, and straightforward transfer to domains that share analogous structural challenges, such as hardware description languages, robotic middleware, or data-centric notebook collections. By enabling agents to understand and integrate code in context, RepoMaster accelerates the virtuous cycle between human contributors and AI systems, fostering continual improvement across the open-source landscape.
10

---

Acknowledgements
This work was supported by the National Natural Science Foundation of China under Grant 62301559 and Shandong Provincial Natural Science Foundation (Grant No. ZR2025QC1570),
References
[1] Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Tool learning with large language models: A survey. Frontiers of Computer Science, 19(8):18343,, 2025.
[2] Zhiruo Wang, Zhoujun Cheng, Hao Zhu, Daniel Fried, and Graham Neubig. What are tools anyway? a survey from the language model perspective. arXiv preprint 15X45 2024.
[3] Zhiruo Wang, Daniel Fried, and Graham Neubig. Trove: Inducing verifiable and efficient toolboxes for solving programmatic tasks. arXiv preprint urXiv:2401.12800 2024.
[4] Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models tool makers. arXiv preprint arXiv:2305.17126. 2023.
[5] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
[6] Ziyi Ni, Yifan Li, Ning Yang, Dou Shen, Pin Lv, and Daxiang Dong. Tree-of-code: A tree- structured exploring framework for end-to-end code generation and execution in complex task handling, arXiv preprint arXhv:2412.15300 2024.
[7] Guangsheng Bao. Hongbo Zhang, Linyi Yang, Cunxiang Wang, and Yue Zhang. Llms with thain-of-thought are non-causal reasoners. arXiv preprint arX1v:2402.16048, 2024.
[8] Matthew Renze and Erhan Guven. Self-reflection in 1lm agents: Effects on problem-solving performance. arXiv preprint arXiv:2405.0666 2024.
[9] Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better Ilm agents. In Forty-first International Conference on Machine Learning, 2024.
[10] Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. arXiv preprint WxXiv22401.07339 2024.
[11] Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai's ability to replicate ai research. arXiv preprint #xXi:2504.00848, 2025.
[12] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 2021.
[13] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-lever code generation with alphacode. Science, 1992--97) 2022.
[14] John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528-5000 2024.
[15] Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagenthench: Evaluating language agents on machine learning experimentation. arXiv preprint 23000:2310.03302, 2023.
11

---

[16] Jun Shem Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents machine learning engineering. arXiv preprint arXiv:2410.07095, 2024.
[17] Bohan Lyu. Xin Cong. Heyang Yu, Pan Yang. Yujia Qin, Yining Ye, Yaxi Lu, Zhong Zhang, Yukun Yan, Yankai Lin, et al. Gitagent: facilitating autonomous agent with github by tool extension. arXiv preprint urXiv:2312.
2023.
[18] Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng. Helan Hu, Kaikai An, Ruijun Huang, et al. MI-bench: Evaluating large language models and agents for machine learning tasks on repository-level code. arXiv preprint arXiv:231 1.09835. 2023.
[19] Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, et Devbench: comprehensive benchmark for software development. CoRR, 2024.
[20] Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308 00352. 3(4):6, 2023.
[21] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, al. Autogen: Enabling next-gen Ilm applications via multi-agent conversation. arXiv preprint 200802230 2023.
[22] Xiao Yu, Lei Liu, Xing Hu, Jacky Wai Keung, Jin Liu, and Xin Xia. Where are large language models for code generation on github? arXiv preprint arXiv:2406. 19544, 2024.
[23] Carlos Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
[24] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for software developers as generalist agents. In The Thirleenth International Conference Learning Representations, 2024.
[25] Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Ja- cenko, and Yuxiang Wu. Aide: Ai-driven exploration the space of code. arXiv preprint arXiv:2502.13 13138, 2025.
[26] GitTaskBench: Anonymous github repository. lattps///nnonmouu wwwpaeomymuus..onnn..ooom.m scwince/
SitTaskBench-EE4T7 Accessed: May 2025.
[27] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
(28) Raymond Li, Yushi Yamaguchi, Sinong Liu, Amir Gal, Matthew Ju, Farhad Mireshghallah, Andrew Wang, Jiayi Gao, Wanjun Wang, Chen Lei, and et al. Starcoder: may the source be with you! CoRR, abs/2305.06161 2023.
[29] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2333.088774 2023.
[30] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- lean-Baptiste Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
[31] Anthropic. Introducing claude.
https: http:///www.atthrrp....c..m//mm/
introducing- claude 2023. Accessed: 2023-10-20.
12

---

[32] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2300. 139771, 2023.
[33] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16009, 2023.
[34] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint arXhv:2401.040
2024.
[35] Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang. Runxin Xu, Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coden vo:er Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406. 11911, 2024.
[36] Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. Repocoder: Repository- level code completion through iterative retrieval and generation. arXiv preprini arXiv:2303. 125700 2023.
[37] Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Jacob Ginesin, Edward Berman, George Chakhnashvili,, Anton Lozhkov, Carolyn Jane Anderson, et al. Can edit? evaluating the ability of large language models to follow code editing instructions. arXiv preprint arXiv:2312. 12450, 2023.
[38] Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Haotian Hui, Weichuan Liu, Zhiyuan Liu, et Debugbench: Evaluating debugging capability of large language models. arXiv preprint arXiv22401.44622 2024.
[39] Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Xuanhe Zhou, Yufei Huang, Chaojun Xiao, et al. Tool learning with foundation models. ACM Computing Surveys, 57(4): 2024.
[40] Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. Advances in neural information processing systems, 36:45870-45894, 2023.
[41] Xiao Yu, Lei Liu, Xing Hu, Jacky Wai Keung, Jin Liu, and Xin Xia. Where are large language models for code generation on github? arXiv preprint arXiv:2406. 19544, 2024.
[42] Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms. arXiv preprint arXivv2304.082444 2023.
[43] Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Qi Zhang, Tao Gui, et al. Tooleyes: fine-grained evaluation for tool learning capabilities of large language models in real-world scenarios. arXiv preprint arXiv:2401.00741 2024.
[44] Yujia Qin, Shihao Liang Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, al. Toolllm: Facilitating large language models master 16000+ real-world apis. arXiv preprint arXiv:2307. 2023.
[45] José Haro Peralta. Microservice APIs: Using Python, Flask, FastAPI, OpenAPI and More. Simon and Schuster, 2023.
[46] Model context protocol: Introduction. introduction Accessed: May 2025.
http:://ooealccneexpppttoo.... 1o/
[47] Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, et al. Repoagent: An llm-powered open-source framework for repository-Ievel code documentation generation. arXiv preprint rxXiv:2402.16660 2024.
[48] Aider: Ai pair programming in your terminal. https://aides chat/ Accessed: May 2025.
13

---

[49] Github copilot: Your ai pair programmer. cessed: May 2025.
Ac-
[50] Cursor: The ai code editor. kittps://ww// cttp://wwwwcrrrrr.ccomoo/ Accessed: May 2025.
[51] Jian Zhang. Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. A novel neural source code representation based abstract syntax tree. In 20/9 JEEE/ACM 4/st International Conference on Software Engineering (ICSE), pages 783-794, 2019.
[52] Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order the web. Technical report, Stanford infolab, 1999.
[53] OpenAL. Hello gpt-4o. fttpss///ppeaaa.commmmeee/////e 2024-05-18.
4pt-40/ 2024. Accessed:
[54] AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1, 2024.
[55] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 2Mf.:2441.11937., 2024.
[56] Ziyi Ni, Huacan Wang. Shuo Zhang, Shuo Lu, Ziyang He, Wang You, Zhenheng Tang, Yuntao Du, Bill Sun, Hongzhang Liu, Sen Hu, Ronghao Chen, Bo Li, Xin Li, Chen Hu, Binxing Jiao, Daxin Jiang, and Pin Lyu. Gittaskbench: A benchmark for code agents solving real-world tasks through code repository leveraging, 2025.
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper claims main contributions and scope in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, the paper discusses the limitations of the work.
15

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our paper mainly works on method and experiments and do include complete proof about the method.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: the paper will open source code on github for other researchers reproduce experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: the paper will provide the open source code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pbllc/guudeeee http://oeeuuiiisssoooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting//etails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, the paper used open sourced dataset and include all the details of the datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, the paper reported error bars properly.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2- 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, the paper includes all the computational resources information.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
18

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, the paper discussed the issue.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We mainly use open sourced models and commercialized model API, These models commonly have safeguards in their own license description.
19

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, our open sourced code will be licensed properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, thcode thcode. com datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We introduced a new method for the problem and will release our code on github if accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
20

---

Answer: [NA]
Justification: Our paper does experiment on existing open sourced dataset and the evaluation code of the method github.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did not crowdsourcing research experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: Our framework conducts a detailed comparison of the of different LLMs, and the performance of the method does not rely on the LLMs themselves.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non stn-standard components.
Please refer to LLM policy fctpee///merrrppcccc///oeeeeeees for what should or should not be described.
21

---

Appendix
A. Description of GirTaskBench
Although existing benchmarks focus predominantly on software engineering tasks, GitTaskBench focuses on everyday user queries drawn from domains such as image enhancement, video analysis, speech processing, physiological signal analysis, office automation, and security/privacy. tasks that are difficult complete without ready-made codebase. For every query, the benchmark provides one specific open source repository and requires the agent to solve the task by reusing and adapting that code. Task queries are presented natural language better simulate genuine user requirements.
To ensure the benchmark reflects realistic and meaningful usage scenarios, each task is paired with carefully selected open-source repository that highly relevant and capable of supporting the task. This design simulates a common real-wer world situation: given a specific user need and an existing codebase, the challenge lies in effectively applying and adapting the provided code to fulfill the requirement. By anchoring each task to concrete repository, GirTaskBench evaluates not only general prblemm-oolving skills but also the practical ability navigate, comprehend, and leverage real-world projects.
To guarantee data quality and comprehensiveness, five computer science students with relevant domain expertise manually reviewed and validated each repository included in GenThskBench. Necessary downloadable resources were supplemented, and any critical information absent from the original repositories was carefully added to ensure dataset completeness and feasibility. This rigorous validation process ensures all tasks are clearly defined, practically feasible, and accurately reflective of real-world development scenarios.
For evaluation, establish standardized multi stage scoring system to comprchensively assess the correctness and completeness solutions generated by various frameworks and L.M.s. Specifi- cally, the scoring system first evaluates technical feasibility by verifying whether the solution can be successfully executed, ensuring the completion of essential GitHub operations. Subsequently, the solutions assessed from the user perspective, examining the extent which actual user requirements have been met. Through this careful and structured design, GitTaskBench provides the research community with valuable resource for advancing the evaluation and development of intelligent agents in real-world scenarios.
More details provided in [56).
Listing 1: Example case of photo restoration task.
# Query
Restore old photo by removing scratches from it.
Input
/ttp://Deeccccaa.o///ppp/eaaaaaa Input file: http:/aaattt.///////eccccc/////s
Repository: DeScratch (located at /ttp///ttttt/eeeeeeeeeeee
jpeg
Type: Image file (JPEG format)
Description: An old photograph containing visible scratches
# Output Requirements
Output directory: /ntt://ssrrtttt//////e////ssssap
Output filename convention:
Single file: output. (extension)
Multiple files: output_01. (extension), (exteesson,,. etc.
Output type: Restored image file
Completion Criteria
1. Code execution completes without errors
2. Dutput file(s) exist in the specified directory
3. Output file(s) follow the required naming convention
4. Output file(s) are readable image files
# Success Criteria
1. Scratches effectively removed from the input image
22

---

2. Image quality is maintained improved
3. Original image content is preserved
B. Repository Search Pipeline
This section presents the condensed logs generated during the repository retrieval process. Listing document search queries, repository rankings, and selection rationales, thereby ensuring transparency and reproducibility in repository selection methodology.
Listing 2: Search for repositories related to the Kaggle APTOS 2019 blindness detection competition.
/* Task Definition Phase */
Search for GitHub repositories related Sgpto220---bilnnhsss-eeeeteeeee Define evaluation criteria for repositories. Specify JSON output format with repository ranking and recommendation.
Intent Analysis Phase
Extract core entities: GitHub repositories, apros200---IImmmeeedddeeeecio Identify keywords: "Githlu repositories", "solution approach", "performance".
Propose search queries: "aptos2019-bll" binnone---idddesssseee.eeeeeeeee maddeessddetccton GitHlub repository".
Plan search browsing strategy.
/* Searching Phase +/
Perform query: "aptos2019 hiidee-bbbinnnneeeeeeeeeee GitHub repository".
Retrieve initial repository list.
/* Brossing Phase */
browsing: ("query": "README", "url": httrep///gttu.cco/////eeet//111
Blindness Soadness-Detectionns
browsing: ("query": "README", "url": httpper//gihal.ccm/////sstttttts Dos9-Bliddess--Deeccttoon
browsing: ("query": "README", "url": httppp///gggtt.....o/////scactes
Blindness- -etteessssDtttiion)
browsing: ("query": "README", "url": http::///gggggtoooooo//////////a -19--nlindees""
browsing: ("query": "README", "url":
Analysing Phase
Analyze search results to determine if there is sufficient information to answer the question:
1.
[Solution Approach]: Utilizes machine learning models for disease detection using retina images.
[Performance]:
No specific performance metrics mentioned.
[Reason] Focuses the competition task with machine learning approach, but lacks detailed performance data.
[Solution Approach]: Aims to detect diabetic retinopathy using retina images, with a focus on automatic screening.
[Performance]: No specific performance metrics mentioned.
[Reason]: Provides comprehensive approach but lacks detailed performance data. 3. gdineshk6174/APTOS 201i--Biinness--Deeeeee
[Solution Approach] Machine learning model to identify diabetic retinopathy outamacllll..
[Performance]:
No specific performance metrics mentioned.
23

---

[ Focuses automatic identification but lacks detailed performance data. 4. filippma/KaglleAAPTGS APP.S-S-019--Linnneeeee
[Solution Approach]: Ensemble of deep learning classifiers (ReaNet33, EfficientNetB6)
trained on original and preprocessed images.
[Performance]: Achieved kappa metric of 0.904
test set.
[Reason]: Provides detailed solution approach with apecific performance metrics.
5. tahsin314/40th. Sacph__paae_______ solutionsoiiinnnooeee htt.eaaoo.o..poto21111---------. aptpo2019-blinnnee--eeee
[Solution Approach]: Information unavailable due to connection error. [Performance]: Information umavailable.
[Reason]: Unable retrieve information.
Results */
"repositories": [
"rank": 1,
"name": "111pppp/aaggl--A/00000000001111 "url": "stars": "N/A"
true, "ooluttion_aprracch" "Ensemble of deep learning classifiers (ResNett34)",
"performance": "Kappa metric of 0.904 on test set.", "reason": "Provides detailed solution approach with specific performance metrics."
"rank": 2, "name": 2atpaa/s5///PS---222---aaaassss "url": httpp://ggthhbb..commneeeeeeeee "stars": "N/A" "can_ soam__oove_tssk" true, Seoluttin_approacct "Machine learning models for disease detection using retinal images.".
1001--1indnees---- "eaddesss-etection".
"performance": "Not specified.", "reason"; "Focuses on the competition task with a machine learning approach, but lacks detailed performance metrica."
}.
"rank": 3, "name": "haaanbbtg//P0002000---laaaeeeee "url": "stars": "N/A",
"can_ solve_task"* true, "apputiionnpppoccch" "Detect diabetic retinopathy using retina images with deep learning.",
"performance": "Not specified.",
"reason": "Provides comprehensive approach but lacks detailed performance metrics."
"rank": "name": "ggiieehheett//pprrs--oAAAaaaaee "url": "nttps://githubbcomt http:h/ww......aaaaaaa////////ee -iw.te---innndss-eeeeee -lindness--Dttettiioo Detection", "stars": "N/A"
"can_soli saa.ssoolssss""""" true,
peelutton_aaproache "Machine learning model to identify diabetic retinopathy from retinal images.", "performance": "Not specified.", "reason": "Focuses automatic identification but lacks detailed performance metrics."
"rank": 5,
"name": "tahssintt4400b_ppaaee. http:pawww...outiiooaaaartoo000r b11000999111111 indness-detectioon

---

"url": http::/...aaaoooo.a...sscs.ssslt
detection",
"stars": "N/A",
"ea__oove_task" false,
esolution_approache "Information unavailable due to connection error.", "performance": "Information unavailable.", "reason": "Unable to retrieve information." }
1,
"recommendation" "The repository 'filipmm/Kaggle-APPOO AP100-------- Bliinneends is recommended provides detailed solution approach with specific performance metrics. Let me know you need further assistance or this concludes your request."
The above log shows the results of our repository selection using the methodology in Section 3.1 For each competition task in our study, we followed a similar procedure to identify the most appro- priate implementation repositories. The repositories evaluated based their documentation completeness, solution approach sophistication, and reported performance metrics when available.
C. Example of Task Prompt
Listing 3 presents the prompt for the Old Photo Restoration task, including the task requirements, the input/output formats and paths, and the repository used.
Listing 3: The prompt for old photo restoration.
Task
I need you analyze the following provided code repository and use your powerful programming abilities to complete the user 's task:
**Task Description**
<task>
Please complete the following tasks based the given repository:
## Task Description
Given an old photo with scratches, remove its scratches.
Available Repository
**Name**; DeScratch
**Path (absolute)**:
#*Understanding Guide**: ['Read README. md to understand the project's basic
functionality and usage']
## File Path
Input file patha and descriptions:
[
"path": http://///ctttt/////////c/cct/te
jpeg",
"description"; "An old photo in need of restoration"
## Dutput
**Output directory**; http////crcaca..rrprpp//aatttttt there is only file, name it "output" there are multiple files, start naming with "output_01", "output_02: etc., format as needed.
Additional Notes
25

---

**Core Objective**: Quickly understand and analyze the code repository, generate and execute the necessary code tool calls to efficiently and accurately complete the user's specified task.
Core Components**:
Saved output results must be named starting with "output" (e.g., "output.txt",
output.var"
there are multiple files, name them "output_01 , "output_02" 02", etc.
Enaure they are saved in a first-level subdirectory (e.g., "ftarget_outpuue output_path //otpput 3/output.txt"). txt""), as subsequent tasks will match the results based on this field in that subdirectory for testing task completion effectiveness.
</task>
Working Directory (where code should be run)** ** <work_ amt/DeScratcc, htte////erraachhoo/rro///////eee 01/rapo/DeeSratchh </work_dir>
Repository Address*
<repo>
http///esratadd../rre///eeeeee </repo>
D. Example of Initial Repository Content
Listing presents the initial repository context for the DeScratch @ codebase used in the GitTaskBench dataset, serving as the agent's 's launchpad for understanding DeScratch's 's functionality and usage.
Listing 4: Initial repository context for the DeScratch.
**Overview of important parts of the code base**
(a) README md
file_pathi README. md
file_content: # 0ld Photo Restoration (Official PyTorch Implementation)
The framework supports the restoration of high-resolution input.
Training code is available welcome to have try and learn the training details.
## Requirement
The code is teated on Ubuntu with Nvidia GPUs and CUDA installed. Python>=3. is required to run the code.
Installation
Clone the
repository for
## rrcket: How to use?
**Note**: CPU can be set 0 0,1,2 0,2; use -1 for CPU
For images without scratches:
python run.py -oinett_folddr gest__mage_folder_ppe \
--ooututt_folder [output_path]
httppp//gtaabbcccccccrroo/ttgtgp Photoo
Pii--Phoos--ck--tte Back- to-ffe
26

---

CPU 0
For scratched images:
python run.py -- --input_foodee folder [test_image__ hosst_inae_ffoldr_ppae \
---ouppt__foode [output_path]
G-GPU 0
--with_srratch
License
The codes and the pretrained model in this repository are under the MIT license as specified the LICENSE file. We use our labeled dataset train the scratch detection model.
README.
(b) Summary of key files
Module Function Summary**
Module Path
Function Description
run.py GUI
Main entry point, orchestrates the restoration flow interface entry point Cog interface wrapper for the predictor
Provides the graphical
predict.py Global/test.py
Global image restoration entry point Facial image enhancement testing Core model implementation
Face Bote__haanneeetttteeeeee
pix2pix#D_mmode caciiimmomepppp.p py sapping.modell.p
Domain mapping model
networks. py
Neural network architecture definitions Summary of key files-
(c) Core components
python
# predict.py
class (rog.ctorccog.Peeee Predictor)
def predict (self, image, HR-False, sithh
ppythn
# Gioball/ooeelsppix2iiii..
htw.d2pppi___mmdeel.e_1 model.pp py class Pix2PixHDModel (BaseModel): def inference(sel)) label, inst, image-None, feat=None):
Core components-
(d) Other module paths
text
Giobal//nooeell/ppppiinnmmee.ll
Global//ataa/mmlne__aaaaat dftaaae_eeeaaattforrdddoro for_s__ffrr________oo__eeeedddle haald_photss___ py
Face
dlobal/ggtionn/baaeepppi.......
Face
Cobal//eessctooo.utt////ippp
Face
cioba//itil//rssulleeer.yyy
cLobal/brain_aaapinng..p
Clobal/braain_ddmiin__....
clobal/traiin.dmain...
27

---

Other module paths
E. Detailed Experiment Execution Logs
This appendix presents the detailed execution logs for both the MLE-R and GitTaskBench experiments. These logs document the end-to-end experimental workflows, including task execution traces, error handling, repeated trials for failed steps, and verification of final outputs. The format follows top-tier conference standards, with line widths controlled for readability. All logs are presented in English.
E.1 Example Demonstration MLE-R: Cancer Detection Experiment Log
The following log outlines the execution representative task from the MLEER suite, specifically the histopathologic cancer detection task:
Listing 5: Cancer detection model training log.
User:
need you to analyze the following code repository and use your programming skills to complete the user task:
Task Description:
Kaggle Task: histopathollgiic--nne------de
Overvies:
In this competition, you must create an algorithm to identify metastatic cancer in small image patches taken from larger digital pathology scans... [Full task and data description omitted for brevity]
Working Directory:
Repository Path:
Assistant:
Thank you for the detailed information. have understood the requirements and the structure of the repository. I will create a PyTorch-based script named train_and prediccttp for training and inference.
System:
File not found:
Assistant:
The file does exist. Let's it.
System:
Error: Function edit_file found.
Assistant:
Sorry, cannot directly edit files. Let's use Python's built-in functions create and write the file.
28

---

User: [Python code to create arein_and_predict py" executed]
System:
Code output: File created content written to File successfully created.
Assistant:
Great! The file has been created. Let's s run the script to start training and prediction.
System:
exitcode: 1 (execution failed)
Code output: Traceback (most recent last):
File
line 195, in <module>
main()
File " htt.://rain.andppeeee.. aanddddddctt.pp' line 131, in main
train_dataset = CancerDataset (csv_fileetrain_dar root_dir=TRAIN_DD DIR, transform= transform)
TypeError:
argument of type 'method' is not iterable
Using device: cuda
Assistant:
I the issue: passed DataFrame instead of CSV file path to the CancerDataset class. Let's update code accordingly.
[Assistant guides through code updates using Python scripts patch the file .]
System:
Code output: File .../trrannand_ppedi... has been updated successfully.
Assistant:
Now let's s try running the script again.
System:
exitcode: 1 (execution failed)
Code output: IndentatiionEroo:: unexpected indent
Assistant:
We introduced indentation Let's fix it with another script.
System:
29

---

Code output: File .../brain_amd_prediict py has been updated and indentation fixed.
Using device: cuda
Epoch 1/8, Train Loss: 0.3771, Loss: 2913,,, Val AUC: 9455
Epoch 2/8, Train Loss: 0.2883, Val Loss: 0.2473, Val AUC: 0.9614
Epoch 3/8, Train Loss: 0.2501, Val Loss: 0. 0.2000, Val AUC: 0.9697
Epoch 4/8, Train Loss: 0.2246, Val Loss: 0.2143, Val AUC: 0.9718
Epoch 5/8, Train Loss: 0.2101, Val Loss: 0090, Val AUC: 0.9718
Epoch 6/8, Train Loss: 0.1978, Val Loss: 09122, Val AUC: 0.9771
Epoch 7/8, Train Loss: 0. 0.1863, Val Loas: 0. 2019 Val AUC: 0.9739
Epoch 8/8, Train Loss: 0.1774, Val Loss: 0. 0.212 Val AUC: 0. 9767
Training and prediction completed. Results saved in tresult_submissionn.co Result file exists and has the correct format.
Assistant:
Excellent! The script ran auccessfully, trained the model for 8 epochs,
produced a valid reault file. Validation AUC reached 0.9766. The workflow
included:
Initial file creation and coding.
Error handling and iterative code patching.
Final successful execution result verification.
E.2 Example Demonstration of GirTTasBBench: Old Photo Scratch Removal Experiment Log
Below is the detailed log for automated scratch removal task using the DeScratch codebase from the Gerch suite. This log demonstrates the full process, including environment setup, error handling, dependency management, repeated attempts, and final verification:
Listing 6: Old photo scratch removal experiment log.
User:
Task: Remove scratches from an photo using the DeScratch repository.
Input: tast28/DeScratch_011 wttp..../eeeraacc//iiigogoee http:/aaaaaaaaatttaaaattttteee/e
0ip___01__n______ee11e jpeg
Output directory: coding Saaiig..unn/aak22//eeeeeett 00l___01/.._.. httpccot///.o.otttttereleel /output_reeuult
Assistant:
1. Check input image existence.
Set up environment variables.
3. Run main script to process image.
4. Verify output.
System:
Input file exista but is an image (cannot display directly).
Assistant:
Set PYTHONPATH and prepare environment.
System:
Environment setup completed.
Assistant: Run main DeScratch processing script via subprocess.
30

---

System:
exitcode: 1 (execution failed)
Error: FileNoctoundBrror: No such file directory: './Global'
Command output: Running Stage 1: Overall restoration
Assistant:
Likely not running the correct directory. Change working directory to repo root and rerun.
System:
exitcode: 0 (execution succeeded)
Command output:
You using NL Res
wtt.hh/coo....eecteoeea.aaaaaaaa not exists yet!
not exists yet!
mat.aiiittttttta.iainntnenttttt pth not exists
yet! Running Stage Overall restoration Finish Stage ...
All the processing is done. Please check the results. Image processing completed successfully.
Output files:
Assistant:
Script ran, output directory empty. Warnings indicate missing pretrained model checkpoints.
Assistant:
Check output folder structure, input files, and checkpoint existence.
System:
Output folder structure:
output_result/
final_output/
stage_2
stage_3_face__uttuu
Lhnge___restorr_outtptt.
origin/
input_imaged
Westored_imas
No image files found in output folder.
Input folder: ['.
@0bbrratch_01_npppe .pegt ]]
Checkpoints missing.
Assistant:
README indicates pretrained models muat be downloaded. Download and required checkpoints.
31

---

System:
Checkpoints downloaded and extracted successfully. Zip files removed.
Assistant:
Rerun processing script.
System:
exitcode: 0 (execution succeeded)
Command output:
You using NL Res
Running Stage Overall restoration Finish Stage ...
All the processing is done. Please check the results. Image processing completed successfully.
Output files:
Assistant:
Still no output files. Check output folder structure and run. log for clues.
System:
Output folder structure:
output_result/
Final_outpute
stage_3_face_
[hage___restorr_outtpttt
origin/
hmputeimage/
pastored_image
Input folder: [' iipyab_checkpoinns''
11Scratch_O1__npiitie jpeg' ]
run log file not found
Assistant:
Try to directly call run.py processing steps for more debug output.
System:
exitcode: 0 (erecution succeeded)
Traceback (most recent call last):
File "maacrattch/llaaaldaeeettto py", line 17, <module> from detection_modell import networks
ModuleelooFFomddEror:: No module named 'detection_mddela sync_batchnorm!
ModuleelotFoomdErrro:: No module named 'dlib'
Output files: 0
stage_ Letee__rretttreeuuue output: ['origin', 'input_image', 'restored_imag '] stage_2 weeeeeeeienouutteeeeeee output: []
stage_ 3_ f_cfa_ooupuut.
32

---

Assistant:
Missing Python modules: detection_modelt batpslsstcnn_.cchooooeeeeee dlib, redtorss sync_ sync_ batchhorm. Install dependencies.
System:
All dependencies installed successfully.
Assistant:
Prepare input/output folders. Confirm input image exists.
System:
Files in imput folder: [' chyykbbbcccko 'DeScratch_ Saaiccho___ippppp jpeg') Input output folders prepared.
Assistant:
Rerun processing, focus on Stage
System:
exitcode: 0 (execution succeeded)
Error executing detection. ModulellotFommdrrror: No module named intection_modelll pondess.ync__aaee sync_batchnorm
batchnorm
Test output: You using NL Res
Now you are processing voScratch__1_input.. jpeg
Finish Stage
output_resull
Final_outpute
DeScratch_ 01_ttttt.nnnppppppe jpeg
[rage___restorr_outppic
origin/
DeScratch__1_inputt
suput_image//
DeScrratch_O1_inpput jpeg
restored_image/
DoScratcch_01_inppr masks/
mask/
DeScratch_O1_inptee png input/
DeScratch_O0 jpeg
Assistant:
Still missing batchnomcn..m Clone sync_ batchnorm.
and copy
System:
Synchronized BatchNorm module installed successfully.
33

---

Assistant:
Rerun process, ModuleliosFoundErroo: No module named 'easydict', Install easydict and other dependencies.
System:
Additional dependencies installed successfully.
Assistant:
Rerun processing script with extra debug output.
System:
exitcode: 0 (erecution succeeded)
Running Stage 1: Overall restoration
Copied input image ...//asks/input Created blank mask at .../aaskk/maask///sscrccch o1.input.pmi Executing detection. py --test_ppath ... ..../inuttddatasee Command output: initializing the dataloader model weights loaded
processing Divcraacch01_inuut.... Executing test.py
uad___aaitt_reetorr Command output: You are using NL + Res Now you are processing DeScratch_01_21 Siattcch_t__nnutt..___ Now you processing asSrratch_01_inppe jpeg
Copied DeScratch, _11__01_____111 01_input.. png final output directory
Copied DeScratch_O0 Biattchh____inn Si_phhh0._ninttnpe___ee
final output directory
Finish Stage
output_result//
final_output/
DeScratch_01_imput. png
DeScratch_.1_iiput: jpeg
origin/
DeScratch_0rt Descratchh.11iinnut..ppr
DeScrattch_01_ipptt.p.. jpeg
image/
DeScratch__1_iipttt png
DaScratcch_01_inutt..pp
restored_image/
DeScratch_.1__Ipprt png
DoScratthh.01.inuut....
masks/
mask/
Cettratch_01_inppiii png
Diticraacc_O1_iippiii input/
01_ png
DeScratch_ 01_ 01_input.. png DeScratch, DeScratch_O1_input jpeg
Assistant:
Processing succeeded. Compare input and output images.
System:
Comparison image saved as 'image_compariaon. png' Input image size: (559, 878) Output image size: (556, 876)
34

---

Input image format: JPEG Output image format: JPEG
Assistant:
Processing completed. Input and output images available for visual comparison. The pipeline handled errors through iterative debugging, resolved missing dependencies, and produced restored images as required.
These logs collectively demonstrate the robustness and reproducibility of our experimental framework. steps, including failures and subsequent recovery, are documented ensure transparency and facilitate future benchmarking.
F. Details of Module-Level Scoring Features
Dependency. As shown in Algorithm|] I to measure a module's centrality in the module dependency graph, we apply Personalized PageRank, emphasizing the target module with personalized vector. The algorithm iteratively computes the PageRank score using damping factor of 0.85, and the resulting score is scaled for comparability with other metrics.
Complexity. The feature calculates the code complexity of module by evaluating two factors: the frequency of branching and looping structures, and the depth of function nesting. The occurrence frequency of branching and looping structures (such if, for, while, and except) is normalized to prevent the complexity score from becoming excessively high. The function nesting depth is quantified based on the indentation level, and the resulting score scaled by a proportional factor. The final complexity score is composite of these two factors, with each factor's 's score normalized to maximum value of 1.0, providing a comprehensive and balanced measure of the module's code complexity, as described in Algorithm 2
Usage. This feature calculates the number of times other modules import the current module, well as the number times the functions and classes within the module are called, using these metrics as indicators of the module's significance.
Semantic. we conduct a semantic importance analysis based on the naming conventions modules to identify key files. Commonly recognized names such __init__. app. settings, conf igg, utils, and constants are considered to be of higher significance. In addition to this, we examine the presence of crucial semantic keywords within the code, including terms like main, core, engine, api, service, processor, factory, builder, scheduler, and conf ig. The occurrence of these keywords in module function names serves as important factor in determining the module's ' relevance and is used as a scoring criterion our analysis.
Doc. This feature calculates score based on the existence and quality of the module's docstring to assess the completeness of the module's documentation. The score is initially determined by the length of the docstring. Additionally, the feature checks for the presence of specific sections within doestring, such as descriptions parameters, return values, and examples. If first two sections are present, the score increases for each, and if examples are included, the score also increases. This approach provides a comprehensive evaluation of the documentation" 's completeness.
Git. This feature calculates whether the file is part of a Git repository or not. If the file is not in Git repository, score of 0.0 is returned. Next, it counts the number of commits associated with the file. The score is initially based on the number of commits. The feature also takes into account the time of the last commit, and recently modified files are considered more important, with the time factor accounting for 30% of the final score. The final importance score is weighted combination of number of submissions and the -ime-of-submissoon score to ensure that both the frequency of modifications and the freshness of the file are taken into account.
F1 Computational Cost of Hierarchical Repository Analysis
Our static analysis preprocessing pipeline utilizes highly optimized libraries (e.g., tree-sitter for AST parsing) and supports multi- multi-threaded execution.
35

---

For typical repositories (1-3k files): The entire preprocessing completes on average within 4.8 seconds, including:
AST construction: ~2.1 seconds (parallelized across CPU cores)
Dependency/call graph generation: 1. seconds
Core component identification: ~~0.8 seconds
For complex repositories: For repositories where components exceed the context threshold (8k tokens), we employ an LLM-based summarization approach using DeepSeek V3, which takes approximately 10 seconds. Even for complex repositories, overall preprocessing time remains within 15 seconds.
F22 Weight Configuration for Importance Scoring
The weights in our importance scoring scheme (Equation 2 are set to w; (equal weighting) for simplicity and interpretability. This configuration was determined through empirical analysis rather than exhaustive optimization. Our experiments show that while different weight combinations may affect the initial recall rate of important files, the impact on overall end-to-end task success is minimal (see Appendix G.4 for detailed sensitivity analysis). The equal weighting scheme offers a good balance between performance and simplicity, avoiding the need for task-specific tuning.
Table Token Usage by Agent Across LLMs on MLE-R (in Millions of Tokens).
Agent
GPT-4o (M)
Claude (M) Deepseek (M)
Openhands
5.26
4.73
4.93
SWE-Agent
1.29
1.51
1.48
RepoMaster
0.36
0.32
0.18
Algorithm 1 Calculate Personalized PageRank Score for Module
Input: Module dependency graph Gm. target module
2: Output: Personalized PageRank score score
3: score f- 0.0
4: if [V(G))] 0 then
5: personalizationin [n] 1.0 for n E V(mm)
6: personalization(m) + 2.0
7: pagerank +- PageRank a 0,85,persoodile persoamammmliiii
8: score + pagerank (ncerueen(ood 10
9: end if
10: return score
Algorithm 2 Analyze Code Complexity of a Module
1: Input: target module
2: Output: Code complexity score score
score <- 0.0
4: content + m.content
5: branchCount
Exe exceee) count(k; in content)
6: score +
55.00, 1.0)
7: matches count(def, in content)
8: if matches + then
9: LndentLevel max(matches)/4
10: score + score + min(indenntLevll/5.00 1.0) x 0.3
end if
12: return score
36

---

G. Detailed Experimental Results and Analysis
G.1 Detailed Results on MLE-R
To ensure transparency and facilitate reproducibility, we present the complete raw results of our experiments in this section. Tables and presentt RepoMaster's scores on all 22 MLE-R tasks when paired with GPT-4o, Claude 3.5, and Deepseek v3, respectively, reporting for each task the achieved score, relevant medal-category thresholds, and binary indicators of whether performance criteria are met, thereby enabling detailed comparisons between models and supporting further analysis of their capabilities. Additionally, Tables and 12 show the results of Openhands and SWE-Agent on MLE-R with Claude 3.5, respectively.
Table 4 presents the token usage of each agent across GPT-4o, Claude 3.5, and Deepseek v3 when handling tasks from the MI.E-R dataset. shows that Openhands consumes the most tokens (around 5M per task), SWE-Agent uses moderate amount (1.3-1.5M), and RepoMaster is by far the most efficient (0.18 1118-0.360 0.36M), with all three agents exhibiting slight reduction token usage when moving from GPT-4o Claude-3.5 to Deepseek V3; this pattern highlights how model-specific tokenization and interaction protocols affect consumption, with RepoMaster's lightweight design offering significant cost savings for large-scale MLE-R tasks.
G.2 Detailed Results on GitTaskBench
We report the detailed results(especially the number of tokens) of different frameworks with different models on GannesBee in Tables 5
G.3 Ablation Study
To more intuitively represent the role of each core component, we present the results of the Ablation Study visually, shown in Figure 4
Performance vs. Token Usage in RepoMaster
60
Code Execution Rate (%)
Task Pass Rate (%)
300k
Token Usage (K)
274k
265k
40
250k
240k
Performance
48.14
46.29
44.44
46.29
SBss Loken
30
180k
40.74
38:65
38:58
29.62
35:18
20
120k
24:07
RepoMaster (Ful)
wro Hierarchical
wio Context-aware Code Navigation
wio Information Selection
Base Code-Agent
Repository Analysis
(wlo all)
Figure 4: Performance comparison of different ablation configurations RepoMaster.
G.4 Hyperparameter Selection and Sensitivity Analysis
This section provides detailed experimental validation for the hyperparameter choices used in Repo- Master's s core component identification (Section 3.2.2) and context initialization strategies.
G.4.1 Top-k Module Selection To determine the optimal number of top-ranked modules to include in the initial context, manually constructed core file modules from several repositories our test
37

---

Table 5: Comparison of Different Frameworks on GitTaskBench.
Framework
LLM
Execution Completion Rate (%)
Task Pass Rate (%)
#Total Tokens
GPT-4o
17.58
10.19
276807
SWE-Agent
Claude 3.5
41.67
22.23
456284
DeepSeck V3
18.52
12.04
414302
GPT-4o
21,30
14.82
764521
OpenHands
Claude
53.70
40.74
2882928
DeepSeek V3
45.37
26.85
4749735
GPT-4o
48.14
40.74
250245
RepoMaster
Claude
75.92
154913
DeepSeek V3
61.11
44.44
255998
set and empirically tuned to maximize the recall important file sets. We gradually increased k values and measured recall rates. As shown in Table 6 the recall rate increases with k; but the marginal gain diminishes significantly beyond i = 20. While top-30 achieves slightly higher recall (73.3%), the 3.3% improvement
top-20 (70.0%) does not justify the additional context consumption. Therefore, we selected = 20 as it balances recall performance with context efficiency.
Table 6: Effect of top-k module selection on important file recall rate.
Top-k
Recall Rate (%)
Relative Improvement
Top-5
46.6
Top-10
60.0
+13.4%
Top-20
70.0
+10.0%
Top-30
73.3
+3.3%
The top-10 class selection threshold was determined based on the average length of each class and the total token budget allocated for classes in the initial context (approximately 8k tokens per interaction).
G.4.2 Weight Sensitivity Analysis for Importance Scoring To validate our gqual-weighting strategy (w; in Equation [ we conducted sensitivity experiments by varying the weights of six features. We selected 5 repositories with manually identified core files, tuned different weight combinations to optimize important file recall, and evaluated 10 corresponding end-to-end tasks using DeepSeek V3.
Table presents the experimental results. The first row shows the baseline with equal weights, achieving 70% recall, 60% execution completion, and 30% task pass rate. Rows 2-7 show ablation results where each feature's 's weight is increased to 5 while others remain at 1. The last three rows show optimized weight combinations that maximize recall.
Key observations:
Dependency, Usage, and Git features show greater impact recall rate for important files compared to other features. Increasing their weights (rows 2, 4, 7) improves recall by 6.67-10% over baseline.
End-to-end task completion quality is less sensitive to weight variations. Despite recall differences up to 20% (60% to 80%), task pass rates remain stable at 30-40%.
Execution time bottleneck
in file identification. End-to-end execution each task takes approximately 20 minutes, compared to only ~4.8 seconds to retrieve important files for typical repositories (1-3k files). This indicates that while entry-point file identification provides useful starting point, the effectiveness of subsequent exploration and localization strategies is far more critical for overall performance.
These findings validate our choice of equal weighting: (1) it offers competitive performance with minimal complexity, (2) initial file recall has limited impact on final task success, and (3) the exploration phase, not the initialization phase, is the primary determinant of end-to-end performance.
38

---

Table 7: Sensitivity analysis of feature weights in importance scoring. Equal weights (baseline) vs.
varied weights.
Feature Weights
Performance Metrics
Dep.
CmpIx.
Usage
Sem.
Doc
Git
Recall (%)
Exec. Comp. (%)
Task Pass (%)
70.00
60
30
1
1
76.67
60
30
1
5
1
1
66.67
50
30
1
73.33
70
30
1
I
5
1
63.33
50
30
I
5
60.00
60
30
I
1
76.67
60
30
6
2
1
76.67
70
40
2
1
80.00
70
30
6
2
1
10
76.67
60
30
Table 8: RepoMaster with GPT-4o: Raw Results.
Competition
Score
Gold Thresheld
Silver Threshold
Brenze Threshold
Median Threshold
Any
Gold
Silver
Bronze
Abose
Valid
Medal
Medal
Medal
Median
Sub.
serial cactus identifiction
L,00000
1.00000
1,00000
1.00000
0.99910
TRUE
TRUC
FALSE
RALSE
TRUE
TRUC
blindness desection
0,89501
0.93051
0.1966
0.91499
TRUE
chail hindi and camil question unswering
0.68616
0.76305
0.73984
0.33725
0,72756
DALSC
TALSE
FALSE
RALSE
DALSE
TRUC
denoiiing--iry ceaeiiinnnnttt--ii distunnnt
0.01994
0/2600
0.4517
0.03255
FALSE
FATSE
FALSE
FALSE
SALSE
FALSE
dng-hoodd--dene mooddddeiiiiiitioon
0.72540)
0.04588
FALSE
PALSE
FALSE
dogs va- cots redus. Lemels-cdtion edition
0,10353
0.03822
0U06127
0.12116
FALSE
FALSE
PALSE
PALSE
TRUE
TRUE
jigsen-toxic sene---mmeeeeeeeee comnncn--claaiillettiie ciw.ittteriieeccllllgg
0,98639
058668
0.9833
099079
PALSE
TRUE
satoppthooogic-coo cuncer dotection
0,99112
0198350
0179900
0593390
094770
TRUE
TRUE
PALSE
PALSE
TRUE
TRUE
leaf clasefication
TRUE
leaming saoney- Iab-anteemtted
0,76242
0.83563
0.83188
0.8371
0.82527
FALSE
FALSE
FALSE
RALSE
FALSE
TRUE
imsys-dhabhet-aeen
1.08268
mlsp 2013-hirds
0.91527
0 50033
0.87522
0.86572
FALSE TALSE
FALSE
RALSE
FALSE
FALSE
new-yorke weitt-ttsii-fss-pelllie
4.70087
nomad201 & preciet pocdie--rrnsppcee crasspaeers conductors
0.166692
0,05599
006229
0U06532
0.06988
FALSE PALSE
PALSE
PALSE
TRLE
TRUE
pathology 2020 fgec?
0,94049
0.97836
0.97663
0.97361
0.96352
BALSE RALSE
RALSE
BALSE
TRUE
rundem acts of---of-izzza
0
0.97988
0.76482
0.69210
0.59960
FALSE TALSE
FALSE
RALSE
TRUE
TRUE
spooky author idencification
0.16806
0.26996
0.29381
0.41879
BALSE BALSE
FALSE
BALSE
tabular phyground series dec-2021 2021
0,95711
0.96600
0.96558
0.95658
0.95342
TRUE TRUE FALSE RALSE
TRUE
TRUE
0.84015
0.93357
0.97152
0,92000
0.96750
FALSE FAISE FALSE
FALSE
FALSE
TRUE
tubula-plog playproundggrettteee. 4aay-2022
0,95056
0L59818
097268
FALSE PALSE
FALSE
test normalization challenge russian. language
0.00000
0.99012
0.98232
0.97992
0.97291
FALSE PALSE PALSE
BALSE
BALSE
TRUE
igs- wol.oalddetttcaaionnclllll
0.52210
0385918
0816655
FALSE
Table 9: RepoMaster with Claude 3.5: Raw Results.
Competition
Scure
Gold
Silver Threshold
Bronze
Median Threshold
Any Medal
Gold
Silver
Bronze
Above
Valid
Threchold
Threshold
Medal
Median
1.00000
1,00000
1 1,0000
1.00000
0.99910
TRUE
TRUI FALSE FALSE
TRUE TRUI
aptos2019 (et.://-iiddeee--eeeeeee
0.90685
0.11449
chall hindi and cami--am---a question asswering
0.67455
0.76315
0.7994
0L75725
0.72756
FALSE
PALSE
FALSE
PALSE
FALSE
TRUE
denoising ditty documents
0.01994
002609
0.45117
0.07225
BALSE
RALSE
PALSE
RALSE
BALSE
FALSE
dog- breed ideniiication
0.58526
0,00500
000539
0U0859
0.47205
FALSE
PALSE
PALSE
PALSE
PALSE
TRUE
redux kemels edition
0,06683
0.08822
005038
0.06127
0.12216
BALSE
RALSE
PALSE
RALSE
TRUE
TRUC
jigsaw toxic cooo-ccmmm----ee chweeffccciii--cclllegggg
0,98029
0.97400
0.9863
0.96519
0.98779
TRUE
TRUB
FALSE
RALSE
TRUE
TRUE
histoputhlee bettppulldlgiooccnnne Regio-canort-deteeee
0.99354
0.98350
0.7990
0.97300
099470
TRUE
TRUE
FALSE
FAISE
TRUE
TRUE
leat-clssi
2.02279
0.01536
0.10835
BALSE
RALSE
BALSE
leaming-egoney seaiinngggnee-----nnee htt.----sttteettteeeeeeeooo www.soooo----oo---oeeeeeeeeeee
0,78064
0.83583
033518
083471
082227
FALSE
PALSE
FALSE
FALSE
PALSE
TRUE
musys-dhutbet-arccn
2.26734
1002222
FALSE
PALSE
FALSE
misp 2013 birds
0
0,93527
0.90138
0373722
0.86722
FALSE FALSE
PALSE
BALSE
BALSE
TRUE
york city
prediction
2.937LL
momouf00 & mmmiiettrrmmppenn-oeeeeeee
0.06479
0.05899
0.02200
006533
0.9988
TRUE SAL
FALSE
TRUE
TRUE
TRUI
pothology 2020--fg0e fgee?
0.95388
random--ctee amndom-actt-offiiee
0.63403
0.97908
0.76482
0,69200
0.59960
FALSF FAI
FALSE
RALSE
TRUE
TRUI
0.5079%
022038
tabular playground series dec 2021
0.96227
039660
0.9658
0U5633
0.95342
TRUE TRUE FALSE
PALSE
TRLE
TRUE
clip cutheter line clustificttoon
0.91278
0.93577
0.91522
0.97900
0.96750
TALSE RALSE FALSE RALSE
BALSE
TRUD
uabula playground scries may 2022
0.99373
0.99823
0199222
0.99818
0.97268
FALSE PALSE FALSE PALSE
TRLE
TRUE
test normulization challenge rasian
0,18164
0.99012
0.96222
0.97992
0.97591
TALSE TALSE FALSE FALSE
BALSE
TRUC
0.52210
0.80897
0.7763
0.89018
0.81665
FALSE PALSE
FALSE
FALSE
FALSE
39

---

Table 10: RepoMaster with Deepseek V3: Raw Results.
Competition
Score
Gold Threshold
Silwr Threshold
Bronce
Median Threshold
Any
Gold
Silver
Brunze
Abose
Threshold
Medal
Medal
Medal
Medal
Mediam
Sall.
zerial cacres adentification
0,99988
1,00000
1,00000
0.99910
FALSE
FALSE
FALSE
BALSE
TRUE
sptecc0 1S-einnreee-Snnccion
0.87384
093051
031965
091449
0.88811
FATSE
SA
SR
SA
FALSE
TRUE
chali hindi and tamil question answering
0.31169
0.76308
0.73944
0.33225
0.72756
FALSE
FALSE
BALSE
BNLSG
BALSE
TRUE
drreksing-ditt--o ditt----du-----entt documents.
001794
0,2999
004517
007325
FALSE
FALSE
FALSE
FALSB
FALSE
FALSE
0.84673
0/00500
0,00119
004598
047306
FALSE
FALSE
FALSE
SA
FALSE
TRUE
doga-va cats redux- -eracc--oiiionnn
0.84717
003832
000000
0.06227
012216
TRUE
FALSE
TRUE
FALSE
TRUE
TRUE
0.90100
0.9850
0.7760
097380
094770
TRUIF
TRUE
FAISSR
SA
TRUE
TRUE
igua dosic cesiiiconiinnnellaalliiii chullcie
0,96374
099740
098668
099633
099079
FALSE
BALSE
FALSE
FALSE
TRUE
leaf classifiation
0.07526
0,0000
0.00791
0.01226
0.10835
PALSE
PALSE
FALSE
BALSE
TRUE
TRUE
learning apency lab sutomated essay scoring:
0.89558
0.3585
0.35188
0.3371
0
TRUE TRUE
FALSE
FNLSC
TRUE
TRUD
Imays- saass-chatbct-reee
1899199
098392
0.99277
100233
105212
FALSE
BALSE
FALSE
BALSE
BALSE
TRUE
misp 2013 kirds
0.98277
0,90088
0.87372
0.36522
FALSB
RALSE
TALSE
BALSE
FALSE
FALSE
new-yan new-yank-eiity farii----eeeeee prodiction
1064668701
2.3377
2
3.97411
FALSE
FALSE
FALSE
FALSE
FALSE
nomud 20 --ppriic-tarrgaeeee-mmeeee
0.86468
B05589
0.06220
0.06582
006988
TRUE FALSE
FALSE
TRUE
TRUE
TRUE
plan pathology 2000 -grc?
0,92683
0.72366
0.97665
0.9761
0.94852
FALSE FALSE
FALSE
FALSE
FALSE
TRUE
0.90187
0,9708
0,76422
0/69100
0.59900
FALSE
SALSE
FAISE
FALSS
FALSE
TRUE
masees-llpp- httpp-/ottteerrii----eeeeeeeeeee wew.-ii----eceeettiiiiii
0197357
0971522
0.7700
0.97500
FALSE FALSE
FALSE
FALSE
FALSE
FALSE
spooky author AGentification
0.62357
0.16506
0226966
0
0445799
FALSE BALSE
FALSE
BALSE
BALSE
TRUE
0.96084
(.95660
0.5565
0.96588
0193342
TRUE TRUE FALSE
FALSE
TRUE
TRUE
tabular playground--ddoooo scrics- 2022
0.9916]
0.99822
0199818
0197288
FALSE BALSE
FALSE FALSE
TRUE
TRUE
text normalization challenge rassian language
0.01451
0.99012
0,98232
0.91592
0.97591
FALSE FALSE FALSE
FALSE
FALSE
TRUO
0,50440
0.39087
0.87163
0.85918
031665
FALSE FALSE FALSE
FALSE
FALSE
TRUE
Table 11: Openhands with Claude 3.5: Raw Results.
Competition
Score
Gold
Silver Threshold
Bronse
Median Threshold
Any Medall
Gold
Silver
Brunze
Abuve
Valid
Threshold
Threshold
Medal
Medal
Modal
Median
Sub.
serial cactus identification
0,99951
1,00000
1.00000
0.99910
FALSE
FALSE
aptos201 9 bibiiiees--tttcciin
0,93051
0.91965
0.01449
0.88991
FAI SE FALSE
FALSE
FALSE
FALSE
FALSE
0,76305
0.73964
0.73725
0.72756
FALSE
FALSE
FALSE
SALSE
FAI
FALSE SE
denotiingg diety documents
0.01794
0.02609
0
007322
FALSE
FALSE
BALSE
FALSE
FALSE
FALSE
dog besed. idemtification
0,00050
0,00339
0
0.4208
PALSE
FALSE
FALSE
PALSE
RALSE
PALSE
ss-et redux kernels edition
0,03882
0.06038
0.06127
0.12216
RALSE
FALSE
FALSE
PALSE
PALSE
FALSE
0.98350
0.9790
0,9780
0,94770
FAISSE
FAI SF
FALSE
FALSF
SAL
FALSE SE
jigsan-trnic-e commennt comment-elassittttiinn hastrmmtelasstttiiiiiillliiaggoo
0.95740
0.9668
0.96339
0.98179
FALSE
FALSE
FALSE
FALSE
FALSE
FALSE
leaf classiacation
4,59035
0,00000
0,00791
0.01526
0.10835
FALSE
FALSE
FALSE
FALSE
FALSE
TRUE
leaming- agency-bal automated essay scoring
0,00000
0.83583
0.83188
0.63471
0
PALSE
FALSE
Imsys chatbot arena
0,98392
0.99277
100083
108212
FALSE FALSE
FALSE
FALSE
FALSE
FALSE
mlsp 2013 birds
0,93527
0.90388
0,56772
FALSE
FALSE
now-york--iity-te catt--iyyyaii-----------e cas--aar--peiiittion
13:42166
2.83377
2
2.93371
3.59741
RAI SE
FAI SF
FALSE
PALSE
FA
TRUI
0.23121
0,05589
0.06229
0,06322
FALSE
FALSE
FALSE
plan puthology 200ggyy22---eee
0,97833
0.97865
0
0194552
FALSE FALSE
PALSE
FALSE
FALSE
FALSE
random pizza
0,54052
0,97908
0.76482
0
0.59900
PALSE FALSO
FALSE
PALSE
FALSE
TRUE
mmmer-clp--cahterr line- chseiiiiioiiiiiiii
0.97357
0.97152
0.97990
0.96500
FAISE FALSE
RALSE
PALSE
FALSE
FALSE
pooky-anthe--dentn suthe--dentiiiiaiio
0:92421
0,16506
0.26993
0,29381
0.41799
FALSSE FALSE
FALSE
FALSE
FALSE
TRUE
tubular- play ground 2012
0:95837
039560
0.95658
0.95658
U.95342
TRLE TRUE PALSE
FALSE
TRUE
TRUE
rabuiar playgronnd series may 2022
0,93330
0,99823
0.99822
0
0.97268
FALSE FALSE FALSE
FALSE
FALSE
TRUE
text zormalization challenge english language
0.98232
0,97592
0.97591
FALSE
0.44300
0.80997
0.37163
0.89018
0.81665
SE
SALSE
FALSE
FALSE
FAISE
TRUIF
Table 12: SWE-Agent with Claude 3.5: Raw Results.
Competition
Score
Gold
Bronze Threshold
Median Threshold
Any
Bronze Medal
Above
Threshel
Threshold
Medal
Medal
Medal
Median
Sob.
serial cactus Jdenntiicttion
0,999117
1,0000
10001000
1.000000
0.99910
FALSE
FALSE
FALSE
PALSE
TRLE
TRUE
2019 blindness desection
0.93051
0.11655
0.91449
0.88891
FALSE FALSE
FALSE
FALSE
PALSE
FALSE
chall- hindi- uad-aaii--qusttai ausstion-atswriing
0,1149955
0.76311
0.3984
0.33225
0.72756
FALSE
FALSE
PALSE
PALSE
FALSE
denoising disty documents
0,01794
0026009
0.04517
0.07225
FALSE
FALSE
FALSE
FALSE
PALSE
FALSE
dug breed- ideniiscation
0.00550
000539
0.04988
0.47005
FALSE FALSE
FALSE
FALSE
FALSE
FALSE
0.03882
005038
0.06227
0.12316
FALSE
PAISE
FAL.SE
RALSE
FALSE
FALSE
histoputhelggie-cne wtwppt/hoioioiecene---eeeeee dotteciio
0:98350
0579900
057330
039477
FALSE PALSE
PALSE
PALSE
BALSE
PALSE
jisso-ttsiie-omee ceriii--ommmett-ee weteeeeooneeooiiiieeeeeeeliiee
0.97151
0.98740
0.9668
0.98670
0.98079
FALSE
PAISE
PALSE
RALSE
FALSE
TRUE
leat- costciissiiiatiion
2.74648
0,0000
0.00911
0B1536
0.10833
FALSE PALSE
PALSE
PALSE
BALSE
TRUE
leaming agency automated essay scoring 2
0.83583
0.33118
0383471
0.22277
BALSE PALSE
PALSE
BALSE
BALSE
BALSE
Imsys churbot-aren arena
0.98392
0:99277
100083
L00222
FALSE TALSE
FALSE
FALSE
DALSE
FALSE
mlsp-201 1--hird
0.60338
0.95277
0.90388
088372
FALSE FALSE
FAISE
FALSE
new york city tasi- fare prediction
2,83377
2
299271
3.59741
FALSE TALSE
FALSE
FALSE
DALSE
FALSE
nomodd00 6- mmwmiiettrmmmmeee--oonnnenn
0.09346
0/05899
0,6533
0
FALSE FALSE
FALSE
FALSE
0.74045
0:97356
897465
0.73661
0.94852
FALSE PALSE
FALSE
BALSE
FALSE
TRUE
random pliza
0.68057
0.979s
0.76482
0L69210
0.59900
PALSE
TRLE
TRUE
ranter clip cutheter line clusificctionn
0.91557
0.97152
0,97990
0.96750
DALSE PALSE
FALSE
RALSE
DALSE
FALSE
spooky author idencitictiion
LU8483
0.16506
0126996
0.29381
0.41879
PALSE
BALSE
TRUE
tabular playground series dec 2021
0
0.9660
0.96553
0.95658
0.95342
TRUE TRUE FALSE FALSE
TRUE
TRUD
rébular nhulin--lyygruun plyyround--erie 2012
0.99823
0.99822
0.99818
0.97268
FALSE FALSE FALSE FALSE FALSE FALSE
0.99012
0.98332
0.97592
0.97591
FALSE PALSE
FALSE
FALSE
FALSE
FALSE
tps aa-salt-deeniiieationn challonge
0.06620
0.89897
0.37163
0359118
081665
FALSE FALSE FALSE
PALSE
BALSE
TRUE
40


Reviews:
[{"review_id": "bmh4KkJtK7", "reviewer": "Reviewer_TDN5", "summary": "This paper proposes RepoMaster, an autonomous agentic framework designed to solve complex coding tasks by exploring and reusing existing GitHub repositories. Instead of generating repository-level code from scratch, the framework focuses on a more practical \"reuse and adapt\" approach. It addresses this via three-stages: (a) Repository Search: Identifies relevant GitHub repositories based on the user's task description (intent). (b) Hierarchical Repository Analysis: Before interacting with the LLM, the framework performs a static analysis of the chosen repository and constructs a Hierarchical Code Tree (HCT), a Module Dependency Graph (MDG), and a Function Call Graph (FCG) to map the codebase's structure. It then uses a scoring system to identify the most critical \"core components\" (files and classes).\n(c) Autonomous Exploration & Execution: The agent uses the pre-analyzed structural information to guide its exploration. It iteratively interacts with the codebase using specialized tools for viewing code, analyzing dependencies, and searching. To manage the LLM's context, it employs an information selection strategy that prunes code, documents, and logs to retain only the most essential information.  \n\nThe authors evaluate RepoMaster on two benchmarks: an adjusted version of MLE-Bench called MLE-R and a newly introduced benchmark named GitTaskBench. The results show that RepoMaster significantly outperforms strong baselines like OpenHands and SWE-Agent.", "questions": "See the weaknesses", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\na) The concept of performing a static, structural analysis before dynamic exploration is innovative. The creation of hybrid structural maps (HCT, MDG, FCG) provides the agent with a foundational understanding.\n\nb) The introduction of GitTaskBench is a good contribution to the research community. \n\nc) The authors conduct a robust evaluation using multiple state-of-the-art LLMs (GPT-4o, Claude 3.5, DeepSeek V3) and two strong baselines. The comprehensive ablation study effectively isolates the contribution of each of RepoMaster's core components, confirming that the performance gains are directly attributable to the proposed mechanisms. The case study is a plus. \n\nWeaknesses:\n\na) The success of the entire workflow is highly dependent on the initial \"Repository Search\" stage. This stage relies on analyzing user intent, README files, and star counts. The paper could benefit from a deeper discussion of the limitations of this search phase and how the agent might recover if an unsuitable repository is initially selected.\n\nb) The initial \"Hierarchical Repository Analysis\" involves parsing all source files to build ASTs, dependency graphs, and call graphs. While effective, the paper does not discuss the computational cost or time required for this preprocessing step.\n\nc) The module-level importance scoring aggregates six features using equal weights. This seems somewhat arbitrary. The paper would be strengthened by a sensitivity analysis on these weights or a more principled method for determining them.", "was_revised": true, "final_justification": "I have read the response and it partly addresses my questions. I'd like to keep my original score.", "rebuttal": "Hi, thank you very much for your thoughtful and encouraging review. We truly appreciate your recognition of our key contributions—\n>including our **innovative combination of static structural analysis and dynamic exploration, the creation of hybrid structural mappings (HCT, MDG, FCG), the introduction of GitTaskBench, and our robust evaluation methodology with comprehensive ablations and case studies**.\n\nWe're especially grateful for your careful attention to *detail and interest* in exploring potential extensions of our work.\n\nIn this rebuttal, we've provided detailed clarifications and comprehensive supplemental information to fully address your questions. We sincerely hope these further strengthen your confidence in the rigor, contributions, and practical value of our paper, and increase your overall satisfaction.\n\n## **To Weaknesses a):**\nThank you for your insightful observation. We acknowledge that the repository search stage is crucial to the success of our framework. To address this concern, we have implemented several mechanisms:\n\n**1. Advanced Search Strategy Based on \"Search and Think\" Paradigm**\n\nWe have developed a sophisticated deep search mechanism that employs an iterative \"search and think\" approach, rather than relying on a single search attempt. In each search iteration, the agent:\n\n- Dynamically formulates and refines search queries based on search and analysis results\n- Performs real-time relevance assessment while browsing repository content\n- Maintains a ranked list of top-k candidate repositories with confidence scores\n\nThis cognitive search process mimics how human developers explore GitHub, gradually deepening understanding through active exploration rather than passive filtering.\n\n**2. Robust Failure Recovery through Comprehensive Multi-Agent Architecture**\n\nTo handle cases of selecting unsuitable repositories, we designed a hierarchical multi-agent system with specialized recovery agents:\n\n```\nSchedule Agent (Orchestrator)\n├── DeepSearch Agent: Repository discovery and ranking\n├── Code Agent: Task execution with repository\n├── Issue Fix Agent: Analyzes GitHub issues for known problems/solutions\n├── Dependency Agent: Handles environment setup and dependency conflicts\n```\n\nWhen execution failures occur, our Schedule Agent coordinates these specialized agents.\n\n**3. Automatic Failure Recovery Mechanisms**\n\nWhen the selected repository is unsuitable, the Code Agent may encounter a greater range of issues, some of which may not be fully resolvable. To address this, our system employs an automatic Failure recovery mechanism: when the Code Agent faces execution failures (e.g., dependency conflicts, API mismatches, missing functionality), the Schedule Agent automatically:\n\n- (i) Analyzes failure patterns to generate structured feedback\n- (ii) Uses exploratory tools to inspect related files/code, extract structural information, and iteratively attempt recovery strategies\n- (iii) Adjusts selection criteria based on failure patterns (e.g., if a computer vision task fails due to a lack of GPU support, it prioritizes CPU-compatible alternatives)\n- (iv) If initial attempts fail, we implement reflection-based recovery strategies, see 4. below.\n- (v) Seamlessly switches to the next candidate repository\n\n**4. Reflection Trajectory Optimization**\n\n**4.1 Execution Trajectory Analysis:** The system analyzes past execution trajectories to identify recurring failure patterns\n\n**4.2 Optimal Path Extraction:** We extract exploration history, retaining only the most informative execution trajectories while pruning redundant or failed attempts\n\n**4.3 Context Optimization:** This ensures the LLM's limited context window focuses on high-value information in the next attempt\n\nThis adaptive multi-agent approach ensures the system can recover from various failure modes, from simple dependency issues to fundamental repository mismatches, significantly enhancing the framework's practical applicability.\n\n## **To Weaknesses b):**\nThank you for your valuable feedback. We acknowledge that the **computational cost of the preprocessing step** indeed requires more thorough elaboration. We will add comprehensive experimental data on this aspect in the revision and appendix.\n\n**1. Hierarchical Repository Analysis**\n\nOur static analysis preprocessing pipeline utilizes highly optimized libraries (e.g., **tree-sitter** for AST parsing) and supports multi-threaded execution.\n\n**For typical repositories (1-3k files):**\nThe entire preprocessing completes on average within **4.8 seconds**, including:\n\n1. **AST construction**: ~2.1 seconds (parallelized across CPU cores)\n2. **Dependency/call graph generation**: ~1.9 seconds\n3. **Core component identification**: ~0.8 seconds\n\n**2. Additional Processing for Complex Repositories**\n\nFor repositories where core components exceed the context threshold (**8k tokens**), we employ an **LLM-based summarization approach**:\n- Using Deepseek V3, this takes approximately **10 seconds**\n- Even for complex repositories, overall preprocessing time remains **within 15 seconds**\n\n## **To Weaknesses c):**\nThank you for the detailed suggestion.\n\nFirst, we want to clarify that identifying the six features themselves—based on experimental observation and domain expertise—is fundamentally more important than their precise relative weighting.\n\nOur empirical experiments demonstrated that *although weights may influence the recall rate of initially important files, this effect does not significantly impact the final end-to-end task success rate*. For this reason, we did not emphasize fine-grained weight sensitivity in our main study.\n\nWe acknowledge that, due to page constraints, we were unable to present the detailed experimental process behind our module-level importance scoring in the current manuscript. \nWe will supplement these details, including additional experimental analysis, in the appendix of future versions. \n\nBelow, we outline our experimental design and feature selection process:\n\n### **Experimental Design and Feature Selection**\n\n**1. Test Set Construction & Weight Optimization**\n- First, we manually constructed the core file modules from several repositories as our test set\n- Then we performed empirical optimization through the performance of different weight combination strategies in RepoMaster to improve the recall rate of important file sets\n\n**2. Feature Screening & Ablation**\n- Through experimental comparison, we removed scores from several evaluation dimensions with high overlap\n- We conducted simple ablation experiments on each single dimension strategy\n- Finally, we retained the six evaluation dimensions presented in the paper and gave them equal weights for both simplicity and interpretability.\n\nIn fact, this experimental process also applies to the selection and tuning of other hyperparameters used in constructing the initial context.\n\n### **Regarding Weight Sensitivity Analysis**\n\nWhile we recognize the value of a comprehensive weight sensitivity analysis, systematically quantitatively evaluating all weight combinations would require substantial experimental cost, which we considered disproportionate to the incremental benefit for our main research focus. Nevertheless, we will include further analysis in the appendix of our revised submission.\n\nAdditionally, we want to detail that in our overall algorithm framework design, the role of the evaluated **core modules** is to help the agent have a comprehensive understanding of the entire repository under limited context windows. This also **serves as the RepoMaster's autonomous exploration entry point** for the entire repository, autonomously deciding whether to expand the search and read adjacent node files from the current file.\nTask-relevant code snippets or file information are incorporated into the agent’s context only after they are identified, supporting autonomous end-to-end exploration and execution.\n\n```\nSearch → Understand → Code Generation/Editing → Execute → Debug → Convergence → Generate Verifiable Output\n```\n\nThank you again for the insightful feedback. We believe these clarifications and our planned appendix additions will further strengthen the paper.\n\n---\n\nIf you have any further questions or need additional clarification, we're more than happy to discuss them. We hope our responses further strengthen your confidence in the quality and contribution of our work! Thanks again."}, {"review_id": "lpHBXZsB2G", "reviewer": "Reviewer_aqw3", "summary": "This paper proposes RepoMaster, an automated agent framework that solves complex real-world tasks by reusing GitHub repositories. It combines structural hierarchy modeling, core component identification, and context-aware code exploration to help LLMs operate efficiently under context limits. This papers also introduce a new benchmark, GitTaskBench, and show that RepoMaster outperforms strong baselines like OpenHands and SWE-Agent on both GitTaskBench and MLE-R.", "questions": "1. The implementation filters to Python files. Would the approach extend easily to, e.g., multi-language projects (C++, Java) on GitHub?\n2. Will the authors release the GitTaskBench tasks and code for RepoMaster? \n3. How were the thresholds (e.g. top-20 modules, top-10 classes, context window sizes) chosen? Is performance sensitive to these?", "limitations": "The proposed GitTaskBench contains only 18 repositories and 54 tasks (line 230-231). While it covers diverse domains, the relatively small size may limit the generality of the conclusions. A larger benchmark would better demonstrate the robustness of the method.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. This paper presents a comprehensive static analysis pipeline that models repositories via hierarchical code trees, function-call graphs, and module dependency graphs. It further ranks modules and classes using multiple features (e.g., centrality, complexity, usage, doc quality) to identify core components, allowing the agent to focus its limited LLM context on the most relevant parts of the codebase.\n2.  RepoMaster achieves much higher success rates than OpenHands and SWE-Agent on multiple tasks and LLMs. \n3.  The authors introduce GitTaskBench, a suite of diverse tasks each tied to a concrete GitHub repo, and evaluate on multiple LLMs and agents.\n\n**Weaknesses:**\n\n1.  Many of RepoMaster’s ideas e.g.repository graphs and context selection overlap with recent work. For example, the concurrent RepoGraph [1] system also builds a repository-level graph structure to guide LLM code agents. The CGM [2] approach similarly integrates a code graph into an LLM’s attention. RepoMaster’s novelty seems to lie mostly in the engineering of combining known techniques (AST traversal, call/dependency graphs, heuristic scoring) rather than introducing fundamentally new algorithms. As a result, its contribution feels somewhat incremental relative to these related methods.\n2. The experiments only compare with OpenHands and SWE-Agent, but do not include newer repository-level methods like RepoGraph [1] or other Agents works, such as Agentless [3]. Adding such baselines would better position RepoMaster within the current literature.\n3.  The proposed GitTaskBench contains only 18 repositories and 54 tasks (line 230-231). While it covers diverse domains, the relatively small size may limit the generality of the conclusions. A larger benchmark would better demonstrate the robustness of the method.\n\n\n[1] Ouyang, S., Yu, W., Ma, K., Xiao, Z., Zhang, Z., Jia, M., ... & Yu, D. (2024). RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph. arXiv preprint arXiv:2410.14684.\n\n[2] Tao, Hongyuan, et al. \"Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks.\" arXiv preprint arXiv:2505.16901 (2025).\n\n[3] Xia, Chunqiu Steven, et al. \"Agentless: Demystifying llm-based software engineering agents.\" arXiv preprint arXiv:2407.01489 (2024).", "was_revised": true, "final_justification": "Some of my concerns have been addressed, and I'll raise my score accordingly.", "rebuttal": "## **Response to W1:**\nHi, thank you for noting the connections with RepoGraph[1] and CGM[2]. We appreciate these parallel works and look forward to potential future integration.\n\n**However, RepoMaster differs fundamentally from RepoGraph/CGM in task problem definition, method design, and technical contributions**:\n\n- RepoMaster targets real, end-to-end task completion via open-source project reuse, evaluated based on MLE-R/GitTaskBench with multi-domain, verifiable tasks. NOT just single-repo code repair (SWE-bench) as in RepoGraph/CGM. Task complexity drives distinct technical approaches.\n- Ours is application-driven: all methods serve practical use. RepoMaster's novelty lies **NOT** in code graph construction, **but in using structured understanding as a tool for context-aware exploration**.  Core contributions are *task-driven static-dynamic collaboration, autonomous exploration, information selection*, enabling a closed loop in real-world E2E tasks.\n  ```\n  Retrieve suitable repo → Understand its functionality → Configure environment → Code generation → Execute debugging → Convergence → Generate verifiable output\n  ```\n\n**Key Technical Contributions**\n\nRepoMaster is designed for reusing repositories to solve real E2E tasks—requiring both repo understanding and full task execution under context constraints:\n\n**(1) Hybrid Structural Repository Mapping**\n\nAST traversal extracts modules/classes/functions, constructs three complementary structures (HCT/FCG/MDG), and identifies critical modules as task entry points under context limits.\n\n**(2) Autonomous Exploration-Execution Closed Loop**\n\nRather than statically embedding graphs into attention, agents dynamically switch between:\n- Code viewing ↔ Dependency tracking\n- Code generation/modification ↔ Execution debugging \n\nunder a limited context.\n\nFig. 2 shows how this iterative process gradually locates and resolves practical issues like missing models and dependency errors.\n\n**(3) Context-aware Information Selection**\n\nTo address the key challenge of limited context in agent applications, we propose programmer-inspired, multi-level information selection with reduction strategies for code, docs, and feedback logs.\n\n\n**3. Significant Performance Advantages**\n\n- Execution completion: 75.92% (vs OpenHands 48.15%)\n- Task pass: 62.96% (vs 24.07%)\n- Token use reduction: 95% (154k vs 3094k)\n\nAblation experiments (Table 3) prove all components' contribution, validating the effectiveness of our algorithmic framework.\n\n```\nBesides, reference CGM[2] is a concurrent work that was made public on arXiv AFTER our submission (22 May 2025).\n```\n## **Response to W2:**\nWe fully agree that more recent, powerful comparisons could further clarify RepoMaster's positioning. \n**However, for real-world E2E tasks in our benchmarks**—which require comprehensive agent capabilities—**many frameworks can not handle such tasks well.**\n\nRepoGraph[1] is a technique that could be integrated into general agent frameworks as a module, but lacks true end-to-end task-solving capability. Agentless[3] is an earlier \"procedural framework\", NOT \"agent framework\" (per [1]), LESS powerful/general than our baselines: OpenHands and SWE-Agent (identified as agent frameworks in [1]).\n\nBaseline selection rationale:\n\n**1. SOTA Agent Frameworks**\n\nOpenHands and SWE-Agent consistently lead the SWE-bench Verified leaderboard (70.4%, 66.4%), while Agentless variants score substantially lower (only 50.8% task resolution rate), and RepoGraph is not even on the leaderboard (lacks full pipeline support).\nIn [1], Agentless + RepoGraph was ONLY 29.67% on SWE-Bench-Lite, far below our baselines.\nThus, we use OpenHands and SWE-Agent as **\"must-align\" strong baselines** for fair, representative positioning of RepoMaster.\n\n**2. End-to-End Task Capability**\n\nOur task setting requires full E2E execution on complete repositories.\n\n- RepoGraph: Primarily focuses on code repair, *unable to handle complete E2E scenarios*\n- OpenHands & SWE-agent: **Currently the most-used, daily-updated, mature general Agent systems** in the open-source community, long dominating the SWE-bench Verified, **with proven E2E capability**.\n\n## **Response to W3 & Limitations:**\nFirst, we'd like to kindly remind you that our RepoMaster evaluation covers **NOT only GitTaskBench but also MLE-R**—a revision of MLE-Bench-Lite—comprising 22 ML Kaggle tasks and 22×3 repositories. \n\n**In total, we evaluate 76 comprehensive, real-world tasks across 120 repositories.**\n\nSecond, beyond task domain diversity, **the properties of our selected full-stack repositories in GitTaskBench are also highly varied and general:**\n- Repo Files: 7–1157 (Avg. 204)\n- Intra-repo Dependencies: 33–6979 (Avg. 1242.7)\n- Intra-repo Calls: 180–40,552 (Avg. 8,651)\n- Functions: 25–4,915 (Avg. 1,274.8)\n- Lines of Code: 0.58k–351.42k (Avg. 52.63k)\n\nThese statistics demonstrate that GitTaskBench is diverse not only in task domains, but also in the repositories themselves—a critical aspect for evaluating repository-centric tasks.\n\nThird, we acknowledge your concern about the task/scale. However, **existing comparable benchmarks at this complexity and comprehensiveness level are similarly scoped**:\n- Full MLE-Bench[1]: 72 ML tasks\n- ML-Bench-A[2]: 55 (for script) + 13 (for code) repo-level ML-related tasks\n- MLAgentBench[3]: 13 ML experiments\n- SWE-Bench[4] (smaller-granularity tasks, repair program): 12 Python repos\n- M3ToolEval[5] (multi-tool, multi-turn calling): 82 tasks, 5 domains\nWhile more tasks can further enhance generality, our method’s capabilities are rigorously validated on two strong, diverse benchmarks, supporting robust and generalizable conclusions.\n- PaperBench[6]: 20 tasks/repos/papers.\n\n>[1] MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. 2024  \n[2] ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code. 2023  \n[3] MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. 2023  \n[4] SWE-bench: Can Language Models Resolve Real-World GitHub Issues? 2023  \n[5] Executable Code Actions Elicit Better LLM Agents. 2024  \n[6] PaperBench: Evaluating AI's Ability to Replicate AI Research. 2025\n\n### **To Q1:**\nThanks for raising this insightful question on multi-language extensibility.\n\nFirst, we apologize for any confusion. \nOur approach parses all file types for tree-based structure construction; we only emphasized “Python files” *simply because our benchmark repositories and tasks (ML/DL) are Python-based*.\n\n (Most mainstream repositories we found in the DL area are Python-based, especially for ML tasks.)\n\nSecond, our approach is designed to be **language-agnostic** and **readily extensible**.\nWhile we evaluated on Python due to its ML dominance, Abstract Syntax Tree (AST) parsing-based HCT/FCG/MDG construction also applies to other mainstream languages (C++, Java, JavaScript, etc).\n\nThird, we use two complementary exploration strategies:\n\n- **Graph-based Exploration**: For languages with rich structural information (e.g., Python, Java, C++)\n- **Tree-based Hierarchical Exploration**: As the primary exploration method for scripting languages (.sh) or structurally simple project files (.yaml, .json)\n\nThis dual approach ensures robust performance across different language paradigms. Based on our empirical observations, RepoMaster adaptively selects the most appropriate strategy, uses:\n- tree-based search for simpler repositories\n- graph-based analysis for complex projects with intricate dependencies\n\nWe acknowledge this is an important direction and plan to extend the evaluation to multi-language repos in future work. RepoMaster's modular design allows straightforward extension, and we anticipate similar gains across languages.\n\n### **To Q2:**\nHi, thank you for your interest. Note that GitTaskBench is already open source; its specific anonymous link has been placed in the current paper as reference [26]. However, it may have been missed as it wasn’t in a footnote. Please refer to [26]! All tasks and code are available there!\n\nRepoMaster code has also been uploaded to GitHub, and will be made public in future paper versions.\n\n### **To Q3:**\nThanks for your suggestion. Due to space limits, parameter selection and sensitivity analysis details were omitted from the main text. We agree they are important and will provide full experimental details in the appendix of future versions.\n\nOur experiments show that while hyperparameters (e.g., top-k modules/classes) may affect initial file recall, their impact on E2E task success is minimal, so we did not focus on fine-grained sensitivity.\n\n**Top-k Module Selection:**\n- Manually built core file modules of several repositories as our test set, and tuned k for the recall of important file sets.\n- gradually increased k values; top-20 recall reached 70%; larger k offered limited gains.\n\n| Top-k  | Recall Rate (%) | Relative Improvement |\n|--------|----------------|---------------------|\n| Top-5  | 46.6           | -                   |\n| Top-10 | 60.0           | +13.4%              |\n| Top-20 | 70.0           | +10.0%              |\n| Top-30 | 73.3           | +3.30%               |\nTop-20 balances recall and efficiency.\n\n**Top-k Class Selection:**\nk=10 set based on the avg. length of each class and the total token budget allocated for classes in the initial context.\n\n**Context Window:**\nEmpirically set as \"max LLM context length÷ avg execution rounds (~8k tokens)\". When context per interaction exceeds this, we perform focused information refinement. When history exceeds 2/3 max context, we extract optimal exploratory paths and retain only the most effective execution trajectories, enabling the LLM to think of better solutions for new rounds. This design reflects our empirical observation: LLM reasoning degrades once total context exceeds a threshold. Context management and trajectory optimization help maintain reasoning quality in long, complex tasks."}, {"review_id": "daSdOAJAuG", "reviewer": "Reviewer_77DM", "summary": "This paper introduces RepoMaster, a novel automated framework designed to effectively leverage code repositories for solving complex real-world tasks end-to-end. RepoMaster aims to comprehend code in a goal-oriented, human-like manner by integrating multiple advanced techniques. Specifically, it combines hybrid structural hierarchy modeling, core component identification, and context-aware code exploration with efficient information selection strategies to navigate and analyze large codebases autonomously.\n\nRepoMaster operates in three main stages: (1) Repository Search to identify relevant repositories based on user intent, (2) Hierarchical Repository Analysis to construct a detailed structural map of the code, and (3) Autonomous Exploration & Execution to dynamically interact with the repository, perform task-specific actions, and refine its understanding through iterative exploration.\n\nThe paper demonstrates the effectiveness and efficiency of RepoMaster by conducting experiments on diverse, complex tasks drawn from the MLE-R and GitTaskBench datasets. In comparison with existing frameworks such as OpenHands and SWE-Agent, RepoMaster achieves superior performance in automating complex repository exploration and execution, validating its potential to address real-world software development challenges.", "questions": "Q1. Why restrict analysis to .py files only? Many practical projects involve configurations (.yaml, .json), scripts (.sh), or compiled extensions (.cpp). Does this limit applicability?\n\nQ2. Were the weights in the importance scoring scheme manually set or learned? How are the weights (and the corresponding contribution) determined for each feature?\n\nQ3. Did the authors consider using GNNs or unsupervised learning for structural importance prediction? Are there any negative insights or learnings from such experiments?\n\nQ4. What is the fallback strategy when errors persist (e.g., missing dependencies, ambiguous import paths)?", "limitations": "I do not find an explicit discussion on the limitations of the framework. Highly encourage the authors to present a discussion on the limitations as it would help the future works to understand the challenges effectively. Answering some of the questions I have asked earlier could give you a discussion on the limitations of the framework.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The framework is inspired from human programmer behaviour to explore the unfamiliar codebases. The integration of context-aware code exploration tools with interactive execution and log filtering is a practical step for working within LLM context constraints while mimicking real developer behaviour.\n\nThe use of ASTs and graphs (HCT, FCG, MDG) provides a principled structure-aware foundation to the framework.\n\nHowever their are some weaknesses:\n\n(1) No mention of how the model adapts or generalizes to diverse languages or non-Python repositories. \n(2) The importance scoring seems entirely heuristic-based—lacking learning-based validation or ablation\n(3) How does RepoMaster deal with stale or broken dependencies found during exploration?\n(4) What are the assumptions on the quality of README or internal documentation? What happens when the README is missing or incorrect?", "was_revised": true, "final_justification": "The paper is technically solid. The authors have shown detailed evaluation of the proposed method. However, generalization of the framework beyond non-python repositories remains to be explored.", "rebuttal": "Hi, thank you sincerely for your valuable and detailed feedback. We have provided comprehensive responses to all your comments, hoping this further increases your confidence in our work and overall satisfaction.\n\n## **To W1 & Q1:**\nWe apologize for any confusion caused by our wording. In fact, our implementation is not limited to Python files—all types of files (including .yaml, .json, .sh, .cpp) are parsed for tree-based structure construction. **Our information selection strategies also apply across diverse formats**.\n\nWe only emphasized “Python files” *SIMPLY because our selected repositories are all Python-based, and the tasks themselves are in the machine learning domain* (most mainstream repositories we found in the deep learning area are Python-based, especially for ML tasks).\n\nSpecifically, our framework employs two complementary exploration strategies:\n\n- Graph-based Exploration: For languages with rich structural information (e.g., Python, Java, C++)\n- Tree-based Hierarchical Exploration: As the primary exploration method for scripting languages (.sh) or structurally simple project files (.yaml, .json)\n\nWhile we chose Python for evaluation—given its dominance in deep learning/machine learning and alignment with our benchmark tasks—our core approach is designed fundamentally language-agnostic with broad applicability.\n\nThe construction of the Hierarchical Component Tree (HCT), Function Call Graph (FCG), and Module Dependency Graph (MDG) relies on **Abstract Syntax Tree (AST) parsing, which applies to most mainstream languages, including C++, Java, JavaScript, etc.** \n\nIn principle, our methods can be directly adapted to diverse languages and non‑Python repositories.\n\nWe acknowledge this multi-language extensibility as an important direction for future work and plan to extend the evaluation scope to other-language repositories in the next version. **RepoMaster's modular design makes this extension straightforward**, and we anticipate similar performance improvements across different programming languages.\n\n## **To W2 & Q2 & Q3:**\nThe importance scoring scheme was developed through extensive empirical trials and observations. We apologize for not providing a systematic, detailed validation or ablation study in the main paper—primarily because \n- (1) our experiments found this aspect was not critical to end-to-end performance, and\n- (2) space constraints. We will include the experimental design and additional results in the appendix. The discussion will be added to the Limitations.\n\nThe weights in our importance scoring scheme were manually set, guided by domain expertise and numerous small-scale experiments, rather than by large-scale optimization or systematic ablation. We initially explored various weight combinations and adjusted them to maximize the recall of important files during identification. \n**However, comprehensive experimental validation showed that, while weights may affect the initial recall rate of important files, this has minimal impact on the overall end-to-end task success rate.** As a result, we did not prioritize fine-grained weight sensitivity in our main study, and ultimately adopted equal weighting, which achieved satisfactory recall performance while keeping the scheme simple and interpretable.\n\nOur experimental process included:\n\n**1. Test Set Construction and Weight Optimization**\n\nWe manually constructed a test set of core file modules from several repositories, and empirically optimized the weights to improve the recall of important files within RepoMaster.\n\n**2. Feature Screening and Ablation Experiments**\n\nWe systematically removed redundant or overlapping feature dimensions and conducted ablation experiments on each single feature.  This led to retaining the six features described in the paper, all assigned equal weights for both simplicity and effectiveness.\n\n### **GNNs or unsupervised learning for structural importance prediction**\nIt is a direction with great complementary potential, and we also look forward to exploring the possibilities of future technical integration. However, our current considerations are mainly from the following dimensions:\n\n**a. Interpretability:** Our feature-based approach clearly explains why certain modules are considered important, which is crucial for debugging.\n\n**b. Computational Efficiency:** Our method runs with small time overhead, completing the entire preprocessing on average within 4.8 seconds for typical repositories (1-3k files)\n\n**c. Generalization Ability:** Rule-based features have better generalization across different programming languages, including C++, Java, JavaScript, etc.\n\n\n## **To W3 & W4:**\nThank you for raising these important practical concerns. Indeed, handling stale dependencies and unreliable documentation are critical challenges for real-world repository utilization.\n\n**Handling Stale/Broken Dependencies**: RepoMaster incorporates robust error recovery mechanisms specifically designed for such scenarios. As demonstrated in our GitTaskBench experiments (Section E.2), when encountering missing dependencies (e.g., \"ModuleNotFoundError\"), RepoMaster:\n1. Automatically identifies and installs missing packages through iterative error analysis;\n2. Adapts execution strategies when dependencies fail (e.g., switching from GPU to CPU versions);\n3. Leverages our hierarchical repository analysis to find alternative implementations when primary paths fail.\n\n**README Quality Assumptions**: While READMEs provide valuable initial guidance, **RepoMaster is explicitly designed NOT to rely solely on documentation**.\nAs highlighted in the Introduction (Lines 39–43), encountering missing or unreliable documentation is a common challenge. Our hierarchical repository analysis (Section 3.2) builds three complementary structural representations (HCT, FCG, MDG) that provide ground-truth understanding and prepare for exploratory tools, **all independently of documentation quality**. Ultimately, it is the combination of task understanding and **effective, efficient repository exploration that matters most**. This allows RepoMaster to significantly outperform baseline methods that depend primarily on README instructions or attempt to process the entire codebase, which quickly exhausts context windows.\n\n**Missing/Incorrect README**: When documentation is absent or misleading, RepoMaster's exploration tools (Section 3.3.1) enable autonomous discovery through:\n1. Structural analysis to identify entry points and core components;\n2. Dependency tracing to understand component relationships;\n3. Iterative execution with feedback-based learning.\nThe case study in Fig. 3 and Fig. 2 illustrates this capability: despite incomplete documentation, RepoMaster successfully completed the task through structural understanding and adaptive exploration, while baselines failed.\n\n## **To Q4:**\nThank you for raising this important question about error handling strategies. Our RepoMaster framework implements a systematic multi-layer fallback mechanism that can effectively handle persistent errors:\n\n**1. Robust Failure Recovery through Comprehensive Multi-Agent Architecture**\n\nTo handle cases of missing dependencies, our designed **hierarchical multi-agent system** was equipped with specialized recovery agents:\n\n```\nSchedule Agent (Orchestrator)\n├── DeepSearch Agent: Repository discovery\n├── Code Agent: Task exploration and execution with repository\n├── Issue Fix Agent: Analyzes GitHub issues for known problems/solutions\n├── Dependency Agent: Handles environment setup and dependency conflicts\n```\n\nWhen the Code Agent encounters execution failures—such as dependency conflicts, ambiguous import paths, or missing functionality—the Schedule Agent coordinates the following recovery workflow:\n\n- (i) Analyzes failure patterns to generate structured feedback\n- (ii) Uses exploratory tools to inspect related files/code, extract structural information, and iteratively attempt recovery strategies\n- (iii) Adjusts criteria based on the specific failure patterns (e.g., if a computer vision task fails due to a lack of GPU support, it prioritizes CPU-compatible alternatives)\n- (iv) If initial attempts fail, we implement **reflection**-based recovery strategies, see step 2 below.\n- (v) Graceful Termination: If errors persist after up to 5 attempts (configurable), the system safely ends the current task, logs the unresolved issue, and continues to the next, ensuring pipeline robustness.\n\nAll recovery steps are documented for future improvement. This adaptive multi-agent fallback both maximizes error recovery and enhances overall reliability and practical applicability.\n\n**2. Reflection Trajectory Optimization**\n\n**2.1 Execution Trajectory Analysis:** The system analyzes past execution trajectories to identify recurring failure patterns\n\n**2.2 Optimal Path Extraction:** We extract exploration history, retaining only the most informative execution trajectories while pruning redundant or failed attempts\n\n**2.3 Context Optimization:** This ensures the LLM's limited context window focuses on high-value information in the next attempt\n\n**3. Empirical Effectiveness**\n\nAs demonstrated in our **case study (Fig. 3)**, this strategy is crucial for complex tasks. For example, in the 3D pose estimation task:\n- **RepoMaster**: Successfully recovered from multiple dependency errors through intelligent backtracking\n- **Baseline methods**: Either failed completely or exhausted resources in futile attempts (OpenHands, ~140 iterations)\n\nThis multi-layer approach ensures **RepoMaster** maintains both robustness and efficiency: Achieving our reported **62.96%** task success rate while using **95%** fewer tokens than baselines.\n\n## **To Limitation**\nWe fully agree that an explicit discussion of the framework’s limitations would be helpful. Thanks for your valuable suggestion. We will include a detailed discussion addressing all your earlier questions in the appendix of the next version."}, {"review_id": "lxlsFcd8JJ", "reviewer": "Reviewer_LTVq", "summary": "This paper proposes a novel agent, RepoMaster, that is used to solve code tasks like the ones found in MLE-Bench. RepoMaster is able to search through many GitHub repositories to use published code in its solutions. It also incorporates clever strategies to help the LLM navigate code repositories, thereby improving its performance and saving tokens. Additionally, the paper proposes a new benchmark, GitTaskBench, comprised of tasks including super resolution, style transfer, source separation and more.", "questions": "1. How was the subset use for MLE-R chosen? Why was a subset taken and not the entire MLE-Bench?\n2. As for other baselines, why wasn't AIDE [1] used? It holds the current state of the art on MLE-Bench.\n3. I'm interested in the tasks in GitTaskBench, and would like to know more. For example, in some of the tasks, it seems like there could be several correct answers (image coloring, style transfer). How are correct solutions determined?\n4. In terms of GitTaskBench -- how would the results change if the underlying models were changed to o4-mini or Gemini 2.5 Pro? These are competitive cost-wise with the models used in the paper, and would probably yield  scores of 70-75% on GitTaskBench, given that the current state of the art gets nearly 63%. In this case, GitTaskBench would be very close to being solved, already on release. How would you address this? I really like the premise of GitTaskBench, but practically speaking, it seems close to being solved.\n\n\n[1] Jiang, Zhengyao, et al. \"Aide: Ai-driven exploration in the space of code.\" arXiv preprint arXiv:2502.13138 (2025).", "limitations": "Though limitations weren't explicitly discussed, I do not think they're of particular relevance here.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nI think that the premise of the paper, i.e., how can we leverage published code to solve general machine learning tasks, is interesting. I also think that efficient agents, like RepoMaster, are of interest to the community. \nThe components of RepoMaster, Hybrid Hierarchical Analysis, Code Exploration, and Information Selection, make a lot of sense, and they seem to work in practice, both by improving performance and saving tokens. \nThe paper is clear and well-structured.\n\nWeaknesses: \n1. How was the subset use for MLE-R chosen? Why was a subset taken and not the entire MLE-Bench?\n2. As for other baselines, why wasn't AIDE [1] used? It holds the current state of the art on MLE-Bench. \n3. I would have liked to see a more comprehensive description of GitTaskBench in the paper. Looking through the Appendix/repo, I found the tasks to be interesting, and I think a comprehensive description in the paper is warranted. I'm interested in the tasks in GitTaskBench, and would like to know more. For example, in some of the tasks, it seems like there could be several correct answers (image coloring, style transfer). How are correct solutions determined?\n\nLess significant, but still noteworthy:\n\nIn general I like the style of the figures, but I think they're too information-dense. They would be easier to understand if they were simpler in my opinion. \n\nThe descriptions sometimes lack detail. In Table 3's description, it should specify that the ablation study was done on RepoMaster + 4o on GitTaskBench.\n\nI think that seeing RepoMaster results on SWE-Bench would be very interesting. Many of the repos that make up SWE-Bench contain many files and lines of code, and it seems like an agent that uses Hybrid Hierarchical Analysis + Code Exploration could achieve high performance.\n\n\n[1] Jiang, Zhengyao, et al. \"Aide: Ai-driven exploration in the space of code.\" arXiv preprint arXiv:2502.13138 (2025).", "was_revised": true, "final_justification": "Initially, I felt like there may have been some missing baselines, and that the benchmark was perhaps too easy. After the authors' detailed response, in which they ran more SOTA language models, I'm more convinced that the benchmark is challenging and progress on it would be useful for real world coding. Though revisions cannot be posted, if the rebuttal is included in the final paper, and the minor weaknesses are fixed, I would find this to be a well rounded paper. I therefore raise my score to 5.", "rebuttal": "Thank you sincerely for your supportive and insightful comments. We hope the responses below fully address all of your questions and clarify any uncertainties. \n\n## **To W1 & Q1:**\nThank you for raising this important question.  Our rationale and method for MLE-R subset selection are as follows:\n\n**1. Computational Considerations**\n\nThrough preliminary testing, we found that each task requires **10+ hours** (including repo search, structural analysis, codegen, and multiple iterative debugging sessions). Full evaluation across **3 different LLMs** and **3 frameworks** across all 72 MLE-Bench tasks would require **~6,480 hours** and substantial API costs. Thus, we selected the smaller **MLE-Bench-lite (22 tasks), also officially provided by OpenAI**, as our test set.\n\n**2. Adopting Standardized Subset**\n\nMLE-Bench-lite is a representative, domain-diverse subset, ensuring standardized and comparable evaluation in the ML engineering area.\n\nDuring use, 2 tasks (detecting-insults-in-social-commentary, the-icml-2013-whale-challenge-right-whale-redux) were excluded due to Kaggle data access issues. To maintain evaluation completeness, we replaced them with tasks (chaii-hindi-and-tamil-question-answering, tgs-salt-identification-challenge), which are comparable in domain coverage and difficulty, from MLE-Bench to maintain 22 total.\n\n**3. Representativeness Guarantee**\n\nDespite being a subset, our MLE-R still covers diverse ML domains (CV, NLP, time series, etc.), comprehensively evaluating agent repository utilization capabilities. The specific competition IDs have been listed in Appendix Tables 6-10.\n\nWe believe this choice balances feasibility and coverage while remaining comparable to prior work. In future work, we are willing to extend to more tasks for evaluation.\n\n## **To W2 & Q2:**\nThank you for the question regarding AIDE. First, note that the task requirements in MLE-R differ from MLE-Bench-Lite (reuse the repo, NOT generate all the code from scratch). While AIDE excels on the original MLE-Bench, its exclusion from our baselines was deliberate:\n\n**1. Fundamental Task Scope Differences**\nAIDE is tailored for **scratch code generation** for Kaggle competitions with highly customized workflows to meet the specific requirements of Kaggle tasks. \nIn contrast, RepoMaster focuses on reusing open-source repositories to solve real-world tasks E2E, covering search, understanding, code editing/generation, execution, and debugging in a constrained context.\n\n**2. Architectural Limitations**\n\n**AIDE** lacks the essential functions for solving tasks E2E for complex code projects, including:\n- **a)** Structural analysis of existing codebases\n- **b)** Cross-file dependency tracking and navigation\n- **c)** Selective code modification and generation rather than pure code generation\n\n**3. Baseline Selection Rationale**\n\nSWE-Agent, OpenHands, and our RepoMaster are all general-purpose code agent frameworks, enabling fair comparison.\n\nNotably, OpenHands and SWE-Agent consistently lead the SWE-bench leaderboard on Verified, making them our \"must-align\" strong baselines to ensure fair and representative positioning of RepoMaster.\n\n**4. Supplementary Experimental Results**\n\nFinally, to address your concerns about comparability,  we conducted additional experiments with AIDE using DeepSeek V3 on the same 22 tasks on **MLE-bench** (instead of MLE-R, since AIDE  is not designed for such tasks as noted above):\n\n- Valid Submissions: RepoLearner 86.36% vs AIDE 63.64%\n- Gold Medals: RepoLearner 13.64% vs AIDE 4.55%\n\nOur advantage is obvious, and we will add these explanations to the paper.\n\n**Experimental comparison results:**\n\n| Model | Make Submission | Valid Sub | Median | Bronze | Silver | Gold | Total |\n|-------|------------|-----------|---------|--------|--------|------|-------|\n| RepoMaster | 95.45% | 86.36% | 36.36% | 4.55% | 4.55% | 13.64% | 22.73% |\n| AIDE | 63.64% | 63.64% | 18.18% | 0% | 0% | 4.55% | 4.55% |\n\n## **To W3 & Q3:**\nHi, thank you for your interest in GitTaskBench. But we'd like to kindly remind you that our GitTaskBench is already open source and has been included in the submitted paper as reference [26].\nPlease see reference [26]: GitTaskBench: Anonymous GitHub repository. All tasks and code are available in the anonymous GitHub link.\n\nAdditionally, we are happy that you want to learn more. We can provide you with our evaluation methods for **execution completion (EC) (finish)** and **task pass (TP) (correct)** for these tasks:\n\n**EC** measures the proportion of tasks where the agent successfully executes the target repo and generates valid, non-empty outputs in the required format (e.g., .jpg or .png for image processing tasks).\n\n**TP** directly quantifies the output quality. It is determined by formulating test functions and defining task-specific success criteria using established metrics, drawing on standards recognized within the domain developer community. TP requires the agent's outputs to satisfy predefined quality standards, such as functional correctness, result completeness, or achieving specific task objectives.\n\n**Task-specific success criteria examples:**\n- Image coloring: CIEDE2000 colour difference ≤ 2.0 and NIQE ≤ 7.0 on the output image\n- Video Style Transfer: average SSIM between input and output frames is ≥ 0.7, FID ≤ 400.\n- Speech enhancement: PESQ ≥ 2.0 and a SNR ≥ 15 dB\n\nTasks not meeting these thresholds are marked as failures.\n\nDue to space limits and our focus on framework design, we sincerely apologize for not adding a \"comprehensive description\" in this version. However, we promise that in the revised version of this paper, we will add this content to the appendix.\n\nThank you again for your interest!\n\n## **To Else in Weakness:**\n> About the figs \n\nThank you for your suggestion and appreciation. Due to space limits, we combined multiple figures and compressed their layout. We fully agree that expanding the figure area would help. If accepted, we will present clearer, more spacious figures.\n\n> About the table caption\n\nThank you for noting this. Currently, these experimental details are only explained in Section 4.4. We agree that Table 3 should specify that the ablation was on RepoMaster + 4o with GitTaskBench. We will clarify this in the next version.\n\n> About the results on SWE-Bench\n\nThank you for your suggestion and interest in our work. Your perspective on testing RepoMaster on SWE-Bench is very insightful. We agree that SWE-Bench's large, complex codebases are ideal for evaluating our **Hybrid Hierarchical Analysis + Code Exploration** method.\n\nWe have added SWE-Bench validation to our near-term roadmap and will submit our results upon completion.\nOngoing work also includes training RepoMaster with reinforcement learning to further enhance its performance on complex repositories. Experiments are underway, and we look forward to reporting more progress soon.\n\n## **To Q4:**\nThank you for thoughtfully considering GitTaskBench's scalability. The Claude series is widely recognized as a leading closed-source LLM family for code. Following your suggestion, we ran additional experiments with Gemini-2.5-pro (stable).\n\nInterestingly, on the more comprehensive and practical GitTaskBench benchmark, Gemini-2.5-pro achieved lower results than Claude 3.5 in our experiments. This suggests that Claude 3.5may already be approaching the current performance upper bound on GitTaskBench.\n\n| Model | Execution Completion Rate (%)| Task Pass Rate (%)| \n|-------|------------|-----------|\n| **RepoMaster + Gemini-2.5-Pro** | **72.22%** | **55.56%** |\n| **RepoMaster + Claude 3.5** | **75.92%** | **62.96%** | \n\nWe have further evaluated the potential impact of newer models:\n\n**1. Long-tail Distribution Characteristics of Task Difficulty**\n\n**GitTaskBench** reflects real-world task difficulty distribution: from simple PDF parsing to complex VideoPose3D pose estimation, **with exponential growth in difficulty**. Even if the overall completion rate reaches 70%, the remaining 30% tasks still require deep code understanding, complex dependency management, and end-to-end task resolution capabilities. This long-tail distribution ensures the benchmark's continued relevance, where **70% is merely a passing grade**.\n\n**2. Diminishing Returns of Performance Improvements**\n\nOur experimental analysis reveals a key insight. When we upgraded the model from GPT-4o to Claude 3.5:\n- Task pass rate improved from 40.74% to 62.96%\n\nHowever, in-depth analysis of this 22% improvement revealed:\n- **>50%** of the improvement came from increased success rates in environment configuration and dependency installation\n- **<20%** of the improvement came from enhanced core codebase exploration and task execution capabilities\n\nThis indicates that models still have enormous room for improvement in handling autonomous exploration and end-to-end execution of complex codebases.\n\n**3. Critical Role of Algorithm Design**\n\n**Using the same Claude 3.5 model**, different frameworks showed dramatic performance differences:\n\n| Framework | Task Pass Rate (%) | Token Consumption |\n|-----------|---------------|-------------------|\n| **RepoMaster** | **62.96%** | **154k tokens** |\n| **OpenHands** | **24.07%** | **3094k tokens** |\n\nSignificant token efficiency difference and performance gap demonstrate that even as underlying model capabilities improve, **Agent algorithm framework design and efficiency constraints remain crucial**.\n\nReferencing **SWE-Bench**: Gemini 2.5 Pro achieved **63.8%**, Claude4 approaches **80%**, indicating pure code repair tasks are being rapidly solved. \n\nInstead, **GitTaskBench**, by requiring agents to handle complete codebase understanding, dependency management, error diagnosis, and end-to-end task resolution workflow, provides more comprehensive evaluation dimensions. This represents the **core optimization direction for code agents in the next phase**, and we will regularly introduce new tasks to maintain the challenge level."}]

Author rebuttal:

Meta-review: The paper presents RepoMaster, an approach to identifying and reusing existing code repositories when solving complex coding tasks. It includes three stages: (1) searching for relevant GitHub repositories, (2) multiple forms of static analysis on the selected repository, and (3) an LLM agent to autonomously explore the repository using tools. The paper also introduces a new benchmark GitTaskBench. Experiments show that RepoMaster outperforms SOTA baselines (OpenHands, SWE-Agent) on GitTaskBench and (existing) MLE-R benchmarks.

Reviewers agreed on the paper's strengths:

* Reviewers appreciated the use of static analysis prior to more expensive LLM-based exploration.
* GitTaskBench is a nice artifact.
* The experimental results are quite strong.

Reviewers mentioned some weaknesses/questions, most of which appear resolved after rebuttals and discussions, but perhaps the most important remaining weakness is that the _evaluated benchmark tasks are in the machine learning domain, and thus the selected repositories are all Python-based_. It is _plausible_ that the impressive results would extend beyond this domain and language, but no concrete evidence is provided. Nevertheless, the presented experimental results are sufficiently strong on their own.

All reviewers are unanimous in recommending to **accept** this paper.

Furthermore, I am recommending **spotlight** primarily due to the very impressive results, outperforming OpenHands and SWE-Agent which lead the SWE-bench leaderboard, with multiple LLMs, on two benchmarks (GitTaskBench and MLE-R).
Decision: Accept (spotlight)