HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale

Authors: Phan Nhật Huy, Tien N Nguyen, Nghi D. Q. Bui
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Multi-Agent, Large Language Model, Software Engineering

Abstract:
Large Language Models (LLMs) have revolutionized software engineering (SE), demonstrating remarkable capabilities in various coding tasks. While recent efforts have produced autonomous software
agents based on LLMs for end-to-end development tasks, these systems are typically designed for
specific SE tasks. We introduce HyperAgent , a novel generalist multi-agent system designed to
address a wide spectrum of SE tasks across different programming languages by mimicking human
developers’ workflows. Comprising four specialized agents—Planner, Navigator, Code Editor, and Executor—HyperAgent manages the full lifecycle of SE tasks, from initial conception to final verification.
Through extensive evaluations, HyperAgent achieves state-of-the-art performance across diverse
SE tasks: it attains a 26.00% success rate on SWE-Bench-Lite and 33.00% on SWE-Bench-Verified for
GitHub issue resolution, surpassing existing methods. Furthermore, HyperAgent demonstrates
superior performance in code generation at repository scale (RepoExec), and in fault localization and
program repair (Defects4J), often outperforming specialized systems. This work represents a significant
advancement towards versatile, autonomous agents capable of handling complex, multi-step SE tasks
across various domains and languages, potentially transforming AI-assisted software development
practices.

Decision: reject
Reviewer scores: 6, 5, 5, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730679349297

thread_type:
official_review

<thread_object>
creator: reviewer_gpre

<content>
summary:
The paper focusses on the recently increasingly popular area of developing autonomous software agents for coding tasks. In particular, the paper focusses on the development of a software agent for generalizability across different tasks (github issue, repair etc) and languages (python, java). To this end, the paper proposes Hyper-Agent which uses four specialized agents - planner, navigator, code editor and finally code executor for scaling to generic SWE tasks. The planner agent acts as the main agent and is responsible to collaborating with multiple specialized agent in an asynchronous manner, in order to perform the final task. The specialized agents improve subtask execution performance while the asynchronous queue based execution improves throughput efficiency. Quantitative evaluations for three SWE benchmarks are provided in order to prove the efficacy of the proposed approach.

strengths:
* The paper focusses on an important problem of developing generalist agents without being tailored for specific SWE or coding tasks (e.g., competition coding etc)
* The combination of subagents with asynchronous message queues for improving subtask efficiency and parallel execution (e.g., parallel editing across multiple files) is also interesting and novel
* Quantitative evaluations across multiple benchmarks are provided to demonstrate the generalizability across different coding tasks and programming languages.

weaknesses:
While the key idea and problem is interesting, I have some questions regarding some experimental results and details,
* The idea of using the same base-LLM to act as different subagents has been previously explored before [1,2]
* How are the subagents specialized? Is the difference primarily form using different input contexts / system prompts, tools etc.
* The paper claims that the final agent is generalist and applicable across diverse tasks. However, its not clear if the context and implementation for each task is specialized or not? 
      - For instance, when solving the swebench like tasks, the original swe-agent implementation provides task specific instructions / tips (e.g., reproducing the issue) in the input context. Is a similar strategy also applied for  hyper-agent?
* Also regarding the experimental evaluations, are the baselines and the proposed approach using the same base-llm (e.g, Tab. 1)?
    - In particular, according to Tab. 1, 7: the best performing hyperagent configuration using claude-3-sonnet, however some of the baseline results are reported with gpt4o.
* Also the numbers with some of the baselines seem off: autocoderover with gpt4o performs 30.67% on swebench-lite and 38.40% on swebench-verified (both higher than hyperagent). However, the reported numbers for the same in Tab. 1 seem to be much lower.
* Finally, while a minor point, I had some concerns regarding the usability in terms of avg. cost which is for instance 1.82 on swebench for the best hyperagent configuration. In contrast, a simple 3 stage model (localization, repair, rerank) with agentless achieves $0.34 per instance on average while achieving better solve-rate. 
* How are the ablation studies performed in Sec. 6? For instance, w/o navigator which agent is used for code navigation the planner agent or any other subagent.
* Are there some incontext examples teaching which subagent should be called by the planner agent? If not, is it mostly reliant on the functon calling capability of the main planner agent?

References:

[1] Meta{GPT}: Meta Programming for A Multi-Agent Collaborative Framework, Hong et al, 2024

[2] OpenDevin: An Open Platform for AI Software Developers as Generalist Agents, Wang et al, 2024

questions:
Please see weakness section above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714777730

thread_type:
official_review

<thread_object>
creator: reviewer_hbbp

<content>
summary:
This paper proposes a multi-agent framework to address general issues in software engineering development. The multi-agent system is designed with four components: an agent for planning implementation, an agent for retrieving file content within the project, an agent for making code modifications, and an agent for executing unit tests. The overall process control of these agents is managed by a planner, and corresponding APIs are defined for each agent to interact with the project. Experiments were conducted on three datasets: SWE-bench, RepoExec, and Defects4J. However, the final experimental results did not demonstrate that the proposed method is significantly stronger than other baseline methods.

strengths:
1. This paper designs a multi-agent system with four components: an agent for planning implementation, an agent for retrieving file content within the project, an agent for making code modifications, and an agent for executing unit tests.
2. The overall process control of these agents is managed by a planner, and corresponding APIs are defined for each agent to interact with the project.
3. This paper conducts relatively solid experiments on three datasets.

weaknesses:
1. Compared to previous work, such as ACR and SWE-Agent, the innovation in this paper is primarily the addition of an Execution Agent component; the other agents are almost identical to those in previous work.
2. The HypeAgent proposed in this paper does not show significant improvements over previous methods in the final experimental results.
3. The HypeAgent proposed in this paper does not compare its fault localization performance against SWE-Agent.

questions:
1. For the executor, how does it find appropriate unit tests to execute? If defect fixing requires introducing unit tests that do not currently exist in the project, how does this agent function in such scenarios? This is a major shortcoming of the paper, and it lacks a detailed analysis in this area.
2. In Section 5.1, how does HypeAgent perform compared to other models on SWE-agent?
3. In Section 5.3, how do other open-source agents in Section 5.1 perform on this dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730742899701

thread_type:
official_review

<thread_object>
creator: reviewer_b9au

<content>
summary:
The paper introduces HyperAgent, a multi-agent system consisting of 4 primary agents (Planner, Navigator, Code Editor and Executor) organised in a Star-topology, to achieve generalizable software engineering capabilities across languages (Java and Python) and tasks (Issue-PR resolution over SWE-Bench, Bug localization and Repair over Defects4J, and repository-level code generation over RepoExec.). The planner agent creates subtasks and issues messages over async queues to each of the other 3 agents, which then try to execute the subtask. An LLM summarizer condenses the child agent trajectory to be sent back to the planner.

The different agents in the system can be instantiated with different LLMs, providing the ability to achieve a cost/capability tradeoff depending on the difficulty of the task.

The authors present results on 3 different software engineering tasks (end-to-end issue resolution, bug localization, bug repair, code generation) to demonstrate the generalizability of the proposed system.

strengths:
The paper demonstrates a generalized software engineering agent. As such, generalizability of the approach, and strong demonstration (even if not state-of-the-art on individual benchmarks) across 3 different tasks is the main strength of the paper. The proposed multi-agent breakdown is simple, while achieving good performance, with ablations supporting the need for each component as well. 

The paper also demonstrates the usefulness of multi-LLM multi-agent system through the HyperAgent-Lite configurations.

The application of dynamic message queues for asynchronous communication between the planner and child agents allows parallelizability and reduction in runtimes.

Further, the authors also identify the difficulty faced by LLM agents in using various IDE provided tools (like go_to_definition, which for example, actually reduced the resolve rate), and came up with augmentations to improve the tool (w/ search ablation in go_to_definition, etc.), which is instructive. This is particularly insightful, and points to further research in the direction of building better abstractions around existing code-tools that will be more suitable for LLMs.

The paper is written clearly, and easy-to-understand.

weaknesses:
I have two primary concerns with the paper:
- *Effectiveness of multi-agent setup*: The proposed system drives the complete workflow with an LLM agent (the planner), which is in direct contrast to the proposal by Agentless, which achieves better results than the best HyperAgent configuration on SWEBench-Verified, while also being much more cost effective (\\$0.34 for Agentless vs \\$1.82 for HyperAgent). This calls into question whether a comparatively complex agentic system is required for the software engineering task.
- *Evaluation baseline for Defects4J* - I would suggest the authors compare HyperAgent to ChatRepair [1], which reports 114 solved cases on D4J v1.2. I seek to understand how HyperAgent compares to ChatRepair and if the proposed multi-agent approach offers benefits above the ChatRepair. 

[1] Xia, C. S., & Zhang, L. (2023). Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. arXiv preprint arXiv:2304.00385.

questions:
- Can the authors compare HyperAgent to Agentless (on SWEBench-Verified) and ChatRepair (on Defects4J)? Specifically, could you include average runtimes for Agentless and SWEAgent on SWEBench?
- For both RepoExec and Defects4J, can the authors include the results for at least one of the Full configurations? This will help with understanding the scalability of the approach to better models.

Minor clarifications which I believe could enhance the readability of the paper:

> For example, multiple Navigator instances can explore different parts of a large codebase in parallel, the Editor can apply changes across multiple files simultaneously, and the Executor can run tests concurrently, accelerating validation.

- Can the authors clarify if there are specific design decisions made to ensure synchronization between agents preventing them from modifying the same files?

> Failed tasks are requeued for reliability
> HYPERAGENT also has a problem of early exit (due to hallucination that the task has been
solved)

- As pointed out by the authors, it could be challenging for the LLMs to determine success/failure of task/subtasks. Can you clarify how are failures detected for requeuing?

- What are the details of HyperAgent-lite-3?

- Explored the sharing of previous context with a specific agent?

- Can the authors clarify if they intend to make the HyperAgent source code open source?

- Can the authors include a discussion about LSPToolKit and if there are specific design choices that were made to better align it as a tool for LLMs? Specifically, can the authors compare with some of the existing approaches that use LSP to build tools for LLMs. This could be instructive for devtool builders.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730761435495

thread_type:
official_review

<thread_object>
creator: reviewer_wb3u

<content>
summary:
This work introduces HyperAgent, a multi-agent software engineering system that leverages Language Models. The HyperAgent system can be adapted to work on multiple Software Engineering benchmarks, specifically SWE-bench, RepoExec, and Defects4J. HyperAgent consists of one “parent” agent (the Planner) that can invoke one of three “child” agents (Navigator, Code Editor, Executor). These child agents can be invoked in parallel in any order by the Planner. The implementation also includes a suite of tools tailored to each child agent, along with flexible problem templating that allows for easy adaptation of HyperAgent to different coding tasks. HyperAgent performs comparably on several benchmarks, and the authors put forth several ablations examining how HyperAgent performs on such tasks.

strengths:
The HyperAgent’s ability to generalize to multiple benchmarks is its main strong suite in my opinion. The adaptation of existing multi-agent methods to a larger span of software engineering tasks is great to see, and it is a good confirmation of the methodologies suggested by prior works. This work explicitly identifies clear stages of software engineering, then demonstrates how a multi-agent framework designed around these stages is extensible to many tasks. In addition to the implementation, HyperAgent demonstrates comparable performance to existing solutions across three benchmarks.

weaknesses:
While the performance improvement of HyperAgent is admirable and the task coverage is quite impressive, I think it’s not quite clear what distinguishes HyperAGent from prior work. The methodology feels like a slight adaptation of existing multi-agent systems (e.g., MetaGPT) for the SWE-bench benchmark. The ablations, while interesting, look very similar to the ablations presented in SWE-agent. It is great that HyperAgent demonstrates comparable performance on several software engineering and coding benchmarks, and while this feels like a good engineering contribution, it remains unclear to me what new research questions or conclusions this exploration can offer up. I believe that more discussion about how HyperAgent’s architecture and methodology differs from prior work, along with analyses that back up these conclusions would make this paper a stronger research contribution.

- Section 2 Related Work: I would’ve liked more investment into a discussion of how HyperAgent differs from prior works, particularly in the second half of Section 2.2. After reading Sections 1 and 2, it is still not clear to me what new research questions HyperAgent addresses or what it does that truly distinguishes it from prior work.
- Section 3: Several comments - (1) How is this approach different from existing work such as MetaGPT, which arguably has more complex multi-agent systems following the Standard Operating Procedure framework? (2) Why is Multi-agent better than existing single agent approaches? How does having multiple agents lower inference costs or eliminate redundant information (wouldn’t the same information have to be communicated to multiple agents several times?) Also, it sounds like multiple child agents can be spawned in parallel - how is this more efficient than single agents, and why is the performance better because of this?
= The engineering details discussed in Section 3 sound quite impressive, particularly the use of the Message Queue and the general coordination of multiple agents. However, it is left to the reader to determine why these tools or multi-agent systems are interesting or novel relative to prior work. For instance, for Section 3.3, it is cool that these tools are made available, but unclear why this is better than the search and localization tools that prior works introduce (e.g. edit, search_dir, search_file in SWE-agent).
- Section 6 - These analyses are almost identical to the ones in SWE-agent. While this is interesting information, the paper may be stronger if there were more novel examinations that demonstrate how HyperAgent is distinct from prior work. For instance, “6.2 Analysis of Tool Design” is very similar to Table 3 in the SWE-agent paper. “6.3 Agent Behavior” is the same kind of graph as SWE-agent Figure 7. Analysis 6.1 is interesting, but it would be nice to know why the performance drops are not equal, and how certain tasks are still resolved even when there is no Editor, Navigator, or Executor.

questions:
- Line 42: “their claim of addressing general software engineering tasks is overstated”. Why is this the case? Given that the downstream tasks you evaluate on are all evaluated on Python or Java, how does HyperAgent resolve the shortcomings of more versatility across tasks, languages, and dev. Scenarios?
- Lines 45 - 104: You introduce a workflow and claim that this is representative of most of software engineering - what evidence is there to reflect this? This seems very reminiscent of the issue resolution pipeline that prior works already introduce. Also, “Although different SE tasks require varied approaches, they all follow a similar workflow.” - Are there citations or prior evidence from existing SE works that reflect that the workflow you discuss is in fact the most reflective? This part of the introduction feels quite anecdotal.
- Line 99 - How are your three advantages made clear in your evaluation? Aside from higher performance, I do not see why existing works are less generalizable than HyperAgent, or why these agents are more efficient or scalable.
- Line 116 - What is SWE-bench-Python, RepoExec-Python, and Defects4J-Java? Are these variants of the original benchmark?
- Line 118 - SWE-agent and OpenDevin were both evaluated on non SWE-bench tasks, including HumanEval, HumanEvalFix, Bird, BioCoder, WebArena, and much more. The claim that this is the first system to work on many SE tasks feels like an over-claim.
- Line 254 - “A task can *be* defined…”
- Line 251 - Missing space between “HyperAgent” and “is”
- Line 328 - What does processing time account for? Does this include or exclude the calls to the API? What’s the variance on these numbers?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730779230191

thread_type:
official_review

<thread_object>
creator: reviewer_coah

<content>
summary:
The paper proposes an LLM agent approach for software engineering tasks. It identifies that SE tasks often involve a combination of localisation, editing and execution, all orchestrated by some higher-level plan. A framework made up of modular components for each of these is proposed where each component is a ReAct style LLM agent loop with access to specialised tools. Through positive results on software generation (RepoExec) and bug-fixing (SWE-bench and Defects4J), the paper aims to show generalisability and efficiency (in terms of API costs).

strengths:
The paper provides a clear framework for decomposing software engineering tasks and solving these in a specialised manner. Specifically, the tools proposed by the paper, as well as how they are called (writing python code) are useful contributions. The ablations show tool design adds significant value to the framework, particularly the choice of what controls are given to the LLM (e.g. code_search, keyword summaries, and repair editor). The paper provides broad evaluation on both generation tasks (RepoExec) as well as bug-fixing (Defects4J and SWE-bench) which support claims of generalisability.

weaknesses:
1. The paper lacks thorough comparison to prior approaches that employ similar multi-stage approaches. CodeR [1] uses a multiagent approach for with seperate manager, localiser, editor and verifier (very similar to the components of Hyperagent), achieving 28.3% on SWE-bench-lite. MASAI [2] uses a modular architecture with similar components (without a manager) achieving 28.3% on SWE-bench-lite. Approaches such as AutoCodeRover (v20240620) (30%) [3] and Aider (26.3%) [3] both have phases for localisation and editing. Overall, the novelty of such a multi-stage approach is questionable, especially since the performance on SWE-bench is comparable to similar approaches. 

2. The paper also does not provide sufficient evidence as to why a multi-stage approach in general is more "generalisable". The performance numbers quoted for agentless (24.3%) seem out of date, with Agentless + GPT 4o (2024-05-13) achieving 27.3%. While it is understandable that comparing with all new entries on swebench is unreasonable, comparing with one of the many strong non multi-stage approaches both quantitatively and qualitatively would do much to strengthen the papers claims. 

In general, while the paper demonstrates positive results of their approach on two different kinds of tasks, the novelty of the approach as well as comparisons with prior work are lacking.

[1] https://arxiv.org/pdf/2406.01304
[2] https://arxiv.org/pdf/2406.11638
[3] https://www.swebench.com/

questions:
1. Is the Planner sequential or can it also dispatch parallel queries to multiple agents.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731049082598

thread_type:
official_review

<thread_object>
creator: reviewer_ahw1

<content>
summary:
This paper presents LLM based multi-agent system for various SE tasks, consisting of Navigator, Editor, Executor and Planner to control there 3 agent modules.
The paper shows the versatility of HyperAgent by applying 3 tasks, i.e. issue resolution, code generation, fault localization and repair, along with comparison with SOTA models, and how it outperforms other methods. The authors also provide a detailed analysis of agent roles, tool design and behaviour,

strengths:
It showed the avility to solve multipole software engineering tasks such as issue resolution, code generation, fault localizaiton and repair. The design enable users to use lightweight (cost efficient) LLM for navigation agent and strong LLM for other agents, depending on the problem.

weaknesses:
Implimantation details is not clear until the appendix. For example, Footenote 1 on  page 2 says" details of each agent are writtin in Sec4". But  Sec.4 is too short to understand. This is not "detail", at least "Implimentation summary" or "overview". Please add  more details on the architecture of each agent, their interactions, or the key algorithms/prompt used, for example.

How much the system is scalable and how did it implimented is not clear. From sec 3.2 denoted it is scalable because of using message queue. But it is not clear how it is related to scalablity. I understand that MessageQueue is designed for large system integration, but not show how it is related to "numerous subtasks, handlig complix tasks efficiently", which the authors wrote on Introduction.

questions:
The result and analysis shows the developed system works well, how ever it is not clear how generalizability and scalability is presented in the paper. It is recommend that authors clearly and explicitly explain how you achieved generality, efficiency, and scalability, what experiments you conducted, and how you evaluated the result section. Or/and add  section on generalizability and scalability experiments, because as for "scalability", there are 3.2 AGENT COMMUNICATION AND SCALABILITY, but ther eare no such description on other 2 features.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732549454301

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We are deeply grateful to the reviewers for their detailed and constructive feedback, which has significantly enhanced the quality of our work. We are encouraged by the reviewers’ recognition of HyperAgent’s contributions, particularly its ability to generalize across diverse software engineering tasks, its modular multi-agent design, and its effective evaluations on benchmarks like SWE-Bench, RepoExec, and Defects4J.

Several reviewers highlighted the novelty of combining specialized subagents with an asynchronous message queue system to enable parallel execution and improve task efficiency. They also appreciated the analysis of agent behavior, tool design, and the potential for HyperAgent to extend to additional tasks and languages.

We address the key concerns and suggestions raised by the reviewers below:

1. **Implementation Details and Scalability (AhW1, B9Au):**  
   We recognize the need to provide a clearer explanation of HyperAgent’s architecture and scalability mechanisms. We have expanded Section 3.2 with concrete examples and detailed the role of the message queue in handling complex tasks and ensuring parallelism.

2. **Comparison with Prior Work (coAh, hBBP, gPre):**  
   While HyperAgent shares similarities with systems like MetaGPT, SWE-Agent, and CodeR, it distinguishes itself through dynamic task orchestration, modularity, and scalability. We clarified these distinctions in Sections 2 and 3, emphasizing the Planner’s adaptive reasoning and the system’s versatility across diverse software engineering tasks. We also addressed concerns regarding task-specific baselines, highlighting HyperAgent’s superior generalization and scalability across multiple benchmarks.

3. **Performance and Cost Concerns (coAh, B9Au, gPre):**  
   HyperAgent balances advanced reasoning and adaptability with competitive performance. While agentless systems like SWE-Agent achieve lower costs for specific tasks, HyperAgent’s multi-agent framework supports more complex workflows like repository-level code generation, bug localization and program repair. We introduced cost-optimized configurations (e.g., HyperAgent-Lite) to address cost concerns while maintaining strong performance.

4. **Fault Localization and Unit Testing (hBBP, B9Au):**  
   We clarified HyperAgent’s fault localization approach and its comparison with ChatRepair. Unlike ChatRepair, which assumes prior localization, HyperAgent autonomously handles the full debugging pipeline. For unit testing, the Executor agent identifies and creates test cases as needed, which we have elaborated in Section 4.

5. **Experimental Results and Baselines (coAh, gPre):**  
   To ensure a fair comparison, we included configurations that use the same base LLMs as baselines like Agentless and AutoCodeRover. Discrepancies in previously reported baseline results have been clarified, and we updated the manuscript to reflect these corrections.

6. **Generalizability and Tool Integration (AhW1, gPre):**  
   HyperAgent’s generalizability stems from its modular design and integration with language server-based tools. These features enable seamless adaptation to new tasks and languages with minimal configuration. We provided additional details in Appendix A.3 on how tools like LSPToolKit align with LLM capabilities and compared them with existing approaches.

Based on the reviewers’ suggestions, we have revised the manuscript to clarify our contributions, address raised concerns, and provide additional supporting evidence. We hope these updates strengthen the case for HyperAgent as a scalable and generalist framework for real-world software engineering tasks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732590480059

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
HyperAgent’s Real-World Focus and Balanced Evaluation Across Benchmarks

comment:
In addition to the previous general response, we want to emphasize a critical focus of our work: **HyperAgent is designed to tackle realistic software engineering settings**, where tasks are defined and resolved at the **repository-level** rather than isolated, function-level examples. Realistic settings in software engineering involve challenges such as navigating large codebases, understanding interdependencies across files, and performing comprehensive testing—all of which mirror the daily workflows of professional developers. Benchmarks used to evaluate systems in such settings must reflect these complexities, going beyond competitive-level, single-file tasks like those in **HumanEval**, **HumanEvalFix**, or **BioCoder**.  

### Defining Realistic Benchmarks: Repository-Level Challenges  
Realistic benchmarks are those that evaluate a system’s ability to solve end-to-end tasks within a **repository context**, requiring navigation, dependency resolution, and multi-file understanding. The benchmarks we selected—**RepoExec**, **Defects4J**, and **SWE-Bench**—reflect this real-world complexity:  

1. **RepoExec**  
   RepoExec is one of the newest benchmarks designed for **repository-level code completion**. It evaluates tasks that require systems to generate code that integrates seamlessly into an existing codebase, navigating dependencies across files, and ensuring functionality in the broader repository context. RepoExec introduces a rigorous evaluation mechanism to verify the correctness of generated code, a critical aspect of real-world software engineering.  

2. **Defects4J**  
   Defects4J is a long-standing benchmark that challenges systems to localize and repair bugs in real-world Java repositories. Tasks in Defects4J require systems to work with real-world bugs in large, interconnected codebases, making it a prime example of a repository-level benchmark that prioritizes realistic settings over isolated snippets.  

3. **SWE-Bench**  
   SWE-Bench evaluates GitHub issue resolution, focusing on understanding and resolving practical software engineering issues. While valuable, SWE-Bench primarily addresses issue resolution within a limited scope and does not fully capture the repository-level challenges found in RepoExec and Defects4J.  

### Encouraging Balanced Evaluation Across Benchmarks  
We recognize the popularity and relevance of SWE-Bench, and we appreciate the reviewers’ focus on this benchmark. However, we would like to highlight that **RepoExec and Defects4J are equally complex and realistic benchmarks** that evaluate critical aspects of software engineering beyond issue resolution. RepoExec, for example, pushes systems to tackle repository-wide code completion, while Defects4J remains one of the most challenging and widely used benchmarks for fault localization and repair in real-world Java projects.  

Given the diverse strengths and contributions of these benchmarks, we kindly suggest that HyperAgent’s achievements across all three benchmarks be considered with equal weight. This would provide a more balanced evaluation of its generalization and effectiveness across a broad range of realistic software engineering tasks.  

### Realism in Evaluation and Multi-Language Generalization  
By selecting these benchmarks, we ensure that HyperAgent is evaluated in scenarios that require handling **realistic software engineering tasks** involving **entire repositories** rather than simplistic, single-file examples. Moreover, these benchmarks span multiple programming languages—**Python** (SWE-Bench, RepoExec) and **Java** (Defects4J)—demonstrating HyperAgent’s ability to generalize across languages.  

### Framework for Future Adaptation  
To further promote realism and scalability, we provide an **open-source framework** that allows researchers to adapt HyperAgent to new programming languages and tasks. This contribution ensures that HyperAgent not only excels in its current benchmarks but also serves as a foundation for expanding to additional real-world challenges in software engineering.  

### Evidence-Based Generalization  
We are the **first to provide comprehensive, result-driven evidence** demonstrating HyperAgent’s generalization capabilities across repository-level benchmarks. While **Reviewer Wb3u** mentioned that other works might be as generalizable as HyperAgent, such claims have not been substantiated with rigorous evaluations across diverse tasks. HyperAgent validates its adaptability with evidence-based performance on **repository-level benchmarks** like RepoExec and Defects4J, setting a new standard for systems tackling real-world software engineering problems.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The primary finding of the paper is that an LLM agent architecture which improves benchmark performance on repository-level software engineering tasks. The primary strength is that it it is better than baselines. The primary weakness is that it is conceptually similar to prior agents, and not strong enough empirically relative to baselines to warrant publication at this stage. I worry, like other reviewers, that these slim SOTA claims are fragile, and derive more from benchmark specifics than broad scientific principles or novel conceptual insight.