HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale

## Abstract

Abstract Large Language Models (LLMs) have revolutionized software engineering (SE), demonstrating remarkable capabilities in various coding tasks. While recent efforts have produced autonomous software agents based on LLMs for end-to-end development tasks, these systems are typically designed for specific SE tasks. We introduce HyperAgent , a novel generalist multi-agent system designed to address a wide spectrum of SE tasks across different programming languages by mimicking human developers’ workflows. Comprising four specialized agents—Planner, Navigator, Code Editor, and Executor— HyperAgent manages the full lifecycle of SE tasks, from initial conception to final verification. Through extensive evaluations, HyperAgent achieves state-of-the-art performance across diverse SE tasks: it attains a 25.01% success rate on SWE-Bench-Lite and 31.40% on SWE-Bench-Verified for GitHub issue resolution, surpassing existing methods. Furthermore, HyperAgent demonstrates superior performance in code generation at repository scale (RepoExec), and in fault localization and program repair (Defects4J), often outperforming specialized systems. This work represents a significant advancement towards versatile, autonomous agents capable of handling complex, multi-step SE tasks across various domains and languages, potentially transforming AI-assisted software development practices.

## 1 Introduction

In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities in assisting with various coding tasks, ranging from code generation and completion to bug fixing and refactoring. These models have transformed the way developers interact with code, providing powerful tools that can understand and generate human-like code snippets with impressive accuracy. However, as software engineering tasks grow in complexity, there is an emerging need for more sophisticated solutions that can handle the intricacies of real-world software development.

To address these challenges, software agents built on top of LLMs have emerged as a promising solution. These agents are designed to automate and streamline complex software engineering tasks by leveraging the advanced reasoning and generative capabilities of LLMs. They can perform tasks such as code generation, bug localization, and even orchestrating multi-step development processes. Despite their potential, current software agents remain limited in scope. Most existing agents are designed to tackle a specific SE task with limited capability, such as resolving GitHub issues (Jimenez et al., ; Chen et al., ; Arora et al., ; Xia et al., ; Zhang et al., ; Yang et al., ) using benchmarks like SWE-bench (Jimenez et al., ). Others (Huang et al., ) focus on competitive code generation benchmarks, such as APPS (Hendrycks et al., ), HumanEval (Chen et al., ), and MBPP (Austin et al., ). Another line of software agents (Qian et al., ; Hong et al., ; Nguyen et al., ) focuses on create complicated software given a set of requirements. While these specialized agents demonstrate impressive capabilities within their domains, the broader claim of solving general Software Engineering tasks may not fully materialize in practice. The diversity and interconnected nature of real-world SE challenges often require a more versatile approach that can seamlessly adapt to various tasks, programming languages, and development scenarios.

To address such drawbacks, we propose HyperAgent, a generalist multi-agent system designed to resolve a broad spectrum of SE tasks. Our design philosophy is rooted in the workflows that software engineers typically follow in their daily routines—whether it’s implementing new features in an existing codebase, localizing bugs in a large project, or providing fixes for reported issues and so on. While developers may use different tools or approaches to tackle these tasks, they generally adhere to consistent workflow patterns. We illustrate this concept through a workflow that represents how developers typically resolve coding tasks. Although different SE tasks require varied approaches, they all follow a similar workflow. Figure [1](#S0.F1) illustrates a typical workflow for a software engineer when resolving a task from the backlog, which is a list of tasks to be completed within a specific period.

- 1.
Analysis & Plan: The developer starts by understanding the task requirements through documentation review and stakeholder discussions. A working plan is then formulated, outlining key steps, potential challenges, and expected outcomes. This plan remains flexible, adjusting as new insights are gained or challenges arise. For simple tasks, this plan may remain a mental checklist rather than a detailed written document.
- 2.
Feature Localization: With a plan in place, the developer navigates the repository to identify relevant components, known as feature localization (Michelon et al., ; Martinez et al., ; Castro et al., ). This involves locating classes, functions, libraries, or modules pertinent to the task. Understanding dependencies and the system’s overall design is crucial here, ensuring the developer is ready to make informed modifications in the next phase.
- 3.
Edition: The developer edits the identified code components, implementing changes or adding new functionality. This phase also involves ensuring smooth integration with the existing codebase, maintaining code quality, and adhering to best practices. New or updated unit tests are written to ensure functionality and reliability.
- 4.
Execution: After editing, the developer tests the modified code to verify it meets the plan’s requirements. This includes running unit and integration tests, as well as conducting manual testing or peer reviews. If issues are found, the process loops back to previous phases until the task is fully resolved.

These four steps are repeated iteratively until the developer confirms that the task is complete. Note that the exact steps may vary depending on the task and the developer’s skill level. And the task may be completed in a single phase, or may require multiple iterations, i.e., after the Execution step, if the developer is not satisfied with the results, the developer may need to repeat the entire process. In HyperAgent, the framework is organized around four primary agents: Planner, Navigator, Code Editor, and Executor. An illustration of HyperAgentis depicted in Figure [2](#S3.F2). Each agent corresponds to a specific step in the overall workflow depicted in Figure [1](#S0.F1), though the actual workflow of each agent may differ slightly from how a human developer might approach similar tasks (^1^11Details about each agent, along with how these advantages are achieved, are provided in Sections [5](#S5)). Our design emphasizes three main advantages over existing methods:

- 1.
Generalizability: The framework is designed to easily adapt to a wide range of tasks with minimal configuration changes and little additional effort required to implement new modules into the system.
- 2.
Efficiency: Each agent is optimized to manage processes with varying levels of complexity, requiring different degrees of intelligence from LLMs. For example, a lightweight and computationally efficient LLM can be employed for navigation, which, while less complex, involves the highest token consumption. Conversely, more complex tasks, such as code editing or execution, require more advanced LLM capabilities.
- 3.
Scalability: The framework is built to scale effectively when deployed in real-world scenarios where the number of subtasks is significantly large. For instance, a complex task in the SWE-bench benchmark may require considerable time for an agent-based system to complete, and HyperAgentis designed to handle such scenarios efficiently.

We conducted extensive evaluations across diverse benchmarks to assess HyperAgent ’s effectiveness in various software engineering tasks, demonstrating its versatility and superior performance. In GitHub issue resolution, HyperAgent outperformed strong baselines such as AutoCodeRover (Zhang et al., ) and SWE-Agent (Yang et al., ) on the SWE-bench benchmark (Jimenez et al., ), while for code generation at repository-level scale, it surpassed strong retrieval-augmented generation (RAG) baselines, including WizardLM2 (Luo et al., ) and GPT-3.5-Turbo on the RepoExec benchmark (Hai et al., ). In fault localization and program repair, HyperAgent achieved state-of-the-art performance on the Defects4J dataset (Sobreira et al., ), outperforming SOTA baselines such as DeepFL Li et al. (), AutoFL (Kang et al., ), RepairAgent (Bouzenia et al., ), and SelfAPR (Ye et al., ). Notably, HyperAgent is the first system to evaluate SWE-Bench using open-source models such as Llama-3, offering a more cost-effective solution compared to closed-source alternatives while maintaining competitive performance across a wide range of software engineering tasks.

These evaluations underscore HyperAgent ’s unique position as the first system designed to work off-the-shelf across various SE tasks and programming languages, often exceeding the performance of specialized systems. This versatility highlights HyperAgent ’s potential as a transformative tool in real-world software development scenarios, capable of adapting to diverse challenges without the need for task-specific fine-tuning.
In summary, the key contributions of this work include:

- •
Introduction of HyperAgent , a novel generalist multi-agent system that closely mimics typical software engineering workflows. It incorporates stages for analysis, planning, feature localization, code editing, and execution/verification, enabling it to handle a broad spectrum of software engineering tasks across different programming languages.
- •
Extensive evaluation demonstrating superior performance across various software engineering benchmarks, including Github issue resolution (SWE-Bench-Python), repository-level code generation (RepoExec-Python), and fault localization and program repair (Defects4J-Java). To our knowledge, HyperAgent is the first system designed to work off-the-shelf across diverse SE tasks in multiple programming languages without task-specific adaptations.
- •
Insights into the design and implementation of scalable, efficient, and generalizable software engineering agent systems, paving the way for more versatile AI-assisted development tools that can seamlessly integrate into various stages of the software lifecycle.

## 2 Related Work

### 2.1 Deep Learning for Automated Programming

In recent years, applying deep learning to automated programming has captured significant interest within the research community (Balog et al., ; Bui and Jiang, ; Bui et al., ; Feng et al., ; Wang et al., ; Allamanis et al., ; Bui et al., ; Guo et al., , ). Specifically, Code Large Language Models (CodeLLMs) have emerged as a specialized branch of LLMs, fine-tuned for programming tasks (Wang et al., , ; Feng et al., ; Allal et al., ; Li et al., ; Lozhkov et al., ; Guo et al., ; Pinnaparaju et al., ; Zheng et al., ; Roziere et al., ; Nijkamp et al., ; Luo et al., ; Xu et al., ; Bui et al., ). These models have become foundational in building AI-assisted tools for developers, aiming to solve competitive coding problems from benchmarks such as HumanEval (Chen et al., ), MBPP (Austin et al., ), APPs (Hendrycks et al., ) and CRUXEval Gu et al. ().

Despite achieving good results with benchmark tasks, these models often struggle to generate real-world software that requires complex logic and detailed acceptance criteria, which are essential for practical applications (Hong et al., ; Qian et al., ; Nguyen et al., ). This limitation has led to the development of more sophisticated benchmarks and evaluation methods.

### 2.2 Benchmarks for Software Engineering

Subsequent works have introduced new SE benchmarks that expand the scope and complexity of evaluation criteria. These efforts include translating problems across programming languages (Cassano et al., ; Wang et al., ), incorporating third-party libraries (Lai et al., ; Liu et al., ), introducing derivative code completion tasks (Muennighoff et al., ), test coverage (Liu et al., ), modifying edit scope (Ding et al., ; Yu et al., ; Du et al., ), and enhancing robustness to dataset contamination (Naman Jain et al., ). However, these code generation problems remain largely self-contained, with short problem descriptions ($\sim$100 lines) and correspondingly brief solutions, typically requiring only basic language primitives. As language models (LMs) rapidly evolve, many of these benchmarks are becoming saturated, highlighting the need for more realistic and challenging tasks that demand complex reasoning and problem-solving skills.

Real-world software engineering often requires understanding the broader context of a problem, identifying relevant parts of a repository, and generating code that meets specific acceptance criteria. This complexity has led to the development of repository-level code generation benchmarks (Liu et al., ; Zhang et al., ; Li et al., ; Hai et al., ). Furthermore, recognizing that software engineering encompasses diverse tasks beyond code generation, researchers have developed benchmarks to address various aspects of the field. SWE-bench (Jimenez et al., ) mimics the workflow of resolving GitHub issues, while Defects4J (Just et al., ) and BugsInPy (Widyasari et al., ) are popular benchmarks for fault localization and program repair. CodeXGlue (Lu et al., ) offers a comprehensive benchmark covering multiple software engineering tasks. These developments reflect the ongoing efforts to create more holistic and challenging evaluations that better represent the complexities of real-world software development.

### 2.3 Autonomous Coding Agents

The emergence of open-source software development tools based on large language models (LLMs) has revolutionized the field of autonomous coding. These tools leverage LLMs’ capabilities to plan, self-critique, and extend functionality through function calls. By integrating such tools into development workflows, researchers have observed dramatic performance improvements in code generation tasks on popular benchmarks such as HumanEval (Chen et al., ). Notable advancements in this area include works by Huang et al. (), Chen et al. (), Shinn et al. (), Islam et al. (), Chen et al. (), and To et al. (). A parallel line of research focuses on generating complex software systems comprising multiple executable code files from input software requirements. Significant contributions in this domain include MetaGPT (Hong et al., ), AgileCoder (Nguyen et al., ), and ChatDev (Qian et al., ). These approaches aim to automate larger portions of the software development process, moving beyond single-file code generation.

Recently, there has been growing interest in employing coding agents to automatically resolve GitHub issues, a task that more closely mimics real-world software engineering challenges. This trend is evident in works such as SWE-Agent (Yang et al., ), SWE-bench (Jimenez et al., ), AutoCodeRover (Zhang et al., ), and agentless approaches (Xia et al., ). The integration of interaction and code generation has spawned novel applications where code serves as the primary modality for actions (Yang et al., ; Wang et al., ), tool construction (Wang et al., ; Zhang et al., ; Gu et al., ), and reasoning (Zelikman et al., , ; Shinn et al., ). Beyond general software engineering tasks, code language agents have found applications in specialized domains such as offensive security (Yang et al., ; Shao et al., ) and theorem proving (Thakur et al., ). This evolution towards agent-based models represents a significant step in bridging the gap between academic benchmarks and real-world software engineering challenges. By mimicking human-like problem-solving processes in coding tasks, these autonomous agents are paving the way for more sophisticated and practical AI-assisted development tools, potentially transforming the landscape of software engineering.

## 3 Problem Formulation

To formally define the software engineering tasks that HyperAgent is designed to address, we introduce the following notation and definitions:

##### Software Engineering Task

Let $\mathcal{R}=\{r_{1},r_{2},...,r_{n}\}$ be a software repository consisting of $n$ code files. Each file $r_{i}$ is a sequence of code tokens. A Software Engineering (SE) task $T$ is defined as a tuple $T=(D,\mathcal{R},F)$, where:

- •
$D$ is a natural language description of the task requirements
- •
$\mathcal{R}$ is the software repository on which the task is to be performed
- •
$F:\mathcal{R}\rightarrow\mathcal{R}^{\prime}$ is the desired transformation function that modifies the repository to fulfill the task requirements

The goal of an SE agent is to approximate the function $F$ given $D$ and $\mathcal{R}$, producing a modified repository $\mathcal{R}^{\prime}$ that satisfies the task requirements. We can further categorize SE tasks into several types based on their specific objectives.

##### Issue Resolution Task

An Issue Resolution Task $T_{IR}=(I,\mathcal{R},F_{IR})$ is an SE task where $I$ is a description of a GitHub issue, and $F_{IR}$ is a function that modifies $\mathcal{R}$ to resolve the issue.

##### Code Generation Task

A Code Generation Task $T_{CG}=(S,\mathcal{R},F_{CG})$ is an SE task where $S$ is a specification for new code to be generated, and $F_{CG}$ is a function that adds new code to $\mathcal{R}$ according to $S$.

##### Fault Localization Task

A Fault Localization Task $T_{FL}=(B,\mathcal{R},F_{FL})$ is an SE task where $B$ is a description of a bug or failing test case, and $F_{FL}:\mathcal{R}\rightarrow\mathcal{L}$ is a function that identifies a set of locations $\mathcal{L}$ in $\mathcal{R}$ where the bug is likely to be present.

##### Program Repair Task

A Program Repair Task $T_{PR}=(B,\mathcal{R},F_{PR})$ is an SE task where $B$ is a description of a bug or failing test case, and $F_{PR}$ is a function that modifies $\mathcal{R}$ to fix the bug while preserving correct functionality.

##### Generalist Software Engineering Agent

Given a set of diverse SE tasks $\mathcal{T}=\{T_{1},T_{2},...,T_{m}\}$, where each $T_{i}$ can be any type of SE task (e.g., $T_{IR}$, $T_{CG}$, $T_{FL}$, or $T_{PR}$), design an agent $A$ that can effectively perform all tasks in $\mathcal{T}$ by approximating the corresponding transformation functions $F_{i}$ for each task $T_{i}$. The challenge in designing such a generalist agent lies in creating a unified framework that can: (1) Understand and interpret diverse task descriptions $D_{i}$; (2) Navigate and comprehend different repository structures $\mathcal{R}_{i}$; (3) Generate appropriate code modifications or analysis results for each task type; (4) Verify the correctness and effectiveness of the solutions.

HyperAgent addresses this challenge by employing a multi-agent architecture that mimics the workflow of human developers. Each agent in the system specializes in a specific aspect of the software engineering process, allowing for a modular and adaptable approach to diverse SE tasks.

Figure: Figure 2: Overview of HyperAgent: A scalable, multi-agent system for software engineering tasks. The workflow illustrates the central Planner agent coordinating with specialized child agents (Navigator, Editor, and Executor) through an asynchronous Message Queue. This architecture enables parallel processing of subtasks, dynamic load balancing, and efficient handling of complex software engineering challenges.
Refer to caption: /html/2409.16299/assets/x2.png

## 4 HyperAgent : A Generalist Software Agent Framework

Figure [2](#S3.F2) illustrates the HyperAgent framework. The key design principle of HyperAgent is the centralization of advanced reasoning in the Planner agent, with delegation of computationally intensive but conceptually simpler tasks to specialized child agents. This approach optimizes inference costs and overall performance by eliminating redundant information processing outside the Planner’s context.

### 4.1 Centralized Multi-Agent System

The HyperAgent framework comprises four primary agents:

- •
Planner: The Planner agent serves as the central decision-making unit. It processes human task prompts, generates resolution strategies, and coordinates child agent activities. The Planner operates iteratively, generating plans, delegating subtasks, and processing feedback until task completion or a predefined iteration limit is reached.
- •
Navigator:
The Navigator agent specializes in efficient information retrieval within the codebase. Equipped with IDE-like tools such as go_to_definition and code_search, it traverses codebases rapidly, addressing challenges associated with private or unfamiliar code repositories. The Navigator is designed for speed and lightweight operation, utilizing a combination of simple tools to yield comprehensive search results.
- •
Editor: The Editor agent is responsible for code modification and generation across multiple files. It employs tools including auto_repair_editor, code_search, and open_file. Upon receiving target file and context information from the Planner, the Editor generates code patches, which are then applied using the auto_repair_editor.
- •
Executor: The Executor agent validates solutions and reproduces reported issues. It utilizes an interactive_bash_shell for maintaining execution states and open_file for accessing relevant documentation. The Executor manages environment setup autonomously, facilitating efficient testing and validation processes.

### 4.2 Agent Communication and Scalability

Inter-agent communication in HyperAgent is structured to minimize information loss, ensure effective task delegation, and enable scalable, parallel processing of complex software engineering tasks. To achieve these goals, we implement an asynchronous communication model using a distributed Message Queue system based on Redis.

The Planner communicates with child agents using a standardized message format comprising Context and Request fields. The Context field provides relevant background information and rationale for the requested action, while the Request field contains specific, actionable instructions for the child agent. This structure enables the Planner to convey both high-level context and specific directives efficiently.

When the Planner needs to delegate a task, it decomposes it into subtasks and publishes messages containing these subtasks to appropriate queues in the Message Queue system. Child agents, including multiple instances of the Navigator, Editor, and Executor, continuously monitor these queues and process tasks asynchronously. This approach allows for parallel processing of subtasks, significantly improving the system’s efficiency and scalability.

For instance, when exploring a large codebase, multiple Navigator instances can simultaneously investigate different sections, dramatically reducing exploration time. Similarly, the Editor can parallelize large-scale code changes across multiple files, and the Executor can run multiple tests concurrently, greatly accelerating the validation process.

To mitigate information loss in child agent reports, we implement a lightweight LLM summarizer. This component compiles intermediate results from the child agent’s execution log, generating a concise yet comprehensive summary. Upon task completion, child agents publish these summarized results back to a designated queue for the Planner to aggregate and process. This approach preserves critical details about code snippets, explored objects, and codebase structure, reducing the risk of information degradation or hallucination in the Planner over multiple iterations. The Message Queue-based architecture offers several advantages:

- •
Parallel Processing: Multiple instances of each agent type can work on different subtasks simultaneously, significantly improving overall system throughput.
- •
Load Balancing: Tasks can be dynamically distributed among multiple instances of each agent type, allowing for efficient resource utilization.
- •
Fault Tolerance: If an agent instance fails, unprocessed tasks remain in the queue and can be redistributed to other available instances, enhancing system reliability.
- •
Scalability: The system can easily scale horizontally by adding more instances of child agents to handle increased workload, without modifying the core architecture.
- •
Decoupling: The Message Queue decouples the Planner from the child agents, allowing for independent scaling and maintenance of each component.

This scalable, asynchronous communication model enables HyperAgent to efficiently handle complex software engineering tasks in large-scale, distributed environments. It adapts dynamically to varying workloads and task complexities, making it well-suited for real-world software development scenarios where task volume and complexity can fluctuate significantly.

### 4.3 Tool Design

The efficacy of HyperAgent is significantly enhanced by its specialized tool design. Key considerations in tool development include feedback format, functionality, and usability. Tools provide succinct, informative, and LLM-interpretable output. Each tool is optimized for its specific role in the software engineering process. Input interfaces are designed for intuitive interaction with LLMs, minimizing the risk of incorrect usage.

The Navigator employs a suite of navigation tools. The code_search tool utilizes a trigram-based search engine (Zoekt) (^2^22[https://github.com/google/zoekt](https://github.com/google/zoekt)) with symbol ranking. IDE-inspired features such as go_to_definition, get_all_references, and get_all_symbols enable precise code navigation. The get_tree_structure tool visualizes codebase structure, while open_file displays source code with integrated keyword search functionality. A proximity search algorithm is implemented to address LLM limitations in providing precise positional inputs.

The Editor utilizes the repair_editor tool, which applies and refines code patches, automatically addressing common syntax and indentation issues. It also employs navigation tools for context-aware editing.

The Executor employs an interactive_shell that maintains execution states for command sequences. It also uses open_file and get_tree_structure to facilitate access to testing and setup documentation.

**Table 1: HyperAgent : Specialized Tool Design by Agent**
| Agent | Tool | Description |
| --- | --- | --- |
| Navigator | code_search | Trigram-based search engine (Zoekt) with symbol ranking |
| go_to_definition | Locates and displays the definition of a given symbol |  |
| get_all_refs | Finds all references to a specific symbol in the codebase |  |
| get_all_symbols | Lists all symbols (functions, classes, etc.) in a given file or module |  |
| get_tree_struc | Visualizes the codebase structure as a tree |  |
| open_file | Displays source code with integrated keyword search functionality |  |
| Editor | repair_editor | Applies and refines code patches, addressing syntax and indentation issues |
| Navigation tools | Employs Navigator’s tools for context-aware editing |  |
| Executor | interactive_shell | Maintains execution states for command sequences |
| open_file | Accesses testing and setup documentation |  |
| get_tree_struc | Visualizes structure of test suites and configuration files |  |

## 5 Implementation Details

For summarizer in HyperAgent , we used Mixtral 8x7B (Jiang et al., ). To examine the flexibility of our framework and measure robustness, we employed a variety of language models (LMs) across different configurations. We tested four main configurations of HyperAgent , each utilizing different combinations of LMs for the Planner, Navigator, Editor, and Executor roles. Table [3](#S5.T3) presents these configurations.

**Table 3: HyperAgent Configurations**
| Configuration | Planner | Navigator | Editor | Executor |
| --- | --- | --- | --- | --- |
| HyperAgent-Lite-1 | Claude-3-Sonnet | Claude-3-Haiku | Claude-3-Sonnet | Claude-3-Haiku |
| HyperAgent-Lite-2 | Llama-3-70B | Llama-3-8b | Llama-3-70B | Llama-3-8b |
| HyperAgent-Full-1 | Claude-3-Sonnet | Claude-3-Sonnet | Claude-3-Sonnet | Claude-3-Sonnet |
| HyperAgent-Full-2 | GPT-4o | GPT-4o | GPT-4o | GPT-4o |
| HyperAgent-Full-3 | Llama-3-70B | Llama3-70B | Llama-3-70B | Llama-3-70B |

An advantage of our design is that we can choose the most suitable LLMs for each agent type, optimizing performance and accuracy. For instance, the Planner, serving as the brain of the whole system, requires a powerful model with superior reasoning capabilities to effectively orchestrate complex tasks. The Editor also demands a strong model with robust coding capability to edit and generate code accurately, given the inherent complexity of real-world codebases. In contrast, the Navigator and Executor agents can utilize less powerful models with smaller footprints and faster inference times, as their tasks are more straightforward and require less complex reasoning. This flexible architecture enables efficient allocation of computational resources, ensuring optimal performance across different agent types while balancing the trade-offs between model capability and computational cost. Such a design also allows for easier updates and improvements to individual components without necessitating a complete system overhaul.

As a result, we can implement various configurations of HyperAgent as shown in Table [3](#S5.T3), utilizing both open-source and closed-source models. For closed-source models, we designate GPT-4 and Claude-3 Sonnet as strong models, while Claude-3 Haiku serves as the weak model. In the open-source domain, Llama-3-70B acts as the strong model, with Llama-3-8B serving as the weak model. We believe that HyperAgent is the first system to evaluate SWE-Bench using open-source models such as Llama-3, offering a more cost-effective solution compared to closed-source alternatives while maintaining competitive performance across a wide range of software engineering tasks.

## 6 Evaluations

We conducted extensive evaluations of HyperAgent across diverse benchmarks to assess its effectiveness in various software engineering tasks. Our criteria for selecting SE tasks and corresponding benchmarks were based on complexity and real-world relevance. Each task must require multiple reasoning steps to complete, such as retrieving relevant contexts from the repository, editing code, and executing tests. We focused on well-known tasks including GitHub issue resolution, code generation at repository-level scale, fault localization, and program repair.

- •
GitHub Issue Resolution: This task requires complex reasoning steps to understand the issue description, explore the code repository and identify the relvant contexts, edit the code and evaluate if the issues have been resolved.
- •
Code Generation at Repository-Level Scale: While code generation is a popular task for evaluating AI systems’ coding capabilities, many benchmarks use competitive programming tasks (Hendrycks et al., ; Chen et al., ; Austin et al., ) that don’t reflect real-world software development scenarios. Instead, we choose benchmarks that evaluate code generation at the repository level, requiring the system to retrieve relevant contexts, understand the codebase, and generate code consistent with existing structures.
- •
Fault Localization: This task involves identifying the specific location of a bug within a program. It requires analyzing program behavior, understanding code structure, and pinpointing the exact lines or components responsible for faulty behavior. Our benchmark assesses the system’s ability to accurately locate bugs in complex codebases, simulating real-world debugging scenarios.
- •
Program Repair: Building upon fault localization, program repair involves automatically fixing identified bugs. This task challenges the system to not only locate the bug but also generate appropriate patches. Our benchmark evaluates the quality and correctness of proposed fixes, considering factors such as maintaining original program functionality and adhering to coding standards.

### 6.1 GitHub Issue Resolution

#### 6.1.1 Dataset

We evaluated HyperAgent using the SWE-bench benchmark (Jimenez et al., ), which comprises 2,294 task instances derived from 12 popular Python repositories. SWE-bench assesses a system’s capability to automatically resolve GitHub issues using Issue-Pull Request (PR) pairs, with evaluation based on verifying unit tests against the post-PR behavior as the reference solution. Due to the original benchmark’s size and the presence of underspecified issue descriptions, we utilized two refined versions: SWE-bench-Lite (300 instances) and SWE-bench-Verified (500 instances). The Lite version filters samples through heuristics (e.g., removing instances with images, external hyperlinks, or short descriptions), while the Verified version contains samples manually validated by professional annotators. These streamlined versions offer a more focused and reliable evaluation framework, addressing the limitations of the original benchmark while maintaining its core objectives.

#### 6.1.2 Baselines

We compared HyperAgent to several strong baselines: SWE-Agent (Yang et al., ), a bash interactive agent with Agent-Computer Interfaces; AutoCodeRover (Zhang et al., ), a two-stage agent pipeline focusing on bug fixing scenarios; Agentless (Xia et al., ), a simplified two-phase approach that outperforms complex agent-based systems in software development tasks; and various Retrieval Augmented Generation (RAG) baselines as presented in (Jimenez et al., ). These baselines represent a diverse range of approaches to software engineering tasks, providing a comprehensive evaluation framework for our method.

#### 6.1.3 Metrics

We evaluate this task using three key metrics: (1) percentage of resolved instances, (2) average time cost, and (3) average token cost. The percentage of resolved instances measures overall effectiveness, indicating the proportion of SWE-bench tasks where the model generates solutions passing all unit tests, thus fixing the described GitHub issue. Average time cost assesses efficiency in processing and resolving issues, while average token cost quantifies economic efficacy through computational resource usage. These metrics collectively provide a comprehensive evaluation of each tool’s performance in addressing real-world software problems, balancing success rate with time and resource utilization.

#### 6.1.4 Results

**Table 4: Performance comparison on SWE-Bench datasets. Verified (%) and Lite (%) columns show the percentage of resolved instances (out of 500 for Verified, 300 for Lite). Avg Time is in seconds, and Avg Cost is in US dollars.**
| Method | Verified (%) | Lite (%) | Avg Time | Avg Cost ($) |
| --- | --- | --- | --- | --- |
| AutoCodeRover + GPT-4o | 28.80 | 22.7 | 720 | 0.68 |
| SWE-Agent + Claude 3.5 Sonnet | 33.60 | 23.00 | – | 1.79 |
| SWE-Agent + GPT-4o | 23.20 | 18.33 | – | 2.55 |
| Agentless + GPT-4o | 33.20 | 24.30 | – | 0.34 |
| RAG + Claude 3 Opus | 7.00 | 4.33 | – | – |
| HyperAgent-Lite-1 | 27.33 | 21.67 | 132 | 0.45 |
| HyperAgent-Lite-2 | 16.00 | 11.00 | 108 | 0.76 |
| HyperAgent-Full-1 | 31.00 | 24.67 | 320 | 1.82 |
| HyperAgent-Full-2 | 31.40^* | 25.00 | 210 | 2.01 |
| HyperAgent-Full-3 | – | – | – | – |

The results presented in Table [4](#S6.T4) demonstrate the competitive performance of HyperAgentacross different configurations on the SWE-Bench datasets. Several key observations can be made:

- 1.
Performance: HyperAgent-Full-2 achieves a strong success rate of 31.40% on the SWE-Bench Verified dataset. This performance is competitive with top-performing methods such as SWE-Agent + Claude 3.5 Sonnet (33.60%) and Agentless + GPT-4o (33.20%). On the SWE-Bench Lite dataset, HyperAgent-Full-2 achieves the best performance among all methods with a 25.00% success rate, closely followed by HyperAgent-Full-1 at 24.67%. This outperforms strong baselines like Agentless + GPT-4o (24.30%) and SWE-Agent + Claude 3.5 Sonnet (23.00%).
- 2.
Efficiency: HyperAgent-Lite configurations demonstrate impressive efficiency. HyperAgent-Lite-1 and HyperAgent-Lite-2 have average processing times of 132 and 108 seconds respectively, significantly faster than AutoCodeRover + GPT-4o (720 seconds).
- 3.
Cost-Effectiveness: HyperAgent-Lite-1 offers an excellent balance of performance (27.33% on Verified, 21.67% on Lite) and cost ($0.45). This makes it substantially more cost-effective than several baselines, including SWE-Agent + Claude 3.5 Sonnet ($1.79) and SWE-Agent + GPT-4o ($2.55).
- 4.
Scalability: The performance spectrum across HyperAgent-Lite and HyperAgent-Full configurations demonstrates the system’s adaptability to different performance-cost trade-offs, providing flexibility for various use cases.

Overall, HyperAgentdemonstrates strong and competitive performance across both datasets. HyperAgent-Full-2 achieves the best performance on the Lite dataset and is highly competitive on the Verified dataset. The system’s ability to achieve high success rates while offering various configurations for efficiency and cost-effectiveness positions HyperAgentas a versatile and practical solution for automated GitHub issue resolution. Its performance is particularly noteworthy given its use of open-source models, offering a compelling alternative to systems relying on closed-source models.

### 6.2 Repository-Level Code Generation

#### 6.2.1 Dataset

We evaluate our task using RepoExec (Hai et al., ), a benchmark for Python for assessing repository-level code generation with emphasis on executability and correctness. Comprising 355 samples with automatically generated test cases (96.25% coverage), RepoExec typically provides gold contexts extracted through static analysis. The gold contexts are splitted into different richness level, including full context, medium context and small context. The richness level of contexts represent for different way to retrieve the contexts, such as import, docstring, function signature, API invocaction, etc. However, to measure HyperAgent’s ability to navigate codebases and extract contexts independently, we omit these provided contexts in our evaluation.

#### 6.2.2 Baselines

We compared HyperAgent against strong retrieval-augmented generation (RAG) baselines, including WizardLM2 + RAG, GPT-3.5-Turbo + RAG, WizardLM2 + Sparse RAG, and GPT-3.5-Turbo + Sparse RAG. These baselines represent state-of-the-art approaches in combining large language models with information retrieval techniques. Sparse RAG represents for using BM25 retriever and RAG stands for using UnixCoder Guo et al. () as context retriever. We used chunking size of 600 and python code parser from Langchain (^3^33https://github.com/langchain-ai/langchain) allowing us to parse the context in a syntax-aware manner. Additionally, we included results from CodeLlama (34b and 13b versions) and StarCoder models when provided with full context from RepoExec, serving as upper bounds for performance with complete information.

#### 6.2.3 Metrics

We used pass@1 and pass@5 as our primary metric, which measures the percentage of instances where all tests pass successfully after applying the model-generated patch to the repository.

**Table 5: RepoExec Results Comparison: HyperAgent-Lite-3 achieves comparable or superior performance to models provided with full context, particularly in Pass@5 (53.33%). It outperforms RAG-based models, demonstrating effective automatic context retrieval from codebases. This highlights the potential of end-to-end solutions like HyperAgentin real-world scenarios where manual context provision is impractical.**
| Model | Context Used | Pass@1 | Pass@5 | Cost ($) |
| --- | --- | --- | --- | --- |
| CodeLlama-34b-Python | Full | 42.93% | 49.54% | – |
| CodeLlama-13b-Python | Full | 38.65% | 43.24% | – |
| StarCoder | Full | 28.08% | 33.95% | – |
| WizardLM2 + RAG | Auto-retrieved | 33.00% | 49.16% | 0.04 |
| GPT-3.5-Turbo + RAG | Auto-retrieved | 24.16% | 35.00% | 0.02 |
| WizardLM2 + Sparse RAG | Auto-retrieved | 34.16% | 51.23% | 0.05 |
| GPT-3.5-Turbo + Sparse RAG | Auto-retrieved | 25.00% | 35.16% | 0.03 |
| HyperAgent-Lite-3 | Auto-retrieved | 38.33% | 53.33% | 0.18 |

As shown in Table [5](#S6.T5), the RepoExec benchmark results reveal insightful comparisons between different code generation approaches. CodeLlama-34b-Python, given full context, achieves the highest Pass@1 rate at 42.93%. Notably, our HyperAgent-Lite-3, which automatically retrieves relevant contexts, outperforms all models in Pass@5 at 53.33%, demonstrating its effective codebase navigation. In contrast, RAG-based models show limited effectiveness in capturing complex code relationships, underperforming both HyperAgentand full-context models. These findings highlight the potential of end-to-end solutions like HyperAgentfor real-world scenarios where manual context provision is impractical, emphasizing the need for sophisticated context retrieval methods in code generation tasks.

### 6.3 Fault Localization

#### 6.3.1 Dataset

We evaluated HyperAgent on the Defects4J dataset (Sobreira et al., ; Just et al., ), a widely used benchmark for fault localization and program repair tasks. Our evaluation encompassed all 353 active bugs from Defects4J v1.0.

#### 6.3.2 Baselines

We compared HyperAgent against several strong baselines, including DeepFL Li et al. (), AutoFL (Kang et al., ), Grace (Lou et al., ) DStar (Wong et al., ), and Ochiai (Zou et al., ). DeepFL, AutoFL and Grace represent more recent approaches that leverage deep learning methods for fault localization. In contrast, DStar and Ochiai are traditional techniques that employ static analysis-based methods to identify faults.

#### 6.3.3 Metrics

We follow AutoFL (Kang et al., ) to use acc@k metric which measures the
We adopt the acc@k metric from AutoFL to evaluate bug localization performance. This metric measures the number of bugs for which the actual buggy location is within a tool’s top k suggestions. We choose this metric because previous research indicates that developers typically examine only a few suggested locations when debugging, and it’s widely used in prior work. To handle ties in the ranking, we employ the ordinal tiebreaker method instead of the average tiebreaker, as we believe it more accurately reflects a developer’s experience when using a fault localization tool.

#### 6.3.4 Results

**Table 6: Comparison of Acc@1 across Different Fault Localization Methods on the Defects4J dataset. HyperAgent-Lite-1 significantly outperforms all baselines, achieving 59.70% accuracy on this widely-used benchmark. It surpasses the next best method, AutoFL, by 8.7 percentage points, and more than doubles the performance of traditional methods like Dstar and Ochiai. This demonstrates the effectiveness of HyperAgent’s approach in precisely locating faults on the first attempt in real-world Java projects, potentially reducing debugging time and effort for developers.**
| Method | Acc@1 | Cost ($) |
| --- | --- | --- |
| Ochiai (Zou et al., ) | 20.25% | – |
| DeepFL (Li et al., ) | 33.90% | – |
| Dstar (Wong et al., ) | 33.90% | – |
| Grace (Zou et al., ) | 49.36% | – |
| AutoFL (Kang et al., ) | 51.00% | – |
| HyperAgent-Lite-1 | 59.70% | 0.18 |

The fault localization results in Table [6](#S6.T6) on the Defects4J dataset demonstrate  HyperAgent superior performance, achieving an Acc@1 of 59.70%. This significantly outperforms all other methods, surpassing the next best performer, AutoFL, by 8.7 percentage points (51.00%) and more than doubling the accuracy of traditional methods like Ochiai (20.25%).  HyperAgent’s ability to correctly identify the buggy location on its first attempt for nearly 60% of the bugs suggests a potentially substantial reduction in debugging time and effort in real-world scenarios. The wide performance range across methods (20.25% to 59.70%) highlights both the challenges in fault localization and the significant improvement  HyperAgentrepresents. While there’s still room for improvement, these results indicate that  HyperAgent’s approach is more effective than existing deep learning and traditional static analysis-based methods for fault localization in Java projects.

### 6.4 Program Repair

#### 6.4.1 Dataset

We also utilize the Defects4J dataset (Sobreira et al., ; Just et al., ). This dataset is particularly suitable as it provides gold-standard fixes and test cases, which are crucial for evaluating the effectiveness of repair techniques once faults are localized and fixes are applied.

#### 6.4.2 Baselines

We compared HyperAgent with configuration Lite-1 against state-of-the-art baselines: RepairAgent (Bouzenia et al., ), SelfAPR (Ye et al., ), and ITER (Ye and Monperrus, ). ITER and SelfAPR are learning-based methods, while RepairAgent is a multi-agent system leveraging LLMs to autonomously plan and execute bug fixes. RepairAgent interleaves information gathering, repair ingredient collection, and fix validation, dynamically selecting tools based on gathered information and previous fix attempts.

#### 6.4.3 Metrics

As in previous studies Bouzenia et al. (); Hidvégi et al. (), we provide both the count of plausible and correct patches. A fix is considered plausible if it passes all the test cases, but this doesn’t guarantee its correctness. To assess if a fix is correct, we automatically verify if its syntax aligns with the fix created by the developer via exactly matching Abstract Syntax Tree (AST) between fixes.

#### 6.4.4 Results

The results presented in Table [7](#S6.T7) on the Defects4J dataset demonstrate HyperAgent’s strong performance compared to existing repair tools. HyperAgentachieved 192 correct fixes out of 835 bugs, outperforming its closest competitors: RepairAgent (164 correct fixes) and SelfAPR (110 correct fixes). Additionally, HyperAgentgenerated 249 plausible fixes, indicating its capability to produce a high number of test-passing patches.
The performance metrics of HyperAgentare noteworthy, with 249 plausible fixes (29.8%) and 192 correct fixes (23%) out of 835 bugs. These results can be attributed to HyperAgent’s four-phase repair strategy: planning, localization, fix verification via testing, and implementation. This approach enables HyperAgentto address the complex challenges of automated program repair effectively.
HyperAgent’s performance extends to the expanded Defects4Jv2 dataset, where it correctly fixed 110 out of 440 bugs. This demonstrates HyperAgent’s ability to maintain high performance across a broader range of modern Java projects, highlighting its versatility and scalability in automated program repair tasks.

**Table 7: Results on Defects4J dataset comparing HyperAgentwith other repair tools. The table includes the number of bugs, and for HyperAgent, both plausible and correct fixes. For RepairAgent, ITER, and SelfAPR, only the number of correct fixes is shown. Note that ITER does not have results for Defects4Jv2. HyperAgentachieves the best performance with 249 plausible fixes and 192 correct fixes (highlighted in blue).**
| Project | Bugs | HyperAgent | RepairAgent | ITER | SelfAPR |  |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | Plausible | Correct | Correct | Correct | Correct |
| Chart | 26 | 20 | 14 | 11 | 10 | 7 |
| Cli | 39 | 18 | 10 | 8 | 6 | 8 |
| Closure | 174 | 30 | 24 | 27 | 18 | 20 |
| Codec | 18 | 12 | 9 | 9 | 3 | 8 |
| Collections | 4 | 1 | 1 | 1 | 0 | 1 |
| Compress | 47 | 12 | 9 | 10 | 4 | 7 |
| Csv | 16 | 8 | 7 | 6 | 2 | 1 |
| Gson | 18 | 5 | 4 | 3 | 0 | 1 |
| JacksonCore | 26 | 6 | 6 | 5 | 3 | 3 |
| Jacksondatabind | 112 | 21 | 14 | 11 | 0 | 8 |
| JacksonXml | 6 | 1 | 1 | 1 | 0 | 1 |
| Jsoup | 93 | 26 | 24 | 18 | 0 | 6 |
| JxPath | 22 | 3 | 2 | 0 | 0 | 1 |
| Lang | 63 | 24 | 19 | 17 | 0 | 10 |
| Math | 106 | 36 | 32 | 29 | 0 | 22 |
| Mockito | 38 | 20 | 12 | 6 | 0 | 3 |
| Time | 26 | 6 | 4 | 2 | 2 | 3 |
| Defects4Jv1.2 | 395 | 119 | 82 | 74 | 57 | 64 |
| Defects4Jv2 | 440 | 130 | 110 | 90 | – | 46 |
| Total | 835 | 249 | 192 | 164 | 57 | 110 |
| Percentage |  | (29.8%) | (22.9%) | (19.64%) | (6.82%) | (13.17%) |

## 7 Analysis

### 7.1 Ablation Studies on Agent Roles

We conducted experiments using SWE-bench Tiny to evaluate the contribution of each agent role to overall performance. This was done by replacing each child agent with the planner itself, requiring the planner to directly utilize the eliminated agent’s toolset. Table [8](#S7.T8) illustrates a significant cost increase for all configurations when any agent role is removed. The resolving rate also decreases, with the magnitude varying based on which role is eliminated. Removing the Navigator causes the most substantial performance drop, followed by the Editor and the Executor, respectively.
Notably, when a medium-long context length LLM such as WizardLM2 acts as the Planner and replaces the role of Editor or Navigator, we observe a more severe drop in the resolving rate. This is attributed to these roles requiring continuous interaction with the environment, necessitating a long context.

**Table 8: Ablation study on different agent role’s contribution on SWE-bench Tiny**
|  | Model | SWE-bench Tiny |  |
| --- | --- | --- | --- |
|  | % Pass Rate | $ Avg. Cost |  |
| GPT-4o | HyperAgent | 15.00 | 0.42 |
| w/o Navigator | 7.00 | 2.81 |  |
| w/o Editor | 11.00 | 1.92 |  |
| w/o Executor | 14.00 | 0.75 |  |
| WizardLM2 | HyperAgent | 13.00 | 0.31 |
| w/o Navigator | 4.00 | 1.21 |  |
| w/o Editor | 7.00 | 0.51 |  |
| w/o Executor | 13.00 | 0.38 |  |

### 7.2 Analysis of Tool Design

We investigated the improvements brought by our major design choices in the tool’s interface and functionality. An ablation study was conducted on the functionalities of go_to_definition, auto_repair_editor, open_file, and code_search using SWE-bench Tiny. For each tool, we evaluated the overall performance when the tool is utilized versus when it is not, as shown in Table [9](#S7.T9).

**Table 9: Ablation result on resolving performance on SWE-Bench Tiny with different key tool designs**
| go_to_definition | open_file | code_search | auto_repair_editor |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Used | $9.00_{\downarrow{6.0}}$ | Used | $9.00_{\downarrow{6.0}}$ | Used | $8.00_{\downarrow{6.0}}$ | Used | $8.00_{\downarrow{7.0}}$ |
| w/ search | 15.00 | w/ annotated lines | $11.00_{\downarrow{4.0}}$ | w/ preview | $11.00_{\downarrow{3.0}}$ | w/ linting feedback | $11.00_{\downarrow{4.0}}$ |
| No usage | $12.0_{\downarrow{3.0}}$ | w/ keyword summary | 15.00 | w/ ranking | 14.00 | w/ repairing | 15.00 |
|  |  | No usage | $4.0_{\downarrow{11.0}}$ | No usage | $3.0_{\downarrow{11.0}}$ | No usage | $1.0_{\downarrow{14.0}}$ |

A crucial finding for go_to_definition is that the LLM agent struggles to effectively use this IDE-like feature. It requires exact line and column numbers and the precise symbol name, which demands precise localization of character positions. Despite supporting annotated line numbers, the agent often fails and retries multiple times. However, incorporating a proximity-based search process, allowing the agent to approximate specifications, significantly improves performance (from 9% without search to 15% with search).
For open_file, small LLMs like Claude Haiku tend to scroll up and down multiple times to find desired snippets by continuously increasing start_line and end_line, leading to out-of-context length issues. We addressed this by adding an additional input field keywords, allowing the LLM to search keywords inside the file. This enables the tool to quickly localize the positions of keywords inside the file and display the surrounding lines, increasing the resolving rate by 3%.
Without code_search, the Navigator faces significant challenges in swiftly identifying necessary objects, resulting in a substantially lower performance rate of 3% compared to 8% when the tool is employed. Enhancing the output to include partial surrounding context around the keyword enables the Navigator to make more informed decisions, improving performance from 8% to 11%. Prioritizing search results for key objects such as functions and classes, and re-ranking these results further enhances overall performance, increasing it from 11% to 14%.
We observed a substantial enhancement (8% to 11%) when providing Python linting feedback(^4^44We use [flake8](https://flake8.pycqa.org/en/latest/) for providing syntax errors.) to the Editor whenever it produces a patch via auto_repair_tool. By enabling an internal LLM of the tool to autonomously refine the generated patch and attempt to fix any encountered errors, the quality of the patch improves, leading to a 4% increase in resolving rate.

### 7.3 Agent Behavior

We analyzed the frequency of each agent role requested by the Planner throughout the issue resolution process. Figure [3](#S7.F3) illustrates a typical pattern where the Planner is most active at the beginning of the resolution process, gathering relevant information about the codebase environment. Subsequently, the Editor is frequently used to generate patches, often immediately following the Navigator, with notable peaks at Iterations 4 and 8. Finally, the Executor is requested more frequently in the later iterations to verify the results by executing tests. It is noteworthy that, in the first iteration, there is a small peak indicating that the Executor is requested to reproduce the issue.

Figure: Figure 3: Frequency of agent role requests by the Planner throughout the issue resolution process. The graph illustrates the typical pattern of agent activities: the Planner is most active at the beginning, gathering codebase information; the Editor is frequently used for patch generation, often following the Navigator, with peaks at Iterations 4 and 8; the Executor is more active in later iterations for result verification, with a small peak in the first iteration for issue reproduction.
Refer to caption: /html/2409.16299/assets/x3.png

## 8 Conclusion

In this paper, we introduced HyperAgent , a generalist multi-agent system designed to address a wide range of software engineering tasks. By closely mimicking typical software engineering workflows, HyperAgent incorporates stages for analysis, planning, feature localization, code editing, and execution/verification. Our extensive evaluations across diverse benchmarks, including GitHub issue resolution, code generation at repository-level scale, and fault localization and program repair, demonstrate that HyperAgent not only matches but often exceeds the performance of specialized systems.

The success of HyperAgent highlights the potential of generalist approaches in software engineering, offering a versatile tool that can adapt to various tasks with minimal configuration changes. Its design emphasizes generalizability, efficiency, and scalability, making it well-suited for real-world software development scenarios where tasks can vary significantly in complexity and scope.

Future work could explore the integration of HyperAgent with existing development environments and version control systems to further streamline the software engineering process. Additionally, investigating the potential of HyperAgent in more specialized domains, such as security-focused code review or performance optimization, could expand its applicability. Enhancing the system’s explainability and providing more detailed insights into its decision-making process could also improve trust and adoption among developers. Finally, exploring techniques to continually update and refine the system’s knowledge base with the latest programming paradigms and best practices could ensure its long-term relevance in the rapidly evolving field of software engineering.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: The result and analysis shows the developed system works well, how ever it is not clear how generalizability and scalability is presented in the paper. It is recommend that authors clearly and explicitly explain how you achieved generality, efficiency, and scalability, what experiments you conducted, and how you evaluated the result section. Or/and add  section on generalizability and scalability experiments, because as for "scalability", there are 3.2 AGENT COMMUNICATION AND SCALABILITY, but ther eare no such description on other 2 features.
soundness: 3
strengths: It showed the avility to solve multipole software engineering tasks such as issue resolution, code generation, fault localizaiton and repair. The design enable users to use lightweight (cost efficient) LLM for navigation agent and strong LLM for other agents, depending on the problem.
summary: This paper presents LLM based multi-agent system for various SE tasks, consisting of Navigator, Editor, Executor and Planner to control there 3 agent modules.
The paper shows the versatility of HyperAgent by applying 3 tasks, i.e. issue resolution, code generation, fault localization and repair, along with comparison with SOTA models, and how it outperforms other methods. The authors also provide a detailed analysis of agent roles, tool design and behaviour,
weaknesses: Implimantation details is not clear until the appendix. For example, Footenote 1 on  page 2 says" details of each agent are writtin in Sec4". But  Sec.4 is too short to understand. This is not "detail", at least "Implimentation summary" or "overview". Please add  more details on the architecture of each agent, their interactions, or the key algorithms/prompt used, for example.

How much the system is scalable and how did it implimented is not clear. From sec 3.2 denoted it is scalable because of using message queue. But it is not clear how it is related to scalablity. I understand that MessageQueue is designed for large system integration, but not show how it is related to "numerous subtasks, handlig complix tasks efficiently", which the authors wrote on Introduction.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Is the Planner sequential or can it also dispatch parallel queries to multiple agents.
soundness: 3
strengths: The paper provides a clear framework for decomposing software engineering tasks and solving these in a specialised manner. Specifically, the tools proposed by the paper, as well as how they are called (writing python code) are useful contributions. The ablations show tool design adds significant value to the framework, particularly the choice of what controls are given to the LLM (e.g. code_search, keyword summaries, and repair editor). The paper provides broad evaluation on both generation tasks (RepoExec) as well as bug-fixing (Defects4J and SWE-bench) which support claims of generalisability.
summary: The paper proposes an LLM agent approach for software engineering tasks. It identifies that SE tasks often involve a combination of localisation, editing and execution, all orchestrated by some higher-level plan. A framework made up of modular components for each of these is proposed where each component is a ReAct style LLM agent loop with access to specialised tools. Through positive results on software generation (RepoExec) and bug-fixing (SWE-bench and Defects4J), the paper aims to show generalisability and efficiency (in terms of API costs).
weaknesses: 1. The paper lacks thorough comparison to prior approaches that employ similar multi-stage approaches. CodeR [1] uses a multiagent approach for with seperate manager, localiser, editor and verifier (very similar to the components of Hyperagent), achieving 28.3% on SWE-bench-lite. MASAI [2] uses a modular architecture with similar components (without a manager) achieving 28.3% on SWE-bench-lite. Approaches such as AutoCodeRover (v20240620) (30%) [3] and Aider (26.3%) [3] both have phases for localisation and editing. Overall, the novelty of such a multi-stage approach is questionable, especially since the performance on SWE-bench is comparable to similar approaches. 

2. The paper also does not provide sufficient evidence as to why a multi-stage approach in general is more "generalisable". The performance numbers quoted for agentless (24.3%) seem out of date, with Agentless + GPT 4o (2024-05-13) achieving 27.3%. While it is understandable that comparing with all new entries on swebench is unreasonable, comparing with one of the many strong non multi-stage approaches both quantitatively and qualitatively would do much to strengthen the papers claims. 

In general, while the paper demonstrates positive results of their approach on two different kinds of tasks, the novelty of the approach as well as comparisons with prior work are lacking.

[1] https://arxiv.org/pdf/2406.01304
[2] https://arxiv.org/pdf/2406.11638
[3] https://www.swebench.com/

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Line 42: “their claim of addressing general software engineering tasks is overstated”. Why is this the case? Given that the downstream tasks you evaluate on are all evaluated on Python or Java, how does HyperAgent resolve the shortcomings of more versatility across tasks, languages, and dev. Scenarios?
- Lines 45 - 104: You introduce a workflow and claim that this is representative of most of software engineering - what evidence is there to reflect this? This seems very reminiscent of the issue resolution pipeline that prior works already introduce. Also, “Although different SE tasks require varied approaches, they all follow a similar workflow.” - Are there citations or prior evidence from existing SE works that reflect that the workflow you discuss is in fact the most reflective? This part of the introduction feels quite anecdotal.
- Line 99 - How are your three advantages made clear in your evaluation? Aside from higher performance, I do not see why existing works are less generalizable than HyperAgent, or why these agents are more efficient or scalable.
- Line 116 - What is SWE-bench-Python, RepoExec-Python, and Defects4J-Java? Are these variants of the original benchmark?
- Line 118 - SWE-agent and OpenDevin were both evaluated on non SWE-bench tasks, including HumanEval, HumanEvalFix, Bird, BioCoder, WebArena, and much more. The claim that this is the first system to work on many SE tasks feels like an over-claim.
- Line 254 - “A task can *be* defined…”
- Line 251 - Missing space between “HyperAgent” and “is”
- Line 328 - What does processing time account for? Does this include or exclude the calls to the API? What’s the variance on these numbers?
soundness: 2
strengths: The HyperAgent’s ability to generalize to multiple benchmarks is its main strong suite in my opinion. The adaptation of existing multi-agent methods to a larger span of software engineering tasks is great to see, and it is a good confirmation of the methodologies suggested by prior works. This work explicitly identifies clear stages of software engineering, then demonstrates how a multi-agent framework designed around these stages is extensible to many tasks. In addition to the implementation, HyperAgent demonstrates comparable performance to existing solutions across three benchmarks.
summary: This work introduces HyperAgent, a multi-agent software engineering system that leverages Language Models. The HyperAgent system can be adapted to work on multiple Software Engineering benchmarks, specifically SWE-bench, RepoExec, and Defects4J. HyperAgent consists of one “parent” agent (the Planner) that can invoke one of three “child” agents (Navigator, Code Editor, Executor). These child agents can be invoked in parallel in any order by the Planner. The implementation also includes a suite of tools tailored to each child agent, along with flexible problem templating that allows for easy adaptation of HyperAgent to different coding tasks. HyperAgent performs comparably on several benchmarks, and the authors put forth several ablations examining how HyperAgent performs on such tasks.
weaknesses: While the performance improvement of HyperAgent is admirable and the task coverage is quite impressive, I think it’s not quite clear what distinguishes HyperAGent from prior work. The methodology feels like a slight adaptation of existing multi-agent systems (e.g., MetaGPT) for the SWE-bench benchmark. The ablations, while interesting, look very similar to the ablations presented in SWE-agent. It is great that HyperAgent demonstrates comparable performance on several software engineering and coding benchmarks, and while this feels like a good engineering contribution, it remains unclear to me what new research questions or conclusions this exploration can offer up. I believe that more discussion about how HyperAgent’s architecture and methodology differs from prior work, along with analyses that back up these conclusions would make this paper a stronger research contribution.

- Section 2 Related Work: I would’ve liked more investment into a discussion of how HyperAgent differs from prior works, particularly in the second half of Section 2.2. After reading Sections 1 and 2, it is still not clear to me what new research questions HyperAgent addresses or what it does that truly distinguishes it from prior work.
- Section 3: Several comments - (1) How is this approach different from existing work such as MetaGPT, which arguably has more complex multi-agent systems following the Standard Operating Procedure framework? (2) Why is Multi-agent better than existing single agent approaches? How does having multiple agents lower inference costs or eliminate redundant information (wouldn’t the same information have to be communicated to multiple agents several times?) Also, it sounds like multiple child agents can be spawned in parallel - how is this more efficient than single agents, and why is the performance better because of this?
= The engineering details discussed in Section 3 sound quite impressive, particularly the use of the Message Queue and the general coordination of multiple agents. However, it is left to the reader to determine why these tools or multi-agent systems are interesting or novel relative to prior work. For instance, for Section 3.3, it is cool that these tools are made available, but unclear why this is better than the search and localization tools that prior works introduce (e.g. edit, search_dir, search_file in SWE-agent).
- Section 6 - These analyses are almost identical to the ones in SWE-agent. While this is interesting information, the paper may be stronger if there were more novel examinations that demonstrate how HyperAgent is distinct from prior work. For instance, “6.2 Analysis of Tool Design” is very similar to Table 3 in the SWE-agent paper. “6.3 Agent Behavior” is the same kind of graph as SWE-agent Figure 7. Analysis 6.1 is interesting, but it would be nice to know why the performance drops are not equal, and how certain tasks are still resolved even when there is no Editor, Navigator, or Executor.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - Can the authors compare HyperAgent to Agentless (on SWEBench-Verified) and ChatRepair (on Defects4J)? Specifically, could you include average runtimes for Agentless and SWEAgent on SWEBench?
- For both RepoExec and Defects4J, can the authors include the results for at least one of the Full configurations? This will help with understanding the scalability of the approach to better models.

Minor clarifications which I believe could enhance the readability of the paper:

> For example, multiple Navigator instances can explore different parts of a large codebase in parallel, the Editor can apply changes across multiple files simultaneously, and the Executor can run tests concurrently, accelerating validation.

- Can the authors clarify if there are specific design decisions made to ensure synchronization between agents preventing them from modifying the same files?

> Failed tasks are requeued for reliability
> HYPERAGENT also has a problem of early exit (due to hallucination that the task has been
solved)

- As pointed out by the authors, it could be challenging for the LLMs to determine success/failure of task/subtasks. Can you clarify how are failures detected for requeuing?

- What are the details of HyperAgent-lite-3?

- Explored the sharing of previous context with a specific agent?

- Can the authors clarify if they intend to make the HyperAgent source code open source?

- Can the authors include a discussion about LSPToolKit and if there are specific design choices that were made to better align it as a tool for LLMs? Specifically, can the authors compare with some of the existing approaches that use LSP to build tools for LLMs. This could be instructive for devtool builders.
soundness: 3
strengths: The paper demonstrates a generalized software engineering agent. As such, generalizability of the approach, and strong demonstration (even if not state-of-the-art on individual benchmarks) across 3 different tasks is the main strength of the paper. The proposed multi-agent breakdown is simple, while achieving good performance, with ablations supporting the need for each component as well. 

The paper also demonstrates the usefulness of multi-LLM multi-agent system through the HyperAgent-Lite configurations.

The application of dynamic message queues for asynchronous communication between the planner and child agents allows parallelizability and reduction in runtimes.

Further, the authors also identify the difficulty faced by LLM agents in using various IDE provided tools (like go_to_definition, which for example, actually reduced the resolve rate), and came up with augmentations to improve the tool (w/ search ablation in go_to_definition, etc.), which is instructive. This is particularly insightful, and points to further research in the direction of building better abstractions around existing code-tools that will be more suitable for LLMs.

The paper is written clearly, and easy-to-understand.
summary: The paper introduces HyperAgent, a multi-agent system consisting of 4 primary agents (Planner, Navigator, Code Editor and Executor) organised in a Star-topology, to achieve generalizable software engineering capabilities across languages (Java and Python) and tasks (Issue-PR resolution over SWE-Bench, Bug localization and Repair over Defects4J, and repository-level code generation over RepoExec.). The planner agent creates subtasks and issues messages over async queues to each of the other 3 agents, which then try to execute the subtask. An LLM summarizer condenses the child agent trajectory to be sent back to the planner.

The different agents in the system can be instantiated with different LLMs, providing the ability to achieve a cost/capability tradeoff depending on the difficulty of the task.

The authors present results on 3 different software engineering tasks (end-to-end issue resolution, bug localization, bug repair, code generation) to demonstrate the generalizability of the proposed system.
weaknesses: I have two primary concerns with the paper:
- *Effectiveness of multi-agent setup*: The proposed system drives the complete workflow with an LLM agent (the planner), which is in direct contrast to the proposal by Agentless, which achieves better results than the best HyperAgent configuration on SWEBench-Verified, while also being much more cost effective (\\$0.34 for Agentless vs \\$1.82 for HyperAgent). This calls into question whether a comparatively complex agentic system is required for the software engineering task.
- *Evaluation baseline for Defects4J* - I would suggest the authors compare HyperAgent to ChatRepair [1], which reports 114 solved cases on D4J v1.2. I seek to understand how HyperAgent compares to ChatRepair and if the proposed multi-agent approach offers benefits above the ChatRepair. 

[1] Xia, C. S., & Zhang, L. (2023). Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. arXiv preprint arXiv:2304.00385.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. For the executor, how does it find appropriate unit tests to execute? If defect fixing requires introducing unit tests that do not currently exist in the project, how does this agent function in such scenarios? This is a major shortcoming of the paper, and it lacks a detailed analysis in this area.
2. In Section 5.1, how does HypeAgent perform compared to other models on SWE-agent?
3. In Section 5.3, how do other open-source agents in Section 5.1 perform on this dataset?
soundness: 2
strengths: 1. This paper designs a multi-agent system with four components: an agent for planning implementation, an agent for retrieving file content within the project, an agent for making code modifications, and an agent for executing unit tests.
2. The overall process control of these agents is managed by a planner, and corresponding APIs are defined for each agent to interact with the project.
3. This paper conducts relatively solid experiments on three datasets.
summary: This paper proposes a multi-agent framework to address general issues in software engineering development. The multi-agent system is designed with four components: an agent for planning implementation, an agent for retrieving file content within the project, an agent for making code modifications, and an agent for executing unit tests. The overall process control of these agents is managed by a planner, and corresponding APIs are defined for each agent to interact with the project. Experiments were conducted on three datasets: SWE-bench, RepoExec, and Defects4J. However, the final experimental results did not demonstrate that the proposed method is significantly stronger than other baseline methods.
weaknesses: 1. Compared to previous work, such as ACR and SWE-Agent, the innovation in this paper is primarily the addition of an Execution Agent component; the other agents are almost identical to those in previous work.
2. The HypeAgent proposed in this paper does not show significant improvements over previous methods in the final experimental results.
3. The HypeAgent proposed in this paper does not compare its fault localization performance against SWE-Agent.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Please see weakness section above.
soundness: 3
strengths: * The paper focusses on an important problem of developing generalist agents without being tailored for specific SWE or coding tasks (e.g., competition coding etc)
* The combination of subagents with asynchronous message queues for improving subtask efficiency and parallel execution (e.g., parallel editing across multiple files) is also interesting and novel
* Quantitative evaluations across multiple benchmarks are provided to demonstrate the generalizability across different coding tasks and programming languages.
summary: The paper focusses on the recently increasingly popular area of developing autonomous software agents for coding tasks. In particular, the paper focusses on the development of a software agent for generalizability across different tasks (github issue, repair etc) and languages (python, java). To this end, the paper proposes Hyper-Agent which uses four specialized agents - planner, navigator, code editor and finally code executor for scaling to generic SWE tasks. The planner agent acts as the main agent and is responsible to collaborating with multiple specialized agent in an asynchronous manner, in order to perform the final task. The specialized agents improve subtask execution performance while the asynchronous queue based execution improves throughput efficiency. Quantitative evaluations for three SWE benchmarks are provided in order to prove the efficacy of the proposed approach.
weaknesses: While the key idea and problem is interesting, I have some questions regarding some experimental results and details,
* The idea of using the same base-LLM to act as different subagents has been previously explored before [1,2]
* How are the subagents specialized? Is the difference primarily form using different input contexts / system prompts, tools etc.
* The paper claims that the final agent is generalist and applicable across diverse tasks. However, its not clear if the context and implementation for each task is specialized or not? 
      - For instance, when solving the swebench like tasks, the original swe-agent implementation provides task specific instructions / tips (e.g., reproducing the issue) in the input context. Is a similar strategy also applied for  hyper-agent?
* Also regarding the experimental evaluations, are the baselines and the proposed approach using the same base-llm (e.g, Tab. 1)?
    - In particular, according to Tab. 1, 7: the best performing hyperagent configuration using claude-3-sonnet, however some of the baseline results are reported with gpt4o.
* Also the numbers with some of the baselines seem off: autocoderover with gpt4o performs 30.67% on swebench-lite and 38.40% on swebench-verified (both higher than hyperagent). However, the reported numbers for the same in Tab. 1 seem to be much lower.
* Finally, while a minor point, I had some concerns regarding the usability in terms of avg. cost which is for instance 1.82 on swebench for the best hyperagent configuration. In contrast, a simple 3 stage model (localization, repair, rerank) with agentless achieves $0.34 per instance on average while achieving better solve-rate. 
* How are the ablation studies performed in Sec. 6? For instance, w/o navigator which agent is used for code navigation the planner agent or any other subagent.
* Are there some incontext examples teaching which subagent should be called by the planner agent? If not, is it mostly reliant on the functon calling capability of the main planner agent?

References:

[1] Meta{GPT}: Meta Programming for A Multi-Agent Collaborative Framework, Hong et al, 2024

[2] OpenDevin: An Open Platform for AI Software Developers as Generalist Agents, Wang et al, 2024

Meta-review: The primary finding of the paper is that an LLM agent architecture which improves benchmark performance on repository-level software engineering tasks. The primary strength is that it it is better than baselines. The primary weakness is that it is conceptually similar to prior agents, and not strong enough empirically relative to baselines to warrant publication at this stage. I worry, like other reviewers, that these slim SOTA claims are fragile, and derive more from benchmark specifics than broad scientific principles or novel conceptual insight.

Decision: Reject