Integrating Expertise of Software Engineering Agents

## Abstract

Abstract Large language model (LLM) agents have shown great potential in solving real-world software engineering (SWE) problems. The most advanced open-source SWE agent can resolve over 27% of real GitHub issues in SWE-Bench Lite. However, these sophisticated agent frameworks exhibit varying strengths, excelling in certain tasks while underperforming in others. To fully harness the diversity of these agents, we propose DEI (Diversity Empowered Intelligence), a framework that leverages their unique expertise. DEI functions as a meta-module atop existing SWE agent frameworks, managing agent collectives for enhanced problem-solving. Experimental results show that a DEI -guided committee of agents is able to surpass the best individual agent’s performance by a large margin.
For instance, a group of open-source SWE agents, with a maximum individual resolve rate of 27.3% on SWE-Bench Lite, can achieve a 34.3% resolve rate with DEI , making a 25 % percent 25 25\% 25 % improvement and beating most closed-source solutions. Our best-performing group excels with a 55% resolve rate, securing the highest ranking on SWE-Bench Lite. Our findings contribute to the growing body of research on collaborative AI systems and their potential to solve complex software engineering challenges.

## 1 Introduction

Recent advancements in large language models (LLMs) have transformed software engineering (SWE) and other domains. Originally developed as chatbots (Schulman et al., ; Team, ), LLMs have evolved into the core of AI agents, capable of understanding and generating human-like conversations, as well as autonomously executing actions in both real-world and digital environments. SWE agents, a specialized subset of these AI agents, integrate these capabilities with software engineering tools and techniques for tasks like code generation, automated testing, and project management, aiming to identify and resolve practical software issues (Zhang et al., ).

In this paper, we study one specific task of SWE agents – resolving real-world GitHub issues based on their descriptions. Automatically fixing a bug in a code repository is an extremely challenging task that involves navigating extensive codebases, understanding complex function interactions, detecting subtle errors, and generating the correct fix patch. The large action space of SWE agents, together with long trajectories, inevitably result in the diversity of Github issue solutions, as shown in Figure 1. We have observed that different SWE agents resolve very different sets of issues (the colored girds in Figure 1a), despite having similar resolve rates (Figure 1b).
This is probably due to different skill sets of SWE agents.
For instance, OpenDevin (Wang et al., ) explicitly instructs the LLM to first replicate the bug in an issue and executes its replication in a development workspace to provide feedback for its generated patches, but other agents like Moatless Tools (Örwall, ) and Agentless (Örwall, ) do not actually execute code in the issue-specific repository.

A garden’s beauty never lies in one flower. Diversity in all its forms is the path to greatness.

Similarly, the trend in the SWE agent community reflects this diversity—no single agent framework dominates in all capabilities. It is the flourishing variety within this community that sparks new ideas and leads to the development of better agents.

The variety in SWE agent capabilities inspires us to develop DEI, Diversity Empowered Intelligence, a framework that leverages the strengths of diverse agents. DEI aims to harness these varied skills to tackle a broader range of problems more effectively with a multi-agent ensemble system and a re-ranking pipeline, as showcased in Figure 1c. DEI functions as a meta-module that can be integrated with any existing agent framework, enabling scalable management and collaboration among agents to form a more powerful multi-agent software engineering organization.

Figure: Figure 1: Different SWE agents (Aider, Moatless, Agentless, OpenDevin) resolve very different sets of issues (the colored girds in Fig 1a), despite having similar resolve rates (Fig 1b). Our proposed DEI Committee takes candidates patches and tries to select the best, oracle choice (Fig 1c), improving the resolve rate significantly to be better than any single agent in the committee.
Refer to caption: x1.png

We evaluate DEI on 7 groups of candidate agents on SWE-Bench Lite.
3 of the 7 are different runs of a single open-source SWE agent.
The other 4 are different agents that are on the SWE-Bench Lite leaderboard, including a group that contains only open-source agents.
Through experiments, we find that different agents show a great level of diversity in the issues they resolve: a group of agents with an average resolve rate of 26.6% can collectively solve 54.3% of issues if there is an oracle reviewer that can consistently select the best candidate.
DEI, as a first step towards harnessing such diversity, can improve the group’s resolve rate to 34.3% ($\uparrow 25\%$), suggesting that LLMs are great code reviewers.
These findings mirror the benefits of diversity in the tech industry, where diverse perspectives and skills lead to greater innovation and problem-solving capabilities.

To summarize, our contributions are the following:

- •
For the first time, we comprehensively evaluate the diversity of solutions provided by SWE agents, revealing significant differences in the types of GitHub issues resolved by various agents, despite similar overall resolve rates. These findings suggest a substantial potential to improve overall performance by more effectively leveraging the diverse expertise of these agents together.
- •
This paper introduces DEI, a multi-agent meta-policy module designed to harness the diversity of SWE agents and seamlessly facilitate collaboration among agents with different specialties. By employing a multi-stage rating and re-ranking pipeline, DEI consistently improves issue resolution, demonstrating a 25% performance boost on the SWE-Bench Lite leaderboard.

## 2 Related Work

We review the work in fundamental language agent architecture, recent developments for SWE agents, and multi-agent or ensemble methods in this section.

##### Fundamental Language Agent

Pioneering AI agent methods along this line of work include ReAct (Yao et al., ), Reflexion (Shinn et al., ), CodeAct (Wang et al., ), etc., in which ReAct interprets the user query, generates functional API calls, and gets the tool outputs in real time; Reflexion further appends failed trial experience into the memory, enabling effective retrials to prevent repetitive errors. CodeAct (Wang et al., ), instead of generating function calls, uses code generation to consolidate AI agents’ actions into a unified action space.

##### Software Engineering Agent

We present the SWE agents which have disclosed the technical details on the SWE-bench Lite leaderboard. Alibaba Lingma Agent (Ma et al., ) constructs a repository knowledge graph to represent code and dependencies, using a Monte Carlo tree search-based strategy for repository exploration, and generates patches to address real-world GitHub issues. AutoCodeRover (Zhang et al., ) adds advanced code search tools, such as abstract syntax trees, and spectrum-based fault localization to the agent for enhancing context understanding and issue resolution. Code-R (Chen et al., ) chooses a multi-agent framework with pre-defined task graphs to resolve Github issues. Agentless (Xia et al., ), is a simplified two-phase approach for solving software development problems. It focused on localization and repair without relying on LLMs to make decisions, highlighting the potential of straightforward techniques in autonomous software development. OpenDevin (Wang et al., ) is a hub of community-contributed agents including CodeAct (Wang et al., ), browser agent, GPTSwarm (Zhuge et al., ), and task-specific micro agents. Finally, SWE-agent (Yang et al., ) developed agent-computer interface that consists of LM-friendly commands and environment feedback to enable LM agents to autonomously use computers to
solve software engineering tasks.

##### Multi and Ensemble Agents

Recent works observe that organizing multiple specialized AI agents (Hong et al., ; Li et al., ; Liu et al., ) enable the task decomposition ability of an agent system, which improves the task-resolving performance.
Current multi-agent frameworks are categorized into three types based on their execution patterns.
Firstly, static agent working flow (Wu et al., ; Github, ), which pre-defines the agent execution flows and ignites agent transitions via specified conditions.
Controlling a multi-agent system with pre-determined states is robust, though losing flexibility in terms of unseen states or conditions.
Secondly, ensemble via group chatting (Wu et al., ; Hong et al., ; Wang et al., ; Chen et al., ).
This is built upon an environment where
multiple agents send messages to each other in a group channel such that their thoughts are ensembled.
Variants of group chatting includes debating (Liang et al., ; Chan et al., ) and model-wise ensembling (Wang et al., ).
Last but not least, hierarchical task assignment (Liu et al., ; ).
Organizing multi-agent in a hierarchical structure benefits the top-down task decomposition and thus enables efficient multi-agent collaboration.

## 3 Integrating Expertise of SWE Agents

### 3.1 Background

Resolving issues in SWE-Bench.
One important task in software engineering is to resolve issues raised by developers.
SWE-Bench curates instances of this task by collecting successfully resolved issues from open-source repositories on Github.
Each instance in SWE-Bench consists of a textual issue description, a version of the repo just before the issue was resolved, and (hidden) unit tests that went from fail to pass after the human-written patch.
To resolve an instance, the model is required to generate a patch that can pass these unit tests.

SWE Agents. In this paper, we use the term “SWE agents”(^1^11According to our definition, SWE-agent (Yang et al., ) is an instance of SWE agents, and Agentless (Xia et al., ), despite the name, is another.)
to refer to any LLM-based system that generates patches to solve issues in a code base, e.g., an instance in SWE-Bench.
While the specific implementation varies, a typical SWE agent usually gives their underlying LLM several tools in the form of callable functions to navigate through the code base, find relevant context, edit files, and run tests.
The workflow of SWE agents often involves multiple LLM calls, each taking some or all outputs from previous steps as input.

### 3.2 Diversity of SWE Agents

We consider two types of diversity: intra-agent diversity and inter-agent diversity.

Intra-agent diversity is defined as the degree to which different runs of the same agent solve different problem instances.
It is most likely from the non-determinism of the underlying LLM due to sampling in decoding and mixture-of-experts architecture (Chann, ).
Since the workflow of SWE agents involves multiple steps and LLM calls, a slight difference in an earlier step can easily propagate and result in significant differences in the final outcome.

Inter-agent diversity is defined as the degree to which different agents solve different problem instances.
Besides sharing the potential causes of intra-agent diversity, inter-agent diversity is also largely because of differences in agent design, including different tools, workflows, and prompts.

### 3.3 Approach

#### 3.3.1 SWE Agent Problem Formulation

We formulate the SWE agent problem under the contextual Markov decision process (CMDP) framework (Hallak et al., ), represented by the tuple $\mathcal{M}=(\mathcal{S},\mathcal{C},\mathcal{A},\mathcal{R},\mathcal{P},p_{0}
,\rho)$. Here, $\mathcal{S}$ denotes the state space, which encompasses all possible states the agent could encounter, such as the current status of files. The context space, $\mathcal{C}$, includes relevant repository information and issue descriptions. The action space, $\mathcal{A}$, represents all potential actions or tools the SWE agent can utilize, such as search or editing. The context-dependent reward function, $\mathcal{R}:\mathcal{S}\times\mathcal{A}\times\mathcal{C}\to\mathbb{R}$, assigns scores based on the actions taken by the agent. For instance, the reward is high if the agent successfully addresses an issue, while it is low if the action results in new bugs in the repository.
The context-dependent transition function, $\mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{C}\to\Delta(\mathcal{S})$, defines how the state of the repository or information changes following a specific action. The context-dependent initial state distribution is denoted by $p_{0}:\mathcal{C}\to\Delta(\mathcal{S})$, and $\rho\in\Delta(\mathcal{C})$ represents the context distribution.

Given the initial context $c\sim\rho$ and initial state $s_{0}\sim p_{0}(\cdot\mid c)$, at each time step $t$, the agent follows a policy $\pi:\mathcal{S}\times\mathcal{C}\to\Delta(\mathcal{A})$ to select an action $a_{t}\sim\pi(s_{t},c)$ and receives a reward $\mathcal{R}(s_{t},a_{t},c)$. The environment then transitions to the next state $s_{t+1}\sim\mathcal{P}(\cdot\mid s_{t},a_{t},c)$, providing the agent with a new state observation.
As the iteration progresses to time $T$, a sampled trajectory $\tau:=\{s_{t},a_{t},r_{t}\}_{t=0}^{T}$ is obtained. The objective of an SWE agent is to maximize the cumulative reward along the trajectory, which is captured by the value function:

$$ $\max_{\pi}V^{\pi}(\rho)=\max_{\pi}\mathbb{E}_{\tau}\left[\sum_{t=0}^{T} \mathcal{R}(s_{t},a_{t},c)\mid c\sim\rho;\pi\right]$ (1) $$

#### 3.3.2 Our Framework: Diversity Empowered Intelligence (DEI)

Many efforts have been made to implement sophisticated agent systems that aim to achieve the objective described in Eq. 1. However, as discussed in Section 1, these systems often exhibit varying levels of effectiveness across different contexts. It is challenging to devise a single agent that can consistently perform well across all possible contexts.

Formally, suppose there are $N$ agent policies, denoted as $\{\pi_{1},\pi_{2},\dots,\pi_{N}\}$, where each policy is tailored to address a specific context $\{\rho_{1},\rho_{2},\dots,\rho_{N}\}$. The union of these contexts is a subset of the entire context space, i.e., $\rho_{1}\cup\rho_{2}\cup\dots\cup\rho_{N}\subseteq\rho$. For each agent policy $\pi_{i}$, the objective is:

$$ $\pi_{i}=\max_{\pi}\mathbb{E}_{\tau}\left[\sum_{t=0}^{T}\mathcal{R}(s_{t},a_{t} ,c)\mid c\sim\rho_{i};\pi\right].$ (2) $$

However, an agent policy $\pi_{i}$ may perform poorly in a different context $\rho_{j}$ (where $j\neq i$). To address this limitation, we propose our framework: Diversity Empowered Intelligence (DEI). The DEI framework leverages the strengths of each agent in their respective contexts to enhance overall performance across all contexts.

We introduce a meta-policy, denoted as $\pi_{\text{DEI}}$, which aims to optimally select among the available agent policies based on the context. The goal of $\pi_{\text{DEI}}$ is defined as:

$$ $\pi_{\text{DEI}}=\max_{\pi}\mathbb{E}_{c\sim\rho}\left[\mathbb{E}_{\tau}\left[ \sum_{t=0}^{T}\mathcal{R}(s_{t},a_{t},c)\mid c;\pi(c)\right]\right],$ (3) $$

where $\pi(c)$ denotes the selection of the optimal agent policy from $\{\pi_{1},\pi_{2},\dots,\pi_{N}\}$ based on the observed context $c$. By dynamically choosing the most suitable agent policy for each context, the DEI framework seeks to maximize the expected cumulative reward across all possible contexts.

#### 3.3.3 DeiBase : A Simple yet Effective Implementation

We present DeiBase, a simple yet powerful implementation of the DEI framework, tailored for SWE-Bench like problems. The context in the setup includes the repository, along with relevant files and issue descriptions. The meta-policy’s action space consists of the final patches generated by different agent frameworks, each specialized in addressing various aspects of the problem.

DeiBase utilizes a Large Language Model (LLM) as a code review committee. The LLM evaluates candidate patches by analyzing the state of the code base before and after the proposed changes, in conjunction with the contextual information from the issue descriptions. It produces detailed explanations for each patch, justifying the modifications based on the identified issues, the context, and the specific changes made.

While other methods of code review and scoring, such as rule-based approaches, can be incorporated into our framework, the use of an LLM-based committee offers a unique advantage. LLMs often excel at evaluating solutions when evaluation is easier than generation. DeiBase thus serves as an effective baseline for LLM-based SWE evaluation, highlighting potential performance variations among diverse SWE agents and showcasing the capabilities of our method.

Figure: Figure 2: Framework Overview. DEI first examines the code base before and after a candidate patch, along with other relevant contexts. Then, it generates an explanation for the issue, the context, and the patch and tries to justify the patch. With its own explanation, it scores the candidate patches and picks the top-scoring ones as more likely to be correct.
Refer to caption: x2.png

As demonstrated in Figure 2, DeiBase is given multiple candidate patches for a single issue.
These patches might be from running a single SWE agent multiple times or running multiple SWE agents.
DeiBase gives each candidate patch a score and then selects the top-scoring candidates as the patches most likely to work.

Step 1: Input Construction. Four inputs are given to DeiBase for each patch: the issue description itself, relevant context (code snippets identified by an SWE agent as relevant to the issue), code before the patch, and code after the patch.
The form of inputs reflects two design choices. First, the entire repository is often too large to fit directly in the context limit of LLMs, so we use the relevant context instead to save token costs and help the model focus.
Second, the format of a patch is not the easiest for an LLM to read as it switches back and forth between the pre-change code and the changed code, so we give the code before and after the patch separately to the model for easier understanding.
In practice, we directly use the relevant code spans identified by Moatless Tools, an open-source SWE Agent (Örwall, ).
There might be potential ways of improving the quality of relevant code spans by making them specific to both the issue and the candidate patch, rather than solely dependent on the issue itself.

Step 2: Explanation Generation.
To help the model better “understand” the patch before scoring, we instruct it to generate various explanations regarding the patch in a specified order.
The order is decided so that the earlier explanations can also help the later ones.
We describe each explanation in the order they are generated here:
1) Issue explanation explains what the issue is about and what problem it may be causing.
2) Context explanation explains how and why each relevant code span (there might be many of these) is relevant to the issue.
3) Location explanation explains if and why the patch is modifying the correct part of the code that’s faulty.
4) Patch explanation explains if and how the patch is fixing the issue.
5) Conflict detection is about checking whether the patch conflicts with other relevant code snippets.
We explicitly instruct the model to refer back to the earlier explanations while generating the later ones.

Step 3: Patch Scoring. Based on its own explanations, the model is asked to give the candidate patch a score of 1 to 10. We give the model detailed rubrics of what violations/mistakes lead to higher score deductions and what should only be considered minor violations.
For example, if the model finds the modification location to be wrong, it is considered a serious mistake.

## 4 Experiments

We aim to answer two research questions with our experiments: 1) How diverse are LLM-based SWE agents in terms of intra- and inter-agent diversity? 2) To what extent can DEI harness the diversity and increase the performances of these SWE agents?

### 4.1 Experiment Setup

#### 4.1.1 Benchmark and Agents

Benchmark. We conduct our experiments on SWE-Bench Lite, a 300-instance subset sampled from the full SWE-Bench for providing a more self-contained evaluation of functional bug fixes (Jimenez et al., ).
Compared to the full SWE-Bench, SWE-Bench Lite has significantly more submissions on the leaderboard for us to conduct a more comprehensive analysis of inter-agent diversity.

Agents.
For intra-agent diversity, we consider three well-performing open-source agents on the SWE-Bench Lite leaderboard: Agentless (Xia et al., ), Moatless (Örwall, ), and Aider (Gauthier, ) by running them 10 times with the same parameters.
For inter-agent diversity, we consider 10 agents (open-source or not) that have similar resolve rates, all between 26.0% and 31.0% on the leaderboard by directly using their submitted patches to the SWE-Bench issues.
For the evaluation of DeiBase on different agents, we consider 3 groups of agents that form different DEI Committees, including one group consisting of only open-source agents.
For the evaluation of DeiBase on multiple runs of a single agent, we use the generations of the three aforementioned agents – Agentless, Moatless Tools, and Aider.
More details can be found in Appendix A.1.

#### 4.1.2 Evaluation Metrics

We use the same set of metrics for both intra- and inter-agent diversity as these metrics are defined for multiple candidate solutions without requiring them to come from the same candidate:

Resolve rate measures how good a SWE agent is. It is defined as the percentage of issues resolved by the agent. We measure both single SWE agents and DEI with it to see how much DEI helps.

Union@k measures the best case performance had the agents been perfectly consistent by counting the number of problems solved by any of the $k$ solutions.
In the ideal case where the agents are perfectly consistent, Union@k should be the same as Union@1.
Union@k can also be considered as the case where we have an oracle reward function $\mathcal{R}^{\text{oracle}}$ that always selects the correct candidate.

Intersect@k measures the worst case performance by computing the number of problems solved by all $k$ solutions.
The assumption is that a problem is only consistently solved if it’s always solved.
Yao et al. () calls this metric pass^k.
Intersect@k can also be considered as the case where we have an adversarial reward function $\mathcal{R}^{\text{adv}}$ that tries to pick an incorrect candidate if there is one.

Average@k measures the average case performance by computing the average number of problems solved. It corresponds to the case of a random reward function $\mathcal{R}^{\text{random}}$ that uniformly samples a candidate solution for each problem.

n@k measures the performance of any reranking mechanism by computing the number of problems solved by $n$ chosen submissions from a total of $k$ samples.
The better a reranking mechanism is at telling good solutions from bad ones, the higher n@k is.
Note that for an oracle that always picks the correct solution over incorrect ones, n@k is the same as Union@k.
For a random reranker that picks a random solution uniformly, n@k is the same as Union@n.
In our case, we evaluate $n=1$.

Our research questions can be answered by the gaps between these metrics.
Union@k - Intersect@ measures how diverse the agents are, while n@k - Average@k measures how much DEI helps in selecting the correct candidate from these agents.
Note that the order – in which different runs are added – matters as $k$ gets larger, especially when the $k$ candidate solutions come from $k$ different agents.
In our experiments, we add candidate solutions from the single agent according to the order they are generated, while we add solutions from different agents in a fixed order (see Appendix A.1).

### 4.2 Main Results

#### 4.2.1 Research Question 1: How diverse are LLM-based SWE agents?

To answer this question, we report the “@k” metrics of 10 different agents and 10 runs of single agents in Figure 3.
The detailed values of these metrics can also be found in Table 2.

Figure: Figure 3: How different metrics change as more candidate solutions are involved. In all 4 scenarios, there is a huge gap between Union@k and Average@k.
Refer to caption: x3.png

Several observations can be made about the results:

SWE agents resolve very different sets of issues across agents and agent runs. Their full potential is far from fully released.
In all four subfigures, the gap between Union@k and Average@k, as well as between Average@k and Intersect@k, is large.
As $k$ – the number of candidates – gets larger, the gap also gets larger.
For 2 of the 4 settings, Union@k is more than 2x larger than Average@k for $k=10$.
The other 2, Union@k is more than 1.5x larger than Average@k for $k=10$.
This indicates that current SWE agents are potentially capable of resolving a lot more issues, as long as we have a reranker that can tell which candidates are correct.

Different agents resolve more distinct issues than different runs of a single agent. In other words, diversity does empower intelligence.
The absolute/relative difference between Union@k and Average@k is much larger in the first subfigure than in the following three subfigures.
For the “10 different agents” setting, as $k$ approaches 10, the distinct issues resolved are $2\times$ the average number of issues resolved by a single agent in the group.

**Table 1: Resolve rates of top-ranking submissions on SWE-Bench Lite. We evaluate 3 DEI Committees formed by different groups of agents. Each DEI Committee outperforms the best agent in it significantly. DeiBase-Open, a committee formed by 4 open-source agents can beat many closed-source agents.**
| Dei Group | % Resolve | System | Open Src | Trajs | Open<br>Candidates | Backend LLM |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | 55.0 | Salesforce Research DeiBase-1 | ✓ | ✓ | ✗ | gpt4o |
| 1 | 50.6 | Cosine Genie | ✗ | ✗ | - | “Fine-tuned OpenAI” |
| 1 | 43.0 | CodeStory Aide | ✗ | ✗ | - | gpt4o, Claude 3.5 Sonnet |
| - | 38.0 | AbenteAI MentatBot | ✗ | ✗ | - | gpt4o |
| 2 | 37 | Salesforce Research DeiBase-2 | ✓ | ✓ | ✗ | gpt4o |
| Open | 34.3 | Salesforce Research DeiBase-Open | ✓ | ✓ | ✓ | gpt4o |
| - | 34.0 | Bytedance MarsCode | ✗ | ✗ | - | gpt4o |
| - | 33.0 | Alibaba Lingma | ✗^1 | ✗ | - | gpt-4-1106-preview |
| 2 | 31.3 | Factory Code Droid | ✗ | ✗ | - | “Anthropic and OpenAI” |
| 2 | 30.6 | AutoCodeRover | ✗^2 | ✗ | - | gpt4o |
| 2 | 29.6 | Amazon Q Dev. | ✗ | ✗ | - | Unknown |
| 2 | 28.3 | CodeR | ✗^1 | ✗ | - | gpt-4-1106-preview |
| 2 | 28.0 | MASAI | ✗^1 | ✗ | - | Unknown |
| - | 27.6 | SIMA | ✗^1 | ✓ | ✓^3 | gpt4o |
| Open | 27.3 | Agentless | ✓ | ✓ | - | gpt4o |
| Open | 26.6 | Moatless Tools | ✓ | ✓ | - | Claude 3.5 Sonnet |
| - | 26.6 | IBM Research Agent | ✗ | ✗ | - | Unknown |
| Open | 26.3 | Aider | ✓ | ✗ | - | gpt4o, Claude 3 Opus |
| Open | 26.0 | OpenDevin + CodeAct | ✓ | ✓ | - | gpt4o |

#### 4.2.2 Research Question 2: How much does DEI help?

We apply DeiBase to the candidates in Figure 3 as they are added to the group.
Our findings are:

DeiBase helps in most cases. For most values of $k$ in all subfigures, we observe a significant improvement of n@k over Average@k, indicating that DeiBase selects correct candidates much better than a random baseline.

DeiBase helps more when the candidates come from different agents.
This finding resonates with a similar finding from research question one:
Since candidates from multiple agents have a larger potential for improvement (Union@k - Average@k), the actual improvements created by DeiBase (n@k - Average@k) are also larger.
This suggests that given a limited budget of candidates, it would be better to choose a diversity of agents over multiple runs of the same agent.

As $k$ gets larger, DeiBase’s improvement first increases and then plateaus. While larger $k$ generally indicates higher n@k, the margin gets smaller and there are cases when an increase in $k$ results in a slight drop in performance. This suggests that the current DeiBase is not ideal for a large group of agents and there is still room for a better reranking mechanism.

Based on the lessons above, we propose three DeiBase groups in which each candidate is from a different agent and no more than 5 candidates exist for each instance.
The members of these DeiBase groups and their performance are reported in Table 1.
DeiBase-1 consists of the top 2 agents.
DeiBase-2 consists of 5 closed-source agents that have high performance on the leaderboard.
DeiBase-Open consists of 4 open-source agents so that we know future researchers can run the entire pipeline.
As Table 1 shows,
all three DeiBase instances outperform the best candidate in the group.
Surprisingly, DeiBase-Open shows a 7% increase in resolve rates and beats most of the closed-source systems.

**Table 2: How different metrics change as more candidate solutions are involved. As the number of candidates $k$ gets larger, the improvement from DEI also increases significantly.**
| System | $k$ | Intersect@k | Average@k | 1@k (DEI, ours) | Improvement from DEI | Union@k |
| --- | --- | --- | --- | --- | --- | --- |
| 10 Agents | 1 | 26.7 | 26.7 | 26.7 | +0.0 | 26.7 |
| 2 | 18.3 | 27.3 | 28.7 | +1.3 | 36.3 |  |
| 3 | 15.7 | 27.7 | 32.3 | +4.7 | 42.3 |  |
| 4 | 13.0 | 27.3 | 34.0 | +6.7 | 45.0 |  |
| 5 | 10.3 | 27.3 | 35.0 | +7.7 | 48.3 |  |
| 6 | 8.3 | 26.9 | 34.3 | +7.4 | 49.0 |  |
| 7 | 6.7 | 26.8 | 35.3 | +8.6 | 52.0 |  |
| 8 | 6.7 | 26.9 | 34.7 | +7.8 | 52.7 |  |
| 9 | 6.3 | 27.3 | 35.7 | +8.4 | 53.3 |  |
| 10 | 4.7 | 26.6 | 35.7 | +9.1 | 54.3 |  |
| 10 Runs from Agentless | 1 | 20.3 | 20.3 | 20.3 | +0.0 | 20.3 |
| 2 | 14.3 | 19.5 | 22.0 | +2.5 | 24.7 |  |
| 3 | 13.7 | 20.3 | 24.0 | +3.7 | 28.0 |  |
| 4 | 12.0 | 20.3 | 22.0 | +1.7 | 29.3 |  |
| 5 | 11.7 | 20.5 | 23.7 | +3.2 | 31.3 |  |
| 6 | 11.0 | 20.4 | 23.7 | +3.3 | 32.3 |  |
| 7 | 10.7 | 20.2 | 24.3 | +4.1 | 33.0 |  |
| 8 | 9.3 | 20.0 | 25.7 | +5.6 | 34.0 |  |
| 9 | 9.0 | 20.4 | 26.0 | +5.6 | 34.0 |  |
| 10 | 9.0 | 20.4 | 26.0 | +5.6 | 34.7 |  |
| 10 Runs from Aider | 1 | 22.3 | 22.3 | 22.3 | +0.0 | 22.3 |
| 2 | 17.7 | 22.3 | 22.3 | +0.0 | 27.0 |  |
| 3 | 15.7 | 21.6 | 22.0 | +0.4 | 29.0 |  |
| 4 | 14.7 | 22.2 | 25.0 | +2.8 | 32.3 |  |
| 5 | 14.0 | 22.1 | 25.3 | +3.2 | 35.3 |  |
| 6 | 13.0 | 21.8 | 25.3 | +3.6 | 35.3 |  |
| 7 | 12.0 | 21.7 | 24.3 | +2.6 | 36.0 |  |
| 8 | 12.0 | 21.8 | 24.3 | +2.5 | 37.0 |  |
| 9 | 11.7 | 21.7 | 24.3 | +2.6 | 37.7 |  |
| 10 | 10.7 | 21.7 | 24.7 | +3.0 | 38.0 |  |
| 10 Runs from Moatless | 1 | 14.0 | 14.0 | 14.0 | +0.0 | 14.0 |
| 2 | 8.7 | 15.0 | 20.0 | +5.0 | 21.3 |  |
| 3 | 5.7 | 15.9 | 23.0 | +7.1 | 26.0 |  |
| 4 | 3.3 | 15.8 | 24.0 | +8.2 | 28.3 |  |
| 5 | 2.0 | 15.5 | 25.7 | +10.1 | 30.3 |  |
| 6 | 1.7 | 16.1 | 26.3 | +10.2 | 32.7 |  |
| 7 | 1.7 | 16.0 | 26.3 | +10.3 | 34.0 |  |
| 8 | 1.0 | 16.0 | 26.3 | +10.4 | 34.7 |  |
| 9 | 1.0 | 16.2 | 26.7 | +10.5 | 35.0 |  |
| 10 | 1.0 | 15.9 | 26.3 | +10.4 | 35.3 |  |

### 4.3 Ablation and Analyses

In this subsection, we demonstrate some ablation studies to investigate the effectiveness of different components in the framework, in order to answer the following questions.
To advocate for open science, all the ablation experiments are conducted on either our own reproduction of open-source SWE agents or their official generations.

Question 1: Does DEI get better with more votes?

Answer 1: Yes. Arguably, DEI itself has the same potential characteristics as SWE agents that may cause diverse outputs.
So it is important for us to harness the diverse outputs of DEI as well.
However, unlike SWE agents whose outputs are code patches, DEI’s output is an integer score, which can easily be aggregated and averaged.
This is why we give DEI more votes and rerank the candidates according to the average of scores.
In most DeiBase experiments, we allow 10 votes for each candidate patch.
To investigate whether more votes lead to better patch reviewing, we directly take the scores generated for DeiBase-Open, DeiBase-Agentless, DeiBase-Aider, and DeiBase-Moatless, and check for various values of $m$, how the first $m$ scores can help us find the best patch.

As demonstrated in Figure 4, more votes generally lead to better resolve rates.
Another finding is that for 3 out of the 4 evaluation settings, DeiBase was able to get much better performance than the average candidate with only one vote.
Even when DeiBase wasn’t able to get better than average with one vote, it managed to get an improvement with only three votes.
These results suggest that DeiBase itself also produces diverse outputs, but it is easier to aggregate them via score averaging.

Figure: Figure 4: How the performance of DeiBase changes as the LLM is given more votes for scoring.
Refer to caption: x7.png

Question 2: Are the explanations necessary?

Answer 2: Yes. We remove the part about asking for explanations from the prompt and compare DeiBase-Open, DeiBase-Agentless, DeiBase-Aider, and DeiBase-Moatless under the same evaluation setting with and without explanations. We report their resolve rates in Table 3. For all 4 settings we evaluated, DeiBase with explanations performs slightly better than DeiBase without explanations.

**Table 3: Comparing DeiBase’s resolve rates with and without explanations.**
|  | Open Agents | Agentless | Aider | Moatless |
| --- | --- | --- | --- | --- |
| DeiBase w/ expl. | 34.6 | 26.0 | 24.6 | 25.6 |
| DeiBase w/ o expl. | 32.3 | 23.0 | 23.3 | 25.3 |

## 5 Conclusion

In this paper, we present Diversity Empowered Intelligence (DEI), a meta-policy module designed to integrate with any existing SWE agent frameworks to enable scalable management and collaboration among specialized agents, thereby fostering a more powerful software engineering organization. Through extensive evaluations, we find that different agents show a great level of diversity in the issues they resolve: a group of agents with an average resolve rate of 26.6% can actually solve 54.3% of the issues if we have an oracle that selects the correct candidate.
DEI, as our first step towards harnessing such diversity, can improve the group’s resolve rate to 34.3% (+7%), suggesting that LLMs are great code reviewers.
These findings mirror the benefits of diversity in the tech industry, where diverse perspectives and skills lead to greater innovation and problem-solving capabilities.

##### Broader Impacts.

DEI represents our initial step toward realizing a fully automated organizational AI. We believe that the full potential of multi-agent AI systems extends beyond enhancing task completion accuracy with agentic workflows, which is the current focus of most industry practices. Instead, DEI offers a horizontal, scaling-out approach that facilitates the collaboration and integration of existing diverse agents without necessitating refactoring of engineering work. This capability not only optimizes and speeds up immediate software development tasks but also sets the groundwork for future innovations in AI-driven organizational management.

## Appendix A Appendix

### A.1 Agents Evaluated

We add the following agents to the DEI Committee (the one in Figure 3) in the order they appear (the order is generated by randomly shuffling their chronological order using python’s random shuffle function with a random seed of 42):

- 1.
20240612 IBM Research Agent101
- 2.
20240612 MASAI gpt4o
- 3.
20240604 CodeR
- 4.
20240523 aider
- 5.
20240630 agentless gpt4o
- 6.
20240617 moatless gpt4o
- 7.
20240725 opendevin codeact v1.8 claude35sonnet
- 8.
20240706 sima gpt4o
- 9.
20240621 autocoderover-v20240620
- 10.
20240509 amazon-q-developer-agent-20240430-dev

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Could you provide a deeper analysis of why different agents excel at different issues? For example, have you analyzed patterns in the types of issues each agent solves successfully or investigated how specific agent architectures (e.g., execution vs non-execution-based agents) relate to their issue-solving capabilities?
- Could DEI's architecture be extended to provide feedback to the agents about why their patches failed, potentially enabling them to generate better patches?
- Could you provide a detailed analysis of DEI's failure cases? For instance, are there specific types of patches or issues where the scoring mechanism consistently makes wrong decisions?
soundness: 3
strengths: - The paper is the first to investigate and establish that different SWE agents solve different types of issues.
- Proposes novel DEI meta-framework that uses diversity to build a better resolver than any single agent.
- Achieves 55% resolve rate on SWE-Lite beating baselines significantly.
- DEI-Base Open using open source LLMs achieves a 33% resolve rate outperforming many closed source LLMs.
- Well motivated with thorough analysis, and ablation studies.
- The release of code and prompts  makes the research reproducible.
summary: This paper introduces DEI (Diversity Empowered Intelligence), a framework designed to leverage the diverse capabilities of software engineering (SWE) agents. Through comprehensive analysis, the authors demonstrate that different SWE agents solve distinct sets of issues despite having similar overall performance metrics. The paper introduces DEI as a meta-module framework that harnesses this diversity through multi-stage rating and re-ranking, rather than attempting to build a single "best" agent. Their experimental results show significant improvements, with their best group achieving a 55% resolve rate on SWE-Bench Lite. The framework is well-documented with ablation studies and analysis.
weaknesses: - While the authors demonstrate that different SWE agents solve different sets of issues, they provide no analysis of why this occurs (e.g., no investigation of which types of issues each agent excels at, or how agent architectures relate to their success patterns). This limits understanding of the relationship between agent design and issue-solving capabilities.

- The framework uses fixed agent groups (DEIBASE-1/2/Open) where all agents in a set contribute patches that are then scored. There is no learning mechanism to dynamically select agents based on past performance or issue characteristics. As a result, the system relies entirely on inherent diversity and fails to leverage agent specialization.

- DEI employs a fixed sequential review process (issue -> patches -> scoring) without iterative refinement or feedback loops between components. This linear approach potentially misses opportunities for improved patch selection that could be achieved through multi-round evaluation.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1) It seems to me that Eq. 3 expects the context to never change. Is that correct? (You fix the context, then you select the optimal policy for that context and then you use the expected reward from that policy.)
2) How do you select the optimal policy for a given context? Do you compute the expected reward for every policy given that reward and selected the highest? Or is this what is done in Step 3 Patch scoring?
3) Is there any learning involved in computing the DEI policy? It seems to me that the policies are given and then it is a deterministic computation to select the best of those.
4) Would it be possible to identify what the missing expertise is? I.e., if there was an agent capable of doing X, then the score would be significantly higher. There seems to be a potential to identify contexts that are far away from the contexts covered by the current policies. 
5) Have you considered composed policies? Maybe a but is complex and would require two different patches, i.e. first apply partial solution by agent a_i and then apply partial solution by agent a_j.
6) If you have as many members on the committee as you have different agents, wouldn't you always get the oracle answer? 
7) If the statement above is correct, wouldn't the interesting question be: What is the minimal number of members on the committee to get an acceptable score? What is the cost ($) benefit (resolve %) trade-off? The results would be much more interesting if the cost of running the different options would be included. It seems that you should be able to get a very clear analysis which would allow developers to estimate how much it would be worth to them to fix an issue.
soundness: 3
strengths: * Important problem with significant real world value.
* The paper is well written and easy to follow.
* The evaluation clearly shows both the added value of the approach and how much better it could be.
* Considers both intra and inter agent diversity.
* Good potential to extend the approach, see questions.
summary: The paper proposes using an esamble of agents to increase the number of software engineering (SWE) problems that can be solved showing that using different agents for different problems is better than using a single agent for all. The approach is based on a reinforcement learning formulation.
weaknesses: * It is unclear what new knowledge we have gained from the paper. That diversity is good is expected. To select the best candidate for each individual problem seems also expected. What is the unexpected or novel result of the paper? Or is the main result that you managed to get to work? - The authors clarified that it is how easy it is to achieve diversity and how significant it is.
* The significance of the approach might be on the low end. Since the approach is effective but straight forward, more elaborate analysis and/or experimentation would be expected. - Based on the authors response, the significance is higher than I initially estimated, but still somewhat limited.
* Some things are unclear, see questions. - Most of these were addressed by the authors.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Sec 4.4: Are these results statistically significant?

- Is it possible to add human developers to the committee? (Fig 1c.)
soundness: 2
strengths: - The topic is important. There is a clear emerging need for systematic methods for using LLMs, especially for software engineering.
- The method seems to be novel.
- The quality of the experiments is solid.
summary: The paper focuses on software engineering (SWE) agents, i.e., AI agents, in support of various software engineering tasks, such as code generation, automated testing, and project management. Specifically, in this work, the authors focus on AI agents' ability to resolve GitHub issues based on their descriptions. Observing that different SWE agents resolve different sets of issues successfully despite having similar resolve rates, the authors propose a method for organizing various SWE agents into ensembles with the intent to use the union of their expertise in resolving GitHub issues. The proposed method, Diversity Empowered Intelligence (DEI), is evaluated using realistic issue tracker data. The results show superior performance compared to single SWE agents' performances.
weaknesses: - The method is not exactly systematic, which limits its applicability and forces an ad-hoc way of working with ensembles of AI agents.

  - Sec 3.3.2, Equation 3: It is clear at this point that diversity is not purposefully engineered but rather: it's an emergent property of the system. One then has to wonder if this approach can be purposefully used in any problem or if its success will be rather non-deterministic and dependent on the problem at hand and its parameters (~ context).

  - Sec 3.3.2, L214-215: "The DEI framework leverages the strengths of each agent in their respective contexts to enhance overall performance across all contexts." - How does the framework account for intra-agent diversity? It seems that Figure 1 is rather deterministic, though some non-determinism is likely.

- Often vague formal underpinnings

  - Sec 3.3.1 is rather vague to provide formal underpinnings to the contributions. For example, L182: "The context space, C, includes relevant repository information and issue descriptions." What does "relevant information" entail here?

- Potentially limited mitigation of inconsistent agent behavior
  
  - Sec 3.2: "We consider two types of diversity: intra-agent diversity and inter-agent diversity." - I don't entirely agree with the proposed term of "intra-agent diversity". Diversity seems to be an artifact of non-determinism in this case, i.e., basically a temporal inconsistency. As such, it requires different means to mitigate, compared to the diversity stemming from the number of different agents. The method does not seem to acknowledge this.


- Typo: L057, citation "Team, 2024"

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: [1] In “4.1.2 Evaluation Mertics” for Union@k: “Solved by any of the k solutions.”, what is the mathematical definition of “any”?

[2] Likewise, for Intersect@k, “solved by all k solutions”: is k tied to the number of agents at hand, or is it a free parameter? If independent, which agents are chosen from the (unordered) set of agents?
soundness: 3
strengths: [1] The authors systematically explore the benefits of ensembling of SWE agents and show significant improvement over single-agent solutions.

[2] Figure 1 nicely illustrates the idea behind DEI. Table 1 is very insightful and provides strong evidence of the value of DEI.
summary: The authors propose “Diversity Empowered Intelligence,” a method for aggregating the solutions of several SWE agents by means of LLM prompting. They show a significant increase in scores on SWE-Bench Lite and attribute this to the fact that different SWE agents can solve different subsets of the problems.
weaknesses: [1] The results are presented for SWE-Bench only. Even though SWE-Bench is a fairly reliable and diverse benchmark, validating DEI on another SWE benchmark could solidify the claim.

[2] The improvement in scores comes at the cost of significantly higher token usage (or compute).

[3] The following papers are closely related but not cited:

a) SELF-CONSISTENCY IMPROVES CHAIN OF THOUGHT REASONING IN LANGUAGE MODELS

https://arxiv.org/pdf/2203.11171

Since a common decision is made based on a diverse set of candidates.

b) Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

https://arxiv.org/abs/2306.05685

Since the scoring by an LLM is prompted, which is essentially LLM-as-a-Judge.

[4] The metrics in 4.1.2 are tied to the order in which the agents (and, by extension, their proposed solution patches) are added to the list. This is unacceptable for a metric as the solutions are fundamentally an unordered set. The metrics should be refined to be permutation invariant towards the set of agents. Whereas the proposed metrics are flawed in the general case, the main results in section 4.2 that are collected with the specific ordered list of agents are not flawed and have analytical value.

Meta-review: An ensemble method for software engineering. Interesting new approach to an important problem with very good results on an important benchmark. Clear accept.

The diversity approach makes me wonder whether have looked at quality-diversity methods such as MAP-Elites. QD methods have been used for ensemble classification before, and also separately together with LLMs for code generation, e.g. LLMatic by Nasir et al.

PS: Fun acronym, and having a box labeled "DEI committee review" in your system diagram is hilarious :)

Decision: Accept (Poster)