Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents
Authors: Kexun Zhang
Weiran Yao
Zuxin Liu
Yihao Feng
Zhiwei Liu
Rithesh R N
Tian Lan
Lei Li
Renze Lou
Jiacheng Xu
Bo Pang
Yingbo Zhou
Shelby Heinecke
Silvio Savarese
Huan Wang
Caiming Xiong
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
LLM agents
software engineering
Primary area: foundation or frontier models, including LLMs

Abstract: Large language model (LLM) agents have shown great potential in solving real-world software engineering (SWE) problems. The most advanced open-source SWE agent can resolve over 27% of real GitHub issues in SWE-Bench Lite. However, these sophisticated agent frameworks exhibit varying strengths, excelling in certain tasks while underperforming in others. To fully harness the diversity of these agents, we propose DEI (Diversity Empowered Intelligence), a framework that leverages their unique expertise. DEI functions as a meta-module atop existing SWE agent frameworks, managing agent collectives for enhanced problem-solving. Experimental results show that a DEI-guided committee of agents is able to surpass the best individual agent's performance by a large margin. For instance, a group of open-source SWE agents, with a maximum individual resolve rate of 27.3% on SWE-Bench Lite, can achieve a 34.3% resolve rate with DEI, making a 25% improvement and beating most closed-source solutions. Our best-performing group excels with a 55% resolve rate, securing the highest ranking on SWE-Bench Lite. Our findings contribute to the growing body of research on collaborative AI systems and their potential to solve complex software engineering challenges.

Published as a conference paper at ICLR 2025
DIVERSITY EMPOWERS INTELLIGENCE: INTEGRAT- ING EXPERTISE OF SOFTWARE ENGINEERING AGENTS
Kexun Zhang 1.2 Weiran Yao Zuxin Liu', Yihao Feng's Zhiwei Liu', Rithesh Murthy', Tian Lan', Lei Li,, Renze Lou', Jiacheng Xu', Bo Pang', Yingbo Zhou', Shelby Heinecke', Silvio Savarese', Huan Wang', Caiming Xiong'
Salesforce Al Research, "Carnegie Mellon University
ABSTRACT
Large language model (LLM) agents have shown great potential in solving real- world software engineering (SWE) problems. The most advanced open-source SWE agent can resolve over 27% % of real GitHub issues in SWE-Bench Lite. How- ever, these sophisticated agent frameworks exhibit varying strengths, excelling certain tasks while underperforning in others. To fully harness the diversity of these agents, we propose DEI (Diversity Empowered Intelligence), a framework that leverages their unique expertise. DEI functions meta mota-module atop isting SWE agent frameworks, managing agent collectives for enhanced problem- solving. Experimental results show that DEI-guided committee of agents is able to surpass the best individual agent's 's performance by a large margin. For instance, group of open-source SWE agents, with maximum individual resolve 27.3% on SWE-Bench Lite, can achieve a 34.3% resolve rate with DEI, making a 25% improvement and beating most closed-soune -ousce solutions. Our best best-performing group excels with a 55% resolve rate, securing the highest ranking on SWE-Bench Lite. Our findings contribute to the growing body of research on collaborative Al systems and their potential to solve complex software engineering challenges
of
Each column is an issue in SWE-Bench Lite. A colored grid means the issue is resolved.
(a)
matplotlib
sphinx
django
Aider
Moatless
Agentless
OpanDevin
DEI Jours!
Oracle
(b)
40
Resolve %
30
Agent
Candidate Patch
Committee Review
Github Codebase
Candidate Patch 2
Agent 2
Candidate Petch 3
10-
Candidate
Agentless
Moatless
Aider
OpenDevin
DEL (ours)
Grade
Issue Description
Agent
Cancidate Patch
Best Patch
Figure 1: Different SWE agents (Aider, Moatless, Agentless, OpenDevin) resolve very different sets of issues (the colored girds in Fig la), despite having similar resolve rates (Fig 1b). Our proposed DEI Committee takes candidates patches and tries to select the best, oracle choice (Fig 1c), improv- ing the resolve rate significantly to be better than any single agent in the committee.
Codee data, and generations released at http:///tthcaeossssseeeeeeeeealr

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Recent advancements in large language models (LLMS) have transformed software engineering (SWE) and other domains. Originally developed as chatbots (Schulman 2024). LLMs have evolved into the core of Al agents, capable of understanding and generating human- conversations, as well as autonomously executing actions in both real-world and digital environments. SWE agents, specialized subset of these AI agents, integrate these capabilities with software engineering tools and techniques for tasks like code generation, automated testing, and project management, aiming to identify and resolve practical software issues (Zhang et al. 2024).
2022 OpenAl-Team
In this paper, we study one specific task of SWE agents resolving real-world GitHub issues based on their descriptions. Automatically fixing a bug in code repository is an extremely challenging task that involves navigating extensive codebases, understanding complex function interactions, de- tecting subtle errors, and generating the correct fix patch. The large action space of SWE agents, together with long trajectories, inevitably result in the diversity of Github issue solutions, as shown in Figure We have observed that different SWE agents resolve very different sets of issues (the colored girds in Figure Th), despite having similar resolve rates (Figure 1p). This is probably due to different skill sets of SWE agents. For instance, OpenDevin (Wang al. 2024c) explicitly instructs the LLM to first replicate the bug in an issue and executes its replication in a development workspace provide feedback for its generated patches, but other agents like Moatless Tools (Örwall, 2024) and Agentless (Xia et al. 2024) do not actually execute code in the issue-specific repository.
A garden's 's beauty never lies in one flower. Diversity in all its forms is the path to greatness.
Similarly, the trend in the SWE agent community reflects this diversity single agent framework dominates in all capabilities. ideas and leads to the development of better agents.
the flourishing variety within this community that sparks new
The variety SWE agent capabilities inspires us to develop DEI, Diversity Empowered Intelligence, a framework that leverages the strengths of diverse agents. DEI aims to harness these varied skills to tackle a broader range of problems more effectively with multi aulti-agent ensemble sys- tem and re-ranking pipeline, showcased in Figure Ip. DEI functions a meta neta-module that can be integrated with any existing agent framework, enabling scalable management and collaboration among agents form more powerful multi-agent software engineering organization.
We evaluate DEI on 7 groups of candidate agents on SWE-Bench Lite. 3 of the 7 are different runs of a single open-source SWE agent. The other 4 are different agents that are on the SWE-Bench Lite leaderboard, including a group that contains only open-source agents. Through experiments, we find that different agents show a great level of diversity the issues they resolve: a group of agents with an average resolve rate of 26.6% can collectively solve 54.3% of issues if there is an oracle reviewer that can consistently select the best candidate. DEI, as a first step towards harnessing such diversity, can improve the group's resolve rate to 34.3% (t 25%), suggesting that LLMSs are great code reviewers. These findings mirror the benefits of diversity in the tech industry, where diverse perspectives and skills lead greater innovation and problem-solving capabilities.
To summarize, our contributions are the following:
For the first time, we comprehensively evaluate the diversity of solutions provided by SWE agents, revealing significant differences in the types of GitHub issues resolved by various agents, despite similar overall resolve rates. These findings suggest a substantial potential improve overall performance by effectively leveraging the diverse expertise of these agents together.
This paper introduces DEI, multi-agent meta-policy module designed to harness the diversity of SWE agents and seamlessly facilitate collaboration among agents with different specialties. By employing mtuti-stage rating and re-ranking pipeline, DEI consistently improves issue resolu- tion, demonstrating a 25% performance boost on the SWE-Bench Lite leaderboard.
2 RELATED WORK
We review the work in fundamental language agent architecture, recent developments for SWE agents, and multi-agent ensemble methods in this section.
2

---

Published as a conference paper at ICLR 2025
Fundamental Language Agent Pioneering Al agent methods along this line of work include ReAct (Yao et al., 2023), Reflexion (Shinn et al.] 2023). CodeAct (Wang et al. 2024b), etc.. in which ReAct interprets the user query, generates functional API calls, and gets the tool outputs in real time; Reflexion further appends failed trial experience into the memory, enabling effective retrials to prevent repetirive errors. CodeAct (Wang et al. 2024b). instead of generating function calls, uses code generation to consolidate AI agents' actions into a unified action space.
Software Engineering Agent We present the SWE agents which have disclosed the technical de- tails on the SWE-bench Lite leaderboard. Alibaba Lingma Agent (Ma et al., (024) constructs a repository knowledge graph to represent code and dependencies, using Monte Carlo tree search- based strategy for repository exploration, and generates patches to address real-world GitHub issues. AutoCodeRover (Zhang al.
2024) adds advanced code search tools, such abstract syntax trees, and spectrum-based fault local ization to the agent for enhancing context understanding and issue res- olution. Code-R (Chen al., 2024) chooses multi-agent framework with pre-defined task graphs to resolve Github issues. Agentless (Xia et al. 2024), is a simplified two-phase approach for solv- ing software development problems. focused on localization and repair without relying on LLMs to make decisions, highlighting the potential of straightforward techniques in autonomous software development. OpenDevin (Wang et al., [ (024c) is a hub of community contuiitt-tttibbteedd agents including CodeAct (Wang et al., 2024b), browser agent, GPTSwarm (Zhuge et al.) (024), and task-specific micro agents. Finally, SWE SWE-agent (Yang et al. 2024) developed agent-computer interface that con- sists of LM AM-friendly commands and environment feedback to enable LM agents to autonomously use computers to solve software engineering tasks.
Multi and Ensemble Agents Recent works observe that organizing multiple specialized AI agents (Hong et al. 2024 Li et al. 2023 Liu al., 2024) enable the task decomposition ability of an agent system, which improves the task-resolving performance. Current multi-agent frame- works are categorized into three types based on their execution patterns. Firstly, static agent working flow (Wu et al.) 2224 Githubl 2023). which pre-defines the agent execution flows and ignites agent transitions via specified conditions. Controlling a multi-agent system with pre-determined states is robust, though losing flexibility in terms of unseen states or conditions. Secondly, ensemble via self-consistency (Wang et al. 2023). LLM-as-a-judge (Zheng et al. 2023), group chatting (Wu et al.) 2023 Hong et al.) 2024 Wang al. 2024a Chen et al. 2023). This is built upon an environment where multiple agents send messages to each other in a group channel such that their thoughts are ensembled. Variants of group chatting includes debating (Liang et al. 2023 Chan al. (202)) and model-wise ensembling (Wang et al. (024a). Last but not least, hicrarchical task assignment (Liu
al., 2024 2023). Organizing multi-agent in a hierarchical structure benefits the top-down task
decomposition and thus enables efficient multi-agent collaboration.
3 INTEGRATING EXPERTISE OF SWE AGENTS
3.1 BACKGROUND
Resolving issues in SWE-Bench. One important task in software engincering is to resolve issues raised by developers. SWE-Bench curates instances of this task by collecting successfully resolved issues from open- open-source repositories on Github. Each instance in SWE- issue description, a version of the repo just before the issue was resolved, and (hidden) unit tests that went from fail to pass after the Haman--ritted patch. To resolve an instance, the model is required to generate a patch that can pass these unit tests.
consists of a textual
SWE Agents. In this paper, we use the term "SWE agents to refer to any L--M- system that generates patches to solve issues in a code base, c.g., an instance in SWE-Bench. While the specific implementation varies, typical SWE agent usually gives their underlying LLM several tools in the form of callable functions to navigate through the code base, find relevant context, edit files, and run tests. The workflow of SWE agents involves multiple LLM calls, each taking some all outputs from previous steps as input.
According to our definition, SWE-agent Yang et al. 2024 is an instance of SWE agents, and Agentless Xia al. 2024), despite the name, another.
3

---

Published as a conference paper at ICLR 2025
3.2 DIVERSITY OF SWE AGENTS
We consider two types of diversity: intra-agent diversity and inter-agent diversity.
Intra-agent diversity is defined as the degree to which different runs of the same agent solve dif- ferent problem instances. It is most likely from the non-determinism the underlying LLM due to sampling in decoding and mixture-of-expeer
architecture (Chann) 2023). Since the workflow of SWE agents involves multiple steps and LLM calls, slight difference in an earlier step can easily propagate and result in significant differences in the final outcome.
Inter-agent diversity is defined as the degree to which different agents solve different problem in- stances. Besides sharing the potential causes of intra-agent diversity, inter-agent diversity is also largely because differences in agent design, including different tools, workflows, and prompts.
3.3 APPROACH
3.3. SWE AGENT PROBLEM FORMULATION
We formulate the SWE agent problem under the contextual Markov decision process (CMDP) frame- work (Hallak et al. 2015), represented by the tuple M = (S,C. C, A, R,P, P.po p). Here, denotes the state space, which encompasses all possible states the agent could encounter, such the current status of files. The context space, C, includes relevant repository information and issue descrip- tions. The action space, A, represents all potential actions or tools the SWE agent as search or editing. The context-dependent reward function, R. : S X A X C - R, as- signs scores based on the actions taken by the agent. For instance, the reward is high if the agent successfully addresses an issue, while it is low if the action results in new bugs in the repository. The context-dependent transition function. P S x A repository or information changes following a specific action. The context- mentext-dependent initial state distribution is denoted by Po - AS)), and p E (CC) represents the context distribution.
Po:
utilize, such
ASS), defines how the state of the
Given the initial context ~ p and initial state so Po(- c), at each time step f, the agent follows policy π S x C (AA to select an action a; π ( e) and receives a reward R(s,, ag c). The environment then transitions to the next state 8141 PP- 8t, dc, providing the agent with new state observation. As the iteration progresses to time T, sampled trajectory T == {st, de; T' Y is obtained. The objective of SWE agent is to maximize the cumulative reward along the trajectory, which is captured by the value function:
T Σ
max (T) max E, π
R(si, pmppp
3.3.2 OUR FRAMEWORK: DIVERSITY EMPOWERED INTELLIGENCE (DEI)
Many efforts have been made to implement sophisticated agent systems that aim to achieve the objective described Eq. [ However, as discussed in Section I these systems often exhibit vary- ing levels of effectiveness across different contexts. It is challenging to devise single agent that can consistently perform well across all possible contexts. Formally, suppose there are N agent where each policy is tailored to address specific con- text (P1.P2...... .......,PN'. PN The union of these contexts subset of the entire context space, i.e., P1 U P2 U p. For each agent policy n, the objective is:
policies, denoted as 1 .T.. ? T2,
n, max E,
Σ R(5e, ((5,,, c~ P=: 1=0
(2)
However, an agent policy n. may perform poorly in a different context Pj (where j * i). To address this limitation, we propose framework: Diversity Empowered Intelligence (DEI). The DEI framework leverages the strengths of each agent in their respective contexts to enhance overall performance across all contexts.
4

---

Published as a conference paper at ICLR 2025
We introduce meta-policy, denoted as TDEI, which aims to optimally select among the available agent policies based on the context. The goal of TDEi is defined as:
DEI max Bowu E, Σ R((s ar (() 1c
where Tcc) denotes the selection of the optimal agent policy from { #2. ........ nN} based on the observed context By dynamically choosing the most suitable agent policy for each context, the DEI framework seeks maximize the expected cumulative reward across all possible contexts.
3.3.3 DEIBASE: A SIMPLE YET EFFECTIVE IMPLEMENTATION
We present DEIBASE, a simple yet powerful implementation of the DEI framework, tailored for SWE-Bench like problems. The context in the setup includes the repository, along with relevant files and issue descriptions. The meta meta-policy's 's action space consists the final patches generated by different agent frameworks, each specialized in addressing various aspects of the problem.
DEIBASE utilizes a Large Language Model (LLM) as a code review committee. The LLM evaluates candidate patches by analyzing the state of the code base before and after the proposed changes, in conjunction with the contextual information from the issue descriptions. It produces detailed explanations for each patch, justifying the modifications based the identified issues, the context, and the specific changes made.
While other methods of code review and scoring, such as rule-based approaches, can be incorporated into our framework, the use of an LLM-based committce offers a unique advantage. LLMs often excel evaluating solutions when evaluation is easier than generation. DEIBASE thus serves as an effective baseline for LLM LLMdbased SWE evaluation, highlighting potential performance variations among diverse SWE agents and showcasing the capabilities of our method.
Agant
Github Codebase
Localiation
Contect
Canrleod
Explanaion
Patch
Score
O lssue Description
Candidate Patch
git-
Afbe Peich
Putch Explanation
comety nated Compos-ettooee
Candidate Patch
Committee Review
DEI
Scoring
Submit
Candidate Patch
Agent
Candidate Patch
Agent N
Figure 2: Framework Overview, DEI first examines the code base before and after a candidate patch, along with other relevant contexts. Then, it generates an explanation for the issue, the context, and the patch and tries to justify the patch. With its own explanation, it scores the candidate patches and picks the top-scoring ones as more likely be correct.
As demonstrated in Figure 2 DEIBASE is given multiple candidate patches for single issue. These patches might be from running single SWE agent multiple times running multiple SWE agents. DEIBASE gives each candidate patch score and then selects the top-scoring candidates as the patches most likely to work. Prompts and example inputs/outputs can be found in Appendix
Step 1: Input Construction. Four inputs are given to DEIBASE for each patch: the issue descrip- tion itself, relevant context (code snippets identified by an SWE agent as relevant to the issue), code before the patch, and code after the patch. The form of inputs reflects two design choices. First, the entire repository is often too large to fit directly in the context limit of LLMs, relevant context instead to save token costs and help the model focus. Second, the format of a patch not the easiest for LIMM to read switches back and forth between the pre-change code and
the

---

Published as a conference paper at ICLR 2025
the changed code, so give the code before and after the patch separately to the model for easier understanding. In practice, we directly use the relevant code spans identified by Moatless Tools, an open-source SWE Agent (Orwall 2024). There might be potential ways of improving the quality of relevant code spans by making them specific to both the issue and the candidate patch, rather than solely dependent the issue itself.
Step 2: Explanation Generation. To help the model better "understand" the patch before scoring, instruct generate various explanations regarding the patch in specified order. The order is decided so that the earlier explanations can also help the later ones. We describe each explanation in the order they are generated here: Issue explanation explains what the issue is about and what problem it may be causing. 2) Context explanation explains how and why each relevant code span (there might many of these) is relevant to the issue. 3) Location explanation explains if and why the patch is modifying the correct part of the code that's faulty. 4) Patch explanation explains if and how the patch is fixing the issue. 5) Conflict detection about checking whether the patch conflicts with other relevant code snippets. We explicitly instruct the model to refer back to the earlier explanations while generating the later ones.
Step 3: Patch Scoring. Based on its own explanations, the model is asked to give the candidate patch a score of to 10. We give the model detailed rubrics of what violations/misttkes lead to higher score deductions and what should only be considered minor violations. For example, if the model finds the modification location to he wrong, it is considered a serious mistake. We score each candidate multiple times and average the score. To encourage diverse outputs, an inference temperature of 1.2
4 EXPERIMENTS
We aim to answer two research questions with our experiments: 1) How diverse are LLM-based SWE agents in terms of intra- and inter-agent diversity? 2) To what extent can DEI harness the diversity and increase the performances of these SWE agents?
4.1 EXPERIMENT SETUP
4.1.1 BENCHMARK AND AGENTS
Benchmark. We conduct our experiments on SWE-Bench Lite, 300-instance subset sampled from the full SWE-Bench for providing more self-contained evaluation of functional bug fixes (Jimenez al. 2024). Compared the full SWE-Bench, SWE-Bench Lite has significantly more submissions on the leaderboard for us to conduct more comprehensive analysis of inter-agent diversity.
Agents. For intra-agent diversity, we consider three well performing open-source agents on the SWE-Bench Lite leaderboard: Agentless (Xia et al., 2024). Moatless (Órwall 2024), and Aider Gauthier 2024) by running them 10 times with the same parameters. For inter-agent diversity, we consider 10 agents that have similar resolve rates, all between 26.0% and 3 31.0% on the leaderboard by directly using their submitted patches to the SWE-Bench issues. For the evaluation of DEIBASE on different agents, consider 3 groups of agents that submitted SWE-Bench Lite before August 15, 2024, including one group consisting of only open-source agents. For the evaluation of DEIBASE on multiple runs of single agent, we use the generations of the three aforementioned agents Agentless, Moatless Tools, and Aider. More details can be found in Appendix A.4
4.1.2 EVALUATION METRICS
We use the same set of metrics for both intra- and inter-agent diversity as these metrics are defined for multiple candidate solutions without requiring them to from the same candidate. We assume there are n candidates in total, and we are computing "@k" metrics over all of its subsets with candidates.
Resolve rate measures how good a SWE agent is. is defined as the percentage of issues resolved by the agent. We measure both single SWE agents and DEI with it to see how much DEI helps.
Union @k measures the best case performance of & candidates, assuming an oracle reranker to always pick the best candidate. For any set of size k. if an issue is resolved by one of the k candidates,

---

Published as a conference paper at ICLR 2025
it is counted solved for Union @ k. We compute the average of resolve rates all possible sets of size k.
Intersect@ @k measures the worst performance of A candidates, assuming adversarial reranker that always picks the worst candidate. An issuc is considered resolved only all candi- dates resolve it.
Average@k measures the average case performance by computing the average number problems solved by & candidates, assuming random reranker that picks random candidate.
DEI@K k measures the performance our method by computing the number of problems solved by the top-1 candidate deemed by DEI. The better reranking mechanism is at telling good solutions from bad ones, the higher DEI @K is.
Our research questions be answered by the gaps between these metrics. Union @ k Intersect @ measures how diverse the agents are, while DEI@K Average@ k measures how much DEI helps in selecting the correct candidate. Note that when there possible subsets of size & when computing the "@k" benchmarks. We report the mean and standard deviation of the benchmarks over all subsets.
than & candidates, we consider all
4.2
MAIN RESULTS
4.2.
RESEARCH QUESTION 1: HOW DIVERSE ARE LLM-BASED SWE AGENTS?
To answer this question, report the "@k metrics of 10 different agents and 10 runs of single agents in Figure 3
Uniongk (Oracle)
Unien@k (Oraclel
DR (Ours)
Reselve
Reselve
Drikk (Ours)
30
20
Avarage (Random)
Averageggk (Random)
20
Intersecte (Aduersary)
Kersettitt (Adversary)
Agents
(Agentless)
Unionge (Oracte)
Uniongk (Grade
30
Drigk TOurs]
Resolve
Resolve
Diloo Yours
Averageik (Randomn)
Averagegk (Random)
15
ntersertikk (Adversory)
Intersectin (Adversary)
Number (Aider)
Number Runs (Moatiess)
Figure 3: How different metrics change as more candidate solutions are involved. In all 4 scenarios, there is huge gap between Union @ and Average@k. The concrete dots are expectation over all possible subsets of size k. while the shaded area corresponds to the standard deviation.
Several observations be made about the results:
SWE agents resolve very different sets issues across agents and agent runs. Their full poten- tial is far from fully released. In all four subfigures. the gap between Union @k and Average @ as well as between Average@k and Intersect@k, is large. As the number of candidates gets larger, the gap also gets larger. For of the settings, Union @k k is more than 2x larger than Aver-

---

Published as a conference paper at ICLR 2025
age@k for k 10. The other 2, Union @k is more than 1.5x larger than Average@k for 10. This indicates that current SWE agents are potentially capable of resolving a lot more issues, if we have reranker that can tell which candidates correct.
agents resolve more distinet issues than different runs a single agent. In other words, diversity does empower intelligence. The absolute/relative difference between Union @k and Average@k @k is much larger in the first subfigure than in the following three subfigures. For the "10 different agents" setting, as k approaches 10, the distinct issues resolved are 2x the average number of issues resolved by single agent in the group.
Different
Table 1: Resolve rates of top submissions on SWE-Bench
(cutoff date: Aug 15, 2024). We evaluate of agents. Each DEI Committee outperforms the best agent in
DEI Committees formed by different groups significantly. DEIBASE-OPe Open, formed by 4 open source agents can beat many glosed-soure agents.
Group
Resolve
System
Open
Trajs
Open Candidates
L.M
55.0
DEIBASE-1
gptle
50.6
Cosine Genie
X
X
"Hine tuned OpenAI"
CodeStory Aide
X
x
gpi4o, Claude 3.5 Sonnet gptdo
38.0
AbenteAl MentatBot
X
X
2
37
DEIBASK-2
X
gni4o
Open
34.3
DEIBASE-Open
gpt4o
34.0
Bytedance MarsCode
gpt4o
33.0
Alibabu Lingma
x
X
gpt-4-1 106-preview
2
31.3
Factory Code Droid
X
X
"Anthropic and OpenAl"
2
30.6
AutoCodeRover
X2
X
gpt4o
2
29.6
Amazon Q Dev.
X
X
Unknown
2
28.3
CodeR
X
X
gpt-4-1 106-previww
2
28.0)
MASAI
X
X
Unknnwn
27.6
SIMA
X
gpt4o
Open
27.3
Agentless
gptdo
Open
26.6
Moatless Tools
Claude 3.5 Sonnet
26.6
IBM Research Agent
X
Unknown
Open
26.3
Aider
X
gpt4o, Claude Opus gpi4o
Open 26.0)
Open CodeAct
Their repo has code yet.
An earlier version is open source. The current one is not. Candidates are generated by "modification of moatless tools".
4.2.2 RESEARCH QUESTION 2: How MUCH DOES DEI HELP?
We apply DEIBASE to the candidates in Figure as they are added to the group. Our findings are:
DEIBASE helps in most cases. For most values of i in all subfigures, observe significant improvement of DEI@K over Average@k, indicating that DEIBASE selects correct candidates much better than random baseline.
DEIBASE helps more when the candidates come from different agents. This finding resonates with a similar finding from research question one: Since candidates from multiple agents have larger potential for improvement (Union@k Average@k), k), the actual improvements created by DEIBASE (DEI@K- Average@k) are also larger. This suggests that given limited budget of candidates, it would be better to choose a diversity of agents over multiple runs of the same agent.
As l gets larger, DEIBASE'S improvement first increases and then plateaus. While larger generally indicates higher DEI@K, the margin gets smaller and there are cases when an increase in results slight drop in performance. This suggests that the current DEIBASE not ideal for large group of agents and there is still room for better reranking mechanism.
Based on the lessons above, we propose three DEIBASE groups in which each candidate is from a different agent and no more than 5 candidates exist for each instance. The members of these DEIBASE groups and their performance are reported Table I DEIBASE-1 consists of the top agents. DEIBASE-2 consists of 5 closed-source agents that have high performance on the leader- board. LABBAS--Open consists of 4 open- source agents so that we know future researchers can run the entire pipeline. As Table I shows, all three DEIBASE instances outperform the best candidate in the group. Surprisingly. DEIBASE-Ope shows a 7% increase resolve rates and beats most of the closed-source systems.
8

---

Published as a conference paper at ICLR 2025
Open Agents
Agentless Runs
26.0
34.0
33.0
25.0
32.0
24.0
31.0
Resolve
Res
30.0
23.0
29.0
22.0
28.0
21.0
27.0
Average@4
Average@10
8
10
2
4
10
Votes
of Votes
Aider Runs
Moatless Runs
25.0
26.0
24.0
24.0
22.0
Resolve
23.0
20.0
22.0
Average@10
18.0
21.0
16.0
Average@10
of Votes
# of Votes
Figure 4: How the performance of DEIBASE changes the LLM is given more votes for scoring.
4.3 ABLATION AND ANALYSES
In this subsection, we demonstrate some ablation studies to investigate the effectiveness of different components the framework, in order to answer the following questions. To advocate for open science, all the ablation experiments are conducted on either our own reproduction of open-source SWE agents or their official generations.
Question 1: Does DEI get better with more votes?
Answer I: Yes. Arguably, DEI itself has the same potential characteristics as SWE agents that may cause diverse outputs. However, unlike SWE agents whose outputs are patches, DEI's output is an integer score, which can easily be aggregated and averaged. This is why we give DEI more votes and rerank the candidates according to the average score. In most DEIBASE experiments, 10 votes for each candidate patch. To investigate whether more votes lead to better patch reviewing, directly take the scores generated for DEIBASE-Open, DEIBASR-Agentee, DEIBASE-Aider and DEIBASE-Moattes, and check for various values of m, how the first m scores can help us find the best patch.
allow
As demonstrated in Figure 4, more votes generally lead to better resolve rates. Another finding is that for 3 out of the 4 evaluation settings, DEIBASE was able get much better performance than the average candidate with only one vote. Even when DEIBASE wasn't 't able to get better than average with one vote, it managed get an improvement with only three votes. These results suggest that DEIBASE itself also produces diverse outputs, but it is easier aggregate them via score averaging.
Question 2: Are the explanations necessary?
Answer 2: Yes. We remove the part about asking for explanations from the prompt and compare DEiBAsE-Open, DEE:AASF-Aeentles, DEIBASE-Aider, and HEIBASFE-Maatles under the same evaluation setting with and without explanations. We report their resolve rates in Table 2] For all settings we evaluated, DEIBASE with explanations performs slightly better than DEIBASE without explanations.
Table 2: Comparing DEIBASE'S resolve rates with and without explanations.
Open Agents
Agentless
Aider
Moatless
DEIBASE w/ expl.
34.6
26.0 23.0
24.6
25.6
DEIBASE w/ expl.
32.3
23.3
25.3

---

Published as a conference paper at ICLR 2025
5 CONCLUSION
In this paper, we present Diversity Empowered Intelligence (DEI), a meta-policy module designed to integrate with any existing SWE agent frameworks to enable scalable management and collaboration among specialized agents, thereby fostering more powerful software engineering organization. Through extensive evaluations, we find that different agents show great level of diversity in the issues they resolve: group of agents with an average resolve rate of 26.6% 6%%% can actually solve 54.3% of the issues if have an oracle that selects the correct candidate. DEI, as our first step towards harnessing such diversity, can improve the group's resolve rate to 34.3% (+7%), suggesting that LLMSs are great code reviewers. These findings mirror the benefits of diversity in the tech industry, where diverse perspectives and skills lead to greater innovation and Soovven-oolvingg capabilities.
DEI represents our initial step toward realizing a fully automated organizational AL. We believe that the full potential of multi-agent Al systems extends beyond enhancing task completion accuracy with agentic workflows, which is the current focus most industry practices. Instead, DEI offers a horizontal, scaling- out approach that facilitates the collaboration and integration of existing diverse agents without necessitating refactoring of engineering work. This capability not only optimizes and speeds up immediate software development tasks but also sets the groundwork for future innovations in Al-driven organizational management,
REFERENCES
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better Ilm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07200 2023.
Seherman Chann. Non-determinism in gpt-4 is caused by sparse moe. Accessed on August, 5:2023, 2023.
Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, al. Coder: Issue resolving with multi-agent and task graphs. arXiv preprint arXiv:2406.01304 2024.
Weize Chen, Yusheng Su. Jingwei Zuo, Cheng Yang. Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308. 10848, 2023.
Paul Gauthier. Aider ai for customer service and support, 2024. URL StRL:ttps:/ //a//aideerrrre chat/ Accessed: 2024-07-16.
Github. Babyagi. Github - babyagi, 2023.
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv: ISI.::1002.0259, 2015.
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang. Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for a multi-agent collab- orative framework. In The Twelfth International Conference Learning Representations, 2024. net. fttt://frummmmmo www.:mideddee/t/eeT... /
URL
Carlos E. Jimenez, John Yang, and Jiayi Geng. Swe-hench lite: A canonical subset for efficient evaluation of language models as software engineers, March 19 2024. URL hnttps://www
ssweennh. com wwt.cctmmmmm...............e..ee wiiteeetm html Accessed: 2024-07-16.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang. Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi- agent debate. arXiv preprint arXiv:2305. 191/8 18, 2023.
10

---

Published as a conference paper at ICLR 2025
Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng, Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, et al. Bolaa: Benchmarking and orchestrating Ilm- aungauggeetedd autonomous agents. arXiv preprint (tX.::2338000960000 2023.
Zhiwei Liu, Weiran Yao, Jianguo Zhang, liangwei Yang. Zuxin Liu, Juntao Tan, Prafulla K Choubey, Tian Lan, Jason Wu, Huan Wang, et al. Agentlite: A lightweight library for build- ing and advancing task oriente IIm agent system. arXiv preprint arXiv:2402.15536 2024.
Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang. and Yongbin Li. How to under- stand whole software repository? arXiv preprint arXiv:2406.0 2024.
Xiangxin Meng, Zexiong Ma, Pengfei Gao, and Chao Peng. An empirical study on Ilm Immdd agents for automated bug fixing, 2024. URL htt.:/ss/aaax....rg//a////////// 10213
OpenAl-Team. Gpt-4 technical report, 2024. URL https: BtLLtttts///axiv..ooo wwt.:/rgggabb///eae
08774
John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Fe- lipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, et al. Introducing chatgpt. OpenAl Blog. 2022.
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sen- sitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. arXiv preprint arXiv:23 20. 11324, 2023.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.1 / 1366, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1)), 2008.
Junlin Wang. Jue Wang. Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agent enhances large language model capabilities. arXiv preprint arXiv:2406. XX01:2406000692,, 2024a.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Exe- cutable code actions clicit better Ilm agents. In ICML, 2024b.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, al. Opendevin: An open platform for ai software developers as generalist agents. arXiv preprint 24X.:24077.1641,, 2024c.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc V Le, H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
Qingyun Wu, Gagan Bansal, Jicyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang. and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi- agent conversation framework. arXiv preprint 23xiv:2308..8/55 2023.
Yiran Wu, Tianwei Yue, Shaokun Zhang, Chi Wang, and Qingyun Wu. Stateflow: Enhancing I1m task-solving through state-driven workflows. arXiv preprint arXiv:2403. 1322, 2024.
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llmebased software engineering agents. arXiv preprint arXiv:2407.01489 2024.
John Yang. Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405. 1.0.::22005.1799, 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement. arXiv preprint arXiv:2404.05423 2024.

---

Published as a conference paper at ICLR 2025
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt- at--bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jurgen Schmidhuber. Language agents as optimizable graphs. arXiv preprini 2ttp:rrmmaurxx2222000663333,,,00
Albert Örwall. Moatless tools, June 2024.
URL https://github.ccm//orraall]
moatless-tools Accessed: 2024-07-16.
12

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ANALYZING DIFFERENT SWE AGENTS
In this section, we analyze the issues in SWE-Bench Lite resolved by the 4 open-source agents in DEIBASE-Open and try to attribute why they are solving different issues. We have access to the implementation of these open agents and, therefore, can better analyze their performance.
A.1.1 WHAT ISSUES DIFFERENT AGENTS RESOLVE?
When combined, the agents resolved 137 / 300 issues, A Venn Diagram of the sets of resolved issues is shown Figure 5 Table shows the number of issues resolved each repository. From the table, we observe that the issues uniquely solved by only one agent are about 10% of all issues resolved by that agent.
By further looking at the number of uniquely resolved issues in each repository, find that the uniquely resolved issues for aider, moatless, and agentless are more evenly distributed across repositories, while those for opendevin are skewed towards django, indicating larger difference between opendevin and other open-source agents.
The larger difference between opendevin and the other three is also shown in Table 4 where we list the number of commonly solved issues for all agent pairs. The commonality between opendevin n and the others smaller than that within the others. This difference is reflected by the length of generated patches and the number of locations edited in Table 5 as opendevin is editing significantly more files and creating significantly longer patches.
aider moatless
agentless opendevin
11
8
79
78
5
10
12
13
27
Figure 5: Venn diagram of the solved issues.
Table Commonly resolved issues between pairs of agents.
aider
moatless
agentless
opendevin
aider
53
52
44
moatless
53
51
47
agentless
52
51
48
opendevin
13

---

Published as a conference paper at ICLR 2025
Table 3: Number resolved issues in each repository. p/g indicates that there are issues resolved in total by that agent, while p indicates there are p issues uniquely solved by that agent and solved by others.
aider
moatless
agentless
opendevin
astropy/astropy
0/0
0/1
0/1
0/2
django/django
4/37
4/41
1/37
7/46
maplodihh/mappltiib
1 1/4
1/5
0/5
1/3
mwaskom/scaborm
0/1
0/1
0/2
0/2
pallets/llask
0/0
0/0
0/0
0/0
pst/requests
0/3
0/3
0/1
pydat//xarray
0/1
0/1
0/1
0/0
pylint-dev/pyling
0/0
1/1
1/1
0/0
pytest-dev/pytest
1/6
0/3
1/6
0/3
corkit-lcam/scikii-eeeeee
0/10
1/8
0/8
2/5
sphinx-doc/sphne
0/2
0/2
0/3
1/3
sympy/sympy
3/15
1/
4/
2/13
Total
9/79
11/80 80
8/82
13/78
Table 5: Mean and standard deviation of generation lengths (by character) and number of locations edited.
aider
moatless
agentless
opendevin
patch length mean patch length std
1140.2
1005.3
750.6
57890.8
499.3
442.2
152.7
30557.0
num. locations edited mean
1.7
1.4
1.2
4.6
num. locations edited std
0.7
0.6
0.4
1.9
We further analyze if there is observable pattern of issue description that correlates with the uniquely solved issues. To do that, we create vector embeddings for the issue statements using voyage-code-
and project them to 2-dimension using t-SNE (Van der Maaten & Hinton, 2008). As shown in Figure the representations of the uniquely resolved issues are not really sep- arated. At least from this figure, we do not observe a clear pattern correlating issue description embeddings and agents' ability to solve them.
-20
-2
-40
-45
moatless
agentless
opendevin
5.5
6/2
8.0
Figure 6: t-SNE visualization of the issues uniquely resolved by the agents.
However, this could also be because of the quality of embeddings, which is why we need qualitative analysis the resolved issues.
An empirical analysis of multiple SWE agents (Meng et al. 2024) found that the commonly solved issues generally better in quality than the less commonly solved ones. The quality metrics include reproducible examples, resolve solutions, quality of file-level location, quality of line-level location, These metrics are annotated by human annotators.
14

---

Published as a conference paper at ICLR 2025
In the next section, we analyze qualitatively the issues uniquely resolved by the open agents and attribute why that is the case. We try to answer the question: why do different agents resolve different issues?
A.1.2 WIYY DO DIFFERENT AGENTS RESOLVE DIFFERENT ISSUES?
Different tool sets for bug localization. Among the four open agents we tested, moat t I less is the only one that uses embedding- based retrieval to find which file and which line to make changes to the code. The other agents provide LLMs with tools like file viewer, AST search and string search and also prompt the model with a summary the file structure. The extra tool of embedding based retrieval gives moatless some chance to better localize the bug. For example, sympy-23262 is an issue only resolved by moat less. The other three agents were not able to correct locate the file to edit: sympy /ut weliiiiis//ii. ammd/ i fy PY, while moatless did. This is also the case for (hango-15790, where the issue description mentions "template", The files modified by the other three agents all have "template" in their paths. However, moatless was the only agent that found the correct file to modify: dj django/ core/ sereecks/temlaattt
Bug reproduction and iterative testing. Among the agents, OpenDevin is the only one that tries to reproduce the bug and create test script before fixing it. is also the only one with larger cost limit and more rounds model calls. While this is making its generations much longer and cost more, it also allows OpenDevin n to solve 13 unique issues, especially the one with reproducible tests their description. For example, for the following issue sympy-15609, although all agents located the bug in sympy warintiig//eee of the reproduction test in the following issue description and eventually resolve the issue. This is also the case for sphinx-8435, which also contains reproducible tests.
py, OpenDevin was the only one that made use
sympy-15609, Issue description:
Indexed matrix-expressieg LaTeX printer is not compilable "python
i, j, k = symbols(") j k")
M Matrixsymbal ("M", k, k)
Matrixsymbal ("N", , k, k) latex (M+N) [i, i1)
The LaTeX string produced by the last command is:
( (k 1) Mii,, 11 NLLL_1, 1
LaTeX complains about double subscript '_'. This expression won't render in MathJax either.
Plausibility tests and mixing models. ai aider runs multiple iterations with different models gpt- 4o and Claude 3 Opus - when one iteration does not produce plausible patches. It checks plausibility by using a python linter find syntax errors and running pre-existing tests make sure the changes do not break the code base. With these plausibility tests, aider can run multiple rounds to generate patches for single issue until it solves them.
Other ad hoc factors and spurious features. SWE agents are complex systems with LLM calls, agent tool implementations, and other components chained together. This makes re- ally hard to attribute their performance differences for many reasons: 1) LLMs are sensitive to even very slight changes in prompts (Selar al.) (023). 2) LLMs are probabilistic and non-deterministic, and 3) software engineering issues are complicated problems. For exam- ple, although aider and agentless are conceptually identical in bug localization, there are still many cases where they found different files modify. For sympy-13971, aider found the correct file sympy/Si / sttseeeeeee/eeeeeeeeeeeeeeeeeese hmmpp//srriesseeeennces sequences. py while agent found the wrong file sympy http:/prinnnnnnntteee
15

---

Published as a conference paper at ICLR 2025
A.1.3 CAN WE DETERMINE WHAT AGENT TO USE FOR EACH ISSUE IN ADVANCE?
Given the explanations in the previous section, it might seem tempting to learn classifier to select the best agent for an issue instead of selecting the best generation of multiple agents. However, such classifier doesn't work well. We tried training several classifiers with the embeddings of issue description input and outputs which of the four agents to use. We trained on a randomly sampled subset with 150 issues in SWE-Bent and evaluated it on the remaining 150 issues. As listed in Table 6 the classifiers we trained are either worse or slightly better than randomly picking an agent terms of resolve rate.
While there are many factors that may impact an agent's performance, they might not be fully re- flected in the issue description. Also, the representations of the issues might not capture all these factors, as shows. We recognize the potential of training a DEI policy using pretrained LLM and a carefully curated dataset. Such a trained model could serve as a better judge model, dy- namically composing and sequencing different policies" partial solutions to address complex issues.
Table Resolve Rates with Different Issue Classifiers
input
output
accuracy
random baseline
which agent use which agent to use
26.6%
issue description classifier issue description + file structure
classifier
25.6%
which agent to use
27.0%
DEIBASE
issue desc. file structure generations
which patch best 33.3%
A.2 CAN DEI PROVIDE FEEDBACKS FOR PATCH REFINEMENT?
To some extent, yes.
We conduct experiments on Agentless generations to evaluate whether DEI explanations can help the model refine its potentially wrong generation.
We sampled 10 issues that were not solved by Agentless and got low scores from DEL We modified the bug- mig-fixing part of the agentless framework to include DEI's output and refined the patches for at most 5 rounds.
Once a patch gets more than 5 points from DEI, we stop refining it. With this refine process, the number of fixed issues among the 10 went from 0 to 1, 2, 3, 5, 5 in the 5 rounds. This result indicates that DEI feedback is not just useful for selecting the best candidate, but also for refining incorrect patches.
A.3 WHEN DOES DEI FAIL?
DEI follows our rubric to score the patches. Each stage in its analysis corresponds to some points in the scoring rubric and, therefore, needs to be analyzed separately.
We analyzed 20 of 35 failure cases of DEI in DEI-Open by manually annotating the stages (location explanation, patch explanation, conflict detection) where DEI failed to make the correct decision. For each casc, analyze for one false positive patch and one false negative patch. We list the results in 7
Note that each row can up to more than 20, because there can be multiple stages where DEI makes mistakes. From this table, we find that DEI tends to misled during the patch explanation stage of incorrect patch. For correct patch, it tends to mistakenly "find" conflicts with the existing code. Fewer errors are made during the location explanation stage. This indicates that DEI is better telling if the patch is modifying the correct file.
16

---

Published as a conference paper at ICLR 2025
Table 7: The number of DEI failures in different stages for different types of errors, among 20 annotated issues.
Location Explanation
Patch Explanation
Conflict Detection
False Positive
10
False Negative
2
15
A.4 AGENTS EVALUATED
We add the following agents to the DEI Committee (the one in Figure 3) in the following order (the order is generated by randomly shuffling their chronological order using python's 's random shuffle function with a random seed of 42):
20240612 Research Agent101
20240612 NABAl spt4o
20240601 CodeR
202 20223523 aider
20240630 agentless gp:4a
20240617 moatless 9pt-4
20243725 opendevin codeact claude35aomnet
20240706 sime gp
20240621 aur:ocoderrovrr=72220022
10.
A.5 PROMPTS AND EXAMPLES
A.5.1 PROMPTS
System Prompl:
You expert python for software engineering and code review: Your responsibility is review the patches generated by language models to fix some issues and provide feedback on the quality of their code.
17

---

Published as a conference paper at ICLR 2025
User Prompt: want you to evaluate LLM
task, you are provided LLM-ggneratee generated candidate putch that tries to resolve un issue in cadebuse.
assist you
following information:
You are given an issue text pithub repository (wrapped with descripeccpppioo -ttssc//ssseeeeeee description
issue. Each code span is wrapped with <code_span lile path=FILE PXTH span id=SPAN >D> </odde spun >tags where FILE PATH is to the lile containing the code span, and SPANJID is the unique ilentifier for the code span. Each code span also comes with the line numbers for you better understand context.
You are also given some identified code spans are relevant
You are given the candidate putch that tries to resalve the larget issue. For your convenience, you are given the hunks of original applying the patch. code after the patch is wrapped with <'after, pateerrppphhh>-- patchhr Note that the file names before.pateh starts with "a' and the file names atter-patch starts with "bv.
patch wrapped <hefore patch </hhefore patch?>and the
Iller's what you want do:
Explain in your own words what the issue is about. Ourput your explanation in <is- suc_exp>> <//ss//ecspppppppp expssecsppssgss...
Understand the issue.
2. Understand the identified code spans. First provide a list of the span ids, Then explain how each of the identified code spans are relevant to the issue. Output your explanation in <codee span exp> <foode. expl>tags.
3. Understand the candidate patch. First curate list of modified bunks. For each modified bunk, explain what it's doing. Output your explanation in the <pulch_exp>> </pptcch.ssppp lield.
Check if the patch fixing the correct function
Output your explanation in the
rect rect_lacatio_eep>> ///ommettlocatioooeee www.aionnnepp-ffeeeeeeo.eeee field.
especially contradicts with any of the identified code spans. Output your explanation in the <new ismes exp> </heww issues expl=field.
Chock the patch introducing any
issuc. Compare generated patch agains the common mistakes made by LLMs see if Talls into any of the categories. Be ruthless point potential mistakes. Output your explanation the lix expp <t/lis, issue uxpue field.
Check if the patch can
7. Finally,
your
Wrip your in <<sonee </score>>tags Make include in these tags only integer,
nothing clse.
Here's scoring rubric:
Your score should be an integer between 0 and 10, where higher scores indicate better quality. should give a score if you think the patch is invalid or there something wrong with it. For every contradiction between the identilied code spans and the patch. you should deduct point from the score. you think the putch not fixing the correct function, you should give O.
If you think the patch is introducing
you should deduct 2 points from
Your scoring should only be about the correctness of the patch, not about its quality or style. description>> > fissue text} </issue description>>
<chefore.pathh patch > [before patch) <c/before.patch>
<aftee-pptcch> {after {after-patch] </after_patch>>
{cude scode_spaas]
Again, make sure your output ends with <<senre>> Checore tags containing only an integer. For example, your score is 8, the final part of outpur should look like this: <'score's 8 <iscore's should not contain any other information characters.
18

---

Published as a conference paper at ICLR 2025
A.5.2 EXAMPLE 1
We list candidate patches for the same issue (sympy-133) 13971) and the evaluation for the candi- dates generated by DEL. In this case, DEI was able to give a high score to the correct candidate and lower score the incorrect candidate.
Candidate Patch (Incorrect):
diff --git a/ h//pyppypppittttttaeee //yppy//prrniiig/lltta.....mm py b/symmyy/printiinngaaaoo index 364b24alc5. .35522441c5..44455155555 100644
43d57015c3
s/sympy//rzittin///aeexxxx
b/symp//printtinn//tte..x
ee -1655, 6 +1655, 10 ee def _print_SegPormula (aelf, 8)
else:
printset tuple(s)
return (r"["
+ r", (el.iiisee..prii (el) for el in printset)
+ + r"]")
return (r"]left[" r", ".joi (rrnisseeep for el in printset) [""[righ]]")
Candidate Patch (Correct):
diff --git
h/smmy/prrnniing/laaxxxx
index 364b24ale5. 9edbaado30 100644 a/sympy/prrittiin///ttee.
+++ b/symp//printiin//laettxx
ee -1655, 9 +1655, 9 ee def (srntt,eegorrmiaaseell,,, s)
else:
printset (s)
return [e"]left["
return [e"]]"
r", ".join(sel=..rrinteee for el in printset) r"(right]") r'ightt))")
print.SecPer = print.SeeFormui
print.SegAdd .print..ee@Frmu.l
19

---

Published as a conference paper at ICLR 2025
DEI Output for Candidate (Incorrect Candidate, Final 2):
Understanding the issuc:
<issue. <issue_exp>> exp>>
The issue pertains to how the "SeqFormula(" command in the SymPy library is rendered in Jupyter Notebooks. Currently, the sequence output is backslash-escaping the square brackets producing expressions like "leeft][O, 4, ..... Aright' which do not render properly markdown cells. The desired oulput is an expression that does not backslash-escape the square brackets, '[0, 1, 4, 9, ...J'. </issue_exp>>>
Understanding the identified code spans:
List of span ids;
Pratxer print SeqFuuuula
Explanation of each code span:
<code_span_eep_>
The identified code span LateepPiiieer..iinneeeeoooaa the LaTeX formatting of sequence output. Specifically, it creates a LaTeX string representing the sequence by encasing it in "Left\.... "right,", which introduces backslash-escape square brackets because of the "V...' LaTeX format convention. This contributed to the issuc's improper rendering in markdown cells when copied from Jupyter. </code <ooode_spa__ep>>>>
'sympy/priating/ltexp"" handles
3. Understanding the candidate patch: List of modified hunks:
Hunk modifying return statements in Explanation each modified hunk: <patch_exp3
The candidate patch modifies the return statement of print SeqFormula" which previously returned the sequence encased in [Jeeit(...)riihtt]]]]
The patch adds return statement that just '[...]' for the sequence (without the "lleft" "rigght),, preventing the eeckslash-sscaping of square brackets. The original line is retained (but will not execute, given the placement of the new return line above it). </patch.exp>
Checking if the patch is fixing the correct function: <correct crrroct_loattoo_eee _epp>>io_eep>>>>>>>>>>>>>>>>>>>> Yes, the patch modifies the correct function print SegFormula" where the issue arises from. This function is responsible for rendering LaTeX sequence outputs. </correct location exp>
5. Checking if the patch introduces any new issues: <new _issuus_exp>>>>> The patch introduces redundancy and maintains the old behavior, since both modified hunks present. The original 'retum (r" [e"t", code kept after the new 'return (r"l" added in the patch. This means the second return statement ("Ileft)[' one) is unreachable. Ideally, to correct the issue properly, the original return should be replaced rather than decomposinn//uncechabbe placement. </new issuus.eess...
Checking if the patch can fix the issue:
<fix_issue_exp _exp>>
The patch can fix the issue because it generates sequence in "I ]r without hackslash-escapes hy rendering directly. However, since the problematic return code \( (wallaerr ]preserved (UnPerf=167**, the cleaned refactor redundannyyogexx---fff/f//s stylrplacement..
unnamedto exactly folding
Concerning issues commonly undermadyce fix suitable patch for combating template, artistry consensus mismanagement LLM tactics).
</fix_issue_ess
<score>-<<scoree>
20

---

Published as a conference paper at ICLR 2025
DEI Output for Candidate 2 (Correct Candidate, Final Score
<issue_expp "esse
The issue deseribed that when using the "SeqFormula" function from the sympy library, the output sequence renders with backslash-escaper brackets in Jupyter. This escaping does not render properly when copied markdown cell. Therefore, the issue requests that the sequence output should not backslash-escape square brackets, least resolve it in such way that copying markdown renders correctly.
</issuc_expp
<code_span_exp>
Span DDs: 'I "Latex Saaeeerriieer.rrntSeegeeo
The referenced code span is the core implementation of the " print SeqFormula" method within the sympy library. This function defines how sequences are formatted in LaTeX by wrapping them with "Jeff[" and "right]'. which causes the backslash-escape of brackets. The issue directly references the behavior observed span. <'ooe__span_exp>>>
<patch_exp>>
Modified Hunk:
Returning formatted sequence in -print,Seqformulaa
The only change made to modify the opening bracket from "Vleft[" to "(left[" the string returned. Specifically:
"'diff
return (e"
return (r"(|eft("
Explanation:
The patched hunk changes LaTeX syntax for sequences the priint SeqFormula' method from using backslash-escaped brackets to normal brackets. This change aims to stop the LaTeX parser from backslash-escapnng the brackets when rendering sequences. </patch.exp>
wwrnnct.occatio.coo
Yes, the patch modifies the correct function. The issue is with the pprint,SeqForuite method in the 'latex.py" file, and the line change appropriately reflects the adjustments needed to stop hackslash-escaping </correct location exp>
<new _issues_exp>>>>>
The patch does not introduce issues directly. Its primary operation is straightforwardly changing the escape mechanism, and this adjustment does not inherently conflict with any the identified required behavior in the span. </new aasueseexp"
<fix_issue.cd ixx_issueeexp>>>
The patch indeed addresses the immediate issue preventing backslash -escspin for square brackets in the LaTeX output. This adjustment should ensure that the sequence can be rendered properly in markdown, consistent with the issue's request. Common LLM mistakes such as incorrect function signature transformations or syntax errors aren't present here. </fi issue exp>>
<score's Soare>9</see 9///scor>>>>
21

---

Published as a conference paper at ICLR 2025
A.5.3 EXAMPLE 2
We list candidate patches for the same issue (django 13028) and the evaluation for the candidates generated by DEL. In this case, DEI was able to give a high score to the correct candidate and lower score to the incorrect candidate.
-
Candidate Patch (Incorrect);
diff --git a/ hww.a///oo///ooel//oo/cooo/////e /quury/uuuypppppp/ h/dhjango/cbbmmcd/ee/////// /uer index cc18098fd2. 034...33333333 100644
ee -1125, 10 +1125, 00 glass Query (Haseixpressioon)
def check filterable (self, expression):
"""Raise an error if expression cannet be used in WHERE clause. *M
if not getattr(e 'filterable', True): raise NotsupporrteEEror 'clause.'
is disallowed in the filter
sarnings.warnt
is marked as non-filterable.
"Tii may lead to unexpected behavicr.',
jagorr/ stacklevel=2,
return
if hasattr (expression, "get_sourreeeppressiini ) for expr in expression source expressions ()
(expr)
Candidate Patch 2 (Correct):
diff --git htc/janggo/a//oodeeee///// /quur//uuryyggre index ce hijjangg//b//mmddlsss//g///eerr
100644
+++ b/django/
ee -1,3 +1,5 @0 +Erom django. impert models
Croate SQL statements for QuerySets.
ee - +1125, ee class Query (Bscexxppxpr self hea.r..ccccckueer.ooffectt.p.p. (v, opts, field)
def fileccfiitteaaleecc (self, expression):
"""Raise an error if expression cannet be used in WHERE clause. *** ***
Raise an error if expression cannet be used in WHERE clause.
For model instances, the check is skipped to allow filtering by
model instances. ***
+ if isinstance (expression, models, Modell
return
if not getattr (expression, 'filtcrable', True):
raise NotSuppportedrror)
+ is disallowed in the filter
22

---

Published as a conference paper at ICLR 2025
DEI Output for Candidate (Incorrect Candidate, Final Score 4):
<issue_exp>>
The issue pertains to a 'NotSuporteeEEroo" being raised when attempting to filter a queryset where the right-hand hand side (RHS) of the filter contains a field that has the attribute "filterable" set to 'False". The error occurs while filtering objects of the 'ProductMetaData" model based on related ProductMetaDataTTys instance. The field 'filterable' having the attribute 'fiterable=False leads error during query execution. </issuc_exs exp>>
<span_ids>
Query.check filterable c
Query.build_filet
</span ids>>
<code_span_exp_>
Query..heck..e filterable': : This method is responsible raising an error expression in filter not filterable. The issue aligns with this method as it throws the 'NotSupportedEEroo which leading to the problem when a non-filterable field is involved.
"Query.buile buery.build_fllerr.. filter': This method constructs the necessary SQL for filter condition includ- ing checks whether the expressions can used filters. leverages "check_filterable to ensure the expressions provided are eligible for the filtration clause. </code span exp>
<patch_exp>>
Line range modified: 1125-1137 in djjnggodb//moddvv////varrryy
Explanation: The patch modifies the method 'check_file _filterable....... Instead of raising a 'NotSupport- edError" when expression marked non-filteralle is encountered, it now issues warning using "warnings. warn(' The warning informs that the expression is non-filterable and may lead to unexpected behavior. The warning category used is "RenovedanDaanoooVYrringg After issuing warning, it returns rather than aborting with an error. </patch.exp>
coorrect.location _exp>
Yes, the patch addressing the correct location in the codebase. The the one responsible for validating whether an expression can be used in a filter clause, which is the function that relates to the described issue. </correct location exp>
method
<new mie._isuueseexp>
Yes, the patch may introduce new issues. By changing the error to a warning and returning immediately, the system might still proceed with non-filterable expression in the query, leading to unexpected behavior or runtime errors down the line, even if the immediate 'NotSupportedError avoided.
</new issues_exps
ffixisssu.exxp)
The patch does not adequately fix issue. While prevents immediate 'NotSupportedErroo: transitioning to a warning does not solve the underlying problem of handling non-filten filterable fields in queries. Subsequent parts of the database query logic might still fail yield incorrect results. This approach sidesteps the problem rather than addressing how non-filterable fields should he properly handled alerted in safe manner. This fall into the common mistake of an overly simplistic fix that avoids one error but leads to potential other unseen errors in operational use. </lix issue exp>
<score <c/scre<
23

---

Published as a conference paper at ICLR 2025
DEI Oulput for Candidate 2 (Correct Candidate, Final Score 9): <issue exp>
The issue is about "NotSuportedErree that occurs when filtering 'PodductMttaDtta' with "metadata_type that has the "iltcrabll=False" attribute Django 3.0.7. The user tried to perform filter query and encountered this error indicating that fields with "iilerable=False" their models cunnot be used liller clauses. The worked around the issue by changing the field name 'label' to 'illtrrble__est but this change is not documented or an official fix. <xxp
<asan
Query-chokk cieccchhck.fiterbbll Query.build buily filter <</span ids
<conle.span.esp>
1. "Query checkk filterable": This code contains the definition of the 'check filterahle' method, which raises 'NotSupprrted Error' expression during the filter operation.
Faiterable==alse', function relevant because responsible error encountered
Cberyybuild_.iter" This code span includes the 'build_fiter" method, which constructs filter clause for SQL query. uses the "check_filterable" method to ensure that the values being used for filtering are allowed, making directly related the as </code span
<_putcheepp>>>
Modiiication 'check. filierable" method: The patch changes "check filterable' bypuss the check instamces "madels. Model'. This prevents the 'NaSupportedFFmor from heing raised when filtering using model instances have 'illeraleeFalse..
Original:
"python
def Dite..lillaaalleeeel.. expression):
Raise an emor expression cannat used in WHERE clause."
if getattriexpression, 'ilterable', Trie): raise NotSuportsdErroct
expression. class name disallowed filter"
"clause.'
basattreexpressinn, expresssion""
for expr in httpssiionngtttoureeeeessseeee wwwhhk.fitterallteee...
Modified:
""python
def Beteck.iilterbbleeeeee expression):
" Raise expression cannot used WHERE clause.
For model instances, the check skipped allow filtering by model instances.
ssinstance(exppeeete exesssion, Model:
return
il not geialtr(expressionn, 'lilterable' True):
raise NusSupportedEErro
ex pleessin...clas_______.____ _ame__ disallowed in the filter"
'clause.'
haspstriexpresssonn, 'get source ceceeespppppessssseee
for expr wwt.ssoo.gggsouuueeeeeeeeeeeeeee
chlckc filterable(espr)
</patch.sxp>>
<currect location expl>
patch modifying correct location. change directly targets 'check_fiteralle method, the "NolSappartedErnee" raised when using filter clauses with filtetabe attribules, described in the <fcorret location exp>>
responsible
<Ceww www._issussorr xxp
introduce issues and contrudict identilied spans. merely check bypass filterable check for model instances, which aligns with the current design construints seen in the previously analyzed code. </new jsspes_exxp>
palch
<ffx issme exp²
The patch adoquately addresses the ensuring model instances themselves subject 'filterable' constraint check. effectively preventing the "NotSugpotteBBeorr This solution appears sensible for the described problem since it allows maintaining liltering restrictions while enabling legitimate lilter queries containing instances with 'ilterable=='ake <//ix, issue_exp>>
<Sscore 9 9ccsuur
24


Reviews:
[{"review_id": "rur0VQ9ozm", "reviewer": "Reviewer_nvyo", "summary": "This paper introduces DEI (Diversity Empowered Intelligence), a framework designed to leverage the diverse capabilities of software engineering (SWE) agents. Through comprehensive analysis, the authors demonstrate that different SWE agents solve distinct sets of issues despite having similar overall performance metrics. The paper introduces DEI as a meta-module framework that harnesses this diversity through multi-stage rating and re-ranking, rather than attempting to build a single \"best\" agent. Their experimental results show significant improvements, with their best group achieving a 55% resolve rate on SWE-Bench Lite. The framework is well-documented with ablation studies and analysis.", "questions": "- Could you provide a deeper analysis of why different agents excel at different issues? For example, have you analyzed patterns in the types of issues each agent solves successfully or investigated how specific agent architectures (e.g., execution vs non-execution-based agents) relate to their issue-solving capabilities?\n- Could DEI's architecture be extended to provide feedback to the agents about why their patches failed, potentially enabling them to generate better patches?\n- Could you provide a detailed analysis of DEI's failure cases? For instance, are there specific types of patches or issues where the scoring mechanism consistently makes wrong decisions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is the first to investigate and establish that different SWE agents solve different types of issues.\n- Proposes novel DEI meta-framework that uses diversity to build a better resolver than any single agent.\n- Achieves 55% resolve rate on SWE-Lite beating baselines significantly.\n- DEI-Base Open using open source LLMs achieves a 33% resolve rate outperforming many closed source LLMs.\n- Well motivated with thorough analysis, and ablation studies.\n- The release of code and prompts  makes the research reproducible.", "weaknesses": "- While the authors demonstrate that different SWE agents solve different sets of issues, they provide no analysis of why this occurs (e.g., no investigation of which types of issues each agent excels at, or how agent architectures relate to their success patterns). This limits understanding of the relationship between agent design and issue-solving capabilities.\n\n- The framework uses fixed agent groups (DEIBASE-1/2/Open) where all agents in a set contribute patches that are then scored. There is no learning mechanism to dynamically select agents based on past performance or issue characteristics. As a result, the system relies entirely on inherent diversity and fails to leverage agent specialization.\n\n- DEI employs a fixed sequential review process (issue -> patches -> scoring) without iterative refinement or feedback loops between components. This linear approach potentially misses opportunities for improved patch selection that could be achieved through multi-round evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mCuWU5Xsqt", "reviewer": "Reviewer_wcfY", "summary": "The paper proposes using an esamble of agents to increase the number of software engineering (SWE) problems that can be solved showing that using different agents for different problems is better than using a single agent for all. The approach is based on a reinforcement learning formulation.", "questions": "1) It seems to me that Eq. 3 expects the context to never change. Is that correct? (You fix the context, then you select the optimal policy for that context and then you use the expected reward from that policy.)\n2) How do you select the optimal policy for a given context? Do you compute the expected reward for every policy given that reward and selected the highest? Or is this what is done in Step 3 Patch scoring?\n3) Is there any learning involved in computing the DEI policy? It seems to me that the policies are given and then it is a deterministic computation to select the best of those.\n4) Would it be possible to identify what the missing expertise is? I.e., if there was an agent capable of doing X, then the score would be significantly higher. There seems to be a potential to identify contexts that are far away from the contexts covered by the current policies. \n5) Have you considered composed policies? Maybe a but is complex and would require two different patches, i.e. first apply partial solution by agent a_i and then apply partial solution by agent a_j.\n6) If you have as many members on the committee as you have different agents, wouldn't you always get the oracle answer? \n7) If the statement above is correct, wouldn't the interesting question be: What is the minimal number of members on the committee to get an acceptable score? What is the cost ($) benefit (resolve %) trade-off? The results would be much more interesting if the cost of running the different options would be included. It seems that you should be able to get a very clear analysis which would allow developers to estimate how much it would be worth to them to fix an issue.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Important problem with significant real world value.\n* The paper is well written and easy to follow.\n* The evaluation clearly shows both the added value of the approach and how much better it could be.\n* Considers both intra and inter agent diversity.\n* Good potential to extend the approach, see questions.", "weaknesses": "* It is unclear what new knowledge we have gained from the paper. That diversity is good is expected. To select the best candidate for each individual problem seems also expected. What is the unexpected or novel result of the paper? Or is the main result that you managed to get to work? - The authors clarified that it is how easy it is to achieve diversity and how significant it is.\n* The significance of the approach might be on the low end. Since the approach is effective but straight forward, more elaborate analysis and/or experimentation would be expected. - Based on the authors response, the significance is higher than I initially estimated, but still somewhat limited.\n* Some things are unclear, see questions. - Most of these were addressed by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GA1s02VYfX", "reviewer": "Reviewer_aexB", "summary": "The paper focuses on software engineering (SWE) agents, i.e., AI agents, in support of various software engineering tasks, such as code generation, automated testing, and project management. Specifically, in this work, the authors focus on AI agents' ability to resolve GitHub issues based on their descriptions. Observing that different SWE agents resolve different sets of issues successfully despite having similar resolve rates, the authors propose a method for organizing various SWE agents into ensembles with the intent to use the union of their expertise in resolving GitHub issues. The proposed method, Diversity Empowered Intelligence (DEI), is evaluated using realistic issue tracker data. The results show superior performance compared to single SWE agents' performances.", "questions": "- Sec 4.4: Are these results statistically significant?\n\n- Is it possible to add human developers to the committee? (Fig 1c.)", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The topic is important. There is a clear emerging need for systematic methods for using LLMs, especially for software engineering.\n- The method seems to be novel.\n- The quality of the experiments is solid.", "weaknesses": "- The method is not exactly systematic, which limits its applicability and forces an ad-hoc way of working with ensembles of AI agents.\n\n  - Sec 3.3.2, Equation 3: It is clear at this point that diversity is not purposefully engineered but rather: it's an emergent property of the system. One then has to wonder if this approach can be purposefully used in any problem or if its success will be rather non-deterministic and dependent on the problem at hand and its parameters (~ context).\n\n  - Sec 3.3.2, L214-215: \"The DEI framework leverages the strengths of each agent in their respective contexts to enhance overall performance across all contexts.\" - How does the framework account for intra-agent diversity? It seems that Figure 1 is rather deterministic, though some non-determinism is likely.\n\n- Often vague formal underpinnings\n\n  - Sec 3.3.1 is rather vague to provide formal underpinnings to the contributions. For example, L182: \"The context space, C, includes relevant repository information and issue descriptions.\" What does \"relevant information\" entail here?\n\n- Potentially limited mitigation of inconsistent agent behavior\n  \n  - Sec 3.2: \"We consider two types of diversity: intra-agent diversity and inter-agent diversity.\" - I don't entirely agree with the proposed term of \"intra-agent diversity\". Diversity seems to be an artifact of non-determinism in this case, i.e., basically a temporal inconsistency. As such, it requires different means to mitigate, compared to the diversity stemming from the number of different agents. The method does not seem to acknowledge this.\n\n\n- Typo: L057, citation \"Team, 2024\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XIoWMwtTqy", "reviewer": "Reviewer_zt1v", "summary": "The authors propose “Diversity Empowered Intelligence,” a method for aggregating the solutions of several SWE agents by means of LLM prompting. They show a significant increase in scores on SWE-Bench Lite and attribute this to the fact that different SWE agents can solve different subsets of the problems.", "questions": "[1] In “4.1.2 Evaluation Mertics” for Union@k: “Solved by any of the k solutions.”, what is the mathematical definition of “any”?\n\n[2] Likewise, for Intersect@k, “solved by all k solutions”: is k tied to the number of agents at hand, or is it a free parameter? If independent, which agents are chosen from the (unordered) set of agents?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[1] The authors systematically explore the benefits of ensembling of SWE agents and show significant improvement over single-agent solutions.\n\n[2] Figure 1 nicely illustrates the idea behind DEI. Table 1 is very insightful and provides strong evidence of the value of DEI.", "weaknesses": "[1] The results are presented for SWE-Bench only. Even though SWE-Bench is a fairly reliable and diverse benchmark, validating DEI on another SWE benchmark could solidify the claim.\n\n[2] The improvement in scores comes at the cost of significantly higher token usage (or compute).\n\n[3] The following papers are closely related but not cited:\n\na) SELF-CONSISTENCY IMPROVES CHAIN OF THOUGHT REASONING IN LANGUAGE MODELS\n\nhttps://arxiv.org/pdf/2203.11171\n\nSince a common decision is made based on a diverse set of candidates.\n\nb) Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena\n\nhttps://arxiv.org/abs/2306.05685\n\nSince the scoring by an LLM is prompted, which is essentially LLM-as-a-Judge.\n\n[4] The metrics in 4.1.2 are tied to the order in which the agents (and, by extension, their proposed solution patches) are added to the list. This is unacceptable for a metric as the solutions are fundamentally an unordered set. The metrics should be refined to be permutation invariant towards the set of agents. Whereas the proposed metrics are flawed in the general case, the main results in section 4.2 that are collected with the specific ordered list of agents are not flawed and have analytical value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)