BOAD: Discovering Hierarchical Software Engineering Agents via Bandit Optimization
Authors: Iris Xu
Guangtao Zeng
Zexue He
Charles Jin
Aldo Pareja
Dan Gutfreund
Chuang Gan
Zhang-Wei Hong
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-armed bandit
Model selection
Software engineering
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Large language models (LLMs) have shown strong reasoning and coding capabilities, yet they struggle to generalize to real-world software engineering (SWE) problems that are long-horizon and out-of-distribution. Existing systems often rely on a single agent to handle the entire workflow—interpreting issues, navigating large codebases, and implementing fixes—within one reasoning chain. Such monolithic designs force the model to retain irrelevant context, leading to spurious correlations and poor generalization. Motivated by how human engineers decompose complex problems, we propose structuring SWE agents as orchestrators coordinating specialized sub-agents for sub-tasks such as localization, editing, and validation. The challenge lies in discovering effective hierarchies automatically: as the number of sub-agents grows, the search space becomes combinatorial, and it is difficult to attribute credit to individual sub-agents within a team. We address these challenges by formulating hierarchy discovery as a multi-armed bandit (MAB) problem, where each arm represents a candidate sub-agent and the reward measures its helpfulness when collaborating with others. This framework, termed Bandit Optimization for Agent Design (BOAD), enables efficient exploration of sub-agent designs under limited evaluation budgets. On SWE-bench-Verified, BOAD outperforms single-agent and manually designed multi-agent systems. On SWE-bench-Live, featuring more recent and out-of-distribution issues, our 36B system ranks second on the leaderboard at the time of evaluation, surpassing larger models such as GPT-4 and Claude. These results demonstrate that automatically discovered hierarchical multi-agent systems significantly improve generalization on challenging long-horizon SWE tasks. Code is available at https://github.com/iamxjy/BOAD-SWE-Agent.

Published as a conference paper at ICLR 2026
BOAD: DISCOVERING HIERARCHICAL SOFTWARE EN- GINEERING AGENTS VIA BANDIT OPTIMIZATION
Iris Xu'; Guangtao Zeng", Zexue Het, Charles Jin Aldo Pareja', Dan Gutfreund?, Chuang Gan 2.5, Zhhng-Waar Wee Hong²
ABSTRACT
Large language models (LLMs) have shown strong reasoning and coding capa- bilities, yet they struggle to generalize to real-world software engineering (SWE) problems that are long-horizon and out-of-distribution. Existing systems often rely on single agent to handle the entire ing large codebases, and implementing fixes- within one reasoning chain. Such monolithic designs force the model to retain irrelevant context, leading to spu- rious correlations and poor generalization. Motivated by how human engineers decompose complex problems, we propose structuring SWE agents as orchestra- tors coordinating specialized sub-agents for sub-tasks such as localization, editing, and validation. The challenge lies in discovering effective hierarchies automati- cally: as the number of sub-agents grows, the search space becomes combinatorial, and it is difficull to attribute credit to individual sub-agents within a team. We address these challenges by formulating hierarchy discovery as a multi-armed bandit (MAB) problem, where each arm represents candidate sub-agent and the reward measures its helpfulness when collaborating with others. This frame- work, termed Bandit Optimization for Agent Design (BOAD), enables efficient exploration of sub-agent designs under limited evaluation budgets. On SWE-bench- Verified, BOAD outperforms single-agent and manually designed multi- systems. On "WE-bench-Live featuring more recent and auttof-dditribution is- sues, our 36B system ranks second on the leaderboard the time of evaluation, surpassing larger models such as GPT-4 and Claude. These results demonstrate that automatically discovered hierarchical multi-agent systems significantly im- prove generalization challenging long-houe monghooizoon SWE tasks. Code is available at https: / thu..htuu..... com/ iamx jy BBADD SWE Agent.
issues, navigat-
INTRODUCTION
Large language models (LLMs) have achieved remarkable progress in natural language processing [38] and reasoning [21], and are increasingly adopted in solving complex coding problems [58]. Yet solving real-world software engineering (SWE) problems remains challenging [25] for LLMs, particularly for issues that fall outside the training distribution [54], Despite strong results on SWE- bench-Verified [25], state-of-the-art systems struggle on more recent and out-of-distribution issues in SWE-bench-Live [54].
We hypothesize that one agent to solve a problem within a single reasoning chain. Solving a software engineering task requires handling multiple sub-tasks- -uch locating relevant files, editing code, and running tests each of which depends only on a subset of information. For example, during code editing, the model only needs know which line modify, not how that file was previously located. Retaining such irrelevant context introduces spurious correlations [57; 36), leading models trained on long reasoning chains to overfit to training distributions and degrade on duittf-diitribution problems.
current SWE agents generalize poorly is their reliance a single
Motivated by the need to limit irrelevant context during long reasoning processes, we introduce explicit hierarchical structure to LLM- agents. This design choice mirrors how human engineers
"Correspondence: irisxo Emi t. edu, Massachusetts Institute of Technology", MIT-IBM Watson Al Lab? Independent researcher" Stanford UMass 2mherst

---

Published as a conference paper at ICLR 2026
solve complex problems coonniivve science studies show that humans reduce cognitive load by decomposing complex tasks into manageable sub-tasks and integrating their outcomes to form the final solution [32; 37; 33J. Recent work [55] demonstrates that decomposing complex task into smaller sub-tasks improves LLM performance. In software engineering, such decomposition commonly maps stages like bug reproduction, fault localization, code modification, and validation [45; 24; 31j.
This same idea appears as temporal abstraction in hierarchical reinforcement learning (HRL) [U]] instead of solving problems step by step, the agent delegates control to reusable sub-agents, each defined as policy for a specific sub-task [40]. An orchestrator coordinates these sub-agents by selecting which one to activate, and each sub sub-agent runs until it completes its sub-task [40; 19; 10]. Planning at the level of sub-agents shortens the effective decision horizon and abstracts away low- level details, allowing the orchestrator to focus on high-level structure. This abstraction supports generalization because sub-agents capture task- invarint Skills-such as repo navigation, information retrieval, or code modification-ttae can be recomposed across different tasks and contexts, rather than overfitting to task-specific action sequences.
Although hierarchical multi-agent systems for SWE tasks appear promising, prior attempts have achieved only limited success. Existing approaches [8; 34; 14] typically rely on manually constructed systems, where sub-agents assigned to predefined sub-tasks (c.g., locating faulty files) and coordinated by a human-designed orchestrator. These designs depend heavily on human intuition to decompose tasks and assign agent roles, but such decompositions often fail to align with how LLM- based agents actually reason and behave. Because LLM behavior is difficult to predict, workflows that appear logical to humans can perform poorly in practice (Section 5).
This motivates automating multi-agent design, which removes the need for manual specification and enables the discovery of effective agent designs that may not be obvious to human designers. While prior work has explored optimizing the design of single-agent SWE systems [51]. these methods do not address the additional challenges of multi-agent coordination, where agents must interact and share information dynamically. Other approaches optimize fixed, multi-step workflows involving multiple L.LM calls [26; 52; 23], but they still rely on predefined pipelines. In contrast, SWE agents operate in open-ended environments where execution order and interactions are not fixed in advance.
Automating multi-agent design introduces two main challenges. First, as the number of sub-agents grows, the design space expands combinatorially, and evaluating each multi-agent design on SWE tasks is far slower than on simpler reasoning benchmarks like multi-hop QA [23]. Second, the evaluation signal is coarse and noisy- each sub-agent's 's contribution. A team of sub-agents may succeed even if one sub-agent consistently makes mistakes and others compensate, making it difficult to assign credit accurately.
failure of the multi-agent system reveals little about
To address these challenges, we take a bottom-up approach: rather than exhaustively searching over all combinations of sub-agents, first identify promising sub-agents individually and then compose them into a team. This makes the search space grow linearly instead of exponentially with the number of sub-agents. The key difficulty is credit assignment-2 a sub-agent's 's value cannot be measured in isolation since it must collaborate with others to solve an issue (e.g., localizer alone cannot edit code). Instead of judging success by whether the entire team solves a problem, we estimate each sub-agent's 's helpfulness, measuring how much it contributes to solving the problem when in combination with other sub-agents. We use -udd-e-- [29] to assess this contribution from the agent's 's behavior during the problem-solving trajectory, yielding more informative signal than binary success or failure outcome.
Once helpfulness is defined, the next question is how to efficiently discover effective sub-agents. Evaluating a sub-agent once is insufficient because its performance depends on its teammates and the task instance, both of which vary stochastically. Yet repeatedly testing all sub-agents would be prohibitively expensive. To balance exploration (trying under-tested sub-agents) and exploitation (focusing those that have performed well), we formulate multi-agent design bandit (MAB) problem [17; 1; 20; 50], where each arm corresponds to a sub-agent and the reward is its helpfulness. Well-established exploration algorithms in multi-armed bandits (MAB) [28] provide an effective toolkit for adaptively selecting sub-agents to evaluate and efficiently discovering a set of helpful ones. We refer to this framework as Bandit Optimization for Agent Design (BOAD).
multi-armed
2

---

Published as a conference paper at ICLR 2026
We evaluate our BOAD on SWE-bench- Verified [25], benchmark for software engineering tasks grounded in real GitHub issues. On SWE-bench- Verified, our method consistently outperforms single-agent systems and manually designed multi-agent systems. On SWE-bench-Live [30], which includes more recently collected issues and presents out-of-distribution challenges, our system ranked second the leaderboard at the time of evaluation using 36B model- outperforming larger- scale systems based on Claude and GPT-4. To our best knowledge, our work is the first to show generalization improvements using automatically discovered hierarchical multi-agent systems on challenging long-horizon interactive tasks like SWE-bench.
2 RELATED WORKS
Prior works [8; 34; 14] built multi-agent systems for SWE tasks by manually designing the prompts of each sub-agent and the orchestrator that coordinates them. Each sub-agent handled specific sub-task (e.g., issue localization). While intuitive, these systems required heavy engineering and yielded only marginal gains, leading to limited follow-up work in this direction. Other efforts [30; 53] leveraged diversity across multiple agents or models to improve accuracy through ensembling, where each agent generated patch and an aggregator selected the final solution. In contrast, we aim to automatically learn a hierarchical multi-agent pestem-compriie uub-agents-withee manually defining their roles or coordination.
an orchestrator and multiple
Automating multi-agen design is closely related to prior work on optimizing single-agent systems. Existing approaches have focused on improving an agent's prompt [26; 2: 22] or its scaffold [51], including both tools and prompts, by iteratively refining these components based on task performance. These methods typically optimize single agent for a given task. In contrast, we study the automated design of multi-agent systems, where different agents are specialized for distinct sub-tasks and must coordinate to solve the overall problem.
Closer to multi-agent systems, several prior works study how optimize the workflow among agents. In these approaches, a workflow may be represented as a Python function that orchestrates multiple LLM calls Hu et al. [23], or graph of agents with distinet roles and prompts [52; 56]. While effective in their respective settings, such workflow optimization methods have seen limited adoption in SWE tasks. One likely reason that SWE tasks often demand fine-grained, dynamic coordination that evolves as the task unfolds. In contrast to workflows that are largely specified prior to execution, multi-agent systems for SWE typically require agents adapt and coordinate on the fly in response to intermediate outcomes and newly discovered information.
Prior work has explored automatically generating multi-agent systems [16; 15; 49], but these ap- proaches typically assume fixed execution scheme, such as predefined agent ordering [16; 15] or parallel aggregation of agent outputs [49]. In contrast, our work focuses on optimizing how agents coordinate with one another. In addition, existing studies primarily consider simple, inexpensive tasks, whereas we target SWE tasks that involve long-horizon, multi-turn interactions and are costly to evaluate, motivating the need for higher sample efficiency.
3 PRELIMINARIES
Software engineering agents We study the problem of using LLMs to resolve real-world GitHub issues, where each issue consists of textual description and corresponding code repository. Since issues are not self-contained, solving them requires identifying and modifying relevant parts of the codebase. In this work, we focus exclusively on agentic methods [48], where an LLM interacts with a runtime environment through tool use. Such agents can browse files, execute shell commands, run tests, and edit code directly, giving them the flexibility to tackle long-horizon tasks end-to-end.
Markoy Decision Process (MDP) We model agent- envirtnenvioomenn interaction as a finite-horizon Markov decision process (MDP) [35], M = A, r, H). At each step f, the agent observes state 8; E S, consisting of the issue description and the history of prior tool interactions h4-1 = (01, .1..... Q4-1. 0:-1). The agent samples an action a E AA from its policy <(a 8t). where AA includes all available tools and commands. Executing a, yields observation o, E O (e.g.. logs, diffs, or test results), updating the state to $ - A trajectory T = ($o. op..... Sr, 3) terminates T < H when the agent submits patch y (forced at 7 H if none is submitted earlier). Rewards
3

---

Published as a conference paper at ICLR 2026
sparse:
r(st, aT) if passes all tests,
((t, for T,
otherwise.
The agent's goal is to maximize the expected rewards J(rr = (r(st, With sparse rewards and long horizons, discovering successful trajectories is challenging.
Temporal Abstraction via Semi-MDP (SMDP) The Semi-Markov Decision Process (SMDP) framework [39] is widely used to mitigate long-horizer sparse reward problems. Instead of issuing primitive actions a, E A. the orchestrator selects temporally extended action (option) wt 92 Each option corresponds to sub-agent that executes a sequence of actions (at..... at+m-1) until termination, after which control returns at St+m, where denotes the duration of sub-agent. This reduces decision frequency and simplifies planning.
Multi-Armed Bandit (MAB) Multi-armed bandit (MAB) [28] is a special case of an MDP with a single state and no transitions. At each round t, the learner selects arm a E A. receives stochastic reward Tt E [0, 1] drawn from an unknown distribution, and seeks to maximize the cumulative reward El Tt over fixed interaction budget B. The MAB framework captures decision-making under uncertainty when only a limited number of trials are available.
4 METHOD: BANDIT OPTIMIZATION FOR AGENT DESIGN (BOAD)
Our goal is to automatically discover set of K sub-agents Ω = {Wi, wk} together with an orchestrator æ that maximizes expected task performance (i.e., reward) r. straightforward approach is to directly optimize over both components using evolutionary [13] or random search [12]:
max Eg EgnDanngssrr [r(sr, ar)) T..
where Daesign denotes a design set of representative problem instances. However, evaluating a candidate pair (π,, .2) requires executing full interaction trajectories and repeatedly querying the reward function r. software engineering (SWE) tasks, such evaluations are long-horizen -oorizon, multi-turn, and computationally expensive, making naive joint search over (л, Q2) impractical.
Agent design as a multi-armed bandit. To address this challenge, we reformulate agent discovery multi-armed bandit (MAB) problem [28]. In this formulation, each corresponds to can- didate sub-agent w,, and at each round f, a subset of K sub-agents is selected and evaluated under an orchestrator. Casting agent design as a bandit problem allows vation-exploitation strategies [27; 9]: these algorithms can allocate more evaluations to promising sub-agents while still exploring alternatives. This substantially reduces wasted computation on poor designs and makes automatic discovery of multi-agen systems feasible under the high evaluation costs characteristic of SWE tasks. We present the formal bandit formulation Section 4. 1. Towards this end, we must solve the following problems:
leverage principled explo-
1. The space of possible orchestrators and sub-agent sets is vast and initially unknown, making it infeasible to pre-enumerate arms, required by standard bandit algorithms.
Even evaluate sub-agent along with an orchestrator and the other sub-agents, credit assignment is ambiguous: some sub-agents may succeed only by "free-riding" on others, so the observed reward does not necessarily reflect their individual contribution.
Next, we illustrate how we tackle these challenges in the following sections.
4.1 AGENT DESIGN AS A MULTI-ARMEI BANDIT PROBLEM
The space of orchestrator
pairs (T, Ω?) is vast and infeasible to enumerate. To make the search tractable, we maintain an archive of candidate sub-agents. Instead treating each subset of sub-agents Ω as arm, we treat each sub-agent across different sub-agents (will be explained below). At each round f, the algorithm selects a subset Ω C T by choosing K arms, instantiates orchestrator ne for 2, evaluates (mz, P2)) on example
as arm, enabling information sharing
4

---

Published as a conference paper at ICLR 2026
problems from design set, and propagates feedback all participating sub-agents. Feedback calculated for each sub-agent independently to tackle the credit assignment problem (details in 4.2). Because sub-agents appear in multiple subsets, each evaluation, even if unsuccessful, provides signal for multiple subsets at the same time. This formulation supports efficient credit assignment, reduces redundant exploration, and progressively refines estimates tw for each E Т. Algorithm 1 summarizes the procedure, with further details provided below.
Bootstrapping a sub-agent archive We begin with an initial archive To of candidate sub-agents. This archive generated by prompting an LLM with the template in Appendix A.1.2. However, simply generating sub-agents is insufficient because the orchestrator may not know how to invoke them. We present sub-agents as tools to the orchestrator and adopt the standard tool-calling protocol from SWE-agent [48]. To call sub-agent, the orchestrator must parse its docstring to understand functionality and supply the required inputs. For example, an issue-localizer sub-agent requires the issue summary as input; without it, the sub-agent cannot operate. To ensure this, we introduce a warm-up stage that rewrites each generated sub-agent's docstring into a precise specification of its imputs and outputs, enabling the orchestrator to integrate it correctly. Details are provided in Appendix A.1.1.
Sub-agent evaluation At each round I, we select a set of K sub-agents Ω4 = (wi,..... wk) C Tr- Given this set, orchestrator ne is instantiated by prompting LLM (see Appendix A. 1.3,,,, and the system (Te, Stt) is evaluated on subset of example problems from design set. This evaluation yields performance score uw [o, 1] for each sub-agent E S4, A straightforward choice for uw is the success rate of the system on the design set, but as discussed in Section 4.2, this metric suboptimal and we propose more effective alternative.
Balancing exploration and exploitation on sub-agent selection After bootstrapping the archive, the next challenge is deciding which sub-agents to evaluate in each round. To balance exploration with exploitation, we adopt the Upper Confidence Bound (UCB) [28] strategy. For each sub-agent E I1-1, we track its empirical mean (t) of the performance score of uw and selection count (t) up to round t. The UCB score sub-agent w at round t defined as
2 lnt
UC (1)
(l)
The first term favors sub-agents with high observed performance, while the second term gives optimism bonus to under-sampled sub-agents (i.e., exploration). At each round I, we select the top-K sub-agents based on their UCB scores, ensuring that evaluations increasingly focus on strong candidates while still allocating time to uncertain ones.
Expanding the archive A fixed archive risks stagnation: once UCB identifies a few strong sub- agents, it will repeatedly exploit them, leaving little opportunity to discover new behaviors. To address this, we expand the archive dynamically using Chinese Restaurant Process (CRP) [3; 44]. At each round t, we prompt the LLM to generate a new sub-agent distinct from those the current archive r (see Appendix A.1.2). The probability of introducing a new sub-agent is
0
Pr(new at t)
0 + |
where 0 > 0 is concentration parameter. This mechanism ensures diversity: when the archive is small, new sub-agents are frequently added; as the archive grows, the probability decreases, shifting the emphasis toward reuse of existing ones. Over time, the expected number of distinct sub-agents after T rounds grows as ( log T",, providing unbounded but controlled expansion. We also run the warmup stage (Appendix A.1.1) to ensure the sub-agent is usable by the orchestrator.
4.2 HINDSIGHT CREDIT ASSIGNMENT
central challenge in our framework is defining the performance score u. of individual sub-agents A simple approach is the score of sub-agent to the success rate of all trajectories that
5

---

Published as a conference paper at ICLR 2026
Algorithm Bandit Optimization for Agent Design (BOAD)
Require: budget B, number of sub-agents to select K, concentration 0
I: Initialize archive To < BOOTSTRAP.
2: for 1 ........ do
3: With probability 0+ create a new sub-agent whew and set Pe + Te-1 U {weew):
otherwise set I, + PT-1-
4:
for each w T do
5:
if new (t 1) then
6:
UCB=(() <- +00
force initial exploration
7:
else
8:
UCB + in 1) +
2 2
(t-11
9:
end if
end for
11: Select top-K sub-agents based on UCB scores as set of sub-agents Ωz
Instantiate orchestrator и, conditioned Ω4.
13:
Evaluate (Tr, S2y) on a subset of training problems; observe performance score u. E [0, (Sec. 4.2).
14:
Update (t) and (t) (t) for each w e 2z-
15:
end for
include it:
Uw
Σ TET3
1(r is successful),
[
where Ty is the set of trajectories round / in which w is used by the orchestrator However, this suffers from a "free-rider" problem: a sub-agent may appear effective simply because it often co-occurs with strong sub-agents, even if it contributes little itself.
To overcome this, we adopt a hindsight-baace -based credit assignment strategy. The idea to reward sub-agent whenever its actions help the orchestrator make progress toward solving the problem, even the orchestrator ultimately fails. Thus, sub-agents that provide useful intermediate steps credited, while those that do
penalized, regardless of the outcomes. Concretely, let r= (a1, 01...., aT,, OT) denote the trajectory of actions and observations produced during problem solving. For each sub-agent that appears in query LLM judge (Appendix A.1.4) with the trajectory and obtain a binary label (T) E {0,1), where lu (T) = 1 indicates that the LLM judge deems w's contribution in the trajectory as helpful. The performance score of sub-agent then defined as the empirical average over all evaluated trajectories:
Σ (T) TET!
The
This hindsight-basee score uw E [0, 1] provides a more reliable estimate of the utility of w than success rates. By directly linking credit to judged contributions, avoids free-riding effects.
EXPERIMENTS
Our experiments address the central question: Can properly designed hierarchical multi-agent systems improve the generalization performance of SWE agents? We further analyze how the systems discovered by our algorithm differ from human-designed ones and examine the contribution of each design choice to the overall performance gains.
5.1 SETUP
Task format and datasets. We evaluate on the SWEEBENCH benchmarks: SWE-BENCS BENCH VERIFIED (500 instances) [25] and SWE-BENCH LIVE (300 instances) [54]. VERIFIED is a curated, frozen set real GitHub issues, while LIVE continuously adds newly collected, human-verified issues from active repositories, making it more resistant data contamination [46] and better suited for

---

Published as a conference paper at ICLR 2026
testing generalization to uut-of-distribution problems. Each instance includes GitHub issue, sepssitrryspeccfiic container image, and an executable test harness. The agent must interact with the repository (files and, when available, history) and produce patch that resolves the issue by passing all tests (pass-to-pal pass and fai mall-to-pass.
To avoid overfitting and limit design-time compute, construct small yet diverse design set by sampling one random issue per repository (12 total) from VERIFIED, which we find to be sufficient based on an ablation over design-set size (A.3.1). The design set is disjoint from all issues in LIVE, All results in Tables 2 and 3 are reported on VERIFIED and LIVE (lite) splits. We also report the result of BOAD on VERIFIED (HELD OUT) that exclude the 12 issues used in the design set.
Optimization Details During BOAD optimization, we run B 20 rounds of the bandit loop (Algorithm 1) (testing on up to 100 rounds shows that sub-agents created after around 20 rounds are generally worse, and 20 iterations is enough to converge rankings align). Each time new sub-agent is created, the sub-agent first goes through warm- up process, which uses randomly sampled instances from the design set to iteratively refine the documenttaioon/nsstacce prompt of the sub-agent, ensuring that the sub-agent is usable. We use W = 4 rounds for the warm-up process. Each round samples K = 3 sub-agents and evaluates them the design set. The optimization took around 12 hours and 440GB RAM. For running SWE-agent (orchestrator and sub-agent LLM inference), we deploy Boed-OSS-36B--nnstee
state where helpful rate and UCB
machine with 56 CPU cores
on one H100 GPU node. Note that by the tenth iteration, the top-2 most helpful subagents are the same as the ones converged on later. Running B = 10 iterations took less than hours.
Implementation. All experiments use the SWE-AGENT scaffold with a set of default tools from SWE-agent [48]: edit anthroicc (file viewing/editing), bash (restricted shell commands), and submit. t. The orchestrator calls sub-agents through the same XMI.-based tool-calling API, passing information via context parameter; sub-agents return outputs through this channel without access to the orchestrator's ' history. We use Claude-4 to generate candidate designs (Section 4.1) with temperature 0.0 and evaluate sub-agent helpfulness (Section 4.2). For execution, both orchestrator and sub-agents use Seed-OSS 36B-Instruct with temperature 0.0, unless specified, a strong instruction- following model that is not heavily tuned on SWE tasks. This choice ensures improvements reflect the benefit of orchestration rather than fine-tuning on SWE-task specific data. Each sub-agent is equipped with prompts discovered by BOAD, defined with docstrings and argument specs, and given access to the same default tools as the orchestrator.
Finally, for evaluation, we the two subagents with the highest helpfulness score (the hindsight- based score) found during the subagent discovery (Appendix 1.2.1). We also ablate by varying k in the top-k selection and by using success rate instead of helpfulness as the ranking metrie (Section 5.3).
5.2 MAIN RESULTS
Success Rate Table shows that with Seed-OSS-360 2-SS-66BIInstrcct BOAD resolves 20.0% % of issues on LIVE, ranking second on the leaderboard at the time of evaluation and outperforming larger models in popular scaffolds (e.g., GPT-4o. DeepSeek- V3, GLM-4.5-Air, Claude 3.7 Sonnet). This is a 63 % improvement the same model with default SWE-agent tools. On VERIFIED BOAD achieves 13.12%, surpassing many larger models (e.g.. GPT-40, Claude 3.5 Sonnet OpenHands, DeepSeek-RI. DeepSeek-V3) and setting a new state of the art among smaller models (c.g.. Swwe3-Cooder-0B-AA. Instruct, Devstral- Soalll-Smll--2500 with a 13.4% 4 gain over the default SWE-agent. Interestingly, adding manually designed sub-agents (Appendix A.2.4) from prior work [8: 34; 14] lowers performance, indicating that human-crafted roles can be misaligned with LLM behavior. Overall, these results demonstrate that BOAD automatically discovers wichestrator-sub-agerr structures that not only boost in-distribution performance but also generalize more effectively to out-of-distribution tasks.
Comparison with Evolutionary Baseline We also implement an evolutionary search for multi- agent system adapted from Automated Design of Agentic Systems [23], as baseline to compare BOAD against. Implementation details for the evolutionary search provided in Appendix A.4.1. When using the same number of evaluation instances (i.e., number of SWE-Bench patches generated), find that the sub-agents generated by the evolutionary search achieve worse performance (17.0%
7

---

Published as a conference paper at ICLR 2026
vs 20.0% on SWE-Bench-Live than BOAD. Additionally, for the same number of iterations, the cost of Claude API calls is more than double that of BOAD due to the need of generating many sub-agents at each iteration, whereas BOAD reuses subagents across iterations.
Table I: Success rate on SWE-BENCH VERIFIED and S SWE-BENCH LIVE
Scale
Model
Scaffolld
Verified Resolved (%)
Live Resolved
GPT-4o [48]
SWE-agent
23.0
10.0
GPI-4o [47]
Agentless
38.8
11.7
Claude 3.5 Sonnel [5]
Agentless
50.8
Claude 3.5 Sonnet
OpenHands
53.0
Large
Claude 3.7 Sonnet
SWE-aagen
62.4
13.7
Claude 4.0 Sonnet [6]
66.8
Claude 4.0 Sonnet
OpenHands
70.4
DeepSeek-RI [21]
Agentless
49.2
VeepSeek-V3
Agentless
GLM-4.5-Air [41]
OpenIlands
57.5
GLM-4.5--ir [41]
SWE-Agent
17.7
Qwen3-Coder 480B/A35B Instruct [43]
OpenIlands
69.6
24.7
Owenl-Codde:3BBBAABB--eme
SWE-agent
Small
-odeee--ooe-3---33------ [43]
Open
51.6
Devstral-Small-550/
OpenHands
Seed-OSS-36B-Instee [42]
SWE-agent (baseline)
49.8
12.3
Seed-OSSS-68-Lnsttuutt [-2]]
SWE-ageet Manual Sub-agent
47.4
14.0
Berdl-00S--66B--nstrudd [42]
SWE SWE-agent Evolutionary Search
46.0
17.0
Sedd-OSS-36B-Inttuutt [42]
SWE-agent BOAD
53.2
20.0
Token Analysis In addition to success rate, we analyze the test-time token usage of the hierar- chical multi-agent system discovered by BOAD in comparison to the default single-agent system. Hierarchical multi-agent systems introduce communication overhead, since agents must exchange information, but they can also reduce context length: sub-agents focus on specialized sub-tasks while the orchestrator handles high-level coordination without low-level details. Table 2 compares token usage between SWE-agent and BOAD. Total tokens refer to the average sum of input and output tokens per issue, while max input tokens capture the average maximum input length per instance. Surprisingly, the total token count is comparable-and even lower on SWE-bench-live- the original SWE-agent. Moreover, BOAD consistently reduces input tokens, confirming that task decomposition shortens context length.
in
Table 2: Token usage. BOAD lowers input token counts, thus shortening the model's input context length.
Metric
Setting
Verified
Live
SWE-agent SWE-agent + BOAD
0.92
1.49
Total tokens (M)
0.93 (+0.7%)
1.13 (-23.%%
SWE-agent BO-A
34.6k
49. 19.0k
Max input tokens
30.5k (-11.6%)
36. 7k (-25.0%)
5.3 ABLATION STUDIES AND ANALYSIS
Does prompt optimization explain the gains? One possible explanation for BOAD 's performance improvement is that it simply arises from better prompt optimization SWE-agent. To test this, we introduce a baseline that optimizes the SWE-agent prompt without adding sub-agents (w/o Sub-agent). We run 10 iterations: in each, a new SWE-agent prompt is generated by prompting Claude-4 with the template shown in A.1.5, evaluated on 12 issues (the same setting BOAD). The first iteration is initialized without history, and from the second onward, prompt generation is conditioned on the top five prompts from previous rounds, ranked by performance. Results in Table show that prompt optimization alone does not reach the performance of BOAD, indicating that the gains are not solely due to prompt tuning but from the discovery of effective sub-agents and orchestration.
'The SWE-bench-live leaderboard score was 17.7, 7.7, based on an earlier issue set from April 2025. 253.1 the sWe-hench-veriffed excluding the 12 issues used in the design set.
8

---

Published as a conference paper at ICLR 2026
Table Ablation studies and analysis, Each corresponds to research question. Results reported on SWE-bench Live using Seed-OSS OE---SSS------ 46B-Instruct unless otherwise specified.
Research Question
Configuration
SWE-Bench Live (%)
Does prompt optimization explain the gains?
w/o Sub-agent Sub-agent
16.3
20.0
Top-5 sub-agene sub-agents
13.7
Top-4 sub-agents
16.7
Do more sub-agents improve performance?
Top-3 sub-agents
Top-2 sub-agents
20.0
Top-1 sub-agent
16.3
Do we need to customize the orchestrator?
w/o customization customization
16.7
20.0
Is expanding the sub-agent archive needed?
w/o expansion
17.0
expansion
20.0
Top-3 subagents (success rate)
11.3
Is hindsight eredit assignment necessary?
Top-3 subagents (helpfuiness)
16.3
Top-2 subagents (success rate)
15.3
Top-2 subagents (helpfulness)
20.0
discovered sub-agents transferable to other models?
Claude Sonnet
+ Top-2 sub-agents (helpfulness)
16.3
Do more sub-agents improve performance? One might expect performance to improve as more sub-agents added, since each specialize. To test this, we vary the number of top-K sub-agents from to 5 based on the helpfulness score (Section 4.2) and evaluate on LIVE. Surprisingly, Table 3 shows that performance peaks with exactly two sub-agents, achieving 60/300 (20.0%). single sub-agent (49/300) fails to leverage specialization, while larger teams of three (49/300), four (50/300), or five (41/300) reduce performance due to communication and coordination overhead. These results suggest that small, focused teams strike the best balance, outperforming both minimal and overly large teams of sub-agents.
Do we need to customize the orchestrator? We next ask whether gains come solely from sub-agent discovery or the orchestrator must also adapt to team. We compare two prompting strategies: (i) generic prompt encouraging sub-agent calls (Appendix A.1.1), and (ii) customized prompt generated by Claude-4 that explicitly references the top two sub-agents (selected by helpfulness scores) and outlines plan for using them. Both settings use the same sub-agent customized prompt allows the orchestrator to reason about and plan calls to specific sub-agents. Results in Table show the customized orchestrator (w customization) achieves 60/300 (20.0%), versus 50/300 (16.7%) (w/o customization) for the generic one. This indicates that while sub-agents provide new capabilities, the orchestrator must also be specialized effectively coordinate them.
but only the
Is expanding the sub-agent archive needed? As discussed in Section 4.1, the initial archive may be limited, and adding new sub-agents during the design process could be necessary discover stronger ones. To test this, we compare orchestrator performance using (i) sub-agents from the initial archive (w/o expansion) and (ii) sub-agents selected the end of the design process (w expansion). Both settings use two sub-agents, consistent with our best configuration in Section 5.2, and the orchestrator generated as described Section 1. Results in Table 3 show that final sub-agents outperform those from the initial archive, highlighting the importance of expanding the archive over time.
Is hindsight credit assignment necessary? To address free-riding issues in sub-agent selection (Section 4)1,, use a helpfulness score to measure each sub-agent's contribution. To test its importance, we compare orchestrator performance when sub-agents are selected by (i) individual success versus (ii) helpfulness score. As shown in Table 3, helpfulness-bant based selection consis- tently outperforms success- ratcess-rate selection, indicating that hindsight credit assignment (Section 4.2) is essential for identifying useful sub-agents.
Are discovered sub-agents transferable to other models? Since BOAD optimizes sub sub-agent agents for specific model, ask whether the best sub-agents differ across models and whether effective

---

Published as a conference paper at ICLR 2026
sub-agenn sub-agents can transfer. To test this, apply the sub-agents from Section 5.2 SWE-agent+Clauce 3.7-Sonnet. As shown in Table 3, the discovered sub-agents do transfer to some extent, though the gains smaller than those achieved with L---OOSSS---B-tttie the model used for sub-agent optimization.
5.4 QUALITATIVE ANALYSIS OF SINGLE- VS MULTI-AGENT OUTCOMES
We manually inspected trajectories in which the single- and multi-agent systems produced different outcomes. Three recurring patterns emerged:
1. Over-editing (multi-agent advantage). Single agents frequently produced extremely long patches, including attempts to create new tests and edits outside the scope of the bug. Such patches inflate apply time and increase the chance of failing pass-to-pan to address the primary fault. In contrast, the multi-agent system tended to emit short, localized patches, highlighting the advantage of separating phases like localization and editing/testing
even if the agent is able
2. Multi-site fixes and coverage (multi-agent advantage). When the fix required edits at multiple call sites or modules, the single agent often either over-edited unrelated regions or missed one or more necessary locations. The hierarchical system mitigated both omission and extraneous edits by delegating to sub-agent for localization, which did thorough analysis of the repository before making any targeted modifications.
Error propagation from unvalidated sub-agent outputs (multi-agent failure mode). minority of cases, the multi-agent system failed while the single agent succeeded. Inspecting these outputs, we found that erroneous sub-agent outputs (e.g.. incomplete span identification, misinter- pretation of the issue) were accepted as ground truth by the orchestrator, leading subsequent steps astray. Because there is no intermediate validation or self-checking, the orchestrator has limited ability to recover from such upstream errors.
These observations align with our hypothesis: hierarchical delegation constrains edit scope and improves coverage of multi-site fixes, but introduces a new dependency the quality of sub-agent handoffss. Incorporating lightweight verification (e.g., span cross-checks, invariant tests, or dual-read localization) is a promising mitigation for the third failure mode.
DISCUSSION & CONCLUSION
We present BOAD, a framework that formulates hicrarchical multi-agent design as a sequential, online decision making problem to automatically discover multi-agent systems for long-horizon software engineering tasks. Our experiments show that automatically discovered sub-agents, when combined with customized orchestrator, outperform single-agent and manually designed multi-agent systems on both S SWE-BENCH VERIFIED and SWE-BENCH LIVE.
Limitations and Future Work: We find that discovered sub-agents transfer across models only partially and failure cases highlight error propagation when orchestrators unconditionally accept sub-agent outputs. Future work should explore evolution on large models, adaptive team sizing, verification, as well as extending the framework to domains beyond software engineering.
ACKNOWLEDGMENT
We thank members of the Improbable AI Lab at MIT and MIT-IBM Watson Al Lab for helpful discussions and feedback.
AUTHOR CONTRIBUTION
Iris Xu: Led the project, implemented and designed the core algorithm, conducted all the experi- ments, and wrote the experiment section.
Guangtao Zeng: Implemented the baselines, explored the preliminary design of the ideas this work, and wrote the related work section.
10

---

Published as a conference paper at ICLR 2026
Zexue He: Explored the preliminary design of the ideas of this work and helped revise the manuscript.
Charles Jin: Assisted and supported the implementation of our method.
Aldo Pareja: Assisted and supported writing.
Dan Gutfreund: Coordinated the compute and infrastructure.
Chuang Gan: Coordinated the compute and infrastructure.
Zhang- Wei Hong: Led the writing and ideation of the project, coordinated the team and resources, and explored the preliminary design of the ideas this work.
ETHICS STATEMENT
Coding agents hold strong promise for automating code generation and bug fixing, but they also carry risks of unintended harmful outputs. For instance, LLMM-bssed agent may produce commands that could compromise a system (e.g.. downloading unauthorized packages or deleting user files with rm -rf). To mitigate these risks in our study, all experiments were conducted within Docker containers, providing isolated and sandboxed environments that prevent harmful commands from impacting real user devices and substantially reducing the potential for actual harm. Our implementation also builds upon the SWE-Agent framework, which has been previously published and reviewed under established ethical standards. We carefully follow its curated protocols and licensing requirements.
Nevertheless, as with any Al-based coding agent framework, there remains the risk of deliberate misuse, for example, malicious prompting the system to generate harmful hacking code. While our contribution is centered on advancing the technical design of hicrarchical coding agents, emphasize that real-world deployment should be coupled with responsible auditing and oversight, so that potential misuse and unintended consequences can effectively mitigated.
REPRODUCIBILITY STATEMENT
For all open-source LLMs (e.g., on their official releases. Commercial LLMs and LLM--asee tools are accessed through their official APIs and reference implementations. We provide detailed implementation notes of BOAD, including the prompt design for meta-agents, sub-agents, and L.LM judges, Section 5. To further support reproducibility and foster future research, we will also release all of our code, used data, and prompts.
we rely
REFERENCES
[1] Alekh Agarwal, Haipeng Luo, Behnam Neyshabur, and Robert E Schapire. Corralling band of bandit algorithms. In Conference on Learning Theory, pp. 12-38. PMLR, 2017.
[2] Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl- Ong, Arnav Singhvi, Herumb Shandilya, Michael Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. Gepa: Reflective prompt evolution can outperform reinforcement learning, 2025. URL https: //arxiv. org abs
[3] David Aldous. Exchangeability and related topics. In École É'é de Probabilités de Saint- Flour XIII-1933, pp. 1-198. Springer, 2006.
[4] Anthropic. Claude 3.7 sonnet and claude code, 2025, URL UBLLttpss/www.atthrrg. com/news/ hawdee//wlladedeeeeeeee 3 3
[5] Anthropic. Claude 3.5 sonnet, 2025. URL httpss//www.rtthoopccccooo/e/eaa
claude-3-55 sonnet.
[6] Anthropic. Introducing claude 4, 2025. URL httpss//ww.antttrooocccooo claude-4.
11

---

Published as a conference paper at ICLR 2026
[7] Anthropic. Devstral, 2025. URL /ttaappss/mmtssrallaaaae///s///s
[8] Daman Arora, Athary Sonwane, Nalin Wadhwa, Abhav Mehrotra, Saiteja Utpala, Ramakrishna Bairi, Aditya Kanade, and Nagarajan Natarajan. Masai: Modular architecture for software- engineering agents. arXiv preprint arXiv:2406.11633 2024.
[9] Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal computing, 32(1):48-77, 2002.
[10] Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems, 13(4):341-379, 2003.
[11] Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13(4):341-37, 2003.
[12] James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. The journal of machine learning research, 13(1):281-305, 2012. 2012.
[13] Hans-Georg Beyer and Hans-Paul Schwefel. Evolution strategies comprehensive introduction. Natural compuling, 1(1):--52, 2002.
[14] Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, et al. Coder: Issue resolving with multi-agent and task graphs. arXiv preprint arXiv:2406. 01 304, 2024.
[15] Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje Karlsson, Jie Fu, and Yemin Shi. Autoagents: A framework for automatic agent generation. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pp. 22-30. ijcai.org, 2024. URL https: //ww://wwwww////////// ijcai.osoo org/ Sooccediing//22220
[16] Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In ICLR, 2024.
[17] Chris Dann, Claudio Gentile, and Aldo Pacchiano. Data-driven online model selection with regret guarantees. In International Conference on Artificial Intelligence and Statistics, pp. 1531-1539. PMLR, 2024.
[18] DeepSeek-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, Wang. Tao Yun. Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang. Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha,
12

---

Published as a conference paper at ICLR 2026
Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu. Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang. Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025. URL https: /ttpss///axiv...a/ aatpaaxiivvvvoo...a//aaa org/ abs/2412.... 19437.
[19] Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decomposition. In Proceedings of the 17th International Conference on Machine Learning (ICML), pp. 118-126. Morgan Kaufmann, 2000.
[20] Dylan J Foster, Akshay Krishnamurthy, and Haipeng Luo. Model selection for contextual bandits, Advances in Neural Information Processing Systems, 32, 2019.
[21] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint urXiv:2501.22945 2025.
[22] Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In The Twelfth International Conference on Learning Representations, 2024. htLs://:://
/ttpt::s///pppeeeiiiwee net / forum? i d= ZG3RaNI
[23] Shengran Hu, Cong Lu, and Jeff Clune, Automated design of agentic systems. arXiv preprint arXiv:2408.0888 2024.
[24] Yu Jiang, Ke Wang, Xin Wang, Yanyan Zhao, and Zhiqiang Zhang. Extracting concise bug- fixing patches from human-written patches in version control systems. In Proceedings of the 43rd International Conference Software Engineering (ICSE), pp. 1351-1362. IEEE/ACM, 2021.
[25] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310006770. 2023.
[26] Jaehun Kim, Shunyu Yao, Howard Chen, Karthik Narasimhan, et al. Promptbreeder: Self- referential self-improvement via prompt evolution. In International Conference on Learning Representations (ICLR), 2024.
[27] Levente Koesis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning. pp. 282-293. Springer, 2006.
[28] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. 2020.
[29] Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Llms-as Lims-as-judges: comprehensive survey on blsed evaluation methods. arXiv preprint arXiv:2412.05579 2024.
[30] Han Li, Yuling Shi, Shaoxin Lin, Xiaodong Gu, Heng Lian, Xin Wang, Yantao Jia, Tao Huang, and Qianxiang Wang. Swe-debate: Competitive multi-agent debate for software issue resolution. arXiv preprini arXiv:2507.223:: 223448..... 2025.
[31] Fernanda Madeiral, Thomas Durieux, Matias Martinez, and Martin Monperrus. Bears: An extensible java bug benchmark for automatic program repair studies. In Proceedings of the 26th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 468 478. IEEE, 2019.
[32] George A Miller, The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63(2):81, 1956.
[33] Allen Newell and Herbert A Simon. Computer science as empirical inquiry: Symbols and search. Communications the ACM, 1933):11--126, 1976.
13

---

Published as a conference paper at ICLR 2026
[34] Huy Nhat Phan, Tien N Nguyen, Phong X Nguyen, and Nghi DQ Bui. Hyperagent: Generalist software engineering agents to solve coding tasks at scale. arXiv preprint arXiv:2409.16299. 2024.
[35] Martin I. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[36] Melanie Sclar, Yejin Choi. Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design or: How learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.
2023.
[37] Herbert A Simon. The structure of ill structured problems. Artificial intelligence, 4(3-4): 181-201, 1973.
[38] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3022 2020.
[39] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2): 181-211, 1999.
[40] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2): 181-211, 1999.
[41] 5 Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang. Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang, Zilin Zhu, Bohan Zhang, Bosi Wen, Bowen Wu, Bowen Xu, Can Huang, Casey Zhao, Changpeng Cai, Chao Yu, Chen Li, Chendi Ge, Chenghua Huang, Chenhui Zhang. Chenxi Xu, Chenzheng Zhu, Chuang Li, Congfeng Yin, Daoyan Lin, Dayong Yang, Dazhi Jiang, Ding Ai, Erle Zhu, Fei Wang, Gengzheng Pan, Guo Wang, Hailong Sun, Haitao Li, Haiyang Li, Haiyi Hu, Hanyu Zhang, Hao Peng. Hao Tai, Haoke Zhang, Haoran Wang. Haoyu Yang. He Liu, He Zhao, Hongwei Liu, Hongxi Yan, Huan Liu, Huilong Chen, Ji Li, Jiajing Zhao, Jiamin Ren, Jian Jiao, Jiani Zhao, Jianyang Yan, Jiaqi Wang, Jiayi Gui, Jiayue Zhao, Jie Liu, Jijie Li, Jing Li, Jing Lu, Jingsen Wang. Jingwei Yuan, Jingxuan Li, Jingzhao Du, Jinhua Du, Jinxin Liu, Junkai Zhi, Junli Gao, Ke Wang, Lekang Yang, Liang Xu, Lin Fan, Lindong Wu, Lintao Ding, Lu Wang. Man Zhang, Minghao Li, Minghuan Xu, Mingming Zhao, Mingshu Zhai, Pengfan Du, Qian Dong, Shangde Lei, Shangqing Tu, Shangtong Yang. Shaoyou Lu, Shijie Li, Shuang Li, Shuang-Li, Shuxun Yang, Sibo Yi, Tianshu Yu, Wei Tian, Weihan Wang, Wenbo Yu. Weng Lam Tam, Wenjie Liang, Wentao Liu, Xiao Wang, Xiaohan Jia, Xiaotao Gu, Xiaoying Ling, Xin Wang, Xing Fan, Xingru Pan, Xinyuan Zhang, Xinze Zhang, Xiuqing Fu, Xunkai Zhang, Yabo Xu, Yandong Wu, Yida Lu, Yidong Wang, Yilin Zhou, Yiming Pan, Ying Zhang, Yingli Wang, Yingru Li, Yinpei Su, Yipeng Geng, Yitong Zhu, Yongkun Yang, Yuhang Li, Yuhao Wu, Yujiang Li, Yunan Liu, Yunqing Wang, Yuntao Li, Yuxuan Zhang, Zezhen Liu, Zhen Yang, Zhengda Zhou, Zhongpei Qiao, Zhuoer Feng, Zhuorui Liu, Zichen Zhang, Zihan Wang, Zijun Yao, Zikang Wang. Ziqiang Liu, Ziwei Chai, Zixuan Li, Zuodong Zhao, Wenguang Chen, Jidong Zhai, Bin Xu, Minlie Huang, Hongning Wang, Juanzi Li, Yuxiao Dong, and Jie Tang. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025. URL https: /ttps:///axvvv.. .org/
06471.
[42] ByteDance Seed Team. Seed-oss open-source models. https://githube SytteDanceSSeed/eeee oss, 2025.
[43] Qwen Team. Qwen3 technical report, 2025. URL https::/axxvv..zg// 09388.
[44] Yee Whye Teh, Michael Jordan, Matthew Beal, and David Blei. Hierarchical dirichlet processes, Journal of the american statistical association, 156661)):566-11555 2006.
14

---

Published as a conference paper at ICLR 2026
[45] Dong Wang, Yu Li, Ming Jiang. Lu Zhang, and Zhiqiang Chen. A systematic mapping study of bug reproduction and fault localization. Information and Software Technology, 167:107316, 2024.
[46] Mingqi Wu, Zhihao Zhang, Qiaole Dong, Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Huijie Lv, Ming Zhang, et al. Reasoning or memorization? unreliable results of reinforcement learning due data contamination. arXiv preprint arXiv:2507.10532, 2025.
[47] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying /407.
Ilm-based software engineering agents, 2024. URL 01489.
[48] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528-50652. 2024.
[49] Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Dongsheng Li, and Deqing Yang. Evoagent: Towards automatic multi-agent generation via evolutionary algorithms. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 6192-6217, 2025.
[50] Chen Bo Calvin Zhang, Zhang- Wei Hong, Aldo Pacchiano, and Pulkit Agrawal. Orso: Ac- celerating reward design via online reward selection and policy optimization. arXiv preprint arXiv:2410.13837, 2024.
[51] Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin godel machine: Open-ended evolution of self-improving agents. arXiv preprint arXiv:2505.2255,, 2025.
[52] Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762, 2024.
[53] Kexun Zhang, Weiran Yao, Zuxin Liu, Yihao Feng, Zhiwei Liu, Rithesh Murthy, Tian Lan, Lei Li, Renze Lou, Jiacheng Xu, et al. Diversity empowers intelligence: Integrating expertise of software engineering agents. arXiv preprint arXiv:2408.07000 2024.
[54] Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, et Swe-bench goes live! arXiv preprint arXiv:2505.224. 23419, 2025.
[55] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe Le, et al. Least-to- most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625. 2022.
[56] Han Zhou, Xingchen Wan, Ruoxi Sun, Hamid Palangi. Shariq Iqbal, Ivan Vulić, Anna Korhonen, and Sercan Ank. Multi-agent design: Optimizing agents with better prompts and topologies. arXiv preprint arXiv:2502.02533, 2025.
[57] Yuhang Zhou, Paiheng Xu, Xiaoyu Liu, Bang An, Wei Ai, and Furong Huang. Explore spurious correlations at the concept level in language models for text classification. arXiv preprint arXiv:2311.08644 2023.
[58] Terry Yue Zhuo Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.1887 15877, 2024.
15

---

Published as a conference paper at ICLR 2026
A EXPERIMENT DETAILS
A.1 PROMPT TEMPLATES
A.1.1 PROMPT TEMPLATE FOR REFINING SUR-AGENT DURING WARMUP STAGE
Prompt template for refining subagent during warmup stage
improving sebegert'e prorpts/config 4 on reeent run trajectories. The sussgent is used by an AI main agent to address cods issucs,
eng neening
systen,
CONTEXT
receive trajeciory summaries belcx, starting with
agent's "s trojectory, tollowed by
any subagent trajectories in call order,
Each sunnary shows what the agent did, what was observed, and how far progressed.
Analyze the subagent's performance and sugaest inprovenents to make it:
1. Koze siecoverable the mzin agent lshen apprepriate) Hore reliable in its betavior outpet
3.
ANALYSIS PRAMENORK
main agent discover use ousegent it should have? bid tha subagent behave as expected and return useful informaticn? Ner: there missed opportenities incfciciont rehaviors?
INEROVEMENT
socus on oze or more of taese areas:
1. =esocstringa= Make the subagen: easier for the main agent to ciscover and choose appropriately,
includa "[subagent]" the berinning docstring. Improve coscripticn b> cleaser and more helpful
2. CRITICAL:
"sontext' argunent itae only arqunent)
3. "vinstanco_templateee Bezter incorporation of context, clearer franing for each problen instance
that docstring
only
ageat,
+ insttance_templaat is visible only to the subzqent.
Taus, it ihe subagent was not calles, you should not esit instance_ccrppe complate. similarly, it the only the
trajectory, caacxx_deecription.
called, do edit doestring
PSINCIPLES
improvements Pewswive exiscing style and capabilities. only edit componenia
surcical,
broad
improvenen..
Focus sn siarity, disesverbilltty and roliability
Ensure gcnersiity. Avoid zeps- izsue-specitle assumptions.
calTICAL: HOT ANYTEING SEECIPIC TO THE PARTICOIAR
PROJECT, DONAIN.
CUIPUT FORMCC
Fizat, explain ceasoning abou: what issuea noticed with the previded Lea Jectory and what Laprovements
nakiae. Then, INNORTAANT.. Onlyy suggest adise when you identify a clear, speci problen, the antiow subagent <> is working well, use an empsy updates diet dintionary.
the YAME code block.
ouiputa:
r*then inprovements are neededive
Explaia your zeasoning hare.
"*yyal
docstring: "clegroved dinstring n
contwat_ dewereepiiiin "cipprooed concext argunent destre destription iption if neededs Instance "cipproved instance temolate if needads"
ROLES
Only include keys that incend to change,
Staci "updaleas" the Lop key. It thece ugdallia make, the value "updaaiea" should be an empty dietiorary
You may ugdate any combiration of the three fields rdorstring, context description
inat ance camplats), bat only include rield iC aweds improvemett.
explaracions
Keep each field poncise but complete
WRURESTICS
<>Disuseery
Strengthen docstring viInsutisiien context passed to rubagentir; Improve context dercription with clearer argunent expl snst on
invoke
USubagent behavier and (Incurrect subagent orajectoory return informt ion) Inpruve instance pestsnce_teepilatee better instructione output speciticacions
1 [TRAJRCTORRRS):
16

---

subegerts Published as a conference paper at ICLR 2026
A.1.2 META AGENT PROMPTS
Prompt for generating a subagent configuration
You aze an expert at desigring custon tools for ExB-agent, an aurchonous agent tast can resolve code 1esues in lerge regositories.
+
Invent susegent tool for SMr--ggent.
The subagent should enable the main agent to better perform its task of aut-onesically resalving code static repositories. issues in applicabi
4
besign
fell workflow, cntice step et the pipeline, such as:
code loselization
regroducl isaues and rurning s/teats
cade ecising/patching
code testing
The subegents created should OxY be focused on correctnesy of the Finel patch le-gy style,
camplexity cede
natter)
RFFICIRKY: Design concise,
subagents
tokene, verbose
explanaticas or rodundznz informstion that wastes tekcas.
you see cubagent tast icoks good but and bad token otticiony, you may generate similar with the furezion but better inp implerentation.
Ihe susagent takes SINGLS arquneas
stxing,
NOVEL! Think caretully about how to help tae nain agent pertern one ot its subtaske, atriciency
Example e subegents include local patca editor, or code tester,
sisuation).
In your reseoning, you must explicitly lest which steps the subagent supporte texemples: explore,
+ rsad/search, edit, run, velidate) and the expected outputs for each supported step. CRITICAL: Output exectly
the tool
key, which is the
The name of the toel should be simple and descristive.
The docstring for eaca tool should be compreheneive and descrime what the output contains, as well rapository after subarent is finished files will edited
the state
etc.).
The struzzere must be exactly as follows: Add reassning **"yan
MEY design susgaent....
tool_nama:
signature: "tocl_namc <conzeNs>*
decstring: doestring: "corprehensive description this
and the
repositery on completion. Starts "[sabaqent]:"
arguiente:
name: concext
type:
desesiption "dstailed description what context pacanieer should contain"
requized: tree
cubagenz: true
Sampls output:
It be usuful to have patch editos subagent. This would well with previous aubagenta and help africiantly the
+
""yanl
patch_edi:os:
sienature: "patch_editar <conzox:>*
doustring: "[subegent]] Fixes specitic part code that escurs. Outputs changes made with reasoning. After calling, the cocrect changes are already implemented reposicary."
arqunents:
context.
type:
description: "A string containing specific file path to make edits in,
where edits
newd to be nade, compeehensive description of the issue with the cude (do not axsune the subagent has information about the zopesitozy prehiem statement), and what to edit."
required:
sabageni: Lrow
[PRREVOOUR, 1 I'ERATION I"TRATION_PPRAAA PREBACK))
Prompt for generating subagent templates
You are an expert al creating Lng SME agent subagent confiigratiin flles for urogaainn code code-patoo patching Lasks by in large GicHub repositaries.
sabagens, you need generate the systan Lnstanok_tenplatee Given ipsiription used the subagent contiguration. parts that
INFCRTANT PORMACTING RULES:
that decails your thinking prosess for creating the templates. outpuz yanl with besh complatas.
Dse KINDAL spacing avoid excessive blank lines
Use only SIXGLE blank lirea between sectioas (naver double or criple spacing)
unnecessary whitespace YAML literal bluck synlex with and I fawe exanple below) be NOT use cuoted strings use literal blocks to avoid quates in cucput
Keep templetes conpect reedsble
17

---

Published as a conference paper at ICLR 2026
Replace oNLY cext in II MisE sext specicic to the subagent, Do NOT MODIFY any other parte. EXACTLY other including call cunctions (0.4.
cutput format:
[8cassoing
""yanl
Wisten_sampltte
You are helpful [xels] ascistant that can interaet with computee to [nair. tass].
<IMFORIANT>
instead, you should explore the tile systen to rind the tile betoro working en it. working dizectozy.
user provides a past, you should Hoz asauns relativa
</DMORTAN'5>
sccess tollowing functione:
(Jeammnn_ddeaa))
Provida any zeasoning fuaction,
reply in the fullosing fornat with auffix: function sall hore.
cfunction exampooeeeexpee..o..eooo saample_fenctioon renction_rance>
pracanelecmexaage paw.pppllpppammeee.e..... perrametes__vvae Dovalu 10tpa///ataa
cparamsterexxanol srcaanteraxaaoi__paaaatttt
for secord porancter
Chat can span muitiple lines
</parameter>
</funct-o07
(You must use the exact text functions* and "paranetars" for each function ard arqunent, respestively, perranttereuumamii ppppaalleeaaaaaaarrra.
beninder:
Function calls NOST follow the specified fornat, stazt with sfunctions and end with s/function> specified
Required parameters funet lon at Line
CITTICAL: Only call
Alwaya provide zeasoning for your Sunction call in natural language BEFORE the function call Inot aftwr)
</DPORTANC>>
der_ciettiptt inns
</p= descriptioon
CRITICAL: Use the subni L_ dbagen, tunol lun to provide the ausults when you ase Fiaished with your <
responsible your speci essigned pe_descciitton only your Lask. Your goall la to complete your task in the NENTHAL Nomee of steps. Resolve the lasce fast and cell >
atterpt resolve entiiw
subrit subnit_aabepe subagent
possible.
Instance template:
Your
(Provide led, step-hy-step instructionn your essigned subagent teilored your ple role. The inatruct ions must ze seferencw subagent ta Sunct iar..
[It context argunent is provides, you MUST include its contents by incerting "[scontext i!" here and aining what the parameter i8.)
<
**RRITIGAL: STAY IN YOUR LAME+
You are DXLY resgonsible le for your apeci ansigned task issue
solving the other subagent Pocus ExcrasTvELy on your axe axsigred tase and nothing else
zesponsible
reeponsible
CRITICAL: call EXCTLY cne tunction in your output!
CRRTIIAL: you tirished, innediately call submit produce addit octput.
call other tools
Focus exclusively on your ansigned task and strictly follow these rustioos bo not attempt outside specidic
address unrelated
the ER you tinished your specitic task provide complete sumnary of your findings or changes.
gubmis_puzagent
Your thinking should be thorough and it's tine it it's very long.
Roles for gencrating templstes:
1. The eysten eysten_terplaa template should clearly detine the subagent's 's role and capabilities based on the
4 available toels
instance_temp
provide iastructions
3. Both terp_ates should maintair consistent tornstting with the base template
4. Ensura the templates saccurags thorough anclysis and cleas cacumentation literai block eyptax: wysten_terplase templates.
cettanso_tenplatee
use quoted strings
7. You should copy the civen systen template exactly other than the first sentence.
9. Kodity the systom tomplete in the spots with [].
[[EREVIOUS, 1.EERIOOS 1.TEMIIIN
F3603MC3] [TEMMII
18

---

and Published as a conference paper at ICLR 2026
A.1.3 CUSTOM ORCHESTRATOR PLAN PROMPT
Prompt template for generating custom orchestrator plan given set of subagents
You asи nastes workflow asclii.ed. autonated sofleare engineering. Your job ia design inaovative, strategic workflows that maxinize the atfectiveness of availabie tcols.
CONTEXT:
AI assistant that solves assistant seceives problem descriptian (Like bug report, feacure request, or code sasue) and needs to systenatically work through the codebaze be understard, fix, and val validdae the solucion. The assistant has acceas to apecialized "aubagenta" each designed for specific vorkfiow.
You"sw designing vorkflows
xupositories,
of the
You will be given toolkit of apecialiaed "aubagenta" each with unique capabilities. Your challeage
b is to:
adiresses the coding 2. =eIncagracers subagenta atracegloally share they add aost value your workflov
sxDexignes comprshenaive problen sulving
syatenatiooall
3. +*optinizove the sequeace and wozding of the plan to ninimiza the number of szops that the MI
( assisian: Lakes while cemaining ive
Think senior engineer designing a solut strategy. Corsider: What ase the key phases ceeded to solve this lype of cuding prublen??
Shich subagenca would be nost valuable for specific phases?
you cmmii subagent
problen-sol solvingg phases?
the most logical progeession integrate sabageni iapul and uukpat aolve issum?
Bov can you ucillze aubaceata to mininize language nodel token usage and number of aceps?
INFUT
The las:lable subagents will be provided inline betveen the fo. lowing tags: Cavail abll ablle
(Iaabagesss
<favai subagents>
The consens savei aubeeletteeeeeeeettt lista each subagent wish its nene short doos
ag faumnacy/desessa (iginerrigginn...). Treal at as the authoritatiio souice for Lool names and parposes.
Dutpat ORLY your strategic plar as plain text Ino YAM,, no code bences, no aeaders). Each phase MUST start with number and peried, e-g. "), ...*,
Por aubegent phases,
the exact form: tiee the Kharee subagent to
phases, describe appl fed to any problen.
zlearly
mentioning subagente, encuring can
Be creative and stralegic design workflous that combine different approsciess effectively total, but nake
parposetul
vel_-reasoned
steps Atter you have solved the issue, deleto any test tiles or tamporary files you created. Use the suanit tool to subnit the changes to the regonstory.
mention Funstion-call formats
details
EXANPEE losttrttt only; adapt to the given input)
Kavai lavailable lalllll sobagent
locaiieoc Identity tiles
regions relevant iseue,
reproducer Repromuce feiling nanz: todo_tester Run costs/commando to verify the fix and regressions.
and capture comands/ostpaas..
c/availab.e__ubbe ooubegentt_
Expacted octput
(plair. sex: only):
1. Use the seruo_loociiee sabagent to nap tae problen space and identity all potentially attected files and cose regions.
+
problen discription exanine identifsed files understend requirements.
3. Use the error reproducer subagent to create a reproducible test case and capture the exect failure
+ coaditions.
implement
based analysis, tocusing specitic tiles
Lientified.
5. Uc2 the sose_tester subsgent to validate the fix against the original failure case and zun
+ regression cests.
selved
delets
files zemperazy siles crezted.
Usa the submit tcol to cubnis the changes repositozy.
NoN, based on the provided subagcazs, produce CNLY the aumbered plan as plain toxt:
A.1.4 PROMPT TEMPLATE FOR CHECKING IF SUB-AGENT WAS HELPFUL A GIVEN INSTANCE
Prompt template for checking if a subagent was helpful in a given instance
CONTEXI:
Theae trajectories show a software angineering agent trying to fix a bug or impl ement Seature. agent can
Lools including subagente (specialized assistants) to help solve the problen.
TRAJECTORTES: [
19

---

Published as a conference paper at ICLR 2026
TOOL TO ANALY23: ((TOOL)AME)
Your Lask is Lo determine if the subagent "[[TOCL_ NAME) ) was helpful in this set of trajecteries.
tool in considered helpful if:
1. was called/inwoke invoked by the main agent the main agent trajectory
2. It provided useful information, analysis, oz inaighta that contributed to solving the + prohlem
3. The main
made progresa ofter using this tool (e.g., identified the issue, made + code changes, validated results, etc.) completed
as intended [followed proper anslysis process, just got lucky results)
Leok fes positive evidence such
The subagent being called with appropriate perameters
The subagent providing insights that led aada changes problam understarding
The main agent referencing or building upen the subagent's output
subagent output baing used subsequent reascning actions
Look for segative evidence such ast
The subagent net being called the main agent, called incorrectly
The subagent providing irrelevent or incorrect information that was not later used agent closer resalving
The aubagent responae
valid but did
+ the problem.
to execute properly or had many arvers during the rur. The aubagent output appeared correct, but its trajectory die not actually achleve those results (e.g., claimed to test code but just reported all tests passed) and
subagent
The agent had call the subagent
again get the proper results. The subagent trajestory was unnezastarily long or verboge, taking many stepa to
corplete
The main agent trajectory became inefficient dua to excessive subagent colls or overly verbose subagens reaponses
The subagent resulta did not actually help the main agent make progress in resolve the issue. If a subagent dié not contribute to preducing the correct patch, e.g. only improved performance, style, documentation, this is NOT helpful.
Respond
texactly);
"'yaml
helpful: true/talse
reasoning:
Brief explanation of why the tool was or wasn't helpful, including specifis evidence from the trajectorias
Always use the block scaler "I" for "reasening" and indent its text by two spaces. Only zeepend with the YAML block; no additional text before or after.
A.1.5 OR PROMPT
Prompt template for generating orchestrator prompt under orchestrator only settings
mostor wocktlow aschicess autometed sоttнarо enginecring. job sesign ettective workftews that solve eoding problens afficiently.
b
CONTEXT:
You denigning workflows
AI assistant that solves problems
repositories, assistant zeceives problen description (like bug resort, feacure request, oz code sesuc) and needs to work through the codcbase to understand, impicneat chances, and volidate Lhe sulucion.
Your g0al create vockflows are both effective at solving probioms and officiene thoir sag
execution. Pocua on des designing ateategic appcoacliez that lead to succesaful problem cesolution.
CONTEXT FOR USAGE: Your generated plan
into this agent tamplate context:
uploaded python code repository directory (1w))) Conaider the following dascripticn:
pes_deacrsptioo ('proble s/pr_dwaer lonk
20

---

Published as a conference paper at ICLR 2026
Can you help ne implement the necessary changes to the reporitory so that the requirements specitied in "ps_diseription
g've sizeady srr_dcscription>
any the riles have modity testing logic or any
means you the tests in any say! Your cass is to nake the minimal charges to non-test tiles in the ([wosxing_diz); directory to anzure the -pr_description is satisfled. when solving the task, by breaking needed underatand the problen, problen into zelevant cods, implement changea, verify the solution.
dascribed
s*tisst create
think systensticoally
+ Fellow this process:
((plan) <-- YOUR FLAM S0BS HERE
MUST follow
exactly.
AVAILABLE TOOLS:
command searching, rumning exploring codabase precise steing roplacement capabilitios
sct_rep_ase__dite view, create, submit: Sabmic the final solucion
LSAINING HISTORY:
<panptee__mppasee
wttpcceeeemmpatss_ssrarr s/sampled_templalee
histories: templates provides acove, idcatity what made highest-scoriat appreaches
succeasful Cul and what caused fallures. Look for patteria in tool usage, alep afficiency, and problen-sclvine stratories. If history exlats, design broakthzough approactas that challengs iunal soltware engineering surkflovs.
KHAT тO CUTPUT:
plan that AI agent execule syssenat inaliclly Create cleasly apecify usage ("Uae bash
"Use ats_replaoo_ditor to....) and
expected cutconas
besign for Pychon irgusitories and Dh based problen salving
Focua mininal, targsted changes rather than broad exploration
steps ()., Final sieg must use subrit Loul
logicel flow
the solatian Make each step actionable and specific enouch for precise execution plain text (no fornstting, headers, explanation iors)
Output GNLY the numbered
A.2 SUB-AGENTS
A.2.1 BOAD TOP DISCOVERED SUB-AGENT CONFIGURATIONS FOR SEED-OSS-360 INSTRUCT
issue_analyzer configuration
docstring:
[subagent] Analyzes and structures issue descriptions, buz reports, or feature requests to extract key intormation for systamatic resolucion planning. Best used at the beginning et iseue resolution to technical details, struztered anslysis iaciading problen cumnary, atfected components, reproduction criteria, success concitions, and recommended invescigation approsch, Reposstory state uncharged only snalyzes the pzovided issue context.
understand requirements,
investication
argunent:
context (stringl [rqquirad] : The complete issue description, bug report, feature request, oz problen + statzment text that needs to be analyzed and structursd. inslude the full oriciasl text to casure technical aystanatic anslysis.
implementation
instance template:
provided description systematic resolution planning, on the provises contexts [Icontext]]
Analyze
information
Follox these steps:
Lype (bug repoci, requeat, 4 oz failure descriptisaz Affected functionaiity oz companents
description
infosnations
ethancement, stc.)
acatement symotens Expected
behavioz Brvor nessages
2. Identify Lechnical detaila: Fils petha, funclioo ion namsa, code suierenees mencioned Stack tracas
information anvironmet
confiquration
issu25
3. Briucture seproduct ion critaria: Stage to reproduce the issue Requized conditions os setup Inpet daca tast cases reeded Expacted fallure modes or syaptons Clear
Befine
resolution Expected belavior
Verification ion nethods os cesta needed Bestostance or quality requirenenta,, Recemend invessiastion appreach: Priority creas to exanins first Suquesred deburgira or analysis methods
Related compenntts that be affected Pateat root cause Categorized techntcal detaila Provide scructured analysis including: Concise problen
soproduction steps Heasurable succass criteria Syscematic investiqation plan Risk assesament and impact analysis
CATTICAL: SIAS IN YOU LAM3<r- You are ONLY responsible for your specific assigrad task- You are NOT responsible for subegent
responeible solving
on your agsigred Lask nothing CRITICAL: Call EXACTLY one faresion in your output!-
21

---

Published as a conference paper at ICLR 2026
CRITISAL;
When you are tinisaed, imeolatell call subnit, subagent, Do not call sny other tools or additionsi outpuz. exclusively parts
assigned task strictly follow these
instruct-cas. be not attemp: address
pertorm
your
specitis subagent role.
Use the vubnit_aubaqon, tool after you are rinished with specific task to provida cleaz and rindings changts.
thinking
thozough and so
it's vasy long.
code navigaaott configuration
docatring:
[suzagent] Exploses and maps rolevant codo structure large rspesitories. Outputs structured intornation about key tiles. runctions, clasees, and their relationships, Repository state unchanged y reads and aralyses code without nodi fi fontone.
argunent:
context (stringl [required] : string containiag the issuc description, ceror messares, stack traces, + oz specitic code elements to investigate. Saould include relevant sile paths, ruaction nanes, keywords that night looate the problenstic
instance template:
Your tack: Explore and nap the relevant code strecture in the repository based on the provided context:
4 [Icontext]]
these steps:
1. Parse the provided context to idencify key elements to inveetigate (Sile paths, function names,
+
class cancs, ercor mescages, etc.)
Navigate through the repesitory structure Loeate relevant tiles and directories.
porposes Important functions and stair signatures Classes and their methods/tttrbbree Module dapendencies and imports Code relationshipe and call hierarchies relationships Provide scructured inforaaion. including:
Examine analyie
idencified coda clements including: Kay files
+
locations
roles cosebase
Function/cles actinitione and their resgonsibilitien Depensencies between modcles ccomppeenessseent Code patzerns and arcaitectural insights Relevant code snippets that relate to the context finist your summary your findiaga.**
+
**URITICAL:
innediatoly call submit_subagent
comprchmeeiv
CRETICALL; SIAY IR YOUR LANE+ You are ONLY responsihis for your specific assicned task You are NOz rorponsible for solving the entire issue You are NO7 rerponsizle for other susagent tasks EXCLUSIVELY
+
address unrelated Use submit__ubbgget to provide clear and complete summary at your tindings when finiched.
parts the partorn outside your specific subagant rele
A.2.2 ALL BOAD DISCOVERED SUB-AGENTS
Here we report all sub-agents discovered over 100 iterations of BOAD. For each sub-agent, we report the iteration number where it was first proposed (Generated Iteration), the number of times was selected during optimization (n), and its final hindsight helpfulness rate.
Sub-agent
Generated Iteration
Helpfulness
code navigator
1140
0.933
test runner code fixer fix_validator
120
0.625
36
0.361
0.333
issue_rerr
564
0.768
dependency resolveer
24
0.125
test rest.analyzer
6
24
0.083
issue_anat analyzer precision editorr
1116
0.982
120
0.642
multi_file coordinator
10
24
0.042
code deeecctive config manager
11
60
0.500
14
0.000 12
git_resolver
0.000
error_debugger
32
12
0.000
api_analyzer
36
24
0).208
performance analyzer
46
36
0.250
compatibility checker
48
36
0.333
spec analyzer
0.361
data_flow annalyzer
60
96
0.562
refactor.architect
96
24
0.042
Table 4: Sub-agent statistics sorted by exp.num after 100 iterations of BOAD
22

---

Published as a conference paper at ICLR 2026
A.2.3 ALL EVOLUTION DISCOVERED SUB-AGENTS
Here we report all bundles discovered over 20 iterations of BOAD (one for each iteration). For each sub-agent, we report its iteration number, the generated sub-agents, and the bundle's s success rate. Note that sub-agents with the same name across different iterations may have different system/instance templates because sub-agents not reused. On evaluation, we choose the bundle with the highest success rate (latest bundle if tied).
Sub-agents focused on problem analysis file localization- such such issue analyzer (0.968 average helpfulness), code wareeenvviiee navigator (0.917), and issue reproduce: (0.817) consistently appear useful. Our observation is that these agents provide value independently of how later stages unfold: even code editing or testing fails, identifying the correct files and clarifying the underlying issue almost always beneficial.
In contrast, sub-agents whose usefulness depends on earlier steps tend to show lower average helpfulness. For example, test_ana lazenll (0.167) is only useful after the system has already identified the faulty files and produced a candidate patch; if either prerequisite fails, it cannot contribute, which naturally lowers its average helpfulness.
Specialized sub-agents such as dependency www.ddennyrreooleerree rreooveer (0.250) and conf i g mannager (0.000) also have low average helpfulness, largely because dependency or configuration issues appear in only a small fraction of tasks.
Finally, once core analysis completed and the faulty files are correctly located, the orchestrator can often complete the remaining code-editing steps on its own. This explains why code-editing sub-agents like code_ fi xer not exhibit high average helpfulness.
Iteration
Sub-agents
Success Rate
issue localiec,,,,,r, issue_ reproducer, patch_editor fix fix_validator, fix fx_planner, code_explorer issue localizer, test runner, code patcher repo mapper, code searcher, fix validator code_editor, issue reproducer, issue localizer fix_validator, coce.locator, problem_analyzer test_runner, rode_editor codc_analyzer code locator, fix_validator, issuc dsscereprodueer code addeee repo analannay jest_runnd code locator, issue reproducer, solution, _paanner
0.333
2
0.500
3
0,417
4
0,500
5
0,182
6
0,583
7
0.583
8
0.417
9
0.250
10
0.417
11
code_patce patcher test_ validato,, repo.exy expoexxlooer issue reproduer,, code locator, locatoo fix. implmmeeeeterr
0.417
0.364
13
fix validator, code aanalanay code_editor issue reproducer, code locator, bug analyzer test_validator, patch_editor, repo_explorer
0,500
14
0,417
15
0,500
16
code_locator, issue repurrdur,, hug_analyzer
0.455
17
test_ test_validator, patch_edite _atch_ codebase eepllrer
0.583
18
issue reproducer, bug localizer, hug analyzer code patcher, fix validator, issue analyzer code locator, reproducer, code editor
0.250
19
0.417
20
0.417
Table 5: Sub-agents and success rates per iteration for the evolutionary baseline.
A.2.4 MANUALLY DESIGNED SUB-AGENT CONFIGURATIONS
issue localizer configuration
docstring:
subagent that localizes the issue in the zepository. Takes a context string specifying the boaef dexcriptioon of the ixsue. Gutguta briwf report about which Files and lines relevant to iasue.
argunent:
context tstring) [requiced] A steing containing the brief deser Sption of the isxue.
template:
Issuee descripl iption: [consextt]
23

---

Published as a conference paper at ICLR 2026
Pleaze identity shich ciles and specitic lines tunctions are nest relevant this issue, Outgut
paths
Sins numbers or tunction names
à oné-sentence explanation for way each location is zelevant
Keep repozt
on helping agenta wosk correct paccs
4 sapository.
rrrrruuerr configuration
doestring: aubagent that crea:es and exacutes test scripts to varity zoported errora. Outputa the - rasult of the tasta and locations of test files created.
argunent: context fatrina: [required]
steing containing arzor details, paths nunbers of code relevant to the ezzez, and expected va actual behavior.
instance template:
Erzor conzext:
[[sontext]]
and Create cemporasy files (prefixed with 'top_")
reproduction
verity this chould:
Includa only what's 's neaded to zapzoduce the arrer
Rapart wbether the crror reprocuens exactly descrized
any
behavior
Cutgut shart, clear report that nen Jons:
Besult of the tests
Socations of files created
code editor configuration
docstring:
implements code zepesitorv. asocific
code changes mado, file pathe numbers edited,
to varity the
argunent:
context (atring) [required] A string containing the code snippet to modily and file (s)
changes appliied.
instance template:
Context for eode changes:
implenen: specified changas sapository.
Identify the relevant filea and code sections
precise edita
spesification
Maintain onde quality consiacency Outpat a shoes, cleas zeport thas mentions:
Pile paths/1 ine numbers edited verify Fix
should
code_tester tester configuration
docatring: eubagent Ouiipta seaull Lhe Lesta,
verify works correct ly,
argunent:
wont (string! (required] striing containing the apeci fil code charges made, fill patha, original
instance template: Codle changes made::
Hoontext
Please Leat the codle aleue the wdiie Lo verily the fix works coctsctly. You should:
existing files if available (prefixed with "eagall
needed
Test the speciifie fig funct lonal iny that was changed
beternine whecher not the original weror is Fixed, Dutpat shert,
report shat mentione:
bests run and recults the testa Whether the originel werus is fixed, and 12 any new Rerors werw ini roduredd
24

---

Published as a conference paper at ICLR 2026
A.3 ADDITIONAL EXPERIMENTAL RESULTS
A.3.1 EFFECT OF DESIGN-SET SIZE
We evaluated whether the size of the design set affects the performance of the discovered sub-agents and found no meaningful impact. We sample unique problems from SWE-Bench- BeBech--rrrfiied and sample one problem from each to make a small design set of 6 problems, and sample two problems from each to make a large design set 24 problems, Across different design-set sizes, the resulting sub-agents achieve similar performance shown in Table 6.
Design Set Size
Resolution Rate
6
21 %
12
20%
24
19%
Table 6: Performance of discovered sub-agents across different design-set sizes.
A.4 ADDITIONAL EXPERIMENTAL SETUP DETAILS
A.4.1 IMPLEMENTATION DETAILS FOR EVOLUTIONARY BASELINE
For comparison with BOAD, implemented an evolutionary multi-agent design baseline where the orchestrator and three sub-agent prompts are bundled together and treated as a single evolutionary individual. We use three sub-agents paired with an orchestrator because BOAD chooses three sub- agents when optimizing the sub-agents in the experiments (Section 5.2). At each iteration of the evolution, the system generates three new sub-agents sequentially, given the previous round sub-agent configurations and their measured helpfulness and success rates (using the same prompts as BOAD, provided in 1.1.)). Next, both the sub-agent warm-up and the orchestrator construction follow the same procedures used in BOAD. The generated orchestrator and sub-agents are then run the same design set to get the helpfulness and success rate for the next iteration.
A.4.2 API COST COMPARISON WITH BOAD
The evolutionary baseline also requires Claude calls both propose new orchestrator/sub-agen prompts and to perform LLM-as-judge scoring. Under the same setup, each evolution iteration costs $2.33, whereas each BOAD iteration costs $0.96, indicating that evolution is substantially more expensive to Evolution incurs higher cost because it mutates the bundle of orchestrator and sub-agents at each iteration.
B THE USAGE OF LARGE LANGUAGE MODELS (LLMS)
In our work, we used large language models (LLMs) for two purposes: (1) as general writing assistant check the grammar of the manuscript for readability, and (2) as the model component of our proposed system BOAD, where LLMs function as the evolution engine, meta- evaluation judges in experiments with mainstream coding agents. All research ideas, methodological contributions, and conclusions are solely those of the authors, who take full responsibility for the content of this work.
and
25


Reviews:
[{"review_id": "yIMP1pF3w3", "reviewer": "Reviewer_5qkT", "summary": "Previous work in agentic frameworks for code generation have shown a worrying trend -- performance drops significantly from in-distribution benchmarks (SWE-Bench Verified) to out-of-distribution benchmarks (SWE-Bench-Live). The authors posit that one of the root causes for this is the long-horizon and complex nature of SWE-tasks, which makes it hard for single-agent frameworks to delegate and solve tasks. Furthermore, current multi-agent systems rely on expensive evolutionary optimization algorithms, which is impractical in the current domain. \n\nThe authors hypothesize that a multi-armed bandits (MAB) presents an efficient framework to build and optimize such multi-agent systems to solve such tasks. Specifically, they propose BOAD, which treats hierarchical multi-agent systems as a sequential decision-making process with the reward signal for evaluating each sub-agent (each arm in the MAB) coming from an LLM-as-a-judge.\n\nBOAD achieves impressive performance on SWE-Bench Verified and SWE-Bench-Live.", "questions": "Look at weaknesses.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is very well written and I thank the authors for explicitly stating the MAB formulation in the context of the code generation task.\n- In terms of novelty, I was pretty surprised but a multi-arm bandit optimizer hasn't been tried before for software engineering tasks.", "weaknesses": "**Methodology:**\n* **Training using 12 problems**: My understanding is that the SOTA accuracy number was reached after including 12 problems from SWE-Bench-Verified in the design set for BOAD. This raises two concerns:\n\t* Why 12 problems specifically? Does increasing or decreasing the set of design problems drastically effect final performance?\n\t* None of the other baselines are automatically tuning the multi-agent system. I understand that an evolutionary agent here might be less efficient than BOAD, but including a result for how BOAD fairs against an evolutionary agent would undoubtedly present a useful point of comparison to understand the tradeoff.\n\n* **Efficiency Experiment:** One of the motivating points of using BOAD (from the introduction) is that it is hypothesized to be more efficient than evolutionary multi-agent frameworks at the same task. However, the Token Analysis experiment is not enough to justify the efficiency for two reasons:\n\t* There isn't a direct comparison against an evolutionary multi-agent framework.\n\t* Efficiency can be achieved by either reducing token usage OR by reducing the underlying model size:\n\t\t* Specifically, In terms of floating point operations, querying a smaller model more times is more efficient than querying a larger model less times.\n\t\t* For example (be advised: this is a rough analysis without access to underlying data):\n\t\t\t* Assuming equivalence in all other aspects, a `30B` model can generate `20%` more tokens than a similar `36B` model yet have the same total cost.\n\t\t\t* Under this logic, the OpenHands single-agent that uses `Qwen3-coder-30B` and achieves a 51.6% resolution rate for SWE-bench Verified in is actually *more efficient*  than SWE-Agent+BOAD using the `OSS-36B` model which achieves a 53.2% resolution rate.\n\t\t* My recommendation:\n\t\t\t* Getting access to TFLOP data is extremely hard. Instead, most works use normalized cost, which is defined as the fractional cost of querying a smaller model per token compared to the cost of querying the largest model. e.g. if cost for 36B model is `1` per token, then cost for a 30B model will be `0.8334`.\n\t\t\t* Then, such methods will show an efficiency-performance tradeoff curve.\n\t\t\t* The key insight is to demonstrate that the efficient-agent algorithm is at the pareto froont of the tradeoff curve for different configurations.\n\t\t\t* Look at Figure 3 of this paper [https://arxiv.org/abs/2504.07247](https://arxiv.org/abs/2504.07247) for an example of how this is computed and analyzed. This might be a good paper for the related works as well.\n\n\n**Overall:** I'm slightly leaning towards rejecting the paper. The results are pretty impressive and I generally haven't seen a similar mutli-agent system optimized with bandits before, but there are some issues with the experimental setup that need to be better understood before proceeding. I'm happy to discuss these points with the authors during the discussion period.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Qb3rC6zkg", "reviewer": "Reviewer_Mz7z", "summary": "This paper proposes BOAD, a method for discovering and selecting subagents to use as tools for an orchestrator agent in software engineering tasks. It considers subagents in a set as arms of a multi-armed bandit and assign credits with both the test-based final outcome reward and LLM-as-a-judge process reward. After running this bandit optimization on 12 issues from SWE-Bench Verified, the discovered multi-agent system can perform well on both SWE-Bench Verified, and SWE-Bench Live, which doe not have repository-level overlap with the problems in SWE-Bench Verified.", "questions": "1. Any plans to compare against manual optimization conducted by human engineers in the loop?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Interesting formulation of the multi-agent system discovery problem as a multi-armed bandit. This creates balance between exploration and exploitation while selecting and evolving subagents.\n\n2. Great performance on SWE-Bench Live demonstrates the effectiveness of the method.\n\n3. Comprehensive ablation showing the effectiveness of having the subagents, customizing the orchestrator, and using hindsight helpfulness for credit assignment.", "weaknesses": "1. Lack of details about the actual optimization process. How many agents are there in the final set of subagents? How many of the top agents are from the expanded set or the initial set of subagents? What are the final top agents selected? Qualitatively why are they better than other subagents? Readers need these details to get a better idea of the final discovered system.\n\n2. Single run evaluation is insufficient. The non-determinism of LLM agents results on lots of randomness in every agent run, which impacts the optimization process. Is BOAD always effective as reported or will its performance fluctuate across multiple runs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vEEWQcsAmZ", "reviewer": "Reviewer_rVvm", "summary": "This paper searches for prompts of sub-agents for software engineering. The resulting agent consists of a set of sub-agents and an orchestrator (all parameterized by prompts). During inference, the agent will ask the orchestrator to call sub-agents sequentially to solve a problem. To search for the set of sub-agents, it maintains a pool of candidate sub-agents, picks and evaluates a subset of candidates each time, and updates the sub-agents' information according to the evaluation results afterward. The paper uses UCB to balance exploration and exploitation during search. It can also add a new sub-agent each round to improve diversity. LLM-judge is used to assign credits to each sub-agent as well. The searched agent is evaluated on two popular benchmarks and demonstrated strong performance.", "questions": "* Are there results of the naive evolution search?\n* Can you provide more experimental details such as the variance of evaluation, the cost for each method, and meta-prompts for each method? \n* Can you share the final best-performance prompts of the searched sub-agents?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The method is intuitive and interesting, although it largely relies on the abilities of LLMs for judging and proposing sub-agents. We do need various ways to balance exploration and exploitation. The searched agent demonstrates strong performance on popular benchmarks as well. \n\nThe paper is generally well-written and easy to understand.", "weaknesses": "* Missing naive baseline, such as evolution search. One can treat all prompts of sub-agents and/or orchestrators as parameters and use LLMs + evolution search to optimize them. There are various prior works that balance exploration and exploitation for naive LLM tree search as well. The authors discussed this baseline in the method section, claiming it is prohibitively expensive with no experimental results. \n* I'm not sure if the comparison with baselines is fair, missing experimental details. It would be great to include the costs of calling Claude-4 (which model?) for each method. I am not sure if the method is stochastic or deterministic either. If not deterministic (hard to imagine LLM to be deterministic even with temperature=0), how noisy is the evaluation? \n* The meta-prompts (e.g., for proposing and refining sub-agents) look long and detailed. Can the method discover novel sub-agents that are not expected, given the meta-prompts? \n* It would be great to include the searched prompts for various methods as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6N5N9MGdsC", "reviewer": "Reviewer_xwJx", "summary": "The paper addresses the problem of automated development of SWE-Agents. They consider a multi-agent setup structured such that an orchestrator agent calls multiple sub-agents. The main concern of the paper is optimizing the start-up prompts for the orchestrator and subagents. They treat this as a bandit problem, maintaining and growing a set of sub-agents over time. Each sub-agent is scored using a with an LLM credit assignment approach. The paper's evaluations show strong results on SWE-Bench Verified and Live.", "questions": "1. Related to W1, can this idea be extended to agent design as well as prompt content? \n\n2. Related to W2:\na) Is there quantitative evidence for multi-agent systems being better in the sorts of ways described in 5.4\nb) What sorts of agents are \"discovered\" and is there a way to characterise the trajectory of the system as is discovers these?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The major strength of the work is the novel formulation of the problem as an MAB problem and the demonstration different aspects of the  formulation leads to improvement of performance.\n\n1. The framing of optimising multi-agent systems as multi-armed bandits is novel in the context of SWE-Agents. The work successfully formulates a UCB strategy for this context as well as a way to grow the number of subagents. Ablations demonstrate how optimising sub-agents explicitly outperforms static approach.\n\n2. The use of LLM based credit assignment technique in the multi-armed bandit setup is novel and very interesting here. It shows a way to learn from specific mistakes the system might be making in a modular way. Ablations also support the importance of this metric.\n\n3. Evaluation show improvements on standard benchmarks, especially on SWE-Bench Live.", "weaknesses": "1. The major limitation of this work is that it is limited to only improving the starting prompt of the system. From the difference in performance of scaffolds present on https://www.swebench.com/, it is reasonable to believe that design decisions can impact the behavior of agent, as much or maybe more than prompts.\n\n2. The analysis of the behaviour of the system is limited to short qualitative analysis. This is currently not substantiated with example or quantitative metrics. There is also no exposition of what sorts of agents are \"discovered\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)