Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems
Authors: Shangbin Feng
Zifeng Wang
Palash Goyal
Yike Wang
Weijia Shi
Huang Xia
Hamid Palangi
Luke Zettlemoyer
Yulia Tsvetkov
Chen-Yu Lee
Tomas Pfister
Venue: NeurIPS 2025 poster ()
Keywords: model collaboration
multi-LLM systems
Primary area: applications

Abstract: We propose Heterogeneous Swarms, an algorithm to design multi-LLM systems by jointly optimizing model roles and weights. We represent multi-LLM systems as directed acyclic graphs (DAGs) of LLMs with topological message passing for collaborative generation. Given a pool of LLM experts and a utility function, Heterogeneous Swarms employs two iterative steps: role-step and weight-step. For role-step, we interpret model roles as learning a DAG that specifies the flow of inputs and outputs between LLMs. Starting from a swarm of random continuous adjacency matrices, we decode them into discrete DAGs, call the LLMs in topological order, evaluate on the utility function (e.g. accuracy on a task), and optimize the adjacency matrices with particle swarm optimization based on the utility score. For weight-step, we assess the contribution of individual LLMs in the multi-LLM systems and optimize model weights with swarm intelligence. We propose JFK-score to quantify the individual contribution of each LLM in the best-found DAG of the role-step, then optimize model weights with particle swarm optimization based on the JFK-score. Experiments demonstrate that Heterogeneous Swarms outperforms 17 role- and/or weight-based baselines by 18.5% on average across 12 tasks. Further analysis reveals that Heterogeneous Swarms discovers multi-LLM systems with heterogeneous model roles and substantial collaborative gains, and benefits from the diversity of language models.

HETEROGENEOUS SWARMS: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems
Shangbin Feng"* Zifeng Wang² Palash Goyal Yike Wang' Weijia Shi!
Huang Xia Hamid Palangi' Luke Zettlemoyer' Yulia Tsvetkor Yuen-Yu Lee² Tomas Pfister*
'University of Washington ?Google Cloud AI Research Gooole
shangbinncs. www.sesssswsssissnnnoeee {zifengu, chenyuulee@@googgee.co Bgg
Abstract
We propose HETEROGENEOUS SWARMS, algorithm to design multi-LLM sys- tems by jointly optimizing model roles and weights. We represent multi-LLM systems as directed acyclic graphs (DAGs) of LLMs with topological message passing for collaborative generation. Given a pool LLM experts and a utility function, HETEROGENEOUS SWARMS employs two iterative steps: role-step step and weight-step. For role-step, we interpret model roles as learning a DAG that specifies the flow of inputs and outputs between LLMs. Starting from swarm of random continuous adjacency matrices, we decode them into discrete DAGs, call the L.MSs in topological order, evaluate the utility function (e.g. accuracy task), and optimize the adjacency matrices with particle swarm optimization based on the utility score. For weight-step, we assess the contribution of individual LLMs in the multi-LLM systems and optimize model weights with swarm intelligence. We propose JFK-score quantify the individual contribution of each LLM in the best-found DAG of the role-step, then optimize model weights with particle swarm optimization based on the Jcoree Experiments demonstrate that HETEROGE- NEOUS S SWARMS outperforms 17 role- and/or weight-based baselines by 18.5% on average across 12 tasks. Further analysis reveals that HETEROGENROUS SWARMS discovers multi- systems with heterogeneous model roles and substantial collaborative gains, and benefits from the diversity of language models."
Introduction
Advancing beyond training single general- purpose LLM [6, 87]. recent research recognizes the importance of multi-LLM collaboration and advances multi-LLM. -LM sys- tems, where diverse models serve in a collaborative system complement each other and expand model capabilities [58, 79]. Models often have different roles in multi-LLM collaboration, governing the subtask and functionality of individual LLMs; adapting the model weights of these LLMs also identified important for models com- plement each other. Existing methods to develop multi- LLM systems are often fixed-weight and/or fixed-ron role and could not flexibly adapt to diverse tasks and contexts.
utility
LLMs
optimize role
optimine
weight
Input
output
Figure I: Our objective: given pool of LLMs and task utility function f. dis- cover multi-LLM system with graph- based model roles and adapted model weights tailored to f.
weighh systems employ static and often black-box LLMs and contextualize their roles through textual inter-
"Work done student researcher Google Cloud AI Research. 2 Resources available at http::/ggilub..om/Buustet Sunnuseenengghettrggeeeeessssss swaaleerrrggennuuussssa... swarm.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Algorithm 1: Particle Swarm Optimization step (PSO)
Input:
vectors {x:)== and the utility values (f(x;)) )== by utility function f Hyperparameters: step length A, inertia 6r, cognitive coeff. Φp. social coeff. Φg. repel coeff. ow State variables: each vector x; has velocity v; and personal best P:. global best and worst g and
gw
for i to n parallel do walk randomness Tu, Tp Uw Iw Uniform(0,1)
V; < ج [rooo, (P: x;) r00g( xi) ro (Bw x,]]. where normalization
term C = ro + Toop 0g + &w
x; 4 x; + AV;
end
Update personl/global information pi, g, and ga: return xx )x>)==1> Xbext = arg maxy [x)
action [21, 120]. Despite the diversity of tasks and inputs, these static models are repeated across model roles and contexts, becoming the bottleneck of flexible adaptation. Fixed-role systems usually orchestrate LLMs in fixed workflow and seed LLMs with different hand-crafted prompts and enable their interaction of message passing based on these prompt- cenpt-induued roles [80, 25]. However, new tasks and domains would require substantial prompt engineering, which heavily depends on prior knowledge of the given task. Hence the static roles become the bottleneck in automating and scaling multi-LLM systems unseen tasks and contexts [49, 88, 50]. As such, a flexible approach that jointly optimizes the weights and roles of multi-LLM systems is crucial for adapting diverse LLM experts for wide-ranging purposes.
We propose HETEROGENEOUS SWARMS to search for adapted roles and weights guided by utility function (e.g. performance on task) via swarm intelligence. Inspired by the success of LLMs and particle swarm optimization (PSO) [48, 26], an algorithm to optimize continuous tensors via collective search, we employ two interleaved steps: role-step and weight-step.
For role-step, we use the heterogeneous expertise of models for varying roles: we interpret roles as input-output relations in multi-LLM system. Role optimization is graph learning problem, where natural language messages are passed between LLMSs organized in a directed acyclic graph (DAG). While previous methods considered heuristics beuse and hand-crafted structures such star chain [52], finding the optimal structure of multi-LLMM systems requires task task-specific adaptation [37]. We use particle swarm optimization learn LLM DAGS: Given
of LLM experts, we randomly initialize a swarm of continuous adjacency matrices of size n X n. indicating the likelihood of having directed edge (i, We propose G-DECODE to decode these continuous adjacency matrices into discrete DAGs of models. We call LLMs in the topological order of the DAG and evaluate their performance: the swarm of adjacency matrices are then optimized based on the performance scores via PSO, where matrices collectively "move" in the matrix search space to adapt to f.
For weight-step, propose JFK JFK-scon K-coree to quantify what individual models can do for the multi-LLM system: assigning the pool of LLMs to positions in the DAG multiple times, run inference, and evaluate different assignments. The JFK-score of one model is the aggregated performance across assignments weighted by their frequency (i.e. number of times it appears in a multi-LLMM system). The swarm of LLMSs is then optimized based on the JFK-scores via PSO, where models update their weights to adapt to f.
adapt models the task and roles represented by the multi-LLM network. We
Role-step and weight-step utility f does not improve or a maximum iteration limit is met. the end, we obtain a multi-LLM network where the graph structure and model weights are both optimized for utility f. (Figure 1)
iteratively employed to jointly optimize roles and weights, until the
HETEROGENEOUS SWARMS outperforms 15 role/weight- based approaches by 18.5% on average across 12 tasks spanning knowledge, reasoning, and agent contexts. HETEROGENEOUS SWARMS also enables inference-time scaling [82, 5. 100] of smaller language models through topological collaborative generation, with a 27. improvement on average when scaling the collaboration from two to ten L.LMs. (Figure 6) Further analysis reveals that roles and weights could have varying levels of importance for different tasks, HETEROGENEOUS S SWARMS benefits from the diversity of LLM

---

Initialization
Role-Step
Weight-Step
15
Swarm of Graphs
assignment
optimize
Swarm of Models
0.42
0.71
0.68
0.85
utility
19021
0.63
Unilty
-0.66
-0.55
m -0.73
00.67
Figure 2: Overview of HETEROGENEOUS SWARMS: starting with a swarm of graphs represented by continuous adjacency matrices and swarm of LLMs, HETEROGENEOUS SWARMS rotates between role-step and weight-step. graphs, call the LLMs in topological order to fulfill task, evaluate on the utility function, and optimize the adjacency matrices with particle swarm optimization. In the weight-step, models are randomly assigned to positions in the best-found network, evaluated by their individual contribution through the JFK-Score, and then optimized with particle swarm optimization. PSO denotes particle swarm optimization (Sec 2), G-Decode denotes Algorithm 2, and denotes the utility function.
the role-step, we decode continuous adjacency adjacencies into discrete
experts, the optimization could speed up with increased sparsity and "dropout", and discover multi-LLM systems with heterogeneous roles and collaborative gains.
Preliminary
Multi-LLM Systems Multi-LLM collaboration could take many forms, featuring information exchange the API [36], text [21], logit [58], and weight levels [102]. In this work, we define multi- LLM systems as directed acyclic graphs (DAG) of LLMs and a multi-LLM system is represented
by two variables: a set of LLMs X' = [x]. ........ Xn.] and adjacency matrix A { Jaxn. where ag E {0, 1} and a.j- = 1 indicates that the output text of model x, becomes part of the input of model x, with models in total.
X then denotes a multi-LLM system, where A defines the structure of the DAG and each position in the graph is instantiated by LLM in X. Given an input, models in X' called in the topological order of A to interact and collaborate with each other. As such, optimizing roles and input-output relations becomes optimizing the graph structure A, and optimizing weights becomes optimizing the weights of models in X.
Swarm Intelligence Particle swarm optimization (PSO) an optimization algorithm based on the collective behavior individual systems for oin-different contexts [48]. Guided by its initial success in LLMs [26], we adapt PSO to optimize swarms of A and x.
The input of PSO is set of continuous vectors {x,} } scalar score f (x). Each vector x, has two attributes: velocity v, of the same dimension, initialized as O; personal best P., the best-found location x; based on utility function f in its search history. The collective swarm has two other attributes: global best/worst g and gw. indicating the best/worst-found checkpoint in all of the previous )x, These attributes would provide signals for vector x, to move in the search space guided by personal/global information.
and utility function giving vectors
One PSO step changes model velocity v, and takes step towards the adjusted velocity direction:
C [ rp (p: x;)+
r 1 x.) Twn aw (g.. - x4)
where C = re + Trrpppptgee Two is normalization term. The adjusted velocity is the weighted average of four terms: v, makes the model keep part its current velocity (i.e. inertia); (p: x;)
draws the model towards its personal best; (g xx) draws towards the global best; -(g.. x;), repel from the global worst. 6r, Φp, &gs the are hyperparameters and r.: Tp, Ta. ~0, 1) are randomness factors, governing how much x; is impacted by personal/global information. Vectors
3

---

then take step towards the new velocity: x; + x; + Av;, where À is the step length hyperparameter. New vectors are then evaluated on update personal/global best and worst. We formulate a PSO step in algorithm I and will use {x: [x 1 Xheur = PSPOS (xx [x ) (f(x:))==1 to denote PSO step to optimize {xi) based on their scores (f(x;))==
3 Methodology
We propose HETEROGENEOUS SWARMS, jointly optimizing model roles and weights for multi-LLM systems. We employ PSO for the optimization, since it uniquely enables us to leverage multiple diverse collaboration patterns and specialized model checkpoints, composing their expertise and strengths in multi-LLM systens. Starting from a pool of LLM experts as input, HeTeRoGENEOUS SWARMS discovers directed acyclic graph of models representing heterogeneous roles with adapted model weights, iteratively employing role-step and weighl-step (Figure 2).
3.1 Role-step
Unlike existing work that often manually specifies model roles through prompt engineering [25, 116), we propose to learn a directed acyclic graph of input-output relationships. To this end, HETEROGE NEOUS SWARMS operationalizes the optimization of model roles as a graph optimization problem: discovering and optimizing directed acyclic graphs (DAGs) of LLMs to call them in the topological order for collaborative generation. We envision swarm intelligence as a viable and flexible tool by letting multiple potential DAGs explore the search space of graphs to adapt to given task and utility function.
To make discrete graphs compatible with particle swarm optimization (Sec 2) that optimizes con- tinuous tensors, we first represent graphs by continuous adjacency matrices A | where aij denotes the likelihood of a directed edge from model x, to model Xj. We propose G-decode, an algorithm decode discrete DAGs out of continuous As. We first select an end node & based on inverse out degrees k = top-p((1/ ΣØ= Giy P==)), where top-p denotes top-p sampling [35]. We then iteratively select a remaining node based on out degrees
top-p ([E)= (iz)) and add an edge between and any existing node with probability Eice exp(a_.)> until all nodes consid- ered. Graphs decoded with G-decode are directed since we only add directed edges and they are acyclic since new nodes only connect existing nodes in the graph. The resulted DAGs define input-output mapping: given an input, we call LLMs in the topological order of G-decode( A), and the output of the end node becomes the output of the multi-LLM system (Algorithm 4).
exp(owe)
We evaluate the utility of the continuous adjacency matrix by decoding it into a DAG, call the models in topological order, and evaluate its performance on the utility function f. Concretely, initialize a swarm of N continuous adjacency matrices (A') N Unxx (0, 1). For one role-step, we decode and evaluate their utility (G-decooee ") and optimize the continuous As through swarm intelligence:
{A' N Ji=1 Abest < PSO((A') [f(G-decodd( ( (A
The resulting {AI after a single PSO step will represent graphs that slightly better adapt the task f.
3.2 Weight-step
While existing work often uses static models across tasks and contexts [120, 21], we posit that model weights could be optimized to adapt to the task as well as model roles in the DAG. However, it is challenging to quantify the utility of a single LLM in multi-LLM system, so that model weight optimization is compatible with swarm intelligence. We propose JFK-score, algorithm to evaluate individual contribution in multi-I.L.M systems.
Concretely, given the best-found DAG in the role-step Abes and the pool of LLM experts (x we randomly select an LLM for each position in Abest to obtain an assignment X, where Abest X
2 "Ask not what the multi-LLM LLM system can do for you, ask what you can do for the multi-LLM LLM system." Authors, 2025

---

Algorithm 2: JFK-score
Input: adjacency matrix Abest and models (x,
for i = to M do
sample assignment X' by randomly select models in {x+ to fill each position in Abea
utility f(x*) = f(G-decode( Abeet x)))
end
for i -ari=l to do
JFK-score(x;) D1=1 enty, x xf(2')
cut,, denotes the frequency of x; in assignment X²
2 cnt.,
end
return {JFK-scoro(x)))
Algorithm 3: Heterogeneous Swarms
Input: language models {x, 1 utility function f
sample (A) Ji=1 ~ (0, 1)
while f is improving do
// role-step
ii=1 {A') Aben + PSOS((AA)))) Ji=1s (((G-decode(A)))
// noight-ster
[x:) Xbest f PSO (x; (JFK-score(x;))2:
end
return G-decode(Ar) (sea {x))==))
represents instantiated multi-LLM system. We repeat assignment M times obtain {x*) j and evaluate their utility f(x') = f(G-decode(A' (A' X²)). The individual contribution model x; should then be the aggregate of utility weighted by model i's frequency: denote the frequency of x; in X ent, Ekm 1 x;) and the individual contribution score should be:
JFK-scorc(x,) Ej=1 enl., x [(x))
Σj cnt,,j
In this way, JFK-score(x:) quantifies the individual contribution of model x; across multiple roles and collaboration partners. (Algorithm 2) We optimize model weights guided by their individual contribution with swarm intelligence:
{x:) Xbear +- PSO( (JFK-score(x)))2]
HETEROGENEOUS SWARMS alternates between role-step and weight-step. iteratively optimizing model roles and weights in the multi-LLM system adapt the task f. We present the overall procedure in algorithm 3.
4 Experiment Settings
Models and Implementation We implement prototype of HETEROGENEOUS SWARMS with GEMMA-7B (gongle/gemma-7bbi)) [29] in the main paper and also employ other LLMs such as MISTRAI--BB Table 5. We employ the pool of 10 LLM experts in Feng et al. [26] for fair comparison, fine-tuned from GEMMA-7B using 10 domains in Tulu-v2 [44] spanning reasoning, code, general instruction following, and more: We employ = 0.8 for top-p sampling, N = 10, M 10, search patience max iteration 20, while running grid search over other hyperparameters and report performance of the best-found multi-LLM systems.
Baselines compare with baselines across 5 categories that focus on optimizing model roles and/or weights.
5

---

Knowlodge
Reasoning
Agent
Misccilancoos
prL
K-Cress
COM2
GSMSK
NLGraph
Normad
GAIA-texi
AB-kg
AB-ltp
Qusper
AbstainQA
WoW
SINGLE
0.231
0.346
0,488
0.237
0.535
0.545
0.107
0.383
0.120
0.174
0.065
0,415
PRED. MERGR
0.173
0.309
0.391
0.074
0.502
0.481
0.036
0.225
0.471
DATA MERGE
0.176
0.370
0.377
0.143
0.423
0.415
0.071
0.242
0.112
0.147
.0.025
0.461
UNIFORM SOUP
0.295
0.500
0.392
0.003
DARE-TIES
0.230
0.572
0.476
0.307
0.544
0.427
0.071
0,300
0.108
0.137
0.140
0.515
GREEDY SOUP
0.219
0.355
0.539
0.330
0.530
0.543
0.071
0.333
0.114
0.184
0.014
0.568
PACK OF LLMS
0.235
0.352
0.512
0.327
0.532
0.543
0.143
0.392
0.106
0.157
0.095
0.545
LORAHUB
0.231
0.291
0.502
0.354
0.568
0.548
0.071
0.375
0,106
0.169
0.064
0.530
MODEL SWARMS
0.254
0.428
0.505
0.459
0.672
0.554
0.107
0.358
0.135
0.225
0.175
0.540
CHAIN
0.216
0.310
0.495
0.295
0.462
0.489
0.143
0.325
0.148
0.218
0.014
0.493
STAR
0.250
0.342
0.508
0.333
0.545
0.518
0.033
0.283
0.130
0.216
0.125
0.499
GPT--WWARM
0.216
0.320
0.460
0.334
0.611
0.510
0.143
0,333
0.134
0.216
0.023
0.492
META-AGENT
0.212
0.276
0.477
0.435
0.515
0.369
0.071
0.325
0.112
0.167
0.016
0.472
AGEEN--RRUNE
0.214
0.321
0.497
0.180
0.460
0.467
0,107
0.333
0.122
0.211
-0.005
0,470
GNNS
0.201
0.339
0.479
0.364
0.593
0.530
0.071
0.308
0.148
0,203
0.076
0,503
AGENTVERSE
0.309
0.501
0.403
0.633
0.367
MACNET
0.252
0.323
0,517
0.409
0.617
0.537
0.143
0.383
0.138
0.207
0.127
0.512
H-SWARMS
0.312
0.450
0,579
0,481
0.660
0,588
0,250
0.425
0.215
0,266
0.220
0.590
Table I: Performance on the 12 datasets, best in bold and second-best in underline. HETEROGE- NEOUS SWARMS outperforms TRIVIAL, STATIC WEIGITT DYNAMIC WEIGHTT STATIC ROLE, and DYNAMIC ROLE approaches by 18.5% on average across tasks.
Delawe by
Delasd:: C012
Oakast KOrmort
Nitast: incored correct
Poblan Durt
150
150
200
1.50
0.79
0.75
0.35
0.76
0.50
023
0.25
15
Numbar www.
Curmc
edhicul LUVA Cored
Figure 3: Evaluating collaborative gains: we create problem buckets by how many out of the 10 individual LLMs could solve it correctly. Top row: the problem count as well as whether the multi- LLM correctly solves the problems in each bucket. Bottom row: accuracy of the multi-LLM system in each bucket and expected accuracy denoted by the dotted line. HETEROGENEOUS SWARMS achieves collaborative gains (C-Gain) of 0.143, 081, 0.101, and 0.426 on the four datasets, all >> 0, and demonstrate consistent collaborative gains.
trivial baselines: 1) Best Single expert, essentially arg maxx fxx for E {x,]] 2) Prediction Merge, where the predictions of {x are ensembled via plurality vote (if applicable).
static weight: these approaches conduct model merging independent of the task and utility function f: Data Merge, Uniform Soup [97], and Dare-Ties [108, 103].
dynamic weight: these approaches optimize model weights based on the utility function f: Greedy Soup 197], Pack of LLMs [66]. LoraHuub [40], and Model Swarms [26].
static role: we employ two structures in
agent systems [52]: chain and star.
dynamic role: these approaches optimize model roles and connections based on f: GPT-Swarm [120], Meta-Agent [37], Agent-Prune [109], GNNs [110], AgentVerse [1]], and MACNet [73].
We compare HETEROGENEOUS SWARMS against baselines on 12 datasets spanning categories: 1) knowledge: MMLU-pro [94], Knowledge Crosswords [19], and COM2 [23]; 2) reasoning: GSM8k [13], GL. [89], and Normad [76]: 3) agent: GAIA-text [67], the knowledge graph and lateral thinking puzzle subtasks of AgentBench [61]; 4) miscellaneous: long-context with Qasper [16], reliability with AbstainQA [25], and judeae-a-udde with WoW [106].
Data and Evaluation

---

Results
We present the performance of HATEROGENFOUS SWARMS and baselines the 12 tasks in Table 1.
HETEROGENEOUS SWARMS consistently discovers state-of-the-art multi-LLM systems. HET- EROGENEOUS SWARMS achieves the best performance 11 of the 12 datasets, outperforming the second-best approach by 18.5% on average. This indicates that starting from the same pool of initial LLMs, HETEROGENEOUS SWARMS could flexibly discover multi-LLM systems that adapt diverse tasks and contexts spanning knowledge, reasoning, and more.
Role and weight are disproportionately important for different tasks. For knowledge tasks, weight baselines (STATIC WEIGHIT and DYNAMIC WEIGITT) outperform role baselines (STATIC ROLE and DYNAMIC ROLE) by 4.3% average. However, for agent tasks, role baselines are 9.2% better. This indicates that role and weight have varying importance different tasks: by jointly optimizing roles and weights in multi-LLM systems, HETEROGENEOUS SWARMS flexibly adapts to both scenarios. We further investigate their importance in Section
Dynamic adaptation works better than static engineering. We find that DYNAMIC WEIGHT approaches outperform STATIC WEIGHT by 30.1% 10.1% across tasks, while DYNAMIC ROLE outperforms STATIC ROLE by up to 8.2%. This indicates that instead of hand-crafted ways to design roles or merge weights, dynamic adaptation approaches guided by utility function offer better adaptation. HETEROGENEOUS SWARMS employs role-step and weight-step to dynamically adapt both for S. resulting flexible multi-LLM systems adapted to diverse tasks and applications.
Analysis
Collaborative Gains To justify the cost of calling multiple LLMs in topological order, multi-LLM, system should unlock 1 1+ effects: the multi-LLM, collaboration should produce collaborative gains compared to employing single LLM. Concretely:
For problem q, if p% of the component LLMs could solve it individually, then the multi-LLM system should have a larger than p% likelihood solving it.
To quantify this, we group problems in dataset D by how many component LLMs could solve individually; problem q E B, if of the N component LLMs could solve it. We then calculate the metric Collaborative Gain as:
N
Bn D
C-Gain
(Aco(Ba) - EA(B.))
R=1
where Ace(B,, denotes the accuracy of the multi-I.LM systems for problems in B,, F.A (B₂) denotes the Expected Accuracy n/N, [B,] and (D) denote the number of problems in the bucket and the dataset. If multi-LLM system satisfies the principle, then Acc( (B) EA(B,) and C-Gain 0.
Knowledge
Reasoning
Agent
Miscellancous
MML.U-poo
K-Cross
COM2
GSMB
NI.Graph
Normad
GAIA-text AB-kg.
AB-ltp
Qusper
AbstainQA
WoW
Role Baselines
0.218
0.318
0.486
0.323
0.531
0.480
0.095
0.318
0.132
0.205
0.042
0.488
Weight Baselines
0,222
0.352
0.490
0.325
0.538
0.494
0.082
0.342
0.112
0.169
0.067
0.516
Ours w/o Role
0.242
0.352
0.515
0.392
0.530
0.564
0.107
0.317
0.140
0.222
0.133
0.539
Ours Weight
0.237
0.492
0.363
0.557
0.143
0.241
0.119
Dans Full
0.312
0.450
0.579
0.481
0.660
0.588
0.250
0.425
0.215
0.266
0.220
0.590
Consistent?
TRUE
TRUE
TRUE
TRUE
FALSE
TRUE
TRUE
FALSE
TRUE
TRUE
TRUE
TRUE
Table 2: Ablation study of removing the role-step or weight-step in HETEROGENROUS SWARMS, comparing whether the importance of role/weight is consistent between baselines and approach. The pattern is consistent in 10 out of 12 datasets, confirming that model roles and weights could have different levels of importance for varying tasks.
We present the collaborative gains of HETEROGENEOUS SWARMS in Figure HRTEROGENEOUS S SWARMS achieves consistent positive collaborative gains with average of 0.213. For problems
7

---

Mig
310%
DLTS
Neteo
0.60
8.00
0.00
0.75
1.00
Opérization Progress
Figure 4: Analyzing the roles in Multi-LLM systems. Top left: the percentage of LLM roles aggregated per dataset. Bottom left: the change of LLM roles in the optimization process for NLGraph. Right: Per-LLM role distribution in the best-found multi-LLM system for NLGraph. Together these figures demonstrate the heterogeneous roles in the multi-LLM systems by HETEROGENEOUS SWARMS.
in bucket Bo, i.e. none of the initial LLMs could solve individually, HETEROGENEOUS, SWARMS discovers multi-LLM systems that solve 18. 1% them on average. We additionally find that HETEROGENEOUS SWARMS has greater C-Gain than baseline in Table 12. This indicates that HETEROGENEOUS SWARMS could find adapted multi-LLM systems with new compositional skills and substantial collaborative gains.
Importance of Role and Weight through Ablation Study In section 5, we discover that model roles and weights could be disproportionately important for different tasks based baseline per- formance. We compare the trend with our approach, specifically through disabling either the role step or the weight step in HETEROGENEOUS SWARMS (w/o Role and w/o Weight). Let B, and B., denote the average performance of role/weight based baselines, then the importance of role/weight is consistent when the following logic expression is True:
(wwo Role < w/o Weight) AND (B, > Bw)) OR ((w/o Role w/o Weight) AND (B, < Bwv))
We present performance of the ablated settings and the value of the logic expression across 12 datasets in Table 2. Results demonstrate that roles and weights could have varying importance (e.g. weight is more important for knowledge tasks while role is more important for agent) and such importance is consistent in 10 of the 12 tasks. In addition jointly adapting model roles and weights, HETEROGENEOUS SWARMS offers insights into their importance for the task at hand.
Role Statistics We manually examine the input/output of multi-LLM systems, identifying four potential roles of individual models in the multi-LLLM system: 1) divide, where the L.M identifies and solves part of the problem; 2) refine, where the LLM proposes new (sub)answer based on previous steps; 3) feedback, where the LLM provides feedhack on previous steps; 4) irrelevant, where the LLM fails to generate relevant text. We employ Gemini-as-a-jude [87] (GEMINI- 1.NNN--1.--0 1.5-FLASH) to automatically identify the role individual LLMs and conduct three analysis in Figure 4.
We analyze the variation of model roles across tasks in Fig.4, top left. For the graph reasoning task NLGraph, there is greater divide and conquer to solve part of the problem: for the knowledge tasks such as K- K-Crosswords, there is greater feedback to identify knowledge gaps in existing answers. Guided by different fs, HETEROGENEOUS SWARMS discover multi-LLM systems with different role distributions.
We investigate the roles of LLMss in different positions of the DAG NL.rraph in Fig.4, right. We find that individual LLMs do have heterogeneous role distributions given their topological position: for the branching nodes there is often higher divide, while for the converging nodes there is often higher refine and feedback. This indicates that HETEROGENEOUS SWARMS successfully discovers multi-LLM systems where individual LLMs play heterogeneous roles.

---

Abeaings
Acrecoannts
CavO
0.6
0.4
0.0
Expert Pool
88:88 5"2
88:88
1'10
255
10"1
Figure 5: HETEROGENEOUS SWARMS with increasing levels of diversity in initial LLMs. Results show general upward trend and an 89% increase on average from the least most diverse models.
NLGraph
kg
We plot the change of model roles in the optimization process in Figure 4, bottom left, when adapting to NL-
Acc
Specdlep
Ace
Speslup
Aoc
Specdhep
original
0.0%
Graph. We observe that irrelevant roles gradually decrease
0.05
0.438
3.8%
0.628
4,9%
0.392
9.28
while the three other functional roles consistently increase in the course of optimization. This indicates that by inte- grating weight optimization, HETEROGENEOUS S SWARMS improves the quality of LLMs to adapt to the task at hand.
0.392
12.4%
0,809
9.3%
0.375
19.8%
0.352
0.383
21.4%
0.01
0.436
1.2%
0.642
0.9%
0.400
3.4%
0.05
01425
3.5%
0.614
2.3%%
0.383
9.18
0.1
0.425
7.1%
0.598
2.5%
0.392
12.8%
Table 3: Encouraging sparsity in multi- LLM systems with thresholed pruning (T) or normalization (A). These strategies bring various tradeoffs bewteen perfor- mance and inference speedup.
Encouraging Sparsity We observe that if the network dense, i.c. when one LLM takes the output of too many LLMs as input, it might lead to irrelevant outputs. In ad- dition, encouraging sparsity in the network would also reduce context lengths and hence inference costs. To this end, encourage sparsity in the multi-LLLM network by means: 1) by setting threshold
E [0, 1] and pruning the continuous adjacency A with A' = {a:j 1 [max(a;; - T, 0]]} 2) by adding C1 normalization term the utility function: = f A|| We employ various values o and A: Table presents the performance and speedup calculated by the percentage of reduced connections in the network. Results demonstrate that L1 better retains performance while thresholding presents larger reduction in inference cost.
Diversity Matters HETEROGENEOUS SWARMS operates on the assumption that the multiple large language models would complement each other based their diverse expertise. To investigate whether this model diversity is crucial, we conduct an controlled experiment where we fix the LLM pool size as 10, but with distinct models each repeated b times. We denote this 1 x 10, x 5, 5 x 2, and 10 x (default setting) with increasing diversity. Figure illustrates that the diversity of initial LLMs does help greatly, with an average relative improvement of 89% from the least to most diverse.
b and employ
7 Related Work
growing line of research focuses on multi-LLM collaboration, where multiple models collaboate and complement each other. These approaches focus either model roles or weights.
Role-based approaches typically rely on assigning roles to LLMs through prompt engineering [21, 25]. Multiple LLMs, or even just a single LLM seeded with different prompts, then collaborate with their prompt-induced roles through exchanging generated texts. For example, specialized LLMs could augment responses to collectively self-refine [25, 7]; (multi- Jagent systems could divide and conquer complex problems [98, 33, 91, 60, 118, 71, 47, 65, 55, 99, 43, 56, 92, 86, 53, 75, 68, 9, 17, 119]; multiple LLMs could debate and compete with each other to find better answers [57, 21J. These approaches are often hindered by the need for prompt engineering and the effectiveness of prompts for model steerability [83, 77], thus HETEROGENEOUS SWARMS uniquely interprets model roles as input-output relationships, optimizing directed acyclic graphs of LLMs to learn contextual roles and specialization.
model [24, 79]: LLMs could generate feedback for each other's 's

---

Weight- based approaches typically focus on adapting the logits/ of multiple LLMs, notably through -efeerrr- or model merging [102]. The hidden states or logit distributions of multiple models could be selected, routed, and aggregated based on various MoE mechanisms [54, 30]. In addition, static [108, 103, 45] and dynamic [66, 1, 40] model merging approaches incorporate the diverse expertise of heterogeneous LLMS into single model in zero-shot and adaptation settings. We continue to believe that weight adaptation is crucial for specializing individual LLMs in multi-LLM systems: guided by the first successes of evolutionary algorithms in weight-baset collaboration [26] and LLMs in general [1. 27, 32J, HETEROGENEOUS SWARMS employs swarm intelligence to optimize model weights guided by each LLM's individual contribution to the multi-LLM system.
HETEROGENEOUS SWARMS uniquely offers a flexible methodology to jointly optimize the roles and weights of diverse LLMs, discovering and adapting novel multi-LLM, systems.
Conclusion
We propose HETEROGENEOUS SWARMS, an algorithm to jointly optimize model roles and weights for multi-LLM systems and collaborative generation. By rotating between role-step and weight- step to optimize the network of LLMs as well as model weights, HETEROGENEOUSS SWARMS discovers directed acyclic graphs of LLMs that could be called in topological order to adapt to given task. HETEROGENEOUS SWARMS outperforms role and weight- baselines on tasks, demonstrating collaborative gains and heterogeneous roles through multi-LLM collaboration.
Acknowledgements
This research was developed with funding from the Defense Advanced Research Projects Agency's s (DARPA) SciFy program (Agreement No. HR001 12520300). The views expressed are those of the author and do not reflect the official policy position of the Department of Defense or the U.S. Government. This material is based upon work supported by the Defense Advanced Research Projects Agency and the Air Force Research Laboratory, contract number(s): (A8650-23-C-7733 Any opinions, findings and conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of AFRL or DARPA. Shangbin Feng would like to thank the support of the IBM PhD Fellowship and Jane Street Graduate Research Fellowship.
References
[1] Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. arXiv preprint arXiv:2403.13187, 2024.
[2] Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, et al. Agentharm: A benchmark for measuring harmfulness of 11m agents. arXiv preprint arXiv:2410.09024 2024.
[3] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyezyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682-17690, 2024.
[4] Xiaohe Bo, Zeyu Zhang, Quanyu Dai, Xueyang Feng. Lei Wang, Rui Li, Xu Chen, and Ji-Rong Wen. Reflective multi-agent collaboration based on large language models. Advances in Neural Information Processing Systems, 37:138595-11388 2024.
[5] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787 2024.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
10

---

[7] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker. Leo Gao, Leopold As- chenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Forty-first International Conference on Machine Learning, 2024.
[8] Souradip Chakraborty, Sujay Bhatt, Udari Madhushani Sehwag, Soumya Suvra Ghosal, Jiahao Qiu, Mengdi Wang, Dinesh Manocha, Furong Huang, Alec Koppel, and Sumitra Ganesh. Collab: Controlled decoding using mixture of agents for Ilm alignment. arXiv preprint arXiv:2503.21720, 2025.
[9] Ma Chang, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn Ilm agents. Advances in Neural Information Processing Systems, 37:74325-74366 2024.
[10] Justin Chen, Swarnadeep Saha, Elias Stengel-Eskin, and Mohit Bansal. Magdi: Structured distillation of multi-agent interaction graphs improves reasoning in smaller language models. In Forty-first International Conference on Machine Learning, 2024.
[1]] Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations, 2024.
[12] Wonje Choi, Woo Kyung Kim, Minjong Yoo, and Honguk Woo. Embodied cot distillation from IIm to off-the thellf agents. In Forty-first International Conference on Machine Learning, 2024.
[13] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reüchhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
[14] Nicholas Crispino, Kyle Montgomery, Fankun Zeng, Dawn Song, and Chenguang Wang. Agent instructs large language models to be general zero-shot reasoners. In Forty-first International Conference on Machine Learning, 2024.
[15] Logan Cross, Violet Xiang, Agam Bhatia, Daniel LK Yamins, and Nick Haber. Hypothetical minds: Scaffolding theory mind for multi-agent tasks with large language models. arXiv preprini
2024.
[16] Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. In Pro- ceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599-4610 2021.
[17] Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Kellner, Marc Fischer, and Florian Tramèr. Agentdojo: dynamic environment evaluate prompt injection attacks and defenses for Iim agents. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[18] Gang Ding, Zeyuan Liu, Zhirui Fang, Kefan Su, Liwen Zhu, and Zongqing Lu. Multi-agent coordination via multi-leve communication. Advances in Neural Information Processing Systems, 37:1 18513-1 118539, 2024.
[19] Wenxuan Ding, Shangbin Feng, Yuhan Liu, Zhaoxuan Tan, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Knowledge crosswords: Geometric knowledge reasoning with large language models. In Findings of the Association for Computational Linguistics ACL 2024, 2024.
[20] Darko Drakulic, Sofia Michel, and Jean-Mare Andreoli. Goal: A generalist combinatorial optimization agent learner. The Thirteenth International Conference on Learning Represen- tations, 2024.

---

[21] Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2024.
[22] Andrew Estornell, -aanFFaancois Ton, Yuanshun Yao, and Yang Liu. Acc-debate: An actor- critic approach to multi-agent debate. In The Thirteenth International Conference on Learning Representations, 2025.
[23] Tianqing Fang. Zeming Chen, Yangqiu Song, and Antoine Bosselut. Complex reasoning over logical queries on commonsense knowledge graphs. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), August 2024.
[24] Shangbin Feng, Weijia Shi, Yuyang Bai, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Knowledge card: Filling Ilms' knowledge gaps with plug-in specialized language models. In The Twelfth International Conference Learning Representations, 2024.
[25] Shangbin Feng Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don't hallucinate, abstain: Identifying LLM knowledge gaps via multi-LLM collaboration, In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024.
[26] Shangbin Feng, Zifeng Wang, Yike Wang, Sayna Ebrahimi, Hamid Palangi, Lesly Miculi- cich, Achin Kulshrestha, Nathalie Rauschmayr, Yejin Choi, Yulia Tsvetkov, et al. Model swarms: Collaborative search to adapt Ilm experts via swarm intelligence. arXiv preprint arXiv:2410.11163, 2024.
[27] Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. In Forty-firsi International Conference on Machine Learning, 2024.
[28] Dayuan Fu, Keqing He, Yejie Wang, Wentao Hong, Zhuoma Gongque, Weihao Zeng, Wei Wang. Jingang Wang. Xunliang Cai, and Weiran Xu, Agentrefine: Enhancing agent generaliza- tion through refinement tuning. arXiv preprint 1000:::200000002... 2025.
[29] Gemma Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08293 2024.
[30] Nikolas Gritsch, Qizhen Zhang, Acyr Locatelli, Sara Hooker, and Ahmet Üstün. Nexus: Specialization meets adaptability for efficiently training mixture of experts. arXiv preprint arXiv:2408. /5901, 2024.
[31] Zhenyu Guan, Xiangyu Kong, Fangwei Zhong, and Yizhou Wang. Richelieu: Self-evolving llmeb agents for ai diplomacy. Advances in Neural Information Processing Systems, 37: 123471-123497, 2024.
[32] Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In The Twelfth International Conference on Learning Representations, 2024.
[33] Taicheng Guo, Xiuying Chen, Yaqi Wang. Ruidi Chang. Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.0/ 0860, 2024.
[34] Yiwei Guo, Shaobin Zhuang, Kunchang Li, Yu Qiao, and Yali Wang. Transagent: Transfer vision-language foundation models with heterogencous agent collaboration. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024.
[35] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference Learning Representations, 2020.
12

---

[36] Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. Routerbench: A benchmark for multi-llm routing system. arXiv preprint arXiv:2403. 12031, 2024.
[37] Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. In NewrlPS 2024 Workshop on Open-World Agents, 2024.
[38] Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, et al. Infiagent-dabench. Evaluating agents on data analysis tasks. In Forty-first International Conference on Machine Learning, 2024.
[39] Yue Hu, Yuzhu Cai, Yaxin Du, Xinyu Zhu, Xiangrui Liu, Zijie Yu, Yuchen Hou, Shuo Tang, and Siheng Chen. Self-evolving multi-agent collaboration networks for software development. 2024.
arXiv preprint arXiv:2410.
[40] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lo- rahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint 1xXiv:3307113266 2023.
[41] Jen-tse Huang. Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Michael Lyu. Competing large language models in multi-agent gaming environments. The Thirteenth International Conference on Learning Representations, 2025.
[42] Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. In Forty-first International Conference on Machine Learning, 2024.
[43] Yizhe Huang, Xingbo Wang, Hao Liu, Fanqi Kong, Aoyang Qin, Min Tang, Xiaoxi Wang, Song-Chun Zhu, Mingjie Bi, Siyuan Qi, et al. Adasociety: An adaptive environment with social structures for multi-agent decision-making. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[44] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, al. Camels in a changing climate:
Enhancing im adaptation with tulu 2. arXiv preprint arXiv:231 1.22111110722. 2023.
[45] Dong- Swag-Hwan Jang, Sangdoo Yun, and Dongyoon Han. Model stock: All we need is just a few fine-tuned models. arXiv preprint arXi::2403.19522 2024.
[46] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel. Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0663 2023.
[47] Beomseok Kang, Priyabrata Saha, Sudarshan Sharma, Biswadeep Chakraborty, and Saibal Mukhopadhyay. Online relational inference for evolving multi-agent interacting systems. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[48] James Kennedy and Russell Eberhart. Particle swarm optimization. In Proceedings of CENN'955inneeratiin conference neural nelworks, volume 4, pages 1942-1948. ieee, 1995.
[49] Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate--sarch-peee -predict: Composing retrieval and language models for knowledge-intensive NLP. arXiv preprint arXiv:22 12. 2022.
[50] Omar Khattab, Arnay Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-improving pipelines. 2024.
[51] Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, and Yulia Tsvetkov. Language generation models can cause harm: So what can we do about it? an actionable survey. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3229-3321, 2023.
13

---

[52] LangGraph. Langgraph, 2024. URL https://langchain- www.aaaa..ggiiiii.i.. waabbb.oo/aagggee /ttbuub..///agggapp//
[53] Bin Lei, Yi Zhang, Shan Zuo, Ali Payani, and Caiwen Ding. Maem: Utilizing a multi-agent system for condition mining in solving complex mathematical problems. In The ehirt Annual Conference on Neural Information Processing Systems, 2024.
[54] Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch- merge: Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208. 033006 2022.
[55] Xinran Li, Ling Pan, and Jun Zhang. Kalcidoscope: Learnable masks for heterogeneous multi- agent reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[56] Yang Li, Wenhao Zhang, Jianhong Wang, Shao Zhang, Yali Du Ying Wen, and Wei Pan. Aligning individual and collective objectives in multi-agent cooperation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[57] Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujuu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.0 19118, 2023.
[58] Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A Smith. Tuning language models by proxy. In First Conference on Language Modeling, 2024.
[59] Jie Liu, Pan Zhou, Yingjun Du, Ah-Hwee Tan, Cees GM Snoek, Jan-Jakob Sonke, and Efstratios Gavves. Capo: Cooperative plan optimization for efficient embodied multi-agent cooperation. arXiv preprint arXiv:2411.0467, 2024.
[60] Wei Liu, Chenxi Wang, YiFei Wang, Zihao Xic, Rennai Qiu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, and Chen Qian. Autonomous agents for collaborative task under information asymmetry. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[61] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms as agents. In The Twelfth International Conference on Learning Representations, 2024.
[62] Yexiang Liu, Jie Cao, Zekun Li, Ran He, and Tieniu Tan. Breaking mental improve reasoning through diverse multi-agent debate. In The Thirleenth International Conference on Learning Representations, 2025.
[63] Zhihan Liu, Hao Hu, Shenao Zhang, Hongyi Guo, Shuqi Ke, Boyi Liu, and Zhaoran Wang. Reason for future, act for now: A principled architecture for autonomous 11m agents. In Forty-first International Conference Machine Learning, 2024.
[64] Yougang Lyu, Lingyong Yan, Zihan Wang, Dawei Yin, Pengjie Ren, Maarten de Rijke, and Zhaochun Ren. Macpo: Weak-to-strong alignment via multi-agent contrastive preference optimization. arXiv preprint arXiv:2410.07672. 2024.
[65] Hao Ma. Tianyi Hu, Zhiqiang Pu, Liu Boyin, Xiaolin Ai, Yanyan Liang, and Min Chen. Coevolving with the other you: Fine-tuning Ilm with sequential cooperative multi-agent reinforcement learning. Advances Neural Information Processing Systems, 17:15497-15525, 2024.
[66] Costas Mavromatis, Petros Karypis, and George Karypis. Pack of Ums: Model fusion at test-time via perplexity optimization. arXiv preprint arXiv:2404. 11531, 2024.
[67] Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. The Twelfth International Conference on Learning Representations, 2024.
14

---

[68] Sumeet Motwani, Mikhail Baranchuk, Martin Strohmeier, Vijay Bolina, Philip Torr, Lewis Hammond, and Christian Schroeder de Witt. Secret collusion among ai agents: Multi-agent deception via steganography. Advances in Neural Information Processing Systems, 37:73439- 73486, 2024.
[69] Boye Niu, Yiliao Song, Kai Lian, Yifan Shen, Yu Yao, Kun Zhang, and Tongliang Liu. Flow: Modularized agentic workflow automation. In The Thirteenth International Conference on Learning Representations, 2025.
[70] Jing-Cheng Pang, Si-Hang Yang, Kaiyuan Li, Jiaji Zhang, Xiong-Hui Chen, Nan Tang, and Yang Yu. Kalm: Knowledgeable agents by offline reinforcement learning from large language model rollouts. Advances in Neural Information Processing Systems, 37: 37:126620-126635 2024.
[71] Feng Peiyuan, Yichen He, Guanhua Huang, Yuan Lin, Hanchong Zhang, Yuchen Zhang, and Hang Li. Agile: A novel reinforcement learning framework of Ilm agents. Advances in Neural Information Processing Systems, 37:5244-522 5284, 2024.
[72] Giorgio Piatti, Zhijing Jin, Max Kleiman Kleiman-Weiner, Bernhard Schölkopf, Mrinmaya Sachan, and Rada Mihalcea. Cooperate or collapse: Emergence of sustainable cooperation society of Ilm agents. Advances in Neural Information Processing Systems, 11111111111799 2024.
[73] Chen Qian, Zihao Xie, Yifei Wang. Wei Liu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang. Zhiyuan Liu, and Maosong Sun. Scaling gagge-laggage-modllbbieeee multi-agent collaboration. arXiv preprint arXiv:2406.0715. 2024.
[74] Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Benchmarking agentic workflow generation. arXiv preprint arXiv:2410.0788 07869, 2024.
[75] Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. Advances in Neural Information Processing Systems, 37:55249-55285, 2024.
[76] Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. Normad: A benchmark for measuring the cultural adaptability of large language models. arXiv preprint arXiv:2404. 12464, 2024.
[77] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations, 2024.
[78] Yu Shang, Yu Li, Keyu Zhao, Likai Ma, Jiahe Liu, Fengli Xu, and Yong Li. Agentsquare: Automatic Ilm agent search in modular design space. arXiv preprint arXiv:2410.06/53, 2024.
[79] Zejiang Shen, Hunter Lang, Bailin Wang, Yoon Kim, and David Sontag. Learning to decode collaboratively with multiple language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024.
[80] Chenglei Si, Weijia Shi, Chen Zhao, Luke Zettlemoyer, and Jordan Boyd-Graber. Getting more out of mixture of language model reasoning experts. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages
2023.
[81] Andries Petrus Smit, Nathan Grinsztajn, Paul Duckworth, Thomas D Barrett, and Arnu Pretorius. Should we be going mad? a look at multi-agent debate strategies for 1lms. In Forty-first International Conference on Machine Learning, 2024.
[82] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 1lm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.032: 033144 2024.
15

---

[83] Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye. Kyle Mahowald, and Greg Durrett. To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning. arXiv preprint arXiv:2409. 283,, 2024.
[84] Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ank. Learn-by- interact: A data-centric framework for self-adaptive agents in realistic environments. arXiv preprint arXiv:2501. 10893, 2025.
[85] Vighnesh Subramaniam, Yilun Du Joshua B Tenenbaum, Antomio Torralba, Shuang Li, and Igor Mordatch. Multiagent finetuning: Self improvement with diverse reasoning chains. arXiv preprint arXiv:2501.0570; 2025.
[86] Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. Magis: LIm-based multi-agent framework for github issue resolution. Advances in Neural Information Processing Systems, 37:51963- 2024.
[87] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
[88] Xingchen Wan, Ruoxi Sun, Hootan Nakhost, and Sercan O Arik. Teach better or show smarter? instructions and exemplars in automatic prompt optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[89] Heng Wang. Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language? Advances in Neural Informa- tion Processing Systems, 36, 2024.
[90] Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities. arXiv preprint arXiv:2406. 04692, 2024.
[91] Junyang Wang. Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-Ve v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[92] Xihuai Wang. Shao Zhang, Wenhao Zhang, Wentao Dong, Jingxiao Chen, Ying Wen, and Weinan Zhang. Zsc-eval: An evaluation toolkit and benchmark for multi-agent zero-shot coordination. Advances Neural Information Processing Systems, 37:47344 47377, 2024.
[93] Xingyao Wang. Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better Ilm agents. In Forty-first International Conference Machine Learning, 2024.
[94] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint 2xXiv:2406.01574 2024.
[95] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[96] Muning Wen, Ziyu Wan, Jun Wang, Weinan Zhang, and Ying Wen. Reinforcing I1m agents via policy optimization with action decomposition. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[97] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. International conference on machine learning, pages 29965-23998, PMLR, 2022.
16

---

[98] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang. Shaokun Zhang, Jiale Liu, al. Autogen: Enabling next-gen Ilm applications via multi-agent conversation. In ICLR 2024 Workshop Large Language Model (LJM) Agents, 2024.
[99] Shirley Wu, Shiyu Zhao, Qian Huang, Kexin Huang, Michihiro Yasunaga, Kaidi Cao, Vassilis Ioannidis, Karthik Subbian, Jure Leskovec, and James Zou. Avatar: Optimizing Ilm agents for tool usage via contrastive reasoning. Advances in Neural Information Processing Systems, 37:25981 26010, 2024.
[100] Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408. 00724, 2024.
[101] Zelai Xu, Chao Yu, Fei Fang. Yu Wang, and Yi Wu. Language agents with reinforcement learning for strategic play in the werewolf game. In Forty-first International Conference on Machine Learning, 2024.
[102] Prateek Yadav, Colin Raffel, Mohammed Mugeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on model moerging: Recycling and routing among specialized experts for collaborative learning, arXiv preprint arXiv:2408. 07057, 2024.
[103] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024.
[104] Wenkai Yang, Xiaohan Bi, Yankai Lin, Sishuo Chen, Jie Zhou, and Xu Sun. Watch for your agents! investigating backdoor threats to Ilm-based agents. Advances in Neural Information Processing Systems, 97:100938-100964 2024.
[105] Fan Yao, Yuwei Cheng, Ermin Wei, and Haifeng Xu. Single-agent poisoning attacks suf- fice to ruin multi-agent learning. In The Thirteenth International Conference on Learning Representations, 2025.
[106] Jihan Yao, Wenxuan Ding, Shangbin Feng, Lucy Lu Wang, and Yulia Tsvetkov. Varying shades of wrong: Aligning llms with wrong answers only. arXiv preprint 2rXiv:2410.11055. 2024.
[107] Xinjie Yao, Yu Wang, Pengfei Zhu, Wanyu Lin, Jialu Li, Weihao Li, and Qinghua Hu. So- cialized learning: making each other better through multi-agent collaboration. In Forty-first International Conference on Machine Learning, 2024.
[108] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang. and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024.
[109] Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. Cut the crap: An economical communication pipeline for based multi-agent systems. arXiv preprint arXiw:2410002506 2024.
[110] Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, and Dawei Cheng. G-designer: Architecting multi-agent communication topologies via graph neural networks. arXiv preprint arXiv:2410.11782.. 2024.
[111] Hangfan Zhang, Zhiyao Cui, Qiaosheng Zhang, and Shuyue Hu. Multi-llm-agents debate- performance, efficiency, and scaling challenges. In The Fourth Blogpost Track at ICLR 2025, 2025.
[112] Hongxin Zhang, Zeyuan Wang, Qiushi Lyu, Zheyuan Zhang, Sunli Chen, Tianmin Shu, Behzad Dariush, Kwonjoon Lee, Yilun Du, and Chuang Gan. Combo: compositional world models for embodied multi-agent cooperation. arXiv preprint
2024.
17

---

[113] Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge. Xin Cheng, Sirui Hong. Jinlin Wang, al. Aflow: Automating agentic workflow generation. arXiv preprint arXiv:2410. 10762, 2024.
[114] Shaokun Zhang, Jieyu Zhang, Jiale Liu, Linxin Song, Chi Wang, Ranjay Krishna, and Qingyun Wu. Offline training of language model agents with functions as learnable weights. In Forty-first International Conference Machine Learning, 2024.
[115] Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Arik. Chain of agents: Large language models collaborating on long-context tasks. Advances in Neural Information Processing Systems,
2024.
[116] Qinlin Zhao, Jindong Wang, Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, and Xing Xie. Competeai: Understanding the competition dynamics of large language model-based agents. In Forty-first International Conference on Machine Learning, 2024.
[117] Yangheng Zhao, Zhen Xiang, Sheng Yin, Xianghe Pang, Siheng Chen, and Yanfeng Wang. Malicious agent detection for robust multi-agent collaborative perception. arXiv preprint arXiv:2310. 19901, 2023.
[118] Hang Zhou, Ychui Tang, Haochen Qin, Yujie Yang. Renren Jin, Deyi Xiong, Kai Han, and Yunhe Wang. Star-agents: Automatic data optimization with Ilm agents for instruction tuning. Advances in Neural Information Processing Systems, 37:4575 2024.
[119] Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang. Shuyi Guo, Zhe Wang, Zhen- hailong Wang, Cheng Qian, Xiangru Tang. Heng Ji, et Multiagentbench: Evaluating the collaboration and competition of Imm agents. arXiv preprint 25X3::550300133 2025.
[120] Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. GPTSwarm: Language agents as optimizable graphs. In Proceedings of the st International Conference on Machine Learning, 2024.
18

---

NeurIPS Paper Checklist
Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims in the abstract and introduction are supported by the experiments and results in Sections and 6.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to vi- olations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope the claims made, e-g., if the approach was only tested on a few datasets with a few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by review- ers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
19

---

Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 4 and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
example, the contribution
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be a way access this model for reproducing the results or way reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We used open-access and publicly available data (Section D), code will be made publicly available upon acceptance.
20

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
PuiidssCodeSuube ssicnpolcyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //mpps.///i..... gttp///ddee/ae// https///oooSeeeeeseeeeee ssi.ssolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Section 4 and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Table 8.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not show tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
21

---

If error bars are reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [Yes]
Justification: Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, every respect, with the NeurIPS Code of Ethics https: /ttp:::////urrpss. httppublic//thhicuuuueellll..aad
Answer: [Yes]
Justification:
We discuss the ethical considerations Appendix
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
22

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
We use open data and models already publicly available.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or -ual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer:
[Yes]
Justification: We use open data and models allowed for academic research.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cwm/ conddataeess has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: No new asset introduced in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
23

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or human subjects in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: No human subjects in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage LLMSs if it is an important, original, or non- standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
LLMs were not used the writing of this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ http:::///euuiip... http:///c.eceeeeeeeeeee/e/////// /LLM) for what should or should not be described.
24

---

A Limitations
HETEROGENEOUS SWARMS by default operates with a pool of LLMs sharing the same model architecture: this is to ensure that models share the same parameter space for weight-step optimiza- tion. the pool of LLMs contain different architectures, then HETEROGENEOUS SWARMS could either operate with only the role-step to optimize the multi-LLM network, or employ token-level collaboration for the weight step similar to Liu et al. [58] and Feng et al. [26].
Both the optimization and inference process of HETEROGENEOU'S SWARMS features calling multiple L.MMs, increasing its potential cost. We propose various strategies to alleviate this: by encouraging sparsity in the multi-LLM network in Table 3, by analyzing the time complexity in Section C, and by employing Dropout- W/R strategies in Figure 9, we show that HETEROGENEOUS SWARMS could speed up with one or multiple strategies used in conjunction.
While we conduct experiments where all the nodes in the network are neural language models, we highlight several other possibilities we weren't I able to evaluate: employing black-box models in the LLM pool. which compatible if we skip these models the weight-step; employing tools and APIs as nodes in more agentic setups [59, 28, 39, 78, 73, 113, 69, 74, 20, 105, 90, 112, 64, 22, 8, 111, 85, 62, 15, 41], as long as they define an ontutt relationship.
We envision two extra steps to take to further expand the expressive power of HETEROGENFOUS SWARMS: allowing for cycles in the multi-LLM network in case one model needs to be called multiple times, together with an exit function to decide when to pull out from the cycle; allowing for instance-level changes the multi-LLM network with learnable edit function.
B Ethics Statement
Since multiple LLMs are called in topological order of the multi-LLM system, we envision potential risks when one or several of the component LLMs are unintentionally or intentionally compromised. For example, if one of the LLMs were to exhibit substantial social biases [51] or malicious intent [117], it might pass on to future models in the network and result in cascading effects. There are further risks agentic [34, 104, 96, 72, 18. 31, 70, 115, 10, 12, 14, 38, 42, 63, 81, 93, 101, 107, 114, 84, 2] versions of HETEROGENEOUS SWARMS where one model could be biased or malicious. We thus argue that HETEROGENEOUS SWARMS should be initialized with carefully selected pool of specialized LLMs and envision future work on identifying harmful//malicouus components in multi-LLM collaboration.
C Analysis (cont.)
Scaling Multi-LLM Collaboration Recent research explored the inference-time scaling of a single, often gargantuan, LLM [82, 5, 100). However, it is unclear whether inference-time scaling is possible for small language models, while we believe HETEROGENEOUS SWARMS offers a way of scaling through topological message passing and collaborative generation. We conduct scaling study by employing 2, 4, 6, 8, and 10 (default) of initial LLM checkpoints as the starting model swarn and run HETEROGENEOUS S SWARMS: results in Figure 6 demonstrates that HETEROGENEOUS SWARMS indeed offers an inference-time scaling paradigm for smaller language models through multi-LLM collaboration.
Time Complexity Given a pool of n LLMss, we employ swarm of N graphs and A model assignment instances, We measure the complexity by the number of model in- ferences as they are the main computational cost. At opti- mization time, the role step calls model inference O(nN) times and the weight step (n.M) times, so the overall op- timization cost per iteration is (n + If where If] denotes the adaptation data size, linear to n, N, and M. inference time, the cost is ((n) where the adapted LLMSs are called in topological order of the network. We further
0.4
Figure 6: Scaling the number of LLM experts from 2 to 10 results in consistent improvements across 4 datasets.
illustrate the empirical complexity by plotting the time per
25

---

12
10
# Hours
2
4
10
GPUs
Figure 7: Optimization time changes with the number of A100 40GB GPUs employed. We employ 1, 5, and 10 GPUs since they could be divided by 10, the amount of LLMs.
HS(C-10)
HS12-10)
Best Single
0.6
0.4
0.2
0.0
K-Cresa
COM2
NLGroph
AB-ig
Figure 8: The collaboration of the all-but-top-1 and bottom-hall LLMs through HETEROGENEOUS SWARMS could beat the top-1 individual LLM.
iteration given different amount of A100 40GB GPUs in Figure We present the optimization and inference cost of approaches in Table C: HETEROGENEOUS SWARMS is more expensive at optimiza- tion time and on-par with most baselines at inference time, while the performance improvements and collaborative gains justify the additional optimization cost.
HS(weak)) > strong successful collaboration of multi- ple weak models can beat stronger LLM: we withhold the top-1 LLM in the given task and discover a multi-LLM sys- tem out of the remaining models with HETEROGENEOUS SWARMS (HS(2-10)). Additionally, we also withhold the top half and evaluate the collaboration of the bottom half (HS(6-10)). Figure 8 demonstrates that HETEROGENEOUS SWARMS enables the collaboration of weaker models outperform the top-1 individual L.M,, enabling the weak- to-strong transition through collaborative generation.
Approach
optimization
inference
single
pred. merge
static weight
dynamic weight static role dynumic role ours
1
π
ON n(N+M)
I
Table 4: Optimization and inference cost of different types of approaches. is the number models, is the number of graphs, and M is the number of graph- model assignments.
Accelerating with Dropout-R/W To further speedup the optimization process, we process to randomly skip the role-step or weight-step with d, % or d % likelihood. We illustrate the performance with dr, dm. {0.2, 0.5, 0.8) in Figure 9. It is illustrated that the acceleration only comes with minor performance drop, showing the flexibility of HETEROGENEOUS SWARMS to adapt to different computational budget.
Another LLM We by default employ GEMMA-7B as the component LLMs. We additionally employ MISTRAL--B (Imistralai/Mistralll) 133-Insrruct-vO.. [46] for HETEROGENEOUS SWARMS and present performance in Table 5. Results show that HETEROGENEOUS SWARMS is compatible with MISTRAL-77 7B too, outperforming the best single Mistral-based LLM.
26

---

K-Creea
0.6
0.5
0.4
0.3
0.2
Fal
DeepR-02 22
Drepgas 5.5
DiepR-)
Dropi-822 22
Dropw-0.5
Dropt-988
Figure 9: By randomly skipping the role-step and weight-step through Drop-R and Drop-W, W, we could speed up the optimization process.
single
CoT
GoT
Ours
0.6
0.4
K-Cross
GSMBK
NLGraph
AbstainQA
Figure 11: Our approaches outperform a single model empowered with reasoning enhancement
approaches.
Comparison with a Larger Model We com-
Sotting
MMLU-pro
K-Cross
GSM8k
NLGraph
AbstainQA
pare the collaboration of gemma-2-9h models against single gemmaa--2274 model, as well the collaboration of gemma-3-4b models against a single gemma-3-2777 3-b7b model in Figure 10. It is demonstrated that through HETEROGE-
Best Single
0,146
0.364
0.303
0.325 0.505
0.081
Ours
0.212
0.477
0.313
0.123
Table 5: Performance with 10 Mistral LMs.
NEOUS SWARMS, the multi-LLM systems of smaller models could outperform larger models.
NLGraph
Comparison with Enhanced Reasoning Approaches We compare our approach against chain-of-thought [95] and graph-of-thought [3] with the best initial LLM expert Figure 1.. The performance improvements suggest that by having multiple models collaborate, their expertise could complement each other and achieve more.
0.5
Generalization Unseen Tasks We optimize multi- LLM systems based on the Knowledge Crosswords dataset and evaluate them on WikiDYK [?], two datasets in the en- cyclopedic knowledge domain, in Table C. Results demon- strate that our approach better generalizes to unseen tasks.
Figure 10: Multi-LLMM systems of mul- tiple smaller models could outperform a single larger model, with similar amounts of total parameters.
Collaborative Gains of Baselines We compare the col- laborative gains of our approach with two model merging baselines Figure
Approach
K-Cross
WikiDYK
greedy soup
0.355
0.525
demonstrated that HETERO- GENEOUS S SWARMS does yield higher collaboration gains the optimized multi-LLM systems.
pack of llms
0.352
0.513
lorahub
0.291
0.501
model swarms
0.428
0.527
gpa-swarm meta-agent
0.320
0.472
0.276
0.475
Collaboration Different Model Sizes
We run the role-step of HETEROGENEOUS SWARMS with different
agent-prune
0.321
0.431
gnns ours
0.339
0.365
0.450
0.566
27
Table 6: Generalization unseen tasks: optimizing on K-Cross and evaluating the held-out task of WikiDYK.

---

ouPs pred. ensemble
uniform soup
0.5
0.4
0.2
abkg
com2
k-cross
nigraph
Figure 12: HETEROGENEOUS SWARMS achieves higher collaborative gains compared to baselines.
Setting
K-Cross
GSM8k
NLGraph
AbstainQA
full 7b
0.450
0.481
0.660
0.220
half 7b half 2b
0.318
0.371
0.492
0,094
full 2b
0,265
0,215
0.392
0.082
single 7b
0.346
0.237
0.535
0.065
single 2b
0.194
0.148
0.364
0.023
Table 7: Collaboration of mixed-size models, compared to fixed-size and best single models.
model size settings, with Gemma-7b 7b and Gemma 2b mod-
els. I) full 7b (default). 2) half 7b half 2b, 3) full 2b, 4) best single 7b, 5) best single 2b. We present results in Table C: half 7b half 2b outperforms both best single 7b and full 2b indicates that mixed-size collaboration works under HETEROGENEOUS SWARMS as well.
D Experiment Details
Dataset Details We employ 12 datasets to evaluate HETEROGENEOUS SWARMS and base- lines spanning knowledge, reasoning, agent, and miscellaneous capabilities. We filter examples in GAIA to retain examples where the human- provided tool use contexts could support the fi- nal answer for GAIA-text. We incorporate each tool use context for the input of one LLM in the multi-LLM network respectively. We trun- cate the context in Qasper in ten-fold and in- corporate each chunk as context for an indi- vidual LLM. We by default sample 200 exam- ples for optimization and 1,000 for evaluation, while downsampling if there's 's enough data. We present data statisties in Table 8. MMLU- pro, Knowledge Crosswords, COM2, Normad, AgentBench-KG, AbstainQA, and WoW are evaluated in multiple-choice settings. GSM8K, NLGraph, and GAIA-text are evaluated via ex- match. AgentBench-LTP and Qasper are evaluated by Gemini for answer similarity on scale of to 10, normalized employ the z-test with the one-ta hypothesis and present statistical significance test results on the datasets.
Size
Dataset
Source
dev
test
MMLU-pro***
Wang et al, [94]
70
1000
K- k-Crosswords***
Ding et al. [19]
200
1000
20M2ss
Fang et al. [23]
317
1000
GSM8k
Cobbe et al. [13]
200
1000
NLGraph
Wang et al. [89]
200
1000
Rao et al. [76]
500
2000
GALA-text
Mialon al. [67]
13
28
AgemtBench-KC
al. [61]
50
AgemtBench--ITP
Liu et al. [61]
20
50
Qasper
Dasigi et al. [16]
100
100
AbstainQA*** WoW
Feng al. [25]
200
Yao et al. [106]
200
1000
Table 8: Statistics of employed datasets. * ** and #** indicates the improvement on this dataset is statistically significant with p 0.1, p 0.05. and < 0.01 with one-tailed z-test.
to We also
28

---

Implementation Details We employ the 10 LLM experts in Feng et al. [26] as the initial swarm of models. We employ the Gemma-basel [29] versions for experiments in the main pa- per for a fair comparison and employ the Mistral-based [46] versions for experiments in Ta- ble 5. Aside from the hyperparmaters specified in Section 4, we run grid search over other hy- perparameters and report the best-found expert based on utility function f. Specifically, &. (0.1, 0.3), og E (0.1, 0.2, 0.3, 0.4, 0.5), Og E {0.2, 0.3, 0.4, 0.5, 0.63. &w E {0.01, 0.05, 0.1]. E {0.5, 0.6, 0.7, 0.8, 0.9, 1.0). We run up to 50 to 200 runs by randomly choosing over these hyperparameter search settings and report the best-found expert on utility function f. Experiments performed on cluster with 16 A100 GPUs each with 40 GB memory.
For best single, prediction merge, data merge, uniform soup, dare-ties, greedy soup, pack of LLMs, lorahub, and model swarms, follow the settings in Feng et al. [26]. For chain, we organize the 10 initial LLMs into a chain, randomly permute their assignments for 20 times, and report the best-found chain on f. For star, organize them into 1-8-1 structure, randomly permute their assignments for 20 times, and report the best-found star on f. For gpt-swarm, employ its method optimize the network structure and do not consider prompt optimization for a fair comparison. To make the optimization compatible to our non-differentiable setting, we modify it into interpolation of adjacency matrices based on utility values. For mela-agent, employ Gemini-pro (gemini-pro- (o0o--.5-0/)) with the original prompt to iteratively suggest multi-LLLM structures, while representing each individual LLM by natural language description of their training data and expertise. For agent-prume, we randomly sample multi-LLM structures, conduct pruning and re-evaluate, repeat this process for 20 times and report the best-found structure on f across runs and pruning degrees. For GNNs, we employ graph attention networks to encode the multi-LLM network with randomly initialized node features, and learn a link prediction task based on whether adding this edge between two LLMs would lead to a performance increase on f or not. At inference-tinge the GNN conducts link prediction for
Baseline Details
edges obtain multi -iMiii-LLMM- network.
Prompts We present the prompts employed in various contexts in Tables 9, 10, and 11. We make dataset-specific changes the general prompt format if necessary.
Qualitative Examples We present two working examples of the multi-LLM systems in HETERO- GENEOUS SWARMS, on Knowledge Crosswords [19] and NLGraph [89] datasets in Figures 13 and 14. These examples demonstrate that HetrRogeNeoUS SWARMS discovers multi-LLLM systems to adapt the task and play heterogeneous roles through collaborative generation.
Illustrating the Optimization We illustrate the changes in the best f utility values for the swarm of graphs and swarm of LLMs in Figure 15: it illustrated that there is co-improvement of roles and weights thanks to the alternating optimization algorithm.
29

---

prompt for the first/entry-point LLM "Please answer the following question." "
prompt for the middle//iterrmeddate LLMs
"Please answer the following question with the help of previous responses, feel free to ignore wrong or unhelpful responses."
> prompt for the final/end-point LLM
"Please answer the following question with the help of previous responses, feel free to ignore wrong or unhelpful responses. Make sure to provide final and definitive answer."
Table Default LLM instructions the multi-LLMM system.
"Please evaluate how similar the following response to the ground truth answer. Please rate the response on a scale of to 10, where is the worst and 10 is the best. Please respond with Rating: 2/10 first and then provide your reason.
Ground truth: [ground truth] Generated response: [generated response]"
Table 10: Default Gemini-as-an as-j-ud evaluation prompt.
"Below is the output of model solving part of problem. Please judge whether the output is one of the four scenarios: 1. solving part of the problem, 2. refining the previous answer, 3. providing feedback, 4. irrelevant.
The problem is: [problem text]
The output is: fintermediate output]"
Table 11: Prompt for Gemini to evaluate the roles of intermediate model outputs in multi-LLM systems.
30

---

Algorithm 4: DAG Decoder (G-decode)
Input:
continuous adjacency matrix A E R where aij denotes the likelihood of a directed edge from model x; to model x; edges G = Ø, remaining node set R (1, , . existing node set 3 = Ø
select end node based on inverse out degrees top sapp--aamlingg (1/ Σj=1 Oij dij)=m) R. R.remove(k), (kadd(k) while R * do
select remaining node based on out degrees u = top-p-sampling (atpssaaaping(aaaa (ej (ER
exp(avn) Gaadd((u - v)) with prob. Eies exp(aus))
Vu E 3
R.remove(u), (.)
end
return directed acyclic graph G, defines onput-output mapping based on {x+)==
Universtty
Ribbert Harnecy
(question roxtj= Entily I groduoted untity Robern Kernady grodunted from entrty Daniel Gorennein groducted from entity 2. Work out entity 3 and
University,
booktoon
pedtend from
genwrited
Stall
Turing sward
Daniel Gorendlein
Paane solve part the tctbwingg problem and colegate rest fature aleps.
input
ddogget the rost fularo stips.
(Tosatton Sedj
"Biep ideecity conditatess of entiity Turing John Nesh, Androw and Dana Soot..
Piea
Please sake part of the fallwingg problem leggee the futur sheps cardiiaates Turing who also graduated from Jone Naat. (eusstion
Morvin Mirsky
Turtig Award
Princation Lintvrriyy
Please work out the possible encyy 1 and delogote
stope.
2 Idennily didanco entty And You, Dana Boot.
Please ie part of tallowing problem and Posoibio ond 1:
Ploase
fol
Prenkous resprnee "Mole identiyy who also griduated from Join Nash, Androw You, and Dane Bood..
delegeis the neat kitur cieps
Please sohre part the falloing
proticm and delegare the rest to futuro
Souestio
Fresious reeporas
Prescreus meeporse 2o
Previouss resporse Si...
Please we the following Pherous requonee I: Eniiyy whoald be Neat, Dans Seatt Mrevin Minaly.
Mareen Mamsdy
Prencate by
Turnng Award
Kennedy
Androw You det not graduste from Prinaaion
Univeriity staps shoulld sonaitae the John
Dana
Exilly sehoold Nash, Dana
Please work our the possiile entiy andd and Jaueshon
Robert F Konnody de and Doniel Universtyy amould Hannord University.
Please scive part of the fot owing prosiamn and de delaggat the
resporse Turing Rward
should
Syuestion Hevd)
Future consider which option grodusted from Homord Unkveniity
Princeion Univers by ar wiongg Fukin elepe
ahould conoader Conematann
Turing Award Aeciliiiall Insul ensollgence.
fulure slops.
owing
based provius Preecus respores Tr... Freveum reaporss
provide the
folloiing problem
Provfous resporse T....
Predltse respores cornect
Enity 1is Mancin nally, Entty Harwar University
output
Figure 13: Working example one of HETEROGENFOUS SWARMS.
31

---

Cof Co
Please scloe part falluinng prbble
generated
node wec seciones shomadd par benween
Stail annwver
mode torm mode rade sitt weght inmmridd
worh
wight demene
mode weight langh the pacmation lux)
the focoro daipe
input
Plecse par falowiing poblee and
Product resporioo Tho shonnss pars cataeee
Co lodi wuth un gir mode to
node node with weyh anditom T12 mode wiggh
Foom sole Tos sode Euih ueig 1, bode with wiggt The of ahorhess Sovestonruu!
veiigin frvt iado drr with weth
Plaas aahve clibe bolowing prodier -Lau our respossa fo... jeittegg
Pro reporiee ia lscirecc Theie is ndge edge with weigh 1. Irom net la mode withs.
lguile the not kutan sapp. (baralics land
The part iyprovious supposss EEvalE
sohss part oheee telowing procles
Plevouue respres
detepala the rest fator (barddio hol
Henrs pevers conned rode
la with
toroue
home
Thene rode trroeh woigt wnag
koliuus stiee sur))
Frr mecevoo nois tonoee with hooe
lebweing priblem
Isumnaaree tortor paw ropossos
delegiaa bikur
shortett path Foomnods sode sith weight..
Hrrvens repporee
pre our reaprves shockee
Thillngt of pobite
Pleaae par falowing publim Callggr paett
Foom Eins wode with in nols wiight
The angth teterterdda.
stortal dioter repoome
output
116
Figure 14: Working example two HETEROGENEOUS SWARMS.
32

---

graph_g
model_g g
0.42
0.38
0.4
0.37
0.36
0.38
Step
10
15
graph_g
model_
0.54
0.515
0.58
0.505
Stop
0.54
0
graph
model
0.25
0.3
0.22
0.25
00
Step
Figure 15: Illustrating how the utility of the role-step (left) and weight-step (right) changes through time three datasets, Knowledge Crosswords, NLGraph, and AB-kg. We observe co-improvement in the utility of graphs and model weights.
33


Reviews:
[{"review_id": "lVwZS2BcUT", "reviewer": "Reviewer_y4wm", "summary": "This paper introduces Heterogeneous Swarms, an algorithm that automatically designs multi-LLM systems by jointly optimizing two key dimensions: (i) model roles, represented as a directed acyclic graph (DAG) defining inter-model communication, and (ii) model weights, quantifying each LLM’s contribution. The method alternates between a role-step (discovering the optimal DAG via particle swarm optimization over continuous adjacency matrices) and a weight-step (estimating individual model contributions via a proposed JFK-score and optimizing weights accordingly). The approach is tested on 12 diverse tasks, outperforming 17 strong baselines with an 18.5% average improvement, and demonstrating consistent collaborative gains from heterogeneous multi-LLM systems.", "questions": "1. Could the authors clarify how G-DECODE ensures DAG validity in extreme cases where adjacency probabilities are nearly uniform? Does this lead to highly variable graph structures between runs?\n\n2. How does the method perform if the initial pool of LLMs is not diverse (e.g., identical models)? Could the authors quantify the dependency on model diversity in more detail?\n\n3. What happens when the utility function f is noisy or discontinuous? Does PSO still converge reliably?\n\n4. Could the authors discuss scalability: what are the computational and memory costs when the LLM pool size grows beyond 10 or 20 experts?\n\n5. Are there practical strategies (e.g., early stopping heuristics) to reduce the potentially high inference cost during optimization without losing significant performance?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths*\n\n1. The paper is well-written with clear explanations of complex components (e.g., G-DECODE, JFK-score). Figures (e.g., Figure 2) are illustrative and help convey the method’s workflow.\n\n2. The framing of roles as learnable DAG structures optimized with swarm intelligence is novel.  Combining both roles and weights in a unified optimization loop is a fresh contribution compared to prior fixed-role or fixed-weight strategies.\n\n3. The paper advances the field of multi-LLM collaboration by addressing both flexible role discovery and weight adaptation—two aspects often studied separately. The empirical gains suggest substantial practical impact.\n\n*Weaknesses*\n\n1.  The method’s reliance on repeated LLM inference during optimization could make it impractical for very large models without distributed compute.\n\n2. The paper builds on particle swarm optimization in a relatively straightforward way; while well-applied, PSO itself is not a new method.\n\n3. A few sections, especially the PSO update rule and G-DECODE sampling process, could benefit from more intuitive examples or step-by-step illustrations.", "was_revised": true, "final_justification": "address most of my concerns", "rebuttal": "We would like to thank the reviewer for their thoughtful comments and feedback.\n\n> The method’s reliance on repeated LLM inference during optimization could make it impractical for very large models without distributed compute. Could the authors discuss scalability: what are the computational and memory costs when the LLM pool size grows beyond 10 or 20 experts? Are there practical strategies (e.g., early stopping heuristics) to reduce the potentially high inference cost during optimization without losing significant performance?\n\nWe discuss time/space complexity in Appendix C. Briefly summarize:\n\nGiven a pool of $n$ LLMs, we denote the data size behind a utility function as $|f|$ and run H-Swarm with $N$ graphs and $M$ model assignments. The optimization cost is then $O(n(N+M)|f|)$ and the inference cost $O(n)$. As $N$ and $M$ are hyperparameter constants and $|f|$ is a fixed dataset, both costs scale linearly with the LLM pool size $n$.\n\nAs for memory costs, you don’t need to load all $n$ models at once. At the bare minimum, you only need 1 GPU that could load 1 model, do its inference, and switch it out for the next model. At the maximum, you could use $n$ GPUs to load one model per GPU. Any amount of GPU/memory is workable through space-time tradeoff, and our implementation supports any (#model, #GPU) combinations.\n\nWe implement a quick early-stopping heuristic: for the k-th model’s response, we parse it to determine whether an answer is reached (“the answer is / the final answer is / the correct answer / the solution is / …”) and exit if found, stopping future models from further critiquing and refining the answer. This lowers inference cost from O(n) to O(k): we report performance and the average k across three tasks:\n\n|                | K-Cross | NLGraph | AbstainQA |\n|:--------------:|:-------:|:-------:|:---------:|\n| full inference |  0.450  |  0.660  |   0.220   |\n| early stopping |  0.437  |  0.591  |   0.205   |\n|     avg. k     |   6.38  |   4.53  |    7.22   |\n\nWe observe minor drops on K-Cross and AbstainQA with a 32% speedup, while on the reasoning task NLGraph performance dropped more with a higher 55% speedup. This suggests that different tasks might require different early stopping strategies, which we will further explore in the future.\n\n> Could the authors clarify how G-DECODE ensures DAG validity in extreme cases where adjacency probabilities are nearly uniform? Does this lead to highly variable graph structures between runs?\n\nNo matter what the adjacency probs are, we use top-p sampling to sample one node at a time and produce an (sampled_node -> existing_node_in_the_graph) edge, so it will always be acyclic regardless of underlying probabilities of selecting the node. If A is near uniform, there could be great randomness in graph decoding: to curb this you could use p=0 (greedy decoding) so that the produced graphs are deterministic. In practice, we find that optimized A is almost never near uniform and enabling this variation and randomness in graphs actually helps to sample and explore diverse model collaboration patterns.\n\n> How does the method perform if the initial pool of LLMs is not diverse (e.g., identical models)? Could the authors quantify the dependency on model diversity in more detail?\n\nFigure 5, leftmost, presents the setting where all 10 models are identical (i.e. the same model checkpoint, or “best single” in Table 1). We observe steady upward trends from this no-diversity setting to the most-diversity setting where the 10 models are all different: on K-Cross performance improved from 27.1 to 45.0 and on AbstainQA from 9.4 to 22.0, indicating major improvements thanks to the collaboration of diverse model checkpoints.\n\n> What happens when the utility function f is noisy or discontinuous? Does PSO still converge reliably?\n\nWe simulate noise in $f$ by randomly flipping 5% and 10% labels in the development set (used in $f$ and optimization) while keeping the test set intact.\n\n|         | 0% noise | 5% noise | 10% noise |\n|:-------:|:--------:|:--------:|:---------:|\n| K-Cross |   0.450  |   0.418  |   0.421   |\n|  Normad |   0.588  |   0.559  |   0.544   |\n\nThere is a slight drop from 0% to 5% and a negligible change from 5% to 10%, indicating that the PSO methodology is moderately robust to dataset noise."}, {"review_id": "qjeQtgVvss", "reviewer": "Reviewer_zL7f", "summary": "This paper focuses on multi-LLM systems and figures out the fixed-weight/fixed-role limitations in the existing methods. To overcome these limitations, the authors propose an algorithm named HETEROGENEOUS SWARMS to jointly optimizes the collaborative structure (a DAG) and individual model weights via Particle Swarm Optimization (PSO). The proposed method shows impressive outperformance over numerous baselines across 12 diverse tasks, with analysis confirming substantial collaborative gains. Though I concern about its generalizability and computation cost, this work presents a powerful framework for designing adaptive multi-LLM systems and marks a valuable contribution to the field.", "questions": "1. It is a little counter-intuitive for me that BEST SINGLE usually achieves better performance than PRED. MERGE as shown in Table 1. Can you briefly explain about it?\n2. Is the utility function optimized per task or not? This point is ambiguous in the paper.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "* Strength\n1. The paper formulate multi-LLM system problem-solving as jointly optimizing model roles (DAG) and weights. This novel modeling provides a structured alternative to prior methods which often treated these components separately or relied on manual heuristics, thereby creating a more comprehensive optimization target.\n2. The proposed algorithm offers a practical solution to a complex search problem by employing PSO combined with two specific components—G-decode for graph generation and the JFK-score for credit assignment. These methods successfully navigate the high-dimensional, non-differentiable search space of possible graph structures and model parameters.\n3. The authors conduct a broad set of experiments across 12 different tasks and compare their method against 17 baselines. The consistent performance improvements demonstrated the effectiveness of this algorithm. Furthermore, the analysis and ablation study is insightful and confirms the necessity of each design modules.\n\n* Weakness\n\nIMO the topological structures of the multiple LLMs are task-dependent, i.e., differ task by task.  While the appendix presents a single experiment on generalization (Table 6), this evidence is insufficient to fully support the claim of adaptability. The experiment shows generalization from one knowledge-based task (K-Cross) to another similar one (WikiDYK). This demonstrates in-domain generalization but does not address the more challenging case of out-of-domain generalization (e.g., a system optimized for reasoning being applied to an agentic task). The high degree of task-specific optimization inherent in the method raises concerns about how brittle these specialized systems might be when faced with novel problem types.", "was_revised": true, "final_justification": "The author's rebuttal clarifies some points and makes me understand this paper better. I believe my current score (4) has shown my positive feedback on your paper and will keep it.", "rebuttal": "We would like to thank the reviewer for their thoughtful comments and feedback.\n\n> IMO the topological structures of the multiple LLMs are task-dependent, i.e., differ task by task. While the appendix presents a single experiment on generalization (Table 6), this evidence is insufficient to fully support the claim of adaptability. The experiment shows generalization from one knowledge-based task (K-Cross) to another similar one (WikiDYK). This demonstrates in-domain generalization but does not address the more challenging case of out-of-domain generalization (e.g., a system optimized for reasoning being applied to an agentic task). The high degree of task-specific optimization inherent in the method raises concerns about how brittle these specialized systems might be when faced with novel problem types.\n\nYes, it is task-dependent: we believe there won’t be general-purpose model/agent collaboration structures that fit every task and context, so what we did is to provide an optimization algorithm to discover what works for each task. This is in line with general practice (*general* method to discover *task-specific* collaboration patterns). [1-2] We don’t think demanding a task-specific structure to work when out-of-domain is fair, as there won’t be a single one-size-fits-all structure.\n\nWhat might be possible is grouping tasks by domains (e.g. knowledge, reasoning, etc.) and hope to discover structures that work across this domain: we demonstrated that in the knowledge domain in Table 6.\n\n> It is a little counter-intuitive for me that BEST SINGLE usually achieves better performance than PRED. MERGE as shown in Table 1. Can you briefly explain about it?\n\nThis indicates that “truth is in the hands of the few”: because the models are diverse (exposure to different domains, tasks, etc.), without any adaptation only a few (or even just one in some cases) could solve a task-specific problem. The plurality vote of the many is then often incorrect compared to the model that best specializes in this task.\n\nThis highlights the importance of the weight-step: to prepare models for the current task in question and adapt their model weights and skill set to what would be helpful for this task and collaboration.\n\n> Is the utility function optimized per task or not? This point is ambiguous in the paper.\n\nYes, it is per task for a task adaptation setting. We will state this in Section 4, line 179.\n\n[1] Zhuge, Mingchen, et al. \"Gptswarm: Language agents as optimizable graphs.\" ICML.\n\n[2] LangGraph. https://www.langchain.com/langgraph"}, {"review_id": "Vz8S5F33jb", "reviewer": "Reviewer_wwh1", "summary": "This paper aims to use multiple LLMs to solve a task by organizing them as the nodes of a DAG and calling them in the topological order where the output of the parents of a node becomes part of that node’s input.\nThe paper employs the technique of Particle Swarm Optimization (PSO) to optimize the DAG topology and the LLMs’ weights for the given task.", "questions": "In addition to the questions in the Strengths And Weaknesses section:\n\nQ1. (Line 103) What's x_best? Is it the same as g (the global best)?\n\nQ2. Isn't that the DAG itself already has an assignment? If so, why do we “randomly select an LLM for each position” (Line 145)? \n\nQ3. How difficult is it to combine a prior dynamic weight technique vs a prior dynamic role?\n\nQ4. Are the optimization rounds (role-step and weight-step) performed independently for every dataset or datapoint in Table 1?\n\nQ5. Why can the proposed method alleviate prompt engineering issues? What prompts does the proposed method use? How about the baselines", "limitations": "Limitations are discussed in the appendix.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**     \nThe paper addresses a well-motivated problem that is worth more attention.\nIt is overall written with good clarity and easy to follow.\nThe empirical improvement seems significant.\nThe paper provides analysis on the learned roles. \n\n**Weaknesses**   \nW1. The G-decode algorithm is not well justified. In what sense does it use matrix $A$ as edge probabilities? What is top-p sampling and why is it used here? And there are DAG parameterization techniques that automatically ensure acyclicility:\nCharpentier, Bertrand, Simon Kibler, and Stephan Günnemann. \"Differentiable DAG Sampling.\" ICLR 2022.\n\nW2. JFK-score is inherently limiting because it averages out an individual LLM’s performance across many teams. In general, an LLM as a team member performs well only with certain teammates in a certain team. Ideally we would like to identify the best team as a whole. Such limitations are not discussed in the paper.\n\nW3. The innovation seems incremental from Model Swarms [26], which already used PSO for weight update. The paper does not clearly describe the difference from [26].\n\nW4. Using PSO for weight update is confusing.\nPSO tries to identify a single $x$, but ideally the multiple LLMs should have their own best weight. It is unmotivated to use PSO for the weight update.\nMoreover, PSO is only applicable when the LLMs have the same architecture, but in general an multi-LLM system involves LLMs of various kinds.\n\nW5. It is unclear how meaningful it is to compare with the chosen baselines.\nHow many baselines do multi-round optimization as the proposed method does? \nIf prior works have done either dynamic weight or dynamic role, can we do a baseline that does both by combining two prior works (e.g., Model Swarms + GPT-Swarm), which could serve as a stronger baseline to test the effectiveness of the proposed role-step/weight-step?", "was_revised": true, "final_justification": "Please see my comment.", "rebuttal": "We would like to thank the reviewer for their thoughtful comments and feedback.\n\n> The G-decode algorithm is not well justified. In what sense does it use matrix as edge probabilities? What is top-p sampling and why is it used here? And there are DAG parameterization techniques that automatically ensure acyclicility.\n\nFor a pool of n models, adjacency matrix A is size $n \\times n$. $a_{ij}$ denotes the likelihood of a directed edge from model i to j. (lines 121-122)\n\nTop-p sampling is a widely employed strategy for decoding, particularly in language models [1]. Instead of greedily selecting the $argmax$ element or sampling from the whole distribution, top-p sampling only considered the highest-prob candidates with a cumulative probability of p. This strikes a nice balance between randomness and quality, in our specific case it helps to decode multiple good-quality structures from A to better explore diverse collaboration patterns.\n\nThank you for the important reference. We are citing it and adding “other strategies such as Charpentier et al. could also produce DAGs from A” in the revised paper.\n\n> JFK-score is inherently limiting because it averages out an individual LLM’s performance across many teams. In general, an LLM as a team member performs well only with certain teammates in a certain team. Ideally we would like to identify the best team as a whole. Such limitations are not discussed in the paper.\n\nUsing the team analogy: the weight-step improves the individual while the role-step improves the team. By using JFK-score in the weight-step, models become versatile for any potential role in the given task. These models then form the best team in the role-step through graph optimization and orchestration. JFK-score fulfills this motivation by providing a signal of whether the model is generally helpful for a given task, not role-dependent or placement-dependent.\n\n> The innovation seems incremental from Model Swarms [26], which already used PSO for weight update. The paper does not clearly describe the difference from [26]. Using PSO for weight update is confusing. PSO tries to identify a single but ideally the multiple LLMs should have their own best weight. It is unmotivated to use PSO for the weight update. Moreover, PSO is only applicable when the LLMs have the same architecture, but in general an multi-LLM system involves LLMs of various kinds.\n\nWe summarize the key differences:\n\n|  Method    |                                       Optimization                                       |                 Inference                 |                                             Applications                                             |\n|:----:|:----------------------------------------------------------------------------------------:|:-----------------------------------------:|:----------------------------------------------------------------------------------------------------:|\n| [26] |                             weights of individual models only                            |               a single model              |                                tasks where a single model could tackle                               |\n| ours | different utility f for weight & new role-step that orchestrates model collaboration | a DAG of multiple models in collaboration | above, plus agentic and especially multi-agent scenarios where the collaboration is helpful/required |\n\nDifferent models actually have different $x$ because their movement is guided by different velocity $v$: if you look at the velocity update equation between line 95 and 96: signals $g$ and $g_w$ are shared across the swarm while $v_i$ and $p_i$ are model-specific (as evident by the subscript i). This allows multiple models to separately explore while having something in common, i.e. the task that models are collaborating for.\n\n*PSO could run with different model architectures* if you move the optimization from the weight space to the logit space. For a pool of n models, each model starts with a one-hot vector v_i = (0, …, 1, …, 0) of size n. The i-th model generates text by decoding from logit distribution $p_i’ = sum_{j=1}^n v_{i,j} p_j$, learning an aggregate distribution of itself and other models. In the beginning, as v_i is one-hot $p_i’ = p_i$ and it only employs itself. As optimization goes, v_i becomes dense and models learn to collaborate by contributing logit distributions.\n\nThis type of logit-level collaboration is successful across many scenarios [2,3, and also [26] itself]. We carried it out for a pool of 5 7b and 5 2b models so the pool contains heterogeneous architectures: Table 7 shows that it works and improves over various other settings of 2b/7b models.\n\n> It is unclear how meaningful it is to compare with the chosen baselines. How many baselines do multi-round optimization as the proposed method does? If prior works have done either dynamic weight or dynamic role, can we do a baseline that does both by combining two prior works (e.g., Model Swarms + GPT-Swarm), which could serve as a stronger baseline to test the effectiveness of the proposed role-step/weight-step? How difficult is it to combine a prior dynamic weight technique vs a prior dynamic role?\n\nMulti-round: dynamic weight and dynamic role baselines, 10 in total, are all multi-round, which repeatedly evaluate on f. We use 20 max iterations across all approaches, including ours.\n\nBoth role and weight: we are the first to the best of our knowledge, as this is one of our contributions.\n\nThere won’t be a natural way to combine a role-only and weight-only approach as they have different utility and goals. We reconcile this gap through G-Decode, through JFK-score, through the swarm methodology, to present a seamless role-AND-weight approach where mutual information between the two steps reinforces each other. But if you have to, you could force the same goal upon these two approaches. We try this out where both approaches are strictly optimized no matter single-model or multi-model:\n\n|                | K-Cross | AB-ltp | AbstainQA |\n|:--------------:|:-------:|:------:|:---------:|\n|  Model Swarms  |  0.428  |  0.135 |   0.175   |\n|    GPT-Swarm   |  0.320  |  0.134 |   0.023   |\n| forced combine |  0.376  |  0.109 |   0.112   |\n|      ours      |  0.450  |  0.215 |   0.220   |\n\nIt isn’t better than ours, in fact it is even worse than the two component methods on AB-ltp, potentially due to the multi-agent nature of this task. We believe our role-AND-weight approach is uniquely effective beyond forcibly gluing two parts together.\n\n> (Line 103) What's x_best? Is it the same as g (the global best)?\n\nx_best is the best-found location of this step, while global best g is the best-found across all steps. We will add this sentence to line 104.\n\n> Isn't that the DAG itself already has an assignment? If so, why do we “randomly select an LLM for each position” (Line 145)?\n\nYes, the default assignment to a DAG is “model i as node i” and each model appears exactly once. We do new assignments to 1) better explore different collaboration possibilities and 2) allow one model to potentially appear multiple times in the DAG while models unhelpful to the task don’t need to appear at all.\n\n> Are the optimization rounds (role-step and weight-step) performed independently for every dataset or datapoint in Table 1?\n\nYes, it is per dataset for a task adaptation setting. We will state this in Section 4, line 179.\n\n> Why can the proposed method alleviate prompt engineering issues? What prompts does the proposed method use? How about the baselines\n\nIn many multi-agent/model approaches, you have to manually design prompts for models to play different roles. For example, in one baseline Meta-Agent, one of the many prompts (and there are many more like this) is as follows and I’m sure has gone through many manual iterations:\n\n```\nYou are an expert machine learning researcher testing various agentic systems. Your objective is to\ndesign building blocks such as prompts and workflows within these systems to solve complex tasks.\nYour aim is to design an optimal agent performing well on [Brief Description of the Domain].\n[Framework Code]\n[Output Instructions and Examples]\n[Discovered Agent Archive] (initialized with baselines, updated at every iteration)\n# Your task\nYou are deeply familiar with prompting techniques and the agent works from the literature. Your goal is\nto maximize the specified performance metrics by proposing interestingly new agents.\nObserve the discovered agents carefully and think about what insights, lessons, or stepping stones can\nbe learned from them.\nBe creative when thinking about the next interesting agent to try. You are encouraged to draw inspiration\nfrom related agent papers or academic papers from other research areas.\nUse the knowledge from the archive and inspiration from academic literature to propose the next interesting agentic system design.\nTHINK OUTSIDE THE BOX.\n```\n\nIn our case (Table 9), the prompt is only:\n\n```\n“Please answer the following question with the help of previous responses, feel free to ignore wrong or unhelpful Responses.”\n```\n\nInstead of manual design, we count on optimization so models can automatically learn and discover what I should do/what my role is, given the responses of previous models. Figure 4 illustrates that models do discover roles such as divide and conquer, feedback, refine, etc., despite not being specifically asked for in the prompt.\n\n[1] Holtzman, Ari, et al. \"The Curious Case of Neural Text Degeneration.\"\n\n[2] Liu, Alisa, et al. \"Tuning Language Models by Proxy.\"\n\n[3] Mavromatis, Costas, Petros Karypis, and George Karypis. \"Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization.\""}, {"review_id": "ROwfmzpU4F", "reviewer": "Reviewer_yVca", "summary": "This paper presents Heterogeneous Swarms (HS), an algorithm that optimizes multi-LLM systems by representing them as DAGs where models exchange messages. Guided by a given utility function, the method uses Particle Swarm Optimization to jointly optimize both the LLM connection structure and their individual parameters. The optimization alternates between a \"role-step\", which learns the graph topology, and a \"weight-step\", which adapts model parameters based on a novel \"JFK-score\" metric designed to quantify each LLM's individual contribution. Through extensive experiments, the proposed method is shown to outperform 17 baselines by an average of 18.5% across 12 diverse tasks, demonstrating substantial collaborative gains.", "questions": "1. Could you provide a more detailed comparison of the resource requirements for HS versus key baselines? Specifically, a breakdown of metrics such as total optimization time, cost for optimization and inference (e.g., total tokens processed or GPU hours), would be very helpful for understanding the practical trade-offs involved.\n2. The optimization addresses a very high-dimensional search space. How does the framework ensure adequate exploration of the search space within 20 iteration limit to find effective solutions? Was any specific initialization strategy employed beyond random initialization to guide the search?\n3. In Table 1, the \"Prediction Merge\" baseline performs worse than the \"Best Single\" expert. This is somewhat counterintuitive for an ensemble method. Could the authors offer some insight into why a simple plurality vote might degrade performance?\n4. In table 2, the results for \"Ours w/o Role\" can be interpreted as a dynamic weight optimization method on a fixed graph. This approach still outperforms many dedicated dynamic weight baselines. What is the key algorithmic difference that leads to this superior performance?\n5. In appendix, the paper demonstrates that a collaboration of weaker models can outperform a stronger single model. How does the total inference cost of the \"HS(2-10)\" or \"HS(6-10)\" swarms compare to the inference cost of the single top-1 model they outperform? Is there a trade-off between collaborative performance gain and increased inference budget?\n6. Could you clarify how the utility function $f$ was defined for each of the 12 benchmarks? While accuracy is a likely metric for many, a detailed breakdown in the appendix specifying the evaluation metric for each dataset is needed to enhance the clarity.", "limitations": "Yes. Limitations have been thoroughly discussed.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The experiment setup is comprehensive. The authors compare HS against 17 baselines that cover a wide spectrum of approaches, including static/dynamic roles and static/dynamic weights. This rigorous comparison across 12 diverse tasks provides strong evidence for the method's effectiveness.\n2. The paper is well-written, with clear explanations of its complex methodology. The analysis is thorough and insightful. For instance, the ablation study disentangles the relative importance of roles and weights for different task types, justifying the joint optimization approach.  Furthermore, the analysis of \"collaborative gains\" provides a quantitative justification for the use of multi-LLM systems by showing that the swarm can solve problems that no single constituent model can. \n3. While the concept of multi-LLM collaboration is not new, the explicit formulation of this as a joint optimization problem over graph structures and model parameters is a significant contribution. The methods and metrics introduced, such as the iterative two-step optimization process and the JFK-score for credit assignment, are well-grounded in the problem they aim to solve. This provides a flexible yet powerful framework for designing multi-LLM systems.\n\nWeaknesses:\n1. The primary limitation of HS is its computational expense. The optimization process involves numerous inferences across a swarm of graphs and models. While the authors acknowledge this and propose sparsity and dropout strategies to mitigate the cost, the optimization phase still remains expensive.\n2. The weight-step of the algorithm, in its default implementation, requires all LLMs in the pool to share the same model architecture to ensure they share the same parameter space for optimization. While the authors suggest potential workarounds, this is a notable constraint that limits the diversity of truly \"heterogeneous\" models that can be incorporated, especially proprietary models which cannot be fine-tuned.\n3. The role-step optimizes continuous adjacency matrices, which is a very high-dimensional search space ($n^2$ for $n$ models). While PSO is suited for non-differentiable problems, its effectiveness can degrade in such large search spaces. The paper could benefit from a deeper discussion on why PSO is the right choice over other black-box optimization techniques (e.g., evolutionary strategies, Bayesian optimization) and how it scales with an increasing number of LLMs.", "was_revised": true, "final_justification": "I vote for acceptance of the paper. It is a good paper with strong technical contribution and solid empirical validation.", "rebuttal": "We would like to thank the reviewer for their thoughtful comments and feedback.\n\n> The primary limitation of HS is its computational expense. The optimization process involves numerous inferences across a swarm of graphs and models. While the authors acknowledge this and propose sparsity and dropout strategies to mitigate the cost, the optimization phase still remains expensive.\n\nWe discuss time/space complexity in Appendix C. Briefly summarize:\n\nGiven a pool of $n$ LLMs, we denote the data size behind a utility function as $|f|$ and run H-Swarm with $N$ graphs and $M$ model assignments. The optimization cost is then $O(n(N+M)|f|)$ and the inference cost $O(n)$. As $N$ and $M$ are hyperparameter constants and $|f|$ is a fixed dataset, both costs scale only linearly with the LLM pool size $n$.\n\nAs for memory costs, you don’t need to load all $n$ models at once. At the bare minimum, you only need 1 GPU that could load 1 model, do its inference, and switch it out for the next model. At the maximum, you could use $n$ GPUs to load one model per GPU. Any amount of GPU/memory is workable through space-time tradeoff,s and our implementation supports any (#model, #GPU) combinations.\n\nEmpirically, it isn’t actually expensive if you refine the implementation. We employ multiprocessing and distributed generation: if multiple GPUs are available, models are separately loaded into memory, and through rotational generation the throughput is very high. If you look at Figure 7: 10 7B models, 1000 data points in f, 5 A40 GPUs, then in 0.43 to 3.02 hours you get a collaboration structure. We personally wouldn’t say 5 GPUs for 3 hours is too “expensive”.\n\n> The weight-step of the algorithm, in its default implementation, requires all LLMs in the pool to share the same model architecture to ensure they share the same parameter space for optimization. While the authors suggest potential workarounds, this is a notable constraint that limits the diversity of truly \"heterogeneous\" models that can be incorporated, especially proprietary models which cannot be fine-tuned.\n\n*Open models with different architectures*\n\nWe move the optimization from the weight space to the logit space. For a pool of n models, each model starts with a one-hot vector v_i = (0, …, 1, …, 0) of size n. The i-th model generates text by decoding from logit distribution $p_i’ = sum_{j=1}^n v_{i,j} p_j$, learning an aggregate distribution of itself and other models. In the beginning, as v_i is one-hot $p_i’ = p_i$ and it only employs itself. As optimization goes, v_i becomes dense and models learn to collaborate by contributing logit distributions.\n\nThis type of logit-level collaboration is successful across many scenarios [1,2]. We carried it out for a pool of 5 7b and 5 2b models so the pool contains heterogeneous architectures: Table 7 shows that it works and improves over various other settings of 2b/7b models.\n\n*Closed models that cannot be finetuned*\n\nH-Swarm is compatible with closed models too: run inference on them as usual, in the weight-step just skip the PSO weight updates for them and update other open models only.\n\n> The role-step optimizes continuous adjacency matrices, which is a very high-dimensional search space (for models). While PSO is suited for non-differentiable problems, its effectiveness can degrade in such large search spaces. The paper could benefit from a deeper discussion on why PSO is the right choice over other black-box optimization techniques (e.g., evolutionary strategies, Bayesian optimization) and how it scales with an increasing number of LLMs. The optimization addresses a very high-dimensional search space. How does the framework ensure adequate exploration of the search space within 20 iteration limit to find effective solutions? Was any specific initialization strategy employed beyond random initialization to guide the search?\n\nThe search space is high-dimensional, but swarm for LLMs doesn’t need to explore the vast search space compared to traditional optimization problems:\n\n|    Setting    |                               Initialization                               |                            Exploration                            |\n|:-----------:|:--------------------------------------------------------------------------:|:-----------------------------------------------------------------:|\n| traditional |                           many and mostly random                           |   extensive and it is unclear where in the search space is good   |\n|  LLM world  | a few, but they are already of good quality since they are LLM checkpoints | mostly the convex hull and linear mesh of the initial checkpoints |\n\nWe use t-sne for dimensionality reduction to plot the movement of LLMs in the 20 iterations (couldn't show figures here). We find that they do not (and don’t need to) go all over the place, instead they converge mostly within the convex hull of models that are already of moderate quality, just not fully adapted to this task.\n\n> Could you provide a more detailed comparison of the resource requirements for HS versus key baselines? Specifically, a breakdown of metrics such as total optimization time, cost for optimization and inference (e.g., total tokens processed or GPU hours), would be very helpful for understanding the practical trade-offs involved.\n\nWe summarize the optimization/inference complexity in Table 4 in the appendix, to recap:\n\n|      Method      | Optimization | Inference |\n|:----------------:|:------------:|:---------:|\n|    best single   |       /      |    O(1)   |\n| prediction merge |       /      |    O(n)   |\n|   static weight  |       /      |    O(1)   |\n|  dynamic weight  |     O(n)     |    O(1)   |\n|    static role   |       /      |    O(n)   |\n|   dynamic role   |     O(nN)    |    O(n)   |\n|       ours       |   O(n(N+M))  |    O(n)   |\n\nSince our approach does more (role AND weight) naturally the optimization cost is higher, while the inference time is on par with the stronger baselines.\n\nFor an empirical cost, we present the time (in hours) using A40 GPUs for a few approaches:\n\n|      Method      | Optimization (per iter.) | Inference |\n|:----------------:|:------------------------:|:---------:|\n|    Best Single   |             /            |   0.012   |\n| Prediction Merge |             /            |   0.138   |\n|    Greedy Soup   |           1.68           |   0.012   |\n|   Model Swarms   |           1.83           |   0.014   |\n|    Meta-Agent    |           2.07           |   0.264   |\n|       GNNs       |           1.73           |   0.185   |\n|       ours       |           3.02           |   0.204   |\n\nThere are many complexities behind this empirical time: some approaches only need 1 GPU and approaches like ours could scale well with multiple GPUs using multiprocessing; approaches like Meta-Agent require black-box models to guide structure designs and introduce additional latency, etc. Overall, our approach has a reasonable cost, could be further reduced with strategies in Table 3, and leads to better utility through the co-evolution of role and weight.\n\n> In Table 1, the \"Prediction Merge\" baseline performs worse than the \"Best Single\" expert. This is somewhat counterintuitive for an ensemble method. Could the authors offer some insight into why a simple plurality vote might degrade performance?\n\nThis indicates that “truth is in the hands of the few”: because the models are diverse (exposure to different domains, tasks, etc.), without any adaptation only a few (or even just one in some cases) could solve a task-specific problem. The plurality vote of the many is then often incorrect compared to the model that best specializes in this task.\n\nThis highlights the importance of the weight-step: to prepare models for the current task in question and adapt their existing skill set to what would be helpful for this task and collaboration.\n\n> In table 2, the results for \"Ours w/o Role\" can be interpreted as a dynamic weight optimization method on a fixed graph. This approach still outperforms many dedicated dynamic weight baselines. What is the key algorithmic difference that leads to this superior performance?\n\nThis is because swarm intelligence and PSO is a strong solution for weight optimization. In Table 1, model swarms is very often the best weight-based approach. While “ours w/o role” uses different utility signals for weight updates (JFK-score, as opposed to individual model performance), they both operate with PSO. By additionally having the role orchestration step, we further outperform Model Swarms.\n\n> In appendix, the paper demonstrates that a collaboration of weaker models can outperform a stronger single model. How does the total inference cost of the \"HS(2-10)\" or \"HS(6-10)\" swarms compare to the inference cost of the single top-1 model they outperform? Is there a trade-off between collaborative performance gain and increased inference budget?\n\nIf the inference cost of best single is x, then HS(2-10) is ~9x and HS(6-10) ~5x. Yes there is a trade-off: weaker models, if compute allows to be used in collaboration, could become strong multi-model systems.\n\n> Could you clarify how the utility function was defined for each of the 12 benchmarks? While accuracy is a likely metric for many, a detailed breakdown in the appendix specifying the evaluation metric for each dataset is needed to enhance the clarity.\n\nWe provide details in Appendix D:\n\nMMLU-pro, Knowledge Crosswords, COM2, Normad, AgentBench-KG, AbstainQA, and WoW are evaluated in multiple-choice settings. GSM8k, NLGraph, and GAIA-text are evaluated via exact match. AgentBench-LTP and Qasper are evaluated by Gemini-1.5-pro for gold answer similarity on a scale of 1 to 10, normalized to 0 to 1.\n\n[1] Liu, Alisa, et al. \"Tuning Language Models by Proxy.\"\n\n[2] Mavromatis, Costas, Petros Karypis, and George Karypis. \"Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization.\""}, {"review_id": "F3sDJU9nfW", "reviewer": "Reviewer_cehQ", "summary": "This paper proposes Heterogeneous Swarms, a method for jointly optimizing model roles and weights in multi-LLM systems. The approach represents multi-LLM systems as directed acyclic graphs (DAGs) and uses particle swarm optimization (PSO) to iteratively optimize both the graph structure (role-step) and model weights (weight-step). The method introduces G-DECODE for converting continuous adjacency matrices to discrete DAGs and JFK-score for evaluating individual model contributions. Experiments across 12 tasks show an 18.5% average improvement over 17 baselines.", "questions": "1. **Theoretical Justification**: Can you provide theoretical analysis of when and why PSO should work well for this problem? What are the convergence guarantees?\n2. **Scalability Analysis**: How does performance scale with the number of expert models (beyond 10)? What are the practical limits of the approach?\n3. **Architecture Heterogeneity**: How could the method be extended to handle expert models with different architectures? This seems crucial for practical deployment.\n4. **Hyperparameter Robustness**: How sensitive is the method to PSO hyperparameters? Could you show performance across a range of settings rather than just the best-found configuration?\n5. **Computational Efficiency**: Beyond the proposed speedup strategies, are there more fundamental ways to reduce the computational overhead while maintaining performance?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. **Novel Joint Optimization Approach**: The paper addresses an important limitation in existing multi-LLM systems by jointly optimizing both roles (graph structure) and weights, rather than treating them as separate problems.\n2. **Comprehensive Experimental Evaluation**: The evaluation spans 12 diverse datasets across knowledge, reasoning, agent, and miscellaneous tasks, with comparisons against 17 baselines across multiple categories.\n3. **Practical Algorithmic Contributions**:\n    - G-DECODE provides a principled way to convert continuous optimization to discrete DAG structures\n    - JFK-score offers a novel metric for quantifying individual model contributions in multi-LLM systems\n4. **Thorough Analysis**: The paper includes extensive ablation studies, collaborative gain analysis, role distribution analysis, and scaling experiments that provide valuable insights.\n5. **Strong Empirical Results**: Consistent improvements across tasks with statistically significant results on most datasets.\n\n### Weaknesses\n\n1. **Limited Theoretical Foundation**: The paper lacks theoretical analysis of convergence properties, optimality guarantees, or conditions under which the approach should work well. The choice of PSO and the specific algorithmic design appear largely empirical.\n2. **Computational Overhead**:\n    - The optimization process requires O(n(N+M)) model inferences per iteration, which could be prohibitively expensive for large model pools\n    - Limited analysis of computational cost vs. performance trade-offs\n    - The speedup strategies (sparsity, dropout) help but may not address fundamental scalability issues\n3. **Architecture Constraints**: The requirement that all models share the same architecture for weight optimization significantly limits practical applicability, as real-world scenarios often involve heterogeneous model architectures.\n4. **Evaluation Limitations**:\n    - Only tested with relatively small models (7B parameters)\n    - Limited to 10 expert models - scalability to larger pools unclear\n    - Some datasets have small evaluation sets (e.g., GAIA-text with 28 examples)\n5. **Methodological Concerns**:\n    - The continuous-to-discrete conversion in G-DECODE may lose important structural information\n    - PSO hyperparameter sensitivity not thoroughly analyzed\n    - The JFK-score formulation could be sensitive to random assignment variations\n\n### Technical Issues\n\n1. **G-DECODE Algorithm**: While novel, the algorithm's reliance on top-p sampling for both end node selection and edge creation introduces stochasticity that may affect reproducibility. The paper would benefit from analysis of this sensitivity.\n2. **JFK-Score Robustness**: The individual contribution metric depends on random model assignments. More analysis on the stability and reliability of this metric across different assignment strategies would strengthen the work.", "was_revised": true, "final_justification": "The authors' rebuttal has satisfactorily resolved my concerns regarding scalability and hyper-parameter sensitivity, making the work feel more robust. Consequently, I am raising my score from 4 to 5.", "rebuttal": "We would like to thank the reviewer for their thoughtful comments and feedback.\n\n> The paper lacks theoretical analysis of convergence properties, optimality guarantees, or conditions under which the approach should work well. The choice of PSO and the specific algorithmic design appear largely empirical.\n\nThere are theoretical analyses of PSO convergence in evolutionary algorithm research such as [1], while the assumptions made in the classic optimization problems are not applicable to LLMs. For example, the initial particles in classic PSO are often initialized randomly or in a grid, while the initial LMs in the swarm cannot be random/arbitrary (taking a random point in the 7-billion-dimensional space would almost certainly be a failed language model). Instead they are seeded with curated model checkpoints with different training and fine-tuning data mixtures. The utility functions here (often performance on a dataset) are also much more complex than the assumptions made in the analysis of traditional optimization problems.\n\nGiven these differences, we decided to take an empirical route to analyze convergence and stability properties. We run H-Swarm for 20 times on three datasets and report its average, worst, best, as well as standard deviation.\n\n|         |  avg  |  max  |  min  |   std  |\n|:-------:|:-----:|:-----:|:-----:|:------:|\n| K-Cross | 0.447 | 0.500 | 0.406 | 0.0279 |\n|   COM2  | 0.587 | 0.617 | 0.560 | 0.0196 |\n| NLGraph | 0.655 | 0.713 | 0.608 | 0.0330 |\n\nResults show that ths standard deviation are all < 0.05, indicating that empirically the algorithm is mostly stable across runs.\n\n> The optimization process requires O(n(N+M)) model inferences per iteration, which could be prohibitively expensive for large model pools. Limited analysis of computational cost vs. performance trade-offs. The speedup strategies (sparsity, dropout) help but may not address fundamental scalability issues\n\nWe discuss time/space complexity in Appendix C. Briefly summarize:\n\nGiven a pool of $n$ LLMs, we denote the data size behind a utility function as $|f|$ and run H-Swarm with $N$ graphs and $M$ model assignments. The optimization cost is then $O(n(N+M)|f|)$ and the inference cost $O(n)$. As $N$ and $M$ are hyperparameter constants and $|f|$ is a fixed dataset, both costs scale only linearly with the LLM pool size $n$.\n\nAs for memory costs, you don’t need to load all $n$ models at once. At the bare minimum, you only need 1 GPU that could load 1 model, do its inference, and switch it out for the next model. At the maximum, you could use $n$ GPUs to load one model per GPU. Any amount of GPU/memory is workable through space-time tradeoff,s and our implementation supports any (#model, #GPU) combinations.\n\nEmpirically, it isn’t actually expensive if you refine the implementation. We employ multiprocessing and distributed generation: if multiple GPUs are available, models are separately loaded into memory, and through rotational generation the throughput is very high. If you look at Figure 7: 10 7B models, 1000 data points in f, 5 A40 GPUs, then in 0.43 to 3.02 hours you get a collaboration structure. We personally wouldn’t call 5 GPUs for 3 hours “prohibitively expensive”.\n\n> The requirement that all models share the same architecture for weight optimization significantly limits practical applicability, as real-world scenarios often involve heterogeneous model architectures.\n\nWe could run with different model architectures if you move the optimization from the weight space to the logit space. For a pool of n models, each model starts with a one-hot vector v_i = (0, …, 1, …, 0) of size n. The i-th model generates text by decoding from logit distribution $p_i’ = sum_{j=1}^n v_{i,j} p_j$, learning an aggregate distribution of itself and other models. In the beginning, as v_i is one-hot $p_i’ = p_i$ and it only employs itself. As optimization goes, v_i becomes dense and models learn to collaborate by contributing logit distributions.\n\nThis type of logit-level collaboration is successful across many scenarios [2,3]. We carried it out for a pool of 5 7b and 5 2b models so the pool contains heterogeneous architectures: Table 7 shows that it works and improves over various other settings of 2b/7b models.\n\n> Only tested with relatively small models (7B parameters); Limited to 10 expert models - scalability to larger pools unclear\n\nUsing 7B LMs is common practice for research on this topic. In fact, most of the baseline papers employed 7/8B language models in their experiments when open-access models are employed.\n\n10 models is a sufficiently large number for research on this topic. For baselines where multiple models are employed: dare used 3, model swarms used 10, ties used 11, greedy soups used 12, pack of llms used 2-6, etc. Optimization/inference cost scales linearly with the amount of models as explained above.\n\n> PSO hyperparameter sensitivity not thoroughly analyzed\n\nWe separately change various hyperaprameters into five different candidates, run for 5 times each, and report the standard deviation on two tasks:\n\n|                  | NLGraph | AbstainQA |\n|:----------------:|:-------:|:---------:|\n|      inertia     |  0.012  |   0.023   |\n| cognitive coeff. |  0.010  |   0.015   |\n|   social coeff.  |  0.010  |   0.012   |\n|   repel coeff.   |  0.009  |   0.014   |\n|    step length   |  0.012  |   0.033   |\n|      overall     |  0.011  |   0.018   |\n\nH-Swarm is not sensitive to the major hyperparameters in PSO, with an overall std < 0.02 across runs.\n\n> JFK-Score Robustness: The individual contribution metric depends on random model assignments. The JFK-score formulation could be sensitive to random assignment variations.\n\nThe goal *is* for JFK-scores to change when the assignment varies. In this way, by sampling M assignments, we obtain signals about a model’s general helpfulness for a task and how versatile it could be across different roles and contexts.\n\n> G-DECODE Algorithm: While novel, the algorithm's reliance on top-p sampling for both end node selection and edge creation introduces stochasticity that may affect reproducibility. The paper would benefit from analysis of this sensitivity.\n\nTo test the impact of algorithmic randomness, we fix the hyperparameters as the best-found set and run H-Swarms for 20 times on three datasets and report its average, worst, best, as well as standard deviation.\n\n|         |  avg  |  max  |  min  |   std  |\n|:-------:|:-----:|:-----:|:-----:|:------:|\n| K-Cross | 0.447 | 0.500 | 0.406 | 0.0279 |\n|   COM2  | 0.587 | 0.617 | 0.560 | 0.0196 |\n| NLGraph | 0.655 | 0.713 | 0.608 | 0.0330 |\n\nResults show that ths standard deviation are all < 0.05, indicating that empirically the algorithm is mostly stable across runs.\n\nFor deterministic outcomes, you could employ greedy decoding in G-Decode to always get the same graph structure from any A. Empirically we found that having this randomness actually helps to sample diverse structures and explore more potential collaboration patterns.\n\n[1] Van den Bergh, Frans, and Andries P. Engelbrecht. \"A study of particle swarm optimization particle trajectories.\" Information sciences.\n\n[2] Liu, Alisa, et al. \"Tuning Language Models by Proxy.\"\n\n[3] Mavromatis, Costas, Petros Karypis, and George Karypis. \"Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization.\""}]

Author rebuttal:

Meta-review: This paper proposes Heterogeneous Swarms, an algorithm for optimizing multi-LLM systems by representing them as directed acyclic graphs (DAGs). The proposed algorithm contains two iterative steps: (1) a role step, which learns the DAGs for inputs/outputs flows, and (2) a weight step, which estimates individual contributions and optimizes weights accordingly. Experimental results demonstrate the effectiveness of the proposed approach. After the rebuttal, all reviewers leaned toward acceptance.
Decision: Accept (poster)