Multi-agent Architecture Search via Agentic Supernet
Authors: Guibin Zhang
Luyang Niu
Junfeng Fang
Kun Wang
LEI BAI
Xiang Wang
Venue: ICML 2025 oral (2025)
Keywords: LLM-agent
Agentic Workflows
Multi-agent System
Primary area: deep_learning->large_language_models

Abstract: Large Language Model (LLM)-empowered multi-agent systems extend the cognitive boundaries of individual agents through disciplined collaboration and interaction, while constructing these systems often requires labor-intensive manual designs. Despite the availability of methods to automate the design of agentic workflows, they typically seek to identify a static, complex, one-size-fits-all system, which, however, fails to dynamically allocate inference resources based on the difficulty and domain of each query. To address this challenge, we shift away from the pursuit of a monolithic agentic system, instead optimizing the \textbf{agentic supernet}, a probabilistic and continuous distribution of agentic architectures. We introduce \textbf{MaAS}, an automated framework that samples query-dependent agentic systems from the supernet, delivering high-quality solutions and tailored resource allocation (\textit{e.g.}, LLM calls, tool calls, token cost). Comprehensive evaluation across six benchmarks demonstrates that MaAS \textbf{(I)} requires only $6\\sim45\\%$ of the inference costs of existing handcrafted or automated multi-agent systems, \textbf{(II)} surpasses them by $0.54\\%\sim11.82\\%$, and \textbf{(III)} enjoys superior cross-dataset and cross-LLM-backbone transferability.

Multi-agent Architecture Search via Agentic Supernet
Guibin Zhang Luyang Niu
Junfeng Fang'
Kun Wang"
Lei Bai Xiang Wang
Abstract
et al., 2024; Li et al., 2024), code generation (Shinn et al., 2023), web navigation (Deng et al., 2024), and data syn- thesis (Butt et al., 2024), by equipping LLMs with high- level features, including persona (Wang et al.. 2023b; Chen et al., 2024), tools (Shen et al., 2024; Richards & et al., 2023), planning (Qiao et al., 2024; Wu et al., 2024; He et al., 2023), and memory (Zhong et al., 2024; Hatalis et al., 2023; Packer et al., 2023). Building upon the success of single agents, researchers have demonstrated that combining mul- tiple agents, either cooperatively (Zhuge et al., 2024) or competitively (Zhao et al., 2023), can surpass the cognitive and intellectual capabilities of individuals (Du et al., 2023; Liang et al., 2023; Wang et al., 2023b; Jiang et al., 2023; Wu et al., 2023; Zhang et al., 2024a), showcasing the collective intelligence in a society LLM-agents (Piatti et al., 2024).
Large Language Model (LLMM)-epowwred multi- agent systems extend the cognitive boundaries of individual agents through disciplined collabo- ration and interaction, while constructing these systems often requires labor-intensive manual de- signs. Despite the availability of methods automate the design of agentic workflows, they typically seek to identify a static, complex, one- size-fits-all system, which, however, fails to dy- namically allocate inference resources based on the difficulty and domain of each query. To ad- dress this challenge, we shift away from the pur- suit of monolithic agentic system, instead op- timizing the agentic supernet, probabilistic and continuous distribution of agentic architec- tures. We introduce MaAS, an automated frame- work that samples query-dependent agentic sys- tems from the supernet, delivering high-quality solutions and tailored resource allocation (e.g., LLM calls, tool calls, token cost). Comprehen- sive evaluation across six benchmarks demon- strates that MaAS (I) requires only 6 ~ 45% of the inference costs of existing handcrafted or automated multi-agent systems, (II) surpasses them by 0.54% ~ 16.89%, and (III) enjoys superior cross-dataset and Moos-LLLM-backiiee transfcrability. The code is available at https: //github. com www.boiggeee////oecccoo//eeee./e MMaarrekkkyMMaS.....
Early multi-agent systems, such as CAMEL (Li al., 2023), AutoGen (Wu et al., 2023), and MetaGPT (Hong et al., 2023), while delivering specialized capacity, often heav- ily rely on manual configurations, including prompt engi- neering, agent profiling, and inter-agent communication pipelines (Qian et al., 2024). This dependency significantly limits the rapid adaptation of multi-agent systems to di- verse domains and application scenarios (Tang et al., 2023; Zhang et al., 2024c). More recently, the research com- munity has shifted toward automating multi-agent system design. For instance, DsPy (Khattab et al., 2023) and Evo- Prompting (Guo et al., 2023) automate prompt optimization, GPTSwarm (Zhuge et al., 2024) and G-Designer (Zhang et al., 2024b) optimize inter-agent communication, and EvoAgent (Yuan al., 2024) and AutoAgents (Chen et al., 2023a) self-evolve agent profiling. Nevertheless, they typ- ically focus on automating specific aspects of the system. Subsequently, ADAS (Hu et al., 2024a), AgentSqure (Shang et al., 2024), and AFlow (Zhang et al., 2024c) broaden the design search space. These state-of-the-ar (SOTA) meth- ods optimize a single, complex (multi- Jagent workflow for a given dataset via different search paradigms, e.g., heuristic search (Hu et al., 2024a), Monte Carlo tree search (Zhang et al., 2024c), and evolution (Shang et al., 2024), surpassing the performance of manually designed systems.
1. Introduction
Large Language Model (LLLM)-based agents (Richards & et al., 2023; Nakajima, 2023; Reworkd, 2023) have made remarkable strides in a spectrum of domains, such as ques- tion answering (Zhu et al., 2024a), data analysis (Hong
'Equal contribution National University of Singapore Tonggi University "Nanyang Technological University 'Shanghai AI Lab- oratory University of Science and Technology of China. Cor- respondence to: Kun Wang <wang kun ntu. edu.s Lei Bai <baisanshi @ gomi..com>.
Although the paradigm of searching for a one-size-fite fits- multi-agent system appears sufficient to optimize performance-related metrics such as accuracy and pass @k, its performance is largely constrained on resource-related
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Multi-agent Architecture Search via Agentic Supernet
Building Blocks
Simple arithmetic
1/0 ReAct
solution
CoT
Reflexion
ReAct
Agentic Supernet
12 21 =
Task
Debate
O 1/O
Web search
Highschool Physics
1/O+ Refleeeffoinnnee Debste
Evaluator-optimizze
Exxi
Exit
Row much work is required to otarge 19 aF capasitor to potential di "Fenence of
Task
solution
O Debate
Rellexion
oirpir
Yeeellonkk
critipoe
Complex Coding
Complicated system
feedback
Trais
ReAct
yelp. higi enent nser
O CoT
authertication and buthorizatson. Dreate product pagee.
Tusk
[--+
aeseorator
Figure 1. (Left) The building blocks of MaAS; (Right) When confronting different queries, the agentic supemeet adaptively samples tailored multi-agent architecture in a query-dependent manner.
metrics, such as token cost, LLM calls, and inference la- tency (Dilemma 1). Specifically, contemporary methods tend to optimize for a complex and resource-intensive agen- tic system, often involving dozens of LLM API calls and external tool usage (Liu et al., 2023). However, this is far from an optimal solution: for example, in mathematical benchmarks (Hendrycks et al., 2021), Ph.D.-level abstract algebra may indeed require complicated, ooken-heavy sys- tems, while simple elementary-level arithmetic works well with a single zero-shot I/O, This paradigm becomes even more problematic when applied to benchmarks across mul- tiple task domains (Dilemma 2): for instance, in the GAIA benchmark (Mialon et al., 2023), there is no single sys- tem that is optimal for both file reading and web searching tasks, leaving practitioners with no alternative but to split the benchmark and optimize separately (Zhuge et al., 2024). These dilemmas unveil that, the paradigm of automatically optimizing single multi-agent architecture fails to meet the dynamic and evolving demands of agentic deployment.
During inference, for different queries, MaAS samples a suitable multi-agent system delivering satisfactory resolu- tion and appropriate inference resources, thereby achieving custmmust collective intelligence.
We conduct comprehensive evaluations on seven widely adopted benchmarks, covering diverse use cases in code generation (HumanEval, MBPP), mathematical reason- ing (GSM8K, MATH, SVAMP), and diverse tool usage (GAIA). Empirical results demonstrate that MaAS is O nigh-performing, surpassing existing handcrafted or auto- mated multi-agent systems by 0.54% ~ 16.89%: @ token- economical, outperforming the SOTA baseline AFlow on the MATH benchmark with 15% of the training cost and 25% of the inference cost; 3 transferable across datasets and ALM-baccbonee; inductive, demonstrating strong generalizability to unseen agentic operators.
Briefly put, our key contributions are summarized as follows:
To address the above challenges, we propose Multi-agent Architecture Search (MaAS), which, instead of search- ing for plausible (possibly non-existent) optimal solution, generates distribution of multi-agent systems. Techni- cally, we model the optimization of MaAS on the agentic supernet, a probabilistic, continuous agentic architecture distribution that encompasses a vast number of possible multi-agent candidates. The agentic supernet can be seen as a cascaded layti-layer workflow, including 1 multiple agentic operators (e.g., CoT (Wei et al., 2022), Multi-agent Debate (Du al., 2023), ReAct (Yao et al., 2023)), as well as @ the parameterized probability distributions of operators across layers. During training, MaAS leverages controller network to sample multi-agent architectures conditioned on input queries. The distribution parameters and opera- tors are jointly updated based on environmental feedback, with the former's gradients approximated via Monte Carlo sampling and the latter's via textual gradient estimation.
Paradigm Reformulation: We introduce the concept of agentic supernet, a probabilistic, continuous agentic ar- chitecture distribution, which transforms the paradigm of optimizing single optimal multi-agent system into optimizing the distribution of multiple architectures.
Practical Solution: We propose MaAS, an agentic supernet-based framework that automatically evolves powerful multi-agent systems and adaptively allocates high-eerforing and resource-efficient solutions for user queries with varied difficulty, domain and features.
Experimental Evaluation: Extensive evaluations six benchmarks demonstrate that our framework discovers novel agentic systems with 0.54% ~ 16.89% higher performance, significantly lower training/inference costs, transfcrability across benchmarks and LLMs, and superior inductive capacity.
2

---

Multi-agent Architecture Search via Agentic Supernet
Varied benchmark
Comditioned sampling
Tp @ Ty + Tx
Valogp(a q O,
ni thul dt
Aapic: Probability
I1 (cVVa (W
e(alg) Qo n. 0) dg
Complicated system
Textual gradient
game spinner dov
parts ibbeled
Ny: 1r:--Ve
feedback
Tool
adli munonioo '." -he Bellaheestmmee speraor
updine gattin frndtio
(apinerr anding A
gradient agent
Task
Topic: Web novigation [iiffouity Serttnrrrn
According gthub, when
Controller
Regression numpy polyoomll
generato
exoontor
I/0 ReAct
Emviroment
feetbback
(Popic: summarize
1ti finull by tysdnhrrnnrr (Wha's the lost lno of haadso visillo the background the photo of oldes: (Bavor's headstone...
solution
ihymee
Agentic Supernet
Task
Fool
V logp(a 3,
Figure 2. The overall framework our proposed MaAS.
2. Related Work
AutoML. Automating the design of agentic systems is an emerging topic, yet the history of AutoML (He et al., 2021) provides clear precedents. Notably, the progression of agentic automation mirrors that of neural architecture search (NAS) (Ren et al., 2021). Core NAS techniques, such as reinforcement learning (Zoph, 2016). evolution- ary algorithms (Liu al., 2021), Bayesian optimization (BO) (White al., 2021), and MCTS (Wang al., 2021), have inspired analogous approaches in agentic automation, from policy gradient in (Zhuge et al., 2024) to evolutionary search in (Yuan et al., 2024), BO in (Shang et al., 2024), and MCTS in (Zhang et al., 2024c). In NAS, however, these black-box methods were eventually eclipsed by efficient su- pernet training (White al., 2023), culminating in seminal works like DARTS (Liu et al., 2018) and SNAS (Xie et al., 2018). Inspired by this, we introduce the first MAS search- ing framework leveraging an agentic supernet, posing new paradigms and challenges for agentic automation.
LLM-Agents and Agentic Systems. Building on the suc- cess of single agents (Shen et al., 2024; Zhu al., 2024b; Zhong et al., 2024), studies have shown that grouping multi- ple LLLM-based agents into multi-agent systems (MAS) can substantially enhance individual model capabilities (Wang et al., 2024a), as demonstrated in early attempts such as AutoGen (Wu et al., 2023), LLM-Debate (Du et al., 2023), and AgentVerse (Chen et al., 2023b). However, they heav- ily relied on manually crafted designs, which constrained the adaptability and flexibility of agents in addressing un- foreseen challenges (He et al., 2023; Chen et al., 2023b). As a result, automated agentic system design has gained increasing attention in the academic community.
Automating Agentic Systems. Efforts to automate the design of agent-based systems can be broadly classified into the following categories: (I) Prompt Optimiza- tion, such as PromptBreeder (Fernando et al., 2023), DsPy (Khattab et al., 2023), and EvoPrompt (Guo et al., 2023); (II) Inter-agent Communication, which focuses on orchestrating interactions between agents, including GPTSwarm (Zhuge et al., 2024), DyLAN (Liu et al., 2023), EvoMAC (Hu et al., 2024b), AgentPrune (Zhang et al., 2024a) and G-Designer (Zhang et al., 2024b); and (II) Agent Profiling, represented by AgentVerse (Chen et al., 2023b), EvoAgent (Yuan et al., 2024), and AutoA- gents (Chen et al., 2023a). Further, ADAS (Hu et al., 2024a) and AgentSquare (Shang et al., 2024) provide more comprehensive automation for single-agent design, while AFlow (Zhang et al., 2024c) achieves multi-agent work- flow automation using Monte Carlo tree search (MCTS). However, these nigh-performing methods still follow the paradigm of searching for a single final system, whereas MaAS searches for distribution of architectures with lower average inference costs (LLM calls, token cost, etc.).
3. Methodology
Figure illustrates the overall workflow of our method. MaAS takes diverse and varying difficulty queries as input and leverages a controller to sample a subnetwork from the agentic supernet for each query, corresponding tomized multi-agent system. After the sampled system exe- cutes the query, MaAS receives environment feedback and jointly optimizes the supernet's parameterized distribution and agentic operators. In the following sections, Section 3.1 formally defines the search space and optimization objec- tive of MaAS, Section 3.2 details how the controller query- dependently samples multi-agent structures, and Section 3.3 details the optimization of MaAS.
cus-
3.1. Preliminary
Search Space. We first define the basic unit of MaAS's search space, namely the agentic operator as follows:
3

---

Multi-agent Architecture Search via Agentic Supernet
Definition 3.1 (Agentic Operator). An agentic operator O is composite LLM-ageet invocation process that involves multiple LLM calls and tool usage:
3.2. Agentic Architecture Sampling
The core of MaAS lies in tailoring customized multi-agent system for each user query, which may vary in difficulty and domain, to deliver a satisfactory solution:
O = II. a M2 E M. EEPEETTT
(L)
p(alg, T, = c(alg) Co म, Og
(6)
where M and M. correspond to LLM backbones and the set of available LLMs, respectively. Similarly, P and T represent prompts and tools. m and denote the number of LLM-aagets and tools invoked in the operator, respectively.
where Q6 represents the controller network, which takes the query q. the parameterized distribution т, and the available operators O and outputs the sampled agentic architecture G. Qo is parameterized by ø. and e(+l-) denotes producing solution via executing G. we implement Qo follows:
Most existing single/multi-agent workflows can be viewed as agentic operators: CoT (Wei et al., 2022) can be con- denoted as Ocor:
sidered one with m and Self-RAG (Asai al., 2023) similarly involves agent allocation, but is equipped with 1 retrieval en- gine, denoted as ORAG:; Multi-agent debate (Du et al., 2023) involves multiple LLM-agent, multi-turn calls, denoted as Detane- The feasible set of agentic operators is denoted as O, and we discuss the initialization of O in Section 4.1 and Appendix B. We define a multi-agent system as:
Qo(9,g, x,, 0) II Te (Vela, (VWD
where VA denotes the selected operators at layer h. The selection of Ve is conditionally dependent on the query q and the operators from the previous layers. However, not all queries require execution across L layers. As discussed in Section 1, many questions can be resolved with simple zero-shot I/O (Zhang et al., 2024b), rendering L layers unnecessarily redundant. To address this, we introduce an early-exit operator, denoted as Oexi.. During sampling, if Oxxit is encountered, the process exits early:
g = (V,E,, V VCO, O, & VVVVV,
where V is the set of selected operators in 9 and & denotes their connectivity. G is constrained as direct acyclic graph (DAG). Finally, we define the agentic supernet: Definition 3.2 (Agentic Supernet). The agentic supernet denoted as AA { 0} ([= (O))ooo) where:
L
(=(9
= II E=1
Te (Vell,, (Vil1=)) To
(8)
Zy (O) = p(0 Al+-1), OEO, A ((=x(0))0ee)
(3)
+ IOWEn ((--
where (O) represents the probability of operator O present at layer f, conditioned on the preceding layers A1.-11 The supernet induces a joint distribution over all possible multi- multi-layer operator configurations:
where fexir denotes the layer at which Oexit appears, and 8(.) is the Kronecker delta function. We implement the sampling process T with a Mixture-of -ofture-of-Expertt Expert (Myle network (Shazeer et al., 2017; Huang et al., 2024):
L II II TOOoov,
Tee : q + Ve, V2- t= arg min kE(1... (00) jec
{O01. O22..... .Oah
(())
(4)
Σs thres,
(9)
OEO
where Loev, is the indicator function for the inclusion of O in the set of active operators V at layer f.
where S = sS, and S E [R|0| [Si, ... SSol represents the activation scores of all feasible operators w.r.t. q. Note that thrrs is a threshold value that governs operator activation. Operators are activated sequentially, starting from the one with the highest score, and the process continues until the cumulative score exceeds thres. This ensures that the number of selected operators per layer is query query-dependent,, allowing MaAS to dynamically allocate resources based on task complexity. S is given by: S, = FFN(v(g) Σoen, v(O)]I ...]l Σoer, v(O)), where v(-) denotes the embedding function using lightweight models like MiniLM (Wang et al., 2020) and Sentence- Bert (Reimers, 2019), and || represents concatenation. The detailed implementation of v(-) is placed in Appendix B.2.
Problem Formulation. Given benchmark D comprising multiple queries q and their corresponding oracle answers/- solutions a, the objective of MaAS is not to identify a single optimal agentic system like previous practices (Zhang et al., 2024c; Zhuge et al., 2024), but to optimize conditional probability distribution as follows:
max E Egga) ())0:gg))~A-C(4- P(Gla) (-r(G14)
s.t. C A (5)
where p( is a distribution that generates query-
dependent agentic architectures. U-) and C(.) represent the utiulity/peffrmmace and cost of G for query q, respectively, and À trade-off parameter.

---

Multi-agent Architecture Search via Agentic Supernet
Upon completing the sequential sampling procedure in MaAS, task-specific multi-agent system & is generated and executed to produce the answer ä. In the next section, we elucidate the process of updating the agentic supernet based on environmental feedhack.
Algorithm 1 Algorithm workflow of MaAS
Input :A dataset DD containing training set Drain and test set Dwn. Operator set O, Randomly initialized distribution ग, Controller network Q
Output: : Well-optimized agentic supernet, composed of dis- tribution π and operators 0
3.3. Cost-constrained Supernet Optinization
for (q,a) a) in Duain do
We present the optimization objective of MaAS as follows:
/+ Sample query-dependent MAS
*
for layer 4- 1 to L do
min Blg.a)DDGsq. I-p(alg, n, 0) + À C():)) (10) R,O
Vj - n (Velg. (Vr 1(-1);
b Eq. 9
if = L or oxir E Ve then break// Exit when reaching
where C() evaluates the cost of multi-agent systems. repre- sented by token cost, and X is the trade-off parameter. The term p(alg. O) in Equation (10) corresponds to Equa- tion (6), where the calculation of e(a(g) often involves external tools or API-based LLM calls, rendering it non- differentiable. Therefore, we employ an empirical Bayes Monte Carlo procedure (Carlin & Louis, 2000; Yan et al., 2021) to estimate the gradient w.r.t the distribution æ::
maximal sampling depth or encountering the early-exit operator
Obtain f (V,,-.., Vi) for query qi
D Eq.
/+ Execute sampled MAS
*/
Execute G and obtain ā + e(agg);
b Eq. 6
x/ /* Self-evolve agentic supernet
1 Σ Σ
mR V 7 (G
Compute loss w.r.t. т, VzL;
b Eq. 11
VAC ~
K
Estimate loss w.r.t O via textual gradient
b Eq. 12
(11)
Update π and O accordingly;
> Eq. 10
plaly, GR)
C(G C(G.g))
my E.p(alg 9.)
where my denotes the cost-aware importance weights of the agentic architecture. Intuitively, the distribution æ is updated to favor multi-agent systems that generate high- quality solutions with minimal token cost.
(such as merging, splitting, altering, etc.), respectively. See prompts in Appendix B.3. In this way, the core components of the agentic supernet, namely the agentic operators and their connectivity, are jointly updated, enabling the fully automated evolution of multi-agent systems. We summarize the notations in Table and the algorithm in Algorithm 1,
Operator Gradient add debater debate operator
4. Experiments 4.1. Experiment Setup
Sampled MAS
Textual Gradient
Task
Tasks and Benchmarks. We evaluate MaAS on six public benchmarks covering three domains: (1) math reasoning, GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), and MultiArith (Roy & Roth, 2016); (2) code gener- ation, HumanEyal (Chen et al., 2021) and MBPP (Austin et al., 2021)); and (3) tool use, GAIA (Mialon et al., 2023). For the MATH benchmark, we follow (Hong et al., 2024) in selecting 617 problems from four typical problem types (Combinatorics & Probability, Number Theory, Pre-algebra, Pre-calculus). The dataset statistics are in Appendix C.I.
Ensemble
feedback
Temperature Gradient lowes Lie snssrdle Lik's cosperature for atability
Prompt Gradient
Eunraboos
Figure 3.
The demonstration of textual gradient.
However, the gradient w.r.t operators V F cannot be com- puted similarly. As shown in Equation (1), operators include black-box box tool usage and natural language prompts, mak- ing numerical gradient updates infeasible. To address this, we utilize agent- based textual gradient (Hao et al., 2023; Liu al., 2023; Hu et al., 2024b; Zhou et al., 2024) to approximate the backpropagation for agentic operators, visualized in Figure 3 and formalized as follows:
Baselines. We compare MaAS with three series of agentic baselines: (1) single agent execution methods, including CoT (Wei al., 2022), ComplexCoT (Fu al., 2022), Self- Consistency (Wang et al., 2023a); (2) hand-craft multi- agent systems, including MultiPersona (Wang et al., 2023b), L.LM-Debate (Du et al., 2023), L.LM-Blender (Jiang et al., 2023), DyLAN (Liu et al., 2023), AgentVerse (Chen et al., 2023b) and MacNet (Qian et al., 2024); (3) (partially or fully) autonomous multi-agent systems, including GPTSwarm (Zhuge al., 2024), AutoAgents (Chen et al.,
VoL = Tp + TT 3 TN, T2 E T,x E (P,T,N) (12)
where Tp,Ty Ty represent agent-generater gradient analyses in textual format, corresponding to updates of the prompt, model temperature, and operator node structure
5

---

Multi-agent Architecture Search via Agentic Supernet
Table Performance comparison with single agent, hand-craft multi-agent systems, and automated agentic workflows. The base L.MM consistently set as apt-4o-mini for all baselines. We bold the best results and underline the runner-ups.
Method
GSM8K
MATH
MultiArith
HumanEval
MBPP
Avg.
Vanilla
87.45
46.29
96.85
87.08
71.83
77.50
CoT (Wei al., 2022)
80.10
16.401o.11
g6.31
88.13
71.83.0.00
77.95
ComplexCoT (Fu et al., 2022)
86.89.o.56 10.56
46.53
96.70
87.49
72.36
78.00
(C)Tx5) (Wang et al., 2023a)
87.57 10.12
17.91
96.58.0027
88.60
73.60
78.85
MultiPersona (Wang et al., 2023b)
87. po.50
45,43
97.49
88.32
73.19
78.39
LLM-Debate (Du et al., 2023)
89.4772.02 f2.02
48.54
97.33+
88,68 18.68
70.29 70.29
78.86
LLM-Blender (Jiang ct al., 2023)
88.35
46.92
97.29
88.80
77.05
79.68
DyLAN (Liu et al.. 2023)
89.98f2.53
48.63
97.12f0.27
90.42
77.30
80.69
VeentVerse (Chen ct al., 2023b)
89.91 89.91 f2.46
47.3511.06
97.50fo.66
89.29
74.28
79.67
MacNet (Qian et al., 2024)
87. (o.55
45,18
96.03.o.8e
B4.57722.55
65.28
75.00
AutoAgents (Chen et al., 2023a)
87.69
45.32,00.97
96.42jo.44
87.64
71.95;0.12
77.80
GPTSwarm (Zhuge et al., 2024)
89.14
47.88
96.79.o.or
89.32
77.4315.60
80.11
ADAS (Hu et al., 2024a)
86. j2.33
43.18.a.11 1....11
96.02 j0.03
84. B4..992..0
68. 13,370
75. 13
AgentSquare (Shang et al., 2024)
87.62 87.6220.17 r0.17
48.51
97.77
89.08 (2.0882.00
78.46
80.29
AFlow (Zhang et al., 2024c)
91.16
51.28
96.22 96.22.0063
90.93
81.67
82.25
MaAS (Ours)
92.30 14.85
51.82
98.80
92.85
82.17
83.50
task domains. The multi-agent system optimized by MaAS outperforms manually designed methods by an average of 3.90 6.40% and existing automated methods by 2.07 8.26%. Overall, as for mathematical reasoning and code generation, MaAS achieves an average best score of 83.59%. demonstrating its versatility and superiority. Table 2 shows a comparison of MaAS with automated systems and three additional baselines, including Auto- GPT (Richards & et al., 2023), TapeAgent (Bahdanau et al., 2024), and Sibyl (Wang et al., 2024b) on the GAIA bench- mark. GAIA encompasses tasks from various domains such as web browsing, file reading, and multimodal understand- ing. making it challenging to pursue single optimal multi- agent system for all tasks. Thus, the modest improvements of AFlow and ADAS over vanilla LLMs (only 3.35% 1 and 2. 0404% 1 on average) are understandable. contrast, MaAS can adaptively sample customized agentic systems for differ- ent domains, achieving 18.38% and 17.61% improvements on Level 1 and 2 tasks, respectively.
Table Performance on GAIA benchmark. The best and runner- up results are bolded and underlined, respectively.
Method
Level 1
Level 2
Level 3
Avg.
GPT-4o-mini
7.53
4.40
0
4.65
GPT-4
9.68
1.89
2.08
4.05
AutoGPT
13.21
0
3.85
4.85
TapeAgent
23.66
14.47
10.20
16.61
Sibyl
21.51
15.72
4.08
15.61
AutoAgents
16.13
0
0
5.16
GPTSwarm
23.66
16.35
2.04
16.33
ADAS
13.98
4.40
0
6.69
AgentSquare
22.58
15.72
6.25
16.34
AFlow
10.75
8.81
4.08
8.00
MAAS
25.91
22.01
6.25
20.69
2023a), ADAS (Hu et al., 2024a), AgentSquare (Shang et al., 2024) and AFlow (Zhang et al., 2024c). More details on baseline setups are provided in Appendix C.2.
Implementation details. We leverage both close-source LLM 100--44--mnni-0111 (OpenAI, 2024)) and open- (Yang al.,
source LLM 2024) and 1lama- 3 10b (Dubey et al., 2024)). All models are accessed via APIs with the temperature set to 1. We set the number of layers as = 4, the cost penalty coefficient as E {le - 3, 5e - 3, le 2}. and the sampling times = 4. thres = 0.3 for Equation (9).
4.3. Cost Analysis
To answer RQ2, we demonstrate that MaAS is both train- ing/inference cost-efticient from the following three dimen- sions: (1) token cost, (2) API cost, and (3) wall-clock time, as shown in Table and Figure 4. We observe:
Obs. MaAS's optimization resource-friendly. As shown in Figure (Training Tokens), among the various optimization-orientee agentic workflows, MaAS achieves the highest accuracy with the least training token consump- tion. While AFlow's accuracy is comparable to that of MaAS, its training cost reaches 22.50$, which is 6,8 that of MaAS (merely 3. 3.38$). Additionally, existing agentic
4.2. Performance Analysis
We compare MaAS with 14 baselines on the GSM8K, MATH, MultiArith, HumanEval, and MBPP benchmarks in Table I, and with 10 baselines on GAIA in Table 2. The following observations can be made:
Obs. O MaAS achieves optimal performance across all
6

---

Multi-agent Architecture Search via Agentic Supernet
Table Efficiency comparison hetween MaAS and state-of-the-art baselines the MATH Renchmark, We shade the values of the lowest aike//cot/wwall-looss time and the highest performance.
Method
Training
Inference
Overall
Prompt token
Completion
Wall-clock cost (S) time (min)
Total
Prompt token
Completion
Total
Wall-clock time (min)
Acc.
token
token
cost (S)
(%)
LLM-Debate
3, 275, 764
459, 097
6.76S
92
48.54
DyLAN
22, 152, 407 16, 147, 052
13.018
508
6, 0808, 483
3,303, 522
2.89$
39
48.63
MacNet
7,522, 057
2, 24043, 600
2.35$
47
45.18
GPTSwarm
21, 325, 266
6, 3,3699.5 884
7.028
129
3, 105, 571
788, 273
0.93$
30
47.88
AFlow
33, 80888,,,200 239
29, 051, 840
22.50$
184
2, 5050,, 944
2, 151, 981
1.66$
23
51.28
MaAS
3, 3,052, 159
2, 380, 505
3.38$
53
1,311, 669
853, 116
0.42$
19
51.82
Training
Inference Token
Token
Debaee
CaT-sC
Resemble
RoAct
Refine
Early-oiit
LlM-Redder
#'okrnees (10"s)
ADAS
Aflaw
Thbrres (10"s) s)
Market
DyLaN
Eaay query
Easy query
PylAM
ADAS
Anto doerts
1G0 hepe on odd warler peotive de lorc?
focte e
AgeniVerse
0.20
puse
s.an
0.20
busa
0.00
Agruzigriane
36
(Ours)
Artitgeents
bu20
pudb
bust
0.00
0ub4
Dusb
2:13
4B Accuracy (%)
4l Accuracy (%)
Inference
LLLM-Debate
2.5
API Cost (9)
(b) Easy
LLM Blender
API Cooe (USD)
2.0
MacNet+
(a) Easy case
wether whibearrdd Every "ime nome i number Si F1S
Hund queryy
DyLAN
Medium quecry
AntoAgents
re foir cir dladd ou What is the webut Express your frutien the the fanger" sumber)
whitt when mome E euriph the sumar The or and narber with son banden En hat the and of morth and is wour on the
take
ADAS
MultiPersona
GPTSwarm
AFlow
AgentVerse AgentSquare
MaAS (ORIs)
0.37
pyde
bubo
2.20
Accuracy (%)
0.oc
0.32
0.31
2.05
0,25
0.14
Figure 4. The cost analysis of MaAS on MATH benchmark.
ouco
0.12
5.17
0.02
5.02
0.34
c.20
0.14
0.20
0.1::
automation pipelines are relatively time-consuming, with DyLAN taking 508 minutes and GPTSwarm taking 129 minutes. In contrast, the optimization wall-clock time of MaAS requires only 53 minutes.
(c) Medium case
(d) Hard case
Figure 5. The visualization of MaAS's 's operator sampling process.
4.4. Case Study
Obs. 2 Agentic supernet enjoys superior token economy during inference. As shown in Figure 4 (Inference API Cost), MaAS achieves the highest accuracy with an API cost of 0.42$, demonstrating its high performance and to- ken economy. Although AgentSquare's 's API cost is slightly lower than MaAS's, this is due to its limitation to single- agent search, which severely restricts its performance (re- sulting in a 4% drop compared to MaAS). Table further highlights that MaAS has the lowest promp//completion token consumption, the lowest API cost, and the shortest wall-clock time during inference. These advantages can be attributed to the agentic supernet's ability to dynamically allocate resources based on the difficulty of the query.
In this section, we explore and visualize the intrinsic mech- anisms of the agentic supernet. Figure 5 showcases the probability distributions of the agentic supernet when faced with different queries, and Figure 6 presents the multi-agent systems designed by MaAS for queries from the MATH, GAIA, and HumanEval benchmarks. We have:
Obs. . MaAS learns to query-aware early exit from the reasoning process. As shown in Figure 5, when faced with the easy queries (a) and (b), MaAS exits multi-agent archi- tecture sampling at the second layer with probabilities of 0.37 and 0.47, respectively, selecting the early-exit operator. Notably, query (b) chose two agentic operators the first
7

---

Multi-agent Architecture Search via Agentic Supernet
criangle Py: And high reture anaa PAA
CoT Prompting
Buse ding sake say Bocan stul kar monnly ad moes
Query
arrddd 33
Workflow
QuerT
splutisn
Query
(from (ATA bearts Fovel binky souns)
Summarzie
Cuch triangle 30 60-33 ortangli,, and the typocaease of awe oriangle 1a coattn was the cutuer better the levpth the Tonger the trtanghe?
query
solution.
Query
Workflow
Relet
exvcntt
Debate
Ensemble
query
scradiiee by Jaca raslrax
nython
Wab scarchl
Work- flow
Mulriple
Rnaemble
Refine
Query
seericar none for the type saee the bedee Nystary
m Workflow
QUerY
Self considtrncy
solution
Summmarie
solution
Figure 6. Case study and visualization for MaAS. Queries from HvmanEval, MATH and GAIA benchmarks.
Peffrrater Jef And Cor eapyylllgy....oo J
Decbomuane Jold query (Kiglt
Performame (lat Ani percurry
early-exit operator in Equation (8); and (3) w/o C(-), elimi- nating the cost constraint in Equation (10). We observe from Table 4 that removing the textual gradient causes the largest performance drop, as it disables MaAS's self-evolving ca- pability. Removing Oexir and C(-) results in little impact performance, but it weakens MaAS's s query query-dependent nature and unnecessarily increases the inference cost.
of
Suiigling Linio
Figure 7. Parameter sensitivity analysis of MaAS. The unit of cost per query (right) and performance (left) is 10-3 $ and pass@1 (%), respectively.
Table 4. Ablation study of MaAS.
Dataset
HumanEval
MATH
@ass (%)
Cost
Accuracy (%)
Cost (10-3 $)
Metric
layer: direct I/O and ReAct, demonstrating MaAS's abil- ity to dynamically allocate different operators at each layer (corresponding to Equation (9)). For the more challenging queries (c) and (d), MaAS sampled additional layers, further proving its ability to customize the multi-agent system based on query awareness. This is also visualized by Figure 8, in which the probability of Oxxi becomes increasingly high with the supernet depth increases.
(10 3 $)
Vanilla MaAS
92.85
1.01
51.82
0.86
MaAS w/o VoL
90.17
0.90
48.23
0.84
MaAS w/o Oexi
91.44
1.67
51.53
1.04
MaAS w/o C(-)
92.94
1.38
51.19
1.28
Transferability Analysis. We evaluate whether the agen- tic supernet of MaAS is (1) model-agnostic and (2) gener- alizable across datasets, with results presented in Tables 7 and 8. As shown, the agentic supernet optimized by MaAS transfers well to models such as Qwen-2. 5-7bb, with 4. 1.98% ~ 5.50% 1 in performance, while also demonstrat- ing strong cross-dataset generalization.
4.5. Framework Analysis
Sensitivity Analysis We analyze the sensitivity of MaAS to three core parameters: the number of layers in the agentic supernet L, the cost penalty coefficient À in Equation (10), and the sampling count K in Equation (11). The results are presented in Figure 7. For the parameter L, we observe a significant performance improvement as L. increases from 2 to 4 (89.5% 92.8%). However, further increases yield only marginal performance gains while incurring higher per-query inference costs. Considering both performance and cost, we select L = 4. For the parameter X, we find that larger values lead MaAS to favor more cost-efficient solutions, albeit with some performance degradation. For the parameter K, we note that performance is suboptimal with highest variance when K 2. Increasing K to 4 effectively achieves satisfactory low-variance estimation.
Inductive Analysis. To evaluate whether MaAS possesses inductive capabilities, i.e., the ability to generalize to un- seen agentic operators, we select the Debate (Du et al., 2023) operator as holdout. We then compare the operator distri- bution of MaAS during inference with and without Debate in Figures and 9. The results demonstrate that MaAS can still reasonably activate and utilize the unseen operator at an appropriate proportion.
5. Conclusion
In this paper, we the first time shift the paradigm of auto- mated multi-agent system design from seeking a (possibly non-existent) single optimal system to optimizing a prob- abilistic, continuous distribution of agentic architectures,
We perform an ablation study on three key components of MaAS: (1) w/o VoL, removing the tex- tal gradient in Equation (12); (2) w/o Dexit, removing the
Ablation Study
8

---

Multi-agent Architecture Search via Agentic Supernet
termed the agentic supernet. Building on this concept, we propose MaAS, which dynamically samples multi-agent systems that deliver satisfactory performance and token effi- ciency for user queries across different domains and varying levels of difficulty. We believe that MaAS paves the way toward fully automated, sell-organizing, and self-evolving collective intelligence.
Carlin, B. P. and Louis, T. A. Empirical bayes: Past, present and future. Journal of the American Statistical Associa- tion, 128652):18661289 2000.
Chen, G., Dong, S., Shu, Y., Zhang, G., Sesay, J., Karls- son, B. F., Fu, J., and Shi, Y. Autoagents: A frame- work for automatic agent generation. arXiv preprint arXiv:2309.1720 2023a.
Acknowledgements
Chen, J., Wang, X., Xu, R., Yuan, S., Zhang, Y., Shi, W., Xie, J.. Li, S., Yang, R., Zhu, T., et al. From persona to personalization: A survey on role-playing language agents. arXiv preprint arXiv:2404. 18231, 2024.
This research is supported by the National Natural Science Foundation of China (No. 92270114) 14), and the Shanghai Municipal Science and Technology Major Project.
Chen, M., Tworek, J., Jun, H., Yuan, Q. Ponde de Oliveira Pinto, H., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L.., Bavarian, M., Winter, C., Tillet, P., Petroski Such, F, Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Hebgen Guss, W., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain. S., Saunders, W., Hesse, C.,, Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., Mc- Grew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code, July 01, 2021 2021.
Impact Statement
Ethical Considerations. We believe that our proposed MaAS framework raises no ethical concerns regarding its motivation, design, implementation, or data usage. The method is designed to advance the automation of multi- agent systems in a principled and resource-efficint manner, ensuring responsible development while adhering ethical guidelines in AI research.
Societal Implications. MaAS introduces a new paradigm in multi-agent system design by replacing static, one-size- fits-all architectures with a dynamic and adaptive agentic supernet. This approach enables fine-grained resource allo- cation tailored to query difficulty and domain, significantly improving efficiency while maintaining high-quality outputs. By reducing inference costs and enhancing the flexibility of multi-agent workflows, MaAS has the potential to de- mocratize access to intelligent automation across diverse applications, including education, research, and industry.
Chen, W., Su, Y., Zuo, J., Yang, C., Yuan, C., Qian, C., Chan, C.-M., Qin, Y., Lu, Y., Xie, R., Liu, Z., Sun, M., and Zhou, J. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents, 2023b.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv prepring, abs/21 14164. 2021.
References
Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. Self- rag: Learning retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511 2023.
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards general- ist agent for the web, Advances in Neural Information Processing Systems, 36, 2024.
Austin, J. Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., al. Program synthesis with large language models. arXiv preprint arXiv:2108 108.0732, 2021.
Du, Y., Li, Torralba, A., Tenenbaum, J. B., and Mordatch, I. Improving factuality and reasoning in language models through multiagent debate. CoRR, abs/2305. 14325,, 2023,
Bahdanau, D., Gontier, N., Huang, G., Kamalloo, E.,, Pardi- nas, R., Piché, A., Scholak, T., Shliazhko, O., Tremblay, J. P., Ghanem, K., et al. Tapeagents: a holistic framework for agent development and optimization. arXiv preprint arXiv:2412.08445, 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21778 2024.
Butt, N., Chandrasekaran, V., Joshi, N.. Nushi, B., and Balachandran, V. Benchagents: Automated bench- mark creation with agent interaction. arXiv preprint MxXiv22410.22540 2024.
Feng, T., Shen, Y., and You, J. Graphrouter: A graph-based router for Ilm selections. arXiv preprint urXiv:2410.0335 2024.
9

---

Multi-agent Architecture Search via Agentic Supernet
Fernando, C., Banarse, D., Michalewski, H., Osindero, S., and Rocktäschel, T. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797 2023.
Huang, D., Bu, Q.. Zhang. J. M., Luck, M., and Cui, H. Agentcoder: Multi-agent- based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010 2023.
Fu. Y., Peng, H., Sabharwal, A., Clark, P., and Khot, T. Complexity- based prompting for multi-step reasoning. In The Eleventh International Conference Learning Representations, 2022.
Huang, Q., An, Z., Zhuang, N., Tao, M., Zhang, C., Jin, Y., Xu, K., Chen, L., Huang, S., and Feng, Y. Harder tasks need more experts: Dynamic routing in moe models. arXiv preprint arXiv:2403. 07652, 2024.
Guo, Q. Wang, R., Guo, J., Li, B., Song. K., Tan, X., Liu, G., Bian, J., and Yang. Y. Connecting large language mod- els with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08.0 2023.
Jiang. D., Ren, X., and Lin, B. Y. LLM-blender: Ensembling
large language models with pairwise ranking and genera- tive fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 14165-14178, Toronto, Canada, July 2023. Association for Computational Linguisties.
Hao, R., Hu, L.., Qi, W., Wu, Q. Zhang, Y., and Nie, L.. Chatllm network: More brains, more intelligence, April 01, 2023 2023.
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., San- thanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint urXiv:2310.037/4 2023.
Hatalis, K., Christou, D., Myers, J., Jones, S., Lambert, K., Amos-Binks, A., Dannenhauer, Z.,, and Dannenhauer, D. Memory matters: The need to improve long-term memory in Ilm-agents. In Proceedings of the AAAI Symposium Series, volume 2, pp. 277-280, 2023.
Li, G., Hammoud, H., Itani, H., Khizbullin, D., and Ghanem, CAMEL: communicative agents for "mind" explo- ration of large language model society. In NeurlPS, 2023.
He, X., Zhao, K., and Chu, Automl: survey of the state-of-the-arr. Knowledge-bate based systems, 212:106622, 2021.
Li, Z., Zang. Q., Ma, D., Guo, J., Zheng, T., Liu, M., Niu, X., Wang, Y., Yang. J., Liu, J., al. Autokaggle: multi-agent framework for autonomous data science com- petitions. arXiv preprint arXiv:2410.20424 2024.
He, Z., Cao, P., Chen, Y., Liu, K., Li, R., Sun, M., and Zhao, Lego: A multi-agent collaborative framework with role- playing and iterative feedback for causality explanation generation. In Findings of the Association for Computa- tional Linguistics: EMNLP 2023, pp. 9142 9163, 2023.
Liang. T., He, Z., Jiao, W., Wang. X., Wang. Y., Wang, R., Yang. Y.. Tu, Z., and Shi, S. Encouraging divergent thinking in large language models through multi-agent debate. CoRR, abs/2305.191 191188. 2023.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. NeurIPS, 2021.
Liu, H., Simonyan, K., and Yang, Y. Darts: Differentiable architecture search. arXiv preprint 18x6:/800.00055 2018.
Hong, S., Zheng, X., Chen, J., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, K. S., Lin, Z., Zhou, L., Ran, C., Xiao, L., and Wu, C. Metagpt: Meta programming for multi-agent collaborative framework, August 01, 2023 2023.
Liu, Y., Sun, Y., Xue, B., Zhang, M., Yen, G. G., and Tan, K. C. A survey on evolutionary neural architecture search. IEEE transactions neural networks and learning sys- lems, 34(2):550-570, 2021.
Hong, S., Lin, Y., Liu, B., Liu, B., Wu, B., Zhang, C., Wei, C., Li, D., Chen, J., Zhang, J., al. Data inter- preter: An I1m agent for data science. arXiv preprint 18777 2024.
Liu, Z., Zhang, Y., Li, P., Liu, Y., and Yang, D. Dynamic Ilm- agent network: An Ilm-agent collaboration framework with agent team optimization. CoRR, abs/2310.021700 2023.
Hu, S., Lu, C., and Clune, J. Automated design of agentic systems. arXiv preprint arXiv:2408.084: 2024a.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S.. Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prab- humoye, S., Yang. Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. Self-refine: Iterative refinement with self- feedback. NeurIPS, 2023. URL http: http::/paperss /papers. nips. h wttccc/cppppa... saaeer_files/ hallles/paper/2225 /
Hu, Y., Cai, Y., Du Y., Zhu, x., Liu, x., Yu, Z., Hou, Y., Tang, S., and Chen, S. Self-evolving multi-agent collaboration networks for software development. arXiv preprint arXiv:2410. 16946, 2024b.
10

---

Multi-agent Architecture Search via Agentic Supernet
9leafffo7333tt111555555333eeee html.
Ne Zhao, K., Ma, L.,, Liu, J., Xu, F,, and Li, Y. Agentsquare: Automatic Ilm agent search in modular design space. arXiv preprint arXiv:2410.06133 2024.
Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., and Scialom, T. Gaia: benchmark for general assistants. arXiv preprint arXiv:2311.1298, 2023.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le,
Q., Hinton, G. and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint
Nakajima, Babyagi. https: / h/ggghttb.cco com/
2017.
yoheinakajjmmm/bbbyyaaa 2023.
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36, 2024.
OpenAl. Gpt-4o mini: Advancing cost-efficient intelli-
gence, 2024. URL httpt: ctwp://naaa.ooii/ee/eeeeeeeeeeee gpt-4o-ming -attommmi---ddaaaninn---o---tt--
Packer, C., Wooders, S., Lin, K., Fang. V., Patil, S. G., Stoica, I., and Gonzalez, J. E. Memgpt: Towards llms as operating systems. arXiv preprint arX1v:2310.08560, 2023.
Shinn, N., Labash, B., and Gopinath, A. Reflexion: an autonomous agent with dynamic memory and self- reflection, arXiv preprint, abs/2303. 11366,...1.66.... 11366, 2023. doi: 10.
48550/arXXi.2233..1166 URL https: / http//di.orroo/ooooo/// 10 48.48550/arrk 2303 11.30311100 11366.
Piatti, G., Jin, Z., Kleiman-Weiner, M., Schölkopf, B., Sachan, M.. and Mihalcea, R. Cooperate or collapse: Emergence of sustainability behaviors in a society of Ilm agents. Xrxiv preprint arXiv:2404. /6698, 2024.
Tang, N., Yang, C.,, Fan, J., Cao, I.., Luo, Y., and Halevy, Verifai: verified generative ai. arXiv preprint arXiv:2307.02790 2023.
Qian, C., Xie, Z.,, Wang, Y., Liu, W., Dang, Y., Du, Z.,, Chen, W., Yang, C., Liu, Z., and Sun, M. Scaling large- multi-agent collaboration. arXiv preprint arXiv:2406. 00555..5., 2024.
Wang, L., Xie, S., Li, T., Fonseca, R., and Tian, Y. Sample- efficient neural architecture search by learning actions for monte carlo tree search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5503-555 5515, 2021.
manguage-model-Beee
Qiao, S., Zhang, N., Fang, R., Luo, Y., Zhou, W., Jiang, Y. E., Lv, C., and Chen, H. Autoact: Automatic agent learning from scratch via self self-planning. In ACL. As- sociation for Computational Linguistics, 2024. URL https: / /at./vivvv.../////v///aa////aavv org/ abs/ 2401. 05268.
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y.. Zhao, W. X., Wei, Z., and Wen, J. A survey large language model based autonomous agents. Front. Comput. Sci., 18, 2024a.
Reimers, N. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.1 10084, 2019.
Wang, W., Wei, F. Dong, L., Bao, H., Yang, N., and Zhou, M. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33: 5776-5788, 2020.
Ren, P., Xiao, Y., Chang, X., Huang, P.-Y,, Li, Z., Chen, X., and Wang, X. A comprehensive survey of neural architec- ture search: Challenges and solutions. ACM Computing Surveys (CSUR), 54(4):1-34 2021.
Wang. X., Wei, J., Schuurmans, D., Le, Q. V., Chi, H., Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. In The Eleventh International Conference on Learning Representations, 2023a.
Reworkd. Agentgpt. reworkd/ Aaa.d//ggentt..... 2023.
com/
Richards, T. B. and et al. Auto-gpt: An au- tonomous gpt-4 experiment. Signif f cant
com/ 2023.
Wang. Y., Shen, T., Liu, L., and Xie, J. Sibyl: Simple yet effective agent framework for complex real-world reasoning. arXiv preprint #7Xiv:2407.10718. 2024b.
Roy, S. and Roth, D. Solving general arithmetic word problems. arXiv preprint arXiv:1608.0144: 2016.
Wang, 7.,, Mao, S., Wu, W., Ge, T., Wei, F., and Ji, H. Unleashing cognitive synergy large language mod- els: A task-solving agent through multi-persona self- collaboration, July 01, 2023 2023b. work in progress.
Saad-Falcon, J., Lafuente, A. G., Natarajan, S., Maru, N., Todorov, H., Guha, E., Buchanan, E. K., Chen, M., Guha, N., Ré, C., et al. Archon: An architecture search framework for inference-ting techniques. arXiv preprint arXiv:2409. 12544, 2024.
Wei, J.. Wang, X., Schuurmans, D.,, Bosma, M., Ichter, B., Xia, F, Chi, E., Le, Q., and Zhou, D. Chain-of-thought

---

Multi-agent Architecture Search via Agentic Supernet
prompting elicits reasoning in large language models, January 01, 202200222222 2022.
Zhang, J., Xiang, J., Yu, Z., Teng, F., Chen, X., Chen, J., Zhuge, M., Cheng, X., Hong. S., Wang, J., al. Aflow: Automating agentic workflow generation. arXiv preprint 2.Xiv:2410.107622 2024c.
White, C., Neiswanger, W., and Savani, Y. Bananas: Bayesian optimization with neural architectures for neu- ral architecture search. In Proceedings of the AAAI con- ference on artificial intelligence, volume 35, pp. 10293- 10301, 2021.
Zhang, Z., Zhang, A., Li, M., and Smola, A. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.0324 034993,, 2022.
White, C., Safari, M., Sukthanker, R., Ru, B., Elsken, T., Zela, A., Dey, D., and Hutter, F. Neural architec- ture search: Insights from 1000 papers. arXiv preprint arXiv:2301.087777 2023.
Zhao, Q. Wang. J., Zhang, Y., Jin, Y., Zhu, K., Chen, H., and Xie, X. Competeai: Understanding the competition behaviors in large language model-based agents. arXiv preprint arXiv:2310.1777 17512, 2023.
Wu, Q.. Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E.,, Li, B., Jiang, L.., Zhang, X., and Wang. C. Autogen: Enabling next-gen I1m applications via multi-agent conversation framework, August 01, 2023 2023.
Zhong, W., Guo, L.,, Gao, Q. Ye, H., and Wang. Y. Memo- rybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, volume 38, pp. 19724-19733 2024.
Zhou, W., Ou, Y., Ding, S., Li, L., Wu, J., Wang, T.,
Wu, X., Shen, Y., Shan, C., Song, K., Wang, S., Zhang, B., Feng, J., Cheng, H., Chen, W., Xiong, Y., et al. Can graph learning improve planning in Ilm-based agents? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Chen, J., Wang, S., Xu, X., Zhang, N., et al. Symbolic learning enables self-evolving agents. arXiv preprint 2024.
Zhu, J.-P., Cai, P., Xu, K., Li, L., Sun, Y., Zhou, S., Su, H., Tang. L., and Liu, Q. Autotqa: Towards autonomous tabular question answering through multi-agent large lan- guage models. Proceedings of the VLDB Endowment, 17 (12):3920-3933, 2024a.
Xie, S., Zheng, H., Liu, C., and Lin, L. Snas: stochastic neu- ral architecture search. arXiv preprint 2018.
Yan, Z., Dai, X., Zhang, P., Tian, Y., Wu, B., and Feiszli, M. Fp-nas: Fast probabilistic neural architecture search. In Proceedings of the IEFF/CV Conference on Computer Vision and Pattern Recognition, pp. 15139-15148, 2021.
Zhu, Y., Qiao, S., Ou, Y., Deng, S., Zhang, N., Lyu, S., Shen, Y., Liang, L., Gu, J., and Chen, H. Knowagent: Knowledge-and eugee-agmeetted planning for l1m-based agents. arXiv preprint arXiv:2403.0311 2024b.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint 24X1v22412.15115, 2024.
Zhuge, M., Wang, W., Kirsch, L.,, Faccio, F., Khizbullin, D., and Schmidhuber, J. Gptswarm: Language agents op- timizable graphs. In Forty-first International Conference Machine Learning, 2024.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023.
Zoph, B. Neural architecture search with reinforcement learning. arXiv preprint arXiv:6611001578 2016.
Yuan, S., Song, K., Chen, J., Tan, X., Li, D., and Yang, D. Evoagent: Towards automatic multi-agent gen- eration via evolutionary algorithms. arXiv preprint arXiv:2406. 14228, 2024.
Zhang, G., Yue, Y., Li, Z.,, Yun, S., Wan, G., Wang, K., Cheng, D., Yu, J. X., and Chen, T. Cut the crap: An economical communication pipeline for Ilm-baser multi- agent systems. arXiv preprint arXiv: arXiv:2410.02506, 2024a.
Zhang, G., Yue, Y., Sun, X., Wan, G., Yu, M., Fang. J., Wang, K., and Cheng, D. G-designer: Architecting multi- agent communication topologies via graph neural net- works. arXiv preprint arXiv:2410.11 1782, 2024b.
12

---

Multi-agent Architecture Search via Agentic Supernet
A. Notations
Table 5. Notations and Definitions
Notation
Definition
O
{M,)) P. PATT
An agentic operator comprising set of LLM instances, textual prompt, and set of temperature settings.
M
An individual LLM instance.
M
The set of all feasible LLMs.
P
A textual prompt used as input to the LLM.
P
The feasible space of prompts.
T
The temperature setting of the LLM.
C
The set of all feasible agentic operators.
9 = [V.E)
A multi-agent system represented as graph with vertices V and edges E.
A {T,, 0} = (m)00)0))))
An L-layer probabilistic agentic supernet, consisting distribution π and a set of feasible operators O.
The distribution associated with the agentic supernet.
a)
The utility evaluator of with respect query q and answer The cost evaluator of G with respect to query g and answer a. The controller network parameterized by Φ. Execution of g to produce the answer a. The selected operators at layer & of the agentic supernet A.
a)
Co
e(all))
Ve
Oxi
The early-exit operator.
v()
The text embedding function. The gradient of the loss L with respect to the distribution æ.
V.L
VoL
The textual gradient of the loss L with respect to the operators O.
B. Technical Details
B.1. Operator Space
In this section, we detail the initialization of operator nodes as follows:
1. Chain-of-Thought (CoT), CoT (Wei et al., 2022) reasoning encourages the LLM to think step by step rather than
directly outputting an answer. This approach enhances its capability to solve complex problems through intermediate reasoning steps, improving task handling and providing greater transparency in the decision-make makinn process.
2. LLM-Debate. LLM-Debate (Du et al., 2023) allows multiple LLMs debate, leveraging diverse perspectives to identify better solutions. In practice, initialize three debaters and permit up to debate rounds.
3. Seff-Consistency. Adopting the methodology from Wang al. (2023a), this operator aggregates five CoT reasoning paths and determines the final answer through majority voting.
4. Self-Refine. Following Madaan et al. (2023), this operator initially generates answer using CoT reasoning, then prompts the agent to self-reflect iteratively. We set maximum of five refinement iterations.
5. Ensemble. Inspired by LLM-Blender (Jiang et al., 2023), this operator involves three LLM-pooered agents from different sources outputting answers to the same query. The pairwise ranking is used to evaluate and aggregate their responses into a final solution.
6. Testing, Following the test designer in AgentCoder (Huang et al., 2023), this operator is used for generating test cases for generated code.
7. ReAct. Following (Yao et al., 2023), this operator enables the agent to leverage versatile tools, including code interpreter, web searching, external knowledge database, etc., to handle diverse user demands.
13

---

Multi-agent Architecture Search via Agentic Supernet
8. Early exit. We introduce the early exit operator, which interrupts the multi-agent architecture sampling process and enables the depth of the agentic supernet be query-dependent.
We respectfully note that the selection of these operators is highly customizable, allowing users the flexibility to incorporate their desired operators into the operator repository of MaAS.
B.2. Embedding Function
Following established practices (Feng et al., 2024), we first employ an LLM to generate a comprchensive profile description for each operator. Subsequently, lightweight text embedding model (in our case, MiniLM (Wang et al., 2020)) is used to encode the profile into fixed-dimensional embedding. The prompt for generating the operator profile is as follows:
Embedding Prompt
prompt "*uYou are a highly proficient expert in designing and defining operators for large language models (LIMs) Your primary objective is to aaticulousi generate the 'description" and 'interface' fields for specified operator based on its provided Python empeementtiion The generated content must be accurate, efficient, and precisely reflect the functionality of the operator's coce.
To ensure consistency, quality, and adherence to best practices, refer to the following examples previously defined operators:
"Generate": {
"descriptien": "Generates anything based on customized input and instruction " "interface"; "generate (input: str, instructions str) dict with key response' of type str"
"ScEnsemble";
"descriptien" in" "Uses sel self-conssstence to select the solution that appears most frequently in the solution list, improving the selection to enhance the choice of the best solution." ", "interface"; sec_emleelle (solutions: List problem: str) -> dict with key 'response' of type str"
Now, given the follewing operator code. This code encompasses the function signature parameters with type annotations, internal logic, and return statements essential for comprehensively understanding the operator's purpose and behavior, Please provide its 'description' and 'interface' fields in the same format. loperator coce]
B.3. Textaul Gradient
The implementation of the textual gradient component is partially adapted from the repositories https://github com/ SShengranHaaADASS /DAS/ and https: //ghthub...//////////////t com/tsinnn com/tsinghua www-lfillallaaaaaae labbaagetsquare, We would like to explicitly acknowledge this contribution and express our sincere gratitude to the authors for their open-source efforts.
Textual Gradient
base =
# Overview
You expert machine Learning researcher specializing in designing agentic systems. Your objective is create building blocks such as prompts and control flows within these systems to solve complex tasks. specifically, you aim to
14

---

Multi-agent Architecture Search via Agentic Supernet
design an optimal agent that periorms exceptionally on the HumanEval benchmark, The Humanival dataset eval uates code generation capabilities in Al systema, consisting of 164 hand-crafted Python programming problems. Each problem mude function signature with docstring describing the task Test cases to verify functional correctress
# Example Questicn from HumanEval
[An example questien from HumanEval dataset here]
Operator code template: cl aas Operator:
def _init__seff, 11m: LIM, name: str):
self.name name
self.ilm 11m
def _aall_ _ (sels, *args, **kwargs) raise NotimplementedBrror
async def _fill_nndeesellf op_class, prompt, mode-None, **extra_kwarge)= kaa_gs ("context"; prompt, "1lm": self.1lm}
if mode: (all__wargs("nenn ["mode"] mode
fill_kwargs update (extra_kwarga
node await ActionNode. from_pydanti (op_clasp .iill (+ffil1_laaaar return node. instrnutt_oonteeetoo oam ()
cl class Generate0p (
response: str Field (default="", description-"Your YYour solution for this problem")
class Generate (Operator):
SENERATE_PRONE PROMPT =
You are tasked with solving the following Python programming problem. Generate complete, systactically correct Python function that strictly adheres to the given zequirements.
Problem: (input)
Follow these steps:
1. Analyze the problem requirements and identify edge cases
2. Design solutien that passes all implied test cases and comments
3. Implement the function with clear variable
Ensure:
The code directly implements the requested functionality
All parameters and return types match the problem specification
Exception handling for edge cases included when necessary "
def _iitt_ (sclf, 11m: LIM, name: str = "Gonerate"):
super init i1it__mmm, name)
async def call (self, input: str, mode: str None) prompt response = await self. fill nede (GenerateOp, prompt, mode="xml (ill")
BL__GGNERAT___RRRRT format (input=input)
return response
# Discovered architecture archive Here is the archive of the discovered operator architectures: [ARCHIVE]
Output Instruction and Example:
The output should be JSON object with the folloxing structure. The first key should be ("thought"), and it should capture your thought process for designing the
15

---

Multi-agent Architecture Search via Agentic Supernet
next operator. The second key ("description") corresponds to the brief description of your next operator. Minally, the last key ("code") correspends to the exact operator and its prompt in Python code that you would like to try. You must write COMPLETE CODE in "code": Your code will be part of the entire project, so please implement complete, reliable, reusable code snippets.
thought: Captures your thought process for designing the next operator.
Roason about what the next interesting operator should be.
Desoribe your reasoning and the overall concept behind the operator design. Detail the implementation steps.
description: A brief description of your next operator.
code: The exact operator and its prompt in Python code. Ensure the code is complete, reliable, and reusable.
Here is an example of the output format for the next operator: [operator_example]
You must strictly follow the exact input/outpa interface used above. Also, it could be helpful to set the LIM's s role and temperature to further control the LLM's response. DON'T try to use sone function that doesn't exist. In __aal_00)) (0,, you need to specify the instruction, input information, the prompt and the required output fields class for operators do their specific part of the architecture.
# Your task
You are highly praficient in prompting techniques and well-versed with agentic systems from academic literature. Your goal is to maximize performance metrics by proposing innovative and effective new operators. Instrustions:
1. Analyze the Discovered Operators: Carefully review the operators in the archive
to identify strengths, weaknesses, and areas for improvement.
2. Draw Insights: Extract lessons and insights from existing operators to inform the design of next operator.
3. Innovate: Think creatively to design an operator that addresses current limitations or explores new functionalities drawing inspiration from related agent papers or other research areas.
Design the Operator: Propose the
operator's 'thought', 'description', and code' following the speci fied format.
5. Ensure Completeness: The generated code must be complete, reliable, and reusable, fitting seamlessly into the existing architecture.
Execution Steps:
1. Insert Operator Code: Replace the '[ARCHIVE] and [[operator_example] placeholders with actual content as needed.
2. Generate Output: Produce the 'thought', 'description', and 'code' fields for the operator, ensuring adherence the guidelines.
3. Validate Output: Ensure the generated JSCN is correctly formatted and the code syntactically and functionally correct.
THINK OUTSIDE THE BOX and leverage interdisciplinary insights to enhance the agentic system's capatilities.
C. Experimental Details
C.1. Dataset Statistics
Building upon established methodologies in workflow automation (Saad-Falcon et al., 2024; Hu et al., 2024a; Zhang et al., 2024c), we divide each dataset into training and test sets using a TRAIN:TEST ratio of 1:4. For the MATH benchmark, we adhere to (Hong et al., 2024), selecting subset of 617 harder problems spanning four representative categories, Combinatorics Probability, Number Theory, Pre-algebra, and Pre-caleulus, all at difficulty level The dataset statistics
16

---

Multi-agent Architecture Search via Agentic Supernet
included in Table 6.
Table 6. Dataset Statistics.
Domain
Dataset
#Train
#Test
Metric
HumanEval
33
131
pass@1
Code Generation
MBPP
86
341
pass@1
GSM8K
264
1055
Accuracy
Math Reasoning
MATH
119
486
Accuracy
MultiArith
150
600
Accuracy
Tool use
GAIA
94
372
Accuracy
C.2. Baseline Setups
In this section, we provide a detailed description of the configurations for baseline methods:
1. COT. Chain-of-Thought (CoT) prompting guides LLM agents to break down reasoning into sequential steps rather than generating direct answers. We employ the implementation from (Zhang et al., 2022).
2. ComplexCoT. We follow the official implementation available at https: htttps:/gittub..om www.hhuwb...oo/raaaeeee.e Prased-Promptine Promptig /main.
3. Self-consistency. To enhance robustness, aggregate five CoT-generated solutions.
4. L.M--Deate. We instantiate five I.L.-agents, each assigned a distinct role, which participate in up to two rounds of debate, after which the final decision is determined via majority voting. The implementation based https: 1ebet///emmmdddeeee_.mee debate.
5. LLM-BBender. We choose two gpt gpt-4o-ming one Qwen-2.5-725, and one 1lama-3. 3-10bb 11a70 to empower LLM-BBender (Jiang al., 2023).
6. DyLAN, We directly utilize the implementation from (Liu et 2023).
7. AgentVerse. The experimental setup follows the original implementation from (Chen et al., 2023b).
8. MacNet. For MacNet (Qian et al., 2024), we adopt the "MacNet-MESH" variant, which corresponds to fully
connected network topology.
9. GPTSwarm. The method is implemented in accordance with the original settings described in (Zhuge et al., 2024).
10. AutoAgents. We adhere to the official configuration specified (Chen et al., 2023a).
11. ADAS. The implementation details are directly inherited from (Hu et al., 2024a).
12. AgentSquare. We utilize the modular search framework introduced (Shang et al., 2024). The base LLM remains fixed gpt-4o-ming with early stopping a patience of iterations.
13. AFlow. In (Zhang et al., 2024c), AFlow operates with both gpt opt-4o-min 4o-mini and claude-3, 3. 5-sonnet. To maintain fairness under homogeneous conditions, we restrict AFlow to -ot-4o-ttt 4o-mini and set 2TX.ITERA
D. Supplementary Results
We have visualized the evolution of operator sampling trends as the sampling count increases, in Figure 10. MaAS learns to avoid overly confident early stopping and instead prioritizes testing and self-refinement in deeper layers.
17

---

Multi-agent Architecture Search via Agentic Supernet
Table Cross-model transferability of MaAS. We optimize the agentic supernet with gpt apo--4ommii and report the performances before and after equipping the LLM backbones with optimized agentic supernet.
Dataset
HumanEval
LLM Backbone
gpt apt-4o-mi gpt-4o-mini Owen-2.5-725 5-72b
11ama-3 3. 3.1....0000 1-70b
vanilla
87.08
85.60
80.06
+MaAS
92.85
90.14
85.26
Dataset
MATH
LLM Backbone
gpt-4o-ming -4o-mini
Qwen 50b
11ama -3, 1..0-
vanilla
46.29
63.80
31.93
+MaAS
51.82
69.35
42.97
Table Cross-dataset transferability of MaAS. "MATHI->GSMBE denotes optimizing the agentic supernet on MATH and evaluating it on GSM8K, with similar notation applied other cases.
Transfer
ESM8K->AAT HumanEval MATH
GPTSwarm 89.96
45.18
47.92
AFlow
91.95
49.39
47.15
MaAS
92.80
51.02
50.27
Layer 1
Layer 2
18.32%
16.03%
32.82%
42.75%
13. 74%
16.79%
0%
8.4%
14.5%
24.43%
12.21 %
1/O
CoT
SC
Ensemble
Testing
Refine
1/O
CoT
SC
Ensemble
Testing
Refine
EarlyExit
EarlyExit
Layer
Layer 4
10%8.75%
1.14%
12.5%
16.88%
28.13%
42.05%
15.91%
19.38%
7.5%
14.37%
13.61%
CoT
SC
Ensemble
Testing
Refine
VO
CoT
SC
Ensemble
Testing
Refine
Early
EarlyExit
Figure 8. The wise distribution of MaAS on HumanEval benchmark without Debate operator.
18

---

Multi-agent Architecture Search via Agentic Supernet
Layer (Debate operator added)
Layer 4 (Debate operator added)
13. 13.79%12.0799
0.95%
10.48%
9.48%
22.4%%
13.33%
35
24000
13.79%
6.03% 14.66%67.669
11.3%
18.19%
NO
CoT
SC ExilyExit
Ensemble
Testing
Refine
DO
CoT
SC EarlyExit
Ensemble
Testing
Refine
Debate
Debate
Figure The layer-wise distribution MaAS on HumanEval benchmark with Dehate operator. Note that the agentic supernet optimized with other operators, while the Debate operator is introduced only during the inference stage. can be observed that, despite not being exposed to this operator during training. MaAS can still reasonably select it during the multi-agent architecture sampling process.
Operator Probuldity Deasly Distribation
Operalor Probalaily Deasit Distrilation
Layer I is Layer (heempl 188)
Layer Ias Layer 3thanppl 200)
6:5
0.1
666
Operator Deadly
Operator Beadly
Layer vel Laye (homple 518)
Layer 1c (kampl 116)
Eapt
Leer
6.0
Pr
600
pedaa
cose
Figure 10. The evolution of operator sampling trends the sampling count increases.
19


Reviews:
[{"review_id": "XaHQoIRkWS", "reviewer": "Reviewer_ZEuX", "summary": "This paper introduces a novel mechanism called the \"Agentic Supernet\" to enable dynamic inference within multi-agent systems. Unlike traditional fixed agentic systems, the supernet and its subnet agents, which are instantiated through parameterized sampling, allowing for adaptive inference across a variety of tasks and difficulty levels. This represents a significant contribution, as it attempts to address a core limitation of workflow-based agentic systems, which often lack sufficient adaptability to diverse tasks and complexities. Extensive experimental results demonstrate the method’s effectiveness, showcasing its ability to deliver both high-performing and cost-efficient agentic systems owing to its adaptive nature.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**`Weakness 1: Insufficient Clarity in Presentation`**\n\nThank you for the insightful comment! Each layer shares the same set of operators, except for the first layer, where the early-exit operator is excluded. The operators in each layer produce outputs in parallel, which are then concatenated and passed as input prompts to the next activated operator, following standard practices in Mixture-of-Agents (ICLR 2025) and GPTSwarm (ICML 2024).\n\n---\n>**`Weakness 2: Lessons from DARTS+`**\n\nThank you for this highly insightful discussion and enlightenment! \n\n- **Collapse Issue** We empirically demonstrate that the agentic supernet does not encounter the collapse issue observed in traditional DARTS, as presented in **Table-collapse** (https://anonymous.4open.science/r/maas-rbt/table-collapse.md).\n- **Overfitting Issue** Intuitively, we argue that MaAS's agentic supernet is inherently resistant to overfitting due to two key factors:  (1) **Query-aware sampling**. In fact, prior work on customizable NAS like GRACES [1], has demonstrated the advantage of input-dependent supernet in out-of-distribution generalizability.  (2) **Cross-domain training data**. The training data itself can span multiple domains (e.g., GAIA benchmark includes web searching and file analysis), inherently promoting cross-domain generalization.\n- **Biased Sampling Issue** We commit to incorporating this intriguing discussion in our revised manuscript, borrowing lessons from FairNAS/DARTS-/DARTS-PT.\n\n[1] Graph Neural Architecture Search Under Distribution Shifts, ICML'22\n\n---\n>**`Weakness 3.1: Transferability of agentic supernet`**\n\nFollowing your suggestion, beyond the numerical transferability study in Table 8, we further visualize the underlying mechanism of MaAS’s transferability, with results and analysis presented in **Figure-transfer** (https://anonymous.4open.science/r/maas-rbt/figure-transfer.md).  \n\n---\n>**`Weakness 3.2: Cross-domain optimization of agentic supernet`**\n\nWe would first like to point out that, the GAIA benchmark we used inherently falls under cross-domain optimization (web searching + file reading). To further address your concerns, we report the results of training the agentic supernet under a math/coding cross-domain setting:\n\n*(M->MATH, G->GSM8K, H->HumanEval)*\n|Train on|Test on|Perf.|\n|-|-|-|\n|M|M|51.82|\n||G|92.80|\n|M+G|M|51.66|\n||G|93.70|\n|H|H|92.85|\n||M|50.27|\n|H+M|H|93.05|\n||M|52.69|\n\n\nNotably, when trained on the HumanEval+GSM8K mixture, the performance surpasses that of training on HumanEval or MATH alone.\n\n---\n>**`Weakness 4: Ablation on (1) supernet and (2) query-based sampling`**\n\nTo validate that the driving force of MaAS relies not only on query-based sampling but also on the agentic supernet, we construct a baseline called Agent-Archive, which consists of an agentic system archive populated with operators and agentic workflows. Results are in **Table-archive** (https://anonymous.4open.science/r/maas-rbt/table-archive.md).\n\n---\n>**`Comment 1`** How agentic supernet shifts the focus from cost reduction to adaptability\n\nWe respectfully state that MaAS's adaptability stems from its query-aware supernet sampling mechanism. \n\nBy analogy, MaAS removes DARTS's final one-shot pruning step that determines a fixed CNN. Instead, during actual usage, it dynamically customizes each layer’s kernel size/skip connections/pooling operators, as well as the  network depth, based on each input (e.g., image). This allows MaAS to retain DARTS's advantage of reducing training costs while simultaneously enhancing adaptability in agentic scenarios.\n\n---\n>**`Comment 2: Possibility of supergraph`**\n\nThank you for your inspiring thoughts! We will incorporate this interesting discussion in our updated manuscript.\n\n---\n>**`Comment 3: Specifying underlying LLM of baselines`**\n\nThank you! This has been specified in Line 277 and Appendix C.2. Besides, in Table 2, GPt-4o-mini is used for TapaAgent/Sibyl and GPT-4 for AutoGPT.\n\n---\n>**`Comment 4: Circle size in Figure 4`**\n\nThank you! The circle size in Figure 4 is proportional to the value of y-axis.\n\n---\n>**`Comment 5: Removal of textual gradient in Table 4`** \n\n\nAfter carefully reviewing Table 4, we identified a typo that exaggerated the impact of removing textual gradient on inference cost. Specifically, 0.09 should be corrected to 0.90. To provide a more detailed analysis, we present Table 4 with standard deviation included for finer observation (https://anonymous.4open.science/r/maas-rbt/table4-stdev.md). We believe the cost reduction occurs because textual gradient increases the length of prompts in MaAS.\n\n---\n>**`Comment 6: Learning visualization of agentic supernet`**\n\nFollowing your insightful suggestion, we have visualized the evolution of operator sampling trends as the sampling count increases (https://anonymous.4open.science/r/maas-rbt/figure-learning.pdf). It learns to avoid overly confident early stopping and instead prioritizes testing and self-refinement in deeper layers.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No proofs.", "experimental_designs_or_analyses": "yes. All experimental designs are sound and valid.", "relation_to_broader_scientific_literature": "The work is related recently emerging research topic of automating agent design. Additionally, the concept of the Agentic Supernet draws inspiration from supernet techniques in Neural Architecture Search (NAS).", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strengths**\n\n- The proposal of an Agentic Supernet to facilitate adaptive inference in multi-agent systems across diverse tasks and difficulty levels is both novel and compelling. This approach introduces a fresh perspective on addressing the adaptability challenges inherent in traditional agentic frameworks.\n- The experimental design is sound, following practices from prior work while expanding the scope through the inclusion of additional benchmarks and comprehensive analyses.\n- The results and examples effectively demonstrate the method’s capability to adapt agent configurations to varying task difficulties, achieving high performance while maintaining cost efficiency. This underscores the practical utility of the proposed framework besides its novelty in scientific ideas.\n\n**Weaknesses**\n\n1. Insufficient Clarity in Presentation\n   - The description of the supernet’s layer structure lacks clarity, particularly regarding whether each layer contains all operators from the operator space.\n   - The process of handling multiple activated operators within a layer and how their outputs are integrated as inputs to subsequent layers remains ambiguous.\n\n2. Drawing from supernet-related work in NAS, such as Liang et al. \"DARTS+\", some learned lessons can be discussed:\n     - Collapse Issue: While the absence of \"skip\" operations in the proposed supernet may avoid the collapse issue in the proposed method, this could be validated through longer runs to confirm its robustness.\n     - Overfitting is also a known challenge in NAS supernets like DARTS. Seemingly the adaptive sampling network to the query will mitigate this issue. But some discussions on it will be interesting.\n     - It is also known that DARTS has a biased sampling issue (e.g. discussed in Chu et al. \"Fair DARTS\"). I.e. Early over-sampling of certain paths (nodes) could bias the optimization process. Because these paths (nodes) are sampled and updated at the early stage, the subsequent sampling will further favor those paths (nodes), even other paths (nodes) may have more potential. A discussion of this risk, along with potential investigation and mitigation strategies as future work, would enhance the paper’s depth.\n\n3. The ability of the resulting agentic system to dynamically adapt to multiple tasks is a highly appealing feature, briefly evidenced in Table 8 through transferability experiments. However, further elaboration and concrete examples would provide deeper insight into this capability. Additionally, an experiment training the supernet across multiple domains simultaneously—rather than a single domain—could be interesting to reveal whether such a setup improves generalization compared to domain-specific training.\n\n4. The paper presents two mechanisms that enable adaptive inference in agentic systems: (1) the Agentic Supernet and (2) query-based sampling during inference. The current evaluation showcases the combined effect of both (1+2), suggesting that the supernet’s effectiveness may depend on query-based sampling. However, an alternative approach—relying solely on (2) with an archive of agentic systems tailored to specific queries—could also achieve adaptive inference without the supernet. This raises a critical question: Is the supernet the primary driver of the method’s superior performance, or does the query-based sampling mechanism play the dominant role? The paper would benefit from a discussion or ablation study disentangling these contributions. Such an analysis would enhance the work’s depth and bolster the persuasiveness of the central claim regarding the supernet’s significance.", "other_comments_or_suggestions": "- In NAS, supernets are primarily employed to reduce evaluation costs during architecture optimization. In contrast, this work leverages the supernet concept to enable dynamic inference tailored to varying queries. A detailed discussion highlighting this distinction—particularly how it shifts the focus from cost reduction to adaptability—would enrich the paper’s contribution and clarify its novelty within the broader literature.\n- The choice of a feed-forward supernet is interesting. Potentially, as the information flow does not appear to strictly depend on a layer-by-layer structure, it might be also interesting to have a directed acyclic graph as a \"supergraph\". Including ideas like this, some future works can be discussed to inspire other researchers.\n- Several baselines lack specificity regarding the underlying LLM used.\n- In Figure 4, the meaning of circle size is not explained.\n- Table 4 reveals an intriguing result: removing the textual gradient leads to a significantly lower-cost agentic system on HumanEval, despite a performance drop. Discussing more on this could provide valuable insights into the method’s cost-performance dynamics.\n- The learning progress of the proposed algorithm is not shown. Including a visualization or analysis of the optimization process (e.g., performance convergence, operator sampling trends over time) would offer a deeper understanding of how the Agentic Supernet evolves and adapts.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DeCmmNxuJF", "reviewer": "Reviewer_43f8", "summary": "The paper proposed a novel automated multi-agent framework through agentic supernet (MaAS), which both delivery satisfactory performance and resource allocation efficiency for user queries across different domains. The framework was comprehensively evaluated on six benchmark tasks with comparison to about 15 baselines and state-of-the-art agentic systems. Coding scripts are also provided through  anonymous GitHub.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Sincere thanks for the thoughtful and constructive reviews of our manuscript! Based on your questions and recommendations, we give point-by-point responses to your comments.\n\n---\n>**`Weakness 1: Lack of variance estimation`**\n\nThank you for your insightful suggestion! In fact, all results in Table 1 represent the average of three runs. Following prior practices in GPTSwarm (ICML 2024) and AgentPrune (ICLR 2025), we chose to present the performance difference from the vanilla LLM in the main result table (rather than reporting standard deviations).\n\nTo further address your concern, we provide a version of Table 1 with standard deviations, as shown in **Table1-stdev** (https://anonymous.4open.science/r/maas-rbt/table1-stdev.png). We sincerely hope this resolves your concern.\n\n\n---\n>**`Weakness 2`** It seems the experiments are not cross-validated (a single train-test split) or randomized with multiple attempts.\n\nThank you! The results are reported as the average of three runs, and we have supplemented them with standard deviations in our response to Weakness 1.\n\n---\n>**`Weakness 3: Clarification on Hyperparameters`** Could you pls clarify which benchmark task yield the optimal parameter set in Figure 7? And are the optimal parameters subject to change given a different task?\n\nThank you for your insightful inquiry! In fact, our proposed MaAS has not undergone extensive hyperparameter tuning, and the current setting ($L=4, K=4$) is not necessarily the optimal one in terms of performance. As shown in Figure 7, increasing $L$ from 4 to 8 leads to a 0.9% improvement in pass@1.  \n\nHowever, we have consistently observed the following trends across multiple benchmarks:  \n\n- Beyond $L=4$, the performance gain from increasing the supernet depth becomes marginal.  \n- Beyond $K=4$, the performance gain from increasing the sampling times also plateaus.  \n\nAbove is the rationale behind our choice of $L=4, K=4$. Since our current sensitivity study is based on HumanEval, we further provide a sensitivity analysis on GSM8K to substantiate our findings:\n\n|$L$|2|4|6|8|\n|-|-|-|-|-|\n|Perf.|89.09|92.30|92.88|93.50|\n|Inf. Cost|0.34|0.48|0.67|0.89|\n|**K**|**2**|**4**|**6**|**8**|\n|Perf.|91.6|92.30|92.35|92.83|\n|Inf. Cost|0.48|0.48|0.49|0.49|\n\n---\n>**`Comment 1: Typos and Minors`**\n\nThanks immensely for pointing out! We will add the missing left parenthesis and underline the tied runner-up in Table 2.\n\n---\n>**`Suggestion 1: Ablation & Sensitivity analysis on other benchmarks`**\n\nThank you for your valuable insights! In our response to **Weakness 3**, we have supplemented the sensitivity analysis for GSM8K. Due to time constraints and API resource budget, we were unable to promptly provide ablation/sensitivity analyses for all datasets. However, we sincerely commit to including the corresponding analyses for other datasets in the appendix of the camera-ready version. Once again, we truly appreciate your feedback!\n\n---\n>**`Suggestion 2: More details on transferability and inductive analysis`**\n\n\nThank you for your valuable feedback! We sincerely commit to including additional details on transferability and inductive analysis in the appendix. \n\n\n\n\n---\n>**`Question 1`** How will the proposed framework scale with number of agentic operators? Could you elaborate more on the potential impact of search efficiency?\n\n\nTo address your question, we gradually increase the number of operators on HumanEval and report the performance, cost, and time-related metrics as follows:  \n\n|operator_num|score |cost ($) |infer-time|Train-time|\n|-|-|-|-|-|\n| 3| 89.31 | 0.00093 | 6min47s | 18min |\n|4|91.6|0.00088|8min|20min|\n|5|92.13|0.00121|8min|22min|\n|6|92.36|0.00117|7min|25min|\n|7|92.85|0.00101|10min|26min|\n|8|93.89|0.00113|11min|29min|\n\n\nAs observed, with more operators, performance exhibits a steady improvement, while cost remains largely stable, and training/inference time does not increase significantly. We believe this demonstrates the efficiency advantage of the agentic supernet.\n\n---\n>**`Question 2`** In Figure 7, the performance of sampling times K does not monotonically increasing, e.g. dip at K=6. What's the possible reason and indication?\n\nYour keen insight is truly appreciated! To further investigate this phenomenon, we conducted a finer-grained hyperparameter analysis on $K$, with results summarized in the table below:  \n\n|K|2|3|4|5|6|7|8|9|  \n|-|-|-|-|-|-|-|-|-|  \n|Perf. (3-run avg.)| 89.50 | 91.04 | 92.28 | 91.38 | 91.17 | 92.01 | 92.45 | 91.55 |  \n| Perf. (5-run avg.) | 89.13 | 91.10 | 92.05 | 91.30 | 91.71 | 92.10 | 92.05 | 92.14 |  \n\nThe results indicate that when $K\\geq4$, MaAS's performance stabilizes within the range of 91.20–92.20 on HumanEval, suggesting that the benefit of additional sampling saturates at $K=4$, with subsequent fluctuations remaining within a normal range. We hope this properly addresses your concern.", "claims_and_evidence": "Yes, most claims are well-supported by either reference literature or experiment results.\n\nThe only complain lies in lack of variance estimation or statistical significance of performance results in Table 1, given some of the best and runner-ups are relatively close to each other, which might weaken the claims that the proposed MaAS wins over all other baselines. The experiments are not cross-validated or randomized with multiple attempts.", "methods_and_evaluation_criteria": "Yes. The proposed methods and evaluation criteria make sense for the problem. Evaluation metrics of benchmark datasets and tasks are well referenced. The benchmarks covers various datasets and tasks. The baseline methods include single agent system, hand-craft multi-agent systems and autonomous multi-agent systems. \n\nEvaluation is comprehensive including not only task performances and computation costs of training and inference, but also parameter sensitivity analysis and ablation analysis.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "Yes. \n- It seems the experiments are not cross-validated (a single train-test split) or randomized with multiple attempts. Thus the results in table 1, 2 and 3 lack the confidence intervals. The paper is very close to strong accept if CIs are provided.\n- It seems the optimal parameter set L, $\\lambda$, and K are set the same for all tasks. Could you pls clarify which benchmark task yield the optimal parameter set in Figure 7? And are the optimal parameters subject to change given a different task?", "relation_to_broader_scientific_literature": "The key contributions of the paper lie in proposing a novel multi-agent framework with agentic supernet, along with comprehensive evaluations with regards to state-of-the-art agentic AI frameworks, which could potentially be a paradigm shift from seeking a single optimal system (e.g. CoT, ComplexCoT) to dynamically and autonomously optimizing a distribution of agentic architectures (various LLMs and tools). The computational efficiency of the proposed framework is also of great practical value, which could be inspiring future agentic AI research.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The paper is well-structured with logic flow and nice graph illustrations. I enjoy reading it very much. My questions raised during reading are well addressed in the subsequent sections. Literature review and existing state-of-art methods are well-summarized in a comparative fashion with close relation to the proposed framework. The evaluation is comprehensive with comparison to meaningful baseline methods on popular benchmark tasks. Analysis and experiment results in the supplementary also provide inspiring viewpoint in results interpretation.", "other_comments_or_suggestions": "Comments on typos and minor issues:\n- Definition 3.2, missing a left parenthesis \n- Missing underlines for some tied runner-ups in table 2.\n\nSuggestions:\n- It seems the cost analysis (Figure 4,Table 3), parameter sensitivity analysis (Figure 7) and ablation analysis (Table 4) is limited to MATH and HumanEval benchmarks only. If it is the case, it would be great if the authors could share the corresponding analysis for all benchmarks in supplementary for a thorough and complete benchmark analysis. If not, please help clarify in the paper if they were averaged across benchmarks. \n- Some analysis in Part 4.5 Framework analysis might be too compressed even with the supplementary materials, especially transferability analysis and inductive analysis. If more details could be provided on experiment settings in supplementary, it'll be a good reference for other researchers.", "questions_for_authors": "- How will the proposed framework scale with number of agentic operators? Could you elaborate more on the potential impact of search efficiency? \n- In Figure 7, the performance of sampling times K does not monotonically increasing, e.g. dip at K=6. What's the possible reason and indication?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NMBBVILpFN", "reviewer": "Reviewer_hocx", "summary": "This paper introduces MAAS (Multi-agent Architecture Search), an innovative framework for automating the design of multi-agent systems powered by Large Language Models (LLMs). MaAS addresses the limitations of existing methods that seek to identify a single, static, and complex multi-agent architecture, which often fails to dynamically allocate resources based on the difficulty and domain of each query. Instead, MaAS proposes the concept of an \"agentic supernet\", a probabilistic and continuous distribution of multi-agent architectures that can be sampled to tailor the system to specific queries.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere respect for your insightful review! In response to your comments, we have carefully prepared a point-by-point reply:\n\n---\n>**`Essential References Not Discussed`**\n\nThank you for the valuable supplement! We have added this important citation in our revised manuscript.\n\n---\n>**`Weaknessm 1: Additional benchmarks`** One thing I can suggest is to test on additional benchmarks such as ALFWorld, SciWorld, and ToolBench.\n\nThank you immensely for the instructive advice! Our performance on ALFWorld is summarized in the table below. Since ALFWorld involves multiple trials, MaAS samples an architecture for each episode, meaning the same agentic workflow is used across multiple trials. Other trainable baselines, such as GPTSwarm and AFlow, are implemented similarly.\n\n| Method | Perf. (Max_trial=20) | Perf. (Max_trial=30) | Avg. Cost (10^-3 $) |\n|--------|---------------------|---------------------|-------------------|\n| Vanilla GPT-4o-mini | 48.71 | 50.12 | 4.68 |\n| CoT | 49.92 | 51.82 | 4.95 |\n| LLM-Debate | 54.68 | 56.90 | 17.30 |\n| GPTSwarm | 53.19 | 57.40 | 12.14 |\n| AgentSquare | 66.42 | 69.75 | 7.66 |\n| AFlow | 59.16 | 60.81 | 13.02 |\n| **MaAS (Ours)** | **68.14** | **72.66** | 9.15 |\n\nAs observed, MaAS achieves a performance improvement of up to 22.54% on the embodied task ALFWorld while maintaining a relatively low average cost, demonstrating its cost-effectiveness.", "claims_and_evidence": "The following primary claims in this paper are well-supported:\n\n1. ​**Pursuing an agentic supernet rather than a one-size-fits-all MAS.**​ The authors advocate for optimizing their proposed agentic supernet instead of the previously attempted giant, high-latency MAS. Their experiments support this claim: MAAS outperforms the SOTA baseline AFlow with only 15% of the training cost and 25% of the inference cost. The task adaptiveness emphasized by MAAS aligns well with intuition.\n2. ​**Comprehensive automation is essential for MAS.**​ The authors provide a clear and insightful overview in Section 2 of the evolution of MAS from fully manual setups to partial automation and, finally, to full automation. The mapping of the technical trajectory from neural architecture search to automated MAS is particularly interesting. The MAAS framework achieves comprehensive automation, from prompts to communication topologies.", "methods_and_evaluation_criteria": "The concept of the agentic supernet introduced in MAAS is novel. While it appears to be inspired by works like DARTS in NAS, it follows a completely different technical approach. The probabilistic sampling and the introduction of the early-exit operator effectively fulfill the authors’ vision of a task-dynamic MAS.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "MAAS is evaluated across six benchmarks, covering domains such as mathematics, coding, and tool usage. The authors also emphasize resource consumption metrics, including token count, API cost, and wall-clock time. MAAS is comprehensively demonstrated, and I find no significant flaws in the evaluation.", "relation_to_broader_scientific_literature": "Automated MAS is an emerging and highly relevant research direction, with works like AgentVerse, GPTSwarm, and later ADAS and AFlow falling under the category of one-size-fits-all MAS. MAAS represents a new paradigm in this field and is closely connected to broader areas such as collaborative AI and autonomous AI.", "essential_references_not_discussed": "I recommend that the authors include Flow[1] (ICLR 2025), which also focuses on automating agentic workflows.\n\n[1] Flow: Modularized Agentic Workflow Automation, ICLR 2025", "other_strengths_and_weaknesses": "Strength:\n\n1. The proposed paradigm shift is innovative and significant for this field. Regardless of whether the agentic supernet becomes the mainstream form of MAS in the future, I believe this is an important contribution.\n2. The evaluation is thorough, covering performance, token cost, API cost, and time consumption.\n\nWeakness:\n\nI find this work well-organized and convincing, with no apparent shortcomings. One thing I can suggest is to test on additional benchmarks such as ALFWorld, SciWorld, and ToolBench.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FspJxWxqJS", "reviewer": "Reviewer_BAH6", "summary": "This paper introduces the concept of “agentic supernet”, which transforms the automatic LLM-based multi-agents design paradigm from a static, one-size-fit-all approach to a dynamitic and adaptive framework. Their MaAS framework samples components from the supernet to assemble appropriate multi-agent systems according to the difficulty and domain of the given query. Sufficient experimental results demonstrate their method effectively reduces API costs while improving performance across a wide range of tasks and LLM backbones.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your careful comments and thorough understanding of our paper! Here we give point-by-point responses to your comments and describe the revisions we made to address them.\n\n---\n>**`Weakness 1`: Clarification on parameter $\\phi$** The paper lacks clarity regarding the parameter $\\phi$ in Eq(6), where \"$Q\\_\\phi$ is parameterized by $\\phi$.\"\n\nThank you for pointing out! The $\\phi$ represents the parameters of the controller network $\\mathbb{Q}\\_\\phi$, which is essentially composed of the sampling functions for each layer, $\\pi\\_l: q \\rightarrow \\mathcal{V}\\_\\ell$ (where the definition of $\\pi_\\ell$ is given in Eq. (9)). They are MoE-style networks that select the activated operators for the $\\ell$-th layer based on the query. Therefore, $\\phi$ can also be expressed as $\\phi = \\\\{\\pi\\_1, \\cdots, \\pi\\_\\ell\\\\}$.\n\nWe hope this clarifies the nature of the controller network.\n\n---\n>**`Weakness 2: The selection of threshold value`** What’s the reason the threshold value thres in Eq(9) is set as 0.3?\n\nWe supplement the parameter sensitivity analysis of threshold value in Eq. (9) as follows:\n\n|Dataset|Metric|0.1|0.2|0.3|0.4|0.5|0.6|0.7|\n|-|-|-|-|-|-|-|-|-|\n|HumanEval|Perf.|90.07|91.60|92.85|93.02|92.36|92.36|90.83|\n||Avg. Cost|0.69|0.92|1.01|1.121|1.113|2.590|4.300|\n|GSM8K|Perf.|90.99|91.46|92.30|92.22|92.70|92.09|91.75|\n||Avg. Cost|0.38|0.44|0.48|0.55|0.70|0.77|0.92|\n\nWe observe that while increasing the threshold value leads to some performance gains, the improvement plateaus beyond 0.3. Additionally, a higher threshold increases inference costs due to the activation of more operators per layer. Therefore, we ultimately set $thres = 0.3$.\n\n---\n>**`Weakness 3: Performance discrepancy`** There appears to be a discrepancy in the reported performance improvements.\n\nThank you for your detailed review! After thorough inspection, we found that 11.82% was a typo, as the main text and experimental results were not properly synchronized. It should be corrected to 16.89\\%, which is derived from the performance difference between MaAS and MacNet on MBPP dataset.\n\n---\n>**`Weakness 4`** the improvements highlighted in red are derived from comparisons with the Vanilla baseline in Table 1, while the actual improvements over “handcrafted or automated multi-agent systems” baselines such as AFlow appear modest.\n\nThank you for your insightful suggestion! We chose to present the improvements over vanilla LLMs in Table 1 following prior practices in GPTSwarm (ICML 2024) and AgentPrune (ICLR 2025), which adopt the same approach.  \n\nTo address your concerns, we have:  \n\n1. Provided a version of Table 1 with standard deviations, replacing the subscript values with the standard deviations from three runs, as shown in **Table1-stdev** (https://anonymous.4open.science/r/maas-rbt/table1-stdev.png).  \n2. Explained the key advantages of MaAS over the SOTA baseline AFlow:  \n   - Although MaAS achieves moderate improvements over AFlow on certain benchmarks, it shows clear advantages on a broader set of benchmarks (e.g., +1.14% on GSM8K, +2.58% on MultiArith).  \n   - MaAS achieves these gains with significantly lower computational costs—its training cost is only 15% of AFlow’s, and inference cost is just 25%.  \n\nWe sincerely hope this demonstrates MaAS’s superiority over AFlow in both cost efficiency and performance.\n\n---\n>**`Minor Typos`**\n\nThank you for the meticulous review! We have repaired the mentioned typos in our revised manuscript.", "claims_and_evidence": "The paper is well-organized and well-written, enhancing the clarity and impact of their proposed method and findings.", "methods_and_evaluation_criteria": "This work presents a meaningful advancement by dynamically designing multi-agent system based on different queries through optimization of probabilistic, continuous distribution of agentic architectures, and the MaAS framework's pipeline is theoretically well-defined.\n\nThe paper lacks clarity regarding the parameter ϕ in Eq(6), where \"Q_ϕ is parameterized by ϕ.\" The initialization and nature of this controller parameter remain unexplained.\n\nWhat’s the reason the threshold value thres in Eq(9) is set as 0.3? The selection of this parameters would benefit from theoretical justification or empirical analysis, similar to the Sensitivity Analysis presented in Section 4.5.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "The authors conducted extensive experiments across multiple benchmarks to evaluate their proposed MaAS, the amount of their experiments is fair and convincing.\n\nThere appears to be a discrepancy in the reported performance improvements. The authors mentioned MaAS “surpassing existing handcrafted or automated multi-agent systems by 0.54%∼11.82%” in terms of performance, but these statistics do not in sync with the results shown in Table 1, I wonder where the value 11.82% is derived from. Besides, the improvements highlighted in red are derived from comparisons with the Vanilla baseline in Table 1, while the actual improvements over “handcrafted or automated multi-agent systems” baselines such as AFlow appear modest.", "relation_to_broader_scientific_literature": "improve the performance on benchmark dataset", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Please refer to the weaknesses outlined above. Besides, there are some minor typos in this paper:\n1. Caption of Table 1: get-4o-mini -> gpt-4o-mini\n2. L81: MATh benchmark -> MATH benchmark", "other_comments_or_suggestions": "see above", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel paradigm—Agentic Supernet—for enabling adaptive inference in multi-agent systems. The reviewers consistently praised the originality of the framework, the comprehensive and rigorous experimental evaluation, and the practical implications for automated agent design. While some concerns were raised about clarity in architectural details and the disentangling of performance gains, the consensus is that the paper marks a significant contribution to the field. I strongly recommend acceptance.
Decision: Accept (oral)