G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks
Authors: Guibin Zhang
Yanwei Yue
Xiangguo Sun
Guancheng Wan
Miao Yu
Junfeng Fang
Kun Wang
Tianlong Chen
Dawei Cheng
Venue: ICML 2025 spotlightposter (2025)
Keywords: Multi-agent communication
Graph machine learning
LLM-based agent
Primary area: deep_learning->graph_neural_networks

Abstract: Recent advancements in large language model (LLM)-based agents have demonstrated that collective intelligence can significantly surpass the capabilities of individual agents, primarily due to well-crafted inter-agent communication topologies. Despite the diverse and high-performing designs available, practitioners often face confusion when selecting the most effective pipeline for their specific task: \textit{Which topology is the best choice for my task, avoiding unnecessary communication token overhead while ensuring high-quality solution?} In response to this dilemma, we introduce G-Designer, an adaptive, efficient, and robust solution for multi-agent deployment, which dynamically designs task-aware, customized communication topologies. Specifically, G-Designer models the multi-agent system as a multi-agent network, leveraging a variational graph auto-encoder to encode both the nodes (agents) and a task-specific virtual node, and decodes a task-adaptive and high-performing communication topology. Extensive experiments on six benchmarks showcase that G-Designer is: \textbf{(1) high-performing}, achieving superior results on MMLU with accuracy at $84.50\\%$ and on HumanEval with pass@1 at $89.90\\%$; \textbf{(2) task-adaptive}, architecting communication protocols tailored to task difficulty, reducing token consumption by up to $95.33\\%$ on HumanEval; and \textbf{(3) adversarially robust}, defending against agent adversarial attacks with merely $0.3\\%$ accuracy drop.

G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks
Guibin Zhang 12 Yanwei Yue"
Xiangguo Sun * Guancheng Wan Miao Yu Junfeng Fang? Kun Wang Tianlong Chen Dawei Cheng'
Abstract
CHAIN
DYNAMIC
Recent advancements in large language model (LLM)-based agents have demonstrated that col- lective intelligence can significantly surpass the capabilities of individual agents, primarily due to well-crafted inter-agent communication topolo- gies. Despite the diverse and tegh-eerfommig designs available, practitioners often face confu- sion when selecting the most effective pipeline for their specific task: Which topology is the best choice for my task, avoiding unnecessary com- munication token overhead while ensuring high- quality solution? In response to this dilemma, we introduce G-Designer, cient, and robust solution for multi-agent de- ployment, which dynamically designs task-aware, customized communication topologies. Specif- ically, G-Designer models the multi-agent sys- tem as a multi-agent network, leveraging vari- ational graph auto-encoder encode both the nodes (agents) and task-specific virtual node, and decodes a task-adaptive and high-performing communication topology. Extensive experiments on six benchmarks showease that G-Designer is: (1) high-performing, achieving superior re- sults on MMLU with accuracy at 84.50% and on HumanEval with pass@1 89.90%: (2) task- adaptive, architecting communication protocols tailored task difficulty, reducing token con- sumption by up to 95.33% on HumanEval; and (3) adversarially robust, defending against agent ad- versarial attacks with merely 0.3% accuracy drop. The code is available at https: /iihbbb. com/ yaaneaawee yue/ GDes inee... gner.
DyLAN
GRAPH
Complete Gogh
Tayered Graph
Random Gregh
GirShomm
Figure I. Existing practices for LLM-based multi-agent communi- cation topology design.
adaptive, effi-
1. Introduction
An LLM- L.M--ased agent, which integrates the language gen- eration capabilities of LLMs with decision-making and action-execution functionalities (Richards & et al., 2023; Nakajima, 2023; Reworkd, 2023), has exhibited impressive performance across a wide range of tasks, from reason- ing (Yao al., 2023b) and code generation (Shinn et al., 2023) even more complex applications like video gam- ing (Wang al., 2023) and autonomous driving (Jin et al., 2023), Even more exciting, researchers have discovered that combining multiple LLM-based agents-whether implicitly or explicitly-into a team can outperform individual agents when tackling complex tasks (Du et al., 2023; Liang et al., 2023; Wang et al., 2023b; Jiang al., 2023; Shinn et al., 2023; Zheng al., 2023; Wu al., 2023), demonstrating form of collaborative intelligence reminiscent of human teamwork in multi-agent systems (Zhang et al., 2023b). This emergence of human-esque collective intelligence is fundamentally driven by the design of their topology, iee, how multi aalti-agents are connected, and how they transmit, exchange, and assimilate information reciprocally.
'Equal contribution Tongii University NNUS *CUHKK "UCLA 'USTC 'NTU "UNC- Hill. Correspondence to: Kun Wang <wang.kuu kun @www..dd..gs... <deheng@tongji.co @tongg..edd...>..
In practice, prior research has extensively explored how multiple instances of LLMs, referred to as agents (Wang et al., 2024; Xi et al., 2023; Gao et al., 2023; Cheng et al., 2024), should be structured and organized to converse, collaborate, debate, or even compete. Various topologi- cal designs have been investigated, such as chain (Wei et al., 2022; Hong et al., 2023), tree (Yao et al., 2023a; Wu
Dawei Cheng
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

G-Designer
To design communication topology that ideally adheres to the MACP principles, we propose effective, adaptive, and robust LLM-powered multi-agent communication graph designer, termed G-Designer. Technically, G-Designer first architects a multi-agent graph, where each agent, along with its specific properties (c.g., profile (Li et al., 2023a), external API tools (Zhuang et al., 2023), or knowledge base (Chen et al., 2024a)), is represented as node, and com- munication between agents forms the edges. G-Designer employs a variational graph auto-encoder to encode the nodes (agents) along with task-specific information, and to decode the resulting collaboration network between agents. This input- paradigm allows G-Designer to design aask-ddap,ive high- -erffrrrronnnng communication topology, which is, at the same time, assured of efficiency and robustness with sparsity regularization. Unlike previous LLM-based multi-agent topology designs, which rely on static structure for all queries/tasks, G-Designer adap- tively crafts customized topologies for different domains and tasks, serving as a fully autonomous and flexible assis- tant for multi-agent system establishment and deployments.
Grrtwarn
Tokm Comanakk
Eosy Scyhschool Biolegy?
Toean CaenCConee Cannure
Chain
Hard (College Hath))
Achili Gough
Accunay (90) on MILLO (Hialachonn Baolog)
Aucunsy (Nil on MAMLU (Codegs Matk)
Figure 2. The token consumption and accuracy of different multi- agent protocols on two subsets of MMLU dataset, "Highschool Bi- ology" and "College Math", tested with four pppt-4-base agents. et al., 2023), star (Wu et al., 2023), complete graphs (Qian et al.. 2024), random graphs (Qian et al., 2024), optimiz- able graphs (Zhuge et al., 2024; Zhang et al., 2024), and LLM-based networks (Hao et al., 2023; Liu et al., 2023). These elaborately designed communication topologies have demonstrated remarkable performance with minimal human supervision, bridging the gap between individual and collec- tive intelligence. Faced with numerous structures available, an inquisitive practitioner might ask: how should I select or design a topology that best suits my task at hand?
The question posed above is non-trivial and, at times, per- plexing. A piece of experimental evidence is presented in Figure 2, where we evaluated the performance of different multi-agent structures on the MMLU dataset (Hendrycks et al., 2021), collection of multiple-choice questions across various subjects. The results reveal that even within the same dataset, the suitability of different communication topolo- gies varies. O Simpler Case: in the simpler "High School Biology" subset, the chain structure performs comparably to the complex GPTSwarm, while consuming significantly fewer tokens (0.5k versus 7.8k). In this case, the chain structure is clearly more economical choice. @ Harder Case: However, for the more challenging "College Mathe- matics" subset, GPTSwarm outperforms the chain structure by 8.75%, primarily attributed to its intricate topology and prompt optimization. In summary, practitioners often find challenging to effortlessly identify the most efficient and complexity-ddaptio multi-agent topology for a given task.
Our contribution can be summarized as follows:
Protocol Proposal. We propose the first communica- tion protocol tailored for LLM-ppoered multi-agent sys- tems, MACP, which comprehensively regulates multi- agent topology design across three dimensions: perfor- mance, adaptability, and robustness, and incisively high- lights the shortcomings of existing designs.
Practical Solution. We present G G-Desi an effective, adaptive, and robust designer of L-M-eed multi- agent communication graphs. By leveraging a varia- tional graph auto-encoder to construct and process the multi-agent network, G-Designer decodes task-adaptive and high-performing agent communication, which is also equipped with strong robustness against agent-rooted ad- versarial attacks via dynamic topology adjustment.
Experimental Validation. Extensive experiments across six benchmarks show that G-Designer is: (1) high- performing, surpassing state-of-the-arr topologies by 0.20% 4.10% on MMLU and HumanEval; (2) task- adaptive, dynamically adjusting topology complexity with task awareness, outperforming state-of-the-a the meth- ods on MMLU with a cost of merely 1.5e + 5 compared to their 2.6e + 6, reducing token consumption by up to 92. 94%: and (3) adversarially robust, defending against agent adversarial attacks with merely 0.3% accuracy drop.
In light of this dilemma, we propose the L.LM-based Multi- agent Communication Protocol (MACP), establishing standardized guidance for LLM-MA topology design:
Multi-agent Communication Protocol (MACP): Given a task/query q. an optimal LLM-MA communication topol- ogy for q should satisfy the following protocol logics: (I) Effectiveness: The communication structure must effec- tively produce the qualified solution for q; (2) Complexity- adaptiveness: The topology should dynamically adjust to the complexity of the task, minimizing communica- tion overhead; (3) Adversarial robustness: The topology should maintain reliable under adversarial attacks.
2. Related Works
LLM-agent Collaboration Recent research has explored various multi-agent communication topologies, including: (1) Non-interactive, where agents operate independently without inter-agent communication, as employed in systems like LATM (Zhang et al., 2023a) and L.L.M-Debate (Du
The formal definition of MACP is provided in Section 3.3.
2

---

G-Designer
al., 2023); (2) Chain, where agents are arranged in a sequential structure, each receiving the output from its pre- decessor and passing information to its successor, utilized by ChatDev (Qian et al., 2023). MetaGPT (Hong et al., 2023), and 1.2MAC (Holt et al., 2024); (3) Star, where a central administrative agent (often referred to as a com- mander, teacher, etc.) directs subordinate agents, seen in AutoGen (Wu et al., 2023), SecurityBot (Yan et al., 2024), and MiniGrid (Zhou et al., 2023); (4) Tree, where root agent hierarchically manages multiple child agents, as in SoA (Ishibashi & Nishimura, 2024); and (5) Graph, encom- passing complete graphs (Qian et al., 2024; Zhuge et al., 2024) and random graphs (Qian et al., 2024), among others.
tools or plugins available U, such as web searchers (Ma et al., 2023), code compilers (Richards & al., 2023; Wu et al., 2023; Hong et al., 2023; Bouzenia et al., 2024; Ishibashi & Nishimura, 2024), or file readers (Zhuge et al., 2024; Richards & et al., 2023). Each LLM--bsed agent iü receives prompt P and generates response R₁:
R = (P) = U; (Psyss Pust).
where Pays {Role,, State:) represents the system prompt encompassing its role and state, and Pusr denotes the user prompt, which possibly includes the given tasks, responsev/nssrrucionss from other agents and externally re- trieved knowledge.
Multi-agents as Graphs Graphs, as a fundamental data structure for organizing and representing relationships be- tween entities (Zhang & Chartrand, 2006), are widely adopted in the pre-LLM era as a powerful tool to facilitate effective communication in multi-agent reinforcement learn- ing (MARL) (Pesce& & Montana, 2023; Hu et al., 2024; Liu et al., 2022). With the rise of L.LMs and the proliferation of LLM-based agents (Chen et al., 2023a; Cohen et al., 2023; Hua et al., 2023), researchers have similarly recognized that interactions among multiple agents can naturally be mod- eled from a -raph-based perspective (Chen et al., 2023b; Zhuge al., 2024; Qian et al., 2024; Liu et al., 2023). Early attempts are implicit, like ChatEval (Chan et al., 2023), Au- toGen (Wu et al., 2023), and DSPy (Khattab et al., 2023).
The connectivity of G can also be characterized by (non- symmetrie) adjacency matrix A E {0, 1,NN,, where A J = if eij ( Uy) E, otherwise 0. Each edge Gig E ε represents the flow of information from is to Uj.
3.2. Communication Pipeline
Given a query/problem Q, the multi-agent system engages in K rounds of interactive utterances, which collaboratively drive the agents toward producing the final solution (k) based on their cumulative dialogue exchanges. At the be- ginning of the i-th dialogue round, mapping function & is applied to determine the execution index for each agent:
Φ: o, o [Vor Uog: Nawl
(3)
More recent practices including ChatLLM (Hao et al., 2023), DyLAN (Liu et al., 2023), GPT'Swamm (Zhuge et al., 2024), and MacNet (Qian et al., 2024), have explicitly represented the organization of multiple agents as graph. However, all these attempts, whether predefined or iteratively optimized, remain input-independent. Consequently, they fail to be task-aware and adaptively design topologies that suit the complexity of the specific task.
s. Vi > j. Vo. $ NMm(vo,),
where a is the execution sequence of agents, (m)()r() denotes the in-neighborhood of to(j), and the constraint ensures that an agent va(i) can only execute after any agent (z) from which it receives information. Once the execu- tion order is determined, each agent proceeds to perform input-output operations sequentially:
3. Formalization
R = (;(()t) sys ust
ust = (2, U2 (v, R( (4)
This section establishes the notation, formalizes key con- cepts from a topology perspective, and formally defines our proposed multi-agent communication protocol.
where R,C represents the output of t; which could be a rationale, an answer, or a partial solution, depending on the specific context. The output R, is generated based on the system prompt Piy and the context prompt, consisting of the query Q and messages from other agents. At the end of each dialogue, aggregation function is adopted to generate the answer/solution a!!):
3.1. Topology Structure
We model the multi-agent system as a directed graph g = (V,E), where
{ei...., EN represents the set of nodes (with N \1) and & denotes the set of edges. Each node us E V corresponds to an agent, formalized as:
(tt) Aggregate (R.() R(
R
(5)
0: = (Base,, Role, State,, Plginn;) (I)
The implementation of the Aggregate function is flexible, with possible options including majority voting (Chen et al., 2024b; Zhuge et al., 2024; Li et al., 2024), aggregating all agents' responses and delegating one agent to provide the final answer (Wu et al., 2023; Jiang et al., 2023; Liu et al., 2023; Zhang et al., 2024), or simply using the output of
where each agent is composed of four key elements:
(1) Base, the language model instance powering Uy; (2) Role; the agent's pre-assigned role or function; (3)
State,, representing the agent's 's accumulated knowledge and interaction history; and (4) P1 Pluging n,. a set of external
3

---

G-Designer
Materials
Construct
Design
Optimize
Query/Problem 2
Profile Assignment
D=C]
Graph Auto-encoder
-
Optimization Config
Encoder 9: 9: ..> Hayer
NNumber dialogues:
An clectric motor has a label on it that reads: Input: 120V AC, 1.0 Amps, Efficiency- 75%. At what constant speed can the motor lift up a kg mass?
Mark Analyst
Programmer
Plyetes
Chief Scientivi
p. Hayor
9co
Exccution order. (May
Communication Graph
Workf low
Gere (V,E") (X sgnnt:
Inpar query
graph
Encode Decode
Hemeive dialegge
Node Encoder
Sentance Bent MinilM:
Andhar Regelarize
[|A, 2W2 18
Sparse Regulorate
www 3 [|WI]
ATTI caceaaa an cornet
The upa Pncippppponn d. edladiedd
Agent (node) set V
creatio Pos
c: [Base, Role,, State, Plugin,)
0.33
Deraaton rund to K
Task-specific Btask ERD
Agent representation Xagent C RR
Iteration
Instontiated by gpt-4/Lloma 3.I....
Virtual node
Mullt-agent Network
Role/Profile pool
s H
Mathematicion
Physiost
gelterrtee by gpi-4
Decode:
P.(S H)
p. (Sylete By Plaae)
Tescher Nath Anclyat
hark
Tool/Plugin set
Ouput
Texi-lo-specch translator
Python complier t File reader
Vtask
M
jo(H
Encode:
Hugee R
Selifise g1
ofpeprcciunii mik spoed till olgan..
Programmer
Chief
Image question answerer
Mathematical calculator
Multi-ogent Network: G (V, () (X, A)
v(H 9 II" (hu (9) 2(b, ((b, ding(e)))
ang max Boane (u(Gemm(1)) 6.0 Cussay fanater Eegorse
Anchor structure M2 either user user-defined generated by LLMc as on ancher
(other extemal kxale)
Figure 3. The designing workflow of proposed G-Designer.
the last agent Ray (Qian et al., 2024). Through K rounds of utterances, either predefined (Qian al., 2024) or deter- mined by an early-stopping mechanism (Liu et al., 2023), the overall system G produces the final answer a (K) for Q.
cally, the process begins with few "raw materials"; the input query Q. the agent set V, the profile pool, and the toolset. In the Construct stage, G-Designer leverages node encoder to construct a multi-agent network along with a task-specific virtual node. In the Design stage, a graph auto-encoder is employed to decode the communication graph topology Geom. which leveraged for multi-round inter-agent collaboration in the Optimize stage.
3.3. MACP Protocol
We give the formal definition MACP Protocol as follows:
Definition 3.1 (Multi-agent Communication Protocol). Given an LLM-based multi-agent system (V,E), we
4.1. Multi-agent Network Construction
establish the following objective optimization principle:
Given an input query Q and a set of LLM-agents V. G-Designer aims to design a task-adaptive and effective communication topology Goom. We begin by assigning each agent unique role and profile, as previous research (Wang et al., 2023b) has shown that assigning distinct personas roles to LLM-based agents can enhance cognitive synergy. Based on these roles, different external tools are allocated to the agents (e.g.. Mathematica for a math analyst, Python compiler for programmer). Thus, we successfully initial- ize each agent v as {Base,, Role, State,, P Plugin;). as defined in Equation (1).
min (G(()) +3, ||||| B2 |G(() -(0)]] (6) GEGL
where G represents the feasible parameter space of G, u(-) is the utility evaluator, ||||| measures the computational and communication overhead of the entire graph, and Q and denote the query description and the multi-agent system after adversarial perturbation, respectively. The first term in Equation (6) corresponds high performance, aiming maximize the utility of the system's 's output; the second term addresses task lask-adaptiveness. seeking to minimize system complexity to reduce power consumption and economic cost; and the third term focuses on robustness, constraining the deviation of system output under adversarial attacks.
We proceed construct structured multi-agent network input to G-Designer, represented as G = (Xagent A), where X Xagent E N is the node (agent) feature matrix and A E NN represents the connectivity matrix. For the feature matrix Xagent, we employ a node encoder to transform each agent's 'ss unique profile into a fixed-length
4. G-Designer
Figure 3 illustrates how G-Designer adaptively designs communication topologies for any given query. Specifi-

---

G-Designer
embedding representation:
where fg is the encoder-decoder architecture with parame- ters (() is the encoder module, p(-) is the decoder mod- ule. The encoder utilizes posterior probabilities to encode the node embeddings into low- cow-dimnssional latent vector representations H Hagents which can be formulated as:
7(-)
x; f NodeEncoder (T(Base)), Role; ((Plugin))) (7)
where T-)) extracts the textual description of the agent's LLM backbone and its assigned plugins, and Nodeencoder can be realized using small and lightweight text embed- ding models (Reimers, 2019). After encoding the indi- vidual agents, we aim to ensure that the multi-agent net- work incorporates information related to the query 2, this query-dependent approach enables G-Designer to be task-aware and adaptive. To this end, we introduce an ad- ditional task-specific virtual global node Ttask. which is bidirectionally connected to all agent nodes, enabling a global "storage sink" and facilitating smoother information flow among agents (Shirzad al., 2023; Tan et al., 2023; Rosenbluth et al., 2024). This task node is encoded by the NodeEncoder follows: Xtask NodeEncoder Q).
q( H (Hagent X, A Annchr))
II
[9( X, Aanchor); (10)
<hb X, Annchor) = Nhi, U diag(o?));
where μ GNN, (X, Aanchori w₂) () is the matrix of mean vectors μ4; similarly log(o) = GNN (X, Aanchori O²). The choice of GNN backbone can be customized as needed; here we utilize simple two-layer GCN (Kipf & Welling, 2017). h; M4 and o, denote the i-th column of H. μ., and respectively. The encoder q(-) is parameterized by O. = {or, θz}. Following the encoding phase, the decoder employs the latent representations to generate a comprehensive blueprint for multi-agent communication.
After obtaining the agent node features Xagent [x1, K2,,,,,..,xx XN] and the task-specific embedding Xtasks we provide a simple anchor topology Aanchor E {0. NNN, which serves as a starting point for G-Designer's topology design process. For instance, given a code generation task with three agents: man- ager/programmerrcode reviewer, the anchor topology could be configured as a chain structure, ie., "manager > pro- grammer + reviewer", reflecting the typical workflow of code completion. The anchor topology, being either user-defined or automatically generated by LLMs, is often simple and sub-optimal'. However, it provides founda- tional reference and prior knowledge for G-Designer's subsequent optimization process. We incorporate the task- specific vertex ttask and its corresponding edges and obtain Aunehor E {0, 1)(N++)x(N++) Consequently, we estab- lish a spec-fpec multi-agent network G:
More specifically, the decoder g(-) = qe o qss first constructs parameterized, sketched graph S, which is then refined into the final multi-agent communication topology:
p(Gcom Hagent) = J PG [S)...S H Hagens) dS.
(11)
At the first step, p-(+) constructs the sketched adjacency matrix S from the latent representation Hagent:
N (S h. hy. heski Oa), (12)
(S H Hagent)
II
whose detailed derivation is as follows:
Ps (S. 1 h, hy, heask) ) = g(h,, h2, heast), (iggoid((og(ee) log(1 E) + ww))/T),
g Xagent
Aanchor) (),
(13)
X xlack
(8)
where w FFFaa h,, htask]) with FFN, parame- terized by O2. Uniform 1), and denotes the temperature coefficient. When approaches zero, Equa- tion (13) essentially return the Bernouli sampling result for Sij. The resulting matrix S E [0, 1|NxNN represents a densely-commected, non-negative graph distribution, indi- cating an overly complex and -intersve pair-wis, communication structure, which is not yet suitable for guid- ing multi-agent collaboration. To align with G-Designer's objectives of task adaptiveness and minimizing costs, ap- ply refinement decoder Pc(-) to refine the sketched S into a compact, sparse, and highly informative communication graph, instantiated by a regularization objective:
= UVU (. E U {(1,, lla E V))).
where speene]] can also be jointly denoted as X.
4.2. Designing Communication Topology
Building upon the task-specific multi-agent network G. G-Designer seeks to establish more fine-grained and precise communication topology Goom Drawing inspi- ration from the variational graph auto-encoder (VGAE) framework (Kipf & Welling, 2016; Zhao & Zhang, 2024), G-Designer employs a VGAE-based encoder-decoder fo to generate the multi-agent interaction topology:
Goom f.( p(ecom
H) X, Aunchor), (9)
Pe arg max 1/21/S ZWZ 112 + < [||+ SES
'e discuss the substantial performance improvement of G-Designer over the anchor topology in Section 5.4.
(14)
1 1/2 A anchor
ZWZ 1|? s. t. 5 = ZwzT,
5

---

G-Designer
where Ze RR-x is the top-r columns of left singular ma- trix S. ( is a coefficient hyperparameter, W E R''x is optimizable weight matrix. -|- || denotes the Frobenius norm and ||]|= = Ee>, A; where A is the i-th singular value of W. $ TR is the desired sparse topology, which is decomposed as ZWZ In Equation (14), the first and second terms are jointly denoted as anchor regulariza- tion, which encourage the learned S to maintain similarity with both the original S and the anchor topology. The third term, denoted as sparsity regularization, though appearing to minimize the nuclear norm of W, essentially sparsifies S, since ||S||. |. fore, Equation (14) achieves two key goals: (1) producing sparse, refined communication topology, and (2) constrain- ing the design to remain grounded in practical intuition. The resulting communication can be represented as follows:
pability strategically design the agent network, achieving truly task-customized multi-agent topology design.
Optimization configuration The overall training objec- tive of our method is formulated as Le-Deeiguer = Lusslity + fanchor + Eeparse where Lutility represents the optimiza- tion target from Equation (16), Lanchor corresponds to the first and third terms in Equation (14), and Lsparse is the second term. Given a benchmark {Q1) consisting of B queries, G-Designer begins by optimizing with a small subset of B' queries and fixes the learned parameters for testing on the remaining (B B¹) queries. The whole algo- rithm workflow of G-Designer is depicted in Algorithm I.
[ holds due to Z Z = Iyxr- There-
5. Experiments
5.1. Experimental Setup
Datasets and Metries We evaluate G-Designer on three categories of datasets:
Goom = (V, Erom); Ecom Eoom = ((..)) | S., * O (i.J) E E)). (15) At this stage, we have successfully distilled lightweight and informative collaboration network Foom from the sketched task-specific network G. which is now ready to guide inter-agent message passing in the following process.
General Reasoning:
MMLU (Hendrycks et al.. 2021); Mathematical Rea- soning: GSM8K (Cobbe et al., 2021). MultiArith (Roy & Roth, 2016), SVAMP (Patel et al., 2021), and AQuA (Ling et al., 2017); Code: HumanEval (Chen et al., 2021). We include the dataset statisties in Table 4.
4.3. Optimizing G-Designer
Baselines For single-agent approaches, we select COT (Wei et al,, 2022). ComplexCoT (Fu et al., 2022), Self- Consistency (Wang et al., 2023a), and PHP (Zheng et al., 2023). For multi-agent topologies, we select Chain, Star, and Tree (formally defined in (Qian et. 2024)), Complete Graph and Random Graph AutoGen (Wu et al., 2023), MetaGPT (Hong et al., 2023), LLM-Debate (Du et al., 2023), LLM-Blender (Jiang et al., 2023), DyLAN (Liu et al., 2023), and GPTSwarm (Zhuge et al., 2024).
Upon obtaining Goom the multi-agent utterances and di- alogues can proceed as usual using Geom. as detailed in Section 3.2. After K rounds of interaction, the agents con- verge a final solution (K)) Geom (Q). We then give the following optimization objective:
arg min Eo. O.
(16)
where O and O are the parameters of the encoder q(-) and decoder p()), respectively, f2 is the parameter space and B(-) denotes the mathematical expectation. Equation (16) aims to maximize the utility of the generated solution, but it is inherently intractable and non diffffffreetiible, as u(-) often depends on external API calls (Li et al., 2023b; Hendryeks et al., 2021). To address this, following standard approaches in multi-agent structure design (Zhuge et al., 2024; Zhang et al., 2024), apply policy gradient (Williams, 1992) to approximate and optimize Equation (16):
Implementation Details We access the GPT via the Ope- nAI API, and mainly test on (gpt-4) and set temperature to 0 for the single execution and sin- gle agent baselines and 1 for multi-agent methods. We set a summarizer agent to aggregate the dialogue history and produce the final solution akK), with K = 3 across all experiments. The NodcEncoder(() is implemented using all-MiniLM-16-VV (Wang et al., 2020), with the em- bedding dimension set to D = 384. The anchor topology Agnchor is predefined as a simple chain structure. The sam- pling times M are set as 10, and 7 = 1e - 2 and Ç = 1e are set for all experiments. We provide explicit agent pro- filing for multi-agent methods, following the classical con- figurations in LLM-MA systems (Liu et al., 2023; Zhuge et al., 2024; Yin et al., 2023), and use gpt-4 to generate agent profile pools. For all benchmarks, we merely use B' E {40, 80} queries for optimization.
(gpt-3. 5).. We
1 M M
~ Velana (Q1)]=
Σ k=1
uaa
(K) (P((G))) (17)
where O = (0,, Oz). { JM are indepently samples from Geom, and (ah ( I are the corresponding out- put. P(Gx) calculates the probability of GR being sam- pled, which can be expressed as P(Gx) = II |= Siz. Through iterative optimization guided by Equations (14) and (16) over a limited set of queries as the "training set", C-Designer efficiently develops aask-awsreness and the ca-
5.2. Main Results
In this section, we conduct extensive experiments across six benchmarks to verify that G-Designes gneer is:
6

---

G-Designer
Table 1. Performance comparison with three types of baselines, including single-agent execution, spatial communication, and temporal communication. The best results are in bold, and the runner-ups ups are underlined. All methods, except for the single-agent category. utiliize five --t-d- agents. "Mul.". "Ada.", and "Rob." indicate whether the method supports a multi-agent setting, whether it
task-adaptive, and whether it is adversarially robust, respectively. X, A and / signifies no/partial/full support in these aspects.
Method
Mul.
Ada.
Rob.
MMLU
GSM8K
MultiArith
SVAMP
AQuA
HumanEval
Avg.
Vanilla
x
82.14
85.40
93.15
87.18
70.34
71.68
81.65
CoT
X
x
x
82.65
87.17
94.79
88.32-1.14
73.97
75.52
83.73
ComplexCoT
x
x
x
83.78
p7.62
95.86
90.17-2.99
77.58
74.94
84.99
SC (CoT)
x
x
X
82.66,0.52
87.93 y2.53
96.88
88.69 1.51
75.08
77.30
84.75
SC (ComplexCoT)
X
x
X
83.65+1.51
16.74
f3.94
89.72-2.64
77.69
77.94
85.35
PHP
x
X
83.45
11.31
95.50
p10.1
98.10
(8.10 |2.84
90.02 -3.44
79.00
62.96,11.88
88.17
Chain
x
82.35
p0.21
10.17
94.38 11.23
83,41.a.77 La.77
70.94
80.88
82.92
Star
x
80.79
11:35
85.55 10.15
93.79.0.64
10.64
88.09
-0.01
68.57.1.77
68.57
11.77
p3.99
82.07
Tree
x
81.8910.22 10:25
84.56,0.84
94.60
89.25-2.07
72.84
77.38p5.70
83,42
Complete Graph
x
83.15
11.09
p7.20
89.48
-9.40 -2.30
79.21
79.21 18.87
83.75
86.55
Random Graph
x
83.76
16.74
95.46,2.31
85,41 .1.77
74.07
82.66
84.58
AutoGen
x
82.13 po.01
90.06
93.80
88.44.1.80 .122
73.65
85.41 113.73
85,58
MetaGPT
x
x
85.90
84.90
LLM-Blender
x
x
81.22 10.92
89.17 13:77
94.27 11.12
88.77-1.60
77.05
86.09
LLM-Debate
x
83.69 y1.55
90.23 14.83
f3.12
90.56 -3.38
77.52
83.79/12.11
87.01
DyLAN
80.16 11.98
88.16
94.2771.12
87.40-0.20 0.222
74.16
89.70
85.64
GPTSwarm
$3.98/1.84
89.74p4.34
97.84
86.42 .0.76
78.16
88.49
87.32
G-Designer
84.50 +2.36
95.07 95.0779.67
19.67
98.30 +5.15
91.85
91.85~4.67
79.47.9.13
r9.13
89.90
89.84
The experimental results from Table demonstrate that G-Designer is effective in designing powerful LLM-MA topologies. Concretely, G-Designer achieves the best performance in five out of six benchmarks, and GSM8K, it trails only PHP with 9.67% accuracy improvement. On the HumanEval benehmark, G-Designer surpasses MetaGPT, specialized multi-agent code gen- eration framework, by 4.0% at pass@ 1, and outperforms state-of-the-art multi-agent collaboration frameworks like GPTSwarm and DyLAN by margins of 0.20% ~ 1.41%.
Hlihhpperformig
MMLU
Eval
Annrrry
Pasog
SVAMP
E. 3 1
Thilan www.sepiiiitoooooo
Task-adaptive Figure 6 visualizes the different topolo- gies designed by G-Designer for varying query difficul- ties on HumanEval and GSM8K. As shown in Figure 6, the multi-agent topologies generated by G-Designer are highly dependent on the specific task context and its diffi- culty. In Case a, despite having five -4 agents available as design resources, G-Designer identified the task of de- signing a st sten (string) function as relatively simple. streamlined the topology by removing unnecessary agents and retained only a minimal "Algorithm Designer- Pro- grammer" structure to solve the problem. In contrast, for the more complex Case and Case G-Designes inner crafted more intricate communication graph. These cases highlight the strong task-adaptiveness of G-Designer.
Ancurary (K1
Aurrmyy (96
Figure 4. Visualization of the performance metrics and prompt token consumption of different multi-agent communication topolo- gies across MMLU, HumanEval, GSM8K, and SVAMP. The di- ameter of cach point is proportional to its y-axis value.
while the complete graph and GPTSwarm incur an over- whelming token cost at 20 agents (5.6 ~ 30.3M tokens), G- D-Designer achieves superior performance with merely 6. 11% of GPTSwarm's prompt token consumption, surpass- ing it by 2.44% t. These results decisively demonstrate the scalability and potential of G-Designer in advancing large-scale autonomous multi-agent systems.
Scalable To evaluate the scalability of G-Designer to a larger number of agents, we report its performance across 5 ~ 20 agents, as presented in Table 6. No- tably, G-Designer exhibits a steeper performance gain than GPTSSarmm as the agent count increases. More importantly,
Token-economical (inference) A key benefit of G-Designer's adaptivity is that it prevents the use of overly complex structures for simple tasks, thus minimizing
7

---

G-Designer
bdbor
ala stack
Table 3. Ablation study of G-Designer's four variants, tested on
MMLU benchmark.
MMLU
GSM8K
Variant
Clean
Attack
Clean
Attack
vanilla G-Designer
84.5
84.2
95.0
92.5
Clais
Star
Ime
Gangite
Kanlon
ceept
IyLAN
GriSnans Aatoaa G DDagnn
w/o SR
84.1
83.2
94.4
90.7
Figure 5. We compare the accuracy (%) of various multi-agent frameworks before and after prompt attacks MMLU.
w/o Anchor
84.0
83.8
94.7
92.0
w/o NodeEncoder()
83.2
82.4
92.8
87.4
w/o truk
81.3
82.0
90.3
87.7
unnecessary communication costs-in the case of L.M-- MA, reducing token consumption. Figure It illustrates the differences in prompt token consumption between G-Designer and several representative multi-agent designs. We observe that simpler topologies, such as complete graphs and random graphs, consume fewer tokens but show significantly weaker performance. More complex communication structures, like GPTSwarm and DyLAN, achieve superior performance, albeit at the cost of excessive token consumption. For instance, DyLAN's cost GSM8K is 2.82x that of the random graph, reaching substantial 2.2e + 7. In contrast, G-Designer elegantly balances both efficiency and task performance, achieving the highest performance across all four benchmarks while maintaining the lowest token cost. For example, on SVAMP, G-Designer surpasses DyLAN by 4% while using only 23.7% of DyLAN's token cost.
well, with DyLAN and AutoGin showing accuracy drops of 6.2% and 9.9%, respectively. Remarkably, G-Designer demonstrates exceptional robustness against adversarial at- tacks, maintaining nearly identical performance pre- and post-attack. This resilience can be attributed to its agent encoding capability, which, during optimization, can detect malicious inputs and prune the corresponding edges.
5.4. Framework Analysis
Ablation Study. We report results for two variants of ourmethod: (1) w/o SR, which removes the sparsity regular- ization in Equation (14), (2) w/o Anehor, which excludes the anchor structure Aanchor (3) w/o NodeEncoder, re- moving node encoder Equation (7), and (4) w/o Vinsk in Equation (8). As shown in Table removing the task virtual node disrupts G G-Designes task- addtaidativeessss, lead- ing to the most significant performance drop. The removal of A Aanchor consistently leads to performance degradation, while the absence of sparsity regularization makes the sys- tem more vulnerable to adversarial attacks.
Resource-efficient (training) We validate G-Designer's training process is resource-friendly from three dimen- sions: GPU cost, token cost, and wall-clock time. Table 5 showcases that training G-Designer with up to 1000 agents requires less than 4GB of memory. Table 2 unveils that G-Designer not only attains the highest accuracy but also exhibits superior token efficiency and reduced wall-clock time compared to existing baselines, underscoring its effec- tiveness in multi- agent collaboration.
Discussion on anchor topology. Given that G-Desi is initialized with the anchor topology Aunchor introduced in Section 4.1, one may question whether the perfor- mance gains of G-Designes primarily stem from Aunchor itself. In response, we emphasize that the anchor topol- ogy corresponds to the simple Chain structure in Table 1, where G-Designer achieves substantial improvements over it, specifically 9.50% t on GSMSK and 8. 8.44% 1 on SVAMP. Thus, we assert that the superior performance of G-Designer is predominantly attributed to its adaptive topology design rather than the anchor topology itself.
Table 2. Efficiency amalysis. We compare the training/inference
wall-clock time and token consumption between G-Designer and other hirh-periorming baselines on the GSM8K dataset.
Training
einference
*Overall
Training Time
Inference Time
Mathod
Perf,
Token
Token
Token
Complete
86.4
9.8 10"
9.8 x 10
2/th
DyLAN
88.1
9.6 x 10
1 x 10
2.2 x 102
288h
4.6h
GPTSwarm
89.7
5.5 x 10
8.4 10
14 10"
2.1h
2.8h
6. Conclusion
G-Deaignar
95.0
2.7 10
x 10
8.5 10
0.3h
2.5h
In this paper, we first present the LLM- Multi-agent Communication Protocol (MACP), which aims to provide insightful guidance for designing complex multi-agent sys- tems. Furthermore, we propose an effective, adaptive, and robust LLM- -----weeeed multi-agent communication graph de- signer, termed G- G-Designen to facilitate the automated de- sign of collaborative Al systems. G-Designez is highly task- aware, dynamically crafting compact and robust com- munication topologies based on the complexity of the task at hand. We hope that G-Designer will inspire future research
5.3. Robustness Analysis
Following (Zhuge et al., 2024), we simulate system prompt attack on one of the five agents. As seen in Figure 5, many trivial structures, such chain complete graph, expe- rience significant performance degradation under partial system attacks, with drops as high as 11.0%. Among more sophisticated structures, GPTSwarm, benefiting from its spe- cialized node optimization mechanism, only suffers a minor 0.3% accuracy decline. However, other methods fare less
8

---

G-Designer
on the emergence of self-organizing and self-evolving col- lective intelligence.
B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser,
L., Bavarian, M., Winter, C.,, Tillet, P., Petroski Such, F,, Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Hebgen Guss, W., Nichol, A., Paino, A. Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., Mc- Grew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code, July 01, 2021 2021.
Acknowledgements
The work is supported by the National Science Foundation of China (62472317).
Impact Statement
Ethical impacts. We confidently affirm that our paper is free of ethical concerns across its motivation, design, experiments, and data usage. The proposed G-Designes method aims to advance the fields multi-agen systems and communication topologies design automation, contribut- ing positively and responsibly the scientific community.
Chen, W., Su, Y., Zuo, J., Yang, C., Yuan, C., Qian, C., Chan, C.-M., Qin, Y., Lu, Y., Xie, R., Liu, Z.., Sun, M., and Zhou, J. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents, 2023b.
Expected societal implications. Our paper presents sig- nificant advancement in multi-agent systems. G-Designer addresses the challenge of choosing the right communi- cation topology for specific tasks. G-Designer not only provides a practical solution for multi-agent deployment but also paves the way for future studies on collective intelli- gence systems. Furthermore, as long as the base LLM used aligned with human values, our system will not generate harmful content.
Cheng, Y., Zhang, C., Zhang. Z., Meng, X., Hong, S., Li, W., Wang, Z., Wang, Z., Yin, F., Zhao, J., and He, X. Explor- ing large language model based intelligent agents: Defi- nitions, methods, and prospects. CoRR, Abs/2400.003428, 2024.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv prepring, abs/2110.14166, 2021.
References
Bouzenia, I., Devanbu, P., and Pradel, M. Repairagent: An autonomous, Ilm-based agent for program repair. arXiv preprint arXiv:2403.17134 2024.
Cohen, R., Hamri, M., Geva, M., and Globerson, A. Lm vs. Im: Detecting factual errors via cross examination. arXiv preprini arXiv:2305. 13281, 2023.
Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., and Liu, Z. ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate. arXiv e-prints, August 2023.
Du, Y., Li, S., Torralba, A., Tenenbaum, B., and Mordatch, I. Improving factuality and reasoning in language models through multiagent debate. CoRR, abs/2305. 143255 2023.
Fu, Y., Peng, H., Sabharwal, A., Clark, P., and Khot, T. Complexity- based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, 2022.
Chen, D., Wang, H., Huo, Li, Y., and Zhang, H. Gamegpt: Multi-agent collaborative framework for game develop- ment. arXiv preprint arXiv:2310. 00.08067, 2023a.
Chen, J., Lin, H., Han, X., and Sun, L. Benchmarking large language models in retrieval-augmennté generation. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 38, pp. 17754-17762, 2024a.
Gao, C., Lan, X., Li, N., Yuan, Y., Ding, J., Zhou, Z., Xu, F., and Li, Y. Large language models empowered agent-based modeling and simulation: A survey and per- spectives. CoRR, abs/2312.11970. 2023.
Chen, L., Davis, J. Q., Hanin, B., Bailis, P., Stoica, I., Za- haria, M., and Zou, J. Are more I1m calls all you need? towards scaling laws of compound inference systems. arXiv preprint ar 423:42243..02419 2024b.
Hao, R., Hu, L., Qi, W., Wu, Q. Zhang, Y., and Nie, L. Chatllm network: More brains, more intelligence, April 01, 2023 2023.
Chen, M., Tworek, J. Jun, H., Yuan, Q. Ponde de Oliveira Pinto, H., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan,
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.

---

G-Designer
Holt, S., Luyten, M. R., and der Schaar, M. L2mac: Large language model automatic computer for extensive code generation. In The Twelfth International Conference on Learning Representations, 2024.
Li, M., Zhao, Y., Yu, B., Song, F, Li, H., Yu, H., Li, Z.., Huang, F., and Li, Y. Api-bank: A comprehen- sive benchmark for tool-augmented Ilms, arXiv preprint arXiv:2304.088 082444, 2023b.
Hong, S., Zheng, X., Chen, J., Cheng, Y., Wang, J., Zhang, C., Wang, Z.,, Yau, S. K. S., Lin, Z.,, Zhou, L., Ran, C., Xiao, L., and Wu, C. Metagpt: Meta programming for multi-agent collaborative framework, August 01, 2023 2023.
Liang, T., He, Z., Jiao, W. Wang, X., Wang, Y., Wang, R., Yang, Y., Tu, Z., and Shi, S. Encouraging divergent thinking in large language models through multi-agent debate. CoRR, abs/2305. 19115, 2023.
Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. Pro- gram induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv: / /705 00005.44100 04/.0044446.. /46, 2017.
Hu, S., Shen, L., Zhang, Y., and Tao, D. Learning multi- agent communication from graph modeling perspective. arXiv preprint wwxi::24050085500 2024.
Hua, W., Fan, I.., Li, L.., Mei, K., Ji, J., Ge, Y., Hemphill, L.., and Zhang, Y. War and peace (waragent): Large language model-based multtiagent simulation of world wars. arXiv preprint arXiv:231 172227, 2023.
Liu, Y., Dou, Y., Li, Y., Xu, X., and Liu, D. Temporal dynamic weighted graph convolution for multi-agent re- inforcement learning. Proceedings of the Annual Meet- ing of the Cognitive Science Society, volume 44, 2022.
Ishibashi, Y. and Nishimura, Self-organized agents: A I1m multi-agent framework toward ultra large-scale code generation and optimization. arXiv preprint arXiv:2404.02183,, 2024.
Liu, Z., Zhang, Y., Li, P., Liu, Y., and Yang, D. Dynamic I1m- agent network: An Ilm-agent collaboration framework with agent team optimization. CoRR, abs/2310.021700 2023.
Jiang, D., Ren, X., and Lin, Y. LLM--bender: Ensembling large language models with pairwise ranking and genera- tive fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 14165-14178, Toronto, Canada, July 2023. Association for Computational Linguistics.
Ma, K., Zhang, H., Wang, H., Pan, X., Yu, W., and Yu, D. Laser: Lim agent with state-space exploration for web navigation. arXiv preprint arXiv:2309.08177 2023.
Nakajima, Y. Babyagi. https://gitth: com/
voheinakaajima/aaaa
2023.
Jin, Y., Shen, X., Peng, H., Liu, X., Qin, J., Li, J., Xie, Gao, P., Zhou, G., and Gong, J. Surrealdriver: Designing generative driver agent simulation framework in urban contexts based on large language model, 2023.
Patel, A., Bhattamishra, S., and Goyal, N. Are nip models really able to solve simple math word problems? arXiv preprint 103.v:2103.07/1 2021.
Khattab, O., Singhvi, A., Maheshwari, P., Zhang. 7.,, San- thanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., et al. Dspy: Compiling declarative language model calls into self- improving pipelines. arXiv preprint arXiv:2310.033 1.0-3344, 2023.
Pesce, E. and Montana, G. Learning multi-agent coordi- nation through connectivity-driven communication. Mu- chine Learning, 12(2):483-514, 2023.
Qian, C., Cong, X., Yang, C., Chen, W., Su, Y., Xu, J., Liu, Z., and Sun, M. Communicative agents for software development, July 01, 2023 2023. 25 pages, 9 figures, 2 tables.
Kipf, T. N. and Welling, M. Variational graph auto-encoders. arXiv preprint arXiv: 1611:0761100 2016.
Kipf, N. and Welling, M. Semi-supervised classifica- tion with graph convolutional networks. 5th Interna- fional Conference on Learning Representations. OpenRe- view.net, 2017.
Qian, C., Xie, Z., Wang, Y., Liu, W., Dang, Y., Du, Z., Chen, W., Yang, C., Liu, Z., and Sun, M. Scaling large- minguage-model-basee multi-agent collaboration. arXiv preprint arXiv:2406.07155, 2024.
Li, G., Hammoud, H., Itani, H., Khizbullin, D., and Ghanem, B. CAMEL: communicative agents for "mind" explo- ration of large language model society. In NeurIPS, 2023a.
Reimers, N. Sentence-bert. Sentence embeddings using siamese hert-networks. arXiv preprint arXiv:/909 10084, 2019.
Li, J.. Zhang, Q., Yu, Y., Fu, Q., and Ye, D. More agents is all you need. CoRR, abs/2402.05 05 0500,, 2024.
Reworkd. Agentgpt.
https:://gtttubccc. reworkd/Agent.or 2023.
10

---

G-Designer
Richards, T. B. and et al. Auto-gpt: An tonomous gpt-4 experiment Significant Gravitas/Auto- GPI, 2023.
Williams, R. J. Simple statistical gradient-folowing algo- rithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
com/
Rosenbluth, E., Tönshoff, J., Ritzert, M., Kisin, B., and Grohe, M. Distinguished in uniform: Self attention vs. virtual nodes. arXiv preprint arXiv:2405. 1951, 2024.
Wu, Q. Bansal, G., Zhang. J., Wu, Y., Zhang, S., Zhu, E., Li, B., Jiang, L., Zhang, X., and Wang, C. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework, August 01, 2023 2023.
Roy. S. and Roth, D. Solving general arithmetic word problems. arXiv preprint arXiv: /eti:::608..11333 2016.
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang. M., Wang, J., Jin, S., Zhou, E.,, Zheng, R., Fan, X., Wang, X., Xiong, L.,, Zhou, Y., Wang, W., Jiang, C., Zou, Y., Liu, X., Yin, Z., Dou, S., Weng, R., Cheng, W. Zhang. Q. Qin, W., Zheng, Y., Qiu, X., Huan, X., and Gui, T. The rise and potential of large language model based agents: A survey. arxiv preprint, abs/2309.07864 2023.
Shinn, N., Labash, B., and Gopinath, A. Reflexion: an autonomous agent with dynamic memory and self- reflection. arXiv preprint, abs/2303.11366, 2023. doi: 10. 48550/arxiv:2303. 11366. URL hwtps:tpps... /rrg 10. 00.448550/aa http://x...222...113333
Shirzad, H., Velingker, A.. Venkatachalam, B., Sutherland, D. J., and Sinop, A. K. Exphormer: Sparse transform- ers for graphs. In International Conference on Machine Learning, pp. 31613-31632. PMLR, 2023.
Yan, Y., Zhang, Y., and Huang, K. Depending on yourself when you should: Mentoring Ilm with rl agents to be- come the master in cybersecurity games. arXiv preprint arXiv:2403. 17674, 2024.
Tan, Z., Guo, R., Ding, K., and Liu, H. Virtual node tuning for few-shot node classification. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2177-2188, 2023.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models, May 01, 2023 2023a.
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An Open- Ended Embodied Agent with Large Language Models. arXiv e-prints, art. arXiv:2305. 16291, May 2023.
Yao, S., Zhao, J., Yu, D., Du N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023b.
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., Zhao, W. X., Wei, 7., and Wen, J. A survey on large language model based autonomous agents. Front. Comput. Sci., 18, 2024.
Yin, Z., Sun, Q., Chang, C., Guo, Q., Dai, J., Huang, X.- J., and Qiu, X. Exchange-of-thougee language model capabilities through cross-model com- munication. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15135-15153, 2023.
Enhancing large
Wang, W., Wei, F., Dong, I.,, Bao, H., Yang. N., and Zhou, M. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33: 5776-5788, 2020.
Zhang, G.. Yue, Y., Li, Z., Yun. S., Wan, G., Wang, K., Cheng, D., Yu, J. X., and Chen, T. Cut the crap: An economical communication pipeline for llmebased multi- agent systems. arXiv preprint arXiv:2410.02506, 2024.
Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. The Eleventh International Conference on Learning Representations, 2023a.
Zhang, J., Xu, X., and Deng, S. Exploring collaboration mechanisms for Ilm agents: A social psychology view. arXiv preprint arXiv:2310.02124 2023a.
Wang. Z., Mao, S., Wu, W., Ge, T., Wei, F., and Ji, H. Unleashing cognitive synergy in large language mod- els: task-solving agent through multi-persona self- collaboration, July 01, 2023 2023b. work in progress.
Zhang, J., Xu, X., and Deng, S. Exploring collaboration mechanisms for Ilm agents: A social psychology view. arXiv preprint 10.002231 2023b.
Zhang, P. and Chartrand, G. Introduction graph theory. Tata McGraw- 2:2-1, 2006.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models, January 01, 2022 2022.
Zhao, K. and Zhang, L. Causality -inapiee spatial-temporal explanations for dynamic graph neural networks. The

---

G-Designer
Twelfth International Conference Learning Represen- fations, 2024.
Zheng, C., Liu, Z., Xie, E.,, Li, Z., and Li, Y. Progressive- hint prompting improves reasoning in large language models, April 01, 2023 2023, Tech Report.
Zhou, Z., Hu, B., Zhao, C., Zhang, P., and Liu, B. Large language model as a policy teacher for training reinforce- ment learning agents, arXiv preprint 1. 2023.
Zhuang, Y., Chen, X., Yu, T., Mitra, S., Bursztyn, V., Rossi, R. A., Sarkhel, S., and Zhang, C. Toolchain Efficient action space navigation large language models with ** search. arXiv preprint arXIv:2310.13227. 2023.
Zhuge, M.. Wang, W., Kirsch, L., Faccio, F., Khizbullin, D., and Schmidhuber, J. Gptswarm: Language agents as op- timizable graphs. In Forty-first International Conference on Machine Learning. 2024.
12

---

G-Designer
A. Algorithm Workflow
Algorithm Designing workflow of G-Designer
Input :Input query Q, Graph auto-encoder fo composed of encoder q(-) and decoder p(-) (parameterized by O₂ and e),
learning rate
for query d in 21,2. D do
* Establish multi-agent. network
node in (1,2, N do
x; t Nodenn (T (T( Role, T(Plugin:))
end
Obtain agent embeddings Xagent f- X1, X2, xx]
Obtain task-specific node Trask NodeEnocderr(). (ua)
Set an anchor topology Aanchor // In our experiments, the anchor topology is simply set as the chain structure
Obtain task-specific multi-agent network ğ = ( Xagent Aanchor) // Note that Aanchor here Xyask
contains rictrectionn rectional edges added by the task node Ctask Design communicat ion topology
*
*/
Encode 9 into latent agent representations H. Hagent: g (Hagent X, Aanchor) = II 7(h; X, Aanchor)
Decode (phase 1) and obtain the sketched graph S: (S H Hagent) = II I1 Ps (Sig h; hy. htaski Oa),
Decode (phase 2) and obtain the communication graph Gcom = (V, Ecom) Ecom = [(t,] Sig #0A E EE)) * Guide mult aaagegett system collaboration for iteration in (1,2.-.. _K] do
for node in (Goom) do
Agent e generates R!d) +- vy p(u)) P(C = (2, (2, R,) end
/ Aggregate solution
(+) + Aggregate(R R , R((
end
* Update G-Designer parameters
@d+1 + @d aV DeDeegger
end
B. Dataset Statistics
We conclude the dataset statistics in Table
C. Case Study
Figure visualizes the different topologies designed by G-Designer for varying query difficulties on the HumanEval and GSM8K benchmarks.
D. Supplementary Results
13

---

G-Designer
Table 4. Dataset descriptions and statistics.
Category
Dataset
Answer Type
Metric
#Test
License
General reasoning
MMLU
Multi-choice
Acc.
153
MIT License
GSM8K
Number
Acc.
1,319
MIT License
MultiArith
Number
Acc.
600
Unspecified
Math reasoning
SVAMP
Number
Acc.
1,000
MIT License
AQuA
Multi-choice
Acc.
254
Apache-2.0
Code generation
HumanEval
Code
Pass
164
MIT License
HumanEval (case
HumanEval (case b)
GSMBK (case
def strrenn (string: str) -> int:
def the differenceeIst: Lisi[ [loat]) -> int:
Background:
Return length of given string strlen(")
A robe takes bolls of blue fiber and half that much
Given a mumbers, retim sum squares the numbers in the list that odd. Ignore numbers that are negutive not integers. If the inpul is emptly, return O.
white fiber.
Question:
strlen('abe')
How many bolts total does take?
3
Algonithmm Designer
Bug Fixer
Anolyst
Math
Math Solver
Algorithm Designer
Programming Expert
Programming
Programming Expert
Test Andlyst
Inspector
KB
Expert
HumanEval (case
GSM8K (case e)
def de, elgebra(eperator: Liststr]] operand: Lis((nt) > int:
Question:
John drives for 3 hours speed of 60 and then around because he realines he forgot something very important home. tries home hours but spends the first 2 hours in standstill traffic. He spends next half-hour driving speed of 30mph, before being able to drive the remaining time the hours going 80 mph. How he from home at end of those hours?
Given two lists operator, and operand. The first list has basic algehra operations, and the second is list integers, Use two given lists build the algebraic expression and retur the evaluation expression. The basic operations: Addition( Suberaction(::: Multiplication! ") Floor division(/1:
Exponeniaaionn "). Example: operator"""' v, "I; Note: 1. The length of operator list is equal to the length of operand list minus one. Operand non-negative integers. Operator list has lcast
[2. 3. S]reosutt - 2+
Sp-0 resull 9
of and operand list has least two operands.
Algonithm
Bug Fixer
Math Anclyst
Math
Designer
Harder Example:
Solver
G- G-Designer is complexity- awore and capable of designing task-adoptive topologies
Project Manager
Programming Expert
S+
Programmingg Expert
Inspector
Analyst
Figure Case study the communication topologies designed by G-Designer on HumanEval GSM8K benchmarks.
Table 5. The GPU cost of G-Designer with increasing number of agents.
#Agents
50
100
1000
Memory (GB)
2.7
2.9
3.0
3.8
14

---

G-Designer
Table Comparison of accuracy, time, token consumption, and cost across different agent configurations. We use the MMLU benchmark and gpt-3 3. 5-turbo as the base LLM.
#Agents
5
10
20
Chain
Accuracy (%)
70.59
71.24
71.98
Time (min)
15.73
30.20
56.18
#Prompt Tokens
351,802
702,164
1,378,328
Cost (USD)
0.5228
1,0434
2.0482
Complete Graph
Accuracy (%)
71.90
72.16
72.51
Time (min)
16.85
34.21
66.47
#Prompt Tokens
545,984
1,669,451
5,648,834
Cost (USD)
0.7161
2.1770
7.3662
GPTSwarm
Accuracy (%)
72.55
73.86
75.38
Time (min)
62.14
186.86
412.18
#Prompt Tokens
3,055,236
9,048,465
30,317,341
Cost (USD)
4.2190
12.4961
41,4235
G-Designer
Accuracy (%)
73.20
74.51
77.82
Time (min)
19.26
36.04
68.89
#Prompt Tokens
452,329
885,332
1,852,538
Cost (USD)
0.6036
1,2768
2.6713
15


Reviews:
[{"review_id": "dz7EhFg8kO", "reviewer": "Reviewer_3iyh", "summary": "The paper discusses the advancements in collective intelligence among large language model-based agents, highlighting the challenge of selecting effective communication topologies for specific tasks. To address this, the authors introduce G-Designer, an adaptive solution that dynamically creates task-aware communication topologies using a variational graph auto-encoder. This approach aims to balance efficiency and performance by customizing the inter-agent communication network according to the task requirements.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **`Weakness 1`**: What happens to the communication protocol if a graph has a directed cycle?\n\nThank you for the insightful inquiry! In Equation (15), we extract an enforced acyclic graph $\\mathcal{G}\\_{com}$ from the potentially cyclic topology $\\tilde{\\mathbf{S}}$. Specifically, in our provided anonymous code repository, the function `check_cycle()` in `GDesigner/graph/graph.py` is responsible for avoiding cycles when constructing $\\mathcal{G}\\_{com}$. We will emphasize this point more clearly in the revised version.\n\n---\n> **`Weakness 2`**: Why are these 3 criteria considered?\n\n\n1. **Effectiveness**: Extraordinary problem-solving is one of the core motivations behind MAS.  \n2. **Complexity Adaptiveness** aims to achieve optimal performance with minimal resource consumption. This is emphasized or pursued by many recent multi-agent papers, such as AgentPrune (ICLR 2025), RouterDC (NeurIPS 2024), and GraphRouter (ICLR 2025).\n3. **Adversarial Robustness**: As attack & defense mechanisms in Agent/MAS gain increasing attention (e.g., TrustAgent (EMNLP 2024)), we believe safety and robustness will be critical for the trustworthy deployment of MAS in real-world applications.  \n\nWe respectfully acknowledge that while we have highlighted what we consider critical, we do not claim this framework to be exhaustive. We would greatly appreciate any additional insights you might suggest and would be happy to incorporate them into our future work.\n\n---\n> **`Weakness 3`**: The starting anchor is simple.\n\nWe would like to respectfully clarify that the anchor is not an \"almost random guess\". Instead, as stated in line 247, it incorporates prior knowledge of the workflow sequence (for example, a simple but standard coding procedure). In Section 5.4 and Table 3, we demonstrate that introducing the starting anchor results in a practical performance improvement. \n\n---\n> **`Weakness 4`**: Equation 14 are not correctly referred.\n\nThank you for your thorough review! We will correct the reference to Eq. (14) in Lines 278–288 in the revised version.\n\n---\n> **`Weakness 5 & 7`**: No details about the training of the proposed method are given.\n\n- Training set: For both our method and the baselines that require training, we use the same training set. As mentioned in Line 321 (Right) of the paper: 40 samples for MMLU/MultiArith/HumanEval, 80 samples for GSM8K/SVAMP/AQuA;\n- Test set: provided in Table 4;\n- Hyperparameter tuning: We use a validation set of size 100 for each benchmark and apply a grid search to find the optimal parameters. To reduce the complexity of hyperparameter tuning for users, we provide a set of generalized hyperparameters in Section 5.1.\n\n---\n> **`Weakness 6`**: The paper lacks proper motivation.\n\nThank you for bringing this issue to our attention! We humbly provide a more intuitive explanation of each component:\n\n- Section 4.1: Makes it possible to process MAS with VGAE.\n- VGAE encoder $q(\\cdot)$: Captures the semantic relationships among different agents regarding a given task.\n- VGAE decoder $p(\\cdot)$: Directly leveraging complex MetaGPT/LLM-Debate/GPTSwarm will cause unnecessary costs on simple queries.\n- Equation 14: If the optimized topology is not regularized, it may suffer from malicious edges or unnecessary edges. This practice has been well-validated in the traditional graph structure learning literature, such as NeuralSparse (ICML 2020), PTDNet (WSDM 2021).\n\n---\n> **`Weakness 7`**: If the baseline approaches need additional training?\n\nIn the baselines of our paper, DyLAN, LLM-Blender, and GPTSwarm require training.\n\nWe compare with other training-free methods because **all previously published training-required methods (including DyLAN (COLM'24), LLM-Blender (ACL'23) and GPTSwarm (ICML'24)) have compared with training-free methods**. Therefore, we follow the same setting as theirs for a comprehensive evaluation. \n\nBesides, we respectfully argue our approach has the following key advantages **in a fair way**: \n\n(1) G-Designer has **better performance** than training-free/required topologies; (2) its **training cost** is an order of magnitude lower than all training-required methods (Table 2). Additionally, our **inference cost** is among the lowest (Figure 4).\n\n---\n\n> **`Weakness 8`**: Clarify the details of all the agents used in Table 1.\n\nThe profiles and tools of different agents vary significantly, resulting in distinct encoded node features. Consequently, the constructed graph varies significantly. We provide a case study (Figure 6) to illustrate this. Examples of different agent role/tool descriptions can be found in our provided code in `GDesigner/prompt/gsm8k_prompt_set.py`.\n\n---\n> **Weakness 9**: The variance of G-Designer is high in Table 1?\n\nWe would like to clarify a possible misunderstanding: the subscripts in Table 1 indicate improvements over the vanilla baseline rather than variance. \n\nBesides, we would like to emphasize that all results in Table 1 are averaged over three runs.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Yes, the contributions of the paper are related to the broader scientific literature of agentic frameworks of LLMs.", "essential_references_not_discussed": "All the essential references are discussed.", "other_strengths_and_weaknesses": "Strengths:\n\n1. It addresses an important problem of learning the communication network structure in a multi-agent system of LLMs.\n\n2. Proposing a formal multiagent communication protocol is important. This paper attempts it.\n\n3. Some analysis done in the paper w.r.t. the number of agents and number of tokens are very insightful.\n\n\nWeaknesses / Confusions:\n\n1.  In Section 3.2, what happened to the communication protocol if a graph has a directed cycle?\n\n2. It is not clear why these 3 criteria are considered as the Mukti agent communication protocol? Is this list exhaustive?\n\n3. The need of the simple starting anchor matrix \\tilta{A} is not clear. If A is an almost random guess, why do you want your final topology to be close to it in Equation 14?\n\n4. In Line 279, terms in Equation 14 are not correctly referred.\n\n5. Equations 14 and 16 are both trainable optimisation functions with hyperparameters. Good amount of training data will be needed to optimize these objectives. The authors just mentioned that limited training data is required without explaining details.\n\n6. The overall paper looks like an overkill and is often made complex without any proper motivation / intuition about where existing simpler things would fail?\n\n7. In the implementation details, no details about the training of the proposed method is given. You need to provide training data name, size, validation and hyperparameter tuning details. Also, it is not clear if the baseline approaches need additional training. If yes, are they also being trained on the same datasets? If no, how do you ensure fairness in your experiments?\n\n8. Can you clarify what all agents being used in Table 1? If they are all the same(GPT4 based), will all the nodes in the graph have similar node features and the final topology may boil down to a symmetric graph structure wrt the nodes?\n\n9. The variance of G-Designer is high in Table 1? Please throw some light on the statistical significance of the results? Also, is it because of the additional uncertainties because of the training?\n\n\n\n\"**Update after rebuttal**\"\n\nI have checked the rebuttal. I thank the authors for their rebuttal. Some of my concerns are addressed. However, concerns around the correctness of the equations, motivation about some steps of the algorithm and details of the training are still there. I will hold my overall score.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L7gVAnxO2q", "reviewer": "Reviewer_LUTJ", "summary": "This paper introduces G-Designer, an innovative solution for multi-agent communication topology design in LLM-MAS. The authors first propose the Multi-agent Communication Protocol (MACP), which sets standards for LLM-MAS topology design in terms of effectiveness, complexity-adaptiveness, and adversarial robustness.\nG-Designer models MAS as a graph, using a variational graph auto-encoder. It constructs a multi-agent network with a task-specific virtual node and then decodes an optimized communication topology. This is achieved through a process of encoding agents and task information, decoding a sketched graph, and refining it with anchor and sparsity regularization.\nExtensive experiments on six benchmarks demonstrate G-Designer's superiority.\n\n## update after rebuttal\nThe authors address most of my concerns. So I raised my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the thoughtful and constructive reviews of our manuscript! Based on your questions and recommendations, we give point-by-point responses to your comments and describe the revisions we made to address them.\n\n---\n> **`Weakness 1`**: Temperature Settings in the Experiments\n \nThank you very much for your thorough review! In the multi-agent system, to facilitate more diverse communication among different agents, we followed DyLAN and set the temperature to 1. For the Single Agent scenario, to ensure more reproducible experimental results, we set the temperature to 0. \n\nFor a fairer comparison, we have also presented the performance of the Single Agent method with the temperature set to 1 in `Table A`.  Experimental results show that the change in temperature from 0 to 1 does not significantly affect the average accuracy of single-agent methods.\n\n*Table A. Performance of single agent methods with temperature set to 1.*\n| Method | MMLU | GSM8K | HumanEval |\n|:-:|:-:|:-:|:-:|\n| Vanilla | 82.14 | 85.40 | 71.68 |\n| CoT | 82.55 | 86.81 | 75.80 |\n| ComplexCoT | 83.70 | 87.04 | 75.66 |\n\n---\n\n> **`Weakness 2`**: The reason for setting the number of dialogue rounds.\n\nYour question is very valuable! In Table B, we present the changes in accuracy and cost with different numbers of dialogue rounds on the HumanEval dataset. Intuitively, increasing the number of optimization rounds leads to more refined and accurate results, yielding substantial performance improvements, but also comes with increased cost. To balance performance with token savings, we consistently set $K = 3$.\n\n*Table B. We report the performance of G-Designer on HumanEval benchmark with different $K$ values.*\n| K | 1 | 3 | 5 | 7 |\n|:-:|:-:|:-:|:-:|:-:|\n| Acc | 88.21 | 89.90 | 90.66 | 91.08 |\n| Cost | 0.1325 | 0.2298 | 0.3339 | 0.4097 |\n\nAs regards the automation of $K$, we respectfully argue that G-Designer can easily achieve this via incorporating existing early-stopping mechanisms like in MacNet or DyLAN. Nevertheless, G-Designer not particularly leverage these, as its core contribution lies in the automation of MAS topology.\n\n---\n\n> **`Weakness 3`**: The impact of different anchor settings on the results.\n\nThe suggestion you proposed is very thoughtful! In Table C, we explored the impact of different initial topologies on the experimental results.\n\n*Table C: Different Topologies for anchor graph initialization on the GSM8K dataset.*\n| Anchor | Chain | Layered | Star | Random | FullConnected |\n| :-: | :-: | :-: | :-: | :-: | :-: | \n| Accuracy | 95.07 | 94.92 | 94.47 | 95.15 | 95.22 |\n| Token | $8.5\\times 10^6$ | $9.3\\times 10^6$ | $8.9\\times 10^6$ | $9.2\\times 10^6$ | $1.2\\times 10^7$ | \n\nUsing different topologies as initial anchors shows minimal performance differences, although the cost is higher. This aligns with our analysis in Section 5.4, where we concluded that the success of G-Designer is mainly attributed to its efficient communication graph generation.\n\n---\n\n> **`Weakness 4`**: Have the authors considered using the latest GPT or LRMs?\n\nTo address your concern, we use the newer and popular closed-source model GPT-4o-mini and the open-source model DeepSeek-V3 as base models, and compare the experimental results of our method with Vanilla and GPTSwarm in Table D. \n\n*Table D. Performance results with more advanced LLMs.*\n| Method | LLM | MMLU | GSM8K | HumanEval |\n|:-:|:-:|:-:|:-:|:-:|\n| Vanilla | GPT-4o-mini | 77.12 | 92.67 | 85.71 |\n| Vanilla | DeepSeek-V3 | 86.27 | 93.25 | 87.67 |\n| GPTSwarm | GPT-4o-mini | 78.43 | 93.78 | 86.28 |\n| GPTSwarm | DeepSeek-V3 | 86.93 | 94.38 | 89.72 |\n| G-Designer | GPT-4o-mini | 79.73 | 94.23 | 90.32 |\n| G-Designer | DeepSeek-V3 | 89.54 | 95.52 | 91.93 |\n\nDue to the time constraints of the rebuttal, we were unable to compare our method with more baselines. However, the experimental results presented above demonstrate that our method is also applicable to newer LLMs.\n\n---\n\n> **`Weakness 5`**: Have the authors considered testing with more agents?\n\nThank you for this thoughtful inquiry! In Table 6 of our paper, we provide a comparison of accuracy, time, token consumption, and cost across different agent number configurations. The experimental results show that G-Designer is still able to maintain its cost efficiency with more agents.", "claims_and_evidence": "The core claim of this paper is that the topology of MAS should be task-dynamic. The authors first support this through the method design: G-Designer is trained to design a task-customized communication graph for different domains and user queries of varying difficulties. Secondly, experimental verification supports this: Fig 4 shows that different graphs for different queries can significantly reduce the average token consumption while maintaining good performance. Finally, case studies verify this: Fig 6 shows that G-Designer can indeed design different topologies for different tasks.", "methods_and_evaluation_criteria": "This paper is the first to use graph neural networks for topology design in LLM-MAS, which is novel. It uses the most commonly used datasets and evaluation metrics.", "theoretical_claims": "No new theorems or proofs provided.", "experimental_designs_or_analyses": "The benchmarks used in this paper are standard, and the authors comprehensively compared single-agent and multi-agent baselines. I have the following concerns:\n\n - In Section 5.1, why is the temperature set to 0 for single agents and 1 for other MAS? Could this introduce inconsistencies when reporting the performance?\n - Section 5.1 mentions that the number of dialogue rounds is K=3. Can the authors explain this? Would other values significantly affect performance? Can this value be automated rather than mannual defined?\n - The authors state that the anchor topology is set as a chain structure. Have the authors tried other structures?", "relation_to_broader_scientific_literature": "This paper is highly relevant to collaborative AI and multi-agent systems. It is the first to advocate for the task dynamics of LLM-MAS and proposes a unified protocol to regulate future LLM-MAS designs, which I believe is important for the community.", "essential_references_not_discussed": "The authors have comprehensively discussed relevant papers.", "other_strengths_and_weaknesses": "### Strength\n\n - Using graph neural networks and variational autoencoding for adaptive MAS topology design seems quite novel.\n - The visualizations are all well-designed and clearly presented.\n - The experimental results of the method are good. Detailed ablation studies validate the method's advantages in performance, token efficiency, and adversarial robustness.\n\n### Weakness\n\n - The authors only used GPT-3.5 and GPT-4. Have they considered using the latest GPT or LRMs?\n - Table 1 is limited to five agents. Have the authors considered testing with more agents (100 or 1000)? Can G-Designer maintain its cost efficiency with more agents?", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aLTL7xQ40L", "reviewer": "Reviewer_gFC8", "summary": "This paper introduces G-Designer, an framework for designing task-aware, adaptive, and robust communication topologies for multi-agent systems powered by large language models (LLMs). G-Designer dynamically generates communication structures for specific user requests using a variational graph auto-encoder (VGAE), which encodes both agent profiles and task-specific information to decode an optimal interaction graph. This method aims to balance performance, adaptability, and robustness by minimizing communication overhead while maintaining high accuracy and resilience against adversarial attacks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our deepest respect for your meticulous review! In response to your efforts, we have carefully prepared a point-by-point reply:\n\n---\n> **`Weakness 1`**: Supplement more agent-specific benchmarks.\n\nThank you very much for your insights. We conducted experiments on the more agent-specific GAIA benchmark to assess the effectiveness of our method. The experimental results are as follows:\n\n*Table A. Performance comparison on GAIA benchmark.*\n|Method|Level 1|Level 2|Level 3|Avg.|\n|-|-|-|-|-|\n|Vanilla GPT-4|9.68|1.89|2.08|4.05|\n|DyLAN|13.98|4.40|0|6.69|\n|GPTSwarm|23.66|16.35|2.04|16.33\n|**G-Designer**|**25.16**|**18.87**|**2.04**|**16.94**|\n\n\nThis demonstrates that our method effectively outperforms baselines even in more agentic benchmarks like GAIA. We sincerely hope this addresses your concerns.\n\n---\n> **`Weakness 2`**: Include the recent multi-agent papers.\n\nYour erudition and wisdom have been immensely helpful to us! We commit to including the papers you mentioned in the revised manuscript.\n\n---\n> **`Weakness 3`**: Are the authors planning to test more advanced or open-source LLMs?\n\nTo address your concern, we use the newer and popular closed-source model GPT-4o-mini and the open-source model DeepSeek-V3 as base models, and compare the experimental results of our method with Vanilla and GPTSwarm in Table B. \n\n*Table B. Performance results with more advanced LLMs.*\n| Method | LLM | MMLU | GSM8K | HumanEval |\n|:-:|:-:|:-:|:-:|:-:|\n| Vanilla | GPT-4o-mini | 77.12 | 92.67 | 85.71 |\n| Vanilla | DeepSeek-V3 | 86.27 | 93.25 | 87.67 |\n| GPTSwarm | GPT-4o-mini | 78.43 | 93.78 | 86.28 |\n| GPTSwarm | DeepSeek-V3 | 86.93 | 94.38 | 89.72 |\n| G-Designer | GPT-4o-mini | 79.73 | 94.23 | 90.32 |\n| G-Designer | DeepSeek-V3 | 89.54 | 95.52 | 91.93 |\n\nDue to the time constraints of the rebuttal, we were unable to compare our method with more baselines. However, the experimental results presented above demonstrate that our method is also applicable to newer LLMs.\n\n---\n> **`Weakness 4`**: Do the authors plan to evaluate G-Designer’s defense capabilities against a broader range of attacks?\n\nTo address your concerns, we evaluated G-Designer under the PoisonRAG [1] attack, with results presented in the table below:\n\n*Table C. Performance comparison on MMLU. We employ the configuration from PoisonRAG, inserting erroneous messages into the contextual memory of the attacker agent, enabling them to disseminate conclusions derived from these messages to others.*\n|Method|Chain|Star|Complete Graph|DyLAN|GPTSWarm|G-Designer|\n|-|-|-|-|-|-|-|\n|Before attack|82.3|80.7|83.1|83.4|84.0|84.5|\n|After attack|73.6|65.8|75.2|77.9|81.0|83.6|\n\nThe results indicate that even under memory attacks, G-Designer effectively mitigates interference from malicious agents through its dedicated sparsification mechanism.\n\n[1] Poison-rag: Adversarial data poisoning attacks on retrieval-augmented generation in recommender systems.\n\n\n---\n> **`Weakness 5`**: Do the authors plan to explore alternative anchor graph initialization?\n\n\nThe suggestion you proposed is very thoughtful! In `Table C`, we explored the impact of different initial topologies on the experimental results.\n\n*Table D: Different Topologies for anchor graph initialization on GSM8K dataset.*\n\n|Anchor|Accuracy|\n|-|-|\n|Chain|95.07|\n|Layered|94.92|\n|Star|94.47|\n|Random|91.15|\n|FullConnected|95.22|\n\nUsing different topologies as initial anchors shows minimal performance differences, although the cost is higher. This aligns with our analysis in Section 5.4, where we concluded that the success of G-Designer is mainly attributed to its efficient communication graph generation.", "claims_and_evidence": "The main claims by this paper (i.e., task-awareness, adaptiveness, robustness, high-performance) are well supported by the methodology and experiments. I particularly appreciate the emphasis on task-adaptiveness in G-Designer, a feature that has been largely absent in prior LLM-based multi-agent systems. Systems such as MetaGPT and ChatDev adhere to static SOPs, and MacNet exhibits poor scaling capabilities due to its fixed structure. Although GPTSwarm incorporates dynamic training, it relies on a fixed topology during testing. In the latest ICLR 2025 papers, works like AFlow, AgentPrune, and ADAS fail to dynamically allocate inference resources based on task complexity. Therefore, I think G-Designer’s advocacy for task-adaptiveness and its proposed Multi-agent Communication Protocol (MACP) represent a significant contribution that warrants attention from the research community.", "methods_and_evaluation_criteria": "The authors primarily employed three categories of benchmarks. Overall, the results on these benchmarks support the superiority of their approach. However, I recommend that the authors include more agent-specific benchmarks, such as API-Bank [1] and AgentBench [2].\n\n[1] API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs\n\n[2] AgentBench: Evaluating LLMs as Agents", "theoretical_claims": "No significant theoretical issues identified.", "experimental_designs_or_analyses": "G-Designer is compared against vanilla LLM, single-agent, and multi-agent methods. In the cost-efficiency scatter plot (Figure 4), G-Designer achieves the best performance with the lowest token cost among all methods. Additionally, I highly appreciate the case study and scalability study presented in Figure 6 and Table 6, which I strongly suggest be moved to the main text. Figure 6 provides an intuitive understanding of G-Designer’s operational logic, and Table 6 suggests that G-Designer may offer preliminary insights into agent scaling laws. Current graph-based multi-agent systems, such as MacNet and GPTSwarm, suffer from poor scaling laws, with marginal performance improvements as the number of agents increases (MacNet is especially criticized by reviewers for this issue). G-Designer explicitly highlights this challenge, namely the quadratic growth in communication edges, for which the authors propose a lightweight solution.", "relation_to_broader_scientific_literature": "This paper builds on the research trajectory of GPTSwarm, MacNet, and AgentPrune, which use DAGs to model multi-agent systems, and is highly relevant to the current line of research on (automated) LLM-based multi-agent systems.", "essential_references_not_discussed": "I recommend that the authors include the following recent multi-agent papers. Incorporating these references would further emphasize the novelty of their contribution:\n\n[1] AFlow: Automating Agentic Workflow Generation, ICLR 2025\n\n[2] AgentSquare: Automatic LLM Agent Search in Modular Design Space, ICLR 2025\n\n[3] Automated Design of Agentic Systems, ICLR 2025\n\n[4] Flow: Modularized Agentic Workflow Automation, ICLR 2025\n\n[5] Talk Structurally, Act Hierarchically: A Collaborative Framework for LLM Multi-Agent Systems\n\n[6] Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence", "other_strengths_and_weaknesses": "Strength:\n\n - The motivation and methodology presented in this paper are clear and accessible. Figure 3 is particularly well-designed and illustrative.\n\n - The proposed method is well-structured and methodologically rigorous, ensuring robustness, adaptability, and high performance.\n\nWeakness:\n\n - The experiments were conducted only on GPT-4 and GPT-3.5, which are no longer state-of-the-art models. Are the authors planning to test more advanced or open-source LLMs?\n\n - The system prompt attack described in Section 5.3 requires more detailed elaboration. Moreover, there are more advanced agent attack methods available, such as those in [1]. Do the authors plan to evaluate G-Designer’s defense capabilities against a broader range of attacks?\n\n[1] Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents", "other_comments_or_suggestions": "No specific comments.", "questions_for_authors": "- Do the authors plan to test more advanced or open-source LLMs?\n\n - In Section 5.3, do the authors intend to incorporate more advanced agent attack methods?\n\n - Are the authors considering using more agent-specific benchmarks to evaluate G-Designer?\n\n - G-Designer employs a chain-based anchor graph as its starting point. Do the authors plan to explore alternative anchor graph initialization?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pi0vhrNnc5", "reviewer": "Reviewer_e1AK", "summary": "Recent research has shown that large language model (LLM)-based multi-agent systems outperform traditional single-agent methods. This paper focuses on the challenge of choosing effective communication topologies for multi-agent systems. It puts forward the LLM-based Multi-agent Communication Protocol (MACP), which serves as a guiding principle for topology design, and presents G-Designer, an adaptable solution for multi-agent deployment.\nG-Designer represents the multi-agent system as a graph. It uses a variational graph auto-encoder to encode agents, along with a task-specific virtual node, and then decodes a communication topology that is tailored to the task at hand. The study finds that the performance of different topologies varies with task complexity.\nExperimental evaluations across six benchmarks verify that G-Designer exhibits high performance, task adaptability, and adversarial robustness. In conclusion, G-Designer offers a practical and efficient approach for multi-agent system deployment, paving the way for further research in collective intelligence systems. ## update after rebuttal", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your careful comments and thorough understanding of our paper! Here we give point-by-point responses to your comments and describe the revisions we made to address them.\n\n---\n> **`Weakness 1`**: Would switching between different agent roles have an impact on the experimental results?\n\nThank you for the insightful comment! To address your concerns, we analyze the transferability of G-Designer to unseen agent roles as follows:  \n\n*Table A. Generalization of G-Designer to unseen agent roles. We train G-Designer on the \"dataset for optimization\" and evaluate it on the \"dataset for test,\" where the test set may include agent roles unseen during training.*  \n\n| Dataset for Optimization | Roles | Dataset for Test | (Unseen) Roles | Performance |  \n|--------------------------|-------|------------------|----------------|-------------|  \n| GSM8K | - | GSM8K | - | 95.07 |  \n| HumanEval | Project Manager, Algorithm Designer, etc. | GSM8K | Math Solver, Mathematical Analyst, etc. | 93.98 |  \n| HumanEval | - | HumanEval | - | 89.90 |  \n| MMLU | Knowledgeable Expert, Mathematician, etc. | HumanEval | Project Manager, Algorithm Designer, etc. | 88.72 |  \n\nThe results indicate that G-Designer demonstrates strong transferability across datasets, effectively adapting from HumanEval to GSM8K and from MMLU to HumanEval, among others.\n\n---\n> **`Weakness 2`**: Typos\n\nThank you very much for your careful review of our manuscript. We have carefully addressed the typos you pointed out and made the necessary corrections throughout the document. Your feedback has been very helpful in improving the clarity and accuracy of the paper.\n\nWe appreciate your attention to detail and your valuable contribution to the review process.", "claims_and_evidence": "This paper mainly puts forward the following ideas. All of these claims seem to be adequately supported:\n\n1. It posits that LLM-based multi-agent systems can surpass traditional single-agent approaches. Moreover, the topologies of multi-agent systems can be optimized to yield superior outcomes across a variety of tasks. This assertion is substantiated by prior research and the experiments conducted in this study.\n2. The paper regulates multi-agent topology design across three dimensions: effectiveness, complexity-adaptiveness, and adversarial robustness. These logics are validated through in-depth analysis, demonstrating their effectiveness in optimizing communication structures.\n3. The study introduces G-Designer, a novel solution for multi-agent deployment. G-Designer models the multi-agent system as a graph and employs a variational graph auto-encoder to encode agents and a task-specific virtual node. The algorithmic details of G-Designer are clearly presented and supported by theoretical derivations and experimental results.", "methods_and_evaluation_criteria": "I believe that the methods and ideas proposed in this paper hold significant practical value for the advancement and application of multi-agent systems. The article introduces a novel multi-agent communication protocol based on large language models, termed the LLM-based Multi-agent Communication Protocol (MACP), which can serve as a guiding principle for the topological design of multi-agent systems. Furthermore, the paper presents an effective, adaptive, and robust LLM-driven multi-agent communication graph designer, G-Designer, to facilitate the automated design of collaborative AI systems. For me, G-Designer has the potential to inspire future research on the emergence of self-organizing and self-evolving collective intelligence.", "theoretical_claims": "I have reviewed the formulas and derivations in Section 4 and found no explicit flaws. Formula (9) employs the variational graph auto-encoder (VGAE) framework to generate the multi-agent interaction topology. Formula (10) presents the specific form of the encoder module within the VGAE, where the innovative introduction of a task-specific virtual node and agent nodes, along with the use of GNN to learn the representations of agent nodes, results in a reasonable representation of the agent collaboration graph.", "experimental_designs_or_analyses": "The experimental design of the paper is thorough and comprehensive. The authors conducted extensive experimental evaluations of G-Designer across six benchmarks, validating its high performance, task adaptability, and adversarial robustness in multi-agent systems. It is worth noting that the authors performed adversarial attack experiments to verify the robustness of G-Designer, which expands the application prospects of the method. This ensures that in scenarios where there are poisoned agents or less intelligent agents, G-Designer can detect them and maintain usability.", "relation_to_broader_scientific_literature": "I believe this paper can be primarily linked to the Agent Scaling Law mentioned in Macnet[1]. Macnet suggests that as the number of agents increases, the performance of multi-agent systems gradually improves, but this improvement is associated with the collaboration patterns among the agents. The G-Designer proposed in this paper can automatically design appropriate collaboration patterns based on the complexity of the task, thereby better adapting to the requirements of different tasks and enhancing the performance of multi-agent systems. Therefore, I think applying G-Designer to a larger number of agents could potentially better validate the effectiveness of the Agent Scaling Law.\n[1] Scaling Large Language Model-based Multi-Agent Collaboration ICLR'25", "essential_references_not_discussed": "To the best of my knowledge, all relevant related works that should be mentioned have been included.", "other_strengths_and_weaknesses": "1. The methodology is innovative. While the abstraction of multi-agent collaboration into graph structures has been mentioned in previous papers, this paper introduces VGAE on top of that. The combination of these two aligns with intuition and leverages the powerful graph representation capabilities of GNNs, making it highly novel.\n2. The paper is clearly articulated. Particularly, the illustrations and case studies are visually appealing.\n3. The training burden is relatively light. I observed that the experimental section not only discusses the effectiveness and efficiency during inference but also mentions the training overhead in `Table 2`. This is crucial for large-scale applications, as excessive training costs can limit the model's applicability. The overhead of this model appears to be lower compared to previous methods.\nHowever:\n1. The paper does not seem to provide specific examples of agent profiles. I am aware that the profile generation method in the paper follows the classical configurations in LLM-MA systems. However, I am curious about the impact of different qualities and contents of profiles on the results. Is the dynamic collaboration graph design sufficiently adaptable to different profiles?", "other_comments_or_suggestions": "Typos:\n1. In Table 1, the average accuracy of `MetaGPT` appears to be miscalculated.\n2. In Figure 3, 'Ouput solution' -> 'Output solution'.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work introduces G-Designer, an adaptive method that dynamically creates task-aware communication topologies using a variational graph auto-encoder. Through experiments, the authors show that the proposed method is promising in terms of performance, communication overhead, task-adaptive, and robustness.

The strengths of this work are summarized as follows.

1. It addresses an important problem of learning the communication network structure in a multi-agent system of LLMs. 
2. The proposed method, G-Designer, can dynamically design task-aware, customized multi-agent communication topologies.
3. G-Designer is the first to use graph neural network and  variational autoencoder for topology design in LLM-MAS.
4. G-Designer is promising in terms of accuracy/pass@1 and communication tokens. Moreover, G-Designer is robust in face of adversarial attack.
Decision: Accept (spotlight poster)