Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems
Authors: Guibin Zhang
Yanwei Yue
Zhixun Li
Sukwon Yun
Guancheng Wan
Kun Wang
Dawei Cheng
Jeffrey Xu Yu
Tianlong Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-agent collaboration
sparsification
LLM agents
Primary area: foundation or frontier models, including LLMs

Abstract: Recent advancements in large language model (LLM)-powered agents have shown that collective intelligence can significantly outperform individual capabilities, largely attributed to the meticulously designed inter-agent communication topologies. Though impressive in performance, existing multi-agent pipelines inherently introduce substantial token overhead, as well as increased economic costs, which pose challenges for their large-scale deployments. In response to this challenge, we propose an economical, simple, and robust multi-agent communication framework, termed $\texttt{AgentPrune}$, which can seamlessly integrate into mainstream multi-agent systems and prunes redundant or even malicious communication messages. Technically, $\texttt{AgentPrune}$ is the first to identify and formally define the $\textit{Communication Redundancy}$ issue present in current LLM-based multi-agent pipelines, and efficiently performs one-shot pruning on the spatial-temporal message-passing graph, yielding a token-economic and high-performing communication topology.
Extensive experiments across six benchmarks demonstrate that $\texttt{AgentPrune}$ $\textbf{(I)}$ achieves comparable results as state-of-the-art topologies at merely $\\$5.6$ cost compared to their $\\$43.7$, $\textbf{(II)}$ integrates seamlessly into existing multi-agent frameworks with $28.1\\%\sim72.8\\%\downarrow$ token reduction, and $\textbf{(III)}$ successfully defend against two types of agent-based adversarial attacks with $3.5\\%\sim10.8\\%\uparrow$ performance boost. The source code is available at \url{https://github.com/yanweiyue/AgentPrune}.

Published as a conference paper at ICLR 2025
CUT THE CRAP: AN ECONOMICAL COMMUNICATION PIPELINE FOR LLM-BASED MULTI-AGENT SYSTEMS
Guibin Zhang'1, 1, Yanwei Yue't, Zhixum Li Sukwon Yun?, Guancheng Wan",
Kun Wang Dawei Cheng 1.6 Jeffrey Xu Yu², Tianlong Chen"
TTongji University 2hhe Chinese University of Hong Kong
University of North Carolina at Chapel Hill *Wuhan University
"Nayyang Technological University Shanghai Al Laboratory
ABSTRACT
Recent advancements large language model that collective intelligence can significantly outperform individual capabilities, largely attributed to the meticulously designed inter-agent communication topolo- gies. Though impressive in performance, existing multi-agent pipelines inher- ently introduce substantial token overhead, as well as increased economic costs, which pose challenges for their large-scale deployments. In response this challenge, we propose economical, simple, and robust multi-agent commu- nication framework, termed AgentPrune, which can seamlessly integrate into mainstream multi-agent systems and prunes redundant or even malicious com- munication messages. Technically, AgentPrune is the first to identify and for- mally define the communication redundancy issue present in current LLM-based multi-agent pipelines, and efficiently performs one-shot pruning on the spatial- temporal pessaae-passsing graph. yielding token-economic and high-performing communication topology. Extensive experiments across six benchmarks demon- strate that AgentPrune (I) achieves comparable results as state-of-the-un topolo- gies at merely $5.6 cost compared to their $43.7, (II) integrates seamlessly into 72.8% token reduction, and
agents have shown
existing multi-agent frameworks with 28. (III) successfully defend against two types of agent-based adversarial attacks with 3.5% 10.8% 1 performance boost. The source code is available https: / ////tttthh//// com crom/yanweiyue /Agenntrrnnee.......rrrrrrr
INTRODUCTION
Task Task/Query
Solution
Large Language Model (LLM) based agents (Richards & et al., 2023; Naka- jima, 2023; Reworkd, 2023) have demon- strated strong performance across a di- verse range of tasks, including reasoning (Yao et al., 2023b), code generation (Shinn al., 2023), and even more complex ap- plications like video gaming (Wang et al., 2023) and autopilot systems (Jin et al., 2023). Recent endeavors have shown that combining implicitly or explicitly differ- ent LLLM-baed agents into team out- perform a single agent in handling com- plex tasks (Du et al., 2023b; Liang al., 2023; Wang et al., 2023e; Jiang et al., 2023: Shinn al., 2023; Zheng et al.. 2023; Wu et al., 2023), which supports the presence of -enue collaborative in- telligence in multi-agent systems (Zhang et al., 2023b). In practice, previous research has explored approaches in which instances of LLMs, referred to as agents (Wang et al., 2024b; Xi et al., 2023;
Alraatinn
Iee k+1
Iter
LLM-bused
Intra-dialogue
dialogue commmunication
Agents
communication
Figure The workflow of existing L..M- (problem- solving) multi-agent systems. The agent-agent communica- tion occurs at both intra- and inter-dialogue stages.
Kun Wang the corresponding author, denotes equal contributions.

---

Published as a conference paper at ICLR 2025
Vinailaa Chuil
Gritwans tpeePone
Fested on
Bucaatic Conmuniiatioo
gpt---iod-peesiew
artibvere Ca Catg token))
D-Si-Debatee Jr Seoo vileed)
WHIL
Token consunpttion inn Cio
Dilloge
DiIAN (oken))
Gesplex
(ittr dalloaa coneussicaiee)
przziig
Yaudss PIPP
LLM-2dak BylAN
AgustFraa:
Apentrrazeet Senuo
MAGU Jeunn
Jansken Gouph (4imitt
AgectP tolocs)
12.5
Chai Hag balonn)
Ravo
R50
54.5
(1)
Figure 2: (Left) The accuracy comparison on MMLU (Hendrycks et al., 2021) among (1) a single gpt-3. 1.tt-3.5turbb 5-turbe, (2) three aareegg---uubbbb.. 5-turbo agents equipped with intra-dialogue communication struc- tures like chain and tree (Qian et al., 2024), complete graph, and GPTSwarm (Zhuge al., 2024), and (3) those equipped with inter-dialogue communication structures like PHP (Zheng et al., 2023), L.MM-Dehate (Du et al., 2023b), DyLAN (Liu et al., 2023b) and our AgentPrune. (Middle) The prompt token consumption comparison on MMLU between different methods. (Right) The overview of our proposed AgentPrune.
Gao et al., 2023; Cheng et al., 2024; Ma et al., 2024), collaborate synergistically (e.g., through de- bate or reflection) to complete tasks (Du et al., 2023a; Pezeshkpour et al., 2024; Guo et al., 2024; Du et al., 2024; Han et al., 2024) via diverse communication topologies (e.g., chain (Wei et al., 2022), tree (Yao et al., 2023a), complete graph (Qian et al., 2024), random graph (Qian et al., 2024), optimizable graph (Zhuge et al., 2024), LLM-based network (Hao et al., 2023; Liu et al., 2023b)). The exceptional performance of these cooperative agents significantly benefits from their interac- tive communication and collaboration, specifically how agents transmit, exchange, and assimilate information (Chan et al., 2023; Wang et al., 2023c; Liu et al., 2023b).
Taking closer look into the communication mechanisms in existing multi-agent systems, they typically involve two key types (as shown in Figure I): ! Intra-dialogue communication: For a given query/task, multiple agents interact-whether by cooperating (Du et al., 2023a; Wu et al., 2023; Hong et al., 2024), teaching (Zhang et al., 2024d), or competing (Zhao et al., 2023; Fu et al., 2023)- produce solution within a dialogue round; @ Inter-dialogue communication: In specific manner-wwe manner-whether by summarizing (Chan et al., 2023; Shen et al., 2024), replicating (Yin et al., 2023: Du al., 2023b), or filtering (Liu al., 20232)22222-2000 the content of the current dialogue is passed to the next round of interaction reference, initiating new cycle of collaborative efforts.
To better illustrate the power of agent communication, Figure (Left) compares the performance of a single gpt-3. 5-turbo with three agents equipped with different inter/intra-dialogue commu- nication structures. The results demonstrate that even the simplest communication framework sig- nificantly leads to a notable accuracy improvement, which vividly showcases the social intelligence and collaborative capabilities of LLMs (Mei et al., 2024). However, the success of multi-agents comes at the cost of significantly increased token consumption, imposing substantial economic bur- dens (Wang et al., 2024a), which are detrimental to the widespread application of multi-agent sys- tems, deployment on edge devices does not accommodate excessively costly inference (Liu al., 2023a). A piece of empirical evidence is in Figure 2 (Middle), where various communication meth- ods result 2 11.8x increase in token consumption compared the simple chain structure, severely undermining the token economy of existing multi-agent systems.
In the light of this limitation, we for the first time identify significant phenomenon of Communica- tion Redundancy (Meyer et al., 2021) 1) within existing LLM-based multi-agent (LLM-MA) communi- cation topologies, where a substantial portion of message passing does not contribute meaningfully to the collaborative intelligence. With this finding, we introduce an economical and versatile com- munication pruning framework for L.M -.w-ered multi-agent systems, dubbed AgentPrune, which can be smoothly incorporated within various existing LLM-MA systems, offering comparable rea- soning and planning performance as well as significantly lower token consumption. Practically, AgentPrune treats the entire LLM-MA framework which each agent, along with its unique properties (e.g.. profile (Li et al., 2023a), external API tools (Zhuang et al., 2023), or knowledge base (Chen al., 2024a)), is packaged nication between agents within the same dialogue forms spatial edges, and communication across dialogues forms temporal edges. By training a www-rank-rrinciple-guiee graph mask, AgentPrune
spatial-temporal communication graph, in
node, commu-
2

---

Published as a conference paper at ICLR 2025
efficiently identifies the important graph connectivities (i.e., message passing through edges). This comes with a one-shot pruning to derive a sparse yet informative communication graph (in Figure (Right)), which is then fixed the communication topology for subsequent token-economic and efficient reasoning. Our contributions can be summarized as follows:
System Discovery. We present a spatial-temporal graph paradigm to describe the communication topology of contemporary LLM-MA frameworks, and further identify and define the Communi- cation Redundancy issue in current systems, wherein a significant portion of spatial and temporal edges, i.e., communication, does not contribute to collaborative intelligence.
Pratical Solution. We propose AgentPrune, an economical, simple, and robust multi-agent com- munication pruming pipeline. By leveraging a trainable communication graph mask, AgentPrune identifies key message exchanges and prunes non-essential components in a one-shot manner, re- sulting in sparse, token-economical, and highly informative communication graph. Notably, AgentPrune employs low-rank principle to guide the graph mask training, successfully robus- tifying LLM-MA systems against various agent-targeted adversarial attacks.
Experimental Validation. Extensive experiments on six henchmarks show that AgentPrune is: (1) high-performing. achieving comparable performance MMLU at $5.6 cost, that of state- of-the-art communication topologies at $43.7: (2)
integrating seamlessly into popular multi-agent frameworks including AutoGen and GPTSwarm reducing their token cost by 28.1% ~ 72.8% Ji and (3) adversarially robust, successfully defending against two types of agent adversarial attacks, with 3.5% 10.8% t performance improvement.
2 LLM-MA AS
GRAPHS
Notations We describe the whole multi-agent system graph (V,E), with {1I, U22 ildl } being the node set and & being the edge set. Each node E V represents an agent, which can further interpreted as follows:
(1) where an agent consists of: (1) Base;, the language model instance used by ey; (2) Role,, the pre-defined role or responsibility of the agent; (3) State,, the state of the agent, encapsulating the accumulated knowledge and experience from previous interactions; (4) P Pluginge a set of P external plugins available to agent fy where each plugin is defined by its functionalities F, (e.g.. web search, python compiler) and configurations Cj.
0; = [Bases, Role; State;, P Plugins:]:
(F):
For the edges, we divide them into two subsets: intra-dialogue (spatial) edges E C V(0) x (0) and
inter-dialogue (temporal) edges ET S y(t-1) x V(O. For each spatial edge eq = (Mij. O.y). ). represents the information flow from agent U/ to agent 1j within the same utterance, composed of the message content Mij, as well as the associated operation Oij (e.g., task assignments, requests). For each temporal edge e.y- it denotes the message passing between two ufterance rounds, ie., whether the output from agent i; in the (t - 1) round should be passed on to agent Uy in the f-th round. Further, we define the temporal/spatial (in-) )neighbors for each agent follows:
NT (vx) = {wy (i,1) E ET), ( = (w, (1, E E.).
Multi-agent Communication We provide a graph-based description of the reasoning process in task-oriented multi-agent systems. Given query/task q, is sequentially fed to each agent, which produces its output. To maintain an orderly sequence of agent interactions, we utilize topological ordering (Bondy et al., 1976) to ensure that each node is processed only after all its dependencies have been addressed. This necessitates that the spatial communication graph GS (V, 8 be structured as directed acyclic graph (DAG). Formally, for go, the following condition holds:
V(e, e(),,,)) I(wx) < I(ey) I(vy),
(3)
where 1(x) denotes the execution order of æ. For each agent of at round t, it produces its rationale or answers, uniformly denoted as M,, as follows:
temporal
spatial
M ~ Po M, q, Role!". State,", O O.jeNT M,. U Mj
(4)
where agent i; responds based on the query q. its current role and state, temporal and spatial mes- sages, and certain prompting instruction Pa. Typically, after K rounds of dialogue, summarizer agent or an answer aggregation mechanism (e.g., voting) is employed to produce the final solution (K) for the given query q. We conclude the general pipeline in Algorithm 1.
3

---

Published as a conference paper at ICLR 2025
Algorithm 1: Execution pipeline of LLM-MA systems from spatial-temporal graph perspective
Input:
Query Communication graph 9 = {g G"), Maximum number of iterations
for iteration & +- to N do
if Meet KndCondition then
break // Extra stopping criteria like agent consensus end
for in TopologicalSort()) do
m n <- (M, by E E (u:)) Messages from temporal in-neighbors
m [M,] o E (m:) (x))) // Measages from spatial in-neighbors
M Po q, Role, 19 State, (9) m² m²)// Generate rationale or answer
end
( AggregatesSluttionMM" M. MD) // Depending the specific syatem, possible implementations of AggregateSolution include (but are not limited to) majority voting or using the output of summarizer agent.
11 end
12 return (1) as the final solution
Problem Formulation In this section, we explore and define the communication redundancy issue within existing multi- agent communication pipelines. Specifi- cally, we examine two representative munication topologies: (1) for spatial communication, the fully-connected mesh graph from MacNet (Qian et al., 2024),
Tampoesi
b0MLa: Accurney (6)
MMLA Aucurnac
20-
68-
80
66-
which exemplifies densely structured intra-utterance communication, and (2) temporal communication, the LLM- Debate (Du al., 2023b), where the start of each dialogue round, an agent receives all responses from the previous round as input. Us- ing four gpt 5-turbo agents, assess system performance on MMLU after randomly pruning a certain proportion of connections. As illustrated Figure 3, when randomly removing 10% 30% of the communication connectivity, the performance actually gains up to 2.83% im- provement. This suggests that, in both spatial and temporal information flow, a substantial portion of messages does not contribute the task-solving process, which we formally define as follows:
10
500
Pruning Ratio (N)
Pruning Ratio (%)
Figure 3: The performance of mesh graph and LLM-Debate structure under different random pruning ratios on MMLU.
Definition (Communication Redundancy), For any LLM-based multi-agent communication graph G = (V,E$ UET), ), the following condition holds:
3 = (V,E' UE") CGG where E'
s. (( Z ((G),
(5)
where s(-) represents utility function that measures the solution quality achieved by the system.
The redundant components in the communication topology, denoted as (Est {E') U (E \ {E"), are referred to as the communication redundancy in LLM-MA systems.
We further outline the objective of this study as follows:
arg maxg 5.0 geub, s.t. lo(gubb) ((9) S,
(6)
where represents the allowable threshold for performance variation. Equation (6) aims to minimize communication redundancy with performance guarantee.
METHODOLOGY
Figure illustrates how our method applied within an LLM-MA system. Specifically, given an input query, AgentPrune first performs spatial pruning by eliminating redundant spatial messages within dialogue round, followed by temporal pruning to discard unnccessary dialogue history. In the following sections, will first explain how AgentPrune facilitates efficient multi-round communication based on an optimizable spatial-temporal communication graph (D> Sections 3.1 and 3.2), leverages one-shot pruning to derive sparse interaction topology (D> Section 3.3), and finally, detail the optimization paradigm for the entire framework (C> Section 3.4).
4

---

Published as a conference paper at ICLR 2025
{Question}
How many urithmetic sequences of consecutive odd integers sum to 240?
iterations
Original Communication Topology
After applying AgentPrune
Thinker
Thinker 2
Thinker 1
Thinker 2
Peperst
((Queation)
Sepert:
((uesstion)
IB
Deteens Theckess all
BI Ourpec: Beed growes
(Qunetion?
(Anoowe
1B Getperti Seppene fhr and moctir sogeneec cor Thrttme ahe reparee ax andi-u +1 Poorterr server
(Ouoeaiio))
Pitters: We
Thes relacs
Alke dlobkig per
Spatial pruning
mailer the 26 Thariii dhe
(Answerr)
(Asomcr
Duusnerr
yo9 calaaa
Thinker 3
Summarizer
Thinker
Summarizer
Jeper:
(Question)
ETI Inpoc: (Qucddaon)
(Question)
(Qucation)
(Anowwer
(Ansuurr
(nnwer
Arnmer s)
(Aavoserr
(Chnewerr)
filzbber 13
(Anoweer aj:
curpec:
panently speoves
ia occpar: nnnver serent
(Anowee
which rowands to 11
produn, theer
io Cukkig cank opatin
12.12 sopectrcly They
mil velle
U. repuctiily onnonn Enepellie
Aeveer crnnnn the tovarooti
un www.iiiiienee picaenis
cavcn prsilimnis
Cuastusion)
(auewer
Conelnion
Tempoonn arruniig
(Utterance history)
(Utterance history)
Aaawner s)
Aaxwer st
(Coneluiie))
(Condiaton))
Prompt for next round: Given the last round's 's utterance: (Utterance history), please
...
Intra-iteration token: 4332x22632)))
2,295 tokens
Intra-itrcation token: 37881,77-2-22009
3:425 tokens
Inter-iteration tokan: 1388---22--2---888))
Inter iteration token: (299-187)x<
Figure 4: The overview of our proposed AgentPrune.
3.1 SPATIAL-TEMPORAL GRAPH COMMUNICATION
Given arbitrary LLM-MA system and its corresponding spatial-temporal communication graph G. the task of AgentPrune is to discover its sparse yet equally high-performing counterpart gLub The objective is essentially a graph sparsification problem (Spielman & Srivastava, 2008; Chen et al., 2023b), whose goal is to identify the essential graph connections and discard the less critical ones. To achieve this, following classical practices in graph sparsification (Chen et al., 2021 Wang al., 2023a; Zhang et al., 2024a), we relax the original binary communication graph 9 by transforming its edge elements from binary values to continuous variables, denoted as G. We have:
A() = (AS, AT). A(¹) = A .TT)) = (A @ SS, ATOSST
where Ag) obtains the adjacency matrix of input graph 9, and A AT E 10. 1 represent the spatial and temporal adjacency matrices, respectively. Specifically, A"] = indicates that eij E 8", and 0 otherwise. It is important to note that both A and A are predefined by the LLM-MA system. $ S E | are differentiable graph masks. As mentioned in Section 2, require the interaction topology to be DAG ensure that agent input/output
(I/O) can be processed sequentially. Therefore, we leverage a DACSamp ling function to transform the original gs into a DAG: gs < DAGSampl (ğS), whose procedure is described in Algorithm 2.
3.2 OPTIMIZING SPATIAL-TEMPORT
CONNECTIVITY
With ÿs and GT in hand, we aim to optimize them toward both high-performance and token ef- ficiency. To this end, we introduce two optimization objectives for ğs and GT. O distribution approximation, ensuring accurate estimation of their underlying probability distributions, and e low-rank sparsity, which promotes a more efficient and sparse structure (Li et al., 2024). The first objective ensures that the magnitudes of the graph masks correctly reflect the importance of different communication channels, facilitating subsequent redundancy pruning, and the second ensures that the learned connectivity remains sparse and robust. Formally, we define the following objective: distribation approximation
low rank sparsity
arg max Egg.gr gT (0 (105.7)) $ STE
rank s.t. Σ |A
X
F
S 6,
(8)
xE(S,T)
xE(S,T)
5

---

Published as a conference paper at ICLR 2025
where S and G represent the viable parameter space, d(.) serves the utility evaluator for the input multi-agent framework, rank(.) calculates the rank of matrix, and is the noise level. Next, we will provide detailed explanation of the implementation of these two optimization objectives.
Distribution Approximation The first term in Equation (8) encourages {S*, (TS) towards the maximization of the system's utility. However, since d()) often depends on external APIs (Li al., 2023b) or compilers (Chen et al., 2021a) for evaluation, it is generally non-differentiable. Therefore, we employ policy gradient (Williams, 1992) to make Equation (8) tractable:
M
Vs Eg or 1o (0(119.97)
1
Σ
Φ (192 (T)) Vs log (ps(1 1)),
(9)
M
ps ( (
9h
(II Jessess S (t.il)
S" (T(.))))
(10)
where = (s$, S , are independently sampled from ( ( ps(1 9 calculates the probability of the sampled structure, and 1(-) is an indicator function.
Low-rank Sparsity The second term in Equation (8) promotes the graph masks { ST) to be low-rank, which not only filters out informative agent communications but also aids in removing redundant, noisy, and even malicious messages, which has been demonstrated in recent studies, showing that low-rank graphs are more robust to network attacks (Entezari et al., 2020; Ennadir al., 2024). We will empirically validate AgentPrune's 's ability to enhance multi-agent robustness in Section 4.4. However, directly optimizing the rank minimization is NP-hard, so we replace the rank function with the nuclear norm as alternative, reformulating this term as follows:
arg min Exe s $5,STES ES
| ||,, s. || S |F
(11)
where - oi and o, represents the i-th singular value of S. Guided by Equation (8), iteratively optimize the spatial-temporal connectivity in conjunction with the multi-agent conversa- tion over K' rounds, where K" < K.
3.3 ONE-SHOT PRUNING
We dynamically optimize {S$, } for only K' iterations, rather than the full K iterations, because prior work on Barly-bird (EB) and Graph EB has demonstrated that limited training can also con- struct high-quality benchmarks reflecting the topology distribution (Achille et al., 2018; You et al., 2019; Zhang et al., 2024b), which also aligns with AgentPrune's token-saving initiation. To elim- inate redundancy in the current communication structure, we perform one-shot magnitude pruning on the optimized graph masks S (either S or S)):
+ 0 A Topk (s. |A x (1 (1-p%) )):
B A
(12)
where TopK(S, return the largest 2% elements in matrix S, and p% is the pruming ratio. By applying the binary masks to the original topology, we obtain sparse, compact, and communication- minimizing connectivity goub, where A (gab) = (A S BB, A @T J. In the subsequent (K - KK") rounds, the entire framework's message passing pipeline is strictly constrained by guub, and agents are continuously optimized to refine the solution for query
3.4 APPLICATION AND ANALYSIS
Algorithm Pipeline As a plug-and and---dd-pay module, AgentPrune can be harmoniously embedded in mainstream multi-agent frameworks to facilitate token-efficiemt communication, provided that the number of agents exceeds three and the communication structure is moderately organized (e.g., chain or direct-output structures are too simple to be applicable). When combined with AgentPrune, mul- tiple agents first undergo K' rounds of interactions alongside trainable graph masks, which are then one-shot pruned to yield the sparse Gruh, leveraged for the subsequent (K K') rounds of opti- mization. We summarize all the notations used in Appendix B and the comprchensive algorithmic workflow in Appendix C.
Multi-Query Training For complex tasks like repository-Level code generation (Qian et al., 2023; Liu et al., 2024), multi-turn dialogues often inevitable. However, for simpler tasks that involve

---

Published as a conference paper at ICLR 2025
Table 1: Performance comparison with three types of baselines, including single-agent execution, spatial com- munication and temporal communication. The best results are highlighted in bold, and runner-ups are underlined. All methods, except for the single-agent category, utilize five gpt - based agents.
Method Spa.
Tem.
MMLU
GSM8K
MultiArith
SVAMP
AQuA
HumanEval
Avg.
Vanilla
82,14
85,40
93.15
87.18
70.34
71.68
81,65
CoT
x
x
82.65
87.17
94.79
88.32 11.4
73.91-3.57
75.52
83.73
ComplexCoT
X
X
83.78
82.62
95.86-2.71
90.1772.99
77.58
74.24
84,99
SC (CoT)
X
X
82.66
87.93
96.88-a.
88.69
75.08
77.30
84.67
SC (ComplexCoT)
X
83.65
36.14
96.94-a.79
82.72
77.69
77.94
85,35
Chain
82.35
85.57
94.38 1,23
83.41 13.77
70,94
80.88
82.92
Star
80.79 11.35
85.55
85.55
10.16
93.79.
88.09 10.91
68.57 .1.77
75.65 (3.97
82.07
Tree
81.89 1o:25
84.56
10.84
94.60-1.46
89.25
72.84
77.38
83.42
Complete Graph
83.15
86.49
97.20-4.u6
89.48
79.2
19.2118..77
83.75/12.07
86.55
Laycred Graph
/
78.41 13:73
85.3410.06
95.04-1.8 1.89
88.61
73.18 81.61
11.43
-2.88
80.38 18.70
83.49
Random Graph
/
X
83.76 1.66
86.1410.74
95.46-2.8
85.41 p1.77
74.07
82.66 18.98
84.58
LLM-Blender
/
81.22 (0.02
89.17 19.77
94.27-1.12
88.77
77.05m6.71
86.10
GPTSwarm
/
83,98
89.74
97.84-4.69
86.42
78.16-7.82
88.49
86.77
L.M-Debate
X
83.69 1..5
90.23 14.83
96.27-3.12
90.56
77.52-7.18
83.79 112.11
87.01
PHP
X
83.45
92.45 f7.06
96.41-3.26
90.62 y3.44
76.25-5.91
82.96 18
87.02
DyLANN
X
80. 11.08
88.16
94.27-1.12
87.40
74. 74.16-3.92
89.70
84.48
AgentPrune-C
84.72 12.58
95.62
97.25-4. -.10
91.85
|4.67
79.47-9.1
89.38
1338815.707 70
89.72
AgentPrune-L.
83.50 1:38
93.78
18.38
96.39
3324
89.58
12.40
78.44.8.u 8.110
88.61
116 0..0
88.38
AgentPrune-R
83.94
95.83-10.43
96.30-a.15
91.68
14.00
78.60
90.30;
+18.02
89.44
large number of queries, such as multiple choice answering (Agashe et al., 2023; Qian et al., 2024), typically only one two dialogue rounds are needed, according to previous practices (Yin et al., 2023). Under such circumstances, optimizing the connectivity for each query independently can be unnecessarily costly. Therefore, we give multi-query training paradigm for AgentPrune, which optimizes and prunes the spatial-temporal topology using merely Q' < Q) queries, given a dataset composed of Q queries. See details in Appendix D.
Cost Analysis In this section, we quantify the difference in token consumption between AgentPrune and the vanilla pipeline. Given communication graph G and K dialogue rounds, suming that the average token count per spatia//tempora//query message is cs, CT, Cg. respectively, then the total token consumption of the vanilla system is Cg = K [cs|E*| + -T| CV] The token consumption after applying AgentPrune is divided into two stages. The first stage involves MK' |c|| +
cr cT! C. [ while in the second stage, after the topology is fixed, the con-
sumption becomes (K KK) [1 p%) (cs|E$1 cT( + G,D1I. Therefore, the total token savings A achieved by AgentPrune can be expressed as:
A A= (1 +%) K (M p%) K')
css (cs|E5 | ( +)(+))+++11- (on-MM)'G,,11
(13)
We present the cost analysis of AgentPrune in multi-query training in Appendix E. We will empir- ically evaluate the substantial token savings gained by AgentPrune in Section 4.2.
4 EXPERIMENTS
In this section, we conduct extensive experiments to answer the following research questions: (RQ1) How does AgentPrune perform with respect to task completion and token efficiency? (RQ2) Can AgentPrune reduce the economical cost of existing multi-agent systems without compromising performance? (RQ3) Is AgentPrune effective defending against adversarial attacks on agents? (RQ4) How sensitive is AgentPrune its key components parameters?
4.1 EXPERIMENTAL SETUP
Tasks and Benchmarks our experiments, we test the performance of AgentPrune on three types of reasoning tasks and the corresponding logically challenging benchmarks: (1) General Reasoning: We opt for MMLU (Hendryeks al., 2021) dataset; (2) Mathematical Reasoning: We select GSM8K (Cobbe et al., 2021), Multi Arith (Roy & Roth, 2016), SVAMP (Patel et al., 2021) and AQuA (Ling et al., 2017) to verify the mathematical reasoning capacity; (3) Code Generation: We use the HymanEval (Chen et al., 202 la) to test the function-level code generation ability.
7

---

Published as a conference paper at ICLR 2025
MMLU
HumanEval
GSMBK
Tolee comse mep.eummooooomommmmm (100">)
1000
Dn cell dime))
Tokoe coo.tmmpeeen
INEAN
0:30
1un
Accuracy
Passgin
Accuracy
5: Visualization of performance and prompt token consumption. This scatter plot illustrates the per- formance metries and prompt token consumption of different multi agent communication topologies across MMLU, HumanEval, and GSMSK. The diameter of each point is proportional to its y-axis value.
Baselines We compare AgentPrune with three series of multi-agent communication paradigms, namely: (1) Single agent execution methods, including Chain-of-Thouggh prompting (CoT; Wei prompting (ComplexCoT: Fu et al. (2022)), and (3) Self- Consistency (SC; Wang et al. (2023b)); (2) Spatial communication methods, including chain, tree, star, complete graph, layered graph and random graph' from MacNet (Qian et al., 2024), LLM- Blender (Jiang et al.. 2023), and GPTSwarm (Zhuge et al., 2024); (3) Temporal communication methods, including PHP (Zheng et al., 2023), LLM-Debate (Du et al., 2023b), DyLAN (Liu et al,, 2023b). Detailed introductions and implementations of the baselines are in Appendix G.I.
et al. (2022)), (2) Complexity-base
Implementation Details We accessed the GPT models via the OpenAl API, and mainly tested on - 1...3.55-.0 5-turbo-0301 (gpt-3. 5) and gpt-4-1106-previce (gpt-4). We set the temperature at 1 during the generation. We set the dialogue round K = 2 for mathematical and general reasoning tasks, and K = for code generation tasks. For multi-query settings, we vary Q' E {5, 10, 20) and fix M 10. We generate differrntt agent profiles using gpt-4. The pruning ratio is chosen among (00%, 30%}. More experimental details are in Appendix G.2.
4.2 PERFORMANCE & COST COMPARISON (RQ1)
To evaluate whether AgentPrune achieves a dual benefit of token savings and task completion, we integrate it with three predefined spatial commu- nication topologies: the complete graph, layered graph, and random graph, denoted AgentPrune- C, AgentPrune-L, and AgentPrune-R, respec- tively. For the temporal communication topology, we consistently employ the fully connected LLM- Debate-style structure. Tables 1 and presents a performance comparison of various communica- multi-agent
Table 2: Performance the Hvaa Benckmark with more baselines. The best and runner-up results are bolded and underlined, re- spectively.
Method
Pass
Vanilla
71.68
AutoGen [2023]
85.41
111.97
Reflexion [2023]
91.40
119.72
CodeT+ [2023a]
85.10
113.42
MetaGPT [2023]
85.90
114.22
tion paradigms within 9pt-4-batt systems, and Figures 5 and 17 to 19 visualizes the performance and token cost of different methods. Our observations (Obs.) are as follows: Obs. Not all multi-agent topologies consistently deliver collective intelligence. As illustrated in Table 1, certain topologies, such star/tree structures, fail to consistently improve performance for multi- agent systems, even resulting in performance drops of 0.17% ~ 3.97%. In contrast, single-agent prompting methods like CoT or ComplexCoT demonstrate much more stable and significant im- provements. Obs. The high performance of existing multi-agent systems comes at a substan- tial economical cost. From Table 1, we observe that the tor-perfogiing baselines, GPTSwarm and DyLAN, achieve pass @ / improvements of 16.81% and 18.02% on HumanEval, respectively; how- ever, this is accompanied by extremely high economic costs. As shown in Figure 5, the prompt token consumption of GPTSwarm and DyLAN is 2.4 ~ 5.3x that of the random graph structure. Obs.0 AgentPrune achieves double win in economic savings and utility. Among the three variants, AgentPrune-R R delivers consistently impressive performance, achieving 90.3% on HumanEval and 95.8% on GSM8K. Importantly, this performance does not come at a high token cost: on both Hu- manEval and GSM8K. the token consumption of AgentPrune is less than 40% that of DyLAN. Overall, AgentPrune excels in both task completion and token efficiency.
ANPL [2024]
86.60
114.92
AgentPrune-C
89.38
117.70
AgentPrune-R
90.30
118.2
Detailed explanations of these topologies are placed in Appendix F.
8

---

Published as a conference paper at ICLR 2025
Table 3: Performance and cost comparison before/after combining AgentPrune. We evaluated the per- formance and economical cost AgentPrune in conjunction with two classic multi- systems, under a five gpt-4-base setting. "#f Prompt tokens" refers to the total number of tokens input, while "# Completion tokens" accounts for the total number of tokens output by the API.
Dataset
Method
Performance
# Prompt Tokens
# Completion Tokens
Cost (USD)
AutoGen
82.13
486, 034
89, 224
$7.537
MMLU
+AgentPrune
82.78(1 0.65)
349, 349.583(1 583(71.99)
86, 582
$6.093 (80.8%
AutoGen
85.41
492,
130, 196
$8. 828
HumanFval
+ AgentPrunee
86.65(1 1.24)
315, 105(64.0%)
139. 714
(73342(83..) 1%)
AutoGen
90.06
327, 740
998, 042
$73.21 21
GSM8K
AggentPrune
92.85 2.79)
3, 791, (59.9%)
1, 156, 884
359.60(81.499)
GPTSwarm
83.98
3, 055, 230
569, 124
847 .60
MMLU
AAgentPrune
83 33.05(L 0.93)
990, (3..312222444))
430, 551
$23.05(48 (23.05(48.4%)
GPTSwarm
88.49
2, 736, 136
1, 004, 616
$37 .9
HumanFval
AAgentPrune
88 (6 0,47)
745, 617 (27.2))
745, 926
$29.80(51 (8.880((18)))
GPTSwarm
89.74
14, 005,
3, 156, 916
$234.76
GSM8K
+ AggentPrume
90.58(1 0.84)
3, 526, (335(39.49)
730, 552
857.17(24.3%)
95
90
before attack
after attack
85
847833
81.08u2
84.0g.66
83.089.2
8o.5
Bo.7
80-
76.2
75
70
66
61.5
60
Chain Chain-hike
Star-like
Tree-like
Complete Graph
Randum Graph
AgeatProne--
AguntPrune-R
DyLAN
GPTSwart
GPTSwarm w/ Antocion
AutoGon w/ AP
Figure Performance under adversarial attack. We compare the accuracy (%) of various multi-agent frameworks before and after prompt attacks on MMLU, "w/ AP" indicates the integration with AgentPrune.
4.3 PLUG-IN INTO EXISTING FRAMEWORKS (RQ2)
As plug-in, AgentPrune can be seamlessly combined with mainstream multi-agent pipelines, ef- fectively reducing the economic costs associated with LLM token throughput while maintaining the original performance levels. To validate our argument, we combined AgentPrune with two repre- sentative LLM-MA frameworks, AutoGen and GPTSwarm. With the results presented in Table and Table 5, we offer the following two key observations: Obs. Scaling multi-agent collabora- tion is costly. Comparing Table and Table 5, we observe that for the GPTSwarm on the GSM8K dataset, optimizing three-agent system incurs a cost of $97 29,22, while the expense for a five-agent system skyrockets to $234.76, with the total token count reaching 1.7c + 7. AutoGen, on the other hand, has relatively lower costs because it does not involve the iterative optimization of the commu- nication topology as extensively as GPTSwarm (Zhuge al., 2024). Nevertheless, still requires $73.21 on the GSM8K benchmark, which comprises up 8.5K data entries. Obs.6 Agent Prune is an economically friendly assistant. When applied to vaumanEval+ a 36% reduction in prompt tokens and saves $1.486. In tasks with larger datasets, the economic sav- ings become even more pronounced: on GSMSK-GGTSSaamm, AgentPrune reduces 60.6% of the prompt token consumptions and saves cost of up $177.58, with even performance increase of 0.84%. Overall, AgentPrune serves as a token-efficient plug-in, effectively fostering the develop- ment of larger and more cost-effective multi-agent systems.
AgentPrune achieves
4.4 ROBUSTNESS VERIFICATION (RQ3)
AgentPrune can not only eliminate unnecessary communications but also remove malicious mes- sages. To validate this, we design two types of adversarial attacks for the multi-agent frameworks: the agent prompt attack and agent replacement attack. The former attacks the role prompts of the agents, while the latter attacks the LLM's generation process, with detailed implementation elaborated in Appendix G.4. observe that: Obs. Existing LLM-MA frameworks often lack adversarial robustness. Despite variations in performance, most frameworks experience significant declines when subjected both types of attacks. As shown in Figures 6 and 20, the chain- like struc- ture suffers performance drop of up to 20.8% due to its oversimplistie topology. AutoGen and Dy- LAN similarly experience accuracy declines ranging from 3.2% to 6.2%. Obs.0 AgentPrune sig- nificantly enhances multi-agent robustness. Figure demonstrates that combining AgentPrune

---

Published as a conference paper at ICLR 2025
with complete graph not only improves performance (83.1%- - 84.7%), but also increases robust- ness under agent prompt attacks (78.4% + 83.9%). Additionally, AgentPrune successfully boosts the robustness of DyLAN and AutoGen by up to 6.3%. The impact of AgentPrune on GPTSwarm is relatively marginal, due to its inherent defenses against adversarial agents. Overall, AgentPrune serves an easy-to- enhancer for multi-agent robustness.
4.5
EXPERIMENTAL ANALYSIS
Ablation Study We ablate agent profiling and low-rank regularization in AgentPrune, with de- tails presented in Table 7 and Appendix H.4.1. Our key finding is that (1) the utility of agent profiling varies across different datasets, demonstrating a more pronounced effect on general reasoning and code generation tasks, while being relatively less significant in math reasoning; (2) low-rank sparsity consistently facilitates the optimization of the communication topology.
Sensitivity Analysis and Case Study We present the parameter sensitivity analysis concerning three hyper- peeer-arameters, [V], Q' and p% in Appendix H.4.2, and provide extensive visualizations on AgentPrune 's pruning process and optimized communication structure in Appendix I.
5 RELATED WORK
Collaboration between multiple LLM-based agents has emerged as a promising approach to enhance the capabilities of individual LLMs (Du et al., 2023b; Liang et al., 2023; Wang et al., 2023c; Zhang et al., 2024c; Niu et al., 2025). As stated in Section 1, current multi-agent communication methods can be categorized into two types: O Intra-dialogue (spatial) communication focuses on how different agents exchange messages within a single dialogue round. Common structures include (1) Direct outpur, where functioning agents do not communicate with each other, adopted by systems like LATM (Zhang et al., 2023a), LLM--ebate (Du et al., 2023b); (2) Chain, employed by ChatDev (Qian et al., 2023), MetaGPT (Hong et al., 2023) and L2MAC (Holt et al., 2024); (3) Tree, where an administrative agent (usually referred to as commander, manager, efc.) controls subordinate agents, adopted by AutoGen (Wu et al., 2023), SecurityBot (Yan et al., 2024), and MiniGrid (Zhou et al., 2023); and (4) Graph, employed by LLM-Blender (Jiang et al., 2023), ChatEval (Chan al., 2023), MacNet (Qian et al., 2024) and GPTSwarm (Zhuge al., 2024); @ Inter-dialogue (temporal) communication focuses on how information is passed between dif- ferent rounds of utterances. Common topologies include (1) Full transmission, where every agent receives the utterances of all agents from the previous round, used by LLM-Debate (Du et al., 2023b); (2) Partial transmission, where some responses are filtered through scoring or rating mech- anisms, adopted by PHP (Zheng et al., 2023) and DyLAN (Liu et al., 2023b); (3) Summarization, where dialogue history is compressed and summarized for the next round of communication, as seen in Reflexion (Shinn et al., 2023), ICL-AIF (Fu et al., 2023), Agent Verse (Chen et al., 2023a), CoMM (Chen et al., 2024b), Corex (Sun et al., 2023), and MAD (Liang et al., 2023).
LLM-agent Collaboration
Agents as Graphs Learning to facilitate communication via learning graph connectivity is a long- standing and viable approach to enhance multi-agent cooperation (Pesce & Montana, 2023; Hu al., 2024). In the pre-L.LM era, numerous efforts explored optimal communication graph struc- tures for reinforcement learning-basee based multi-agents with graph diffusion (Pesce & Montana, 2023), weighted GNN (Liu al., 2022), or transformers (Hu et al., 2024). In the emerging wave of LLM- powered agents, attempts that leverage graphs for modeling agent-agent interaction also exist: Chat- Eval (Chan et al., 2023) and AutoGen (Wu al., 2023) implicitly adopt graph structures to de- scribe "simultaneous talk", and STOP (Zelikman et al., 2023b) and DSPy (Khattab et al., 2023) optimize both the prompts and the inference structure together. MacNet (Qian et al., 2024) and GPTSwarm (Zhuge et al., 2024) model agent communication via directed acyelic graphs (DAG). However, none of these approaches simultancously optimize both intra- and inter-dialogue commu- nication structures, and they often result in even increased token consumption.
6 CONCLUSION
This paper makes the first attempt towards a nigh-performance and token-efficient L.LM-powert multi-agent system. We propose an economical, simple, and robust multi-agent communication pipeline, termed AgentPrune, which can be harmoniously embedded into mainstream multi-agent frameworks while effectively pruning the communication redundancy that we have identified and defined. AgentPrune achieves performance comparable to, or even superior to, the original systems with significantly smaller token throughput and economic costs. We believe that AgentPrune can facilitate the advancement toward larger-scale collective intelligence.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In International Conference on Learning Representations, 2018.
Saaket Agashe, Yue Fan, and Xin Eric Wang. Evaluating multi-agen coordination abilities in large language models. arXiv preprint arXiv:2310.33903, 2023.
John Adrian Bondy, Uppaluri Siva Ramachandra Murty, et al. Graph theory with applications, volume 290. Macmillan London, 1976.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Evaluators through Multi-Agent Debate.
Zhiyuan Liu. ChatEval: Towards Better arXiv e-prints, art. arXiv:2308.0777 07200,, August 2023.
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmennedd generation. In Proceedings the volume 38, pp. 17754-17766 17762, 2024a.
the AAAI Conference on Artificial Intelligence,
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, July 01, 2021 2021a. corrected typos, added references, added authors, added acknowledgements.
Pei Chen, Boran Han, and Shuai Zhang. Comm: Collaborative multi-agent, multi-reassoing--aae prompting for complex problem solving. arXiv preprint arXiv:2404.1777 17729, 2024b.
Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. In International conference on machine learning, pp- 1695-1706. PMLR, 2021b.
Weize Chen, Yusheng Su. Jingwei Zuo, Cheng Yang. Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facili- tating multi-agent collaboration and exploring emergent behaviors in agents, 2023a.
Yuhan Chen, Haojie Ye, Sanketh Vedula, Alex Bronstein, Ronald Dreslinski, Trevor Mudge, and Nishil Talati. Demystifying graph sparsification algorithms in graph properties preservation. Pro- ceedings of the VLDB Endowment, 17(3):2274440, 2023b.
Yuheng Cheng, Ceyao Zhang. Zhengwen Zhang, Xiangrui Meng, Sirui Hong, Wenhao Li, Zihao Wang, Zekai Wang, Feng Yin. Junhua Zhao, and Xiuqiang He. Exploring large language model based intelligent agents: Definitions, methods, and prospects. CoRR, abs/2401. 003428, 2024. URL https://arxiy:: org/ abss//////e///// / albs//21111 24011 03428.
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv prepring, Abs/2110.14163
2021. URL httph:tt///ett
org/ abs /2100 14168.
Hung Du Srikanth Thudumu, Rajesh Vasa, and Kon Mouzakis. A survey context-aware multi- agent systems: Techniques, challenges and future directions. CoRR, abs/2402.01168, 2024. URL
https: htt//v.ivvv..////.///.////..avvv wwtoo/////aao.
http:///wwwwwww..//a//ass/e/s/// 2402. 01968.
Yali Du, Joel Z. Leibo, Usman Islam, Richard Willis, and Peter Sunehag. A review of cooperation in multi-agent learning. CoRR, abs/2312.05162, 2023a. doi: 10.4655/AARXIV2331..5112.. URL https://dee wwweiioogr/1..0 48550/ www.v w.w... 2312..011111

---

Published as a conference paper at ICLR 2025
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. CoRR, abs/2305 14325,, 2023b. doi: 10.48550/v2Xxv.2355544333 URL /ttpp/////....or....885550/////0 14325.
Sofiane Ennadir, Yassine Abbahaddou, Johannes Lutzeyer, Michalis Vazirgiannis, and Henrik Boström. A simple and yet fairly effective defense for graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 21063-21071, 2024.
Negin Entezari, Saba A Al-Sayouri, Amirali Darvishzadeh, and Evangelos E Papalexakis. All you need is low (rank) defending against adversarial attacks on graphs. In Proceedings of the 13th international conference on web search and data mining, pp. 169-177, 2020.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-basse prompting for multi-step reasoning. The Eleventh International Conference on Learning Representations, 2022.
Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with self-play and in-context learning from feedback, May 01, 2023 2023. Preprint. Code at https://github.com//// www.wuan.ooo///aaaaccaa Baw.inxXao///PPPPaoo PaaooGGG--aaggiingg.....
Chen Gao, Xiaochong Lan, Nian Li, Yuan Yuan, Jingtao Ding, Zhilun Zhou, Fengli Xu, and Yong Li. Large language models empowered agent-based modeling and simulation: survey and per- spectives. CoRR, abs/2312.1197// 2023. URL https: arxiv. wwwp/a/g/////eoeeaeee//o/eaaeeee http://p//a/////..111119../
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. CoRR, abs/2402.016800 2024. URL https: //t:///axxe arxiv. org/ abs/ / 222.. 01 680.
Shanshan Han, Qifan Zhang, Yuhang Yao, Weizhao Jin, Zhaozhuo Xu, and Chaoyang He. Llm multi-agent systems: Challenges and open problems. CoRR, abs/2402.03578, 2024. URL
attpss//ae /ttp:s/////vvvv/a//ae orgg ats//bb//22 2402 03578.
Rui Hao, Linmei Hu, Weijian Qi, Qingliu Wu, Yirii Zhang, and Liqiang Nie. Chatllm network: More brains, more intelligence, April 01, 2023 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICIR), 2021.
Samuel Holt, Max Ruiz Luyten, and Mihaela van der Schaar. L2mac: Large language model au- tomatic computer for extensive code generation. In The Twelfth International Conference Learning Representations, 2024.
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, and Chenglin Wu. Metagpt: Meta programming for multi-agent collaborative framework, August 01, 2023 2023.
Sirui Hong, Yizhang Lin, Bang Liu, Banghang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, et al. Data interpreter: An Ilm agent for data science. arXiv preprint arXiv:2402. 18679, 2024.
Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Learning multi-agent communication from graph modeling perspective. arXiv preprint arXiv:2405.0 085500 2024.
Di Huang, Ziyuan Nan, Xing Hu, Pengwei Jin, Shaohui Peng, Yuanbo Wen, Rui Zhang, Zidong Du, Qi Guo, Yewen Pu, et al. Ampl: towards natural programming with interactive decomposition. Advances in Neural Information Processing Systems, 36, 2024.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-biender: els with pairwise ranking and generative fusion. In Proceedings of the 61st Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14165- 14178, Toronto, Canada, July 2023, Association for Computational Linguistics. URL https: //aclanthor ogy. orgg.org//023 acl a long. 792.
Ensembling large language mod-
12

---

Published as a conference paper at ICLR 2025
Ye Jin, Xiaoxi Shen, Huiling Peng, Xiaoan Liu, Jingli Qin, Jiayang Li, Jintao Xie, Peizhong Gao, Guyue Zhou, and Jiangtao Gong. Surrealdriver: Designing generative driver agent simulation framework in urban contexts based on large language model, 2023.
Omar Khattab, Arnv Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.0371 2023.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: communicative agents for "mind" exploration of large language model society. In NeurlPS, 2023a. URL http://papers: nips cc cappapprrppppecpeeeeeeeeeeee semper__flelse www.s//apweeeeeee//ee Confeetnceceeeeeceeeeee html.
hash/aa
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for -uggeneeeet llms. arXiv preprint arXiv:2304.0822. 08244, 2023b.
Zhixun Li, Xin Sun, Yifan Luo, Yanqiao Zhu, Dingshuo Chen, Yingtao Luo, Xiangxin Zhou, Qiang Liu, Shu Wu, Liang Wang, et Gslb: the graph structure learning benchmark. Advances in Neural Information Processing Systems, 36, 2024.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi- agent debate. CoRR, abs/2305. 2023, doi: 10.48550kaxxi..2335.10000 19118. URL https:
(idoi.oogg/0.. 404448555//a/ www.vi..ivvv...////a 2305 19118.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale gener- ation: Learning to solve and explain algebraic word problems. arXiv preprint 13KYv:1705.0116: 2017.
Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. Evaluat- ing language models for efficient code generation. arXiv preprini arXiv:2408.064550 2024.
Yuntao Liu, Yong Dou, Yuan Li, Xinhai Xu, and Donghong Liu. Temporal dynamic weighted graph convolution for multi-agent reinforcement learning. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 44, 2022.
Zichang Liu, Jue Wang. Tri Dao. Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient Ilms at inference time. In International Conference on Machine Learning, pp. 22137-22176. PMLR, 2023a.
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic Ilm-agent network: An Ilm-agent collaboration framework with agent team optimization. CoRR, abs/2310.021700 2023b. doi: 10.4455//AXXIV22210007770 URL https: : doi. 40gg/10..455//ark/ 02170.
Qun Ma. Xiao Xue, Deyu Zhou, Xiangning Yu, Donghua Liu, Xuwen Zhang, Zihan Zhao, Yifan Shen, Peilin Ji, Juanjuan Li, Gang Wang, and Wanpeng Ma. Computational experiments meet large language model based agents: A survey and perspective. CoRR, abs/2402.00262, 2024.
URL httphttt///// anttps://arxin /arxivv..../ http://axxvvvl../.../.// aww./4/2.... 00222
Qiaozhu Mei, Yutong Xie, Walter Yuan, and Matthew O. Jackson. turing test of whether ai chatbots are behaviorally similar to humans. Proceedings of the National Academy of Sciences, 12119):e231322512, 2024. doi: 1..0073/paa.231139233 1073/pnas. 2313925121
URL htttps://doi..e.
Meyer, A Zill, Dilba, and Voermans. Entspann dich, deutschland! tk R-strrsssttudie 2021, 2021.
Yohei Nakajima. Babyagi. https://gitthee httppa//ggttuub...m/yyoeeeeaaaa com/yboeio//oociaaaaaaa im hm/abbyaagi..... 2023.
13

---

Published as a conference paper at ICLR 2025
Boye Niu, Yiliao Song. Kai Lian, Yifan Shen, Yu Yao, Kun Zhang, and Tongliang Liu. Flow: A modular approach to automated agentic workflow generation. arXiv preprint arXiv:2501.07834, 2025.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able solve simple math word problems? arXiv preprini wxXiv:2103.0719/ 2021.
Emanuele Pesce and Giovanni Montana. Learning multi-agent coordination through connectivity- driven communication. Machine Learning, 1 1222))483-514, 2023.
Pouya Pezeshkpour, Eser Kandogan, Nikita Bhutani, Sajjadur Rahman, Tom Mitchell, and Este- vam Hruschka. Reasoning capacity in multi-agent systems: Limitations, challenges and human- centered solutions. CoRR, abs/2402.01 108, 2024. 01108.
Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development, July 01, 2023 2023. 25 pages, figures, 2 tables.
Chen Qian, Zihao Xie, Yifei Wang, Wei Liu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Scaling large-language language-model model- multi-agent collaboration. arXiv preprini arXiv:2406.07/55, 2024.
Reworkd. Agentgpt.
com/ www.wkk//weettteee./ 2023.
Toran Bruce Richards and al. Auto-gpt: An autonomous gpt-4 experiment, GPT, 2023.
thub.
com/ com/Signte Stt//iinnnifii www.iffiiinttrtaaaaa
Subhro Roy and Dan Roth. Solving general arithmetic word problems.
arXiv preprint
arXiv:1608.01413, 014113 2016.
Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Zhang, and Fei Huang. Small Ilms are weak tool learners: A multi-llm agent. 2rXiv:2401.073244 2024.
arXiv preprint
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint, abs/2303.1 13666 1366, 2023. doi: 10.48550/axXw.23300. 11366. URL T /tt.://.ooo......//////.... 4w.4448550///// 2303.
Daniel A Spielman and Nikhil Srivastava. Graph sparsification by effective resistances. In Proceed- ings of the fortieth annual ACM symposium on Theory of computing, pp. 563-568, 2008
Qiushi Sun, Zhangyue Yin, Xiang Li, Zhiyong Wu, Xipeng Qiu, and Lingpeng Kong. Corex: Pushing the boundaries of complex reasoning through multi-model collaboration. arXiv preprint MrXiv:2310.00288 2023.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An Open-Enddee Ended Embodied Agent with Large Language Models. arXiv e-prints, art. S1KV::2305.12299 May 2023.
Junlin Wang, Siddhartha Jain, Dejiao Zhang, Baishakhi Ray, Varun Kumar, and Ben Athiwaratkun. Reasoning in token economies: Budget-aware evaluation of Ilm reasoning strategies. arXiv preprint arXiv:2406. 06461, 2024a.
Kun Wang, Yuxuan Liang, Xinglin Li, Guohao Li, Bernard Ghanem, Roger Zimmermann, Huahui Yi, Yudong Zhang, Yang Wang, al. Brave the wind and the waves: Discovering robust and gen- eralizable graph lottery tickets. IEEE Transactions on Patlern Analysis and Machine Intelligence, 2023a.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang. Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. A survey large language model based autonomous agents. Front. Comput. Sci., 18, 2024b. doi: 10.1007/ s1 024443313- URL https:::: htt.:t/ps//bb...rr//0000000////0
14

---

Published as a conference paper at ICLR 2025
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023b. URL https: haapp:://pperreee /tppppppeevvveeeeeeeeeeeeeeeeeee net/ ceretfforumm forum? d=1PLINTMME.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing cognitive synergy in large language models: A task-solving agent through multi- self- collaboration, July 01, 2023 2023c. work in progress.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi- persona self-collaborationn. In NAACL. Association for Computational Linguistics, 2024c. URL https:: pettps://arivve /ttpiv.iiivvv.////i///../aaaatva org/ http:/aa///aaa 2307 05300.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou, Chain-of-thought prompting elicits reasoning in large language models, January 01, 2022 2022.
Ronald J Williams. Simple statistical srudient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229 -229-226, 1992.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi- agent conversation framework, August 01, 2023 2023. 28 pages.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Jun- zhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang. Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xu- anjing Huan, and Tao Gui. The rise and potential of large language model based agents: A sur- vey. arxiv preprint, abs/2309.07864, 2023. URL https: doi. org/ 48550/arxiv 2309. 07864.
Yikuan Yan, Yaolun Zhang, and Keman Huang. Depending on yourself when you should: Mentoring IIm with rl agents to become the master in cybersecurity games. arXiv preprint arXiv:2403. 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, May 01, 2023 2023a. Code repo with all prompts: http:://iituh..omlyyyymyhhtttitt offthuhhht-- Ilm.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. The Eleventh International Con- ference on Learning Representations, 2023b. URL https: openreveiiv ew net. / forum?
id=WE_vluYUL-2
Zhangyue Yin, Qiushi Sun, Cheng Chang, Qipeng Guo. Junqi Dai, Xuan-Jing Huang, and Xipeng Qiu. Exchange-oftthought Enhancing large language model capabilities through cross-model communication. In Proceedings of the 2023 Conference Empirical Methods in Natural Lan- guage Processing, pp. 15135-15153, 2023.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang. Xiaohan Chen, Richard G Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing barly-bird tickets: Towards more efficient training of deep networks. arXiv preprint arXiv:1909, 1957. 2019.
Eric Zelikman, Qian Huang, Gabriel Poesia, Noah Goodman, and Nick Haber. Parsel: Algorithmic reasoning with language models by composing decompositions. Advances in Neural Information Processing Systems, 36:31466- 1664466563333 2023a.
Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-taught optimizer (stop): Recursively self-improving code generation. arXiv preprint arXiv:2310.02300 2023b.
15

---

Published as a conference paper at ICLR 2025
Guibin Zhang, Kun Wang. Wei Huang, Yanwei Yue, Yang Wang, Roger Zimmermann, Aojun Zhou, Dawei Cheng, Jin Zeng, and Yuxuan Liang. Graph lottery ticket automated. In The Twelfth International Conference Learning Representations, 2024a.
Guibin Zhang, Yanwei Yue, Kun Wang, Junfeng Fang, Yongduo Sui, Kai Wang, Yuxuan Liang, Dawei Cheng, Shirui Pan, and Tianlong Chen. Two heads are better than one: Boosting graph sparse training via semantic and topological awareness, arXiv preprint arXivv2402.01222,, 2024b.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong. Jinlin Wang, et al. Aflow: Automating agentic workflow genera- tion. arXiv preprint :xXv:2410.10066. 2024c.
Jintian Zhang, Xin Xu, and Shumin Deng. Exploring collaboration mechanisms for Ilm agents: A social psychology view. arXiv preprint arXiv:2310.02//24 2023a.
Jintian Zhang, Xin Xu, and Shumin Deng. Exploring collaboration mechanisms for Ilm agents: A social psychology view. arXiv preprint arXiv:2310.02114, 2023b.
Zheyuan Zhang, Daniel Zhang-Li, Jifan Yu, Linlu Gong, Jinchang Zhou, Zhiyuan Liu, Lei Hou, and Juanzi Li. Simulating classroom education with llm-empowered agents. arXiv preprint arXiv:2406. 19226, 2024d.
Qinlin Zhao, Jindong Wang, Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, and Xing Xie. Com- peteai: Understanding the competition behaviors in large language model-bart based agents. arXiv preprint arXiv:2310.17511 2023.
Chuanyang Zheng, Zhengying Liu, Enze Xie. Zhenguo Li, and Yu Li. Progressive-hint prompting improves reasoming in large language models, April 01, 2023 2023. Tech Report.
Zihao Zhou, Bin Hu, Chenyang Zhao, Pu Zhang, and Bin Liu. Large language model as policy teacher for training reinforcement learning agents. arXiv preprint arXiv:2311.1337 2023.
Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor Bursztyn, Ryan A Rossi, Somdeb Sarkhel, and Chao Zhang. Toolchain Efficient action space navigation large language models with a* search. arXiv preprint arXiv:2310.1322. 2023.
Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: Language agents optimizable graphs. In Forty-first International Conference on Machine Learning, 2024.
A DAG SAMPLING FUNCTION
Algorithm 2: Sample DAG from spatial communication graph
Input: Spatial communication graph g (V,E5)
Output: A directed acyclic graph Gs
G + g // Create copy of gs
while not is is.acyclic (9) do
/* Use DFS to locate cycle
cycle find (9)
e t random_choice (cycle) // Randomly select an edge from the cycle
9 < geeee (e) // Remove the selected edge from G'
return gs ف
B NOTATIONS
We conclude the commonly used notations in Table for reference.
16

---

Published as a conference paper at ICLR 2025
Table 4: The notations that are commonly used throughout the manuscript.
Notation
Definition
9 = (V,8) = {99, }'}
the spatial temporal communication graph the set of nodes (agents) the overall edge set the spatial edge set the temporal edge set the LLM base utilized by agent us the predefined responsibilliiies or roles of agent es the state of agent v; the plugins available agent 9s the spatial edge from ej the temporal edge from v; to by the temporal (in-)neight) Jneighbors of o the spatial (in )neghborrs of
V {ui, v2.... tldla
= g UE
8. g x vio
ET "V((->)> x pl0)
Base;
Role:
State;
Plugins, = CE D==
6 (Mij, Ogj)
5
c/
(v:) {0j (1.1) E ET
(v.) {0, (i.1) E E
O 9th (V.EL
the rationale or answers provided by ev the 1-th epoch
the sparsified communication topology
8 ST ERMMN
the spatial and temporal graph masks
A
E {0, M
the predefined spatial communication topology
AT (0, 1iilI
the predefined temporal communication topology
Gs
the parameterized spatial graph
gs
the parameterized spatial graph after DAG sampling
GT
the parameterized temporal graph
(()
the utility evaluation function
_ E {0, 1 PIXIII
the obtained binary spatial mask
B E (0. M
the obtained binary temporal mask the total number dialogue rounds the dialogue round after which pruning takes place the total number of queries the number of queries after which pruning takes place
K
K'
Q'
C ALGORITHM WORKFLOW
We conclude the overall algorithm workflow of AgentPrune in algorithm 3.
D MULTI-QUERY TRAINING OF AgentPrune
For complex tasks such repository-level code generation (Qian al., 2023), multi-turn dialogues (K > 5) are often essential. In such cases, utilizing K" E (1,2) rounds to optimize the topology and subsequently continue the dialogue for K K' rounds is reasonable. However, for simpler tasks that involve numerous queries, such as multiple-choice answering (Agashe et al., 2023) or basic mathematical problems (Cobbe et al., 2021), previous studies (Yin et al., 2023; Qian et al., 2024) suggest that typically only to 2 dialogue rounds are needed. In this context, prior dialogue- level optimization is no longer applicable. To better adapt AgentPrune to such cireumstances, we propose a query-level optimization paradigm for AgentPrune.
Given a benchmark consisting of Q queries, any LLM-MA framework processes these Q queries se- quentially to provide solutions one by one. We utilize the initial Q' << Q) queries as "training phase," " collaboratively optimizing the spatio-temporal communication topology while leveraging multiple agents for reasoning and evaluation. Following this, we perform one-shot pruning as de- scribed in Equation (12). The fixed topology grub is then employed for the reasoning and evaluation of the remaining (Q Q') queries. We also refer to this approach query-level optimization, in contrast to the dialogue-level optimization discussed in the main text. The distinction between the two lies in their focus: the latter concentrates on resolving single query by utilizing several initial
17

---

Published as a conference paper at ICLR 2025
Algorithm 3: Execution pipeline of LLM-MA systems combined with AgentPrune.
Input: Query q. Communication graph 9 = {g G7), Maximum rounds of iterations K,
Rounds for optimization K", Initial masks S S
A( (T)) (A 3 S$ AT [
/+ Optimizing spatial-tempora to K' do
communication topology
iteration
ون = (V.& UET) + DAGSampling($9)
for in Tepologi www.oggicaalrr@ do
Obtain temporal (in- meighbeeggbbss... (T + fuy (11)
Obtain spatial (in- au-))neighbors. N (w;) t {u; (j.i)
m (M, v E ET (w:) m (M, 10 (v))
M Po(M; g, Role (9) State (6) mm,m².
end
( AggregateSolutioonNNN M M
10
Update $ ST according to Equation (8)
end
13 B = =IAAS * A TS A x commuiccation topology
/- One-shot pruning ppatialtteporral
14 B =I(A * A (Sp |AT x (11-p%)
15 Obtain g where A(gub) = (A$ @ B A O B7)
/- Fixing the topology for subsequent iterations
16 for iteration t K' to K do
17 Use Gaub for multi-agent dialogues as Algorithm
end
19 return (*)) as the final solution
utterances derive the topology, while the former considers the entire benchmark, employing few early queries to inform the topology.
COST ANALYSIS
In Section 3.4, we present token-saving analysis in single-query setting. In this section, pro- vide cost analysis for AgentPrune in multi-query optimization context. Given a communication graph G and a benchmark with Q queries, we assume that the LLM-MA framework iterates for K dialogue rounds for each query. Furthermore, we denote the average token count per spatial, tem- poral, and query message as cs. CT. and cg. respectively. Hence, the total token consumption of the vanilla system can be expressed as:
Cg = QKK c CC|11
(14)
When utilizing AgentPrune, the LLM-MA framework processes the initial Q' queries while simul- taneously optimizing the spatial-temporal connectivity. The token cost for this phase
MQ'K K [cs|E5] + c7 C2
(15)
After pruning g and GT, we use the obtained Guub to solve the remaining Q - Q' queries, with cost of:
(Q Ø'K (1 p%)- (cs|E*1 + or( + CVIl
(16)
Overall, the token savings of AgentPrune in multi-query setting can be expressed as:
A (p% Q p% M)Q') 1 +T(E + (1 M)Q'KG,IV]
(17)
F EXISTING SPATIAL COMMUNICATION TOPOLOGIES
In this section, we introduce several existing spatial communication topologies, including chain, tree, star, complete graph, layered graph, random graph, and M-Blnddrd
18

---

Published as a conference paper at ICLR 2025
F.1 CHAIN STRUCTURE
The chain structure (in Figure 7) is one of the most widely utilized communication architectures in contemporary multi-agent systems, as demonstrated by its application in ChatDev (Qian al., 2023), MetaGPT (Hong et al., 2023), and L2MAC (Holt et al., 2024). In this architecture, the first agent receives imput from the user, transforms into new instruction, and subsequently forwards to the next agent. For instance, in MetaGPT, user instructions are initially sent to the first agent, termed the "product manager," with information progressively relayed to subsequent agents, such as the architect agent, engineer agent, and QA engineer agent. Generally, the final agent in the chain provides a solution to the user's request.
Figure 7: Demonstration of chain structure as spatial communication topology.
F.2 TREE STRUCTURE
In tree-like multi-agent pipeline, as shown in Figure 8, an administrative agent (commonly referred to as teacher, commander, manager, etc.) oversees subordinate agents, which typically have distinct responsibilities. Ultimately, these subordinate agents submit their outputs to the administrative agent for final evaluation. Notable works employing this structure include AutoGen (Wu et al., 2023), SecurityBot (Yan al., 2024), and MiniGrid (Zhou et al., 2023). For instance, in AutoGen (A4: Multi-Agent Coding), there exists a Commander agent alongside a Safeguard agent. The Writer is responsible for crafting the code and its interpretation, the Safeguard ensures safety (e.g., preventing information leaks and avoiding malicious code), and the Commander executes the code.
Mu
Figure 8: Demonstration tree structure as spatial communication topology.
F.3 STAR STRUCTURE
The star structure resembles the tree structure and can essentially be viewed as a tree with depth of two. When utilizing the star configuration for spatial communication, the central administra- tive agent receives queries from the user and dispatches instructions to subordinate agents. Upon completing their tasks using various tools, these subordinate agents return all outputs the admin- istrative agent, which then compiles a final summary, as illustrated in Figure
F.4 COMPLETE GRAPH STRUCTURE
In the main text, we refer to the structure shown in Figure 10 as a complete graph. However, this complete graph differs from the traditional definition, i.e., undirected graph where each vertex connected to every other vertex. Instead, it is a directed graph that would represent a complete graph if converted to undirected form. This distinction is necessary because the execution of the
19

---

Published as a conference paper at ICLR 2025
M
Figure 9: Demonstration of star structure as spatial communication topology.
multi-agent system relies on topological ordering (Qian et al., 2024; Zhuge al., 2024), requiring the spatial communication topology to be DAG. In MacNet (Qian et al., 2024), this structure is also referred to as "Mesh graph." " After executing in the order determined by topological sorting, the final agent summarizes the dialogue and provides concluding output or reflection.
Figure 10: Demonstration of complete graph structure as spatial communication topology.
F.5 LAYERED GRAPH STRUCTURE
A layered graph, proposed by Qian et al. (2024), refers to the structure illustrated in Figure 11, resembling stacked configuration similar a multilayer perceptron (MLP). The query is first provided to all agents in the first layer, whose outputs serve prompts that, along with the query, are then fed to the agents in the second layer. The final layer consists of single agent that receives information from the previous layer and generates the ultimate solution.
F.6 RANDOM GRAPH STRUCTURE
random graph refers to sparse graph randomly sampled from a complete graph, as illustrated in Figure 12. Irregular random structures have been shown to outperform regular fully connected struc- tures (Qian et al., 2024), which is attributed to the presence of random edge connections. Analogous to social networks, these connections can link "unacquainted" agents through direct shortcuts, trans- forming them into adjacent "acquaintances" and implicitly reducing the average path length, thereby exhibiting small-world characteristies.
20

---

Published as a conference paper at ICLR 2025
MA
Figure 11: Demonstration of layered graph structure as spatial communication topology.
Figure Demonstration random graph structure as spatial communication topology.
F.7 LLM-BLENDER STRUCTURE
The structure of LLM-Blender relatively straightforward. It feeds query to multiple LLM- powered agents from different sources, employing a PairRanker mechanism to score each agent's 's output. The top K responses are then selected and merged using LLM called GenFuser, as illustrated in Figure 13.
NLA
Figure 13: Demonstration of L.MM-Blender structure as spatial communication topology.
G EXPERIMENTAL DETAILS
G. BASELINES
In this section, we will provide a detailed overview of the various baselines mentioned in Section 4.1 and their adaptations for our evaluation.
21

---

Published as a conference paper at ICLR 2025
G.1.1 SPATIAL COMMUNICATION BASELINES
The methods described below fall under the category of spatial communication, meaning they are designed to regulate how different agents interact and exchange information within the same dia- logue round. Unless stated otherwise, we do not employ explicit inter-dialogue message passing and limit iterations two rounds (i.e., == 2).
Chain Given the diversity of benchmarks employed (including mathematical reasoning, code generation, etc.), we organized distinct agent pools for different categories of reasoning tasks, as detailed in Appendix G.3. In addition personalized prompts for each agent, also designed a universal prompt template applicable to agents for generating outputs. Taking the MMLU benchmark as example:
Prompt Template for Agents the Chain
will ask you a question and 4 answers enumerated A, B, C and D.
Only one answer out of the offered You must choose the correct answer to the question from your perspective.
correct.
Using the reasoning from other agents additional advice with critical thinking, can you give an updated answer? You are strictly prohibited from imitating the analysis process of other agents.
Your reply must be less than 100 words but include your answer and a brief step-by-step analysis of the question. The first line of your reply must contain only one letter (for example A, B, C or D)
We utilize the output from the final agent the decision for the entire system.
Star When employing the star structure, described in Appendix F.3, designate one agent as the administrative agent and utilize the other agents as subordinates. The administrative agent ultimately collects outputs from the subordinate agents to make decision. During the final decision- making process, the prompt is as follows:
Prompt Template for Decision Making
You are the top decision iasion-makezz maker and are good at analyzing and summarizing other people's opinions, finding errors and giving final answers.
Tree We reduce the tree structure to binary tree and sequentially assign agents based on the binary tree indexing, depending on the number of agents. The outputs from all non-root nodes are ultimately relayed to the root node's agent for the final decision, with the prompt template remaining same as the Star structure.
Complete Graph We employ the structure outlined in Figure 10 and execute the input/output for each agent node through topological sorting. Before performing topological sorting, it may be necessary to apply the DAG sampling method discussed Section 3.1 to ensure that the spatial communication graph is DAG. Given the relative complexity of the graph structure, which does not possess straightforward core agent like a chain tree, we introduce an additional summarizer node to which all other nodes direct their outputs. Naturally, this summarizer node is executed last in the topological order, positioning it as final decision-male making expert.
Layered Graph As discussed in Appendix F.5, we arrange the agents MLP-like layered structure, ensuring that the final layer contains only one agent. This agent receives outputs from all agents the preceding layer and produces the final solution.
22

---

Published as a conference paper at ICLR 2025
Random Graph The implementation of random graph is similar to that of a complete graph. It also begins with DAG sampling. followed by execution through topological sorting, and concludes with summarizer agent that generates the overall response. We generate the random graph with connection probability set as 0.5.
LLM-Blender LLM-Blender (Jiang et al., 2023) was originally designed to consolidate responses from various LLM architectures. In this context, treat it spatial message- passgggg-gnnggggg paradigm, standardizing all agents to utilize either opt-3: 33 or gpt-4, 4, with the final output from GenFuser serving the solution. Notably, LLM-BBender is specifically tailored for single-turn dialogues; thus, we do not employ multi-turn dialogues in conjunction with LLM-BAN -.M-Bledder.
GPTSwarm GPTSwarm (Zhuge et al., 2024) conceptualizes the connections among all agents as a parameterized, dense adjacency matrix, which is continuously optimized to enhance collabo- rative performance. In the original paper, distinct internal structures were customized for differ- ent agents, such as configuring specific agent to first receive a query, followed by performing F FileAnalya yze and WebSearch, and ultimately outputting the results. To ensure fair com- parison, we did not utilize such configurations in Section 4; instead, we assigned each agent dif- ferent profiles, similar to the other structures mentioned above, along with possible external tools like a Python compiler or Wikipedia searcher. Our implementation is based on the resources avail- able at https:::: qi thub. com/ me Bawauutaai/GeSSaaaaa. open-sourced code, the multi-agent collaboration for MMLU dataset only transmitted the options A/B/C/D during dialogues, without including the content of agents' reasoning process, which is not consistent with the description in Section 2.2 their manuscript. To ensure a fair comparison and maintain consistency with the original description, we modified their code transmit both the choices and the reasoning process.
Important Note: in the originally
G.1.2 TEMPORAL COMMUNICATION BASELINES
LLM-Debate LLM-Debate (Du et al., 2023b) is designed for multiple agents to engage de- bate, where in each round, every agent receives the outputs of all agents from the previous round before making their own statements. Consequently, it essentially forms fully connected temporal communication graph, illustrated in Figure 14.
Figure 14: Demonstration of LLM-DDeate structure as temporal communication topology.
PHP PHP (Zheng et al., 2023) progressively improves prompts by utilizing the entirety of his- torical dialogue, offering potential "hint" prompts. this context, we adapt this setting to the multi-agent collaboration framework, using the decisions made after each round of dialogue as hint prompts for all agents the subsequent round, as depicted in Figure 15.
DyLAN DyLAN (Liu et al., 2023b) primarily focuses on optimizing temporal communication and reducing redundancy by employing specific scoring mechanism to eliminate low-quality outputs between every two rounds dialogue. We utilize the official implementation available at https: //git tigtthbb.... cwm/saaa/eeoooooo SALTT-NLR /DYLAN.
23

---

Published as a conference paper at ICLR 2025
Hint
Hint
67
Figure 15: Demonstration of PHP structure as temporal communication topology.
Manager
Manager
query
inpul
Solution
Solution
Query
Engineer
Critic
Engineer Critic Executor User Proxy
Figure 16: The AutoGen system design with three/five ALM-based agents. The dashed line indicates the agent will propagate its rationale output back the manager for its final decision-making.
G.1.3 OTHERS
In Section 4.3, we integrated AgentPrune with AutoGen and GPTSwarm under both three-agent and five-agent configurations. Here, we elaborate on how we implemented AutoGen, which is inher- ently customizable framework. Based the setup from AutoGen (A5: Dynamic Group Chat), we define five roles: user proxy, manager, engineer, critic, and code executor. For the three-agent con- figuration, we condense these roles into manager, engineer, and critic. The detailed communication structure is depicted in Figure 16.
G.2 EXPERIMENTAL CONFIGURATINSS
In this section, we supplement a detailed description of the experimental setup for AgentPrune:
Evaluation function ø: The implementation of á varies across tasks. For multiple-choice bench- marks such as MMLU and AQuA, é represents accuracy. For HumanEval, it corresponds to the pass@1 metric for code execution. For other mathematical benchmarks, á computes accuracy by comparing the system-generated answers with the ground truth using string matching.
Initialization of graph masks: The graph masks S $ ST are initialized as 0.5 1p and is an all-one matrix.
Training queries Q': As noted in Section 4.1, Q' is set to one of (5, 20).
Temperature setting: The temperature parameter for gpt 3.5 and gpt -4 is consistently set to 1 during generation.
Samples per query M: We set M 10 for each training query.
G.3 AGENT PROFILING
Previous works (Wang et al., 2024c) have formally established that assigning different personas or roles to LLM-based agents can enhance cognitive synergy among agents. Consequently. we utilized gpt-4 4 to generate a series of agent profiles for various tasks, thereby promoting diversity and collective intelligence in multi-agent setting.
24

---

Published as a conference paper at ICLR 2025
G.3.1
PROFILE EXAMPLES FOR GENERAL REASONING
Below are some examples of agent profiles tailored for general reasoning tasks:
Knowledge Expert
You knowledgeable expert in question answering.
Please give several key entities that need to be searched in Wikipedia to sclve the problem.
Key entitics that need to be searched are included between two "g' when output, for example: &catfish effecte, Ebroken window effecte, eShakespearce.
If there no entity in the questien that needs to be searched in Wikipedia, you don't have to provide it
Wiki Searcher
You will be given question and Wikipedia overview of the key entities within it.
Please refer them step by step to give your answer.
And point out potential issues in other agent's s analysis.
Critic
You are an excellent critic.
Please point out potential issues in other agent's s analysis point by point.
Mathematician
You are a pathematician who is good at math games, arithmetic calculation, and long-term planning.
Programmer
You are good at computer science, engineering, and physics. You have experience in designing and developing computer software and hardware.
You especially good writing code complex programs with Python, C++, MATLAB, JAVA, etc.
Doctor
You are doctor and come up with creative treatments illnesses or discases.
You are able to recommend conventional medicinos, herbal remedics and other natural alternatives.
You also consider the patient's s age, lifestyle and medical history when providing your recommendations.
Economist
You are good at economics, finance, and business. You have experience on understanding charts while interpreting the macroeconomic environment prevailing across world economies.
G.3.2 PROFILE EXAMPLES FOR MATHEMATICAL REASONING
Below examples of agent profiles tailored for mathematical reasoning tasks:
25

---

Published as a conference paper at ICLR 2025
Math Solver
You math expert.
You will be given math problem and hints from other agents.
Give your own solving process step by step based on hints.
The last 1ine of your output contains enly the final result without any units, for example: The answer is 140.
You will given some examples you may refer to.
Mathematical Analyst
You are mathematical analyst.
You will given math problem, analysis and code from other agents.
You need to first analyze the problem-solving process step by step, where the variables are represented by letters.
Then you substitute the values into the analysis process to perform calculations and get the results.
The last line of your output contains only the final result without any units, for example: The answer is 140
You will be given some examples may refer to.
Programming Expert
You are a programming expert.
You will be given math problem, analysis and code from other agents. Integrate step-by-ster step reascning and Python code to solve math problems.
Analyze the questien and write functiens to solve the problem. The function should not take any arguments and use the final result as the return value.
The last line of code calls the functien you wrote and assigns the return value the (answer
) variable. Use a Python code block to write your response. For example: <some
python code>
Do not incluce anything other than Python code blocks in your response. You will be given some examples you may refer to.
G.3.3 PROFILE EXAMPLES FOR CODE GENERARION
Below are some examples of agent profiles tailored for code generation tasks:
Project Manager
"You are project manager.
"You will given function signature and its docstring by the user.
"You are responsible for overseeing the overall structure of the code, ensuring that the code is structured to complete the task Implement code concisely and correctly without pursuing over-ongineering."
"You need to suggest optimal design patterns to ensure that the code follows best practices for maintainabilic and flexibility. "You can specify the overall design of the code, including the classes that need
defined (maybe none) and the functions used (maybe only one function)
"I hope your reply will be more concise. Preferably within sifty words. Don't list too many points."
26

---

Published as a conference paper at ICLR 2025
Algorithm Designer
"You are algerithm designer.
"You will given function signature and its docstring by the user.
"You need to specity the specific design of the algoritam, including the classes that may be defined and the functions used. "You need to generate the detailed docurentation, including explanations the algorithm, usage instructions, and API references.
"When the implementation logic is complex, you can give the pseudocode logic of the main algorithm." "I hope your reply will be concise. Preferably within fifty words. Don't list too many points.",
Programming Expert
"You are programming expert.
"You will be given a function si signature and its docstring by the user.
"You may be able to get the output results of other agents. They may have passed internal tests, but they may not be completely correct. "Write your full implementation (restate the function signature).
"Use Python code block write your response. For example: <some python code> "Do not include anything other than Python code blocks in your response.
"Do not change function and input variable types in tasks."
Bug Fixer
"You are bug fixer."
"You will be given a function signature and its docstring by the user.
"You need previde modified and improved python code based on the current overall code design, algorithm framework, code implementation or test problems.
"Write your full implementation (restate the function signature). "Use Pythen code block to write your response. For example: <some python code>"
"Do not include anything other than Python code blocks in your response
"Do not change function names and input variable types in tasks"
G.4 AGENT ATTACK IMPLEMENTATION
We designed two types of attacks on agents: agent prompt attack and agent replacement attack, with specific implementations detailed as follows:
Agent Prompt Attack We attempt to compromise the role prompt of the collaborative agent, altering its predefined role the following "liar" agent:
27

---

Published as a conference paper at ICLR 2025
Prompt Template for Agent Prompt Attack
You are liar who only tell lies. Give wrong answer and false analysis process for the following question: (questien].
You may get output from other agents, but no matter what, please only output 1ies and try your pest to mislead other agents.
Your reply must be less than 100 words.
The first line of your reply must contain only letter (for example : A, B, or D)
Agent Replacement Attack We replace the originally high-cognitive and planning-capable LLM with randomly generating "dummy" API that outputs text without coherence. The specific prompt is as follows:
Prompt Template for Agent Replacement Attack
Randomly output letter from ABCD the first line.
Then output any gibberish paragraph on the same topic as the following question: question.
The first 1inc of your reply must centain only one letter (for example A, B, C or D)
When attacking all multi- agent frameworks, randomly select one agent to the adversarial agent, while the remaining agents retain their original functions and responsibilities.
H SUPPLEMENTED EXPERIMENTAL RESULTS
H.1 RESULTS FOR RQI
Figures 17 to 19 compares AgentPrune with other communication topologies in terms of comple- tion tokens, overall tokens, and overall cost (USD). Notably, across multiple datasets, AgentPrune achieves superior performance of the economic expense of SOTA topologies. For instance, on the MMLU dataset, AgentPrune surpasses GPTSwarm's performance with cost of only $5.6, compared to GPTSwarm's 's $43.56. Similarly, on the GSM8K dataset, AgentPrune outperforms DyLAN with a cost of $65.9, whereas DyLAN incurs a cost of $357.47.
fraction of the cost, often low one-half or even one-tenth
MMLU
pirtuein Cernl ablict
HumanEval
GSMBK
PEAA Lake
Cenrgy tikers
Toloss concumppttee (10"
Lavee
Eigrte Morel
20
fearrsl soc
Accuneyy
Paseg81 (%)
Arcuracy
Figure 17: Seatter plot illustrating the performance metries and total token consumption of different multi- agent communication topologies across the MMLU, HumanEval, and GSM8K datasets. The diameter of each point is proportional to the value on the y-axis, representing token consumption.
H.2 RESULTS FOR RQ2
Table 5 presents a comparison the performance, prompt/completion token consumption, and cost between AgentPrune integrated with three-agent-basee AutoGen and GPTSwarm frameworks. Ta- ble 6 showcases the training and total (training + inference) token costs AgentPrune when com- bined with various L.LM-MAS backbones. As shown, AgentPrune requires only around 2% 6%
28

---

Published as a conference paper at ICLR 2025
EvmanEval
Toluso enicis cos.smmyoooo (10"
PLAM
Tolee comne topoonn
CAarinng
Npm or
OfTiway
Accuracy
Pa (%)
Accurasy
Figure 18: Scatter plot illustrating the performance metrics and completion token consumption different multi-age agent communication topologies across the MMLU, HvlanEval, and GSM8K datasets. The diameter of each point is proportional to the value the y-axis, representing token consumption.
MILU
LE LE-S
HumanEval
and
GSM8K
TAXI CARcN
GETO
Cough Unioie
d4 90s EXEN
150
Tose Dototott 1940
$5
Accuracy (55)
PioDEL (5))
Accuracy (%)
Figure 19: Seatter plot illustrating performance metrics and total cost (USD) different multi-agent communication topologies across the MMLU, Hvaa and GSM8K datasets. The diameter of each point is proportional to the value on the y-axis. representing the economic cost.
Table Performance and cost comparison before and after combining AgentPrune. We evaluated performance and economical cost comparison of AgentPrune in conjunction with two classic multi-agent systems, AutoGen and GPTSwarm, under three gpt-4 4-hased setting.
Dataset
Method
Performance
# Prompt Tokens
# Completion Tokens
Cost
AutoGen
81.94
346, 028
66, 204
$5.446
MMLU
AgentPrune
82 $2.20(1 0.26)
274, 665(79.4%)
66, 803
$4.750(87.255
AutoGen
83.66
351, 985
90, 762
$6. 242
HumanEval
AgentPrune
35.04(1 1.38)
254, 203(72.2%)
89, 362
83.282(844.66)
AutoGen
87.23
2,317, 937
624, 055
$4 $41.92
GSM8K
AgentPrune
88.51(f 1.23)
481, 780(63.99% 9%)
629, 771
833. (800.44))
GPTSwarm
83.32
1,521,500 504
325, 994
$24.99
MMLUU
AgentPrune
83.66(t 0.34)
554, 638(35.8%)
336, 887
$15.65(62.655
GPTSwan
83.62
1, 1,478, 312
612, 815
$33.16
HumanEvvl
AgentPrune
84.74(t 1.12)
432, 42.408009..00 (29.2%)
598, 367
$22.07(66.59%)
GP''Sarmm
87.85
6, 27,, 665
186, 510
$68.34
GSMSK
Agenntruuee
88.30(t 0.45)
3. 0000, (15(47.9
173, 296
$35.29(51.085)
Table 6: Training and total token costs of AgentPrune (AP) applied to different LLM-MAS backbones on the GSM8K dataset.
# Training # Total
# Training
Total
LLM-MAS
Prompt Tokens
Prompt Tokens
Ratio (%)
Completion Tokens
Completion Tokens
Ratio (3)
Complete Graph+AP
274.821
8,526.035
3.22
69,808
2,022,560
3.45
Random Graph+AP
269,732
7,495.738
3.59
63,054
1,796.603
3.50
AutoGen+AP
158.474
3,791,251
4.18
78,899
1.156,884
6.82
GPPSSarm=-AP
91,192
3,526.035
2,80
22,982
730,552
3.10
of token consumption to complete the spatial-temporal connectivity optimization. Once this opti- mization achieved, all subsequent inferences benefit from a token-efficient and economical com- munication structure.
29

---

Published as a conference paper at ICLR 2025
before attack
after attack
90
81.8
85.4
83-7
84:7
Rg.6
R4.88g.7
83.5
B2.1
S0.1
80
MM.5
75
24--2
70-3
71-3
65
Chain like
Star-like
Tive-like
Complets Gruph
Random Graph
AgontPrume-
AgentPrune-R
DyLAN
GP GPTSwarm
GrTSwarm w/ AP
Autoonn
AutoGien AP
Figure 20: Performance under adversarial attack. We compare the performance of various multi-agent frameworks before and after agent replacement attacks. "w/ AP" indicates the integration with AgentPrune.
Table 7: Abltion study of AgentPrune. "w/o profile" denotes not assigning agents with different roles and profiles; "w/o low-rank" denotes not using the low-rank regularization described in Equation (11).
Variant
MMLU
GSM8K
MultiArith
SVAMP
AQuA
HumanEval
AgentPrune-C
84.72
95.62
97.25
91.85
79.47
89.38
w/o profile
84.37
93.70
96.21
91.79
79.19
87.83
low-rank
84.65
94.55
96.84
91.12
79.57
88.90
AgentPrune-R
83.94
95.83
96.30
91.68
78.60
90.30
w/o profile
83.36
95.67
95.72
91.75
78.75
88.60
w/o low-rank
83.52
95.46
96.05
91.30
78.49
89.39
H.3 RESULTS FOR RQ3
We supplement the performance of various topologies before and after being perturbed by agent replacement attack in Figure 20.
H.4 RESULTS FOR RQ4
H.4.1 ABLATION STUDY
We develop two variants of AgentPrune: "AgentPrune w/o profile", which assigns unique roles to each agent, and "AgentPrune w/o low-rank", which does not implement the low-rank regularization described Equation (11). The results are presented in Table 7.
H.4.2 SENSITIVITY ANALYSIS
We amalyze the sensitivity of AgentPrune to two parameters: the number of agents | and the early stopping round K. The findings are presented in Figure 21. Notably, as the number of agents increases, there is significant performance enhancement initially (from 3 to 5 agents), while sub- sequent improvements (from to agents) in accuracy become relatively marginal. The early stopping round Q' indicates the number of queries used optimize spatial-temporal connectivity in a multi-query training setting. Intuitively, increasing the number of optimization rounds leads to more refined and accurate graph mask, resulting in substantial performance gains from multi-agent collaboration, along with reduced performance fluctuations. To balance performance with token sav- ings, we consistently set Q' (5. 10, 20). As for the pruning ratio, we can observe that, with p% increasing, the performance exhibits an overall decreasing trend. Taking both the token efficiency and performance into consideration, we set p% E (50%, 30%).
H.5 COMPARISON WITH RANDOM PRUNING
To better illustrate the differences between AgentPrune and random pruning, supplement the analysis with results on six datasets: MMLU, GSM8K, MultiArith, SVAMP, AQuA, and Hu- manEval, shown in Table 8
30

---

Published as a conference paper at ICLR 2025
Acturaty 45
Accuracy (%)
Accurticy
Number aggents [V]
Early Stopping Round Q'
Proming Ratio pi
Figure 21: Parameter sensitivity analysis AgentPrune.
vary the number of agents [V]
{3, 6, 7,9,9 the early stopping round Q' E {5, 10, 15, 15, 20, 25} and the pruning ratio p% (10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%} on MMLU dataset.
Table 8: Performance comparison between AgentPrune and random pruning. The pruning ratio is fixed at 50%, and five gpt-4-baser agents are employed.
LLM-MAS
Pruning Method
MMLU
GSMSK
MultiArith
SVAMP
AQuA
HumanEval
Complete Graph
N/A
83.15
86.49
97.20
89.48
79.21
83.75
+ AgentPrunee
84.72
95.62
97.25
91.85
79.47
89.38
+ RRandom Pruning
82.30
85.60
95.80
83.90
74.50
82.70
Random Gruph
83.76
86.14
95.46
85.41
74.07
82.66
AgentPrune
83,94
95.83
96.30
91.68
78.60
90.30
+Random Pruning
83.20
86.40
92.90
82.70
71.10
77.40
GPTSwarm
83.98
89.74
97.84
86.42
78.16
88.49
AAgentrrune
83.05
90.58
97.11
88.41
78.50
88.96
+Random Pruning
83.10
85.21
96.30
83.60
75.30
82.70
Table 9: Transfcrability analysis of AgentPrune, with the backbone LLM-MAS being a complete graph.
Structure From Performance Tested On
AQuA
MultiArith
SVAMP
GSMSK
w/o AgentPrune
79.21
97.20
89.48
93.80
Optimized From AQuA
79.47
Optimized From MultiArith
97.25
Optimized From SVAMP
78.92
96.54
91.85
94.80
Optimized From GSMSK
80.50
97.73
91.68
95.62
The results demonstrate that random pruning consistently underperforms AgentPrune across var- ious LLM-MAS backbones, causing notable performance degradation. While this degradation is relatively modest in dense communication structures like the Complete Graph, it becomes more pronounced in inherently sparse structures such Random Graph and GPTSwarm. For example, random pruning results in a significant 5.79% drop in pass @ on HumanEval-GGPSSwam trast, AgentPrune achieves 0.47% performance gain while requiring only 27.2% of the vanilla GPTSwarm's 's prompt tokens. This improvement can be attributed to AgentPrune's 's refined optimiza- tion strategy and precise connection importance evaluation.
In con-
H.6 TRANSFERABILITY OF AgentPrune
We conducted transferability experiments across four mathematical reasoning datasets, evaluating the performance of communication graphs optimized by AgentPrune when transferred to other datasets without further optimization. The results summarized in Table 9. We draw several key conclusions:
1. AgentPrune-optimizee communication graphs exhibit strong transferability on sim- ilar tasks. For instance, directly applying the structure optimized on GSM8K AQuA outperforms the structure specifically optimized on AQuA itself.
2. Transferability is influenced by the knowledge capacity of the dataset. GSM8K, with thousands of mathematical queries, yields communication graphs that generalize well to
31

---

Published as a conference paper at ICLR 2025
other datasets. For example, the GSMSK- structure achieves ter performance compared to vanilla optimization on MultiArith and AQuA, including a 1.03% improvement AQuA. In contrast, SVAMP, composed of merely elementary-level queries, demonstrates relatively limited generalization capacity. Its optimized structure sometimes leads to minor performance drops, such 0.71% decrease when transferred to MultiArith.
bet-
I CASE STUDY
1.1 SPATIAL TOPOLOGY VISUALIZATION
In this section, we will demonstrate how AgentPrune operates on the predefined spatial commu- nication structure and the resulting structure after one-shot pruning. It is important to note that the pruned sparse graph we present is likely not a Directed Acyclic Graph (DAG). This is because, during the practical application, still need to utilize the DAGSampling function on the pruned graph to ensure it conforms to DAG properties before sequentially executing agent I/O.
GPTSwarm + MMLU Figure 22 illustrates how AgentPrune applies one-shot pruning within five-agent GPTSwarm framework, where three agents serve as simple I/O agents, and the remaining two are TOT (Tree of Thoughts) agents. Notably, AgentPrune prunes many in-edges for the I/O agents, while preserving many for TOT agents. This behavior likely stems from the stronger reasoning capabilities of TOT agents, which makes them better suited for synthesizing the agents' discussions and providing the final solution. Figure 23 showcases how AgentPrune completely prunes the adversarial agent out of the system by deleting all its in/out-edges.
GPTSwarm MMLU
I/O
I/O agent
agent
I/O agent
I/O agent
I/O agent
I/O agent
one-shot pruming
TOT agent
TOT agent
TOT
TOT
agent
agent
Figure 22: The pruning process of AgentPrune on five-agent GPTSwarm framework when tested on MMLU,
AutoGen + HumanEval Figure 24 illustrates how AgentPrune applies one-shot pruning within
five-agent AutoGen framework.
Figure illustrates how Agent.ruuee performs one-shot pruning to achieve compact sparse spatial communication graph, given five predefined agent roles: knowledge expert, critic, historian, mathematician, and psychologist, along with a predefined complete graph structure on MMLU dataset. It is evident that in the pruned graph, the Critic has high number of incoming edges, while the Knowledge Expert has a high number of outgoing edges. This aligns with their respective functions: the Critic is expected receive broad range of external information and provide feedback, whereas the Knowledge Expert should output useful knowledge based on their knowledge base.
Complete Graph + MMLU
32

---

Published as a conference paper at ICLR 2025
GPTSwarm MMLu
I/O agent
1/O
adversarial
I/O agent
adversarial agent
agent
1/O agent
agent
one-shot proning
M
TOT agent
TOT agent
TOT agent
TOT agent
Figure 23: The pruning process of AgentPrune live-agent GPTSwarm framework when tested MMLU, where one the five agents set the adversarial agent.
AutoGen + HumanEval
Manager
Manager
My
one-shot pruming
Engineer Critic Executor Proxy
Engineer Critic Executor Proxy
Figure 24: The pruning process of AgentPrume on a five-agent AutoGen framework when tested on MMLU.
Complete Graph MMLu
knowledge
knowledge expert
expert
Critic
Historian
Critic
Historian
one-shot pruning
Mathematician
Psychologist
Mathematician
Psychologist
Figure pruning process AgentPrune on a five-agent complete- graphe-rraaakkk-a framework when tested on MMLU dataset.
Random Graph MMLU Figure 26 illustrates how AgentPrune performs one-shot pruning, given five predefined agent roles along with predefined random graph structure MMLU
33

---

Published as a conference paper at ICLR 2025
dataset. The pruned graph structure is similarly aligned with that in Figure 25, with the Critic having many incoming edges and the Knowledge Expert having many outgoing edges. Notably, the absence of outgoing edges for the psychologist in Figure 26 may suggest that this role has limited utility within the overall system.
Random Graph MMLu
knowledge expert
knowledge
expert
Critic
Historian
Critic
Historian
one-shot pruming
Mathematician
Psychologist
Mathematician
Psychologist
Figure The visualization of how AgentPrune prunes five-agent randon-graph-like framework when tested on MMLU dataset.
Random Graph + HumanEval Figure 27 demonstrates how AgentPrune implements one-shot pruning based on five predefined agent roles: project manager, algorithm designer, bug fixer, test an- alyst, and programming expert, utilizing predefined complete graph structure the HumanEval dataset. Notably, the outer edges of the graph are retained, aligning with the standard workflow for code completion, which is consistent with the designs of multi-agent code generation frameworks like MetaGPT (Hong et al., 2023). The Bug Fixer has no outgoing edges, as it represents the fi- nal step in the code completion process. This effectively highlights AgentPrune's capability for autonomous optimization of multi-agent collaborative topologies.
Complete Graph HumanEval
Project Manager
Project Manager
Algorithm
Algorithm Designer
Designer
Bug Fixer
Bug Fixer
one-shot pruning
Programming Expert
Test
Programming Expert
Test Analyst
Analyst
Figure 27: The visualization of how AgentPrune prunes five-agent complete-graph-iee framework when tested the HumanFval dataset.
Complete Graph + HumanEval Figure 27 illustrates how AgentPrune executes one-shot prun- ing based four predefined agent roles: math solver, math analyst, programming expert, and in- spector, utilizing a predefined complete graph structure on the GSM8K dataset. In this scenario, designated two agents math solvers. Interestingly, the pruned graph structure reveals explicit
34

---

Published as a conference paper at ICLR 2025
Random Graph HumanEval
Project Manager
Project Manager
Algorithm Designer
Algorithm Designer
Bug Fixer
Bug Fixer
one-shot pruning
Programming Test
Programming Expert
Test Analyst
Expert
Analyst
Figure The visualization how AgentPrune prunes five-agent random-graph-like framework when tested on the HumanEval dataset.
differentiation in roles for the two math solvers: one is responsible for preliminary solving, while the other is tasked with final solving. The final solver gathers information from all other nodes and has no outgoing edges.
Complete Graph + GSM8K Figure 29 illustrates how AgentPrune performs one-shot pruning based on four predefined agent roles and predefined random graph structure on the GSM8K dataset. We observe distinct differentiation in node characteristics: the agents focused on problem analysis exhibit high out-degree with no incoming edges, while the agents responsible for problem- solving demonstrate a high in-degree and a low out-degree.
Complete Graph + GSMak
Math Solver
Math Solver
Math Analyst
Math Solver
Math Analyst
Math Solver
one-shot pruming
Programming Expert
Programming Expert
Inspector
Inspector
Figure 29: The visualization of how AgentPrune prunes five-agent complete-graph-lik framework when tested the GSMSK dataset.
1.2 TEMPORAL TOPOLOGY VISUALIZATION
Figures 31 to demonstrate how AgentPrunt operates on the predefined temporal communication structure and the resulting structure after one-shot pruning.
35

---

Published as a conference paper at ICLR 2025
Random Graph + GsMak
Math Solver
Math Solver
Math Analyst
Math Solver
Math Analyst
Math Solver
one-shot pruning
Programming Expert
Programming Expert
Inspector
Inspector
Figure The visualization how AgentPrune prunes five-agent random-grass -ialoo--raah-lkeee framework when
tested on the GSM8K dataset.
Temporal Pruning HumonEval
Project
Project Manager
Project
Project Manager
Manager
Manages
Algorithm Designer
Algorithun Designer
Algorithm Designer
Algorithm Designer
Program Expert
Program Expert
one shat pruniny
Program Expert
Program Expert
Test
Test Analyst
Test Analyst
Analyst
Analyst
Bug Pixer
Bug Fixer
Bug Fixer
Bug Fixer
T-
T
T-
Figure 31: The visualization of how AgentPrune temporally prunes five-agent I...-Dehatt-like framework when tested the HumanEval dataset.
I.3 SPATIAL CONNECTIVITY OPTIMIZATION
Figures 34 to 36 illustrate the optimization trajectory of spatial connectivity before applying one-shot pruning. As time progresses, the initially identical graph masks begin to show increasing differenti- ation, with their magnitudes serving as a crucial reference for subsequent pruning decisions.
1.4 THE PRUNING ELEMENT ANALYSIS
In this section, we summarize the pruning mechanism of AgentPrune, focusing on two primary cat- egories of messages that it targets: malicious misleading information and redundant content. For the first category, as illustrated in Figure 38, Web Browser acts as an adversarial agent by introducing direct prompt injection. This leads to misleading information, such the claim that
36

---

Published as a conference paper at ICLR 2025
Temporal Pruning + GsMaK
Math Solver
Math Solver
Math Solver
Math Solver
Math Analyst
Math Analyst
Math Analyst
Math Analyst
one-shot pruniny
Program Expert
Program Expert
Program Expert
Program Expert
Inspector
Inspector
Inspecter
Inspector
T-
T
T-
T
Figure 32: The visualization of how AgentPrune temporally prunes four-agent LLM- Debate-like Mike framework when tested the GSM8K dataset.
Temporal Pruning + GsMak
Math Solver
Math Solver
Math Solver 1
Math Sulver
Math Analyst
Math Analyst
Math Analyst
Math Analyst
one--shot pruming
Program Expert
Program Expert
Program Expert
Program Export
Inspector
Insgector
Inspector
Inspector
Math
Math Solver
Math
Math Solver
T-1
T
T-
Figure 33: The visualization how AgentPrune temporally prunes a five-agent L. framework when tested on the GSM8K dataset.
the "Top Silliest Animal Moments" includes the fictitious "kakapo parrot." Consequently, down- stream agents such as inspectors and aggregators are misled, failing to provide the correct answer. For the second category, redundant information more prevalent. As shown in Figure 39, the math analyst, after receiving input from the math solver, fails to contribute novel insights and instead nearly replicates the solver's content. In such scenarios, AgentPrune eliminates most of the outgo- ing edges from the redundant agent, thereby reducing unnecessary information flow and minimizing token consumption.
37

---

Published as a conference paper at ICLR 2025
0.47
0.00
0.90
0.50
0.00
0.50
0.42
0.00
0.50
0.54
0.g0
0.g9
0.00
0.g0
0.g0
0.59
0.50
0.49
0.59
0.g9
0.48
0.52
0.00
0.68
0.36
0.50
0.99
0.50
0.90
0.50
0.50
0.50
0.50
0.41
0.59
0.57
0.48
Round
Round 2
Round 3
0.42
0.00
0.47
0.53
0.42
0.00
0.45
0.52
0.49
0.50
0.00
0.66
0.33
0.48
0.50
0.00
0.69
0.39
0.40
0.45
0.40
0.64
0.63
0.44
Round 4
Round 5
Figure 34: The demonstration of how spatial connectivity evolves and optimizes over time on MMLU,
0.00
0.50
0.50
0.48
0.00
0.46
0.50
0.00
0.55
0.55
0.42
0.50
0.50
0.00
0.50
0.50
0.46
0.46
0.00
0.54
0.48
0.46
0.43
0.00
0.55
0.47
0.go
0.g0
0.50
0.g0
0.54
0.54
0.52
0.g0
0.53
0.58
0.52
0.54
Round
Round 2
Round 3
0.48
0.59
0.40
0.39
0.00
0.56
0.58
0.00
0.55
0.61
0.96
0.65
0.44
0.42
0.00
0.57
0.45
0.43
0.42
0.00
0.57
0.45
N
0.43
0.41
0.00
0.57
0.44
0.52
0.54
0.00
0.54
0.49
0.64
0.56
0.57
0.48
0.67
0.58
0.00
0.47
0.47
Round
Round 5
Round 6
Figure 35: The demonstration of how spatial connectivity evolves and optimizes over time on HumanFval.
38

---

Published as a conference paper at ICLR 2025
0.00
0.59
0.90
0.50
0.90
0.00
0.50
0.50
0.50
0.50
0.00
0.40
0.44
0.42
0.46
0.50
0.50
0.60
0.50
0.50
0.00
0.50
0.50
0.50
0.55
0.36
0.53
0.54
0.47
0.39
0.50
0.50
0.55
0.00
0.50
0.50
0.50
0.50
0.00
0.50
0.50
0.50
0.50
0.00
0.58
0.42
0.59
0.47
0.00
Round
Round 2
Round
0.43
0.00
0.41
0.46
0.37
9.44
0.90
0.61
0.97
0.05
0.66
0.56
0.69
0.61
0.60
0.45
0.00
0.68
0.49
0.45
0.48
0.63
0.44
0.00
0.70
0.47
0.55
0.39
0.52
0.00
0.57
0.54
0.37
0.56
0.36
0.36
0.52
0.00
0.56
0.88
0.59
0.61
0.36
0.59
0.49
0.00
0.34
0.59
0.49
Round 4
Round 5
Round 6
Figure 36: The demonstration how spatial connectivity evolves optimizes over time on GSM8K dataset.
3.0
2.5
Absolute Difference
1.5
0
2
4
6
8
10
12
14
16
18
Round
Figure 37: The absolute difference of consecutive spatial graph masks.
39

---

Published as a conference paper at ICLR 2025
Query: On the BBC Earth YouTlabe video of the Tog 5 Silliest Animal Moments, what species of bird feutured?
Web browser
Web browser
Inspector
Aggregator
In the BBC Eanh's YouTube
The "Top Silliest Animal Moments" by BBC Earth features penguin as the highlighted bird species. Penguins often appear lighthearted conrexts in RBC Earth due to their amusing and relatable behaviors.
Both answers identify
The BBC Earth YouTube video "Top Silliest Animal Moments" features multiple bird species, including the kakapo, a lorge. flightless parnor, and penguins, known for their humorous and relatable behaviors. get comprehensive view of all featured species, you can watch the video on the BBC Earth chamnel.
video 'Top Silliest Animal
species (kakapo in anewer and penguin in answer being fearured the video, creating conflicting cluims. Neither answer provides specific timestamps detailed descriptions to confirm their assertions, and both rely on generic context about the species. To determine accuracy. would need to watch the video themselves.
Moments," one of the for rured bird spocies
Prun
large, flightless for
bel
can full video on the BBC Earth channel for more detaill.
You watch the video for more entertaining highlights
adversarial
benign
mislead
mislcad
Figure 38: Pruning focus of AgentPrune: Malicious or misleading information. This example highlights how adversarial agents, such as Web Browser 1, inject misleading content (e.g., referencing a fictitious "kakapo parrot"), which propagates errors through subsequent agents.
Query: Christina planoing birthday party and needs .75 gift hags per invited guest, because 1/4 of attendees don't show up. She invited 16 friends. hags each. How much will she spend!!
Math Solver
Math Analyst
solve
The solution from math solver agent correct and methodical. Here's quick recap problem solving steps and calculations: 1. Calculite the number of guests who will attend Christina invited 16
Calculate the number of who will atrnn:
Christina friends, bur of them won't show up. This means & will attend:
friends, Since them won't It show $ of them will attend:
Atlendees 16 12
A Attenclees 16 x -12
2. Determine the total number of gift eled. Christina needs
Prune
2. Determine total number of gift bugs needed: Christina plans
0.75 per guest:
0,75 gill per
0.75
Gift bags needed
3. Calculate the total oan of the gift bags. Each gift bag costs $2:
3. Caleulate the total cost of the gift bags: Each gift bag costs S2:
=
Total cast x 2 18
Final Verification:
Answer: Christina apend **gig**
The ressening aligns with the problem's conditions, and all calculations correct. Christina will spend **518** gift hags.
Programmer
Inspector
Here is the Python code solve the problem:
The problem has been tharoughly analyzed, and the solition correct. Here's the concise breakdown:
Number invited guests invited, gueses 16
Fraction guests who
attendance rute 0,75
Christina invited 16 friends, and will not allend, (12 guests) will atlend. 2. Each guest roquires 0.75 gift bugs, resalting 0.75 gill bags. 52 per gift bag, total cost
Gaft bags required per guest
bage, per gguesttttttt 0.75
Cost per gift bag
Calculate the number of guests attending
9 xx 2 18.
allending guests- invited puests atiendance rale
Calculate the
of bugs
gift. hags needed - attending_ Bocnddiggggessssst guests gift Enggs_per_goee
Final Answer: * *Christina will spend S18 gift bags,
Calculate the sotal cost
total, cost gift. bags needed cast_per_ bag
Print the result
print(" Christina spend S (ootal cost:2f7; gift bogs.")
Python cade provided this result, confiming the caleulations
ste Outyut
you run code, the output will be:
Christina will spend $18.00 bags.
This accounts for the .75 gift bage per attending guest and the $2 cost per bug.
Figure 39; Pruning focus of AgentPrune: Redundant information.
40


Reviews:
[{"review_id": "4K6REDPhNk", "reviewer": "Reviewer_FgBo", "summary": "The paper introduces AgentPrune, a communication framework designed for LLM-MA systems. AgentPrune addresses inefficiencies in traditional multi-agent communication by reducing redundant messaging, which can lead to excessive token usage and increased costs. The framework formalizes the communication between agents as a spatial-temporal message-passing graph and optimizes its token usage without compromising task performance. Key findings include a significant reduction in token overhead and improved robustness against adversarial attacks. Through extensive testing on various benchmarks, AgentPrune demonstrated cost-effective, high-performance capabilities, achieving similar or better results compared to conventional models but at a fraction of the token cost.", "questions": "I would like to know the required quantity of data to be generated as well as the number and types of mask configurations necessary for training in Sections 3.2 and 3.3.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The connectivity optimization methods (distribution approximation and low-rank sparsity) appear highly effective. By applying a straightforward policy gradient approach and sparsity optimization on the graph masks, the framework efficiently enhances the performance and robustness across various multi-agent frameworks and tasks while reducing the token cost.", "weaknesses": "1. It appears necessary to fix the number and roles of agents before implementing AgentPrune. \n2. To effectively train each version of the framework, a substantial amount of data is required to optimize the connectivity parameters through sufficient rollouts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PrFp5IHqIE", "reviewer": "Reviewer_CryL", "summary": "This work focuses reducing communication between multi-agent LLM systems. The goal is to reduce spatial and temporal communication required between multiple LLM agents while keeping high accuracy. The authors propose learning a graph mask using distribution approximation and low-rank sparsity. The proposed method is evaluated on a wide variety of benchmark in terms of communication and cost required and the accuracy achieved.", "questions": "- What's the training time for this proposed method. Does it overfit to the training data? How does the K' affects the performance.\n- Does the mask changes depending on the input data?\n- How does the proposed method perform compared to random pruning? \n- Can distribution approximation and low-rank sparsity work in isolation? what's their effect on the accuracy and communication reducing?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work looks into an important problem of reducing cost when using multi-agent LLM system. \n- The paper is well written and relatively easy to follow.", "weaknesses": "- The paper start with showing random pruning can provide performance improvement, but the proposed method is not compared to the random pruning, so it is hard to understand how much improvement compared to random pruning we have.\n- It is not unclear how much training and query/tokens is need to train the system, and how different number of training (one-shot) is needed and how they affect the performance.\n- Missing ablation study about the benefit of each of the propose methods (i.e. distribution approximation, low-rank sparsity)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XyR9a83oob", "reviewer": "Reviewer_sHy9", "summary": "The paper studies the communication redundancy problem, i.e. the fact that when multiagent networks solve a task, several exchanged messages are redundant. To address this problem, they model communication as a spatiotemporal graph (i.e. a graph that models both which agents interact with each other and what context should be included) and introduce AgentPrune, an optimization technique that reduces the level of redundancy while preserving performance. This technique involves three steps:\n* In the \"training\" phase ($K^\\prime$ rounds), the spatiotemporal graph is optimized according to a metric that rewards assigning higher weights to important connections while at the same time increasing sparsity;\n* The connections are then pruned according to a threshold;\n* From that point onwards ($K - K^\\prime$ rounds), only the non-pruned connections are used.\n\nThe authors then study the performance of AgentPrune both in its standalone form and in conjunction with other multiagent frameworks, finding a reduction in overall cost; they also find that AgentPrune leads to improvements in robustness against agent-targeted adversarial attacks.\n\nOverall, the paper studies an important problem and proposes a reasonable (though far from perfect) approach to reduce communication redundancy. In particular, the technique requires an expensive training step; however, for large enough datasets, this leads to long-term efficiency gains. Aside from some issues that raised a few eyebrows, the experimental analysis is well done and the model also offers some robustness gains as a bonus. ~For this reason, my recommendation is 6 (Marginally above acceptance threshold).~\n\n~I would be willing to raise my score should the authors take the following steps (see the Weaknesses section):~\n* ~Presenting the results in Table 2 and 3 in a more balanced light;~\n* ~Adding the missing experimental info;~\n* ~Answering the questions reported in the Questions section (assuming that the results showcase the transferability of AgentPrune).~\n\nUpdate: the authors' rebuttal has addressed essentially all of my concerns, which is why I'm raising my score to 8 (Accept).", "questions": "* What’s the minimum scale (in terms of e.g. dataset size and number of queries) where AgentPrune is more cost-effective than other techniques?\n* Do networks trained on a task transfer to similar tasks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper does a good job of formalizing the problem, which I agree is an important one;\n* Modelling the task as a spatiotemporal graph is also a pretty solid idea, and the proposed technique can be seen as an intuitive extension of this approach;\n* The cost reduction is significant;\n* I appreciated the parameter sensitivity analysis and the ablation study, as these types of studies, while expensive, are very important for proper science.", "weaknesses": "Note: Most of these weaknesses have been addressed in the rebuttal.\n\nIt is unclear to me how applicable AgentPrune is in a real-world context. AP makes sense in situations where the deployer already has a training set for a given task and is willing to invest resources into optimizing the communication graph for future queries. In other words, using AP is reasonable from an economics point of view only at large scales, where there are enough training examples and enough expected future queries to warrant such an optimization step. To be clear, this is not a dealbreaker: several real-world applications fit this description, and my critique of AP can also be applied to any other network optimization technique (e.g. GPTSwarm). Still, it’s hard to make a case for widely applicable cost savings, especially since each graph appears to be task-specific and thus not transferable. Comparing with optimization-free baselines (e.g. CoT) is also a bit unfair, since they do not require this initial training step.\n\nMoreover, emphasizing the reduction in prompt tokens instead of the cost itself (e.g. in Table 3) is slightly misleading, since a) deployers care about the cost, not the prompt tokens b) the cost reductions, while still good, are often less impressive than the prompt token reduction (e.g. -17% cost vs -36% tokens for AG+HumanEval, -55% cost vs -78% tokens for GPTSwarm+MMLU, -52% cost vs -73% tokens for GPTSwarm+HumanEval). In the same vein, reporting in Table 2 the performance gain w.r.t. Vanilla sort of hides the fact that AgentPrune has slightly worse performance compared to Reflexion prompting (which is not a dealbreaker, but it should still be noted).\n\nFinally, there are some missing experimental details in the paper, such as:\n* What function is used as $\\phi$ in the experiments?\n* What connection probability is used to generate the random graph?\n* What’s the diversity penalty for the OpenAI models?\n\nWhile I found the answers to these questions in the code, given the non-static nature of repositories these values should also be reported in the experimental setup appendix of the code.\n\nMinor notes:\n* In Table 6 there's a typo related to the word \"ablation\";\n* Again in Table 6, the original scores have 4 significant digits, but the ones after ablation only have 3;\n* The title of Appendix F contains a misspelled \"Existing\";\n* Figure 30 has a typo in the word HumanEval;\n* In G.3 “liar\" is misspelled as “lier”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "42pf13Bwqm", "reviewer": "Reviewer_rYs7", "summary": "The paper introduces AgentPrune, a framework aimed at optimizing communication in LLM-based multi-agent systems by reducing unnecessary message exchanges, which in turn lowers token usage and economic costs. In MAS, agents collaborate through complex communication topologies, which, while enhancing collective intelligence, can lead to significant token overhead and high operational expenses.\n\nAgentPrune addresses this challenge by introducing a \"communication pruning\" mechanism, where redundant or non-essential messages within the spatial (intra-dialogue) and temporal (inter-dialogue) message-passing graph are pruned. \n\nExperiments demonstrate AgentPrune's effectiveness across several benchmarks, including MMLU, GSM8K, and HumanEval. Experimental results indicate that AgentPrune achieves comparable performance to existing MAS topologies at a fraction of the cost, with a reported token reduction of up to 72.8% and an increased robustness to basic adversarial attacks by 3.5% to 10.8%", "questions": "Regarding point 2: If you could include experiments that require high contextual awareness, demonstrating that your model can improve both quality and token efficiency in such scenarios, I would be happy to raise my evaluation score.\n\nAnd I would appreciate a deeper analysis of the types of errors or ineffective communications that AgentPrune eliminates, as this would clarify the practical implications of its pruning method. Such insights would better illustrate AgentPrune’s strengths and limitations and help inform users of any potential side effects.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. It addresses an important problem in MAS involving LLMs: the significant economic costs associated with token usage. By focusing on token efficiency, it highlights an often-overlooked aspect of MAS design, especially relevant as these systems scale up. The emphasis on reducing token consumption could encourage future MAS research to adopt similar cost-effective approaches, making it a useful reference for economical MAS deployment.\n\n2. It borrows idea from the spatial-temporal graphs to manage agent communication provides a structured way to visualize and optimize message passing in MAS. This graph-based approach brings an analytical perspective to inter-agent communication, offering insights into which interactions can be minimized without sacrificing performance. Such structured pruning could inspire other MAS research to explore graph-based techniques for optimizing agent communication.\n\n3. It formally define \"communication redundancy\" in LLM-powered MAS, bringing attention to the often excessive and unnecessary communications within these systems. By identifying this redundancy, the authors shed light on a specific inefficiency that had not been well-formalized in previous MAS research, thus contributing a valuable perspective to MAS optimization.", "weaknesses": "1. Inadequate Justification for Pruning Methodology: The one-shot pruning approach, though efficient, lacks theoretical grounding in the context of MAS. The decision to prune a fixed percentage of tokens and communication paths across all tasks does not account for the varying communication needs of different agents or tasks. This oversimplified pruning criterion may lead to suboptimal performance in scenarios where more dynamic and context-sensitive pruning is required.\n\n2. The paper provides little analysis of how pruning affects the quality and depth of agent interactions or task accuracy, especially in more nuanced MAS tasks. **For instance, the pruning’s impact on collaborative tasks requiring high contextual awareness is underexplored.**\n\n3. The number of colors and the format of text in Figure 1 is excessive, and the text is too lengthy, making the annotations unclear and distracting from the overall understanding of the figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)