G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systems
Authors: Shilong Wang, Guibin Zhang, Miao Yu, Guancheng Wan, Fanci Meng, chongye guo, Kun Wang, Yang Wang
Accepted at: acl25

G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systems

Shilong Wang*†  Guibin Zhang*†  Miao Yu*  Guancheng Wan*  Fanci Meng*
Chongye Guo*  Kun Wang**  Yang Wang***
*USTC  **NUS  *UCLA  *Shanghai University  *NTU
wk520529@mail.ustc.edu.cn  angyan@ustc.edu.cn

Abstract
Large Language Model (LLM)-based Multi-agent Systems (MAS) have demonstrated remarkable capabilities in various complex tasks, ranging from collaborative problem-solving to autonomous decision-making. However, as these systems become increasingly integrated into critical applications, their vulnerability to adversarial attacks, misinformation propagation, and unintended behaviors has raised significant concerns. To address this challenge, we introduce G-Safeguard, a topology-guided security lens and treatment for robust LLM-MAS, which leverages graph neural networks to detect anomalies on the multi-agent utterance graph and employs topological intervention for attack remediation. Extensive experiments demonstrate that G-Safeguard: (I) exhibits significant effectiveness under various attack strategies, recovering over 40% of the performance for prompt injection; (II) is highly adaptable to diverse LLM backbones and large-scale MAS; (III) can seamlessly combine with mainstream MAS with security guarantees. The code is available at https://github.com/wslong20/G-safeguard.

1 Introduction
Autonomous agents (Wang et al., 2024), while inheriting the general task-solving and instruction comprehension capabilities of Large Language Models (LLMs) (Chang et al., 2024; Minaee et al., 2024), are equipped with external units such as tools (Liu et al., 2024b; Tang et al., 2023) and memory (Zhang et al., 2024h), extending the capability boundaries of LLMs. Multi-agent systems (MAS) further integrate collective intelligence through agent interactions, enhancing the capabilities of individual agents. This enables MAS to be recognized as intelligent entities capable of tackling more complex tasks, such as environment perception and embodied actions (Guo et al., 2024; Zhao et al., 2025). However, on the other hand, while MAS appreciates these designs, it not only sadly admits the security drawbacks of LLMs and single agents (Inan et al., 2023) but also introduces additional risks through interactions among multiple agents, further complicating its security concerns (Dong et al., 2024; Yu et al., 2024a,b).

Existing attacks on agents primarily target their external units (e.g., tool, memory) (Tian et al., 2023; Zhang et al., 2024g) and main body (Liu et al., 2023a). For single-agent, direct prompt injection (Perez and Ribeiro, 2022; Kang et al., 2024; Liu et al., 2023a; Toyer et al., 2023) manipulates agents by embedding harmful knowledge or biases into their foundation LLM, influencing decision-making and generating harmful responses. Indirect prompt injection exposes agents to a vast amount of potentially harmful information when interacting with external interfaces (Greshake et al., 2023; Zhan et al., 2024; Yi et al., 2023), indirectly enabling them to acquire malicious instructions. Without custom designs or any modifications, MAS falls into the predicament of single agents and introduces communication as a new point of vulnerability (Yu et al., 2024b; Tian et al., 2023). For example, NetSafe (Yu et al., 2024b) take the first step to identify the existence of topology-based misinformation and bias propagation.

![The paradigm comparison between single agent safeguard and multi-agent safeguard.](page_1012_370_388_246.png)

Figure 1: The paradigm comparison between single agent safeguard and multi-agent safeguard.
Compared to attacks, defense in MAS becomes significantly more challenging, encompassing both anomalies detection and remediation (Andriushchenko et al., 2024), also shown in Figure 1. Detection focuses on diagnosing the location of the issue, while remediation involves mitigating the negative impact. Unfortunately, the literature primarily focuses on designing customized defenses for attacks targeting specific units, overlooking the features of MAS:

★ Topology-Aware. Existing defenses primarily focus on single-agent, neglecting the critical topology characteristics inherent in MAS (Zhang et al., 2024b,c). Due to the interaction mechanisms, anomaly detection must take into account the relationships and behaviors of neighboring agents to effectively address the unique challenges posed by the interconnected nature.

★ Inductive transferability. When combining the number of agents with the design of external units, MAS can exhibit an extensive variety of combinations and configurations. Custom-designed solutions hinder transferability (Fu et al., 2024; Hua et al., 2024), which even unnerves small-size MAS, let alone large-size MAS-based applications (Zhang et al., 2024e).

To tackle these challenges, we propose G-Safeguard, a topology-guided lens for attack detection and treatment for attack remediation. Technically, G-Safeguard performs security assessments at the conclusion of each dialogue round within a multi-turn MAS. It first constructs a multi-agent utterance graph, which sufficiently encodes both agent-wise communicative interactions and topological dependencies. Building upon this foundation, G-Safeguard formulates the attack detection as an anomaly detection problem on this graph, leveraging an edge-featured graph neural network (GNN) to pinpoint high-risk agents. Finally, G-Safeguard enforces topological interventions to disrupt the propagation of misleading or adversarial information, thereby materializing robust MAS against a broad spectrum of agent-oriented attacks, i.e., prompt injection, memory poisoning, and tool attacks. More importantly, inheriting the inductive ability of GNNs, G-Safeguard can scale to arbitrary-scale MAS without resource-intensive retraining, meanwhile exhibiting remarkable cross-LLM-backbone generalizability.

We conduct extensive experiments to validate our method’s effectiveness: G-Safeguard (I) prevents malicious information spread across various MAS topologies, blocking 12.50% ~ 33.23% of infections in chain structures and 10% ~ 38.52% in star structures on the MMLU dataset; (II) defends against multiple attacks, reducing attack success rate (ASR) by 21.38% and 22.01% for prompt injection on CSQA and MMLU, 12.67% for tool attack, and 16.27% for memory poison; (III) scales seamlessly to large-scale MAS, maintaining stable performance with 19.50% ~ 39.23% ASR reduction under prompt injection settings.

Our contributions can be concluded as follows:
• Paradigm Proposal. We pioneer the detection and remediation paradigm for adversarial defense within LLM-MAS, which emphasizes topology-aware diagnosis and intervention of misleading or malicious information as it propagates and proliferates across the multi-agent system.
• Practical Solution. We introduce G-Safeguard, a topology-guided framework for attack detection and remediation, enabling lightweight and real-time identification of adversarial entities on multi-agent utterance graphs and contamination-free communication via graph pruning.
• Empirical Evaluation. The results across various LLM backbones, MAS frameworks, and attack strategies show that G-Safeguard provides effective protection in all these scenarios. Also, G-Safeguard is adaptable to arbitrary-scale MAS and can integrate into mainstream MAS to enhance their defensive capabilities.

2 Preliminary

In this section, we establish the notation and formalize key concepts for the attack and defense of multi-agent systems from a topology perspective.

Multi-agent System. Consider a multi-agent system composed of \( N = |\mathcal{V}| \) agents, which we conceptualize as a graph \( \mathcal{G} = (\mathcal{V}, \mathcal{E}) \), where \( \mathcal{V} = \{C_1, \ldots, C_N\} \) denotes the agent (node) set, while the edge set \( \mathcal{E} = \mathcal{V} \times \mathcal{V} \) encodes their connectivity. Each agent \( C_i \in \mathcal{V} \) is characterized as:

\[
C_i = \{\text{Base}_i, \text{Role}_i, \text{Mem}_i, \text{Plugin}_i\},
\]

where (1) Base_i denotes the underlying LLM instance; (2) Role_i is a functional role or persona; (3) Mem_i denotes the memory of \( C_i \), generally encapsulating its previous interactions and external knowledge; and (4) Plugin_i represents a repertoire of external tools augmenting its operational reach, like web search engines and document parsers.
Figure 2: The designing workflow of our proposed G-Safeguard. (a): Input example and attack methods. (b): The MAS starts with a small fraction of compromised agents, while the other agents remain benign. As multi-turn interactions progress, the majority of agents in the MAS eventually become infected by the attacker agents. (c): We take attacker detection in the second-round dialogue as an example. Specifically, for each agent node, we encode the previous and current interactions with adjacent agents into node and edge features, constructing a multi-agent utterance graph.

Execution Logic. Given a user query \( Q \), the multi-agent system engages in \( K \) iterative rounds of dialogues, converging upon the final solution \( a^{(K)} \). At the onset of the \( t \)-th dialogue round, an ordering function \( \phi \) is applied to orchestrate the execution sequence of agents:

\[
\phi : \mathcal{G} \mapsto \sigma, \quad \sigma = [v_{\sigma_1}, v_{\sigma_2}, \ldots, v_{\sigma_N}],
\]
\[
\text{s.t.} \quad \forall i > j, \quad v_{\sigma_i} \notin \mathcal{N}_{\text{in}}(v_{\sigma_j}),
\]

where \( \sigma \) dictates the order of agent activations. Equation (2) ensures that any agent \( v_{\sigma(i)} \) can only execute after all agents in its in-neighborhood \( \mathcal{N}_{\text{in}}(v_{\sigma(j)}) \) have completed their operations. Following the prescribed execution sequence, each agent sequentially processes inputs and generates outputs as follows:

\[
\mathbf{R}_i^{(t)} = C_i(\mathcal{P}_{\text{sys}}^{(t)}, \{q, \cup_{j \in \mathcal{N}_{\text{in}}(C_i)} \mathbf{R}_j^{(t)}\})
\]

where \( \mathbf{R}_i^{(t)} \) denotes the output of agent \( C_i \), which may manifest as a rationale, an intermediate result, or a direct solution. This output is derived from two key components: the system prompt \( \mathcal{P}_{\text{sys}}^{(t)} \), which encodes global instructions like Role_i, and the real-time context, integrating the query \( q \) and insights from preceding agents.

At the conclusion of each dialogue round, an aggregation function \( \mathcal{A}(\cdot) \) is employed to synthesize the final solution or answer \( a^{(t)} \):

\[
a^{(t)} \leftarrow \mathcal{A}(\mathbf{R}_1^{(t)}, \mathbf{R}_2^{(t)}, \ldots, \mathbf{R}_N^{(t)}).
\]

The possible implementations of \( \mathcal{A}(\cdot) \) include majority voting (Zhuge et al., 2024), agent-based summarization (Wu et al., 2023; Zhang et al., 2024b), and directly utilizing the final output \( \mathbf{R}_{\sigma_N}^{(t)} \) as answer (Qian et al., 2024). Through \( K \) rounds of iterative interaction, whether predefined (Qian et al., 2024) or early-stopped (Liu et al., 2023b), the system \( \mathcal{G} \) outputs the final solution \( a^{(K)} \) in response to the query \( Q \).

MAS Attack. Multi-agent systems (MAS) are susceptible to adversarial interventions at multiple levels, including prompt manipulation, memory corruption, and tool exploitation. These attacks can distort agent outputs, leading to biases, misinformation, or operational failures. \( \bullet \) Prompt Injection, either direct or indirect (Greshake et al., 2023), involves manipulating the system prompt \( \mathcal{P}_{\text{sys}} \) of part of agents in \( \mathcal{G} \) by injecting adversarial content, leading to degraded system performance, adopted by ASB (Zhang et al., 2024d) and NetSafe (Yu et al., 2024b); \( \bullet \) Memory Poison refers to attacking the Mem component of agents, including injecting false conversational records (Nazary et al., 2025) and poisoning external databases (Chen et al., 2024); For \( \bullet \) Tool Attack, external tools (Plugin_i) expand an agent’s capabilities but can also be leveraged for malicious intent like data stealing and user harm (Zhan et al., 2024). We denote the attacked MAS as \( \tilde{\mathcal{G}} \), wherein the set of attacked agents is denoted as \( \mathcal{V}_{\text{atk}} \subseteq \mathcal{V} \).
Defense. To safeguard the system, we define two core defense objectives: ■ Attack Detection is to accurately identify the attacked agent set \( \mathcal{V}_{\text{atk}} \):

\[
\arg\max_{\mathcal{V}'} \mathcal{D}(\mathcal{V}' = \mathcal{V}_{\text{atk}} \mid \{\{\mathbf{R}_i^{(t)}\}_{i=1,t=1}^{N,K}\}),
\]

where \( \mathcal{D}(\cdot|\cdot) \) is an attack detector that computes the posterior probability over the attack set given observable system behaviors. ■ Attack Remediation. Once \( \mathcal{V}_{\text{atk}} \) is identified, an attack remediation \( \mathcal{R} \) is leveraged to minimize the negative impact of compromised agents:

\[
\min_{\mathcal{G}'} \text{Difference}(\mathcal{R}(\tilde{\mathcal{G}}), \mathcal{G}),
\]

where Difference(\(\cdot,\cdot\)) quantifies the differences between MAS with regard to utility, safety, cost, etc.

3 Methodology

Figure 2 illustrates (a) various attack strategies targeting MAS, (b) how attacks propagate across the multi-agent network, and (c) how G-Safeguard dynamically identifies both *attacked* or *infectious* agents within the network, executing timely interventions. Specifically, in MAS, one or more agents may carry misleading or malicious information due to prompt injection, memory poisoning, or tool attacks, which, over multiple rounds of dialogue, spread and contaminate the entire system. In response to this, G-Safeguard collects the outputs of agents from previous dialogue, along with their topological connectivity, to construct a multi-agent utterance graph (> Section 3.1). A graph neural network is then employed to materialize **attack detection**, namely identifying anomalous agents and toxic information flows (> Section 3.2). Furthermore, edge sparsification is utilized for **attack remediation**, mitigating the attack’s impact and ensuring contamination-free inter-agent communication (> Section 3.3).

3.1 Multi-agent Utterance Graph

To enable real-time monitoring of agent attacks, G-Safeguard constructs a *multi-agent utterance graph*, capturing both agent-wise discourse dynamics and their topologies. Formally, let \( \mathcal{M}^{(t)} = (\mathbf{X}^{(t)}, \mathbf{E}^{(t)}) \) denote the utterance graph for utterance round \( t \), where \( \mathbf{X}^{(t)} \in \mathbb{R}^{N \times D} \) and \( \mathbf{E}^{(t)} \in \mathbb{R}^{E^{(t)} \times D} \) denotes the node and edge embeddings respectively, and \( E^{(t)} \) denotes the number of edges in round \( t \). Each agent (node) \( C_i \in \mathcal{V} \) is associated with a node representation \( \mathbf{h}_i^{(t)} \), derived from the agent’s historical records via a text embedding function \( \mathcal{T} : \mathbb{T} \to \mathbb{R}^D \), *i.e.,*

\[
\mathbf{h}_i^{(t)} := \mathbf{X}_i^{(t)} = \mathcal{T}(\mathbf{R}_i^{(t)}, \bigcup_{k=1}^{t-1} \mathbf{R}_i^{(k)}),
\]

where \( \mathbb{T} \) denotes the space of textual utterances and \( D \) is the embedding dimension. We instantiate \( \mathcal{T}(\cdot) \) with text embedding models like MiniLM (Wang et al., 2020). Each edge \( e_{ij}^{(t)} \in \mathcal{E}^{(t)} \) encodes the interaction history between agents \( C_i \) and \( C_j \), namely \( [\mathbf{R}_{i \to j}^{(1)}, \cdots, \mathbf{R}_{i \to j}^{(K)}] \), where \( K \leq t \) denotes the occurring times of interaction \( C_i \to C_j \) and \( \mathbf{R}_{i \to j}^{(t)} \) represents the utterance transmitted from \( C_i \) to \( C_j \) at round \( t \). Note that since \( \mathcal{E}^{(t)} \) can be time-varying in many adaptive multi-agent pipelines (Chen et al., 2023b; Ishibashi and Nishimura, 2024), \( K \) does not necessarily coincide with \( t \). To ensure G-Safeguard’s generality, we employ a learnable permutation-invariant fusion function \( \mathcal{F} : \mathbb{T}^K \to \mathbb{R}^D \) to distill the interaction history into a fixed-dimensional representation:

\[
\mathbf{e}_{ij}^{(t)} = \mathcal{F}\left([\mathcal{T}(\mathbf{R}_{i \to j}^{(1)}), \ldots, \mathcal{T}(\mathbf{R}_{i \to j}^{(K)})]\right).
\]

Based on this, the edge embeddings \( \mathbf{E}^{(t)} \) can be expressed as \( \mathbf{E}^{(t)} = [\mathbf{e}_{ij}^{(t)}]_{e_{ij}^{(t)} \in \mathcal{E}^{(t)}} \). Upon constructing \( \mathcal{M}^{(t)} \), which sufficiently encodes each agent’s utterance dynamics and inter-agent discourse, we will detail how G-Safeguard formalizes multi-agent adversarial detection as a *node classification* problem on \( \mathcal{M}^{(t)} \) in the next section.

3.2 Graph-based Attack Detection

At the end of interaction round \( t \), G-Safeguard first seeks to identify the set of attacked agents \( \mathcal{V}_{\text{atk}}^{(t)} \subseteq \mathcal{V} \). Notably, while adversarial attacks often initially affect only a small subset of agents \( \mathcal{V}_{\text{atk}}^{(0)} \), their misleading or harmful utterances propagate through the system (Yu et al., 2024b), leading to a cascading effect that corrupts a broader subset of agents, *i.e.*, \( \mathcal{V}_{\text{atk}}^{(0)} \subseteq \mathcal{V}_{\text{atk}}^{(1)} \subseteq \cdots \subseteq \mathcal{V}_{\text{atk}}^{(t)} \). This underscores the necessity of per-utterance attack detection, a fundamental principle in our design.

G-Safeguard formalizes attack detection in MAS as a node classification problem on \( \mathcal{M}^{(t)} \). Thus, GNN has become a natural solution owing to its tremendous success in classification tasks (Wu et al., 2020; Zhang et al., 2024a). Specifically, to propagate both structural and semantic dependencies, we iteratively update the node representations
through an \( L \)-layer GNN:

\[
h_i^{(t,l)} = \text{COMB}\left(\mathbf{h}_i^{(t,l-1)}, \text{AGGR}\{\psi(\mathbf{h}_j^{(t,l-1)}, e_{ij}^{(t)}) : C_j \in \mathcal{N}_{in}^{(t)}(C_i)\}\right), \quad 0 \leq l \leq L,
\]

where \( \mathcal{N}_{in}^{(t)}(C_i) \) denotes the set of neighboring agents, and \( \psi : \mathbb{R}^D \times \mathbb{R}^D \to \mathbb{R}^D \) is a transformation mechanism that encodes edge-aware neighbor information, for which we follow (Chen and Chen, 2021). AGGR(\cdot) and COMB(\cdot) represent aggregating neighborhood information and combining ego- and neighbor-representations.

Unlike conventional agent safeguard methods such as LlamaGuard (Inan et al., 2023) and WildGuard (Han et al., 2024), which operate solely at the input-output level for individual agents, G-Safeguard is featured with inter-agent information flow perception, thereby facilitating topology-aware attack detection. Upon message propagation, each agent \( C_i \) is assigned an attack probability via a soft probabilistic classifier:

\[
p(C_i \in \mathcal{V}_{atk}^{(t)} \mid h_i^{(t,L)}) = \sigma \left(f_\theta(h_i^{(t,L)})\right),
\]

where \( f_\theta : \mathbb{R}^D \to \mathbb{R} \) is a learnable scoring function, and \( \sigma(\cdot) \) denotes the sigmoid activation. We denote the set of risky agents identified by G-Safeguard at round \( t \) as \( \tilde{\mathcal{V}}_{atk}^{(t)} \). Moving forward, G-Safeguard intervenes on these high-risk nodes to mitigate their detrimental impact.

3.3 Edge Pruning for Remediation

Given \( \tilde{\mathcal{V}}_{atk}^{(t)} \), G-Safeguard institutes a topological intervention by excising their outgoing edges. Formally, we redefine the next round’s interaction topology as follows:

\[
\mathcal{E}^{(t+1)} \leftarrow \mathcal{E}^{(t+1)} \setminus \bigcup_{C_i \in \tilde{\mathcal{V}}_{atk}^{(t)}} \{e_{ij}^{(t)} \mid C_j \in \mathcal{V}\}.
\]

This targeted edge pruning effectively suppresses the propagation of adversarial messages.

Beyond topological intervention, remediation strategies can be highly customizable per user request. For instance, filtering mechanisms, such as AWS Bedrock (Amazon, 2025), may be deployed to sanitize the content generated by compromised agents, or precautionary alerts can be issued to users, proactively mitigating potential harm.

3.4 Optimization

To optimize G-Safeguard for attack detection, we employ a cross-entropy loss function, formulated as the expected negative log-likelihood over the attack labels:

\[
\mathcal{L} = -\mathbb{E}_{C_i \sim \mathcal{V}, t \sim [1,K]} \Big[ y_i \log p(C_i \in \mathcal{V}_{atk}^{(t)} \mid h_i^{(t,L)}) 
+ (1-y_i) \log \left(1 - p(C_i \in \mathcal{V}_{atk}^{(t)} \mid h_i^{(t,L)})\right) \Big],
\]

where \( y_i \in \{0, 1\} \) is the ground-truth attack label for agent \( C_i \). Equation (12) effectively guides G-Safeguard to discern adversarial agents with high fidelity.

4 Experiment

In this section, we conduct extensive experiments to answer the research questions:

■ (RQ1) Can G-Safeguard detect and defend malicious agents under various attacks?
■ (RQ2) Does G-Safeguard have the inductiveness and transferability, enabling it to be easily integrated into MAS of different scales?
■ (RQ3) Can G-Safeguard be migrated to real-world MAS applications to guarantee safety?

4.1 Experiment Setup
Dataset Construction. We perform MAS interaction across various attack scenarios for verifying the defense ability of G-Safeguard. Concretely, we first generate different structures under edge density \( D_e = \{0.2, 0.4, 0.6, 0.8, 1.0\} \). When \( D_e = 1 \), it corresponds to a complete graph. We consider three type of attacks. ① Direct prompt attack. We sample questions from the CSQA (Talmor et al., 2018), MMLU (Hendrycks et al., 2020), and GSM8K (Cobbe et al., 2021) datasets respectively. Then, we randomly select 800 samples after combining these questions with the structures. Based on these samples, we construct MAS communication data, setting the number of communication rounds to \( K \) and the number of agents to \( N \) (marked in specific part). During this process, we assign the roles of certain agents as attackers, whose responsibility is to inject misinformation into others. ② Tool attack. We construct communication data based on the InjecAgent dataset (Zhan et al., 2024). First, we extract cases that fail to attack GPT-4o-mini. Similarly to ①, we construct adversarial scenarios in MAS. ③ Memory attack. We employ the configuration from PoisonRAG (Nazary et al., 2025), inserting erroneous messages into the contextual memory of the attacker agent, enabling them to disseminate conclusions derived from these messages to others.
<table>
  <tr>
    <th rowspan="2">Topology</th>
    <th rowspan="2">Dataset</th>
    <th colspan="4">PI (CSQA)</th>
    <th colspan="4">PI (MMLU)</th>
    <th colspan="4">PI (GSM8K)</th>
    <th colspan="4">TA (InjectAgent)</th>
    <th colspan="4">MA (PostionRAG)</th>
  </tr>
  <tr>
    <th>Model</th>
    <th>R0</th>
    <th>R3-R5</th>
    <th>R3-R5</th>
    <th>R0</th>
    <th>R3-R5</th>
    <th>R0</th>
    <th>R3-R5</th>
    <th>R0</th>
    <th>R3-R5</th>
    <th>R0</th>
    <th>R3-R5</th>
    <th>R0</th>
    <th>R3-R5</th>
    <th>R0</th>
    <th>R3-R5</th>
    <th>R0</th>
    <th>R3-R5</th>
    <th>R0</th>
    <th>R3-R5</th>
    <th>R0</th>
    <th>R3-R5</th>
  </tr>
  <tr>
    <td rowspan="6">Chain</td>
    <td>GPT-4o-mini</td>
    <td>29.06</td>
    <td>45.03/27.50</td>
    <td>8.60</td>
    <td>19.59</td>
    <td>34.46/21.96</td>
    <td>12.50</td>
    <td>11.25</td>
    <td>15.24/9.58</td>
    <td>5.60</td>
    <td>3.07</td>
    <td>36.52/29.96</td>
    <td>23.58</td>
    <td>8.78</td>
    <td>18.13/9.05</td>
    <td>10.58</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>22.00</td>
    <td>34.17/23.33</td>
    <td>8.58</td>
    <td>15.00</td>
    <td>27.33/14.09</td>
    <td>13.24</td>
    <td>14.16</td>
    <td>10.83/10.00</td>
    <td>8.33</td>
    <td>5.00</td>
    <td>16.15/5.38</td>
    <td>10.77</td>
    <td>8.78</td>
    <td>16.38/11.70</td>
    <td>16.68</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70B</td>
    <td>27.40</td>
    <td>52.22/35.33</td>
    <td>8.80</td>
    <td>19.41</td>
    <td>43.68/19.34</td>
    <td>24.34</td>
    <td>13.40</td>
    <td>11.74/5.55</td>
    <td>6.19</td>
    <td>50.00</td>
    <td>69.61/60.77</td>
    <td>8.84</td>
    <td>8.19</td>
    <td>49.14/41.62</td>
    <td>26.32</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>26.25</td>
    <td>50.00/28.84</td>
    <td>8.16</td>
    <td>18.80</td>
    <td>38.00/15.00</td>
    <td>23.00</td>
    <td>6.67</td>
    <td>7.08/2.67</td>
    <td>0.81</td>
    <td>5.83</td>
    <td>28.83/29.16</td>
    <td>25.33</td>
    <td>11.01</td>
    <td>50.88/36.60</td>
    <td>12.08</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>23.75</td>
    <td>55.21/31.25</td>
    <td>8.96</td>
    <td>16.36</td>
    <td>43.68/10.45</td>
    <td>8.33</td>
    <td>10.00</td>
    <td>8.75/2.27</td>
    <td>2.15</td>
    <td>24.67/42.67</td>
    <td>8.19</td>
    <td>29.83/16.90</td>
    <td>12.57</td>
  </tr>
  <tr>
    <td rowspan="6">Tree</td>
    <td>GPT-4o-mini</td>
    <td>29.06</td>
    <td>45.31/31.87</td>
    <td>8.34</td>
    <td>18.88</td>
    <td>29.72/18.53</td>
    <td>20.79</td>
    <td>12.5</td>
    <td>16.66/9.58</td>
    <td>7.08</td>
    <td>4.16</td>
    <td>47.5/4.16</td>
    <td>43.34</td>
    <td>8.19</td>
    <td>18.72/10.36</td>
    <td>9.36</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>18.66</td>
    <td>34.00/24.66</td>
    <td>9.34</td>
    <td>10.56</td>
    <td>18.31/11.26</td>
    <td>7.07</td>
    <td>7.91</td>
    <td>6.25/1.66</td>
    <td>0.00</td>
    <td>13.11/1.67</td>
    <td>10.43</td>
    <td>8.19</td>
    <td>19.89/10.53</td>
    <td>9.36</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70B</td>
    <td>27.40</td>
    <td>52.22/35.33</td>
    <td>8.80</td>
    <td>19.41</td>
    <td>43.68/19.34</td>
    <td>24.34</td>
    <td>13.40</td>
    <td>11.74/5.55</td>
    <td>6.19</td>
    <td>50.00</td>
    <td>69.61/60.77</td>
    <td>8.84</td>
    <td>8.19</td>
    <td>49.14/41.62</td>
    <td>26.32</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>28.70</td>
    <td>42.95/29.74</td>
    <td>8.12</td>
    <td>22.00</td>
    <td>46.67/25.33</td>
    <td>21.34</td>
    <td>7.08</td>
    <td>6.67/7.08</td>
    <td>4.10</td>
    <td>4.31</td>
    <td>25.86/29.31</td>
    <td>13.45</td>
    <td>36.26/26.32</td>
    <td>20.81</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>24.68</td>
    <td>63.43/28.75</td>
    <td>8.56</td>
    <td>7.00</td>
    <td>31.33/8.02</td>
    <td>23.37</td>
    <td>7.08</td>
    <td>10.42/7.08</td>
    <td>3.34</td>
    <td>36.84</td>
    <td>47.37/50.87</td>
    <td>5.50</td>
    <td>8.78</td>
    <td>38.60/15.79</td>
    <td>22.81</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>29.06</td>
    <td>48.75/29.06</td>
    <td>9.00</td>
    <td>18.67</td>
    <td>30.00/20.00</td>
    <td>12.91</td>
    <td>19.58/9.58</td>
    <td>10.00</td>
    <td>2.67</td>
    <td>40.18/3.57</td>
    <td>36.61</td>
    <td>10.48</td>
    <td>13.81/11.43</td>
    <td>14.00</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>28.57</td>
    <td>49.05/29.06</td>
    <td>8.90</td>
    <td>7.50</td>
    <td>20.8/8.33</td>
    <td>9.47</td>
    <td>10.59</td>
    <td>7.20/7.20</td>
    <td>0.00</td>
    <td>0.83</td>
    <td>6.67/0.83</td>
    <td>5.84</td>
    <td>8.78</td>
    <td>22.81/8.77</td>
    <td>11.04</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70B</td>
    <td>31.93</td>
    <td>56.54/34.01</td>
    <td>8.63</td>
    <td>15.67</td>
    <td>42.61/20.13</td>
    <td>22.48</td>
    <td>7.76</td>
    <td>9.38/4.26</td>
    <td>5.12</td>
    <td>49.14</td>
    <td>70.69/49.14</td>
    <td>8.54</td>
    <td>50.61/20.22</td>
    <td>20.39</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>25.97</td>
    <td>56.87/30.25</td>
    <td>8.62</td>
    <td>20.61</td>
    <td>43.24/19.58</td>
    <td>23.66</td>
    <td>6.25</td>
    <td>5.83/5.00</td>
    <td>0.83</td>
    <td>6.67</td>
    <td>16.67/25.00</td>
    <td>8.33</td>
    <td>11.70</td>
    <td>47.20/36.10</td>
    <td>11.04</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>24.88</td>
    <td>76.25/28.27</td>
    <td>8.47</td>
    <td>10.41</td>
    <td>49.33/15.26</td>
    <td>17.86</td>
    <td>7.08</td>
    <td>10.42/7.08</td>
    <td>3.34</td>
    <td>36.84</td>
    <td>47.37/50.87</td>
    <td>5.50</td>
    <td>8.78</td>
    <td>38.60/15.79</td>
    <td>22.81</td>
  </tr>
  <tr>
    <td rowspan="6">Star</td>
    <td>GPT-4o-mini</td>
    <td>25.75</td>
    <td>54.23/29.77</td>
    <td>8.27</td>
    <td>18.98</td>
    <td>38.87/20.27</td>
    <td>17.25</td>
    <td>11.72/11.67</td>
    <td>7.33</td>
    <td>3.33</td>
    <td>20.16/3.33</td>
    <td>18.19</td>
    <td>14.02/11.11</td>
    <td>17.01</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>20.00</td>
    <td>44.06/21.56</td>
    <td>7.26</td>
    <td>14.63</td>
    <td>29.68/24.54</td>
    <td>9.32</td>
    <td>7.63/5.08</td>
    <td>2.65</td>
    <td>0.83</td>
    <td>3.33/4.16</td>
    <td>7.02</td>
    <td>16.38/11.70</td>
    <td>16.68</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70B</td>
    <td>26.24</td>
    <td>53.95/36.75</td>
    <td>8.56</td>
    <td>18.77</td>
    <td>51.35/15.38</td>
    <td>15.97</td>
    <td>12.91</td>
    <td>7.11/3.62</td>
    <td>1.49</td>
    <td>48.33</td>
    <td>65.00/53.33</td>
    <td>11.67</td>
    <td>10.47</td>
    <td>44.66/19.77</td>
    <td>27.67</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>26.62</td>
    <td>41.14/27.15</td>
    <td>8.99</td>
    <td>23.33</td>
    <td>49.33/23.36</td>
    <td>26.00</td>
    <td>7.08</td>
    <td>10.33/7.58</td>
    <td>5.83</td>
    <td>5.00</td>
    <td>17.5/21.46</td>
    <td>9.66</td>
    <td>9.95</td>
    <td>41.53/30.17</td>
    <td>11.36</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>23.75</td>
    <td>76.25/28.27</td>
    <td>8.47</td>
    <td>9.37</td>
    <td>45.71/15.11</td>
    <td>9.00</td>
    <td>7.91/7.51</td>
    <td>4.14</td>
    <td>24.16</td>
    <td>50.00/26.67</td>
    <td>12.03</td>
    <td>13.25</td>
    <td>31.32/12.09</td>
    <td>19.27</td>
  </tr>
  <tr>
    <td rowspan="6">Random</td>
    <td>GPT-4o-mini</td>
    <td>25.75</td>
    <td>54.23/29.77</td>
    <td>8.27</td>
    <td>18.98</td>
    <td>38.87/20.27</td>
    <td>17.25</td>
    <td>11.72/11.67</td>
    <td>7.33</td>
    <td>3.33</td>
    <td>20.16/3.33</td>
    <td>18.19</td>
    <td>14.02/11.11</td>
    <td>17.01</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>20.00</td>
    <td>44.06/21.56</td>
    <td>7.26</td>
    <td>14.63</td>
    <td>29.68/24.54</td>
    <td>9.32</td>
    <td>7.63/5.08</td>
    <td>2.65</td>
    <td>0.83</td>
    <td>3.33/4.16</td>
    <td>7.02</td>
    <td>16.38/11.70</td>
    <td>16.68</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70B</td>
    <td>26.24</td>
    <td>53.95/36.75</td>
    <td>8.56</td>
    <td>18.77</td>
    <td>51.35/15.38</td>
    <td>15.97</td>
    <td>12.91</td>
    <td>7.11/3.62</td>
    <td>1.49</td>
    <td>48.33</td>
    <td>65.00/53.33</td>
    <td>11.67</td>
    <td>10.47</td>
    <td>44.66/19.77</td>
    <td>27.67</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>26.62</td>
    <td>41.14/27.15</td>
    <td>8.99</td>
    <td>23.33</td>
    <td>49.33/23.36</td>
    <td>26.00</td>
    <td>7.08</td>
    <td>10.33/7.58</td>
    <td>5.83</td>
    <td>5.00</td>
    <td>17.5/21.46</td>
    <td>9.66</td>
    <td>9.95</td>
    <td>41.53/30.17</td>
    <td>11.36</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>23.75</td>
    <td>76.25/28.27</td>
    <td>8.47</td>
    <td>9.37</td>
    <td>45.71/15.11</td>
    <td>9.00</td>
    <td>7.91/7.51</td>
    <td>4.14</td>
    <td>24.16</td>
    <td>50.00/26.67</td>
    <td>12.03</td>
    <td>13.25</td>
    <td>31.32/12.09</td>
    <td>19.27</td>
  </tr>
</table>

Table 1: Attack success rate (ASR$^{\dagger}$) of different LLMs under our attack settings. We consider three types attack: Prompt injection, tool attack and memory attack. “PI” denotes Prompt Injection, “TA” denotes Tool Attack, “MA” denotes Memory Attack. “GS” represents G-Safeguard model. We showcase results after round 3 communications and the additional results are placed in Appendix A.

The other Settings are the same as ① and ②. In the three attack scenarios of the main experiment, we set N to 8 and K to 3. After obtaining the communication data, we then transfer them to graphs for training G-Safeguard. Specifically, we adapt SentenceBERT (Reimers, 2019) to process the textual messages in the communication data for generating embeddings, which will serve as edges for constructing graphs. For detailed prompts, please refer to Appendix C.

Experiment Settings. We evaluate the defense effectiveness of G-Safeguard under diverse attack methods, various topological structures, and different LLMs. Specifically, in terms of attack methods, we employ direct prompt, tools, and memory attack to disrupt the MAS, thereby verifying the broad applicability of G-Safeguard. Regarding topologies, we select three fixed structures commonly found in MAS (Qian et al., 2024): chain, tree, and star. Additionally, we use random graph to test the generalization ability. For LLMs, we included a wide range of open-source (llama-3.1-70B (Dubey et al., 2024) and deepseek-v3 (Liu et al., 2024a)) and closed-source models (GPT-4o, GPT-4o-mini, and claude-3.5-haiku). Furthermore, we verify that G-Safeguard can be directly transferred to larger-size MAS without retraining, which demonstrates the inductive ability (Wu et al., 2020). Finally, we conduct experiments on the well-known multi-agent framework, Camel (Li et al., 2023), which represents practical interaction architectures in real-world scenes.

We employ Graph Attention Networks (GAT) as our foundational GNN architecture. The training configuration is specified as follows: for each dataset, we generate 800 training samples and 60 test samples. We use Adam as the optimizer, with the learning rate set to 1e-3, weight decay set to 2e-4, dropout set to 0.2, and batch size set to 32.

4.2 Effectiveness of G-Safeguard

In this section, we create three types of attack targeting prompt, tool and memory in MAS to verify the effectiveness of G-Safeguard. We showcase the different rounds of attack success rate (ASR) across different LLMs and topologies. The results are placed in Table 1 and Figure 3, from which we can list the observations (Obs):

Obs 1. G-Safeguard can prevent the spread of malicious information after identifying the attackers. As shown in Table 1, with the implementation of G-Safeguard, MAS under various settings has exhibited more robust and secure behavior. Especially on the CSQA and MMLU, the MAS equipped with G-Safeguard exhibits a significant decrease in the ASR after three rounds of dialogue across various topologies. For example, in low-connectivity topologies (Chain & Tree), the average decreases about ~18.01% for CSQA and ~20.01% for MMLU. In high-connectivity settings, the average decrease is about ~24.74% and ~24.90%, respectively. On the GSM8K, with G-Safeguard added, the system’s performance remains largely unaffected by attackers. It is noteworthy that without G-Safeguard, MAS with high-connectivity topologies based on Deepseek-V3
Figure 3: The accuracy of MAS on the CSQA (left) and MMLU (right) datasets after each turn of dialogue. We use majority voting as the strategy to select the final answer. Dashed line: no G-safeguard; Solid line: with G-safeguard.

or LLaMA on the CSQA essentially collapses, with problem-solving performance failing to reach 50%, as illustrated in Figure 3.

We also test commonly used LLM-based defense methods, but they did not yield satisfactory results. For details, please refer to Appendix B.

Figure 4: The attacker’s recognition accuracy of G-Safeguard for MAS with different topological structures composed of various LLMs on PoisonRAG dataset.

Obs 2. G-Safeguard can generalize across different LLMs and topologies. Although we trained it using communication data generated by the GPT-4o-mini-based MAS, its training results can easily generalize to different LLMs. As illustrated in Figure 4, under the memory attack strategy, the recognition accuracy of G-Safeguard on MAS constructed by LLaMA-3.1-70b and Claude-3.5-haiku can surpass that of GPT-4o-mini-based MAS. Even though its performance on GPT-4o and Deepseek-V3 is relatively weaker, the majority of results still exceed 75%. This indicates that training G-Safeguard does not require special LLM and training data generated from readily available models can be generalized to other LLM. Additionally, due to the inductive properties of GNNs, G-Safeguard can be easily transferred to MAS with unseen topologies. As shown in Table 1, G-Safeguard provides effective defense across different topological structures. For instance, LLaMA-3.1-70b achieve a reduction in ASR 3.49% ~ 35.97% across different topologies, demonstrating G-Safeguard’s transferability.

4.3 Scalability of G-Safeguard

In this section, we examine the inductive capability and scalability of G-Safeguard towards larger-scale MAS. Due to the inductive properties of GNNs, we propose that a G-Safeguard trained on small-scale MAS can be applied directly to larger-scale MAS without the necessity of generating training data from large-scale MAS, which would incur prohibitive costs. For this analysis, we train the G-Safeguard using data generated from an MAS composed of eight agents and subsequently apply it to MAS comprising {20, 35, 50, 65, 80} agents to evaluate its effectiveness. In this experiment, the MAS we construct is no longer in the setting of multi-agent debate, but involves communication between all agents that have edges connecting them. All other settings remain consistent with the main experiment.

<table>
  <tr>
    <th rowspan="2">Agent Num.</th>
    <th colspan="5">Rounds</th>
  </tr>
  <tr>
    <th>R0/R0+GS</th>
    <th>R1/R1+GS</th>
    <th>R2/R2+GS</th>
    <th>R3/R3+GS</th>
  </tr>
  <tr>
    <td>20</td>
    <td>0/0</td>
    <td>6.67/0</td>
    <td>18.67/0</td>
    <td>25.93/0</td>
  </tr>
  <tr>
    <td>35</td>
    <td>0/0</td>
    <td>10.37/2.33</td>
    <td>23.71/2.96</td>
    <td>26.66/3.04</td>
  </tr>
  <tr>
    <td>50</td>
    <td>0/0</td>
    <td>7.00/0</td>
    <td>15.00/0</td>
    <td>23.00/3.50</td>
  </tr>
  <tr>
    <td>65</td>
    <td>9.62/6.92</td>
    <td>32.69/8.46</td>
    <td>44.62/9.62</td>
    <td>50.77/11.54</td>
  </tr>
  <tr>
    <td>80</td>
    <td>0.31/0.31</td>
    <td>5.29/1.25</td>
    <td>17.50/2.50</td>
    <td>22.81/2.19</td>
  </tr>
</table>

Table 2: ASR each round on MAS with different numbers of agents. GS stands for G-Safeguard.

Obs 3. G-Safeguard can be directly transferred to larger-scale MAS without the need for retraining. As shown in Table 2 and fig. 5, G-Safeguard-guided MAS demonstrates better robustness, which indicates that G-Safeguard trained on data generated from small-scale MAS does not suffer performance degradation when applied to larger MAS with more agents and different topologies. For example, in an MAS composed of 65 agents, a performance recovery of 39.23% was achieved. With this observation, we answer the question of RQ2. Due to the transferability of G-Safeguard across different topologies and scales of MAS, we can reasonably posit
that G-Safeguard can generalize to scenarios with constantly changing topologies in MAS.

![A line chart showing the reply accuracy of agents on MAS with different number of nodes.](page_186_324_579_312.png)

Figure 5: The reply accuracy of agents on MAS with different number of nodes.

4.4 Real-world Application

In this section, we explore the application of G-Safeguard to multi-role scenarios. In real-world settings, the agents in a MAS may consist of agents with different roles. Based on the role-playing framework of CAMEL (Li et al., 2023), we configure agents with various roles in our MAS and place the MAS in a scenario with attackers. For detailed prompts, please refer to the paper (Li et al., 2023). We measured the attacker recognition accuracy of G-Safeguard in MAS constructed by different LLMs on the CSQA and MMLU datasets.

![A circular diagram showing attacker recognition accuracy on camel built with various LLMs, evaluated on CSQA and MMLU.](page_186_1042_579_312.png)

Figure 6: Attacker recognition accuracy on camel built with various LLMs, evaluated on CSQA and MMLU.

Obs 4. G-Safeguard can be seamlessly integrated into real-world MAS pipelines, enhancing the defensive capabilities. In our constructed multi-role multi-agent system, G-Safeguard can still accurately identify attackers within the system and stably adapt to MAS systems built with various LLMs. As shown in Figure 6, G-Safeguard achieves an identification accuracy of over 80% on both the CSQA and MMLU datasets, effectively preventing a large number of attackers from compromising the MAS and avoiding system collapse.

Based on the above observations, we can answer the questions posed at the very beginning of Section 4, which proves the effectiveness of the G-Safeguard.

5 Related Work

Agent Safety. LLM-based agent safety has garnered significant attention. It can be broadly divided into (1) single-agent safety and (2) multi-agent safety. Unlike foundation LLMs, agents are designed with distinct roles, memory, and tool invocation to enhance functionality (Guo et al., 2024; Wang et al., 2024). While promising, these features also introduce vulnerabilities, as attacks can inject malicious instructions into tools (Greshake et al., 2023; Liu et al., 2023a; Tian et al., 2023) or memory (Zhang et al., 2024g). To address this, studies (Inan et al., 2023; Xie et al., 2023; Liu et al., 2024c; Zhang et al., 2024f;i; Phute et al., 2023) have focused on improving security alignment and protective measures for both agent parameters and external entities. Extending beyond single agents, MAS enhance task-solving through collaboration (Li et al., 2023; Qian et al., 2023), but this interaction also risks toxicity transmission (Tian et al., 2023; Chern et al., 2024; Yu et al., 2024b; Gu et al., 2024). An attacked agent not only performs malicious actions but can also spread toxicity, potentially paralyzing the entire MAS and triggering collective malicious behavior.

Multi-agent as Graphs. With the widespread application of MAS (Chan et al., 2023; Chen et al., 2023a; Cohen et al., 2023; Chen et al., 2023b; Hua et al., 2023; Park et al., 2023), researchers have recognized that multi-agent interactions can be effectively modeled using graphs (Chen et al., 2023b; Liu et al., 2023c; Qian et al., 2024; Zhuge et al., 2024). Studies like ChatEval (Chan et al., 2023), AutoGen (Wu et al., 2023), and DyLAN (Liu et al., 2023c) utilize predefined or hierarchical graph structures to facilitate agent communication and collaboration. Others, such as GPTSwarm (Zhuge et al., 2024) and AgentPrune (Zhang et al., 2024b), optimize graph topologies for efficiency and performance. NetSafe (Yu et al., 2024b) investigates toxicity propagation in MAS under attacks across various topological structures. Inspired by these, we model MAS with graphs and employ
GNNs to detect malicious nodes, leveraging their inductive capabilities to adapt to diverse structures. In this work, we adapt the graph-based foundation to uncover the detection and inductive skill of attacked MAS, which provide valuable insights for safer designs of future frameworks.

6 Conclusion
In this paper, we address, for the first time, the critical issue of anomaly detection and security protection for individual modules within MAS. We introduce the G-Safeguard framework, designed to enhance the inductive learning capabilities of models. This framework pioneers the ability to train on small-scale MAS and seamlessly transfer defensive mechanisms to larger-scale MAS architectures. Through extensive experimentation across various system configurations (e.g., tree, chain, graph) and under diverse attack scenarios (e.g., prompt injection, memory attack), we demonstrate that G-Safeguard not only provides superior defense against attacks but also facilitates effortless transfer of protective capabilities across different base LLMs. These findings open new avenues for future research in MAS security.

7 Acknowledgement
This work is partially supported by the National Natural Science Foundation of China (Grant No. 12227901). The authors gratefully acknowledge this financial support.

Limitation
Although G-Safeguard demonstrates robust capabilities in anomaly detection and mitigation within an attacked Multi-Agent System (MAS), it is important to note that G-Safeguard cannot preemptively prevent the MAS from being compromised. As a defense mechanism reliant on communication data analysis, G-Safeguard primarily functions to curtail the further dissemination of malicious information within the MAS. However, by the time G-Safeguard identifies an attacker, certain nodes within the MAS have already been successfully compromised. Consequently, the proactive prevention of attacks in MAS environments emerges as a pivotal direction for our future research endeavors.

References
Amazon. 2025. Build Generative AI Applications with Foundation Models - Amazon Bedrock - AWS — aws.amazon.com. https://aws.amazon.com/bedrock/?nc1=h_ls. [Accessed 15-02-2025].

Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, et al. 2024. Agentharm: A benchmark for measuring harmfulness of llm agents. arXiv preprint arXiv:2410.09024.

Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201.

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1–45.

Dake Chen, Hanbin Wang, Yunhao Huo, Yuzhao Li, and Haoyang Zhang. 2023a. Gamept: Multi-agent collaborative framework for game development. arXiv preprint arXiv:2310.08067.

Jun Chen and Haopeng Chen. 2021. Edge-featured graph attention network. arXiv preprint arXiv:2101.07671.

Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. 2023b. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308.10848, 2(4):6.

Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. 2024. Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases. arXiv preprint arXiv:2407.12784.

Steffi Chern, Zhen Fan, and Andy Liu. 2024. Combating adversarial attacks with multi-agent debate. arXiv preprint arXiv:2401.05998.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168.

Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. 2023. Lm vs lm: Detecting factual errors via cross examination. arXiv preprint arXiv:2305.13281.

Zhichen Dong, Zhanhui Zhou, Chao Yang, Jing Shao, and Yu Qiao. 2024. Attacks, defenses and evaluations for llm conversation safety: A survey. arXiv preprint arXiv:2402.09283.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Xiaohan Fu, Shuheng Li, Zihan Wang, Yihao Liu, Rajesh K Gupta, Taylor Berg-Kirkpatrick, and Earlene Fernandes. 2024. Imprompter: Tricking llm agents into improper tool use. arXiv preprint arXiv:2410.14923.

Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, pages 79–90.

Xiangming Gu, Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Ye Wang, Jing Jiang, and Min Lin. 2024. Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast. arXiv preprint arXiv:2402.08567.

Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680.

Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. 2024. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. arXiv preprint arXiv:2406.18495.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.

Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei, Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang. 2023. War and peace (waragent): Large language model-based multi-agent simulation of world wars. arXiv preprint arXiv:2311.17227.

Wenyue Hua, Xianjun Yang, Mingyu Jin, Zelong Li, Wei Cheng, Ruixiang Tang, and Yongfeng Zhang. 2024. Trustagent: Towards safe and trustworthy llm-based agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10000–10016.

Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674.

Yoichi Ishibashi and Yoshimasa Nishimura. 2024. Self-organized agents: A llm multi-agent framework toward ultra large-scale code generation and optimization. arXiv preprint arXiv:2404.02183.

Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. 2024. Exploiting programmatic behavior of llms: Dual-use through standard security attacks. In 2024 IEEE Security and Privacy Workshops (SPW), pages 132–143. IEEE.

Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991–52008.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengji Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.

Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, et al. 2024b. Toolace: Winning the points of llm function calling. arXiv preprint arXiv:2409.00920.

Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, et al. 2023a. Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499.

Zichuan Liu, Zefan Wang, Linjie Xu, Jinyu Wang, Lei Song, Tianchun Wang, Chunlin Chen, Wei Cheng, and Jiang Bian. 2024c. Protecting your llms with information bottleneck. arXiv preprint arXiv:2404.13968.

Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. 2023b. Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.02170.

Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. 2023c. Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.02170.

Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlou, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196.

Fatemeh Nazary, Yashar Deldjoo, and Tommaso di Noia. 2025. Poison-rag: Adversarial data poisoning attacks on retrieval-augmented generation in recommender systems. arXiv preprint arXiv:2501.11759.

Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22.

Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527.
Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau. 2023. Llm self defense: By self examination, llms know they are being tricked. arXiv preprint arXiv:2308.07308.

Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. 2023. Communicative agents for software development. arXiv preprint arXiv:2307.07924, 6(3).

Chen Qian, Zihao Xie, Yifei Wang, Wei Liu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. 2024. Scaling large-language-model-based multi-agent collaboration. arXiv preprint arXiv:2406.07155.

N Reimers. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937.

Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. 2023. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301.

Yu Tian, Xiao Yang, Jingyuan Zhang, Yinpeng Dong, and Hang Su. 2023. Evil geniuses: Delving into the safety of llm-based agents. arXiv preprint arXiv:2311.11855.

Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, et al. 2023. Tensor trust: Interpretable prompt injection attacks from an online game. arXiv preprint arXiv:2311.01011.

Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345.

Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776–5788.

Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. Auto-gen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155.

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. 2020. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1):4–24.

Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 5(12):1486–1496.

Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2023. Benchmarking and defending against indirect prompt injection attacks on large language models. arXiv preprint arXiv:2312.14197.

Miao Yu, Junfeng Fang, Yingjie Zhou, Xing Fan, Kun Wang, Shirui Pan, and Qingsong Wen. 2024a. Llm-virus: Evolutionary jailbreak attack on large language models. arXiv preprint arXiv:2501.00055.

Miao Yu, Shilong Wang, Guibin Zhang, Junyuan Mao, Chenlong Yin, Qijiong Liu, Qingsong Wen, Kun Wang, and Yang Wang. 2024b. Netsafe: Exploring the topological safety of multi-agent networks. arXiv preprint arXiv:2410.15686.

Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents. arXiv preprint arXiv:2403.02691.

Guibin Zhang, Xiangguo Sun, Yanwei Yue, Chonghe Jiang, Kun Wang, Tianlong Chen, and Shirui Pan. 2024a. Graph sparsification via mixture of graphs. arXiv preprint arXiv:2405.14260.

Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. 2024b. Cut the crap: An economical communication pipeline for llm-based multi-agent systems. arXiv preprint arXiv:2410.02506.

Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, and Dawei Cheng. 2024c. G-designer: Architecting multi-agent communication topologies via graph neural networks. arXiv preprint arXiv:2410.11782.

Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. 2024d. Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents. arXiv preprint arXiv:2410.02644.

Xinyu Zhang, Huiyu Xu, Zhongjie Ba, Zhibo Wang, Yuan Hong, Jian Liu, Zhan Qin, and Kui Ren. 2024e. Privacyassist: Safeguarding user privacy in tool-using large language model agents. IEEE Transactions on Dependable and Secure Computing.

Yuqi Zhang, Liang Ding, Lefei Zhang, and Dacheng Tao. 2024f. Intention analysis prompting makes large language models a good jailbreak defender. arXiv preprint arXiv:2401.06561.
Zaibin Zhang, Yongting Zhang, Lijun Li, Hongzhi Gao, Lijun Wang, Huchuan Lu, Feng Zhao, Yu Qiao, and Jing Shao. 2024g. Psysafe: A comprehensive framework for psychological-based attack, defense, and evaluation of multi-agent system safety. arXiv preprint arXiv:2401.11880.

Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. 2024h. A survey on the memory mechanism of large language model based agents. arXiv preprint arXiv:2404.13501.

Ziyang Zhang, Qizhen Zhang, and Jakob Foerster. 2024i. Parden, can you repeat that? defending against jailbreaks via repetition. arXiv preprint arXiv:2405.07932.

Zhonghan Zhao, Wenhao Chai, Xuan Wang, Boyi Li, Shengyu Hao, Shidong Cao, Tian Ye, and Gaoang Wang. 2025. See and think: Embodied agent in virtual environment. In European Conference on Computer Vision, pages 187–204. Springer.

Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. 2024. Gptswarm: Language agents as optimizable graphs. In Forty-first International Conference on Machine Learning.
A  Additional results of G-Safeguard

We present more detailed results on CSQA, MMLU, GSM8K, InjecAgent and PoisonRAG benchmarks in Appendix A.

B  Experiments on LLM-based Defense Methods

We provide the comparison with three defense methods (LlamaGuard [1], WildGuard [2], ThinkGuard [3]) on prompt injection (MMLU) and random topologies, using GPT-4o-mini as the base model. The experimental results are presented in Table 8.

<table>
  <tr>
    <th>Method/Round</th>
    <th>R0</th>
    <th>R1</th>
    <th>R2</th>
    <th>R3</th>
  </tr>
  <tr>
    <td>LlamaGuard</td>
    <td>18.32</td>
    <td>26.99</td>
    <td>32.46</td>
    <td>37.33</td>
  </tr>
  <tr>
    <td>WildGuard</td>
    <td>17.21</td>
    <td>22.36</td>
    <td>28.55</td>
    <td>31.63</td>
  </tr>
  <tr>
    <td>ThinkGuard</td>
    <td>18.55</td>
    <td>23.84</td>
    <td>26.67</td>
    <td>28.12</td>
  </tr>
  <tr>
    <td>G-Safeguard</td>
    <td>18.30</td>
    <td>19.59</td>
    <td>21.28</td>
    <td>20.27</td>
  </tr>
</table>

Table 8: ASR after each round of conversation under different defense methods.

C  Detailed Prompts

Figure 7: Under prompt injection attacks, the prompt for normal nodes on CSQA and MMLU datasets.

Figure 8: Under prompt injection attacks, the prompt for attacker nodes on CSQA and MMLU datasets.

Figure 9: Under prompt injection attacks, the prompt for normal nodes on GSM8K dataset.

Figure 10: Under prompt injection attacks, the prompt for attacker nodes on GSM8K dataset.

Figure 11: Under tool attacks, the prompt for normal nodes on injecagent dataset.

Figure 12: Under tool attacks, the prompt for attacker nodes on injecagent dataset.

Figure 13: Under memory attacks, the prompt for normal nodes on poisonrag dataset.

Figure 14: Under memory attacks, the prompt for attacker nodes on poisonrag dataset.
<table>
  <tr>
    <th rowspan="2">Topology</th>
    <th rowspan="2">Dataset</th>
    <th colspan="4">PI (CSQA)</th>
  </tr>
  <tr>
    <th>Model</th>
    <th>R0/R0+GS</th>
    <th>R1/R1+GS</th>
    <th>R2/R2+GS</th>
    <th>R3/R3+GS</th>
  </tr>
  <tr>
    <td rowspan="6">Chain</td>
    <td rowspan="6">CSQA</td>
    <td>GPT-4o-mini</td>
    <td>29.06/26.88</td>
    <td>38.12/27.50</td>
    <td>44.06/29.06</td>
    <td>45.93/27.50</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>22.00/20.67</td>
    <td>28.67/20.67</td>
    <td>32.67/22.00</td>
    <td>34.67/23.33</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70b</td>
    <td>27.40/25.08</td>
    <td>36.52/32.33</td>
    <td>44.03/33.33</td>
    <td>52.22/35.33</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>26.25/26.33</td>
    <td>40.31/26.95</td>
    <td>48.13/28.52</td>
    <td>50.00/28.84</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>23.75/23.75</td>
    <td>39.68/30.00</td>
    <td>50.31/29.69</td>
    <td>55.31/31.25</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>29.06/29.38</td>
    <td>38.75/32.19</td>
    <td>43.43/31.87</td>
    <td>45.31/31.87</td>
  </tr>
  <tr>
    <td rowspan="6">Tree</td>
    <td rowspan="6">CSQA</td>
    <td>GPT-4o</td>
    <td>18.66/23.33</td>
    <td>28.66/24.00</td>
    <td>33.33/24.00</td>
    <td>34.00/24.67</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70b</td>
    <td>33.91/32.60</td>
    <td>46.75/39.13</td>
    <td>54.34/37.55</td>
    <td>56.33/39.13</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>28.70/28.70</td>
    <td>36.99/27.67</td>
    <td>22.01/29.78</td>
    <td>42.95/29.47</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>24.68/23/13</td>
    <td>42.81/28.44</td>
    <td>59.06/27/81</td>
    <td>63.43/28.75</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>29.06/29.06</td>
    <td>39.37/28.75</td>
    <td>46.56/28.75</td>
    <td>48.75/29.06</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>28.57/26.67</td>
    <td>30.47/25.71</td>
    <td>33.33/25.71</td>
    <td>40.95/24.76</td>
  </tr>
  <tr>
    <td rowspan="6">Star</td>
    <td rowspan="6">CSQA</td>
    <td>LLaMA-3.1-70b</td>
    <td>31.93/30.71</td>
    <td>45.61/31.96</td>
    <td>51.05/32.51</td>
    <td>55.64/34.01</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>25.97/26.92</td>
    <td>52.24/29.37</td>
    <td>55.91/28.98</td>
    <td>56.81/30.25</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>24.68/25.31</td>
    <td>48.43/28.75</td>
    <td>66.25/29.69</td>
    <td>74.37/29.06</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>28.75/29.78</td>
    <td>45.45/30.63</td>
    <td>51.56/30.63</td>
    <td>54.23/29.37</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>20.00/20.00</td>
    <td>27.19/20.94</td>
    <td>35.94/21.25</td>
    <td>44.06/21.56</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70b</td>
    <td>26.24/27.00</td>
    <td>44.44/30.79</td>
    <td>50.00/34.77</td>
    <td>53.59/36.75</td>
  </tr>
  <tr>
    <td rowspan="6">Random</td>
    <td rowspan="6">CSQA</td>
    <td>Claude-3.5-haiku</td>
    <td>26.62/26.62</td>
    <td>40.06/26.88</td>
    <td>41.14/27.18</td>
    <td>41.14/27.15</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>23.75/25.00</td>
    <td>46.56/29.38</td>
    <td>70.31/30.63</td>
    <td>76.25/32.18</td>
  </tr>
</table>

§ ASR: In our work, ASR represents the proportion of agents that exhibit malicious or incorrect behaviors.

Table 3: ASR after each round of conversation for MAS constructed by different LLMs on CSQA Dataset.

<table>
  <tr>
    <th rowspan="2">Topology</th>
    <th rowspan="2">Dataset</th>
    <th colspan="4">PI (MMLU)</th>
  </tr>
  <tr>
    <th>Model</th>
    <th>R0/R0+GS</th>
    <th>R1/R1+GS</th>
    <th>R2/R2+GS</th>
    <th>R3/R3+GS</th>
  </tr>
  <tr>
    <td rowspan="6">Chain</td>
    <td rowspan="6">MMLU</td>
    <td>GPT-4o-mini</td>
    <td>19.59/19.59</td>
    <td>26.35/21.62</td>
    <td>29.39/20.61</td>
    <td>34.46/21.96</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>15.00/14.55</td>
    <td>19.54/14.09</td>
    <td>25.00/14.09</td>
    <td>27.73/14.09</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70b</td>
    <td>19.41/19.41</td>
    <td>33.57/20.72</td>
    <td>40.07/17.45</td>
    <td>43.69/19.34</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>18.00/16.67</td>
    <td>34.67/15.33</td>
    <td>37.33/15.00</td>
    <td>38.00/15.00</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>16.36/8.58</td>
    <td>33.57/8.95</td>
    <td>40.07/9.75</td>
    <td>43.68/10.45</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>18.88/19.58</td>
    <td>23.08/21.32</td>
    <td>28.32/18.18</td>
    <td>29.72/18.53</td>
  </tr>
  <tr>
    <td rowspan="6">Tree</td>
    <td rowspan="6">MMLU</td>
    <td>GPT-4o</td>
    <td>10.56/10.56</td>
    <td>12.67/11.97</td>
    <td>16.20/13.38</td>
    <td>18.31/11.26</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70b</td>
    <td>17.22/18.08</td>
    <td>29.09/20.88</td>
    <td>38.73/17.58</td>
    <td>38.18/16.84</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>22.00/24.67</td>
    <td>40.73/26.00</td>
    <td>47.33/25.00</td>
    <td>46.67/25.33</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>7.00/6.02</td>
    <td>18.00/7.67</td>
    <td>30.33/7.33</td>
    <td>31.33/8.03</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>18.67/19.67</td>
    <td>26.00/17.33</td>
    <td>28.00/18.67</td>
    <td>30.00/20.00</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>7.50/7.50</td>
    <td>12.50/6.67</td>
    <td>17.50/7.50</td>
    <td>20.80/8.33</td>
  </tr>
  <tr>
    <td rowspan="6">Star</td>
    <td rowspan="6">MMLU</td>
    <td>LLaMA-3.1-70b</td>
    <td>15.67/16.43</td>
    <td>32.99/19.09</td>
    <td>40.55/19.79</td>
    <td>42.61/20.13</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>20.61/19.25</td>
    <td>34.80/20.27</td>
    <td>39.53/19.32</td>
    <td>43.24/19.58</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>6.68/8.00</td>
    <td>21.07/7.33</td>
    <td>37.13/7.67</td>
    <td>45.82/7.33</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>18.98/18.30</td>
    <td>26.35/19.59</td>
    <td>34.80/21.28</td>
    <td>38.83/20.27</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>14.63/14.63</td>
    <td>15.24/14.02</td>
    <td>21.95/10.97</td>
    <td>29.26/8.54</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70b</td>
    <td>18.77/15.82</td>
    <td>39.86/14.04</td>
    <td>45.61/15.10</td>
    <td>51.35/15.38</td>
  </tr>
  <tr>
    <td rowspan="6">Random</td>
    <td rowspan="6">MMLU</td>
    <td>Claude-3.5-haiku</td>
    <td>23.33/23.33</td>
    <td>40.33/24.33</td>
    <td>46.67/23.33</td>
    <td>49.33/23.33</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>3.97/3.97</td>
    <td>21.02/5.11</td>
    <td>37.50/5.68</td>
    <td>45.71/5.11</td>
  </tr>
</table>

§ ASR: In our work, ASR represents the proportion of agents that exhibit malicious or incorrect behaviors.

Table 4: ASR after each round of conversation for MAS constructed by different LLMs on MMLU Dataset.
<table>
  <tr>
    <th rowspan="2">Topology</th>
    <th rowspan="2">Dataset</th>
    <th colspan="4">PI (GSM8K)</th>
  </tr>
  <tr>
    <th>Model</th>
    <th>R0/R0+GS</th>
    <th>R1/R1+GS</th>
    <th>R2/R2+GS</th>
    <th>R3/R3+GS</th>
  </tr>
  <tr>
    <td rowspan="6">Chain</td>
    <td rowspan="6">Dataset</td>
    <td>GPT-4o-mini</td>
    <td>11.25/13.75</td>
    <td>9.17/9.58</td>
    <td>12.91/9.17</td>
    <td>15.42/9.58</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>14.16/14.16</td>
    <td>10.92/10.00</td>
    <td>10.83/10.83</td>
    <td>10.83/10.00</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70b</td>
    <td>13.40/9.56</td>
    <td>9.38/7.83</td>
    <td>11.27/7.37</td>
    <td>11.74/5.55</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>6.67/7.08</td>
    <td>7.08/6.66</td>
    <td>7.08/7.50</td>
    <td>7.08/6.27</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>8.33/9.16</td>
    <td>7.91/7.91</td>
    <td>11.25/11.25</td>
    <td>10.00/8.75</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>12.50/10.83</td>
    <td>10.41/10.00</td>
    <td>15.41/10.00</td>
    <td>16.66/9.58</td>
  </tr>
  <tr>
    <td rowspan="6">Tree</td>
    <td rowspan="6">Dataset</td>
    <td>GPT-4o</td>
    <td>7.91/8.75</td>
    <td>5.83/6.67</td>
    <td>8.75/7.50</td>
    <td>7.91/6.25</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70b</td>
    <td>13.79/14.14</td>
    <td>8.83/6.25</td>
    <td>8.33/8.17</td>
    <td>10.59/5.76</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>7.08/7.08</td>
    <td>6.67/6.67</td>
    <td>6.67/7.08</td>
    <td>6.67/7.08</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>7.08/7.08</td>
    <td>5.42/7.08</td>
    <td>10.00/10.83</td>
    <td>10.42/7.08</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>12.91/11.67</td>
    <td>10.41/9.17</td>
    <td>14.58/10.42</td>
    <td>19.58/9.58</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>10.59/8.05</td>
    <td>6.36/6.35</td>
    <td>5.93/7.20</td>
    <td>7.20/7.20</td>
  </tr>
  <tr>
    <td rowspan="6">Star</td>
    <td rowspan="6">Dataset</td>
    <td>LLaMA-3.1-70b</td>
    <td>7.76/10.96</td>
    <td>7.51/5.24</td>
    <td>9.38/3.79</td>
    <td>9.38/4.26</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>6.25/6.67</td>
    <td>5.83/5.42</td>
    <td>5.83/5.42</td>
    <td>5.83/5.00</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>8.89/8.05</td>
    <td>5.51/6.78</td>
    <td>7.20/11.02</td>
    <td>6.36/8.47</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>11.25/10.41</td>
    <td>10.00/10.00</td>
    <td>12.50/10.83</td>
    <td>17.92/11.67</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>9.32/10.17</td>
    <td>5.51/5.08</td>
    <td>5.93/5.08</td>
    <td>7.63/5.08</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70b</td>
    <td>12.91/10.50</td>
    <td>6.22/5.43</td>
    <td>8.00/3.60</td>
    <td>7.11/3.62</td>
  </tr>
  <tr>
    <td rowspan="6">Random</td>
    <td rowspan="6">Dataset</td>
    <td>Claude-3.5-haiku</td>
    <td>7.08/7.08</td>
    <td>11.66/7.92</td>
    <td>8.75/7.50</td>
    <td>10.83/7.50</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>9.16/7.50</td>
    <td>6.25/6.25</td>
    <td>9.16/9.16</td>
    <td>7.91/7.50</td>
  </tr>
</table>

§ ASR: In our work, ASR represents the proportion of agents that exhibit malicious or incorrect behaviors.

Table 5: ASR after each round of conversation for MAS constructed by different LLMs on GSM8k Dataset.

<table>
  <tr>
    <th rowspan="2">Topology</th>
    <th rowspan="2">Dataset</th>
    <th colspan="4">TA(InjecAgent)</th>
  </tr>
  <tr>
    <th>Model</th>
    <th>R0/R0+GS</th>
    <th>R1/R1+GS</th>
    <th>R2/R2+GS</th>
    <th>R3/R3+GS</th>
  </tr>
  <tr>
    <td rowspan="6">Chain</td>
    <td rowspan="6">Dataset</td>
    <td>GPT-4o-mini</td>
    <td>3.07/1.92</td>
    <td>23.08/2.69</td>
    <td>33.46/2.69</td>
    <td>36.54/2.69</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>5.00/4.61</td>
    <td>13.46/5.00</td>
    <td>15.00/5.00</td>
    <td>16.15/5.38</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70b</td>
    <td>50.00/43.07</td>
    <td>64.61/56.38</td>
    <td>68.46/59.23</td>
    <td>69.61/60.77</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>5.83/5.83</td>
    <td>20.83/23.33</td>
    <td>20.83/26.67</td>
    <td>20.83/29.16</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>27.15/29.43</td>
    <td>40.08/49.12</td>
    <td>41.81/50.00</td>
    <td>42.67/50.88</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>4.16/4.16</td>
    <td>29.16/4.16</td>
    <td>42.50/4.16</td>
    <td>47.50/4.16</td>
  </tr>
  <tr>
    <td rowspan="6">Tree</td>
    <td rowspan="6">Dataset</td>
    <td>GPT-4o</td>
    <td>0.00/1.67</td>
    <td>8.33/1.67</td>
    <td>11.67/1.67</td>
    <td>12.50/1.67</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70b</td>
    <td>37.50/41.67</td>
    <td>60.83/54.16</td>
    <td>69.17/58.33</td>
    <td>70.83/58.33</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>4.31/6.89</td>
    <td>24.14/20.69</td>
    <td>25.86/25.86</td>
    <td>25.86/29.31</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>24.11/25.00</td>
    <td>36.84/28.75</td>
    <td>47.37/38.79</td>
    <td>47.37/50.87</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>2.67/0.89</td>
    <td>24.11/2.67</td>
    <td>36.61/3.57</td>
    <td>40.18/3.57</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.83/0.83</td>
    <td>3.33/0.83</td>
    <td>5.83/0.83</td>
    <td>6.67/0.83</td>
  </tr>
  <tr>
    <td rowspan="6">Star</td>
    <td rowspan="6">Dataset</td>
    <td>LLaMA-3.1-70b</td>
    <td>49.14/38.79</td>
    <td>64.65/48.28</td>
    <td>69.83/49.14</td>
    <td>70.69/49.14</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>6.675.00</td>
    <td>15.83/20.00</td>
    <td>16.67/24.17</td>
    <td>16.67/25.00</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>17.86/25.00</td>
    <td>50.00/25.00</td>
    <td>67.86/25.00</td>
    <td>67.86/25.00</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>3.73/2.50</td>
    <td>18.33/3.33</td>
    <td>25.83/3.33</td>
    <td>26.16/3.33</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.83/1.67</td>
    <td>3.33/3.33</td>
    <td>3.33/3.33</td>
    <td>3.33/4.16</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>LLaMA-3.1-70b</td>
    <td>48.33/45.00</td>
    <td>60.83/52.50</td>
    <td>64.17/53.33</td>
    <td>65.00/53.33</td>
  </tr>
  <tr>
    <td rowspan="2">Random</td>
    <td>Claude-3.5-haiku</td>
    <td>5.00/5.83</td>
    <td>14.17/20.83</td>
    <td>17.50/23/33</td>
    <td>17.50/24.16</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>28.33/24.16</td>
    <td>47.66/26.67</td>
    <td>46.67/26.67</td>
    <td>50.00/26.67</td>
  </tr>
</table>

§ ASR: In our work, ASR represents the proportion of agents that exhibit malicious or incorrect behaviors.

Table 6: ASR after each round of conversation for MAS constructed by different LLMs on InjecAgent Dataset.
<table>
  <tr>
    <th rowspan="2">Topology</th>
    <th rowspan="2">Model</th>
    <th colspan="4">Dataset MA(PoisonRAG)</th>
  </tr>
  <tr>
    <th>R0/R0+GS</th>
    <th>R1/R1+GS</th>
    <th>R2/R2+GS</th>
    <th>R3/R3+GS</th>
  </tr>
  <tr>
    <td rowspan="4">Chain</td>
    <td>GPT-4o-mini</td>
    <td>8.78/8.19</td>
    <td>14.04/9.36</td>
    <td>16.96/10.53</td>
    <td>18.13/9.95</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>8.78/8.19</td>
    <td>13.45/11.11</td>
    <td>13.45/12.87</td>
    <td>16.38/11.70</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70b</td>
    <td>8.19/9.36</td>
    <td>24.57/14.62</td>
    <td>31.58/14.62</td>
    <td>40.94/14.62</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>11.11/13.45</td>
    <td>15.21/14.62</td>
    <td>28.08/18.72</td>
    <td>50.88/38.60</td>
  </tr>
  <tr>
    <td rowspan="4">Tree</td>
    <td>GPT-4o-mini</td>
    <td>8.19/8.19</td>
    <td>21.05/13.45</td>
    <td>26.32/15.21</td>
    <td>29.83/16.96</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>8.19/5.27</td>
    <td>12.29/9.95</td>
    <td>16.38/10.53</td>
    <td>19.89/10.53</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70b</td>
    <td>17.08/11.59</td>
    <td>33.45/15.86</td>
    <td>37.87/14.02</td>
    <td>43.29/14.02</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>13.45/9.36</td>
    <td>16.38/10.53</td>
    <td>28.01/11.11</td>
    <td>36.26/26.32</td>
  </tr>
  <tr>
    <td rowspan="4">Star</td>
    <td>GPT-4o-mini</td>
    <td>10.48/11.43</td>
    <td>10.93/13.33</td>
    <td>11.91/11.91</td>
    <td>13.81/11.43</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>8.78/7.02</td>
    <td>14.62/8.78</td>
    <td>16.96/9.36</td>
    <td>22.81/8.77</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70b</td>
    <td>8.54/14.63</td>
    <td>33.54/18.30</td>
    <td>42.58/17.73</td>
    <td>50.61/20.22</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>11.70/10.53</td>
    <td>15.20/13.45</td>
    <td>45.81/33.92</td>
    <td>47.20/36.16</td>
  </tr>
  <tr>
    <td rowspan="4">Random</td>
    <td>GPT-4o-mini</td>
    <td>8.78/8.19</td>
    <td>25.15/11.70</td>
    <td>32.16/12.28</td>
    <td>42.96/12.28</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>7.02/8.19</td>
    <td>9.95/9.36</td>
    <td>12.87/11.2–</td>
    <td>16.38/11.70</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70b</td>
    <td>10.70/11.93</td>
    <td>28.30/15.72</td>
    <td>38.36/16.98</td>
    <td>44.66/16.99</td>
  </tr>
  <tr>
    <td>Claude-3.5-haiku</td>
    <td>9.95/11.70</td>
    <td>16.38/15.21</td>
    <td>63.92/22.81</td>
    <td>41.53/30.17</td>
  </tr>
  <tr>
    <td rowspan="1">Deepseek-V3</td>
    <td>13.25/12.05</td>
    <td>20.48/13.25</td>
    <td>25.30/22.05</td>
    <td>31.32/12.05</td>
  </tr>
</table>

§ ASR: In our work, ASR represents the proportion of agents that exhibit malicious or incorrect behaviors.

Table 7: ASR after each round of conversation for MIAS constructed by different LLMs on PoisonRAG Dataset.