Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
Authors: Sunwoo Lee
Jaebak Hwang
Yonghyeon Jo
Seungyul Han
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Multi-Agent Reinforcement Learning
Robustness
Adversarial Learning
Primary area: reinforcement_learning->multiagent
TL;DR: We propose the Wolfpack Adversarial Attack framework to enhance robustness in multi-agent reinforcement learning, which targets an initial agent and its assisting agents to disrupt cooperation.

Abstract: Traditional robust methods in multi-agent reinforcement learning (MARL) often struggle against coordinated adversarial attacks in cooperative scenarios. To address this limitation, we propose the Wolfpack Adversarial Attack framework, inspired by wolf hunting strategies, which targets an initial agent and its assisting agents to disrupt cooperation. Additionally, we introduce the Wolfpack-Adversarial Learning for MARL (WALL) framework, which trains robust MARL policies to defend against the proposed Wolfpack attack by fostering system-wide collaboration. Experimental results underscore the devastating impact of the Wolfpack attack and the significant robustness improvements achieved by WALL. Our code is available at https://github.com/sunwoolee0504/WALL.

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
Sunwoo Lee' Jaebak Hwang'
Yonghyeon Jo
Seungyul Han
Abstract
CTDE methods face challenges from exploration inefficien- cies (Mahajan et al., 2019; Jo et al., 2024) and mismatches between training and deployment environments, leading to unexpected agent behaviors and degraded performance (Moos et al., 2022; Guo et al., 2022). Thus, enhancing the robustness of CTDE remains a critical research focus.
Traditional robust methods in multi-agent re- inforcement learning (MARL) often struggle against coordinated adversarial attacks in co- operative scenarios. To address this limitation, we propose the Wolfpack Adversarial Attack framework, inspired by wolf hunting strategies, which targets an initial agent and its assisting agents to disrupt cooperation. Additionally, we introduce the Wolfpack- Brrtpacck-AAversaan Adversarial Learning for MARL (WALL) framework, which trains ro- bust MARL policies to defend against the pro- posed Wolfpack attack by fostering systemwide collaboration. Experimental results underscore the devastating impact of the Wolfpack attack and the significant robustness improvements achieved by WALL. Our code is available http://gghabbccmm/ssoooe//////a
To improve learning robustness, single-agent RL methods have explored strategies based on game theory (Yu et al., 2021), such max-min approaches and adversarial learning (Goodfellow et al., 2014; Huang et al., 2017; Pattanaik et al., 2017; Pinto et al., 2017). In multi-agent systems, simulta- neous agent interactions introduce additional uncertainties (Zhang et al., 2021b). To address this, methods like per- turbing local observations (Lin al., 2020), training with adversarial policies for Nash equilibrium (Li al., 2023a), adversarial value decomposition (Phan et al., 2021), and attacking inter-agent communication (Xue et al., 2021) have been proposed. However, these approaches often target a single agent per attack, overlooking interdependencies in cooperative MARL, making them vulnerable to scenarios where multiple agents are attacked simultaneously.
1. Introduction
Multi-agent Reinforcement Learning (MARL) has gained attention for solving complex problems requiring agent operation (Oroojlooy& & Hajinezhad, 2023) and competition, such as drone control (Yun et al., 2022), autonomous nav- igation (Chen et al., 2023), robotics (Orr& & Dutta, 2023), and energy management (Jendoubi & Bouffard, 2023). To handle partially observable environments, the Centralized Training and Decentralized Exccution (CTDE) framework (Oliehoek et al., 2008) trains global value function cen- trally while agents execute policies based on local obser- vations. Notable credit-assigmment methods in CTDE in- clude Value Decomposition Networks (VDN) (Sunehag et al., 2017), QMIX (Rashid et al., 2020), which satisfies the Individual- Siwiiiiua--Gooo--Mar (IGM) condition ensuring that optimal joint actions align with positive gradients in global and individual value functions, and QPLEX (Wang et al., 2020b), which encodes IGM into its architecture. However,
To overcome the vulnerabilities posed by coordinated ad- versarial attacks in MARL, we propose the Wolfpack adver- sarial attack framework, inspired by wolf hunting strategies. This approach disrupts inter-agent cooperation by target- ing a single agent and subsequently attacking the group of agents assisting the initially targeted agent, resulting in more devastating impacts. Experimental results reveal that traditional robust MARL methods highly susceptible to such coordinated attacks, underscoring the need for new defense mechanisms. In response, also introduce the Learning for MARL (WALL) frame- work, a robust policy training approach specifically designed to counter the Wolfpack Adversarial Attack. By fostering system- wide collaboration and avoiding reliance on spe- cific agent subsets, WALL enables agents defend effec- tively against coordinated attacks. Experimental evaluations demonstrate that WALL significantly improves robustness compared to existing methods while maintaining high per- formance under a wide range of adversarial attack scenarios. The key contributions of this paper in constructing the Wolf- pack Adversarial Attack are summarized as follows:
'Graduate School of Artificial Intelligence, UNIST, UI- san, South Korea. Correspondence to: Scungyul Han <sy- han @ un
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
A novel MARL attack strategy, Wolfpack Adversarial Attack, is introduced, targeting multiple agents simul-

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
taneously to foster stronger and more resilient agent cooperation during policy training.
(Wang al., 2020c; Kobayashi, 2024). Transition models have been leveraged to improve robustness (Mankowitz et al., 2019; Ye et al., 2024; Herremans et al., 2024), and offline setups have been explored for robust training (Rigter et al., 2022; Bhardwaj et al., 2024). In multi-agent systems, model- approaches address challenges like construct- ing worst-case sets (Shi et al., 2024) and managing transition kernel uncertainty (He et al., 2022).
The follow-up agent group selection method is pro- posed to target agents with significant behavioral ad- justments an initial attack, enabling subsequent se- quential attacks and amplifying their overall impact.
A planner-based attacking step selector predicts fu- ture Q-value reductions caused by the attack, enabling the selection of critical time steps to maximize impact and improve learning robustness.
3. Background
3.1. Dec-POMDP and Value-based CTDE Setup
2. Related Works
A fully cooperative multi-agent enviromment is modeled as a decentralized partially observable Markov decision pro- cess (Dec-POMDP) (Olichoek et al., 2016), defined by the tuple M = (W, A, P, 2, O, R, 7). N = =1, 1.....;" 12 is the set of agents, & the global state space, A A x... x A" the joint action space, P the state transition probabil- ity, Ω the observation space, R the reward function, and Y E [0, 1) the discount factor. At time t, each agent i ob- serves of = Ost, i) E S2 and takes action of E A' based on its individual policy x (-(rr)) where 57 is the agent's 's trajectory up to t. The joint action a, = (at..... aaf)) sam- pled from the joint policy π := (T).... ,T") leads to the next state S14-1 P(-|s,, a.) and reward r, == R(s,, az). MARL aims to find the optimal joint policy that maxi- mizes Et= 7"r,. As noted, this paper adopts the central- ized training with decentralized execution (CTDE) setup, where the joint value Q' ax) is learned using temporal- difference (TD) learning. Through credit assignment, indi- vidual value functions Q' q'(T,, at) are learned, guiding indi- vidual policies z5 to select actions that maximize Q', i.e., n' arg max Q'(TE, :).
Robust MARL Strategies: Recent research has focused on robust MARL to address unexpected changes in multi-agent environments. Max-min optimization (Chinchuluun et al., 2008; Han & Sung, 2021) has been applied to traditional MARL algorithhs for robust learning (Li al., 2019; Wang 2022). Robust Nash equilibrium has been redefined to better suit multi-agent systems (Zhang et al., 2020b; Li et al., 2023a). Regularization-bain based approaches have also been explored to improve MARL robustness (Lin et al., 2020; Li et al., 2023b; Wang et al., 2023; Bukharin et al., 2024), alongside distributional reinforcement learning methods to manage uncertainties (Li et al., 2020; Xu et al., 2021: Du et al., 2024; Geng et al., 2024).
Adversarial Attacks for Resilient RL: To strengthen RI.,
numerous studies have explored adversarial learning to train
policies under worst-case scenarios (Pattanaik et al., 2017:
Tessler et al., 2019; Pinto et al., 2017; Chac et al., 2022). These attacks introduce perturbations to various MDP com- ponents, including state (Zhang et al., 2020a; 2021a; Everett et al., 2021; Li et al., 2023c; Qiaoben et al., 2024), action (Tan et al., 2020; Lee et al., 2021; Liu et al., 2024), and reward (Wang et al., 2020a; Zhang et al., 2020c; Rakhsha et al., 2021; Xu et al., 2022; Cai et al., 2023; Bouhaddi & Adi, 2023; Xu et al., 2024; Bouhaddi & Adi, 2024). Adver- sarial attacks have recently been extended to multi-agent setups, introducing uncertainties to state observation (Han al., 2022; He et al., 2023; Zhang et al., 2023; Zhou al., 2023), actions (Yuan et al., 2023), and rewards (Kardeş et al., 1). Further research has applied adversarial attacks to value decomposition frameworks (Phan et al., 2021), se- lected critical agents for targeted attacks (Yuan et al., 2023; Zhou et al., 2024), and analyzed their effects on inter-agent communication (Xue et al., 2021: Tu et al., 2021; Sun al., 2023; Yuan et al., 2024).
3.2. Robust MARL with Adversarial Attack Policy
Among various methods for robust learning in MARL, Yuan et al. (2023) defined an adversarial attack policy Tadv and implemented robust MARL. by training multi-agent policies to defend against attacks executed by Tadv : S x A x N + A. A cooperative MARL environment with an adversarial policy Radv can be described as a Limited Policy Adversary Dec-POMDP LLPA-Dec-PMMDE M, defined as follows:
Definition 3.1 (Limited Policy Adversary Dec-POMDP). Given a Dec-POMDP M and a fixed adversarial policy Tadv, we define Limited Policy Adversary Dec-POMDP (LPA-Dec-POMD) M (N, K, Q, O, R. 7) where K is the maximum number of attacks, Tadv (-1sc, a,, kz) executes joint action ā to disrupt the original action a, chosen by n, and he S K indicates the number of remaining attacks.
Model-based Frameworks for Robust RL: Model-based methods have been extensively studied to enhance RL ro- bustness (Berkenkamp et al., 2017; Panaganti & Kalathil, 2021; Curi et al., 2021; Clavier et al., 2023; Shi & Chi, 2024; Ramesh et al., 2024), including adversarial extensions
Here, if Tady selects an attack action à, different from the original action a,, the remaining number of attacks h; de-
2

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
t+1
++t+1
Wollpack
Anitial Aftack
Help(Heal)
Attack
Wohpaok Aragk
Help(Guard)
(a)
(b)
(c)
Initial Agent
Responding (follow-up) Agents
Original Action
Figure 1. Visualization of Wolfpack attack strategy during combat in the StarCraft II environment: (a) The initial agent is attacked, disrupting its original action (b) Responding (follow-up) agents to help the initially attacked agent and (c) Wolfpack adversarial attack that disrupts help actions of follow-up agents.
creases by 1. Once k, reaches O, no further attacks can be performed. In this framework, Yuan et al. (2023) also demon- strated that the APA-Dec-POMDD can be represented as an- other Dec-POMDP induced by Tadv. with the convergence of the optimal policy x= under M guaranteed. particular, considers an adversarial attack targeting a chosen agent by minimizing its individual value function Q' proposed by (Pattanaik et al., 2017), i.e., ot arg mi Q'(T', a). Additionally, to enhance the attack's 's effectiveness, an evolu- tionary generation of attacker (EGA) approach is proposed, which combines multiple adversarial policies.
initially, followed by targeting the group of follow-up agents that respond to defend against the initial attack, as shown in Fig. (cc). Leveraging this approach, we aim to develop robust policies capable of effectively countering Wolfpack adversarial attack, thereby significantly enhancing the over- all resilience of the learning process.
4.2. Wolfpack Adversarial Attack
In this section, we formally propose the Wolfpack adver- sarial attack, as introduced in the previous sections. The Wolfpack attack consists of two components: initial attacks, where a single agent is targeted at a specific time step linit. and follow-up group attacks, where the group of agents re- sponding to the initial attack is selected and targeted over the subsequent steps fiiit + 1,... Liait + twp. Over the course of an episode, a maximum of KwP Wolfpack attacks can be executed. Consequently, the total number of attack steps is given by K Kwp (twP 1). The Wolfpack adversarial attacker TWP can then be defined as follows:
4. Methodology
4.1. Motivation of Wolfpack Attack Strategy
Existing adversarial attackers typically target only single agent per attack, without coordination or relationships be- tween successive attacks. In a cooperative MARL setup, such simplistic attacks enable non-targeted agents to learn effective policies to counteract the attack. However, we observe that policies trained under these conditions are vul- nerable to coordinated attacks. As illustrated in Fig. (a)), single agent is attacked at time t. In Fig. 1(b), at the next step + 1, responding agents adjust their actions, such as healing or moving to guard, to protect the initially attacked agent. In contrast, Fig. I(c) demonstrates a coordinated at- tack strategy that targets the agents responding to the initial attack. Such coordinated attacks render the learned policy ineffective, preventing it from countering the attacks en- tirely. This highlights that coordinated attacks are far more detrimental than existing attack methods, and current robust policies fail to defend effectively against them.
Definition 4.1 (Wolfpack Adversarial Attacker). A Wolf- m WP
pack adversarial attacker A is defined as â, Jady (st, ar, kt), where ai arg min (tat at. a, ") for all i E NN, and at = of otherwise. Here, a, represents the joint actions of all agents excluding the i-th agent, and Na, denotes the set of agents targeted for adversarial attack, defined as
aadv : S A x N + WP
0
if kr
{i}
else if = tiuit, i ~ Unif(AA)
Ne
Niollow-up up
else if = timit + 1,-.. tinit + twe, otherwise,
As depicted in Fig. 1(c), targeting agents that respond to an initial attack aligns with the Wolfpack attack strategy, a tactic widely employed in traditional military operations, as discussed in Section To adapt this concept to a cooper- ative multi-agent setup, we define Wolfpack adversarial attack as a coordinated strategy where one agent is attacked
where Unif(.) is the Uniform distribution, Niollow {ūn, fm) C N is the group of agents selected for follow-up attack, and m is the number of follow-up agents.
Here, note that k, decreases by 1 for every attack step such that å & a,, as in the ordinary adversarial attack policy, and the total value function Qtot is used for the attack instead
3

---

In the Wolfpack adversarial attacker, we aim to identify the follow-up agent group N Noollow-up that actively responds to WWP the initial attack π ady (Stmas: atma kt...) and target them in subsequent steps. To do this, we define the difference Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
Pm((Sof((@))SSff((())
AQ!"
A
1--
Oc+1
2++
2++2
S Stttp
O+tly
Heal
Guard
Transformer
A²
2
Heal
S2 0
a
|-1
O++1
a
St+twp-1
Oeteyp-
detenp-1
2
23 follow-up
N
Guard
i= [1, - - Soft(@)) = (1,3)
iQ
Figure 2. Visualization of follow-up agent group selection method: Agent is initially attacked, and the m agents exhibiting the largest changes Q' are selected from (1, 2, 3) (m- = 2).
Figure 3. Planning with Transformer
maximize Q', making distributional differences easier to compute. The follow-up agent group is determined by se- lecting the m agents that maximize the Kullback-Leiblee (KI.) divergence DKL. between these distributions:
WP of Q' The proposed Wolfpack adversarial attacker т. ady is a special case of the adversarial policy defined in Definition 3.1. Consequently, the proposed attacker forms an LPA-Dec- POMDP M induced by WP, and as demonstrated Yuan et al. (2023), the convergence of MARL within the LPA- Dec-POMDP can be guaranteed. The proposed Wolfpack attack involves two key issues: how to design the group of follow-up agents Niallow-u and when to select linit. The following sections address these aspects in detail.
EDEI(
Nfollow-wr
arg max NENIN' ( m.jENN.
(2)
Sofl(Q" (Timi - ()]l() (Trma))
Using the proposed method, the follow-up agent group is identified as the agents whose policy distributions expe- rience the most significant changes following the initial attack. Fig. 2 illustrates this process. After the initial at- tack, Q Q-differences
4.3. Follow-up Agent Group Selection Method
computed for the remaining agents 1, 2, 3. and those with the largest changes in individual value functions are selected as the follow-up agent group. These agents are targeted over the next fwP time steps to prevent them from effectively responding. Section 5, we analyze how the proposed method enhances attack criticalness by comparing it to naive selection methods based solely on observation distances.
between the Q-functions from the original action and the initial attack time as:
AQ = =Q0o- (81, at) (so āt);
where AQtat 2 0 for all such that attack # 0, because å, minimizes Qiot for the agent indices selected by wWP. Assuming the i-th agent is the target of the initial attack, updating Qrat based on e vidual value function Q' to increase Q for all j A i e N, in accordance with the credit assignment principle in CTDE algorithms (Sunehag et al., 2017; Rashid et al., 2020), shown below:
4.4. Planner-based Critical Attacking Step Selection
In the proposed Wolfpack adversarial attacker a the follow-up agent group is defined, leaving the task of de- termining the timing of initial attacks linit, executed Kwr times within an episode. While Random Step Selection in- volves choosing time steps randomly, existing methods show that selecting steps to minimize the rewards of the execu- tion policy æ leads to more effective attacks and facilitates robust learning (Yuan et al., 2023). However, in coordinated attacks like Wolfpack, targeting steps that cause the greatest reduction in the Q-function value AQ ensures a more devastating and lasting impact on the agents' ability to re- cover and respond. Thus, we propose selecting initial attack times based on the total reduction in AQ!T defined as:
fuit adjusts each agent's 's indi-
dsQtot Eyntt
(' Q' Oir aQ (Ti a)
Baik
(1)
where Our is the learning rate. As agents select actions based on Q", changes in Q' indicate adjustments in their policies in response to the initial attack. Agents with the largest changes in Q' are identified as follow-up agents, while the i-th agent is excluded it is already under attack and cannot respond immediately.
AQUP
AQ
To identify the follow-up agent group. the updated Q and original Q² are transformed into distributions using the Soft- max function Soft(). This transformation softens the de- terministic policy n², which directly selects an action to
where the Wolfpack attack is performed from (initial at- tack) to t + 1,...... ,t + twp (follow-up attacks). Initial at- tack time steps Iii are chosen to maximize AQ which

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
t 9
t
14
[+
1.19
ENV
Attacking Step Selector
10
27
2.3
3.
Zt
T=10
Policy
T=1
10
ăe!
4Q
aLt+WP
at
T=0.1
18
35
13
11
10
0
0
Finit Lattack
Wolfpack Adversarial Attack
Critical Step
Timesteps
t= Centt
Probability (9%)
40
Initial Attack
T= 10
30
T=1
@<+1:++PP
follow-uu
Follow-up Group Selection
001
10
Figure 5. Illustration of the proposed WALL framework
5
10
15
20
Timesteps
environments. Since the proposed method involves planning at every evaluation, we also train separate model to predict ÂQPr significantly reducing computational complexity. Details of this approach and the Transformer training loss functions are provided in Appendix B.
Figure Attacking step probabilites
captures the total Q-value reduction caused by the attack over twp + 1 steps, enhancing the criticalness of the attack. However, computing AQ!P for every time step is computa- tionally expensive as it requires generating attacked samples through interactions with the environment. To mitigate this, stored buffer is utilized to plan trajectories of future states and observations for the attack.
4.5. WALL: A Robust MARL Algorithm
Similar to other robust MARI. methods, we propose the Wolfpack- Aaippack--dersaarialll Learning for MARL (WALL) frame- work, a robust policy designed to counter the Wolfpack attack by performing MARL the L.A-DDc-POMDP M with the Wolfpack attacker π While the proposed Wolf- pack framework is broadly applicable to most CTDE algo- rithms, we primarily applied it to well-known value-based CTDE methods, including QMIX (Rashid et al., 2020), VDN (Sunehag et al., 2017), and QPLEX (Wang et al., 2020b). Detailed implementations, including loss functions for the planner Transformer and the value functions, are provided in Appendix B.2. The proposed WALL framework is illustrated in Fig. 5 and summarized in Algorithm 1.
For planning, we employ Transformer (Vaswani, 2017), commonly used in sequential learning, which leverages an attention mechanism for efficient learning. As shown in Fig. 3, the Transformer learns environment dynamics P using replay buffer trajectories predict future states and observations (fe+1; ..... trp Ör+twe), where o, represents the joint observation used to compute Qtat Ac- WP "ndy for [ - 1, + Iwp, with & = 8t. Using the plan- ner, we estimate the Q-value reduction ÁQNP caused by the Wolfpack attack. For L time steps = f, ...... L 1, we compute future Q-differences ÁQ@ and select thai based on the initial attack probability Pt,tttack:
tions å; ady (s), at, k)) for ~ are generated by
Algorithm 1 WALL framework
Initialize: Value function Qta Planning Transformer for each training iteration do
Pt,ttaack = {Soft (AQ.1/ZZ /T. AQUE-
3: for each environment step t do
(3)
4: Sample the action a,: of e-greedy(Q*)
where x; indicates the I-th element of x, and T is the temperature. In this paper, we set L = 20 it provides an appropriate attack period. After selecting Kwp initial attacks, no further attacks are performed. Fig. 4 shows how step probabilities are distributed for different values (T 0.1, 10). At each time I, the planner predicts ÁQWP for tot+L- forming soft initial attack probabilities. A larger
5: Compute Pt Pt.tttack using Planner and sample finit
6: if fiii then
7:
WP Perform the initial attack: äy ~ ग adv else if /init + S linit + lwp then Select the follow-up agent group Niollow-up Perform the follow-up attack: ā, n: WP ady
10:
11:
else
I' results in more uniform probabilities, while a smaller T increases the likelihood of targeting critical steps where
12:
Execute the original action
13:
end if
ÁQUP, is highest. These critical steps are selected with the highest probabilities for initial attacks. In Section 5, we analyze the effectiveness this method in delivering more critical attacks compared to Random Step Selection and examine the impact of T on performance practical
14:
end for
15:
Update the Qtot using a CTDE algorithm
16: Update the Planning Transformer
17:
end for
5

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
predator
prey obstacle
(b)
(d)
Figure 6. MARL benchmarks used in our experiments: (a) PP PP_3/1 and (b) PP_6/ PP_6/2 6/2 in MPE, and (c) 8m and (d) MMM scenarios in SMAC..
5. Experiments
Method
Secmario
PP_31
PP.62
PP PP.IV3
Mene
In this section, we evaluate the proposed methods on two standard benchmarks in MARL research: the Multi-Agent Particle Environment (MPE) (Lowe et al., 2017) and the StarCraft II Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019), as illustrated in Fig. 6. Specifically, we com- pare: (1) the impact of the proposed Wolfpack adversarial at- tack against other adversarial attacks, and (2) the robustness of the WALL framework defending against such attacks compared to other robust MARL methods. Also, ablation study analyzes the effect of the proposed components and hyperparameters on robustness. All results are reported as the mean and standard deviation (shaded areas for graphs and + values for tables) across 5 random seeds. Our code is available at https///gthbb..mmmmwooolle/////
Vanilla
1.3
453.4
Nitural
ROMANCE
RANDOM
175.0 0.7
178.5 1.0
648.6
083.8
8.9
721.4
0...0.1. 3.5
7.5
515.0
503.0
WALL (cuI)
202.9 1.7
076.0 + 8.8
802.6 +
560
Vanilla
3.0
170.1 1.3
633.5
488.0
Attack
ROMANCE
175.4
1.4
621.2
8.5
701.7
499.7 2.4
WALL (OUIS)
202.9
1.7
064.0 8 2.1
802.0 =:
063 1G8.141
Vanilia
2.0
11.5
412.8
ROMANCE
167.1 2.4
003.
994.
455-5-1
WAT T. (eum))
166.5 1.7
686 5 5.0
682.7 0.8
511.5
Wolfpack
Vanilla QMIX
RANDOM
185.7 1.5
1.4
115.0 8,4
824.3
S11.7
450.9
367.6
Attack (ours)
WALL (ours)
ROMANCE
171.5
1.5
509.0
9.4
698.1 1
570.3
489.5
Table I. Average cumulative rewards of robust MARL policies under various attack settings in the MPE environments.
formed; Random Attack, where time steps, agents, and actions are randomly selected to execute attacks; Evolution- ary Generation of Attackers (EGA) (Yuan et al., 2023), which combines multiple single-agent-targe
5.1. Environmental Setup
attackers generated from various seeds as described in Section 3; and the proposed Wolfpack Adversarial Attack. For fair comparison, adversarial attackers are trained on independent seeds to execute unseen attacks.
The MPE environment provides multi-agent setting where agents interact through simple physical dynamics. We con- duct experiments on three predator-prey prey (PP) scenarios with varying agent-to-target ratios: PP_3/1, PP PP_6/2, 2, and PP_9/3. In these tasks, multiple predator agents must co- ordinate to capture one or more prey agents moving adver- sarially. The SMAC environment serves as a challenging benchmark requiring effective agent cooperation to defeat opponents. We evaluate the proposed method across six sce- narios: 2s32, 3m, 3s_vs__ _____32, 8m, MMM, and 123 We perform parameter searches for the number of follow-up agents m, total Wolfpack attacks K Kwr, and attack duration twp, using optimal settings for comparisons. To ensure re- alistic constraints, we set m m n-1 where n is the maximum number of allied units. We provide details environment setups and experimental configurations, includ- ing hyperparameter settings, in Appendices A and C. All MARL methods are evaluated on the QMIX baseline, with comparison for other CTDE baselines in Appendix E.
To compare the severity of at- tack basclines and the robustness of policies trained under adversarial attack scenarios, we evaluate QMIX-trained poli- cies under the following attack conditions: Vanilla QMIX, assuming no adversarial attacks; RANDOM, using Ran- dom Attack; RARL (Pinto et al., 2017), where adversarial attackers tailor attacks to the learned policy; RAP (Vinit- sky et al., 2020), an extension of RARL that uniformly samples attackers to prevent overfitting and introduce diver- sity; ROMANCE (Yuan et al., 2023), an RAP extension countering diverse EGA attacks; ERNIE (Bukharin et al., 2024), enhancing robustness via adversarial regularization in observations and actions; and the proposed WALL. All robust MARL methods follow author-provided methodolo- gies and parameters. Further details on the MARL baselines are available in Appendix D. All policies are trained for 3M timesteps, starting from a pretrained Vanilla QMIX model trained for 1 1M timesteps.
Robust MARL Baselines:
Adversarial Attacker Baselines: To compare the severity of different attacks, we consider the following 4 scenarios: Natural, representing the where attacks per-
6

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
Scenario
2s32
3m
35 3z
8m
MMM
1c3s5z
Mean
Method
Vanilla QMIX
98.0 1.5
99.2 : 1.0
99.2 = 1.6
97.6
99.2 = 0.5
98.1 1.1
98.7 0.5
RANDOM
99.7 0.5
99.1
1.2
99.0 = 0.8
99.2
99.6 0.6
99.3 + 1.0
99.3 : 0.2
RARL
97.8 2.0
93.8 + 3.2
93.1 17.4
3.7
90.6 + 20.8
84.0 33.7
92.5 5.3
Natural
RAP
98.8 1.3
96.8 1 4.4
99.5 1.00
14.7.66.77
95.5 1: 12.1
84.2 1: 16.9
94.7
ERNIE
98.2 1 1.3
99.2 1 1.2
99.8 99.8 0.4
99.8
0.5
98.5 = 1.7
99.2 4 1.0
99.1 10.1::00 0.5
ROMANCE
96.4 2.9
93.6 13.7
99.7 0.5
99.6
0.6
96.4 6.6
96.5 4 4.3
97.0 17 2.1
WALL (ours)
99.4 + 0.6
99.7 0.8
09.8401
0.6
99.0
99.5 0.6
99.4 = 0.5
Vanilla QMIX
80.4 =0.43.22
69.6 10.4
4.6
69.0
66.4
918 3.4
78.6
RANDOM
3.8
76.4
0.6
4.9
95.4 5.3
96.0
89.4
RARL
$6.8
58.7 15.4
89.2 20.4
5.8
84.2 5.4
79.1 22.2
78.0 7.2
Random Attack
RAP
9.77
1.1
97.8
114
86.7 -
86.6 12.5
83.0 3.T
ERNIE
6.9
6.92
90.2
76.2 = 11.8
86.0 18.2
95.6 96.61322 3.2
82.7 5.2
ROMANCE
9.3
+
0.6
8..
86.8 - 16.3
94.0
87.8
WALL (ours)
94.6 4.5
87.4
0.5
9
99.4
98.6 1.6
95.9 = 0.5
Vanilla QMIX
54.0 7.6
66.5
72.4 15.1
71.2 20.0
14
$3.0
09.6
RANDOM
6 3.3
70.6
08.8 23.5
87.5
84.4
84.5 +
76.9 5.4
RARL
62.6 9.4
88.4 12.4
78.
83.4 11.9
80.1 11.4
77.9
10.0
EGA
RAP
70.4 13.0
7.3
86.2 15.8
83.9 16.4
80.2 5.4
81.5 1 4.3
ERNIE
52.4 9.6
20.1
9.7
8.5
85.0 4.6
93.6 2.2
76.0 + 3.0
ROMANCE
2.8
86,8
9.1
9..9
87.8 + 11.7
89.6 + 2.9
87.5 + 1.6
WALL (ours)
88.6 5.4
87.0
98.7
95.8
2.9
94.2 1: 3.8
97.0 1.3
93.6 1.5
Vanilla QMIX
7.5
118
8.74
11.4 13.3
10.4 14.3
59.2
39.4
RANDOM
60.4 27.4
37.4
9..
40.4 14.8
63.6
68.4 19.6
63.5 3.1
Wolfpack
RARL
52.4 15.8
31.1
00.0+17.4
9..7
51.1
36.8
75.9 13.6
52.5 8.0
Adversarial
RAP
60.0 = 10.3
37.3 10.4
9.9
35.6 = 14.4
47.0 36.9
75.7 26.1
58.5 5.7
Attack (ours)
ERNIE
43.2 = 13.0
35.4 6.2
44
26.4 10.4
26,2 17.3
77.0 9.3
50.5 1 6.6
ROMANCE
6.1
31.8
14.3
9.6
28.6 = 14.2
48.8 - 17.9
81.2 +
59. 2.0
WALL (ours)
92.2 3.7
90.8 4.9
99.8 0.5
83.6 5.0
95.0
98.8 1.6
93.4 1.1
Table 2. Average test win rates of robust MARL policies under various attack settings in the SMAC environments.
Test Win Rate (%)
100
WALL ONIX
8m
MMM
ing superior policy quality even without external threats.
80
80
For SMAC, Table 2 presents the average win rates over the last 100 episodes of MARL policies under differ- ent attack baselines. The results show that the proposed Wolfpack adversarial attack significantly more pow- erful than existing methods such EGA and Random Attack. For example, EGA reduces the performance of Vanilla QMIX by 98.7 69.6 = 29.1% and RANDOM by 99.3 - 76.9 22.4% compared to the natural scenario. In contrast, the Wolfpack attack reduces Vanilla QMIX performance by 98.7 -9.4 59.3% and RANDOM by 99.3 63.5 35.8% demonstrating its greater impact. In addition, the proposed WALL framework, which is trained to defend against the Wolfpack attack, outperforms other robust MARL methods under all attack types, showcasing its superior robustness. Notably, although RANDOM is trained specifically against Random Attack and ROMANCE against EGA, WALL achieves better performance against both attack types. These results highlight the effectiveness of WALL in enabling robust learning under diverse adversar- ial scenarios. Fig. 7 further illustrates this in the 8m and MMM environments, where performance differences with existing methods are most pronounced, showing the average win rate of each policy over training steps under unseen Wolfpack adversarial attacks. The results reveal that WALL not only achieves higher robustness but also adapts more quickly to attacks. Similar trends are observed for other CTDE algo-
RAP
60
BOLE
40
40
20
20
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2,5
3.0
Timesteps (10%)
Timesteps (109)
Figure 7. Learning curves of MARL methods for Wolfpack attack
5.2. Performance Comparison in MPE and SMAC
Table presents the average cumulative rewards over the last 100 episodes under various attack settings in the MPE environments. The results show that the proposed Wolf- pack adversarial attack is significantly more disruptive than existing methods such as EGA and Random Attack. For example, for Vanilla QMIX, the average cumulative re- ward across the three predator-prey scenarios drops by 455.4 - 367.6 = 87.8 under the proposed Wolfpack at- tack. compared to 455.4- 412.8 42.6 under EGA and 455.4 445.9 9.5 under Random Attack. These results demonstrate that the Wolfpack attack imposes much more severe degradation in policy performance. In contrast, the proposed WALL framework consistently achieves the best performance across all attack types. Notably, in the MPE scenarios, WALL outperforms all basclines not only under adversarial attacks but also in the natural setting, demonstrat-

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
t = 6 (mint))
7-9
10-19
20~29
1= Terminal
fintt (wp)
WICTOR
Kill Enemy
Ally
Enemy
Hit Enemy
Enemy
Kill
(b)
Guard
Killed by Enemy
DE EAT
Move to Hit
Hit Enemy
Worback Bltack
Killea by Enemy
(c)
Guard
Kill Enemy
Kill Enemy
VICTORY
Move
Hit. Enemy
Initial Agent
Follow-up Agents
Other Ally Agents
Original Action
Figure& 8. Attack comparison on 2x3z task in the SMAC: (a) QMIX/Natural, (b) QMIX/Wolfpack attack, and (c) WALL/Wolfppck attack
rithms, such as VDN and QPLEX, as detailed in Appendix E.1, confirming the robustness of the proposed method.
5.4. Ablation Study
To evaluate the impact of each component and hyperpa- rameter in the proposed Wolfpack adversarial attack, we conduct an ablation study focusing on the following aspects: component evaluation, step selection temperature T, and the number of follow-up agents m. The ablation study is conducted in the 8m and MMM environments, where the per- formance differences are most pronounced. Additionally, more ablation studies on other hyperparameters, such as the total number of Wolfpack attacks Kwr and the attack duration (wp. are provided in Appendix F.
To support a more practical evaluation, we assess compu- tational complexity and general robustness under common perturbations in Appendix E.4 and Appendix E.5, respec- tively. WALL incurs about 30% higher training cost than ROMANCE but achieves substantially better performance due to its critical step selection. For robustness, we consider perturbations including Gaussian noise in observations and test-time parameter shifts such as reduced allied HP, under which WALL still outperforms existing baselines. These re- sults demonstrate the practical effectiveness of WALL under both computational and environmental challenges.
To evaluate the impact of each proposed component on attack severity and policy robust- ness, we consider five setups: 'Default', which uses all pro- posed components as designed; 'Init. agent (min)', where the initial target agent is selected to minimize Qtat, i.c. = arg min, min,, Q*'"" af a instead of random selection; 'Follow-up (L2)', which selects m agents closest to the initial agent based on I.2 distance instead of the proposed follow-up selection method; 'Step (Random)', which uses random step selection instead of the proposed step selection method, while keeping the same total number of attacks; and 'Agents & Step (Random)', which randomly selects both m follow-up agents and attack steps.
Component Evaluation:
5.3. Visualization of Wolfpack Adversarial Attack
To analyze the superior performance of the Wolfpack attack, we provide visualization of its execution in the SMAC en- vironment. Fig. 8 illustrates a scenario where the proposed step selector identifies & = 6 as a critical initial step to ini- tiate the attack. Prior to = 6, all setups are assumed to follow the same trajectory. Fig. 8(a) shows Vamilla QMIX in Natural scenario without attack, where our agents suc- cessfully defeat all enemy agents, achieving victory. Fig. 8(b) demonstrates Vanilla QMIX under the Wolfpack adver- sarial attack, with follow-up agents targeted during = 7 to t = 9. This leaves other agents unable to effectively defend against the adversarial attack, resulting in defeat as all agents are killed by the enemy. Fig. 8(c) highlights policy trained with the WALL framework. Despite the same follow-up agents are targeted during WALL trains non-attacked agents back up and protect the attacked agents, enabling ally agents to eliminate enemy agents and secure victory. This visualization demonstrates how the Wolfpack attack disrupts agent coordination and how the WALL framework robustly defends against such attacks. Visualizations of other SMAC tasks and detailed follow-up agent selection are provided in Appendix G.
For each setup, we train the Wolfpack adversarial attack and the corresponding robust policy of WALL. Fig. 9(a) shows the robustness of WALL trained under each setup when exposed to the default Wolfpack attack, while Fig. 9(b) illustrates how each attack setup degrades the performance of 'Vanilla QMIX' compared to its 'Natural' performance. Randomly selecting the initial agent yields more robust poli- cies than selecting the agent minimizing Qlot ('Init. agent (min)'), as random selection introduces diversity in attack scenarios. While selecting the "o--mmmmmizing agent may slightly enhance attack severity in cases like MMM, the added diversity from random selection generally improves robust-
to / 9,
8

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
Test Win Rate (96)
100
100
100
90-
Test Win Rate (96)
80
Tast Win Rate (%K
80
60
WALL (m--3)
50
WALL (m
8m
MMM
8.0
2.0 3.0
8.0
(a) Robustness of WALL variations
Timesteps (10%)
Timesteps (109)
Test Win Rate (%)
(a) 8m
(b) MMM
90-
Figure 10. Number follow-up agents m
80
100
100
Tast VAin Rate (9)
Test Van (%6
60-
80-
80-
50
60
60
8m
MMM
40
Default
Step (Random)
Init. agent (min)
Agents & Step (Random)
20
WAL
20
Follow-up (L2)
0.0
0.5
0.0
0.5
1.5 2.0 Timesteps (10")
1.0
Timesteps (10")
(b) Degradation of Vanilla QMIX under each attack setup
(a) 8m
(b) MMM
Figure 9. Component evaluation of our WALL/Wolffpcc attack
Figure 11. Step selection temperature T
ness. Comparing 'Default' and 'Follow-up (L2)' shows that the proposed follow-up selection method enables more se- vere attacks and trains more robust policies than simply targeting agents closest to the initial agent. Similarly. 'De- fault' outperforms 'Step (Random) in both attack severity and robustness, demonstrating that the proposed planner effectively identifies critical steps minimize Qtot, pro- ducing stronger policies. Finally, 'Default' achieves signifi- cantly better robustness and more critical attacks compared to 'Agents& & Step (Random)', highlighting the combined effectiveness of the proposed components.
mance of WALL policies trained with varying values of T against the default Wolfpack attack. In both the 8m and MMM environments, a very small causes the attack to target only specific steps, leading to policies that are less robust against diverse attacks. Conversely, a very large leads to overly uniform attacks, failing to target critical steps effectively, which also results in less robust policies. Based on these findings, we determined that T = 0.5 strikes balance be- tween targeting critical steps and maintaining robustness, and we set this as the default parameter.
Number of Follow-up Agents m: To analyze the impact of the hyperparameter m, which determines the number of follow-up agents, on robustness, Fig. 10 shows how WALL trained with different values of defend against the default Wolfpack attack. To prevent excessive attack that could cause learning to fail, we assume m < [== The results indicate that in the 8m environment, when m is small, only a few agents defending against the initial attack are targeted, leading to reduced robustness. Conversely, when m = 4, too many agents are attacked, causing learning deteriorate. Therefore, m 3 yields the most robust performance and is considered the default hyperparameter. Similarly, in the MMM environment, m = 4 results in the most robust performance and is set as the default. Notably, when m = 1, the attack becomes a single-agent attack. As discussed in Section 4. performing coordinated multi-agent attack (m > 1) enables much more robust learning, demonstrating the effectiveness and superiority of the proposed Wolfpack attack framework.
6. Limitations
While the proposed WALL significantly improves robust- ness in MARL, it has few limitations. The first is the additional computational overhead introduced by training the Transformer for identifying critical steps. However, as shown in our analysis, this overhead is justified given that other baselines fail to achieve comparable performance even with extended training. Another limitation is the need for hyperparameter tuning construct the Wolfpack attack. Nevertheless, the method is not highly sensitive these hyperparameters, and the provided ablation study offers practical guidelines for selecting appropriate configurations.
7. Conclusions
In this paper, we propose the Wolfpack adversarial attack. a coordinated strategy inspired by the Wolfpack tactic used in military operations, which significantly outperforms ex- isting adversarial attacks. Additionally, we develop WALL,, robust MARL method designed to counter the proposed attack, demonstrating superior performance across various SMAC environments. Overall, our WALL framework en- hances the robustness of MARL algorithms.
Step Selection Temperature T: T is hyperparameter in Eq. (3) that controls the temperature of the initial at- tack probability. A larger T results in more random attacks across steps, while a smaller T focuses attacks on critical steps with higher probability, Fig. 11 illustrates the perfor-

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
Acknowledgment
Chae, J., Han, S., Jung. W., Cho, M., Choi, S., and Sung, Y. Robust imitation learning against variations in environ- ment dynamics. In International Conference on Machine Learning, pp. 2828-2288222 2852.. PMLR, 2022.
This work was supported by Institute Information Com- munications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.RS-2022- I1220469, Development of Core Technologies for Task- oriented Reinforcement Learning for Commercialization of Autonomous Drones), Innovative Human Resource De- velopment for Local Intellectualization program through the Institute of Information & Communications Technology Planning Evaluation (IITP) grant funded by the Korea and
Chen, J., Ma, R., and Oyekan, inforcement learning framework for autonomous aerial navigation to grasping points on loads. Robotics and Autonomous Systems, 167:104489, 2023.
deep multi multi-agent re-
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 1441441155097 2021.
2022-RSS-0222001556300 Institute of Information & Communications Technology Planning & Evaluation (IIP)) grant funded by the Korea government (MSIT) (No.RS RS-20200-1201333 Artificial In- telligence graduate school support (UNIST)).
government (MSIT)
Chinchuluun, A., Migdalas, A., Pardalos, M., and Pit- soulis, L. Pareto optimality, game theory and equilibria, volume 17. Springer New York, 2008
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Clavier, P., Pennec, E. L.., and Geist, M. Towards minimax optimality of model-based robust reinforcement learning. arXiv preprint arXiv:2302.000
2023.
Curi, S., Bogunovic, I., and Krause, A. Combining pes- simism with optimism for robust and efficient model- based deep reinforcement learning. In International Con- ference Machine Learning, pp. 2254-2266 PMLR, 2021.
References
Berkenkamp, F., Turchetta, M., Schoellig, A., and Krause, A. Safe model-based reinforcement learning with stability guarantees. Advances in neural information processing systems, 30, 2017.
Du, X., Chen, H., Wang, C., Xing, Y., Yang. J., Philip, S. Y., Chang. Y., and He, Robust multi- ageet reinforcement learning via bayesian distributional value estimation. Pat- tern Recognition, 145:109917, 2024.
Bhardwaj. M., Xie, T., Boots, B., Jiang, N., and Cheng, C.- A. Adversarial model for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Everett, M., Lütjens, B., and How, P. Certifiable robust- ness adversarial state uncertainty in deep reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 33(9):4184 4198, 2021.
Bouhaddi, M. and Adi, K. Multi-enviromment training against reward poisoning attacks deep reinforcement learning. In SECRYPT, pp. 870-875, 2023.
Geng, W., Xiao, B., Li, R., Wei, N., Wang, D., and Zhao, Z. Noise distribution decomposition based multi-agent distributional reinforcement learning. IEEE Transactions Mobile Computing. 2024.
Bouhaddi, M. and Adi, K. When rewards deceive: Coun- teracting reward poisoning on online deep reinforcement learning. In 2024 IEEE International Conference on Cyber Security and Resilience (CSR), pp. 38 44. IEEE, 2024.
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explain- ing and harnessing adversarial examples. arXiv preprint arXiv: 2ee..41226552 2014.
Bukharin, A., Li, Y., Yu, Y., Zhang, Q., Chen, Z., Zuo, S., Zhang, C., Zhang, S., and Zhao, T. Robust multi-agent re- inforcement learning via adversarial regularization: The- oretical foundation and stable algorithms. Advances in Neural Information Processing Systems, 36. 2024.
Guo, J., Chen, Y., Hao, Y., Yin, Z., Yu, Y., Li S. Towards comprehensive testing on the robustness of cooperative multi-agent reinforcement learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 115-122, 2022.
Cai, K., Zhu, X., and Hu, Z. Reward poisoning attacks in deep reinforcement learning based on exploration strate- gies. Neurocomputing,
Han, S. and Sung, Y. max-min entropy framework for reinforcement learning. Advances in Neural Information Processing Systems, 34:25732-25745, 2021.
2023.
10

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
Han, S., Su, S., He, S., Han, S., Yang, H., Zou, S., and Miao, F. What is the solution for state-adversarial multi-agent re- inforcement learning? arXiv preprint arXiv:2212.00700, 2022.
Li, S., Xu, R., Guo, J., Feng, P., Wang, J., Liu, A., Yang, Y., Liu, X., and Lv, W. Mir2: Towards provably robust multi-agent reinforcement learning by mutual informa- tion regularization. arXiv preprint arXiv:2310.09833 2023b.
He, S., Wang, Y., Han, S., Zou, S., and Miao, F. A ro- bust and constrained multi-agent reinforcement learning framework for electric vehicle amod systems. Dynamics, 8(10), 2022.
Li, X., Li, Y., Feng, Z., Wang, Z., and Pan, Q. Ats-o2ar A state-based adversarial attack strategy deep rein- forcement learning. Computers & Security, 129: 12:1103259, 2023c.
He, S., Han, S., Su, S., Han, S., Zou, S., and Miao, F. Robust multi-agent reinforcement learning with state uncertainty. arXiv preprint (tXXi:23307.121122 2023.
Lin, J., Dzeparoska, K., Zhang, S. Q.. Leon-Garcia, A., and Papernot, N. On the robustness of cooperative multi- agent reinforcement learning. In 2020 IEEE Security and Privacy Workshops (SPW), pp. 62-68. IEEE, 2020.
Herremans, S., Anwar, A., and Mercelis, S. Robust model- based reinforcement learning with an adversarial auxiliary model. arXiv preprint arXiv:2406.0999 2024.
Liu, Q., Kuang, Y., and Wang. Robust deep reinforcement learning with adaptive adversarial perturbations action space, arXiv preprint arXiv:2405. 11982, 2024.
Huang, S., Papernot, N., Goodfellow, I., Duan, Y., and Abbeel, P. Adversarial attacks on neural network policies. arXiv preprini arXiv: T0XYv:/702.02255 702.02284, 2017.
Lowe, R., Wu, Y. I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I. Multi-agent actor-critic for mixed cooperative computiiieei environments. Advances in neu- ral information processing systems, 30, 2017.
Jendoubi, I. and Bouffard, F. Multi-agent hierarchical re- inforcement learning for energy management. Applied Energy, 20500, 2023.
Jo, Y., Lee, S., Yeom, J., and Han, S. Fox: Formation-aware exploration in multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pp. 12985-12999 2024.
Mahajan, A., Rashid, T., Samvelyan, M., and Whiteson, S. Maven: Multi-agent variational exploration. Advances in neural information processing systems, 32, 2019.
Mankowitz, D. J., Levine, N., Jeong, R., Shi, Y., Kay, J., Abdolmaleki, A., Springenberg, J. T., Mann, T., Hester, T., and Riedmiller, M. Robust reinforcement learning for continuous control with model misspecification. arXiv preprint 190iv0/900..70 2019.
Kardeş, E., Ordóñez, F., and Hall, R. W. Discounted robust stochastic games and application to queueing control. Operations research, 59(2):365-382, 2011.
Kobayashi, T. Lira: Light-robust adversary for model-based reinforcement learning in real world. arXiv preprint 2rXiv:2409.1900 2024.
Moos, J., Hansel, K., Abdulsamad, H., Stark, S., Clever, D., and Peters, J. Robust reinforcement learning: A review of foundations and recent advances. Machine Learning and Knowledge Extraction, 4(1):276-33) 2022.
Lec, X. Y.. Esfandiari, Y., Tan, K. L., and Sarkar, S. Query-based targeted action-space adversarial policies on deep reinforcement learning agents. In Proceedings of the ACM/IEEE 12th international conference on cyber- physical systems, pp. 87-97, 2021.
Oliehoek, F. A., Spaan, M. T., and Vlassis, N. Optimal and approximate q g-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008.
Li, R., Wang, R., Tian, T., Jia, F., and Zheng, 7. Multi-agent reinforcement learning based on value distribution. In Journal of Physics: Conference Series, volume 1651, pp- 012017. IOP Publishing, 2020.
Olichoek, F. A., Amato, C., et al. A concise introduction to
decentralized POMDPs, volume 1. Springer, 2016.
Li, S., WU, Y., Cui, X., Dong, H., Fang, F., and Russell, S. Robust multi multi-agent reinforcement learning via minimax deep deterministic policy gradient. In Proceedings of the AAAI conference artificial intelligence, volume 33, pp.
Oroojlooy, A. and Hajinezhad, D. A review of coopera- tive multi-agent deep reinforvement learning. Applied Intelligence, 53(11): 2023.
4220- 2019.
Orr, and Dutta, A. Multi-agent deep reinforcement leam- ing for multi-robot applications: A survey. Sensors, 23 (7):3625, 2023.
Li, S., Guo, J., Xiu, J., Xu, R., Yu, X., Wang, J., Liu, A., Yang, Y., and Liu, X. Byzantine robust cooperative multi- agent reinforcement learning as a bayesian game. arXiv preprint Xrvi:22005 12872, 2023a.
Panaganti, K. and Kalathil, D. Sample complexity of model- based robust reinforcement learning. 202/ 60th IEEE

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
Conference on Decision and Control (CDC), pp. 2240- 2245. IEEE, 2021.
the face of environmental uncertainty. arXiv preprint MAXiv:240411909 2024.
Pattanaik, A., Tang, Z. Liu, S., Bommannan, G., and Chowdhary, G. Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv:1712.03632, 2017.
Sun, Y., Zheng, R., Hassanzadeh, P., Liang, Y., Feizi, S., Ganesh, S., and Huang, F. Certifiably robust policy learn- ing against adversarial multi-agent communication. In The Eleventh International Conference on Learning Rep- resentations, 2023.
Phan, T., Belzner, L., Gabor, T., Sedlmeier, A., Ritz, F., and Linnhoff- Linnhoff-Popien, C. Resilient multi-agent reinforce- ment learning with adversarial value decomposition. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 35, pp. 11308-115 11316, 2021.
Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zam- baldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint 1.X.v:1706.0529900 2017.
Pinto, L., Davidson, J., Sukthankar, R., and Gupta, A. Ro- bust adversarial reinforeement learning. In International conference on machine learning, pp. 2817-2826. PMLR, 2017.
Tan, K. L.,, Esfandiari, Y., Lee, X. Y., Sarkar, S., et al. Ro- bustifying reinforcement learning agents via action space adversarial training. In 2020 American control conference (ACC), pp. 3959-3964. IEEE, 2020.
Qiaoben, Y., Ying, C., Zhou, X., Su, H., Zhu, J., and Zhang, B. Understanding adversarial attacks on observations in deep reinforcement learning, Science China Information Sciences, 67(5):1-15, 2024.
Terry, J., Black, B., Grammel, N., Jayakumar, M., Hari, A.. Sullivan, R., Santos, L. S., Dieffendahl, C., Horsch, C., Perez- Vicente, R., et al. Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34: 15032-15043, 2021.
Rakhsha, A., Zhang, X., Zhu, X., and Singla, A. Reward poisoning reinforcement learning: Attacks against un- known learners in unknown environments. arXiv preprint arXiv:2102.08492, 2021.
Tessler, C., Efroni, Y., and Mannor, S. Action robust rein- forcement learning and applications in continuous control. International Conference on Machine Learning, pp. 6215 6224.. PMLR, 2019.
Ramesh, S. S., Sessa, P. G., Hu, Y., Krause, A., and Bo- gunovic, I. Distributionally robust model-based reinforce- ment learning with large state spaces. In International Conference Artificial Intelligence and Statistics, pp- 100 108. PMLR, 2024.
Tu, J., Wang. T., Wang. J., Manivasagam, S., Ren, M., and Urtasun, Adversarial attacks on multi-agent commu- nication. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7768-7777, 2021.
Rashid, T., Samvelyan, M., De Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S. Monotonic value function factorisation for deep multi-agent reinforcement learning- Journal of Machine Learning Research, 21(178):1-51, 2020.
Vaswani, A. Attention all you need. Advances Neural Information Processing Systems, 2017.
Vinitsky, E.,, Du, Y., Parvate, K., Jang, K., Abbeel, P., and Bayen, A. Robust reinforcement learning using adversar- ial populations. arXiv preprint arXiv:2008.014: 01825, 2020.
Rigter, M., Lacerda, B., and Hawes, N. Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in neural information processing systems, 35: 16082-16097, 2022.
Wang, J., Liu, Y., and Li, B. Reinforcement learning with perturbed rewards. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 34, pp. 6202-6209, 2020a.
Samvelyan, M., Rashid, T., De Witt, C. S., Farquhar, G., Nardelli, N., Rudner, G., Hung, C.-M., Torr, P. H., Foerster, J., and Whiteson, S. The starcraft multi-agent 2019.
challenge. arXiv preprint
Wang. J., Ren, Z., Liu, T., Yu, Y., and Zhang, C. Qplex: Duplex dueling multi-agent g-learning. arXiv preprint arXi::2008.01062 2020b.
Shi, I.. and Chi, Y. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. Journal of Machine Learning Research, 25 (200):1-91, 2024.
Wang. S., Chen, W., Huang, L., Zhang, F., Zhao, Z., and adapted anderson acceleration for
Qu H. Regularization catllariizatio-aaaapee multi-agent reinforcement learning. Knowledge-Band Based Systems, 275: 10709, 2023.
Shi, L., Mazumdar, E., Chi, Y., and Wierman, A. Sample- efficient robust multi-agent reinforcement learning in
12

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
Wang, X., Nair, S., and Althoff, M. Falsification-based robust adversarial reinforcement learning. In 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 205-212 IEEE, 2020c.
against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 33: 2 21024-21037, 2020a.
Zhang. H., Chen, H., Boning, D., and Hsieh, C.-J. Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint arXiv:2 1IXv:22001.0852, 202
Wang, Y., Wang, Y., Zhou, Y., Velasquez, A., and Zou, S. Data-driven robust multi-agent reinforcement learning- In 2022 IEEE 32nd International Workshop Machine Learning for Signal Processing (MLSP), Pp. 1-6. IEEE, 2022.
Zhang, K., Sun, T., Tao, Y., Genc, S., Mallya, S., and Basar, T. Robust multi-agent reinforcement learning with model uncertainty. Advances in neural information processing systems, 13:10571-10581 2020b.
Xu, Y., Zeng, Q.. and Singh, G. Efficient reward poisoning attacks online deep reinforcement learning. arXiv preprint arXiv:2205. (400:22,,.444,, 2022.
Zhang, K., Yang. Z., and Başar, T. Multi-agent reinforce- ment learning: selective overview of theories and algo- rithms. Handdoook of reinforcement learning and control, pp. 321-384--444 2021b.
Xu, Y., Gumaste, R., and Singh, Reward poisoning attack against offline reinforcement learning. arXiv preprint 3xXIv:2402.09695.. 2024.
Xu, Li, D., Bai, Y., and Fan, G. Mmd-mix: Value function factorisation with maximum mean discrepancy for coop- erative multi-agent reinforcement learning. In 2021 Inter- national Joint Conference Neural Networks (IJCNN), pp. 1-7. IEEE, 2021.
Zhang, X., Ma, Y., Singla, A., and Zhu, X. Adaptive reward- poisoning attacks against reinforcement learning. In In- ternational Conference on Machine Learning, pp. 11225- 11234. PMLR, 2020c.
Zhang, Z., Sun, Y., Huang, F., and Miao, Safe and ro- bust multi agent reinforcement learning for connected autonomous vehicles under state perturbations. arXiv preprint arXiv:2309. 2023.
Xue, W., Qiu, W., An, B., Rabinovich, Z., Obraztsova, S., and Yeo, C. K. Mis-spoke or mis-lead: Achieving ro- bustness in multi-agent communicative reinforcement learning. arXiv preprini S0fi::2188.03880 2021.
Zhou, Z., Liu, G., and Zhou, M. A robust mean-field actor- critic reinforcement learning against adversarial pertur- bations on agent states. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Ye, C., He, J., Gu, Q. and Zhang, T. Towards robust model- based reinforcement learning against adversarial corrup- tion. arXiv preprint arXiv:2402.0899) 2024.
Yu, J., Gehring, C.,, Schäfer, F., and Anandkumar, A. Robust reinforcement learning: constrained game-theoretic approach. In Learning for Dynamics and Control, pp- 1242-1254. PMLR, 2021.
Zhou, Z., Liu, G., Guo, W., and Zhou, M. Adversarial attacks on multiagent deep reinforcement learning mod- els in continuous action space. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2024.
Yuan, L., Zhang, Z., Xue, K., Yin, H., Chen, F., Guan, C., Li, L., Qian, C., and Yu, Y. Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attack- ers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11753-11762 2023.
Yuan, L., Jiang, T., Li, L., Chen, F., Zhang. Z., and Yu, Y. Robust cooperative multi-agent reinforcement learn- ing via multi-view message certification. Science China Information Sciences, 67(4): 2024.
Yun, J., Park, S., Kim, J., Shin, M., Jung, S., Mohaisen, D. A., and Kim, J.-H. Cooperative multiagent deep reinforcement learning for reliable surveillance via au- tonomous multi-uav control. IEEE Transactions on In- dustrial Informatics, 18(10):7086-700 2022.
Zhang, H., Chen, H., Xiao, C., Li, B., Liu, M., Boning, D., and Hsieh, C.-J. Robust deep reinforcement learning
13

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
A. Environmental Setup
conduct experiments MPE (Lowe et al., 2017) and SMAC (Samvelyan et al., 2019) environments. This section provides detailed descriptions of their setup and features.
A.1. Multi-Agent Particle Environments (MPE)
(a) PP PF13/1
(b) PP_6/2
(c) PP.9/3
Figure A.I. Visualization of PP scenarios in the MPE environment
The Multi-Agent Particle Environment (MPE) (Lowe et al., 2017) is widely used benchmark suite consisting of multi-agent scenarios. Agents are modeled particles capable of movement and interaction, governed by simple physical dynamics. MPE includes both cooperative and competitive tasks, with each scenario sharing a continuous state space and typically partial observability. A standardized implementation of MPE is available through the PettingZoo library (Terry et al., 2021).
Scenarios
The MPE benchmark includes variety of multi-agent scenarios. Among them, we focus on the predator-prey environment, which is well-suited for analyzing the impact of attacks on the cooperative structures among agents. We consider multiple variants denoted as PP_X/Y, where X represents the number of predator agents and Y the number of prey agents. In all scenarios, prey agents follow random policy. Detailed configurations for the three selected variants are summarized in Table A. and Fig. A.I.
State and observation spaces
Each agent in the MPE environment receives a partial observation, which includes its own position and velocity, the relative positions and velocities of other predators, and the relative positions of landmarks. The global state is constructed by concatenating the local observations of all agents.
Action space
The action space is discrete. Each agent choose of the four cardinal directions do nothing.
Reward function
The reward function R assigns positive value to predator agents upon successful collision with a prey:
Σ
I(collision(g, predator))
gEprey
where II(-) is the indicator function, returning if the condition inside is true and 0 otherwise. Here, collision(i.)) denotes whether agent i and agent j are in physical contact.
Map
Predators
Prey
State Dimension
Obs Dimension
Num. of Actions
PP_3/1
3 Agents
Agent
48
16
5
PP_6/2
6 Agents
Agents
156
26
5
PP PP-9/3
9 Agents
Agents
324
36
5
Table A.J. The numher of agents, the dimensions of state and observation spaces, and the number of actions in MPE scenarios
14

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
A.2. The StarCraft Multi-Agent Challenge (SMAC)
(a) 3m
(b) 3s_vs_3z
(c) 2s3z
(d) 8m
(e) 1c3s52
(L) MMM
Figure A.2. Visualization of SMAC scenarios
The StarCraft Multi-Agent Challenge (SMAC) serves benchmark for cooperative Multi-Agent Reinforcement Learning (MARL), focusing decentralized micromanagement tasks. Based the real-time strategy game StarCraft II, SMAC requires each unit to be controlled by independent agents acting solely on local observations. It offers variety of combat scenarios to evaluate MARL methods effectively.
Scenarios
SMAC scenarios involve combat situations between an allied team controlled by learning agents and an enemy team managed by a scripted AI. These scenarios vary in complexity, unit composition, and terrain, challenging agents to use advanced micromanagement techniques such focus fire, kiting, and terrain exploitation. Scenarios end when all units on one side are eliminated or when the time limit is reached. The objective is to maximize the win rate of the allied agents. Detailed descriptions of the scenarios and unit compositions are provided in Table A.2 and Fig. A.2.
State and observation spaces
In the SMAC environment, each agent receives partial observations that contain information about visible allies and enemies within a fixed sight range of These observations do not include any global state and are specifically designed to support decentralized decision-making based only on each agent's local view of the environment.
The global state, which used during centralized training, is constructed by aggregating the features of all agents and enemies. It consists of three primary components. The ally state includes each agent's relative and y positions, health, energy, shield (if applicable), and unit type. The enemy state is similar but excludes energy, containing relative positions, health, shield, and unit type of enemies. The last action component records the most recent action taken by each agent, represented as a one-hot encoded vector. The full global state is formed by concatenating these components, and its dimensionality depends on the number agents, enemies, and available actions.
Each agent's observation vector is separately constructed from the following elements. The movement features indicate the four cardinal directions the agent can move in, resulting in fixed size of 4. The enemy features describe each observed enemy, including available action flag, distance to the agent, relative and positions, health, shield (if applicable), and unit type. The ally features encode the same types of information for all visible allies, excluding the observing agent. Finally, the own features contain the observing agent's own health, shield, and unit type.
The precise dimensions of the observation and state vectors vary depending on the specific map scenario and unit composition, and are summarized in Table A.2.
Action space
Agents can perform discrete actions, including movement in four cardinal directions (North, South, East, West), attacking specific enemy units within shooting range of 6 units, and specialized actions such as healing for units like Medivacs. Additionally, agents can perform a stop or a no-op action, the latter being restricted to dead units.
The size the action space varies depending on the scenario and is defined as Ractions = 6 + Neneeiies where represents movement, stop, or a no-op action. The inclusion of Dememies accounts for the need to specify which enemy unit is targeted
15

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
when performing an attack action. The exact size of the action space varies across different maps, summarized in Table A.2.
Reward function
SMAC uses shaped reward function R to guide learning, including components for damage dealt (Rdanage), enemy units killed Renemyykkllld and scenario victory (Rwin). The total reward defined as:
(Hcalth(e))
Σ
I((Health(e) = 0}) Rewardaeath + [I( True}) Rewardwin
eCenemies
Cenemies
Here, Health(e) represents the health of an enemy unit e, and AHealth(e) is the reduction in its health during a timestep. The indicator function I(-) returns if the condition inside is true and O otherwise. The parameters Rewarddeat and Rewardwin are scaling factors for rewards when an enemy unit is killed and when the agents win the scenario, set to 10 and 200, respectively.
Map
Ally Units
Enemy Units
State Dimension
Obs Dimension
Num. of Actions
3m
3 Marines
3 Marines
48
30
9
38_v8_3z
3 Stalkers
3 Zealots
54
36
9
2s3z
2 Stalkers,
2 Stalkers,
120
80
11
3 Zealots
3 Zealots
8m
8 Marines
8 Marines
168
80
14
lc3s5z
Colossus,
1 Colossus, 270
162
15
3 Stalkers,
3 Stalkers,
5 Zealots
5 Zealots
MMM
Medivac,
1 Medivac, 290
160
16
2 Marauders,
2 Marauders,
7 Marines
7 Marines
Table A.2. The number of agents, the dimensions state and observation spaces, and the number actions SMAC scenarios
16

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
B. Implementation Details
In this section, we provide detailed implementation of the proposed WALL framework. Section B. outlines the imple- mentation of the Transformer used efficiently identify critical initial steps. Section B.2 elaborates on the components constituting the Wolfpack attack and provides an in-depth explanation of the reinforcement learning implementation.
B.1. Practical Implementation of Planner Transformer
$ s++
0
1+1
st+
0 t+2
$ Stttwp
① Otttwp
Planning Transformer
St
a
S t+1
1+1
a++
S B
Otttwp-1
a acttwp-1
(a) Planning Transformer
A@WP
DeW?
AQWP
AQWP
Q - -diffference Transformer
St
St+1
(b) Q-difference Transformer
Figure B. I. Structure of Transformers
Critical attacking step selection introduced in Section 4.4 requires planning to compute the reduction in Q-function values, AQMP (l=t,.... + 1), over multiple time steps. To facilitate this process, Transformer model is employed. During training, the Transformer predicts (&++1, 0,+1, Stttwes Óe+swe) the current step compute the target ÁQ However, performing this planning process each evaluation step calculate the probability of initial attack, Pt,ttaacks is computationally expensive.
To address this, the single Transformer is split into two components: planning Transformer and Q-difference Transformer. The planning Transformer predicts ($ 6(+1) and used only during training, while the Q-difference Transformer predicts AQUPP t, ,t + J. - 1) and is employed exclusively during evaluation. This separation enables efficient computation of Pr.attack during evaluation, significantly reducing computational costs. Both Transformers adopt the Decision Transformer structure (Chen et al., 2021), consisting of Transformer decoder layers. The planning Transformer is parameterized by Oplanning and the Q-difference Transformer by Aqdift Their respective loss functions are defined as:
Eplanning ( (planning) = E [||s111 $141 I1? + ||0:41 6 (Oplaniinnn))
(B.1)
Lqgitt (Deditt) E || | WP ÁQ,
WP (Qqaitt) |12] VI.
(B.2)
As shown in Fig. B.I, the estimated state 1 and observations 6,41 are generated by the planning Transformer, which takes previous trajectories as input. Similarly, the estimated Q-difference ÁQWP is produced by the Q-diff Transformer, using previous states input. The loss function Cplauning minimizes the prediction error for the next state $t+1 and observation o.+1, while Lediff minimizes the prediction enror for AQMP resulting from the Wolfpack attack. Fig. B.1 illustrates the architectures of the Transformers, where (a) represents the planning Transformer and (b) represents the Q-diff Transformer.
B.2. Detailed Implementation of WALL
The WALL framework trains robust MARI policies to counter the Wolfpack adversarial attack by employing a Q-learning approach within the CTDE paradigm. Each agent computes its individual Q-values, ("(Ti, at), i= 1, using separate Q-networks. These individual values, along with the global state, are combined through mixing network to produce the total Q-value, Qaat, parameterized by 6. This joint value function ensures effective coordination among agents under adversarial scenarios.
17

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
To achieve robustness, the training process minimizes the temporal difference (TD) loss, which incorporates the observed rewards, state transitions, and target Q-values parameterized by separate target network, 0- By leveraging this target network, the CTDE frameworks stabilize learning and mitigate the impact of the Wolfpack attack. The TD loss is defined as:
(0) = Ess..r,s' (re max (8141, - Q8 av) )
(B.3)
a'
where the target network parameter 0- is updated by applying the exponential moving average (EMA) to 0. This training mechanism allows agents to adapt and develop robust policies capable of resisting the coordinated disruptions caused by the Wolfpack adversarial attack, ensuring enhanced performance and resilience MARL scenarios. In addition, we use value-based CTDE algorithms as baselines for the WALL framework: QMIX, VDN, and QPLEX. Below, we provide an outline of the key details of these baseline algorithms:
Palue-Decompositionn Networks (VDN)
VDN (Sunehag et al., 2017) is a Q-learning algorithm designed for cooperative MARL. It introduces an additive decomposi- tion of the joint Q-value into individual agent Q-values, enabling centralized training and decentralized execution. The joint action-value function, Qtot, is expressed as:
(so ar) = a()
(B.4)
i=1
allowing agents to act independently during execution by relying only on their local Q' values.
QMIX
QMIX (Rashid et al., 2020) extends VDN by introducing a more expressive, non-linear representation of the joint Q- value, while maintaining monotonic relationship between Qtat and individual agent Q-values, Q' (²(",aa). This ensures individual-global-maa (IGM) condition:
aQ
0, Vi,
(B.5)
aQ'
guaranteeing consistency between centralized and decentralized policies. Specifically;
arg aaggaxx. (* at),
arg max (Se, ar)
(B.6)
arg m
QMIX combines agent networks, a mixing network, and hypemetworks, where hypemeeworrss dynamically parameterize the mixing network based on the global state 8t. The weights generated by the hypernetworks are constrained to be non-negative to enforce the monotonicity constraint.
QPLEX
QPLEX (Wang et al., 2020b) introduces duplex dueling architecture enhance the representation of joint action-value functions while adhering to the IGM principle. QPLEX reformulates the IGM principle in an advantage-bia based form:
aarg max," A (' aa),
arg max A (T, (T, a)
(B.7)
arg m n d"(r",a")
where to and A are the advantage functions for joint and individual action-value functions, respectively. The joint action-value function is expressed as:
Qiol ar) ('(,Taa)) + - (T,A"),aa)
(B.8)
where A1(T, a) > 0 are importance weights generated using multi-head attention mechanism to enhance expressiveness.
Here, VDN and QMIX are implemented using the PyMARL codebase http://ggitubbcoommovvvvh//ppppp while QPLEX implemented using official codebase
18

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
C. Experimental Details
All experiments this paper are conducted on a GPU server equipped with an NVIDIA GeForce RTX 3090 GPU and AMD EPYC 7513 32-Core processors running Ubuntu 20.04 and PyTorch. We follow the implementations and loss scales provided by the CTDE algorithms and focus on parameter searches for hyperparameters related the proposed Wolfpack adversarial attack. Comparisons performed using the optimal hyperparameter setup, with ablation study available in Appendix F.
C.1. Hyperparameter Setup
We conduct parameter search for the number of Wolfpack attacks Kwp 2,, the attack duration tyr [1, 2,,2,3 2, 3,, the number of follow-up agents and the temperature T E 0.,1, 0.5, 1.0]. The total number of attacks K is then determined based on K = Kwp x (twp + 1). separated into training and testing setups. During training, K is selected through hyperparameter sweeping to ensure optimal performance. For testing, K is unified across all adversarial attack setups, including Random Attack, EGA, and the Wolfpack Adversarial Attack, to ensure fair comparisons. Additionally, the attack period chosen based on the average episode length of SMAC scenarios and the total number of attacks, with L 20 is fixed appropriate. Transformer hyperparameters, shared between the Planning Transformer and Q-difference Transformer, such as the number of heads, decoder layers, embedding dimensions, and input sequence length, selected to balance accuracy and computational efficiency.
The Q-learning hyperparameters (shared across all CTDE methods) and those specific to the CTDE algorithms are detailed in Table C.1 and Table C.2, respectively. The Wolfpack adversarial attack-related hyperparameters for the WALL framework, shared across all SMAC scenarios and scenario scenario-specific specific setups, are presented in Table C.3.
Hyperparameter
Value
Epsilon
1.0 -> 0.05
Epsilon Anneal Time
50000 timesteps
Train Interval
episode
Gamma
0.99
Critic Loss
MSE Loss
Buffer Size
5000 episodes
Batch Size
32 episodes
Agent Learning Rate
0.0005
Critic Learning Rate
0.0005
Optimizer
RMSProp
Optimizer Alpha
0.99
Optimizer Eps
1e-5
Gradient Clip Norm
10.0
Num GRU Layers
RNN Hidden State Dim
64
Double Q
True
Table C.I. Common Q-learning hyperparameters
19

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
Hyperparameter
VDN
QMIX
QPLEX
Mixer
VDN
QMIX
QPLEX
Mixing Embed Dim.
32
32
Hypernet Layers
2
Hypernet Embed Dim.
64
64
Adv Hypernet Layers
1
Adv Hypemet Embed Dim.
64
Num. Kernel
2
Table C.2. VDN, QMIX, QPLEX hyperparameters
Common Hyperparameters
Value
Attack duration (twP)
Temperature (T)
0.5
Attack Period (L)
20
Num. Transformer Head
Num. Transformer Decoder Layer
Transformer Embed Dim.
64
Input Sequence Length
20
Scenario
PP_3/1
PP_6/2
PP_9/3
Num. Total Attacks (Train) (K)
4
Num. Total Attacks (Test) (K)
4
4
Num. Wolfpack Attacks (Kwr)
1
1
Num. Follow-up Agents (m)
3
5
Scenario
3m
3s_vs. 3z
2s3z
8m
Ic385z
MMM
Num. Total Attacks (Train) (K)
16
12
16
16
Num. Total Attacks (Test) (K)
4
8
8
Num. Wolfpack Attacks (Kwp)
2
4
3
4
4
Num. Follow-up Agents (m)
2
4
4
Table C.3, Wolfpack hyperparameters shared across scenarios and -cenario-specific values
20

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
D. Details of Other Robust MARL Methods
In this section, we detail various robust MARL methods compared against the proposed WALL framework, as below:
Robust Adversarial Reinforcement Learning (RARL)
RARL (Pinto al., 2017) enhances policy robustness by training protagonist agent and adversary in two-player zero-sum Markov game, At each timestep the agents observe state and take actions at ((ss) and az ~ n(r)), where is the protagonist's s policy, and y is the adversary's s policy. The state transitions follow:
Sc+1
P(se, di az).
(D.9)
The protagonist maximizes its cumulative reward R², while the adversary minimizes it:
R. min max R (H, max min R (H,
(D.10)
Robustness via Adversary Populations (RAP)
RAP (Vinitsky et al., 2020) improves robustness by training agents against a population adversaries, reducing overfitting specific attack patterns. During training, adversary is sampled uniformly from the population Tél Rdgi .... a The objective is:
min (1, .n....
Σ
E'rrss, a, ani) Te, Th.
(D.11)
1==
where Te is the agent's 's policy, To, is the i-th adversary, and a controls adversary strength.
Robust Multi-Agent Coordination via Evolutionary Generation of Auxiliary Adversarial Attackers (ROMANCE) ROMANCE (Yuan al., 2023) generates diverse auxiliary adversarial attackers improve robustness in CMARL. Its objective combines attack quality and diversity:
Tapa
T. (1b).
(D.12)
nhp
where Lopt minimizes the ego-system's 's return, Ldiv promotes diversity using Jensen- Censen-Shannon Divergence, and np the number of adversarial policies. ROMANCE evolutionary mechanism to explore diverse attacks.
We implement RARL and RAP for multi-agent systems, as well as ROMANCE with EGA, using the ROMANCE codebase available at https//jiitbb.com/zzzbb//MAAA
ERNIE
ERNIE (Bukharin et al., 2024) improves robustness by promoting Lipschitz continuity through adversarial regularization. minimizes discrepancies between policy outputs under perturbed and non-perturbed observations:
(ox: 0x) max D(TO, (or + +5) no. (ox));
(D.13)
where oę is the agent's 's observation, is bounded perturbation, and D measures divergence (e.g.. KL-divergencee. ERNIE reformulates adversarial training as Stackelberg game and extends its framework mean-field MARL for scalability in large-agent settings. We evaluate ERNIE using its official codebase at http::/gittub.om//bbbkhennnnnt
21

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
E. Additional Experiments Results
This section presents additional experimental results. E.1 reports comparison results for other CTDE algorithms, E.2 provides learning curves across additional SMAC scenarios, E.3 presents a performance comparison for EGA, E.4 discusses computational cost, and E.5 includes general robustness experiments.
E.1. Comparison Results for Other CTDE Algorithms
Our proposed Wolfpack attack is compatible with various value-based MARL algorithms. Experimental results in this section demonstrate its significant impact on robustness, not only in QMIX, discussed in the main text, but also in VDN and QPLEX. The hyperparameters used in these experiments follow those outlined in Appendix C.I, with WALL hyperparameters remaining consistent across all algorithms, including QMIX.
VDN Results: Table E. shows the average win rates for various robust MARL methods with VDN against attacker baselines. Both models and attackers are trained using the IDN-based algorithm. Results indicate that the proposed Wolfpack attack more detrimental than Random Attack or EGA for VDN, For example, while EGA reduces Vanilla VDN's performance by 95.6% 73.8% 21.8%. the Wolfpack attack causes a larger reduction 95.6% 44.1% 51.5%, demonstrating its severity. Additionally, the WALL framework outperforms other baselines under both Natural conditions and adversarial attacks, showeasing its robustness.
Secnario
Method
2s3z
3m
3s.vs.34
8m
MMM
Ic3s5z
Mean
Vanilla VDN
98.5 1.4
96.0 3.0
0.5
97.8
2.0
0.5
83.5 + 9.5
95.6 1.7
RANDOM
99.0
0.5
98.9
97.6 0.5
98.9 0.4
RARI.
95.3+
3.4
0.5
96.3 1.5
91.8 0.3
94.7 0.6
Natural
93.4
0.5
98.7 1.0
97.3
97.3 0.5
97.1
ERNIE
9.9
99.4 0.5
97.1 0.5
98.4 1.4
98.8 0.8
96.5
97.5
ROMANCE
9.1
97.9 1.1
99.4 0.5
93.5 5.4
97.9
93.0
96.6
WALL (ours)
0.1
99.4 0.5
99.9 0.1
96.9 1: 2.0
99 0.5
100.0
99.2 - 0.2
Vanilla VDN
1.5
96.5 0.5
52.5 10.5
95.0 1.0
76.5
77.8 1.9
RANDOM
83.0+
94.5
1.5 1.5
0.7
0.7
88.5 + 0.5
2.5
93.5+11
91.8 0.5
RARI.
4.2
64.5
97.3 0.5 0.5
72.9 72.9
76.3
92.0+00.2
80.7 2.1
Random Attack
RAP
8.1
37.9
9.9
8.5
9..
92.3 0.5
88.7 + 1.6
ERNIE
2.0
66.9
93.1
67.1
89.1
90.4 4.5
81.1 +
ROMANCE
91.0 5.0
6.0
98.4
54.0 5.0
0.5
92.0
85.3 0.6
WALL (ours)
95.5+11
86.0
9..4+
90.0 + 3.0
98.5 + 0.5
98.1 0.1
94.6 1.4
Vanilla VDN
69.5
54.5
12.5
94.5 1.5
59.5 12.5
90.5
74.5 9.5
73.8 1.0
RANDOM
56.0+66.0
6.0
84.0+13.0
84.5 6.5
85.5
86.5 3.5
78.3
RARL
1.2
7..0
96.5 0.7
7..7 4.0
75.9
81.3 4.5
77.9
EGA
RAP
3.1
4.6
0.5
85.7
91.8
87.2 1.5
89.3
ERNIE
62.0
6.2
89.1
74.7 1.6
89.3 2.5 2.5
82.2 1.9
76.6 3.6
ROMANCE
86.5
93.8 3.0
98.0 0.2
76.5 0.5
95.5 2.5 2.5
92.0 0.1
90.3
WALL (ours)
91.5 0.5
91.0++2.0
99.0 1.0
90.0 + 4.0
97.5 + 0.5
95.5 + 0.5
94.1 0.1
Vanilla VDN
54.0+ 4.0
20.5 5.5
91.3 0.5 0.5
24.5 12.5
18.5 9.5
55.5 2.5
2.5
44.1 1.1
RANDOM
4.0
69.0
90.0
41.0
18.5 5.5
83.5 0.5
61.5 2.7
Wolfpuck
RARL
8.7
41.3 1 16.4
96.8 0.9 0.9
13.1 1: 2.0
24.3 1 6.5
61.6 11.6-117 11.7
49.4 1 1.8
Adversarial
RAP
3.5
67.7 33.8
98.9 0.1
931
53.3 1 2.5
82.1 5.4
65.0 1 1.7
Attack (ours)
ERNTE
33.1 = 3.6
25.8
93.0 4.5
17.2
23.5
9.3
66.4 13.5
43.2 + 1.7
ROMANCE
63.0 10.0
46.5 20.5
97.5 0.5
19.5 7.5
38.0 +.0
83.0 + 4.0
57.9 3.1
WALL (ours)
91.5 + 2.5
91.5 2.5
100.0 + 0.0
71.5 + 1.5
98.0 1.0
93.5 4.5
91.0 0.8
Table E.I. Average test win rates of robust MARL policies under various attack settings (VDN)
22

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
Similarly, Table E.2 reports the average win rates various robust MARL methods with QPLEX against attacker baselines. Both models and attackers are trained using the QPLEX-based algorithm. Results reveal that the Wolffack attack is also more detrimental for QPLEX compared to Random Attack and EGA. For instance, EGA reduces Vanilla QPLEX's performance by 98.4% 57.2% = 41.2 41.2%, whereas the Wolfpack attack results in a larger reduction of 98.4% 33. 13.1% 1% = 65.3%. The WALLL framework again demonstrates superior robustness, performing well against all attacks, including the Wolfpack attack.
QPLEX Results:
Scenario
2s3z
3m
38_vs
8m
MMM
Ic3s5z
Mean
Method
Vanilla QPLEX
97.1 + 1.3
99.2 + 0.5
99.4 0.5
97.4 0.3
99.5
97.1 + 0.9
98.4 + 0.1
RANDOM
97.7 = 2.1
98.7 = 0.8
99.6
0.1
99.1
1 0.9
9.2
98.5 + 1.3
98.7 0.5
RARL.
94.4 + 4.5
89.7 + 2.0
88.4 6.8
97.8 + 1.6
9..
94.6 + 1.6
94.0 1.4
Natural
RAP
96.5 1.7
96.6 1: 2.5
93.2 0.9
99.1 1 0.5
9..
97.8 1 0.8
97.7 0.4
ERNIE
96.7
98.8 0.8
99.7
98.8
1.4
0.5
98.4
98.6 - 0.6
ROMANCE
2.9
93.2
94.7 3.2
99.7
99.0
1.2
(ours)
0.5
97.7 2.1
99.
0.1
9.T0
99.5
0.3
Vanilla QPLEX
2.5
36.2 6.6
1
4.8
6..
7..0
62.3
RANDOM
85.9 0.9
60.0
96.2=16
8.7
91.2
9
87.8 $7.8 0.7
RARI.
81.2 13.6
61.3++13. 13.9
74.7+
73.7+2116
92.0+0088
9.9
79.1 79.1+44. 4.3
Random Attack
RAP
90.8 = 2.8
7.0
92.1
4.5
9.
9.9
85.0 1.9
ERNIE
$2.0 2.0
69.0
93.0=16
8..0
0.9
93.0
83
ROMANCE
89.2
64.9 13.4
93.7
51.6.15.3
92.2
1.0
76.0
WALL (ours)
97.4 0.9
85.4
9.5
9.5
1.2
98.6
0.7
Vanilla QPLEX
48.1
5.1
12.5
16.8
7.0
1.5
57.2 6.2
RANDOM
60.5 6.8
61.4+14.9
82.0=6.6
8..2
85.3
88.8
77.4+3.1 3.1
RARL
63.4 0.5
6..7
71.0=82
6..0
89
84.7
76.7
EGA
RAP
7
8..
8.34
89.0
92.7
96.7
88.0 0.7
ERNIE
7.0
10.33
51.7 9.5
8..7
8..0
4.5
74.2 3.0
ROMANCE
6.0
80.3 1.7
9.3
8.3
95.2
92.5
85.6 1.0
WALL (ours)
89.0 3.8
83.9 2.9
99.6
1.2
9.2
96.1=0.7
93.2 0.7
Vanilla QPLEX
30.8
11.8 7.5
63.7 24.
11.7
12.5
6.1
33.1
RANDOM
50.5
6.8
89.5
45.6
34.0 14.1
82.5 11.3
53.2 4.9
Wolfpack
RARL
55.5
27.3 9.2
78.0 4.3
44.
2133
79.7
51.3
Adversarial
RAP
59.0
50.3 + 14.7
86.7 86.7+8.2
33.7
4.00
93.7 9.4
56.3 3.2
Atlack (ours)
ERNIE
52.9 6.3
49.0 1: 9.9
76.0 9.9
42.7 10.6
20.7 10.4
78.7 6.7
53.3 3.6
ROMANCE
57.3 17.3+280 S.4
38.5
+ 13.9
89.7 1.3
28.6
46.2 10.6
83.5 + 5.5
50.8 0.9
WALL (ours)
88.3 1.2
87.6 4.7
99.7 0.5
84.5 1.7 1.7
96.3 3.9
99.3 0.9
92.6 1.3
Table Average test win of robust MARL policies under various attack settings (QPLEX)
23

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
Learning Curves for VDN and QPLEX: We also analyze training curves and average test win rates across different CTDE algorithms. Graphs for the 8m and MMM environments illustrate the average win rates of each policy training steps under unseen Wolfpack adversarial attacks. Fig. E. presents training curves for VDN, while Fig. E.2 shows results for QPLEX. These curves highlight that WALL not only achieves greater robustness but also adapts more quickly to attacks across VDN and QPLEX, further confirming its effectiveness beyond QMIX.
100
100
Test Win Rate (%)
WALL
Test Win Rate (%)
80
VDN
80
RANDOM
60
RARL
60
RAP
ERNIE
40
40
ROMANCE
20
20
8.0
8.0
0.5
1.0
1.5
2.0
2.5
3.0
0.5
1.0
1.5
2.0
2.5
3.0
Timesteps (10%)
Timesteps (106)
(a) 8m
(b) MMM
Figure E.I. Learning curves of MARL methods for Wolfpack attack (VDN)
100
100
Test Win Rate (%)
WALL
Test Win Rate (%)
80
QPLEX
80
RANDOM
60
RARL
60
RAP
ERNIE
40
40
ROMANCE
20
20
0
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.5
1.0
1.5
2.0
2.5
3.0
Timesteps (106)
Timesteps (106)
(a) 8m
(b) MMM
Figure E.2, Learning curves of MARL methods for Wolfpack attack (QPLEX)
24

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
E.2. Learning Curves Across Additional SMAC Scenarios
In this section, we provide the training performance of WALL under Wolfpack adversarial attack across additional SMAC scenarios beyond the 8m and MMM environments, which emphasized in the main text for their significant performance differences. Fig. E.3 illustrates the training curves for 6 scenarios: 3m, 3s_vs_3z, 2a3z, 8m, MMM, and 1c3s5z. The results demonstrate that WALL consistently outperforms baseline methods, achieving superior win rates across all scenarios. Additionally, policies trained with WALL adapt more quickly to the challenges posed by Wolfpack attack, showing robust and efficient performance across environments of varying complexity. These findings highlight the effectiveness of WALL in enhancing the robustness of MARL policies against coordinated adversarial attacks.
100
100
100
Test Win Rate (%)
WALL QMIX
Test Win Rate (%)
Test Win Rate (%)
60
RARL
60
60
RAP
40
40
40
ROMANCE
8.0
0.5 1.
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Timesteps (109)
Timesteps (10%)
Timesteps
(10%)
(a) 3m
(b) 3s.vs 3z
(c) 2s32
100
100
100
Test Win Rate (%)
Test Win Rate (%)
Test Win Rate (%)
80
60
60
60
40
40
40
20
20
20
8.0
8.0
8.0
0.5
1.0
1.5
2.0
2.5
3.0
0.5
1.0
1.5
2.0
2.5
3.0
0.5
1.0
1.5
2.0
2.5
3.0
Timesteps (10%)
Timesteps (10%)
Timesteps (10%)
(d) 8m.
(e) MMM
(1) 1c3s5z
Pigure E.3. Learning curves MARL, methods Wolfpack attack across SMAC scenarios (QMIX)
25

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
E.3. Performance Comparison for EGA
This section presents the training performance of WALL under the existing Evolutionary Generation-based Attackers (EGA) (Yuan et al., 2023) method across six SMAC scenarios, shown in Fig. E.4. The results demonstrate that WALL consistently achieves superior robustness across all scenarios, even against unseen EGA adversaries that not included in its training process.
The EGA framework generates a diverse and high-quality population of adversarial attackers. Unlike single-adversary methods, EGA maintains an evolving archive of attackers optimized for both quality and diversity, ensuring robust evaluations against various attack strategies. During training, attackers are randomly selected from the archive simulate diverse attack scenarios. The archive iteratively updated by replacing low-quality or redundant attackers with newly generated ones. For evaluation, attacker policy is randomly selected from the archive. The chosen attacker identifies critical attack steps and targets specific victim agents, introducing action perturbations to reduce their individual Q-values. WALL demonstrates strong resilience in these challenging environments, effectively mitigating the impact of EGA adversaries and maintaining high performance across all evaluated scenarios.
100
Test Win Rate (%)
WALL
Test Win Rate (%)
Test Win Rate (%)
80
80
80
Random
60
MARL
60
60
RAP
40
ERNIE
40
40
ROMANCE
20
20
20
0.0
0.5
1.5 2.0 2.5
0.0
0.5
2.0 2.5
0.0
0.5
1.5
2.0 2.5
Timesteps (10%)
Timesteps (10%)
Timesteps (10%)
(a) 3m
(b) 3s.vs... vv.32
(c) 2s32
Test Win Rate (%)
Test Win Rate (%)
Test Win Rate (%)
80
80
80
60
60
60
40
40
40
20
20
20
8.0
8.0
8.0
0.5
2.0
2.5
3.0
0.5
1.5
2.0
2.5 3.0
0.5
1.5
2.0 2.5
Timesteps (10%)
Timesteps (10%)
Timesteps (10%)
(d) 8m
(c) MMM
(f) c3352
Figure E.4. Learning curves of MARL methods EGA across SMAC scenarios (QMIX)
26

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
E.4. Computational Cost
We measured the total training time (for 3M steps) required for each algorithm in the SMAC environments. The results are summarized in Table E.3. WALL requires approximately 30% more time than ROMANCE, primarily due to the use of the Transformer. However, despite this additional cost, the Transformer-ben effectiveness of our method, as it enables precise identification of attack timing where coordinated perturbations are most disruptive. This capability allows the robust policy to anticipate and defend against strategically timed adversarial threats, ultimately resulting in stronger robustness. In general, training on the MMM map takes longer than on the 8m map due to the increased number agents, which leads to higher computational costs as result of scalability.
critical step selection plays essential role in the
Scenario
8m
MMM
Method
Vanilla QMIX
6h 30m
7h 30m
RANDOM
6h 30m
7h 35m
RARL
11h 10m
15h 50m
Training Time
RAP
14h 35m
17h 25m
ERNIE
16h 30m
20h 55m
ROMANCE
16h 35m
18h 45m
WALL (ours)
21h 05m
23h 30m
Table E.3. Training time comparison on 8m and MMM
E.5. General Robustness Experiments
We conducted additional experiments to evaluate robustness under other commonly used criteria in the 8m and MMM environments. Specifically, we considered: (1) Gaussian observation noise: Standard Gaussian noise (mean 0, standard deviation 1) is injected into agents' local observations, where attack steps (total attack steps for 8m, 16 for MMM) and the attack group selected randomly; and (2) Different parameterization in the SMAC test environment: Robustness is evaluated under perturbed unit attributes by reducing the initial health of allied units by 10%, 15%, and 20% in the test setup, compared to the training configuration. These types of noise and perturbations are standard in robustness evaluations and enable us to assess how well the proposed method generalizes beyond action-level perturbations. The Table E.4 below summarizes the results. In both settings, WALL demonstrates consistently stronger performance than existing baselines, suggesting that generalizes well to broader forms of distributional shift.
Scenario
8m
MMM
Mean
Method
Vanilla QMIX
62.6 2.3
75.3 75.3 3.1
69.0 2.1
RANDOM
72.3 4.2
79.3 19.3:3.0 3.9
75.8 + 7.2
Gaussian Obs. Noise
ROMANCE
69.6 11.2
76.6 7.5
73.1 + 6.6
WALL (ours)
91.3 3.3
97.3 + 1.7
94.3 + 3.3
Vanilla QMIX
52.8
88.4 2.7
70.6 1.8
RANDOM
50.8
9..4
73.1 5.2
Ally HP L 10%
ROMANCE
16.0
4.5
7
WALL (ours)
7..2
98.6+1.1
85.9 3.7
Vanilla QMIX
47.4
6..0+5.4
58.2 5.8
RANDOM
4.52
81.4+5.5
65.3 + 4.6
Ally HP 15%
ROMANCE
57.6 15.7
3..
69.4 + 7.8
WALL (ours)
69.2 10.5
94.0
5.7
Vanilla QMIX
0.4
4.5
20.8 2.1
RANDOM
0.0
6.70
32.5 1 1.3
Ally HP L. 20%
ROMANCE
0.44
8.90
29.7 4.3
WALLI (ours)
1.9
89.6 + 5.6
47.0 + 3.3
Table E.4. Average test rates robust MARL policies under various perturbation settings
27

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
F. Additional Ablation studies
In this section, we provide additional ablation studies on number of Wolfpack adversarial attacks Kwp and the attack duration twp in the 8m and MMM environments, where the performance differences between WALL and other robust MARL methods are most pronounced.
Number of Wolfpack Attacks KwP: The hyperparameter Kwp determines the number of Wolfpack attacks, with each attack consisting of an initial attack and follow-up attacks over Iwp 3 timesteps. The total number of attack steps K for Wolfpack attack is then calculated as K = 4 x Kwp. In this section, we conduct a parameter search for KwP E [1, 2, 3, 4]. Fig. F.I illustrates the robustness of WALL policies trained with different KwP values under the default Wolfpack attack in and MMM. both environments, having too small Kwp results in insufficiently severe attacks, which leads to reduced robustness of the WALL framework. Conversely, in the 8m environment, excessively large Kwr values create overly devastating attacks, making it difficult for CTDE methods learn strategies counter the Wolfpack attack, which degrades learning performance. Therefore, optimal Kw exists in both environments: Kwp for 8m and Kwp 4 for MMM, which we choose as the default hyperparameters.
Test Win Rate (%)
(Kar
Test Win Rate (%)
(Kar
80
80
(Kar
(Kwr
60
60
40
40
20
20
8.0
8.0
0.5
1.0
1.5
2.0
2.5
3.0
0.5
1.0
1.5
2.0
2.5
3.0
Timesteps (106)
Timesteps (106)
(a) 8m
(b) MMM
Figure F.I. Number of Wolfpack attacks (Kwp)
Attack duration twr: The hyperparameter twr determines the duration of follow-up attacks after the initial attack. To analyze its impact robustness, Fig. F.2 compares performance for (wp [1, 2, in the 8m and MMM environments. As shown in the figure, similar to the case of Kwr, setting twr too low results in insufficient follow-up attacks on assisting agents, reducing the severity of the attack and lowering the robustness of WALL. On the other hand, excessively high twp values lead to overly severe attacks, making challenging for WALL to learn effective defenses against the Wolfpack attack. Both environments demonstrate that /wp yields optimal performance and is selected as the best hyperparameter.
Test Win Rate (%)
Test Win Rate (%)
80
80
60
60
40
40
WALL 16 1)
WALL 16w 2)
20
20
WALL 16e -
WALL 16 4)
8.0
8.0
0.5
1.0
1.5
2.0
2.5 3.0
0.5
1.0
1.5
2.0
2.5
3.0
Timesteps (106)
Timesteps (106)
(a) 8m
(b) MMM
Figure F.2. Attack duration (twp)
28

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
G. Additional Visualizations of Wolfpack Adversarial Attack
G.I. Visualization of Wolfpack Adversarial Attack Across Additional SMAC Scenarios
(a)
5 (Eiit))
t 6-8
10~19
20-29
Terminal
(mi + 1, , Liitt
Lup
Kill Enemy
VICTORY
Kill Enemy
Ally
SSenmy
Kill Enemy
(b)
Hit Enemy
Killed by
DEFEAT
Killed by Enemy
Enemy
Move to Hit
Guard
Killed by
Atsak
Woltpack Attack
Enemy
(c)
Enemy
Hit Enemy
VICTORY
Move to Hit
Guard
Kill Enemy
Rttak
Wolfpack Attack
Kill Enemy
Figure G.I. Attack comparison 8m task in the SMAC: (a) QMIX/Natural, (b) QMIX/Wolfpack attack, and (c) WALL/Wolfpack attack
To analyze the superior performance of the Wolfpack attack, we provide visualization of its execution in various SMAC environments. Fig. 8 illustrates the 2s3z task, while Fig. G.1 visualizes the 8m task, and Fig. G.2 presents the MMM task.
Fig. Ga) illustrates Vanilla QMIX operating in natural scenario without any attack, where the agents successfully defeat all enemy units and achieve victory. In this scenario, agents with low health continuously move to the backline to avoid enemy attacks, while agents with higher health position themselves at the frontline to absorb damage. This dynamic coordination enables the team to manage their resources effectively, withstand enemy attacks, and secure a successful outcome.
Fig. Gbb depicts Vanilla QMIX under the Wolfpack adversarial attack, where initial attack is launched at / 5, and follow-up agents targeted between agents, preventing them from guarding the targeted ally or engaging the enemy effectively, Between = 10 and / 19, the initial agent continues to take focused enemy fire, eventually succumbing to the attacks and being eliminated. This disruption renders the remaining agents ineffective in defending against the adversarial attack, leading to a loss as all agents are defeated.
and 8. Agents with higher remaining health selected follow-up
Fig. Gc) shows the policy trained with the WALL framework. During selected follow-up agents and subjected the Wolfpack attack, limiting their ability to guard or engage the enemy. Nevertheless, the non-attacked agents adjust by forming wider formation vertically, effectively dispersing enemy firepower while delivering coordinated attacks. Additionally, agents with higher health move forward to guard the initial agents, ensuring that the initial agents do not die. This tactical adaptation enables the team to eliminate enemy units and secure victory.
8, the same agents as in (b) are
29

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
(miit)
7-9
t 10~19
t 20-29
t= Terminal
Fintt
twp
VICTORY
Hit Enemy
Hit Enemy
Kill Enemy
Ally
Eremmy
(b)
SHeal
Hit
Ally
Enemy
Killed by Enemy
Killed by Enemyy
DEFEAT
Move to Hit
Guard
Lolk
Killed by Enémy
(c)
Heal
Hit
Hit
Ally
Enemy
Kill Enemy
Enemy
VICTORY
Move to Hit
Guard
Kill Enemy
Atack
Figure G.2. Attack comparison on MMM task in the SMAC: (a) QMIX/Natural, (b) QMIX/Wolfpack attack, and (c) WALL/Wolfpack attack
Fig. G.2(a) showcases Vanilla QMIX operating in a natural scenario without any adversarial interference, where all enemy units are successfully eliminated, leading to a decisive victory. During this process, agents with lower health retreat to the back while the Medivac agent provides healing, and agents with higher health move forward absorb enemy attacks. This coordinated strategy enables the team to secure victory efficiently.
Fig. G.2(b) illustrates Vanilla QMIX under the Wolfpack adversarial attack. An initial attack is launched t = 6, followed by the targeting of four follow-up agents between f 7 and t 9. The follow-up agents selected include one healing agent attempting to heal the initial agent, one guarding agent positioned to protect the initial agent, and two agents actively engaging the enemy targeting the initial agent. Due to the Wolfpack attack, the initial agent failed to receive critical healing or guarding support at = 10 and = 19, leading to its climination. This disruption severely hinders the remaining agents' ability to defend against the adversarial attack, ultimately resulting in a loss as agents are defeated.
Fig. G.2(c) illustrates the policy trained with the WALL framework. During / selected follow-up agents and subjected the Wolfpack attack, restricting their actions such as healing, guarding, targeting the enemy effectively. However, the non-attacked agents adapt by positioning themselves ahead of the initial agent to provide protection and focus their fire on the enemies targeting the initial agent. This strategic adaptation enables the team to successfully repel the adversarial attack, eliminate enemy units, and secure victory. Notably, in the terminal timesteps, more agents survive under the WALL framework compared to the natural scenario depicted (a), highlighting the enhanced robustness and stability of the policy learned with WALL.
9, the same agents in (b)
30

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
G.2. Additional Analysis of Follow-up Agent Group Selection
Heal
Heal
Ally
Ally
Hit Enemy
Killed by Enemy
Hit Enemy
Guard
Iniraa Ar
Walfoack Btack
Hollpack
Guard
(a) Initial attack
(h) Follow-up (1.2)
(c) Follow-up agent group selection
Pigure G.3. Visualization of follow-up agent group selection comparison for the MMM task in SMAC
In this section, we demonstrate that the proposed uollow-up Agent Group Selection method effectively identifies responding agents that protect the initially attacked agent. Fig- G.3 visualizes the process of selecting the follow-up agent group after an initial attack. By comparing the proposed method with baseline method, Follow-up (L2), which selects agents closest the initial agent based on observation 1.2 distance, show that method better identifies responding agents, enabling more impactful Wolfpack attack.
Fig. G.3(a) illustrates an initial attack on agent 8, preventing it from performing its original action of hitting the enemy and forcing it to move forward, exposing it to enemy attacks. Fig. G.3(b) shows the Follow-up (1.2) method selecting agents 3, 4, 7, as the follow-up agents based on their proximity to the initial agent. Despite the follow-up attack, non-attacked agents, which is far from the initial agent in terms of observation L2 distance, such as agent 10, heal the initial agent, while agent 2 guards it, effectively protecting the initial agent from the attack.
Fig. G.3(c) illustrates the follow-up agents selected using our proposed Follow-up Agent Group Selection method. The selected group includes agents and 10, which are responsible for healing and guarding the initial agent, and agents and 9, which are hitting enemies targeting the initial agent. These agents are subjected to the follow-up attack, preventing them from performing their protective actions. As a result, the initial agent is left vulnerable, succumbs to enemy attacks, and ultimately eliminated.
31

---

Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning
220
SoftiO)
Soft10")
20
222
Probabiity
15
Ofference
1.
215
Probactiio
15
1.5
Actions
Actions
Actions
Actions
Actions
(a) Agent=1
(b) Agent=2 (selected)
(c) Agent=3
(d) Agent=4
(c) Agent=5
25
25
25
25-
Trababilt (86
Probabiity (9)
20
2220
20
15
115
15
15
Actions
Actions
Actions
Actions
Actions
(f) Agent=6 (selected)
g
Agent=7
(h) Agent=8
(i) Agent=9 (selected)
() Agent=10 (selected)
Figure G.4. Soft(Q') and Soft(Q') for each agent, along with the difference hetween the distributions.
0.06
0.05
0.04
DKL
0.03
0.02
0.01
0.00
3
6
7
9
10
Agents
Figure G.5. KL divergence values for each agent, representing the difference between Soft(Q') and Soft(Q') distributions.
Fig. G.4 illustrates the Soft(Q')) distribution and the updated Soft(Q') distribution based on Equation 1, highlighting the differences between the two distributions. It is evident that agents 2, 6, 9, 10 exhibit the largest differences in their distributions. This suggests that, following the initial attack, these agents show noticeable policy changes to adapt and defend against Additionally, Fig. G.5 presents the KL divergence values between these two distributions, further confirming that agents 2, 6, 6,9,11 10 have the highest KL divergence values.
Consequently, based on Equation 2, agents 6, 9, 10 are selected as the follow-up agent group. This aligns with the visualization results shown in Fig. G.3, demonstrating consistency in the SMAC environment.
32


Reviews:
[{"review_id": "uVC9T8RJay", "reviewer": "Reviewer_rqE9", "summary": "This paper introduces a new kind of adversarial attack, Wolfpack, and trains policies against those attacks. They find that trained models are not only more robust against Wolfpack attacks but are also more robust to other adversarial attacks (EGA) and noise injection (Random Attack).\n\nWolfpack builds upon previous work (EGA) that attacks a single agent from the MARL set-up: after the first agent is attacked (has its action replaced), “exposed” agents are identified and are also attacked. In most settings, previous robustness methods (that are not designed to be robust to Wolfpack attacks) fail to be robust to the attack.\n\nThe paper provides empirical evidence with comprehensive evidence from the Starcraft II Multi-Agent Challenge environment.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive comments. We greatly appreciate the opportunity to address the main concerns regarding general robustness and environmental diversity, and we provide detailed responses below.\n\n**General robustness beyond action perturbations:**\n\nWe appreciate the reviewer’s insightful suggestion to consider broader notions of robustness beyond action-level perturbations. While our work primarily focuses on robustness against coordinated adversarial attacks and random action replacement, we fully agree that a comprehensive understanding of robustness should also account for other forms of distributional shift, such as observation noise and parameterization mismatches in Dec-POMDPs, which are critical for real-world deployment.\n\nOur choice to focus on action-level adversarial attacks is based on their prevalence and practical relevance in the MARL literature. Prior works in robust MARL (e.g., [1–2]) commonly adopt this setting, as action perturbations directly affect decentralized decision-making, where each agent selects actions based solely on local observations and individually learned policies. Building upon this foundation, our contribution introduces coordinated multi-agent action perturbations, an attack model designed to reveal cooperation-disrupting vulnerabilities not exposed by existing single-agent or uncoordinated attacks.\n\nTo address the reviewer’s concern, we conducted additional experiments to evaluate robustness under other commonly used criteria in the 8m and MMM environments. Specifically, we considered: **(1) Gaussian observation noise:** Standard Gaussian noise (mean 0, standard deviation 1) is injected into agents’ local observations, where attack steps (total attack steps $K$: 8 for 8m, 16 for MMM) and the attack group are selected randomly; and **(2) Different parameterization in the SMAC test environment:** Robustness is evaluated under perturbed unit attributes by reducing the initial health of allied units by 10%, 15%, and 20% in the test setup, compared to the training configuration. These types of noise and perturbations are standard in robustness evaluations and enable us to assess how well the proposed method generalizes beyond action-level perturbations.\n\nThe Table R.1 below summarizes the results. In both settings, WALL demonstrates consistently stronger performance than existing baselines, suggesting that it generalizes well to broader forms of distributional shift.\n\n**Table R.1:** Performance comparison under various perturbation setups in SMAC environments.\n\n| Setting | Methods | 8m  | MMM  | Mean |\n| --- | --- | --- | --- | --- |\n| Gaussian Obs. Noise | Vanila QMIX | 62.6 | 75.3 | 69.0  |\n|  | RANDOM | 72.3 | 79.3 | 75.8  |\n|  | ROMANCE | 69.6 | 76.6 | 73.1  |\n|  | WALL (ours) | **91.3** | **97.3** | **94.3**  |\n| Ally HP ↓ 10% | Vanila QMIX | 52.8 | 88.4 | 70.6  |\n|  | RANDOM | 50.8 | 95.4 | 73.1  |\n|  | ROMANCE | 56.8 | 92.4 | 74.6  |\n|  | WALL (ours) | **73.2** | **98.6** | **85.9**  |\n| Ally HP ↓ 15% | Vanila QMIX | 47.4 | 69.0 | 58.2  |\n|  | RANDOM | 49.2 | 81.4 | 65.3  |\n|  | ROMANCE | 57.6 | 81.2 | 69.4  |\n|  | WALL (ours) | **69.2** | **94.0** | **81.6**  |\n| Ally HP ↓ 20% | Vanila QMIX | 0.3 | 41.2 | 20.8  |\n|  | RANDOM | 0.0 | 65.0 | 32.5  |\n|  | ROMANCE | 2.3 | 57.0 | 29.7  |\n|  | WALL (ours) | **4.3** | **89.6** | **47.0**  |\n\n[1] Yuan et al., \"Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attackers,\" AAAI 2023.\n\n[2] Li et al., \"Byzantine robust cooperative multi-agent reinforcement learning as a bayesian game,\" ICLR 2024.\n\n**Environment diversity:** (Common response with Reviewer YyTB)\n\nThank you for raising the important point regarding environment diversity. In line with ICML’s policy of one rebuttal per reviewer and encouraging shared responses where applicable, we kindly refer you to our detailed response under Reviewer YyTB, where we provide additional performance comparisons on the cooperative Multi-Agent Particle Environment (MPE) [3], a widely adopted benchmark in MARL. We appreciate your understanding.\n\n[3] Lowe et al., \"Multi-agent actor-critic for mixed cooperative-competitive environments,\" NeurIPS 2017.\n\n**Minor issues:**\n\nWe thank the reviewer for pointing out the typo and will correct it.\n\nOnce again, we sincerely thank the reviewer for their careful reading, insightful comments, and positive recommendation. We hope that the additional experiments and clarifications meaningfully address the reviewer’s concerns and further strengthen the impact of our work.", "claims_and_evidence": "The paper claims that WALL “significantly improves robustness compared to existing methods”. It provides evidence that WALL is more robust to action-replacement-based adversarial attacks (Wolfpack, EGA, Random Attack) than two prior methods for action-replacement adversarial attacks (ERNIE, ROMANCE) and non-action-replacement methods (e.g,. RARL and RAP). If these other methods are designed to deal with other kinds of adversaries, evidence that other methods are not robust to Wolfpack attacks may not, in itself, substantiate that WALL improves “robustness” in a broader sense, based on evidence from Wolfpack attacks.\n\nHowever, the authors also show that WALL is more robust to random noise in action replacement than other methods, which I believe is significantly more compelling evidence that the method is more robust in general. Overall, I find the range of alternative methods, environments, and attack types to provide convincing evidence that WALL is more robust.\n\nThe authors claim that their framework is novel, and they support this claim with a comprehensive review of the literature.", "methods_and_evaluation_criteria": "One main stated motivation of the work is “enhancing the robustness of CTDE”, especially to address “challenges from mismatches between training and deployment environments”. However, the evaluation of the new method is largely against adversarial attacks, particularly those where actions are replaced. The authors do measure robustness against random noise (Random Attack), but this is only against the random replacement of attacks and does not consider other threat models like noisy observations or distributional shift in the underlying Dec-POMDP between training and deployment.\nWhile this may be a common move in the literature, if the primary contribution is improved robustness to adversarial attacks, I’d like to see either an argument or evidence that this helps with generalization to deployment. Alternatively, the authors could justify why their attack model is relevant.\n\nThe choice of the SMAC environment and the choice of benchmark training methods is appropriate. However, I think a more convincing demonstration would use additional environments (beyond SMAC)  to increase diversity.\n\nOverall, I find that the paper’s proposed evaluation criteria are for justifying robustness when restricted to some adversarial attacks and random noise.", "theoretical_claims": "No theoretical claims are provided.", "experimental_designs_or_analyses": "I did not find any problems in the soundness or validity of the experimental design.", "relation_to_broader_scientific_literature": "The relation to broader scientific literature is thorough, with a wide range of previous work being explained and methods compared across benchmarks.", "essential_references_not_discussed": "No known missing references.", "other_strengths_and_weaknesses": "Strengths:\n* A major strength of this paper’s contributions is the improvement of WALL above state-of-the-art methods on Random Attack. Given the differences between Random Attack and the Wolfpack adversarial attacks, the generalisation to improve performance is impressive.\n* The paper thoroughly investigates the robustness of WALL, with substantial evidence provided in a relevant and challenging environment.\n* The breadth of environment, attack type, and robust training methods is particularly compelling.\n* The paper was presented extremely clearly, with a thorough analysis of previous literature and a clear and full explanation of the relevant background.\n\nWeaknesses:\n* As stated above, I think that the paper could have shown its contributions were even more significant for improving robustness to train-test distribution shift. For example, they could compare their method to others in a SMAC environment with different parameterization during test-time than during train-time.", "other_comments_or_suggestions": "There are some typos, for example: “Notably, despite RANDOM is trained against Random Attack and ROMANCE against EGA, WALL achieves better performance against both attack types.”", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mef1jMp4Qf", "reviewer": "Reviewer_YyTB", "summary": "This paper proposes a Wolfpack adversarial attack to train robust MARL systems. Specifically, it modifies the attack process to attack an agent and its follow-up teammates. Experiments on SMAC benchmarks demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive comments. We greatly appreciate the opportunity to address the main concerns regarding computational cost and environmental diversity, and we provide detailed responses below.\n\n**Extra computational cost:**\n\nThank you for pointing this out. To address the reviewer’s concern, we measured the total training time (for 3M steps) required for each algorithm in the SMAC environments. The results show that vanilla QMIX takes 6h 30m, ROMANCE requires 16h 36m, and WALL takes 21h 05m on average. Compared to QMIX, ROMANCE and WALL incur a 2.46× and 3.23× increase in training time, respectively. Also, WALL requires approximately 30% more time than ROMANCE, primarily due to the use of the Transformer. However, despite this additional cost, the Transformer-based critical step selection plays an essential role in the effectiveness of our method, as it enables precise identification of attack timing where coordinated perturbations are most disruptive. This capability allows the robust policy to anticipate and defend against strategically timed adversarial threats, ultimately resulting in stronger robustness. \n\nAs shown in our component ablation study (Fig. 8), removing this module and replacing it with random step selection results in a substantial drop in robustness across all SMAC scenarios. This clearly demonstrates that the Transformer-based planner is not merely an architectural enhancement, but a core component contributing to the success of WALL. Furthermore, as illustrated in the training curves in Fig. 6, other methods do not reach WALL’s level of performance even with extended training time, indicating that the proposed method offers superior robustness despite its slightly higher computational cost. We hope this helps clarify and address the reviewer’s concern.\n\n**Environment diversity:** (Common response with Reviewer rqE9)\n\nWe also appreciate the reviewer’s valuable suggestion to expand beyond SMAC in our evaluations. Our initial choice of SMAC was motivated by its widespread use as a benchmark in MARL, its cooperative partial observability, and its complex unit-level coordination dynamics. These characteristics make SMAC a rigorous and well-established testbed for adversarial robustness in decentralized learning.\n\nTo further address the reviewer’s concern and support the broader applicability of our method, we conducted additional experiments in the Multi-Agent Particle Environment (MPE) [1], a widely adopted benchmark in MARL, focusing on three cooperative scenarios: tag 3/1, tag 6/2, and tag 9/3. These environments require agents to coordinate spatially to pursue adversarial targets, which closely aligns with the core motivations of our work. For comparison, attack-based methods were evaluated using their respective optimal hyperparameter setups, ensuring a fair and controlled evaluation across environments.\n\nAs summarized in Table R.2, WALL consistently outperforms prior methods in these diverse MPE tasks. These results are included to address the reviewer’s concerns regarding environment diversity and to clarify that the effectiveness of our approach is not limited to the SMAC benchmark.\n\n**Table R.2:** Performance comparison on MPE envrionments.\n\n|Attacks|Methods|tag 3/1|tag 6/2|tag 9/3|Mean|\n|---|---|---|---|---|---|\n|Random Attack|Vanila QMIX|158.9|522.6|656.4|445.9|\n||RANDOM|170.1|638.7|657.9|488.9|\n||ROMANCE|173.4|624.2|701.7|499.7|\n||WALL|**202.9**|**654.0**|**802.5**|**553.1**|\n|EGA|Vanila QMIX|153.7|499.8|585.1|412.8|\n||RANDOM|157.6|604.6|589.7|450.6|\n||ROMANCE|167.1|605.1|594.4|455.5|\n||WALL|**166.5**|**685.5**|**682.7**|**511.5**|\n|Wolfpack attack|Vanila QMIX|135.7|415.6|551.7|367.6|\n||RANDOM|157.5|571.0|624.3|450.9|\n||ROMANCE|159.3|554.6|570.3|428.0|\n||WALL|**171.5**|**599.0**|**698.1**|**489.5**|\n\nAdditionally, in response to Reviewer rqE9’s suggestion regarding broader definitions of robustness, we also evaluated the robustness of WALL under observation noise and parameter noise conditions. For completeness, we kindly refer the reviewer to our response to Reviewer rqE9 for the full details of these experiments. We include these results to directly address the reviewer’s concern and will incorporate both the findings and discussion in the revised version.\n\n[1] Lowe et al., \"Multi-agent actor-critic for mixed cooperative-competitive environments,\" NeurIPS 2017.\n\n**Presentation issues**\n\nWe thank the reviewer for pointing out the issues related to clarity and formatting. We will revise the abstract to more clearly highlight our key contributions and results, and we will correct the formatting errors in Equation 1 and Line 186.\n\nOnce again, we sincerely thank the reviewer for their careful reading, insightful comments, and positive recommendation. We hope that the additional experiments and clarifications adequately address the raised concerns and further reinforce the value of our work.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There is no formulated theoretical claims in this paper.", "experimental_designs_or_analyses": "Yes. I checked Section 5. Experiments.", "relation_to_broader_scientific_literature": "The key contributions extend and refine ideas from MARL, robustness in machine learning, and open-environment learning. By addressing the challenge of robustness in MARL, this paper advances these fields and provides a framework for developing more robustly collaborative AI systems.", "essential_references_not_discussed": "The purterbation of observations in MARL is an open-MARL problem, where the key factors of the learning process change. A survey of this problem [1] should be discussed.\n\n[1] A Survey of Progress on Cooperative Multi-agent Reinforcement Learning in Open Environment. Lei Yuan, et al. 2023.", "other_strengths_and_weaknesses": "## Strengths  \n1. **Clarity and Accessibility**: The paper is well-written and easy to follow. The proposed method, WALL, is presented in a clear and straightforward manner, making it accessible to readers.  \n2. **Novelty**: The idea of transforming single-step, fixed-agent attacks into consecutive attacks on follow-up agents is quite novel.\n3. **Comprehensive Experiments**: The experiments are thorough and span diverse multi-agent attack scenarios and robust baselines. The results demonstrate consistent improvements over strong baselines, underscoring the effectiveness of the proposed method.  \n\n## Weaknesses  \n1. **Computational Cost**: WALL introduces a transformer-based module to select follow-up agents, which likely increases the computational overhead. However, the paper does not discuss or quantify this additional cost, leaving a gap in the practical evaluation of the method.  \n2. **Limited Environments**: While the experiments are robust, they are only conducted in the SMAC environment. Including additional environments or benchmarks would strengthen the generalizability and significance of the results.  \n3. **Presentation Issues**:  \n   - The abstract could be more informative, providing additional details about the key contributions and results.  \n   - Equation 1 appears to have an extra $\\Delta$ symbol, which should be corrected.  \n   - In Line 186, the notation $\\tilde{a}_t$ is incorrectly written as $\\tilde{a}t$.", "other_comments_or_suggestions": "None.", "questions_for_authors": "See Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YkQY33CZHv", "reviewer": "Reviewer_z86J", "summary": "This paper propose Wolfpack adversarial attack for robust MARL, which targets not only the initial perturbed agent, but also the agents that follow-up to help this perturbed agent. Based on wolfpack attack, the author further propose the wolfpack-adversarial learning for MARL, which learns robust policy based on wolfpack attack. Experiment results show the defense is more robust, and the attack is more harmful.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive comments. We greatly appreciate the opportunity to address the main concerns regarding the scope of our method, the perturbation budget, and the theoretical contributions. We provide detailed responses to each of these points below.\n\n**Scope of our method:**\n\nOur proposed Wolfpack adversarial attack introduces a novel class of adversarial attacks in MARL that we believe goes beyond existing approaches. Unlike ROMANCE, which focuses on building a population of diverse attackers targeting a single agent (EGA), Wolfpack explicitly models **multi-step attacks targeting multi-agents**. Specifically, it targets not only a single agent but also its cooperating teammates in follow-up time steps, thereby exposing relational vulnerabilities in agent coordination that are not captured by single-agent or uncoordinated perturbations.\n\nA key innovation also lies in our use of a **Transformer-based attack planner** for critical step selection. This module allows the attacker to identify time steps where coordinated interventions lead to maximal long-term disruption, particularly important when attacks are spread over multiple steps. ROMANCE, by contrast, does not perform planning-based or temporally extended attacks. We hope this distinction clarifies that our method is not a straightforward extension of ROMANCE but rather proposes a new class of coordinated adversarial behavior in MARL.\n\nAlthough we adopt the LPA-Dec-POMDP formulation in our training setup, we emphasize that this choice is primarily for structuring the learning process with a finite attack budget and enabling convergence guarantees. Importantly, our method is not inherently restricted to the LPA-Dec-POMDP setting. Since our planner enables the policy to anticipate attacks based on learned dynamics, WALL remains effective even without explicitly encoding the attack budget, suggesting general applicability beyond this specific framework. We hope this addresses the reviewer’s concern regarding the generality of our approach.\n\n**Theoretical contributions:**\n\nWe understand and appreciate the reviewer’s concern regarding the lack of formal convergence or equilibrium analysis. As discussed above, our work emphasizes practical effectiveness in attacker modeling and defense under realistic MARL coordination settings. In particular, we focus on demonstrating how temporally extended, learning-based attackers can expose structural weaknesses in existing robust MARL methods.\n\nWhile some works in robust MARL pursue theoretical guarantees, a substantial body of literature (e.g., [1–2]) actively explores attacker design and empirical robustness under realistic conditions. Our approach aligns with this latter direction: the use of Transformer-based planning makes formal convergence analysis challenging, but reflects the state of practice in robust MARL where empirical insights are often more actionable than theoretical ones. However, we agree that developing theoretical underpinnings for planning-based attacks is a valuable and non-trivial direction. We consider this a promising line of future research.\n\n[1] Gleave et al., \"Adversarial policies: Attacking deep reinforcement learning,\" ICLR 2020.\n\n[2] Ao et al., \"Eat-c: environment-adversarial sub-task curriculum for efficient reinforcement learning,\" ICML 2022.\n\n**Perturbation budget comparisons:**\n\nWe appreciate the reviewer’s concern about the role of perturbation budget. However, as our experiments show, attack budget does not directly correlate with robustness. For example, in both ROMANCE (Yuan et al.) and our own method, increasing the number of attacks can degrade policy learning, as overly perturbed environments prevent effective strategy formation.\n\nIn our case, as shown in Fig. 9 and Fig. F.1, robustness does not improve monotonically with more attack steps or follow-up agents. Instead, we perform parameter sweeps for each method and compare their performance under optimal attack configurations. Thus, our comparisons are fair and based on each method’s best-case performance. The success of Wolfpack arises not from larger attack volume, but from coordinated timing and targeting, which prior defenses fail to counter.\n\n**Applicability to MAPPO:**\n\nAlthough our work focuses on value-based CTDE methods, the Wolfpack framework can be extended to actor-critic approaches like MAPPO, which also follow the CTDE paradigm. While some architectural adjustments (e.g., adapting the $Q$-value estimation) would be needed, the core ideas of coordinated attack modeling and robust training remain applicable. We will note this in the revised version as a potential direction for future work.\n\nOnce again, we sincerely thank the reviewer for their careful reading and insightful feedback. We hope that our clarifications and responses adequately address the raised concerns and help situate our contributions within the broader scope of robust MARL research.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes, albeit in a narrow scope.", "theoretical_claims": "The author do not make theoretical claims.", "experimental_designs_or_analyses": "Yes, experiments are comprehensive and solid. However, I do have concerns on the perturbation budget used.", "relation_to_broader_scientific_literature": "The paper is a part of robust MARL literature, but more specifically related to the LPA-Dec-POMDP paper in a reference.", "essential_references_not_discussed": "Most important references are discussed.", "other_strengths_and_weaknesses": "Strength.\n\n1. I would appreciate the strong empirical focus of this paper, which identifies the current weakness of many robust MARL algorithms. While one agent is being attacked, other agents trained by robust MARL would come to help it. Wolfpack attack responds to this by also targeting these agents as well, which is an interesting insight provided in this paper.\n\n2. I appreciate the extensive empirical evaluation in the paper, which indicates the effectiveness of the method proposed in this paper.\n\n3. The paper is clearly written and easy to follow.\n\nWeakness.\n\n1. The biggest weakness I believe is the scope of this paper. While robust MARL is already a narrow field, the threat model proposed by LPA-Dec-POMDP is one of the many threats in robust MARL. The definition of Wolfpack attack, while focusing on action uncertainties, is another narrower field: it first assumes the LPA-Dec-POMDP threat model, then base its attack on it. What if other form of threat model and formulations is applied? How can Wolfpack attack based on LPA-Dec-POMDP formulation generalize to other settings?\n\n2. While I do not deny wolfpack attack is interesting, the entire paper is more like a follow-up work of Yuan et al (2023) and its journal version, ie, using same framework and just an updated attack which requires more perturbation budget. Since the attacker is using larger perturbation budget, it is natural that existing defenses are broken by the attack and the proposed attack is the most robust. What about these robust methods trained with larger perturbation budgets? or any discussions?\n\n3. While I appreciate the strong empirical performance of the paper, I do believe there is a lack of theoretical analysis, including convergence, equilibrium, etc, which better fits the scope of ICML.\n\n4. This is minor. The definition applies only to QMIX-family, the application to MAPPO family is not straightforward, but can be imagined. Maybe adding a discussion in conclusion or section 3.1 helps.", "other_comments_or_suggestions": "Please see \"Other Strengths And Weaknesses\"", "questions_for_authors": "Please see \"Other Strengths And Weaknesses\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}]

Author rebuttal:

Meta-review: In this submission, the authors propose the Wolfpack Adversarial Attack framework, which targets an initial agent and its assisting agents to disrupt cooperation in multi-agent settings.  They also proposed the Wolfpack-Adversarial Learning for MARL (WALL) approach that can be used to train MARL policies to defend against these types of attacks. The authors provide empirical results which showcase the effectiveness of the proposed methods, both in terms of attacking as well as defending against it.

All of the reviewers believe that this work demonstrates strong empirical performance. They also highlight the novelty of the approach and the comprehensive set of experiments included in the manuscript. The reviewers also noted various limitations of the original paper, including the inclusion of only a single environment, the lack of discussion on computational costs, and the robustness of the approach (beyond action perturbations). The authors have addressed many of the reviewers' concerns during the rebuttal, which has led to an increase in scores.

Overall Assessment: Based on the reviewers' comments and discussions during the rebuttal, I believe this paper makes a strong technical contribution. I hope the reviewers’ comments and the results shown during the rebuttal can help the authors prepare an improved version of this submission for camera-ready.
Decision: Accept (poster)