Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
Authors: Ziyuan Zhou
Guanjun Liu
Mengchu Zhou
Weiran Guo
Venue: ICML 2025 poster (2025)
Keywords: Multi-Agent Reinforcement Learning
Adversarial Training
Stochastic Adversary
Entropy Maximization
Primary area: reinforcement_learning->multiagent

Abstract: The performance of models trained by Multi-Agent Reinforcement Learning (MARL) is sensitive to perturbations in observations, lowering their trustworthiness in complex environments. Adversarial training is a valuable approach to enhance their performance robustness. However, existing methods often overfit to adversarial perturbations of observations and fail to incorporate prior information about the policy adopted by their protagonist agent, i.e., the primary one being trained. To address this important issue, this paper introduces Adversarial Training with Stochastic Adversary (ATSA), where the proposed adversary is trained online alongside the protagonist agent. The former consists of Stochastic Director (SDor) and SDor-guided generaTor (STor). SDor performs policy perturbations by minimizing the expected team reward of protagonists and maximizing the entropy of its policy, while STor generates adversarial perturbations of observations by following SDor's guidance. We prove that SDor's soft policy converges to a global optimum according to factorized maximum-entropy MARL and leads to the optimal adversary. This paper also introduces an SDor-STor loss function to quantify the difference between a) perturbations in the agent's policy and b) those advised by SDor. We evaluate our ATSA on StarCraft II tasks and autonomous driving scenarios, demonstrating that a) it is robust against diverse perturbations of observations while maintaining outstanding performance in perturbation-free environments, and b) it outperforms the state-of-the-art methods.

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
Ziyuan Zhou' Guanjun Liu
MengChu Zhou 23
Weiran Guo
Abstract
1. Introduction
The performance of models trained by Multi- Agent Reinforcement Learning (MARL) is sen- sitive to perturbations in observations, lowering their trustworthiness in complex environments. Adversarial training is a valuable approach to enhance their performance robustness. How- ever, existing methods often overfit to adver- sarial perturbations of observations and fail to incorporate prior information about the policy adopted by their protagonist agent, i.e., the pri- mary one being trained. To address this im- portant issue, this paper introduces Adversar- ial Training with Stochastic Adversary (ATSA), where the proposed adversary is trained online alongside the protagonist agent. The former con- sists of Stochastic Director (SDor) and SDor- guided generaTor (STor). SDor performs policy perturbations by minimizing the expected team reward of protagonists and maximizing the en- tropy of its policy, while STor generates adver- sarial perturbations of observations by follow- ing SDor's guidance. We prove that SDor's soft policy converges to a global optimum accord- ing factorized maximum-entropy MARL and leads to the optimal adversary. This paper also introduces an SDor-STor loss function to quan- tify the difference between a) perturbations in the agent's policy and b) those advised by SDor. We evaluate our ATSA on StarCraft tasks and au- tonomous driving scenarios, demonstrating that a) it is robust against diverse perturbations of ob- servations while maintaining outstanding perfor- mance in perturbation-fiee environments, and b) it outperforms the state-of of-the-art methods.
Deep reinforcement learning leverages the representational power of deep learning alongside the decision- maannnn ca- pabilities of reinforcement learning, allowing agents to learn and adapt through interactions with dynamic environ- ments (Sutton & Barto, 2018). In Multi-Agent Reinforce- ment Learning (MARL), agents are not isolated because they actively interact with and adapt to environments in both cooperative and/or competitive frameworks. This in- teraction enables them to develop their sophisticated strate- gies and allows them to optimize their behaviors not only individual level but also on the system level, thereby enhancing their ability to handle complex tasks. MARL proves particularly valuable for data-driven applications (Cai et al., 2023; Yuan et al., 2023a) like autonomous driv- ing (Wang et al., 2024a; Zhang et al., 2023) and recom- mendation systems (Deng et al., 2023; Gui et al., 2019), where adaptive and decentralized decision-making is cru- cial. However, models trained by deep learning often lack robustness to dynamic and noisy environments because they are typically trained under the assumption of indepen- dent and identically distributed training data (Goodfellow et al., 2014). This assumption is rarely met in real-world scenarios, leading to performance degradation when mod- els encounter data distribution that differs from the train- ing set's (Wang et al., 2024b; Zhou et al., 2024a).
In MARL, deep neural network module exacerbates this robustness issue, causing them to make incorrect or sub- optimal decisions in critical situations. It is essential to enhance the robustness of MARL-trained models. Much progress has been made on doing so deep learning for classification tasks such as adversarial training (Goodfel- low et al., 2014; Madry et al., 2018; Wong al., 2020) where the goal of the adversary is to identify and use the strongest adversary. Yet directly applying these approaches to Single-Agent Reinforcement Learning (SARL) scenar- ios introduces some great challenges: 1) Mismatch be- tween short-term and long-term goals. Aversarial train- ing like the Fast Gradient Sign Method (FGSM) (Good- fellow et al., 2014) and projected gradient descent (Madry et al., 2018), borrowed from classification tasks, perturb the agent's observation at each time step (Huang et al., 2017). However, one-step misclassification does not equate to minimized long-term rewards. 2) Difficulty in high-
'The School of Computer Science and Technology, Tongji University "The School of Information and Electronic Engineer- ing, Zhejiang Gongshang University 'The Helen and John C. Hartmann Department of Electrical and Computer Engineering, New Jersey Institute of Technology. Correspondence to: Guanjun Liu <liuguanjun edu..i.edu.en...
Proceedings of the 1nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
dimensional environments. Existing studies (Zhang et al., 2020; 2021a) prove that training adversary to generate an adversarial perturbation of observation can be modeled as Markov decision process, in which the reward received by the adversary is the negative counterpart of the reward received by the protagonist agent, and its action space is the observation space of the protagonist agent. Then they use reinforcement learning to train an adversary. How- ever, high-dimensional action space in reinforcement learn- ing presents great implementation difficulties. The work in (Sun et al., 2022b) attempts to address these challenges by training adversaries by using policy adversarial actor and director. The latter provides suggestions for policy per- turbations of the protagonist agent to achieve its worst- case performance, while the former generates an adversar- ial perturbation of observation.
quantify the difference between perturbations sug- gested by SDor and those implemented by STor. This function is used to refine the coherence between STor and SDor, ensuring that the protagonist agent's s policy changes align with the director's intended modifica- tions.
3) To evaluate the robustness of models trained by ATSA, this study conducts extensive experiments on StarCraft II tasks and autonomous driving scenarios. The results show that ATSA is robust to adversarial perturbations of observations generated by four meth- ods while maintaining strong performance in random and perturbation-fie environments.
2. Preliminaries
In MARL, cooperation among agents means that an adver- sary for one protagonist agent must consider its impact on team rewards. The above adversarial training methods to MARL can improve the trained model's robustness (Guo et al., 2024), but there are limitations: 1) Overfitting to adversarial perturbations. In MARL, the strongest ad- versary often causes models overfit adversarial perturba- tions of observations, losing performance when handling clean ones due to suboptimal data collection. 2) Misalign- ment between actor and director. When applying the policy adversarial actor and director framework from ro- bust SARL to MARL, there are gaps between the actor and the director. The adversarial perturbations of observations generated by the former do not always align with the lat- ter's intended influence on the protagonist agent's policy, causing instability in training. To address these issues, we propose Adversarial Training with a Stochastic Adversary (ATSA) as a novel framework designed to enhance robust- ness in MARL with discrete action space and deterministic policies. This study makes the following contributions:
In this section, we introduce the definitions of Decentral- ized Partially Observable Markov Decision Process (Dec- POMDP), Observation-adveeraria Dec-POMDP (OD- POMDP), and Policy-adversarral Dec-POMDP (PD- POMDP). Some important concepts, abbreviations, and symbols related to this paper are described in Appendix B.
2.1. Dec-POMDP
A Dec-POMDP (Oliehoek & Amato, 2016) is typically de- scribed as follows:
g a (S, 10'O (A' ti',, P.7)
where S is the state space, E S is state, i N (1....NN N} is protagonist agent, A' is the action space for protagonist agent i, af E A is the action taken by protagonist i, a E XieArA is the joint action of all protagonist agents, Oi is the observation space of protag- onist agent i, o E O' is the observation of protagonist agent calculated using the observation function Z' (s. i): S x VV O', S XieN A' IR is the reward func- tion, P 8 XieN A' - A (S) is the transition proba- bility function, and y E [0.1] is the discount factor. Un- der the Centralized Training with Decentralized Execution (CTDE) paradigm, each protagonist agent learns its policy n² (afl7"> T x A' - [0. 1] based on its trajectory The T : O² A'. From the centralized perspective, the joint policy of all protagonist agents is represented as Th (a T"), where 7" E X'ENT'r is the joint trajectory of all protagonist agents.
1) It proposes ATSA. Its stochastic adversary consists of two components: a) Stochastic Director (SDor) that suggests policy perturbations by minimizing the ex- pected team reward of the protagonist agent while maximizing entropy to promote exploration and b) SDor-guided generaTTor (STor) that generates adver- sarial perturbations of observations based on SDor's guidance. This framework allows agents to dynam- ically adjust to both adversarial and clean condi- tions, thus greatly improving the robustness of MARL models. By leveraging factorized maximum-entroppy MARL, we theoretically demonstrate that the soft pol- icy of SDor converges global optimum, and when combined with STor, SDor's optimal policy induces the optimal adversary.
2.2. OD-POMDP
An OD-POMDP (Zhou et al., 2024c) extends Dec-POMDP by introducing adversarial perturbations affecting the agents' observations. Its formal definition is as follows:
2) It designs an SDor-STor loss function, which is new,
(
)exs,P.)
2

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
where B for Of ||ō o'Iloc s e} denotes set of adversarial observation of protagonist agent adversarial observation o' Bi is within 20 -iistacce from the clean observation o'. ..M- S W is the set of observation adversary, and ô - [ is the joint action of observation adversary.The difference be- tween OD-POMDP and Dec-POMDP is the inclusion of observation adversary policy v² which aims modify the protagonist's observation. If the observation adversary pol- icy is stochastie, i.e., (-((rx)) : T'p x B - [0, I]. then v - [u']em represents the joint policy of the ob- servation adversary, and = (-17")
PD-POMDP
OD-POMDP
Environment
Environment
e-bounded
H
SDor
hlt
Observation Adversary
STor
T (-17"), where FP [F']eN is the joint trajectory of protagonist agents, piv f**I(i M) + Téol(i & .)) is the trajectory of the protagonist agent which may be perturbed by adversary, and f'v E Ttv : B x A' is composed of the adversarial observation and the action. ~ (-I7²) denotes the joint policy of the protagonist agent under this adversary, where ō [ō*)) Jee,, and o² 0'I(i M) o'I(i $ M).
ot
of
af
Protagonist
Protagonist
Figure 1. The figure illustrates the evolution from OD-POMDE to PD-POMDP. Under N, i.e., all protagonist agents are per- turbed by adversaries. In OD-POMDP, the protagonist agent re- ceives ohservation modified by the observation adversary, makes decisions based on the adversarial observation, and then receives rewards from the environment based on its actions. PD-POMDP generates adversarial observations via SDor and STor. SDor gen- erates policy perturbation suggestions, while STor generates ad- versarial perturbations based on these suggestions. Unlike OD- POMDP, where the adversary's 's goal is only to minimize the pro- tagonist agent's reward, PD-POMDP modifics the SDor's objec- tive to include maximizing policy entropy.
2.3. PD-POMDP
Given a Dec-POMDP fixed joint stochastic policy of protagonist agents wl, and perturbation budget 0, PD-POMDP (Sun et al., 2022b; Zhou & Liu, 2023) is defined as
&pm (s,('')em HeM, 12 JieM, P.7)
where A' (d 1-1, プトー d, 0} is the action space and represents the policy perturbations of the protag- onist agent. The relationship between and P (resp. é and r) in PD-POMDP and Dec-POMDP, respectively, is
of. The constraint ensures that the adversarial observation achieves the maximum possible impact in the direction of at. Following Theorem in (Sun et al., 2022b), know that an optimal policy adversary h induces an optimal ob- servation adversary against n². In other words, the opti- mal observation adversary vi in Goa can be constructed by combining hi, in PD-POMDP 9pa with the function g². In the CTDE paradigm, the objective of finding an optimal observation adversary v. in Gon is conducted adversarial training with the protagonist agent, thereby enhancing the robustness of the protagonist agent while this objective can be transformed into the task of finding an optimal joint pol- icy h in 9pm-
P(st+1|5 à.) d I (M) x.EA'x--A""
(81, (e) = - (arlg (ār, ; (8t, ar).
a+EAt X..AA
The function 9' generates the corresponding adversarial ob- servations based on the action of the policy adversary a', with g - [g'lee If & M, then g$ = of. and otherwise,
g' (at, of) argmaxaçees
x. (-16% - (106)
3. Adversarial Training with Stochastic Adversary
s.t. (Tr (110 x/=1 -T' (10) r/=;)) T at (10g x1m ||a|||
In this section, we introduce ATSA. As illustrated in Fig. 1, our framework consists of two components: an observation adversary and protagonist agent.
r (100), 2/p T"
(1) where T' (-) represents the policy function of the protag- onist agent i. This optimization seeks the adversarial ob- servation oj within the allowable set Bi that maximally deviates the policy's behavior from the clean observation
3.1. Objective Function of SDor
In the adversarial training process, using the strongest ad- versary may lead to instability of training the protagonist
3

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
agent or overfitting the strongest adversarial perturba- tions, ultimately failing to achieve robustness. To address this problem, we use stochastic adversary. PD-POMDP Gpa is viewed as probabilistic graphical model. Following (Levine, 2018), we introduce an additional optimality vari- able X₂ which takes binary values to indicate the optimality of the joint actions taken by all policy adversaries. Specif- ically, the probability p(x; _[ âe) x exp(r(st. a.)) models the likelihood such that a partieular joint action is optimal. We define the objective of SDor by using max- imum entropy. The related derivation process is given in Appendix C.
where Qs and V are the optimal soft Q-function and value function for the agent i, respectively.
By plugging (4) and (5) into (3), the optimal joint and the individual soft-Q-functions should satisfy
(T)))= - - + +V2((T) (6)
3.2. SDor and STor
In this section, we provide an explanation of SDor's learn- ing process and STor's computation method. We then theo- retically demonstrate that combining SDor and STor results in an optimal stochastic observation adversary.
Definition 3.1 (Ohjective of SDor). The maximum entropy-based objective function of SDor, based on vari- ational inference, is defined as:
1. SDOR
J (1*)
=1 R p
Σ
(f (s), +a7l (11) (172))
We follow the factorized optimal joint policy (Zhang et al., 2021b) to learn the optimal policy of SDor. In the joint soft policy evaluation step, applying a soft Bell- man backup operator F iteratively to update the joint soft Q-function of SDor Qj as T Q" (Tz', a.) & Ar + yBrr, [V" (T(-1) where the soft value function is de- fined as V (T En (T," â,) - alogh (ā. [T')] This formulation provides foundation for the convergence of the joint soft policy. The individual soft policy is up- dated according to Proposition 3.3, with more details avail- able in Appendix E. Alternating between joint soft policy evaluation and individual soft policy improvement leads to convergence, as formalized in the following theorem:
(2)
where a temperature parameter and H(-) denotes the entropy of SDor's joint policy.
In Definition 3.1, controls the trade-off between entropy and reward maximization. As a + oo, the SDor's pol- icy becomes completely random. By adjusting the value of a, the level of randomness in the policy can be effectively controlled. Following the CTDE paradigm, the soft pol- icy of an individual SDor is denoted by ht (-17'=), where r E TTi O² x Á represents its trajectory. The joint soft policy of all SDors is denoted by 12" (.17"), where T" E X'eMT" represents their joint trajectory. To en- sure consistency. we assume that the individual optimal soft policies satisfy the andividual-gloaal-opiinn condition (Zhang et al., 2021b):
Theorem 3.4. (Factorized Soft Policy Iteration of SDor) Consider joint soft policy of all SDors that can be fac- torized as he = IIIEMM h-. By repeatedly applying joint soft policy evaluation and individual soft policy improve- ment, this process converges to a policy I8 such that 2he (T= à) 2 L'r(T" à) for all [h' E 11"M and (7", XiEMT" XieM A' with XIEM < co.
Nº (alTT) = II Ni (a'l7a). REM
Then we derive the forms of the optimal joint soft policy and the optimal individual soft policy that maximize this objective. The derivation process is given in Appendix D. Proposition 3.2 (Optimal Joint Policy of SDors under Maximum Entropy-basee based Objective Function). The opti- mal joint soft policy of all SDors denoted as Ht which max- imizes the regulai-e objective, is given by:
Based Theorem H.6, know that the individual soft policies of SDor guarantee convergence to the global op- timum. The proof is given in Appendix E. To learn the SDor's soft policies, we employ a deep neural network. During training, the relationship between the joint soft pol- icy and the individual soft policy of SDor is formulated as:
1t (a) ")) = exp (a (Q". (T", aa) - v (T")), (4)
Q" (T"" = Σ Agn (T" (T" [Dim. (F',a')
where Qt is the optimal joint soft Q-function and V is the optimal joint soft value function.
iEM
- -Vh., (F7) + Vift (Ta)
Proposition 3.3 (Optimal Individual Policy of SDor under Maximum Emtropy-basee -based Objective Function). For each SDor i, the individual optimal soft policy, which conditions only on its own trajectory r'o, is given by:
where d", qa, 6',, and 0 represent the parameters of the weight network, the joint soft value network, the individual soft value network, and the individual soft-Q network, re- spectively. The individual soft-Q network, joint soft value
né, (a*) =xxp (a, (Q' (7'=, a²) - Vv. (rdx))), (5)
4

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
network, and weight network are trained by minimizing the temporal-die cinporalddiffrrence error:
We apply the FGSM (Goodfellow et al., 2014) solve this optimization problem:
1(0',, ,," -09 = -E(ra, (a a') [( (T";
g" (o') clip (o' - Bsgn (V. (o!))
(13)
(8)
where / is the step size, and clip(-) ensures that the per- turbed observation o' remains within a valid range.
- (f ++7 7 (otL (T ,a' a logo hdt (d'IT"
where D is the replay buffer of SDor, Que is target net- work. To train the individual value networks of SDor, the following loss function is minimized:
3.3. SDor-STor Loss Function
From Theorem 3.5, we know that the optimal stochastic observation adversary requires collaboration between SDor and STor. However, previous work (Sun et al., 2022b) does not fully account for this aspect, which may result in deci- sions by the protagonist agent, based the adversarial ob- servations generated by STor, that do not align with SDor's intentions. Therefore, we propose an SDor-STor loss func- tion to measure this gap. If the protagonist policy is de- terministic and discrete, we use the cross-entropy loss to measure this difference:
C ((0)) Bori--er [(T a(logh
(9)
Finally, the individual soft policies of SDor optimized using the policy gradient method, where the objective is:
(Er)) (a")') - (tra''
(10)
3.2.2. STOR
( De (oa'"""") (a' (14)
Once the optimal joint policy of SDor is obtained, STor seeks to generate adversarial observations that lead the protagonist agent to take actions aligned with SDor's in- tentions. The adversarial observation of each protagonist agent can be determined individually based on its clean ob- servations and SDor's actions. The optimization objective for STor i (i E M) is given in (1). We consider the case where the protagonist agent's policy is deterministic and SDor's policy is stochastic, which modifies the optimiza- tion objective function to minimize the following:
In conjunction with the individual soft policy objective function of SDor (10), optimize the individual soft pol- icy network by updating it with the gradient of
L' (x²) - J (5²) + +L (1²)
(15)
where k is a hyperparameter that regulates the impact of our loss function on SDor's individual policy. For a com- prehensive explanation of the training process and com- plexity analysis, see Appendix G.
DKL (Earnns (m (a')o*,++>) higs (+oot+h) (11)
4. Experiments
where fp' represents the parameters of the policy network of protagonist agent i. We then demonstrate that if SDor's joint soft policy is optimal and STor has obtained the op- timal solution to (11), this results in the optimal stochastic observation adversary.
4.1. Experimental Settings
4.1.1. ENVIRONMENT SETTINGS
We evaluate our adversarial training framework on two challenging benchmarks: the StarCraft Multi-Agent Chal- lenge (SMAC) (Samvelyan et al., 2019) and a Connected and Autonomous Vehicles (CAV) environment (Chen et al., 2023). We conduct experiments on three SMAC maps con- taining 3 Marines (3m), 3 Stalkers_vs_3 Zealots (3s_3z), and 8 Marines (8m) as well as one CAV scenario with three autonomous vehicles and 1-4 human-driven ones. More details about the SMAC and CAV environments are in- cluded in Appendix H.1
Theorem 3.5 (Optimality of SDor-STor). For any PD- POMDP ppa any fixed joint deterministic policy of protag- onist agents x", and any attack budget e 2 O, optimal joint soft policy Ve of SDor in 9po induces an optimal optimal soft policy of the observation adversary against mil in Goa-
Theorem 3.5 demonstrates that if SDor learns the optimal joint soft policy within Spe. can effectively collaborate with STor to produce the optimal stochastic observation ad- versary. The relevant proof is detailed in Appendix F.
4.1.2. BENCHMARK METHODS
All benchmarks implemented based on the classical MARL methods: Value Decomposition Network (VDN) (Sunchag et al., 2018) and Q-MIXing network (QMIX) (Rashid et al., 2020). We use the following as bench- marks: No Adversary (NoAdv), Random Noise (RN), ad- versarial training methods including FGSM (Goodfellow
However, (11) is intractable for optimization. In practice, we minimize the expected KI. divergence over samples:
Ci (6') = Eaa [KKL (T (a'10', n/as (a'1o)78))
(12)
5

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
Table 1. Performance comparison in SMAC. The columns represent various adversary algorithms, and the rows indicate the protagonist agent's training methods. AVG represents the average performance of the protagonist agent under six adversaries. The values in the table represent the win rates over 500 episodes (higher is better). The upper bound of the perturbation size is set to 0.25 for 3m and 3s 3z, and 0.1 for 8m. Bold numbers mean the best results.
VDN
QMIX
Euvs
Protagonist
NoAdy
RN
FGSM
ATLA
PAAD
ATSA
AVG
NoAdv
RN
FGSM
ATLA
PAAD
ATSA
AVG
NoAdy
0.99
0.23
0.00
0.02
0.00
0.00
0,21=0.36
0.99
0.24
0.00
0.00
0.00
0.00
0.20=0.36*
RAND
0.99
0.94
0.99
0.98
0.98
0.35
O.344
FGSM
0.96
0.96
0.91
0.96
0.79
0.57
0.86-0.14
0.67
0.86
0.96
0.80
0.26
0.92
0.74-0.24*
ATLA
0.86
0.36
0.06
0.98
0.00
0,00
0.38.10.40
0.99
0.68
0,00
0.98
0.00
0.00
0.44_00.4**
PAAD
0.93
0.94
0.87
0.96
0.81
0.89
0.90=
0.93
0.95
0.97
0.90
0.98
0.94
0.94:10.03
3m
PR
0.51
0.49
0.61
0.80
0.78=
PR-REP
0.61
0.70
0.99
0.85
0.76
0.93
0.81=0.13
0.85
0.87
0.91
0.78
0.84
0.81
0,84=0.04*
ERNIE
0.83
0.87
0.95
0.77
0.73
0.37
0.75=0.19
0.85
0.82
0.87
0.40
0.44
0.77
0.69+-0.19
RAP
0.97
0.90
0.45
0.99
0.83
0.83=0.18
0.99
0.96
0.94
0.97
0.96
0.95
0,96:+0.02
ROMANCE-P
0.86
0.31
0.00
0.09
0.00
0.00
0.21
0.85
0.33
0.00
0.28
0.00
0.00
0.34-0.300*
ROMANCE
0.93
0.19
0.00
0.05
0.00
0.00
0.20-0034
0.98
0.42
0.00
0.01
0.00
0.00
0
ATSA
0.97
0.95
0.96
0.84
0.77
0.98
0,91 0..99
0.99
1.00
0.99
1.00
0.96
1.00
0.99100.02
NoAdy
0.99
0.00
0.24
0.00
0..38
0.49
0.38 0.40*
RAND
1,00
0.99
0.00
0.76
0.00
0.00
0.46-0..6
1.00
1.00
0.00
0.95
0.00
0.00
0.49-+0.49
FGSM
0.92
0.93
0.68
0.84
0.36
0.38
0.69 1..94
1.00
1.00
0.99
0.99
0.62
0.53
0.86.10.20
ATLA
0.88
0.73
0.00
0.99
0.00
0.00
0.43 44
1,00
0.90
0.00
1.00
0.00
0.00
0.48:10.48
PAAD
0.86
0.81
0.97
0.97
0.45
0.69=0.29>
3,,3x
PR
0.25
0.39
0.48
0.42
0.04
0.04
0.27=0.18*
0,40
0.42
0.34
0.27
0.01
0.01
0,24=0.17
PR-REP
0.96
0.98
0.91
0.84
0.63
0,52
0.81=0.17
0.85
0.84
0.95
0.97
0.35
0.24
0,70=0.29*
ERNIE
0.99
0.98
0.92
0.92
0.31
0,36
0.75=00.29
1.00
1.00
0.98
1.00
0.83
0.84
19. 19:11-30
RAP
0:99
0.94
0.64
0.92
0.63
0.64
0.79110.16
0.98
0.98
0.77
0.95
0.78
0.70
0.11
ROMANCE-P
0.94
0.45
0.00
0.00
0.00
0.00
0.23++0.36 36
1.00
0.94
0.00
0.90
0.00
0.00
0.47-0.47
ROMANCEs
0.97
0.41
0.00
0.05
0.00
0.00
0.24
1.00
0.89
0.00
0.63
0.00
0.00
0.42-_0.43*
ATSA
0.99
0.89
0.99
0.79
0.90::0.11
1.000
0.96
1.00
0.96
0 0.00
NoAdy
0.96
0,86
0.00
0.34
0.00
0.00
0.36=0.41
0.98
0.95
0.00
0.92
0.00
0.01
0
RAND
0.98
0.98
0.02
0.91
0.01
0.05
0.49..0.47
1.00
1.00
0.02
1.00
0.06
0.09
0.53.10.47
FGSM
0.99
0.98
0.93
0.97
0.79
0.77
0.90+00.08
0.84
0.89
0.83
0.53
0.89
0.52
0.75-0.16
ATLA
0.47
0.95
0.48=0.47*
PAAD
1,00
1.00
0.94
0.99
0.92
0.92
0.96=
0.96
0.94
0.90
0.95
0.89
0.92
0.93=0.03+
8m
PR
0.98
0.99
0.89
0.98
0.78
0,81
0.90=0008*
0.96
0.95
0.93
0.97
0.47
0.09
0.73=0.34*
PR-REP
0.99
0.98
0.94
0.93
0.31
0,76
0.82:40.24 24
0.98
0.98
0.91
0.95
0.35
0.34
0.75=0.29*
ERNIE
0.99
0,95
0.95
0.90
0.94=0.05
0.94
0.91
0.84
0.57
0.46
0.78=0.19*
RAP
0.97
0.98
0.93
0.96
0.79
0.94
0.93-+0.06
0.96
0.97
0.87
0.95
0.91
0.91
0
ROMANCE-P
0.96
0.95
0.00
0.27
0.00
0.00
4.34
0.98
0.99
0.08
0.99
0.03
0.02
0.52
ROMANCES
0.97
0.00
0.02
0.00
0.32+-0.5
1.00
0.97
0.00
0.49 0.49- 0.49*
ATSA
1.00
1.00
0.93
1.00
0.92
0.94
0.97
1.00
1.00
1.00
1,00
0.97
0.95
0.99.10.02
# indicates statistically significant improvement of ATSA over the corresponding method (p < 0.05, Wilcexon rank-sum text).
et al., 2014), Alternate Training protagonist agents with Learned Adversaries (ATLA) (Zhang et al., 2021a) and the Policy Adversarial Actor and Director (PAAD) (Guo et al., 2025; Sun et al., 2022b); robust leaning baselines including Policy Regularization (PR) (Guo et al., 2024), Repetitive PR (PR-REP) (Zhou et al., 2024c), and ad- vErsarially Regularized multiageNt reInforcement lEar- ing (ERNIE) (Bukharin et al., 2023); adversarial training with stochastic adversaries, including Robustness via Ad- versary Populations (RAP) (Vinitsky et al., 2020), and RO- bust Multi-AgeNt Coordination via Evolutionary genera- tion (ROMANCE-P(s)) (Yuan et al., 2023b). These methods cover a range of adversarial training strategies, from basic noise injection to more sophisticated adversarial training paradigms. Appendix H.2 includes further details the benchmark methods.
nario, and 0.05 for the CAV environment. The perturbation range is determined based on the difficulty of the task and the scale of the protagonist agent's observation space. For instance, in the 3m and 3z.3z scenarios, there are fewer agents, resulting in longer distances between agents. Con- sequently. a larger perturbation range can be used com- pared to the 8m scenarios. The objective is to select the largest possible perturbation range while maintaining the validity of the task, ensuring the robustness of the model can generalize to a broader range of perturbations.
4.1.4. HYPERPARA METTRS
We set the same hyperparameters as those used in the orig- inal VDN (Sunehag al., 2018) and QMIX (Rashid al., 2020). For SDor, the actor and critic networks consist of two MLP layers with a GRU (hidden size 64) inserted be- tween them. RMSprop (Hinton, 2012; Wen & Zhou, 2024) is used to optimize all parameters, with both actor and critic employing a learning rate of 0.0005. The target networks are updated every 200 episodes. The temperature parame-
3.1.3. PERTURBATION SETTINGS
The perturbation is defined as an E. norm with ranges of 0.25 for the 3m and 3s_3z scenarios, 0.1 for the 8m sce-
6

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
Table 2. Performance comparison in CAV. The columns represent various adversary algorithms, and the rows indicate the protagonist agent's training methods. AVG denotes the average performance of the protagonist agent across six adversaries. Reward indicates the average cumulative reward over 500 episodes (higher is better), while CR represents the crash rate over the same episodes (lower better). The upper bound of the perturbation size is 0.05. Bold numbers mean the best results.
Adversary
NuAdy
RN
FGSM
ATLA
FAAD
ATSA
AYG
Protagunist
Reward
CR
Rewarl
CR
Rewari
CR
Reward
CR
Reward
CR
Reward
CR
Reward
CR
NoAdy
48.07329966 6.
0,00
51.21
26.25
0.00
19.5 19,51E51.3 11.20
0.31
30.01
0.60
22.11153:44
0.07
16.91
250.11
0.04
34,87
15.39 39
0,00
RN
67.31=17.78
0.00
76
19.04
0.00
39.77=51.55
0.05
67 21:24=214
0.00
3.96469.22 23
17.26=58
26
43.72+25.66
0.05=00.66
FGSM
56.32+35.166
0.01
58.41-1::0
31.77
0.01
50 30.53+42.55
87
0.04
50 26264-2966 61
0.01
53.09+52.90
0.05
42.96
0.06
50.03+9.11 13
0.03-0.024
ATLA
57.54.35.72
0.02
60,23
14.39
0.02
10.8 58.63
0.07
61. 61.10.132.54
0.01
00667144.77
0.20
20.29.16.63
64.63
0.08
39.86
0.07_.0.07
PAAD
51.18:23.87 87
0.01
52.38
22.77
0.01
46
06.90:333
13 86
0.01
52. 52:97:22133
0.00
30.85
53
0.04
30.78 39 58
0.03
44.18 9.64
0.02:0011
VDN
PR-REP
PR
69. 09.78=17.32
62.23
0.00
70.20-16.50
65.21-20.77
16.94
0.00
0.00
61 01.26-22.882
60.74
0.00
0.09
60.74+11742
16.11719907
0.01
0.00
56.32+335.13
66.55 73.91
001
14.13157.00
$2.46+228 74
0.00
0.09
38.58
63.29-7.09
0.00-0.00
0.07-0007
ERNIE
67. 21.36-21.22 72
0.00
67.73 19.93
0.00
0.01
67 65+:20.77
0.00
44. 49.49: 50.03
0.03
43.67::3466 61
0.00
58.15-
0.01-061
RAP
66-21-16.9
06..6-111
0.00
57.03-12.38 54
66 16.95
45.41+31.4
58.55-8
0.00-0.00
ROMANCE-P
43.82.335.22 23
0.02
43:46 33.36
0.00
24.171.49.90
0.07
43.06.134.99
0.02
7.42.160.47
0.14
4.211154.69
0.11
26.29.19.06"
0.06_0005
ROMANCES
70.49L 18.60
0.00
24.10
0.01
24.411:05.55 65.56
0.31
20,08:
0.00
-5.412:33.25
0.20
1.61 67 03
0,11
38.58:232.77
0.07-0007
ATSA
08.57=22.00
68.76
0.00
56.05-37.115
0.02
19:42+199.8
40.76+550101
55.32+36.00
59.81=10.38
0.1-0002
NoAdy
30.50:30.97
0.01
61.75:31.71
0.02
36 53.25
0.06
62. 62.188129.36 36
0.01
16.48366544 48
0.12
9.8316366 63
0.08
41.15
0.0
RN
71.49 18.54
0.00
70.51-222.71
0.01
23.52.664.88
0.07
11.32119988
0.00
3..541.68.24
0.16
2.85.174.44
0.15
28.38.33..1
0.06__0.06
FGSM
58.57=27.18
59.72=27.12
36.72=33.077
0.02
59 29.06-+28.006
39,30=442.99
53.32=7.75'
0.02-0.011
AILA
45,96:245200
0.02
43.21=41.11
0,05
30.00
0.95
29.0031-6.774
0.04
41.141552.0
0106
29.20
008
44.75=7.89
0.05=0000
PAAD
57.43+23.45
0.00
58.75-25.11
1.00
47.96-35.80 80
0.02
59 36++21.0
0.00
57.44+23.45
0.00
35.56+30.47
0.01
52.75-8.58
0.01-0011
QMIX
PR-REP
PR
39, 39,92:129.26
08.52:22.78
0.01
60.88 28.02
68.99-19.39
0.00
0.01
52.58.136.21
61.67-226.22
0.01
0.02
60.58.1228088
19.79117.48
0.00
0.01
54.01
58.92.150.13
0.06
$1,27.34429
25.57:157.13
0.08
0.00
13,24113,26
61.95_16.97
0.03
0.00-00.00
ERNIE
59.12.27778
0.00
59.72 28.36
0.00
19.121.36.88
0.01
59 59.911228.08
0.00
45.98.137..55
0.00
23.27.146.80
0.02
19.52112.95
0.01 0.01
KAP
08.59 16.29
0.00
70.06= 15.15
0.00
63.11::26.19
0.01
16.55:16.50
0
59.18=228.3
0.01
47,04 44
0.03
63.14=8.26 26
0.010001
ROMANCE-P
24.93-29.11
58.66-26.22
0.00
54.222
60 00+00333336 36
51.06--53.32
12.05+66.44
41.47+17.71
0.05-0.05
ROMANCE+
23.26
0.00
64.69 64.68
0.01
28.
02.442..8.222
28
0.06
65 51121.92
0.00
15.66
0.08
-1.011611
0.09
39.42.12648
0.04-0.04
ATSA
70.17:19 19.56
0.00
70.54
18.57
0.00
62.01
26.66 66
0.01
69
10+ 14
0.00
56
56.01+31.56 56
0.02
53.94
30.66
0.01
63.74 6,87
0.01-001
indicates a statistically significant improvement of ATSA over the corresponding method (p < 0.05,
Wilcoxon runk-sum test).
ter a and the set Jee follow the same configuration as in (Zhang et al., 2021b). Additionally, our framework in- troduces an extra hyperparameter, K, which regulates the influence of the SDor-STor loss function on the SDor's policy. In our experiments, for VDN-based protagonist agent is selected from the set {0.001, 0.005, 0.01}, while for SMIX-baseed protagonist agent is chosen from the set {0.001, 0.005, 0.01, 0.025], depending on the environment to optimize performance.
sarial agents, which promotes exploration and reduces lo- cal optima risks. Under adversarial observations, NoAdy and RN fail handle attacks, while RN shows better per- formance against weaker adversaries like ATLA. PR per- forms well against FGSM but struggles under stronger at- tacks like PAAD and ATSA. Among all evaluated frame- works, ATSA achieves the highest average win rate 1e0--o0 out- performing 11 baselines across 3 scenarios with 42 statis- tically significant wins according to the Wilcoxon signed- rank test. Though it doesn't always perform best in ev- ery scenario (e.g.. 3s with QMIX), it remains generally superior across different adversarial conditions. In CAV, for instance, PR-REP performs well under the VDN frame- work, but fails under QMIX, indicating difficulty in tuning the regularization weight. Overall, ATSA surpasses most baselines in both cumulative reward and CR. In conclu- sion: 1) NoAdy and RN fail to enhance robustness. 2) Ro- bust learning methods (PR, PR-REP and ERNIE) face chal- lenges in balancing adversarial and standard losses. 3) Ad- versarial training methods (FGSM and PAAD) destabilize training under clean conditions; ATLA struggles with large action spaces, reducing its effectivenesss 4) Stochastic ad- versaries: ROMANCE requires attack budget constraints to stabilize training, which is incompatible with state ro- bustness learning, and RAP avoids constraints via adver- sary population, but introduces unstable duitt--disttibuttin states and incurs high memory cost. 5) ATSA overcomes overfitting issues by using stochastic adversaries, ensur- ing strong performance across clean and adversarial condi- tions. The more detailed experimental results are presented in Appendix H.3. The results and analysis regarding dif- ferent perturbation ranges are presented in Appendix H.5,
4.2. Results Analysis
The experimental results on SMAC are presented in Ta- ble 1, and those on CAV are shown in Table 2. The pro- tagonist model in each cell is trained by using the method specified by the row, while the adversary is trained by using the method specified by the column. Importantly, no re- training is performed for each cell, i.e., the adversary is not retrained specifically to target each protagonist. In SMAC, we use Win Rate (WR) as the metric to evaluate perfor- mance. To measure robustness, we calculate the average WR under different perturbations. In CAV, we use reward and Crash Rate (CR) to evaluate model performance, with the average reward and CR under different scenarios also calculated.
For perturbation-free and random noise cases, models trained with FGSM and PAAD show strong performance under adversarial conditions but perform poorly in clean or random noise scenarios. In contrast, our proposed method maintains stable even improved performance, particu- larly in CAV, likely due to the soft policy used by adver-
7

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
OIVDN)
ATSAIVDN)
((QMIX)
ATSAIQMIX)
NoAdv BBC
FGSM
PAAD
ATSA
1.0
1.0
1.0
0.8
0.8
0.6
Win
Win
0.4
0.4
0.2
0.2
0.2
0.2
0.0
NoAdv
FGSM
PAAD
ATSA
0.0
NoAdv FGSM
PAAD
ATSA
0.0
FGSM(VDN)
PAAD(VDN)
0.0
FGSM[QMIX)
PAAD(QMIX)
(a)
3m
(a) 3m
1.0
1.0
0.8
HIT
0.8
0.8
Rate
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
NoAdv FGSM PAAD ATSA
0.0
NoAdv FG5M PAAD ATSA
0.0
FGSM(VDN)
PAAD(VDN)
0.0
FGSM[QMIX]
PAAD(QMIX)
(b) 3s.3z
(b) 3s 3z
Figure 2. Ablation study. The x-axis represents four types of ad- versaries. The y-axis indicates the win rate of the models under different adversaries. In the legend, k = 0 represents the results after removing the SDor-STor loss function.
Figure Comparison of attack performance across adversaries and scenarios. The x-axis shows models adversarially trained with FGSM and PAAD. NoAdy indicates performance without perturhations. Bars labeled FGSM, PAAD, and ATSA represent win rates under attacks from these methods.
demonstrating the generalization ability of ATSA from the perspective of perturbation size. Appendix H.6 provides an analysis of the potential for applying ATSA to continuous action spaces.
As shown in Fig. 3a, when attacking QMIX-based agents, our method does not always produce the strongest at- tack results. However, when attacking FGSM(VDN)- agents, our approach is capable of achieving optimal attack performance. In the 3s_3z scenario, adversaries trained using our framework generally outperform other adversar- ial attack methods. This is because our method generates more diverse perturbations, thanks to the stochastic nature of the adversaries. For instance, as shown in Fig. 3b, QMIX-based agents trained with FGSM adversarial train- ing show strong defense against FGSM-gemerated pertur- bations. However, when attacked by our method, their performance drops significantly. This suggests that mod- els trained with FGSM-based adversarial training tend to overfit to their own perturbation patterns. In summary, our proposed adversarial training framework demonstrates the ability to create adversaries that are effective attackers. These adversaries help expose the weaknesses of existing models, providing a broader perspective for evaluating ro- bustness against adversarial perturbations.
4.3. Ablation Study
Fig. 2 presents the results of the ablation study on the ATSA method, evaluated under environments: (a) 3m and (b) 3z. = 0 represents the results after removing the SDor-STor loss function. It is evident that the ATSA method demonstrates strong robustness across various ad- versaries. Particularly in the PAAD and ATSA adversaries, its win rate significantly higher than that of the x = 0 counterparts, highlighting the crucial role in maintaining the consistency between SDor and STor, which ensures more stable training of the adversary. This, in turn, en- hances the adversarial robustness of the protagonist agent.
4.4. Attacking Performance Analysis
The results in Fig. 3, derived from Table 1, focus scenarios: 3m and 3s.3z. Protagonist agents are trained with adversaries like FGSM and PAAD. The comparisons include adversaries trained by ATSA alongside FGSM and PAAD. We selected FGSM and PAAD for comparison because they have shown strong attack performance in prior studies, and models trained using these methods ex- hibit better robustness against perturbations. Thus, we use ATSA to evaluate its attack capability.
5. Conclusion and Future Work
In this paper, we propose an ATSA framework that contains an SDor-STor structure and a novel loss function. ATSA trains the protagonist agent with stochastic adversary. ad- dressing the issues of adversarial observation overfitting and instability in MARL adversarial training. Furthermore, we theoretically prove that in the factorized maximum- entropy framework, the soft policy of SDor converges to global optimum, and this structure can derive the optimal stochastic observation adversary. To overcome the limita- tion of prior methods that fail exploit the policy infor-
It is important to note that adversaries trained using our method are stochastic, and the optimal adversarial observa- tions they generate adhere to soft constraints. This means the perturbations they create are not always the strongest.
8

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
mation of the protagonist agents, we introduce the SDor- STor loss function, which leverages the policy information of the protagonist agents when training the adversary poli- cies. Our future work aims to extend the framework to con- tinuous action space algorithms (Lowe et al., 2017; Zhong et al., 2024; Zhou et al., 2024b). Model-based reinforce- ment learning can also employed by considering pro- tagonist agents' policies and environments as a unified sys- tem. This approach allows the adversary to fully leverage the protagonist agents' information when learning the en- vironment model. ATSA's application to various systems (Hu et al., 2025; Lou et al., 2025) should be pursued.
Deng, Y., Li, Y., Ding, B., and Lam, W. Leveraging long short-term user preference in conversational recommen- dation via multi-agent reinforcement learning. IEEE Transactions on Knowledge and Data Engineering, 35 (11):11541-11555 2023.
Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradi- ents. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018.
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explain- ing and harnessing adversarial examples. arXiv preprint arXiv: 1et..6112.65 2014.
Acknowledgements
Gui, T., Liu, P., Zhang, Q. Zhu, L.,, Peng, M., Zhou, Y., and Huang, X. Mention recommendation in twitter with cooperative multi-agent reinforcement learning. In Pro- ceedings of the 42nd International ACM SIGIR Confer- ence Research and Development in Information Re- trieval, SIGIR' 19, pp. 535-544, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450361729.
This work was supported by the National Nature Science Foundation of China under Grant 62172299 and Grant 62032019, and the Fundamental Research Funds for the Central Universities under Grant 3023-4-YB-05
Impact Statement
This paper presents ATSA, method designed to train robust models against diverse observation perturbations. ATSA is applicable to wide range of real-world scenar- ios, including autonomous driving and other safety-critical systems.
Guo, J., Li, A., and Liu, C. Backdoor detection in rein- forcement learning. arXiv preprint arXiv:2202.03609 2022.
Guo, W., Liu, G., Zhou, Z., Wang, L., and Wang, J. En- hancing the robustness of qmix against state-adversarial attacks. Neurocomputing, 572:127191, 2024. ISSN 0925-2312.
References
Bukharin. A., Li, Y., Yu, Y., Zhang, Q. Chen, Z., Zuo, S., Zhang, C., Zhang, S., and Zhao, T. Robust multi- agent reinforcement learning via adversarial regulariza- tion: Theoretical foundation and stable algorithms. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Informa- lion Processing Systems, volume 36, pp. 681 68121-68133 Curran Associates, Inc., 2023.
Guo, W., Liu, G., Zhou, Z.,, Wang. J., Tang, Y., and Wang, M. Robust training in multiagent deep reinforcement learning against optimal adversary. IEEE Transactions Systems, Man, and Cybernetics: Systems, pp. 1-12, 2025.
Hinton, G. Lecture 6.5 rmsprop: Divide the gradi- ent by a running average of its recent magnitude. https //::://wwee /tww//wwwcooo cs. toranto. edw/ csc321/slides wtliiideslectur____e htts//eeeturesideessseeeeeeeeeee
Cai, Q., Cui, C., Xiong, Y., Wang, W., Xie, Z., and Zhang, M. A survey on deep reinforcement learning for data processing and analytics. IEEE Transactions on Knowl- edge and Data Engineering, 35(5):446 4465, 2023.
.df, 2012. Coursera: Neural Networks for Machine Learning.
Hu, X., Jiang, X., Zhang, J., Wang, S., Zhou, M., Zhang, B., Gan, Z., and Yu. B. Electric vehicle charging network security: A survey. Journal of Systems Architecture, 159: 103337, 2025. ISSN 1383-7621.
Chen, D., Hajidavalloo, M. R., Li, Z., Chen, K., Wang. Y., Jiang, L., and Wang, Y. Deep multi-agent reinforce- ment learning for highway on-ramp merging in mixed traffic. IEEE Transactions Intelligent Transportation Systems, 24(11):11623-1165) 2023.
Huang, S. H., Papernot, N., Goodfellow, J., Duan, Y., and Abbeel, P. Adversarial attacks on neural network poli- cies. In 5th International Conference on Learning Rep- resentations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. URL https: https:// opwpreview.eeeeeepeeeee net Forum? id=ryvl id=ryv1RyBK1.
Chen, S., Liu, G., Zhou, 7.,, Zhang, K., and Wang, J. Robust multi-agent reinforcement learning method based on ad- versarial domain randomization for real-world dual-uav cooperation. JEEE Transactions on Intelligent Vehicles, 91):1615-1627, 2024.

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
Levine, S. Reinforcement learning and control as probabilistic inference: Tutorial and review. CoRR, 00011805.00000 2018.
Oliehoek, A. and Amato, C. A Concise Introduction to Decentralized POMDPs. Springer Briefs in Intelligent Systems. Springer, 2016. ISBN 978-3-319-28927-44
Peng. B., Rashid, T., Schroeder de Witt, C.,, Kamienny, P.-.., Torr, P., Boehmer, W., and Whiteson, S. FAC- MAC: Factored multi-agent centralised policy gradients. In Ranzato, M., Beygelzimer, A., Dauphin. Y., Liang, P., and Vaughan, W. (eds.), Advances in Neural Informa- tion Processing Systems, volume 34, pp. 12208-12221. Curran Associates, Inc., 2021.
Li, S., Wu, Y. Cui, X.. Dong, H., Fang, F., and Russell, S. Robust multi-agent reinforcement learning via mini- max deep deterministic policy gradient. In The Thirty- Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artifi- cial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intel- ligence, FAAI 2019, Honolulu, Hawaii, USA, January 27 -eeruary 2019, pp. 4213-4220, AAAI Press, 2019.
Rashid, T., Samvelyan, M., De Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S. Monotonic value function factorisation for deep multi-agent reinforcement learn- ing. J. Mach. Learn. Res., 21(1), jan 2020. ISSN 1532- 4435.
Li, S., Xu, R., Guo, J., Feng, P., Wang. J., Liu, A., Yang. Y., Liu, X., and Lv, W. MIR2: towards provably robust multi-agent reinforcement learning by mutual informa- tion regularization. CoRR, 10.0933,,, 2023.
Reddi, A., Tölle, M., Peters, J., Chalvatzaki, G., and D' D'Ermmo C. Robust adversarial reinforcement learn- ing via bounded rationality curricula. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024. URL https:// wwwprpeeie..neeeeeeeeeee forum? 1=tp:iddaFOoOoaaaaao
Liang Y. Sun, Y., Zheng, R., and Huang, F. Efficient ad- versarial training without attacking: Worst-case-awaee robust reinforcement learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Sys- tems, volume 35, pp. 22561.22561. Curran Associates, Inc., 2022.
Samvelyan, M., Rashid, T., Schroeder de Witt, C., Far- quhar, G., Nardelli, N., Rudner, T. G. J., Hung, C.-M., Torr, P. S., Foerster, J., and Whiteson, S. The starcraft multi-agent challenge. In Proceedings of the 18th Inter- national Conference on Autonomous Agents and Multi- Agent Systems, AAMAS '19, pp. 2186-2188, Richland, SC, 2019. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781 01811550660999
Lou, S., Hu, Z., Zhang, Y., Feng, Y., Zhou, M., and Lv, C. Human-cyber-physiinn system for industry 5.0: A re- view from human-centric perspective. IEEE Transac- tions on Automation Science and Engineering, 22:494- 511, 2025.
Lowe, R., WU, Y., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I. Multi-agent actor-critic for mixed cooperative-compettive environments. In Guyon, I., Luxburg. U. V.. Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347 2017.
Shi, H., Liu, G., Zhang. K., Zhou, Z., and Wang, J. Marl sim2real transfer: Merging physical reality with digi- tal virtuality in metaverse. IEEE Transactions Sys- tems, Man, and Cybernetics: Systems, 53(4):2107-21177 2023.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: //open net fort///ormmmmm/ iitmmmdd=rizzii d
Son, K., Kim, D., Kang, W. J., Hostallero, D. E., and Yi, Y. QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learn- ing, volume 97 of Proceedings of Machine Learning Re- search, pp. 5887-5896. PMLR, 09-15 Jun 2019.
Oikarinen, T. P., Zhang, W., Megretski, A., Daniel, L., and Weng, T. Robust deep reinforcement learning through adversarial loss. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NeurlPS' 21, pp. 26156-26167. Curran Associates, Inc., 6-14 Dec 2021.
Sun, C., Kim, D., and How, J. P. ROMAX: certifiably robust deep multiagent reinforcement learning via con- vex relaxation. In 2022 International Conference on Robotics and Automation, ICRA 2022, Philadelphia, PA. IEEE, 2022a.
USA, May 23-27, 2022, pp.
10

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
Sun, Y., Zheng, R., Liang, Y., and Huang. F. Who is the strongest enemy? towards optimal and effi cient evasion attacks in deep RL. In The Tenth In- ternational Conference on Learning Representations, ICIR 2022, Virtual Event, April 25-29, 2022. Open- Review.net, 2022b. URL https:/ /openrevin...... net/ forum? wwt.:/oouuuuduaiooe ddd-MM2K..WVII.
Wang. Y., Li, K., Luo, Y., Li, G., Guo, Y., and Wang. Z. Fast, robust and interpretable participant contribution estimation for federated learning. In 40th IEEE Inter- national Conference on Data Engineering. ICDE 2024, Utrecht, The Netherlands, May 13-16, 2024, pp- 2298- 2311. IEEE, 2024b.
Wen, X. and Zhou, M. Evolution and role of optimizers in training deep learning models. IEEE/CAA Journal of Automatica Sinica, 11(10):2039-2422 2024.
Sun. Y.. Zheng, R., Hassanzadeh, P., Liang, Y., Feizi, S., Ganesh, S., and Huang, F. Certifiably robust pol- icy learning against adversarial multi-agent communi- cation. In The Eleventh International Conference Learning Representations, 2023. URL https:// openreview. www.um forum?
Wong, E., Rice, L.,, and Kolter, J. Z. Fast is better than free: Revisiting adversarial training. In 8th International Con- ference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: openreview net/ forum? id=BJx040EFW
Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., and Graepel, T. Value- decomposition networks for cooperative multi-agent learning based team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '18, pp. 2085-2087, Richland, SC, 2018. International Foundation for Au- tonomous Agents and Multiagent Systems.
Yang, M., Liu, G., Zhou, Z., and Wang. J. Probabilistic automata- based method for enhancing performance of deep reinforcement learning systems. JEEE/CAA Jour- nal of Automatica Sinica, 11(11):237722333 2024.
Yang, Y., Luo, R., Li, M., Zhou, M., Zhang, W., and Wang, J. Mean field multi-agent reinforcement learning. In Dy, J. and Krause, A. (eds.). Proceedings of the 35th Inter- national Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5571 5580. PMLR, 10-15 Jul 2018.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Tampuu, A., Matüisen, T., Kodelja, D., Kuzovkin, I., Kor- jus. K., Aru, J., Aru, J., and Vicente, R. Multiagent coop- eration and competition with deep reinforcement learn- ing. PLOS ONE, 12,4):1-15, 04 2017.
Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., and WU, Y. The surprising effectiveness of ppo in operative multi-agent games. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Sys- tems, volume 35, pp. 24611 1-62242 Curran Associates, Inc., 2022.
Vinitsky, E., Du, Y., Parvate, K., Jang, K., Abbeel, P., and Bayen, A. Robust reinforcement learning using ad- versarial populations. arXiv preprint arXiv:2008.0182) 2020.
Yuan, L., Zhang, Z., Li, L., Guan, C., and Yu, Y. A sur- vey of progress cooperative multi-agent reinforce- ment learning in open environment. arXiv preprint arXiv:2312.01008, 2023a.
Wang, J., Ren, Z., Liu, T. Yu, Y., and Zhang, C. QPLEX: Duplex dueling multi-agent q-learning. In International Conference on Learning Representations, 2021. URL https: http:////eneeeeeeeeeeeeeo id=RcmkOxxIQv.
Yuan, L., Zhang, Z., Xue, K., Yin, H., Chen, F., Guan, C., Li, L.,, Qian, C., and Yu, Y. Robust multi-agent coordina- tion via evolutionary generation of auxiliary adversarial attackers. Proceedings of the AAAI Conference on Arti- ficial Intelligence, 17(10):11733-1177
Wang, J., Ye, D., and Lu, Z. More centralized training, still decentralized execution: Multi-agent conditional policy factorization. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.nee 2023.
Jun. 2023b.
Zhang, H., Chen, H., Xiao, C., Li, B., Liu, M., Boning, D.,, and Hsieh, C.-J. Robust deep reinforcement learning against adversarial perturbations In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 6-12 Dec 2020. Curran Associates Inc ISBN 978171382927
Wang, S., Tang, S., Ruan, S., Long, C., Liang, Y., Li, Q. Yuan, Z., Bao, J., and Zheng, Y. Urban sensing for multi- destination workers via deep reinforcement learning. In 2024 IEEE 40th International Conference on Data En- gineering (ICDE), pp. 4177-4179, 2024a.
state observations.
11

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
Zhang, H., Chen, H., Boning, D. S., and Hsieh, C. Robust reinforcement learning state observations with learned optimal adversary. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. Open- Review.net, 2021a. URL https:/ /ttp/////ppee.....eeee net/ dt
Zhang, T., Li, Y., Wang, C., Xie, G., and Lu, Z. Fop: Fac- torizing optimal joint policy of maximum-entropy multi- agent reinforeement learning. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 12491-12500. PMLR, 18-24 Jul 2021b.
Zhang, W. Liu, H., Xiong, H., Xu, T., Wang, F, Xin, H., and Wu, H. Rlcharge: Imitative multi-agent spa- tiotemporal reinforcement learning for electric vehicle charging station recommendation. IEEE Transactions on Knowledge and Data 3 35())6290-63334 2023.
Zhong, Y., Kuba, J. G., Feng, X., Hu, S., Ji, J., and Yang. Y. Heterogeneous- agentt reinforcement learning. Journal of Machine Learning Research, 25(1-67):1, 2024.
Zhou, T., Liu, N., Song, B., Lv, H., Guo, D., and Liu, Robil: Robust federated learning via feature center sep- aration and malicious center detection. In 40th IEEE International Conference Data Engineering, ICDE 2024, Utrecht, The Netherlands, May 13-16, 2024, pp- 926-938. IEEE, 2024a.
Zhou, Z. and Liu, G. Robustness testing for multi-agent reinforcement learning: State perturbations on critical agents. In Gal, K., Nowé, A., Nalepa, G. J., Fairstein, R., and Radulescu, R. (eds.), ECAI 2023 - 26th European Conference on Artificial Intelligence, September 30 Oc- tober 2023, Kraków, Poland Including 12th Confer- ence on Prestigious Applications of Intelligent Systems (PAIS 2023), volume 372 of Frontiers in Artificial In- telligence and Applications, pp. 3131-3139. IOS Press, 2023.
Zhou, Z. Liu, G.. Guo, W. and Zhou, M. Adversarial attacks on multiagent deep reinforcement learning mod- els in continuous action space. IEEE Transactions Systems, Man, and Cybernetics: Systems, 54(12):7633- 7646, 2024b.
Zhou, Z., Liu, G., and Zhou, M. robust mean-field actor- critic reinforcement learning against adversarial pertur- bations on agent states. IEEE Transactions Neurul Nenworks and Learning Systems, 35(10):14370-1438 2024c.
12

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
A. Related Work
A.1. Aulti-Agent Reinforcement Learning
Centralized Training with Decentralized Execution (CTDE) is a popular paradigm to address the challenges of Decentral- ized Partially Observable Markov Decision Processes (Dec-POMDPs). During the training phase, all agents share global information for centralized training to optimize the overall policy. In the execution phase, each agent independently makes decisions based its local observations, performing decentralized execution. This paradigm effectively addresses the non-stationarity and partial observability issues in MARL (Foerster et al., 2018; Lowe et al., 2017). Independent deep Q-network (Tampuu et al., 2017) is simple approach but struggles with complex scenarios due to challenges in reward assignments. Value-based methods like (Rashid et al., 2020; Son et al., 2019; Sunehag et al., 2018) progressively address these limitations by introducing shared rewards, non-linear structures, and broader applicability. More recent methods such as (Wang et al., 2021; 2023; Zhang et al., 2021b) further enhance coordination by leveraging advanced factorization tech- niques, improving performance across diverse environments and action spaces. However, these classical methods remain sensitive to observation perturbations.
A.2. Robust Single-Agent Reinforcement Learning
In the field of SARL, improving model robustness relies on two approaches: robust learning and adversarial training. The works in (Liang al., 2022; Oikarinen et al., 2021; Zhang et al., 2020) are currently the most representative method in robust learning. Zhang et al. (Zhang et al., 2020) develop policy regularization principle that can be broadly applied to various SARL algorithms. In addition, the State- Adivesrrraaai Markov Decision Process (SA-MDP) framework incorpo- rates adversarial perturbations in states, providing theoretical foundation for robust SARL. In constructing the adversarial regularization term, Liang et al. (Liang et al., 2022) consider not only adversarial perturbations of states but also the sig- nificance of the current state within the entire episode, This approach enables the policy network to focus more on the performance of critical states during updates, further enhancing the model's robustness. Unlike the previous two methods, the work in (Oikarinen et al., 2021) employs the concept of interval-bound propagation to construct regularization term. This approach enhances model robustness by optimizing the boundaries of the decision network. While robust learning can enhance model's robustness, models trained this way are still sensitive to more potent attacks. Yang et al. (Yang et al., 2024) improves the robustness of reinforcement learning systems by leveraging monitoring mechanisms based on probabilistic automata to guide real-time correction of critical actions Zhang et al. (Zhang et al., 2021a) use reinforce- ment learning to Alternate Training protagonist agents with Learned Adversaries (ATLA). This approach performs well in scenarios with small observation spaces. However, as the agent's 's observation space grows, leading to larger action space for the adversary, training the adversary model becomes challenging. Sun et al. (Sun et al., 2022b) demonstrate that observation perturbations are equivalent to policy perturbations. Based this theoretical foundation, the Policy Ad- versarial Actor and Director (PAAD) is proposed, decomposing the solution of the observation adversary into two steps: the director determines the strongest policy perturbation, and then the actor generates the optimal observation perturbation based on this policy perturbation. This approach effectively addresses the issues present in ATLA. PAAD aims to enhance a model's 's performance against the strongest adversaries. However, this approach may lead to overfitting to these adversarial perturbations of observations, thereby reducing performance on clean ones.
A.3. Robust Multi-Agent Reinforcement Learning
In the multi-agent domain, most existing studies primarily focus on action manipulation (Li et al., 2019; 2023; Sun et al., 2022a), policy-level attacks (Guo et al., 2022; Reddi et al., 2024), or adversarial communication (Sun et al., 2023). Other works adopt domain randomization to address the sim-to-real gap (Chen et al., 2024; Shi et al., 2023). In contrast, relatively little attention has been paid to adversarial perturbations on observations. By extending SA-MDP multi-agent fields, Zhou et al. (Zhou et al., 2024c) propose state-adversarial stochastic game and discuss its properties and propose robust training framework based on mean-field actor-critic (Yang et al., 2018), which includes an action loss based on the difference in action distributions between clean and perturbation-free observations, along with the iterative regularization method for action loss. However, this method falls within the scope of robust learning and is also sensitive to stronger adversaries. Guo et al. (Guo et al., 2024; 2025) extend SARL robustness techniques (Goodfellow et al., 2014; Sun et al., 2022b; Zhang et al., 2020; 2021a) to multi-agent scenarios, where multi-agent PAAD achieves notable results. However, it tends to overfit the perturbation generated by the strongest adversary, compromising performance in other situations. Stochastic adversary has been studied in policy robustness in some papers, such as ROMANCE (Yuan et al., 2023b) and
13

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
RAP (Vinitsky al., 2020). However, there are significant differences between poliey robustness and state robustness in terms of both formulation and motivation. Formally, policy robustness assumes that the adversary directly perturbs the agent's ss actions, modifying the output of the policy. In contrast, state robustness introduces perturbations to the agent's s input states. This fundamental difference in formulation directly leads to distinct objectives. In multi-agent settings, policy robustmess aims to ensure that the overall system performance remains stable even when some agents' decisions are perturbed. To facilitate stable training under such settings, methods like ROMANCE (Yuan et al., 2023b) introduce a sparse action attack budget to limit the number of adversarial interventions, thereby preserving controllability. In contrast, state robustness focuses on enabling each agent to make correct decisions even when its input states are perturbed. The goal is to make every agent robust to state perturbation. Therefore, the sparse action perturbation constraints used in ROMANCE (Yuan et al., 2023b) are not applicable to state perturbations.
Table 3. List of Abbreviations and Their Full Terms
Abbreviation
Full Term
ATLA
Alternate Training Protagonist Agents with Learned Adversaries Adversarial Training with Stochastic Adversary
ATSA
CAV
Connected and Autonomous Vehicles
CR
Crash Rate
CTDE
Centralized Training with Decentralized Execution
Dec-POMDP
Decentralized Partially Observable Markov Decision Process Fast Gradient Sign Method
FGSM
MARL
Multi-Agent Reinforcement Learning
NoAdv
No Adversary
OD-POMDP
inpervation-adversarii Rec-POMDP
PAAD
Policy Adversarial Actor and Director
PD-POMDP
Policy-adversarial Dec-POMDP
PR
Policy Loss Regularization
QMIX
Q-MIXing network
RN
Random Noise
SARL
Sigle-Agent Reinforcement Learning
Sdor
Stochastic Director
SMAC
StarCraft Multi-Agent Challenge
Stor
SDor- -Dorrguided generaTor
VDN
Value Decomposition Network
WR
Win Rate
B. Concepts, Abbreviations, and Symbols
B.1. Concepts
Some important concepts related to this paper are described as follows:
Protagonist agent:
The primary agent whose learning objective is to maximize the cumulative expected reward. Clean observations: The accurate observation information directly obtained by an agent from the environment.
Protagonist policy: The decision- makinn strategy of the protagonist, designed to guide it in selecting optimal actions based on its observations.
Observation adversary: An adversary that generates adversarial perturbation to the agent's 's observations information, aiming to mislead the protagonist agent into making suboptimal decisions.
Policy adversary: An adversary to modify the protagonist's 's's policy often chooses actions that minimize the protagonist agent's cumulative expected reward.
14

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
Adversarial observations: clean observation.
The observation via adding perturbations generated by the observation adversary to the
Adversarial policy: the protagonist policy.
The decision-making strategy of the policy adversary, specifically designed to disrupt mislead
B.2. Abbreviations
The abbreviations in the paper are listed in Table 3.
B.3. Symbols
The symbols in the paper are listed in Table 4.
Table Symbols and Explanations
Category
Symbol
Explanation
State space
Action space for protagonist
Action laken by protagonist Joint action of all protagonists
Environment
O
Observation space of protagonist i
of
Observation of protagonist
Z
Observation function
Reward function
Transition probability function
Discount factor
A set of protagonist agent
Policy of protagonist
r
Trajectory of protagonist
m
Joint policy of all protagonists
Joint trajectory of all protagonists Action laken by protagonist agent i based on the clean observation Parameters of the policy network of protagonist agent
Protagonist Agents
g
me
Joint policy of protagonist agents under the adversary Adversarial observation protagonist
Joint observation of protagonist agents, with some agents perturbed
Joint trujectory of protagonist agents, where some agents are perturbed
g'
The adversarial observation generated for the agent hased on the action of the policy adversary
B
A set of adversarial observations for protagonist
M
Set of observation adversaries
Joint action of observation adversary
Policy of adversary
Adversaries
Joint policy udversary Action space for adversary
A
P
Transition probability function of adversary
Reward function of adversary
Dec-POMDP
Models Frameworks
Goa
OD-POMDP
PD-POMDP
Temperature parameter
Learning Parameters
Step size
SDor's trajectory
Joint soft policy of all SDors Joint trajectory of all SDors
7
SDor's Parameters and Trajectories
Q
Optimal joint soft Q-function
Optimal joint soft value function
Bellman backup operator
d
Parameters of the weight network
g SDor's Network Parameters
Parameters of the joint soft value network
g
Parameters of the individual soft value network
gi
Parameters of the individual soft-Q network
Replay buffer SDor
Replay Buffer and Target Networks
Q
Target network of joint soft-Q network
15

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
C. Maximum Entropy-based Objective Function of SDor
The derivation process of Definition 3.1 as follows.
Proof: To derive the maximum entropy objective, we approximate the true distribution of trajectories P given by
P (sctalee, an) exp Σ
P())
(() II
(16)
1=1
1=1
with a variational distribution, ④ (7), as
@ (T) (s1) II
(944155 âr) (ār(s:)
t-1
(17)
T
q (ss) II
g (s ae) II q (a/lse)
where P (st-1/82, â.) and / (st+1|82, a+) share the same dynamics, i.e. ((1) = (81) and ((s+t1lst,d.) d:) = q (8t+1182. âz) and á (814 185, âs) = IT q² (āfls)) following the CTDE paradigm.
The variational lower bound on the log-lroobe peg-probability of the optimality sequence Xi. given by:
logE (Kir) = log / / (Kirr, 81., a1.r) dsi:rdār.
log / //((2 (XK: SI.T a1.))
á (11.T.117) dsp:rd
① (81.7, a1.T)
(18)
p (Ki,,, 81:T a1.T)
(o amr)
(81.7, @IT))
[logp (Xirr, 81.7, åir)) logd (sI.T, a1.7) )]
Given that á ($t+1152, a) = á (8tt1l1s, a). and substituting (16) and (17) into the bound, we obtain:
T
A ( āt) - Elogg' logg' (ailss) )
logE (Aur)) 2 E ((a
Σ
(19)
Optimizing this lower bound with respect the adversarial policy ((â,{se) corresponds exactly the following maximum entropy objective (2).
D. Proof of Propositions 3.2 and 3.3
The derivation process for Propositions 3.2 and 3.3 is the same; here, we only present the derivation for Proposition 3.2.
Proof: To derive the optimal joint adversarial policy hh,,, start by considering the following constrained policy optimiza- tion problem:
max N
Ea-an [Q" (T",a)] a
Σ
12" (2ll") (álT")
aEXieMAi
(20)
s.l. Σ
1 ('(TT")=
aEXiemA:
To solve this, we construct the Lagrangian:
C(he", 2) = Ea-ia [Q" (T (7" -a Σ logh" Σ htth (a]7=)) - 1)
dEXiEMA'
dEXiEMA'
(E ( -a Σ 12th (al7") (a]7")- Σ 12th (a(7 - 1)
dEXiEMAi
aEXiEMÁi
âEXIEMA
16

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
Taking the derivative of L (W', A) with respect 1td (ál7) and setting zero yields:
(L(N,,) Qt (7", à) alogh" (alT") a == Oh (a(T*)
(21)
Solving this for the optimal policy li!, get
A (a-1)
1 (a|+") = exp (ap (7",")))) exp
(22)
Using the normalization condition
Σ Eaexiendi hit (ālTT)) = 1, we find the optimal A,
(1
1 A.
exp
exp (a QQ" (T", a))
(23)
aEXieMA
Thus,
d+
1 log
Σ
exp Qa (T, (T, a
(24)
àc aEXimAA
Substituting X, back, the optimal policy het becomes:
exp (a- (T" (T",a)
No (āls)
Ene exp (a- Qut (Tm, a)
(T², a)). we conclude that: Since V22l (7¹) alog E Eaexiend exp (a=1 (T
12 (a]T")= exp (a (Q (T", a) - 122 (T9))
(25)
E. Proof of Theorem 3.4
To prove Theorem H.6, we need the following Lemmas, the proof's of which follow a similar methodology to that presented in (Zhang et al., 2021b).
Lemma E.1 (Joint Soft Policy Evaluation of SDor). Consider the soft Bellman backup operator Di and a mapping L S XieM A + R with | XiEM A < cc. Define Qi_ A N Then, the sequence Q¹ will converge to the joint soft Q-function under the policy het as + co.
This result guarantees the convergence of the joint soft Q-function for a given joint policy of SDor h" Following this, the joint soft policy is updated based the individual soft policies I ie. under the CTDE paradigm. To ensure consistency and improvement in the individual policies, the following constraint is imposed during the policy update:
h how arg min DKL. (M (-17f=) exp (a5 ( (rih,-) - Vila (7
(26)
This formulation ensures that the new individual policy minimizes the divergence from exponential transformation of the Q-function. The improvement achieved by this update is formalized in the following lemma:
Lemma F.2. [Individual Soft Policy Improvement of SDor] Let Was E TT'h and Whew be the optimizer of the minimization problem defined in (26). Then, we have ei N (T.. a.) 2 Qe R2 (Tf a.) for all (Tf, a.) E "XEMMT XieM A with
XIEM Al < 0o, where V IIiE Way and bilt ITE h neo
17

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
This lemma demonstrates that each iteration of the individual soft policy update improves the value of the joint soft Q-function. By alternating between the joint soft policy evaluation and individual soft policy improvement, the overall factorized soft policy iteration (Zhang al., 2021b) achieves convergence. This is established in Theorem 3.4, The proof of Theorem H.6 is as follows:
Proof: Let kyg represent the soft joint policy at iteration k. Based on Lemma E.2, the sequence Q is monotonically R/
increases. Since 2% is bounded above for all hd" llighh this sequence converges to some N."
At convergence, we must have:
Jet ( (-17)) I Jhe (h (-17")) vh + h
(27)
Using the same iterative argument as in the proof of Lemma E.2, we conclude that: 2h (T", ( à) E XieMT' XieM A Therefore, Aett is optimal in the product space Iie..
S (7", a)
>
F. Proof of Theorem 3.5
Proof: The logic of the proof is summarized as introducing an intermediate problem GG to connect Gya and G. leveraging equivalence and proof by contradiction to ensure the optimality of the policy, thereby validating the theorem's 's conclusion.
Definition F.1 (Policy Perturbation Dec-POMDP). For a given Dec-POMDP G. a fixed joint deter- ministic protagonist policy n², and an attack define a policy perturbation Dec-POMDP as 9p A (S, (O') heM, (4pheM, (Z') KEM,, Pp:7), where Ai. = A', and Vs E S. E A
rp(s, ap) ) - (= if
(28)
00
otherwise
(p)) = >e>0=>aapp (ap aEA
(29)
where ap is the protagonist's policy under policy perturbations and A A XiEN is the joint action space of protagonist.
Let lif. denote the optimal joint policy in 9p. We define the cntropy-augmentee reward as ip (st, ap.) - rp (8t, ap,) + then the soft Bellman equation for
P H (nB (17 where H,g - E
H h (17?))
P
nip. is given by
1/- ñax F hB (T")) +7 Σ
Pp (6'ls, Vi
S'ES
max- Σ h)((al")r(ss,d) +7 Σ
Σ
( + Hay
(30)
acA
s'E
max
Σ (1((a)T))
-r(s,a)+ +1 Σ
P(d
+ H
hp asA
e'ES asA
In the original Dec-POMDP G, an optimal policy adversary the for Thll is defined as
V
min
Σ ((((TT acA
((s,a)+ + P(s' a) (6)
ΣΣ
H
(31)
s'ESaEA
By comparing (30) and (31), we have the following lemma,
Lemma F.2. The optimal joint policy in Gp is an optimal joint policy of observation adversary for n² in G.
Let 12 denote the optimal joint policy in 9pa- This policy hit induces a joint policy 12) in 9p- Assume, for contradiction,
that 1/ is not an optimal joint policy in G,. Then there exists an alternative optimal joint policy, denoted ni in G, such
that 1/ (T²) > Vi (TP) ) for at least one state s E S.
18

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
Algorithm ATSA
Initialize SDor's policy h/ _ NeM including d", , 1 ,'','',, ,p hem and protagonist's policy x²"; budget e. for t = 0, 1, 2,. do
For adversary, sample adversarial policy based on SDor's joint soft policy hl and generate the adversarial observation based on STor's function (13);
Protagonist takes action a, T +"):
Put (st. Os, ae s fz) in the adversary's 's replay buffer D and (st, Ōz, a, S241, (2)) in the protagonist's replay buffer DP;
if Time to update then
Update SDor's network parameters based on (7), (8), (9) and (15);
Update the protagonist's 's network parameters based on its algorithm:
end if end for
We now construct another joint soft policy lt in 9mm by setting: We (-17a) = 11 (.172). This means 1/2 follows the same action distribution hhl in Gp. In our framework, the policy he and 1942 are both stochastic, each of satisfies (1I), which implies,
1t:2 (-IT") T (g (1 (-17"), .0)) +0)
(32)
For the values of the policies, we have:
V/2² = +
PV + +HW
(33)
S'ES
Tp
p Vp
+H
(34)
8'E8
Since h matches h in terms of action distributions at each state, we conclude Vi (7²)= = VWa (T*). Therefore, we have:
V/ v
This leads to contradiction, indicating that an optimal soft joint policy in Spa must induce an optimal soft joint policy in 9p. Hence, we conclude the lemma:
Lemma F.3. An optimal soft joint policy in 0pa induces an optimal soft joint policy in Gp.
By combining Lemma F.2 and Lemma F.3, we conclude that optimal joint policy in Gya induces optimal stochastic observation adversary for the protagonist n² G. Therefore, SDor learns the optimal policy in Spa. it can effectively collaborate with STor generate the optimal stochastic observation adversary.
G. Complexity Analysis
The training process of ATSA is shown in Algorithm 1. We analyze its overall time complexity.
Theorem G.I. If the time complexity of sampling sample adversarial policy based on SDor's joint soft policy 1e' is Dpy computing the adversarial observation based STor's function (13) is Dg., and taking the joint action of protagonist interacting with the environment is Dp2 respectively, the time complexity of sampling process is
D1 = OT + +Dp + +Dg)))
where T is the total training time steps.
If the time complexity of computing the gradient of (7), (8), (9) and (15) is Dap- D ga. Dg, and Dap respectively, and the protagonist's 's network training is D the training time complexity of updating networks is
D2 0
(T T (Dga + Dg + Dg, + Dgo + Dga))
T
19

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
where Tj is the interval between neural network parameter updates. The time complexity of ATSA O (D1 D2).
H. Experimental Results
H.1. Environment settings
We evaluate our adversarial training framework on two challenging benchmarks: the StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019) and a Connected and Autonomous Vehicles (CAV) environment (Chen et al., 2023).
SMAC, distributed real-time strategy game, is a widely adopted benchmark for evaluating MARL performace It features dynamic, partially observable environments with high-dimensional state and action spaces, requiring agents to coordinate effectively. This makes SMAC ideal testbed for assessing robustness under adversarial conditions. In contrast, the CAV environment simulates real-world scenarios where connected and autonomous vehicles must collaborate to navigate traffic, avoid collisions, and optimize performance amidst uncertainties. We conduct experiments on three SMAC maps containing 3 Marines (3m), 3 Stalkers_ _v_3 Zealots (3s_3z), and 8 Marines (8m) and one scenario on CAV where three autonomous driving vehicles and 1-4 human-driven ones.
In SMAC, we adopt the default episode horizon provided by each environment. The observation space, action space, and reward structure are described follows.
Observation space: At each timestep, agents receive localized observations within their field of view. These observa- tions include information about both allies and enemies, such as distance, relative coordinates (I, y), health, shield, and unit type. The environment's global state provides a complete view all units on the map, including agent po- sitions relative to the map center and other relevant features from their observations. The global state is exclusively used during centralized training. In testing, only the agents' local observations are perturbed in adversarial scenarios.
Action space: Each agent has four discrete actions available: movement, attacking, stopping, no-operation. sttppiinggstooppnng and Move- ment is limited to the four cardinal directions (north, south, cast, and west). Agents attack enemies if they within range. The stop action leaves the agent idle, while the no-operation (no-op) action is only applicable when the agent is eliminated.
The primary objective is to maximize the agents' win rate. SMAC employs a shaped reward system, where agents carn rewards based on the damage they inflict during each timestep. Additional rewards are granted for defeat- ing individual opponents and achieving victory by eliminating all enemy units. All rewards are normalized, ensuring that the maximum cumulative reward for an episode does not exceed 20.
Reward:
In the CAV environment, we set the episode horizon to 100 timesteps. The corresponding observation space, action space, and reward structure are summarized as follows:
Observation space: Each agent (vehicle) observes nearby vehicles within a 150-meter range along the longitudinal axis. The observations include features such as whether a vehicle is present, its relative longitudinal and lateral positions (x, y). and its relative longitudinal and lateral speeds By). Only the nearest neighboring vehicles are considered observable due to local observability constraints.
Action space: Agents make high-level control decisions, including turning left, turning right, cruising, speeding up, and slowing down. These decisions are translated into low-level steering and throttle commands by the controller. The joint action space of the system is the Cartesian product of all agents' individual action spaces.
The reward function incentivizes agents to exhibit desired behaviors, such as safely and efficiently merging into traffic. Rewards are based on several factors: collision avoidance, maintaining stable speeds, minimizing headway time, and successful merging. Each component is weighted to balance the objectives.
Reward:
H.2. Benchmark methods
All benchmarks are implemented based on the classical MARL, Value Decomposition Network (VDN) (Sunehag et al., 2018) 88) and Q-MIXing network (QMIX) (Rashid al., 2020). We use the following methods as the benchmark:
20

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
NoAdv. This baseline uses the standard training approach with VDN and QMIX, without introducing any perturba- tions during training.
RN. In this setup, the protagonist is trained with random noise applied to its observations.
FGSM (Goodfellow et al., 2014). The protagonist is trained with adversarial observation generated using the FGSM. These perturbations are calculated based on the individual protagonist policy, but their impact on the team reward is not considered.
ATLA (Zhang et al., 2021a). In single-agent scenarios, the observation adversary is trained using Proximal Policy Optimization (PPO) (Schulman al., 2017), while Multi-Agent PPO (MAPPO) (Yu al., 2022) is employed as the adversary in multi-agent scenarios. For the MAPPO adversary, clean observations are used as input, adversarial observations are selected as actions, and the reward is defined as the negative of the protagonist's ' reward. The training alternates between the protagonist and the MAPPO-based observation adversary.
PAAD (Guo et al., 2025; Sun al., 2022b). The PAAD adversary has the same network architecture as the protag- onist. The adversary operates deterministically the protagonist's observations and aims identify its worst-case performance.
PR (Guo et al., 2024). This robust learning method employs an adversarial policy loss regularization strategy to enhance the robustness of the protagonist models against perturbations. PR formulates the policy loss as L Creguler + pLadvv where μ controls the trade-off between standard and adversarial performance.
PR-REP (Zhou et al., 2024c). μ is repeatedly increased from 0 to 0.1 three times during training.
ERNIE (Bukharin et al., 2023). It promotes Lipschitz continuity in policies concerning state observations and actions through adversarial regularization.
RAP (Vinitsky et al., 2020). It uses a population of adversaries to perturb actions indirectly. To fairly compare with RAP, we have adopted its core idea of promoting adversary diversity by selecting from the population of deterministic adversaries and applying within the PAAD framework.
ROMANCE-p's (Yuan et al., 2023b). This method evolves diverse auxiliary adversarial attackers to introduce policy perturbations during training, thereby enhancing the robustness of multi-agent coordination under uncertainty and po- tential adversarial threats. We have conducted two sets of experiments: ROMANCE-p, using the original ROMANCE policy adversary, and ROMANCE-S, where our SDor module is replaced with ROMANCE adversaries.
H.3. Results analysis
For the case of perturbation-fiee free and random noise, the data show that models trained using FGSM and PAAD adversarial frameworks, while performing well under adversarial conditions, tend to exhibit poor performance in clean or random noise scenarios. In contrast, our proposed method does not lead to significant performance degradation under these conditions and even enhances performance in certain clean scenarios, such as in the CAV environment. This improvement may be attributed to the soft policy employed by the adversarial agents in our framework. These stochastic adversaries encourage the protagonist agent explore broader range of policies, reducing the likelihood of falling into local optima.
For the case of adversarial observations, the NoAdv and RN baselines do not account for adversarial observations during training, resulting in poor performance under adversarial attacks. However, RN considers random noise during train- ing, allowing it to perform relatively well under weaker adversarial attacks such as those generated by ATLA. PR, a method, incorporates an adversarial loss term during training. However, its performance under stronger adversarial attacks, such as PAAD and ATSA, remains suboptimal. PR can occasionally achieve the best perfor- mance under FGSM attacks, such as in the 8m scenario with QMIX-based methods. The experimental results demonstrate that proposed ATSA framework achieves the highest average win rate across various scenarios and adversarial obser- vations when compared to FGSM, ATLA,, PAAD, and RN methods. While ATSA does not achieve the best performance against all adversaries in every scenario, it demonstrates overall robustness. For example, in the 3s 3z scenario during alternate training with QMIX, ATSA'S performance against FGSM as the adversary is not optimal. However, even in this case, the best-pernoniing framework, FGSM, only surpasses ATSA's win rate by 3%, and its performance against other adversaries is not remarkable. A similar pattern is observed in the CAV environment, where ATSA consistently maintains
robust regularization-base
21

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
robust performance. Moreover, ATSA achieves the best average rewards and collision rates under multiple adversarial conditions, underscoring its superior robustness compared the other frameworks. From the above analysis, we have the following conclusions:
NoAdv and RN fail to enhance model robustness under adversarial conditions.
PR, PR-REP, and ERNIE improve robustness under adversarial conditions by optimizing an adversarial regularization term. However, balancing the adversarial loss with the standard loss proves challenging. This often leads either a significant performance drop in clean conditions ineffective adversarial training.
FGSM and PAAD aim to identify the strongest adversary, optimizing the performance of the protagonist agent under such conditions. While this approach enhances robustness under adversarial observations, frequently destabilizes training and compromises performance in clean ones. In extreme cases, models trained with these methods may perform equally poorly on both clean and adversarial observations.
ATLA also seeks to identify the strongest adversary, but the large action space of adversaries makes difficult to find an optimal solution, reducing its effectiveness in improving robustness.
ROMANCE as policy robustness learning method has to constrain adversarial agents, i.c., introducing sparse action attack budget to limit the number of adversarial interventions, order to preserve the stability of a training process. These constrains do not work on our state robustness learning method, because if our robustness learning method also takes these constrains, then only part of agents are possibly robust while others are not.
RAP as another policy robustness learning method does not use these constraints, and it takes a population of adver- saries to deal with overfitting. However, our state robustness learning method also takes this method to deal with overfitting, lots of -i--off-dsttbuution states are generated (so that the training process is not stable) and the memory cost is increased significantly.
ATSA, our proposed framework, introduces stochastic adversaries to address the overfitting issues inherent in adver- sarial training. This approach ensures that models achieve strong performance in both clean and adversarial observa- tions encouraging the protagonist agent to generalize to diverse perturbations.
H.4. Ablation study
== le -30
ATSA(QMIX)
1.0
Win Rate
0.8
0.6
0.4
0.2
0.0
NoAdv
FGSM
PAAD
Figure 4. Ablation study 8m
The Fig. shows that when a approaches 0, it leads to a decline in robustness. In this case, the policy becomes de- terministic, which results in performance similar to that of PAAD-showe showing good results under PAAD attacks but poor performance under FGSM attacks.
H.S. Different perturbation range
To test the robustness of the model against perturbations of different ranges, we select the upper bound of perturbation sizes in the range [0, 0.25) with increments of 0.05. shown in Fig. 5, we compare the model trained by ATSA with those
22

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
FGSM
ATLA
PAAD
ATSA
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
Win rate
0.6
0.6
0.6
0.4
0.4
0.4
0.4
NoADV
0.2
PAAD
0.2
0.2
0.2
ATSA
0.0
0.0
0.0
0.1
0.2
0.0
0.1
0.1
0.2
0.0
0.1
0.2
Perturbation size
Perturbation size
Perturbation size
Perturbation size
(a) VDN-hased models against adversarial perturbations of varying ranges.
FGSM
ATLA
PAAD
ATSA
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
Win
0.4
0.4
0.4
NoADV
0.2
PAAD
0.2
0.2
0.2
ATSA
0.0
0.0
0.0
0.0
0.0
0.1
0.2
0.0
0.1
0.2
0.0
0.1
0.2
0.0
0.1
0.2
Perturbation size
Perturbation size
Perturbation size
Perturbation size
(b) QMIX-based models against adversarial perturbations of varying ranges.
Figure S. Robustness evaluation of VDN-based and QMIX-based models against adversarial perturbations varying ranges. FGSM, ATLA, PAAD, and ATSA are four adversarial attack methods used to evaluate the robustness of models against perturbations. In the legend, AoAdv refers to the baseline model trained without any adversarial training. The models labeled PAAD and ATSA are obtained through adversarial training.
trained by PAAD and NoAdv. We evaluate the model's robustness against perturbations generated using four adversaries focusing on the 3m scenario.
Fig. 5a shows the performance of VDN-based models obtained through training with adversaries. The green solid line represents the model trained using our ATSA method with the VDN framework. For the NoAdv model, its win rate drops to nearly 0 when the perturbation size exceeds 0.05, highlighting its lack of robustness. The only exception is under ATLA attacks, where the performance does not completely degrade due to ATLA's weaker attack capability. Both PAAD and ATSA exhibit stable performance across different perturbation ranges. However, PAAD shows occasional instability, such as at a perturbation size of 0.15, where the performance is significantly better than at 0.05 under the FGSM adversary. This deviation indicates that PAAD-trained models may overfit specific perturbation ranges, leading to unexpected results under varying perturbation sizes.
Fig. 5b depicts the performance of QMIX-based models trained with adversaries. The green dashed line represents the model trained using our ATSA method with the QMIX framework. Under this framework, our method consistently demonstrates superior performance across all attack methods and perturbation ranges. Similar to the observations in Fig. 5a, PAAD-trained models frequently show lower performance under small perturbations than under larger ones. This suggests that PAAD-based training may overfit to specific noise ranges, achieving strong performance within predefined ranges but failing against others. In contrast, our method, which employs stochastic adversaries during adversarial training. successfully counters diverse set of perturbation-geenraate methods and ranges. This diversity effectively mitigates the overfitting problem observed in PAAD-trained models.
In summary, our ATSA method, by leveraging stochastic adversarial training, equips the model with robustness against a broader range of perturbation sizes. Compared to PAAD, our approach resolves the overfitting issue, ensuring stable and strong performance across varying perturbation ranges.
23

---

Robust Multi-Agent Reinforcement Learning with Stochastic Adversary
H.6. Continuous action space
Table 5. Performance of ATSA in continuous control tasks under the MPE environment
MPE
NoAdy
RN
FGSM
ATLA
PAAD
ATSA
AVG Reward
MADDPG
103.13+66..8
110:40770.55
97.88+66..01
96.16+666.7
107.27=36.33
106.67+-68.06
103.58+5512**
FGSM
121.01160.14
697.47=69.62
129.09113.41
115.76:163.58
119.60=55..45
119.70:12.42
123.77=7.32
ATLA
100.81+71.05 113.94-75.32
98.38++69.64
113.13+71.87
116.97=-1.37
115.15+33.21
109.73++.330*
PAAD
130.00-168.30 118.89-68664
127.88116.67
128.78171.11
121.62.667.54
618.38166..6
124.26
ATSA
130.30-172.24 136.36_72.98
120.40167.00 09
130.20-16..37
121.62.16..00
133.64.171.4
128.75-5.87
FACMAC
204.74190.11 193.23-86.40
179.29118.17
191.01189.56
171.82-82.32
169.20.186.00
184.88-12.55
FGSM
138.79+85.003 159.90-88.11
153.43++00.43
151.01+72.73
158.98-81.75
160.81+-8.22
153.82+7.60
ATLA
864.34482.48 170.10-76.52
160.71+80.44
183.54+90.47
175.35-93.00
1.8899++7.558
168.84+8.60
PAAD 752.63+76.65 87..81=87.62
167.271:0.00
153.94115.38 164.14484.18
155.76=79.81
155.861171.99
156.05=5.32
ATSA
166.66++6.119
158.99-71.77
79.480++99442
666666-85.66
153.23++-69998
161.55-+4.76
indicates statistically significant improvement of ATSA over the corresponding method (p < 0.05, Wilcoxon rank-sum test).
To demonstrate the potential applicability of ATSA to continuous control tasks, we conduct experiments based on MAD- DPG (Lowe et al., 2017) and FACMAC (Peng et al., 2021) in predator-prey scenarios where there 3 agents and 1 prey based on Multiagent Particle Environments (MPE) (Lowe et al., 2017). As shown in Table 5, method under the MAD- DPG framework shows some improvement compared to existing baselines such as FGSM, PAAD, and ATLA. However, the performance under the FACMAC framework is not particularly strong under both clean and adversarial conditions. This indicates that our method has limitations in continuous action spaces and needs further improvement.
We suspect that this due to two main reasons: 1) In multi-agent environments, inducing coordinated worst-case behaviors via gradient-basee based attacks challenging. 2) For step-wise adversaries like PAAD and ATSA, difficult to generate compound perturbations that follow the intended direction of policy disruption. In future work, plan to further extend method to more challenging continuous action space environments. This includes improving adversarial optimization techniques for continuous domains and enhancing policy robustness against high-dimensional perturbations.
24


Reviews:
[{"review_id": "gcHNpi3Dqr", "reviewer": "Reviewer_K6AV", "summary": "This paper proposes Adversarial Training with Stochastic Adversary (ATSA) for training Multi-Agent Reinforcement Learning (MARL) models, where the adversary is trained simultaneously with the protagonist agent. ATSA reduces the models' sensitivity to perturbations in observations, while addressing issues with existing adversarial training methods (such as FGSM and PAAD), which tend to overfit to their own perturbations, resulting in models that perform poorly in a no-adversary (clean) setting. This is achieved via a Stochastic Director\n(SDor) that performs policy perturbations, and a SDor-guided generator (STor) that generates observation perturbations given SDor's suggestion. SDor is then trained to both minimize the protagonist's reward and maximize the entropy of its policy, with the latter encouraging exploration. The authors also introduce a SDor-STor loss function to quantify and penalize deviations between SDor-suggested perturbations and those produced by STor, thereby aligning STor's perturbations with SDor's intention. Theoretical proofs show that under factorized maximum-entropy MARL, SDor's soft policy achieves a global optimum, resulting in an optimal observation adversary.\n\nExperimental results using an extensive set of baseline techniques (no adversary, random noise, FGSM, ATLA, PR, and PAAD) show that ATSA is robust in a variety of settings. Overall, ATSA achieves the best average performance among all baseline methods, while existing methods such as FGSM and PAAD perform poorly in clean scenarios or against different adversaries. A number of ablation studies are also performed, demonstrating the contribution of the SDor-STor loss function to ATSA's performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the suggestion. Additional results—including continuous (Table 1), non-deterministic (Table 2) cases, and further details for Q5 (Tables 1 and 3)—are available at the following link: https://anonymous.4open.science/r/icml25-9974-8E33/9974.pdf\n\nQ1: Deterministic and discrete protagonist policy limits its applicability.\n\nWe have now added experiments on continuous action spaces and non-deterministic policies. In continuous action space tasks, ATSA demonstrates a certain level of effectiveness, but there is room for improvement. Please refer to the response to Reviewer 5nJY, Q5 for more details. In the non-deterministic setting (i.e., FOP), our method outperforms baselines. However, our theoretical analysis assumes a deterministic and discrete policy, and extensions to other settings remain future work. We plan to explicitly state this assumption in the introduction and provide further discussion in the future work section.\n\nQ2: Statistical significance tests.\n\nWe have conducted Wilcoxon Signed-Rank Tests to assess the statistical significance of ATSA's performance. In Table 1, ATSA shows significant improvement in 24 out of 36 baseline comparisons. In Table 2, ATSA significantly outperforms baselines in 10 out of 12 cases for Reward and 9 out of 12 for CR.\n\nWe plan to mark significant results (p < 0.05) with an asterisk (*) in the AVG columns of Tables 1 and 2 in the final version.\n\nQ3: Are the protagonist models retrained for each cell in Tables 1 and 2 based on the specified training method and adversary, or is a shared model used across settings?\n\nThe protagonist model in each cell is trained by using the method specified by the row, while the adversary is trained by using the method specified by the column. Importantly, no retraining is performed for each cell, i.e., the adversary is not retrained specifically to target each protagonist. For example, when evaluating PAAD (as the protagonist) against ATSA (as the adversary), the PAAD model is trained independently using its own adversarial training procedure, and the ATSA adversary is also trained independently using its own procedure—not against the PAAD-trained policy specifically. Similarly, the PAAD protagonist is not trained using the ATSA adversary. We will clarify this setup in Section 4.1.2 of the future version.\n\nQ4: Do the confidence intervals in Tables 1 and 2 represent standard deviations, and over how many episodes are they computed?\n\nYes, the values are standard deviations computed over 500 evaluation episodes. \n\nQ5: More justification for the claims that (1) PR struggles to balance adversarial and standard losses, and that (2) ATLA is less effective in large action spaces.\n\n(1) PR formulates the policy loss as $\\mathcal L = \\mathcal L_{regular} + \\mu \\mathcal L_{adv}$, where $\\mu$ controls the trade-off between standard and adversarial performance. \n\nFollowing [1], we set $\\mu=0$ for the first 2M steps and $\\mu=0.1$ thereafter (denoted PR-0.1). To further investigate this trade-off, we have run two additional experiments: PR-0.05, where $\\mu=0.05$ in the final 2M steps and Rep-PR [2], where $\\mu$ is repeatedly increased from 0 to 0.1 three times during training. Results on the 3m (QMIX-based) setting are available at the provided link.\n\nThese results show that small changes in $\\mu$ have notable impacts. While Rep-PR helps on clean inputs, it is less robust under attack. This illustrates the difficulty PR faces in balancing clean and adversarial performance. We plan to include Rep-PR as a benchmark method and report full results in the future version.\n\n(2) ATLA models the adversary as an RL agent that learns to perturb the environment state. This means the adversary's action space is tied to the dimensionality of the state space. As stated in [3]: “If the original state space is high-dimensional, learning a good policy in the adversary’s MDP (i.e., ATLA) may become computationally intractable.”\n\nTo illustrate this, we have conducted an experiment in the MPE environment, where the state dimensionality is relatively small (16). Based on FACMAC, the results of ATLA and PAAD-trained protagonists against PAAD are 175.35 and 155.76.\n\nThese results show that ATLA performs well in low-dimensional settings. However, in our main experimental environments, the state dimension ranges from 30 to 80, which significantly increases the adversary's action space and training complexity. As a result, ATLA becomes less effective. This supports our claim that ATLA struggles with large state spaces, reducing its effectiveness in high-dimensional environments. \n\n[1] Enhancing the robustness of QMIX against state-adversarial attacks, 2024\n\n[2] A Robust Mean-Field Actor-Critic Reinforcement Learning Against Adversarial Perturbations on Agent States, 2024\n\n[3] Who is the strongest enemy? Towards optimal and efficient evasion attacks in deep RL, 2022", "claims_and_evidence": "The paper's claim that ATSA improves robustness is supported by the experimental results, achieving the highest average performance in all the examined scenarios. The claim that existing methods such as FGSM and PAAD overfit to their own perturbations and therefore perform poorly in clean environment and also against other adversarial methods (such as ATSA itself) is also supported by the experimental results. The authors also theoretically prove the optimality of SDor. Ablation studies further justify the use of the SDor-STor loss function and how it leads to higher performance.\n\nThe only claims that are somewhat unclear to me are claims 2 and 4 in Section 4.2, please see the relevant question below.", "methods_and_evaluation_criteria": "The authors compare against multiple benchmark RL environments, MARL methods, and adversarial/clean settings. The experiments are therefore sound/comprehensive and support the authors' claims.", "theoretical_claims": "I did not check theoretical proofs.", "experimental_designs_or_analyses": "The experimental design and setup are appropriate for the problem being studied. I have one clarifying question regarding the configuration of different adversaries and training methods, which I've included below.", "relation_to_broader_scientific_literature": "The paper adds to existing literature by designing a novel method that incorporates information about the policy of protagonist agents, addressing a limitation of prior techniques that do not perform well in scenarios other than their own. This ultimately results in more robust MARL models that perform optimally in a more diverse set of conditions.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "This is a great paper with a good mixture of novelty, theoretical proofs, and experimental results. I particularly appreciate the depth/diversity of the experimental results, and the ablation study at the end. The overall motivation behind ATSA is insightful and the authors provide a sufficient amount of evidence that their proposed method outperforms state-of-the-art methods.\n\nOne limitation of the proposed method is the assumption that the protagonist's policy is deterministic and discrete. However this still covers a large set of problems, and the continuous setting is left for future work.\n\nA slight weakness of the experimental results is that the performance metrics tends to be noisy in some cases, especially for the Connected and Autonomous Vehicles (CAV) environment. This can undermine the statistical significance of the results. However, the fact that ATSA achieves the best average performance across all settings is a strong justification.", "other_comments_or_suggestions": "1. I suggest the authors perform a statistical test on results in the average columns of Tables 1 and 2, to determine rows where the lower average performance is statistically significant. This could strengthen the paper's claim that ATSA achieves a better \"across-the-board\" performance.\n2. It would also be helpful to clearly state in the introduction that the approach assumes a discrete policy.", "questions_for_authors": "1. Could you elaborate on how the experiments were conducted for Tables 1 and 2, particularly when pitching one adversary algorithm against a different training method for the protagonist agent? Are models retrained for each cell in the tables (i.e., the protagonist's policy in each cell is trained using the training method specified by the row, but in an environment where the adversary specified by the column is present)?\n2. In Tables 1 and 2, are the confidence intervals standard deviations? How many episodes were they computed over?\n3. Could you provide more details/justification regarding the following claims in Section 4.2: \"PR faces challenges in balancing adversarial and standard losses\" and \"ATLA struggles with large action spaces, reducing its effectiveness\". For the latter, why is ATLA (and not other methods) specifically called out for struggling with large action spaces?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "65YDmsSloS", "reviewer": "Reviewer_9h5o", "summary": "This paper proposes Adversarial Training with Stochastic Adversary (ATSA) to fortify the robustness of models trained by multi-agent reinforcement learning. It addresses the overfitting problem of existing methods by training the proposed adversary online alongside the protagonist agent. ATSA implements an SDor-STor structure that performs policy perturbation and uses a loss function that leverages the protagonist agent's policy information when training the adversary policies.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We have added additional experiments on the MPE environment (Table 1), the QTRAN (Table 4) and non-deterministic policy (Table 2), ERNIE (Table 5), and results for Q5 (Table 6). Please refer to https://anonymous.4open.science/r/icml25-9974-8E33/9974.pdf\n\nQ1: MPE and QTRAN.\n\nAs suggested, we have extended our evaluation to include additional benchmarks. The analysis of the MPE experimental results can be found in our response to Reviewer 5nJY, Q5. QTRAN achieves better performance than other methods in the 8m scenario.\n\nQ2: Essential References Not Discussed.\n\nIn the final version, we plan to include a discussion of ERNIE in Section 1 and add it as a benchmark method in Section 4. We have conducted a preliminary experiment with ERNIE under the 3m scenario (VDN-based). As seen, although ERNIE performs well in clean and random noise settings, ATSA outperforms ERNIE in average win rate under different types of perturbations. \n\nQ3: Time complexity and scalability\n\nThe additional time complexity introduced by adversarial training is indeed inevitable. In the 8m scenario, training VDN with ATSA for 4M steps takes approximately 14 hrs 9 mins, while training with PAAD takes around 13 hrs and 38 mins. This indicates that ATSA incurs a comparable training cost to other adversarial training methods such as PAAD.\n\nRegarding scalability, it should be noted that our method is designed within the CTDE framework, which has inherent scalability limitations. We suspect that challenges remain in both the theoretical convergence of entropy-based Mean-Field RL to the Nash Q-value and in the practical implementation of such methods, which require further investigation.\n\nQ4: Typo\n\nWe will correct it in the final version.\n\nQ5: Is it necessary for every agent to have a dedicated adversarial policy?\n\nYes, it is necessary for each agent to have its own adversarial policy. To test this, we have trained ATSA-based (QMIX) protagonists in the 3m and 8m environments with only two adversarial policies that rotate among different agent combinations at each timestep. We have compared these with settings where all agents are adversarial (i.e., fully coordinated). The results show that using only two adversarial agents in rotation (i.e., ATSA-2) does not consistently improve the robustness of the protagonist.\n\nQ6: How does a stochastic adversary promote stable and generalizable training, and are its perturbations less extreme than those from other methods?\n\nOur method introduces an entropy term with a temperature coefficient $\\alpha$ to encourage diverse adversarial behaviors. Early in training, a higher $\\alpha$ promotes broader exploration, preventing overly disruptive perturbations and stabilizing learning. As $\\alpha$ decays, the adversary focuses more on reward optimization. This controlled exploration leads to less extreme, more varied perturbations, reducing overfitting and training collapse—making our method more stable than others.\n\nHence yes, compared to other approaches, the perturbations are indeed less extreme.\n\nQ7: VDN+ATLA performance and large standard deviations.\n\nATLA struggles to learn effective adversarial policies in high-dimensional state spaces, often behaving like random noise. This results in weak protagonist training and degraded performance, even without attacks. The large standard deviations in Table 2 stem from the CAV reward design, where severe penalties for collisions cause high return variance, especially under unstable or adversarial conditions.\n\nQ8: How practical is assuming a deterministic discrete policy in StarCraft, and what if the policy is stochastic?\n\nThis setting is widely adopted in the CTDE framework, where many classic multi-agent methods—such as VDN, QMIX, and QTRAN—are also based on discrete action spaces with deterministic policies. Moreover, these methods are commonly evaluated on benchmarks like SMAC/StarCraft, which naturally align with this assumption.\n\nTo further explore the applicability in stochastic policy settings, we have also conducted experiments under the non-deterministic policy (i.e., FOP). The preliminary results indicate that our method is also applicable to non-deterministic policies.\n\nQ9: SDor architectures and parameters.\n\nThe actor and critic networks of SDor are composed of two MLP layers with a GRU (hidden size 64) inserted between two layers. RMSprop is used to optimize all parameters.\n\nQ10: What performance trade-offs exist between non-adversarial settings and adversarial training when using ATSA?\n\nIn our eight experimental settings, ATSA does not achieve the best performance in three cases under the NoAdv condition. However, it outperforms other methods in most settings and achieves the highest average performance across both clean and adversarial conditions.", "claims_and_evidence": "The 1st claim is the mathematical proof proving the SDor’s soft policy converges to a global optimum according to factorized maximum-entropy MARL and leads to the optimal adversary. It is generally supported by the theoretical proof. \n\nThe 2nd claim is that the proposed ATSA demonstrates robustness against diverse perturbations of observation while maintaining outstanding performance in environments without perturbation. This is generally supported by the experiments whereas several questions remain, please see below.", "methods_and_evaluation_criteria": "The proposed method is reasonable considering the adversarial training in MARL. \n\nThe evaluation is conducted on 2 common benchmarks for MARL the StarCraft and the connected autonomous vehicles, 2 MARL methods (value decomposition network and Q-MIXing network), and 6 baseline agent training methods.", "theoretical_claims": "Read through, but not carefully verify them", "experimental_designs_or_analyses": "The experimental design and soundness will be better if more benchmarks are included such as Multi-Agent Particle Environment (MPE). The MARL methods used can be more powerful such as QTRAN.", "relation_to_broader_scientific_literature": "This paper related to adversarial robustness and multi-agent reinforcement learning.", "essential_references_not_discussed": "This is reference is not discussed in the paper. \n\n**Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms. NeurIPS 2023**", "other_strengths_and_weaknesses": "One of the weakness is the time complexity and the issue of scalability of the proposed method. The introduction of adversarial agents will lead to higher time complexity and limited scalability.", "other_comments_or_suggestions": "There is a typo in Table 1's \"NoAdv\" column - it should be labeled \"WR\" instead. \n\nIs it necessary for each agent to have an adversarial policy? For example, in a system with 3 agents, could we use only 2 adversarial agent policies that rotate between different combinations of agents? Would such an approach be effective?", "questions_for_authors": "How does the stochastic adversary specifically help prevent destabilization of the protagonist agent's training and overfitting to extreme adversarial perturbations? Are the perturbations from the stochastic adversary less extreme compared to other approaches? \n\nIn Table 2, under the VDN row and ATLA column, why is the performance (61.10) significantly lower than RN and PR when using the same agent training method and ATLA adversarial attack? Additionally, what explains the large standard deviations in Table 2?\n\nLine 221 states that the considered case involves a deterministic and discrete protagonist agent policy. Given evaluation tasks like StarCraft, how practical is this setting? What are the implications when the policy is non-deterministic?\n\nWhat specific deep neural network architectures and parameters are used to learn the soft Q-functions and soft policies for SDor?\n\nWhat performance trade-offs exist between non-adversarial settings and adversarial training when using ATSA?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CYapk0UxDR", "reviewer": "Reviewer_5nJY", "summary": "The paper proposes replacing workst-case adversary to stochastic adversary to improve robustness in multi-agent reinforcement learning (MARL). The proposed adversary model consists of a director and an actor where the former predicts a direction of manipulation and the latter translates this action into a manipulated observation. The paper proves the convergence and optimality of the training dynamics. It also conducts experiments on StarCraft II and autonomous driving datasets that the proposed model effectively address the issues of adversarial overfitting and director-actor misalignment, which are commonly seen in the adversarial MARL literature.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your question. Based on your suggestion, we have provided additional experimental results, including ablation studies on $\\alpha$ (Fig. 1) and continuous control tasks (Table 1), available at the following link: https://anonymous.4open.science/r/icml25-9974-8E33/9974.pdf\n\nQ1: Some unclear statement\n\nThe sentence in line 431: You are correct. We have corrected it.\n\nThe sentence on line 1096: We define instability as the variation in performance under different perturbation levels. Although ATSA shows a similar trend in this specific figure, the overall results across all eight evaluation plots indicate that ATSA is more stable than PAAD, with consistently smaller performance fluctuations under varying perturbation levels.\n\nThe last sentence of section 2.2: Yes, the correct notation should be $\\hat o \\sim v(\\cdot|\\tau)$, where $v$ is the joint policy of the adversary.\n\n\nQ2: Training parameters\n\nWe run 4,000,000 environment steps across all environments and methods. For SDor, both actor and critic use a learning rate of 0.0005. The target networks are updated every 200 episodes. In SMAC, we use the default episode horizon provided by each environment. For the CAV environment, the horizon is set to 100 timesteps.\n\nQ3: Some ablation tests on $\\alpha$.\n\nWe have added ablation experiments on the temperature parameter $\\alpha$. The results show that when $\\alpha$ approaches 0, it leads to a decline in robustness. In this case, the policy becomes deterministic, which results in performance similar to that of PAAD—showing good results under PAAD attacks but poor performance under FGSM attacks.\n\nQ4: Essential References\n\nWe plan to add relevant references on adversarial learning and the use of randomization techniques to balance robustness and nominal performance.\n\nQ5: The applicability of ATSA to continuous control (e.g., MADDPG) remains undiscussed.\n\nWe have conducted experiments based on MADDPG and FACMAC in the MPE environment to demonstrate the potential applicability of ATSA to continuous control tasks. Our method under the MADDPG framework shows some improvement compared to existing baselines such as FGSM, PAAD, and ATLA. However, the performance under the FACMAC framework is not particularly strong under both clean and adversarial conditions. This indicates that our method has limitations in continuous action spaces and needs further improvement.\n\nWe suspect that this is due to two main reasons: 1) In multi-agent environments, inducing coordinated worst-case behaviors via gradient-based attacks is challenging. 2) For step-wise adversaries like PAAD and ATSA, it is difficult to generate compound perturbations that follow the intended direction of policy disruption. We plan to include this discussion in Section 4 of the future version.", "claims_and_evidence": "Most claims are supported by clear and convincing evidence, with one perhaps unclear statement:\n- The sentence in line 431 starting with \"However...\" says the proposed ATSA method achieves the **best** attack performance, but Fig. 3(a) PAAD(VDN) seems to imply the ATSA attacker cannot degrade the agent performance as much as the PAAD attacker, right?", "methods_and_evaluation_criteria": "Both StarCraft II and Autonomous Driving problems are valid and common problems in MARL. The evaluation metrics are the cumulative rewards and win rate (resp. crash rate) for StarCraft II (resp. Autonomous Driving). The metrics are reasonable.\n\nBenchmarks are a list of other adversarially trained MARL models, as well as the case where the adversay is absent. These benchmarks are also reasonable.", "theoretical_claims": "I checked all proofs and theoretical claims and not spotted any issues, provided that the cited results are correct.", "experimental_designs_or_analyses": "I checked the experiments on metric comparisons, ablation tests, attacker performance, and the perturbation effect. The experiment design is valid and the results support their claim on the proposed advantages. \n\n- Some details on the training parameters is missing, such as epochs, learning rate, horizon, etc., which may influence the reproducibility of the results.\n\n- Some ablation tests on $\\alpha$ could make the experiments more complete.", "relation_to_broader_scientific_literature": "This work belongs to the multi-agent reinforcement learning and adversarial learning literature. The key finding advances the state-of-the-art performance in the robust MARL area.", "essential_references_not_discussed": "No missing essential references recognized.\n\nHowever, the Related Work section only contains discussions with different robust reinforcement learning algorithms. Perhaps some discussions on adversarial learning would help understand the context more. The idea of using randomization to simultaneously detering attacks and maintaining naive-case performance should exist somewhere in the literature.", "other_strengths_and_weaknesses": "This paper is strong both in originality and technical contents. It innovatively proposed a training approach with stochastic adversaries. This method is itself novel and original and proved effective in ensuring high RL model performance under different types of adversaries. Both theory and experiments are rich enough to support the claimed superiority of the proposed model. \n\nThe weakness is in the discrete and finite environment setting. The applicability of ATSA to continuous control (e.g., MADDPG) remains undiscussed.", "other_comments_or_suggestions": "The last sentence of section 2.2: The notations are a little confusing. Do the authors mean $\\hat{o}\\sim v(\\cdot|\\tau)$ instead of  $\\tilde{o}\\sim v(\\cdot|\\tau)$ as $v$ is $\\mathcal{M}$-dimensional (line 120) and $\\tilde{o}$ is $\\mathcal{N}$-dimensional (line 128). Also, is it the \"joint policy of the protagonist agent\" or the joint policy of the adversary?", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5a4UkueQ0R", "reviewer": "Reviewer_UH2H", "summary": "This paper proposes a soft-policy-based MARL observation adversary consisting of a director module and an attack generator. The convergence guarantee of the director module is provided. Experiments on SMAC and CAV benchmarks demonstrate the effectiveness of the proposed method.\n\n## update after rebuttal\n\nMy concerns are largely addressed. I raise my score from 2 to 3. Thank you to the authors for their response and the additional experiments. It is good to see the improvements made to the manuscript. Please make sure the improvements are integrated in the next version.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful suggestion. We have added RAP and ROMANCE as benchmark methods in the 8m VDN-based setting, as shown in Table 7 (https://anonymous.4open.science/r/icml25-9974-8E33/9974.pdf).\n\nQ1: Essential References Not Discussed\n\nWe plan to include a discussion of the references in the introduction and incorporate ROMANCE as a baseline method in our experiments to enable a more comprehensive comparison. \n\nQ2: Incremental algorithmic contribution\n\nAlthough the core idea of training robust agents via adversaries has been explored in prior work, there are still some open and challenging problems, such as overfitting, which has been highlighted in recent papers and is a central focus of our study. Certainly, the concept of stochastic adversaries is not new, as it has been studied in the context of policy robustness in several works, including ROMANCE and RAP. However, there are significant differences between policy robustness and state robustness in terms of both formulation and motivation. \n\nFormally, policy robustness involves perturbing agents' actions to evaluate their robustness, whereas state robustness involves perturbing the agent's input states without directly altering its actions.\n\nROMANCE, as a policy robustness method, requires constraints on adversarial agents, i.e., introducing a sparse action attack budget to limit the number of adversarial interventions, in order to preserve training stability. These constraints are not applicable to our state robustness training method. If our method takes these constraints, only some agents might become robust, while others would remain sensitive to state perturbation. Our experiments also show this phenomenon. \n\nRAP, as another policy robustness method, does not use these constraints, and it takes a population of adversaries to deal with overfitting. However, if our state robustness learning method adopts a similar approach to address overfitting, it generates numerous out-of-distribution states (resulting in destabilization of the training process) and significantly increases memory costs.\n\nTherefore, although stochastic adversaries have been used for policy robustness, they do not translate well to the setting of state robustness. In contrast, our paper provides a feasible way. We’ll add this discussion in the future version.\n\nQ3: Underdeveloped justification for stochastic adversaries.\n\nWe have added both RAP and ROMANCE as baselines in our experiments for comparison.\n\nTo fairly compare with RAP, we have adopted its core idea of promoting adversary diversity by selecting from the population of deterministic adversaries and applied it within the PAAD framework. Experimental results show that our method achieves better performance across diverse adversarial scenarios, while being more memory-efficient by noting that RAP requires training and maintaining multiple adversaries in parallel.\n\nFor ROMANCE, we have conducted two sets of experiments: ROMANCE-p, using the original ROMANCE policy adversary, and ROMANCE-s, where our SDor module is replaced with ROMANCE adversaries. Results indicate that ROMANCE does not effectively help the protagonist learn state-robust policies.\n\nQ4: Overly complex formalisms with limited clarity\n\nIn the definitions, we have used superscripts $p$ and $a$ to distinguish between the states, actions, and policies of the protagonist and the adversary, respectively. We agree that Figure 1 is overly large and under-explained, and we plan to replace it with the linked Figure 2 in the final version.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes, I checked Theorem 3.4 and 3.5.", "experimental_designs_or_analyses": "Yes. I checked Section 4. Experiments.", "relation_to_broader_scientific_literature": "The key contributions extend and refine ideas from MARL, robustness in machine learning, and open-environment learning. By addressing the challenge of robustness in MARL, this paper advances these fields and provides a framework for developing more robustly collaborative AI systems.", "essential_references_not_discussed": "The purterbation of observations in MARL is an open-MARL problem, where the key factors of the learning process change. A survey of this problem [1] should be discussed. Also, the idea of learning an adversary and proving its optimality is similar with [2], which should be discussed and compared as a baseline.\n\n[1] A Survey of Progress on Cooperative Multi-agent Reinforcement Learning in Open Environment. Lei Yuan, et al. 2023.\n\n[2] Robust Multi-agent Coordination via Evolutionary Generation of Auxiliary Adversarial Attackers. Lei Yuan, Ziqian Zhang, et al. 2023.", "other_strengths_and_weaknesses": "## Strengths  \n1. **Rigorous Theoretical Foundation:** The paper provides a thorough theoretical analysis of adversary optimality, including formal proofs for convergence on the adversary's optimal policy. This strengthens the methodological grounding of the approach.  \n2. **Comprehensive Empirical Evaluation:** The experiments span diverse multi-agent benchmarks and demonstrate consistent improvements over strong baselines.\n\n## Weaknesses\n1. **Incremental Algorithmic Contribution:** While the integration of stochastic adversaries is technically sound, the core idea of training robust agents via adversaries is well-explored in prior work. The novelty lies primarily in the specific instantiation rather than a conceptual leap.  \n2. **Underdeveloped Justification for Stochasticity:** The claim that stochastic adversaries mitigate overfitting is not sufficiently substantiated. Population-based methods like RAP [1] and ROMANCE [2] explicitly address this via diverse adversary ensembles, whereas the paper's stochastic policy approach lacks a direct comparison or analysis to validate this advantage.  \n3. **Overly Complex Formalisms with Limited Clarity:** The OD-POMDP/PD-POMDP formulations introduce heavy notation without commensurate payoff in insight. Figure 1 occupies too much space but is not clearly explained.\n\n[1] Robust Reinforcement Learning Using Adversarial Populations. Eugene Vinitsky, et al. 2020.\n\n[2] Robust Multi-agent Coordination via Evolutionary Generation of Auxiliary Adversarial Attackers. Lei Yuan, Ziqian Zhang, et al. 2023.", "other_comments_or_suggestions": "None.", "questions_for_authors": "See Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes the use of stochastic adversary to improve robustness of multi-agent reinforcement learning (MARL).  Both theoretical and experimental contributions are good.  The reviews are generally positive. The reviewer of the original low score (2) was happy with the rebuttal and expressed willingness to raise their score.
Decision: Accept (poster)