Evaluating Robustness of Cooperative MARL
Authors: Nhan Pham
Lam M. Nguyen
Jie Chen
Thanh Lam Hoang
Subhro Das
Tsui-Wei Weng
Venue: ICLR 2022 Submitted (2022)
Keywords: cooperative multi-agent reinforcement learning
adversarial attack
continuous action

Abstract: In recent years, a proliferation of methods were developed for multi-agent reinforcement learning (MARL). In this paper, we focus on evaluating the robustness of MARL agents in continuous control tasks. In particular, we propose the first model-based approach to perform adversarial attacks for cooperative MARL. We design effective attacks to degrade the MARL agent's performance by adversarially perturbing the states of agent(s) and solving an optimization problem. In addition, we also developed several strategies to select the most vulnerable agents that help to further decrease the team reward of MARL. Extensive numerical experiments on multi-agent Mujoco tasks verify the effectiveness of our proposed approach.

Under review as a conference paper at ICLR 2022
EVALUATING ROBUSTNESS OF COOPERATIVE MARL
Anonymous authors
Paper under double-blind review
ABSTRACT
In recent years, a proliferation of methods were developed for multi-agent rein- forcement learning (MARL). In this paper, we focus on evaluating the robustness of MARL agents in continuous control tasks. In particular, we propose the first model-based approach to perform adversarial attacks for continuous MARL. The attack aims at adversarially perturbing the states of agent(s) to misguide them to take damaging actions that lower team rewards. A deep neural network is trained to represent the dynamics of the environment. We then solve an optimization prob- lem with the learned dynamics model to yield small perturbations. In addition, we discuss several strategies to optimally select adversary agents for the attack. Nu- merical experiments on multi-agent Mujoco tasks verify the effectiveness of our proposed approach.
INTRODUCTION
Deep neural networks are known to be vulnerable to adversarial examples, where a small and often imperceptible adversarial perturbation can easily fool the state-of-the art deep neural network clas- sifiers (Szegedy et al.) 2013, Nguyen et al.] 2015 Goodfellow et al.) 2014] Papernot et al.) 2016). Since then, a wide variety of deep learning tasks have been shown to also be vulnerable to adversar- ial attacks, ranging from various computer vision tasks to natural language processing tasks (Jia &
Liang 2017
Zhang et al. 2020b Jin et al. 2020 Alzantot et al. 2018).
Perhaps unsurprisingly, deep reinforcement learning (DRL) agents also vulnerable to adversarial attacks, as first shown in Huang et al. (2017) for atari games DRL agents. In Huang et al, (2017), the authors study the effectiveness of adversarial examples a policy network trained Atari games under the situation where the attacker has access to the neural network of the target policy. In Lin attack by attacking trained agents on
al. (2017), the authors further investigate a strategically-tinng Atari games at a subset of the time step. Meanwhile, Kos & Song (2017) use the fast gradient sign method (FGSM) to generate adversarial perturbation on the A3C algorithm (Mnih al. (016)) and explore training with random noise and FGSM perturbation to improve resilience against adversarial examples. While the above research endeavors focused on actions that take discrete values, another line of research tackles a more challenging problem on DRL with continuous action spaces (Weng al., 2019: Gleave et al., 2019). Specifically, |Weng et al. (2099) consider a two-step algorithm which determines adversarial perturbation be closer to target state using learnt dynamics model, and Gleave et al. (211 propose a physically realistic thread model and demonstrates the existence of adversarial policies in zero-sum simulated robotics games.
While most of the existing DRL attack algorithms focus on the single DRL agent setting"] in this work we propose to study the vulnerability of multi-agent DRL, which has been widely applied in many safety-criticn real-world applications including swarm robotics (Dudek et al. 1993). electric- ity distribution, and traffic control (OroojlooyJadde Jadid & Hajinezhad 2019). In particular, the collaborative multi-agen reinforcement learning (c-MARL) setting, where group of agents is trained generate joint actions to maximize certain (team) reward function. We note that c-MARL is a more challenging yet interesting setting than the single DRL agent setting, as the interactions between agents commands consideration of additional layers of complications.
focus
Contribution. Our contribution can be summarized as follows:
We are aware of only one recent work (Lin al. 2020) attacking c-MARI. agents with discrete action space, rather than continuous action space considered our work.

---

Under review as a conference paper at ICLR 2022
We propose new adversarial attack framework on MARL with continuous action space, where we name it cMBA (model-based attack on c-MARL). The attack comprises two steps: learning representation of the environment dynamics and then determining proper observation perturbations based on the learned dynamics model.
We formulate the process selecting an victim agent as a mixed-integer programming problem and propose an approximate formulation that can be efficiently solved by first- order method.
We evaluate our attack framework with two baselines and four multi-agent MuJoCo tasks (Peng et al. 2020), with number of agents ranging from 2 to 6, to verify the effectiveness our approach. Our model-based attack consistently outperforms the two baselines in all tested environments.
Paper outline. Section discusses related works in adversarial attacks for MARL along with the problem settings. We describe the main attack framework cMBA in Section including the training of the dynamics model and solving the sub-optimization problem. In Section 3.3 we introduce a mixed-integer program to find the optimal set of victim agents and propose an approximate formu- lation that can be efficiently solved using a first-order optimization method. Section presents the evaluation of our approach on several multi-agent MuJoCo environments. Therein, we also study the performance of different variants of our model-based attack as discussed in Section 3.3
2 RELATED WORK
Adversarial attacks for DRL agents. In the c-MARL setting, although there exists significant literature focusing on the adversarial training of MARL agents (Phan et al. 20200 2021 Zhang et al.] 2020a). yet very few of them addresses the scenario of adversarial attacks in c-MARL. Lin et al. 2220 propose a two-step attack on a single agent in a c-MARL environment, where they extend existing methods to generate adversarial examples guided by learned adversarial policy. The attack in Lin et al. (2020) is evaluated under the StarCraft Multi-Agent Challenge (SMAC) environment (Samvelyan et al. 2019) where the action spaces are discrete. To the best of our knowledge, there has not been work considering adversarial attacks on the c-MARL setting with continuous action spaces. The closest approach ours is the enchanting attack Lin al.] (2017) and the model- based attack in Weng et al. (2099; however, these two works focused on single DRL agent setting. The enchanting attack in Lin et al. (2017) uses a video prediction model to predict the future state and finds perturbation that minimizes a distance function between the predicted future state and target state. Similarly, Wenn et al. (2019) consider two formulations for observation and action perturbation that incorporates a learned dynamic model for future state prediction. In addition to the model- based approach, in this paper, we consider the selection victim agents, which is unique to the multi-agen agent DRL setting.
Training c-MARL agents. One simple approach to train c-MARL agents is to let each agent learn their own policy independently as in Independent Q-Learning (IQL) (Tan 1993 Tampuu et al.) 2017). However, this training strategy does capture the interaction between agents. Altema- tively, one can follow a common paradigm called centralized training with decentralized execution
(CTDE) (Oliehock al. 2008 Olichock & Amato 2016;; Kraemer & Banerjee, 2016). Gupta et al. (017) extend DQN (Mmih et al. 2015). TRPO (Schulman et al. 2015). and DDPG (Liliicrap et al.
2015) to c-MARL, where a parameter sharing neural network policy is trained in centralized fash- ion but still allows decentralized behavior across agents. Similar to this, COMA (Foerster et al.) 2018 is an actor-critic method that uses a centralized critic to estimate counterfactual advantage for decentralized policies. While COMA uses single critic for all agents, MADDPG (Lowe et al.) 2017) learns a centralized critic for each agent and evaluates on continuous control. Instead of hav- ing fully centralized critic, DOP (Wang al.) 2020b) and FACMAC (Peng et al., (0200) apply the value decomposition idea (Sunehag et al. 2017 Rashid et al. 2018 2020 Son et al.] 21199 Wang et al. 2020a) into multi-agent actor-critic by having centralized but factorized critic.
2

---

Under review as a conference paper at ICLR 2022
3
CMBA: MODEL-BASED ATTACK FOR C-MARL
3.1 PROBLEM SETTING
We consider multi-agent tasks with continuous action spaces modeled as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) (Oliehoek & Amato, 2016). A Dec-POMDP has finite set of agents N {1.... _n} associated with set of states S describing global states, set of continuous actions A,, and set of individual partial observation O, for each agent i N. Given the current observation of E O, the action oj A, is selected by parametrized policy Ai The next state agent i moves to is determined by the state transition function P. : S x A + S. After that, agent receives a reward r calculated from a reward function R4 : S x A. R and observes local observation 0/41 O, correlated with the global state s*+1 S. In addition, Dec-POMDP is associated with initial state distribution Po and a discount factor y. In this setting, each agent tries to find the policy that maximizes its own total expected
To: O
return R = E >²r² where T is the sampling horizon.
3.2 PROBLEM FORMULATION
Our goal is to generate adversarial perturbations imposed to the victim agents' input (state) order to deteriorate the total team reward. The added noise encourages the victim agents' observation to be close to a desired adversarial state corresponding to low reward. To avoid sampling from the environment,
use pre-trained model that learns the dynamics of the environment to predict the next state from the perturbed observation and current action, then find the suitable noise that minimizes the distance between the predicted next state and a predefined target state. This process can be formulated as an optimization problem as follows.
Formally, consider a multi-agent setting with [W] agents, each agent M observes state st locally and takes action a,. Let = (s).... ..)) E S be the global state at time step t which concatenated from local states sf for each agent I.... n.. We also denote the joint action a, (a). , a.") concatenated from each agent's action at. Let L. E N be the set of victim agents time step i, i.e. the set of agents that can be attacked. Let fa function that approximates the dynamics of the environment, where A is the set of concatenated actions, one from each A,, We assume that a pre-trained MARL policy πT.) is given. Let Starget be the target state which can lead to poor performance to the agent. We denote as upper bound on budget constraint w.r. some p-norm I1-1lp- The state perturbation As = (As', ......))) (we suppress the dependence on As to avoid overloading the notation) to s; is the solution to the following problem
A - S be a parametrized
min As") s.t.
d(Br+1, 1, Starget)
A==(As)
$2+1 fa(s af = (s; + As") As* 0, Vi Le Lg 8t + As us
(1)
|| | Sc
where 0 is zero vector, and the observation space is constrained within the (vectorized) intervals between ls and us.
Let us first provide some insights for the formulation ( For each agent, using the trained policy n we can compute the corresponding action of given its (possibly perturbed) local state si or s; + As' From the concatenated state-action pair (se, 0z), we can predict the next state 8 via the learned dynamics model fa. Then by minimizing the distance between the predicted next state and the target state under the budget constraint, next transition. We also note that the last equality constraint indicates that we only allow perturbing agents within the victim set L.
are forcing the agent to move closer to damaging state in the
Problem I can be efficiently solved by caatana--rddden--baae based methods. Firstly, by substituting the definition of a into 81+1, is equívalent
min s.t. I I Cs
d(fa(se, T(s + x)),, starget)
3

---

Under review as a conference paper at ICLR 2022
where C/ {ar (I1,.... .In) |>>1| S is S us St and 2; for & Li). If
we choose the distance function as d(a. b) = ||a -bll" we can use the projected gradient descent (PGD) algorithm (Nesterov 2003) to solve The PGD iteration update yk at iteration can be described as
bll?
Mk+1 = projey lyk v/ vdd(ff(ss (s; + y)), Starget)]
(3)
where projep() (-) the projection the convex C and 7) is the learning rate. The projection is simple to calculate since C₁ is the intersection of a unit ball in the p-norm and a box.
The whole attack process can be summarized in Algorithm !
Algorithm cMBA algorithm at timestep /
1: Initialization:
Given St. Starget, n, fd. Li: initialize Asy; choose learning rate > do
2: For k == 0,...
3: Compute a₃ ... a (
(sf + Ast) if L. otherwise.
aj
(s{))
4: Compute 8441 Sa(st, at)
5: Update As as
Ask-1 =proje, [Ask 11 (,d($t+1,starpee
6: End For
Learning dynamics model. The most important factor that affects the solution of 2 is the quality of the learned dynamics model fa- the dynamics is known, we can solve 2 exactly so that the solution is indeed the optimal perturbation. However, it is hardly the in practice because we often approximate the dynamics using function approximators such as neural networks. The parameter for is the solution of the following optimization problem
min Σ Ifals w)
Snert
(4)
RED
where 'D is a collection of state transition {( ne where Snezt the actual state that the environment transitions to after taking action determined by given policy. In particular, we separately collect transitions using the pre-trained policy æ and a random policy to obtain Derain and D Drondom- Then the dataset D is built as D Dtrain U Drandom- Problem (4) is standard supervised learning problem and can be solved using gradient-bard based learning. We describe the full process of training the dynamics model in Algorithm 2 where the Gradient BasedUpdate step any
update.
Algorithm Training dynamics model
1: Initialization:
Given pre-trained policy wer and random policy Tydi initialize wo
2: Form D : Derain U Drandom by collecting a set of transitions Dtrasn and D Drandom using policy Wer and wyd. respectively.
3: For h: := 0. do
Wk++ Graddien
4: End For
3.3
VICTIM AGENT SELECTION
In this subsection, we discuss different strategies to select victim agents when performing adversarial attacks in the MARL setting. Suppose that we would like to perform an attack on any agent but the budget the victim agents is na In other words, attacker only choose subset of

---

Under review as a conference paper at ICLR 2022
agents to attack during each time step, and the goal of an attacker is to choose the most vulnerable agents so that the total reward is utmost affected. We note that this scenario is unique in the setting of multi-agen DRL setting, as in the single DRL agent setting the time. To start with, we first formulate a mixed-integer program to perform the attack on set of optimally victim agents as below:
only attack the same agent all
min
d 8 Starget)
S.C.
81+1 = [a(St, az)
a = (5 + (As, w))
fg 8, As S us
(5)
|| | S
We (0,1)
Σ, w. Ra
where we introduce a new binary variable to select the appropriate agents' input to perturb and na is the total number of victim agents we can attack.
Due to the existence of the new binary variable, problem is much harder to solve than before. We instead solve proxy of 5 as follows
minAs,0 s.t.
d( Starget)
st+1 = fa(s1, at)
a, 5(5 + (As, (s.,, 0)))
(6)
Es S $1 + As I us
| p Se
0 S (st,0) 0) 1
where (-) is a function parametrized by that takes current state as input and outputs the weight to distribute the noise to each agent. Suppose we represent (..88 by a neural network, can rewrite the formulation (6) as
minl
d(fa(s,, + (As, (ss,, 0)))), Starget)
s.t.
As E C.
because the last constraint in (6) can be enforced by using softmax activation in the neural network (), As result, (7) can be efficiently solved by using PGD. We present the pseudo-code of
the attack in Algorithm 3 After K steps of PGD update, we define the index in-j the j-th
largest value within W (8t; 0k) E R", i.e. we have Win. (8,, 0x) Z Witn-n (Be, 0k) 2 Z WeO (sc, 0g). Let I, be the index set of top-j largest outputs of the W (st, 0x) network. The final perturbation returned by our victim agent selection strategy will be As ((As)),... (As)a) where (As)² = 0 if i & In, and (As) (Ask)² if In..
Algorithm 3 cMBA with victim agent selection at timestep
Initialization:
Given St. Starget, fa. na; initialize Aso: choose learning rate > 0 > 0.
2: For == ..... K- do
3: Compute at x(3 + (As, ((s,, 0)))
4: Compute 8141 = fa(ss, a1)
5: Update As: Ask+1 = projey [Ask v/ A, Starget)
6: Update 0: 0x+1 = 0 - AVo (8 Starget)
7: End For
8:
Compute In. = {i(m). ... i(a { such that
Winm (se, 0k) 2 Witn-n (in-,;($p, 0x) ... W 0k)
9: Return As= ((As))
(As) such that
(As) (Ask)* if In. otherwise
Remark 3.1 For this attack, we assume each agent has access to the other agent's observation to form the concatenated state 8t-

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS
We perform the attack on various multi-agent MuloCo (MA-MuJoCo) environments including Walker2d (2x3), HalfCheetah (2x3), HalfCheetah (6x1), and Ant (4x2), The pair envi name (config) indicates the name of MuJoCo environment along with the agent partition, where a configuration of 2x3 means there are in total 2 agents and each agent has 3 actions. Note that we cannot directly apply methods such as the fast gradient sign method (FGSM) or JSMA (an attack using saliency map) as in Lin et al.] (2020), since those methods require a "target" action which is not available in continuous action space. Therefore, we consider two baselines: Uni form and Gaussian baselines where the perturbation follows either Uniform distribution U(-e, e) or Normal distribution ((, e).
Variants of model-based attack. We consider the following variants of our model-based attack:
Model-based attack on fixed agents: perform A using a fixed set of victim agents
L.
Best model-based attack on fixed agents: among the model-based attack on fixed agents, pick the subset that achieves the best performance.
Model-based attack random agents: perform Algorithm with Lt, which is sampled uniformly from NN such that G2 = no (ng is the total victim agents).
Model-basee attack with learned victim selection: perform Algorithm 3 to select vulnerable agents and perform the attack on them.
Greedy victim selection: perform a sweep over subsets of agents with size nhe For each subset, perform Algorithm to obtain proper perturbation and retrieve the corresponding objective value (distance between predicted state and target state). Select the subset corre- sponding to the lowest objective value as the victim agents.
Experiment setup. We first use MADDPG (Lowe et al. 2017) train MARL agents for the four MA-MuJoCo environments listed above. Using the trained agents, we collect datasets con- taining one million transitions to train the dynamics model for each environment. The dynamics model is fully connected neural network with three hidden layers of 1000 neurons. We also use a colly-connected neural network for W (st 8) in (6) with two hidden layers of 200 neurons. We use AdamW (Loshchilov & Hutter 2017) ) as the optimizer and select the best learning rate from (1,5) x 110-9,10-2 1O 10-3} (the best learning rate the one achieving lowest prediction error on test set of 80, 000 samples). For model-based attack, we run PGD for K 30 steps to solve ) and (5) We perform each attack over 16 episodes then average the rewards. We also illustrate the standard deviation of rewards using the shaded area in the plots.
Model-free baselines vs model-based attack. In this experiment, we run the two baseline attacks along with model-based attack on the four MA-MuJoCo environments, in which there is only one victim agent (n,, = 1). Figure illustrates the performance when we perform these attacks on specific agents under multiple budget levels using foc-norm. For fixed agent, our model-based attack consistently outperforms the two basclines. In particular, our model-based attack yields much lower rewards under low budget constraints (small e) compared to the two baselines.
To better visualize the performance difference, Iillustrates the environment with and without attacks captured different time-steps. From Figure 2 our model-based attack is able to make the MuJoCo agent flip, which terminates the episode at the 409-th timestep. The episode length and total rewards for each variant are: No attack(1000, 2644.60). Uniform( (1000, 1758.97). Gaussian (8911, 1399.51), Ours( (409, 287.06).
We also investigate how the state values change during these attacks. Figure S presents different recordings of state values under adversarial attacks compared to no attack. Consider state index 8, which represents the horizontal velocity of the agent. For the Hal fCheetah environment, as the goal is to make the agent move forward as fast as possible, we expect the reward to be proportional to this state value. From Figure 3. all three attacks have fairly sparse red fractions across timesteps, which result in much lower reward compared the no-attack setting. Among the three attacks, our model- based ones appear to have the most sparse red fractions leading to the lowest rewards. In addition, the model-based attack appears to show its advantage in environments with more agents.

---

Under review as a conference paper at ICLR 2022
Mean rewards Walker2d (2x3)
Mean rewards HalfCheetah (2x3)
2500
2000
Nienn Bowardd
Moom Roward
1500
2000
1.000
1000
0.0
0.1
0.2
0.
0.4
0.5
0.1
0.3 Noise Level. s.
0,2
0.4
0.5
Level
[|td]
Uniform
Uniform noie Agent
cMIAA (ours)
Agent
Uniform noise
Agent
cMBA (ours)
Agert
Ga ussian noise Agent
cMBA (ours) Agent
Goussian roise
Agent 0
cMBA (ours) Agent
Mean rewards HalfCheetah (6x1)
Mean rewards HalfCheetah (6x1)
3000
3000
Noan Rewwers
2000
Moan Reward
2000
1500
2500
1000
500
8,0
0.1
0.2
0.3
0.4
0.5
0.1
0.3 Noise Level, [|As]), st
0.2
0.4
0.5
Noise Level, [[at] sc
Uniterm
Gaussian
Uniform noise
Agent
Gaussian
(ours)
Agent
Agent
Uniform neise Agant
ZMBA (ours) Agent
Unifom noise noise-Agerts Agent 5
cMBA (ours] Agent
Ga ussiaan nosse Agent cMBA (ours) Agent
Gaussian roise Agent 3
cMBA (ours) Agent
Agent
Mean rewards Ant (4x2)
Mean rewards Ant (4x2)
3000
3000
2000
2000
Mean Roward
1000
1000
-1000
-100
-20g0
0.025
0.050
0.575
0.100 8.325 [[
0.150
0.175
0.200
-2000
0.025
0,050
0.075 0.100 0.125
0.150
0.175
0.200
[|88]]
Unifomm noise Agent 0
Sauzdon nobe Agent1
Uniform notee Agent
Gousston noise Agent
Uniform noise - Agent 1
cMBA tours) Agent
Uniterm nakee Agent 3
CHBA (ours) Agent
Gecosian noise Agent
<MBA (ours)
Gaussion nokee Agent 2
cMBA (ours) Agent
Figure 1: Model based attack baseline attacks fixed agents.

---

Under review as a conference paper at ICLR 2022
Step 0
Step 80
Step 160
Step 240
Step 320
Step 400
Step 409
No attack
Uniform
Gaussian
Figure 2: Various attacks on Agent 0 in Ant (4x2) environment with || S 0.1. It can be seen that the agent flip at the end of episode under our model-based attack, demonstrating the effective- ness of our algorithm.
In particular, our approach results lower rewards under a smaller budget as seen in HalfCheetah (6x1) and Ant (4x2) environments.
Cheethh 2x3 (Agent O), Reward: 4536.39
HalfCheetah Uniform moise, delli ds (Agent Reward: 2209.57
353
HalfCheetah, Gaussian nolse, [idr] s (Agent O), Reward: 1782.00
2E3
HairCheetah, 2xd CMBA (ours), [[AT]] 02 (Agent 0), Reward: 1112.37
180
Figure 3: Recordings of state values an episode under different attack on Agent 0 in HalfCheetah( 2x3) environment.
8

---

Under review as a conference paper at ICLR 2022
Effectiveness of learned victim selection. We further demonstrate the performance of our model- based attack with and without learned victim selection on one victim agent. Figure illustrates the performance of four model-based variants: the best performing model-based attack the fixed agent (Best Fixed Agent), model- model--based attack on the random agent (Random Agent), model-based with learned victim selection in Algorithm (Learned victim Selection), and model-based with greedy victim selection (Greedy). Surprisingly, even though the greedy approach employs a brute- force strategy to select the agent with the closest distance, its performance reveals that selecting victim agents greedily might not be good strategy. The learned victim selection seems to be better than the random or greedy strategy and comparable with the random one in HalfCheetah environment. The best-fixed agent variant has good performance under low budget constraints in HalfCheetah (6x1) and Ant (4x2) environments. We want to emphasize that although the learned victim selection strategy in Section 3.3 is sub-optimal, it works well in these four environments.
Our cMBA Variants
Walker2d (2x3)
Our cMBA Varlants HalfCheetah (2x3)
3000
5000
2500
MNan Relerdd
Moon Roward
2000
1500
1.000
500
1000
0.0
0.2
0.4
0.0
0.2
0.3
0.4
Nolse Level liádl s
Nose Level, [Ad]]
Agent
Greedy
Agont
Greasy
Our cMBA Variants- HalfCheetah (6x1)
Our cMBA Varlants Ant (4x2)
2009
1000
1000
500
-1000
-200
0.525
0.056
0.375 0.200 6,075
6.100
8.12
0150
Lovol, [|osti.
Nolsel Level, [| St
Best Food Agent
Learned Victim Selection
Agent
Selection
Random Agent
Greddy
Random Agent
Greedy
Figure 4: Performance of different variants of our proposed model-based attack (c-MBA) with dif- ferent choices on victim agents.
CONCLUSIONS
In this paper, we propose a new attack algorithm named cMBA for evaluating the robustness of c-MARL environment with continuous action space. Our cMBA algorithm is the first to consider the c-MARL and the adversarial perturbation computed by solving an optimization problem with learned dynamics models. The cMBA approach outperforms the other baselines attack by large margin under 4 multi-agent MuJoCo environments especially ones with larger number of agents. Unique multi-agent setting, also study different strategies select victim agents. Extensive experiment results on standard c-MARL benchmarks show that our proposed model-based attack and victim-selection strategy can successfully degrade the performance of well-trained c-MARL agents while outperforming other baselines by a large margin. A future direction is to consider timed attack strategy where the perturbation is added at certain timesteps. We can also consider longer planning step, i.e. trying to reach target state within the next T steps instead of 1.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
This paper does not contain ethics concerns.
REFERENCES
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho. Mani Srivastava, and Kai-Wei Chang. Generating natural language adversarial examples. arXiv preprint arXiv:1804.0777 2018.
Gregory Dudek, Michael Jenkin, Evangelos Milios, and David Wilkes. A taxonomy for swarm robots. In Proceedings of 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems (13)S)93), volume 1, pp. 441-447. IEEE, 1993.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv: 10615, 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint
2014.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pp. 66-83. Springer, 2017.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv: 7202.0224, 2017.
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. arXiv preprint (007.07777.0322 2017.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 8018-8025, 2020.
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. arXiv preprint 170.06620,. 2017.
Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82-94, 2016.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:/509.0297) 2015.
Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Garn-Gareia, and Nicolas Papernot. On the robustness of cooperative multi-agent reinforcement learning. In 2020 IEEE Security and Privacy Workshops (SPW), pp. 62-68. IEEE, 2020.
Yen-Chen Lin, Zhang-Wer Wei Hong. Yuan-Hong Liao, Meng- Shih, Ming- Yu Liu, and Min Sun. Tac- tics of adversarial attack on deep reinforcement learning agents. arXiv preprint arXiv:1703..06 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.01101, 2017.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Aueti-agent actor- critic for mixed cooperative-comppttiiee environments. arXiv preprint arXiv: / 7006 0200000275, 2017.
10

---

Under review as a conference paper at ICLR 2022
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning, nature, 518(7540):529- 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937 PMLR, 2016.
Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2003.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Frans A Olichock and Christopher Amato. A concise introduction to decentralized POMDPs.
Springer, 2016.
Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value func- tions for decentralized pomdps. Journal Artificial Intelligence Research, 32:289-353, 2008.
Afshin OroojlooyJadid and Davood Hajinezhad. A review of cooperative multi-agent deep rein- forcement learning. arXiv preprint 1908.
2019.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations deep learning in adversarial settings. 2016 IEEE European sympo- sium on security and privacy (EuroS&P), P), pp. 372-387. IEFF, 2016.
Bei Peng. Tabish Rashid, Christian A Schroeder de Witt, Torr. Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gradients. arXiv preprint arXiv:2003.06709, 2020.
Kamienny, Philip HS
Thomy Phan, Thomas Gabor, Andreas Sedlmeier, Fabian Ritz, Bernhard Kempter, Cornel Klein, Horst Sauer, Reiner Schmid, Jan Wieghardt, Mare Zeller, et al. Learning and testing resilience in cooperative multi-agent systems. In Proceedings of the 19th International Conference on Au- tonomous Agents and MultiAgent Systems, pp. 1055-1063, 2020.
Thomy Phan, Lenz Belzner, Thomas Gabor, Andreas Sedlmeier, Fabian Ritz, and Claudia Linnhoff- Popien. Resilient multi-agent reinforcement learning with adversarial value decomposition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 1 11308-11316. 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforce- ment learning. In International Conference Machine Learning, pp. 4295-4304. PMLR, 2018.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob N Fo- erster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent rein- forcement learning. J. Mach. Learn. Res., 21:178-1, 2020.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/ abs/1902.00443 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning. pp. 1897. PMLR, 2015.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Interna- fional Conference on Machine Learning, pp. 5887-5896. PMLR, 2019.

---

Under review as a conference paper at ICLR 2022
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. decomposconnoiiionnn networks for cooperative multi-agent learning. arXiv preprint arXiv: 1 1ado:::00000000000 702.09996, 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprini arXiv:13/2.6199 2013.
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4):c0172335, 2017.
Ming Tan. Multi-agent reinforcement learning: Independent cooperative agents. In Proceedings of the tenth international conference on machine learning, pp. 330-337, 1993.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008. 01062, 2020a.
Yihan Wang, Beining Han, Tonghan Wang. Heng Dong. and Chongjie Zhang. Off-policy multi- agent decomposed policy gradients. arXiv preprint arXiv:2007. 12322, 2020b.
Isui-Wei Weng, Krishnamurthy Dj Dvijotham, Jonathan Uesato, Kai Xiao, Sven Gowal, Robert Stanforth, and Pushmeet Kohli. Toward evaluating robustness of deep reinforcement learning with continuous control. International Conference on Learning Representations, 2019.
Kaiqing Zhang, Tao Sun, Yunzhe Tao, Sahika Genc, Sunil Mallya, and Tamer Basar. Robust multi- agent reinforcement learning with model uncertainty. In NeurIPS, 2020a.
Wei Emma Zhang, Quan Z. Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial attacks deep-learning models natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology (TIST), 11(3):1 41, 2020b.
12

---

Under review as a conference paper at ICLR 2022
A MODE DETAILS ON EXPERIMENT SETUP IN SECTION 4
Specifying target state for each environment. To perform our model based attack, we need to specify a target state that potentially worsens the total reward. In multi-agent MuloCo environments, each agent has access to its own observation of the agent consisting the position-related and velocity- related information. The position-related information includes part of æ; y, 2 coordinates and the quarternion that represents the orientation of the agent. The velocity-related information contains global linear velocities and angular velocities for each joint in MuJoCo agent. We refer the reader to Todorov et al. (2012) for more information about each MuJoCo environment. Now we describe the design of this target state for each environment as follows:
Walker (2x3) environment: Since the episode ends whenever the agent falls, i.c. the coordinate falls below certain threshold. In this environment, the target state has value of 0 for the index that corresponds to the coordinate of the MuloCo agent (index 0).
HalfCheetah (2x3) and HalfCheetah (6x1) environments: As the goal is to make agent moves fast as possible, we set the index corresponding to the linear velocities to 0 (index 8).
Ant (4x2) environment: As the agent can move freely in 2D-plan, we set the index corresponding to the z, linear velocities to (indices 13 and 14).
B ADDITIONAL EXPERIMENTS
In this section, we present experimental results in addition ones presented in Section 4
Figure illlsttrate the environment with and without attacks captured at different timesteps. From Figure our model based attack is able to make the MuJoCo agent fall down which terminates the episode at the 65-th timestep. The episode length and total rewards for each variant are: No attack(478, 1736.44). Uniform(382, 1037. (3), Gaussian(90) 32.24). Ours(63, 344.26). Figure 6 illustrates how the state values change during an episode. These state values correspond to the agent shown in Figure 5 If we look at how the noise values change in Figure 7 the noise generated by our approach appears to maximize the permissible noise budget.
In addition the -nom budget constraint, also evaluate adversarial attacks using the 41- norm constraint. Note that using f--norm for budget constraint is more challenging as the attack needs to distribute the noise across states while in the & -norm the computation of perturba- tion for individual state is independent. Figure illustrates an episode of Walker agent with and without attack. Our cMBA approach able make the agent fall the 255-th timestep with the episode length and rewards for each setting as: No attack 2513.57), Uniform 403, 1 1144.77), Gaussian(4555 1637.16). O2rs(253, 759.85). From the noise values Figure 7 our cMBA method appears to put noise on few selective states at each timestep, similar to the Gausian noise setting.
Additional experiments using the approach in Lin et al. (2020) for continuous action spaces: In this experiment, we follow the approach in Lin al. (2020) in the Ant (4x2) environment where train an adversarial policy for one agent trying minimize the total team reward while the remaining agents use the trained MARL policy. The adversarial policy is trained for I million timesteps. The results using this approach compared with approach and the two baselines presented in Figure
From Figure II our cMBA approach outperforms Lin et al. 2020 's approach as is able to achieve
lower team rewards when attacking the same agent. We also note that Lin et al. (2020) 's approach
better than the two baselines using Uniform and Gaussian noise.
Effect of using [|-II, budget constraint: In this experiment, we replace the 11-11-o by the ||-II; for budget constraint. Using [| constraint will produce significant different adversarial noise pattern
13

---

Under review as a conference paper at ICLR 2022
Step 0
Step
Step
Step
Step 60
Step 65
No attack
Uniform
Gaussian
ours
Figure 5: Various attacks on Agent 0 in Walker (2x3) environment with | S 0.2.
Walker 2x3 No attack (Ag (Agant 0), Reward: 1736.44
Walker 2x3 Uniiform noise, [|As]] Adlls50z 502 (Agent 0), Reward: 1037.73
Walker 2x3 Gaussian neise, |Asile = 0.2 (Ag ent 0), Reward: 32.24
155
Walker 2x3 cMBA (ours), [AAd]) s0222 (Agent 0), Reward: -34.26
11
Figure 6: Record of state values in an episode under different attack on Agent in Walker (2x3) environment with |||| < 0.2.
14

---

Under review as a conference paper at ICLR 2022
Walker 2x3 Uniform noise, [[aill a 0.2 (Agent o), Reward: 1037.73
Walleer 2x3 Gaussian nolse, [issil = 0Z (Ag ent 0), Reward: 32.24
Trmmmee
(ours), |AASl,, (Agent 0),
Figure 7: Record of noise values in episode under different attack Agent 0 in Walker (2x3) environment with with ||As|| 2 0.2.
Step
Step
Step 100
Step 150
Step 200
Step 250
Step
No attackk
Uniform
Gaussian
Ours
Figure 8: Various attacks on Agent 0 in Walker (2x3) environment with |As|], < 0.5.
15

---

Under review as a conference paper at ICLR 2022
Walker 2x3 No attack (Agent 0), Reward: 2513.57
16-
12.
Indok
50
100
150
200
250
Timesteps
Walker 2x3 Uniform noise, || 0.5 (Agent 0), Reward: 1144.77
16
14-
Index
10-
50
100
150
200
250
-10
Timesteps
Walker 2x3 Gaussian noise, ||As|| 5.5 (Agent 0), Reward: 1637.16
10
14
12:11
0
2-
150
250
-10
Walker 2x3 cMBA (ours), ||As|h 0.5 (Agent O), Reward: 759.85
16.
12
Index
50
100
150
200
250
Timesteps
Figure 9: Record of state values in an episode under different attack on Agent in Walker (2x3) environment with with ||S||| 0.5.
16

---

Under review as a conference paper at ICLR 2022
Walker 2x3
Uniform noise, ||As|| s0.5 (Agent Reward: 1144.77
16
12
0.2
Indes
10
-022
100
150
200
250
Timestops
Walker 2x3 Gaussian noise, || s 0.5 (Agent 0), Reward: 1637.16
0.4
0.2
Indox
10-
0.2
100
150
200
250
Timestaps
Walker 2x3 cMBA (ours), || s 0.5 (Agent 0), Reward: 759.85
14
0.4
Index
0.0
-0.4
50
150
200
Timesteps
Figure 10: Record of noise values in episode under different attack on Agent O in Walker (2x3) environment with with ||As|| S 0.5.
(4x2)
Ant (4x2)
2000
2000
Maan Naward
Rowara
-1000
-1000
-200
0.005
0.27%
0.100
0.325
6.150
6.200
-2000
0.595
0.325
0.375
[|/M]|=
[|An]|-
Agarto
(2029) Agant
Agant
Uniform roise Agert
Un et (2029)
Agont
Unitorm nose Agent
Lnn et (2020) Ag cm
Gaursion nobe Agent
Agent 0 CHBA (ours) Agent
dMBA (ouns)
Gaussian noire Agent
cHBA (ours) Agent
Gaussian nose Agent
Gaussian naice Agent
CHBA (ours) Agent
Figure 11: Adversarial attacks using |l budget constraint in Ant (4x2) environment.
17

---

Under review as a conference paper at ICLR 2022
compared ll as producing adversarial noise when using |1-11 is independent for each state while is not the case for I|-Il We run different attacks on two environments, HalfCheetah (6x1) and Ant (4x2), and the results presented in Figure and Figure I3 From these figures, cMBA outperforms other baselines. In particular, able to achieve much lower rewards under smaller budget constraint which shows the advantage of our approach.
Mean rewards HalfCheetah (6x1)
Mean rewards
HalfCheetah (6x1)
3000
3000
2500
Mean Rewwre
2000
Mcan
1500
1000
1000
10
10
Noire Levde sAS
Notso laadia scc
Uniferm noise Agent
Gaussian noise
Unilsrm raise Agent
Caussian noise Agent
Uniform
Unikem
Uniferm Agent
<MBA (nurs) Agert!
Uniferm raise Agert
cMBA (ours) Agent
Gaussian noise Agent
<MBA (ours) Agent
Gaussian noise Agent
cMBA (ours) AAgart
Caussian noise Agent
Gaussian neise Agenz
Figure 12: Adversarial attacks using |1-11, budget constraint in HalfCheetah (6x1) environment.
Mean rewards (4x2)
(4x2)
2500
2500
2000
2000
Mean Revard
1000
1000
Moan
300
500
500
500
-1000
-1000
Nobse [|88]]
Nobse
Agent
Agent (ours) Agennt
Agent
Agent
Unilam naise- Agent
Uniform Agent
cMBA (ours) Agentt
Gatsssia noise - Agunt C
200 <MBA (ours) Agant
Gatussian noise Agunt 2
Box cMBA (ours) - Agant
Figure 13: Adversarial attacks using ||-1|/ budget constraint in Ant (4x2) environment.
Adversarial attacks using dynamics model with various accuracy: this test, we compare the performance of our attack when using trained dynamics model with different mean-squared error (MSE). We use the Ant (4x2) environment and consider the following 3 dynamies models:
The first model is trained using million samples for 100 epochs and select the model with the best performance. We denote this model "M Best epoch".
The second model trained using million samples for only epoch. We denote this model "1M - 1st epoch".
We further reduce the number of samples used for training the dynamic model to only 200,000 and train for only epoch. We denote this model "200K 1st epoch".
18

---

Under review as a conference paper at ICLR 2022
These models are evaluated predefined test set consisting of 100,000 samples. The test MSE of these models are 0.33, 0.69, 0.79, respectively, with the initial test MSE of 1.241.
Effect dynamics model accuracy (4x2)
Effect dynamics model accuracy (4x2)
2000
2000
Mean Noward
1500
500
500
-1000
8.025
9.e50
0.100 0.125 hoise Level, [|oal] m*
0.150
0.175
02000
-1000
Neive Lavel. [talls mx
1M Best spech
IM 1st spoch
200K. 1ot spach
IM Best spach
1M 12 spech
200K: 1st spach
Figure 14: Adversarial attacks using dynamics models Ant (4x2) environment.
Figure depicts the attacks using these three models on the same agent in Ant (4x2) environment using F budget constraint. Interestingly, the dynamics model trained with only 200,000 sam- ples for epoch can achieve comparable performance with the other two models using much more samples.
Comparison between attacking all agents attacking 1 agent: In this experiment, we compare performance of different approaches when attacking one victim agent or simply perturbing all agents under the same budget constraint. We compare different approaches in the Ant (4x2) environment. In particular, we use the two baselines and our cMBA approach to attack each agent out of 4 agent, denoted as (agent i) for = 0, 1, 2, We also use these approach which we simultaneously perturb the inputs to all agents, denoted as (4 agents). In addition, we also illustrate the performance of our victim selection scheme when we only want to attack cither 1 or two agents at the same time. We report the mean and standard deviation of rewards using these approach with several values of budget level e in Table [ where we use || Ilx for the budget constraint. From Table ] the performance is getting better the attacker is stronger, i.e. more agents are attacked at the same time. This makes sense as for || Iloo, the adversarial noise for each agent can be computed independently.
Now, we consider the -| budget constraint. This is a more interesting setting as the total noise budget is fixed the adversarial noise of agent will affect from the other. The results shown using several values of budget level in Table 2 We can observe that in this case attacking more agents is not always effective.
19

---

Under review as a conference paper at ICLR 2022
Table 1: Comparison between adversarial attacks on a single agent and all agents in Ant (4x2)
environment under - || I budget constraint.
Rewards: Mean (standard deviation)
Methods
0.025
0.05
é 0.075
0.1
Uniform noise (agent 0)
2332 (594)
2296 (88)
1773 (941)
1542 (554)
Uniform noise (agent 1)
2233 922)
2033 (565)
1579 (856)
1293 (385)
Uniform noise (agent 2)
2028 (695)
1713 (649)
988 (650)
322 (652)
Uniform noise (agent 3)
2579 (66)
2484 (73)
2159 (623)
2208 (279)
Uniform noise (4 agents)
2089 (450)
1093 (753)
299 (781)
62 (921)
Gaussian noise (agent 0)
2450 (134)
2030 (405)
1660 (663)
1067 (536)
Gaussian noise (agent 1)
2204 (709)
1992 (337)
1447 (406)
1141 (381)
Gaussian noise (agent 2)
2256 (297)
1313 (977)
287 (923)
154 (570)
Gaussian noise (agent 3)
2550 (83)
2331 (567)
1927 (1250)
2115 (312)
Gaussian noise (4 agents)
2022 (303)
761 (1083)
86 (688)
-133 (809)
cMBA (agent
923 (340)
244 (737)
516 (737)
-8 (306)
cMBA (agent 1)
1417 (48)
553 (269)
108 (303)
-60 (309)
cMBA (agent
227 (955)
-652 (1097)
-241 (660)
-631 (911)
cMBA (agent 3)
2024 (87)
861 (1298)
567 (1012)
-172 (725)
cMBA (4 agents)
1116 (379)
314 (734)
72 (407)
-239 (281)
Leaned Victim Selection (1 agent)
-165 (805)
624 (103)
256 (201)
-2 (123)
Learned Victim Selection agent)
405 (308)
-176 (853)
-588 (1026)
-325 (730)
Table 2: Comparison between adversarial attacks on a single agent and all agents in Ant (4x2)
environment under ||1 < budget constraint.
Rewards: Mean (standard deviation)
Methods
= 1.5
= 2.0
= 2.5
3.0
Uniform noise (agent 0)
2189 (151)
2093 (148)
1615 (524)
1421 (780)
Uniform noise (agent 1)
2039 (139)
1715 (282)
1387 (586)
1243 (303)
Uniform noise (agent 2)
1542 (826)
1073 (755)
581 (920)
436 (317)
Uniform noise (agent 3)
2468 (75)
2332 (282)
2306 (131)
2136 (319)
Uniform noise (4 agents)
2224 (353)
2037 (412)
1734 (541)
1473 (558)
Gaussian noise (agent 0)
1752 (877)
1676 (650)
1431 (774)
1493 (554)
Gaussian noise (agent 1)
1775 (344)
1442 (619)
1257 (487)
1036 (554)
Gaussian noise (agent 2)
1326 (766)
1049 (666)
776 (763)
516. (842)
Gaussian noise (agent 3)
2103 (559)
2319 (104)
2086 (480)
1896 (603)
Gaussian noise (4 agents)
1614 (776)
1.553 (793)
1544 (568)
1470 (476)
cMBA (agent 0)
2068 (145)
1699 (169)
1432 (136)
1282 (136)
cMBA (agent 1)
2005 (355)
1682 (346)
1505 (222)
1189 (585)
cMBA (agent 2)
1554 (781)
1080 (792)
433 (1202)
513 (987)
cMBA (agent 3)
2302 (80)
2032 (528)
2099 (92)
1604 (1069)
cMBA (4 agents)
2212 (225)
1909 (332)
1671 (212)
1504 (201)
Leaned Victim Selection (1 agent)
706 (1118)
355 (635)
-326 (931)
-139 (608)
Learned Victim Selection (2 agent)
1274 (779)
720 (939)
152 (787)
156 (638)
20


Reviews:
[{"review_id": "ypF4gZ1Qq-x", "reviewer": "Reviewer_jFMh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "***Soundness***\n\nThe main concern of the reviewer is about the novelty and applicability of the proposed method.\n\n1) The proposed attacking method depends on a pre-trained dynamics model. Training such a model may be costly in multi-agent settings. \n\n2) The most frustrating point about the proposed method is that the target state needs to be predefined, which largely limits the applicability of the proposed method.\n\n3) For selecting the victim agents, the proposed method needs local information of all the agents. This assumption is less realistic in multi-agent settings. Moreover, together with 2), this assumption simplifies the difficulty of attacking MARL algorithms.\n\n4) The presented results are not surprising because the instability of MARL has been well studied. As reported in the paper, random noise and randomly selected victim agents can also reduce performance significantly.The reviewer thinks that perturbing local observations is not a promising research direction. The reviewer holds that a simple attacking method where the gradients of observation with respect to value function is used to perturb the observations can have significant influence.\n\n5) The proposed method has access to the policy of all agents. Such a white-box attacking method is less applicable.\n\n***Evaluation***\n\nThe baselines are not powerful. Please show the results of the baseline described in Soundness (4).\n\n***Clarity***\n\nThis paper is not well written, and there are various grammatical errors throughout the paper. For example, in section 3.2, above equation 1, “$A$ is the set of concatenated states.” I guess this should be actions? ", "summary_of_the_paper": "The paper proposes a method to attach cooperative MARL strategies on tasks with continuous action spaces. A dynamics model is trained, based on which agents’ local observations are perturbed so that agents select actions that lead to the least-rewarding states.", "summary_of_the_review": "The problem under research is interesting, but the proposed method can access parameters of all agents’ policies, can modify observations of all agents, and depends on pre-defined target states. These assumption hurts the novelty and applicability of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WKsd7N2vS1c", "reviewer": "Reviewer_1ANY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the idea of using a learned dynamics model to reduce the need for environment interactions when computing proposed attacks.\n\nI'm not so sure about the constraint that you need to prespecify a target state that leads to low performance. In some sense, if you have to pre-specify this, you have to predeterimine the type of attack you want your adversary to make, as opposed to letting it learn what are the best types of attacks given the domain. The paper against which you compare, Lin et al. (2020), seems more general to me in that they try to minimize the reward of the team. This is a broader approach in that your reward function could simply be I(s ==s_target) if you wanted your adversary to reach a certain state as in your work.\n\n\"Note that we cannot directly apply methods such as the fast gradient sign method (FGSM) or JSMA (an attack using saliency map) as in Lin et al. (2020), since those methods require a “target” action which is not available in continuous action space.\" — why is the case? If I understand correctly, the target action in Lin et al. (2020) is proposed by a RL agent policy, which could easily be a continuous action policy no? To that end, it's unclear to me why their method wouldn't work in a continuous action space if you substitute policy gradient for q-learning. That being said, it could be interesting to compare to their method with and without a learned model to replace the need for interactions with the environment.\n\nWhy is victim selection important? What the real world instances where you'd have an adversary be able to select a different agent at each timestep but not be able to simple perturb all of them?\n\nThis method would be more interesting to me if it allowed for and experimented with discrete or mixed continuous-discrete action spaces.\n\nSmall comments:\n\nOverall I found the grammar and writing to be somewhat poor and distracting.\n\nTo me, using language \"dynamic model\" is more unclear than \"dynamics model\". The latter I think is more commonly used.\n\n\"neural network of target policy\" → \"neural network of **the** target policy\"\n\n\"another line of research tackle\" → \"another line of research tackle**s**\"\n\n\"problem on DRL with continuous action space**s**\"\n\n\"where A is the set of concatenated states\" — I think this should be \"concatenated actions\".", "summary_of_the_paper": "Proposes an adversarial attack on continuous cooperative multi-agent settings. Given a target low performing state, the adversary chooses an observation perturbation by trying to minimize the distance from the next state (as predicted by a dynamics model) to the target state via projected gradient descent. They also experiment with optimizing for and choosing a limited number of agents to perform the adversarial action at each timestep. There experiments are in multi-limbed mujoco tasks such as Walker, HalfCheetah, and Ant, and show that their method causes the agent to receive lower reward than a random adversary.", "summary_of_the_review": "Overall I felt this paper was below the acceptance threshold primarily because I felt the experiment domains rather limited and disappointing that the only baseline was random noise (and that the method seems to do only marginally better than random noise). I also felt that prior methods could work in continuous action spaces if very slightly modified, so I felt this paper could be much stronger if it experimented with and compared to those variants.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QvY4ulKBXr", "reviewer": "Reviewer_55ss", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The following are some of my concerns and questions:\n1. The main issue is related to the novelty of this work. Previous works (Gleave et al. 2020, Lin et al. 2020) have looked at attacking MARL. Thus it is no surprise that MARLs are vulnerable to perturbation of the agents’ observation. In fact, as shown in the evaluation section, random noise seems to have comparable (or sometimes better) performance compared to the proposed method, which further emphasizes the instability of the cooperative MARL algorithms. In fact, the brittleness of cooperative strategies is richly documented in game theory literature, which is not addressed in this paper. \n\n2. From a practical point of view, this work also has a few shortcomings. The attack method proposed in this work is a white-box attack, which requires the attacker to have access to the model’s parameter, which limits the applicability of the attack.\n\n3. The proposed method in fact does not directly minimize the team reward, but the users need to manually define a target state as part of the attack objective. Defining the target states can be challenging for some use cases, and hence it is in fact non-trivial to evaluate the robustness of the MARL algorithm.\n\n4. Gradient-based attack methods such as FGSM and JSMA are not compared in this work. Although the authors mentioned that the target action is not available in the continuous action space, one can actually back prop from the centralized critic toward to actor policy with an objective to minimize the team reward. How do those methods compare to your proposed method?\n\n5. One of the contributions in this work is the victim selection during the attack. However, it seems the attacker needs to have observations of all agents and be able to modify the observation of any agent. This can be challenging during the execution phase of MARL where the agents are physically distributed. Can you provide some concrete applications of this attack? What are the assumptions of the attack?\n\n6. This attack requires training a model-based state prediction model. How much data is needed to train such a model? How much time is required for training? Does it assume the attacker has access to the underlying transition function?\n", "summary_of_the_paper": "This work focuses on attacking cooperative multi-agent reinforcement learning using a model-based method. First, a model is trained to predict the next state based on the current state and action. Then PGD is used to find a small perturbation on the current state so that the next state is closed to a target state. The target state is selected using heuristic methods and they are used as a surrogate objective for minimizing the team reward. A victim agent selection method is also proposed for selecting the best target action.\n", "summary_of_the_review": "Although this work is looking at an important problem related to the robustness issue of MARL, its technical contribution fail short in term of the theoretical analysis, novelty, and evaluations. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IAs8aRs8I_o", "reviewer": "Reviewer_j4AK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.\tThe method proposed in the paper has marginal novelty. It looks like the authors combined different existing method, such as model-based attack, MARL, and PGD to come up with a novel framework for evaluating the robustness of the cooperative MARL. While such a combination has not been explored, the work presented in the paper did not really show sufficient and significant contributions, either theoretical or empirical. In terms of theory, this work is lack of some analysis that can guarantee the optimality or convergence. It is okay to not have such analysis, which means that the effectiveness of the proposed scheme highly depends on the empirical evidence. However, the numerical results shown in the paper to me are not very convincing. \n2.\tDue to the DECENTRALIZED MARL, typically, method would require communication protocol among various agents. However, in the paper, it looks like the author only summed up all agents and solved the optimization problem to attain the optimal solutions. That way, how to guarantee the optimality?\n3.\tIn the paper, the author leveraged the model-based attack. While a learned model for the environment is helpful, how does the model accuracy affect the performance of the proposed approach? Though using deep net to represent the dynamics may be decent for the state prediction, while the nonlinearity could significantly affect the optimization problems. How to evaluate such effect and empirically solve such a potential issue?\n4.\tIn the numerical results, the authors showed different results for Mujoco environment. However, the baselines to me look simple and in the introduction they have mentioned some existing works, so they should have used one or two in their comparison. Additionally, it is unknown to me whether the proposed method can always outperform those two baselines under different constraint budgets. Since I have only seen 0.2 from the supplementary materials. We can also see that the authors only showed a few agents. What about 20? 30? Even more? Though with more agents, it could be difficult to scale in the Mujoco environment. The authors could use the Gridworld to show that. Moreover, what about topologies? For a decentralized method, it is critical to see the performance with different topologies. While this is missing in the work.\n5.\tThe author mentioned in Algorithm 1 the target state. How to determine the target state in theory and practice?\n6.\tWhat is $\\mathcal{D}_{random}$ and why did the authors need that to form the buffer?\n7.\tAt the beginning, the authors said that each agent observes its own local state. While in remark 3.1, the authors assumed each agent had access to the other agent’s observation. Why not just assuming at the beginning each agent could observe the global state?\n\n****************************************\nAfter carefully reviewing the authors' responses and comments from other reviewers, I decide to keep the score. I think the current draft is still below the acceptance bar and it requires more work to make it technically solid and sound, particularly in how to determine the target state in a more efficient way, mathematically analysis for the framework, and the quantitative impact of dynamic accuracy on the performance.\n", "summary_of_the_paper": "This paper presented a novel method for cooperative MARL to evaluate the robustness under the adversarial states. Specifically, the authors leveraged a model-based approach to perform adversarial attacks on states for MARL with continuous action and state spaces. To represent well the environment, they used a deep neural network to predict the future states. Subsequently they solved an optimization problem with the learned dynamics to produce small perturbations. They also discussed a few strategies to optimally select the victim agents for the attack. To validate the proposed method, the authors implemented numerical experiments on multi-agent Mujoco tasks to show the effectiveness of the proposed method. The investigated topic in this work is quite interesting yet challenging, which remains an active research area. Overall, the paper is easy to follow and well written. However, the authors should pay attention to a few points I have raised in the following to make the paper more technically solid and sound.", "summary_of_the_review": "Overall, I think the paper still requires substantial work to make sufficient contributions. Particularly, the authors need to address some aforementioned issues, such as marginal novelty and weak numerical results, to make the paper technically solid and sound. Hopefully my comments can help out.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although the problem studied in the paper is interesting, all the reviewers believe that the current draft has limited technical contributions. Moreover, there are serious issues with the writing and presentation of the work. Also the experiments are rather limited and their results are not significant. I strongly recommend the authors to take the reviewers' comments into account and improve different aspects of their work for future conferences.
Decision: Reject