Evaluating Robustness of Cooperative MARL: A Model-based Approach
Authors: Nhan H Pham
Lam M. Nguyen
Jie Chen
Hoang Thanh Lam
Subhro Das
Tsui-Wei Weng
Venue: Submitted to ICLR 2023 (2023)
Keywords: robust c-MARL
model-based adversarial attack

Abstract: In recent years, a proliferation of methods were developed for cooperative multi-agent reinforcement learning (c-MARL). However, the robustness of c-MARL agents against adversarial attacks has been rarely explored. In this paper, we propose to evaluate the robustness of c-MARL agents via a model-based approach, named c-MBA. Our proposed formulation can craft much stronger adversarial state perturbations of c-MARL agents to lower total team rewards than existing model-free approaches. In addition, we propose the first victim-agent selection strategy and the first data-driven approach to define targeted failure states where each of them allows us to develop even stronger adversarial attack without the expert knowledge to the underlying environment. Our numerical experiments on two representative MARL benchmarks illustrate the advantage of our approach over other baselines: our model-based attack consistently outperforms other baselines in all tested environments.

Under review as a conference paper at ICLR 2023
EVALUATING ROBUSTNESS OF COOPERATIVE MARL: A MODEL-BASED APPROACH
Anonymous authors Paper under double-blind review
ABSTRACT
In recent years, proliferation of methods were developed for cooperative multi- agent reinforcement learning (c-MARL). However, the robustness of c-MARL agents against adversarial attacks has been rarely explored. In this paper, we pro- pose to evaluate the robustness of c-MARL. agents via a model-based approach, named e-MBA. Our proposed formulation can craft much stronger adversarial state perturbations of c-MARL agents to lower total team rewards than existing model-free approaches. In addition, propose the first victim-agent selection strategy and the first data-driven approach to define targeted failure states where each of them allows expert knowledge to the underlying environment. Our numerical experiments on representative MARL benchmarks illustrate the advantage of our approach over other baselines: our model- based attack consistently outperforms other base- lines in all tested environments.
develop even stronger adversarial attack without the
INTRODUCTION
Deep neural networks are known to be vulnerable to adversarial examples, where a small and often imperceptible adversarial perturbation can easily fool the state-of-the-arr deep neural network clas- sifiers (Szegedy al., 2013 Nguyen et al. 2015 Goodfellow et al. 2014 Papernot et al.) 2016). Since then, a wide variety of deep learning tasks have been shown to also be vulnerable to adversar- ial attacks, ranging from various computer vision tasks to natural language processing tasks (Jia &
Liang 2017 Zhang et al. 2020 Jin et al. 2020 Alzantot et al. 2018
Perhaps unsurprisingly, deep reinforcement learning (DRL) agents are also vulnerable to adversarial attacks, as first shown in (Huang et al., 2017) for atari games DRL agents. (Huang et al. (017) study the effectiveness of adversarial examples on a policy network trained on Atari games under the situation where the attacker has access to the neural network of the victim policy. In (Lin et al.) 2017]. the authors further investigate a strategically-timie Atari games at subset of the time- steps. Meanwhile, (Kos & Song] 2017) use the fast gradient sign method (FGSM) (Goodfellow et al. 2014) to generate adversarial perturbation on the A3C agents (Mnih et al., 2016) and explore training with random noise and FGSM perturbation to improve resilience against adversarial examples. While the above research endeavors focus on actions that take discrete values, another line of research tackles more challenging problem on DRL with continuous action spaces (Weng et al. 2019 Gleave et al. 2019). Specifically, (Weng et al.) 2019)) consider a two-step algorithm which determines adversarial perturbation be closer targetted failure state using a learnt dynamics model, and (Gleave et al. 2019) propose physically realistic threat model and demonstrate the existence of adversarial policies in zero-sum simulated robotics games. However, all the above works focused on the single DRL setting.
attack when attacking victim agents on
While most of the existing DRL attack algorithms focus on the single DRL agent setting, in this work we propose to study the vulnerability of multi-agent DRL, which has been widely applied in many safety-critical real-world applications including swarm robotics (Dudek et al. 1993). electric- ity distribution, and traffic control (OroojlooyJadid & Hajinezhad 2019). In particular, we focus on the collaborative multi- reinforcement learning (c-MARL) setting, where group of agents is trained to generate joint actions to maximize the team reward. We note that c-MARL is more chal- lenging yet interesting setting than the single DRL agent setting, as now one also needs to consider the interactions between agents, which makes the problem becomes more complicated.

---

Under review as a conference paper at ICLR 2023
Step
Step 80
Step 180
Step 240
Step 320
Step 400
Step 440
unitor artack
agent still perform well (ottack failed)
Baselines:
model-free attack
agent flippedt (nttack succeeded)
This work: moddll-assd attack (cMBA)
We illustrate the proposed model-based attack is powerful while other model-free attacks failed on attacking Agent 0 in Ant (4x2) environment. The episode ends after 440 time steps under our c-MBA (the agent flips), demonstrating the effectiveness of our algorithm.
Figure 1:
Our contribution can summarized as follows:
In this work, propose the first model-based adversarial attack framework on c-MARL, where we name it c-MBA (Model-Based Attack on c-MARL), We formulate the attack into a two-step process and solve for adversarial state perturbation efficiently by existing proximal gradient methods. We show that our model-based attack is stronger and more effective than all existing model-free baselines. Besides, we propose a novel adaptive victim selection strategy and show that it could further increase the attack power of c-MBA by decreasing the team reward even more.
To alleviate the dependence on the knowledge the c-MARL. environment, we also pro- pose the first data-driven approach to define the targeted failure state based on the collected data for training the dynamics model. Our numerical experiments illustrate that c-MBA with the data-driven failure state is comparable and even outperforms c-MBA with the expert-defined failure state in many cases. Therefore, our data-driven approach a good proxy the optimal failure state when we have little no knowledge about the state space of the c-MARL environments.
We show on both the multi-agent MuJoCo and muutti-aeent particle environments that our c-MBA consistently outperforms the SOTA baselines in all tested environments. We show that c-MBA can reduce the team reward up to 8-9x when attacking the c-MARL. In outperforms
addition, e-MBA with the proposed victim selection strategy matches other e-MBA variants in all environments with up to 80% of improvement on reducing team reward.
2 discusses related works in adversarial attacks for DRL and present general background in c-MARL setting. We describe our proposed attack framework c-MBA in Section 3.1 for fixed set of victim agents. In addition, mine the failure state for c-MBA in Section 3.2 After that, we detail an adaptive strategy to design stronger attack by selecting the most vulnerable victim agents in Section 3.3 Section [ presents the evaluation of our approach on several standard c-MARL benchmarks. Finally, we summarize our results and future directions in Section
Paper outline. Section
propose alternative data-drive approach to deter-
RELATED WORK AND BACKGROUND
Related work. Most existing adversarial attacks on DRL agents are on single agent (Huang et al. 2017: Lin et al. 2017 Kos & Song 2017 Weng et al. 2019) while there is only two other works (Lin et al. 2020: Hu & Zhang 2022) that focus on the c-MARL setting. Whereas (Hu & Zhang 2022) considers different problem than ours where they want to find an optimally "sparse" attack by finding an attack with minimal attack steps, (Lin et al. 2020) proposes a two-step attack procedure generate state perturbation for c-MARI setting which is the most relevant to our work.
2

---

Under review as a conference paper at ICLR 2023
However, there are two major differences between our work and (Lin et al. 2020 (1) their attack is only evaluated under the StarCraft Multi-Agent Challenge (SMAC) environment (Samvelyan et al. 2019) where the action spaces are discrete; (2) their approach is model-free as they do not involve learning the dynamics of the environment and instead propose to train an adversarial policy for a fixed agent minimize the the total team rewards. The requirement on training an adversarial policy is impractical and expensive compared to learning the dynamics model. To the best of our knowl- edge, there has not been any work considering adversarial attacks on the c-MARL setting using model-based approach on continuous action spaces. In this paper, we perform adversarial attacks on agents trained using MADDPG (Lowe et al., 2017) on two multi-agent benchmarks including multi-agent MuloCo and multi-agent particle environments. Note that in the setting of adversarial attacks, once the agents are trained, policy parameters will be frozen and we do not require any retraining of the c-MARI. agents during our attack.
Background in c-MARL. We consider multi-agent tasks with continuous action spaces modeled as Decentralized Partially Observable Markov Decision Process (Dec-POMDP) (Oliehoek Am- ato 2016). A Dec-POMDP has a finite set of agents N {1, .n} associated with a set of states S describing global states, a set of continuous actions A4. and a set of individual state S; for each agent i E .. Given the current state si S,, the action of A, is selected by pa- rameterized policy n' S + A,, The next state for agent i is determined by the state transition function P, S x A, - S, and agent will receive reward ri calculated from reward function : A; + R and observe a new state s1+1 E S. In addition, Dec-POMDP is associated with an initial state distribution Po and discount factor 7. Training a c-MARL agent is to find a joint policy that maximize the total team rewards Σi, rÊ. Note that for the ease of exposition, we do not differentiate state and observation in this work and use them interchangeably throughout the paper.
3 C-MBA: MODEL-BASER ATTACK FOR C-MARL
3.1 PROBLEM FORMULATION AND C-MBA ATTACK
Our goal is to generate adversarial perturbations imposed to the victim agents' input (state) in order to deteriorate the total team reward. The added perturbations encourages the victim agents' state to be close to desired failure state corresponding to low reward. To avoid sampling from the environment, we use a pre-trained model that learns the dynamics of the environment to predict the next state from the perturbed state and current action, then find the suitable noise that minimizes the distance between the predicted next state and a predefined target state. For now, we assume the target state is given and show in Section 3.2 that this target state actually be learned directly from the data. The overall attack can be formulated as an optimization problem as follows.
Formally, we consider a multi-agent setting with [W = n agents, each agent E W receives state of locally and takes action ot following the pre-trained c-MARL policy (s{)). Let 8t (s).... sx)) E S be the joint global state at time step which concatenated from local states sf for each agent E .Ñ. We also denote the joint action
(a) , a²)) concatenated from each agent's s action af. Let V/ S N be the set of victim agents at time step f, i.e. the set of agents that can be attacked. Let f : S x A + S be parameterized function that approximates the dynamics of the environment, where A is the set of concatenated actions, one from each A,. Let foil be the targeted failure state which can lead to poor performance to the agent. We denote bound on budget constraint w.r. some fn-norm II-Ilpp The state perturbation As = (Ast,... As") (we suppress the dependence on i of As to avoid overloading the notation) to 8, the solution to the following problem:
upper
min As (Ast Asn)
d( MBr+1 8fait)
s.t.
1 = f(s), at) af = T'(s + Ast),
Vi E N vi $ V.
(I)
As' 0,
Es < 8t + As S us
As 1l, Se,
Vi V.
where 0 is zero vector, and the state vector follows boxed constraint specified by ls and us.
3

---

Under review as a conference paper at ICLR 2023
Let us first provide some insights for the formulation ( For each agent i, using the trained policy T², we can compute the corresponding action a; given its (possibly perturbed) local state sj or st + As'. From the concatenated state-action pair (st, az), we can predict the next state 6,+1 via the learned dynamics model f. Then by minimizing the distance between 8 and the targeted failure state ffait subject the budget constraint, damaging failure state in the next time step leading to low team reward.
forcing the victim agents to move closer
Note that problem ① can be reformulated as a constrained nonconvex problem which can be effi- ciently solved by first-order method to obtain stationary point. We defer the details to Appendix [ Here, the convergence guarantee is that the perturbation found by solving the optimization problem (I) will make the next state (predicted by our dynamics model) closest to the failure state given the budget constraint. Finally, the full attack algorithm of c-MBA at timestep can be summarized in Alg.
Algorithm c-MBA algorithm at timestep t
1:
Initialization:
2: Given 8t, Sfall. n, f. Vú; initialize As = sign(x) for ~ N 1), attack budget E, p; choose learning rate >
3: For k 0, K do
4: Compute d2 (a), ,af) where of (s{ + Ast) if V2 and of (sf)) otherwise.
5: Compute $141 = f(s,, az).
6: Update As as AsH+1 proje,... [A ((
7:
End For
Learning dynamics model. One of the key enabler to solve 11 is the availability of the learned dynamics model f. If the dynamics is known, we can solve ( casily with proximal gradient meth- ods. However, in practice we often do not have the full knowledge of the environment and thus if we would like to solve ( we can learn the dynamics model via some function approximator such as neural networks. The parameter w for f is the solution of the following optimization problem
min
Σ11(s ar; +) 81+1
LED
where 'D is a collection of state-action transitions {(s,, at, 8e4 1) HED and 8141 is the actual state that the environment transitions to after taking action a₂ determined by a given policy. In particular, we separately collect transitions using the pre-trained policy Ter and random policy xnd to obtain Dirain and Drendom- The motivation of using the random policy to sample to avoid overfitting the dynamics model to the trained policy. Then the dataset 'D is built as D = DDtrain U U'DDandoo- Since (2) is standard supervised learning problem, the dynamics model f can be solved by existing gradient-based methods. We describe the full process of training the dynamics model in Alg. A where the bradient BasedUpdate step could be any gradient-descent-tyee update. In our experiments, notice that the quality of the dynamics model does not significantly affect the result as seen in Figure [I] We find that even when we use a less accurate dynamics model, our proposed c-MBA attack is still effective. Our attack using the dynamics model trained with only 0.2 M samples for only 1 epoch is comparable with ones using more accurate dynamics model (1M for and 100 epochs) in the Ant (4x2) environment under both loo and L budget constraint. We describe the whole process of training the dynamics model in Algorithm 4 in Appendix B
Discussion: difference between the baseline (Lin al.] 2020). We note that the most closely related to our work is (Lin et al.) 2020). where they also propose an attack algorithm to destroy c-MARL. However, there are two major differences between their approach and ours: (1) Their method is a model-free approach based on training extra adversarial policies, which could be im- practical as it requires a of samples and may difficult to collect the required "bad" trajec- tories to minimize the team reward (this requires the full control of all the agents in the c-MARL. setting which may not be available in practice). On the other hand, our c-MBA is a model-based ap- proach, where we only need to have a rough surrogate of the environment. This is more practical scenario, and even very crude dynamics model could make c-MBA effective (see Experiment (III) in Section 4). (2) They did not leverage the unique setting in MARL to select most vulnerable agent to attack, while in the next section, we show that with the victim agent selection strategy, we could make c-MBA an stronger attack algorithm (also Experiment (IV) for more details).
4

---

Under review as a conference paper at ICLR 2023
We note that our approach is applicable for the black-box attacks where adversary does not have access to the policy parameter. The reason is because Alg. 7 and Bondynnny need to have mechanism access the gradient of the output action with respect to the state inpus, which could be estimated by finite difference methods similar to the black-box attacks in the image classification setting. As the baseline method in (Lin et al. 2020) focuses on white-box attack, we also implement our approach under the white-box setting for fair comparison.
Remark 3.1
3.2 LEARNING FAILURE STATES FROM DATA
In order to solve I we need to specify the failure state fail which often requires prior knowledge of the state definition of the c-MARL environment. To make method more flexible, propose the first data-driven approach to learn the failure state. As our c-MBA attack involves training dynamics model of the environment by collecting the transition data, we can learn the failure state from directly the pre-collected dataset without extra overhead. Based on the observation that the a failure state should be resulting state where the reward corresponding to that transition is low, we sort the collected transition (st, at. $t, rz) by ascending order of the reward and choose the failure state to be & that corresponds to the lowest ry in the dataset. This process described in Algorithm
Algorithm Learning failure state from collected data
1: Input:: dataset D = {(st, Gg, St+1)heD collection of transitions.
2: Sort the transition by ascending order of reward r,
Determine ( ((),T(1)) as the transition corresponding the minimum reward.
4: Set fail = $(1).
Using this data-driven strategy, we show that performing c-MBA attack with the learned failure state either matches or performs better than the expert-defined failure state for multi-agent MuJoCo environments. In addition, the data-driven approach demonstrates its advantage in the multi-agent particle environment where we not have expert knowledge of the state space. Overall, c-MBA using the learned failure state shows its superior performance over other model-free baselines in most cases experiments.
3.3 CRAFTING STRONGER ATTACK WITH C-MBA VICTIM AGENT SELECTION STRATEGY
In this subsection, we propose new strategy to select most vulnerable victim agents with the goal to further increase the power of our cMBA. We note that this scenario is unique in the setting of multi-agent DRL setting, as in the single DRL agent setting we can only attack the same agent all the time. To the best of our knowledge, as (Lin et al, 2020) only use one fixed agent to perform the attack. As a result, we can develop a stronger attack by selecting appropriate set of "vulnerable" agents. This strategy can be effective in the "sparse attack" setting when only a few agents in the team are attacked (Hu & Zhang (202)). To start with, first formulate mixed-integer program perform the attack set of victim agents as below:
work is the first to consider the victim selection strategy
minAs s.t.
d( d(s 8 fail)
$ f( at)
af n. (s) + w; As*),
ViEW ViEM N Vi N
fg < si +As uss
(3)
As* SE.
W; E (0,1),
NEN
Σ W n.p
where we introduce set of binary variables {w;) } to select the suitable agent attack.
5

---

Under review as a conference paper at ICLR 2023
Due to the existence of the new binary variables, problem 3 is much harder to solve. Therefore, we instead propose to solve proxy of as follows
min s.t.
d( 8 ffait
1111 f(8, (z)
at = n²' (s; + (9( (09;;0)A00 As'),
E Vi E V
(4)
Ls S si + us.
A P Se,
Vi N
S (st; 1
Wi WEN
where Wss;), 0) S - R" is a function parametrized by that takes current state as input and returns the weight to distribute the noise to each agent. Suppose represent (s; #) by neural network, we can rewrite the formulation
as (5) because the last constraint in (4) can be enforced by using a softmax activation in the neural network W (s;;))
min
d(f(s, (((s; Wi(s;: 0) As)).)), stai)) s.t. As E Cp.c.t
where the notation x ({s{ + (i (): As*), denotes the concatenation of action vectors from each agent to form a global action vector [T (s) + Wi (8::0) As*),...,,?* " _ + (s;; 0) As")]. As result, (5) can be efficiently solved by using PGD. We present the pseudo-code of the attack in Alg. 3 After K steps of PGD update, we define the i(n-j+1) as index of the j-th largest value
within W (s;; 0x) E RR" , i.e. Winss (st; 0x) 2 W Wi (8;;0k) 2 ... 2 Winh (5;;0k) 0x). Let Zj be the index set of top-j largest outputs of the W (st; Ox) network. The final perturbation returned by victim agent selection strategy will be As ((As))... (As)*) where (As)* 0 if i i In.
and (As)* 4 (Ask)* if E In..
Algorithm cMBA with victim agent selection at time-step
1: Initialization: Given 8t, sfaits T, ʃ. ny; initialize Aso; choose learning rate 1,, A 0.
2: For 0, , do
3: Compute dę ((1s) + Wi(se; 0) }.)).).
4: Compute $441 = f(s), az).
5: Update As Ask+1 proje A W
6: Update as 0x+1 0g A Vod((S141 sfait).
7: End For
(((m)." (n-nn))) such that Wi,.,) (s,, 0k) ... .2.2 W (8t, 0k). 9: Return As ((As),.... (As)a)) where (As)* (Ask)* if E In, and (As)² 0 other- wise.
8: Compute In..
Remark 3.2 For this attack, we assume each agent has access to the other agent's state to form the joint state St. If the set of victim agents is pre-selected, we do not need this assumption and the adversarial attack can be performed at each agent independently of others.
Remark 3.3 We can also use Alg. 3 in conjunction with Alg. Dio perform the attack using Alg. 7 on the set of victim agents returned by Alg. Our results for Experiment (IV) in Section $ indicate that using Alg. 3 and Alg. | together produces an even stronger cMBA attack: e.g. it can further decrease the team reward up to 267% more than using only Alg. in Ant (4x2) environment.
We highlight that combining cMBA with victim agent selection strategy can potentially make cMBA much stronger as shown in our comparison with other selection strategy such as random selection or greedy selection in Section Our numerical results show that selecting the appropriate set of victim agents at each time-step constantly outperform cMBA attack with random/greedy agent selection. For example, with victim agent selection strategy, cMBA further lower the team reward up to 267%, 486%. 30% more than when using cMBA on randomly selected, greedily selected, or fixed agent in Ant (4x2) environment.

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
We perform the attack on both multi-agent MuJoCo (MA-MuJoCo) environments (de Witt 2020) including Ant(4x2), HaltCheetah(2x33, HallCheetah(6xl), and Walker2d(2x3), The pair name(config) indicates the name of MuloCo environment along with the agent partition, where configuration of 2x3 means there are in total 2 agents and each agent has 3 actions. We provide more details on the construction of these agents in Appendix C We also demonstrate the effectiveness of the learned failure state approach using the multi-agent particle environment, denoted as MPE(3x5) (Lowe et al.) 2017 Mordatch & Abbeel (017) where we do not have expert knowledge of the failure state. We consider the following model-free baselines:
1. Uniform: the perturbation follows the Uniform distribution U(-s,E).
2. Gaussian: the perturbation follows the Normal distribution (0,
3. Lin et al. (2020) iFGSM: Since there is no other work performing adversarial attack for continuous action space in c-MARL, we adapt the approach in (Lin et al.) 2020) to form another baseline. In particular, we train an adversarial policy for one agent to minimize the total team reward while the remaining agents use the trained MARL policy. This adversarial policy is trained for million timesteps. We then use this trained policy to generate a "target" action and use iterative FGSM method (Kurakin et al. 2016, Goodfellow al. (014)) to generate the adversarial observation perturbation for the agents' input. Note that the adversarial policy trained on the same agent that is being attacked.
In our experiments, consider two variants of c-MBA follows.
1. c-MBA-F:
we perform c-MBA using an expert-defined failure state where we have the knowledge of the state definition the c-MARL environment. This variant used on the MA-MuJoCo benchmark there are available documentation (AI) on its state definition.
2. c-MBA-D: we perform c-MBA attack using failure state learned from the collected data as described Section 3.2
Due to space constraint, only present part of our results here and we defer the full results in Appendix D We evaluate c-MBA comprehensively with the following 5 experiments:
Experiment (I) model-free baselines vs model-based attack e-MBA using &o-- constrained perturbation: we compare c-MBA-F and c-MBA-D with other baselines when attacking individual agent under fox constraint.
Experiment (II) effectiveness of learned adaptive victim selection: we evaluate the performance of Alg. with other heuristicly selected variants of c-MBA. Our results show that in HalfCheetah environment, under ε 0.05 and Ezo budget constraint, the amount of team reward reduction of our learned agents selection scheme can be up to 80% more than the cases when using heuristic strategy to select victim agent.
Experiment (III) attacking multiple agents using model-free baselines model- based attack e-MBA with fo perturbation: we report results on attacking multiple agents simultaneously. This setting is not previously considered in (Lin et al. 2020).
Experiment (IV) model-free baselines vs model-based attack e-MBA in MPE(3x5)
environment: we compare c-MBA-D with other basclines under &oo and f constraint.
Experiment (V) -V)-model-fre
model-free baselines vs model-based attack c-MBA on é perturba-
tion: we compare c-MBA-I and c-MBA-D with other baselines when attacking individual agent under & constraint. The results of this experiment are presented in Appendix D Experiment (VI) adversarial attacks using dynamics model with various accuracy: we illustrate the performance of c-MBA when using dynamics model with various accu- racy. This experiment is deferred Appendix D
Experiment (I) model-free baselines vs model-based attack c-MBA on foo perturbation. In this experiment, we run the baseline attacks along with two variants our model-based attack on the four MA-MuJoCo environments with victim agent (n, 1) in coherence with (Lin) jet al., 2020).Fig. 2 illustrate the performance when we perform these attacks on each agent with different attack budget using norm. For fixed agent, model-based attack outperforms all
7

---

results show that Under review as a conference paper at ICLR 2023
Meen rewards - Ant (4x2), agent
Mean rewards HalfCheetah (6x1), agent o
3000
Mean
Un (222) FGM
2000
(M36-7 tous!
tous!
Moan Roward
Mean Reward
2000
1000-
1500
1000
-1000
0.15
Noise Level, [|ās'||
0.1
0.2
0.3
0.4
0.5
Noise Level, [|A+|]
Figure 2: c-MBA vs baselines when attacking one agent in 2 MA-MuJoCo environments Exp. (I).
the other baselines. In particular, our model-based attack yields much lower rewards under relatively low budget constraints (when E [0.05, 0.2) compared to other baselines. For instance, under budget level & 0.05, the amount of team reward reduction of our c-MBA attack is (on average) 66%, 503%, and 806% more than the three model-free (Lin et al. (2020) + iFGSM, Gaussian, and Uniform, respectively) in Ant(4x2) environment. We also observe that c-MBA-D either matches or performs better than c-MBA-F with the expert-defined failure state.
Experiment (II) - effectiveness learned adaptive victim selection. To evaluate our proposed victim-agent selection strategy, we consider the following variants of our model-based attack:
M-Aeeef agents): attack a fixed set of victim agents with Alg. We use e-MBA (best
fixed agents) to denote the best result among fixed agents.
2. crnnoorandom agents): uniformly randomly select victim agents to attack with Alg. I
3. (greed agents selection): for each time step, sweep all possible subsets of agents with size n,, and perform Alg. [ Select the final victim agents corresponding to the objec- tive value (distance between predicted observation and target failure observation).
4. C-MBA(learned agents selection): use Alg. 3 to attack the most vulnerable agents.
5. c-MBA(learned agents selection + Alg. 1): use Alg. to select the most vulnerable agents to attack then perform the attack with the selected agents with Alg. [
We conducted experiments on all 4 MA-MuJoCo environments and part of the results are plotted in Fig. 3. shows that (learned agents selection) and C-MBA Alg. 1) perform better than other variants in both environments. It is interesting to observe that c-MBA(learned agents selection + Alg. 1) is either comparable or better than c-aanele agents selection) which shows that running Alg. [using agents selected by Alg. 3 be beneficial. We show that the victim selection technique is really important as randomly choosing the agents to attack (i.e. c-MBA(random agents)) or even choosing agents greedily (i.e. c-MBA(greedy agents selection)) cannot form an effective attack and are in fact, even worse than the attacking on fixed agents (i.e c-MBA (best fixed agents)) in most cases. For example, in TalfCheetah(6x1) environment, under E 0.05 and Eoo budget constraint, the amount of team reward reduction of our learned agents selection scheme is 33%, 80%, 35% more than the cases when attacking fixed agents, random agents, or greedily selected agents, respectively.
agents selection
Experiment (III) attacking two agents using model-free baselines model-based attack c-MBA using foo constrained: We conduct experiments using model-free and model-based ap- proaches to simultaneously attack two agents in Ant(4x2) environment. Fig. 4 illustrate the per- formance of various attacks. We observe that our c-MBA attack outperforms other baselines. For instance, at ε = 0.025, the team reward reduction of c-MBA 68%, 522%, 713% more than the three model-free baselines. In addition, our c-MBA-D even outperforms c-MBA-F to achieve lower reward at all budget levels.
Experiment (IV) model-free baselines vs model-based attack c-MBA in MPE(3x5) environ- ment: We perform the same procedure as in Experiment (III) to attack different agents in the MPE(3x5) environments where attack two three agents at the same time. Since we do not have expert knowledge about the failure state in this environment, we compare c-MBA-D with other model-free baselines. The results depicted in Figure where c-MBA-D perform slightly better
8

---

Under review as a conference paper at ICLR 2023
2500
Our cMBA Variants- Walker2d (2x3)
Our cMBA Variants- - HalfCheetah (6x1)
3000
cWaoijod agent(
2000
(aEAlp agents susccenn
2500
(iBAlpuuly agenta weecien
Mean Reward
2.500
MM Alhamed oprta sele
hurred uprtt soliction
Mean Roward
2000
oltAA samde uprtt schelioe)
AMrAri ugrtt scliction 1:
1000
1500
1000
500
500
0
0.3 Noise Level. [|As 5c
0.3 Noise Level. ||As =e
Figure 3: Performance of c-MBA with different victim agent selection strategies Exp. (II)
Mean rewards- Ant (4x2), agent [0,2]
Mean rewards- - Ant (4x2), agent [1,2]
3000
3000
Guestin un al FGM
Guastin
Un 2000 FGMM
2000
(PE5S D1ear))
2000
(PE5S Dlean))
Mean Roward
Me Roward
1000
1000
0
1000
-1000
0.000
0.025
0.050
0.075 0.100
0.125
0.150
0.000
0.025
0.050
0.075
0.100 0.125 0.150
Noise Level. [|68
Noise Level, ||68
Figure 4: c-MBA vs baselines in Ant(4x2) when attacking two agents Exp. (III).
than other random noise baselines when attacking agents and significantly outperforms other base- lines when attacking three agents simultaneously. We note that (Lin et al., (2020) does not perform well in this experiment as we observe during training the adversarial policy that it could not lower the team reward effectively.
Mean rewards Particle, agent [0,2]
Mean rewards Particle, all agents
180
200
Unttum
Untom
160
(222)) IRO0M
175
Ui (2328) IROO 798 O teum
150
Mean Reward
120.
Reward
125-
100
100
80
75:
50
60
25:
D-
28.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Noise Level, [|a='|]_ sc
Noise Level, ||as sc
Figure 5: c-MBA vs baselines in MPE(3x5) when attacking two agents (3 leftmost) and three agents (rightmost) simultancously using foo constraint Exp. (IV).
5 CONCLUSIONS
In this paper, we propose a new attack algorithm named c-MBA for evaluating the robustness of c-MARL environment. Our c-MBA algorithm is the first model-hased attack to craft adversarial observation perturbations and we have shown that c-MBA outperforms existing model-free base- lines attack by a large margin under both multi-agent MuJoCo and multi-agent particle benchmarks. Unique to multi-agent setting, we also propose a new victim-selection strategy to select the most vulnerable victim agents given the attack budget, which has not been studied in prior work. We show that with the vietim-agent selection strategy, c-MBA attack becomes stronger and more ef- fective. We also propose the first data-driven approach to determine the failure state based on the pre-collected data without extra overhead making our attack more flexible.
9

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We present additional numerical results in Appendix C We also provide an example of how the state or adversarial state perturbation evolves in one of MA-MuJoCo environment. To promote reproducibility, we have uploaded the source code with full instructions to reproduce the results in this paper. We also add an example notebook visualize the results. If our paper gets accepted, will make our code publicly.
REFERENCES
Open AI.
Mujoco documentation. envi ronments s moco//
URL
htttpe:///w..gymllbbrrary
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. Generating natural language adversarial examples. arXiv preprint arXiv:1804.07998, 2018.
C. S. de Witt. Multi agent mujoco. Btteer//ggtuuu.cco//cchcrrorrrrd mu multiagentt_m mujjuuojoo 2020.
Gregory Dudek, Michael Jenkin, Evangelos Milios, and David Wilkes. A taxonomy for swarm robots. In Proceedings of 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems (IR0S'93), volume 1, pp. 441-447, IEEE, 1993.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv:1905. 10615, 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.663: 2014.
Yizheng Hu and Zhihua Zhang. Sparse adversarial attack multi-agent reinforcement learning. arXiv preprint arXiv:2205.09366 2022.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint urXiv:1702.02288. 2017.
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv: 1707.07328, 2017.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 8018-8025, 2020.
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. arXiv preprint arXiv:1705.06452, 2017.
Alexey Kurakin, Ian Goodfellow, Samy Bengio, et al. Adversarial examples in the physical world, 2016.
Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Leon-Garcia, and Nicolas Papernot. On the robustness of cooperative multi-agent reinforeement learning. 2020 IEEE Security and Privacy Workshops (SPW), pp. 62-68. IEEE, 2020.
Yen-Chen Lin, Zhang- Wei Hong, Yuan- Hong Liao, Meng- Shih, Ming- Yu Liu, and Min Sun. Tac- tics of adversarial attack on deep reinforcement learning agents. arXiv preprint arXivv 2017.
Yuan-Hong
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-commpetiiee environments. arXiv preprint arXiv: (00:::700000000 0220002275 2017.
10

---

Under review as a conference paper at ICLR 2023
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforeement learning. In International conference machine learning, pp. 1928-19331 PMLR, 2016.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations, arXiv preprint TrXivv/703. 1703.04908, 04908, 2017.
Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2003.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Frans A Olichoek and Christopher Amato. A concise introduction to decentralized POMDPs.
Springer, 2016.
Afshin OrodjlooyJadid and Davood Hajinezhad. review of cooperative multi-agent deep rein- forcement learning. arXiv preprint 1908v09998.03999 2019.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson. Z Berkay Celik, and Ananthram Swami. The limitations deep learning in adversarial settings. 2016 IEEE European sympo- shum security and privacy (EuroS&P), pp. 372-387. IEEE, 2016.
Bei Peng. Tabish Rashid, Christian A Schroeder de Witt, Pierre-Alexandre Kamienny, Philip HS Torr, Wendelin Böhmer, and Shimon Whiteson. Faemac: Factored multi-agent centralised policy gradients. arxiv preprint wrXiv:2003.06709 2020.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/ 1e///1922004433 2019.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312.6616 2013.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Tsui- Wei Weng, Krishnamurthy Dj Dvijotham, Jonathan Uesato, Kai Xiao, Sven Gowal, Robert Stanforth, and Pushmeet Kohli. Toward evaluating robustness of deep reinforcement learning with continuous control. International Conference on Learning Representations, 2019.
Wei Emma Zhang, Quan Z. Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial attacks on deep-learning models in natural language processing: A survey, ACM Transactions Intelligent Systems and Technology (TIST), 1 113)::1-1,, 2020.

---

Under review as a conference paper at ICLR 2023
Appendix
A DETAILS ON HOW TO SOLVE (1) EFFICIENTLY
Problem ① can be efficiently solved by proximal-gren caaeanal--ggdien--baaaa based methods (Nesterov 2003). Firstly, by substituting and 8141 with their definitions,
① is equivalent to
min d(f(st, (st + x)), sfait)
s.t. I Cp,,tt
where Cp.r.t == fxr = (x², ,.. xn) ls us st, ||=0|1, S e for E V. and x² O for i ViJ. When and are represented by neural network and we choose the distance func- tion as d(a, b) = |la 6112 (6) is a constrained nonconvex problem, then We can use the projected gradient descent (PGD) algorithm (Nesterov) 2003) to solve (6). The PGD iteration to update Ty at iteration l starting from zo can be described as
1k++ 1 proic,...e [a* - V (8 n (st + x)), sfaa]
where proic,... (-) is the projection to the convex set Cp.et and is the learning rate. The projection is simple to calculate since Cp.s,e c is the intersection of a Ep -n-mm ball and boxed constraint.
B DETAILS ON TRAINING DYNAMICS MODEL
Algorithm Training dynamics model
1: Initialization: Given pre-trained policy Thy and a random policy nyα' initialize dynamics model
parameter do-
2: Form D = Deroin Uaranoo by collecting set of transitions Dtrain and Drandom using policy Ter and Tyd. respectively.
3: For h = 1. do
Ø++1 = Graadienttaaeeetuucttee....
4: End For
For each environment, we collect million transitions using the trained MARL policy mer and random policy Zyd- We partition the collected data into train and test with ratio 90-10. The dynamics model is represented by fully-connected neural network. The network contains 4 hidden layers with 1000 neurons each layer and the activation function for each hidden layer ReLUU. We train the network for 100 epochs using Adam W (Loshchilov & Hutter 2017) with early stopping where the learning rate tuned in the set {0.001, 0.0005, 0.0001, 0.00005, 0.00001} } to obtain the model with the best test mean squared error. Please refer our submitted code for further details.
C MORE DETAILS ON EXPERIMENT SETUP IN SECTION 4
Experiment setup. We use MADDPG (Lwwe et al. 2017) train MARL agents for the four MA- MuloCo environments as well as the multi-agent particle environment. Using the trained agents, we collect datasets containing
million transitions train the dynamics model for each environment. The dynamics model a fully connected neural network with three hidden layers of 1000 neurons. We also fully-connected neural network for W W(st, 0) in (4) with two hidden layers of 200
12

---

Under review as a conference paper at ICLR 2023
the optimizer and select the best learning (the best learning rate is the one achieving lowest prediction error test set of 80, 000 samples). For model-baser attack,
neurons. We use Adam (Loshchilov & Hutter 2017
rate from {1, 5) x {10-6, 10 1000100000
10
run PGD algorithm for
K = 30 steps to solve 6 and (3). We perform each attack over 16 episodes then average the rewards. We also illustrate the standard deviation of rewards using the shaded area in the plots.
Agent partitioning for MA-MuJoCo environments. Each of the original MuJoCo agent in the single-agent setting contains multiple joints and the way these joints are partitioned will lead to different multi-agent configurations. These configurations are described as follows:
Walker (2x3) environment: this environment has 6 joints, 3 for each leg and the whole
agent is divided into group of joints {1, 2. and {4, 6) representing two legs (Peng ct al. 2020 Fig. 4F).
HalfCheetah(2x3) environment: there are two agents, each represents a front or rear leg with joints 2,, and (4, 5, 6) (Pegg et al. 2020 Fig. 4C).
(6x environment: each agent represents each of the total 6 joints (Peng al. 2020 Fig. 4D).
Ant(4x2) environment: each agent controls one leg with two joints out of legs (Peng et al. 2020 Fig. 4J).
Specifying target observation for each environment. To perform our model based attack, we need to specify target observation that potentially worsens the total reward. Currently, do not have a general procedure to specify this target observation. We specify the target observations based on prior knowledge about the environments as follows. In multi-agent MuJoCo environments, each agent has access to its own observation of the agent consisting the position-related and velocity- related information. The position-relaee related information includes part of z, y, 2 coordinates and the quarternion that represents the orientation of the agent. The velocity-related information contains global linear velocities and angular velocities for each joint MuJoCo agent. We refer the reader to (Todorov et al. 2012) for more information about each MuloCo environment. Now we describe the design of this target observation for each environment as follows:
Walker(2x3) environment: Since the episode ends whenever the agent falls, i.c. the coordinate falls below certain threshold. In this environment, the target observation has value of for the index that corresponds the coordinate of the MuJoCo agent (index 0).
HalfCheetah(2x33 and HalfCheetah( environments: As the goal is to make agent moves fast possible, we the value at index corresponding the linear velocity to 0 (index 8).
Ant(4x2) environment: As the agent can move freely in a 2D-plan, we set the index corre- sponding to the x, y linear velocities to 0 (indices 13 and 14).
D ADDITIONAL EXPERIMENTS
In this section, we present experimental results in addition ones presented in Section 4
Full results for Experiment (I) model-free baselines vs model-based attack c-MBA on foo perturbation We first show the full results on running c-MBA and three other baselines MA-MuloCo environments under -nomm budget constraint in Fig. 6 c-MBA performs the best in all cases and significantly outperforms other baselines in majority of the cases.
To better visualize the performance difference. Fig. illustrates the environment with and without attacks captured at different time-steps. From Fig. [ our model-based attack is able to make the MuloCo agent flip, which terminates the episode at the 409-th timestep. The results with episode length and total rewards for each variant are: No attack (00ck, 2645.64), Uniform(1000 1663.5), Gaussian(8911 1548.49), Lin al. (2020) + iFGSM(1000, 902.53), and Ours(738, 366.55).
Full results of Experiment (II) effectiveness learned adaptive victim selection. We con- ducted experiments all 4 MA-MuJoCo environments and the results plotted in Fig. It
13

---

Under review as a conference paper at ICLR 2023
Meen rewards. (4x2), agent 0
Meen rew (4x2), agent
Mean rewards Art (4z2), agent
5000
3005
3002
2000
ORAN cool)
2000
(9AA rork)
2009-
Maun Banerd
Meaan Teeoerr
1000
-1009
-3000
-1009
0.10 Nolse [|oF]]
Nolse Level [|AF|]
Noise
[|AF
Moon rewardss (4x2), agent
5000
Moan rewards #CChetahh (233), agent
5000
Mean rewards Ha HatCheetah (2x3), agent
caffare
Laffee
2000
2200.
2000
1000
1000
-1000
3.00
0,55
0,30
Noke Lovel [|62 Il.
||
3000
Mean rewards- HCheetah (6x1), agent
3000 Mean rewerds- #Cheelsh (8x1), agent
3000
Meen rewards He HCheetsh (8x3), agant
25.0
2500
crand (oul))
2500
2750
orand (oal))
Mein ROMME
2000
Mean Naan/aaa
2000
1500
2000
1000
1.500
1755
1500
8.0
500
100g
0.2
||0e|]
||
[|br|]
Mean rewards Ha #CChettah (6x11, agent
Moan rewards Ha IfCheetthh (6x1), agent
Moan
Moan
Mean rewards Ha #Chhetah (6x1), agent
2500
2530
busk
2600
18/10
2600
Newr Belave 28
2000
2400
1500
15500
2200
1000
2000
1000
500
1600
0.3
022
0.1
0.2
Lovel, ||as| |.
Level, ||ss
Nobsc Lovel. [|As
Mean rewards
We Buer2d (2x3)), agent O
3000
Mean rewards We Iker2d (2x33) agent
2500
2000 POM
2500
2030
2000
2000
3500
1500
2000
Mean
1000
0.0
0.2
0.5
0.4
0.5
0.0
0.1
0.2
0.4
0.5
Nobu Larvel [|Ax]]
Noou Larvel, ||oa 11-
Figure 6: c-MBA vs basclines when attacking one agent in 4 MA-MuJoCo environments Exp. (I).
shows that c-MBA agents selection) and C-MBA agents selection + Alg. 1) ap- pear to be better than the random or greedy strategy and is comparable with the randomly selected one in HalfCheetah(2x3) environment. It is interesting to observe that A.MBA(leareed agents se- lection + Alg. 1) is either comparable or better than c-MBA(learned agents selection) which shows that running Alg. | Jusing agents selected by Alg. Jean be beneficial. We show that the victim selec- tion technique is really important as randomly choosing the agents to attack (i.e. C-MBA(random agents)) or even choosing agents greedily (i.e. c-MBA(random agents)) cannot form an effec- tive attack and are in fact, even worse than the attacking on fixed agents (i.e c-MBA (best fixed agents)) in most cases. For example, our results show that in HalfCheetah(6x1] environment, un- der ε 0.05 and for budget constraint, the amount of team reward reduction of our learned agents
14

---

Under review as a conference paper at ICLR 2023
selection scheme is 33%, 80%, 35% more than the cases when attacking fixed agents, random agents, or greedily selected agents, respectively.
2000
3003
2000
2530
$2000
1000
550
3830
Lord, Lafi
Lendl
Figure 7: Performance c-MBA with different victim agent selection strategies Exp. (II).
Full results of Experiment (III) - attacking two agents using model-free baselines vs model- based attack c-MBA using & constrained: We conduct experiments using model-free and model- based approaches to simultaneously attack two agents in Ant(4x2) environment. Fig. |illustrate the performance of various attacks. We observe that our c-MBA attack outperforms other baselines in out of 6 settings, and especially it's able to achieve low reward (close to or below 0) at lower attack budget levels. For example, at & 0.025, the team reward reduction of c-MBA is 68%, 522%, 7 713% more than the three model-free baselines.
dek Medk
1000
12:0
1290
125 3050 LID
Li Ea E.
beurliet D-1.2
Mean rewards Ant (4x2), agent [1,3]
Mean rewards Ant (4x2], agent [2.3]
3000
3000
2000
20:00
Moan Nessors
2020
-1000
-1000
3,000
Nobie Level [
Noke Level [67)
Figure 8: c-MBA vs baselines in Ant(4x2) when attacking two agents- Exp. (III.
Full resuts of Experiment (IV)- Model-free baselines vs model-based attack c-MBA in MPE(3x5) environment: We follow the same procedure as in Experiment (III) to attack differ- ent agents in the MPE(3x5) environments where we attack two or three agents at the same time. Since we do not have expert knowledge about the failure state in this environment, c-MBA-D with other model-free baselines. The results are depicted in Figure 9 where e-MBA-D perform slightly better than other random noise baselines when attacking 2 agents and significantly outperforms other baselines when attacking three agents simultaneously. We note that (Lin et al, 2020) does not perform well in this experiment as observe during training the adversarial policy that it could not lower the team reward effectively.
compare
Meurn remardd Partila
Maur rwwands Partaca, agend
Plear EROANES PlotUc agend
Moas CRNGUSS PL-BICA Aggets
300
375
332
O
Ho'ee Level Loft
Lovel Foll-ez
Lovel, Eofl-.e
Figure 9: c-MBA vs baselines in MPE(3x5) when attacking two agents (3 leftmost) and three agents (rightmost) simultancously using Loo constraint Exp. (IV).
Experiment (V): model-free baselines vs model-based attack c-MBA on L perturbation. In addition to the norm budget constraint, also evaluate adversarial attacks using the foomr
15

---

Under review as a conference paper at ICLR 2023
constraint. Note that using 4n-nrrm for budget constraint is more challenging the attack needs to distribute the perturbation across all observations while in the -nom the computation of pertur- bation for individual observation is independent. Fig. illlstrrat the effect of different attacks on HalfCheetah(6xI) and Walker2d(233 environments, respectively. Our c-MBA is able to outper- form other approaches in almost all settings. In HalfCheetah(6xl), using constraint, the amount of total team reward reduced by c-MBA variants is up to 156%, 37%, and 42% more than Lin et al. (2020) iFGSM, Gaussian, and Uniform baselines, respectively.
1.0 under & budget
we pleas
320
330
25:2
2500
20:0
290
3009
120
Suce Lemt ber
(IM]ES Meas coward Welbehh (AAt)
Plen menerde
comeett Walherhh Daaks
2004
3850
3800
2204
2004
1000
1000
23 25
24
13
20
25 24
Figure 10: c-MBA vs baselines in HallChectah(6xI) and Walker2d(2x3) Exp. (V).
Experiment (VI) - adversarial attacks using dynamics model with various accuracy. We com- pare our attack when using trained dynamics model with different mean-squared error (MSE). We use Ant(4x2) environment and consider the following three dynamics models:
(1Mel(1M, 100 epochs) is trained using I million samples for 100 epochs and select the model with the best performance.
(1,, epoch) is trained using million samples for only I epoch.
(.2M, 1 epoch) is trained using less number of samples to only 200k and we train it for only epoch.
These models are evaluated on a predefined test set consisting of IM samples. The test MSE of these models are 0.33, 0.69, and 0.79, respectively, with the initial (w/o training) test MSE of 1.241. Fig. II depicts the attacks using these three models on the same agent in Ant(4x2) environment using IT-II budget constraint. Interestingly, the dynamics model trained with only 0.2M samples for epoch can achieve comparable performance with the other two models using 5 x more samples.
In addition to the visualization in Fig. 1 we investigate how the state values change during these attacks. Fig. 12] presents different recordings of state values under adversarial attacks compared to no attack. Consider state index 8, which represents the horizontal velocity of the agent. For the HalfCheetah(2x3) environment, as the goal is to make the agent move forward as fast as possible, we expect the reward to be proportional to this state value. From Fig. [2] all three attacks have fairly sparse red fractions across time-steps, which result in much lower reward compared to the no-attack setting. Among the three attacks, our model-based ones appear to have the most sparse red fractions leading to the lowest rewards. In addition, the model- badel-based attack appears to show its advantage in environments with more agents as our approach results in lower rewards under a smaller budget as seen in HalfCheetah(6xI) and Ant(4x2) environments. Additionally, we record the evolution of noise values and the results are shown Fig. 13
In summary, our c-MBA attack is able to shows its advantage in all tested multi-agent environment where it achieves lower reward with smaller budget level. Moreover, c-MBA with the victim agent selection has been shown to constantly performs better than the original c-MBA variant as seen in Figure 3
16

---

Under review as a conference paper at ICLR 2023
Dynamics model accuracy Ant (4x2), agent
Dynamics model accuracy Ant (4x2), agent 0
3000
28. 120 eoech
3000
1H. 300 spach
IK esoch
1H spoch
2500
9.2M 1 esoch
2500
0.2M spuch
Mean Reward
2000
Reward
2000
1500
1500
1000
1000
500
500
0
-500
-500
0.00
0.05
0.10 Noise Level, ||si|]
0.15
0.20
0
2
6
8
10
Noise Level, ||AS|]
Figure 11: Effect of using three dynamics models in Ant(4x2) when attacking Agent out of four agents in Ant(4x2) Exp. (VI).
HallCheetah (2x3) No attack (Agent 0), Reward: 4536.39
HallCheetah (2x3) Unitfomm noise, [|Mt s02 (Agent 0), Reward: 2209.57
353
HatfCheetah (2x)) Gaursian noise, [Mel] 200.2 (Agent o), Reward: 1782.00
HairCheetah (2x3) Lin et + IFGSM, M, [|tc] 0.1 (Agent o), Reward: 1575.39
HaifCheetah (2x3) CMBA (ours), Bedile = 0.2 (Agentt 0), Reward: 1112.37
140
Figure 12: Recordings state values in an episode under different attacks one out of two agents (Agent 0) in HallCheetah 2x3) environment.
17

---

Under review as a conference paper at ICLR 2023
HalfCheetah (2x3) No attack (Agent Reward: 4536.39
5 0 5 5
100
HalfCheetah (2x3) Uniform moise, ||Ad| =t (Agent Reward: 2209.55
HalfChentah (2x3) Gauesian nolie, [ós]]
(Agent O), Reward: 1782.00
HairCheetah (2x3) Lin et
IFGSM. LASIL x 0.1 (Agent 0), Reward: 1575.38 38
(2x3) (ours), lióHli- 2.2 (Agent O), Reward:
Figure 13: Recordings of noise values in episode under different attacks Agent in HalfChee-
tah( 2x3) environment.
18


Reviews:
[{"review_id": "u8T9SxDVTR", "reviewer": "Reviewer_giiS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author propose a novel model-based method to attack the team performance in cooperative Multi-agent reinforcement learning. By changing the input state of the victim agent, the goal is to reduce the reward function of the whole team. In addition, the authors also propose a  victim-agent selection strategy to choose the most vulnerable agent to attack. The evaluation results shows that the proposed method can consistently outperforms other baselines in all tested environments.", "summary_of_the_review": "Overall, I think secure MARL is an interesting and important topic. The paper did bring some novelty to the field -- proposing an algorithm to select the victim agent which is the most vulnerable. However, the author should justify the applicability of their approaches, as mentioned in the Strength And Weaknesses section.", "strength_and_weaknesses": "Strong point:\n+ This paper tries to design a model-based attacking algorithm for MARL, this problem has not been explored too much before.\n+ A novel victim selection algorithm is proposed, which is also novel.\n+ Evaluation results shows that the performance is great.\n\nWeak point:\n- It seems that MARL is very easy to attack, according to Figure 5 and 6. This is also mentioned in the early work (Lin et al. 2020). The author should justify if their approach is still necessary given its relevant high implementation cost.\n- The other problem is training this attack method requires the attacker to have access to the RL environment and model parameters, which may seriously limited the application of this approach. \n- Compared with the early work (Lin et al. 2020) The novelty is not that obvious.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "cRIMsKHKxA", "reviewer": "Reviewer_Rtpm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a model-based adversarial attack framework (c-MBA) on continuous action spaces of cooperative MARL. The core idea is to find the best perturbations to the states of the agents' policy networks using a transition model such that the state after the transition will be the closest to the failure state. The idea of finding the most vulnerable victim agents is interesting.", "summary_of_the_review": "This paper proposes a model-based adversarial attack framework method to evaluate the robustness of multi-agent cooperative RL methods. The idea is novel and interesting. But some parts of the method designs can be further improved. The reviewer also thinks that there exist more simple but effective baselines. So, the reviewer currently recommends a weak reject.", "strength_and_weaknesses": "**Strength**\n  * The research topic is interesting and the paper is well-motivated and clearly written.\n\n**Weaknesses**\n  * The definition of the failure states in Section 3.2 is too simple. Since the target of RL is to maximize the expected cumulative reward, simply minimizing the immediate reward the agent can obtain is not a good practice and may get in suboptimal solutions. Instead, we should find a target state which could minimize the Q-value, i.e., the expected cumulative reward, of the RL agent.\n  * I think there exist more simple but effective baselines. Since the authors perform adversarial attacks on agents trained using MADDPG. The most straightforward way to apply adversarial attacks to MADDPG is:\n    * $\\min _{\\Delta s=\\left(\\Delta s^1, \\cdots, \\Delta s^n\\right)} Q(\\mathbf{s}, \\mathbf{\\hat{a}})$, s.t., $\\mathbf{s}=\\text{concat}(s_1, \\ldots, s_N)$ and $\\mathbf{a}=\\text{concat}(\\hat{a_1}, \\ldots, \\hat{a_N})$, where $\\hat{a_i}=\\pi^i\\left(s_i+\\Delta s^i\\right), \\forall i \\in \\mathcal{N}$ and $Q(\\mathbf{s}, \\mathbf{\\hat{a}})$ is the centralized critic Q-function of MADDPG.\n    * In other words, the idea is to modify the input state to the policy network of the agents such that the $Q(\\mathbf{s}, \\mathbf{\\hat{a}})$ value of MADDPG can be minimized. This optimization problem can be implemented via SGD methods. As this method is very simple to implement, I'd like to see how well does this simple algorithm perform compared with the proposed (a little more complicated) one.\n  * What's the performance of the proposed method on some more complicated MARL benchmarks, e.g., SMAC?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XA7wyDhOmq", "reviewer": "Reviewer_SrTs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper mainly studies a method to evaluate the robustness of c-MARL agents. The main contributions of this paper are using model-based appoach rather than the model-free approach. The most critical benefit of this approach is that the learned dynamic model can be used to generate the next failure state. In addition, the authors also formulate a mixed integer problem to locate the vulnerable agent (which is named as victim-agent in the paper) to inject stronger attacks to damage the whole multi-agent system. ", "summary_of_the_review": "This paper describes a methodology to attack c-MARL with model-based approach, but there are still multiple concerns about the proposed approach which need to be addressed.", "strength_and_weaknesses": "## Strength\n1. The description of the proposed method is concrete and comparison to the related work is clear.\n2. The experimental design is comparatively fair to demonstrate the effectiveness of the proposed method.\n\n\n## Weaknesses\n1. There are multiple editing errors which should be addressed in the modified version.\n2. The authors claim that in the setting of Dec-POMDP each agent receive an individual reward. However, as far as I know there should be only one team reward. It should be a factual error.\n3. The author also claim that \"The requirement on training an adversarial policy is impractical and expensive compared to learning the dynamics model.\", which has not been clearly verified. For example, how is the sample efficiency between the off-policy methods and the model-based method?\n4. The authors claim that \"In our experiments, we notice that the quality of the dynamics model does not significantly affect the result as seen in Figure 11.\". To me, it is so wierd. I wonder whether learning a dynamic model is a key factor in this method. For example, I wish the authors can show whether a linear model with randomized parameters that characterizes the relationship between state-action pair and next state can still achieve a comparatively good result.\n5. About Eq. (5), if I understand it correctly, the $W_{i}$ obtained is only for ranking the victim-agents and the state perturbations are not penalised with $W_{i}$. In other words, the coefficient prior to each perturbation is either 0 or 1. If it is correct, this is inconsisitent with the formulation in Eq. (4). I wish the authors can give more explanation for it.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LMtrZpjJYv5", "reviewer": "Reviewer_oW7x", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method that attack cooperative multi-agent reinforcement learning. The major features are\n- Model-based: predict future states and find those with low rewards.\n- Continuous action space: distinguish the paper from those on discrete action space.", "summary_of_the_review": "The research direction is interesting and promising, but the proposed method relies on less realistic assumptions and models.", "strength_and_weaknesses": "The main concern of the reviewer is about the novelty and applicability of the proposed method.\n\n- The proposed attacking method depends on a pre-trained dynamics model. Training such a model may be costly in multi-agent settings.\n\n- The most frustrating point about the proposed method is that the target state needs to be collected, which largely limits the applicability of the proposed method to sparse reward settings and tasks with large action-oberversation space.\n\n- The proposed method has access to the policy of all agents. Such a white-box attacking method is less applicable.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "024kOHKzj1", "reviewer": "Reviewer_Hx5n", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new attack method, called c-MBA, for evaluating the robustness of cooperative multi-agent reinforcement learning (c-MARL) environment. c-MBA algorithm is a model-based attack to craft adversarial observation perturbations. The authors empirically demonstrate c-MBA’s superior performances against model-free baselines in two domains: multi-agent MuJoCo and particle benchmarks.", "summary_of_the_review": "This is an interesting paper where the authors propose model-based approach for adversarial attacks on c-MARL settings with continuous actions (for the first time). It includes extensive empirical analysis and ablation studies. I think the paper could also use discrete action environments and compare c-MBA directly with other baselines (designed specifically for those settings).", "strength_and_weaknesses": "## Strengths\n\n- I think the problem and approach are interesting and novel. A model-based approach for adversarial attacks on c-MARL settings seems promising. Furthermore, leveraging the unique setting in MARL to select the most vulnerable agent to attack is novel and intuitive.\n- The authors present numerous experiments to demonstrate the superior performance of their method against simple and model-free baselines for adversarial attacks.\n\n## Weaknesses\n\n- I understand the method is designed for continuous action tasks. I believe it should, with a straightforward change, also work on discrete action settings. If so, it would be beneficial to apply it to SMAC and compare it directly with Lin et al. (2020). I think this would strengthen the arguments made in the paper.\n- Applying it to problems would a larger number of agents would also strengthen the paper.\n\nNote: My score is not final and can change (in either direction) based on the authors' responses or the feedback of other reviewers.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject