Year: 2021

Abstract:
In recent years, a proliferation of methods were developed for multi-agent reinforcement learning (MARL). In this paper, we focus on evaluating the robustness of MARL agents in continuous control tasks. In particular, we propose the first model-based approach to perform adversarial attacks for continuous MARL. The attack aims at adversarially perturbing the states of agent(s) to misguide them to take damaging actions that lower team rewards. A deep neural network is trained to represent the dynamics of the environment. We then solve an optimization problem with the learned dynamics model to yield small perturbations. In addition, we discuss several strategies to optimally select adversary agents for the attack. Numerical experiments on multi-agent Mujoco tasks verify the effectiveness of our proposed approach.

heading: 1 INTRODUCTION
text: Deep neural networks are known to be vulnerable to adversarial examples, where a small and often imperceptible adversarial perturbation can easily fool the state-of-the-art deep neural network classifiers (Szegedy et al., 2013; Nguyen et al., 2015; Goodfellow et al., 2014; Papernot et al., 2016). Since then, a wide variety of deep learning tasks have been shown to also be vulnerable to adversarial attacks, ranging from various computer vision tasks to natural language processing tasks (Jia & Liang, 2017; Zhang et al., 2020b; Jin et al., 2020; Alzantot et al., 2018).
Perhaps unsurprisingly, deep reinforcement learning (DRL) agents are also vulnerable to adversarial attacks, as first shown in Huang et al. (2017) for atari games DRL agents. In Huang et al. (2017), the authors study the effectiveness of adversarial examples on a policy network trained on Atari games under the situation where the attacker has access to the neural network of the target policy. In Lin et al. (2017), the authors further investigate a strategically-time attack by attacking trained agents on Atari games at a subset of the time step. Meanwhile, Kos & Song (2017) use the fast gradient sign method (FGSM) to generate adversarial perturbation on the A3C algorithm (Mnih et al., 2016) and explore training with random noise and FGSM perturbation to improve resilience against adversarial examples. While the above research endeavors focused on actions that take discrete values, another line of research tackles a more challenging problem on DRL with continuous action spaces (Weng et al., 2019; Gleave et al., 2019). Specifically, Weng et al. (2019) consider a two-step algorithm which determines adversarial perturbation to be closer to a target state using a learnt dynamics model, and Gleave et al. (2019) propose a physically realistic thread model and demonstrates the existence of adversarial policies in zero-sum simulated robotics games.
While most of the existing DRL attack algorithms focus on the single DRL agent setting1, in this work we propose to study the vulnerability of multi-agent DRL, which has been widely applied in many safety-critical real-world applications including swarm robotics (Dudek et al., 1993), electricity distribution, and traffic control (OroojlooyJadid & Hajinezhad, 2019). In particular, we focus on the collaborative multi-agent reinforcement learning (c-MARL) setting, where a group of agents is trained to generate joint actions to maximize certain (team) reward function. We note that c-MARL is a more challenging yet interesting setting than the single DRL agent setting, as the interactions between agents commands consideration of additional layers of complications.
Contribution. Our contribution can be summarized as follows: 1We are aware of only one recent work (Lin et al., 2020) on attacking c-MARL agents with discrete action space, rather than continuous action space considered in our work.
• We propose a new adversarial attack framework on MARL with continuous action space, where we name it cMBA (model-based attack on c-MARL). The attack comprises two steps: learning a representation of the environment dynamics and then determining proper observation perturbations based on the learned dynamics model.
• We formulate the process of selecting an victim agent as a mixed-integer programming problem and propose an approximate formulation that can be efficiently solved by a firstorder method.
• We evaluate our attack framework with two baselines and four multi-agent MuJoCo tasks (Peng et al., 2020), with number of agents ranging from 2 to 6, to verify the effectiveness of our approach. Our model-based attack consistently outperforms the two baselines in all tested environments.
Paper outline. Section 2 discusses related works in adversarial attacks for MARL along with the problem settings. We describe the main attack framework cMBA in Section 3, including the training of the dynamics model and solving the sub-optimization problem. In Section 3.3, we introduce a mixed-integer program to find the optimal set of victim agents and propose an approximate formulation that can be efficiently solved using a first-order optimization method. Section 4 presents the evaluation of our approach on several multi-agent MuJoCo environments. Therein, we also study the performance of different variants of our model-based attack as discussed in Section 3.3.

heading: 2 RELATED WORK
text: Adversarial attacks for DRL agents. In the c-MARL setting, although there exists a significant literature focusing on the adversarial training of MARL agents (Phan et al., 2020; 2021; Zhang et al., 2020a), yet a very few of them addresses the scenario of adversarial attacks in c-MARL. Lin et al. (2020) propose a two-step attack on a single agent in a c-MARL environment, where they extend existing methods to generate adversarial examples guided by a learned adversarial policy. The attack in Lin et al. (2020) is evaluated under the StarCraft Multi-Agent Challenge (SMAC) environment (Samvelyan et al., 2019) where the action spaces are discrete. To the best of our knowledge, there has not been work considering adversarial attacks on the c-MARL setting with continuous action spaces. The closest approach to ours is the enchanting attack in Lin et al. (2017) and the modelbased attack in Weng et al. (2019); however, these two works focused on single DRL agent setting. The enchanting attack in Lin et al. (2017) uses a video prediction model to predict the future state and finds perturbation that minimizes a distance function between the predicted future state and a target state. Similarly, Weng et al. (2019) consider two formulations for observation and action perturbation that incorporates a learned dynamic model for future state prediction. In addition to the model-based approach, in this paper, we consider the selection of victim agents, which is unique to the multi-agent DRL setting.
Training c-MARL agents. One simple approach to train c-MARL agents is to let each agent learn their own policy independently as in Independent Q-Learning (IQL) (Tan, 1993; Tampuu et al., 2017). However, this training strategy does not capture the interaction between agents. Alternatively, one can follow a common paradigm called centralized training with decentralized execution (CTDE) (Oliehoek et al., 2008; Oliehoek & Amato, 2016; Kraemer & Banerjee, 2016). Gupta et al. (2017) extend DQN (Mnih et al., 2015), TRPO (Schulman et al., 2015), and DDPG (Lillicrap et al., 2015) to c-MARL, where a parameter sharing neural network policy is trained in a centralized fashion but still allows decentralized behavior across agents. Similar to this, COMA (Foerster et al., 2018) is an actor-critic method that uses a centralized critic to estimate counterfactual advantage for decentralized policies. While COMA uses a single critic for all agents, MADDPG (Lowe et al., 2017) learns a centralized critic for each agent and evaluates on continuous control. Instead of having fully centralized critic, DOP (Wang et al., 2020b) and FACMAC (Peng et al., 2020) apply the value decomposition idea (Sunehag et al., 2017; Rashid et al., 2018; 2020; Son et al., 2019; Wang et al., 2020a) into multi-agent actor-critic by having a centralized but factorized critic.

heading: 3 CMBA: MODEL-BASED ATTACK FOR C-MARL

heading: 3.1 PROBLEM SETTING
text: We consider multi-agent tasks with continuous action spaces modeled as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) (Oliehoek & Amato, 2016). A Dec-POMDP has a finite set of agents N ≡ {1, · · · , n} associated with a set of states S describing global states, a set of continuous actions Ai, and a set of individual partial observation Oi for each agent i ∈ N . Given the current observation oit ∈ Oi, the action ait ∈ Ai is selected by a parametrized policy πθi : Oi → Ai. The next state agent i moves to is determined by the state transition function Pi : S × Ai → S . After that, agent i receives a reward rit calculated from a reward function Ri : S × Ai → R and observes a local observation oit+1 ∈ Oi correlated with the global state st+1 ∈ S. In addition, Dec-POMDP is associated with an initial state distribution P0 and a discount factor γ. In this setting, each agent tries to find the policy that maximizes its own total expected return Ri = ∑T t=0 γ trit where T is the sampling horizon.

heading: 3.2 PROBLEM FORMULATION
text: Our goal is to generate adversarial perturbations imposed to the victim agents’ input (state) in order to deteriorate the total team reward. The added noise encourages the victim agents’ observation to be close to a desired adversarial state corresponding to low reward. To avoid sampling from the environment, we use a pre-trained model that learns the dynamics of the environment to predict the next state from the perturbed observation and current action, then find the suitable noise that minimizes the distance between the predicted next state and a predefined target state. This process can be formulated as an optimization problem as follows.
Formally, we consider a multi-agent setting with |N | = n agents, each agent i ∈ N observes state sit locally and takes action a i t. Let st = (s 1 t , · · · , snt ) ∈ S be the global state at time step t which is concatenated from local states sit for each agent i = 1, · · · , n. We also denote the joint action at = (a 1 t , · · · , ant ) concatenated from each agent’s action ait. Let Lt ∈ N be the set of victim agents at time step t, i.e. the set of agents that can be attacked. Let fd : S × A → S be a parametrized function that approximates the dynamics of the environment, where A is the set of concatenated actions, one from each Ai. We assume that a pre-trained MARL policy π(·) is given. Let starget be the target state which can lead to poor performance to the agent. We denote as an upper bound on budget constraint w.r.t some p-norm ‖·‖p. The state perturbation ∆s = (∆s1, · · · ,∆sn) (we suppress the dependence on t of ∆s to avoid overloading the notation) to st is the solution to the following problem
min ∆s=(∆s1,··· ,∆sn) d(ŝt+1, starget)
s.t. ŝt+1 = fd(st, at) ait = π(s i t + ∆s
i) ∆si = 0, ∀ i /∈ Lt `S ≤ st + ∆s ≤ uS ‖∆s‖p ≤
(1)
where 0 is a zero vector, and the observation space is constrained within the (vectorized) intervals between `S and uS .
Let us first provide some insights for the formulation (1). For each agent, using the trained policy π, we can compute the corresponding action ait given its (possibly perturbed) local state s i t or s i t+∆s
i. From the concatenated state-action pair (st, at), we can predict the next state ŝt+1 via the learned dynamics model fd. Then by minimizing the distance between the predicted next state and the target state under the budget constraint, we are forcing the agent to move closer to a damaging state in the next transition. We also note that the last equality constraint indicates that we only allow perturbing agents within the victim set Lt. Problem (1) can be efficiently solved by proximal-gradient-based methods. Firstly, by substituting the definition of at into ŝt+1, (1) is equivalent to
min x d(fd(st, π(st + x)), starget)
s.t. x ∈ Cpt (2)
where Cpt := {x = (x1, · · · , xn) : ‖x‖p ≤ , `S − st ≤ x ≤ uS − st and xi = 0 for i /∈ Lt}. If we choose the distance function as d(a, b) = ‖a− b‖2, we can use the projected gradient descent (PGD) algorithm (Nesterov, 2003) to solve (2). The PGD iteration to update yk at iteration k can be described as
yk+1 = projCpt [yk − η∇yd(fd(st, π(st + y)), starget)] (3)
where projCpt (·) is the projection to the convex set C p t and η is the learning rate. The projection is simple to calculate since Cpt is the intersection of a unit ball in the p-norm and a box. The whole attack process can be summarized in Algorithm 1.
Algorithm 1 cMBA algorithm at timestep t 1: Initialization:
Given st, starget, π, fd, Lt; initialize ∆s0; choose learning rate η > 0 2: For k := 0, · · · ,K − 1 do 3: Compute at = (a1t , · · · , ant ) as
ait =
{ π(sit + ∆s
i) if i ∈ Lt π(sit) otherwise.
4: Compute ŝt+1 = fd(st, at) 5: Update ∆s as ∆sk+1 = projCt [∆sk − η∇∆sd(ŝt+1, starget)] 6: End For
Learning dynamics model. The most important factor that affects the solution of (2) is the quality of the learned dynamics model fd. If the dynamics is known, we can solve (2) exactly so that the solution is indeed the optimal perturbation. However, it is hardly the case in practice because we often approximate the dynamics using function approximators such as neural networks. The parameter w for fd is the solution of the following optimization problem
min w ∑ i∈D ∥∥fd(sicur, w)− sinext∥∥2 (4) where D is a collection of state transition {(sicur, sinext)} |D| i=1 where snext is the actual state that the environment transitions to after taking action at determined by a given policy. In particular, we separately collect transitions using the pre-trained policy π and a random policy to obtain Dtrain and Drandom. Then the dataset D is built as D = Dtrain ∪ Drandom. Problem (4) is a standard supervised learning problem and can be solved using gradient-based learning. We describe the full process of training the dynamics model in Algorithm 2 where the GradientBasedUpdate step is any gradient-descent-type update.
Algorithm 2 Training dynamics model 1: Initialization:
Given pre-trained policy πtr and a random policy πrd; initialize w0 2: Form D := Dtrain ∪ Drandom by collecting a set of transitions Dtrain and Drandom using
policy πtr and πrd, respectively. 3: For k := 0, 1, · · · do
wk+1 = GradientBasedUpdate(D, wk)
4: End For

heading: 3.3 VICTIM AGENT SELECTION
text: In this subsection, we discuss different strategies to select victim agents when performing adversarial attacks in the MARL setting. Suppose that we would like to perform an attack on any agent but the budget on the victim agents is na ≤ n. In other words, an attacker can only choose a subset of
agents to attack during each time step, and the goal of an attacker is to choose the most vulnerable agents so that the total reward is utmost affected. We note that this scenario is unique in the setting of multi-agent DRL setting, as in the single DRL agent setting we can only attack the same agent all the time. To start with, we first formulate a mixed-integer program to perform the attack on a set of optimally victim agents as below:
min∆s,w d(ŝt+1, starget) s.t. ŝt+1 = fd(st, at)
at = π(st + 〈∆s, w〉) `S ≤ st + ∆s ≤ uS ‖∆s‖p ≤ wi ∈ {0, 1}∑ i wi = na
(5)
where we introduce a new binary variable w to select the appropriate agents’ input to perturb and na is the total number of victim agents we can attack.
Due to the existence of the new binary variable, problem (5) is much harder to solve than before. We instead solve a proxy of (5) as follows
min∆s,θ d(ŝt+1, starget) s.t. ŝt+1 = fd(st, at)
at = π(st + 〈∆s,W (st, θ)〉) `S ≤ st + ∆s ≤ uS ‖∆s‖p ≤ 0 ≤W (st, θ) ≤ 1
(6)
where W (·, θ) is a function parametrized by θ that takes current state as input and outputs the weight to distribute the noise to each agent. Suppose we represent W (·, θ) by a neural network, we can rewrite the formulation (6) as
min∆s,θ d(fd(st, π(st + 〈∆s,W (st, θ)〉)), starget) s.t. ∆s ∈ Cpt
(7)
because the last constraint in (6) can be enforced by using a softmax activation in the neural network W (·, θ). As a result, (7) can be efficiently solved by using PGD. We present the pseudo-code of the attack in Algorithm 3. After K steps of PGD update, we define the index in−j as the j-th largest value within W (st, θK) ∈ Rn, i.e. we have Wi(n)(st, θK) ≥ Wi(n−1)(st, θK) ≥ · · · ≥ Wi(1)(st, θK). Let Ij be the index set of top-j largest outputs of the W (st, θK) network. The final perturbation returned by our victim agent selection strategy will be ∆̂s = ((∆̂s)1, · · · , (∆̂s)n) where (∆̂s)i = 0 if i /∈ Ina and (∆̂s)i = (∆sK)i if i ∈ Ina .
Algorithm 3 cMBA with victim agent selection at timestep t 1: Initialization:
Given st, starget, π, fd, na; initialize ∆s0; choose learning rate η > 0 λ > 0. 2: For k := 0, · · · ,K − 1 do 3: Compute at = π(st + 〈∆s,W (st, θ)〉) 4: Compute ŝt+1 = fd(st, at) 5: Update ∆s: ∆sk+1 = projCpt [∆sk − η∇∆sd(ŝt+1, starget)] 6: Update θ: θk+1 = θk − λ∇θd(ŝt+1, starget) 7: End For 8: Compute Ina = {i(n), · · · , i(n−na)} such that
Wi(n)(st, θK) ≥Wi(n−1)(st, θK) ≥ · · · ≥Wi(1)(st, θK)
9: Return ∆̂s = ((∆̂s)1, · · · , (∆̂s)n) such that
(∆̂s)i =
{ (∆sK)
i if i ∈ Ina 0 otherwise
Remark 3.1 For this attack, we assume each agent i has access to the other agent’s observation to form the concatenated state st.

heading: 4 EXPERIMENTS
text: We perform the attack on various multi-agent MuJoCo (MA-MuJoCo) environments including Walker2d (2x3), HalfCheetah (2x3), HalfCheetah (6x1), and Ant (4x2). The pair environment name (config) indicates the name of MuJoCo environment along with the agent partition, where a configuration of 2x3 means there are in total 2 agents and each agent has 3 actions. Note that we cannot directly apply methods such as the fast gradient sign method (FGSM) or JSMA (an attack using saliency map) as in Lin et al. (2020), since those methods require a “target” action which is not available in continuous action space. Therefore, we consider two baselines: Uniform and Gaussian baselines where the perturbation follows either Uniform distribution U(− , ) or Normal distribution N (0, ). Variants of model-based attack. We consider the following variants of our model-based attack:
• Model-based attack on fixed agents: perform Algorithm 1 using a fixed set of victim agents L.
• Best model-based attack on fixed agents: among the model-based attack on fixed agents, pick the subset that achieves the best performance.
• Model-based attack on random agents: perform Algorithm 1 with Lt, which is sampled uniformly from N such that |Lt| = na (na is the total victim agents).
• Model-based attack with learned victim selection: perform Algorithm 3 to select vulnerable agents and perform the attack on them.
• Greedy victim selection: perform a sweep over subsets of agents with size na. For each subset, perform Algorithm 1 to obtain proper perturbation and retrieve the corresponding objective value (distance between predicted state and target state). Select the subset corresponding to the lowest objective value as the victim agents.
Experiment setup. We first use MADDPG (Lowe et al., 2017) to train MARL agents for the four MA-MuJoCo environments listed above. Using the trained agents, we collect datasets containing one million transitions to train the dynamics model for each environment. The dynamics model is a fully connected neural network with three hidden layers of 1000 neurons. We also use a fully-connected neural network for W (st, θ) in (6) with two hidden layers of 200 neurons. We use AdamW (Loshchilov & Hutter, 2017) as the optimizer and select the best learning rate from {1, 5} × {10−5, 10−4, 10−3} (the best learning rate is the one achieving lowest prediction error on a test set of 80, 000 samples). For our model-based attack, we run PGD for K = 30 steps to solve (2) and (5). We perform each attack over 16 episodes then average the rewards. We also illustrate the standard deviation of rewards using the shaded area in the plots.
Model-free baselines vs model-based attack. In this experiment, we run the two baseline attacks along with our model-based attack on the four MA-MuJoCo environments, in which there is only one victim agent (na = 1). Figure 1 illustrates the performance when we perform these attacks on specific agents under multiple budget levels using `∞-norm. For a fixed agent, our model-based attack consistently outperforms the two baselines. In particular, our model-based attack yields much lower rewards under low budget constraints (small ) compared to the two baselines.
To better visualize the performance difference, 2 illustrates the environment with and without attacks captured at different time-steps. From Figure 2, our model-based attack is able to make the MuJoCo agent flip, which terminates the episode at the 409-th timestep. The episode length and total rewards for each variant are: No attack(1000, 2644.60), Uniform(1000, 1758.97), Gaussian(891, 1399.51), Ours(409,287.06).
We also investigate how the state values change during these attacks. Figure 3 presents different recordings of state values under adversarial attacks compared to no attack. Consider state index 8, which represents the horizontal velocity of the agent. For the HalfCheetah environment, as the goal is to make the agent move forward as fast as possible, we expect the reward to be proportional to this state value. From Figure 3, all three attacks have fairly sparse red fractions across timesteps, which result in a much lower reward compared to the no-attack setting. Among the three attacks, our model-based ones appear to have the most sparse red fractions leading to the lowest rewards. In addition, the model-based attack appears to show its advantage in environments with more agents.
In particular, our approach results in lower rewards under a smaller budget as seen in HalfCheetah (6x1) and Ant (4x2) environments.
Effectiveness of learned victim selection. We further demonstrate the performance of our modelbased attack with and without learned victim selection on one victim agent. Figure 4 illustrates the performance of four model-based variants: the best performing model-based attack on the fixed agent (Best Fixed Agent), model-based attack on the random agent (Random Agent), model-based with learned victim selection as in Algorithm 3 (Learned victim Selection), and model-based with greedy victim selection (Greedy). Surprisingly, even though the greedy approach employs a bruteforce strategy to select the agent with the closest distance, its performance reveals that selecting victim agents greedily might not be a good strategy. The learned victim selection seems to be better than the random or greedy strategy and is comparable with the random one in HalfCheetah environment. The best-fixed agent variant has good performance under low budget constraints in HalfCheetah (6x1) and Ant (4x2) environments. We want to emphasize that although the learned victim selection strategy in Section 3.3 is sub-optimal, it works well in these four environments.

heading: 5 CONCLUSIONS
text: In this paper, we propose a new attack algorithm named cMBA for evaluating the robustness of c-MARL environment with continuous action space. Our cMBA algorithm is the first to consider the c-MARL and the adversarial perturbation is computed by solving an optimization problem with learned dynamics models. The cMBA approach outperforms the other baselines attack by a large margin under 4 multi-agent MuJoCo environments especially ones with larger number of agents. Unique to multi-agent setting, we also study different strategies to select victim agents. Extensive experiment results on standard c-MARL benchmarks show that our proposed model-based attack and victim-selection strategy can successfully degrade the performance of well-trained c-MARL agents while outperforming other baselines by a large margin. A future direction is to consider a timed attack strategy where the perturbation is added at certain timesteps. We can also consider longer planning step, i.e. trying to reach a target state within the next T steps instead of 1.
ETHICS STATEMENT
This paper does not contain ethics concerns.

heading: A MODE DETAILS ON EXPERIMENT SETUP IN SECTION 4
text: Specifying target state for each environment. To perform our model based attack, we need to specify a target state that potentially worsens the total reward. In multi-agent MuJoCo environments, each agent has access to its own observation of the agent consisting the position-related and velocityrelated information. The position-related information includes part of x, y, z coordinates and the quarternion that represents the orientation of the agent. The velocity-related information contains global linear velocities and angular velocities for each joint in a MuJoCo agent. We refer the reader to Todorov et al. (2012) for more information about each MuJoCo environment. Now we describe the design of this target state for each environment as follows:
• Walker (2x3) environment: Since the episode ends whenever the agent falls, i.e. the z coordinate falls below certain threshold. In this environment, the target state has a value of 0 for the index that corresponds to the z coordinate of the MuJoCo agent (index 0).
• HalfCheetah (2x3) and HalfCheetah (6x1) environments: As the goal is to make agent moves as fast as possible, we set the index corresponding to the linear velocities to 0 (index 8).
• Ant (4x2) environment: As the agent can move freely in a 2D-plan, we set the index corresponding to the x, y linear velocities to 0 (indices 13 and 14).

heading: B ADDITIONAL EXPERIMENTS
text: In this section, we present experimental results in addition to ones presented in Section 4.
Figure 5 illustrates the environment with and without attacks captured at different timesteps. From Figure 5, our model based attack is able to make the MuJoCo agent fall down which terminates the episode at the 65-th timestep. The episode length and total rewards for each variant are: No attack(478, 1736.44), Uniform(382, 1037.73), Gaussian(90, 32.24), Ours(63,−34.26). Figure 6 illustrates how the state values change during an episode. These state values correspond to the agent shown in Figure 5. If we look at how the noise values change as in Figure 7, the noise generated by our approach appears to maximize the permissible noise budget.
In addition to the `∞-norm budget constraint, we also evaluate adversarial attacks using the `1- norm constraint. Note that using `1-norm for budget constraint is more challenging as the attack needs to distribute the noise across all states while in the `∞-norm the computation of perturbation for individual state is independent. Figure 8 illustrates an episode of a Walker agent with and without attack. Our cMBA approach is able to make the agent fall at the 255-th timestep with the episode length and rewards for each setting as: No attack(661, 2513.57), Uniform(403, 1144.77), Gaussian(455, 1637.16), Ours(253,759.85). From the noise values in Figure 7, our cMBA method appears to put noise on a few selective states at each timestep, similar to the Gausian noise setting.
Additional experiments using the approach in Lin et al. (2020) for continuous action spaces: In this experiment, we follow the approach in Lin et al. (2020) in the Ant (4x2) environment where we train an adversarial policy for one agent trying to minimize the total team reward while the remaining agents use the trained MARL policy. The adversarial policy is trained for 1 million timesteps. The results using this approach compared with our approach and the two baselines are presented in Figure 11.
From Figure 11, our cMBA approach outperforms Lin et al. (2020)’s approach as it is able to achieve lower team rewards when attacking the same agent. We also note that Lin et al. (2020)’s approach is better than the two baselines using Uniform and Gaussian noise.
Effect of using ‖·‖1 budget constraint: In this experiment, we replace the ‖·‖∞ by the ‖·‖1 for budget constraint. Using ‖·‖1 constraint will produce a significant different adversarial noise pattern
compared to ‖·‖∞ as producing adversarial noise when using ‖·‖∞ is independent for each state while it is not the case for ‖·‖1. We run different attacks on two environments, HalfCheetah (6x1) and Ant (4x2), and the results are presented in Figure 12 and Figure 13. From these figures, cMBA outperforms other baselines. In particular, it is able to achieve much lower rewards under smaller budget constraint which shows the advantage of our approach.
Adversarial attacks using dynamics model with various accuracy: In this test, we compare the performance of our attack when using trained dynamics model with different mean-squared error (MSE). We use the Ant (4x2) environment and consider the following 3 dynamics models:
• The first model is trained using 1 million samples for 100 epochs and select the model with the best performance. We denote this model ”1M - Best epoch”.
• The second model is trained using 1 million samples for only 1 epoch. We denote this model ”1M - 1st epoch”.
• We further reduce the number of samples used for training the dynamic model to only 200,000 and train for only 1 epoch. We denote this model ”200K - 1st epoch”.
These models are evaluated on a predefined test set consisting of 100,000 samples. The test MSE of these models are 0.33, 0.69, 0.79, respectively, with the initial test MSE of 1.241.
Figure 14 depicts the attacks using these three models on the same agent in Ant (4x2) environment using ‖·‖∞ budget constraint. Interestingly, the dynamics model trained with only 200,000 samples for 1 epoch can achieve comparable performance with the other two models using much more samples.
Comparison between attacking all agents vs attacking 1 agent: In this experiment, we compare performance of different approaches when attacking one victim agent or simply perturbing all agents under the same budget constraint. We compare different approaches in the Ant (4x2) environment. In particular, we use the two baselines and our cMBA approach to attack each agent out of 4 agent, denoted as (agent i) for i = 0, 1, 2, 3. We also use these approach in which we simultaneously perturb the inputs to all agents, denoted as (4 agents). In addition, we also illustrate the performance of our victim selection scheme when we only want to attack either 1 or two agents at the same time. We report the mean and standard deviation of rewards using these approach with several values of budget level ε in Table 1 where we use ‖ · ‖∞ for the budget constraint. From Table 1, the performance is getting better as the attacker is stronger, i.e. more agents are attacked at the same time. This makes sense as for ‖ · ‖∞, the adversarial noise for each agent can be computed independently.
Now, we consider the ‖ · ‖1 budget constraint. This is a more interesting setting as the total noise budget is fixed the adversarial noise of one agent will affect one from the other. The results are shown using several values of budget level ε in Table 2. We can observe that in this case attacking more agents is not always effective.

Review summaries:
This paper presented a novel method for cooperative MARL to evaluate the robustness under the adversarial states. Specifically, the authors leveraged a model-based approach to perform adversarial attacks on states for MARL with continuous action and state spaces. To represent well the environment, they used a deep neural network to predict the future states. Subsequently they solved an optimization problem with the learned dynamics to produce small perturbations. They also discussed a few strategies to optimally select the victim agents for the attack. To validate the proposed method, the authors implemented numerical experiments on multi-agent Mujoco tasks to show the effectiveness of the proposed method. The investigated topic in this work is quite interesting yet challenging, which remains an active research area. Overall, the paper is easy to follow and well written. However, the authors should pay attention to a few points I have raised in the following to make the paper more technically solid and sound.

Proposes an adversarial attack on continuous cooperative multi-agent settings. Given a target low performing state, the adversary chooses an observation perturbation by trying to minimize the distance from the next state (as predicted by a dynamics model) to the target state via projected gradient descent. They also experiment with optimizing for and choosing a limited number of agents to perform the adversarial action at each timestep. There experiments are in multi-limbed mujoco tasks such as Walker, HalfCheetah, and Ant, and show that their method causes the agent to receive lower reward than a random adversary.

This work focuses on attacking cooperative multi-agent reinforcement learning using a model-based method. First, a model is trained to predict the next state based on the current state and action. Then PGD is used to find a small perturbation on the current state so that the next state is closed to a target state. The target state is selected using heuristic methods and they are used as a surrogate objective for minimizing the team reward. A victim agent selection method is also proposed for selecting the best target action.