Evaluating Robustness of Cooperative MARL

1 INTRODUCTION . Deep neural networks are known to be vulnerable to adversarial examples , where a small and often imperceptible adversarial perturbation can easily fool the state-of-the-art deep neural network classifiers ( Szegedy et al. , 2013 ; Nguyen et al. , 2015 ; Goodfellow et al. , 2014 ; Papernot et al. , 2016 ) . Since then , a wide variety of deep learning tasks have been shown to also be vulnerable to adversarial attacks , ranging from various computer vision tasks to natural language processing tasks ( Jia & Liang , 2017 ; Zhang et al. , 2020b ; Jin et al. , 2020 ; Alzantot et al. , 2018 ) . Perhaps unsurprisingly , deep reinforcement learning ( DRL ) agents are also vulnerable to adversarial attacks , as first shown in Huang et al . ( 2017 ) for atari games DRL agents . In Huang et al . ( 2017 ) , the authors study the effectiveness of adversarial examples on a policy network trained on Atari games under the situation where the attacker has access to the neural network of the target policy . In Lin et al . ( 2017 ) , the authors further investigate a strategically-time attack by attacking trained agents on Atari games at a subset of the time step . Meanwhile , Kos & Song ( 2017 ) use the fast gradient sign method ( FGSM ) to generate adversarial perturbation on the A3C algorithm ( Mnih et al. , 2016 ) and explore training with random noise and FGSM perturbation to improve resilience against adversarial examples . While the above research endeavors focused on actions that take discrete values , another line of research tackles a more challenging problem on DRL with continuous action spaces ( Weng et al. , 2019 ; Gleave et al. , 2019 ) . Specifically , Weng et al . ( 2019 ) consider a two-step algorithm which determines adversarial perturbation to be closer to a target state using a learnt dynamics model , and Gleave et al . ( 2019 ) propose a physically realistic thread model and demonstrates the existence of adversarial policies in zero-sum simulated robotics games . While most of the existing DRL attack algorithms focus on the single DRL agent setting1 , in this work we propose to study the vulnerability of multi-agent DRL , which has been widely applied in many safety-critical real-world applications including swarm robotics ( Dudek et al. , 1993 ) , electricity distribution , and traffic control ( OroojlooyJadid & Hajinezhad , 2019 ) . In particular , we focus on the collaborative multi-agent reinforcement learning ( c-MARL ) setting , where a group of agents is trained to generate joint actions to maximize certain ( team ) reward function . We note that c-MARL is a more challenging yet interesting setting than the single DRL agent setting , as the interactions between agents commands consideration of additional layers of complications . Contribution . Our contribution can be summarized as follows : 1We are aware of only one recent work ( Lin et al. , 2020 ) on attacking c-MARL agents with discrete action space , rather than continuous action space considered in our work . • We propose a new adversarial attack framework on MARL with continuous action space , where we name it cMBA ( model-based attack on c-MARL ) . The attack comprises two steps : learning a representation of the environment dynamics and then determining proper observation perturbations based on the learned dynamics model . • We formulate the process of selecting an victim agent as a mixed-integer programming problem and propose an approximate formulation that can be efficiently solved by a firstorder method . • We evaluate our attack framework with two baselines and four multi-agent MuJoCo tasks ( Peng et al. , 2020 ) , with number of agents ranging from 2 to 6 , to verify the effectiveness of our approach . Our model-based attack consistently outperforms the two baselines in all tested environments . Paper outline . Section 2 discusses related works in adversarial attacks for MARL along with the problem settings . We describe the main attack framework cMBA in Section 3 , including the training of the dynamics model and solving the sub-optimization problem . In Section 3.3 , we introduce a mixed-integer program to find the optimal set of victim agents and propose an approximate formulation that can be efficiently solved using a first-order optimization method . Section 4 presents the evaluation of our approach on several multi-agent MuJoCo environments . Therein , we also study the performance of different variants of our model-based attack as discussed in Section 3.3 . 2 RELATED WORK . Adversarial attacks for DRL agents . In the c-MARL setting , although there exists a significant literature focusing on the adversarial training of MARL agents ( Phan et al. , 2020 ; 2021 ; Zhang et al. , 2020a ) , yet a very few of them addresses the scenario of adversarial attacks in c-MARL . Lin et al . ( 2020 ) propose a two-step attack on a single agent in a c-MARL environment , where they extend existing methods to generate adversarial examples guided by a learned adversarial policy . The attack in Lin et al . ( 2020 ) is evaluated under the StarCraft Multi-Agent Challenge ( SMAC ) environment ( Samvelyan et al. , 2019 ) where the action spaces are discrete . To the best of our knowledge , there has not been work considering adversarial attacks on the c-MARL setting with continuous action spaces . The closest approach to ours is the enchanting attack in Lin et al . ( 2017 ) and the modelbased attack in Weng et al . ( 2019 ) ; however , these two works focused on single DRL agent setting . The enchanting attack in Lin et al . ( 2017 ) uses a video prediction model to predict the future state and finds perturbation that minimizes a distance function between the predicted future state and a target state . Similarly , Weng et al . ( 2019 ) consider two formulations for observation and action perturbation that incorporates a learned dynamic model for future state prediction . In addition to the model-based approach , in this paper , we consider the selection of victim agents , which is unique to the multi-agent DRL setting . Training c-MARL agents . One simple approach to train c-MARL agents is to let each agent learn their own policy independently as in Independent Q-Learning ( IQL ) ( Tan , 1993 ; Tampuu et al. , 2017 ) . However , this training strategy does not capture the interaction between agents . Alternatively , one can follow a common paradigm called centralized training with decentralized execution ( CTDE ) ( Oliehoek et al. , 2008 ; Oliehoek & Amato , 2016 ; Kraemer & Banerjee , 2016 ) . Gupta et al . ( 2017 ) extend DQN ( Mnih et al. , 2015 ) , TRPO ( Schulman et al. , 2015 ) , and DDPG ( Lillicrap et al. , 2015 ) to c-MARL , where a parameter sharing neural network policy is trained in a centralized fashion but still allows decentralized behavior across agents . Similar to this , COMA ( Foerster et al. , 2018 ) is an actor-critic method that uses a centralized critic to estimate counterfactual advantage for decentralized policies . While COMA uses a single critic for all agents , MADDPG ( Lowe et al. , 2017 ) learns a centralized critic for each agent and evaluates on continuous control . Instead of having fully centralized critic , DOP ( Wang et al. , 2020b ) and FACMAC ( Peng et al. , 2020 ) apply the value decomposition idea ( Sunehag et al. , 2017 ; Rashid et al. , 2018 ; 2020 ; Son et al. , 2019 ; Wang et al. , 2020a ) into multi-agent actor-critic by having a centralized but factorized critic . 3 CMBA : MODEL-BASED ATTACK FOR C-MARL . 3.1 PROBLEM SETTING . We consider multi-agent tasks with continuous action spaces modeled as a Decentralized Partially Observable Markov Decision Process ( Dec-POMDP ) ( Oliehoek & Amato , 2016 ) . A Dec-POMDP has a finite set of agents N ≡ { 1 , · · · , n } associated with a set of states S describing global states , a set of continuous actions Ai , and a set of individual partial observation Oi for each agent i ∈ N . Given the current observation oit ∈ Oi , the action ait ∈ Ai is selected by a parametrized policy πθi : Oi → Ai . The next state agent i moves to is determined by the state transition function Pi : S × Ai → S . After that , agent i receives a reward rit calculated from a reward function Ri : S × Ai → R and observes a local observation oit+1 ∈ Oi correlated with the global state st+1 ∈ S. In addition , Dec-POMDP is associated with an initial state distribution P0 and a discount factor γ . In this setting , each agent tries to find the policy that maximizes its own total expected return Ri = ∑T t=0 γ trit where T is the sampling horizon . 3.2 PROBLEM FORMULATION . Our goal is to generate adversarial perturbations imposed to the victim agents ’ input ( state ) in order to deteriorate the total team reward . The added noise encourages the victim agents ’ observation to be close to a desired adversarial state corresponding to low reward . To avoid sampling from the environment , we use a pre-trained model that learns the dynamics of the environment to predict the next state from the perturbed observation and current action , then find the suitable noise that minimizes the distance between the predicted next state and a predefined target state . This process can be formulated as an optimization problem as follows . Formally , we consider a multi-agent setting with |N | = n agents , each agent i ∈ N observes state sit locally and takes action a i t. Let st = ( s 1 t , · · · , snt ) ∈ S be the global state at time step t which is concatenated from local states sit for each agent i = 1 , · · · , n. We also denote the joint action at = ( a 1 t , · · · , ant ) concatenated from each agent ’ s action ait . Let Lt ∈ N be the set of victim agents at time step t , i.e . the set of agents that can be attacked . Let fd : S × A → S be a parametrized function that approximates the dynamics of the environment , where A is the set of concatenated actions , one from each Ai . We assume that a pre-trained MARL policy π ( · ) is given . Let starget be the target state which can lead to poor performance to the agent . We denote as an upper bound on budget constraint w.r.t some p-norm ‖·‖p . The state perturbation ∆s = ( ∆s1 , · · · , ∆sn ) ( we suppress the dependence on t of ∆s to avoid overloading the notation ) to st is the solution to the following problem min ∆s= ( ∆s1 , ··· , ∆sn ) d ( ŝt+1 , starget ) s.t . ŝt+1 = fd ( st , at ) ait = π ( s i t + ∆s i ) ∆si = 0 , ∀ i /∈ Lt ` S ≤ st + ∆s ≤ uS ‖∆s‖p ≤ ( 1 ) where 0 is a zero vector , and the observation space is constrained within the ( vectorized ) intervals between ` S and uS . Let us first provide some insights for the formulation ( 1 ) . For each agent , using the trained policy π , we can compute the corresponding action ait given its ( possibly perturbed ) local state s i t or s i t+∆s i . From the concatenated state-action pair ( st , at ) , we can predict the next state ŝt+1 via the learned dynamics model fd . Then by minimizing the distance between the predicted next state and the target state under the budget constraint , we are forcing the agent to move closer to a damaging state in the next transition . We also note that the last equality constraint indicates that we only allow perturbing agents within the victim set Lt . Problem ( 1 ) can be efficiently solved by proximal-gradient-based methods . Firstly , by substituting the definition of at into ŝt+1 , ( 1 ) is equivalent to min x d ( fd ( st , π ( st + x ) ) , starget ) s.t . x ∈ Cpt ( 2 ) where Cpt : = { x = ( x1 , · · · , xn ) : ‖x‖p ≤ , ` S − st ≤ x ≤ uS − st and xi = 0 for i /∈ Lt } . If we choose the distance function as d ( a , b ) = ‖a− b‖2 , we can use the projected gradient descent ( PGD ) algorithm ( Nesterov , 2003 ) to solve ( 2 ) . The PGD iteration to update yk at iteration k can be described as yk+1 = projCpt [ yk − η∇yd ( fd ( st , π ( st + y ) ) , starget ) ] ( 3 ) where projCpt ( · ) is the projection to the convex set C p t and η is the learning rate . The projection is simple to calculate since Cpt is the intersection of a unit ball in the p-norm and a box . The whole attack process can be summarized in Algorithm 1 . Algorithm 1 cMBA algorithm at timestep t 1 : Initialization : Given st , starget , π , fd , Lt ; initialize ∆s0 ; choose learning rate η > 0 2 : For k : = 0 , · · · , K − 1 do 3 : Compute at = ( a1t , · · · , ant ) as ait = { π ( sit + ∆s i ) if i ∈ Lt π ( sit ) otherwise . 4 : Compute ŝt+1 = fd ( st , at ) 5 : Update ∆s as ∆sk+1 = projCt [ ∆sk − η∇∆sd ( ŝt+1 , starget ) ] 6 : End For Learning dynamics model . The most important factor that affects the solution of ( 2 ) is the quality of the learned dynamics model fd . If the dynamics is known , we can solve ( 2 ) exactly so that the solution is indeed the optimal perturbation . However , it is hardly the case in practice because we often approximate the dynamics using function approximators such as neural networks . The parameter w for fd is the solution of the following optimization problem min w ∑ i∈D ∥∥fd ( sicur , w ) − sinext∥∥2 ( 4 ) where D is a collection of state transition { ( sicur , sinext ) } |D| i=1 where snext is the actual state that the environment transitions to after taking action at determined by a given policy . In particular , we separately collect transitions using the pre-trained policy π and a random policy to obtain Dtrain and Drandom . Then the dataset D is built as D = Dtrain ∪ Drandom . Problem ( 4 ) is a standard supervised learning problem and can be solved using gradient-based learning . We describe the full process of training the dynamics model in Algorithm 2 where the GradientBasedUpdate step is any gradient-descent-type update . Algorithm 2 Training dynamics model 1 : Initialization : Given pre-trained policy πtr and a random policy πrd ; initialize w0 2 : Form D : = Dtrain ∪ Drandom by collecting a set of transitions Dtrain and Drandom using policy πtr and πrd , respectively . 3 : For k : = 0 , 1 , · · · do wk+1 = GradientBasedUpdate ( D , wk ) 4 : End For

Summary: This paper presented a novel method for cooperative MARL to evaluate the robustness under the adversarial states. Specifically, the authors leveraged a model-based approach to perform adversarial attacks on states for MARL with continuous action and state spaces. To represent well the environment, they used a deep neural network to predict the future states. Subsequently they solved an optimization problem with the learned dynamics to produce small perturbations. They also discussed a few strategies to optimally select the victim agents for the attack. To validate the proposed method, the authors implemented numerical experiments on multi-agent Mujoco tasks to show the effectiveness of the proposed method. The investigated topic in this work is quite interesting yet challenging, which remains an active research area. Overall, the paper is easy to follow and well written. However, the authors should pay attention to a few points I have raised in the following to make the paper more technically solid and sound.