Evaluating Robustness of Cooperative MARL

1 INTRODUCTION . Deep neural networks are known to be vulnerable to adversarial examples , where a small and often imperceptible adversarial perturbation can easily fool the state-of-the-art deep neural network classifiers ( Szegedy et al. , 2013 ; Nguyen et al. , 2015 ; Goodfellow et al. , 2014 ; Papernot et al. , 2016 ) . Since then , a wide variety of deep learning tasks have been shown to also be vulnerable to adversarial attacks , ranging from various computer vision tasks to natural language processing tasks ( Jia & Liang , 2017 ; Zhang et al. , 2020b ; Jin et al. , 2020 ; Alzantot et al. , 2018 ) . Perhaps unsurprisingly , deep reinforcement learning ( DRL ) agents are also vulnerable to adversarial attacks , as first shown in Huang et al . ( 2017 ) for atari games DRL agents . In Huang et al . ( 2017 ) , the authors study the effectiveness of adversarial examples on a policy network trained on Atari games under the situation where the attacker has access to the neural network of the target policy . In Lin et al . ( 2017 ) , the authors further investigate a strategically-time attack by attacking trained agents on Atari games at a subset of the time step . Meanwhile , Kos & Song ( 2017 ) use the fast gradient sign method ( FGSM ) to generate adversarial perturbation on the A3C algorithm ( Mnih et al. , 2016 ) and explore training with random noise and FGSM perturbation to improve resilience against adversarial examples . While the above research endeavors focused on actions that take discrete values , another line of research tackles a more challenging problem on DRL with continuous action spaces ( Weng et al. , 2019 ; Gleave et al. , 2019 ) . Specifically , Weng et al . ( 2019 ) consider a two-step algorithm which determines adversarial perturbation to be closer to a target state using a learnt dynamics model , and Gleave et al . ( 2019 ) propose a physically realistic thread model and demonstrates the existence of adversarial policies in zero-sum simulated robotics games . While most of the existing DRL attack algorithms focus on the single DRL agent setting1 , in this work we propose to study the vulnerability of multi-agent DRL , which has been widely applied in many safety-critical real-world applications including swarm robotics ( Dudek et al. , 1993 ) , electricity distribution , and traffic control ( OroojlooyJadid & Hajinezhad , 2019 ) . In particular , we focus on the collaborative multi-agent reinforcement learning ( c-MARL ) setting , where a group of agents is trained to generate joint actions to maximize certain ( team ) reward function . We note that c-MARL is a more challenging yet interesting setting than the single DRL agent setting , as the interactions between agents commands consideration of additional layers of complications . Contribution . Our contribution can be summarized as follows : 1We are aware of only one recent work ( Lin et al. , 2020 ) on attacking c-MARL agents with discrete action space , rather than continuous action space considered in our work . • We propose a new adversarial attack framework on MARL with continuous action space , where we name it cMBA ( model-based attack on c-MARL ) . The attack comprises two steps : learning a representation of the environment dynamics and then determining proper observation perturbations based on the learned dynamics model . • We formulate the process of selecting an victim agent as a mixed-integer programming problem and propose an approximate formulation that can be efficiently solved by a firstorder method . • We evaluate our attack framework with two baselines and four multi-agent MuJoCo tasks ( Peng et al. , 2020 ) , with number of agents ranging from 2 to 6 , to verify the effectiveness of our approach . Our model-based attack consistently outperforms the two baselines in all tested environments . Paper outline . Section 2 discusses related works in adversarial attacks for MARL along with the problem settings . We describe the main attack framework cMBA in Section 3 , including the training of the dynamics model and solving the sub-optimization problem . In Section 3.3 , we introduce a mixed-integer program to find the optimal set of victim agents and propose an approximate formulation that can be efficiently solved using a first-order optimization method . Section 4 presents the evaluation of our approach on several multi-agent MuJoCo environments . Therein , we also study the performance of different variants of our model-based attack as discussed in Section 3.3 . 2 RELATED WORK . Adversarial attacks for DRL agents . In the c-MARL setting , although there exists a significant literature focusing on the adversarial training of MARL agents ( Phan et al. , 2020 ; 2021 ; Zhang et al. , 2020a ) , yet a very few of them addresses the scenario of adversarial attacks in c-MARL . Lin et al . ( 2020 ) propose a two-step attack on a single agent in a c-MARL environment , where they extend existing methods to generate adversarial examples guided by a learned adversarial policy . The attack in Lin et al . ( 2020 ) is evaluated under the StarCraft Multi-Agent Challenge ( SMAC ) environment ( Samvelyan et al. , 2019 ) where the action spaces are discrete . To the best of our knowledge , there has not been work considering adversarial attacks on the c-MARL setting with continuous action spaces . The closest approach to ours is the enchanting attack in Lin et al . ( 2017 ) and the modelbased attack in Weng et al . ( 2019 ) ; however , these two works focused on single DRL agent setting . The enchanting attack in Lin et al . ( 2017 ) uses a video prediction model to predict the future state and finds perturbation that minimizes a distance function between the predicted future state and a target state . Similarly , Weng et al . ( 2019 ) consider two formulations for observation and action perturbation that incorporates a learned dynamic model for future state prediction . In addition to the model-based approach , in this paper , we consider the selection of victim agents , which is unique to the multi-agent DRL setting . Training c-MARL agents . One simple approach to train c-MARL agents is to let each agent learn their own policy independently as in Independent Q-Learning ( IQL ) ( Tan , 1993 ; Tampuu et al. , 2017 ) . However , this training strategy does not capture the interaction between agents . Alternatively , one can follow a common paradigm called centralized training with decentralized execution ( CTDE ) ( Oliehoek et al. , 2008 ; Oliehoek & Amato , 2016 ; Kraemer & Banerjee , 2016 ) . Gupta et al . ( 2017 ) extend DQN ( Mnih et al. , 2015 ) , TRPO ( Schulman et al. , 2015 ) , and DDPG ( Lillicrap et al. , 2015 ) to c-MARL , where a parameter sharing neural network policy is trained in a centralized fashion but still allows decentralized behavior across agents . Similar to this , COMA ( Foerster et al. , 2018 ) is an actor-critic method that uses a centralized critic to estimate counterfactual advantage for decentralized policies . While COMA uses a single critic for all agents , MADDPG ( Lowe et al. , 2017 ) learns a centralized critic for each agent and evaluates on continuous control . Instead of having fully centralized critic , DOP ( Wang et al. , 2020b ) and FACMAC ( Peng et al. , 2020 ) apply the value decomposition idea ( Sunehag et al. , 2017 ; Rashid et al. , 2018 ; 2020 ; Son et al. , 2019 ; Wang et al. , 2020a ) into multi-agent actor-critic by having a centralized but factorized critic . 3 CMBA : MODEL-BASED ATTACK FOR C-MARL . 3.1 PROBLEM SETTING . We consider multi-agent tasks with continuous action spaces modeled as a Decentralized Partially Observable Markov Decision Process ( Dec-POMDP ) ( Oliehoek & Amato , 2016 ) . A Dec-POMDP has a finite set of agents N ≡ { 1 , · · · , n } associated with a set of states S describing global states , a set of continuous actions Ai , and a set of individual partial observation Oi for each agent i ∈ N . Given the current observation oit ∈ Oi , the action ait ∈ Ai is selected by a parametrized policy πθi : Oi → Ai . The next state agent i moves to is determined by the state transition function Pi : S × Ai → S . After that , agent i receives a reward rit calculated from a reward function Ri : S × Ai → R and observes a local observation oit+1 ∈ Oi correlated with the global state st+1 ∈ S. In addition , Dec-POMDP is associated with an initial state distribution P0 and a discount factor γ . In this setting , each agent tries to find the policy that maximizes its own total expected return Ri = ∑T t=0 γ trit where T is the sampling horizon . 3.2 PROBLEM FORMULATION . Our goal is to generate adversarial perturbations imposed to the victim agents ’ input ( state ) in order to deteriorate the total team reward . The added noise encourages the victim agents ’ observation to be close to a desired adversarial state corresponding to low reward . To avoid sampling from the environment , we use a pre-trained model that learns the dynamics of the environment to predict the next state from the perturbed observation and current action , then find the suitable noise that minimizes the distance between the predicted next state and a predefined target state . This process can be formulated as an optimization problem as follows . Formally , we consider a multi-agent setting with |N | = n agents , each agent i ∈ N observes state sit locally and takes action a i t. Let st = ( s 1 t , · · · , snt ) ∈ S be the global state at time step t which is concatenated from local states sit for each agent i = 1 , · · · , n. We also denote the joint action at = ( a 1 t , · · · , ant ) concatenated from each agent ’ s action ait . Let Lt ∈ N be the set of victim agents at time step t , i.e . the set of agents that can be attacked . Let fd : S × A → S be a parametrized function that approximates the dynamics of the environment , where A is the set of concatenated actions , one from each Ai . We assume that a pre-trained MARL policy π ( · ) is given . Let starget be the target state which can lead to poor performance to the agent . We denote as an upper bound on budget constraint w.r.t some p-norm ‖·‖p . The state perturbation ∆s = ( ∆s1 , · · · , ∆sn ) ( we suppress the dependence on t of ∆s to avoid overloading the notation ) to st is the solution to the following problem min ∆s= ( ∆s1 , ··· , ∆sn ) d ( ŝt+1 , starget ) s.t . ŝt+1 = fd ( st , at ) ait = π ( s i t + ∆s i ) ∆si = 0 , ∀ i /∈ Lt ` S ≤ st + ∆s ≤ uS ‖∆s‖p ≤ ( 1 ) where 0 is a zero vector , and the observation space is constrained within the ( vectorized ) intervals between ` S and uS . Let us first provide some insights for the formulation ( 1 ) . For each agent , using the trained policy π , we can compute the corresponding action ait given its ( possibly perturbed ) local state s i t or s i t+∆s i . From the concatenated state-action pair ( st , at ) , we can predict the next state ŝt+1 via the learned dynamics model fd . Then by minimizing the distance between the predicted next state and the target state under the budget constraint , we are forcing the agent to move closer to a damaging state in the next transition . We also note that the last equality constraint indicates that we only allow perturbing agents within the victim set Lt . Problem ( 1 ) can be efficiently solved by proximal-gradient-based methods . Firstly , by substituting the definition of at into ŝt+1 , ( 1 ) is equivalent to min x d ( fd ( st , π ( st + x ) ) , starget ) s.t . x ∈ Cpt ( 2 ) where Cpt : = { x = ( x1 , · · · , xn ) : ‖x‖p ≤ , ` S − st ≤ x ≤ uS − st and xi = 0 for i /∈ Lt } . If we choose the distance function as d ( a , b ) = ‖a− b‖2 , we can use the projected gradient descent ( PGD ) algorithm ( Nesterov , 2003 ) to solve ( 2 ) . The PGD iteration to update yk at iteration k can be described as yk+1 = projCpt [ yk − η∇yd ( fd ( st , π ( st + y ) ) , starget ) ] ( 3 ) where projCpt ( · ) is the projection to the convex set C p t and η is the learning rate . The projection is simple to calculate since Cpt is the intersection of a unit ball in the p-norm and a box . The whole attack process can be summarized in Algorithm 1 . Algorithm 1 cMBA algorithm at timestep t 1 : Initialization : Given st , starget , π , fd , Lt ; initialize ∆s0 ; choose learning rate η > 0 2 : For k : = 0 , · · · , K − 1 do 3 : Compute at = ( a1t , · · · , ant ) as ait = { π ( sit + ∆s i ) if i ∈ Lt π ( sit ) otherwise . 4 : Compute ŝt+1 = fd ( st , at ) 5 : Update ∆s as ∆sk+1 = projCt [ ∆sk − η∇∆sd ( ŝt+1 , starget ) ] 6 : End For Learning dynamics model . The most important factor that affects the solution of ( 2 ) is the quality of the learned dynamics model fd . If the dynamics is known , we can solve ( 2 ) exactly so that the solution is indeed the optimal perturbation . However , it is hardly the case in practice because we often approximate the dynamics using function approximators such as neural networks . The parameter w for fd is the solution of the following optimization problem min w ∑ i∈D ∥∥fd ( sicur , w ) − sinext∥∥2 ( 4 ) where D is a collection of state transition { ( sicur , sinext ) } |D| i=1 where snext is the actual state that the environment transitions to after taking action at determined by a given policy . In particular , we separately collect transitions using the pre-trained policy π and a random policy to obtain Dtrain and Drandom . Then the dataset D is built as D = Dtrain ∪ Drandom . Problem ( 4 ) is a standard supervised learning problem and can be solved using gradient-based learning . We describe the full process of training the dynamics model in Algorithm 2 where the GradientBasedUpdate step is any gradient-descent-type update . Algorithm 2 Training dynamics model 1 : Initialization : Given pre-trained policy πtr and a random policy πrd ; initialize w0 2 : Form D : = Dtrain ∪ Drandom by collecting a set of transitions Dtrain and Drandom using policy πtr and πrd , respectively . 3 : For k : = 0 , 1 , · · · do wk+1 = GradientBasedUpdate ( D , wk ) 4 : End For

Summary: This work focuses on attacking cooperative multi-agent reinforcement learning using a model-based method. First, a model is trained to predict the next state based on the current state and action. Then PGD is used to find a small perturbation on the current state so that the next state is closed to a target state. The target state is selected using heuristic methods and they are used as a surrogate objective for minimizing the team reward. A victim agent selection method is also proposed for selecting the best target action.