Robust Multi-Agent Reinforcement Learning Driven by Correlated Equilibrium

1 INTRODUCTION . Recently , reinforcement learning ( RL ) has achieved remarkable success in many practical sequential decision problems , such as Go ( Silver et al. , 2017 ) , chess ( Silver et al. , 2018 ) , real-time strategy games ( Vinyals et al. , 2019 ) , etc . In real-world , many sequential decision problems involve more than one decision maker ( i.e . multi-agent ) , such as auto-driving , traffic light control and network routing . Cooperative multi-agent reinforcement learning ( CMARL ) is a key framework to solve these practical problems . Existing MARL methods for cooperative environments include policybased methods , e.g . MADDPG ( Lowe et al. , 2017 ) , COMA ( Foerster et al. , 2017 ) , and value-based methods , e.g . VDN ( Sunehag et al. , 2018 ) , QMIX ( Rashid et al. , 2018 ) , QTRAN ( Son et al. , 2019 ) . However , before we actually apply CMARL ’ s policy into real world applications , a question must be asked : are these learned policies safe or robust to be deployed ? What will happen if some agents made mistakes or behaved adversarially against other agents ? It will be most likely that the entire team might fail to achieve their goal or perform extremely poorly . ( Lin et al. , 2020 ) demonstrates the unrobustness in CMARL environment , where a learnt adversarial of one agent can hugely decrease the team ’ s performance . Therefore , in practice , we expect to have a multi-agent team policy in a fully cooperative environment that is robust when some agent ( s ) make some mistakes and even behave adversarially . To the best of knowledge , very few existing works on this issue mainly use vanilla adversarial training strategy . Klima et al . ( 2018 ) considered a two-agent cooperative case , in order to make the policy robust , agents become competitive with a certain probability during training . Li et al . ( 2019 ) provided a robust MADDPG approach called M3DDPG , where each agent optimizes its policy based on other agents ’ influenced sub-optimal actions . Most state-of-the-art MARL algorithms utilize centralized training and decentralized execution ( CTDE ) routine , since this setting is common in real world cases . The robust MARL method M3DDPG also followed the CTDE setting . However , existing works on team mini-max normal form or extensive form games show that if the environment contains an adversarial agent , then the decentralized equilibrium from CTDE routine can be significantly worse than the correlated equilibrium . We furthermore extend this finding into stochastic team mini-max games . Inspired by this important observation , if we can urge agents to learn a correlated equilibrium ( i.e . the non-adversarial agents jointly make the decision when execution ) , then we may achieve better performance than those following CTDE in robust MARL setting . In this work , we achieve the robust MARL via solving correlated equilibrium motivated by the latent variable model , where the introduction of a latent variable across all agents could help agents jointly make their decisions . Our contributions can be summarized as follows . • We demonstrate that in stochastic team mini-max games , the decentralized equilibrium can be arbitrarily worse than correlated one , and the gap can be significantly larger than in normal or extensive form games . • With this result , we point out that learning correlated equilibrium is indeed necessary in robust MARL . • We propose a simple strategy to urge agents to learn correlated equilibrium , and show that this method can yield significant performance improvements over vanilla adversarial training . 2 RELATED WORKS . Robust RL The robustness in RL involves the perturbations occurring in different cases , such as state or observation , environment , action or policy , opponent ’ s policy , etc . 1 ) For the robustness to state or observation perturbation , most works focused on adversarial attacks of image state/observation . Pattanaik et al . ( 2018 ) used gradient-based attack on image state , and vanilla adversarial training was adopted to obtain robust policy ; Fischer et al . ( 2019 ) first trained a normal policy , and distilled it on adversarial states to achieve robustness ; Ferdowsi et al . ( 2018 ) applied adversarial training to autonomous driving tasks that interfered agent ’ s input sensors based on environment , and then conducted adversarial training ; 2 ) For the robustness to environment , robust Markov decision process ( MDP ) could be used to formulate this problem . Many works ( e.g . Wiesemann et al . ( 2013 ) ; Lim et al . ( 2013 ) ) have studied this model and provided both theoretical analysis and algorithmic design . In deep RL scenario , Rajeswaran et al . ( 2016 ) used Monte Carlo approach to train agent , while Abdullah et al . ( 2019 ) ; Hou et al . ( 2020 ) adopted adversarial training to obtain a robust agent to all environments within a Wasserstein ball . Mankowitz et al . ( 2019 ) conducted adversarial training in MPO algorithm to optimize the performance in the worst performance environment . 3 ) To be against the perturbation of action or policy , Tessler et al . ( 2019 ) ; Gu et al . ( 2018 ) ; Vinitsky et al . ( 2020 ) considered the case that agent ’ s action may be influenced by another action , and conducted adversarial training . 4 ) For the robustness to opponent , Pinto et al . ( 2017 ) ; Ma et al . ( 2018 ) focused on the case that agent ’ s reward may be influenced by another agent , and adversarial training was implemented to solve the two-agent game to obtain a robust agent . Correlated Equilibrium Correlated equilibrium is a more general equilibrium in game theory compared to Nash equilibrium . In a cooperative task , if the team agents jointly make decisions together , then the optimal team policy is correlated equilibrium . Correlated equilibrium is widely studied in game theory ( e.g . Hart & Mas-Colell ( 2001 ; 2000 ) ; Neyman ( 1997 ) ) . In team mini-max game , solving the team ’ s correlated equilibrium in normal form game is straightforward ( just treat the team as a single agent ) ; Celli & Gatti ( 2017 ) ; Zhang & An ( 2020 ) ; Farina et al . ( 2018 ) proposed various algorithms to solve correlated equilibrium in extensive formal games . In deep RL scenario , Celli et al . ( 2019 ) applied vanilla hidden variable model to solve correlated equilibrium in simple repetitive environments , while information loss with hidden variable model was used in Chen et al . ( 2019 ) to solve correlated equilibrium in normal multi-agent environment . 3 ROBUST MARL . 3.1 BACKGROUND . A typical cooperative MARL problem can be formulated as a stochastic Markov game ( S , { Ai } ni=1 , r , P ) , where S denotes the state space , { Ai } is i-th agent ’ s action space . The en- vironment starts at state s0 based on some initial distribution p0 . At each time step t , agents select a joint action at ∈ ×ni=1Ai based on some policy πteam ( at|st ) , and receive reward r ( st , at ) . The environment will transfer to a new state st+1 ∼ P ( ·|st , at ) . The goal is to maximize agents ’ expected accumulated reward : maxπteam Es0∼p0 , at∼πteam , st+1∼P [ ∑ t r ( st , at ) ] . Most state-of-the-art MARL algorithms utilize CTDE routine , that is , each agent selects its own action independently . If the environment is fully observable , then at each timestep t , each agent selects an action ait ∈ Ai based on some policy πi ( ait|st ) , and if the environment is partially observable , then at each timestep t , each agent will receive an observation oit based on st , and select action based on policy πi ( ait|oit ) . And the goal becomes maxπ1 : n Es0∼p0 , a1 : n , t∼π1 : n , st+1∼P [ ∑ t r ( st , a1 : n , t ) ] . 3.2 ROBUST MARL AND VANILLA ADVERSARIAL TRAINING . The motivation of our work is to obtain a policy that is robust when one agent makes some mistakes . In normal MARL algorithm , the team can guarantee to achieve high reward only when all agents accurately execute their optimal strategies . However , this may not always be true in real world scenarios . Real world agents may occasionally make mistakes ( e.g . machine malfunctioning ) . To achieve the robustness to this kind of mistakes , we propose to solve the worst case mini-max problem : Fixed i or ∀ i , max π1 : n min πi , mis Es0∼p0 , ait∼πi , mis , a−i , t∼π−i , st+1∼P [ ∑ t r ( st , a1 : n , t ) ] s.t . D ( πi , mis||πi ) ≤ ε ( 1 ) where { −i } means all except i. πi , mis means the mistaken policy that the mistaken agent actually perform . D is some kind of distance measure , since we can not expect that the team policy can still be robust when one agent makes very big mistakes . Unfortunately , this mini-max problem is hard to solve , since it contains two MDPs nested with each other . On the other side , the common case in real world is that , agents make mistakes randomly , i.e . these mistakes are most likely not related to the team ’ s goal and other agents ’ policies . Also , agents typically make mistakes only occasionally , since agents that always/frequently makes mistakes are not allowed to be deployed in practice . So following these considerations , we consider a simpler case and instantiate the robust cooperative MARL into QMIX algorithm . We consider a weaker worst case mini-max problem . Since we assume agent only make mistakes occasionally , we consider the case that the mistaken agent executes its “ worst ” action in a certain probability ε . Also we assume that the mistakes are most likely not related to the team ’ s goal and other agents ’ policies . In QMIX , if an agent i doesn ’ t consider other agents ’ policies , then its worst action is the one that minimizes its own Qi function ( since in QMIX ∂Qtot∂Qi ≥ 0 , lower Qi will lead to lower Qtot ) . In summary , we let the mistaken agent i perform ai , mis = { argmaxaQi ( s , a ) with prob . 1− ε argminaQi ( s , a ) with prob . ε ( 2 ) and apply vanilla adversarial training to obtain a robust policy . The detailed algorithm framework 2 is described in Appendix A.1 . The performance of vanilla adversarial training will be considered as a baseline . 4 ROBUST MARL WITH CORRELATED EQUILIBRIUM . 4.1 ROBUST MARL REQUIRES CORRELATED EQUILIBRIUM . In this part , we will show that with naive adversarial training in a centralized training and decentralized execution fashion , the learned policy might be sub-optimal in adversarial settings , thus requiring a correlated equilibrium . In typical MARL settings , if the environment is fully cooperative , then the algorithms with centralized training and decentralized execution ( e.g . QMIX/MADDPG/COMA , etc ) can achieve stateof-the-art performance in certain environments . This indicates that at least for these environments , correlation in execution is not necessary . Furthermore , Lauer & Riedmiller ( 2000 ) proved that decentralized executed policy can achieve optimal performance for fully observable and fully cooperative RL . However , in the robust MARL setting , some agent ( s ) become adversarial , indicating that the environment is not fully cooperative now . So the question is : whether the correlation in execution is necessary in adversarial scenario ? For the settings of Eq . ( 1 ) , the problems actually become a team mini-max game . The works on team mini-max normal form or extensive form games ( von Stengel & Koller , 1997 ; Basilico et al. , 2016 ; Celli & Gatti , 2017 ) , pointed out that the decentralized equilibrium can be significantly worse than correlated equilibrium at least for some games . In normal form team mini-max game , Basilico et al . ( 2016 ) proved that the gap between correlated and decentralized equilibrium is at most O ( mn−2 ) , where n is the number of agents ( includes the adversarial ) and m is each agent ’ s number of action . And the bound is tight . During this section , we simply define the “ correlated equilibrium ” as the equilibrium of optimal “ correlated policy ” , that is , the team learns the optimal policy π∗team ( a|st ) together , and “ decentralized equilibrium ” as the equilibrium of optimal “ decentralized policy ” learned by CTDE algorithm . We denote Ecor as the team ’ s expected reward under their optimal correlated policy , and Edec as that in their optimal decentralized policy . In MARL , agents play a stochastic game . Since a repeated normal form game is a special case of stochastic games , so EcorEdec can be at least m n−2 in stochastic games . Moreover , we find that this gap can be even larger in stochastic games than that in normal form games , because stochastic game is a sequential game , and agents ’ previous actions can influence the future state and therefore affect the future reward . Proposition 1 . There exists a stochastic game that EcorEdec > m n−2 . Proof . Consider this example : S = { S1 , S2 } , initial state is S1 . Agents 1 · · ·n − 1 is a team , n is adversary . Ai = { 1 , · · · , m } , i = 1 · · ·n . Discount factor γ < 1 . The team ’ s reward function r ( S1 , a ) = 0 , ∀a . r ( S2 , a ) = { 1 a1 = · · · = an 0 otherwise . Deterministic state transition function T ( S1 , a ) = { S2 a1 = · · · = an S1 otherwise ; T ( S2 , a ) = { Game ends a1 = a2 = · · · = an S2 otherwise . In each state , the team ’ s optimal correlated policy is to perform correlated action { 1 , · · · , 1 } , · · · , { m , · · · , m } with equal probability 1m . Because if the team perform any of these actions with probability less than 1m , the adversary will perform that action , which will reduce the team ’ s reward . However , if the team plays in a decentralized way , then each agent ’ s optimal policy is to perform each m actions with equal probability 1m . We can prove that in this example , EcorEdec ≥ m 2n−4 ( 1− r ) 2 . The detailed derivation can be found in Appendix A.2 . In fact , the gap between correlated and decentralized equilibrium in stochastic team mini-max game can be arbitrarily larger than normal form game , elaborated in the following proposition . The detailed derivation can be found in Appendix A.2 . Proposition 2 . ∀ fixed k ∈ Z+ , there exists a stochastic game in which EcorEdec ≥ O ( m k ( n−2 ) ) . Therefore , in robust MARL settings , CTDE algorithms may no longer achieve optimal performance . To achieve better robustness in adversarial settings , we need to design some methods that can urge agents to learn correlated equilibrium .

Summary: This paper considers a case in the cooperative multiagent reinforcement learning where one single agent can behave adversarially. It claims that the performance of the whole MARL system  would deteriorate significantly  and demonstrate this phenomenon both theoretically and empirically. Then the author proposes a solution, i.e. , solving the correlated equilibrium in the game.  To seek  the correlated equilibrium, the author introduces a global random variable z and adds a mutual information regularization term.  Indeed this idea is easy to follow since it directly follows the definition of the correlated equilibrium. At last, it tests the algorithm in the SMAC environment and compares it with QMIX algorithm.