Robust Multi-Agent Reinforcement Learning Driven by Correlated Equilibrium

1 INTRODUCTION . Recently , reinforcement learning ( RL ) has achieved remarkable success in many practical sequential decision problems , such as Go ( Silver et al. , 2017 ) , chess ( Silver et al. , 2018 ) , real-time strategy games ( Vinyals et al. , 2019 ) , etc . In real-world , many sequential decision problems involve more than one decision maker ( i.e . multi-agent ) , such as auto-driving , traffic light control and network routing . Cooperative multi-agent reinforcement learning ( CMARL ) is a key framework to solve these practical problems . Existing MARL methods for cooperative environments include policybased methods , e.g . MADDPG ( Lowe et al. , 2017 ) , COMA ( Foerster et al. , 2017 ) , and value-based methods , e.g . VDN ( Sunehag et al. , 2018 ) , QMIX ( Rashid et al. , 2018 ) , QTRAN ( Son et al. , 2019 ) . However , before we actually apply CMARL ’ s policy into real world applications , a question must be asked : are these learned policies safe or robust to be deployed ? What will happen if some agents made mistakes or behaved adversarially against other agents ? It will be most likely that the entire team might fail to achieve their goal or perform extremely poorly . ( Lin et al. , 2020 ) demonstrates the unrobustness in CMARL environment , where a learnt adversarial of one agent can hugely decrease the team ’ s performance . Therefore , in practice , we expect to have a multi-agent team policy in a fully cooperative environment that is robust when some agent ( s ) make some mistakes and even behave adversarially . To the best of knowledge , very few existing works on this issue mainly use vanilla adversarial training strategy . Klima et al . ( 2018 ) considered a two-agent cooperative case , in order to make the policy robust , agents become competitive with a certain probability during training . Li et al . ( 2019 ) provided a robust MADDPG approach called M3DDPG , where each agent optimizes its policy based on other agents ’ influenced sub-optimal actions . Most state-of-the-art MARL algorithms utilize centralized training and decentralized execution ( CTDE ) routine , since this setting is common in real world cases . The robust MARL method M3DDPG also followed the CTDE setting . However , existing works on team mini-max normal form or extensive form games show that if the environment contains an adversarial agent , then the decentralized equilibrium from CTDE routine can be significantly worse than the correlated equilibrium . We furthermore extend this finding into stochastic team mini-max games . Inspired by this important observation , if we can urge agents to learn a correlated equilibrium ( i.e . the non-adversarial agents jointly make the decision when execution ) , then we may achieve better performance than those following CTDE in robust MARL setting . In this work , we achieve the robust MARL via solving correlated equilibrium motivated by the latent variable model , where the introduction of a latent variable across all agents could help agents jointly make their decisions . Our contributions can be summarized as follows . • We demonstrate that in stochastic team mini-max games , the decentralized equilibrium can be arbitrarily worse than correlated one , and the gap can be significantly larger than in normal or extensive form games . • With this result , we point out that learning correlated equilibrium is indeed necessary in robust MARL . • We propose a simple strategy to urge agents to learn correlated equilibrium , and show that this method can yield significant performance improvements over vanilla adversarial training . 2 RELATED WORKS . Robust RL The robustness in RL involves the perturbations occurring in different cases , such as state or observation , environment , action or policy , opponent ’ s policy , etc . 1 ) For the robustness to state or observation perturbation , most works focused on adversarial attacks of image state/observation . Pattanaik et al . ( 2018 ) used gradient-based attack on image state , and vanilla adversarial training was adopted to obtain robust policy ; Fischer et al . ( 2019 ) first trained a normal policy , and distilled it on adversarial states to achieve robustness ; Ferdowsi et al . ( 2018 ) applied adversarial training to autonomous driving tasks that interfered agent ’ s input sensors based on environment , and then conducted adversarial training ; 2 ) For the robustness to environment , robust Markov decision process ( MDP ) could be used to formulate this problem . Many works ( e.g . Wiesemann et al . ( 2013 ) ; Lim et al . ( 2013 ) ) have studied this model and provided both theoretical analysis and algorithmic design . In deep RL scenario , Rajeswaran et al . ( 2016 ) used Monte Carlo approach to train agent , while Abdullah et al . ( 2019 ) ; Hou et al . ( 2020 ) adopted adversarial training to obtain a robust agent to all environments within a Wasserstein ball . Mankowitz et al . ( 2019 ) conducted adversarial training in MPO algorithm to optimize the performance in the worst performance environment . 3 ) To be against the perturbation of action or policy , Tessler et al . ( 2019 ) ; Gu et al . ( 2018 ) ; Vinitsky et al . ( 2020 ) considered the case that agent ’ s action may be influenced by another action , and conducted adversarial training . 4 ) For the robustness to opponent , Pinto et al . ( 2017 ) ; Ma et al . ( 2018 ) focused on the case that agent ’ s reward may be influenced by another agent , and adversarial training was implemented to solve the two-agent game to obtain a robust agent . Correlated Equilibrium Correlated equilibrium is a more general equilibrium in game theory compared to Nash equilibrium . In a cooperative task , if the team agents jointly make decisions together , then the optimal team policy is correlated equilibrium . Correlated equilibrium is widely studied in game theory ( e.g . Hart & Mas-Colell ( 2001 ; 2000 ) ; Neyman ( 1997 ) ) . In team mini-max game , solving the team ’ s correlated equilibrium in normal form game is straightforward ( just treat the team as a single agent ) ; Celli & Gatti ( 2017 ) ; Zhang & An ( 2020 ) ; Farina et al . ( 2018 ) proposed various algorithms to solve correlated equilibrium in extensive formal games . In deep RL scenario , Celli et al . ( 2019 ) applied vanilla hidden variable model to solve correlated equilibrium in simple repetitive environments , while information loss with hidden variable model was used in Chen et al . ( 2019 ) to solve correlated equilibrium in normal multi-agent environment . 3 ROBUST MARL . 3.1 BACKGROUND . A typical cooperative MARL problem can be formulated as a stochastic Markov game ( S , { Ai } ni=1 , r , P ) , where S denotes the state space , { Ai } is i-th agent ’ s action space . The en- vironment starts at state s0 based on some initial distribution p0 . At each time step t , agents select a joint action at ∈ ×ni=1Ai based on some policy πteam ( at|st ) , and receive reward r ( st , at ) . The environment will transfer to a new state st+1 ∼ P ( ·|st , at ) . The goal is to maximize agents ’ expected accumulated reward : maxπteam Es0∼p0 , at∼πteam , st+1∼P [ ∑ t r ( st , at ) ] . Most state-of-the-art MARL algorithms utilize CTDE routine , that is , each agent selects its own action independently . If the environment is fully observable , then at each timestep t , each agent selects an action ait ∈ Ai based on some policy πi ( ait|st ) , and if the environment is partially observable , then at each timestep t , each agent will receive an observation oit based on st , and select action based on policy πi ( ait|oit ) . And the goal becomes maxπ1 : n Es0∼p0 , a1 : n , t∼π1 : n , st+1∼P [ ∑ t r ( st , a1 : n , t ) ] . 3.2 ROBUST MARL AND VANILLA ADVERSARIAL TRAINING . The motivation of our work is to obtain a policy that is robust when one agent makes some mistakes . In normal MARL algorithm , the team can guarantee to achieve high reward only when all agents accurately execute their optimal strategies . However , this may not always be true in real world scenarios . Real world agents may occasionally make mistakes ( e.g . machine malfunctioning ) . To achieve the robustness to this kind of mistakes , we propose to solve the worst case mini-max problem : Fixed i or ∀ i , max π1 : n min πi , mis Es0∼p0 , ait∼πi , mis , a−i , t∼π−i , st+1∼P [ ∑ t r ( st , a1 : n , t ) ] s.t . D ( πi , mis||πi ) ≤ ε ( 1 ) where { −i } means all except i. πi , mis means the mistaken policy that the mistaken agent actually perform . D is some kind of distance measure , since we can not expect that the team policy can still be robust when one agent makes very big mistakes . Unfortunately , this mini-max problem is hard to solve , since it contains two MDPs nested with each other . On the other side , the common case in real world is that , agents make mistakes randomly , i.e . these mistakes are most likely not related to the team ’ s goal and other agents ’ policies . Also , agents typically make mistakes only occasionally , since agents that always/frequently makes mistakes are not allowed to be deployed in practice . So following these considerations , we consider a simpler case and instantiate the robust cooperative MARL into QMIX algorithm . We consider a weaker worst case mini-max problem . Since we assume agent only make mistakes occasionally , we consider the case that the mistaken agent executes its “ worst ” action in a certain probability ε . Also we assume that the mistakes are most likely not related to the team ’ s goal and other agents ’ policies . In QMIX , if an agent i doesn ’ t consider other agents ’ policies , then its worst action is the one that minimizes its own Qi function ( since in QMIX ∂Qtot∂Qi ≥ 0 , lower Qi will lead to lower Qtot ) . In summary , we let the mistaken agent i perform ai , mis = { argmaxaQi ( s , a ) with prob . 1− ε argminaQi ( s , a ) with prob . ε ( 2 ) and apply vanilla adversarial training to obtain a robust policy . The detailed algorithm framework 2 is described in Appendix A.1 . The performance of vanilla adversarial training will be considered as a baseline . 4 ROBUST MARL WITH CORRELATED EQUILIBRIUM . 4.1 ROBUST MARL REQUIRES CORRELATED EQUILIBRIUM . In this part , we will show that with naive adversarial training in a centralized training and decentralized execution fashion , the learned policy might be sub-optimal in adversarial settings , thus requiring a correlated equilibrium . In typical MARL settings , if the environment is fully cooperative , then the algorithms with centralized training and decentralized execution ( e.g . QMIX/MADDPG/COMA , etc ) can achieve stateof-the-art performance in certain environments . This indicates that at least for these environments , correlation in execution is not necessary . Furthermore , Lauer & Riedmiller ( 2000 ) proved that decentralized executed policy can achieve optimal performance for fully observable and fully cooperative RL . However , in the robust MARL setting , some agent ( s ) become adversarial , indicating that the environment is not fully cooperative now . So the question is : whether the correlation in execution is necessary in adversarial scenario ? For the settings of Eq . ( 1 ) , the problems actually become a team mini-max game . The works on team mini-max normal form or extensive form games ( von Stengel & Koller , 1997 ; Basilico et al. , 2016 ; Celli & Gatti , 2017 ) , pointed out that the decentralized equilibrium can be significantly worse than correlated equilibrium at least for some games . In normal form team mini-max game , Basilico et al . ( 2016 ) proved that the gap between correlated and decentralized equilibrium is at most O ( mn−2 ) , where n is the number of agents ( includes the adversarial ) and m is each agent ’ s number of action . And the bound is tight . During this section , we simply define the “ correlated equilibrium ” as the equilibrium of optimal “ correlated policy ” , that is , the team learns the optimal policy π∗team ( a|st ) together , and “ decentralized equilibrium ” as the equilibrium of optimal “ decentralized policy ” learned by CTDE algorithm . We denote Ecor as the team ’ s expected reward under their optimal correlated policy , and Edec as that in their optimal decentralized policy . In MARL , agents play a stochastic game . Since a repeated normal form game is a special case of stochastic games , so EcorEdec can be at least m n−2 in stochastic games . Moreover , we find that this gap can be even larger in stochastic games than that in normal form games , because stochastic game is a sequential game , and agents ’ previous actions can influence the future state and therefore affect the future reward . Proposition 1 . There exists a stochastic game that EcorEdec > m n−2 . Proof . Consider this example : S = { S1 , S2 } , initial state is S1 . Agents 1 · · ·n − 1 is a team , n is adversary . Ai = { 1 , · · · , m } , i = 1 · · ·n . Discount factor γ < 1 . The team ’ s reward function r ( S1 , a ) = 0 , ∀a . r ( S2 , a ) = { 1 a1 = · · · = an 0 otherwise . Deterministic state transition function T ( S1 , a ) = { S2 a1 = · · · = an S1 otherwise ; T ( S2 , a ) = { Game ends a1 = a2 = · · · = an S2 otherwise . In each state , the team ’ s optimal correlated policy is to perform correlated action { 1 , · · · , 1 } , · · · , { m , · · · , m } with equal probability 1m . Because if the team perform any of these actions with probability less than 1m , the adversary will perform that action , which will reduce the team ’ s reward . However , if the team plays in a decentralized way , then each agent ’ s optimal policy is to perform each m actions with equal probability 1m . We can prove that in this example , EcorEdec ≥ m 2n−4 ( 1− r ) 2 . The detailed derivation can be found in Appendix A.2 . In fact , the gap between correlated and decentralized equilibrium in stochastic team mini-max game can be arbitrarily larger than normal form game , elaborated in the following proposition . The detailed derivation can be found in Appendix A.2 . Proposition 2 . ∀ fixed k ∈ Z+ , there exists a stochastic game in which EcorEdec ≥ O ( m k ( n−2 ) ) . Therefore , in robust MARL settings , CTDE algorithms may no longer achieve optimal performance . To achieve better robustness in adversarial settings , we need to design some methods that can urge agents to learn correlated equilibrium .

Summary: The paper addresses the issue of robustness in cooperative multi-agent RL setups, where the inclusion at test time of an agent that makes error or is even adversarial can drastically decrease performance. The main idea is to compute a correlated equilibrium, by allowing all agents policies to depend on a common signal. To encourage the actions of the agents to correlate, they add a mutual information loss (i.e. a retrodiction that encourages the global latent to be predictable given the action taken).