Greedy-based Value Representation for Efficient Coordination in Multi-agent Reinforcement Learning

1 INTRODUCTION . By taking advantage of the deep learning technique , cooperative multi-agent reinforcement learning ( MARL ) shows great scalability and excellent performance on challenging tasks ( Vorotnikov et al. , 2018 ; Wu et al. , 2020 ) such as StarCraft unit micromanagement ( Foerster et al. , 2018 ) . As an efficient paradigm of cooperative MARL , centralized training with decentralized execution ( CTDE ) ( Oliehoek et al. , 2008 ; Foerster et al. , 2016 ; Lowe et al. , 2017 ) gains growing attention . A simple and effective approach to adopt CTDE in value-based cooperative MARL is linear value decomposition ( LVD ) or monotonic value decomposition ( MVD ) . However , both LVD and MVD suffer from relative overgeneralization ( Panait et al. , 2006 ; Wei et al. , 2018 ) due to the representation limitation of the joint Q value function . As a result , they can not guarantee optimal coordination . Recent works address the problem from two different perspectives . The first kind of method aims to solve the representation limitation directly through value functions with complete expressiveness capacity ( e.g. , QTRAN ( Son et al. , 2019 ) and QPLEX ( Wang et al. , 2020 ) ) . However , learning the complete expressiveness is impractical in complicated MARL tasks because the joint action space increases exponentially with the number of agents . The other kind of method tries to overcome relative overgeneralization by learning a bias ( e.g. , WQMIX ( Rashid et al. , 2020 ) and MAVEN ( Mahajan et al. , 2019 ) ) , which lacks theoretical and quantitative analysis of the problem and is only applicable in specific tasks . As a result , these methods are insufficient to guarantee optimal coordination . More discussions about related works are provided in Appendix A . Value decomposition is a popular approach to assign credit for individual agents in fully cooperative MARL tasks , where the main concern is the optimality of coordination . In a successful case of credit assignment via value decomposition , individual agents act according to their local policies and achieve the best team ’ s performance . To evaluate the value decomposition , we propose the optimal consistency , a criterion concerning the optimality of coordination . The optimal consistency can be decomposed into two conditions : Individual-Global-Max ( IGM ) and True-Global-Max ( TGM ) . In this paper , to achieve the optimal consistency efficiently , we investigate the requirements of the TGM condition and go deep into the mechanism of the value representation for LVD and MVD , where the IGM condition always holds . We first derive the expression of the joint Q value function of LVD and MVD , by which we draw some interesting conclusions . Firstly , LVD and MVD share the same expression of the joint Q value function . Secondly , the correspondence between the joint greedy action and the maximal Q true value ( i.e. , the team ’ s best performance ) depends heavily on the task-specific reward function for LVD and MVD . Thirdly , there may be multiple stable points of the joint greedy action . As a result , the joint policy may converge to different results . More importantly , in some of the stable points , the true Q value of the joint greedy action is not maximal , which is the root cause of non-optimal coordination and relative overgeneralization . To ensure the TGM condition for LVD and MVD , the stable point satisfying the TGM condition ( we call it the optimal stable point ) is required to be the unique stable point , which is the target problem to be solved in this paper . To solve the target problem , we propose the greedy-based value representation ( GVR ) . According to previous conclusions , the stable points are task-specific due to their dependency on the reward function , for which we propose the inferior target shaping ( ITS ) . ITS dynamically modifies the true Q value of inferior samples ( i.e. , the samples worse than the current greedy ) according to current greedy Q value , which is theoretically proved to stabilize the optimal point under any reward function . Besides , under ITS , the stability of a non-optimal point depends only on the probability ratio of superior samples ( i.e. , the samples better than the current greedy ) to the non-optimal sample , where the non-optimal stable points can be eliminated under a large enough ratio ( Eq.7 ) . We prove two simple ways applied by previous works ( i.e . applying weight on the superior samples ( Rashid et al. , 2020 ) and improving exploration ( Mahajan et al. , 2019 ) ) are both inapplicable to raise the ratio because the probability of superior samples decreases exponentially with the number of agents . Therefore , we further propose the superior experience replay ( SER ) , which achieves almost constant probability of superior samples by saving them in a superior buffer . SER is theoretically proved to eliminate the non-optimal stable points under ITS . We have three contributions in this work . ( 1 ) This is the first work to derive the exact expression of the joint Q value function for LVD and MVD . ( 2 ) We point out the root cause of non-optimal coordination and further propose the target problem to be solved for LVD and MVD . ( 3 ) We propose the GVR method , which is proved theoretically to ensure the optimal consistency under sufficient exploration , and our method outperforms state-of-the-art baselines in various benchmarks . 2 PRELIMINARIES . 2.1 DEC-POMDP . We model a fully cooperative multi-agent task as a decentralized partially observable Markov decision process ( Dec-POMDP ) described by a tuple G = < S , U , P , r , Z , O , n , γ > ( Guestrin et al. , 2001 ; Oliehoek & Amato , 2016 ) . s ∈ S denotes the true state of the environment . At each time step , each agent a ∈ A ≡ { 1 , 2 , · · · , n } receives a local observation za ∈ Z produced by the observation function O : S × A → Z , and then chooses an individual action ua ∈ U according to a local policy πa ( ua|τa ) : T × U → [ 0 , 1 ] , where τa ∈ T ≡ ( Z × U ) ∗ denotes the local action-observation history . The joint action of n agents u results in a shared reward r ( s , u ) and a transition to the next state s′ ∼ P ( ·|s , u ) . γ ∈ [ 0 , 1 ) is a discount factor . We denote the joint variable of group agents with bold symbols , e.g. , the joint action u ∈ U ≡ Un , the joint action-observation history τ ∈ T ≡ Tn , and the joint interactive policy ( i.e. , the policy interacts with environment to generate trajectories ) π ( u|τ ) . The true Q value of π ( ut|τt ) is denoted by Qπ ( st , ut ) = Est+1 : ∞ , ut+1 : ∞ [ Rt|st , ut ] , whereRt = ∑∞ i=0 γ irt+1 is the discounted return . The action-state value function of agent a and the group of agents are defined as utility function Ua ( ua , τa ) and joint Q value function Q ( u , τ ) respectively . The true Q value is the target of the joint Q value in training , serving as the unique external criterion of the team ’ s performance . The greedy action u∗ : = argmaxuQ ( u , τ ) is defined as the joint action with the maximal joint Q value . The optimal action uopt : = argmaxuQ ( s , u ) is defined as the joint action with the best team ’ s performance . For brevity , we sometimes omit the prefix ” joint ” for the joint variables . 2.2 OPTIMAL CONSISTENCY AND TGM CONDITION . In CTDE paradigm , agents are expected to act individually according to their local policies ( i.e. , the individual greedy actions ) while achieve the optimal coordination ( i.e. , the maximal true Q value ) . Here we define the correspondence between the individual greedy actions and the maximal true Q value as the optimal consistency . Definition 1 ( Optimal consistency ) . Given a set of utility functions { U1 ( u1 , τ1 ) ) , · · · , Un ( un , τn ) } , and the true Q value Q ( s , u ) , if the following holds { argmax u1 U1 ( u1 , τ1 ) , · · · , argmax un Un ( un , τn ) } = argmax u Q ( s , u ) ( 1 ) then we say the set of utility functions { U1 ( u1 , τ1 ) ) , · · · , Un ( un , τn ) } satisfies the optimal consistency . For simplicity , we ignore situations with non-unique optimal actions . The optimal consistency can be decomposed into two conditions : Individual-Global-Max ( IGM ) and True-Global-Max ( TGM ) . The IGM condition proposed by QTRAN ( Son et al. , 2019 ) is defined on the correspondence between individual greedy actions and the joint greedy actions ( formally , { argmaxu1 U1 ( u1 , τ1 ) , · · · , argmaxun Un ( on , τn ) } = argmaxuQ ( u , τ ) ) . To achieve the optimal consistency , the correspondence between the joint greedy action and the maximal true Q value is required , for which we define the TGM condition : Definition 2 ( TGM ) . Given a joint value function Q ( u , τ ) , and the true Q value Q ( s , u ) , if the following holds argmax u Q ( u , τ ) = argmax u Q ( s , u ) ( 2 ) then we say the joint value function Q ( u , τ ) satisfies the TGM condition . For simplicity , we ignore situations with non-unique optimal actions . 3 INVESTIGATION OF THE TGM CONDITION FOR LVD & MVD . Linear value decomposition ( LVD ) and monotonic value decomposition ( MVD ) are simple and naturally meet the IGM condition . To achieve the optimal consistency , we investigate the requirements of the TGM condition for LVD and MVD . According to Def.2 , the TGM condition is related to the joint Q value function Q ( u , τ ) . In this section , we first derive the expression of Q ( u , τ ) for LVD and MVD under −greedy visitation . The expression indicates there may be non-optimal stable points that violate the TGM condition , which is the root cause of non-optimal coordination . 3.1 EXPRESSION OF THE JOINT Q VALUE FUNCTION FOR LVD & MVD . Firstly , take two-agent linear value decomposition as an example , where the joint Q value function Q ( u1i , u 2 j , τ ) is linearly factorized into two utility functions Q ( u 1 i , u 2 j , τ ) = U1 ( u1i , τ1 ) + U2 ( u2j , τ2 ) . u1i , u 2 j ∈ { u1 , · · · , um } denote the individual actions of agent 1,2 respectively , where { u1 , · · · , um } is the discrete individual action space . Specially , we denote the individual greedy action of agent 1,2 with u1i∗ , u 2 j∗ respectively . For brevity , Q ( u 1 i , u 2 j , τ ) and Ua ( uai , τa ) are represented by Qij and Uai ( a ∈ { 1 , 2 } ) respectively . Through the derivation provided in Appendix B.1 , Qij can be represented by the true Q values as Qij = m m∑ k=1 ( Qik +Qkj ) + ( 1− ) ( Qi∗j +Qij∗ ) − 2 m2 m∑ i=1 m∑ j=1 Qij − ( 1− ) m m∑ k=1 ( Qi∗k +Qkj∗ ) − ( 1− ) 2Qi∗j∗ ( 3 ) Verification of the expression is provided in Appendix B.2 . For monotonic value decomposition , the expression is identical to Eq.3 ( the proof is provided in Appendix C ) , which indicates the coefficients and bias on utility functions do not affect the joint Q value function . For situations with more than two agents , by referring to the derivation in Appendix B.1 and C , the expression of joint Q values can also be obtained .

Summary: The paper studies the problem of value decomposition, which decomposes the joint-Q function into some linear or monotonic transformations of individual factored Q functions for each of the agents. The paper identifies limitations with previous linear/monotonic forms of the decomposition, in particular that the joint greedy action (which matches the greedy joint action for these decompositions) might not match the maximum true Q value. This condition is termed "True-Global-Max", and the paper introduces two techniques (inferior target shaping, superior experience replay) to satisfy this condition and improve upon previous suggested Q-value decompositions in literature. The paper walks through a toy matrix game example, a predator-prey experiment, and the Starcraft Multi-Agent challenge environment.