Greedy-based Value Representation for Efficient Coordination in Multi-agent Reinforcement Learning

1 INTRODUCTION . By taking advantage of the deep learning technique , cooperative multi-agent reinforcement learning ( MARL ) shows great scalability and excellent performance on challenging tasks ( Vorotnikov et al. , 2018 ; Wu et al. , 2020 ) such as StarCraft unit micromanagement ( Foerster et al. , 2018 ) . As an efficient paradigm of cooperative MARL , centralized training with decentralized execution ( CTDE ) ( Oliehoek et al. , 2008 ; Foerster et al. , 2016 ; Lowe et al. , 2017 ) gains growing attention . A simple and effective approach to adopt CTDE in value-based cooperative MARL is linear value decomposition ( LVD ) or monotonic value decomposition ( MVD ) . However , both LVD and MVD suffer from relative overgeneralization ( Panait et al. , 2006 ; Wei et al. , 2018 ) due to the representation limitation of the joint Q value function . As a result , they can not guarantee optimal coordination . Recent works address the problem from two different perspectives . The first kind of method aims to solve the representation limitation directly through value functions with complete expressiveness capacity ( e.g. , QTRAN ( Son et al. , 2019 ) and QPLEX ( Wang et al. , 2020 ) ) . However , learning the complete expressiveness is impractical in complicated MARL tasks because the joint action space increases exponentially with the number of agents . The other kind of method tries to overcome relative overgeneralization by learning a bias ( e.g. , WQMIX ( Rashid et al. , 2020 ) and MAVEN ( Mahajan et al. , 2019 ) ) , which lacks theoretical and quantitative analysis of the problem and is only applicable in specific tasks . As a result , these methods are insufficient to guarantee optimal coordination . More discussions about related works are provided in Appendix A . Value decomposition is a popular approach to assign credit for individual agents in fully cooperative MARL tasks , where the main concern is the optimality of coordination . In a successful case of credit assignment via value decomposition , individual agents act according to their local policies and achieve the best team ’ s performance . To evaluate the value decomposition , we propose the optimal consistency , a criterion concerning the optimality of coordination . The optimal consistency can be decomposed into two conditions : Individual-Global-Max ( IGM ) and True-Global-Max ( TGM ) . In this paper , to achieve the optimal consistency efficiently , we investigate the requirements of the TGM condition and go deep into the mechanism of the value representation for LVD and MVD , where the IGM condition always holds . We first derive the expression of the joint Q value function of LVD and MVD , by which we draw some interesting conclusions . Firstly , LVD and MVD share the same expression of the joint Q value function . Secondly , the correspondence between the joint greedy action and the maximal Q true value ( i.e. , the team ’ s best performance ) depends heavily on the task-specific reward function for LVD and MVD . Thirdly , there may be multiple stable points of the joint greedy action . As a result , the joint policy may converge to different results . More importantly , in some of the stable points , the true Q value of the joint greedy action is not maximal , which is the root cause of non-optimal coordination and relative overgeneralization . To ensure the TGM condition for LVD and MVD , the stable point satisfying the TGM condition ( we call it the optimal stable point ) is required to be the unique stable point , which is the target problem to be solved in this paper . To solve the target problem , we propose the greedy-based value representation ( GVR ) . According to previous conclusions , the stable points are task-specific due to their dependency on the reward function , for which we propose the inferior target shaping ( ITS ) . ITS dynamically modifies the true Q value of inferior samples ( i.e. , the samples worse than the current greedy ) according to current greedy Q value , which is theoretically proved to stabilize the optimal point under any reward function . Besides , under ITS , the stability of a non-optimal point depends only on the probability ratio of superior samples ( i.e. , the samples better than the current greedy ) to the non-optimal sample , where the non-optimal stable points can be eliminated under a large enough ratio ( Eq.7 ) . We prove two simple ways applied by previous works ( i.e . applying weight on the superior samples ( Rashid et al. , 2020 ) and improving exploration ( Mahajan et al. , 2019 ) ) are both inapplicable to raise the ratio because the probability of superior samples decreases exponentially with the number of agents . Therefore , we further propose the superior experience replay ( SER ) , which achieves almost constant probability of superior samples by saving them in a superior buffer . SER is theoretically proved to eliminate the non-optimal stable points under ITS . We have three contributions in this work . ( 1 ) This is the first work to derive the exact expression of the joint Q value function for LVD and MVD . ( 2 ) We point out the root cause of non-optimal coordination and further propose the target problem to be solved for LVD and MVD . ( 3 ) We propose the GVR method , which is proved theoretically to ensure the optimal consistency under sufficient exploration , and our method outperforms state-of-the-art baselines in various benchmarks . 2 PRELIMINARIES . 2.1 DEC-POMDP . We model a fully cooperative multi-agent task as a decentralized partially observable Markov decision process ( Dec-POMDP ) described by a tuple G = < S , U , P , r , Z , O , n , γ > ( Guestrin et al. , 2001 ; Oliehoek & Amato , 2016 ) . s ∈ S denotes the true state of the environment . At each time step , each agent a ∈ A ≡ { 1 , 2 , · · · , n } receives a local observation za ∈ Z produced by the observation function O : S × A → Z , and then chooses an individual action ua ∈ U according to a local policy πa ( ua|τa ) : T × U → [ 0 , 1 ] , where τa ∈ T ≡ ( Z × U ) ∗ denotes the local action-observation history . The joint action of n agents u results in a shared reward r ( s , u ) and a transition to the next state s′ ∼ P ( ·|s , u ) . γ ∈ [ 0 , 1 ) is a discount factor . We denote the joint variable of group agents with bold symbols , e.g. , the joint action u ∈ U ≡ Un , the joint action-observation history τ ∈ T ≡ Tn , and the joint interactive policy ( i.e. , the policy interacts with environment to generate trajectories ) π ( u|τ ) . The true Q value of π ( ut|τt ) is denoted by Qπ ( st , ut ) = Est+1 : ∞ , ut+1 : ∞ [ Rt|st , ut ] , whereRt = ∑∞ i=0 γ irt+1 is the discounted return . The action-state value function of agent a and the group of agents are defined as utility function Ua ( ua , τa ) and joint Q value function Q ( u , τ ) respectively . The true Q value is the target of the joint Q value in training , serving as the unique external criterion of the team ’ s performance . The greedy action u∗ : = argmaxuQ ( u , τ ) is defined as the joint action with the maximal joint Q value . The optimal action uopt : = argmaxuQ ( s , u ) is defined as the joint action with the best team ’ s performance . For brevity , we sometimes omit the prefix ” joint ” for the joint variables . 2.2 OPTIMAL CONSISTENCY AND TGM CONDITION . In CTDE paradigm , agents are expected to act individually according to their local policies ( i.e. , the individual greedy actions ) while achieve the optimal coordination ( i.e. , the maximal true Q value ) . Here we define the correspondence between the individual greedy actions and the maximal true Q value as the optimal consistency . Definition 1 ( Optimal consistency ) . Given a set of utility functions { U1 ( u1 , τ1 ) ) , · · · , Un ( un , τn ) } , and the true Q value Q ( s , u ) , if the following holds { argmax u1 U1 ( u1 , τ1 ) , · · · , argmax un Un ( un , τn ) } = argmax u Q ( s , u ) ( 1 ) then we say the set of utility functions { U1 ( u1 , τ1 ) ) , · · · , Un ( un , τn ) } satisfies the optimal consistency . For simplicity , we ignore situations with non-unique optimal actions . The optimal consistency can be decomposed into two conditions : Individual-Global-Max ( IGM ) and True-Global-Max ( TGM ) . The IGM condition proposed by QTRAN ( Son et al. , 2019 ) is defined on the correspondence between individual greedy actions and the joint greedy actions ( formally , { argmaxu1 U1 ( u1 , τ1 ) , · · · , argmaxun Un ( on , τn ) } = argmaxuQ ( u , τ ) ) . To achieve the optimal consistency , the correspondence between the joint greedy action and the maximal true Q value is required , for which we define the TGM condition : Definition 2 ( TGM ) . Given a joint value function Q ( u , τ ) , and the true Q value Q ( s , u ) , if the following holds argmax u Q ( u , τ ) = argmax u Q ( s , u ) ( 2 ) then we say the joint value function Q ( u , τ ) satisfies the TGM condition . For simplicity , we ignore situations with non-unique optimal actions . 3 INVESTIGATION OF THE TGM CONDITION FOR LVD & MVD . Linear value decomposition ( LVD ) and monotonic value decomposition ( MVD ) are simple and naturally meet the IGM condition . To achieve the optimal consistency , we investigate the requirements of the TGM condition for LVD and MVD . According to Def.2 , the TGM condition is related to the joint Q value function Q ( u , τ ) . In this section , we first derive the expression of Q ( u , τ ) for LVD and MVD under −greedy visitation . The expression indicates there may be non-optimal stable points that violate the TGM condition , which is the root cause of non-optimal coordination . 3.1 EXPRESSION OF THE JOINT Q VALUE FUNCTION FOR LVD & MVD . Firstly , take two-agent linear value decomposition as an example , where the joint Q value function Q ( u1i , u 2 j , τ ) is linearly factorized into two utility functions Q ( u 1 i , u 2 j , τ ) = U1 ( u1i , τ1 ) + U2 ( u2j , τ2 ) . u1i , u 2 j ∈ { u1 , · · · , um } denote the individual actions of agent 1,2 respectively , where { u1 , · · · , um } is the discrete individual action space . Specially , we denote the individual greedy action of agent 1,2 with u1i∗ , u 2 j∗ respectively . For brevity , Q ( u 1 i , u 2 j , τ ) and Ua ( uai , τa ) are represented by Qij and Uai ( a ∈ { 1 , 2 } ) respectively . Through the derivation provided in Appendix B.1 , Qij can be represented by the true Q values as Qij = m m∑ k=1 ( Qik +Qkj ) + ( 1− ) ( Qi∗j +Qij∗ ) − 2 m2 m∑ i=1 m∑ j=1 Qij − ( 1− ) m m∑ k=1 ( Qi∗k +Qkj∗ ) − ( 1− ) 2Qi∗j∗ ( 3 ) Verification of the expression is provided in Appendix B.2 . For monotonic value decomposition , the expression is identical to Eq.3 ( the proof is provided in Appendix C ) , which indicates the coefficients and bias on utility functions do not affect the joint Q value function . For situations with more than two agents , by referring to the derivation in Appendix B.1 and C , the expression of joint Q values can also be obtained .

Summary: This paper aims to improve value-factorization in cooperative multi-agent reinforcement learning settings under centralized training and decentralized execution (CTDE) framework. The proposed method (GVR) attempts to ensure both Individual-Global-Max(IGM) and True-Global-Max(TGM) conditions without learning a completely expressive (CEC) value function (which can represent all joint- actions). IGM ensures the consistency between joint action selection and local greedy action selection (Individual-Global-Max, IGM), and TGM ensures that the joint-action value function correctly represents optimal values. The GVR method proposed by the authors claims to ensure both IGM and TGM conditions without CEC.  The main contribution is greedy-value based representation (GVR) which consists of two parts. First, Inferior Target Sampling (ITS) ensures that the greedy joint action is stable such that if the greedy action corresponds to the optimal joint action, then for other joint actions, the gradient will be negative, thereby selectively focusing on representing the unique optimal joint action. If the greedy action does not correspond to the unique optimal joint action, then higher exploration is required in order to destabilize it. However, the lower bound for exploration can become quite high (close to 1) with increasing agents/joint action space, therefore the authors propose a prioritized experience replay buffer. The prioritized buffer assigns higher priority to non-greedy actions which have higher values than a state-based critic.  Experimental results on matrix game, predator-prey and hard/super-hard maps from Starcraft II benchmark show improvements over baseline methods.