QPLEX: Duplex Dueling Multi-Agent Q-Learning

1 INTRODUCTION . Cooperative multi-agent reinforcement learning ( MARL ) has broad prospects for addressing many complex real-world problems , such as sensor networks ( Zhang & Lesser , 2011 ) , coordination of robot swarms ( Hüttenrauch et al. , 2017 ) , and autonomous cars ( Cao et al. , 2012 ) . However , cooperative MARL encounters two major challenges of scalability and partial observability in practical applications . The joint state-action space grows exponentially as the number of agents increases . The partial observability and communication constraints of the environment require each agent to make its individual decisions based on local action-observation histories . To address these challenges , a popular MARL paradigm , called centralized training with decentralized execution ( CTDE ) ( Oliehoek et al. , 2008 ; Kraemer & Banerjee , 2016 ) , has recently attracted great attention , where agents ’ policies are trained with access to global information in a centralized way and executed only based on local histories in a decentralized way . Many CTDE learning approaches have been proposed recently , among which value-based MARL algorithms ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ; Wang et al. , 2019b ) have shown state-of-the-art performance on challenging tasks , e.g. , unit micromanagement in StarCraft II ( Samvelyan et al. , 2019 ) . To enable effective CTDE for multi-agent Q-learning , it is critical that the joint greedy action should be equivalent to the collection of individual greedy actions of agents , which is called the IGM ( Individual-Global-Max ) principle ( Son et al. , 2019 ) . This IGM principle provides two advantages : 1 ) ensuring the policy consistency during centralized training ( learning the joint Q-function ) and decentralized execution ( using individual Q-functions ) and 2 ) enabling ∗Equal contribution . 1Videos available at https : //sites.google.com/view/qplex-marl/ . scalable centralized training of computing one-step TD target of the joint Q-function ( deriving joint greedy action selection from individual Q-functions ) . To realize this principle , VDN ( Sunehag et al. , 2018 ) and QMIX ( Rashid et al. , 2018 ) propose two sufficient conditions of IGM to factorize the joint action-value function . However , these two decomposition methods suffer from structural constraints and limit the joint action-value function class they can represent . As shown by Wang et al . ( 2020a ) , the incompleteness of the joint value function class may lead to poor performance or potential risk of training instability in the offline setting ( Levine et al. , 2020 ) . Several methods have been proposed to address this structural limitation . QTRAN ( Son et al. , 2019 ) constructs two soft regularizations to align the greedy action selections between the joint and individual value functions . WQMIX ( Rashid et al. , 2020 ) considers a weighted projection that places more importance on better joint actions . However , due to computational considerations , both their implementations are approximate and based on heuristics , which can not guarantee the IGM consistency exactly . Therefore , achieving the complete expressiveness of the IGM function class with effective scalability remains an open problem for cooperative MARL . To address this challenge , this paper presents a novel MARL approach , called duPLEX dueling multiagent Q-learning ( QPLEX ) , that takes a duplex dueling network architecture to factorize the joint action-value function into individual action-value functions . QPLEX introduces the dueling structure Q = V + A ( Wang et al. , 2016 ) for representing both joint and individual ( duplex ) action-value functions and then reformalizes the IGM principle as an advantage-based IGM . This reformulation transforms the IGM consistency into the constraints on the value range of the advantage functions and thus facilitates the action-value function learning with linear decomposition structure . Different from QTRAN and WQMIX ( Son et al. , 2019 ; Rashid et al. , 2020 ) losing the guarantee of exact IGM consistency due to approximation , QPLEX takes advantage of a duplex dueling architecture to encode it into the neural network structure and provide a guaranteed IGM consistency . To our best knowledge , QPLEX is the first multi-agent Q-learning algorithm that effectively achieves high scalability with a full realization of the IGM principle . We evaluate the performance of QPLEX in both didactic problems proposed by prior work ( Son et al. , 2019 ; Wang et al. , 2020a ) and a range of unit micromanagement benchmark tasks in StarCraft II ( Samvelyan et al. , 2019 ) . In these didactic problems , QPLEX demonstrates its full representation expressiveness , thereby learning the optimal policy and avoiding the potential risk of training instability . Empirical results on more challenging StarCraft II tasks show that QPLEX significantly outperforms other multi-agent Q-learning baselines in online and offline data collections . It is particularly interesting that QPLEX shows the ability to support offline training , which is not possessed by other baselines . This ability not only provides QPLEX with high stability and sample efficiency but also with opportunities to efficiently utilize multi-source offline data without additional online exploration ( Fujimoto et al. , 2019 ; Fu et al. , 2020 ; Levine et al. , 2020 ; Yu et al. , 2020 ) . 2 PRELIMINARIES . 2.1 DECENTRALIZED PARTIALLY OBSERVABLE MDP ( DEC-POMDP ) . We model a fully cooperative multi-agent task as a Dec-POMDP ( Oliehoek et al. , 2016 ) defined by a tuple M = 〈N , S , A , P , Ω , O , r , γ〉 , where N ≡ { 1 , 2 , . . . , n } is a finite set of agents and s ∈ S is a finite set of global states . At each time step , every agent i ∈ N chooses an action ai ∈ A ≡ { A ( 1 ) , . . . , A ( |A| ) } on a global state s , which forms a joint action a ≡ [ ai ] ni=1 ∈ A ≡ An . It results in a joint reward r ( s , a ) and a transition to the next global state s′ ∼ P ( ·|s , a ) . γ ∈ [ 0 , 1 ) is a discount factor . We consider a partially observable setting , where each agent i receives an individual partial observation oi ∈ Ω according to the observation probability function O ( oi|s , ai ) . Each agent i has an action-observation history τi ∈ T ≡ ( Ω ×A ) ∗ and constructs its individual policy πi ( a|τi ) to jointly maximize team performance . We use τ ∈ T ≡ T n to denote joint action-observation history . The formal objective function is to find a joint policy π = 〈π1 , . . . , πn〉 that maximizes a joint value function V π ( s ) = E [ ∑∞ t=0 γ trt|s0 = s , π ] . Another quantity of interest in policy search is the joint action-value function Qπ ( s , a ) = r ( s , a ) + γEs′ [ V π ( s′ ) ] . 2.2 DEEP MULTI-AGENT Q-LEARNING IN DEC-POMDP . Q-learning algorithms is a popular algorithm to find the optimal joint action-value function Q∗ ( s , a ) = r ( s , a ) +γEs′ [ maxa′ Q∗ ( s′ , a′ ) ] . Deep Q-learning represents the action-value function with a deep neural network parameterized by θ. Mutli-agent Q-learning algorithms ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ; Yang et al. , 2020 ) use a replay memory D to store the transition tuple ( τ , a , r , τ ′ ) , where r is the reward for taking action a at joint action-observation history τ with a transition to τ ′ . Due to partial observability , Q ( τ , a ; θ ) is used in place of Q ( s , a ; θ ) . Thus , parameters θ are learnt by minimizing the following expected TD error : L ( θ ) = E ( τ , a , r , τ ′ ) ∈D [ ( r + γV ( τ ′ ; θ− ) −Q ( τ , a ; θ ) ) 2 ] , ( 1 ) where V ( τ ′ ; θ− ) = maxa′ Q ( τ ′ , a′ ; θ− ) is the one-step expected future return of the TD target and θ− are the parameters of the target network , which will be periodically updated with θ . 2.3 CENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION ( CTDE ) . CTDE is a popular paradigm of cooperative multi-agent deep reinforcement learning ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Wang et al. , 2019a ; 2020b ; c ; d ) . Agents are trained in a centralized way and granted access to other agents ’ information or the global states during the centralized training process . However , due to partial observability and communication constraints , each agent makes its own decision based on its local action-observation history during the decentralized execution phase . IGM ( Individual-Global-Max ; Son et al. , 2019 ) is a popular principle to realize effective value-based CTDE , which asserts the consistency between joint and local greedy action selections in the joint action-value Qtot ( τ , a ) and individual action-values [ Qi ( τi , ai ) ] n i=1 : ∀τ ∈ T , argmax a∈A Qtot ( τ , a ) = ( argmax a1∈A Q1 ( τ1 , a1 ) , . . . , argmax an∈A Qn ( τn , an ) ) . ( 2 ) Two factorization structures , additivity and monotonicity , has been proposed by VDN ( Sunehag et al. , 2018 ) and QMIX ( Rashid et al. , 2018 ) , respectively , as shown below : QVDNtot ( τ , a ) = n∑ i=1 Qi ( τi , ai ) and ∀i ∈ N , ∂QQMIXtot ( τ , a ) ∂Qi ( τi , ai ) > 0 . Qatten ( Yang et al. , 2020 ) is a variant of VDN , which supplements global information through a multi-head attention structure . It is known that , these structures implement sufficient but not necessary conditions for the IGM constraint , which limit the representation expressiveness of joint action-value functions ( Mahajan et al. , 2019 ) . There exist tasks whose factorizable joint action-value functions can not be represented by these decomposition methods , as shown in Section 4 . In contrast , QTRAN ( Son et al. , 2019 ) transforms IGM into a linear constraint and uses it as soft regularization constraints . WQMIX ( Rashid et al. , 2020 ) introduces a weighting mechanism into the projection of monotonic value factorization , in order to place more importance on better joint actions . However , these relaxations may violate the exact IGM consistency and may not perform well in complex problems . 3 QPLEX : DUPLEX DUELING MULTI-AGENT Q-LEARNING . In this section , we will first introduce advantage-based IGM , equivalent to the regular IGM principle , and , with this new definition , convert the IGM consistency of greedy action selection to simple constraints on advantage functions . We then present a novel deep MARL model , called duPLEX dueling multi-agent Q-learning algorithm ( QPLEX ) , that directly realizes these constraints by a scalable neural network architecture .

Summary: The paper proposed a multi-agent Q Learning algorithm with an entire IGM function class for cooperative games. The key idea is to leverage a duplex dueling network architecture to factorize the joint action-value function into individual action-value functions. The main contributions of the work lie in that the proposed method offered an highly scalable algorithms for cooperative tasks. Empirical results show that the method could achieve significant improvement in StarCraftII tasks.