QPLEX: Duplex Dueling Multi-Agent Q-Learning

1 INTRODUCTION . Cooperative multi-agent reinforcement learning ( MARL ) has broad prospects for addressing many complex real-world problems , such as sensor networks ( Zhang & Lesser , 2011 ) , coordination of robot swarms ( Hüttenrauch et al. , 2017 ) , and autonomous cars ( Cao et al. , 2012 ) . However , cooperative MARL encounters two major challenges of scalability and partial observability in practical applications . The joint state-action space grows exponentially as the number of agents increases . The partial observability and communication constraints of the environment require each agent to make its individual decisions based on local action-observation histories . To address these challenges , a popular MARL paradigm , called centralized training with decentralized execution ( CTDE ) ( Oliehoek et al. , 2008 ; Kraemer & Banerjee , 2016 ) , has recently attracted great attention , where agents ’ policies are trained with access to global information in a centralized way and executed only based on local histories in a decentralized way . Many CTDE learning approaches have been proposed recently , among which value-based MARL algorithms ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ; Wang et al. , 2019b ) have shown state-of-the-art performance on challenging tasks , e.g. , unit micromanagement in StarCraft II ( Samvelyan et al. , 2019 ) . To enable effective CTDE for multi-agent Q-learning , it is critical that the joint greedy action should be equivalent to the collection of individual greedy actions of agents , which is called the IGM ( Individual-Global-Max ) principle ( Son et al. , 2019 ) . This IGM principle provides two advantages : 1 ) ensuring the policy consistency during centralized training ( learning the joint Q-function ) and decentralized execution ( using individual Q-functions ) and 2 ) enabling ∗Equal contribution . 1Videos available at https : //sites.google.com/view/qplex-marl/ . scalable centralized training of computing one-step TD target of the joint Q-function ( deriving joint greedy action selection from individual Q-functions ) . To realize this principle , VDN ( Sunehag et al. , 2018 ) and QMIX ( Rashid et al. , 2018 ) propose two sufficient conditions of IGM to factorize the joint action-value function . However , these two decomposition methods suffer from structural constraints and limit the joint action-value function class they can represent . As shown by Wang et al . ( 2020a ) , the incompleteness of the joint value function class may lead to poor performance or potential risk of training instability in the offline setting ( Levine et al. , 2020 ) . Several methods have been proposed to address this structural limitation . QTRAN ( Son et al. , 2019 ) constructs two soft regularizations to align the greedy action selections between the joint and individual value functions . WQMIX ( Rashid et al. , 2020 ) considers a weighted projection that places more importance on better joint actions . However , due to computational considerations , both their implementations are approximate and based on heuristics , which can not guarantee the IGM consistency exactly . Therefore , achieving the complete expressiveness of the IGM function class with effective scalability remains an open problem for cooperative MARL . To address this challenge , this paper presents a novel MARL approach , called duPLEX dueling multiagent Q-learning ( QPLEX ) , that takes a duplex dueling network architecture to factorize the joint action-value function into individual action-value functions . QPLEX introduces the dueling structure Q = V + A ( Wang et al. , 2016 ) for representing both joint and individual ( duplex ) action-value functions and then reformalizes the IGM principle as an advantage-based IGM . This reformulation transforms the IGM consistency into the constraints on the value range of the advantage functions and thus facilitates the action-value function learning with linear decomposition structure . Different from QTRAN and WQMIX ( Son et al. , 2019 ; Rashid et al. , 2020 ) losing the guarantee of exact IGM consistency due to approximation , QPLEX takes advantage of a duplex dueling architecture to encode it into the neural network structure and provide a guaranteed IGM consistency . To our best knowledge , QPLEX is the first multi-agent Q-learning algorithm that effectively achieves high scalability with a full realization of the IGM principle . We evaluate the performance of QPLEX in both didactic problems proposed by prior work ( Son et al. , 2019 ; Wang et al. , 2020a ) and a range of unit micromanagement benchmark tasks in StarCraft II ( Samvelyan et al. , 2019 ) . In these didactic problems , QPLEX demonstrates its full representation expressiveness , thereby learning the optimal policy and avoiding the potential risk of training instability . Empirical results on more challenging StarCraft II tasks show that QPLEX significantly outperforms other multi-agent Q-learning baselines in online and offline data collections . It is particularly interesting that QPLEX shows the ability to support offline training , which is not possessed by other baselines . This ability not only provides QPLEX with high stability and sample efficiency but also with opportunities to efficiently utilize multi-source offline data without additional online exploration ( Fujimoto et al. , 2019 ; Fu et al. , 2020 ; Levine et al. , 2020 ; Yu et al. , 2020 ) . 2 PRELIMINARIES . 2.1 DECENTRALIZED PARTIALLY OBSERVABLE MDP ( DEC-POMDP ) . We model a fully cooperative multi-agent task as a Dec-POMDP ( Oliehoek et al. , 2016 ) defined by a tuple M = 〈N , S , A , P , Ω , O , r , γ〉 , where N ≡ { 1 , 2 , . . . , n } is a finite set of agents and s ∈ S is a finite set of global states . At each time step , every agent i ∈ N chooses an action ai ∈ A ≡ { A ( 1 ) , . . . , A ( |A| ) } on a global state s , which forms a joint action a ≡ [ ai ] ni=1 ∈ A ≡ An . It results in a joint reward r ( s , a ) and a transition to the next global state s′ ∼ P ( ·|s , a ) . γ ∈ [ 0 , 1 ) is a discount factor . We consider a partially observable setting , where each agent i receives an individual partial observation oi ∈ Ω according to the observation probability function O ( oi|s , ai ) . Each agent i has an action-observation history τi ∈ T ≡ ( Ω ×A ) ∗ and constructs its individual policy πi ( a|τi ) to jointly maximize team performance . We use τ ∈ T ≡ T n to denote joint action-observation history . The formal objective function is to find a joint policy π = 〈π1 , . . . , πn〉 that maximizes a joint value function V π ( s ) = E [ ∑∞ t=0 γ trt|s0 = s , π ] . Another quantity of interest in policy search is the joint action-value function Qπ ( s , a ) = r ( s , a ) + γEs′ [ V π ( s′ ) ] . 2.2 DEEP MULTI-AGENT Q-LEARNING IN DEC-POMDP . Q-learning algorithms is a popular algorithm to find the optimal joint action-value function Q∗ ( s , a ) = r ( s , a ) +γEs′ [ maxa′ Q∗ ( s′ , a′ ) ] . Deep Q-learning represents the action-value function with a deep neural network parameterized by θ. Mutli-agent Q-learning algorithms ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ; Yang et al. , 2020 ) use a replay memory D to store the transition tuple ( τ , a , r , τ ′ ) , where r is the reward for taking action a at joint action-observation history τ with a transition to τ ′ . Due to partial observability , Q ( τ , a ; θ ) is used in place of Q ( s , a ; θ ) . Thus , parameters θ are learnt by minimizing the following expected TD error : L ( θ ) = E ( τ , a , r , τ ′ ) ∈D [ ( r + γV ( τ ′ ; θ− ) −Q ( τ , a ; θ ) ) 2 ] , ( 1 ) where V ( τ ′ ; θ− ) = maxa′ Q ( τ ′ , a′ ; θ− ) is the one-step expected future return of the TD target and θ− are the parameters of the target network , which will be periodically updated with θ . 2.3 CENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION ( CTDE ) . CTDE is a popular paradigm of cooperative multi-agent deep reinforcement learning ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Wang et al. , 2019a ; 2020b ; c ; d ) . Agents are trained in a centralized way and granted access to other agents ’ information or the global states during the centralized training process . However , due to partial observability and communication constraints , each agent makes its own decision based on its local action-observation history during the decentralized execution phase . IGM ( Individual-Global-Max ; Son et al. , 2019 ) is a popular principle to realize effective value-based CTDE , which asserts the consistency between joint and local greedy action selections in the joint action-value Qtot ( τ , a ) and individual action-values [ Qi ( τi , ai ) ] n i=1 : ∀τ ∈ T , argmax a∈A Qtot ( τ , a ) = ( argmax a1∈A Q1 ( τ1 , a1 ) , . . . , argmax an∈A Qn ( τn , an ) ) . ( 2 ) Two factorization structures , additivity and monotonicity , has been proposed by VDN ( Sunehag et al. , 2018 ) and QMIX ( Rashid et al. , 2018 ) , respectively , as shown below : QVDNtot ( τ , a ) = n∑ i=1 Qi ( τi , ai ) and ∀i ∈ N , ∂QQMIXtot ( τ , a ) ∂Qi ( τi , ai ) > 0 . Qatten ( Yang et al. , 2020 ) is a variant of VDN , which supplements global information through a multi-head attention structure . It is known that , these structures implement sufficient but not necessary conditions for the IGM constraint , which limit the representation expressiveness of joint action-value functions ( Mahajan et al. , 2019 ) . There exist tasks whose factorizable joint action-value functions can not be represented by these decomposition methods , as shown in Section 4 . In contrast , QTRAN ( Son et al. , 2019 ) transforms IGM into a linear constraint and uses it as soft regularization constraints . WQMIX ( Rashid et al. , 2020 ) introduces a weighting mechanism into the projection of monotonic value factorization , in order to place more importance on better joint actions . However , these relaxations may violate the exact IGM consistency and may not perform well in complex problems . 3 QPLEX : DUPLEX DUELING MULTI-AGENT Q-LEARNING . In this section , we will first introduce advantage-based IGM , equivalent to the regular IGM principle , and , with this new definition , convert the IGM consistency of greedy action selection to simple constraints on advantage functions . We then present a novel deep MARL model , called duPLEX dueling multi-agent Q-learning algorithm ( QPLEX ) , that directly realizes these constraints by a scalable neural network architecture .

Summary: This paper proposes a novel value decomposition approach to learn decentralized Q function in multi-agent setting. This idea is to follow Individual-Global-Max (IGM) principle. The main contribution is to use dueling structure (Q_i = V_i+A_i) for each agent i, and separately combining advantage/value terms respectively to form a centralized Q and A terms for training. Such combinations keep the positive correlation constraint between Q_tot and individual agent Q_i in QMix (i.e,. \partial Q_{tot} / \partial Q_i > 0) via positive trainable module in neural network, implemented by multi-head attention, etc.