Towards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning

1 INTRODUCTION . Cooperative multi-agent reinforcement learning ( MARL ) has great promise for addressing coordination problems in a variety of applications , such as robotic systems ( Hüttenrauch et al. , 2017 ) , autonomous cars ( Cao et al. , 2012 ) , and sensor networks ( Zhang & Lesser , 2011 ) . Such complex tasks often require MARL to learn decentralized policies for agents to jointly optimize a global cumulative reward signal , and post a number of challenges , including multi-agent credit assignment ( Wolpert & Tumer , 2002 ; Nguyen et al. , 2018 ) , non-stationarity ( Zhang & Lesser , 2010 ; Song et al. , 2019 ) , and scalability ( Zhang & Lesser , 2011 ; Panait & Luke , 2005 ) . Recently , by leveraging the strength of deep learning techniques , cooperative MARL has made a series of great progress ( Sunehag et al. , 2018 ; Baker et al. , 2020 ; Wang et al. , 2020b ; a ) , particularly in value-based methods that demonstrate stateof-the-art performance on challenging tasks such as StarCraft unit micromanagement ( Samvelyan et al. , 2019 ) . Sunehag et al . ( 2018 ) proposed a popular approach called value-decomposition network ( VDN ) based on the paradigm of centralized training with decentralized execution ( CTDE ; Foerster et al. , 2016 ) . VDN learns a centralized but factorizable joint value function Qtot , represented as the summation of individual value functions Qi . During the execution , decentralized policies can be easily derived for each agent i by greedily selecting actions with respect to its local value function Qi . By utilizing this decomposition structure , an implicit multi-agent credit assignment is realized because Qi is learned by neural network backpropagation from the total temporal-difference error on the single global reward signal , rather than on a local reward signal specific to agent i . This decomposition technique significantly improves the scalability of multi-agent Q-learning algorithms and fosters a series of subsequent works , including QMIX ( Rashid et al. , 2018 ) , QTRAN ( Son et al. , 2019 ) , and QPLEX ( Wang et al. , 2020a ) . In spite of the empirical success in a broad class of tasks , multi-agent Q-learning with linear value decomposition has not been theoretically well-understood . Because of its limited representation complexity , the standard Bellman update is not a closed operator in the joint action-value function class with linear value decomposition . The approximation error induced by this incompleteness is known as inherent Bellman error ( Munos & Szepesvári , 2008 ) , which usually deviates Q-learning to an unexpected behavior . To develop a deeper understanding of learning with value decomposition , this paper introduces a multi-agent variant of the popular Fitted Q-Iteration ( FQI ; Ernst et al. , 2005 ; Levine et al. , 2020 ) framework and derives a closed-form solution to its empirical Bellman error minimization . To the best of our knowledge , it is the first theoretical analysis that characterizes the underlying mechanism of linear value decomposition in cooperative multi-agent Q-learning , which can serve as a powerful toolkit to establish follow-up profound theories and explore potential insights from different perspectives in this popular value decomposition structure . By utilizing this novel closed-form solution , this paper formally reveals two interesting insights : 1 ) Learning linear value decomposition implicitly implements a classical multi-agent credit assignment method called counterfactual difference rewards ( Wolpert & Tumer , 2002 ) , which draws a connection with COMA ( Foerster et al. , 2018 ) , a multi-agent policy-gradient method . 2 ) Multi-agent Q-learning with linear value decomposition potentially suffers from the risk of unbounded divergence from arbitrary initialization . On-policy data distribution or richer Q function classes can provide local or global convergence guarantees for multi-agent Q-learning , respectively . Finally , we set up an extensive set of experiments to demonstrate the realizability of our theoretical implications . Besides the FQI framework , we also consider deep-learning-based implementations of different multi-agent value decomposition structures . Through didactic examples and the StarCraft II benchmark , we design several experiments to illustrate the consistency of our closed-form formulation with the empirical results , and that online data distribution and richer Q function classes can significantly alleviate the limitations of VDN on the offline training process ( Levine et al. , 2020 ) . 2 RELATED WORK . Deep Q-learning algorithms that use neural networks as function approximators have shown great promise in solving complicated decision-making problems ( Mnih et al. , 2015 ) . One of the core component of such methods is iterative Bellman error minimization , which can be modelled by a classical framework called Fitted Q-Iteration ( FQI ; Ernst et al. , 2005 ) . FQI utilizes a specific Q function class to iteratively optimize empirical Bellman error on a dataset D. Great efforts have been made towards theoretically characterizing the behavior of FQI with finite samples and imperfect function classes ( Munos & Szepesvári , 2008 ; Farahmand et al. , 2010 ; Chen & Jiang , 2019 ) . From an empirical perspective , there is also a growing trend to adopt FQI for empirical analysis of deep offline Q-learning algorithms ( Fu et al. , 2019 ; Levine et al. , 2020 ) . In MARL , the joint Q function class grows exponentially with the number of agents , leading many algorithms ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ) to utilize different value decomposition structures with limited expressiveness to improve scalability . In this paper , we extend FQI to a multi-agent variant as our grounding theoretical framework for analyzing cooperative multi-agent Q-learning with linear value decomposition . To achieve superior effectiveness and scalability in multi-agent settings , centralized training with decentralized executing ( CTDE ) has become a popular MARL paradigm ( Oliehoek et al. , 2008 ; Kraemer & Banerjee , 2016 ) . Individual-Global-Max ( IGM ) principle ( Son et al. , 2019 ) is a critical concept for value-based CTDE ( Mahajan et al. , 2019 ) , that ensures the consistency between joint and local greedy action selections and enables effective performance in both training and execution phases . VDN ( Sunehag et al. , 2018 ) utilizes linear value decomposition to satisfy a sufficient condition of IGM . The simple additivity structure of VDN has achieved excellent scalability and inspired many follow-up methods . QMIX ( Rashid et al. , 2018 ) proposes a monotonic Q network structure to improve the expressiveness of the factorized function class . QTRAN ( Son et al. , 2019 ) tries to realize the entire IGM function class , but its method is computationally intractable and requires two extra soft regularizations to approximate IGM ( which actually loses the IGM guarantee ) . QPLEX ( Wang et al. , 2020a ) encodes the IGM principle into the Q network architecture and realizes a complete IGM function class , but it may also have potential limitations in scalability . Based on the advantages of VDN ’ s simplicity and scalability , linear value decomposition becomes very popular in MARL ( Son et al. , 2019 ; Wang et al. , 2020a ; d ) . This paper focuses on the theoretical and empirical understanding of multi-agent Q-learning with linear value decomposition to explore its underlying implications . 3 NOTATIONS AND PRELIMINARIES . 3.1 MULTI-AGENT MARKOV DECISION PROCESS ( MMDP ) . To support theoretical analysis on multi-agent Q-learning , we adopt the framework of MMDP ( Boutilier , 1996 ) , a special case of Dec-POMDP ( Oliehoek et al. , 2016 ) , to model fully cooperative multi-agent decision-making tasks . MMDP is defined as a tuple M = 〈N , S , A , P , r , γ〉 . N ≡ { 1 , . . . , n } is a finite set of agents . S is a finite set of global states . A denotes the action space for an individual agent . The joint action a ∈ A ≡ An is a collection of individual actions [ ai ] ni=1 . At each timestep t , a selected joint action at results in a transition st+1 ∼ P ( ·|st , at ) and a global reward signal r ( st , at ) . γ ∈ [ 0 , 1 ) is a discount factor . The goal for MARL is to construct a joint policy π = 〈π1 , . . . , πn〉 maximizing expected discounted rewards V π ( s ) = E [ ∑∞ t=0 γ tr ( st , π ( st ) ) |s0 = s ] , where πi : S 7→ A denotes an individual policy of agent i . The corresponding action-value function is denoted asQπ ( s , a ) = r ( s , a ) +γEs′∼P ( ·|s , a ) [ V π ( s′ ) ] . We useQ∗ and V ∗ to denote the action-value function and the state-value function corresponding to the optimal policy π∗ , respectively . Dec-POMDP ( Oliehoek et al. , 2016 ) is a generalized model of MMDP with the consideration of partial observability . In Dec-POMDPs , each agent can only access to its local observations rather than the full information of global states . As infinite-horizon Dec-POMDPs is undecidable in general ( Madani et al. , 1999 ) , this paper focuses theoretical analyses on settings with full observability . In practice , partial observability is not a hard constraint . A Dec-POMDP can be transformed to an MMDP when communication is available . Many prior efforts have been made to construct efficient communication protocols for exchanging information among agents ( Foerster et al. , 2016 ; Das et al. , 2019 ; Wang et al. , 2020c ) . By constructing belief states through extended observation scopes , these methods can approximately transform learning problems in Dec-POMDPs to that in MMDPs . From this perspective , we consider MMDP as a simplification of notations to make the underlying insights more accessible . 3.2 CENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION ( CTDE ) . Most deep multi-agent Q-learning algorithms with value decomposition adopt the paradigm of centralized training with decentralized execution ( Foerster et al. , 2016 ) . In the training phase , the centralized trainer can access all global information , including global states , shared global rewards , agents ’ polices , and value functions . In the decentralized execution phase , every agent makes individual decisions based on its local observations . Note that this paper considers MMDP as a simplified setting which rules out the concerns of partial observability . Thus our notations do not distinguish the concepts of states and observations . Individual-Global-Max ( IGM ) ( Son et al. , 2019 ) is a common principle to realize effective decentralized policy execution . It enforces the action selection consistency between the global joint action-value Qtot and individual action-values [ Qi ] ni=1 , which are specified as follows : ∀s ∈ S , arg max a∈A Qtot ( s , a ) = 〈 arg max a1∈A Q1 ( s , a1 ) , . . . , arg max an∈A Qn ( s , an ) 〉 . ( 1 ) As stated in Eq . ( 2 ) , the additivity constraint adopted by VDN ( Sunehag et al. , 2018 ) is a sufficient condition for the IGM constraint stated in Eq . ( 1 ) . However , this linear decomposition structure is not a necessary condition and induces a limited joint action-value function class because the linear number of individual functions can not represent a joint action-value function class , which is exponential with the number of agents . ( Additivity ) Qtot ( s , a ) = n∑ i=1 Qi ( s , ai ) . ( 2 )

Summary: This paper is largely a theoretical undertaking and focuses on bringing new insights into the currently popular value decomposition schemes like VDN, QMIX etc. for multi-agent reinforcement learning. They find two major implications: 1) linear value decomposition leads to implicit difference based credit assignment, and 2) richer Q function classes like those of QTRAN, QPLEX can improve convergence because otherwise there is a risk of unbounded divergence.