Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization

1 Introduction . Cooperative multi-agent reinforcement learning ( MARL ) has great promise for addressing coordination problems in a variety of applications , such as robotic systems [ 1 ] , autonomous cars [ 2 ] , and sensor networks [ 3 ] . Such complex tasks often require MARL to learn decentralized policies for agents to jointly optimize a global cumulative reward signal , which posts a number of challenges , including multi-agent credit assignment [ 4 , 5 ] , non-stationarity [ 6 , 7 ] , and scalability [ 3 , 8 ] . Recently , by leveraging the strength of deep learning techniques , cooperative MARL has made great progress [ 9–19 ] , particularly in value-based methods that demonstrate state-of-the-art performance on challenging tasks such as StarCraft unit micromanagement [ 20 ] . Value factorization is a popular approach to effectively scaling up cooperative multi-agent Q-learning in complex domains . One fundamental problem of factorized multi-agent algorithms is the tradeoff between the learning scalability and the representational capacity of value functions . As a representative method , value-decomposition network ( VDN ) [ 10 ] is proposed to obtain excellent scalability based on a popular paradigm called centralized training with decentralized execution ( CTDE ) [ 21 ] , which learns a centralized but factorizable joint value function Qtot represented as the summation of individual value functionsQi . During the execution , decentralized policies can be easily derived by greedily selecting individual actions from the local value function Qi . An implicit multiagent credit assignment is realized because Qi is learned by neural network backpropagation from ∗Equal contribution . †Work done while Zhizhou was an undergraduate at Tsinghua University . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the total temporal-difference error on the single global reward signal . This linear value factorization structure can significantly improve the scalability of multi-agent joint policy training and individual policy execution . This linear value factorization adopted by VDN also realizes a sufficient condition for an important principle of CTDE , the IGM ( Individual-Global-Max ) principle [ 13 ] , which asserts the consistency between joint and individual greedy action selection and ensures the consistent policy learning under the CTDE paradigm . Following this principle , most recent advances focus on enriching the function expressiveness of the factorization from Qtot to Qi . QMIX [ 12 ] uses a monotonic function to represent the joint value function Qtot through local values Qi , whose function expressiveness is further improved by QTRAN [ 13 ] and QPLEX [ 19 ] that aim to achieve full function expressiveness induced by the IGM principle . These approaches show promise and achieve state-of-the-art performance in complicated cooperative multi-agent domains . Despite these impressive empirical successes , theoretical understandings for these MARL approaches are still limited . To bridge this gap , this paper is the first to consider a general framework for theoretical studies . We formulate Factorzed Multi-Agent Fitted Q-Iteration ( FMA-FQI ) for formally analyzing cooperative MARL with value factorization . This framework generalizes single-agent Fitted Q-Iteration , a popular model for studying Q-learning algorithms with function approximation [ 22–25 ] . FMA-FQI models the iterative training procedure of multi-agent Q-learning using empirical Bellman error minimization . Under this framework , we investigate two popular value factorization methods : the linear factorization used by VDN [ 10 ] , and the IGM factorization adopted by QTRAN [ 13 ] and QPLEX [ 19 ] . Our analyses reveal algorithmic properties of these approaches and provide insights on fundamental questions : Why do they work ? Do they converge ? When may they not perform well ? The main contribution of this paper can be summarized as follows : 1 . We formalize Factorized Multi-Agent Fitted Q-Iteration ( FMA-FQI ) as a general theoretical framework for analyzing cooperative multi-agent Q-learning with value factorization . 2 . Based on FMA-FQI , we study linear value factorization and derive a closed-form solution to its Bellman error minimization . We then reveal two novel insights that : 1 ) linear value factorization implicitly realizes a powerful counterfactual credit assignment and 2 ) on-policy training is beneficial to its stability and local convergence near the optimal solutions . 3 . Using FMA-FQI , we also study IGM value factorization and prove its global convergence and optimality . 4 . Empirical analysis is conducted to connect our theoretical implications to practical scenarios by evaluating state-of-the-art deep MARL approaches . 2 Related Work . Deep Q-learning algorithms that use neural networks as function approximators have shown great promise in solving complicated decision-making problems [ 26 ] . One of the core components of such methods is iterative Bellman error minimization , which can be modelled by a classical framework called Fitted Q-Iteration ( FQI ) [ 27 ] . FQI utilizes a specific Q-function class to iteratively optimize empirical Bellman error on a dataset D with respect to a frozen target . Great efforts have been made towards theoretically characterizing the behavior of FQI with finite samples and imperfect function classes [ 22–24 ] . From an empirical perspective , there is also a growing trend to adopt FQI for empirical analysis of deep offline Q-learning algorithms [ 25 , 28 ] . In MARL , the joint Qfunction class grows exponentially with the number of agents , leading many algorithms [ 10 , 13 , 19 ] to utilize different value factorization structures to balance scalability and optimality . In section 4 , we generalize FQI to formalize a factorized multi-agent fitted Q-iteration framework for analyzing cooperative multi-agent Q-learning algorithms with value factorization . To achieve superior effectiveness and scalability in multi-agent settings , centralized training with decentralized executing ( CTDE ) has become a popular MARL paradigm [ 29 , 30 ] . Individual-GlobalMax ( IGM ) principle [ 13 ] is a critical concept for value-based CTDE [ 31 ] , that ensures the consistency between joint and local greedy action selections and enables effective performance in both training and execution phases . VDN [ 10 ] utilizes linear value factorization to satisfy a sufficient condition of IGM . This simple linear structure of VDN has become very popular in MARL due to its excellent scalability [ 13 , 19 , 32 ] and inspired many follow-up methods . QMIX [ 12 ] proposes a monotonic Q-network structure to improve the expressiveness of the factorized function class . QTRAN [ 13 ] aims to realize the entire IGM function class , but its proposed objective is computationally intractable and requires two extra soft regularizations to approximate IGM , which actually loses rigorous IGM guarantees . QPLEX [ 19 ] , the state-of-the-art multi-agent Q-learning algorithm , encodes the IGM principle into the Q-network architecture and realize a complete IGM function class , but may have potential limitations in scalability . This paper focuses on the theoretical and empirical understanding of multi-agent Q-learning with linear and IGM value factorization to explore the underlying implications of their corresponding factorized value structures . 3 Notations and Preliminaries . 3.1 Decentralized Rich-Observation Markov Decision Process ( Dec-ROMDP ) . To support theoretical analysis on cooperative multi-agent Q-learning , we consider the problem formulation of Decentralized Rich-Observation Markov Decision Process ( Dec-ROMDP ) . DecROMDP is an interpolation of ROMDP [ 33 , 34 ] and Dec-POMDP [ 35 ] , in which the latent state space is finite , and the observation space could be arbitrarily large . A Dec-ROMDP is defined as a tupleM = 〈N , S , X , A , P , Λ , r , γ〉 . N ≡ { 1 , . . . , n } is a finite set of agents . S is a finite set of global latent states . X denotes the observation space ( a.k.a . context space ) which is larger than the latent state space . A denotes the action space for an individual agent . The joint action a ∈ A ≡ An is a collection of individual actions [ ai ] ni=1 . At each timestep t , a selected joint action at results in a latent-state transition st+1 ∼ P ( ·|st , at ) and a global reward signal r ( st , at ) . The observation of agent i is generated from the emission distribution xt , i ∼ Λ ( ·|i , st ) . The major difference between Dec-ROMDP and Dec-POMDP is the concept of rich observation . Dec-ROMDP ensures that the observation emission distributions are disjoint across latent states , i.e. , there exists an inverse function Λ−1 : N × X → S that decodes full information of latent states from observations . This assumption enables us to consider reactive function classes [ 36 , 37 ] , which only takes the latest observation as inputs . Formally , the goal for MARL in Dec-ROMDP is to construct a joint policy π = 〈π1 , . . . , πn〉 maximizing expected discounted rewards V π ( x ) = E [ ∑∞ t=0 γ tr ( st , π ( xt ) ) |x0 = x ] , where πi : X 7→ A denotes an individual policy of agent i which depends on its local observation . The corresponding action-value function is denoted as Qπ ( xt , at ) = E [ r ( st , at ) + γV π ( xt+1 ) |xt ] . More discussions are deferred to Appendix A . Why Dec-ROMDP ? The most widely-used environment setting in deep MARL for empirical studies is Dec-POMDP . However , as implied by prior work , finding an optimal policy in infinitehorizon Dec-POMDPs is an incomputable problem in the classical computation model [ 38 ] . There is no algorithm that can achieve strong theoretical guarantees in the Dec-POMDP setting . Following recent advances in reinforcement learning with function approximation [ 33 , 34 , 39 , 40 ] , we consider the rich-observation setting to derive implications from a theoretical perspective . This setting is also adopted by a concurrent work to establish algorithm analysis for cooperative multi-agent reinforcement learning [ 41 ] . 3.2 Centralized Training with Decentralized Execution ( CTDE ) . Most deep multi-agent Q-learning algorithms with value factorization adopt the paradigm of centralized training with decentralized execution [ 21 ] . In the training phase , the centralized trainer can access all global information , including joint trajectories , shared global rewards , agents ’ policies , and value functions . In the decentralized execution phase , every agent makes individual decisions based on its local information . Individual-Global-Max ( IGM ) [ 13 ] is a common principle to realize effective decentralized policy execution . It enforces the action selection consistency between the global joint action-value Qtot and individual action-values [ Qi ] ni=1 , which are specified as follows : ( IGM ) arg max a∈A Qtot ( x , a ) = 〈 arg max a1∈A Q1 ( x1 , a1 ) , · · · , arg max an∈A Qn ( xn , an ) 〉 . ( 1 ) QPLEX [ 19 ] is the state-of-the-art deep multi-agent Q-learning algorithm that realizes a complete IGM factorization . QTRAN [ 13 ] is another method to approximate the IGM principle by two extra soft regularizations . Moreover , as stated in Eq . ( 2 ) , the linear value factorization adopted by VDN [ 10 ] is a sufficient condition for the IGM constraint stated in Eq . ( 1 ) but is not a necessary condition , which induces a limited joint action-value function class . ( Additivity ) Qtot ( x , a ) = n∑ i=1 Qi ( xi , ai ) . ( 2 ) 4 Factorized Multi-Agent Fitted Q-Iteration . Constructing a specific factorized value function class satisfying the IGM condition is a critical step to realize the centralized training with decentralized execution ( CTDE ) paradigm . In the literature of Q-learning with function approximation , Fitted Q-Iteration ( FQI ) [ 27 ] is a popular framework for both theoretical and empirical studies [ 22–25 , 28 ] . In this section , we generalize single-agent FQI to formalize a Factorzed Multi-Agent Fitted Q-Iteration ( FMA-FQI ) framework , which enables us to characterize the algorithmic properties of different factorization structures in the CTDE paradigm . For multi-agent Q-learning with value factorization , we use Qtot to denote the global but factorized value function , which can be represented as a function of individual value functions [ Qi ] ni=1 . In other words , we can use [ Qi ] ni=1 to represent Qtot . For brevity , we overload Q = 〈Qtot , [ Qi ] ni=1〉 to denote their association . Following fitted Q-iteration , FMA-FQI considers an iterative optimization framework based on a given dataset D = { ( x , a , r , x′ ) } as presented in Algorithm 1 . Algorithm 1 Factorized Multi-Agent Fitted Q-Iteration ( FMA-FQI ) 1 : Input : dataset D , the number of iterations T , factorized multi-agent value function class QFMA 2 : Randomly initialize Q ( 0 ) from QFMA 3 : for t = 0 . . . T − 1 do 4 : Iteratively update Q ( t+1 ) ← T FMAD Q ( t ) ≡ arg min Q∈QFMA E ( x , a , r , x′ ) ∼D ( ŷ ( t ) ( x , a , x′ ) −Qtot ( x , a ) ) 2 ( 3 ) where ŷ ( t ) ( x , a , x′ ) = r + γmaxa′∈AQ ( t ) tot ( x ′ , a′ ) denotes the one-step TD target . 5 : Construct policies by individual value functions ∀i ∈ N , π ( t+1 ) i ( xi ) = arg max ai∈A Q ( t+1 ) i ( xi , ai ) ( 4 ) 6 : Return : [ π ( T ) i ] ni=1 Formally , FMA-FQI specifies the Q-function class through a multi-agent value factorization structure : QFMA = { Q ∣∣∣ ∃f ∈ FFMA , Qtot ( x , a ) = f ( x , a , [ Qi ] ni=1 ) , [ Qi ] ni=1 ∈ R|X×A|n } , ( 5 ) where FFMA denotes the function class characterizing the factorization structure . For example , the IGM constraint stated as Eq . ( 1 ) and the additivity constraint stated as Eq . ( 2 ) correspond to the value factorization structures used by QPLEX [ 19 ] and VDN [ 10 ] , respectively . The empirical Bellman error minimization in FMA-FQI is established using the global reward signals . To facilitate further discussions , we rewrite the regression objective as follows : Q ( t+1 ) ← T FMAD Q ( t ) ≡ arg min Q∈QFMA E ( x , a ) ∼D ( y ( t ) ( x , a ) −Qtot ( x , a ) ) 2 ( 6 ) where y ( t ) ( x , a ) = r+γEx′ [ maxa′ Q ( t ) tot ( x ′ , a′ ) ] denotes the expected one-step TD target . Assume the dataset is adequate , the equivalence between Eq . ( 3 ) and Eq . ( 6 ) is proved in Appendix A.3 . Modeling CTDE by FMA-FQI . Different from the single-agent case , FMA-FQI considers the multi-agent value factorization structure , Qtot and [ Qi ] ni=1 , to support effective training and scalable execution . In the global reward setting , the shared reward signal can only supervise the training of the joint value function Qtot ( see Eq . ( 6 ) ) . The learned joint value function Qtot would be automatically decomposed to individual value functions [ Qi ] ni=1 through the factorized function structure , which can be further used to perform decentralized execution ( see Eq . ( 4 ) ) . In practice , FMA-FQI illustrated in Eq . ( 5 ) provides the end-to-end learning of [ Qi ] ni=1 as the iterative optimization of joint value function Qtot . The greedy action selection can be searched in the individual action space A rather than the joint action space An , which significantly reduces the computation costs . Relation to Prior Work . Most prior work of FQI focus on the single-agent RL with finite-sample analyses and exploring minimum requirements for performance guarantees ( e.g. , data distribution and function capacity ) in a general sense [ 22–24 ] . In comparison , this paper is motivated by recent advances in the multi-agent area . To our best knowledge , we are the first to generalize single-agent FQI to the multi-agent setting ( i.e. , FMA-FQI ) for analyzing value factorization . For instance , we characterize the algorithmic properties of two specific value function classes , linear value factorization and IGM value factorization , which are widely used in deep MARL [ 10 , 13 , 19 ] . These factorization structures are implemented by certain network architectures and thus can be well modelled by FMA-FQI from the perspective of function approximation .

Summary: The authors present a theoretical framework based on the fitted Q-iteration (FQI) method as a common tool in analyzing two popular value factorization approaches in cooperative deep multi-agent Q-learning. Linear value factorization (LVF) and Individual-Global-Max (IGM) value factorization methods are studied from the FQI perspective, and their convergence properties are explained in theoretical manner. Some didactic experiments support their findings, and numerous empirical performance figures are reported to be consistent with those in previous works as well as with the theoretical proofs delivered in the appendix.