Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization

1 Introduction . Cooperative multi-agent reinforcement learning ( MARL ) has great promise for addressing coordination problems in a variety of applications , such as robotic systems [ 1 ] , autonomous cars [ 2 ] , and sensor networks [ 3 ] . Such complex tasks often require MARL to learn decentralized policies for agents to jointly optimize a global cumulative reward signal , which posts a number of challenges , including multi-agent credit assignment [ 4 , 5 ] , non-stationarity [ 6 , 7 ] , and scalability [ 3 , 8 ] . Recently , by leveraging the strength of deep learning techniques , cooperative MARL has made great progress [ 9–19 ] , particularly in value-based methods that demonstrate state-of-the-art performance on challenging tasks such as StarCraft unit micromanagement [ 20 ] . Value factorization is a popular approach to effectively scaling up cooperative multi-agent Q-learning in complex domains . One fundamental problem of factorized multi-agent algorithms is the tradeoff between the learning scalability and the representational capacity of value functions . As a representative method , value-decomposition network ( VDN ) [ 10 ] is proposed to obtain excellent scalability based on a popular paradigm called centralized training with decentralized execution ( CTDE ) [ 21 ] , which learns a centralized but factorizable joint value function Qtot represented as the summation of individual value functionsQi . During the execution , decentralized policies can be easily derived by greedily selecting individual actions from the local value function Qi . An implicit multiagent credit assignment is realized because Qi is learned by neural network backpropagation from ∗Equal contribution . †Work done while Zhizhou was an undergraduate at Tsinghua University . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the total temporal-difference error on the single global reward signal . This linear value factorization structure can significantly improve the scalability of multi-agent joint policy training and individual policy execution . This linear value factorization adopted by VDN also realizes a sufficient condition for an important principle of CTDE , the IGM ( Individual-Global-Max ) principle [ 13 ] , which asserts the consistency between joint and individual greedy action selection and ensures the consistent policy learning under the CTDE paradigm . Following this principle , most recent advances focus on enriching the function expressiveness of the factorization from Qtot to Qi . QMIX [ 12 ] uses a monotonic function to represent the joint value function Qtot through local values Qi , whose function expressiveness is further improved by QTRAN [ 13 ] and QPLEX [ 19 ] that aim to achieve full function expressiveness induced by the IGM principle . These approaches show promise and achieve state-of-the-art performance in complicated cooperative multi-agent domains . Despite these impressive empirical successes , theoretical understandings for these MARL approaches are still limited . To bridge this gap , this paper is the first to consider a general framework for theoretical studies . We formulate Factorzed Multi-Agent Fitted Q-Iteration ( FMA-FQI ) for formally analyzing cooperative MARL with value factorization . This framework generalizes single-agent Fitted Q-Iteration , a popular model for studying Q-learning algorithms with function approximation [ 22–25 ] . FMA-FQI models the iterative training procedure of multi-agent Q-learning using empirical Bellman error minimization . Under this framework , we investigate two popular value factorization methods : the linear factorization used by VDN [ 10 ] , and the IGM factorization adopted by QTRAN [ 13 ] and QPLEX [ 19 ] . Our analyses reveal algorithmic properties of these approaches and provide insights on fundamental questions : Why do they work ? Do they converge ? When may they not perform well ? The main contribution of this paper can be summarized as follows : 1 . We formalize Factorized Multi-Agent Fitted Q-Iteration ( FMA-FQI ) as a general theoretical framework for analyzing cooperative multi-agent Q-learning with value factorization . 2 . Based on FMA-FQI , we study linear value factorization and derive a closed-form solution to its Bellman error minimization . We then reveal two novel insights that : 1 ) linear value factorization implicitly realizes a powerful counterfactual credit assignment and 2 ) on-policy training is beneficial to its stability and local convergence near the optimal solutions . 3 . Using FMA-FQI , we also study IGM value factorization and prove its global convergence and optimality . 4 . Empirical analysis is conducted to connect our theoretical implications to practical scenarios by evaluating state-of-the-art deep MARL approaches . 2 Related Work . Deep Q-learning algorithms that use neural networks as function approximators have shown great promise in solving complicated decision-making problems [ 26 ] . One of the core components of such methods is iterative Bellman error minimization , which can be modelled by a classical framework called Fitted Q-Iteration ( FQI ) [ 27 ] . FQI utilizes a specific Q-function class to iteratively optimize empirical Bellman error on a dataset D with respect to a frozen target . Great efforts have been made towards theoretically characterizing the behavior of FQI with finite samples and imperfect function classes [ 22–24 ] . From an empirical perspective , there is also a growing trend to adopt FQI for empirical analysis of deep offline Q-learning algorithms [ 25 , 28 ] . In MARL , the joint Qfunction class grows exponentially with the number of agents , leading many algorithms [ 10 , 13 , 19 ] to utilize different value factorization structures to balance scalability and optimality . In section 4 , we generalize FQI to formalize a factorized multi-agent fitted Q-iteration framework for analyzing cooperative multi-agent Q-learning algorithms with value factorization . To achieve superior effectiveness and scalability in multi-agent settings , centralized training with decentralized executing ( CTDE ) has become a popular MARL paradigm [ 29 , 30 ] . Individual-GlobalMax ( IGM ) principle [ 13 ] is a critical concept for value-based CTDE [ 31 ] , that ensures the consistency between joint and local greedy action selections and enables effective performance in both training and execution phases . VDN [ 10 ] utilizes linear value factorization to satisfy a sufficient condition of IGM . This simple linear structure of VDN has become very popular in MARL due to its excellent scalability [ 13 , 19 , 32 ] and inspired many follow-up methods . QMIX [ 12 ] proposes a monotonic Q-network structure to improve the expressiveness of the factorized function class . QTRAN [ 13 ] aims to realize the entire IGM function class , but its proposed objective is computationally intractable and requires two extra soft regularizations to approximate IGM , which actually loses rigorous IGM guarantees . QPLEX [ 19 ] , the state-of-the-art multi-agent Q-learning algorithm , encodes the IGM principle into the Q-network architecture and realize a complete IGM function class , but may have potential limitations in scalability . This paper focuses on the theoretical and empirical understanding of multi-agent Q-learning with linear and IGM value factorization to explore the underlying implications of their corresponding factorized value structures . 3 Notations and Preliminaries . 3.1 Decentralized Rich-Observation Markov Decision Process ( Dec-ROMDP ) . To support theoretical analysis on cooperative multi-agent Q-learning , we consider the problem formulation of Decentralized Rich-Observation Markov Decision Process ( Dec-ROMDP ) . DecROMDP is an interpolation of ROMDP [ 33 , 34 ] and Dec-POMDP [ 35 ] , in which the latent state space is finite , and the observation space could be arbitrarily large . A Dec-ROMDP is defined as a tupleM = 〈N , S , X , A , P , Λ , r , γ〉 . N ≡ { 1 , . . . , n } is a finite set of agents . S is a finite set of global latent states . X denotes the observation space ( a.k.a . context space ) which is larger than the latent state space . A denotes the action space for an individual agent . The joint action a ∈ A ≡ An is a collection of individual actions [ ai ] ni=1 . At each timestep t , a selected joint action at results in a latent-state transition st+1 ∼ P ( ·|st , at ) and a global reward signal r ( st , at ) . The observation of agent i is generated from the emission distribution xt , i ∼ Λ ( ·|i , st ) . The major difference between Dec-ROMDP and Dec-POMDP is the concept of rich observation . Dec-ROMDP ensures that the observation emission distributions are disjoint across latent states , i.e. , there exists an inverse function Λ−1 : N × X → S that decodes full information of latent states from observations . This assumption enables us to consider reactive function classes [ 36 , 37 ] , which only takes the latest observation as inputs . Formally , the goal for MARL in Dec-ROMDP is to construct a joint policy π = 〈π1 , . . . , πn〉 maximizing expected discounted rewards V π ( x ) = E [ ∑∞ t=0 γ tr ( st , π ( xt ) ) |x0 = x ] , where πi : X 7→ A denotes an individual policy of agent i which depends on its local observation . The corresponding action-value function is denoted as Qπ ( xt , at ) = E [ r ( st , at ) + γV π ( xt+1 ) |xt ] . More discussions are deferred to Appendix A . Why Dec-ROMDP ? The most widely-used environment setting in deep MARL for empirical studies is Dec-POMDP . However , as implied by prior work , finding an optimal policy in infinitehorizon Dec-POMDPs is an incomputable problem in the classical computation model [ 38 ] . There is no algorithm that can achieve strong theoretical guarantees in the Dec-POMDP setting . Following recent advances in reinforcement learning with function approximation [ 33 , 34 , 39 , 40 ] , we consider the rich-observation setting to derive implications from a theoretical perspective . This setting is also adopted by a concurrent work to establish algorithm analysis for cooperative multi-agent reinforcement learning [ 41 ] . 3.2 Centralized Training with Decentralized Execution ( CTDE ) . Most deep multi-agent Q-learning algorithms with value factorization adopt the paradigm of centralized training with decentralized execution [ 21 ] . In the training phase , the centralized trainer can access all global information , including joint trajectories , shared global rewards , agents ’ policies , and value functions . In the decentralized execution phase , every agent makes individual decisions based on its local information . Individual-Global-Max ( IGM ) [ 13 ] is a common principle to realize effective decentralized policy execution . It enforces the action selection consistency between the global joint action-value Qtot and individual action-values [ Qi ] ni=1 , which are specified as follows : ( IGM ) arg max a∈A Qtot ( x , a ) = 〈 arg max a1∈A Q1 ( x1 , a1 ) , · · · , arg max an∈A Qn ( xn , an ) 〉 . ( 1 ) QPLEX [ 19 ] is the state-of-the-art deep multi-agent Q-learning algorithm that realizes a complete IGM factorization . QTRAN [ 13 ] is another method to approximate the IGM principle by two extra soft regularizations . Moreover , as stated in Eq . ( 2 ) , the linear value factorization adopted by VDN [ 10 ] is a sufficient condition for the IGM constraint stated in Eq . ( 1 ) but is not a necessary condition , which induces a limited joint action-value function class . ( Additivity ) Qtot ( x , a ) = n∑ i=1 Qi ( xi , ai ) . ( 2 ) 4 Factorized Multi-Agent Fitted Q-Iteration . Constructing a specific factorized value function class satisfying the IGM condition is a critical step to realize the centralized training with decentralized execution ( CTDE ) paradigm . In the literature of Q-learning with function approximation , Fitted Q-Iteration ( FQI ) [ 27 ] is a popular framework for both theoretical and empirical studies [ 22–25 , 28 ] . In this section , we generalize single-agent FQI to formalize a Factorzed Multi-Agent Fitted Q-Iteration ( FMA-FQI ) framework , which enables us to characterize the algorithmic properties of different factorization structures in the CTDE paradigm . For multi-agent Q-learning with value factorization , we use Qtot to denote the global but factorized value function , which can be represented as a function of individual value functions [ Qi ] ni=1 . In other words , we can use [ Qi ] ni=1 to represent Qtot . For brevity , we overload Q = 〈Qtot , [ Qi ] ni=1〉 to denote their association . Following fitted Q-iteration , FMA-FQI considers an iterative optimization framework based on a given dataset D = { ( x , a , r , x′ ) } as presented in Algorithm 1 . Algorithm 1 Factorized Multi-Agent Fitted Q-Iteration ( FMA-FQI ) 1 : Input : dataset D , the number of iterations T , factorized multi-agent value function class QFMA 2 : Randomly initialize Q ( 0 ) from QFMA 3 : for t = 0 . . . T − 1 do 4 : Iteratively update Q ( t+1 ) ← T FMAD Q ( t ) ≡ arg min Q∈QFMA E ( x , a , r , x′ ) ∼D ( ŷ ( t ) ( x , a , x′ ) −Qtot ( x , a ) ) 2 ( 3 ) where ŷ ( t ) ( x , a , x′ ) = r + γmaxa′∈AQ ( t ) tot ( x ′ , a′ ) denotes the one-step TD target . 5 : Construct policies by individual value functions ∀i ∈ N , π ( t+1 ) i ( xi ) = arg max ai∈A Q ( t+1 ) i ( xi , ai ) ( 4 ) 6 : Return : [ π ( T ) i ] ni=1 Formally , FMA-FQI specifies the Q-function class through a multi-agent value factorization structure : QFMA = { Q ∣∣∣ ∃f ∈ FFMA , Qtot ( x , a ) = f ( x , a , [ Qi ] ni=1 ) , [ Qi ] ni=1 ∈ R|X×A|n } , ( 5 ) where FFMA denotes the function class characterizing the factorization structure . For example , the IGM constraint stated as Eq . ( 1 ) and the additivity constraint stated as Eq . ( 2 ) correspond to the value factorization structures used by QPLEX [ 19 ] and VDN [ 10 ] , respectively . The empirical Bellman error minimization in FMA-FQI is established using the global reward signals . To facilitate further discussions , we rewrite the regression objective as follows : Q ( t+1 ) ← T FMAD Q ( t ) ≡ arg min Q∈QFMA E ( x , a ) ∼D ( y ( t ) ( x , a ) −Qtot ( x , a ) ) 2 ( 6 ) where y ( t ) ( x , a ) = r+γEx′ [ maxa′ Q ( t ) tot ( x ′ , a′ ) ] denotes the expected one-step TD target . Assume the dataset is adequate , the equivalence between Eq . ( 3 ) and Eq . ( 6 ) is proved in Appendix A.3 . Modeling CTDE by FMA-FQI . Different from the single-agent case , FMA-FQI considers the multi-agent value factorization structure , Qtot and [ Qi ] ni=1 , to support effective training and scalable execution . In the global reward setting , the shared reward signal can only supervise the training of the joint value function Qtot ( see Eq . ( 6 ) ) . The learned joint value function Qtot would be automatically decomposed to individual value functions [ Qi ] ni=1 through the factorized function structure , which can be further used to perform decentralized execution ( see Eq . ( 4 ) ) . In practice , FMA-FQI illustrated in Eq . ( 5 ) provides the end-to-end learning of [ Qi ] ni=1 as the iterative optimization of joint value function Qtot . The greedy action selection can be searched in the individual action space A rather than the joint action space An , which significantly reduces the computation costs . Relation to Prior Work . Most prior work of FQI focus on the single-agent RL with finite-sample analyses and exploring minimum requirements for performance guarantees ( e.g. , data distribution and function capacity ) in a general sense [ 22–24 ] . In comparison , this paper is motivated by recent advances in the multi-agent area . To our best knowledge , we are the first to generalize single-agent FQI to the multi-agent setting ( i.e. , FMA-FQI ) for analyzing value factorization . For instance , we characterize the algorithmic properties of two specific value function classes , linear value factorization and IGM value factorization , which are widely used in deep MARL [ 10 , 13 , 19 ] . These factorization structures are implemented by certain network architectures and thus can be well modelled by FMA-FQI from the perspective of function approximation .

Summary: This paper investigates Q-learning algorithms that factorise the joint-action Q-values in a cooperative MARL setting under CTDE. Specifically, a fitted Q-iteration framework is used in order to focus on the theoretical properties associated with these approaches. The theoretical analysis provides a better understanding of linear value factorisation: how it can be viewed as performing implicit credit assignment, situations where it can diverge, and the stabilising effect of training with on-policy data. Additionally, the paper theoretically demonstrates the benefits of being able to represent a wider class of joint-action Q-values by proving convergence in the same setting.