VAST: Value Function Factorization with Variable Agent Sub-Teams

1 Introduction . Many real-world problems can be defined as cooperative multi-agent system ( MAS ) , where multiple autonomous agents collaborate to achieve a common goal like fleet management [ 20 , 21 ] , industry 4.0 [ 9 , 29 , 43 ] , or communication networks [ 26 , 51 ] . Multi-agent reinforcement learning ( MARL ) seems promising to realize such cooperative MAS by learning local policies for each autonomous agent [ 3 , 27 , 37 , 40 ] . Multi-agent credit assignment is an important challenge , where all agents only observe a single global reward , which makes the deduction of individual agent contributions difficult , especially in large MAS with many agents . This can lead to poor policies , since it is unclear which agent policy needs to adapt to what extent in order to improve global MAS behavior [ 6 , 10 , 39 ] . Value function factorization ( VFF ) via end-to-end deep learning is a popular approach to MARL in order to address the credit assignment problem [ 31 , 32 , 36 , 39 , 44 ] . A centralized value function is learned from global rewards and factorized into local value functions , which can be used to realize coordinated local policies via multi-armed bandits [ 32 , 40 ] or local actor-critic learning [ 30 , 38 , 45 ] . Despite the popularity of VFF , most approaches have been only evaluated in domains with a handful of agents . We hypothesize that this is due to the flat factorization scheme of current VFF approaches ( Fig . 1a ) . With an increasing number of agents , the centralized VFF operator becomes a performance bottleneck , where it gets difficult to provide sufficiently informative training signal for each agent . To alleviate this performance bottleneck problem , we propose VFF with variable agent sub-teams ( VAST ) . Instead of directly factorizing a centralized value function for each agent , VAST approximates 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . a factorization for agent sub-teams which can be defined in an arbitrary way and vary over time , e.g. , to adapt to different situations . The sub-team values are then linearly decomposed for all sub-team members as illustrated in Fig . 1b . Therefore , VAST can learn on a more focused and compact input representation of the original VFF operator . Our contributions are as follows : • We formulate VAST and show that VAST maintains decentralizability like state-of-the-art VFF given any sub-team assignment and depending on the sub-team based VFF operator . • We propose a meta-gradient approach to optimize sub-team assignments in order to adapt and improve VAST . We also briefly discuss alternative sub-team assignment strategies . • We empirically evaluate different variants of VAST in three multi-agent domains and show that VAST can significantly outperform flat state-of-the-art VFF approaches by alleviating the performance bottleneck problem , when the number of agents is sufficiently large . 2 Background . We model cooperative MAS as partially observable Markov game M = 〈D , S , A , P , R , Z , Ω〉 , where D = { 1 , ... , N } is a set of agents i , S is a set of states st at time step t , A = 〈Ai〉i∈D is the set of joint actions at = 〈at , i〉i∈D = 〈at,1 , ... , at , N 〉 , P ( st+1|st , at ) is the transition probability , rt = R ( st , at ) ∈ R is the global reward , Z is a set of local observations zt , i for each agent i , and Ω ( st , at ) = zt+1 = 〈zt+1 , i〉i∈D ∈ ZN is the subsequent joint observation . Each agent i maintains a local history τt , i ∈ ( Z×Ai ) t and τt = 〈τt , i〉i∈D is the joint history . π ( at|τt ) = ∏ i∈D πi ( at , i|τt , i ) is the ( joint ) action probability of joint policy π , where πi is the local policy of agent i. π can be evaluated with a value function Qπ ( st , at ) = Eπ [ Rt|st , at ] , ∀st ∈ S , ∀at ∈ A , where Rt = ∑∞ c=0 γ crt+c is the return with γ ∈ [ 0 , 1 ) . The goal is to find an optimal joint policy π∗ = 〈π∗i 〉i∈D with Qπ ∗ = Q∗ = maxπQ π . If Q∗ is known , then π∗ can be obtained by greedily maximizing Q∗ . Note : Expressions of the form 〈ei〉i∈I denote unordered sets , where ei is mapped to exactly one identifier ( e.g. , an agent ) i ∈ I . Thus , we implicitly assume the order of agents to be irrelevant . 2.1 Independent Learning of Value Functions . Q∗ can be approximated independently by each agent i ∈ D using naive decentralized MARL on at , i and τt , i [ 10 , 18 , 40 ] . These local approximations Qi ∼ Q∗ can be used to realize local polices πi for each agent i by using , e.g. , multi-armed bandits on Qi [ 32 , 40 ] or actor-critic learning with Qi as critic [ 10 ] . Independent Learning ( IL ) offers optimal scalability w.r.t . N but violates the Markov assumption due to non-stationarity caused by simultaneously learning agents [ 17 , 36 ] . 2.2 Value Function Factorization . For many problems , training usually takes place in a laboratory or in a simulated environment , where global information is available . State-of-the-art MARL exploits this fact to approximate a centralized value function Qtot ∼ Q∗ , which conditions on joint histories τt and joint actions at ( and optionally on global states st ) . However , Qtot is only required during training in order to realize local policies πi , which can be used in a decentralized way because they only condition on the local history τt , i . This paradigm is known as centralized training and decentralized execution ( CTDE ) [ 10 , 32 , 36 ] . Qtot can be factorized into local value functions 〈Qi〉i∈D via a VFF operator Ψ as shown in Fig . 1a : Qtot ( τt , at ) = Ψ ( Q1 ( τt,1 , at,1 ) , ... , QN ( τt , N , at , N ) ) ( 1 ) In practice , Ψ is realized with deep neural networks , such that 〈Qi〉i∈D can be learned end-to-end via backpropagation by minimizing the mean squared TD ( λ ) ( temporal difference ) error [ 32 , 36 , 39 ] . A VFF operator Ψ is decentralizable when satisfying the IGM ( Individual-Global-Max ) such that [ 36 ] : argmaxat∈AQtot ( τt , at ) = 〈argmaxat , i∈AiQi ( τt , i , at , i ) 〉i∈D ( 2 ) VDN ( Value Decomposition Networks ) [ 39 ] formulates ΨVDN as linear sum such that Qtot ( τt , at ) = ΨVDN ( · ) = ∑ i∈D Qi ( τt , i , at , i ) , which satisfies the IGM for Qtot and 〈Qi〉i∈D [ 36 ] . QMIX [ 32 ] formulates ΨQMIX as a nonlinear monotonic combination of 〈Qi〉i∈D with a mixing network . The mixing network is generated by hypernetworks [ 12 ] and has nonnegative weights to satisfy the monotonicity condition δQtotδQi ≥ 0 , ∀i ∈ D to maintain consistency w.r.t . the IGM [ 32 , 36 ] . Weighted QMIX further improves QMIX by weighting the transition losses w.r.t . over- and underestimation of Qtot [ 31 ] QTRAN [ 36 ] avoids the additivity and monotonicity constraints of VDN and QMIX respectively by formulating the more general ΨQTRAN , which aims to satisfy∑ i∈D Qi ( τt , i , at , i ) −Qtot ( τt , at ) + Vtot ( τt ) = { = 0 , at = at ≥ 0 , at 6= at ( 3 ) where at = 〈at , i〉i∈D = 〈at,1 , ... , at , N 〉 with at , i = argmaxat , i∈AiQi ( τt , i , at , i ) and Vtot ( τt ) = maxat∈AQtot ( τt , at ) − ∑ i∈D Qi ( τt , i , at , i ) , in order to be consistent w.r.t . the IGM . 3 Related Work . MARL is a long-standing research area with rapid progress towards complex domains [ 3 , 11 , 40 , 42 ] . Most state-of-the-art approaches are based on CTDE to learn Qtot for actor-critic learning [ 10 , 22 ] or VFF [ 31 , 32 , 36 , 39 , 44 ] . VFF approaches like VDN , QMIX , and QTRAN use a flat factorization scheme , where Qtot is directly factorized into 〈Qi〉i∈D as illustrated in Fig . 1a . We introduce a hierarchical VFF approach based on agent sub-teams which can vary over time , e.g. , to adapt to different situations . With that , we can improve performance in large MAS , where flat VFF approaches could fail due to Ψ becoming a performance bottleneck . Prior work on hierarchical MARL has mainly focused on temporal abstraction , where the MAS attempts to solve tasks based on temporal subgoals or roles [ 24 , 49 ] . We focus on VFF applied to agent sub-teams , which can be regarded as an abstraction of agents . These abstractions or sub-teams can vary over time , depending on the sub-team assignment strategy which may be chosen arbitrarily . Approaches based on coordination graphs enrich VFF with agent relationship information and focus on pairwise interactions of agents to learn local value functions [ 1 , 19 ] . The maximization of Qtot scales at least quadratically w.r.t . N based on the graph structure . Our approach simplifies VFF via agent abstraction and maximizes Qtot with linear complexity . The abstraction is based on variable agent sub-teams , which are not restricted to pairwise interactions . There is some prior work on sub-team assignments and agent-based hierarchization in MAS : The relationship between coordination , complexity , and performance depending on predefined organizational MAS structures was studied in [ 4 , 8 , 15 , 34 , 35 ] . [ 20 ] proposed a contextual MARL framework for fleet management , where the spatial environment is partitioned into fixed cells with locally assigned rewards and the number of agents per cell can vary over time . [ 16 ] proposed an attention-based mechanism for self-interested MAS to focus on different contextual information per agent in order to approximate Qi . Mean field MARL was introduced in [ 50 ] , where Qi is learned based on the mean field approximation of the joint action of all neighbor agents , where the definition of `` neighborhood '' is domain dependent . Different clustering approaches for agents , communication messages , etc . w.r.t . some similarity criteria have been proposed in [ 5 , 25 , 46 ] . Our approach addresses the performance bottleneck problem of flat VFF approaches . It can be used with any sub-team assignment strategy like random assignments , clustering , or meta-learning to structure the MAS dynamically while satisfying the IGM for Qtot and 〈Qi〉i∈D like flat state-of-the-art VFF approaches [ 32 , 36 , 39 ] . Unlike [ 16 , 20 , 50 ] , our approach does not depend on predefined local rewards per agent or region but automatically approximates local value functions Qi from global rewards via sub-team based VFF .

Summary: The paper introduces VAST, a framework to train decentralised MARL agents to each learn local policies from a single global reward. In contrast to VFF, where each agent calculates a local Q function which is then aggregated in a flat way into a total Q function to which Deep Q-Learning may be applied, VAST first assigns agents into variable-agent sub-teams, linearly aggregates the local Q function of each sub-team into a group Q function, and then aggregates the group function into a total Q function in the manner of VFF. The authors hypothesise that VFF underperforms with large numbers of agents due to a performance bottleneck that limits the information received by each agent, and empirically demonstrate a range of experiments where VAST achieves superior performance to flat versions of VFF when training large numbers of agents. They further discuss different ways of assigning agents to sub-teams, which may vary between time steps, and introduce an effective meta-gradient based approach.