DOP: Off-Policy Multi-Agent Decomposed Policy Gradients

1 INTRODUCTION . Cooperative multi-agent reinforcement learning ( MARL ) has achieved great progress in recent years ( Hughes et al. , 2018 ; Jaques et al. , 2019 ; Vinyals et al. , 2019 ; Zhang et al. , 2019 ; Baker et al. , 2020 ; Wang et al. , 2020c ) . Advances in valued-based MARL ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ; Wang et al. , 2020e ) contribute significantly to the progress , achieving state-ofthe-art performance on challenging tasks , such as StarCraft II micromanagement ( Samvelyan et al. , 2019 ) . However , these value-based methods present a major challenge for stability and convergence in multi-agent settings ( Wang et al. , 2020a ) , which is further exacerbated in continuous action spaces . Policy gradient methods hold great promise to resolve these challenges . MADDPG ( Lowe et al. , 2017 ) and COMA ( Foerster et al. , 2018 ) are two representative methods that adopt the paradigm of centralized critic with decentralized actors ( CCDA ) , which not only deals with the issue of nonstationarity ( Foerster et al. , 2017 ; Hernandez-Leal et al. , 2017 ) by conditioning the centralized critic on global history and actions but also maintains scalable decentralized execution via conditioning policies on local history . Several subsequent works make improvements to the CCDA framework by introducing the mechanism of recursive reasoning ( Wen et al. , 2019 ) or attention ( Iqbal & Sha , 2019 ) . Despite the progress , most of the multi-agent policy gradient ( MAPG ) methods do not provide satisfying performance , e.g. , significantly underperforming value-based methods on benchmark tasks ( Samvelyan et al. , 2019 ) . In this paper , we analyze this discrepancy and pinpoint three major issues that hinder the performance of MAPG methods . ( 1 ) Current stochastic MAPG methods do not support off-policy learning , partly because using common off-policy learning techniques is computationally expensive in multi-agent settings . ( 2 ) In the CCDA paradigm , the suboptimality of one agent ’ s policy can propagate through the centralized joint critic and negatively affect policy learning of other agents , causing catastrophic miscoordination , which we call centralized-decentralized mismatch ( CDM ) . ( 3 ) For deterministic MAPG methods , realizing efficient credit assignment ( Tumer et al. , 2002 ; Agogino & Tumer , 2004 ) with a single global reward signal largely remains challenging . ∗Equal Contribution . Listing order is random . In this paper , we find that these problems can be addressed by introducing the idea of value decomposition into the multi-agent actor-critic framework and learning a centralized but factorized critic . This framework decomposes the centralized critic as a weighted linear summation of individual critics that condition on local actions . This decomposition structure not only enables scalable learning on the critic , but also brings several benefits . It enables tractable off-policy evaluations of stochastic policies , attenuates the CDM issues , and also implicitly learns an efficient multi-agent credit assignment . Based on this decomposition , we develop efficient off-policy multi-agent decomposed policy gradient methods for both discrete and continuous action spaces . A drawback of an linearly decomposed critic is its limited representational capacity ( Wang et al. , 2020b ) , which may induce bias in value estimations . However , we show that this bias does not violate the policy improvement guarantee of policy gradient methods and that using decomposed critics can largely reduce the variance in policy updates . In this way , a decomposed critic achieves a great bias-variance trade-off . We evaluate our methods on both the StarCraft II micromanagement benchmark ( Samvelyan et al. , 2019 ) ( discrete action spaces ) and multi-agent particle environments ( Lowe et al. , 2017 ; Mordatch & Abbeel , 2018 ) ( continuous action spaces ) . Empirical results show that DOP is very stable across different runs and outperforms other MAPG algorithms by a wide margin . Moreover , to our best knowledge , stochastic DOP provides the first MAPG method that outperforms state-of-the-art valuedbased methods in discrete-action benchmark tasks . Related works on value decomposition methods . In value-based MARL , value decomposition ( Guestrin et al. , 2002b ; Castellini et al. , 2019 ) is widely used . These methods learn local Q-value functions for each agent , which are combined with a learnable mixing function to produce global action values . In VDN ( Sunehag et al. , 2018 ) , the mixing function is an arithmetic summation . QMIX ( Rashid et al. , 2018 ; 2020 ) proposes a non-linear monotonic factorization structure . QTRAN ( Son et al. , 2019 ) and QPLEX ( Wang et al. , 2020b ) further extend the class of value functions that can be represented . NDQ ( Wang et al. , 2020e ) addresses the miscoordination problem by learning nearly decomposable architectures . A concurrent work ( de Witt et al. , 2020 ) finds that a decomposed centralized critic in QMIX style can improve the performance of MADDPG for learning in continuous action spaces . In this paper , we study how and why linear value decomposition can enable efficient policy-based learning in both discrete and continuous action spaces . In Appendix F , we discuss how DOP is related to recent progress in multi-agent reinforcement learning and provide detailed comparisons with existing multi-agent policy gradient methods . 2 BACKGROUND . We consider fully cooperative multi-agent tasks that can be modelled as a Dec-POMDP ( Oliehoek et al. , 2016 ) G=〈I , S , A , P , R , Ω , O , n , γ〉 , where I is the finite set of agents , γ ∈ [ 0 , 1 ) is the discount factor , and s ∈ S is the true state of the environment . At each timestep , each agent i receives an observation oi ∈ Ω drawn according to the observation function O ( s , i ) and selects an action ai ∈ A , forming a joint action a ∈ An , leading to a next state s′ according to the transition function P ( s′|s , a ) and a reward r = R ( s , a ) shared by all agents . Each agent learns a policy πi ( ai|τi ; θi ) , which is parameterized by θi and conditioned on the local history τi ∈ T ≡ ( Ω × A ) ∗ . The joint policy π , with parameters θ = 〈θ1 , · · · , θn〉 , induces a joint action-value function : Qπtot ( τ , a ) =Es0 : ∞ , a0 : ∞ [ ∑∞ t=0 γ tR ( st , at ) | s0=s , a0=a , π ] . We consider both discrete and continuous action spaces , for which stochastic and deterministic policies are learned , respectively . To distinguish deterministic policies , we denote them by µ = 〈µ1 , · · · , µn〉 . Multi-Agent Policy Gradients The centralized training with decentralized execution ( CTDE ) paradigm ( Foerster et al. , 2016 ; Wang et al. , 2020d ) has recently attracted attention for its ability to address non-stationarity while maintaining decentralized execution . Learning a centralized critic with decentralized actors ( CCDA ) is an efficient approach that exploits the CTDE paradigm . MADDPG and COMA are two representative examples . MADDPG ( Lowe et al. , 2017 ) learns deterministic policies in continuous action spaces and uses the following gradients to update policies : g = Eτ , a∼D [ ∑ i ∇θiµi ( τi ) ∇aiQ µ tot ( τ , a ) |ai=µi ( τi ) ] , ( 1 ) and D is a replay buffer . COMA ( Foerster et al. , 2018 ) updates stochastic policies using the gradients : g = Eπ [ ∑ i ∇θi log πi ( ai|τi ) Aπi ( τ , a ) ] , ( 2 ) where Aπi ( τ , a ) = Q π tot ( τ , a ) − ∑ a′i Qπtot ( τ , ( a-i , a ′ i ) ) is a counterfactual advantage ( a-i is the joint action other than agent i ) that deals with the issue of credit assignment and reduces variance . 3 ANALYSIS . In this section , we investigate challenges that limit the performance of state-of-the-art multi-agent policy gradient methods . 3.1 OFF-POLICY LEARNING FOR MULTI-AGENT STOCHASTIC POLICY GRADIENTS . Efficient stochastic policy learning in single-agent settings relies heavily on using off-policy data ( Lillicrap et al. , 2015 ; Wang et al. , 2016 ; Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) , which is not supported by existing stochastic MAPG methods ( Foerster et al. , 2018 ) . In the CCDA framework , off-policy policy evaluation—estimating Qπtot from data drawn from behavior policies β = 〈β1 , . . . , βn〉—encounters major challenges . Importance sampling ( Meuleau et al. , 2000 ; Jie & Abbeel , 2010 ; Levine & Koltun , 2013 ) is a simple way to correct for the discrepancy between π and β , but , it requires computing ∏ i πi ( ai|τi ) βi ( ai|τi ) , whose variance grows exponentially with the number of agents in multi-agent settings . An alternative is to extend the tree backup technique ( Precup et al. , 2000 ; Munos et al. , 2016 ) to multi-agent settings and use the k-step tree backup update target for training the critic : yTB = Qπtot ( τ , a ) + k−1∑ t=0 γt ( t∏ l=1 λπ ( al|τl ) ) [ rt + γEπ [ Qπtot ( τt+1 , · ) ] −Qπtot ( τt , at ) ] , ( 3 ) where τ0 = τ , a0 = a . However , the complexity of computing Eπ [ Qπtot ( τt+1 , · ) ] is O ( |A|n ) , which becomes intractable when the number of agents is large . Therefore , it is challenging to develop off-policy stochastic MAPG methods . 3.2 THE CENTRALIZED-DECENTRALIZED MISMATCH ISSUE . In the centralized critic with decentralized actors ( CCDA ) framework , agents learn individual policies , πi ( ai|τi ; θi ) , conditioned on the local observation-action history . However , the gradients for updating these policies are dependent on the centralized joint critic , Qπtot ( τ , a ) ( see Eq . 1 and 2 ) , which introduces the influence of actions of other agents . Intuitively , gradient updates will move an agent in the direction that can increase the global Q value , but the presence of other agents ’ actions incurs large variance in the estimates of such directions . Formally , the variance of policy gradients for agent i at ( τi , ai ) is dependent on other agents ’ actions : Vara-i∼π-i [ Q π tot ( τ , ( ai , a-i ) ) ∇θi log πi ( ai|τi ) ] =Vara-i∼π-i [ Q π tot ( τ , ( ai , a-i ) ) ] ( ∇θi log πi ( ai|τi ) ) ( ∇θi log πi ( ai|τi ) ) T , ( 4 ) where Vara-i [ Q π tot ( τ , ( ai , a-i ) ) ] can be very large due to the exploration or suboptimality of other agents ’ policies , which may cause suboptimality in individual policies . For example , suppose that the optimal joint action under τ is a∗= 〈a∗1 , . . . , a∗n〉 . When Ea-i∼π-i [ Qπtot ( τ , ( a∗i , a-i ) ) ] < 0 , πi ( a∗i |τi ) will decrease , possibly resulting in a suboptimal πi . This becomes problematic because a negative feedback loop is created , in which the joint critic is affected by the suboptimality of agent i , which disturbs policy updates of other agents . We call this issue centralized-decentralized mismatch ( CDM ) . Does CDM occur in practice for state-of-the-art algorithms ? To answer this question , we carry out a case study in Sec . 5.1 . We can see that the variance of DOP gradients is significantly smaller than COMA and MADDPG ( Fig . 2 left ) . This smaller variance enables DOP to outperform other algorithms ( Fig . 2 middle ) . We will explain this didactic example in detail in Sec . 5.1 . In Sec . 5.2 and 5.3 , we further show that CDM is exacerbated in sequential decision-making settings , causing divergence even after a near-optimal strategy has been learned .

Summary: In the context centralized training distributed execution in cooperative multi-agent reinforcement learning (MARL), the paper proposes an architecture to learn a decomposed action value function expressed as a weighted sum of the agent's individual functions (plus an additional weight). Those weights are themselves learned and depend on the observed history. Thanks to this decomposition, gradients can be decomposed over each agent. The authors propose to use a combination of off-policy (using tree backup) and on-policy (using TD(\lambda) methods for estimating the decomposed critic. They formulate both a deterministic and stochastic decomposed policy gradients, which are analyzed theoretically to some extent and evaluated experimentally.