Divergence-Regularized Multi-Agent Actor-Critic

1 INTRODUCTION . Regularization is a common method for single-agent reinforcement learning ( RL ) . The optimal policy learned by traditional RL algorithm is always deterministic ( Sutton and Barto , 2018 ) . This property may result in the inflexibility of the policy facing with unknown environments ( Yang et al. , 2019 ) . Entropy regularization is proposed to settle this problem by learning a policy according to the maximum-entropy principle ( Haarnoja et al. , 2017 ) . Moreover , entropy regularization is beneficial to exploration and robustness for RL algorithms ( Haarnoja et al. , 2018 ) . However , entropy regularization is imperfect . Eysenbach and Levine ( 2019 ) pointed out maximum-entropy RL is a modification of the original RL objective because of the entropy regularizer . Maximum-entropy RL is actually learning an optimal policy for the entropy-regularized Markov Decision Process ( MDP ) rather than the original MDP , i.e. , the converged policy may be biased . Nachum et al . ( 2017 ) analysed a more general case for regularization in RL and proposed what we call divergence regularization . Divergence regularization can avoid the bias of the converged policy as well as be beneficial to exploration . Wang et al . ( 2019 ) employed divergence regularizer and proposed a single-agent RL algorithm , DAPO , which prevents the altering-objective drawback of entropy regularization . Regularization can also be applied to cooperative multi-agent reinforcement learning ( MARL ) ( Agarwal et al. , 2020 ; Zhang et al. , 2021 ) . However , most of cooperative MARL algorithms do not use regularizer ( Lowe et al. , 2017 ; Foerster et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ; Jiang et al. , 2020 ; Wang et al. , 2021a ) . Only few cooperative MARL algorithms such as FOP ( Zhang et al. , 2021 ) use entropy regularization , which may suffer from the drawback aforementioned . Divergence regularization , on the other hand , could potentially benefit cooperative MARL . In addition to its advantages mentioned above , divergence regularization can also help to control the step size of policy update which is similar to conservative policy iteration ( Kakade and Langford , 2002 ) in single-agent RL . Conservative policy iteration and its successive methods such as TRPO ( Schulman et al. , 2015 ) and PPO ( Schulman et al. , 2017 ) can stabilize policy improvement ( Touati et al. , 2020 ) . These methods use a surrogate objective for policy update , but decentralized policies in centralized training with decentralized execution ( CTDE ) paradigm may not preserve the properties of the surrogate objective . Moreover , DAPO ( Wang et al. , 2019 ) can not be trivially extended to cooperative MARL settings . Even with some tricks like V-trace ( Espeholt et al. , 2018 ) for off-policy correction , DAPO is essentially an on-policy algorithm and thus may not be sample-efficient in cooperative MARL settings . In the paper , we propose and analyze divergence policy iteration in general cooperative MARL settings and a special case combined with value decomposition . Based on divergence policy iteration , we derive the off-policy update rule for the critic , policy , and target policy and propose divergenceregularized multi-agent actor-critic ( DMAC ) , a novel off-policy cooperative MARL framework . We theoretically show that DMAC guarantees a monotonic policy improvement and is not biased by the regularization . Besides , DMAC is beneficial to exploration and stable policy improvement by applying our update rule of target policy . Moreover , DMAC is a flexible framework and can be combined with many existing cooperative MARL algorithms to substantially improve their performance . We empirically investigate DMAC in a didactic stochastic game and StarCraft Multi-Agent Challenge ( Samvelyan et al. , 2019 ) . We combine DMAC with five representative MARL methods , i.e. , COMA ( Foerster et al. , 2018 ) for on-policy multi-agent policy gradient , MAAC ( Iqbal and Sha , 2019 ) for off-policy multi-agent actor-critic , QMIX ( Rashid et al. , 2018 ) for value decomposition , DOP ( Wang et al. , 2021b ) for the combination of value decomposition and policy gradient , and FOP ( Zhang et al. , 2021 ) for the combination of value decomposition and entropy regularization . Experimental results show that DMAC indeed induces better performance , faster convergence , and better stability in most tasks , which verifies the benefits of DMAC and demonstrates the advantages of divergence regularization over entropy regularization in cooperative MARL . 2 RELATED WORK . MARL . MARL has been a hot topic in the field of RL . In this paper , we focus on cooperative MARL . Cooperative MARL is usually modeled as Dec-POMDP ( Oliehoek et al. , 2016 ) , where all agents share a reward and aim to maximize the long-term return . Centralized training with decentralized execution ( CTDE ) ( Lowe et al. , 2017 ) paradigm is widely used in cooperative MARL . CTDE usually utilizes a centralized value function to address the non-stationarity for multi-agent settings and decentralized policies for scalability . Many MARL algorithms adopt CTDE paradigm such as COMA , MAAC , QMIX , DOP and FOP . COMA ( Foerster et al. , 2018 ) employs the counterfactual baseline which can reduce the variance as well as settle the credit assignment problem . MAAC ( Iqbal and Sha , 2019 ) uses self-attention mechanism to integrate local observation and action of each agent and provides the structured information for the centralized critic . Value decomposition ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ; Yang et al. , 2020 ; Wang et al. , 2021a ; b ; Zhang et al. , 2021 ) is a popular class of cooperative MARL algorithms . These methods express the global Q-function as a function of individual Q-functions to satisfy Individual-Global-Max ( IGM ) , which means the optimal actions of individual Q-functions are corresponding to the optimal joint action of global Q-function . QMIX ( Rashid et al. , 2018 ) is a representative of value decomposition methods . It uses a hypernet to ensure the monotonicity of the global Q-function in terms of individual Q-functions , which is a sufficient condition of IGM . DOP ( Wang et al. , 2021b ) is a method that combines value decomposition with policy gradient . DOP uses a linear value decomposition which is another sufficient condition of IGM and the linear value decomposition helps the compute of policy gradient . FOP ( Zhang et al. , 2021 ) is a method that combines value decomposition with entropy regularization and uses a more general condition , Individual-Global-Optimal , to replace IGM . In this paper , we will combine DMAC with these algorithms and show its improvement . Regularization . Entropy regularization was first proposed in single-agent RL . Nachum et al . ( 2017 ) analyzed the entropy-regularized MDP and revealed the properties about the optimal policy and the corresponding Q-function and V-function . They also showed the equivalence of value-based methods and policy-based methods in entropy-regularized MDP . Haarnoja et al . ( 2018 ) pointed out maximum-entropy RL can achieve better exploration and stability facing with the model and estimation error . Although entropy regularization has many advantages , Eysenbach and Levine ( 2019 ) showed entropy regularization modifies the MDP and results in the bias of the convergent policy . Yang et al . ( 2019 ) revealed the drawbacks of the convergent policy of general RL and maximum-entropy RL . The former is usually a deterministic policy ( Sutton and Barto , 2018 ) which is not flexible enough for unknown situations , while the latter is a policy with non-zero probability for all actions which may be dangerous in some scenarios . Neu et al . ( 2017 ) analyzed the entropy regularization method from several views . They revealed a more general form of regularization which is actually divergence regularization and showed entropy regularization is just a special case of divergence regularization . Wang et al . ( 2019 ) absorbed previous result and proposed an on-policy algorithm , i.e. , DAPO . However , DAPO can not be trivially applied to MARL . Moreover , its on-policy learning is not sample-efficient for MARL settings and its off-policy correction trick V-trace ( Espeholt et al. , 2018 ) is also intractable in MARL . There are some previous studies in single-agent RL which use similar target policy to ours , but their purposes are quite different . Trust-PCL ( Nachum et al. , 2018 ) introduces a target policy as a trust region constraint for maximum-entropy RL , but the policy is still biased by entropy regularizer . MIRL ( Grau-Moya et al. , 2019 ) uses a distribution which is only related to actions as the target policy to compute a mutual-information regularizer , but it still changes the objective of the original RL . 3 PRELIMINARIES . Dec-POMDP is a general model for cooperative MARL . A Dec-POMDP is a tuple M = { S , A , P , Y , O , I , n , r , γ } . S is the state space , n is the number of agents , γ is the discount factor , and I = { 1 , 2 · · ·n } is the set of all agents . A = A1 ×A2 × · · · ×An represents the joint action space where Ai is the individual action space for agent i. P ( s′|s , a ) : S × A × S → [ 0 , 1 ] is the transition function , and r ( s , a ) : S×A→ R is the reward function of state s and joint action a. Y is the observation space , and O ( s , i ) : S× I → Y is a mapping from state to observation for each agent . The objective of Dec-POMDP is to maximize J ( π ) = Eπ [ ∑ t=0 γ tr ( st , at ) ] , and thus we need to find the optimal joint policy π∗ = arg maxπ J ( π ) . To settle the partial observable problem , history τi ∈ Ti = ( Y × Ai ) ∗ is often used to replace observation oi ∈ Y . As for policies in CTDE , each agent i has an individual policy πi ( ai|τi ) and the joint policy π is the product of each πi . Though we calculate individual policy as πi ( ai|τi ) in practice , we will use πi ( ai|s ) in analysis and proofs for simplicity . Entropy regularization adds the logarithm of current policy to the reward function . It modifies the optimization objective as Jent ( π ) = Eπ [ ∑ t=0 γ t ( r ( st , at ) − λ logπ ( at|st ) ) ] . We also have the corresponding Q-function Qπent ( s , a ) = r ( s , a ) + γE [ V πent ( s′ ) ] and V-function V πent ( s ) = E [ Qπent ( s , a ) − λ logπ ( a|s ) ] . Given these definitions , we can deduce an interesting property V πent ( s ) = E [ Qπent ( s , a ) ] + λH ( π ( ·|s ) ) , whereH ( π ( ·|s ) ) represents the entropy of policy π ( ·|s ) . V πent ( s ) includes an entropy term which is the reason it is called entropy regularization .

Summary: This paper suggests a DMAC learning framework by introducing a divergence penalty between the learning policy between a target policy into the RL framework instead of the commonly used entropy penalty. It can be shown that the proposed objective can be easily combined with various MARL algorithms and can eventually lead to the optimal policy of the underlying unmodified MDP with properly set target policies. The paper rigorously studies the mathematical foundation of the algorithm following a standard workflow and presents a few experiments showing that DMAC could improve the learning efficiency of various MARL algorithms.