Divergence-Regularized Multi-Agent Actor-Critic

1 INTRODUCTION . Regularization is a common method for single-agent reinforcement learning ( RL ) . The optimal policy learned by traditional RL algorithm is always deterministic ( Sutton and Barto , 2018 ) . This property may result in the inflexibility of the policy facing with unknown environments ( Yang et al. , 2019 ) . Entropy regularization is proposed to settle this problem by learning a policy according to the maximum-entropy principle ( Haarnoja et al. , 2017 ) . Moreover , entropy regularization is beneficial to exploration and robustness for RL algorithms ( Haarnoja et al. , 2018 ) . However , entropy regularization is imperfect . Eysenbach and Levine ( 2019 ) pointed out maximum-entropy RL is a modification of the original RL objective because of the entropy regularizer . Maximum-entropy RL is actually learning an optimal policy for the entropy-regularized Markov Decision Process ( MDP ) rather than the original MDP , i.e. , the converged policy may be biased . Nachum et al . ( 2017 ) analysed a more general case for regularization in RL and proposed what we call divergence regularization . Divergence regularization can avoid the bias of the converged policy as well as be beneficial to exploration . Wang et al . ( 2019 ) employed divergence regularizer and proposed a single-agent RL algorithm , DAPO , which prevents the altering-objective drawback of entropy regularization . Regularization can also be applied to cooperative multi-agent reinforcement learning ( MARL ) ( Agarwal et al. , 2020 ; Zhang et al. , 2021 ) . However , most of cooperative MARL algorithms do not use regularizer ( Lowe et al. , 2017 ; Foerster et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ; Jiang et al. , 2020 ; Wang et al. , 2021a ) . Only few cooperative MARL algorithms such as FOP ( Zhang et al. , 2021 ) use entropy regularization , which may suffer from the drawback aforementioned . Divergence regularization , on the other hand , could potentially benefit cooperative MARL . In addition to its advantages mentioned above , divergence regularization can also help to control the step size of policy update which is similar to conservative policy iteration ( Kakade and Langford , 2002 ) in single-agent RL . Conservative policy iteration and its successive methods such as TRPO ( Schulman et al. , 2015 ) and PPO ( Schulman et al. , 2017 ) can stabilize policy improvement ( Touati et al. , 2020 ) . These methods use a surrogate objective for policy update , but decentralized policies in centralized training with decentralized execution ( CTDE ) paradigm may not preserve the properties of the surrogate objective . Moreover , DAPO ( Wang et al. , 2019 ) can not be trivially extended to cooperative MARL settings . Even with some tricks like V-trace ( Espeholt et al. , 2018 ) for off-policy correction , DAPO is essentially an on-policy algorithm and thus may not be sample-efficient in cooperative MARL settings . In the paper , we propose and analyze divergence policy iteration in general cooperative MARL settings and a special case combined with value decomposition . Based on divergence policy iteration , we derive the off-policy update rule for the critic , policy , and target policy and propose divergenceregularized multi-agent actor-critic ( DMAC ) , a novel off-policy cooperative MARL framework . We theoretically show that DMAC guarantees a monotonic policy improvement and is not biased by the regularization . Besides , DMAC is beneficial to exploration and stable policy improvement by applying our update rule of target policy . Moreover , DMAC is a flexible framework and can be combined with many existing cooperative MARL algorithms to substantially improve their performance . We empirically investigate DMAC in a didactic stochastic game and StarCraft Multi-Agent Challenge ( Samvelyan et al. , 2019 ) . We combine DMAC with five representative MARL methods , i.e. , COMA ( Foerster et al. , 2018 ) for on-policy multi-agent policy gradient , MAAC ( Iqbal and Sha , 2019 ) for off-policy multi-agent actor-critic , QMIX ( Rashid et al. , 2018 ) for value decomposition , DOP ( Wang et al. , 2021b ) for the combination of value decomposition and policy gradient , and FOP ( Zhang et al. , 2021 ) for the combination of value decomposition and entropy regularization . Experimental results show that DMAC indeed induces better performance , faster convergence , and better stability in most tasks , which verifies the benefits of DMAC and demonstrates the advantages of divergence regularization over entropy regularization in cooperative MARL . 2 RELATED WORK . MARL . MARL has been a hot topic in the field of RL . In this paper , we focus on cooperative MARL . Cooperative MARL is usually modeled as Dec-POMDP ( Oliehoek et al. , 2016 ) , where all agents share a reward and aim to maximize the long-term return . Centralized training with decentralized execution ( CTDE ) ( Lowe et al. , 2017 ) paradigm is widely used in cooperative MARL . CTDE usually utilizes a centralized value function to address the non-stationarity for multi-agent settings and decentralized policies for scalability . Many MARL algorithms adopt CTDE paradigm such as COMA , MAAC , QMIX , DOP and FOP . COMA ( Foerster et al. , 2018 ) employs the counterfactual baseline which can reduce the variance as well as settle the credit assignment problem . MAAC ( Iqbal and Sha , 2019 ) uses self-attention mechanism to integrate local observation and action of each agent and provides the structured information for the centralized critic . Value decomposition ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ; Yang et al. , 2020 ; Wang et al. , 2021a ; b ; Zhang et al. , 2021 ) is a popular class of cooperative MARL algorithms . These methods express the global Q-function as a function of individual Q-functions to satisfy Individual-Global-Max ( IGM ) , which means the optimal actions of individual Q-functions are corresponding to the optimal joint action of global Q-function . QMIX ( Rashid et al. , 2018 ) is a representative of value decomposition methods . It uses a hypernet to ensure the monotonicity of the global Q-function in terms of individual Q-functions , which is a sufficient condition of IGM . DOP ( Wang et al. , 2021b ) is a method that combines value decomposition with policy gradient . DOP uses a linear value decomposition which is another sufficient condition of IGM and the linear value decomposition helps the compute of policy gradient . FOP ( Zhang et al. , 2021 ) is a method that combines value decomposition with entropy regularization and uses a more general condition , Individual-Global-Optimal , to replace IGM . In this paper , we will combine DMAC with these algorithms and show its improvement . Regularization . Entropy regularization was first proposed in single-agent RL . Nachum et al . ( 2017 ) analyzed the entropy-regularized MDP and revealed the properties about the optimal policy and the corresponding Q-function and V-function . They also showed the equivalence of value-based methods and policy-based methods in entropy-regularized MDP . Haarnoja et al . ( 2018 ) pointed out maximum-entropy RL can achieve better exploration and stability facing with the model and estimation error . Although entropy regularization has many advantages , Eysenbach and Levine ( 2019 ) showed entropy regularization modifies the MDP and results in the bias of the convergent policy . Yang et al . ( 2019 ) revealed the drawbacks of the convergent policy of general RL and maximum-entropy RL . The former is usually a deterministic policy ( Sutton and Barto , 2018 ) which is not flexible enough for unknown situations , while the latter is a policy with non-zero probability for all actions which may be dangerous in some scenarios . Neu et al . ( 2017 ) analyzed the entropy regularization method from several views . They revealed a more general form of regularization which is actually divergence regularization and showed entropy regularization is just a special case of divergence regularization . Wang et al . ( 2019 ) absorbed previous result and proposed an on-policy algorithm , i.e. , DAPO . However , DAPO can not be trivially applied to MARL . Moreover , its on-policy learning is not sample-efficient for MARL settings and its off-policy correction trick V-trace ( Espeholt et al. , 2018 ) is also intractable in MARL . There are some previous studies in single-agent RL which use similar target policy to ours , but their purposes are quite different . Trust-PCL ( Nachum et al. , 2018 ) introduces a target policy as a trust region constraint for maximum-entropy RL , but the policy is still biased by entropy regularizer . MIRL ( Grau-Moya et al. , 2019 ) uses a distribution which is only related to actions as the target policy to compute a mutual-information regularizer , but it still changes the objective of the original RL . 3 PRELIMINARIES . Dec-POMDP is a general model for cooperative MARL . A Dec-POMDP is a tuple M = { S , A , P , Y , O , I , n , r , γ } . S is the state space , n is the number of agents , γ is the discount factor , and I = { 1 , 2 · · ·n } is the set of all agents . A = A1 ×A2 × · · · ×An represents the joint action space where Ai is the individual action space for agent i. P ( s′|s , a ) : S × A × S → [ 0 , 1 ] is the transition function , and r ( s , a ) : S×A→ R is the reward function of state s and joint action a. Y is the observation space , and O ( s , i ) : S× I → Y is a mapping from state to observation for each agent . The objective of Dec-POMDP is to maximize J ( π ) = Eπ [ ∑ t=0 γ tr ( st , at ) ] , and thus we need to find the optimal joint policy π∗ = arg maxπ J ( π ) . To settle the partial observable problem , history τi ∈ Ti = ( Y × Ai ) ∗ is often used to replace observation oi ∈ Y . As for policies in CTDE , each agent i has an individual policy πi ( ai|τi ) and the joint policy π is the product of each πi . Though we calculate individual policy as πi ( ai|τi ) in practice , we will use πi ( ai|s ) in analysis and proofs for simplicity . Entropy regularization adds the logarithm of current policy to the reward function . It modifies the optimization objective as Jent ( π ) = Eπ [ ∑ t=0 γ t ( r ( st , at ) − λ logπ ( at|st ) ) ] . We also have the corresponding Q-function Qπent ( s , a ) = r ( s , a ) + γE [ V πent ( s′ ) ] and V-function V πent ( s ) = E [ Qπent ( s , a ) − λ logπ ( a|s ) ] . Given these definitions , we can deduce an interesting property V πent ( s ) = E [ Qπent ( s , a ) ] + λH ( π ( ·|s ) ) , whereH ( π ( ·|s ) ) represents the entropy of policy π ( ·|s ) . V πent ( s ) includes an entropy term which is the reason it is called entropy regularization .

Summary: This paper develops a new multiagent framework with regulated divergence to address the learning of biased policies in the maximum-entropy approach. Specifically, the main contributions include the developments of the divergence policy iteration for general cooperative MARL settings and the off-policy divergence regularized multiagent actor-critic framework (DMAC). Empirical results show that DMAC can improve the performance of existing MARL frameworks in various multiagent domains.