Divergence-Regularized Multi-Agent Actor-Critic
Authors: Su Kefan
Zongqing Lu
Venue: ICLR 2022 Submitted (2022)
Keywords: multi-agent reinforcement learning

Abstract: Entropy regularization is a popular method in reinforcement learning (RL). Although it has many advantages, it alters the RL objective and makes the converged policy deviate from the optimal policy of the original Markov Decision Process. Though divergence regularization has been proposed to settle this problem, it cannot be trivially applied to cooperative multi-agent reinforcement learning (MARL). In this paper, we investigate divergence regularization in cooperative MARL and propose a novel off-policy  cooperative MARL framework, divergence-regularized multi-agent actor-critic (DMAC). Mathematically, we derive the update rule of DMAC which is naturally off-policy, guarantees a monotonic policy improvement and is not biased by the regularization. DMAC is a flexible framework and can be combined with many existing MARL algorithms. We evaluate DMAC in a didactic stochastic game and StarCraft Multi-Agent Challenge and empirically show that DMAC substantially improves the performance of existing MARL algorithms. 

Under review as a conference paper at ICLR 2022
DIVERGENCE-REGULLRIZZI AGENT ACTOR- CRITIC
Anonymous authors Paper under double-blind review
ABSTRACT
Entropy regularization is popular method in reinforcement learning (RL). Al- though it has many advantages, it alters the RL objective and makes the converged policy deviate from the optimal policy of the original Markov Decision Process. Though divergence regularization has been proposed to settle this problem, it can- not be trivially applied to cooperative multi-agent reinforcement learning (MARL). In this paper, we investigate divergence regularization in cooperative MARL and propose a novel off- policy cooperative MARL framework, divergence-regullrizee multi-agent actor-critic (DMAC). Mathematically, derive the update rule of DMAC which is naturally off-policy, guarantees a monotonic policy improvement and is not biased by the regularization. DMAC is flexible framework and can be combined with many existing MARL algorithms. We evaluate DMAC in a didactic stochastic game and StarCraft Multi-Agent Challenge and empirically show that DMAC substantially improves the performance of existing MARL algorithms.
INTRODUCTION
Regularization is a common method for single-agent reinforcement learning (RL). The optimal policy learned by traditional RL algorithm is always deterministic (Sutton and Barto, 2018). This property may result in the inflexibility of the policy facing with unknown environments (Yang et al., 2019). Entropy regularization is proposed settle this problem by learning policy according the maximum-entroopy principle (Haarnoja et al., 2017). Moreover, entropy regularization is beneficial to exploration and robustness for RL algorithms (Haamojj al., 2018). However, entropy regularization is imperfect. Eysenbach and Levine (2019) pointed out maximum-entre original RL objective because of the entropy regularizer. Maximum-entropp RL. is actually learning an optimal policy for the entropy- regulla-eee Markov Decision Process (MDP) rather than the original MDP, i.e., the converged policy may be biased. Nachum et al. (2017) analysed a more general case for regularization in RL and proposed what call divergence regularization. Divergence regularization can avoid the bias of the converged policy as well as be beneficial to exploration. Wang et al. (2019) employed divergence regularizer and proposed aggett RL algorithm, DAPO, which prevents the altering-objective brawback of entropy regularization.
RL is a modification of the
Regularization can also be applied to cooperative multi-agent reinforcement learning (MARL) (Agarwal et al., 2020; Zhang et al., 2021). However, most of cooperative MARL algorithms do not regularizer (Lowe et al., 2017: Foerster et al., 2018; Rashid et al., 2018; Son al., 2019; Jiang et al., 2020; Wang et al., a021 Only few cooperative MARL algorithms such as FOP (Zhang et al., 2021) use entropy regularization, which may suffer from the drawback aforementioned. Divergence regularization, on the other hand, could potentially benefit cooperative MARL. In addition to its advantages mentioned above, divergence regularization can also help control the step size of policy update which is similar to conservative policy iteration (Kakade and Langford, 2002) in single-agent RL. Conservative policy iteration and its successive methods such as TRPO (Schulman al., 2015) and PPO (Schulman et al., 2017) can stabilize policy improvement (Touati et al., 2020). These methods use a surrogate objective for policy update, but decentralized policies in centralized training with decentralized execution (CTDE) paradigm may not preserve the properties of the surrogate objective. Moreover, DAPO (Wang et al., 2019) cannot be trivially extended to cooperative MARL settings. Even with some tricks like V-trace (Espeholt et al., 2018) for off off-policy correction, DAPO essentially an on on-policy algorithm and thus may not be sample-efficient in cooperative MARL settings.

---

Under review as a conference paper at ICLR 2022
In the paper, propose and analyze divergence policy iteration in general cooperative MARL settings and a special case combined with value decomposition. Based on divergence policy iteration, we derive the off off-policy update rule for the critic, policy, and target policy and propose divergence- regularized multi-agent actor-critic (DMAC), a novel off-policy cooperative MARL. framework. We theoretically show that DMAC guarantees a monotonic policy improvement and is not biased by the regularization. Besides, DMAC is beneficial to exploration and stable policy improvement by applying our update rule of target policy. Moreover, DMAC is a flexible framework and can be combined with many existing cooperative MARL algorithms to substantially improve their performance.
We empirically investigate DMAC in didactic stochastic game and StarCraft Multi-Agent Challenge (Samvelyan et al., 2019). We combine DMAC with five representative MARL methods, i.e., COMA (Foerster et al., 2018) for on-policy multi-agent policy gradient, MAAC (Igbal and Sha, 2019) for off-policy multi-agent actor-critic, QMIX (Rashid et al., 2018) for value decomposition, DOP (Wang et al., 2021b) for the combination of value decomposition and policy gradient, and FOP (Zhang et al., 2021) for the combination of value decomposition and entropy regularization. Experimental results show that DMAC indeed induces better performance, faster convergence, and better stability in most tasks, which verifies the benefits of DMAC and demonstrates the advantages of divergence regularization over entropy regularization in cooperative MARL.
RELATED WORK
MARL. MARL has been a hot topic in the field of RL. In this paper, we focus on cooperative MARL. Cooperative MARL is usually modeled as Dec-POMDP (Oliehoek et al., 2016), where all agents share a reward and aim to maximize the long-term return. Centralized training with decentralized execution (CTDE) (Lowe et al., 2017) paradigm widely used cooperative MARL.. CTDE usually utilizes a centralized value function to address the non-stationarity for multi-agent settings and decentralized policies for scalability. Many MARL algorithms adopt CTDE paradigm such as COMA, MAAC, QMIX. DOP and FOP. COMA (Foerster et al., 2018) employs the counterfactual baseline which can reduce the variance as well as settle the credit assignment problem. MAAC (Iqbal and Sha, 2019) uses self-attention mechanism to integrate local observation and action of each agent and provides the structured information for the centralized critic. Value decomposition (Sunehag et al., 2018; Rashid al., 2018: 8: Son et al., 2019; Yang et al., 2020; Wang et al., 2021a;b; Zhang et al., 2021) is a popular class of cooperative MARL algorithms. These methods express the global Q-function as a function of individual O-functions to satisfy Individual-Globar Global--aa (IGM), which means the optimal actions of individual Q-functions are corresponding to the optimal joint action of global Q-function. QMIX (Rashid et al., 2018) is a representative of value decomposition methods. It uses hypernet to ensure the monotonicity of the global Q-function in terms of individual Q Q-functions, which is a sufficient condition of IGM. DOP (Wang et al., 2021b) is method that combines value decomposition with policy gradient. DOP uses linear value decomposition which is another sufficient condition of IGM and the linear value decomposition helps the compute of policy gradient. FOP (Zhang et al., 2021) is method that combines value decomposition with entropy regularization and uses more general condition, Individual SorddulllGloal----e Global- -pttmmll,,, to replace IGM. In this paper, we will combine DMAC with these algorithms and show its improvement.
Regularization. Entropy regularization was first proposed in single-agent RL. Nachum et al. (2017) analyzed the rnguopy-regularrzed MDP and revealed the properties about the optimal policy and the corresponding Q-function and V-function. They also showed the equivalence of basue methods and policy-based methods in entropy-regularized MDP. Haarnoja et al. (2018) pointed out maximum-entropy RL can achieve better exploration and stability facing with the model and estimation error. Although entropy regularization has many advantages, Eysenbach and Levine (2019) showed entropy regularization modifies the MDP and results in the bias of the convergent policy. Yang et al. (2019) revealed the drawbacks of the convergent policy of general RL. and maximum-entropp RL. The former is usually deterministic policy (Sutton and Barto, 2018) which is not flexible enough for unknown situations, while the latter is a policy with non-zero probability for all actions which may be dangerous in some scenarios. Neu et al. (2017) analyzed the entropy regularization method from several views. They revealed a more general form of regularization which is actually divergence regularization and showed entropy regularization is just special case of divergence regularization. Wang et al. (2019) absorbed previous result and proposed an on polooiic algorithm, i.e. DAPO. However, DAPO cannot be trivially applied to MARL.. Moreover, its on-policy learning is
2

---

Under review as a conference paper at ICLR 2022
not sample-efficient for MARL settings and its off off-po correction trick V-trace (Espeholt et al., 2018) is also intractable in MARL. There are some previous studies in single-agent RL. which use similar target policy ours, but their purposes introduces a target policy as a trust region constraint for maximum-entropy RI., but the policy is still biased by entropy regularizer. MIRL (Grau-Moya et al., 2019) uses distribution which is only related to actions as the targel policy to compute a mutual-informattie regularizer, but it still changes the objective of the original RL.
quite different. Trust-PCL (Nachum et al., 2018)
PRELIMINARIES
Dec-POMDP is general model for cooperative MARL. A Dec-POMDP is tuple M {S, A, P.Y. O. n. n. 7)- S is the state space, n is the number of agents, y is the discount fac- tor, and I {1, 2...2... . is the set of all agents. A = Ai x A2 space where A; is the individual action space for agent i. P(s 18, a) S x A x S - [0, 1] is the transition function, and r(s, a) : S x A + R is the reward function of state and joint action the observation space, and (s,, i) 5xxI- Y is a mapping from state to observation for each agent. The objective of Dec-POMDP
x A, represents the joint action
Y is
maximize J(T) Ex ? ar)], and thus we need to find the optimal joint policy π" = arg max, J(r). To settle the partial observable problem, history T E T; (Y A:) often used to replace observation or E Y. As for policies in CTDE, each agent has an individual policy n, (a:l;)) and the joint policy is the product of each т. Though we calculate individual policy as T (a; (T)) in practice, we will use my (a:(s) in analysis and proofs for simplicity.
Entropy regularization adds the logarithm of current policy to the reward function. It mod- ifies the optimization objective as Jent(1) = Ex 15 7t (r(s), a.) Alog π (agss)))] We also have the corresponding Q-function (s, = r(. a) + ?E [Vent (s') and V-function deduce interesting property V (s) (s)= E EQ (s, a)] + XH (TT-)s)), where H (T(-(s) represents the entropy of policy (-1s). Vam (s) includes an entropy term which is the reason it called entropy regularization.
ent (s) E [ (s. a) Algg π (a]s)]. Given these definitions,
4 METHOD
In this section, we first give the definition of divergence regularization, Then we propose and analyze divergence policy iteration. Finally, based on divergence policy iteration, we derive the update rules of the critic, policy, and target policy for divergence- MARL.
4.1 DIVERGENCE REGULARIZATION
We maintain a target policy Pt for each agent i, which is different from the policy n.. Then we have a joint target policy p = III Pi. This joint target policy p modifies the objective function =(a.(s:) That is, a regularizer log miar]ss). which p(a.)s.) describes the discrepancy between policy π and target policy p. is added to the reward function just like entropy regularization.
as (T) Ex Ei-o 7d (r(st, az) A log p(a+)+.)
Given we can define corresponding V-function and Q-function for divergence regularization as follows.
7 (8) p(ar (s)) )800 =
VT((s) (5) = Ex Σ 7 (r(s), Allog
07 (8, a) = r(s. a) + B. (s'))
(2)
Further, by simple deduction, we have
π (a|s) p(a)s)
V Eaw (s. Alog
= Ea-m(1s) [07 (s, a)] >DRL (T(()s))(P(((ss)
VI (s) includes an extra term which is the KL divergence between π and p. and thus this regularizer referred to as divergence regularization.
3

---

Under review as a conference paper at ICLR 2022
4.2 DIVERGENCE POLICY ITERATION
From the perspective of policy evaluation, we can define an operator I as
(7 = r(s, P.ggp ( a.) [Q(s' a') x log ( p(a' (s)
(3)
and have the following lemma. Note that all the proofs are given Appendix
Lemma (Divergence Policy Evaluation) For any initial Q-finction Q (s, a) : S - R, we define a sequence { given operator 1 as Q++1 =r7 Then, the sequence will converge to O7 as k xc.
After the evaluation of the policy, we need method to improve the policy. We have the following lemma about policy improvement.
Lemma 2 (Divergence Policy Improvement) If we define new satisfying
Tnew (-18) = arg min DRL. (T (-)))))((11)))
(4)
where u(.ls) p(-ls) exp (oppa (o,))/A and a (s) is normalization term, then for all actions ZMold
and all states s we have TTe (s, a) 2 O a).
Lemma and 2 could be seen as corollaries of the conclusion of Haarnoja et al. (2018). Lemma indicates that given a policy Told. if find policy Nnww according (4), then the policy nnew better than Told.
Lemma does not make any assumption and for general settings. Further, the policy improvement can be established and simplified based on value decomposition. In the following, we give an example for linear value decomposition like DOP (i.e., Q(x,a) Σ, ks()Q a:) b(s)) (Wang al., 202
Lemma 3 (Divergence Policy Improvement with Linear Value Decomposition) If Q-functions satisfy (s) E. ks)) (s, a:) + b(s) and we define Thew satisfying
now (-1s) = arg min DKL (m((-15)ll<<((1s)) ViE as
exp (
where ui (-(ss) pr(-(s)
and Zra (s) is a normalization term, then for all actions a and all states 8 we have Quae " (8, a) 2 Qu (s. a).
20 (s)
Lemma 3 further tells that if the MARL setting satisfies the linear value decomposition condition, then each agent can optimize its individual policy with an objective of its own individual Q-function, which immediately improves the joint policy. By combining divergence policy evaluation and divergence policy improvement, we have the following theorem of divergence policy iteration.
Theorem (Divergence Policy Iteration) By iteratively using Divergence Policy Evaluation and Divergence Policy Improvement, we will get sequence (Q*) and this sequence will converge the optimal Q-function Q' and the corresponding policy sequence will converge to the optimal policy n
Theorem shows that with repeated application of divergence policy improvement and divergence policy evaluation, the policy be monotonically improved and converge the optimal policy. Let TT (s). and (s, a) denote the optimal policy, Q-function, and V-function respectively, given a target policy p. We have the following proposition.
Proposition If T = arg max JT( and Vi) = (' and (2 a) = QT a) are respectively the corresponding Q-function and V-function of Tr then they satisfy the following
4

---

Under review as a conference paper at ICLR 2022
properties:
(alss)) p(als) exp ((r(s,a) +.By-p(ha) (V((s /2)
(5)
()) Alog Σ p(als) exp (()(s,a) + 71B [V)(s))) /2)
(6)
()(s,a) r(s,a) PP(a.))) log Elaals) (Q)(s', (''''a'))//) a
With all these results above, we have enough tools to obtain the practical update rule of the critic, policy and target policy of DMAC.
4.3 SHERGENCE-REGULAAEE CRITIC
From Proposition 1, we obtain
TH (als)) p(a exp ((r(s, a) + (B (s)) /A) p(als) exp (s.(s))/A1)
p(bls) exp ((r(s,b) (P. (s) /A) exp (V)())/A)
-p(als) exp (Qppss, a) - (s)) /2).
(8)
By rearranging the equation, we have
(a(ss)
vi(() 07 a) A Alog
(9)
p(als)
which is tenable for all actions A. Therefore, we have the following corollary.
Corollary
Ta(a'ls')
(s(s,a) = r(s, a) + y1E. P P(laa) (2((', -Alog p(a'ls')
is tenable for all actions a' E A.
Corollary gives an iterative formula for (p a). with which we can design a loss function and update rule for learning the critic,
Lo = E (Qo(s, w)²] where r(s, a) + (og( (8) Alog
π (a' (a'ls')
(10)
p (aalls)
where and are respectively the weights of Q-function and target Q-function. The update of Q-function is similar to that in general MDP, except that the action for next state could be chosen arbitrarily while must be the action that maximizes Q-function for next state in general MDP. This property greatly enhances the flexibility of learning Q-function, e.g., can easily extend it to TD()).
4.4 DIVERGENCE- REGULARIZED
RNVERGENCERREGULARIZEE ACTORS
DAPO (Wang et al., 2019) analyzes the divergence-regullrizee MDP from the perspective of policy gradient theorem (Sutton et al., 2000) and gives an on-policy update rule for single-agent RL. Unlike cxisting work, we focus on a different perspective and derive an off-policy update rule by taking into consideration the characteristics of MARL..
From Lemma 2, we can obtain an optimization target for policy improvement,
arg min DKL. (<-1ss)|p(-|s)) exp (Qp (s, )/A)
(a)ss) arg Σ T(als) (s) Alog p(als)
2 (s)
Then, can define the objective of the actors,
T((ss) (z) - log T(ass) s)
Cx EEND
(11)
p(als)
where D is the replay buffer. Suppose each individual policy n; has a corresponding parameterization 0,. We can obtain the following policy gradient for each agent with some derivation and the detail is given in Appendix A.6,
V Lr = E.nD.am Ve. log (a(lss)) (x) - Allg T
x)
2)]
(12)
p(a.s)
5

---

Under review as a conference paper at ICLR 2022
We need to point out that the key to off-policy update is that Lemma does not limit the state distribution. only requires the condition is satisfied for each state. Therefore, we can maintain a replay buffer to cover different states as much possible, which is a common practice in off- off-policy learning. DAPO uses a similar formula to ours, but it obtains the formula from policy gradient theorem, which requires the state distribution of the current policy.
Further, we can add counterfactual baseline to the gradient. First, we have the following equation about the counterfactual baseline (Foerster al., 2018),
BawD,awn Vo, log n. (1: (s)(),))(sss)())))))) a_.)] 00
(13)
where a- denotes the joint action of all agents except agent i. Next, we take the baseline as
(aas)
b(s, as)a Ba, 11) (s. A Alog p(a|s) A)].
(14)
Then, the gradient for each agent can be modified follows.
Alog T(als)
Ve, -r = VV, log (agg
p(als) b(s, a-:))]
(((a;)s)) Bajon (7 a)] + (
Vo, log n. (a. (() Alog
(a(a:(s)
In addition to variance reduction and credit assignment, this counterfactual baseline eliminates the policies of other agents from the gradient. This property makes it convenient to calculate the gradient and easy to select the target policy for each agent. Moreover, if the linear value decomposition condition satisfied, have the following gradient formula,
Ca = V. log क. (alsss(()) (s, a;) -loo ((als) + (DKL(T(()))lo((s))) (15)
(:(a:s))
where Ag (s, ax) 27 (s, a:) Eg (8, ā.)
4.5 TARGET POLICY
We have discussed the update rule of the critic and actors, and now we focus on the selection and update rule of the target policy. With the update rules above, we can obtain divergence policy iteration given fixed target policy p. Then we need to devise the update rule of p prevent the bias of regularization and benefit the learning procedure.
p(ayles) could help to balance exploration and exploitation. For example, for some action a, if p(als) > T(als), then the regularizer is equivalent adding positive value to the reward and vice versa. Therefore, if we choose previous policy as target policy, the regularizer will encourage agents to take actions whose probability has decreased and discourage agents to take actions whose probability has increased. Additionally, the regularizer actually controls the discrepancy between current policy and previous policy, which could stabilize the policy improvement (Kakade and Langford, 2002; Schulman et al., 2015; 2017).
Intuitively, this regularizer log x(a.(s.)
To derive the update rule of target policy, we need to further analyze the regularizer theoretically. Let Mi denote the state-action distribution given a policy π. That is, (x a) d (s (a|s), where
d= (s) Ei= 7º Pr(s; s|Tr) is the stationary distribution of states given π. With Prs we can rewrite the optimization objective JP(T)) as follows,
(s) log T(aas)
JØ(TT) Σ (s,
p(als)
3,a
3.a
(16)
Σ (x 8.0
ADc (uzllpp))
where Do (M = E..o Mr (s a) log plals) fore, the objective of the divergence regularized MDP can be represented as (17)
#ials)
Bregman divergence (Neu al., 2017). There-
arg Max
u as) a) ADc (umllHp)
(17)
8.0

---

Under review as a conference paper at ICLR 2022
With this property, similar to Neu et al. (2017) and Wang al. (2019), we use the following iterative process,
arg max
Σ ( a) ADc (H
(18)
8,0
This iteration is a mirror descent process (Neu et al., 2017), so the convergence of the policy is guaranteed. This process also guarantees that when the policy converges, De (f7 [lpx=)) ) + 0; i.e. the regularizer will vanish. Moreover, we can obtain the following inequalities:
J( 2 J
ADc (1oe44 [| 2 J(T') - XDc (Par [|/xx)))))))))) J(T',,
The first inequality is from De (M> 11pp) ) 2 and the second inequality is from the definition of a This conclusion means the policy sequence obtained by this iteration improves monotonically in the original MDP. With these deductions, we actually obtain the following theorem.
Theorem By iteratively applying the divergence policy iteration and taking p nh ,he policy sequence {π²} will converge and improve monotonically in the original MDP.
However, it is intractable to perform this update rule in practice because every iteration in (18) needs convergent policy. Thus, propose alternative approximate method. For each agent, update the policy n, and the target policy 0 as 0g 0; + VV and 0, (1 7)0 70,, where is the learning rate, Ó, is the weights of pt. and 7 is the hyperparameter for soft update. Here we use one gradient step to replace the max operator in (18). From Theorem and previous discussion, we know that optimizing Cnr can maximize JT(TT) so we use Vo, Crr in the gradient step for off- off-policy training instead of the gradient step directly optimizing JP(T) in (16). Moreover, as the convergence of (17) guaranteed only if the target policy p is fixed, we softly update the target policy as the moving average of the policy to prevent the instability caused by the large change of the target policy and hence obtain stable policy improvement.
Now we have all the update rules of DMAC. The training of DMAC is a typical off-policy learning process, which given Appendix B for completeness.
EXPERIMENTS
In this section, we first empirically study the benefits of DMAC and investigate how DMAC improves the performance of existing MARL algorithms in didactic stochastic game and five SMAC tasks. Then, we demonstrate the advantages of divergence regularizer over entropy regularizer in cooperative MARL.
5.1 IMPROVEMENTS OF EXISTING METHODS
DMAC is a flexible framework and can be combined with many existing MARL algorithms. In the experiments, we choose four representative algorithms for different types of methods: COMA (Foerster et al., 2018) 1) for on-policy multi-agent policy gradients, MAAC (Igbal and Sha, 2019) for off-policy multi-agent actor-critic, QMIX (Rashid et al., 2018) for value decomposition, DOP (Wang et al., 2021b) for the combination of value decomposition and policy gradient. These algorithms need minor modifications to the framework of DMAC. We denote these modified algorithms as COMA+DMAC, MAAC+DMAC. DMIX+DMAC, and DOP+DMAC. Generally, our modification is limited and tries to keep the original architecture so as to fairly demonstrate the improvement of DMAC. The details of the modifications are included in Appendix C.I. More details about hyperparameters are available in Appendix C. All the curves in our plots correspond to the mean value of five training with different random seeds, and shaded regions indicate 95% confidence interval.
5.1.1 A DIDACTIC EXAMPLE
We first test the four groups of methods in a stochastic game where agents share the reward. The stochastic game is generated randomly for the reward function and transition probabilities with 30 states, 3 agents and actions for each agent. Each episode contains 30 steps in this game. The performance of these methods is illustrated in Figure We could find that DMAC performs
7

---

Under review as a conference paper at ICLR 2022
325
110
02MA-OMAC
-PXC-04CC
25
NAAC-MAA
26
D00-26AC
2500 1900
1900
19009
20093 4900 1900
(0009)
20090 0r60s 8900
3990e
50893
109000
156930
20093
136
Figure The learning curves terms of episode rewards of COMA, MAAC, QMIX and DOP groups in the randomly generated stochastic game.
better than the baseline at the end of the training in all the four groups. Moreover, COMA+DMAC, DMIX+ and MAAC+DMAC learn faster than their baselines. Though DOP learns faster than DOP+DMAC at the start, it falls into sub-optimal policy and DOP+DMAC finds better policy in the end.
We also show the benefit of exploration in this stochastic game for the convenience of statistics. We evaluate the exploration in terms of the cover rate of all state-action pairs, i.e., the ratio of the explored state-action pairs to all state-action pairs. The cover rates of COMA and COMA+DMAC are illustrated in Figure 2. We use COMA here as a representation of the traditional policy gradient method in cooperative MARL. We could find that the cover rate of COMA+DMAC is higher than COMA, which can be an evidence for the benefit of exploration of DMAC. The cover rates of other three groups of algorithms are available in Appendix D.
stoovart: uame
0.6
COVA-CMAC
100799
Figure 2: The learning curves in terms of cover rates of COMA and COMA+DMAC in the randomly generated stochastic game.
5.1.2 SMAC
We test all the methods in five tasks of StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019). The introduction of the SMAC environment and the training details are included in Appendix C. The learning curves in terms of win rate of all the methods in the five SMAC tasks are illustrated in Figure (four columns for four groups of algorithms and five rows for five different maps in SMAC). In addition, the learning curves in terms of rewards are available in Appendix D. We show the empirical result of DAPO in the map of 3m in the first figure of the second column in Figure 3. It can be seen that DAPO cannot obtain good performance in the simple task of SMAC, so we skip it in other SMAC tasks. The reason for the low performance of DAPO may be that DAPO omits the correction for di, (s)) /dz, (s) in policy update which introduces bias in the gradient of policy, and uses V-trace as off- policy correction which however is biased. These drawbacks may be magnified
0.38
346
1.00
233-
100
1.70
0.5
Figure 3: Learning curves in terms of win rates of COMA, MAAC, QMIX and DOP groups five SMAC maps (each row corresponds to map and each column corresponds to group).
8

---

Under review as a conference paper at ICLR 2022
Mw
0.3
3.0
0.1-
0.2
3.2
FOP-DMAC
TOP- DMAC
TOP
0.5
3.0
2.0
0.0
1.0
2.0
0.0
1.0
3.5
<tep
dep
step
Figure 4: The learning curves of win rates of FOP+DMAC and FOP in three SMAC maps.
in MARL settings. The superiority of our naturally off-policy method over the biased off-policy correction method can be partly seen from the large performance gap between COMA+DMAC and DAPO.
In all the five tasks, MAAC+DMAC outperforms MAAC significantly, but MAAC+DMAC does not change the network architecture of MAAC, which shows the benefits of divergence regularizer. As for the result of COMA and COMA+DMAC. We find that COMA+DMAC has higher win rates than COMA in most cases at the end of the training, which can be attributed to the benefits of off-policy training and exploration of divergence regularizer. Though in some cases COMA learns faster than COMA+DMAC, it falls into sub-optimal in the end. This phenomenon can be observed more clearly in the plots of episode rewards in Appendix D, especially in the hard tasks like 3s5z. This can be an evidence for the advantage of divergence regularizer which helps the agents find a better policy.
The stable policy improvement of divergence regularizer can be showed in the variance of the learning curves especially in the comparison between QMIX and QMIX+DMAC. In most tasks, we find that QMIX+DMAC learns substantially faster than QMIX and gets higher win rates in harder tasks. The results of DOP groups are illustrated in the fourth column of Figure 3. DOP+DMAC learns faster than DOP in most cases and finally obtains better performance. The difference of DOP and DOP+DMAC can also partly show the advantage of naturally off off-ccllcy method to the off off-policy correction method, as DOP+DMAC replaces the tree backup loss with off- policy TD(A).
DMAC improves the performance and/or convergence speed of the evaluated algorithms in most tasks. This empirically demonstrates the benefits of divergence regularizer. Moreover, the superiority of our naturally off off-policy learning over the biased off- aff-policy correction method can be partly witnessed from the empirical results.
5.2 COMPARISON WITH ENTROPY REGULARIZATION
FOP (Zhang et al., 2021) combines the value decomposition with entropy regularization, which obtained the ffte--ofttheall performance in SMAC tasks. FOP has a well tuned scheme for the temperature parameter of the entropy, so we take FOP as a strong baseline for entropy-regullrizzed methods in cooperative MARL. We compare FOP and FOP+DMAC in three SMAC tasks, 3__vss____________ 3z, 2c_vs_ vs_ vs__4zg, and MMM2, which respectively correspond to the three levels of difficulties (i.e., easy, hard, and super hard) for SMAC tasks. These tasks are taken from the original paper of FOP. The modifications FOP+DMAC are also included in Appendix C. The win rates of FOP and FOP+DMAC are illustrated in Figure 4. We could find that FOP+DMAC learns much faster than FOP in 3s vs sy_3,, while it performs better than FOP in other two harder tasks. These results could be an evidence for the advantages of DMAC and the bias of entropy regularization.
CONCLUSION
We propose multi-agent actor-critic framework, DMAC, for cooperative MARL. We investigate divergence regularization, derive divergence policy iteration, and deduce the update rules for the critic, policy, and target policy multi-agent settings. DMAC is a naturally off-policy framework and the divergence regularizer is beneficial to exploration and stable policy improvement. Unlike entropy regularizer, the divergence regularizer will not bias the converged policy. DMAC is also a flexible framework and can be combined with many existing MARL algorithms with limited modification. It is empirically demonstrated that combining DMAC with existing MARL algorithms can improve the performance and convergence speed in a stochastic game and SMAC tasks.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Akshat Agarwal, Sumit Kumar, Katia Sycara, and Michael Lewis. Learning transferable cooperative behavior in multi-agent team. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2020.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures, In International Conference on Machine Learning (ICML), 2018.
Benjamin Eysenbach and Sergey Levine. If maxent is the answer, what is the question? arXiv preprint arXiv:1910.01913, 2019.
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
Jordi Grau-Moya, Felix Leibfried, and Peter Vrancx. Soft q-learning with mutual-information regularization. In International Conference on Learning Representations (ICLR), 2019.
Tuomas Haarnoja, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning (ICML), 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning (ICML), 2018.
Shariq Iqbal and Fei Sha. Actor-attention-rrin critic for multi-agent reinforcement learning. In Interna- tional Conference on Machine Learning (ICML), 2019.
Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforcement
learning. In International Conference on Learning Representations (ICLR), 2020.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning (ICML), 2002.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperatieecomppttivve environments. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Ofr Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Trust-pel: An off-policy trust region method for continuous control. In International Conference Learning Representations (ICLR), 2018
Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of retropy-regularizee markov decision processes. arXiv preprint 1rXiv:1705.07798, 2017.
Frans A Oliehock, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung. Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: 0902,0404, 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning (ICMI., 2015.
10

---

Under review as a conference paper at ICLR 2022
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2019.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius Flores Zam- baldi, Max Jaderberg. Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, al. Value- decomposition networks for cooperative multi-agent learning based on team reward. In Interna- tional Conference on Autonomous Agents and MultiAgent Systems (AAMAS). 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour, Policy gradient meth- ods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NeurIPS), 2000.
Ahmed Touati, Amy Zhang, Joelle Pineau, and Pascal Vincent. Stable policy optimization via off-policy divergence regularization. arXiv preprint :Xiv:2003.04108, 2020.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling
multi-agent q-learning. In International Conference on Learning Representations (ICLR), 2021a.
Qing Wang, Yingru Li, Jiechao Xiong, and Tong Zhang. Divergence--uugmentte policy optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Dop: Off-policy multi-
agent decomposed policy gradients. In International Conference on Learning Representations (ICLR), 2021b.
Wenhao Yang, Xiang Li, and Zhihua Zhang. A regularized approach sparse optimal policy in reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03939, 2020.
Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning. In International Conference Machine Learning, pages 12491-12500 PMLR, 2021.

---

Under review as a conference paper at ICLR 2022
A PROOFS
A.1
LEMMA
Proof.
We define reward function
pp [s, r(s, a) -B [DLL (T((|s')]l(( s)))
then we can rewrite the definition of operator r
r (s, s, a) r (s, + atsa [Q(s', a')]
With this formula, apply the traditional convergence result of policy evaluation in Sutton and Barto (2018).
A.2
LEMMA
For the proof of Lemma 2, we need the following lemma (Haarnoja al., 2018) about improving policy in antropy-regularyy MDP.
Lemma 4 If have new policy Nnew and
Nnew arg min DKL. [T(-s))] exp (Qreld (s, ))/1) Z
where Tot (s) represents the normalization term, then have
QTeen (s,a) 2 Tet (s, Vs E S,EE A.
With Lemma 4, we have the following proof of Lemma 2.
Proof. Let Q be the same the definition in Proof A.5. Then we have Q (s, llog plals), VT.
According to Lemma 4,
nnw (8,-) arg min DRL [-|ss)]] exp (ToaI (s, 1//1)
ZToi (s)
Qtww (8, a) mold (s, a), Va E A.
With the definition, we have
= DKL. (-(1s)][p(-11) exp(O% (s, 1/21) DRL. T--|s)]| exp ( Told 1/2)
Rold (8)
nooil (8)
π new new
(s, nhww (8, a) = Naw (s, a) - Allog p(als) 2 (s, - -Alog p(als) = (Troodds a).
A.3
LEMMA 3
Proof. From the equation
Σ (ailss (a A log (a((ss)) pi 7r(a;(s) 8)
new arg max We
we can obtain
Σ Thow (ails) (20 - -Alog
T The (atls) pi(a.(s)
(19)
2 Σ (a
(a AAlo (a,ls)
Tada (asls)
12

---

Under review as a conference paper at ICLR 2022
By taking expectation on the both side of (19), we can obtain the followings.
neww (a,18)
Σ # (a-1(__18))
Σ new (ails)
(al)) - A Alog
Pi(ails) (s)
Toia (cils) pr(a,(s)
(20)
Σ π (a=:a_1)))
Taaa (a:ls)
(s)
Alog
ViEI -カ=
Moreover, easily have the following derivation,
Σ nnw (arls)
(m(ss) (s. a.) + b(s) Alog
new (a:ls) pi(a:ls)
Σ un (a_i))))) Σ (ails) (2(s)) (s, (s. + b(s)
Alog T new (a, (8)
Di (a(a1()))
(21)
a,
= Σ u2 (a==)))))
new (ails) (=(()) (s. + b(s)) Σ
Allo new (ag (s)
Ps (a:18)
Vun. U22
Then we have
nnw (a|s)
(s, - -log
E n now
p(als)
Allog (new(a
(ails)
Σ new (als)
Σ
Q Vola (s, b
pi(as(s)
Σ nnww (als)
Σ
ki k Toi (s, (s) ( (s))))+b(()) - Alog
new (a=ls)
p;(a;(s)
Tnew (ails)
ΣΣ
T new (a=i(s) Σ Fnew(arls (ails) (5 as) + b(s) - -Alog
(22)
Di (ails))
new (ayls)
ΣΣ Told (a_18))
Σ new (asls)
(((s)) (o (8, + b(s)) > Alog
Ps
To (a:1s)
Todd (amis))
Σ Taiaaalk
(s) (s) (s. b(s) - Alog b(s)
p,(a,1s)
(s) (s)
The fourth equation is from (21) and the fifth inequality is from (20).
By repeatedly applying (22) and the relation Op (s, a) r(s. a) + NE, (VT we can complete the proof as followings.
Quaa (s, a) = r(s, a) + EE
Mold (s')l
Tnew (a'lss))
< <r(s, a) + +yB.
Ex' Calnraoo
QTTolal (s aa.aa')oo -log
p(alls))
π nnow (a'ls')
= =s(s, a) + 7/Bg'
Ex' r(s',a') a') + 71Err [Vraa - -log
p(a'
SQ²T T (s,
13

---

Under review as a conference paper at ICLR 2022
A.4 THEOREM
Proof.
First, we will show that Divergence Policy Iteration will monotonically improve the policy. From Lemma 2, we know that
T new (a|s)
now (8) = Ba-maer(-11
2 (s, a)
Alog
p(als)
new (a
Ba Bannon (nnw
ppal (s, a)
Alog
p(als)
p (8, a)) - Alog Told(als) s)
p(a ()
V (s).
The first inequality is from the conclusion of Lemma 2 that
Quee (8, a) 2 0 (s, Va E A, and the second inequality is from the definition of new that
[T|ss)] exp (Q (s. 1/2))
New arg min DKL
ZTll (8)
Here we have VTow (s) Volld (s), Vs E S, and thus Jp(Tnew) 2 Jo(TToa). So, Divergence Policy Iteration will monotonically improve the policy.
Since the OF is bounded above (the reward function is bounded), the sequence of Q-function { of Divergence Policy Iteration will converge and the corresponding policy sequence will also converge to some policy couvv We need to show Tcouv
p
Tonny (a(s)
V Teann (8) ES Ban
O Toomn (s, a)
Alog p(a 8)
(alss)
E Ea)t())
@ -llog
p(a|s)
QTcone (s, (5)m 1 Allo (aals'))
A log T(als) p(als)
r(s,a) + +7Ba'
p(a'
2 (s).
The first inequality is from the definition of Weonv that
(-|ss)] exp (QT.ome (s, 1/2)
Toony
arg min DKL.
and all the other inequalities are just iteratively using the first inequality and the relation of Qp and VE With iterations, we replace all the Teonv with in the expression of V / (s) and finally get (s). Therefore, we have
Reaar (s) (s) V VT II
p
Jp( (Tonv)))) 2 Jp(TT) VT II Tonn T.
A.5 PROPOSITION
Before the proof of Proposition 1, need some results about the optimal Q-function Q the optimal V-function V and the optimal policy nht in entropy-regularize MDP. We have the following lemma (Nachum et al., 2017).
14

---

Under review as a conference paper at ICLR 2022
Lemma
(s, a) exp ((r(s, a) + -((aa)) Ven (s'))) /2)
Vent (en((s) (s) = Alog Σ exp (()(s (BBxp((aao) |Ven (s')1) /2)
( = r(s,a) + yABs
log Σ exp (Qeme(s',aa)/))
With Lemma 5, we can complete the proof of Proposition 1.
Proof.
Let a) = r(s, a) + Alog p(als), we consider the objective function
J(TT) Ent Σ
(f($), - - (logg(a))))))
t=
Let #" (alss))v*(s) and Q'(s,, a) be the corresponding optimal policy, V-function and Q-function of J (T).. By definition we can obtain
#a(as) - (a(ss)
(s))s = E a~i (11))))
f(s, + >V²((s') Alog #े (als))
E EanT)(()))) r(s,a) +>)))) Alog [p(a|s) s) p(a(s)
= (s)
(s, a) =f(s,a) a) + -(ss)()
=s(s a) E. EPPe()
[%((')] + plags(als)
(s a) + x log p(als).
According to Lemma 5, we have
(als)) = # (als) x exp + [F-(3]) /2)
=(aass) exp
+ + (s)() /2)
(8) (s)
= Alog Σ exp (F(s,a)+ (()))) /2)
=Alog] Ep(als) exp ((r(s, a) + + (s'))) /A)
a) = (x, - A Alog p(als)
exp (Q ('(6) Allog Alogg(aass) p(als)
= f(s, + >>AE
log
a'
p(als) exp (0) (5((''))//A) a')/A))
= =r(s, a) + P((3.))
log Σ
a'
15

---

Under review as a conference paper at ICLR 2022
A.6 DERIVATION OF GRADIENT
Σ (ols)( (0) -Alog T(als)) + (Tals)Vo, (aass)
Vo, fr ESND
p(a|s)
p(a
T(als)
EgND Σ T(als)Vo, log (ayls)
()) p(als)
-AT(alsVVo, log (
Va. (a (3) Alog x(als) p(als)
EggD
B ALGORITHM
Algorithm gives the training procedure of DMAC.
Algorithm DMAC
1:
for episode = to m do
2: Initialize the environment and receive initial state
3: fo = to max-episoode-lenghh do
4: For each agent i, select action a; (T((-s))
5: Execute joint-action a = (a), 02, ,an) and observe reward r and next state
6: Store (s,a,r, s') in replay buffer D
7:
end for
8: Sample random minibatch of K samples from D, {(kk ag, Tke (()3
9: for agent = 1 to do
Update policy ny: 0; = 0, 0;
11:
Update target policy p;: 0; = (1 r)0, + =0,
12:
end for
13:
Update critic: Φ = Φ a
14: Update target critie: à = (1 - 7) T
end for
C IMPLEMENTATION DETAILS
SMAC is a MARL environment based on the game StarCraft II (SC2). Agents control different units in SC2 and can attack, move take other actions. The general mode of SMAC tasks is that agents control a team of units to counter another team controlled by built-in AL. The target agents is to wipe out all the enemy units and agents will gain rewards when hurting killing enemy units. Agents have an observation range and can only observe information of units in this range, but the information of all the units can be accessed in training. We test all the methods in totally 8 tasks/maps: 3m, 2s3z, 3s5z, lc3s522.33ss vs vs___________ 32,2cc 32,2c__v__4222 vs_64zg, and MMM2.
In SMAC tasks, we evaluate 20 episodes in every 10000 training steps in the one million steps training procedure for COMA, MAAC, QMIX and in every 20000 time steps for DOP and FOP. In evaluation, select greedy actions for QMIX and FOP (following the setting in the FOP paper) and sample actions according to action distribution for stochastic policy (COMA, MAAC, DOP and divergence-regularizee methods).
C.1 MODIFICATIONS OF THE BASELINE METHODS
The modifications of the baseline methods, COMA,MAAC.QQIX and FOP, are as follows.
COMA. We keep the original critic and actor networks and add target policy network with the same architecture as the actor. As COMA on-policy but COMA+DMAC is off-policy, we add replay buffer for experience replay.
MAAC already has target policy for stability, so we do not need to modify the network architecture. We only change the update rule for the critic and actors.
16

---

Under review as a conference paper at ICLR 2022
66--
0.-
04-
04
COMMAOOAC
114 SHOMAC
02
PARL-OMM
04-
DOP
00
03
0.3
00
20083
48000
60000
2008
230003
2000
48000 50003
2008
130033
20003
4000
50083
68000
100403
50093
160000
150000
200433
Figure 5: Learning curves in terms of cover rates COMA, QMIX, MAAC and DOP groups the randomly generated stochastic game.
QMIX is value-based method, need to add policy network and target policy network for each agent. We keep the original individual Q-functions to learn the critic in QMIX+DMAC. In divergence-reggulaiized MDP, the max operator is not needed in the critic update, the hypernet and use an MLP, which takes individual Q-values and state as input and produces the joint Q-value. This architecture is simple and its expressive capability is not limited by QMIX's IGM condition.
abandon
DOP. We keep the original critic and actor networks and add target policy network with the same architecture the actor. We keep the value decomposition structure and use off-policy TD(A) for all samples in training to replace the tree backup loss and on-policy TD(A) loss.
FOP. We replace the entropy regularizers with divergence regularizers in FOP and use the update rules of DMAC. We keep the original architecture of FOP.
C.2 HYPERPARAMETERS
As all tasks in our experiments are cooperative with shared reward, so we use parameter-sharing policy network and critic network for MAAC and MAAC+DMAC to accelerate training. Besides, we add RNN layer to the policy network and critic network in MAAC and MAAC+DMAC to settle the partial observability.
All the policy networks are the same as two linear layers and one GRUCell layer with ReLU activation and the number hidden units is 64. The individual Q-networks for QMIX group is the same as the policy network mentioned before. The critic network for COMA group is MLP with three 128-unit hidden layers and ReIU activation. The attention dimension in the critic networks of MAAC group is 32. The number of hidden units of mixer network in QMIX group is 32. The learning rate for critic is 10-3 and the learning rate for actor is 10 We train all networks with RMSprop optimizer. The discouted factor is 7 0.99. The coefficient regularizer
0.01. The td Jambda factor used in COMA group is 0.8. The parameter used for soft updating target policy is 7 0.01. Our
Figure 6: Learning curves in terms mean episode reward of COMA, MAAC, QMIX, DOP
groups in five SMAC maps (each row corresponds to map and each column corresponds to group).
17

---

Under review as a conference paper at ICLR 2022
20
20.0
20
420x000 tuwaros
35-
17.5
35-
15.0
TDP4DMAC
10.0
TOP4DWAC
FOP4DMAC
10P
10P
7DP
1.5
2.0
1.0
0.5
1e6
Figure Learning curves in terms of mean episode rewards of FOP+DMAC and FOP three SMAC maps (each column corresponds to map).
CDM
CDM
episode rewards
COMA+DMAC
DOP+ DMAC
-10
COMA
-10
DOP
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
step
step
Figure 8: Learning curves terms of mean episode rewards of COMA and DOP groups in the CDM environment used by the DOP paper.
code is based on the implementation of PyMARL (Samvelyan et al., 2019), MAAC (Iqbal and Sha, 2019), DOP (Wang et al., 2021b), FOP (Zhang et al., 2021) and open source code for algorithms in SMAC (https: / wtt.ggttthhhcooooo com wtw.ttttttttttotottooooeo.....ee sttssstarrryyyyyyyyy 6688/ (terrr-saatt///ttrrrrfftt
C.3 EXPERIMENT SETTINGS
We trained each algorithms for five runs with different random seeds. In SMAC tasks, we train each algorithm for one million time steps in each run for COMA, QMIX, MAAC, and DOP groups and two million timesteps for FOP groups We the experiments by a server with NVIDIA A100 GPUs.
D ADDITIONAL RESULTS
6.6
0.4
2.44
6.2
500
COVES
146
Figure 9: Learning curves of QMIX, DOP, COMA and MAAC groups the map MMM2 in SMAC.
Figure 5 shows the learning curves terms cover rates of COMA, QMIX, MAAC and DOP groups in the randomly generated stochastic game.
Figure 6 shows the learning curves of COMA, MAAC, QMIX and DOP groups in terms of mean episode rewards in each SMAC map.
Figure shows the learning curves of FOP+DMAC and FOP in terms of mean episode rewards in 3s_vs_32, _e_vs___________________________ vs_64zg and MMM2.
Figure shows the learning curves in terms mean episode rewards of COMA and DOP groups in the CDM environment used by the DOP paper.
Figure shows the learning curves of QMIX and DOP groups in terms of win rate, and COMA and MAAC groups in terms of episode reward on the map MMM2 in SMAC.
18


Reviews:
[{"review_id": "y0TImMHjifM", "reviewer": "Reviewer_sxWU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nTheoretical derivations have been included in the analysis of the algorithm in addition to the experiments. \nExperiments conducted are comprehensive, covering most of the state-of-the-art algorithmic benchmarks. \n\nWeaknesses:\nThe motivation for the target policy rho is not well-explained. What exactly is rho, and why do we need to maintain a rho to make learning more effective? Does it represent prior knowledge of team level policy pre-defined by the user or something else? If we already know what the optimal policy is, why do we need DMAC? If we don’t necessarily know the optimal policy, how can rho be chosen such that rho will help learning? Neither theoretical or empirical evaluations seem to address these concerns.\nThe degree of novelty seems unclear. On a high level, DMAC looks like a slight modification of the soft actor-critic(SAC) algorithm with joint action spaces, particularly in the definition of the objective function. The update rule is also standard in RL literature, and it would be ideal if the authors can further emphasize how their method differs from existing benchmarks. \n\n", "summary_of_the_paper": "This paper proposes DMAC, a general off-policy actor-critic framework with entropy-based divergence regularization for multi-agent reinforcement learning to better coordinate multiple agents in a complicated environment. Theoretical derivations and empirical investigations on several state-of-the-art algorithmic benchmarks over the SMAC environment demonstrate the effectiveness of DMAC.  ", "summary_of_the_review": "DMAC does provide improvements over existing algorithmic benchmarks, but the level of novelty seems limited as the DMAC method resembles SAC and several components of DMAC has questionable motivation. I’d be happy to change my score, if the authors address these issues accordingly. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "sIidNLsFur", "reviewer": "Reviewer_b5Li", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n1. The paper is well written, organized, and explains the main insights well. \n2. The proposed framework has a very appealing and important benefit that DMAC is flexible and can be easily combined with previous works. This benefit is highlighted well by combining with four representative MARL algorithms in the challenging SMAC domain. \n3. DMAC makes a conclusion in a principled manner, where the benefit of DMAC is thanks to the improved exploration without introducing the bias (Section 4.5). \n\n**Questions:**\n1. As pointed in the paper, the theoretical study in Section 4.2 is assuming the fixed target policy. However, in practice, the target joint policy is non-stationary/updated (see Algorithm 1). Hence, I wonder whether the theoretical conclusions (i.e., monotonic improvement, optimal policy convergence) would still hold with the non-stationary target joint policy.\n2. To address the partial observability, an RNN layer is used. However, it is unclear from reading the main paper and appendix, how the hidden state is handled in the off-policy implementation. Is the hidden state stored in the replay buffer as in Hausknecht and Stone, 2015?\n3. What are the possible future directions of DMAC? Can DMAC be extended to general-sum games (other than cooperative settings), and if not, what are the expected challenges?\n\n**Reference:**\nMatthew Hausknecht, Peter Stone. Deep Recurrent Q-Learning for Partially Observable MDPs. AAAI, 2015.", "summary_of_the_paper": "This paper develops a new multiagent framework with regulated divergence to address the learning of biased policies in the maximum-entropy approach. Specifically, the main contributions include the developments of the divergence policy iteration for general cooperative MARL settings and the off-policy divergence regularized multiagent actor-critic framework (DMAC). Empirical results show that DMAC can improve the performance of existing MARL frameworks in various multiagent domains.", "summary_of_the_review": "Initially, I vote for a score of 6. While some parts of the theoretical studies are based on related works (e.g., SAC, DOP), DMAC shows the appealing benefit that the framework can be easily combined with prior MARL works, so it has much applicability. After reading the authors' responses to my questions, I am open to raising my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "nAG4kztRro5", "reviewer": "Reviewer_nt66", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Regarding related work, it seems that closely related is Wang et al. 2019. I propose the give more details on the differences to the current work, as to why it cannot be applied to the cooperative MARL setting the authors study (because intuitively, cooperative MARL is verly close to the single-agent case).\n\nAm I understanding correctly that the theoretical guarantees (monotonic improvement) only hold for a fixed target policy rho? I.e., not for the additional update rule for rho in 4.5?  If so, that would be OK, it is clear that one cannot understand everyhting in one paper. However, it may be worth making more explicit if there is a theoretical guarantee also for this general method or not, and if there in fact is, then explicitly state this as a theorem.\n\nI suggest to give a bit more overview and motivation of what is being done where in the paper and how things are connected. For instance, it remained unclear to me how rho is defined, or what is intuitively is, until sec 4.5.", "summary_of_the_paper": "For cooperative multi-agent reinforcement learning, they propose an off-policy(?) learning method that is reqularized by the KL divergence to some target policy, which, in contrast to classic max entropy RL, allows to circumvent(?) the bias that the regularizer introduces (eq18).\nThey give several special cases / tricks of how the learning can be somewhat decoupled between the agent (Lem 3, end of sec4.4). They paper comprises theory about convergence of the method, as well as experiments in toy envs.", "summary_of_the_review": "MARL is a very challenging task, and I agree that entropy regularization introduces problematic biases, so they address a relevant and challenging problem. The paper is strong in presenting both theory (I took a brief look at the proofs, the authors seem to know what they are doing, but I did not go through details) and experiments. \n\nWhat I'm not so sure about is how deep the contribution is compared to previous work, in particular Wang et al. 2019, or, for instance, (Haarnoja et al., 2018), which is cited in the proof of Lem 2. Additionally, it seems the guarantees are only for fixed rho, while the full strength (nonbiasedness) of the method seems to come into play only when updating rho as well.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H7fL6AVZZPU", "reviewer": "Reviewer_GUrq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strong points\n1. In general, I think the proposed framework is at least new in the MARL literature. It is interesting to read a paper with a full deviation of such a divergence formulation of the learning objective.\n2. The mathematical justifications are completely and sufficient for the purpose of presentation, although I do feel some of them could be possible shrank or moved to the appendix to save some space for more discussions and experiments. For example, Lemma 2 & 3 look just the same to me, and the discussions on applying DMAC to COMA look nothing special to the audience familiar with COMA.\n3. I really appreciate seeing that the proposed methods can be really combined with multiple algorithms and gain improvement naturally.\n\n### Weak points\n1. Yes, the method is new in MARL but I'm still a bit concerned about the novelty of all the technical content. \n\nThere are sufficient mathematical discussions but I can still see most of the techniques are somewhat straightforward from existing works. For example, the value decomposition policy improvement theorem seems direct to obtain from the DOP paper; the mirror descent analysis seems very similar to the analysis in the DAPO paper. The authors state in the introduction section that \"DAPO cannot be trivially extended to cooperative MARL settings\". However, I don't really get this argument. It seems that all the mirror decent analysis simply holds directly with the assumption of linear-value decomposition. So, at least from the current form of the paper, the authors would have been done a better job explaining why those methods are challenging to be applied in MARL and what technical ideas really makes this paper distinct (rather simply saying that paper A is working on single-agent RL while paper B is on MARL). I hope the authors make more clarifications and discussions in the next revision. \n\n2. Experiments are not sufficiently convincing.\n\n(1) For Fig.1, why not run the same experiment setting as DOP so that we can have a straightforward comparison? From the curves in Fig.1, the behavior of COMA and DOP are contradicting the curves in the DOP paper. \n\n(2) The paper utilizes COMA as its on-policy baseline. It is a fair choice, but it is been known that COMA has less satisfying performances in many domains. Is it possible to include some strong on-policy baselines like PPO or IMPALA?\n\n(3) Except for FOP, the SMAC maps used in the experiments are less challenging. It would be more convincing if more hard maps can be examined. Also, for FOP, why not perform experiments on those simpler maps? The authors state that \"COMA+DMAC has higher win rates than COMA in most cases at convergence\" and \"DOP+DMAC learns faster than DOP in most cases and finally converges to better performance\". I agree that DMAC enhancement makes learning faster but I cannot really see the performances at convergence ---- at least from the visualization, DOP and COMA are still improving on some maps. \n\n\n### Writing \n1. It would be better if the authors could include some justifications at the beginning of section 4 on why such a ratio penalty is preferred. What is the motivation for this choice? Is this simply due to that we can eventually derive a nice KL-penalized value formulation?\n\n2. \"Besides, DMAC is **benefited** for exploration and stable policy improvement.\" --> beneficial\n\n3. \"**we** find that COMA+DMAC has higher win rates than COMA in most cases at convergence,\" --> capital w", "summary_of_the_paper": "This paper suggests a DMAC learning framework by introducing a divergence penalty between the learning policy between a target policy into the RL framework instead of the commonly used entropy penalty. It can be shown that the proposed objective can be easily combined with various MARL algorithms and can eventually lead to the optimal policy of the underlying unmodified MDP with properly set target policies. The paper rigorously studies the mathematical foundation of the algorithm following a standard workflow and presents a few experiments showing that DMAC could improve the learning efficiency of various MARL algorithms. ", "summary_of_the_review": "This paper presents a new algorithmic framework with sufficient mathematical discussions, although I'm a bit concerned about its current form. However, the paper can be improved if more novelty discussions can be presented and more thorough experiments can be conducted. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents multi-agent RL framework that uses the divergence between the learned policies and a target policy as a penalty that pushes the agent to learn cooperative strategies. The proposed method is built on top of an existing one (DAPO, Wang et al., 2019). Empirical experiments clearly show the advantage of the proposed method.

The reviews for this paper are mixed and borderline. The reviewers appreciate the experiments reported in the paper and that indicate the advantage of the proposed method. But two reviewers do not think that the proposed analysis is sufficiently novel compared to an existing one (DAPO). The responses provided by the authors were appreciated, but did not dissipate these concerns.
Decision: Reject