A Maximum Mutual Information Framework for Multi-Agent Reinforcement Learning
Authors: Woojun Kim
Whiyoung Jung
Myungsik Cho
Youngchul Sung
Venue:  (2021)
Keywords: Multi-agent reinforcement learning
coordination
mutual information

Abstract: In this paper, we propose a maximum mutual information (MMI) framework for multi-agent reinforcement learning (MARL) to enable multiple agents to learn coordinated behaviors by regularizing the accumulated return with the mutual information between actions. By introducing a latent variable to induce nonzero mutual information between actions and applying a variational bound, we derive a tractable lower bound on the considered MMI-regularized objective function. Applying policy iteration to maximize the derived lower bound, we propose a practical algorithm named variational maximum mutual information multi-agent actor-critic (VM3-AC), which follows centralized learning with decentralized execution (CTDE). We evaluated VM3-AC for several games requiring coordination, and numerical results show that VM3-AC outperforms MADDPG and other MARL algorithms in multi-agent tasks requiring coordination.

Under review as a conference paper at ICLR 2021
A MAXIMUM MUTUAL INFORMATION FRAMEWORK FOR MULTI-AGENT REINEORCEMENT LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
In this paper, we propose maximum mutual information (MMI) framework for multi-agent reinforcement learning (MARL) to enable multiple agents to learn coordinated behaviors by regularizing the accumulated return with the mutual information between actions. By introducing latent variable to induce nonzero mutual information between actions and applying a variational bound, we derive a tractable lower bound on the considered MMI-regularized objective function. Applying policy iteration to maximize the derived lower bound, we propose a practical algorithm named variational maximum mutual information multi-agent actor-critic (VM3-AC), which follows centralized learning with decentralized exe- cution (CTDE). We evaluated VM3-AC for several games requiring coordination, and numerical results show that VM3-AC outperforms other MARI. algorithms multi-agent -ggett tasks requiring coordination.
INTRODUCTION
With the success of RL in the single-agent domain (Mnih et al. (2015); Lillicrp et al. (2015), ), MARL is being actively studied and applied to real-world problems such traffic control systems and connected self driviniiing cars, which can be modeled as multi-agent systems requiring coordinated control (Li et al. (Andriots & Papakonstantinou/ (2019). The simplest approach to MARL is independent learning, which trains each agent independently while treating other agents as a part of the environment. One such example is independent Q-Iearning (IQL) (Tan (I993)), which an extension of Q-lcarning to multi-agent setting. However, this approach suffers from the problem of non-stationarity of the environment. common solution to this problem to use fully-centralized critic in the framework of centralized training with decentralized execution (CTDE) (OroollooyJadii Hajinezhad (2019): Rashid et al. (2018)). For example, MADDPG (Lowe al. (2017)) uses a centralized critic to train a decentralized policy for each agent, and COMA (Foerster et al. (2018) uses a common centralized critic to train all decentralized policies. However, these approaches assume that decentralized policies are independent and hence the joint policy is the product of each agent's policy. Such non-correlated factorization of the joint policy limits the agents to learn coordinated behavior due to negligence of the influence of other agents (Wen et al. (2019): de Witt et al. (2019) Thus, learning coordinated behavior is one of the fundamental problems in MARL (Wen et al. (2019)
Liu et al. (0220)).
In this paper, we introduce a new framework for MARL to learn coordinated behavior under CTDE. Our framework is based regularizing the expected cumulative reward with mutual information among agents' actions induced by injecting a latent variable. The intuition behind the proposed framework is that agents can coordinate with other agents if they know what other agents will do with high probability, and the dependence between action policies can be captured by the mutual informa- tion. High mutual information among actions means low uncertainty of other agents' actions. Hence, by regularizing the objective of the expected cumulative reward with mutual information among agents' actions, we can coordinate the behaviors of agents implicitly without explicit dependence enforcement. However, the optimization problem with the proposed objective function has several difficulties since we consider decentralized policies without explicit dependence or communication in the execution phase. In addition, optimizing mutual information is difficult because of the intractable conditional distribution. We circumvent these difficulties by exploiting the property of the latent variable injected to induce mutual information, and applying variational lower bound on the mutual

---

Under review as a conference paper at ICLR 2021
information. With the proposed framework, we apply policy iteration by redefining value functions to propose the VM3-AC algorithm for MARL with coordinated behavior under CTDE.
2 RELATED WORK
Learning coordinated behavior in multi-agent systems is studied extensively the MARL community. To promote coordination, some previous works used communication among agents (Zhang & Lesser (2013); Foerster et al. (2016); Pesce& & Montana (2019). For example, Foerster et al. (2016) proposed the DIAL algorithm to learn a communication protocol that enables the agents coordinate their behaviors. Instead of relying on communication, Jaques et al. (2018) proposed social influence intrinsic reward which is related to the mutual information between actions to achieve coordination. The purpose of the social influence approach is similar to our approach and the social influence yields good performance in social dilemma environments. The difference between our algorithm and the social influence approach will be explained in detail and the effectiveness of our approach over the social influence approach will be shown in Section 6 Wang al. (2019) proposed an intrinsic reward capturing the influence based on the mutual information between an agent's 's current actions/states and other agents' next states. In addition, they proposed an intrinsic reward based on a pocision-theoreti measure. Although they used the mutual information to enhance exploration, our approach focuses on the mutual information between simultaneous actions capturing policy correlation not influence. Besides, they considered independent policies, whereas policies are correlated in our approach.
Some previous works considered correlated policies instead of independent policies. For example, Liu et al. (2020) proposed explicit modeling of correlated policies for multi-agent imitation learning, and Wen et al. (2019) proposed a recursive reasoning framework for MARL to maximize the expected return by decomposing the joint policy into own policy and opponents' policies. Going beyond adopting correlated policies, our approach maximizes the mutual information between actions which measure of correlation.
Our framework can be interpreted as enhancing correlated exploration by increasing the entropy of own policy while decreasing the uncertainty about other agents' actions. Some previous works proposed other techniques to enhance correlated exploration (Zheng & Yue (2018); MMhajan et al. (2019). ). For example, MAVEN addressed the poor exploration problem of QMIX by maximizing the mutual information between the latent variable and the observed trajectories (Mahajan et al. (2019)).
However, MAVEN does not consider the correlation among policies.
BACKGROUND
We consider a Markov Game LLttman 1994)), which is an extention of Markov Decision Process (MDP) to multi-agent setting. An N-agent Markov game is defined by an environment state space S, action spaces for N agents AI. .N.
state transition probability T S x A S + [0, 1]. where A= II A, is the joint action space, and a reward function R x A + R. At each time step t, Agent executes action aj E A, based on state 8; E S. The actions of all agents a: = (at.... ... ag yield next state $1+1 according to T and yield shared common reward r; according to R under the assumption of coolyrcooeee MARL. The discounted return is defined as R, = Di Y rr, where 7 [0,) 1) is the discounting factor.
We assume CTDE incorporating resource asymmetry between training and execution phases, widely considered in MARL (Lowe al. (2017): Iqbal & Sha (2018) Foerster al. (2018)) Under CTDE, each agent can access all information including the environment state, observations and actions of other agents in the training phase, whereas the policy of each agent can be conditioned only on its own cottionobbeervation history Fi or observation of in the execution phase. For given joint policy (iT) .... N), the goal of fully cooperative MARL is to find the optimal joint policy π that maximizes the objective J(x) For RR]
Maximum Entropy RL The goal of maximum entropy RL is to find an optimal policy that maximizes the entropy-regularize objective function, given by
((T) = En
Σ 7 (r((s) ar) + (((ss,a))+Hll 1=0
+ +zH(((((1)))))
(1)
2

---

Under review as a conference paper at ICLR 2021
It is known that this objective encourages the policy enhance exploration in the state and action spaces and helps the policy avoid converging to local minimum. Soft actor-critic (SAC), which is based the maximum entropy RL principle, approximates soft policy iteration to the actor-critic method. SAC outperforms other deep RI. algorithms in many continuous action tasks (Haarnoja et al. (2018))
We can simply extend SAC to multi-agent setting in the manner of independent learning. Each agent trains its decentralized policy using decentralized critic maximize the weighted sum of the cumulative return and the entropy of its policy. We refer to this method as Independent SAC (I-SAC). Adopting the framework of CTDE, we can replace decentralized critic with centralized critic which incorporates observations and actions of all agents. We refer to this method as multi-agent soft actor-critic (MA-SAC). Both I-SAC and MA-SAC are considered as baselines the experiment section.
4 THE PROPOSED MAXIMUM MUTUAL INFORMATION FRAMEWORK
We assume that the environment is fully observable, i.e., each agent can observe the environment state s, for theoretical development in this section, and will consider partially observable environment for practical algorithm construction under CTDE in the next section.
Under the proposed MMI framework, aims to find the policy that maximizes the mutual informa- tion between actions in addition to cumulative return. Thus, the MMI-regularized objective function for joint policy is given by
J(T) FOr
Σ
(1(s), + [
(15)
where of ~~ (-(ss) and the temperature parameter that controls the relative importance of the mutual information against the reward.
As aforementioned, coordinated behavior. By regularization with mutual information in the proposed objective function (2) the policy of each agent implicitly encouraged to coordinate with other agents' policies without explicit dependency by reducing the uncertainty about other agents' policies. This can be seen as follows: Mutual information is expressed in terms of entropy and conditional entropy as
assume decentralized policies and want the decentralized policies exhibit
1( [ (-1st))); T/
(-1se) m
(-(ss) H (+))))))))))) (+(s)))
If the knowledge of x <-(ss) does not provide any information about n/ (-18x), the conditional en- tropy reduces to the unconditional entropy, i.e., (-1++))==+(())) information becomes zero. Maximizing mutual information is equivalent to minimizing the un- certainty about other agents' policies conditioned on the agent's 's own policy, which can lead the agent learn coordinated behavior based the reduced uncertainty about other agents' policies.
H (-18:)). and the mutual
However, direct optimization of the objective func- tion ( is not easy. Fig. a[)) shows the causal diagram of the considered system model described in Section
3 in the case of two agents with de- centralized policies. Since we consider the case of no explicit dependency, the two policy distribu- tions can expressed as π (a! (st) and x (a st). Then, for given environment state observed by both agents, (aa st)) and (a]])=) are condi- tionally independent and the mutual information I( (-[se): (-(st)) = 0. Thus, the MMI objective (2) reduces to the standard MARL objective of only the accumulated return. In the following subsec- tions, we present our approach to circumvent this difficulty and implement the MMI framework and its operation under CTDE.
Figure 1: Causal diagram in 2-agent Markov
Game: (a) Standard MARL, (b) Introducing the latent variable to the standard MARL
3

---

Under review as a conference paper at ICLR 2021
4.1 INDUCING MUTUAL INFORMATION USING LATENT VARIABLE
First, in order to induce mutual information among agents' policies under the considered system causal diagram shown in Fig. aa). a). introduce latent variable 2t. For illustration, consider the new diagram with latent variable Ee in Fig. [b). Suppose that the latent variable = has prior distribution p(ze), and assume that both actions aj and a; are generated from the observed random variable 8t and the unobserved random variable 2, Then, the policy of Agent i is given by the marginal distribution r'((1s)) = J-= x <-(-18 2)p(=)dz marginalized over 2. With the unobserved latent random variable 2, the conditional independence does not hold for of and až and the mutual information can be positive, i.e., (- (-1s,)) > O. Hence, we can induce the mutual information between actions without explicit dependence by introducing the latent variable. In the general case of N agents, we have Traa, a Is) Ex [x (a' 2,
....... .....""" TN (a 2)]. Note that in this case we inject common latent variable z into all agents' policies.
4.2 VARIATIONAL. BOUND OF MUTUAL INFORMATION
Even with non-trivial mutual information [( (-Is;); T (-Iss))), it difficult to directly compute the mutual information. Note that we need the conditional distribution of a; given (a), 8x) to compute the mutual information seen in 4 but it is difficult to know the conditional distribution directly, To circumvent this difficulty, use variational distribution g(a)) a s,) to approximate p(a/la), sa) and derive a lower bound on the mutual information T( (-1se) x (-1x,) =: Tig((4 as
g(a//aa (aallaj,s) st) p(a²)
[i5(st) = pptot log
+ Eptablee)
2 H( (-1st) E. (fa)) [log a(al (ai, st)]
(4)
where the incquality holds because KL divergence is always non non-negative. The lower bound becomes tight when q(az (af, st) approximates p(aş la), st) well. Using the symmetry of mutual information, we can rewrite the lower bound as
1 (+ + H(T (-1s)) + E Ppiafaffeed [log 2
[Iog c(a/la / 8t) + log g(ailaf st) s (5)
(x)) 2
Then, we can maximize this lower bound of mutual information by using the tractable approximation ((a))a),s.).
4.3 MODIFIED POLICY ITERATION
In this subsection, we develop policy iteration for the MMI framework. First, we replace the original MMI objective function (2) with the following tractable objective function based the variational
lower bound 55)
N x (rolsr, ar) + aN Σ t=
N
J(TT,q) q) = Em
Σ
H- +
log (o)
(6)
1=1
where (laglaf,* 8t) is the variational distribution to approximate the conditional distribution pla lat, st). Then, we determine the individual objective function Jr('', 4) for Agent as the sum of the terms in (6) associated with Agent i's policy n² or action af, given by j'('''
B jft
log plgalaaa/ st) + log (oga(a/(oo. s.) 1)] (7)
(e(s), ae) + ( En Σ
(0)
where = a. is the temperature parameter. Note that maximizing the term (a) in implies that each agent maximizes the weighted sum of the policy entropy and the return, which can be interpreted as extension of maximum entropy RL to maximizing the term (b) with respect to x' means that we update the policy n' so that Agent j well predicts Agent i's action by the first term in (b) and Agent well predicts Agent j's action by the
setting. On the other hand,
4

---

Under review as a conference paper at ICLR 2021
second term in (b). Thus, the objective function can be interpreted as the maximum entropy MARL objective combined with predictability enhancement for other agents' actions. Note that predictability is reduced when actions are uncorrelated. Since the policy entropy term H HTr' (- (-1s,) enhances individual exploration due to maximum entropy principle (Haarnoja et al. (2018) and the term (b) in (7) enhances predictability correlation among agents' actions, the proposed objective function (7) can be considered as one implementation of the concept of correlated exploration in MARL Mahajan et (2019)).
Now, in order to learn policy x' to maximize the objective function 7 we modify the policy iteration in standard RL. For this, we redefine the state and state-action value functions for each agent as follows:
00 o C r++ B11(T (1++)(((()))))))) +
B N
(8) 2 Ex
Σ
log q (1:5) (a), az 8e) =
(8)
(5, a) & Ex ro + (s," 50 5,00
(9)
where (a)) al si) - q(a//al s s)) s1). Then, the Bellman operator corresponding to Vi" and Q is given by
( 7 (V('s))
(10)
where
Vi(+) Eaan
(a'ls) N Σ
(11)
In the policy evaluation step, we compute the value functions defined in (4) and 15) by applying the modified Bellman operator 7 repeatedly any initial function Q
Lemma 1. (Variational Policy Evaluation). For fixed π and the variational distribution q. consider the modified Bellman operator T in (16) and an arbitrary initial function Q :S x A R, and define Q.++1 = = Then, Q converges to Q defined in 15
Proof. See Appendix A.
In the policy improvement step, we update the policy and the variational distribution by using the value function evaluated in the policy evaluation step. Here, each agent updates its policy and variational distribution while keeping other agents' policies fixed as follows: (TE+1)
Blog (a' -
Σ log (a', .2)) (12) jfi
arg max (a (0) r'4g
+
N
where a a (al,... (a'......""" Ma'). Then, we have the following lemma regarding the improvement step.
Lemma 2. (Variational Policy Improvement). Let Theu and qnew be the updated policy and the
variational distribution (30) in Appendix A. Then, Q n. Flld (s, a..)== 2 Q2 Tai (s, a) for all (s, a) E (S x A).
Proof. See Appendix A.
The modified policy iteration is defined as applying the variational policy evaluation and variational improvement steps in an alternating manner. Each agent trains policy, critic and the variational distribution to maximize its objective function
5 ALGORITHM CONSTRUCTION
Summarizing the development above, we now propose the variational maximum mutual information multi-agent actor-critic (VM3-AC) algorithm, which be applied to continuous and partially
5

---

Under review as a conference paper at ICLR 2021
Centralizeed training
Operation of o
Decentralized execution
N(O.I)
X2
M( of.oo)
log loga(a"aaa
@ Vr
Figure 2: Overall operation of the proposed VM3-AC. We only need the operation in the red box after training.
observable multi-agent environments under CTDE. The overall operation of VM3-AC is shown in Fig. Under CTDE, each agent's policy is conditioned only local observation, and centralized critics are conditioned on either the environment state or the observations of all agents, depending on the situation (Lowe al. (2017). Let æ denote either the environment state s or the observations of all agents (01, .NN). ), whichever is used. In order to deal with the large continuous state-action spaces, we adopt deep neural networks to approximate the required functions. For Agent i, we parameterize the variational distribution with ç as qę: (a* (aalaaa o², o²), the state-value function with us as V (at), two action-value functions with 0+.1 and 00.2 as Q (az, a)), Qg. 2 (x., a), and the policy with of as T (alo")
E of (alo', We assume normal distribution for the latent variable which plays a key role in inducing coordination among agents' policies, i.c., 21 ~ (O, 1), and further assume that the variational distribution is Gaussian distribution with constant variance o², ie.. qe: (a* la or, o²) ( of, o²); )²), where Me (a', o', o²) is the mean of the distribution.
Centralized Training As aforementioned, the policy is the marginalized distribution over the latent variable 2, where the policies of all agents take the same 2 generated from NN I) as an input variable. We perform the required marginalization based Monte Carlo numerical expectation as follows:
π <(aas) = Fez (m (a'1s, 2). T (a 8
2)
Σ L
T (a (,,22) (a
(a (1, 2'),
(13)
and we use L = 1 for simplicity. The parameterized value functions, policy, and variational distributions are trained similarly to the training in SAC. Due to space limitation, training detail and pseudo code are provided in Appendices B and C.
Decentralized Execution In the centralized training phase, we pick actions (a' ........ N by using Monte Carlo expectation based on common latent variable z' generated from zero-mean Gaussian distribution, as seen T3p. We consider two methods achieve the same operation the decentral- ized execution phase. First, this can be done by making all agents have the same Gaussian random sequence generator and distributing the same seed to this random sequence generator only once in the beginning of the execution phase. This eliminates the necessity of communication for sharing the latent variable. fact, this way of sharing z' can be applied the centralized training phase too. Second, exploit the property of zero-mean Gaussian latent variable z'. That is, simply replace z! with zero vector with the matching dimensions in the decentralized exccution phase. This substitution method does not deteriorate the performance much seen in Section 2 since the latent variable distribution is zero-mean Gaussian and zero has the highest density. Thus, the proposed algorithm is fully operative under CTDE.
EXPERIMENT
In this section, we provide numerical results to evaluate VM3-AC, We considered four baselines: 1) MADDPG (Lowe al. (2017)) extension of DDPG with centralized critic to train

---

Under review as a conference paper at ICLR 2021
(a) MW (N=3)
(b) MW (N=4)
(c) PP (N=2)
(d) PP (N=3)
(e) PP (N=4)
(f) CN (N=3)
Figure 3: Performance of MADDPG (blue), MA-AC (green), MAVEN (purple), IMOA (black), and VM3-AC (the proposed method, red) multi-walker environments (a)-(b), predator-prey (c)-(e), and cooperative navigation (f). (MW, PP, and CN denote malki- predator-pre prey, and cooperative navigation, respectively)
decentralized policy for each agent. 2) Multi-agent actor-critic (MA-AC)- variant of VM3-AC (8 without the latent variable. 3) Multi-agent variational exploration (MAVEN) MMahjjan et al. (2019) Similarly to VM3-AC, MAVEN introduced latent variable and variational approach for optimizing the mutual information. However, MAVEN does not consider the mutual information between actions but consider the mutual information between the latent variable and trajectories of the agents. 4) Influence MOA (IMOA) also known as social influence (SI) (Jaques et al. (2018)). The IMOA method models p(a lat, st), where agents j and are influencee and influencer, respectively, and adds intrinsic reward given by the mutual information I [af; (ss) between influencer's current action and influencee's 's next timestep action not the mutual information I (at; af l/ss) between two agents' simultaneous actions, which is considered in our MMI framework. Both MAVEN and IMOA are implemented on the top of MA-AC since we consider continuous action-space environments. We evaluated the proposed algorithm and the baselines in three multi-agent environments with the varying number of agents: multi-walker (Gupta et al. (2017)), predator-prey (Lowe et al. (2017)), and cooperative navigation (Lowe et al. (017)). We modified the original environments to require further coordination among agents. For example, we increased the size of the agent and collision reward in the cooperative navigation. Hence, the agent should consider other agents more while achieving its goal. The detailed setting of each environments is provided in Appendix D.
6.1 RESULT
Fig. shows the learning curves for the considered three environments with the different number of agents. The y-axis denotes the average of all agents' rewards averaged over 7 random seeds, and the x-axis denotes time step. The hyperparameters including the temperature parameter 3 and the dimension of the latent variable are provided in Appendix E. As shown in Fig. 3 VM3-AC outperforms the basclines in the considered environments. Especially, in the case of the multi- environment, VM3-AC has large performance gain. This because the agents in the multi-walker environment are required especially to learn coordinated behavior to obtain high rewards. In addition, the agents in the predator-pres pedly environment, where the number of agents is four, should spread out in groups of two to get more reward. In this environment, VM3-AC also has large performance gain. Thus, is seen that the proposed MMI framework improves performance in complex multi-agent tasks requiring high-quality coordination. It is observed that both MAVEN and IMOA outperform the basic algorithm MA-AC but not VM3-AC. Hence, the mutual information between the latent variable and trajectory (used in MAVEN) and the mutual information between the action of the agent and the
7

---

Under review as a conference paper at ICLR 2021
Figure 4: The positions of four agents after five time-steps after the episode begins in the early stage of the training: 1st row VM3-AC and 2nd row MA-SAC. The figures in column correspond to different seed. The black squares are the preys and each color except black shows the position of each agent.
next time step action of other agents (used in IMOA) are not as effective for coordinated behavior as the mutual information between agents' simultaneous actions used for VM3-AC.
6.2 ABLATION STUDY
In this section, we provide ablation study on the major techniques and hyperparameter of VM3-AC: 1) mutual information versus entropy 2) the latent variable, the temperature parameter /3 and 4) injecting zero vector instead of the latent variable z to policies the execution phase.
The proposed MMI framework maximizes the sum of entropy and variational conditional probability, which provides a lower bound of mutual information between actions. As aforementioned, maximizing the entropy and variational conditional probability enhance exploration and predictability for other agents' actions, respectively. Hence, the proposed MMI framework enhances correlated exploration among agents. We compared VM3-AC with MA-SAC, which is an extension of maximum entropy RI. to multi-agent setting. We performed an experiment the predator-prey environment with four agents where the number of required agents to catch the prey is two. In this environment, the agents started at the center of the map. Hence, the agents should spread out the group of two to catch preys efficiently. Fig shows the positions of the four agents at five time-steps after the episode starts. The first row and the second row in Fig show the results of VM3-AC and MA-SAC the early stage of the training, respectively. It is seen that the agents of VM3-AC explore in the group of two while the agents of MA-SAC tend to explore independently. We provided the performance comparisons of VM3-AC with MA-SAC in Fig (a) and (b).
Mutual information versus entropy:
Latent variable:
The role of the latent variable is to induce mutual information among actions and promote coordinated behavior. We compared VM3-AC and VM3-AC without the latent variable (implemented by setting dim(z) 0) in the multi-walkes environment. In both cases, VM3-AC yields better performance that VM3-AC without the latent variable as shown in Fig 5 a) and ((b).
Injecting vector instead of the la- tent variable: As mentioned in Section [] we replace the latent variable with zero vector to execute actions without com- munication in the execution phase. We compared the performance of decentral- ized policies which use zero vector and decentralized policies which use the la- tent variable assuming communication. We used deterministic evaluation based on 20 episodes generated by the corresponding deterministic policy, i.c., each agent selects action using the mean network of Gaussian policy T We averaged the return over 7 seeds, and the result
Table I: Impact of replacing the latent variable z
(O, I) with zero vector 2 = in the execution phase
PP (N=2)
PP (N=3)
PP (N=4)
2 N(.
413
734
1123
2
409
743
1147
8

---

Under review as a conference paper at ICLR 2021
(a) PP (N=4)
(b) MW (N=4)
(c) MW (N=3)
(d) MW (N=4)
Figure 5: (a) and (b): VM3-AC (red), VM3-AC without latent variable (orange), and MA-SAC (cyan) and (c) and (d): performance with respect to the temperature parameter
seen that the zero vector replacement method yields almost the same performance and enables fully decentralized execution without performance loss.
shown Table 3
Temperature parameter 3: The role of temperature parameter is control the relative im- portance between the reward and the mutual information. We evaluated VM3-AC by varying == [0, 0.05, 0.1, 0.15] in the multi-walker environment with N 3 and 4. Fig. Stc) and 5)d) show that VM3-AC with the temperature value around [0.05, 0.1] yields good performance.
7 CONCLUSION
In this paper, have proposed the MMI framework for MARL to enhance multi-agen coordinated learning under CTDE by regularizing the cumulative return with mutual information among actions. The MMI framework is implemented practically by using latent variable and variational technique and applying approximate policy iteration. Numerical results show that the derived algorithm named VM3-AC outperforms other baselines, especially in multi-agent tasks requiring high coordination among agents. Furthermore, the MMI framework can be combined with other techniques for cooperative MARL, such as value decomposition (Rashid al. (2018)) to yield better performance.
REFERENCES
Praveen Agarwal, Mohamed Jllli, and Bessem Samet. Fixed point theory in metric spaces. Recent Advances and Applications, 2018
CP Andriotis and KG Papakonstantinou. Managing engineering systems with large state and action spaces through deep reinforcement learning, Reliability Engineering & System Safety, 191:106483, 2019.
Christian Schroeder de Witt, Jakob Foerster, Gregory Farquhar, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Multi-agent common knowledge reinforcement learning. In Advances in Neural Information Processing Systems, pp.
2019.
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in neural information processing systems, pp. 2137-2145, 2016..
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Thirty-second AAAI conference on artificial intelligence, 2018.
Gerald B Folland. Real analysis: modern techniques and their applications, volume 40. John Wiley & Sons, 1999.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 1802.09477, 2018.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi- agett control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pp. 63-83. Springer, 2017.

---

Under review as a conference paper at ICLR 2021
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.012 2018.
Shariq lqbal and Fei Sha. Actor-attention-citiie for multi-agent reinforcement learning. arXiv preprint arXiv:1810.022 029102..0......... 2018.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro A Ortega, DJ Strouse, Joel Z. Leibo, and Nando De Freitas. Social influence intrinsic motivation for multi-agent deep reinforcement learning. arXiv preprint arXiv: /8 10.006647, 2018.
Woojun Kim, Myungsik Cho, and Youngchul Sung. Message-dropout An efficient training method for multi-agent deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6079-6086, 2019.
Langley. Crafting papers on machine learning. Pat Langley (ed.), Proceedings of the 17th International Conference Machine Learning (ICMI. (/0M),000)), pp. 1207-1216, Stanford, CA,, 2000. Morgan Kaufmann.
Minne Li. Zhiwei Qin, Yan Jiao, Yaodong Yang. Jun Wang, Chenxi Wang, Guobin Wu, and Jieping Ye. Efficient ridesharing order dispatching with mean field multi-agent reinforcement learning. In The World Wide Web Conference, pp. 983-994, 2019.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.022 50.00999,, 2015.
Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
Minghuan Liu, Ming Zhou, Weinan Zhang, Yuzheng Zhuang, Jun Wang, Wulong Liu, and Yong Yu. Multi-agent interactions modeling with correlated policies. arXiv preprint mxXiv:200..03415 2020.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed pnperaaiive-ompetttivv environments. Advances in Neural Information Processing Systems, pp. 6379-6390 2017.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. In Advances in Neural Information Processing Systems, pp. 761 1-7622, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-555 533, 2015.
Afshin OroojlooyJadid and Davood Hajinezhad. A review of cooperative multi-agent deep reinforce- ment learning. arXiv preprint
2019.
Emanuele Pesce and Giovanni Montana. Improving coordination small-scale multi-agent deep reinforcement learning through memory-driven communication. arXiv preprint arXiv:1901.03880 2019.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv:/803: /485, 2018.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pp. 330-337, 1993.
Tonghan Wang, Jianhao Wang. Yi Wu, and Chongjie Zhang. Influence- based multi-agent exploration. 2019.
arXiv preprint
10

---

Under review as a conference paper at ICLR 2021
Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning, arXiv preprint arXiv: 2019.
Chongjie Zhang and Victor Lesser. Coordinating multi-agent reinforcement learning with limited communication. In Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems, pp. 1101-1108, 2013.
Stephan Zheng and Yisong Yue. Structured exploration via hierarchical variational policy networks. 2018.

---

Under review as a conference paper at ICLR 2021
APPENDIX A: VARIATIONAL POLICY EVALUATION AND POLICY IMPROVEMENT
In the main paper, we defined the state and state-action value functions for each agent as follows:
(s) - ET
T (
+
(1
+
B
log (0 (a): at. si)) 30 =s
(14)
N
it
80 s, 0,00 = @7 0) - Ex ro + vV, (3,
(15)
Lemma 3. (Variational Policy Evaluation). For fixed π and the variational distribution q. consider
the modified Bellman operator T in (16) and an arbitrary initial function Q S x A IR and define Oi+1 =T"Qu Then, Q converges to Q defined in 15
TQ((s,a) + -Es
(16)
where
Vi(+) Fann
a((s,a)- Blg (a'ls)
Σ
(17)
JF
Proof.
Define the mutual information augmented reward as T (i(8), =
= = az) + - Q(41)+1,0411) - log (a +
Σ
log (( (htt 8t)
N
jfz
(18)
B
ai) + t7
Blg n' (ailss) +
Σ
log gl ((w)) (af, (a).
log
(19)
N
iti
(=(
"Bunropa Q0(8441,0442)
(20)
rs az) + -a (((5))))4411)
(21)
Then, can apply the standard convergence results for policy evaluation. Define
(w) R- + +pp=
(22)
for = (Q(s; a)S.ES. Then, the operator T is a 7-contraction.
(T^(')) - (w) = (R + +P* (R" + +>D"u)lloo
(23)
1yp (u u) 1lx
(24)
S |yP"||aell w/loo
(25)
S >rllu~ flloc
(26)
Note that the operator T has an unique fixed point by the contraction mapping theorem, and we define the fixed point as Q (s, a)) Since
|[ (s, a.) as) S x "s,,),"(ssa)))) QT, $ QT, |)
(27)
we have
lim -)) = 0
(28)
*
12

---

Under review as a conference paper at ICLR 2021
and this implies
lim (s, = (s, V(s,a) E (8 x A).
(29)
We proved the variational policy evaluation in the finite state-action sets. We can expand it to the infinite state-action sets by assuming follows:
Assume that Q functions for are in L infinity
From Folland 1999). I. infinity is Banach space
From Agarwal al. 2018 by Banach fixed point theorem, Q function should converge
to a unique point in infinity space and that is the Q function of given
Lemma 4. (Variational Policy Improvement). Let Thow and Quew be the updated policy and the variational distribution from 330) Then, FemewtTld a) 2 Q Jolaraala (s, a) for all (s, a)
(S A).
1141100411 = arg max F( t)m(tii 3)
0:th
(s, (s,
Bllo (a'l''(a')s))
(30)
x'.
Σ
log ( (a a s))
(31)
N
jti
Proof. Let Trem be determined as
nnew (new) = arg max E E(aj,a ( widd
Qua a.) - 3 log x (afl)s)
(32)
xi.
log g (1:)) (a)) ai ((,
(33)
jz
Then, the following inequality is hold
( (otd ax) - -Blog τं i (a(lse)
Σ
log ((an)) (ai, af st))
(34)
N
2 B(aaa ((a
(aTai (st, ar) - Bog +
Σ
(1-5 log gold (at, a. st))
(35)
N
(36)
13

---

Under review as a conference paper at ICLR 2021
From the definition of the Bellman operator, Q, (St, at) = r(t,, 0t) + 7 7 (++Ll(((+1))
(37)
I r(s,, az) + y 7 (ajy) (Them Q.all (st11,
Bloog new (01+118441) + 8
Σ
log (ii1 (ait2) (4p1
(38)
821
rst,, az) + E Bl (
r't (((11,,
Blogg neew (c1+119441) + +,8 Σ
log hiel (a/+1, ae+1, 9(+1) + VV, ($1+2)
jzi
(39)
Q new Jold (5c, Ut).
(40)
14

---

Under review as a conference paper at ICLR 2021
APPENDIX B: ALGORITHM CONSTRUCTION
7.1 CENTRALIZED TRAINING
The value functions Vil. (z). Q6, (xe, a) are updated based on the modified Bellman operator defined
in 16 and (17) The state-value function Vi, (a) is trained to minimize the following loss function:
(v(r²) FRODD (2))
Vi.. (2
(41)
where
Vi ((r())))
Qman ar)
B log T (aflot)
8 N
(3-5 (at ar or of) D is the replay buffer that stores the transitions (ær, azs Tt, Tt+1). Ezz log qet
and Qim (2:;, at) min
Qig. (Ix,, a)), 2g.a (at, af) the minimum of the two action-value functions to prevent the overestimation problem Fujimoto et al. (2018). The two action-value functions are updated by minimizing the loss
1/1 (0 ai) - Q ai))
(o E(er,.))DD
(42)
where
Q(x,, a.) = (1 ax) + YE VV (I--1)
(43)
and Vi is the target value network, which updated by the exponential moving average method. We implement the reparameterization trick to estimate the stochastic gradient of policy loss. Then, the action of agent is given by a' for (s; 2), where c¹ ~ (0,, and ~ 1). The policy for agent and the variational distribution are trained to minimize the following policy improvement loss,
C G =
(z,, (x), a) + 3 log ग (d
3 Σ N
log
(1:) (a' of of) qe
D. N
j
(44)
where
(6,5) (ai, of,0) (atlait of) (gc(a (ablai, of). 4
ge (a)
(45)
Since approximation of the variational distribution is not accurate in the early stage of training and the learning via the term (a) in equation 45 is more susceptible to approximation error, we propagate the gradient only through the term (b) in equation 45 to make learning stable. Note that minimizing log ge (a' Ja .t) is equivalent to minimizing the mean-squared error between a and M (a o², o²) due our Gaussian assumption on the variational distribution.
15

---

Under review as a conference paper at ICLR 2021
APPENDIX C: PSEUDO CODE
Algorithm 1 VM3-AC (L=1)
Centralized training phase Initialize parameter Φ¹, 0', , 10 1. ,5' Vi (1,... for episode 2, ... do Initialize state So and each agent observes of for < T and s, terminal do
Generate 24 ~ N0, I) and select action at ~ =- zt) for each agent Execute at and each agent receives r; and of_
Store transitions in D
end for
for each gradient step do
Sample a minibatch from and generate 2 ~ 1), for each transition.
Update 0² ve by minimizing the loss (42) and 43
end for Update $², by minimizing the loss
Update & using the moving average method end for
Decentralized execution phase Initialize state so and each agent observes of for each environment step do
Select action at T (-lof, 2)) where 0 (or sample from the Gaussian random sequence generator with the same seed)
Execute de and each agent receives o/41
end for
16

---

Under review as a conference paper at ICLR 2021
APPENDIX D: ENVIRONMENT DETAIL
Multi-walker The multi-walker environment, which was introduced in Gupta et al. (2017). is a modified version of the Bipedal environment in OpenAl gym to multi-agent setting. The environment consists of N bipedal walkers and a large package. The goal of the environment is to move forward together while holding the large package on top of the walkers. The observation of each agent consists of the joint angular speed, the position of joints and so on. Each agent has 4-dimensional continuous actions that control the torque of their legs. Each agent receives shared reward Ri depending on the distance over which the package has moved and receives negative local compensation Rg if the agent drops the package or falls to the ground. An episode ends when one of the agents falls, the package is dropped or I time steps elapse. To obtain higher rewards, the agents should learn coordinated behavior. For example, one agent only tries to learn to move forward, ignoring other agents, then other agents may fall. In addition, the different coordinated behavior is required as the number of agents changes. We set T = 500, R₂ = 10 and R₁ = 10d, where is the distance over which the package has moved. We simulated this environment in three cases by changing the number of agents (N- 2, N = 3, and N 4).
All algorithms used neural networks to approximate the required functions. In the algorithms except I-SAC, we used the neural network architecture proposed in Kim et al. (2019) to emphasize the agent's own observation and action for centralized critics. For agent i, we used the shared neural network for the variational distribution q:= (al laf, of. of) for j (1..... N and the network takes the one-hot vector which indicates as input. Experimental details are given in Appendix E.
Predator-prey The predator-prey environment, which is a standard task for MARL, consists of N predators and M preys. We used a variant of the predator-prey environment into the continuous domain. The initial positions on the predators are randomly determined, and those of the preys are in the shape of a square lattice as shown in f (b). The goal of the environment is to capture many preys as possible during a given time T. A prey is captured when C predators catch the prey simultancously. The predators get team reward Ri when they catch prey. After all of the preys are captured and removed, we set the preys to respawn in the same position and increase the value of R1. Thus, the different coordinated behavior needed as N and C' change. The observation of each agent consists of relative positions between agents and other agents and those between agents and the preys. Thus, each agent can access to all information of the environment state. The action of each agent is two-dimensional physical action. We set
Dwe
(a)
(b)
(c)
Figure 6: Considered environments: (a) Multi-
walker, (b) Predator-pree prey, and (c) Cooperative navigation
R1 = 10 and T = 100. We simulated the environment with three cases: (N = 2, M = 16, C = 1), (N 3, M = 16, C' = 1) and (N 4, MM = 16, C 2).
Cooperative navigation Cooperative navigation, which was proposed in Lowe et al. (017)). consists of N agents and L landmarks. The goal of this environment is occupy all landmarks while avoiding collision with other agents. The agent receives shared reward R₁ which is the sum of the minimum distance of the landmarks from any agents, and the agents who collide each other receive negative reward- - R2. In addition, all agents receive R3 if all landmarks are occupied. The observation of each agent consists of the locations of all other agents and landmarks, and action is two-dimensional physical action. We set R2 = 10, R3- = 1, and T = 50. We simulated the environment in the cases
of (N 3, L = 3).
17

---

Under review as a conference paper at ICLR 2021
APPENDIX E: HYPERPARAMETER AND TRAINING DETAIL
The hyperparameters for MA-AC, MA-SAC, MADDPG, and VM3-AC are summarized Table 2
Table 2: Hyperparameters of all algorithms
MA-AC
MA-SAC
MADDPG
VM3-AC
REPLAY BUFFER SIZE
5 10
5 10
x 10
5 10
DISCOUNT FACTOR
0.99
0.99
0.99
0.99
MINI-BATCH SIZE
128
128
128
OPTIMIZER
ADAM
ADAM
ADAM
ADAM
LEARNING RATE
0.0003
0.0003
0.0003
0.0003
TARGET
NUMBER HIDDEN COEFFICIENT
0.005
0.005
0.005
0.005
(ALL NETWORKS)
NUMBER OF HIDDEN UNITS PER LAYER
128
128
128
128
ACTIVATION FUNCTION FOR HIDDEN LAYER
RELU
RELU TANII
RELU
RELU
ACTIVATION FUNCTION FOR FINAL LAYER
TANHI
TANH
TANH
Table The temperature parameter 3 and the dimension of the latent variable for VM3-AC on the considered environments. Note that the temperature parameter /3 in I-SAC and MA-SAC controls the relative importance between the reward and the entropy. whereas the temperature parameter /3 in VM3-AC controls the relative importance between the reward and the mutual information.
VM3-AC
DIM(Z)
MW (N=3)
0.05
8
MW (N=4)
0.1
8
(N=2)
PP (N=3)
0.1
8
PP (N=4)
0.2
8
CN (N=3)
0.1
8
18


Reviews:
[{"review_id": "V9X3jZDWahP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose to include the mutual information between agents' simultaneous actions in the objective to encourage coordinated behaviour. To induce positive mutual information, the authors relax the assumption that the joint policy can be decomposed as the product of each agent's policy, independent of each other given the state, and they achieve so by introducing a latent variable that correlates agents behaviours. Since the mutual information is difficult to compute, the authors proposed to maximise a parametric lower bound. The algorithm is theoretically motivated as a policy iteration variation in its exact tabular form. But experiments are performed with neural network approximations on some environments. Numerical results show improvements over previous similar techniques.\n\nStrong points:\nThe mutual information objective, the latent variable, the variational lower bound, and the algorithm are well motivated.\nPaper is well written. \nSimulation results seem convincing.\n\nWeak points:\nThe decentralised execution relying on having random generators with the exact same seed is not a robust solution.\nThe presentation as modified policy iteration is OK, but quite straightforward. The characterisation as a contraction is similar to many other works, maybe a citation would have helped.\n\n\nQuestions:\nIs always coordination desirable? Could be some adversarial example of an environment where the optimal policy requires lack of coordination? Wouldn't the current method suffer in such case?\nDoes the paper assume finite state-action sets? If so, please say it explicitly.\nSince the environment has stochastic transitions, aren't more assumptions needed in order to allow gamma = 1 and still ensure existence of optimal policy?\nRegarding the rightmost term of (b) in (7), is it missing from (6) and (8)?\n\nComments that didn't influence the score:\nLast paragraph of page 2, \"to explore widely\" seems loose --> \"to enhance exploration\" might be more accurate.\nThe authors use the term causal diagram, but it seems to refer a Bayesian network, which indicates correlation rather than causality, is that right?\nThird line of the conclusions paragraph in Sec. 7, wouldn't be more accurate to say \"applying approximate policy iteration\"?", "extra_scores": {}, "extra_text": {}}, {"review_id": "U3lsfXmHsjx", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes regularizing the conventional MARL learning objective with a mutual information term to encourage more correlated behaviors among different agents. The contribution is clearly stated. However, the similarity to previous works is not sufficiently discussed, and the paper leaves out some important related works.\n\nThe paper maximizes the mutual information between agents' policies, given the current states. To allow policies to be conditional dependent, the authors assume there exists a dummy variable. The contributions of the paper about the lower bound of the mutual information (Sec. 4.2) and policy update (Sec. 4.3) are not significant. The lower bound of the mutual information has recently been extensively explored in multi-agent settings, used for encouraging role emergence, minimized communication, and exploration. The policy update and policy improvement guarantee can be easily obtained based on soft reinforcement learning literature. \n\n** Major concern: About related works. My major concern is not about the two contributions mentioned above. Instead, I think that the first and main contribution of this paper is a subset of previous works' contributions. EDTI [1] discusses how to maximize the mutual information between the \\emph{trajectories} of different agents. Their discussion already covers the correlation of policies of different agents at \\emph{a single timestep}. More importantly, the authors of that paper also point out that only optimizing mutual information between trajectories is not enough because the reward signal has to be considered for better policy learning. They even discuss the second-order influence between these mutual-information-based intrinsic rewards. The contribution of this paper seems to be the first part of [1]. Frans Oliehoek and other researchers also did lots of excellent works on this topic [2,3,4,5]. However, the discussion about these related works is absent from this paper.\n\nAdditionally, the difference between the proposed method and Jaques's paper (social influence) is not significant. The only difference is whether the action of other agents, $a_j$, is $a_j^{t}$ or $a_j^{t+1}$ when calculating the mutual information (In Jaques's paper, they prove that their formulation is equivalent to a mutual information formulation). I do not think the author's definition is an improvement of that of Jaques. At least, the authors should provide a more serious discussion about this point, perhaps providing a matrix game to show that different timesteps do indeed make a difference. The authors may argue that their experiments show that their method has better performance. The problem is that SSD used by Jaques is a more challenging task than those used in this paper. Moreover, social influence is sensitive to hyperparameter settings and needs fine-tuning to reach its full potential. I will change my mind regarding the experiments if the authors could provide SSD results (adopting their methods to tasks with discrete actions is not very difficult). Besides, Jaques et al. also discuss how to make condition dependency clear between agents and how to carry out influential communications, which are not discussed in this paper. \n\n[1] Wang, T., Wang, J., Wu, Y. and Zhang, C., 2019. Influence-based multi-agent exploration. ICLR 2020 spotlight\n\n[2] F. A. Oliehoek, S. Witwicki, and L. P. Kaelbling. Influence-based abstraction for multiagent systems. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, pages 1422–1428, July 2012\n* Also see a recent longer version: https://arxiv.org/abs/1907.09278\n\n[3] R. Becker, S. Zilberstein, V. Lesser, and C. V. Goldman. Transition-independent decentralized Markov decision processes. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 41–48, 2003.\n\n[4] Miguel Suau de Castro, Elena Congeduti, Rolf A.N. Starre, Aleksander Czechowski, and Frans A. Oliehoek. Influence-Based Abstraction in Deep Reinforcement Learning. In Proceedings of the AAMAS Workshop on Adaptive Learning Agents (ALA), May 2019.\n\n[5] Frans A. Oliehoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs, SpringerBriefs in Intelligent Systems, Springer, May 2016.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AX7MgmhWayl", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper can be seen as a modification of SAC, in a multiagent setup by adding the conditional entropy $H(\\pi_i|\\pi_j)$ as a second set of regularization on top of $H(\\pi_i)$. The overall idea and intuition appear to be interesting. \n\n\n1.The first question is whether the mutual information is informative enough. Mutual means two $a$, how about more, e.g., $H(a^i|a^j, a^k)$. When Eq.(2) is still our target, the underlying assumption is $a^j$ and $a^k$ would independently affect $a^i$. Is that true? And how the conditional distribution of $a^j$ on $a^k$ and vice versa would influence the objective?\n\n2.The motivation for inducing mutual information using latent variable is not well motivated. I can only find the first sentence in Section 4.1. \n\n3.Would the new regularization benefit other RL algorithms? Say, would the regularization combined with the baseline algorithms shown in the experiment be better compared with those without regularization?  \n\n\n4.The same latent variable $z$ is shared by all the policies. As shown in Fig. 1(b), would $z_{ij}$ make more sense? Essentially, would decentralized training be more reasonable?\n\n\n5.In the experiment, multi-agent soft actor-critic is missing, i.e., only keep $H(\\pi^j)$ in the regularization.\n\n6.How is $\\alpha$ selected?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4Zpzd6xPPkt", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n- This paper proposes a Maximum Mutual Information framework for cooperative MARL. Following the insight that mutual information of agents’ policies is the indicator of coordination, this paper proposes VM3-AC, an MA-AC algorithm that optimizes long-term reward as well as a variational lower bound of mutual information in the paradigm of CTDE. Experimental results show superiority of proposed algorithm in comparison with a few benchmark approaches.\n\n\nDetailed comments:\n- I appreciate the idea of utilizing mutual information (MI) of agents’ policies to facilitate the coordination of cooperative multiple agents. In my personal opinion, the MI of agents’ policy can be viewed as an indicator of coordination, although the coordination may not be a good/optimal one (the results of temperature parameter experiments also demonstrate this. I will mention this point later below). \n- My most concern is about the latent variable $z$. I agree that the latent variable $z$ should be an unobserved variable which reflects some information of coordination, e.g., such a variable can be implicitly induced during the learning process of policies of agents. In this paper, the latent variable $z$ in both learning and execution. Especially, the authors use a common and pre-determined sequence of $z$ is generated before execution for agents. In my opinion, this violates the CTDE paradigm which is claimed by the authors, since such a common sequence of $z$ is more like explicit signals of how to perform coordination (not truly decentralized). I am willing to hear the further understanding about the practical role of $z$ from the authors.\n- Moreover, it seems that the two of three environments, i.e., Predator-prey and Cooperative Navigation, are not partially observable (PO). It would be better to provide more evaluation under PO environments.\n\n\nQuestions:\n- Can the authors further discuss the reason to stop the gradient of term (a) in Equation 45, since such a mechanism results in a practical optimization objective different from variation approximate MI. \n- As in Algorithm 1, in decentralized execution phase, the latent $z$ is set to zero-vector, which conflicts the random sequence generated from the same random process as described in Section 5. Which one is the practical way and what is the difference in learning performance?\n- Below Equation 13, it says that $L = 1$ is used for MC expectation. Is $L = 1$ is sufficient for good performance? And how can larger values of $L$ influence the learning process?\n\nSuggestions:\n- As mentioned above, I view MI as an indicator of coordination, however, the coordination may not be preferred. Therefore, maximize the MI sometimes may trap the agents in a sub-optimal coordination (although the reward-objective may help them out to some degree). One potential evidence is, a larger temperature parameter (0.15) hampers the learning performance, as shown in Figure 5 (c)-(d). Besides, the stabilization induced by stop partial gradients of Equation 45 may also be explained by this point. Therefore, a more sophisticated mechanism for adaptive optimization of MI can be considered.\n- One possible concern (also possible future work) of proposed approach is the current modeling of MI is pair-wise modeling, of which the apparent drawback is the computational complexity can increase quadratically to the agent number. As in this paper, the experiments contain environments with up to 4 agents. A development towards more agents should be considered.\n\n\nMinors:\n- In Figure 5, last two sub-plots are both labeled as (c). Temperature parameters are denoted by \\alpha in legends but \\beta in paragraph.\n\n\nOverall, I think this paper takes a good attempt to study MI in cooperative MARL, proposing a reasonable algorithm with promising experimental results.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overview:
This paper introduces a maximum mutual information method for helping to coordinate RL agents without communication.

Discussion:
Some reviewers leaned towards accept, but I found the two reviewers recommending rejecting to be more convincing.

Recommendation:
This is an important research topic and I'm glad this paper is focusing on the problem. Hopefully the reviews will help improve a future version of this paper. I agree that this is a new way of using mutual information, but it seems more like a small improvement rather than a very significant step forward. 

In addition, I think the setting needs to be better motivated. This is a centralized training with decentralized execution (CTDE) setting, and this paper helps the agents coordinate. In CTDE, the agents work in the environment and then pool their information to train before deploying on the next episode. I don't understand why, e.g., in multiwalker, agents would not be able to communicate while walking, can communicate after they succeed or drop the object (the episode ends), and then cannot communicate once the next episode starts.
Decision: Reject