Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper can be seen as a modification of SAC, in a multiagent setup by adding the conditional entropy $H(\pi_i|\pi_j)$ as a second set of regularization on top of $H(\pi_i)$. The overall idea and intuition appear to be interesting. 


1.The first question is whether the mutual information is informative enough. Mutual means two $a$, how about more, e.g., $H(a^i|a^j, a^k)$. When Eq.(2) is still our target, the underlying assumption is $a^j$ and $a^k$ would independently affect $a^i$. Is that true? And how the conditional distribution of $a^j$ on $a^k$ and vice versa would influence the objective?

2.The motivation for inducing mutual information using latent variable is not well motivated. I can only find the first sentence in Section 4.1. 

3.Would the new regularization benefit other RL algorithms? Say, would the regularization combined with the baseline algorithms shown in the experiment be better compared with those without regularization?  


4.The same latent variable $z$ is shared by all the policies. As shown in Fig. 1(b), would $z_{ij}$ make more sense? Essentially, would decentralized training be more reasonable?


5.In the experiment, multi-agent soft actor-critic is missing, i.e., only keep $H(\pi^j)$ in the regularization.

6.How is $\alpha$ selected?


review_title: More clarificaitons and experiments on MMI regularization

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
The authors propose to include the mutual information between agents' simultaneous actions in the objective to encourage coordinated behaviour. To induce positive mutual information, the authors relax the assumption that the joint policy can be decomposed as the product of each agent's policy, independent of each other given the state, and they achieve so by introducing a latent variable that correlates agents behaviours. Since the mutual information is difficult to compute, the authors proposed to maximise a parametric lower bound. The algorithm is theoretically motivated as a policy iteration variation in its exact tabular form. But experiments are performed with neural network approximations on some environments. Numerical results show improvements over previous similar techniques.

Strong points:
The mutual information objective, the latent variable, the variational lower bound, and the algorithm are well motivated.
Paper is well written. 
Simulation results seem convincing.

Weak points:
The decentralised execution relying on having random generators with the exact same seed is not a robust solution.
The presentation as modified policy iteration is OK, but quite straightforward. The characterisation as a contraction is similar to many other works, maybe a citation would have helped.


Questions:
Is always coordination desirable? Could be some adversarial example of an environment where the optimal policy requires lack of coordination? Wouldn't the current method suffer in such case?
Does the paper assume finite state-action sets? If so, please say it explicitly.
Since the environment has stochastic transitions, aren't more assumptions needed in order to allow gamma = 1 and still ensure existence of optimal policy?
Regarding the rightmost term of (b) in (7), is it missing from (6) and (8)?

Comments that didn't influence the score:
Last paragraph of page 2, "to explore widely" seems loose --> "to enhance exploration" might be more accurate.
The authors use the term causal diagram, but it seems to refer a Bayesian network, which indicates correlation rather than causality, is that right?
Third line of the conclusions paragraph in Sec. 7, wouldn't be more accurate to say "applying approximate policy iteration"?
review_title: Proposed technique seems effective and it is nicely motivated and well written 

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
- This paper proposes a Maximum Mutual Information framework for cooperative MARL. Following the insight that mutual information of agents’ policies is the indicator of coordination, this paper proposes VM3-AC, an MA-AC algorithm that optimizes long-term reward as well as a variational lower bound of mutual information in the paradigm of CTDE. Experimental results show superiority of proposed algorithm in comparison with a few benchmark approaches.


Detailed comments:
- I appreciate the idea of utilizing mutual information (MI) of agents’ policies to facilitate the coordination of cooperative multiple agents. In my personal opinion, the MI of agents’ policy can be viewed as an indicator of coordination, although the coordination may not be a good/optimal one (the results of temperature parameter experiments also demonstrate this. I will mention this point later below). 
- My most concern is about the latent variable $z$. I agree that the latent variable $z$ should be an unobserved variable which reflects some information of coordination, e.g., such a variable can be implicitly induced during the learning process of policies of agents. In this paper, the latent variable $z$ in both learning and execution. Especially, the authors use a common and pre-determined sequence of $z$ is generated before execution for agents. In my opinion, this violates the CTDE paradigm which is claimed by the authors, since such a common sequence of $z$ is more like explicit signals of how to perform coordination (not truly decentralized). I am willing to hear the further understanding about the practical role of $z$ from the authors.
- Moreover, it seems that the two of three environments, i.e., Predator-prey and Cooperative Navigation, are not partially observable (PO). It would be better to provide more evaluation under PO environments.


Questions:
- Can the authors further discuss the reason to stop the gradient of term (a) in Equation 45, since such a mechanism results in a practical optimization objective different from variation approximate MI. 
- As in Algorithm 1, in decentralized execution phase, the latent $z$ is set to zero-vector, which conflicts the random sequence generated from the same random process as described in Section 5. Which one is the practical way and what is the difference in learning performance?
- Below Equation 13, it says that $L = 1$ is used for MC expectation. Is $L = 1$ is sufficient for good performance? And how can larger values of $L$ influence the learning process?

Suggestions:
- As mentioned above, I view MI as an indicator of coordination, however, the coordination may not be preferred. Therefore, maximize the MI sometimes may trap the agents in a sub-optimal coordination (although the reward-objective may help them out to some degree). One potential evidence is, a larger temperature parameter (0.15) hampers the learning performance, as shown in Figure 5 (c)-(d). Besides, the stabilization induced by stop partial gradients of Equation 45 may also be explained by this point. Therefore, a more sophisticated mechanism for adaptive optimization of MI can be considered.
- One possible concern (also possible future work) of proposed approach is the current modeling of MI is pair-wise modeling, of which the apparent drawback is the computational complexity can increase quadratically to the agent number. As in this paper, the experiments contain environments with up to 4 agents. A development towards more agents should be considered.


Minors:
- In Figure 5, last two sub-plots are both labeled as (c). Temperature parameters are denoted by \alpha in legends but \beta in paragraph.


Overall, I think this paper takes a good attempt to study MI in cooperative MARL, proposing a reasonable algorithm with promising experimental results.

review_title: An Interesting Mutual Information Based Learning Framework for Multiagent Cooperation

final_score: rating: 3: Clear rejection
review_content: This paper proposes regularizing the conventional MARL learning objective with a mutual information term to encourage more correlated behaviors among different agents. The contribution is clearly stated. However, the similarity to previous works is not sufficiently discussed, and the paper leaves out some important related works.

The paper maximizes the mutual information between agents' policies, given the current states. To allow policies to be conditional dependent, the authors assume there exists a dummy variable. The contributions of the paper about the lower bound of the mutual information (Sec. 4.2) and policy update (Sec. 4.3) are not significant. The lower bound of the mutual information has recently been extensively explored in multi-agent settings, used for encouraging role emergence, minimized communication, and exploration. The policy update and policy improvement guarantee can be easily obtained based on soft reinforcement learning literature. 

** Major concern: About related works. My major concern is not about the two contributions mentioned above. Instead, I think that the first and main contribution of this paper is a subset of previous works' contributions. EDTI [1] discusses how to maximize the mutual information between the \emph{trajectories} of different agents. Their discussion already covers the correlation of policies of different agents at \emph{a single timestep}. More importantly, the authors of that paper also point out that only optimizing mutual information between trajectories is not enough because the reward signal has to be considered for better policy learning. They even discuss the second-order influence between these mutual-information-based intrinsic rewards. The contribution of this paper seems to be the first part of [1]. Frans Oliehoek and other researchers also did lots of excellent works on this topic [2,3,4,5]. However, the discussion about these related works is absent from this paper.

Additionally, the difference between the proposed method and Jaques's paper (social influence) is not significant. The only difference is whether the action of other agents, $a_j$, is $a_j^{t}$ or $a_j^{t+1}$ when calculating the mutual information (In Jaques's paper, they prove that their formulation is equivalent to a mutual information formulation). I do not think the author's definition is an improvement of that of Jaques. At least, the authors should provide a more serious discussion about this point, perhaps providing a matrix game to show that different timesteps do indeed make a difference. The authors may argue that their experiments show that their method has better performance. The problem is that SSD used by Jaques is a more challenging task than those used in this paper. Moreover, social influence is sensitive to hyperparameter settings and needs fine-tuning to reach its full potential. I will change my mind regarding the experiments if the authors could provide SSD results (adopting their methods to tasks with discrete actions is not very difficult). Besides, Jaques et al. also discuss how to make condition dependency clear between agents and how to carry out influential communications, which are not discussed in this paper. 

[1] Wang, T., Wang, J., Wu, Y. and Zhang, C., 2019. Influence-based multi-agent exploration. ICLR 2020 spotlight

[2] F. A. Oliehoek, S. Witwicki, and L. P. Kaelbling. Influence-based abstraction for multiagent systems. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, pages 1422–1428, July 2012
* Also see a recent longer version: https://arxiv.org/abs/1907.09278

[3] R. Becker, S. Zilberstein, V. Lesser, and C. V. Goldman. Transition-independent decentralized Markov decision processes. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 41–48, 2003.

[4] Miguel Suau de Castro, Elena Congeduti, Rolf A.N. Starre, Aleksander Czechowski, and Frans A. Oliehoek. Influence-Based Abstraction in Deep Reinforcement Learning. In Proceedings of the AAMAS Workshop on Adaptive Learning Agents (ALA), May 2019.

[5] Frans A. Oliehoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs, SpringerBriefs in Intelligent Systems, Springer, May 2016.
review_title: The contribution of this paper largely overlaps with previous works.

Meta-review: Overview:
This paper introduces a maximum mutual information method for helping to coordinate RL agents without communication.

Discussion:
Some reviewers leaned towards accept, but I found the two reviewers recommending rejecting to be more convincing.

Recommendation:
This is an important research topic and I'm glad this paper is focusing on the problem. Hopefully the reviews will help improve a future version of this paper. I agree that this is a new way of using mutual information, but it seems more like a small improvement rather than a very significant step forward. 

In addition, I think the setting needs to be better motivated. This is a centralized training with decentralized execution (CTDE) setting, and this paper helps the agents coordinate. In CTDE, the agents work in the environment and then pool their information to train before deploying on the next episode. I don't understand why, e.g., in multiwalker, agents would not be able to communicate while walking, can communicate after they succeed or drop the object (the episode ends), and then cannot communicate once the next episode starts.
Decision: Reject