Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper tackles the collaborative multi-agent RL problem as the problem of finding almost-decentralized value functions, where the loss tries to minimize communications between the agents. The core idea is around maximizing the mutual information between each massage and the existing knowledge of its receiver. This way, redundant messages are naturally removed. The authors then assert an entropy regularization to (almost) prevent the agents from *cheating*. The paper is in general well-written and motivated. There are however certain issues that should be addressed. [The second one is my main issue.]  

1. The term *message* has been used repeatedly but not defined. You should define precisely what you mean by a message in the background section. In particular, it is not clear from the text what a message looks like mathematically. You may also consider giving intuitive examples of how different ways of designing a message alters the behaviour in a given context before start talking about how to optimise them. 

2. Section 3.1 needs a revisit. I assume by "optimal action policy of agent j" you mean "optimal policy of agent j". Formally, an optimal policy is greedy to the optimal value function and is deterministic unless there exist multiple actions with same optimal value at a given state. Therefore, the mutual information is not mathematically well-defined since most of the time the optimal policy is deterministic and does not induce a probability distribution. 

2.1 What is the optimal value function of an agent? The agents are not optimizing their own value function, so even if your complex model converges, it does not imply optimality of agent-level value functions (and they shouldn’t be locally optimal). If by that you mean the agent-level value functions *after* convergence of $Q_tot$, then you need to clarify it to avoid the confusion. Even so, it is still not clear how you conceive the agent-level policies from these value functions. 

2.2. Minor: As for the notation, I found $A_j$ quite strange for a policy; you may want to consider something like $\pi_j$. 

3. If $f_m$ is learnt, then message encoding is not stationary at least in the training time. It may make the training potentially become unstable. Specifically, there is no condition to assure stability of your model. Suggestion: your entropy regularization might be sufficient to induce stability. More discussion on this (or a simple experiment to show if an instability exists and will be alleviated with regularization) would be quite helpful.  

4. If I understand it correctly [and it wasn't clear from the text], the encoder $f_m$ is conditioned on local history, which induces that all the agents must share same history shape (tensor-wise), which in turn means that all the local agents has to have same local state shape (computationally, they should for example have same output/internal-state shape in their neural networks). This sounds like a limitation, and it may only be OK in domains where agents are homogenous. 

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: Authors propose a new method for multi-agent reinforcement learning by using nearly decomposable value functions. The main idea is to have local (agent specific) value function and one global value function (for all agents). They also try minimizing the required communication need for multi-agent setup. To this end they deploy variational inference tools and support their method with an experimental study.

I found the paper interesting and empirical evaluation is good. However, my knowledge in the field is quite limited.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The authors propose a framework for combining value function factorization and communication learning in a multi-agent setting by introducing two regularizers, one for maximizing mutual information between decentralized Q functions and communication messages and the other for minimizing the entropy of messages between agents. The authors also discuss a method for dropping non-informative messages. They illustrate their approach on sensor and hallway tasks and evaluate their method on the decentralized StarCraft II benchmark. The paper addresses an interesting problem, and the authors show that their approach gives good performance compared to alternative approaches even when a large percentage of communication is cut off between the agents.

Questions/Comments:
- Implementation details (e.g., hyper-parameters, model size) are missing from the paper.
- The results are average over only 3 seeds, is this enough to compare different algorithms?
- How should beta should be determined? 
- The authors present results when 80% of messages are cut off. What is the performance of the model when all communication is cut off for comparison?
- How does the approach work in competitive environments?
- The experimental results section is not well organized. The authors mention five question on page 6, but it is not very clear with examples/set of experiments address which question.
- There are many spelling/grammar errors in the paper.
review_title: Official Blind Review #2

Meta-review: The paper extends recent value function factorization methods for the case where limited agent communication is allowed. The work is interesting and well motivated. The reviewers brought up a number of mostly minor issues, such as unclear terms and missing implementation details. As far as I can see, the reviewers have addressed these issues successfully in their updated version. Hence, my recommendation is accept.
Decision: Accept (Poster)