Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: # Summary
This paper proposes a pairwise communication between agents using a shared neural network. The idea is to define the joint action-value function as the sum of individual agent's values + pair-wise payoff between agents, which is based on the prior work [Castellini et al.]. In particular, this paper proposes to share the parameters of the pairwise payoff function to improve efficiency. The result on a grid-world domain shows that the proposed method performs better than baselines that either do not have pairwise communication (VDN) or learn fully joint action value function (QTRAN). 

# Originality
The main novelty seems to be coming from the idea of parameter sharing between pairwise payoffs, but the overall architecture seems to be the same as [Castellini et al.]. 

# Quality
- This paper is missing an important baseline which is DCG without parameter sharing. Given that parameter sharing is the main new component from [Castellini et al.], it would be important to show the benefit of parameter sharing.
- Although the result against several baselines looks good, it would be much more convincing to show some qualitative analysis of the proposed method. For example, showing that the learned payoff captures reasonable and intuitive knowledge would strengthen the paper.

# Clarity
- The paper is well-written, and the figures are very clear. 
- It would be better to show a figure that illustrates the domain and task. 

# Significance
- Although the paper presents a new idea very well, the overall idea seems a bit incremental. This paper overall looks like a straightforward extension of [Castellini et al.] by adding parameter sharing and evaluating it on a more complex domain. In addition, showing more in-depth analysis would make the paper stronger. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: 1. Summary

Teh authors propose to learn value functions that are a sum of utility (single-agent) and payoff (2-agent) components. The weights between all function are shared (common RNN). This stands in contrast to VDNs (only uses utility functions or centralized value functions. The authors evaluate on predator-prey, where they argue that e.g. VDN fails to learn.

(Note that other work has looked at reward shaping to learn *decentralized* agents for that problem.)

1. Decision (accept or reject) with one or two key reasons for this choice.

Weak reject.

- The comparison between different topologies is nice, but implies that the structure of the graph has to be fixed manually. This seems to be a severe and unscalable constraint. It would be better if the authors would propose and evaluate a method to determine / learn what the right graph structure should be.
- Weight sharing between the various agent components makes the problem closer to a single-agent problem. What happens if the agents are decentralized and the (shared-weight) pairwise functions are separate?
- Authors only evaluate on a predator-prey problem.

4. Supporting arguments

N/A

5. Additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.

6. Questions
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper introduces the deep coordination graph for collaborative multi-agent reinforcement learning aimed to solve predator-prey tasks by preventing relative overgeneralization during the exploration of agents. 

In general, this paper gives a detailed and comprehensible depiction of the Introduction, Related work and Background section. However, I have two concerns about their method and experiments. 

The presentation of the “Method” section is not clear enough to evaluate the contribution of this paper. Specifically: 
(1). In the "Method" section, your method incorporates three ideas: 1. restricting the payoffs; 2. sharing parameters; 3. allowing generalization; 
In my understanding, both idea 1 and idea 2 come from VDN [4]. Idea 3 is not implemented in this work. Then, what is your contribution?

(2). According to the announcement, the key benefit for your work is "prevent relative overgeneralization during exploration of agents". It is hard to access if the proposed method can prevent overgeneralization. Do you have some theoretical or empirical justifications?

(3). I think it would be helpful if you could give a description according to your algorithm in the appendix. It would be better to draw a diagram to show what is your model.

I think the experiments are too simple and not convincing. 
(1). I notice recent multi-agent reinforcement papers ([1], [2] closely related to your work) evaluate their work on the challenging set of StarCraft II micromanagement tasks and achieved the evident result. 
 
(2). In fig.2, why did you only compare your model with VDN [4]?

(3). In fig.3, why does the return value of QTRAN first decrease and then increase? From [3], it seems their return curve continuously grows. 

(4). In fig.4, why QTRAN fail to this task?

[1]. Rashid, Tabish, et al. "QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning." arXiv preprint arXiv:1803.11485 (2018).
[2]. Wang, Tonghan, et al. "Learning Nearly Decomposable Value Functions Via Communication Minimization." arXiv preprint arXiv:1910.05366 (2019).
[3]. Son, Kyunghwan, et al. "QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning." arXiv preprint arXiv:1905.05408 (2019).
[4]. Sunehag, Peter, et al. "Value-decomposition networks for cooperative multi-agent learning." arXiv preprint arXiv:1706.05296 (2017).
review_title: Official Blind Review #1

Meta-review: This work extends previous work (Castellini et al) with parameter sharing and  low-rank approximations, for pairwise communication between agents. 
However the work as presented here is still considered too incremental, in particular when compared to Castellini et al.
The advances such as parameter sharing and low-rank approximation are good but not enough of a contribution. Authors' efforts to address this concern did not change reviewers' judgment.
Therefore, we recommend rejection.
Decision: Reject