Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a method of learning a hierarchical communication graph for improving collaborative multi-agent reinforcement learning, particularly with large numbers of agents. The method is compared to a suitable range of baseline approaches across two complex environments. The initial results presented seem promising, but further work is needed to ensure the results are reproducible and repeatable.

To enable reproducibility, please include details of all hyperparameters used for all approaches in both domains. These should include justification of how the hyperparameters were tuned. Without understanding how these values were set I cannot support acceptance.

To ensure the results are repeatable, repeated runs of training should be completed and the variation in performance quantified in the results. These repeats may have already been performed as Table 3 and Figure 5 discuss average results, but if not they must be completed before the work can be published due to the known issues with high variance in performance that commonly occur in deep RL.

I would also argue against the justification of excluding ATOC from the StarCraft II experiments as its performance in MAgent with 25 agents is comparable to the other baseline methods that were tested. However, this is lower priority than the issues above provided there is no significant change in the relative performance of methods when the variance across multiple runs is documented in all existing experiments.

Minor Comments:
The following are suggestions for improvements if the paper is accepted or for future submissions. 

In Section 2, centralised critic methods are grouped as "communication-free" however I don't think this is the best term to explain this approach as each agent has to communicate both its observations and actions to a centralised node (e.g. COMA) or all other agents (e.g. MADDPG). I also think this section should include coverage of other methods of utilizing graph neural networks in multi-agent reinforcement learning - e.g. "Deep Multi-Agent Reinforcement Learning with Relevance Graphs." Malysheva et al. Deep RL Workshop @ NeurIPS 2018 and "Relational Forward Models for Multi-Agent Learning" Tacchetti et al. ICLR 2019.

In Section 3, the acronyms CBRP and HCOMM are used on page 4 before they are introduced in full on page 5 for CBRP and never for HCOMM. HCOMM is also not used in Figure 4 or in the text description of the method. I believe it is the module described in Section 3.2 but this should be made clearer.

Many claims in the paper are worded too strongly and should be revised. In Section 2, it is claimed that DQN "is one of the few RL algorithms applicable for large-scale MARL" - However, there are now many successful applications of deep RL to multi-agent systems (some of which are cited earlier in this same section) that use a variety of algorithms other than DQN. It is also claimed in this section that "DQN has excellent sample efficiency" despite the sample efficiency of deep RL being a known issue, open research question and a barrier to its widespread use in practice. 

In Section 3.2 the authors conclude "Overall, our LSC algorithm has advantage in the communication efficiency" despite in the same section noting two cases where ATOC has better efficiency (N_msg and N_b-r). I would suggest removing this sentence entirely as the paragraph above already contains a balanced account of the relative merits of each approach. 

On pages 8 and 9 the authors make references to guarantees and in the Appendix to proofs that are not supported by theory only empirical results. Without supporting theory these words should be avoided.

The writing is also often informal to the detriment of presenting important information clearly. Notably, on page 3 "due to explosive growing number of agents" and on page 4 "The overall task of the MARL problem can be solved by properly objective function modeling." The second of these, particularly the word "solved" is also related to the issue above of using the words guarantee and prove.

Finally, the paper would benefit from a thorough grammar check. I note the following issues as simple changes that can be made to improve the readability of the paper:
- In the abstract, the sentence "but also communication high-qualitative" does not parse. Perhaps this could be shortened to the brackets following? i.e. "is not only scalable but also learns efficiently."
- Page 2, "or employing the LSTM to" -> "or use the LSTM to"
- Page 2, "still hinder the" -> "still hinders the"
- Page 2, "the communication structure need be jointly" -> needs to be jointly
- Page 6, "policy module motioned below" -> discussed below
- Page 10, "the map size are is 1920x1200" -> the map size is 1920x1200
- Page 10, "We do not compare with ATOC because its poor performance in MAgent" -> because of its
- Page 10, "we will to improve" -> we will improve
- Page 10, "practical constrains" -> practical constraints
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Summary:
The paper proposes a method for improving the scalability of communication-based cooperative multi-agent reinforcement learning. While existing approaches assume a fixed underlying network topology over which agents communicate, in the proposed method, this network topology is dynamic (changes at each time step) and learnable (by assigning a weight to each node and "rewiring" nodes in a particular way based on these weights).

Authors highlight the importance of having a topology that is roughly similar to a collection of star-topologies. The center of stars (central nodes) further form a complete graph. They argue that such a topology can achieve global cooperation while reducing the number of messages exchanged as compared to the case where all agents can communicate with each other.

To learn a dynamically changing topology, the method assigns a weight (an integer between 0 and 4) to each agent based on its local observation. An existing method (CBRP) is then used to establish connections between agents based on weights assigned to them. 

A graph neural network (GNN) is used for computing the messages that are exchanged among agents. Communication uses the following 3 steps: (i) agents talk to the central agent(s) to which they are connected, (ii) central agents exchange information among themselves, and (iii) central agents transmit information to the agents that are connected to them. 

Each agent uses a deep Q-network - the parameters of this network are shared across agents. This Q-network receives rewards from the environment. Gradients flowing through Q-network are also used to update GNN. Since CBRP is non-differentiable, the parameters for network that computes weights for all the agents are also updated using a Q-network that gets the same reward from the environment as the first Q-network.

Experiments done on MAgent environment demonstrate that: (i) communication is useful and (ii) method scales well as number of agents increases. Additional qualitative studies have also been performed to understand the content of messages and the learned strategies. Authors have also experimented with the StarCraftII environment.


Comments:
The paper deals with an interesting problem, however, the presentation can be significantly improved as there are multiple grammatical mistakes in the manuscript. Unfortunately, the work does not position very well with the existing literature. The motivation and the impact of the contributions are not very clear. I would rate contributions as marginal. 

It is not clear what POSs* terms in Algorithm 1 mean.

It would be interesting to see which agents become central agents over time. As central agents form a complete graph, if there are many central agents then the approach will be inefficient.

Under the message visualization heading on p9, it is not clear how one decides whether a message was a "move" message or an "attack" message.


Questions to the Authors:

1. On p2, it is written that when concatenation or mean operation is used for aggregation, then inter-relationship between agents are not captured. What does this mean? Why does this problem not apply to GNN based solution which may also use mean for aggregation?

2. On p10, first line, it is written that all methods were made to react to the same initial state. How was this state chosen?

A few questions are also embedded in the comments above.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: 1. Summary

The authors learn structured communication patterns between multiple RL agents. Their framework uses a Structured Communication Network Module and Communication-based Policy Module. These use a hierarchical decomposition of the multi-agent system and a graph neural network that operates over the resulting abstract agent (groups). The authors evaluate on two environments, where this approach outperforms other ways to communication protocols.

2. Decision (accept or reject) with one or two key reasons for this choice.

Weak accept.

3. Supporting arguments

Scalable communication with many agents will require a (learned) trade-off between structural priors and learning representations of communicaton (protocols). This work seems like an interesting step in analyzing how to implement this. 

4. Additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.

5. Questions
review_title: Official Blind Review #1

Meta-review: The paper focuses on large-scale multi-agent reinforcement learning and proposes Learning Structured Communication (LSC) to deal issues of scale and learn sample efficiently. Reviewers are positive about the presented ideas, but note remaining limitations. In particular, the empirical validation does not lead to sufficiently novel insights, and additional analysis is needed to round out the paper.
Decision: Reject