Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper addresses the problem of coordination in the multi-agent reinforcement learning setting. It proposes the value function factorization similar to independent Q-learning conditioning on the output of the graph convolutional neural network, where the graph topology is based on the agents’ nearest neighbours. The paper is interesting and has some great ideas, for example, KL term to ensure temporal cooperation consistency. However, the paper has its drawbacks and I feel obliged to point them out below. I vote for the weak acceptance of this paper.

One of the main drawbacks of the paper is that it is extremely hard to grasp. Even the Abstract and Introduction are hard to understand without having a pass over the whole paper. The authors often use vague terms such as 'highly dynamic environments' or 'dynamics of the graph' which make it hard to understand what they mean. The paper would benefit from a more precise language. Some of the important notions of the paper are used before they are introduced, which make the general picture very hard to understand and to relate the work to the existing research.

'Related Work' section seems to be missing some recent work applying graph neural networks to multi-agent learning settings:
• Malysheva, Aleksandra, Tegg Taekyong Sung, Chae-Bong Sohn, Daniel Kudenko, and Aleksei Shpilman. "Deep Multi-Agent Reinforcement Learning with Relevance Graphs." arXiv preprint arXiv:1811.12557 (2018).
• Agarwal, Akshat, Sumit Kumar, and Katia Sycara. "Learning Transferable Cooperative Behavior in Multi-Agent Teams." arXiv preprint arXiv:1906.01202 (2019).

My questions to the authors:

• In section 3 you mention 'a set of neighbours ..., which is determined by distance or other metrics'. Can you elaborate on that? What are these metrics in your case?
• Just before the Section 3.1, you say 'In addition, in many multi-agent environments, it may be costly and less helpful to take all other agents into consideration.' Have you run any experiments on that? In the appendix, you show, that making the neighbourhood smaller negatively affects the performance, but what if you make it bigger? Ideally, I would like to see an extended version of Figure 8 and 9 in the main part of the paper since they are very interesting and important for the claims the paper makes.
• At the end of Section 3.1, you mention the soft update of the target network. Later, in 3.3, you say that that you do not use the target network. Can you elaborate more on that?
• In Equation 4, is it a separate KL for each of the attention heads? If yes, this is not clear from the formula.
• It will be useful to see the ablation experiments for all of the testbeds, not only for Battle.
• Why do you think the DQN performance drops in the second half of the training in Figure 4 for all of the runs?
• Have you tried summation instead of the mean aggregation step?

I will put comments for particular parts of the paper below.

ABSTRACT

>>> ...environments are highly dynamic

What do you mean precisely here?

>>> ...graph convolution adapts to the dynamics of the underlying graph of the multi-agent environment

What is the 'dynamics of the underlying graph'? What is the 'graph of the multi-agent environment'?

>>> 'coordination is further boosted'

Not sure that 'boosted' is the right word here.

INTRODUCTION

>>> '...where mutual interplay between humans is abstracted by their relations'

Not sure what it means.

>>> we consider the underlying graph of agents...

The agent graph has not been introduced yet.

>>> DGN shares weights among all agent(s) making it easy to scale

What do you mean precisely by 'easy to scale'? Can you support this claim?

>>> We empirically show the learning effectiveness of DGN in jungle

Needs a reference to the testbed.

>>>  ... interplay between agents and abstract relation representation

What is 'abstract relation representation?

>>> We consider partially observable environments.

What do you mean precisely by that? What is the MDP formalism most suitable for your problem statement? What is objective under your formalism?

>>> However, more convolutional layers will not increase the local region of node i.

What do you mean by that?

>>> As the number and position of agents vary over time, the underlying graph continuously changes, which brings difficulties to graph convolution.

What kind of difficulties?

>>> As the action of agent can change the graph at next timestep which makes it hard to learn Q function.

Why does it make it hard?

>>> DGN can also be seen as a factorization of a centralized policy that outputs actions for all the agents to optimize the average expected return.

It would be useful for the reader to compare your approach with all the others type of the value function factorization. To me, your approach looks like a more sophisticated version of independent Q-learning, is that true?

Minor comments:

* In 3.2 it would be very helpful to put the dimensions for all of the variables for easier understanding.
* The brackets in the equation 3 are not very clear (what are you concatenating across?)
* In section 4, when describing an environment you say 'local observation that contains a square view with 11x11 grids'. What is the total size of the environment?
* The performance plots for Battle include ablations before the ablation subsection is introduced. This is a bit confusing.
* All figures/tables captions should be more detailed and descriptive.
* ‘However, causal influence is not directly related to the reward of environment.’ Should be ‘of the environment’.


review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper proposes an algorithm allowing "cooperation" between agents in multi-agent reinforcement learning, modeling agents as nodes in a graph. Each agent having only a partial view of the environment, the proposed algorithm uses multi-head attention as a (graph) convolution kernel but otherwise remains similar to the DQN algorithm. Performance is evaluated on three tasks using the MAgent framework.

The paper is reasonably well motivated, grounded and written. It addresses an interesting question: how to make agents cooperate in an efficient way? It does so by combining ideas from two lines of work, bringing incremental novelty. 

My main concern relates to the experiments. It seems that ATOC and TarMAC would be the best baselines to compare against for a fair evaluation of the algorithm. Could they be added?

One question for the authors: at the beginning of Section 3, it is stated that "it may be costly and less helpful to take all other agents into consideration". It seems counter intuitive that DGN with several convolutional layers (to have a large receptive field) would be less costly than directly receiving global information? And isn't, in a sense, DGN also making use of global information when it has a large enough receptive field, even if indirectly? In this case, would it also make sense to more thoroughly compare DGN with RFM or other global state algorithms? Can this be clarified?

Finally, readability is somewhat hindered by several small issues:
- Acronyms used in the paper should really be introduced, at least when they are first used. DGN is never introduced, DGN-R/DGN-M are introduced several paragraphs after being first mentioned and BL needs some guessing.
- Re-citing the same paper several times when mentioned in different sections is good practice. I found myself going over and over back to the related work section to find references and acronyms.

Some typos:
Page 1: among all agent -> among all agents
Page 3: of S -> of size S
Page 4: weighed -> weighted, concate -> concatenate
Page 5: respecitvely -> respectively
Page 6: regularation

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper introduces Graph Convolutional Reinforcement Learning (referred to as DGN). DGN is a Deep Q-Learning (DQN) agent structured as a graph neural network / graph convolutional network with multi-head dot product attention as a message aggregation function. Graphs are obtained based on spatial neighborhoods (e.g. k nearest neighbors) or based on network structure in the domain. DGN considers a multi-agent setting with a centralized learning algorithm and shared parameters across all (controlled) agents, but individually allocated reward. Further, the paper considers environments where other non-learning agents are present which follow a pre-trained, stationary policy. In addition to the attention-based multi-agent architecture, the paper introduces a regularizer on attention weights similar to the use of target networks in DQN, to stabilize training. Results demonstrate that the proposed model architecture outperforms related earlier agent architectures that do not use attention or use a fully-connected graph.

Overall, this paper addresses an interesting problem, introduces a novel combination of well-established architecture/agent building blocks and introduces a novel regularizer. The novelty and significance of the contributions, however is limited, as many recent works have explored using graph-structured representations and attention in multi-agent domains (e.g. VAIN: Hoshen (NeurIPS 2017), Zambaldi et al. (ICLR 2019), Tacchetti et al. (ICLR 2019)). The combination of these blocks and the considered problem setting is novel, but otherwise incremental. Nonetheless, the results are interesting, the overall architecture is simple (which I consider to be a good sign), and the attention regularizer is novel, hence I would rate this paper as relevant to the ICLR audience.

My main concern with this paper is clarity of writing: I have the feeling that important details are missing and some modeling decisions and formulas are difficult to understand. For example, I found section 3.3 difficult to read. The following sentences/statements need revision or further explanation:
* “Intuitively, if the relation representation produced by the relation kernel of upper layer truly captures the abstract relation between surrounding agents and itself, such relation representation should be stable/consistent” (Please clarify)
* “We use the attention weight distribution in the next state as the target for the current attention weight distribution” (What is the reasoning behind this? Would an exponential moving average of attention logits/weights work as well?)
* “While RNN/LSTM forces consistent action, regardless of cooperation” (unclear)
* “Since we only focus on the self-consistent of the relation representation based on the current feature extraction network we apply current network to the next state to produce the new relation representation instead of the target network as in deep Q learning” (unclear)
* The KL term in Eq. 4 is odd: z_i is defined as G^K and vice versa, neither of them appear to be distributions. I suppose one of the two arguments of the KL term should be the attention distribution for the current time step and the other argument for the next time step (if I understood the motivation in the earlier paragraph correctly), but this is not evident from Eq. 4.
* KL is not symmetric -- what motivates the particular ordering in your case? Did you consider symmetric divergences such as Jensen-Shannon divergence (JSD)?

I also wonder about the necessity of assembling adjacency matrices per node to create an intermediate ordered representation of the neighborhood on which, afterwards, an order-invariant operation such as mean pooling or self-attentive pooling is applied. Wouldn't it be more efficient to implement these operations directly using sparse scatter/gather operations as most recent GNN frameworks implement these techniques (e.g. PyTorch Geometric or DeepMind's graph_nets library)?

Further, important experimental details are missing, e.g., how observations / node features are represented / obtained from the environment and preprocessed. Do you encode position (continuous/discrete) and normalize in some way? It should further be mentioned that some of the baselines are trained with a different training algorithm and do not only differ in agent architecture (e.g. CommNet) — what is the effect of this?

Experimentally, the results seem sound, but the variance in the results is suprisingly low (see e.g. Figure 7 DQN) — did you change the random seed between runs (both environment seed and the seed for initializing the agent weights)?

Overall, this paper is interesting but needs revision in terms of clarity. Novelty is incremental, but if the paper would otherwise be very well written, I think it could qualify for acceptance. In its current state, I recommend a weak reject.


--- UPDATE AFTER REVISION ---
The clarity in the revised manuscript is significantly improved and I feel confident in recommending acceptance of the paper. 
review_title: Official Blind Review #3

Meta-review: The work proposes a graph convolutional network based approach to multi-agent reinforcement learning. This approach is designed to be able to adaptively capture changing interactions between agents. Initial reviews highlighted several limitations but these were largely addressed by the authors. The resulting paper makes a valuable contribution by proposing a well-motivated approach, and by conducting extensive empirical validation and analysis that result in novel insights. I encourage the authors to take on board any remaining reviewer suggestions as they prepare the camera ready version of the paper.
Decision: Accept (Poster)