Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a memory layer for Graph Neural Networks (GNNs) and two deep models for hierarchical graph representation learning. The proposed memory layer models the memory as a multi-head array of keys with a soft clustering mechanism and applies a convolution operator over the heads. The proposed models are experimentally evaluated on seven graph classification and regression tasks. 

Generally, the paper is technically justified. The proposed technique is well motivated and properly presented. A novel clustering-convolution mechanism is proposed  for memory augmentation and graph pooling. However, there are still some rebuttal requests. 1- Some details are insufficient. For the multi-head mechanism, it is not stated clearly whether for each head an independent query is computed or a shared query is used for all heads.  
2- Additionally, a related work published in NIPS 2016 should be cited and discussed. 
Jack Rae et al. Scaling memory-augmented neural networks with sparse reads and writes.  

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The authors introduce a method for adding memory layers to a graph neural network which can be used for representation learning and pooling. Two variants, the MemGNN and GMN are proposed which use the memory layer. The authors evaluate their models over 7 datasets that cover classification and regression tasks. They obtain SOTA on 6/7 datasets; perform ablation analysis and introspect the clusters for chemical significance.

Overall this paper is well written and easy to read. The motivation, equations and illustrations are clear and helpful. The model is technically novel, building up from existing approaches in a progressive way. Given the generality of the approach, the impact is also likely to be high.

In order to bolster their results the authors may run their approach on a few other datasets in Wu et. al. 2018. 

Minor issues:
 - Provide error bars for the tables
 - Sec 4.2 typo : “datastes” 

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper proposes to use memory network (self-attention) to summarize information from graphs. The proposed approach "soft clusters" the node embeddings into coarser representation and eventually fed to a final MLP for classification. The empirical results on some standard datasets show promising gains of the algorithm.

The proposed approach stacks a few layers of self-attention on top of either some features of the nodes (including projected edge connections, parametrized by W_0) or the node embeddings of some form of graph neural network. And this stacking seems to be simple combination of existing approaches, without fully integrating them. In fact, the training process is also separated: "task-specific loss are back-propagated batch-wise while the gradients of the unsupervised loss are applied
epoch-wise". It makes me wonder whether we can just separate it into two stages, i.e., first learn a node embedding using graph neural network, then learn this self attention transformation. Due to the above issues, I feel the novelty of the approach is limited and incremental.

Another issue with the paper is that the notations seem to be messed up and some concepts are not explained clearly. For example, the C_{i,j} soft assignment matrix is normalized row-wise, then Eqn (3) seems very suspicious, because it averages the queries using weights along the other direction, thus not normalized. Also, the dimension of the MLP weights do not align well with inputs, for instance in Eqn (4), it should be written as V^(l) W.

There are more questions that are not clearly specified in the current manuscript. For example, where does the keys K come from? From the text description, it seems to be cluster results, and do you do the clustering on every gradient update? Or are they learned from scratch? The distribution P defined in Eqn (11) also seems to be difficult to optimize since it depends on C_{i,j} and is connected to different entries. Is simple SGD sufficient to optimize over P? Moreover, in the experiment section, it is unknown how many layers of self-attention is applied and what are the important parameters. For better comparison, the experiment section should include some estimate of parameter size as well.

The experiment results seem interesting since the approach indeed achieves good performance across many datasets. Also the visualized keys are interesting as well because it captures some meaningful patterns from the data.

There is some related work that you should cite:
Hanjun Dai, Bo Dai and Le Song. Discriminateive Embeddings of Latent Variable Models for Structured Data. International Conference on Machine Learning (ICML) 2016.


============================================================
Based on the authors' reply and other reviews, I have changed my rating to "Weak accept".

The authors' reply has clarified the important detail of how the key matrix is learned and now it is clear that the algorithm is not just two-stage separate learning. In light of this clarification, I think the proposed algorithm is novel enough and the jointly training mechanism is also beneficial for the state-of-the-arts results reported in the experiments.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper presents "memory layer" to simultaneously do graph representation learning and pooling in a hierarchical way. It shares the same spirit with the previous models (DiffPool and Mincut pooling) which cluster nodes and learn representation of the coarsened graph. In DiffPool, Graph convolutional Neural Networks (GCNs) with 2-6 iterations of “Message passing” is used to learn node embedding, followed by graph pooling. By contrast, the proposed model circumvent the inefficiency of using message passing by using their proposed memory layer.

Pros:
* The paper is generally written well, but some important details are missing.
* Clear visualization of the results.
* Useful ablation study.

Cons:
* Missing information, which can be critical for the success of the model:
(a) the estimation of the keys, and 
(b) how does the convolutional layer in Eq (2) work, given that the input for it is concatenation of matrices, which has no spatial structure?
* Experiments on graph classification lack diversity, where CoLLAB is the only non-chemical dataset in the experiment. 
* The paper argues that interactive message passing is not efficient. But do you have any explanation on why MemGNN with message passing in initial embedding learning performs better than GMN without message passing in D&D dataset?
* I have some reservation for calling something "memory", which is meant to store information for later processing. For this work, the network is a feed-forward architecture for processing graphs, where the middle layers (the queries) are matrices, which can be studied on their own right (e.g., see [1]).

At this point, the ideas for graph representation are plentiful, but there have not been a coherent story on how and why new architectures should work better than previous ones. This paper can be made stronger by offering insights along this line.

[1] Do, K., Tran, T., & Venkatesh, S. (2017). Learning deep matrix representations. arXiv preprint arXiv:1703.01454.



review_title: Official Blind Review #3

Meta-review: Four reviewers have assessed this paper and they have scored it as 6/6/6/6 after rebuttal. Nonetheless, the reviewers have raised a number of criticisms and the authors are encouraged to resolve them for the camera-ready submission.
Decision: Accept (Poster)