Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary

The authors propose a new pooling layer, LaPool, for hierarchical graph representation learning (Ying et al., 2019) by clustering nodes around centroids that are selected based on "signal intensity variation". The signal intensity variation of node x is defined as sum_{y in HOP(x, h)} ||x - y||  where HOP(x, h) is the set of nodes reachable from x within h hops. Once top k maximizers are selected as centroids (k can be predetermined or dynamically chosen), a sparse cluster assignment distribution is computed for each node using sparsemax (Laha et al., 2018), and the affinity matrix and the node embeddings are coarsened as in Ying et al. (2019). The authors show that LaPool can improve performance in various graph-related tasks over baselines and generate interpretable clusters. 


Strengths

- Explicit centroid selection based on signal intensity variation seems like an intuitive idea worth investigating. 

- LaPool seems to be empirically effective, in particular outperforming Graph U-Net which is probably the most relevant baseline (also k-max pooling for hierarchical graph representation learning). But I'm not an expert on the considered tasks, so I cannot judge how significant these results are. 


Weaknesses: 

- The paper has issues with clarity. There seem to be many sloppy notations as well as unclear (possibly wrong) arguments.

1. Isn't equation (1) true for unnormalized graph Laplacian (D - A), not normalized? 

2. In the proof of Theorem 1, why is it that C C' X = X (i.e., X is in range(C))? This is a crucial step that I'm not sure why is true. I might be missing something, but it's not clear to me in the current version. 

3. Sloppy notations. What do "A^h", "||L X||_{R^d}", "top_k(V|L S)" mean in equation (4)? I can infer their meaning, but do I have to? This also interferes with my other confusions: back to question #1, is it true that S = ||L X||_{R^d}? 

4. Is the number of clusters fixed or dynamic for these experiments? Figure 2 seems to be dynamic based on the main text, but I cannot tell if it's the case for other experiments. Based on the provided code, it seems the default number of clusters is always 10% of the number of nodes.

- The paper doesn't do a good job of contextualizing itself. Graph U-Net seems to be the most relevant previous work, but there's no discussion of how this work relates and why it's better. For instance, is it the case that Graph U-Net is not interpretable because there's no explicit clustering? It'd be much clearer to spell out such differences. 

- It's unclear why certain design choices are made and why they're better. For instance, is swapping entropy minimization with sparsemax necessarily better? I understand not every design decision should be (or needs to be) justified, but it's helpful if it is to understand what helps and what doesn't.


Summary

The core idea of the paper, clustering nodes based on signal intensity for hierarchical graph representation learning, is interesting and seems to be useful in practice, but the paper has issues with clarity and the rest of the framework is a bit limited in novelty (DiffPool + sparsemax + GIN).
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper introduces a new pooling approach "Laplacian pooling" for graph neural networks, which the authors claim is able to better preserve information about the local structure, and to provide interpretability.  Namely, the pooling approach is based on finding centroids (nodes having high signal variation compared to their neighbors, via graph-Laplacian) and assigning other nodes to be "followers" based on a soft-attention mechanism. The authors add these new pooling layers to existing GNN architectures and show improved performance on problems of classification and generative modeling of molecular graphs. The paper also extends CNN interpretability techniques (integrated gradients) to GNNs.

I am borderline on the paper -- I'll give a weak accept rating for now. The proposed Laplacian-pooling ideas could be interesting, and the results encouraging -- but I found the mathematical motivation to be not very convincing, and has various (mostly correctable issues). The paper can be viewed as more of an engineering effort, which attempts to find practical tricks aimed at modeling molecular structures. What I like about the paper is that the authors make an earnest attempt to model the domain (biochemistry) -- for example they realize that a graph formalism for molecular structures is rather simplistic, and misses many important details -- such as different types of bonds (which require different types of edges). The authors also realize that typical neighborhood smoothing (diffusion) that makes sense for say spatial or social network graphs may not make sense for molecular graphs, where specific substructures (e.g. presence of a benzene ring) may be highly indicative for some classification tasks. The paper also contains a collection of interesting practical heuristics and observations (mainly in appendix) to help train GNN models for classification and generative modeling -- which researchers in the field may find valuable. I am not sure if the idea of coarsening (via hierarchical pooling) can be meaningfully applied to a wide-variety of natural molecules - but it does seem to make sense for some organic molecules -- e.g. protein chains. 

Detailed comments: 
1. The paper makes an analogy between band-pass filtering and the proposed approach. In my opinion the analogy is rather weak -- while it may carry over to spatial graphs (e.g. grid graphs) but may not apply to more complex graphs -- e.g. graphs where each node is at most a few hops away from any other node. It's not clear in what sense (3) corresponds to high-pass filtering. Can you show that it's somehow related to filtering-out the large (low-pass) eigenvalues of the graph-laplacian?

2. There is a typo in equation (1) -- equality of the quadratic form f'L f requires an unnormalized definition of the graph laplacian D - A, instead of I - D^{-1} A. 

3. Notation in equation (3) -- is unclear and undefined.  What is ||L X||_{R^d} ? is that a norm (giving a scalar), or concatenation?  What is Top_k(V | L*S) -- the readers have to guess. You may be relying on notation from existing papers -- but still need to set up notation to be self-consistent. Subtle comment: "signal intensity variation" sounds like the variation of signal intensity -- e.g. something like difference of signal norms. Perhaps intensity of signal variation is a better term.  Sparsemax is also undefined. What is A^h -- the h-hop adjacency matrix? 

4. I do not understand what do you mean by "information preservation" after pooling -- and I did not understand the importance of the "structure-aware feature content" definition, and the value of theorem 1.  Matrix C in the derivation is undefined. 

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper proposes a graph down-sampling component named LaPool. It uses the graph signal to dynamically select some "important" centroids and learn a sparse assignment matrix for clustering the remaining nodes.

This paper should be rejected due to the following reasons.

1. Strange new results compared with its previous version

The paper has a previous arxiv version. While the method does not change, the performance in this current submission has dramatic change compared with previous version: the proposed model seems much improved, while some important baselines (that outperform the proposed model) only have 50% of it previous performance in this submission.

For example, for Table 1 in the current version, the proposed model reached almost perfect scores in both F1 and AUC. However in previous version for the same setting and experiment (in Table 2 and 3 of its arxiv version), the performance are much lower especially the Structural alert prediction results.  Their previous results, which show the model does not perform better than DiffPool .

For the structural alert prediction results on DiffPool, below I copied and pasted the results from the arxiv version
                                  

Table 3: Structural alert prediction results 
------------------------------------------------------------------------------------------------------------
                                             Tox21                   |               ChEMBL
                       F1-macro F1-micro ROC-AUC F1-macro F1-micro ROC-AUC
------------------------------------------------------------------------------------------------------------
GIN                     78.9        68.3             72.6           93.6         76.7          59.2
DiffPool              79.2        68.0             75.6           94.5        83.3          59.3
Graph U-net      71.1        47.6             67.9           92.9        68.1          59.3
LaPooldistance 80.6        74.2             73.5           95.2        81.3          59.5
LaPoolunreg     81.3        72.8             74.1           94.1        75.8          58.9
LaPool3hop       79.1        71.6             74.8           93.8        75.0          59.1
------------------------------------------------------------------------------------------------------------
One example: 
(1) in the ICLR version, The F1 score for DiffPool is only 48.638 ± 9.916 on ChEMBL data (about 50% of its previous level) but F1 for the proposed method is improved. Why is that? 
(2) same as baseline  Graph U-net , the ICLR version reports F1 37.585 ± 2.978, why is that?
(3) same as GIN, in the ICLR version, F1 is only 31.759 ± 3.728, less than 50% of its previous level in arxiv version.

The author needs to justify this dramatic change.

2. Although LaPool can dynamically select centroids, for a dense graph such as a complete graph, only one centroid will remain there since there is only one node which has larger signal variation than all its neighbors, as shown in Eq. 4. This consequently hurts the model performance on more dense graphs. That may be why in Table 2, on dense data "DD", LaPool performs much worse than baseline DiffPool. Also on another dense data "FRANKENSTEIN", LaPool does not performs significantly better than DiffPool.
 

3, the evaluation of interpretability is not convincing. This paper considers  "interpretability as the degree to which a human (in this context, a medicinal chemist) can understand the cause of the model’s decision". Therefore, the conclusion that this model is more interpretable is based on only one person's subjective judgement.  Even so, from the scores the model does not outperform baseline "GIN" that much.

review_title: Official Blind Review #1

Meta-review: The paper introduces a new pooling approach "Laplacian pooling" for graph neural networks and applies this to molecular graphs. While the paper has been substantially improved from its original form, there are still various concerns regarding performance and interpretability that remain unanswered. In its current form the paper is not ready for acceptance to ICLR-2020.
Decision: Reject