Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper presented a new pooling method for learning graph-level embeddings. The key idea is to use the initial node attributes to compute all-pair attention scores for each node pair and then use these attention scores to formulate a new graph adjacency matrix beyond the original raw graph adjacency matrix. As the result, each node can average these attention score edges to compute the overall importance. Based on these scores, the method chooses top-k nodes to perform graph coarsening operation. In addition, a graph connectivity term is proposed to address the problems of isolated nodes. Experiments are performed to validate the effectiveness of the proposed method. 

Although I found this paper is generally well written and well motivated, there are several concerns about the novelty of paper, computational expenses, and the experimental results as listed below:

1) The idea of using attention score is simple yet seems effective. But using attention score to identify the importance of nodes has been first presented in GAT [Velicˇkovic ́ et al., 2019] and further studied by a lot of subsequent works.  

2) In this paper, authors considered using raw node features to obtain good attention scores between each node pair. However, the effectiveness of this will heavily depend on how informative these original node features are. In the extreme case, the original node features are completely random, then the proposed method will definitely fail since there are no meaningful similarity scores that can be learnt from. Surprisingly, I did not see any discussion or any ablation study on this aspect. Also, why only using initial node attributes? How about the computed node embeddings to compute the similarity scores?

3) The pair-wise similarity scores (similarity or attention matrix) are very expensive to compute and score, which easily renders quadratic complexity in terms of the number of nodes O(N^2) for both computation and memory. This makes it scale to really large graph. In this sense, the proposed scheme is not promising in real applications. 

4) The experiments are really problematic. There are no descriptions on how the graph data are split in train/val/test. Following the traditional graph kernel settings, it should be 9/1/1. Also, there are no information about how many runs are performed on each dataset. I noticed that authors basically collected all performance results from published related works except for several baselines. Different data split and number of runs could result quite different performance report, leading to apple-and-orange comparisons. 

5) The experimental results are also problematic as well. For example, WL baseline has extremely large variance for all of datasets, which are completely different from the reported number from other literature [Zhang et al, neurIPS'18] and my personal experience. In general, the variance of WL could be 1/10 smaller than what are reported in this paper. Also, there is no clear explanations why the proposed method achieved quite large margin compared to G-UNet and GIN as shown in Table 1. 

RetGK: Graph Kernels based on Return Probabilities of Random Walks, [Zhang et al, neurIPS'18]

6) In Table 3, authors tried to show the performance difference between different pooling methods. I was so surprised about the results after I looked back to Table 1. For instance, Net_top-k (G-Unet) achieved 71.5 on PTC while GUnet achieved 64.7 on PTC in Table 2. It is really hard to believe this almost 7 points difference are just due to slight different model choices in other parts beyond the main contribution - new pooling component. I feel the whole experimental results are not convincing or at least no well explanations what's going on here. Of course, based on very large variances shown in table 1 and 2, table 3 and table 4 are meaningless to look without seeing the variances for each ablation study. 


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a topology-aware pooling method on graph data, which explicitly encodes the topology information when computing ranking scores. More specifically, the proposed method uses an attention operator to compute similarity scores between each node and its neighborhood nodes, and then uses the average similarity score of each node as the ranking score in the node selection process. This topology-aware pooling technique can be applied to graph neural networks on downstream tasks such as graph classification. Experimental results demonstrate the effectiveness of the proposed method, which outperform previous state-of-the-art models consistently.

Overall, this paper is well-organized and the contributions are clear. Although the methodology of adding a graph connectivity term is relatively simple, it is effective to improve the performance on graph classification task and should be straightforward to be applied to other graph learning tasks. Based on the above reasons, I would like to recommend a weak accept for this paper.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposed a topology-aware-pooling method to generate ranking scores for each node and so the pooling (or coarsening) of the graph can be achieved by picking those nodes with higher aggregated attention scores. The ranking scores for each node are computed by taking the average of the scores of its neighbors, by doing which the authors claim that the topological information of the graph is taken into direct account in the pooling steps. 

The proposed method indeed shares some common idea with “Hierarchical Graph Representation Learning with differentiable pooling” by Xing et al. There, a differentiable pooling procedure is used to progressively coarsen the graph to achieve global representation, and the assignment matrix is learned in each hierarchies. This is very similar to the proposed method, since those nodes with higher attention scores are those with more neighbors, which can be considered as ``local cluster centers’’ on a graph. In other words, both methods perform some kind of hierarchical clustering to coarsen the graph. The authors claim that an assignment matrix may cause (1) overfitting, and (2) the coarsened graph may have a different structure than the original graph, which I believe are not justified criticisms. In many cases of clustering the assignment matrix  is auxiliary variables; and even if the assignment matrix is learned as a whole matrix variable, as long as the loss terms is properly defined (such as reconstruction loss in k-means), it can be learned very well because it does not need labels as a unsupervised term.  Second, computing the assignment matrix has more flexibility, especially considering that the coarsening step is not just to perform clustering but rather to facilitate the final classification, therefore locating ``important’’ nodes (in terms of fully representing the graph) is not the only goal, but locating both important and discriminative nodes (i.e., those nodes that can lead to informative features for final classification) is. In this sense, learning an assignment matrix obviously has the benefit of receiving back-propagated gradients from class labels, while the proposed method does not have this flexibility in the pooling step since the neighborhood structure of the graph is fixed throughout the iterations. 

Another important observation is that The proposed method seems to connect each hierarchy directly to the final layer, which is similar to skip-connections. Can this be the main reason why the performance shows improvement, as has been validated in extensive studies in the computer vision community? Some ablation studies are needed to verify that the performance gains are mainly due to the key idea of TAP but not due to the skip connections that have been widely used in computer vision tasks.

Some minor comments:
How do you implement the ranking_k() function? Is it differentiable?

review_title: Official Blind Review #3

Meta-review: This paper proposes to incorporate graph topology into pooling operations on graphs, to better define the notion of locality  necessary for pooling.  While the paper tackles an important problems, and seems to be also well-written, the reviewers agree that there are several issues regarding the contribution and empirical results that need to be addressed before this paper is ready for publication.
Decision: Reject