Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an unsupervised framework to perform graph representation learning. The local-instance structure is learned by first gets patch-level and graph-level representations for each graph, then maximize the mutual information between both correlated patches and correlated graphs, which are decided by attribute masking strategy. The global-semantic structure is maintained by leveraging RPCL to derive hierarchical prototypes of the representation and maximizing the mutual information between correlated graph representation and the searching path in the prototypes.

Strengths:
+  This paper presents a framework to jointly consider the local instance structure and global-semantic structure of graphs. It is a meaningful direction and could be beneficial for explainability.
+  The experimental results are quite thorough with comparisons to several baseline methods. Moreover, the ablation study of different mechanisms is provided in the experiments.

Weaknesses:
-  The proposed model seems like a simple combination of several existing techniques and thus lacks novelty.
-  The performance of this model seems to heavily rely on the attribute masking strategy as all the operations are built upon the correlated graph pairing from the attribute masking strategy. But how reliable is this technique? It seems to be a bottleneck of the model, and I think there should be an explanation on this either theoretically or experimentally.

Overall, the proposes a reasonable model for learning hierarchical graph representations. However, the novelty is limited since the proposed method seems like a simple combination of several existing techniques.

Questions:

1.  As I mentioned earlier, I wonder how reliable the attribute masking strategy is. As graph matching is an extremely hard problem, can this strategy provide a reliable pairing between correlated graphs?
2.  It is not clear how to leverage prototypes in classification tasks? I understand that the prototypes serve to ensure a better structure of the embeddings, but when classifying graphs, I wonder whether embeddings and prototypes are both used or not?
3.  In the Constraint for the global-semantic structure part, the loss for a graph embedding includes both the representations for its correlated graph and its searching path consisting of several prototypes. When minimizing the loss, I wonder both representations of the correlated graph and prototypes are updated together? Or the prototypes are only updated in eq. (11) and (12).

review_title: Official Blind Review #2

final_score: rating: 6: Marginally above acceptance threshold
review_content: The motivation and novelty of the proposed method are good. However, the validation is kind of
weak.
I can understand that this papers follows the validation in Hu et al (2019), however, I feel that
two tasks (one on chemical benchmark and one on biological benchmark) may not be sufficient
to give a detailed idea of the improvement of the proposed GraphLoG over other baselines. I
think 3-5 tasks are much better.
For the ablation study in Section 5.4, these ablated items are good. However, I more would like
to see fluctuated parts in the proposed GraphLoG. One example may be: is there any different
choice/option for hierarchical prototype? Which one is good/bad? What is the reason. Or other
potential and similar examples exist in the proposed GraphLoG. I think this will help us to
understand GraphLoG more.
review_title: This paper proposes GraphLoG for self-supervised graph-level representation learning. It can learn both local-instance and global-semantic information. Experiments are conducted on chemical and biological benchmark.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Pros:
- The paper proposed a novel self-supervised learning method to embed graphs to vector space. Different from previous methods, the method proposed a global-semantic learning strategy to encourage the embeddings to form a hierarchical clustering structure.  Both the embedding network and the hierarchical structure can be jointly learned.

- Authors have provided extensive and convincing comparison results and numerical analysis to show the effectiveness of the method.

- The paper is well-organized and clearly written. To the best of my knowledge, the proposed method is technically feasible.

Cons:
- The number of prototypes is determined by RPCL and can not be adjusted in training.　
- Clustering algorithms are usually not very robust. Since the prototypes of GraphLoG is initialized by RPCL, is the performance of GraphLoG robust? 
review_title: An interesting work for self-supervised graph representation learning

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a method for self-supervised graph-level representation learning. The main idea is to enforce both the instance level smoothness embedding constraints, and a so-called global, semantic grouping structures across all instance graphs in the training data set.  To achieve this goal, the authors have adopted a global clustering framework to encourage the embedding of the graphs belonging to the same clusters to be close to each other, and by using a hierarchically organized set of prototypes. The proposed method is applied to pre-train GNN on massive unlabeled graphs, which is then fine-tuned to downstream learning tasks.

Enforcing a global clustering structure can be useful in capturing the distribution of large number of graphs in the training data set and hopefully carry the learned representations over to other tasks. However, it appears to me that the paper has combined the carefully devised ideas from too many existing work, each of which alone has shown great success in improving the learning performance. Therefore it can be difficult to judge which part of the choices really leads to the final improvement, and in particular whether it is  the local and global structure preserving part, which seems to be the core theme of the paper (with the other theme being sel-supervised learning), that can fully explain the result.

In more detail, the authors have used (1) masking strategy by Hu et al., 2019 to generate correlated graph pairs, (2) mutual information estimation technique InfoNCE to enforce the correlation between paired graphs, and (3) Rival Penalized Competitive Learning (RPCL) as the main building block for hierarchical prototype-based learning. Therefore, a natural question to ask is, if one uses plain GNN architecture of each graph and plug it in a (hierarchical) clustering framework (without self-supervised learning and RPCL), whether similar improvements in the learning performance can still be obtained? If not, then the gains are merely due to the effectiveness of these specially designed components from the literatures and not by the general idea of local and global structures. 

With regard to this concern, I would suggest the authors to clarify on what is the main theme of the work and demonstrate that the empirical performance gains are truly due to a novel, focused idea they propose rather than by combining some of the  existing algorithms which have shown great impact and performance gains in their respective context. In the current form, the novelrity of the work seems less significant by introducing so many components from other works.

review_title: A combination of too many existing ideas that shadows the novelty and makes it hard to judge who should be given the credit of the empirical performance gains

Meta-review: This paper proposes a self-supervised learning method for learning representations for graph-structured data, with both local and global objectives. The local objective aims to maximize the mutual information between two correlated graphs generated with attribute masking [Hu et al. 19], with the InfoNCE loss [van den Oord et al. 18], and the global objective aims to cluster the graphs using the RPCL [Xu et al. 93] objective, which pulls the sample toward the closest cluster while pushing it away from the rival clusters. The proposed method is validated on standard graph classification benchmarks by training a linear classifier on top of the GNN pre-trained with it, and the results show that it largely outperforms existing graph pre-training methods. 

This paper fell into a borderline case, receiving split reviews with two of the reviewers learning toward rejection, and two others proposing to accept. The reviewers in general agreed that the experimental validation is thorough (except for one reviewer), and some of the reviewers mentioned that the proposed idea of performing self-supervised learning at both local and global level makes sense. However, the negative reviewers were concerned with the limited novelty of the proposed method, since the proposed method seems like a simple combination of two objectives each of which are based on existing ideas (although the latter has not been explored for GNN pre-training). The reviewers had interactive discussions with the authors, and the authors provided detailed feedback. Yet, the reviewers were not convinced that the method has sufficient novelty to warrant publication even after the internal discussion period, and decided to keep their negative ratings.

I believe that this is a simple yet effective pre-training method for GNNs on graph-structured data. The proposed method of combining the local and global objective seems like a promising solution to learn a metric space that well-captures the graph-level similarity and also is well-separated for discriminative classification, and it may have some practical impact given its good performance on benchmark datasets. However, as the two negative reviewers mentioned, the paper in its current form is presented as a simple combination of existing approaches. The local objective is a slight modification of attribute masking strategy of [Hu et al. 19], and the global objective of clustering has been explored in self-supervised learning of CNNs for image data [Asano et al. 20]. Thus, I lean toward rejecting the paper, considering its relative novelty and quality. 

However, I find the proposed work highly promising, and encourage the authors to further develop the method while also improving on the paper writing. I suggest the authors to focus more on the main idea of learning with both local and global objectives, without specifically tying each objective to any of the existing methods. The authors may consider various techniques for both local and global objectives (such as hinge loss-based contrastive loss with k-means clustering as shown in the response to R3), and suggest the proposed work as a more general framework.  

[Asano et al. 20] Self-Labeling via Simultaneous Clustering and Representation Learning, ICLR 2020
Decision: Reject