Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

This paper proposes a self-distillation based graph augmentation mechanism to alleviate the drawbacks of existing MI based models w.r.t. their high dependency towards negative sampling. Quantitatively the proposed model achieves encouraging results. However it would have been better if the system designs and significant difference of IGSD from existing work are discussed.


Strength:

- This work has clearly discussed a drawback of existing unsupervised MI based models which is the leading approach in graph classification
- They propose a mechanism to address this issue with satisfiable quantitative results on unsupervised setting and extended semi-supervised setting with self-training also supported quantitatively.
- Paper is clear in general, with a clear research problem, proposes mechanism for unsupervised/semi-supervised graph representation domain and encouraging quantitative results.

Weakness:

-There is a lack of qualitative analysis and discussion of the proposed method. 
-In Section 4.3 "Performance with different amount of negative pairs", it is not clear the reasoning of the provided observation from Figure 3a.
-It is not clear the motivation behind selecting a teacher-student network for obtaining different views of the graph. These networks are normally used for knowledge transfer, but here used for contrastive learning. How is this more beneficial than an ensemble model w/o knowledge transfer step of Eq. 3. 
-The core difference of IGSD from CMC-graph is that CMC uses MI based contrastive between local patch representation and graph rep, while IGSD uses L2 based contrastive between 2 graph representations. Input, Encoders and projections are the same for both architectures. It could be useful to add some analysis to discuss these differences and their contributions to clearly understand the significance of IGSD.
-This paper seems to have state-of-the-art results (although it is based on graph kernel). Why the results are not included?

Convolutional Kernel Networks for Graph-Structured Data, ICML-2020



=======================

after rebuttal:


I thank the authors for the response. I still have concerns re their comparison with GCKN (ICML-2020). The reproduced results by the authors are different significantly from the published one in Table 1 of GCKN paper (~5%). E.g. MUTAG is 92.8 in original paper, but the authors report 87.2 for GCKN. The difference is significant.

Therefore, I will keep my original rating.
 

review_title: Motivation of key elements and significance of the idea are unclear

final_score: rating: 6: Marginally above acceptance threshold
review_content: Overall Comments:
Learning graph-level representations with only labels has been explored by many works. However, it's not easy to annotate every graph. This paper applies the ideas from semi-supervised classification task to improve the representation quality learned by graph neural network. Specifically the proposed solution combines several kinds of existing techniques including diffusion graph augmentation, mean teacher consistency, debiased contrastive loss and pseudo class consistency. Finally they are combined together to act as a regularization term by utilizing the unlabelled data. From this point of view, the novelty of this work is incremental, but it's still an interesting work for improving graph-level representations.

Clarity:
The presentation is not clear enough. There exists many claims that are not clear, shown as follows:

1. In the last sentence of 3rd paragraph in introduction section, it's difficult to get the connection between negative samples mining and self-distillation strategy. Why using the self-distillation can alleviate the dependency on negative samples mining? The unsupervised objective in Equation 4 still depends on negative samples. 
2. In Section 2.1, the notation for augmentations. Why are the graphs G_L attached without labels after being augmented?
3. In Section 2.3, authors firstly use PPR to augment node features, then randomly remove edges to create a corrupted graphs. According to the description, the question is how many views that will be used in follow sections? I guess that the graph feature from original graph will be fed to student network, and the augmented corrupted graph will be fed to teacher network.

Questions for Rebuttal:
1. Please clarify the mentioned questions above.

2. The proposed method contains an encoder, projector, and predictor. The question is why we need a projector g to get a higher dimension z? Does it have a big influence on the performance? Could you please give the complete definition of function g and h?

3. The definition of L^{con}  in Equation 2 is for positive sample extracted from the same graph G_i. However, the complete unsupervised loss needs negative samples G_j. Could you please also give the definition for L^{G_i, G_j}?

4. The overall loss consists of supervised and unsupervised loss. The L^{sup} has conflict to the first term in Equation 7. Both of them use labels but it's difficult to tell which one should be aligned with the supervised loss shown in the ablation study (Table 2). The SupCon has never been shown in the main content before. Please pay attention to make it clear.

review_title: An interesting paper

final_score: rating: 5: Marginally below acceptance threshold
review_content: ------------------------------------

update after reading the authors' response.

The authors didn't address my question "Did the authors perform a significance study?" A significance test such as double-sided t-test is needed to verifying whether the proposed method is significantly better than baselines.

-------------------------------

This paper proposed a distillation approach for unsupervised graph representation learning. The approach partially builds upon contrastive self-supervised learning which contrasts pairs of augmented graphs. The approach is extended to the semi-supervised setting. The authors performed evaluation in graph classification and regression tasks. 

I recommend to reject this paper, due to the following major concerns: 1) experimental results are not strong; 2) important baselines were not compared; 3) important details such as optimal hyperparameter values are missing. 

My major concerns of this paper include:
1. The improvement of the proposed approach over baselines seem not significant. For example, in Table 1, comparing the mean and standard deviation of the proposed approach and CMC-GRAPH, it seems that the difference is not statistically significant. Did the authors perform a significance study?
2. In the experiments, why the authors didn't compare with GCC, which is a contrastive self-supervised learning approach applied to graph classification?
3. There are many other unsupervised graph representation learning methods. The authors need to compare with more to substantiate this work.
4. In hyperparameter tuning, the authors gave the range of hyperparameters tuned, but didn't give the optimal value of the hyperparameters, which make the paper difficult to reproduce.
5. In table 1, the authors excluded some results since they need more than 1 day to obtain. It is common for deep learning models to run several days to obtain results. I don't think it is proper to exclude these results simply because the runtime is more than 24 hours.

However, the paper does have a few strong points.
1. The ablation studies are well designed and the results are insightful.

2. The paper is well-written and easy to follow, with a clear organization.

3. The experiments were conducted on a rich collection of datasets. 

Other comments.
1. In equation (3), the authors can draw a connection with MoCo.
2. In Table, why didn't report mean and standard deviation of the results?
3. For this result "When batch size is
greater than 32, IGSD outperforms CMC-Graph and the performance gap becomes larger as the batch
size increases.",  can the authors provide a reason that can possibly explain this phenomenon?
4. The authors can add some statistics of the datasets used in Figure 2.




review_title: Experiments need to be improved

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed a method for learning graph-level representation in an unsupervised contrastive way. Instead of contrasting between graph-level representation and patch representation like InfoGraph [1], they contrast graph-level representation of a graph to its augmented variation using a teacher-student framework.

* why use InfoNCE objective instead of the Jensen-Shannon mutual information objective used in InfoGraph [1] ? 

* The major concern about this paer is that the proposed method encourages the closeness of augmented views from the same graph instances but provide no guarantee that the transformation used (graph diffusion and sparsification with PPR + random remove edges in this paper) would be label-preserving. For example, in molecular datasets, if we drop an edge and that edge happens to be in a structural motif, it will drastically change the attributes/labels of the molecule. 

* $v$ represents nodes in section 2.1 and 2.2 and it represents graph instances in section 3.1 and Figure 1. This can be confusing I suggest changing the notation in section 3.1 and Figure 1.1 to $G$

[1] Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semisupervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908.01000, 2019.

review_title: Official Blind Review #3

Meta-review: This paper proposes an unsupervised graph learning method [Iterative Graph Self-Distillation (IGSD)] by iteratively performing self-distillation to contrast graph pairs under different augmented views. This idea is then extended to semi-supervised setting where via a supervised contrastive loss and self-training. The method is empirically evaluated on some semi-supervised graph classification and molecular property prediction tasks, and has achieved promising results.

Reviewers agree that the method is interesting and the paper is well-written. The biggest concern from reviewers related to experimental evaluations of the method. The authors responded to this and included additional experiments. Although the reviewers appreciate the provided results and explanations, at the end they were not convinced about the empirical assessments. In particular, R1's post rebuttal comment indicates concerns about the reported performance of GCKN, which is different from the published one in Table 1 of GCKN paper. I encourage authors to improve on these experimental discrepancies and resubmit. 
Decision: Reject