Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: - The position of this work relates to graph-level contrastive learning. The paper aims to introduce a better way to sample the positive instances, avoiding illegal or poor-quality positive instances.

- The authors propose to select positive graph instances directly from existing graphs in the training set, which can maintain the legality and similarity to the target graphs. 

- The positive sample selection is done on domain-specific pair-wise similarity measurements, sampling from a hierarchical graph encoding similarity relations among graphs. 

- A side idea using adaptively select nodes to mask for even distribution has been proposed.

- Extensive experiments on various graph classification and node classification benchmark datasets have shown the effectiveness of this simple design’s impact on the final pre-train performance. 
main_review: ### Interesting point

This paper mainly aim to discuss the impact of the quality of the positive sample when using contrastive objective to pre-train GNN models. This research angel is interesting and allows the pre-train GNN step to learn more effectively with the quality improved positive samples. 

### Main concern

**Methodology aspect**:
- The approach proposed in the paper seems to be a small incremental change on top of the previous GNN pre-train work. The novelty aspect is low.

- A lot of important points only discussed in the appendix. Going back and forth between the main paper and the appendix makes the reader a bit hard to read through the paper.
- Fails to discuss a easy fix to the poor positive sample quality. If we increase the positive sample number, will it be able to match the performance with the proposed high quality (similarity aware) positive instance sampling strategy.

**Experimental result**:

- Fail to discuss the trade off between the performance improvement gain vs. the introduced computation complexity by computing the similarity  measurement between the positive sampled vs. the original graph. 





limitations_and_societal_impact: - The authors have discussed the limitation of their work to some degree. But the authors fail to discuss one big limitation with is their limitation in terms of the computation complexity. I think the authors should address the trade-off between the performance improvement gain vs. the introduced computation complexity by computing the similarity measurements. 

- I do not believe this work has a negative social impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a new method for graph pre-training based on instance contrastive learning. They exploit graphs in the training set, to select positive instances that are similar to target graphs. They also propose a new strategy for node-level pre-training. 

main_review: The main contribution is the novel pre-training strategy introduced. The work has potential high impact in the research area, given that the issue of selecting appropriate graphs and datasets for pre-training is still an open and relevant question. The proposed strategy ensured that structurally similar graphs are used for pre-training. An important experimental contribution is also the transfer across domains. 

Overall, while technical details are properly provided, I found the general structure of the paper difficult to follow. As a general comment, I would recommend to make the text more smooth and highlight the key steps from sampling to pre-training. 

Detailed comments

1) The sampling strategy and concept of the neighbour graph should be better described. The relations with similarity score and specific pre-training strategy should be highlighted clearly. 

2) It would also be useful to describe in more details, how the sampling strategy is practically used in pre-training. Maybe a schematic figure would help. 

3) I do not see the need of Table 4. The non pre-trained model can be included in table 1. This would also make easier for the reader to validate the effectiveness of the method. 

4) The results for no pre-training of table 2 and 3 seem to be missing

---------------------------------------------------------

After authors response

Thank you for your response. I still think the paper needs some restructuring in order to better describe the contribution and improve the clarity of the text. I will keep my score as it is.
limitations_and_societal_impact: The limitation and societal (broader) impact are adequately addressed
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, the authors argue that existing graph unsupervised pre-training models may miss some important information when selecting positive samples, or the reconstructed graphs do not meet some specifications. Based on this motivation, the authors design a similarity-aware sampler to automatically select positive samples from the existing samples for graph contrastive learning. Experiments are also conducted to demonstrate the effectiveness of the method.
main_review: Strengths: The problem faced by the paper is interesting and timely and the proposed approach seems reasonable. The article is well written, the method is clearly described, and the overall quality is good. The authors also provide the source code to facilitate experimental replication.

Weakness: 
1.	Regarding the adaptive masking part, the authors' work is incremental, and there have been many papers on how to do feature augmentation, such as GraphCL[1], GCA[2]. The authors do not experiment with widely used datasets such as Cora, Citeseer, ArXiv, etc. And they did not compare with better baselines for node classification, such as GRACE[3], GCA[2], MVGRL[4], etc. I think this part of the work is shallow and not enough to constitute a contribution. The authors should focus on the main contribution, i.e., graph-level contrastive learning, and need to improve the node-level augmentation scheme.
2.	In the graph classification task, the compared baseline is not sufficient, such as MVGRL[4], gpt-gnn[5] are missing. I hope the authors could add more baselines of graph contrastive learning and test them on some common datasets.
3.	I am concerned whether the similarity-aware positive sample selection will accelerate GNN-based encoder over-smoothing, i.e., similar nodes or graphs will be trained with features that converge excessively and discard their own unique features. In addition, whether selecting positive samples in the same dataset without introducing some perturbation noise would lead to lower generalization performance. The authors experimented with the transfer performance of the model on the graph classification task, though it still did not allay my concerns about the model generalization. I hope there will be more experiments on different downstream tasks and across different domains.
Remarks: 1. The authors seem to have over-compressed the line spacing and abused vspace. 
2. Table 5 is collapsed.


[1] Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen, “Graph contrastive learning with augmentations,” Advances in Neural Information Processing Systems, vol. 33, 2020.
[2] Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang, “Graph contrastive learning with adaptive augmentation,” arXiv preprint arXiv:2010.14945, 2020.
[3] Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang, “Deep graph contrastive representation learning,” arXiv preprint arXiv:2006.04131, 2020.
[4] Hassani, Kaveh, and Amir Hosein Khasahmadi. "Contrastive multi-view representation learning on graphs." International Conference on Machine Learning. PMLR, 2020.
[5] Hu, Ziniu, et al. "Gpt-gnn: Generative pre-training of graph neural networks." Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020.

limitations_and_societal_impact: The authors elaborate on the article's limitations and social impacts.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors of the paper proposed to select positive graph instances (in graph pre-training) directly from existing graphs
in the training set in order to make them more legitimate.  The proposed scheme is based on pair-wise similarity
measurements as well as sampling from a hierarchical graph encoding similarity relations among graph data sets. An adaptive node-level pre-training method is used to for noe-;evel pretraining tasks as well. Extensive experiments on 13 graph classification and node classification benchmark datasets from various domains are reported to demonstrate the usefulness of the methods. 
main_review: The authors proposed to use exisent samples in the graph data sets to select positive samples, by carefully designing a similarity-based sampling strategy, in order to improve the resultant pre-training step.  The idea is also used in node-level pre-training tasks. The idea is simple and makes some sense, but there are a numbe rof concerns

(1) There are a number of grammar mistakes and the writing needs to be improved. Sentences are too long to understand quickly and precisely.

(2) Is there any advantage of perturbation-based method for existing method in generating positive samples? For example, is the proposed method for generating positive pairs limited by the size of the graph data set?

(3) The proposed method can be deemed as applying node-level contrastive learning methods (such as deep walk, t-SNE, etc.) in graph-level pre-training tasks; the only difference is that  rather than using proximity relation (such as spatial distance between two nodes in a graph, or co-occurrence of two image patches in a single image), the similarity metric has to be used, since there is no clear prosximity relation defined among graph-level objects. But this is exactly what makes the proposed method very similar to existing embedding methods like t-SNE, and the only difference  is that how to exploit the similarities are defined in more refined manner. Therefore, the novelty of the proposed method is limited. Please note that proximity-based positive/negative sample pairs are very much different from similariy-based ones; in the former case, proximity and similarity can be consistent or conflictive, and they interact with each other; but in the latter case, there is only similarity to be discussed and its scope is therefore not much different than embedding method that relies solely on similarities. 

limitations_and_societal_impact: As mentioned above, the proposed idea is more or less a reuse of similarity-based embedding methods. Although authors claim that it is a contrastive learning framework, it is indeed lacking the important interaction between proximity and similarity relations, therefore its close resemblance with embedding methods such as t-SNE, deep walk makes the novelty of the paper very limited.
review_title: null

Meta-review: The paper proposes to select positive graph instances directly from existing graphs in the training set for constrastive learning, using pair-wise similarity measurements as well as sampling from a hierarchical graph encoding similarity relations. It also develops an adaptive node-level pre-training method. Experimental results show that the GNN models pre-trained by the proposed strategies can outperform baselines. 

The reviewers pointed several concerns/aspects for improvement, mainly:
- relative contribution over existing work
- discussion about the baseline of using more positive instances
- discussion on the tradeoff between the gain and the computational overhead
- clarity of presentation
- experiments: more baselines and datasets

The authors have made good responses that significantly strengthened the work. After the response, the consensus is:
- The authors have provided detailed and meaningful discussions on the baseline of using more positive instances, and the tradeoff between the gain and the computational overhead
- The presentation needs improvement
- The authors have provided results of two more baselines that are helpful, while more results will further strengthen the work.
- The novelty/relative contribution of the work remains a concern for the reviewers. Better presentation (importance of the positive samples' quality,relation with existing work) and more thorough and organized experimental evaluation will help illustrate the contribution. 


Decision: Reject