Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper looks into the effect of self-supervised GNN pre-training in graphs used for molecular representation. The paper assesses the performance of various popular graph pre-training objectives and finds that the benefit is negligible in many cases. The paper also finds that when improvement is observed with supervised pre-training, the improvement may diminish with richer features or more balanced data splits. Lastly, the paper shows that hyper-parameters can have a larger impact on accuracy of downstream tasks than the pretraining tasks. Paper offers examples of when pre-training might and might not be useful, based on experimental findings. Takeaways in this paper are contradictory to some related work in the field of GNNs which claims that, similarly to NLP models, GNNs might benefit from self-supervised pre-training. 
strengths_and_weaknesses: - The research question is interesting: does pretraining help in GNNs?
- Paper is sound & authors design and perform an extensive number of experiments to support their claims
- The main question of assessing the value of pretraining is conceptually significant. The findings that pretraining might have negligible gains is interesting. However, one of the takeaways of the paper is that benefit of pretraining diminishes with richer/more high quality features. I believe this is to be expected - particularly on small datasets, as the ones used in this paper (authors also mention this under Limitations). If you have a small graph and rich enough features, then there probably isn’t much additional information to get from pretraining. Moreover, one of the benefits of GNNs is being able to learn complex representations from the graph structure itself, rather than having to use rich features. For small enough graphs, there probably isn’t that much representational information to learn from. Authors acknowledge this under Section 4.6, but I think it should be made clear that the findings in the paper apply to small graphs

questions: - Have the authors thought about whether or not findings in this paper would translate to GNNs pretrained on graphs coming from different distributions - e.g. more general domain?

- Minor Miscellaneous Edits
- Line 18: extra space (GNNs) , 
- Line 21: typo: nurturing?
- Line 184: typo:  scenarios


limitations: Authors adequate address limitations in their paper under ‘Limitations of current study’
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a comprehensive empirical study on pretraining GNNs for downstream molecule property prediction tasks, investigating aspects including features, GNN architectures, pretraining datasets, pretraining objectives, downstream task data split, and fine-tuning hyperparameters etc. It shows that self-supervised pretraining alone generally bring non-significant improvement in performance alone, while supervised pretraining helps when downstream tasks do not have richer features or more balanced data splits.
strengths_and_weaknesses: #Strengths
- The empirical study is novel, systematic and comprehensive, probing a range of design choices made by previous works. 
- The central claims, especially the set of observations in section 5, are valuable summaries for future works and well supported by the experiments.
- The paper is clearly written and easy to follow

#Weaknesses
- The specific contrastive method evaluated is not the known best fit. As shown in GraphCL(https://arxiv.org/pdf/2010.13902.pdf), the choice of data augmentation is crucial. Specifically, this work shows that the suitable augmentations for biochemical molecules should be node dropping and subgraph, instead of attribute masking used in this empirical study. 
- The experiments do not consider combined self-supervision objectives.
- The paper does not include more scalable architectures (e.g. deep GNN and various transformer-based GNNs)

questions: It would be beneficial to also include 
1) another contrastive learning baseline with node dropping and subgraph augmentation
2) the performance when using several or all the self-supervised pretraining objectives
3) some more scalable architectures.
limitations: The limitations are adequately addressed.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Self-supervised pre-training of GNNs on molecule data can be very useful for tasks in computational chemistry, e.g., drug discovery. This has motivated several works proposing pretraining methods that allegedly lead to higher downstream fine-tuning performance over non-pretrained (i.e. randomly initialized) models.

The authors of the reviewed paper challenge this claim and conduct systematic experiments revealing that the aforementioned SSL methods do not have statistically significant advantages over non-pretrained methods in many settings. Their main conclusion is that current SSL methods produce "less transferable knowledge for downstream tasks."  

strengths_and_weaknesses: # Strengths
The paper 
* systematically investigates the sensitivity of choices in different stages of the pre-training pipeline by changing one factor at a time. 
* proposes to split downstream task data not randomly like previous works but based on scaffolds. This allows for better OoD generalization evaluation since evaluating on a single split can be subject to biases. 
* further examines the impact of using rich features [1], i.e., additional node features not commonly used in many previous papers. 
* The paper reveals several surprising results, e.g., Table 2 shows that no pretraining method performs significantly better than non-pre-trained models.  
* The paper complements other recent works reporting positive results for GNN pretraining on molecules with some negative results [1,2,3,4]. This is not to say that one must be right and the other wrong. More diversity in terms of investigated settings or experimental configurations (data split, hyper-parameters, etc.) is useful for the community overall. 

# Weaknesses
Assuming that the authors conducted experiments properly and with genuine efforts to reproduce previous work's results (see my questions below), I don't see any major weaknesses in this empirical study. 

Some minor writing suggestions which do not significantly impact my scores:
* tables and figures are quite misplaced; e.g., Table 3 is shown on Page 4 but eventually discussed on Page 7. Figure 2 is shown right next to section 4.7; it isn't discussed there but in section 6. 
* The paper looks pretty crammed. For example, section 4.4. should be split up into multiple paragraphs.  I also see that the authors removed vertical space/margins of the section titles. I understand that this is likely due to the page limit. However, I suspect that some sentences might not add much information and can be removed to make the key messages stand out more clearly. Alternatively, maybe more details can be moved to the appendix. 

[1] Self-Supervised Graph Transformer on Large-Scale Molecular Data, Rong et al, NeurIPS 2020. 
[2] Graph Contrastive Learning with Augmentations, You et al, Neurips 2020.
[3] Graph Contrastive Learning Automated, You et al, ICML 2021. 
[4] Evaluating Self-Supervised Learning for Molecular Graph Embeddings, Wang et al, arXiv:2206.08005.
questions: * Are the two pre-training datasets you're using (ZINC15,SAVI) commonly used by the pre-training methods you investigate? For example, [1,2] use NCI1, and [3] uses the GEOM dataset. While the two datasets you use seem to be appropriate based on your description, one possibility leading to negligible pre-training gains might still be that they are simply not well-suited for pre-training?

[1] Graph Contrastive Learning with Augmentations, You et al, Neurips 2020.
[2] Graph Contrastive Learning Automated, You et al, ICML 2021. 
[3] Evaluating Self-Supervised Learning for Molecular Graph Embeddings, Wang et al, arXiv:2206.08005.
limitations: Yes, the authors describe two relevant limitations of their work.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: In this paper, the authors perform a comprehensive empirical study on GNN pre-training for molecular representations from the perspectives of pretraining objectives, data splitting methods, input features, pretraining dataset scales, and GNN architectures. They find that self-supervised pretraining alone does not provide statistically significant improvements over non-pretrained methods on downstream tasks. Also, data splitting methods, input features, hyperparameters may be more important than the choice of pre-training strategies. Additionally, they think the insufficient complexity of pre-training on molecular graphs is the key factor for the poor performance.
strengths_and_weaknesses: **Strengths**

- The paper is well-presented and easy to follow.
- The experimental results are meaningful to the community.

**Weakness**
- The authors only evaluate the performance on the downstream task of molecular property prediction. However, their used MoleculeNet benchmark is potentially brittle because the performance of pre-trained models on these datasets is extremely unsteady with diverse random seeds, which may bias their conclusions. I suggest the authors to try more recent benchmarks such as Alchemy [1] for property prediction. Additionally, the learned molecular representations can also be applied in more molecule-related downstream tasks such as drug-target Interaction [2]. Above-mentioned benchmarks or tasks may help avoid the instability of MoleculeNet. 
- The conclusion of existing pre-training strategy cannot work in molecular representations may not be convincing. Firstly, the authors only evaluate some naïve strategies which pre-train GNNs at the level of either entire graphs or individual nodes. As illustrated in a representative work [3], these tasks often give limited improvement and can even lead to negative transfer on many downstream tasks. Moreover, the authors show that 'Masking Node + Supervised' or 'Context Prediction + Supervised' pre-training tasks can bring notable improvements over 'no pre-training', which aligns the conclusions of [3] that combining node-level and graph-level pre-training will be conducive indeed. Additionally, the authors only try the vanilla molecular graph augmentations (adding Gaussian noise to input node features) in contrastive learning, which may alter the semantics of molecules dramatically due to their discrete nature and thus incur negative transfer. I would suggest the authors to try more advanced molecular augmentations such as automated augmentations [4], hidden feature augmentations [5] and knowledge-guided augmentations [6].
- The discussions of the reasons why pretraining does not help are neither adequate nor convincing. Moreover, how can we make the molecular pretraining more difficult?

[1] Alchemy: A Quantum Chemistry Dataset for Benchmarking AI Models

[2] Pre-training Molecular Graph Representation with 3D Geometry (ICLR 2022)

[3] Strategies for Pre-training Graph Neural Networks (ICLR 2020)

[4] Graph Contrastive Learning Automated (ICML 2021)

[5] SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation (WWW 2022)

[6] MoCL: Data-driven Molecular Fingerprint via Knowledge-aware Contrastive Learning from Molecular Graph (KDD 2021)

questions: See the weakness above.
limitations: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper benchmarks the performance of graph pretraining under different settings. In particular, the paper performs thorough ablation studies on the key components of GNN pretraining, including pretraining objectives, data splitting methods, input features, pretraining dataset scales, and GNN architectures. They find that self-supervised graph pretraining does not have statistically significant advantages over non-pretraining methods in many settings. Although improvement can be observed with additional supervised pretraining, the improvement may diminish with richer features or more balanced data splits. The authors hypothesize the complexity of pretraining on molecules is insufficient, leading to less transferable knowledge for downstream tasks.
strengths_and_weaknesses: Strength:
1. The idea of benchmarking graph pretraining under different settings is original and deserves a thorough study. 
2. This paper considers many different settings including pretraining objectives, data splitting methods, input features, pretraining dataset scales, and GNN architectures. The settings are extensive and well studied in the experiments. 
3. The organization and writing are sound and satisfying. 

Weakness:
1. The authors use ZINC/SAVI for self-supervised learning, ChEMBL for supervised pretraining, and evaluate the performance on BBBP, BACE, TOX21, TOXCAST, and SIDER. I am not sure whether the five datasets BBBP, BACE, TOX21, TOXCAST, and SIDER are representative enough. It seems that the choice of downstream tasks is a bit ad-hoc. For example, these three datasets ClinTox, MUV, and HIV are also widely used for evaluating graph pre-training. As far as I know, the performance of graph pretraining varies on different datasets. The results on some datasets are insignificant and may be worse than random initialization. The authors may need to provide more evidence about why these five datasets are chosen and why other datasets are not considered. Otherwise, the evaluation is not comprehensive enough.
2. The authors show that graph pretraining does not bring significant improvements in many settings. Then the authors hypothesize that the complexity of pretraining on molecules is insufficient, leading to less transferable knowledge for downstream tasks. However, the paper does not provide any insights into why graph pretraining fails in many different settings. On the other hand, graph pretraining does improve the performance in some settings. Although the paper hypothesizes that the similarity between downstream tasks and pretraining tasks might play a role here, the paper does not provide significant insights into why pretraining can help in these settings. Some empirical analysis might be needed to improve the paper. 
questions: 1. Why the five downstream datasets are used? Why others are not used?

2. What the reason behind the success and failure of graph pretraining? 
limitations: Yes.
review_title: null

Meta-review: metareview: The reviewers were split about this paper: on one hand they appreciated the sensitivity analyses and surprising findings, on the other they were concerned about the overall contribution of the work. After going through it and the discussion I have decided to vote to accept given the clear and convincing author response. I urge the authors to take all of the reviewers changes into account (if not already done so). Once done this paper will be a nice addition to the conference!
confidence: Less certain
Decision: Accept