Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: *Summary

This paper presents a novel motif-level GNN pretraining called "Motif-based Graph Self-Supervised Learning (MGSSL)" and multi-level self-supervised pre-training. Most existing self-supervised pre-training frameworks for GNNs are only defined as node-level or graph-level tasks. But molecular graphs would follow a compositionality principle, and the complex structures are defined by their parts (motifs/fragments/scaffolds) and how to combine them.

It leverages the BRICS fragmentation algorithm [4] and further refinements to decompose a molecule into a "motif tree" representation that describes parts-level compositions of the given molecule in a tree form. Along with atom-level self-supervised tasks by atom-label masking, topology and label prediction for this 'motif tree' defines a nice self-supervised task at a motif level. 

Extensive experiments on multiple downstream tasks demonstrated that the proposed pretraining framework outperforms all other state-of-the-art baselines.

main_review: Overall I liked the idea of this paper, and it provides a very interesting approach to GNN pretraining for molecular tasks. Related work is also quite relevant, and extensive experiments for multiple downstream tasks with scaffold-split are very nice to see. Also, it is also nice to see that several common GNN architectures (GCN, GIN, RGCN, DAGNN, GraphSAGE) are fed to the proposed pretraining framework, and observed performance gains for all cases. Comparisons to vocabularies of vanilla BRICS as well as JT-VAE fragments are very informative, too.

The following are some points unclear to me, and it will be nice to be in a clear description.

1.
GNN models are described just as function f_theta, but this symbol is not used explicitly in the main paper. Because of this, it is a bit unclear what is assumed as the input to GNNs, and how we can use the pretrained GNNs for downstream tasks such as property prediction.   

We have two graphs, input molecular graph G + the corresponding 'motif tree' T(G). Are GNNs pretrained as a function G -> vector? Or T(G) to vector, and when we use it in downstream tasks, T(G) needs to be constructed, keeping the same BRICS+alpha vocabulary to label T(G) accordingly??

Furthermore, multi-level cases, do we need two GNNs pretrained for both atom-level and motif-level? Also, use both two GNNs for transfer to downstream tasks?

It is appreciated to clearly describe the input-output of GNNS using the defined symbol f_theta, how to use the output vector for pretraining, and how to use the pretrained GNNs for downstream tasks. 

2.
Similarly, line 196 says "the question is ... how to model the conditional probability log p_theta(..) ", but this probability is never explicitly defined. So this implies minimizing the loss eq.(13) corresponds to maximizing the log likelihood (the conditional probability?). Any explicit and consistent descriptions are appreciated.

limitations_and_societal_impact: Any limitations are not clearly described. It is appreciated to summarize them somewhere in the manuscript?
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary:  This paper proposes Motif-based Graph Self-supervised Learning, which pre-trains GNNs with a novel motif generation task. The authors first use a retrosynthesis-based algorithm with two additional rules to fragment molecule graphs and get meaningful motifs. Next they propose a motif generative pre-training framework, where in each step the pretrained GNN is required to make topology and motif label predictions.
main_review: Strengths:
1. The idea of pretraining GNN with motif prediction is interesting;
2. The paper is well-written and easy to follow.

Weaknesses:
1. The authors mentioned that they use BRICS algorithm with two added rules to segment molecule graphs. However, my concern is that if there is a standard definition or criteria on the granularity of functional groups. For example, should C(=O)OH be seen as a whole or two functional groups: C=O and OH? Should C6H5-OH (phenol) be treated as a whole or two functional groups: benzene ring and -OH? Will this granularity affect the performance of the proposed model?
2. BERT can use this pretraining strategy, i.e., predicting the masked word using its context, because adjacent words are seen as related. However, I was wondering if motifs in a molecule also have such property. Given all motifs but one in a molecule graph, can we really predict what the missing motif is? Is there any relationship between adjacent functional groups in a molecule? I suggest that the authors conduct an empirical study on this to show correlation of motifs.
3. According to Table 4, the proposed model does not show significant improvement over baselines.

limitations_and_societal_impact: The authors do not mention limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors present a novel pre-training method (Motif-based Graph Self-Supervised Learning, or MGSSL) for graph neural networks specific to molecular property prediction. The pre-training involves constructing synthetic tasks to predict atom information, generation of the molecule, and motif identity. They demonstrate this pre-training improves GNN model performance on several MoleculeNet tasks across 5 different GNN models.
main_review: ## Originality
Rating: High

- [+] The use of BRICS to generate motifs for self-supervised pre-training is surprising and novel. The authors compare this approach of generating motifs to the fragments proposed by the JT-VAE and found that using the larger BRICS fragment library was helpful for pre-training the model.


## Quality
Rating: Moderate to High
- [+] The MGSSL approach is applied to 5 different graph neural network models, and compared against the performance of 5 self-supervised pre-training methods as well as GNNs without pretraining
- [+] The authors perform ablation studies to demonstrate the effects of different parts of the pre-training, atom-level and sequential pre-training.
- [+] The authors analyze the impact of the fragmentation method/motif vocabulary size on model improvement after pre-training.
- In Table 1, can you provide an explanation for why the MGSSL pre-training approach does not not show as much improvement over the other SSL methods?

## Clarity
Rating: Moderate to High
- [+] The description of MGSSL pre-training was very clear; I was able to follow the approach taken by the authors.
- For the baseline GNN models, did you train the GNN models for additional epochs? Or were they given the same number of epochs as the pre-trained models? Based on the graphs shown in Figure 4, it seems that the base GNN model (not pretrained) has not yet reached convergence on some of the tasks.
- In table 1, please specify what the values right of the $\pm$ are. I see from 4.1 you mention these are standard deviations of 3 runs of scaffold splitting (I assume?) but it might be helpful to mention this in the table.
- Please include additional architectural information (embedding sizes, full atom/bond feature set) in the supplementary information, if not in the main text.


## Significance
Rating: High

MGSSL improves performance on the selected tasks for a variety of GNN models. Assuming the authors provide code for MGSSL, or the pre-trained model reported in this work, MGSSL could become a very useful starting point for any molecular property prediction project.

I have some questions about the generalizability of MGSSL outside of drug-like chemical space, which I discuss in the Limitations section below. 


## Other notes/comments
Section 2 on molecular property prediction, the authors state 'Recently, ... many works regard molecules as graphs and explore the graph convolutional network for property prediction.' 

While the use of graph convolutional networks are recent, I would argue that molecules have been regarded as graphs for a long time. Molecular fingerprints such as [ECFP](https://pubs.acs.org/doi/10.1021/ci100050t) and other [cheminformatics descriptors](http://www.talete.mi.it/products/dragon_molecular_descriptors.htm) contain on graph-based features of molecules, such as path length, topological indices. In my view, GNNs provides a way of learn which of these descriptors are most important, but it is still working with the same features.

limitations_and_societal_impact: Overall, the authors did a good job of comparing against baselines and analyzing the effect of different components of MGSSL (i.e. pre-training components, motif library creation).

For societal impact, have you considered the computational costs/savings associated with pretraining on MGSSL?

An additional limitation to consider: I would be interested to know how well MGSSL applies to areas of chemical space outside of drug-like molecules. The authors currently use the ZINC dataset, a collection of drug-like molecules, and the fragments are generated using BRICS, which generated fragments for retrosynthesis of drug-like molecules. The [ZINC dataset itself contains biases](https://www.pnas.org/content/116/24/11624); I wonder whether MGSSL also captures these biases.

For applications in other areas of chemical space, such as organic photovolataic molecules, would MGSSL have a lower rate of improvement, or even hurt the model's prediction?

How well do you think MGSSL would work for new areas of drug-like chemical space that haven't been explored before?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a new self-supervised pre-training framework for Graph Neural Networks (GNNs) called Motif-based Graph Self-supervised Learning (MGSSL) to predict molecular properties. First, the method uses BRICs, a molecule fragmentation method, to create a Graph Motif vocabulary. Next, it performs pre-training using a model with three components -  1) a motif generation module that uses GRUs to recursively generate motifs given the previously generated motifs. 2)  a motif labeling module that uses dense layers followed by a softmax to predict the label of the motif from the vocabulary created in the previous step. 3) a GNN module that generates node and edge embeddings which are finally used to predict node and edge attributes of a molecule graph. MGSSL is trained as a multi-level model (with atom and motif layers) and it minimizes a hybrid loss consisting of Motif, Atomic, and Bond prediction losses, with weights adjusted by the Frank-Wolfe Algorithm. The results report improved performance for molecular property predictions over existing pre-training frameworks 7/8 datasets. 

main_review: Strengths:

++ This paper is a good addition to the recent literature focusing on accurately modeling domain-specific (chemistry) knowledge using deep learning methods. It introduces context-specific mechanisms (incorporating the molecular structure) to provide a performance improvement over established techniques. 

++ The different components and levels of the model are interesting and their roles make sense for the task the paper aims to solve. 

++ It presents results for 8 different datasets and 6 relevant baselines. It also provides results for different GNN architectures, the influence of the size of motif vocabulary on the performance, and ablation studies to justify the final MGSSL model.  

++ The paper places the work with respect to existing methods, lists the contributions, and clearly defines motivations behind them.


Weaknesses:

-- The methodology has multiple and extensive steps, however, the paper does not discuss any limitations related to computational or time cost compared to existing methods. Is the cost/performance tradeoff reasonable? 

-- It is unclear whether hyperparameter tuning was performed for all the baseline models when reporting performance. If not, was the choice of default settings fair?

-- In terms of applicability, the MGSSL model seems very specific to the molecular graph domain. Even though the paper presents a good use case for molecular property prediction, its contributions to the wider community are unclear. For example, the BRICS algorithm does not apply to other graph learning tasks. It would be helpful if the paper clarifies its contributions for the wider use of the method. 
 
Minor points:

-An overview figure for the whole model with the components like GRUs and GNNs and connections to the method section would be very useful.
  
-Figure 4 presents 4 benchmark datasets due to page limits, the rest can be adjusted in the supplementary

Errata: Line 220: “parent motif i generate → generates”


limitations_and_societal_impact: Did not find any specific discussion on the limitations of the work and potential negative societal impacts.
review_title: null

Meta-review: This paper presents a novel motif-level GNN pretraining. While most existing self-supervised pre-training frameworks for GNNs are only defined as node-level or graph-level tasks, molecular graphs would follow a compositionality principle, and the complex structures are defined by their parts (motifs/fragments/scaffolds) and how to combine them. The authors show that by leveraging the BRICS fragmentation algorithm to decompose a molecule into a "motif tree" representation, the pretraining could become more effective. 
This paper triggered a lot of discussions between the authors and the reviewers, as well as among the reviewers themselves. One reviewer raised very serious concerns on the technical novelty (whether leveraging an existing algorithm for molecule decomposition could provide sufficient novelty to the pretraining task), and on the choice of GIN as the backbone model in the experiments (many stronger GNN models were developed recently). On the other hand, other reviewers believed that the introduction of motif to molecule graph learning was novel by its own, especially for this inter discipline, and the use of GIN makes sense and does not affect the main conclusion. Although a consensus was not successfully reached at the end, I tend to agree that this paper has its value in representation of molecule graph (and shed some lights on AI for molecular science), and there is no serious issue with the choice of GIN. Therefore, my recommendation is ACCEPT as a poster.

Decision: Accept (Poster)