Motif-based Graph Self-Supervised Learning for Molecular Property Prediction
Authors: ZAIXI ZHANG
Qi Liu
Hao Wang
Chengqiang Lu
Chee-Kong Lee
Venue: NeurIPS 2021 Poster (2021)
Keywords: Graph Neural Networks
Self-supervised Learning
Molecular Property Prediction

Abstract: Predicting molecular properties with data-driven methods has drawn much attention in recent years. Particularly, Graph Neural Networks (GNNs) have demonstrated remarkable success in various molecular generation and prediction tasks. In cases where labeled data is scarce, GNNs can be pre-trained on unlabeled molecular data to first learn the general semantic and structural information before being finetuned for specific tasks. However, most existing self-supervised pretraining frameworks for GNNs only focus on node-level or graph-level tasks. These approaches cannot capture the rich information in subgraphs or graph motifs. For example, functional groups (frequently-occurred subgraphs in molecular graphs)  often carry indicative information about the molecular properties. To bridge this gap, we propose Motif-based Graph Self-supervised Learning (MGSSL) by introducing a novel self-supervised motif generation framework for GNNs. First, for motif extraction from molecular graphs, we design a molecule fragmentation method that leverages a retrosynthesis-based algorithm BRICS and additional rules for controlling the size of motif vocabulary. Second, we design a general motif-based generative pretraining framework in which GNNs are asked to make topological and label predictions. 
This generative framework can be implemented in two different ways, i.e., breadth-first or depth-first. 
Finally, to take the multi-scale information in molecular graphs into consideration, we introduce a multi-level self-supervised pre-training. Extensive experiments on various downstream benchmark tasks show that our methods outperform all state-of-the-art baselines.

Motif-based Graph Self-Supervised Learning for Molecular Property Prediction
Zaixi Zhang', Qi Liu",, Hao Wang', Chengqiang Lu', Chee-Kong Lee
1: Anhui Province Key Lab of Big Data Analysis and Application, School of Computer Science and Technology,
University of Science and Technology of China
2: Tencent America
(zaixi, wanghao3, lunar) @ maw..ustc.edu.... edu.en, qiliuql @ustc.edu.cn checkonglee@@ tencent.com
Abstract
Predicting molecular properties with data-driven methods has drawn much attention in recent years. Particularly, Graph Neural Networks (GNNs) have demonstrated remarkable success in various molecular generation and prediction tasks. In cases where labeled data is scarce, GNNs can be pre-trained on unlabeled molecular data to first learn the general semantic and structural information before being finetuned for specific tasks. However, most existing self-supervised pre-training frameworks for GNNs only focus on node-level or graph-level tasks. These approaches cannot capture the rich information in subgraphs or graph motifs. For example, functional groups brequentty-occurrge
subgraphs in molecular graphs) often carry indicative information about the molecular properties. To bridge this gap, we propose Motif- based Graph Self-supervised Learning (MGSSL) by introducing a novel self- supervised motif generation framework for GNNs. First, for motif extraction from molecular graphs, we design molecule fragmentation method that leverages a retrosynthesis-basee algorithm BRICS and additional rules for controlling the size of motif vocabulary. Second, we design a general motif- generative pre-training framework in which GNNs are asked to make topological and label predictions. This generative framework can be implemented two different ways, i... breadth-firs' depth-first. Finally, to take the multi-scale information in molecular graphs into consideration, we introduce a multi-level self-supervised pre- tre-training. Extensive experiments various downstream benchmark tasks show that our methods outperform all state-of saate-of-the-ne the-art art baselines.
1
Introduction
The past decade has witnessed the remarkable success deep learning in natural language processing (NLP) (5), computer vision (CV) [3], and graph analysis [19. Inspired by these developments, researchers in the chemistry domain have also tried to exploit deep learning methods for molecule- based tasks such retrosynthesis 47 and drug discovery [ To preserve the internal structural information, molecules can be naturally modeled as graphs where nodes represent atoms and edges denote the chemical bonds. Recently, some works applied Graph Neural Network (GNN) and some of its variants for molecular property prediction and obtained promising results g 25).
Though GNNs have achieved remarkable accuracy on molecular property prediction, they are usually data-hungry, i.e. large amount of labeled data (i.e., molecules with known property data) is required for training g 12] However, labeled molecules only occupy an extremely small portion of the
Qi Liu and Chee-Kong Lee are corresponding authors.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

enormous chemical space since they can only be obtained from wet-lab experiments or quantum chemistry calculations, which are time-consuming and expensive. Moreover, directly training GNNs on small labeled molecule datasets in supervised fashion is prone to over-fitting and the trained GNNs can hardly generalize to out-o--distiiuttion data.
Similar issues have also been encountered in natural language processing and computer vision. Recent advances in NLP and CV address them by self-supervised learning (SSL) where a model is first pre-trained on a large unlabeled dataset and then transferred to downstream tasks with limited For example, the pre-trained BERT language model [5] is able learn expressive contextualized word representations through reconstructing the input text-next sentence and masked language predictions so that can significantly improve the performance of downstream tasks. Inspired by these developments, various self-supervisee pre- pre-training methods of GNNs have been proposed [14 15
labels 13
32, 10 30 Based on how the self-supervised tasks are constructed, these methods can be classified into two categories, contrastive methods and predictive methods. Contrastive methods foree views from the same graph (e.g., sampling nodes and edges from graphs) to become closer and push views from different graphs apart. On the other hand, the predictive methods construct prediction tasks by exploiting the intrinsic properties of data. For example, Hu et.al. 14] designed node-level pre-training tasks such as predicting the context of atoms and the attributes of masked atoms and bonds. [I5] introduced an attributed graph reconstruction task where the generative model predicts the node attributes and edges to be generated at each step.
52
51
However, we argue that existing self sule-supe learning tasks on GNNs are sub-optimal since most of them fail to exploit the rich semantic information from graph motifs. Graph motifs can be defined as significant subgraph patterns that frequently occur [27]. Motifs usually contain semantic meanings and are indicative of the characteristics of the whole graph. For example, the hydroxide (-OH) functional group in small molecules typically implies higher water solubility. Therefore, it is vital to design motif-level self-supervised learning tasks which can benefit downstream tasks such as molecular property prediction.
Designing lovif-level supervuue learning tasks brings unique challenges. First, existing motif mining techniques could not be directly utilized to derive expressive motifs for molecular graphs because they only rely on the discrete count of subgraph structures and overlook the chemical validity
1 118. Second, most graph generation techniques generate graphs node-by-node 23 50), which are not suitable for our task of motif generation. Finally, how to unify multi-level self-supervised pre-training tasks harmoniously brings a new challenge. One naive solution to do pre-training tasks sequentially may lead to catastrophic forgetting similar to continual learning 77].
To tackle the aforementioned challenges, we propose Motif-based Graph Supe-Supe Learning (MGSSL) and Multi-level self-supervised pre-training in this paper. Firstly, MGSSL introduces a novel motif generation task that empowers GNNs capture the rich structural and semantic information from graph motifs. To derive semantically meaningful motifs and construct motif trees for molecular graphs, we leverage the BRICS algorithm which based on retrosynthesis from the chemistry domain. Two additional fragmentation rules are further introduced to reduce the redundancy of motif vocabulary. Secondly, general motif-based generative pre-training framework is designed to generate molecular graphs motif-by-ma my-motif. The pre-trained model is required to make topology and attribute predictions at each step and two specific generation orders are implemented (breadth-firs and firth-first). Furthermore, to take the multi-scale regularities of molecules into consideration, we introduce Multi-level self-supervised pre-training for GNNs where the weights of different SSL tasks are adaptively adjusted by the Frank-Wolff algorithm [16). Finally. by pre-training GNNs on the ZINC dataset with our methods, the pre-trained GNNs outperforms all the state-of-the-arr baselines on various downstream benchmark tasks, demonstrating the effectiveness of our design. The implementation is publicly available at http:///thubb.oom//xiiaiaangg///
2 Preliminaries and Related Work
2.1 Molecular Property Prediction
Prediction of molecular properties is central research topic in physics, chemistry, and materials science 44 Among the traditional methods, density functional theory (DFT) is the most popular one and plays a vital role in advancing the field (2T]. However, DFT is very time-consuming and its complexity could be approximated as (NV3), where N denotes the number of particles. To find
2

---

more efficient solutions for molecular property prediction, various machine learning methods have been leveraged such as kernel ridge regression, random forest, and convolutional neural networks 6 26 36 To fully consider the internal spatial and distance information of atoms in molecules, 25] regard molecules as graphs and explore the graph convolutional network for property prediction. To better capture the interactions among atoms, [ proposes message passing framework and 220 48) extend this framework to model bond interactions. [25) builds a hierarchical GNN to capture multilevel interactions. Furthermore, [32] integrates GNNs into the Transformer-style architecture to deliver a more expressive model.
many works 35 48
2.2 Preliminaries of Graph Neural Networks
Recent years have witnessed the success of Graph Neural Networks (GNNs) for modeling graph data [ 19 40
43 42 155. Let G (V, E') denotes a graph with node attributes X, for v E V and edge attributes Euv for (u,v) E E. GNNs leverage the graph connectivity as well as node and edge features learn a representation vector (i.e., embedding) hy for each node v E G and vector lig for the entire graph G. Generally, GNNs follows a message passing paradigm, in which representation of node v iteratively updated by aggregating the representations of e's neighboring nodes and edges. Specifically, there are two basic operators for GNNs; AGGREGATE() and COMBINE()). AGGREGATE()) extracts the neighboring information of node v; COMBINE(.) serves as the aggregation function of the neighborhood information. After k iterations of aggregation, v's representation captures the structural information within its k-hop network neighborhood. Formally, the k-th layer of a GNN is:
46
( COMBINE (N( (.--DGAGGGGAATEE (1 (k-1)) ewm)) : ((w))))) (1)
where h/ denotes the representation of node at the k-th layer, euv is the feature vector of edge between u and and and (vv) represents the neighborhood set of node v. h,()) initialized with X,.. Furthermore, to obtain the representation of the entire graph ha. READOUT functions are designed to pool node representations at the final iteration K,
hg READOUT (
READOUT is a permutation-invante function, such as averaging, sum, max or more sophisticated graph-level pooling functions 49 53
2.3 Self-supervised Learning of Graphs
Graph Self-supervised learning aims to learn the intermediate representations of unlabeled graph data that are useful for unknown downstream tasks. [4 52 5T
32 10 30 39
28, 41] Traditional
graph embedding methods [ 29 define different kinds of graph proximities, i... the vertex proximity,
as the self- supervuuee objective to learn vertex representations. Furthermore, 39 28 41
proposed to
use the mutual information maximization as the optimization objective for GNNs. Recently, more 51
30 Based on how the self-supervised tasks constructed, these models be classified into categories, namely contrastive models and predictive models. Contrastive models try to generate informative and diverse views from data instances and perform node-to-conter. [14), node-to- to-graph
self-supervised tasks for GNNs have been proposed 14 15| 52
32
10
54
39 or motif-to- to-ghh [5] contrastive learning. On the other hand, predictive models are trained in a supervised fashion, where the labels are generated based on certain properties of the input graph data, i.e., node attributes
32 or by selecting certain parts of the graph [5] 32
However, few works try to leverage the information of graph motifs for graph self supervisd learning. Grover 32 use traditional softwares to extract motifs and treat them as classification labels. MICRO- Graph 54 exploits graph motifs for motif-gro -otif-graph contrastive learning. Unfortunately, both methods fail to consider the topology information of motifs.
3 Motif-based Graph Self-supervised Learning
In this section, we introduce the framework of motif-based graph self-supervised learning (Figure I Generally, the framework consists of three parts: chemistry-nnppired molecule fragmentation, motif generation and multi-level self- sele-suuer pre-training for GNNs.
3

---

Topology Prediction
Motif Prediction
Motif layer
Spond
types
Atom layer
Nill
Liopo
Cyred
meek
Lutorn
C.N.O.S,C Cl...
Figure 1: Illustration of Motif-based Graph Self-supervised learning (MGSSL). The multi-level pre- pre-training consists of two layers, Atom layer and Motif layer. the Atom layer, we mask node/edge attributes and let GNNs predict those attributes based on neighboring structures. In the Motif layer, we construct motif trees and perform motif generative pre- tre-training. In each step, based on existing motifs and connections, topology and motif predictions are made iteratively.
3.1 Chemistry-inspired Molecule Fragmentation
Given a dataset of molecules, the first step of our method is to decompose molecules into several fragments/mottifs Based these fragments, molecule graph can be converted into motif tree structure where each node represents motif and the edges denotes the relative spatial relationships among motifs. We choose represent the structure of motifs generation task. Formally, given a molecule graph G = (V, E), a motif tree T(G) = (V.& X) is connected labeled tree whose node set is V [M1, ...y Mn and edge set E. X' refers to the induced motif vocabulary. Each motif M; = (V,, E,) is a subgraph of G. There are many ways to fragment given graph while the designed molecule fragmentation method should achieve the following goals: 1) In motif tree T(G), the union of all motifs M₁ should equals G. Formally, V.V. and E, U& E. 2) In a motif tree T(G), the motifs should have no intersections. That is M; n M4 0. 3) The induced motifs should capture semantic meanings, e.g., similar to meaningful functional groups in the chemistry domain. 4) The occurrence of motifs the dataset should be frequent enough so that the pre-trained GNNs can learn semantic information of motifs that can be generalized downstream tasks. After the molecule fragmentation, the motif vocabulary of the molecule dataset should have a moderate size.
tree because it facilitates the motif
In Figure 2 we show the overview of molecule fragmentation. Generally, there are three procedures, BRICS fragmentation, further decomposition, and motif tree construction. A motif vocabulary can be built via preprocessing the whole molecule dataset following the molecule fragmentation procedures.
To fragment molecule graphs and construct motif trees, we firstly use the Breaking of Retrosyntheti- cally Interesting Chemical Substructures (BRICS) algorithm 44] that leverages the domain knowledge from chemistry. BRICS defines 16 rules and breaks strategic bonds in a molecule that match a set of chemical reactions. "Dummy" atoms are attached to each end of the cleavage sites, marking the location where two fragments can join together. BRICS cleavage rules are designed to retain molecular components with valuable structural and functional content, e.g. aromatic rings.
However, we find BRICS alone cannot generate desired motifs for molecule graphs. This because BRICS only breaks bonds based on limited set of chemical reactions and tends to generate several large fragments for a molecule. Morcover, due to the combination cxplosion of graph structure, we find BRICS produces many motifs that are variations of the same underlying structure (c.g., Furan ring with different combinations of Halogen atoms). The motif vocabulary is large (more than 100k unique fragments) while most of these motifs appear less than 5 times in the whole dataset.
To tackle the problems above, we introduce a post-processing procedure BRICS. combination explosion, we define two rules operating on the output fragments of BRICS: (1) Break the bond where one end atom in a ring while the other end not. (2) Select non-ring atoms with three or more neighboring atoms as new motifs and break the neighboring bonds. The first rule reduces the number of ring variants and the second rule break the side chains. Experiments show that these rules effectively reduce the size of motif vocabulary and improve the occurrence frequency of motifs.
To alleviate the

---

OCIIO
ChiPloonc
OOCINO
OCIIOO
Tue
BRICS Fragmentation
Further Decomposition
Motif Tree Construction
OH
Matif Vocabulary
Figure 2: Overview of molecule fragmentation. Generally, there three steps: Firstly molecule graph is cleaved based on BRICS. 2) Further decomposition to reduce the redundancy motifs 3) Construct motif trees from molecule graphs. A motif vocabulary is built after preprocessing the whole molecule dataset.
3.2 Motif Generation
Here, we present the framework of motif generation for the generative self-supervised pre-training. The goal of the motif generation task is let GNNs learn the data distribution of graph motifs so that the pre-trained GNNs can easily generalize to downstream tasks with several finetuning steps on graphs from similar domains.
Given a molecule graph = EV and a GNN model fa. we first convert the molecule graph motif tree T(G) EV, Then we can model the likelihood GNN model as p(T(G); 0), representing how the motifs are labeled and connected. Generally, method aims to pretrain the GNN model via maximizing the likelihood of motif trees, i.e., 0r = argmax; (G'; 0). To model the likelihood of motif trees, special predictions heads for topology and motif label predictions are designed (as shown the following sections) and optimized along with fe. After pre-training, only the GNN model fa is transferred to downstream tasks.
this motif tree by the
We note that most existing works graph generation 17 23 follow the auto-regressive manner to factorize the probability objective, i.e., (T)) in this work. For each molecular graph, they decompose it into sequence of generation steps. Similarly in this paper, interleave the addition of a new motif, and the addition of bonds to connect the newly added motif to the existing partial motif tree. We denote permutation vector determine the motif ordering, in which i" denotes the motif ID of i-th position in permutation æ. Therefore, the probability (GT( 0) is equivalent to the expected likelihood over all possible permutations, i.c.,
(T(()): E,, (ps (V" [V="
where y denotes the permuted motif labels and Ed denotes the edges among motifs.
Our formalism permits variety of orders. For simplicity, assume that any node ordering has an equal probability and we also omit the subscript æ when illustrating the generative process for one permutation in the following sections. Given permutation order, the probability of generating motif tree (GG) can be decomposed as follows:
log pe (V,E)
Σ log po (V,, & Ven,
At each step i, motif attributes Ves and structures Eei of all motifs generated before to generate a new motif V; and its connection with existing motifs E,.
Equation describes the autoregressive generative process of motif trees. Then the question how to choose an efficient generation order and how to model the conditional probability log pa (V., & Vei, Exs). In the following sections, introduce two efficient generation orders,
5

---

DFS order
2
3)
Bedkbok
BFS order
2;
3)
4)
Ags
Figure 3: Illustration of the motif generation orders. The first row is the illustration of DFS order and the second row is the BFS order.
breadth-first (BFS) and depth-first (DFS) orders and show the corresponding auto-regressive genera- tive models.
To generate motif from scratch, we need to first choose the root of motif tree. In our experiments, we simply choose the motif with the first atom in the canonical order [34]. Then MGSSL generates motifs in DFS or BFS orders (see Figure B). the DFS order, for every visited motif, MGSSL first makes a topological prediction: whether this node has children to be generated. If new child motif node is generated, we predict its label and recurse this process. MGSSL backtracks when there no more children to generate. As for the BFS order, MGSSL generates motif nodes layer-wise. For motifs nodes in the k-th layer, MGSSL makes topological predictions and label predictions. If all the children nodes of motifs in the k-th layer are generated, MGSSL will move to the next layer. We also note that the motif node ordering BFS and DFS are not unique as order within sibling nodes ambiguous. In the experiments, we pre- vee-rriin in order and leave this issue and other potential generation orders for future exploration.
At each time step, motif node receives information from other generated motifs for making those predictions. The information is propagated through message vectors hij when motif trees are incrementally constructed. Formally, let &, be the set of message at time L. The model visits motif i time and v; denotes the embedding of motif which can be obtained by pooling the atom embeddings in motif i. The message hig is updated through previous messages:
baj GRU (IS,, (bk,d) (K.1)E..
(5)
where GRU is Gated Recurrent Unit 3 adapted for motif tree message passing:
8t,j = Σ
(6)
(K:)) KA
Zi.j = oW xi + U + b²)
(7)
Tk o(W",z 2, + Uh + b")
(8)
hig tanh Wz: + U Σ
Tk ha.e)
(9)
k=N
big = -1 (1-2;) 2 8ij Zij hay
(10)
Topology Prediction: When MGSSL visits motif it needs to make binary predictions whether it has children to be generated. We compute the probability via a one hidden layer network followed by sigmoid function taking messages and motif embeddings into consideration:
Pt =0
Td TWTT W Σ hes)
(11)
(K.))E
where d the dimension of the hidden layer.
Motif Label Prediction: When motif generate a child motif j. we predict the label of child with:
g softmax(U's (aammax('T(W"hh))))
(12)

---

where 43 is the distribution over the motif vocabulary A' and is the hidden layer dimension. Let P. {0, } and 9j be the ground truth topoloical and motif label values, the motif generation loss is the sum of cross-entropy losses of topological and motif label predictions:
m - Cropol Pc) Cproa(
(13)
In the optimization process, minimizing the above loss function corresponds to maximizing the log likelihood in equation 4 Note that in the training process, after topological and motif label prediction at each step. we replace them with their ground truth so that the MGSSL makes predictions based on correct histories.
3.3 Multi-level Self-supervised Pre-training
To capture the multi-scale information in molecules, MGSSL designed to be a hierarchical framework including Atom-level and Motif-level tasks (Figure I For Atom-level pre-training, we leverage attribute masking to let GNNs firstly learn the regularities of the node/edge attributes. In attribute masking, randomly sampled node and bond attributes (e.g., atom numbers, bond types) are replaced with special masked indicators. Then we apply GNNs to obtain the corresponding node/edge embeddings (edge embeddings be obtained end nodes). Finally, a fully connected layer on top of the embeddings predicts the node/edge attributes. The cross-entropy prediction losses are denoted as Cutom and Loond respectively.
combination of node embeddings of the edge's
To avoid catastrophic forgetting in sequential pre-training, we unify multi-level tasks and aim to minimize the hybrid loss in pre-training:
Lest = Aiotif + X2hatom Aahhonds
(14)
where A weights of losses. However, it is time-consuming to do grid search to determine the optimal weights. Here, we adapts the MGDA-UB algorithm (37] from multi-task learning to efficiently solve the optimization problem (Equation [4]. Since MGDA-UB calculates the weights X; by Frank- Wolfe algorithm [16] at each training step, we do not have to give weights explicitly. The pseudo codes of the training process is included in the Appendix.
4 Experimental Results
4.1 Experimental Settings
Datasets and Dataset Splittings. In this paper, we mainly focus on the molecular property prediction tasks, where large-scale unlabeled molecules are abundant whereas downstream labeled data is scarce. Specifically, we use 250k unlabeled molecules sampled from the ZINC15 database (38] for self-supervised pre-training tasks. As for the downstream finetune tasks, we consider 8 binary classification benchmark datasets contained in MoleculeNet 45 The detailed dataset statistics
are summarized in the Appendix. We use the open-source source package RDKit (2 to preprocess the
SMILES strings from various datasets. To mimic the real- sal-world use case, we split the downstream dataset by scaffold-split [14 31), which splits the molecules according to their structures. We apply 3 independent runs on random data splitting and report the means and standard deviations.
Baselines. We comprehensively evaluate the performance of MGSSL against five state-of-the-art self-supervised pre-training methods for GNNs:
Deep Graph Infomax 41 maximizes the mutual information between the representations of the whole graphs and the representations of its sampled subgraphs.
Attribute masking [4 masks node/edge features and let GNNs predict these attributes.
designs the pretraining task as discriminating ego-networks sampled from certain node ego-networks sampled from other nodes.
GCC [30]
Grover (32] predicts the contextual properties based on atom embeddings encode contex- tual information into node embeddings.
GPT-GNN 1. is a generative pretraining task which predicts masked edges and node attributes.
7

---

Table I: Test ROC-AUC (%) performance on molecular property prediction benchmarks using differ- ent pre-training strategies with GIN. The rightmost column averages the mean of test performance across the 8 datasets. The best result for each dataset are bolded.
SSL methods
clintox
sider
hiv
lux
buce
luscast
bkhp
Asg
pretrain
71.74:2.3
2.2
57.020-7
75421.5
74.32:0.5
70.0:22.5
63.321.5
65.5=115
67.0
Infomex
75.142..8
73.03.2
58.200.5
76.51.6
75.240.3
75.6
62.820.6
68.12
70.6
Altribute masking
74.741.9
7754:3.1
59620.7
77991.2
7722:0.4
78.35
63.3=0.8
65.620.9
71.8
14.14:.4
73242.6
98020.8
75.20.8
7662:0.5
75.02..
63.220.4
66.220.7
7504:2.5
74942.7
59.340.8
770-1.7
76.120.4
78.5=0.9
63.1-0.5
67.5=1.3
71,4
Gruver
7.8
7.911.0
60.71.0.5
77821.4
76.32.0.6
79.5=1.11
63422..6
68.0=
72.3
MOSSI (DFS)
78.12:.8
79.7422.2
60.520.7
79521
76420.4
79.20..8
63822.33
70.51
73.5
80.7122.1
7.811122
79.120.9
69.72.0.9
hous
65%
Spachs
Epoera
Eooets
Spects
clintox
(b) sider
(c) hiv
bace
Figure 4: Training and testing curves of different pre-trainnt pre-training strategies on GINs. Solid and dashed lines indicate training and testing curves respectively.
In experiments, we also consider GNN without pre- pre-trainn vre-training (direct supervised finetuning) and MGSSL with different generation orders (BFS and DFS).
Model Configuration. In the following experiments, we select five-layer Graph Isomorphism Networks (GINs) the backhone architecture, which is one of the state-of-the-an GNN methods. Mean pooling used as the Readout function for GIN. MGSSL, sum pooling used to get the embedding of graph motifs. Atom number and chirality tag are imput node features and bond type and direction are regarded as edge features. In the process of pre-training, GNNs are pre-trained for 100 epochs with Adam optimizer and learning rate 0.001. In the finetuning stage, we train for 100 epochs and report the testing score with the best cross- cross- validation performance. The hidden dimension is set to 300 and the batch size is set to 32 for pre-training and finetuning. The split for vain/validation/test sets is 80% : 10% 10%. All experiments are conducted on Tesla V100 GPUs.
4.2 Results and Analysis
Results Downstream Tasks, In Table show the testing performance downstream molecular prediction benchmarks using different self-supervised pre-training strategies with GIN. We have the following observations: 1) Generally, GNN models can benefit from various self- supervised pre-training tasks. The average prediction ROC-AUC of all the pre-trained models are better than GNN with direct finetuning. 2) MGSSL methods achieve the best performance out of 8 benchmarks, demonstrating the effectiveness of motif-based self-supervised pre-training. 3) We also show MGSSL with motif generation orders in Table T Both methods show significant performance improvements on downstream tasks and BFS has a small edge over DFS on average. This may be explained by the fact that MGSSL is required to generate motifs layer-wise in BFS orders, which helps GNNs learn more structural information of motifs. In the following experiments, we use BFS order as the default setting.
In Figure we further show the training and testing curves of MGSSL (BFS and DFS) and the selected baselines. Due to the page limits, select 4 benchmark datasets here. Beyond predictive performance improvement, our pre-trained GNNs have faster training and testing convergence than baseline methods. Since the pre- pre-training is one-time-effort, pre-trained GNNs can be used for various downstream tasks with minimal finetuning overhead.
pre-trained with MGSSL, the
Influence of the Base GNN. In Table we show that MGSSL is agnostic to the GNN architectures by trying five popular GNN models including GIN [46), GCN [19], RGCNEE), Gra and DAGNN 24). We report the average ROC-AUC and the relative gains on 8 benchmarks. We

---

Table 2: Compare pre-training gains with different GNN architectures, averaged ROC-AUC (%)
benchmark datasets
Model
GCN
GIN
RGCN
DAGNN
GraphSAGE
No pretrain
68.8
67.0
68.3
67.1
68.3
MGSSL (BFS)
72.7
73.7
73.0
72.3
73.4
Relative gain
5.7%
10.0%
6.9%
7.7 %
7.5%
observe that all these GNN architectures can benefit from motif-based pre-training tasks, Moreover, GIN achieves the largest relative gain and the best performance after pre-training.
Influence of Molecule Fragmentation. Here we show proper molecule frag- mentation methods vital for the motif-based pre-training. Given dif-
8:00
ferent molecule fragmentation methods, different motif vocabulary are gener- ated with varying sizes. Other than the fragmentation method introduced in this paper, we also try other fragmenta- tion schemes with different granularities [17 4. BRICS alone [4] tends to gener- ate motif's with large numbers of atoms. Due to the combinatorial explosion, its generated motif vocabulary has size over 100k while more than 90% motifs have frequencies less than 5. On the other hand, VAE [7] fragments molecules into rings and bonds and has a motif vocabulary size of less than 800. Our methods generate around 12k distinct motifs. By combining different fragmentation strategies, we are able to fragment molecules with intermediate granularities. In Figure 5] we show the influence of the size of motif vocabulary on 5 benchmark datasets. We can observe that the pre-trained models achieve the optimal performance with the motif vocabulary generated by our method. This may be explained by the following reasons: 1) When the motif segmentation is too coarse and the motif vocabulary is too large, the generated motif trees have fewer nodes. It is harder for GNNs to capture the structural information of motifs. Moreover, the generated motifs have low occurrence frequencies, which prevents GNNs from learning the general semantic information of motifs that be generalized downstream tasks. 2) When the motif segmentation too fine, many generated motifs are single atoms or bonds, which inhibits GNNs from learning higher level semantic information through motif generation tasks.
235
(a) MGSSL (BFS)
(b) MGSSL (DFS)
Figure 5: Influence of the size of motif vocabulary
Ablation Studies on Multi-level Self-
supervised Pre-training. We perform ablation studies to show the effectiveness the multi-level pre-training. In Table w/o atom-level denotes pre-training GNNs with Motif-level tasks only and the sequential pre-training denotes perform- ing the Motif-level tasks after the Atom-level tasks. As observed from Table 3 the multi-level pre- reeetraining has larger average ROC-AUC than the variants. We have the following interesting insights: 1) The Atom-level pre-training tasks enables GNNs to first capture the atom-level information, which benefit higher level, i.e., motif-level tasks. 2) Since our multi-level pre-training unifies multi-scale pre-training tasks and adaptively assigns the weights for hierarchical tasks, can achieve better performance than sequential pre-training.
Methods
Avg. ROC-AUC
w/o atom-level Sequential pre-training 73.4
73.0
Multi-level
73.7
Table 3: Ablation studies on multi-level self- supervised pre-training.
5 Conclusion and Future Works
In this paper, we proposed Motif-based Graph Self-supervised Learning (MGSSL), which pre-trains GNNs with a novel motif generation task. Through pre-training, MGSSL empowers GNNs to capture the rich semantic and structural information in graph motifs. First, a retrosynthesis-baved

---

algorithm with two additional rules are leveraged to fragment molecule graphs and derive semantic meaningful motifs. Second, a motif generative pre-training framework designed and two specific generation orders are considered (BFS and DFS). At each step, the pre-trained GNN is required to make topology and motif label predictions. Furthermore, we designed a multi-level pre-training to unify hierarchical self-supprrised tasks. Finally, we conducted extensive experiments to show that MGSSL overperforms all the state-of-the-ar baselines on various downstream benchmark tasks. Interesting future work includes 1) Designing more self-supervised pre-training tasks based on graph motifs. 2) Exploring motif-based pre-training in other domains other than molecules.
Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for valuable feedback. This research was supported by grants from the National Natural Science Foundation of China (Grants No. 61922073 and U20A20229) and 2021 Tencent Rhino-Bire Bird Research Elite Training Program.
References
[1] Jin Chen, Wynne Hsu, Mong Li Lee, and See-Kiong Ng. Nemofinder: Dissecting geddme-wide protein-protein interactions with meso-scale network motifs. In SIGKDD, pages 106-115, 2006.
[2] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607. PMLR, 2020.
[3] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling. arXiv preprini arXiv: 1412.3555 2014.
[4] Jörg Degen, Christof Wegscheid-Geree Gerlach,, Andrea Zaliani, and Matthias Rarey. On the art of compiling and using' drug- like chemical fragment spaces. ChemMedChem:: Chemistry Enabling Drug Discovery, 3(10), 1507-1507, 2008
[5] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ACL, 2018.
Ming-Wei
[6] Felix A Faber, Luke Hutchison, Bing Huang, Justin Gilmer, Samuel Schoenholz, George Dahl, Oriol Vinyals, Steven Kearnes, Patrick F Riley, and O Anatole Von Lilienfeld. Prediction eπrоrs of molecular machine learning models lower than hybrid dft error. Journal of chemical theory and computation, 13(11):5255-5244 2017.
[7] Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 188):128-135, 1999.
[8] Justin Gilmer, Samuel S Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In ICML, pages 1263-1272. PMLR, 2017.
[9] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In SIGKDD, pages 855-864, 2016.
[10] Hakim Hafidi, Mounir Ghogho, Philippe Ciblat, and Ananthram Swami. Graphel: Contrastive self-supervised learning of graph representations. arXiv preprint arXiv:2007.08025, 2020.
[11] William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. NeurIPS, 2017.
[12] Zhongkai Hao, Chengqiang Lu, Zhenya Huang, Hao Wang, Zheyuan Hu, Qi Liu, Enhong Chen, and Cheekong Lee. Asgn: An active semi-supervised graph neural network for molecular property prediction. KDD '20, 2020.
[13] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9729-9738, 9738, 2020.
[14] Weihua Hu* Bowen Liu Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In ICLR, 2020.
10

---

[15] Ziniu Hu, Yuxiao Dong, Kuansan Wang, Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks, In SIGKDD, pages 1857-1867, 2020.
[16] Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In ICML, pages 427-435 PMLR, 2013.
[17] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction variational autoencoder for molecular graph generation. In ICML, pages 2323-2332. PMLR, 2018.
[18) Nadav Kashtan, Shalev Itzkovitz, Ron Milo, and Uri Alon. Efficient sampling algorithm for estimating subgraph concentrations and detecting network motifs. Bioinformatics, 20(11):1746- 1758, 2004.
[19] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ICLR, 2017.
[20] Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. In ICLR, 2020.
[21] Walter Kohn and Lu Jeu Sham. Self-consistent equations including exchange and correlation effects. Physical review, 140(4A):A1133, 1965.
[22] Greg Landrum. Rdkit documentation. Release, 1(1-79):4, 2013.
[23] Renjie Liao, Yujia Li, Yang Song. Shenlong Wang, Charlie Nash, William I. Hamilton, David Duvenaud, Raquel Urtasun, and Richard $ Zemel. Efficient graph generation with graph recurrent attention networks. NeurlPS, 2019.
[24] Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In SIGKDD), pages 338-348, 2020.
[25] Chengqiang Lu, Qi Liu, Chao Wang, Zhenya Huang, Peize Lin, and Lixin He. Molecular prop- erty prediction: A multilevel quantum interactions modeling perspective. In AAAI, volume 33, pages 1052-1060. 2019.
[26] James L McDonagh, Arnaldo Silva, Mark Vincent, and Paul LA Popelier. Machine learning of dynamic electron correlation energies from topological atoms. Journal of chemical theory and computation, 14(1):216-2244 2017.
[27] Ron Milo, Shai Shen-Orr, Shalev Itzkovitz, Nadav Kashtan, Dmitri Chklovskii, and Uri Alon. Network motifs: simple building blocks of complex networks. Science, 89845574) 2002.
[28] Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. Graph representation learning via graphical mutual information maximization. In Proceedings of The Web Conference 2020, pages 259-270. 2020.
[29] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In SIGKDD, pages 701-710, 2014.
[30] Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gee: Graph contrastive coding for graph neural network pre-training. In SIGKDD, pages 1150-1160, 2020.
[31] Bharath Ramsundar, Peter Eastman, Patrick Walters, and Vijay Pande. Deep learning for the life sciences: applying deep learning to genomics, microscopy, drug discovery, and more. O' O'Reilly Media, Inc.". 2019.
[32] Yu Rong, Yatao Bian, Tingyang Xu. Weiyang Xie. Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. NeurIPS, 33, 2020.
[33] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European semantic web conference, pages 593-607. Springer, 2018.

---

[34] Nadine Schneider. Roger A Sayle, and Gregory A Landrum. Get your atoms in order an open- source implementation of novel and robust molecular canonicalization algorithm. Journal of chemical information and modeling, (5(10)2211122222 2015.
[35] Kristof T Schütt, Farhad Arbabzadah, Stefan Chmiela, Klaus R Müller, and Alexandre Tkatchenko. Quantum-chemical insights from deep tensor neural networks. Nature com- munications, 801):1-8, 2017.
[36] Kristof T Schütt, Pieter-Jan Kindermans, Huziel E Sauceda, Stefan Chmiela, Alexandre Tkatchenko, and Roburt Müller. Schnet: A continuous- convolutional neural network for modeling quantum interactions. NewrIPS, 2017.
[37] Ozan Sener and Vladlen Koltun. Multi-task learning multi-objective optimization. NeurIPS, 2018.
[38] Teague Sterling and John J Irwin. Zinc 15-ligand discovery for everyone. Journal of chemical information and modeling, 55(11):2324-2337, 2015.
[39] Fan-Yun Sun, Jordan Hoffman, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In ICLR, 2019.
[40] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 1rXiv:1710.10990 2017.
[41] Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR, 2019.
[42] Hao Wang, Defu Lian, Hanghang Tong, Qi Liu, Zhenya Huang, and Enhong Chen. Hypersorec: Exploiting hyperbolic user and item representations with multiple aspects for social-aware recommendation. ACM Trans. Inf. Syst. 2021.
[43] Hao Wang, Tong Xu, Qi Liu, Defu Lian, Enhong Chen, Dongfang Du, Han Wu, and Wen Su. Mene: An end-to-end framework for learning multiple conditional network representations of social network. KDD '19 2019.
[44] Junmei Wang and Tingjun Hou. Application of molecular dynamics simulations in molecular property prediction ii: diffusion coefficient. Journal of computational chemistry, 2011.
[45] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9():513-5330, 2018
[46] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.
[47] Chaochao Yan, Qianggang Ding, Peilin Zhao, Shuangjia Zheng, JINYU YANG, Yang Yu, and Junzhou Huang. Retroxpert: Decompose retrosynthesis prediction like a chemist. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, NeurlPS, volume 33, pages 11248 1258. Curran Associates, Inc., 2020.
[48] Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Fiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molec- ular representations for property prediction. Journal of chemical information and modeling, 59(8):3370-3355 2019.
[49] Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, NeurlPS, volume 31, 2018.
[50] Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. Graphrnn: Gener- ating realistic graphs with deep auto-regressive models. In ICML., pages 5708-5717, PMIR, 2018.
12

---

[51] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. NeurIPS, 33, 2020.
[52] Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. When does self-supervision help graph convolutional networks? In ICML. PMLR, 2020.
[53] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In AAAI, volume 32, 2018.
[54] Shichang Zhang, Ziniu Hu, Arjun Subramonian, and Yizhou Sun. Motif-driven contrastive learning of graph representations. arXiv preprint arXiv:2012.12533, 2020.
[55] Zaixi Zhang, Qi Liu, Zhenya Huang. Hao Wang, Chengqiang Lu, Chuanren Liu, and Enhong Chen. Graphmi: Extracting private graph data from graph neural networks. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21.
13


Reviews:
[{"review_id": "jYNFbAegnqD", "reviewer": "Reviewer_FWqv", "summary": " This paper proposes Motif-based Graph Self-supervised Learning, which pre-trains GNNs with a novel motif generation task. The authors first use a retrosynthesis-based algorithm with two additional rules to fragment molecule graphs and get meaningful motifs. Next they propose a motif generative pre-training framework, where in each step the pretrained GNN is required to make topology and motif label predictions.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The idea of pretraining GNN with motif prediction is interesting;\n2. The paper is well-written and easy to follow.\n\nWeaknesses:\n1. The authors mentioned that they use BRICS algorithm with two added rules to segment molecule graphs. However, my concern is that if there is a standard definition or criteria on the granularity of functional groups. For example, should C(=O)OH be seen as a whole or two functional groups: C=O and OH? Should C6H5-OH (phenol) be treated as a whole or two functional groups: benzene ring and -OH? Will this granularity affect the performance of the proposed model?\n2. BERT can use this pretraining strategy, i.e., predicting the masked word using its context, because adjacent words are seen as related. However, I was wondering if motifs in a molecule also have such property. Given all motifs but one in a molecule graph, can we really predict what the missing motif is? Is there any relationship between adjacent functional groups in a molecule? I suggest that the authors conduct an empirical study on this to show correlation of motifs.\n3. According to Table 4, the proposed model does not show significant improvement over baselines.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not mention limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"ethical_concerns": "n.a.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "MTyGV9Us-tB", "reviewer": "Reviewer_8tNd", "summary": "This paper presents a new self-supervised pre-training framework for Graph Neural Networks (GNNs) called Motif-based Graph Self-supervised Learning (MGSSL) to predict molecular properties. First, the method uses BRICs, a molecule fragmentation method, to create a Graph Motif vocabulary. Next, it performs pre-training using a model with three components -  1) a motif generation module that uses GRUs to recursively generate motifs given the previously generated motifs. 2)  a motif labeling module that uses dense layers followed by a softmax to predict the label of the motif from the vocabulary created in the previous step. 3) a GNN module that generates node and edge embeddings which are finally used to predict node and edge attributes of a molecule graph. MGSSL is trained as a multi-level model (with atom and motif layers) and it minimizes a hybrid loss consisting of Motif, Atomic, and Bond prediction losses, with weights adjusted by the Frank-Wolfe Algorithm. The results report improved performance for molecular property predictions over existing pre-training frameworks 7/8 datasets. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n++ This paper is a good addition to the recent literature focusing on accurately modeling domain-specific (chemistry) knowledge using deep learning methods. It introduces context-specific mechanisms (incorporating the molecular structure) to provide a performance improvement over established techniques. \n\n++ The different components and levels of the model are interesting and their roles make sense for the task the paper aims to solve. \n\n++ It presents results for 8 different datasets and 6 relevant baselines. It also provides results for different GNN architectures, the influence of the size of motif vocabulary on the performance, and ablation studies to justify the final MGSSL model.  \n\n++ The paper places the work with respect to existing methods, lists the contributions, and clearly defines motivations behind them.\n\n\nWeaknesses:\n\n-- The methodology has multiple and extensive steps, however, the paper does not discuss any limitations related to computational or time cost compared to existing methods. Is the cost/performance tradeoff reasonable? \n\n-- It is unclear whether hyperparameter tuning was performed for all the baseline models when reporting performance. If not, was the choice of default settings fair?\n\n-- In terms of applicability, the MGSSL model seems very specific to the molecular graph domain. Even though the paper presents a good use case for molecular property prediction, its contributions to the wider community are unclear. For example, the BRICS algorithm does not apply to other graph learning tasks. It would be helpful if the paper clarifies its contributions for the wider use of the method. \n \nMinor points:\n\n-An overview figure for the whole model with the components like GRUs and GNNs and connections to the method section would be very useful.\n  \n-Figure 4 presents 4 benchmark datasets due to page limits, the rest can be adjusted in the supplementary\n\nErrata: Line 220: “parent motif i generate → generates”\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Did not find any specific discussion on the limitations of the work and potential negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "GGX4SI2Qxjg", "reviewer": "Reviewer_QYy7", "summary": "The authors present a novel pre-training method (Motif-based Graph Self-Supervised Learning, or MGSSL) for graph neural networks specific to molecular property prediction. The pre-training involves constructing synthetic tasks to predict atom information, generation of the molecule, and motif identity. They demonstrate this pre-training improves GNN model performance on several MoleculeNet tasks across 5 different GNN models.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\nRating: High\n\n- [+] The use of BRICS to generate motifs for self-supervised pre-training is surprising and novel. The authors compare this approach of generating motifs to the fragments proposed by the JT-VAE and found that using the larger BRICS fragment library was helpful for pre-training the model.\n\n\n## Quality\nRating: Moderate to High\n- [+] The MGSSL approach is applied to 5 different graph neural network models, and compared against the performance of 5 self-supervised pre-training methods as well as GNNs without pretraining\n- [+] The authors perform ablation studies to demonstrate the effects of different parts of the pre-training, atom-level and sequential pre-training.\n- [+] The authors analyze the impact of the fragmentation method/motif vocabulary size on model improvement after pre-training.\n- In Table 1, can you provide an explanation for why the MGSSL pre-training approach does not not show as much improvement over the other SSL methods?\n\n## Clarity\nRating: Moderate to High\n- [+] The description of MGSSL pre-training was very clear; I was able to follow the approach taken by the authors.\n- For the baseline GNN models, did you train the GNN models for additional epochs? Or were they given the same number of epochs as the pre-trained models? Based on the graphs shown in Figure 4, it seems that the base GNN model (not pretrained) has not yet reached convergence on some of the tasks.\n- In table 1, please specify what the values right of the $\\pm$ are. I see from 4.1 you mention these are standard deviations of 3 runs of scaffold splitting (I assume?) but it might be helpful to mention this in the table.\n- Please include additional architectural information (embedding sizes, full atom/bond feature set) in the supplementary information, if not in the main text.\n\n\n## Significance\nRating: High\n\nMGSSL improves performance on the selected tasks for a variety of GNN models. Assuming the authors provide code for MGSSL, or the pre-trained model reported in this work, MGSSL could become a very useful starting point for any molecular property prediction project.\n\nI have some questions about the generalizability of MGSSL outside of drug-like chemical space, which I discuss in the Limitations section below. \n\n\n## Other notes/comments\nSection 2 on molecular property prediction, the authors state 'Recently, ... many works regard molecules as graphs and explore the graph convolutional network for property prediction.' \n\nWhile the use of graph convolutional networks are recent, I would argue that molecules have been regarded as graphs for a long time. Molecular fingerprints such as [ECFP](https://pubs.acs.org/doi/10.1021/ci100050t) and other [cheminformatics descriptors](http://www.talete.mi.it/products/dragon_molecular_descriptors.htm) contain on graph-based features of molecules, such as path length, topological indices. In my view, GNNs provides a way of learn which of these descriptors are most important, but it is still working with the same features.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Overall, the authors did a good job of comparing against baselines and analyzing the effect of different components of MGSSL (i.e. pre-training components, motif library creation).\n\nFor societal impact, have you considered the computational costs/savings associated with pretraining on MGSSL?\n\nAn additional limitation to consider: I would be interested to know how well MGSSL applies to areas of chemical space outside of drug-like molecules. The authors currently use the ZINC dataset, a collection of drug-like molecules, and the fragments are generated using BRICS, which generated fragments for retrosynthesis of drug-like molecules. The [ZINC dataset itself contains biases](https://www.pnas.org/content/116/24/11624); I wonder whether MGSSL also captures these biases.\n\nFor applications in other areas of chemical space, such as organic photovolataic molecules, would MGSSL have a lower rate of improvement, or even hurt the model's prediction?\n\nHow well do you think MGSSL would work for new areas of drug-like chemical space that haven't been explored before?\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}, {"review_id": "5mLX7k7f6TU", "reviewer": "Reviewer_VNdp", "summary": "*Summary\n\nThis paper presents a novel motif-level GNN pretraining called \"Motif-based Graph Self-Supervised Learning (MGSSL)\" and multi-level self-supervised pre-training. Most existing self-supervised pre-training frameworks for GNNs are only defined as node-level or graph-level tasks. But molecular graphs would follow a compositionality principle, and the complex structures are defined by their parts (motifs/fragments/scaffolds) and how to combine them.\n\nIt leverages the BRICS fragmentation algorithm [4] and further refinements to decompose a molecule into a \"motif tree\" representation that describes parts-level compositions of the given molecule in a tree form. Along with atom-level self-supervised tasks by atom-label masking, topology and label prediction for this 'motif tree' defines a nice self-supervised task at a motif level. \n\nExtensive experiments on multiple downstream tasks demonstrated that the proposed pretraining framework outperforms all other state-of-the-art baselines.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I liked the idea of this paper, and it provides a very interesting approach to GNN pretraining for molecular tasks. Related work is also quite relevant, and extensive experiments for multiple downstream tasks with scaffold-split are very nice to see. Also, it is also nice to see that several common GNN architectures (GCN, GIN, RGCN, DAGNN, GraphSAGE) are fed to the proposed pretraining framework, and observed performance gains for all cases. Comparisons to vocabularies of vanilla BRICS as well as JT-VAE fragments are very informative, too.\n\nThe following are some points unclear to me, and it will be nice to be in a clear description.\n\n1.\nGNN models are described just as function f_theta, but this symbol is not used explicitly in the main paper. Because of this, it is a bit unclear what is assumed as the input to GNNs, and how we can use the pretrained GNNs for downstream tasks such as property prediction.   \n\nWe have two graphs, input molecular graph G + the corresponding 'motif tree' T(G). Are GNNs pretrained as a function G -> vector? Or T(G) to vector, and when we use it in downstream tasks, T(G) needs to be constructed, keeping the same BRICS+alpha vocabulary to label T(G) accordingly??\n\nFurthermore, multi-level cases, do we need two GNNs pretrained for both atom-level and motif-level? Also, use both two GNNs for transfer to downstream tasks?\n\nIt is appreciated to clearly describe the input-output of GNNS using the defined symbol f_theta, how to use the output vector for pretraining, and how to use the pretrained GNNs for downstream tasks. \n\n2.\nSimilarly, line 196 says \"the question is ... how to model the conditional probability log p_theta(..) \", but this probability is never explicitly defined. So this implies minimizing the loss eq.(13) corresponds to maximizing the log likelihood (the conditional probability?). Any explicit and consistent descriptions are appreciated.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Any limitations are not clearly described. It is appreciated to summarize them somewhere in the manuscript?", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel motif-level GNN pretraining. While most existing self-supervised pre-training frameworks for GNNs are only defined as node-level or graph-level tasks, molecular graphs would follow a compositionality principle, and the complex structures are defined by their parts (motifs/fragments/scaffolds) and how to combine them. The authors show that by leveraging the BRICS fragmentation algorithm to decompose a molecule into a "motif tree" representation, the pretraining could become more effective. 
This paper triggered a lot of discussions between the authors and the reviewers, as well as among the reviewers themselves. One reviewer raised very serious concerns on the technical novelty (whether leveraging an existing algorithm for molecule decomposition could provide sufficient novelty to the pretraining task), and on the choice of GIN as the backbone model in the experiments (many stronger GNN models were developed recently). On the other hand, other reviewers believed that the introduction of motif to molecule graph learning was novel by its own, especially for this inter discipline, and the use of GIN makes sense and does not affect the main conclusion. Although a consensus was not successfully reached at the end, I tend to agree that this paper has its value in representation of molecule graph (and shed some lights on AI for molecular science), and there is no serious issue with the choice of GIN. Therefore, my recommendation is ACCEPT as a poster.

Decision: Accept (Poster)