Motif-based Graph Self-Supervised Learning for Molecular Property Prediction
Year: 2021

Abstract:
Predicting molecular properties with data-driven methods has drawn much attention in recent years. Particularly, Graph Neural Networks (GNNs) have demonstrated remarkable success in various molecular generation and prediction tasks. In cases where labeled data is scarce, GNNs can be pre-trained on unlabeled molecular data to first learn the general semantic and structural information before being finetuned for specific tasks. However, most existing self-supervised pre-training frameworks for GNNs only focus on node-level or graph-level tasks. These approaches cannot capture the rich information in subgraphs or graph motifs. For example, functional groups (frequently-occurred subgraphs in molecular graphs) often carry indicative information about the molecular properties. To bridge this gap, we propose Motifbased Graph Self-supervised Learning (MGSSL) by introducing a novel selfsupervised motif generation framework for GNNs. First, for motif extraction from molecular graphs, we design a molecule fragmentation method that leverages a retrosynthesis-based algorithm BRICS and additional rules for controlling the size of motif vocabulary. Second, we design a general motif-based generative pre-training framework in which GNNs are asked to make topological and label predictions. This generative framework can be implemented in two different ways, i.e., breadth-first or depth-first. Finally, to take the multi-scale information in molecular graphs into consideration, we introduce a multi-level self-supervised pre-training. Extensive experiments on various downstream benchmark tasks show that our methods outperform all state-of-the-art baselines.

heading: 1 Introduction
text: The past decade has witnessed the remarkable success of deep learning in natural language processing (NLP) [5], computer vision (CV) [13], and graph analysis [19]. Inspired by these developments, researchers in the chemistry domain have also tried to exploit deep learning methods for moleculebased tasks such as retrosynthesis [47] and drug discovery [8]. To preserve the internal structural information, molecules can be naturally modeled as graphs where nodes represent atoms and edges denote the chemical bonds. Recently, some works applied Graph Neural Network (GNN) and some of its variants for molecular property prediction and obtained promising results [8, 25].
Though GNNs have achieved remarkable accuracy on molecular property prediction, they are usually data-hungry, i.e. a large amount of labeled data (i.e., molecules with known property data) is required for training [8, 12]. However, labeled molecules only occupy an extremely small portion of the
∗Qi Liu and Chee-Kong Lee are corresponding authors.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
enormous chemical space since they can only be obtained from wet-lab experiments or quantum chemistry calculations, which are time-consuming and expensive. Moreover, directly training GNNs on small labeled molecule datasets in a supervised fashion is prone to over-fitting and the trained GNNs can hardly generalize to out-of-distribution data.
Similar issues have also been encountered in natural language processing and computer vision. Recent advances in NLP and CV address them by self-supervised learning (SSL) where a model is first pre-trained on a large unlabeled dataset and then transferred to downstream tasks with limited labels [13, 2, 5]. For example, the pre-trained BERT language model [5] is able to learn expressive contextualized word representations through reconstructing the input text-next sentence and masked language predictions so that it can significantly improve the performance of downstream tasks. Inspired by these developments, various self-supervised pre-training methods of GNNs have been proposed [14, 15, 52, 51, 32, 10, 30]. Based on how the self-supervised tasks are constructed, these methods can be classified into two categories, contrastive methods and predictive methods. Contrastive methods force views from the same graph (e.g., sampling nodes and edges from graphs) to become closer and push views from different graphs apart. On the other hand, the predictive methods construct prediction tasks by exploiting the intrinsic properties of data. For example, Hu et.al. [14] designed node-level pre-training tasks such as predicting the context of atoms and the attributes of masked atoms and bonds. [15] introduced an attributed graph reconstruction task where the generative model predicts the node attributes and edges to be generated at each step.
However, we argue that existing self-supervised learning tasks on GNNs are sub-optimal since most of them fail to exploit the rich semantic information from graph motifs. Graph motifs can be defined as significant subgraph patterns that frequently occur [27]. Motifs usually contain semantic meanings and are indicative of the characteristics of the whole graph. For example, the hydroxide (–OH) functional group in small molecules typically implies higher water solubility. Therefore, it is vital to design motif-level self-supervised learning tasks which can benefit downstream tasks such as molecular property prediction.
Designing motif-level self-supervised learning tasks brings unique challenges. First, existing motif mining techniques could not be directly utilized to derive expressive motifs for molecular graphs because they only rely on the discrete count of subgraph structures and overlook the chemical validity [1, 18]. Second, most graph generation techniques generate graphs node-by-node [23, 50], which are not suitable for our task of motif generation. Finally, how to unify multi-level self-supervised pre-training tasks harmoniously brings a new challenge. One naive solution to do pre-training tasks sequentially may lead to catastrophic forgetting similar to continual learning [7].
To tackle the aforementioned challenges, we propose Motif-based Graph Self-Supervised Learning (MGSSL) and Multi-level self-supervised pre-training in this paper. Firstly, MGSSL introduces a novel motif generation task that empowers GNNs to capture the rich structural and semantic information from graph motifs. To derive semantically meaningful motifs and construct motif trees for molecular graphs, we leverage the BRICS algorithm [4] which is based on retrosynthesis from the chemistry domain. Two additional fragmentation rules are further introduced to reduce the redundancy of motif vocabulary. Secondly, a general motif-based generative pre-training framework is designed to generate molecular graphs motif-by-motif. The pre-trained model is required to make topology and attribute predictions at each step and two specific generation orders are implemented (breadth-first and depth-first). Furthermore, to take the multi-scale regularities of molecules into consideration, we introduce Multi-level self-supervised pre-training for GNNs where the weights of different SSL tasks are adaptively adjusted by the Frank-Wolfe algorithm [16]. Finally, by pre-training GNNs on the ZINC dataset with our methods, the pre-trained GNNs outperforms all the state-of-the-art baselines on various downstream benchmark tasks, demonstrating the effectiveness of our design. The implementation is publicly available at https://github.com/zaixizhang/MGSSL.

heading: 2 Preliminaries and Related Work

heading: 2.1 Molecular Property Prediction
text: Prediction of molecular properties is a central research topic in physics, chemistry, and materials science [44]. Among the traditional methods, density functional theory (DFT) is the most popular one and plays a vital role in advancing the field [21]. However, DFT is very time-consuming and its complexity could be approximated as O(N3), where N denotes the number of particles. To find
more efficient solutions for molecular property prediction, various machine learning methods have been leveraged such as kernel ridge regression, random forest, and convolutional neural networks [6, 26, 36]. To fully consider the internal spatial and distance information of atoms in molecules, many works [8, 35, 48, 25] regard molecules as graphs and explore the graph convolutional network for property prediction. To better capture the interactions among atoms, [8] proposes a message passing framework and [20, 48] extend this framework to model bond interactions. [25] builds a hierarchical GNN to capture multilevel interactions. Furthermore, [32] integrates GNNs into the Transformer-style architecture to deliver a more expressive model.

heading: 2.2 Preliminaries of Graph Neural Networks
text: Recent years have witnessed the success of Graph Neural Networks (GNNs) for modeling graph data [11, 19, 40, 46, 43, 42, 55]. Let G = (V,E) denotes a graph with node attributes Xv for v ∈ V and edge attributes euv for (u, v) ∈ E. GNNs leverage the graph connectivity as well as node and edge features to learn a representation vector (i.e., embedding) hv for each node v ∈ G and a vector hG for the entire graph G. Generally, GNNs follows a message passing paradigm, in which representation of node v is iteratively updated by aggregating the representations of v’s neighboring nodes and edges. Specifically, there are two basic operators for GNNs: AGGREGATE(·) and COMBINE(·). AGGREGATE(·) extracts the neighboring information of node v; COMBINE(·) serves as the aggregation function of the neighborhood information. After k iterations of aggregation, v’s representation captures the structural information within its k-hop network neighborhood. Formally, the k-th layer of a GNN is:
h(k)v = COMBINE (k) ( h(k−1)v ,AGGREGATE (k) ({ (h(k−1)v , h (k−1) u , euv) : u ∈ N (v) })) , (1)
where h(k)v denotes the representation of node v at the k-th layer, euv is the feature vector of edge between u and v and andN (v) represents the neighborhood set of node v. h(0)v is initialized with Xv . Furthermore, to obtain the representation of the entire graph hG, READOUT functions are designed to pool node representations at the final iteration K,
hG = READOUT ({ h(K)v | v ∈ G }) . (2)
READOUT is a permutation-invariant function, such as averaging, sum, max or more sophisticated graph-level pooling functions [49, 53].

heading: 2.3 Self-supervised Learning of Graphs
text: Graph Self-supervised learning aims to learn the intermediate representations of unlabeled graph data that are useful for unknown downstream tasks. [14, 15, 52, 51, 32, 10, 30, 39, 28, 41]. Traditional graph embedding methods [9, 29] define different kinds of graph proximities, i.e., the vertex proximity, as the self-supervised objective to learn vertex representations. Furthermore, [39, 28, 41] proposed to use the mutual information maximization as the optimization objective for GNNs. Recently, more self-supervised tasks for GNNs have been proposed [14, 15, 52, 51, 32, 10, 54, 30]. Based on how the self-supervised tasks are constructed, these models can be classified into two categories, namely contrastive models and predictive models. Contrastive models try to generate informative and diverse views from data instances and perform node-to-context [14], node-to-graph [39] or motif-to-graph [54] contrastive learning. On the other hand, predictive models are trained in a supervised fashion, where the labels are generated based on certain properties of the input graph data, i.e., node attributes [32], or by selecting certain parts of the graph [15, 32].
However, few works try to leverage the information of graph motifs for graph self-supervised learning. Grover [32] use traditional softwares to extract motifs and treat them as classification labels. MICROGraph [54] exploits graph motifs for motif-graph contrastive learning. Unfortunately, both methods fail to consider the topology information of motifs.

heading: 3 Motif-based Graph Self-supervised Learning
text: In this section, we introduce the framework of motif-based graph self-supervised learning (Figure 1). Generally, the framework consists of three parts: chemistry-inspired molecule fragmentation, motif generation and multi-level self-supervised pre-training for GNNs.

heading: 3.1 Chemistry-inspired Molecule Fragmentation
text: Given a dataset of molecules, the first step of our method is to decompose molecules into several fragments/motifs. Based on these fragments, a molecule graph can be converted into a motif tree structure where each node represents a motif and the edges denotes the relative spatial relationships among motifs. We choose to represent the structure of motifs as a tree because it facilitates the motif generation task. Formally, given a molecule graph G = (V,E), a motif tree T (G) = (V, E ,X ) is a connected labeled tree whose node set is V = {M1, ...,Mn} and edge set is E . X refers to the induced motif vocabulary. Each motif Mi = (Vi, Ei) is a subgraph of G. There are many ways to fragment a given graph while the designed molecule fragmentation method should achieve the following goals: 1) In a motif tree T (G), the union of all motifs Mi should equals G. Formally,⋃ i Vi = V and ⋃ iEi ⋃ E = E. 2) In a motif tree T (G), the motifs should have no intersections. That is Mi ∩Mj = ∅. 3) The induced motifs should capture semantic meanings, e.g., similar to meaningful functional groups in the chemistry domain. 4) The occurrence of motifs in the dataset should be frequent enough so that the pre-trained GNNs can learn semantic information of motifs that can be generalized to downstream tasks. After the molecule fragmentation, the motif vocabulary of the molecule dataset should have a moderate size.
In Figure 2, we show the overview of molecule fragmentation. Generally, there are three procedures, BRICS fragmentation, further decomposition, and motif tree construction. A motif vocabulary can be built via preprocessing the whole molecule dataset following the molecule fragmentation procedures.
To fragment molecule graphs and construct motif trees, we firstly use the Breaking of Retrosynthetically Interesting Chemical Substructures (BRICS) algorithm [4] that leverages the domain knowledge from chemistry. BRICS defines 16 rules and breaks strategic bonds in a molecule that match a set of chemical reactions. "Dummy" atoms are attached to each end of the cleavage sites, marking the location where two fragments can join together. BRICS cleavage rules are designed to retain molecular components with valuable structural and functional content, e.g. aromatic rings.
However, we find BRICS alone cannot generate desired motifs for molecule graphs. This is because BRICS only breaks bonds based on a limited set of chemical reactions and tends to generate several large fragments for a molecule. Moreover, due to the combination explosion of graph structure, we find BRICS produces many motifs that are variations of the same underlying structure (e.g., Furan ring with different combinations of Halogen atoms). The motif vocabulary is large (more than 100k unique fragments) while most of these motifs appear less than 5 times in the whole dataset.
To tackle the problems above, we introduce a post-processing procedure to BRICS. To alleviate the combination explosion, we define two rules operating on the output fragments of BRICS: (1) Break the bond where one end atom is in a ring while the other end not. (2) Select non-ring atoms with three or more neighboring atoms as new motifs and break the neighboring bonds. The first rule reduces the number of ring variants and the second rule break the side chains. Experiments show that these rules effectively reduce the size of motif vocabulary and improve the occurrence frequency of motifs.

heading: 3.2 Motif Generation
text: Here, we present the framework of motif generation for the generative self-supervised pre-training. The goal of the motif generation task is to let GNNs learn the data distribution of graph motifs so that the pre-trained GNNs can easily generalize to downstream tasks with several finetuning steps on graphs from similar domains.
Given a molecule graph G = (V,E) and a GNN model fθ, we first convert the molecule graph to a motif tree T (G) = (V, E ,X ). Then we can model the likelihood over this motif tree by the GNN model as p(T (G); θ), representing how the motifs are labeled and connected. Generally, our method aims to pretrain the GNN model fθ via maximizing the likelihood of motif trees, i.e., θ∗ = argmaxθ p(T (G); θ). To model the likelihood of motif trees, special predictions heads for topology and motif label predictions are designed (as shown in the following sections) and optimized along with fθ. After pre-training, only the GNN model fθ is transferred to downstream tasks.
We note that most existing works on graph generation [17, 23] follow the auto-regressive manner to factorize the probability objective, i.e., p(T (G); θ) in this work. For each molecular graph, they decompose it into a sequence of generation steps. Similarly in this paper, we interleave the addition of a new motif, and the addition of bonds to connect the newly added motif to the existing partial motif tree. We denote a permutation vector π to determine the motif ordering, in which iπ denotes the motif ID of i-th position in permutation π. Therefore, the probability p(T (G); θ) is equivalent to the expected likelihood over all possible permutations, i.e.,
p(T (G); θ) = Eπ [pθ(Vπ, Eπ)] , (3)
where Vπ denotes the permuted motif labels and Eπ denotes the edges among motifs. Our formalism permits a variety of orders. For simplicity, we assume that any node ordering π has an equal probability and we also omit the subscript π when illustrating the generative process for one permutation in the following sections. Given a permutation order, the probability of generating motif tree T (G) can be decomposed as follows:
log pθ(V, E) = |V|∑ i=1 log pθ(Vi, Ei | V<i, E<i). (4)
At each step i, we use motif attributes V<i and structures E<i of all motifs generated before i to generate a new motif Vi and its connection with existing motifs Ei. Equation 4 describes the autoregressive generative process of motif trees. Then the question is how to choose an efficient generation order and how to model the conditional probability log pθ(Vi, Ei | V<i, E<i). In the following sections, we introduce two efficient generation orders,
breadth-first (BFS) and depth-first (DFS) orders and show the corresponding auto-regressive generative models.
To generate a motif tree from scratch, we need to first choose the root of motif tree. In our experiments, we simply choose the motif with the first atom in the canonical order [34]. Then MGSSL generates motifs in DFS or BFS orders (see Figure 3). In the DFS order, for every visited motif, MGSSL first makes a topological prediction: whether this node has children to be generated. If a new child motif node is generated, we predict its label and recurse this process. MGSSL backtracks when there is no more children to generate. As for the BFS order, MGSSL generates motif nodes layer-wise. For motifs nodes in the k-th layer, MGSSL makes topological predictions and label predictions. If all the children nodes of motifs in the k-th layer are generated, MGSSL will move to the next layer. We also note that the motif node ordering in BFS and DFS are not unique as the order within sibling nodes is ambiguous. In the experiments, we pre-train in one order and leave this issue and other potential generation orders for future exploration.
At each time step, a motif node receives information from other generated motifs for making those predictions. The information is propagated through message vectors hij when motif trees are incrementally constructed. Formally, let Êt be the set of message at time t. The model visits motif i at time t and xi denotes the embedding of motif i, which can be obtained by pooling the atom embeddings in motif i. The message hi,j is updated through previous messages:
hi,j = GRU ( xi, {hk,i}(k,i)∈Êt,k 6=j ) , (5)
where GRU is Gated Recurrent Unit [3] adapted for motif tree message passing: si,j = ∑
(k,i)∈Êt,k 6=j
hk,i (6)
zi,j = σ(W zxi +U zsi,j + b z) (7) rk,i = σ(W rxi +U rhk,i + b r) (8)
h̃i,j = tanh(Wxi + U ∑
k=N (i)\j
rk,i hk,i) (9)
hi,j = (1− zij) sij + zij h̃i,j . (10) Topology Prediction: When MGSSL visits motif i, it needs to make binary predictions on whether it has children to be generated. We compute the probability via a one hidden layer network followed by a sigmoid function taking messages and motif embeddings into consideration:
pt = σ Ud · τ(W d1 xi +W d2 ∑ (k,i)∈Êt hk,i)  , (11) where d is the dimension of the hidden layer.
Motif Label Prediction: When motif i generate a child motif j, we predict the label of child j with:
qj = softmax(U lτ(W lhij)), (12)
where qj is the distribution over the motif vocabulary X and l is the hidden layer dimension. Let p̂t ∈ {0, 1} and q̂j be the ground truth topological and motif label values, the motif generation loss is the sum of cross-entropy losses of topological and motif label predictions:
Lmotif = ∑ t Ltopo(pt, p̂t) + ∑ j Lpred(qj , q̂j). (13)
In the optimization process, minimizing the above loss function corresponds to maximizing the log likelihood in equation 4. Note that in the training process, after topological and motif label prediction at each step, we replace them with their ground truth so that the MGSSL makes predictions based on correct histories.

heading: 3.3 Multi-level Self-supervised Pre-training
text: To capture the multi-scale information in molecules, MGSSL is designed to be a hierarchical framework including Atom-level and Motif-level tasks (Figure 1). For Atom-level pre-training, we leverage attribute masking to let GNNs firstly learn the regularities of the node/edge attributes. In attribute masking, randomly sampled node and bond attributes (e.g., atom numbers, bond types) are replaced with special masked indicators. Then we apply GNNs to obtain the corresponding node/edge embeddings (edge embeddings can be obtained as a combination of node embeddings of the edge’s end nodes). Finally, a fully connected layer on top of the embeddings predicts the node/edge attributes. The cross-entropy prediction losses are denoted as Latom and Lbond respectively. To avoid catastrophic forgetting in sequential pre-training, we unify multi-level tasks and aim to minimize the hybrid loss in pre-training:
Lssl = λ1Lmotif + λ2Latom + λ3Lbond, (14)
where λi are weights of losses. However, it is time-consuming to do a grid search to determine the optimal weights. Here, we adapts the MGDA-UB algorithm [37] from multi-task learning to efficiently solve the optimization problem (Equation 14). Since MGDA-UB calculates the weights λi by Frank-Wolfe algorithm [16] at each training step, we do not have to give weights explicitly. The pseudo codes of the training process is included in the Appendix.

heading: 4 Experimental Results

heading: 4.1 Experimental Settings
text: Datasets and Dataset Splittings. In this paper, we mainly focus on the molecular property prediction tasks, where large-scale unlabeled molecules are abundant whereas downstream labeled data is scarce. Specifically, we use 250k unlabeled molecules sampled from the ZINC15 database [38] for self-supervised pre-training tasks. As for the downstream finetune tasks, we consider 8 binary classification benchmark datasets contained in MoleculeNet [45]. The detailed dataset statistics are summarized in the Appendix. We use the open-source package RDKit [22] to preprocess the SMILES strings from various datasets. To mimic the real-world use case, we split the downstream dataset by scaffold-split [14, 31], which splits the molecules according to their structures. We apply 3 independent runs on random data splitting and report the means and standard deviations.
Baselines. We comprehensively evaluate the performance of MGSSL against five state-of-the-art self-supervised pre-training methods for GNNs:
• Deep Graph Infomax [41] maximizes the mutual information between the representations of the whole graphs and the representations of its sampled subgraphs.
• Attribute masking [14] masks node/edge features and let GNNs predict these attributes. • GCC [30] designs the pretraining task as discriminating ego-networks sampled from a
certain node ego-networks sampled from other nodes. • Grover [32] predicts the contextual properties based on atom embeddings to encode contex-
tual information into node embeddings. • GPT-GNN [15] is a generative pretraining task which predicts masked edges and node
attributes.
In experiments, we also consider GNN without pre-training (direct supervised finetuning) and MGSSL with different generation orders (BFS and DFS).
Model Configuration. In the following experiments, we select a five-layer Graph Isomorphism Networks (GINs) as the backbone architecture, which is one of the state-of-the-art GNN methods. Mean pooling is used as the Readout function for GIN. In MGSSL, sum pooling is used to get the embedding of graph motifs. Atom number and chirality tag are input as node features and bond type and direction are regarded as edge features. In the process of pre-training, GNNs are pre-trained for 100 epochs with Adam optimizer and learning rate 0.001. In the finetuning stage, we train for 100 epochs and report the testing score with the best cross-validation performance. The hidden dimension is set to 300 and the batch size is set to 32 for pre-training and finetuning. The split for train/validation/test sets is 80% : 10% : 10%. All experiments are conducted on Tesla V100 GPUs.

heading: 4.2 Results and Analysis
text: Results on Downstream Tasks. In Table 1, we show the testing performance on downstream molecular prediction benchmarks using different self-supervised pre-training strategies with GIN. We have the following observations: 1) Generally, GNN models can benefit from various selfsupervised pre-training tasks. The average prediction ROC-AUC of all the pre-trained models are better than GNN with direct finetuning. 2) MGSSL methods achieve the best performance on 7 out of 8 benchmarks, demonstrating the effectiveness of motif-based self-supervised pre-training. 3) We also show MGSSL with two motif generation orders in Table 1. Both methods show significant performance improvements on downstream tasks and BFS has a small edge over DFS on average. This may be explained by the fact that MGSSL is required to generate motifs layer-wise in BFS orders, which helps GNNs learn more structural information of motifs. In the following experiments, we use BFS order as the default setting.
In Figure 4, we further show the training and testing curves of MGSSL (BFS and DFS) and the selected baselines. Due to the page limits, we select 4 benchmark datasets here. Beyond predictive performance improvement, our pre-trained GNNs have faster training and testing convergence than baseline methods. Since the pre-training is a one-time-effort, once pre-trained with MGSSL, the pre-trained GNNs can be used for various downstream tasks with minimal finetuning overhead.
Influence of the Base GNN. In Table 2 we show that MGSSL is agnostic to the GNN architectures by trying five popular GNN models including GIN [46], GCN [19], RGCN[33], GraphSAGE[11] and DAGNN [24]. We report the average ROC-AUC and the relative gains on 8 benchmarks. We
observe that all these GNN architectures can benefit from motif-based pre-training tasks. Moreover, GIN achieves the largest relative gain and the best performance after pre-training.
Influence of Molecule Fragmentation. Here we show proper molecule fragmentation methods are vital for the motif-based pre-training. Given different molecule fragmentation methods, different motif vocabulary are generated with varying sizes. Other than the fragmentation method introduced in this paper, we also try other fragmentation schemes with different granularities [17, 4]. BRICS alone [4] tends to generate motifs with large numbers of atoms. Due to the combinatorial explosion, its
generated motif vocabulary has a size over 100k while more than 90% motifs have frequencies less than 5. On the other hand, JT-VAE [17] fragments molecules into rings and bonds and has a motif vocabulary size of less than 800. Our methods generate around 12k distinct motifs. By combining different fragmentation strategies, we are able to fragment molecules with intermediate granularities. In Figure 5, we show the influence of the size of motif vocabulary on 5 benchmark datasets. We can observe that the pre-trained models achieve the optimal performance with the motif vocabulary generated by our method. This may be explained by the following reasons: 1) When the motif segmentation is too coarse and the motif vocabulary is too large, the generated motif trees have fewer nodes. It is harder for GNNs to capture the structural information of motifs. Moreover, the generated motifs have low occurrence frequencies, which prevents GNNs from learning the general semantic information of motifs that can be generalized to downstream tasks. 2) When the motif segmentation is too fine, many generated motifs are single atoms or bonds, which inhibits GNNs from learning higher level semantic information through motif generation tasks.
Ablation Studies on Multi-level Selfsupervised Pre-training. We perform ablation studies to show the effectiveness of the multi-level pre-training. In Table 3, w/o atom-level denotes pre-training GNNs with Motif-level tasks only and the sequential pre-training denotes performing the Motif-level tasks after the Atom-level tasks. As observed from Table 3, the multi-level pre-training has larger average ROC-AUC than the two variants. We can have the following
interesting insights: 1) The Atom-level pre-training tasks enables GNNs to first capture the atom-level information, which can benefit higher level, i.e., motif-level tasks. 2) Since our multi-level pre-training unifies multi-scale pre-training tasks and adaptively assigns the weights for hierarchical tasks, it can achieve better performance than sequential pre-training.

heading: 5 Conclusion and Future Works
text: In this paper, we proposed Motif-based Graph Self-supervised Learning (MGSSL), which pre-trains GNNs with a novel motif generation task. Through pre-training, MGSSL empowers GNNs to capture the rich semantic and structural information in graph motifs. First, a retrosynthesis-based
algorithm with two additional rules are leveraged to fragment molecule graphs and derive semantic meaningful motifs. Second, a motif generative pre-training framework is designed and two specific generation orders are considered (BFS and DFS). At each step, the pre-trained GNN is required to make topology and motif label predictions. Furthermore, we designed a multi-level pre-training to unify hierarchical self-supervised tasks. Finally, we conducted extensive experiments to show that MGSSL overperforms all the state-of-the-art baselines on various downstream benchmark tasks. Interesting future work includes 1) Designing more self-supervised pre-training tasks based on graph motifs. 2) Exploring motif-based pre-training in other domains other than molecules.
Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for valuable feedback. This research was supported by grants from the National Natural Science Foundation of China (Grants No. 61922073 and U20A20229) and 2021 Tencent Rhino-Bird Research Elite Training Program.

Review summaries:
The authors present a novel pre-training method (Motif-based Graph Self-Supervised Learning, or MGSSL) for graph neural networks specific to molecular property prediction. The pre-training involves constructing synthetic tasks to predict atom information, generation of the molecule, and motif identity. They demonstrate this pre-training improves GNN model performance on several MoleculeNet tasks across 5 different GNN models.

 This paper proposes Motif-based Graph Self-supervised Learning, which pre-trains GNNs with a novel motif generation task. The authors first use a retrosynthesis-based algorithm with two additional rules to fragment molecule graphs and get meaningful motifs. Next they propose a motif generative pre-training framework, where in each step the pretrained GNN is required to make topology and motif label predictions.

This paper presents a new self-supervised pre-training framework for Graph Neural Networks (GNNs) called Motif-based Graph Self-supervised Learning (MGSSL) to predict molecular properties. First, the method uses BRICs, a molecule fragmentation method, to create a Graph Motif vocabulary. Next, it performs pre-training using a model with three components -  1) a motif generation module that uses GRUs to recursively generate motifs given the previously generated motifs. 2)  a motif labeling module that uses dense layers followed by a softmax to predict the label of the motif from the vocabulary created in the previous step. 3) a GNN module that generates node and edge embeddings which are finally used to predict node and edge attributes of a molecule graph. MGSSL is trained as a multi-level model (with atom and motif layers) and it minimizes a hybrid loss consisting of Motif, Atomic, and Bond prediction losses, with weights adjusted by the Frank-Wolfe Algorithm. The results report improved performance for molecular property predictions over existing pre-training frameworks 7/8 datasets.  

*Summary  This paper presents a novel motif-level GNN pretraining called "Motif-based Graph Self-Supervised Learning (MGSSL)" and multi-level self-supervised pre-training. Most existing self-supervised pre-training frameworks for GNNs are only defined as node-level or graph-level tasks. But molecular graphs would follow a compositionality principle, and the complex structures are defined by their parts (motifs/fragments/scaffolds) and how to combine them.  It leverages the BRICS fragmentation algorithm [4] and further refinements to decompose a molecule into a "motif tree" representation that describes parts-level compositions of the given molecule in a tree form. Along with atom-level self-supervised tasks by atom-label masking, topology and label prediction for this 'motif tree' defines a nice self-supervised task at a motif level.   Extensive experiments on multiple downstream tasks demonstrated that the proposed pretraining framework outperforms all other state-of-the-art baselines.