Molecular Graph Representation Learning via Heterogeneous Motif Graph Construction
Authors: Zhaoning Yu
Hongyang Gao
Venue: ICLR 2022 Submitted (2022)
Keywords: Molecular Graph Representation
Graph Neural Networks
Heterogeneous

Abstract: We consider feature representation learning of molecular graphs. Graph Neural Networks have been widely used in feature representation learning of molecular graphs. However, most proposed methods focus on the individual molecular graph while neglecting their connections, such as motif-level relationships. We propose a novel molecular graph representation learning method by constructing a Heterogeneous Motif graph (HM-graph) to address this issue. In particular, we build an HM-graph that contains motif nodes and molecular nodes. Each motif node corresponds to a motif extracted from molecules. Then, we propose a Heterogeneous Motif Graph Neural Network (HM-GNN) to learn feature representations for each node in the HM-graph. Our HM-graph also enables effective multi-task learning, especially for small molecular datasets. To address the potential efficiency issue, we propose an edge sampler, which significantly reduces computational resources usage. The experimental results show that our model consistently outperforms previous state-of-the-art models. Under multi-task settings, the promising performances of our methods on combined datasets shed light on a new learning paradigm for small molecular datasets. Finally, we show that our model achieves similar performances with significantly less computational resources by using our edge sampler.

Under review as a conference paper at ICLR 2022
MOLECULAR GRAPH REPRESENTATION LEARNING VIA HETEROGENEOUS MOTIF GRAPH CONSTRUCTION
Anonymous authors Paper under double-blind review
ABSTRACT
We consider feature representation learning problem of molecular graphs. Graph Neural Networks have been widely used in feature representation learning of molecular graphs. However, most existing methods deal with molecular graphs individually while neglecting their connections, such as motif-level relationships We propose novel molecular graph representation learning method by construct- ing a heterogeneous motif graph to address this issue. In particular, we build a heterogeneous motif graph that contains motif nodes and molecular nodes. Each motif node corresponds to motif extracted from molecules. Then, propose a Heterogeneous Motif Graph Neural Network (HM-GNN) to learn feature rep- resentations for each node in the heterogeneous motif graph. Our heterogeneous motif graph also enables effective multi-task learning, especially for small molec- ular datasets. To address the potential efficiency issue, we propose use an edge sampler, which can significantly reduce computational resources usage. The ex- perimental results show that our model consistently outperforms previous state- of-the art models. Under multi-task settings, the promising performances of our methods on combined datasets shed light on a new learning paradigm for small molecular datasets. Finally, we show that our model achieves similar perfor- mances with significantly less computational resources by using our edge sampler.
INTRODUCTION
Graph neural networks (GNNs) have been proved to effectively solve various challenging tasks in graph embedding fields, such as node classification (Kipf & Welling, 2016), graph classification (Xu et al. 2018). and link prediction (Schlichtkrull et al.) (018). which have been extensively applied social networks, molecular properties prediction, natural language processing, and other fields. Compared with hand-crafted features in the molecular properties prediction field, GNNs map molecular graph into dimensional Euclidean space using the topological information among the nodes in the graph (Scarselli et al., 2008). Most existing GNNs the basic molecular graphs topology to obtain structural information through neighborhood feature aggregation and pooling methods (Kipf & Welling 2016;; Ying al. 2018 Gao & Ji 2019). However, these methods fail to consider connections among molecular graphs, specifically the sharing of motif patterns in the molecular graph.
One of the critical differences between molecular graphs and other graph structures such as social network graphs and citation graphs is that motifs, which be seen common sub-graphs in molecular graphs have special meanings. For example, an edge in a molecule represents a bond, and cycle represents ring. One ground truth that has been widely used explanation of GNNs is that carbon rings and NO2 groups tend to be mutagenic (Debnath et al., 1991). Thus, motifs deserve more attention when designing GNNs for motif-level feature representation learning. To this end, we propose novel method to learn motif-level feature embedding for molecular graphs. We first extract motifs from molecular graphs and build a motif vocabulary containing all these motifs. Then we construct heterogeneous motif graph containing all motif nodes and molecular nodes. We can apply GNNs to learn letif representations for each molecular graph based on the heterogeneous motif graph. The message passing scheme in a heterogeneous motif graph enables interaction between motifs and molecules, which helps exchange information between molecular graphs. The experimental results show that the learned motif-level embedding can dramatically

---

Under review as a conference paper at ICLR 2022
Molecule Graph
Motifs
Motif Dictionary
Tosic Acid
OH
Atom Type:
Bead Type:
8 O
Saule
Double
Figure 1: Example building motifs vocabulary. Given a molecule Tosic Acid, we first extract bonds and rings from its atom graph. After removing duplicates, we add five unique motifs into the vocabulary. Blue, red, and purple nodes represent Carbon, Sulfur, and Oxygen atoms, respectively.
improve the representation of molecule, and our model can significantly outperform other state- of-the-art GNN models on a variety of graph classification datasets.
2 HETEROGENEOUS MOTIF GRAPH NEURAL NETWORKS
In this section, we propose a novel method to construct a motif-based heterogeneous graph, which can advance motif-based feature representation learning on molccular graphs. Then, we use two separate graph neural networks to learn atom-level and motif-level graph feature representations, respectively.
2.1 MOTIF VOCABULARY OF MOLECULAR GRAPHS
In molecular graphs, motifs are sub-graphs that appear repeatedly and are statistically significant. Specific to biochemical molecule graphs, motifs can be bonds and rings. Analogously, an edge in the graph represents a bond, and a cycle represents a ring. Thus, we can construct a molecule from sub-graphs motifs out of motif vocabulary. To represent molecule by motifs, we first build motif vocabulary that contains valid sub-graphs from given molecular graphs.
To build the motif vocabulary, search all molecular graphs and extract important sub-graphs. In this work, we only keep bonds and rings to ensure a manageable vocabulary size. However, the algorithm be easily extended include different motif patterns. We then remove all duplicate bonds and rings. Some motifs may appear in most of molecules, which carry little information for molecule representation. To reduce the impact of these common motifs, frequency-inverse document frequency (TF-IDF) algorithm (Ramos et al., 2003). In particular, term frequency measures the frequency of motif in molecule, and inverse document frequency refers to the number of molecules containing a motif. We average the TF-IDFs of those molecules that contain motif as the TF-IDF value of the motif. By sorting the vocabulary by TF-ID,, we keep the most essential motifs as our final vocabulary. Figure illustrates the procedure of building motifs vocabulary.
employ the term
2.2 HETEROGENEOUS MOTIF GRAPH CONSTRUCTION
Based on the motif vocabulary, we build a heterogeneous graph that contains motif nodes and molec- ular nodes. In this graph, each motif node represents motif in the vocabulary, and each molecular node is a molecule. Then, we build two types of edges between these nodes; those are motif- molecule edges and motif-motif edges. We add motif-molecule edges between molecule node and motif nodes that represent its motifs. We add a motif-motif edge between two motifs they share at least one atom in any molecule. In this way, can build heterogeneous graph containing all motifs in the vocabulary and all molecules connected by two kinds of edges. Appendix contains detailed pseudocode of constructing Heterogeneous Motif Graph.
2

---

Under review as a conference paper at ICLR 2022
Phenol
Styrene
Toluene
Alom Type:
Cl
Bond Type:
Single
Dushle
Molecular Type:
Moncashuttuted Beraene
m- Cresoo
3-Chlorophenol
Figure 2: Example of heterogeneous motif graph. In this graph, there five molecular nodes: Phenol, Styrene, Toluene, m-Cresol, and 3-Chlorophenol. Here, we have five motifs the vocab- ulary. We connect a molecular node with motif node if the molecule contains this motif. For example, Phenol has Benzene and carbon-oxygen bond. Thus, we connect the Phenol node with the Benzene node and carbon-oxygen bond node. We connect two motif nodes they share at least one atom in molecules. In this graph, we connect the Benzene node and the carbon-oxygen bond node since they share carbon atom.
One thing to notice is that different motifss have different impacts. We assign different weights to edges based on their ending nodes. particular, for edges between motif node and a molecule node, use the TF-IDF value of the motif as the weight. For the edges between two motif nodes, we use the co-occurrence information point-wise mutual information (PMI), which popular correlation measure in information theory and statistics Yao al. 2019). Formally, the edge weight Ag between node i and node j is computed as
PMI TF-IDE 0,
if i, j are motifs if or j a motif Otherwise
Au
(I)
The TF-IDF value of an edge between motif node i and molecular node is computed as
TF-IDF, C(i); log 1 M
N() +1)
where C(i), the number of times that the motif appears in the molecule j, M is the number of molecules, and (2) is the number of molecules containing motif i.
The PMI value of edge between motif nodes is computed
PMI, log p(i.j) p(i)p())
where p(i,j) is the probability that molecule contains both motif and motif j. p(1) is the proba- bility that molecule contains motif i, and p(j) is the probability that molecule contains motif j. We use following formulas to compute these probabilities.
p(i.j) N(i,)) M
N(i) M
N(j) p()
p(1)
4)
M
where j) is the number of molecules that contain both motif i and motif j. Figure provides an example heterogeneous motif graph construction. Note that we assign zero weight for motif' node pairs with negative PMI value.
2.3 HETEROGENEOL: MOTIF GRAPH NEURAL NETWORKS
In this part, we build HM-GNN to learn both atom-level and motif-level graph feature represen- tations. In Section 2.2 construct heterogeneous motif graph that contains all motif nodes and

---

Under review as a conference paper at ICLR 2022
OH
Melecalor Griph
Plenol
Final Foture batbodding
HAIN
Heterngrneus Motif Gmph
Heteragenecus Morif Griph
Figure 3: Example of our HM-GNN. Given input Phenol, we first apply GNN on its atom-level graph structure to learn its atom-level feature embedding. Mcanwhile, we add it into heteroge- neous motif graph and use heterogencous GNN to learn its motif-level graph embedding. In the heterogeneous motif graph, Phenol one of the molecular nodes. Finally, we concatenate graph embeddings from two GNNs and feed them into a MLP for prediction.
molecular nodes. Here, we first initiate features for each motif node and molecular node. We use the one-hot encoding to generate features for motif nodes. In particular, each motif node has a feature vector X, of length [VI, where V represents the motif vocabulary we obtained in Section 1. Given the unique index i of the motif the vocabulary, we set [ [i] = and other positions O. For molecular nodes, we use bag-of-words method to populate their feature vectors. We consider each motif as a word and each molecule as a document. By applying the bag-of-words model, we can obtain feature vectors for molecular nodes. Based on this heterogeneous graph, a heterogeneous graph neural network can be applied to learn the motif-level feature embedding for each molecule in the graph.
At the same time, each molecule can be easily converted into a graph by using atoms as nodes and bonds edges. The original molecule graph topology and node features contain atom-level graph information, which can supplement motif-level information. Thus, we employ another graph neural network learn the atom-level feature embedding. Finally, we concatenate feature embeddings from two graph neural networks and feed them into a multi-layer perceptron (MLP) for prediction. Figure shows an example of our HM-GNN model.
2.4 MULTI-TASK LEARNING VIA HETEROGENEOUS MOTIF GRAPHI
This part will show that our heterogeneous motif graph can help with graph deep learning models on small molecular datasets via multi-task learning. It well known that deep learning methods require a significant amount of data for training. However, most molecular datasets are relatively small, and graph deep learning methods can easily over-fit these datasets. Multi-task learning (Caruana 1997) has been shown to effectively reduce the risk of over-fitting and help improve the generaliza- tion performances of all tasks (Zhang & Yang 2017). can effectively increase the size of train data and decrease the influence of data-dependent noise, which leads to a more robust model. However, it is hard directly apply multi-task learning on several molecular datasets due to the lack of plicit connections among different datasets. Based on our heterogeneous motif graph, we can easily connect a set of molecular datasets and form multi-task learning paradigm.
Given N molecular datasets Di.... DN. each dataset D; contains n, molecules. We first construct a motif vocabulary V that contains motifs from N molecular datasets. Here, the motif only needs to be shared in some datasets but not all of them. Then, we build heterogeneous motif graph that contains motifs in the vocabulary and molecules from all datasets. We employ our HM-GNN to leamn both graph-level and motif-level feature representations for each molecule based on this graph. The resulting features of each dataset are fed into a separate MLP for prediction. In this process, the motif nodes can be considered as connectors connecting molecules from different datasets or tasks. Under the multi-task training paradigm, our motif-hhterogeneouu graph can improve the feature representation learning all datasets.
4

---

Under review as a conference paper at ICLR 2022
Figure 4: Example of generating sub- sub-graph for 3-layer HM-GNN via edge sampler. A sam- pling rule is we sample all edges, one edge, two edges for each layer, respectively (We select motif- motif edges first). In this graph, have four molecular nodes and seven motifs nodes. We solid lines to represent selected edges and dashed lines to indicate unselected edges. We randomly choose node S as the "starting" node. In the first hop, we keep all edges connecting the node and motif nodes. We sample one motif-motif edge for each motif node connecting to node S in the second hop. the third hop, two motif-motif edges are selected for each newly added motif node in the last hop. Finally, the resulting sub-graph contains all nodes and edges we sampled.
2.5 EFFICIENT TRAINING VIA EDGE SAMPLING
In Section 2.2 we construct a heterogeneous motif graph that contains all motif nodes and molec- ular nodes. As the number of molecular nodes increases, there can be issue with computational resources. To address this issue, we propose to use an edge sampler to reduce the size of the het- erogeneous motif graph. Due the special structure of heterogeneous motif graph that it has two kinds of nodes and two kinds of edges, we can efficiently generate a computational subgraph by using the type of edge.
We show how sampling edges can save computational resources. Most GNNs follow a neighborhood aggregation learning scheme. Formally, the b-th layer GNN can be represented by
f (z%, of i E N()))))
where xf+1 is the new feature vector of node i, f a function that combines the ě-th layer's features and the aggregated features, & is the function that aggregate all neighbors' feature vectors of node i, eja is the weight of edge,s, and (i) is the set of node i's neighbors. This equation shows that time and space complexity are both O(E)), where |E| is the number of edges in the graph. This means we can reduce the usage of computational resources by removing some edges from the graph. Thus, employ edge sampler that samples edges from the graph. And sampling rule is that we prioritize motif-molecule edges.
To sample edges, first randomly select some molecular nodes "starting" nodes. We breadth-first algorithm to conduct a hop-by-hop exploration on the heterogeneous motif graph start- ing from these nodes. In each hop, we randomly sample fixed size of edges based edge type. Note that the first-hop neighbors of each molecular node are the motif nodes, which play essen- tial roles in our heterogeneous motif graph. Thus, we retain all first-hop edges to ensure effective learning of feature representations for motff nodes. Starting from the second hop, we only sam- ple motif-motif edges retain as much motif information as possible. Figure shows an example of sampling sub-graph for 3-layer HM-GNN. Appendix [B contains complete sample rules and pseudocode.
EXPERIMENTAL STUDIES
In this section, we evaluate our proposed methods on graph classification tasks. We compare our methods with previous ofte--of-hhearr models on various benchmark datasets. Datasets details and experiment settings are provided the Appendix and
3.1 PERFORMANCE STUDIES ON MOLECULAR GRAPH DATASETS
We first evaluate our model on five popular bioinformatics graph benchmark datasets from TU- Dataset (Morris et al., 2020). which includes four molecule datasets and one protein dataset. Here, PTC, MUTAG, NCII, and Mutagenicity are molecule datasets, and PROTEINS is protein dataset. Following GIN (Xu et al. 2018), we use node labels provided by TUDataset as the initial node features. To learn graph feature representations in heterogeneous motif graphs, use 3-yeyer
5

---

Under review as a conference paper at ICLR 2022
Table 1: Graph classification accuracy (%) various TUD graph classification tasks. Some results for GraphSAGE and GCN are reported from (Xu et al. 2018) and (Zhang et al., 2019). The best performer on each dataset are shown in bold. means there no reported accuracy on this dataset in original papers
METHODS
PTC
MUTAG
NCI1
PROTEINS
MUTAGENICITY
PatchySAN
60.0 4.8
92.6 = 4.2
78.6
75.9
GCN
64.2
85.6 + 5.8
8..
76.0
79.8 + 1.6
GraphSAGE
6.7
7.66
1..7
7..9
78.8 1.2
DGCNN
2.5
85.8
0.54
75.5
GIN
6.0
8.66
8.77
76.2
PPGN
66.2 - 6.5
8.7
8.12
77.2
CapsGNN
6.9
1.6
76.3 3.6
WEGL
64.6 166-10 7.4
8.1
1.7
76.1 3.3
GraphNorm
64.9 + 7.5
91.6
81.4
77.4 + 4.9
OURS
78.8 6.5
96.3 2.6
83.6
79.9 + 3.1
83.0 1.1
GIN. To utilize atom-level information like node and edge features, we use another GIN on each atom-level graphs. Specifically. it has GNN layers and 2-layer MLPs. Batch normalization (loffe & Szegedy 2015) is applied to each layer. and dropout (Srivastava et al. 2014) is applied to all lay- ers except the first layer. To evaluate the performance
our model, we strictly follow the settings in (Yanardag& & Vishwanathan, 2015, Niepert et al., 2016) 6, Xu et al., 2018 Gao & 2019). For each cross-validass validation with random splitting on the entire dataset. We report the mean and standard deviation of validation accuracy from ten folds.
dataset, we perform 10-fold cross-var
We compare our model on five datasets with six state-of-the-ard GNN models for graph classifi- cation tasks: PATCHY-SAN (Niepert et al., (016)), Graph Convolution Network (GCN) (Kipf & Welling, 2016), GraphSAGE (Hamilton et al. (017). Deep Graph CNN (DGCNN) (Zhang et al.
2018), Graph Isomorphism Network (GIN) (Xu et al. 2018), Provably Powerful Graph Networks
(PPGN) (Maron et al. 2019), Capsule Graph Neural Network (CapsGNNN (Xinyi & Chen 2018), Wasserstein Embedding for Graph Learning (WEGL) (Kolouri al. 2020). and GraphNorm (Cai et al. 2021). For baseline models, we report the accuracy from their original papers. The compari- son results are summarized in Table From Table [ our model consistently outperforms baseline models on all five datasets. The superior performances on four molecular datasets demonstrate that the motif nodes constructed from the motif vocabulary can help GNN learn better motif-level feature representations of molccular graphs. On the protein dataset, our model also performs the best, which shows that the motifs in protein molecules also contain useful structural information.
3.2 PERFORMANCE STUDIES ON LARGE-SCALE DATASETS
To evaluate our methods on large-scale
Table 2: Graph Classification Results (%) on two
datasets,
two bioinformaties datasets
open graph benchmark datasets. The results for GCN and GIN are reported from (Hu et al., 2020).
from the Open Graph Benchmark (OGB) (Hu
al. 2020): egbg-molhiv and ogbg-molpeba. These two molecular property prediction datasets are adopted from the MOLECU-
METHODS
ogbg-molhiv
ogbg-pcba 24.24 + 0.34
GCN
75.99 1.19
GIN
77.07 + 1.49
27.03 + 0.23
LENET (Wu et al., 2018). They are all pre-processed by RDKIT (Landrum et al. 2006). Each molecule has mine-dimensional node features and three-dimensional edge
PNA
79.05 1.32
OURS
79.03 = 0.92
28.70 + 0.26
PNA
80.20 + 1.18
features. molg-y is binary classification dataset, while Ogbg-molpcha is a multi-class classification dataset. We report the Receiver Operating Characteristic Area Under the Curve (ROC-AUC) for Ogbg-molhiv, and the Average Precision (AP) for Ogbg-molpcba, which are more popular for the situation of highly skewed class balance (only about 1.4% of data is positive). Following Wu et al. (2018); Hu al.) (2020). we adopt the scaffold splitting procedure to split the dataset. This evaluation scheme is more challenging, which requires the distribtuin generalization capability.

---

Under review as a conference paper at ICLR 2022
Table 4: Results the PTC dataset with three different training settings. The first row report the performances of only using the PTC dataset. The second row and third row show the results of training on combined vocabularies and datasets with PTC MM and PTC FR, respectively. We report the motif vocabulary size (Vocab Size) of the dataset and the Overlap Ratio, which indicates the overlap ratio of motif vocabularies between two datasets. The last three columns represent the performances of using different sizes of training sets. For example, 90% means we use 90% of
dataset as the training set and 10% of dataset as the testing set.
Dataset
Vocab Size
Overlap Ratio
90%
50%
10 %
PTC
97
71.8 + 4.1
65.1 + 0.8
59.9 + 1.9
PTC MM
111
83.5% 94.8%
76.5 + 3.3
69.2 + 0.8
66.7 1.9
+ PTC
110
84.3 + 3.8
77.3 + 0.8
74.0 + 1.7
In this part, we compare our model with GIN (Xu et al., and PNA (Corso et al. 2020). We report the mean and standard deviation of the results by using 10 different seeds (0-9). Table shows the ROC-AUC results on Ogbg-molhiv and AP results on belg-molpeba. It can be observed from the results that our approach outperforms GIN, GCN, and PNA by significant margins. The results demonstrate our model's superior generalization ability on large-scale datasets.
2018), GCN (Kipf Welling 2016).
3.3 ABLATION STUDIES ON HETEROGENEOUS MOTIF GRAPH NEURAL NETWORKS
In Section 3.1 and Section 3.2 our HM-
Table 3: Graph classification accuracy (%) of GIN and
GNNs use GINs to learn atom-level infor- mation for molecular graphs, which can be good complement motif-level feature
our model on three datasets: PTC, MUTAG, and PRO- TEINS. Models
PTC
MUTAG
PROTEINS
representations. To demonstrate the effec-
GIN
64.6 7.0
89.4 = 5.6
76.2 2.8
tiveness of motif-level feature learning our HM-GNNs, we remove the heteroge-
OURS
78.8 + 6.5
96.3 = 2.6
79.9 + 3.1
neous graphs and the corresponding GNNs from HM-GNNs, which reduces to GINs. We compare our LMM-NNNs with GINs on three popular bioinformatics datasets: PTC, MUTAG, and PROTEINS. The comparison results are summarized in Table From the results, our model significantly out- performs GIN by margins of 14.2%, 6.9%, and 3.7% on PTC, MUTAG, and PROTEINS datasets, respectively. This demonstrates that motif-level features are critical for molecular feature represen- tation learning.
3.4 RESULTS OF MULTI-TASK LEARNING ON SMALL MOLECULAR DATASETS
In the section 2.4 we introduce a new multi-task learning paradigm by constructing a mixed het- erogencous motif graph that contains molecular nodes from different datasets. Here, conduct experiments to demonstrate the effectiveness of this new multi-task learning paradigm. To this end, another two PTC datasets: PTC MM and PTC FR. Here, both PTC and PTC FR rats datasets and PTC_MM a mice dataset. By combining PTC with PTC_MM and PTC_FR sepa- rately, we can create another two datasets: PTC PTC.MM and PTC + PTC.FR. The statisties of these new datasets are summarized in Table In this table, we report the vocabulary sizes and the overlap ratios with the original PTC dataset. The overlap ratios show that these datasets share most of their motifs. In particular, the PTC dataset has 94.8% and 83.5% overlapped motifs with PTC_FR _RR and PTC MM, respectively. We construct separate heterogeneous motif graphs and evaluate our HM-GNNs on them. Here, we conduct the evaluations on three settings: 90%, 50%, and 10% for training and 10%, 50%, 90% for testing. The models trained on smaller training datasets have higher risk of overfitting, which can better reveal the impacts performances.
The comparison results are summarized in the last three columns in Table We can observe from the results that combing PTC with PTC.MM and PTC_FR can consistently bring performance im- provements in three settings. Notably, combing PTC.FR brings even larger performance boost than using
This is because PTC has larger motif vocabulary overlap with PTC_FR. Thus, Com- bining datasets with similar motif vocabularies will benefit multi-tase task learning on small molecular datasets.
7

---

Under review as a conference paper at ICLR 2022
3.5 COMPUTATIONAL EFFICIENCY STUDY
In Section 2.5 we propose to use an edge sam- pler to improve the training efficiency of our HM-GNNs by reducing the size of edges in the heterogeneous graph. In our algorithm, if we fixed the sampling rule, the number of starting nodes is a nyper-paramete the sampled heterogeneous motif graph. As an essential syper-parameter. the number of start- ing nodes can influence both training efficiency and model performance. In this part, con- duct experiments to investigate its impact on the Ogbg-mollig dataset. In particular, we change the number of starting nodes and report the cor- responding model performances in terms of the ROC-AUC.
13500
11500
Memor Usage MB
ROGGAUC (%)
9500
control the size of
77.5
7500
77-
5500
76.5
3500
ROC-AUC Usage
1500
0.5
2
2.5
Number of Starting Nodes
x10
Figure 5: Results of ROC-AUC and memory us- age using different number of starting nodes. We vary the number of starting nodes from 5,000 to 29,000. The ROC-AUC performances and mem- ory usages of different batch sizes illustrated on blue and red lines, respectively.
Figure 5 shows the performance of our model as the number of starting nodes changes. The blue line represents the ROC-AUC value, and the red line shows the memory usage. The blue line shows that the model performance boosts significantly when the number of starting nodes changes from 5,000 to 15,000. Starting from 15,000, the improvement of ROC-AUC gradually slows down until it converges. The red line shows that the memory usage increases almost linearly as the number of starting nodes increases. Thus, the model can achieve the best utility efficiency when choosing 25,000 as the number of starting nodes. At this point, model can achieve high performance with relatively fewer computational resources.
3.6 MOTIF VOCABULARY SIZE. STUDY
In Section 2.1 we propose filtering noisy motifs from the motif vocabulary by their TF-IDF values, which can improve our model's generalization abil- ity and robustness. By using different keeping ratios, we can have different vocabulary sizes. In this part, we conduct experiments to investigate the impact of varying keeping ratios on the model performance. obtain differ- Using different keeping ratios, molhiv dataset, ent motif vocabularies on the leading to different heterogeneous motif graph con- struction outputs. Here, we apply our HM-GNNs on the resulting heterogeneous motif graphs and report the performances in terms of the ROC-AUC, Here, vary the keeping ratio from 50% 100%, which indicates the portion of top motifs using as final mo- tif vocabulary.
79.1
79-
78.9
78.8
UC
78.7
78.6
78.5
50
60
70
80
90
100
Keeping Ratio (%)
Figure 6: The impact of the motif keeping ra- tio. We evaluate our method using different keeping ratio of the original motif vocabu- lary.
We summarize the result in Figure 6 From the fig- ure, we can observe that the model performance im- proves as the increase of keep ratios. A higher keeping ratio increases the motifs in the vocabulary, which leads to better motif-level feature propagation. And molecules in the graph have more con- nectors (motifs) to communicate with other molecules. The model performance starts to decrease when the keeping ratio larger than 80%, which indicates the last 20% motifs are noisy and can hurt the model generalization and robustness. Notably, in the vocabulary, our model can still outperform GIN by a margin of 1.37%, which demonstrates the significant contribution of letif feature representations.
with 50% of the most important motifs
8

---

Under review as a conference paper at ICLR 2022
4 RELATED WORK
In this section, we provide a briefly introduction to some related works on graph representation, motif learning, and deep molecular graph learning.
GNNs (Micheli 2009, Scarselli et al. 2008 have became the most major tool in machine learning on graph-related tasks. Many GNN variants have been proposed (Battaglia et al. 2016 Defferrard al.
2016 Duvenaud et al. [2115] Hamilton et al. [0177] Kipf & Welling 2016] at. 2015 Veličkovid
al. 2017 Santoro et al., 2017; Xu et al. 2018). They have different neighborhood aggregation
and graph pooling method. In factual, these models have achieved state-of-the-art baselines in many graph-related tasks like node classification, graph classification, and link prediction.
A motif can be a simple block of a complex graph and is highly related to the function of the graph (Alon 2007). (Prill al.) 2005) demonstrates that dynamic properties are highly correlated with the relative abundance of network motifs in biological networks. MCN (Lee et al. 2019) proposes a weighted multi-hop motif adjacency matrix to capture higher-order neighborhoods and uses an attention mechanism to select neighbors. HierG2G (Jin et al. (2200) employs larger and more flexible motif's as basic building blocks to generate a hierarchical graph encoder-decode
MICRO-
Graph (Zhang al., (2200) applies motif learning to help contrastive learning of GNN. Some GNN explainers also use motif knowledge to generate subgraphs to explain GNNs (Ying et al.) 2019 Yuan
al. 1021
Specific to the bioinformatics field, GNNs have been widely used in molecular graphs related tasks. Duvenaud al. (2015) introduces molecular feature extraction method based on the idea of circu- lar fingerprints. MPNNs (Gilmer et al. 2017) reformulate existing models and explores additional novel variations. (Chen al., 2017) introduces a non-backtracking operator defined on the line graph of edge adjacencies to enhance GNNs. DimeNet (Klicpera et al. 2020) proposes directional message passing embed the message passing between atoms instead of the atoms themselves. HIMP (Fey et al., (2020) develops a method to learn associated junction trees of molecular graphs and exchange messages between junction tree representation and the original representation to detect cycles. Some other works introduce permutations of nodes into models (Murphy et al. [2019] Al-
booych et al. 2019). Pre-training and self self-supervised learning schemes have been proved that they can effectively increase performance for downstream tasks (Hu et al.) 2019; Sun et al. 2019 Rong ct al. 2020 Hassani & Khasahmadi 2020 Zhang et al. 2020 Sun et al.] (222]). DGN [eani ct al,
2021 introduces a globally consistent anisotropic kernels for GNNs to overcome over-smoothing
Issue. Another line of work focuses on graph pooling. DiffPool (Ying et al., 2018) addresses a
differentiable graph pooling module which can build hierarchical representations of graphs. Graph U-Nets (Gao & Ji 2019) introduces pooling and up-sampling operations into graph data.
However, most existing works focus on the molecular graph itself, and they have considered connections among molecular graphs, such as motif-level relationships. Because different molecules may share the same motif in their structures, we use a heterogeneous motif graph neural network model to extract and learn motif representation of molecular graphs. By learning motif-level feature representations, our proposed methods can increase the expressiveness of deep molecular graph representation learning.
5 CONCLUSION
In this work, we propose a novel heterogeneous motif graph and HM-GNNs for molecular graph rep- resentation learning. We construct a motif vocabulary that contains all motifs in molecular graphs. To remove the impact of noise motifs, we select the essential motifs with high TF-IDF values, which leads to more robust graph feature representations. Then, we build a heterogeneous motif graph that contains motif nodes and molecular nodes. We connect two molecules by jointly owned motifs in the heterogeneous motif graph, enabling message passing between molecular graphs. We use one HM-GNN to learn the heterogeneous motif graph and get the motif-level graph embedding. And we use another GNN to learn the original graph's atom-level graph embedding. This two-GNNs model can learn graph feature representation from both atom-level and motif-level. Experimental results demonstrate that our HM-GNN can significantly improve performance compared to previous state-of-te -hetthh-art GNNs on graph classification tasks.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Marjan Albooyeh, Daniele Bertolini, and Siamak Ravanbakhsh, Incidence networks for geometric deep learning. arXiv preprint wrXiv:995.114660 2019.
Uri Alon. Network motifs: theory and experimental approaches. Nature Reviews Genetics, 8(6): 450-461, 2007.
Peter W Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. arXiv preprint arXiv:1612.00222, 2016.
Dominique Beani, Saro Passaro, Vincent Létourneau, Will Hamilton, Gabriele Corso, and Pietro Liò. Directional graph networks. International Conference on Machine Learning, pp. 748- 758. PMLR, 2021.
Karsten M Borgwardt, Cheng Soon Ong, Stefan Schönauer, SVN Vishwanathan, Alex Smola, and Hans-Peter Kriegel. Protein function prediction via graph kernels. Bioinformatics, 21(suppl 1): i47-i56, 2005.
Tianle Cai. Shengjie Luo, Keyulu Xu, Di He. Tie-yan Liu, and Liwei Wang. Graphnorm: A prin- cipled approach to accelerating graph neural network training. In International Conference on Machine Learning, pp. 1204-1215. PMLR, 2021.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Zhengdao Chen, Xiang Li, and Joan Bruna. Supervised community detection with line graph neural networks. arXiv preprint arXiv: /705.0845, 2017.
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbourhood aggregation for graph nets. arXiv preprint arXiv:2004.05718, 2020.
Asim Kumar Debnath, Rosa L Lopez de Compadre, Gargi Debnath, Alan J Shusterman, and Cor- win Hansch. suueture-activity relationship of mutagenic aromatic and heteroaromatic nitro com- pounds. correlation with molecular orbital energies and hydrophobicity. Journal of medicinal chemistry, 34(2)788--777, 1991.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29:3844-3852, 2016.
David Duvenaud, Dougal Maclaurin, Jorge Jaarraauppprrrgurrree, Rafael Gómez-Bombarelli, Tim- othy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. arXiv preprint arXiv: 1509.09929. 2015.
Matthias Fey, Jan-Gin Yuen, and Frank Weichert. Hierarchical inter-message passing for learning molecular graphs. arXiv preprint arXiv:2006.12177 12179, 2020.
Hongyang Gao and Shuiwang Ji. Graph u-nets. In international conference machine learning. pp. 2083-2092 PMLR, 2019.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference Neural Information Processing Systems, pp. 11055-1355. 2017.
Kavch Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference Machine Learning, pp. 4116-4126. PMLR, 2020.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitmik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre- pre-training graph neural networks. arXiv preprint 1905, 12265, 2019.
10

---

Under review as a conference paper at ICLR 2022
Weihua Hu, Matthias Fey, Marinka Zitiik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXih:2005.00668 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 488-456. PMLR, 2015.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In International Conference on Machine Learning, pp. 4839-4848. PMLR, 2020.
Jeroen Kazius, Ross McGuire, and Roberta Bursi. mutagenicity prediction. Journal of medicinal chemistry, 48(13332--320 2005.
Derivation and validation of toxicophores for
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net-
works. arXiv preprint arXiv: (oop:020099 2016.
Johannes Klicpera, Janek GroB, and Stephan Günnemann. Directional message passing for molec- ular graphs. arXiv preprint arXiv:2003.08 /23, 2020.
Soheil Kolouri, Navid Naderializadeh, Gustavo K Rohde, and Heiko Hoffmann. Wasserstein em- bedding for graph learning. arXiv preprint 004::20000.09430, 2020.
Greg Landrum al. Rdkit: Open-source cheminformatics. 2006.
John Boaz Lee, Ryan A Rossi, Xiangnan Kong, Sungchul Kim, Eunyee Koh, and Anup Rao. Graph convolutional networks with motif-based attention. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 499-499999 1999-508,1 2019.
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. networks. arXiv preprint arXiv: 15::::11110000 1.10,5593, 2015.
Gated graph sequence neural
Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. arXiv preprint arXiv:1955.11136, 2019.
Alessio Micheli. Neural network for graphs: A contextual constructive approach. IEEE Transactions on Neural Networks, 20(3):498-511, 2009.
Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benehmark datasets for learning with graphs. In ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020), 2020. URL www. graphl graphlearning earninggniin
Ryan Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. Relational pooling for graph representations. In International Conference on Machine Learning, pp. 4663-4673.
PMLR, 2019.
Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural net- works for graphs. In International conference on machine learning, pp. 2014-2023. PMLR, 2016.
Robert J Prill, Pablo Iglesias, and Andre Levchenko. Dynamic properties of network motifs contribute to biological network organization. PLoS biology, 3(11) 2005.
Juan Ramos et al. Using if-idf determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, volume 242, pp. 29-48. Citeseer, 2003,
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. arXiv preprint arXiv:2007.02835. 2020.
Adam Santoro, David Raposo, David GT Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. arXiv preprint arXiv: / 01..70066.1122 2017.
11

---

Under review as a conference paper at ICLR 2022
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-80, 2008
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European semantic web conference, pp. 593-607. Springer, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1955 2014.
Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv: 01000, 2019.
Mengying Sun, Jing Xing, Huijun Wang. Bin Chen, and Jiayu Zhou. Mocl: Contrastive learning on molecular graphs with multi-level domain knowledge. arXiv preprint arXiv:2/06.044 2021.
Hamnu Toivonen, Ashwin Srinivasan, Ross D King, Stefan Kramer, and Christoph Helma. Statistical evaluation of the predictive toxicology challenge 2000-2001. Bioinformatics, 19(10): 1100:::188-199,,, 2003.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 2017.
Nikil Wale, lan A Watson, and George Karypis. Comparison of descriptor spaces for chemical com- pound retrieval and classification. Knowledge and Information Systems, 14(3):347-375, 2008.
Zhengin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learn- ing. Chemical science, 9(2):513-530, 2018.
Zhang Xinyi and Lihui Chen. Capsule graph neural network. In International conference on learning representations, 2018
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826. 2018.
Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1365-1374, 2015.
Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convolutional networks for text classification. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 7370-7377, 2019.
Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. arXiv preprint 1rxiv:1806.08804. 2018
Rex Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gan explainer: A tool for post-hoc explanation of graph neural networks. arXiv preprint 1XXXv:1903.03800 2019.
Hao Yuan, Haiyang Yu, Jie Wang, Kang Li, and Shuiwang Ji. explainability of graph neural networks via subgraph explorations. arXiv preprint a0Xiv:2 152, 2021.
Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Shichang Zhang, Ziniu Hu, Arjun Subramonian, and Yizhou Sun. Motif-driven contrastive learning of graph representations, arXiv preprint arXiv:2012.12533. 2020.
Yu Zhang and Qiang Yang. A survey on multi-task learning. arXiv preprint arXiv:1707.0811
2017.
Zhen Zhang, Jiajun Bu, Martin Ester, Jianfeng Zhang, Chengwei Yao. Zhi Yu, and Can Wang. Hierarchical graph pooling with structure learning. arXiv preprint 191 2019.
12

---

Under review as a conference paper at ICLR 2022
A PSEUDOCODE OF GENERATING HETEROGENEOUS MOTIF GRAPH
Algorithm is a pseudocode to show how to construct a Heterogeneous Motif Graph. Lines 1-19 Algorithm| generate all motif-molecule edges. Lines 20-24 build motif-motif edges. Line 25 first calculates TF-IDF for all molecules containing motif, then averages these TF-IDFs as the TF-IDF value of the motif. Line 26 calculates the TF-IDF of molecule as a molecule-motif weight, then calculates PMI value as motif-motif weight. Note that we assign zero weight to motif node pairs with negative PMI. Lines 27-28 generate feature vectors for motif nodes and molecular nodes by one-hot encoding and bag-of- words method, respectively.
Algorithm 1: An algorithm building a Heterogcncous Motif Graph
Input
A set of graph data G (91, 92, 9k: Gv Wk (1,.... NJ, ne and ex denote the numbers of nodes and edges in graph GR. respectively.
Output
A large Heterogeneous Motif graph (M E). V mooeecue Pmotif.
E = moot))) U [(motif, f)-
Initialization:
Initialize an empty set of motif vocabulary M = 0
An empty Heterogeneous Motif graph (V, V = -
An empty dictionary B stores how many times a motif appears in molecule An empty dictionary C' stores how many molecules contain for each motif.
for k 1,..., N do
Search Graph (v(V,, Ex)
Generate set of edges Mg (e1, C,, e ). Vi E {I.....} and set of basic cycles Mc {ci, (2,02,,,, Gj c.), Vj E {l,...,
for e, Mg do
e; is part of where Oj E Mc then
Delete e; from Mr
end end
Unify motifs M₁-- ME U Mc
Add generated motifs into motif vocabulary M = MU M Add molecular node U2 into 9M
11
13
for motif's in Mr do
Add motif node Us into set GM Add an edge between the motif node u. and the molecular node into set E T donates how many s does Graph Gy have and add Bks T B Update C,,
15
16
18
end
19
end
for motif and motif r in M do
21
s and r are sharing at least one atom then
Add edge between the motif node and the motif node
23
end
24 end
Calculate TF-IDF for each motif. then select essential motifs based some threshold.
26
Calculate molecule-motif edge weights based on dictionary B and motif-motif edge weights based on and then add them Graph GM
27 Generate one-hot fcature vectors for motif nodes. {x {0, 10,13; E. 2, = 1}, p the length feature vector.
28 Generate feature vectors for molecular nodes using bag-of-word method.
13

---

Under review as a conference paper at ICLR 2022
B PSEUDOCODE OF OUR MINI-BATCH HETEROGENEOUS MOTIF GRAPH NEURAL NETWORK
Algorithm pseudocode of minibatch HM-GNN. Lines 3-6 of Algorithm _ correspond to the sampling stage. R(k) is the sampling rule each layer, and it fixes the sample size in each layer. To retain as much important motif information as possible, we sample all edges in the first hop, and starting from the second hop, randomly sample edges from motif-motif edges. In this way, the computational graph contains essential motff nodes in sampling. Lines 7-12 correspond to a motif- level embedding learning stage. We can choose any kind of AGGREGATE and COMBINE( here. Lines 14-19 correspond to atom-level embedding learning stage. Line 21 concatenates the atom-level embedding and the motif-level embedding as final graph embedding.
Algorithm 2: Mini-batch HM-GNN algorithm
Input
A set of graph data G (91, 92, .... 9N AA (1...., N), me and ex denote the number of nodes and edges
Input features { Vg G}
Heterogeneous Motif Graph Gas V= Vmotecull Vmotis
Input feature of Heterogeneous Graph (he Yo E V}
Depth K
AGGREGATE(#)) (x)
COMBINE(()()
Edge sampling rule R(k)
Batch size S
A subset G, we use this G' to generate starting molecular nodes in HM graph
sampling
GNN model p(-) for atom-level learning
Vector representations for all molecular nodes computational sampled graph Initialization: An empty computational graph for motif learning (Vo.,,E)) Ve 0. & = 0
Output
Generate starting molecular nodes ((K) G'
Add all nodes in (K) into V/
for & = K...1 do
Randomly sample edges based on rule (kk) and starting nodes ((k) from 9m, and then add these edges into Ec. add all end nodes of edges into Vc
All end nodes of these edges become new set of starting nodes (kk- 1) for next iteration
end
Learning computational graph Go for motif embedding
for k = 1...K do
for v E Bb)) do
an (k)
(GGREGATE(A) (nh(t :u N(u)))
:uE(()))
(R) COMBINE(M) (1.5 (k---1) ,a
10
11
end
12 end
13 Motif embedding em +- hy, Ve E B(K) // Learning atom-level embedding
14 for u E ((K) do
15 Find the corresponding atom-level graph
16
ha = p(g. xn)
17 end
18 Atom-level embedding e, + has Va E (K)
Final graph embedding & ealle eallem
14

---

Under review as a conference paper at ICLR 2022
C DETAILS OF DATASETS
We give detailed descriptions of datasets used in our experiments. Further details can be found in Yanardag & Vishwanathan (2015), Zhang al. (2019), and Wu al. (2018).
MUTAG (Debnath et al. 1991) is dataset that contains 188 mutagenic aromatic and heteroaromatic nitro compounds. The task to predict their mutagenicity Salmonella typhimurium. It has discrete labels. PTC (Tivonen et al. 2003) is dataset that contains 344 chemical compounds that reports the carcinogenicity for male and female rats with 19 discrete labels. NCII (Wale al., 2008) is made publicly available by the National Cancer Institute (NCT). It is a subset of screened compound balance dataset designed to inhibit inhibit the growth of a group of human tumor cell lines with 37 discrete labels. PROTEINS (Borgwardt et al. 2005) is a dataset in which nodes are secondary structure elements (SSE). two nodes are adjacent nodes an amino acid sequence or 3D space, there are edges between them. It has discrete labels, which represent spirals, slices, or turns. Mutagenicity (Kazius et al. 2005) is a dataset of compounds for drugs, which can divided into two categories: mutagens and non-mutagenic agents.
The Ogbg-molhiv dataset was introduced by the Drug Therapy Program (DTP) AIDS Antiviral Screen, which tested the ability of more than 40,000 compounds inhibit HIV replication. The screening results were evaluated and divided into three categories: Confirmed Inactivity (CT), Con- firmed Activity (CA), and Confirmed Moderate Activity (CM). PubChem BioAssay (PCBA) is database consisting of the biological activities of small molecules produced by high-throughput screening. The Ogbg-peba dataset is subset of PCBA. It contains 128 bioassays and measures more than 400,000 compounds. Previous work was used to benchmark machine learning methods.
15

---

Under review as a conference paper at ICLR 2022
D DETAILS OF EXPERIMENT SETTINGS
We give detailed descriptions of experiment settings used in our experiments.
TUDatasets. For all configurations, 3 GNN layers applied, and all MLPs have 2 layers. Batch normalization is applied on every hidden layer. Dropout is applied to all layers except the first layer. 2000. We use Adam optimizer with initial weight decay 0.0005. The hyper-
The batch size is parameters we tune for each dataset are: (1) the learning rate E {0.01, 0.05}; (2) the number of hidden units E {16, 64, 1024); (3) the dropout ratio E {0.2, 0.5}.
Open Graph Benchmark. For all configurations, 3 GNN layers are applied, and all MLPs have layers. Batch normalization is applied on every hidden layer. Dropout is applied all layers except the first layer. We use Adam optimizer with initial weight decay 0.0005, and decay the learning rate by 0.5 every 20 epochs if validate performance not increase. The
we tume for each dataset are: (1) the learning rate E (0.01, 0.001 J: (2) the number of hidden units E {10, 16); (3) the dropout ratio E (0.5, 0.7, 0.9} (4) the batch size E { {128, 5000, 28000).
16


Reviews:
[{"review_id": "qmGzrpBsgBh", "reviewer": "Reviewer_swsR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\nThe paper proposes a method of constructing a heterogeneous graph based on motif, which can effectively utilize motif-based features.\n\n### Weaknesses\n\nThe proposed method of the paper is interesting, but the baseline for comparison is too weak. For example, the two datasets of OGB shown in Table 2 compare the methods that are ranked very low on the OGB leaderboard, so it is difficult to prove the power of the method. At the same time, the paper did not give a detailed explanation on why GIN was chosen as the feature representation of the learning graph, and from the results in Table 1, GIN does not have obvious advantages compared with other methods. And when doing atom-level feature embedding learning, the edge feature (Bond Type, etc.) is not used, and the edge feature should have a certain degree of contribution to the prediction of molecular properties.\n", "summary_of_the_paper": "Aiming at the feature representation learning problem of molecular graphs, this paper proposes a heterogeneous motif graph composed of motif nodes and molecular nodes to realize the information interaction between motifs and molecules. The paper uses graph neural networks to learn and combine atom-level and motif-level graph feature representations, so as to improve the representation of molecules.", "summary_of_the_review": "The motif-based heterogeneous graph construction method proposed in this paper is interesting, but the experimental results are difficult to prove the effectiveness of the method. The network architecture is slightly simpler, the chemical bond information of the molecules is not used in the atom-level feature extraction. Therefore, I am not in favor of recommending this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "d3cyCPe86tu", "reviewer": "Reviewer_6PpB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n- The idea of adopting multi-task learning via heterogenous motif graph to address lack of data problem of deep learning for molecular data is interesting. Moreover, the empirical performance through multi-task learning is promising.\n- The paper is well organized.\n\nCons\n1. The paper lacks a clear justification on whether two molecules sharing motifs imply special meaning.\n\t- In other words, is sharing a motif a clear indication that two molecules share common properties? \n\t- In addition, a motif-motif edge is generated if at least one atom is shared. Does sharing an atom imply anything chemically?\n\n2. The exact message passing process needs to be described. For example, the sizes of the feature vectors of a motif node and a molecular node should be different, and how they are put together for message passing is not clear.\n\n3. Why are there blanks in Table 1 regarding \"MUTAGENICITY.\"\n\n4. Regarding the edge sampler, \n\t- Is BFS done until all the nodes are covered?\n\t- How efficient is HM-GNN compared with GIN in terms of memory consumption? \n\t- The edge sampler seems to be not much different from what is done in GraphSAGE.\n\n5. The literature survey is not sufficient. \n\t- There are many recent work on GNNS for molecular graphs especially along the line of self-supervised learning, such as [1,2,3,4]. HM-GNN should be compared with a few methods among them, if not, they should be mentioned in the related work.\n\t- Moreover, since this is not the first paper to consider \"motifs\" for GNNs, related work, such as [5,6,7,8] should have been described in the paper.\n\n\n[1] MoCL: Contrastive Learning on Molecular Graphs with Multi-level Domain Knowledge\n\n[2] InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization\n\n[3] Contrastive Multi-View Representation Learning on Graphs\n\n[4] Self-Supervised Graph Transformer on Large-Scale Molecular Data\n\n[5] Graph Convolutional Networks with Motif-based Attention\n\n[6] Hierarchical Generation of Molecular Graphs using Structural Motifs\n\n[7] Gnnexplainer: Generating explanations for graph neural networks\n\n[8] Motif-Driven Contrastive Learning of Graph Representations\n\n6. Minor comments\n\t- For reproducibility, best performing hyperparameters could be mentioned.\n\t- The start of the sentence should be capitalized.\n", "summary_of_the_paper": "The paper proposes a novel molecular graph representation learning method by constructing a heterogeneous motif graph. In this graph, molecules and motifs are considered as nodes, and thus forming a heterogeneous graph. Motifs are extracted manually and important motifs are selected by TF-IDF. Moreover, the paper demonstrates that a multi-task learning framework is beneficial, thanks to sharing motifs across different datasets.", "summary_of_the_review": "Overall, I think this paper is at the borderline. Although the paper is among the first to use motif for molecular graph representation learning, the paper lacks sufficient literature survey. Moreover, it would have been better if the paper had had discussions about why motifs are helpful in a data-driven perspective.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9LPtAWsCqsD", "reviewer": "Reviewer_FtFN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The core strength of the paper is the strong experimental results. They obtained results that are much better than existing baselines on graph classifications. In terms of their ideas I find their TD-IDF approach natural, and I find the idea of taking into account molecular motifs very appealing and natural as well. \n\nThere are several concerns that I believe the authors should address for this paper to be of publishable quality: \n\n1. One of my major concerns is that the authors did not provide a sufficiently detailed/rigorous description of the HM-GNN model/any of the related procedures like the edge sampling. The high level description of the model is based on very two short paragraphs in section 2.3, as well as a graphical illustration in figure 3. Lots of details of the model are left out/undescribed. I expect at the very least some pseudocode to describe some details, i.e. the neural network architecture/training, the initialization etc. The GNN for the atom level embedding is also glossed over. I believe that for publication in a venue like ICLR, the authors should at the very least give a more precise and detailed description of the model that they are proposing. There is only a very short appendix provided and many details (e.g. details on the proportion that they used for edge sampling etc)  are nowhere to be found unless the reader delve into their code. \n\n2. Another one of my concerns is about the Edge sampling scheme. I have no doubts that it will actually speed things up (the complexity depends on the number of edges, so of course if we only select a subset of the edges the process is sped up). What the authors did not provide is any kind of justification, whether from a mathematical or from a molecular perspective, for why they picked this scheme/how they know their scheme isn't deleting important information? The authors propose to \"randomly select some molecular nodes as starting nodes\" (how many nodes? why random? Wouldn't some nodes be more important than others, especially in molecular settings?) and conduct a BFS where they sample a fixed portion of edges hop by hop (why fixed proportion? how does one pick the proportion?) I understand this is a heuristic that works well in practice but I expect a little more justification and details.  \n\nOverall, I will say that this is a paper with a lot of potential, and many of the ideas are likely to be useful in graph machine learning and molecular representation learning in general. However, the authors wrote the paper in a way where many of the core procedures are described in short paragraphs via words, and where lots of the important details, design decisions, hyperparameters etc are completely left out and are not sufficiently motivated/justified. Similarly ambiguous language is used in the experimental section. I would highly suggest the authors to include more precise descriptions and details in the paper, and to provide more justifications to their heuristics. ", "summary_of_the_paper": "This paper propose the Heterogeneous Motif Graph Neural Network (HM-GNN) which is based on a motif level graph representation that takes into account commonly occurring motifs like rings in molecules. They apply this to multi-task settings and obtained good experimental performances. They propose an edge sampling scheme to reduce the computational costs needed for training. ", "summary_of_the_review": "A paper with a lot of potential, but lots of important details are left out and lots of decisions are stated without sufficient justifications. As of right now, this paper is not in publishable state. After some revision however and some improvement in their exposition, I believe this can be a good paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7Gy3ryIgWL", "reviewer": "Reviewer_VxSx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The idea of constructing a motif-molecule heterogeneous graph is new;\n2. The experimental results show that the proposed method can improve the performance significantly;\n3. The paper is basically well-written and easy to follow.\n\nWeaknesses:\n1. Motif has been explored extensively in the literature, which makes the contribution of novelty for this paper is marginal;\n2. Though the idea of constructing a motif-molecule heterogeneous graph is new, it seem unnecessary to introduce motif nodes in this graph;\n3. The technical contribution of this paper includes three parts, but the three parts are not necessarily related to each other, which makes the paper very \"loose\" and more like a combination of small tricks;\n4. The part of edge sampling seems trivial;\n5. The authors do not test their method on some datasets that are commonly used for molecule property prediction, which makes it hard to comprehensively compare their method with the literature;\n6. Some of the technical details are not clearly stated (see below for detail);\n7. The quality of figures can be improved.\n\n\nDetails for weaknesses:\n1. This paper studies the problem of using motifs (functional groups) in molecule representation learning. However, it is well known that molecule motif is an essential part of molecule structure, which have been extensively explored in the literature.\n2. In the molecule-motif graph constructed by the authors, there are motif-motif links (co-occurrence relation) and motif-molecule links (part-whole relation), but no molecule-molecule links. However, motifs are more like a bridge connecting molecules in this big graph, and what the authors finally use is only molecule representation. If this is the case, it seems more direct and reasonable to build a molecule-molecule graph, whose edge weights are calculated based on the motifs they consists of, then the message can pass directly among molecules in GNN. I think the authors can at least take this as a baseline method? Moreover, it is also interesting to investigate if there is any theoretical relationship between the two graphs in terms of running GNN on them.\n3. The technical contribution of this paper consists of three parts: constructing a molecule-motif graph, multi-task learning framework, and edge sampling. However, they are not necessarily related to each other. In other words, even if you do not use the molecule-motif graph, you can still build another molecule-molecule graph and use your multi-task and edge sampling strategy on top of this graph. The three parts of contributions are loosely coupled together, making this paper more like a combination of separate tricks.\n4. The part of edge sampling seems trivial to me: of course you can randomly delete some edges from a graph to improve the time efficiency, but you do loss information from the original graph. Therefore, it is always a trade-off, which should be discussed in the paper. Moreover, it is also weird why the authors highlight a BFS method on sampling edges: It is easy to implement a method that directly removes edges from the graph while ensuring that all nodes have at least one edge left.\n5. There are a lot of molecule property datasets proposed in the literature, for example, BBBP, BACE, HIV, Tox21, ToxCast, SIDER, QM9, ESOL, FreeSolv, etc, but the authors do not test their methods on these commonly used datasets.\n6. Some of the technical details are not clearly stated: (1) In section 2.1 the authors say that \"By sorting the vocabulary by TF-IDF, we keep the most essential motifs as our final vocabulary\", but TF-IDF measures the importance of a motif to a specific molecule, and it is not an independent feature of motifs. How you get the TF-IDF value of motifs? Did you average the TF-IDF values of a motif to all molecules and treat this as the TF-IDF value of this motif? (2) In section 2.2, the authors construct a heterogeneous graph and define edge weights as in Eq. (1), however, PMI could be positive or negative, and TF-IDF can only be positive. How do you deal with negative PMI values in GNN? Should these two type of edge weights be normalized or calibrated to make sure that they are at the same scale in the unified graph?\n7. Though the figures in this paper are clear and easy to read, I suggest that the authors reorganize (do not leave too blanks), resize (make sure that texts and shapes across these figures are basically of the same size), and recolor the graphs (do not use colors that are too bright, try to color your graphs using the same style) to make them more beautiful and professional.", "summary_of_the_paper": "This paper proposes learning molecule representations by using motif-level information. The authors construct a heterogeneous graph which consists of molecules and motifs, then learn representations of them using graph neural networks. The learned features are concatenated with the molecule feature learned from a traditional atom-level graph neural networks and fed into an MLP for property prediction.", "summary_of_the_review": "In summary, I think the idea of using motif in molecule representation learning is interesting, but the propose method has a lot of design issues that are not well-justified or well-explained. Moreover, the proposed method is not tested on commonly-used datasets. Overall, I think this paper is below the borderline and I tend to reject this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a graph neural network for molecules which takes into account motif-level relationships. The paper received borderline reviews, with three reviewers voting for reject, and one for accept.  After the rebuttal, the reviewers did not change their scores. Overall, it seems that the paper has some merit, with good experimental results. Nevertheless, it suffers from two issues (i) the positioning with respect to other motif-based approaches is not clear enough, making the novelty hard to assess; (ii) there is a lot of room for improvement in terms of clarity. Therefore, the area chair follows the majority of the reviewers' recommendations and recommends a reject.
Decision: Reject