Fragment-based Pretraining and Finetuning on Molecular Graphs
Authors: Kha-Dinh Luong
Ambuj Singh
Venue: NeurIPS 2023 poster (2023)
Keywords: Self-supervised Learning
Graph Neural Network
Molecule
TL;DR: Pretraining GNNs on molecular graphs at fragment level.

Abstract: Property prediction on molecular graphs is an important application of Graph Neural Networks (GNNs). Recently, unlabeled molecular data has become abundant, which facilitates the rapid development of self-supervised learning for GNNs in the chemical domain. In this work, we propose pretraining GNNs at the fragment level, a promising middle ground to overcome the limitations of node-level and graph-level pretraining. Borrowing techniques from recent work on principal subgraph mining, we obtain a compact vocabulary of prevalent fragments from a large pretraining dataset. From the extracted vocabulary, we introduce several fragment-based contrastive and predictive pretraining tasks. The contrastive learning task jointly pretrains two different GNNs: one on molecular graphs and the other on fragment graphs, which represents higher-order connectivity within molecules. By enforcing consistency between the fragment embedding and the aggregated embedding of the corresponding atoms from the molecular graphs, we ensure that the embeddings capture structural information at multiple resolutions. The structural information of fragment graphs is further exploited to extract auxiliary labels for graph-level predictive pretraining. We employ both the pretrained molecular-based and fragment-based GNNs for downstream prediction, thus utilizing the fragment information during finetuning. Our graph fragment-based pretraining (GraphFP) advances the performances on 5 out of 8 common molecular benchmarks and improves the performances on long-range biological benchmarks by at least 11.5%. Code is available at: https://github.com/lvkd84/GraphFP.

Fragment-based Pretraining and Finetuning on Molecular Graphs
Kha-Dinh Luong, Ambuj Singh Department of Computer Science University of California, Santa Barbara Santa Barbara, CA 93106 {vluong, ambuj J&cs. ucsb. edu
Abstract
Property prediction on molecular graphs is an important application of Graph Neu- ral Networks (GNNs). Recently, unlabeled molecular data has become abundant, which facilitates the rapid development of self-superviee learning for GNNs the chemical domain. In this work, we propose pretraining GNNs the fragment level, a promising middle ground to overcome the limitations of node-level and graph-level pretraining. Borrowing techniques from recent work on principal sub- graph mining, we obtain compact vocabulary prevalent fragments from a large pretraining dataset. From the extracted vocabulary, we introduce several fragment- based contrastive and predictive pretraining tasks. The contrastive learning task jointly pretrains two different GNNs: one on molecular graphs and the other on fragment graphs, which represents higher-order connectivity within molecules. By enforcing consistency between the fragment embedding and the aggregated em- bedding of the corresponding atoms from the molecular graphs, we ensure that the embeddings capture structural information at multiple resolutions. The structural information of fragment graphs is further exploited to extract auxiliary labels for graph-level predictive pretraining. We employ both the pretrained molecular-basee and fragment-based GNNs for downstream prediction, thus utilizing the fragment information during finetuning. Our graph fragment-based pretraining (GraphFP) advances the performances on 5 out of 8 common molecular benchmarks and improves the performances on long-range biological benchmarks by at least 11.5%. Code is available at: https//gtthbb.com//kk4//////aagp
Introduction
The rise of Graph Neural Networks (GNNs) has captured the interest of the computational chemistry community [II]. Representing molecules and chemical structures as graphs beneficial because the topology and connectivity are preserved and can be directly analyzed during learning [13) leading to state-of-the-arr performance. However, there is a problem that while highly expressive GNNs are kata-hungry, the majority of molecular datasets available challenge to the generalization of GNNs. Following the steps that were taken in other domains such text and images, potential solution is to pretrain GNNs on large-scale unlabeled data via self-supervised pretraining 3 [5]. How to do so effectively still an open question since graphs complex than images or text, and the chemical specificity of molecular graphs needs to be considered when designing graph-based pretraining tasks.
considerably small, posing critical
A large number of pretraining methods have been proposed for molecular graphs 38 40). For
example, [15] uses node embeddings to predict attributes of masked nodes, [17] predicts graph structural properties, and [16] generatively reconstructs graphs via node and edge predictions. Another
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

45 46], in which multiple views of the same graph are mapped closer each other the embedding space. These views are commonly generated via augmentations to the original graphs R5 45 46]. However, without careful adjustments, there a great risk of violating the chemical validity or changing the chemical properties of the original molecular graph after applying the augmentations. Other methods such as (3 [1] avoid this problem by contrasting 2D-graphs against 3D-graphs. Unfortunately, privileged information such as 3D coordinates is often expensive to obtain. Node versus graph contrastive learning has also been investigated [2] however, this approach may encourage over-smoothing among node embeddings.
popular direction is contrastive learning [26 32 B5|
In general, the majority of works pretrain embeddings at either node-level or graph-level. Node- level pretraining may be limited to capturing local patterns, neglecting the higher-order structural arrangements while graph-level methods may overlook the finer details. As such, motif-based or fragment- based pretraining is a new direction that potentially overcomes these problems 27 48 49]. Still, existing fragment-based methods use either suboptimal fragmentation or fragmentation embeddings. GROVER (27 predicts fragments from node and graph embeddings, however, their fragments are k-hop subgraphs that cannot account for chemically meaningful subgraphs with varying sizes and structures. MICRO-Graph [48) contrastively learns subgraph embeddings versus graph embeddings; however, these embeddings may not effectively capture global patterns. MGSSL [49] utilizes graph topology via depth-fint first search or breadth-firg first search to guide the fragment- bageeett-baedd generation of molecules; however, their multi-step fragmentation may overly decompose molecules, thereby losing the ability to represent higher-order structural patterns.
In this paper, we propose GraphFP, novel fragment-level contrastive pretraining framework that captures both granular patterns and higher-order connectivity. For each molecule, we obtaín two representations, molecular graph and fragment graph, each of which is processed by separate GNN. The molecular GNN learns node embeddings that capture local patterns while the fragment GNN learns fragment embeddings that encode global connectivity. The GNNs are jointly trained Via a contrastive task that enforces consistency between the fragment embedding and the aggregated embedding of the corresponding atoms from the molecular graphs. Unlike previous work, we pretrain the fragment- based aggregation of nodes instead of individual nodes. With the fragment embeddings encoding the global patterns, contrasting them with the aggregation of node embeddings reduces the risk of mer-smothhin and allows ficxibility in learning the appropriate node latent space. Since aggregation of nodes in this context also be considered kind of fragment representation, our framework essentially contrasts views of fragments, each of which captures structural information at different scale. Moreover, as the molecular graphs and the fragment graphs are chemically faithful, our framework requires no privileged information or augmentation.
Exploiting the prepared fragment graphs, we further introduce two predictive pretraining tasks. Given a molecular graph as input, the molecular GNN is trained to predict structure-related labels extracted from the corresponding fragment graph. These tasks enhance the structural understanding of the molccular GNN and can be conducted together with contrastive pretraining.
To generate vocabulary of molecular fragments, we utilize Principal Subgraph Mining 20 to extract optimized prevalent fragments that span the pretraining dataset. Compared to fragmentations used in previous works, this approach can produce a concise and diverse vocabulary of fragments, each of which is sufficiently frequent without sacrificing fragment size.
We evaluate GraphFP on benchmark chemical datasets and long-range biological datasets. Inter- estingly, both the molecular and the fragment GNNs can be used in downstream tasks, providing enriched signals for prediction. Empirical results and analysis show the effectiveness of our proposed methods. In particular, our pretraining strategies obtain the best results on out of 8 common chemical benchmarks and improve performances by 11.5% and 14% on long-range biological datasets.
Related Works
2.1 Representation Learning on Molecules
Representation learning on molecules has made use of fixed hand-crafted representations such as descriptors and fingerprints (0|43. string-based representations such as SMILES and InChl [2 2x and molecular images [47. Currently, state-of-the methods rely on representing molecules as graphs and couple them with modern graph-based learning algorithms like GNNs [[]. Depending

---

on the type of compounds (small molecules, proteins, crystals), different graph representations Compared to graph-based representation, molecular descriptors and fingerprints cannot encode structural information effectively while string-based representations require an additional layer of syntax which may significantly complicate the learning problem.
can be constructed [TP 39
2.2 Graph Neural Networks
Given graph G = (V,E) E) with node attributes F, for v E V and edge attributes cuv for (u, E, GNNs learn graph embedding hG and node embedding he At each iteration, a node updates its embedding by gathering information from its neighborhood, including both the neighboring nodes and the associated edges. This process often involves an aggregating function Mj and an updating function Us [II. After the k-th iteration (layer), the embedding of node v is, he Uh M4 (E 1 1e(
N())))), where ((e) the set of neighbors of v and (0) Tv. The aggregating function M& pools the information from e's neighbors into an aggregated message. Next, the updating function Us updates the embedding of v based on its previous embedding h ((-1) and the aggregated message. An additional readout function R combines the final node embeddings into graph embedding he R((h |u E V)), where K is the number of iterations (layers). Since there is usually no ordering among the nodes in a graph, the readout function R often chosen be order-invariant [4I]. Following this framework, variety of GNNs has been proposed 12
eu (uv))l
some specifically for molecular graphs 9 (29]
2.3 Pretraining on Graph Neural Networks
To alleviate the generalization problem of graph- based learning in the chemical domain, graph in order to take advantage of 45 46
pretraining has been actively explored [15] [3] 26
31
32
35
large databases of unlabeled molecules 100
In terms of the pretraining paradigm, existing methods can be categorized into being predictive contrastive (23 26
generative [16 49]. Predictive methods often require moderate to large labeled datasets for pretraining. In the case of an unlabeled large dataset, chemically or topologically generic labels can be generated. Despite the simple setup, predictive methods are prone to negative transfer [15). On the other hand, contrastive methods aim to learn robust embedding space by using diverse molecular views 23
15 27 77
31
35
45
46),
46 Generative methods
intend to learn the distribution of the components constituting molecular graphs 16
49
At the pretraining level, methods can be sorted into node-level [7). graph-level [5
23
26
31
35 45 46), and more recently motif-level [27 48 49 Node-level methods only learn chemical
semantic patterns the lowest granularity, limiting their ability capture higher-order molecular arrangements, while graph-level methods may miss the granular details. Motif-based or fragment- based pretraining has emerged possible solution to these problems (7 48 49 : however, existing methods use suboptimal fragmentation or suboptimal fragment embeddings. In this work, we learn fragment embeddings that effectively capture both local and global topology. We exploit the fragment
information at every step of the framework, from pretraining to finetuning.
3 Methods
In this section, we introduce the technical details of our pretraining framework GraphFP. We begin with discussing molecular fragmentation and fragment graph construction based on an extracted vocabulary. Then, we describe our fragment-based pretraining strategies, including one contrastive task and two predictive tasks. Finally, we explain our approach for combining pretraining strategies and the pretrained molecular-based and tragment-basee GNNs for downstream predictions.
3.1 Molecule Fragmentation
Graph fragmentation plays a fundamental role the quality of the learning models because it dictates global connectivity patterns. Existing methods rely on variations of rule-based procedures such as BRICS 4 or RECAP 21). Though chemistry-inspired, the extracted vocabulary is often large, to the order of the size of the pretraining dataset, and contains unique or low- cow-frequency fragments,
the
3

---

Molecular Graph
(())
M
GNNM
FragPool
InfoNCE
(8) F
G
GNNF
Fragment Graph
Negartve
Palis
Palis
Figure I: Fragment- based contrastive pretraining framework. GNN As processes molecular graphs while G processes fragment graphs. The fragment-basse pooling function FRAGPOOL ag- gregates node embeddings into a combined embedding that forms a positive contrastive pair with the corresponding fragment embedding. Notice that the two -OH groups, blue and light pink, considered distinct. Therefore, the aggregated node embedding corresponding to the blue fragment and the embedding of the light pink fragment form negative pair.
posing a significant challenge for pattern recognition. To overcome these drawbacks, some works further break down fragments [49], even to the level of rings and bonds [18]. However, reducing the sizes of fragments forfeits their ability to capture higher-order arrangements.
3.1.1 Principal Subgraph Extraction
In this work, borrow the Principal Subgraph Mining algorithm from 20 alleviate the above- mentioned problems. Given graph G (V, F),, subgraph of G is defined V C and É C E. In vocabulary of subgraphs. S is principal subgraph if VS' intersecting with S in any molecule, either S' C S or c(S") < c(S) where c(-) counts the occurrences of a fragment among molecules. Intuitively, principal subgraphs are fragments with both larger sizes and more frequent occurrences. The algorithm heuristically constructs vocabulary of such principal subgraphs via few steps:
(V, R.). where
Initialize:
Initialize the vocabulary with unique atoms.
Merge:
For each molecular graph, for all pairs of overlapping fragments the graph, merge the fragment in the pair and update the occurrences of the resulting combined fragment.
Update: Update the vocabulary with the merged fragment with the highest occurrence. Repeat the last two steps until reaching the predefined vocabulary size.
For more detailed description, we refer readers 20
We investigate the effect of vocabulary sizes
on the performance of pretrained models in Section 4.4 and Appendix C Similarly, we investigate
the effect of fragmentation strategies in Section 4.2
3.1.2 Fragment-graph Construction
Given an extracted vocabulary of principal fragments, for each molecular graph, we construct
corresponding fragment graph. Let F' = {{(()) 5(1) ...., ....,((m)) S(m) be the fragmentation of a molecular
graph GM (VM. EM), where g(i) = (V()) E(t) is fragment subgraph, (((1) n (70) 0. and U (i) VM- We denote the fragment graph as Gr = (Vr, Er), where Vvl = FF] and each node ( Vr corresponds to a fragment S(1) An edge exists between two fragment nodes of G if there exists at least bond interconnecting atoms from the fragments. Formally,

---

Ef- = =(i,))) v, edge features the fragment graph. As a result, fragment graph purely represents the higher- order connectivity between the large components within molecule. Fragment node features are embeddings from an optimizable lookup table. Vocabulary and fragment graphs statisties are provided in Appendix
V E VO), (u E E}. For simplicity, in this work, we withhold
3.2 Fragment-based Contrastive Pretraining
Figure Illustrates our contrastive framework. We define two separate encoders, GNNM M and GNN GNN M processes molecular graphs and produces node embeddings while GNN P processes fragment graphs and produces fragment embeddings. Since GNNs can capture structural information, the node embeddings encode local connectivity of neighborhoods surrounding atoms. Similarly, the fragment embeddings encode global connectivity and positions of fragments within the global context.
We apply contrastive pretraining at the fragment level. Contrastive pretraining learns robust latent spaces by mapping similar views closer to each other in the embedding space while separating dissimilar views. Learning instances are processed in pairs, in which similar views form the positive pairs while dissimilar views form the negative pairs. In this case, a positive pair consists of fragment from a fragment graph and the collection of atoms constituting this fragment from the corresponding molecular graph. On the other hand, a fragment and any collection of atom nodes constituting different fragment instance form a negative pair. Notice that different occurrences of the same type of fragment in the same or different molecules are considered distinct instances because when structural contexts are taken into account, the embeddings of these instances are dissimilar. The -OH groups in Figure M illusttatee one such case.
To obtain the collective embedding of atom nodes corresponding to a fragment, we define a function FRAGPOOL(.) that combines node embeddings. Contrasting the combined embedding of nodes against the fragment embedding allows flexibility within the latent representations of individual nodes. Intuitively, the learning task enforces consistency between fragment embeddings and collective node embeddings, incorporating higher-order connectivity information. Through the learning process, the information is optimally distributed among the nodes so that each node only holds a part of this collective knowledge, Arguably, such setup is semantically reasonable because nodes and fragments represent different orders of connectivity. A node does not necessarily hold the same structural information as the corresponding fragment. Instead, it is sufficient for a group of nodes to collectively represent such higher-order information. Essentially, the fragment embeddings and the collective embeddings of atom nodes can be considered different views of molecular fragments.
Extending the notations in 3.1.2 given a pretraining dataset of molecular graphs DMs = (G G!3) , ",,),,.,,h,,, G.N)), ). obtain
of fragmentations F (F(1), p(2) F(N]} and a set of corresponding fragment graphs Dr {GP , ".."....G+)) with M = (V/6) E and G( = (v ) More specifically, v(()) is the set of atom nodes in the molecular graph Gl and ((( is the set of fragment nodes in the fragment graph G.. For the i-th molecule, let M Ixd and H E RIV/"xxd be the final node embeddings and fragment embeddings after applying GNN and GNNp, respectively, i.e:
H GNN M (V FESi)
H GNN GV Ef")
We further compute the fragment- based aggregation of node embeddings:
H A
FRAGPOOL F())),
(3)
where 110) E 1 Ixd has the same dimensions those of H The r-th rows, h (10)) E H and E E H from both matrices are embeddings of the same fragment from the original molecule.
In our contrastive framework, ( is the positive example for the anchor h (6) The negative learning examples are sampled from:
X (H # #ivg
(4)
5

---

fragment Existence Prediction
GNNM
Tree Structure Prediction
Figure 2: Fragment-base based predictive pretraining. GNN M processes molecular graphs and produces graph-level embeddings used for prediction. The right upper box shows unique fragments that exist the input molecule while the right lower box other backbones are also visualized for comparison.
shows the ground-truth structural backbone. few
We minimize the contrastive learning objective based the InfoNCE loss
25
exp((h (2) FF.)
Lc = -Eir
(5)
(8) exp(( A,r P. NXC exp (h Am k)))
3.3 Fragment-based Predictive Pretraining
We further define two predictive pretraining tasks of which labels are extracted from the properties and the topologies of the fragment graphs. Only the molecular GNN is pretrained in this case. The labels provided by the fragment graphs guide the graph-based pretraining of the molecular GNNs, encouraging the model to learn higher-order structural information.
Fragment Existence Prediction A multi-label prediction task that outputs vocabulary-size binary vector indicating which fragments exist in the molecular graph. Thanks to the optimized fragmentation procedure (20) that we use, the output dimension is compact without extremely rare classes or fragments, resulting in more robust learning.
Fragment Graph Structure Prediction We predict the structural backbones of fragment graphs. The number of classes is the number of unique structural backbones. Essentially, a backbone is a fragment graph with no node or edge attributes. Graphs share the same backbone if their fragments are arranged similarly. For example, fragment graphs in which three fragments connect in a line correspond to the same structural backbone, line graph with three nodes. This task also benefits from the optimized fragmentation. As the extracted fragments are sufficiently large, the fragment graphs are small enough that the enumeration of all unique backbones feasible.
The predictive tasks pretrain the graph embedding hG- In particular, one task injects local community information of fragments while the other task injects the arrangements of fragments into he- We train both tasks together, optimizing the objective C. = Lr, + Lr,, where Lr, and Lr₃ are the predictive objective of each task. The predictive tasks are illustrated in Figure
We can combine the predictive pretraining with the contrastive pretraining presented Section 3.2 We found parallel pretraining more effective than the sequential pretraining described in [15). With being the weight hyperparameter, the joint pretraining objective then:
C = aLp +1 a)Cc
(6)
3.4 Combining Models based on Fragment Graphs and Molecule Graphs
We further propose utilize both the molecule encoder GNN M and the fragment encoder GNN F for downstream prediction. Following the procedure described in Section we can easily fragment and extract molecular graph and fragment graph representations from any unseen molecule. Such convenience is not always the case. For example, existing works that contrast 2D and 3D molecular graphs 23 3il can only finetune the 2D encoder since 3D information expensive to obtain for unseen molecules. Other works that rely graph augmentation
[35 45 46) cannot utilize the

---

3 The pretrained models are augmented views as signals for downstream prediction since they are not faithful representations of the original molecule. In contrast, the representations extracted by our framework are faithful views that do not require expensive information other than the vanilla molecular structure.
Let has and hr be the graph embeddings produced by GNN as and GNNr, respectively. We obtain the downstream prediction by applying fully-connected layer on the concatenation of heyr and hp.
Experiments
4.1 Experimental Settings
We pretrain GNNs according to the process discussed in section evaluated on various chemical benchmarks.
Datasets We use a processed subset containing 456K molecules from the ChEMBL, database for pretraining. A fragment vocabulary of size 800 is extracted as described in Section To ensure possible fragmentation of unseen molecules, we further complete the vocabulary with atoms not existing in the pretraining set, totaling 908 unique fragments. For downstream evaluation, we consider binary graph classification tasks from MoleculeNet (36] with scaffold split [15. Moreover, to assess the ability of the models in recognizing global arrangement, we consider two graph prediction tasks on large peptide molecules from the Long-range Graph Benchmark 77]. Long-range graph benchmarks are split using stratified random split. More information regarding datasets is provided
in Appendix
Models For graph classification benchmarks, we model our molecular encoders GNN with the 5-layer Graph Isomorphism Network (GIN) 4 as in previous works [15] 23], using the same featurization. Similarly, we model the fragment encoder GNN with a shallower 2 2-layer GIN since fragment graphs are notably smaller. For long-range benchmarks, both GNN and GNN F are GIN with 5 layers, with the featurization of molecular graphs based on the Open Graph Benchmark (14) All encoders have hidden dimensions of size 300. For more details, please refer to Appendix A
Pretraining and Finetuning All pretrainings are done in 100 epochs, with AdamW optimizer, batch size 256, and initial learning rate 1 x We reduce the learning rate by a factor of 0.1 every 5 epochs without improvement. We use the models at the last pretraining epoch for finetuning. On graph classification benchmarks, to ensure comparability, our finetuning setting is mostly similar to that of previous works [15] 23): 100 epochs, Adam optimizer, batch size 256, initial learning rate 1 x 10-3, and dropout rate chosen from (0.0, 0.5). We reduce the learning rate by a factor of 0.3 every 30 epochs. On long-range benchmarks, the setting is similar except that we finetune for 200 epochs and factor the learning rate by 0.5 every 20 epochs without improvement. We find that using averaging as FRAGPOOL(-) works well. All experiments are run on individual Tesla V 100 GPUs.
Baselines We compare to several notable pretraining baselines, including predictive methods (AttrMask& & ContextPred [5]. G-Motif & G-Contextual (GROVER) (27). generative method (GPT- GNN (6)), contrastive methods (GraphLoG (4), GraphCL 46). JOAO, JOAOvs 451.. contrastive
method with privileged knowledge (GraphMVP (23), and fragment-based method (MGSSL [49]
For long-range prediction, we compare our method with popular GNN architectures: GCN [9]
GCNII 2 GIN [1), and GatedGCN II with and without Random Walk Spatial Encoding 6
Our goal is to benchmark the quality of our proposed pretraining methods with existing work. The settings in our experiments are chosen to fulfill this objective. In general, our settings follow closely those from previous works ensure comparability. Specifically, we use the same embedding model (5-layer GIN), featurization, and overall similar hyperparameters and amount of pretraining data as 49). We limit the amount of hyperparameter tuning for the same reason. In general, self supervisd learning and property prediction on molecular graphs are important research topics and a wide variety of methods have been proposed terms of both the pretraining task and the learning model, resulting in impressive results X 50 51. Some of these performances stem from deeper neural network design, extensive featurization, and large-scale pretraining. Pretraining at the scale of [8] 51] requires hundreds of GBs or even a TB of memory.
those in the baselines 15 23
7

---

Table I: Test ROC-AUC on binary molecular property prediction benchmarks using different pre- training strategies in GraphFP. The top-3 performances on each dataset are shown red color, with red being the best result, red being the second best result, and red being the third best result. The 2 rightmost column shows the average performance ranking (lower value means better ranking) and the average AUC. The last 5 rows show the performances of our methods, with C, P, and F indicate contrastive pretraining, predictive pretraining, and inclusion of fragment encoders in downstream prediction, respectively.
Pretraining Strategies
BBBP
Tox21
ToxCast
SIDER
ClinTox
MUV
HIV
BACE
Avg. Rank
Arg. AUC
AusMasking
[5]
64.3 a 2.8
76.7 s 0.4
6.5
6.70
7 1.1
74.7 1.4
77.2 1.1
79.3 &
7,88
71.15
ContexiPred
68.0
75.7
0.
1
1.0
G-Matif
2+
66.9
0.7
6..3
61
73.0 2.7
73.8
73.0
14,25
70.16
G-Contextual
27
69.9
0.5
0.78
58.7110
60.6.:22
1.71
76.3 1.5
79.3 L1
11.8
69.34
GPT-GNN
16
64.5
74.90..
62.
58.1
58.1552
2.
6.1
1.2
13.63
67.16
Graphlaxi
75.1+10
624+02
1.5
653+12
GraphCC.
6.7
0.7
605+09
76
2.7
1.2
75.411
12.13
70.78
10A0 [S]
70.2 1.0
0.3
02.9
000008
21.
71.7+14
76.7=1.
01.
9.56
71.89
JOAO+2 45
0.9
7..3
0.52
0.55
73.7 1.
77.5
1.2
75.5 1.3
8.94
72.14
GraphMMP
6.25+0.2
0.
62.7+0.1
62,3+16
2.
1.
748
76.8-11
71.70
MGSSL
57.
73.2
75.7
10.94
GraphiFP JTc
0.55.00.
0.5
0..6
12.0
4..
76.0
75.6
1.0
79.7
6.13
72.66
GraphFFPJTer
70.2 1.7
0.88
6.
1.
4..9
13
73.0 1.9
74.2
13.56
70.21
GraphiPP:
L.
63.8 0.4
61.4099
17 7.76
76.3 1.0
5.50
72.81
GraphFP,
6..2
63.2
59.310
71.5
76.7
GraphFPop
71.3 1.7
0.5
64.7
61
399
76.6 1.8
76.3
81.3 2.2
5.19
72.59
GraphFPPr
70.1 1.8
0.3
0.8
64.7
87.2
74.5 188
2.0
2.1
7.25
73.73
GraphFPoPP
22.0 1.7
0.77
6.99
63
75.1 1.9
1.
80.5 1.8
4.56
74,01
Due to limited resources, we leave the evaluation of GraphFP and other baselines when pretrained on such larger datasets for future work.
4.2 Results on Graph Classification Benchmarks
Table reports our results on chemical graph classification benchmarks. For each dataset, we report the mean and error from 10 independent runs with predefined seeds. Except for GraphLoG
42
and MGSSL [49], the results of other baselines are collected from the literature [15 23
37
45 46
To ensure comprehensive evaluation, we conduct experiments with all possible combinations of the proposed strategies, which include contrastive pretraining (denoted as C)., predictive pretraining (denoted P). and inclusion of fragment encoders in downstream prediction (denoted as F). Because requires C, all possible combinations of these components are (C, CP CF, CPF), with the corresponding pretrained models GraphFPc. GraphFP,,. p. GraphFP GraphFP and GraphFPerr- For GraphFPors we choose a = 0.3 and for GraphFPepr, we choose a = 0.1. The choices of a are reported in Appendix A We also compare GraphFP-JTc and GraphFP-JTcr, which are variations of our models pretrained with the fragmentation from [18. The fragments used by 18] are generally smaller, resulting in larger fragment graphs. We found 5-layer GIN models encode these larger graphs better. Our models are competitive in all benchmarks, obtaining the best performance in out of 8 downstream datasets. We report the average ranking and the average AUC of the models. As reported in Table ! the GNNs with our proposed tragment-basee pretraining and finetuning strategies achieve the best average rankings and average AUCs across baselines. Moreover, adding the strategies successively, i.e. C, C.P.P and CPF, improves the average downstream rankings, confirming the individual effectiveness of each strategy in supporting the learning on molecular graphs. We attribute the remarkable performances of our methods to the capability of the pretrained embeddings and the fragment graph encoder in capturing higher-order structural patterns. For instance, the BBBP benchmark requires predicting blood-brain barrier permeability, where the molecular shape, size, and interaction with the transporting proteins play significant roles in determining the outcome. Capturing such information necessitates deep understanding of the global molecular structure, which is the goal our pretraining strategies.
4.3 Results on Long-range Chemical Benchmarks
In Table we compare fragment-based pretraining and finetuning of GraphFP with GNN baselines on two long-range benchmarks: PEPTIDE-FUNC containing 10 classification tasks regarding peptide functions and PEPTIDE-STRUCE containing five regression tasks regarding 3D structural information [7] Because fragment graphs of peptides are extremely large for effective extraction of structural backbones, we exclude predictive pretraining in this experiment. On both benchmarks, the results
8

---

Table 2: Performances on PEPTIDE-FUNC (graph classification) and PEPTIDE-STRUCE (graph re- gression). These tasks require capturing long-range interactions within large peptide molecules. Best performances are colored red.
Peptide-func Test AP
Peptide-struct Test MAE
Methods
GCN
0.3930 + 0.0023
0.3496 + 0.0013
GCNII
0.5543 0.0078
0.3471 0.0010
GIN
0.5498 0.0079
0.3547 0.0045
GatedGCN
0.5864 0,0077
0.3420 0,0013
GatedGCN+RWSSE
0.6069 0.0035
0.3357 = 0.0006
GraphFPer
0.6267 + 0.0073
0.3137 + 0.0019
Table 3: Effects on varying the size of the vocabulary
Fragmentation
ROC-AUC
Models
Avg Frag Size
Avg Frag Graph Size
SIDER
ClinTox
HIV
GraphFP s00
2.40
4.61
61.4 0.9
78.6 2
76.3 + 1.0
GraphFP 600
2.47
4.15
60.4 0.5
76.4
75.3 + 1.6
GraphFP3220n
2.53
3.73
60.6 + 0.5
7.95
75. + 1.3
show that the proposed method outperforms other GNN baselines, including GatedGCN+RWSE with positional encoding. In particular, our pretrained model GraphFP obtains about 14% improvement on PEPTIDE-FUNC and 11.5% improvement on PEPTIDE-STRUCE compared to vanilla GIN. As the tasks require recognizing long-range structural information, these improvements indicate the effectiveness of our strategies in capturing the global arrangement of molecular graph components.
4.4 Analysis
Pretraining with Various GNN Architectures In Table report the performances of some other GNN architectures besides GIN. The pretraining and finetuning conditions are similar to those described in Section I.T and Table All architectures share the same input features, hidden dimensions, and the number layers. On average, GIN, being the most expressive GNN among the shown, performs the best. This observation agrees with the conclusions from previous works 15
Figure 3: (left) I-SNE plot showing the embeddings of common fragment. Each color corresponds structural backbone. (right) I-SNE plot showing the embeddings of atoms in a molecule with colors distinguishing fragment membership. Round markers indicate atom embeddings while square markers indicate fragment-ban based pooling of atom embeddings.

---

Table 4: Downstream performances with models pretrained with various GNN architectures.
Architectures
BBBP
Tox21
ToxCast
SIDER
ClinTox
MUV
HIV
BACE
Avg. Rank
GIN
0.8
0.3
6.4
61.3 0.4
63.2 1.0
7..2220
2 74.6
82.5 1,9
188
GCN
70.5
7.33+0.3
6,,3
61.5
78.1
778
76.2211
2.13
GraphSage
1.
0.6
0.3
6.8
69.6
13
77.1
7.60
2,00
Effects on Varying the Size of the Vocabulary The size of the fragment vocabulary likely influences the quality of pretraining and finetuming since it dictates the resolution fragment graphs. To investigate such effects, we prepare two additional vocabularies with size 1, 600 and size 3, 200. We repeat the same contrastive pretraining as in Section 4.1 using the new vocabularies. As the vocabulary size increases, more unique and larger fragments are discovered, increasing the average fragment size and reducing the average fragment graph size. Table TO shows that the performances worsen on some downstream benchmarks the vocabulary size grows larger. We conjecture few possible reasons. Firstly, a larger vocabulary means more parameters to optimize. Second, smaller fragment graphs represent excessively loose view of the graph, resulting in loss of structural information. In general, vocabulary size is an important hyperparameter that greatly affects the quality of self-supervised learning. In (20), the authors gave discussion selecting optimal vocabulary size according to the antropy-sparsiie trade-off.
Visualizing the Learned Embeddings As shown in Figure via t-SNE B3 The left plot illustrates the embeddings produced by the pretrained fragment encoder GNN of a common fragment. Each dot corresponds to molecule in which the fragment appears. The color of a dot indicates the structural backbone of the molecule it represents. For visibility, we only show the most common backbones. The embeddings the structural backbones, indicating that the fragment embeddings capture higher-order structural information. Notice that the fragment encoder GNN is not directly trained to recognize structural backbones. The right plot shows the embeddings produced by the molccule encoder GNN M of atoms within molecule. The embeddings of atoms within the same fragment are clustered together. Interestingly, some clusters, shown in green, purple, and black, are arranged similarly to how they appear in the original molecule. This observation confirms that the collective embedding of nodes can capture higher-order connectivity.
visualize the learned embeddings
reasonably separated according
5 Conclusions and Future Work
In this paper, we proposed contrastive and predictive learning strategies for pretraining GNNs based on graph fragmentation. Using optimized fragment vocabulary, we pretrain two separate encoders for molecular graphs and fragment graphs, thus capturing structural information at different resolutions. When benchmarked chemical and long-range peptide datasets, our method achieves competitive or better results compared to existing methods. Moving forward, we plan to further improve the pretraining via larger datasets, representations. We also plan to extend the fragment-based techniques to other learning problems, including novel molecule generation and interpretability.
extensive featurizations, better fragmentations, and more optimal
Acknowledgments and Disclosure of Funding
This project is supported by the BioPACIFIC Materials Innovation Platform of the National Science Foundation under Award # DMR-193348" and high-perffrmance computing infrastructure from the California NanoSystem Institute at the University of California Santa Barbara.
References
[1] Xavier Bresson and Thomas Laurent. Residual gated graph convnets. arXiv preprint arXiv:1711007533, 2017.
[2] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In Proceedings of the 37th International Conference on Machine Learning, 2020.
10

---

Pre-training [3] Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference Machine Learning. 2020.
[4] Jörg Degen, Christof Bressheid--Grlacee Andrea Zaliani, and Matthias Rarey. On the art of compiling and using drug-like chemical fragment spaces. ChemMedChem: Chemistry Enabling 2008.
Drug Discovery,
[5] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT:
of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, Volume I (Long and Short Papers), 2019.
[6] Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph neural networks with learnable structural and positional representations. In International Conference on Learning Representations, 2022.
[7] Vijay Prakash Dwivedi, Ladislav Rampášek, Michael Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long range graph benchmark. Advances in Neural Information Processing Systems, 2022.
[8] Xiaomin Fang, Lihang Liu, Jieqiong Lei, Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. Geometry-enhanced molecular representation learning for property prediction. Nature Machine Intelligence, 4(2):127-134 2022.
[9] Johannes Gasteiger, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations, 2020.
[10] Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvnnne Light, Shaun McGlinchey. David Michalovich, Bissan Al-Lazikani, et al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic Acids Research, 40(DI):D1100- DI 2012.
[11] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning. 2017.
[12] William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, 2017.
[13] Huahai He and Ambuj K. Singh. Query language and access methods for graph databases. In Charu C. Aggarwal and Haixun Wang. editors, Managing and Mining Graph Data, volume 40 of Advances Database Systems, pages 125-160. Springer, 2010.
[14] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs Advances in Neural Information Processing Systems, 2020.
[15] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for vre-training graph neural networks. In International Conference on Learning Representations, 2020.
[16] Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre- praitrannnn of graph neural networks. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
[17] Dasol Hwang, Jinyoung Park, Sunyoung Kwon, KyungMin Kim, Jung-Woo Ha, and Hyun- woo Kim. Self-supervised auxiliary learning with meta-paths for heterogeneous graphs. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 10294-10305. Curran Associates. Inc.. 2020.
[18) Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International Conference on Machine Learning, 2018.
[19] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
[20] Xiangzhe Kong, Wenbing Huang, Zhixing Tan, and Yang Liu. Molecule generation by principal subgraph mining and assembling. Advances in Neural Information Processing Systems, 2022.
11

---

[21] Xiao Qing Lewell, Duncan B Judd, Stephen P Watson, and Michael M Hann. Recap retrosyn- thetic combinatorial analysis procedure: a powerful new technique for identifying privileged molecular fragments with useful applications in combinatorial chemistry. Journal of Chemical Information and Computer Sciences, 38(3):11--522, 1998.
[22] Juncai Li and Xiaofei Jiang. Mol-bert: An effective molecular representation with bert for molecular property prediction. Wireless Communications and Mobile Computing, 2021:1-7, 2021.
[23] Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre-training molecular graph representation with 3d geometry. In International Conference on Learning Representations, 2022.
[24] Andreas Mayr, Günter Klambauer, Thomas Unterthiner, Marvin Steijaert, Jörg K Wegner, Hugo Ceulemans, Djork-Arné Clevert, and Sepp Hochreiter. Large-scale comparison of machine learning methods for drug target prediction on chembl. Chemical Science, 4(44):541--541, 2018.
[25] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.0378333 2018.
[26] Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gec: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
[27] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems, 2020.
[28] Jerret Ross, Brian Belgodere, Vijil Chenthamaraksham, Inkit Padhi, Youssef Mrouch, and Payel Das. Large-scale chemical language representations capture molecular structure and properties. Nature Machine Intelligence, 4 4(12):1256-126) 2022.
[29] K. T. Schütt, H. E. Sauceda, P.-J. Kindermans, A. Tkatchenko, and K.-R. Müller. SchNet A deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24), 03 2018. 241722.
[30] Jie Shen and Christos A Nicolaou. Molecular property prediction: recent trends in the era of artificial intelligence. Drug Discovery Today: Technologies, 32-33:29-36, 2019.
[31] Hannes Stärk, Dominique Beaini, Gabricle Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Lió. 3D infomax improves GNNs for molecular property prediction. In Proceedings of the 39th International Conference on Machine Learning. 2022.
[32] Fan-Yur Yun Sun, Jordan Hoffman, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations, 2019.
[33] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(1 1), 2008
[34] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.
[35] Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):279- 287, 2022.
[36] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical Science, 9():5133-330 2018.
[37] Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z. Li. Mole-BERT: Rethinking pre-training graph neural networks for molecules. In International Conference on Learning Representations, 2023.
[38] Jun Xia, Yanqiao Zhu, Yuanqi Du, Yue Liu, and Stan Li. A systematic survey of chemical pre-trained models. International Joint Conference Artificial Intelligence, 2023.
12

---

[39] Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical Review Letters, 120(14):14530. 2018.
[40] Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: A unified review. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(2):2412-2429, 2023.
[41] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference Learning Representations, 2019.
[42] Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph- level representation learning with local and global structure. In Proceedings of the 38th International Conference on Machine Learning, 2021.
[43] Ling Xue and Jurgen Bajorath. Molecular descriptors in chemoinformatics. computational combinatorial chemistry, and virtual screening. Combinatorial Chemistry & High Throughput Screening, 3(5):363-372, 2000.
[44] Ronghui You, Shuwei Yao, Hiroshi Mamitsuka, and Shanfeng Zhu. Deepgraphgo: graph neural network for lcage-scale, multispecies protein function prediction. Bioinformatics, (Suppee ment 1):1262- -2221, 2021.
[45] Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In Proceedings of the 38th International Conference Machine Learning, 2021.
[46] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 2020.
[47] Xiangxiang Zeng, Hongxin Xiang, Linhui Yu, Jianmin Wang, Kenli Li, Ruth Nussinov, and Feixiong Cheng. Accurate prediction of molecular properties and drug targets using self- supervised image representation learning framework. Nature Machine Intelligence, 4(1 4(11):1004 1016, 2022.
[48] Shichang Zhang, Ziniu Hu, Arjun Subramonian, and Yizhou Sun. Motif-driven contrastive learning of graph representations. arXiv preprint arXiv:2012.12553: 2020.
[49] Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee- Lee. Motif-based graph self-supervised learning for molecular property prediction. In Advances in Neural Information Processing Systems, 2021.
[50] Jinhua Zhu, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Wengang Zhou, Houqiang Li, and Tie Yan Liu. Unified 2d and 3d pre-training of molecular representations. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2626-2633 2022.
[51] Jinhua Zhu, Yingce Xia, Lijun Wu, Shufang Xie, Wengang Zhou, Tao Qin, Houqiang Li, and Yan Liu. Dual-view molecular pre-training- In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3615-3627, 2023.
13

---

A Experimental Settings
A.1 Graph Isomorphism Network
Given graph = (V, F) with node attributes T, for v E V and edge attributes Cuv for (u, E F. after the k-th layer of the GNN, the embedding of node is updated as:
h U2(h MMM(((h (((=1), Ewo)le N())))),
where (v) is the set of neighbors of e and 1 = In. The graph embedding lag is obtained via:
hG R( E)),
(8)
where K is the number of layers and R is the readout function. The overall graph embedding he is used for graph-level prediction, as in Section 4.1 In our work as well as some previous works, often used as the underlying
Graph Isomorphism Network (GIN) with edge encoding [5 architectures of learning models. In particular:
23 42
))
4 ReLU MLP(K)
Σ
h
1)-114
Σ
uEN(ojute}
eEf(u,e) (EN())U(
where h/ is the learnable edge embedding the k-th layer. The mean readout is used obtain the graph embedding:
ha = (p
(10)
A.2 Model and Training Configurations
The configurations of GNN M and GNNF provided in Table The pretraining and finetuning setups are provided in Table 6 general, our parameters are comparable to those of previous works with which we compare much fewer than that in g 50 from {0.0, 0.5).
[5 23 42 Note that the number of node features and edge features is On downstream tasks, we only finetune the dropout rate, selected
Table 5: Model Configuration
Methods
Parameters
Values
Convolution type
GIN
Number of atom features
Number of atom features for Long-range benchmarks
9
GNN
Number of edge features
2
Number of edge features for long-range benchmarks
3
Dimension of hidden embeddings
300
Aggregation
SUM
Number of layers
5
Readout
MEAN
Convolution type
GIN
Number of fragment features Dimension of hidden embeddings
1
300
GNN
Aggregation
SUM
Number of layers
Number of layers for long-range benchmarks Readout
5
MEAN
A.3 Combination of Predictive and Contrastive Pretraining
We combine the contrastive pretraining and predictive pretraining proposed in Section 3 by optimizing the joint objective function:
L = ==aLp + -a)fe,
(11)
14

---

Table 6: Training Configuration
Phases
Hyperparameters
Values
Number of epochs
100
Batch size
256
Optimizer
AdamW
Weight decay
0.01
Pretraining
Learning rate
0.001
Learning rate decay
Reduce on plateau
Learning rate decay factor Learning rate decay patience
0.1
epochs
Number of epochs
100
Batch size
256
Dropout
[0.0,0.5]
Optimizer
Adam
Finetuning, chemical
Weight decay
0.0
Learning rate
0.001
Learning rate decay
Step decay
Number of steps before decay
30
Learning rate decay
0.3
Number of epochs
100
Batch size
128
Dropout
(0.0,0.5)
Optimizer
Adam
Finetuning, long-range
Weight decay
0.0
Learning rate
0.001
Learning rate decay
Reduce on plateau 0.5
Learning rate decay factor Learning rate decay patience
epochs
where Cp is the predictive objective, Lo is the contrastive objective, and a is the weight hyperparam- eter. We select a from {0.1, 0.2, 0.3, 0.4, 0.5), using finetuning results as the performance indicator. 7 the performances using models trained with different values of a.
Table 7: Downstream performances with models pretrained using different values of the weight hyperparameter a. Subscripts C. P, and /' indicate contrastive pretraining, predictive pretraining, and inclusion of fragment encoders downstream prediction, respectively. Best results for either CP or CPF models are shown in red. The rightmost column shows the average performance ranking for either CP CPF models (lower value means better ranking).
Models
BBBP
Tox21
ToxCast
SIDER
ClinTox
MUV
HIV
BACE
Avg. Rank
CP. 0.1
0.
0.3
6..9
6.4
63.2
76.1 7.
82.5
2,88
CP. 0.2
70,8 1.9
0.3
64.
62.1
64
76.5 1.2
76.0 1.1
83.2 1.4
2.63
CP. 0.3
1.73
0.5
64
61.3+06
73.7=39
76.6
1.0
81.
2.13
CP. 0.1
2.31
0.41
6,30
6..2106
60.1
7.
75.7
82.1
3.81
CP. 0.5
70.6 1.1
76.1 0.3
64.6
59.2
65.7
74.1
7.66
81.9 1.3
3.56
CPF:
72.0
7.70
63.909
6..6+1.2
84.7
7..4
78.0
CPF 0.2
69.4 1.3
7.87
6..5
63.3
76.9332
7..0
77.6
1..
4.06
CPF. a 0.3
70.0
0.
65.020.5
62.64099
83.0
7.72
77.6
7..2
3.25
CPF, 0.4
0.
0.
64.6205
63.5
830069
3.03
2.1
7.88
3.06
CPF. a 0.5
7.9
7..2
64.8
62.7
79,3
73.4228
76.7
80.1 1.2
3.00
B
Datasets
Table and Table show statistics regarding the benchmark datasets used to evaluate our models. In
Table we list the number of learning instances and the number binary tasks each chemical dataset. Out of datascts, are small-size and 2 are medium-size. Table presents Long-range
15

---

datasets and predictive tasks on peptides. The peptide datasets have similar learning instances but with different predictive tasks.
Table 8: Binary Chemical Benchmarks
Datasets
Descriptions
Number of Graphs
Number of Tasks
BBBP
Blood-brain barrier permeability
2039
1
Tox21
Toxicology on biological targets
783
12
ToxCast
Toxicology measurements via thrhuhhruughpu screening
8575
617
SIDER
Adverse drug reactions of marketed medicines
1427
27
Clin'Tox
Drugs that failed clinical trials for toxicity reasons
1478
2
MUV
Validation of virual screening techniques
93087
17
HIV
Ability to inhibit HIV replication
41127
1
BACE
Binding results for inhibitors of human B secretase 1
1513
1
Table Long-range Peptide Benchmarks
Datasets
Peptides-fune
Peptides-struet
Number of Graphs
15535
15535
Number of Tasks
1
5
Number of Classes
10
N/A Multi-label regression
Task types Descriptions
Multi-label classification
Peptide functions: antibacterial, antiviral, etc 3D properties: length, sphericity, etc
To maintain comparability with prior works, we different featurizations for molecules from the binary chemical benchmarks and the long-range benchmarks. For data from Table we follow the fcaturization used by [15), in which there 2 input atom features and 2 input bond features For
data from Table 9] we use the standard fcaturization provided in Open Graph Benchmark [14], in which there are 9 input atom features and 3 input bond features The latter featurization scheme is more up-to-date and adopted by recent works on machine learning in the chemical space [ 50
C Vocabulary
C.1 Vocabulary Statistics
In Figure 4 we present statistics on fragments in the extracted vocabulary, molecular graphs, and fragment graphs. Subplot shows the distribution of fragments based on their sizes. The smallest fragments consist of singleton atoms, while the largest one contains 21 atoms. The majority of fragments comprise 1 to 14 atoms. Although the extracted vocabulary is precise, its fragments are both large and highly prevalent, which is not the case for fragmentations used in previous works. In fact, the most prevalent fragments are size-independent. As shown in subplot common fragments are well distributed among all fragment sizes. This high prevalence throughout the vocabulary fosters pattern recognition. thereby contributing to the competitive results achieved by our models. The presence of large fragments also positively impacts the size of the fragment graphs. Subplot a and subplot depict the size distributions of the fragment graphs and the molecular graphs, respectively. Relative to the molecular graphs, fragment graphs are more sparse and considerably smaller in size. This property, combined with the optimized fragments, enables them to capture the higher-order connectivity patterns of graph components effectively.
C.2 Vocabulary Size and Downstream Performance
The vocabulary size may influence the performance of pretrained models on downstream tasks. In To] we report this effect when varying the vocabulary size from 200 to 3, 200, doubling the size with each step. The results suggest that in general, for each task, there is an optimal vocabulary size. For instance, GraphFP& performs the best on Clin Tox and HIV when pretrained with vocabulary of size 800. The optimal vocabulary size for Tox21 is 400 while the optimal size for BBBP falls
HH et al, ICLR 2020, Appendix C http:///hw...ooooooaaaaoottoobbe
16

---

340000
a
b
100000
Number G/02'6
100000
Number Crease
20000
80000
00000
10000
00000
5000
AMAJI
1.5
20
40
70
Size
Number Pregments
d
50000
200000
1088 fraiinot 9p2
Nurbord of Frogmonts
Nummer ol toagimett-
40300
250000
20300
20003
20009
464
30
20
320
205
320
4000
500
600
700
800
Fragment Size
Fragment ID
Figure 4: (a) Size distribution of fragment graphs (b) Size distribution of molecular graphs (c) Size distribution of fragments from the extracted vocabulary (d) Blue plot shows the occurrences of fragments in the vocabulary and purple plot shows the sizes of the fragments.
between 400 to 800. SIDER favors smaller vocabulary sizes while MUV favors larger vocabulary sizes. Varying vocabulary sizes seem to have little impact on the performance of ToxCast. BACE perceives the most interesting trend when the performance drops at vocabulary size 400 and rises again when the size increases.
Table 10: Downstream performances with GIN pretrained on vocabulary of various sizes.
Vocab Size
BBBP
Tox21
ToxCast
SIDER
Clin'Txx
MUV
HIV
BACE
200
69.8
0.8
6.85
61.3 0.7
74,8 4.3
74.9 = 1.8
76.9 1.0
79.4 + 1.6
400
71.6 1.4
0.5
63.8
6.73
6.6
2.5
75.9
77.5
2,5
800
1.6
0.4
63.8
61.4009
78.6227
1.5
7..
78.2 2,4
1600
71.1
7.84
63.9
60.4
7.64
75.3 2.1
79.0 4.3
3200
7.83
7.4
6.6
6..6
7.9
1.9
7.3
4.5
C.3 Fragment Graph Statistics
We report several statistics regarding the fragment graphs from the prepared pretraining dataset described in Section 4.1 Table [T lists the number of graphs for each fragment graph size. The smallest fragment graphs are of size (standalone fragment) while the largest fragment graphs are of size 30. The majority of fragment graphs are smaller sizes. This indicates that the fragmentation algorithm was able to extract large and frequent fragments, resulting small fragment graphs that can capture higher-order connectivity.
To illustrate the connectivity within fragment graphs, for each fragment graph size, we report the average number of edges in Table [2 To save space, we only consider fragment graphs with maximum size of 10. Since our molecular graphs are connected, our fragment graphs are also connected (i.e., no disconnected island). From the above table, we can see that, given a fragment graph with size the average number of edges connecting the fragments varies from around n
17

---

Table 11: Number of Fragment Graphs by Size.
Size
4
5
7
8
10 11
Num Graphs
167
8433
76024
144940
112426
52607
20999
7742
1570
788
to n. When the fragment graphs are small, the average number of edges is closer to 1, indicating that the fragment graphs are mostly tree-like. As the fragment graph size increases, more loops appear and thus the average number of edges deviates further from n 1.
Table 12: Average number of edges by graph size.
Size
3
5
8
10
Average Num Edges
0.00
1.00
2.02
3.07
4.17 5.33
6.51
7.74
8.88
10.11
Finally, in Table 13 with varying vocabulary sizes, report the average fragment graph size and the average number of edges for the whole pretraining dataset. As the vocabulary size increases, the size of fragment graphs decreases in general since larger vocabularies contain larger fragments.
Table 13: Average graph size and average number of edges with varying vocabulary size.
Vocabulary Size
200
400
800
1600
3200
Average Graph Size
5.92
5.17
4.61
4.15
3.73
Average Num Edges
5.23
4.40
3.78
3.28
2.84
18


Reviews:
[{"review_id": "4y1cDr3WLd", "reviewer": "Reviewer_RaR1", "summary": "This paper proposes a contrastively and predictively strategy for pretraining GNNs based on graph fragmentation. Specifically, it leverages a frequency-based method for extracting molecule fragments, and performs fragment-based contrastive and predictive tasks to jointly pretrain two GNNs on a molecule graph and a fragment graph. It also enforces the consistency between fragment embeddings and atom embeddings for multi-solution structural information.", "questions": "1.\tThe results of MGSSL [1] is different from what they report in the original paper. Why?\n2.\tDo you kekulize the molecules? Or how do your deal with the possible breakage of aromatic systems?\n3.\tWhy do you use a subset of ChEMBL for pretraining? How is the dataset processed?\n\n[1] Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph self-supervised learning for molecular property prediction. NeurIPS 2021.\n", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is easy to follow.\n2.\tThe paper investigates an interesting fragmentation strategy for pretraining tasks.\n3.\tThe proposed method enforces the consistency between fragment embeddings and atom embeddings for multi-solution structural information, which is a promising trick. Experiments demonstrate the effectiveness of this method.", "weaknesses": "1.\tThe technical novelty is limited, because it is a combination of existing methods. While the performance improvement is not very remarkable.\n2.\tThe authors may want to conduct ablation studies on the effect of molecule fragmentation strategy and the pretraining strategy, respectively.\n3.\tTable 3 shows that the performances worsen on some downstream benchmarks as the vocabulary size grows larger. The authors may want to investigate a smaller vocabulary size. When it reaches 1, the method is the same as without fragments.\n4.\tA related work [1]---which leverages a similar frequency-based motif extractor and uses contrastive learning for generative training---is missing.\n\n[1] Zijie Geng Z, Shufang Xie, Yingce Xia, et al. De Novo Molecular Generation via Connection-aware Motif Mining. ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful suggestions! We are happy to address your concerns.\n\n(1) **The technical novelty is limited, because it is a combination of existing methods**\n\nWe would like to argue that our work is not simply a combination of existing methods. Besides Micro-Graph [1], ours is the only work that does fragment-level contrastive pretraining. Our proposed fragment-based pretraining and finetuning strategies are novel and efficient. We used an existing fragmentation procedure, however, the fragmentation component used is not fixed and can be replaced with other frequency-based fragmentation methods. Please refer to point **(1)** from the main rebuttal for a detailed discussion regarding our contributions.\n\n(2) **The performance improvement is not very remarkable.**\n\nFor comparison, **Table B** from the main rebuttal the average AUC across 8 common chemical benchmarks of our models against strong baselines. The combination of our proposed pretraining strategies obtained 2.62%, 1.72%, and 1.29% relative improvement over JOAOv2, GraphMVP-G, and GraphMVP-C, respectively. When considering only the contrastive component, our model GIN-C (72.81) performs better than other contrastive models, including GraphCL (70.78), JOAOv2 (72.12), and GraphMVP (71.69).\n\n(3) **The authors may want to conduct ablation studies on the effect of molecule fragmentation strategy and the pretraining strategy.**\n\nTo clarify, we have conducted certain ablation on the components proposed in the paper, which are:\n- C: contrastive pretraining.\n- P: predictive pretraining.\n- F: including fragment GNN in downstream prediction.\n\nBecause F requires C, all possible combinations are {C, P, CP, CF, CPF}. In the paper, we included 3/5 combinations (C, CP, CPF) and showed that more components successively improve the performance. We provide the full results with 5 combinations (**Table A** in the main rebuttal). Given the limit of the rebuttal period and the focus of our paper on the pretraining and finetuning instead of the fragmentation, we leave the ablation on the fragmentation strategies for future work.\n\n(4) **The authors may want to investigate a smaller vocabulary size. When it reaches 1, the method is the same as without fragments.**\n\nTo clarify, the vocabulary size cannot go to 1 since it should at least contain unique singular atom fragments. We added 2 additional experiments with smaller vocabulary sizes (200,400) and show the full results on 8 benchmarks (**Table C** in the main rebuttal). Thank you for your suggestion.\n\n(5) **A related work [1] ... is missing.**\n\nThank you for the pointer. We will include this citation in the revision.\n\n(6) **The results of MGSSL [1] is different from ... the original paper. Why?**\n\nWe rerun MGSSL and GraphLoG using the codes provided by the authors to ensure the results are comparable to other baselines (10 runs). The results from MGSSL paper used 3 runs and the results from GraphLoG are from the last epoch, not the best validation epoch.\n\n(7) **Do you kekulize the molecules? Or how do your deal with the possible breakage of aromatic systems? Why do you use a subset of ChEMBL for pretraining? How is the dataset processed?**\n\nThe benchmark datasets and pretraining dataset are curated data from existing works. These datasets have been cleaned in a standard manner, including kekulization. The size of our pretraining dataset (456k) is comparable to those used to pretrain the baseline methods and is optimal for our limited computing resource. As far as we know, the motif mining algorithm does not produce fragments with incomplete aromatic system. Even so, our pipeline is robust to such cases because of how the fragment graphs are constructed. We have a single edge connecting two neighboring fragments. This edge is non-featurized and no matter the type of connection between two fragments (single bond, multiple bonds, overlapping atoms, etc), there is only one edge connecting them. Essentially, the fragment graphs only contain high-level arrangement within molecules and not granular bonding information. As a result, the pipeline is robust to cases in which fragments may contain incomplete aromatic system, such as fragmentation based on functional groups.\n\n\n[1] Zhang, S., Hu, Z., Subramonian, A., & Sun, Y. (2020). Motif-driven contrastive learning of graph representations. arXiv preprint arXiv:2012.12533."}, {"review_id": "lKjgi1B853", "reviewer": "Reviewer_Vdiv", "summary": "The authors propose a novel method for generating representations for molecule graphs where two GNNs are contrastively learned. Using this new represntations, the authors achieve good results compared to a variety of baseline methods.", "questions": "i realise that many molecular benchmarks are based on similar organic compounds, but I am curious how the method would behave over a wider-range of molecules in material science. \n\nWould it be possible to consider fragments are even larger common structures? Would this aid in longer range predictions?\n\n> We reduce the learning rate by a factor of 0.1 every 5 epochs without improvement. \nI'm curious if a different learning rate schedule would give better results. How is no improvement defined?\n\nLong range structural information may involve 3D information, where. things that appear far apart on the graph may not be. I'm curious if this could be measured in some way? Could this method be applied to domains where 3D information is more directly used? ", "limitations": "N/A", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper and method are presented clearly.\n\nThe results are strong and the method is interesting + well explained ", "weaknesses": "I found the presentation of Figure 3 a bit confusing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time spent reviewing our paper! We appreciate the thoughtful comments and are happy to address your questions regarding the wider applicability of the model. \n\n(1) **I realise that many molecular benchmarks are based on similar organic compounds, but I am curious how the method would behave over a wider-range of molecules in material science.**\n\nThis is an interesting question! Indeed learning and pretraining on material science compounds is an underexplored area compared to that of organic molecules. There are some differences between the two domains.\n\nThere are a wide variety of materials and its hard to discuss all of them given the limitation of this forum. In general, we can say that while organic compounds are small and are stand-alone structures, materials have consituting units. Their properties are defined by both the arrangement of these units and the units themselves (for example: polymers with monomers, lattice with unit cells, and metal-organic frameworks). For application of GNNs to this domain, the graph representation needs to reflect these special characteristics. For example, [1] construct graphs for crystal lattices based on the connectivity within a unit cell. There have been investigations regarding the performance of GNNs on various types of material property prediction [2]. Most recently, several works have explored pretraining on materials, such as metal-organic frameworks [3] [4]. \n\nOur method can be applied at different scale, to capture structural information either within the constituting units or of the unit arrangement. However, more research is needed to find an effective way of combining and utilizing these multiscale arrangement.  It would be interesting to see the performance of the method when applied to domains in which patterns are different than those in organic compounds but surely, some domain specific adjustments are needed. We are confident that it would be easier to adapt to materials in which organic patterns are presented such as polymers or metal-organic frameworks, but more investigation will need to be done for other types of compounds. In our work, we pretrained on 2 levels of topological resolution, but it is quite straightforward to extend to more levels.  We can have a separate loss to handle each pair of layer $n$ and layer $n+1$. However, as pretraining information can be distilled beyond 2 levels, we need a mechanism to control and tune this flow.\n\n(2) **Would it be possible to consider fragments are even larger common structures? Would this aid in longer range predictions?**\n\nIt would be quite interesting to do so. However, a major problem is that as the size of substructures grows, they become rarer and more unique because in general, smaller substructures are more likely to repeat (think singular atoms, bonds, or simple rings). Even when we could extract high-frequency large fragments, this is not always useful, as we discussed in Section 4.4 and showed in Table 3. Larger fragments may overly \"blur\" the higher order connectivity, leading to worsen performance.\n\nThere are cases when we indeed want much larger fragments, such as when working with macromolecules. In those cases, the substructure rarity problem still exists. One possible idea is to look for common substructures within the fragment graphs and graphs with even higher order of connectivity and enforce this information up and down the hierarchy. This expansion may lead to better performance on long-range prediction.\n\n(3) **I'm curious if a different learning rate schedule would give better results. How is no improvement defined?**\n\nNo improvement means no new minima in the loss is found after the given number of epochs (5 in this case). \n\nWe believe the performance will likely be better with learning rate tuning. However, due to limited resource, we generally went with hyperparameters that seems to work well. The hyperparameters we used are similar and comparable to those of the baselines. The factor value of 0.1 is typical in practice, and the patient value of 5 epochs came from our observation of a few initial training cycles.\n\n(4) **Long range structural information may involve 3D information, where. things that appear far apart on the graph may not be. I'm curious if this could be measured in some way? Could this method be applied to domains where 3D information is more directly used?**\n\n3D coordinates are valuable information that would likely help property prediction. However, such information is expensive to obtain on a large scale. When it is available, pretraining can help distill such knowledge into the molecule's embeddings, like in GraphMVP [5] or 3D-InfoMax [6]. In that case, our learned embeddings, which encode long-range information, likely adapt well to incorporating 3D information.\n\n[1] Xie, T., & Grossman, J. C. (2018). Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review letters, 120(14), 145301.\n\n[2] Fung, V., Zhang, J., Juarez, E., & Sumpter, B. G. (2021). Benchmarking graph neural networks for materials chemistry. npj Computational Materials, 7(1), 84.\n\n[3] Cao, Z., Magar, R., Wang, Y., & Barati Farimani, A. (2023). Moformer: self-supervised transformer model for metal–organic framework property prediction. Journal of the American Chemical Society, 145(5), 2958-2967.\n\n[4] Kang, Y., Park, H., Smit, B., & Kim, J. (2023). A multi-modal pre-training transformer for universal transfer learning in metal–organic frameworks. Nature Machine Intelligence, 5(3), 309-318.\n\n[5] Liu, S., Wang, H., Liu, W., Lasenby, J., Guo, H., & Tang, J. (2021). Pre-training molecular graph representation with 3d geometry. arXiv preprint arXiv:2110.07728.\n\n[6] Stärk, H., Beaini, D., Corso, G., Tossou, P., Dallago, C., Günnemann, S., & Liò, P. (2022, June). 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning (pp. 20479-20502). PMLR."}, {"review_id": "KAmzEEoiM0", "reviewer": "Reviewer_MPQy", "summary": "Based on the belief that learning with fragments can help capture structural information at multiple resolutions, this paper proposes a fragment-based strategy for pretraining and fine-tuning. \n\nFirst, the paper extracts fragments by an existing heuristic algorithm called Principle Subgraph Mining algorithm to obtain fragments from a large molecular dataset (i.e.,  ChEMBL database). Then, the paper uses two separate GNNs (one for molecules and one for fragments) to learn the node embeddings. In particular, the node embeddings obtained by molecular-based GNN are pooled into fragment embeddings. \nThe model is trained with respect to three tasks: a contrastive task between fragment embeddings obtained by molecular-based GNN and  fragment-based GNN, a fragment existence prediction task, and a fragment graph structure prediction task. \n\nThe experiments are done on 8 binary graph classification tasks from MoleculeNet and 2 graph prediction tasks on large peptide molecules from the Long-range Graph Benchmark.", "questions": "1. The fragment extraction strategy can be described and compared with more details. The authors choose Principle Subgraph Mining algorithm to extract fragments. Do you try others? Any empirical evidence supports this choice? Like BRICKS or RECAP?\nOn line 53, the authors comment MGSSL as \"their multi-step fragmentation may overly decomposes molecules, losing the ability to represent high-order structural patterns\". Can you expand it? Why? \nOn line 122, the authors wrote that \"however, existing methods use suboptimal fragmentation or fragment embeddings\".  Likewise, can you provide more discussion about it? Why their methods obtain suboptimal fragments while yours can? Any theoretical proof? \n\n\n2. The method assigns different fragment embeddings for two identical fragments appearing int the same molecule. Have you trying forcing them to be the same? I get that the local neighborhood of these two identical fragments can be different. But will it also help capture generic information if fragment is uniquely represented?\n\n\n3. On lie 159, the authors wrote that \"An edge exists between two fragment F nodes of GF is there exist at least a bond interconnecting atoms from the fragments.\" Will these cause too many edges between fragments? I suspect that most fragments are connected in this case. Therefore, the topology of fragment graph is lost. \n\n4. How to obtain ground-truth structural backbones?\n\n5. More experimental details of fragment extraction algorithm are needed. Any hyperparameters? On line 308, the authors wrote that \"we prepare two additional vocabularies with size 1600 and size 3200\". How do you prepare that?\n\n6. Why results on long-range tasks can be much improved than existing works? Why different tasks use different baselines?\n\n7. Figure 4(d) is hard to follow. How do you assign fragment ID? Why you need fragment ID as x axis?", "limitations": "No potential negative societal impact of their work as far as I know.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ The proposed method is easy to follow and conduct. \n+ The results on Long-range Graph Benchmark are particularly good. \n+ Figure 1 clearly shows the method.\n+ Codes are provided. Appendix further provides more details.", "weaknesses": "The idea of using molecular fragment can be interesting. The proposed method shows some effectiveness, although how it obtains can be less illuminating. \n\nMany design choices need to be further described. Please reply to my questions below.\n\nIn addition, some writing issues exist. Sentences cannot start with \"[reference]\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful questions! \n\n(1) **The fragment extraction strategy can be described and compared with more details. ..., the authors comment MGSSL as \"their multi-step fragmentation may overly decomposes molecules, losing the ability to represent high-order structural patterns\"?**\n\nRule-based methods (BRICS, RECAP) produce:\n- Very large vocabularies.\n- Most fragments have very low occurrence (like Kipf distribution).\n- Fragments with high occurrence are mostly small.\n\nA very large vocabulary with mostly rare and unique fragments is a major challenge for learning embeddings. One solution is further fragmenting the large and rare fragments to form a more concise vocabulary with mostly small fragments (MGSSL). However, if a fragment graph is constructed with small fragments, then this fragment graph is not much sparser than the original molecular graph. This means that the connectivity of the fragment graph is not much more global or higher-level than that of the molecular graph, which is the reason for our comment on MGSSL.\n\nWe set the requirements for a fragment vocabulary to have precise size and contain larger fragments with good occurrences. Principal Subgraph Mining provides these qualities, so it is suitable for our study.\n\n(2) **On line 122, the authors wrote that \"however, existing methods use suboptimal fragmentation or fragment embeddings\". Likewise, can you provide more discussion about it?**\n\nWe discuss these points from line 48 - 54. We'd like to further clarify them here:\n- The fragments in GROVER are k-hop subgraphs surrounding atom nodes, which limits the kind of patterns these fragments can represent because chemical patterns come in various sizes and shapes.\n- In MICRO-Graph, the fragment embeddings are contrasted with the graph embeddings. This encourages smoothing among the embeddings of fragments from the same graph since they all form positive pairs with the graph.\n- MGSSL, as stated above, overly decompose the fragments, hence their fragment graph cannot effectively represent high-order connectivity.\n\nWe will add these explanations after line 122 in the revision.\n\n(3) **The method assigns different fragment embeddings for two identical fragments appearing in the same molecule. Have you tried forcing them to be the same?**\n\nCapturing structural information and multiple resolution and enforcing the consistency between node embeddings and fragment embeddings are our main objectives. The fragment embeddings distill onto the node embeddings 2 pieces of information: the local neighborhood, which comes from the fragment type, and the positional information with respect to the global arrangement, which comes from the embedding of higher-order connectivity via applying a GNN on the fragment graph. Both are arguably important for downstream prediction. Forcing the embeddings of 2 fragments of the same type to be similar will strengthen the first piece of information and weaken the second. Instead, we leaved the decision on weighting these 2 pieces of information to the contrastive learning and to the need of the downstream task. Generic information regarding each fragment is always captured because we keep an optimizable dictionary of fragment embeddings, which is tuned via learning.\n\n(4) **... I suspect that most fragments are connected in this case. Therefore, the topology of fragment graph is lost.**\n\nTo clarify, there will be only 1 edge between a pair of connected fragments in the fragment graph no matter how many bonds there are connecting the fragments. To answer the reviewer's question: no, there will not be too many edges between fragments. Generally, fragment graphs are sparse and tree-like.\n\n(5) **How to obtain ground-truth structural backbones?**\n\nWe record a dictionary of all possible structural backbones from the pretraining dataset with NetworkX's graph hashing algorithm. To obtain ground-truth structural backbone on an input graph, we remove all node features, hash the empty fragment graph, and look up the hash value in the dictionary.\n\n(6) **...the authors wrote that \"we prepare two additional vocabularies with size 1600 and size 3200\". How do you prepare that?**\n\nAfter each iteration of the fragmentation algorithm, a new larger fragment is added to the vocabulary by merging smaller ones, prioritizing frequency. The process repeats until the size of vocabulary reach a predefined value. To prepare larger vocabulary, we run the algorithm with more iterations.\n\n(7) **Why results on long-range tasks can be much improved than existing works? Why different tasks use different baselines?**\n\nOur method can effectively embed higher-order structural information. GNNs are inefficient in capturing long-range connectivity beyond a node's local neighborhood. In Table 2, GatedGCN with RWSE performs better than other GNNs because graph topological information (RWSE) is explicitly added to the node embeddings. Our pretraining effectively enforces this information into both node embeddings and fragment embeddings. To illustrate this point, Figure 3 shows that the fragment embeddings alone can well distinguish different structural backbones and node embeddings agree with the spatial arrangement within a molecule.\n\nFor long-range tasks, we compared with GNNs to show that our pretrained GNN overcome the short-range problem of GNNs.\n\n(8) **Figure 4(d): How do you assign fragment ID? Why you need fragment ID as x axis?**\n\nThe fragments are arranged based on decreasing size. The ID on the x-axis simply shows unique fragments in this order. The purple line shows the sizes of the fragments on the x-axis. Each horizontal plateau shows a collection of fragments with the same size. The blue plot is a histogram showing the frequency of each fragment with the spikes showing the most frequent fragments. These spikes are distributed across the x-axis, showing that most frequent fragments are of all sizes and not just small fragments."}, {"review_id": "5LtXZWkXzv", "reviewer": "Reviewer_Br6k", "summary": "This paper presents a novel approach to pretrain Graph Neural Networks (GNNs) at the fragment level for property prediction on molecular graphs. By utilizing a compact vocabulary of prevalent fragments and introducing fragment-based contrastive and predictive pretraining tasks, the authors overcome the limitations of node-level and graph-level pretraining. Two different GNNs are pretrained to capture structural information at multiple resolutions, and the fragment information is utilized during finetuning. The proposed models show improved performances on both common molecular and long-range biological benchmarks.", "questions": "See weakness.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to follow.\n- The idea that using motif for pretraining is novel and reasonable.\n\n", "weaknesses": "- Empirical performance is not strong enough. Especially in Table 2, it's hard to distinguish the absolute gain over the baselines. The authors are encouraged to report the average score over all tasks in molecular property prediction.\n- The authors are also encouraged to compare with stronger baselines. For example, the authors can also compare JOAO V2 in addition to JOAO.\n- Missing ablations: the authors add many components and loss functions in the system. It would be interesting know how each contribute to the performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time reviewing our paper and your thoughtful insights! We are happy to address your concerns.\n\n(1) **Empirical performance is not strong enough. The authors are encouraged to report the average score over all tasks in molecular property prediction.**\n\nWe understand that the average score is often reported in the literature to give an evaluation of the overall performance across benchmarks. However, we argue that this metrics can be problematic since the values and variations of the scare can be quite different among benchmarks and a few benchmarks can skew the average value. Instead, we opt for rankings per benchmark and report the average ranking to evaluate overall performance. \n\nWe do agree with the reviewer that absolute values give a better impression of the performance. As a result, we will add an extra column reporting this metric. Our 3 models, GIN_C, GIN_CP, and GIN_CPF, are the top 3 regarding both AUC and Ranking.\n\n| Baselines | AttrMasking | ContextPred | G-Motif | G-Contextual | GPT-GNN | GraphLoG | GraphCL |  JOAO | GraphMVP | MGSSL |   GIN_C   |   GIN_CP  |  GIN_CPF  |\n|:---------:|:-----------:|:-----------:|:-------:|:------------:|:-------:|:--------:|:-------:|:-----:|:--------:|:-----:|:---------:|:---------:|:---------:|\n| Avg. Rank |     5.81    |     5.44    |  10.06  |     8.69     |  10.38  |   8.94   |   8.69  |  7.31 |   6.00   |  8.13 |  **4.31** |  **3.69** |  **3.56** |\n|  Avg. AUC |    71.15    |    70.89    |  70.16  |     69.34    |  67.16  |   69.70  |  70.78  | 71.89 |   71.70  | 70.41 | **72.81** | **72.59** | **74.01** |\n\nIn terms of the strength of the empirical evaluation, we argue that our models achieved substantial improvement over existing methods. Please see our response to the next concern. \n\n(2) **The authors are also encouraged to compare with stronger baselines. For example, the authors can also compare JOAO V2 in addition to JOAO.**\n\nThe following table compares our models with JOAO and JOAOv2:\n\n|         | BBBP           | Tox21          | ToxCast        | SIDER          | ClinTox        | MUV            | HIV            | BACE           | Ave AUC |\n|---------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|---------|\n| JOAO    | 70.2 ± 1.0     | 75.0 ± 0.3     | 62.9 ± 0.5     | 60.0 ± 0.8     | 81.3 ± 2.5     | 71.7 ± 1.4     | 76.7 ± 1.2     | 77.3 ± 0.5     | 71.89   |\n| JOAOv2  | 71.4 ± 0.9     | 74.3 ± 0.6     | 63.2 ± 0.5     | 60.5 ± 0.7     | 81.0 ± 1.6     | 73.7 ± 1.0     | 77.5 ± 1.2     | 75.5 ± 1.3     | 72.12   |\n| GIN_C   | 71.5 ± 1.6     | **75.5 ± 0.4** | 63.8 ± 0.6     | 61.4 ± 0.9     | 78.6 ± 2.7     | **77.2 ± 1.5** | 76.3 ± 1.0     | 78.2 ± 3.4     | 72.81   |\n| GIN_CPF | **72.0 ± 1.7** | 74.0 ± 0.7     | **63.9 ± 0.9** | **63.6 ± 1.2** | **84.7 ± 5.8** | 75.4 ± 1.9     | **78.0 ± 1.5** | **80.5 ± 1.8** | **74.01**   |\n\nIn addition, Table B from the main rebuttal shows the average AUC across 8 chemical benchmarks of our models against strong baselines. The combination of our proposed pretraining strategies obtained 2.62%, 1.72%, and 1.29% relative improvement over JOAOv2, GraphMVP-G, and GraphMVP-C, respectively. When considering only the contrastive component, our model GIN-C (72.81) performs better than other contrastive models, including GraphCL (70.78), JOAOv2 (72.12), and GraphMVP (71.69).\n\n(3) **Missing ablations: the authors add many components and loss functions in the system. It would be interesting know how each contribute to the performance.**\n\nTo clarify, we have conducted certain ablation on the components proposed in the paper. These components are:\n- C: contrastive pretraining.\n- P: predictive pretraining.\n- F: including fragment GNN in downstream prediction.\n\nBecause F requires C, all possible combinations are {C, P, CP, CF, CPF}. In the paper, we included 3 out of 5 combinations (C, CP, CPF) and showed in Table 1 of the paper that more components successively improve the performance. We added results with all 5 combinations in Table A of the main rebuttal."}, {"review_id": "ShyC1oaCFp", "reviewer": "Reviewer_wcQm", "summary": "In this paper, the authors proposed contrastively and predictively strategies for pretraining GNNs based on graph fragmentation. Using principle subgraph extraction, the authors pretrain two separate encoders for molecular and fragment graphs, capturing structural information at different resolutions.", "questions": "Please see the weakness.", "limitations": "The limitation is not clearly discussed. Please provide the discussion in the rebuttal period. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized and easy to follow.\n2. The authors conduct comprehensive comparisons with baselines to show their advantages.\n3. The fragment-level contrastive pretraining framework is novel, which captures both granular patterns and higher-order connectivity.\n4. The t-SNE visualization in Figure 3 clearly shows the effectiveness of the authors' design.", "weaknesses": "1. The technical contribution is limited. For example, the principle subgraph extraction module is borrowed from [19].\n2. The authors do not clearly state how to choose hyperparameter alpha.\n3. In Table 3, the effects of the vocabulary size are only explored on three datasets.\n4. The pertaining time is not reported in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and insightful review. We are happy to answer your concerns.\n\n(1) **The technical contribution is limited. For example, the principle subgraph extraction module is borrowed from [19].**\n\nThe focus of the paper is on developing pretraining and finetuning strategies based on the fragmen graphs. Our strategies are elegant and efficient, achieving improvement over recent strong baselines. The fragmentation component is not fixed and can be replaced with other similar frequency-based fragmentation methods. Please refer to point **(1)** from the main rebuttal for a detailed discussion regarding our contribution.\n\n(2) **The authors do not clearly state how to choose hyperparameter alpha.**\n\nWe choose alpha from [0.1,0.5] with step size 0.1. The detailed performance on each value of alpha is reported in Table 6 from the Appendix.\n\n(3) **In Table 3, the effects of the vocabulary size are only explored on three datasets.**\n\nIn **Table C** of the main rebuttal, we show the effect of varying vocabulary sizes on all 8 chemical benchmarks. We also added smaller vocabulary sizes (200, 400) as suggested by reviewer **RaR1**. We found interesting trends among datasets. Thank you for your suggestion.\n\n(4) **The pretaining time is not reported in the paper.**\n\nIn general, it takes 4-5 hours to do one round of pretraining  (100 epochs) for any of our models on one V100 GPU. We did not find a significant difference in training time between the pretraining variations in our work. We will report the training time in the revision.\n\n(5) **The limitation is not clearly discussed.**\n\nThis is a valid concern. In general, we do not observe any potential negative societal impact from our work. We do recognize a few limitations:\n- Our paper is empirical. The theoretical work for our paper is limited. We concluded the effectiveness of our designs based on downstream performances, ablation of training components, and visualization of the learned embeddings.\n- Due to resource constraints, we did not fully optimize the hyperparameters during pretraining and finetuning. We generally went with hyperparameters that seem to work well, referring to those used in previous work. Still, our models compare competitively with results from other works.\n- We pretrain on a subset of ChEMBL containing 456k molecules. Given that public database containing millions of molecules are available, the size of our pretraining data could be a bottleneck for performance. However, this pretraining size ($10^5$) is reasonable to compare with other baselines. GraphMVP used 50k molecules with 3D information for pretraining while others use pretraining set ranging from $10^5$ to $10^6$ in size. \n- We mainly evaluated on 8 popular binary prediction datasets. Molecular property prediction is a considerably rich field with a variety of datasets suitable for benchmarking. Even so, the benchmarks we used are the most popular among existing works.\n- We did not experiment with a variety of fragmentation methods. A recent paper [1], as pointed out by Reviewer **RaR1**, is another frequency-based fragmentation strategy comparable to the one used in our paper. It would be interesting to see how our methods perform with a variety of fragmentation methods, however, we'd like to leave this for future work.\n\n[1] Zijie Geng Z, Shufang Xie, Yingce Xia, et al. De Novo Molecular Generation via Connection-aware Motif Mining. ICLR 2023.\n"}]

Author rebuttal:
We want to thank the reviewers for their time and insightful comments!
We would like to use this space to summarize and address some common concerns. Citations follow those in the paper.

**(1)** Reviewer **RaR1** and Reviewer **wcQm** raise concerns about the level of contribution of our work.  Reviewer **wcQm** reasoned that the lack of novelty comes from the borrowed fragmentation technique from another work [19]. We used [19] because this algorithm is suitable for our requirements. The fragmentation component is not fixed and can be replaced with other frequency-based fragmentation methods. Our contribution is in the pretraining and finetuning strategies with fragment graphs. For that reason, we respectfully disagree with reviewer **RaR1**'s statement that the paper is a combination of existing methods. We studied fragment-level pretraining, an interesting direction where few prior works exist [26], [46], [47]. Ours is the only beside [46] that does fragment-level contrastive pretraining. Compared to previous work, we proposed to:
- Conduct cross-level contrastive learning between node embeddings and fragment embeddings.
- Learn separate sets of embeddings for node and fragments and train separate models for molecular graphs and fragment graphs.
- Contrast fragment embeddings with aggregated embeddings of nodes corresponding to the fragments. This design allows much more flexibility in learning multi-resolution topology compared to those of previous works.
- Utilize the fragment graphs in both predictive pretraining and downstream prediction.

Our methods are not only elegant, but also efficient. Compared to [47] which took 20 hours to train, each of our models took less than 5 hours to train in total (using similar hardware) while obtaining superior performances.

**(2)** Reviewer **RaR1** and Reviewer **Br6k** indicated a lack of ablation study regarding the various components we proposed. In particular, we have 3 unique components:
- C: contrastive pretraining using separate molecule GNN and fragment GNN.
- P: predictive pretraining utilizing information from fragment graphs.
- F: including fragment GNN in downstream prediction. 

To clarify, we have conducted considerable ablation on these components. Because F requires C, all possible combinations of these components are {C, P, CP, CF, CPF}. In the paper, we included 3/5 combinations (Table 1: C, CP, CPF) and showed that more components successively improve the downstream performance, confirming the positive contribution of each component. We report results on all combination in Table A. We only include the average AUC and ranking (among the combinations) here and will include the full details in the final revision.

**Table A**

|Model|GIN_C|GIN_P|GIN_CP|GIN_CF|GIN_CPF|
|:-:|:-:|:-:|:-:|:-:|:-:|
|Avg AUC|72.81|69.05|72.59|73.73|74.01|
|Avg Rank|2.88|3.69|2.75|3.19|2.50|

C and F are stronger contributors than P, however, when combined with others, P has positive impacts on the overall performance. Disparity between AUC and Rank is because strong performance on a few benchmarks can skew the average AUC. Between GIN_CP and GIN_CF, GIN_CF has better AUC because it did particularly well on ClinTox and ToxCast but GIN_CP has better overall ranking because it has more pretraining than GIN_CF.

**(3)** Reviewer **RaR1** and Reviewer **Br6k** raised questions regarding the strength of our results and improvements. For comparison, the following Table B shows the average AUC across 8 common chemical benchmarks. GraphMVP-G and GraphMVP-C are recent strong baselines, and JOAO-v2 is a strong contrastive baseline suggested by Reviewer **Br6k**.

**Table B**
|Baselines|GraphCL|MGSSL|JOAOv2|GraphMVP|GraphMVP-G|GraphMVP-C|GIN_C|GIN_CPF|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|Avg AUC|70.78|70.41|72.12|71.69|72.76|73.07|72.81|74.01|

The combination of our proposed pretraining strategies in GIN_CPF obtained 2.62%, 1.72%, and 1.29% relative improvement over JOAOv2, GraphMVP-G, and GraphMVP-C, respectively. Notice that GraphMVP relies on 3D information, which is expensive to obtain and difficult to scale. Compared to generative methods (MGSSL and GraphMVP-G), our strategies are much faster to train. MGSSL takes 20 hours while GIN-CPF takes less than 5 hours to train. Considering only the contrastive component, our model GIN-C (72.81) performs better than others, including GraphCL (70.78), JOAOv2 (72.12), and GraphMVP (71.69).

**(4)** Reviewer **wcQm** and reviewer **RaR1** recommend expanding the analysis on the effects of the vocabulary size. This is a valuable suggestion and we are happy to address it. In particular, we added 2 smaller vocabulary with sizes 200 and 400 to the analysis and report the downstream performances of GIN_C on all 8 datasets in Table C.

**Table C**

| Vocab Size |BBBP|Tox21|ToxCast|SIDER|ClinTox|MUV|HIV|BACE|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|200|69.8±1.5|75.6±0.8|63.5±0.8|61.3±0.7|74.8±4.3|74.9±1.8|76.9±1.0|79.4±1.6|
|400|71.6±1.4|75.8±0.5|63.8±0.4|61.3±0.7|75.2±6.6|75.7±2.5|75.9±1.0|77.5±2.5|
|800|71.5±1.6|75.5±0.4|63.8±0.6|61.4±0.9|78.6±2.7|77.2±1.5|76.3±1.0|78.2±3.4|
|1600|71.1±1.6|75.4±0.8|63.9±0.9|60.4±0.5|76.4±4.6|76.1±2.1|75.3±1.6|79.0±4.3|
|3200|71.3±0.8|75.4±0.4|63.7±0.6|60.6±0.5|75.5±5.9|77.1±1.9|75.1±1.3|79.4±4.5|

The size of the vocabulary directly influences the size of fragments. Small fragments may result in fragment graphs that are too granular to capture high-level connectivity effectively. On the other hand, fragments being too large may lead to an overly loose view of the graph and a loss of structural information. The results suggest that in general, for each task, there is an optimal sweet spot for vocabulary size for which it should be finetuned. For example, the optimal vocabulary size for Tox21 is 400 while the optimal size for BBBP falls between 400 to 800. SIDER favors smaller vocabulary sizes while MUV favors larger vocabulary sizes.

Meta-review: The authors proposed a contrastive learning algorithm for molecular representation learning. To split the molecular graph into fragments, the authors use a method similar to BPE in NLP. The constrastive learning is conducted between fragment graphs and molecule graphs. 

Following thorough discussions among reviewers, the consensus leans towards acceptance. Consequently, I have decided to adhere to the collective decision and recommend acceptance.

However, I DO have the following concerns and I hope the authors should discuss / address in the camera ready version:

- The rationale of the constrastive learning between a molecular graph and fragment graph is not clear. What is the advantage over the constrastive learning within molecular graphs? Any experimental results?


- L246: 456K molecules and 5-layer GIN for pre-training: This is not convincing to me. Pre-training should be conducted on large amount of data and large models, while the results in this paper are not convincing.

- The results on MoleculeNet seem to be far behind SOTA.

  -  https://www.nature.com/articles/s42256-021-00438-4

   - https://arxiv.org/pdf/2207.08806.pdf or https://dl.acm.org/doi/10.1145/3534678.3539368

    -  I understand the above two works use both 2D and 3D information. However, given so many molecular pre-training papers, will the proposed method in this paper benefit and influence more readers?

   - Even if for the work not using 3D information, https://dl.acm.org/doi/pdf/10.1145/3580305.3599317, this works gives a much stronger baseline than that in the current submission.

  - I think the authors should discuss with the above works.
Decision: Accept (poster)