Contrastive Learning of Molecular Representation with Fragmented Views
Authors: Seojin Kim
Jaehyun Nam
Junsu Kim
Hankook Lee
Sungsoo Ahn
Jinwoo Shin
Venue: Submitted to ICLR 2023 (2023)
Keywords: Molecule representation learning

Abstract: Molecular representation learning is a fundamental task for AI-based drug design and discovery. Contrastive learning is an attractive framework for this task, as also evidenced in various domains of representation learning, e.g., image, language, and speech. However, molecule-specific ways of constructing good positive or negative views in contrastive training under consideration of their chemical semantics have been relatively under-explored. In this paper, we consider a molecule as a bag of meaningful fragments, e.g., functional groups, by disconnecting a non-ring single bond as the semantic-preserving transformation. Then, we suggest to construct a complete (or incomplete) bag of fragments as the positive (or negative) views of a molecule: each fragment loses chemical substructures from the original molecule, while the union of the fragments does not. Namely, this provides easy positive and hard negative views simultaneously for contrastive representation learning so that it can selectively learn useful features and ignore nuisance features. Furthermore, we additionally suggest to optimize the torsional angle reconstruction loss around the fragmented bond to incorporate with 3D geometric structure in the pre-training dataset. Our experiments demonstrate that our scheme outperforms prior state-of-the-art molecular representation learning methods across various downstream molecule property prediction tasks.

Under review as a conference paper at ICLR 2023
CONTRASTIVE LEARNING OF MOLECULAR REPRESENTATION WITH FRAGMENTED VIEWS
Anonymous authors Paper under double-blind review
ABSTRACT
Molecular representation learning is fundamental task for Al-based drug design and discovery. Contrastive learning is an attractive framework for this task, as also evidenced in various domains of representation learning, e.g., image, language, and speech. However, molecule-specific ways of constructing good positive or negative views in contrastive training under consideration of their chemical se- mantics have been relatively ender-explored. In this paper. we consider molecule as bag of meaningful fragments, e.g., chemically informative substructures, by disconnecting non-ring single bond as the semantic- preeeeiiiigss transformation. Then, we suggest to construct a complete (or incomplete) bag fragments as the positive (or negative) views of molecule: each fragment loses chemical sub- structures from the original molecule, while the union of the fragments does not. Namely, this provides easy positive and hard negative views simultaneously for contrastive representation learning so that it can selectively learn useful features and ignore nuisance features. Furthermore, the torsional angle reconstruction loss around the fragmented bond to incorporate with 3D geometric structure in the pretraining dataset. Our experiments demon- strate that our scheme outperforms prior state-of-the-art molecular representation learning methods across various downstream molecule property prediction tasks.
additionally suggest to optimize
INTRODUCTION
Obtaining discriminative representations of molecules is a long-standing research problem in chem- istry (Morgan, 1965). Such a task is critical for many applications such as drug discovery (Capecchi et al., 2020) and material design (Gómez-Boo Bombbrelll et al., 2018), since it is a fundamental build- ing block for various downstream tasks, c.g., molecule property prediction (Duvenaud et al., 2015) and molecule generation (Mahmood al., 2021). Over the past decades, researchers have focused on handcrafting the molecular fingerprint representation which encodes the presence or absence of chemically meaningful substructures, e.g., functional groups, molecule (Rogers & Hahn, 2010).
Recently, graph neural networks (GNNs) (Kipf & Welling, 2016) have gained much attention as a framework learn the molecular representation due to its remarkable performance for learning predict chemical properties (Wu et al., 2018). However, they suffer from overfitting without much labeled training data (Rong et al., 2020b). To resolve this issue, researchers have investigated self- supervised learning to generate supervisory signals from a large amount of unlabeled molecules.
A notable approach this line of work is contrastive learning. which learns discriminative rep- resentation by maximizing the agreement of representations of "similar" positive views while mini- mizing the agreement of "dissimilar" negative views (Chen al., 2020a); it has widely demonstrated its effectiveness for representation learning not only for molecules (Wang et al., 2021; 2022), but also for other domains, e.g., image (Chen et al., 2020a; He al., 2019), video (Pan et al., 2021), language (Wu et al., 2020), and speech (Chung et al., 2021). Here, the common challenge for learning good representation how to construct effective positive and negative views a self-supervised manner.
For molecule contrastive representation learning, most prior works have utilized graph-augmentation techniques, e.g., edge/node drop, to produce positive views (You et al., 2020; 2021). However, such augmentations often fail to generate proper positive views of molecule graph, losing impor- tant chemical semantics from the anchor molecule, e.g., randomly inserting an edge of a graph may generate non-realistic molecule (Fang et al., 2021b). Thus, sementic-preservingg transformation

---

Under review as a conference paper at ICLR 2023
Complete bag fragments
Frugmentation
Positive
Cut non-ring single bond
Torsional angle reconstruction
Negative
Negative
Incomplete bag Incomplete bag fragments
fragments
(a) Tragment-basse based contrastive learning
(b) Torsion reconstruction from fragments
Figure 1: Illustration of FragCL: contrastive learning of molecular representation with fragmented
views. (a) Fragment-based view construction: We construct bag of fragments from molecule via fragmentation. complete (or incomplete) bag of fragments is regarded as positive (or negative) view of the original molecule. (b) Additionally, 3D contextual information can be learned by recon- structing the torsional angle around the fragmented bond.
should be carefully designed on molecule space to produce a "confident (or easy) positive" view of molecule. On the other hand, despite the effectiveness of hard-to-diseriminate negative samples has been widely evidenced in the contrastive representation learning literature of other domains (Robin- et al., 2020; Lee & Shin, 2022), such "hard negative" view construction is also not well-explored on molecule space yet.
Contribution. In this paper, we propose Fragment-based molecule Contrastive Learning (FragCL.), novel contrastive learning method using fragments to simultaneously generate easy positive and hard negative views of molecule graph. FragCL consists the following key ingredients with overall illustration provided in Figure
Fragment-based positive view construction: We construct new positive view of molecules by decomposing it into bag of meaningful fragments. We propose disconnecting non-ring single bond of molecule in half as the semantic-preservinn transformation, since such trans- formation preserves most of the chemically informative substructure, e.g., the number of het- eroatoms and existence chemically informative substructures. Then, we suggest to regard complete bag of resulting fragments an easy positive view of a molecule.
Fragment-based negative view construction: For negative views of molecule, we consider (a) incomplete bag fragments of its own and (b) complete bag of fragments of other molecules (in mini-batch). Here, (a) is of strikingly different choice from prior molecule contrastive learning methods (You al., 2020; Wang al., 2021); existing works use the subgraphs of incomplete bag of fragments, which special
molecule only positive views, while we kind of subgraph, as a negative view. Our intuition that (a) becomes a hard-to-diiscimminte fragment molecule in half and incomplete bag of fragments, i.e., roughly
negative view half of important substructures of the anchor molecule are lost.
Torsional angle reconstruction from fragments: We additionally propose pretext task to incorporate 3D geometric context with fragments. We note that torsional angle, i.e., the angle between planes through two sets of three atoms having two atoms common, defines several 3D contextural properties. e.g., the energy surface around the fragmented bond (Smith, 2008). Thus, the 2D graph encoder is able to leam meaningful 3D contextual information by reconstructing from the fragments during training.
2

---

Under review as a conference paper at ICLR 2023
To demonstrate the effectiveness of FragCL, we extensively evaluate method under various downstream molecule property prediction tasks on widely-used MoleculeNet (Wu et al., 2018) and ATOM3D (Townshend et al., 2020) benchmarks using 2D GNN pretrained by FragCL. Following Stärk et al. (2022) and Liu et al. (2022), we consider the most practical scenario: we pretrain with 21) and 3D paired unlabeled molecule dataset, and fine-tune with 2D downstream labeled molecule dataset." We observe that FragCL outperforms prior state-of-the-an molecular representation learn- ing method. For example, fragment-based molecule contrastive learning achieves the state-of-the-an performance in out of 8 molecule property classification tasks on MoleculeNet, and all 7 molecule property regression tasks on ATOM3D.
PRELIMINARIES
2.1 GRAPH NEURAL NETWORKS FOR MOLECULES
In this work, we graph neural networks (GNNs) obtain discriminative representations of molecules. At a high-level, GNNs learn the molecular representation by applying (a) iterative neigh- borhood aggregation to acquire node representations from the molecule graph and (h) readout func- tion to create a graph representation from the node representations.
be represented as 2D topological graph g (V, 8) associated with a set of nodes (atoms) V and a set of edges (bonds) &. Given a 2D graph G, the 8-th iteration of the neighborhood aggregation scheme is formulated as follows:
GNN for 2D molecule graphs. A molecule
( < Aggregate (1 (( (b(-") .h ()-1)) euv) E ((o))),,
(1)
h Combine(h h ((-1))).
(2)
Here, (v)) a set of adjacent nodes of v, and h the f-th representation of node u where h the atom attribute associated with node v. Next, Eue is the bond attribute corresponding to the edge {u, v} E E. The Aggregate and Combine functions are GNN-specific components.
GNN for 3D molecule graphs. A molecule can be represented as 3D geometric graph G (V, R) associated with a set of nodes (atoms) V and a set 3D positions of atoms R. Given a 3D graph G. the (-th iteration of the neighborhood aggregation scheme is formulated as follows:
h (8) < Aggregate (hl ("-",, h(( (f-")), (r, : uEE V)),
(3)
( Combine ((()) h ((-1)). g
(4)
Here, h is the l-th representation of node where h is the atom attribute associated with node v. Next, Ta and ru is the coordinate of u and v, respectively. The Aggregate and Combine functions GNN-specific components.
Molecular representation extraction. After L times of the neighborhood aggregation, the 2D or 3D graph representation be extracted from the final node representation h. (L. follows:
f(g) Readout ((h(") (1(4))))) )evv)
where Readout is permutation invariant function such as node-wise mean-pooling
2.2 CONTRASTIVE LEARNING
In general, contrastive learning aims to learn meaningful data representation by minimizing its distance to positive data representations and maximizing distance to negative data representations. The key idea here is to incorporate the human prior in representation learning by carefully setting the positive and the negative relation between representations.
To this end, number of contrastive learning objectives have been proposed, e.g., max-margin trastive loss (Hadsell et al., 2006), triplet loss (Weinberger & Saul, 2009), or metric maximization
Sinne obtaining exact 3D geometric structure of molecule is costly due iterations of quantum calculations, is highly likely that 3D information is available in downstream tasks.
3

---

Under review as a conference paper at ICLR 2023
of local aggregation (Zhuang et al., 2019). The normalized lemperature-scaled cross entropy loss (NT-Xent) (Chen et al., 2020a) is one of the widely used contrastive learning losses (Chen et al., 2020a;b; You et al., 2020). In this work, we follow GraphCL (You et al., 2020), which uses a variant of NT-XXent defined as follows:
Xen- (z, xt, 1T)) = -- -log exp(sim(z,27))
(5)
Σz- exp(sim(z,z )/T)²
where sim(2. 2) =z =zT2/|z[|22 is the cosine similarity, is temperature-scaling hyperparame- ter, (z,2t) 2 and (z, x z7) are positive and negative pairs of latent representations, respectielly
3 FRAGCL: FRAGMENTATION- MOLECULE CONTRASTIVE LEARNING
In this work, we propose FragCL: Fragment-based molecule Constrastive Learning, a novel self- supervised learning framework for pretraining molecular representations. As mentioned in Section 1, we train a 2D GNN (20(.) as a molecular representation extractor by jointly training fso (.) on a pretraining dataset DD containing both 21) and 3D molecule graphs.
Our key idea is to combine contrastive learning with molecule fragmentation: decomposing a molecule into a bag of meaningful fragments, e.g., chemically informative substructures. We con- struct a complete and incomplete bag of fragments as easy positive and hard negative views of a molecule, respectively. Additionally, we propose torsional angle reconstruction pretext task around the fragmented bond to further incorporate 3D geometric context in the pretraining dataset. The overall description of FragCI. illustrated in Figure 1. For each training step, FragCI, performs the following operations:
1. Molecule fragmentation: by disconnecting non-ring single bond whose absence likely to preserve chemically infor- mative substructures, i.e., G and {g', g") shares chemically informative substructures.
Given a molecule G E D, we obtain the 'bag of fragments' {
Contrastive learning with bag of fragments: We optimize the 2D GNN fzo(+) and 3D GNN f-)(-) along with the projection heads (-p(-) and (s)(-) via contrastive learning with our pro- posed positive/negative views, while maintaining the consistency of the representations from (-n(-) and (-) To this end, we regard the complete (or incomplete) bag of fragments positive (or negative) view of each molecule graph (see Figure la). Additionally, we use the complete bag of fragments of other molecules in mini- mani-bbtch&& as negative views.
Torsional angle reconstruction from fragments: Given a bag of fragments "") obtained by removing non-ring single bond (u, v) from G with u E g' and E G", reconstruct the torsional angle defined by a quartet of atoms (s, u, U. t), linearly connected via covalent bonds in order in G, incorporate 3D contextual information of original molecule (see Figure 1b).
In the rest of this section, we provide details and rationale of our three components: molecule frag- mentation in Section 3.1, contrastive learning with bag of fragments Section 3.2, and torsional angle reconstruction in Section 3.3.
3.1 MOLECULE FRAGMENTATION
Our FragCL crucially relies on the molecule fragmentation to generate a bag of fragments from a molecule. While there exist various ways to break molecule into disconnected components, we consider removal of a bond that (a) has the bond order of one and (b) is not a member of any ring- Note that double or higher-order bonds are capable of many reaction pathways such as nucleophilic attack and hydrogen addition (Smith, 2008), i.c., they are directly involved in chemically informative substructures. Thus, removing those bonds may alter the semantics of molecules seriously. On the other hand, single bonds does not determine the core semantic of the functional groups in general. For example, functional groups such as ketone, ester, and amide are categorized by the "carbonyl group", whose main functionalities are determined by C=O double bond. Also, the chemical func- tionalities of the ether and thioether come from the electronegative oxygen and sulfur atoms (not from the single bond), respectively (Smith, 2008). Thus, choose only single bonds to fragment molecule, and it is verified to be effective empirically in Section 4.3. Also, further discussion about the fragmentation strategy can be found in Appendix F.
4

---

Under review as a conference paper at ICLR 2023
The fragmentation of 2D molecule G20 (V.E) by single non-ring bond & produces complete bag of fragments (22n 9 where G2 and 9% are components in the graph induced on the edge set E \ {e). The edge E is selected minimize the difference of the number of atoms in 2o and G%. i.e., we split the original molecule in half. This is to prevent one of the fragments from containing almost all semantics of the original molecule because consider incomplete bag of fragments as negative pair of the original molecule in Section 3.2.
The fragments on 3D molecule graph Gap are defined by S'p (V", R') and 9% (V", R"), where V and y" are the sets of nodes of 2o and 9 and R' and R² are the corresponding coordinates of V and V", When the context is clear, we drop the subscripts 2D and 3D.
3.2 CONTRASTIVE OBJECTIVE WITH BAG FRAGMENTS
We introduce a strategy to obtain positive and negative pairs of molecular representations and con- trastive objectives based on the positive and negative pairs. what follows, we describe the strategy to construct the pairs given a training batch ( where n denotes the batch size.
Positive pairs. The key insight is that our fragmentation preserves chemically meaningful sub- set of all chemically informative substructures in G. Then, each fragment G' and g", from our fragmentation scheme described in Section 3.1, satisfies F(G.) F(I))UFFF) with a high probability. Based on this insight, we consider (9,, (9), 9 ) a positive pair, i.e., positive pair consists of the original molecule and its complete bag of fragments. To facilitate the representation attraction of a molecule 9; and bag of fragments (9 9.') we use simple mixing strategy between r/ == f(g') and r. == fgg") as r n'tn" (n' x x + n" xr.) [9'). and n" | we attract r, f(G.) and r,1x.
structures; for clarity, denote F(G)
with n'
Negative pairs. For given molecule Gú we construct negative pairs with (a) arbitrary complete bag of fragments (9), 9"] with i + j. and (b) its own incomplete bag of fragments {G]) {g") }. Intuitively, the negative pairs constructed using (a) is dissimilar to each other since arbitrary com- plete bag of fragments (G'.g") has similar semantics with Gj. which is distinct from the original molecule G. Furthermore, we consider (b) "hard" negative since deleting large fragment sig- nificantly modifies the molecular semantics. Also, this pair is hard-to-discriminate since Gj yields a common substructure its original molecule Gi.
Using positive and negative pairs, we train both 2D and 3D GNNs fz0(.) and fao(.) along with projection heads (-)(+) and (-o(-) using following contrastive loss:
L2n,
Lur (Z20,;; 20,,. (220.) Jive U (2/20 diT),
nix nix
(6)
1=1
L3D.Com
Cur-Xent(Zsp nix [znix Jof U (Z'D. 1:T).
Y2
where 2,, 2/,, z!,, and 2, denote latent representations projected by 9(-) from r,, r/ r. and r, respectively (Chen et al., 2020a; You et al., 2020). Namely, we plug in our proposed positive and negative samples into the NT-Xent loss of equation 5.
To facilitate the representation learning on the multi-modality of 2D and 3D graphs of molecules, consider each 2D and 3D view of molecule as positive pair, following the common practice to deal with the multi-modal representation learning (Radford et al., 2021; Stärk et al., 2022; Liu al., 2022):
1
((20,30).oo =
(Lur-xeat (Z20,;; Zap.s: (2.0. (7=07)))=1i7) + Lur-xent (Rap, Z2D,11 (220.) ):==):))))))))))
2n i=1
(8)
Finally, we obtain our contrastive objective combining above:
Leen LzD,com + L30 + L120,sD).com
(9)
After pretraining with the loss using (unlabeled) molecule dataset, we transfer pretrained 2D GNN (-)()) (-) for fine-tuning downstream task; note that fao (-), gzo (+). and 9sD(.) are not transferred.
A component of graph connected subgraph that is not part of any larger connected subgraph.
5

---

Under review as a conference paper at ICLR 2023
3.3 TORSIONAL ANGLE RECONSTRUCTION FROM FRAGMENTS
We propose an additional pretext task to incorporate the 3D contextual information into our 2D GNN (-). To this end. (20(-) learns reconstruct the torsional angle around the fragmented bond from fragments of a 2D molecule graph 22 and 9
Our reconstruction target, torsional angle, is defined by a quartet of atoms. Let (s, u, v, t) be tuple of vertices (atoms) in G, which is linearly connected via covalent bonds in order. The torsional angle defined by the angle between the planes defined by (s, u, v) and (u, t), and it encodes important 3D contextual properties, e.g., energy surface around the atoms (Smith, 2008). Thus, by the reconstruction of the torsional angle from fragments {G', 9"} obtained by the removal of (u, v) from G, the 2D GNN (-)(-) would learn the 3D contextual properties around the fragmented bond. The loss function is defined as follows:
Zyot, (8, U, U, b) == Grot (bap ai o, 6 h2n,,,s))
(10)
abb (8, M, U, f) == Jabs e,, a, b,0 b2)..,))
(11)
Ltorsion =
Lace (Zrot, u, U. 1), Vrot,e) + Caaaaaaa,i u, U, Yaba,))
(12)
n
+ L
(t, CZ v, u, s), Mzot,a) + Cee Zabe, 0, u4 8), Babe..).
(13)
where h2v, ( is the representation of {s, u, t 'th node from fzo (-). respectively. And Frot. binary label for the rotation direction of torsional angle, i.e., clockwise or counter-clockwsse, and Vaba, is a binned label for the absolute torsional angle value between 0 and of i-th ground state molecule, Also, frot and Jabs are projection functions, and Lace and Lce are binary cross-entropy and cross-entropy loss function, respectively. Since the torsional angle defined by (s, u, v) and (t, u, u, s) are the same, we learn to reconstruct the torsional angle from both tuples.
3.4 OVERALL TRAINING OBJECTIVE
From the discussion of Section 3.2 and 3.3, we finally propose our training loss function. sum- mary, set (a) the complete bag of fragments as positive view of molecule, (b) incomplete bag of its own and the complete bag of fragments of other molecules as negative views molecule while maximizing the consistency of the outputs of 2D and 3D GNNs. Additionally, we incorporate 3D contextual information by reconstructing the torsional angle around the fragmented bond. The specific loss function follows:
Fraagel Leon + aLtorsion
(14)
where is hyperparameter to control the contribution of the torsion reconstruction objective, and we simply use a = 1 for all experiments. The ablation study on a can be found in Appendix
4 EXPERIMENTS
In this section, we evaluate the effectiveness of our FragCL. To this end, we extensively compare FragCL. with the existing molecular graph representation learning methods. We evaluate FragCL and baselines on various downstream molecule property prediction tasks after pretraining on (unlabeled) molecule dataset. Also, we perform the extensive ablation study on FragCL to confirm that each of our components plays an important role for effectively discriminating the molecules.
4.1 EXPERIMENTAL SETUP
In what follows, we describe our experimental setup, where we largely follow prior works (Hu et al., 2019; Liu et al., 2022). Detailed information can be found in Appendix A.
Datasets. For self-supervised pretraining, 3D paired molecule graphs from GEOM (Axelrod & Comez-Bombarelll 2022). After pretraining, transfer learning is performed (a) binary classification tasks from MoleculeNet benchmark (Wu
50k unlabeled molecule dataset with 2D and

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison of test ROC-AUC score on MoleculeNet downstream property classification benchmarks. We report mean and standard deviation over 3 different seeds. We mark the best mean score bold. Additionally, we mark the average scores within one standard deviation from the highest average score to be bold. We remark by (*) when we use the score reported by Liu et al. (2022). Otherwise, we reproduced scores under the same setup.
Methods
BBBP
Tox21
ToxCast
Sider
Clintox
MUV
HIV
Bace
65 4224
74.9=us
61.6a12
58.0+24
58.8s5s
71.0a25
75.3
72.6s49
Pretrained on 2D molecule graph
EdgePred*
64.5
74.5
60.8
56.7
55.8
73.3
75.1
64.6 244
*
70.2
74.2:ns
62.5
60.41oa
68.6:98
73.9
74.3
77.2
GPT-GNN"
64.5
75.3
62.2
57.5
57.8
76.1
75.1
77.62as
Infomax
69.2aoa
73.0=n7
62.0.os
59.21ea
75.1
74.0
74.5-18
73.9
ContextPred
71.2.109
73.3
62.8
59.3114
73.7
72.5
75.8-11
78.6214
GraphLoo
67.8
73.0.ns
62.2.104
57.4123
62.02
73.1
73.4_m
78.8
G-Contextual
70.3
75.2.ns
62.6
58,4108
59.9a82
72.3104
75.9_m9
79.2
G-Motif
66.4
73.2.n8
62.6105
60.611
77.8
73.3130
73.8-14
73.4140
GraphCL*
67.5
75.0=a3
62.8=02
60.1+1a
78.9=42
77.1+1A
75.0=a4
68.7273
JOAO*
66.0
74.4=07
62.7=a6
60.7
66.3=3>
77.0
76.6=05
72.9=20
JOAOv2
67.2#34
75.0=a7
63,5
60,6104
77.1
73.4s34
77.7=11
71.7
MGSSL
67.3
74.5=02
63.6s04
58,4102
75.4=a8
73.9+14
77.2=25
76.2+1a
MolCLR
67.6.06
74.4=13
62.9.02
58.7.1.1
$7.9=30
70.8=2a
75.4-12
74.6
2D FragCL (Ours)
68.2.30
74.7=m3
62.4
60.3
82.7213
77.9
76.1
77.2
Pretrained
21) and 3D molecule graph
GruphMVP#
68.5
74.5=m4
62.7aa.1
62.3
79.0=2s
75.0
74.8=14
76.8
GruphMVP-G*
70.8
75.9=ns
63.1
60.2
79.1
77.7
79.3 76.0=a1
GraphMVP-C#
72.4
74.4=n2
63.1
63.9
77.5
75.0=1o
77.0=12
81.2sao
3D-InfoMax
67.9
75.3
64.6204
59.6107
89.7
76.7
73.4-12
79.9
FragCL (Ours)
70.9
76.2=m2
64.2.05
61.9
89.9
77.8
77.8
80.9
et al., 2018), and (b) regression tasks from ATOM3D benchmark (Townshend et al., 2020). Detailed explanations for each downstream task can be found in Appendix D.
Baselines. We follow the baselines considered in GraphMVP (Liu et al., 2022), EdgePred (Hamil- ton et al., 2017), AttrMask (Hu et al., 2019), GPT-GNN (Hu et al., 2020), Infomax (Sun et al. 2019), ContextPred (Hu et al., 2019), GraphLoG (Xu et al., 2021), G- G-|CCntextual,, Motif) (Rong et al., 2020a), GraphCL (You et al., 2020), and JOAO (You et al., 2021). Additionally, we com- pare with newly-proposed ID-Imfomax (Stärk al., 2022). GraphMVP. GraphMVP-(G.C), and 3D-Infomax utilize both 2D and 3D molecule graphs in pretraining, while other baselines are based on 2D molecule graph. Detailed information of baselines can be found in Appendix A. For more comparison. we add 2D-FragCL, which is pretrained on only 2D molecule graphs with C2o con in cquation More comparison on 2D-FragCL and 2D molecule graph representation learning meth- ods on ZINC15 (Sterling & Irwin, 2015) pretraining dataset can be found in Appendix B.
Experimental details. We consider 5-layer graph isomorphism network (GIN) (Xu et al., 2019) and 6-layer SchNet (Schütt et al., 2017) with mean pooling as readout function of our 2D GNN (-) (-) and 3D GNN (.D(.). The configuration is drawn from GraphMVP (Liu et al., 2022) for a fair comparison. Note that only transfer f2o(-) to fine-tune on 2D molecule graphs in the downstream tasks. Further details can be found in Appendix A.
4.2
MAIN RESULTS
In our experiments, we achieve state-of-the-ee performance in downstream molecule property pre- diction tasks. In both MoleculeNet and ATOM3D downstream tasks, FragCL consistently achieves the state-of-the-art ROC-AUC and MAE score, respectively.
MoleculeNet classification task. As reported in Table FragCL achieves the best average test ROC-AUC score when transferred to MoleculeNet (Wu et al., 2018) downstream tasks. Specifically, method achieves the state-of-thh-ard performance out of 8 downstream tasks. We emphasize
7

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison of test MAE score on ATOM3D downstream quantum property regression benchmarks. We report mean and standard deviation over 3 different seeds. We mark the best mean score bold. Additionally, we mark the average scores within one standard deviation from the highest average score to be bold.
Methods
ZPVE
C,
LUMO
HOMO
Fym
R²'
Uo
Uni
Hass
Gyss
49.7
0-128en.022
0.255:tuxe
$4.8
65.6
124
28.8:51
74,9293
68.3t1122
72.02106
71,2
CP
30.7:21
0.416aa0022
0633:3302
0.219:uue
K3.6s25
867729
124
25.4.033
60.8511
65.2s111
60.8:114
GrapaCt
27.2
0.419ea.0s
03589:3103
K5.0a23
85.3-93
121
25.1
57,2642
58.9-19
53.1
59,9444
CVppiMVP-::
22.7
0.425
0.19
85.2,36
86.5.91
134_1
15,2
37,433
386
39.2:1
41,6133
30 Infomax
22-2-16
0,692
0.202:1004
84.9
832=39
121
24.8
40,3e13
39.7-12
38.7
38,0+13
FragCI. (Ouns)
23:4::
0.409
0,488
0.201:10Mm
80.9
8300as
118
24.4
39,1s3
40,0-16
40.9
57,7e22
that the improvement of FragCL is consistent over downstream tasks. For example, 3D-InfoMax (Liu et al., 2022) achieves the best performance on ToxCast, while fails to generalize on Sider, resulting in even lower ROC-AUC score compared to several baselines pretrained on 2D dataset. On the other hand, FragCL shows the best average performance with no such failure case, i.e., FragCL learns well-generalizable representations over several downstream tasks. Also, 2D-FragCL even performs better than GraphMMP (pretrained on both 2D and 3D dataset) on 4 out of tasks. This verifies the effectiveness of our proposed view construction strategy between molecule and its bag of fragments.
ATOM3D regression task. Table shows the overall results of transfer learning on ATOM3D (Townshend et al., 2020) molecule property regression benchmark. Overall, our FragCL outperforms the baselines, achieving the ofate-of-the-art performance over all considered downstream tasks of ATOM3D. We emphasize that our FragCL. shows the best performance when fine-tuned on both MoleculeNer and ATOM3D downstream dataset, which verifies that FragCL learns representations generally applicable over distinct fine-tuning data distributions.
4.3 ABLATION STUDY
Fragment-based view construction. To recognize the effectiveness of our view construction strat- egy, we conduct an ablation study on regarding the complete (incomplete) bag of fragments positive (negative) view in Table 3. We start our ablation by considering an incomplete bag of frag- ments, i.e., a subgraph, as a positive view of the original molecule, which is similar to the prior methods (You et al., 2020; 2021; Wang et al., 2021). The improvement 70.4 + 71.7 - 72.1 is from our careful easy positive and hard negative view construction strategy via molecule fragmentation from its own molecule, which is different from existing molecular contrastive representation learn- ing methods. Also, regarding the incomplete bag of fragments as a neutral view, i.e., complete bag of fragments of other molecules as only negatives, greatly improves its non-pretraining counterpart by 67.2 + 71.9 9 and it verifies the effectiveness of our negative view construction in a batch.
Strategy to construct fragments. To verify the effectiveness of our dind-disconnecting strategy to obtain the bag of fragments, we conduct experiments with an alternative strategy: cutting random non-ring bonds, i.e., including single, double and triple bond. As shown in Table 3, our single-hond disconnecting strategy improves disconnecting a random non-ring bond by 72.1 + 72.4 for L20,con and 73.3 + 74.0 for Lcon. Note that the margin larger in Leon since Lcon utilizes both 2D and 3D fragments, while L2D, is based on only 2D fragments. The results verify our semantic-preservie transformation, i.e., disconnecting a non-ring single bond, successfully creates an easy positive view by maintaining informative substructures of molecules.
The proposed loss. Our loss design Lereee (equation 14) combines several components proposed in Section 3, and here we validate that each of the components has an individual effect in improving the performance. Table shows that 2D-FragCLI trained with Lzn.con as its loss function, effectively learss discriminative representations from fragmented 2D views by obtaining 67.2 + 72.4 from its non-pretraining counterpart. Also, jointly training 3D GNN with 2D GNN improves 2D-FragCL by 72.4 + 74.0, and incorporating 2D GNN with 3D torsional angle further boosts the performance by 74.0 + 75.0. This confirms that each component of our FragCL plays its role effectively. Also, the effectiveness of the torsional angle reconstruction task is further verified in the advanced torsion- aware architecture SphereNet (Liu et al., 2021), by improving the average performance by 74.4 - 7..1. The detailed results with SphereNet can be found in Appendix G.
8

---

Under review as a conference paper at ICLR 2023
Table 3: Ablation of FragCL with the average ROC-AUC score of 8 downstream tasks on Molecu- leNet. Detailed results can be found in Appendix E.
Complete bag of fragments
Incomplete bag
Disconnecting bond
Pretraining Joss
of fragments
Avg
67.2
Neutral Positive
Positive Positive
Random
Cap.eaa
70.4
Random
Lan.on
71.7
Positive
Negative
Random
200,0o
72.1
Positive
Negative
Single
220,coa
72.4
Positive Positive
Neutral
Single Single
L
71.9
Negative
Lz0,co c00
72.4
Positive Positive
Negutive
Random Single Single
Leon
73.3
Negutive
Leo
74.0
Positive
Negutive
Crrage
75.0
5
RELATED WORKS
Molecular representation learning. Researchers have paid attention learning molecular repre- sentations for downstream tasks given a massive unlabeled molecule dataset. self-supervised molec- ular representation learning. One of the self-supervvsed learning techniques for molecular repre- sentation learning is generative learning. For example, those methods reconstruct the corrupted in- put as pre-defined pretext tasks (Hamilton al., 2017; Hu et al., 2019; Rong al., 2020a; Zhang et al., 2021). Another large portion of graph self-supervised learning consists of contrastive learn- ing, which aims to learn a representation by pulling positive samples together and pushing negative samples apart. Zhu et al. (2021a) considers SMILES ID string and 2D molecule of molecule positive pair. You et al. (2020; 2021); Wang et al. (2021); Zhang et al. (2020) utilize augmentation schemes produce positive view of molecule graphs, Fang al. (2021a); Sun al. (2021); Wang et al. (2022) mitigate the effect of semantically similar molecules in the negative samples (Zhu et al., 2021b). Recently, there are pioneering attempts incorporate 2D topology of molecule with paired 3D geometric structures (Stärk et al., 2022; Liu et al., 2022).
Molecule fragmentation. Recently, several machine learning researches for molecule generation (Maziarz et al., 2021; Jin et al., 2018; ; 2020) regard a molecule as a combination of semantically important pieces, e.g., fragment. Such approach natural chemical sense, since the property of molecule is largely determined by its chemically important substructures; not by atom-level features (Smith, 2008). Zhang et al. (2021) has adapted this concept to molecular representation learning by considering the reconstruction of substructures as a generative pretext task. However, there are remaining challenges: since the dictionary of substructure grows as the size of pretraining dataset increases, the reconstruction is infeasible in large scale pretraining dataset.
the other hand, some research on contrastive molecular representation learning has utilized sub- structure of molecule in their training objective. For example, You al. (2020); Wang et al. (2021); Zhang et al. (2020) construct a positive view of molecule as its substructure. These works differ from ours such that they focus maximizing similarity between representations of molecule and its single fragment, i.e., positive view, while FragCL. considers a single fragment as the (hard) nega- rive view. Such opposite strategy is possible for FragCL. as it instead considers complete bag of fragments as the (easy) positive view. In addition, Wang et al. (2022) contrasts the representations of molecule-molecule and frammentfragment pairs, while FragCL considers molecule-fragment pairs.
6 CONCLUSION
We present FragCL., a new contrastive molecular representation learning method based on chemical prior: A molecule can be viewed as "bag of fragments", which consists of chemically meaningful structure. Based on this insight, we regard the complete bag fragment as an easy positive view of given molecule. Moreover, its own molecule since they differ in chemical property, while sharing some chemical substructures. Additionally, FragCL learns 3D contextual information by reconstructing the torsional angle around the fragmented bond. Experiments show that FragCL outperforms in predicting molecule property thanks to our pretraining strategy guided by clever fragment-based views of molecule graphs.
set incomplete bag of fragments to be hard negative view of

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
This work will facilitate research in molecular representation learning, which can speed up the pro- cessing of many important downstream tasks such as molecule property prediction and molecule generation. However, malicious use of well-learned molecular expressions poses a potential threat of creating hazardous substances, such as toxic chemical substances or biological weapons. On the other hand, molecule representation is also essential for creating defense mechanisms against harm- ful substances, so the careful use of our work, FragCI., can lead more positive effects.
REPRODUCIBILITY STATEMENT
We describe the implementation details for self-supervised pretraining and fine-tuning Ap- pendix A, and provide source code in the supplementary material. For 2D-Frag use single NVIDIA GeForce RTX 2080 Ti GPU and 40 CPU cores (Intel(R) (e CPU E5-2630 v4 @ 2.20GHz), and we use single NVIDIA-A100-SXM GPU and 124 CPU cores (AMD EPYC 7542 32-Core Processor) for FragCL.
REFERENCES
Simon Axelrod and Rafael for property prediction and molecular generation. Scientific Data, 911::1-14, 2022.
Geom, energy-annotated molecular conformations
Alice Capecchi, Daniel Probst, and Jean-Louis Reymond. One molecular fingerprint to rule them all: drugs, biomolecules, and the metabolome. Journal of cheminformattis, 12(1):1-15, 2020.
Lei Chen, Wei-Ming Zeng, Yu-Dong Cai, Kai-Yan Feng, and Kuo-Chen Chou. Predicting anatom- ical therapeutic chemical (atc) classification of drugs by integrating chemical-chemical interac- tions and similarities. PloS one, 7(4):435254, 2012.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0420 2020b.
Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self- supervised speech pre- training. In 2021 IEEE Automatic Speech Recognition and Understanding IEEE, 2021.
Workshop (ASRU), pp.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alan Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015. URL https: http://pcceeeeeeeeeeeeee/e neurips httpsccccappeecc/////ae (aweeeeeessllllaae1aaaaaaa Pappaee- pdf.
Yin Fang, Haihong Yang, Xiang Zhuang, Xin Shao, Xiaohui Fan, and Huajun Chen. Knowledge- aware contrastive molecular graph learning. arXiv preprint arXiv:2103. 13047, 2021a
Yin Fang. Qiang Zhang, Haihong Yang, Xiang Zhuang, Shumin Deng, Wen Zhang, Ming Qin, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Molecular contrastive learning with chemical element knowledge graph. CoRR, abs/2112 12.00544, b021 URL https::/// /ttpps///axaaee /aaxiaiiiivv///i/ org/ abs//1/s222 00544.
Rafael .ombar-Bmm..rlli, Jennifer Wei, David Duvenaud, José Miguel Hernández-Lobate Benjamín Sánchez-Lengelingg Dennis Sheberla, Jorge eguilera-Iparragene Ryan P Adams, and Alán Aspuru-Guza Guzik. Automatic chemical design using a data-driven contin- representation of molecules. ACS central science, 4(2):268-27) 276, 2018.
Timothy D Hirzel,
10

---

Under review as a conference paper at ICLR 2023
R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning invariant map- ping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition doi: 10.1109/CVPR.2066.J
(CVPR'06) 00), volume 2. pp. 1735-1742, 2006.
William L Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. arXiv preprint arXiv:1709.05530 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for un- supervised visual representation learning, 2019. URL / http:/iwwwrr/abb//////ea/ 05722.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks, 2019. URL https::/aaaat org/abs//1 905. 12265.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre- training of graph neural networks, 2020. URL
5437.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning, pp. 2323-2332. PMLR, 2018
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hicrarchical generation of molecular graphs using structural motifss In International conference on machine learning, pp. 4839-4848, PMLR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412. 6980, 2014.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: ww.....9.22200 020000 2016.
Kyungmin Lee and Jinwoo Shin. Rényicl: Contrastive representation learning with skew rényi di- vergence. arXiv preprint arXiv:2208.062770 2022.
Shengehao Liu, Hanchen Wang. Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre-training molecular graph representation with 3d geometry. In International Confer- ence on Learning Representations, 2022. URL https:/ /tt///ppeevveeeeeeeeeeeeeeeeeeee net forum?id xQUe1pOKPam.
Yi Liu, Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d graph networks. arXiv preprint arXiv:2102.05013, 2021.
Omar Mahmood, Elman Mansimov, Richard Bonneau, and Kyunghyun Cho. Masked graph model- ing for molecule generation. Nature communications, 12(1): 2021.
Krzysztof Maziarz, Henry Jackson-Flux, Pashmina Cameron, Finton Sirockin, Nadine Schneider, Nikolaus Stiefl, Marwin Segler, and Mare Brockschmidt. Learning to extend molecular seaffolds with structural motifs. arXiv preprint arXiv: 103. 1et.22103.0866 03864, 2021.
H. L. Morgan. The generation of a unique machine description for chemical structures-a technique developed at chemical abstracts service. Journal of Chemical Documentation, 5(2):107-113, 1965. doi: 10.1021/c160017000
URL https::/ /ttps:///d...o www.or/////ooo http://.11222/11100011200
Tian Pan, Yibing Song, Tianyu Yang, Wenhao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11205-1122 11244, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
11

---

Under review as a conference paper at ICLR 2023
Joshua Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592, 2020.
David Rogers and Mathew Hahn. Extended-comeee connectivity fingerprints. Journal of Chemical Informa- tion and Modeling, (554:744-755, 2010. doi: 10.1021/c1100050. URL https: /ttttpps:/doi.oroo 10. 10.1021/ http://c1100000 PMID: 20426451.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer large-scale molecular data, 2020a. URL https: / //aaxxiv.rrg/aabbbb.e 2200 02835.
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In International Conference on Learning Repre- sentations, 2020b. URL https: wwwpreveee.ee....ee............m nww.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
Janice Smith. Organic chemistry. McGraw-Hill, 2008
Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tosson, Christian Dallago, Stephan Günnemann, and Pictro Liò. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning, pp. 20479-20502. PMLR, 2022.
Teague Sterling and John J Irwin. Zinc 15-ligand discovery for everyone. Journal of chemical information and modeling, (5(11))2324-233)) 2015.
Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. arXiv preprins arXiv: 1908..9908.000
1908.10 0/000, 2019.
Mengying Sun, Jing Xing, Huijun Wang, Bin Chen, and Jiayu Zhou Mocl: data-driven molecular fingerprint via knowledge-aware contrastive learning from molecular graph. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 3585-3594, 2021.
Raphael JL Townshend, Martin Vögele, Patricia Suriana, Alexander Derry, Alexander Powers, Yianni Laloudakis, Sidhika Balachandar, Bowen Jing. Brandon Anderson, Stephan Eismann, al. Atom3d: Tasks molecules in three dimensions. arXiv preprint arXiv:2012.04033 2020.
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molclr: Molecular con- trastive learning of representations via graph neural networks. arXiv preprint 10X0v:2102.10056. 2021.
Yuyang Wang, Rishikesh Magar, Chen Liang, and Amir Barati Farimani. Improving molecular contrastive learning via faulty negative mitigation and decomposed fragment contrast. Journal of Chemical Information and Modeling, 2022.
Kilian Q Weinberger and Lawrence K. Saul. Distance metric learning for large margin nearest neighbor classification. J. Mach. Learn. Res., 10:207-244 jun 2009. ISSN 1532-4435.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learn- ing. Chemical science, 9(2):513-533 2018.
Zhuofeng Wu, Sinong Wang. Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466 2020.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful graph neural networks? In International Conference on Learning Representations, 2019, URL https: //ooenreviee net. formm/ wowormmddrryaaasroooor d=ryGa A5Km.
12

---

Under review as a conference paper at ICLR 2023
Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure, 2021. URL https: pttpp::/arrxiv.rg/ abs/2106. 04113.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33:5812-5823, 2020.
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning auto- mated. In International Conference on Machine Learning, pp. 12121-12132. PMLR, 2021.
Shichang Zhang, Ziniu Hu, Arjun Subramonian, and Yizhou Sun. Motif-driven contrastive learning of graph representations. arXiv preprint urXiv:20/2.12533 2020.
Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lec. Motif-based graph self- supervised learning for molecular property prediction. Advances in Neural Information Process- ing Systems, 34, 2021.
Jinhua Zhu, Yingce Xia, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yar Liu. Dual-view molecule pre-training. arXiv preprint arXiv:2 10234, 2021a.
Yanqiao Zhu, Yichen Xu, Hejie Cui, Carl Yang, Qiang Liu, and Shu Wu. Structure-aware hard negative mining for heterogeneous graph contrastive learning, arXiv preprint arXh::2108.1388 13886, 2021b.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings, 2019. URL https: wwt./bs//g/a/aooooo.ooooo 12355.
13

---

Under review as a conference paper at ICLR 2023
A EXPERIMENTAL DETAILS
Training details. We follow the training setup considered in GraphMVP (Liu al., 2022): Specifi- cally, we use a batch size of 256 and no weight decay. Also, we set the temperature in equation 5 as 0.1 for overall experiments. We use (Nodedrop, Amarm,,,, identity] randomly, i.e., 3 probability for each fragment and the original 2D molecule graphs, and Gaussian noise (0, I) to each coordinate of 3D molecule graphs. When pretraining, we excluded about 0. 1% of molecules that does not satisfy our fragmentation scheme (51 out of 50k molecules in GEOM 50k dataset. When Nodedrop or Attrmass sk is used, we drop/mask the portion of 0.1 vertices from the total vertices. For self-supervised pretraining, we train for 100 epochs using Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0.001 and no dropout. Our code is based on open-source codes of GraphMVP and GraphCL*.
For 2D-FragCL and other reproduced 2D baselines, we exclude implicit hydrogens in molecule graph, following the common frameworks of (You et al., 2020; 2021). For FragCL and 3D-InfoMax, we include implicit hydrogens into molecule graph, following (Liu et al., 2022) that utilizes the 3D coordinates of hydrogen atoms provided in GEOM dataset (Axelrod & Gomez-Bombarellli 2022). We exclude an incomplete bag of fragments G' from the negative pair set of the anchor molecule G when | 2 0.7 x [G], to prevent the incomplete bag of fragments G' from becoming a semantically positive view. For torsional angle prediction task, we construct the quartet of atoms (s, u, v, t) for the fragmented bond (u, u) so that s, f are non-hydrogen atoms, and the binning of Yabs splits 0 to into 9 uniform bins. We use 2-layer and 3-layer MLP for (roe(-) and Jabs (+), respectively. Also, grot() (-) and gahs (-) shares the first layer of MLP.
Evaluation on MoleculeNet downstream tasks. Following the baselines, we use scaffold split (Chen al., 2012), which splits the molecules based on their substructures. We use the split ratio vraininalidatiionntes 80:10:10 10 for each downstream task dataset to evaluate the performance. For the consistency of the input graphs in pretraining and fine-tuning, we exclude implicit hydrogen atoms of molecules in fine-tuning dataset for 2D-FragCL and other reproduced 2D baselines and include implicit hydrogen atoms of molecules in fine-tuning dataset for FragCL. and MD-InfoMax. Experimental detail follows GraphMVP (Liu et al., 2022); initialized linear projection layer for 100 epochs with Adam optimizer and a learning rate of 0.001, and dropout probability of 0.5. Our results are calculated by the test ROC-AUC score of the epoch with the best validation ROC-AUC score. Besides the ROC-AUC score of individual downstream tasks, we also report the average ROC-AUC score across downstream datasets.
tnn-tune& a pretrained 2D GNN with an
Evaluation on ATOM3D downstream tasks. Following (Townshend et al., 2020), we split the molecules ATOM3D SMP (Small Molecule Properties) dataset into 103,547 molecules for train- ing, 12,943 molecules for validation, and 12.943 molecules for test. We normalize the regression label by the mean and the standard deviation of the labels of training set. Our result is calculated by the test MAE score of the epoch with the best validation MAE score. We pretrain on ATOM3D training dataset and fine-tune a pretrained 2D GNN with an initialized linear projection layer for 100 epochs with Adam optimizer and Reduce-LR- -R--nn----------- Plateau scheduler with reduction parameter 0.7, and patience 3, and initial learning rate 0.001.
Hardwares. For Fr, we use single NVIDIA GeForce RTX 2080 Ti GPU and 40 CPU cores (Intel(R) Xeo CPU E5-2630 v4 @ 2.20GHz). and single VUDIA--A10-SXI GPU and 124 CPU cores (AMD EPYC 7542 32-Core Processor) for FragCL. Experiments for ZINCI5 dataset were performed using single GPU (Tesla V100) and 32 CPU cores (Intel Xeon Gold 5120).
Baselines. We compare our method with an extensive list of baseline methods in the literature of graph representation learning:
No pretraining trains a model from scratch for downstream task.
EdgePred (Hamilton et al., 2017) uses regestrcconttttctoonn as a pretext task.
AurMask (Hu et al., 2019) learns representation by recovering the vertex features after masking them.
Phttps://git http:/cccccccco22///cc.mm Anttps:/ thub. com htte//Shnn--aa Labb /ab///aa///raaa
14

---

Under review as a conference paper at ICLR 2023
Table 4: Comparison of ROC-AUC (%) score on downstream molecular property prediction benchmarks when pretrained on ZINC15 dataset. Following the baselines, we report mean and stan- dard deviation over 10 different seeds. We mark the best mean score bold. Additionally, we mark the second-best score bold when its mean is within one standard deviation from the highest score.
Pretraining
BBBP
Tox21
ToxCast
Sider
Clintox
MUV
HIV
Bace
Avg
65.8
74.0aos
63.4
57.3216
58.0244
71.8
75.3a19
70.1
67.0
Infomax
68.8
75.3
62.7=a4
58.4ans
69.9.ae
75.3+25
76.0sa7
75.9
70.3
EdgePred
67.3
76.0ano
64.1
60.4aa7
64. 137
74.1
76.3a1u
79.9
70.3
AttMask
64.3
76.7
64.2.aa5
61.0.a7
71.8=41
74.7=14
77.2
79.3
71.2
ContextPred
68.0aze
75.7
63. 63.92ns
60.9ane
65.9asx
75.8
77.3
79.6a12
70,9
GraphCL
69.7
73.9
62.4#as
60.5
76.0=2n
69.8s17
78.5
75,4s14
70.8
JOAO
70.2
75.0ao.
62,9aas
60.0 Ozns
81.3
71.7a14
76.7a12
77.3
71.9
JOAOv2
71.4
74.3
63,2
60.5
81.0=1s
73.7a11
77.5a12
75.5
72.1
2D-FragCL (Ours)
67.6a17
76.1 1
63.6 63.6an3
61.32m9
80.0az1
75.4
78.1
81.2
72.9
GPT-GNN (Hu et al., 2020) uses the graph generation task as a pretext task.
Infomax (Sun et al., 2019) maximizes mutual information between global representations (i.e.,
graph representations) and local representations (i.e. path representation).
ContextPred (Hu et al., 2019) learns representation by predicting surrounding subgraph of spe- cific node edge.
GraphLoG (Xu et al., 2021) discriminates graph and subgraph pairs from their opposing pairs to preserve local similarity between various graphs, which leads to the embedding aligmment of correlated graphs.
G- C-Contextual (Rong et al., 2020a) learns representations by randomly masking local subgraphs of target nodes (or edges) and predicting these contextual properties from node embeddings.
G-Motif (Rong al., 2020a) predicts the occurrence of the semantic motifs extracted by using chemical prior.
GraphCL (You et al., 2020) is generic graph contrastive learning method based on their graph- agnostic augmentation schemes, which do not use any molecule-specifie knowledge.
JOAO (You et al., 2021) and JOAOv2 (You et al., 2021) propose min-max optimization processes to learn optimal data augmentation strategies dynamically from pre-fixed candidate set of aug- mentations.
MGSSL (Zhang et al., 2021) introduces generative self-supervised objective to reconstruct motif-tree.
MolCLR (Wang et al., 2021) performs a contrastive learning with NT-Xent (Chen et al., 2020a), constructing positive views of molecule by proposed molecule augmentation schemes.
3D-InfoMax (Stärk et al., 2022) proposes to consider 2D topological molecule graph and 3D geometric molecule graph from the same molecule as a positive view of each other.
B ADDITIONAL RESULTS OF 2D-FRAGCL
To further emphasize the effectiveness of our view construction strategy, we report the performance of 2D-FragCL on the several downstream tasks on MoleculeNet (Wu et al., 2018) when pretrained on ZINC15 dataset with basclines. In Table 4, we observe that FragCL significantly improves non- pretrained counterpart. Also, CIagCL. achieves at least the second-best ROC-AUC score for out of 8 tasks, FragCL. achieves an average ROC-AUC of 72.9, while second-best baseline (i.e., JOAOv2) performs 72.1. As shown in Table 4, JOAOv2 (with respect to average ROC-AUC) often performs even worse than the vanilla without using self-supervised pretraining, e.g., in the ToxCast task. However, FragCL does not have such a failure case and achieves the robust performance across all tested downstream tasks. Overall, FragCL achieves average ROC-AUC of 72.9, while second- best baseline (i.c., JOAOv2) performs 72.1. which indeed demonstrates the generalization ability of FragCI..
15

---

Under review as a conference paper at ICLR 2023
C GRAPH NEURAL NETWORKS
Graph Isomorphism Network (GIN), We provide a detailed description of architecture of graph isomorphism network (GIN) (Xu et al., 2019), which mainly consider the feature extractor fzn(-) in this paper. Particularly, GIN learns representation h! by:
h (LP (h, ()) +
Σ nEN(2)
(h + eue e
(15)
while (f-1) the embedding corresponding the attribute of edge {u, v} E. SchNet. We consider SchNet (Schütt et al., 2017), which a very strong 3D graph neural network under fair comparison (Liu et al., 2022) our (-n(.) in this paper.
D DATASET DETAILS
We pretrain feature extractor GEOM (Axelrod & KomeezBombarelli, 2022) and ZINC15
(Sterling & Irwin, 2015). Then, we perform transfer-learning on eight benchmark binary classi- fication datasets from MoleculeNet (Wu et al., 2018). More information on downstream tasks is described below, and the statistics are reported in Table 5 and 6.
BBBP contains data on whether the compound is permeable to the blood-brain barrier.
Tox2/ measures the toxicity of compound and used in the 2014 Tox21 Data Challenge.
ToxCast includes multiple toxicity annotations of compounds collected after performing high- throughput sereening tests.
Sider refers to side effect resources, i.e., data on the marketed drugs and their side effects.
Clintox is dataset of comparison results between drugs approved through the FDA and drugs removed because of toxicity during clinical trials.
MUV is validation dataset of virtual screening technology. Specifically, it is subsampled in the PubChem BioAssay using refined nearest neighborhood analysis.
HIV consists of data about capability prevent HIV replication.
Bace is collected dataset of compounds that could prevent (BACE-I).
Table 5: Pretraining dataset statistics
Dataset
GEOM
ZINC15
Number of molecules
50,000
2,000,000
Avg. Node
25.35
26.62
Avg. Degree
54.72
57.72
Table 6: MoleculeNet downstream classification dataset statistics
Dataset
BBBP
Tox21
ToxCast
Sider
Clintox
MUV
HIV
Bace
Number of molecules
2,039
7,831
8.575
1,427
1,478
93,087
41,127
1,513
Number of tasks
12
617
27
2
17
1
Avg. Node
24.06
18.57
18.78
33.64
26.15
24.23
25.51
34.08
Avg. Degree
51.90
38.58
38.52
70.71
55.76
52.55
54.93
73.71
16

---

Under review as a conference paper at ICLR 2023
We also perform transfer-leariigg seven benchmark binary classification datasets from ATOM3D (Townshend et al., 2020). More information on downstream tasks described below, and the statis- ties reported in Table 7 and 8.
Table 7: ATOM3D downstream regression tasks
Task
Summary
Unit
ZPVE
Zero point vibrational energy
meV
Dipole moment
D
Isotropic polarizability
bohr
C
Heat capacity 298.15K
sal/mol K
LUMO
Lowest unoccupied molecular orbital energy
meV
HOMO
Highest occupied molecular orbital energy
meV
ggpp
Gap between HOMO and LUMO
meV
R
Electronic spatial extent Internal energy at OK Internal energy at OK
bohr²
Un
meV
U2i8
meV
H2gs
Enthalpy at 0K
meV
G298
Gibbs energy at 0K
meV
Table 8: ATOM3D downstream dataset statistics
Dataset
ATOM3D
Number of molecules 129,433
Avg. Node
18.0
Avg. Degree
18.6
E ABLATION DETAILS
Table 9: Detailed results on ablation of a with the average score on MoleculeNet.
Complete bag
Incomplete hag
Disconnecting
Prermining
Downstream taoks
fragments
Tragments
band
loss
BHBP
Tox21
ToxCast
Sider
Clintox
MUV
HIV
Baec
Avg
63.4ss
67.0
Neutral Posilive
Positive
Random
65.4e1a
73.8am
63.3ssc
$7.3
74.4.s4
74.2
75.0sas
79.8-a
70.4
Positive
Random
72.2440
74.6w0
63.1
58.6.as
77.9
76.5us
73.6ut
77.5
71.7
Positive
Negative
Randoen
71.0.es
73.8.sx
63.31s1
59.9.03
81.5
75.7
75.1
76.4.1
72.1
Positive
Negalive
Single
68.2.3
74,7am
62.4st
60.3
82.7a1
77.9kt
76.1a14
77.2a13
72.4
Neutral
Positive
Negative
Single
68.2gs
74.7am
62.4sr
60.3gm
82.7.:
77.9gm
76.1.r
71.2
72.4
Positive
Negative
Random
Com
66.4es
75.7am
63.7am
60.7cm
89.6
73.8
76.1am
80.6gm
73.3
Positive
Negative
Single
68.4
63.5asa
90.5.ca
80.8
74.0
Posétive
Negative
Singlo
Crese
70.9.1a
76.2
64.2Gs
61.9Gs
89.9.1
77.8
77,8
80.9
75.0
Table 10: Detailed results on ablation of FragCL with the average score on MoleculeNet.
Qc
BBBP
Tox21
ToxCast
Sider
Clintox
MUV
HIV
Bace
Avg
0.0
68.4
76.3
63.5
61.2406
90.5
76,3
75.0aa7
80.8
74.0
0.5
69.1 29.1
76.0=02
63.5
60.2
92.4112
77.3415
77.6117
80.
74.5
1.0
70.9=16
76.2-02
64.2
61.9
89.9a12
77.8406
77.8
80.9+10
75.0
2.0
69.2 29.2202
75.3
64.0aa.
60.3
92.012n
76.6114
77.2106
79.610s
74.3
17

---

Under review as a conference paper at ICLR 2023
F DISCUSSION ABOUT THE FRAGMENTATION STRATEGY
The extent cmmantic-preservation can be controlled by a more sophisticated fragmentation strat- egy to control. In Table 11, we report the performance with the fragmentation strategy 'disconnect non-ring C-C bond" to prevent altering the named groups such as ether, ester, and amide. In the table below, modified strategy obtains improvements on some downstream dataset (e.g.. 89.9 92.1 in Clintox). However, we note that our original (and extremely simple) strategy also shows comparable performance for downstream tasks, verifying the validity of 'disconnect non con-ring single bond' as semantic-preserving transformation.
Table 11: Detailed results on the fragmentation strategy of FragCL. with the average score on MoleculeNet.
Fragmentation strategy
BBBP
Tox21
ToxCast
Sider
Clintox
MUV
HIV
Bace
Avg
Single bond
70.9
76.2=a2
64.22us
61.92n9
89.9 39.9
77.8=as
77.8205
80.9
75.0
C-C single bond
70.4
75.6=0s
64.6=u6
61.5=12
92. 1a19
77.5=16
77.6=12
80.4
75.0
G RESULTS ON SPHERENET AS A 3D ENCODER
Our architectural choice of Table is for the fair comparison with GraphMVP (one of our main baselines). In Table 12, verify torsional angle reconstruction task is still meaningful even with a corsion-angle-awann 3D encoder, i.c., SphereNet (Liu et al., 2021). by improving the performance by 74.4 + 75.1.
Table 12: Detailed results SphereNet 3D encoder of FragCL with the average MoleculeNet.
Pretraining loss
BBBP
Tox21
ToxCast
Sider
Clintox
MUV
HIV
Bace
Avg
Lean
69.8
75.6gas
64.92n6
62.0gsa
89.9217
25.9
76.42as
80.3
74.4
Leange
71.5214
75.7
65.72na
61,3210
91.4214
78.122a
76. 116
80.7
75.1
18

---

Under review as a conference paper at ICLR 2023
H TRAINING PROCEDURE OF FRAGCL
Algorithm Fragment- based molecule Contrastive Learning (FragCL)
Input: Molecule graphs {9) 2D GNN fzo()), 3D GNN fao(-), and projection heads g20 920
Cao Arot (.), and (aba
1: for sampled bhini-batch of N molecule graphs (G do
2: for 1 N do
3:
(9mm F a.s..: + Fragmentation(G:
4:
// Obtain 2D graph representations
5:
120.1; T20. 120.1 (w() (g) fz²""),
20,1 +- mitn? (n/ x fwo(") + nf x fg)
7:
7z0.1, 220 720.11 20. 920 (r22.)))) 9zm (1 920 (r20. 02 (x2()))
Obtain 3D graph representations
9:
Ts 10 1s..: + f20(9))), fzo (g")
10:
Fo0, n/ta? (n² fm=(9') + n x (g"))
11:
2, 20 2z0. 2:0,1 + 930 900 (1 . 93n (1) 950 (r 9rn (130.)
12:
Contrastive objectives
define Can,eon, == LAT-Xone (Z20, ZoD (z Jitt (K 20.
14:
define L3p.cca.i = LuT tent (700 730, 1 Ju U 2gp, 720,i)
15:
define (1(20,30),com com.i == (Lrr (Z20.i Zz0, in [Zao.) 1==1) + Ler Lorrxxett (Z30. Ze0.5, [220. }==)))
16:
define Leca,i = L2D coa.1 + Lop.com.i [220,50)
Torsional angle reconstruction (s, u, v. around fragmented edge (u, v)
18:
Zrot. (3, U, 8, t) + 9cot [2. bao.u. hap...i bap...
19:
( U, v, f) + Jaba h20... bap.u.i b20.0.5 b20.1.
20:
Zabe, (t, U, u,
Jabe h20.0,6 ba h20,
h2o.s,
21:
Zabe, u.
Gane [brzo,e,33 b, h20,0,1; b2o.s.
22:
Cror, <- Lycu iroot, (8, u, n, t), Mros,s) Locel iZ U, u, 8), yror,))
23:
Laba,i <- Lace (8, u, v, t), Yeos,s) + Lace (aaaa (f, U. u, 8), year,s)
24:
define Lrorsion,; = Lior, + Luba,i
25:
Definition of FragCL loss define Leragot, = Leom, + o.Leorsonn. endfor
26:
27:
28:
define Cyrage. 1 Σn Leragah,
29:
Update fo(.) and g6 (:) minimize Lerage.
30:
endfor
31:
return fo(-)
19

---

Under review as a conference paper at ICLR 2023
I FRAGMENTATION EXAMPLES
We present several examples of fragmentation strategy based on our algorithm. Our fragmentation scheme almost preserves the chemically informative substructures, keeping the semantics of the original molecule.
Table 13: Fragmentation examples
Molecule
Bag of Fragments
20


Reviews:
[{"review_id": "v8Wj80fCMl7", "reviewer": "Reviewer_ov3H", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper develops a method for constructing representations of small molecules using contrastive learning. For a particular molecule, the molecule is split into two equal-sized fragments by breaking a single non-ring bond. The method attempts to maximize similarity between the original molecule’s representation and the weighted average of the representations of the two fragments. It also attempts to minimize similarity between the original molecule’s representation and the individual fragments on their own or the average of other molecules’ fragments. The authors also train two separate neural networks—a 2D GNN and a 3D GNN—and encourage representations learned between the two GNNs to be similar, and also add on a loss to predict torsional angle of the cut bond.\n\nThe authors use this method to pretrain and then attempt to predict several benchmark molecular properties, comparing to other similar methods of generating molecular representations. They demonstrate modest improvements in some of the tasks. ", "summary_of_the_review": "The work is interesting, and there are some modest improvements in some of the benchmarks shown. However, there is limited novelty in the technical contributions. Additionally, the method is very complex, combining 2 GNNs and 3 – 4 distinct types of loss functions. Even with the complex architectures and training, the improvements in the benchmarks are not particularly impressive. Combined with the numerous issues with clarity and incorrect justifications of the fragmentation scheme (one of the main focuses of the paper), it is very difficult to support its passage.", "strength_and_weaknesses": "The paper proposes a novel scheme for fragmentation (as opposed to previous fragmentation schemes) for contrastive learning on molecules. The scheme is interesting, and worth publicizing (at least on arXiv). The improvements on benchmarks are also nice to see.\n\nBelow are some weaknesses which I believe need to be addressed before publication:\n### The complexity of the objective suggests a lack of robustness\nIn addition to the fact that there are two separate GNNs, there are a lot of loss functions in the objective (i.e. contrastive loss for the 2D GNN, contrastive loss for the 3D GNN, contrastive losses between the two GNNs, and a complex loss function for the torsion angle which consists of both regression and multiclass classification objectives. Not only does this greatly increase training and inference time (and computational resource use), it also casts doubt on the robustness of the neural network. Especially because the improvements in downstream prediction tasks are only modest (if present at all), it suggests that all this extra complexity may be the result of overfitting to these specific benchmarks.\n\nFor example, it is difficult to justify predicting the torsion angle both as a regression problem and as a multiclass classification problem (over bins of the angle). What happens if the binned objective is removed?\n\nThe ablation analysis also suggests that each piece of this complex framework is necessary, even for the few modest improvements.\n### Incorrect chemical justification for the proposed fragmentation scheme\nThe authors attempt to justify the fragmentation scheme using several incorrect statements. For example, the claim is made that fragmenting a molecule along a single bond is “likely to preserve chemically informative substructures”. They also claim that “single bonds are not directly involved in reaction pathways in general”, and that the \"fragmentation scheme described in Section 3.1, satisfies $F(G\\_{i}) = F(G'\\_{i})\\cup F(G''_{i})$ with a high probability” (where $F(G)$ is the set of functional groups in $G$). These statements are all dubious if not incorrect. \n\nSingle bonds are oftentimes highly important in reaction pathways. Nucleophilic attacks such as SN1 and SN2 rely on the breaking of single bonds almost ubiquitously, not double or triple bonds as the authors claim. If anything, single bonds are perhaps even more likely to participate in reaction chemistry (compared to double or triple bonds) because the bond energy is usually lower.\n\nBreaking single bonds also can very easily break functional groups. For example, consider dibenzyl ether. Based on the fragmentation scheme proposed by the authors, this molecule would be split into two fragments which completely destroy the ether functional group. Groups such as esters, amides, and ketones would also easily be split up. The statement that $F(G\\_{i}) = F(G'\\_{i})\\cup F(G''_{i})$ with high probability should be quantified over the dataset if it is true (although I have very strong suspicions that it is not).\n\nFurthermore, the distinction between single and multiple bonds in organic molecules is not as clear as the authors suggest. For example, consider a highly conjugated alkene capped with aldehydes (or another resonant functional group). Although the graph representation of this molecule will have several single bonds, the distinction between the single and double bonds in reality are actually minimal, as the entire structure is in resonance (which would be broken by fragmentation). Thus, the choice to fragment molecules by single bonds only is not particularly well justified.\n\nIt is also good to note that there are many molecules which would not satisfy the fragmentation scheme suggested by this paper, such as benzene, naphthalene, anthracene, etc. These molecules could not be fragmented at all using this scheme.\n### Areas lacking in explanation, clarity, or typos\n- In the penultimate sentence of Section 3.1, $G''_{2D}$ is a typo, I think\n- Eq. 10 – 11 should specify what the $h$s mean. I assume they are the node representations from the last layer of the 2D GNNs\n- In Table 1, the Avg column is not informative and should be removed. These are very different benchmarks with varying difficulties, and averaging over the scores doesn’t make sense. After all, a model could get a very high Avg just by optimizing very specifically for a single very easy task.\n- The bolding in Table 1 and 2 is not clear. Why are there multiple numbers bolded in a single column?\n- In Section 4.3, clarify what the loss value is. Is this just the NT-Xent value? If so, this metric does not make much sense because the fragmentation scheme is different between different experiments, and so the NT-Xent values will be based on different inputs and therefore are not comparable across ablation experiments\n- In Appendix A, the definition of $r^{mix}_{i}$ contradicts the main text\n- There are typos in the title of Appendix B, and in the Table 9 caption\n- What is the form of the projectors $g$ which map from node embeddings to torsional angle? I assume they are single linear projection heads (i.e. a single linear layer) on the concatenation of node embeddings\n- Once pretraining is done, how are the final tasks (e.g. the ATOM3D regression tasks) trained? Is it another linear projection head on the average node representations? This doesn’t seem to be explained anywhere\n- The computation of the score of these benchmarks should be explained: how are these scores computed?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eRM3fP2GTl_", "reviewer": "Reviewer_EnEZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThe paper proposes a fragment-based contrastive learning algorithm for learning the discriminative representations of the molecules. They propose to decompose a molecule into two fragments by breaking a single bound (none ring). The bound is chosen such that two fragments have a similar number of atoms. Then the complete bag of fragments of the molecule is considered as a positive view while the incomplete or the complete bag of fragments of the other molecule is considered as a negative view. They also  Incorporated 3D structure by including a reconstruction loss on the torsional angle. ", "summary_of_the_review": "Overall, the paper is clearly written and easy to follow, however, I have some detailed questions as following:\n\n1. How often does the assumption \" breaking a none-ring single bound preserve the  chemically informative substructures\" \n\n2, Is the negative view which is the complete fragments of another molecule really useful, seems to be a very obvious negative view that is easy to discriminate.\n\n3. It is not very clear here: \"Note that the semantic of a fragment g_i is ensured to be sufficiently different from the original molecule g'_i  since ....... \".  According to the paper, F(g'_j) and F(g''_j)  are complementary, not necessarily similar. And I would think also that the negative view which is constructed by the incomplete bag of fragments is not hard to discriminate from science dropping half of the molecule could change the functional group of the original molecule drastically. \n\n4. When looking at the experimental results in Table1, it feels to me that the model did not have much advantage over the other models when working on 2D representation,  but it shows better performance when trained on both 2D and 3D. Could it be possible that the improvement comes from the fact that the model also tries to incorporate the reconstruction of the torsional angle but less due to introduced strategies about constructing a positive and negative view of the molecule? \n\n5. The ablation study section is a bit confusing to me, so the author first used the incomplete fragment as a positive view, and on top of that used the complete fragments as a positive view, and then on top of that used the incomplete fragment as a negative view? Also, it would be interesting to see what is the effect of the complete fragments of another molecule as a negative view.\n\n6. As table 3 shows, when all settings are kept the same and only changing where we cut seems not to have such a big difference in terms of performance,e 72.1 to 72.4 (on average), instead adding the loss on torsion angle seem to improve a lot (74 to 75).\n\n7. It would be great to see what the value of alpha is in the best model, and what happens if we set alpha to zero which means there is no reconstruction of the torsional angle", "strength_and_weaknesses": "Strength:\nThe paper addresses an interesting problem such as how to define a good positive and negative view when we apply contrastive learning on the molecule. They also propose an interesting idea that seems to be an easy direct way to incorporate 3D information about molecules. \n\nWeakness: \nExperimental results seem weak especially when trained on 2D molecule graphs. \nCompare to the existing baselines, the proposed scheme to construct a negative and positive view of molecules does not seem to have many advantages.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "X5rkGIB1vH4", "reviewer": "Reviewer_yCZe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors develop a graph neural network model that breaks a module up into fragments over non-ring single bonds, and trains a both a contrastive bag-of-fragments objective as well as prediction of the torsion angle of the respective broken bond.", "summary_of_the_review": "I find the approach novel, pragmatic, and of general interest. I also find the work well written and easy to understand.", "strength_and_weaknesses": "I appreciated the number of baselines presented, and I think the proposed model results are rather good.\n\nI really like that the objective is straightforward and pragmatic. The approach is straightforward and more generally applicable. Since the model inputs and objectives are quite interpetable, I think Section E (Table 9) is useful as an ablation study.\n\nQuestions: \n\nWhat is the distribution of 3D bond angles used in the objective? Was only a single bond angle used, and where was this generated? Would it be possible to learn using the energetically favorable distribution of bond angles?\n\nThe loss for the torsion angle is interesting and pragmatic. However, by using a binned loss for the torsion angles, it is not radially symmetric, and may not be ideal. Can the angle loss be parameterized with a von Mises distribution?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SzP6RA65ZN", "reviewer": "Reviewer_NCgJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper decomposes the 2D and 3D molecular graphs into fragments. The pretraining objective is composed of the fragment-based CL on 2D and 3D graphs, as well as a torsion angle reconstruction from 2D to 3D.\n", "summary_of_the_review": "I think the main motivation of this paper is reasonable, but the experimental set-ups are not valid. The authors may double-check the SE(3)-invariant/equivariant modeling, and the QM9 dataset.", "strength_and_weaknesses": "Strength:\n- The existing works on utilizing the domain knowledge to help molecule representation is either focusing on the 2D structure (like fragments) or general 3D and 2D structure. This paper introduces using the fragments from two views, which is interesting and promising.\n- This paper further introduces using the torsion angle reconstruction, which can help the molecule representation intuitively.\n\n\nWeakness:\n- There exists information leakage between the SSL objective and backbone model.\n  - The authors are using SchNet for the geometric backbone. It is an SE(3)-invariant model, using the pairwise distance information, and adding torsion angle to it has been expected to be helpful. This has been verified in GemNet[1] and SphereNet[2].\n  - Meanwhile, there are more advanced SE(3)-invariant GNNs (like GemNet[1] and SphereNet[2]). They all explicitly use the torsion angles directly in the backbone models. Other SE(3)-equivariant GNNs are also preferred, like TFN [3], since the angle information is simultaneously modeled in the spherical harmonics basis.\n  -  In other words, using the SchNet as backbone is not sufficient to support the effect of using torsion angle reconstruction. Now the information released in the experiments is that for geometric model with distance modeling only, adding torsion angle pretraining is beneficial. This is different from the main claim of the paper, because this may fail if the backbone models [1,2,3] explicitly/implicitly encodes the torsion angle.\n  - For improvements, the authors should use more advanced backbone models with torsion angle modeling, either implicitly or explicitly. Then it can better verify the effectiveness of using torsion angle for pretraining.\n\n- The experiments are not solid for the 3D downstream.\n  - First, the 7 tasks are actually from QM9, with 12 tasks and 130K molecules.\n  - So the pretraining has 50K molecules, and the downstream has 130K molecules. I don’t think this is a valid pretraining and finetuning setting.\n  - The data preprocessing has been widely used on QM9 (SchNet, DimeNet, SphereNet) etc, but the authors are not applying that. So this result can be hardly compared with other existing works.\n  - There are 12 tasks, I’m wondering why only 7 tasks are reported here?\n  - Thus, combined all the 4 points above, I am not convinced by the validness of QM9 downstream results.\n\n- Questions on 2D downstream. I found that authors are reporting results on 2D-FragCL, FragCL, and 3D InfoMax, and the other results are from GraphMVP paper, which is fine. But I noticed that the main benefits between the newly reported results and existing baselines are from Clintox, which is >0.1 ROC-AUC better. Other than this, the improvements are quite modes, within 0.01 ROC-AUC. Do the authors have any hypothesis for this? Could it be the hyper-parameter issue (seed, learning rate, etc)?\n\n[1] Gasteiger, Johannes, Florian Becker, and Stephan Günnemann. \"Gemnet: Universal directional graph neural networks for molecules.\" Advances in Neural Information Processing Systems 34 (2021): 6790-6802.\n\n[2] Liu, Yi, et al. \"Spherical message passing for 3d graph networks.\" arXiv preprint arXiv:2102.05013 (2021).\n\n[3] Thomas, Nathaniel, et al. \"Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds.\" arXiv preprint arXiv:1802.08219 (2018).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject